Feat/interledger refactor

use bigchaindb mixin for examples
move ledger under accounts
multi-load assets of accounts on init
composite accountList/accountDetail
escrow endpoint (doesnt really belong here, but hey)
avoid adding ledger listeners on account switch

Dependencies:

[ ] #18

overruled by #18


feat: add form agent notebook

remove keys
Thank you for contributing to LangChain!

[ ] PR title: "package: description"

Where "package" is whichever of langchain, community, core, experimental, etc. is being modified. Use "docs: ..."  for template changes, "infra: ..." for CI changes.
Example: "community: add foobar LLM"

[ ] PR message: Delete this entire checklist and replace with

Description: a description of the change
Issue: the issue # it fixes, if applicable
Dependencies: any dependencies required for this change
Twitter handle: if your PR gets announced, and you'd like a mention, we'll gladly shout you out!

[ ] Add tests and docs: If you're adding a new integration, please include

a test for the integration, preferably unit tests that do not rely on network access,
an example notebook showing its use. It lives in docs/docs/integrations directory.

[ ] Lint and test: Run make format, make lint and make test from the root of the package(s) you've modified. See contribution guidelines for more: https://python.langchain.com/docs/contributing/

Additional guidelines:

Make sure optional dependencies are imported within a function.
Please do not add dependencies to pyproject.toml files (even optional ones) unless they are required for unit tests.
Most PRs should not touch more than one package.
Changes should be backwards compatible.
If you are adding something to community, do not re-import it in langchain.

If no one reviews your PR within a few days, please @-mention one of baskaryan, efriis, eyurtsev, hwchase17.

could you provide a description of what this pr does

few linting errors that need to be resolved, can check them locally by running
make format
make lint

this is cool but should be a langgraph example/tutorial


Use Node v14.19.0!!

Because node-gyp v3.8.0 is a dependency, you have to use Node.js 14. The LTS for node v14 is v14.19.0. Please add this in the README. I love this project and I hope this solves any issues. Cheers!

Hey there! Why has this been closed without adding it to the project's README?
I've been struggling with errors for almost 2 hours and this fixed it, thank you so much!


wrong parameter name in RestApiExample.java for resource query

on line 126
resourceTypes= must be resource=

not working:
URL methodURL =new URL(baseUrl,"resources?resourceTypes="+ resourceTypes+
"&attributeFilter="+attributeFilter);

correct working:
URL methodURL =new URL(baseUrl,"resources?resource="+ resourceTypes+
"&attributeFilter="+attributeFilter);

Original issue reported on code.google.com by<EMAIL_ADDRESS>on 21 Aug 2014 at 1:45

Inspected the code should work now. Please Confirm


Add register storage

Closes: #???
Description
Add register storage.
AllocateSlabIndex shouldn't be needed in this use case. If it is we will add it later.

For contributor use:

[ ] Targeted PR against master branch
[ ] Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
[ ] Code follows the standards mentioned here.
[ ] Updated relevant documentation
[ ] Re-reviewed Files changed in the Github PR explorer
[ ] Added appropriate labels

optimization: owner is always the same, not using it for keys have two benefits: 1. less storage used, 2. better balancing of LSM trees

@zhangchiqing @peterargue Sorry to do this, but while integrating it with the rest, I noticed that in all of the redesigns I forgot that the register storage should implement a types.StorageProvider so I split out a cached snapshot into a separate struct. Which also makes the delta a bit simpler since it does not have to think about caching.
Also did some renaming as I didn't like it.


[SPARK-37509][CORE] Improve Fallback Storage upload speed by avoiding S3 rate limiter

What changes were proposed in this pull request?
This PR aims to improve Fallback Storage upload speed by randomizing the path in order to avoid S3 rate limiter.
Why are the changes needed?
Currently, Fallback Storage is using a single prefix per shuffle. This PR aims to randomize the upload prefixes even in a single shuffle to avoid S3 rate limiter.

https://aws.amazon.com/premiumsupport/knowledge-center/s3-request-limit-avoid-throttling/

Does this PR introduce any user-facing change?
No. This is used internally during the runtime.
How was this patch tested?
Pass the CIs to verify read and write operations. To check the layout, check the uploaded path manually with the following configs.
spark.decommission.enabled                          true
spark.storage.decommission.enabled                  true
spark.storage.decommission.shuffleBlocks.enabled    true
spark.storage.decommission.fallbackStorage.path     file:///tmp/fallback/

Start one master and worker. Connect with spark-shell and generate shuffle data.
scala> sc.parallelize(1 to 11, 10).map(x => (x % 3, 1)).reduceByKey(_ + _).count()
res0: Long = 3

Invoke decommission and check. Since we have only one worker, the shuffle data go to the fallback storage directly.
$ kill -PWR <CoarseGrainedExecutorBackend JVM PID>
$ tree /tmp/fallback
/tmp/fallback
└── app-20211130135922-0001
    └── 0
        ├── 103417883
        │   └── shuffle_0_7_0.data
        ├──<PHONE_NUMBER>
        │   └── shuffle_0_4_0.data
        ├──<PHONE_NUMBER>
        │   └── shuffle_0_7_0.index
        ├──<PHONE_NUMBER>
        │   └── shuffle_0_6_0.index
        ├──<PHONE_NUMBER>
        │   └── shuffle_0_3_0.data
        ├──<PHONE_NUMBER>
        │   └── shuffle_0_2_0.index
        ├──<PHONE_NUMBER>
        │   └── shuffle_0_8_0.data
        ├──<PHONE_NUMBER>
        │   └── shuffle_0_9_0.index
        ├──<PHONE_NUMBER>
        │   └── shuffle_0_6_0.data
        ├──<PHONE_NUMBER>
        │   └── shuffle_0_1_0.data
        ├──<PHONE_NUMBER>
        │   └── shuffle_0_4_0.index
        ├── 227534966
        │   └── shuffle_0_2_0.data
        ├── 266114061
        │   └── shuffle_0_3_0.index
        ├── 413944309
        │   └── shuffle_0_5_0.index
        ├── 581811660
        │   └── shuffle_0_0_0.data
        ├── 705928743
        │   └── shuffle_0_5_0.data
        ├── 713451784
        │   └── shuffle_0_8_0.index
        ├── 861282032
        │   └── shuffle_0_0_0.index
        ├── 912764509
        │   └── shuffle_0_9_0.data
        └── 946172431
            └── shuffle_0_1_0.index

Test build #145776 has started for PR 34762 at commit 7cdcb44.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50249/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50249/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50249/

Could you review this please, @viirya ?

Test build #145776 has finished for PR 34762 at commit 7cdcb44.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145776/

lgtm

Thank you so much, @viirya !

Only just seen this.
How much throttling what you actually seeing, and, assuming the s3a client, have you set directory marker retention to keep? It is often the actual attempted delete of dir markers which trigger the problem ... Effectively it is a form of write amplification.
I'm actually thinking that feature to go into s3a this year should be configurable rate limiting through the guava RateLimiter; I'm using this in the abfs committer to keep committer io below limits where throttling starts to cause problems with renames. This is all per process; things like random filenames are still going to be critical to spread load on s3.

I didn't set any configurations. When a customer does not make a special request, the pre-defined throttling is applied by default (3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per). In the worst notebook scenario, 3500 executors may start to decommission at the same time and it causes throttling.

OK. you should try enabling directory marker retention everywhere and see what that does to your work. The auditing stuff in 3.3.2  will help you identify which jobs and operations are generating the IO.

They are orthogonal dimension from this PR, aren't they?
This PR will give us benefit on top of whatever Apache Hadoop delivers, @steveloughran .

yeah, you should do both. this is best as it will spread across all shards a bucket has


Alpine Expression Error: undefined for form within livewire component

Package
filament/filament
Package Version
v3.1.42
Laravel Version
10.40.0
Livewire Version
v3.3.5
PHP Version
8.2.12
Problem description
Alpine Expression Error: undefined for form within livewire component.
I get error on the picture, bcs of the error the form doesnt work and submiting refreshes the page. (I am calling this component inside modal)

<?php

namespace App\Livewire;

use Filament\Forms\Contracts\HasForms;
use Livewire\Component;
use Filament\Forms\Components\RichEditor;
use Filament\Forms\Form;
use Filament\Forms\Concerns\InteractsWithForms;

class Chat extends Component implements HasForms
{
    use InteractsWithForms;

    public ?array $data = [];

    public function mount(): void
    {
        $this->form->fill();
    }

    public function render()
    {
        return view('livewire.chat');
    }

    public function form(Form $form): Form
    {
        return $form
            ->schema([
                RichEditor::make('body')
            ])
            ->statePath('data');
    }
    public function send (): void
    {
        dd($this->form->getState());
    }
}

Expected behavior
Bind form field to $data array.
Steps to reproduce
https://filamentphp.com/docs/3.x/forms/adding-a-form-to-a-livewire-component
Reproduction repository
https://github.com/soixt/demo-for-testing
Relevant log output
Alpine Expression Error: undefined

Expression: "richEditorFormComponent({
                            state: $wire.$entangle('data.body', false),
                        })"

I've the same problem... Basically, what I'm doing is creating a new RichEditor component within a form. It works perfectly when I'm using complete resources (with CRUD pages). However, for some reason – I don't know if it's a bug or if I'm doing something wrong – when using it in a simple resource generated with php artisan filament:resource TestResource --simple,' it doesn't work…
public static function form(Form $form): Form
    {
        return $form
            ->schema([
                RichEditor::make('body')
                    ->label('Description')
                    ->required(),
            ]);
    }

Returns: Uncaught Could not find Livewire component in DOM tree on console

F*ck, I figure it out....  I had two elements in my livewire component on root level.


Some small fixes

I'm using the amazing tool in a personal project and I implement some fixes I needed.
The fixes are

Add static keyword on member functions
Fix static keyword string
Fix inheritance string.
Close file where c++ code is written in the destruct of the cpp code generator (pease, check if this is ok)

I would like to add also the possibility of adding custom types to classes, like using/ typedef. What would be the best methodology?

Thank you for your contribution!
Is it okay if I ask you to add a small unit test?
The project suffers from a lack of meaningful test cases.
You can add class CppUsing, e.g.
using = CppArray(name='vec', value='std::vector<int, alloc<int>>') which can be rendered as  using vec = std::vector<int, Alloc>;`
In the class declaration it can be rendered between enums and nested classes.

@rafaelrojasmiliani If you don't have time for tests now it's fine of course
Just tell me, I will accept your PR anyway

I Added a small test on the file creation. But I don't have an idea of how to test for a code generation.

@rafaelrojasmiliani
You can take a look at existing tests
Creating C++ file:
https://github.com/yuchdev/code_generator/blob/master/tests/test_cpp_file.py
Creating in-memory-writer
https://github.com/yuchdev/code_generator/blob/master/tests/test_cpp_function_writer.py

I think it's fine, I approve the PR


Cannot enter text in character editor

Describe your bug here.
I can type the character name and press enter, but when I go to enter the pose name it does not let me enter text. Psych 0.52h
Command Prompt/Terminal logs (if existing)
No response
Are you modding a build from source or with Lua?
Lua
What is your build target?
Windows
Did you edit anything in this build? If so, mention or summarize your changes.
I did not.

Duplicate of #6623


Static methods do not work for CaseType interfaces

Static methods for interface work fine for Java8 backend
shared interface NonCaseInterface {
    shared static void method() {}
}

But static methods for interface of caseTypes produces error: "Error:ceylon: Ceylon backend error: method does not override or implement a method from a supertype"
shared interface CaseInterface of case1 {
    shared static void method() {}
}
object case1 satisfies CaseInterface {}

Fixed, thanks for reporting @aedeph.


update devops.md

Describe your changes
I have seen that some of the peoples only speak hindi. So for that I have tried to make some changes.
Related issue number or link (ex: resolves #issue-number)
Checklist before opening this PR (put x in the checkboxes)

[x] This PR does not contain plagiarism

don’t copy other people’s work unless you are quoting and contributing it to them.

[x] I have signed off on all commits

signing off (ex: git commit -s) is to affirm that commits comply DCO. If you are working locally, you could add an alias to your gitconfig by running git config --global alias.ci "commit -s".

This PR contains irrelevant commits.
Please refer to https://github.com/cncf/glossary/pull/1550#issuecomment-1367175073.


Demo codes and files?

bamgineer/demo_codes/wes_example.sh is a blank file

I was also wondering if there are any demo files to try out bamgineer with?  The directory bamgineer/tcga_experiments/inputs/ only contains bed files of genes.  As per the scripts used in  "bamgineer/tcga_experiments/scripts/generation/T1", I can not find any of the files mentioned:  "inputs/vcfs/normal_varscan.vcf", "inputs/bams/Normal.bam" or "inputs/beds/exons.bed"

This is the contents of debug.log, not sure what to look for.  Anything stand out to you?
pipeline started!
 --- Initializing input files  ---
--- initialization complete ---
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr1_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr2_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr3_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr4_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr5_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr6_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr15_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr7_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr9_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr8_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr13_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr14_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr19_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr16_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr10_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr17_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr12_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr18_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr11_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr20_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr21_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr22_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrX_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrY_exons_in_roi.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr1_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr2_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr3_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr5_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr6_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr7_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr4_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr9_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr18_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr8_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr10_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr11_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr21_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr12_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr13_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr14_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr15_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr16_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr17_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr19_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr20_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr22_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrX_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrY_het_snp.bed.bed does not exist!
 ***** pipeline finished in 0.0 minutes *****

I suspect some of the created VCF files (in phasedvcfs directory) do not have variants in them( the ".bed.bed" extension in the log file should be fixed but I don't think that is causing the issue). Can you confirm they actually have variants in them? The empty VCF file will lead to blank snp_het file and hence the program will exit without creating the Bam files. I'll have a closer look at the demo example and let you know if I can figure it out.

I had a closer looks and it seems that the normal_het.vcf file that is generated is not sorted. When I run Beagle I get the error "java.lang.IllegalArgumentException: markers not in chromosomal order". However this error will only be displayed on the console and not recorded in the log file. Running Beagle will generate phased_beagle.vcf that contains only the "header". So you need to sort the VCF (after removing Indels) and re-run the code.

good catch; seems to stem from the input_preparation.md line:
tabix genome.vcf.gz chr21 > chr22.vcf
and
 bedtools merge -i chr22.gt50.bedgraphh > exons_chr22.bed
The exons.bed was only being made on chr21 due to the typo on .bedgraphh; and chr22.vcf was being created by tabix-selecting chr21.
I made those fixes, removed the indels, sorted the vcf, dedupped using the awk script (and double checked with a bash command), and still get no bam files in my output
## Pre-filtered line-count
(bamgineer_test)Badger:81318Mb [quever@samwise inputs]$ wc -l normal_het.vcf
13898 normal_het.vcf

## Awk Filter
(bamgineer_test)Badger:81570Mb [quever@samwise inputs]$ awk '!seen[$0]++' normal_het.vcf > in_dedup.vcf
(bamgineer_test)Badger:81318Mb [quever@samwise inputs]$ wc -l in_dedup.vcf
13898 in_dedup.vcf

## Bash Filter
(bamgineer_test)Badger:81317Mb [quever@samwise inputs]$ sort in_dedup.vcf  | uniq | wc -l
13898

For your reference, these are the commands I've updated to using after remaking exons.bed
vcftools --vcf chr21_22_het_genome.vcf --bed exons.bed --out normal_het --recode
vcftools --vcf normal_het.recode.vcf --remove-indels --recode --recode-INFO-all --out snps_only
vcf-sort snps_only.recode.vcf > snps_only.recode.sort.vcf
awk '!seen[$0]++' snps_only.recode.sort.vcf > snps_only.recode.sort.dedup.vcf
mv snps_only.recode.sort.dedup.vcf normal_het.vcf

Running the command line version of beagle threw no java.lang.IllegalArgumentException: markers not in chromosomal order error for me, so I think that's working correctly.  The normal_het_phased.vcf.gz contains information.  Maybe it does have something to do with the missing/curious .bed.bed files?  I'll tinker around with that and see if it addresses the issue

I figured what is going on here. By default, I used to add the event as an extension to the copy number event in the previous version. For instance, the file "cnv1.amp" used to denote the amplification event. Now the name of your cnv here is "gain.bed" (as opposed to previous version where it was "cnv.gain"). So when Bamgineer split this file, it thinks that the extension (".bed") is the event and hence you get ".bed.bed" in the output. However, since in the new version, I only use one bed file, and the type of event is inferred from the absolute copy number, this won't be necessary anymore. To get around it please just change your gain.bed to cnv.gain for now. I will update and push the right code shortly.

I think I found the bug (line 255 utils.py that assumed a specific format for cnv event files). Pushed the changes. Could you pull and retry please?

Looks like it's getting closer.  After I pulled the new repo, the run_example1.sh script runs a bit longer before it gets bombarded with "Exception!"
Parameters as interpreted:
	--vcf /mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/phasedvcfdir/hap2_het.vcf
	--thin 50
	--out /mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/phasedvcfdir/hap2_het_filtered
	--recode

After filtering, kept 0 out of 0 Individuals
Outputting VCF file...
After filtering, kept 5424 out of a possible 6905 Sites
Run Time = 0.00 seconds
Exception!
Exception!
Exception!
...
Exception!
Exception!
Exception!
Exception!

The debug.log is as follows  (not quite sure what I should be looking for here):
pipeline started!
 --- Initializing input files  ---
--- initialization complete ---
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr1_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr2_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr3_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr4_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr5_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr6_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr15_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr7_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr9_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr8_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr13_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr14_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr19_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr16_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr10_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr17_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr12_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr18_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr11_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr20_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr21_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr22_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrX_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrY_exons_in_roi.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr1_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr2_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr3_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr5_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr6_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr7_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr4_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr9_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr18_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr8_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr10_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr11_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr21_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr12_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr13_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr14_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr15_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr16_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr17_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr19_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr20_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr22_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrX_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrY_het_snp.bed.bed does not exist!
 ***** pipeline finished in 0.0 minutes *****
pipeline started!
 --- Initializing input files  ---
--- initialization complete ---
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr3_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr13_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr1_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr14_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr6_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr4_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr7_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr20_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr2_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr15_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr5_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr10_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr17_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr12_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr11_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr22_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr8_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr21_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrX_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr19_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr16_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrY_exons_in_roi.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr9_exons_in_roi.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr3_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr18_exons_in_roi.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr1_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr2_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr4_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr5_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr6_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr7_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr15_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr8_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr9_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr10_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr11_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr12_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr13_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr14_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr17_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr16_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr19_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr18_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr20_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr21_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr22_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrX_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrY_het_snp.bed.bed does not exist!
 ***** pipeline finished in 0.0 minutes *****
pipeline started!
 --- Initializing input files  ---
--- initialization complete ---
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr4_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr2_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr5_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr14_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr6_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr3_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr18_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr9_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr10_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr8_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrX_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr11_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrY_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr1_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr21_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr17_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr7_exons_in_roi.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr22_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr13_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr15_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr16_exons_in_roi.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr20_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr12_exons_in_roi.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr19_exons_in_roi.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr4_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr15_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr6_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr12_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr9_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr2_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr13_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr7_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr11_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr10_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr14_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr8_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr1_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr21_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr16_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrX_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr3_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr17_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr5_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr22_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr19_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr20_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrY_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr18_het_snp.bed.bed does not exist!
 ***** pipeline finished in 0.1 minutes *****
pipeline started!
 --- Initializing input files  ---
--- initialization complete ---
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr2_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr1_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr5_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr6_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr3_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr4_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr7_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr18_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr9_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr13_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr21_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr10_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr12_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr15_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr11_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrY_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr17_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr8_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr20_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr14_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrX_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr16_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr19_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr22_exons_in_roi.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr4_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr2_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr1_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr5_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr6_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr10_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr7_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr8_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr11_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr22_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr13_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr9_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr12_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr17_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr14_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr21_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrX_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr15_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr3_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr19_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrY_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr18_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr16_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr20_het_snp.bed.bed does not exist!
 ***** pipeline finished in 0.1 minutes *****
pipeline started!
 --- Initializing input files  ---
--- initialization complete ---
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr2_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr1_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr4_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr13_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr6_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr15_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr7_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr20_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr9_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr19_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr5_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr16_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr12_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr11_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr3_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr18_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr17_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr14_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr22_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr10_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrX_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr21_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrY_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr8_exons_in_roi.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr2_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr4_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr1_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr9_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr8_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr3_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr7_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr6_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr16_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr12_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr11_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr21_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr14_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr5_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr10_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr18_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr19_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr20_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr17_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr13_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr15_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrY_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr22_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrX_het_snp.bed.bed does not exist!
 ***** pipeline finished in 0.1 minutes *****
pipeline started!
 --- Initializing input files  ---
--- initialization complete ---
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr3_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr4_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr2_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr14_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr5_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr6_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr7_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr13_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr19_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr1_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr16_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr10_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr22_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr12_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr15_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr17_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr18_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr9_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr20_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr8_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrX_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrY_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr11_exons_in_roi.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr21_exons_in_roi.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr3_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr6_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr1_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr5_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr2_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr8_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr4_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr7_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr11_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr10_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr12_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr13_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr9_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr15_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr20_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr17_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr18_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr19_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr22_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrY_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr14_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr16_het_snp.bed.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr21_het_snp.bed.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrX_het_snp.bed.bed does not exist!
 ***** pipeline finished in 0.1 minutes *****
pipeline started!
 --- Initializing input files  ---
--- initialization complete ---
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr1_exons_in_roigain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr4_exons_in_roigain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr2_exons_in_roigain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr3_exons_in_roigain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr5_exons_in_roigain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr7_exons_in_roigain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr6_exons_in_roigain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr13_exons_in_roigain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr12_exons_in_roigain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr8_exons_in_roigain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr10_exons_in_roigain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr14_exons_in_roigain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr11_exons_in_roigain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr15_exons_in_roigain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr16_exons_in_roigain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr9_exons_in_roigain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr17_exons_in_roigain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr18_exons_in_roigain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr20_exons_in_roigain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr19_exons_in_roigain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr22_exons_in_roigain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrX_exons_in_roigain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrY_exons_in_roigain.bed does not exist!
find_roi_bam complete successfully for chr21gain
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr2_het_snpgain.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr1_het_snpgain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr3_het_snpgain.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr5_het_snpgain.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr4_het_snpgain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr15_het_snpgain.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr9_het_snpgain.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr8_het_snpgain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr7_het_snpgain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr6_het_snpgain.bed does not exist!
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr12_het_snpgain.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr10_het_snpgain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr13_het_snpgain.bed does not exist!
___ Bamgineer main engine started ___
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr11_het_snpgain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr14_het_snpgain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr16_het_snpgain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr17_het_snpgain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr18_het_snpgain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr20_het_snpgain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr19_het_snpgain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chr22_het_snpgain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrX_het_snpgain.bed does not exist!
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs/LUAC/haplotypedir/chrY_het_snpgain.bed does not exist!
+++ coverage ratio for: chr21_roigain.sorted.bam: 0.278378637301
not enough reads for chr21_roigain.sorted.bam
 ***** pipeline finished in 3.4 minutes *****

Never mind those "Exceptions"; they're printed out everytime there is a problem with an individual read (line 362 method.py). I will delete the print line as it really isn't informative. Perhaps I can write the read names for which the exception happens to a seperate file. However the line before the last (not enough reads for ...), shows that Bamgineer was not able to "re-pair" enough reads to generate new ones to obtain the desired CNV state (for CNV=3 Bamginner needed ~0.5 coverage for new reads compared to the old, but found 0.27 and hence complains). In that case, one needs to put a less stringent criteria on new paired read generation.
That said, I suspect there won't indeed be enough read for this example. I will try running it with the examples in the tutorial and see if I get the same result. Anyhow, I do recommend running it on the actual Bam/Bed/regions and ref files as for the the case explained here, I use a lot of assumption and as explained it's  mainly to perform a  dry-run test. Will keep you posted.

It's good to know that Bamgineer ran to completion correctly then!!
However, I'm not too clear on this answer (is BioRxiv article updated? is that the most recent version of this manuscript?)?  When you say Bamgineer needs 0.5 coverage, are you talking about allelic fraction?  Not sure what 0.5 coverage means... GATK DepthofCoverage shows that Chr21 and Chr22 are sequenced to an average of ~32x and ~36x respectively.
Also, not quite sure what you mean by "running it on the actual bam/bed/regions and ref files as for the case explained here".  I ran it on the PGP chromosomes that you suggested if that's the case you were talking about?

Hi Soroush, can you comment on your last reply?  Would it be feasible to run bamgineer on a PGP sample or is there just not enough coverage to run with your tool?

Hi Rene, as I mentioned in my data preparation document, The dataset used from PGP does not provide the target regions (i.e. exons) used to generate the Bam files, and as such I came up with an approximation approach to generate exon bed file by making a bed file from all contiguous regions above a certain coverage threshold. I suspect, this may cause the problem with coverage. That's why I did suggest, it's far more reliable to run Bamgineer on your costum Bam file with known target regions and so on.
Also w.r.t my comment about "X0.5" additional coverage, I meant since the input CNV bed file has the absolute CN of 3, the total number of newly generated reads will have to be 1.5 times of the original coverage(i.e ~1.5x32 for chr21 and 1.5x36 for chr22). So if all the steps are done correctly, the "not enough read" message from Bamgineer means that Bamgineer was unable to find enough reads to "re-pair" and regenerate new reads (needed 0.5 times total but found 0.28 according to message) .
Anyhow, I will still follow the step-by-step guide in data prep section and see if I can reproduce your coverage problem. Will keep you posted.

Hey Soroush,
I'm starting to look back into bamgineer and we intend on running bamgineer on PGP samples due to the large number of normals available to simulate different types of tumors.  Do you suggest that there is not enough coverage in these samples to be usable for Bamgineer?
I've re-ran the pipeline with exons.bed using an absolute CN of 2 and one with an absolute CN of 1:
For the absolute CN of 1, it throws a division by zero error in the debug.log:
Exception in find_roi_bam float division by zero
Traceback (most recent call last):
  File "/mnt/work1/users/home2/quever/git/bamgineer/src/methods.py", line 499, in implement_cnv
    ratio_kept = float(countReads(bamsortfn)) / float(countReads(bamfn))
ZeroDivisionError: float division by zero
 ***** pipeline finished in 2.4 minutes *****

For the absolute CN of 2, it shouldnt need to add any new reads so we wouldn't run into the same error where not enough reads were present?  However, I still get the division by zero error:
Exception in find_roi_bam float division by zero
Traceback (most recent call last):
  File "/mnt/work1/users/home2/quever/git/bamgineer/src/methods.py", line 499, in implement_cnv
    ratio_kept = float(countReads(bamsortfn)) / float(countReads(bamfn))
ZeroDivisionError: float division by zero
 ***** pipeline finished in 2.5 minutes *****

In neither case do I get a bam outputted.

Hi Rene, Can you please provide the exact CNV file(s) you are using as the input?

PDIR='/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test'

[quever@samwise scripts]$ cat ${PDIR}/inputs/cnv.bed
chr21	30227447	         47076809	1

Thanks, I will try to replicate the error and will let you know.

Hi Rene,
I tested the code and there was a typo in the loss function. Please pull from master branch and try re-running the script. Please note that I commented out the line where Bamgineer deleted temporary bam files "..outputs/tmpbams/" for debugging purposes. The folder can get too large and should be deleted (line 623 in methods.py).
Regarding CN=2 (LOH) event, I will perform some additional tests and commit the latest code within the next couple of days. Once tested I will add back the commented line to delete tmpbams folder. Anyhow, you should now see the output bam file generated in the "finalbams" directory.
  You removed importing handle from handlers, but you still call the object on line 562 of methods.py
/mnt/work1/users/pughlab/projects/shallowWGS_benchmark/bamgineer_test/outputs
Traceback (most recent call last):
  File "/mnt/work1/users/home2/quever/git/bamgineer/src/simulate.py", line 69, in <module>
    main(args)
  File "/mnt/work1/users/home2/quever/git/bamgineer/src/simulate.py", line 39, in main
    run_pipeline(results_path)
  File "/mnt/work1/users/home2/quever/git/bamgineer/src/methods.py", line 562, in run_pipeline
    haplotype_path, cancer_dir_path, tmpbams_path, finalbams_path, log_path, logfile = handle.GetProjectPaths(
NameError: global name 'handle' is not defined

Thanks Rene, I had mistakenly removed some imports. FIxed :)

Also I think you should be able to run "gain" (CN=3) without a problem (just tested and the number of reads seems to be fine)

Unfortunately, with the latest pull, I can not run the "gain" like you did?
When I run the gain cnv bed: chr21	30227447	47076809	3; I still fall short on coverage ratio and end up with no final bams
+++ coverage ratio for: chr21_roigain.sorted.bam: 0.382933236524
not enough reads for chr21_roigain.sorted.bam
 ***** pipeline finished in 3.6 minutes *****

The only one I do get a final bam for is the loss cnv bed, chr21	30227447	47076809	1.  Since I assume we are using the same PGP sample (huA2692E) and working on the same chromosomes (Chr21, Chr22) and all other parameters being the same, I would think that you just selected a different region of the genome that has adequate coverage for it to simulate a gain?  What bed did you use to simulate your gain?  Also, how did you test the number of reads, is it just a samtools view with a wc -l?
Just to clarify, did you clean up your output folder after running the "loss" (CN=1) script before running the "gain" (CN=3) script?  I've noticed that if you run them sequentially, then the "gain" (CN=3) will leverage existing temporary files and reproduce the preceeding runs results (maybe should be addressed with run-specific temporary identifiers that get cleaned at the end?)

Hey soroush,
Following up on this again.

What bed did you use to simulate your gain?
Also, how did you test the number of reads, is it just a samtools view with a wc -l?
did you clean up your output folder after running the "loss" (CN=1) script before running the "gain" (CN=3) script?

I'll add a fix shortly for creating run-specific temporary identifiers and cleaning up between runs to prevent reuse of existing files.

Hey Rene,
Thank you for the congratulations and also thank for your support and contributions to make Bamgineer better; much appreciated. Here's the answer to your questions :
Bed file used               :     chr21	30227447	47076809	3
Final number of reads:     9047080 (note that it the final bam file should be named test.bam as indicated by the run script. So if you're not getting test.bam in the output, it means probably the final merge hasn't taken place)
output cleaned?         :      Yes, I ran CN=3 from scratch on a different folder
Seems like we both used the same CNV region; so it's strange that you didn't get enough reads for that region. Have you pulled the latest version?
Hope it helps,
Soroush

After checking over some of your inputs, it seems like your exons.bed (specified in your config file) wasn't made according to your specifications outlined in step  8. Create "exons.bed" file in input_preparation.md.  Can you double check your pipeline for this and try again?
For reference, these are the md5sums for our input files for Step 8:
# My md5sums
3eb30670cfa3dd184931d8e27480bf6f`  chr21.bam
5fa9d9defca46111fe5c6dbf4cdaa302  chr22.bam
434f950f81ebd1a77fecff0c0154550a  bedit.sh

# Your md5sums
3eb30670cfa3dd184931d8e27480bf6f  chr21.bam
5fa9d9defca46111fe5c6dbf4cdaa302  chr22.bam
434f950f81ebd1a77fecff0c0154550a  bedit.sh

However, when I compared my exons.bed to your exons.bed, this is what I got:
# My exons.bed
md5sums:     38f824cc251c50036f2ef3c1af56e7ee  exons.bed
line-count:    47399 exons.bed
size:              1134589 exons.bed

# Your exons.bed (also includes Chr1?)
md5sums:     785f73d276920ea82f2c9dad2cddb44f  exons.bed
line-count:    216812 exons.bed
size:              5182092 Feb 26 15:08 exons.bed

Hi,
I am getting error while running run_example1.sh using example data set. I have recreated exons.bed as explained in input_preparation.md file and modified config file with appropriate path.
File structure:
|-- inputs
|   |-- cnv.bed
|   |-- config.cfg
|   |-- config_ORI.cfg
|   |-- exons.bed
|   |-- exons_ORI.bed
|   |-- normal_het.vcf
|   -- normal_het.vcf.gz |-- outputs |   |-- cnv_dir |   |   -- gain.bed
|   |-- finalbams
|   |-- haplotypedir
|   |   |-- bedtool.log
|   |   |-- chr21_exons_in_roigain.bed
|   |   |-- chr21_het_snpgain.bed
|   |   |-- exons_in_roigain.bed
|   |   -- het_snpgain.bed |   |-- logs |   |   -- debug.log
|   |-- phasedvcfdir
|   |   |-- hap1_het_filtered.bed
|   |   |-- hap1_het_filtered.log
|   |   |-- hap1_het_filtered.recode.vcf
|   |   |-- hap1_het.vcf
|   |   |-- hap2_het_filtered.bed
|   |   |-- hap2_het_filtered.log
|   |   |-- hap2_het_filtered.recode.vcf
|   |   |-- hap2_het.vcf
|   |   |-- normal_het_phased.log
|   |   |-- normal_het_phased.vcf.gz
|   |   |-- normal_het_phased.warnings
|   |   -- PHASED.BED |   -- tmpbams
|-- scripts
|   |-- beagle.log
|   -- run_example1.sh |-- splitbams |   |-- chr21.bam |   |-- chr21.bam.bai |   |-- chr21.byname.bam |   |-- chr22.bam |   |-- chr22.bam.bai |   -- chr22.byname.bam
Here is the output:
/opt/installers/bamgineer/examples/outputs
___ phasing vcf file ___
beagle.09Nov15.d2a.jar
Copyright (C) 2014-2015 Brian L. Browning
Enter "java -jar beagle.jar" for a summary of command line arguments.
Start time: 01:04 PM IST on 25 Apr 2018
Command line: java -Xmx3641m -jar beagle.jar
gt=/opt/installers/bamgineer/examples/inputs/normal_het.vcf.gz
out=/opt/installers/bamgineer/examples/outputs/phasedvcfdir/normal_het_phased
No genetic map is specified: using 1 cM = 1 Mb
reference samples:       0
target samples:          1
Window 1 [ chr21:9414112-48119669 ]
target markers:      10225
Starting burn-in iterations
Window=1 Iteration=1
Time for building model:         0 seconds
Time for sampling (singles):     0 seconds
DAG statistics
mean edges/level: 2      max edges/level: 2
mean edges/node:  2.000  mean count/edge: 1
Window=1 Iteration=2
Time for building model:         0 seconds
Time for sampling (singles):     0 seconds
DAG statistics
mean edges/level: 2      max edges/level: 2
mean edges/node:  2.000  mean count/edge: 1
Window=1 Iteration=3
Time for building model:         0 seconds
Time for sampling (singles):     0 seconds
DAG statistics
mean edges/level: 2      max edges/level: 2
mean edges/node:  2.000  mean count/edge: 1
Window=1 Iteration=4
Time for building model:         0 seconds
Time for sampling (singles):     0 seconds
DAG statistics
mean edges/level: 2      max edges/level: 2
mean edges/node:  2.000  mean count/edge: 1
Window=1 Iteration=5
Time for building model:         0 seconds
Time for sampling (singles):     0 seconds
DAG statistics
mean edges/level: 2      max edges/level: 2
mean edges/node:  2.000  mean count/edge: 1
Window=1 Iteration=6
Time for building model:         0 seconds
Time for sampling (singles):     0 seconds
DAG statistics
mean edges/level: 2      max edges/level: 2
mean edges/node:  2.000  mean count/edge: 1
Window=1 Iteration=7
Time for building model:         0 seconds
Time for sampling (singles):     0 seconds
DAG statistics
mean edges/level: 2      max edges/level: 2
mean edges/node:  2.000  mean count/edge: 1
Window=1 Iteration=8
Time for building model:         0 seconds
Time for sampling (singles):     0 seconds
DAG statistics
mean edges/level: 2      max edges/level: 2
mean edges/node:  2.000  mean count/edge: 1
Window=1 Iteration=9
Time for building model:         0 seconds
Time for sampling (singles):     0 seconds
DAG statistics
mean edges/level: 2      max edges/level: 2
mean edges/node:  2.000  mean count/edge: 1
Window=1 Iteration=10
Time for building model:         0 seconds
Time for sampling (singles):     0 seconds
DAG statistics
mean edges/level: 2      max edges/level: 2
mean edges/node:  2.000  mean count/edge: 1
Starting phasing iterations
Window=1 Iteration=11
Time for building model:         0 seconds
Time for sampling (singles):     0 seconds
DAG statistics
mean edges/level: 2      max edges/level: 2
mean edges/node:  2.000  mean count/edge: 1
states/marker:    1.0
Window=1 Iteration=12
Time for building model:         0 seconds
Time for sampling (singles):     0 seconds
DAG statistics
mean edges/level: 2      max edges/level: 2
mean edges/node:  2.000  mean count/edge: 1
states/marker:    1.0
Window=1 Iteration=13
Time for building model:         0 seconds
Time for sampling (singles):     0 seconds
DAG statistics
mean edges/level: 2      max edges/level: 2
mean edges/node:  2.000  mean count/edge: 1
states/marker:    1.0
Window=1 Iteration=14
Time for building model:         0 seconds
Time for sampling (singles):     0 seconds
DAG statistics
mean edges/level: 2      max edges/level: 2
mean edges/node:  2.000  mean count/edge: 1
states/marker:    1.0
Window=1 Iteration=15
Time for building model:         0 seconds
Time for sampling (singles):     0 seconds
DAG statistics
mean edges/level: 2      max edges/level: 2
mean edges/node:  2.000  mean count/edge: 1
states/marker:    1.0
Window 2 [ chr22:16066867-51239065 ]
target markers:       3455
Starting burn-in iterations
Window=2 Iteration=1
Time for building model:         0 seconds
Time for sampling (singles):     0 seconds
DAG statistics
mean edges/level: 2      max edges/level: 2
mean edges/node:  1.999  mean count/edge: 1
Window=2 Iteration=2
Time for building model:         0 seconds
Time for sampling (singles):     0 seconds
DAG statistics
mean edges/level: 2      max edges/level: 2
mean edges/node:  1.999  mean count/edge: 1
Window=2 Iteration=3
Time for building model:         0 seconds
Time for sampling (singles):     0 seconds
DAG statistics
mean edges/level: 2      max edges/level: 2
mean edges/node:  1.999  mean count/edge: 1
Window=2 Iteration=4
Time for building model:         0 seconds
Time for sampling (singles):     0 seconds
DAG statistics
mean edges/level: 2      max edges/level: 2
mean edges/node:  1.999  mean count/edge: 1
Window=2 Iteration=5
Time for building model:         0 seconds
Time for sampling (singles):     0 seconds
DAG statistics
mean edges/level: 2      max edges/level: 2
mean edges/node:  1.999  mean count/edge: 1
Window=2 Iteration=6
Time for building model:         0 seconds
Time for sampling (singles):     0 seconds
DAG statistics
mean edges/level: 2      max edges/level: 2
mean edges/node:  1.999  mean count/edge: 1
Window=2 Iteration=7
Time for building model:         0 seconds
Time for sampling (singles):     0 seconds
DAG statistics
mean edges/level: 2      max edges/level: 2
mean edges/node:  1.999  mean count/edge: 1
Window=2 Iteration=8
Time for building model:         0 seconds
Time for sampling (singles):     0 seconds
DAG statistics
mean edges/level: 2      max edges/level: 2
mean edges/node:  1.999  mean count/edge: 1
Window=2 Iteration=9
Time for building model:         0 seconds
Time for sampling (singles):     0 seconds
DAG statistics
mean edges/level: 2      max edges/level: 2
mean edges/node:  1.999  mean count/edge: 1
Window=2 Iteration=10
Time for building model:         0 seconds
Time for sampling (singles):     0 seconds
DAG statistics
mean edges/level: 2      max edges/level: 2
mean edges/node:  1.999  mean count/edge: 1
Starting phasing iterations
Window=2 Iteration=11
Time for building model:         0 seconds
Time for sampling (singles):     0 seconds
DAG statistics
mean edges/level: 2      max edges/level: 2
mean edges/node:  1.999  mean count/edge: 1
states/marker:    1.0
Window=2 Iteration=12
Time for building model:         0 seconds
Time for sampling (singles):     0 seconds
DAG statistics
mean edges/level: 2      max edges/level: 2
mean edges/node:  1.999  mean count/edge: 1
states/marker:    1.0
Window=2 Iteration=13
Time for building model:         0 seconds
Time for sampling (singles):     0 seconds
DAG statistics
mean edges/level: 2      max edges/level: 2
mean edges/node:  1.999  mean count/edge: 1
states/marker:    1.0
Window=2 Iteration=14
Time for building model:         0 seconds
Time for sampling (singles):     0 seconds
DAG statistics
mean edges/level: 2      max edges/level: 2
mean edges/node:  1.999  mean count/edge: 1
states/marker:    1.0
Window=2 Iteration=15
Time for building model:         0 seconds
Time for sampling (singles):     0 seconds
DAG statistics
mean edges/level: 2      max edges/level: 2
mean edges/node:  1.999  mean count/edge: 1
states/marker:    1.0
Number of markers:               13680
Total time for building model: 1 second
Total time for sampling:       1 second
Total run time:                2 seconds
End time: 01:04 PM IST on 25 Apr 2018
beagle.09Nov15.d2a.jar finished
VCFtools - v0.1.12a
(C) Adam Auton and Anthony Marcketta 2009
Parameters as interpreted:
--vcf /opt/installers/bamgineer/examples/outputs/phasedvcfdir/hap1_het.vcf
--thin 50
--out /opt/installers/bamgineer/examples/outputs/phasedvcfdir/hap1_het_filtered
--recode
After filtering, kept 0 out of 0 Individuals
Outputting VCF file...
After filtering, kept 5324 out of a possible 6773 Sites
Run Time = 0.00 seconds
VCFtools - v0.1.12a
(C) Adam Auton and Anthony Marcketta 2009
Parameters as interpreted:
--vcf /opt/installers/bamgineer/examples/outputs/phasedvcfdir/hap2_het.vcf
--thin 50
--out /opt/installers/bamgineer/examples/outputs/phasedvcfdir/hap2_het_filtered
--recode
After filtering, kept 0 out of 0 Individuals
Outputting VCF file...
After filtering, kept 5424 out of a possible 6905 Sites
Run Time = 0.00 seconds
[bam_header_read] EOF marker is absent. The input is probably truncated.
[bam_header_read] invalid BAM binary header (this is not a BAM file).
[main_samview] fail to read the header from "/opt/installers/bamgineer/examples/outputs/tmpbams/chr21_r
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/opt/installers/bamgineer/src/helpers/handlers.py", line 76, in receive
record = self.queue.get(True, self.polltime)
File "/usr/lib/python2.7/multiprocessing/queues.py", line 135, in get
res = self._recv()
TypeError: ('init() takes exactly 2 arguments (1 given)', <class 'pysam.utils.SamtoolsError'>, ())