Running CWL on a cluster with Cromwell

Hello,
I'm trying to run a CWL variant calling analysis on an SGE cluster with Cromwell and a local bcbio install and I get the following error:
Traceback (most recent call last):
  File "/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/tools/bin/bcbio_vm.py", line 354, in <module>
    args.func(args)
  File "/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/anaconda/envs/bcbiovm/lib/python2.7/site-packages/bcbio/cwl/tool.py", line 317, in run
    _TOOLS[args.tool](args)
  File "/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/anaconda/envs/bcbiovm/lib/python2.7/site-packages/bcbio/cwl/tool.py", line 186, in _run_cromwell
    "-Dconfig.file=%s" % hpc.create_cromwell_config(args, work_dir, json_file)]
  File "/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/anaconda/envs/bcbiovm/lib/python2.7/site-packages/bcbio/cwl/hpc.py", line 31, in create_cromwell_config
    main_config = {"hpc": (HPC_CONFIGS[scheduler] % conf_args) if scheduler else "",
ValueError: incomplete format key

Running locally with Cromwell works fine. This is the command I'm using:
bcbio_vm.py cwlrun cromwell --no-container testingVC-merged-workflow -q campus-haswell.q -s sge -r pename=openmp

Thanks for the report and sorry about the issue. This was a typo in the SGE configuration where I had a close bracket instead of parentheses. I've fixed in development and will release a new bcbio-nextgen conda package in the next couple of days that resolves this. If you want the fix immediately you can manually install using the bcbiovm python (/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/anaconda/envs/bin/python) or make the small fix:
/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/anaconda/envs/bcbiovm/lib/python2.7/site-packages/bcbio/cwl/hpc.py
Thanks again for reporting the issue.

Thanks for looking into this. Manually editing the hpc.py file as shown in the previous commit has fixed the problem. However, I'm now running into a different error, shown below.
I did not specify any memory settings for SGE. The command I used to run Cromwell is in the first post, above. I'm using the same input files from the archive uploaded here: https://github.com/bcbio/bcbio-nextgen/issues/2688#issue-412394726, with the only difference that I'm trying to run things on a cluster, using Cromwell.
[2019-02-25 14:15:30,76] [info] MaterializeWorkflowDescriptorActor [9a5400e7]: Parsing workflow as CWL v1.0
[2019-02-25 14:15:32,65] [info] MaterializeWorkflowDescriptorActor [9a5400e7]: Call-to-Backend assignments: variantcall_batch_region -> SGE, merge_split_alignments -> SGE, pipeline_summary -> SGE, combine_sample_regions -> SGE, multiqc_summary -> SGE, alignment_to_rec -> SGE, qc_to_rec -> SGE, postprocess_alignment -> SGE, summarize_vc -> SGE, concat_batch_variantcalls -> SGE, compare_to_rm -> SGE, prep_align_inputs -> SGE, batch_for_ensemble -> SGE, batch_for_variantcall -> SGE, postprocess_variants -> SGE, process_alignment -> SGE, get_parallel_regions -> SGE, prep_samples -> SGE, postprocess_alignment_to_rec -> SGE, combine_calls -> SGE, prep_samples_to_rec -> SGE
[2019-02-25 14:15:32,87] [error] Error parsing generated wdl:
task submit {

String job_id
String job_name
String cwd
String out
String err
String script
String job_shell

        Int cpu = 1
        Int memory_mb = 2048
        String queue = "campus-haswell.q"
        String pename = "openmp"
        String memtype = "mem_type"

        Int? cpuMin
        Int? cpuMax
        Int? memoryMin
        Int? memoryMax
        String? outDirMin
        String? outDirMax
        String? tmpDirMin
        String? tmpDirMax

command {

        qsub -V -w w -j y -N ${job_name} -wd ${cwd}         -o ${out} -e ${err} -q ${queue}         -pe ${pename} ${cpu} ${"-l " + mem_type + "=" + memory_mb + "m"}         /usr/bin/env bash ${script}

}
}

task kill {

String job_id
String docker_cid
String job_shell

command {
qdel ${job_id}
}
}

task check_alive {

String job_id
String docker_cid
String job_shell

command {
qstat -j ${job_id}
}
}
java.lang.RuntimeException: Error parsing generated wdl:
task submit {

String job_id
String job_name
String cwd
String out
String err
String script
String job_shell

        Int cpu = 1
        Int memory_mb = 2048
        String queue = "campus-haswell.q"
        String pename = "openmp"
        String memtype = "mem_type"

        Int? cpuMin
        Int? cpuMax
        Int? memoryMin
        Int? memoryMax
        String? outDirMin
        String? outDirMax
        String? tmpDirMin
        String? tmpDirMax

command {

        qsub -V -w w -j y -N ${job_name} -wd ${cwd}         -o ${out} -e ${err} -q ${queue}         -pe ${pename} ${cpu} ${"-l " + mem_type + "=" + memory_mb + "m"}         /usr/bin/env bash ${script}

}
}

task kill {

String job_id
String docker_cid
String job_shell

command {
qdel ${job_id}
}
}

task check_alive {

String job_id
String docker_cid
String job_shell

command {
qstat -j ${job_id}
}
}
	at cromwell.backend.impl.sfs.config.ConfigWdlNamespace.<init>(ConfigWdlNamespace.scala:55)
	at cromwell.backend.impl.sfs.config.ConfigInitializationActor.configWdlNamespace$lzycompute(ConfigInitializationActor.scala:39)
	at cromwell.backend.impl.sfs.config.ConfigInitializationActor.configWdlNamespace(ConfigInitializationActor.scala:39)
	at cromwell.backend.impl.sfs.config.ConfigInitializationActor.declarationValidations$lzycompute(ConfigInitializationActor.scala:42)
	at cromwell.backend.impl.sfs.config.ConfigInitializationActor.declarationValidations(ConfigInitializationActor.scala:41)
	at cromwell.backend.impl.sfs.config.ConfigInitializationActor.runtimeAttributesBuilder$lzycompute(ConfigInitializationActor.scala:53)
	at cromwell.backend.impl.sfs.config.ConfigInitializationActor.runtimeAttributesBuilder(ConfigInitializationActor.scala:52)
	at cromwell.backend.standard.StandardInitializationActor.coerceDefaultRuntimeAttributes(StandardInitializationActor.scala:82)
	at cromwell.backend.BackendWorkflowInitializationActor.initSequence(BackendWorkflowInitializationActor.scala:155)
	at cromwell.backend.BackendWorkflowInitializationActor.initSequence$(BackendWorkflowInitializationActor.scala:153)
	at cromwell.backend.standard.StandardInitializationActor.initSequence(StandardInitializationActor.scala:44)
	at cromwell.backend.BackendWorkflowInitializationActor$$anonfun$receive$1.$anonfun$applyOrElse$1(BackendWorkflowInitializationActor.scala:146)
	at cromwell.backend.BackendLifecycleActor.performActionThenRespond(BackendLifecycleActor.scala:44)
	at cromwell.backend.BackendLifecycleActor.performActionThenRespond$(BackendLifecycleActor.scala:40)
	at cromwell.backend.standard.StandardInitializationActor.performActionThenRespond(StandardInitializationActor.scala:44)
	at cromwell.backend.BackendWorkflowInitializationActor$$anonfun$receive$1.applyOrElse(BackendWorkflowInitializationActor.scala:146)
	at akka.actor.Actor.aroundReceive(Actor.scala:517)
	at akka.actor.Actor.aroundReceive$(Actor.scala:515)
	at cromwell.backend.standard.StandardInitializationActor.aroundReceive(StandardInitializationActor.scala:44)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
	at akka.actor.ActorCell.invoke(ActorCell.scala:557)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: wdl.draft2.parser.WdlParser$SyntaxError: ERROR: Variable mem_type does not reference any declaration in the task (line 28, col 132):

        qsub -V -w w -j y -N ${job_name} -wd ${cwd}         -o ${out} -e ${err} -q ${queue}         -pe ${pename} ${cpu} ${"-l " + mem_type + "=" + memory_mb + "m"}         /usr/bin/env bash ${script}
                                                                                ^

Task defined here (line 1, col 6):

task submit {
     ^

	at wdl.draft2.model.WdlNamespace$.$anonfun$apply$54(WdlNamespace.scala:391)
	at scala.collection.TraversableLike$WithFilter.$anonfun$map$2(TraversableLike.scala:739)
	at scala.collection.immutable.List.foreach(List.scala:389)
	at scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:738)
	at wdl.draft2.model.WdlNamespace$.$anonfun$apply$51(WdlNamespace.scala:389)
	at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:241)
	at scala.collection.Iterator.foreach(Iterator.scala:944)
	at scala.collection.Iterator.foreach$(Iterator.scala:944)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1432)
	at scala.collection.IterableLike.foreach(IterableLike.scala:71)
	at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
	at scala.collection.TraversableLike.flatMap(TraversableLike.scala:241)
	at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:238)
	at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
	at wdl.draft2.model.WdlNamespace$.$anonfun$apply$50(WdlNamespace.scala:388)
	at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:241)
	at scala.collection.Iterator.foreach(Iterator.scala:944)
	at scala.collection.Iterator.foreach$(Iterator.scala:944)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1432)
	at scala.collection.IterableLike.foreach(IterableLike.scala:71)
	at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
	at scala.collection.TraversableLike.flatMap(TraversableLike.scala:241)
	at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:238)
	at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
	at wdl.draft2.model.WdlNamespace$.apply(WdlNamespace.scala:387)
	at wdl.draft2.model.WdlNamespace$.$anonfun$load$1(WdlNamespace.scala:160)
	at scala.util.Try$.apply(Try.scala:209)
	at wdl.draft2.model.WdlNamespace$.load(WdlNamespace.scala:160)
	at wdl.draft2.model.WdlNamespace$.loadUsingSource(WdlNamespace.scala:156)
	at cromwell.backend.impl.sfs.config.ConfigWdlNamespace.<init>(ConfigWdlNamespace.scala:53)
	... 27 common frames omitted

Thank you for continuing to debug this and sorry about the problems. We haven't had an SGE cluster to test this on so appreciate you working through the bugs. I pushed another small fix for this typo that you can patch in the same approach as previously (mem_type -> memtype). I hope that will resolve all of the issues and get this running for you. Please let us know if you get stuck anywhere else and thanks again for the patience debugging.

Thanks for the update. After modifying the hpc.py file, the CWL run ends up with another long list of errors, partially reproduced below.
As suggested in https://github.com/bcbio/bcbio-nextgen/issues/2701#issuecomment-467387879, I also tried running bcbio_nextgen.py directly on the login node (as opposed to running it through qsub), but it did not make any difference in this case.
[2019-02-26 14:26:02,40] [info] DispatchedConfigAsyncJobExecutionActor [d44c20ccalignment_to_rec:NA:1]: 'bcbio_nextgen.py' 'runfn' 'alignment_to_rec' 'cwl' 'sentinel_runtime=cores,1,ram,3072' 'sentinel_parallel=multi-combined' 'sentinel_outputs=alignment_rec:resources;description;config__algorithm__trim_reads;rgnames__lb;rgnames__sample;config__algorithm__variant_regions;config__algorithm__adapters;reference__bwa__indexes;config__algorithm__bam_clean;config__algorithm__align_split_size;reference__fasta__base;files;config__algorithm__aligner;rgnames__rg;rgnames__lane;config__algorithm__mark_duplicates;rgnames__pu;analysis;rgnames__pl' 'sentinel_inputs=files:var,analysis:var,config__algorithm__align_split_size:var,reference__fasta__base:var,rgnames__pl:var,rgnames__sample:var,rgnames__pu:var,rgnames__lane:var,rgnames__rg:var,rgnames__lb:var,reference__bwa__indexes:var,config__algorithm__aligner:var,config__algorithm__trim_reads:var,config__algorithm__adapters:var,config__algorithm__bam_clean:var,config__algorithm__variant_regions:var,config__algorithm__mark_duplicates:var,resources:var,description:var' 'run_number=0'
[2019-02-26 14:26:02,48] [error] DispatchedConfigAsyncJobExecutionActor [d44c20ccalignment_to_rec:NA:1]: Error attempting to Execute
common.exception.AggregatedMessageException: Error(s):
Could not evaluate expression: "-l " + memtype + "=" + memory_mb + "m": Cannot perform operation: -l mem_type= + WomLong(3072)
	at common.validation.Validation$ValidationTry$.toTry$extension1(Validation.scala:77)
	at common.validation.Validation$ValidationTry$.toTry$extension0(Validation.scala:73)
	at cromwell.backend.impl.sfs.config.ConfigAsyncJobExecutionActor.writeTaskScript(ConfigAsyncJobExecutionActor.scala:107)
	at cromwell.backend.impl.sfs.config.ConfigAsyncJobExecutionActor.writeTaskScript$(ConfigAsyncJobExecutionActor.scala:55)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.writeTaskScript(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.impl.sfs.config.ConfigAsyncJobExecutionActor.processArgs(ConfigAsyncJobExecutionActor.scala:43)
	at cromwell.backend.impl.sfs.config.ConfigAsyncJobExecutionActor.processArgs$(ConfigAsyncJobExecutionActor.scala:39)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.processArgs$lzycompute(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.processArgs(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.makeProcessRunner(SharedFileSystemAsyncJobExecutionActor.scala:174)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.makeProcessRunner$(SharedFileSystemAsyncJobExecutionActor.scala:171)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.makeProcessRunner(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.$anonfun$execute$2(SharedFileSystemAsyncJobExecutionActor.scala:145)
	at scala.util.Either.fold(Either.scala:188)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute(SharedFileSystemAsyncJobExecutionActor.scala:144)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute$(SharedFileSystemAsyncJobExecutionActor.scala:139)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.execute(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$executeAsync$1(StandardAsyncExecutionActor.scala:637)
	at scala.util.Try$.apply(Try.scala:209)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync(StandardAsyncExecutionActor.scala:637)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync$(StandardAsyncExecutionActor.scala:637)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.executeAsync(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover(StandardAsyncExecutionActor.scala:952)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover$(StandardAsyncExecutionActor.scala:944)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.executeOrRecover(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.$anonfun$robustExecuteOrRecover$1(AsyncBackendJobExecutionActor.scala:65)
	at cromwell.core.retry.Retry$.withRetry(Retry.scala:38)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.withRetry(AsyncBackendJobExecutionActor.scala:61)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.cromwell$backend$async$AsyncBackendJobExecutionActor$$robustExecuteOrRecover(AsyncBackendJobExecutionActor.scala:65)
	at cromwell.backend.async.AsyncBackendJobExecutionActor$$anonfun$receive$1.applyOrElse(AsyncBackendJobExecutionActor.scala:88)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
	at akka.actor.Actor.aroundReceive(Actor.scala:517)
	at akka.actor.Actor.aroundReceive$(Actor.scala:515)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.aroundReceive(ConfigAsyncJobExecutionActor.scala:211)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
	at akka.actor.ActorCell.invoke(ActorCell.scala:557)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[2019-02-26 14:26:02,49] [error] DispatchedConfigAsyncJobExecutionActor [d44c20ccprep_samples_to_rec:NA:1]: Error attempting to Execute
common.exception.AggregatedMessageException: Error(s):
Could not evaluate expression: "-l " + memtype + "=" + memory_mb + "m": Cannot perform operation: -l mem_type= + WomLong(3072)
	at common.validation.Validation$ValidationTry$.toTry$extension1(Validation.scala:77)
	at common.validation.Validation$ValidationTry$.toTry$extension0(Validation.scala:73)
	at cromwell.backend.impl.sfs.config.ConfigAsyncJobExecutionActor.writeTaskScript(ConfigAsyncJobExecutionActor.scala:107)
	at cromwell.backend.impl.sfs.config.ConfigAsyncJobExecutionActor.writeTaskScript$(ConfigAsyncJobExecutionActor.scala:55)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.writeTaskScript(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.impl.sfs.config.ConfigAsyncJobExecutionActor.processArgs(ConfigAsyncJobExecutionActor.scala:43)
	at cromwell.backend.impl.sfs.config.ConfigAsyncJobExecutionActor.processArgs$(ConfigAsyncJobExecutionActor.scala:39)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.processArgs$lzycompute(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.processArgs(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.makeProcessRunner(SharedFileSystemAsyncJobExecutionActor.scala:174)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.makeProcessRunner$(SharedFileSystemAsyncJobExecutionActor.scala:171)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.makeProcessRunner(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.$anonfun$execute$2(SharedFileSystemAsyncJobExecutionActor.scala:145)
	at scala.util.Either.fold(Either.scala:188)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute(SharedFileSystemAsyncJobExecutionActor.scala:144)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute$(SharedFileSystemAsyncJobExecutionActor.scala:139)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.execute(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$executeAsync$1(StandardAsyncExecutionActor.scala:637)
	at scala.util.Try$.apply(Try.scala:209)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync(StandardAsyncExecutionActor.scala:637)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync$(StandardAsyncExecutionActor.scala:637)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.executeAsync(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover(StandardAsyncExecutionActor.scala:952)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover$(StandardAsyncExecutionActor.scala:944)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.executeOrRecover(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.$anonfun$robustExecuteOrRecover$1(AsyncBackendJobExecutionActor.scala:65)
	at cromwell.core.retry.Retry$.withRetry(Retry.scala:38)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.withRetry(AsyncBackendJobExecutionActor.scala:61)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.cromwell$backend$async$AsyncBackendJobExecutionActor$$robustExecuteOrRecover(AsyncBackendJobExecutionActor.scala:65)
	at cromwell.backend.async.AsyncBackendJobExecutionActor$$anonfun$receive$1.applyOrElse(AsyncBackendJobExecutionActor.scala:88)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
	at akka.actor.Actor.aroundReceive(Actor.scala:517)
	at akka.actor.Actor.aroundReceive$(Actor.scala:515)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.aroundReceive(ConfigAsyncJobExecutionActor.scala:211)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
	at akka.actor.ActorCell.invoke(ActorCell.scala:557)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Sorry about the continued problems and thanks for the patience debugging this. Apologies, it is hard to do this right without an SGE cluster to test on. I've pushed another speculative fix based on your traceback and hope that this one will get the submission working. Please let us know if not and we can continue to fix. Thanks again.

No worries and thanks a lot for your help with this. I'm just hoping to get things working with CWL and IPython on the SGE cluster before the next bcbio release.
After the new modification, the outcome is a different error. The contents of the two referenced stderr files is also posted below. This seems to be a problem with the scheduling system, but any pointers you have about troubleshooting it would be much appreciated, in case you've seen this kind of error before.
To keep things somewhat readable, in a second post I will also paste the relevant script files and script.submit files corresponding to the two stderr files mentioned below.
Please see also this comment on an issue related to IPython, just in case both of these two issues might somehow related to the cluster setup.
[2019-02-27 16:28:46,17] [error] WorkflowManagerActor Workflow a19ad0e5-8d63-4aff-aefc-76e6b40afeef failed (during ExecutingWorkflowState): java.lang.RuntimeException: Unable to start job. Check the stderr file for possible errors: /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-prep_samples_to_rec/execution/stderr.submit
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.$anonfun$execute$2(SharedFileSystemAsyncJobExecutionActor.scala:149)
	at scala.util.Either.fold(Either.scala:188)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute(SharedFileSystemAsyncJobExecutionActor.scala:144)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute$(SharedFileSystemAsyncJobExecutionActor.scala:139)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.execute(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$executeAsync$1(StandardAsyncExecutionActor.scala:637)
	at scala.util.Try$.apply(Try.scala:209)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync(StandardAsyncExecutionActor.scala:637)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync$(StandardAsyncExecutionActor.scala:637)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.executeAsync(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover(StandardAsyncExecutionActor.scala:952)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover$(StandardAsyncExecutionActor.scala:944)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.executeOrRecover(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.$anonfun$robustExecuteOrRecover$1(AsyncBackendJobExecutionActor.scala:65)
	at cromwell.core.retry.Retry$.withRetry(Retry.scala:38)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.withRetry(AsyncBackendJobExecutionActor.scala:61)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.cromwell$backend$async$AsyncBackendJobExecutionActor$$robustExecuteOrRecover(AsyncBackendJobExecutionActor.scala:65)
	at cromwell.backend.async.AsyncBackendJobExecutionActor$$anonfun$receive$1.applyOrElse(AsyncBackendJobExecutionActor.scala:88)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
	at akka.actor.Actor.aroundReceive(Actor.scala:517)
	at akka.actor.Actor.aroundReceive$(Actor.scala:515)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.aroundReceive(ConfigAsyncJobExecutionActor.scala:211)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
	at akka.actor.ActorCell.invoke(ActorCell.scala:557)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

java.lang.RuntimeException: Unable to start job. Check the stderr file for possible errors: /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-alignment_to_rec/execution/stderr.submit
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.$anonfun$execute$2(SharedFileSystemAsyncJobExecutionActor.scala:149)
	at scala.util.Either.fold(Either.scala:188)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute(SharedFileSystemAsyncJobExecutionActor.scala:144)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute$(SharedFileSystemAsyncJobExecutionActor.scala:139)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.execute(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$executeAsync$1(StandardAsyncExecutionActor.scala:637)
	at scala.util.Try$.apply(Try.scala:209)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync(StandardAsyncExecutionActor.scala:637)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync$(StandardAsyncExecutionActor.scala:637)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.executeAsync(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover(StandardAsyncExecutionActor.scala:952)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover$(StandardAsyncExecutionActor.scala:944)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.executeOrRecover(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.$anonfun$robustExecuteOrRecover$1(AsyncBackendJobExecutionActor.scala:65)
	at cromwell.core.retry.Retry$.withRetry(Retry.scala:38)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.withRetry(AsyncBackendJobExecutionActor.scala:61)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.cromwell$backend$async$AsyncBackendJobExecutionActor$$robustExecuteOrRecover(AsyncBackendJobExecutionActor.scala:65)
	at cromwell.backend.async.AsyncBackendJobExecutionActor$$anonfun$receive$1.applyOrElse(AsyncBackendJobExecutionActor.scala:88)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
	at akka.actor.Actor.aroundReceive(Actor.scala:517)
	at akka.actor.Actor.aroundReceive$(Actor.scala:515)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.aroundReceive(ConfigAsyncJobExecutionActor.scala:211)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
	at akka.actor.ActorCell.invoke(ActorCell.scala:557)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

And the two stderr files contain essentially the same thing:
$ cat /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-prep_samples_to_rec/execution/stderr.submit
Unable to run job: Script length does not match declared length.
Exiting.

and
$ cat /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-alignment_to_rec/execution/stderr.submit
Unable to run job: Script length does not match declared length.
Exiting.

These are the script and script.submit files corresponding to the two stderr.submit files from the previous post.
First, the files for the call-prep_samples_to_rec stage:
$ cat /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-prep_samples_to_rec/execution/script
#!/bin/bash

cd /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-prep_samples_to_rec/execution
tmpDir=$(mkdir -p "/export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-prep_samples_to_rec/tmp.1bd893ee" && echo "/export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-prep_samples_to_rec/tmp.1bd893ee")

export _JAVA_OPTIONS=-Djava.io.tmpdir="$tmpDir"
export TMPDIR="$tmpDir"
export HOME="/export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-prep_samples_to_rec/execution"
(
cd /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-prep_samples_to_rec/execution

)
outa19ad0e5="${tmpDir}/out.$$" erra19ad0e5="${tmpDir}/err.$$"
mkfifo "$outa19ad0e5" "$erra19ad0e5"
trap 'rm "$outa19ad0e5" "$erra19ad0e5"' EXIT
tee '/export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-prep_samples_to_rec/execution/stdout' < "$outa19ad0e5" &
tee '/export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-prep_samples_to_rec/execution/stderr' < "$erra19ad0e5" >&2 &
(
cd /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-prep_samples_to_rec/execution
export MPLCONFIGDIR="."

'bcbio_nextgen.py' 'runfn' 'prep_samples_to_rec' 'cwl' 'sentinel_runtime=cores,1,ram,3072' 'sentinel_parallel=multi-combined' 'sentinel_outputs=prep_samples_rec:resources;description;rgnames__sample;config__algorithm__variant_regions;reference__fasta__base;config__algorithm__svcaller' 'sentinel_inputs=rgnames__sample:var,config__algorithm__svcaller:var,config__algorithm__variant_regions:var,reference__fasta__base:var,resources:var,description:var' 'run_number=0'
)  > "$outa19ad0e5" 2> "$erra19ad0e5"
echo $? > /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-prep_samples_to_rec/execution/rc.tmp
(
# add a .file in every empty directory to facilitate directory delocalization on the cloud
cd /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-prep_samples_to_rec/execution
find . -type d -exec sh -c '[ -z "$(ls -A '"'"'{}'"'"')" ] && touch '"'"'{}'"'"'/.file' \;
)
(
cd /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-prep_samples_to_rec/execution
sync
# make the directory which will keep the matching files
mkdir /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-prep_samples_to_rec/execution/glob-b34dfc006a981a93d6da067cf50036fe

# create the glob control file that will allow for the globbing to succeed even if there is 0 match
echo "This file is used by Cromwell to allow for globs that would not match any file.
By its presence it works around the limitation of some backends that do not allow empty globs.
Regardless of the outcome of the glob, this file will not be part of the final list of globbed files." > /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-prep_samples_to_rec/execution/glob-b34dfc006a981a93d6da067cf50036fe/cromwell_glob_control_file

# hardlink or symlink all the files into the glob directory
( ln -L /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-prep_samples_to_rec/execution/cwl.output.json /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-prep_samples_to_rec/execution/glob-b34dfc006a981a93d6da067cf50036fe 2> /dev/null ) || ( ln /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-prep_samples_to_rec/execution/cwl.output.json /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-prep_samples_to_rec/execution/glob-b34dfc006a981a93d6da067cf50036fe )

# list all the files (except the control file) that match the glob into a file called glob-[md5 of glob].list
ls -1 /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-prep_samples_to_rec/execution/glob-b34dfc006a981a93d6da067cf50036fe | grep -v cromwell_glob_control_file > /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-prep_samples_to_rec/execution/glob-b34dfc006a981a93d6da067cf50036fe.list

)
mv /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-prep_samples_to_rec/execution/rc.tmp /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-prep_samples_to_rec/execution/rc

$ cat /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-prep_samples_to_rec/execution/script.submit
#!/bin/bash
qsub -V -w w -j y -N cromwell_a19ad0e5_prep_samples_to_rec -wd /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-prep_samples_to_rec         -o /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-prep_samples_to_rec/execution/stdout -e /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-prep_samples_to_rec/execution/stderr -q campus-haswell.q         -pe openmp 1 -l mem_type=3072m         /usr/bin/env bash /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-prep_samples_to_rec/execution/script

Second, the files for the call-alignment_to_rec stage:
$ cat /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-alignment_to_rec/execution/script
#!/bin/bash

cd /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-alignment_to_rec/execution
tmpDir=$(mkdir -p "/export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-alignment_to_rec/tmp.84c9975b" && echo "/export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-alignment_to_rec/tmp.84c9975b")

export _JAVA_OPTIONS=-Djava.io.tmpdir="$tmpDir"
export TMPDIR="$tmpDir"
export HOME="/export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-alignment_to_rec/execution"
(
cd /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-alignment_to_rec/execution

)
outa19ad0e5="${tmpDir}/out.$$" erra19ad0e5="${tmpDir}/err.$$"
mkfifo "$outa19ad0e5" "$erra19ad0e5"
trap 'rm "$outa19ad0e5" "$erra19ad0e5"' EXIT
tee '/export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-alignment_to_rec/execution/stdout' < "$outa19ad0e5" &
tee '/export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-alignment_to_rec/execution/stderr' < "$erra19ad0e5" >&2 &
(
cd /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-alignment_to_rec/execution
export MPLCONFIGDIR="."

'bcbio_nextgen.py' 'runfn' 'alignment_to_rec' 'cwl' 'sentinel_runtime=cores,1,ram,3072' 'sentinel_parallel=multi-combined' 'sentinel_outputs=alignment_rec:resources;description;config__algorithm__trim_reads;rgnames__lb;rgnames__sample;config__algorithm__variant_regions;config__algorithm__adapters;reference__bwa__indexes;config__algorithm__bam_clean;config__algorithm__align_split_size;reference__fasta__base;files;config__algorithm__aligner;rgnames__rg;rgnames__lane;config__algorithm__mark_duplicates;rgnames__pu;analysis;rgnames__pl' 'sentinel_inputs=files:var,analysis:var,config__algorithm__align_split_size:var,reference__fasta__base:var,rgnames__pl:var,rgnames__sample:var,rgnames__pu:var,rgnames__lane:var,rgnames__rg:var,rgnames__lb:var,reference__bwa__indexes:var,config__algorithm__aligner:var,config__algorithm__trim_reads:var,config__algorithm__adapters:var,config__algorithm__bam_clean:var,config__algorithm__variant_regions:var,config__algorithm__mark_duplicates:var,resources:var,description:var' 'run_number=0'
)  > "$outa19ad0e5" 2> "$erra19ad0e5"
echo $? > /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-alignment_to_rec/execution/rc.tmp
(
# add a .file in every empty directory to facilitate directory delocalization on the cloud
cd /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-alignment_to_rec/execution
find . -type d -exec sh -c '[ -z "$(ls -A '"'"'{}'"'"')" ] && touch '"'"'{}'"'"'/.file' \;
)
(
cd /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-alignment_to_rec/execution
sync
# make the directory which will keep the matching files
mkdir /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-alignment_to_rec/execution/glob-b34dfc006a981a93d6da067cf50036fe

# create the glob control file that will allow for the globbing to succeed even if there is 0 match
echo "This file is used by Cromwell to allow for globs that would not match any file.
By its presence it works around the limitation of some backends that do not allow empty globs.
Regardless of the outcome of the glob, this file will not be part of the final list of globbed files." > /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-alignment_to_rec/execution/glob-b34dfc006a981a93d6da067cf50036fe/cromwell_glob_control_file

# hardlink or symlink all the files into the glob directory
( ln -L /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-alignment_to_rec/execution/cwl.output.json /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-alignment_to_rec/execution/glob-b34dfc006a981a93d6da067cf50036fe 2> /dev/null ) || ( ln /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-alignment_to_rec/execution/cwl.output.json /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-alignment_to_rec/execution/glob-b34dfc006a981a93d6da067cf50036fe )

# list all the files (except the control file) that match the glob into a file called glob-[md5 of glob].list
ls -1 /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-alignment_to_rec/execution/glob-b34dfc006a981a93d6da067cf50036fe | grep -v cromwell_glob_control_file > /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-alignment_to_rec/execution/glob-b34dfc006a981a93d6da067cf50036fe.list

)
mv /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-alignment_to_rec/execution/rc.tmp /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-alignment_to_rec/execution/rc

$ cat /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-alignment_to_rec/execution/script.submit
#!/bin/bash
qsub -V -w w -j y -N cromwell_a19ad0e5_alignment_to_rec -wd /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-alignment_to_rec         -o /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-alignment_to_rec/execution/stdout -e /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-alignment_to_rec/execution/stderr -q campus-haswell.q         -pe openmp 1 -l mem_type=3072m         /usr/bin/env bash /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/a19ad0e5-8d63-4aff-aefc-76e6b40afeef/call-alignment_to_rec/execution/script

Thanks for continuing to help with this and sorry about the problems. That error is completely baffling to me and when searching I don't find many clues as to what is going wrong. My suggestion would be to try and run the qsub command Cromwell is generating outside of bcbio/Cromwell. Does it still fail with the same error? If so, you could try tweaking command line options to see if there's something problematic we're passing that's causing the problem. If you're able to figure out what is happening with the submission we can definitely fix on the bcbio side.
It looks like this is happening upstream of the issues you're seeing in #2701. I think Cromwell submission will work around that problem since it only interacts with the cluster and doesn't need direct communication between the workers and main process.
Thanks again.

You're right that the CWL and IPython issues were unrelated, as the latter was fixed, while the former is still ongoing.
Manually running the qsub commands results in the same error, so I've had a go at trying to change the qsub parameters. Removing the /usr/bin/env bash bit before the script path at the end got rid of the original error Unable to run job: Script length does not match declared length, but resulted in a different error:
$ qsub -V -w w -j y -N cromwell_fc40d97e_alignment_to_rec -wd /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/fc40d97e-33a8-4785-8a4c-b2585c1e4a84/call-alignment_to_rec         -o /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/fc40d97e-33a8-4785-8a4c-b2585c1e4a84/call-alignment_to_rec/execution/stdout -e /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/fc40d97e-33a8-4785-8a4c-b2585c1e4a84/call-alignment_to_rec/execution/stderr -q campus-haswell.q         -pe openmp 1 -l mem_type=3072m         /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/fc40d97e-33a8-4785-8a4c-b2585c1e4a84/call-alignment_to_rec/execution/script
Unable to run job: unknown resource "mem_type".
Exiting.

Changing mem_type to h_vmem (as per this page: http://www.softpanorama.org/HPC/Grid_engine/Resource_quotas/index.shtml) got the job submitted successfully:
$ qsub -V -w w -j y -N cromwell_fc40d97e_alignment_to_rec -wd /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/fc40d97e-33a8-4785-8a4c-b2585c1e4a84/call-alignment_to_rec         -o /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/fc40d97e-33a8-4785-8a4c-b2585c1e4a84/call-alignment_to_rec/execution/stdout -e /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/fc40d97e-33a8-4785-8a4c-b2585c1e4a84/call-alignment_to_rec/execution/stderr -q campus-haswell.q         -pe openmp 1 -l h_vmem=3072m         /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/fc40d97e-33a8-4785-8a4c-b2585c1e4a84/call-alignment_to_rec/execution/script
Your job 1048762 ("cromwell_fc40d97e_alignment_to_rec") has been submitted

Not sure if this is the best approach, but at least the script was submitted with no qsub errors.

Thanks much for digging into this. Apologies, my SGE is incredibly rusty and it looks like this should have been specified as mem_free which is the default on most systems. You can configure this to something else with -r memtype on the command line but if the default worked with IPython it should also do here. It's another small change if you want to patch as you've been doing. Fingers crossed this gets it working for you, and please let us know if you hit any other issues. Thanks again.

Thanks for the update. Unfortunately, after editing the file /export/home/ncit/external/a.mizeranschi/bcbio_nextgen/anaconda/envs/bcbiovm/lib/python2.7/site-packages/bcbio/cwl/hpc.py as per the above commit, I still get the following error:
$ cat /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/0b234010-da29-4fb6-893b-5c0fad674261/call-prep_samples_to_rec/execution/stderr.submit
Unable to run job: Script length does not match declared length.
Exiting.

Is the /usr/bin/env bash bit really necessary in the file /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/0b234010-da29-4fb6-893b-5c0fad674261/call-prep_samples_to_rec/execution/script.submit? I'm not too knowledgeable about this kind of stuff, but I noticed that each of the scripts (script, script.submit) already contain the following shebang on the first line:
#!/bin/bash

I guess the /usr/bin/env bash bit was needed, after all. I tried changing line number 352 in /export/home/ncit/external/a.mizeranschi/bcbio_nextgen/anaconda/envs/bcbiovm/lib/python2.7/site-packages/bcbio/cwl/hpc.py from
        /usr/bin/env bash ${script}

to
        ${script}

but all that managed was to change the error into the one below.
Not sure what else to try next.
[2019-03-01 16:07:31,38] [info] DispatchedConfigAsyncJobExecutionActor [cefb3c55prep_samples_to_rec:NA:1]: Status change from - to Running
[2019-03-01 16:07:31,38] [info] DispatchedConfigAsyncJobExecutionActor [cefb3c55alignment_to_rec:NA:1]: Status change from - to Running
[2019-03-01 16:07:32,69] [warn] DispatchedConfigAsyncJobExecutionActor [cefb3c55alignment_to_rec:NA:1]: Fatal exception polling for status. Job will fail.
java.util.concurrent.ExecutionException: Boxed Error
	at scala.concurrent.impl.Promise$.resolver(Promise.scala:83)
	at scala.concurrent.impl.Promise$.scala$concurrent$impl$Promise$$resolveTry(Promise.scala:75)
	at scala.concurrent.impl.Promise$KeptPromise$.apply(Promise.scala:402)
	at scala.concurrent.Promise$.fromTry(Promise.scala:138)
	at scala.concurrent.Future$.fromTry(Future.scala:635)
	at cromwell.backend.standard.StandardAsyncExecutionActor.pollStatusAsync(StandardAsyncExecutionActor.scala:697)
	at cromwell.backend.standard.StandardAsyncExecutionActor.pollStatusAsync$(StandardAsyncExecutionActor.scala:697)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.pollStatusAsync(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.standard.StandardAsyncExecutionActor.poll(StandardAsyncExecutionActor.scala:989)
	at cromwell.backend.standard.StandardAsyncExecutionActor.poll$(StandardAsyncExecutionActor.scala:983)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.poll(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.$anonfun$robustPoll$1(AsyncBackendJobExecutionActor.scala:76)
	at cromwell.core.retry.Retry$.withRetry(Retry.scala:38)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.withRetry(AsyncBackendJobExecutionActor.scala:61)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.cromwell$backend$async$AsyncBackendJobExecutionActor$$robustPoll(AsyncBackendJobExecutionActor.scala:76)
	at cromwell.backend.async.AsyncBackendJobExecutionActor$$anonfun$receive$1.applyOrElse(AsyncBackendJobExecutionActor.scala:89)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
	at akka.actor.Actor.aroundReceive(Actor.scala:517)
	at akka.actor.Actor.aroundReceive$(Actor.scala:515)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.aroundReceive(ConfigAsyncJobExecutionActor.scala:211)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
	at akka.actor.ActorCell.invoke(ActorCell.scala:557)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: scala.NotImplementedError: This should not happen, please report this
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.pollStatus(ConfigAsyncJobExecutionActor.scala:281)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.pollStatus(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$pollStatusAsync$1(StandardAsyncExecutionActor.scala:697)
	at scala.util.Try$.apply(Try.scala:209)
	... 25 more

[2019-03-01 16:07:32,87] [warn] DispatchedConfigAsyncJobExecutionActor [cefb3c55prep_samples_to_rec:NA:1]: Fatal exception polling for status. Job will fail.
java.util.concurrent.ExecutionException: Boxed Error
	at scala.concurrent.impl.Promise$.resolver(Promise.scala:83)
	at scala.concurrent.impl.Promise$.scala$concurrent$impl$Promise$$resolveTry(Promise.scala:75)
	at scala.concurrent.impl.Promise$KeptPromise$.apply(Promise.scala:402)
	at scala.concurrent.Promise$.fromTry(Promise.scala:138)
	at scala.concurrent.Future$.fromTry(Future.scala:635)
	at cromwell.backend.standard.StandardAsyncExecutionActor.pollStatusAsync(StandardAsyncExecutionActor.scala:697)
	at cromwell.backend.standard.StandardAsyncExecutionActor.pollStatusAsync$(StandardAsyncExecutionActor.scala:697)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.pollStatusAsync(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.standard.StandardAsyncExecutionActor.poll(StandardAsyncExecutionActor.scala:989)
	at cromwell.backend.standard.StandardAsyncExecutionActor.poll$(StandardAsyncExecutionActor.scala:983)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.poll(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.$anonfun$robustPoll$1(AsyncBackendJobExecutionActor.scala:76)
	at cromwell.core.retry.Retry$.withRetry(Retry.scala:38)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.withRetry(AsyncBackendJobExecutionActor.scala:61)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.cromwell$backend$async$AsyncBackendJobExecutionActor$$robustPoll(AsyncBackendJobExecutionActor.scala:76)
	at cromwell.backend.async.AsyncBackendJobExecutionActor$$anonfun$receive$1.applyOrElse(AsyncBackendJobExecutionActor.scala:89)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
	at akka.actor.Actor.aroundReceive(Actor.scala:517)
	at akka.actor.Actor.aroundReceive$(Actor.scala:515)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.aroundReceive(ConfigAsyncJobExecutionActor.scala:211)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
	at akka.actor.ActorCell.invoke(ActorCell.scala:557)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: scala.NotImplementedError: This should not happen, please report this
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.pollStatus(ConfigAsyncJobExecutionActor.scala:281)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.pollStatus(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$pollStatusAsync$1(StandardAsyncExecutionActor.scala:697)
	at scala.util.Try$.apply(Try.scala:209)
	... 25 more

[2019-03-01 16:07:33,11] [error] WorkflowManagerActor Workflow cefb3c55-9742-4988-907c-46306e82869c failed (during ExecutingWorkflowState): java.util.concurrent.ExecutionException: Boxed Error
	at scala.concurrent.impl.Promise$.resolver(Promise.scala:83)
	at scala.concurrent.impl.Promise$.scala$concurrent$impl$Promise$$resolveTry(Promise.scala:75)
	at scala.concurrent.impl.Promise$KeptPromise$.apply(Promise.scala:402)
	at scala.concurrent.Promise$.fromTry(Promise.scala:138)
	at scala.concurrent.Future$.fromTry(Future.scala:635)
	at cromwell.backend.standard.StandardAsyncExecutionActor.pollStatusAsync(StandardAsyncExecutionActor.scala:697)
	at cromwell.backend.standard.StandardAsyncExecutionActor.pollStatusAsync$(StandardAsyncExecutionActor.scala:697)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.pollStatusAsync(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.standard.StandardAsyncExecutionActor.poll(StandardAsyncExecutionActor.scala:989)
	at cromwell.backend.standard.StandardAsyncExecutionActor.poll$(StandardAsyncExecutionActor.scala:983)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.poll(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.$anonfun$robustPoll$1(AsyncBackendJobExecutionActor.scala:76)
	at cromwell.core.retry.Retry$.withRetry(Retry.scala:38)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.withRetry(AsyncBackendJobExecutionActor.scala:61)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.cromwell$backend$async$AsyncBackendJobExecutionActor$$robustPoll(AsyncBackendJobExecutionActor.scala:76)
	at cromwell.backend.async.AsyncBackendJobExecutionActor$$anonfun$receive$1.applyOrElse(AsyncBackendJobExecutionActor.scala:89)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
	at akka.actor.Actor.aroundReceive(Actor.scala:517)
	at akka.actor.Actor.aroundReceive$(Actor.scala:515)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.aroundReceive(ConfigAsyncJobExecutionActor.scala:211)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
	at akka.actor.ActorCell.invoke(ActorCell.scala:557)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: scala.NotImplementedError: This should not happen, please report this
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.pollStatus(ConfigAsyncJobExecutionActor.scala:281)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.pollStatus(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$pollStatusAsync$1(StandardAsyncExecutionActor.scala:697)
	at scala.util.Try$.apply(Try.scala:209)
	... 25 more

java.util.concurrent.ExecutionException: Boxed Error
	at scala.concurrent.impl.Promise$.resolver(Promise.scala:83)
	at scala.concurrent.impl.Promise$.scala$concurrent$impl$Promise$$resolveTry(Promise.scala:75)
	at scala.concurrent.impl.Promise$KeptPromise$.apply(Promise.scala:402)
	at scala.concurrent.Promise$.fromTry(Promise.scala:138)
	at scala.concurrent.Future$.fromTry(Future.scala:635)
	at cromwell.backend.standard.StandardAsyncExecutionActor.pollStatusAsync(StandardAsyncExecutionActor.scala:697)
	at cromwell.backend.standard.StandardAsyncExecutionActor.pollStatusAsync$(StandardAsyncExecutionActor.scala:697)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.pollStatusAsync(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.standard.StandardAsyncExecutionActor.poll(StandardAsyncExecutionActor.scala:989)
	at cromwell.backend.standard.StandardAsyncExecutionActor.poll$(StandardAsyncExecutionActor.scala:983)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.poll(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.$anonfun$robustPoll$1(AsyncBackendJobExecutionActor.scala:76)
	at cromwell.core.retry.Retry$.withRetry(Retry.scala:38)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.withRetry(AsyncBackendJobExecutionActor.scala:61)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.cromwell$backend$async$AsyncBackendJobExecutionActor$$robustPoll(AsyncBackendJobExecutionActor.scala:76)
	at cromwell.backend.async.AsyncBackendJobExecutionActor$$anonfun$receive$1.applyOrElse(AsyncBackendJobExecutionActor.scala:89)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
	at akka.actor.Actor.aroundReceive(Actor.scala:517)
	at akka.actor.Actor.aroundReceive$(Actor.scala:515)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.aroundReceive(ConfigAsyncJobExecutionActor.scala:211)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
	at akka.actor.ActorCell.invoke(ActorCell.scala:557)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: scala.NotImplementedError: This should not happen, please report this
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.pollStatus(ConfigAsyncJobExecutionActor.scala:281)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.pollStatus(ConfigAsyncJobExecutionActor.scala:211)
	at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$pollStatusAsync$1(StandardAsyncExecutionActor.scala:697)
	at scala.util.Try$.apply(Try.scala:209)
	... 25 more

I found something else that seems relevant in SGE's qsub man page: the option -b y. Here's what the manual says:
       -b y[es]|n[o]
              Available for qsub, qrsh only. Qalter does not allow changing this option. This option cannot be embedded in the script file itself.

              Gives  the  user  the possibility to indicate explicitly whether command should be treated as binary or script. If the value of -b is 'y', then command  may be a binary or
              script.  The command might not be accessible from the submission host.  Nothing except the path of the command will be transferred from the submission host to  the  execu‐
              tion host. Path aliasing will be applied to the path of command before command will be executed.

              If  the  value  of  -b  is  'n' then command needs to be a script and it will be handled as script. The script file has to be accessible by the submission host. It will be
              transferred to the execution host. qsub/qrsh will search directive prefixes within script.

              qsub will implicitly use -b n whereas qrsh will apply the -b y option if nothing else is specified.

              The value specified with this option or the corresponding value specified in qmon will only be passed to defined JSV instances if the value is yes.  The name of the param‐
              eter  will  be  b.  The  value  will  be y also when then long form yes was specified during submission.  (see -jsv option below or find more information concerning JSV in
              jsv(1))

              Please note that submission of command as script (-b n) can have a significant performance impact, especially for short running jobs and big job scripts.   Script  submis‐
              sion adds a number of operations to the submission process: The job script needs to be
              - parsed at client side (for special comments)
              - transferred from submit client to qmaster
              - spooled in qmaster
              - transferred to execd at job execution
              - spooled in execd
              - removed from spooling both in execd and qmaster once the job is done
              If job scripts are available on the execution nodes, e.g. via NFS, binary submission can be the better choice.

I tried adding this to the qsub command from the file script.submit and this got it to run fine when executed manually.
I then added back the /usr/bin/env bash bit on line 352 and I changed line 349 to:
        qsub -b y -V -w w -j y -N ${job_name} -wd ${cwd} \

Unfortunately, this resulted in the same error (java.util.concurrent.ExecutionException: Boxed Error) and identical stack trace as in the previous post.

Thanks for all this help and debugging. This is really helpful. It sounds like two things were problematic:

The submission command either needed -b y and bash, or just to remove the bash call. I've done the latter to try and make it simpler.
You're then hitting this bug in the latest Cromwell release (https://github.com/broadinstitute/cromwell/issues/4560 and https://github.com/broadinstitute/cromwell/issues/4651).

So I pushed a fix for the first issue and you can avoid the second by downgrading Cromwell:
bcbio_conda install -n bcbiovm -c conda-forge -c bioconda -y cromwell=0.36

and then I hope it will work for you. Please let me know if you hit any issues after this, and thanks again for your work helping debug this.

OK, I'm running the bcbio_conda command now and, while running, I noticed that it's downgrading a bunch of other packages (including bcbio-nextgen) along with cromwell:
The following packages will be DOWNGRADED:

  atk                                 2.25.90-hb9dd440_1002 --> 2.25.90-hf2eb9ee_1001
  bcbio-nextgen                               1.1.4a-py27_0 --> 1.1.0a-py27_8
  bcbio-nextgen-vm                             0.1.3-py27_1 --> 0.1.0a-py27_80
  cairo                                1.16.0-ha4e643d_1000 --> 1.14.12-h80bd089_1005
  cromwell                                           0.37-0 --> 0.36.1-0
  cryptography                           2.5-py27hb7f436b_1 --> 2.5-py27h9d9f1b6_1
  glib                                 2.58.2-hf63aee3_1001 --> 2.56.2-had28632_1001
  gobject-introspec~               1.56.1-py27h2da5eee_1002 --> 1.56.1-py27h9e29830_1001
  graphviz                             2.38.0-hf68f40c_1011 --> 2.38.0-hcf1ce16_1009
  gtk2                                2.24.31-hb68c50a_1001 --> 2.24.31-h5baeb44_1000
  harfbuzz                                 2.3.1-h6824563_0 --> 1.9.0-he243708_1001
  pango                               1.40.14-h4ea9474_1004 --> 1.40.14-hf0c64fd_1003
  pycurl                            <IP_ADDRESS>-py27hb7f436b_0 --> <IP_ADDRESS>-py27h1ba5d50_0
  samtools                                   1.9-h57cc563_6 --> 1.9-h91753b0_5

Apologies, sorry about that. I'm not sure why it's dragging the other tools down, we don't want that. I think if you add --no-deps it won't do that:
bcbio_conda install -n bcbiovm -c conda-forge -c bioconda --no-deps cromwell=0.36

We'd want the latest bcbio/bcbio-vm but with the older cromwell. Apologies, the version resolution starts to get complex, I'm hoping we can get a fixed cromwell soon without this issue.

No worries and thanks for the update. It was too late to cancel the previous command (I was running it directly on the cluster's NFS system). Luckily, I had a back-up on a cloud VM.
Running bcbio_conda with the --no-deps option didn't downgrade any other packages, thanks for mentioning it. I've rsynced bcbio_nextgen from the VM back to the cluster's NFS system and tested the CWL workflow again.
This time, it results in the following error:
[2019-03-02 19:48:31,25] [info] Slf4jLogger started
[2019-03-02 19:48:31,33] [info] Pre Processing Workflow...
[2019-03-02 19:48:31,59] [info] Pre-Processing /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/main-testingVC-merged.cwl
[2019-03-02 19:48:39,31] [info] Pre Processing Inputs...
Exception in thread "main" cromwell.CromwellEntryPoint$$anon$1: ERROR: Unable to submit workflow to Cromwell::
running cwltool on file /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/main-testingVC-merged.cwl failed with ammonite.ops.ShelloutException: CommandResult 1
Traceback (most recent call last):
  File "/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/extra2/bin/cwltool", line 7, in <module>
    from cwltool.main import run
  File "/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/extra2/lib/python2.7/site-packages/cwltool/main.py", line 24, in <module>
    from schema_salad import validate
  File "/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/extra2/lib/python2.7/site-packages/schema_salad/__init__.py", line 10, in <module>
    from .utils import onWindows
  File "/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/extra2/lib/python2.7/site-packages/schema_salad/utils.py", line 8, in <module>
    from typing_extensions import Text  # pylint: disable=unused-import
  File "/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/extra2/lib/python2.7/site-packages/typing_extensions.py", line 4, in <module>
    from typing import (
ImportError: cannot import name Counter

	at cromwell.CromwellEntryPoint$.$anonfun$validOrFailSubmission$1(CromwellEntryPoint.scala:241)
	at cats.data.Validated.valueOr(Validated.scala:48)
	at cromwell.CromwellEntryPoint$.validOrFailSubmission(CromwellEntryPoint.scala:241)
	at cromwell.CromwellEntryPoint$.validateRunArguments(CromwellEntryPoint.scala:237)
	at cromwell.CromwellEntryPoint$.runSingle(CromwellEntryPoint.scala:61)
	at cromwell.CromwellApp$.runCromwell(CromwellApp.scala:14)
	at cromwell.CromwellApp$.delayedEndpoint$cromwell$CromwellApp$1(CromwellApp.scala:25)
	at cromwell.CromwellApp$delayedInit$body.apply(CromwellApp.scala:3)
	at scala.Function0.apply$mcV$sp(Function0.scala:34)
	at scala.Function0.apply$mcV$sp$(Function0.scala:34)
	at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
	at scala.App.$anonfun$main$1$adapted(App.scala:76)
	at scala.collection.immutable.List.foreach(List.scala:389)
	at scala.App.main(App.scala:76)
	at scala.App.main$(App.scala:74)
	at cromwell.CromwellApp$.main(CromwellApp.scala:3)
	at cromwell.CromwellApp.main(CromwellApp.scala)

Thanks for fixing and re-testing. This looks like you're picking up a broken cwltool install from outside of the standard bcbio install:
/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/extra2/bin/cwltool

It appears to have an incompatible schema-salad dependency with the cwltool. Cromwell calls out to cwltool as an external dependency for pre-processing the inputs. If you remove the problem cwltool from your PATH, or fix that install, hopefully it'll run cleanly. Thanks again.

Thanks for noticing that. Not sure how this happened, but there doesn't seem to be a cwltool executable inside bcbio_nextgen/anaconda/bin or bcbio_nextgen/tools/bin. That's why it's picking up the one inside bcbio_nextgen/extra2/bin.
How can I reinstall cwltool (and, possibly, toil as well) in the standard bcbio install?

cwltool should get installed inside the bcbio environment where you have cromwell at:
/home/ncit/external/a.mizeranschi/bcbio_nextgen/anaconda/envs/bcbiovm/bin

If it's missing after all the updates/changes we've been doing, you can re-install with:
bcbio_conda install -n bcbiovm -c conda-forge -c bioconda cwltool

Hope this fixes it for you.

OK, I found the cwltool executable inside bcbio_nextgen/anaconda/envs/bcbiovm/bin. However, there was no symlink to it inside bcbio_nextgen/tools/bin, which is what I had set up in my PATH, along with the directory bcbio_nextgen/extra2/bin mentioned above.
At first sight, it appears that none of the executables from bcbio_nextgen/anaconda/envs/bcbiovm/bin were symlinked inside bcbio_nextgen/tools/bin.
I've manually added the directory bcbio_nextgen/anaconda/envs/bcbiovm/bin to the PATH variable in what I think should be the right order (i.e. after bcbio_nextgen/tools/bin and bcbio_nextgen/anaconda/bin, but before the directory bcbio_nextgen/extra2/bin), so that the which cwltool command points to the right place:
$ which cwltool
/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/anaconda/envs/bcbiovm/bin/cwltool

Then, I reran the analysis using Cromwell. However, I still got the cwltool error from above (ImportError: cannot import name Counter), which suggests that Bcbio's version of cwltool got broken as well.
I tried reinstalling this with --no-deps, to prevent conda from downgrading bcbio-nextgen and bcbio-nextgen-vm:
bcbio_conda install -n bcbiovm -c conda-forge -c bioconda cwltool --no-deps

and it installed a newer version:
The following packages will be UPDATED:

  cwltool                      1.0.20181217162649-py27_1001 --> 1.0.20190228155703-py27_0

However, even having done this, I still got the error from above (ImportError: cannot import name Counter).
Finally, I went ahead and created a new installation from scratch and copied over the genome data from the broken one.
The cwltool and other bcbio-vm-related executables still weren't symlinked into bcbio_nextgen/tools/bin.
I've redone the modifications to the file /export/home/ncit/external/a.mizeranschi/bcbio_nextgen/anaconda/envs/bcbiovm/lib/python2.7/site-packages/bcbio/cwl/hpc.py and then tested again with running through Cromwell on the cluster, after making sure that the directory bcbio_nextgen/anaconda/envs/bcbiovm/bin is listed in $PATH.
I ran into the java.util.concurrent.ExecutionException: Boxed Error again, and after downgrading Cromwell to v. 0.36 with bcbio_conda install -n bcbiovm -c conda-forge -c bioconda --no-deps cromwell=0.36, things ended with a different error, as below.
Note: this time, the mentioned stderr file is empty (/export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/84d4fab6-5188-44e2-b8f9-2cd4f9e28d82/call-prep_samples_to_rec/execution/stderr.submit). The stdout file, however, is not:
$ cat /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/84d4fab6-5188-44e2-b8f9-2cd4f9e28d82/call-prep_samples_to_rec/execution/stdout.submit
warning: no suitable queues
Your job 1048860 ("cromwell_84d4fab6_prep_samples_to_rec") has been submitted

This is the error I get:
[2019-03-03 22:52:51,85] [error] DispatchedConfigAsyncJobExecutionActor [84d4fab6prep_samples_to_rec:NA:1]: Error attempting to Execute
java.lang.RuntimeException: Could not find job ID from stdout file. Check the stderr file for possible errors: /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/84d4fab6-5188-44e2-b8f9-2cd4f9e28d82/call-prep_samples_to_rec/execution/stderr.submit
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.getJob(ConfigAsyncJobExecutionActor.scala:231)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.$anonfun$execute$2(SharedFileSystemAsyncJobExecutionActor.scala:143)
	at scala.util.Either.fold(Either.scala:188)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute(SharedFileSystemAsyncJobExecutionActor.scala:136)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute$(SharedFileSystemAsyncJobExecutionActor.scala:131)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.execute(ConfigAsyncJobExecutionActor.scala:213)
	at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$executeAsync$1(StandardAsyncExecutionActor.scala:628)
	at scala.util.Try$.apply(Try.scala:209)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync(StandardAsyncExecutionActor.scala:628)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync$(StandardAsyncExecutionActor.scala:628)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.executeAsync(ConfigAsyncJobExecutionActor.scala:213)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover(StandardAsyncExecutionActor.scala:943)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover$(StandardAsyncExecutionActor.scala:935)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.executeOrRecover(ConfigAsyncJobExecutionActor.scala:213)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.$anonfun$robustExecuteOrRecover$1(AsyncBackendJobExecutionActor.scala:65)
	at cromwell.core.retry.Retry$.withRetry(Retry.scala:38)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.withRetry(AsyncBackendJobExecutionActor.scala:61)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.cromwell$backend$async$AsyncBackendJobExecutionActor$$robustExecuteOrRecover(AsyncBackendJobExecutionActor.scala:65)
	at cromwell.backend.async.AsyncBackendJobExecutionActor$$anonfun$receive$1.applyOrElse(AsyncBackendJobExecutionActor.scala:88)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
	at akka.actor.Actor.aroundReceive(Actor.scala:517)
	at akka.actor.Actor.aroundReceive$(Actor.scala:515)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.aroundReceive(ConfigAsyncJobExecutionActor.scala:213)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
	at akka.actor.ActorCell.invoke(ActorCell.scala:557)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[2019-03-03 22:52:51,85] [error] DispatchedConfigAsyncJobExecutionActor [84d4fab6alignment_to_rec:NA:1]: Error attempting to Execute
java.lang.RuntimeException: Could not find job ID from stdout file. Check the stderr file for possible errors: /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/84d4fab6-5188-44e2-b8f9-2cd4f9e28d82/call-alignment_to_rec/execution/stderr.submit
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.getJob(ConfigAsyncJobExecutionActor.scala:231)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.$anonfun$execute$2(SharedFileSystemAsyncJobExecutionActor.scala:143)
	at scala.util.Either.fold(Either.scala:188)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute(SharedFileSystemAsyncJobExecutionActor.scala:136)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute$(SharedFileSystemAsyncJobExecutionActor.scala:131)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.execute(ConfigAsyncJobExecutionActor.scala:213)
	at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$executeAsync$1(StandardAsyncExecutionActor.scala:628)
	at scala.util.Try$.apply(Try.scala:209)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync(StandardAsyncExecutionActor.scala:628)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync$(StandardAsyncExecutionActor.scala:628)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.executeAsync(ConfigAsyncJobExecutionActor.scala:213)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover(StandardAsyncExecutionActor.scala:943)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover$(StandardAsyncExecutionActor.scala:935)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.executeOrRecover(ConfigAsyncJobExecutionActor.scala:213)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.$anonfun$robustExecuteOrRecover$1(AsyncBackendJobExecutionActor.scala:65)
	at cromwell.core.retry.Retry$.withRetry(Retry.scala:38)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.withRetry(AsyncBackendJobExecutionActor.scala:61)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.cromwell$backend$async$AsyncBackendJobExecutionActor$$robustExecuteOrRecover(AsyncBackendJobExecutionActor.scala:65)
	at cromwell.backend.async.AsyncBackendJobExecutionActor$$anonfun$receive$1.applyOrElse(AsyncBackendJobExecutionActor.scala:88)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
	at akka.actor.Actor.aroundReceive(Actor.scala:517)
	at akka.actor.Actor.aroundReceive$(Actor.scala:515)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.aroundReceive(ConfigAsyncJobExecutionActor.scala:213)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
	at akka.actor.ActorCell.invoke(ActorCell.scala:557)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Thanks for all this work and the detailed report. It sounds like you got cwltool working with the re-install but in case you hit this in the future the likely cause is that you don't have compatible cwltool and schema-salad versions. I'd suggest updating both with:
bcbio_conda install -n bcbiovm -c conda-forge -c bioconda cwltool schema-salad

since they need to stay in sync.
For your current error, it looks like it's now launching and sending jobs but the SGE run does not find a queue to launch the job in (warning: no suitable queues) so it's not actually making it to the grid and then Cromwell complains. I'd suggest trying the current qsub command line to see if you can identify the issues (queue name? resource specification?) that prevents it from being able to schedule. Sorry to not have more specific advice but this one is hard to debug remotely. Hope this helps point you in the right direction.

Note: I also saw the message warning: no suitable queues while testing with ipython-parallel (see: https://github.com/bcbio/bcbio-nextgen/issues/2701#issuecomment-467533829). However, the problem there was unrelated to the queue and the jobs were starting up fine in spite of the warning message, so I'm guessing that it could just be a red herring.
This could be the case here as well. I tried to manually run the qsub command from the script.submit file. The job starts up and appears to be running (its state is r), so it definitely makes it to the cluster:
$ cat /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/84d4fab6-5188-44e2-b8f9-2cd4f9e28d82/call-prep_samples_to_rec/execution/script.submit
#!/bin/bash
qsub -V -w w -j y -N cromwell_84d4fab6_prep_samples_to_rec -wd /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/84d4fab6-5188-44e2-b8f9-2cd4f9e28d82/call-prep_samples_to_rec         -o /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/84d4fab6-5188-44e2-b8f9-2cd4f9e28d82/call-prep_samples_to_rec/execution/stdout -e /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/84d4fab6-5188-44e2-b8f9-2cd4f9e28d82/call-prep_samples_to_rec/execution/stderr -q campus-haswell.q         -pe openmp 1 -l mem_free=3221m         /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/84d4fab6-5188-44e2-b8f9-2cd4f9e28d82/call-prep_samples_to_rec/execution/script

$ qsub -V -w w -j y -N cromwell_84d4fab6_prep_samples_to_rec -wd /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/84d4fab6-5188-44e2-b8f9-2cd4f9e28d82/call-prep_samples_to_rec         -o /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/84d4fab6-5188-44e2-b8f9-2cd4f9e28d82/call-prep_samples_to_rec/execution/stdout -e /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/84d4fab6-5188-44e2-b8f9-2cd4f9e28d82/call-prep_samples_to_rec/execution/stderr -q campus-haswell.q         -pe openmp 1 -l mem_free=3221m         /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/84d4fab6-5188-44e2-b8f9-2cd4f9e28d82/call-prep_samples_to_rec/execution/script
warning: no suitable queues
Your job 1048863 ("cromwell_84d4fab6_prep_samples_to_rec") has been submitted

$ qstat -u a.mizeranschi
job-ID  prior   name	   user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
1048863 0.50500 cromwell_8 a.mizeransch r     03/04/2019 08:42:29 campus-haswell.q@haswell-wn35.     1

I found a similar error on the Cromwell forums (including the empty stderr file): https://gatkforums.broadinstitute.org/wdl/discussion/13716/could-not-find-job-id-from-stdout-file-cromwell-with-an-unusual-backend. Here they mentioned it could be a Cromwell bug, so this may have been fixed in v. 0.37, however I've downgraded to v. 0.36 yesterday, in order to bypass a different bug (java.util.concurrent.ExecutionException: Boxed Error).
I also found another related issue here: https://github.com/ENCODE-DCC/atac-seq-pipeline/issues/40. The reply from there mentioned the mem_mb setting. I tried adding this as -r memtype=mem_mb, but it doesn't look like this is supported. I got a slightly different error this time: java.lang.RuntimeException: Unable to start job and, this time, the stderr files specifically complained about the mem_mb option:
$ cat /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/d41133ee-5182-4add-9ede-78df80af2caf/call-prep_samples_to_rec/execution/stderr.submit
Unable to run job: unknown resource "mem_mb".
Exiting.

Update: this previous error (java.lang.RuntimeException: Could not find job ID from stdout file) seems to be a Cromwell-related issue.
I gave Toil v. 3.14 a try, as I had it installed in a separate conda environment. I've added the option -- --batchSystem gridEngine to bcbio_vm.py and set up environment variables for the SGE parallel environment and queue, as mentioned in the Toil documentation.
 Some of the Toil workers were failing:
2019-03-04 10:55:28,805 - toil.leader - WARNING - V/f/jobdniYB4    WARNING:cwltool:[job prep_samples.cwl] completed permanentFail
haswell-wn42.grid.pub.ro 2019-03-04 10:55:28,805 MainThread WARNING toil.leader: V/f/jobdniYB4    Traceback (most recent call last):
2019-03-04 10:55:28,805 - toil.leader - WARNING - V/f/jobdniYB4    Traceback (most recent call last):
haswell-wn42.grid.pub.ro 2019-03-04 10:55:28,806 MainThread WARNING toil.leader: V/f/jobdniYB4      File "/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/extra2/lib/python2.7/site-packages/toil/worker.py", line 309, in workerScript
2019-03-04 10:55:28,806 - toil.leader - WARNING - V/f/jobdniYB4      File "/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/extra2/lib/python2.7/site-packages/toil/worker.py", line 309, in workerScript
haswell-wn42.grid.pub.ro 2019-03-04 10:55:28,806 MainThread WARNING toil.leader: V/f/jobdniYB4        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
2019-03-04 10:55:28,806 - toil.leader - WARNING - V/f/jobdniYB4        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
haswell-wn42.grid.pub.ro 2019-03-04 10:55:28,806 MainThread WARNING toil.leader: V/f/jobdniYB4      File "/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/extra2/lib/python2.7/site-packages/toil/job.py", line 1328, in _runner
2019-03-04 10:55:28,806 - toil.leader - WARNING - V/f/jobdniYB4      File "/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/extra2/lib/python2.7/site-packages/toil/job.py", line 1328, in _runner
haswell-wn42.grid.pub.ro 2019-03-04 10:55:28,806 MainThread WARNING toil.leader: V/f/jobdniYB4        returnValues = self._run(jobGraph, fileStore)
2019-03-04 10:55:28,806 - toil.leader - WARNING - V/f/jobdniYB4        returnValues = self._run(jobGraph, fileStore)
haswell-wn42.grid.pub.ro 2019-03-04 10:55:28,806 MainThread WARNING toil.leader: V/f/jobdniYB4      File "/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/extra2/lib/python2.7/site-packages/toil/job.py", line 1273, in _run
2019-03-04 10:55:28,806 - toil.leader - WARNING - V/f/jobdniYB4      File "/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/extra2/lib/python2.7/site-packages/toil/job.py", line 1273, in _run
haswell-wn42.grid.pub.ro 2019-03-04 10:55:28,806 MainThread WARNING toil.leader: V/f/jobdniYB4        return self.run(fileStore)
2019-03-04 10:55:28,806 - toil.leader - WARNING - V/f/jobdniYB4        return self.run(fileStore)
haswell-wn42.grid.pub.ro 2019-03-04 10:55:28,806 MainThread WARNING toil.leader: V/f/jobdniYB4      File "/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/extra2/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 464, in run
2019-03-04 10:55:28,806 - toil.leader - WARNING - V/f/jobdniYB4      File "/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/extra2/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 464, in run
haswell-wn42.grid.pub.ro 2019-03-04 10:55:28,807 MainThread WARNING toil.leader: V/f/jobdniYB4        raise cwltool.errors.WorkflowException(status)
2019-03-04 10:55:28,807 - toil.leader - WARNING - V/f/jobdniYB4        raise cwltool.errors.WorkflowException(status)
haswell-wn42.grid.pub.ro 2019-03-04 10:55:28,807 MainThread WARNING toil.leader: V/f/jobdniYB4    WorkflowException: permanentFail
2019-03-04 10:55:28,807 - toil.leader - WARNING - V/f/jobdniYB4    WorkflowException: permanentFail
haswell-wn42.grid.pub.ro 2019-03-04 10:55:28,807 MainThread WARNING toil.leader: V/f/jobdniYB4    ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn39.grid.pub.ro
2019-03-04 10:55:28,807 - toil.leader - WARNING - V/f/jobdniYB4    ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn39.grid.pub.ro
haswell-wn42.grid.pub.ro 2019-03-04 10:55:28,807 MainThread WARNING toil.leader: V/f/jobdniYB4    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl V/f/jobdniYB4 with ID V/f/jobdniYB4 to 1
2019-03-04 10:55:28,807 - toil.leader - WARNING - V/f/jobdniYB4    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl V/f/jobdniYB4 with ID V/f/jobdniYB4 to 1
haswell-wn42.grid.pub.ro 2019-03-04 10:55:28,937 MainThread INFO toil.leader: Issued job 'file:///export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl V/f/jobdniYB4 with job batch system ID: 44 and cores: 1, disk: 2.0 G, and memory: 3.0 G
2019-03-04 10:55:28,937 - toil.leader - INFO - Issued job 'file:///export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl V/f/jobdniYB4 with job batch system ID: 44 and cores: 1, disk: 2.0 G, and memory: 3.0 G

However, other jobs seemed to be working:
haswell-wn42.grid.pub.ro 2019-03-04 11:02:52,552 MainThread INFO toil.leader: Job ended successfully: 'file:///export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/merge_split_alignments.cwl' bcbio_nextgen.py runfn merge_split_alignments cwl 6/v/jobLcMfKr
2019-03-04 11:02:52,552 - toil.leader - INFO - Job ended successfully: 'file:///export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/merge_split_alignments.cwl' bcbio_nextgen.py runfn merge_split_alignments cwl 6/v/jobLcMfKr
haswell-wn42.grid.pub.ro 2019-03-04 11:03:40,075 MainThread INFO toil.leader: Issued job 'file:///export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/prep_align_inputs.cwl' bcbio_nextgen.py runfn prep_align_inputs cwl p/o/jobqvCvT3 with job batch system ID: 68 and cores: 16, disk: 2.0 G, and memory: 48.0 G
2019-03-04 11:03:40,075 - toil.leader - INFO - Issued job 'file:///export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/prep_align_inputs.cwl' bcbio_nextgen.py runfn prep_align_inputs cwl p/o/jobqvCvT3 with job batch system ID: 68 and cores: 16, disk: 2.0 G, and memory: 48.0 G
haswell-wn42.grid.pub.ro 2019-03-04 11:05:51,698 MainThread INFO toil.leader: Job ended successfully: 'file:///export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/process_alignment.cwl' bcbio_nextgen.py runfn process_alignment cwl s/0/jobXmr4NI
2019-03-04 11:05:51,698 - toil.leader - INFO - Job ended successfully: 'file:///export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/process_alignment.cwl' bcbio_nextgen.py runfn process_alignment cwl s/0/jobXmr4NI

Update2: the previous Toil errors were my fault, due to using the wrong variant_regions file:
ValueError: Contigs in BED file /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/toil_work/tmp2CVwN1/stgbeed72d8-073e-43d8-96c7-595c7647a9fe/variant_regions.bed not in reference genome:
 ['7']
This is typically due to chr1 versus 1 differences in BED file and reference.

After fixing this and rerunning, the engines don't error out anymore. The process_alignment phase just finished and merge_split_alignments is starting.
However, the whole thing seems to be running quite a bit slower than when using IPython-parallel, even with a very small input dataset (the same as used here: https://github.com/bcbio/bcbio-nextgen/issues/2709). Is this expected?

Update3: the Toil job ran for a while and eventually ended with a memory-related error (java.lang.OutOfMemoryError: unable to create new native thread):
[job concat_batch_variantcalls.cwl] /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/toil_work/out_tmpdirjfXSoN$ bcbio_nextgen.py \
    runfn \
    concat_batch_variantcalls \
    cwl \
    sentinel_runtime=cores,1,ram,3072 \
    sentinel_parallel=batch-merge \
    sentinel_outputs=vrn_file \
    sentinel_inputs=batch_rec:record,region_block:var,vrn_file_region:var \
    run_number=0
INFO:cwltool:[job concat_batch_variantcalls.cwl] /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/toil_work/out_tmpdirjfXSoN$ bcbio_nextgen.py \
    runfn \
    concat_batch_variantcalls \
    cwl \
    sentinel_runtime=cores,1,ram,3072 \
    sentinel_parallel=batch-merge \
    sentinel_outputs=vrn_file \
    sentinel_inputs=batch_rec:record,region_block:var,vrn_file_region:var \
    run_number=0
Traceback (most recent call last):
  File "/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/tools/bin/bcbio_nextgen.py", line 223, in <module>
    runfn.process(kwargs["args"])
  File "/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/runfn.py", line 57, in process
    out = fn(*fnargs)
  File "/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/anaconda/lib/python2.7/site-packages/bcbio/utils.py", line 55, in wrapper
    return f(*args, **kwargs)
  File "/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/multitasks.py", line 276, in concat_batch_variantcalls
    return genotype.concat_batch_variantcalls(*args)
  File "/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/genotype.py", line 397, in concat_batch_variantcalls
    dd.get_ref_file(items[0]), items[0]["config"])
  File "/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/vcfutils.py", line 371, in concat_variant_files
    out_file = _run_concat_variant_files_gatk4(input_file_list, out_file, config)
  File "/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/vcfutils.py", line 399, in _run_concat_variant_files_gatk4
    broad_runner.run_gatk(params)
  File "/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/anaconda/lib/python2.7/site-packages/bcbio/broad/__init__.py", line 363, in run_gatk
    log_error=log_error)
  File "/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 26, in run
    _do_run(cmd, checks, log_stdout, env=env)
  File "/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 106, in _do_run
    raise subprocess.CalledProcessError(exitcode, error_msg)
subprocess.CalledProcessError: Command 'set -o pipefail; unset JAVA_HOME && export PATH=/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/anaconda/bin:$PATH && gatk --java-options '-Xms681m -Xmx1g -XX:+UseSerialGC -Djava.io.tmpdir=/export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/toil_work/out_tmpdirjfXSoN/bcbiotx/tmpINxAlu' GatherVcfs -I /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/toil_work/out_tmpdirjfXSoN/samtools/testingVC-files.list -O /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/toil_work/out_tmpdirjfXSoN/bcbiotx/tmpS64q6S/testingVC.vcf.gz
Using GATK jar /export/home/ncit/external/a.mizeranschi/bcbio_nextgen/anaconda/share/gatk4-<IP_ADDRESS>-0/gatk-package-<IP_ADDRESS>-local.jar
Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -Xms681m -Xmx1g -XX:+UseSerialGC -Djava.io.tmpdir=/export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/toil_work/out_tmpdirjfXSoN/bcbiotx/tmpINxAlu -jar /export/home/ncit/external/a.mizeranschi/bcbio_nextgen/anaconda/share/gatk4-<IP_ADDRESS>-0/gatk-package-<IP_ADDRESS>-local.jar GatherVcfs -I /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/toil_work/out_tmpdirjfXSoN/samtools/testingVC-files.list -O /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/toil_work/out_tmpdirjfXSoN/bcbiotx/tmpS64q6S/testingVC.vcf.gz
Error occurred during initialization of VM
java.lang.OutOfMemoryError: unable to create new native thread
' returned non-zero exit status 1
[job concat_batch_variantcalls.cwl] Job error:
Error validating output record. the `vrn_file` field is not valid because
  is not a dict
 in {
    "vrn_file": []
}

Thanks for all this work debugging and sorry about the problems with both runners. All of my recent experience has been with Cromwell, so trying to debug more there is our best strategy.
As best as I can tell with the Toil issue, it appears to not correctly pass memory to that job. Even though the specification asks for 3072Mb of memory, the actual GATK java command only gets 1Gb which results in the failure you're seeing.
For Cromwell, my suggestion is to try and raise and isolate a small test case and report as a follow up to the issue with 0.37. Our best strategy is to try and raise importance of that and hopefully get a clean fix so we don't have to keep backtracking on versions into potentially old issues. If you're able to cleanly reproduce these on any of the small test cases (https://github.com/bcbio/test_bcbio_cwl) that might make it easier to debug and identify a fix.
Sorry about all these issues running on SGE. I appreciate your help working through them.

Regarding Cromwell, just to clarify, do you mean I should try to reproduce the issue using the small test cases and the latest Bcbio development install (incl. Cromwell 0.37 and your latest updates)?
If so, then I'll give this a try as soon as I manage to update bcbio with -u development. I tried doing this yesterday in order to get a recent commit (https://github.com/bcbio/bcbio-nextgen/commit/5afdc1f6df88e7c94fd1891e17f52115a0ce0161), but it failed, for some reason. I've created a separate issue report for that.
As for Toil, what do you think could be the cause for java getting only 1GB (-Xmx1g)? I mean, would this be a Toil-related problem, or an SGE-related problem?

There is a new Cromwell 0.38 release which hopefully fixes the issues you were seeing with that. If you update with:
bcbio_conda install -n bcbiovm -c conda-forge -c bioconda cromwell

you should hopefully get this version and it'll work cleaner for those runs. Fingers crossed this will improve the run situation for you on SGE, and please let us know if you keep hitting issues. Thanks again for the patience debugging.

I made another bcbio upgrade with -u development and it got the most recent version of Cromwell, as well as the SGE-related modifications you made earlier to the file hpc.py.
Things are working well with Cromwell now on the SGE cluster, thanks a lot for your help and the Cromwell devs for the new version. However, it seems that Cromwell (just like Toil) is quite a bit slower and less efficient during my tests, compared to running with IPython parallel.
I'm running a test with 100 small FASTQ files. I notice it spawns a batch of dozens of worker processes, but only 3 at a time are actually running, while the rest just sit there, waiting. When the three running processes finish, the next 3 start running, and when they finish the next 3 start, and so on. Is this expected? Or could this be caused by something else in my setup?
The queue I'm running the worker processes on contains nodes with 24 cores each, and I've set up 8 cores per job in the bcbio_system.yaml file, so it would be expected to see 3 worker processes running on each node.
job-ID  prior   name	   user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
1050060 0.52079 run-this.s a.mizeransch r     03/09/2019 02:08:29 campus-haswell.q@haswell-wn42.     4
1050405 0.54184 cromwell_a a.mizeransch r     03/09/2019 11:00:59<EMAIL_ADDRESS>   8
1050406 0.54184 cromwell_d a.mizeransch r     03/09/2019 11:00:59<EMAIL_ADDRESS>   8
1050407 0.54184 cromwell_d a.mizeransch r     03/09/2019 11:00:59<EMAIL_ADDRESS>   8
1050408 0.54184 cromwell_5 a.mizeransch qw    03/09/2019 06:36:18                                    8
1050409 0.54184 cromwell_0 a.mizeransch qw    03/09/2019 06:38:30                                    8
1050410 0.54184 cromwell_1 a.mizeransch qw    03/09/2019 06:51:54                                    8
1050411 0.54184 cromwell_e a.mizeransch qw    03/09/2019 06:52:23                                    8
1050412 0.54184 cromwell_9 a.mizeransch qw    03/09/2019 06:53:00                                    8
1050413 0.54184 cromwell_b a.mizeransch qw    03/09/2019 07:05:52                                    8
1050414 0.54184 cromwell_f a.mizeransch qw    03/09/2019 07:05:59                                    8
1050415 0.54184 cromwell_4 a.mizeransch qw    03/09/2019 07:09:31                                    8
1050416 0.54184 cromwell_1 a.mizeransch qw    03/09/2019 07:22:25                                    8
1050417 0.54184 cromwell_3 a.mizeransch qw    03/09/2019 07:23:31                                    8
[...]

Thanks for the positive report, it's great to hear things are now running and submitting cleanly on SGE with Cromwell. I appreciate all your patience getting this going.
In terms of what you're seeing on queuing and efficiency of the run, this will depend on your specific cluster and node availability. The snippet you showed indicates that you have availability for 28 total cores (4 for the main process and 8 for each of the 3 subproceses that are running). The other jobs are waiting for your SGE cluster to allocate them resources. If you're cluster is busy and no other resources are available, it will cycle through what you can use as you specify. In terms of packing onto nodes, I think qstat -r might give node information to tell you where the jobs get allocated.
The Cromwell queuing approach will be less efficient for large numbers of small jobs than bcbio's IPython approach if the cluster is busy and you have to wait for resources. bcbio only waits a single time, then holds onto the resources for the remainder of the job. This results in less waiting, but is also more wasteful since it keeps all of these cores even if only a single process is running, like one long running task. The CWL Cromwell approach is meant to avoid this wastefulness and be a better cluster citizen.
Hope this helps explain what you're seeing.

The cluster queue is definitely not busy. There are 312 cores available right now on that queue (ibm-nehalem.q). This is why I was surprised that only one 24-core node appears to actually be used:
CLUSTER QUEUE                   CQLOAD   USED    RES  AVAIL  TOTAL aoACDS  cdsuE
--------------------------------------------------------------------------------
all.q                             0.11      0      0      0   1000      0   1000
campus-haswell.q                  0.03      4      0    412    448      0     32
hp-sl.q                           0.60     49      0     71    120      0      0
ibm-cell-int.q                    -NA-      0      0      0    200      0    200
ibm-cell-qs22.q                   -NA-      0      0      0     36      0     36
ibm-dp.q                          0.06      0      0     96     96      0      0
ibm-nehalem.q                     0.05     24      0    312    336      0      0

OK, it looks like this was a cluster-specific issue after all, although I'm still not sure about the exact cause.

Update: regarding the test I mentioned earlier with 100 small FASTQ files, it eventually crashed with the following errors. However, it ran fine when using IPython parallel instead of CWL. Any idea what I can do to prevent this?
[2019-03-09 20:30:59,10] [info] DispatchedConfigAsyncJobExecutionActor [bcabb07bpipeline_summary:562:1]: Status change from Running to Done
[2019-03-09 20:30:59,30] [info] checkpointClose end
[2019-03-09 20:30:59,30] [info] Checkpoint end - txts: 31636204
[2019-03-09 20:30:59,31] [info] lob file truncated to usage
[2019-03-09 20:30:59,32] [error] WriteMetadataActor Failed to properly process data
java.sql.BatchUpdateException: lob is no longer valid
	at org.hsqldb.jdbc.JDBCPreparedStatement.executeBatch(Unknown Source)
	at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:128)
	at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java)
	at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction.$anonfun$run$14(JdbcActionComponent.scala:532)
	at slick.jdbc.JdbcBackend$SessionDef.withPreparedStatement(JdbcBackend.scala:386)
	at slick.jdbc.JdbcBackend$SessionDef.withPreparedStatement$(JdbcBackend.scala:381)
	at slick.jdbc.JdbcBackend$BaseSession.withPreparedStatement(JdbcBackend.scala:448)
	at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl.preparedInsert(JdbcActionComponent.scala:501)
	at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$MultiInsertAction.run(JdbcActionComponent.scala:526)
	at slick.jdbc.JdbcActionComponent$SimpleJdbcProfileAction.run(JdbcActionComponent.scala:30)
	at slick.jdbc.JdbcActionComponent$SimpleJdbcProfileAction.run(JdbcActionComponent.scala:27)
	at slick.dbio.DBIOAction$$anon$4.$anonfun$run$3(DBIOAction.scala:239)
	at scala.collection.Iterator.foreach(Iterator.scala:944)
	at scala.collection.Iterator.foreach$(Iterator.scala:944)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1432)
	at scala.collection.IterableLike.foreach(IterableLike.scala:71)
	at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
	at slick.dbio.DBIOAction$$anon$4.run(DBIOAction.scala:239)
	at slick.dbio.DBIOAction$$anon$4.run(DBIOAction.scala:237)
	at slick.dbio.SynchronousDatabaseAction$$anon$7.run(DBIOAction.scala:486)
	at slick.basic.BasicBackend$DatabaseDef$$anon$2.liftedTree1$1(BasicBackend.scala:275)
	at slick.basic.BasicBackend$DatabaseDef$$anon$2.run(BasicBackend.scala:275)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

The following seems to be the main issue:
[2019-03-09 20:49:36,03] [error] WorkflowManagerActor Workflow bcabb07b-ae08-426e-88c2-6ec0387b7f19 failed (during ExecutingWorkflowState): java.lang.RuntimeException: Unable to start job. Check the stderr file for possible errors: /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/bcabb07b-ae08-426e-88c2-6ec0387b7f19/call-pipeline_summary/shard-376/execution/stderr.submit
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.$anonfun$execute$2(SharedFileSystemAsyncJobExecutionActor.scala:165)
	at scala.util.Either.fold(Either.scala:188)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute(SharedFileSystemAsyncJobExecutionActor.scala:160)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute$(SharedFileSystemAsyncJobExecutionActor.scala:155)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.execute(ConfigAsyncJobExecutionActor.scala:215)
	at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$executeAsync$1(StandardAsyncExecutionActor.scala:637)
	at scala.util.Try$.apply(Try.scala:209)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync(StandardAsyncExecutionActor.scala:637)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync$(StandardAsyncExecutionActor.scala:637)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.executeAsync(ConfigAsyncJobExecutionActor.scala:215)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover(StandardAsyncExecutionActor.scala:952)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover$(StandardAsyncExecutionActor.scala:944)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.executeOrRecover(ConfigAsyncJobExecutionActor.scala:215)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.$anonfun$robustExecuteOrRecover$1(AsyncBackendJobExecutionActor.scala:65)
	at cromwell.core.retry.Retry$.withRetry(Retry.scala:38)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.withRetry(AsyncBackendJobExecutionActor.scala:61)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.cromwell$backend$async$AsyncBackendJobExecutionActor$$robustExecuteOrRecover(AsyncBackendJobExecutionActor.scala:65)
	at cromwell.backend.async.AsyncBackendJobExecutionActor$$anonfun$receive$1.applyOrElse(AsyncBackendJobExecutionActor.scala:88)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
	at akka.actor.Actor.aroundReceive(Actor.scala:517)
	at akka.actor.Actor.aroundReceive$(Actor.scala:515)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.aroundReceive(ConfigAsyncJobExecutionActor.scala:215)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
	at akka.actor.ActorCell.invoke(ActorCell.scala:557)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

This is the contents of the stderr file mentioned in the message above:
$ cat /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/bcabb07b-ae08-426e-88c2-6ec0387b7f19/call-pipeline_summary/shard-376/execution/stderr.submit
error: commlib error: got select error (Connection reset by peer)
Unable to run job: failed receiving gdi request response for mid=1 (got syncron message receive timeout error)..
Exiting.

I found a seemingly related error report (https://github.com/aws/aws-parallelcluster/issues/26) where someone mentioned that this could be a problem with the cluster load. A different page (https://www.t3.gsic.titech.ac.jp/en/node/176) confirms this:
    Unable to run job: failed receiving gdi request response for mid=1 (got syncron message receive timeout error).  or  Unable to run job: got no response from JSV script"/apps/t3/sles12sp2/uge/customize/jsv.pl".

Communication with the job scheduler will time out and the above error message may be displayed if the management node becomes in a state of high load due to a large amount of job input in a short time. The state of high load is temporary. Please try again after waiting a while.

In case it's relevant, here is the output of ulimit -a from one of the compute nodes. Could the open files limit be the cause of these errors?
$ ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 510853
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) 16777216
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 510853
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Thanks for the follow up and all this testing. The error you saw looks like an issue communicating with the scheduler. Do you see this consistently, or intermittently during runs?  This is is another one where I can't offer better advice than to retry and see if it works cleanly on a re-run; hope it works more consistently going forward.

Things worked fine with 10 samples (small FASTQ files with reads mapping to just one gene). However, I'm consistently getting the messages [error] WriteMetadataActor Failed to properly process data and java.sql.BatchUpdateException: lob is no longer valid when running with 100 small samples. These events don't seem to crash the whole analysis, so can they be safely ignored?
The second error mentioned above (java.lang.RuntimeException: Unable to start job) didn't reappear.
There were several Java memory-related errors near the end (in the MultiQC stage) which seemed surprising, given that I'm using very small FASTQ files:
2019-03-10 16:39:43,04] [warn] SingleWorkflowRunnerActor: received unexpected message: QueueWeight(0) of type cromwell.core.actor.BatchActor.QueueWeight in state RequestingMetadata
[2019-03-10 16:39:49,43] [error] Uncaught error from thread [cromwell-system-akka.dispatchers.engine-dispatcher-967]: Java heap space, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[cromwell-system]
java.lang.OutOfMemoryError: Java heap space
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.growArray(ForkJoinPool.java:1090)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1978)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[2019-03-10 16:39:50,96] [error] Uncaught error from thread [cromwell-system-akka.dispatchers.engine-dispatcher-971]: Java heap space, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[cromwell-system]
java.lang.OutOfMemoryError: Java heap space
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.growArray(ForkJoinPool.java:1090)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1978)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

[2019-03-10 16:52:07,77] [warn] SingleWorkflowRunnerActor: received unexpected message: QueueWeight(0) of type cromwell.core.actor.BatchActor.QueueWeight in state RequestingMetadata
[2019-03-10 16:52:17,87] [error] Uncaught error from thread [cromwell-system-akka.dispatchers.engine-dispatcher-983]: Java heap space, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[cromwell-system]
java.lang.OutOfMemoryError: Java heap space

In the end, the run still crashed:
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
[2019-03-10 18:41:36,18] [warn] CoordinatedShutdown from JVM shutdown failed: Futures timed out after [5430000 milliseconds]
Running: export PATH=$PATH:/export/home/ncit/external/a.mizeranschi/bcbio_nextgen/anaconda/envs/bcbiovm/bin && cromwell -Xms1g -Xmx3g run --type CWL -Dconfig.file=/export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/bcbio-cromwell.conf -Dbackend.default=SGE --metadata-output /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/testingVC-merged-metadata.json --options /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/testingVC-merged-options.json --inputs /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/main-testingVC-merged-samples.json /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/main-testingVC-merged.cwl 2>&1 | tee -a /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/testingVC-merged-cromwell.log

How can I get this to work with the 100 small FASTQ files as well?

On a new try, the whole thing crashed due to running out of memory. I double checked the settings from bcbio_system.yaml and they should match the configuration of the execution nodes (which have 32 cores and 128 GB of memory each).
resources:
  # default options, used if other items below are not present
  # avoids needing to configure/adjust for every program
  default:
    memory: 3.5G
    cores: 16
    jvm_opts: ["-Xms750m", "-Xmx3500m"]
  gatk:
    jvm_opts: ["-Xms500m", "-Xmx3500m"]

Even though I'm using very small FASTQ files (with less than 100 reads each), Java seems to be running out of memory. What could be causing this?
[2019-03-10 21:34:09,31] [warn] SingleWorkflowRunnerActor: received unexpected message: QueueWeight(0) of type cromwell.core.actor.BatchActor.QueueWeight in state RequestingMetadata
Exception in thread "db-12" java.lang.OutOfMemoryError: GC overhead limit exceeded
Uncaught error from thread [cromwell-system-akka.dispatchers.engine-dispatcher-712]: GC overhead limit exceeded, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[cromwell-system]
java.lang.OutOfMemoryError: GC overhead limit exceeded

Thanks for all this testing and sorry about the continued problems. It seems like the latest issues are that the main Cromwell processing is hitting memory issues. Is that right? If so, that might be due to scheduling all of the pieces with the 100 samples. We also see this with IPython at higher number of samples, since you need to schedule out all the work, independent of the size of the files.
If that's what's failing, increasing the memory used to run Cromwell beyond the set 3g might resolve it:
https://github.com/bcbio/bcbio-nextgen/blob/62b1b79bb0145c44b8aca6fbe242cd0882685efb/bcbio/cwl/tool.py#L205
Could you try adjusting this manually and see if that resolves the issue? Is so, we can try to come up with a more general fix going forward. Thanks again.

OK, I edited the file bcbio_nextgen/anaconda/envs/bcbiovm/lib/python2.7/site-packages/bcbio/cwl/tool.py and changed -Xmx3g to -Xmx100g.
I also changed the number of CPU cores requested by the main SGE job that I submit (the one that runs bcbio) from 4 to 32, hoping that this would also increase the overall memory available to Cromwell.
After about 3 hours, the run crashed with the following message:
[2019-03-11 09:58:22,49] [error] WorkflowManagerActor Workflow 0d35763d-1bd3-4d23-a7e8-59c099ca3d31 failed (during ExecutingWorkflowState): java.lang.RuntimeException: Unable to start job. Check the stderr file for possible errors: /export/home/ncit/external/a.mizeranschi/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/0d35763d-1bd3-4d23-a7e8-59c099ca3d31/call-pipeline_summary/shard-430/execution/stderr.submit
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.$anonfun$execute$2(SharedFileSystemAsyncJobExecutionActor.scala:165)
	at scala.util.Either.fold(Either.scala:188)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute(SharedFileSystemAsyncJobExecutionActor.scala:160)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute$(SharedFileSystemAsyncJobExecutionActor.scala:155)
	at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.execute(ConfigAsyncJobExecutionActor.scala:215)
	at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$executeAsync$1(StandardAsyncExecutionActor.scala:637)
	at scala.util.Try$.apply(Try.scala:209)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync(StandardAsyncExecutionActor.scala:637)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync$(StandardAsyncExecutionActor.scala:637)