Error in sandbox when used predefined model

I coppied the model Will said to use in the training directory of my run. This is the result (error). This error repeats itself and it doesn't stop trying to go through it
`submitted 3000/3113 job(s)
submitted 3025/3113 job(s)
submitted 3050/3113 job(s)
submitted 3075/3113 job(s)
submitted 3100/3113 job(s)
submitted 3113/3113 job(s)
submitted 3113 htcondor job(s)
03:47:31: all: 7155, pending: 1648 (+1648), running: 1465 (+1465), finished: 4042 (+0), retry: 0 (-3113), failed: 0 (+0)
^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A03:48:33: all: 7155, pending: 1614 (-34), running: 1499 (+34), finished: 4042 (+0), retr
y: 0 (+0), failed: 0 (+0)
03:49:35: all: 7155, pending: 1571 (-43), running: 1525 (+26), finished: 4059 (+17), retry: 0 (+0), failed: 0 (+0)
03:50:38: all: 7155, pending: 1402 (-169), running: 1553 (+28), finished: 4200 (+141), retry: 0 (+0), failed: 0 (+0)
03:51:52: all: 7155, pending: 1289 (-113), running: 1591 (+38), finished: 4275 (+75), retry: 0 (+0), failed: 0 (+0)
03:56:23: all: 7155, pending: 1261 (-28), running: 1602 (+11), finished: 4292 (+17), retry: 0 (+0), failed: 0 (+0)
03:58:32: all: 7155, pending: 0 (-1261), running: 1519 (-83), finished: 5636 (+1344), retry: 0 (+0), failed: 0 (+0)
03:59:33: all: 7155, pending: 0 (+0), running: 1186 (-333), finished: 5969 (+333), retry: 0 (+0), failed: 0 (+0)
INFO: luigi-interface - [pid 1713650] Worker Worker(salt=413637447, workers=5, host=dgx1, username=vasileios.skliris, pid=1628977) running   SandboxExport(image=/home/vasileios.skliris
/aframe/images/export.sif, weights=, fduration=1.0, kernel_length=1.5, inference_sampling_rate=4.0, sample_rate=2048.0, repository_directory=/home/vasileios.skliris/aframe/runs/my-firs
t-run/results/model_repo, batch_file=, streams_per_gpu=2, aframe_instances=1, preproc_instances=1, clean=True, batch_size=128, psd_length=64.0, highpass=32.0, q=, fftlength=0, ifos=["H
1", "L1"], train_task=Train)
INFO:luigi-interface:[pid 1713650] Worker Worker(salt=413637447, workers=5, host=dgx1, username=vasileios.skliris, pid=1628977) running   SandboxExport(image=/home/vasileios.skliris/af
rame/images/export.sif, weights=, fduration=1.0, kernel_length=1.5, inference_sampling_rate=4.0, sample_rate=2048.0, repository_directory=/home/vasileios.skliris/aframe/runs/my-first-r
un/results/model_repo, batch_file=, streams_per_gpu=2, aframe_instances=1, preproc_instances=1, clean=True, batch_size=128, psd_length=64.0, highpass=32.0, q=, fftlength=0, ifos=["H1",
"L1"], train_task=Train)
04:03:51: all: 7155, pending: 0 (+0), running: 958 (-228), finished: 6197 (+228), retry: 0 (+0), failed: 0 (+0)
=============================== entering sandbox ===============================
task   : SandboxExport_1__128_c0df290c87
sandbox: aframe::/home/vasileios.skliris/aframe/images/export.sif
/home/vasileios.skliris/.bash_profile: line 24: /cvmfs/oasis.opensciencegrid.org/ligo/sw/conda/etc/profile.d/conda.sh: No such file or directory
Traceback (most recent call last):
File "/usr/local/bin/law", line 10, in
exit_code = run()
File "/law_forward/py/law/cli/cli.py", line 74, in run
return mods[prog].execute(*exec_args)
File "/law_forward/py/law/cli/run.py", line 57, in execute
mod = import(modid, globals(), locals(), [cls_name])
File "/opt/aframe/aframe/pipelines/sandbox/init.py", line 1, in
from .sandbox import Sandbox, Tune
File "/opt/aframe/aframe/pipelines/sandbox/sandbox.py", line 6, in
from aframe.tasks.infer import InferLocal
File "/opt/aframe/aframe/tasks/infer/init.py", line 1, in
from .infer import InferLocal, InferRemote
File "/opt/aframe/aframe/tasks/infer/infer.py", line 59, in
class InferBase(AframeSandboxTask):
File "/opt/aframe/aframe/tasks/infer/infer.py", line 67, in InferBase
env_path = get_poetry_env(INFER_DIR)
File "/opt/aframe/aframe/tasks/infer/infer.py", line 26, in get_poetry_env
output = subprocess.check_output(
File "/usr/local/lib/python3.10/subprocess.py", line 421, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/usr/local/lib/python3.10/subprocess.py", line 503, in run
with Popen(*popenargs, **kwargs) as process:
File "/usr/local/lib/python3.10/subprocess.py", line 971, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/local/lib/python3.10/subprocess.py", line 1863, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: PosixPath('/opt/aframe/projects/infer')
=============================== leaving sandbox ================================
task   : SandboxExport_1__128_c0df290c87
sandbox: aframe::/home/vasileios.skliris/aframe/images/export.sif
ERROR: luigi-interface - [pid 1713650] Worker Worker(salt=413637447, workers=5, host=dgx1, username=vasileios.skliris, pid=1628977) failed    SandboxExport(image=/home/vasileios.skliri
s/aframe/images/export.sif, weights=, fduration=1.0, kernel_length=1.5, inference_sampling_rate=4.0, sample_rate=2048.0, repository_directory=/home/vasileios.skliris/aframe/runs/my-fir
st-run/results/model_repo, batch_file=, streams_per_gpu=2, aframe_instances=1, preproc_instances=1, clean=True, batch_size=128, psd_length=64.0, highpass=32.0, q=, fftlength=0, ifos=["
H1", "L1"], train_task=Train)
Traceback (most recent call last):
File "/home/vasileios.skliris/miniconda3-tmp/envs/aframe-EMPJJKiR-py3.9/lib/python3.9/site-packages/luigi/worker.py", line 210, in run
new_deps = self._run_get_new_deps()
File "/home/vasileios.skliris/miniconda3-tmp/envs/aframe-EMPJJKiR-py3.9/lib/python3.9/site-packages/luigi/worker.py", line 138, in _run_get_new_deps
task_gen = self.task.run()
File "/home/vasileios.skliris/miniconda3-tmp/envs/aframe-EMPJJKiR-py3.9/lib/python3.9/site-packages/law/sandbox/base.py", line 460, in run
raise Exception(
Exception: sandbox 'aframe::/home/vasileios.skliris/aframe/images/export.sif' failed with exit code 1, please see the error inside the sandboxed context above for details
ERROR:luigi-interface:[pid 1713650] Worker Worker(salt=413637447, workers=5, host=dgx1, username=vasileios.skliris, pid=1628977) failed    SandboxExport(image=/home/vasileios.skliris/a
frame/images/export.sif, weights=, fduration=1.0, kernel_length=1.5, inference_sampling_rate=4.0, sample_rate=2048.0, repository_directory=/home/vasileios.skliris/aframe/runs/my-first-
run/results/model_repo, batch_file=, streams_per_gpu=2, aframe_instances=1, preproc_instances=1, clean=True, batch_size=128, psd_length=64.0, highpass=32.0, q=, fftlength=0, ifos=["H1"
, "L1"], train_task=Train)
Traceback (most recent call last):
File "/home/vasileios.skliris/miniconda3-tmp/envs/aframe-EMPJJKiR-py3.9/lib/python3.9/site-packages/luigi/worker.py", line 210, in run
new_deps = self._run_get_new_deps()
File "/home/vasileios.skliris/miniconda3-tmp/envs/aframe-EMPJJKiR-py3.9/lib/python3.9/site-packages/luigi/worker.py", line 138, in _run_get_new_deps
task_gen = self.task.run()
File "/home/vasileios.skliris/miniconda3-tmp/envs/aframe-EMPJJKiR-py3.9/lib/python3.9/site-packages/law/sandbox/base.py", line 460, in run
raise Exception(
Exception: sandbox 'aframe::/home/vasileios.skliris/aframe/images/export.sif' failed with exit code 1, please see the error inside the sandboxed context above for details
`

Hey @VasSkliris It looks like you are on an old version of the code. Can you

pull the latest changes
Rebuild the infer container?

Let me know how it goes

I did, althouth I am not sure if it really rebuild all the images. No I get this
`(base) [vasileios.skliris@dgx1 aframev2]$ bash ~/aframe/runs/my-first-run/run.sh
INFO: luigi-interface - Informed scheduler that task   Sandbox__99914b932b   has status   PENDING
WARNING: luigi-interface - Will not run SandboxSV(image=/home/vasileios.skliris/aframe/images/plots.sif, ifos=["H1", "L1"], mass_combos=[[35, 35], [35, 20], [20, 20], [20, 1
0]], source_prior=priors.priors.end_o3_ratesandpops, dt=inf, output_dir=/home/vasileios.skliris/aframe/runs/my-first-run/results/plots, train_task=Train) or any dependencies
due to error in deps() method:
Traceback (most recent call last):
File "/home/vasileios.skliris/miniconda3-tmp/envs/aframe-EMPJJKiR-py3.9/lib/python3.9/site-packages/luigi/worker.py", line 884, in _add
deps = task.deps()
File "/home/vasileios.skliris/miniconda3-tmp/envs/aframe-EMPJJKiR-py3.9/lib/python3.9/site-packages/luigi/task.py", line 685, in deps
return flatten(self._requires())
File "/home/vasileios.skliris/miniconda3-tmp/envs/aframe-EMPJJKiR-py3.9/lib/python3.9/site-packages/luigi/task.py", line 657, in _requires
return flatten(self.requires())  # base impl
File "/home/vasileios.skliris/aframev2/aframe/pipelines/sandbox/sandbox.py", line 19, in requires
reqs["ts"] = TestingWaveforms.req(self)
File "/home/vasileios.skliris/miniconda3-tmp/envs/aframe-EMPJJKiR-py3.9/lib/python3.9/site-packages/law/task/base.py", line 167, in req
return cls(**cls.req_params(inst, **kwargs))
File "/home/vasileios.skliris/miniconda3-tmp/envs/aframe-EMPJJKiR-py3.9/lib/python3.9/site-packages/law/task/base.py", line 350, in call
inst = super(Register, cls).call(*args, **kwargs)
File "/home/vasileios.skliris/miniconda3-tmp/envs/aframe-EMPJJKiR-py3.9/lib/python3.9/site-packages/luigi/task_register.py", line 87, in call
param_values = cls.get_param_values(params, args, kwargs)
File "/home/vasileios.skliris/miniconda3-tmp/envs/aframe-EMPJJKiR-py3.9/lib/python3.9/site-packages/law/task/base.py", line 143, in get_param_values
values = super(BaseTask, cls).get_param_values(params, args, kwargs)
File "/home/vasileios.skliris/miniconda3-tmp/envs/aframe-EMPJJKiR-py3.9/lib/python3.9/site-packages/luigi/task.py", line 432, in get_param_values
raise parameter.MissingParameterException("%s: requires the '%s' parameter to be set" % (exc_desc, param_name))
luigi.parameter.MissingParameterException: TestingWaveforms[args=(), kwargs={'log_file': 'NO_STR', 'dev': False, 'gpus': '0', 'container_root': '/home/vasileios.skliris/afra
me/images/', 'ifos': ('H1', 'L1')}]: requires the 'num_signals' parameter to be set
INFO: luigi-interface - Informed scheduler that task   SandboxSV_inf___H1____L1____ad5eb76d68   has status   UNKNOWN
INFO: luigi-interface - Done scheduling tasks
INFO: luigi-interface - Running Worker with 5 processes
INFO: luigi-interface - Worker Worker(salt=9376474673, workers=5, host=dgx1, username=vasileios.skliris, pid=3130659) was stopped. Shutting down Keep-Alive thread
INFO: luigi-interface -
===== Luigi Execution Summary =====
Scheduled 2 tasks of which:

1 failed scheduling:

1 SandboxSV(...)

1 were left pending, among these:

1 had dependencies whose scheduling failed:

1 Sandbox(...)

Did not run any tasks
This progress looks :( because there were tasks whose scheduling failed
===== Luigi Execution Summary =====
`

Ah yeah - there are a couple configuration changes that need to be made. If you remind me of your run directory and machine I can enumerate them here.

This is the run directory (/home/vasileios.skliris/aframe/runs/my-first-run) I am running on dgx-1 on ligo hanford cluster

[luigi_TimeslideWaveforms] --> [luigi_TestingWaveforms]
Under [luigi_TestingWaveforms] : num_injections --> num_signals
Under [luigi_SandboxInfer]: add workflow = htcondor
[luigi_SandboxExport] --> [luigi_ExportLocal]

@VasSkliris Let me know if any of these changes are confusing

It looks for a file that looks like a model (.h5)? I only have trasfered a model.pt in the training directory
ERROR: luigi-interface - [pid 3869922] Worker Worker(salt=5151869991, workers=5, host=dgx1, username=vasileios.skliris, pid=3853386) failed    ExportLocal(i[73/117$ me/vasileios.skliris/aframe/images/export.sif, fduration=1.0, kernel_length=1.5, inference_sampling_rate=4.0, sample_rate=2048.0, batch_file=, streams_per_gpu=2, a$ rame_instances=1, preproc_instances=1, clean=True, batch_size=128, psd_length=64.0, highpass=32.0, q=, fftlength=None, ifos=["H1", "L1"], repository_directory=/hom$ /vasileios.skliris/aframe/runs/my-first-run/results/model_repo, train_task=Train)                                                                                Traceback (most recent call last):                                                                                File "/law_forward/py/luigi/worker.py", line 210, in run                                                                                new_deps = self._run_get_new_deps()                                                                                File "/law_forward/py/luigi/worker.py", line 138, in _run_get_new_deps                                                                                task_gen = self.task.run()                                                                                File "/opt/aframe/aframe/tasks/export/export.py", line 70, in run                                                                                export(                                                                                File "/opt/aframe/projects/export/export/main.py", line 137, in export                                                                                with open_file(batch_file, "rb") as f:                                                                                File "/opt/aframe/libs/utils/utils/s3.py", line 13, in open_file                                                                                return open(path, mode)                                                                                FileNotFoundError: [Errno 2] No such file or directory: '/home/vasileios.skliris/aframe/runs/my-first-run/training/batch.h5'                                         ERROR:luigi-interface:[pid 3869922] Worker Worker(salt=5151869991, workers=5, host=dgx1, username=vasileios.skliris, pid=3853386) failed    ExportLocal(image=/home/ vasileios.skliris/aframe/images/export.sif, fduration=1.0, kernel_length=1.5, inference_sampling_rate=4.0, sample_rate=2048.0, batch_file=, streams_per_gpu=2, afram e_instances=1, preproc_instances=1, clean=True, batch_size=128, psd_length=64.0, highpass=32.0, q=, fftlength=None, ifos=["H1", "L1"], repository_directory=/home/va sileios.skliris/aframe/runs/my-first-run/results/model_repo, train_task=Train)                                                                                Traceback (most recent call last):                                                                                File "/law_forward/py/luigi/worker.py", line 210, in run                                                                                new_deps = self._run_get_new_deps()                                                                                File "/law_forward/py/luigi/worker.py", line 138, in _run_get_new_deps                                                                                task_gen = self.task.run()                                                                                File "/opt/aframe/aframe/tasks/export/export.py", line 70, in run                                                                                export(                                                                                File "/opt/aframe/projects/export/export/main.py", line 137, in export                                                                                with open_file(batch_file, "rb") as f:                                                                                File "/opt/aframe/libs/utils/utils/s3.py", line 13, in open_file                                                                                return open(path, mode)                                                                                FileNotFoundError: [Errno 2] No such file or directory: '/home/vasileios.skliris/aframe/runs/my-first-run/training/batch.h5'                                         INFO: luigi-interface - This progress looks :( because there were failed tasks                                                                                INFO:luigi-interface:This progress looks :( because there were failed tasks

Ah yeah - we use a saved training batch to infer some things for the export task. Can you copy this path to the same directory as model.pt?  /home/ethan.marx/aframe/o3-production-search/training/batch.h5

After using your training directory above in the run.sh and remaking the run directory and the images from scratch, I get this:
(this is just the end of the file: /home/vasileios.skliris/aframe/runs/my-first-run_pretrained/run.out)
`=============================== entering sandbox ===============================
task   : ExportLocal_6__128_de5506789b
sandbox: aframe::/home/vasileios.skliris/aframe/images/export.sif
/home/vasileios.skliris/.bash_profile: line 24: /cvmfs/oasis.opensciencegrid.org/ligo/sw/conda/etc/profile.d/conda.sh: No such file or directory
INFO: luigi-interface - Done scheduling tasks
INFO: Done scheduling tasks
INFO: luigi-interface - Running Worker with 1 processes
INFO: Running Worker with 1 processes
INFO: luigi-interface - [pid 817825] Worker Worker(salt=2603756573, workers=5, host=dgx1, username=vasileios.skliris, pid=696537) running   ExportLocal(image=/home/vasileios.skliris/aframe/images/export.sif, fduration=1.0, kernel_length=1.5, inference_sampling_rate=4.0, sample_rate=2048.0, batch_file=, streams_per_gpu=6, aframe_instances=6, preproc_instances=1, clean=True, batch_size=128, psd_length=64.0, highpass=32.0, q=, fftlength=0.0, ifos=["H1", "L1"], repository_directory=/home/vasileios.skliris/aframe/runs/my-first-run_pretrained/results/model_repo, train_task=Train)
INFO: [pid 817825] Worker Worker(salt=2603756573, workers=5, host=dgx1, username=vasileios.skliris, pid=696537) running   ExportLocal(image=/home/vasileios.skliris/aframe/images/export.sif, fduration=1.0, kernel_length=1.5, inference_sampling_rate=4.0, sample_rate=2048.0, batch_file=, streams_per_gpu=6, aframe_instances=6, preproc_instances=1, clean=True, batch_size=128, psd_length=64.0, highpass=32.0, q=, fftlength=0.0, ifos=["H1", "L1"], repository_directory=/home/vasileios.skliris/aframe/runs/my-first-run_pretrained/results/model_repo, train_task=Train)
ERROR: luigi-interface - [pid 817825] Worker Worker(salt=2603756573, workers=5, host=dgx1, username=vasileios.skliris, pid=696537) failed    ExportLocal(image=/home/vasileios.skliris/aframe/images/export.sif, fduration=1.0, kernel_length=1.5, inference_sampling_rate=4.0, sample_rate=2048.0, batch_file=, streams_per_gpu=6, aframe_instances=6, preproc_instances=1, clean=True, batch_size=128, psd_length=64.0, highpass=32.0, q=, fftlength=0.0, ifos=["H1", "L1"], repository_directory=/home/vasileios.skliris/aframe/runs/my-first-run_pretrained/results/model_repo, train_task=Train)
Traceback (most recent call last):
File "/law_forward/py/luigi/worker.py", line 195, in run
raise RuntimeError('Unfulfilled %s at run time: %s' % (deps, ', '.join(missing)))
RuntimeError: Unfulfilled dependency at run time: Train____home_vasileios__12_e02b2ae116 (LocalFileTarget(fs=local_fs, path=/home/ethan.marx/aframe/o3-production-search/training/model.pt))
ERROR: [pid 817825] Worker Worker(salt=2603756573, workers=5, host=dgx1, username=vasileios.skliris, pid=696537) failed    ExportLocal(image=/home/vasileios.skliris/aframe/images/export.sif, fduration=1.0, kernel_length=1.5, inference_sampling_rate=4.0, sample_rate=2048.0, batch_file=, streams_per_gpu=6, aframe_instances=6, preproc_instances=1, clean=True, batch_size=128, psd_length=64.0, highpass=32.0, q=, fftlength=0.0, ifos=["H1", "L1"], repository_directory=/home/vasileios.skliris/aframe/runs/my-first-run_pretrained/results/model_repo, train_task=Train)
Traceback (most recent call last):
File "/law_forward/py/luigi/worker.py", line 195, in run
raise RuntimeError('Unfulfilled %s at run time: %s' % (deps, ', '.join(missing)))
RuntimeError: Unfulfilled dependency at run time: Train____home_vasileios__12_e02b2ae116 (LocalFileTarget(fs=local_fs, path=/home/ethan.marx/aframe/o3-production-search/training/model.pt))
INFO: luigi-interface - This progress looks :( because there were failed tasks
INFO: This progress looks :( because there were failed tasks
=============================== leaving sandbox ================================
task   : ExportLocal_6__128_de5506789b
sandbox: aframe::/home/vasileios.skliris/aframe/images/export.sif
ERROR: luigi-interface - [pid 817733] Worker Worker(salt=2603756573, workers=5, host=dgx1, username=vasileios.skliris, pid=696537) failed    ExportLocal(image=/home/vasileios.skliris/aframe/images/export.sif, fduration=1.0, kernel_length=1.5, inference_sampling_rate=4.0, sample_rate=2048.0, batch_file=, streams_per_gpu=6, aframe_instances=6, preproc_instances=1, clean=True, batch_size=128, psd_length=64.0, highpass=32.0, q=, fftlength=0, ifos=["H1", "L1"], repository_directory=/home/vasileios.skliris/aframe/runs/my-first-run_pretrained/results/model_repo, train_task=Train)
Traceback (most recent call last):
File "/home/vasileios.skliris/miniconda3-tmp/envs/aframe-EMPJJKiR-py3.9/lib/python3.9/site-packages/luigi/worker.py", line 210, in run
new_deps = self._run_get_new_deps()
File "/home/vasileios.skliris/miniconda3-tmp/envs/aframe-EMPJJKiR-py3.9/lib/python3.9/site-packages/luigi/worker.py", line 138, in _run_get_new_deps
task_gen = self.task.run()
File "/home/vasileios.skliris/miniconda3-tmp/envs/aframe-EMPJJKiR-py3.9/lib/python3.9/site-packages/law/sandbox/base.py", line 460, in run
raise Exception(
Exception: sandbox 'aframe::/home/vasileios.skliris/aframe/images/export.sif' failed with exit code 40, please see the error inside the sandboxed context above for details
ERROR: [pid 817733] Worker Worker(salt=2603756573, workers=5, host=dgx1, username=vasileios.skliris, pid=696537) failed    ExportLocal(image=/home/vasileios.skliris/aframe/images/export.sif, fduration=1.0, kernel_length=1.5, inference_sampling_rate=4.0, sample_rate=2048.0, batch_file=, streams_per_gpu=6, aframe_instances=6, preproc_instances=1, clean=True, batch_size=128, psd_length=64.0, highpass=32.0, q=, fftlength=0, ifos=["H1", "L1"], repository_directory=/home/vasileios.skliris/aframe/runs/my-first-run_pretrained/results/model_repo, train_task=Train)
Traceback (most recent call last):
File "/home/vasileios.skliris/miniconda3-tmp/envs/aframe-EMPJJKiR-py3.9/lib/python3.9/site-packages/luigi/worker.py", line 210, in run
new_deps = self._run_get_new_deps()
File "/home/vasileios.skliris/miniconda3-tmp/envs/aframe-EMPJJKiR-py3.9/lib/python3.9/site-packages/luigi/worker.py", line 138, in _run_get_new_deps
task_gen = self.task.run()
File "/home/vasileios.skliris/miniconda3-tmp/envs/aframe-EMPJJKiR-py3.9/lib/python3.9/site-packages/law/sandbox/base.py", line 460, in run
raise Exception(
Exception: sandbox 'aframe::/home/vasileios.skliris/aframe/images/export.sif' failed with exit code 40, please see the error inside the sandboxed context above for details
INFO: luigi-interface - Informed scheduler that task   ExportLocal_6__128_de5506789b   has status   FAILED
INFO: Informed scheduler that task   ExportLocal_6__128_de5506789b   has status   FAILED
09:54:00: all: 159, unsubmitted: 31 (-1), pending: 0 (+0), running: 5 (+0), finished: 123 (+1), retry: 0 (+0), failed: 0 (+0)
going to submit 1 htcondor job(s)
submitted 1 htcondor job(s)
09:54:12: all: 159, unsubmitted: 30 (-1), pending: 0 (+0), running: 5 (+0), finished: 124 (+1), retry: 0 (+0), failed: 0 (+0)
going to submit 1 htcondor job(s)
submitted 1 htcondor job(s)
09:54:24: all: 159, unsubmitted: 29 (-1), pending: 0 (+0), running: 5 (+0), finished: 125 (+1), retry: 0 (+0), failed: 0 (+0)
going to submit 1 htcondor job(s)
submitted 1 htcondor job(s)
09:54:37: all: 159, unsubmitted: 28 (-1), pending: 0 (+0), running: 4 (-1), finished: 127 (+2), retry: 0 (+0), failed: 0 (+0)
going to submit 2 htcondor job(s)
submitted 2 htcondor job(s)
09:54:49: all: 159, unsubmitted: 26 (-2), pending: 2 (+2), running: 4 (+0), finished: 127 (+0), retry: 0 (+0), failed: 0 (+0)
09:55:01: all: 159, unsubmitted: 26 (+0), pending: 0 (-2), running: 6 (+2), finished: 127 (+0), retry: 0 (+0), failed: 0 (+0)
09:55:13: all: 159, unsubmitted: 26 (+0), pending: 0 (+0), running: 5 (-1), finished: 128 (+1), retry: 0 (+0), failed: 0 (+0)
going to submit 1 htcondor job(s)
submitted 1 htcondor job(s)
09:55:25: all: 159, unsubmitted: 25 (-1), pending: 0 (+0), running: 5 (+0), finished: 129 (+1), retry: 0 (+0), failed: 0 (+0)
going to submit 1 htcondor job(s)
submitted 1 htcondor job(s)
09:55:37: all: 159, unsubmitted: 24 (-1), pending: 0 (+0), running: 6 (+1), finished: 129 (+0), retry: 0 (+0), failed: 0 (+0)
09:55:49: all: 159, unsubmitted: 24 (+0), pending: 0 (+0), running: 4 (-2), finished: 131 (+2), retry: 0 (+0), failed: 0 (+0)
going to submit 2 htcondor job(s)
submitted 2 htcondor job(s)
09:56:02: all: 159, unsubmitted: 22 (-2), pending: 0 (+0), running: 5 (+1), finished: 132 (+1), retry: 0 (+0), failed: 0 (+0)
going to submit 1 htcondor job(s)
submitted 1 htcondor job(s)
09:56:14: all: 159, unsubmitted: 21 (-1), pending: 0 (+0), running: 5 (+0), finished: 133 (+1), retry: 0 (+0), failed: 0 (+0)
going to submit 1 htcondor job(s)
submitted 1 htcondor job(s)
09:56:26: all: 159, unsubmitted: 20 (-1), pending: 0 (+0), running: 6 (+1), finished: 133 (+0), retry: 0 (+0), failed: 0 (+0)
09:56:38: all: 159, unsubmitted: 20 (+0), pending: 0 (+0), running: 6 (+0), finished: 133 (+0), retry: 0 (+0), failed: 0 (+0)
09:56:50: all: 159, unsubmitted: 20 (+0), pending: 0 (+0), running: 5 (-1), finished: 134 (+1), retry: 0 (+0), failed: 0 (+0)
going to submit 1 htcondor job(s)
submitted 1 htcondor job(s)
09:57:02: all: 159, unsubmitted: 19 (-1), pending: 0 (+0), running: 5 (+0), finished: 135 (+1), retry: 0 (+0), failed: 0 (+0)
going to submit 1 htcondor job(s)
submitted 1 htcondor job(s)
09:57:14: all: 159, unsubmitted: 18 (-1), pending: 0 (+0), running: 6 (+1), finished: 135 (+0), retry: 0 (+0), failed: 0 (+0)
09:57:27: all: 159, unsubmitted: 18 (+0), pending: 0 (+0), running: 3 (-3), finished: 138 (+3), retry: 0 (+0), failed: 0 (+0)
going to submit 3 htcondor job(s)
submitted 3 htcondor job(s)
09:57:39: all: 159, unsubmitted: 15 (-3), pending: 0 (+0), running: 6 (+3), finished: 138 (+0), retry: 0 (+0), failed: 0 (+0)
09:57:51: all: 159, unsubmitted: 15 (+0), pending: 0 (+0), running: 5 (-1), finished: 139 (+1), retry: 0 (+0), failed: 0 (+0)
going to submit 1 htcondor job(s)
submitted 1 htcondor job(s)
09:58:03: all: 159, unsubmitted: 14 (-1), pending: 0 (+0), running: 5 (+0), finished: 140 (+1), retry: 0 (+0), failed: 0 (+0)
going to submit 1 htcondor job(s)
submitted 1 htcondor job(s)
09:58:15: all: 159, unsubmitted: 13 (-1), pending: 0 (+0), running: 4 (-1), finished: 142 (+2), retry: 0 (+0), failed: 0 (+0)
going to submit 2 htcondor job(s)
submitted 2 htcondor job(s)
09:58:27: all: 159, unsubmitted: 11 (-2), pending: 0 (+0), running: 5 (+1), finished: 143 (+1), retry: 0 (+0), failed: 0 (+0)
going to submit 1 htcondor job(s)
submitted 1 htcondor job(s)
09:58:39: all: 159, unsubmitted: 10 (-1), pending: 0 (+0), running: 5 (+0), finished: 144 (+1), retry: 0 (+0), failed: 0 (+0)
going to submit 1 htcondor job(s)
submitted 1 htcondor job(s)
09:58:52: all: 159, unsubmitted: 9 (-1), pending: 0 (+0), running: 5 (+0), finished: 145 (+1), retry: 0 (+0), failed: 0 (+0)
going to submit 1 htcondor job(s)
submitted 1 htcondor job(s)
09:59:05: all: 159, unsubmitted: 8 (-1), pending: 0 (+0), running: 5 (+0), finished: 146 (+1), retry: 0 (+0), failed: 0 (+0)
going to submit 1 htcondor job(s)
submitted 1 htcondor job(s)
09:59:17: all: 159, unsubmitted: 7 (-1), pending: 0 (+0), running: 3 (-2), finished: 149 (+3), retry: 0 (+0), failed: 0 (+0)
going to submit 3 htcondor job(s)
submitted 3 htcondor job(s)
09:59:29: all: 159, unsubmitted: 4 (-3), pending: 0 (+0), running: 6 (+3), finished: 149 (+0), retry: 0 (+0), failed: 0 (+0)
09:59:41: all: 159, unsubmitted: 4 (+0), pending: 0 (+0), running: 6 (+0), finished: 149 (+0), retry: 0 (+0), failed: 0 (+0)
09:59:53: all: 159, unsubmitted: 4 (+0), pending: 0 (+0), running: 5 (-1), finished: 150 (+1), retry: 0 (+0), failed: 0 (+0)
going to submit 1 htcondor job(s)
submitted 1 htcondor job(s)
10:00:05: all: 159, unsubmitted: 3 (-1), pending: 0 (+0), running: 5 (+0), finished: 151 (+1), retry: 0 (+0), failed: 0 (+0)
going to submit 1 htcondor job(s)
submitted 1 htcondor job(s)
10:00:17: all: 159, unsubmitted: 2 (-1), pending: 0 (+0), running: 6 (+1), finished: 151 (+0), retry: 0 (+0), failed: 0 (+0)
10:00:30: all: 159, unsubmitted: 2 (+0), pending: 0 (+0), running: 5 (-1), finished: 152 (+1), retry: 0 (+0), failed: 0 (+0)
going to submit 1 htcondor job(s)
submitted 1 htcondor job(s)
10:00:42: all: 159, unsubmitted: 1 (-1), pending: 0 (+0), running: 6 (+1), finished: 152 (+0), retry: 0 (+0), failed: 0 (+0)
10:00:54: all: 159, unsubmitted: 1 (+0), pending: 0 (+0), running: 4 (-2), finished: 154 (+2), retry: 0 (+0), failed: 0 (+0)
going to submit 1 htcondor job(s)
submitted 1 htcondor job(s)
10:01:07: all: 159, unsubmitted: 0 (-1), pending: 0 (+0), running: 4 (+0), finished: 155 (+1), retry: 0 (+0), failed: 0 (+0)
10:01:19: all: 159, unsubmitted: 0 (+0), pending: 0 (+0), running: 4 (+0), finished: 155 (+0), retry: 0 (+0), failed: 0 (+0)
10:01:31: all: 159, unsubmitted: 0 (+0), pending: 0 (+0), running: 4 (+0), finished: 155 (+0), retry: 0 (+0), failed: 0 (+0)
10:01:44: all: 159, unsubmitted: 0 (+0), pending: 0 (+0), running: 3 (-1), finished: 156 (+1), retry: 0 (+0), failed: 0 (+0)
10:01:56: all: 159, unsubmitted: 0 (+0), pending: 0 (+0), running: 3 (+0), finished: 156 (+0), retry: 0 (+0), failed: 0 (+0)
10:02:08: all: 159, unsubmitted: 0 (+0), pending: 0 (+0), running: 3 (+0), finished: 156 (+0), retry: 0 (+0), failed: 0 (+0)
10:02:20: all: 159, unsubmitted: 0 (+0), pending: 0 (+0), running: 3 (+0), finished: 156 (+0), retry: 0 (+0), failed: 0 (+0)
10:02:32: all: 159, unsubmitted: 0 (+0), pending: 0 (+0), running: 3 (+0), finished: 156 (+0), retry: 0 (+0), failed: 0 (+0)
10:02:44: all: 159, unsubmitted: 0 (+0), pending: 0 (+0), running: 2 (-1), finished: 157 (+1), retry: 0 (+0), failed: 0 (+0)
10:02:56: all: 159, unsubmitted: 0 (+0), pending: 0 (+0), running: 1 (-1), finished: 158 (+1), retry: 0 (+0), failed: 0 (+0)
10:03:08: all: 159, unsubmitted: 0 (+0), pending: 0 (+0), running: 1 (+0), finished: 158 (+0), retry: 0 (+0), failed: 0 (+0)
10:03:20: all: 159, unsubmitted: 0 (+0), pending: 0 (+0), running: 1 (+0), finished: 158 (+0), retry: 0 (+0), failed: 0 (+0)
10:03:32: all: 159, unsubmitted: 0 (+0), pending: 0 (+0), running: 1 (+0), finished: 158 (+0), retry: 0 (+0), failed: 0 (+0)
10:03:44: all: 159, unsubmitted: 0 (+0), pending: 0 (+0), running: 1 (+0), finished: 158 (+0), retry: 0 (+0), failed: 0 (+0)
10:03:57: all: 159, unsubmitted: 0 (+0), pending: 0 (+0), running: 1 (+0), finished: 158 (+0), retry: 0 (+0), failed: 0 (+0)
10:04:09: all: 159, unsubmitted: 0 (+0), pending: 0 (+0), running: 0 (-1), finished: 159 (+1), retry: 0 (+0), failed: 0 (+0)
polling took 40 minutes, 6 seconds
INFO: luigi-interface - [pid 697039] Worker Worker(salt=2603756573, workers=5, host=dgx1, username=vasileios.skliris, pid=696537) done      FetchTest(effective_workflow=htcondor, branch=-1, image=/home/vasileios.skliris/aframe/images/data.sif, job_log=, accounting_group_user=vasileios.skliris, accounting_group=ligo.dev.o4.cbc.allsky.aframe, request_disk=1024KB, request_memory=6GB, request_cpus=1, start=1241443783.0, end=1244035783.0, ifos=["H1", "L1"], flag=DATA, min_duration=128.0, sample_rate=2048.0, max_duration=20000.0, prefix=background, channels=["H1", "L1"], condor_directory=/home/vasileios.skliris/aframe/runs/my-first-run_pretrained/condor/fetch_test, data_dir=/home/vasileios.skliris/aframe/runs/my-first-run_pretrained/data/test/background, segments_file=/home/vasileios.skliris/aframe/runs/my-first-run_pretrained/data/test/segments.txt, workflow=htcondor)
INFO: [pid 697039] Worker Worker(salt=2603756573, workers=5, host=dgx1, username=vasileios.skliris, pid=696537) done      FetchTest(effective_workflow=htcondor, branch=-1, image=/home/vasileios.skliris/aframe/images/data.sif, job_log=, accounting_group_user=vasileios.skliris, accounting_group=ligo.dev.o4.cbc.allsky.aframe, request_disk=1024KB, request_memory=6GB, request_cpus=1, start=1241443783.0, end=1244035783.0, ifos=["H1", "L1"], flag=DATA, min_duration=128.0, sample_rate=2048.0, max_duration=20000.0, prefix=background, channels=["H1", "L1"], condor_directory=/home/vasileios.skliris/aframe/runs/my-first-run_pretrained/condor/fetch_test, data_dir=/home/vasileios.skliris/aframe/runs/my-first-run_pretrained/data/test/background, segments_file=/home/vasileios.skliris/aframe/runs/my-first-run_pretrained/data/test/segments.txt, workflow=htcondor)
INFO: luigi-interface - Informed scheduler that task   FetchTest_ligo_dev_o4_cbc__vasileios_skliri__1_b11b95a829   has status   DONE
INFO: Informed scheduler that task   FetchTest_ligo_dev_o4_cbc__vasileios_skliri__1_b11b95a829   has status   DONE
INFO: luigi-interface - [pid 862949] Worker Worker(salt=2603756573, workers=5, host=dgx1, username=vasileios.skliris, pid=696537) running   DeployTestingWaveforms(effective_workflow=htcondor, branch=-1, image=/home/vasileios.skliris/aframe/images/data.sif, job_log=, accounting_group_user=vasileios.skliris, accounting_group=ligo.dev.o4.cbc.allsky.aframe, request_disk=1024KB, request_memory=6GB, request_cpus=1, num_signals=1350000, sample_rate=2048.0, waveform_duration=8.0, prior=priors.priors.end_o3_ratesandpops, minimum_frequency=20.0, reference_frequency=50.0, waveform_approximant=IMRPhenomPv2, coalescence_time=4.0, start=1241443783.0, end=1244035783.0, ifos=["H1", "L1"], shifts=[0, 1], spacing=16.0, buffer=16.0, highpass=32.0, snr_threshold=4.0, psd_length=64.0, seed=1122, output_dir=/home/vasileios.skliris/aframe/runs/my-first-run_pretrained/data/test, condor_directory=/home/vasileios.skliris/aframe/runs/my-first-run_pretrained/condor/testing_waveforms, workflow=htcondor)
INFO: [pid 862949] Worker Worker(salt=2603756573, workers=5, host=dgx1, username=vasileios.skliris, pid=696537) running   DeployTestingWaveforms(effective_workflow=htcondor, branch=-1, image=/home/vasileios.skliris/aframe/images/data.sif, job_log=, accounting_group_user=vasileios.skliris, accounting_group=ligo.dev.o4.cbc.allsky.aframe, request_disk=1024KB, request_memory=6GB, request_cpus=1, num_signals=1350000, sample_rate=2048.0, waveform_duration=8.0, prior=priors.priors.end_o3_ratesandpops, minimum_frequency=20.0, reference_frequency=50.0, waveform_approximant=IMRPhenomPv2, coalescence_time=4.0, start=1241443783.0, end=1244035783.0, ifos=["H1", "L1"], shifts=[0, 1], spacing=16.0, buffer=16.0, highpass=32.0, snr_threshold=4.0, psd_length=64.0, seed=1122, output_dir=/home/vasileios.skliris/aframe/runs/my-first-run_pretrained/data/test, condor_directory=/home/vasileios.skliris/aframe/runs/my-first-run_pretrained/condor/testing_waveforms, workflow=htcondor)
ERROR: luigi-interface - [pid 862949] Worker Worker(salt=2603756573, workers=5, host=dgx1, username=vasileios.skliris, pid=696537) failed    DeployTestingWaveforms(effective_workflow=htcondor, branch=-1, image=/home/vasileios.skliris/aframe/images/data.sif, job_log=, accounting_group_user=vasileios.skliris, accounting_group=ligo.dev.o4.cbc.allsky.aframe, request_disk=1024KB, request_memory=6GB, request_cpus=1, num_signals=1350000, sample_rate=2048.0, waveform_duration=8.0, prior=priors.priors.end_o3_ratesandpops, minimum_frequency=20.0, reference_frequency=50.0, waveform_approximant=IMRPhenomPv2, coalescence_time=4.0, start=1241443783.0, end=1244035783.0, ifos=["H1", "L1"], shifts=[0, 1], spacing=16.0, buffer=16.0, highpass=32.0, snr_threshold=4.0, psd_length=64.0, seed=1122, output_dir=/home/vasileios.skliris/aframe/runs/my-first-run_pretrained/data/test, condor_directory=/home/vasileios.skliris/aframe/runs/my-first-run_pretrained/condor/testing_waveforms, workflow=htcondor)
Traceback (most recent call last):
File "/home/vasileios.skliris/miniconda3-tmp/envs/aframe-EMPJJKiR-py3.9/lib/python3.9/site-packages/luigi/worker.py", line 195, in run
raise RuntimeError('Unfulfilled %s at run time: %s' % (deps, ', '.join(missing)))
RuntimeError: Unfulfilled dependency at run time: FetchTest_ligo_dev_o4_cbc__vasileios_skliri__1_b11b95a829 (TargetCollection(len=159, threshold=159.0))
ERROR: [pid 862949] Worker Worker(salt=2603756573, workers=5, host=dgx1, username=vasileios.skliris, pid=696537) failed    DeployTestingWaveforms(effective_workflow=htcondor, branch=-1, image=/home/vasileios.skliris/aframe/images/data.sif, job_log=, accounting_group_user=vasileios.skliris, accounting_group=ligo.dev.o4.cbc.allsky.aframe, request_disk=1024KB, request_memory=6GB, request_cpus=1, num_signals=1350000, sample_rate=2048.0, waveform_duration=8.0, prior=priors.priors.end_o3_ratesandpops, minimum_frequency=20.0, reference_frequency=50.0, waveform_approximant=IMRPhenomPv2, coalescence_time=4.0, start=1241443783.0, end=1244035783.0, ifos=["H1", "L1"], shifts=[0, 1], spacing=16.0, buffer=16.0, highpass=32.0, snr_threshold=4.0, psd_length=64.0, seed=1122, output_dir=/home/vasileios.skliris/aframe/runs/my-first-run_pretrained/data/test, condor_directory=/home/vasileios.skliris/aframe/runs/my-first-run_pretrained/condor/testing_waveforms, workflow=htcondor)
Traceback (most recent call last):
File "/home/vasileios.skliris/miniconda3-tmp/envs/aframe-EMPJJKiR-py3.9/lib/python3.9/site-packages/luigi/worker.py", line 195, in run
raise RuntimeError('Unfulfilled %s at run time: %s' % (deps, ', '.join(missing)))
RuntimeError: Unfulfilled dependency at run time: FetchTest_ligo_dev_o4_cbc__vasileios_skliri__1_b11b95a829 (TargetCollection(len=159, threshold=159.0))
INFO: luigi-interface - Informed scheduler that task   DeployTestingWaveforms_ligo_dev_o4_cbc__vasileios_skliri__1_6836e28346   has status   FAILED
INFO: Informed scheduler that task   DeployTestingWaveforms_ligo_dev_o4_cbc__vasileios_skliri__1_6836e28346   has status   FAILED
INFO: luigi-interface - Informed scheduler that task   DeployTestingWaveforms_ligo_dev_o4_cbc__vasileios_skliri__1_6836e28346   has status   PENDING
INFO: Informed scheduler that task   DeployTestingWaveforms_ligo_dev_o4_cbc__vasileios_skliri__1_6836e28346   has status   PENDING
INFO: luigi-interface - Informed scheduler that task   FetchTest_ligo_dev_o4_cbc__vasileios_skliri__1_b11b95a829   has status   PENDING
INFO: Informed scheduler that task   FetchTest_ligo_dev_o4_cbc__vasileios_skliri__1_b11b95a829   has status   PENDING
INFO: luigi-interface - Informed scheduler that task   Query_1244035783_0_DATA___H1____L1___c10d7b66a3   has status   DONE
INFO: Informed scheduler that task   Query_1244035783_0_DATA___H1____L1___c10d7b66a3   has status   DONE
INFO: luigi-interface - [pid 862990] Worker Worker(salt=2603756573, workers=5, host=dgx1, username=vasileios.skliris, pid=696537) running   DeployTestingWaveforms(effective_workflow=htcondor, branch=-1, image=/home/vasileios.skliris/aframe/images/data.sif, job_log=, accounting_group_user=vasileios.skliris, accounting_group=ligo.dev.o4.cbc.allsky.aframe, request_disk=1024KB, request_memory=6GB, request_cpus=1, num_signals=1350000, sample_rate=2048.0, waveform_duration=8.0, prior=priors.priors.end_o3_ratesandpops, minimum_frequency=20.0, reference_frequency=50.0, waveform_approximant=IMRPhenomPv2, coalescence_time=4.0, start=1241443783.0, end=1244035783.0, ifos=["H1", "L1"], shifts=[0, 1], spacing=16.0, buffer=16.0, highpass=32.0, snr_threshold=4.0, psd_length=64.0, seed=1122, output_dir=/home/vasileios.skliris/aframe/runs/my-first-run_pretrained/data/test, condor_directory=/home/vasileios.skliris/aframe/runs/my-first-run_pretrained/condor/testing_waveforms, workflow=htcondor)
INFO: [pid 862990] Worker Worker(salt=2603756573, workers=5, host=dgx1, username=vasileios.skliris, pid=696537) running   DeployTestingWaveforms(effective_workflow=htcondor, branch=-1, image=/home/vasileios.skliris/aframe/images/data.sif, job_log=, accounting_group_user=vasileios.skliris, accounting_group=ligo.dev.o4.cbc.allsky.aframe, request_disk=1024KB, request_memory=6GB, request_cpus=1, num_signals=1350000, sample_rate=2048.0, waveform_duration=8.0, prior=priors.priors.end_o3_ratesandpops, minimum_frequency=20.0, reference_frequency=50.0, waveform_approximant=IMRPhenomPv2, coalescence_time=4.0, start=1241443783.0, end=1244035783.0, ifos=["H1", "L1"], shifts=[0, 1], spacing=16.0, buffer=16.0, highpass=32.0, snr_threshold=4.0, psd_length=64.0, seed=1122, output_dir=/home/vasileios.skliris/aframe/runs/my-first-run_pretrained/data/test, condor_directory=/home/vasileios.skliris/aframe/runs/my-first-run_pretrained/condor/testing_waveforms, workflow=htcondor)
ERROR: luigi-interface - [pid 862990] Worker Worker(salt=2603756573, workers=5, host=dgx1, username=vasileios.skliris, pid=696537) failed    DeployTestingWaveforms(effective_workflow=htcondor, branch=-1, image=/home/vasileios.skliris/aframe/images/data.sif, job_log=, accounting_group_user=vasileios.skliris, accounting_group=ligo.dev.o4.cbc.allsky.aframe, request_disk=1024KB, request_memory=6GB, request_cpus=1, num_signals=1350000, sample_rate=2048.0, waveform_duration=8.0, prior=priors.priors.end_o3_ratesandpops, minimum_frequency=20.0, reference_frequency=50.0, waveform_approximant=IMRPhenomPv2, coalescence_time=4.0, start=1241443783.0, end=1244035783.0, ifos=["H1", "L1"], shifts=[0, 1], spacing=16.0, buffer=16.0, highpass=32.0, snr_threshold=4.0, psd_length=64.0, seed=1122, output_dir=/home/vasileios.skliris/aframe/runs/my-first-run_pretrained/data/test, condor_directory=/home/vasileios.skliris/aframe/runs/my-first-run_pretrained/condor/testing_waveforms, workflow=htcondor)
Traceback (most recent call last):
File "/home/vasileios.skliris/miniconda3-tmp/envs/aframe-EMPJJKiR-py3.9/lib/python3.9/site-packages/luigi/worker.py", line 195, in run
raise RuntimeError('Unfulfilled %s at run time: %s' % (deps, ', '.join(missing)))
RuntimeError: Unfulfilled dependency at run time: FetchTest_ligo_dev_o4_cbc__vasileios_skliri__1_b11b95a829 (TargetCollection(len=159, threshold=159.0))
ERROR: [pid 862990] Worker Worker(salt=2603756573, workers=5, host=dgx1, username=vasileios.skliris, pid=696537) failed    DeployTestingWaveforms(effective_workflow=htcondor, branch=-1, image=/home/vasileios.skliris/aframe/images/data.sif, job_log=, accounting_group_user=vasileios.skliris, accounting_group=ligo.dev.o4.cbc.allsky.aframe, request_disk=1024KB, request_memory=6GB, request_cpus=1, num_signals=1350000, sample_rate=2048.0, waveform_duration=8.0, prior=priors.priors.end_o3_ratesandpops, minimum_frequency=20.0, reference_frequency=50.0, waveform_approximant=IMRPhenomPv2, coalescence_time=4.0, start=1241443783.0, end=1244035783.0, ifos=["H1", "L1"], shifts=[0, 1], spacing=16.0, buffer=16.0, highpass=32.0, snr_threshold=4.0, psd_length=64.0, seed=1122, output_dir=/home/vasileios.skliris/aframe/runs/my-first-run_pretrained/data/test, condor_directory=/home/vasileios.skliris/aframe/runs/my-first-run_pretrained/condor/testing_waveforms, workflow=htcondor)
Traceback (most recent call last):
File "/home/vasileios.skliris/miniconda3-tmp/envs/aframe-EMPJJKiR-py3.9/lib/python3.9/site-packages/luigi/worker.py", line 195, in run
raise RuntimeError('Unfulfilled %s at run time: %s' % (deps, ', '.join(missing)))
RuntimeError: Unfulfilled dependency at run time: FetchTest_ligo_dev_o4_cbc__vasileios_skliri__1_b11b95a829 (TargetCollection(len=159, threshold=159.0))
INFO: luigi-interface - Informed scheduler that task   DeployTestingWaveforms_ligo_dev_o4_cbc__vasileios_skliri__1_6836e28346   has status   FAILED
INFO: Informed scheduler that task   DeployTestingWaveforms_ligo_dev_o4_cbc__vasileios_skliri__1_6836e28346   has status   FAILED
INFO: luigi-interface - Worker Worker(salt=2603756573, workers=5, host=dgx1, username=vasileios.skliris, pid=696537) was stopped. Shutting down Keep-Alive thread
INFO: Worker Worker(salt=2603756573, workers=5, host=dgx1, username=vasileios.skliris, pid=696537) was stopped. Shutting down Keep-Alive thread
INFO: luigi-interface -
===== Luigi Execution Summary =====
Scheduled 10 tasks of which:

1 complete ones were encountered:

1 Train(...)

2 ran successfully:

1 FetchTest(...)
1 Query(...)

2 failed:

1 DeployTestingWaveforms(...)
1 ExportLocal(...)

5 were left pending, among these:

5 had failed dependencies:

1 DeployInferLocal(...)
1 Sandbox(...)
1 SandboxInfer(...)
1 SandboxSV(...)
1 TestingWaveforms(...)

This progress looks :( because there were failed tasks
===== Luigi Execution Summary =====
INFO:
===== Luigi Execution Summary =====
Scheduled 10 tasks of which:

1 complete ones were encountered:

1 Train(...)

2 ran successfully:

1 FetchTest(...)
1 Query(...)

2 failed:

1 DeployTestingWaveforms(...)
1 ExportLocal(...)

5 were left pending, among these:

5 had failed dependencies:

1 DeployInferLocal(...)
1 Sandbox(...)
1 SandboxInfer(...)
1 SandboxSV(...)
1 TestingWaveforms(...)

This progress looks :( because there were failed tasks
===== Luigi Execution Summary =====`


fix(bin/ofs): build failed when target_os != linux

We should add some basic README for ofs project and mention that ofs doesn't support windows yet.


add doc for https://github.com/mautic/mautic/pull/8196

closes https://github.com/mautic/mautic-documentation/issues/253

Looks great 👍,  thank you @npracht  I just left a small suggestion.

@RCheesley please review


Wrong iphdr and tcphdr mapping

Describe the bug
https://github.com/l3af-project/eBPF-Package-Repository/pull/54 is failing which tries to use iphdr and tcphdr in cross-platform code.
https://github.com/microsoft/ebpf-for-windows/blob/main/include/net/ip.h#L94 defines it as _IPV4_HEADER, but the field names in that struct don't match the expected field names in iphdr which is causing the windows test failures in https://github.com/l3af-project/eBPF-Package-Repository/pull/54.
OS information
No response
Steps taken to reproduce bug
Try to compile https://github.com/l3af-project/eBPF-Package-Repository/pull/54
Expected behavior
Build should succeed
Actual outcome
Build fails
Additional details
D:/a/eBPF-Package-Repository/eBPF-Package-Repository/ratelimiting/ratelimiting_kern.c(103,27): error G1A4676F8: no member named 'ihl' in 'struct _IPV4_HEADER' [D:\a\eBPF-Package-Repository\eBPF-Package-Repository\build\ratelimiting\ratelimiting_kern.vcxproj]
      __u8 l4_offset = iph->ihl * 4; // ipv4 header length
                       ~~~  ^
D:/a/eBPF-Package-Repository/eBPF-Package-Repository/ratelimiting/ratelimiting_kern.c(114,31): error G1A4[67](https://github.com/l3af-project/eBPF-Package-Repository/actions/runs/5384061159/jobs/9771541350?pr=54#step:5:68)6F8: no member named 'doff' in 'struct tcphdr' [D:\a\eBPF-Package-Repository\eBPF-Package-Repository\build\ratelimiting\ratelimiting_kern.vcxproj]
      __u16 data_offset = tcph->doff * 4; // tcp header length
                          ~~~~  ^

https://github.com/microsoft/ebpf-for-windows/blob/main/include/net/tcp.h#L11 has data_offset instead of doff

@dthaler Do you want to add Linux-compatible field names for the IPv6 header, too, while we are at it?


I want to cancel timer.wait from another thread, is this possible?

    timers = Timers::Group.new
    five_second_timer = timers.after(10) { Thread.kill(runningThread) }
    runningThread = Thread.new do
      # this task cost me 5 seconds, then I want to cancel the timer from status of waiting ,is this possible?
    end
    timers.wait

The timers gem presently does not have cross-thread support. If you would like something like this, I would suggest using Celluloid

Perhaps you can explain a bit more what you are trying to achieve?

And feel free to take this to the Google Group where we discuss things like this all the time.

Okay, I'll try Celluloid, thank you.@tarcieri


FOX ESS Battery

This manufacturer seems not yet included in the list of devices. Interfaces are RS485, DRM and CAN.
Is any other device similar or can somebody design the connection for EVCC?

Does is have Modbus over RS485? Do you have docs?

this is the only details I could find regarding the interface:
Fox ESS.pdf

That is unfortunately not nearly enough. Happy to look into this again if you can find a description of the modbus registers.

Actually, this looks decent: https://github.com/StealthChesnut/HA-FoxESS-Modbus/wiki/Data-Register-Reference---H1-AC1

	Super! Danke
	Mit besten Grüßen / With best regardsMartin Gruß / GrussMobile +4915781590839

Von: andig @.>Gesendet: Sonntag, Januar 8, 2023 3:00 PMAn: evcc-io/evcc @.>Cc: grusslev @.>; Author @.>Betreff: Re: [evcc-io/evcc] FOX ESS Battery (Issue #5690)
Actually, this looks decent: https://github.com/StealthChesnut/HA-FoxESS-Modbus/wiki/Data-Register-Reference---H1-AC1
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>


"Master did not receieve heartbeat" on long ingest

Not sure if this is an issue anymore. Closing until it's observed again.

Still a problem.


fix(deps): update module github.com/onsi/ginkgo/v2 to v2.22.1

This PR contains the following updates:

Package
Type
Update
Change

github.com/onsi/ginkgo/v2
require
patch
v2.22.0 -> v2.22.1

Release Notes

onsi/ginkgo (github.com/onsi/ginkgo/v2)
v2.22.1
Compare Source
2.22.1
Fixes
Fix CSV encoding

Update tests [aab3da6]
Properly encode CSV rows [c09df39]
Add test case for proper csv escaping [96a80fc]
Add meta-test [43dad69]

Maintenance

ensure *.test files are gitignored so we don't accidentally commit compiled tests again [c88c634]
remove golang.org/x/net/context in favour of stdlib context [4df44bf]

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

ℹ Artifact update notice
File name: go.mod
In order to perform the update(s) described in the table above, Renovate ran the go get command, which resulted in the following additional change(s):

9 additional dependencies were updated

Details:

Package
Change

github.com/onsi/gomega
v1.36.0 -> v1.36.1

github.com/google/pprof
v0.0.0-20241029153458-d1b30febd7db -> v0.0.0-20241210010833-40e02aabc2ad

golang.org/x/crypto
v0.29.0 -> v0.30.0

golang.org/x/net
v0.31.0 -> v0.32.0

golang.org/x/sync
v0.9.0 -> v0.10.0

golang.org/x/sys
v0.27.0 -> v0.28.0

golang.org/x/term
v0.26.0 -> v0.27.0

golang.org/x/text
v0.20.0 -> v0.21.0

golang.org/x/tools
v0.27.0 -> v0.28.0


Add support for code engine

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.2 out of 5 committers have signed the CLA.:white_check_mark: balazs-marjan:white_check_mark: hiltol:x: sonkatomi:x: eugene-koo:x: kzhang-2017eugene-koo seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


Runtime exception using getShow and providing wrong type arguments.

The following produces a runtime exception with no type errors.
import { getShow } from '@unsplash/sum-types-fp-ts';
import * as Sum from '@unsplash/sum-types';

type S = Sum.Member<'A'> | Sum.Member<'B'>;
const S = Sum.create<S>();

const show = getShow<S>({
  A: 'Custom',
});

console.log(show.show(S.mk.A));

https://stackblitz.com/edit/typescript-5afkt4?file=index.ts
I believe if supplied, A must be an instance of Show but TS doesn't check for it.
Side note, I find it a little confusing prepending the sum member key in the show output here https://github.com/unsplash/sum-types-fp-ts/blob/master/src/index.ts#L153 . In the following code I'm expecting the show output to be CustomA but instead I get A CustomerA
const show = getShow<S>({
  A: {
    show: () => "CustomA"
  }
});

This appears to only be an issue with nullary sums. Add any non-nullary member and A will be flagged as an unknown property.


No auto-push of symbols with 4.5.0

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

README.md
:white_check_mark:Succeeded

docs/Create-Packages/Creating-a-Package.md
:white_check_mark:Succeeded

docs/Create-Packages/Publish-a-package.md
:white_check_mark:Succeeded

docs/Create-Packages/Symbol-Packages.md
:white_check_mark:Succeeded

docs/Release-Notes/Index.md
:white_check_mark:Succeeded

docs/Release-Notes/NuGet-4.5-RTM.md
:white_check_mark:Succeeded

docs/Schema/nuspec.md
:white_check_mark:Succeeded

docs/Tools/cli-ref-pack.md
:white_check_mark:Succeeded

docs/What-is-NuGet.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/Create-Packages/Publish-a-package.md
:white_check_mark:Succeeded

docs/Create-Packages/Symbol-Packages.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/Create-Packages/Publish-a-package.md
:white_check_mark:Succeeded

docs/Create-Packages/Symbol-Packages.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/Create-Packages/Publish-a-package.md
:white_check_mark:Succeeded

docs/Create-Packages/Symbol-Packages.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Add basic support for building with GCC

This doesn't resolve all of the issues experienced when building the toolchain with GCC, but at least it allows the build script to start building LLVM. This is a prerequisite for building the whole toolchain on Alpine Linux with Musl.

Related to SR-4632.

@swift-ci please smoke test

I'm not sure that this is valuable in its current state - the gcc build is horribly broken as of my last test (~2y ago).  The generated compiler (after fixing the build - which will need to happen again) miscompiles the standard library.  I would rather that we hold off on this until the other fixes are merged.

CC: @etcwilde

miscompiles the standard library. I would rather that we hold off on this until the other fixes are merged.

Are you referring to the Swift standard library here? If so, that's not the configuration I'm fixing here. The goal is not to compile swiftc with GCC, but to compile LLVM/clang with GCC, which in turn can then compile swiftc on their own. Wouldn't that avoid miscompiles then?

@swift-ci please test macOS platform

Can you describe why apk add clang is insufficient for this?  Adding this new build model adds a dependency on the split swift build which has a ~10% build penalty (compared to the current system which would allow for the build-script to build with the unified swift build).  The current instructions for Ubuntu and CentOS recommend that you install the host's clang packages.  We should document the same for Alpine and recommend that you install the clang package.


feat(alarm): show keypad only for disarming

Description
Add a new configuration property in the alarm card to show the keypad only for disarming (when the alarm is not disarmed).
Related Issue
787 - [Feature]: show keypad only for disarming in alarm card
Motivation and Context
To improve the alarm card. There are alarms that can be configured only for requiring the code for disarming (e.g., securitas)
How Has This Been Tested
With my Home Assistant instance.
Types of changes

[ ] 🐛 Bug fix (non-breaking change which fixes an issue)
[X] 🚀 New feature (non-breaking change which adds functionality)
[ ] 🌎 Translation (addition or update a translation)
[ ] ⚠ Breaking change (fix or feature that would cause existing functionality to change)

Checklist

[X] My code follows the code style of this project.
[X] My change requires a change to the documentation.
[X] I have updated the documentation accordingly.
[X] I have tested the change locally.

Can we transform the boolean to a select (always, never, disarm) ? (empty default value will be never)

Sure! Done in 5a0f502.

@rogelio-o was this actually released? No show_keypad_disarm option on my end.

@piitaya I've rebased the branch and fixed the conflicts. Can you take a look? Thanks


Slow searching in powertoys run

Microsoft PowerToys version
v0.85.0
Installation method
PowerToys auto-update
Running as admin
None
Area(s) with issue?
PowerToys Run
Steps to reproduce
https://github.com/user-attachments/assets/bae6c180-0530-4a84-9810-fbbca6b7bb1c
✔ Expected Behavior
Faster searching
❌ Actual Behavior
the searching of powertoys run is significantly slower.
used to be much much faster but it has been like this for the previous three versions
tried updating, downloading from github page, reinstalling, running as admin. even tried some of the powertoys run settings but none could fix this issue.
i have another device with the same issue of powertoys run. but it resolved with the latest update
Other Software
No response

/bugreport , might give us an idea what plugin might be taken longer. Also, can you please try to turn off the plugins one by one? Might help in debugging which plugin might be causing this behavior for you.

PowerToysReport_2024-10-02-19-29-02.zip

idk which plugin is causing this issue. as far as i remember I didnt change any plugin the time I started experiencing this issue.
also the issue on my other device resolved immediately after an update without changing any settings.
I tried running powertoys as an administrator but it didnt change anything

tried turning off every plugin. didnt change anything
https://github.com/user-attachments/assets/9f2e93a5-0da1-4461-89de-8c109a3bcbb8

Hi @ahmedusmanshahid,

Can you check these settings in PTRun settings?
If the settings are not the same as in the screenshot, can you change them and try again? If you see same issue again can you try again by increasing these values? For example, you can set "Background execution plugins" to 500.

https://github.com/user-attachments/assets/83c69487-aef2-4b00-a9a3-9dfd7950aa86
yup same settings.
nvm i found the issue. It was a bug in the program plugin.

Hi @ahmedusmanshahid,
When I examine the logs, the problem appears to be in the program plugin.
When I tried to reproduce the issue, I could not reproduce it. Can you share your PC specs with me?
While trying again, can you open Task Manager and check if there is an application slowing down your CPU and memory? I think a program is affecting PTRun. If we can find it, I can install and test the same app and find out what's causing the problem.

Laptop: Dell latitude 5320 with touch support
Windows Specification: (copied from about section in windows>system)
Edition	Windows 11 Pro
Version	23H2
Installed on	03/08/2024
OS build	22631.4169
Experience	Windows Feature Experience Pack 1000.22700.1034.0
https://github.com/user-attachments/assets/6fd6094b-a719-4d60-ae96-bb3436f2a02f
https://github.com/user-attachments/assets/35f27976-2f75-4229-8b98-30ce4782b914

sorry for the quick video. github only allows 10 mb or less videos. feel free to slow it down or pause it

sorry for the quick videos. github only allows 10 mb or less videos. feel free to slow it down or pause it

Laptop: Dell latitude 5320 with touch support
Windows Specification: (copied from about section in settings>system)
Edition Windows 11 Pro
Version 23H2
Installed on 03/08/2024
OS build 22631.4169
Experience Windows Feature Experience Pack 1000.22700.1034.0
https://github.com/user-attachments/assets/e96b4cc5-a898-4e2f-ae64-b02a065801f1
https://github.com/user-attachments/assets/aed5316b-2746-4ead-a9ad-055275122e0e

I have had problems similar to @ahmedusmanshahid's for about a year or so. When my computer first boots up, I let all of the apps start up and then wait until everything is finished. When all the startup apps have loaded, I press the PowerToys Run toggle and it takes 15-17 seconds every single time. I had to stop using PowerToys Run and switch to Flow Launcher because I can't disable the "Programs" plugin. That's one of the main reasons I want to use PowerToys Run.
I watched the video of his Task Manager and I noticed that I have Everything.exe installed too. Could it be possible that this has something to do with the Search Indexer or something? I have that disabled because I use Everything.exe.
I really hope that you guys find the solution to this because this is the single reason that I have not been able to use PowerToys Run, and I want to use it so badly!

I tried closing everything.exe and the issue still persists.
@hkpD I use the regular windows 11 search bar to search for programs and use Powertoys run for websearches, for the time being.


Update chroma.py

provided support for user provided embeddings in add_texts and add_images function
Thank you for contributing to LangChain!

[ ] PR title: "package: description"

Where "package" is whichever of langchain, community, core, experimental, etc. is being modified. Use "docs: ..."  for template changes, "infra: ..." for CI changes.
Example: "community: add foobar LLM"

[ ] PR message: Delete this entire checklist and replace with

Description: a description of the change
Issue: the issue # it fixes, if applicable
Dependencies: any dependencies required for this change
Twitter handle: if your PR gets announced, and you'd like a mention, we'll gladly shout you out!

[ ] Add tests and docs: If you're adding a new integration, please include

a test for the integration, preferably unit tests that do not rely on network access,
an example notebook showing its use. It lives in docs/docs/integrations directory.

[ ] Lint and test: Run make format, make lint and make test from the root of the package(s) you've modified. See contribution guidelines for more: https://python.langchain.com/docs/contributing/

Additional guidelines:

Make sure optional dependencies are imported within a function.
Please do not add dependencies to pyproject.toml files (even optional ones) unless they are required for unit tests.
Most PRs should not touch more than one package.
Changes should be backwards compatible.
If you are adding something to community, do not re-import it in langchain.

If no one reviews your PR within a few days, please @-mention one of baskaryan, efriis, eyurtsev, hwchase17.

Hey Baskaran,
could you please do the same thanks for the help...

should be added in the langchain_chroma package


Can I have timestamp too in the fileNamePattern when using RollingFilelogger

Now I am able to put in DATE format as a filenamepattern. But, i really want timestamp to be included in the filename pattern. is it possible ??

yes, for the simple-file-logger but no for the rolling logger.  rolling uses the date format to determine when to roll, so a timestamp would roll each minute--probably not what you want.


update AAMAS 2023

AAMAS '23 https://aamas2023.soton.ac.uk/

@jault should reopen this? Seems the AAMAS conference needs updated?


Revert "update global dictionary to 124."

Reverts istio/mixerclient#90 temprorarily to unblock istio/proxy#398

Jenkins job mixerclient/presubmit passed

Jenkins job mixerclient/presubmit passed

Jenkins job mixerclient/presubmit passed


Docker images for BigBen

It would be awesome to provide docker images or at least instructions how to create proper BigBen docker deployments.
Ideally, include docker-compose.yml example in readme.
I'm gonna try that and in case of success I will create proper PR for that.

That would be great. Thanks Andrii for the help. Let me know if you need something.
Thanks,
Sandeep

@sandeepmalik regarding lib/src/main/resources/hz.template.xml.
It should be somehow mounted into hazelcast image?

Hi Andrii,
We are running hz in embedded mode so no separate mounting is required.
This error usually indicates that real cause is elsewhere. Most likely
server didn't start properly which caused hazelcast to shutdown as well and
when scheduler tried to access hz in a separate thread it threw 'not
active' error.
If you can send me the complete logs then I'll be able to help more.
Thanks
Sandeep
On Mon, Dec 3, 2018, 12:29 AM Andrii<EMAIL_ADDRESS>wrote:

I am facing the following exception:
com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.hazelcast.spi.impl.proxyservice.impl.ProxyRegistry.getOrCreateProxyFuture(ProxyRegistry.java:152)
at com.hazelcast.spi.impl.proxyservice.impl.ProxyRegistry.getOrCreateProxy(ProxyRegistry.java:135)
at com.hazelcast.spi.impl.proxyservice.impl.ProxyServiceImpl.getDistributedObject(ProxyServiceImpl.java:147)
at com.hazelcast.instance.HazelcastInstanceImpl.getDistributedObject(HazelcastInstanceImpl.java:376)
at com.hazelcast.instance.HazelcastInstanceImpl.getMap(HazelcastInstanceImpl.java:182)
at com.hazelcast.instance.HazelcastInstanceProxy.getMap(HazelcastInstanceProxy.java:96)
at com.walmartlabs.bigben.core.BucketManager.(BucketManager.kt:61)
at com.walmartlabs.bigben.core.ScheduleScanner.init(ScheduleScanner.kt:83)
at com.walmartlabs.bigben.utils.hz.ClusterSingleton$1.run(ClusterSingleton.kt:69)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Which information I could provide to resolve this issue?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/walmartlabs/bigben/issues/7#issuecomment-443627745,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AD-w-sayos4d-IDWPRF4CI1Mz5gdmPemks5u1ODWgaJpZM4Y9nEN
.

Hi,
I collected all logs. Maybe it is somehow related to docker networks.. but still strange.
Because BigBen sees Cassandra with no issues.
hz-issue.zip
I can create patch / WIP PR if needed.

My main goal is to prepare docker-compose at least for development / trying purposes

Ahh I see now, in your yaml file, you have removed the modules but left the key "modules" there (which yaml parser interprets as having null value). The code throws a null pointer on server start up and shuts down hazelcast. I'll fix and push the fix. For the time being, just remove the "modules" key as well from the yaml file and it should be fine.

As a matter of fact, I fixed this issue some time back in dev but haven't merged the changes back to master.
https://github.com/walmartlabs/bigben/commit/9a4f776c79cc9d24d225970d9dd7a8a5cae0d48b#diff-24b3a0cc809cf5576c60c4377309fc63R119
I'll do that soon, for the time being just remove the "modules" key and redeploy. Let me know if any issues.

Awesome, it fixed the issue.
04-Dec-2018 07:30:58.430 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 24124 ms

Hi @aaabramov, can't find docker-compose file.
Could you pls share, if you have implemented already?


fix text/html to concat multiline strings

For #7873

[x] Pull request represents a single change (i.e. not fixing disparate/unrelated things in a single PR)
[x] Title summarizes what is changing
[x] Has a news entry file (remember to thank yourself!)
[ ] Appropriate comments and documentation strings in the code
[ ] Has sufficient logging.
[ ] Has telemetry for enhancements.
[ ] Unit tests & system/integration tests are added/updated
[ ] Test plan is updated as appropriate
[ ] package-lock.json has been regenerated by running npm install (if dependencies have changed)
[ ] The wiki is updated with any design decisions/details.

Codecov Report

Merging #8328 into master will decrease coverage by 0.06%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##           master   #8328      +/-   ##
=========================================
- Coverage   58.66%   58.6%   -0.07%
=========================================
  Files         505     505
  Lines       22778   22778
  Branches     3740    3740
=========================================
- Hits        13363   13349      -14
- Misses       8522    8527       +5
- Partials      893     902       +9

Impacted Files
Coverage Δ

src/client/common/utils/platform.ts
64.7% <0%> (-23.53%)
:arrow_down:

src/client/common/platform/pathUtils.ts
66.66% <0%> (-5.56%)
:arrow_down:

src/client/linters/pydocstyle.ts
84.61% <0%> (-5.13%)
:arrow_down:

src/client/datascience/debugLocationTracker.ts
79.24% <0%> (-3.78%)
:arrow_down:

src/client/interpreter/locators/helpers.ts
70.73% <0%> (-2.44%)
:arrow_down:

src/client/common/process/proc.ts
13.28% <0%> (-1.57%)
:arrow_down:

src/client/common/installer/productInstaller.ts
76.71% <0%> (-0.53%)
:arrow_down:

src/client/datascience/constants.ts
99.66% <0%> (-0.34%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0774fb8...e74fd86. Read the comment docs.


Your app contains exposed Google Cloud Platform (GCP) API keys

Your app contains exposed Google Cloud Platform (GCP) API keys, the Classes:
MainActivity->onCreate
Please refer to the following article on how to fix the issue:
https://support.google.com/faqs/answer/9287711
I am getting this alert from my google play store console.  I have added restrictions to my apikey. but i have used same line as demo
**String apiKey = getResources().getString(R.string.google_api_key);
if (!Places.isInitialized()) {
Places.initialize(getApplicationContext(), apiKey);
}**
this is cause of this issue. How to rectify this. Can you please explain

@DurgaArunkumarSmitiv I'll try to take a look at this but you may get a faster response on StackOverflow with tag google-maps. See here for more information: https://developers.google.com/maps/documentation/android-sdk/get-api-key#key-std-details

Did you have any solution for this issue?

HI @DurgaArunkumarSmitiv what method did you use to include the API key in your app? Did you add the API key via gradle.properties (which will autogenerate an entry in .xml) or are you storing the API key in .xml directly?

@arriolac
I get the same issue and I have implemented my keys via gradle.properties which is the safe way. I believe because its actually called google_maps_key somehow that triggers the warning which is not a wise method for detecting flaws in the app.

Hi
Yes even from gradle faced same issue. Getting the keys from server
helped me.
Regards
Durga
Android Developer
On Fri, Feb 28, 2020 at 11:34 AM Ali Mojarrad<EMAIL_ADDRESS>wrote:

@arriolac https://github.com/arriolac
I get the same issue and I have implemented my keys via gradle.properties
which is the safe way. I believe because its actually called
google_maps_key somehow that triggers the warning which is not a wise
method for detecting flaws in the app.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/googlemaps/android-places-demos/issues/29?email_source=notifications&email_token=ALJYQGXKFB2OG5RRRT5FZFTRFCSPLA5CNFSM4IXKNDE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENHDVUQ#issuecomment-592329426,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ALJYQGWQHDU7AFGUUHRQSPLRFCSPLANCNFSM4IXKNDEQ
.

I can confirm that the warning shows even with the build time gradle.properties key/value property approach (see instructions here). While this approach is certainly better than storing the API key in source control, please refer to our API key best practices to ensure your API key is absolutely secure. To remove the warning itself from the Play Console, you'll have to completely remove the API from your code (@DurgaArunkumarSmitiv approach works or you can store an encrypted version of the API key in code).

Today while uploading a repo I got a warning about an MAPS API key that I replaced already back in January. Why is that? I can't remove it from the repo as it is part of very old commits only in files long since heavily modified and replaced/moved, with the key removed from themin the process..

I am having the same issue, I put the key in a property file, and reading that file in the gradle, but that key will be in BuildConfig.java once it's generate, and that is what is complaining about it, I am not sure if is truly a problem or just like false-positive.
Did anyone find a solution to this issue?

@DurgaArunkumarSmitiv did you have a solution for this problem ?
Thanks!

The warning in the Play Console can be safely ignored assuming the best practices outlined in Securing an API key are applied. Specifically, make sure to apply API key restrictions and ensure that the API key is not checked into source code (see PR #78 as an example for how to do this).

Our FAQ page for Remediation for Exposed GCP API keys has been updated to reflect that the warning can be safely ignored.

Add restrictions to your API key so that only your apps are allowed to use the API key. More details on adding restrictions to API keys can be found here. (Please Note: If you have already added restrictions to your API key, you can ignore this warning.)

Please note that this does not resolve the problem I reported here: a warning about publishing an API key that I replaced in GCP console with an unpublished one already.

I also faced the same problem. I couldn't find any proper solution for this. After lot of searching I found a solution. So I am giving a complete guide for this issue.
Link : https://stackoverflow.com/a/71155071/13387867

I'm also facing the same problem, with the play console.


⚠ Spryr has degraded performance

In c928e4a, Spryr (https://spryr.com) experienced degraded performance:

HTTP code: 200
Response time: 5295 ms

Resolved: Spryr performance has improved in de92b0c after 7 minutes.


Conda package?

Hi, after some experimentation with codon, i could not make it work. Probably some unsatisfied
dependency (installing python3-dev amd build-essential seemed to help with the install).
Ultimately I think the error is due to the fact that I use anaconda for Python. I believe an easy to install conda package could be created for codon as there are already packages for gcc for example.
Have the maintainers considered this? This is what conda was designed to solve  and lots of people in the data science community use conda, so I think it could be a useful addition.

+1

When I have to use Python, I use mamba as my package manager. A conda-forge package for Codon I think would be a nice easy way of being able to try it out, and also containing some of the dependency problems that might come out - and if mamba is used, then there's no need for a python version to run conda itself.

+1


CosmosDB support for TransactionalBatch

Is your feature request related to a problem? Please describe.
I would like to make atomic bulk requests to cosmosdb, like in the .NET SDK: https://devblogs.microsoft.com/cosmosdb/introducing-transactionalbatch-in-the-net-sdk/
Describe the solution you'd like
items.bulk() bulkOptions should support atomic but I'm unsure about the interaction with continueOnError.
I suppose an enum would be better: mode: 'atmic' | 'bulk' | 'bulkStopOnError' or so.
Notes
I ran into this https://github.com/Azure/azure-sdk-for-js/blob/f44959a46f3a5f7b16a9fe37ee1e9fae31dfe42b/sdk/cosmosdb/cosmos/src/ClientContext.ts#L581
Would it be that simple? (Not sure how I could override the header atm)

Hey @patroza we have open issues for this. TransactionalBatch isn't supported yet but is in the pipeline

Closing in favor of: https://github.com/Azure/azure-sdk-for-js/issues/13368


Added Dockerfile + fixes related to angular bower version

I have added Dockerfile for this project so people can run it inside container.
How to test.
git clone https://github.com/i63/pos.git
cd pos
docker build -t  pos  .
docker run -p 8080:8080 pos

Why is the angular version requiring a specific version resolution in bower.json? What conflicts with it?


Merge developent into master

Development is stable, merging into master.

Merging all the development branch so far.
It is stable !


NullReferenceException thrown while using MAML transformer

Steps to reproduce
var markdown = @"
# Get-Foo

## EXAMPLES
### --EXAMPLE1--

```
# PS code here
```
Remarks
"
var parser = new MarkdownParser();
var doc = parser.ParseString(new string[] { markdown });
var mamlCommand = NodeModelToMamlModelV2(doc).ToArray();

Note: there're some extra whitespaces below ### --EXAMPLE1--
Expected behavior
Transform successfully or report friendly error message to help fix the content.
Actual behavior
Thrown System.NullReferenceException: 'Object reference not set to an instance of an object.'
Environment data
Base on latest code in master branch

Oh good catch

We believe this is no longer an issue in the current version 0.14.1.


fal installation instructions with dbt 1.0

Trying fal now after a recommendation through Slack. I installed dbt 1.0 by pip install dbt-core, as well as fal by pip install fal.
I ran into a few snags when installing fal in a clean python 3.9 environment, and it might be worth adding a note about dbt 1.0 to the fal readme.

fal currently doesn't install dbt. I assume this is because of the change from dbt to dbt-core.
fal relies on bigquery here, which means that one needs to run pip install dbt-bigquery. This should be explicitly stated on the readme, and preferably not be a requirement.

For the second point, it would be much better to use:
from dbt.adapters.factory import load_plugin
load_plugin("bigquery") # returns dbt.adapters.bigquery.connections.BigQueryCredentials

Hi @thomasaarholt! Thanks for raising this issue. We are actively working on supporting dbt 1.0 (see #69) and should release the new version soon. You're right, we should add a note about dbt 1.0 in README.

Ah! I searched for 1.0 and didn't find the PR (I had to type 1.0.0...) :)

My bigquery fix above works on 0.20 as well, I just checked:

Should I make a PR?

Ah, I see you have a fix as well in the 1.0.0 PR :) I'll leave that to you!

@thomasaarholt, we have just released a version compatible with dbt 1.0.0.
pip install fal==0.1.35

Please let us know how it goes and if anything else comes up!


Math does not update in display if you change TeX and press ENTER

Needs investigation. I think the data is there and saved, but no rerender (at least of the KaTeX part) is done.

Also, we when you have some text selected (property selection) we should use that as the TeX code instead of overriding with f(x). That way you can type sqrt(42), select it and turn it into math.

Rerender problem fixed via https://github.com/substance/texture/commit/ed30a74b9b8f6120d32fa0fdf3380ee54f7f2d7a

pls check #496


defmt-0.3.9 failed to build

Crate name
defmt
Build failure link
https://docs.rs/crate/defmt/0.3.9/builds/1566395
Additional details
I don't think we changed anything substantial since defmt-0.3.8, which built OK.

there wasn't any related change in docs.rs, though I can only imagine it's either related to your code, or perhaps that the new nightly version changed something.

Maybe -Zbuild-std is the one in fault here?

this is the diff: https://github.com/knurling-rs/defmt/compare/defmt-v0.3.8...defmt-v0.3.9

ah, I remember there is also diff.rs: https://diff.rs/defmt/0.3.8/defmt/0.3.9/Cargo.toml

Do you use -Zbuild-std? Because I don't think we do. And yes, on thumbv6m-none-eabi for example, I'm pretty sure you have to say -Zbuild-std=core
➜  defmt git:(release-defmt-0.3.10) rustc +nightly --version
rustc 1.83.0-nightly (06bb8364a 2024-10-01)
➜  defmt git:(release-defmt-0.3.10) cargo +nightly build --target=thumbv6m-none-eabi
   Compiling bitflags v1.3.2
   Compiling defmt v0.3.9 (/Users/jonathan/Documents/knurling-rs/defmt/defmt)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.23s
➜  defmt git:(release-defmt-0.3.10) cargo +nightly build --target=thumbv6m-none-eabi -Zbuild-std
   Compiling gimli v0.29.0
error[E0432]: unresolved import `alloc::sync`
 --> /Users/jonathan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/gimli-0.29.0/src/read/dwarf.rs:2:12
  |
2 | use alloc::sync::Arc;
  |            ^^^^ could not find `sync` in `alloc`

error[E0432]: unresolved import `alloc::sync`
 --> /Users/jonathan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/gimli-0.29.0/src/read/abbrev.rs:4:12
  |
4 | use alloc::sync::Arc;
  |            ^^^^ could not find `sync` in `alloc`

For more information about this error, try `rustc --explain E0432`.
error: could not compile `gimli` (lib) due to 2 previous errors
➜  defmt git:(release-defmt-0.3.10) cargo +nightly build --target=thumbv6m-none-eabi -Zbuild-std=core
   Compiling bitflags v1.3.2
   Compiling defmt v0.3.9 (/Users/jonathan/Documents/knurling-rs/defmt/defmt)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.25s
➜  defmt git:(release-defmt-0.3.10)

The error message appears to come from here: https://github.com/rust-lang/cargo/blob/81e0670dddefc0b78b272e58decc0eea9def3720/src/cargo/core/compiler/standard_lib.rs#L80

Which was added last week: https://github.com/rust-lang/cargo/commit/5e4a1db935cbc1e1bb24ab0e48f0ba0e9d79aeb8

Yup:
➜  defmt git:(release-defmt-0.3.10) RUSTC=~/Downloads/rust-nightly/rust-nightly-aarch64-apple-darwin/rustc/bin/rustc  ~/Downloads/rust-nightly/rust-nightly-aarch64-apple-darwin/cargo/bin/cargo build --target=thumbv6m-none-eabi -Zbuild-std
error: building std is not supported on the following targets: thumbv6m-none-eabi

➜  defmt git:(release-defmt-0.3.10) RUSTC=~/Downloads/rust-nightly/rust-nightly-aarch64-apple-darwin/rustc/bin/rustc  ~/Downloads/rust-nightly/rust-nightly-aarch64-apple-darwin/cargo/bin/cargo build --target=thumbv6m-none-eabi -Zbuild-std=core
error: "/Users/jonathan/Downloads/rust-nightly/rust-nightly-aarch64-apple-darwin/rustc/lib/rustlib/src/rust/library/Cargo.lock" does not exist, unable to build with the standard library, try:
        rustup component add rust-src

No idea how to add rust-src to a tarball installation, but that's the error.

I sent https://crates.io/crates/thejpster-test-docs-target down the line, which is an inoffensive test crate. If you can't build it then I'm pretty sure it's a docs.rs but and not a defmt bug.

Maybe -Zbuild-std is the one in fault here?

@GuillaumeGomez you mean we need add it do the docs.rs metadata to use https://github.com/rust-lang/docs.rs/pull/2107 ?

I sent https://crates.io/crates/thejpster-test-docs-target down the line, which is an inoffensive test crate. If you can't build it then I'm pretty sure it's a docs.rs but and not a defmt bug.

there was no docs.rs change that might cause this, so I imagine the nighty version changes you linked could cause the error.
If you have a test crate that errors too (and IMO it likely will with the targets) we might try it with cargo-args=["-Zbuild-std"]

AFAIK will need -Zbuild-std=core when building for a target that doesn't have libstd. I guess most of your crates build for the host machine, but the docs.rs metadata for this one forces you to build it for thumbv6m-none-eabi and thumbv7em-none-eabihf, because it's known to not work on Windows/macOS/Linux.

https://crates.io/crates/thejpster-test-docs-target is perpetually stuck at the back of the queue, perhaps because you've rated as a crate of low value (it is, it's terrible).

it's not deprioritized, so has the same prio as normal releases (deprioritized is shown in braces after the crate, if we do it, like (priority: -1))

So why is everything else being added to the queue in front of it?

I can't Check the logs right now, my guess is build-attempts.
For a certain type of build errors ( like the one from the original crate) we actually try the build up to 5 times with a delay in between.

Ah, it did build but it went back in the queue!!
and it failed: https://docs.rs/crate/thejpster-test-docs-target/0.1.0/builds/1567394

error: building std is not supported on the following targets: thumbv7em-none-eabihf, thumbv6m-none-eabi

So yup, not defmt's fault.

I want to create a bug-report for cargo, @jonathanpallant could you push the test crate to github so I can include it?

I wonder if cargo-args = ["-Zbuild-std=core"] in the metadata might help...

I started with a zulip thread

https://github.com/jonathanpallant/thejpster-test-docs-target

the final fix is still in progress, but since there were some other releases that needed a newer version I un-froze the nightly version.
Feel free to ping me if you need to build another release, and we can handle it together. It might also be worth trying to add Zbuild-std=core as done here

coming from the zulip discussion I would say this is fixed now.
Feel free to reopen if it breaks again, if it fails please first try adding -Zbuild-std=core or -Zbuild-std to the docs.rs metadata


Prefix based on hyphen prop(not camel case)

Is it possible to make inline-style-prefixer prefix based on hyphen prop instead of camelCase? By doing so, I dont have to convert users' input from hyphen to camel-case to hyphen again, thus enhancing the overall performance

Apparently, no. This library aims to prefix inline-style and/or CSS-in-JS style objects which are written in camelCase as that's what the browser APIs default to. In what use-case do you work with kebab-cased properties that need prefixes?
I guess you could check out the prefixer from stylis, which operates on kebab-cased properties instead.

Apparently, no. This library aims to prefix inline-style and/or CSS-in-JS style objects which are written in camelCase as that's what the browser APIs default to. In what use-case do you work with kebab-cased properties that need prefixes?
I guess you could check out the prefixer from stylis, which operates on kebab-cased properties instead.

Hi @robinweser thank you for your answer. For browser APIs, which one are you referring to? Currently I found camelCase is only used for setting inline styling for component, and I have to use kebab-case set css on a stylesheet with js, through insertRule().
https://developer.mozilla.org/en-US/docs/Web/API/CSSStyleSheet/insertRule
Ok I think the meaning of the name of this package again after I have written that, I guess I get your point now.

Hi @robinweser thank you for your answer. For browser APIs, which one are you referring to? Currently I found camelCase is only used for setting inline styling for component, and I have to use kebab-case set css on a stylesheet with js, through insertRule().
https://developer.mozilla.org/en-US/docs/Web/API/CSSStyleSheet/insertRule
Ok I think the meaning of the name of this package again after I have written that, I guess I get your point now.

So, this package was originally created for React inline styles which are written in camelCase. Then it got adapted by many CSS-in-JS libraries (especially those operating on objects) and all of those use camelCase properties as well, mainly because it's more readable and easy-to-type in JavaScript.
If you explicitly aim to prefix CSS strings like the ones you pass to insertRule this package might be the wrong solution. In this case the stylis prefixer should be great!

So, this package was originally created for React inline styles which are written in camelCase. Then it got adapted by many CSS-in-JS libraries (especially those operating on objects) and all of those use camelCase properties as well, mainly because it's more readable and easy-to-type in JavaScript.
If you explicitly aim to prefix CSS strings like the ones you pass to insertRule this package might be the wrong solution. In this case the stylis prefixer should be great!


packages.md describes how to use Cloudant package which does not exist in OpenWhisk

Please see issue 336 as this might resolve the issue; however it has been backlogged for 21 days as of this authoring.
The file docs/packages.md assumes that both cloudant and alarms packages are available (neither are).
Therefore steps:
2. Get a list of entities in the /whisk.system/cloudant package.
$ wsk package get --summary /whisk.system/cloudant
and:
3. Get a description of the /whisk.system/cloudant/read action.
$ wsk action get --summary /whisk.system/cloudant/read
of course, will never work for the user (if it wasn't apparent from step 1. which shows a package list that does not include these missing packages).
In addition, for open source, we should likely have couchdb be the default example (and provide cloudant as an alternative).  In any event, we need to be aware that the openwhisk user doc on packages is not usable until this the corresponding package is made available.
I am happy to open another issue to default to couchdb over cloudant if you like.

@csantanapr since we now have a cloudant package in the open can we address this high priority issue? otherwise remove the label?

cloudant repo and docs https://github.com/openwhisk/openwhisk-package-cloudant


Community section update

Changed the image, position of the section and a bit of the content

/lgtm

/lgtm

/approve


Shablona website broken?

I can't find it on RTD: https://readthedocs.org/projects/shablona/
since #32 is already an issue, maybe a good time to switch to gh-pages? I can make a PR setting up gh-pages to build with travis, then you could insert the uwescience github API in a subsequent PR?

I've been meaning to move to gh-pages for ages. Please do make a PR for that! Thank you!

Cool - see #64


perf: use faster statement type check

As discussed offline with @sokra, this change is meant to allow faster statement type lookup's. Based on https://jsperf.com/exact-pattern-vs-dot-star/1
What kind of change does this PR introduce?
perf
Did you add tests for your changes?
N/A
Does this PR introduce a breaking change?
No
What needs to be documented once your changes are merged?
N/A

Thanks


Wave equation demo

Added wave equation demo in /demos/. Also set the project expand mode to canvas_items.

Looks good, thanks!


POS PATTERN for italian language?

First of all @TimSchopf  I would like to thank you so much for this powerful and very easy to use, simple library. I have tested it with some english articles and every keyword returned makes absolutely sense. Yay!
However, I was trying to follow the guide to use it with other languages, which in my case, I want to see how the library works with the italian language. I got a little lost at the pos_pattern parameter, since I've been reading a lot of the spaCy documentation and I don't seem to find the specific pos pattern for the Italian language. I already have the italian spaCy model, which is it_core_news_lg, and a list of italian stopwords, however as I said before I'm a little stuck here because of the pos pattern for the italian language and I wanted to know if you could help me with this simple part😅.
Thanks in advance!

Hello guys, I already found the solution to own question. In order to use the spacy pipeline with the italian language you have to input the following pos_pattern: '<A.*>*<S.*>+'
For the vectorizer part, the one that I used and obtained good results, I used it like this:
vectorizer = KeyphraseCountVectorizer(spacy_pipeline='it_core_news_lg', pos_pattern='<A.*>*<S.*>+', stop_words=stop_words)
I even made a simple function if anyone wants to obtain a list of keywords:
# We initialize the model outside the function given the fact that it takes ∼ 5 seconds to load.
ita_kw_model = KeyBERT(model=SentenceTransformer('paraphrase-multilingual-mpnet-base-v2'))

def keybert_keyphrase_vectorizers_italian(kw_model, article, top_n=20, stop_words=ita_stopwords):
    """Performs Grammatically correct Keyword Extraction using KeyBERT with KeyphraseVectorizers using the spaCy italian pipeline.
    kw_model should be a KeyBERT model with a SentenceTransformer model that includes the italian language. ita_stopwords is
    my custom list of italian stopwords.
    """

    article = clean_from_html(article)
    vectorizer = KeyphraseCountVectorizer(spacy_pipeline='it_core_news_lg', pos_pattern='<A.*>*<S.*>+', stop_words=stop_words)
    keywords = kw_model.extract_keywords(docs=article, vectorizer=vectorizer, top_n=top_n)

    return keywords

And to use the function, just use the following code where sample_doc is your italian document.
keybert_keyphrase_vectorizers_italian(kw_model=ita_kw_model, article=sample_doc)

@spolo96 How did you figure out that?


How to delete shapes programmatically?

Is there a way to delete a shape programmatically? I've looked at the API, maybe not thoroughly enough, but is there a way for me to do this?

Same question.

I figured out how to do it (kind of) - you'll need to copy in the following code into your literallycanvas JavaScript file. Put it underneath LiterallyCanvas.prototype.clear:
LiterallyCanvas.prototype.rerender = function(newShapes) {
    var oldShapes;
    oldShapes = this.shapes;
    this.setShapesInProgress([]);
    this.execute(new actions.ClearAction(this, oldShapes, newShapes), false);
    this.repaintLayer('main');
  };

Then you'll just need to write something like:
lc.rerender(shapes);
Where shapes is a shape object containing all your shapes.


Update author order for the preprint

As suggested in #86

Should we add the shared first authorship with asterisk or something, I know it's just a preprint, but still ...

That would be good if you can find how to do it

Example here from one of the other BioHackathon projects
https://github.com/elixir-europe/biohackathon-projects-2023/blob/main/20/biohackrxiv/paper.md?plain=1#L129

Example here from one of the other BioHackathon projects
https://github.com/elixir-europe/biohackathon-projects-2023/blob/main/20/biohackrxiv/paper.md?plain=1#L129

Added something like this. Thanks @supernord


Add restart file coarsening regression tests

This PR adds some regression tests for coarse-graining restart files using the high-level interfaces provided in coarsen_restarts.py.  They already turned up one bug when using the blended approach, but not coarse-graining the A-grid winds, which I went ahead and fixed.  I tried to write things in a way such that these could facilitate refactoring the somewhat repetitive code in coarsen_restarts.py.
Timing information
Since being refactored to use synth these tests now take less than 1.5 seconds each.  They are now more on par with the current slowest tests in vcm (test_grid_builder).
Implementation note
Unfortunately we cannot use a traditional checksum based system, because mappm does not produce bit-for-bit identical results on my VM when compared with CircleCI, so I resorted to adding reference data to compare against in GCS.
Requirement changes

synth is now required to run the vcm tests.

[x] Tests added

Coverage reports (updated automatically):

test_unit: 84%

I'm concerned about the slowness of these tests. For comparison, the entire test suite for vcm (550 tests) is about 2 minutes in a recent CI test, and runs in as little as 15s on my mac. So this would be a very significant increase in test time for vcm.
Do you know what makes the test slow? Is it the interaction with GCS, or the actual coarsening? It seems like we should be able to write this test without relying on GCS.

It looks like I need to install synth in the CI environment.

Should add synth to vcm test requirements.

I know you did add it to the tox config, but for completeness probably good to also include in setup.py.

Should add synth to vcm test requirements.

Or to be as general as possible, you could skip these tests if synth is not installed. Just thinking from perspective of VCM being a useful standalone package.

Alright we should be good now (had a bad rebase there for a moment, but I reset things).  I added a module-level importorskip to the test_coarsen_restarts.py file so that those tests get skipped if synth is not installed, making synth fully optional.  I'm not sure if we want to add it in the tests_require section of setup.py or not (technically it's not really required for running the tests, just some of them, but I generally agree it's probably good to have it listed somewhere other than in the tox config).

Yeah, leaving it out of setup.py seems fine to me. Having the importorskip is good.

Sounds good.  I'll go ahead and merge.


DIFFERENCE function is not working

Hi,
It seems there is a problem with DIFFERENCE function: example doesn't work well here and I can't use it well in my warpscript.
KannarFR.

// Converts the two LISTS into sets
[ 'foo' 'bar' ] ->SET
[ 'foo' 'bar' 'bob' ] ->SET
// Compute the difference
DIFFERENCE
// convert back the SET into LIST
SET->

It seems swap list make it works, strange behavior.
// Converts the two LISTS into sets
[ 'foo' 'bar'  'bob' ] ->SET
[ 'foo' 'bar' ] ->SET
// Compute the difference
DIFFERENCE
// convert back the SET into LIST
SET->

Is working!

Oh I get the point it's normal... Sorry.

I tihnk the example is confusing, you should swap lines to illustrate your point. WDYT?

A bug was recently corrected in DIFFERENCE see #275

The doc has been updated, thanks for the head up.