running on other OS than Linux

@constantinpape @martinschorb
I did not manage to run the conversion to OME-Zarr on my Mac (see MoBIE mattermost channel).
This raises a general question: on which OS does the code here run?

In principle everything should run on Mac and Windows and there are unit tests for it that run through without issues.
https://github.com/mobie/mobie-utils-python/actions/runs/7186625802/job/19572434858
Maybe these don't cover all the cases for ome-zarr conversion?! To fix this I would need a minimal self-contained reproducible example where it fails.

Hi! I'm facing a very similar issue right now where code works as a .py on linux but not on macOS or windows. However, the code works in mac and windows when run as .ipynb file. It seems to get stuck on the downscaling when run as a .py file containing the exact same code. Potentially jupyter notebook is adding a layer of virtualisation?
The exact error traceback:
`/Users/najmie/miniconda3/envs/mobie3/bin/python /Users/najmie/Downloads/mobie_python_project/project_creation.py
najmie@CH7H1X6L5F mobie_python_project % /Users/najmie/miniconda3/envs/mobie3/bin/python /Users/najmie/Downloads/mobie_python_project/project_creation.py
Only one channel found in data/em_20nm_z_40_145.tif - no conversion needed :)
/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/luigi/parameter.py:296: UserWarning: Parameter "dtype" with value "None" is not of type string.
warnings.warn('Parameter "{}" with value "{}" is not of type string.'.format(param_name, param_value))
DEBUG: Checking if DownscalingWorkflow(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, target=local, dependency=DummyTask, input_path=data/em_20nm_z_40_145.tif, input_key=, scale_factors=[[1, 2, 2], [1, 2, 2], [1, 2, 2], [1, 2, 2]], halos=[[1, 2, 2], [1, 2, 2], [1, 2, 2], [1, 2, 2]], dtype=None, int_to_uint=False, metadata_format=ome.zarr, metadata_dict={"resolution": [10.0, 10.0, 10.0], "unit": "nanometer", "setup_name": "em_20nm_z_40_145"}, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key_prefix=, force_copy=False, skip_existing_levels=False, scale_offset=0) is complete
/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/luigi/parameter.py:296: UserWarning: Parameter "scale_factor" with value "(1, 2, 2)" is not of type string.
warnings.warn('Parameter "{}" with value "{}" is not of type string.'.format(param_name, param_value))
DEBUG: Checking if WriteDownscalingMetadata(tmp_folder=tmp_dataset1_em_20nm_z_40_145, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, scale_factors=[[1, 2, 2], [1, 2, 2], [1, 2, 2], [1, 2, 2]], dependency=DownscalingLocal, metadata_format=ome.zarr, metadata_dict={"resolution": [10.0, 10.0, 10.0], "unit": "nanometer", "setup_name": "em_20nm_z_40_145"}, output_key_prefix=, scale_offset=0, prefix=downscaling) is complete
INFO: Informed scheduler that task   DownscalingWorkflow_tmp_dataset1_em__DummyTask_None_f01a3395a1   has status   PENDING
DEBUG: Checking if DownscalingLocal(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, input_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, input_key=s3, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key=s4, scale_factor=(1, 2, 2), scale_prefix=s4, halo=[1, 2, 2], effective_scale_factor=[1, 16, 16], dimension_separator=/, dependency=DownscalingLocal) is complete
INFO: Informed scheduler that task   WriteDownscalingMetadata_DownscalingLocal___resolution_____ome_zarr_784397d098   has status   PENDING
DEBUG: Checking if DownscalingLocal(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, input_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, input_key=s2, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key=s3, scale_factor=(1, 2, 2), scale_prefix=s3, halo=[1, 2, 2], effective_scale_factor=[1, 8, 8], dimension_separator=/, dependency=DownscalingLocal) is complete
INFO: Informed scheduler that task   DownscalingLocal_tmp_dataset1_em__DownscalingLocal___d657c21b11   has status   PENDING
DEBUG: Checking if DownscalingLocal(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, input_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, input_key=s1, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key=s2, scale_factor=(1, 2, 2), scale_prefix=s2, halo=[1, 2, 2], effective_scale_factor=[1, 4, 4], dimension_separator=/, dependency=DownscalingLocal) is complete
INFO: Informed scheduler that task   DownscalingLocal_tmp_dataset1_em__DownscalingLocal___d82adb898c   has status   PENDING
DEBUG: Checking if DownscalingLocal(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, input_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, input_key=s0, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key=s1, scale_factor=(1, 2, 2), scale_prefix=s1, halo=[1, 2, 2], effective_scale_factor=[1, 2, 2], dimension_separator=/, dependency=CopyVolumeLocal) is complete
INFO: Informed scheduler that task   DownscalingLocal_tmp_dataset1_em__DownscalingLocal___7fa9471af6   has status   PENDING
DEBUG: Checking if CopyVolumeLocal(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, input_path=data/em_20nm_z_40_145.tif, input_key=, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key=s0, prefix=initial_scale, dtype=None, int_to_uint=False, fit_to_roi=False, effective_scale_factor=[], dimension_separator=/, dependency=DummyTask) is complete
INFO: Informed scheduler that task   DownscalingLocal_tmp_dataset1_em__CopyVolumeLocal___452731ba01   has status   PENDING
DEBUG: Checking if DummyTask() is complete
INFO: Informed scheduler that task   CopyVolumeLocal_tmp_dataset1_em__DummyTask___07bedbbf06   has status   PENDING
INFO: Informed scheduler that task   DummyTask__99914b932b   has status   DONE
INFO: Done scheduling tasks
INFO: Running Worker with 1 processes
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 7
INFO: [pid 11765] Worker Worker(salt=8065394207, workers=1, host=CH7H1X6L5F, username=najmie, pid=11765) running   CopyVolumeLocal(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, input_path=data/em_20nm_z_40_145.tif, input_key=, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key=s0, prefix=initial_scale, dtype=None, int_to_uint=False, fit_to_roi=False, effective_scale_factor=[], dimension_separator=/, dependency=DummyTask)
Only one channel found in data/em_20nm_z_40_145.tif - no conversion needed :)
/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/luigi/parameter.py:296: UserWarning: Parameter "dtype" with value "None" is not of type string.
warnings.warn('Parameter "{}" with value "{}" is not of type string.'.format(param_name, param_value))
DEBUG: Checking if DownscalingWorkflow(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, target=local, dependency=DummyTask, input_path=data/em_20nm_z_40_145.tif, input_key=, scale_factors=[[1, 2, 2], [1, 2, 2], [1, 2, 2], [1, 2, 2]], halos=[[1, 2, 2], [1, 2, 2], [1, 2, 2], [1, 2, 2]], dtype=None, int_to_uint=False, metadata_format=ome.zarr, metadata_dict={"resolution": [10.0, 10.0, 10.0], "unit": "nanometer", "setup_name": "em_20nm_z_40_145"}, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key_prefix=, force_copy=False, skip_existing_levels=False, scale_offset=0) is complete
/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/luigi/parameter.py:296: UserWarning: Parameter "scale_factor" with value "(1, 2, 2)" is not of type string.
warnings.warn('Parameter "{}" with value "{}" is not of type string.'.format(param_name, param_value))
DEBUG: Checking if WriteDownscalingMetadata(tmp_folder=tmp_dataset1_em_20nm_z_40_145, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, scale_factors=[[1, 2, 2], [1, 2, 2], [1, 2, 2], [1, 2, 2]], dependency=DownscalingLocal, metadata_format=ome.zarr, metadata_dict={"resolution": [10.0, 10.0, 10.0], "unit": "nanometer", "setup_name": "em_20nm_z_40_145"}, output_key_prefix=, scale_offset=0, prefix=downscaling) is complete
INFO: Informed scheduler that task   DownscalingWorkflow_tmp_dataset1_em__DummyTask_None_f01a3395a1   has status   PENDING
DEBUG: Checking if DownscalingLocal(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, input_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, input_key=s3, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key=s4, scale_factor=(1, 2, 2), scale_prefix=s4, halo=[1, 2, 2], effective_scale_factor=[1, 16, 16], dimension_separator=/, dependency=DownscalingLocal) is complete
INFO: Informed scheduler that task   WriteDownscalingMetadata_DownscalingLocal___resolution_____ome_zarr_784397d098   has status   PENDING
DEBUG: Checking if DownscalingLocal(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, input_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, input_key=s2, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key=s3, scale_factor=(1, 2, 2), scale_prefix=s3, halo=[1, 2, 2], effective_scale_factor=[1, 8, 8], dimension_separator=/, dependency=DownscalingLocal) is complete
INFO: Informed scheduler that task   DownscalingLocal_tmp_dataset1_em__DownscalingLocal___d657c21b11   has status   PENDING
DEBUG: Checking if DownscalingLocal(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, input_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, input_key=s1, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key=s2, scale_factor=(1, 2, 2), scale_prefix=s2, halo=[1, 2, 2], effective_scale_factor=[1, 4, 4], dimension_separator=/, dependency=DownscalingLocal) is complete
INFO: Informed scheduler that task   DownscalingLocal_tmp_dataset1_em__DownscalingLocal___d82adb898c   has status   PENDING
DEBUG: Checking if DownscalingLocal(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, input_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, input_key=s0, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key=s1, scale_factor=(1, 2, 2), scale_prefix=s1, halo=[1, 2, 2], effective_scale_factor=[1, 2, 2], dimension_separator=/, dependency=CopyVolumeLocal) is complete
INFO: Informed scheduler that task   DownscalingLocal_tmp_dataset1_em__DownscalingLocal___7fa9471af6   has status   PENDING
DEBUG: Checking if CopyVolumeLocal(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, input_path=data/em_20nm_z_40_145.tif, input_key=, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key=s0, prefix=initial_scale, dtype=None, int_to_uint=False, fit_to_roi=False, effective_scale_factor=[], dimension_separator=/, dependency=DummyTask) is complete
INFO: Informed scheduler that task   DownscalingLocal_tmp_dataset1_em__CopyVolumeLocal___452731ba01   has status   PENDING
INFO: Informed scheduler that task   CopyVolumeLocal_tmp_dataset1_em__DummyTask___07bedbbf06   has status   DONE
INFO: Done scheduling tasks
INFO: Running Worker with 1 processes
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 6
INFO: [pid 11783] Worker Worker(salt=2341089979, workers=1, host=CH7H1X6L5F, username=najmie, pid=11783) running   DownscalingLocal(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, input_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, input_key=s0, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key=s1, scale_factor=(1, 2, 2), scale_prefix=s1, halo=[1, 2, 2], effective_scale_factor=[1, 2, 2], dimension_separator=/, dependency=CopyVolumeLocal)
ERROR: [pid 11783] Worker Worker(salt=2341089979, workers=1, host=CH7H1X6L5F, username=najmie, pid=11783) failed    DownscalingLocal(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, input_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, input_key=s0, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key=s1, scale_factor=(1, 2, 2), scale_prefix=s1, halo=[1, 2, 2], effective_scale_factor=[1, 2, 2], dimension_separator=/, dependency=CopyVolumeLocal)
Traceback (most recent call last):
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/luigi/worker.py", line 210, in run
new_deps = self._run_get_new_deps()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/luigi/worker.py", line 138, in _run_get_new_deps
task_gen = self.task.run()
^^^^^^^^^^^^^^^
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/cluster_tools/cluster_tasks.py", line 95, in run
raise e
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/cluster_tools/cluster_tasks.py", line 81, in run
self.run_impl()
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/cluster_tools/downscaling/downscaling.py", line 184, in run_impl
self.submit_jobs(n_jobs, self.scale_prefix)
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/cluster_tools/cluster_tasks.py", line 567, in submit_jobs
tasks = [pp.submit(self._submit, job_id, job_prefix) for job_id in range(n_jobs)]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/cluster_tools/cluster_tasks.py", line 567, in
tasks = [pp.submit(self._submit, job_id, job_prefix) for job_id in range(n_jobs)]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/concurrent/futures/process.py", line 808, in submit
self._adjust_process_count()
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/concurrent/futures/process.py", line 767, in _adjust_process_count
self._spawn_process()
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/concurrent/futures/process.py", line 785, in _spawn_process
p.start()
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
^^^^^^^^^^^^^^^^^
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/multiprocessing/context.py", line 288, in _Popen
return Popen(process_obj)
^^^^^^^^^^^^^^^^^^
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 32, in init
super().init(process_obj)
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/multiprocessing/popen_fork.py", line 19, in init
self._launch(process_obj)
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 42, in _launch
prep_data = spawn.get_preparation_data(process_obj._name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/multiprocessing/spawn.py", line 164, in get_preparation_data
_check_not_importing_main()
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/multiprocessing/spawn.py", line 140, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

    To fix this issue, refer to the "Safe importing of main module"
    section in https://docs.python.org/3/library/multiprocessing.html

DEBUG: 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   DownscalingLocal_tmp_dataset1_em__CopyVolumeLocal___452731ba01   has status   FAILED
DEBUG: Asking scheduler for work...
DEBUG: Done
DEBUG: There are no more tasks to run at this time
DEBUG: There are 6 pending tasks possibly being run by other workers
DEBUG: There are 6 pending tasks unique to this worker
DEBUG: There are 6 pending tasks last scheduled by this worker
INFO: Worker Worker(salt=2341089979, workers=1, host=CH7H1X6L5F, username=najmie, pid=11783) was stopped. Shutting down Keep-Alive thread
INFO:
===== Luigi Execution Summary =====
Scheduled 7 tasks of which:

1 complete ones were encountered:

1 CopyVolumeLocal(...)

1 failed:

1 DownscalingLocal(...)

5 were left pending, among these:

5 had failed dependencies:

3 DownscalingLocal(...)
1 DownscalingWorkflow(...)
1 WriteDownscalingMetadata(...)

This progress looks :( because there were failed tasks
===== Luigi Execution Summary =====
Could not add em_20nm_z_40_145 to project :(
ERROR: [pid 11765] Worker Worker(salt=8065394207, workers=1, host=CH7H1X6L5F, username=najmie, pid=11765) failed    CopyVolumeLocal(tmp_folder=tmp_dataset1_em_20nm_z_40_145, max_jobs=1, config_dir=tmp_dataset1_em_20nm_z_40_145/configs, input_path=data/em_20nm_z_40_145.tif, input_key=, output_path=test_project/data/dataset1/images/ome-zarr/em_20nm_z_40_145.ome.zarr, output_key=s0, prefix=initial_scale, dtype=None, int_to_uint=False, fit_to_roi=False, effective_scale_factor=[], dimension_separator=/, dependency=DummyTask)
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/concurrent/futures/process.py", line 261, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/cluster_tools/cluster_tasks.py", line 547, in _submit
assert os.path.exists(script_path), script_path
AssertionError: tmp_dataset1_em_20nm_z_40_145/copy_volume.py
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/cluster_tools/cluster_tasks.py", line 81, in run
self.run_impl()
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/cluster_tools/copy_volume/copy_volume.py", line 163, in run_impl
self.submit_jobs(n_jobs, self.prefix)
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/cluster_tools/cluster_tasks.py", line 568, in submit_jobs
[t.result() for t in tasks]
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/cluster_tools/cluster_tasks.py", line 568, in
[t.result() for t in tasks]
^^^^^^^^^^
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
AssertionError: tmp_dataset1_em_20nm_z_40_145/copy_volume.py
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/luigi/worker.py", line 210, in run
new_deps = self._run_get_new_deps()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/luigi/worker.py", line 138, in _run_get_new_deps
task_gen = self.task.run()
^^^^^^^^^^^^^^^
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/cluster_tools/cluster_tasks.py", line 92, in run
self._write_log("task failed in run_impl with %s" % msg)
File "/Users/najmie/miniconda3/envs/mobie3/lib/python3.11/site-packages/cluster_tools/cluster_tasks.py", line 281, in _write_log
with open(log_file, 'a') as f:
^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'tmp_dataset1_em_20nm_z_40_145/copy_volume_initial_scale.log'
DEBUG: 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   CopyVolumeLocal_tmp_dataset1_em__DummyTask___07bedbbf06   has status   FAILED
DEBUG: Asking scheduler for work...
DEBUG: Done
DEBUG: There are no more tasks to run at this time
DEBUG: There are 7 pending tasks possibly being run by other workers
DEBUG: There are 7 pending tasks unique to this worker
DEBUG: There are 7 pending tasks last scheduled by this worker
INFO: Worker Worker(salt=8065394207, workers=1, host=CH7H1X6L5F, username=najmie, pid=11765) was stopped. Shutting down Keep-Alive thread
INFO:
===== Luigi Execution Summary =====
Scheduled 8 tasks of which:

1 complete ones were encountered:

1 DummyTask()

1 failed:

1 CopyVolumeLocal(...)

6 were left pending, among these:

6 had failed dependencies:

4 DownscalingLocal(...)
1 DownscalingWorkflow(...)
1 WriteDownscalingMetadata(...)

This progress looks :( because there were failed tasks
===== Luigi Execution Summary =====
Could not add em_20nm_z_40_145 to project :( `
The code being run is in this repo: https://github.com/eschalnajmi/load_mobie_project
If anyone has figured out a way to fix this easily I would very very much appreciate it :)


Build with JDK 17 instead of 13

JDK 13 was non-lts release and has been deprecated years ago.
Current LTS release is 17 so it makes sense to build with 17 instead of 13.
Also maybe drop 9 as it was also non-lts version.

If you agree I can add removal of v9. It will makes builds faster as they won't wait for runners that are being occupied by v9 builds


TABLE EXTRACTOR

Description of the new feature / enhancement
like your text extractor, this would be specific to data in a table. where you can snippet the table from an image or pdf and then be able to paste in a spreadsheet. a key feature would be a prompt to allow user to define the data that was copied if its text, numbers, or symbols. and in case of the numbers, is it decimal or is it fractions, because most conversions have a hard time distinguishing 'O' from 0 or 'l" from 1.
Scenario when this would be used?
i just got a 5 page pdf of a table with data, call it a 150 rows. there are 2 only headers with information I really need, call it length and width.  instead of going through excels table extractor, which is doesn't really get me what I want unless I finesse the results, i should be able to take snippets of the data paste and then a prompt asking me "what am i pasting" and/or "how would you like to paste"
Supporting information
No response

Text Extractor is not meant to be a complete data cleaning product. My other tool Text Grab is built with more features like what you are describing. Adding all of those features to Text Extractor overload the simple nature of the tool today.


fix: better error handling in account_info RPC

High Level Overview of Change
This PR adds a check for whether the account and ident params in several account-based RPCs are strings. If not, it returns invalidParams.
Note: This PR looks a lot larger than it is; it is really just the same change(s) applied over and over again across several RPCs.
Context of Change
rippled currently returns Internal error in this scenario.
The error in the logs is Caught throw: Type is not convertible to string
Type of Change

[x] Bug fix (non-breaking change which fixes an issue)

API Impact

[ ] Public API: New feature (new methods and/or new fields)
[x] Public API: Breaking change (in general, breaking changes should only impact the next api_version)
[ ] libxrpl change (any change that may affect libxrpl or dependents of libxrpl)
[ ] Peer protocol change (must be backward compatible or bump the peer protocol version)

This is a bugfix, not a breaking change, but there isn't an option for that.
Test Plan
Added unit tests that fail before this patch and now pass.
Future Tasks
Audit other RPCs for this sort of issue.

This change could also be done by editing parseBase58 to return an empty std::optional instead of crashing (which would result in an rpcACT_MALFORMED instead of rpcINVALID_PARAMS, which I believe differs from what Clio returns in this case).

Closing in favor of #5078


Should allow blank headers of depth greater than 1

https://github.github.com/gfm/#example-49
> JSON.stringify(window.RDMD.mdast("## "), null, 2)
"{
  "type": "root",
  "children": [
    {
      "type": "heading",
      "depth": 1,
      "children": []
    }
  ]
}"

This now works.


[publish] Blog on JDBCToJDBC

There is already a blog for this - https://medium.com/google-cloud/migrating-data-from-one-databases-into-another-via-jdbc-using-dataproc-serverless-c5336c409b18


Ignore CPU & RAM for machine allocation

This fixes the issue where tasks that have CPU+RAM requirements can't get scheduled even in the single-node gpu setup, because we don't currently find the number of CPUs/amount of RAM for the primary vm host or the secondary vm host nodes. These resource requirements are only passed to the docker daemon which keeps track of how much of those resources are available.
Additionally this adds some code to throw errors in undesirable cases of non-GPU workloads getting allocated to GPU machines (this seemed to happen, but I don't know how it could have :/ ).
Despite the name of the branch, I didn't end up pursuing my initial thought of making the primary vm-host be implicit (not stored in the DB) since it would mean I'd have to drop the foreign key constraint on workload machines, and that seems pretty silly. Plus I still don't really see how the initialization code could have caused allocations to fail in the way that they did in prod...
Watch out:
n/a
Documentation:
n/a yet
Testing:

covered by automated tests

Additionally this adds some code to throw errors in undesirable cases of non-GPU workloads getting allocated to GPU machines (this seemed to happen, but I don't know how it could have :/ ).

Works for me except for this. I need to think further if this is going to be an issue. Depends on how long until we can switch to this completely.

Actually, this is still not working correctly, unless I have something misconfigured. Server deployed on paperspace gets:
TRPCError: No machine available for task (error: Error: Not implemented in NoopCloud (called with [{"quantity":2,"kind":"gpu","subkind":"h100"}]))

It does seem like non-GPU workloads could get allocated to the GPU-having primary VM host, in the existing Voltage Park + Docker Compose setup. Does that sound right? If so, is this just not a problem because the Voltage Park + Docker Compose setup doesn't use this code path? Or is it a problem?

I don't think we should be restricting non-GPU workloads from "GPU-having primary VM host, in the existing Voltage Park + Docker Compose setup". We know what tasks we're running, some of them don't need GPUs and that should be fine.


hubble/relay: fix unavailable nodes count on ServerStatus

The number of listed unavailable nodes is bounded. However, the count of
unavailable nodes definitely should not be...
Fix: cefe1458010dd82a166c8d3e15761bc541bc944d

test-me-please

Coverage increased (+0.03%) to 37.208% when pulling 31fce064b7f30d3a97346b23fb737a8656858940 on pr/rolinh/hubble-relay-fix-unavailable-nodes-report into cefe1458010dd82a166c8d3e15761bc541bc944d on master.


Fixing vulnerabilities for issue #55

This pull fixes https://github.com/auth0-samples/auth0-angularjs-samples/issues/55

@mikearias3  You're doing a major version upgrade. Did you check the breaking changes for those versions?

Yeah, I just checked a few websites and there's a lot of breaking changes in the class names, so just upgrading the bootstrap version will certainly break everything inside the app 😬


QueryDSL not compatible with Spring Boot 3.1 / Hibernate 6.2

Observed vs. expected behavior
When using QueryDSL 5.0.0 with Spring Boot 3.1 which includes Hibernate 6.2, the following stack trace is observed:
java.lang.NoClassDefFoundError: Could not initialize class com.querydsl.jpa.hibernate.HibernateUtil
	at com.querydsl.jpa.hibernate.AbstractHibernateQuery.createQuery(AbstractHibernateQuery.java:105)
	at com.querydsl.jpa.hibernate.AbstractHibernateQuery.createQuery(AbstractHibernateQuery.java:97)
	at com.querydsl.jpa.hibernate.AbstractHibernateQuery.fetch(AbstractHibernateQuery.java:185)
        ...
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.NoClassDefFoundError: org/hibernate/type/ByteType [in thread "ForkJoinPool-1-worker-1"]
	at com.querydsl.jpa.hibernate.HibernateUtil.<clinit>(HibernateUtil.java:44)
	...

Steps to reproduce
Any querydsl JPA invocation that creates a query seems to trigger this.
Environment

Querydsl version: 5.0.0
Querydsl module: querydsl-jpa-jakarta
Database: Postgres
JDK: 17

At the moment I am still using Spring Boot 3.0, but I had success with this fork: https://github.com/marok39/querydsl

me too.

@cmuchinsky  It seems weird. I'm already using Querydsl with Spring Boot 3.1 and Hibernate 6.2 -> see my repo https://github.com/arnosthavelka/spring-advanced-training/tree/develop/sat-jpa. I guess I've seen this issue when I was migrating to Spring Boot 3. You can check my article https://dzone.com/articles/upgrade-guide-to-spring-boot-3-for-spring-data-jpa-3-and-querydsl-5. Have you switched to Jakarta? I guess it's not the issue, but it's a good start 😉.


Release & Debug version of the app asking keychain permissions

Related to https://github.com/firebase/firebase-ios-sdk/issues/5540
When launching the app customers are reporting that they are asking for sensitive key-chain access.
We had to include Firebase Core in order to get FCM working..
Anyway around this? It's not acceptable to ask 8 times for Keychain access everytime the customer launches and many refuse to use "Always allow" which is completely reasonable.

@paulb777 disabling the library stops with Platform.isMac stops it asking for keychain access.

One more thought since the issue sounds similar to a development-only workflow we've seen for macOS.  How is the app distributed? We would expect App Store signing to obviate the need for any keychain permission popups.

@ollydixon
If I understand correctly, the keychain permission is being asked for macOS after adding firebase_core / firebase_messaging plugin ?

@darshankawar indeed. Multiple times on every launch. Something that is no acceptable for a user journey.

@darshankawar sorry for the delay. We removed firebase in this project and are using an alternative push notifications system because of this bug.
I’m sure the sample app will give the same bug for MacOS :-)

@ollydixon Incase you have more info for us regarding how to reproduce the issue, feel free to write back.

@maheshmnj I've given enough info for you to replicate. Just use the sample app and run it in MacOS.

@ollydixon
I ran the release version of flutter counter app targeting macOS with the following dependencies
 firebase_core: ^1.17.1
 firebase_messaging: ^11.4.1

logs
mahesh@Maheshs-MacBook-Air-M1 sample % flutter run -d macos --release
Launching lib/main.dart on macOS in release mode...
Running pod install...                                           1,332ms
objc[16507]: Class AMSupportURLConnectionDelegate is implemented in both /usr/lib/libamsupport.dylib (0x1e4faf640) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x1086802c8). One of the two will be used. Which one is undefined.
objc[16507]: Class AMSupportURLSession is implemented in both /usr/lib/libamsupport.dylib (0x1e4faf690) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x108680318). One of the two will be used. Which one is undefined.
objc[16507]: Class AppleTypeCRetimerRestoreInfoHelper is implemented in both /usr/lib/libauthinstall.dylib (0x1e4faf458) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x1086804f8). One of the two will be used. Which one is undefined.
objc[16507]: Class AppleTypeCRetimerFirmwareAggregateRequestCreator is implemented in both /usr/lib/libauthinstall.dylib (0x1e4faf4a8) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x108680548). One of the two will be used. Which one is undefined.
objc[16507]: Class AppleTypeCRetimerFirmwareRequestCreator is implemented in both /usr/lib/libauthinstall.dylib (0x1e4faf4f8) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x108680598). One of the two will be used. Which one is undefined.
objc[16507]: Class ATCRTRestoreInfoFTABFile is implemented in both /usr/lib/libauthinstall.dylib (0x1e4faf548) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x1086805e8). One of the two will be used. Which one is undefined.
objc[16507]: Class AppleTypeCRetimerFirmwareCopier is implemented in both /usr/lib/libauthinstall.dylib (0x1e4faf598) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x108680638). One of the two will be used. Which one is undefined.
objc[16507]: Class ATCRTRestoreInfoFTABSubfile is implemented in both /usr/lib/libauthinstall.dylib (0x1e4faf5e8) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x108680688). One of the two will be used. Which one is undefined.
--- xcodebuild: WARNING: Using the first of multiple matching destinations:
{ platform:macOS, arch:arm64, id:00008103-0019553C36F2001E }
{ platform:macOS, arch:x86_64, id:00008103-0019553C36F2001E }
Building macOS application...

Flutter run key commands.
h List all available interactive commands.
c Clear the screen
q Quit (terminate the application on the device).
2022-06-13 10:40:16.870 sample[17552:136674] The operation couldn’t be completed. (OSStatus error 13.)

But I don't see app asking for keychain permissions.
Let me know if I am missing anything.
Thanks

@maheshmnj was their an update?
Anyhow, you have probably previously accepted all. It remembers forever. Go to the keychain and clear any references to Google, flutter (there’s many other names but I can’t remember).
I was getting this permission on two laptops and the QA personelle are reporting it as a bug.

@maheshmnj You need to connect to the Firebase service (your own) to fully run the test as you have not even initialized the package it will do nothing.
await Firebase.initializeApp(
      options: DefaultFirebaseOptions.currentPlatform,
    );

^^ Will trigger the keychain requests on MacOS.

Thanks for the info @ollydixon, I ran the below cloud_firestore  sample code in release mode targeting macOS, And I was able to reproduce the issue, The keychain permission prompts were shown at least 7-8 times.
https://user-images.githubusercontent.com/31410839/173536225-494aa506-a933-49c2-a7da-c8489da47e9c.mp4
I am quite not sure if this from flutter, firebase plugins/sdk , Labeling this issue for further insights from the team.

logs
mahesh@Maheshs-MacBook-Air-M1 cloud_firestore_example % flutter run -d macos --release
Launching lib/main.dart on macOS in release mode...
Running pod install...                                             12.7s
objc[98739]: Class AMSupportURLConnectionDelegate is implemented in both /usr/lib/libamsupport.dylib (0x1fd99f640) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x1082802c8). One of the two will be used. Which one is undefined.
objc[98739]: Class AMSupportURLSession is implemented in both /usr/lib/libamsupport.dylib (0x1fd99f690) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x108280318). One of the two will be used. Which one is undefined.
objc[98739]: Class AppleTypeCRetimerRestoreInfoHelper is implemented in both /usr/lib/libauthinstall.dylib (0x1fd99f458) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x1082804f8). One of the two will be used. Which one is undefined.
objc[98739]: Class AppleTypeCRetimerFirmwareAggregateRequestCreator is implemented in both /usr/lib/libauthinstall.dylib (0x1fd99f4a8) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x108280548). One of the two will be used. Which one is undefined.
objc[98739]: Class AppleTypeCRetimerFirmwareRequestCreator is implemented in both /usr/lib/libauthinstall.dylib (0x1fd99f4f8) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x108280598). One of the two will be used. Which one is undefined.
objc[98739]: Class ATCRTRestoreInfoFTABFile is implemented in both /usr/lib/libauthinstall.dylib (0x1fd99f548) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x1082805e8). One of the two will be used. Which one is undefined.
objc[98739]: Class AppleTypeCRetimerFirmwareCopier is implemented in both /usr/lib/libauthinstall.dylib (0x1fd99f598) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x108280638). One of the two will be used. Which one is undefined.
objc[98739]: Class ATCRTRestoreInfoFTABSubfile is implemented in both /usr/lib/libauthinstall.dylib (0x1fd99f5e8) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x108280688). One of the two will be used. Which one is undefined.
--- xcodebuild: WARNING: Using the first of multiple matching destinations:
{ platform:macOS, arch:arm64, id:00008103-0019553C36F2001E }
{ platform:macOS, arch:x86_64, id:00008103-0019553C36F2001E }
/Users/mahesh/Desktop/triage-examples/flutterfire_examples/cloud_firestore_example/macos/Pods/Pods.xcodeproj: warning: The macOS deployment target 'MACOSX_DEPLOYMENT_TARGET' is set to 10.7, but the range of supported deployment target versions is 10.9 to 12.2. (in target 'leveldb-library' from project 'Pods')
Building macOS application...

Flutter run key commands.
h List all available interactive commands.
c Clear the screen
q Quit (terminate the application on the device).
2022-06-14 14:16:20.683 cloud_firestore_example[7449:457108] The operation couldn’t be completed. (OSStatus error 13.)

code sample
// Copyright 2020, the Chromium project authors.  Please see the AUTHORS file
// for details. All rights reserved. Use of this source code is governed by a
// BSD-style license that can be found in the LICENSE file.

import 'dart:math';
import 'package:flutter/material.dart';
import 'package:firebase_core/firebase_core.dart';
import 'package:cloud_firestore/cloud_firestore.dart';

/// Requires that a Firestore emulator is running locally.
/// See https://firebase.flutter.dev/docs/firestore/usage#emulator-usage
bool USE_FIRESTORE_EMULATOR = false;

Future<void> main() async {
  WidgetsFlutterBinding.ensureInitialized();
  await Firebase.initializeApp();
  if (USE_FIRESTORE_EMULATOR) {
    FirebaseFirestore.instance.useFirestoreEmulator('localhost', 8080);
  }
  runApp(MyApp());
}

final messagesCollection = FirebaseFirestore.instance.collection('messages');

class MessagesList extends StatefulWidget {
  const MessagesList({Key? key}) : super(key: key);

  @override
  _MessagesListState createState() => _MessagesListState();
}

class _MessagesListState extends State<MessagesList> {
  @override
  Widget build(BuildContext context) {
    final query =
        messagesCollection.where('message', isEqualTo: 'Hello World').orderBy(
              'sentDateTime',
              descending: true,
            );
    return Scaffold(
      floatingActionButton: FloatingActionButton(
        onPressed: () {
          final minutes = Random().nextInt(60);
          final DateTime dateTime =
              DateTime.now().add(Duration(minutes: minutes));
          messagesCollection.add({
            'message': 'Hello World',
            'sentDateTime': Timestamp.fromDate(dateTime)
          });
        },
        child: Icon(Icons.add),
      ),
      appBar: AppBar(
        title: const Text('Firestore Messages'),
      ),
      body: StreamBuilder<QuerySnapshot>(
        stream: query.snapshots(),
        builder: (context, snapshot) {
          if (snapshot.hasError) {
            return Center(
              child: Text(snapshot.error.toString()),
            );
          }

          if (!snapshot.hasData) {
            return const Center(child: CircularProgressIndicator());
          }

          final data = snapshot.data!.docs;

          return ListView.builder(
            itemCount: data.length,
            itemBuilder: (context, index) {
              final dateTime =
                  (data[index]['sentDateTime'] as Timestamp).toDate();
              return ListTile(
                title: Text('${data[index]['message']}'),
                trailing: Text(dateTime.toString()),
              );
            },
          );
        },
      ),
    );
  }
}

class MyApp extends StatelessWidget {
  const MyApp({Key? key}) : super(key: key);
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Demo',
      theme: ThemeData(
        primarySwatch: Colors.blue,
      ),
      home: const MessagesList(),
    );
  }
}

flutter doctor -v (mac)
[✓] Flutter (Channel stable, 3.0.2, on macOS 12.4 21F79 darwin-arm, locale en-IN)
    • Flutter version 3.0.2 at /Users/mahesh/Documents/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision cd41fdd495 (5 days ago), 2022-06-08 09:52:13 -0700
    • Engine revision f15f824b57
    • Dart version 2.17.3
    • DevTools version 2.12.2

[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.0-rc4)
    • Android SDK at /Users/mahesh/Library/Android/sdk
    • Platform android-32, build-tools 33.0.0-rc4
    • ANDROID_HOME = /Users/mahesh/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 13.2.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • CocoaPods version 1.11.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2021.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840)

[✓] IntelliJ IDEA Community Edition (version 2021.2.1)
    • IntelliJ at /Applications/IntelliJ IDEA CE.app
    • Flutter plugin version 61.2.4
    • Dart plugin version 212.5080.8

[✓] VS Code (version 1.67.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.42.0

[✓] Connected device (3 available)
    • sdk gphone arm64 (mobile) • emulator-5554 • android-arm64  • Android 11 (API 30) (emulator)
    • macOS (desktop)           • macos         • darwin-arm64   • macOS 12.4 21F79 darwin-arm
    • Chrome (web)              • chrome        • web-javascript • Google Chrome 102.0.5005.61

[✓] HTTP Host Availability
    • All required HTTP hosts are available

• No issues found!

@maheshmnj awesome now you see the bug. It's also the same for release mode. We cannot use Firebase on MacOS because of this, not customer is going to accept a high suspicious key-chain request 8 times.

I have the same trouble. The first launch of the application without trying keychain.  Only core and analytics installed.

The error was fixed after configuring the application signature via xcode.

The error was fixed after configuring the application signing via xcode.

Can you elaborate a bit on this please?

@Aksi0 yes Aksi0, because we are signing a release version via the store. So I don't think this is a fix.

Problem reproduced when using crashalytics (so it's not only plugin:analytics), the error is probably in plugin:core.

Throw Exception() in initState()
Keychain access is triggered with crashalytics is sending the error to the FB backends.

@tommienu we're not using crashalytics fyi, just Firebase messaging.

@tommienu we're not using crashalytics fyi, just Firebase messaging.

Even more indication that the error is somewhere plugin:core then.

@tommienu indeed, this library, not good that they are forcing us to use this library to do Push Notifications on Android (because we had the nice and simple GCM). Otherwise I'd never put Firebase into any of our projects. Ah well.

I am having this issue with macOS as well - every time my app is run, in both debug and release modes, I am prompted for keychain access, and as mentioned above, this is not acceptable behavior for a user journey. In fact, this issue is actively preventing me from being able to release an app I've been working on.
I have tried the following things to fix the issue, with no success:

Changing app signing from "Sign to run locally" to "Development" (this causes the app to crash after allowing keychain access)
Turning off app sandbox (this actually breaks the app completely - it doesn't crash, but it also doesn't work)

If anyone has any insight into fixing this issue, it would be appreciated. Again, this is a release-blocking bug and unacceptable UX.

The irony of this bug I can reproduce this in my own project

Honestly I know this isn't productive. But Firebase iOS libraries have been a bane for so many years. Even in the React Native days they were plagued with build issues and crashes.
I made a suggestion that perhaps the Flutterfire team write the whole library in Dart and not use the native libraries.

Any chance we can get this into triage? Especially since this blocks any firebase release on macOS.
I'll spend some time this weekend trying older versions for this library to see if this is a recent regression. And if so, in which version it's introduced.

I've just ran the Firebase Messaging example & the Cloud Firestore example and I didn't get the keychain access request in either instance. Out of curiosity, could the users who are experiencing this issue let me know if you are initializing your app from dart code or do you have the GoogleService-Info.plist in your Runner directory?
If you wish to add the service file easily, run flutterfire configure (use latest version of FlutterFire CLI) in the root of your app and it should write the service file into your macOS Runner directory.

@russellwheatley used the CLI.

Out of curiosity, could the users who are experiencing this issue let me know if you are initializing your app from dart code or do you have the GoogleService-Info.plist in your Runner directory?

We might be on to something here. I'm actually able to run the app without any error by doing the following:

Remove all GoogleService-Info.plist files from my projects (i accidentally had one in /macos as well)
Compile and run the app, ensure that it fails with the GoogleService-Info.plist-file missing error.
Run flutterfire configure and let it create the GoogleService-Info.plist-file.

I'm now able to run the app without the aforementioned errors. I'll send this build to a few testers to confirm that it solves the problem.
My guess is that I might have been doing 50% old style setup (downloading the GoogleService-Info.plist file manually) and then running the new style setup (flutterfire configure). Or the other way around.

@tommienu have you tried creating a new app on the Firebase Console, updating the bundle id in Xcode and writing the service file again? I just noticed this and wonder whether there is something to it 🤔

I don't get the error if I change the version parameter back to the previous value in pubspec.yaml.

Do you mean changing the version of the app?

We used the CLI as I mentioned before, we created 2 files one for DEV and one for PROD for real time switching. PROD is default.
This doesn't work for Android, you need to restart the app and have the service file, so that's broken but off topic for this iOS issue.
Just checking here, you are testing with MacOS right and not iOS? The bug is related to MacOS.

Yes, macOS. My response was to @tommienu.

@tommienu have you tried creating a new app on the Firebase Console, updating the bundle id in Xcode and writing the service file again? I just noticed this and wonder whether there is something to it 🤔

I didn't try this, but I did the full renaming of the app, including SKU, new distribution cert and BUNDLE_ID. And that didn't work.
I'll try and create a new Firebase project from the flutterfire-tool and see if this solves the issue.

I don't get the error if I change the version parameter back to the previous value in pubspec.yaml.

Do you mean changing the version of the app?

Yes, correct.

@tommienu have you tried creating a new app on the Firebase Console, updating the bundle id in Xcode and writing the service file again? I just noticed this and wonder whether there is something to it 🤔

I didn't try this, but I did the full renaming of the app, including SKU, new distribution cert and BUNDLE_ID. And that didn't work.
I'll try and create a new Firebase project from the flutterfire-tool and see if this solves the issue.

The same issue, unfortunately.

I removed all keys from the keystore (to avoid collisions)
Deleted GoogleService-Info.plist from Runner-folder.
Ran flutter clean
Ran flutterfire configure and created a new Firebase project for MacOS via the command line tool.
Ran the app, first run ran fine (since it wrote the initial keys to the keystore)
Subsequent runs caused the keystore-access errors.

Out of curiosity, could the users who are experiencing this issue let me know if you are initializing your app from dart code or do you have the GoogleService-Info.plist in your Runner directory?`

In our project we have the GoogleService-Info.plist in the Runner directory and unfortunately experience the same issue.
@russellwheatley did you add/try the step @ollydixon mentioned after @maheshmnj first couldn't reproduce it using the sample project either? once he did those steps he was able to reproduce the issue so maybe the same works for you too (added a screenshot of the specific comments)

Hey folks, I've spent a couple of hours this morning trying to reproduce this on a fresh app install. I installed firebase_core,  firebase_analytics & firebase_crashlytics and updated the main.dart like the below:

code sample
```dart
import 'dart:async';
import 'package:flutter/material.dart';
import 'firebase_options.dart';
import 'package:firebase_core/firebase_core.dart';
import 'package:firebase_crashlytics/firebase_crashlytics.dart';
void main() async {
runZonedGuarded<Future>(() async {
WidgetsFlutterBinding.ensureInitialized();
await Firebase.initializeApp(
options: DefaultFirebaseOptions.currentPlatform);
FlutterError.onError = FirebaseCrashlytics.instance.recordFlutterError;
runApp(const MyApp());

}, (error, stack) => FirebaseCrashlytics.instance.recordError(error, stack));
}
class MyApp extends StatelessWidget {
const MyApp({Key? key}) : super(key: key);
// This widget is the root of your application.
@override
Widget build(BuildContext context) {
return MaterialApp(
title: 'Flutter Demo',
theme: ThemeData(
// This is the theme of your application.
//
// Try running your application with "flutter run". You'll see the
// application has a blue toolbar. Then, without quitting the app, try
// changing the primarySwatch below to Colors.green and then invoke
// "hot reload" (press "r" in the console where you ran "flutter run",
// or simply save your changes to "hot reload" in a Flutter IDE).
// Notice that the counter didn't reset back to zero; the application
// is not restarted.
primarySwatch: Colors.blue,
),
home: const MyHomePage(title: 'Flutter Demo Home Page'),
);
}
}
class MyHomePage extends StatefulWidget {
const MyHomePage({Key? key, required this.title}) : super(key: key);
// This widget is the home page of your application. It is stateful, meaning
// that it has a State object (defined below) that contains fields that affect
// how it looks.
// This class is the configuration for the state. It holds the values (in this
// case the title) provided by the parent (in this case the App widget) and
// used by the build method of the State. Fields in a Widget subclass are
// always marked "final".
final String title;
@override
State createState() => _MyHomePageState();
}
class _MyHomePageState extends State {
int _counter = 0;
void _incrementCounter() {
setState(() {
// This call to setState tells the Flutter framework that something has
// changed in this State, which causes it to rerun the build method below
// so that the display can reflect the updated values. If we changed
// _counter without calling setState(), then the build method would not be
// called again, and so nothing would appear to happen.
_counter++;
});
}
@override
Widget build(BuildContext context) {
// This method is rerun every time setState is called, for instance as done
// by the _incrementCounter method above.
//
// The Flutter framework has been optimized to make rerunning build methods
// fast, so that you can just rebuild anything that needs updating rather
// than having to individually change instances of widgets.
return Scaffold(
appBar: AppBar(
// Here we take the value from the MyHomePage object that was created by
// the App.build method, and use it to set our appbar title.
title: Text(widget.title),
),
body: Center(
// Center is a layout widget. It takes a single child and positions it
// in the middle of the parent.
child: Column(
// Column is also a layout widget. It takes a list of children and
// arranges them vertically. By default, it sizes itself to fit its
// children horizontally, and tries to be as tall as its parent.
//
// Invoke "debug painting" (press "p" in the console, choose the
// "Toggle Debug Paint" action from the Flutter Inspector in Android
// Studio, or the "Toggle Debug Paint" command in Visual Studio Code)
// to see the wireframe for each widget.
//
// Column has various properties to control how it sizes itself and
// how it positions its children. Here we use mainAxisAlignment to
// center the children vertically; the main axis here is the vertical
// axis because Columns are vertical (the cross axis would be
// horizontal).
mainAxisAlignment: MainAxisAlignment.center,
children: [
ElevatedButton(onPressed: (){
FirebaseCrashlytics.instance.crash();
}, child: const Text('press me'),),
const Text(
'You have pushed the button this many times:',
),
Text(
'$_counter',
style: Theme.of(context).textTheme.headline4,
),
],
),
),
floatingActionButton: FloatingActionButton(
onPressed: _incrementCounter,
tooltip: 'Increment',
child: const Icon(Icons.add),
), // This trailing comma makes auto-formatting nicer for build methods.
);
}
}
</details>

I've also updated the version ran in debug and release mode and I still couldn't get a keychain access prompt.

Interestingly, when I ran this command:

```bash
flutter create mac_keychain_repro

I noticed this "signing iOS app" log:
Signing iOS app for device deployment using developer identity: "Apple Development: Michael Diarmid (88BU8MA94Y)"
Creating project mac_keychain_repro...
Running "flutter pub get" in mac_keychain_repro...

At this point, I'm not spending anymore time trying to reproduce. If you can create a repro for me, I will happily look into it. I currently suspect a configuration issue and would advise ensuring you've fully complied with the macOS documentation for Flutter.

@russellwheatley just to be sure, did you also create a firebase project in firebase and set it up in the project?

I used an existing firebase project but it created a new macOS app on the project.

Ok cool, then I would indeed expect that you would also see these keychain prompts, from what I can see comparing our GoogleService-Info plist files, the iOS and MacOS ones have different values for the GOOGLE_APP_ID and one of the prompts is always for the "idvalue;*". I will make a repo with a sample project then, maybe then you can spot what is different/ gone wrong and hopefully with that project you will be able to reproduce the problem.

@TammiLion Appreciated  🙏

Made a sample project and managed to reproduce it. Some (for me easily overlooked steps). In Xcode setting the signing to your own signing (so not Sign to run Locally). Adding the Firebase.configure() into the AppDelegate.swift (if this is wrong please let me know). In our project we use Firebase Crashlytics and Firebase Messaging and I saw the same for @maheshmnj so I also added it in the pubspec. I made a video of the entire process but the interesting stuff only happens at the end of the video (besides uploading the repo to git :p).
https://github.com/TammiLion/macos_keychain_firebase_attempt2
Here is the google drive link to the 40+ min video. https://drive.google.com/file/d/13G3FWqlz3r_FwbfoBciJQCILqvxzozf1/view?usp=sharing
(Google drive is processing it, so it might not be available immediately)
The "interesting" stuff start around the 33 minute mark. This is the point where I realize I also can't reproduce it immediately just from a sample project. I then open the XCode workspace of the project in which I can reproduce it and start adding the Firebase.configure() to the AppDelegate.swift.

@TammiLion Thank you for sharing your setup. The video is only 26 seconds long, but I didn't need to watch it anyway.
You do not need to add FirebaseApp.configure() to your AppDelegate. The firebase_core plugin already takes care of configuration here.
@tommienu did you also add FirebaseApp.configure() to your AppDelegate?

@tommienu did you also add FirebaseApp.configure() to your AppDelegate?

Nope, here's my AppDelegate.swift:
import Cocoa

@NSApplicationMain
class AppDelegate: FlutterAppDelegate {
  override func applicationShouldTerminateAfterLastWindowClosed(_ sender: NSApplication) -> Bool {
    NSApp.setActivationPolicy(.accessory)
    return false
  }
}

I believe I've configured this according to the documentation.

FWIFW, this sounds like a regression from this old issue from 2020. Which was fixed in Firebase release 6.26.0, according to this comment.
Understanding what changed there might assist with understanding why it's now resurfacing.

Folks, I have been following this conversation closely over the last few days and I'm super confused as to whether there is a resolution for this issue or not. @russellwheatley seems to feel that this is a user configuration issue, yet @TammiLion is able to reproduce the issue, and @tommienu thinks there is a regression. Do we know what's going on here yet or is this still an ongoing investigation?

@russellwheatley thanks for the answer. I removed the Firebase.configure() from the AppDelegate.swift in the sample project. Unfortunately the problem still occurs.
Here's the actual video (once again, might not be immediately available due to drive processing it, will also edit the old comment): https://drive.google.com/file/d/1oXZEAc0U7abGL2Fe1ygDUPUXf-WOFVBL/view?usp=sharing

Hi everyone,
Apologies for the trouble here and thank you for all the useful information. The firebase-ios-sdk team is aware of the issue and we have been actively working on a fix to resolve this. This issue has popped up in a few GitHub issues so I'm going to centralize communications in firebase-ios-sdk/#9392, but I will also update this thread when I have a more substantial update.
I will share another update soon!

Hi everyone, I have an update here.
We have a fix that is currently planned for release in Firebase 9.6.0 next week. Thanks!

Hi everyone,
Just a quick fyi that Firebase 9.6.0 has been released. This release should unblock this issue @darshankawar.
Please continue the conversation here or in firebase-ios-sdk/#9392 if the issue persists.
As noted in this comment and in the release notes, please enable the Keychain Sharing capability for your macOS targets to make sure everything works properly.
Thanks!

@ollydixon Please check above comment and re-verify to confirm.

@ncooke3 do customers need to enable keychain sharing for their apps? As that won’t be an option.
Also; what is the reason for us enabling it? I don’t see the point.

Hi @ollydixon
Thank you for the questions. I have answered each one below.

do customers need to enable keychain sharing for their apps?

A developer building a Mac app using Firebase needs to do two things to ensure their app’s users do not see keychain access prompts:

Use a Firebase version equal or greater than 9.6.0
Add the Keychain Sharing capability to the provisioning profile they use to sign their app

I don’t see other MacOS apps requesting this; and if they do I reject.

I understand your concern. It is likely the case that such Mac apps either do not access the keychain or use the solution introduced in Firebase 9.6.0.

Apple will reject our apps without valid reason for keychain access.

There is a reason that the Keychain Sharing capability needs to be added. Firebase uses the keychain to store SDK data. For example, Firebase Installations is an
SDK that is depended on by other Firebase products. Firebase Installations manages Installation IDs, which map to an installed instance of a Firebase app. The implementation of this SDK is shared across platforms. At the time of its implementation, the Keychain was chosen to store these unique identifiers.

What is the reason for us enabling it? I don’t see the point and it’s not explained. Non of our apps ‘SDKs’ use the keychain ..

Please refer to these places for additional documentation regarding this issue:

Firebase 9.6.0 release notes
Firebase for Apple Platforms FAQ
Additionally, I posted this post to give an explanation as to why it should be enabled. Apologies for not posting it here; I posted it in firebase-ios-sdk/#9392. I'm using firebase-ios-sdk/#9392 as the central channel to track this issue and reduce cross-posting.

Additionally, the Firebase team is planning on moving away from using the keychain as a storage container where possible. It will take some work to do this, so it would be targeted for a future Firebase 10.x.0 release. For now, the fix in Firebase 9.6.0 will unblock your Mac app(s) from prompting users for permission to access the keychain. All you need to do is upgrade and add the Keychain Sharing capability to the provisioning profile you use to sign your Mac app(s).

Apple provides a very simple push notifications service/key like on iOS; why is Firebase not utilizing this?

I understand, but custom solutions are sometimes needed to reach the desired functionality. And for what it's worth, Firebase Messaging does make use of APNs.

Why is it using the keychain in the first place?

Please refer to answer 4.

Please let me know if you have further questions.

@ncooke3
Thanks for the info.

There is a reason that the keychain Sharing capability needs to be added. Firebase uses the keychain to store SDK data. For

Why not use library folder/cache or app folder for this. Storing it inside the key-chain seems a bit strange? You're asking us expose sensitive data, our key-chains might contain passwords and other items that we don't want to expose to any SDK's inside out app.

Apple will reject our apps without valid reason for keychain access.

If we get an awkward reviewer which sometimes happens, we still have no reason to use the keychain. Our answer can't be 'cause some SDK inside our app uses it; they won't accept that.

Why not use library folder/cache or app folder for this.

Firebase uses the keychain to store various bits of SDK data. For example, Firebase Auth allows you to manage user accounts. Account information is a good example of something that the Keychain is designed to store. Storing such information in the keychain is done to protect such information.
When SDKs are designed, a storage solution is chosen based on the requirements at that time. That said, we are working on reviewing our use of the Keychain, and if we find that the Keychain's security is not needed, then we will migrate away as necessary. We will be working on this and any resulting changes can be expected in a Firebase 10.X.0 release.

Storing it inside the key-chain seems a bit strange?

The Keychain can store all sorts of data. And it has its own trade-offs just like any other storage solution (File System, User Defaults, database, etc.).
While I was not involved in its design, I believe Firebase Installation tokens were stored in the keychain rather than an unprotected storage container out of an abundance of caution. And as I said in my previous answer, it's worth us examining whether or not the SDK needs to be that defensive. From an implementation and testability perspective, the Keychain can be difficult to deal with because of its API surface. So there are multiple benefits to not using the Keychain (as long as the data does not pose a risk by being stored in an unsecure container).

You're asking us expose sensitive data, our key-chains might contain passwords and other items that we don't want to expose to any SDK's inside our app.

Nobody is asking that. Just because an SDK uses the keychain does not mean that it can access such items. Each Keychain items has a dictionary of attributes that are unique to that item. These are defined by you, the developer, and Firebase has no idea what those attributes are. It is only aware of the Keychain items that it creates.
I'm happy to point you to code in the Firebase SDK that accesses the keychain. Everything is here.

If we get an awkward reviewer which sometimes happens, we still have no reason to use the keychain. Our answer can't be 'cause some SDK inside our app uses it' they won't accept that.

I understand. I think the justification can be more detailed as this is ultimately something that is trying to improve your app's user experience. Firebase Installations provides a table that provides a justification for why the SDK is used for each product. For instance, if you are using Firebase Messaging, it uses Installations to target devices for message delivery. The keychain is used to securely store the ID used to target that particular device.

Apologies if you are still not convinced. The fix in Firebase 9.6.0 was intended to quickly unblock Firebase developers targeting macOS apps. I will have more info in the coming weeks regarding what Firebase SDKs are ok to move away from the Keychain.

@ncooke3
Thanks for the reply.
Still not convinced; have many more questions and security pokes; but I won’t take anymore time.
Really it’s a shame Google can’t give us a simple messaging service without using Firebase (which was GCM on Android and APN on Apple). Forcing developers to go though Firebase to increase sales; seems antitrust. Us developers always get screwed by this mobile duopoly 🤪
For us the keychain issue is still a deal breaker and a risk not worth taking. That keychain code could change at anytime exposing users sensitive data.
We decided to go with our own solution on MacOS 😎
Thanks for your help.

@ncooke3 it seems like package:cloud_firestore is still using version 9.5.0 of the SDK. When can we expect to see an update that uses 9.6.0?

Check this PR : https://github.com/firebase/flutterfire/pull/9531

@darshankawar that's great, but firebase_core needs to be published to pub with 9.6.0 because cloud_firestore depends on it from pub, so depending on cloud_firestore from git and running pod install still brings in the 9.5.0 sdk. Really it would just be better if both packages were updated on pub. When can we expect this to happen?


Improving Resolution

Hi @noriakis,
Is there any way to improve the resolution of a graph made using the overlay_raw_map function? I made a pathway diagram using some of your code from the documentation, but the resolution is not high enough to be published. Thank you for your help.

Hi @Ramirj,
Thanks for raising this important point. What code did you use to save the plot?
The plot can be saved via ggsave the same as the formal ggplot object. However, as we use annotation_raster for overlay_raw_map, the resolution would be unclear in some environments.
Also, if you would prefer the sharp image, you could disable interpolation of annotation_raster by interpolate=FALSE in overlay_raw_map (please reinstall the devel branch using devtools::install_github).
I will try to implement some functions to save the plot by using the dimensions of the raw image anyway (ggkeggsave, presumably).

Hi, ggkeggsave is implemented in the devel branch of the repository. Please try the function and see if it helps.

Hi, could you reinstall the library on the devel branch of the repository, and try again? Also, lowering the wscale and hscale parameters could help, but the resolution would not be better than the original PNG from the KEGG. KEGG provides the 2x resolution image per pathway, so supporting this will be an important option (I'll be working on this!).

Hi @Ramirj, for this issue, if you have still problems, please consider checking #19.

Hi @noriakis,
The quality looks great in the examples you provided. I tried to implement the "output_overlay_image" code you provided to my own code and received this error message:
Error in base::nchar(wide_chars$test, type = "width") :
lazy-load database '/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/cli/R/sysdata.rdb' is corrupt
In addition: Warning messages:
1: In base::nchar(wide_chars$test, type = "width") :
restarting interrupted promise evaluation
2: In base::nchar(wide_chars$test, type = "width") :
internal error -3 in R_decompress1
Do you know what could be causing this error? Here is my code:
#########
###########
Load libraries
library(ggkegg)
library(BiocFileCache)
library(viridis)
Fetch and cache RN to EC map
url <- "https://rest.kegg.jp/link/reaction/ec"
bfc <- BiocFileCache()
path <- bfcrpath(bfc, url)
convert <- data.frame(data.table::fread(path, header = FALSE, sep = "\t"))
rntoec <- convert$V1 |>
strsplit(":") |>
vapply("[", 2, FUN.VALUE = "a") |>
setNames(convert$V2)
Define the pathway ID
pathway_id <- "dosa00270"  # Change depending on the pathway you want to map
Create a graph of the pathway
g <- pathway(pathway_id) |>
mutate(ec = rntoec[reaction])
Define a vector of node names to highlight
nodes_to_highlight <- reaction
Add log fold changes (LFCs) to the graph
LFC_vector <- LFCs  # Make sure LFCs is a vector with values for each corresponding reaction number
Ensure that LFC column is created in the graph
g <- g %>%
mutate(LFC = ifelse(reaction %in% nodes_to_highlight, LFC_vector, NA))
Create a pathway diagram with gradient colors to visualize LFCs
gg <- g |>
filter(type %in% c("compound", "gene")) |>
ggraph(g, layout = "manual", x = x, y = y) +
geom_node_rect(aes(fill = LFC), size = 5) +
geom_node_rect(aes(fill1=x, xmin=xmin2, xmax=xmax2, ymin=ymin2, ymax=ymax2, filter=type=="gene"))+
geom_node_rect(aes(fill2=y, xmin=xmin2+width, xmax=xmax2, ymin=ymin2, ymax=ymax2, filter=type=="gene"))+
scale_fill_gradient2(low = "red", high = "green",
breaks = c(-8, -4, -1, 1, 4, 8),
limits = c(-8, 8)) +
theme_minimal() +
theme_void() +
ggtitle("Induced (Green) and Repressed (Red) Genes in Cysteine and Methionine Met Pathway - 8 hr Timepoint")
output_overlay_image(gg, high_res=TRUE, use_cache=TRUE, with_legend_image=TRUE,
res=100, legend_space=100,
out="test4.png")
#> [1] "test4.png"
knitr::include_graphics("test4.png")
gg
############
#########
Thank you for taking the time to add this feature by the way.

I was able to fix what was causing the error message, but when I ran the code I got this result:


syntax error

https://github.com/vperezb/google-api-support/blob/a49f921970ff86cac545bde7155911031926f0cd/GoogleApiSupport/auth.py#L37
bit of syntax errors here in these few lines

Many thanks for the feedback


feat: adding sentieon_terraform

could please remove my PR commit history, i accidentally created PR here

I would be happy to help remove the PR, but it looks like GitHub does not provide a way for repository owners to remove the PR through their user interface and we would need to reach out to the GitHub support team, https://stackoverflow.com/a/18318431/3203719.
Would you like me to reach out to the GitHub support team (with you CC'd)?

yes, please. Thank you, and I apologize for any trouble caused!


Question about potential refactor for get_reward in AIController2D/3D

Currently, the get_reward method is set to need to be implemented in the extended AIController2D/3D:
https://github.com/edbeeching/godot_rl_agents_plugin/blob/5b09dc906eae1e037c4f8b0b09a1ffe11340802f/addons/godot_rl_agents/controller/ai_controller_3d.gd#L25-L27
And for example in the custom environment tutorial it is implemented as:
func get_reward() -> float:
	return reward

Which seems to be returning the reward that is in the base class.
Is this needed to allow some kind of reward shaping before the reward is sent to the RL algorithm, or could we refactor the base class implementation to return reward and remove the requirement to override it in the inherited class?

If the reward will usually be updated in the agent/player class, then the refactor would reduce the amount of methods that need to be overriden, and custom functionality would still be possible when needed (method can still be overridden if needed).
If the reward will usually be calculated in the extended AIController, based e.g. on signals received from the agent/player class, then overriding the method will be frequent.

The main question is should overriding the method be a requirement, or should it have a default implementation which still leaves the option of possible extension when needed.

Hey, good point. I agree.

As one potential solution is in the PR we can consider for the future (it would require env changes), I will close this issue for now.


Function_crouch: add a toggle feature rather than just hold and crouch feature

Just making this issue as a placeholder.  This is already implemented in the godot-xr-avatar repo here:
https://github.com/Godot-Dojo/Godot-XR-The-Dojo/blob/main/addons/godot-xr-tools/functions/Function_Crouch_movement.gd
So once XR Tools 3 is out and stable I can submit a pull request for this change.

Sweet!

@teddybear082 could you confirm this works as desired, and if so close the issue.

This is perfect thanks! (Tested in TPS demo project, works well).


Return SQL error details in VSCode Output tab

Is your feature request related to a problem? Please describe.
I believe it would be more productive if in Output tab would be inserted log with following informations:

SQL error message
number of returned records
commit rollback status
dbms output
etc.

The branch lazy tree has implemented something similar. See:

Closing.


Translation tutorial 3

Description

Related Issue:
Pre-commit steps

[ ] Tested on QEMU and real HW Rasperry Pi.

Not needed if it is just a README change or similar.

[ ] Ran ./contributor_setup.sh followed by ./devtool ready_for_publish

You'll need Ruby with Bundler and NPM installed locally.
If no Rust-related files were changed, ./devtool ready_for_publish_no_rust can be used instead (faster).
This step is optional, but much appreciated if done.

#166

Thanks! Do you have somebody for proofreading?
CC @colachg @readlnh

@colachg Thank you for reviewing, I try to modify

@colachg Hi, I have tried to modify it. Can you review it again? If ok, I will continue to translate the following chapters

@colachg Hi, I have tried to modify it. Can you review it again? If ok, I will continue to translate the following chapters

Of course, can you please resolve all the comments that I left above if you have modified it or even you don't want to modify it?

can you please resolve all the comments

please check, I have modified it and resolve all

Please check the english version, which is kept up-to-date.
Don't have to translate this sentence into Chinese? My understanding is that it can be linked to the English version of diff, which can be seen clearly by users. This sentence should be translated
If necessary, I will make another modification
@colachg What do you think?

Okay for me to translate this sentence 👍

Please open one PR per chapter. That makes reviewing easier for others.
It seems that for tutorial 3, you still have the English sentence for

Please check the english version, which is kept up-to-date.

In tutorial 4, it is translated already.

Thanks a lot!


Use docker compose

Description

Make the dev container use docker compose. Previously, it was built on .devcontainer/Dockerfile. Now it is built on .devcontainer/docker-compose.yml, which builds the dockerfile. Using docker compose allows the dev container to run multiple images.
Verification

[x] Rebuild dev container locally

Resource

https://github.com/devcontainers/templates/tree/main/src/javascript-node-mongo

@hhenry01 FYI see the first commit. We previously discussed that the dev container doesn't need to see CAN, as the mocks won't be using VCAN.
Overall plan is to refactor https://github.com/UBCSailbot/sailbot_workspace/pull/107 into multiple smaller PRs, the first of which is this one.


why are some bool variables default value set to null?

[x] ✋ I have searched the open/closed issues and my issue is not listed.

Please describe your question here
Why are some bool variables default value set to null? Also, their documentation mentions the default is a specific value, e.g., Defaults to false.
For example, the wait_for_jobs variable is declared as, https://github.com/aws-ia/terraform-aws-eks-blueprints-addon/blob/v1.1.1/variables.tf#L197-L201:
variable "wait_for_jobs" {
  description = "If wait is enabled, will wait until all Jobs have been completed before marking the release as successful. It will wait for as long as `timeout`. Defaults to `false`"
  type        = bool
  default     = null
}

Descriptions typically come from the resource https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release#wait_for_jobs
Setting the default to null just means that this module doesn't have an opinion on the default value and therefore its whatever set in the provider or the upstream API

interesting. are theses descriptions automatically updated when the upstream is updated? if not, can't these descriptions induce the user in error when the upstream changes the default?

Is there an issue that you are running into?

No issue at all. I'm just intrigued why is it like that. As it didn't match my expectations. I'm even surprised that null is accepted. Which reveals that I have to read more about this.

its a very common practice, one that we use across all of our modules in https://github.com/terraform-aws-modules

oh its explained at https://developer.hashicorp.com/terraform/language/expressions/types#null. when using a wrapper module to wrap other modules/resources the use of null kinda makes some sense. still, having a wrapper module variable with a description with "Default: false", but it actually uses null, is still odd to me. Please feel free to close this issue if that's ok.

The description is there to make users aware of what value they will get if they don't choose one. If you have multiple layers of abstractions, you would have to dig through those to find out what value will be set
Closing this out for now, but feel free to reach out with any further questions!


@xyflow/system: refactor public stuff

Currently we need to specify everything that we want to export to userland from system in @xyflow/react and @xyflow/svelte explicitly. It would be nice if we could find a way to specify public stuff in system directly.
current re-export in @xyflow/react:
export {
  type Foo,
  type Bar,
  ...
} from '@xyflow/system'

a better way?!:
export * from '@xyflow/system/public'

We decided that it's not worth it to complicate the bundler setup.


Updated nodejs runtime

Addresses issue #21
No other changes required aside from the version bump, code is already compliant with the newer runtime.

Thanks for the PR Simon.


Inherit recipes of the Application Workflow

Connected to #501

Quick thing.  Suggestion to change from builded to built.
✔ ~/code/SlipStreamServer [issue_#501-inherited_targets|…16⚑ 9]
09:13 $ grep -ri builded ./*
./jar-persistence/src/main/java/com/sixsq/slipstream/persistence/ImageModule.java:              public final String buildedOn;
./jar-persistence/src/main/java/com/sixsq/slipstream/persistence/ImageModule.java:                      this.buildedOn = clouds.stream().collect(Collectors.joining(","));
./jar-persistence/src/main/java/com/sixsq/slipstream/persistence/ImageModule.java:      public List<String> getCloudNamesWhereBuilded() {
./jar-persistence/src/main/java/com/sixsq/slipstream/persistence/ImageModule.java:              buildStates.add(new BuildState(image.getResourceUri(), image.getCloudNamesWhereBuilded()));
✔ ~/code/SlipStreamServer [issue_#501-inherited_targets|…16⚑ 9]
09:14 $ grep -ri builded ./* | sed -e 's/uilded/uilt/g'
./jar-persistence/src/main/java/com/sixsq/slipstream/persistence/ImageModule.java:              public final String builtOn;
./jar-persistence/src/main/java/com/sixsq/slipstream/persistence/ImageModule.java:                      this.builtOn = clouds.stream().collect(Collectors.joining(","));
./jar-persistence/src/main/java/com/sixsq/slipstream/persistence/ImageModule.java:      public List<String> getCloudNamesWhereBuilt() {
./jar-persistence/src/main/java/com/sixsq/slipstream/persistence/ImageModule.java:              buildStates.add(new BuildState(image.getResourceUri(), image.getCloudNamesWhereBuilt()));
✔ ~/code/SlipStreamServer [issue_#501-inherited_targets|…16⚑ 9]
09:14 $

Just a general comment.  There were +409/-57 lines of code added/removed, but no tests were added/updated.  There are no regressions with the current tests, but in the future it would be good to supply such big changes with tests.  It would simplify the validation as well by showing the intended usages (with assertions) of the introduced code/features/etc.


Polyfill SubmitEvent for PaleMoon

Fix #28319
It only polyfills if there is no "SubmitEvent" class, so it has no side effect for most users.

Also, I think we should create a web_src/js/polyfills.js and move existing polyfills to it, like this one for Intl.NumberFormat:
https://github.com/go-gitea/gitea/blob/main/web_src/js/webcomponents/polyfill.js

Maybe outsource the polyfill to https://github.com/idea2app/event-submitter-polyfill?

No, event-submitter-polyfill has many problems. The biggest problem is that IIRC PaleMoon doesn't seem having addEventListener(capture) support.

Also, I think we should create a web_src/js/polyfills.js and move existing polyfills to it, like this one for Intl.NumberFormat:

No, it's not feasible at the moment. Our polyfill patch needs some dependencies between functions. For example, the _submitter, and the index.js code needs to call a helper function. Putting the code together is more maintainable. If you put the code into the  webcomponents/polyfill.js, then some polyfill code appears in the webcomponents.js, some polyfill code appears in index.js, it's difficult to read or maintain.

If you put the code into the webcomponents/polyfill.js, then some polyfill code appears in the webcomponents.js, some polyfill code appears in index.js, it's difficult to read or maintain.

Not into webcomponents, but a separate polyfill file in the root. If polyfills are spread out in the code base, it will be harder to remove them later. Prefer to have them all in one file. Ideally only export functions from that file. Can be either ponyfills or polyfills.

If polyfills are spread out in the code base, it will be harder to remove them later.

This polyfill is special, it already "spreads" because there is no transparent approach, the callers have directly dependency with this change. So I do not see it's worth to move the code again.
If you have better ideas, feel free to edit this PR directly, I do not see necessary changes at the moment.


Missing identifier board macro for Uno R4 Minima

When board = uno_r4_minima is selected.
Related to #3, the board identifying macros are missing, which for the Arduino R4 Minima is ARDUINO_MINIMA.

Missing macro was added in https://github.com/platformio/platform-renesas-ra/commit/ef4b59f4edfe161335645c2fb7d8541cdea16d61


Highlight today's date for the Windows cal utility

Is it possible to somehow indicate today's date when running Windows cal utility. Something like highlight today's date or underlining it.
It might look something like this:

That's a neat idea, but, well, Windows makes this somewhat complicated and
I'm unsure if I'd want to spoil my program with that complexity! I'm aware
of two possible options:

Use Win32 console functions to change the console state before and
after printing that part of the output. The output is no longer a stream
of bytes, but a sequence of bytes and commands.

Use ANSI escapes, just as on unix-likes. The "commands" are inline in
the byte stream so the representation is unchanged. The console does not
interpret escapes by default so you have to request the behavior from the
console, but that's not too bad. It's only supported on Windows 10 and up,
so this won't work on older systems. Regardless, there needs to be a flag
to turn it off if the output is not an ANSI-escape-handling device. (Also
true on unix when redirecting to, say, a file.)

If you'd like to try (2), see ENABLE_VIRTUAL_TERMINAL_PROCESSING as used
elsewhere in this repository. Check the return from {G,S}etConsoleMode to
determine whether to actually emit ANSI escapes.


[SPARK-40784][SQL][TESTS] Check error classes in DDLParserSuite

What changes were proposed in this pull request?
This PR aims to replace 'intercept' with 'Check error classes' in DDLParserSuite.
Why are the changes needed?
The changes improve the error framework.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
By running the modified test suite:
$ build/sbt "test:testOnly *DDLParserSuite"

cc @MaxGekk

@panbingkun Please, don't forget about another DDLParserSuite at:
org.apache.spark.sql.execution.command.DDLParserSuite

@panbingkun Please, don't forget about another DDLParserSuite at: org.apache.spark.sql.execution.command.DDLParserSuite

ok.

Can one of the admins verify this patch?

When I work on it, I found typo: https://github.com/apache/spark/pull/38269
cc @MaxGekk

+1, LGTM. Merging to master.
Thank you, @panbingkun.

LGTM!


Undesirable behavior/bug in MiniBatchFileDataSetIterator()

I make a training iteration from an existing (all in memory) DataSet as
MiniBatchFileDataSetIterator trainIter = new MiniBatchFileDataSetIterator(dataSet, minibatchSize)

However, if dataSet.numExamples() < minibatchSize (as it is in my unit test), I get NO iterations.   This is undesirable to me: data are silently dropped.  Is it intentional?
And what happens if dataSet.numExamples() is not evenly divisible by minibatchSize?  I would want there to be a runt minibatch at the end with the remaining data as the usual behavior, i.e. no dropping data (especially if this were a test set iteration and not a train set iteration)

@mbkennel it already handles the left over case. We will add something for the minibatch size though.

https://github.com/deeplearning4j/nd4j/pull/1670


Hi Alexander, how to clasificate hosts by country with tags or hostgroup

Hi Alexander, how to clasificate hosts by country with tags or hostgroup in grafana. For example in grafana I have 2 organizations by country, but in zabbix I have all host with tags PAIS:XXX and have a lot of hostgroups by platform (service1,service2,service3)
how can I separate the hosts by country in grafana with zabbix datasource?

hi, you can create different accounts in zabbix for different countries, than use different datasources for each grafana´s organization.


AssertionError: Number of Block dimensions (1) must equal number of axes (2)

The following code throws AssertionError
df = pd.DataFrame({"x-1": X[:,0], "x-2": X[:,1], "label": np.where(y==1, "Driver's License", "Check")})

What is X?

X = pca.fit_transform(data)
and y is are the label,  def extrct_labels(filename):
n = len(filename)
y = np.zeros(n,dtype = np.int32)

for i,filename in enumerate(filename):
    # If 'check' string is in file name assign '0'
    if 'check' in str(filename):
        y[i] = 0
    else:
        y[i] = 1
return y

Can you give a copy-pastable example that reproduces the problem?

On Nov 1, 2015, at 00:33, ShivangiM<EMAIL_ADDRESS>wrote:
X = pca.fit_transform(data)
and y is are the label, def extrct_labels(filename):
n = len(filename)
y = np.zeros(n,dtype = np.int32)
for i,filename in enumerate(filename):
# If 'check' string is in file name assign '0'
if 'check' in str(filename):
y[i] = 0
else:
y[i] = 1
return y
—
Reply to this email directly or view it on GitHub.

Ya sure:
from scipy.misc import imread
import numpy as np
import os
import numpy as np
from sklearn.decomposition import RandomizedPCA
import pandas as pd
import pylab as pl
img_dir = 'pics/'
pics = [img_dir+ f for f in os.listdir(img_dir)]
def extrct_labels(filename):
n = len(filename)
y = np.zeros(n,dtype = np.int32)

for i,filename in enumerate(filename):
    # If 'check' string is in file name assign '0'
    if 'check' in str(filename):
        y[i] = 0
    else:
        y[i] = 1
return y

data = []
for pic in pics:
img = imread(pic)
img = img.flatten()
data.append(img)
data = np.array(data)
pca = RandomizedPCA(n_components=2)
X = pca.fit_transform(data)
df = pd.DataFrame({"x-1": X[:,0], "x-2": X[:,1], "label": np.where(y==1, "Driver's License", "Check")})

@ShivangiM we don't have your directory of images handy, and using scikit-learn obscures the origin of this problem (see http://sscce.org/). Please try reproducing the issue using only numpy and pandas, if possible.

pls provide a copy-pastable example that is runnable w/o your data (and ideally is just write before the DataFrame creation step).


swap out 600M param model for tiny for faster startup

On a 3090 it saves 2 minutes per run.

This change is

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (8a9b64e) 86.82% compared to head (8211d42) 86.82%.

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #92   +/-   ##
=======================================
  Coverage   86.82%   86.82%
=======================================
  Files         226      226
  Lines       13629    13629
=======================================
  Hits        11833    11833
  Misses       1796     1796

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Remove summary_view

summary_view  was added as a part of  the ticket #771
However, view did not work as expected due to Postgresql rule that Access to tables referenced in the view is determined by permissions of the view owner. View is created under ccbc user. Changing the owner of the view or changing permissions of the ccbc user to bypass RLS would mean opening security hole and effectively disabling row-level security.
Thus, alternative approach was implemented (using Metabase Model) and thus, code that adds summary_view in Postgres DB should be removed.

I can still see them in metabase:

Ok those views are no longer listed in Metabase


Error in notebook "RCPSP-6 Large Neighbourhood Search" with LargeNeighborhoodSearchScheduling

The cell lns_solver = LargeNeighborhoodSearchScheduling(problem=rcpsp_problem) is crashing. It seems that it expects a rcpsp model with special constraints and our case has not such constraints.

fixed in #159


We have to use clipsToBounds = true for our case, how can we do it?

İn our case we have a favorites button for tablecells. When someone clicked on that we set cell heights to zero. When we are not using clipsToBounds = true it is not working as expect.
When we use clipsToBounds = true  the actions are gone.
How can we build it?

Remove the cell instead of hiding it.


ACC (Accumulator) Chip Clarification

ACC chips don't work well with spaced-out attackers, as they only have an effect when multiple attackers enter them at the same time (or close to each other?). The maximum spacing should be clarified to make players more aware of their limitations.

There is no maximum spacing. They wait for the next attacker to enter, no matter how long it takes. You can tell from the thicker frame that there is a number "inside".
Have a look at https://github.com/ochadenas/cpudefense/wiki/ACC for more info.


Getting tons of Error: unknown .loc sub-directive

System Info
I am attempting to install text-generation-inference on our local servers.  Things are blowing up when the install does: cd router && cargo install --path . command during the install process:
   Compiling unicode_categories v0.1.1
   Compiling rand v0.8.5
   Compiling nom v7.1.3
   Compiling nohash-hasher v0.2.0
   Compiling ordered-float v3.9.1
   Compiling clap_builder v4.4.6
   Compiling hostname v0.3.1
   Compiling crossbeam-deque v0.8.3
   Compiling utoipa-swagger-ui v3.1.5
   Compiling sha2 v0.10.8
   Compiling flume v0.11.0
   Compiling mime_guess v2.0.4
The following warnings were emitted during compilation:
warning: /tmp/ccULKIi9.s: Assembler messages:
warning: /tmp/ccULKIi9.s:13: Error: unknown .loc sub-directive `view'
warning: /tmp/ccULKIi9.s:13: Error: junk at end of line, first unrecognized character is `-'
warning: /tmp/ccULKIi9.s:15: Error: unknown .loc sub-directive `view'
warning: /tmp/ccULKIi9.s:15: Error: unknown pseudo-op: `.lvu1'
warning: /tmp/ccULKIi9.s:16: Error: unknown .loc sub-directive `view'
warning: /tmp/ccULKIi9.s:16: Error: unknown pseudo-op: `.lvu2'

Totally stuck on how to resolve this issue.
Information

[ ] Docker
[X] The CLI directly

Tasks

[X] An officially supported command
[ ] My own modifications

Reproduction
Following the instructions:
BUILD_EXTENSIONS=True make install
Expected behavior
not erroring out

Do the instructions in https://github.com/compiler-explorer/compiler-explorer/issues/3239 help?

Thanks @McPatate -
I see them say:
-gas-locview-support
Inform the compiler that the assembler supports view assignment and reset assertion checking in .loc directives.

This option will be enabled by default if, at GCC configure time, the assembler was found to support them.

-gno-as-locview-support
Force GCC to assign view numbers internally, if -gvariable-location-views are explicitly requested.

I have no idea how to use this with the make command in the README

@McPatate - it seems that I had to remove the .cargo/registry and start all over.

I gave up on this - too many problems, I guess i am forced to use docker.


[loadgenerator] start locust with host "<IP_ADDRESS>" (#1437)

starts locust with env LOCUST_WEB_HOST=<IP_ADDRESS>

Why is this change needed?

Why is this change needed?

locust fails to start if ipv6 is disabled. With this env, we force locust to bind ipv4 address.

LOCUST_WEB_PORT

This implementation is fine, but we should be following the pattern for the other LOCUST_* environment variables, which is defining them all in the .env file, and referencing them from the 2 compose files.
SIMILAR to how we LOCUST_WEB_PORT and LOCUST_HOST defined.

I Can't figure out how to change 'LOCUST_WEB_HOST' in .env file, it's already defined in .env as LOCUST_WEB_HOST=loadgenerator.
I guess LOCUST_WEB_HOST in .env is used by other service to get node name (or dns name? i don't known) of loadgenerator, not used to tell locust to bind to this address.
So there are tow different things (in my guess):

how other service find address of loadgenerator
which ip address shoud loadgenerator bind

These two things just use same env name by coincidence.

hmmm... I see now. We did some unfortunate things with the naming of environment variables used for the frontend-proxy.
Let's get this one approved and merged. I'll create a separate issue to track fixing the frontend-proxy.


[FIXED] Release on linux amd64 is the only one dynamically linked

[X] Link to issue
[X] Documentation added (N/A)
[X] Tests added (ran the script on a Linux amd64 box)
[X] Branch rebased on top of current master (git pull --rebase origin master)
[X] Changes squashed to a single commit (described here)
[ ] Build is green in Travis CI
[X] You have certified that the contribution is your original work and that you license the work to the project under the MIT license

Resolves # 470
Changes proposed in this pull request:
We are using the tool from https://github.com/mitchellh/gox
for cross compilation. When we issue a new release, we create a
tag and trigger Travis build which will then invoke the script and
upload the files to GH.
We were affected by https://github.com/mitchellh/gox/issues/55
Invoking the script on a Linux amd64 would produce this:
file pkg/gnatsd-linux-amd64/gnatsd
pkg/gnatsd-linux-amd64/gnatsd: ELF 64-bit LSB executable, x86-64,
version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, stripped
With the fix, you'll get this:
file pkg/gnatsd-linux-amd64/gnatsd
pkg/gnatsd-linux-amd64/gnatsd: ELF 64-bit LSB executable, x86-64,
version 1 (SYSV), statically linked, stripped
/cc @nats-io/core

Coverage decreased (-0.2%) to 90.47% when pulling 67a4a37616f0b2ded6073391c032679d9b4f8e2c on fix_issue_470 into 64dcdd6a7e9225e71c7678a9992802eff98d2e05 on master.


Course with no heading (or not at the top) does not render at all

In the following examples LiaScript renders a blank slide:

Example containing no headers: https://liascript.github.io/course/?https://raw.githubusercontent.com/jhc-at-home/scratch/master/NoHeaders.md

Some text

Some more text

Example with heading after some text: https://liascript.github.io/course/?https://raw.githubusercontent.com/jhc-at-home/scratch/master/WithHeaders.md

Some text

# A heading

Some more __text__

I fixed this, the default behavior is now, that this initial content without a heading gets ignored. Probably this can be used as an extra space for adding comments to a course, which shall/is not visible in LiaScript ...