torch.distributed.DistBackendError when training on multiple GPUs

scvi crashes when trying to train on multiple GPUs (2x Tesla P100-PCIE-16GB)
As attempt to work around https://github.com/Lightning-AI/pytorch-lightning/issues/17212 issue strategy='ddp_find_unused_parameters_true' was set.
def annotate(adata: AnnData, geneset: Dict, out_dir: str = 'out', epochs: int = None, visualize: bool = False, save: bool = False, random_seed=None) -> pd.Series:
    normalized = adata.copy()
    sc.pp.normalize_total(normalized, target_sum=1e4)
    sc.pp.log1p(normalized)
    normalized = normalized[:, geneset['gene_subset']].copy()
    sc.pp.scale(normalized)
    adata.obs["seed_labels"] = generate_seed_labels(normalized, geneset['cell_geneset'])
    base_model_train_ratio, transfer_model_train_ratio = get_train_ratio(adata, unconstrained_train_ratio=0.9)
    torch.set_float32_matmul_precision("medium")
    scvi.model.SCVI.setup_anndata(adata, batch_key=None, labels_key="seed_labels")
    if random_seed is not None:
        scvi.settings.seed = random_seed
    scvi_model = scvi.model.SCVI(adata, n_latent=30, n_layers=2)
    scvi_model.train(max_epochs=epochs, train_size=base_model_train_ratio, accelerator='gpu', devices=-1, strategy='ddp_find_unused_parameters_true')

    scanvi_model = scvi.model.SCANVI.from_scvi_model(scvi_model, 'unknown')
    print('Training transfer model:')
    scanvi_model.train(max_epochs=epochs, train_size=transfer_model_train_ratio, accelerator='gpu', devices=-1, strategy='ddp_find_unused_parameters_true'
    )

Training transfer model:

GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]

Training:   0%|          | 0/100 [00:00<?, ?it/s]
Epoch 1/100:   0%|          | 0/100 [00:00<?, ?it/s]/usr/local/lib/python3.10/dist-packages/lightning/pytorch/trainer/connectors/data_connector.py:441: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=7` in the `DataLoader` to improve performance.

Training:   0%|          | 0/100 [00:00<?, ?it/s]
Epoch 1/100:   0%|          | 0/100 [00:00<?, ?it/s]/usr/local/lib/python3.10/dist-packages/scvi/module/_scanvae.py:304: UserWarning: The value argument must be within the support of the distribution
  reconst_loss = -px.log_prob(x).sum(-1)
/usr/local/lib/python3.10/dist-packages/scvi/module/_scanvae.py:304: UserWarning: The value argument must be within the support of the distribution
  reconst_loss = -px.log_prob(x).sum(-1)
/usr/local/lib/python3.10/dist-packages/scvi/module/_scanvae.py:304: UserWarning: The value argument must be within the support of the distribution
  reconst_loss = -px.log_prob(x).sum(-1)

Epoch 1/100:   1%|          | 1/100 [00:15<26:13, 15.89s/it]/usr/local/lib/python3.10/dist-packages/lightning/pytorch/trainer/connectors/logger_connector/result.py:433: It is recommended to use `self.log('train_loss', ..., sync_dist=True)` when logging on epoch level in distributed setting to accumulate the metric across devices.

Epoch 1/100:   1%|          | 1/100 [00:18<30:16, 18.35s/it][rank1]:[W socket.cpp:432] [c10d] While waitForInput, poolFD failed with (errno: 0 - Success).
Traceback (most recent call last):
  File "/work/demo.py", line 8, in <module>
    run(args)
  File "/work/src/main.py", line 40, in run
    pred = annotate(
  File "/work/src/annotate.py", line 181, in annotate
    scanvi_model.train(
  File "/usr/local/lib/python3.10/dist-packages/scvi/model/_scanvi.py", line 438, in train
    return runner()
  File "/usr/local/lib/python3.10/dist-packages/scvi/train/_trainrunner.py", line 98, in __call__
    self.trainer.fit(self.training_plan, self.data_splitter)
  File "/usr/local/lib/python3.10/dist-packages/scvi/train/_trainer.py", line 219, in fit
    super().fit(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/trainer/trainer.py", line 544, in fit
    call._call_and_handle_interrupt(
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/trainer/call.py", line 44, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/trainer/trainer.py", line 580, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/trainer/trainer.py", line 989, in _run
    results = self._run_stage()
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/trainer/trainer.py", line 1035, in _run_stage
    self.fit_loop.run()
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/loops/fit_loop.py", line 203, in run
    self.on_advance_end()
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/loops/fit_loop.py", line 372, in on_advance_end
    call._call_callback_hooks(trainer, "on_train_epoch_end", monitoring_callbacks=False)
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/trainer/call.py", line 208, in _call_callback_hooks
    fn(trainer, trainer.lightning_module, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/scvi/train/_progress.py", line 85, in on_train_epoch_end
    self.main_progress_bar.set_postfix(self.get_metrics(trainer, pl_module))
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/callbacks/progress/progress_bar.py", line 195, in get_metrics
    pbar_metrics = trainer.progress_bar_metrics
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/trainer/trainer.py", line 1653, in progress_bar_metrics
    return self._logger_connector.progress_bar_metrics
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/trainer/connectors/logger_connector/logger_connector.py", line 245, in progress_bar_metrics
    metrics = self.metrics["pbar"]
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/trainer/connectors/logger_connector/logger_connector.py", line 226, in metrics
    return self.trainer._results.metrics(on_step)
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/trainer/connectors/logger_connector/result.py", line 475, in metrics
    value = self._get_cache(result_metric, on_step)
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/trainer/connectors/logger_connector/result.py", line 439, in _get_cache
    result_metric.compute()
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/trainer/connectors/logger_connector/result.py", line 284, in wrapped_func
    self._computed = compute(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/trainer/connectors/logger_connector/result.py", line 252, in compute
    return self.value.compute()
  File "/usr/local/lib/python3.10/dist-packages/torchmetrics/metric.py", line 1135, in compute
    val_a = self.metric_a.compute() if isinstance(self.metric_a, Metric) else self.metric_a
  File "/usr/local/lib/python3.10/dist-packages/torchmetrics/metric.py", line 1135, in compute
    val_a = self.metric_a.compute() if isinstance(self.metric_a, Metric) else self.metric_a
  File "/usr/local/lib/python3.10/dist-packages/torchmetrics/metric.py", line 610, in wrapped_func
    with self.sync_context(
  File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.10/dist-packages/torchmetrics/metric.py", line 581, in sync_context
    self.sync(
  File "/usr/local/lib/python3.10/dist-packages/torchmetrics/metric.py", line 530, in sync
    self._sync_dist(dist_sync_fn, process_group=process_group)
  File "/usr/local/lib/python3.10/dist-packages/torchmetrics/metric.py", line 434, in _sync_dist
    output_dict = apply_to_collection(
  File "/usr/local/lib/python3.10/dist-packages/lightning_utilities/core/apply_func.py", line 70, in apply_to_collection
    return {k: function(v, *args, **kwargs) for k, v in data.items()}
  File "/usr/local/lib/python3.10/dist-packages/lightning_utilities/core/apply_func.py", line 70, in <dictcomp>
    return {k: function(v, *args, **kwargs) for k, v in data.items()}
  File "/usr/local/lib/python3.10/dist-packages/torchmetrics/utilities/distributed.py", line 122, in gather_all_tensors
    return _simple_gather_all_tensors(result, group, world_size)
  File "/usr/local/lib/python3.10/dist-packages/torchmetrics/utilities/distributed.py", line 93, in _simple_gather_all_tensors
    torch.distributed.all_gather(gathered_result, result, group)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 72, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 2617, in all_gather
    work = group.allgather([tensor_list], [tensor])
torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '1', but store->get('1') got error: Socket Timeout
Exception raised from doWait at ../torch/csrc/distributed/c10d/TCPStore.cpp:550 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fab06181d87 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x15c0e57 (0x7faaee93fe57 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so)
frame #2: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7faaf2c0ece2 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so)
frame #3: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7faaf2c0fb11 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so)
frame #4: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7faaf2bc4f81 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so)
frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7faaf2bc4f81 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so)
frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7faaf2bc4f81 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so)
frame #7: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7faabbe02c69 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #8: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector<c10::Device, std::allocator<c10::Device> > const&, c10d::OpType, int, bool) + 0x22b (0x7faabbe09c5b in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #9: c10d::ProcessGroupNCCL::allgather(std::vector<std::vector<at::Tensor, std::allocator<at::Tensor> >, std::allocator<std::vector<at::Tensor, std::allocator<at::Tensor> > > >&, std::vector<at::Tensor, std::allocator<at::Tensor> >&, c10d::AllgatherOptions const&) + 0xb5c (0x7faabbe2005c in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #10: <unknown function> + 0x583a31d (0x7faaf2bb931d in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so)
frame #11: <unknown function> + 0x5844218 (0x7faaf2bc3218 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so)
frame #12: <unknown function> + 0x4e893cc (0x7faaf22083cc in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so)
frame #13: <unknown function> + 0x1a08a88 (0x7faaeed87a88 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so)
frame #14: <unknown function> + 0x584ba33 (0x7faaf2bcaa33 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so)
frame #15: <unknown function> + 0x5856e1f (0x7faaf2bd5e1f in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so)
frame #16: <unknown function> + 0xca3fae (0x7fab0545afae in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so)
frame #17: <unknown function> + 0x413ea4 (0x7fab04bcaea4 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so)
frame #18: <unknown function> + 0x15a10e (0x59d15614610e in /usr/bin/python3)
frame #19: _PyObject_MakeTpCall + 0x25b (0x59d15613ca7b in /usr/bin/python3)
frame #20: <unknown function> + 0x168acb (0x59d156154acb in /usr/bin/python3)
frame #21: _PyEval_EvalFrameDefault + 0x614a (0x59d156134cfa in /usr/bin/python3)
frame #22: _PyFunction_Vectorcall + 0x7c (0x59d1561469fc in /usr/bin/python3)
frame #23: _PyEval_EvalFrameDefault + 0x2a27 (0x59d1561315d7 in /usr/bin/python3)
frame #24: _PyFunction_Vectorcall + 0x7c (0x59d1561469fc in /usr/bin/python3)
frame #25: _PyEval_EvalFrameDefault + 0x614a (0x59d156134cfa in /usr/bin/python3)
frame #26: _PyFunction_Vectorcall + 0x7c (0x59d1561469fc in /usr/bin/python3)
frame #27: _PyEval_EvalFrameDefault + 0x6bd (0x59d15612f26d in /usr/bin/python3)
frame #28: _PyFunction_Vectorcall + 0x7c (0x59d1561469fc in /usr/bin/python3)
frame #29: PyObject_Call + 0x122 (0x59d156155492 in /usr/bin/python3)
frame #30: _PyEval_EvalFrameDefault + 0x2a27 (0x59d1561315d7 in /usr/bin/python3)
frame #31: _PyFunction_Vectorcall + 0x7c (0x59d1561469fc in /usr/bin/python3)
frame #32: _PyEval_EvalFrameDefault + 0x6bd (0x59d15612f26d in /usr/bin/python3)
frame #33: _PyFunction_Vectorcall + 0x7c (0x59d1561469fc in /usr/bin/python3)
frame #34: _PyEval_EvalFrameDefault + 0x198c (0x59d15613053c in /usr/bin/python3)
frame #35: <unknown function> + 0x1687f1 (0x59d1561547f1 in /usr/bin/python3)
frame #36: _PyEval_EvalFrameDefault + 0x198c (0x59d15613053c in /usr/bin/python3)
frame #37: <unknown function> + 0x1687f1 (0x59d1561547f1 in /usr/bin/python3)
frame #38: _PyEval_EvalFrameDefault + 0x198c (0x59d15613053c in /usr/bin/python3)
frame #39: <unknown function> + 0x200175 (0x59d1561ec175 in /usr/bin/python3)
frame #40: <unknown function> + 0x15ac59 (0x59d156146c59 in /usr/bin/python3)
frame #41: _PyEval_EvalFrameDefault + 0x6bd (0x59d15612f26d in /usr/bin/python3)
frame #42: <unknown function> + 0x168a51 (0x59d156154a51 in /usr/bin/python3)
frame #43: _PyEval_EvalFrameDefault + 0x266d (0x59d15613121d in /usr/bin/python3)
frame #44: _PyFunction_Vectorcall + 0x7c (0x59d1561469fc in /usr/bin/python3)
frame #45: _PyEval_EvalFrameDefault + 0x614a (0x59d156134cfa in /usr/bin/python3)
frame #46: <unknown function> + 0x1687f1 (0x59d1561547f1 in /usr/bin/python3)
frame #47: _PyEval_EvalFrameDefault + 0x614a (0x59d156134cfa in /usr/bin/python3)
frame #48: <unknown function> + 0x1687f1 (0x59d1561547f1 in /usr/bin/python3)
frame #49: _PyEval_EvalFrameDefault + 0x614a (0x59d156134cfa in /usr/bin/python3)
frame #50: <unknown function> + 0x168a51 (0x59d156154a51 in /usr/bin/python3)
frame #51: _PyEval_EvalFrameDefault + 0x2a27 (0x59d1561315d7 in /usr/bin/python3)
frame #52: _PyFunction_Vectorcall + 0x7c (0x59d1561469fc in /usr/bin/python3)
frame #53: _PyEval_EvalFrameDefault + 0x614a (0x59d156134cfa in /usr/bin/python3)
frame #54: _PyFunction_Vectorcall + 0x7c (0x59d1561469fc in /usr/bin/python3)
frame #55: _PyEval_EvalFrameDefault + 0x614a (0x59d156134cfa in /usr/bin/python3)
frame #56: _PyFunction_Vectorcall + 0x7c (0x59d1561469fc in /usr/bin/python3)
frame #57: _PyEval_EvalFrameDefault + 0x8ac (0x59d15612f45c in /usr/bin/python3)
frame #58: <unknown function> + 0x16600e (0x59d15615200e in /usr/bin/python3)
frame #59: _PyObject_GenericGetAttrWithDict + 0x468 (0x59d1561448a8 in /usr/bin/python3)
frame #60: PyObject_GetAttr + 0x4d (0x59d156142e3d in /usr/bin/python3)
frame #61: _PyEval_EvalFrameDefault + 0x5dc1 (0x59d156134971 in /usr/bin/python3)
frame #62: <unknown function> + 0x16600e (0x59d15615200e in /usr/bin/python3)
frame #63: _PyObject_GenericGetAttrWithDict + 0x468 (0x59d1561448a8 in /usr/bin/python3)
. This may indicate a possible application crash on rank 0 or a network set up issue.

Epoch 1/100:   1%|          | 1/100 [30:19<50:01:35, 1819.15s/it]
[rank0]:[E ProcessGroupNCCL.cpp:523] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=115017, OpType=ALLGATHER, NumelIn=1, NumelOut=2, Timeout(ms)=1800000) ran for 1800859 milliseconds before timing out.
[rank0]:[E ProcessGroupNCCL.cpp:537] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank0]:[E ProcessGroupNCCL.cpp:543] To avoid data inconsistency, we are taking the entire process down.
[rank0]:[E ProcessGroupNCCL.cpp:1182] [Rank 0] NCCL watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=115017, OpType=ALLGATHER, NumelIn=1, NumelOut=2, Timeout(ms)=1800000) ran for 1800859 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:525 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7d37dfb81d87 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1e6 (0x7d37958026e6 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x19d (0x7d3795805c3d in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7d3795806839 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0xdc253 (0x7d38071b9253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #5: <unknown function> + 0x94ac3 (0x7d381d459ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #6: <unknown function> + 0x126a40 (0x7d381d4eba40 in /lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
  what():  [Rank 0] NCCL watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=115017, OpType=ALLGATHER, NumelIn=1, NumelOut=2, Timeout(ms)=1800000) ran for 1800859 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:525 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7d37dfb81d87 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1e6 (0x7d37958026e6 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x19d (0x7d3795805c3d in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7d3795806839 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0xdc253 (0x7d38071b9253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #5: <unknown function> + 0x94ac3 (0x7d381d459ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #6: <unknown function> + 0x126a40 (0x7d381d4eba40 in /lib/x86_64-linux-gnu/libc.so.6)

Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1186 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7d37dfb81d87 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0xdf6b11 (0x7d379555cb11 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #2: <unknown function> + 0xdc253 (0x7d38071b9253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #3: <unknown function> + 0x94ac3 (0x7d381d459ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #4: <unknown function> + 0x126a40 (0x7d381d4eba40 in /lib/x86_64-linux-gnu/libc.so.6)

Aborted (core dumped)

Versions:

cuda 11.8
scvi-tools 1.1.1
jaxlib 0.4.23+cuda11.cudnn86

any update ?

It looks like the issue is referring to a timeout in the NCCL library. Could you paste the installed versions of PyTorch, Lightning, and NCCL? Also, are any training steps being run or is it crashing right from the start (I see that it doesn't go past the first training epoch)?

OK looking at the traceback further, looks like there's an issue with syncing losses to display it in the progress bar. Could you try passing in enable_progress_bar=False to scanvi_model.train(...)?

Here is the log with enable_progress_bar=False:
2024-06-13 09:03:53.504676: I external/xla/xla/stream_executor/cuda/cuda_[executor.cc:991](http://executor.cc:991/)] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-06-13 09:03:53.509353: I external/xla/xla/stream_executor/cuda/cuda_[executor.cc:991](http://executor.cc:991/)] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-06-13 09:03:53.510154: I external/xla/xla/service/[service.cc:145](http://service.cc:145/)] XLA service 0x5a219133c0d0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2024-06-13 09:03:53.510223: I external/xla/xla/service/[service.cc:153](http://service.cc:153/)]   StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2024-06-13 09:03:53.510250: I external/xla/xla/service/[service.cc:153](http://service.cc:153/)]   StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2024-06-13 09:03:53.516528: I external/xla/xla/pjrt/gpu/se_gpu_pjrt_[client.cc:738](http://client.cc:738/)] Using BFC allocator.
2024-06-13 09:03:53.516599: I external/xla/xla/pjrt/gpu/gpu_[helpers.cc:105](http://helpers.cc:105/)] XLA backend allocating<PHONE_NUMBER>6 bytes on device 0 for BFCAllocator.
2024-06-13 09:03:53.516649: I external/xla/xla/pjrt/gpu/gpu_[helpers.cc:105](http://helpers.cc:105/)] XLA backend allocating<PHONE_NUMBER>6 bytes on device 1 for BFCAllocator.
2024-06-13 09:03:53.516838: I external/xla/xla/stream_executor/cuda/cuda_[executor.cc:991](http://executor.cc:991/)] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-06-13 09:03:53.517388: I external/xla/xla/stream_executor/cuda/cuda_[executor.cc:991](http://executor.cc:991/)] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
/usr/local/lib/python3.10/dist-packages/scvi/data/fields/_base_[field.py:64](http://field.py:64/): UserWarning: adata.X does not contain unnormalized count data. Are you sure this is what you want?
  self.validate_field(adata)

Training base model:

GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/2
2024-06-13 09:08:55.251742: I external/xla/xla/stream_executor/cuda/cuda_[executor.cc:991](http://executor.cc:991/)] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-06-13 09:08:55.256383: I external/xla/xla/stream_executor/cuda/cuda_[executor.cc:991](http://executor.cc:991/)] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-06-13 09:08:55.257143: I external/xla/xla/service/[service.cc:145](http://service.cc:145/)] XLA service 0x59f238331a40 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2024-06-13 09:08:55.257186: I external/xla/xla/service/[service.cc:153](http://service.cc:153/)]   StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2024-06-13 09:08:55.257199: I external/xla/xla/service/[service.cc:153](http://service.cc:153/)]   StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2024-06-13 09:08:55.258777: I external/xla/xla/pjrt/gpu/se_gpu_pjrt_[client.cc:738](http://client.cc:738/)] Using BFC allocator.
2024-06-13 09:08:55.258847: I external/xla/xla/pjrt/gpu/gpu_[helpers.cc:105](http://helpers.cc:105/)] XLA backend allocating<PHONE_NUMBER>6 bytes on device 0 for BFCAllocator.
2024-06-13 09:08:55.258973: I external/xla/xla/pjrt/gpu/gpu_[helpers.cc:105](http://helpers.cc:105/)] XLA backend allocating<PHONE_NUMBER>6 bytes on device 1 for BFCAllocator.
2024-06-13 09:08:55.259192: I external/xla/xla/stream_executor/cuda/cuda_[executor.cc:991](http://executor.cc:991/)] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-06-13 09:08:55.259678: I external/xla/xla/stream_executor/cuda/cuda_[executor.cc:991](http://executor.cc:991/)] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
/usr/local/lib/python3.10/dist-packages/scvi/data/fields/_base_[field.py:64](http://field.py:64/): UserWarning: adata.X does not contain unnormalized count data. Are you sure this is what you want?
  self.validate_field(adata)

Training base model:

Initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/2
----------------------------------------------------------------------------------------------------
distributed_backend=nccl
All distributed processes registered. Starting with 2 processes
----------------------------------------------------------------------------------------------------

LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]
LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1]
/usr/local/lib/python3.10/dist-packages/lightning/pytorch/trainer/connectors/data_[connector.py:441](http://connector.py:441/): The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=15` in the `DataLoader` to improve performance.

Training:   0%|          | 0/100 [00:00<?, ?it/s]
Epoch 1/100:   0%|          | 0/100 [00:00<?, ?it/s]/usr/local/lib/python3.10/dist-packages/scvi/module/_[vae.py:458](http://vae.py:458/): UserWarning: The value argument must be within the support of the distribution
  reconst_loss = -generative_outputs["px"].log_prob(x).sum(-1)
/usr/local/lib/python3.10/dist-packages/scvi/module/_[vae.py:458](http://vae.py:458/): UserWarning: The value argument must be within the support of the distribution
  reconst_loss = -generative_outputs["px"].log_prob(x).sum(-1)
/usr/local/lib/python3.10/dist-packages/scvi/distributions/_negative_[binomial.py:69](http://binomial.py:69/): UserWarning: Specified kernel cache directory could not be created! This disables kernel caching. Specified directory is /root/.cache/torch/kernels. This warning will appear only once per process. (Triggered internally at ../aten/src/ATen/native/cuda/jit_utils.cpp:1442.)
  + torch.lgamma(x + theta)
/usr/local/lib/python3.10/dist-packages/scvi/module/_[vae.py:458](http://vae.py:458/): UserWarning: The value argument must be within the support of the distribution
  reconst_loss = -generative_outputs["px"].log_prob(x).sum(-1)

Epoch 1/100:   1%|          | 1/100 [00:07<11:58,  7.26s/it]
Epoch 1/100:   1%|          | 1/100 [00:07<11:58,  7.26s/it, v_num=1, train_loss_step=5.37e+3, train_loss_epoch=5.84e+3]
Epoch 2/100:   1%|          | 1/100 [00:07<11:58,  7.26s/it, v_num=1, train_loss_step=5.37e+3, train_loss_epoch=5.84e+3]/usr/local/lib/python3.10/dist-packages/scvi/module/_[vae.py:458](http://vae.py:458/): UserWarning: The value argument must be within the support of the distribution
  reconst_loss = -generative_outputs["px"].log_prob(x).sum(-1)
/usr/local/lib/python3.10/dist-packages/scvi/module/_[vae.py:458](http://vae.py:458/): UserWarning: The value argument must be within the support of the distribution
  reconst_loss = -generative_outputs["px"].log_prob(x).sum(-1)

Epoch 2/100:   2%|▏         | 2/100 [00:12<09:29,  5.81s/it, v_num=1, train_loss_step=5.37e+3, train_loss_epoch=5.84e+3]

[...]

Epoch 98/100:  98%|█████████▊| 98/100 [07:59<00:09,  4.88s/it, v_num=1, train_loss_step=4.48e+3, train_loss_epoch=4.41e+3]
Epoch 98/100:  98%|█████████▊| 98/100 [07:59<00:09,  4.88s/it, v_num=1, train_loss_step=4.03e+3, train_loss_epoch=4.4e+3]
Epoch 99/100:  98%|█████████▊| 98/100 [07:59<00:09,  4.88s/it, v_num=1, train_loss_step=4.03e+3, train_loss_epoch=4.4e+3]/usr/local/lib/python3.10/dist-packages/scvi/module/_[vae.py:458](http://vae.py:458/): UserWarning: The value argument must be within the support of the distribution
  reconst_loss = -generative_outputs["px"].log_prob(x).sum(-1)
/usr/local/lib/python3.10/dist-packages/scvi/module/_[vae.py:458](http://vae.py:458/): UserWarning: The value argument must be within the support of the distribution
  reconst_loss = -generative_outputs["px"].log_prob(x).sum(-1)

Epoch 99/100:  99%|█████████▉| 99/100 [08:04<00:04,  4.89s/it, v_num=1, train_loss_step=4.03e+3, train_loss_epoch=4.4e+3]
Epoch 99/100:  99%|█████████▉| 99/100 [08:04<00:04,  4.89s/it, v_num=1, train_loss_step=4.33e+3, train_loss_epoch=4.38e+3]
Epoch 100/100:  99%|█████████▉| 99/100 [08:04<00:04,  4.89s/it, v_num=1, train_loss_step=4.33e+3, train_loss_epoch=4.38e+3]/usr/local/lib/python3.10/dist-packages/scvi/module/_[vae.py:458](http://vae.py:458/): UserWarning: The value argument must be within the support of the distribution
  reconst_loss = -generative_outputs["px"].log_prob(x).sum(-1)
/usr/local/lib/python3.10/dist-packages/scvi/module/_[vae.py:458](http://vae.py:458/): UserWarning: The value argument must be within the support of the distribution
  reconst_loss = -generative_outputs["px"].log_prob(x).sum(-1)

Epoch 100/100: 100%|██████████| 100/100 [08:09<00:00,  4.87s/it, v_num=1, train_loss_step=4.33e+3, train_loss_epoch=4.38e+3]
Epoch 100/100: 100%|██████████| 100/100 [08:09<00:00,  4.87s/it, v_num=1, train_loss_step=4.2e+3, train_loss_epoch=4.39e+3] `[Trainer.fit](http://trainer.fit/)` stopped: `max_epochs=100` reached.

Epoch 100/100: 100%|██████████| 100/100 [08:09<00:00,  4.89s/it, v_num=1, train_loss_step=4.2e+3, train_loss_epoch=4.39e+3]
/usr/local/lib/python3.10/dist-packages/scvi/data/fields/_base_[field.py:64](http://field.py:64/): UserWarning: adata.X does not contain unnormalized count data. Are you sure this is what you want?
  self.validate_field(adata)
/usr/local/lib/python3.10/dist-packages/scvi/data/fields/_base_[field.py:64](http://field.py:64/): UserWarning: adata.X does not contain unnormalized count data. Are you sure this is what you want?
  self.validate_field(adata)

Training transfer model:

Training transfer model:

INFO     Training for 100 epochs.
INFO     Training for 100 epochs.
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]
/usr/local/lib/python3.10/dist-packages/lightning/pytorch/trainer/connectors/data_[connector.py:441](http://connector.py:441/): The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=31` in the `DataLoader` to improve performance.
/usr/local/lib/python3.10/dist-packages/scvi/module/_[scanvae.py:304](http://scanvae.py:304/): UserWarning: The value argument must be within the support of the distribution
  reconst_loss = -px.log_prob(x).sum(-1)
/usr/local/lib/python3.10/dist-packages/scvi/module/_[scanvae.py:304](http://scanvae.py:304/): UserWarning: The value argument must be within the support of the distribution
  reconst_loss = -px.log_prob(x).sum(-1)
/usr/local/lib/python3.10/dist-packages/scvi/module/_[scanvae.py:304](http://scanvae.py:304/): UserWarning: The value argument must be within the support of the distribution
  reconst_loss = -px.log_prob(x).sum(-1)
/usr/local/lib/python3.10/dist-packages/lightning/pytorch/trainer/connectors/logger_connector/[result.py:433](http://result.py:433/): It is recommended to use `self.log('train_loss', ..., sync_dist=True)` when logging on epoch level in distributed setting to accumulate the metric across devices.
[rank0]:[E ProcessGroupNCCL.cpp:523] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=115017, OpType=ALLGATHER, NumelIn=1, NumelOut=2, Timeout(ms)=1800000) ran for 1800068 milliseconds before timing out.
[rank0]:[E ProcessGroupNCCL.cpp:537] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank0]:[E ProcessGroupNCCL.cpp:543] To avoid data inconsistency, we are taking the entire process down.
[rank0]:[E ProcessGroupNCCL.cpp:1182] [Rank 0] NCCL watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=115017, OpType=ALLGATHER, NumelIn=1, NumelOut=2, Timeout(ms)=1800000) ran for 1800068 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:525 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7b7d508f4d87 in /usr/local/lib/python3.10/dist-packages/torch/lib/[libc10.so](http://libc10.so/))
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1e6 (0x7b7d024026e6 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_[cuda.so](http://cuda.so/))
frame #2: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x19d (0x7b7d02405c3d in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_[cuda.so](http://cuda.so/))
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7b7d02406839 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_[cuda.so](http://cuda.so/))
frame #4: <unknown function> + 0xdc253 (0x7b7dafd48253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #5: <unknown function> + 0x94ac3 (0x7b7e01ffdac3 in /lib/x86_64-linux-gnu/[libc.so](http://libc.so/).6)
frame #6: <unknown function> + 0x126a40 (0x7b7e0208fa40 in /lib/x86_64-linux-gnu/[libc.so](http://libc.so/).6)

terminate called after throwing an instance of 'c10::DistBackendError'
  what():  [Rank 0] NCCL watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=115017, OpType=ALLGATHER, NumelIn=1, NumelOut=2, Timeout(ms)=1800000) ran for 1800068 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:525 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7b7d508f4d87 in /usr/local/lib/python3.10/dist-packages/torch/lib/[libc10.so](http://libc10.so/))
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1e6 (0x7b7d024026e6 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_[cuda.so](http://cuda.so/))
frame #2: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x19d (0x7b7d02405c3d in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_[cuda.so](http://cuda.so/))
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7b7d02406839 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_[cuda.so](http://cuda.so/))
frame #4: <unknown function> + 0xdc253 (0x7b7dafd48253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #5: <unknown function> + 0x94ac3 (0x7b7e01ffdac3 in /lib/x86_64-linux-gnu/[libc.so](http://libc.so/).6)
frame #6: <unknown function> + 0x126a40 (0x7b7e0208fa40 in /lib/x86_64-linux-gnu/[libc.so](http://libc.so/).6)

Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1186 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7b7d508f4d87 in /usr/local/lib/python3.10/dist-packages/torch/lib/[libc10.so](http://libc10.so/))
frame #1: <unknown function> + 0xdf6b11 (0x7b7d0215cb11 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_[cuda.so](http://cuda.so/))
frame #2: <unknown function> + 0xdc253 (0x7b7dafd48253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #3: <unknown function> + 0x94ac3 (0x7b7e01ffdac3 in /lib/x86_64-linux-gnu/[libc.so](http://libc.so/).6)
frame #4: <unknown function> + 0x126a40 (0x7b7e0208fa40 in /lib/x86_64-linux-gnu/[libc.so](http://libc.so/).6)

[rank1]:[W socket.cpp:432] [c10d] While waitForInput, poolFD failed with (errno: 0 - Success).
Traceback (most recent call last):
  File "/work/[demo.py](http://demo.py/)", line 8, in <module>
    run(args)
  File "/work/src/[main.py](http://main.py/)", line 40, in run
    pred = annotate(
  File "/work/src/[annotate.py](http://annotate.py/)", line 181, in annotate
    scanvi_model.train(
  File "/usr/local/lib/python3.10/dist-packages/scvi/model/_[scanvi.py](http://scanvi.py/)", line 438, in train
    return runner()
  File "/usr/local/lib/python3.10/dist-packages/scvi/train/_[trainrunner.py](http://trainrunner.py/)", line 98, in __call__
    [self.trainer.fit](http://self.trainer.fit/)([self.training](http://self.training/)_plan, [self.data](http://self.data/)_splitter)
  File "/usr/local/lib/python3.10/dist-packages/scvi/train/_[trainer.py](http://trainer.py/)", line 219, in fit
    super().fit(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/trainer/[trainer.py](http://trainer.py/)", line 544, in fit
    call._call_and_handle_interrupt(
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/trainer/[call.py](http://call.py/)", line 44, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/trainer/[trainer.py](http://trainer.py/)", line 580, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/trainer/[trainer.py](http://trainer.py/)", line 989, in _run
    results = self._run_stage()
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/trainer/[trainer.py](http://trainer.py/)", line 1035, in _run_stage
    [self.fit](http://self.fit/)_[loop.run](http://loop.run/)()
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/loops/fit_[loop.py](http://loop.py/)", line 203, in run
    self.on_advance_end()
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/loops/fit_[loop.py](http://loop.py/)", line 376, in on_advance_end
    trainer._logger_connector.on_epoch_end()
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/trainer/connectors/logger_connector/logger_[connector.py](http://connector.py/)", line 187, in on_epoch_end
    metrics = self.metrics
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/trainer/connectors/logger_connector/logger_[connector.py](http://connector.py/)", line 226, in metrics
    return self.trainer._results.metrics(on_step)
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/trainer/connectors/logger_connector/[result.py](http://result.py/)", line 475, in metrics
    value = self._get_cache(result_metric, on_step)
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/trainer/connectors/logger_connector/[result.py](http://result.py/)", line 439, in _get_cache
    result_metric.compute()
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/trainer/connectors/logger_connector/[result.py](http://result.py/)", line 284, in wrapped_func
    self._computed = compute(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/lightning/pytorch/trainer/connectors/logger_connector/[result.py](http://result.py/)", line 252, in compute
    return self.value.compute()
  File "/usr/local/lib/python3.10/dist-packages/torchmetrics/[metric.py](http://metric.py/)", line 1135, in compute
    val_a = self.metric_a.compute() if isinstance(self.metric_a, Metric) else self.metric_a
  File "/usr/local/lib/python3.10/dist-packages/torchmetrics/[metric.py](http://metric.py/)", line 1135, in compute
    val_a = self.metric_a.compute() if isinstance(self.metric_a, Metric) else self.metric_a
  File "/usr/local/lib/python3.10/dist-packages/torchmetrics/[metric.py](http://metric.py/)", line 610, in wrapped_func
    with self.sync_context(
  File "/usr/lib/python3.10/[contextlib.py](http://contextlib.py/)", line 135, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.10/dist-packages/torchmetrics/[metric.py](http://metric.py/)", line 581, in sync_context
    self.sync(
  File "/usr/local/lib/python3.10/dist-packages/torchmetrics/[metric.py](http://metric.py/)", line 530, in sync
    self._sync_dist(dist_sync_fn, process_group=process_group)
  File "/usr/local/lib/python3.10/dist-packages/torchmetrics/[metric.py](http://metric.py/)", line 434, in _sync_dist
    output_dict = apply_to_collection(
  File "/usr/local/lib/python3.10/dist-packages/lightning_utilities/core/apply_[func.py](http://func.py/)", line 70, in apply_to_collection
    return {k: function(v, *args, **kwargs) for k, v in data.items()}
  File "/usr/local/lib/python3.10/dist-packages/lightning_utilities/core/apply_[func.py](http://func.py/)", line 70, in <dictcomp>
    return {k: function(v, *args, **kwargs) for k, v in data.items()}
  File "/usr/local/lib/python3.10/dist-packages/torchmetrics/utilities/[distributed.py](http://distributed.py/)", line 122, in gather_all_tensors
    return _simple_gather_all_tensors(result, group, world_size)
  File "/usr/local/lib/python3.10/dist-packages/torchmetrics/utilities/[distributed.py](http://distributed.py/)", line 93, in _simple_gather_all_tensors
    torch.distributed.all_gather(gathered_result, result, group)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_[logger.py](http://logger.py/)", line 72, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_[c10d.py](http://c10d.py/)", line 2617, in all_gather
    work = group.allgather([tensor_list], [tensor])
torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '1', but store->get('1') got error: Connection reset by peer
Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:667 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x787d33d50d87 in /usr/local/lib/python3.10/dist-packages/torch/lib/[libc10.so](http://libc10.so/))
frame #1: <unknown function> + 0x589525c (0x787d2041425c in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_[cpu.so](http://cpu.so/))
frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2ee (0x787d2040e92e in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_[cpu.so](http://cpu.so/))
frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x787d2040ece2 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_[cpu.so](http://cpu.so/))
frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x787d2040fb11 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_[cpu.so](http://cpu.so/))
frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x787d203c4f81 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_[cpu.so](http://cpu.so/))
frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x787d203c4f81 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_[cpu.so](http://cpu.so/))
frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x787d203c4f81 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_[cpu.so](http://cpu.so/))
frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x787ce9602c69 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_[cuda.so](http://cuda.so/))
frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector<c10::Device, std::allocator<c10::Device> > const&, c10d::OpType, int, bool) + 0x22b (0x787ce9609c5b in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_[cuda.so](http://cuda.so/))
frame #10: c10d::ProcessGroupNCCL::allgather(std::vector<std::vector<at::Tensor, std::allocator<at::Tensor> >, std::allocator<std::vector<at::Tensor, std::allocator<at::Tensor> > > >&, std::vector<at::Tensor, std::allocator<at::Tensor> >&, c10d::AllgatherOptions const&) + 0xb5c (0x787ce962005c in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_[cuda.so](http://cuda.so/))
frame #11: <unknown function> + 0x583a31d (0x787d203b931d in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_[cpu.so](http://cpu.so/))
frame #12: <unknown function> + 0x5844218 (0x787d203c3218 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_[cpu.so](http://cpu.so/))
frame #13: <unknown function> + 0x4e893cc (0x787d1fa083cc in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_[cpu.so](http://cpu.so/))
frame #14: <unknown function> + 0x1a08a88 (0x787d1c587a88 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_[cpu.so](http://cpu.so/))
frame #15: <unknown function> + 0x584ba33 (0x787d203caa33 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_[cpu.so](http://cpu.so/))
frame #16: <unknown function> + 0x5856e1f (0x787d203d5e1f in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_[cpu.so](http://cpu.so/))
frame #17: <unknown function> + 0xca3fae (0x787d32c5afae in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_[python.so](http://python.so/))
frame #18: <unknown function> + 0x413ea4 (0x787d323caea4 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_[python.so](http://python.so/))
frame #19: <unknown function> + 0x15a10e (0x59f22582b10e in /usr/bin/python3)
frame #20: _PyObject_MakeTpCall + 0x25b (0x59f225821a7b in /usr/bin/python3)
frame #21: <unknown function> + 0x168acb (0x59f225839acb in /usr/bin/python3)
frame #22: _PyEval_EvalFrameDefault + 0x614a (0x59f225819cfa in /usr/bin/python3)
frame #23: _PyFunction_Vectorcall + 0x7c (0x59f22582b9fc in /usr/bin/python3)
frame #24: _PyEval_EvalFrameDefault + 0x2a27 (0x59f2258165d7 in /usr/bin/python3)
frame #25: _PyFunction_Vectorcall + 0x7c (0x59f22582b9fc in /usr/bin/python3)
frame #26: _PyEval_EvalFrameDefault + 0x614a (0x59f225819cfa in /usr/bin/python3)
frame #27: _PyFunction_Vectorcall + 0x7c (0x59f22582b9fc in /usr/bin/python3)
frame #28: _PyEval_EvalFrameDefault + 0x6bd (0x59f22581426d in /usr/bin/python3)
frame #29: _PyFunction_Vectorcall + 0x7c (0x59f22582b9fc in /usr/bin/python3)
frame #30: PyObject_Call + 0x122 (0x59f22583a492 in /usr/bin/python3)
frame #31: _PyEval_EvalFrameDefault + 0x2a27 (0x59f2258165d7 in /usr/bin/python3)
frame #32: _PyFunction_Vectorcall + 0x7c (0x59f22582b9fc in /usr/bin/python3)
frame #33: _PyEval_EvalFrameDefault + 0x6bd (0x59f22581426d in /usr/bin/python3)
frame #34: _PyFunction_Vectorcall + 0x7c (0x59f22582b9fc in /usr/bin/python3)
frame #35: _PyEval_EvalFrameDefault + 0x198c (0x59f22581553c in /usr/bin/python3)
frame #36: <unknown function> + 0x1687f1 (0x59f2258397f1 in /usr/bin/python3)
frame #37: _PyEval_EvalFrameDefault + 0x198c (0x59f22581553c in /usr/bin/python3)
frame #38: <unknown function> + 0x1687f1 (0x59f2258397f1 in /usr/bin/python3)
frame #39: _PyEval_EvalFrameDefault + 0x198c (0x59f22581553c in /usr/bin/python3)
frame #40: <unknown function> + 0x200175 (0x59f2258d1175 in /usr/bin/python3)
frame #41: <unknown function> + 0x15ac59 (0x59f22582bc59 in /usr/bin/python3)
frame #42: _PyEval_EvalFrameDefault + 0x6bd (0x59f22581426d in /usr/bin/python3)
frame #43: <unknown function> + 0x168a51 (0x59f225839a51 in /usr/bin/python3)
frame #44: _PyEval_EvalFrameDefault + 0x266d (0x59f22581621d in /usr/bin/python3)
frame #45: _PyFunction_Vectorcall + 0x7c (0x59f22582b9fc in /usr/bin/python3)
frame #46: _PyEval_EvalFrameDefault + 0x614a (0x59f225819cfa in /usr/bin/python3)
frame #47: <unknown function> + 0x1687f1 (0x59f2258397f1 in /usr/bin/python3)
frame #48: _PyEval_EvalFrameDefault + 0x614a (0x59f225819cfa in /usr/bin/python3)
frame #49: <unknown function> + 0x1687f1 (0x59f2258397f1 in /usr/bin/python3)
frame #50: _PyEval_EvalFrameDefault + 0x614a (0x59f225819cfa in /usr/bin/python3)
frame #51: <unknown function> + 0x168a51 (0x59f225839a51 in /usr/bin/python3)
frame #52: _PyEval_EvalFrameDefault + 0x2a27 (0x59f2258165d7 in /usr/bin/python3)
frame #53: _PyFunction_Vectorcall + 0x7c (0x59f22582b9fc in /usr/bin/python3)
frame #54: _PyEval_EvalFrameDefault + 0x614a (0x59f225819cfa in /usr/bin/python3)
frame #55: _PyFunction_Vectorcall + 0x7c (0x59f22582b9fc in /usr/bin/python3)
frame #56: _PyEval_EvalFrameDefault + 0x614a (0x59f225819cfa in /usr/bin/python3)
frame #57: _PyFunction_Vectorcall + 0x7c (0x59f22582b9fc in /usr/bin/python3)
frame #58: _PyEval_EvalFrameDefault + 0x8ac (0x59f22581445c in /usr/bin/python3)
frame #59: <unknown function> + 0x16600e (0x59f22583700e in /usr/bin/python3)
frame #60: _PyObject_GenericGetAttrWithDict + 0x468 (0x59f2258298a8 in /usr/bin/python3)
frame #61: PyObject_GetAttr + 0x4d (0x59f225827e3d in /usr/bin/python3)
frame #62: _PyEval_EvalFrameDefault + 0x5dc1 (0x59f225819971 in /usr/bin/python3)
frame #63: _PyFunction_Vectorcall + 0x7c (0x59f22582b9fc in /usr/bin/python3)
. This may indicate a possible application crash on rank 0 or a network set up issue.
Aborted (core dumped)

It looks like the issue is referring to a timeout in the NCCL library. Could you paste the installed versions of PyTorch, Lightning, and NCCL? Also, are any training steps being run or is it crashing right from the start (I see that it doesn't go past the first training epoch)?

installed versions:
lightning - 2.1.4
pytorch-lightning - 2.2.1
nvidia-nccl-cu12 - 2.19.3
torch - 2.2.1
Yes, please see in the (more complete) log attached below, base training succeeds, it's the transfer training after that crashes.


TypeError: cannot pickle 'generator' object

code:
from rocksdict import Rdict

path = str("./test_dict")
db = Rdict(path)
def f(a, b):
    for n in range(a):
        yield b + b[-1] * n

gen = f(4, 'ay')
x = next(gen)
db['gx'] = gen

error:
TypeError: cannot pickle 'generator' object
thinking:
It should be the cause of pickling, but it doesn't fit the readme description of being able to store anything

Default mode, which allows storing int, float, bool, str, bytes, and other python objects.

Right, only python objects that can be pickled. But you might also use raw_mode to store objects in custom encoding.

in history commit :value: int, float, bool, str, bytes and anything that supports pickle.

i will try,thx

Updated the readme, adding (with Pickle) to clarify this problem.


Rendering preview.html

The PostPreview shows a preview of a post's body using the post.preview tag, see here. This works fine when the markdown file does not specify a preview text (as the example posts do), as then the first <p> element is used (see here, but for me it does not work when a preview text is specified, as this line stringifies the preview. Thus, for me using asterisks for bold ends up shown as **this**. How can I correctly render the preview markdown?

Hm, yeah, that was a feature added here. Seems we never considered parsing that text as markdown.
You could manually run remark (or some markdown formatter) on the preview metadata in posts.js, and pass that down instead.

No problem, thanks for the explanation!


[WIP] Vuex functionalities for explorer

Signed-off-by: Karoly Albert Szabo<EMAIL_ADDRESS>Closes #1865
Description:

Add blocks in append only
action to reset blocks
add peers requested via RPC

❤ Thank you!

[ ] Added entries in CHANGELOG.md with issue # and GitHub username
[ ] Reviewed Files changed in the github PR explorer

can we call the module blocks.js instead of blockchain.js? i think this will be more self evident.

i think if you remove the cache and the peers then we can merge your blocks PR! unless you think we should keep both, then that’s ok too - just remove the commented out code.


new package: emacs27-git

Description of package
I am excited about this, with the new pdumper people with bloated configs will still be able to load fast, in addition we can now enable json features which will improve performance with packages that use it, such as the up and coming lsp-mode.
Checklist

[ ] Latest upstream version.
[ ] I agree to maintain this package (required).
[ ] I agree to notify KISS if I can no longer maintain this package (required).

@dylanaraps I can't get it to checkout the emacs-27 branch, which clearly exists (one can see it by looking online in a web browser if desired...)
-> emacs27-git Extracting sources
-> Checking out emacs-27
fatal: ambiguous argument 'emacs-27': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
!> Commit hash emacs-27 doesn't exist
bash-5.0$

Pushed a package manager update which fixes this. :+1:


Blitz.JS Deployment: TypeError: Cannot read property '$create' of undefined

I have successfully deployed BlitzJS to AWS. However, when I try to login or signup an account I get an Error:

I think this issue is related to https://github.com/blitz-js/blitz/issues/2955
I don't wrap the config like in the issue in 2955, so I don't get why it's not working correctly. BlitzJS is supported with this module, right?
My blitz.config.ts looks like this:
import { sessionMiddleware, simpleRolesIsAuthorized } from "blitz"
//const withPWA = require("next-pwa")

const config = {
  middleware: [
    sessionMiddleware({
      cookiePrefix: "wardrobe",
      isAuthorized: simpleRolesIsAuthorized,
    }),
  ],
 // pwa: {
  //  dest: "public",
  //  swSrc: "service-worker.js",
//    sw: "service-worker.js",
//    reloadOnOnline: true,
//    cacheOnFrontEndNav: true,
    //disable: process.env.NODE_ENV === "development",
//  },
}

module.exports = config

Maybe it has something to do with the next-tf package. I found in "terraform-aws-next-js/packages/tf-next/src/utils/index.ts" that its parsing for next.config.js not blitz.config.js. Maybe because of that blitz can not wrap around next?
Has anybody an idea how I could fix it? Would really appreciate if somebody know a solution!
Note:

Yes, I specifically disabled the wrapping with next-pwa and it doesn't work.
I am building on EC2 instance via Cloud9 on Amazon Linux:

In theory the module should work with Blitz.js.
I am currently not up-to-date with the build process Blitz uses.
My latest info on that is that Blitz can detect if it is built inside a Vercel-like environment and outputs a next.config.js instead of a blitz.config.js.
tf-next itself currently creates a wrapper config to ensure that some configuration is set for deploying for the serverless target.
We are in the process to remove this wrapper in #89.
Work on it is almost finished, but needs some final touches.


[ios] platform_views_layout crashes on subsequent runs

Here's a sample log:
platform_channel.dart:161)
stdout: [   +3 ms] <asynchronous suspension>
stdout: [        ] #2      PlatformViewsService.initUiKitView (package:flutter/src/services/platform_views.dart:219)
stdout: [        ] <asynchronous suspension>
stdout: [        ] #3      _UiKitViewState._createNewUiKitView (package:flutter/src/widgets/platform_view.dart:624)
stdout: [        ] <asynchronous suspension>
stdout: [   +1 ms] [VERBOSE-2:ui_dart_state.cc(166)] Unhandled Exception: PlatformException(create_failed, can't create a view on a headless engine, null)
stdout: [        ] #0      StandardMethodCodec.decodeEnvelope (package:flutter/src/services/message_codecs.dart:572)
stdout: [        ] #1      MethodChannel._invokeMethod (package:flutter/src/services/platform_channel.dart:161)
stdout: [        ] <asynchronous suspension>
stdout: [        ] #2      PlatformViewsService.initUiKitView (package:flutter/src/services/platform_views.dart:219)
stdout: [        ] <asynchronous suspension>

2020-07-15T14:52:19.045273: INFO [task runner] [platform_views_scroll_perf_ios__timeline_summary] stdout: [        ] #3      _UiKitViewState._createNewUiKitView (package:flutter/src/widgets/platform_view.dart:624)
stdout: [        ] <asynchronous suspension>
stdout: [        ] [VERBOSE-2:ui_dart_state.cc(166)] Unhandled Exception: PlatformException(create_failed, can't create a view on a headless engine, null)
stdout: [        ] #0      StandardMethodCodec.decodeEnvelope (package:flutter/src/services/message_codecs.dart:572)
stdout: [        ] #1      MethodChannel._invokeMethod (package:flutter/src/services/platform_channel.dart:161)
stdout: [        ] <asynchronous suspension>
stdout: [        ] #2      PlatformViewsService.initUiKitView (package:flutter/src/services/platform_views.dart:219)
stdout: [        ] <asynchronous suspension>
stdout: [        ] #3      _UiKitViewState._createNewUiKitView (package:flutter/src/widgets/platform_view.dart:624)
stdout: [        ] <asynchronous suspension>

fwiw, the framework fires a message to create the native view, which is handled in https://github.com/flutter/engine/blob/24d2143b98d6b0dfffd52eaad60d3900ea6a5190/shell/platform/darwin/ios/framework/Source/FlutterPlatformViews.mm#L124.
cc @dnfield, @cyanglaz

I'm able to reproduce this locally with a profile build of platform_views_layout. Every so often running it results in failure.  Escalating priority until we determine cause.

Root cause is https://github.com/flutter/engine/pull/19592

Steps to recreate: flutter run --profile in platform_views_layout, exit the app, run it a second time.

this also affects 1.20 as 0dc86cda19d55aba8b719231c11306eeaca4b798 was our engine branch point

this will require a manual merge into flutter-1.20-candidate.7

for posterity the 1.20 cherrypick is https://github.com/flutter/engine/pull/19848


Update documentation_tips.md

Status
Ready/In Progress/In Hold (Reason for hold)
Related Issues
fixes: link to the issue
Description
A few sentences describing the overall goals of the pull request's commits.
Screenshots
Paste here any images that will help the reviewer

@julieschwartz18 Hopefully one more time. Can you approve the graphic? Thanks


Error : gzip: stdin: not in gzip format

Hello,
I have an error when i run this command :MQ_ARCHIVE=IBM_MQ_<IP_ADDRESS>_LINUX_X86-64_TRIAL.tar.gz make build-advancedserver
The output is :

false
DIR_EXTRACT=/tmp/mq
mkdir -p /tmp/mq
cd /tmp/mq
curl -LO http://build:80/IBM_MQ_<IP_ADDRESS>_LINUX_X86-64_TRIAL.tar.gz
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Dload  Upload   Total   Spent    Left  Speed
100  3892  100  3892    0     0   364k      0 --:--:-- --:--:-- --:--:--  380k
tar -zxvf ./IBM_MQ_<IP_ADDRESS>_LINUX_X86-64_TRIAL.tar.gz

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
The command '/bin/sh -c chmod u+x /usr/local/bin/install-mq.sh   && sleep 1   && install-mq.sh' returned a non-zero code: 2
build-server
build
Thanks for your help.

Outside the container i can execute tar -zxvf ./IBM_MQ_<IP_ADDRESS>_LINUX_X86-64_TRIAL.tar.gz successfully inside downloads directory. So i suspect that the curl -LO http://build:80/IBM_MQ_<IP_ADDRESS>_LINUX_X86-64_TRIAL.tar.gz do not download the entire file.
Thanks for your help.

I have the exact same issue. Looks like the issue is with the temporarily create network and/or the webserver that's supposed to host the tar.gz.

Thanks for your response.

Hi @faissalwise,
When we see the error gzip: stdin: not in gzip format it means that the curl command has failed to download the archive file from the temporary nginx http server we have started. The most common cause of this is that the archive file in the downloads directory does not have world read permissions on it so the curl command cannot download it. Please check that the permissions for this archive file allow anyone to read it.
On a side note, if i remember correctly the 9 LTS archive you are using does not contain debian installation files but instead contains rpm install files. This will be a problem later on when the install script attempts to install MQ as it will try and do it using apt install which will not work with rpm install files. A possible workaround would be to change the base image from ubuntu:16.04 to centos:latest which will cause the install scripts to use rpm to install instead.

Hi @parrobe ,
It's resolved,my problem was the proxy in the curl command,so i use curl with noproxy option and that works.
Thanks.

@faissalwise
What you have done in local to resolve this issue ?
Other than changing the base image from ubuntu:16.04 to centos:latest
I am having the same issue still

Can you please clarify the name of the tar.go file you are using, and that you've checked it's downloaded correctly (is really a valid file)?


Replace leaktest with goleak

As Prometheus did we probably want to replace leaktest with https://github.com/uber-go/goleak with per package pattern:
func TestMain(m *testing.M) {
	goleak.VerifyTestMain(m)
}

This way we don't need to put defer on every unit test which we often forget. It will harder to detect WHERE the leak happens but for CI we care more if it happens.

Help wanted!

hi, i'd like to work on this

Please help! :heart:


Proposal: Expanding Interface Function Signature Matching

Shortly
Rather than interfaces matching strictly on the function signature, they would also consider if a signature's arguments, or returns implement an interface for a match.
Or in other words, make this code example compile and work:
package main

import (
	"fmt"
	"io/ioutil"
)

type SomeReader struct {
}

func (sr *SomeReader) Read(p []byte) (in int, err *ErrorSomeReader) {
     return 0, &ErrorSomeReader{}
}

type ErrorSomeReader struct {
}

func (esr *ErrorSomeReader) Error() string {
     return "SomeReaderError"
}

func main() {
       sr := &SomeReader{}
        _, err := ioutil.ReadAll(sr)
	fmt.Println("Error", err)
}

In today's go an interface matches a function only if the full signature matches exactly. So the above fails with the error:
cannot use sr (type *SomeReader) as type io.Reader in argument to ioutil.ReadAll:
	*SomeReader does not implement io.Reader (wrong type for Read method)
		have Read([]byte) (int, *ErrorSomeReader)
		want Read([]byte) (int, error)

However, *ErrorSomeReader implements error and thus *SomeReader seems like it should be an io.Reader, but is not.
This example is usefull to understand how this would work, but not why it should be added.
The Rationale
The real reason to add this arises when you have interfaces defined relative to other interfaces. Consider some code for doing verification of access with users and groups:
package verify

type User interface {
	Group() Group
}

type Group interface {
	Name() string
	PermittedPaths() []string
}

func VerifyAccess(u User, path string) (string, bool) {
	for _, g := range u.Groups() {
		for _, p := range g.PermittedPaths() {
			if p == path {
				return p.Name(), true
			}
		}
	}
	return "", false
}

Because verify.User is defined relative to verify.Group, any use of  package verify must also now define itself with a direct tie into verify. Even when a verify.User isn't what they want. Callers probably have a much richer defintion of the group returned by a user. This breaks one of the nicest things about Go's interfaces: that packages can simply define what they need of an object without placing restrictions on a caller.
With this change any type with a method Group() returning any type with the methods Name() string, PermittedPaths() []string can be used by verify.VerifyAccess. A user of this package does not need to define it's almost undoubtedly richer user and group with the restricted view required by package verify.
With this change packages ask for behaviors of a more complicated nature without tying implementers to a limited view of the world. Thus we can write code which captures more generic and common sets of operations. Code becomes automatically more composable (in the functional sense) as any object which implements desired behaviors is permitted. And we don't loose any type safety. Finally it may encourage returning more "bare" types and less spurious interfaces.
This isn't totally new in realm of language design either. I believe Hakell's TypeClasses allow similar expressions.*
*I am not a Haskell expert by any stretch of the imagination.
The Downsides
Just some obvious ones:

It is less immediately obvious if a type implements an interface.
This might be difficult to understand? I certainly had a bit of trouble reaching a succinct explanation of it.
Compiler complexity considerations. I have no idea how hard this would be to actually implement.

Timeline
This would obviously probably fall into a Go 2.0, though such a change would explicitly not break any currently compiling code.

You need to discuss implementation.  Quite apart from whether this change is desirable or not, it is difficult to implement.

It's easy to implement if we explicitly label implemented interface
for each type, but that's not how Go works.
You need to consider how to generate wrapper functions for yet
unknown interface methods.

Changing the type system is out of scope for Go 1, and we're not accepting proposals for a hypothetical Go 2 yet.
I've flagged this for Go 2 (many people have type system desires), but there are no plans for a Go 2 at the current time.
As this is a proposal, I'll let @adg handle resolution of this issue.

@ianlancetaylor You need to discuss implementation. Quite apart from whether this change is desirable or not, it is difficult to implement.

@minux It's easy to implement if we explicitly label implemented interface
for each type, but that's not how Go works.
You need to consider how to generate wrapper functions for yet
unknown interface methods.

A Naive Implementation
A naive and trivial implementation is to do this the same way we would as software engineers, an adapter pattern. We might write an adapter this way in go:
package main

import (
	"fmt"
	"io/ioutil"
)

type SomeReader struct {
}

func (sr *SomeReader) Read(p []byte) (n int, err *ErrorSomeReader) {
	return 0, &ErrorSomeReader{}
}

type ErrorSomeReader struct {
}

func (esr *ErrorSomeReader) Error() string {
	return "SomeReaderError"
}

type SomeReaderAdapter struct {
	SomeReader *SomeReader
}

func (sra SomeReaderAdapter) Read(p []byte) (n int, err error) {
	return sra.SomeReader.Read(p)
}

func main() {
	sr := &SomeReader{}
	_, err := ioutil.ReadAll(SomeReaderAdapter{sr})
	fmt.Println("Error", err)
}

If instead the compiler generated the adapting types for us a lot of "boiler plate" for this would simply disappear. Technically the act of generating these adapters isn't that difficult. Finding a truly efficient implementation may be.

@bradfitz Changing the type system is out of scope for Go 1, and we're not accepting proposals for a hypothetical Go 2 yet.

Another way to view this is not a fundamental change to the type system, but the automated generation of adapters. And as I mentioned, this does not break compatibility with Go1 code (unlike most Go2 suggestions). I of course realize that it is a larger change then Go1 has done to date, and could be treated simply "too large".

How could the compiler statically determine the set of "adapters" (we
call them "wrapper" methods) to generate?
For example, if *ErrorSomeReader implements N methods, there will
be 2^N interfaces that it could satisfy, and note that the user can type
assert any such interface to another at runtime and in general, the compiler
can't determine which interfaces are needed, so the wrappers can't be
statically generated.

There are 2^N interfaces a type could satisfy, but there are a fixed number of interfaces in a system. In a given codebase you don't have to consider all possible interfaces, just the interfaces defined in the system.  At most you have to generate | T | * | I | wrappers, with T being the set of types and I the set of interfaces. For the entire go source code base there are ~2500 interfaces, so its practical to brute force search and generate that space (but certainly not efficient).
I admit I hadn't considered the runtime type assertion aspect since I avoid it like the plague. As I mentioned above its not impossible to generate a wrapper for every interface in a system a type could satisfy just inefficient. However, the same code/methodology can be used to generate those warper functions/types at runtime that are type asserted instead of assigned to yes?

How could the compiler (when compiling a single package) know the full
set of interface used in a system?

Officially leaving this open as a "Go2" feature request, as it has come up before.
It is not a suitable change for Go1.
--
On a personal note, I am not inspired by the examples in this proposal. In particular, the example showing returning an error is a classic example of bad style (you should always return error values, not values of concrete types that implement error). This weakens the proposal as a whole.

@adg On a personal note, I am not inspired by the examples in this proposal. In particular, the example showing returning an error is a classic example of bad style (you should always return error values, not values of concrete types that implement error). This weakens the proposal as a whole.

That is a good point, perhaps I should alter the example not to use error. It was simply a very simple illustration of the idea. That said, the standard library seems to strongly prefer baretypes for everything but error, and as I mentioned this would encourage that practice.
Here is another, perhaps more compelling example based how we use container/heap today.
package main

import (
	"container/heap"
	"fmt"
)

// An IntHeap is a min-heap of ints.
type IntHeap []int

func (h IntHeap) Len() int            { return len(h) }
func (h IntHeap) Less(i, j int) bool  { return h[i] < h[j] }
func (h IntHeap) Swap(i, j int)       { h[i], h[j] = h[j], h[i] }
func (h *IntHeap) Push(x interface{}) { *h = append(*h, x.(int)) }
func (h *IntHeap) Pop() interface{} {
	old := *h
	n := len(old)
	x := old[n-1]
	*h = old[0 : n-1]
	return x
}

func main() {
	var h IntHeap
	heap.Init(h)
	heap.Push(h, 1)
	heap.Push(h, 2)
	fmt.Println("Hello world: ", heap.Pop(h))

	// TYPE SAFTEY HAS ABANDONED ME!
	heap.Push(h, "1")
	fmt.Println("Hello world: ", heap.Pop(h))
}

YIKES! While a viable implementation all of our beautiful type safety just went out the window.
Now compare this with changing the interface matching.
package main

import (
	"container/heap"
	"fmt"
)

// An IntHeap is a min-heap of ints.
type IntHeap []int

func (h IntHeap) Len() int           { return len(h) }
func (h IntHeap) Less(i, j int) bool { return h[i] < h[j] }
func (h IntHeap) Swap(i, j int)      { h[i], h[j] = h[j], h[i] }
func (h *IntHeap) Push(x int)        { *h = append(*h, x) }
func (h *IntHeap) Pop() int {
	old := *h
	n := len(old)
	x := old[n-1]
	*h = old[0 : n-1]
	return x
}

func main() {
	var h IntHeap
	heap.Init(h)
	heap.Push(h, 1)
	heap.Push(h, 2)
	fmt.Println("Hello world: ", heap.Pop(h))

	// NO PROBLEM, WON'T COMPILE!
	// But since int is an interface{}, we can compile the this once the below line is commented out.
	heap.Push(h, "1")
	fmt.Println("Hello world: ", heap.Pop(h))
}

I will get to answering minux's questions asap.

Even if we somehow implements this proposal,
I don't think heap.Push(h, "1") will be flagged as
a compile error (because heap.Push takes an
interface{} as the 2nd argument.)
Now I realize the implementation difficulty is
even bigger than I previously described. Because
the method could take multiple arguments and
return multiple arguments, each one is subject to
interface conversion! Even if we could devise
a way to get around the problem of generating
all the wrapper for each interface conversion
(we need at least two, eface and iface case), for
N-argument methods, we still need 2^N wrapper
functions.

On a procedural note, I don't want to use the issue tracker and the proposal process to plan Go 2. We have enough work keeping up with Go 1, and we haven't even begun to think about considering possibly planning Go 2. I am closing this issue so that it does not appear in pending proposals, but I'm leaving the Go2 and Proposal labels so we can find the issue if we ever do get to Go 2.
As for the actual content of the proposal, Ian noted that it is difficult to implement. But on top of that, it has a serious semantic problem. Andrew noticed the concrete problem with the error example, but it's not just a style nit.
Let's suppose we have:
type I interface {
	F(X) Y
}

type T whatever

func (T) F(X1) Y1 { ... }

var t T
var x X
var i I = t // proposal magic happens here
var y Y = i.F(x) // or maybe here

It must be that those last two lines of code would behave as the one line:
var y Y = Y(t.F(X1(x)))

I've written the conversions from X to X1 and from Y1 to Y explicitly but of course Go does not require them to be written in code like this. The conversions imply that X1 and Y must be interface types. X and Y1 may or may not be.
Suppose Y is interface{} and Y1 is int and t.F returns 0. Then y ends up being interface{}(0), which seems about right.
Suppose Y is error and Y1 is *MyError and t.F returns (*MyError)(nil). Then y ends up being a non-nil error containing (*MyError)(nil). (See the FAQ entry Andrew mentioned if this does not ring alarm bells.) So when T satisfies I in this case, it cannot return a successful result. It always returns an error. One could of course introduce an exception for typed nil pointers converting to interface{}, but why is the type lost in that one case? Why is a zero pointer different from a zero int? And we surely can't change the way existing interface conversions work. Why is the magic due to interface conversion different from an explicit interface conversion?
The same narrowing of possibility happens for the arguments, and again it might be a problem. If the interface says F(*MyError) but T's method is F(error), again you have the problem that F may need to be called with a nil error and cannot be. It's less of a concern in the argument position perhaps. But it is still likel a real problem.
This issue with nil does not happen if you require that all of X, X1, Y, Y1 are interface types. Then the conversion from nil of one interface type to nil of another interface type works as expected. But that likely restricts the utility of the change. It does not allow the Read example.
I think this is the showstopper.
But then I think there are also other semantic issues and circularities in even deciding whether one recursively defined interface is to be considered to implement a second one. I'm not going to say more about this problem but I think it's there.


Lions and Tigers and Multi-User OH MY!

Description
Adds Multi User support.
PR for visibility only, lots still to do.
Any person who DM's me asking for a release date gets blocked.
Fixes # Various
Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[x] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[x] This change requires a documentation update

How Has This Been Tested?

Checklist:

[ ] My code follows the style guidelines of this project
[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have updated changelog.md with changes made

+2,011 −1,156 in 21 files

ok good luck buddy

+2,011 −1,156 in 21 files

ok good luck buddy

Most of that is because i rejigged api.js .. gotta get dem stats up somehow

@pagermon/collaborators I think it's functionally done...
Lots of code tidying to do still I think, but if you could review and tell me what you think is broken/leaving the barn door open/monumentally stupid, I would appreciate it.
Anyone else with opinions on code and not features, feel free to leave suggestions 😁

Capcode column doesn't hide properly when hidecapcode is on when logged in as a normal user.

Sockets are also messed up, AdminSocket is opening for standard users

Sockets are also messed up, AdminSocket is opening for standard users

This fixed

Capcode column doesn't hide properly when hidecapcode is on when logged in as a normal user.

This is also fixed

,🍆🍆🍆🍆🍆

Still need to

Add brute force protection to login route
Adjust brute force time out for dupe routes
username minumum and maximum length validation
password complexity? Minimum length validation

Then I think it's almost ready..

* password complexity? Minimum length validation

Complexity bad, min length good

* password complexity? Minimum length validation

Complexity bad, min length good

👍 >6?

Sure why not 😁

@pagermon/collaborators I'm out of things i'd like for the initial release.
Review plz

Ok i really think i'm finished now, i've had this running on a test instance for a few days as my primary without any issues so far. @pagermon/collaborators


upgrade to 0.18.0 release

I tried to update the module for the upcoming 0.18 release.
It appears that Json.customDecoder was removed, and got stuck. Would you mind taking a look?
I believe everything else works.

I found a solution on the forums. The Demo pages work with 0.18 now.

Hey! thanks so much!
Could you actually make this PR for elm-menu?
I'm working today on introducing an API that's more drop in. But the current API that you just updated to 0.18 will be available in elm-menu 😄

To clarify, elm-menu will be a dependency of elm-autocomplete and elm-autocomplete will become the AccessibleExample without subscriptions

actually, decided to merge this and release so people can work with the existing API while I work on the new one! Thanks for the contribution!


bugfix: verify otp not working

Veritfy otp api not working for verifying different emails/phones

Currently not working on this issue actively, putting on hold.


Plugin cannot work with Error

There are two errors when the editor loaded:

And I cannot get module when I use getModule() function, it returns an undefined

@hanerx If you use quill-better-table with Vue-quill-editor ?
quill-better-table relies on quilljs v2.0.0-dev.x, not a released version, but vue-quill-editor  relies on quilljs 1.3.6.
So you can not use quill-better-table with vue-quill-editor.
Sorry about this.


Update cri plugin to v1.0.0.

Update cri to v1.0.0.
cri v1.0.0 release note https://github.com/containerd/cri/releases/tag/v1.0.0.
Signed-off-by: Lantao Liu<EMAIL_ADDRESS>

Codecov Report

Merging #2304 into master will decrease coverage by 4.53%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2304      +/-   ##
==========================================
- Coverage   50.07%   45.54%   -4.54%
==========================================
  Files          75       83       +8
  Lines        7269     9185    +1916
==========================================
+ Hits         3640     4183     +543
- Misses       2956     4326    +1370
- Partials      673      676       +3

Flag
Coverage Δ

#linux
50.07% <ø> (ø)
:arrow_up:

#windows
41.24% <ø> (?)

Impacted Files
Coverage Δ

oci/spec.go
40% <0%> (-10%)
:arrow_down:

snapshots/native/native.go
43.89% <0%> (-10%)
:arrow_down:

metadata/snapshot.go
45.8% <0%> (-8.96%)
:arrow_down:

archive/compression/compression.go
43.93% <0%> (-8.9%)
:arrow_down:

remotes/docker/fetcher.go
41.42% <0%> (-7.6%)
:arrow_down:

content/local/writer.go
52.63% <0%> (-7.37%)
:arrow_down:

archive/tar.go
43.05% <0%> (-6.95%)
:arrow_down:

metadata/containers.go
47.97% <0%> (-6.62%)
:arrow_down:

metadata/buckets.go
54.66% <0%> (-5.04%)
:arrow_down:

metadata/images.go
58.46% <0%> (-4.7%)
:arrow_down:

... and 53 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 321c52a...9c9abec. Read the comment docs.


Handy shell script to build all versions of the plugin, and optionally publish to JetBrains.

@long1eu you have some free time to do it? shouldn't be a lot of work with a shell script...

I don't think we need to automate it right now. Just to keep in mind to update it on a new build.


perception_open3d: 0.1.1-1 in 'rolling/distribution.yaml' [bloom]

Increasing version of package(s) in repository perception_open3d to 0.1.1-1:

upstream repository: https://github.com/ros-perception/perception_open3d.git
release repository: https://github.com/ros-gbp/perception_open3d-release.git
distro file: rolling/distribution.yaml
bloom version: 0.11.1
previous version for package: null

Blacklist for RHEL added due to lack of open3d key https://github.com/ros2/ros_buildfarm_config/pull/244

As for versions, it may be better to make the rolling one greater than the humble one.


Option to white-list components

Would be cool to allow an option to white-list certain component names, as ^3.0.0 only applies to native html elements. In my case, allowing <Link> would be great.

Related discussion: https://github.com/insin/babel-plugin-react-html-attrs/issues/12


spell check indicator hidden on active line

When you have the cursor on the line, the spell check indicator gets hidden... Seems to only happen on the standard syntax as both Dark and Light do not have this issue (see below screenshots)

Details:
Mac: 10.12.3
Atom: 1.15.0

Can also confirm

Details:
macOS: 10.12.4
Atom: 1.16.0 x64
atom-material-ui: 1.3.10
atom-material-syntax: 1.0.3


Mapping of CSL container-title from CFF

@sdruskat @hainesr I am probably missing something obvious, but how do you describe in CFF what can be mapped to container-title in CSL? For DataCite DOIs describing software we use the repository name (e.g. Zenodo), but it could also be GitHub.  I don't see anything in the examples.
In Codemeta and DataCite XML we map publisher to container-title, but in CFF I see publisher only under references.
And also consider that for dataset the repository name, or GitHub would be the expected behaviour.

I suspect one for @sdruskat and @jspaaks!
There is the repository field, but that is a URI to an artefact in a repository, not the name of a repository, and so I suspect not what you mean @mfenner.

I think this is probably actually a question for the main CFF repo, so I'll transfer it there.


Add ability to skip aggressive vscode.Uri encoding

Adds a field in the FileDownloadSettings to disable aggressive encoding for the url.toString call on line 118 of FileDownloader.ts.

@microsoft-github-policy-service agree


Update on coordinate transform between cubed sphere panels

summary
-add vs_velocity_rotation.hpp
Defined transformation of velocity from panel 1 into neighboring panels and back.
-add test_vs_velocity_rotation.cpp
Test cases for the transformation, ensure that all inverses are valid.

OK to merge PR.


[DNM] Send message to Slack on failed deploy

Context

Changes proposed in this pull request

Guidance to review

Link to Trello card

Checklist

[ ] Attach to Trello card
[ ] Rebased main
[ ] Cleaned commit history
[ ] Tested by running locally

Superseded by this PR: https://github.com/DFE-Digital/find-a-lost-trn/pull/1048


No response from /deleteprofile

I know it's some where on my end, but all of a sudden all my bots (including new and fresh with 0.10.2) are not reposting nor doing anything for /deleteprofile. This applies to working with BotConnector platform as well as with Emulator.

Emulator doesn't support /deleteprofile command, only connector supports /deleteprofile command.  It works for the test bots that we have. Can you email, your bot ID and the Microsoft ID associated with your bot to<EMAIL_ADDRESS>so we can debug the issue?

done, thanks

Can you share your code? The blank replies imply that you're call session.endDialog() somewhere without sending a message

Before I share the code, it acts the same on examples as well. I do not have any endDialog() without a string.
Simple HelloWorld does same thing

@mikkhait  we did a deployment today with a fix for /deleteprofile. Can you see if it start working for your bot again on production?

Closing this issue. Feel free to reopen if you still see the problem.


List of , need input from other users

These are the domains I found we need to add.
gamingonlinux.com
freebsd.org
zoho.com
protonmail.com
@epsilon-0 looks like you added fastmail? yes no?

What's about gmail.com; outlook.com...

They are already in there, I only listed ones we might want to add to the current list. Here's the list we got so far not including the ones I listed above.
aliexpress.com
amazon.com
ebay.com
github.com
google.com
icloud.com
kijiji.com
linkedin.com
mandrillapp.com
mailchimp.com
microsoft.com
outlook.com
paypal.com
sendgrid.net
t-mobile.com
verizon.com
yahoo.com
intuit.com
facebookmail.com
messagingengine.com
freebsd.org
zoho.com
gamingonlinux.com

outlook.fr


Just a suggestion

Since there's already compatibility with the grandpa demon mod, you should add the grandpa demon difficulties.

Will be added in a later update


DropLevel for event items

Affected event items:

Blood Bone
ArchangelScroll
Devils Key
Devils Eye
Old Scroll
Illusion Sorcerer Covenant

A few issues:

Currently, EventItems are initialized without a drop level ( https://github.com/MUnique/OpenMU/blob/master/src/Persistence/Initialization/Items/EventTicketItems.cs )
Event Items should never exist with item level 0, but currently they drop
Event Items must be dropped for higher levels than currently level 4
The level spacing between even item drop levels are much bigger (see example below) than for normal items (every 3 monster levels is one item level)

Example for Blood Castle event items
(Source: https://wiki.infinitymu.net/index.php?title=Blood_Castle_Event )
Who drop those items?
Items | Drop level | from Monsters
Scroll of Archangel +1 & Blood Bone +1 | Over LV. 2 | Spider
Scroll of Archangel +2 & Blood Bone +2 | Over LV. 32 | Ghost
Scroll of Archangel +3 & Blood Bone +3 | Over LV. 45 | Vepare
Scroll of Archangel +4 & Blood Bone +4 | Over LV. 57 | Death Cow
Scroll of Archangel +5 & Blood Bone +5 | Over LV. 68 | Silver Valkyrie
Scroll of Archangel +6 & Blood Bone +6 | Over LV. 76 | Bloody Wolf
Scroll of Archangel +7 & Blood Bone +7 | Over LV. 85 | Vim Knight

How to fix:

Extension of the ItemDefinition to be able to define a drop level per item level, e.g. as collection.
Adapting the logic in ItemDropGenerator: When such entries exist for an item, the default logic should not be applied, but these entries must be applied.

Related issues: #23, #146

I think this can be fixed in another way:
In the quest branch, I had to extend the DropItemGroup by a min and max drop level (https://github.com/MUnique/OpenMU/commit/0bcfd7c30756e70c935e6ad96f6af3b43873fda6). If we build the fix on it, we could simply add a property of "ItemLevel" and define DropItemGroups for the event ticket items. Additionally, the "DropsFromMonsters" must be changed to false for these items. "DropsFromMonsters" defines, if the item is dropping automatically by the standard random item drop logic. Maybe a rename would make sense, too.


Bug/questionnaire labels

Bug-fixes on the questionnaires to display labels/fields correctly
Includes:

Disabling "Save & Next" button until a form is valid (all required fields are filled in)
Fix hide statement to display follow-up question about partner centers on Evaluation History questionnaire
Fix labels on Current Behaviors and Education questionnaires
Add Text and "Choose not to disclose" options on Clinical Diagnosis questionnaire
(view following comments to see more)

Fixes issue #65

Addresses issues from #67
**Contact:**

How did you hear about us question: option that appears when you select Other – please change prompt in this cell to “please specify how you heard about us”
Evaluation history self and dependent:

Reword “do you self-identify as having autism” to “Do you have autism?” and move it to follow “have you been formally diagnosed with autism?”
-Add a box “none of the above” as response option for question “Have you ever been evaluated at any of the following centers?”

Change “how old was your child” to “how old (in years) was your child”

Add a box “none of the above” as response option for question “Have you ever been evaluated at any of the following centers?”
Education:

Yes automatically selected on Q1 (do you attend an academic program),
Can you change this so neither answer option is selected?

Add “none of the above” checkbox under services currently received through academic program

Under “School Services” change “Please check the following” (Q4) to “please check all school services that you are currently receiving”

Add “Home schooled” under current school/grade placement
Employment:

For “Do you currently receive employment supports” question: add “no but I am interested” option
Supports:

Answer options should be changed to “currently using” (not receiving) “used in past” “interested in using”

More issues from #67
General:

Put the “Get Started Now” blue button at the TOP and the bottom of the first page in joining the registry

Home:

change “who else lives there?” to “who else lives with X?” (Did "who lives with your child?" because the housemate model doesn't know the preferred name)

Current Behaviors:

Add “I choose not to disclose” checkbox option under current behaviors of concern


feat(core): support for workspace solution files

Supports passing --solution {path}, or --solution to add a project to a specific or default solution file in the workspace. To use this behavior for all projects in the workspace, add a section to nx.json like
{
  //...
  generators: {
    //...
    "@nx-dotnet/core:app": {
      "solutionFile": true // or "MySolutionFile.sln" etc
    }
  }
}

Closes #241

LGTM. I especially like the e2e tests, which helped me understand what it was doing better than the documentation did.


Is this project maintained?

I would like to fix error: #1565 but is this project even maintained? Will PR be merged if I fix this annoying error that is in the library for 4 years?

seriously +1
How do I close the datepicker popper when using the Tab key? I tab to the next element (or prev) on the page and I just want the popper to close when focus shifts out of the popper but I can't find a solution. I believe the onBlur function just doesn't work. No matter what I try on the page, onBlur={() => console.log('hello')}  does nothing

If no one cares about the project anymore, maybe someone should create a fork. Unfortunately, I don't have the capacity to do this myself. I wanted to create React Open Source project, but this one has way too many users for me alone to handle everything. But maybe if more people will join we will be able to create a better react date picker.
If anyone is interested in forking and maintaining, reading issues, fixing bugs, and doing code reviews and other things, then leave a comment below, so we can find out.
I can do the fork and create an organization, maybe just a generic @react-forks or something, just to not exclude other similar projects that may be needed to fork in the future. But not that I will not do everything myself, it's way too much work for one person, and I'm already maintainer of few projects.

damn react-forks is taken.

Hi all - This repo is still maintained, but I've been a bit busy recently to keep this running on my own. I could use some help, but happy to keep this alive and spend some extra time on it. There's a lot of activity happening here, which is great but running this next to a full-time day job is challenging.

You should seek more maintainers there are way too many issues open and PR hanging.
I'm afraid that will also not be able to help. Also, it's that interesting to me to spend my free time on it. I have other projects and need this one only for the job where I will only be til the end of February.
Maybe you should update the README with info that you seek maintainers. I'm not that good at creating communities myself. I mostly do solo Open Source projects.
But I'm part of one project where the original author left, his project is a great example because he automated a lot of things. Automatic build, tests, and releases. There are issues with this setup, but I think that automating as much as possible is a way to go.

Released new versions today. Enjoy!


Added BFS Graph Algorithm

Hi @pranav-nb
Thank you for merging PR.
Can you pls add hacktober-accepted label?


SpinePlugin: ignoreIfPlaying is broken for multiple tracks

Version

Phaser Version: 3.20.2

Description
The ignoreIfPlaying flag does not consider the trackIndex being applied to.
It is hardcoded as trackIndex 0, thus failing if the check is being applied to other tracks.
https://github.com/photonstorm/phaser/blob/c25331cf30373d685ba59afef7520486d9212e01/plugins/spine/src/gameobject/SpineGameObject.js#L1003

Thank you for submitting this issue. We have fixed this and the fix has been pushed to the master branch. It will be part of the next release. If you get time to build and test it for yourself we would appreciate that.


Make sure to have all helm charts values defined in values.yaml

In GitLab by @paul.laffitte on Jan 14, 2021, 18:52
and change deamonNodeSelector to deamon.NodeSelector

In GitLab by @paul.laffitte on Jan 21, 2021, 16:25
mentioned in commit 46fb1cd522896bb12d31aa7262ea3fcabe5456d5

In GitLab by @paul.laffitte on Jan 21, 2021, 16:26
mentioned in commit ab0a54c3dfc58d54a8c6a5503de8eacb231c3c27

In GitLab by @paul.laffitte on Jan 21, 2021, 16:41
mentioned in commit bb52683fa60519810bdf329ecebeb1675daa8a8c


[CSE-261] Fix typo

YT issue: https://iohk.myjetbrains.com/youtrack/issue/CSE-261

@sectore Is this one for master or for develop? By the look of it looks like the latter, but I don't have the full context so that's why I'm asking 😉

@adinapoli-iohk It's a hot fix, so we do need it in master.

@adinapoli-iohk It's a hot fix, so we do need it in master.

Right, but at the same time we also need to backport this to develop, if we stick to git-flow.
Anyway, I don't have permissions to merge this, you will have to ping on of the other guys ;)


chore: upgrade ember-cli & dependencies

This PR upgrades ember-cli to 3.6.0, which also removes (tested) support for ember-lts-2.12 in anticipation of #175.
All other deps are updated to the most recent version as well.
It also replaces the integrated ember-cli-eslint and ember-cli-template-lint with their standalone counterparts in accordance with https://github.com/ember-cli/rfcs/pull/121.
btw, I can wholeheartedly recommend enabling Dependabot 🤖 for automated dependency upgrades.

@buschtoens thanks a bunch, looks great! :v: