[BUG] Failed to checkpoint with deepspeed 0.12.4

Describe the bug
I encountered an issue when using DeepSpeed 0.12.4 with the OpenChat trainer, where checkpointing failed and raised an NCCL error. However, the checkpoints work fine when using DeepSpeed 0.12.2-0.12.3. The trainer utilizes the save_pretrained from the Transformers library and deepspeed.checkpoint.utils.clone_tensors_for_torch_save, as seen here.
Interestingly, the checkpoint only fails if the training is conducted with more than a certain number of steps (likely >100 steps) using ZeRO. Training with fewer steps does not result in a failure.
Here is the error message:
[10:38:00<41:53:38, 10.39s/it][E ProcessGroupNCCL.cpp:474] [Rank 7] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=
REDUCE, NumelIn=2, NumelOut=2, Timeout(ms)=1800000) ran for 1800475 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:488] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[E ProcessGroupNCCL.cpp:494] To avoid data inconsistency, we are taking the entire process down.
[E ProcessGroupNCCL.cpp:915] [Rank 7] NCCL watchdog thread terminated with exception: [Rank 7] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=REDUCE, NumelIn=2, NumelOut=2, Timeout(ms)=1800000) ran for 1800475 milliseconds before timing
 out.

Expected behavior
Checkpoint should work correctly with deepspeed 0.12.4
ds_report output
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
fused_adam ............. [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_lion ............... [NO] ....... [OKAY]
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
evoformer_attn ......... [NO] ....... [NO]
fused_lamb ............. [NO] ....... [OKAY]
fused_lion ............. [NO] ....... [OKAY]
inference_core_ops ..... [NO] ....... [OKAY]
cutlass_ops ............ [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
ragged_device_ops ...... [NO] ....... [OKAY]
ragged_ops ............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.1
 [WARNING]  using untested triton version (2.1.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/home/share/tianshikeji/syh18_tsinghua/miniconda3/envs/openchat_deepspeed0124/lib/python3.11/site-packages/torch']
torch version .................... 2.1.1+cu121
deepspeed install path ........... ['/home/share/tianshikeji/syh18_tsinghua/miniconda3/envs/openchat_deepspeed0124/lib/python3.11/site-packages/deepspeed']
deepspeed info ................... 0.12.4, unknown, unknown
torch cuda version ............... 12.1
torch hip version ................ None
nvcc version ..................... 11.8
deepspeed wheel compiled w. ...... torch 2.1, cuda 12.1
shared memory (/dev/shm) size .... 1007.76 GB

System info (please complete the following information):

OS: Ubuntu 20.04.6 LTS
H100x8 NVLink (single machine)
Also observed this issue on A40x8

Launcher context
deepspeed launcher on H100x8
deepspeed --module ochat.training_deepspeed.train \
	--model_type openchat_v3.2_mistral \
	--model_path imone/Mistral_7B_with_EOT_token \
	--data_prefix DATA_TOKENIZED \
	--save_path SAVE_PATH \
	--save_every 1 \
	--epochs 5 \
	--batch_max_len 77824 \
	--deepspeed \
	--deepspeed_config ochat/training_deepspeed/deepspeed_config.json

I'm having the same issue - checkpointing simply hangs with multi GPUs exactly after >100 steps.

Figured this out. This is because NCCL cannot use the memory in the PyTorch memory pool and a CUDA OOM occurs during NCCL collective operation. Set NCCL_DEBUG=INFO to see NCCL OOM.

same question.

I'd like to elevate the priority of this issue. I can reproduce this with both Megatron-DeepSpeed and gpt-neox.

Is there a follow-up solution to this problem?

@Quentin-Anthony , I had no issue saving checkpoints with Megatron DeepSpeed training of the GPT-350M model with both ZeRO-1 and ZeRO-3. Additional configurations of interest are as follows: 8 V100 GPUs (single node), exit_interval=400, save_interval=180. Could you please post your test configuration here?”

I'm also observing the issue when fine-tuning.
Machine: 8xH100 GPUs

Using DS ZeRO-3 + Activation Checkpointing + Flash Attention 2, I can run finetuning on 8 H100 GPUs with 16384 sequence length (the max sequence length from CodeLlama paper).
Code: https://github.com/pacman100/DHS-LLM-Workshop/tree/main/personal_copilot/training. Using the main branch of  Accelerate, Transformers, PEFT and the latest versions of Torch(2.2.0), Flash Attention(2.5.2) and DeepSpeed(0.13.1).
 This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
  Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
/fsx/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/torch/autograd/__init__.py:266: UserWarning: c10d::broadcast_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
  Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
/fsx/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/torch/autograd/__init__.py:266: UserWarning: c10d::broadcast_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
  Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
/fsx/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/torch/autograd/__init__.py:266: UserWarning: c10d::broadcast_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
  Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
/fsx/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/torch/autograd/__init__.py:266: UserWarning: c10d::broadcast_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
  Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
/fsx/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/torch/autograd/__init__.py:266: UserWarning: c10d::broadcast_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
  Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
/fsx/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/torch/autograd/__init__.py:266: UserWarning: c10d::broadcast_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
  Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
/fsx/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/torch/autograd/__init__.py:266: UserWarning: c10d::broadcast_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
  Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
{'loss': 0.9162, 'learning_rate': 5.000000000000001e-07, 'epoch': 0.0}
{'loss': 0.957, 'learning_rate': 1.0000000000000002e-06, 'epoch': 0.01}
{'loss': 0.9032, 'learning_rate': 1.5e-06, 'epoch': 0.01}
{'loss': 1.3991, 'learning_rate': 2.0000000000000003e-06, 'epoch': 0.01}
{'loss': 0.831, 'learning_rate': 2.5e-06, 'epoch': 0.01}
{'loss': 0.8801, 'learning_rate': 3e-06, 'epoch': 0.01}
{'loss': 1.1378, 'learning_rate': 3.5e-06, 'epoch': 0.02}
{'loss': 0.948, 'learning_rate': 4.000000000000001e-06, 'epoch': 0.02}
{'loss': 0.9863, 'learning_rate': 4.5e-06, 'epoch': 0.02}
{'loss': 1.0219, 'learning_rate': 5e-06, 'epoch': 0.03}
{'loss': 0.8341, 'learning_rate': 5.500000000000001e-06, 'epoch': 0.03}
{'loss': 1.0635, 'learning_rate': 6e-06, 'epoch': 0.03}
{'loss': 0.9225, 'learning_rate': 6.5000000000000004e-06, 'epoch': 0.03}
{'loss': 1.0654, 'learning_rate': 7e-06, 'epoch': 0.04}
{'loss': 0.7203, 'learning_rate': 7.500000000000001e-06, 'epoch': 0.04}
{'loss': 0.6427, 'learning_rate': 8.000000000000001e-06, 'epoch': 0.04}
{'loss': 1.0184, 'learning_rate': 8.5e-06, 'epoch': 0.04}
{'loss': 1.0611, 'learning_rate': 9e-06, 'epoch': 0.04}
{'loss': 1.3879, 'learning_rate': 9.5e-06, 'epoch': 0.05}
{'loss': 0.9321, 'learning_rate': 1e-05, 'epoch': 0.05}
  5%|██████▌                                                                                | 100/2000 [26:34<4:27:52,  8.46s/it]***** Running Evaluation *****
  Num examples: Unknown
  Batch size = 1

{'eval_loss': 0.45860880613327026, 'eval_runtime': 183.3016, 'eval_samples_per_second': 0.131, 'eval_steps_per_second': 0.016, 'epoch': 0.05}
{'loss': 0.743, 'learning_rate': 1.0500000000000001e-05, 'epoch': 0.05}
{'loss': 0.8819, 'learning_rate': 1.1000000000000001e-05, 'epoch': 0.06}
{'loss': 1.202, 'learning_rate': 1.15e-05, 'epoch': 0.06}
{'loss': 0.9497, 'learning_rate': 1.2e-05, 'epoch': 0.06}
{'loss': 0.9649, 'learning_rate': 1.25e-05, 'epoch': 0.06}
{'loss': 1.1595, 'learning_rate': 1.3000000000000001e-05, 'epoch': 0.07}
{'loss': 0.6125, 'learning_rate': 1.3500000000000001e-05, 'epoch': 0.07}
  7%|████████▉                                                                                | 136/2000 [34:43<4:19:21,  8.35s/it][rank6]:[E ProcessGroupNCCL.cpp:523] [Rank 6] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=4210, OpType=_REDUCE_SCATTER_BASE, NumelIn=492851200, NumelOut=61606400, Timeout(ms)=600000) ran for 699606 milliseconds before timing out.
[rank2]:[E ProcessGroupNCCL.cpp:523] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=4215, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=600000) ran for 698713 milliseconds before timing out.
[rank1]:[E ProcessGroupNCCL.cpp:523] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=4210, OpType=_REDUCE_SCATTER_BASE, NumelIn=492851200, NumelOut=61606400, Timeout(ms)=600000) ran for 699611 milliseconds before timing out.
[rank5]:[E ProcessGroupNCCL.cpp:523] [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=4211, OpType=_REDUCE_SCATTER_BASE, NumelIn=458762240, NumelOut=57345280, Timeout(ms)=600000) ran for 699463 milliseconds before timing out.
[rank1]:[E ProcessGroupNCCL.cpp:537] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank1]:[E ProcessGroupNCCL.cpp:543] To avoid data inconsistency, we are taking the entire process down.
[rank1]:[E ProcessGroupNCCL.cpp:1182] [Rank 1] NCCL watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=4210, OpType=_REDUCE_SCATTER_BASE, NumelIn=492851200, NumelOut=61606400, Timeout(ms)=600000) ran for 699611 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:525 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fb04ceadd87 in /fsx/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1e6 (0x7fb04e0556e6 in /fsx/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x19d (0x7fb04e058c3d in /fsx/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7fb04e059839 in /fsx/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0xdbbf4 (0x7fb097d4bbf4 in /fsx/sourab/miniconda3/envs/hf/bin/../lib/libstdc++.so.6)
frame #5: <unknown function> + 0x8609 (0x7fb09cdff609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #6: clone + 0x43 (0x7fb09cbca353 in /lib/x86_64-linux-gnu/libc.so.6)

[rank6]:[E ProcessGroupNCCL.cpp:537] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank6]:[E ProcessGroupNCCL.cpp:543] To avoid data inconsistency, we are taking the entire process down.
[rank6]:[E ProcessGroupNCCL.cpp:1182] [Rank 6] NCCL watchdog thread terminated with exception: [Rank 6] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=4210, OpType=_REDUCE_SCATTER_BASE, NumelIn=492851200, NumelOut=61606400, Timeout(ms)=600000) ran for 699606 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:525 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f56bb0f6d87 in /fsx/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1e6 (0x7f56bc29e6e6 in /fsx/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x19d (0x7f56bc2a1c3d in /fsx/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7f56bc2a2839 in /fsx/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0xdbbf4 (0x7f5705f94bf4 in /fsx/sourab/miniconda3/envs/hf/bin/../lib/libstdc++.so.6)
frame #5: <unknown function> + 0x8609 (0x7f570b048609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #6: clone + 0x43 (0x7f570ae13353 in /lib/x86_64-linux-gnu/libc.so.6)

[rank2]:[E ProcessGroupNCCL.cpp:537] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank2]:[E ProcessGroupNCCL.cpp:543] To avoid data inconsistency, we are taking the entire process down.
[rank2]:[E ProcessGroupNCCL.cpp:1182] [Rank 2] NCCL watchdog thread terminated with exception: [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=4215, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=600000) ran for 698713 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:525 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f877a647d87 in /fsx/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1e6 (0x7f877b7ef6e6 in /fsx/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x19d (0x7f877b7f2c3d in /fsx/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7f877b7f3839 in /fsx/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0xdbbf4 (0x7f87c54e5bf4 in /fsx/sourab/miniconda3/envs/hf/bin/../lib/libstdc++.so.6)
frame #5: <unknown function> + 0x8609 (0x7f87ca599609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #6: clone + 0x43 (0x7f87ca364353 in /lib/x86_64-linux-gnu/libc.so.6)

[rank5]:[E ProcessGroupNCCL.cpp:537] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank5]:[E ProcessGroupNCCL.cpp:543] To avoid data inconsistency, we are taking the entire process down.
[rank5]:[E ProcessGroupNCCL.cpp:1182] [Rank 5] NCCL watchdog thread terminated with exception: [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=4211, OpType=_REDUCE_SCATTER_BASE, NumelIn=458762240, NumelOut=57345280, Timeout(ms)=600000) ran for 699463 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:525 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f42a7ba4d87 in /fsx/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1e6 (0x7f42a8d4c6e6 in /fsx/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x19d (0x7f42a8d4fc3d in /fsx/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7f42a8d50839 in /fsx/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0xdbbf4 (0x7f42f2a42bf4 in /fsx/sourab/miniconda3/envs/hf/bin/../lib/libstdc++.so.6)
frame #5: <unknown function> + 0x8609 (0x7f42f7af6609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #6: clone + 0x43 (0x7f42f78c1353 in /lib/x86_64-linux-gnu/libc.so.6)

[2024-02-12 12:43:34,263] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 424472 closing signal SIGTERM
[2024-02-12 12:43:34,264] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 424475 closing signal SIGTERM
[2024-02-12 12:43:34,265] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 424476 closing signal SIGTERM
[2024-02-12 12:43:34,265] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 424479 closing signal SIGTERM
[2024-02-12 12:43:36,145] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: -6) local_rank: 1 (pid: 424473) of binary: /fsx/sourab/miniconda3/envs/hf/bin/python
Traceback (most recent call last):
  File "/fsx/sourab/miniconda3/envs/hf/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/fsx/sourab/accelerate/src/accelerate/commands/accelerate_cli.py", line 47, in main
    args.func(args)
  File "/fsx/sourab/accelerate/src/accelerate/commands/launch.py", line 1008, in launch_command
    deepspeed_launcher(args)
  File "/fsx/sourab/accelerate/src/accelerate/commands/launch.py", line 724, in deepspeed_launcher
    distrib_run.run(args)
  File "/fsx/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/torch/distributed/run.py", line 803, in run
    elastic_launch(
  File "/fsx/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 135, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
=======================================================
train.py FAILED
-------------------------------------------------------
Failures:
[1]:
  time      : 2024-02-12_12:43:34
  host      : ip-26-0-169-207.ec2.internal
  rank      : 2 (local_rank: 2)
  exitcode  : -6 (pid: 424474)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 424474
[2]:
  time      : 2024-02-12_12:43:34
  host      : ip-26-0-169-207.ec2.internal
  rank      : 5 (local_rank: 5)
  exitcode  : -6 (pid: 424477)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 424477
[3]:
  time      : 2024-02-12_12:43:34
  host      : ip-26-0-169-207.ec2.internal
  rank      : 6 (local_rank: 6)
  exitcode  : -6 (pid: 424478)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 424478
-------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-02-12_12:43:34
  host      : ip-26-0-169-207.ec2.internal
  rank      : 1 (local_rank: 1)
  exitcode  : -6 (pid: 424473)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 424473
=======================================================

Another concern is the warning that happens when using latest DeepSpeed with latest Torch:

UserWarning: c10d::broadcast_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass

Output of ds_report:
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
fused_adam ............. [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_lion ............... [NO] ....... [OKAY]
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
evoformer_attn ......... [NO] ....... [NO]
fused_lamb ............. [NO] ....... [OKAY]
fused_lion ............. [NO] ....... [OKAY]
inference_core_ops ..... [NO] ....... [OKAY]
cutlass_ops ............ [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
ragged_device_ops ...... [NO] ....... [OKAY]
ragged_ops ............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.2
 [WARNING]  using untested triton version (2.2.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/fsx/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/torch']
torch version .................... 2.2.0+cu121
deepspeed install path ........... ['/fsx/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/deepspeed']
deepspeed info ................... 0.13.1, unknown, unknown
torch cuda version ............... 12.1
torch hip version ................ None
nvcc version ..................... 12.1
deepspeed wheel compiled w. ...... torch 2.2, cuda 12.1
shared memory (/dev/shm) size .... 999.99 GB

We are seeing this as well. At tricky one for sure. No problems with the training phase, just when saving the checkpoint it seems.
Machine(s): 5x nodes : AMD Rome, 4x A100 40GB each
OS: SLES 15.4
Launcher:
#!/bin/bash
#SBATCH --job-name=sv-huge-deepspeed   # create a short name for your job
#SBATCH --nodes=5                # node count
#SBATCH --ntasks-per-node=1      # total number of tasks per node
#SBATCH --cpus-per-task=40       # cpu-cores per task (>1 if multi-threaded tasks)
#SBATCH --mem=300G                # total memory per node (4 GB per cpu-core is default)
#SBATCH --gres=gpu:4             # number of allocated gpus per node
#SBATCH --time=12:00:00       # total run time limit (HH:MM:SS)

# export MASTER_PORT=$(expr 10000 + $(echo -n $SLURM_JOBID | tail -c 4))
export MASTER_PORT=6000
export WORLD_SIZE=$(($SLURM_NNODES * $SLURM_NTASKS_PER_NODE))
echo "WORLD_SIZE="$WORLD_SIZE

export NCCL_SOCKET_IFNAME=ib

export MASTER_ADDR=$(scontrol show hostname ${SLURM_NODELIST} | head -n 1)
echo "MASTER_ADDR="$MASTER_ADDR

echo "$MASTER_ADDR:$MASTER_PORT"

export PYTHONPATH=$PWD:pytorch-caney
export NCCL_DEBUG=INFO

# do not remove or the training will hang and nodes will be lost w/o this workaround
#export CUDA_LAUNCH_BLOCKING=1

# hide duplicated errors using this hack - will be properly fixed in pt-1.12
#export TORCHELASTIC_ERROR_FILE=torch-elastic-error.json

# force crashing on nccl issues like hanging broadcast
#export NCCL_ASYNC_ERROR_HANDLING=1

#export NCCL_P2P_DISABLE=1

echo $SLURM_JOB_NUM_NODES
echo $SLURM_PROCID
echo $MASTER_ADDR
echo $MASTER_PORT

nnodes=$SLURM_JOB_NUM_NODES

launcher="singularity exec --nv -B /discover,/gpfsm /discover/nobackup/projects/akmosaic/container/nvpt-24.01 python -u -m torch.distributed.run --nnodes=${nnodes} --master_addr ${MASTER_ADDR} --master_port ${MASTER_PORT} --nproc_per_node=4"

echo $launcher

cmd=" pytorch-caney/pytorch_caney/pipelines/pretraining/mim_deepspeed.py --cfg $1 --dataset MODIS --data-paths /discover/nobackup/projects/calebtest/3dclouds/v3 --output . --batch-size 40"
echo $cmd

srun --jobid $SLURM_JOBID bash -c "$launcher --node_rank \$SLURM_PROCID $cmd"

echo "END TIME: $(date)"

Output of ds_report:
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
fused_adam ............. [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_lion ............... [NO] ....... [OKAY]
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
evoformer_attn ......... [NO] ....... [NO]
fused_lamb ............. [NO] ....... [OKAY]
fused_lion ............. [NO] ....... [OKAY]
inference_core_ops ..... [NO] ....... [OKAY]
cutlass_ops ............ [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
ragged_device_ops ...... [NO] ....... [OKAY]
ragged_ops ............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.2
 [WARNING]  using untested triton version (2.1.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/usr/local/lib/python3.10/dist-packages/torch']
torch version .................... 2.2.0a0+81ea7a4
deepspeed install path ........... ['/home/cssprad1/.local/lib/python3.10/site-packages/deepspeed']
deepspeed info ................... 0.13.3, unknown, unknown
torch cuda version ............... 12.3
torch hip version ................ None
nvcc version ..................... 12.3
deepspeed wheel compiled w. ...... torch 2.2, cuda 12.3
shared memory (/dev/shm) size .... 251.56 GB

Log: https://gist.github.com/cssprad1/7f9f8ec52575f038bc6cc36979fc8321
DS config:
    deepspeed_config = {

        "train_micro_batch_size_per_gpu": config.DATA.BATCH_SIZE,

        "steps_per_print": config.PRINT_FREQ,

        "zero_optimization": {
            "stage": 2,
            "overlap_comm": True,
            "reduce_scatter": True,
            "reduce_bucket_size": 5e8,
            "allgather_bucket_size": 5e8,
        },

        # "activation_checkpointing": {
        #     "partition_activations": True,
        #     "cpu_checkpointing": True,
        #     "profile": True,
        # },

        "fp16": {
            "enabled": False,
        },

        "bf16": {
            "enabled": True,
        },

        "optimizer": {
            "type": "AdamW",
            "params": {
                "lr": config.TRAIN.BASE_LR,
            },
            # "type": "ZeroOneAdam",
            # "params": {
            #     "lr": 1e-3,
            #     "weight_decay": 0.01,
            #     "comm_backend_name": "nccl",
            #     "cuda_aware": False,
            # },
        },

        "scheduler": {
            "type": "WarmupLR",
            "params": {
                "warmup_min_lr": config.TRAIN.WARMUP_LR,
                "warmup_max_lr": config.TRAIN.BASE_LR,
            },
        },

        "flops_profiler": {
            "enabled": False,
            "profile_step": 1,
            "module_depth": -1,
            "top_modules": 1,
            "detailed": True,
            "output_file": f'profile_{time.time()}',
        },

    }

Checkpoint code:
            model_engine.save_checkpoint(save_dir=config.OUTPUT,
                                         tag=f'ckpt_epoch_{epoch}.pth')

Figured this out. This is because NCCL cannot use the memory in the PyTorch memory pool and a CUDA OOM occurs during NCCL collective operation. Set NCCL_DEBUG=INFO to see NCCL OOM.

Any updates on this issue? I've figured out a temporary solution:

When checkpointing, copy the weights to CPU first, avoid using model_engine.save_checkpoint as it will make an extra copy on GPU.
Remove other stages (especially evaluation) besides training and checkpointing.

It solves this issue by strictly avoiding any extra GPU memory allocation besides training logic. The reference implementation is openchat trainer.

Figured this out. This is because NCCL cannot use the memory in the PyTorch memory pool and a CUDA OOM occurs during NCCL collective operation. Set NCCL_DEBUG=INFO to see NCCL OOM.

Any updates on this issue? I've figured out a temporary solution:
1. When checkpointing, copy the weights to CPU first, avoid using `model_engine.save_checkpoint` as it will make an extra copy on GPU.

2. Remove other stages (especially evaluation) besides training and checkpointing.

It solves this issue by strictly avoiding any extra GPU memory allocation besides training logic. The reference implementation is openchat trainer.

Would this save the partitioned optimizer states, etc? That would be valuable for resuming model training while side-stepping the extra GPU mem allocs.

I have the same issue when train mixtral 7bx8 with transformers 4.36 and deepseed 0.12.4(0.12.3) zero3 with gradient_checkpointing enable . it hangs after around 1:30 hours traning.

@dumpmemory I am having the same issue with training mixtral 8x7b as well. Did you manage solve your issue ? Thanks !

We are seeing this as well. A tricky one for sure. No problems with the training phase, just when saving the checkpoint it seems.我们也看到了这一点。这肯定是一个棘手的问题。训练阶段没有问题，只是在保存检查点时出现了问题。
Machine(s): 5x nodes : AMD Rome, 4x A100 40GB each机器：5x 节点：AMD Rome，4x A100，每个 40GB OS: SLES 15.4 操作系统：SLES 15.4
Launcher: 启动器：
#!/bin/bash
#SBATCH --job-name=sv-huge-deepspeed   # create a short name for your job
#SBATCH --nodes=5                # node count
#SBATCH --ntasks-per-node=1      # total number of tasks per node
#SBATCH --cpus-per-task=40       # cpu-cores per task (>1 if multi-threaded tasks)
#SBATCH --mem=300G                # total memory per node (4 GB per cpu-core is default)
#SBATCH --gres=gpu:4             # number of allocated gpus per node
#SBATCH --time=12:00:00       # total run time limit (HH:MM:SS)

# export MASTER_PORT=$(expr 10000 + $(echo -n $SLURM_JOBID | tail -c 4))
export MASTER_PORT=6000
export WORLD_SIZE=$(($SLURM_NNODES * $SLURM_NTASKS_PER_NODE))
echo "WORLD_SIZE="$WORLD_SIZE

export NCCL_SOCKET_IFNAME=ib

export MASTER_ADDR=$(scontrol show hostname ${SLURM_NODELIST} | head -n 1)
echo "MASTER_ADDR="$MASTER_ADDR

echo "$MASTER_ADDR:$MASTER_PORT"

export PYTHONPATH=$PWD:pytorch-caney
export NCCL_DEBUG=INFO

# do not remove or the training will hang and nodes will be lost w/o this workaround
#export CUDA_LAUNCH_BLOCKING=1

# hide duplicated errors using this hack - will be properly fixed in pt-1.12
#export TORCHELASTIC_ERROR_FILE=torch-elastic-error.json

# force crashing on nccl issues like hanging broadcast
#export NCCL_ASYNC_ERROR_HANDLING=1

#export NCCL_P2P_DISABLE=1

echo $SLURM_JOB_NUM_NODES
echo $SLURM_PROCID
echo $MASTER_ADDR
echo $MASTER_PORT

nnodes=$SLURM_JOB_NUM_NODES

launcher="singularity exec --nv -B /discover,/gpfsm /discover/nobackup/projects/akmosaic/container/nvpt-24.01 python -u -m torch.distributed.run --nnodes=${nnodes} --master_addr ${MASTER_ADDR} --master_port ${MASTER_PORT} --nproc_per_node=4"

echo $launcher

cmd=" pytorch-caney/pytorch_caney/pipelines/pretraining/mim_deepspeed.py --cfg $1 --dataset MODIS --data-paths /discover/nobackup/projects/calebtest/3dclouds/v3 --output . --batch-size 40"
echo $cmd

srun --jobid $SLURM_JOBID bash -c "$launcher --node_rank \$SLURM_PROCID $cmd"

echo "END TIME: $(date)"

Output of ds_report: ds_report 的输出：
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
fused_adam ............. [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_lion ............... [NO] ....... [OKAY]
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
evoformer_attn ......... [NO] ....... [NO]
fused_lamb ............. [NO] ....... [OKAY]
fused_lion ............. [NO] ....... [OKAY]
inference_core_ops ..... [NO] ....... [OKAY]
cutlass_ops ............ [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
ragged_device_ops ...... [NO] ....... [OKAY]
ragged_ops ............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.2
 [WARNING]  using untested triton version (2.1.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/usr/local/lib/python3.10/dist-packages/torch']
torch version .................... 2.2.0a0+81ea7a4
deepspeed install path ........... ['/home/cssprad1/.local/lib/python3.10/site-packages/deepspeed']
deepspeed info ................... 0.13.3, unknown, unknown
torch cuda version ............... 12.3
torch hip version ................ None
nvcc version ..................... 12.3
deepspeed wheel compiled w. ...... torch 2.2, cuda 12.3
shared memory (/dev/shm) size .... 251.56 GB

Log: https://gist.github.com/cssprad1/7f9f8ec52575f038bc6cc36979fc8321日志： https://gist.github.com/cssprad1/7f9f8ec52575f038bc6cc36979fc8321
DS config: DS配置：
    deepspeed_config = {

        "train_micro_batch_size_per_gpu": config.DATA.BATCH_SIZE,

        "steps_per_print": config.PRINT_FREQ,

        "zero_optimization": {
            "stage": 2,
            "overlap_comm": True,
            "reduce_scatter": True,
            "reduce_bucket_size": 5e8,
            "allgather_bucket_size": 5e8,
        },

        # "activation_checkpointing": {
        #     "partition_activations": True,
        #     "cpu_checkpointing": True,
        #     "profile": True,
        # },

        "fp16": {
            "enabled": False,
        },

        "bf16": {
            "enabled": True,
        },

        "optimizer": {
            "type": "AdamW",
            "params": {
                "lr": config.TRAIN.BASE_LR,
            },
            # "type": "ZeroOneAdam",
            # "params": {
            #     "lr": 1e-3,
            #     "weight_decay": 0.01,
            #     "comm_backend_name": "nccl",
            #     "cuda_aware": False,
            # },
        },

        "scheduler": {
            "type": "WarmupLR",
            "params": {
                "warmup_min_lr": config.TRAIN.WARMUP_LR,
                "warmup_max_lr": config.TRAIN.BASE_LR,
            },
        },

        "flops_profiler": {
            "enabled": False,
            "profile_step": 1,
            "module_depth": -1,
            "top_modules": 1,
            "detailed": True,
            "output_file": f'profile_{time.time()}',
        },

    }

Checkpoint code: 检查点代码：
            model_engine.save_checkpoint(save_dir=config.OUTPUT,
                                         tag=f'ckpt_epoch_{epoch}.pth')

hello,have you solved the problem?


SR_2_POL have all the ValidityStart to 19870101T000000

Dear,
SR_2_POL have all the ValidityStart to 19870101T000000 thus the computation of the SensingApplicability can not be done as for the other type.
I suggest as a first approach to compute it on the same way as for the ERRMAT type ?
Regards

Also catherine told me that this file is incremental thus the last contains the informations from the first adn so on.
Is it possible then to put only the last one and make it valid for the whole satellite period ?
Regards

Also catherine told me that this file is incremental thus the last contains the informations from the first adn so on.
Is it possible then to put only the last one and make it valid for the whole satellite period ?
Regards

Email from S3 MPC send on 14/01/2021.

Email from S3 MPC send on 14/01/2021.


"Conditional MUST" needs better explanation

There are many instances of "Conditional MUST" that are confusing to the reader. Several kinds exist:

a SHOULD requirement that depends on a certain value of a MUST requirement, e.g. CSIP3 and 2;
a MAY requirement that depends on a certain value of a SHOULD requirement, e.g. CSIP5 and 4;
a MUST requirement that depends on the existence of a SHOULD-level element, e.g. CSIP33 and 32, and the flavour of it where both have the same XPath, e.g. CSIP99 and 97 (see also #570);
a MUST requirement that depends on the existence of a MAY-level element, e.g. CSIP46 and 45;
a SHOULD requirement that depends on the existence of a MAY-level element, e.g. CSIP47 and 45;
a SHOULD requirement that has a condition in itself, e.g. CSIP8;
a MUST requirement that has a SHOULD in itself, e.g. CSIP113;
a SHOULD requirement that depends on a MUST requirement that has a secondary dependence, e.g. CSIP97 and 113.

All of these are actually "conditional MUST":

if the precondition is met, the dependent requirements MUST be met;
if the precondition is not met, the dependent requirements are SHOULD, MAY or MUST NOT (the latter in the sense that they cannot be met, e.g. rightsMD/@ID cannot exist if rightsMD does not exist).

While most of these do have a sound logic, it takes a lot of effort to learn that logic. The solution could be an explanatory paragraph somewhere before the METS requirements (e.g. in chapter 5.3) or improved descriptions of each of the specific requirements.
PS The above is valid for CSIP v.2.0.4.

Good work, Koit. I agree. I think a visually logic could help - for example by indentation.

Closed with PR #691


Error trying to enable development mode

Hello,
I'm using the latest version of module. But, when i try to enable development mode i get an error, see:

I'm using zf-apigility-doctrine into my project.
As I could see, the Zend first load the Doctrine libraries (which returns the aforementioned error) before loading the DevelopmentMode class.
Is there a way to suppress premature loading of the Doctrine? (Premature because the developer has not configured the database connection).
And the only way to make it work is going manually and retitle the .dist files.
Thanks for the attention,
Ilson Nóbrega

I'm having the same issue. Development mode does not appear to work with Doctrine:
php public/index.php development enable

PHP Fatal error:  Uncaught exception 'PDOException' with message 'SQLSTATE[HY000] [2002] No such file or directory' in /Users/clemfreeman/Code/secretary/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOConnection.php:43
Stack trace:
#0 /Users/clemfreeman/Code/secretary/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOConnection.php(43): PDO->__construct('mysql:host=loca...', 'xyz', 'abc', Array)
#1 /Users/clemfreeman/Code/secretary/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOMySql/Driver.php(45): Doctrine\DBAL\Driver\PDOConnection->__construct('mysql:host=loca...', 'xyz', 'abc', Array)
#2 /Users/clemfreeman/Code/secretary/vendor/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(360): Doctrine\DBAL\Driver\PDOMySql\Driver->connect(Array, 'xyz', 'abc', Array)
#3 /Users/clemfreeman/Code/secretary/vendor/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(429): Doctrine\DBAL\Connection->connect()
#4 /Users/clemfreeman/Code/secretary/vendor/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(389): Doctrine\DBAL\Connection->getDatabasePlatfo in /Users/clemfreeman/Code/secretary/vendor/zendframework/zendframework/library/Zend/ServiceManager/ServiceManager.php on line 930

This might be related to #16 (configuration caching). Can you update to v2.1.2 and see if the issue is resolved, please?

@weierophinney I was getting this issue after moving my development doctrine connection configuration into config/autoload/devlopment.local.php. When I manually enable that configuration, development enable works fine, but otherwise it exits with the PDOException. 2.1.2 has the same behavior.
Before adding Apigility to our ZF2 project, we were using an environment variable check to enable development configuration, as described in the documentation. For now I'll go back to that method. Is there a reason that development mode has to run the application? At the stage in my application setup process that development enable is useful I do not yet have my vagrant environment up.

Is there a reason that development mode has to run the application?

It no longer does, as of v3.0.0, which uses zf-console, instead of the zend-mvc console integration, to do the work. Try upgrading to that version; however, be aware that the invocation of the tool changes when you do (see the migration note for details).


Custom paging and sorting

Hi, I ran into a possible bug.
If I choose manual paging in the grid, the grid sorts result I send to him anyway.
If I have 100 entries of strings and use manual paging in the grid with page size 10 and I display the first page. The main advantage of manual sorting is obviously that I can sort and page it in the database, so I send sorted and paged result to the grid.
As I found out, the grid sorts the result I sent to him again. This doesn't seem to be a problem at first, but it is, because sorting in a database can be defined else than sorting in C#.
For example when sorting ascending, NHibernate and Oracle tends to leave null entries at the end of the result, while C# leaves them at the beginning. If I have array of strings { "a", "b", "c", "b", null, "d", null, null, null } and I sort it in the database, the ordered result will look like this { "a", "b", "b", "c", "d", null,  null, null, null }. If we page this with pageSize = 3 and take the second page, then we should get {"c", "d", null} and the third page should look like { null, null, null }. However, it seems the framework sorts in C# as well, so the second page will look like { null, "c", "d" } (and the third page will contain {null, null, null} as before). This looks inconsistent, because null entry should be followed up by another null entries, but instead of that it's followed by "c" and "d" and nulls are on the next page.
Imo in case of manual paging there's no reason to sort it by the framework as well.

This is true and it works like that, however grid pager and every column is a different data processor.
Generally grid runs all the column filters and then sorts (pre) and then paging (post).
So setting manual on pager doesn't change the columns.
The only problem I see now is that grid column doesn't have an extension to set processor type to manual, so it has to be done directly on the column
columns.Add(model => model.Name).ProcessorType = GridProcessorType.Manual;

I will try adding the necessary extensions in the next release.

I think I understand, but why would you ever want to let the framework sort the result if you choose manual paging?
If I have manual paging, in your proposed solution (even with that extension method) I would have to literally write that for every column I have in the grid.

There would be a change on .Sortable and .Filterable extensions too, to avoid "on every column" problem. But yes, having a global grid property for "I will do everything manually" would make the most sense and it is in my backlog for a while now, but for reasons it wasn't done till now.
Manual paging itself was added way after the initial grid API was formed, so I chose the least API breaking way to add manual paging (by adding a new processor type), which resulted in a 4 line change at the time.
Moving forward, maybe next major release, a global change could be made if won't find any edge cases.
Some people actually expect filtering and sorting to work after manual paging, but it's not possible without taking a performance hit on larger tables or even having a semantic change like in your sorting example.

there is a performance problem on this grid because it execute entire query on database instead of executing only required page query, for example if i want to take top(100) rows with paging

No there is not, unless you are giving the grid in memory collection and not a proper IQueryable.

I mean how can I achieve skip rows as part of database query, with sorting and filtering, I tried manual paging but it filter and sort current  page not entire page

There's an example on how to do a manual paging. As per this conversation, you also have to make columns manual, and do similar sorting and filtering building in the controller. However filtering is way harder, because of manual parameter parsing needed.

In manual example there is no filtering or paging option how can I achieve them

There is paging option shown in the example page
[HttpGet]
// Take page and rows values from the query, which are provided from the grid.
public ViewResult Index(Int32? page, Int32? rows)
{
    // Do your own total count on the records, this is the same as writing.
    // `SELECT COUNT(1) FROM YOUR_TABLE`
    ViewBag.TotalRows = PeopleRepository.GetPeople().Count();

    // Do your own paging on the records using the provided page values,
    // they have to be null coalesced, because initial grid doesn't have paging information.
    // This is the same as writing `SELECT * FROM YOUR_TABLE LIMIT 3 OFFSET 0`.
    return View(PeopleRepository.GetPeople().Skip((page - 1 ?? 0) * (rows ?? 3)).Take(rows ?? 3));
}

Sorting is basically the same as paging, just that parameters names are sort and order respectively.
Filtering needs to parsed from Request.Query, it is passed in this format
<grid-name-if-any>-<column-name>-<filter-type>=<filter-value>
You can check column grid filter for example parsing.


Treat context deadline exceeded error as a success

What this PR does / why we need it:
Which issue(s) this PR fixes:
Fixes https://github.com/pipe-cd/pipe/issues/1245
Does this PR introduce a user-facing change?:

NONE

/lgtm

After careful consideration, I realized we don't have to propagate the context to Analysis provider clients. They each have their own query timeout, so we should wait for the query to end up even if the analysis duration exceeds.

@nghialv PTAL 😄

@nakabonne

After careful consideration, I realized we don't have to propagate the context to Analysis provider clients. They each have their own query timeout, so we should wait for the query to end up even if the analysis duration exceeds.

Do we have any better way?
I think removing the context is not good because it removes the control of function lifetime from the caller.
The caller has no way to stop the executing one. That may be not good for the case of canceling or piped termination.

For sure. We wanna avoid it as much as possible because it blocks until the HTTP request completed even if we want to terminate Piped right now.
We just want to distinguish errors of parent's deadline exceeded and children's one. I'm looking for a way to find out them.

As we talked about previously, we just want to ignore the deadline exceeded error from the parent. To check that, I settled on using Err() against the parent context.

@nghialv PTAL 😸

Nice.
/lgtm

🏃
/approve


add faucet button to docs site

Add faucet button to the docs site in the top navigation next to the connect metamask button. Uses a faucet icon instead of text. LMK what ya think!

@eshaben I added the target=_blank blob so that it takes you to a new page. Also, what do you think about this metamask logo?

I'll leave it pre-approved, but feel free to choose whichever you like the most


Preparing dataset for training CogVideo1.5 I2V

System Info / 系統信息
Linux, otherwise N/A
Information / 问题信息

[X] The official example scripts / 官方的示例脚本
[ ] My own modified scripts / 我自己修改的脚本和任务

Reproduction / 复现过程
Here it only provides information on how to prepare the dataset for text to video. There is no information for I2V.
Expected behavior / 期待表现
Information on how to prepare a dataset for image to video.

Similarly, I2V means that the first frame of the video is the Image. The code will automatically capture it.

Does any resolution work similarly to how i2v currently works or does it need to be a set resolution?

Need fixed, for example, CogVideoX1.0 is 720 * 480
Regarding CogVideoX1.5, it supports 768-1360 (long edge) and 768 short edge. However, there is currently no manpower available to invest in writing the specific fine-tuning code, and it is expected to continue using CogVideoX-Factory as the fine-tuning framework for open-source models.

Regarding CogVideoX1.5, it supports 768-1360 (long edge) and 768 short edge

Is vertical video (i.e. 768x1360) meant to be supported? It always becomes blurry when I try.

No, only CogVideoX1.5-I2V is supported; other models must strictly adhere to the resolution mentioned in the readme and cannot use portrait orientation.


Problem opencti synchronize

Good morning friend, how are you, thanks for sharing the script, I'm trying to synchronize with my opencti and I have this problem below

Thanks

Hello Friend,
Thank you for reaching out to us and sharing the script. I understand that you are trying to sync with OpenCTI and are running into a problem. Based on the details provided, the solution appears to be in the .lenv copy and .env copy files located in the src folder. These files contain path information for log files and data files. Please follow the steps below to resolve the issue:

Rename the .lenv copy file to .lenv.
Try running the script again.

Best regards,


return on handleCallback

otherwise the scucess handler is called, which results into further missleading errors (e.g. length is not defined)

@simllll thanks for the contribution!


New Nitter Changes

Hello
Apparently X deployed some changes the last few days that broke Nitter again, and then Nitter just deployed changes from their side and the service is back up. However, ntscraper now simply stops after 5 fetches from nitter and that's it.
19-Aug-23 13:43:39 - Error fetching https://nitter.net, trying another random instance 19-Aug-23 13:43:40 - Error fetching https://nitter.net, trying another random instance 19-Aug-23 13:43:42 - Error fetching https://nitter.net, trying another random instance 19-Aug-23 13:43:43 - Error fetching https://nitter.net, trying another random instance 19-Aug-23 13:43:44 - Error fetching https://nitter.net, trying another random instance 19-Aug-23 13:43:45 - Max retries reached. Check your request and try again

I think we should wait for nitter to publish the new version

update: i just try the new update of nitter with the current version of ntscraper and work fine.

when i calling the get_tweets function :
raw_data = scraper.get_tweets("papua", number=5, max_retries=20
somehow the random instances doesnt work and returning error like this :

23-Aug-23 15:43:48 - Error fetching https://nitter.net/, trying another random instance
23-Aug-23 15:43:50 - Error fetching https://nitter.net/, trying another random instance
23-Aug-23 15:43:52 - Error fetching https://nitter.net/, trying another random instance
23-Aug-23 15:43:55 - Error fetching https://nitter.net/, trying another random instance
23-Aug-23 15:43:57 - Error fetching https://nitter.net/, trying another random instance
23-Aug-23 15:43:59 - Error fetching https://nitter.net/, trying another random instance
23-Aug-23 15:44:01 - https://nitter.net/ unreachable, trying another random instance
23-Aug-23 15:44:03 - Error fetching https://nitter.net/, trying another random instance
23-Aug-23 15:44:05 - Error fetching https://nitter.net/, trying another random instance
23-Aug-23 15:44:07 - Error fetching https://nitter.net/, trying another random instance
23-Aug-23 15:44:10 - Error fetching https://nitter.net/, trying another random instance
23-Aug-23 15:44:13 - Error fetching https://nitter.net/, trying another random instance
23-Aug-23 15:44:15 - Error fetching https://nitter.net/, trying another random instance
23-Aug-23 15:44:17 - Error fetching https://nitter.net/, trying another random instance
23-Aug-23 15:44:19 - Error fetching https://nitter.net/, trying another random instance
23-Aug-23 15:44:22 - Error fetching https://nitter.net/, trying another random instance
23-Aug-23 15:44:24 - Error fetching https://nitter.net/, trying another random instance
23-Aug-23 15:44:26 - Error fetching https://nitter.net/, trying another random instance
23-Aug-23 15:44:28 - Error fetching https://nitter.net/, trying another random instance
23-Aug-23 15:44:31 - Error fetching https://nitter.net/, trying another random instance
23-Aug-23 15:44:32 - Max retries reached. Check your request and try again.

Thank you, fixed with the last commit. Apparently the instances page has a slightly different table layout. To get the fix just update the library.

Thank you, fixed with the last commit. Apparently the instances page has a slightly different table layout. To get the fix just update the library.

Apparently the issue still persists, even after updating the library.

I just pushed a small fix for the video urls in regular tweets, but I'm unable to reproduce the original issue you're having. Some instances don't work correctly, but for example the nitter.privacydev.net and nitter.cz instances work without issues.

The below is a snippet of what I am doing, even specifying an instance still produces the same error. If I put the exacy query directly on Nitter's UI, it works well. The error is below the code snippet for your referece.
`#manual run to bring unarchived tweets
else:
print('Manual Run \n')
runTime = datetime.datetime.now(datetime.timezone.utc)
newTweets = []
insertedTweets = []
tweetsList = []

#get all tweets matching the search condition

while True:
    try:
        tweets = nitter.get_tweets(termsManual,
                                   'term',
                                   since=manualDateSince,
                                   until=manualDateUntil,
                                   max_retries=20,
                                   instance="https://nitter.privacydev.net")

    except IndexError: #user has no tweets in the specified time range, just skip this error
        continue
    except Exception as error:
        print("An exception occurred:", type(error).__name__)
        continue
    break

if tweets is not None:
    newTweets = newTweets + tweets['tweets']
    for tweet in tweets['threads']:
        newTweets.append(tweet)`

25-Aug-23 18:57:47 - Empty profile on https://nitter.privacydev.net, trying another random instance 25-Aug-23 18:57:49 - Empty profile on<EMAIL_ADDRESS>trying another random instance 25-Aug-23 18:57:50 - Empty profile on https://nitter.privacydev.net, trying another random instance 25-Aug-23 18:57:51 - Error fetching https://nitter.net, trying https://nitter.net 25-Aug-23 18:57:52 - Error fetching https://nitter.net, trying https://nitter.privacydev.net 25-Aug-23 18:57:54 - Empty profile on https://nitter.privacydev.net, trying another random instance ......... .........

I've added a "cooldown" system that temporarily pauses the current instance for 20 seconds if it has issues and also made the sleep between calls 2 seconds instead of 1 (it seemed more stable when testing it). Please update the library and try it out, currently it gives me no issues with the same code you provided.

I've added a "cooldown" system that temporarily pauses the current instance for 20 seconds if it has issues and also made the sleep between calls 2 seconds instead of 1 (it seemed more stable when testing it). Please update the library and try it out, currently it gives me no issues with the same code you provided.

Done, just updated and ran the script again. From the initial run, it seems to be solved. Let's see how it goes as days pass. Hopefully the tesla guy stops forcing changes too.

@bocchilorenzo there is another issue now. It loops through the "Max retries reached" error and never goes out.

Should be fixed now with the latest commit


Generator: Generate delete-by statements

Lots of repetitive commits here :)
This PR extends DELETE generation to allow deleting by more than just ID, and to allow deleting many rows at once.
Changes:

Added gofmt -s -w ./lxd/db/ to Makefile under update-schema for file consistency.
Added //go:build linux && cgo && !agent comment as part of the generator.
statement generation for delete now requires the field to delete by to be specified, ie: delete-by-name-and-project
method generation can now be one of DeleteOne or DeleteMany. The difference is just an if-statement that returns an error in the former case.
the generated Delete method now takes the Filter struct for its corresponding object as its argument.

Next up will be operations, since I have that done already.
After that will be automatic generation of all possible statement combinations for a Filter struct, so that the -by- will not need to be directly specified.

Looks good to me!


BUG: Password View/Hide Toggle Functionality

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?
When clicking on the "View Password" icon, instead of showing the password, it hides it and vice versa.

I would like to work on this issue. Please assign it to me under GSSoC '24.
Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a GSSOC'24 contributor
[X] I want to work on this issue

I am interested in addressing this issue and confident that my skills and experience make me well-suited to resolve it efficiently. I would appreciate it if this issue could be assigned to me under GSSoC'24.

Hey,
I am interested in addressing this issue and confident that my skills and experience make me well-suited to resolve it efficiently. I would appreciate it if this issue could be assigned to me under GSSoC'24.
Thankyou.
Ok start working on it

**hey i have sorted this bug **
https://github.com/user-attachments/assets/ad22b828-bc2b-4f8c-a39f-52affba9991e


Huge number of retries of unsuccessful requests in athena

One request to the hive metastore to add a partition generates an infinite number of repeated requests to athena. Which seems to prevent all following requests from being run in athena

I researched the error handling code, any unhandled execution error results in an infinite loop of re-sending the request because the completed flag is false
https://github.com/awslabs/aws-glue-catalog-sync-agent-for-hive/blob/6dd72172d3a5969a0b559bad2503c76155e1771a/src/main/java/com/amazonaws/services/glue/catalog/HiveGlueCatalogSyncAgent.java#L165-L170

Yes, this is expected as the error may be retryable. What is the base exception causing the create partition statement to fail?

Yes, this is expected as the error may be retryable. What is the base exception causing the create partition statement to fail?

In my case,exception causing is table not found, but  in a different case, it could be any other and to be generate an infinite number of repeated requests to athena


feat(napi/transform): add define and inject options

part of #6156

#6212  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @Boshen and the rest of your teammates on  Graphite


192.168.29.x closed. error? false.

Hello, I got 5 Yeelight Color 1S bulbs and they all seem to be having an issue.
[10/22/2020, 2:26:11 PM] [Config] Homebridge Config UI X v4.30.0 is listening on :: port 8581
[10/22/2020, 2:26:13 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:13 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:13 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:13 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:13 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:13 PM] [Yeelight] 0x0000000010e34268: failed communication attempt 0 after 100ms.
[10/22/2020, 2:26:13 PM] [Yeelight] 0x0000000012ae4358: failed communication attempt 0 after 100ms.
[10/22/2020, 2:26:13 PM] [Yeelight] 0x000000001259b8b0: failed communication attempt 0 after 100ms.
[10/22/2020, 2:26:13 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:13 PM] [Yeelight] 0x00000000125a2f63: failed communication attempt 0 after 100ms.
[10/22/2020, 2:26:13 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:13 PM] [Yeelight] 0x0000000012ae4373: failed communication attempt 0 after 100ms.
[10/22/2020, 2:26:13 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:13 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:13 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:14 PM] [Yeelight] 0x0000000010e34268: failed communication attempt 1 after 200ms.
[10/22/2020, 2:26:14 PM] [Yeelight] 0x0000000012ae4358: failed communication attempt 1 after 200ms.
[10/22/2020, 2:26:14 PM] [Yeelight] 0x000000001259b8b0: failed communication attempt 1 after 200ms.
[10/22/2020, 2:26:14 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:14 PM] [Yeelight] 0x00000000125a2f63: failed communication attempt 1 after 200ms.
[10/22/2020, 2:26:14 PM] [Yeelight] 0x0000000012ae4373: failed communication attempt 1 after 200ms.
[10/22/2020, 2:26:14 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:14 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:14 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:14 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:14 PM] [Yeelight] 0x0000000010e34268: failed communication attempt 2 after 400ms.
[10/22/2020, 2:26:14 PM] [Yeelight] 0x0000000012ae4358: failed communication attempt 2 after 400ms.
[10/22/2020, 2:26:14 PM] [Yeelight] 0x000000001259b8b0: failed communication attempt 2 after 400ms.
[10/22/2020, 2:26:14 PM] [Yeelight] 0x00000000125a2f63: failed communication attempt 2 after 400ms.
[10/22/2020, 2:26:14 PM] [Yeelight] 0x0000000012ae4373: failed communication attempt 2 after 400ms.
[10/22/2020, 2:26:14 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:14 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:14 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:14 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:14 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:15 PM] [Yeelight] 0x0000000010e34268: failed communication attempt 3 after 800ms.
[10/22/2020, 2:26:15 PM] [Yeelight] 0x0000000012ae4358: failed communication attempt 3 after 800ms.
[10/22/2020, 2:26:15 PM] [Yeelight] 0x000000001259b8b0: failed communication attempt 3 after 800ms.
[10/22/2020, 2:26:15 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:15 PM] [Yeelight] 0x00000000125a2f63: failed communication attempt 3 after 800ms.
[10/22/2020, 2:26:15 PM] [Yeelight] 0x0000000012ae4373: failed communication attempt 3 after 800ms.
[10/22/2020, 2:26:15 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:15 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:15 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:15 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:16 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:16 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:16 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:16 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:16 PM] [Yeelight] 0x0000000010e34268: failed communication attempt 4 after 1600ms.
[10/22/2020, 2:26:16 PM] [Yeelight] 0x0000000012ae4358: failed communication attempt 4 after 1600ms.
[10/22/2020, 2:26:16 PM] [Yeelight] 0x000000001259b8b0: failed communication attempt 4 after 1600ms.
[10/22/2020, 2:26:16 PM] [Yeelight] 0x00000000125a2f63: failed communication attempt 4 after 1600ms.
[10/22/2020, 2:26:16 PM] [Yeelight] 0x0000000012ae4373: failed communication attempt 4 after 1600ms.
[10/22/2020, 2:26:16 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:16 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:16 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:16 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:16 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:16 PM] [Yeelight] 0x000000001259b8b0: failed communication attempt 0 after 100ms.
[10/22/2020, 2:26:16 PM] [Yeelight] 0x0000000010e34268: failed communication attempt 0 after 100ms.
[10/22/2020, 2:26:16 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:16 PM] [Yeelight] 0x00000000125a2f63: failed communication attempt 0 after 100ms.
[10/22/2020, 2:26:16 PM] [Yeelight] 0x0000000012ae4373: failed communication attempt 0 after 100ms.
[10/22/2020, 2:26:17 PM] [Yeelight] 0x0000000012ae4358: failed communication attempt 0 after 100ms.
[10/22/2020, 2:26:17 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:17 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:17 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:17 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:17 PM] [Yeelight] 0x000000001259b8b0: failed communication attempt 1 after 200ms.
[10/22/2020, 2:26:17 PM] [Yeelight] 0x0000000010e34268: failed communication attempt 1 after 200ms.
[10/22/2020, 2:26:17 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:17 PM] [Yeelight] 0x00000000125a2f63: failed communication attempt 1 after 200ms.
[10/22/2020, 2:26:17 PM] [Yeelight] 0x0000000012ae4373: failed communication attempt 1 after 200ms.
[10/22/2020, 2:26:17 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:17 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:17 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:17 PM] [Yeelight] 0x0000000012ae4358: failed communication attempt 1 after 200ms.
[10/22/2020, 2:26:17 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:17 PM] [Yeelight] 0x000000001259b8b0: failed communication attempt 2 after 400ms.
[10/22/2020, 2:26:17 PM] [Yeelight] 0x0000000010e34268: failed communication attempt 2 after 400ms.
[10/22/2020, 2:26:17 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:17 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:17 PM] [Yeelight] 0x00000000125a2f63: failed communication attempt 2 after 400ms.
[10/22/2020, 2:26:17 PM] [Yeelight] 0x0000000012ae4373: failed communication attempt 2 after 400ms.
[10/22/2020, 2:26:17 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:17 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:17 PM] [Yeelight] 0x0000000012ae4358: failed communication attempt 2 after 400ms.
[10/22/2020, 2:26:17 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:18 PM] [Yeelight] 0x000000001259b8b0: failed communication attempt 3 after 800ms.
[10/22/2020, 2:26:18 PM] [Yeelight] 0x0000000010e34268: failed communication attempt 3 after 800ms.
[10/22/2020, 2:26:18 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:18 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:18 PM] [Yeelight] 0x00000000125a2f63: failed communication attempt 3 after 800ms.
[10/22/2020, 2:26:18 PM] [Yeelight] 0x0000000012ae4373: failed communication attempt 3 after 800ms.
[10/22/2020, 2:26:18 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:18 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:18 PM] [Yeelight] 0x0000000012ae4358: failed communication attempt 3 after 800ms.
[10/22/2020, 2:26:18 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:20 PM] [Yeelight] 0x0000000010e34268: failed communication attempt 4 after 1600ms.
[10/22/2020, 2:26:20 PM] [Yeelight] 0x000000001259b8b0: failed communication attempt 4 after 1600ms.
[10/22/2020, 2:26:20 PM] [Yeelight] 0x00000000125a2f63: failed communication attempt 4 after 1600ms.
[10/22/2020, 2:26:20 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:20 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:20 PM] [Yeelight] 0x0000000012ae4373: failed communication attempt 4 after 1600ms.
[10/22/2020, 2:26:20 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:20 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:20 PM] [Yeelight] 0x0000000012ae4358: failed communication attempt 4 after 1600ms.
[10/22/2020, 2:26:20 PM] [Yeelight] <IP_ADDRESS> closed. error? false.
[10/22/2020, 2:26:20 PM] [Yeelight] 0x0000000012ae4358: failed communication attempt 5 after 3200ms.
[10/22/2020, 2:26:20 PM] [Yeelight] 0x0000000012ae4358: failed to send cmd 3 after 5 retries.
[10/22/2020, 2:26:20 PM] [Yeelight] 0x000000001259b8b0: failed communication attempt 5 after 3200ms.
[10/22/2020, 2:26:20 PM] [Yeelight] 0x000000001259b8b0: failed to send cmd 1 after 5 retries.
[10/22/2020, 2:26:20 PM] [Yeelight] 0x0000000010e34268: failed communication attempt 5 after 3200ms.
[10/22/2020, 2:26:20 PM] [Yeelight] 0x0000000010e34268: failed to send cmd 2 after 5 retries.
[10/22/2020, 2:26:20 PM] [Yeelight] 0x00000000125a2f63: failed communication attempt 5 after 3200ms.
[10/22/2020, 2:26:20 PM] [Yeelight] 0x00000000125a2f63: failed to send cmd 5 after 5 retries.
[10/22/2020, 2:26:20 PM] [Yeelight] 0x0000000012ae4373: failed communication attempt 5 after 3200ms.
[10/22/2020, 2:26:20 PM] [Yeelight] 0x0000000012ae4373: failed to send cmd 4 after 5 retries.
[10/22/2020, 2:26:23 PM] [Yeelight] 0x0000000010e34268: failed communication attempt 5 after 3200ms.
[10/22/2020, 2:26:23 PM] [Yeelight] 0x0000000010e34268: failed to send cmd 7 after 5 retries.
[10/22/2020, 2:26:23 PM] [Yeelight] 0x000000001259b8b0: failed communication attempt 5 after 3200ms.
[10/22/2020, 2:26:23 PM] [Yeelight] 0x000000001259b8b0: failed to send cmd 6 after 5 retries.
[10/22/2020, 2:26:23 PM] [Yeelight] 0x00000000125a2f63: failed communication attempt 5 after 3200ms.
[10/22/2020, 2:26:23 PM] [Yeelight] 0x00000000125a2f63: failed to send cmd 10 after 5 retries.
[10/22/2020, 2:26:23 PM] [Yeelight] 0x0000000012ae4373: failed communication attempt 5 after 3200ms.
[10/22/2020, 2:26:23 PM] [Yeelight] 0x0000000012ae4373: failed to send cmd 9 after 5 retries.
[10/22/2020, 2:26:23 PM] [Yeelight] 0x0000000012ae4358: failed communication attempt 5 after 3200ms.
[10/22/2020, 2:26:23 PM] [Yeelight] 0x0000000012ae4358: failed to send cmd 8 after 5 retries.
[10/22/2020, 2:26:23 PM] [Yeelight] All known devices found. Stopping proactive search.
Any idea what's going on?

Upon restarting the bulbs (cutting off the power to them and then reapply power), this seems to fix the issue. However, if I restarted the homebridge server or my raspberry pi, the issue appears again.
And as a note the bulbs appear just fine on Yeelight app (and on Home.app for those added directly to Home, not HB)

I have the same issue. Except I use HOOBS. This plugin is relible when HOOBS or Hb is reliable 100%. If HOOBS or Hb services fails to start for what ever reason, I would have to unplug/plug the lamps to get them working again.
This only happens with the 3 Bedside Lamps. No issues with the 5 colored bulbs.

This seems related to #89 but it will be hard to debug as I don't have a bedside lamp to test. I would suggest trying to tune the number of retries and the timeout in config.json (see readme).
    "connection": {
      "retries": 5, // 3 to 5 should work
      "timeout": 500 // try values from 250 to 1500
    },


Make sure next-head class isn't added to title

Since the title doesn' need the class this makes sure it isn't added
Closes: #7212

Sounds good, added.


Search result

Tell me please... I still don't get it. If the program finds a key pair, how will I know about it? Where they can be seen. In which folder, in which file? You could explain in detail and show.

From the questions and answers described below, I still did not understand exactly where the record of the find would be made.

Hi, it will be shown in the console. If you are using a console pipe to a file like shown in my examples, the output will be redirected to a file: https://github.com/bernardladenthin/BitcoinAddressFinder/blob/a8306f2dca3abc1f97cdce479a599aa98249c626/examples/run_Find_1OpenCLDevice.bat#L14

Thanks, I'll try to figure it out.  Is it possible to limit the CPU usage using some settings? I would like him to complete the task with a load of 50%. Let it take longer, but in a calm mode. The result in this case is more influenced by luck, not speed. I don't use a video card, I don't want to download it. i5 8400 processor, 6 cores, 3.7 GHz.

You can't easily limit the CPU load. If you are using Windows, you can use the power settings to change the maximum power to something like 99% to prevent a clock burst and high energy consumption. If you just want to limit the amount, you can reduce producer threads. The consumer threads will sleep if the consumer doesn't produce key pairs fast enough.

Thanks!

If you can make a small clarification.. Yes, I run the program through the file " run_Find_6 CPU Producer.bat" . So you mean that in this case the result will not be written to some text file, but simply displayed in a window? I'm sorry for being stupid, I'm not a programmer..

If you are using a run-file like this:
https://github.com/bernardladenthin/BitcoinAddressFinder/blob/a8306f2dca3abc1f97cdce479a599aa98249c626/examples/run_Find_8CPUProducer.bat#L14
you are seing this line:
config_Find_8CPUProducer.js >> log_Find_8CPUProducer.txt 2>&1
You can change the line to:
config_Find_8CPUProducer.js
You will see any output in the console and not in the file log_Find_8CPUProducer.txt
Thats how you can switch between direct output on the console or pipe to a file.

One last question, and I'll stop torturing you). Yes, I see these lines and I won't change anything. I will leave the recording channel in the file log_Find_8CPUProducer.txt . But earlier you said that when the keys are detected, the program continues to work. This means that this file continues to be filled with data and increases in size. Is there any understanding where exactly to look for this record, or will it be made at the very beginning?

Your questions are welcome. Exactly, the program is searching all the time until you stop it. You see a hit counter all the time: [Hits: 0]
For testing purposes, you can provoke a hit because there exist some addresses in the lower bit range:
https://github.com/bernardladenthin/BitcoinAddressFinder/blob/a8306f2dca3abc1f97cdce479a599aa98249c626/examples/config_Find_8CPUProducer.js#L24
As an example, change to 20 bit "privateKeyMaxNumBits" : 20
Do you see hits in your console/log?

privateKeyBigInteger: [198669] privateKeyBytes: [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 8, 13]] privateKeyHex: [000000000000000000000000000000000000000000000000000000000003080d] WiF: [KwDiBf89QgGbjEhKnhXJuH7LrciVrZi3qYjgd9M7rFyWkjT5fywW] publicKeyAsHex: [020ce4a3291b19d2e1a7bf73ee87d30a6bdbc72b20771e7dfff40d0db755cd4af1] publicKeyHash160Hex: [ad1e852b08eba53df306ec9daa8c643426953f94] publicKeyHash160Base58: [1GnNTmTVLZiqQfLbAdp9DVdicEnB5GoERE] Compressed: [true] Mnemonic: [abandon, abandon, abandon, abandon, abandon, abandon, abandon, abandon, abandon, abandon, abandon, abandon, abandon, abandon, abandon, abandon, abandon, abandon, abandon, abandon, abandon, account, cage, predict]
Here is the result, the detection record is performed in order. This means that in a text file, which at some point will be large, I will have to try very hard to check for such an entry manually among a large array of monotonous data. I found the answer to the question that was bothering me). Thanks.

Although it is possible to reduce the time interval for writing a report to a file. Not after 10 seconds, but for example 120.

Great news, you are right, just change this parameter:
https://github.com/bernardladenthin/BitcoinAddressFinder/blob/a8306f2dca3abc1f97cdce479a599aa98249c626/examples/config_Find_8CPUProducer.js#L13

Thank you, good luck to us)!

Hi.  There is such a task https://privatekeys.pw/puzzles/bitcoin-puzzle-tx Tell me, if I need to scan the private key space in the 66-bit range, do I just need to change the 256 settings to 66 in the file? And whether the lightweight address database affects the search. It's probably better to download the full version. Is your program suitable for this at all?

The lightweight is enough. Exactly, you can change the address range to e.g., 64 to look in 1 up to 64 bits only, where puzzle transactions will be found. Start with a range of 20 bits or smaller to test your setup first.


Desync on Dedicated Server

Bug Report
Basic Information
(First of all: sorry for my english)
We are experiencing some desync with your mission on Dedicated server, the desync is between 0,5 seg to 1 seg and ocurs randomly between 15 and 50 segs. This is not the first Sandbox Insurgency we have made, we worked before with ALiVe only with some DropFPS.
**Mission version: Liberation Malden 0.96.5
**Map used: Malden
**Mods used: 2035 Russian Armed Forces, 3den Enhanced, Ace, Ace 3 extension (animations and actions), Ace 3 extension (Placeables), Ace Comapt RHS AFRF, Ace Compat RHS USAF, Ace Compat RHS GREF, ACEX, ADV ACE CPR, ADV ACE Splint, Advanced rapelling, Advanced towing, ASR AI3, Backpack on Chest, CBA_A3, CUP ACE3 Compatibility addon - terrains, Cup terrains core, Cup terrains Maps, Enhanced Movement, HiddenIdentityPack V2, JSRS Sound Mod, RHSAFRF, RHSGREF, RHSSAF, RHSUSAF, Specialist Military Arms, TFAR, VSM Collection, Zeus Enhanced.
**Dedicated or local game: Dedicated Server
**Source of the mission file (URL): https://github.com/KillahPotatoes/KP-Liberation

Check the size of your vars file on the server. Anything further to Discord.


[BUG] rc5 ISO Installer Hangs at fb0: switching to i915 from EFI VGA or variant of fb0: switching to i915 from VESA VGA

Describe the bug
This seems entirely dependent on the underling hardware.
So not easy to reproduce. Unless a ThinkCentre M93p is handy.
Also may not be typical use-case for anyone trying Harvester.
Marking as "severity/needed" due to, it seems not common for anyone using Harvester as traditionally those using Harvester would probably be using more performant Hardware.
On an Intel Xeon E3-1265L v3:
https://www.intel.com/content/www/us/en/products/sku/75463/intel-xeon-processor-e31265l-v3-8m-cache-2-50-ghz/specifications.html
Lenovo ThinkCentre M93p
w/ 16GB 1333 MH` DDR Memory installed.
And 1 Kingston SSD 500GB.
The rc5 iso installation fails at boot with either:

Legacy (non UEFI BIOS Boot)
UEFI BIOS Boot from Grub2 ( chainloader /boot/grub2/x86_64/grub.efi & boot from the usb stick)

The "Legacy" non-uefi boot yields the message of:
fb0: switching to i915 from VESA VGA
Whereas, the UEFI boot yields the meassage of:
fb0: switching to i915 from EFI VGA
Both either selecting the Regular or 1024*768 Boot Option.
To Reproduce
Entirely hardware dependent, not easy to reproduce.
Expected behavior
I've had no issues in the past installing and running harvester on the physical machine.  Recently seems like on it's prior master-head installation rke2-server.service seems to not start
Support bundle
No support bundle available, currently rke2-server.service on the prior master-head install is hung - supportconfig -ka yielded:
scc_lenovo_c_04_04_23.tar.gz
Environment

Harvester ISO version: master-head (old, 4M + old)
Underlying Infrastructure: ThinkCentre M93P

Additional context

Hi irishgordo,
I had the same problem and The problem is that the driver of this intel card is not available. I fix it putting "nomodeset"  in the GRUB config.
Specifically, when the GRUB menu appears on boot,  type "e" key and add "nomodeset" at the end "$linux" command line and perform the boot with "CTRL+X or F10" button.
This avoid the switch of graphic card maintaining the "generic VGA controller" and the system starts.
If you want to setup this config permanently after the system starts, follow the isructionthis link configure kernel parameter.
Hope it helps.

I have Same Problem and solved it following above instruction. But Harvester team should take care this issue. It is happening on latest hardware. (No problem with 5-10 years old computer.)


Not a issue but a suggesion.

Would you like to work together. I'm building the mono repo for resizable-panels.
You can integrate your repo into and use the resizable-core also.
A common repo can increase our hit count :) .
https://github.com/BipanKishore/resizable-panes
If you liked it do give it a star and do check the demo.

I guess you don't need that
https://corvu.dev/docs/primitives/resizable/
these guys used my code and integrated it into this library.

ok


Special functions tests

Most functions tested. A few functions added to autodiff.hpp.

Codecov Report

Merging #10 into develop will decrease coverage by 31.49%.
The diff coverage is 83.37%.

@@            Coverage Diff             @@
##           develop     #10      +/-   ##
==========================================
- Coverage      100%   68.5%   -31.5%
==========================================
  Files            3       9       +6
  Lines         1583    2813    +1230
==========================================
+ Hits          1583    1927     +344
- Misses           0     886     +886

Impacted Files
Coverage Δ

test/test_autodiff5.cpp
46.41% <ø> (ø)

include/boost/math/differentiation/autodiff.hpp
94.81% <ø> (-5.19%)
:arrow_down:

test/test_autodiff2.cpp
100% <100%> (ø)

test/test_autodiff.hpp
100% <100%> (ø)

test/test_autodiff1.cpp
100% <100%> (ø)

test/test_autodiff3.cpp
100% <100%> (ø)

test/test_autodiff4.cpp
100% <100%> (ø)

test/test_autodiff6.cpp
51.23% <51.23%> (ø)

...lude/boost/math/differentiation/autodiff_cpp11.hpp
100% <0%> (ø)
:arrow_up:

... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f9d3452...cc8104c. Read the comment docs.


Retrieve BJJ from wallet

If a user wants to get the BJJ associated to his account from the wallet to share it with a third party, it needs to do Wallet->My Account -> view in batch explorer.
I would like the user to be able to retrieve this BJJ without the explorer.

Wrong place to post issue


[WIP] Add angle delta for H_PRED and V_PRED

For reference, these are the changes I made locally while reviewing:

Tidy up matching rules
Fix interaction with chroma-from-luma
De-duplicate angles in tests

I think the interaction between AngleDeltaUV and CflAlphaU/CflAlphaV already exists - adding angle delta for  H_PRED and V_PRED just increases the probability of seeing it in tests.
I would like to align the names and ordering of PredictionMode values in the code with the AV1 spec. I can wait for your PR to land before applying that, if you prefer.

I would like to align the names and ordering of PredictionMode values in the code with the AV1 spec. I can wait for your PR to land before applying that, if you prefer.

Now its merged, please align the names and ordering of PredictionMode values.

Sorry, I have not checked this page.
I only checked PR page and its already merged.

Now its merged, please align the names and ordering of PredictionMode values.

Will be done in: xiph/rav1e#2048
Thank you for adding angle deltas to rav1e.


Allow style can be set on TouchableOpacity elements inside SegmentedItem

Allow style can be set on TouchableOpacity elements inside SegmentedItem elements.
This will solve the issue that bigger size Badges may be covered by the content of adjacent SegmentedItem.
Set zIndex to Badge element won’t help — it seems zIndex on children elements of a TouchableOpacity will be ignored (tested on iOS 10.3 on a iPhone 6 emulator).
Setting zIndex to TouchableOpacity directly will work, however.
If you have better solution, please ignore this pull request and let me know.

It is not the best solution, I have modified the code to solve this bug: https: //github.com/rilyu/teaset/commit/cd66c88f13c4b9258f801e99f36fc43331ec9eb4
It will be published to npm later, you can install it like this: npm install teaset@git+https://github.com/rilyu/teaset.git

Many thanks. Your solution works better. Closed the pull request.


Update Theme Repo Link in Docs

Closes #3402

@MattRudy : Thanks for your contribution! The author(s) have been notified to review your proposed change.

#label:"aq-pr-triaged"

@maggiesMSFT : Thanks for your contribution! The author(s) have been notified to review your proposed change.

Thanks, @MattRudy !
#sign-off


API: expose Fluent APIs for Twitch / IGDB

Once the Twitch client is functional, consider refactoring it as a Fluent API.
Inspiration could be Azure Fluent, with an extension methods-based approach, and a single Authentication interface usable by both the Twitch API, Twitch IRC and IGDB clients.

Twitch and IGDB now refactored as fluent.
For IRC using OAuth token directly, still have to figure out if the abstractions are a good fit.

Dealt with IRC client as well


Add support for Create2 which returns the Stat struct on success.

Add new op and error code.
Implement Create2 as a separate call since it is not backward compatible.
Note that this requires version 3.5.x of the server to exercise the test cases.

Coverage decreased (-0.6%) to 80.6% when pulling 42c751e530995551136a19228e8fe994a8fbadba on msolo:msolo-add-create2-support into 87e1bca4477a3cc767ca71be023ced183d74e538 on samuel:master.


c/c++ functions lexed incorrectly

Using pygments 2.8. In both of these snippets the first has_value function is lexed correctly, but the second has_value is lexed incorrectly as Name.
int has_value();
SomeClassConstructor() {}
int has_value();

int has_value();
~SomeClassDestructor() {}
int has_value();

I think matching constructors and destructors like functions (going to the function state after the match) can fix this.
Another issue is that in the following code func is lexed as Name and not Name.Function and std is lexed as Label:
[[using std: nodiscard]] int func();

I think matching and handling attribute sequences before matching functions can solve this.
Also in both of these lines, value is lexed incorrectly as Name and not Name.Function:
auto value() { return 1; }
unsigned int value() { return 1; }

I added examples for more problematic snippets.
All these cases have a potential 1 line fix / very easy fix (apart from the constructors / destructors snippet which I am not sure how to tackle). @Anteru maybe putting a 'good first issue' label on this can be a good idea?


Fix log level

scipio command doesn't show info log when verbose  is not specified from https://github.com/giginet/Scipio/commit/9fb11bf27ae650576edccafb6696012ba096af09 .
I asked @giginet about the issue in person, and he said it's a bug.
In this PR, I fixed the log level.

Original logic in 18.3.0.
https://github.com/giginet/Scipio/blob/f9a3acc02c8e260ad3e55770d23776ed5646c44c/Sources/ScipioKit/Runner.swift#L38-L42


speedup for make_gaussian_blobs

make_gaussian_blobs is a really nice addition to pyRiemann, but it is computationally expensive. The complexity depends mainly on the number of matrices to generate. The sampling code is a loop that generate one matrix at each iteration. As the samples are indenpendent, we could easily parallelize the process to reduce computation time. This is the object of this PR.
NB: this could reduce a lot the doc building time:
computation time summary:
    - ../examples/simulated/plot_classifier_comparison.py:                235.60 sec   0.0 MB
    - ../examples/stats/plot_oneWay_Manova.py:                             67.49 sec   0.0 MB
    - ../examples/artifacts/plot_detect_riemannian_potato_field_EEG.py:    49.95 sec   0.0 MB
    - ../examples/artifacts/plot_detect_riemannian_potato_EEG.py:          37.31 sec   0.0 MB
    - ../examples/simulated/plot_toy_classification.py:                    15.94 sec   0.0 MB
    - ../examples/ERP/plot_classify_MEG_mdm.py:                            11.12 sec   0.0 MB
    - ../examples/ERP/plot_embedding_MEG.py:                               11.03 sec   0.0 MB
...

@sylvchev can you say how much n_jobs helps by giving running time you obtain? 🙏

@sylvchev can you say how much n_jobs helps by giving running time you obtain? 🙏

for plot_toy_classification.py, with n_jobs=1:
13 s ± 438 ms per loop (mean ± std. dev. of 3 runs, 1 loop each)
and with n_jobs=8:
5.88 s ± 940 ms per loop (mean ± std. dev. of 3 runs, 1 loop each)

@plcrodrigues merge if happy :)


[move-prover] Refactor Abort

Motivation
This PR moves Abort from an Operation into a top-level Bytecode.  This change allows an Abort to end a basic block.  Since such a basic block will have no outgoing edges, any propagation in dataflow analysis is naturally and easily avoided.
Have you read the Contributing Guidelines on pull requests?
Yes.
Test Plan
Existing tests
Related PRs

@bors-libra r=wrwg

:pushpin: Commit f25561c has been approved by wrwg

:hourglass: Testing commit f25561c37c878773e316ac8dc46d9117aecace09 with merge b2d226e5ecac290260e3b9a54dca44dd681d6f65...

:sunny: Test successful - checks-actions_land_blocking_test, checks-circle_commit_workflow
Approved by: wrwg
Pushing b2d226e5ecac290260e3b9a54dca44dd681d6f65 to master...


feat: add sql schema and migrations for the ENS

Description
This PR adds the SQL database tables schema migration files for the sqlx according to the SPECS.
Resolves #405
How Has This Been Tested?
When the server is started the migration procedure is invoked in #412 . The expected result is a successful server start.
Due Diligence

[ ] Breaking change
[ ] Requires a documentation update
[ ] Requires a e2e/integration test update

I think the structure needs to be changed to support historical names. Also "blockchain" to "namespace"

Gave my thoughts in the comment.


How can I add a range type where answer will be right if it's within given range? for eg.- 10<Ans<15

Done it.