[BUG]: Weird OOM bug

🐛 Describe the bug
I used 8 GPUs, 24G*8, to train 7B llama, but reported OOM error.
I run the examples/train_sft.sh to start a supervised instructs fine-tuning.
torchrun --standalone --nproc_per_node=4 train_sft.py
--pretrain "/path/to/LLaMa-7B/"
--model 'llama'
--strategy colossalai_zero2
--log_interval 10
--save_path  /path/to/Coati-7B
--dataset /path/to/data.json
--batch_size 2
--accimulation_steps 8
--lr 2e-5
--max_datasets_size 512
--max_epochs 1 \
ERROR:
steps:   0%|                                                                                | 0/4 [00:00<?, ?it/s]use_cache=True is incompatible with gradient checkpointing. Setting use_cache=False...
steps:  50%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌                                                                                | 2/4 [02:32<02:32, 76.26s/it]Traceback (most recent call last):
File "train_sft.py", line 184, in
train(args)
File "train_sft.py", line 155, in train
trainer.fit(logger=logger, log_interval=args.log_interval)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/coati/trainer/sft.py", line 110, in fit
self.strategy.optimizer_step(self.optimizer)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/coati/trainer/strategies/colossalai.py", line 154, in optimizer_step
optimizer.step()
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/torch/optim/lr_scheduler.py", line 68, in wrapper
return wrapped(*args, **kwargs)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/colossalai/zero/sharded_optim/low_level_optim.py", line 480, in step
self.optim.step()
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/torch/optim/optimizer.py", line 140, in wrapper
out = func(*args, **kwargs)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/colossalai/nn/optimizer/hybrid_adam.py", line 112, in step
state['exp_avg'] = torch.zeros_like(p, dtype=torch.float, device=target_device)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.12 GiB (GPU 4; 23.70 GiB total capacity; 21.98 GiB already allocated; 356.56 MiB free; 22.23 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
wandb: Waiting for W&B process to finish... (failed 1).
Traceback (most recent call last):
File "train_sft.py", line 184, in
train(args)
File "train_sft.py", line 155, in train
trainer.fit(logger=logger, log_interval=args.log_interval)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/coati/trainer/sft.py", line 110, in fit
self.strategy.optimizer_step(self.optimizer)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/coati/trainer/strategies/colossalai.py", line 154, in optimizer_step
optimizer.step()
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/torch/optim/lr_scheduler.py", line 68, in wrapper
return wrapped(*args, **kwargs)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/colossalai/zero/sharded_optim/low_level_optim.py", line 480, in step
self.optim.step()
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/torch/optim/optimizer.py", line 140, in wrapper
out = func(*args, **kwargs)
Traceback (most recent call last):
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/colossalai/nn/optimizer/hybrid_adam.py", line 112, in step
state['exp_avg'] = torch.zeros_like(p, dtype=torch.float, device=target_device)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.17 GiB (GPU 1; 23.70 GiB total capacity; 22.11 GiB already allocated; 270.56 MiB free; 22.31 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
File "train_sft.py", line 184, in
train(args)
File "train_sft.py", line 155, in train
trainer.fit(logger=logger, log_interval=args.log_interval)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/coati/trainer/sft.py", line 110, in fit
self.strategy.optimizer_step(self.optimizer)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/coati/trainer/strategies/colossalai.py", line 154, in optimizer_step
optimizer.step()
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/torch/optim/lr_scheduler.py", line 68, in wrapper
return wrapped(*args, **kwargs)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/colossalai/zero/sharded_optim/low_level_optim.py", line 480, in step
self.optim.step()
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/torch/optim/optimizer.py", line 140, in wrapper
out = func(*args, **kwargs)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/colossalai/nn/optimizer/hybrid_adam.py", line 112, in step
state['exp_avg'] = torch.zeros_like(p, dtype=torch.float, device=target_device)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.12 GiB (GPU 5; 23.70 GiB total capacity; 22.00 GiB already allocated; 292.56 MiB free; 22.29 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
wandb: Waiting for W&B process to finish... (failed 1).
wandb: Waiting for W&B process to finish... (failed 1).
Traceback (most recent call last):
File "train_sft.py", line 184, in
train(args)
File "train_sft.py", line 155, in train
trainer.fit(logger=logger, log_interval=args.log_interval)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/coati/trainer/sft.py", line 110, in fit
self.strategy.optimizer_step(self.optimizer)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/coati/trainer/strategies/colossalai.py", line 154, in optimizer_step
optimizer.step()
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/torch/optim/lr_scheduler.py", line 68, in wrapper
return wrapped(*args, **kwargs)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/colossalai/zero/sharded_optim/low_level_optim.py", line 480, in step
self.optim.step()
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/torch/optim/optimizer.py", line 140, in wrapper
out = func(*args, **kwargs)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/colossalai/nn/optimizer/hybrid_adam.py", line 112, in step
state['exp_avg'] = torch.zeros_like(p, dtype=torch.float, device=target_device)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.17 GiB (GPU 0; 23.70 GiB total capacity; 22.12 GiB already allocated; 250.56 MiB free; 22.33 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
wandb: Waiting for W&B process to finish... (failed 1).
Traceback (most recent call last):
File "train_sft.py", line 184, in
train(args)
File "train_sft.py", line 155, in train
trainer.fit(logger=logger, log_interval=args.log_interval)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/coati/trainer/sft.py", line 110, in fit
self.strategy.optimizer_step(self.optimizer)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/coati/trainer/strategies/colossalai.py", line 154, in optimizer_step
optimizer.step()
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/torch/optim/lr_scheduler.py", line 68, in wrapper
return wrapped(*args, **kwargs)
Traceback (most recent call last):
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/colossalai/zero/sharded_optim/low_level_optim.py", line 480, in step
self.optim.step()
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/torch/optim/optimizer.py", line 140, in wrapper
out = func(*args, **kwargs)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/colossalai/nn/optimizer/hybrid_adam.py", line 112, in step
state['exp_avg'] = torch.zeros_like(p, dtype=torch.float, device=target_device)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.14 GiB (GPU 7; 23.70 GiB total capacity; 22.03 GiB already allocated; 200.56 MiB free; 22.38 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
File "train_sft.py", line 184, in
train(args)
File "train_sft.py", line 155, in train
trainer.fit(logger=logger, log_interval=args.log_interval)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/coati/trainer/sft.py", line 110, in fit
self.strategy.optimizer_step(self.optimizer)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/coati/trainer/strategies/colossalai.py", line 154, in optimizer_step
optimizer.step()
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/torch/optim/lr_scheduler.py", line 68, in wrapper
return wrapped(*args, **kwargs)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/colossalai/zero/sharded_optim/low_level_optim.py", line 480, in step
self.optim.step()
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/torch/optim/optimizer.py", line 140, in wrapper
out = func(*args, **kwargs)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/colossalai/nn/optimizer/hybrid_adam.py", line 112, in step
state['exp_avg'] = torch.zeros_like(p, dtype=torch.float, device=target_device)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.12 GiB (GPU 3; 23.70 GiB total capacity; 21.97 GiB already allocated; 356.56 MiB free; 22.23 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
wandb: Waiting for W&B process to finish... (failed 1).
wandb: Waiting for W&B process to finish... (failed 1).
Traceback (most recent call last):
File "train_sft.py", line 184, in
train(args)
File "train_sft.py", line 155, in train
trainer.fit(logger=logger, log_interval=args.log_interval)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/coati/trainer/sft.py", line 110, in fit
self.strategy.optimizer_step(self.optimizer)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/coati/trainer/strategies/colossalai.py", line 154, in optimizer_step
optimizer.step()
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/torch/optim/lr_scheduler.py", line 68, in wrapper
return wrapped(*args, **kwargs)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/colossalai/zero/sharded_optim/low_level_optim.py", line 480, in step
self.optim.step()
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/torch/optim/optimizer.py", line 140, in wrapper
out = func(*args, **kwargs)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/colossalai/nn/optimizer/hybrid_adam.py", line 112, in step
state['exp_avg'] = torch.zeros_like(p, dtype=torch.float, device=target_device)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.12 GiB (GPU 2; 23.70 GiB total capacity; 21.98 GiB already allocated; 248.56 MiB free; 22.33 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
wandb: Waiting for W&B process to finish... (failed 1).
Traceback (most recent call last):
File "train_sft.py", line 184, in
train(args)
File "train_sft.py", line 155, in train
trainer.fit(logger=logger, log_interval=args.log_interval)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/coati/trainer/sft.py", line 110, in fit
self.strategy.optimizer_step(self.optimizer)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/coati/trainer/strategies/colossalai.py", line 154, in optimizer_step
optimizer.step()
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/torch/optim/lr_scheduler.py", line 68, in wrapper
return wrapped(*args, **kwargs)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/colossalai/zero/sharded_optim/low_level_optim.py", line 480, in step
self.optim.step()
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/torch/optim/optimizer.py", line 140, in wrapper
out = func(*args, **kwargs)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/colossalai/nn/optimizer/hybrid_adam.py", line 112, in step
state['exp_avg'] = torch.zeros_like(p, dtype=torch.float, device=target_device)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.14 GiB (GPU 6; 23.70 GiB total capacity; 22.04 GiB already allocated; 138.56 MiB free; 22.44 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
wandb: Waiting for W&B process to finish... (failed 1).
wandb: - 0.000 MB of 0.000 MB uploaded (0.000 MB deduped)
wandb: Run history:
wandb: batch_id ▁█
wandb:    epoch ▁▁
wandb:     loss █▁
wandb:       lr █▁
wandb:
wandb: Run summary:
wandb: batch_id 15
wandb:    epoch 0
wandb:     loss 0.17278
wandb:       lr 2e-05
wandb:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/qihaoran/coati/ColossalAI-main/applications/Chat/examples/wandb/offline-run-20230410_041849-esiyxc2g
wandb: Find logs at: ./wandb/offline-run-20230410_041849-esiyxc2g/logs
wandb: - 0.000 MB of 0.000 MB uploaded (0.000 MB deduped)
wandb: Run history:
wandb: batch_id ▁█
wandb:    epoch ▁▁
wandb:     loss █▁
wandb:       lr █▁
wandb:
wandb: Run summary:
wandb: batch_id 15
wandb:    epoch 0
wandb:     loss 0.16051
wandb:       lr 2e-05
wandb:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/qihaoran/coati/ColossalAI-main/applications/Chat/examples/wandb/offline-run-20230410_041852-g3a9ilfd
wandb: Find logs at: ./wandb/offline-run-20230410_041852-g3a9ilfd/logs
wandb: - 0.000 MB of 0.000 MB uploaded (0.000 MB deduped)
wandb: Run history:
wandb: batch_id ▁█
wandb:    epoch ▁▁
wandb:     loss █▁
wandb:       lr █▁
wandb:
wandb: Run summary:
wandb: batch_id 15
wandb:    epoch 0
wandb:     loss 0.16887
wandb:       lr 2e-05
wandb:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/qihaoran/coati/ColossalAI-main/applications/Chat/examples/wandb/offline-run-20230410_041854-7efbymx2
wandb: Find logs at: ./wandb/offline-run-20230410_041854-7efbymx2/logs
wandb: - 0.000 MB of 0.000 MB uploaded (0.000 MB deduped)
wandb: Run history:
wandb: batch_id ▁█
wandb:    epoch ▁▁
wandb:     loss ▁█
wandb:       lr █▁
wandb:
wandb: Run summary:
wandb: batch_id 15
wandb:    epoch 0
wandb:     loss 0.19582
wandb:       lr 2e-05
wandb:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/qihaoran/coati/ColossalAI-main/applications/Chat/examples/wandb/offline-run-20230410_041854-hfz6wf7b
wandb: Find logs at: ./wandb/offline-run-20230410_041854-hfz6wf7b/logs
wandb:
wandb: Run history:
wandb: batch_id ▁█
wandb:    epoch ▁▁
wandb:     loss █▁
wandb:       lr █▁
wandb:
wandb: Run summary:
wandb: batch_id 15
wandb:    epoch 0
wandb:     loss 0.17409
wandb:       lr 2e-05
wandb:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/qihaoran/coati/ColossalAI-main/applications/Chat/examples/wandb/offline-run-20230410_041904-5r6581c1
wandb: Find logs at: ./wandb/offline-run-20230410_041904-5r6581c1/logs
wandb: - 0.000 MB of 0.000 MB uploaded (0.000 MB deduped)
wandb: Run history:
wandb: batch_id ▁█
wandb:    epoch ▁▁
wandb:     loss █▁
wandb:       lr █▁
wandb:
wandb: Run summary:
wandb: batch_id 15
wandb:    epoch 0
wandb:     loss 0.16742
wandb:       lr 2e-05
wandb:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/qihaoran/coati/ColossalAI-main/applications/Chat/examples/wandb/offline-run-20230410_041856-fy350tdc
wandb: Find logs at: ./wandb/offline-run-20230410_041856-fy350tdc/logs
wandb:
wandb: Run history:
wandb: batch_id ▁█
wandb:    epoch ▁▁
wandb:     loss █▁
wandb:       lr █▁
wandb:
wandb: Run summary:
wandb: batch_id 15
wandb:    epoch 0
wandb:     loss 0.15266
wandb:       lr 2e-05
wandb:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/qihaoran/coati/ColossalAI-main/applications/Chat/examples/wandb/offline-run-20230410_041850-du00duko
wandb: Find logs at: ./wandb/offline-run-20230410_041850-du00duko/logs
wandb: - 0.000 MB of 0.000 MB uploaded (0.000 MB deduped)
wandb: Run history:
wandb: batch_id ▁█
wandb:    epoch ▁▁
wandb:     loss ▁█
wandb:       lr █▁
wandb:
wandb: Run summary:
wandb: batch_id 15
wandb:    epoch 0
wandb:     loss 0.18481
wandb:       lr 2e-05
wandb:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/qihaoran/coati/ColossalAI-main/applications/Chat/examples/wandb/offline-run-20230410_041854-1vli5lv6
wandb: Find logs at: ./wandb/offline-run-20230410_041854-1vli5lv6/logs
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 345752 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 345753 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 345754 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 345755 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 345757 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 345758 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 345759 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 4 (pid: 345756) of binary: /home/qihaoran/.conda/envs/coati1/bin/python
Traceback (most recent call last):
File "/home/qihaoran/.conda/envs/coati1/bin/torchrun", line 8, in
sys.exit(main())
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 346, in wrapper
return f(*args, **kwargs)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/torch/distributed/run.py", line 762, in main
run(args)
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/torch/distributed/run.py", line 753, in run
elastic_launch(
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 132, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/qihaoran/.conda/envs/coati1/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
train_sft.py FAILED
Environment
torch:1.13.1+cu117
cuda：11.7
coati    1.0.0
transformers  4.28.0.dev0
colossalai   0.2.8

I have already solved this problem by using 'train_sft. py',set  placement_policy='cpu'


WIP: Add NESO package to repo

A Spack package has been written for NESO. This allows Spack to install it. It also allows us to easily ensure Spack will install all of the correct dependencies. Some modifications were made to the dependency packages to help facilitate this.

@will-saunders-ukaea would you mind reviewing this please?

Looking at the NESO package: I made some progress yesterday with setting up the build environment with different sycl implementations here: https://github.com/ExCALIBUR-NEPTUNE/NESO-Spack/blob/dev-neso-particles/packages/neso-particles/package.py.
It's annoying that the intel-oneapi-compilers package doesn't provide sycl. Another issue is that if you set the compiler (for your spack spec) to %oneapi then the find_package(IntelDPCPP REQUIRED) will fail as the intel-oneapi-compilers package doesn't set the CMAKE_PREFIX_PATH when it's used as a compiler. I don't know if this ultimately matters - we've found out that for mkl linking having -DCMAKE_CXX_COMPILER=dpcpp is important (i.e. icpx instead of dpcpp will cause segfaults). For NESO-Particles I've experimentally set the find_package to QUIET for IntelDPCPP and simply set the CXX compiler and it seems to work.
If we really need the compiler to be set to %oneapi and need the find_package(IntelDPCPP) to work then it looks like we're going to end up having (or work out how to avoid) installing intel-oneapi-compilers%oneapi which is messy if you end up installing intel-oneapi-compilers twice.
The Hipsycl builds seem more straightforward - we need a clean way to set the hipsycl-target and go from there.

If we really need the compiler to be set to %oneapi and need the find_package(IntelDPCPP) to work then it looks like we're going to end up having (or work out how to avoid) installing intel-oneapi-compilers%oneapi which is messy if you end up installing intel-oneapi-compilers twice.

There is a better way to do this, as desribed in the spack documentation. You can install the oneapi package and then tell spack to register it as a compiler in its own right.
spack install intel-oneapi-compilers
spack load intel-oneapi-compilers
spack compiler find

I know, that's not the issue here. The issue is getting find_package(IntelDPCPP) to work and if we need it work. Just doing %oneapi does not set the env vars such that find_package(IntelDPCPP) works (maybe you see a way around this).

Your doing all the same sensible things we did, Chris. We found these bugs and work arounds for them. We've done the work arounds because we assume they're easier than changing them oneapi spack package et al.

Trying to build with dpcpp unfortunately causes problems for one of the transitive dependencies (berkeley-db). That package uses autotools to manage the build and there is a known bug when trying to use dpcpp with autotools. Autotools gets a bit confused and thinks some temporary files generated during the configuration stage need to be linked into the shared object library. However, as those object files get deleted after configuration, it can't do this.
I'm trying to think of potential solutions:

provide a wrapper for dpcpp that filters out these files, if present; awkward for users
adjust the configuration files after the configuration step; requires editing the spack package for berkeley-db
anything else?

Is this the berkeley-db dependency of openblas? If so the neso nektar++ package can (tries to) avoid the openblas dependency with +mkl to enable MKL when using the intel tool chain.
If this doesn't work: does berkeley-db build using icc/icpc as I doubt we need to use dpcpp for berkeley-db (seriously doubt it has a sycl interface we use - or any sycl at all).

Is this the berkeley-db dependency of openblas?

Nope, it's a dependency of perl, which is a dependcy of openssl, which is a dependency of cmake.
I'm not sure if I can get spack to mix dependencies compiled using icpc with those compiled using dpcpp in the same build, unfortunately.

Does a spec like:
spack spec neso.nektar %oneapi ^cmake%gcc

Solve the problem for the time being? (seems to have a promising looking spec tree on my spack 0.18.1)

Does a spec like:
spack spec neso.nektar %oneapi ^cmake%gcc

Solve the problem for the time being? (seems to have a promising looking spec tree on my spack 0.18.1)
Thanks, I didn't actually know Spack would let you do that! The spec tree didn't quite work for me when I did it that way (the way I've set things up anyway), but did with

spack spec neso%dpcpp ^berkeley-db%gcc

Now to see if it will actually build successfully...

@jwscook @will-saunders-ukaea Have either of you been able to build numpy with the oneAPI compilers? On my computer it fails (at link-time, I think). I strongly suspect this is just because the linger is a memory-hog and my computer runs out of RAM/swap space. I've confirmed that the same issue occurs when trying to build numpy with clang (on which oneAPI is based), though not with the classic Intel compilers.

Tried it (spack install --reuse py-numpy%oneapi) and some intel-clang binary got to 22GB usage before I killed it to avoid an oom crash.

I haven't tried this, I confess.

I think I have a reasonably sensible approach in place now for building with OneAPI/DPC++. I created a dummy package to represent DPC++/satisfy the SYCL dependency. It sets a bunch of environment variables etc. so that building with it in the NESO package "just works". There are still a few minor things to tidy up (possibly left for another time) and you need to make sure you have some of the right environment variables at run-time (this might be fixable from within the NESO binary). Plus you still have to compile BOOST and Numpy with GCC to avoid the memory problems already mentioned. But I think things are basically working.

Could you augment the readme, please, do add a short copy-pasteable guide to building nektar with common variants e.g. oneapi & mkl vs hipsycl & openblas?

Could you augment the readme, please, do add a short copy-pasteable guide to building nektar with common variants e.g. oneapi & mkl vs hipsycl & openblas?

Do you mean the README of this repo or the main NESO repo? Also, Nektar doesn't use SYCL so the oneapi vs. hipsycl consideration doesn't apply.

Do you mean the README of this repo or the main NESO repo? Also, Nektar doesn't use SYCL so the oneapi vs. hipsycl consideration doesn't apply.

I did mean this repo, but you're right, the NESO README should be brought up-to-date.
Ignore my comment about nektar and SYCL - that's just a hangover from getting it all to work before all this...

I did mean this repo, but you're right, the NESO README should be brought up-to-date.

I have made extensive additions to the NESO README as part of ExCALIBUR-NEPTUNE/NESO#93. If there is anything you think worth adding, probably best to add a note there.
I've expanded the README of this repo a bit now too.

I'm just trying the build described by https://github.com/ExCALIBUR-NEPTUNE/NESO/pull/93 and if that's successful I reckon this is good to merge.

I've successfully spack installed, built and run the tests with hipsycl.

Building the oneapi spec with  ^intel-oneapi-mpi and doing export I_MPI_FABRICS=shm to get it to build (so the compiler works at build time) and export SYCL_DEVICE_FILTER=host before running the tests on my gpu-less intel laptop, it all works. Phew!

I successfully built the intel path with this: https://github.com/will-saunders-ukaea/docker-workspace/blob/11db8f41c01df143d5fcf6dad9204217634e60ee/neso-intel-test-branch/Dockerfile (I added the ^intel-oneapi-mpi to the spec).

As far as I'm concerned, this is ready to merge. @jwscook and/or @will-saunders-ukaea, feel free to hit the button if you are too.


⚠ DDoS-Relay US-W-sfo3-02 has degraded performance

In aee024b, DDoS-Relay US-W-sfo3-02 (http://<IP_ADDRESS>:42069/;csv) experienced degraded performance:

HTTP code: 200
Response time: 104 ms

Resolved: DDoS-Relay US-W-sfo3-02 performance has improved in f2b6e33 after 6 minutes.


'range(at:)' has been renamed?

DOTNETDateTimeFormatter.swift:47:85: 'range(at:)' has been renamed to 'rangeAt(_:)'
and have some same problems, how to fix?

What version are you using? Are you compiling with Swift 4?

@malcommac i'm using version 4.4.1 and have the same issue

The PR-487 to fix this, is added to the 4.4.2 release. Hopefully @malcommac will release it soon to Cocoapods.

Fixed in https://github.com/malcommac/SwiftDate/commit/e993a1b13ea9c9588e29c79b7543c310be88da63

Hey @malcommac the merging around, reverted this change and now it's still broken :/

"Installing SwiftDate 4.4.2 (was 4.1.10)"
Still have the "Pods/SwiftDate/Sources/SwiftDate/DOTNETDateTimeFormatter.swift:47:85: 'range(at:)' has been renamed to 'rangeAt(_:)'" error


[orgmode] Enable to export inline images with file scheme

[ Background ]
Currently we need to refer inline images with dedicated scheme img-url.
However, during editing with orgmode, Emacs orgmode can't recogine the scheme and it is inconvenient because they are invisible on Emacs.
[ Task ]
Support file scheme supported by Emacs orgmode.

Are you willing to provide a patch? (Nikola is not going to support /foo.png or foo.png in the way orgmode expects; if you want to make a file:// scheme, and it can be used with relative URLs, we can accept that patch.)

@Kwpolska
Yes, I have the patch. Could I push my working branch ?

@shishimaru Make a pull request.

@Kwpolska
I made the pull request. Could you check ?


Add From<uid_t> and From<gid_t>

Conversions to/from primitive uid_t and gid_t to newtype Uid and Gid types are valid and infallible, but are only implemented in one direction. This provides the counterparts in the other direction.
These conversions are identical in behavior to the from_raw methods. However, using the more idiomatic From trait enables easier interoperability with other crates (e.g., deserialization with serde)

All comments addressed. Good to go.

bors r+


Big flash of unstyled content (FOUC) during HMR

Describe the bug
Problem
When my styles change, the page flashes without any styling for a second or so, it's verry jarring I thought it was actually reloading.
Cause
After some debugging I found out the cause:
When a css stylesheet is loaded through  (i.e. in the main HTML page), the HMR logic is simply to update its href but that can cause issues, here's what happens in my project:

Multiple css are HMR at the same time, some are imported by JS, the main one is   in index.html.
At HMR time, Vite fetches the imported ones, to later replace

Got a PR up for this, as we ran into the same issue: https://github.com/vitejs/vite/pull/8495

It seems this does not happen (or rarely happens) on chrome. But it happens on firefox.

@sapphi-red For what it's worth, I'm having this issue in Edge Chromium.

@jods4 I confirmed #8495 fixes for me on firefox. Would you check if it fixes for you, too?

@sapphi-red I'll try when it's merged into a published release, I don't have the time to make my own builds right now.


Prevent automatic batching updates in ThemeProvider to ensure client use right colorMode

Closes #2731.
Use ReactDOM.flushSync to prevent automatic batching of state updates since React 18, which causes error described in #2731.
Screenshots
None.
Merge checklist

[x] Added/updated tests
[x] Added/updated documentation
[x] Tested in Chrome
[x] Tested in Firefox
[x] Tested in Safari
[x] Tested in Edge

Take a look at the What we look for in reviews section of the contributing guidelines for more information on how we review PRs.

Thanks for taking the time to make Pull Request @renbaoshuo! 🙌 In your opinion, would there be a good test that we could add to make sure this doesn't happen again in the future?

Thanks for taking the time to make Pull Request @renbaoshuo! 🙌 In your opinion, would there be a good test that we could add to make sure this doesn't happen again in the future?

Sorry, I have no idea.


Read Authenticator keys from a file

What

Option to read authenticator keys from a file and display TOTP

Option to store new authenticator key passed as argument or QR image to a file

To-do

[ ] Read Name/Label and Authenticator Key from file
[ ] Display TOTP for each Key in file
[ ] Display TOTP for requested key name/label from file
[ ] Update readme

hi, can you assign this issue to me? :)


📃: TO do list web application using html , css and js

🔴 Title : TO DO list
🔴 Tech stack : HTML , CSS , jAVASCRIPT
🔴 Objective : To manage daily work schedule
🔴 Summary :A clean and intuitive layout with an input field for new tasks, a button to add tasks, and a list to display tasks.
📸 Screenshots:

✅ Details to Include When Taking the Issue:
Name : Please mention GSSOC extended, Hacktoberfest
Participant Role (Specify the Open Source Program name, e.g., GSSOC, Hacktoberfest, etc.):

Happy Contributing! 🚀
Wishing you all the best on your open source journey. Enjoy! 😎

Sorry, someone is already working on this. Do not be demotivated and raise a new issue.


Dependency conflict

Currently when installing the latest version of WellcomeML v2020.7.0 the following error is thrown
ERROR: tensorflow 2.2.0 has requirement scipy==1.4.1; python_version >= "3", but you'll have scipy 1.5.1 which is incompatible.

This does not cause an error in installation or usage as far as i have seen but it could create problems in packages that depend on WellcomeML. It would be ideal to pin all versions of our packages as a starting point but possibly also add a frozen requirements.txt into setup.py.

I got something similar now:
ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.
We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

botocore 1.17.31 requires docutils<0.16,>=0.10, but you'll have docutils 0.16 which is incompatible.
tensorflow 2.3.0 requires numpy<1.19.0,>=1.16.0, but you'll have numpy 1.19.1 which is incompatible.
tensorflow 2.3.0 requires scipy==1.4.1, but you'll have scipy 1.5.2 which is incompatible.


Expand semantics of Array from MutableSequence to MutableMapping

 Issue by whitequark
Sunday May 12, 2019 at 13:25 GMT
Originally opened as https://github.com/m-labs/nmigen/issues/66

An Array is essentially a mux, and there are several equally valid ways to drive a mux. Currently only straight binary is supported. It would be nice to also have one-hot and priority encodings, since all of these are trivially representable directly in RTLIL and Verilog.
One thing to note is that one-hot and priority are one and the same unless parallel_case is used.

I've never seen anyone express any interest in this feature, and it's not trivial to implement because of the highly special treatment of ArrayProxy today. Because of that I'm closing the issue.
To implement this, Array should be rewritten so that ArrayProxy is a plain UserValue (perhaps we should call those CustomValue or something instead), and then it will be quite easy to expand its semantics, since it will no longer involve nMigen internals.

This has been requested on the Aug 3rd meeting: https://freenode.irclog.whitequark.org/nmigen/2020-08-03#1596478574-1596478045; reopening since the reason for closing was that "seemingly no one wants this".

This was reopened for the benefit of lkcl, who is no longer participating in the project. I think we can safely close this back as "no one (worth caring about) wants this".


switch to ginkgo v2

Thank you for submitting a pull request to the bytefmt repository!
We appreciate the contribution. To help us understand the context for your pull request, please fill out this template to the best of your ability.
Please make sure to complete all of the following steps

Check the Contributing document on how to sign the CLA and run tests.
Submit your PR to this repo.
Submit an accompanying PR Review Request referencing this PR so the Diego Team knows to review your pull request.

Note: this PR will not be reviewed unless you submit the PR Review Request.

Please provide the following information:
What is this change about?
update ginkgo to new major version v2
What problem it is trying to solve?
"outdated" dependency
What is the impact if the change is not made?
"outdated" dependency

@ameowlia are you able to merge these changes in?

Hi @phil9909,
We're seeing some conflicts on this PR. Are you able to resolve these?
Thanks
@MarcPaquette

Looks like someone stole my contribution :disappointed:. Switch to ginkgo v2 was already done in #59 (which was opened over a year after my PR).


feature: Provide a way to detect when navigation to a view-model is completed, even if no navigation-parameter is being passed

Is your feature request related to a problem? Please describe.
The problem I'm currently facing is as follows:

I'm implementing a detail view for recipes. This detail view can be accessed either from a list that shows all available recipes by directly passing a reference to the recipe to show, or by reference to the identifier of the recipe to show - in which case the recipe will be resolved when the view gets activated.
I've implemented WhenNavigatedTo(INavigationParameter) to pass said data - i.e. the recipe-object or its identifier - to the view-model.
If neither recipe-object or identifier has been specified, I want to inform the user of this fact by showing them an error message.

Now I can show the user an error when INavigationParameter doesn't contain the expected parameters, but when the view is pushed without INavigationParameter, WhenNavigatedTo doesn't get called at all. As far as I'm aware there's no way to detect either 1) whether WhenNavigatedTo will be called at some point following activation of the view-model; or 2) that WhenNavigatedTo will not be called at all. That is, other than simply processing WhenNavigatedTo, I can't predict in the view-model constructor or WhenActivated whether to wait for WhenNavigatedTo to occur and have any error messages be handled there, or know that, since WhenNavigatedTo is not going to be called anyway, I should trigger my error from the constructor or WhenActivated.
Describe the solution you'd like
Either:

call WhenNavigatedTo(INavigationParameter) with null to indicate the lack of parameters having been passed, but the navigation event having taken place; or
implement a separate handler, such as WhenNavigatedTo(), WhenNavigatedToWithoutParameter() or WhenNavigationCompleted(ToOrFrom), to indicate that navigation is complete, but without parameters.

Describe alternatives you've considered
Alternatives I've considered are:

Using the view-model's WhenActivated to check if my data has come in. This doesn't work, though, since WhenNavigatedTo consistently occurs after WhenActivated.
Consistently passing an empty NavigationParameter when I want to trigger the intended error message. This, however, defeats the purpose, as the error is supposed to catch exceptions in the program.
Setting a timer to verify whether data has come in. This, however, may provide false-positives, as the data might need to be retrieved through the network and thus end up taking longer than the time-out that was set.

Describe suggestions on how to achieve the feature
As proposed above, the existing interface and method can be used to achieve this effect by simply always calling the WhenNavigatedTo(INavigationParameter)-method when a new view-model is navigated to. This, however, may break existing code that either does a null-check on the incoming parameters, or otherwise simply assumes the parameter to be set. In this case, extending the INavigated-interface with a new method would solve the issue.

Hey @AlexanderMelchers , I was looking the code here and actually what you are requesting can be done already in some way. The WhenNavigatingTo and WhenNavigatedTo are called every time a push happens, the only pre-requisite to have that working is to be using the ParameterViewStackService and call the Push passing in yourself an empty dictionary as a parameter. If you instead call the Push methods without the parameter signature (from the ViewStackService) then indeed those won't be triggered, so I guess that's your case.
Ideally I'd say we should have some kind of special "hook" to have that in a more consise way, there's a feature request I opened for that here #197

I'd say this could be potentially closed and you could keep an eye on the #197 :)

@AlexanderMelchers Apologies for not commenting on this sooner.  What @winterdouglas says is accurate, the methods get called every navigation and there is nothing preventing you from hooking into them with an empty dictionary.  Some base ViewModel class that handles it will get you that information.
We are looking into ways to provide better life cycle methods for the ViewModel.  I feel like this is basically the same thing you are requesting.  I am going to close this issue as it is similar to the other feature request and solving one, should solve both.


[Symfony] Handle single action with __construct on InvokableControllerRector

Given the following code:
use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\Serializer\SerializerInterface;

final class SingleActionWithConstruct extends AbstractController
{
    public function __construct(
        private SerializerInterface $serializer,
    ) {
    }

    public function create(): Response
    {
        return $this->json([], Response::HTTP_CREATED);
    }
}

It currently produce error:

There was 1 error:

1) Rector\Symfony\Tests\Rector\Class_\InvokableControllerRector\InvokableControllerRectorTest::test with data set #1 (Symplify\SmartFileSystem\SmartFileInfo Object (...))
Undefined array key 0

Fixes https://github.com/rectorphp/rector/issues/7365

Fixed 🎉 /cc @Myks92

All checks have passed 🎉 @TomasVotruba it is ready for review.

Thank you :+1:


Dev pariterre

Fully Openable C3D and CSV files :)

csv part reviewed :+1:
Some major modifications were made since the PR (cf. commits).
c3d in production! :smiley:

PR ready to be tested :+1:


NatGrads notebook cleanup

Quick PR. ^^^ subj.

Codecov Report

Merging #696 into master will decrease coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #696      +/-   ##
==========================================
- Coverage   98.09%   98.07%   -0.03%
==========================================
  Files          48       48
  Lines        4105     4105
==========================================
- Hits         4027     4026       -1
- Misses         78       79       +1

Impacted Files
Coverage Δ

gpflow/params/parameterized.py
98.23% <0%> (-0.45%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e2acd61...64d8e4c. Read the comment docs.

Hugh should look at this as well, but I have points:

First mention of "Comparison with ordinary gradients in the non-conjugate case" should be "conjugate case" (since you're using a Gaussian likelihood)
Likelihoods in cells 115 and 116 look wrong.


Remove deprecated sklearn from requirements.txt

The package named sklearn has been deprecated and replaced by scikit-learn which is already in the list of dependencies. This PR removes the deprecated name.

RIP


Twitter Color Emoji (New formula)

Twitter released a set of the custom colored EMOJI they use across
their web and mobile platforms. This project has taken the SVG and
generated a font based on the SVG vectors.

The order of stanzas is wrong. To solve common stylistic issues, run brew cask style --fix {{your_cask_file}} prior to submission.

Thank you for the contribution. It needed some fixes, so they were made in 90e99d625099cbbc3bb0a50bd0bf1611ba0aa1ae. Your contribution is still included (and still credited to you), with the appropriate modifications. Please feel free to ask about any of the changes.

Great, thanks for making those fixes for me :+1:
I didn't see any mention of stanza ordering in the CONTRIBUTIONS guidelines .

I didn't see any mention of stanza ordering in the CONTRIBUTIONS guidelines.

It’s on the one from the main repo, which we tell you to consult first. However, those links were wrong, and I just corrected them.

@0xadada Git rebase, commit amends, or specifically setting git commit author.

Specifically in this case I made the fixes than fixup with rebase. More specifically even, I’ve use a custom-built script that makes it faster and easier.

@victorpopkov The script looks great! I need to implement it in my workflow :+1:

@eosrei brew install vitorgalvao/tiny-scripts/prfixmaster will get it. There’s also a prfixbranch that makes a new branch with the fixes instead of merging directly.

@victorpopkov Brew isn't an option, I'm on Linux :) I can handle it. Thanks!


Undo / redo support

I'm using this lovely little library to try to replace CodeMirror in the Ghost Editor with a plain textarea and I stumbled across the fact that making changes to a textarea with rangyinputs doesn't work with the browsers internal undo/redo.
I know that this is far from a simple problem to solve, and I'm also aware that you probably know far more about it than I do, seeing as researching the problem turned up this :grin:
As the answer to the question "can this be solved?" is "sort of", I was wondering if having support in rangyinputs would be welcomed either by default, or via an option?

hımm


Mac build

Checklist

[x ] Used a personal fork of the feedstock to propose changes
[ ] Bumped the build number (if the version is unchanged)
[x ] Reset the build number to 0 (if the version changed)
[ ] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x ] Ensured the license file is being packaged.

Now builds python-foton for macintosh.

@conda-forge-admin, please rerender


Serialize TargetName in TargetSkippedEventArgs

This was missed in #2777
Closes #2789
Not bumping the version, we didn't even do an official build between then and now.

I took the liberty and merged this :) I need this to update the viewer to take advantage of the latest changes.
Thanks a lot!


[bug]: Icons are not centered in the checkboxes

What version of design-system are you using?
Browser: Arc
What's Wrong?
Icons are not centered in the checkboxes, it's very visible in the intermediate state.

To Reproduce
Find any checkbox (ex: in the edit view table).
Expected Behaviour
Icons should be centered.

hey, could you assign me this issue

I found the issue rooted from the parent container missing
justify-content: center
align-items: center

Path: packages\design-system\src\components\Checkbox\Checkbox.tsx
What was changed:
Inline Styling for CheckboxIndicator
<CheckboxIndicator style={{ display: 'inline-flex', pointerEvents: 'auto', justifyContent: 'center', alignItems: 'center' }} forceMount>
          {checked === true ? <CheckIcon width="1.6rem" fill="neutral0" /> : null}
          {checked === 'indeterminate' ? <Minus fill="neutral0" /> : null}
        </CheckboxIndicator>

Pull request: https://github.com/strapi/design-system/pull/1821

Hey @PlanckConst, I answered in the PR, thanks again :)


[µTVM] Fix two warnings when deprecated forms are used

Hi,
This patchset addresses the following two warnings on microTVM:
[0]
[...]
/home/gromero/git/tvm/python/tvm/micro/contrib/zephyr.py:391: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  doc = yaml.load(f)
[...]

[1]
[...]
/home/gromero/git/tvm/python/tvm/target/target.py:460: UserWarning: tvm.target.create() is being deprecated. Please use tvm.target.Target() instead
  warnings.warn("tvm.target.create() is being deprecated. Please use tvm.target.Target() instead")
[...]

Warning [0] is related to the deprecated form (no loader specified) of yaml.load(input), while the second one is related to the use of  deprecated tvm.target.create().
Both warnings appear  when running micro_tflite.py. The first one [0] when TARGET = tvm.target.target.micro("stm32f746xx") and the second one [1] when TARGET = tvm.target.target.micro("host"), which is the default target.
Thanks for contributing to TVM!   Please refer to guideline https://tvm.apache.org/docs/contribute/ for useful information and tips. After the pull request is submitted, please request code reviews from Reviewers by @ them in the pull request thread.

@areusch Hi. Could you review that small patchset, please? For warning [0] I'm assuming microTVM is not locked with any old PyYAML version (i.e. < 5.1). If it's locked we could simply silence the warning, which is not so nice I think. Thanks!

@areusch Hi. Could you review that small patchset, please? For warning [0] I'm assuming microTVM is not locked with any old PyYAML version (i.e. < 5.1). If it's locked we could simply silence the warning, which is not so nice I think. Thanks!

thanks for fixing these @gromero! @tmoreau89 @tqchen

thanks for fixing these @gromero! @tmoreau89 @tqchen

Thanks @gromero @areusch the PR is merged.

Thanks @gromero @areusch the PR is merged.


RUMM-2607: Use event write context for Logs

What does this PR do?
This change brings the usage of event write context (EWC) to the Logs feature, by doing the following:

Logs feature now does the call to withWriteContext to get DatadogContext and the binary batch writer.
There is a step to make Logs feature extracted into a dedicated module: Crash reports feature and NDK crash reporting don't depend on the Logs Feature explicitly now, but they send the necessary request to write crash log via the message bus.
Since many values to create LogEvent is now taken from DatadogContext which is not available during the feature construction time, many values are moved out from the DatadogLogGenerator constructor.
If LogEvent should be created and written in the synchronous manner, caller should pass the syncWrite parameter with the event.
DataWriter for v2 now should accept 2 arguments for the write method - event which should be serialized, and EventBatchWriter which will write serialized event and provided by the withWriteContext call.

There is some duplication between NetworkInfo and UserInfo classes from v2 and network model packages, it will essentially go away later (we won't expose network models in the public API).
Review checklist (to be filled by reviewers)

[x] Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
[x] Make sure you discussed the feature or bugfix with the maintaining team in an Issue
[x] Make sure each commit and the PR mention the Issue number (cf the CONTRIBUTING doc)

Codecov Report

Merging #1103 (cd86390) into feature/sdkv2 (9b73f57) will decrease coverage by 0.23%.
The diff coverage is 89.86%.

@@                Coverage Diff                @@
##           feature/sdkv2    #1103      +/-   ##
=================================================
- Coverage          82.69%   82.46%   -0.23%
=================================================
  Files                350      351       +1
  Lines              11301    11362      +61
  Branches            1846     1874      +28
=================================================
+ Hits                9345     9369      +24
- Misses              1394     1424      +30
- Partials             562      569       +7

Impacted Files
Coverage Δ

...n/com/datadog/android/core/internal/CoreFeature.kt
91.09% <ø> (-0.35%)
:arrow_down:

...in/com/datadog/android/core/internal/SdkFeature.kt
88.18% <ø> (ø)

...atadog/android/log/internal/domain/LogGenerator.kt
80.00% <ø> (-3.33%)
:arrow_down:

...og/android/v2/core/internal/NoOpContextProvider.kt
3.57% <0.00%> (-0.78%)
:arrow_down:

.../android/v2/log/internal/storage/LogsDataWriter.kt
60.00% <60.00%> (ø)

.../android/error/internal/DatadogExceptionHandler.kt
81.08% <84.62%> (-4.63%)
:arrow_down:

...in/com/datadog/android/log/internal/LogsFeature.kt
87.10% <85.96%> (-12.90%)
:arrow_down:

...g/android/log/internal/logger/DatadogLogHandler.kt
90.91% <86.96%> (-0.76%)
:arrow_down:

.../src/main/kotlin/com/datadog/android/log/Logger.kt
94.02% <93.33%> (+0.12%)
:arrow_up:

...adog/android/error/internal/CrashReportsFeature.kt
100.00% <100.00%> (ø)

... and 18 more


Keyboard shortcut between TI elements

Summary
Quick jump to next selected TI element using a shortcut key, not a click.
Details
The functionality of the injector that lets you click on the default text between TI elements, then automatically selecting it, is excellent. After you're done typing, you then simply click on the next set of text and continue. It would be even better if you could just TAB to the next set of TI elements, like tabbing between fields works. The actual TAB key won't work, but something like the back tick would.
Additional Information

Create new JIRA issue.
Select a different issue type, or use the selected one.
Press TAB to get to the Description field. Or, click on the Description field.
Press the back tick key ```
First set of TI elements is selected.
Start typing your words.
Press the back tick key ```
Next set of TI elements is selected.
Rinse, repeat.

I'm happy to contribute code as I am able to if this gets accepted as a valid enhancement.

This is a great idea. We will work on implementing this. Thanks for the feedback!

This has been implemented. Closing the issue.
Thanks for the great idea!


Set up CI

grp_test
Available groups in the GitHub Actions container:

Linux: docker adm systemd-journal
macOS: staff wheel everyone localaccounts _appserverusr admin _appserveradm _lpadmin _webdeveloper com.apple.access_ssh com.apple.sharepoint.group.1 _appstore _lpoperator _developer _analyticsusers com.apple.access_ftp com.apple.access_screensharing com.apple.access_remote_ae

pwd_test
Fails on macOS:
00:10 +12 ~3 -1: test/src/pwd_test.dart: pwd ... [E]
  Unsupported operation: Operation 'toDartString' not allowed on a 'nullptr'.
  package:ffi/src/utf8.dart 62:7                 Utf8Pointer._ensureNotNullptr
  package:ffi/src/utf8.dart 42:5                 Utf8Pointer.toDartString
  package:posix/src/util/conversions.dart 46:36  copyCBuffToDartString
  package:posix/src/pwd.dart 275:20              _buildPasswd
  package:posix/src/pwd.dart 123:18              getpwnam
  test/src/pwd_test.dart 8:18                    main.<fn>
  test/src/pwd_test.dart 7:19                    main.<fn>

Possibly because of a memory layout mismatch.
https://linux.die.net/man/3/getpwnam
struct passwd {
  char   *pw_name;       /* username */
  char   *pw_passwd;     /* user password */
  uid_t   pw_uid;        /* user ID */
  gid_t   pw_gid;        /* group ID */
  char   *pw_gecos;      /* user information */
  char   *pw_dir;        /* home directory */
  char   *pw_shell;      /* shell program */
};

vs.
https://www.freebsd.org/cgi/man.cgi?query=getpwnam
struct passwd {
  char	   *pw_name;	   /* user name	*/
  char	   *pw_passwd;	   /* encrypted	password */
  uid_t   pw_uid;	   /* user uid */
  gid_t   pw_gid;	   /* user gid */
  time_t  pw_change;	   /* password change time */
  char	   *pw_class;	   /* user access class	*/
  char	   *pw_gecos;	   /* Honeywell	login info */
  char	   *pw_dir;	   /* home directory */
  char	   *pw_shell;	   /* default shell */
  time_t  pw_expire;	   /* account expiration */
  int	   pw_fields;	   /* internal:	fields filled in */
};

sysinfo_test
Fails on macOS
00:10 +12 ~3 -1: loading test/src/sysinfo_test.dart
00:11 +12 ~3 -1: loading test/src/sysinfo_test.dart
00:11 +12 ~3 -1: test/src/sysinfo_test.dart: sysinfo
00:11 +12 ~3 -2: test/src/sysinfo_test.dart: sysinfo [E]
  Invalid argument(s): Failed to lookup symbol 'sysinfo': dlsym(0x117949710, sysinfo): symbol not found
  dart:ffi                         DynamicLibrary.lookup
  package:posix/src/sysinfo.dart   sysinfo
  test/src/sysinfo_test.dart 6:18  main.<fn>
  test/src/sysinfo_test.dart 5:19  main.<fn>

P.S. I've tested the CI with my fork: https://github.com/jpnurmi/dart_posix/pull/1


feat: implementing v-pre transform

v-pre
// template

{{a}}

// generates
export default function render() {
  const _ctx = this
  return (openBlock(), createBlock("div", null, "{{a}}"))
}

// template

{{ a }}

{{ a }}

    <component is="div"></component>

// generates
export default function render() {
  const _ctx = this
  return (openBlock(), createBlock("div", null, [
    createVNode("div", null, "{{ a }}"),
    createVNode("div", null, toString(_ctx.a), 1 /* TEXT */),
    createVNode("div", null, "<component is=\"div\"></component>")
  ]))
}

Notes: I made the implementation on the compiler-core, when started doing that was thinking in not parsing the children if I found the v-pre but it turn to be hard to find where the current node ends without parsing the children. The transformer possible can be moved to compiler-code if is decided to keep the implementation there.
This implementation can be moved to be a NodeTransform if needed

I think it will be better v-pre to be on the parser since we can actually make the parser ignore the content instead of doing at the later stage.
The only issue of having it at the compiler is to parse the children.
Also should this stay at compiler-dom or be moved to compiler-core?

Yes, v-pre should be handled at parser level

Skip interpolation parsing
Parse all elements as plain elements (with tagType: ElementTypes.ELEMENT)
Parse all attributes as plain attributes (with type: NodeTypes.ATTRIBUTE)

This should be in compiler-core, directly in src/parse.ts.

Closed via 5dfb271 - still, thanks for the contribution!


more granular development instructions

I believe the current development instructions https://github.com/yahoo/athenz/blob/master/docs/dev_environment.md are too "monolith" - they tell you to run mvn install.  Would it be possible to break them into more granular pieces - for example, if someone wants to do development on the Go client pieces, what they need and whether it is possible to avoid dealing with the whole Java "bulk"?

Also:  Should I expect to be able to build according to the development instructions on a Mac? Or should I expect success only on a Linux box of some kind?


B 17960 PPM Closeout Calculations Breakout Display Main

Agility ticket
Integration PR
#12072
Summary
This PR updates the PPM Closeout "Review Documents" workflow to include more information about a PPM shipment and the costs/calculations behind the incentives for it. There is now a "Shipment Info" dropdown that shows general info such as move dates, ZIPs, and mileage.
Then the "Incentives/Costs" section breaks down the GCC cost, gross incentive, any advances received, and any remaining incentive owed to the customer.
Finally, the "GCC Factors" section includes relevant info that contributes to the value of the "GCC (Government Constructed Cost)", such as haul prices and packing expenses.
How to test

Access http://milmovelocal:3000/sign-in and sign in as a new customer

Create a new PPM move with one or more PPM shipments. Take note of certain values when making the shipments (not when making profile info), such as

Origin / Destination Zips
Planned move date
Estimated weight
Pro gear self and spouse estimated weight (if applicable)
Estimated incentive
Advance amount requested (if applicable)

If making multiple shipments be sure to note these values for each shipment. Finish up and sign the form to finish creating the move. Take note of the move code.

Login as a SC at http://officelocal:3000/sign-in and click on the new move.

Edit the orders to include needed info.

Click "Edit Shipment" and put in the estimated SIT weight if applicable (NOTE: Be sure to put in the estimated SIT weight, start and end date if the customer marked that SIT is expected, if not these fields will be stuck at null in the DB and SIT related expenses will not be taken into account for calculations).

If an advance was requested, click through to the next screen and approve it.

Click "Submit Move Details"

Go back to the customer with the move and click "Upload PPM documents". Take note of all of the values you put in for these fields throughout the whole process, as they will be needed to verify the calculations later. If you indicated things like "Pro gear" be sure to add weights for those things, and I recommend putting in different "actual" values on these screens than what you put for "estimated", this will help you verify that the calculation screen is showing the "actual" value and not the "estimated" one.

You can also add other misc. expenses that will show in the review screen. These will also contribute to the incentive totals.

After signing for the PPM documents, go back to the SC and access the move under the PPM Closeout tab. Click "Review Documents" on the shipment(s).

Before approving any receipts, you should see a Shipment Info dropdown, click and verify that the values look correct.

After approving all receipts, before clicking "Confirm", you should see two new sections ("Incentives" and "GCC Factors"), check to make sure that these at least make sense (i.e. GCC should always be higher or equal to total incentive).

You should also see the correct breakdown for all documents/tickets:

Test that fields such as GCC update when you go back to the "Trip" weight receipt and update the allowable weight. GCC uses the max allowable weight to calculate the max allowable incentive amount.
for Trip tickets should result in a lower incentive, vice versa for full weight. Rejecting a Trip weight ticket should also lower the amount.

The Happo test is showing an error. See the following:

I don't know if that's an issue or not

Might also need to verify if the INT PR was merged in before the rebase of integration branch. We may have to re-PR that INT branch if so.

main pr verification rpt:
B17960.verify_rpt.txt

Closing for now, will re-open once storybook fixes and main updates are merged into INT and tested.

Verify rpt.
Per the dev, the changes in the pr are correct and complete.


Add buildArgs to service configuration and capture build configuration

Add buildArgs to service configuration and capture build configuration as global property to setup msbuild project.
From a configuration like this
...
services:
- name: projectA
  project: ProjectA/ProjectA.csproj
  buildArgs: --configuration Debug

The build args are appened to the build command line and the supported global property Configuration is read from that command line and used it to create the msbuild project instance.

Instead of build args it should be a dictionary of properties.

Ok, I will implement this with a dictionary. But there are entries in the dictionary that need to be transtaled as build command arguments. In this case, I extracted the "dictionary" only for the supported build configuration from the build args.
A question: What to do with the all properties in the dictionary? I know what to do with the build Configuration. But the others?

Ok, I will implement this with a dictionary. But there are entries in the dictionary that need to be transtaled as build command arguments. In this case, I extracted the "dictionary" only for the supported build configuration from the build args.

They can all be translated via /p:{Key}={Value}, it's just msbuild properties. You can choose to recognize some as first class but I wouldn't at the moment.

I tried something different for the moment, added properties but only build Configuration as first class

Also translated extra properties as /p:{Key}={Value}

This change needs a test, my guess is that this will fail with --docker as well.

This change needs a test, my guess is that this will fail with --docker as well.

Will check this one.

This change needs a test, my guess is that this will fail with --docker as well.

Added support for --docker option, working on the test


object_id is no longer used

object_id is no longer used by the current version of Packer:
==> azure-arm: Running builder ...
    azure-arm: Creating Azure Resource Manager (ARM) client ...
    azure-arm: You have provided Object_ID which is no longer needed, azure packer builder determines this dynamically from the authentication token

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 1bb18731-b0b4-f92b-7ece-031a84c76d7b
Version Independent ID: 27047a2a-a07b-cadb-fab5-ecc928b46717
Content: How to create Windows Azure VM Images with Packer
Content Source: articles/virtual-machines/windows/build-image-with-packer.md
Service: virtual-machines-windows
GitHub Login: @cynthn
Microsoft Alias: cynthn

@jkodroff Thanks for your feedback. I have assigned the issue to the content author to evaluate and update as appropriate.

All the instances of object _id needs to be removed as object_id is no longer needed.

@jkodroff thank you for the contribution! It looks like the object_id was removed at some point, so I believe the topic is correct now. Thank you!
#please-close


Enhancement Request: Could we set a bool to a chart that will display a legend for a pie chart?

 - I would love if I could have a legend on the left or right side of this pie chart if possible

This should be added to all charts.

closed becuase this is included in #11


Output gamma is very different with post-processing on or off

On Android it is very useful to disable post-processing for low-end devices as it practically doubles the frame rate, but without the tone mapping pass the output is very dark.
Post Processing On

Post Processing Off

Are there any obvious workarounds for boosting the darker view without a second pass of all the pixels? I'm presuming it's because the output buffer is in linear color space and we are relying on the a post-processing pass to map it to sRGB, in which case could we use something like GL_FRAMEBUFFER_SRGB?

With newer versions of Filament you should simply use the linear tone mapper on the ColorGrading API to achieve what you want (and turn off all other post-processing options). Using GL_FRAMEBUFFER_SRGB brings a host of other issues we'd rather avoid.


Page recognition never works properly

I have used this app a lot, since it is open source and I want to support it. However, It never correctly recognized the actual document borders. Not once in about 100 scans. I went through the trouble of manually selecting the document many times -- but I canont do it anymore. It's just too much work.
Is this expected behaviour? None of this is documented or reported as a Github Issue, which is hard for me to believe since multiple reviews on the Play Store mention it. If this is a shortcoming -- fair enough, it's an open source project. But this should definitely be documented, to prevent dissappointment in this open source project.
If this is rather an isolated issue that me and a handful of others are experiencing, it would be great to know how to troubleshoot it. There are no "settings" in the app, hence I'm unsure how to change any configuration.
Otherwise a great app -- although I only use it to take pictures and group them in Folders/Create PDFs for now, although it is a scanner app. I want to give this app as many chance as possible, but at some point I may have to switch back to properly functioning proprietary solutions.
Since I am a Java Developer I may also be able to contribute to a bugfix if pointed in the right direction.

Yes, same issue. Otherwise the app is great.
I am afraid the fix will not be so easy. It needs some kind of AI I guess.

The edge detection is kind of tricky. In the live version OpenCV is being used through JNI on the android side of implementation to execute the C++ code which detects the edges.
We are refactoring it to directly use the java implementation of OpenCV but our testing has shown little improvements over the existing edge detection. So we are implementing and testing alternatives such as BoofCV.
Alternatives/suggestions are welcome.
We are also refactoring the code to make it more friendly to contributions, but as you may be able to tell, all of these are huge tasks.

Agreed +1 on this.
Its not just that it doesn't get the edges right, opens can just fails every time I don't have a perfectly rectangular page.
Use case/repro : I'd like to scan my post/mail documents, which are always usually folded to fit inside envelopes. I take out the papers, unfold the and try to straighten. I then try to get good lighting, and place them very flatly on the counter.
I then scan the doc, and instead of just failing edge detection, it quits and restarts the entire app almost every time. Its suuuuuper painful and makes the app unusable in almost all cases.
Even when its a 8.5x11 or A4 printed document it still has trouble, even in good lighting.
I noticed that its exceptionally buggy when I'm trying to scan the top paper of a stack (then the edges are not 100% perfect, as there are edges from papers underneath not perfectly aligned).
Also receipts from restaurants and gas stations often have a ripped edge - which also causes failure


cannot find mechanism to override marshalling behavior

In our project we use some fairly complex typescript types.  When exposing or consuming these types in the generated python, we'd like to be able to control how the marshalling works.  For example, we have a type that can represent something that is similar to  Promise.  We'd like to be able to marshall that to/from a Python asyncio.Task or asyncio.Future.
Is such a thing possible?  I can see in the .net side of thing there is an IJsiiToFrameworkConverter, and it may be possible to construct things (maybe using ServiceContainer) to use our own custom converter.
However, it's unclear to me how i would do the same for python.  Is there a jsii 'escape hatch' as it were to give control over the sorts of common TS concepts that JSII currently can't handle?

Ditto, asyncio support would rock.  In fact, I think if we can this thing batching sufficiently across the ipc/marshaling boundary, combined w/ head use of ValueTask in .net core, and python asyncio Queue, etc., we could get really good throughput.


Downloads for Caltech datasets are broken

🐛 Bug
Download for Caltech101 and Caltech256 is broken.
To Reproduce
from torchvision import datasets

datasets.Caltech101(".", download=True)

collecting ... Downloading https://drive.google.com/file/d/137RyRjvTBkBiIfeYBNZBtViDHQ6_Ewsp/view to ./caltech101/101_ObjectCategories.tar.gz
RuntimeError: File not found or corrupted.

As you can see from the output, the download from http://www.vision.caltech.edu/Image_Datasets/Caltech101/101_ObjectCategories.tar.gz is redirected to Google Drive. download_url is not able to handle this.
This could be resolved by #3245 which dispatches the call to download_file_from_google_drive if necessary.
cc @pmeier

Fixed in #3245.


SCHEDULE_EXACT_ALARM permission support

FAQs

Overview

Android 12 SCHEDULE_EXACT_ALARM permission is not supported

Thx! would you mind sending a PR or proposing an API interface?

I will


[Bug]: web plugin throws exception NoSuchMethodError: tried to call a non-function, such as null: 'this.share'

Platform
Chrome Web
Plugin
share_plus
Version
6.3.0
Flutter SDK
3.3.10
Steps to reproduce
It doesn't want to share a picture from a specified directory on web. Additionally, the landing page for the plugin appears to give examples for deprecated Share.shareFiles.
I couldn't get shareXFiles to share on Android from a specified directory and had to write the picture down to a temporary file first. That does not work on web, path provider isn't supported. The example given on the landing page of pub.dev claims you can just specify a directory for the image as I have done.

It would be great to have a full working example of how to share an image on Flutter web.
It would also be great to have an example of a fallback if the particular browser the op employs doesn't support sharing pictures. Or better have this done automatically - it just falls back to subject and text. I'm not sure falling back to downloading an image is that useful or intuitive.

I'm aware there was some chat about 'Well of course sharing static images doesn't work'. But so far as I can see that's what's proposed on pub.dev's landing page.
Apologies if this is entirely my ineptitude.
Code Sample
try {
                        var path;
                        path = 'assets/slideshow/${presentLocation.pictureList[0]}';

                        await Share.shareXFiles([
                          XFile(path,
                              mimeType:
                                  "image/${presentLocation.pictureList[0].split(".")[1]}",
                              name: presentLocation.name)
                        ],
                                subject: presentLocation.name +
                                    presentLocation.emoji,
                                text: shareStr,
                                sharePositionOrigin:
                                    box!.localToGlobal(Offset.zero) & box.size)
                            .then((shareResult) async => {
                                  if (shareResult.status ==
                                      ShareResultStatus.success)
                                    {
                                      await MyApp.analytics.logShare(
                                          contentType: "URL shared",
                                          itemId: presentLocation.name,
                                          method: "with image")
                                    }
                                });
                      } catch (exception) {
                        print(
                            "Tried to share with image, hit exception: $exception");
}

Logs
Tried to share with image, hit exception: NoSuchMethodError: tried to call a non-function, such as null: 'this.share'

Flutter Doctor
Tried to share with image, hit exception: NoSuchMethodError: tried to call a non-function, such as null: 'this.share'

Checklist before submitting a bug

[X] I Google'd a solution and I couldn't find it
[X] I searched on StackOverflow for a solution and I couldn't find it
[X] I read the README.md file of the plugin
[X] I'm using the latest version of the plugin
[X] All dependencies are up to date with flutter pub upgrade
[X] I did a flutter clean
[X] I tried running the example project

I am facing a similar issue on web platform.
share_plus: 7.2.1
I am creating Xfile from data
return XFile.fromData(
                  fileEntity.bytes!,
                  mimeType: fileEntity.mimeType,
                  length: fileEntity.length,
                  lastModified: fileEntity.lastModifiedDate,
                  name: fileEntity.fileName,
                );

Future<ShareResult> shareXFiles from share_plus_web.dart is called. Here there the file is not empty as bytes length of 386353. The execution fails on await _navigator.share


window size

hello!
I have a (80000*8) data, I want to know how to calculate the size of window

I can slove your problem,please contact me via<EMAIL_ADDRESS>

hello!
I have a (80000*8) data, I want to know how to calculate the size of window

I can slove your problem,please contact me via<EMAIL_ADDRESS>


Fix docs: custom plugin user guide

The gradleApi()'s configuration actually the api not the implementation: JavaGradlePluginPlugin.java
Also it conflicts with correct java_gradle_plugin docs about the configuration.

Fixes #?
Context

Contributor Checklist

[x] Review Contribution Guidelines
[x] Make sure that all commits are signed off to indicate that you agree to the terms of Developer Certificate of Origin.
[ ] Make sure all contributed code can be distributed under the terms of the Apache License 2.0, e.g. the code was written by yourself or the original code is licensed under a license compatible to Apache License 2.0.
[x] Check "Allow edit from maintainers" option in pull request so that additional changes can be pushed by Gradle team
[ ] Provide integration tests (under <subproject>/src/integTest) to verify changes from a user perspective
[ ] Provide unit tests (under <subproject>/src/test) to verify logic
[ ] Update User Guide, DSL Reference, and Javadoc for public-facing changes
[ ] Ensure that tests pass locally: ./gradlew <changed-subproject>:check

Gradle Core Team Checklist

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation
[ ] Recognize contributor in release notes

Thanks @EntryPointKR!


hello-snack from tutorial is not working

Description
https://snack.expo.dev/@wcandillon/hello-snack?platform=web
snack from https://shopify.github.io/react-native-skia/docs/getting-started/web/#snack

yes that's a very good catch, we're in the process of fixing this, in
the meantime you can refer to:
https://staging-snack.expo.dev/eg-j1fbAxvV0D9xt8hMEW
On Wed, Feb 14, 2024 at 9:47 PM JacobDel @.***> wrote:

Description
@.***/hello-snack?platform=web
snack from https://shopify.github.io/react-native-skia/docs/getting-started/web/#snack
—
Reply to this email directly, view it on GitHub or unsubscribe.
You are receiving this email because you are subscribed to this thread.
Triage notifications on the go with GitHub Mobile for iOS or Android.

this has been fixed now, thks to expo :)


Attached done tasks

Add your profile picture to the folder
Add your profile information

please can you rename your markdown file with .md at the end? @LakshithaSamod

Done it


Profile Picture low quality sometimes

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like
A clear and concise description of what you want to happen.
PFP becomes very low picture depending on size

finished


[EXC-8484] Add new cookie "exchange.loginCSRFToken" in "Cookies used by Exchange"

Link to JIRA ticket
Summary
To prevent CSRF attacks in the Exchange login flow, a new cookie exchange.loginCSRFToken has been added.
Related Pull Requests

https://github.com/mulesoft/exchange-ui/pull/4088

Thank you @guillerecalde ! @mcampo would you please check and approve this 1-line change before I merge?


rake assets:precompile fails on production

We've been struggling with this one for almost a day.
We're using a theme from wrap bootstrap, which uses SCSS. It compiles absolutely fine on development, but on production it just doesn't play ball.
I'm really lost on where to get help on this one from, any assistance would be greatly appreciated.
Here's some of my code: https://gist.github.com/charleskirk/5555615

let me know if anything else is needed.

Here's the error and stack trace:
localhost:assets charleskirk$ /Users/charleskirk/.rvm/rubies/ruby-1.9.3-p286/bin/ruby /Users/charleskirk/.rvm/gems/ruby-1.9.3-p286/bin/rake assets:precompile:all RAILS_ENV=production RAILS_GROUPS=assets
(in /Users/charleskirk/jvo-portal)
rake aborted!
Invalid CSS after "&.": expected class name, was "#000"
  (in /Users/charleskirk/jvo-portal/app/assets/stylesheets/action_nav.css.scss)
/Users/charleskirk/jvo-portal/app/assets/stylesheets/action_nav.css.scss:104
/Users/charleskirk/.rvm/gems/ruby-1.9.3-p286/gems/sass-3.2.8/lib/sass/scss/parser.rb:1148:in `expected'
/Users/charleskirk/.rvm/gems/ruby-1.9.3-p286/gems/sass-3.2.8/lib/sass/scss/parser.rb:1084:in `expected'
/Users/charleskirk/.rvm/gems/ruby-1.9.3-p286/gems/sass-3.2.8/lib/sass/scss/parser.rb:1066:in `expr!'

I deployed rails project on Amazon EC2.
but I found below issue
Precompiling asset files        rake aborted!        Sass::SyntaxError: Undefined variable: "$body-background".
finally,
/home/ec2-user/.rvm/gems/ruby-2.4.1/bin/bundle:22:in load'
/home/ec2-user/.rvm/gems/ruby-2.4.1/bin/bundle:22:in <main>'        /home/ec2-user/.rvm/gems/ruby-2.4.1/bin/ruby_executable_hooks:15:in eval'
/home/ec2-user/.rvm/gems/ruby-2.4.1/bin/ruby_executable_hooks:15:in `'
Tasks: TOP => assets:precompile
(See full trace by running task with --trace)
!     ERROR: Deploy failed.
-----> Cleaning up build
Unlinking current
OK
Connection to (server_ip_address) closed.
!     Run Error`
help me!
I can't deploy my new version of this project on production because this issue.


Bug - export data

When reference field is same , but this value is diffrent , this export data's fields will wrong .
example :
nga.field('broker', 'reference').targetEntity(adminUser).targetField(nga.field('name'))
nga.field('broker', 'reference').targetEntity(adminUser).targetField(nga.field('age'))
that export file will own two age field.

ng-admin doesn't support that. Use an interceptor to duplicate the broker field, as explained in https://ng-admin-book.marmelab.com/doc/API-mapping.html


testadd p0 filter case

Proposed changes
add p0 filter case of mtmv
Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


fix Fix thread leaks when doing checkpoint

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall


What to do when a server goes down

The failover documentation explains what RethinkDB will do to try to maintain availability when a server goes down.
It might be useful to mention that although the tables will become available again after automatic failover kicks in, some issues might pop up, such as "Table test.foo is available for all operations, but some replicas are not ready." Users may need to connect the server back to the cluster to regain full health, or adjust the replication settings.

When you say "connect the server back to the cluster to regain full health," what does this mean in terms of user action? Restarting the server with the --join option? And which server -- the one(s) with the replicas that aren't ready? What replication settings would need to be adjusted, the list of replicas in the shard key for a given table in table_config?
Sorry to ask questions about virtually every noun. :)

Pinging @danielmewes for any comments on this when you get a chance. It's worth circling back around to.

When you say "connect the server back to the cluster to regain full health," what does this mean in terms of user action? Restarting the server with the --join option? And which server -- the one(s) with the replicas that aren't ready?

Yes. In the common case, some of the servers will have failed because of maintenance, or a hardware, power or network failure. So the solution is to start the server back up (in the way you mention), or resolve the issue by replacing hardware, reconnecting the network etc.
Once the server comes back and joins the cluster again, the issue will go away.
Note that this assumes that the server's data remains intact.
A server which hard drive broke for example cannot be brought back up in this sense. If that's the case, users should proceed to the second way of resolving the issue:

What replication settings would need to be adjusted, the list of replicas in the shard key for a given table in table_config?

The missing servers that are not expected to come back must be removed from the replicas list in table_config. If the user wants to maintain the same replication factor that they had before the failure, they might want to replace the replica in the configuration by a different server instead of just removing it.
The easiest way to perform these steps is to run reconfigure, or to use the web UI to reconfigure the number of replicas. reconfigure (as well as the web UI which is built on top of it) will only pick servers our of the set of currently connected servers for the new replica list. So the missing servers will automatically be removed from the configuration in the process.
In case a majority of replicas no longer exists, it might be necessary to perform a two-step recovery:

Running reconfigure({emergency_repair="unsafe_rollback"}) on the table to make it configurable by normal means again
Reconfiguring the table through
a) a second call to reconfigure (without emergency_repair)
b) by using the web UI
c) or by removing the missing servers manually from the replicas and nonvoting_replicas list in table_config.


Fixed reference to close_encrypted_device

@buzztroll method is local to this module.

SonarQube analysis reported 1 issue:

 1 minor

Watch the comments in this conversation to review them.
Note: the following issues could not be reported as comments because they are located on lines that are not displayed in this pull request:

 Rename method "tearDown" to match the regular expression ^[a-z_][a-z0-9_]{2,30}$.

:+1:

SonarQube analysis reported 1 issue:

 1 minor

Watch the comments in this conversation to review them.
Note: the following issues could not be reported as comments because they are located on lines that are not displayed in this pull request:

 Rename method "tearDown" to match the regular expression ^[a-z_][a-z0-9_]{2,30}$.


Changed the log file names so that our fluentd picks them.

Also let php-fpm catches the stdout and stderr and stop picking up
php-fpm's access log.
Background: The current fluentd configuration only picks up certain file name patterns. We need to adapt our log file names to their pattern.

LGTM

I'll merge this after the test passes.

The new image will be uploaded shortly :)


:seedling: convert CII Best Practices check to probes

What kind of change does this PR introduce?
feature

[x] PR title follows the guidelines defined in our pull request documentation

What is the new behavior (if this is a feature change)?**
This PR converts the CII Best Practices to 6 different probes.

[x] Tests for the changes have been added (for bug fixes/features)

Which issue(s) this PR fixes
NONE
Does this PR introduce a user-facing change?
No.

@laurentsimon @spencerschrock PTAL again.

I don't think https://github.com/ossf/scorecard/pull/3520#discussion_r1338969512 was addressed. That paragraph is still misleading and "that" hasn't been removed.

The OpenSSF Best Practices badge indicates whether or not that the project uses a set of security-focused best development practices for open source software.

I don't think #3520 (comment) was addressed. That paragraph is still misleading and "that" hasn't been removed.

The OpenSSF Best Practices badge indicates whether or not that the project uses a set of security-focused best development practices for open source software.

Missed that the other probes had the same issue. Fixed now.

To be honest I'm much more interested in getting the wording right :-) It's just that I don't think that badges should be conflated with the best practices. All this check shows is that projects have badges and it doesn't necessarily mean that they actually follow those practices. Projects without badges can follow the practices too.

To be honest I'm much more interested in getting the wording right :-) It's just that I don't think that badges should be conflated with the best practices. All this check shows is that projects have badges and it doesn't necessarily mean that they actually follow those practices. Projects without badges can follow the practices too.

I think it's fine to reword. Any suggestion? I'm not sure which best practices are automatically checked, and which are not (in which case the best practice badge is essentially about education; and that's something we could highlight, eg maintainers have followed an introductory course on ...)
@david-a-wheeler please feel free to suggest

Any suggestion?

To judge from https://github.com/coreinfrastructure/best-practices-badge/blob/main/docs/vetting.md "The best practices badge approach is based on self-certification" with some basic checks so I'd say it's supposed to encourage good practices and let projects self-assess themselves. Personally I think it's too vague to be useful in automated tools like scorecard. I'm not sure why projects with no badges are downgrade either.

that's something we could highlight, eg maintainers have followed an introductory course on

I think it's being discussed in https://github.com/ossf/scorecard/issues/3534

Personally I think it's too vague to be useful in automated tools like scorecard.

They aren't vague at all. They are carefully worded requirements. Most of them can't be directly measured by Scorecard, so the best practices badge provides a way to measure many criteria that otherwise Scorecard cannot measure.

I'm not sure why projects with no badges are downgrade either.

Because a project not pursuing a badge is unlikely to meet its requirements. Once it starts pursuing a badge it's more likely to reach them.

All this check shows is that projects have badges and it doesn't necessarily mean that they actually follow those practices

It does in general mean they follow the practices. If it's automated then it's checked. If it's a human making a claim, then it requires a listed human who is asserting that it's met, typically with a justification. The claim is also posted for all to see. We don't get many lies; if someone does lie, the badge is pulled.

It does in general mean they follow the practices

It does not. I often look at projects with badges and they are usually out-of-date because they were received once when companies felt the need to show their commitment to security by participating in that program and have never been updated since then. Basically it's necessary to manually verify all the claims but if I have the resources and the expertise to verity all those claim I'm not exactly sure how the badge is supposed to help me to evaluate anything.

It does not. I often look at projects with badges and they are usually out-of-date

If you have specific cases, file an issue & let's fix them. We're planning to improve the automation & rechecking of results, that should also help in some cases.
But ignoring the best practices badge information isn't an improvement.

I'm not saying that badges should be ignored. I'm saying that the scorecard documentation shouldn't claim that badges mean more than encouraging good practices and some sort of self-assessment performed at some point.

It's more than a self-assessment, but sure, searching for clearer wording can be helpful.

I looked this file's text over. @evverx - I think you're conflating the decription with the motivation. Here's what this file says:

short: This check determines whether the project has an OpenSSF (formerly CII) Best Practices Badge at the Gold level.
motivation: >
The OpenSSF Best Practices badge indicates whether or not the project uses a set of security-focused best development practices for open source software.

The short description is absolutely correct - it's measuring the presence/absence of a best practices badge level. The "motivation" text is a short explanation of why that measure has value. I think you could add another sentence to the "motivation" section to clarify things further the way you want. Perhaps appending this text in each "motivation" section will help:

This is primarily through a process of self-certification of many specific criteria, along with some automated checks and public display of the claims.

I think as long as self-certification is mentioned somewhere it should be fine. It should help people who actually audit projects to infer the value of that badge.

public display of the claims

On a somewhat related note as far as I can see in its current form scorecard doesn't show links to badges
    {
      "name": "CII-Best-Practices",
      "score": 5,
      "reason": "badge detected: passing",
      "details": null,
      "documentation": {
        "short": "Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.",
        "url": "https://github.com/ossf/scorecard/blob/4edb07802fdad892fa8d10f8fd47666b6ccc27c9/docs/checks.md#cii-best-practices"
      }

and it makes it unnecessarily hard to view those claims.
Also according to https://github.com/coreinfrastructure/best-practices-badge/blob/main/docs/vetting.md

The problems of self-certification are well-known, so we have various mechanisms to mitigate its problems:
...
Reduced incentives. We intentionally reduce the incentives for people to create false information.

It doesn't seem to be true anymore given that scorecard bumps projects with badges and they are included in the SOS reward.
It has nothing to do with this PR probably.

/scdiff generate CII-Best-Practices

/scdiff generate CII-Best-Practices


[READY] - [Infra-Issue511] - add s3 support to term-apply

Adds s3 support to term-apply. It's very raw and will need a 2nd pass, but would like to get a version out that doesn't require manual syncing.
Previous:

applicants.csv and resume pdfs only written locally to pod and needed to be sync'd off

New:

applicants.csv is pulled from s3 on startup and written to s3 every time it's updated locally
resume pdfs are still written locally, but are then copied to s3. the copy fails if either step fails
a new environment variable called TA_BUCKET is now required with the s3 bucket name

Tests:

[x] locally

@sarcasticadmin this is good to go in dev. Once merged I'll re-target the nix-garage build to main and proceed to re-validate dev then start pushing to higher environments.