FineTune Wav2Vec2.0 , CUDA OOM

What is i question?
I am finetune a Wav2Vec 2.0 Large (LV-60) model using my dataset (35 hours) for ASR Task.
Training takes 15-20 epochs, after which the error CUDA OUT OF MEMORY appears.
What have i tried?
I tried changing the max-tokens parameter in the GPU
What's environment?
I use docker container NVidia nemo:0.10

fairseq Version - master

PyTorch Version - 1.4.0

OS - Ubuntu 18.04.4 LTS :

How you installed fairseq -
git clone https://github.com/pytorch/fairseq
cd fairseq
pip install --editable ./

Python version - 3.6.9

CUDA Version: 10.2

GPU models and configuration - I use 4 Titan V 12GB

Run command
python train.py --distributed-world-size 4
ru_manifest/
--save-dir checkpoint_wer/
--fp16
--post-process letter
--valid-subset valid
--no-epoch-checkpoints
--best-checkpoint-metric wer
--num-workers 32
--max-update 400000
--sentence-avg
--task audio_pretraining
--arch wav2vec_ctc
--w2v-path checkpoints_wav2vec/wav2vec_vox.pt
--labels ltr
--apply-mask
--mask-selection static
--mask-other 0
--mask-length 10
--mask-prob 0.65
--layerdrop 0.1
--mask-channel-selection static
--mask-channel-other 0
--mask-channel-length 64
--mask-channel-prob 0.256
--zero-infinity
--feature-grad-mult 0.0
--freeze-finetune-updates 10000
--validate-after-updates 10000
--optimizer adam
--adam-betas '(0.9, 0.98)'
--adam-eps 1e-08
--lr 2e-05
--lr-scheduler tri_stage
--warmup-steps 4000
--hold-steps 8000
--decay-steps 10000
--final-lr-scale 0.05
--final-dropout 0.0
--dropout 0.0
--activation-dropout 0.1
--criterion ctc
--attention-dropout 0.0
--max-tokens 900000
--seed 2337
--log-format json
--log-interval 500
--ddp-backend no_c10d
--tensorboard-logdir tensorboard-wav2vec/

Hey @byebye1  can you guide what are the required files to start fintuning on our own data?
What I understood now is that it requires one dict file(A-Z) which given already in reame only and one manifest file which will have paths for all audio and a labels file which will have transcript of all audios. All for 16khz audio. Although I have 8khz data😑
Am I missing anything?

@byebye1 have found any solution of OOM issue getting similar issue. After 8 epoch its showing OOM

Try setting the --empty-cache-freq option. Alternatively reduce --max-tokens.  Seems to be a GPU RAM fragmentation issue.

I met this issue after finetune 3 epoch for train-clean-100 using one 2080ti. And it's strange that in previous 3 epoch everything is ok, but if I load the ckpt saved and resume from 3 epoch, I get the oom error immediately. @medabalimi and I have reduced the max tokens but seems not work.

@ArtemisZGL did you set the --empty-cache-freq option.

@medabalimi yes I just set it to 100, but I'm confused that I can't find it in any code by searching. And it did't solve the problem yet.

I suggested using the --empty-cache-freq option because that helped me with OOM issues. This helps clear the pytorch cache at specified intervals at the cost of speed. I'm assuming that you're installed Nvidia's Apex as well.
What is the checkpoint size?

@medabalimi Thanks for your reply. I think I did not install apex, will it have some influence? And I can not search the flag " --empty-cache-freq" in any parser option in the code. I even have tried to use 2 gpus to train, but still oom in epoch 3. The ckpt size is 3.6G, is it normal? thanks.

Any resolution for this? I'm facing the same issue after 4 epochs. I even increased the number of GPUs I was training on, but even on resuming it fails. If changing the seed helps for a few epochs, there's some seed for which the batch is too big for any GPU to handle. Even when resumed, the seed is set (differently, but deterministically) according to epoch number  Am I interpreting this right?

@yash-s20 my solution is to reduce the max-tokens and set the update-frequent bigger to keep the batch size. But don’t know the real reason caused this problem.

I'm also facing OOM errors when Fine-Tunning the model.
The server has 10 GPUs but nvidia-smi doesn't seem to be saying there is no memory left. It reports that just between 10% and 15% of it's in use.
I'm using:

batch-size = 10
fp16
The model has 315471520 parameters ===>  315471520*2/2^20 = 602MB so the model fits in the GPU

I don't understand what max-tokens per GPU does, but I don't
2020-11-09 19:24:02 | INFO | fairseq_cli.train | model: Wav2VecCtc
2020-11-09 19:24:02 | INFO | fairseq_cli.train | criterion: CtcCriterion)
2020-11-09 19:24:02 | INFO | fairseq_cli.train | num. model params: 315471520 (num. trained: 315471520)
2020-11-09 19:24:09 | INFO | fairseq.utils | ***********************CUDA enviroments for all 10 workers***********************
2020-11-09 19:24:09 | INFO | fairseq.utils | rank   0: capabilities =  7.5  ; total memory = 10.761 GB ; name = GeForce RTX 2080 Ti
2020-11-09 19:24:09 | INFO | fairseq.utils | rank   1: capabilities =  7.5  ; total memory = 10.761 GB ; name = GeForce RTX 2080 Ti
2020-11-09 19:24:09 | INFO | fairseq.utils | rank   2: capabilities =  7.5  ; total memory = 10.761 GB ; name = GeForce RTX 2080 Ti
2020-11-09 19:24:09 | INFO | fairseq.utils | rank   3: capabilities =  7.5  ; total memory = 10.761 GB ; name = GeForce RTX 2080 Ti
2020-11-09 19:24:09 | INFO | fairseq.utils | rank   4: capabilities =  7.5  ; total memory = 10.761 GB ; name = GeForce RTX 2080 Ti
2020-11-09 19:24:09 | INFO | fairseq.utils | rank   5: capabilities =  7.5  ; total memory = 10.761 GB ; name = GeForce RTX 2080 Ti
2020-11-09 19:24:09 | INFO | fairseq.utils | rank   6: capabilities =  7.5  ; total memory = 10.761 GB ; name = GeForce RTX 2080 Ti
2020-11-09 19:24:09 | INFO | fairseq.utils | rank   7: capabilities =  7.5  ; total memory = 10.761 GB ; name = GeForce RTX 2080 Ti
2020-11-09 19:24:09 | INFO | fairseq.utils | rank   8: capabilities =  7.5  ; total memory = 10.761 GB ; name = GeForce RTX 2080 Ti
2020-11-09 19:24:09 | INFO | fairseq.utils | rank   9: capabilities =  7.5  ; total memory = 10.761 GB ; name = GeForce RTX 2080 Ti
2020-11-09 19:24:09 | INFO | fairseq.utils | ***********************CUDA enviroments for all 10 workers***********************
2020-11-09 19:24:09 | INFO | fairseq_cli.train | training on 10 devices (GPUs/TPUs)
2020-11-09 19:24:09 | INFO | fairseq_cli.train | max tokens per GPU = 1280000 and batch size per GPU = 10
2020-11-09 19:24:09 | INFO | fairseq.trainer | no existing checkpoint found /home/aalvarez/trainings/checkpoint_last.pt
2020-11-09 19:24:09 | INFO | fairseq.trainer | loading train data for epoch 1-11-09 19:24:10 | INFO | fairseq.data.audio.raw_audio_dataset | loaded 1467296, skipped 0 samples
2020-11-09 19:24:14 | INFO | fairseq.optim.adam | using FusedAdam
2020-11-09 19:24:14 | INFO | fairseq.trainer | begin training epoch 1
2020-11-09 19:25:02 | INFO | fairseq.trainer | NOTE: overflow detected, setting loss scale to: 64.0
2020-11-09 19:25:03 | INFO | fairseq.trainer | NOTE: overflow detected, setting loss scale to: 32.0
2020-11-09 19:25:05 | INFO | fairseq.trainer | NOTE: overflow detected, setting loss scale to: 16.0
2020-11-09 19:25:07 | INFO | fairseq.trainer | NOTE: overflow detected, setting loss scale to: 8.0
2020-11-09 19:25:12 | WARNING | fairseq.trainer | OOM: Ran out of memory with exception: CUDA out of memory. Tried to allocate 644.00 MiB (GPU 8; 10.76 GiB total capacity; 7.38 GiB already allocated; 467.56 MiB fr
ee; 7.40 GiB reserved in total by PyTorch)
2020-11-09 19:25:12 | WARNING | fairseq.trainer | |===========================================================================|
|                  PyTorch CUDA memory summary, device ID 8                 |
|---------------------------------------------------------------------------|
|            CUDA OOMs: 1            |        cudaMalloc retries: 3         |
|===========================================================================|
|        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
|---------------------------------------------------------------------------|
| Allocated memory      |    6917 MB |    7561 MB |  101651 MB |   94733 MB |
|       from large pool |    6913 MB |    7557 MB |  101539 MB |   94625 MB |
|       from small pool |       3 MB |       9 MB |     112 MB |     108 MB |
|---------------------------------------------------------------------------|
| Active memory         |    6917 MB |    7561 MB |  101651 MB |   94733 MB |
|       from large pool |    6913 MB |    7557 MB |  101539 MB |   94625 MB |
|       from small pool |       3 MB |       9 MB |     112 MB |     108 MB |
|---------------------------------------------------------------------------|
| GPU reserved memory   |    7578 MB |    7942 MB |   18258 MB |   10680 MB |
|       from large pool |    7570 MB |    7932 MB |   18228 MB |   10658 MB |
|       from small pool |       8 MB |      10 MB |      30 MB |      22 MB |
|---------------------------------------------------------------------------|
| Non-releasable memory |   16790 KB |    1242 MB |  113718 MB |  113701 MB |
|       from large pool |   12584 KB |    1242 MB |  113566 MB |  113554 MB |
|       from small pool |    4206 KB |       4 MB |     151 MB |     147 MB |
|---------------------------------------------------------------------------|
| Allocations           |     458    |     886    |   11608    |   11150    |
|       from large pool |     159    |     306    |    6479    |    6320    |
|       from small pool |     299    |     582    |    5129    |    4830    |
|---------------------------------------------------------------------------|
| Active allocs         |     458    |     886    |   11608    |   11150    |
|       from large pool |     159    |     306    |    6479    |    6320    |
|       from small pool |     299    |     582    |    5129    |    4830    |
|---------------------------------------------------------------------------|
| GPU reserved segments |      44    |      79    |     196    |     152    |
|       from large pool |      40    |      74    |     181    |     141    |
|       from small pool |       4    |       5    |      15    |      11    |
|---------------------------------------------------------------------------|
| Non-releasable allocs |      10    |      32    |    4114    |    4104    |
|       from large pool |       2    |      30    |    3470    |    3468    |
|       from small pool |       8    |      12    |     644    |     636    |
|===========================================================================|

This is how I'm launching the training
python -W ignore train.py /mnt/data/ale/manifest_ml2 \
--save-dir ~/trainings \
--wer-args '("/mnt/data/ale/kenlm-models/openwebtext/v5/4-gram-265M-25-10-2020-pruned-300K-1.bin","/mnt/data/ale/manifest_ml2/dev.lex",2,-1)' \
--post-process letter  \
--valid-subset dev \
--best-checkpoint-metric wer  \
--num-workers 20 \
--max-update 80000  \
--sentence-avg  \
--task audio_pretraining  \
--arch wav2vec_ctc  \
--w2v-path /mnt/data/ale/models/wav2vec_vox_new.pt \
--labels ltr  \
--apply-mask  \
--mask-selection static  \
--mask-other 0  \
--mask-length 10  \
--mask-prob 0.5  \
--layerdrop 0.1 \
--mask-channel-selection static  \
--mask-channel-other 0  \
--mask-channel-length 64  \
--mask-channel-prob 0.5  \
--zero-infinity \
--feature-grad-mult 0.0  \
--freeze-finetune-updates 10000  \
--validate-after-updates 10000  \
--optimizer adam \
--adam-betas '(0.9, 0.98)'  \
--adam-eps 1e-08  \
--lr 2e-05  \
--lr-scheduler tri_stage  \
--warmup-steps 8000  \
--hold-steps 32000 \
--decay-steps 40000  \
--final-lr-scale 0.05  \
--final-dropout 0.0  \
--dropout 0.0  \
--activation-dropout 0.1  \
--criterion ctc \
--attention-dropout 0.0  \
--max-tokens 1280000  \
--seed 2337  \
--log-format json  \
--log-interval 500  \
--ddp-backend no_c10d \
--normalize \
--batch-size 10 \
--fp16 \
--distributed-world-size 10 \
--empty-cache-freq 100

I noticed that long audio file might HAVE CHANGE to cause OOM but not every epoch.
try --max-sample-size 160000 or even smaller to make sure the OOM issue is not due to long audio sample, if OOM doesn't occur again, then you should skip those long audio file according to your devices capability.
To train wav2vec_vox_new.pt such large model, my hardware (GTX 1080 8GB x 2) can only afford --max-sample-size 208000, which is 13 seconds audio and batch-size=1  without --fp16, and I notice that the --batch-size-valid can still be 8, seems training updates step cost much memory than valid.

@mychiux413 is your training going fine on 2*1080? I also have a similar system.

@008karan I am still tracking it, and if it crashes again, I will report it.

Thank you very much for your comment @mychiux413!! It seems that now it's working.
Although, I'm still confused:

In the dataset loading documentation they mention --max-tokens. Is this valid or not? I'm thinking now that this was written for another datatype loading class.
What happens with audio files that are larger than max-sample-size?
How can I estimate how much memory it will occupy? or is it only a matter of trial and error?

Tracing back [udio_pretraining.py](https://github.com/pytorch/fairseq/blob/master/fairseq/tasks/audio_pretraining.py#L94) it uses [FileAudioDataset](https://github.com/pytorch/fairseq/blob/master/fairseq/data/audio/raw_audio_dataset.py#L133) which inherits [RawAudioDataset](https://github.com/pytorch/fairseq/blob/master/fairseq/data/audio/raw_audio_dataset.py#L21) which has a function called [num_tokens](https://github.com/pytorch/fairseq/blob/master/fairseq/data/audio/raw_audio_dataset.py#L110) which returns the min(size, max_sample_size)`, but I don't know where this is used and how.

@alealv

It should be valid, because to train wav2vec_vox_new.pt, I have to set --max-tokens=500000, or it will still OOM.
Those files will be cropped, see self.crop_to_max_size, every larger file size will be --max-sample-size, so the best way is never put large audio in manifest file at the first.
No idea, I haven't studied that how memory grow with multihead-self-attention to difference sequence size
It frustrated me too.

Update:
--batch-size-valid=8, --max-sample-size=208000 can still OOM at validate phase, so I set --batch-size-valid=1 for safety now.

Update:
The memory will increase after the number of updates > --freeze-finetune-updates, so we should --freeze-finetune-updates=0 at fist try to make sure it won't OOM after unfreezing, and maybe that's why the OOM occurred after few epochs, which depends on your data size.
And the other information is, if we enable --update_freq > 1, the distributed process also increase the GPU memory usage.
With GTX1080(8GB) * 2, I can only filter audio duration less than 10 secs to prevent OOM, which drops too many dataset.
Does any one can share the affordable audio duration with GPU > 8GB?

Thanks for the update @mychiux413
I've been experimenting as well with a small training size and I realized something really weird.
Configuration

training size = 8
dev size = 6
batch size = 1
log-interval = 1

Expected output
I believe that with this configuration, I should get 8 log lines for each sample in the training set. Then a validation step which should output 6 logs lines. And finally, the updated parameters followed by a checkpoint.
Ouput
Instead, I get just 1 log line for the training step and the validation step, but on that line the parameter nstences is equal to the training_size and the dev_size.
2020-11-20 18:37:12 | INFO | fairseq_cli.train | max tokens per GPU = None and batch size per GPU = 1
2020-11-20 18:37:12 | INFO | fairseq.trainer | no existing checkpoint found /home/aalvarez/trainings/checkpoint_last.pt
2020-11-20 18:37:12 | INFO | fairseq.trainer | loading train data for epoch 1
2020-11-20 18:37:12 | INFO | fairseq.data.audio.raw_audio_dataset | loaded 8, skipped 0 samples
2020-11-20 18:37:12 | INFO | fairseq.optim.adam | using FusedAdam
2020-11-20 18:37:12 | INFO | fairseq.trainer | begin training epoch 1
2020-11-20 18:37:22 | INFO | train_inner | {"epoch": 1, "update": 1.0, "loss": "255.939", "ntokens": "40", "nsentences": "8", "nll_loss": "51.188", "wps": "0", "ups": "0", "wpb": "40", "bsz": "8", "num_updates": "
1", "lr": "2.02475e-07", "gnorm": "171.406", "loss_scale": "128", "train_wall": "2", "wall": "11"}
2020-11-20 18:37:22 | INFO | fairseq_cli.train | begin validation on "dev" subset
2020-11-20 18:37:48 | INFO | dev | {"epoch": 1, "dev_loss": "188.179", "dev_ntokens": "31", "dev_nsentences": "6", "dev_nll_loss": "36.422", "dev_uer": "100", "dev_wer": "100", "dev_raw_wer": "100", "dev_wps": "0"
, "dev_wpb": "31", "dev_bsz": "6", "dev_num_updates": "1"}
2020-11-20 18:37:48 | INFO | fairseq_cli.train | begin save checkpoint
2020-11-20 18:38:05 | INFO | fairseq.checkpoint_utils | saved checkpoint /home/aalvarez/trainings/checkpoint1.pt (epoch 1 @ 1 updates, score 100.0) (writing took 16.460464287083596 seconds)
2020-11-20 18:38:05 | INFO | fairseq_cli.train | end of epoch 1 (average epoch stats below)
2020-11-20 18:38:05 | INFO | train | {"epoch": 1, "train_loss": "255.939", "train_ntokens": "40", "train_nsentences": "8", "train_nll_loss": "51.188", "train_wps": "0", "train_ups": "0", "train_wpb": "40", "train_
bsz": "8", "train_num_updates": "1", "train_lr": "2.02475e-07", "train_gnorm": "171.406", "train_loss_scale": "128", "train_train_wall": "2", "train_wall": "53"}
2020-11-20 18:38:05 | INFO | fairseq.trainer | begin training epoch 2

So I believe that fairseq is actually loading more samples than the indicated in batch-size and that combine with large audios could make everything explode.
I haven't dig yet into the code to understand what's going on.

Hi folks,
I encountered the same issue, and I just solved it.
In my case, the reason why OOM when reloading is, the checkpoint is reloaded to the CPU first, then broadcasted to other processes by Inter-process communication:
the reloading code is at * fairseq/trainer.py, line 323*:
        if bexists:
            if self.data_parallel_rank == 0:
                state = checkpoint_utils.load_checkpoint_to_cpu(filename)
                last_optim_state = state.get("last_optimizer_state", None)

                # If doing zero_sharding, do not broadcast global optimizer
                # state. Later we will broadcast sharded states to each rank
                # to avoid memory from exploding.
                if (
                        self.cfg.distributed_training.zero_sharding == "os"
                        and "last_optimizer_state" in state
                        and self.data_parallel_world_size > 1
                ):
                    state["last_optimizer_state"] = "SHARDED"
            else:
                last_optim_state = None
                state = None

            if self.data_parallel_world_size > 1:
                group = (
                    self.data_parallel_process_group
                    if self.data_parallel_process_group is not None
                    else torch.distributed.group.WORLD
                )
                state = distributed_utils.broadcast_object(
                    state,
                    src_rank=0,
                    group=group,
                )
                if self.data_parallel_rank > 0:
                    last_optim_state = state.get("last_optimizer_state", None)

2020-11-19T06:55:32.107Z: [1,0]:Traceback (most recent call last):
2020-11-19T06:55:32.107Z: [1,0]:  File "/data/e2easr2/v-naili/proj/fairseq/examples/wav2vec/../../train.py", line 14, in
2020-11-19T06:55:32.108Z: [1,0]:    cli_main()
2020-11-19T06:55:32.108Z: [1,0]:  File "/data/e2easr2/v-naili/proj/fairseq/fairseq_cli/train.py", line 391, in cli_main
2020-11-19T06:55:32.108Z: [1,0]:    distributed_utils.call_main(cfg, main)
2020-11-19T06:55:32.108Z: [1,0]:  File "/data/e2easr2/v-naili/proj/fairseq/fairseq/distributed_utils.py", line 311, in call_main
2020-11-19T06:55:32.109Z: [1,0]:    distributed_main(cfg.distributed_training.device_id, main, cfg, kwargs)
2020-11-19T06:55:32.109Z: [1,0]:  File "/data/e2easr2/v-naili/proj/fairseq/fairseq/distributed_utils.py", line 289, in distributed_main
2020-11-19T06:55:32.109Z: [1,0]:    main(cfg, **kwargs)
2020-11-19T06:55:32.109Z: [1,0]:  File "/data/e2easr2/v-naili/proj/fairseq/fairseq_cli/train.py", line 120, in main
2020-11-19T06:55:32.109Z: [1,0]:    disable_iterator_cache=task.has_sharded_data("train"),
2020-11-19T06:55:32.109Z: [1,0]:  File "/data/e2easr2/v-naili/proj/fairseq/fairseq/checkpoint_utils.py", line 196, in load_checkpoint
2020-11-19T06:55:32.110Z: [1,0]:    reset_meters=reset_meters,
2020-11-19T06:55:32.110Z: [1,0]:  File "/data/e2easr2/v-naili/proj/fairseq/fairseq/trainer.py", line 368, in load_checkpoint
2020-11-19T06:55:32.120Z: [1,0]:    self.optimizer.load_state_dict(last_optim_state, optimizer_overrides)
2020-11-19T06:55:32.121Z: [1,0]:  File "/data/e2easr2/v-naili/proj/fairseq/fairseq/optim/fp16_optimizer.py", line 88, in load_state_dict
2020-11-19T06:55:32.125Z: [1,0]:    self.fp32_optimizer.load_state_dict(state_dict, optimizer_overrides)
2020-11-19T06:55:32.125Z: [1,0]:  File "/data/e2easr2/v-naili/proj/fairseq/fairseq/optim/fairseq_optimizer.py", line 86, in load_state_dict
2020-11-19T06:55:32.145Z: [1,0]:    self.optimizer.load_state_dict(state_dict)
2020-11-19T06:55:32.145Z: [1,0]:  File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 105, in load_state_dict
2020-11-19T06:55:32.145Z: [1,0]:    state_dict = deepcopy(state_dict)
2020-11-19T06:55:32.145Z: [1,0]:  File "/opt/conda/lib/python3.7/copy.py", line 150, in deepcopy
2020-11-19T06:55:32.145Z: [1,0]:    y = copier(x, memo)
2020-11-19T06:55:32.145Z: [1,0]:  File "/opt/conda/lib/python3.7/copy.py", line 241, in _deepcopy_dict
2020-11-19T06:55:32.145Z: [1,0]:    y[deepcopy(key, memo)] = deepcopy(value, memo)
2020-11-19T06:55:32.145Z: [1,0]:  File "/opt/conda/lib/python3.7/copy.py", line 150, in deepcopy
2020-11-19T06:55:32.145Z: [1,0]:    y = copier(x, memo)
2020-11-19T06:55:32.145Z: [1,0]:  File "/opt/conda/lib/python3.7/copy.py", line 241, in _deepcopy_dict
2020-11-19T06:55:32.145Z: [1,0]:    y[deepcopy(key, memo)] = deepcopy(value, memo)
2020-11-19T06:55:32.146Z: [1,0]:  File "/opt/conda/lib/python3.7/copy.py", line 150, in deepcopy
2020-11-19T06:55:32.146Z: [1,0]:    y = copier(x, memo)
2020-11-19T06:55:32.146Z: [1,0]:  File "/opt/conda/lib/python3.7/copy.py", line 241, in deepcopy_dict
2020-11-19T06:55:32.146Z: [1,0]:    y[deepcopy(key, memo)] = deepcopy(value, memo)
2020-11-19T06:55:32.146Z: [1,0]:  File "/opt/conda/lib/python3.7/copy.py", line 161, in deepcopy
2020-11-19T06:55:32.146Z: [1,0]:    y = copier(memo)
2020-11-19T06:55:32.146Z: [1,0]:  File "/opt/conda/lib/python3.7/site-packages/torch/tensor.py", line 52, in deepcopy
2020-11-19T06:55:32.146Z: [1,0]:    new_storage = self.storage().deepcopy(memo)
2020-11-19T06:55:32.146Z: [1,0]:  File "/opt/conda/lib/python3.7/site-packages/torch/storage.py", line 28, in deepcopy
2020-11-19T06:55:32.146Z: [1,0]:    new_storage = self.clone()
2020-11-19T06:55:32.146Z: [1,0]:  File "/opt/conda/lib/python3.7/site-packages/torch/storage.py", line 44, in clone
2020-11-19T06:55:32.146Z: [1,0]:    return type(self)(self.size()).copy(self)
2020-11-19T06:55:32.146Z: [1,0]:  File "/opt/conda/lib/python3.7/site-packages/torch/cuda/init.py", line 480, in _lazy_new
2020-11-19T06:55:32.147Z: [1,0]:    return super(_CudaBase, cls).new(cls, *args, **kwargs)
2020-11-19T06:55:32.147Z: [1,0]:RuntimeError: CUDA out of memory. Tried to allocate 1.18 GiB (GPU 0; 22.38 GiB total capacity; 2.94 GiB already allocated; 1.01 GiB free; 2.96 GiB reserved in total by PyTorch)

state = distributed_utils.broadcast_object should: 1. if current proc rank is 0, send the ckpt to a buffer; 2. if not, received the ckpt from the buffer.
However, if you check the ckpt loaded from the buffer, you find that the tensors in "last_optim_state " (last_optim_state = state.get("last_optimizer_state", None) ) are all on cuda:0! I don't know why this happens, but when reloading it to the processes, each of them will deepcopy it, as shown in the exception traceback, therefore OOM is encountered on cuda:0.
My solution is very simple, just avoid the inter-process communication, and load the ckpt from the disk in each process!
        if bexists:
            state = checkpoint_utils.load_checkpoint_to_cpu(filename)
            last_optim_state = state.get("last_optimizer_state", None)
            # if self.data_parallel_rank == 0:
            #     state = checkpoint_utils.load_checkpoint_to_cpu(filename)
            #     last_optim_state = state.get("last_optimizer_state", None)
            #
            #     # If doing zero_sharding, do not broadcast global optimizer
            #     # state. Later we will broadcast sharded states to each rank
            #     # to avoid memory from exploding.
            #     if (
            #             self.cfg.distributed_training.zero_sharding == "os"
            #             and "last_optimizer_state" in state
            #             and self.data_parallel_world_size > 1
            #     ):
            #         state["last_optimizer_state"] = "SHARDED"
            # else:
            #     last_optim_state = None
            #     state = None
            #
            # if self.data_parallel_world_size > 1:
            #     group = (
            #         self.data_parallel_process_group
            #         if self.data_parallel_process_group is not None
            #         else torch.distributed.group.WORLD
            #     )
            #     state = distributed_utils.broadcast_object(
            #         state,
            #         src_rank=0,
            #         group=group,
            #     )
            #     if self.data_parallel_rank > 0:
            #         last_optim_state = state.get("last_optimizer_state", None)
            #     # check_ckpt_device(self, last_optim_state)

the OOM bug is because max_tokens_valid isnt being correctly defaulted to max_tokens. i will submit a fix, meanwhile just provide --max-tokens-valid and set it to the same thing as --max-tokens

So I believe that fairseq is actually loading more samples than the indicated in batch-size and that combine with large audios could make everything explode.

Update
I discovered that batch-size is the number of samples loaded per GPU, which is really misleading. So care must be taken with this value because all the inputs are loaded first to one GPU and spread to the others as explained here

right, the same is true for max-tokens - these are all per gpu values and must be multiplied by the number of gpus being used  and also by update_freq to compute the effective batch size


improve x-axis breaks/labels

x-axis breaks and labels - can we improve?
burrlioz just has 1, 10, 100, 1000 etc.
"Divisions in x axis are too big.  A label of 3160 seems unusual
and difficult to interpolate by eye.  Why not 10, 100, 1000,
10,000 like Burrlioz?"

now full customization of x breaks


v0.1

Version v0.1

Basic functionalities: entity, attributes,
TCP, KCP, WEBSOCKET
Reload
ext: db/gwmongo, pubsub, msgbox
strip useless code

作者开发很积极啊.. 居然才0.1版本。 期待1.0

Do you plan to remove the use of python? You should be able to do everything around process creation in go instead.

good suggestion for removing python.

How close is v0.1 now?


kured - add psp configuration capabilities

This PR allow the configuration of pod security policy necessary to get kured running when the admission controller is enabled.
updated version of #11603
Checklist

[x] DCO signed
[x] Chart Version bumped
[x] Variables are documented in the README.md

/assign @lachie83

/assign
/ok-to-test
/lgtm
i bumped the minor version as this PR adds a new feature.
thanks for contributing!


Rename peel and unpeel as unpackVec and packVec

I find the names peel and unpeel confusing. Let's think of a better name e.g. unpackVec which takes a SignalType vector and returns a Vector.t and packVec which takes a Vector.t and returns a SignalType vector.
Another question: currently the unpacked type is Vector.t. Should it be list?

I think it may well be worth exploring unpacking Vector.t to list at some point -- but it would break a lot of current proofs, so perhaps best to try it after our phase 1 deadline.

I think it may well be worth exploring unpacking Vector.t to list at some point -- but it would break a lot of current proofs, so perhaps best to try it after our phase 1 deadline.

Some additional name ideas:

bundle / unbundle /  bundleVec / unbundleVec
distribute / collect

I'm happy with any, pack/unpack would get a vote from me just for simplicity.

Some additional name ideas:

bundle / unbundle /  bundleVec / unbundleVec
distribute / collect

I'm happy with any, pack/unpack would get a vote from me just for simplicity.

I like pack and unpack although I worry about name clashes because they are quite common identifier names. For phase 1 leaving them as peel and unpeel might be the best plan.

I like pack and unpack although I worry about name clashes because they are quite common identifier names. For phase 1 leaving them as peel and unpeel might be the best plan.

Let's close this for now and revisit in phase 2.

Let's close this for now and revisit in phase 2.


IE8 layout problem

Hi guys,
I'm developing a website under wordpress with the roots theme based on the foundation framework.
I never user IE and I just decided to have a look to the site with it... http://www.tkwk.be/client/acea/www/home/
The whole layout is upside down, and I can't figure out why.
The size of the class .row seems to be extra-large and it drives me crazy.
Any advice would be much appreciated.
Cheers,
J.

I was unable to reproduce this issue on our IE8 test machine - do you have screenshots or anything we can?


Benchmark tok/sec with other libs

Would be great to benchmark tokens/sec of OpenLM, comparing to other libraries like Mosaic, Metaseq, etc.

just a few datapoints from OpenLM, with default hparams: we get ~2.5K tokens/sec/GPU on 256 A100s for OpenLM-7B, and ~9.5K tokens/sec/GPU on 128 A100s for OpenLM-1B. ~11.5K tokens/sec/GPU on 32 A100s for 1B. 7B model gets ~2700 tokens/sec/GPU on one node

This could also be useful: https://github.com/mosaicml/llm-foundry/tree/main/scripts/train/benchmarking

@achalddave is making fantastic progress here. When we're done, let's add the benchmarking results to the repository so others can compare numbers and check whether they locally get the expected performance.

See #29 for improvements. Basically we are now matching the mosaicml numbers on 1 node (~4000 tok/s/gpu for a 7b model with batch size 16, seq length 2048, on 8 A100). Will close this once we test convergence on a large run.


NIFI-12490 MiNiFi docker graceful shutdown

Summary
NIFI-12490
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking

[x] Apache NiFi Jira issue created

Pull Request Tracking

[x] Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
[x] Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

[x] Pull Request based on current revision of the main branch
[x] Pull Request refers to a feature branch with one commit containing changes

Verification
Please indicate the verification steps performed prior to pull request creation.
Build

[x] Build completed using mvn clean install -P contrib-check

[x] JDK 21

Licensing

[ ] New dependencies are compatible with the Apache License 2.0 according to the License Policy
[ ] New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

[ ] Documentation formatting appears as expected in rendered files

Thank you for the change, double checked and now minifi is in sync with the nifi behavior. Also built a docker image and validated that SIGTERM triggers a minifi.sh stop so minifi is stopped gracefully.
+1 merging


fix: 🤖 (node) support node 14

:tada: This PR is included in version 3.0.2 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Expose directed rounding functions?

In a project, I need to implement the function

Clearly I cannot do it naively because the singularity of the logarithm at 0.  To that aim, directed rounding methods (such as x.add_ru(y), x.mul_rd(y),...) would be useful.  Of course I can always make a singleton with each boundary point by I would be making twice as many computations (and half would be thrown away).  What do you think?

I'm not sure about how you are going to use add_ru, etc., but if you want to experiment with them, you can git clone inari locally, and change inari = "0.x" to inari = { path = "/path/to/inari" } in your project's Cargo.toml (please check Specifying path dependencies.) Then, you will be able to expose any functions from inari.


Blade rendering driver

Depends directly on illuminate/view, no wrapper library.
Inspirations:

https://github.com/ryangjchandler/standalone-blade
https://jenssegers.com/using-blade-outside-of-laravel

This issue made me dislike Blade. This is just a templating engine, and supposedly can work outside of the framework, so why am I obliged to instantiate 5 other classes, use IoC, and worst of all use File IO instead of being able to just input the template a string with data and receive HTML as output 😠


jax.jit crashes only as a class method (calling jax.scipy.linalg.lu_solve)

Description
Hi,
For some research, I have to solve a large set of LU systems (using vmap). I found that my code systematically crashes (on Jupyter "the kernel dies", on a Python script "segmentation fault") when I implement the jit function as a class method and define the first argument as static.
Here is a reduced snippet of the code that fails:
key = jnr.PRNGKey(5)
lhs = jax.random.normal(key, (577, 770, 770))
rhs = jnp.zeros((577, 770))

class Solver:
  def __init__(self, lu):
    self.lu = lu
  def solve(self, rhs_0):
    return jax.vmap(jax.scipy.linalg.lu_solve)(self.lu, rhs_0)

  @partial(jax.jit, static_argnums=(0,))
  def solve_jit(self, rhs_0):
    return jax.vmap(jax.scipy.linalg.lu_solve)(self.lu, rhs_0)

lu = jax.vmap(jax.scipy.linalg.lu_factor)(lhs)

solver = Solver(lu)
sol = solver.solve(rhs)
sol = solver.solve_jit(rhs)

The first solve passes while the second one fails. I use a NVIDIA A100 so GPU memory should not be a problem here.
What is interesting is that I tried implementing the jit function as a classical function (outside of a class) and this runs without any problem.
What jax/jaxlib version are you using?
jax 0.4.13, jaxlib 0.4.13+cuda11.cudnn86
Which accelerator(s) are you using?
GPU
Additional system info
Python 3.10.12 running on Linux
NVIDIA GPU info
No response

Hi - the issue is that you've marked self as static, but self contains a very large array self.lu which is being embedded as a constant as a result of being declared static, and this is causing your code to exhaust the available memory.
To fix this (and in general) I'd recommend avoiding marking self as static when JIT compiling class methods: it can cause problems in other situations as well. You can read more about this at FAQ: How to use JIT with methods?.

Generally speaking it's an antipattern to use classes with raw JAX, for the reasons Jake highlights.
Try Equinox.

Thanks for the quick replies,
Indeed, I was aware that this probably had to do with GPU memory but didn’t know that static args would mean large storage / copies.
It might be something interesting to mention in the doc, where solutions to JIT compiling code are given.
I’ll probably refactor mine to make it more « function based » instead of « class based », it seems to be the way to go.


[cluster-test] Add support for CT run with multi VASP accounts on premainnet

Now, AOS created 16 VASP account for CT, this change make CT can load up to 16 VASP accounts as seeds account to create child VASP, which means for CT run on premainnet, we can create up to 4098 accounts instead of 256.

Motivation
(Write your motivation for proposed changes here.)
Have you read the Contributing Guidelines on pull requests?
(Write your answer here.)
Test Plan

Test with Local swarm:
cargo run -p cluster-test -- --mint-file "/private/var/folders/qd/tg50mr0d7k75qvztr_8yk2km0000gn/T/437cd0ef91b051e08db5c1bfe19d922d/mint.key" --swarm --peers "<IP_ADDRESS>:8080" --emit-tx --chain-id TESTING

2020-10-14T22:00:54.224625Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:234 Will use 10 workers per AC with total 10 AC clients
2020-10-14T22:00:54.224681Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:246 Will create 15 accounts_per_client with total 150 accounts
2020-10-14T22:00:54.225043Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:376 Creating and minting faucet account
2020-10-14T22:00:55.136032Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:394 DD account current balances are<PHONE_NUMBER>854775807, requested 150000000 coins
2020-10-14T22:00:55.136161Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:411 Creating and minting seeds accounts
2020-10-14T22:00:55.785841Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:422 Completed creating seed accounts
2020-10-14T22:00:56.437173Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:480 Completed minting seed accounts
2020-10-14T22:00:56.437204Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:481 Minting additional 150 accounts
2020-10-14T22:01:06.580980Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:518 Mint is done
2020-10-14T22:01:06.581960Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:280 Tx emitter workers started
2020-10-14T22:01:16.583481Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:547 submitted: 60 txn/s, committed: 45 txn/s, expired: 0 txn/s, latency: 2433 ms, p99 latency: 2500 ms
2020-10-14T22:01:26.587577Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:547 submitted: 60 txn/s, committed: 60 txn/s, expired: 0 txn/s, latency: 2324 ms, p99 latency: 2500 ms
2020-10-14T22:01:36.588852Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:547 submitted: 60 txn/s, committed: 60 txn/s, expired: 0 txn/s, latency: 2357 ms, p99 latency: 2450 ms
2020-10-14T22:01:46.590324Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:547 submitted: 60 txn/s, committed: 60 txn/s, expired: 0 txn/s, latency: 2398 ms, p99 latency: 2550 ms
2020-10-14T22:01:56.591437Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:547 submitted: 60 txn/s, committed: 60 txn/s, expired: 0 txn/s, latency: 2435 ms, p99 latency: 2500 ms
2020-10-14T22:02:06.592563Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:547 submitted: 60 txn/s, committed: 60 txn/s, expired: 0 txn/s, latency: 2459 ms, p99 latency: 2500 ms
Total stats: submitted: 3600, committed: 3600, expired: 0
Average rate: submitted: 60 txn/s, committed: 60 txn/s, expired: 0 txn/s, latency: 2400 ms, p99 latency: 2500 ms

Test with premainnet
./target/debug/cluster-test --swarm --emit-tx --premainnet --chain-id=18 --peers=a72ffeb94e81111ea81b70e75131a869-1894457723.ap-northeast-1.elb.amazonaws.com:80,<IP_ADDRESS>:80,ada61ba47e80d11ea9bcd0a7a792c33e-1496143175.ap-southeast-1.elb.amazonaws.com:80,<IP_ADDRESS>:80,libra-breakthrough-full-node.man.bdi.sh:80,ab6b29969132f484db9fdc9d302b92e1-9b15e1987ad0319f.elb.eu-west-1.amazonaws.com:80,<IP_ADDRESS>:80,<IP_ADDRESS>:80,<IP_ADDRESS>:80,ad49a6c1e472c4f82ac8507df2d81f58-e2858d056c4dfef9.elb.us-east-2.amazonaws.com:80,aeb83d6e0bf8f46f7bb25dd68c82f021-db6448367d0c7b7b.elb.us-east-1.amazonaws.com:80,<IP_ADDRESS>:80,premainnet.libra.novi.com:80,ae5dfb87ce81311ea8c230282fe68c3b-1804125671.us-west-2.elb.amazonaws.com:80,<IP_ADDRESS>:80,<IP_ADDRESS>:80,<IP_ADDRESS>:80,libra-slow-full-node.man.bdi.sh:80,<IP_ADDRESS>:80,libra-temasek-full-node.man.bdi.sh:80,a9a46a9e8c0aa4ab0995a36de953d2f5-68825d4ac845d301.elb.us-west-2.amazonaws.com:80,<IP_ADDRESS>:80,a4d1bf61ae81311eaaaf002ebf759cd5-341763703.us-east-2.elb.amazonaws.com:80,a8c8481bcc817472581e91a468c671da-44b7ecf30878f348.elb.eu-west-1.amazonaws.com:80 --accounts-per-client=15 --workers-per-ac=10

2020-10-14T21:43:54.730777Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:234 Will use 10 workers per AC with total 240 AC clients
2020-10-14T21:43:54.730858Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:246 Will create 15 accounts_per_client with total 3600 accounts
2020-10-14T21:43:54.731231Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:388 Loading faucet account from DD account
2020-10-14T21:43:55.144686Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:394 DD account current balances are<PHONE_NUMBER>00000, requested<PHONE_NUMBER> coins
2020-10-14T21:43:55.144934Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:426 Loading VASP account as seed accounts
2020-10-14T21:43:58.350900Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:436 Loaded 16 VASP accounts
2020-10-14T21:44:14.476527Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:480 Completed minting seed accounts
2020-10-14T21:44:14.476583Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:481 Minting additional 3600 accounts
2020-10-14T21:46:37.250433Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:518 Mint is done
2020-10-14T21:46:37.304419Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:280 Tx emitter workers started
2020-10-14T21:46:47.305427Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:547 submitted: 459 txn/s, committed: 256 txn/s, expired: 0 txn/s, latency: 7120 ms, p99 latency: 8250 ms
2020-10-14T21:46:57.306480Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:547 submitted: 443 txn/s, committed: 394 txn/s, expired: 0 txn/s, latency: 6580 ms, p99 latency: 17350 ms
2020-10-14T21:47:07.307720Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:547 submitted: 409 txn/s, committed: 423 txn/s, expired: 0 txn/s, latency: 6138 ms, p99 latency: 16950 ms
2020-10-14T21:47:17.308851Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:547 submitted: 464 txn/s, committed: 461 txn/s, expired: 0 txn/s, latency: 6137 ms, p99 latency: 9900 ms
2020-10-14T21:47:27.309959Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:547 submitted: 430 txn/s, committed: 381 txn/s, expired: 0 txn/s, latency: 6401 ms, p99 latency: 26400 ms
2020-10-14T21:47:37.310943Z [main] INFO testsuite/cluster-test/src/tx_emitter.rs:547 submitted: 420 txn/s, committed: 423 txn/s, expired: 0 txn/s, latency: 6054 ms, p99 latency: 8900 ms
Total stats: submitted: 27990, committed: 27990, expired: 0
Average rate: submitted: 446 txn/s, committed: 446 txn/s, expired: 0 txn/s, latency: 6408 ms, p99 latency: 21550 ms

Related PRs
(If this PR adds or changes functionality, please take some time to update the docs at https://github.com/libra/libra/developers.libra.org, and link to your PR here.)

/land


visualize_retargeting - ERROR: Shadow framebuffer is not complete, error 0x8cd7

Hi
I am trying to use your code for retargeting, and am attempting to view the result of visulization.
I meet a problem when rendering.
“ERROR: Shadow framebuffer is not complete, error 0x8cd7”.
I don't know how to solve this and need help.
I would appreciate any help you have to offer!

Hi @JingboZeng ,
It seems that this is a very general message related to the visualization of your system. It is possible for you to run MuJoCo demo?
Also you can take a look of the issues here. Maybe it can help.

Hi@yzqin
I can run the MuJoCo demo, there is no error. And I can correctly get the rendering result.
I have seen the issues in this link, it does work when running demo. But when using your code, this method does not work.

hi @JingboZeng
Does this error happen for all the DexMV demo or only the retargeting one? Can you please share more information about your system, MuJoCo version?
Thanks

hi @yzqin
Here are the information about my system
OS: Ubuntu 16.04.12
MuJoCo Version: 2.0
MuJoCo-py Version: <IP_ADDRESS>
python version: 3.7.11

Oh I see. The DexMV code is tested on Ubuntu 18.04, 20.04, and 22.04. I am sorry that I do not have a 16.04 system tested for you.
If you can run other mujoco-py programs but fails on DexMV, one possible reason is some GLFW support breaks for Ubuntu 16.04, which will affect this line of code

Thank you for your help.


willdurand/Hateoas#325 support for php attributes

Adds a new service hateoas.configuration.metadata.attribute_driver for the AttributeDriver class (https://github.com/willdurand/Hateoas/pull/326)
Moreover, hateoas.configuration.metadata.attribute_driver and hateoas.configuration.metadata.annotation_driver are combined in a new service hateoas.configuration.metadata.annotation_or_attribute_driver to give the extension feature to both services.
The hateoas.configuration.metadata.attribute_driver will be removed if the PHP version is prior 8.1.0, so the hateoas.configuration.metadata.annotation_or_attribute_driver will have only the annotation driver.
This PR will require a new release version of willdurand/Hateoas

Is anyone handling this PR?

any new about this PR?

Hi @jeandonaldroselin ,
I am still waiting for the final review by the maintainers.
Nonetheless, I appreciate any suggestions that may improve the code.
Same for https://github.com/willdurand/Hateoas/pull/326
Thank you kindly

I'm really sorry if this took so long. I have merged https://github.com/willdurand/Hateoas/pull/326 and tagged https://github.com/willdurand/Hateoas/releases/tag/3.11.0-beta1
Can someone please rebase this pull request?

Hi @goetas
Rebase was done. thanks

So, I see a lot of test failing. if it helps, i'm fine with requiring php ^8.1 and willdurand/Hateoas 3.1@beta.  That would simplify the pipelines and allow you to not worry about some edgecases

I will try to fix the tests this evening.  I will reach out to you again in case I am unable to fix them.
To be honest, my intention was not to be disruptive with the php version.

Hi @goetas
I believe this would be ready for a new try.
Thanks

To me looks good. I'm going to merge it and to tag a beta release.
Please ping me here again after some time so I tag a stable release if there will be no problems.

Great, I'll try the beta release on sylius resource package. Thx a lot

The ci failures are unrelated to your changes

@goetas I don't think that the failures are unrelated.
The tests with PHP 8.0 fail because this PHP version knows about attributes but does not support nested attributes which are used by the test fixtures.
That's why tests with PHP < 8.0 and PHP > 8..0 are green but tests with PHP 8.0 are red.
I encountered a similar issue when working on https://github.com/willdurand/Hateoas/pull/334
The simplest solution to fix that would be to drop support for PHP < 8.1. WDYT?

@loic425  did you try the beta ?

@loic425 Did you also try the latest beta https://github.com/willdurand/Hateoas/releases/tag/3.12.0-beta1 together with https://github.com/willdurand/BazingaHateoasBundle/pull/110 ?
Especially https://github.com/willdurand/Hateoas/commit/b3b5da5cd4222df15e468132f01f830e86ef8974 is important to make willdurand/hateoas usable without annotations.
This commit is part of the latest beta version.

@loic425 great! thanks for letting me know!
@W0rma are you available anywhere for a chat regarding the recent changes you did? you can contact me via any of the possible ways listed on my profile https://github.com/goetas


Unit tests for gem.String

Add unit tests for the remaining functions of gem.String usage and manipulation.

[x] #4
[x] #5
[x] #6
[x] #7
[x] #8
[x] #9
[x] #12
[x] #13
[x] #14

All tasks complete, gem.String is tested! glub.


Use Team working day settings to show days or weekend

Did I not set the working days already in the team settings? Why not use that?
this will avoid showing weekends when not needed.

We're closing old issues so we can get back to actively maintaining the extension. Feel free to reopen if this is still an issue for you.


passing style prop to AIRMap component

I might be missing something but it seemed like the style prop wasn't being consumed anywhere so I couldn't do things like
<MapView style={{flex: 1}} />

@ex1machina styles passed in will already be included.  The {...props} call passes all props passed in to the component.


keyboard+MSC problem

I Implemented an example of the composite device (KB+MSC) using this library in cube-mx for STM32F405. Every thing is Ok and the devices (KB & MSC) is shown in device manager immediately after USB plugin. But there are 2 problems:
1- There is no new drive in my computer.
2- Keyboard will work about 10 minutes after USB plugin.

The problem was solved by myself !
the issue was from the buffer assignment.


Your image should be 100 pixels wide. Browser zoom should be at 100%. - Error doesn't goes away

Challenge Size your Images has an issue.
User Agent is: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36.
Please describe how to reproduce this issue, and include links to screenshots if possible.
My code:
<link href="https://fonts.googleapis.com/css?family=Lobster" rel="stylesheet" type="text/css">

CatPhotoApp

Kitty ipsum dolor sit amet, shed everywhere shed everywhere stretching attack your ankles chase the red dot, hairball run catnip eat the grass sniff.

Purr jump eat the grass rip the couch scratched sunbathe, shed everywhere rip the couch sleep in the sink fluffy fur catnip scratched.

The code above works if the image is given the class "smaller-image" as required to apply the style; there is no issue with the challenge. Please consult the help chat if you get stuck before reporting an issue. Happy coding!

@shadab044 Yup it would pass after the first test passes.
Please consult the Help Chat Room for any assistance to Coding.
Happy Coding!
/cc @BKinahan Thanks.

This is so hard!

Is there anybody able to help me on "Size your Images"?
Thank You!

@xiaji2, You could use the Help chat room for getting challenge related help.
Happy Coding!


add Serious Sam 1 & 2

add Serious Sam 1 & 2  with serious-engine

https://github.com/tx00100xt/SeriousSamClassic
https://github.com/tx00100xt/SeriousSamClassic-VK
will look into using one of these to see what works best

working on this
it fails to build, opened issue ticket with Dev

runs like junk, still working on it, for now only SS1 is building and runnning.
moving on to something else

done


Add data: Sudan Al Gadaref CEO 2020

@bhyeh and @adebowaledaniel, kindly review this when you have the chance. Thanks!


Observable constructor

Coverage remained the same at 87.941% when pulling f50f1d241a50919baf714e757056e114cb5f2031 on observable-constructor into 0388f7833983499c3d903977d6388989e057dde9 on master.


Fix fontspark broken url details on f.ts

fontspark.app does not exist anymore. It is now fontspark.com
Pull Request Template
Please ensure all items below are checked before creating a pull request:

[x] My changes were made in the resources folder, not in the auto-generated README.md file or db folder
[x] My submission is formatted according to the guidelines in the contributing guide
[x] My submission is ordered alphabetically based on the resource name
[x] My submission has a useful description
[x] I have searched the repository for any relevant issues or pull requests

Thank you!


Autocrafter that emits a redstone signal to request producing an output

I still need a good name for this, and need to determine if this goes in a separate block, but it would be cool if you could:

Specify an output resource in the autocrafter
Make it available for autocrafting
And when requesting, the corresponding would emit a redstone signal
... until the requested amount of that output is returned

I would imagine it would be a separate block, similar to a detector. Maybe call it a redstone crafter? Or a farm crafter? Or something?

I think it could be a pattern variant where you set the redstone power (default to 15) and the expected item
someone who can do redstone could use the 9 patterns that way


web apps install

See https://github.com/openmicroscopy/www.openmicroscopy.org/pull/249 points raised by @joshmoore and discussion with @rgozim about poor visibility of OMERO.web app installs on our docs pages.
This adds a section to the OMERO.web install pages (just did one for now - will duplicate to the others once this one is good) to link to the new www.openmicroscopy.org/omero/apps page and make sure sysadmins know about existence of web apps.

Any comments about this ASAP would give me more chance to get it right (today and tomorrow).


Fix imagePullSecrets example.

Fixes #9740
Fixed broken link in docs.
Converted json to yaml.
Tested the examples (substituting my actual credentials for my private image repository).
Improved example.

LGTM but I'm no TL

@bgrant0607 want to do a quick one-over?

Further improvements are possible, but LGTM

I can't see what the Shippable error is. Due to some bug, I can't get any content on their build status pages.

Closing and re-opening to kick shippable.

Shippable failed again. @erictune can you investigate and add this to the flakes list if it is a flake.

Please re-add the ok-to-merge label once the shippable errors have been fixed.

shippable error was legit.
Fixed.

watching shippable and will re-add label when passes

Shippable looks like a flake, as it's been doing all day. Are you worried about this breaking anything?

I don't think this can break anything.

I wouldn't expect it could either. cc @zmerlynn for merging tomorrow

@zmerlynn can I get a merge please.

Sorry, thanks for the ping.


L1, L2, L3 segments are created too when only L4 segment defined

Hi. I am working on architecture that has only L4 (length=4) routing segments. But when I build architecture, I see L1, L2, L3 segments are created on bottom and left sides. What is the reason of this?

Thanks.


Add files via upload

@jackhwarner Please fork SWOT repository, copy file.zip in the root folder of your forked SWOT repository. Then unzip file.zip there with no changes to the directory structure. Then make a new pull request. If you are not able to make the pull request according the instructions, please send us an email to<EMAIL_ADDRESS>Please mention this request URL in the message. I have to close this request without merging. Thank you for your understanding.

Hi, should I add the file before or after unzipping?
On Fri, Jun 7, 2024 at 2:12 PM Philip Torchinsky @.***>
wrote:

@jackhwarner https://github.com/jackhwarner Please fork SWOT
repository, copy file.zip in the root folder of your forked SWOT
repository. Then unzip file.zip there with no changes to the directory
structure. Then make a new pull request. If you are not able to make the
pull request according the instructions, please send us an email to
@.*** Please mention this request URL in the
message. I have to close this request without merging. Thank you for your
understanding.
—
Reply to this email directly, view it on GitHub
https://github.com/JetBrains/swot/pull/21109#issuecomment-2155300336,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ASWML4Q6EOXKFVA7MRGS3LLZGHZYRAVCNFSM6AAAAABI7CICYSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJVGMYDAMZTGY
.
You are receiving this because you were mentioned.Message ID:
@.***>


{Monitor} az afd: Migrate commands to codegen

Related command
All az afd commands
afd
Description
Testing Guide
History Notes
[Component Name 1] BREAKING CHANGE: az command a: Make some customer-facing breaking change
[Component Name 2] az command b: Add some customer-facing feature

This checklist is used to make sure that common guidelines for a pull request are followed.

[ ] The PR title and description has followed the guideline in Submitting Pull Requests.

[ ] I adhere to the Command Guidelines.

[ ] I adhere to the Error Handling Guidelines.

Monitor

If there are breaking changes in this PR, we cannot merge it before next breaking change window (Build event in May, 2024).
Azure CLI only allows breaking changes released in breaking change windows (twice each year).


Remove debug statement

This causes some confusion when attempting to integrate the timeout script into your application

Thanks for raising @said-moj. Fixed by https://github.com/hannalaakso/accessible-timeout-warning/pull/4


updated_cryptography_version

Changes

Updated Cryptography version to be grater than 3.2 to fix the potential security vulnerabilities.

I reverted this merge.  This needs to be reparented to dev.  We don't deliver directly to master (except hot fixes)

I reverted this merge.  This needs to be reparented to dev.  We don't deliver directly to master (except hot fixes)


Creates duplicate bower_components directory.

I came across an issue where gulp-bower generates the bower_components directory at the destination i set but it also creates a bower_components directory in the directory containing the bower.json file outside of the destination. I'm also seeing this via terminal and it seems like the bower task finishes then it runs a caching implementation which creates another directory.
Joshs-MacBook-Pro:dev jbedo$ gulp pinchme
[11:49:10] Using gulpfile ~/Desktop/PINCHme/kendall/kendall/pinchme_mobile/dev/gulpfile.js
[11:49:10] Starting 'less'...
[11:49:10] Finished 'less' after 9.98 ms
[11:49:10] Starting 'js'...
[11:49:10] Finished 'js' after 15 ms
[11:49:10] Starting 'bower'...
../www/js/vendors/
[11:49:10] Using cwd:  /Users/jbedo/Desktop/PINCHme/kendall/kendall/pinchme_mobile/dev
[11:49:10] Using bower dir:  ./bower_components
[11:49:10] Finished 'bower' after 4.53 ms
[11:49:10] Starting 'pinchme'...
[11:49:10] Finished 'pinchme' after 9.35 μs
[11:49:16] bower cached git://github.com/marionettejs/backbone.marionette.git#2.2.2
[11:49:16] bower validate 2.2.2 against git://github.com/marionettejs/backbone.marionette.git#~2.2.2
[11:49:16] bower cached git://github.com/marionettejs/backbone.radio.git#0.8.2
[11:49:16] bower validate 0.8.2 against git://github.com/marionettejs/backbone.radio.git#~0.8.2
[11:49:21] bower cached git://github.com/jashkenas/backbone.git#1.1.2
[11:49:21] bower validate 1.1.2 against git://github.com/jashkenas/backbone.git#>=0.9.9 <=1.1.2
[11:49:21] bower cached git://github.com/jashkenas/underscore.git#1.7.0
[11:49:21] bower validate 1.7.0 against git://github.com/jashkenas/underscore.git#>=1.4.4 <=1.7.0
[11:49:21] bower cached git://github.com/jashkenas/underscore.git#1.6.0
[11:49:21] bower validate 1.6.0 against git://github.com/jashkenas/underscore.git#1.4.4 - 1.6.0
[11:49:21] bower cached git://github.com/jquery/jquery.git#2.1.1
[11:49:21] bower validate 2.1.1 against git://github.com/jquery/jquery.git#^1.8.0 || ^2.0.0
[11:49:21] bower cached git://github.com/marionettejs/backbone.wreqr.git#1.3.1
[11:49:21] bower validate 1.3.1 against git://github.com/marionettejs/backbone.wreqr.git#^1.0.0
[11:49:21] bower cached git://github.com/marionettejs/backbone.babysitter.git#0.1.5
[11:49:21] bower validate 0.1.5 against git://github.com/marionettejs/backbone.babysitter.git#^0.1.0
[11:49:21] bower install backbone.radio#0.8.2
[11:49:21] bower install marionette#2.2.2
[11:49:21] bower install underscore#1.6.0
[11:49:21] bower install backbone.wreqr#1.3.1
[11:49:21] bower install backbone.babysitter#0.1.5
[11:49:21] bower install backbone#1.1.2
[11:49:21] bower install jquery#2.1.1

gulpfile.js
// package.json has the projectPath: '../www/js/vendors/` set
gulp.task('bower', function() {
  console.log(pkg.projectPath);
  return bower()
    .pipe(gulp.dest(pkg.projectPath))
});

You can use .bowerrc for configure bower directory: http://bower.io/docs/config/#directory

The problem occurs when you configure a bower directory within the .bowerrc file. It creates the specified directory and also creates a bower_components directory.

Use:
gulp.task('bower', function() {
  console.log(pkg.projectPath);
  return bower()
});


Accept objects in push and replace

Discussed on Discord a bit, thought I'd get some code in place.
I think this is a bit of a first step toward https://github.com/rackt/react-router/issues/2186, as described in https://github.com/rackt/react-router/issues/2186#issuecomment-157911118.

The immediate benefit of this will be to enable patterns like
history.push({ ...location, query: nextQuery })

Squashed in a couple of test cases showing off the using spreads.
I'm aware this doesn't update the changelog - it takes me a long time to write that sort of thing, so wanted to get feedback on the new functionality before adding a changelog entry.

Ah - this is a little tricky because of our semantics with handling search and query. I added some failing test cases for the moment.

Well - this resolves it. I don't love this implementation, but I don't hate it either.
For a breaking change, it'd be nicer to just drop the search entirely from the location that listen gives to listeners; this would also make the API a bit cleaner, since it means that the search I get back is exactly the search I pass in.
In practice, though, I think this implementation is good enough.

I have a local branch that implements this as well; just haven't had time to push it yet. I'll take a look at this on my flight to the UK this weekend. Thanks for all your work here, @taion. :)

A couple of things to note - I'd like to resolve this TODO sooner rather than later, but it'd require a (mostly non-breaking) 2.x. Should hold off on actually doing a major version bump to aggregate a few more of these technically breaking changes, though.
Also, my real motivation here isn't just that this API feels "cleaner" in some sense. Concrete things:

I strongly suspect that supporting mobile routing is going to require a bit more than just the existing state, path, query tuple, and is going to instead need a few extra keys. For this, it's going to be nicer to pass in an object from the start, rather than mucking with the signature on pushState. cc @skevy
This lets someone implement named routes as a first-class things w/e.g. a useNamedRoutes history enhancer, that allows the user to do e.g. history.push({name: "foo", params: {fooId: 3}}) (and possibly just history.push("foo")). It wouldn't even require a different <Link>, as <Link> could just accept an object for props.to, and pass that to history.push.

cc @jlongster
This also opens up the path toward adding a more generic history.transition (or history.update or whatever) that doesn't presuppose action. Might be a better entry point for an integration like redux-simple-router that just wants to forward stuff onto the history; otherwise for completeness I feel like that library ought to have a replacePath (with the current API).

On second thought, that might be too generic an API. Probably better for integrations that want to do this to just implement both push and replace hooks.

The work in #162 is going to touch a lot of these same points. I think it may be a bit cleaner to do that first.

I disagree - I don't think #162 is on-path. The longer-term goal here is to offer a "generic" API to push and replace that better supports paradigms outside of the state/query/path paradigm, which is very focused toward doing path-based routing in browser. One of the goals here is to move the React Router <Link> API to be something like:
<Link to={{path: '/foo', query: {the: 'query'}, state: {the: 'state'}}>

Where the arguments to link.props.to are passed directly to history.push. This then lets users just sub out the history without changing the <Link> component, which is going to be really great for RN support, as well as for "good" userspace named routes support.

Where the arguments to link.props.to are passed directly to history.push

This is appealing.

Thanks, @taion! Let's be sure to follow up with a note in CHANGES.md.

Ah crud, I should also have cleaned up the commits on this branch.
I want to drop the hack in useQueries ASAP - entirely just a question of what else we want to break first. I'm trying to track possible things on https://github.com/rackt/history/issues/164... possibly at least all the things under "minor", just to get those out of the way.


add style file

As we use the bootstrap, the CSS file is not necessary. I created a style.py file to combine all the styling things for the app.


json file

Hi,
Is there a way to get a json file instead of QR codes for bulk import into Aegis / 2FAS / Ente?

I was initial thinking about this, but the problem is there are different export/import formats. I believe Aegis supports importing the raw com.authy.storage.tokens.authenticator.xml file.
Anyway, I've added really basic JSON export feature, using the Aegis format. It writes a file to the current directory. P.S. not much testing has been done it.


Flame tests are too brittle

It seems that very minor changes can result in our flame allocation tests to break. I suspect that this is due to inference failures, and we should probably fix this. A quick patch for now, to at least alleviate some of the headache is to use a buffer, like we do for the threaded tests (which are inherently flakey).
Thoughts @Sbozzolo?

I think they're even flakey, which is worse. I updated the allocation limits in https://github.com/CliMA/ClimaAtmos.jl/pull/2673, and re-running still resulted in failure. This is another indicator that the issue is related to inference.

#2673 is going to change buffer = occursin("threaded", job_id) ? 1.4 : 1 to buffer = occursin("threaded", job_id) ? 1.4 : 1.1 so that we have a bit of a buffer.

But we should revert this and fix the issue.

I had precisely the same thought and I would proposed the same solution.


This change most likely fixes the LSP deadlock.

Changes LSPFileUpdates to track the range of edit versions that they contain so we:
A) Can properly time-travel to before the set of updates occurred to make fast path decisions in the past.
B) Can avoid accidentally throwing way undo history for merged updates.
I've added tests for these edge cases.

Motivation

LSP was deadlocking on people.
Test plan

See included automated tests.

Thank you for debugging this with urgency and for adding a test


api merge

Hiii @yingliming , thanks for your contribution, but I suggest using init func to register the routes, like we did in
https://github.com/horizoncd/horizon/blob/8534fadd202f92bf1c59ce4b363a112f26bae148/pkg/cluster/cd/workload/rollout/rollout.go#L21
https://github.com/horizoncd/horizon/blob/8534fadd202f92bf1c59ce4b363a112f26bae148/pkg/cluster/registry/registry.go#L22


State Verification

Tasks that relate to the verification of state.

Closing this issue because I don't really see a long term need for this epic. @amerameen feel free to reopen if you'd like


python311Packages.types-pytz: 2<IP_ADDRESS>40203 -> 2<IP_ADDRESS>40417

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python311Packages.types-pytz is: Typing stubs for pytz
meta.homepage for python311Packages.types-pytz is: https://github.com/python/typeshed
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 2<IP_ADDRESS>40417 with grep in /nix/store/9k9xz27j5wzcvjkr33p7gglhwn7a6ndf-python3.11-types-pytz-2<IP_ADDRESS>40417
found 2<IP_ADDRESS>40417 in filename of file in /nix/store/9k9xz27j5wzcvjkr33p7gglhwn7a6ndf-python3.11-types-pytz-2<IP_ADDRESS>40417

Rebuild report (if merged into master) (click to expand)

20 total rebuild path(s)

20 package rebuild(s)

First fifty rebuilds by attrpath
home-assistant-component-tests.ridwell
home-assistant-component-tests.simplisafe
python311Packages.aioridwell
python311Packages.django-stubs
python311Packages.djangorestframework-stubs
python311Packages.layoutparser
python311Packages.pandas-stubs
python311Packages.pdfplumber
python311Packages.pyngo
python311Packages.simplisafe-python
python311Packages.types-pytz
python312Packages.aioridwell
python312Packages.django-stubs
python312Packages.djangorestframework-stubs
python312Packages.layoutparser
python312Packages.pandas-stubs
python312Packages.pdfplumber
python312Packages.pyngo
python312Packages.simplisafe-python
python312Packages.types-pytz

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/9k9xz27j5wzcvjkr33p7gglhwn7a6ndf-python3.11-types-pytz-2<IP_ADDRESS>40417 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python311Packages.types-pytz https://github.com/r-ryantm/nixpkgs/archive/4c777661f7f178c7a505f2bdf3a538dfd6ca9e19.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/4c777661f7f178c7a505f2bdf3a538dfd6ca9e19#python311Packages.types-pytz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/9k9xz27j5wzcvjkr33p7gglhwn7a6ndf-python3.11-types-pytz-2<IP_ADDRESS>40417
ls -la /nix/store/9k9xz27j5wzcvjkr33p7gglhwn7a6ndf-python3.11-types-pytz-2<IP_ADDRESS>40417/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review --extra-nixpkgs-config '{ allowInsecurePredicate = x: true; }' run on x86_64-linux 1

8 packages failed to build:

python311Packages.layoutparser
python311Packages.layoutparser.dist
python311Packages.pdfplumber
python311Packages.pdfplumber.dist
python312Packages.layoutparser
python312Packages.layoutparser.dist
python312Packages.pdfplumber
python312Packages.pdfplumber.dist

30 packages built:

home-assistant-component-tests.ridwell
home-assistant-component-tests.simplisafe
python311Packages.aioridwell
python311Packages.aioridwell.dist
python311Packages.django-stubs
python311Packages.django-stubs.dist
python311Packages.djangorestframework-stubs
python311Packages.djangorestframework-stubs.dist
python311Packages.pandas-stubs
python311Packages.pandas-stubs.dist
python311Packages.pyngo
python311Packages.pyngo.dist
python311Packages.simplisafe-python
python311Packages.simplisafe-python.dist
python311Packages.types-pytz
python311Packages.types-pytz.dist
python312Packages.aioridwell
python312Packages.aioridwell.dist
python312Packages.django-stubs
python312Packages.django-stubs.dist
python312Packages.djangorestframework-stubs
python312Packages.djangorestframework-stubs.dist
python312Packages.pandas-stubs
python312Packages.pandas-stubs.dist
python312Packages.pyngo
python312Packages.pyngo.dist
python312Packages.simplisafe-python
python312Packages.simplisafe-python.dist
python312Packages.types-pytz
python312Packages.types-pytz.dist

Maintainer pings
cc @fabaff for testing.

Already on master.


fix($compile) : Add base#href to the list of RESOURCE_URL context attrs

This will put the same security constraints to bindings on base#ref as on iframes or script srcs, since it changes the behavior of relative URLs across the page. Currently, they don't have a $sce context. Also, it's more generally a good idea in Angular itself, as since #15144 and its fix #15145 we'll consider that baseURI as a trusted origin.
Does this PR introduce a breaking change?
Yup: something like <base href="{{myUrl}}"/> will send myUrl to the $sce's RESOURCE_URL checks. By default, it will break if myUrl isn't same-origin. Also, concatenation in trusted contexts is disabled: "/{{myPathComponent}}/something" won't work at all.
This follows the discussion in #15537 and the initial issue in #15144, fixed in #15145.

@rjamet, what is your take on https://github.com/angular/angular.js/pull/15145#issuecomment-272115379? Is it OK to release #15145 without this change, or should they rather go together (or not at all)?

For anyone wondering this question was answered in https://github.com/angular/angular.js/pull/15145#issuecomment-273113931

both PRs at the same time would be the safest.


New site: WALTON-CLOUD

Summary

Brings a new site to the cloud-info-provider: WALTON-CLOUD

Related issue :

Missing an ops project that we can reference here

Missing an ops project that we can reference here

For which purpose exactly?

Missing an ops project that we can reference here

For which purpose exactly?

We need the project id in OpenStack to be able to map it to a VO (ops) so cloud-info-provider can extract all the information

xref: https://github.com/tdviet/fedcloudclient/pull/179


progess bar and info

Put progress bar and "this project is updating! We will send you an updated email once we are done!" together

fixed.


CLOUD-61260 - Reinstate recipes on Uluwatu UI

@keyki @doktoric pls review

Refer to this link for build results (access rights to CI server needed):
http://ci.sequenceiq.com//job/cloudbreak-pull-request/2813/

Refer to this link for build results (access rights to CI server needed):
http://ci.sequenceiq.com//job/mock-integration-test/304/

Refer to this link for build results (access rights to CI server needed):
http://ci.sequenceiq.com//job/cloudbreak-pull-request/2816/

Refer to this link for build results (access rights to CI server needed):
http://ci.sequenceiq.com//job/mock-integration-test/308/

Refer to this link for build results (access rights to CI server needed):
http://ci.sequenceiq.com//job/cloudbreak-pull-request/2819/

Refer to this link for build results (access rights to CI server needed):
http://ci.sequenceiq.com//job/mock-integration-test/311/

Refer to this link for build results (access rights to CI server needed):
http://ci.sequenceiq.com//job/cloudbreak-pull-request/2823/

Refer to this link for build results (access rights to CI server needed):
http://ci.sequenceiq.com//job/mock-integration-test/315/


Rianhughes/common feeder2core

depends https://github.com/NethermindEth/juno/issues/1146
closes https://github.com/NethermindEth/juno/issues/1147
This PR does two things:

rename feeder2core to sequencer2core, and move it to utils
move network.go from utils to core (to prevent import cycles)

Left a comment on the issue.


$request->withHeader(); doesn't set the header.

$request is an instance of > \Amp\Artax\Request
$request->withHeader("Content-Type", "application/x-www-form-urlencoded");
So the header should be set, but if I do var_dump($request->getHeader("Content-Type"); it says "NULL"

Request is immutable and returns a new instance, so you need $request = $request->withHeader('content-type', 'application/x-www-form-urlencoded');

Oh, ok.
array(1) {
  ["content-type"]=>
  array(1) {
    [0]=>
    string(33) "application/x-www-form-urlencoded"
  }
}

It's ok that "application/x-www-form-urlencoded" is into an array?

Thank you for your help.


Fixed datalab Initlialization actions

Manually added missing GPG keys to fix the apt-get update command.

/gcbrun

/gcbrun


[Feature Request]: Intercept requests made by the worker

Hey! First thanks for this awesome project, We are currently using the one from dollar shave club but that project died :-(
We need a way to strip the added Cloudflare headers added in MiniFlare as it causes Cloudflare to block the request. We need to route to our upstream via cloudflare as there is no direct access to the servers.
So one thought was a way to have some sort of middleware we can write to strip the needed headers to make it work
Or maybe there can be an option to not add any CF headers when the worker makes fetch requests.
Looking forward to hearing your opinion!

Should have thought of that haha. Let me do some testing :-)
Btw do you have github sponsors or something similar?


add simple support for android

doesn't support any parameters but at least swiping works for Android

Good start


hide password when loging in

For some reason it makes me really uncomfortable to see my password when I'm not login in per facebook or google.

Good catch!
Hide password on login but keep it visible on signup? - I'm being lazy and don't want to add a password confirmation field on signup


Updated README

Thanks for your interest in palantir/tslint-react, @screendriver! Before we can accept your pull request, you need to sign our contributor license agreement - just visit https://cla.palantir.com/ and follow the instructions. Once you sign, I'll automatically update this pull request.


Use enterprise-chef-server-schema from oc_erchef

enterprise-chef-server-schema has been merged into oc_erchef.  All
cookbooks and migrations now point at the new location and the old
location is removed from the disk.

:+1:

:+1: and +:100: for removing the make dep


SeedImage: built image doesn't honor cloud-config

reported, checking current Dev branch

Works for me. Closing.


Possible anti-pattern with current weights behaviour

This issue arose from #84. The relevent text is copied, edited an collated from that PR for further discussion here @lukashergt, @appetrosyan
The code samples['weight'][i] = 0 doesn't actually do anything, since samples['weight'] is different from samples.weight, and it is the latter which is used for computing the weighted sums. This possible anti-pattern was introduced in #55 and #56. I'm not sure of the best way to ameliorate this clearly confusing state of affairs. One could:

drop the weight as a column (but this then makes it less clear that it is a weighted frame)

@lukashergt : "I don't like this solution. It is useful to see the weight column when looking at the full table."

make the weight column read-only (don't know if this is easy to do in pandas, and would likely cause no end of bother down the line)

@lukashergt : "Well, it seems like this is already the case, right? Trying to overwrite the weight column actually doesn't work..."

This is only partially true. It's worse than doing nothing, since users might think that they have done something when they haven't. True read-only would raise an exception when a user tried to write to the column.

synchronise the weight column with the weight attribute more effectively. This would require someone with an understanding of how the weighted data frames work in anesthetic/weighted_pandas.py which is currently just me at the moment (AFAIK).

It would be better practice if more than one person understood this portion of the codebase, otherwise the risk is increased that hidden bad behaviour is not being discovered.

The issue is that there is a hidden attribute in the WeightedSeries and WeightedDataFrame called _weight.
The hidden attribute is necessary for WeightedSeries, as if you want the pandas equivalent of a Series you can only have one column, so the weights have to be an attribute. This is forwarded as a readable attribute via a @property, and the correct behaviour is also enforced for slicing.
The same was originally true for WeightedDataFrame, but this was slightly counterintuitive as you would not see the weights in the same way that you see nlive or other derived NestedSamples columns. The contents of _weight is therefore copied into a new column whenever you change _weight using the anesthetic functions, but if you go and change this new column manually it has no effect on the weighted calculations (like mean, std, median etc)

synchronise the weight column with the weight attribute more effectively. This would require someone with an understanding of how the weighted data frames work in anesthetic/weighted_pandas.py which is currently just me at the moment (AFAIK).

Having looked a bit into the issue described in #95 I am wondering whether having the propery weight in WeightedSeries and WeightedDataFrame disjoint from the sample points might be complicating our lives unneccessarily...?
Each sample point comes with a corresponding weight, so having weight in the table with the parameter values and with logL etc. seems more natural to me, than keeping it as something separate.
Were there reasons why we need to make that separation, @williamjameshandley?

The reason we need the separation is you still want it to be a weighted data frame if you do say samples[['A','B']].mean(), so you can't rely on a column of weights. The real issue is that it is also very convenient to be able to see this column of weights alongside the other columns (although strictly not necessary for weighted functionality), which is why this information is duplicated, and therefore prone to not being updated in parallel.

Keeping a column of weights, that example could be rephrased to:
samples[['A', 'B']].mean(weight=samples.weight)  # weighted mean
samples[['A', 'B']].mean(weight=None)  # mean with equal weights

Where a required weight argument could enforce the weighted nature of the Series/Dataframe.
This could simplify our lives. Although, granted, this is not as nice and slim as what you suggest.

For brainstorming purposes. Maybe pandas' MultiIndex could be useful? That way we could have a secondary index which is our weights. This secondary weight index (all indices for that matter) gets displayed alongside the data, but also passed along to subsamples, slices, etc.
import numpy as np
import pandas

index = np.arange(10)
weight = np.linspace(0, 1, index.size + 1)[1:]
indices = pandas.MultiIndex.from_tuples(zip(index, weight), names=['index', 'weight'])

params = ['x0', 'x1', 'x2', 'x3', 'x4']
data = np.arange(index.size * len(params)).reshape(len(params), index.size).T + 100

df = pandas.DataFrame(data, columns=params, index=indices)

> print(df)

               x0   x1   x2   x3   x4
index weight
0     0.1     100  110  120  130  140
1     0.2     101  111  121  131  141
2     0.3     102  112  122  132  142
3     0.4     103  113  123  133  143
4     0.5     104  114  124  134  144
5     0.6     105  115  125  135  145
6     0.7     106  116  126  136  146
7     0.8     107  117  127  137  147
8     0.9     108  118  128  138  148
9     1.0     109  119  129  139  149

> print(df[['x0', 'x1']])

               x0   x1
index weight
0     0.1     100  110
1     0.2     101  111
2     0.3     102  112
3     0.4     103  113
4     0.5     104  114
5     0.6     105  115
6     0.7     106  116
7     0.8     107  117
8     0.9     108  118
9     1.0     109  119

> print(df.x0)

index  weight
0      0.1       100
1      0.2       101
2      0.3       102
3      0.4       103
4      0.5       104
5      0.6       105
6      0.7       106
7      0.8       107
8      0.9       108
9      1.0       109
Name: x0, dtype: int64

> print(df[2:5])

               x0   x1   x2   x3   x4
index weight
2     0.3     102  112  122  132  142
3     0.4     103  113  123  133  143
4     0.5     104  114  124  134  144

I strongly disagree with the first suggestion, but the second may well be truly inspired, as it would also get around a lot of the issues associated with Series and DataFrames.
The fundamental question is whether we can get samples.weight to neatly access the multi index correctly (Pretty sure this should be possible), in order to avoid breaking things in a backward incompatible fashion.

The more I think about this, the more I get a 'why didn't I think of this in the first place?' vibe, which in my book is a good sign. In the interests of full consideration though, it's worth our trying to shoot it down as much as possible before getting started.

Is it going to make slicing an array over the index hard/ugly, if ever we wanted to do that (e.g. in the live_points function)?
Is iteration + weight theoretically in keeping with an 'index'?
Will this make operations on/between arrays hard?

I think that if we were to make this change, we should consider a major version bump to 2.0.0, as it is arguably an API shift (although hopefully not if we get it right).

The other thing worth considering alongside the above points is whether we even need an iteration index, and whether one could just have weight on its own.

The other thing worth considering alongside the above points is whether we even need an iteration index, and whether one could just have weight on its own.

For compatibility with MCMC chains, where the weights are not unique, I think it is better to retain an iteration index.

The fundamental question is whether we can get samples.weight to neatly access the multi index correctly

Make samples.weight a property returning the index?
E.g. along these lines:
class WeightedDataFrame(pandas.DataFrame):

    def __init__(self, data, columns, weight):
        index = np.arange(weight.size)
        indices = pandas.MultiIndex.from_tuples(zip(index, weight), names=['index', 'weight'])
        super(WeightedDF, self).__init__(data=data, index=indices, columns=columns)

    @property
    def weight(self):
        return self.index.get_level_values('weight').to_numpy()

df = WeightedDF(data, columns=params, weight=weight)

> df.weight

array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. ])

> df.weight[0]

0.1

We'd need a way for subsamples to access that property, though.

Is it going to make slicing an array over the index hard/ugly, if ever we wanted to do that (e.g. in the live_points function)?

Not sure what you mean, as shown in my first examples df[2:5] is still possible without any complication. Is that what you mean?

Is iteration + weight theoretically in keeping with an 'index'?

In nested sampling, weights are an ordered list, so they could be almost viewed as an index of their own, right? In MCMC, weights are essentially a counter for line repetitions, which has an index quality to it too...

Will this make operations on/between arrays hard?

Using a MultiIndex definitely makes indexing a bit more complex when it comes to actually using the index (see above: self.index.get_level_values('weight').to_numpy()). However, it shouldn't be necessary to directly use the multi indexing that often, right?

For compatibility with MCMC chains, where the weights are not unique, I think it is safer to retain an iteration index.

Agreed

Not sure what you mean, as shown in my first examples df[2:5] is still possible without any complication. Is that what you mean?

Yes. Neat.

Make samples.weight a property returning the index?

Yes.

In nested sampling, weights are an ordered list, so they could be almost viewed as an index of their own, right? In MCMC, weights are essentially a counter for line repetitions, which has an index quality to it too...

I can definitely buy this reasoning (pandas has a pretty loose notion of index anyway).

Using a MultiIndex definitely makes indexing a bit more complex when it comes to actually using the index (see above: self.index.get_level_values('weight').to_numpy()). However, it shouldn't be necessary to directly use the multi indexing that often, right?

I think this makes sense, or at least not so much a concern to not give it a go.
Who wants to have a go at doing this? I have some time this afternoon to look at it, but if you'd prefer to do it let me know soon.

Who wants to have a go at doing this? I have some time this afternoon to look at it, but if you'd prefer to do it let me know soon.

You are much more familiar with pandas inheritance stuff, so you'll probably have an easier time. I'd need to do more reading on the whole _constructor business. At a quick glance, I wasn't able to figure out how to make a slice/subsample inherit a property (weight) from the main sample. You've already made this happen in the current version, so probably better if you have a go at it.

OK. The only possible remaining issue is what to do about the _rand attribute (used to consistently compress), which is treated similarly to the weights, but not user-accessible. This is not associated with this issue, but may be with the underlying issues in #115. I'll see how I get on.

OK. The only possible remaining issue is what to do about the _rand attribute.

Keep that as it is now? _rand is different from weight in that it's not really something that needs to be displayed (maybe even should not...?). Therefore there is no need of syncing a column with an attribute.

Another index? Though not really something that needs displaying maybe it doesn't hurt...? Then it can be handled alongside the weight index.

Another thing that I've been wondering is whether logL is another candidate for an index. It is not a parameter like the rest in data, rather each logL is associate with a complete parameter set...

Another thing that I've been wondering is whether logL is another candidate for an index. It is not a parameter like the rest in data, rather each logL is associate with a complete parameter set (i.e. the complete row)...

I think if we start adding too much, it will begin to get messy. I'm also not sure I agree that logL isn't of the same status as a derived parameter (it is for the purposes of computing things like the DKL).


reshape on bad data: "segment_ids[0] = -1 is out of range"

This piece of code fails in unsorted_segment_sum inside Reshape op. It works just fine for a few batches and then fails on one specific batch of data: it just so happened (bug in my code) that the length of one line of the input of rnn turned out to be zero (and length param in dynamic_rnn was also zero), and I was passing the resulting tensor to a reshape, so it gave me this error:
InvalidArgumentError: segment_ids[0] = -1 is out of range [0, 100100)
     [[Node: gradients/Reshape_grad/Reshape/tensor = UnsortedSegmentSum[T=DT_FLOAT, Tindices=DT_INT32, _device="/job:localhost/replica:0/task:0/cpu:0"](gradients/Gather_grad/Reshape/_137, gradients/Gather_grad/Reshape_1/_139, gradients/Reshape_grad/Reshape/Squeeze/_141)]]
     [[Node: gradients/Reshape_grad/Reshape/tensor/_143 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:2", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_31_gradients/Reshape_grad/Reshape/tensor", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:2"]()]]
...
  File "/home/usman/anaconda2/envs/tfpy3/lib/python3.5/site-packages/tensorflow/python/ops/gen_array_ops.py", line 1383, in reshape
    name=name)
  File "/home/usman/anaconda2/envs/tfpy3/lib/python3.5/site-packages/tensorflow/python/ops/op_def_library.py", line 455, in apply_op
    as_ref=input_arg.is_ref)
  File "/home/usman/anaconda2/envs/tfpy3/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 620, in convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/home/usman/anaconda2/envs/tfpy3/lib/python3.5/site-packages/tensorflow/python/ops/gradients.py", line 94, in _IndexedSlicesToTensor
    name=name)
  File "/home/usman/anaconda2/envs/tfpy3/lib/python3.5/site-packages/tensorflow/python/ops/gen_math_ops.py", line 2215, in unsorted_segment_sum
    num_segments=num_segments, name=name)
  File "/home/usman/anaconda2/envs/tfpy3/lib/python3.5/site-packages/tensorflow/python/ops/op_def_library.py", line 704, in apply_op
    op_def=op_def)
  File "/home/usman/anaconda2/envs/tfpy3/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2260, in create_op
    original_op=self._default_original_op, op_def=op_def)

here's the piece of code on which it fails:
def last_relevant(output, length):
    batch_size = tf.shape(output)[0]
    max_length = int(output.get_shape()[1])
    output_size = int(output.get_shape()[2])
    index = tf.range(0, batch_size) * max_length + (length - 1)
    flat = tf.reshape(output, [-1, output_size])
    relevant = tf.gather(flat, index)
    return relevant

just wondering if error message could be less obscure.

The main reason this is so confusing is that tf.gather currently doesn't check its indices when running as a GPU op: https://github.com/tensorflow/tensorflow/blob/c8b59c0/tensorflow/core/kernels/gather_op_gpu.cu.cc#L65
If you pass in [-1], as in your case, it will return a tensor containing zeros.
It then continues to run until the gradient gives you an error.
I think once tf.gather checks its inputs when running on the GPU, this issue should be considered fixed.

Automatically closing due to lack of recent activity. Since this issue is old at this point, please reopen the issue if it still occurs when tried with the latest version of Tensorflow. Thank you.


Compatibility: paket vs. Klondike NuGet server

Hello
Intro
We are using a private NuGet server (Klondike) to serve our internal (non-public) NuGet packages and mirror (public) NuGet packages from nuget.org.
Problem
paket is not able to fetch package information from Klondike (Visual Studio's integrated package manager & the nuget command line work flawless).
Analysis
I've debugged paket install and checked the HTTP requests and responses:

followODataLink expects an XML document as answer; Klondike sent JSON
getAllVersionsFromNuGet2 expects an JSON document; Klondike sent XML
..

The content type used for output is like random.
Question
Shouldn't the (web-)client always specify the desired output format?
paket.dependencies configuration
source http://klondike.internal.domain:8080/api/odata

Best regards,
Daniel

Thanks!


Environmental variable are not being set in stacks

Bug description
When deploying stack, deployment environment variables are not being set.
Expected behavior
When deploying a stack, deployment variables show in the stack's environment variable section and/or on the affected containers.
Portainer Logs
No logs were produced during the deployment of the stack in the portainer container.
Steps to reproduce the issue:

Go to 'Stacks'
Click 'Add Stack'
Give Stack Name
Click Repository
Repository URL: https://github.com/hydrian/docker-recipes
Compose Path: graylog-stack.yml
click 'add environment variable' for the following variables and appropriate variables. These variables are set up in the graylog-stack.yml file. You must set them to override them.

GRAYLOG_HTTP_EXTERNAL_URI
GRAYLOG_PASSWORD_SECRET
GRAYLOG_ROOT_PASSWORD_SHA2

Click 'Deploy the stack'... wait for Deployment
Goto 'Containers'
Click on GrayLog container to see details
Check ENV section. All added env variables are the docker-composer.yml default or not set even if you added custom variables.

Technical details:

Portainer version: 2.2.1
Docker version (managed by Portainer): 20.10.6 (API: 1.41)
Kubernetes version (managed by Portainer):
Platform (windows/linux): Linux (Linux Mint 19.4)
Command used to start Portainer: docker run -p 9000:9000 -v portainer portainer/portainer
Browser: Brave Version 1.23.71 Chromium: 90.0.4430.72 (Official Build) (64-bit)
Use Case (delete as appropriate): Using Portainer at Home
Have you reviewed our technical documentation and knowledge base? Yes

Additional context
Running a stand-alone version
Also deployed with cut&pasting the 'docker-compose.yml' file into the text area. It didn't make a difference.

This may be related to #3024

Hi @hydrian
See my examples stack below with ENV variables
version: '3.3'
services:
  db:
    image: mysql:5.7
    restart: always
    environment:
      MYSQL_DATABASE: $MYSQLDB
      MYSQL_USER: $MYSQLUSER
      MYSQL_PASSWORD: $MYSQLPASS
      MYSQL_ROOT_PASSWORD: $MYSQLROOTPASS
    ports:
      - '3306:3306'
    expose:
      - '3306'
    volumes:
      - my-db:/var/lib/mysql

volumes:
  my-db:

and here is the screenshot of my env vars

I've used $VARNAME in the compose file for each of the env vars that I wanted to use.
Hope that makes it clear.
Feel free to reach out if you still have issues.

Seeing as this hasn't been closed I'd just like to add my opinion regarding this: the current behaviour is a little confusing, as there is the need to add references to the vars in at least two places (env form + compose file), where I perhaps naively expected them to act similar to passing an env file.
For what it's worth, you can simply reference the var in the compose file like so:
   environment:
      - MY_VAR

Without the need to initialise it directly, if MY_VAR is within the stacks env config, it will be passed through.


ci: remove github action on pull request

What problem does this PR solve?
remove github action on pull request, jenkins verify ci will run on every pr
What is changed and how it works?
Check List
Tests

No code

Related changes

Need to cherry-pick to the release-4.0 / release-5.0 / release-5.1 / release-5.2

cc @zhouqiang-cl

I think you can just remove the github actions file.

I think you can just remove the github actions file.

jenkins-parser/verify only runs on pull request

I think you can just remove the github actions file.

jenkins-parser/verify only runs on pull request

Hmmm...Then things go really weird that we are in an intermediate stage. Any plan to run this set of test on commits without GitHub Actions? Or we just make sure PR passed and thus commits?

I think you can just remove the github actions file.

jenkins-parser/verify only runs on pull request

Hmmm...Then things go really weird that we are in an intermediate stage. Any plan to run this set of test on commits without GitHub Actions? Or we just make sure PR passed and thus commits?

Done! Now github action on branch master has been removed

/cc @tangenta

/merge

@purelind if you turn on verify ci on all branches, could you please cherry pick this PR to all those branches?


vm/gce: don't recreate VMs every time

A potential optimization: instead of destroying/creating VM each time, we probably could just reboot it. Should be faster.
On a similar note: we could not recreate GCE image on every boot if it's the same image (need to somehow understand that it's the same image), and just use the existing one.

Similarly, it  should be faster if boot qemu test machine once , and use a "fork" vm every time.
I noticed that  TriforceAFL has implement a forkserver.
see here https://github.com/nccgroup/TriforceAFL/blob/master/slides/ToorCon16_TriforceAFL.pdf
Is it pissible to implement it in syzkaller?

Is it pissible to implement it in syzkaller?

I guess it is possible, but qemu boot is already quite fast. So why bother?

I just think it will be faster, but don't know how big the difference will be.