cannot do distributed training

Hello,
Thanks for open sourcing!
I try to run distributed training for pretraining. Without distributed training, it works fine.
I get the below error. I tried with pytorch versions 1.7.0, 1.7.1 and 1.8.0 They get below error. Version 1.9 gets ImportError: cannot import name '_new_empty_tensor' from 'torchvision.ops' **(/work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torchvision/ops/__init__.py)** ``
I tried changing this line to losses.backward(retain_graph=True), it did not fix.
Let me know if you have any suggestions on how to address this issue.
Traceback (most recent call last):
  File "main.py", line 643, in <module>
    main(args)
  File "main.py", line 546, in main
    train_stats = train_one_epoch(
  File "/work/vcirik/mdetr/engine.py", line 100, in train_one_epoch
    losses.backward()
  File "/work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/autograd/__init__.py", line 130, in backward
    Variable._execution_engine.run_backward(
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.LongTensor [1, 10]] is at version 3; expected version 2 instead. Hint: enable anomaly detection to find the operati\
on that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

Similar error happens to me with command

$ python -m torch.distributed.launch --nproc_per_node=2 --use_env main.py --dataset_config configs/pretrain.json --ema

I enabled detect_anomaly and got the error log below.

/home/toku/mdetr/engine.py:53: UserWarning: Anomaly Detection has been enabled. This mode will increase the runtime and should only be enabled for debugging.
with torch.autograd.detect_anomaly():
[W python_anomaly_mode.cpp:104] Warning: Error detected in EmbeddingBackward. Traceback of forward call that caused the error:
File "main.py", line 643, in
main(args)
File "main.py", line 558, in main
model_ema=model_ema,
File "/home/toku/mdetr/engine.py", line 69, in train_one_epoch
memory_cache = model(samples, captions, encode_and_save=True)
File "/usr/local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 705, in forward
output = self.module(*inputs[0], **kwargs[0])
File "/usr/local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/toku/mdetr/models/mdetr.py", line 143, in forward
text_attention_mask=None,
File "/usr/local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/toku/mdetr/models/transformer.py", line 121, in forward
encoded_text = self.text_encoder(**tokenized)
File "/usr/local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.6/site-packages/transformers/models/roberta/modeling_roberta.py", line 833, in forward
past_key_values_length=past_key_values_length,
File "/usr/local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.6/site-packages/transformers/models/roberta/modeling_roberta.py", line 132, in forward
token_type_embeddings = self.token_type_embeddings(token_type_ids)
File "/usr/local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 147, in forward
self.norm_type, self.scale_grad_by_freq, self.sparse)
File "/usr/local/lib/python3.6/site-packages/torch/nn/functional.py", line 1913, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
(function _print_stack)
Traceback (most recent call last):
File "main.py", line 643, in
main(args)
File "main.py", line 558, in main
model_ema=model_ema,
File "/home/toku/mdetr/engine.py", line 101, in train_one_epoch
losses.backward()
File "/usr/local/lib/python3.6/site-packages/torch/tensor.py", line 245, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "/usr/local/lib/python3.6/site-packages/torch/autograd/init.py", line 147, in backward
allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.LongTensor [2, 14]] is at version 3; expected version 2 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

I tried python 3.8 and all torch versions(1.7.0,1.8.0 and 1.9.0) but got same error.
any help?
p.s.
@volkancirik About the error with torch==1.9.0,  #15  might help you.

After fixing 1.9.0 bug, I get a new error.
Traceback (most recent call last):   File "main.py", line 643, in <module>     main(args)   File "main.py", line 546, in main     train_stats = train_one_epoch(   File "/work/vcirik/mdetr/engine.py", line 79, in train_one_epoch     loss_dict.update(criterion(outputs, targets, positive_map))   File "/work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl     return forward_call(*input, **kwargs)   File "/work/vcirik/mdetr/models/mdetr.py", line 679, in forward     losses.update(self.get_loss(loss, outputs, targets, positive_map, indices, num_boxes))   File "/work/vcirik/mdetr/models/mdetr.py", line 655, in get_loss     return loss_map[loss](outputs, targets, positive_map, indices, num_boxes, **kwargs)   File "/work/vcirik/mdetr/models/mdetr.py", line 487, in loss_labels     eos_coef[src_idx] = 1 RuntimeError: linearIndex.numel()*sliceSize*nElemBefore == value.numel()INTERNAL ASSERT FAILED at "/opt/conda/conda-bld/pytorch_1623448278899/work/aten/src/ATen/native/cuda/Indexing.cu":253, please report a bug to PyTorch. number of flat\ tened indices did not match number of elements in the value tensor31 ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 64656) of binary: /work/vcirik/anaconda3/envs/mdetr/bin/python ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed

I tried pretraining with single GPU with the command below, and the error did not happen.

$ python main.py --dataset_config configs/pretrain.json --ema

If the problem is in-place operation, why the error did not happen with single GPU?

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.LongTensor [2, 14]] is at version 3; expected version 2 instead.

This may be the cause of this issue.
https://github.com/pytorch/pytorch/issues/22095
The fix will be
diff --git a/main.py b/main.py
--- a/main.py
+++ b/main.py
@@ -320,7 +320,8 @@ def main(args):
     model_ema = deepcopy(model) if args.ema else None
     model_without_ddp = model
     if args.distributed:
-        model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu], find_unused_parameters=True)
+        model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu], find_unused_parameters=True, broadcast_buffers=False)
         model_without_ddp = model.module
     n_parameters = sum(p.numel() for p in model.parameters() if p.requires_grad)
     print("number of params:", n_parameters)

The error disappeared, but I am not yet sure whether it will affect training process.

@toku-n , after broadcast_buffers=False fix I still get RuntimeError: linearIndex.numel()*sliceSize*nElemBefore == value.numel()INTERNAL ASSERT FAILED (screenshot below)
What's your torch version where single GPU distributed training is able to run.

Hi @volkancirik
I think torch==1.9.0 causes the error  RuntimeError: linearIndex.numel()*sliceSize*nElemBefore == value.numel()INTERNAL ASSERT FAILED
Here is my environment @alcinos
TITAN RTX
python 3.8.7
cuda 11.1.1
cudnn 8.2.1

and I installed python libs with
$ pip3 install torch==1.8.1+cu111 torchvision==0.9.1+cu111 -f https://download.pytorch.org/whl/lts/1.8/torch_lts.html
$ pip3 install -r requirements.txt

and my pip libs are
certifi==2021.5.30
chardet==4.0.0
click==8.0.1
cloudpickle==1.6.0
cycler==0.10.0
Cython==0.29.23
filelock==3.0.12
flatbuffers==2.0
huggingface-hub==0.0.12
idna==2.10
joblib==1.0.1
kiwisolver==1.3.1
matplotlib==3.4.2
numpy==1.21.0
onnx==1.9.0
onnxruntime==1.8.1
packaging==21.0
panopticapi==0.1
Pillow==8.3.1
prettytable==2.1.0
protobuf==3.17.3
pycocotools==2.0
pyparsing==2.4.7
python-dateutil==2.8.1
PyYAML==5.4.1
regex==2021.7.6
requests==2.25.1
sacremoses==0.0.45
scipy==1.7.0
six==1.16.0
submitit==1.3.3
timm==0.4.12
tokenizers==0.10.3
torch==1.8.1+cu111
torchvision==0.9.1+cu111
tqdm==4.61.2
transformers==4.8.2
typing-extensions==<IP_ADDRESS>
urllib3==1.26.6
wcwidth==0.2.5
xmltodict==0.12.0

I changed my code with  #15 and the  broadcast_buffers=False patch I shared in the comment above.
Will this help you?

@toku-n Your env looks fine.  I wouldn't go the  broadcast_buffers=False route because I can't guarantee, from the top of my head, that it will not have unwanted side-effects. And we definitely didn't require it when training with pythorch 1.8.1
My best guess is the transformers version, try downgrading to 4.5.1 as I suggested above and see if this helps.

I can run when nproc_per_node=1 but cannot run when using multiple gpus.
TITAN RTX
python 3.8.10
cuda 11.1
cudnn 8005
pytorch 1.8.0

Log
$ CUBLAS_WORKSPACE_CONFIG=:16:8 python -m torch.distributed.launch --nproc_per_node=4 --use_env main.py --dataset_config configs/pretrain.json --ema
<de=4 --use_env main.py --dataset_config configs/pretrain.json --ema
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
*****************************************
| distributed init (rank 2): env://
| distributed init (rank 1): env://| distributed init (rank 3): env://

| distributed init (rank 0): env://
git:
  sha: N/A, status: clean, branch: N/A

Namespace(GT_type='separate', aux_loss=True, backbone='resnet101', batch_size=1, bbox_loss_coef=5, ce_loss_coef=1, clevr_ann_path='', clevr_img_path='', clip_max_norm=0.1, coco_path='/work/vcirik/all_data/mscoco/', combine_datasets=['flickr'], combine_datasets_val=['flickr'], contrastive_align_loss=True, contrastive_align_loss_coef=1, contrastive_loss=False, contrastive_loss_coef=0.1, contrastive_loss_hdim=64, dataset_config='configs/pretrain.json', dec_layers=6, device='cuda', dice_loss_coef=1, dilation=False, dim_feedforward=2048, dist_backend='nccl', dist_url='env://', distributed=True, do_qa=False, dropout=0.1, ema=True, ema_decay=0.9998, enc_layers=6, eos_coef=0.1, epoch_chunks=-1, epochs=40, eval=False, eval_skip=1, flickr_ann_path='mdetr_annotations', flickr_dataset_path='/work/vcirik/flickr30k/github_flickr30k_entities/', flickr_img_path='/work/vcirik/flickr30k/images/', fraction_warmup_steps=0.01, freeze_text_encoder=False, frozen_weights=None, giou_loss_coef=2, gpu=0, gqa_ann_path='mdetr_annotations', hidden_dim=256, load='', lr=0.0001, lr_backbone=1e-05, lr_drop=35, mask_loss_coef=1, mask_model='none', masks=False, modulated_lvis_ann_path='', nheads=8, no_detection=False, num_queries=100, num_workers=5, optimizer='adam', output_dir='', pass_pos_and_query=True, phrasecut_ann_path='', phrasecut_orig_ann_path='', position_embedding='sine', pre_norm=False, predict_final=False, qa_loss_coef=1, rank=0, refexp_ann_path='mdetr_annotations', refexp_dataset_name='all', remove_difficult=False, resume='', run_name='', schedule='linear_with_warmup', seed=42, set_cost_bbox=5, set_cost_class=1, set_cost_giou=2, set_loss='hungarian', split_qa_heads=False, start_epoch=0, temperature_NCE=0.07, test=False, test_type='test', text_encoder_lr=5e-05, text_encoder_type='roberta-base', vg_ann_path='', vg_img_path='/work/vcirik/all_data/gqa/images/', weight_decay=0.0001, world_size=4)
/work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/__init__.py:421: UserWarning: torch.set_deterministic is deprecated and will be removed in a future release. Please use torch.use_deterministic_algorithms instead
  warnings.warn((
[W Context.cpp:70] Warning: torch.use_deterministic_algorithms is in beta, and its design and functionality may change in the future. (function operator())
/work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/__init__.py:421: UserWarning: torch.set_deterministic is deprecated and will be removed in a future release. Please use torch.use_deterministic_algorithms instead
  warnings.warn((
[W Context.cpp:70] Warning: torch.use_deterministic_algorithms is in beta, and its design and functionality may change in the future. (function operator())
/work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/__init__.py:421: UserWarning: torch.set_deterministic is deprecated and will be removed in a future release. Please use torch.use_deterministic_algorithms instead
  warnings.warn((
[W Context.cpp:70] Warning: torch.use_deterministic_algorithms is in beta, and its design and functionality may change in the future. (function operator())
/work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/__init__.py:421: UserWarning: torch.set_deterministic is deprecated and will be removed in a future release. Please use torch.use_deterministic_algorithms instead
  warnings.warn((
[W Context.cpp:70] Warning: torch.use_deterministic_algorithms is in beta, and its design and functionality may change in the future. (function operator())
Some weights of the model checkpoint at roberta-base were not used when initializing RobertaModel: ['lm_head.dense.bias', 'lm_head.layer_norm.bias', 'lm_head.layer_norm.weight', 'lm_head.dense.weight', 'lm_head.bias', 'lm_head.decoder.weight']
- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of the model checkpoint at roberta-base were not used when initializing RobertaModel: ['lm_head.layer_norm.weight', 'lm_head.bias', 'lm_head.decoder.weight', 'lm_head.layer_norm.bias', 'lm_head.dense.weight', 'lm_head.dense.bias']
- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of the model checkpoint at roberta-base were not used when initializing RobertaModel: ['lm_head.layer_norm.weight', 'lm_head.dense.bias', 'lm_head.dense.weight', 'lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.decoder.weight']
- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of the model checkpoint at roberta-base were not used when initializing RobertaModel: ['lm_head.layer_norm.weight', 'lm_head.bias', 'lm_head.layer_norm.bias', 'lm_head.dense.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight']
- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
number of params: 185160324
loading annotations into memory...
Done (t=16.14s)
creating index...
/work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/utils/data/dataloader.py:474: UserWarning: This DataLoader will create 5 worker processes in total. Our suggested max number of worker in current system is 1, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  warnings.warn(_create_warning_msg(
index created!
/work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/utils/data/dataloader.py:474: UserWarning: This DataLoader will create 5 worker processes in total. Our suggested max number of worker in current system is 1, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  warnings.warn(_create_warning_msg(
/work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/utils/data/dataloader.py:474: UserWarning: This DataLoader will create 5 worker processes in total. Our suggested max number of worker in current system is 1, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  warnings.warn(_create_warning_msg(
/work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/utils/data/dataloader.py:474: UserWarning: This DataLoader will create 5 worker processes in total. Our suggested max number of worker in current system is 1, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  warnings.warn(_create_warning_msg(
loading annotations into memory...
Done (t=0.49s)
creating index...
index created!
Start training
Starting epoch 0
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/THC/generic/THCTensorMath.cu line=29 error=715 : an illegal instruction was encountered
Traceback (most recent call last):
  File "main.py", line 682, in <module>
    main(args)
  File "main.py", line 582, in main
    train_stats = train_one_epoch(
  File "/work/vcirik/mdetr/engine.py", line 68, in train_one_epoch
    memory_cache = model(samples, captions, encode_and_save=True)
  File "/work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 705, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/work/vcirik/mdetr/models/mdetr.py", line 129, in forward
    features, pos = self.backbone(samples)
  File "/work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/work/vcirik/mdetr/models/backbone.py", line 169, in forward
    xs = self[0](tensor_list)
  File "/work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/work/vcirik/mdetr/models/backbone.py", line 74, in forward
    xs = self.body(tensor_list.tensors)
  File "/work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torchvision/models/_utils.py", line 63, in forward
    x = module(x)
  File "/work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/nn/modules/container.py", line 119, in forward
    input = module(input)
  File "/work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torchvision/models/resnet.py", line 128, in forward
    out = self.conv2(out)
  File "/work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 399, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 395, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: cuda runtime error (715) : an illegal instruction was encountered at /opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/THC/generic/THCTensorMath.cu:29
terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: an illegal instruction was encountered
Exception raised from create_event_internal at /opt/conda/conda-bld/pytorch_1614378083779/work/c10/cuda/CUDACachingAllocator.cpp:733 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7ff3253662f2 in /work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x5b (0x7ff32536367b in /work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0x809 (0x7ff3255bf219 in /work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10::TensorImpl::release_resources() + 0x54 (0x7ff32534e3a4 in /work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #4: <unknown function> + 0x6e0dda (0x7ff37c2c2dda in /work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0x6e0e71 (0x7ff37c2c2e71 in /work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #6: <unknown function> + 0x1932c6 (0x55c734ddf2c6 in /work/vcirik/anaconda3/envs/mdetr/bin/python)
frame #7: <unknown function> + 0x15878b (0x55c734da478b in /work/vcirik/anaconda3/envs/mdetr/bin/python)
frame #8: <unknown function> + 0xe81c4 (0x55c734d341c4 in /work/vcirik/anaconda3/envs/mdetr/bin/python)
frame #9: <unknown function> + 0x15893b (0x55c734da493b in /work/vcirik/anaconda3/envs/mdetr/bin/python)
frame #10: <unknown function> + 0x193141 (0x55c734ddf141 in /work/vcirik/anaconda3/envs/mdetr/bin/python)
frame #11: <unknown function> + 0x15878b (0x55c734da478b in /work/vcirik/anaconda3/envs/mdetr/bin/python)
frame #12: <unknown function> + 0xe81c4 (0x55c734d341c4 in /work/vcirik/anaconda3/envs/mdetr/bin/python)
frame #13: <unknown function> + 0x15893b (0x55c734da493b in /work/vcirik/anaconda3/envs/mdetr/bin/python)
frame #14: <unknown function> + 0x193141 (0x55c734ddf141 in /work/vcirik/anaconda3/envs/mdetr/bin/python)
frame #15: <unknown function> + 0x15878b (0x55c734da478b in /work/vcirik/anaconda3/envs/mdetr/bin/python)
frame #16: <unknown function> + 0xe81c4 (0x55c734d341c4 in /work/vcirik/anaconda3/envs/mdetr/bin/python)
frame #17: <unknown function> + 0x15893b (0x55c734da493b in /work/vcirik/anaconda3/envs/mdetr/bin/python)
frame #18: <unknown function> + 0x193141 (0x55c734ddf141 in /work/vcirik/anaconda3/envs/mdetr/bin/python)
frame #19: <unknown function> + 0x15878b (0x55c734da478b in /work/vcirik/anaconda3/envs/mdetr/bin/python)
frame #20: <unknown function> + 0xe81c4 (0x55c734d341c4 in /work/vcirik/anaconda3/envs/mdetr/bin/python)
frame #21: <unknown function> + 0x15893b (0x55c734da493b in /work/vcirik/anaconda3/envs/mdetr/bin/python)
frame #22: <unknown function> + 0x193141 (0x55c734ddf141 in /work/vcirik/anaconda3/envs/mdetr/bin/python)
frame #23: <unknown function> + 0x1592ac (0x55c734da52ac in /work/vcirik/anaconda3/envs/mdetr/bin/python)
frame #24: <unknown function> + 0x158e77 (0x55c734da4e77 in /work/vcirik/anaconda3/envs/mdetr/bin/python)
frame #25: <unknown function> + 0x158e60 (0x55c734da4e60 in /work/vcirik/anaconda3/envs/mdetr/bin/python)
frame #26: <unknown function> + 0x176057 (0x55c734dc2057 in /work/vcirik/anaconda3/envs/mdetr/bin/python)
frame #27: PyDict_SetItemString + 0x61 (0x55c734de33c1 in /work/vcirik/anaconda3/envs/mdetr/bin/python)
frame #28: PyImport_Cleanup + 0x9d (0x55c734e21aad in /work/vcirik/anaconda3/envs/mdetr/bin/python)
frame #29: Py_FinalizeEx + 0x79 (0x55c734e53a49 in /work/vcirik/anaconda3/envs/mdetr/bin/python)
frame #30: Py_RunMain + 0x183 (0x55c734e55893 in /work/vcirik/anaconda3/envs/mdetr/bin/python)
frame #31: Py_BytesMain + 0x39 (0x55c734e55ca9 in /work/vcirik/anaconda3/envs/mdetr/bin/python)
frame #32: __libc_start_main + 0xf5 (0x7ff3c0c50c05 in /lib64/libc.so.6)
frame #33: <unknown function> + 0x1e21c7 (0x55c734e2e1c7 in /work/vcirik/anaconda3/envs/mdetr/bin/python)

Killing subprocess 17070
Killing subprocess 17071
Killing subprocess 17072
Killing subprocess 17073
Traceback (most recent call last):
  File "/work/vcirik/anaconda3/envs/mdetr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/work/vcirik/anaconda3/envs/mdetr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/distributed/launch.py", line 340, in <module>
    main()
  File "/work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/distributed/launch.py", line 326, in main
    sigkill_handler(signal.SIGTERM, None)  # not coming back
  File "/work/vcirik/anaconda3/envs/mdetr/lib/python3.8/site-packages/torch/distributed/launch.py", line 301, in sigkill_handler
    raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)
subprocess.CalledProcessError: Command '['/work/vcirik/anaconda3/envs/mdetr/bin/python', '-u', 'main.py', '--dataset_config', 'configs/pretrain.json', '--ema']' died with <Signals.SIGABRT: 6>.

Hi @alcinos,
The error has gone with transformers==4.5.1 !
Now I don't need the patch of broadcast_buffers=False.
Thank you so much !

@alcinos, thank you for your suggestion, but the error seems to have remained... I ran finetuning on the "all" split and got:
��G�error�X6Traceback (most recent call last):
  File "/home/pchelintsev/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/submitit/core/submission.py", line 53, in process_job
    result = delayed.result()
  File "/home/pchelintsev/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/submitit/core/utils.py", line 128, in result
    self._result = self.function(*self.args, **self.kwargs)
  File "run_with_submitit.py", line 98, in __call__
    detection.main(self.args)
  File "/home/pchelintsev/MDETR/mdetr/main.py", line 546, in main
    train_stats = train_one_epoch(
  File "/home/pchelintsev/MDETR/mdetr/engine.py", line 73, in train_one_epoch
    loss_dict.update(criterion(outputs, targets, positive_map))
  File "/home/pchelintsev/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/pchelintsev/MDETR/mdetr/models/mdetr.py", line 679, in forward
    losses.update(self.get_loss(loss, outputs, targets, positive_map, indices, num_boxes))
  File "/home/pchelintsev/MDETR/mdetr/models/mdetr.py", line 655, in get_loss
    return loss_map[loss](outputs, targets, positive_map, indices, num_boxes, **kwargs)
  File "/home/pchelintsev/MDETR/mdetr/models/mdetr.py", line 487, in loss_labels
    eos_coef[src_idx] = 1
RuntimeError: linearIndex.numel()*sliceSize*nElemBefore == value.numel()INTERNAL ASSERT FAILED at "/pytorch/aten/src/ATen/native/cuda/Indexing.cu":253, please report a bug to PyTorch. number of flattened indices did not match number of elements in the value tensor51

in 10562_0_result.pkl file in experiments/ directory.

@TopCoder2K This error is pretty uninformative. Please try debugging on cpu first (--device cpu) to see if you get more information.

@alcinos, thank you for the advice! I got the following output in job_number_0_log.err:
/home/pchelintsev/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/torch/__init__.py:471: UserWarning: torch.set_deterministic is deprecated and will be removed in a future release. Please use torch.use_deterministic_algorithms instead
  warnings.warn((
/home/pchelintsev/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /pytorch/c10/core/TensorImpl.h:1156.)
  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
/home/pchelintsev/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/torch/_tensor.py:575: UserWarning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (Triggered internally at  /pytorch/aten/src/ATen/native/BinaryOps.cpp:467.)
  return torch.floor_divide(self, other)
submitit ERROR (2021-09-18 16:41:20,879) - Submitted job triggered an exception
Traceback (most recent call last):
  File "/home/pchelintsev/anaconda3/envs/mdetr_env/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/pchelintsev/anaconda3/envs/mdetr_env/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/pchelintsev/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/submitit/core/_submit.py", line 11, in <module>
    submitit_main()
  File "/home/pchelintsev/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/submitit/core/submission.py", line 71, in submitit_main
    process_job(args.folder)
  File "/home/pchelintsev/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/submitit/core/submission.py", line 64, in process_job
    raise error
  File "/home/pchelintsev/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/submitit/core/submission.py", line 53, in process_job
    result = delayed.result()
  File "/home/pchelintsev/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/submitit/core/utils.py", line 128, in result
    self._result = self.function(*self.args, **self.kwargs)
  File "run_with_submitit.py", line 98, in __call__
    detection.main(self.args)
  File "/home/pchelintsev/MDETR/mdetr/main.py", line 546, in main
    train_stats = train_one_epoch(
  File "/home/pchelintsev/MDETR/mdetr/engine.py", line 69, in train_one_epoch
    outputs = model(samples, captions, encode_and_save=False, memory_cache=memory_cache)
  File "/home/pchelintsev/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/pchelintsev/MDETR/mdetr/models/mdetr.py", line 154, in forward
    hs = self.transformer(
  File "/home/pchelintsev/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/pchelintsev/MDETR/mdetr/models/transformer.py", line 168, in forward
    hs = self.decoder(
  File "/home/pchelintsev/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/pchelintsev/MDETR/mdetr/models/transformer.py", line 232, in forward
    output = layer(
  File "/home/pchelintsev/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/pchelintsev/MDETR/mdetr/models/transformer.py", line 448, in forward
    return self.forward_post(
  File "/home/pchelintsev/MDETR/mdetr/models/transformer.py", line 383, in forward_post
    tgt2 = self.cross_attn_image(
  File "/home/pchelintsev/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/pchelintsev/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/torch/nn/modules/activation.py", line 1031, in forward
    attn_output, attn_output_weights = F.multi_head_attention_forward(
  File "/home/pchelintsev/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/torch/nn/functional.py", line 5082, in multi_head_attention_forward
    attn_output, attn_output_weights = _scaled_dot_product_attention(q, k, v, attn_mask, dropout_p)
  File "/home/pchelintsev/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/torch/nn/functional.py", line 4830, in _scaled_dot_product_attention
    attn = dropout(attn, p=dropout_p)
  File "/home/pchelintsev/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/torch/nn/functional.py", line 1168, in dropout
    return _VF.dropout_(input, p, training) if inplace else _VF.dropout(input, p, training)
  File "/home/pchelintsev/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 28712) is killed by signal: Killed.

What can be wrong with the dataloader?

The worker may have been called because of the host is out of ram.
I’d suggest running locally first:
python main.py --dataset_config configs/gqa.json --ema --epochs 125 --epoch_chunks 25 --do_qa --split_qa_heads --lr_drop 150 --load pretrained_resnet101_checkpoint.pth  --batch_size 4 --no_aux_loss --qa_loss_coef 25 --lr 1.4e-4 --lr_backbone 1.4e-5 --text_encoder_lr 7e-5 --device cpu --num_workers 0

The worker may have been called because of the host is out of ram.
I’d suggest running locally first:
python main.py --dataset_config configs/gqa.json --ema --epochs 125 --epoch_chunks 25 --do_qa --split_qa_heads --lr_drop 150 --load pretrained_resnet101_checkpoint.pth  --batch_size 4 --no_aux_loss --qa_loss_coef 25 --lr 1.4e-4 --lr_backbone 1.4e-5 --text_encoder_lr 7e-5 --device cpu --num_workers 0

Hmmm, I also encountered on that advice in pytorch issues, so I checked the RAM usage with free -m during the job was running and found out that 60 of 64 Gbs were used. Suspiciously...
I also tried to run the command with --num_workers 0 as you suggested, here the full output:
Not using distributed mode
git:
  sha: dda257d51a9944ee3e4201e7e52e50e5f9faec60, status: has uncommited changes, branch: main

Namespace(aux_loss=False, backbone='resnet101', batch_size=4, bbox_loss_coef=5, ce_loss_coef=1, clevr_ann_path='', clevr_img_path='', clip_max_norm=0.1, coco_path='', combine_datasets=['gqa'], combine_datasets_val=['gqa'], contrastive_align_loss=True, contrastive_align_loss_coef=1, contrastive_loss=False, contrastive_loss_coef=0.1, contrastive_loss_hdim=64, dataset_config='configs/gqa.json', dec_layers=6, device='cpu', dice_loss_coef=1, dilation=False, dim_feedforward=2048, dist_url='env://', distributed=False, do_qa=True, dropout=0.1, ema=True, ema_decay=0.9998, enc_layers=6, eos_coef=0.1, epoch_chunks=25, epochs=125, eval=False, eval_skip=1, fraction_warmup_steps=0.01, freeze_text_encoder=False, frozen_weights=None, giou_loss_coef=2, gqa_ann_path='mdetr_annotations/', gqa_split_type='all', hidden_dim=256, load='pretrained_resnet101_checkpoint.pth', lr=0.00014, lr_backbone=1.4e-05, lr_drop=150, mask_loss_coef=1, mask_model='none', masks=False, modulated_lvis_ann_path='', nheads=8, no_detection=False, num_queries=100, num_workers=0, optimizer='adam', output_dir='', pass_pos_and_query=True, phrasecut_ann_path='', phrasecut_orig_ann_path='', position_embedding='sine', pre_norm=False, predict_final=False, qa_loss_coef=25.0, remove_difficult=False, resume='', run_name='', schedule='linear_with_warmup', seed=42, set_cost_bbox=5, set_cost_class=1, set_cost_giou=2, set_loss='hungarian', split_qa_heads=True, start_epoch=0, temperature_NCE=0.07, test=False, test_type='test', text_encoder_lr=7e-05, text_encoder_type='roberta-base', vg_ann_path='', vg_img_path='images/', weight_decay=0.0001, world_size=1)
/home/pchelintsev/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/torch/__init__.py:471: UserWarning: torch.set_deterministic is deprecated and will be removed in a future release. Please use torch.use_deterministic_algorithms instead
  warnings.warn((

number of params: 185879918
loading annotations into memory...
^[[A^[[BDone (t=233.60s)
creating index...
index created!
loading annotations into memory...
Done (t=45.16s)
creating index...
index created!
Splitting the training set into {args.epoch_chunks} of size approximately  652688
loading annotations into memory...
Done (t=0.45s)
creating index...
index created!
loading from pretrained_resnet101_checkpoint.pth
Start training
Starting epoch 0, sub_epoch 0
/home/pchelintsev/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /pytorch/c10/core/TensorImpl.h:1156.)
  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
/home/pchelintsev/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/torch/_tensor.py:575: UserWarning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').

Hmmmmm, I really confused... It seems I have different mistakes simulteneously, that's why I see different output every time...
I've just rerun the command python main.py --dataset_config configs/gqa.json --ema --epochs 125 --epoch_chunks 25 --do_qa --split_qa_heads --lr_drop 150 --load pretrained_resnet101_checkpoint.pth  --batch_size 4 --no_aux_loss --qa_loss_coef 25 --lr 1.4e-4 --lr_backbone 1.4e-5 --text_encoder_lr 7e-5 --device cpu --num_workers 0 and got
Not using distributed mode
git:
  sha: dda257d51a9944ee3e4201e7e52e50e5f9faec60, status: has uncommited changes, branch: main

Namespace(aux_loss=False, backbone='resnet101', batch_size=4, bbox_loss_coef=5, ce_loss_coef=1, clevr_ann_path='', clevr_img_path='', clip_max_norm=0.1, coco_path='', combine_datasets=['gqa'], combine_datasets_val=['gqa'], contrastive_align_loss=True, contrastive_align_loss_coef=1, contrastive_loss=False, contrastive_loss_coef=0.1, contrastive_loss_hdim=64, dataset_config='configs/gqa.json', dec_layers=6, device='cpu', dice_loss_coef=1, dilation=False, dim_feedforward=2048, dist_url='env://', distributed=False, do_qa=True, dropout=0.1, ema=True, ema_decay=0.9998, enc_layers=6, eos_coef=0.1, epoch_chunks=25, epochs=125, eval=False, eval_skip=1, fraction_warmup_steps=0.01, freeze_text_encoder=False, frozen_weights=None, giou_loss_coef=2, gqa_ann_path='mdetr_annotations/', gqa_split_type='all', hidden_dim=256, load='pretrained_resnet101_checkpoint.pth', lr=0.00014, lr_backbone=1.4e-05, lr_drop=150, mask_loss_coef=1, mask_model='none', masks=False, modulated_lvis_ann_path='', nheads=8, no_detection=False, num_queries=100, num_workers=0, optimizer='adam', output_dir='', pass_pos_and_query=True, phrasecut_ann_path='', phrasecut_orig_ann_path='', position_embedding='sine', pre_norm=False, predict_final=False, qa_loss_coef=25.0, remove_difficult=False, resume='', run_name='', schedule='linear_with_warmup', seed=42, set_cost_bbox=5, set_cost_class=1, set_cost_giou=2, set_loss='hungarian', split_qa_heads=True, start_epoch=0, temperature_NCE=0.07, test=False, test_type='test', text_encoder_lr=7e-05, text_encoder_type='roberta-base', vg_ann_path='', vg_img_path='images/', weight_decay=0.0001, world_size=1)
/home/pchelintsev/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/torch/__init__.py:471: UserWarning: torch.set_deterministic is deprecated and will be removed in a future release. Please use torch.use_deterministic_algorithms instead
  warnings.warn((
number of params: 185879918
loading annotations into memory...
Done (t=216.72s)
creating index...
index created!
loading annotations into memory...
Done (t=42.99s)
creating index...
index created!
Splitting the training set into {args.epoch_chunks} of size approximately  652688
loading annotations into memory...
Done (t=0.44s)
creating index...
index created!
loading from pretrained_resnet101_checkpoint.pth
Start training
Starting epoch 0, sub_epoch 0
/home/pchelintsev/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /pytorch/c10/core/TensorImpl.h:1156.)
  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
/home/pchelintsev/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/torch/_tensor.py:575: UserWarning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').  DefaultCPUAllocator: can't allocate memory: you tried to allocate 244470272 bytes. Error code 12 (Cannot allocate memory)

And indeed the memory ran out (I saw it with free -m), so it's not enough 64 Gb. But why do strange mistakes attack me earlier than Epoch: [0]  [    40/163172] when I use GPU?

I also tried to run evaluation on CLEVR, which is much smaller, using python main.py --batch_size 64 --dataset_config configs/clevr.json --num_queries 25 --text_encoder_type distilroberta-base --backbone resnet18  --resume https://zenodo.org/record/4721981/files/clevr_checkpoint.pth  --eval and again got RuntimeError: linearIndex.numel()*sliceSize*nElemBefore == value.numel()INTERNAL ASSERT FAILED at "/pytorch/aten/src/ATen/native/cuda/Indexing.cu":253, please report a bug to PyTorch. number of flattened indices did not match number of elements in the value tensor1421.
But I have 64 Gb RAM and 32 Gb GPU, so the whole dataset and the models can be easily allocated. It seems this is not a bug with resources... And then I run on CPU, it seems to work, consuming only about 11 Gb RAM.

Try setting this line to false:
https://github.com/ashkamath/mdetr/blob/0b747b99e2995c3c429f1391cb8e6104eaec7f21/main.py#L309
Also, could you paste the output of python -m torch.utils.collect_env ?

@alcinos, thank you for your support!

I changed the line on torch.set_deterministic(False) and the evaluation on CLEVR went successfully!

Accumulating evaluation results...
DONE (t=106.84s).
IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.828
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.990
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.988
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.718
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.830
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.907
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.373
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.872
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.872
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.786
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.874
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.934
{'test_clevr_loss': 43.043012584842515, 'test_clevr_loss_ce': 5.731913995213883, 'test_clevr_loss_bbox': 0.05494903914992771, 'test_clevr_loss_giou': 0.18080522204549035, 'test_clevr_loss_contrastive_align': 0.739596603851473, 'test_clevr_loss_ce_0': 6.165245272397181, 'test_clevr_loss_bbox_0': 0.06370836580744324, 'test_clevr_loss_giou_0': 0.21175039811345905, 'test_clevr_loss_contrastive_align_0': 1.6258955893866438, 'test_clevr_loss_ce_1': 5.9863008091881005, 'test_clevr_loss_bbox_1': 0.06160141016855683, 'test_clevr_loss_giou_1': 0.20656590048007592, 'test_clevr_loss_contrastive_align_1': 1.2556593710804556, 'test_clevr_loss_ce_2': 5.867624813379281, 'test_clevr_loss_bbox_2': 0.059281312062726084, 'test_clevr_loss_giou_2': 0.19940767474091095, 'test_clevr_loss_contrastive_align_2': 1.0192198974077205, 'test_clevr_loss_ce_3': 5.771324536906167, 'test_clevr_loss_bbox_3': 0.05730514128536698, 'test_clevr_loss_giou_3': 0.194069205697486, 'test_clevr_loss_contrastive_align_3': 0.817971428532039, 'test_clevr_loss_ce_4': 5.741090293833827, 'test_clevr_loss_bbox_4': 0.05582752994519147, 'test_clevr_loss_giou_4': 0.18599585877292799, 'test_clevr_loss_contrastive_align_4': 0.7558641342681423, 'test_clevr_loss_answer_type': 3.604563896833151e-06, 'test_clevr_loss_answer_binary': 0.00632767150950992, 'test_clevr_loss_answer_reg': 0.025332900922053852, 'test_clevr_loss_answer_attr': 0.0023749297052867175, 'test_clevr_loss_ce_unscaled': 5.731913995213883, 'test_clevr_loss_bbox_unscaled': 0.010989807827442681, 'test_clevr_loss_giou_unscaled': 0.09040261102274517, 'test_clevr_cardinality_error_unscaled': 0.01639825085324232, 'test_clevr_loss_contrastive_align_unscaled': 0.739596603851473, 'test_clevr_loss_ce_0_unscaled': 6.165245272397181, 'test_clevr_loss_bbox_0_unscaled': 0.012741673153303818, 'test_clevr_loss_giou_0_unscaled': 0.10587519905672953, 'test_clevr_cardinality_error_0_unscaled': 1.9568324677976732, 'test_clevr_loss_contrastive_align_0_unscaled': 1.6258955893866438, 'test_clevr_loss_ce_1_unscaled': 5.9863008091881005, 'test_clevr_loss_bbox_1_unscaled': 0.012320282033075652, 'test_clevr_loss_giou_1_unscaled': 0.10328295024003796, 'test_clevr_cardinality_error_1_unscaled': 1.115309238815267, 'test_clevr_loss_contrastive_align_1_unscaled': 1.2556593710804556, 'test_clevr_loss_ce_2_unscaled': 5.867624813379281, 'test_clevr_loss_bbox_2_unscaled': 0.011856262410399679, 'test_clevr_loss_giou_2_unscaled': 0.09970383737045548, 'test_clevr_cardinality_error_2_unscaled': 0.5807346361705162, 'test_clevr_loss_contrastive_align_2_unscaled': 1.0192198974077205, 'test_clevr_loss_ce_3_unscaled': 5.771324536906167, 'test_clevr_loss_bbox_3_unscaled': 0.01146102826025197, 'test_clevr_loss_giou_3_unscaled': 0.097034602848743, 'test_clevr_cardinality_error_3_unscaled': 0.1629644974624363, 'test_clevr_loss_contrastive_align_3_unscaled': 0.817971428532039, 'test_clevr_loss_ce_4_unscaled': 5.741090293833827, 'test_clevr_loss_bbox_4_unscaled': 0.01116550598355525, 'test_clevr_loss_giou_4_unscaled': 0.09299792938646399, 'test_clevr_cardinality_error_4_unscaled': 0.04800581955033078, 'test_clevr_loss_contrastive_align_4_unscaled': 0.7558641342681423, 'test_clevr_loss_answer_type_unscaled': 3.604563896833151e-06, 'test_clevr_accuracy_answer_type_unscaled': 1.0, 'test_clevr_loss_answer_binary_unscaled': 0.00632767150950992, 'test_clevr_accuracy_answer_binary_unscaled': 0.9983339897855964, 'test_clevr_loss_answer_reg_unscaled': 0.025332900922053852, 'test_clevr_accuracy_answer_reg_unscaled': 0.9925868332182588, 'test_clevr_loss_answer_attr_unscaled': 0.0023749297052867175, 'test_clevr_accuracy_answer_attr_unscaled': 0.9996434087283376, 'test_clevr_accuracy_answer_total_unscaled': 0.9974136092150171, 'test_clevr_coco_eval_bbox': [0.8280577103267962, 0.9900612713060782, 0.9877238191805104, 0.7180399374430324, 0.8298110005371897, 0.9071876701127315, 0.37329648287548844, 0.8716828824051552, 0.8719843337521758, 0.7862159823816266, 0.8740682349128326, 0.9336712955657251], 'n_parameters': 111200939}

Here is the output of python -m torch.utils.collect_env:

Collecting environment information...
PyTorch version: 1.9.0+cu102
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.27

Python version: 3.8 (64-bit runtime)
Python platform: Linux-4.15.0-156-generic-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: 9.1.85
GPU models and configuration: GPU 0: Tesla V100-PCIE-32GB
Nvidia driver version: 470.63.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.20.3
[pip3] torch==1.9.0
[pip3] torchvision==0.10.0
[conda] blas                      1.0                         mkl
[conda] mkl                       2021.3.0           h06a4308_520
[conda] mkl-service               2.4.0            py38h7f8727e_0
[conda] mkl_fft                   1.3.0            py38h42c9631_2
[conda] mkl_random                1.2.2            py38h51133e4_0
[conda] numpy                     1.20.3           py38hf144106_0
[conda] numpy-base                1.20.3           py38h74d4b33_0
[conda] torch                     1.9.0                    pypi_0    pypi
[conda] torchvision               0.10.0                   pypi_0    pypi

I also try VQA2 again a bit later but this gave me hope! Thank you :)


Added dataflows within GEH

Description
This adds a model showing what data is needed in various domains, and what data each domain publishes.
This does not indicate which events publish what data, and will be described separately in an .md accompanying this picture once it has been defined.

I thought we should use the model from ENTSO-E? I.e. no mRID's...

@MartinFHansen Was this an option for us in the integration events (ie. illustrated here), or just internally in the domains?

I think this can be useful for developers without deep business knowledge 👍
I'm a bit unsure about the publishing. What does it mean that aggregations domain publishes metering point ID (which btw. market points domain needs - but surely not from the aggregations domain)?
Also perhaps needs could be grouped by which domain the individual data comes from.

Yes, good idea with the grouping, easier than having to look for which domain publishes it. "Publishing" basically means it is data which is output as events from that specific domain. I have not enumerated which events publish what, as I believe this can be done as a team activity in a design session. Alternatively I can make supporting documentation encompassing all events per domain, but that's gonna take some time :)

"Publishing" basically means it is data which is output as events from that specific domain. I have not enumerated which events publish what, as I believe this can be done as a team activity in a design session. Alternatively I can make supporting documentation encompassing all events per domain, but that's gonna take some time :)

I'm not sure it' worth creating supporting documentation. Regarding published data I would limit to the data owned by the domain. E.g. aggregations publishing metering point IDs is probably more confusing than helping. I know that in reality a lot more data might be actually published in the events eventually. But that will primarily be for convenience or e.g. identifiers to identify to which entities the published data belongs.

"Publishing" basically means it is data which is output as events from that specific domain. I have not enumerated which events publish what, as I believe this can be done as a team activity in a design session. Alternatively I can make supporting documentation encompassing all events per domain, but that's gonna take some time :)

I'm not sure it' worth creating supporting documentation. Regarding published data I would limit to the data owned by the domain. E.g. aggregations publishing metering point IDs is probably more confusing than helping. I know that in reality a lot more data might be actually published in the events eventually. But that will primarily be for convenience or e.g. identifiers to identify to which entities the published data belongs.

It needs to publish MP ID so that timeseries knows which MP the data pertains to. Otherwise it's just random data which you can't place

It is limited to data owned by the domain. It needs to publish MP ID so that timeseries knows which MP the data pertains to. Otherwise it's just random data which you can't place

Exactly. So there is a clear distinction between published owned data on published "necessary" identifiers (or other supporting data). It would be nice if that could be made clear in the diagram :-)

Why is this diagram needed?
What gap is it supposed to fill?
Could it be done another way?
In my opinion it would make more sense that each sub-domain documented what data/events they could provide. This should be done in a consistent way across all sub-domains.
Second - each team needs to known what the other domains can offer. It is OK that a team does not know everything, but then it needs to reach out to the other teams and ask for guidance / assistance. Through this dialog the contracts should be defined and the dependencies documented.
Another concern is - who is responsible for keeping this document updated after this pull request?

Why is this diagram needed?
What gap is it supposed to fill?
Could it be done another way?
In my opinion it would make more sense that each sub-domain documented what data/events they could provide. This should be done in a consistent way across all sub-domains.
Second - each team needs to known what the other domains can offer. It is OK that a team does not know everything, but then it needs to reach out to the other teams and ask for guidance / assistance. Through this dialog the contracts should be defined and the dependencies documented.
Another concern is - who is responsible for keeping this document updated after this pull request?

Currently there is confusion in regards to what data is offered from where, and which domain needs what data. This is to alleviate that issue, and is only appropriate to be in the core repo. Completely agree that each subdomain should document what data they can provide, and we should go in this direction as we start defining the contracts between domains. It is important that each contract/event is well documented once defined (and updated when changed).
This also offers a quick overview for "outsiders". As it is open source, it is all our responsibility to update any and all diagrams when we spot inconsistencies.

Currently there is confusion in regards to what data is offered from where, and which domain needs what data. This is to alleviate that issue, and is only appropriate to be in the core repo. Completely agree that each subdomain should document what data they can provide, and we should go in this direction as we start defining the contracts between domains. It is important that each contract/event is well documented once defined (and updated when changed).

If there is confusion as to how the domains interact then I don't think that adding another diagram is the right way to go. I firmly believe that this information should be provided through clear communication. The end goal is that every one knows the concepts of the architecture that has been chosen.

Besides above we already have simple diagrams documenting the context streams. This is an elaboration on that meant to replace the initial diagrams, which didn't actually document what data flowed between the domains, only the directions.

If we already have started documenting the context streams, why not extend those with the details in this diagram?

Currently there is confusion in regards to what data is offered from where, and which domain needs what data. This is to alleviate that issue, and is only appropriate to be in the core repo. Completely agree that each subdomain should document what data they can provide, and we should go in this direction as we start defining the contracts between domains. It is important that each contract/event is well documented once defined (and updated when changed).

If there is confusion as to how the domains interact then I don't think that adding another diagram is the right way to go. I firmly believe that this information should be provided through clear communication. The end goal is that every one knows the concepts of the architecture that has been chosen.

Besides above we already have simple diagrams documenting the context streams. This is an elaboration on that meant to replace the initial diagrams, which didn't actually document what data flowed between the domains, only the directions.

If we already have started documenting the context streams, why not extend those with the details in this diagram?

I believe this contributes to clear communication. The existing diagrams need to be completely reworked. This is that rework.
If this is not something that is beneficial to the project we can remove it, it's just my suggestion to solving one of the deeper issues we have :)

As I see it the issue is not that information is not available - it is that people/teams are not aware of it.

We need to ensure that the teams known where to look.
Align on the understanding of the architecture.
Have an understanding of who can help them, and who to reach out to, when they need certain data.

It is more a culture change we need in my opinion.


Fix GET/POST errors when using 1811

We updated Spartacus to work with cmsoccaddon for 1905, which uses GET instead of POST for getting cms information.
We found one issues when using 1811, after May 24 release:
405 errors returned when GET is used, and this results in severa "unknown errors" displaying in the frontend.
"TypeError: You provided 'undefined' where a stream was expected. You can provide an Observable, Promise, Array, or Iterable.
...."
We have dev6 and dev7 we can use to test 1811.
https://wiki.hybris.com/display/acc/ROT+DataCenter+VMs

PR: GH-2709

For testing:

point backend to https://dev-com-7.accdemo.b2c.ydev.hybris.com:9002
make electronics as the default base site
verify in the network tab that GET fails and POST returns 200 OK
No global error message showing

Will be fixed and retested with GH-2746


Update page weights in API Access Control reference section

contributes to https://github.com/kubernetes/website/issues/35093
@reylejano @natalisucks @bradtopol
Updating section to have distinct page weight. (these do not appear to be auto-generated pages in the reference section)

/retitle Update page weights in API Access Control reference section

/lgtm
/approve

/kind cleanup
/lgtm

Hi @a-mccarthy, looks like the deploy preview failed. Sometimes making an empty commit will kick Netlify and make it retry.


Added Dockerfile to individual components

This PR adds the following make targets:

docker -- to build the Docker images locally, based on the newly created Dockerfiles. It does not build docker images for the existing images, as those are handled by other tasks (all-in-one and crossdock)
docker-push -- to push the images built by the docker target to a repository. This requires a previous docker login.

The Travis changes were made kind of in a blind mode, as I'm not sure how I can test it. The bash parts were tested as much as I could.
I'm also not quite sure how I can remove the submodule updates to this commit, as I never really dealt with submodules. Any hints there is highly appreciated :)

Coverage remained the same at 100.0% when pulling fc807c5d956dbde9c4e0ee42dbfb01ba95019b29 on jpkrohling:JPK-Dockerfiles into baa60f654d03bfa0da5f26c94b0cf21f515678a9 on uber:master.

Coverage remained the same at 100.0% when pulling a55c7bfb30dba3045fd2113a7ad94d29fc915a35 on jpkrohling:JPK-Dockerfiles into 49dbd3c2820c71003d386c5bd9bbc49ca0b6151b on uber:master.

@yurishkuro , @black-adder , I just updated this PR, rebasing it on the latest master and adding a couple of fixes based on the comment reviews. If this looks good, I'll squash it and it would be ready for merging.
There are still a couple of items open for Cassandra, mostly related to tuning it to be more resilient and reliable (replication factor, schema creation, ...) , but I'd like to have that as another follow up task, instead of blocking this PR. I'll then ask @jsanda for help on that task, as he's far more experienced with Cassandra than I am :-)

Coverage remained the same at 100.0% when pulling 972b25b4a9bc6a5fc060be7b02634bef6f6e34d7 on jpkrohling:JPK-Dockerfiles into 437ec418e597d21b49122fb388728e75a524f1f9 on uber:master.

I am not clear why the build failed, no output in the Travis logs.
By default the new images are published to jaegertracing/jaeger-{component}, I assume we need to create repos on Docker hub?
The images are missing any sort of tags, usually we at least add -t $REPO:$COMMIT
Can we exclude changes to jaeger-ui from this PR?

You will certainly run into problems by just pinging the the pod. Cassandra start up times will vary, particularly when you take commit log replay into consideration. That can slow down start up significantly. I suggest using nodetool statusbinary.  You can grep the output for running.
To avoid overly long start up times and to avoid commit log corruption, I strongly recommend running nodetool drain on shutdown.

Coverage remained the same at 100.0% when pulling 99238f77e055a54a23b084f26de4c2813e62c9c5 on jpkrohling:JPK-Dockerfiles into 7a965e984fac3e7beb4684913a48c1affd76024b on uber:master.

Coverage remained the same at 100.0% when pulling 99238f77e055a54a23b084f26de4c2813e62c9c5 on jpkrohling:JPK-Dockerfiles into 7a965e984fac3e7beb4684913a48c1affd76024b on uber:master.

Coverage remained the same at 100.0% when pulling b122fce8d4f522fcbbe5b317adcfb51447610c79 on jpkrohling:JPK-Dockerfiles into 7a965e984fac3e7beb4684913a48c1affd76024b on uber:master.

Coverage remained the same at 100.0% when pulling 56eb6f59ab5ed142dadcd220ce117e64c943687e on jpkrohling:JPK-Dockerfiles into 7a965e984fac3e7beb4684913a48c1affd76024b on uber:master.

Coverage remained the same at 100.0% when pulling fee8371061078f445107144bf86c15dd93234792 on jpkrohling:JPK-Dockerfiles into 7a965e984fac3e7beb4684913a48c1affd76024b on uber:master.

I updated the PR to use another command for checking Cassandra's state. I decided not to use nodetool, as it would require exposing the JMX port just for this purpose. If we do need JMX in the future, that's fine, but it sounded too much for this specific purpose.

I am not clear why the build failed, no output in the Travis logs.

It works now. I had to add a dedicated conditional on .travis.yml/install for $DOCKER, instead of using a || on the $ALL_IN_ONE part.

By default the new images are published to jaegertracing/jaeger-{component}, I assume we need to create repos on Docker hub?

They are created on demand on the first build, provided the account has push permissions on the organization.

The images are missing any sort of tags, usually we at least add -t $REPO:$COMMIT

It's actually done on the upload-to-docker.sh script, called at the end of the script. It's the same script the other ones execute.

Can we exclude changes to jaeger-ui from this PR?

I tried every git trick I know, but the only thing that helped was to do a diff against master, create a new branch off of master and apply the new diff onto this new branch :/
I also opened #175 to track the points to improve in our usage of Cassandra.

Coverage remained the same at 100.0% when pulling 2abe6ef9f6e547d16eac73223c830fd562bea92e on jpkrohling:JPK-Dockerfiles into 7a965e984fac3e7beb4684913a48c1affd76024b on uber:master.

Rebased with the latest master. This is ready for a re-review or eventual merge.

Coverage remained the same at 100.0% when pulling 585a670ffb50c8d8d7f267bfd6caf8a317518bca on jpkrohling:JPK-Dockerfiles into 894cf6e925bc8aa6e822255dbf0552866b19567d on uber:master.


Fix the display format for file sizes

Description

Replace humansize with a function format_file_size in src/directory_listing.rs
Related Issue

https://github.com/static-web-server/static-web-server/issues/370
How Has This Been Tested?

Add some test cases in src/directory_listing.rs
Compared and tested with local files

Looking good, and I do feel sorry about all the nitpicking.

It's all part of helping me improve, and I appreciate your attention to detail.

@miroim FYI, there was an issue with the code that clippy complained about (you have to run it with --tests command line flag). I fix it in #382.


[Bug]: RuntimeError: Cannot return the results in a contigious 2D array. Probably ef or M is too small

What happened?
I have the following code to initialize the db client and get the collections:
def initialize_vectordb(client_path,collections_name):
    vectordb_client = chromadb.PersistentClient(path=client_path)
    collections = vectordb_client.get_or_create_collection(name=collections_name)
    return vectordb_client,collections

and here is the code to add documents to my db:
collections.add(
        documents=docs,
        embeddings=embeddings,
        metadatas=metadatas,
        ids=ids
    )

and following code to query the db:
column_len = len(collections.get(where={"source":"column"})["ids"])
file_len = len(collections.get(where={"source":"file"})["ids"])
text_embeddings = embeddings_model(text)
file_results = collections.query(
    query_embeddings=text_embeddings,
    n_results=file_len,
    where={"source":"file"}
)

Everytime when I perform initialization of db, adding documents to db followed by query in the same session, it works perfectly fine. But when I terminate the session, then initialize the db followed by query, it will shows the error:
RuntimeError: Cannot return the results in a contigious 2D array. Probably ef or M is too small

Basically,
initialize_new_db -> add_docs to new db -> query db✅ Works fine
start new session -> initialize_persistent_client_db -> query db ❌ shows error
Versions
Chroma v0.4.2
Python 3.8.15
Relevant log output
No response

@HammadB can you take a look at this one?

Hi @JacksonCakes, can you share more about where you are running your code (Environment, Platform etc). What files does your "client_path" contain?

I also got this issue at some point but I can't reproduce it for now. It seems like there is a more general issue with the way the database gets initialized from persistence. The other case I got, the search function just returned nothing but the db didn't seem to have any missing doc. In both cases I had to remove and recreate the whole db/collection from scratch.
For context, I can confirm that these issues only happened after doing some writes (Chroma.from_texts() in langchain, and add_text()) as well as some updates using ids. It only happens after reloading from persistence. I don't know if that happens when the operations are done after loading from persistence or the other way around, as I can't reproduce it consistently.

Thanks everyone, sorry to hear about this. We do have pretty extensive testing that tests things like persisting, loading, etc. Please do report back if there is any way to reproduce this!

Hi,
I solved this problem with this
client.get_or_create_collection(COLLECTION_NAME,metadata={"hnsw:M": 64})
you can pass M size with metadata
good luck

Hi, I solved this problem with this client.get_or_create_collection(COLLECTION_NAME,metadata={"hnsw:M": 64})
you can pass M size with metadata and resize until when error is gone. good luck

In my case {"hnsw:M": 1024} All error was gone.

I have this problem too. Google Colab
!pip install openai
!pip install chromadb
import chromadb
from chromadb.utils import embedding_functions
from chromadb.config import Settings
import openai
!pip install chroma
os.environ["OPENAI_API_KEY"] = openai_api_key
openai.api_key = openai_api_key
openai_ef = embedding_functions.OpenAIEmbeddingFunction(
api_key=openai.api_key ,
model_name="text-embedding-ada-002"
)
client = chromadb.PersistentClient(path=idex_path)
collection = client.get_or_create_collection(name="test_collection", embedding_function=openai_ef)
And all OK.
Then i disconnect application and do this
os.environ["OPENAI_API_KEY"] = openai_api_key
openai.api_key = openai_api_key
openai_ef = embedding_functions.OpenAIEmbeddingFunction(
api_key=openai.api_key ,
model_name="text-embedding-ada-002"
)
client = chromadb.PersistentClient(path=idex_path)
client.get_collection(name="test_collection", embedding_function=openai_ef)
Error: Collection test_collection does not exist.

Hi, I solved this problem with this client.get_or_create_collection(COLLECTION_NAME,metadata={"hnsw:M": 64})
you can pass M size with metadata and resize until when error is gone. good luck

In my case {"hnsw:M": 1024} All error was gone.

client.get_or_create_collection(COLLECTION_NAME,metadata={"hnsw:M": 64})

Don't help in my case. Return empty collection

May be i can install previos version chroma?
does anyone know if this is possible?

May be i can install previos version chroma? does anyone know if this is possible?

Are you sure you are getting
RuntimeError: Cannot return the results in a contigious 2D array. Probably ef or M is too small
this error?

Error: Collection test_collection does not exist.

I'm also experiencing the issue of an empty collection return when reloading with PersistentClient() and a path to the store. I created the collection in Google Colab (python 3.10.2) and can reload the store just fine from my google drive in another Colab session, HOWEVER, when I try to load a local copy in my local VS CODE project environment (python 3.11.1), the return is coming up as an empty list with no collections.
Going to test again with a local python 3.10 environment.

@kevon217 im not sure how durable colab storage is. how are you setting the path for your persistent client? how are you connecting VS code to a local copy?

Are you sure you are getting

No. my Error: Collection __ does not exist.
may be this errors is connected, may be not
One result - i can get collection

@kevon217 im not sure how durable colab storage is. how are you setting the path for your persistent client? how are you connecting VS code to a local copy?

Creating and reloading chroma.sqlite3 works just fine on Colab. However, I downloaded the persistent storage to my local (chroma.sqlite3 and 2 directories with bin/pickle files corresponding to my two created collections) and tried running the following code in a Jupyter kernel in VSCode:
import chromadb
client = chromadb.PersistentClient(path="/path_to_local_copy")
client.list_collections()

This just returns an empty list, but I'm expecting 2 collections. Do I need to install any additional dependencies or configure the Settings differently?
Currently running python 3.10.2 and chromadb==0.4.4

@kevon217
i couldnt get header.bin, linked_lists.bin, or length.bin to download from colab.. were you able to?

Yeah I was able to download those directories from Colab. Not sure what they are needed for though. I wish there was more detailed API documentation on local chromadbs. I'm going to just create the collections again on my local.
The indexing does seem to take a while though with only 20k embeddings. Not sure if it is feasible to scale up like I need to.

closing this in favor of https://github.com/chroma-core/chroma/issues/1348

I am facing this issue too and upon closer investigation, I get this error when "allowed_ids" after metadata filtering results in fewer records than the K value in the KNN query.
chroma version: 0.5.5


Crashes with invalid GIF data

Hi, trying to load invalid data as a GIF crashes this code with EXC_BAD_ACCESS. This is due to the unsafe code in delayForImageAtIndex(). Proper checks should be in place and the UIImage.gifWithData() should throw / return nil if invalid data was handed in.
As a quick hack, the following is how I made this 'work' (= not crash):
    class func delayForImageAtIndex(index: Int, source: CGImageSource!) -> Double {
        var delay = 0.1

        guard let cfProperties = CGImageSourceCopyPropertiesAtIndex(source, index, nil) as? NSDictionary,
            gifProperties = cfProperties[kCGImagePropertyGIFDictionary as NSString] as? NSDictionary else {
                return delay
        }

        var delayTime = gifProperties[kCGImagePropertyGIFUnclampedDelayTime as NSString]

        if delayTime?.doubleValue == 0.0 {
            delayTime = gifProperties[kCGImagePropertyGIFDelayTime as NSString]
        }

        if let delayTime = delayTime {
            delay = delayTime.doubleValue
        }

        if delay < 0.1 {
            delay = 0.1 // Make sure they're not too fast
        }

        return delay
    }

Also, building for the device, I get the linker error:
ld: framework not found SwiftGifOrigin for architecture arm64

Matti

Thank you for the information and sample code! :+1:

I tested your code, but the tests are failing due to failed access of cfProperties.
I would be very grateful if you could fix this error and create a PR? :100:

What do you mean 'failed access of cfProperties'? The code I posted compiles & executes without errors on Xcode 7.x build?
It is fairly crucial for our system that the app does not crash when possibly handed a broken .gif, so having this fix is a must.
-Matti

Closed via #25, feel free to reopen if the issue persists.


Is there an under-the-hood tooling to inject services(for DB for example) using DI?

The question says it all.
But what I mean is to be able to inject into the context my custom services?

Which DI container do you use?
It generally is possible to register context objects in DI, although I'm not sure if this will lead to clean code. You also wouldn't register incoming HTTP request objects on it, eh?

Haven't decided yet. And thus want to know is there any tooling like that already present.
I have one more question.
Is it possible when having conversation to put
   const { message: royaltyMessage } = await conversation.waitFor(":text");
into the separate async functions?

Oh wait, do you want to add properties to the context object? I may have misunderstood then, in which case you can use https://grammy.dev/guide/context.html#via-middleware-recommended

Regarding the conversations, see https://grammy.dev/plugins/conversations.html#functions-and-recursion

Hmm, weird.
Thank you.
I will try to go deeper to understand why it doesn't work though in my case

If you are trying to say that you need help, remember that when you say it “doesn't work”, we cannot login to your computer and figure out what's wrong. You need to explain what you are doing, what you are expecting to happen, and what is actually happening. If there's an error, share the full error message, too :)

Yeah, sorry. My bad.
Looked into documentation and see quite no difference between what I wrote.
To be more precise:
Here is the root conversation function.
It invokes other functions
async function editionSettings(
  conversation: BotConversaction,
  ctx: BotContext
) {

  const point = await getButtonName(conversation, ctx);
  console.log("Doesn't reach this point", point);
  await getPayoutAddress(conversation, ctx);
  await getRoyaltyAmount(conversation, ctx);
  await userService.updateUserEditionSettings(
    ctx.chat?.id!,
    ctx.session.editionDraft
  );

  return ctx.reply(EDITION_DEFAULT_SETTINGS_REPLIES_MAP.SETTINGS_UPDATED);
}

Here is the code of getButtonName
export async function getButtonName(
  conversation: BotConversaction,
  ctx: BotContext
) {
  const keyboard = new Keyboard().text("Collect").row().resized().oneTime();

  await ctx.reply(EDITION_DEFAULT_SETTINGS_REPLIES_MAP.ENTER_BUTTON_NAME, {
    reply_markup: keyboard,
  });
  const { message } = await conversation.wait();
  console.log("This message is logging", message);
  ctx.reply("New one");
  const { message: sec } = await conversation.wait();
  console.log("And This message is logging", sec);

  ctx.session.editionDraft.buttonName = message?.text;

  return "just_return_falue_to_check";
}

So the way it works: all console logs inside getButtonName work fine, but the just_return_falue_to_check value is not return into invocation point which is  const point = await getButtonName(conversation, ctx); from the first snippet.
Am I really missing something?

cc KnorpelSenf

Checked it a few times again.
Yeah, it ends the conversation completely when you leave get getButtonName which I guess is a non desired behaviour


Add Ability to Rollback or Deploy Specific Commits of a Project

Currently, there is no way to rollback a deploy or choose a previous version, you have to hope and pray the current build will work or hot fix it. Let's add the ability to rollback the last deploy and/or choose a previous version to deploy in addition to the already existing redeploy option.

Falls outside the scope of the project for v1


Failed to get resource Managed Application Definition after created this resource successfully

Bug Report

Import path of package in question: https://github.com/Azure/azure-sdk-for-go/tree/master/services/resources/mgmt/2018-06-01/managedapplications
SDK version: v35.0.0
Output of go version: go version go1.12.7 windows/amd64

What happened?
It failed to get resource Managed Application Definition and threw below error message after created this resource successfully. After investigated, the property type of isEnabled in local package (https://github.com/Azure/azure-sdk-for-go/blob/781d9fb592a1818cb7fb8e0855ea9c00b095a9e9/services/resources/mgmt/2018-06-01/managedapplications/models.go#L589) is string, however, the property type of isEnabled in the response returned from azure side is boolean. As they are inconsistent, so it encountered property type conversion issue.

What did you expect or want to happen?
It should get resource Managed Application Definition successfully.

How can we reproduce it?

Create a resource group
Create Resource Managed Application Definition
Immediately call "Get" go rest api to get resource Managed Application Definition, which returns below error message

Anything we should know about your environment.
The error message:
Error: Error retrieving Managed Application Definition (Managed Application Definition Name "neiltest2-managedapplicationdefinition" / Resource Group "neiltest2-resources"): managedapplications.ApplicationDefinitionsClient#Get: Failure responding to request: StatusCode=200 -- Original Error: Error occurred unmarshalling JSON - Error = 'json: cannot unmarshal bool into Go struct field ApplicationDefinitionProperties.isEnabled of type string' JSON = '{"properties":{"authorizations":[{"principalId":"xxxx","roleDefinitionId":"xxxxx"}],"isEnabled":true,"lockLevel":"ReadOnly","displayName":"TestManagedApplicationDefinition","description":"Test Managed Application Definition","artifacts":[{"name":"ApplicationResourceTemplate","type":"Template","uri":"https://xxxx/applicationdefinitions/xxxxx/xxx/applicationResourceTemplate.json"},{"name":"CreateUiDefinition","type":"Custom","uri":"https://xxxx/subscriptions/xxx/resourceGroups/neiltest2-resources/providers/Microsoft.Solutions/applicationDefinitions/neiltest2-managedapplicationdefinition/applicationArtifacts/CreateUiDefinition?api-version=2017-09-01"},{"name":"MainTemplateParameters","type":"Custom","uri":"https://management.azure.com/subscriptions/xxxxx/resourceGroups/neiltest2-resources/providers/Microsoft.Solutions/applicationDefinitions/neiltest2-managedapplicationdefinition/applicationArtifacts/MainTemplateParameters?api-version=2017-09-01"}]},"id":"/subscriptions/xxxxxx/resourceGroups/neiltest2-resources/providers/Microsoft.Solutions/applicationDefinitions/neiltest2-managedapplicationdefinition","name":"neiltest2-managedapplicationdefinition","type":"Microsoft.Solutions/applicationDefinitions","location":"eastus","tags":{}}'
on main.tf line 6, in resource "azurerm_managed_application_definition" "example":
6: resource "azurerm_managed_application_definition" "example" {

Hi @olsoro could you please check on this?

This has been fixed in later versions of the SDK:
github.com/Azure/azure-sdk-for-go/services/resources/mgmt/2019-07-01/managedapplications
in v39.0.0

Closed since this get fixed in a higher api-version 2019-07-01


bundle install jf user

so I ran through the documentation again and I cant run bundle install in the api directory as the jf user

Hello @alpha00Zero
What is the error message that you are getting?

<EMAIL_ADDRESS>api]$ bundle install
bash: bundle: command not found

Sounds like the documentation may be missing a gem install bundler command, or it might be out of order.

Nope, I checked the docs, it is right here:
https://github.com/projectjellyfish/api/blob/master/docs/INSTALL-RHEL.md#install-bundler
@alpha00Zero My guess is that some steps got missed.
Chris

Sweet thanks Chris
On Tue, Jul 21, 2015, 10:39 AM Chris Kacerguis<EMAIL_ADDRESS>wrote:

Nope, I checked the docs, it is right here:
https://github.com/projectjellyfish/api/blob/master/docs/INSTALL-RHEL.md#install-bundler
@alpha00Zero https://github.com/alpha00Zero My guess is that some steps
got missed.
Chris
—
Reply to this email directly or view it on GitHub
https://github.com/projectjellyfish/api/issues/854#issuecomment-123376551
.


Remove randomID generation in ingest plugin and add logic back to preprocessing service to pull logs

Aim to finish this on 11/17/2022. But would need some long-term testing

https://forum.opensearch.org/t/can-i-config-opensearch-to-skip-the-same-id-check/12298 posted a discussion here.


useFieldArray - additional documentation on append/prepend/insert focusName format

Thank you for the wonderful project!
Is your feature request related to a problem? Please describe.
The v7 useFieldArray documentation mentions a append/prepend/insert focusName option I would like to use, but I am not what the name should be.
Describe the solution you'd like
An example with focusName in the documentation would be great. Ideally, the option would be more narrowly typed with a helpful typescript error.
Describe alternatives you've considered
I found the solution through the test:
https://github.com/react-hook-form/react-hook-form/blob/master/src/__tests__/useFieldArray/focus.test.tsx#L140
It's obvious in hindsight it would follow the same convention as the rest, but I had tried unsuccessfully a few times before.
Additional context
Feel free to close wontfix, focusName will be searchable in issues now.

can you provide more context on how to improve the doc? Happy to improve on it.

In the useFieldArray api doc, maybe replace "and focus." with "and focus the field corresponding to either: 1) the fully qualified field name if focusName is set; 2) the field specified by focusIndex if it is set and focusName is not; 3) the field created by the action if neither is set.
https://github.com/react-hook-form/react-hook-form/blob/master/src/useFieldArray.ts#L110
One point of confusion for me is how focusIndex (and partial focusNames) should work for nested field arrays. In my use case, it would focus an arbitrary input. I would expect it would focus the first in either the field object keys or in the jsx elements. A note here would be helpful I think.
Another option could be to add an option accepting a union type and deprecate the current fields, although probably something to consider for a later version.
I'll send a PR this weekend if the above looks good.


Expand intro

Closing this down because I haven't heard back. Releasing v1.0.0 here in the next ~hour. Feel free to reopen later and we can adjust.


Drop Python 3.6

Currently advertised supported Python version is:
https://github.com/silx-kit/silx/blob/bf182fd14eb79b0458220f79a5e3b6d27ed7475d/setup.py#L876
but we already advertised drop of Python3.5 in v0.14.0.
Python 3.6 has reached end-of-support at the end of 2021 (https://peps.python.org/pep-0494/).
Question is do we drop it now for the next release or wait for the following one?

Let' drop in the following release


Helm cannot be installed

Describe the bug
helm install  kube-starrocks kube-starrocks -n starrocks
Error: INSTALLATION FAILED: failed to install CRD crds/starrocks.com_starrocksclusters.yaml: unable to decode "": json: cannot unmarshal string into Go value of type unstructured.detector
To Reproduce
cd /starrocks-kubernetes-operator/helm-charts/charts
helm install  kube-starrocks kube-starrocks -n starrocks
Expected behavior
A clear and concise description of what you expected to happen.
Please complete the following information

Operator Version: [v1.9.8]
Chart Name: [kube-starrocks]
Chart Version [v1.9.8]

I can't reproduce your problem.
➜  charts git:(v1.9.8) k delete crd starrocksclusters.starrocks.com
customresourcedefinition.apiextensions.k8s.io "starrocksclusters.starrocks.com" deleted
➜  charts git:(v1.9.8)
➜  charts git:(v1.9.8)
➜  charts git:(v1.9.8) helm install kube-starrocks kube-starrocks -n starrocks

Alias tip: hin kube-starrocks kube-starrocks -n starrocks
walk.go:74: found symbolic link in path: /Users/yandongxiao/github/yandongxiao/starrocks-kubernetes-operator/helm-charts/charts/kube-starrocks/charts/operator/crds/starrocks.com_starrocksclusters.yaml resolves to /Users/yandongxiao/github/yandongxiao/starrocks-kubernetes-operator/deploy/starrocks.com_starrocksclusters.yaml. Contents of linked file included and used
NAME: kube-starrocks
LAST DEPLOYED: Tue Sep  3 13:39:08 2024
NAMESPACE: starrocks
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing kube-starrocks-1.9.8 kube-starrocks chart.
It will install both operator and starrocks cluster, please wait for a few minutes for the cluster to be ready.

Please see the values.yaml for more operation information: https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/helm-charts/charts/kube-starrocks/values.yaml


Update settings handler to support JSON with comments.

[work in progress]
This change will require us to jump to version 0.6.0

Thanks!


Reload rework

Reworks most of the HAProxy reloading code to be a little more testable (and adds tests), while keeping the logic mostly identical. Also adds the features in #158.
The major affect of all this is that

We no longer exit if there is a problem trying to reload the haproxy config
We have better logging in the area
The names of the statsd metrics were changed (well, I added a 2 and changed 1)
We have a nice reliable way to test and cleanup config reloads.

Checklist:

[x] Test
[x] Document
[ ] Test on staging cluster for a while

Do you mind squash the commits to sensible group?
[ ] Update README.md
[ ] Update example/dev config

Done

looks good :+1:  let's test on staging before release.

Seems hunky dory on staging. Merging and deploying.


Threat of Thassarian

[deathknight] fixed all offhand strikes triggered by Threat of Thassarian to use the correct damage formula, reducing the impact of the "flatModifier" by up to 50%, plus minor refactoring

Fixed in #1042


Adding type hints to curry

What's in this pull request?
This pull request adds type hints to the curry functions.
Why merge this pull request?
The swift compiler does not cope very well with generics. The more generics in the type signature, the longer it take to compile (roughly). By adding type hints to the signature, compile times can be sped up.
What's worth discussing about this pull request?
Compile times for Curry.swift (and all source files in general) can be obtained by running:
xcrun xcodebuild clean build OTHER_SWIFT_FLAGS="-Xfrontend -debug-time-function-bodies" | grep [1-9].[0-9]ms > culprits.txt (source swift-profiling)
Compile times for Curry.swift before this pull request are available here
After this pull request Curry.swift does not even show up in the logs
What downsides are there to merging this pull request?
Significantly more characters in Curry.swift

I feel bad about you having to work around the compiler for us like this (I can also vaguely remember doing the reverse of this pull request at some point!).

A general style fix before I merge: we use this form for type signatures: thing : Type.

@CodaFi Don't feel to bad, I generate this via a script. However before merging I suggest you also confirm that the compile time is lowered since this can affect negatively everyone using the library.
I'll try to address the compile time of some other functions in other pull requests.

Also I suggest attaching binaries to releases like so: https://github.com/Carthage/Carthage#archive-prebuilt-frameworks-into-one-zip-file

The problem with prebuilt binaries is that they only work for the version they were compiled with. Has this changed?

I don't think there is Application Binary Interface (ABI) stability yet but anyways the source code will compile only with Swift 2.2

i'm closing this pull request and will submit another with more optimizations including this one.

@blender Good point about the source compatibility. Including a binary for Xcode 7.3/Swift 2.2 would allow easier use and faster build times with Carthage.
@CodaFi Thoughts?

If you decide to go for it it would be a welcome addition to attach binaries also for the other projects. Tyro, Concurrent etc...

We should wait until Swift 3 stabilizes the ABI before providing binary-only releases.  Also need to get this thing working with the package manager which is going to require major dependency tree changes.


Add a 'How to Host This Event' link in the README, with instructions

Going to work on this today.


js-controller 3.2.2

warning:
2021-01-09 11:01:47.843  - warn: proxmox.0 (8650) State "proxmox.0.info.connection" has no existing object, this might lead to an error in future versions

bei sämtlichen State's "has no existing object, this might lead to an error in future versions"

bei sämtlichen State's "has no existing object, this might lead to an error in future versions"

Ich habe gerade eine neue VM angelegt, welche mit folgenden Warnungen im Log auftaucht:
20 Jan 2021 22:01:22.868 proxmox.0: State "proxmox.0.info.connection" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:20.542 proxmox.0: State "proxmox.0.qemu_Docker.disk" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:20.541 proxmox.0: State "proxmox.0.qemu_Docker.disk_lev" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:20.540 proxmox.0: State "proxmox.0.qemu_Docker.maxdisk" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:20.539 proxmox.0: State "proxmox.0.qemu_Docker.cpus" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:20.537 proxmox.0: State "proxmox.0.qemu_Docker.mem" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:20.536 proxmox.0: State "proxmox.0.qemu_Docker.mem_lev" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:20.535 proxmox.0: State "proxmox.0.qemu_Docker.cpu" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:20.533 proxmox.0: State "proxmox.0.qemu_Docker.diskwrite" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:20.532 proxmox.0: State "proxmox.0.qemu_Docker.uptime" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:20.531 proxmox.0: State "proxmox.0.qemu_Docker.status" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:20.529 proxmox.0: State "proxmox.0.qemu_Docker.pid" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:20.525 proxmox.0: State "proxmox.0.qemu_Docker.netin" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:20.524 proxmox.0: State "proxmox.0.qemu_Docker.maxmem" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:20.520 proxmox.0: State "proxmox.0.qemu_Docker.netout" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:15.998 proxmox.0: State "proxmox.0.info.connection" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:15.755 proxmox.0: starting. Version 1.1.0 in /opt/iobroker/node_modules/iobroker.proxmox, node: v12.18.3, js-controller: 3.2.7

Ich habe gerade eine neue VM angelegt, welche mit folgenden Warnungen im Log auftaucht:
20 Jan 2021 22:01:22.868 proxmox.0: State "proxmox.0.info.connection" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:20.542 proxmox.0: State "proxmox.0.qemu_Docker.disk" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:20.541 proxmox.0: State "proxmox.0.qemu_Docker.disk_lev" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:20.540 proxmox.0: State "proxmox.0.qemu_Docker.maxdisk" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:20.539 proxmox.0: State "proxmox.0.qemu_Docker.cpus" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:20.537 proxmox.0: State "proxmox.0.qemu_Docker.mem" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:20.536 proxmox.0: State "proxmox.0.qemu_Docker.mem_lev" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:20.535 proxmox.0: State "proxmox.0.qemu_Docker.cpu" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:20.533 proxmox.0: State "proxmox.0.qemu_Docker.diskwrite" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:20.532 proxmox.0: State "proxmox.0.qemu_Docker.uptime" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:20.531 proxmox.0: State "proxmox.0.qemu_Docker.status" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:20.529 proxmox.0: State "proxmox.0.qemu_Docker.pid" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:20.525 proxmox.0: State "proxmox.0.qemu_Docker.netin" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:20.524 proxmox.0: State "proxmox.0.qemu_Docker.maxmem" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:20.520 proxmox.0: State "proxmox.0.qemu_Docker.netout" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:15.998 proxmox.0: State "proxmox.0.info.connection" has no existing object, this might lead to an error in future versions
20 Jan 2021 22:01:15.755 proxmox.0: starting. Version 1.1.0 in /opt/iobroker/node_modules/iobroker.proxmox, node: v12.18.3, js-controller: 3.2.7

fixed

fixed


alter table drop constraint throws an error.

Hi, the following test does not pass. I would expect this to work.
use sqlparser::*;

#[test]
fn func() {
    let stmt = "ALTER TABLE table_name DROP CONSTRAINT constr_name;";

    let x = parser::Parser::parse_sql(&dialect::GenericDialect {}, stmt);
    dbg!(x).unwrap();
}

(Also,  DropConstraint { name: Ident } should probably also contain fields for cascade/restrict, and possibly if_exists )

Note that when constructing the value by hand and transforming it to a string, we get the expected result:
use sqlparser::ast::{AlterTableOperation, Ident, ObjectName, Statement};

#[test]
fn func() {
    let stmt = Statement::AlterTable {
        name: ObjectName(vec![Ident::new("table_name")]),
        operation: AlterTableOperation::DropConstraint {
            name: Ident::new("constraint_name"),
        },
    };
    println!("{}", stmt);
}

fixed in https://github.com/sqlparser-rs/sqlparser-rs/pull/396


1.c. The “user profile” of Greta Hansen looks funky (on mobile). The boxes are overlapping.

If there isn't a straightforward way to export this properly, I can try including the graphic in a diff format.


Bump patch release to 1.1.2

Submission Checklist

[x] Have you installed and run this change against pre-commit? (pre-commit install)
[X] Are all tests passing? (make tests)
[ ] Have you written unit tests to cover this change?
[X] Is unit test coverage still above 80%?
[X] Have you updated all applicable documentation?
[X] Have you followed the guidelines in our Contributing document?

@heyealex please also review whether the release branch I've created (release-v1.1.1) is the correct one


[graphiql] Unable to input date type

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior
Unable to input date type for "A type representing a formatted java.time.LocalDateTime" in Kotlin with following error: "with value 'StringValue{value='2024-01-31T11:30:59.000Z'}' is not a valid 'DateTime' - Expected valid DateTime literal but was StringValue{value='2024-01-31T11:30:59.000Z'}","
Expected Behavior
It should be working without error
Steps To Reproduce
Spring (Kotlin)
Environment

GraphiQL Version: LTS
OS: Mac & AWS Linux
Browser: any
Bundler:  Kotlin spring
react Version: no react
graphql Version: LTS

Anything else?
??

It should be fixed from source code or documents