qwen2-72B can not be quantized by autoawq

when I use my dataset to quantize qwen2-72B by autoawq, it not works successful.

whether dataset is c4 or my data, it can not be quantized.

This error indicates that the weights end up containing NaN values.

Does it work without using custom data or is it mainly using specific datasets?
Can you show an end-to-end example that triggers the error so I can investigate?

@xiaoxiangshusheng I ran into a similar issue with Llama-2-70B. For me, the issue was because the weights were corrupted during download. Can you try re-downloading the model weights (or otherwise validate the integrity of the weights) and try again?

This error indicates that the weights end up containing NaN values.

Does it work without using custom data or is it mainly using specific datasets?
Can you show an end-to-end example that triggers the error so I can investigate?

@casper-hansen hi，my used model is Qwen2-72B-Instruct，and convert code is
`from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
model_path = '/my_addr/Qwen2-72B-Instruct'
quant_path = '/addr/autoawq/model_Qwen72'
quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }
Load model
model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
Quantize
model.quantize(tokenizer, quant_config=quant_config,
calib_data="/hub/datasets--mit-han-lab--pile-val-backup/snapshots/2f5e46ae6a69cf0dce4b12f78241c408936ca0e4"）
Save quantized model
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)
`
In fact, whether dataset is c4/ptb or my dataset, it can not be quantized.

I guess, the autoawq code requires the dtype of model weights is float16, however, the dtype of Qwen2-72B-Instruct is bfloat16, if I modefiy data dtype as blow, the model can be quantized successful.  I want to know that whether  our autoawq code can use bfloat16 in quantization.

Best wishes for you !

@xiaoxiangshusheng I ran into a similar issue with Llama-2-70B. For me, the issue was because the weights were corrupted during download. IMO you should try re-downloading the model weights (or otherwise validate the integrity of the weights) and try quantization again.

Hi, my used model is download fully，which means wasn't interrupted.
Thank you very much for your advice. I guess that bfloat16 should be worked in quantization.

@casper-hansen Hello, this is the Qwen Team. We also encounter the same problem and use a workaround like this.
https://github.com/yangyo/AutoAWQ/commit/32bf03c3067c5d863a6c7151861f6f954435317d?diff=split&w=1
The problem seems lying in scales.max() == inf and scales.min() == 0 (brought by fp16 conversion?).

May I ask what are your gpu devices type? I am using 4*RTX 4090 to quantize 72B qwen2 model but I always ran into cuda oom error when computing silu acitivations. I used to successfully quantize qwen1.5-72b on Nvidia A10( which has only 22 GB memory). I wonder why it doesn't work this time.

May I ask what are your gpu devices type? I am using 4*RTX 4090 to quantize 72B qwen2 model but I always ran into cuda oom error when computing silu acitivations. I used to successfully quantize qwen1.5-72b on Nvidia A10( which has only 22 GB memory). I wonder why it doesn't work this time.

Hi, @swaggy-TN , it seems that autoawq code can not quantized by muti GPUs. My gpu device is a RTX A6000, and a few custom data to be quantized .in order to avoid meeting the occurrence of OOM situations

@xiaoxiangshusheng Well, autoawq can definitely support multi gpus. nvm I solved it by downgrade the torch version. But it remains unclear why torch2.3.0 doesn't work.

@casper-hansen I'm unsure if the current workaround is an appropriate solution. Our approach just sets scales to 1 upon encountering NaN values. It would be greatly appreciated if you could look into this further. I believe the issue can be reproduced using the code provided by @xiaoxiangshusheng.

@casper-hansen I'm unsure if the current workaround is an appropriate solution. Our approach just sets scales to 1 upon encountering NaN values. It would be greatly appreciated if you could look into this further. I believe the issue can be reproduced using the code provided by @xiaoxiangshusheng.

It's a great workaround that would allow people to use the model anyways. I will investigate what is causing this soon and see if it's fixable.

I think that the dtype of all variables in quantization will be convert into float16 if you load model by fault float16, especially scales which needs to be soloved iteratively. In code:
scales = scales / (scales.max() * scales.min()).sqrt()
Actually, scales will be set in the interval
It is extremely likely to occur the situation that the value of the scales exceeds 65504 when using the qwen2 72B model to complete quantization.  And the number 65504 is the maximum value that fp16 can represent.

@casper-hansen I'm unsure if the current workaround is an appropriate solution. Our approach just sets scales to 1 upon encountering NaN values. It would be greatly appreciated if you could look into this further. I believe the issue can be reproduced using the code provided by @xiaoxiangshusheng.

It's a great workaround that would allow people to use the model anyways. I will investigate what is causing this soon and see if it's fixable.

I'm testing the workaround, 5% 4/80 without problem. will be done in 1:46 hrs, will upload the awq into my huggingface repo

did not work for me, failed at 21%
---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
Cell In[2], [line 12](vscode-notebook-cell:?execution_count=2&line=12)
      [9](vscode-notebook-cell:?execution_count=2&line=9) tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
     [11](vscode-notebook-cell:?execution_count=2&line=11) # Quantize
---> [12](vscode-notebook-cell:?execution_count=2&line=12) model.quantize(tokenizer, quant_config=quant_config)
     [14](vscode-notebook-cell:?execution_count=2&line=14) # Save quantized model
     [15](vscode-notebook-cell:?execution_count=2&line=15) model.save_quantized(quant_path)

File [~/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py:115](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py:115), in context_decorator.<locals>.decorate_context(*args, **kwargs)
    [112](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py:112) @functools.wraps(func)
    [113](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py:113) def decorate_context(*args, **kwargs):
    [114](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py:114)     with ctx_factory():
--> [115](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py:115)         return func(*args, **kwargs)

File [~/.local/lib/python3.10/site-packages/awq/models/base.py:186](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/awq/models/base.py:186), in BaseAWQForCausalLM.quantize(self, tokenizer, quant_config, calib_data, split, text_column, duo_scaling, export_compatible, apply_clip)
    [168](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/awq/models/base.py:168)     self.quant_config.modules_to_not_convert = self.modules_to_not_convert
    [170](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/awq/models/base.py:170) self.quantizer = AwqQuantizer(
    [171](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/awq/models/base.py:171)     self,
    [172](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/awq/models/base.py:172)     self.model,
   (...)
    [184](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/awq/models/base.py:184)     apply_clip=apply_clip,
    [185](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/awq/models/base.py:185) )
--> [186](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/awq/models/base.py:186) self.quantizer.quantize()
    [188](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/awq/models/base.py:188) self.is_quantized = True

File [~/.local/lib/python3.10/site-packages/awq/quantize/quantizer.py:149](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/awq/quantize/quantizer.py:149), in AwqQuantizer.quantize(self)
    [144](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/awq/quantize/quantizer.py:144) # Filter out the linear layers we don't want to exclude
    [145](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/awq/quantize/quantizer.py:145) named_linears = exclude_layers_to_not_quantize(
    [146](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/awq/quantize/quantizer.py:146)     named_linears, self.modules_to_not_convert
    [147](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/awq/quantize/quantizer.py:147) )
--> [149](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/awq/quantize/quantizer.py:149) input_feat = self._get_input_feat(self.modules[i], named_linears)
    [150](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/awq/quantize/quantizer.py:150) clear_memory()
    [152](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/awq/quantize/quantizer.py:152) # [STEP 2]: Compute and apply scale list

File [~/.local/lib/python3.10/site-packages/awq/quantize/quantizer.py:542](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/awq/quantize/quantizer.py:542), in AwqQuantizer._get_input_feat(self, layer, named_linears)
    [535](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/awq/quantize/quantizer.py:535) # get output as next layer's input
    [536](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/awq/quantize/quantizer.py:536)
    [537](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/awq/quantize/quantizer.py:537) # Sanitize the kwargs in case we use transformers version that contains
    [538](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/awq/quantize/quantizer.py:538) # kwargs that are not handled by the module.
    [539](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/awq/quantize/quantizer.py:539) # Useful for trust_remote_code models.
    [540](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/awq/quantize/quantizer.py:540) module_kwargs = self._sanitize_kwargs(self.module_kwargs, layer)
--> [542](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/awq/quantize/quantizer.py:542) self.inps = layer(self.inps, **module_kwargs)[0]
    [543](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/awq/quantize/quantizer.py:543) for h in handles:
    [544](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/awq/quantize/quantizer.py:544)     h.remove()

File [~/.local/lib/python3.10/site-packages/torch/nn/modules/module.py:1532](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/torch/nn/modules/module.py:1532), in Module._wrapped_call_impl(self, *args, **kwargs)
   [1530](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/torch/nn/modules/module.py:1530)     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   [1531](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/torch/nn/modules/module.py:1531) else:
-> [1532](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/torch/nn/modules/module.py:1532)     return self._call_impl(*args, **kwargs)

File [~/.local/lib/python3.10/site-packages/torch/nn/modules/module.py:1541](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/torch/nn/modules/module.py:1541), in Module._call_impl(self, *args, **kwargs)
   [1536](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/torch/nn/modules/module.py:1536) # If we don't have any hooks, we want to skip the rest of the logic in
   [1537](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/torch/nn/modules/module.py:1537) # this function, and just call forward.
   [1538](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/torch/nn/modules/module.py:1538) if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   [1539](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/torch/nn/modules/module.py:1539)         or _global_backward_pre_hooks or _global_backward_hooks
   [1540](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/torch/nn/modules/module.py:1540)         or _global_forward_hooks or _global_forward_pre_hooks):
-> [1541](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/torch/nn/modules/module.py:1541)     return forward_call(*args, **kwargs)
   [1543](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/torch/nn/modules/module.py:1543) try:
   [1544](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/torch/nn/modules/module.py:1544)     result = None

File [~/.local/lib/python3.10/site-packages/accelerate/hooks.py:161](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/hooks.py:161), in add_hook_to_module.<locals>.new_forward(module, *args, **kwargs)
    [160](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/hooks.py:160) def new_forward(module, *args, **kwargs):
--> [161](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/hooks.py:161)     args, kwargs = module._hf_hook.pre_forward(module, *args, **kwargs)
    [162](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/hooks.py:162)     if module._hf_hook.no_grad:
    [163](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/hooks.py:163)         with torch.no_grad():

File [~/.local/lib/python3.10/site-packages/accelerate/hooks.py:356](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/hooks.py:356), in AlignDevicesHook.pre_forward(self, module, *args, **kwargs)
    [345](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/hooks.py:345)             self.tied_pointers_to_remove.add((value.data_ptr(), self.execution_device))
    [347](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/hooks.py:347)         set_module_tensor_to_device(
    [348](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/hooks.py:348)             module,
    [349](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/hooks.py:349)             name,
   (...)
    [353](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/hooks.py:353)             tied_params_map=self.tied_params_map,
    [354](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/hooks.py:354)         )
--> [356](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/hooks.py:356) return send_to_device(args, self.execution_device), send_to_device(
    [357](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/hooks.py:357)     kwargs, self.execution_device, skip_keys=self.skip_keys
    [358](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/hooks.py:358) )

File [~/.local/lib/python3.10/site-packages/accelerate/utils/operations.py:177](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/utils/operations.py:177), in send_to_device(tensor, device, non_blocking, skip_keys)
    [175](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/utils/operations.py:175)         return tensor.to(device)
    [176](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/utils/operations.py:176) elif isinstance(tensor, (tuple, list)):
--> [177](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/utils/operations.py:177)     return honor_type(
    [178](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/utils/operations.py:178)         tensor, (send_to_device(t, device, non_blocking=non_blocking, skip_keys=skip_keys) for t in tensor)
    [179](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/utils/operations.py:179)     )
    [180](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/utils/operations.py:180) elif isinstance(tensor, Mapping):
    [181](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/utils/operations.py:181)     if isinstance(skip_keys, str):

File [~/.local/lib/python3.10/site-packages/accelerate/utils/operations.py:81](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/utils/operations.py:81), in honor_type(obj, generator)
     [79](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/utils/operations.py:79)     return type(obj)(*list(generator))
     [80](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/utils/operations.py:80) else:
---> [81](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/utils/operations.py:81)     return type(obj)(generator)

File [~/.local/lib/python3.10/site-packages/accelerate/utils/operations.py:178](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/utils/operations.py:178), in <genexpr>(.0)
    [175](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/utils/operations.py:175)         return tensor.to(device)
    [176](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/utils/operations.py:176) elif isinstance(tensor, (tuple, list)):
    [177](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/utils/operations.py:177)     return honor_type(
--> [178](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/utils/operations.py:178)         tensor, (send_to_device(t, device, non_blocking=non_blocking, skip_keys=skip_keys) for t in tensor)
    [179](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/utils/operations.py:179)     )
    [180](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/utils/operations.py:180) elif isinstance(tensor, Mapping):
    [181](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/utils/operations.py:181)     if isinstance(skip_keys, str):

File [~/.local/lib/python3.10/site-packages/accelerate/utils/operations.py:158](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/utils/operations.py:158), in send_to_device(tensor, device, non_blocking, skip_keys)
    [156](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/utils/operations.py:156)     tensor = tensor.cpu()
    [157](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/utils/operations.py:157) try:
--> [158](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/utils/operations.py:158)     return tensor.to(device, non_blocking=non_blocking)
    [159](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/utils/operations.py:159) except TypeError:  # .to() doesn't accept non_blocking as kwarg
    [160](https://file+.vscode-resource.vscode-cdn.net/home/dario/~/.local/lib/python3.10/site-packages/accelerate/utils/operations.py:160)     return tensor.to(device)

NotImplementedError: Cannot copy out of meta tensor; no data!

@swaggy-TN the autoawq kernels are compiled in torch 2.2.0. torch does not have backward compatibility which is why the current requirement is 2.2.0. i am looking to make this better and more clear in future versions.
@hzhwcmhf can you create a PR to solve the issues or do you want me to investigate?
@xiaoxiangshusheng multi-GPU is supported, but not concurrent quantization. i am looking to optimize this in the future so that quantization is faster. it works like this: model is loaded onto your CPU and GPU. then the quantization goes layer by layer, moving each layer to a GPU and quantizing it. there is plenty of room for optimization here. NOTE: the extra VRAM is only used if there is not enough RAM to load the model in.
@xiaoxiangshusheng the following may be a problem because you are referencing blobs instead of the actual dataset. might not work as expected. please see the docs for how I quantized a model using custom data: https://casper-hansen.github.io/AutoAWQ/examples/#custom-data
calib_data="/hub/datasets--mit-han-lab--pile-val-backup/snapshots/2f5e46ae6a69cf0dce4b12f78241c408936ca0e4"）

Is there an example on how to quantize with multiple GPUs? Is it possible to quantize Llama 3 70B with 2x3090 24GB?

I implemented the following fix and successfully quantized Qwen2 72B. This will keep the scales that did not overflow, but ignore scaling parts of the tensor that overflowed. I tried playing with the scaling formulation, but couldn't find a way to avoid producing infs or nans, so this is the best workaround for now. #516 brings some new features and this fix.

https://github.com/casper-hansen/AutoAWQ/pull/516/commits/bc37fda11ad12309e25abdcd64e494cf661acc51


Add uninstall helper for simple delete instead of using cmd

Give it a progress bar, and option to run loud (maybe no ui at all for quiet). Loud has uninstall/cancel question, and maybe a tree of files to check what to delete.

Only if you can put it in hidden mode. But still working as usual.
Just a suggestion, do not take it as something mandatory.


"pn" is not "on" - Typo on the resources page.

Small little tweak, changing "they play the game pn pc or xbox..." to "they play the game on pc or xbox..."

Thanks!


596 implement backups for apply for qts

Backup and Restore postgres db for production environment.

 Implement backups for apply for qts


Umbraco V13 support

This should be updated for Umbraco V13

Fixed in V3.0.0 which is on NuGet now


Update check for RTS membership

To address https://github.com/unrvld/ODP.VisitorGroups/issues/22

@daniel-isaacs Thanks for fixing this. I had tested this but obviously not very well.


Experimental API for saving and loading models

The biggest pain point of using lucid seems to be importing models. Most users wish to visualize their own models, and need to get them into a format lucid can use. Unfortunately, this presently involves several steps, which can be unintuitive:

Save a graph
Convert it into a frozen graph
Write a Lucid modelzoo.Model class, filling in values like image_value_range

This PR proposes an alternate API where there is only one, clearly defined step to preparing your model for use in Lucid. This import path is only for TensorFlow users.
We assume that the user can construct an inference graph of their model. This should be easy for anyone training models (because they will have one for tracking accuracy) and for anyone using a model for inference.
A this point, the user simply calls the save_model() function.
# Run this code with inference graph in default graph and session
save_model(
    save_path    = 'gs://.../test.pb',  # Local paths are also fine!
    input_name   = 'input',
    output_names = ['prob/Softmax'],
    image_shape  = [224, 224, 3],
    image_value_range = [0,1]
  )

If the user successfully does this with the correct arguments, they should be done. All metadata is baked into the save model (more on this later), meaning it never needs to be specified again.
To use a model in lucid, the user simple does:
model = load_model('gs://.../test.pb')

And they're ready to go!
Suggesting save code for users
The above import path is still unnecessarily painful, since we can often infer many of the arguments to save_model() with high confidence.
I considered just making save_model() have optional arguments and try to infer unspecified arguments if possible. However, this struck me as an API that would lead to user confusion, since we can't always infer these arguments, aren't completely certain, and some arguments just can't be inferred.
Instead, I went with a suggest_save_code() function, which is simply invoked:
# With inference graph in default graph
suggest_save_code()

If our heuristics can determine arguments, it will print out something like the following:
>>> suggest_save_code()

# Infered: input_name = input  (only Placeholder)
# Infered: image_shape = [224, 224, 3]
# Infered: output_names = ['prob/Softmax']  (Softmax ops)

# Sanity check all inferred values before using this code!
save_model(
    save_path    = 'gs://save/model.pb', # TODO: replace
    input_name   = 'input',
    output_names = ['prob/Softmax'],
    image_shape  = [224, 224, 3],
    image_value_range =                  # TODO (eg. [0, 1], [0, 255], [-117, 138] )
    # WARNING: Incorrect `image_value_range` is the most common cause of feature
    #     visualization bugs! It will fail silently with incorrect visualizations!
  )

In other cases, when it can't be inferred, the output will be something like this, giving the user a n empty template to fill in.
>>> suggest_save_code()

# Sanity check all inferred values before using this code!
save_model(
    save_path    = 'gs://save/model.pb', # TODO: replace
    input_name   =   ,                   # TODO (eg. 'input' )
    output_names = [ ],                  # TODO (eg. ['logits'] )
    image_shape  =   ,                   # TODO (eg. [224, 224, 3] )
    image_value_range =                  # TODO (eg. [0, 1], [0, 255], [-117, 138] )
    # WARNING: Incorrect `image_value_range` is the most common cause of feature
    #     visualization bugs! It will fail silently with incorrect visualizations!
  )

Input Range
My biggest concern with this API -- and any other I can think of -- is misspecified image_value_ranges. Unlike other errors, this will not cause visualization to fail. Instead, it will cause bad or incorrect visualizations to be produced, failing silently.
There isn't any reasonable way for us to catch these errors. (My best thought would be to test accuracy on ImageNet for different common ranges.)
At the moment, trying to warn users that this is a common silent failure mode seems like the best bet.
Storing Metadata
This API depends on us being able to save metadata along with the graph. This is critical to making it into "one save step" instead of two separate steps of saving and then later specifying metadata on import.
For this code, I do something kind of evil to accomplish this. I inject a tf.constant op into the graph named lucid_meta_json, containing a json blob of metadata.  On import, we can detect and extract this node to recover meta data. This is a completely legal TensorFlow graph! But not really being used as intended...
In the future, we might be able to switch over to SavedModel and use their actual supported mechanisms for specifying metadata.

Pull Request Test Coverage Report for Build 461

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.05%) to 81.923%

Totals

Change from base Build 459:
-0.05%

Covered Lines:
1491

Relevant Lines:
1820

💛  - Coveralls

As a novice lucid user I tried this out and saving my model was pretty straightforward. I've not yet tried doing anything with the saved model, but nothing went obviously wrong.
I did find the suggest_save_code interface to be a little strange. More natural to me would be a function called something like save_model_try_infer, which is a wrapper around save_model that does its best to infer any unspecified arguments, raising an exception if there is something that it can't infer.
A minor issue I had is that json.dumps doesn't support numpy dtypes (try json.dumps(np.array([1])[0]) for example), which is a bit of a shame but probably not worth worrying about.

Thanks for the feedback, Jacob!

I did find the suggest_save_code interface to be a little strange. More natural to me would be a function called something like save_model_try_infer, which is a wrapper around save_model that does its best to infer any unspecified arguments, raising an exception if there is something that it can't infer.

Yeah, it's totally weird.
I think my main concern about the save_model_try_infer API is that our heuristics may be wrong. That makes me kind of want to get explicit user sign-off on inferred things, if that makes sense? But maybe that's naive of me, and users will just OK whatever it suggests without looking.

A minor issue I had is that json.dumps doesn't support numpy dtypes (try json.dumps(np.array([1])[0]) for example), which is a bit of a shame but probably not worth worrying about.

Thanks for raising this! Will fix. :)

Another way of doing it that avoids generating code would be to have a suggest_save_kwargs function that returns a dict that you can pass as kwargs to save_model. I can see the benefits of the current interface though.

@colah For me, this API looks very good improvement from old method of loading. Thanks for this! Here are my thoughts :)

From Usability point of view:

This API looks much easier to use than the previous method of importing custom model! Thanks for making this so simple!
import lucid
<Code to train your model>
.
.
.
#save model
lucid.save_model(....)

Then
#load model
lucid.load_model(...)

From Use case point of view:

I have access to the code to train the model

This is very simple and just save_model  and load_model   will take care of most of the work! Makes life easier!

I got the model from internet as as inference graph(pb). This model is not there in model zoo.

If i understand correctly, we have to follow the old method to import this model or write a script to load this model and save it with save_model ? Because this graph will not have lucid_meta_json node so cannot be used with load_model  API. I think this could be improvement point in future to make the model loading more easier. My thought is load_model may not be right place to provide this functionality and new API may be required.

I need to visualize a model which is in modelzoo
It was always easy to do this! Just use the existing API to import and load.

Overall, this change is very encouraging to try out my own model and visualize with lucid.

I had an issue where my input was originally a uint8 placeholder - talked to @colah about this.

@colah I took a shot at how I'd integrate this into our existing API.
You can look at the tests I wrote for usage examples, but the gist is:
path = ...
inferred = Model.suggest_save_args()  # throws if one arg can't be inferred
Model.save(path, **inferred, image_value_range=(0,1))
reloaded_model = Model.load(path)

This unifies the APIs for loading to the following static methods:
Model.load(graphdef_url)
Model.load_from_metadata(graphdef_url, metadata)
Model.load_from_manifest(manifest_url)

I'll still fix Python 2 and remove some commented out code, so please don't merge yet. :-)

@jacobhilton @colah what was the suggested solution in your case? I'll happily write a test for this case, I imagine that might be common. :-)

@colah from 523d4dd Model.suggest_save_args no longer throws. Instead we write to stdout as in your original proposal:
Inferred: input_name = input (because it was the only Placeholder in the graph_def)
Inferred: image_shape = [16, 16, 3]
Inferred: output_names = ['Softmax']  (because those are all the Softmax ops)
# Please sanity check all inferred values before using this code!
Model.save(
    input_name='input',
    image_shape=[16, 16, 3],
    output_names=['Softmax'],
    image_value_range=_,                   # TODO (eg. '[-1, 1], [0, 1], [0, 255], or [-117, 138]')
  )


blockcypher api on gettxout is broken

the translator from "blockcypher style" script types for to legacy must be extended.
https://github.com/gdassori/spruned/blob/master/spruned/services/blockcypher_service.py#L72
this must be done after an investigation on how blockcypher return all the known script types.

current gettxout api is deprecated


attrs package conflict

hi, thanks for great modification.
there’s some packages are conflict each others
such as -
user requested attrs==18.1.0
automat 20.2.0 depends on attrs>=19.2.0
coursera-dl 0.11.5 depends on attrs==18.1.0
so i just remove attars==18.1.0 and also remove the version of coursera-dl from requirements txt

add python-binance and ta-lib in requirements.txt


added support for agent principal auth type

Description
Added support for authenticating with an agent principal (e.g., in order to submit MELT data)
Type of Change

[ ] Bug Fix
[X] New Feature
[ ] Breaking Change
[ ] Refactor
[ ] Documentation
[ ] Other (please describe)

Checklist

[X] I have read the contributing guidelines
[ ] Existing issues have been referenced (where applicable)
[X] I have verified this change is not present in other open pull requests
[X] Functionality is documented
[X] All code style checks pass
[ ] New code contribution is covered by automated tests
[X] All new and existing tests pass

Codecov Report

Merging #53 (8bef05d) into main (1b64f16) will decrease coverage by 0.42%.
The diff coverage is 0.00%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@            Coverage Diff             @@
##             main      #53      +/-   ##
==========================================
- Coverage   33.95%   33.53%   -0.42%
==========================================
  Files          27       27
  Lines        3390     3432      +42
==========================================
  Hits         1151     1151
- Misses       2152     2194      +42
  Partials       87       87

Impacted Files
Coverage Δ

cmd/config/config.go
29.94% <0.00%> (ø)

cmd/config/set.go
9.49% <0.00%> (-0.45%)
:arrow_down:

cmd/config/types.go
0.00% <0.00%> (ø)

platform/api/login.go
15.05% <0.00%> (-0.34%)
:arrow_down:

platform/api/service_principal.go
0.00% <0.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Support x-wso2 Swagger extensions that are already supported by Micro Gateway

Describe your problem(s)
WSO2 Micro gateway currently supports processing several x-wso2 vendor extensions defined at swagger level which contain all the information required to create APIs. Processing these needs to be supported at APIM level in order to allow the same swagger definition to be used to create a complete API on APIM as well as importing a complete API definition as a swagger

Here are the list of extensions[1] supported by microgateway. Its better if we can capture those extensions from swagger when creating the API
[1] - https://github.com/wso2/product-microgateway#microgateway-supported-open-api-extensions

Please find the list of extensions we need to support from APIM side as well

x-wso2-basePath
x-wso2-production-endpoints
x-wso2-sandbox-endpoints
x-wso2-endpoints
x-wso2-throttling-tier
x-wso2-cors
x-wso2-disable-security
x-wso2-transports
x-wso2-application-security
x-wso2-mutual-ssl
x-wso2-auth-header
x-wso2-response-cache


Using a --pid=host container results in unstoppable container

I have been working on a OpenStack Nova compute node container set.  It involves at the moment a nova-network container, and a nova-compute container.  The nova-compute container runs libvirt and nova-compute.  I probably will split these into separate containers at some point, if that would work.
I recently introduced the --pid=host flag to my start script for docker and it results in an unkillable container.  docker stop blocks permanently.  pressing ctrl-c on the docker cli process and running docker stop again result in errors about processes not found.
The worst part is processes in the original container are leaked into the operating system permanently.
To run this container for debugging, it is necessary to stop libvirt in your host OS (because the container will want  to run libvirt in your host OS).
The start command is here:
https://github.com/sdake/compute-upgrade/blob/master/tools/start-compute#L29

The full set of processes in the container
odd that virtlockd/libvirt has nova-compute as a parent
[root@minime cgroup]# ps -ef | grep 6597
root      6597   859  2 09:11 ?        00:00:00 /usr/bin/python /usr/bin/nova-compute
root      6760  6597  0 09:11 ?        00:00:00 /usr/sbin/virtlockd
root      6765  6597  0 09:11 ?        00:00:00 /usr/sbin/libvirtd
root      6900  6282  0 09:12 pts/0    00:00:00 grep --color=auto 6597
The full set of processes with docker as a parent
[root@minime cgroup]# ps -ef | grep 859
root       859     1  0 08:16 ?        00:00:00 /usr/bin/docker -d --selinux-enabled -H fd://
root      6542   859  0 09:11 ?        00:00:00 /usr/bin/python /usr/bin/nova-network
root      6597   859  0 09:11 ?        00:00:00 /usr/bin/python /usr/bin/nova-compute
After a docker stop operation on the nova-compute container (PID 6597) the state of the system is:
root      6765     1  0 09:11 ?        00:00:00 /usr/sbin/libvirtd
root      6760     1  0 09:11 ?        00:00:00 /usr/sbin/virtlockd
On a docker stop, the host system's systemd is reparenting libvirtd and virtlockd to PID 1 (systemd).  For some reason this is blocking docker stop from killing the processes.

@sdake Could you provide docker version and docker info. Also, do you use docker exec? Thanks!

I can confirm that this issue in latest master.

@sdake Hmmm, this is blocking docker stop because of container cgroups, which can't be deleted. Fun that I can't delete them even after docker stopped and libvirt killed.

@sdake what happens if you run docker kill?

@crosbymichael
slightly different behavior
[root@minime tools]# docker kill 1e332b49962f
Error response from daemon: Cannot kill container 1e332b49962f: no such process
FATA[0010] Error: failed to kill one or more containers
processes still remain after kill:
[root@minime tools]# ps -ef | grep libvirt
root      1955     1  0 11:46 ?        00:00:00 /usr/sbin/libvirtd

This seems to fix it, but wouldn't work for the normal case where --pid=host is called I suspect.
Fedora's kernel is reparenting "faster" in some way to PID1 on process kill.  When the reparent happens, /proc/self/pid where pid is a child process in the container disappears from the pid address space because it has been reparented.  This causes the below code to fail to access key data and return errors, blocking the stop.  Workaround is to just access /proc/pid directly.
http://paste.fedoraproject.org/173764/43119142

Dan Walsh suggested a change such as:
http://paste.fedoraproject.org/173792/45803142
Which seems to fix the problem.  No idea why.

Weird that it would come down to something with selinux?
On Friday, January 23, 2015, Steven Dake<EMAIL_ADDRESS>wrote:

Dan Walsh suggested a change such as:
http://paste.fedoraproject.org/173792/45803142
Which seems to fix the problem. No idea why.
—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/issues/10303#issuecomment-71262469.

I'm going to say "that patch is wrong".  Maybe it works around a problem, but it is wrong.  period full stop.  (And I'm the SELinux kernel and userspace libraries maintainer)
I'm starting to play with your reproducer...

Awesome thanks so much for the feedback and help
On Friday, January 23, 2015, Eric Paris<EMAIL_ADDRESS>wrote:

I'm going to say "that patch is wrong". Maybe it works around a problem,
but it is wrong. period full stop. (And I'm the SELinux kernel and
userspace libraries maintainer)
I'm starting to play with your reproducer...
—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/issues/10303#issuecomment-71263854.

We found something on our side actually. That exec.Cmd.Wait is not exiting with your start script + pid namespace even if process died and that's weird :/

@LK4D4 this is likely because libvirtd and virtlockd are still running in the system.  Wait will wait until those processes exit.

@sdake But why? And how can I kill them?

@eparis yup the fact that I don't understand why it seems to fix the problem is a clear sign we haven't found a root cause.
I did want to report I ran docker via strace -f with master and found that an ENOENT was returned during the stop operation via an open syscall directed at /proc/self/tasks/#/attr/exec which likely jammed up the rest of the stop operation.

Fun thing, Wait haging even after killing all processes in cgroup :) Seems like golang bug.

@LK4D4 I killed them via a ssh root shell into my machine - eg killall -15 libvirtd

adding some log messages, we never end up in killAllPids()   which I thought was supposed to happen

We are tracking this down to something around:
https://github.com/golang/go/issues/7990

The rabbit hole is deep with this one.
On Fri, Jan 23, 2015 at 2:13 PM, Michael Crosby<EMAIL_ADDRESS>wrote:

We are tracking this down to something around:
golang/go#7990 https://github.com/golang/go/issues/7990
—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/issues/10303#issuecomment-71275585.

I have confirmed that Dan's cherrypicked repo of LK4D4's patch fixes the problem.
Thanks for the responsive attention!
Regards
-steve


Upgrade github.com/kr/text to solve non go mod indirect dependency

github.com/niemeyer/pretty requires<EMAIL_ADDRESS><EMAIL_ADDRESS>requires<EMAIL_ADDRESS><EMAIL_ADDRESS>has no go.mod file so AFAIU it messes up dependency graphs of any package that indirectly requires github.com/niemeyer/pretty
<EMAIL_ADDRESS>requires<EMAIL_ADDRESS>which has a go.mod file.
If this is accepted an upgrade of github.com/niemeyer/pretty in gopkg.in/yaml.v3 would solve a few go.mod // indirect lines in all packages importing gopkg.in/yaml.v3.

Hum, I've been a bit quick, that is not gopkg.in/yaml.v3 which should have github.com/niemeyer/pretty upgraded but rather gopkg.in/check.v1 which also has no go.mod file.
I made a couple forks of your repos to validate what it does:

github.com/sylr/go-check/v2 is a module built on to top of gopkg.in/check.v1 with a go.mod file which replace github.com/niemeyer/pretty by github.com/sylr/pretty
github.com/sylr/pretty is a fork of github.com/niemeyer/pretty with the patch of this PR.
github.com/sylr/go-yaml is a fork of github.com/go-yaml/yaml v3 branch which uses github.com/sylr/go-check/v2 instead of gopkg.in/check.v1

Here is a the initial graph of one of my module:

You'll see a direct link between the module and gopkg.in/check.v1 and github.com/kr/pretty which should not be there.
After:

Those graphs have been generated using @Helcaraxan gomod tool.

The "indirect" dependencies are actually not (yet) represented correctly by gomod. I realised this myself yesterday while doing some investigations on the kubernetes/kubernetes dependency graph.
I have a fix that I am merging today and which you can already build and use yourself from the project's non-mod-deps branch. It is already working and I simply need to fix up some tests that broke to the nature of the changes that I am introducing.
I'd suggest rerunning your graph generation with the updated tool to correctly reflect the source-code-based dependency graph and not the modified one that Go modules creates to deal with non-module dependencies.

The related issue is Helcaraxan/gomod#68.

It's still worth to have a popular package such as gopkg.in/yaml.v3 have a clean go mod dependency chain, wouldn't you agree ?

Oh definitely. I don't disagree with that idea. I just wanted to make sure that you were aware of the potential fallacy of the graphs that gomod generated (up til its v0.5.0, still have to tag the v0.6.0) when it comes to non-module dependencies.
On the other hand I have not looked into the details of this PR or what it does so I can't comment on its specifics. :)


[Bug]: Unwanted borders around buttons

Describe the bug
There are these borders around the buttons for some reasons
Screenshots

What browser are you using?
Microsoft Edge
Version 113.0.1774.35 (Official build) (64-bit)
What version of AdashimaaTube are you using?
2.2.6

Can you send me the link of the that videos?

You download other extension so I can't help. https://github.com/sapondanaisriwan/AdashimaaTube/issues/5

You download other extension so I can't help. #5

What are u even talking about? It's specific to 2.0.6

could you right-click the border area, select "inspect," and send me the screenshots?

could you right-click the border area, select "inspect," and send me the screenshots?

It was caused by ambient light that overrides the CSS for some reasons, closed


Support for searching

Python lib supports searching.
This is a very handy feature, considering the size of databases.
Any plans?

Hi Smostovoy,
You can use the all commands (Quandl::Database.all() and Quandl::Dataset.all()) in our new package to search for databases and datasets. An example that searches for databases with gold in them could look like: Quandl::Database.all(params: { query: ‘gold' })


Fix missing curly bracket

In the README.md file

No curly bracket is needed there. Take a look at the url property before the form - it's part of the options object.

Oeps, sorry!


Winning Player not receiving coins

Player who won previous round is not receiving pot even when next round starts (not live)
Player with chosen proposal does not see their pot update when they win (live)

Players are not receiving winning coins at all - they start each round with the coins they ended with the previous round, ignoring any winnings, proposal or agenda.

UPDATE: Players are getting points live time for agenda items

I saw this once and then not again. Please try to keep reproducing, noting exact player actions taken.

confirmed fixed on production


[TC-TSTAT-1.1] ClusterRevision value has to be updated from 6 to 7 in Yaml script.

Description:

In TC-TSTAT-1.1, the ClusterRevision value has to be updated from 6 to 7 in step 2.

Script Reference: https://github.com/project-chip/connectedhomeip/blob/master/src/app/tests/suites/certification/Test_TC_TSTAT_1_1.yaml

Test Plan Reference: https://github.com/CHIP-Specifications/chip-test-plans/blob/master/src/cluster/thermostat.adoc#tc-tstat-1-1-global-attributes-with-dut_server

Issue fixed


ERR_OSSL_EVP_UNSUPPORTED

I've just experienced the above error when trying to open a dual funded channel.
I solved by running the following export:
export NODE_OPTIONS=--openssl-legacy-provider
I don't know if it was a problem of bos or nodejs, but I wanted to report this just in case someone else gets the same error.
My environment is:
bos 12.7.0
nodejs 17.9.0 (current latest)
rasblitz node 1.7.2

please try node LTS 16.15.0?

Dear Alex, as I said I solved by running the export, so I'll try the next time I'm the same situation of opening the dual funded channel. All the other things worked.

ok if that works for you long term that is great
this is an issue upstream in nodejs with tls support so hopefully that gets fixed in the latest-features nodejs

perfect, thank you for the clarification.

this should be addressed now


Please tag a stable release

The option -R was included in r131, and the most recent tagged release is 0.2 r128. Could you please tag a new stable release that includes the -R option?
miniasm: illegal option -- R

Currently miniasm -R prints a warning and continues. Could this please instead exit with an error?
Thanks, Heng!

Bump. Time to tag miniasm 0.3, Heng?

I also not able to access the R -option thought my miniasm 0.2-r128.
Since my run with miniasm segfaulted, it would be great to have this option working!
Thanks
Sam

@bioteksampath Hi, Sam. Here's the related issue for Miniasm segfaulting: https://github.com/lh3/miniasm/issues/48

Done. Sorry for the delay.

Thanks, Heng!


Added some modifications into the global functions, and fixed a spelling mistake

Fixed the success sentence, which was outputted when a new phar build was finished,
Added automatic environment variable registration to the server function ("host" and "name" variables),
Added a has function, so that the existence of a parameter can be checked.

I'm awaiting your opinions.

Forgot to mention, if this gets merged in, then the documentation should definitely mention the automatically generated env variables. (host/name)
Same with the has function, it must be listed in the "Functions" section of the docs.

And check new issues here: https://scrutinizer-ci.com/g/deployphp/deployer/inspections/fe07aa87-2431-4754-ab40-813f64962e04

BTW, its better to separate PR into small, independent ones. So it's simple to merge them.

I'll keep that in mind next time. :)
I'll check these issues as soon as I can.

I refactored some things and played around the "host" and "name" environment settings.
I discarded the latter idea because the first - naive - implementation would certainly cause issues in the future.
I'll open a separate issue about the env variables.

After 7 commits, this PR has:

Spelling fix. Fixed the success sentence, which was outputted when a new phar build was finished,
Added a has function, so that the existence of a parameter can be checked.
I think clear more if you re-base before merging code.

Do we really need has function?

I can break this PR into two separate one if you think, one for the spelling and an other for the has function.
I added that global function, since I wanted to check inside a task, whether a global configuration array was set or not. I wanted to implement proper error handling, which is as informative as it can be.
The current get implementation throws an exception if the given key does not exist in the collection, which is fine, but I wanted a different error message.
I can of course put the get call in a try-catch block, and throw my own exception, but then there's the chance of catching exceptions not related to the existence of the global variable I was trying to access.
The task I was trying to implement was a slack notification task, and with the has function I could do the following:
// Checking for the existence of the slack task configuration.
// If it doesn't exist, I throw a custom exception with detailed information
// (Actually it could be even more detailed, by giving the expected array keys,
// but this is only an example.)
if (!has('slack')) {
    throw new \RuntimeException('Missing "slack" parameter for the "slack:push:success" task. Please provide it using the "set(\'slack\', [])" function.');
}

// The slack global variable now surely exist. If this get call throws
// something, that will not be related to the existence of the variable.
$config = get('slack');

// Validating the given configuration options goes here
// ...

In short: I wanted to implement proper error handling inside a given task, and I think a has function can make that very simple.

Looks good. I'm merging.

Thanks. :)
On a related note: would you consider adding a simple "slack" task?
Because I could share what I have.

Take a look at: https://github.com/deployphp/recipes

Now that slack implementation is a perfect example why I wanted to access the "host" server configuration.
Does it work in v3?
It uses the config() function which I think does not exist anymore:
https://github.com/deployphp/recipes/blob/master/recipes/slack.php

Yeahh, need to create issue there.

On it. :)


EKS node on ARM64 is reported ready before its corresponding CSR is signed.

What happened:
When provisioning many nodes at the same time on EKS 1.29 specifically with ARM64 node image on AL 2023 (latest) sometimes a node would come up and report its status as ready before its CSR is signed and approved. This results in a situation where the pods are running on the node but one cannot run kubectl logs <POD_NAME> which returns tls internal error
The condition goes away after a bit by itself.
The below commands were run simultaneously:
dcherniv@lildebbie:~/$ kubectl get nodes
NAME                          STATUS     ROLES    AGE     VERSION
[...]
ip-10-8-12-74.ec2.internal    Ready      <none>   40s     v1.29.6-eks-1552ad0
[...]

dcherniv@lildebbie:~/$ for i in `kubectl get nodes | awk '{print $1}' | grep -v NAME` ; do kubectl get csr | grep $i ; done
[...]
csr-q4dm7   44s     kubernetes.io/kubelet-serving   system:node:ip-10-8-12-74.ec2.internal    <none>              Pending
[...]

After a while the CSR is finally approved and we can run kubectl commands against pods on this node:
dcherniv@lildebbie:~/$ kubectl get csr | grep 12-74
csr-q4dm7   74s     kubernetes.io/kubelet-serving   system:node:ip-10-8-12-74.ec2.internal    <none>              Approved,Issued

What you expected to happen:
The node should not post ready status until we can run kubectl commands against it.
How to reproduce it (as minimally and precisely as possible):
Spin up multiple nodes (20 should be enough) at the same time. And watch for their ready status while simultaneously watching for their CSR. There will be a short time window where the nodes will post ready but CSRs would still be pending.
Anything else we need to know?:
Environment:
- AWS Region: us-east-1
- Instance Type(s): m6g.2xlarge
- EKS Platform version (use `aws eks describe-cluster --name <name> --query cluster.platformVersion`): eks.10
- Kubernetes version (use `aws eks describe-cluster --name <name> --query cluster.version`): 1.29
- AMI Version: amazon-eks-node-al2023-arm64-standard-1.29-v20240828

Small correction, apparently during scale up event, it happens on the amd64 node types too.
ip-10-8-13-89.ec2.internal    Ready      <none>   13s     v1.29.6-eks-1552ad0
csr-txknw   21s     kubernetes.io/kubelet-serving   system:node:ip-10-8-13-89.ec2.internal    <none>              Pending

After a while the CSR is finally approved

How much time are we talking about?
We'd probably need to look at your control plane logs to see what happened, but my guess would be the event queue of the certificate controller is getting backed up when many CSR's are created in a short period of time. If you want to open a case with AWS Support and provide your cluster ARN, we can look into it.

Yea I figured as much. It only happens under load when nodes provisioned simultaneously.
20-30 second delay after node is reported and until CSR is signed.
I can create a case for sure. But it seems there is a fundamental problem here. Should node report itself as ready before its CSR is approved?

Should node report itself as ready before its CSR is approved?

For the serving cert (what you're seeing) I think there's an argument either way. If your workloads depend on kubelet's HTTP server (which is not common at all), then it would make sense to delay pod scheduling until that CSR is approved. But in most cases, kubelet can go ahead and start doing useful work (pulling your container images, getting pods running) while it waits for the CSR to be approved.
The consensus here was that this shouldn't be part of node readiness by default: https://github.com/kubernetes/kubernetes/pull/73047
I'd support an optional toggle for it if you want to reboot the discussion 👍

Ah, sounds like its a non-issue then. Don't want to get into lengthy discussions of what constitutes a ready node. It's way above my paygrade :)
Closing this with WONTFIX.


Zero transitions should be possible

In FRBC, OMBC and DDBC transitions are used but it should be possible to have a single actuator without transitions described.

That makes sense! I've updated the specification.
Since the code is generated I won't use use the pull request.


Kulidai

In Ymesei's song "Alibasosoi," he has the line "Alibasosoi ia ra mei me debora chei / Kulidai  a bub  e kau mometech  a sang."  Yoichi Rengiil and Felix Okabe said that "Kulidai a bub" means "I wove a fish trap" (past tense of weave).  What is the root of that word (to weave):  Olidai?

Melidai: mengaus. Ongeuikl.
On Wed, Jul 10, 2024 at 14:42 JimGeselbracht @.***>
wrote:

In Ymesei's song "Alibasosoi," he has the line "Alibasosoi ia ra mei me
debora chei / Kulidai a bub e kau mometech a sang." Yoichi Rengiil and
Felix Okabe said that "Kulidai a bub" means "I wove a fish trap" (past
tense of weave). What is the root of that word (to weave): Olidai?
—
Reply to this email directly, view it on GitHub
https://github.com/tekinged/missing/issues/504, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AF2DJJC4FDLXJSF5R3WXLNLZLWTEZAVCNFSM6AAAAABKVXLHFOVHI2DSMVQWIX3LMV43ASLTON2WKOZSGQYDCNZZGE3DCOA
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>


Compiler error with private method that returns proc effect method

Acton Version
<IP_ADDRESS>21<IP_ADDRESS>
Steps to Reproduce
Compile:
actor Foo(on_done: action() -> None):
    var callbacks: list[proc() -> bool] = []

    def _push_cb(cb):
        callbacks.append(cb)

    def _head_cb():
        return callbacks[-1]

    def _do_stuff_cb():
        on_done()
        return True

    def poll():
        _head_cb()()

    _push_cb(_do_stuff_cb)

Expected Behavior
Compilation success!
Actual Behavior
Building project in /home/ddjoh/workspaces/act-test-1
Compiling mod1.act for release
actonc: No match in record selector rtail

Returning a proc effect method seems to be the issue even though the 'returner' is private. E.g. inlining _head_cb removes the compilation error:
actor Foo(on_done: action() -> None):
    var callbacks: list[proc() -> bool] = []

    def _push_cb(cb):
        callbacks.append(cb)

    def _do_stuff_cb():
        on_done()
        return True

    def poll():
        callbacks[-1]()

    _push_cb(_do_stuff_cb)

compiles fine.


Update to Craft CMS 3.6 Error

Declaration of flipbox\craft\domains\fields\Domains::inputHtml(flipbox\craft\domains\queries\DomainsQuery $query, bool $static) must be compatible with craft\base\Field::inputHtml($value, ?craft\base\ElementInterface $element = null): string

Looks like they might have added this in 3.5: https://github.com/craftcms/cms/blob/3.5.19/CHANGELOG.md#added-15. Do we just need to rename it to not conflict with the one they added?

Fixed in https://github.com/flipboxfactory/craft-domains/releases/tag/2.0.2


[Bug]: Interface slowdown after extensive usage

What happened?
After running for about 7 days interface will start slowing up.
As each day passes, it takes longer to load tabs like "Logs" under settings. Similarily, "Config" tab will take alot to load.
Eventually after 30ish days of uptime, web interface will not even load.
After restarting application, everything reverts to normal.
I use three Newtek vision mixers which i monitor for tally states simultaniously.
Has anyone had similar experience?
Version
3.0.10
Distribution
Desktop
OS
Windows
What browsers are you seeing the problem on?
Chrome
If applicable, What Listener Clients are You Using?
No response
TallyArbiter configuration
No response
Relevant log output
No response
Error stacktrace (if applicable)
No response

Yes I see this too. I think it’s the massive amount of tally data coming in and trying to be loaded.

Is there perhaps a way to turn off logging? That would make this applications operable 24/7 unattended.

Yep it’s open source so have at it

Ok fair enough.
Thanks!


Set utf8 character encoding for windows generate results steps

Summary
In #5988 we changed the character encoding used in windows CI jobs so
that it was explicitly using utf8 to avoid issues with characters being
emitted from the tests outside the default windows character encoding.
This was coming up because even though python is explicitly utf8 the
shell or the pipes used for interprocess communication in stestr did not
know we were using utf8 and would raise an encoder error when trying to
write out a string with utf8 characters. However, what that PR neglected
was the later step in the windows test jobs that we pipe the subunit
stream for the test run into a junitxml generator which would also
suffer from this problem. However we did not encounter this because only
the test id, runtime, and any failure or error messages are written out
on the process across the pipe so unless we were using characters
outside the default windows code page we wouldn't have any issues. But
as was recently discovered when a test error message included 'π' the
generate results step will fail on windows jobs. To address this edge
case this commit adds the same env variables and chcp command to the
generate results step of the windows test jobs so that if we have error
messages with utf8 characters in the output.
Details and comments

For an example of this failure see: https://dev.azure.com/qiskit-ci/qiskit-terra/_build/results?buildId=25494&view=logs&jobId=7a68dc58-ebf8-506c-1eea-de91a393131c&j=7a68dc58-ebf8-506c-1eea-de91a393131c&t=96673dbe-c4dc-582e-59b9-386ddf9eb8d7 (I didn't include it in the commit message because that link will be short lived when azure pipelines rotates the job logs).


:recycle: Drop support for ruby 1.9

Coverage remained the same at 68.0% when pulling cd566b13b234283f94bf71d1113f1ba9f8fbb636 on chore/drop-ruby-1-9 into 391307db508f29cde46957edf9baba38c7a2933e on master.


HADOOP-16269. ABFS: add listFileStatus with StartFrom.

Add support to list entries in a path from an entry name in lexical order(Azure Storage Service returns the entries in lexical order)
This support is added to AzureBlobFileSystemStore and won't be exposed to FS level api.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
33
Docker mode activated.

_ Prechecks _

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1
mvninstall
1036
trunk passed

+1
compile
33
trunk passed

+1
checkstyle
22
trunk passed

+1
mvnsite
34
trunk passed

+1
shadedclient
670
branch has no errors when building and testing our client artifacts.

+1
findbugs
38
trunk passed

+1
javadoc
20
trunk passed

_ Patch Compile Tests _

+1
mvninstall
25
the patch passed

+1
compile
23
the patch passed

+1
javac
23
the patch passed

-0
checkstyle
14
hadoop-tools/hadoop-azure: The patch generated 4 new + 2 unchanged - 0 fixed = 6 total (was 2)

+1
mvnsite
26
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
675
patch has no errors when building and testing our client artifacts.

+1
findbugs
46
the patch passed

+1
javadoc
17
the patch passed

_ Other Tests _

+1
unit
77
hadoop-azure in the patch passed.

+1
asflicense
23
The patch does not generate ASF License warnings.

2901

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-768/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/768

Optional Tests
dupname  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  shadedclient  findbugs  checkstyle

uname
Linux e1f8804a2a8c 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / a703dae

maven
version: Apache Maven 3.3.9

Default Java
1.8.0_191

findbugs
v3.1.0-RC1

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-768/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-768/1/testReport/

Max. process+thread count
445 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-768/1/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
31
Docker mode activated.

_ Prechecks _

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 3 new or modified test files.

_ trunk Compile Tests _

+1
mvninstall
1088
trunk passed

+1
compile
27
trunk passed

+1
checkstyle
18
trunk passed

+1
mvnsite
32
trunk passed

+1
shadedclient
694
branch has no errors when building and testing our client artifacts.

+1
findbugs
40
trunk passed

+1
javadoc
23
trunk passed

_ Patch Compile Tests _

+1
mvninstall
25
the patch passed

+1
compile
24
the patch passed

+1
javac
24
the patch passed

-0
checkstyle
13
hadoop-tools/hadoop-azure: The patch generated 4 new + 2 unchanged - 0 fixed = 6 total (was 2)

+1
mvnsite
28
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
746
patch has no errors when building and testing our client artifacts.

+1
findbugs
49
the patch passed

+1
javadoc
17
the patch passed

_ Other Tests _

+1
unit
86
hadoop-azure in the patch passed.

+1
asflicense
24
The patch does not generate ASF License warnings.

3045

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-768/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/768

Optional Tests
dupname  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  shadedclient  findbugs  checkstyle

uname
Linux 9e551bc6d3e3 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 3758270

maven
version: Apache Maven 3.3.9

Default Java
1.8.0_191

findbugs
v3.1.0-RC1

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-768/2/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-768/2/testReport/

Max. process+thread count
412 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-768/2/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
527
Docker mode activated.

_ Prechecks _

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 3 new or modified test files.

_ trunk Compile Tests _

+1
mvninstall
1145
trunk passed

+1
compile
35
trunk passed

+1
checkstyle
21
trunk passed

+1
mvnsite
38
trunk passed

+1
shadedclient
696
branch has no errors when building and testing our client artifacts.

+1
findbugs
47
trunk passed

+1
javadoc
21
trunk passed

_ Patch Compile Tests _

+1
mvninstall
27
the patch passed

+1
compile
24
the patch passed

+1
javac
24
the patch passed

-0
checkstyle
14
hadoop-tools/hadoop-azure: The patch generated 6 new + 2 unchanged - 0 fixed = 8 total (was 2)

+1
mvnsite
27
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

-1
shadedclient
109
patch has errors when building and testing our client artifacts.

+1
findbugs
45
the patch passed

+1
javadoc
17
the patch passed

_ Other Tests _

+1
unit
80
hadoop-azure in the patch passed.

+1
asflicense
23
The patch does not generate ASF License warnings.

2967

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-768/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/768

Optional Tests
dupname  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  shadedclient  findbugs  checkstyle

uname
Linux 53320c43a750 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 3758270

maven
version: Apache Maven 3.3.9

Default Java
1.8.0_191

findbugs
v3.1.0-RC1

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-768/3/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-768/3/testReport/

Max. process+thread count
411 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-768/3/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

Yetus is complaining about "shadedclient", but I tried the same command in my local env, there is no issue.
Any ideas?

No idea.
@aw-was-here -your thoughts?

Tested with my US-West account:
non-xns account:
Tests run: 40, Failures: 0, Errors: 0, Skipped: 0
Tests run: 342, Failures: 0, Errors: 0, Skipped: 207
Tests run: 190, Failures: 0, Errors: 0, Skipped: 15
xns-account:
Tests run: 40, Failures: 0, Errors: 0, Skipped: 0
Tests run: 342, Failures: 0, Errors: 0, Skipped: 21
Tests run: 190, Failures: 0, Errors: 0, Skipped: 15

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
30
Docker mode activated.

_ Prechecks _

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 3 new or modified test files.

_ trunk Compile Tests _

+1
mvninstall
1171
trunk passed

+1
compile
28
trunk passed

+1
checkstyle
22
trunk passed

+1
mvnsite
30
trunk passed

+1
shadedclient
685
branch has no errors when building and testing our client artifacts.

+1
findbugs
42
trunk passed

+1
javadoc
20
trunk passed

_ Patch Compile Tests _

+1
mvninstall
28
the patch passed

+1
compile
23
the patch passed

+1
javac
23
the patch passed

-0
checkstyle
15
hadoop-tools/hadoop-azure: The patch generated 5 new + 2 unchanged - 0 fixed = 7 total (was 2)

+1
mvnsite
29
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
736
patch has no errors when building and testing our client artifacts.

+1
findbugs
45
the patch passed

+1
javadoc
18
the patch passed

_ Other Tests _

+1
unit
80
hadoop-azure in the patch passed.

+1
asflicense
26
The patch does not generate ASF License warnings.

3113

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-768/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/768

Optional Tests
dupname  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  shadedclient  findbugs  checkstyle

uname
Linux 9265f7433f05 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / dead9b4

maven
version: Apache Maven 3.3.9

Default Java
1.8.0_191

findbugs
v3.1.0-RC1

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-768/4/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-768/4/testReport/

Max. process+thread count
410 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-768/4/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

This looks pretty good overall. I think we need some documentation and proposed use case (who will call this method). Also a proposal how we integrate this with the greater FileSystem APIs would be good. We generally want to keep the FS implementations on the same API or they can diverge over time and create a mess. Depending on the greater API discussion, the listStatus method probably needs some Private Unstable annotations.

@ajfabbri Because it relies on Azure service to return results in lexical order, so I am not sure if this need to be integrated to FileSystem API. I will add some private Unstable annotations for now.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
30
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 3 new or modified test files.

_ trunk Compile Tests _

+1
mvninstall
1047
trunk passed

+1
compile
27
trunk passed

+1
checkstyle
18
trunk passed

+1
mvnsite
34
trunk passed

+1
shadedclient
177
branch has no errors when building and testing our client artifacts.

+1
javadoc
24
trunk passed

0
spotbugs
54
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
52
trunk passed

_ Patch Compile Tests _

+1
mvninstall
25
the patch passed

+1
compile
24
the patch passed

+1
javac
24
the patch passed

-0
checkstyle
15
hadoop-tools/hadoop-azure: The patch generated 4 new + 2 unchanged - 0 fixed = 6 total (was 2)

+1
mvnsite
29
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
746
patch has no errors when building and testing our client artifacts.

+1
javadoc
21
the patch passed

+1
findbugs
55
the patch passed

_ Other Tests _

+1
unit
81
hadoop-azure in the patch passed.

+1
asflicense
31
The patch does not generate ASF License warnings.

2593

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-768/5/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/768

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux 7924d0c9a0e0 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / d6b7609

Default Java
1.8.0_191

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-768/5/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-768/5/testReport/

Max. process+thread count
411 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-768/5/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

Can anyone help to review or commit this PR?

+1, committed.
thanks
I'm going to leave a warning note on the JIRA about how these internal things are brittle and may go away.


Scan for single commits seems to be broken

Describe the bug
I'm trying to scan single commits. For my understanding that means that only in that commit involved files (added and modified ones) should be scanned. Nothing more. Well, exactly the opposite happens.
To Reproduce
I'll use a test repository to demonstrate:
$ git clone https://github.com/dorneanu/test
$ cd test
$ git log --pretty=format:'%H' -n 5
c787e5a0009184147081c20976f743cac38673f4
eb2fdd2d98dfa4db48b5a71a8515897c590467bc
0434334a161f0ce86ec32549a6c5b1b8525b3efe
2b784a96334bc7bfa5a713eecf331d64fb73eb50
b7c5d1a1a66edc2e4faae1df68984722d3042a18

Let's have a look at the latest commit and which files have been added or modified:
$ git diff-tree --no-commit-id --name-only c787e5a0009184147081c20976f743cac38673f4
test03.md

When I run gitleaks with --commit option this is what I get:
$ gitleaks --commit=c787e5a0009184147081c20976f743cac38673f4 --repo-path . --verbose

{"line":"  \"line\": \"ASIA7QA4AFF6ZZSW4GU7\",","offender":"ASIA7QA4AFF6ZZSW4GU7","commit":"c787e5a0009184147081c20976f743cac38673f4","repo":".","rule":"AWS Manager ID","commitMessage":"update2\n","author":"Victor Dorneanu","email":"xxxxxxxxxx","file":"bla.json","date":"2020-01-22T17:40:01+01:00","tags":"key, AWS"}
{"line":"  \"line\": \"-----BEGIN PGP PRIVATE KEY BLOCK-----\",","offender":"-----BEGIN PGP PRIVATE KEY BLOCK-----","commit":"c787e5a0009184147081c20976f743cac38673f4","repo":".","rule":"Asymmetric Private Key","commitMessage":"update2\n","author":"Victor Dorneanu","email":"xxxxxxxxxx","file":"bla.json","date":"2020-01-22T17:40:01+01:00","tags":"key, AsymmetricPrivateKey"}
{"line":"-----BEGIN PGP PRIVATE KEY BLOCK-----","offender":"-----BEGIN PGP PRIVATE KEY BLOCK-----","commit":"c787e5a0009184147081c20976f743cac38673f4","repo":".","rule":"Asymmetric Private Key","commitMessage":"update2\n","author":"Victor Dorneanu","email":"xxxxxxxxxx","file":"cert.cer","date":"2020-01-22T17:40:01+01:00","tags":"key, AsymmetricPrivateKey"}
{"line":"-----BEGIN PGP PRIVATE KEY BLOCK-----","offender":"-----BEGIN PGP PRIVATE KEY BLOCK-----","commit":"c787e5a0009184147081c20976f743cac38673f4","repo":".","rule":"Asymmetric Private Key","commitMessage":"update2\n","author":"Victor Dorneanu","email":"xxxxxxxxxx","file":"cert2.cer","date":"2020-01-22T17:40:01+01:00","tags":"key, AsymmetricPrivateKey"}
{"line":"ASIA7QA4AFF6ZZSW4GU7","offender":"ASIA7QA4AFF6ZZSW4GU7","commit":"c787e5a0009184147081c20976f743cac38673f4","repo":".","rule":"AWS Manager ID","commitMessage":"update2\n","author":"Victor Dorneanu","email":"xxxxxxxxxx","file":"folder1/aws.conf","date":"2020-01-22T17:40:01+01:00","tags":"key, AWS"}
{"line":"ASIA7QA4AFF6ZZSW4GU7","offender":"ASIA7QA4AFF6ZZSW4GU7","commit":"c787e5a0009184147081c20976f743cac38673f4","repo":".","rule":"AWS Manager ID","commitMessage":"update2\n","author":"Victor Dorneanu","email":"xxxxxxxxxx","file":"neu.java","date":"2020-01-22T17:40:01+01:00","tags":"key, AWS"}
{"line":"-----BEGIN PGP PRIVATE KEY BLOCK-----","offender":"-----BEGIN PGP PRIVATE KEY BLOCK-----","commit":"c787e5a0009184147081c20976f743cac38673f4","repo":".","rule":"Asymmetric Private Key","commitMessage":"update2\n","author":"Victor Dorneanu","email":"xxxxxxxxxx","file":"test.pem","date":"2020-01-22T17:40:01+01:00","tags":"key, AsymmetricPrivateKey"}
{"line":"ASIA7QA4AFF6ZZSW4GU7","offender":"ASIA7QA4AFF6ZZSW4GU7","commit":"c787e5a0009184147081c20976f743cac38673f4","repo":".","rule":"AWS Manager ID","commitMessage":"update2\n","author":"Victor Dorneanu","email":"xxxxxxxxxx","file":"test01.md","date":"2020-01-22T17:40:01+01:00","tags":"key, AWS"}
{"line":"ASIA7QA4AFF6ZZSW4GU7","offender":"ASIA7QA4AFF6ZZSW4GU7","commit":"c787e5a0009184147081c20976f743cac38673f4","repo":".","rule":"AWS Manager ID","commitMessage":"update2\n","author":"Victor Dorneanu","email":"xxxxxxxxxx","file":"test02.md","date":"2020-01-22T17:40:01+01:00","tags":"key, AWS"}
WARN[2020-01-24T17:00:08+01:00] 9 leaks detected. 0 commits audited in 0 seconds%

As you can notice, multiple files were scanned and not only test03.md.
Expected behavior
I would have expected to get results only related to test03.md (the file modified during the commit).
Last year I've submitted a PR (Implemented right way to scan for files in a commit (by using --commit) #236) regarding this behaviour which was also accepted.  I don't know what happened since then.
Screenshots
If applicable, add screenshots to help explain your problem.
Basic Info (please complete the following information):

OS: Linux arch64 5.4.3-arch1-1 #1 SMP PREEMPT Fri, 13 Dec 2019 09:39:02 +0000 x86_64 GNU/Linux
Gitleaks Version: v3.1.0

Additional context
Add any other context about the problem here.
cc @zricethezav

Thanks for the quick fix!


Count of emitted warnings/errors

It would be really nice to to have seelog count the number of Log messages it got for loglevels with small throughput (warn, etc.) so that a main program can set an descriptive exit status based on that.
Currently, you have to do that by yourself incrementing a variable each time you call log.Warn which is not that elegant...

Hi bong0!
I'll think about the implementation

@goodsign :+1: thank you!

@goodsign this is also good for testing, since e.g. goroutines that just output log Warnings cannot be easily observed when you want to test them for bad input handling. With an incrementing counter you could compare those before and after a call.

What do you think we should do, if a log level is unused (disabled for all dispatchers): do we increment the counter for this log level or not?
I think we should, so that the counter holds the value of 'log attempts', despite the fact that some log messages may not appear because of current config.

So, what do you think about 'log attempts' vs 'filtered messages'? I'm going to implement calculation of log attempts, e.g. number of log.Error / Errorf calls, even if the minlevel if set to off, so that you get info about number of errors even if current log config filters them out. What do you think?

@goodsign Sorry for me taking long to answer. Yes I would also count the log attemts. I would expect every call of log.LEVEL results in incrementing. This is independent of the setting whether we write the entry to a file or so. Yes this introduces one more line which is always executed but I think an increment is not that much overhead.


Search result not showing

Search result not showing, how to fix this? i try on localhost. please any documentation or guide ?
i love this theme......

Thanks for using this theme.
I also tried it and it is showing up fine.
can you tell me how I can reproduce it?

ok.. search working.. by adding in config.toml
[outputs]   home = ['HTML', 'JSON']


Sortable entries items missing the "duplicate" action

Thanks for the contribution but this seems to be fixed already


Bug 1904507: Add metrics for Datastore types

NFS datastore type may be deprecated upstream and we want to know how many clusters use them.
(see https://vsphere-csi-driver.sigs.k8s.io/compatiblity_matrix.html)
The metric is collected in CheckStorageClasses(), therefore the default data store must be checked there if a StorageClass does not refer to a specific one. Extra care is made to collect each DS only once.
The simulated vCenter used for unit tests has all datastores with type OTHER. I did not change that to make sure the files can be re-generated later without manual changes.
Example:
# HELP vsphere_datastore_total [ALPHA] Number of DataStores used by the cluster.
# TYPE vsphere_datastore_total gauge
vsphere_datastore_total{type="vsan"} 2

cc @gnufied

/bugzilla refresh

/lgtm

/retest

/override ci/prow/e2e-vsphere
Known DNS bug: https://bugzilla.redhat.com/show_bug.cgi?id=1963912


bearer token instead of cookies

I'd like to have bearer token in browser instead of cookies which will be used further to access backend.
Please share where this is possible

Use secure-cookie: false to turn of cookie, then it uses bearer tokens

Does this work for keycloak gatekeeper ver 11?


Add spec structs

Second try for #290 hopefully fixing #291.
I haven't reviewed all of the original changes yet but also haven't seen any problems so far.
(I've manually set the author of the original changes to @aweinstock314 with a separate commit as I wasn't able to rebase.)

Looks good to me (assuming 614f55490e057ad67ae0cc241b3df604e7b633d6 is a revert of 77294fd3ce4d9657a31ea07894ca6fb3e83c2174). I probably mishandled the rebase of #288's commits for #290.
Would you be willing to add unit tests for the zip64 support (not necessarily as part of this PR)?

Yes, it's supposed to revert the revert. I'm planning to add an exhaustive ZIP64 test and some comments to the code (not necessarily as part of this PR). Not sure when, maybe next weekend.

Btw, I ran into exactly this as well, so I mention it. This screwed up the header and resulted in CRC-32 mismatches:
let mut zip64_extra_field = vec![0; 20];
write_local_zip64_extra_field(&mut zip64_extra_field, file)?;

It doesn't overwrite the 20 bytes of zeros as intended but instead extends them by the extra field. This would do the trick:
let mut zip64_extra_field = vec![0; 20];
write_local_zip64_extra_field(&mut &mut zip64_extra_field[..], file)?;

It's impl Write for Vec<u8> vs impl Write for &mut [u8]. Former extends, latter overwrites.

These structures are not going to be exposed publicly as-is, and I see no need for the changes in the current version. Thanks for the work you've done on this, but we will make these changes as and when we make the API public, so that we can properly motivate the design of them.
(If you think I'm missing something feel free to say ^^)

Even if they're not exposed yet, having the specification structures separated out makes the code clearer (currently, the code mixes concerns between reading/writing the structures as they appear in the file and casting them into an intermediate representation that supports deferred lookups and writes in a way that's agnostic between a few different equivalent ways to represent the same metadata (e.g. data descriptors)), and adds unit tests for the specification structures themselves.
These structures suffice for adding custom metadata via an orphaned LFH (e.g. https://github.com/aweinstock314/zip/blob/8acc05175b11aa372bc5f11d5351e3ce0885c50b/examples/reorder_cd.rs, which makes retrieving the central directory via HTTP range requests more efficient, since some servers give worse response times for range requests with offsets from the end of a file), and I'm confident that they'll generalize to crafting other sorts of files (e.g. overlapping LFHs with file contents to write unit tests for #228).
I'd like these to be exposed as part of the proposed zip-format crate, and would like them to be merged as private types prior to that to avoid needing to rebase around other conflicting changes to reading/writing these structures.

This was and still is a good idea, whoops! The definitions look great, and I'll bring them in as unstable API ASAP

Replaced with https://github.com/zip-rs/zip2/pull/71.


No tensorflow-rocm on pypi for Python 3.11

Issue Type
Build/Install
Have you reproduced the bug with TF nightly?
No
Source
binary
Tensorflow Version
n/a, not installable
Custom Code
Yes
OS Platform and Distribution
Debian 12 (bookworm)
Mobile device
No response
Python version
3.11
Bazel version
No response
GCC/Compiler version
No response
CUDA/cuDNN version
No response
GPU model and memory
No response
Current Behaviour?
I have a Stable Diffusion venv with Python 3.11 and no easy way to downgrade to Python 3.10 on my current Debian stable deployment.
Within venv, running the following errors out:
$ pip install tensorflow-rocm
ERROR: Could not find a version that satisfies the requirement tensorflow-rocm (from versions: none)
ERROR: No matching distribution found for tensorflow-rocm

Looking at the published whls on http://pypi.org/project/tensorflow-rocm/#files I can see there's the following items:

Built Distributions

tensorflow_rocm-<IP_ADDRESS>0-cp310-cp310-manylinux2014_x86_64.whl (456.9 MB view hashes)
Uploaded May 10, 2023 cp310

tensorflow_rocm-<IP_ADDRESS>0-cp39-cp39-manylinux2014_x86_64.whl (456.9 MB view hashes)
Uploaded May 10, 2023 cp39

tensorflow_rocm-<IP_ADDRESS>0-cp38-cp38-manylinux2014_x86_64.whl (456.8 MB view hashes)
Uploaded May 10, 2023 cp38

My current guess is that this is because only 3.8, 3.9 and 3.10 versions are published (and notably there's also no 2.12).
Standalone code to reproduce the issue
n/a

Relevant log output
No response

@ivucica tensorflow-rocm is available for python 3.11 since 10 July 2023 as you can see on PyPI.
I think that this issue can now be closed.

I’ve been using it for a while now. This can be closed, indeed.