How do I turn on logging using Azure CLI for Key Vault?

Documentation pushed me towards PowerShell. :-(

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 199fbc30-6751-5ee6-4fbb-f23f2f5137cc
Version Independent ID: 00bcc33c-c14b-fa5f-74ab-f1bd635dc043
Content: Best Practices to use Key Vault - Azure Key Vault
Content Source: articles/key-vault/general/best-practices.md
Service: key-vault
Sub-service: general
GitHub Login: @msmbaldwin
Microsoft Alias: mbaldwin

@awentzel Thank you for your feedback . We will have this reviewed with the content team and update you .

@msmbaldwin Could this be reviewed ?

Added section in Azure logging document (linked in this document) for an Azure CLI tutorial.
PR submitted, changes will be publicly visible within 48 hrs.
Resolving issue

#please-close


[GLUTEN-6887][VL] Daily Update Velox Version (2024_08_19)

Upstream Velox's New Commits:
54b16e041 by Ke Wang, Refactor WriterOptions processConfigs function (10762)
2b7f559a4 by Pedro Eugenio Rocha Pedreira, Fix std::shared_mutex missing includes (10772)
6b9806069 by xiaoxmeng, Fix flaky FileInputStreamTest.stats (10771)
7fd6273e5 by Chengcheng Jin, Move decimal-to-string API to DecimalUtil (10477)
d23dde5aa by Wei He, Avoid generating unsupported data types for JoinFuzzer with PQR (10662)
d805d31a0 by Jialiang Tan, Apply grow target adjustment to reduce arbitration count (10170)
e7ffae924 by xiaoxmeng, Remove byte input stream legacy code (10761)
44fb107d0 by Wei He, Use ASSERT_STREQ when comparing C-strings (10759)
c701f8308 by Satadru Pan, Match Velox hmac_sha/md5 functions behavior with Java on empty keys (10744)
d08d69469 by xiaoxmeng, Some window operator code cleanup (10764)
bc274f2a8 by Bikramjeet Vig, Ensure flattening an array or map does not result in an oversized values vector (10760)

@GlutenPerfBot benchmark

ACK, will benchmark TPCH/DS on this pull request

#6887

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query
log/native_6910_time.csv
log/native_master_08_18_2024_9fcd4885c7_time.csv
difference
percentage

q1
39.42
40.01
0.589
101.50%

q2
30.56
28.03
-2.535
91.71%

q3
53.89
53.67
-0.219
99.59%

q4
40.88
42.78
1.897
104.64%

q5
108.87
104.21
-4.666
95.71%

q6
11.60
11.23
-0.365
96.85%

q7
117.15
115.52
-1.630
98.61%

q8
114.69
116.72
2.024
101.76%

q9
169.01
168.24
-0.771
99.54%

q10
64.89
66.68
1.795
102.77%

q11
27.76
30.17
2.405
108.66%

q12
33.00
31.20
-1.796
94.56%

q13
51.40
51.65
0.253
100.49%

q14
24.35
26.30
1.947
107.99%

q15
55.54
54.84
-0.700
98.74%

q16
19.87
19.79
-0.081
99.59%

q17
133.11
131.24
-1.867
98.60%

q18
197.98
197.27
-0.706
99.64%

q19
26.20
25.90
-0.301
98.85%

q20
41.59
40.46
-1.127
97.29%

q21
387.36
386.69
-0.666
99.83%

q22
15.60
15.31
-0.288
98.16%

total
1764.74
1757.93
-6.807
99.61%

note: there's one pending fix not included
https://github.com/facebookincubator/velox/pull/10768


Redux-devtools-extension integration

Hi,
I forked react-waterfall-example and started implementing an integration with redux-devtools-extension.
It works very well for viewing, I havn't got the yet to go over the time travel, so I combined your devTool with mine.
Have a see:
https://github.com/elisherer/react-waterfall-example/blob/feature/redux-devtools-extension/src/reduxDevTool.js

UPDATE:
I got time travel working...

Also got the dispatcher working as well, even passing arguments:
For example: "this.increment", "increment" & "increment(23)" will all work (and the last will actually pass the argument)
A thing that I think we should have is logging the action's arguments and those are not shared with the middleware (PR maybe?)

Yes it could be great to have it !

I think we can close this ;)


Inference with replace_method='auto' fails for gpt2-xl generation

I modified the code at "End-to-End GPT NEO 2.7B Inference" by replacing EleutherAI/gpt-neo-2.7B with gpt2-xl, and I get the following stack trace when running on 2 GPUs:
[2021-06-28 13:56:01,937] [ERROR] Uncaught Exception
Traceback (most recent call last):
  File "gpt2-generation.py", line 20, in <module>
    string = generator("DeepSpeed is", do_sample=True, min_length=50)
  File "venv/lib/python3.7/site-packages/transformers/pipelines/text_generation.py", line 148, in __call__
    output_sequences = self.model.generate(input_ids=input_ids, **generate_kwargs)  # BS x SL
  File "venv/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "venv/lib/python3.7/site-packages/transformers/generation_utils.py", line 1026, in generate
    **model_kwargs,
  File "venv/lib/python3.7/site-packages/transformers/generation_utils.py", line 1535, in sample
    output_hidden_states=output_hidden_states,
  File "venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "venv/lib/python3.7/site-packages/deepspeed/inference/engine.py", line 229, in forward
    return self.model_orig_fwd(*inputs, **kwargs)
  File "venv/lib/python3.7/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 956, in forward
    return_dict=return_dict,
  File "venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "venv/lib/python3.7/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 799, in forward
    output_attentions=output_attentions,
  File "venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "venv/lib/python3.7/site-packages/deepspeed/ops/transformer/inference/transformer_inference.py", line 611, in forward
    self.norm_b)
  File "venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "venv/lib/python3.7/site-packages/deepspeed/ops/transformer/inference/transformer_inference.py", line 393, in forward
    self.qkv_merging)
  File "venv/lib/python3.7/site-packages/deepspeed/ops/transformer/inference/transformer_inference.py", line 308, in forward
    output, key_layer, value_layer, context_layer = selfAttention_fp()
  File "venv/lib/python3.7/site-packages/deepspeed/ops/transformer/inference/transformer_inference.py", line 273, in selfAttention_fp
    context_layer, key_layer, value_layer = compute_attention(qkv_out)
  File "venv/lib/python3.7/site-packages/deepspeed/ops/transformer/inference/transformer_inference.py", line 207, in compute_attention
    mixed_query = _transpose_for_scores(mixed_query, False, True)
  File "venv/lib/python3.7/site-packages/deepspeed/ops/transformer/inference/transformer_inference.py", line 148, in _transpose_for_scores
    x_1 = x.view(*new_x_shape)
RuntimeError: shape '[1, 3, 12, 66]' is invalid for input of size 2400

The code works without issues when using the default EleutherAI/gpt-neo-2.7B, and it also runs successfully with gpt2, gpt2-medium, and gpt2-large. Out of the GPT-2 models, only gpt2-xl fails. Setting replace_method to a value that is not 'auto' (such as 'foo') allows it to work, but the DeepSpeed optimizations are not loaded.
This code was run with Python 3.7.9, torch==1.8.1+cu102, transformers==4.8.1, and deepspeed==0.4.2+bc019a5 (compiled using DS_BUILD_OPS=1 and CUDA 10.2.89).
I edited the stack trace to remove the parts of the path outside the virtual environment.

Hi @aphedges
Thanks for trying all these.
I will take a look at this. By the way, we have another test script for this model architecture. I wonder if you see the same issue for that too.
Thanks,
Reza

@RezaYazdaniAminabadi, thanks for looking into this!
The test script you linked worked without issues for 1 and 2 GPUs. However, it seems that removing the --fp16 line from the script causes the following error when using 1 GPU:
Traceback (most recent call last):
  File "run_generation.py", line 349, in <module>
    main()
  File "run_generation.py", line 316, in main
    num_return_sequences=args.num_return_sequences,
  File "venv/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "venv/lib/python3.7/site-packages/transformers/generation_utils.py", line 1026, in generate
    **model_kwargs,
  File "venv/lib/python3.7/site-packages/transformers/generation_utils.py", line 1535, in sample
    output_hidden_states=output_hidden_states,
  File "venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "venv/lib/python3.7/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 965, in forward
    lm_logits = self.lm_head(hidden_states)
  File "venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "venv/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 94, in forward
    return F.linear(input, self.weight, self.bias)
  File "venv/lib/python3.7/site-packages/torch/nn/functional.py", line 1753, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)`

I also got the above error when I ran the code for my initial comment on only 1 GPU, but I didn't mention it yesterday.
I get a different (third) error when running on 2 GPUs:
!!!! kernel execution error. (m: 1600, n: 6, k: 6400, error: 13)
!!!! kernel execution error. (m: 4800, n: 6, k: 1600, error: 13)
Traceback (most recent call last):
  File "run_generation.py", line 349, in <module>
    main()
  File "run_generation.py", line 316, in main
    num_return_sequences=args.num_return_sequences,
  File "venv/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "venv/lib/python3.7/site-packages/transformers/generation_utils.py", line 1026, in generate
    **model_kwargs,
  File "venv/lib/python3.7/site-packages/transformers/generation_utils.py", line 1535, in sample
    output_hidden_states=output_hidden_states,
  File "venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "venv/lib/python3.7/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 956, in forward
    return_dict=return_dict,
  File "venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "venv/lib/python3.7/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 799, in forward
    output_attentions=output_attentions,
  File "venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "venv/lib/python3.7/site-packages/deepspeed/ops/transformer/inference/transformer_inference.py", line 611, in forward
    self.norm_b)
  File "venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "venv/lib/python3.7/site-packages/deepspeed/ops/transformer/inference/transformer_inference.py", line 393, in forward
    self.qkv_merging)
  File "venv/lib/python3.7/site-packages/deepspeed/ops/transformer/inference/transformer_inference.py", line 308, in forward
    output, key_layer, value_layer, context_layer = selfAttention_fp()
  File "venv/lib/python3.7/site-packages/deepspeed/ops/transformer/inference/transformer_inference.py", line 273, in selfAttention_fp
    context_layer, key_layer, value_layer = compute_attention(qkv_out)
  File "venv/lib/python3.7/site-packages/deepspeed/ops/transformer/inference/transformer_inference.py", line 207, in compute_attention
    mixed_query = _transpose_for_scores(mixed_query, False, True)
  File "venv/lib/python3.7/site-packages/deepspeed/ops/transformer/inference/transformer_inference.py", line 154, in _transpose_for_scores
    return x_1.reshape(x.shape)
RuntimeError: CUDA error: an illegal memory access was encountered

I am able to prevent the both errors with the following modifications:
--- /dev/fd/63	2021-06-29 12:44:58.984909630 -0700
+++ run_generation.py	2021-06-29 12:42:08.976235560 -0700
@@ -261,7 +261,7 @@
         model = deepspeed.init_inference(model,
                                          mp_size=1,
                                          dtype=(torch.half if args.fp16 else torch.float),
-                                         injection_policy=injection_policy)
+                                         injection_policy={},replace_method='')
         model = model.module

     args.length = adjust_length_to_model(args.length, max_sequence_length=model.config.max_position_embeddings)

This is consistent with the behavior I noted in my initial comment.
If it matters, I've been running these experiments on 1 and 2 Nvidia Quadro RTX 8000 cards with driver version 440.44.

I forgot to check, but I can confirm that these errors also go away when I switch to gpt2-large.

Hi @aphedges
I think this PR fixes the issue.
Thanks,
Reza

@RezaYazdaniAminabadi, I installed 65f1fe8, and it unfortunately does not fix my issue.

Can you please share your script with me to try on my side?
Thanks.

I tried 7cb65cc, and that also didn't fix it.
Here is my script:
# Filename: gpt2-generation.py
import os
import deepspeed
import torch
import transformers
from transformers import pipeline

local_rank = int(os.getenv('LOCAL_RANK', '0'))
world_size = int(os.getenv('WORLD_SIZE', '1'))
generator = pipeline('text-generation', model='gpt2-xl', device=local_rank, model_kwargs={'cache_dir': 'gpt2_component/.model_cache'})

generator.model = deepspeed.init_inference(generator.model,
                                           mp_size=world_size,
                                           dtype=torch.float,
                                           replace_method='auto')

string = generator("DeepSpeed is", do_sample=True, min_length=50)
if torch.distributed.get_rank() == 0:
    print(string)

The only changes from the version in the tutorial is that it uses gpt2-xl and that I am using a local model cache directory.
I ran it with deepspeed --num_gpus 2 gpt2-generation.py. Here is the stack trace (given that line numbers are different after your changes; also seems to be some interleaving from the two processes):
Traceback (most recent call last):
  File "gpt2-generation.py", line 19, in <module>
    string = generator("DeepSpeed is", do_sample=True, min_length=50)
  File "venv/lib/python3.7/site-packages/transformers/pipelines/text_generation.py", line 148, in __call__
Traceback (most recent call last):
  File "gpt2-generation.py", line 19, in <module>
    string = generator("DeepSpeed is", do_sample=True, min_length=50)
  File "venv/lib/python3.7/site-packages/transformers/pipelines/text_generation.py", line 148, in __call__
    output_sequences = self.model.generate(input_ids=input_ids, **generate_kwargs)  # BS x SL
  File "venv/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    output_sequences = self.model.generate(input_ids=input_ids, **generate_kwargs)  # BS x SL
  File "venv/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "venv/lib/python3.7/site-packages/transformers/generation_utils.py", line 1026, in generate
    return func(*args, **kwargs)
  File "venv/lib/python3.7/site-packages/transformers/generation_utils.py", line 1026, in generate
    **model_kwargs,
  File "venv/lib/python3.7/site-packages/transformers/generation_utils.py", line 1535, in sample
    **model_kwargs,
  File "venv/lib/python3.7/site-packages/transformers/generation_utils.py", line 1535, in sample
    output_hidden_states=output_hidden_states,
  File "venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    output_hidden_states=output_hidden_states,
  File "venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "venv/lib/python3.7/site-packages/deepspeed/inference/engine.py", line 229, in forward
    result = self.forward(*input, **kwargs)
  File "venv/lib/python3.7/site-packages/deepspeed/inference/engine.py", line 229, in forward
    return self.model_orig_fwd(*inputs, **kwargs)
  File "venv/lib/python3.7/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 956, in forward
    return self.model_orig_fwd(*inputs, **kwargs)
  File "venv/lib/python3.7/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 956, in forward
    return_dict=return_dict,
  File "venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    return_dict=return_dict,
  File "venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "venv/lib/python3.7/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 799, in forward
    result = self.forward(*input, **kwargs)
  File "venv/lib/python3.7/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 799, in forward
    output_attentions=output_attentions,
  File "venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    output_attentions=output_attentions,
  File "venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "venv/lib/python3.7/site-packages/deepspeed/ops/transformer/inference/transformer_inference.py", line 615, in forward
    result = self.forward(*input, **kwargs)
  File "venv/lib/python3.7/site-packages/deepspeed/ops/transformer/inference/transformer_inference.py", line 615, in forward
    self.norm_b)
  File "venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    self.norm_b)
  File "venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "venv/lib/python3.7/site-packages/deepspeed/ops/transformer/inference/transformer_inference.py", line 397, in forward
    result = self.forward(*input, **kwargs)
  File "venv/lib/python3.7/site-packages/deepspeed/ops/transformer/inference/transformer_inference.py", line 397, in forward
    self.qkv_merging)
  File "venv/lib/python3.7/site-packages/deepspeed/ops/transformer/inference/transformer_inference.py", line 312, in forward
    self.qkv_merging)
output, key_layer, value_layer, context_layer = selfAttention_fp()
  File "venv/lib/python3.7/site-packages/deepspeed/ops/transformer/inference/transformer_inference.py", line 312, in forward

  File "venv/lib/python3.7/site-packages/deepspeed/ops/transformer/inference/transformer_inference.py", line 277, in selfAttention_fp
        output, key_layer, value_layer, context_layer = selfAttention_fp()
context_layer, key_layer, value_layer = compute_attention(qkv_out, input_mask)
  File "venv/lib/python3.7/site-packages/deepspeed/ops/transformer/inference/transformer_inference.py", line 277, in selfAttention_fp

  File "venv/lib/python3.7/site-packages/deepspeed/ops/transformer/inference/transformer_inference.py", line 203, in compute_attention
        context_layer, key_layer, value_layer = compute_attention(qkv_out, input_mask)mixed_query = _transpose_for_scores(mixed_query, False, True)
  File "venv/lib/python3.7/site-packages/deepspeed/ops/transformer/inference/transformer_inference.py", line 144, in _transpose_for_scores

  File "venv/lib/python3.7/site-packages/deepspeed/ops/transformer/inference/transformer_inference.py", line 203, in compute_attention
    x_1 = x.view(*new_x_shape)
RuntimeError: shape '[1, 3, 12, 66]' is invalid for input of size 2400
    mixed_query = _transpose_for_scores(mixed_query, False, True)
  File "venv/lib/python3.7/site-packages/deepspeed/ops/transformer/inference/transformer_inference.py", line 144, in _transpose_for_scores
    x_1 = x.view(*new_x_shape)
RuntimeError: shape '[1, 3, 12, 66]' is invalid for input of size 2400

Can you please send me the full log of your experiment?
thanks

Here is the complete log: log_2021_07_07_17_39_28.txt. I used the same command as in my previous comment. I hope it helps.

Hi @aphedges
Sorry for the late reply on this thread.
I now see the issue with this model. The problem is that the number of heads is 25 and is not dividable by 2. So, we cannot parallelize it across 2 GPUs. In this case, the smallest model-parallel degree has to be 5, otherwise the kind of parallelism we support cannot work.
I can put an assert to clarify this issue more properly for the inference API.
Thanks,
Reza

Should this issue be closed? An assert (or similar check) would be very useful here by providing a more clear error message, but the linked PR doesn't seem to add one.

Yes, I will do that in a separate PR.
Thanks @aphedges

@RezaYazdaniAminabadi, thanks for figuring this out!
I did not know that model-parallelism had such restrictions. I guess I just can't use gpt2-xl with DeepSpeed on a single 4-GPU machine, then.


SL Micro 6.0 tools for SUSE Manager

What does this PR change?
Use different tools for Uyuni and SUSE Manager for SL Micro 6.0.

Maybe just a question, why are you creating this is_uyuni varaible and not just doing this ?
%{ if product_version contains "uyuni" }

or
 if [[ $product_version =~ uyuni ]]; then

Maybe just one question, why are you creating is_uyuni variable and not just doing this ?
%{ if product_version contains "uyuni" }

I do not see a "contains" operator in https://developer.hashicorp.com/terraform/language/expressions/operators

or
 if [[ $product_version =~ uyuni ]]; then

The code above is shell, and product_version is a terraform variable.
You COULD pass the terraform variable to shell by using this
  if [[ "${{  product_version }}" =~ "uyuni" ]]; then

but this would get expanded to e.g.
  if [[ "4.3-stable" =~ "uyuni" ]]; then

which is not really elegant.
More discussions about my attempts here: https://suse.slack.com/archives/C02D12TNYLS/p1723642166193729

Tested to work on suma-05 with both SUSE Manager and Uyuni products.

I can confirm the new changes work:
uyuni-min-sles15:~ # zypper lr --uri
Repository priorities are without effect. All enabled repositories share the same priority.

# | Alias              | Name               | Enabled | GPG Check | Refresh | URI
--+--------------------+--------------------+---------+-----------+---------+-------------------------------------------------------------------------------------------------------------------
1 | ca_suse            | ca_suse            | No      | ----      | ----    | http://download.opensuse.org/repositories/SUSE:/CA/15.5/
2 | client_tools_repo  | client_tools_repo  | No      | ----      | ----    | http://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Master:/SLMicro6-Uyuni-Client-Tools/SL-Micro6/
3 | os_pool_repo       | os_pool_repo       | No      | ----      | ----    | http://download.suse.de/ibs/SUSE/Products/SL-Micro/6.0/x86_64/product/
4 | test_repo_rpm_pool | test_repo_rpm_pool | No      | ----      | ----    | http://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Test-Packages:/Pool/rpm/
uyuni-min-sles15:~ #

uyuni-min-sles15:~ # salt-call --local grains.item product_version
/usr/lib/python3.11/site-packages/salt/utils/http.py:8: DeprecationWarning: 'cgi' is deprecated and slated for removal in Python 3.13
  import cgi
/usr/lib/python3.11/site-packages/salt/utils/jinja.py:9: DeprecationWarning: 'pipes' is deprecated and slated for removal in Python 3.13
  import pipes
/usr/lib/python3.11/site-packages/salt/grains/core.py:2711: DeprecationWarning: 'locale.getdefaultlocale' is deprecated and slated for removal in Python 3.15. Use setlocale(), getencoding() and getlocale() instead.
  ) = locale.getdefaultlocale()
local:
    ----------
    product_version:
        uyuni-master


Update sql-server-to-sql-on-azure-vm-migration-overview.md

provide link to topic https://learn.microsoft.com/en-us/sql/relational-databases/databases/manage-metadata-when-making-a-database-available-on-another-server

Learn Build status updates of commit d8c8e0c:
:bulb: Validation status: suggestions

File
Status
Preview URL
Details

azure-sql/migration-guides/virtual-machines/sql-server-to-sql-on-azure-vm-migration-overview.md
:bulb:Suggestion

Details

azure-sql/migration-guides/virtual-machines/sql-server-to-sql-on-azure-vm-migration-overview.md

Line 154, Column 122: [Suggestion: docs-link-absolute - See documentation] Absolute link 'https://learn.microsoft.com/sql/relational-databases/databases/manage-metadata-when-making-a-database-available-on-another-server' will be broken in isolated environments. Replace with a relative link.

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings or suggestions unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel

@markjones-msft

Can you review this PR?
IMPORTANT: When this content is ready to merge, you must add #sign-off in a comment or the approval may get overlooked.

#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

Learn Build status updates of commit 5684f53:
:warning: Validation status: warnings

File
Status
Preview URL
Details

azure-sql/migration-guides/virtual-machines/sql-server-to-sql-on-azure-vm-migration-overview.md
:warning:Warning

Details

azure-sql/migration-guides/virtual-machines/sql-server-to-sql-on-azure-vm-migration-overview.md

Line 154, Column 122: [Warning: file-not-found - See documentation] Invalid file link: '../../relational-databases/databases/manage-metadata-when-making-a-database-available-on-another-server.md'.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Learn Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings or suggestions unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel

Learn Build status updates of commit c950e80:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

azure-sql/migration-guides/virtual-machines/sql-server-to-sql-on-azure-vm-migration-overview.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel

#sign-off

@suresh-kandoth
PRMerger will accept sign-off from the listed author only. However, if you are a Microsoft employee authorized to update this content, you can send a request to<EMAIL_ADDRESS>for this PR to be merged. Include your role and an explanation of why the PR should be merged without approval from the listed author.


Remove shebang from test files

As per the discussion in #295, this PR will remove the shebang from all the test files.
Todo:

[x] lib/generator/test_template.erb
[x] bin/generate --all which can be invoked with a -f option, since changes don't effect the tests
[x] exercise/*/*_test.rb removal of shebang line
[x] bin/executable-tests-check should be removed

[x] modify .travis.yml so this file is not called

[x] bin/enable-executable should be removed
[x] bin/local-status-check needs to not reference the removed file
[x] Run bin/setup-git-hoooks if pre-push git hook is desired, and to update with changes
[x] Remove execution bit from all tests files
[x] Update test files to reflect the new templates

How Has This Been Tested?
By running the full test suite.
Type of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] Enhancement (non-breaking change which adds functionality)
[ ] Breaking change (fix or enhancement that would cause existing functionality to change)
[x] Gardening (code and/or documentation cleanup)

References #295
Checklist:

[x] I have read the CONTRIBUTING document.
[x] My change relies on a pending issue/pull request
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] I have added tests to cover my changes.
[x] All new and existing tests passed.

When this is complete, it should close #295 (and hopefully this message will accomplish that.)

Might not be related, but should I remove the shebangs from this project's test/ folder as well?

It should probably be a different PR if you do.  That change would be related to maintaining the track, as opposed to the track itself.

@Insti I have updates the files as you required.
@kotp can you please have a look and let me know if I missed anything. I consider this task completed.

Thank you for all the work done @budmc29.


Ability to execute arbitrary commands for filtered packages

Which project is this feature idea for?
Turborepo
Describe the feature you'd like to request
I apologize if this has been addressed previously, but I couldn't find anything about it.
There is a command available to the Lerna CLI called exec that allows us to execute any arbitrary command in each of the managed packages. Lerna also allows filtering, so, for example, I can execute a common "deploy" command from my CI tool configuration in each of the changed packages.
This functionality does not appear to be available in Turborepo. As a workaround, I've had to create the same scripts in each package, and then trigger them using turbo run from the CI configuration. This setup works, but there are two drawbacks: 1) I need to duplicate the same scripts in all 20+ packages, and 2) it moves scripts and configuration options out of the CI config and into each package (I'd like to keep these concerns separated).
It would be very helpful to have similar functionality available to the Turbo CLI.
Describe the solution you'd like
It would be great if there were an additional CLI command such as exec that would allow me to run an arbitrary command in each package (whether all packages or a filtered subset). The CWD would be set to the package root, not the repo root. Ideally, there would be a way to access some sort of package-specific config or metadata, perhaps the package.json contents like when running a script directly in that package. Something like this:
$ npx turbo exec --filter={./apps/*}[HEAD~1] docker build -f ../../Dockerfile -t $npm_package_config_dockerRepoName:$npm_package_version

This command would filter packages by the "apps" directory, and only those changed in the previous commit. It would execute docker build for each app; the Dockerfile would be common at the project root so -f ../../Dockerfile would tell it use the common config. Then the package.json information would be available as environment variables just as if running an NPM script.
Describe alternatives you've considered
Currently, I've set up shell scripts that each package calls into, but this requires copying multiple NPM scripts into each package and moving functionality into the packages that ideally would just live in the CI config. Here's an example:
{
  "name": "home",
  "version": "1.0.0",
  "scripts": {
    "docker:build": "../../scripts/docker-build.sh --app-name $npm_package_name --docker-repo-name $npm_package_config_dockerRepoName",
    "docker:push": "../../scripts/docker-push.sh --docker-repo-name $npm_package_config_dockerRepoName"
  },
  "config": {
    "dockerRepoName": "home"
  }
}

It's quite verbose, and I've come to ask for the feature because I need to add additional scripts now that could just be a one-liner in my Jenkinsfile.
If using the package.json content as environment variables doesn't work, maybe additional config in turbo.json could be an alternative. Something like:
{
  ...
  "packageMetadata": {
    "home": {
      "dockerRepoName": "home"
    },
    ...
  }
}

I'm open to any suggestions/alternative solutions. Thank you!
UPDATE (2023-04-19)
I've found another workaround, but it's specific to Jenkins. I've used the --dry-run=json flag to output the task metadata which includes all the packages that fit the filters. I can ingest the JSON and loop over the packages to run commands for each of them. To to this, I basically just have a global config that contains information for each package:
def deploymentMap = [
  "@my-scope/home": [
    dirName: "home",
    dockerRepoName: "home",
  ],
  "@my-scope/app2": [
    dirName: "app2",
    dockerRepoName: "app2",
  ],
  //...
]

def getChangedPackages(filterString) {
  def statsJSON = sh(script: "npx turbo build --dry-run=json --filter=${filterString}" , returnStdout: true)
  def stats = readJSON text: statsJSON

  return stats.packages
}

// usage in the pipeline:
step {
  script {
    def changedPackages = getChangedPackages('{./apps/*}[HEAD~1]')

    changedPackages.each({ packageName ->
      def config = deploymentMap[packageName]

      dockerBuild(packageName, config.dirName, config.dockerRepoName, "${version}")
      dockerPush(config.dockerRepoName, "${version}")
    }
  }
}

TURBO-2330

Hey! We have plans to address this.
I don't have a specific release timeline, but we are actively working towards it.

That's great news, thanks @arlyon!
Sounds like a cool feature, though maybe a little larger in scope than I had in mind 😅
Would it cover a case where I want to run something like turbo exec --filter={./apps/*}[HEAD~1] -- docker build . -f ../Dockerfile -t latest? Where everything after the -- would execute in each of packages matching the filter?

Yes, install is a separate but complementary feature :)

One day, we'd like you to able to run turbo exec build on a brand new repo and have it run the appropriate install command (pnpm install or others), figure out your package graph, and execute those commands with absolutely no config.

This sounds nice but I would be happy if Turbo would simply run the script that I tell it to run without anything else, e.g. turbo exec drizzle:push --filter="database"


Nn

a high level NN interface that supports symbolic+imperative use. With a mixed Keras and pytorch flavor.
@jermainewang @pluskid @sxjscience @yajiedesign @tqchen @mli @madjam

Should we try to combine this with mx.mod.Module? Since we have written a lot of codes in Module.

Also, is nn a good name? User may feel confused about Layers in nn and operators in operator/nn folder? @piiswrong @mli

@piiswrong: we are very interested in dynamic graph construction direction MXNet is going in. A question: how flexible (compared to pytorch) is this? For example, can complicated NLP examples be implemented (like https://devblogs.nvidia.com/parallelforall/recursive-neural-networks-pytorch/)?

@sbodenstein It should match pytorch feature by feature. If you see anything missing it should be easy to add it

@sxjscience The existing module API can actually do low level forward backward easily, as a demo shown here for writing your own explicit training loop. But comparing to PyTorch's full interface, it is still a bit basic. I do think we have a lot to learn from PyTorch / Chainer API design by either extending module or have a new API.

Seems we are going to manually define a bunch of NN layers in python, e.g., Pooling, BatchNorm, etc. Can they be generated automatically from c lib, just like the functions for symbol? Other frontend languages can follow the same way to do code generation.

this is a super exciting development. I'm also wondering what the relationship to the existing APIs like the Module API will be in the long run?

This is awesome! My nvidia blogpost has one example of the kind of model that needs this sort of framework, but a more practically useful one that'll demonstrate the full power of this approach in MXNet is https://github.com/facebookresearch/clevr-iep. I think it would be surprisingly easy to port as a demonstration (one benefit of PyTorch not building in any training loop abstractions etc. is that there's less to port when you move it to another framework).

Several comments from previous conversations that I think might be good to post here for public discussions.
Merging Metric and Loss
The current proposal serialize a list of metrics inside a loss object. An alternative proposal is to unify metric and loss. A Metric is an object that

support forward that maps inputs to a scalar
support backward that by default is NOP (i.e. zero gradient)

A Loss is a subclass of Metric, that has meaningful backward implementation.
When constructing module, a list of Metric can be given. During training and scoring, all the metrics will be forward evaluated. During training, all the Loss will be backward evaluated to provide initial gradient for back-prop.
So the effective loss is the sum of all Loss subclasses in the list metrics supplied.
Layer and SimpleLayer
I think it might be good to rename Layer to BaseLayer, and SimpleLayer to Layer, as the user will mostly be interacting with Layer.
The simple_forward function, which supports both NDArray forward and symbolic forward, might be renamed to mixed_forward or generic_forward (name coming from C++ generics).
Whether to provide a built-in fit function
The current fit implementation on Module is actually simple. However, it looks complicated because tooooooo many keyword arguments are added to that function. One way to potentially avoid this is to ask users to always write their own training loop, which should be fairly straightforward.
To make common scenarios simple to use. We can provide a training loop code template, from which user can copy & paste and customize at will. Alternative, we could provide a simplified fit function, with only a few arguments, and state in the doc encouraging the user to write their own training loop if extension is needed.


Change the way we install from backports in the deb builder (to force deps too)

Also, add "libsystemd-journal-dev" to the explicit list (which is what prompted the change in how we install).
Fixes #19668
cc @jfrazelle
This builds successfully. :+1:

you da bomb

LGTM

LGTM


TypeError: Cannot read property 'typehint' of undefined

[Enter steps to reproduce below:]

Install latest Atom
Use sbt plugin 0.2.1
sbt gen-ensime
In atom start ensime

Atom Version: 1.1.0
System: Mac OS X 10.10.4
Thrown From: Ensime package, v0.29.0
Stack Trace
Uncaught TypeError: Cannot read property 'typehint' of undefined
At /Users/waltocha/.atom/packages/Ensime/lib/ensime.coffee:163

TypeError: Cannot read property 'typehint' of undefined
  at Object.module.exports.Ensime.generalHandler (/Users/waltocha/.atom/packages/Ensime/lib/ensime.coffee:163:19)
  at /Users/waltocha/.atom/packages/Ensime/lib/ensime.coffee:201:52
  at SwankParser.onMessage (/Users/waltocha/.atom/packages/Ensime/lib/client.coffee:28:9)
  at SwankParser.handleMessage (/Users/waltocha/.atom/packages/Ensime/lib/lisp/swank-protocol.js:77:8)
  at SwankParser.handleContent (/Users/waltocha/.atom/packages/Ensime/lib/lisp/swank-protocol.js:64:10)
  at SwankParser.execute (/Users/waltocha/.atom/packages/Ensime/lib/lisp/swank-protocol.js:55:20)
  at Socket.<anonymous> (/Users/waltocha/.atom/packages/Ensime/lib/client.coffee:42:15)
  at emitOne (events.js:77:13)
  at Socket.emit (events.js:169:7)
  at readableAddChunk (_stream_readable.js:146:16)
  at Socket.Readable.push (_stream_readable.js:110:10)
  at TCP.onread (net.js:520:20)

Commands
     -2:51.2.0 command-palette:toggle (atom-text-editor.editor.is-focused.hyperclick)
     -2:48.9.0 core:confirm (atom-text-editor.editor.mini.is-focused)
     -2:48.9.0 ensime:start (atom-text-editor.editor.hyperclick)
     -2:15.8.0 command-palette:toggle (atom-text-editor.editor.is-focused.hyperclick)
     -2:13.6.0 core:move-down (atom-text-editor.editor.mini.is-focused)
     -2:13.1.0 core:move-up (atom-text-editor.editor.mini.is-focused)
     -2:12.9.0 core:confirm (atom-text-editor.editor.mini.is-focused)
     -2:12.9.0 ensime:stop (atom-text-editor.editor.hyperclick)
     -1:21 command-palette:toggle (atom-text-editor.editor.is-focused)
     -1:18.9.0 core:move-down (atom-text-editor.editor.mini.is-focused)
     -1:18.8.0 core:confirm (atom-text-editor.editor.mini.is-focused)
     -1:18.8.0 ensime:update-ensime-server (atom-text-editor.editor)
     -0:52.8.0 command-palette:toggle (div.ensime-log-view.scroll-view.native-key-bindings)
     -0:50.4.0 core:confirm (atom-text-editor.editor.mini.is-focused)
     -0:50.4.0 ensime:start (div.ensime-log-view.scroll-view.native-key-bindings)
     -0:33.7.0 core:copy (atom-notification.fatal.icon.icon-bug.native-key-bindings.has-detail.has-close.has-stack)

Config
{
  "core": {
    "themes": [
      "atom-dark-ui",
      "atom-dark-syntax"
    ]
  },
  "Ensime": {
    "sbtExec": "/usr/local/bin/sbt"
  }
}

Installed Packages
# User
Ensime, v0.29.0
hyperclick, v0.0.32
language-scala, v1.1.4

# Dev
No dev packages

---
Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/27955322-typeerror-cannot-read-property-typehint-of-undefined?utm_campaign=plugin&utm_content=tracker%2F14719420&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F14719420&utm_medium=issues&utm_source=github).

payload is undefined here:
https://github.com/ensime/ensime-atom/blob/master/lib/client.coffee#L28
Something is wrong with the wire format I would guess. Have you used ensime before and might have an old version in .ivy local or something that tries to speak swank? Can you try to open dev tools (cmd-alt-i) in atom and see if you can find any communication logs? And paste tail from [project]/.ensime_cache/server.log?

If above guess is indeed the case the following might be relevant:
I think that ivy resolution works by prioritizing local before cache so when you once have publishLocal that's the one it's gonna use since ensime-server has been on 0.9.10-SNAPSHOT since forever. I've used dirty money to clear my local when I needed to touch the server code: https://github.com/sbt/sbt-dirty-money.
@rorygraves Or am I wrong?

I have used ensime before in Emacs, I could try deleting my ivy cache if thats what you're suggesting?

Please try that, but could you ll both local and cache for ensime-server first?

Either it's that you have a old homebuilt one in local or an old downloaded i cache. If it's cache, it feels more problematic. Haven't looked at those issues for a while, but it would be nice if it "just works" for everyone…

oh, this looks old:
:) 09:50:32 Wed Nov 04 waltocha@LDNPWA000000054 ~/.ivy2/cache/org.ensime/server_2.11
$ l
-rw-r--r--   1 waltocha  INTRANET\Domain Users   3.4K 28 Jun 10:48 ivy-0.9.10-SNAPSHOT.xml
-rw-r--r--   1 waltocha  INTRANET\Domain Users   845B 17 Aug 14:12 ivydata-0.9.10-SNAPSHOT.properties

$ l
drwxr-xr-x   7 waltocha  INTRANET\Domain Users   238B 22 May 08:58 0.9.10-SNAPSHOT

So that doesn't seem right at all. I am going to delete them and start again.
Sorry for the false alarm - should have checked first.

No worries :)
I really think it's an issue with how we init the server. Is the second one ls on local? Could you try to just clear out that one and leave cache? Shouldn't SNAPSHOT deps in cache be updated from an sbt build?

What I'm after: if it's an old version in cache that makes the build not download the latest I think it's not a false alarm at all…

Yup, all sorted now (by deleting local).
Thanks for the help.

Once I had deleted local the latest server downloaded properly, so whilst it was an old version I think thats expected SBT behaviour? Local takes precedence?

That's my guess. Just to make it clear: you only deleted local not cache? Then I think we're good.  But just having used a previous version of Ensime (old version in cache) shouldn't break our plugin.

Yes just local.

Awesome! Then I can sleep tonight :)

Thanks for all the help.


Is it possible to tell which PRs hound should check?

The team I'm part of is currently split into sub-teams and we use GitHub's team feature. Is it possible to somehow tell hound to only scan PRs that has been tagged with @my-repo/my-team?
We're currently experimenting with Hound and would love to disable it for other teams that are also working on the same repository as us. Thanks!

There isn't a way in Hound to do that. It processes all open PRs.

Got it. thanks!


v0.8.1_27: Pop up screen is disturbed

SEVERITY: Major
MODULE:  Bubble Chart
Platform:

Windows 7 / Chrome 45.0
Windows 7 / Firefox 42.0
MAC / Safari 8.0
iPad Air 2 iOS(8.3) / Safari 8.0
Nexus 9 (5.0.1) / Chrome 45.0

RELEASE: v0.8.1
STEPS TO REPRODUCE:

Navigate URL: http://static.gapminderdev.org/vizabi/release/v0.8.1/preview/bubblechart.html
Bubble chart page will be loaded
Click “Find”. Small pop up screen is displayed
Now again click “Find”. Small pop up screen is disappeared
Click "Full" or “Landscape” button under RESPONSIVENESS
Again click “Find”
Note that small pop up screen is disturbed

ACTUAL RESULTS:
Small pop up screen is disturbed after clicking “Find” options which is available on the chart.  This happens in another way also. After loading chart on Firefox, click random button under responsiveness until all options are displayed. Then click any options and see that same issue is there.
EXPECTED RESULTS:
Small pop up screen should not be disturbed
SCREENSHOT:

Retest Result: Failed
Release: v0.8.3
Testing Environment: http://static.gapminderdev.org/vizabi/release/v0.8.3/preview/bubblechart.html

fixed


Containers are removed with Container ID, where not with Container Name

HI,
Currently have a problem, like when containers are placed in orchestration and one minion goes down then containers move to other and container ID gets changed and my code fails when i want to remove, Dockerode has not have a container remove with container name?

Just supply the name instead of the ID :)
https://docs.docker.com/engine/api/v1.26/#operation/ContainerDelete


Support to decoration image

Describe your changes

Implement support to DecorationImage

Type of change

New feature (non-breaking change which adds functionality)
Docs (changes to the documentation)

Checklist before requesting a review

[x] I have performed a self-review of my code
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works

@tilucasoli I made some minor adjustments and ran dcm linting. The main change is having an assert to throw an error if the DecorationImageDto does not have an image when resolving since it's required.


Battle Royale Mode In Private server :O

Anyone know anything about this? :o

No, it's not possible. You have to make it yourself

It is possible. The only thing you need to make on the client side is where the zone is etc etc.

I meant that it is not possible in the current state.. and I actually said You have to make it yourself.......

Yes, it's possible @iGhozT

You'll have to make it yourself.

...

Don't bump old issue.


How to generate Firestore security rules?

I'm using this library. Very cool! After I define a schema, like a string that is a minimum of 4 characters and a max of 10 characters, matching a regex pattern, I want to generate a Firebase Firestore security rules file that ensures data in the db matches this definition. The rules file has text in it like...
data.size() >= 4 && data.size()<=10 && data.matches("[a-z]+")

How would I go about converting a struct here to this kind of text?

Am I right that you want to take a string struct that uses built-in refinements like this:
let struct = S.string()->S.String.min(4)->S.String.max(10)->S.String.pattern(%re("[a-z]+"))

And then use it to generate a string with Firestore security rules?
Console.log(Firestore.SecurityRules.make(~struct))
// Output: "data.size() >= 4 && data.size()<=10 && data.matches(\"[a-z]+\")"

Yes but not just for strings. If I'm using the built in .min(3), .max(9), regex standard patterns maybe I can do most of it automatically as long as I can see inside the struct. FireStore lets you do validation on "objects" so I'll need to put together conditions on each field, like field1requirements && field2requirements etc. But if there is custom code in some field I'd like to know that and will probably have to bail/fail or do something custom.

Can you show an example of some FireStore validation object?

Can't do it now but here is a tool  that does it from zod.
https://github.com/yarnaimo/fireschema
The security rules are in a text file with a bunch of stuff unrelated to ReScriptStruct. But there are places in the file where you'll write "allow read, write : if (conditions)" where the condition is based on the rules for what data is allowed.
https://firebase.google.com/docs/firestore/security/rules-structure

Here is a tool where you start from a schema similar to a TypeScript and it generates real TypeScript types AND the security rule file so you can keep the two in sync.
https://github.com/bijoutrouvaille/fireward

Some of the rule conditions are based on authentication. Like this (not syntactically correct but you'll get the idea...
allow read : if not(user is anonymous) && user.role=="admin" && (insert the data rules here)

Oh, now it's super clear to me. Yes, it's possible to do with ReScript struct design. But I've scoped out the feature of getting the information about refinements usage, so for the version 3.1.0 some security rules are not possible:

It's easy to add. I'm preparing the next rescript-struct release in one-two week. I'll also add the information about refinements, so it'll be possible to output all security rules from the screenshot.
As for the implementation of the converter of rescript-structs to security rules, you can take a look at https://github.com/DZakh/rescript-json-schema/blob/9005594ee3ddd10af70d5543513324637b4ae978/src/JSONSchema.res#L156, it's very similar to what you need.

Here's the simplified version of possible code:

let makeSecurityRule = (~localtion=[], struct) => {
  switch struct->S.classify {
  | S.Unknown => "true"
  | S.String => `data${localtion->Location.toSecurityRulePath} is string`
  | S.Bool => `data${localtion->Location.toSecurityRulePath} is bool`
  | S.Float => `data${localtion->Location.toSecurityRulePath} is number`
  | S.Int => `data${localtion->Location.toSecurityRulePath} is int`
  | S.Tuple(structs) => `(data${localtion->Location.toSecurityRulePath} is list && ${structs
      ->Array.mapWithIndex((itemStruct, index) =>
        makeSecurityRule(~location=location->Array.concat([index], itemStruct))
      )
      ->Array.joinWith(" && ")})`
  }
}

Awesome! I think it is possible to write custom JavaScript logic for validation in ReScriptStruct. Is that correct? Will I be able to know if such a custom rule was written? Because those rules can never be automatically translated to Firestore and I'd want to throw and exception or something.

https://github.com/DZakh/rescript-struct/issues/34#issuecomment-1453809361

I've created the PR to solve the problem https://github.com/DZakh/rescript-struct/pull/39

I think it is possible to write custom JavaScript logic for validation in ReScriptStruct. Is that correct?

Yes, it is.

Actually, I assume the custom code refinement might fall into the Unknown category.

Yes, but if somebody uses S.transform, or S.refine the struct tag won't be changed to Unknown

Will I be able to know if such a custom rule was written?

It's not possible right now. As I know, it's also not possible with Zod. I assume they just ignore structs with custom validation logic. But if not, I can add an ability to check whether there's custom validation logic applied.

This looks really good. I can get the refinement kind for each type like int float array and string. So I can see that a string has a minimum length and maximum length for example.
Can I get the larger structure? Like can I know that a struct refers to an "object" with 3 fields - "firstName" string "lastName" string and "age" int?
And can I know if a field is optional?
It may be ok without it. The place where I define the struct I can define the security rules and pull out the metadata without having to "repeat myself". So if I know a field is a primitive I'll use the metadata to get the details. But for the larger picture like what fields are in the struct maybe I'll have to do that manually. This would only work for a struct I built myself where I know what it looks like.
I haven't tried generating security rules yet. Probably a few weeks away.

Maybe the Tuple is the "object" so I scan it recursively? Optional fields are important. Can I know about that?

Can I get the larger structure? Like can I know that a struct refers to an "object" with 3 fields - "firstName" string "lastName" string and "age" int?

Yes, it's already possible in the current version https://github.com/DZakh/rescript-json-schema/blob/9005594ee3ddd10af70d5543513324637b4ae978/src/JSONSchema.res#L205

And can I know if a field is optional?

Yes, it's already possible in the current version https://github.com/DZakh/rescript-json-schema/blob/9005594ee3ddd10af70d5543513324637b4ae978/src/JSONSchema.res#L150

Great. One last thing. Can I determine if something is a choice discriminated union? I use those everywhere. It's an A or B or C.

Yep, https://github.com/DZakh/rescript-json-schema/blob/9005594ee3ddd10af70d5543513324637b4ae978/src/JSONSchema.res#L187

I've released<EMAIL_ADDRESS>with the ability to get information about refinements. You can find a usage example here https://github.com/DZakh/rescript-struct/blob/57b6aa932207c2b27070b37df6d90503afbcfe9f/src/S.res#L3628-L3660
Feel free to reopen/open a new issue if you have any questions or feature requests.


[Bug]: docker.host value in ~/.testcontainers.properties is not used

Testcontainers version
3.5.0
Using the latest Testcontainers version?
Yes
Host OS
windows
Host arch
x64
.NET version
7.0.403
Docker version
Client: Docker Engine - Community
 Version:           20.10.22
 API version:       1.41
 Go version:        go1.18.9
 Git commit:        3a2c30b
 Built:             Thu Dec 15 22:28:22 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.22
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.9
  Git commit:       42c8b31
  Built:            Thu Dec 15 22:26:14 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.14
  GitCommit:        9ba4b250366a5ddde94bb7c9d1def331423aa323
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-buildx" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-buildx: no such file or directory
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-compose" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-compose: no such file or directory
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-dev" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-dev: no such file or directory
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-extension" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-extension: no such file or directory
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-sbom" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-sbom: no such file or directory
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-scan" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-scan: no such file or directory

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 3
 Server Version: 20.10.22
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 9ba4b250366a5ddde94bb7c9d1def331423aa323
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: <IP_ADDRESS>-microsoft-standard-WSL2
 Operating System: Debian GNU/Linux 11 (bullseye)
 OSType: linux
 Architecture: x86_64
 CPUs: 20
 Total Memory: 19.38GiB
 Name: Nero
 ID: 46K3:5DBU:TFGT:UPFZ:36GJ:UCBL:5PGL:EGZT:VD3P:N7L4:YLFB:4SCV
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: giammin
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  <IP_ADDRESS>/8
 Live Restore Enabled: false

WARNING: API is accessible on http://<IP_ADDRESS>:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/
WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

What happened?
I'm running testcontainers from a windows pc using the docker engine running on wsl2 (no docker desktop, custom docker installation directly on wsl2 debian)
It works fine if I set from code WithDockerEndpoint
 var _dbContainer = new MsSqlBuilder()
        .WithDockerEndpoint("tcp://localhost:2375")
        .WithImage("mcr.microsoft.com/mssql/server:2022-latest")
        .Build();

it work if i set environment var DOCKER_HOST
$env:DOCKER_HOST="tcp://localhost:2375"
dotnet test

but if i set docker.host in ~/.testcontainers.properties
docker.host=tcp://localhost:1234

I get the error:

Docker is either not running or misconfigured. Please ensure that Docker is running and that the endpoint is properly configured. You can customize your configuration using either the environment variables or the ~/.testcontainers.properties file. For more information, visit:
https://dotnet.testcontainers.org/custom_configuration/ (Parameter 'DockerEndpointAuthConfig')

I know if it is reading that docker.host value because if I assign non valid url value it gives me an url exception:

System.TypeInitializationException : The type initializer for 'DotNet.Testcontainers.Configurations.TestcontainersSettings' threw an exception.
-------- System.InvalidOperationException : This operation is not supported for a relative URI.

Relevant log output
No response
Additional information
No response

Hi, could you please verify that the environment variable has not been accidentally set to an empty string (please set it to $env:DOCKER_HOST=$null)? If TC can resolve the environment variable, it will favor it.

I did some test and I can confirm that $env:DOCKER_HOST is null

Sorry, I cannot reproduce it. The properties file runs fine on my test host.
Your description confuses me. Since you are saying it picks up the property, I would expect that it really cannot connect to the Docker host. Is the port 1234 a typo? Your other two examples use the port 2375. I do not expect it to be an issue in Testcontainers.

the 1234  port was an error in a copy paste. My .testcontainers.properties file has the correct value:
docker.host=tcp://localhost:2375
how can i test what value is used in code?

Testcontainers tries to resolve various Docker host configurations. It will select the first one it can establish a connection with. These are the configurations we check:
https://github.com/testcontainers/testcontainers-dotnet/blob/871a9cd1d9ae98a02d7778f3e22010180bfb2d8b/src/Testcontainers/Configurations/TestcontainersSettings.cs#L29-L36
According to the error message, none of the providers is able to establish a connection to Docker. Debug into the following class to debug the environment or properties file configuration (it would be interesting which _dockerEngine it chooses):
https://github.com/testcontainers/testcontainers-dotnet/blob/871a9cd1d9ae98a02d7778f3e22010180bfb2d8b/src/Testcontainers/Builders/EnvironmentEndpointAuthenticationProvider.cs#L28-L30
This part attempts to access (ping) the Docker host (if it succeeds, it will pick the endpoint and run the tests against it):
https://github.com/testcontainers/testcontainers-dotnet/blob/871a9cd1d9ae98a02d7778f3e22010180bfb2d8b/src/Testcontainers/Builders/DockerEndpointAuthenticationProvider.cs#L36-L46

I finally discovered what was the problem:
# comments in the .testcontainers.properties
I copy/pasted the example file from here:  https://java.testcontainers.org/features/configuration/
and modified it so my docker.host file line was:
docker.host=tcp://my.docker.host:1234     # Equivalent to the DOCKER_HOST environment variable.
I deleted the comment and now it works as expected

I deleted the comment and now it works as expected

😬
TBH, I am not sure if this is valid. Wikipedia says:

Comment lines in .properties files are denoted by the number sign (#) or the exclamation mark (!) as the first non blank character

But, of course, we can only select the value until the first # or ! occurs.

maybe the host value is not trimmed?

Whitespaces are not a problem, but a non-blank characters following the host address are. The current implementation takes the value as it is. Given that .NET lacks native support for .properties files, we prioritize only the essential features.
https://github.com/testcontainers/testcontainers-dotnet/blob/871a9cd1d9ae98a02d7778f3e22010180bfb2d8b/src/Testcontainers/Configurations/PropertiesFileConfiguration.cs#L43-L49

ok, so the error in my .testcontainers.properties was that the comment was not at the beginning of the line.

modifing the last line with this would prevent my error:

I had something similar in mind :+1:, but we need to take care of (consider) the escape sequences too.
I will close this issue in favor of #1041 (to keep the conversation clean). Thanks for bringing this up.

thanks for your support and for this awesome library


LibC::TimeVal#tv_sec is TimeT

That means it's architecture dependent
This fixes sleep on 32bit platforms, probably among
a host of other stuff.

@jhass I basically did what you proposed, except you had a small typo in the second case.

Sigh, I wish I hadn't done that typo, it was too late :(
Your fix is invalid, t.tv_sec = seconds.to_i64 fails.


add hair variation two

On my server I use modified hairstyles and sometimes the original hair overlaps.
example:

and it should actually look like this.

I can give you more information if you need it, the main problem is that component number two seems to activate the original hairstyle.
Thank you for your time and for your excellent work!

Currently working on this!

Currently finished, here a preview:
https://user-images.githubusercontent.com/47320936/167225661-be961c76-3ca8-4110-837c-1a176dfe1bbe.mp4

Currently finished, here a preview:
uklbk3s.mp4

Perfect, when will you compile a new version to use it?

@lush0n3 I dont know, i need to get approved another pr, then it should be available.

Perfect, thank you very much, thank you for your time and work.


Undefined line number error

You get an 'Undefined line number error' when loading a raw TXT file (source code) if your line numbers are out of sequence. I admit that my code can jump around a bit when I am in mid-development, and I tend to have sections of code that dont follow the line number sequence during dev. I then pull it all together for the production versions, but during dev and test, I rely on the Aquarius to sort the line numbering. For example..
10 rem some stuff
20 rem some more stuff
3 cls
4 poke some stuff
100 print
85 print

LOADing a program in ASCII format requires the line numbers to be an order with no duplications. There are multiple reasons for this, and it will not change.


Citra update instructions

Description
PR to update API key instructions to point to the new developer Create an API Key page.
Links and Data
Issue: kotlin/issues/3981

@shubham7109 Do you have somewhere that tells the developers how to create an API Key and where to use it?

Thanks @maryharvey!
We instruct users to create their own local.properties to locally manage their API Key, we don't provide any doc in the project on how to create the key:
https://github.com/Esri/arcgis-maps-sdk-kotlin-toolkit/blob/e7f0e8cf9c454bea91681982f87c75647766ef63/secrets.defaults.properties#L22

Thanks @maryharvey!
We instruct users to create their own local.properties to locally manage their API Key, we don't provide any doc in the project on how to create the key:
https://github.com/Esri/arcgis-maps-sdk-kotlin-toolkit/blob/e7f0e8cf9c454bea91681982f87c75647766ef63/secrets.defaults.properties#L22

OK understood. The https://developers.arcgis.com/documentation/security-and-authentication/api-key-authentication/ (that you have) takes the dev to the https://developers.arcgis.com/documentation/security-and-authentication/api-key-authentication/tutorials/create-an-api-key/ tutorial. FYI - If you ever wanted to be explicit anywhere in your advice, you could put a link to that Tutorial to Create an API Key.
Thanks

Thanks @maryharvey Added the relevant doc here:
https://github.com/Esri/arcgis-maps-sdk-kotlin-toolkit/blob/462ba3999c631e85736f198899389669f4d4b3ed/secrets.defaults.properties#L19-L23


Project status?

@manojlds are you still interested in developing this project?

Yeah, but it is more a showcase than of real utility.


Add endpoint for getting workspace users

coverage: 98.052% (-0.6%) from 98.621%
when pulling dde9e3bad27a2cea660a7c3bc2dd73537601e570 on workspace-users
into e187d09e605103e122171655c772ae94a696b900 on master.


Add support for compiling assembler (.S) files

Arrgh, still learning my way around github sorry. Feel free to delete this PR (I don't think I can at this point)


Store new libraries to db while updating others

Fixes #6

There are several extra changes in this PR, but the main fix for issue #6 is in db.update_libraries(). In cases where a previously-existing run is being updated in the database, there may be some new libraries along with the previously-existing ones. So some libraries on the run need to be updated, while others need to be created.


Hash scrolling rubber-bands when content changes

Scenario:

Bet a URL /example#foo, scrolled down to the end of the page.
Click on a link #bar.
onhashchange a piece of custom code removes
, adds a
 in its place.
Screen shows the new div, and the page instantly scrolls to the top of
.
With no delay start scrolling down.

Expected:

Page scrolls down.

Actual:

Page scrolls down a small amount and is immediately scrolled to the top of
. This repeats for a better part of a second, after which the scroll is released to act naturally.

I’m sorry to bring the hash scrolling again, you must hate it by now.

Hmmm, If possible, could you spin up a repo with this?

I disabled react-static scrolling:
<Router
          history={history}
          autoScrollToTop={false}
        >
          <Routes />
</Router>

This pattern for scroll to top:
    componentDidMount () {
      if (typeof document !== 'undefined') {
        window.scrollTo(0, 0)
      }

and this pattern instead of #scroll
getRef = div => {
    this.div = div
}

componentDidMount () {
    this.div.scrollIntoView({
        block: 'start',
        behavior: 'smooth'
    })
}

render () {

}

Couldn’t come up with a demo, came up with two fixes #460, #462.


validator rules shouldn't allow unicode

Are validation rules alpha, alpha_dash and alpha_num suppose to be alternative to PHP equivalent ctype_*?
Because the doc does not make it clean these rules will allow utf8 character (and not normalize them), a developer may use alpha_dash to guard a field such as username, without knowing its consequence.
http://laravel.com/docs/validation#rule-alpha
	/**
	 * Validate that an attribute contains only alpha-numeric characters.
	 *
	 * @param  string  $attribute
	 * @param  mixed   $value
	 * @return bool
	 */
	protected function validateAlphaNum($attribute, $value)
	{
		return preg_match('/^[\pL\pN]+$/u', $value);
	}

I have tracked down the commit here, but don't really see a good reason behind it.
https://github.com/laravel/framework/commit/f6c9a74b141ba3551941a813ff162457dc4d10ef

@bitinn Good catch :+1:
For username attribute, we can do this:
//...
use Illuminate\Support\Str;

class User extends Model
{
    //...

    /**
     * Set username attribute in lower case and in ASCII
     *
     * @param string $username
     */
    public function setUsernameAttribute($username)
    {
        $this->attributes['username'] = strtolower(Str::ascii($username));
    }
}

And adding a custom validation rule to return an explicit error message.


Euler: Fix typos & writing

Fixed wrong index value in the comparison between forward and backward Euler schemes
Fixed typos and spelling
Minor visual changes for consistency (newlines, aligning, ...)

Thanks for a lot the typos corrected, a very good bug catch !
@YelyzavetaV Could you merge if you also agree?


Kevin

I had a need to 'serialize' some gwen dsl steps when multiple feature files are run parallelly. In order to do so i have provided 2 dsls :
And I acquire a lock for 60 seconds
And I release a lock
The behaviour is as follows:
Then a lock is acquired for 60 seconds, no other thread will access the lock for at least 60 seconds.
If the locking thread takes more than 60 seconds, then other threads can potentially grab the lock.
If the lockingn thread finishes earlier than 60 seconds, any other thread can immediately get the lock and need not wait for  complete 60 seconds.
Effectively all steps between acquire and release get serialized when multiple threads are running (gwen in --parallel mode)  and the serialization has a guarantee for 60 seconds.

Hi @rkevin99,
I've reimplemented this as per my @Synchronized StepDef proposal above in a separate PR at the Gwen core level here so it can also be used by other engines.
I've also built the following snapshot for you to try it out and let us know how it goes before closing this PR and rolling it out. If it does not meet your needs, let us now also.

gwen-web-2.25.3-SNAPSHOT

Cheers,
Branko

Hi BrankoThanks for the quick response.
This might solve my problem - but i need a small clarification.
What would happen if step 1 passes and step 2 fails ? Would one of the other feature files that were waiting, now seamlessly enter this synchronized StepDef ? Or would they be waiting endlessly for the earlier thread to run to completion ?
RegardsKevin Rodrigues.
On Saturday, 25 August, 2018, 5:50:10 AM IST, Branko Juric<EMAIL_ADDRESS>wrote:
Hi Kevin,
Thanks for the PR. I see that what you're wanting is effectively a synchronization block around some steps. Perhaps an alternative approach to implementing this might be to introduce a @Synchronized annotation for StepDefs. All the steps in a StepDef with this annotation would then run in a single thread when called from any feature.
For example, the 3 steps in the following step def would run for one feature at a time when called by muntiple features running in parallel.
@Synchronized
@StepDef
Scenario: I run in single thread
Given step 1
When step 2
Then step 3
The implementation would be greatly simplified and wouldn't require any wait, timeout, or lock management code.
Let me know if this would suit your needs.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

If step 2 fails it should not block..Feel free to test that out.

Hi Branko
Any updates on this ? A few more issues that i may have with the @synchronized keyword at a StepDef level is that  even when 'parallelization' is NOT required, some scenarios would be  synchronized through the StepDef one at a time, which could be unnecessary.
Also, if we need consecutive execution of a few of the StepDefs  together ( which could be a series of javascript executions ), then too i am constrained to implement @Synchronized for each of those StepDefs rather than one consolidated block of synchronization across the steps.
By providing an acquire and release lock DSL at a feature file level would provide more flexibility/control to the users of gwen framework.
Your framework is beautiful and can be used across a lot across nightly builds of different solutions so testing and 'time' to complete the tests can become very important  - especially if the testing is in sync with the development teams in an 'agile' development mode. One would like to avoid unnecessary synchronizations/delays as far as possible.
Let me know.
Thanks & RegardsKevin Rodrigues.
On Monday, 27 August, 2018, 10:01:34 AM IST, Kevin Rodrigues<EMAIL_ADDRESS>wrote:
Hi BrankoThanks for the quick response.
This might solve my problem - but i need a small clarification.
What would happen if step 1 passes and step 2 fails ? Would one of the other feature files that were waiting, now seamlessly enter this synchronized StepDef ? Or would they be waiting endlessly for the earlier thread to run to completion ?
RegardsKevin Rodrigues.
On Saturday, 25 August, 2018, 5:50:10 AM IST, Branko Juric<EMAIL_ADDRESS>wrote:
Hi Kevin,
Thanks for the PR. I see that what you're wanting is effectively a synchronization block around some steps. Perhaps an alternative approach to implementing this might be to introduce a @Synchronized annotation for StepDefs. All the steps in a StepDef with this annotation would then run in a single thread when called from any feature.
For example, the 3 steps in the following step def would run for one feature at a time when called by muntiple features running in parallel.
@Synchronized
@StepDef
Scenario: I run in single thread
Given step 1
When step 2
Then step 3
The implementation would be greatly simplified and wouldn't require any wait, timeout, or lock management code.
Let me know if this would suit your needs.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Hi Kevin,
These are good points.
We can definitely suppress the sync blocks in the case of serial (non parallel) execution. I'll add a commit for that to the PR.
Regarding supporting an imperative style DSL at thefeature level, we would not wish to add that capability as we feel it would clutter feature files with technical details that really should be confined to meta. With little effort, you can achieve any synchronization you need. For example, to run multiple StepDefs in sync, just create a new higher level StepDef with @synchronozed annotation that calls all those StepDefs you need run synchronized. Each nested StepDef will then run in the synchronized block of the calling StepDef. We can also make a change to ensure that unncessary looks are not created again for nested calls
Please let us know how you go with the snapshot build and we will then roll it out to a release.
Thanks,
Branko

There are many things that could go wrong and lock up threads if we put the onus on the user to demarcate blocks in an imperative manner. We wish to keep Gwen simple and reliable and the annotation approach is flexible and robust.

Ok. Branko. Thx.
One question remains unanswered though:
What would happen if in a synchronized StepDef, step 1 passes and step 2 fails ? Would one of the other feature files that were waiting, now seamlessly enter this synchronized StepDef ? Or would they be waiting endlessly for the earlier (failed)  thread/step to run to completion ?
RegardsKevin Rodrigues.
On Wednesday, 29 August, 2018, 12:12:05 PM IST, Branko Juric<EMAIL_ADDRESS>wrote:
There are many things that could go wrong and lock up threads if we put the onus on the user to demarcate blocks in an imperative manner. We wish to keep Gwen simple and reliable and the annotation approach is flexible and robust.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

I've built the snapshot below containing the change to ignore synchronized step defs in serial (non-parallel) execution mode. This does not however include logic to reuse locks in nested stepdefs from the caller, since iInheriting locks would introduce more overhead and risk in cases where non annotated stepdefs are called both directly and nested at the same time by different features. Therefore the user is responsible for annotating all StepDefs (and there will be no surprises).
Kevin, please retest using snapshot and let us know the outcome.

gwen-web-2.25.3-a4a1d51-SNAPSHOT

Also can you change the title of this PR from Kevin to something more descriptive of the request? Thanks

Closing PR. This is being addressed at the Gwen core level PR 45 instead.


Incrementing the version number to 0.9.0

Increments the version number
Increases the PyQuil requirement to the latest version
Increases the PennyLane version due to API changes

Thanks @antalszava! We'll need to merge this in as soon as PL v0.9 is released and on PyPI


commonfns&vectors&basic:initialize the output buffer to zeros

To ensure bit exactness when reading output buffers using clEnqueueReadBuffer, it's crucial to initialize all output buffers to zero. This ensures they start with a predictable state, preventing potential undefined behavior. The reason for this initialization is that in all tests, the size of the buffers exceeds the number of work items in the kernel. Without initialization, the last portion of the output buffer might contain unpredictable data ("junk").

In our testing environment, we check the bit-exactness of output buffers between our compiler and a reference one. Since our framework isn't aware of what part of the output buffer is actually written by a certain kernel, it compares the entire buffer; if that buffer isn't properly initialized and the kernel doesn't write all of it, some of its content is undefined.

To continue the suggestion discussed in the meeting about initializing the buffer to zeroes during clCreateBuffer, I would like to express my concern that this approach does not fully address the issue of buffers potentially containing unpredictable data.
Currently, we are using the CL_MEM_COPY_HOST_PTR flag with clCreateBuffer, which copies data from a vector or pointer that might not be initialized.
This means that even if we initialize the buffer to zeroes initially, the copied data may still be unpredictable.

Discussed in the June 4th teleconference (I think, may have been the week prior).  Concerns are that this "buffers must be initialized" property is not enforced by the current CTS tests, so even if we merge this PR there is no guarantee that these buffers will continue to be initialized, nor is there a way to have confidence that there are not other tests that would need to be updated to initialize buffers.
Given these concerns, would a better solution be for whatever testing framework that is requiring bit exactness to do its own buffer intialization, vs. doing the buffer intialization explicitly in the tests?
Removing "focused review" while evaluating whether these changes are required.


[lmic-c1.51] How can get a confimation when my node joined a gateway or not

Good day.
I am new and I confured a gateway.
I also have anode with IC880a.
I am using that library with my node
https://github.com/things4u/LoRa-LMIC-1.51/tree/master/libraries/lmic-v1.51
When I am at home, my data are sent to TTN.
Actually I left my place and I am about 8km from my flat.
There is another gateway but I am in a building.
I am sure I can not reach a gateway, but my terminal print me this

Opmode check: EV_TXCOMPLETE, time: 1101
Sent: 1

Someone know that library and some know if there is solution to confirm that my node can reach a gateway?
If my node is out of a gateway range or can not send data throug a gate, I want it to print me a message and desactivate some fonction, as sending data..., and periodicaly chec if a gateway is available.
Thank a lot for any information, you could provide me

See https://github.com/things4u/LoRa-LMIC-1.51/issues/13


kubernetes resource should be eventually consistent-- not working for vpc subnet case

What happened?

How can we reproduce it?

What environment did it happen in?

Crossplane Version: 1.13.1
Provider Version: 0.46
Kubernetes Version:   1.27.1
Kubernetes Distribution:  baremetal

I am closing this because the necessary/sufficient information was not provided for us to understand and reproduce the issue. If the issue recurs in provider version 0.47.1, do not hesitate to reopen it by providing the necessary information to reproduce it.


Docker Hub description (README) is outdated

And it would be nice if it could be automatically updated as well during CI:
https://github.com/peter-evans/dockerhub-description
But that means, currently, that it would require to deactivate 2FA on Docker Hub.

This is causing issues for some beginners trying to use Docker with Deno. Hopefully someone can get around to fixing this. It looks like maybe #279 may resolve this but I haven't looked to far into it.


List all the FeedbackDataset responses not just the current user's

Description
As recently spotted by @nataliaElv, we were just listing the current dataset records with their annotations, instead of all the available record annotations, as we were sending the request to GET /api/v1/me/datasets/{id}/records instead to GET /api/v1/datasets/{id}/records, so removing the prefix /me solves this issue.
Kudos to @nataliaElv 🎉
Type of change

[X] Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested

[X] Current unit tests are passing, and annotations are retrieved for more than one user when applicable

Checklist

[X] I have merged the original branch into my forked branch
[X] follows the style guidelines of this project
[X] I did a self-review of my code

Thanks @frascuchon I've updated the PR title as it was a little bit confusing 😄


No guarantee to finalize DepositTree in absence of Eth1Data cached block height

Description
Since #6095 updating the structure of DepositTreeSnapshot block height is included in it and we may fail to finalize deposit tree in absence of execution block data referenced in finalized state's Eth1Data. We use height from Eth1Data cache, but in some cases, especially when finalization is delayed for many epochs, this data will not be available from the cache.
So, if this data is not found in the cache, we should call EL client to get it and finalize tree when the response is received. This will add an extra complexity to the deposit tree finalization for the side feature. Plus, in active network, we shouldn't ever fall into this, it could be more common for test networks.
Overall it looks like we could miss several DepositTreeSnapshot in test networks relying only on cache, but shouldn't have any issues in mainnet, so we may not need any improvements on this side.

Closing this - it's extremely unlikely to happen in real networks and just means we'll reload deposits from the EL on the next restart so isn't a big issue.


Renaming IndexOfVectorized to IndexOf

Bringing original IndexOf from corefx and naming it IndexOfNon_Vectorized (for comparison).
The default implementation of IndexOf is now vectorized (https://github.com/dotnet/corefx/pull/17143)

@KrzysztofCwalina, is the method name fine?
cc @davidfowl

Is conveying implementation detail? Maybe
IndexOfFastStart

What about calling it IndexOfSequential? or SequentialIndexOf.

looks good.


Add statusbar and remove scenegraph title

Related to: https://github.com/aframevr/aframe-editor/issues/147

I'd probably reverse the order of the actions: <DELETE> <ADD>. You usually want the positive action on the right hand side.

Looks good after that

Yes the delete button should probably be replaced by some of the following alternatives:

Add an option in Menu > Edit > Remove entity
When clicked on the scenegraph, show the delete icon aligned to the right on the same line of the entity.
Contextual menu over the entity both on the viewport and on the scenegraph.
Backspace shortcut

Transforms gizmos:
https://www.dropbox.com/s/zck3dfwqyrunwto/transforms_gizmos.mov?dl=0
And filters:
https://www.dropbox.com/s/x1715fjtkz239q3/filter_entities.mov?dl=0

Please take a look at it, and if it's ok for you we could merge and continue the next features in another PR, not to mix many things on the same PR. In fact I could have split search & transforms in two PR, if you think it could help the review I'll do that! :)

Yeah, can we split?

@ngokevin sure! First one with the filters: https://github.com/aframevr/aframe-editor/pull/172

@ngokevin and the second one https://github.com/aframevr/aframe-editor/pull/173
I close this PR


Code size

I have put together a code size testbench to measure code size. It has 2 test cases, spam logs lots of stuff (based on the qemu test) and smoltcp runs smoltcp's loopback example from latest git master (which has defmt support).
spam  |  smoltcp |  what
===================================
18428 |  60680   |  main branch
  624 |  49780   |  no defmt

spam has 156 log calls and is 18428 bytes (.text size). Commenting the run_test() call in main gives a .text size of 944 bytes. Therefore, defmt's code size is:
Base size (RTT stuff, etc): 944 - 624 = 320 bytes
Log call size: (18428-944)/155 = 112.8 bytes/call
112.8 bytes/call sounds about right. As seen here, a simple info!("defmt test {=u32}", x); is already 74 bytes of code, and the spam code does a mix of simple and complex log calls.
And 112.8 bytes avg, or 74 bytes to print a u32 is A LOT.
For comparison, let's check good old C printf:
void foo(int x) {
    printf("defmt test %d", x);
}

compiled with arm-none-eabi-gcc -c printf.c -Os -mthumb:
   2:	0001      	movs	r1, r0
   4:	4802      	ldr	r0, [pc, #8]	; (10 <foo+0x10>)
   6:	f7ff fffe 	bl	0 <printf>

This is just 8 bytes of code, or 8+14 = 22 bytes if we count the string.
Defmt is 3.5 times larger than printf when counting the string. And this is a generous comparison: arguably the apples-to-apples comparison is not counting the string, since the string pointer is equivalent to defmt's interned string IDs. In which case defmt is 10 times larger.
I'm aware defmt does much more than printf. It's to be expected a log call printing complex nested structs and enums generates bigger code. However, simple log calls (with no arguments or just a handful of integers) are the vast majority in real-world projects, and code size for these really hurts.

@Dirbaio: Thank you for putting in the effort to create this code size benchmark!
Do we know which parts of the code contribute the most bloat?

I've been experimenting with this. Code in my codesize4 branch, testbench updated.

#258 is an easy win.
Removing the bool and needs_tag optimizations is a big win since it allows the compiler to optimize the Formatter to essentially nothing, which avoids all the work of stack-allocating and initializing it at every log call.
Removing leb/zigzag encodings has almost no effect, perhaps unsurprisingly because the compiler wasn't inlining them anyway.

spam         |  smoltcp      |  what
code   tx'd  |  code   tx'd  |
===============================================
18428  2806  |  60680  1098  |  main branch
15460  2806  |  58232  1098  |  remove dyn Write (PR #258)
15764  2806  |  57904  1098  |  remove bool return in acquire
13076  2817  |  57800  1098  |  remove bool compression
13060  3181  |  57728  1458  |  remove leb64/leb128/zigzag. Make tags always 16bit
11828  3729  |  57872  1468  |  remove needs_tag optimization
11428  3729  |  57240  1468  |  make Formatter zero-sized

This is at the cost of increasing the wire size, but:

The needs_tag increase can probably be gained back with this without increasing code size.
The leb128/zigzag increase can be gained back with a "zero-removing" compression such as this

Something I want to try next is to "pack" together multile write calls. For example when printing 4 u32s, instead of doing 4 write calls, put them in a 16-byte buffer then do a single 16-byte write call. I have a WIP version of this but it doesn't improve codesize that much. I'm still investing why.

Updated table with size measurements from my company's firmware:
spam         |  smoltcp      | company's fw |  what
code   tx'd  |  code   tx'd  | code         |
==============================================================
18428  2806  |  60680  1098  | 136336       |  main branch
15460  2806  |  58232  1098  | 127152       |  remove dyn Write (PR #258)
15764  2806  |  57904  1098  | 125832       |  remove bool return in acquire
13076  2817  |  57800  1098  | 120096       |  remove bool compression
13060  3181  |  57728  1458  | 120024       |  remove leb64/leb128/zigzag. Make tags always 16bit
11828  3729  |  57872  1468  | 118280       |  remove needs_tag optimization
11428  3729  |  57240  1468  | 117064       |  make Formatter zero-sized
11428  3729  |  57312  1468  | 117168       |  Remove methods from Formatter
  624        |  49780        | 98716        |  no defmt

2x smaller! :)

I've done a quick test with rzCOBS compression+framing. Results on spam test:
rzCOBS compression: 2290 bytes, 367308 cycles
No compression:     3729 bytes, 337152 cycles

Wire size is better than upstream defmt, while keeping all the code size gains. Code size increase is only ~200 bytes.
This is without needs_tag. Wire size could be even better if this is implemented.
And it provides framing, so probe-run is able to recover from a corrupt stream!

@Urhengulas @jonas-schievink what are your thoughts on this? Is this something you would want to address?
#258 is an easy start, and I'm willing to clean/rework my other patches to PR them if you confirm this is something you'd like to review and merge.

Personally I think this is a good change, inclusing plugging in rzCOBS, but we don't currently plan to make breaking changes to defmt, which complicates things.

In what timeframe would you consider OK making the next breaking release?

In what timeframe would you consider OK making the next breaking release?

We currently want to avoid it, since it will split the ecosystem. We are discussing how to make breaking releases more feasible, mainly looking at https://github.com/knurling-rs/defmt/issues/426.
Another option we discussed is maintaining a next branch for some time. This branch would include all the changes from main and breaking changes, which enables us to start the effort, but delay the breaking change for longer. What would you think about this workaround?

Using defmt from git was extremely painful in the pre-1.0 times.

It forces users to have to git-clone, patch and build probe-run.
For git builds defmt requires firmware and decoder versions to match exactly (#287), so this means constantly rebuilding probe-run every time you upgrade defmt on your firmware.
You can't depend on git crates for crates published on crates.io. This means using this next branch in libraries (eg smoltcp, embassy) is unfeasible. Published crates would be stuck on 0.2, which in turns prevents end users to use next in their firmware if they depend on any such published crate. To get around this users would have to patch every single crate on their tree that depends on defmt 0.2!

Hi @Dirbaio! Now that #492 has been fully implemented, how are we doing in terms of code size? I've seen https://github.com/knurling-rs/defmt/pull/525, but that doesn't seem necessary to consider this issue fixed.

Yeah, #525 should reduce the code size further but in its current state it didn't. In my original experiments branch it did reduce it quite a lot, so I still have to check what's up with that.
Feel free to close this issue if you want. There'll always be some code size reduction possible, this'd stay open forever otherwise :P

Okay, closing then!


save_to_stp() for optics

Main changes:

save_to_stp() implemented for all optics (apertures, crystals, cameras)
For now only save optics as planar polygons (no curvature yet)

Issues:
Fixes, in devel, issue #905
Examples:
pfe = 'C:/Users/dvezinet/Documents/Slides_Posters/Meetings/20240202_SXR_CameraInversions/diagtest.npz'

coll = tf.data.load(pfe)

coll.save_diagnostic_to_stp(
    key_optics=['ap0', 'ap1'],
    factor=1,
    pfe_save='C:/Users/dvezinet/Documents/Slides_Posters/Meetings/20240202_SXR_CameraInversions/dslit_diag.stp',
    overwrite=True,
    color={'ap0': 'r', 'ap1': 'b'},
)

Hello @dvezinet! Thanks for opening this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file tofu/data/_class08_save2stp.py:

Line 54:5: E303 too many blank lines (2)
Line 513:80: E501 line too long (81 > 79 characters)
Line 532:80: E501 line too long (87 > 79 characters)
Line 533:80: E501 line too long (108 > 79 characters)
Line 543:80: E501 line too long (111 > 79 characters)
Line 554:80: E501 line too long (105 > 79 characters)
Line 565:80: E501 line too long (122 > 79 characters)
Line 576:80: E501 line too long (105 > 79 characters)
Line 585:80: E501 line too long (147 > 79 characters)
Line 595:80: E501 line too long (176 > 79 characters)
Line 605:80: E501 line too long (119 > 79 characters)
Line 616:80: E501 line too long (90 > 79 characters)
Line 627:80: E501 line too long (118 > 79 characters)
Line 638:80: E501 line too long (85 > 79 characters)
Line 649:80: E501 line too long (91 > 79 characters)
Line 661:80: E501 line too long (88 > 79 characters)
Line 672:80: E501 line too long (93 > 79 characters)
Line 683:80: E501 line too long (125 > 79 characters)
Line 694:80: E501 line too long (90 > 79 characters)
Line 705:80: E501 line too long (87 > 79 characters)
Line 715:80: E501 line too long (98 > 79 characters)
Line 725:80: E501 line too long (83 > 79 characters)
Line 735:80: E501 line too long (97 > 79 characters)
Line 755:80: E501 line too long (98 > 79 characters)
Line 765:80: E501 line too long (85 > 79 characters)
Line 775:80: E501 line too long (81 > 79 characters)
Line 787:80: E501 line too long (154 > 79 characters)
Line 796:80: E501 line too long (82 > 79 characters)
Line 813:5: E303 too many blank lines (2)
Line 824:80: E501 line too long (83 > 79 characters)
Line 827:1: E122 continuation line missing indentation or outdented
Line 837:80: E501 line too long (111 > 79 characters)
Line 843:80: E501 line too long (81 > 79 characters)
Line 847:80: E501 line too long (91 > 79 characters)
Line 848:80: E501 line too long (80 > 79 characters)
Line 858:1: E122 continuation line missing indentation or outdented
Line 865:80: E501 line too long (137 > 79 characters)
Line 877:80: E501 line too long (86 > 79 characters)
Line 898:15: W292 no newline at end of file