AuthorizeAttribute and RoleStore.IsInRole method use separate methods to check role

AuthorizeAttribute checks role equality by name, but IsInRole method of UserStore checks role equality by normalized name. Example:
[Authorize(Roles = "manager")] // It's will be ok
public async Task<IActionResult> SomeMethod(...)
{
    var isInRole = await _roleStore.IsInRoleAsync(user, "manager", cancellationToken); // It's will be false!
    ...
}

[Authorize(Roles = "MANAGER")] // It's will be not ok
public async Task<IActionResult> SomeMethod(...)
{
    var isInRole = await _roleStore.IsInRoleAsync(user, "MANAGER", cancellationToken); // It's will be true!
    ...
}

I think there need some unification

@HaoK Oh dear. This needs changing, but it may break a bunch of folks

Normalization is an identity concept though today, unless we want to push that concept to Auth?

Curious, wasn't the purpose of normalization just to disallow users or roles to be created that could seem equivalent? Did we make an explicit decision that using it for other comparisons was ok?

Yes so for identity we are case insensitive, but the claim we generate for the role is the actual role name (not normalized).  The implementation in AuthZ uses IsInRole on ClaimsIdentity which is case sensitive.  Identity isn't involved on that side of things, we just generate the claim (via our UserClaimsPrincipalFactory), and we use the normalization service for our APIs, hence the difference in behavior.  RoleManager is case insensitive, but the AuthZ apis don't have any concept of normalization/case insensitivity for the role check

This would be the same issue for any of the other things we normalize, like user name, `UserManager.FindByName("WeirdCaPS") will be true, while Authorize("PolicyCheckingNameEqualsWEIRDCAPS") will be false just like the roles check.
Its not that bad though, since they can work around this by not using Roles = and adding a policy that did a case insensitive string equality check on the role claim instead of using the sugar method.

@HaoK but, as you can see in example, IsInRole method of UserStore is not case insensitive. It expected you pass already normalized role and that is a problem, I think

That is by design, identity is not case sensitive with roles name by default, its just like user name, we don't want them to be considered different.

What IS case sensitive is ClaimsIdentity.IsInRole https://github.com/aspnet/Security/blob/dev/src/Microsoft.AspNetCore.Authorization/Infrastructure/RolesAuthorizationRequirement.cs#L58

Authorize(Roles = "foo") ends up calling that, identity produces a claimsidentity with that claim, its decoupled from the authorization logic by that point

@HaoK I guess, it is little strange that I can't create "manager" and "MANAGER" roles in one app, but if I created "manager" role and then try to check that user is having "Manager" role - i will get "false". I believe, it is design lack.

Two different systems. you either need to use the same casing, or instead of using the built in Roles parameter, use a policy that has a requirement that checks the Role claim using a case insensitive comparison
services.AddAuthorization(options =>
{
    options.AddPolicy("ManagerRole", policy => policy.RequireAssertion(context => String.Equals("Manager", context.User.FindFirst(ClaimTypes.Roles).Value, StringComparision.OrdinalIgnoreCase));
});

@HaoK My fault, it was incorrect example. If I create "manager" role and then check user having "manager" role with UserStore - I will get "false" because UserStore.IsInRoleAsync method expect already normalized name. Shouldn't this method expect unnormalized role?

Oh that sounds like a bug yes

Actually I misread what you wrote, the basic idea is that the stores shouldn't have to deal with normalization, the manager will take care of dealing with making sure the names are normalized (or not), but its up to them, this allows the stores to just do simple equality checks.
Feel free to continue the discussion here, but things appear to be working by design here so I will close this issue

Generally you should never access the stores directly, you should always go through the manager APIs

@HaoK Ok, thank you


Eclipse plugin's Run As -> Wurst fails after 1.28 patch

checking program ... 1719ms
translating program ... 365ms
running compiletime functions ... 309ms
translating program to jass ... 1021ms
Output created in /war-heroes/compiled.j.txt
preparing testmap ...
Starting wc3 ...
running [C:\Program Files (x86)\Warcraft III\Frozen Throne.exe, -window, -loadfile, Maps\wurstTestMap.w3x] // Issue is here

Looks like loadfile points to a directory that's no longer where wc3 expects it - 1.28 patch moves this to My Documents\Warcraft III\Maps\Test

@peq this as well

@Frotty this one as well

fixed by https://github.com/peq/WurstScript/commit/a8c161ce947ab79dda5eede80d2c9f854dfe6f5b


feat: [refactor]move CollectorController to CollectorService

I have moved the collectorController to collectorService.

I see this is a duplicate of #2433 🥲

I see this is a duplicate of #2433 🥲

oh,sorry,i didnt see it.

@2433 merged, you can close it.


Entire Cifs share wiped when container is removed

I am scheduling several container with hashicorp's nomad. After I recreate the nomad job my share is wiped. Maybe a cifs version of #104
root@cell-core-01:~#docker-volume-netshare version
Version: 0.33 - Built: 2017-01-08T22:45:48-08:00

root@cell-core-01:~# docker volume inspect  myserver.my.domain/share
[
    {
        "Driver": "cifs",
        "Labels": null,
        "Mountpoint": "/var/lib/docker-volumes/netshare/cifs/myserver.my.domain/share",
        "Name": "myserver.my.domain/share",
        "Options": {},
        "Scope": "local"
    }
]

root@cell-core-01:~# docker version
Client:
 Version:      17.03.1-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Mon Mar 27 17:14:09 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.1-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Mon Mar 27 17:14:09 2017
 OS/Arch:      linux/amd64
 Experimental: false

cat nomad.job
...
        volumes = [
            "myserver.my.domain/share:/export"
        ]
        volume_driver = "cifs"
...

Probably duplicate of: #122

This has been merged and released under version 0.34


Accelerate crashes unexpectedly with 30b+ Llama models

System Info
I am running on 5 x 8 GPU A100 80GB servers (40 GPUs total), 1TB System RAM each server, Infiniband interconnect.

- `Accelerate` version: 0.18.0.dev0
- Platform: Linux-5.15.0-60-generic-x86_64-with-glibc2.31
- Python version: 3.9.5
- Numpy version: 1.24.2
- PyTorch version (GPU?): 1.13.1+cu116 (True)
- `Accelerate` config passed:
	- compute_environment: LOCAL_MACHINE
	- distributed_type: FSDP
	- mixed_precision: bf16
	- use_cpu: True
	- num_processes: 40
        - main_process_ip: <private>
        - main_process_port: 29401
	- machine_rank: 0
	- num_machines: 5
	- rdzv_backend: static
	- same_network: True
	- main_training_function: main
	- deepspeed_config: {}
	- fsdp_config: {'fsdp_auto_wrap_policy': 'TRANSFORMER_BASED_WRAP', 'fsdp_backward_prefetch_policy': 'BACKWARD_PRE', 'fsdp_offload_params': True, 'fsdp_sharding_strategy': 1, 'fsdp_state_dict_type': 'FULL_STATE_DICT', 'fsdp_transformer_layer_cls_to_wrap': 'LlamaDecoderLayer'}
	- megatron_lm_config: {}
	- downcast_bf16: no
	- tpu_use_cluster: False
	- tpu_use_sudo: False
	- tpu_env: []
	- dynamo_config: {}

Information

[X] The official example scripts
[ ] My own modified scripts

Tasks

[X] One of the scripts in the examples/ folder of Accelerate or an officially supported no_trainer script in the examples folder of the transformers repo (such as run_no_trainer_glue.py)
[ ] My own task or dataset (give details below)

Reproduction
Steps to reproduce, use the official run_clm.py example:
Convert Llama weights/tokenizer with convert script. Then run below.
accelerate launch run_clm.py --model_name_or_path "llama-30b" --tokenizer_name "llama-tokenizer" --per_device_train_batch_size 1 --output_dir "../output" --overwrite_output_dir --fp16 --do_train --max_train_samples 128 --num_train_epochs 1 --dataset_name "wikitext" --dataset_config "wikitext-103-v1" --fsdp "full_shard offload auto_wrap" --fsdp_transformer_layer_cls_to_wrap "LlamaDecoderLayer" --optim "adamw_torch
This works for 13B and below models but for Llama 30b and 65b I get below. The resources I have should be enough to load 30b loading at least with offloading etc right?
I've also tried setting the low_cpu_mem_usage=True option when calling AutoModelForCausalLM.from_pretrained with a --do-eval instead of --do-train flag on that same command above and that also gives the same error. (based on this)
Loading checkpoint shards: 100%|██████████| 61/61 [05:45<00:00,  5.66s/it]
 WARNING  Sending process 14 closing signal SIGTERM         api.py:699
 WARNING  Sending process 15 closing signal SIGTERM         api.py:699
 WARNING  Sending process 16 closing signal SIGTERM         api.py:699
 WARNING  Sending process 17 closing signal SIGTERM         api.py:699
 WARNING  Sending process 20 closing signal SIGTERM         api.py:699
 WARNING  Sending process 26 closing signal SIGTERM         api.py:699
 WARNING  Sending process 29 closing signal SIGTERM         api.py:699
 ERROR    failed (exitcode: -9) local_rank: 5 (pid: 22) of  api.py:673
 binary: /opt/conda/bin/python
056444Z ╭───────────────────── Traceback (most recent call last) ──────────────────────╮
058181Z │                                                                              │
061458Z │ /opt/conda/bin/accelerate:8 in <module>                                      │
063432Z │                                                                              │
065546Z │   5 from accelerate.commands.accelerate_cli import main                      │
067260Z │   6 if __name__ == '__main__':                                               │
069203Z │   7 │   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])     │
086718Z │ ❱ 8 │   sys.exit(main())                                                     │
088482Z │   9                                                                          │
090075Z │ /opt/conda/lib/python3.8/site-packages/accelerate/commands/accelerate_cli.py │
092069Z │ :45 in main                                                                  │
093722Z │                                                                              │
095326Z │   42 │   │   exit(1)                                                         │
096929Z │   43 │                                                                       │
098552Z │   44 │   # Run                                                               │
100115Z │ ❱ 45 │   args.func(args)                                                     │
101728Z │   46                                                                         │
103322Z │   47                                                                         │
104905Z │   48 if __name__ == "__main__":                                              │
106648Z │                                                                              │
108262Z │ /opt/conda/lib/python3.8/site-packages/accelerate/commands/launch.py:906 in  │
109845Z │ launch_command                                                               │
111969Z │                                                                              │
113642Z │   903 │   │   args.deepspeed_fields_from_accelerate_config = ",".join(args.d │
115326Z │   904 │   │   deepspeed_launcher(args)                                       │
116919Z │   905 │   elif args.use_fsdp and not args.cpu:                               │
118532Z │ ❱ 906 │   │   multi_gpu_launcher(args)                                       │
120125Z │   907 │   elif args.use_megatron_lm and not args.cpu:                        │
121688Z │   908 │   │   multi_gpu_launcher(args)                                       │
123252Z │   909 │   elif args.multi_gpu and not args.cpu:                              │
124805Z │                                                                              │
126368Z │ /opt/conda/lib/python3.8/site-packages/accelerate/commands/launch.py:603 in  │
127911Z │ multi_gpu_launcher                                                           │
131909Z │                                                                              │
133652Z │   600 │   )                                                                  │
135246Z │   601 │   with patch_environment(**current_env):                             │
136829Z │   602 │   │   try:                                                           │
138392Z │ ❱ 603 │   │   │   distrib_run.run(args)                                      │
139975Z │   604 │   │   except Exception:                                              │
141558Z │   605 │   │   │   if is_rich_available() and debug:                          │
143242Z │   606 │   │   │   │   console = get_console()                                │
144825Z │                                                                              │
146528Z │ /opt/conda/lib/python3.8/site-packages/torch/distributed/run.py:753 in run   │
148091Z │                                                                              │
149654Z │   750 │   │   )                                                              │
151258Z │   751 │                                                                      │
153061Z │   752 │   config, cmd, cmd_args = config_from_args(args)                     │
154684Z │ ❱ 753 │   elastic_launch(                                                    │
156298Z │   754 │   │   config=config,                                                 │
157891Z │   755 │   │   entrypoint=cmd,                                                │
159524Z │   756 │   )(*cmd_args)                                                       │
161117Z │                                                                              │
163021Z │ /opt/conda/lib/python3.8/site-packages/torch/distributed/launcher/api.py:132 │
164574Z │ in __call__                                                                  │
166137Z │                                                                              │
167721Z │   129 │   │   self._entrypoint = entrypoint                                  │
169424Z │   130 │                                                                      │
171007Z │   131 │   def __call__(self, *args):                                         │
172600Z │ ❱ 132 │   │   return launch_agent(self._config, self._entrypoint, list(args) │
179584Z │   133                                                                        │
181218Z │   134                                                                        │
182781Z │   135 def _get_entrypoint_name(                                              │
184354Z │                                                                              │
185987Z │ /opt/conda/lib/python3.8/site-packages/torch/distributed/launcher/api.py:246 │
187570Z │ in launch_agent                                                              │
189173Z │                                                                              │
190877Z │   243 │   │   │   # if the error files for the failed children exist         │
192460Z │   244 │   │   │   # @record will copy the first error (root cause)           │
194063Z │   245 │   │   │   # to the error file of the launcher process.               │
195837Z │ ❱ 246 │   │   │   raise ChildFailedError(                                    │
197580Z │   247 │   │   │   │   name=entrypoint_name,                                  │
199174Z │   248 │   │   │   │   failures=result.failures,                              │
200747Z │   249 │   │   │   )                                                          │
202370Z ╰──────────────────────────────────────────────────────────────────────────────╯

Expected behavior
Model checkpoint loads and trains CausalLM

What you are telling the computer to do by launching the code this way is to load the model on each GPU, which you don't have the memory for (30B model will take 120GB in full precision). You need to use DeepSpeed ZeRo--3 to fine-tune the model, not sure how to enable the same thing with FSDP but @pacman100 may know.

Thanks for the clarification. I thought this was the case. It would be great to know what's the recommended way of training such (larger) models using Accelerate with FSDP and not just Deepspeed.

Hello, the FSDP command seems correct. I currently don't yet have access to LLAMA weights, have applied for it, can only check once I get access to it. Also as per this comment https://github.com/huggingface/transformers/issues/21509#issuecomment-1422841064, CPU offload with mixed precision seems to hang indefinitely with FSDP. Could you try to use FSDp with CPU offload without mixed precision and see if that solves the issue?

Also, for smaller models till 13B, were you able to successfully use FSDP with CPU offload using mixed precision?

@pacman100
I get the same error as my original post without mixed precision for 30b but am able to run it with lower models like 13b (without mixed precision, with mixed precision 13b hangs like you said in that comment)

Hello @rajammanabrolu, you say accelerate creashes but the below command accelerate launch run_clm.py --model_name_or_path "llama-30b" --tokenizer_name "llama-tokenizer" --per_device_train_batch_size 1 --output_dir "../output" --overwrite_output_dir --fp16 --do_train --max_train_samples 128 --num_train_epochs 1 --dataset_name "wikitext" --dataset_config "wikitext-103-v1" --fsdp "full_shard offload auto_wrap" --fsdp_transformer_layer_cls_to_wrap "LlamaDecoderLayer" --optim "adamw_torch is for using Trainer. I'm a bit confused now and need clarity on your side for a minimal example to run.

It's just the official script
https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_clm.py
with Llama and the command as mentioned in my initial post, it gives the
error I had there. I thought Accelerate was supported with Trainer?
On Mon, Apr 24, 2023, 11:08 PM Sourab Mangrulkar @.***>
wrote:

Hello @rajammanabrolu https://github.com/rajammanabrolu, you say
accelerate creashes but the below command accelerate launch run_clm.py
--model_name_or_path "llama-30b" --tokenizer_name "llama-tokenizer"
--per_device_train_batch_size 1 --output_dir "../output"
--overwrite_output_dir --fp16 --do_train --max_train_samples 128
--num_train_epochs 1 --dataset_name "wikitext" --dataset_config
"wikitext-103-v1" --fsdp "full_shard offload auto_wrap"
--fsdp_transformer_layer_cls_to_wrap "LlamaDecoderLayer" --optim
"adamw_torch is for using Trainer. I'm a bit confused now and need
clarity on your side for a minimal example to run.
—
Reply to this email directly, view it on GitHub
https://github.com/huggingface/accelerate/issues/1214#issuecomment-1521202441,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADNGS3SKGPIPVC3YCI6XPMLXC5S7NANCNFSM6AAAAAAWAKQD7Q
.
You are receiving this because you were mentioned.Message ID:
@.***>

So, you are using accelerate launch for launching distributed setup but using Trainer and trainer args for FSDP, is that understanding correct? Currently, I don't have access to multinodes with such high-end GPUs to test this out. And the fact that there is no error makes debugging very difficult. Are you using 3 nodes for 13B model as well or was that single node? If that was single node, then the issue for >=30B models might be the connection across nodes.

Yes that understanding is correct. I'm using the same number of nodes (3+)
for 13b also and it works with torchrun too. I was thinking it has
something to do with how large models are loaded and want to make sure I'm
not trying to load a model per process based on that code? Unfortunately
that error message is all I have so I can't tell if this is what it is
doing.
On Tue, Apr 25, 2023, 3:20 AM Sourab Mangrulkar @.***>
wrote:

Also, does it work with using torchrun for launching distributed setup
instead of accelerate launcher
—
Reply to this email directly, view it on GitHub
https://github.com/huggingface/accelerate/issues/1214#issuecomment-1521541320,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADNGS3TAV4QKTR3VSHZSXELXC6QQNANCNFSM6AAAAAAWAKQD7Q
.
You are receiving this because you were mentioned.Message ID:
@.***>

Also, to be clear, the error message is posted in the first message in this
error issue, it's an out of memory error.
On Tue, Apr 25, 2023, 4:08 AM Prithviraj Ammanabrolu <
@.***> wrote:

Yes that understanding is correct. I'm using the same number of nodes (3+)
for 13b also and it works with torchrun too. I was thinking it has
something to do with how large models are loaded and want to make sure I'm
not trying to load a model per process based on that code? Unfortunately
that error message is all I have so I can't tell if this is what it is
doing.
On Tue, Apr 25, 2023, 3:20 AM Sourab Mangrulkar @.***>
wrote:

Also, does it work with using torchrun for launching distributed setup
instead of accelerate launcher
—
Reply to this email directly, view it on GitHub
https://github.com/huggingface/accelerate/issues/1214#issuecomment-1521541320,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADNGS3TAV4QKTR3VSHZSXELXC6QQNANCNFSM6AAAAAAWAKQD7Q
.
You are receiving this because you were mentioned.Message ID:
@.***>

I don't think ERROR    failed (exitcode: -9) local_rank: 5 (pid: 22) of  api.py:673 would be OOM error. Shouldn't it be something like "RuntimeError: CUDA out of memory. Tried to allocate 8.60 GiB (GPU 0; 23.70 GiB total capacity; 3.77 GiB already allocated; 8.60 GiB free; 12.92 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF"

Actually, it might be the case that RAM might be running out. Let's take the Llama-30B model as an example. Before the accelerate.prepare call, when loading the pretrained model via model = AutoModelForCausalLM.from-pretrained(model_name_or_path), it first loads the model on the RAM. Now, in a node with 8 GPUs, there would be 8 processes loading the same checkpoint on RAM. By default, FP32 ckpt is loaded, for 32B model it would be around 120GB (approx 4GB per 1B params when using FP32). So, the total RAM by 8 processes would amount to 120*8 = 960GB which might be causing RAM (1TB total per node, available might be a bit lower because of other processes running on the server) to blow up and when it exceeds the limit in my experience the process gets killed (-9) and the server becomes unresponsive.

That makes sense though I'm still not sure how to fix that, should I be
using less processes per node? (But I thought it was one process per GPU
standard, how would I use all the GPUs on a node then?)
On Tue, Apr 25, 2023, 7:12 AM Sourab Mangrulkar @.***>
wrote:

Actually, it might be the case that RAM might be running out. Let's take
the Llama-30B model as an example. Before the accelerate.prepare call,
when loading the pretrained model via model =
AutoModelForCausalLM.from-pretrained(model_name_or_path), it first loads
the model on the RAM. Now, in a node with 8 GPUs, there would be 8
processes loading the same checkpoint on RAM. By default, FP32 ckpt is
loaded, for 32B model it would be around 120GB (approx 4GB per 1B params
when using FP32). So, the total RAM by 8 processes would amount to 120*8 =
960GB which might be causing RAM (1TB total per node, available might be a
bit lower because of other processes running on the server) to blow up and
when it exceeds the limit in my experience the process gets killed (-9) and
the server becomes unresponsive.
—
Reply to this email directly, view it on GitHub
https://github.com/huggingface/accelerate/issues/1214#issuecomment-1521863907,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADNGS3R44CODNUJHGCPN4CTXC7LVVANCNFSM6AAAAAAWAKQD7Q
.
You are receiving this because you were mentioned.Message ID:
@.***>

You can try DeepSpeed which avoid this by using zero_init. Alternatively, one would need to only load the model on rank 0 and then use sync_module_states=True in FSDP object init call. I'll look into how to enable this with Accelerate but can take some time.

Does this issue figure out?

How to run torchrun with accelerate's device_map = auto arguments to implemente pipeline parallelism in a single node and do it for each nodes with distributed training.

Howdy @pacman100 , I'm still seeing this issue with 30B+ Llama models (e.g. codellama 34B). llama 2 7b works fine. Here is a demo script:
# Script from: https://github.com/huggingface/trl/issues/1303
# Run this with DDP with "accelerate launch test_sft.py"
from datasets import load_dataset
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, AutoTokenizer
from peft import LoraConfig
from trl import SFTTrainer
from transformers import TrainingArguments
from accelerate import PartialState

# Load the dataset
dataset_name = "timdettmers/openassistant-guanaco"
dataset = load_dataset(dataset_name, split="train")

# Load the model + tokenizer
# model_name = "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T"
# model_name = "meta-llama/Llama-2-7b-hf"
model_name = "codellama/CodeLlama-34b-hf"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    trust_remote_code=True,
    cache_dir='',
    use_cache = False,
)

# PEFT config
lora_alpha = 8
lora_dropout = 0.1
lora_r = 32
peft_config = LoraConfig(
    lora_alpha=lora_alpha,
    lora_dropout=lora_dropout,
    r=lora_r,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=["k_proj", "q_proj", "v_proj", "up_proj", "down_proj", "gate_proj"],
    modules_to_save=["embed_tokens", "input_layernorm", "post_attention_layernorm", "norm"],
)

# Args
max_seq_length = 512
output_dir = "./results"
per_device_train_batch_size = 8
gradient_accumulation_steps = 2
optim = "adamw_torch"
save_steps = 10
logging_steps = 1
learning_rate = 2e-4
max_grad_norm = 0.3
max_steps = 1 # Approx the size of guanaco at bs 8, ga 2, 2 GPUs.
warmup_ratio = 0.1
lr_scheduler_type = "cosine"
training_arguments = TrainingArguments(
    output_dir=output_dir,
    per_device_train_batch_size=per_device_train_batch_size,
    gradient_accumulation_steps=gradient_accumulation_steps,
    optim=optim,
    save_steps=save_steps,
    logging_steps=logging_steps,
    learning_rate=learning_rate,
    fp16=True,
    max_grad_norm=max_grad_norm,
    max_steps=max_steps,
    warmup_ratio=warmup_ratio,
    group_by_length=True,
    lr_scheduler_type=lr_scheduler_type,
    gradient_checkpointing=True,
    gradient_checkpointing_kwargs = {"use_reentrant": False}, #must be false for DDP
    report_to="wandb",
)

# Trainer
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    peft_config=peft_config,
    dataset_text_field="text",
    max_seq_length=max_seq_length,
    tokenizer=tokenizer,
    args=training_arguments,
)

# handle PEFT+FSDP case
trainer.model.print_trainable_parameters()
if getattr(trainer.accelerator.state, "fsdp_plugin", None):
    from peft.utils.other import fsdp_auto_wrap_policy

    fsdp_plugin = trainer.accelerator.state.fsdp_plugin
    fsdp_plugin.auto_wrap_policy = fsdp_auto_wrap_policy(trainer.model)

# Train :)
trainer.train()

and here is the error:
special_tokens_map.json: 100%|██████████████| 411/411 [00:00<00:00, 1.01MB/s]
Loading checkpoint shards:  29%|████▎          | 2/7 [02:24<05:58, 71.62s/it][2024-04-08 20:36:50,275] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 4040 closing signal SIGTERM
[2024-04-08 20:36:50,276] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 4041 closing signal SIGTERM
[2024-04-08 20:36:50,276] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 4042 closing signal SIGTERM
[2024-04-08 20:36:54,495] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: -9) local_rank: 0 (pid: 4039) of binary: /workspace/ADVANCED-fine-tuning/trelisEnv/bin/python
Traceback (most recent call last):
  File "/workspace/ADVANCED-fine-tuning/trelisEnv/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/workspace/ADVANCED-fine-tuning/trelisEnv/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 46, in main
    args.func(args)
  File "/workspace/ADVANCED-fine-tuning/trelisEnv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1044, in launch_command
    multi_gpu_launcher(args)
  File "/workspace/ADVANCED-fine-tuning/trelisEnv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 702, in multi_gpu_launcher
    distrib_run.run(args)
  File "/workspace/ADVANCED-fine-tuning/trelisEnv/lib/python3.10/site-packages/torch/distributed/run.py", line 803, in run
    elastic_launch(
  File "/workspace/ADVANCED-fine-tuning/trelisEnv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 135, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/workspace/ADVANCED-fine-tuning/trelisEnv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

and here is how I'm doing accelerate config:
In which compute environment are you running?
This machine
----------------------------------------------------------------------------------Which type of machine are you using?
multi-GPU
How many different machines will you use (use more than 1 for multi-node training)? [1]: 1
Should distributed operations be checked while running for errors? This can avoid timeout issues but will be slower. [yes/NO]: NO
Do you wish to optimize your script with torch dynamo?[yes/NO]:NO
Do you want to use DeepSpeed? [yes/NO]: NO
Do you want to use FullyShardedDataParallel? [yes/NO]: yes
----------------------------------------------------------------------------------What should be your sharding strategy?
FULL_SHARD
Do you want to offload parameters and gradients to CPU? [yes/NO]: NO
----------------------------------------------------------------------------------What should be your auto wrap policy?
TRANSFORMER_BASED_WRAP
Do you want to use the model's `_no_split_modules` to wrap. Only applicable for 🤗 Transformers [yes/NO]: yes
----------------------------------------------------------------------------------What should be your FSDP's backward prefetch policy?
BACKWARD_PRE
----------------------------------------------------------------------------------What should be your FSDP's state dict type?
SHARDED_STATE_DICT
Do you want to enable FSDP's forward prefetch policy? [yes/NO]: NO
Do you want to enable FSDP's `use_orig_params` feature? [YES/no]: no
Do you want to enable CPU RAM efficient model loading? Only applicable for 🤗 Transformers models. [YES/no]: YES
How many GPU(s) should be used for distributed training? [1]:4
----------------------------------------------------------------------------------Do you wish to use FP16 or BF16 (mixed precision)?
bf16

and I'm running with 4x A6000 (48 GB VRAM) with 38 vCPU 200 GB RAM and a docker container size of 50 GB.

Ok, I think the issue here was that I had use_reentrant = False and that seems to require a lot more memory.


feat: Disable type to filter for single SelectFields with 10 or fewer options

This provides a better UX for users using iPads where the virtual keyboard gets in the way and is unnecessary for select fields with few options.

Is this worth having a test case? 🤷

Probably. Will write one up quickly.


Remove analytic keyword argument and v0.15 version bump

Thanks @trbromley, good catch with the docs! Have updated all device doc pages.


Remove STAGING Environment from SDK

Summary of changes

Let's remove the STAGING environment from the SDK

Checklist

[x] Added a changelog entry

Authors

List GitHub usernames for everyone who contributed to this pull request.

@sshropshire

Interesting, wasn't some of these changes already done in #171 ?

Interesting, wasn't some of these changes already done in #171 ?

Yes that is interesting. I'm not entirely sure why we didn't merge that work in.

👍 Looks like we have some remnants in the PayPalDataCollectorEnvironment + tests

👍 Looks like we have some remnants in the PayPalDataCollectorEnvironment + tests

Yup! That'll be removed in an upcoming PR. We shouldn't need that enum for data collector.


feat: safe waku

This PR adds a safe wrapper around sending and receiving waku messages. Also moves queueing and retry/resync logic to the safe wrapper from the adapter. The long term goal is to abstract away the connectivity related issues from the application.
Notes:

There are known issues with waku-js (such as this, this and this) that makes it very difficult or impossible to handle all errors gracefully in a wrapper at the moment.
The Svelte Readable subscribe function calls the callback function immediately, which is not needed, that's why there are the firstChatStoreSave and firstObjectStoreSave variables.
This safe wrapper approach does not work very well with the Svelte autoreload functionality. Editing and saving unrelated code may result in having multiple connection issues running in the background. In that case a refresh helps.
Currently the logging is turned on with debug loglevel for the wrapper. This helps building intuition about the connectivity issues and also help to see when to refresh the browser (see the previous point).
It would be good to disable the text inputs in the chat while the message is being sent and re-enable when it is sent. Currently only the send button is disabled but it is possible to send a message by pressing Enter. Maybe disabling sending with Enter can be a solution.
Added preliminary support for multiple peers, however it looks like that if any of the connections are not reliable then it makes the initial connection more flaky. The solution could be to only block until there is at least one peer that is connected, but at the moment it's not clear what is the best way to achieve it. That will be addressed in a separate PR.
The disconnection flow is not heavily tested and there may be issues if there are queued messages. That will be addressed in a separate PR.

@filoozom fixed the resubscribe and the callback, please re-review it.
Also added a safe wrapper around connect.


project website offline (sourceforge)

Google Chrome and Microsoft Edge users who have both the "English (United States)" and "English" languages configured in their browser are not currently able to access the project website https://processhacker.sourceforge.io/
Firefox users are not affected and SourceForge is looking into the issue with support ticket 20363

project website back online

a caching layer had cached a 302 redirect. It's possible that could have happened during a brief instance of the front page being unavailable, but at this stage we can only speculate. Upon purging this cache record, Chrome seems to have proper access to the site again.


Use fixed-size constant_time_eq

The generic constant_time_eq has several branches on the slice length,
which are not necessary when the slice length is known. However, the
optimizer is not allowed to look into the core of constant_time_eq, so
these branches cannot be elided.
Use instead a fixed-size variant of constant_time_eq, which has no
branches since the length is known.

Very snazzy! What prompted you to add these?

Very snazzy! What prompted you to add these?

Looking at the generated assembly in my new 64-bit phone (compiled within Termux). The variable-size constant_time_eq does a comparison between both sizes (for the assert), a comparison of the size with zero (to return immediately), a comparison of the size with 31 (to skip the vectorized code), an AND with a mask, the vectorized XOR/OR code, and a conventional XOR/OR loop for the tail. There's a total of 7 labels in that function (though all of the branches should be easy for the branch predictor when always called with the same size). The constant-size constant_time_eq, on the other hand, has no branches, and should have only the vectorized XOR/OR part.
Unfortunately, the code generated by LLVM (at least on the version Termux has) for the fixed-size variants on this phone is atrocious; it fully unrolls both the 16-byte and the 32-byte variants, somehow managing to spill several registers to the stack. The code for the 64-bit variant, on the other hand, is a fine vectorized code, as is the code for all three variants on x86 (at least on the version Fedora 31 has). Since this is probably an optimization bug in LLVM for aarch64 (it should at least do it without spilling), it should get better with time.


Entity span highlight invisible

Hello,
I'm facing an issue with sequence labeling packages pushed with doccano client where, once the name of the entity type is selected from the drop down menu appearing after having selected a text span, the span is not underlined or highlighted in anyway. This means that the annotator has no visual feedback of what he or she did. It looks exactly like if the example hadn't been annotated. Note that if one downloads the data, the labels added are actually there with the span boundaries and everything, in the .jsonl file.
Also note that if one creates manually a doccano project using the GUI in the browser, and using exactly the same data, labels, and guideline, this bug is NOT present.
Here is the sample code I use:
from doccano_api_client import DoccanoClient
import os

main_dir = os.getcwd()

doccano_url = 'http://<IP_ADDRESS>/'
doccano_client = DoccanoClient(
    baseurl=doccano_url,
    username='admin',
    password='password'
)

data_files_dir = os.path.join(os.getcwd(), 'Toy_data')
data_fn = os.path.join(data_files_dir, 'toy_data.jsonl')

ent_guideline_fn = os.path.join(data_files_dir, 'ent_guideline.md')
with open(ent_guideline_fn, encoding='utf-8', mode='r') as f:
    ent_guideline_string = f.read()

ent_description = 'Entities annotation for toy data'
ent_project_type = 'SequenceLabeling'
ent_response = doccano_client.create_project(name='toy_ent_api',
                                             description=ent_description,
                                             project_type=ent_project_type,
                                             guideline=ent_guideline_string)

ent_docc_proj_id = ent_response['id']

_ = doccano_client.post_doc_upload(project_id=ent_docc_proj_id,
                                   file_name=data_fn,
                                   file_path=data_files_dir,
                                   format='JSONL')

ent_docc_labels_fn = os.path.join(data_files_dir, 'ent_label_config.json')
_ = doccano_client.post_label_upload(project_id=ent_docc_proj_id,
                                     file_name=ent_docc_labels_fn,
                                     file_path=data_files_dir)

The code runs without errors.
toy_data.jsonl contains
{"id": "12345", "text": "I love doccano", "label": []}
{"id": "12346", "text": "Napoleon was French", "label": []}

ent_guideline.md contains
## product
For products
## person
For people

ent_label_config.json contains
[
  {
    "id": 1,
    "text": "product",
    "prefixKey": null,
    "suffixKey": null,
    "backgroundColor": "#FA28FF",
    "textColor": "#ffffff"
  },
  {
    "id": 2,
    "text": "person",
    "prefixKey": null,
    "suffixKey": null,
    "backgroundColor": "#A4DD00",
    "textColor": "#ffffff"
  }
]

I run doccano 1.5.5 in a local container on my computer running with Ubuntu 20.04. I created the image using
docker-compose -f docker-compose.prod.yml --env-file .env build
I ran it using
docker-compose -f docker-compose.prod.yml --env-file .env up
Note that I had to use the fix explained here:
https://github.com/doccano/doccano/pull/1739/files
to modify the files
Dockerfile
nginx/Dockerfile
in order to be able to build the docker image.
I use python 3.8.12 to run my script. Here is the output of pip feeze:
attrs==21.4.0
certifi==2<IP_ADDRESS>
charset-normalizer==2.0.12
doccano-client==1.0.3
idna==3.3
jsonlines==3.0.0
numpy==1.22.4
pandas==1.4.2
python-dateutil==2.8.2
pytz==2022.1
requests==2.27.1
six==1.16.0
urllib3==1.26.9

In particular, I use doccano-client 1.0.3

I found what seems to be the solution!
The problem seems to be related to the type of project that I create with these lines:
ent_project_type = 'SequenceLabeling'
ent_response = doccano_client.create_project(name='toy_ent_api',
                                             description=ent_description,
                                             project_type=ent_project_type,
                                             guideline=ent_guideline_string)

I assumed that it was enough to specify the project_type argument. But it's not. One also needs to specify the value of the resourcetype argument. The code above should be changed to
ent_project_type = 'SequenceLabeling'
ent_response = doccano_client.create_project(name='toy_ent_api',
                                             description=ent_description,
                                             project_type=ent_project_type,
                                             guideline=ent_guideline_string
                                             resourcetype='SequenceLabelingProject')

I guess that this solves the issue!


Power fix

Pull Request Test Coverage Report for Build 185

18 of 22 (81.82%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.4%) to 95.209%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

fluxions/fluxion_node.py
16
20
80.0%

Totals

Change from base Build 184:
-0.4%

Covered Lines:
934

Relevant Lines:
981

💛  - Coveralls


Fix #1918: Fix Long.MinValue / (negative power of 2).

Review by @gzm0, assuming you still have any recollection of how RuntimeLong works ^^

Refer to this link for build results (access rights to CI server needed):
https://scala-webapps.epfl.ch/jenkins/job/scalajs-pr/2220/
https://scala-webapps.epfl.ch/jenkins/job/scalajs-matrix-build/2886/
Test PASSed.

LGTM


ToastOptions should have field for PositionClass

Currently there is no way to change the location of the toasts per toast message. The global ToasterConfiguration must be changed, which changes the position for all toasts. We have a special case where the designer would like to control a particular toast messages position. Is this possible?
Thanks

The position class is applied to the toasts container and there is only one of it so, at the moment, what you’re asking is not possible. I would like to revisit the whole library to make the css transitions better so maybe I’ll keep this request in mind at that time

Actually after I thought about it for a few minutes...  I realized that what I was asking for was not really feasible nor really what I need. For some reason I forgot that the toast display is the display of the toasts in the array. Can't have one here and other over there, unless there are 2 toast containers. I was just worried about the user clicking to fast and seeing the new toast show up in the old location. Thanks for the response. I was going to delete this issue anyway.

I'll close it for now then. Feel free to reopen it in case you want to chat about it more.


MSFT_xWebAppPool Does not allow Managed Service Accounts

I don't seem to be able to use MSFT_xWebAppPool to set an AppPool to Run as a gMSA.
I had hoped it would work using a credential with a blank password like so:
Note that You have to set PSDscAllowPlainTextPassword = $true in the configuration data for this to work.
$pw = (new-object System.Security.SecureString)
$un = "domain\account$"
$cred = New-Object System.Management.Automation.PSCredential ( $un, $pw)

xWebAppPool testpool
{
    Name = "testPool"
    Ensure = "Present"
    enable32BitAppOnWin64 = $true
    autoStart = $true
    managedRuntimeVersion = "v4.0"
    managedPipelineMode = "Integrated"
    startMode = "AlwaysRunning"
    LoadUserProfile = $false
    IdentityType = "SpecificUser"
    Credential = $cred
    restartSchedule = @()
}

Note that You have to set PSDscAllowPlainTextPassword = $true in the configuration data for this to work (well, not work, but you get the Idea... )
This creates the Schema files correctly (I think!)
However I receive the following error when I send the configuration:
The password supplied to the Desired State Configuration resource MSFT_xWebAppPool is not valid. The password cannot be null or empty.
    + CategoryInfo          : InvalidResult: (:) [], CimException
    + FullyQualifiedErrorId : InvalidPassword
    + PSComputerName        : localhost

The SendConfigurationApply function did not succeed.
    + CategoryInfo          : ObjectNotFound: (root/Microsoft/...gurationManager:String) [], CimException
    + FullyQualifiedErrorId : MI RESULT 6
    + PSComputerName        : localhost

Is there any hope for this? Not allowing gMSA's is a deal breaker for my situation, so I would be thankful for any workaround...
I'm pasting a complete test config below to make checking this easier.
Configuration testIIS
{
    # Import the module that defines custom resources
    Import-DscResource -Module xWebAdministration, PSDesiredStateConfiguration

    # Dynamically find the applicable nodes from configuration data
    Node $AllNodes.where{$_.Roles -contains 'Web'}.NodeName
    {
        # Install the IIS role
        WindowsFeature IIS
        {
            Ensure          = 'Present'
            Name            = 'Web-Server'
        }

        # Install the ASP .NET 4.5 role
        WindowsFeature AspNet45
        {
            Ensure          = 'Present'
            Name            = 'Web-Asp-Net45'
        }

        $pw = (new-object System.Security.SecureString)
        $un = "domain\account$"
        $cred = New-Object System.Management.Automation.PSCredential ( $un, $pw)

        xWebAppPool testpool
        {
            Name = "testPool"
            Ensure = "Present"
            enable32BitAppOnWin64 = $true
            autoStart = $true
            managedRuntimeVersion = "v4.0"
            managedPipelineMode = "Integrated"
            startMode = "AlwaysRunning"
            LoadUserProfile = $false
            IdentityType = "SpecificUser"
            Credential = $cred
            restartSchedule = @()
        }

        # Create a new website
        xWebsite testSite
        {
            Ensure          = 'Present'
            Name            = "TestSite"
            State           = 'Started'
            PhysicalPath    = 'D:\Acumen\InetPub\Test-Site'
            DependsOn       = '[xWebAppPool]testpool'
        }

    }

}

$configData = @{
    AllNodes = @(
                    @{
                        NodeName = "*";
                        PSDscAllowPlainTextPassword = $true
                    }
                    @{
                        NodeName = "localhost";
                        Roles=@("web")

                    }

        )
    }

testIIS -ConfigurationData $configData

So it appears that you can't use a credential with an empty password anywhere in DSC, the Community resource gets around this by passing the username as a string and the password as a PSCredential object. Their approach does in fact allow me to use GMSAs so I have a path forward, however it would be nice if I could do the same using this resource.

Interesting, I'll take a look into this as soon as I have a chance.

Looking at this it appears to be an issue with DSC itself.  As I can run the specific command without issue.
"VERBOSE: [COMPUTERNAME]: LCM:  [ End    Test     ]  [[xWebAppPool]testpool]  in 0.0070 seconds.
The password supplied to the Desired State Configuration resource MSFT_xWebAppPool is not valid. The password cannot be null or empty.
    + CategoryInfo          : InvalidResult: (:) [], CimException
    + FullyQualifiedErrorId : InvalidPassword
    + PSComputerName        : localhost
"

Credentials in this case were username: test password: (null)
Set-TargetResource -Name 'testapp' -Ensure 'Present' -Credential $(Get-Credential) -identityType 'SpecificUser' -restartSchedule @()

cmdlet Get-Credential at command pipeline position 1
Supply values for the following parameters:
Credential

@TravisEz13 or @KarolKaczmarek - Do either of you know why we specific disallow blank passwords in credential objects in DSC?
I can modify this to match the community code but it would be inconsistent with how we use credentials in other resources.

Thanks for looking into it, I reached the same conclusion. I'm actually working on modifying several of the MSFT_x... resources to pass the user names separately (Notably xService and xSqlServer .) I'll admit it feels clunky, but it seems to be the best way to keep our deployment  on track for now....

@tysonjhayes
No, I see no reason to disallow blank passwords, but I believe @randomchance  is correct.  Please file user
Please file a user voice issue that DSC should allow this:
https://windowsserver.uservoice.com/forums/301869-powershell?query=password

Done - DSC: Should allow Credentials with Blank Passwords, so you can use GMSA accounts

I believe DSC stops the credential from ever being delivered, preventing you from being able to check the username - but I could be mistaken

@MarvTheRobot not all accounts ending in $ are managed service accounts.  Machine accounts in a domain also end in $, but I don't think they could be used in this way...

I think @MarvTheRobot did something like this Install and Configure a Group Managed Service Account with PowerShell
It doesn't seem to been work done on this yet. The resource still expect to set a password. No check is made to see if it is a managed account (as proposed above). To see if it is a gMSA we could use accountName$ -match '\$$', but it will be true for a computer name as well, so that works only if computer names are not allowed as account names.


make publish_pointclouds param grip for external map updates

Minor fix to prevent pointclouds from being published all the time if they are disabled, draining a lot of compute.

Can one of the admins verify this patch?

Ok to test


Freeze my IDE for about half a minute

Reproduce:
When I open a large project: such as Spring, when I modify the code, switch to another application, and switch back to the IDEA, My IDEA freezes for about half a minute every time.
Find:
So I went to the IDEA folder to find logs, and I found that a lot of logs were printed

There are Atom Material File Icons print information in the log file, so I guess it may be Atom Material File Icons error

Then I disabled Atom Material File Icons to reopen the project, it did not print the frozen log, and it would not cause the IDEA to freeze.
I'm not sure if it is an error of Atom Material File Icons or a conflict about plugins.
So now I have to ban Atom Material File Icons in the project:cry:.
This is the log file:
threadDump-20201215-203351.txt

I'm not really confident about the performance of the plugin, this is not a metric I'm working by. But I've been working with my plugin for a long time now and didn't notice unbearable conditions so far... And I'm working with big projects too.
Maybe try to reduce CPU consumption by excluding folders or by trying to not load all modules in the same window?

thank you very much. I will try and continue to observe.


[Feature] Removing unnecessary JavaScript functions

Problem
There exists two functions that I came across that does nothing in our website
Description
The functions UpdateQueryString() and RemoveQueryString() present in "listing.html" are non-functional. Upon investigation, I have found that these functions are not being called anywhere in the code. After removing them, I tested the website's functionality, and it worked perfectly fine. The intended purpose of these functions was to update the URL as filters are applied and removed, but this functionality is already present in "all.html".

It would be a good approach to remove these functions unless necessary as they are creating confusion in the codebase.
Implementation

[x] I would be interested in implementing this feature.

@possumbilities could you provide a feedback regarding this?

Hi @Xaid-vfx, It's ready to work now, are you still interested in implementing it?

@digi-booster yes, I am taking up this issue.


Release v1.29.0

v1.29.0: New Firewall Rules module & Slurm-GCP v6 Improvements

What's Changed
Key New Features 🎉

Initial commit of firewall-rules module by @tpdownes in https://github.com/GoogleCloudPlatform/hpc-toolkit/pull/2244

New Modules 🧱

Split service account creation from htcondor-setup by @tpdownes in https://github.com/GoogleCloudPlatform/hpc-toolkit/pull/2250

Module Improvements 🔨

Set http_proxy, https_proxy variables for user login and during startup-script by @tpdownes in https://github.com/GoogleCloudPlatform/hpc-toolkit/pull/2237
Update documentation for Packer to include minimum operational requirements by @tpdownes in https://github.com/GoogleCloudPlatform/hpc-toolkit/pull/2241
Modify cloud-storage-bucket to include ability to set bucket viewers by @tpdownes in https://github.com/GoogleCloudPlatform/hpc-toolkit/pull/2247
Add "submit" option to batch-job-template module by @aaronegolden in https://github.com/GoogleCloudPlatform/hpc-toolkit/pull/2210
Prevent usage of placement with static and auto-scale nodes in same nodeset by @nick-stroud in https://github.com/GoogleCloudPlatform/hpc-toolkit/pull/2279

Improvements 🛠

Add login node in spack wrfv3 tutorial example by @alyssa-sm in https://github.com/GoogleCloudPlatform/hpc-toolkit/pull/2207

Version Updates ⏫

Update to Slurm-GCP 5.10.4 by @tpdownes in https://github.com/GoogleCloudPlatform/hpc-toolkit/pull/2251
Bump cryptography from 42.0.0 to 42.0.2 in /community/front-end/ofe by @dependabot in https://github.com/GoogleCloudPlatform/hpc-toolkit/pull/2257
Slurm-GCP Release 6.4.1 by @tpdownes in https://github.com/GoogleCloudPlatform/hpc-toolkit/pull/2230
Updated Slurm v6 version, fix placement error by @nick-stroud in https://github.com/GoogleCloudPlatform/hpc-toolkit/pull/2280
Bump cryptography from 42.0.0 to 42.0.4 in /community/front-end/ofe by @dependabot in https://github.com/GoogleCloudPlatform/hpc-toolkit/pull/2288
Bump Golang version 1.20 -> 1.21 by @mr0re1 in https://github.com/GoogleCloudPlatform/hpc-toolkit/pull/2291

Bug fixes 🐞

Fix bug in validation of variable in PBS module by @tpdownes in https://github.com/GoogleCloudPlatform/hpc-toolkit/pull/2255
Fix for wait-for-startup.yml ansible task by @cdunbar13 in https://github.com/GoogleCloudPlatform/hpc-toolkit/pull/2283

Full Changelog: https://github.com/GoogleCloudPlatform/hpc-toolkit/compare/v1.28.1...v1.29.0
What’s changed gets added automatically by GitHub.

PR-test-e2e IIRC was recently added in trigger after RC was cut. So we can ignore it. @mr0re1


add aggregation pipeline stage AddFields (new in mongodb 3.4)

0.12.2-SNAPSHOT published again with it


How user with Class View ?

How can I use with url MyListView.as_view() ?
Like:
{% breadcrumb "My View" "MyviewListView.as_view()" %}

My url:
url(r'^myview/view/(?P[0-9]+)/$', MyViewDetail.as_view(), name='myview_detail'),
Thank's a lot.

you can pass the name as argument:
{% breadcrumb "My View" "myview_detail" %}

Thank's.


next deploy custom domain?

Codecov Report

Merging #85 (82e2178) into master (ddd06c6) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #85   +/-   ##
=======================================
  Coverage   94.31%   94.31%
=======================================
  Files          14       14
  Lines         422      422
  Branches      133      133
=======================================
  Hits          398      398
  Misses         21       21
  Partials        3        3

Flag
Coverage Δ

eslint-plugin-vality
96.87% <ø> (ø)

vality
93.55% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


PDF file transferring corruption

My Setup

Operating System: Windows 10x64

App Version: 2.10.5

[ ] I have tested with the latest version

[ ] I can simulate the issue easily

Description

Take PDF file.
Press on contact
Drag and Drop file to Contract
Send

Current Behavior
After loading of file on other client it is
a. Locked
b. Contains invalid data.
Expected Behavior
File should not be locked after download completion. File content should not be corrupted.

Please test if the latest release did solve this. If not please reopen the issue or create a new one and tell us. I will be closing this one for now.
Thank you!


fix: default allowed-origins value

📑 Description
Closes https://github.com/goravel/goravel/issues/457
I defined it in allowed-origins because <IP_ADDRESS>:3000 is the default.

@coderabbitai summary

✅ Checks

[ ] Added test cases for my code

Thanks for your suggestion, but we don't want to change it for now. Welcome to other optimizations.

Thank you check this PR.
I'll try other optimizations. 👍


Preserve formatting

Right now edgetest doesn't preserve the formatting when it changes toml or cfg files. This would be a nice feature to get working so it doesn't butcher and preserves the existing formatting of the file.

Hi @fdosani! I would like to contribute to this issue. Could you make me the assignee for this?
Also, would you happen to have any repo examples handy for toml-based and cfg-based projects that I could easily test for this? Thank you so much!


Build src and doc jar

I'm not sure if this because of jitpack.io, but I cant download the sources jar for jurl.
This makes debugging and understanding the library more difficult.
Maybe only an option is missing in the gradle build file.
Thanks for the cool library!

Hey I updated the gradle build file and it should include the sources now. Let me know if it worked! Cheers

It works ! :)
Thanks for the quick fix.


Add support for Python 3.11

AWS Lambda has recently added support for Python 3.11
Can we add this runtime to Lithops?

The changes in #1134 should be enough


discord-canary: 0.0.140 -> 0.0.141

Description of changes
Bump Discord Canary
Except, this time there is a problem.
After changing the version number and updating the hash, trying to run the resulting binary, gives me this output:
[604419:1026/211714.572002:FATAL:fork_and_spawn.cc(196)] posix_spawn: No such file or directory (2)

CC: @Artturin
I don't really know how to debug this as an issue, since it provides no real usable output.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

This also occurs on other distros. I believe it is a Discord issue.

I'll mark this as needing an upstream fix then.

strace -f can be useful for debugging but the output isn't very friendly
there's this but that could have been happening before since we dont have chrome_crashpad_handler at that location
[pid 1182537] execve("/nix/store/cgn42klrqdz6qj3gbisgh9lcr5hnlr4h-discord-canary-0.0.141/opt/DiscordCanary/chrome_crashpad_handler", ["/nix/store/cgn42klrqdz6qj3gbisgh"..., "--monitor-self-annotation=ptype="..., "
--no-rate-limit", "--database=/home/artturin/review"..., "--url=https://sentry.io/api/1463"..., "--annotation=_companyName=Discor"..., "--annotation=_productName=Discor"..., "--annotation=_version=0.0.141", "--ann
otation=prod=Electron", "--annotation=ver=17.4.9", "--initial-client-fd=45", "--shared-client-connection"], 0x8e60032c800 /* 186 vars */) = -1 ENOENT (No such file or directory)
[pid 1182537] exit_group(127)           = ?
[pid 1182537] +++ exited with 127 +++
[pid 1182536] <... clone3 resumed>)     = 1182537
[pid 1182536] wait4(1182537, NULL, 0, NULL) = 1182537
[pid 1182536] munmap(0x7fac3516e000, 36864) = 0
[pid 1182536] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 1182536] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1182537, si_uid=1000, si_status=127, si_utime=0, si_stime=0} ---
[pid 1182536] getpid()                  = 1182536
[pid 1182536] futex(0x5635cf123bb8, FUTEX_WAKE_PRIVATE,<PHONE_NUMBER>) = 0
[pid 1182536] openat(AT_FDCWD, "/nix/store/g8d5bnh9mh1364x17mjyc731fsi9a61m-glibc-2.35-163/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 3
[pid 1182536] newfstatat(3, "", {st_mode=S_IFREG|0444, st_size=2998, ...}, AT_EMPTY_PATH) = 0
[pid 1182536] read(3, "# Locale name alias data base.\n#"..., 3072) = 2998
[pid 1182536] read(3, "", 3072)         = 0
[pid 1182536] close(3)                  = 0
[pid 1182536] openat(AT_FDCWD, "/nix/store/g8d5bnh9mh1364x17mjyc731fsi9a61m-glibc-2.35-163/share/locale/en_US.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 1182536] openat(AT_FDCWD, "/nix/store/g8d5bnh9mh1364x17mjyc731fsi9a61m-glibc-2.35-163/share/locale/en_US.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 1182536] openat(AT_FDCWD, "/nix/store/g8d5bnh9mh1364x17mjyc731fsi9a61m-glibc-2.35-163/share/locale/en_US/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 1182536] openat(AT_FDCWD, "/nix/store/g8d5bnh9mh1364x17mjyc731fsi9a61m-glibc-2.35-163/share/locale/en.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 1182536] openat(AT_FDCWD, "/nix/store/g8d5bnh9mh1364x17mjyc731fsi9a61m-glibc-2.35-163/share/locale/en.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 1182536] openat(AT_FDCWD, "/nix/store/g8d5bnh9mh1364x17mjyc731fsi9a61m-glibc-2.35-163/share/locale/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 1182536] write(2, "[1182536:1027/043546.647542:FATA"..., 101[1182536:1027/043546.647542:FATAL:fork_and_spawn.cc(196)] posix_spawn: No such file or directory (2)

This is very much an issue that needs to be fixed on Discord's end, as far as I know -- I've heard from multiple sources on the matter who've all said the same thing. No ETA, though, and at least we have SKIP_HOST_UPDATE.

Aight, ig i had good timing on the SKIP_HOST_UPDATE script :smile: .

Never mind, can you try setting ELECTRON_ENABLE_STACK_DUMPING=true?
Don't ask me how this was discovered, I have no idea.

Apparently enabling that just causes it to magically work.
What

Not initializing crash reporter because ELECTRON_ENABLE_STACK_DUMPING is set.

A crash reporter causing something to fail to even start would be funny.

I ran into this just now, the issue seems to be a missing chrome_crashpad_handler. It's a messy fix but I was able to fix it by just copying it from Electron 17: https://github.com/InternetUnexplorer/discord-overlay/blob/647652fcbeb05f8ce12953d03495d60a84d1e101/default.nix#L20-L23.

Superseded by #198229


Non-deterministic validator

An utility which checks if module consists any possible non-deterministic operations.

https://github.com/paritytech/wasm-utils/pull/18


TransCompressionCommand Inoperable

TransCompressionCommand in parquet-tools is intended to allow translation of compression types in parquet files.  We are intending to use this functionality to debug a corrupted file, but this command fails to run at the moment entirely.
Running the following command (on the uncorrupted file):

java -jar ./parquet-tools-1.11.1.jar trans-compression ~/Downloads/part-00048-69f65188-94b5-4772-8906-5c78989240b5_00048.c000.snappy.parquet

This results in

Unknown command: trans-compression

I believe this is due to the Registry class silently catching any errors to initialize which subsequently is misinterpreted as an unknown command.
We need to:

Write a test for the TransCompressionCommand to figure out why it's showing up as unknown command

Probably expand these tests to cover all the other commands

This will then unblock our debugging work on the suspect file.

Environment: I am using parquet-tools 1.11.1 on a Mac machine running Catalina, and my parquet-tools jar was downloaded from Maven Central.
Reporter: Shelby Vanhooser
Note: This issue was originally created as PARQUET-1948. Please see the migration documentation for further details.

Gabor Szadovszky / @gszadovszky:
[~vanhooser], this feature is to be released in 1.12.0. See PARQUET-1872. So the error message is correct in the release 1.11.1. Could you please test the feature on master?

Xinli Shang / @shangxinli:
[~vanhooser], glad to see you have the interests of this tool. We have been using it by translating GZIP to ZSTD for existing parquet files. Let me know if you hit any issues.


fix: RadzenDropDownItem behavior on older versions of the blazor.js f…

Unfortunately in older versions of NET CORE 6 the framework blazor.js does not include the passage in SignalIR of the field "type" (as in the figure appears comparing the version 6.0.1 and 6.0.7). The dropdown, if used with the old version, is no longer selectable with the mouse (type='click' is always false as type is always empty).

Please check if this will not brake again this: https://github.com/radzenhq/radzen-blazor/issues/537

Please check if this will not brake again this: #537

I went back to commit a0d49f8 and also in the previous bc66795 to do a check. I don’t seem to be stopping this again: #537. The change works correctly even in case of throw
I saw the merger.
Ok thanks


Sponsorship Popup Does Not Center To User's View

The popup shows in the centre of the top of the screen. Which means, if you are far down the page because you have many tracks in your cart, you don't actually know why you can't progress towards paying.
solution:
setup the popup to be in the centre of the user's view not the centre of the top of the page

@emcniece any suggestions?

Try adding a wrapper div outside of bes_dialog and give it a fixed position. Something like:

...

That should get it to stay within viewport regardless of scroll position.

closed with: fix/106_popup_not_centered


Display only the message of UserMessageException

Exceptions of type UserMessageException are meant to show errors to the users, without any system details.
BTW, the system_errors element may display the place where these exception were created and their stack trace, which is out of scope (see here).
So, what about displaying only the error message?

See for example https://3v4l.org/Th589


release: update changelog and version

Dang, auto-formatter changed some things, I'll fix that

Dang, auto-formatter changed some things, I'll fix that


Remove v1 API (close #652)

The purpose of this PR is to remove the old v1 API.
I haven't introduced big changes in the internal classes. The main goal is just to avoid their exposure in order to refactor them more in deep in future minor releases.
Changes:

Removed builders for events
Removed getScreenState and updateWithPreviousState on ScreenView
EcommerceTransactionItem has orderId field

I've also removed the callbacks from Session (they were deprecated) but a recent discussion on Discourse make me think they are needed for hybrid apps. I'm considering to add them back (maybe with a different format).

Thanks for your pull request. Is this your first contribution to a Snowplow open source project? Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).
:memo: Please visit https://docs.snowplowanalytics.com/docs/contributing/contributor-license-agreement/ to learn more and sign.
Once you've signed, please reply here (e.g. I signed it!) and we'll verify. Thanks.


Fix default group and dir-group for RHEL-derived linux config

Description
This change fixes the issue in #152
Issues Resolved
#152
Note
This patch is hereby public domain.   I committed without standing on the left foot, or including the right magic legal statement, or whatever voodoo is required; and I don't have the gear to prove adequate testing.   And after midnight the additional costs of donated labour start to become noticeable.  :-/

@bish0polis thanks for the fix!  can you pls fix code style violations so this may be merged?

Closing due to inactivity.
If this is still wanted please reopen or open another pull request. Alternatively drop by the #sous-chefs channel on the Chef Community Slack and we'll be happy to help!
Thanks,
Sous-Chefs

It doesn't appear I can re-open it.
But if it sat for 20 months in a ready state, I don't know what re-opening's going to do.

Re-opening means there is still a desire for this feature, when we inherit cookbooks some older prs are no longer valid, this is a way of finding out features wanted/required vs features no longer wanted

Re-opening means there is still a desire for this feature

Neat.  I couldn't find the bit on the UI that allowed for re-opening.  Either I'm used to GL vs GH or an ACL prevented its inclusion.


Update Readme with a code quality badge

Consider adding this badge to show your quality report.

Full disclosure: I'm a co-author of codebeat - please share your feedback 🙃

Thanks, but I'd rather not divert reader's attention.


Add host argument to dev and test commands

Closes #193

@thecb4 would you have a moment to review changes to your testing code?

@thecb4 would you have a moment to review changes to your testing code?

I'm merging this as is, as it fixes CI build failures. @thecb4 feel free to add any comments to the diff, which we can address in a seperate PR.

I'm merging this as is, as it fixes CI build failures. @thecb4 feel free to add any comments to the diff, which we can address in a seperate PR.


[Feature] Hospital and district data for visualisation

Currently, there are fragmented and limited consolidated public data regarding the healthcare systems in South Africa. This extends beyond just knowing about a particular hospital but also relates to functions of the hospital such as specialised care, location, the estimated population in the district, and which hospitals can potentially assist in the COVID-19 pandemic.
To combat this, hospitals with specialised staff/units are available in South Africa, but this knowledge is not always accessible to the general public. Furthermore, this data seems to be in different formats, depending on the province/district and is not consolidated on one platform which makes data analysis challenging related to resource management.
I propose that we collate all the hospital data within South Africa so that we can have a clear visualisation of maximum capacity, and have data available so that other data scientists can perform assessments related to resource management, projections per district and identify what COVID-19 hotspots mean in terms of hospital capacity.
To do this, we need to start by collating data per province, related to the location of the hospital,  and capability of the hospital.

Location of hospital
a.	Name of hospital
b.	GPS coordinates
c.	Province
d.	District
e.	Subdistrict
f.	Estimated population within the district
Capacity and capability
a.	Description of the hospital (each type of hospital has its own detail)
b.	Size (differs per hospital type for example small district is 50 – 150 beds, small tertiary is 400)
c.	Operational functions (services which must be included within the type of hospital)
d.	Type of hospital (district, regional, tertiary, central or specialized)
e.	Maximum capacity
f.	Speciality services (what specialised services are at the hospital, for example, internal medicine and infectious disease units etc.)
g.	Can the hospital currently respond to COVID-19?
h.	Source confirming the hospital can currently respond to COVID-19
i.	CEO of hospital
j.	Postal address of hospital
k.	The physical address of the hospital
l.      Contact number
m.    Email

The last question “Can the hospital currently respond to COVID-19?” is challenging to answer unless public statements have been made. We know that the following hospitals are/were highlighted as quarantine hospitals.
Province	HosptName	                          GPSCoordinant	         TotalHospitalBeds
Limpopo	Polokwane Hospital	         23.8968° S, 29.4574° E	450
Mpumalanga	Rob Ferreira Hospital	25.4769° S, 30.9709° E	461
Gauteng	Charlotte Maxeke Hospital	26.1760° S, 28.0439° E	1088
Gauteng	Steve Biko Academic Hospital  25.7303° S, 28.2044° E	832
Gauteng	Tembisa Hospital	                  25.9830° S, 28.2382° E	840
KwaZulu Natal	Grey's Hospital	          29.5795° S, 30.3643° E	530
North West	Klerksdorp Hospital	26.8789° S, 26.6633° E	890
Northern Cape	Kimberely Hospital	28.7462° S, 24.7730° E	287
Free State	Pelonomi Hospital	         29.1396° S, 26.2447° E	469
Eastern Cape	Livingstone Hospital	 33.9254° S, 25.5697° E	541
Western Cape	Tygerberg Hospital	          33.9106° S, 18.6123° E	1384
We also know, based on articles written about each one of these hospitals, that they experience a heavy load of patients throughout the year – irrespective of COVID-19.
The data that is consolidated can be used as a dashboard for the general public to see where they can go in terms of public healthcare, and additionally, other scientists can use this data to make contextually relevant inferences if they have this data and add it to their analysis.

16_03_2020_KZNHospitalDatabase.xlsx

See health_system_za_public* files in data/
populating with a few examples is needed.

The Department of Health for each South African province has this data, but it is scattered and there is not consistent format between the different departments. Consolidating this information as a dataset will be valuable for visualisation as well as for analytics related to capacity and systems.

See
General: https://github.com/dsfsi/covid19za/blob/master/data/health_system_za_public_hospitals.csv
Detailed: https://github.com/dsfsi/covid19za/blob/master/data/health_system_za_public_hospitals_extended_details.csv
Contacts: https://github.com/dsfsi/covid19za/blob/master/data/health_system_za_public_hospitals_contacts.csv

@MikeMcMalace See latest commit to branch Hospital_data

Think now getting per hospital ICU is next
Also thanks @elolelo

Thank you Nompumelelo, this is great!

You might also want to dig into https://opendataza.gitbook.io/toolkit/open-data-resources/healthcare-data-resources
@elolelo and @MikeMcMalace

Hey @MikeMcMalace how are we doing here?

Her @vukosim , currently with regard to the district data we have 351 hospitals, with complete data except for 2 with missing geodata. Unfortunately, name changes etc. complicate this a bit.
20_03_2020_CurrentHospitalData_12_05pm.xlsx

@MikeMcMalace please see the Hospital Data branch. I would suggest you work on that as it also has info on where the data came from as per @elolelo

See https://w.wiki/Ksn

Fantastic! Is this linked?

The data on that link is from WIkiData. So wikidata would have to be updated. We have a volunteer from Codebridge who says they will update during the week.

and apologies if you already have this, but there's quite detailed descriptions of each facility (down to the ward level? not sure about capacity) here on the NIDS site:
https://dd.dhmis.org/orgunits.html?file=NIDS Integrated&source=nids

@MikeMcMalace Hey Mike, did you get a chance to check this out?

Yes. Unfortunately, there are some issues in the data (unless they updated it within the last day). The biggest problem is district information and bed capacity. Each district hospital (according to the South African DoH) is categorised as either small, medium or large. In addition to this, each district has its own capacity (or estimated capacity). The hospital data have been updated, and thank you @elolelo for your valuable contribution into this matter. The problem right now is we want to have certain information on a map, that details the hospital, what it can and cannot do, and have information about the district included in it.
Something similar to this:  https://codebridge-za.github.io/seeking-shelter/index.html
but with the details of this: https://www.healthestablishments.org.za/Home/Facility
and with the added district data and information about the hospital capacity.
The reason for this is we know which hospitals are COVID-19 ready. We also have amazing tools like the links here, which are very useful, but not if you have a medical emergency other than COVID-19, or COVID-19. When doing this search, I was shocked at how much digging one needs to do, not just for South Africa, but in general in Africa to find this information. The idea is to consolidate information in a way that is useful and usable to a person who is not a tech enthusiast, and who just has a cellphone.

This is being addressed in #115


Update this page to reset credentials for AAD enabled cluster

Please update this page to include the steps to reset credentials for Azure Active Directory enabled cluster. https://docs.microsoft.com/en-us/cli/azure/aks?view=azure-cli-latest#az-aks-update-credentials

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 0b842222-f6ac-5266-227f-7e400610e07b
Version Independent ID: 429cd3bf-2691-f633-2e0d-e70e5d8d0e03
Content: Reset the credentials for an Azure Kubernetes Service (AKS) cluster
Content Source: articles/aks/update-credentials.md
Service: container-service
GitHub Login: @iainfoulds
Microsoft Alias: iainfou

@sampath2121 thanks for reaching out. If you still have feedback feel free to let us know or reopen this issue.

@MicahMcKittrick-MSFT Any news on this ? Is it even possible to do so? I have a customer who requires to update the credentials so that he can integrate Azure AD into his cluster.


TLS/ SSL Support - FTP FileCrawler

Hi,
I tried to initialize a new FTP Crawler and got this messages:
[info] filecrawler initialized
[error] 'NoneType' object has no attribute 'sendall'
[error] 550 SSL/TLS required on the control channel
[error] error connecting to Ftp server on

Thank you very much for your efforts!

Hi @lucajung , check out latest release


Cannot save workspace: path must be a string or Buffer

This issue pertains to the following package(s):

[x] GraphQL Playground - Electron App

What OS and OS version are you experiencing the issue(s) on?
Windows 10 x64
What version of graphql-playground(-electron/-middleware) are you experiencing the issue(s) on?
1.6.0
What is the expected behavior?
Being able to save workspace so I can reopen it next time instead of entering endpoint URL every time. I would assume that if I create .graphqlconfig file by hand and use that, it would probably work, but generally, I don't have any other use for that file so I don't it floating in a project folder.
What is the actual behavior?
Clicking on Workspace -> Save seemingly does nothing. Opening developer console I can see the output.
handleSaveConfig called
calling this.props.onSaveConfig undefined
Uncaught (in promise) TypeError: path must be a string or Buffer
    at Object.fs.openSync (fs.js:646:18)
    at Object.module.(anonymous function) [as openSync] (ELECTRON_ASAR.js:172:20)
    at Object.fs.writeFileSync (fs.js:1303:33)
    at Object.App._this.saveConfig [as onSaveConfig]

Might be some NodeJS incompatibility? I am currently running v10.4.1.
What steps may we take to reproduce the behavior?

Open the app
Select "Endpoint URL" as a new workspace
Enter correct URL
After opening workspace, try to click Workspace -> Save

Having similar issue.
fs.js:646 Uncaught (in promise) TypeError: path must be a string or Buffer
    at Object.fs.openSync (fs.js:646:18)
    at Object.module.(anonymous function) [as openSync] (ELECTRON_ASAR.js:166:20)
    at Object.fs.writeFileSync (fs.js:1303:33)
    at Object.App._this.saveConfig [as onSaveConfig]

If you do not open a workspace from a file, you won't be able to save it. There is no functionality "Save Workspace As" now.
This is related to #892, but not exactly the same, because this issue is about the workspace configuration file and that one about the graphql query file.
If you trigger saving and the current tab was not opened from a file, the workspace configuration will be saved instead. The error on the console is caused by a bug in saveConfig. Instead of checking if the workspace was opened from a file and either showing an error or choosing a name, it just uses the undefined configPath value to call writeFileSync. Obviously, the writeFileSync retaliates:
// v1.8.10/packages/graphql-playground-electron/src/renderer/components/App.tsx, line 490
fs.writeFileSync(_this.state.configPath, configString);

The code should be more robust in supporting the scenario that you start the playground with an endpoint only, not by opening a workspace file. "Save As" for a workspace would be nice.


A lightweight version of Subversion is already installed on FreeBSD a…

…s svnlite (the only difference from normal Subversion use is that the command name is svnlite).

Thanks!


Completely refactors logic for days ago

I found a bug when the date happened between >24h and <48h ago but that date was not yesterday.
For instance, when the date was 34h ago (= 1day and 10h ago) and on the day before yesterday, it returns 10 hours ago instead of 1 day ago. There was a fix for a similar issue some months ago but it still fails on this case.
However, instead of fixing just this, I refactored the whole thing because I found a deeper bug. IMO, the existent logic fails when the dates are more than 24h apart. In fact, when someone says that something happened 2 days ago, they don't really mean that it happened between 48h and 72h ago. They mean that it happened on the day before yesterday.
That's the logic that we implemented and we'll be using at Hole19 from now on.

+1 :D

My apologies for not getting to this sooner. Looks like some tremendous work. I will do a few more tests and then hopefully merge this in!

A big thank you for refactoring this. I am going to take a little while to repair any broken tests.
Best regards,
-Matt

Thanks Matt!


Provide pre-compiled binaries for 64-bit ARMv8

Hello, I try to install sharp on odroid C2 (Amlogic ARM® Cortex®-A53(ARMv8) 2Ghz quad core CPU)  with Ubuntu 16.04
uname -a Linux odroid64 3.14.65-72 #1 SMP PREEMPT Wed Jul 20 21:51:35 BRT 2016 aarch64 aarch64 aarch64 GNU/Linux
nodejs -v v4.2.6
npm -v 3.5.2
When I run npm install sharp i have the error: libvips-cpp.so: error adding symbols: File in wrong format. AFAIF, this error is occurred when mixing 32 and 64 bit libraries together. I were installed libvips-dev from repo and run 'npm install sharp --sharp-cxx11' but have the same error.

Hello, the pre-built binaries for ARM are created on a 32-bit ARMv6 machine, a 1st gen Raspberry Pi, so your suspicion about this being a 32 vs 64 bit problem is probably correct.
Are you able to try with a 32-bit version of node, perhaps one prepared for the ARMv7 arch, to help prove this (I think ARMv8 CPUs can run in an ARMv7 "mode")?

Thank you for answer, Lovell. I can try to install sharp with 32 bit node in the weekend. Right now the device is in use with default system environment and too much of code expects 64 bit. And a little bit problem that Hardkernels team don't provided 32-bit system for this computer. At weekend I can try to build node, glib and g++ for 32 bit and check your suggestions.
But, I have another idea. What do you think if I try to get sharp sources from github and change  sharp/packaging/arm/build.sh
export PATH="${PATH}:${TARGET}/bin"
export CPPFLAGS="-I${TARGET}/include"
export LDFLAGS="-L${TARGET}/lib"
from 32 destination to 64 bit? Or may be you know some options to change this settings in npm command line? My tries to use "env LDFLAGS="-Lpath/to/lib64" were failed.

If you'd like to experiment with the packaging scripts, the CFLAGS and CXXFLAGS here need to include
-march=armv8-a+crc for ARMv8 cross-compilation.
What I can do is ensure the pre-compiled binaries are only used for ARMv6 and ARMv7 machines via something like process.config.variables.arm_version. Then we can think about providing ARMv8 binaries.

Thank you very much, Lovell, you proposal is very useful and now sharp is installed successfully.
Installation instructions for Arm8 on Ubuntu 16.04:
Get sources
mkdir ~/src
cd ~/src
git clone https://github.com/lovell/sharp.git
Change platform flag:
sudo vi sharp/packaging/arm/build.sh
delete line 16,17 contains:
export CFLAGS="-O3"
export CXXFLAGS="-O3"
and add
export CPPFLAGS="-march=armv8-a+crc"
Install libvips and its dependencies:
cd sharp
sudo sh preinstall.sh
Install sharp
cd ~/target_project/node_modules
sudo npm install ~/src/sharp/ --sharp-cxx11=1 --build-from-source
Pre-compiled binaries for Arm8 64:
sharp-o.zip

I notice you used the deprecated preinstall.sh script in there to install libvips globally, which is probably why this is now working. From the commands you list, it looks like the scripts inside the packaging directory aren't being used.
Let's leave this task open whilst I sort out the 32 vs 64 bit ARM binaries and, more importantly, document this process to make things less confusing.
As an aside, it always makes me happy to learn about people using sharp on ARM-based kit :)

I notice you used the deprecated preinstall.sh script in there to install libvips globally
Yes, it's not a problem for me if libvips is installed globally.

From the commands you list, it looks like the scripts inside the packaging directory aren't being used
The 'preinstall.sh' were looked by me superficially and I overlook that this script doesn't fire build.sh from /packaging/arm directory. If 'build.sh' was being running manually,  the error is occurred during the 'make' is performing:
tar (child): /arm/libvips-8.3.1-arm.tar.gz: Cannot open: No such file or directory.
I didn't deeply investigate this problem, because all works fine now. But if you need more details about this, please provide information what should I do to help you.

it always makes me happy to learn about people using sharp on ARM-based kit :)
Your library is fast and clear, arm  device is small and powerful, why not to splice this sweet couple :)

Commit 475f0bf on the pencil branch adds support for ARMv6, v7 and v8 specific pre-compiled binaries.
Running npm install lovell/sharp#pencil should now "just work". Are you able to test this on a 64-bit ARMv8 machine, preferably where libvips is not installed?

I'm on vocation now and haven't Odroid within touch. I shall come to my city at the end of august and when it will happen, I shall do installation on clean system and write about result here.

@rybalchenkoserg Thanks for the update, enjoy your travels!

v0.16.0 is now available via npm and provides separate pre-compiled binaries for ARMv6, v7 and v8. I'll leave this open for further testing and feedback.

Closing due to inactivity but please feel free to re-open if the ARMv8 binaries currently provided aren't working.


Add ability to store multiple cells per HDF5

Fixes #61

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

30 of 31 (96.77%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.2%) to 81.472%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

batdata/data.py
30
31
96.77%

Totals

Change from base Build<PHONE_NUMBER>:
0.2%

Covered Lines:
664

Relevant Lines:
815

💛  - Coveralls

Good point a out the potential annoyance of having to lookup prefixes before reading. How about I add an 'auto' option to the read which will grab data belonging to the first prefix then warn if there are more than one available?

I think the 'auto' idea is good, my question there is: how is the "first" prefix identified? I mean, there's no inherent order to them, right?
I also think there should be a 'all' option, which basically loads all prefixes available. If you think it's wise, the 'all' can also output a warning for memory constraints.

@victorventuri , I opted to add a "load by index" option to supply the need for "any." prefix=0 will load the first prefix, alphabetically.
Does this give you the features you'd need?


send() requires a tokio timer before the future is polled

If a request_timeout is set, creating a future for a request has to be called within a tokio runtime, because it calls tokio::timer::delay_for(). This was surprising to at least one person, who tried to build the Client and Request outside of an async context and then use tokio::Runtime::block_on() on the resulting Future.
Could it be possible to defer calling delay_for() until the request future is first polled, to allow the Request to be built outside an async context?

I have a similar issue where a tokio stream that constructs a reqwest::Response from an http::Response is never polled.
Adding a random get request or tokio::time::delay_for(std::time::Duration::new(1, 0)); before the stream starts in the same tokio runtime fixed the issue.
I am not quite sure if the issue is from reqwest or tokio or maybe me, but the issue above sounds pretty similar.

I've also faced similar problem - I try to create and run the future for request independently and if I use timeout - I get a panic. Here is a minimal example of the problem:
This works:
fn main() {
    let rt = tokio::runtime::Runtime::new().unwrap();
    rt.block_on(reqwest::Client::new().get("https://google.com").send()).unwrap();
}

This panics:
fn main() {
    let rt = tokio::runtime::Runtime::new().unwrap();
    rt.block_on(reqwest::Client::new().get("https://google.com").timeout(std::time::Duration::from_secs(1)).send()).unwrap();
}


JNA issue on MacOS after publishing to TestFlight

Hello,
Thank you for so useful library.
I've faced with issue on MacOS, presumably after update of MacOS and xCode. It's shows an JNA error and then crashes. Here is the text of error:
/Users/user/Library/Containers/com.site.myapp/Data/tmp/jna9436987196270821361.tmp: dlopen(/Users/user/Library/Containers/com.site.myapp/Data/tmp/jna9436987196270821361.tmp, 0x0001): tried:
/Users/user/Library/Containers/com.site.myapp/Data/tmp/jna9436987196270821361.tmp (code signature in <BEFEF0FB-C7C5-305A-8B79-ED4851A90E7F> '/Users/user/Library/Containers/com.site.myapp/Data/tmp/jna9436987196270821361.tmp' not valid for use in process: library load disallowed by system policy),
/System/Volumes/Preboot/Cryptexes/OS/Users/user/Library/Containers/com.site.myapp/Data/tmp/jna9436987196270821361.tmp (no such file),
'/Users/user/Library/Containers/com.site.myapp/Data/tmp/jna9436987196270821361.tmp' not valid for use in process: library load disallowed by system policy

Here is a sample code:
    val result = remember { mutableStateOf(listOf<Path>()) }
    Text(text = result.value.joinToString("\n"))

    val directoryLauncher = rememberDirectoryPickerLauncher(title = Strings.current.app_name) { directory ->
        directory?.file?.absolutePath?.let { result.value = listOf(Path(it)) }
    }
    Button(
        onClick = {
            directoryLauncher.launch()
        },
    ) {
        Text(text = "Open")
    }

Library version 0.8.7.
Could you please help with this issue?
Thank you.

Hello @bartwell! Thank you for your feedback!
I'm trying to reproduce your issue.

What is your macOS and Xcode version?
Does the sample-compose work on your device?

@vinceglb, sorry for not mentioning the key detail earlier: this issue occurs only in sandbox mode after publishing to TestFlight, so I can't reproduce it in the sample application.
I am using macOS Sequoia 15.1 (24B83) and Xcode 16.1 (16B40).

Thank you for the information. Interesting 🤔
The bad news here is I don't have anymore an Apple Developer subscription to test the behavior.
I found this thread that can help us fix the problem:
https://groups.google.com/g/jna-users/c/Bws1h060faA

Thank you for the link. It seems like the solution might be:
System.setProperty("jna.nounpack", "true");
System.setProperty("jna.boot.library.path", "<path to native libs>");

However, could you clarify what path should be specified here? As far as I can tell, the directories inside /Applications/MyApp.app/Contents/ are writable only by root.


Mention replacement can occur recursively

Due to how the mentions of someone's names may occur, there is a chance that the rendering may parse mentions recursively. For example, if you replace @mentions with mentions then try to highlight that, yet the code is @[mentions] then you may end up replacing things over and over.

We have two method to slove this:

return value of function label could be not contain return value encode
use replace by regex instead String.prototype.replace in ngModel.$parsers.push...

Hello for those looking for a solution we used this snippet :
var regex = new RegExp("(?<![\\w])"+_this.label(mention)+"(?![\\w])", 'g');
return value = value.replace(regex, _this.encode(mention));