Add Ascend NPU as a backend

What does this PR do?
Overview
🚀This PR enables the users of torhtune to leverage the Ascend NPU for better performance in inferencing when GPU device is not available.
This PR primarily addresses the initial refactoring of device-independent code. In upcoming changes, we’ll focus on further adjustments, using NPU as an example to refine each recipe and complete the remaining device-independent modifications. For now, this PR only touches on recipe lora_finetune_single_device and full_finetune_single_device.
For more details, see: [#1797].
Environment

OS: ubuntu 20.04
NPU: Atlas 300T A2
CANN: 8.0.RC2
torch-npu: 2.4.0 rc1
torch: 2.4.0

Note
To properly install CANN, see [here] for more details.
The version of torch-npu should match that of torch, see [here] for more details.
In addition, torch_npu has a pre-release version, 2.4.0 RC1, which is also the basis for this test. For more information, please visit [here].
Examples
To start with, the library torch_npu should be correctly installed and imported. Part of the codes are showed below:
torchtune/utils/_device_support.py:
# Copyright (c) Meta Platforms, Inc. and affiliates.
# All rights reserved.
#
# This source code is licensed under the BSD-style license found in the
# LICENSE file in the root directory of this source tree.
import torch

def is_torch_npu_available():
    try:
        import torch_npu # noqa: F401
    except ImportError:
        return False
    return torch.npu.is_available()

Plus, there are some other places of the codes might be adjusted, which won't be too much.
Feel free to leave comments to guide me in further improvements 😊.

Hi @ebsmothers, @RdoubleA:
I hope you’re doing well! Could you please help me review my code?  I would really appreciate it if you could take a look and share any feedback or suggestions. Thank you so much in advance for your time and support! 😊
Best regards

distributed training seems to have problems @noemotiovon

distributed training seems to have problems e.g qat_distributed @noemotiovon function torchtune/training/_distributed.py/load_from_full_model_state_dict sharded_meta_param.device_mesh.device_type is cpu when loading model which would raise error. Are you willing to connect me through my github profile email? maybe we can discuss how to support ascend npu

I would be very happy to! I will contact you via email.

distributed training seems to have problems e.g qat_distributed @noemotiovon function torchtune/training/_distributed.py/load_from_full_model_state_dict sharded_meta_param.device_mesh.device_type is cpu when loading model which would raise error. Are you willing to connect me through my github profile email? maybe we can discuss how to support ascend npu

I would be very happy to! I will contact you via email.

@noemotiovon through 126 email thanks. Looking forward to your email.

Basic Usage Test

A single-device fine-tuning process was performed on the Llama 3.1 8B model using the LoRA (Low-Rank Adaptation) technique.

Recipe: lora_finetune_single_device

Model: Meta-Llama-3.1-8B-Instruct

Config:
# Config for single device LoRA finetuning in lora_finetune_single_device.py
# using a Llama3.1 8B Instruct model
#
# This config assumes that you've run the following command before launching
# this run:
#   tune download meta-llama/Meta-Llama-3.1-8B-Instruct --output-dir /tmp/Meta-Llama-3.1-8B-Instruct --ignore-patterns "original/consolidated.00.pth"
#
# To launch on a single device, run the following command from root:
#   tune run lora_finetune_single_device --config llama3_1/8B_lora_single_device
#
# You can add specific overrides through the command line. For example
# to override the checkpointer directory while launching training
# you can run:
#   tune run lora_finetune_single_device --config llama3_1/8B_lora_single_device checkpointer.checkpoint_dir=<YOUR_CHECKPOINT_DIR>
#
# This config works only for training on single device.

# Model Arguments
model:
  _component_: torchtune.models.llama3_1.lora_llama3_1_8b
  lora_attn_modules: ['q_proj', 'v_proj']
  apply_lora_to_mlp: False
  apply_lora_to_output: False
  lora_rank: 8
  lora_alpha: 16
  lora_dropout: 0.0

# Tokenizer
tokenizer:
  _component_: torchtune.models.llama3.llama3_tokenizer
  path: /home/lcg/.cache/modelscope/hub/LLM-Research/Meta-Llama-3___1-8B-Instruct/original/tokenizer.model
  max_seq_len: null

checkpointer:
  _component_: torchtune.training.FullModelHFCheckpointer
  checkpoint_dir: /home/lcg/.cache/modelscope/hub/LLM-Research/Meta-Llama-3___1-8B-Instruct
  checkpoint_files: [
    model-00001-of-00004.safetensors,
    model-00002-of-00004.safetensors,
    model-00003-of-00004.safetensors,
    model-00004-of-00004.safetensors
  ]
  recipe_checkpoint: null
  output_dir: /home/lcg/tmp/torchtune/Meta-Llama-3.1-8B-Instruct/
  model_type: LLAMA3
resume_from_checkpoint: False
save_adapter_weights_only: False

# Dataset and Sampler
dataset:
  _component_: torchtune.datasets.alpaca_cleaned_dataset
seed: null
shuffle: True
batch_size: 30

# Optimizer and Scheduler
optimizer:
  _component_: torch.optim.AdamW
  fused: False
  weight_decay: 0.01
  lr: 3e-4
lr_scheduler:
  _component_: torchtune.modules.get_cosine_schedule_with_warmup
  num_warmup_steps: 100

loss:
  _component_: torchtune.modules.loss.CEWithChunkedOutputLoss

# Training
epochs: 1
max_steps_per_epoch: null
gradient_accumulation_steps: 64
compile: False

# Logging
output_dir: /home/lcg/tmp/torchtune/Meta-Llama-3.1-8B-Instruct/lcg-test
metric_logger:
  _component_: torchtune.training.metric_logging.DiskLogger
  log_dir: ${output_dir}
log_every_n_steps: 1
log_peak_memory_stats: False

# Environment
device: npu
dtype: bf16

# Activations Memory
enable_activation_checkpointing: True
enable_activation_offloading: False

# Profiler (disabled)
profiler:
  _component_: torchtune.training.setup_torch_profiler
  enabled: False

  #Output directory of trace artifacts
  output_dir: /home/lcg/tmp/torchtune/Meta-Llama-3.1-8B-Instruct/lcg-test

  #`torch.profiler.ProfilerActivity` types to trace
  cpu: True
  cuda: True

  #trace options passed to `torch.profiler.profile`
  profile_memory: False
  with_stack: False
  record_shapes: True
  with_flops: False

  # `torch.profiler.schedule` options:
  # wait_steps -> wait, warmup_steps -> warmup, active_steps -> active, num_cycles -> repeat
  wait_steps: 5
  warmup_steps: 5
  active_steps: 2
  num_cycles: 1

Logs:
INFO:torchtune.utils._logging:Running LoRAFinetuneRecipeSingleDevice with resolved config:

batch_size: 30
checkpointer:
  _component_: torchtune.training.FullModelHFCheckpointer
  checkpoint_dir: /home/lcg/.cache/modelscope/hub/LLM-Research/Meta-Llama-3___1-8B-Instruct
  checkpoint_files:
  - model-00001-of-00004.safetensors
  - model-00002-of-00004.safetensors
  - model-00003-of-00004.safetensors
  - model-00004-of-00004.safetensors
  model_type: LLAMA3
  output_dir: /home/lcg/tmp/torchtune/Meta-Llama-3.1-8B-Instruct/
  recipe_checkpoint: null
compile: false
dataset:
  _component_: torchtune.datasets.alpaca_cleaned_dataset
device: npu
dtype: bf16
enable_activation_checkpointing: true
enable_activation_offloading: false
epochs: 1
gradient_accumulation_steps: 64
log_every_n_steps: 1
log_peak_memory_stats: false
loss:
  _component_: torchtune.modules.loss.CEWithChunkedOutputLoss
lr_scheduler:
  _component_: torchtune.modules.get_cosine_schedule_with_warmup
  num_warmup_steps: 100
max_steps_per_epoch: null
metric_logger:
  _component_: torchtune.training.metric_logging.DiskLogger
  log_dir: /home/lcg/tmp/torchtune/Meta-Llama-3.1-8B-Instruct/lcg-test
model:
  _component_: torchtune.models.llama3_1.lora_llama3_1_8b
  apply_lora_to_mlp: false
  apply_lora_to_output: false
  lora_alpha: 16
  lora_attn_modules:
  - q_proj
  - v_proj
  lora_dropout: 0.0
  lora_rank: 8
optimizer:
  _component_: torch.optim.AdamW
  fused: false
  lr: 0.0003
  weight_decay: 0.01
output_dir: /home/lcg/tmp/torchtune/Meta-Llama-3.1-8B-Instruct/lcg-test
profiler:
  _component_: torchtune.training.setup_torch_profiler
  active_steps: 2
  cpu: true
  cuda: true
  enabled: false
  num_cycles: 1
  output_dir: /home/lcg/tmp/torchtune/Meta-Llama-3.1-8B-Instruct/lcg-test
  profile_memory: false
  record_shapes: true
  wait_steps: 5
  warmup_steps: 5
  with_flops: false
  with_stack: false
resume_from_checkpoint: false
save_adapter_weights_only: false
seed: null
shuffle: true
tokenizer:
  _component_: torchtune.models.llama3.llama3_tokenizer
  max_seq_len: null
  path: /home/lcg/.cache/modelscope/hub/LLM-Research/Meta-Llama-3___1-8B-Instruct/original/tokenizer.model

DEBUG:torchtune.utils._logging:Setting manual seed to local seed<PHONE_NUMBER>. Local seed is seed + rank =<PHONE_NUMBER> + 0
Writing logs to /home/lcg/tmp/torchtune/Meta-Llama-3.1-8B-Instruct/lcg-test/log_1728874769.txt
INFO:torchtune.utils._logging:Model is initialized with precision torch.bfloat16.
INFO:torchtune.utils._logging:Memory stats after model init:
        NPU peak memory allocation: 16.98 GiB
        NPU peak memory reserved: 17.00 GiB
        NPU peak memory active: 16.98 GiB
INFO:torchtune.utils._logging:Tokenizer is initialized from file.
INFO:torchtune.utils._logging:Optimizer and loss are initialized.
INFO:torchtune.utils._logging:Loss is initialized.
Using the latest cached version of the dataset since yahma/alpaca-cleaned couldn't be found on the Hugging Face Hub
WARNING:datasets.load:Using the latest cached version of the dataset since yahma/alpaca-cleaned couldn't be found on the Hugging Face Hub
Found the latest cached dataset configuration 'default' at /home/lcg/.cache/huggingface/datasets/yahma___alpaca-cleaned/default/0.0.0/12567cabf869d7c92e573c7c783905fc160e9639 (last modified on Fri Oct 11 01:21:44 2024).
WARNING:datasets.packaged_modules.cache.cache:Found the latest cached dataset configuration 'default' at /home/lcg/.cache/huggingface/datasets/yahma___alpaca-cleaned/default/0.0.0/12567cabf869d7c92e573c7c783905fc160e9639 (last modified on Fri Oct 11 01:21:44 2024).
INFO:torchtune.utils._logging:Dataset and Sampler are initialized.
INFO:torchtune.utils._logging:Learning rate scheduler is initialized.
WARNING:torchtune.utils._logging: Profiling disabled.
INFO:torchtune.utils._logging: Profiler config after instantiation: {'enabled': False}
  0%|                                                                                | 0/26 [00:00<?, ?it/s]/home/lcg/miniconda3/envs/torchtune/lib/python3.10/site-packages/torch/utils/checkpoint.py:1399: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead.
  with device_autocast_ctx, torch.cpu.amp.autocast(**cpu_autocast_kwargs), recompute_context:  # type: ignore[attr-defined]
  4%|█████▌                                                                                | 1/26 [07:09<2:59:1|1|Loss: 1.7533539533615112:   4%|████▍                                                                                | 1/26 [07:09<2:59:1|1|Loss: 1.7533539533615112:   8%|████████▊                                                                                | 2/26 [14:09<2:49:1|2|Loss: 1.7825285196304321:   8%|████████▊                                                                                | 2/26 [14:09<2:49:1|2|Loss: 1.7825285196304321:  12%|█████████████▎                                                                                | 3/26 [21:17<2:43:1|3|Loss: 1.7610299587249756:  12%|█████████████▎                                                                                | 3/26 [21:17<2:43:1|3|Loss: 1.7610299587249756:  15%|█████████████████▋                                                                                | 4/26 [28:28<2:36:1|4|Loss: 1.7874119281768799:  15%|█████████████████▋                                                                                | 4/26 [28:28<2:36:1|4|Loss: 1.7874119281768799:  19%|██████████████████████                                                                                | 5/26 [35:36<2:29:1|5|Loss: 1.7903798818588257:  19%|██████████████████████                                                                                | 5/26 [35:36<2:29:1|5|Loss: 1.7903798818588257:  23%|██████████████████████████▌                                                                                | 6/26 [42:52<2:23:1|6|Loss: 1.776786208152771:  23%|██████████████████████████▊                                                                                | 6/26 [42:52<2:23:1|6|Loss: 1.776786208152771:  27%|███████████████████████████████▏                                                                                | 7/26 [49:45<2:14:1|7|Loss: 1.7698196172714233:  27%|██████████████████████████████▉                                                                                | 7/26 [49:45<2:14:29, 424.69s/it]
 *  History restored

(torchtune) (base) lcg@lcg-docker:~/github/torchtune$
(torchtune) (base) lcg@lcg-docker:~/github/torchtune$
(torchtune) (base) lcg@lcg-docker:~/github/torchtune$ tune run lora_finetune_single_device --config my_custom_config.yaml
INFO:torchtune.utils._logging:Running LoRAFinetuneRecipeSingleDevice with resolved config:

batch_size: 30
checkpointer:
  _component_: torchtune.training.FullModelHFCheckpointer
  checkpoint_dir: /home/lcg/.cache/modelscope/hub/LLM-Research/Meta-Llama-3___1-8B-Instruct
  checkpoint_files:
  - model-00001-of-00004.safetensors
  - model-00002-of-00004.safetensors
  - model-00003-of-00004.safetensors
  - model-00004-of-00004.safetensors
  model_type: LLAMA3
  output_dir: /home/lcg/tmp/torchtune/Meta-Llama-3.1-8B-Instruct/
  recipe_checkpoint: null
compile: false
dataset:
  _component_: torchtune.datasets.alpaca_cleaned_dataset
device: npu
dtype: bf16
enable_activation_checkpointing: true
enable_activation_offloading: false
epochs: 1
gradient_accumulation_steps: 64
log_every_n_steps: 1
log_peak_memory_stats: false
loss:
  _component_: torchtune.modules.loss.CEWithChunkedOutputLoss
lr_scheduler:
  _component_: torchtune.modules.get_cosine_schedule_with_warmup
  num_warmup_steps: 100
max_steps_per_epoch: null
metric_logger:
  _component_: torchtune.training.metric_logging.DiskLogger
  log_dir: /home/lcg/tmp/torchtune/Meta-Llama-3.1-8B-Instruct/lcg-test
model:
  _component_: torchtune.models.llama3_1.lora_llama3_1_8b
  apply_lora_to_mlp: false
  apply_lora_to_output: false
  lora_alpha: 16
  lora_attn_modules:
  - q_proj
  - v_proj
  lora_dropout: 0.0
  lora_rank: 8
optimizer:
  _component_: torch.optim.AdamW
  fused: false
  lr: 0.0003
  weight_decay: 0.01
output_dir: /home/lcg/tmp/torchtune/Meta-Llama-3.1-8B-Instruct/lcg-test
profiler:
  _component_: torchtune.training.setup_torch_profiler
  active_steps: 2
  cpu: true
  cuda: true
  enabled: false
  num_cycles: 1
  output_dir: /home/lcg/tmp/torchtune/Meta-Llama-3.1-8B-Instruct/lcg-test
  profile_memory: false
  record_shapes: true
  wait_steps: 5
  warmup_steps: 5
  with_flops: false
  with_stack: false
resume_from_checkpoint: false
save_adapter_weights_only: false
seed: null
shuffle: true
tokenizer:
  _component_: torchtune.models.llama3.llama3_tokenizer
  max_seq_len: null
  path: /home/lcg/.cache/modelscope/hub/LLM-Research/Meta-Llama-3___1-8B-Instruct/original/tokenizer.model

DEBUG:torchtune.utils._logging:Setting manual seed to local seed<PHONE_NUMBER>. Local seed is seed + rank =<PHONE_NUMBER> + 0
Writing logs to /home/lcg/tmp/torchtune/Meta-Llama-3.1-8B-Instruct/lcg-test/log_1728878132.txt
INFO:torchtune.utils._logging:Model is initialized with precision torch.bfloat16.
INFO:torchtune.utils._logging:Memory stats after model init:
        NPU peak memory allocation: 16.98 GiB
        NPU peak memory reserved: 17.00 GiB
        NPU peak memory active: 16.98 GiB
INFO:torchtune.utils._logging:Tokenizer is initialized from file.
INFO:torchtune.utils._logging:Optimizer and loss are initialized.
INFO:torchtune.utils._logging:Loss is initialized.
Using the latest cached version of the dataset since yahma/alpaca-cleaned couldn't be found on the Hugging Face Hub
WARNING:datasets.load:Using the latest cached version of the dataset since yahma/alpaca-cleaned couldn't be found on the Hugging Face Hub
Found the latest cached dataset configuration 'default' at /home/lcg/.cache/huggingface/datasets/yahma___alpaca-cleaned/default/0.0.0/12567cabf869d7c92e573c7c783905fc160e9639 (last modified on Fri Oct 11 01:21:44 2024).
WARNING:datasets.packaged_modules.cache.cache:Found the latest cached dataset configuration 'default' at /home/lcg/.cache/huggingface/datasets/yahma___alpaca-cleaned/default/0.0.0/12567cabf869d7c92e573c7c783905fc160e9639 (last modified on Fri Oct 11 01:21:44 2024).
INFO:torchtune.utils._logging:Dataset and Sampler are initialized.
INFO:torchtune.utils._logging:Learning rate scheduler is initialized.
WARNING:torchtune.utils._logging: Profiling disabled.
INFO:torchtune.utils._logging: Profiler config after instantiation: {'enabled': False}
  0%|                                                                                | 0/26 [00:00<?, ?it/s]/home/lcg/miniconda3/envs/torchtune/lib/python3.10/site-packages/torch/utils/checkpoint.py:1399: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead.
  with device_autocast_ctx, torch.cpu.amp.autocast(**cpu_autocast_kwargs), recompute_context:  # type: ignore[attr-defined]
1|26|Loss: 1.427944302558899: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 26/26 [3:04:04<00:00, 418.75s/it]INFO:torchtune.utils._logging:Starting checkpoint save...
INFO:torchtune.utils._logging:Model checkpoint of size 4.98 GB saved to /home/lcg/tmp/torchtune/Meta-Llama-3.1-8B-Instruct/hf_model_0001_0.pt
INFO:torchtune.utils._logging:Model checkpoint of size 5.00 GB saved to /home/lcg/tmp/torchtune/Meta-Llama-3.1-8B-Instruct/hf_model_0002_0.pt
INFO:torchtune.utils._logging:Model checkpoint of size 4.92 GB saved to /home/lcg/tmp/torchtune/Meta-Llama-3.1-8B-Instruct/hf_model_0003_0.pt
INFO:torchtune.utils._logging:Model checkpoint of size 1.17 GB saved to /home/lcg/tmp/torchtune/Meta-Llama-3.1-8B-Instruct/hf_model_0004_0.pt
INFO:torchtune.utils._logging:Adapter checkpoint of size 0.01 GB saved to /home/lcg/tmp/torchtune/Meta-Llama-3.1-8B-Instruct/adapter_0.pt
INFO:torchtune.utils._logging:Adapter checkpoint of size 0.01 GB saved to /home/lcg/tmp/torchtune/Meta-Llama-3.1-8B-Instruct/adapter_model.bin
INFO:torchtune.utils._logging:Adapter checkpoint of size 0.00 GB saved to /home/lcg/tmp/torchtune/Meta-Llama-3.1-8B-Instruct/adapter_config.json
INFO:torchtune.utils._logging:Saving final epoch checkpoint.
INFO:torchtune.utils._logging:The full model checkpoint, including all weights and configurations, has been saved successfully.You can now use this checkpoint for further training or inference.
INFO:torchtune.utils._logging:Checkpoint saved in 65.93 seconds.
1|26|Loss: 1.427944302558899: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 26/26 [3:11:40<00:00, 442.34s/it]

Result: The test results demonstrate the successful completion of a single-device LoRA fine-tuning process on the Llama 3.1 8B model. The configuration included a batch size of 30, gradient accumulation over 64 steps, and one epoch of training on an NPU device using the bf16 data type. Activation checkpointing was enabled, and LoRA fine-tuning was applied to attention modules. The process utilized AdamW as the optimizer with a learning rate of 0.0003 and a cosine learning rate scheduler.

Basic Usage Test

A single-device full fine-tuning process was performed on the Qwen2 0.5B model using the LoRA (Low-Rank Adaptation) technique.

Recipe: full_finetune_single_device

Model: Qwen2-0.5B-Instruct

Config:
# Tokenizer
tokenizer:
  _component_: torchtune.models.qwen2.qwen2_tokenizer
  path: /home/lcg/.cache/modelscope/hub/Qwen/Qwen2-0___5B-Instruct/vocab.json
  merges_file: /home/lcg/.cache/modelscope/hub/Qwen/Qwen2-0___5B-Instruct/merges.txt
  max_seq_len: null

# Dataset
dataset:
  _component_: torchtune.datasets.alpaca_cleaned_dataset
seed: null
shuffle: True

# Model Arguments
model:
  _component_: torchtune.models.qwen2.qwen2_0_5b

checkpointer:
  _component_: torchtune.training.FullModelHFCheckpointer
  checkpoint_dir: /home/lcg/.cache/modelscope/hub/Qwen/Qwen2-0___5B-Instruct
  checkpoint_files: [
    model.safetensors
  ]
  recipe_checkpoint: null
  output_dir: /home/lcg/tmp/torchtune/Qwen2-0.5B-Instruct-finetune
  model_type: QWEN2
resume_from_checkpoint: False

# Fine-tuning arguments
batch_size: 15
epochs: 1
optimizer:
  _component_: torch.optim.AdamW
  fused: False
  lr: 2e-5

loss:
  _component_: torchtune.modules.loss.CEWithChunkedOutputLoss
optimizer_in_bwd: False

max_steps_per_epoch: null
gradient_accumulation_steps: 8
compile: False

# Training environment
device: npu

# Memory management
enable_activation_checkpointing: True

# Reduced precision
dtype: bf16

# Logging
metric_logger:
  _component_: torchtune.training.metric_logging.DiskLogger
  log_dir: ${output_dir}
output_dir: /tmp/Qwen2-0.5B-Instruct-finetune
log_every_n_steps: 1
log_peak_memory_stats: False

Logs:
NFO:torchtune.utils._logging:Running FullFinetuneRecipeSingleDevice with resolved config:

batch_size: 15
checkpointer:
  _component_: torchtune.training.FullModelHFCheckpointer
  checkpoint_dir: /home/lcg/.cache/modelscope/hub/Qwen/Qwen2-0___5B-Instruct
  checkpoint_files:
  - model.safetensors
  model_type: QWEN2
  output_dir: /home/lcg/tmp/torchtune/Qwen2-0.5B-Instruct-finetune
  recipe_checkpoint: null
compile: false
dataset:
  _component_: torchtune.datasets.alpaca_cleaned_dataset
device: npu
dtype: bf16
enable_activation_checkpointing: true
epochs: 1
gradient_accumulation_steps: 8
log_every_n_steps: 1
log_peak_memory_stats: false
loss:
  _component_: torchtune.modules.loss.CEWithChunkedOutputLoss
max_steps_per_epoch: null
metric_logger:
  _component_: torchtune.training.metric_logging.DiskLogger
  log_dir: /tmp/Qwen2-0.5B-Instruct-finetune
model:
  _component_: torchtune.models.qwen2.qwen2_0_5b
optimizer:
  _component_: torch.optim.AdamW
  fused: false
  lr: 2.0e-05
optimizer_in_bwd: false
output_dir: /tmp/Qwen2-0.5B-Instruct-finetune
resume_from_checkpoint: false
seed: null
shuffle: true
tokenizer:
  _component_: torchtune.models.qwen2.qwen2_tokenizer
  max_seq_len: null
  merges_file: /home/lcg/.cache/modelscope/hub/Qwen/Qwen2-0___5B-Instruct/merges.txt
  path: /home/lcg/.cache/modelscope/hub/Qwen/Qwen2-0___5B-Instruct/vocab.json

DEBUG:torchtune.utils._logging:Setting manual seed to local seed<PHONE_NUMBER>. Local seed is seed + rank =<PHONE_NUMBER> + 0
Writing logs to /tmp/Qwen2-0.5B-Instruct-finetune/log_1729914193.txt
/home/lcg/miniconda3/envs/torchtune/lib/python3.10/site-packages/torch_npu/utils/storage.py:38: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  if self.device.type != 'cpu':
INFO:torchtune.utils._logging:Model is initialized with precision torch.bfloat16.
INFO:torchtune.utils._logging:Memory stats after model init:
        NPU peak memory allocation: 1.55 GiB
        NPU peak memory reserved: 1.61 GiB
        NPU peak memory active: 1.55 GiB
INFO:torchtune.utils._logging:Tokenizer is initialized from file.
INFO:torchtune.utils._logging:Optimizer is initialized.
INFO:torchtune.utils._logging:Loss is initialized.
Using the latest cached version of the dataset since yahma/alpaca-cleaned couldn't be found on the Hugging Face Hub
WARNING:datasets.load:Using the latest cached version of the dataset since yahma/alpaca-cleaned couldn't be found on the Hugging Face Hub
Found the latest cached dataset configuration 'default' at /home/lcg/.cache/huggingface/datasets/yahma___alpaca-cleaned/default/0.0.0/12567cabf869d7c92e573c7c783905fc160e9639 (last modified on Fri Oct 11 01:21:44 2024).
WARNING:datasets.packaged_modules.cache.cache:Found the latest cached dataset configuration 'default' at /home/lcg/.cache/huggingface/datasets/yahma___alpaca-cleaned/default/0.0.0/12567cabf869d7c92e573c7c783905fc160e9639 (last modified on Fri Oct 11 01:21:44 2024).
INFO:torchtune.utils._logging:Dataset and Sampler are initialized.
INFO:torchtune.utils._logging:No learning rate scheduler configured. Using constant learning rate.
WARNING:torchtune.utils._logging: Profiling disabled.
INFO:torchtune.utils._logging: Profiler config after instantiation: {'enabled': False}
  0%|                                                                                | 0/431 [00:00<?, ?it/s]/home/lcg/miniconda3/envs/torchtune/lib/python3.10/site-packages/torch/utils/checkpoint.py:1399: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead.
  with device_autocast_ctx, torch.cpu.amp.autocast(**cpu_autocast_kwargs), recompute_context:  # type: ignore[attr-defined]
1|431|Loss: 1.136042833328247: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 431/431 [31:03<00:00,  4.45s/it]INFO:torchtune.utils._logging:Model checkpoint of size 0.99 GB saved to /home/lcg/tmp/torchtune/Qwen2-0.5B-Instruct-finetune/hf_model_0001_0.pt
INFO:torchtune.utils._logging:Saving final epoch checkpoint.
INFO:torchtune.utils._logging:The full model checkpoint, including all weights and configurations, has been saved successfully.You can now use this checkpoint for further training or inference.
1|431|Loss: 1.136042833328247: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 431/431 [31:07<00:00,  4.33s/it]

Hi @noemotiovon thanks for the PR! And apologies for the delay in getting to the review here. A couple other questions I have that don't really fit neatly anywhere inline:

Do we expect compile to work? If so, we should test that. If not, we could raise an error
Do we expect quant-related APIs (e.g. QLoRA or QAT) from torchao to work? Same as point 1: if so we should test or possibly raise an error
PyTorch has now released 2.5 as stable. In general we do not claim to support anything but the latest stable release of PyTorch -- do you know the contract on torch_npu releases here?

Thank you very much for reviewing my code! I’ve made the suggested changes, and the goal of this PR is to accomplish the device-independent modifications for torchtune, using NPU as an example. This will involve adapting all recipes to ultimately make torchtune device-independent, with this PR specifically covering full_finetune_single_device and lora_finetune_single_device. Regarding the third point, torch-npu will release the 2.5.0 RC version on November 7, and I’ll be optimizing the code based on PyTorch’s new features as well!

Hi @ebsmothers,  could you please take a moment to review the code :relaxed: ? This update currently supports full_finetune_single_device and lora_finetune_single_device on NPU, and we’ll be adding support for additional recipes in the future. I really appreciate your help—thank you!
Best regards

Hi @noemotiovon sorry for the delay! I will take a look tomorrow if that's alright. Until then I'll tag @RdoubleA and @joecummings in case either of them gets a minute to take a look

Hi @ebsmothers, when you have a moment, could you take a quick look at the recent changes I made? Your feedback would be greatly appreciated. Thank you!

Thanks @noemotiovon for your patience! I left a handful more comments, please let me know if anything is unclear

Thank you for your review! Your feedback is very clear, and I will make the necessary code changes as soon as possible based on your suggestions. :relaxed:

Hi @ebsmothers, I’ve made the code changes based on your suggestions; could you please review it again?  :relaxed:
Additionally:

For NPU devices, it currently checks for bf16 support based only on the device model, encapsulated in the torch.npu.is_bf16_supported() method.
Support for distributed functionality is still being debugged and will be gradually integrated into another PR.

Best regards

Thanks @noemotiovon for the updates! I left a couple more comments but I think this is pretty close now. It looks like a unit test is failing in CI though, can you take a look? Happy to provide any debugging pointers if you need

Hi @ebsmothers,
Your review comments were excellent, and I’ve learned a lot from you! I’ve made changes based on your feedback and fixed the CI failure issue. Could you please review my code again?
Best regards

Hi @noemotiovon looks like CI is green now but there are merge conflicts. Can you pull from latest main and merge with your changes?

Hi @ebsmothers, I’d be happy to! I’ve resolved the conflicts and made the necessary adjustments. Could you please help merge it? Thank you for your continued support, and wishing you all the best in work and life!  :blush:


Request for abbreviations/acronyms

I'like to request support for abbreviations.
Abbreviations and/or acronyms (in HTML with the <abbr> tag) are not part of CommonMark neither GFM, but they are very useful.
Kramdown docs have a nice example about it.

This is a duplicate of https://github.com/raphlinus/pulldown-cmark/issues/5, so I will close this and respond there instead.


Separate LICENSE.md into LICENSE and NOTICE

There's no particular reason, but this makes it easier for beginners to understand.

ASF 3rd Party License Policy

Agreed.


FLOAT missing from Literal datatypes, but defined in Column datatypes.

FLOAT is missing from Literal datatypes (org.apache.pinot.common.request.Literal) but defined as a column datatype (org.apache.pinot.spi.data.FieldSpec.DataType). This may create type conversion issues since the only way to specify values for FLOAT column is by using DOUBLE literal values.
Literal defines the following datatypes:
BOOL, BYTE, SHORT, INT, LONG, DOUBLE, STRING, BINARY
FieldSpec.DataType defines the following datatypes:
INT, LONG, FLOAT, DOUBLE, BOOLEAN/* Stored as STRING */, STRING, BYTES, STRUCT, MAP, LIST;

The literal type should not matter. We should convert the literal values based on the field data type, and as long as the values can be represented by the literal, we are good. Actually on the server side, literal is always stored as string

From what I saw server is converting a string literal that appears in the predicate at least twice to an integer value and this is after Broker/Calcite has already parsed the string as number and possibly operated on (plus/minus scalar functions). So while it works, there is a bit of cost associated with it. There are also potential correctness issues. For example long to double conversion (which is happening in ScalarFunction.java on the Broker side due to ScalarFunctions.java) is a lossy conversion. None of these issues by themselves are major, but combined together they seem to indicate that some cleanup with type handling would be useful.

I agree that always using string literal is not efficient (legacy behavior carried over from PQL). For literal, we should keep only the necessary types as long as the value can be represented correctly without losing accuracy. Keeping LONG, DOUBLE and STRING should be able to represent all types supported by Pinot. We don't want to add too many literal types because that will complicate the logic of type conversion from literal to pinot type.

To achieve standard SQL behavior, we need to preserve the literal type. Submitted #11697 to represent float literal


Using ring buffer for LIFO

Fix #162

It's for for (int i = 0.... For two compilers listed in travis config file, gcc-4.8 defaults to sth below c99, even though it supports c11 and gnu11.


Example in nixos-container docs is broken

In https://nixos.org/releases/nixos/14.12/nixos-14.12.374.61adf9e/manual/ch-containers.html, this example doesn't work (curl hangs).
$ nixos-container update foo --config 'services.httpd.enable = true; \
  services.httpd.adminAddr = "foo@example.org";'

$ curl http://$(nixos-container show-ip foo)/
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">…

It can be fixed by adding networking.firewall.enable = false; inside the container config.

It's probably better to add networking.firewall.allowedTCPPorts = [ 80 ];

Please submit a PR if this is still a problem


Allow publishing from a specific folder using publishPath attribute

This is to allow publishing only a specific folder from within a package. In order to publish a specific folder, developers can add the following attribute to the relevant package's package.json file:
"publishPath": "dist"

This will run the npm publish command from within the dist folder, allowing only the content of this directory to be published.

Thanks for the PR!
This is a fairly unconventional use case you're asking for, and one that could likely be accomplished by using the conventional mechanism npm uses. Would this not work for you?

Hi @scalvert,
Thank you for your swift response. The problem with the above approach is that you can't publish the content of a folder. Instead, you will be publishing the folder itself.
For example, let's assume that the structure of my repo is as follows:
|_dist
|__index.js
|__index.d.ts
|__package.json
|_src
|__index.ts
|_package.json

I want to publish the content of the dist folder. If I use the approach you suggested, the npm pack will look like this:
|_dist
|__index.js
|__index.d.ts
|__package.json

Instead, I want the npm pack to look like this:
|_index.js
|_index.d.ts
|_package.json

Release It actually supports this using the publishPath config. I would appreciate it if the workspaces plugin also supported something similar. https://github.com/release-it/release-it/blob/main/docs/npm.md#publish-path


particular format of number is matching

Rasa NLU version:0.12.3
Operating system (windows, osx, ...): Ubuntu 18.04
Content of model configuration file:
language: "en"

pipeline:
- name: "intent_featurizer_count_vectors"
- name: "intent_classifier_tensorflow_embedding"
  intent_tokenization_flag: true
  intent_split_symbol: "_"

Issue:
my intent training data has been included the following data
{{ firstName | limitTo: 7 }}{{firstName.length > 7 ? '...' : ''}}
once the data is trained, while testing this intent is matching for the input 20/7/2018

{{ firstName | limitTo: 7 }}{{firstName.length > 7 ? '...' : ''}} i'm not sure what format this training data is? could you describe your problem in more detail please and post your training data if you can too.

i am training some FAQ documents in which questions are my intent and answers are text.
in that document, i have this line which i trained as the question, after training when a user gives input in any of the following formats,
55 34 555
20/7/2018
its matching with that intent... i don't know how it's matching

Which intent? Could you post your training data please? I'm still not sure I understand

i am extracting FAQ questions and answers from here.
the training data is following,
{
"rasa_nlu_data": {
"entity_synonyms": [],
"common_examples": [
{
"text": "  ",
"intent": "default"
},
{
"text": "I have special dietary requirements – will they be catered for?",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_0c8275af-455a-4b33-94de-4acd2e0397dc"
},
{
"text": "Can I drink the water in the countries I visit?",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_0d90e858-4033-42be-a7f1-cab1b96734d6"
},
{
"text": "Is accommodation air-conditioned/ heated?",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_12c88075-71e5-4b79-956a-bb5dc7acdb38"
},
{
"text": "Will I need a sleeping bag/mat/tent?",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_134299b0-90e4-40c2-9637-c06004293447"
},
{
"text": "What's the advantage of a traveller donating to The Intrepid Foundation?",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_14802046-58fc-4223-9bbd-3edf2a995000"
},
{
"text": "What does my policy need to cover?",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_14f7ffeb-09ed-4fdb-83ad-d77eb568724c"
},
{
"text": "Travel agent? Log in here",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_1a72b4a3-0884-4f86-956b-b357f10e965d"
},
{
"text": "Are airport transfers included in the trip price? If not, how much will a taxi cost from the airport? If not, how do I get to and from airports?",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_1ea7a59f-e12c-4144-a23c-c1157b8bc31a"
},
{
"text": "Are there age restrictions on your trips?",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_2049edbc-7caf-400a-8545-90d27c4649e2"
},
{
"text": "Will I be able to charge my electrical appliances?",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_252e8449-0fc2-4f13-a1ec-18d3bd53dbc4"
},
{
"text": "{{ firstName | limitTo: 7 }}{{firstName.length > 7 ? '...' : ''}}",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_719b6873-b318-4c03-b464-e5e9edcc09ba"
},
{
"text": "What's your email?",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_78082392-a470-4a9c-9bf5-6711fa09a7eb"
},
{
"text": "What type of ground transportation is used?",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_7814e085-8f0c-4470-9c14-39e65951f67f"
},
{
"text": "Do you help with Visas?",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_7d864804-8ff1-4792-a5a4-d9a4959d28eb"
},
{
"text": "What type of luggage do you recommend I take?",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_7ed45b0e-d05b-4921-8437-82c07c668b8c"
},
{
"text": "Who will be my group leader?",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_7faac70c-7fd7-4aeb-9fc8-1301c47e0884"
},
{
"text": "Can you provide me a list of places we are staying at?",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_83534b1c-8dc2-477a-8b6c-d7255dab468a"
},
{
"text": "Is tipping commonplace in other parts of the world?",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_839e3959-ed6a-4f57-8eed-2b7bc8646f01"
},
{
"text": "Do I need travel insurance?",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_84aa735e-a52a-4141-8535-c8f943c837cd"
},
{
"text": "Who are Intrepid travellers?.",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_88a20840-f4d2-49bb-87b2-0ff9c5054096"
},
{
"text": "Can I book a single room?",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_8b8c359a-933f-4723-81aa-3209e9366c2b"
},
{
"text": "How do I obtain Trip Notes/itinerary for full details of my trip?.",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_9e125a3d-e763-4e7d-a0c2-277166bd6663"
},
{
"text": "How safe are my belongings whilst on the trip?",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_9e71eb89-f0fa-4ef5-a3c5-0eb43db0207d"
},
{
"text": "Is Intrepid financially secure?.",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_a043882c-c6d7-43fc-8a12-bf9e244f4429"
},
{
"text": "What is the best way to carry money?",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_a4fc7dc9-dc9d-44cf-8f92-da147ca3a96e"
},
{
"text": "Does Intrepid have events/slide nights that I can attend?.",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_a58d97c9-a8e5-4386-9ba6-aa00fc704def"
},
{
"text": "Do I need to bring a sleeping bag?",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_a8282e3d-04d5-4247-9e29-77cd1ff0aecb"
},
{
"text": "What is The Intrepid Foundation?.",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_a910d20b-87b3-4ad1-b125-648a57477a91"
},
{
"text": "What is the name of the hotel where my trips starts?",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_aedd8f38-8b70-4b24-8911-42c51bf7fe07"
},
{
"text": "What do I do if I am unable to open a link on the Intrepid website?.",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_b373e336-3c9d-4bb4-ba4e-fee69c4fe0fb"
},
{
"text": "What will the food be like on my trip?",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_c3db01d4-acb4-44ab-8fd0-00270421af3a"
},
{
"text": "Can I request an early check-in?",
"intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_cf9ea02c-6cfa-47ff-97cb-10eb45dab8aa"
},
  {
    "text": "How can friends and family contact you?",
    "intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_df48fa16-043f-40b5-9e29-76e619b7949c"
  },
  {
    "text": "I have a disability – can I join an Intrepid trip?",
    "intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_e0824cce-0e62-40ae-99e7-b18e6e541174"
  },
  {
    "text": "What facilities are at the campsites?",
    "intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_e10ed082-4ffa-4da4-9f2d-9ef4080cff21"
  },

  {
    "text": "Can I make any changes to the itinerary of my group trip, either before or during the trip? .",
    "intent": "86464e71-5650-422b-8b08-7bf6d5de4c03_fed7b711-8482-4bfa-bf2d-150e8919fb5b"
  }
],
"regex_features": []

}
}

It looks like you only have a single example per intent - that explains why you're getting unstable NLU results. You should start with at least 10, and I'd advise you to name your intents something legible so you can distinguish between them :)

but I am not able to find the rationale behind matching the 20/7/18 only with that intent.

Because it's the only one containing numbers is my guess. As I said though, increase the amount of training data and you'll find more consistent NLU performance.

if so why does it match with the only particular format? like 55 55 525,20/07/2018
if we give something like 55 it doesn't match

as i said, an NLU model with a single example per intent is going to make inconsistent and weird predictions. Please add more examples and retrain your model. If you want that kind of input to match a particular intent, then you should add that kind of training data as well


Unexpected result with findall/3

For the following program in http://tau-prolog.org/sandbox/
test :-
    findall([A,_],p(A,_),L),
    write(L),
    nl.

p(a,b).
p(c,d).
p(e,f).

the query
test.

gives
[[a,__2],[c,__2],[e,__2]]

whereas the expectation is somtehing like
[[a,_2],[c,_3],[e,_4]]

@jariazavalverde Looking at the fix, the bug is only on the top-level printing of query bindings, correct?

the bug is only on the top-level printing of query bindings

@pmoura I'm not sure I'm getting this right. It wasn't just a printing problem: if a variable was subsequently instantiated, all of them were instantiated.
Before:

Now:

I have not yet update my local installation with the fix. I get:
?- assertz(f(_, 1)).
true.

?- assertz(f(_, 2)).
true.

?- findall(Var, f(Var, _), [X, Y]), X \== Y.
true.

?- findall((A1,A2), f(A1,A2), [(B1,1),(B2,2)]), B1 \== B2.
true.

I'm trying to define one or more additional Logtalk tests for the findall/3 predicate based on @josd finding and example code.

In https://github.com/IDLabResearch/Webeye/blob/master/examples/fourcolor.pl#L6
originally had findall([Place,_],'https://idlabresearch.github.io/ns#neighbours'(Place,_),Places),
and that now works fine!

@pmoura Try with a variable not in the goal. You should get something like:
?- assertz(f(1)).
true.

?- assertz(f(2)).
true.

?- findall((B,A), f(A), [(B1,1),(B2,2)]), B1 \== B2.
false.

That clarifies it, thanks. Test added (https://github.com/LogtalkDotOrg/logtalk3/commit/72f5836dc190ccae4a4280c5085ac8981132aa3f).

Couldn't reopen this issue but it is again occuring in 0.3.3

Fixed again. Thanks.

Thanks a lot and now we get the correct results in https://github.com/josd/eyebrow/commit/95946ec831fa8d960900cc73d611c6c89bf12ee0

Tests exist that can prevent these regressions... running them before a new release would be a good idea. The whole testing process can also be automated using a GitHub workflow. A good example is https://github.com/trealla-prolog/trealla/blob/main/.github/workflows/compliance.yml


Configuration for polling

Allow user to configure how often to poll

And per job would be useful for optimising the reaction to build updates.


coqPackages: doc

Motivation for this change
Changing bullet style to fix the rendering
cf https://github.com/NixOS/nixpkgs/pull/105877#issuecomment-757434759
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[ ] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

You can test to see if this worked by running this in your checkout of Nixpkgs.
$ cd doc
$ nix-build && nix-shell -p python3 --run 'python3 -m http.server --directory ./result/share/doc/nixpkgs'

For me it still renders incorrectly.

You can test to see if this worked by running this in your checkout of Nixpkgs.
$ cd doc
$ nix-build && nix-shell -p python3 --run 'python3 -m http.server --directory ./result/share/doc/nixpkgs'

For me it still renders incorrectly.

You may want to change the type of list using the admonitions added by @jtojnar used for instance in
`system`

: This is a two-component shorthand for the platform. Examples of this would be "x86_64-darwin" and "i686-linux"; see `lib.systems.doubles` for more. The first component corresponds to the CPU architecture of the platform and the second to the operating system of the platform (`[cpu]-[os]`). This format has built-in support in Nix, such as the `builtins.currentSystem` impure string.

`config`

: This is a 3- or 4- component shorthand for the platform. Examples of this would be `x86_64-unknown-linux-gnu` and `aarch64-apple-darwin14`. This is a standard format called the "LLVM target triple", as they are pioneered by LLVM. In the 4-part form, this corresponds to `[cpu]-[vendor]-[os]-[abi]`. This format is strictly more informative than the "Nix host double", as the previous format could analogously be termed. This needs a better name than `config`!

`parsed`

: This is a Nix representation of a parsed LLVM target triple with white-listed components. This can be specified directly, or actually parsed from the `config`. See `lib.systems.parse` for the exact representation.

This renders as

@ryantm do you think this is appropriate?

You may want to change the type of list using the admonitions added by @jtojnar used for instance in
`system`

: This is a two-component shorthand for the platform. Examples of this would be "x86_64-darwin" and "i686-linux"; see `lib.systems.doubles` for more. The first component corresponds to the CPU architecture of the platform and the second to the operating system of the platform (`[cpu]-[os]`). This format has built-in support in Nix, such as the `builtins.currentSystem` impure string.

`config`

: This is a 3- or 4- component shorthand for the platform. Examples of this would be `x86_64-unknown-linux-gnu` and `aarch64-apple-darwin14`. This is a standard format called the "LLVM target triple", as they are pioneered by LLVM. In the 4-part form, this corresponds to `[cpu]-[vendor]-[os]-[abi]`. This format is strictly more informative than the "Nix host double", as the previous format could analogously be termed. This needs a better name than `config`!

`parsed`

: This is a Nix representation of a parsed LLVM target triple with white-listed components. This can be specified directly, or actually parsed from the `config`. See `lib.systems.parse` for the exact representation.

This renders as

@ryantm do you think this is appropriate?

I don't understand, despite this PR, https://nixos.org/manual/nixpkgs/unstable/#sec-language-coq is still broken...

I don't understand, despite this PR, https://nixos.org/manual/nixpkgs/unstable/#sec-language-coq is still broken...

Perhaps it did not update yet? You can test it locally by running nix-build pkgs/top-level/release.nix -A manual in the nixpkgs directory.

Perhaps it did not update yet? You can test it locally by running nix-build pkgs/top-level/release.nix -A manual in the nixpkgs directory.


onSuccess callback never triggered

The onSuccess event is never triggered after a successful authentification (from Revolut) and I still stuck on the white screen with the black loader. Any idea ?
To Reproduce
Steps to reproduce the behavior:

Select a bank (Revolut)
Go to the authentification Revolut flow
Success screen from Revolut
Redirect to Plaid view
No success event triggered / infinite loader screen

Expected behavior
Get a onSuccess callback
Screenshots

Expo (please complete the following information):

Version 42.0.0

Smartphone (please complete the following information):

Device: iPhone6 simulator
OS: iOS15

Additional context
the last event trigger(onEvent):
{   "eventName": "OPEN_OAUTH",   "metadata": Object {     "errorCode": null,     "errorMessage": null,     "errorType": null,     "exitStatus": undefined,     "institutionId": "ins_120183",     "institutionName": "Revolut (ES)",     "institutionSearchQuery": null,     "linkSessionId": "xxx",     "mfaType": null,     "requestId": "xxx",     "timestamp": "2021-11-05T00:20:58.054Z",     "viewName": null,   }, }

@jetaix Are you sure it isn't the bank that is failing to respond? Have you tried another bank? Have you tried it in a web flow?

@JBaczuk Yes I'm sure the bank (Revolut) response a success screen, it's work with another bank, I think it's specific to the Revolut response and how the success url/event is manage by the library.

same issue with a other french bank called CIC

We're having the same issue, the component is returning a HANDOFF event, but not triggering Success.
https://www.loom.com/share/1408eb53bf6d462997f56ef8f604a1be

Hello, I am having having the same issue. I can navigate and authorize in Revolut or Barclays, but in the and is not triggering the onSuccess event.

Hello, I am having having the same issue. I can navigate and authorize in Revolut or Barclays, but in the and is not triggering the onSuccess event.

Do you have any idea if there is any network activity, does their server send a reply?

Yes, the last event is OPEN_AUTH. I authenticate my bank , i see the success message from my bank but after that is not triggering the onSuccess event.

Yes, the last event is OPEN_AUTH. I authenticate my bank , i see the success message from my bank but after that is not triggering the onSuccess event.

Anyone know if this is fixed? Deciding where I can use this library

Any word here yet?

Experiencing the same thing. Any thoughts here?

I'm sorry I don't have time to try to reproduce this issue. If anyone would like to contribute, I'm happy to take PRs. If anyone would like to take over the project, please let me know.

Same issue here, onSuccess is not being called. No way to finish linking accounts.

Taking a look at this.


RTK query setup refactor

Need to change file structure and refactor apiSlices

@Muralidhar22 can you add more description to this issue?
cc: @Maheima


⚠ ClamAV has degraded performance

In 3769a90, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 145 ms

Resolved: ClamAV performance has improved in 9978d21.


[RFC][authenticator] Introduce a new trait TransactionAuthenticator

Motivation
This PR introduces a new trait TransactionAuthenticator and updated the code base accordingly. The purpose of the PR is to increase reusability of diem code for other blockchains, specifically for authentication scheme.
As of now, Diem is tightly coupled with TransactionAuthenticator enum which is maintained as part of diem.  If a different blockchain wants to reuse diem code with an authentication scheme not yet supported by Diem (e.g., ECDSA), the blockchain developer have to contribute to Diem repo to add a new variant of TransactionAuthenticator enum. With this PR, we want to
Changes include

Introduce a new trait TransactionAuthenticator
Change the existing TransactionAuthenticator enum implementation to DiemTransactionAuthenticator
Make SignedTransaction generic over TransactionAuthenticator
Add type parameters for DiemTransactionAuthenticator in diem code accordingly

Some check/review points

Using DiemTransactionAuthenticator instead of TransactionAuthenticator would be a breaking change? e.g., No issue for ser/des
How far do we want to push reusability of diem code? i.e., For now, we added DiemTransactionAuthenticator to struct Transaction

Have you read the Contributing Guidelines on pull requests?
Yes

hmm, I was expecting something like
pub enum TransactionAuthenticator {
    /// Single signature
    Ed25519 {
        public_key: vec<u8>,
        signature: vec<u8>,
    },
    /// K-of-N multisignature
    MultiEd25519 {
        public_key: vec<u8>,
        signature: vec<u8>,
    },

so the behavior is implementation specific instead of specification specific (the underlying bytes representation stays the same). or in other words, the choice of eager/lazy should be user agnostic.

LazyEvaluation (what's in the PR currently) is what we have in the design doc. It was called ToBeDeserialized in the doc and @imsungchoi-fb gave it a better name. The advantage of adding a new LazyEvaluation variant over changing the current Ed25519 and MultiEd25519 variants is agility outside of Diem repo. With a more generic adapter interface, other instantiations of Diem can reuse the infrastructure, use LazyEvaluation variant to implement their own authentication scheme and signature verification algorithm without having to change any Diem code.
Do you have a strong preference for implementation specific behavior @zekun000 ? Also if we are changing the variants, will there be backwards compatibility issues?

ah, I thought this is for performance so I was expecting we only change implementation. for agility purpose it makes sense, although the name is a little confusing that makes me feel it's for optimization purpose (just my opinion).
we should be able to maintain the compatibility by changing the pubkey/signature <-> vec, but we probably should do it in another PR.

ah, I thought this is for performance so I was expecting we only change implementation. for agility purpose it makes sense, although the name is a little confusing that makes me feel it's for optimization purpose (just my opinion).
we should be able to maintain the compatibility by changing the pubkey/signature <-> vec, but we probably should do it in another PR.

I've changed the original enum name ToBeSerialized to LazyEvaluation as I thought it is more accurate for what we are doing. I have no strong opinion, but I am willing to change the name if you guys think that ToBeSerialized or another one is making more sense. Feel free to add your input on naming.

I'm not fancy enough, I was thinking about Others lol (which shouldn't support existing variants).
also wonder what's the thought for future variants that are not just pubkey and signatures like the multi-agents variant?

I'm not fancy enough, I was thinking about Others lol (which shouldn't support existing variants).
also wonder what's the thought for future variants that are not just pubkey and signatures like the multi-agents variant?

lol. I prefer to my suggestions against Others and will wait for @emmazzz's input.  Naming is the hardest problem :/
For multi-agents, we have the different authenticator AccountAuthenticator and we added the same variant to the authenticator as follows.
In the future, if we have a new scheme to support (e.g., ECDSA), I think we don't have to add a new variant "ECDSA" to both of authenticators. We just need to add "ECDSA" scheme to Scheme and reuse the LazyEvaluation variant.
@emmazzz Please correct me if I am wrong.
pub enum AccountAuthenticator {
    /// Single signature
    Ed25519 {
        public_key: Ed25519PublicKey,
        signature: Ed25519Signature,
    },
    /// K-of-N multisignature
    MultiEd25519 {
        public_key: MultiEd25519PublicKey,
        signature: MultiEd25519Signature,
    },
    /// Delay Public Key & Signature Deserialization until Verify
    LazyEvaluation {
        scheme: u8, // which scheme to deserialize this into
        public_key: Vec<u8>,
        signature: Vec<u8>,
    },
    // ... add more schemes here
}

For multi-agents, we have the different authenticator AccountAuthenticator and we added the same variant to the authenticator as follows.
In the future, if we have a new scheme to support (e.g., ECDSA), I think we don't have to add a new variant "ECDSA" to both of authenticators. We just need to add "ECDSA" scheme to Scheme and reuse the LazyEvaluation variant.

This sounds good to me.
Naming wise, I don't like ToBeDeserialized, which is just a placeholder name I used in the doc. I like the idea of having the word "opaque" in the name. Maybe something like OpaqueAuthScheme? I don't have strong preference though.

For multi-agents, we have the different authenticator AccountAuthenticator and we added the same variant to the authenticator as follows.
In the future, if we have a new scheme to support (e.g., ECDSA), I think we don't have to add a new variant "ECDSA" to both of authenticators. We just need to add "ECDSA" scheme to Scheme and reuse the LazyEvaluation variant.

This sounds good to me.
Naming wise, I don't like ToBeDeserialized, which is just a placeholder name I used in the doc. I like the idea of having the word "opaque" in the name. Maybe something like OpaqueAuthScheme? I don't have strong preference though.

Yeah I feel that OpaqueAuthScheme would be the best name I also came up with.
@zekun000 If you don't have any concerns on the name, I will update the PR.

Also, we would like to optimize the performance of deserialization by delaying the validation to happen at the verification stage, where transactions can be processed in parallel. This optimization involves storing raw bytes representing public keys and signatures in the transaction authenticator and deserializing only when necessary.

I just want to point out that this PR does not achieve this goal since you are requiring all users to change the way they construct transactions in order to "benefit" from delayed validation.

With a more generic adapter interface, other instantiations of Diem can reuse the infrastructure, use OpaqueAuthScheme variant to implement their own authentication scheme and signature verification algorithm without having to change any Diem code.

I also want to push back on this claim that people will be able to implement their own auth schemes without having to change any diem code. This is not the case since you are defining a mapping between a u8 and a particular scheme, in diem specific code, If i wanted to add another scheme type I would still need to change diem code to introduce a mapping from u8 -> new scheme. If we really wanted to achieve this particular goal we might have a better chance if we introduced a trait to represent authenticators and making diem code generic across a type of authenticator (although that would require quite a bit of work).
I could be missing something though or misunderstanding the intent here. Have you tested that you can introduce a new authenticator type without changing diem's code (built on top of this PR)?

Also, we would like to optimize the performance of deserialization by delaying the validation to happen at the verification stage, where transactions can be processed in parallel. This optimization involves storing raw bytes representing public keys and signatures in the transaction authenticator and deserializing only when necessary.

I just want to point out that this PR does not achieve this goal since you are requiring all users to change the way they construct transactions in order to "benefit" from delayed validation.

You're right. For the existing Ed25519 or MultiEd25519 variants, this PR does not make any performance improvement yet.  I think we should follow up in another PR to bring the benefit to the existing variants. With the description I was trying to explain why we introduced vec for public key and signature in OpaqueAuthScheme. I think the description is a little bit confusing. I will make it more clear.

With a more generic adapter interface, other instantiations of Diem can reuse the infrastructure, use OpaqueAuthScheme variant to implement their own authentication scheme and signature verification algorithm without having to change any Diem code.

I also want to push back on this claim that people will be able to implement their own auth schemes without having to change any diem code. This is not the case since you are defining a mapping between a u8 and a particular scheme, in diem specific code, If i wanted to add another scheme type I would still need to change diem code to introduce a mapping from u8 -> new scheme. If we really wanted to achieve this particular goal we might have a better chance if we introduced a trait to represent authenticators and making diem code generic across a type of authenticator (although that would require quite a bit of work).
I could be missing something though or misunderstanding the intent here. Have you tested that you can introduce a new authenticator type without changing diem's code (built on top of this PR)?

@emmazzz would answer this question better. I will try my best.
I think this OpaqueAuthScheme is a kind of stepping stone for the change emma is working on.
How this PR change is related with emma's change is as follows.
In emma's change, we introduce a new trait VMAdapter.
pub trait VMAdapter {
   fn verify_signature(txn: &SignedTransaction) -> Result<()>;
   ... // and other customizations
}

This SignedTransaction includes an authenticator with the variant OpaqueAuthScheme introduced in this PR. This way other instantiations of Diem can implement VMAdapter trait and use OpaqueAuthScheme variant for the signed transaction, and deserialize the keys and signatures into their favorite schemes in verify_signature function.

I just took a look at that PR and I agree with @sblackshear first reaction that the VMAdapter trait should probably take in a transaction type defined by an associated type parameter, e.g.
pub trait VMAdapter {
   type Transaction: <Maybe some traits here>;
   fn verify_signature(txn: &Self::Transaction) -> Result<()>;
}

Specifically if your goal is to have people be able to implement their own auth schemes without having to change diem code, then the current approach we're taking necessitates that you must support ed25519 (since the Authenticator enum has that as one of the variants).
Also, to be clear, my biggest hesitation with the approach this PR is taking is that it is changing a public facing interface and is not just an implementation detail. I would also expect that when changing some critical public facing interface a DIP would be required (of course i'm not an expert in what requires a dip or not)

Specifically if your goal is to have people be able to implement their own auth schemes without having to change diem code, then the current approach we're taking necessitates that you must support ed25519 (since the Authenticator enum has that as one of the variants).

I'd say having a TransactionTrait is definitely one of the steps we can take down the way. It's not part of the current step since introducing it will incur a lot of refactoring and code changes.

Also, to be clear, my biggest hesitation with the approach this PR is taking is that it is changing a public facing interface and is not just an implementation detail.

I understand the hesitation. Do we have a way to achieve similar performance gains without changing the public facing interface though? One way is to delay the deserialization of SignedTransaction altogether? Is it doable?

we should separate performance gain from adding agility. we can get performance gain simply by changing the enum to
    Ed25519 {
        public_key: vec<u8>,
        signature: vec<u8>,
    },

and delay deserialization to verification stage. (It doesn't break compatibility I think)
This PR is more on addressing the agility problem.

I understand the hesitation. Do we have a way to achieve similar performance gains without changing the public facing interface though? One way is to delay the deserialization of SignedTransaction altogether? Is it doable?

Agreed with @zekun000, this PR does nothing to solve performance issues since it necessitates the end user to change how they construct a transaction for those performance gains to take effect, any performance gains should ideally be done as an implementation detail, perhaps in a way Zekun showed above.

I'd say having a TransactionTrait is definitely one of the steps we can take down the way. It's not part of the current step since introducing it will incur a lot of refactoring and code changes.

I haven't taken a very close look at your VMAdaptor PR, but given the VMAdaptor is new I don't know if it would be significantly more challenging to do this now (depending on how we go about making the change). Of course i've only started looking at this issue so i could very well be wrong.

Depending on how far we want to push for "agility" we could look towards doing something like https://github.com/bmwill/diem/commit/ac4c6a24582537ccc2cd3f6af35cb61c9143099f. This introduces a trait TransactionAuthenticator and makes SignedTransaction generic over A: TransactionAuthenticator. This will, at the end of the day, enable completely switching out the transaction authenticator to something else (maybe a type that doesn't even support ed25519) at the cost of adding generic type parameters in a number of places (I started this here but theres still a ton of spots that would need to be converted). Doing this, and maybe even abstracting a number of other concepts, would enable us to instantiate a blockchain very different from the existing DPN one we have today.

Depending on how far we want to push for "agility" we could look towards doing something like bmwill@ac4c6a2. This introduces a trait TransactionAuthenticator and makes SignedTransaction generic over A: TransactionAuthenticator. This will, at the end of the day, enable completely switching out the transaction authenticator to something else (maybe a type that doesn't even support ed25519) at the cost of adding generic type parameters in a number of places (I started this here but theres still a ton of spots that would need to be converted). Doing this, and maybe even abstracting a number of other concepts, would enable us to instantiate a blockchain very different from the existing DPN one we have today.

I like the approach and I think it is more generic for others to instantiate a blockchain. I am willing to dig into this approach, but I am afraid that this would be overkill for the change we are making.
Some questions on @emmazzz and @sblackshear

Any concerns/thought on this approach?
An alternative option between the current implementation and brandon's proposal? i.e., a stepping stone solution towards brandon's proposal

I am afraid that this would be overkill for the change we are making.

You are absolutely correct, I'm most likely missing context for the intent behind this PR and there might be a simpler way to solve this depending on your goals.

Depending on how far we want to push for "agility" we could look towards doing something like bmwill@ac4c6a2. This introduces a trait TransactionAuthenticator and makes SignedTransaction generic over A: TransactionAuthenticator. This will, at the end of the day, enable completely switching out the transaction authenticator to something else (maybe a type that doesn't even support ed25519) at the cost of adding generic type parameters in a number of places (I started this here but theres still a ton of spots that would need to be converted). Doing this, and maybe even abstracting a number of other concepts, would enable us to instantiate a blockchain very different from the existing DPN one we have today.

I like this approach too! And it actually looks easier to implement than I expected. I actually don't mind doing this for agility and I think it aligns with what VMAdapter trait hopes to achieve.
This way we can also separate the agility and performance conversations.
For performance, it seems that it's better to do the implementation specific approach @zekun000 originally proposed and what @bmwill proposed here, which I think is the same essential idea.

Depending on how far we want to push for "agility" we could look towards doing something like bmwill/diem@ac4c6a2. This introduces a trait TransactionAuthenticator and makes SignedTransaction generic over A: TransactionAuthenticator. This will, at the end of the day, enable completely switching out the transaction authenticator to something else (maybe a type that doesn't even support ed25519) at the cost of adding generic type parameters in a number of places (I started this here but theres still a ton of spots that would need to be converted). Doing this, and maybe even abstracting a number of other concepts, would enable us to instantiate a blockchain very different from the existing DPN one we have today.

I like this approach too! And it actually looks easier to implement than I expected. I actually don't mind doing this for agility and I think it aligns with what VMAdapter trait hopes to achieve.
This way we can also separate the agility and performance conversations.
For performance, it seems that it's better to do the implementation specific approach @zekun000 originally proposed and what @bmwill proposed here, which I think is the same essential idea.

Okay I will try to make change the PR as guided in here [bmwill/diem@ac4c6a2]! Thanks for the feedback.

Please add a test to json-rpc/integration-tests suite for fetching a transaction with new authenticator, and update doc https://github.com/diem/diem/blob/main/json-rpc/docs/type_transaction.md#type-transactiondata as necessary.
Also, I have same concern with @bmwill, I don't see a way you can actually add a new authenticator without changing Scheme definition or TransactionAuthenticator implementation on server side.
It would be great if you could add a test to show how this new OpaqueAuthScheme is used for the purpose of adding new authenticator.

Thanks for the feedback.
I will add a new test in next change and update the doc.
Also I will change the PR as guided in [bmwill/diem@ac4c6a2] which addresses your concerns.

just small nit, it's probably better to have a type alias like DiemSignedTransaction = SignedTransaction.
or rename the SignedTransaction to something else and type alias SignedTransaction = GenericSignedTransaction.

Thanks for taking a look!
I guess that it is for easier refactoring in the future. It would be great if you elaborate more.
I consider adding a type alias like
type DiemSignedTransaction = SignedTransaction<DiemTransactionAuthenticator>;

Do you think that we have to refactor codes to be relied on DiemSignedTransaction instead of SignedTransaction<DiemTransactionAuthenticator>?

Updated.
Added two type aliases, DiemSignedTransaction and DiemSignatureCheckedTransaction.


Fix notifier verbs

Fix notifier verbs
Based on #703 notifier verbs are enable, disable and not start, stop.
Related issue(s)
#703

Can we also fix this comment on this PR: https://github.com/kubeshop/botkube/issues/703#issuecomment-1359960622 ?

I think the help (@Botkube help) should be also updated.
Also, I think we miss another thing:

@Botkube - same as @Botkube help

Source: https://github.com/kubeshop/botkube/issues/703#issuecomment-1308613253
Does it work this way? Looking at the code I don't think so 🤔 If not, please also add it here or in a separate PR, whatever it's easier for you 👍


[UPDATED] Convert to typescript

Convert original javascript to modern TypeScript standards
Add code style lint rules, and auto formatting using Prettier
Add config/main.json validation with yup (see example CONFIG VALIDATION ERROR output below)

Example Validation Error with no config:
Configuration error! See config/main.json
project is a required field
polkadotNetworkId is a required field
chain is a required field
state.project is a required field
polkadotBinary.url is a required field
polkadotBinary.checksum is a required field
nodeExporter.enabled is a required field
nodeExporter.binary.url is a required field
nodeExporter.binary.checksum is a required field
polkadotRestart.enabled is a required field

Example Validation Error with misconfigured validator "nodes" config
Configuration error! See config/main.json
validators.nodes[0].provider must be one of the following values: aws, azure, digitalocean, gcp, packet
validators.nodes[0].count is a required field
validators.nodes[0].location is a required field
validators.nodes[0].machineType is a required field
validators.nodes[0].nodeName is a required field
validators.nodes[0].projectId is a required field
validators.nodes[0].sshUser is a required field

And more...

Can I get some feedback on this? This adds type checking to the code base. Very helpful 🎉

Hello @DoubleOTheven, I really appreciate the effort you put into this, but we're not planing to do any major changes in this repo anymore. It was introduced due to the fact that Polkadot/Kusama used to have sentry nodes, which made the setup a little bit more complicated. Now, however, the setup is quite straight forward and most validators use their own workflow and other architectures anyway, such as Kubernetes.

Ah, ok. K8s is better, but this set up works well for people just getting into it. No worries


How to add user device when login

I want track user logged in current all devices

Please, don't use github issues to ask questions, join our discord or start a discussion


SQLDatabase: adding security disclamer

It might be obvious to most engineers, but I think everybody should be cautious when using such a chain.

this is fantastic! totally agree, disclaimer is super helpful.


Prevent enumeration of absolute path using Browser Plugin

Previously URIs like ?sabreAction=asset&assetName=../../../../../../../../../../../Users/lreschke/Programming/core/3rdparty/sabre/dav/lib/DAV/Browser/assets/sabredav.css could have been used as  realpath does resolve them properly.
As discussed in https://github.com/owncloud/core/commit/1edd6d7d0b15a3fd0ad7b20107ac0c603f63204c#commitcomment-9804349
@evert As discussed.

FILE: /home/travis/build/fruux/sabre-dav/lib/DAV/Browser/Plugin.php
--------------------------------------------------------------------------------
FOUND 1 ERROR(S) AFFECTING 1 LINE(S)
--------------------------------------------------------------------------------
 468 | ERROR | TRUE, FALSE and NULL must be lowercase; expected "false" but
     |       | found "FALSE"
--------------------------------------------------------------------------------
UPGRADE TO PHP_CODESNIFFER 2.0 TO FIX ERRORS AUTOMATICALLY
--------------------------------------------------------------------------------

Let me find out what black magic that is :see_no_evil:

Writing a unittest for this one

Writing a unittest for this one

Need to find out how to start the SabreDAV test suite on my Mac. It failed always hard with:
PHP Warning:  include(/Users/lreschke/Programming/core/3rdparty/sabre/dav/tests/../vendor/autoload.php): failed to open stream: No such file or directory in /Users/lreschke/Programming/core/3rdparty/sabre/dav/tests/bootstrap.php on line 9
PHP Stack trace:
PHP   1. {main}() /usr/local/Cellar/phpunit/4.3.4/libexec/phpunit-4.3.4.phar:0
PHP   2. PHPUnit_TextUI_Command::main() /usr/local/Cellar/phpunit/4.3.4/libexec/phpunit-4.3.4.phar:605
PHP   3. PHPUnit_TextUI_Command->run() phar:///usr/local/Cellar/phpunit/4.3.4/libexec/phpunit-4.3.4.phar/phpunit/TextUI/Command.php:138
PHP   4. PHPUnit_TextUI_Command->handleArguments() phar:///usr/local/Cellar/phpunit/4.3.4/libexec/phpunit-4.3.4.phar/phpunit/TextUI/Command.php:148
PHP   5. PHPUnit_TextUI_Command->handleBootstrap() phar:///usr/local/Cellar/phpunit/4.3.4/libexec/phpunit-4.3.4.phar/phpunit/TextUI/Command.php:651
PHP   6. PHPUnit_Util_Fileloader::checkAndLoad() phar:///usr/local/Cellar/phpunit/4.3.4/libexec/phpunit-4.3.4.phar/phpunit/TextUI/Command.php:817
PHP   7. PHPUnit_Util_Fileloader::load() phar:///usr/local/Cellar/phpunit/4.3.4/libexec/phpunit-4.3.4.phar/phpunit/Util/Fileloader.php:77
PHP   8. include_once() phar:///usr/local/Cellar/phpunit/4.3.4/libexec/phpunit-4.3.4.phar/phpunit/Util/Fileloader.php:93

etc.
Any suggestions? :smile:

The problem is that the unittests expect sabre/dav to be loaded as the main package, not as a dependency.
So the easiest is to just do a separate clone of sabredav, or you could run composer install in the sabre/dav directory (but that will also create another vendor directory there).
Alternatively you could also create your own bootstrap.php that uses the owncloud autoloader instead of PHP's.


Fix bug when re-running expired preflight checks

Fixed a bug where re-running expired preflight checks would hang indefinitely.

@davidmreed - Already created: W-11268662


What should I do after running retrain.py?

After running retrain.py,if I want to predict a image what should I do?

You can use the label_image script for predicting the image.
label_image script

thinnks,but still have some troubles,that i can not find the lable_image.py script in my project,and both tf_files/retrained_graph.pb;tf_files/retrained_labels.txt can not be found.how can i do?

Hi,
the link to label_image script is above. You'll have to git clone the other repository to access this file (or theoretically just download the file directly, if there are no special deps).
As for the other two files, take a look at:
https://github.com/tensorflow/hub/blob/09f45963f6787322967b6fec61459f3ac56fbb27/examples/image_retraining/retrain.py#L1176
https://github.com/tensorflow/hub/blob/09f45963f6787322967b6fec61459f3ac56fbb27/examples/image_retraining/retrain.py#L1197

@laminggg ,
Can you please confirm if you could Predict an image successfully now.

@rmothukuru
yes,it works well now


[docs] update fine_tune_checkpoint specification

update fine_tune_checkpoint specification

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


closeOnSelect is not working if I set the value property

In updateSelectedDate function, the closeCalendar will run if I set "undefined" to value property.
It would be great if the closeCalendar can run based on closeOnSelect even if the value is set. So, I don't need to call closeCalendar manually.
[Edit]
Oops, sorry I just found that I should declare "open" property to force it close all the time.

I will close this as the author seems to have resolved his/her issue (looking at the edit comment).


Feature/usgs

Fixed USGS harvester (pep8 and unused imports)
Included favicon
Included yaml file

@MerlinZhang it looks like the description needs to be parsed (still includes html tags)

We usually keep the raw data as it is, and then leave whatever sources use the data to fix the parsing. For example, osf.io/share sanitizes html before displaying!


[T5A2][T11-B2]Wong Jeremy

Accepting this PR without a review because another of your PRs for this tutorial was reviewed in detail.


set Content-Type to json

Fixes https://github.com/KenanY/strawpoll/issues/15
/cc @rfox90 for review?

I'll take a better good shortly but looks good from mobile :)

LGTM :+1:

@rfox90 Thanks for the review!
Landed and pushed, but according to https://status.github.com/

Our systems are working through a queue backlog. In the meantime, new pushes may take some time to show on GitHub.com.

Waiting on that before releasing 1.0.6.

Actually putting this in 1.1.0 since it's going in along with #13 which is a new feature. Not sure why I never published that before.


All markdown extensions should be supported

To clarify
Add support for the following file extensions:

.md
.markdown


What is the license?

What is the license for this repository, please?
Thank you.

Thanks for your feedback, the BSD license has been added, closed in https://github.com/grucloud/bau/pull/22

the BSD license has been added
I'm not sure I got this part.
https://github.com/grucloud/bau/blob/main/LICENSE lists the license as MIT, not BSD. Which one is, please?

Sorry, I meant MIT


Update infra

### Summary of the changes
Move to scouting queue
Re-enable integration tests
Switch to newer package in support of integration tests

@phil-allen-msft: Still working on this PR or should it be closed?

/azp run


explore weighted random sample

issue #19

@Runjing-Liu120 / @shifwang - feel free to MRG this once you are ready.


Fix issue with only lowercase player name working with PvP combat

I'm sorry, I did not mean to include the BountyGoal.js in this PR but it is present. If you are sensitive about the github audit trail, please let me know and I'll refile this PR.

Nah, it's fine. Are you still working on the bounty goal or is it done? If you're still working on it I'll wait to merge


add rbac prerequisite for using weave net on GKE

fixes #3111

Please can you rebase this on 2.4 so the change will flow through to the website straight away.
I changed the base branch so GitHub is now showing 54 commits :man_facepalming:


gateway: refactor event emissions

Refactor event emissions to be simpler, faster, reduce allocations, and remove duplicate code.
Instead of creating a Vec to store listener IDs to remove, iterating over listeners and determining what to remove, and then iterating over the vec to remove them, use DashMap::retain to directly remove listeners that have had their channels closed.
Additionally, simplify and remove code by abstracting the iteration and checks over the listeners map for the bytes and event emission functions to another function that both call. bytes and event provide a function to create events, and this function is called when sending an event to a listener channel.
bytes now takes a reference to the listeners map and is no longer an async fn.
Additionally, add documentation and tests.

I'm curious to see how well the "clone, then give the final copy away" approach works down the line, but looks good to me.

Is there a specific concern you have? The method of avoiding a final clone was in the original implementation.

Is there a specific concern you have?

Nope, I'm not concerned, I just thought it was an interesting approach is all. I haven't seen that pattern as of until now.

Rebased


Add update_record_with_output function and tests

~Draft as it builds on https://github.com/risinglightdb/sqllogictest-rs/pull/129~
Rationale
I want to be able to update Records given the output of a particular run (see https://github.com/apache/arrow-datafusion/issues/4570)
Changes

Add update_record_with_output function
Tests

I think this is my last planned PR to sqllogictest for a bit (I want to go try and integrated this into DataFusion upstream first)

@xxchan  would you like me to add tests to this PR or do you plan to merge this one and have me add more tests as a follow on PR?

Let's merge it first and improve it later, since it generally looks good and works well in my manual testing.

Let's merge it first and improve it later, since it generally looks good and works well in my manual testing.

Here is a follow on PR https://github.com/risinglightdb/sqllogictest-rs/pull/131#


Save selected wireless interface

This looks like a bug. I don't have a system with the necessary hardware running 2.3 right now to really test, but the code that processes the submit/apply looks for $_POST['if']
Please double-check on suitable hardware!

Looks good to me, especially comparing it to the 2.2.x version of the page.


需要图库自动索引

The current version only creates an index during the first import. It does not update the index database when album pictures are updated, and only does so after uninstalling and reinstalling. It is necessary to add a feature for manual or automatic indexing

Firstly, you can perform a search, and then at the bottom of the search box, you'll see a button that says 'New photos available for indexing.' Clicking it will update the index.
The reason why 'updating the index for each photo' is not supported is that loading the model is too slow. The time it takes to load the model once could be used to build an index for 1000 photos.


[Warning] Multiple input files are attempting to write to the same output file "_themes/_includes/master.html.liquid". Please rename at least 1 of following input files to ensure no duplicate output files: "_themes/_includes/master.html.liquid, _themes/_includes/master.html.liquid".
[Warning] Multiple input files are attempting to write to the same output file "_themes/_includes/mooncakeFooter.html.liquid". Please rename at least 1 of following input files to ensure no duplicate output files: "_themes/_includes/mooncakeFooter.html.liquid, _themes/_includes/mooncakeFooter.html.liquid".
[Warning] Multiple input files are attempting to write to the same output file "_themes/_includes/mooncakeHeader.html.liquid". Please rename at least 1 of following input files to ensure no duplicate output files: "_themes/_includes/mooncakeHeader.html.liquid, _themes/_includes/mooncakeHeader.html.liquid".
[Warning] Multiple input files are attempting to write to the same output file "_themes/_includes/openGraphAndTwitter.html.liquid". Please rename at least 1 of following input files to ensure no duplicate output files: "_themes/_includes/openGraphAndTwitter.html.liquid, _themes/_includes/openGraphAndTwitter.html.liquid".
[Warning] Multiple input files are attempting to write to the same output file "_themes/_includes/productFeedback.html.liquid". Please rename at least 1 of following input files to ensure no duplicate output files: "_themes/_includes/productFeedback.html.liquid, _themes/_includes/productFeedback.html.liquid".
[Warning] Multiple input files are attempting to write to the same output file "_themes/_includes/standardFooter.html.liquid". Please rename at least 1 of following input files to ensure no duplicate output files: "_themes/_includes/standardFooter.html.liquid, _themes/_includes/standardFooter.html.liquid".
[Warning] Multiple input files are attempting to write to the same output file "_themes/_includes/tutorialToc.html.liquid". Please rename at least 1 of following input files to ensure no duplicate output files: "_themes/_includes/tutorialToc.html.liquid, _themes/_includes/tutorialToc.html.liquid".
[Warning] Multiple input files are attempting to write to the same output file "_themes/_includes/versionPicker.html.liquid". Please rename at least 1 of following input files to ensure no duplicate output files: "_themes/_includes/versionPicker.html.liquid, _themes/_includes/versionPicker.html.liquid".

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


aiken uplc encode --to named-debruijn doesn't work properly

What Git revision are you using?
aiken v1.0.10-alpha 85d82e5
What operating system are you using, and which version?

[ ] Linux / Ubuntu
[ ] Linux / Other
[x] macOS
[ ] Windows

Describe what the problem is?
I've found 2 problems regarding the encoding to named-debruijn that was introduced as a result of this issue https://github.com/aiken-lang/aiken/issues/553
Problem 1

Details of problem 1
Having the following validator:
use aiken/hash.{Blake2b_224, Hash}
use aiken/list as list
use aiken/transaction.{ScriptContext, Spend}
use aiken/transaction/credential.{VerificationKey}

type FooDatum {
  owner: Hash<Blake2b_224, VerificationKey>,
}

validator {
  fn validate_foo(datum: FooDatum, _redeemer: Void, ctx: ScriptContext) -> Bool {
    let ScriptContext { transaction, .. } = ctx
    when ctx.purpose is {
      Spend(_) -> list.has(transaction.extra_signatories, datum.owner)
      _ -> False
    }
  }
}

When I build the validator using the following:
aiken build --uplc
And then encode to named-debruijn using the following (where the result of uplc build is in the file artifacts/foo.validate_foo.uplc):
aiken uplc encode --to named-debruijn artifacts/foo.validate_foo.uplc
I get the following binary shown as text in the terminal:
3p�i_18!i_1932i_200i_20i_20!i_21!i_22!i_23S3i_2i_22�&Ai_242i_25S3i_63q�i_24i_23�&fi_21i_21i_25i_230i_0i_227\ai_1i_227Xfi_10
0i_8
��i_1016H
0i_8
    i_14H
         i_8
            i_17�ai_7
                     i_173i_100i_8      i_13H�i_1i_150i_8i_13�X�i_26i_112i_27S3i_6      3p�i_27!i_282i_29S3i_20i_0i_28�T�i_5
                                                                                I3List/Tuple/Constr contains more items than expected7\ai_1i_280i_8i_113i_5
I+Constr index did not match any type variant0i_7i_113i_9i_9H   i_30!i_31!i_32!i_3333i_63p�i_33i_31i_1
                                                                                !i_3433i_30i_303pi_31H�i_0i_34i_33i_32!i_350i_37Ti_35!i_360i_47Ti_36W4�qUΪ��W>��]%

I can see 2 problems there:
List/Tuple/Constr contains more items than expected
...
Constr index did not match any type variant

which seems to me that the encoding failed.

Problem 2

Details of problem 2
I have the following always passing validator:
use aiken/transaction.{ScriptContext}

validator {
  fn validate_smth(_datum: Void, _redeemer: Void, _ctx: ScriptContext) -> Bool {
    False
  }
}

building it using aiken build --uplc and then encoding to flat-namedDeBruijn and store in a binary file using the following:
aiken uplc encode --to named-debruijn artifacts/smth.validate_smth.uplc > smth.flat-namedDeBruijn

Then using the hexdump -C smth.flat-namedDeBruijn I get the following:
00000000  01 00 00 32 01 03 69 5f  30 00 21 03 69 5f 31 00  |...2..i_0.!.i_1.|
00000010  21 03 69 5f 32 00 21 03  69 5f 33 00 53 33 01 03  |!.i_2.!.i_3.S3..|
00000020  69 5f 30 00 04 4a 02 93  0b 2b 9a 01              |i_0..J...+..|
0000002c

Which seems correct. However when I compare it to the encoding that is produced using the plutus tool I get the following hexdump:
00000000  01 00 00 32 01 01 69 00  00 21 01 69 00 00 21 01  |...2..i..!.i..!.|
00000010  69 00 00 21 01 69 00 00  53 33 01 01 69 00 04 4a  |i..!.i..S3..i..J|
00000020  02 93 0b 2b 9a 01                                 |...+..|
0000002

To get the plutus version clone the Plutus repo https://github.com/input-output-hk/plutus, nix develop in the repo and then run the following to get the flat-namedDeBruijn encoded validator:
cabal run uplc convert -- -i /Users/.../smth.flat --if flat -o my_validator.flat-namedDeBruijn --of flat-namedDeBruijn

Where the smth.flat is generated using the aiken uplc flat artifacts/smth.validate_smth.uplc > smth.flat.
Then using the hexdump tool on the my_validator.flat-namedDeBruijn we get the dump above.
The encodings between aiken and plutus differ even for the simplest always passing validator.
And the problem is that the aiken named-debruijn encoding can't be loaded in Plutus resulting in the classical error:
Caused by: BadEncoding (0x000000010a61ec84,S {currPtr = 0x000000010a61ec63, usedBits = 7}) "usedBits /= 0"

Which suggests that the aiken encoding to named-debruijn is incorrect.
FYI. the plutus encoded named-debruijn works properly.

What should be the expected behavior?
I believe that the encoding to flat named-debruijn should be the same when comparing Aiken and Plutus and additionally the Aiken's encoding result shouldn't contain weird error messages which is suggesting that there's also some mistake in the encoding in the Aiken tool.

I see, we'll take a look. Something definitely seems off

I see for number 2 it looks like you are starting from 2 different bases. The first one using aiken --uplc starts from textual which has the original names. The second one you are feeding Plutus the flat version from the aiken artifact. At this point it's just Debruijn which has stripped the textual names.

Any progress on this?

found the issue and I'm pushing it to main

I see the issue. The textual parser is off when you read back in the dumped uplc.

Pushed a fix to main. Let me know if it resolves it.

Also to remove the i_0, i_1, etc. Convert to Debruijn first then NamedDebruijn

Fixed by 42544af7990219850cd1f863e384f415c6de1e77

Pushed a fix to main. Let me know if it resolves it.

Confirming, works nicely on main 👌


clean_build_artifacts should uniq paths before deleting

Fixes #821

Makes sense, thanks for the pull request :+1:


Special glyphs slightly malformed / text slightly blurry with Atlas Engine and Meslo Nerd font

Windows Terminal version
1.16.2523.0
Windows build number
10.0.19044.0
Other Software
PARTIALLY affects MacType (better font AA) but I'm mainly reporting the malformed glyphs
Steps to reproduce
Use WT preview with Atlas Engine and Meslo Nerd Font
Just to make sure, the exact font files MesloLGS-NF.zip
Expected Behavior
No response
Actual Behavior
This is without Atlas Engine (MacType enabled)

This is with Atlas Engine (MacType enabled). Less clear, shapes malformed.

With Atlas (MacType disabled, so only clarity/sharpness is slightly affected)

Without Atlas (MacType disabled, very visible in the Windows glyph. Just for reference)

FYI If you're curious about some of these changes, there's a lengthy discussion over here with the latest update being: https://github.com/microsoft/terminal/issues/13936#issuecomment-1254302780
In short, the blurriness appears to be correct actually.
In the soon to be released next 1.16 update you'll be able to specify precise fractional font sizes which will allow you to restore the previous look of glyphs. Text clarity can be improved if you enable ClearType in your settings.  This is an issue that's specific to Nerd Fonts and is something I'm currently working on as fast as I can.

Good to know, thanks for the fast reply. I'll keep it disabled for the time being and use MacType. Especially the NF symbols (OS logo, folder, git..) look a lot better for the time being

What I don't understand is why exactly some text gets blurry with Atlas AND MacType

FYI I'm going to continue tracking nerd font glyphs in #14022, which was the first issue to report it.


Issue with k8s.io/docs/setup/best-practices/cluster-large/

No more than 110 pods per node
is this a hard limit? I got a powerful hardware that uses only less than 20% of CPU and 15% of RAM, just wonder why the official document suggest not more than 110 pods. I found may post on the internet suggest we can increase this number to higher number  eg. 160 is the default value on DaoCloud that use K8S as the base.
Just wonder should this pods per node settings related to the CPU and RAM resources on the server?

Related page:https://kubernetes.io/docs/setup/best-practices/cluster-large/

/language en

110 is the default limit, you can tune it using the --max-pods (https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/) flag anyway.

There is some discussion about it. refer here

We could update the page to clarify whether these are hard limits.

This issue is already tracking in #42373
Iam closing this issue and suggesting people who are checking this issue to directly refer #42373

/close


[RSDK-3639] Create Location Client

This PR implements our LocationClient class, a class that exposes methods that make calls to the location service, and adds it as a property under out AppClient class. Right now, only the methods that've been labeled as high-priority have been implemented. The class also includes 3 shadow classes that essentially copy proto-defined classes but convert the Structs within those definitions to dictionaries and some Timestamps to Python datetimes. This specific PR also includes docstrings for the class. I'll make separate tickets and PR's for adding a snippet in the example notebook showing how to use the location_client and also some unit tests that depend on a mock location service.

One thing worth double checking is the format of the logs that I output as a result of passing a filepath 'dest' to get_robot_part_logs():

@njooma @stuqdog Okay I made all the changes we've talked about. I've run successful local tests but have yet to write a formal testing suite- that'll come next, and then I'll make the necessary changes to our example notebook and other docs stuff. Thanks!


error XA2002: Can not resolve reference: AsyncAwaitBestPractices

Are you aware of this? Is it intentional or a mistake?
error XA2002: Can not resolve reference: AsyncAwaitBestPractices, referenced by Xam.Shell.Badge.Android. Please add a NuGet package or assembly reference for AsyncAwaitBestPractices, or remove the reference to Xam.Shell.Badge.Android.

Hello, i'm not sure this should be the problem since AsyncAwaitBestPractices is a .NET Standard library, thus should be compatible with android and iOS projects... Anyway, this issue was a good base for me too re-visit Nuget dependencies and clean them up. Changes are available with the PR #30.

It's not that it didn't get added from the dependencies after instating the package. Thanks for the quick follow up


Sharing code between proton-native and react-native

Hi, thank you for your efforts in creating this framework!
I have an existing RN app that is under development and would love to add a desktop (Mac, windows, linux) target to it ... I understand I'd have to build new views but my redux actions, reducers, api, etc could all be re-used by them.
Any tips on how to structure the code so that I can keep it to one code base?

When I used react Native, I mostly used ignite cause it came with a really good structure. I can't remember exactly what it was, but basically you have a folder for state changes, folder for API, folder for stateless pure components and then a folder for state full components. IMHO that would be how I would do it. If you're doing anything that requires differing libraries, make that a separate file.
Basically, separate the different parts of it and make it modular (same API) so that to convert, you just have to change the react Native components and props to proton native components and props.


If a command does not exist, nerd should say so

If a user types nerd project sets test it will return the help message and Not enough arguments.

This behaviour is seemingly hardcoded in the CLI library so seems hard to change: https://github.com/mitchellh/cli/blob/33edc47170b5df54d2588696d590c5e20ee583fe/cli.go#L232 I can add a line in the main.go that checks for 127 but that is brittle and doesn't work on subcommands. I'm removing this from this milestone.


TASVideos Extractor - Beta

This is a beta, blablabla

didn't mean to add this here trying to delete