求助：TypeError: Caught TypeError in DataLoader worker process 0.

在多卡训练的时候，遇到
=====Epoch: 000001=====
<torch.utils.data.dataloader.DataLoader object at 0xfffef44d4040><torch.utils.data.dataloader.DataLoader object at 0xffff03a8af80>
:914: ImportWarning: TEMetaPathFinder.find_spec() not found; falling back to find_module()
:914: ImportWarning: TEMetaPathFinder.find_spec() not found; falling back to find_module()
Traceback (most recent call last):
File "/home/LucaOne-master/src/run.py", line 1305, in
main()
File "/home/LucaOne-master/src/run.py", line 1290, in main
global_step, avg_loss, max_metric_model_info = train(
File "/home/LucaOne-master/src/trainer.py", line 162, in train
for step, batch in enumerate(dataloader):
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 630, in next
data = self._next_data()
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 1345, in _next_data
return self._process_data(data)
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 1371, in _process_data
data.reraise()
File "/usr/local/lib/python3.10/dist-packages/torch/_utils.py", line 694, in reraise
raise exception
TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
data = fetcher.fetch(index)
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py", line 42, in fetch
return self.collate_fn(data)
File "/home/LucaOne-master/src/batch_converter.py", line 250, in call
labels_b, strs_b, input_ids_b, position_ids_b, token_type_ids_b, max_len_b = self.call_single(batch_size, seq_types_b, seqs_b, seq_labels_b)
File "/home/LucaOne-master/src/batch_converter.py", line 209, in call_single
return self.process_label(max_len, labels), strs, input_ids, position_ids, token_type_ids, max_len
File "/home/LucaOne-master/src/batch_converter.py", line 88, in process_label
new_encoded_labels[level1_name][level2_name] = torch.tensor(new_items, dtype=torch.int64)
File "/usr/local/lib/python3.10/dist-packages/torch_npu/contrib/transfer_to_npu.py", line 56, in decorated
return fn(*args, **kwargs)
TypeError: only integer tensors of a single element can be converted to an index
请问TypeError: Caught TypeError in DataLoader worker process 0.这个问题该怎么解决？

ignore_index我没有修改过，就是def __init__里面指定的-100，
用单卡训练可以跑起来，现在想用多卡训练就会报错，感觉跟dataloader有关，麻烦作者帮忙看看，这个问题困扰我好几天了，下面是全部的运行日志。
root@localhost:/home/LucaOne-master/src/training# source /usr/local/Ascend/ascend-toolkit/set_env.sh
root@localhost:/home/LucaOne-master/src/training# sh run_multi_v2.0.sh
[2024-09-04 01:38:13,938] torch.distributed.run: [WARNING]
[2024-09-04 01:38:13,938] torch.distributed.run: [WARNING] *****************************************
[2024-09-04 01:38:13,938] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
[2024-09-04 01:38:13,938] torch.distributed.run: [WARNING] *****************************************
/usr/local/lib/python3.10/dist-packages/torch_npu/contrib/transfer_to_npu.py:209: ImportWarning:
*************************************************************************************************************
The torch.Tensor.cuda and torch.nn.Module.cuda are replaced with torch.Tensor.npu and torch.nn.Module.npu now..
The torch.cuda.DoubleTensor is replaced with torch.npu.FloatTensor cause the double type is not supported now..
The backend in torch.distributed.init_process_group set to hccl now..
The torch.cuda.* and torch.cuda.amp.* are replaced with torch.npu.* and torch.npu.amp.* now..
The device parameters have been replaced with npu in the function below:
torch.logspace, torch.randint, torch.hann_window, torch.rand, torch.full_like, torch.ones_like, torch.rand_like, torch.randperm, torch.arange, torch.frombuffer, torch.normal, torch._empty_per_channel_affine_quantized, torch.empty_strided, torch.empty_like, torch.scalar_tensor, torch.tril_indices, torch.bartlett_window, torch.ones, torch.sparse_coo_tensor, torch.randn, torch.kaiser_window, torch.tensor, torch.triu_indices, torch.as_tensor, torch.zeros, torch.randint_like, torch.full, torch.eye, torch._sparse_csr_tensor_unsafe, torch.empty, torch._sparse_coo_tensor_unsafe, torch.blackman_window, torch.zeros_like, torch.range, torch.sparse_csr_tensor, torch.randn_like, torch.from_file, torch._cudnn_init_dropout_state, torch._empty_affine_quantized, torch.linspace, torch.hamming_window, torch.empty_quantized, torch._pin_memory, torch.autocast, torch.load, torch.Generator, torch.Tensor.new_empty, torch.Tensor.new_empty_strided, torch.Tensor.new_full, torch.Tensor.new_ones, torch.Tensor.new_tensor, torch.Tensor.new_zeros, torch.Tensor.to, torch.nn.Module.to, torch.nn.Module.to_empty
*************************************************************************************************************
warnings.warn(msg, ImportWarning)
:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
/usr/local/lib/python3.10/dist-packages/tensorflow/python/framework/dtypes.py:246: DeprecationWarning: np.bool8 is a deprecated alias for np.bool_.  (Deprecated NumPy 1.24)
np.bool8: (False, True),
/usr/local/lib/python3.10/dist-packages/tensorflow/python/framework/dtypes.py:246: DeprecationWarning: np.bool8 is a deprecated alias for np.bool_.  (Deprecated NumPy 1.24)
np.bool8: (False, True),
:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
Output dir, logger, tb-logger created succeed.
:671: ImportWarning: TBEMetaPathLoader.exec_module() not found; falling back to load_module()
:671: ImportWarning: TBEMetaPathLoader.exec_module() not found; falling back to load_module()
:914: ImportWarning: TEMetaPathFinder.find_spec() not found; falling back to find_module()
:914: ImportWarning: TEMetaPathFinder.find_spec() not found; falling back to find_module()
:914: ImportWarning: SixMetaPathImporter.find_spec() not found; falling back to find_module()
:914: ImportWarning: SixMetaPathImporter.find_spec() not found; falling back to find_module()
:671: ImportWarning: TBEMetaPathLoader.exec_module() not found; falling back to load_module()
:671: ImportWarning: TBEMetaPathLoader.exec_module() not found; falling back to load_module()
world size: 2
Alphabet, tokenizer_dir: ../vocab/lucagplm/v2.0/vocab.txt
Alphabet, tokenizer_dir: ../vocab/lucagplm/v2.0/vocab.txt
Pretrain Tasks:
token_level/gene_mask
token_level/prot_mask
span_level/gene_type
span_level/prot_homo
span_level/prot_site
span_level/prot_domain
seq_level/gene_taxonomy
seq_level/prot_taxonomy
seq_level/prot_keyword
structure_level/prot_structure
Pretrain Tasks:
token_level/gene_mask
token_level/prot_mask
span_level/gene_type
span_level/prot_homo
span_level/prot_site
span_level/prot_domain
seq_level/gene_taxonomy
seq_level/prot_taxonomy
seq_level/prot_keyword
structure_level/prot_structure
n_gpu: 2
n_gpu: 2, use: DataLoader
n_gpu: 2, use: DataLoader
/usr/local/lib/python3.10/dist-packages/urllib3/util/ssl.py:260: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated
context = SSLContext(ssl_version or PROTOCOL_TLS_CLIENT)
/usr/local/lib/python3.10/dist-packages/urllib3/util/ssl.py:260: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated
context = SSLContext(ssl_version or PROTOCOL_TLS_CLIENT)
/usr/local/lib/python3.10/dist-packages/urllib3/connection.py:407: DeprecationWarning: ssl.match_hostname() is deprecated
match_hostname(cert, asserted_hostname)
/usr/local/lib/python3.10/dist-packages/urllib3/connection.py:407: DeprecationWarning: ssl.match_hostname() is deprecated
match_hostname(cert, asserted_hostname)
Has Pair: False
Has Pair: False
Init lr:  0.0002
Peak lr:  0.0002
Scheduler_type: step
Use Warmup, warmup_steps=2000, max_steps=16000000
Init lr:  0.0002
Peak lr:  0.0002
Scheduler_type: step
Use Warmup, warmup_steps=2000, max_steps=16000000
=====Epoch: 000001=====
:914: ImportWarning: TEMetaPathFinder.find_spec() not found; falling back to find_module()
:914: ImportWarning: TEMetaPathFinder.find_spec() not found; falling back to find_module()
Traceback (most recent call last):
File "/home/LucaOne-master/src/run.py", line 1306, in
main()
File "/home/LucaOne-master/src/run.py", line 1291, in main
global_step, avg_loss, max_metric_model_info = train(
File "/home/LucaOne-master/src/trainer.py", line 161, in train
for step, batch in enumerate(dataloader):
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 630, in next
data = self._next_data()
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 1345, in _next_data
return self._process_data(data)
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 1371, in _process_data
data.reraise()
File "/usr/local/lib/python3.10/dist-packages/torch/_utils.py", line 694, in reraise
raise exception
TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
data = fetcher.fetch(index)
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py", line 42, in fetch
return self.collate_fn(data)
File "/home/LucaOne-master/src/batch_converter.py", line 250, in call
labels_b, strs_b, input_ids_b, position_ids_b, token_type_ids_b, max_len_b = self.call_single(batch_size, seq_types_b, seqs_b, seq_labels_b)
File "/home/LucaOne-master/src/batch_converter.py", line 209, in call_single
return self.process_label(max_len, labels), strs, input_ids, position_ids, token_type_ids, max_len
File "/home/LucaOne-master/src/batch_converter.py", line 88, in process_label
new_encoded_labels[level1_name][level2_name] = torch.tensor(new_items, dtype=torch.int64)
File "/usr/local/lib/python3.10/dist-packages/torch_npu/contrib/transfer_to_npu.py", line 56, in decorated
return fn(*args, **kwargs)
TypeError: only integer tensors of a single element can be converted to an index
Traceback (most recent call last):
File "/home/LucaOne-master/src/run.py", line 1306, in
main()
File "/home/LucaOne-master/src/run.py", line 1291, in main
global_step, avg_loss, max_metric_model_info = train(
File "/home/LucaOne-master/src/trainer.py", line 161, in train
for step, batch in enumerate(dataloader):
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 630, in next
data = self._next_data()
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 1345, in _next_data
return self._process_data(data)
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 1371, in _process_data
data.reraise()
File "/usr/local/lib/python3.10/dist-packages/torch/_utils.py", line 694, in reraise
raise exception
TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
data = fetcher.fetch(index)
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py", line 42, in fetch
return self.collate_fn(data)
File "/home/LucaOne-master/src/batch_converter.py", line 250, in call
labels_b, strs_b, input_ids_b, position_ids_b, token_type_ids_b, max_len_b = self.call_single(batch_size, seq_types_b, seqs_b, seq_labels_b)
File "/home/LucaOne-master/src/batch_converter.py", line 209, in call_single
return self.process_label(max_len, labels), strs, input_ids, position_ids, token_type_ids, max_len
File "/home/LucaOne-master/src/batch_converter.py", line 88, in process_label
new_encoded_labels[level1_name][level2_name] = torch.tensor(new_items, dtype=torch.int64)
File "/usr/local/lib/python3.10/dist-packages/torch_npu/contrib/transfer_to_npu.py", line 56, in decorated
return fn(*args, **kwargs)
TypeError: only integer tensors of a single element can be converted to an index
[2024-09-04 01:48:39,568] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 70495 closing signal SIGTERM
[2024-09-04 01:48:40,884] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 1 (pid: 70496) of binary: /usr/bin/python
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launch.py", line 196, in
main()
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launch.py", line 192, in main
launch(args)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launch.py", line 177, in launch
run(args)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/run.py", line 797, in run
elastic_launch(
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
run.py FAILED
Failures:
<NO_OTHER_FAILURES>
Root Cause (first observed failure):
[0]:
time      : 2024-09-04_01:48:39
host      : localhost.localdomain
rank      : 1 (local_rank: 1)
exitcode  : 1 (pid: 70496)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
Process ForkServerProcess-7:
Process ForkServerProcess-2:
Process ForkServerProcess-5:
Process ForkServerProcess-9:
Process ForkServerProcess-3:
Process ForkServerProcess-8:
Process ForkServerProcess-6:
Process ForkServerProcess-4:
Traceback (most recent call last):
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 65, in wrapper
raise exp
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 65, in wrapper
raise exp
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
key, func_name, detail = resource_proxy[TASK_QUEUE].get()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
key, func_name, detail = resource_proxy[TASK_QUEUE].get()
File "", line 2, in get
File "", line 2, in get
Traceback (most recent call last):
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
kind, result = conn.recv()
File "/usr/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
kind, result = conn.recv()
File "/usr/lib/python3.10/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/usr/lib/python3.10/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/usr/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/usr/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/usr/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
raise EOFError
File "/usr/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
raise EOFError
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
EOFError
EOFError
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 65, in wrapper
raise exp
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 65, in wrapper
raise exp
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
key, func_name, detail = resource_proxy[TASK_QUEUE].get()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
key, func_name, detail = resource_proxy[TASK_QUEUE].get()
File "", line 2, in get
File "", line 2, in get
File "/usr/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
kind, result = conn.recv()
File "/usr/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
kind, result = conn.recv()
File "/usr/lib/python3.10/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/usr/lib/python3.10/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/usr/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/usr/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/usr/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
raise EOFError
File "/usr/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
raise EOFError
EOFError
EOFError
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 65, in wrapper
raise exp
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 65, in wrapper
raise exp
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 65, in wrapper
raise exp
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
key, func_name, detail = resource_proxy[TASK_QUEUE].get()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
key, func_name, detail = resource_proxy[TASK_QUEUE].get()
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
key, func_name, detail = resource_proxy[TASK_QUEUE].get()
File "", line 2, in get
File "", line 2, in get
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "", line 2, in get
File "/usr/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
kind, result = conn.recv()
File "/usr/lib/python3.10/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 65, in wrapper
raise exp
File "/usr/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/usr/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
kind, result = conn.recv()
File "/usr/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
raise EOFError
File "/usr/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
kind, result = conn.recv()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
func(*args, **kwargs)
File "/usr/lib/python3.10/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
EOFError
File "/usr/lib/python3.10/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
key, func_name, detail = resource_proxy[TASK_QUEUE].get()
File "/usr/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/usr/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "", line 2, in get
File "/usr/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
raise EOFError
File "/usr/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
raise EOFError
File "/usr/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
kind, result = conn.recv()
File "/usr/lib/python3.10/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
EOFError
EOFError
File "/usr/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/usr/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
raise EOFError
EOFError
root@localhost:/home/LucaOne-master/src/training# :914: ImportWarning: TEMetaPathFinder.find_spec() not found; falling back to find_module()
/usr/local/lib/python3.10/dist-packages/datasets/iterable_dataset.py:279: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/LucaOne-master/dataset/lucagplm/v2.0/train/20231123140614791gwknta8l88o_M2_1_599_0-0_TableSink2-0-.csv' encoding='utf-8'>
yield key, example
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '

作者大大，请问这个模型在GPU上多卡是能跑通的嘛？
我用的是npu，单卡能跑通，多卡不行。


Beta: config variables

I can't seem to get rclone to recognize some of the environment variables outlined here:
https://github.com/ncw/rclone/issues/616
Example.

# env | grep RCLONE
RCLONE_STATS=5s
RCLONE_CONFIG=/home/robert/.rclone.conf

The RCLONE_CONFIG works but the RCLONE_STATS does not.  Also there is no way to tell what variables have been applied.  My suggestion is to include in the INFO or DEBUG output the parameters applied similarly to the echo'd command line already does.  That way we don't need to guess if there was a typo. :)
rclone -vvv ls robacd:
2017/03/02 13:08:13 DEBUG : rclone: Version "v1.35-137-ge2f0feeβ" **starting with parameters** ["rclone" "-vvv" "ls" "robacd:"]

Can you also tell me why RCLONE_STATS doesn't work?  Is that a bug?

The environment vars change the default values of the flags as you can see here
$ rclone -h | grep stats
      --stats duration                    Interval between printing stats, e.g 500ms, 60s, 5m. (0 to disable) (default 1m0s)
$ RCLONE_STATS=5s rclone -h | grep stats
      --stats duration                    Interval between printing stats, e.g 500ms, 60s, 5m. (0 to disable) (default 5s)

So you can check that they are applied like that.
The --stats is a slightly special case as commands which don't normally give stats (like ls) won't give stats unless you actually pass the --stats parameter.
Logging the variables read from the environment is complicated by the fact that you need them set up to log stuff...

ahhhhh.  Beautiful.  Adding that tidbit to the DOCs may make sense for the rclone -h to verify. Great stuff.

I've done exactly that - thanks for the suggestion :-)

I realize you said that --stats was a special case:

The --stats is a slightly special case as commands which don't normally give stats (like ls) won't give stats unless you actually pass the --stats parameter.

I would, however, expect that this would print stats.  But it is not.
-># export RCLONE_STATS=5m
-># export RCLONE_CONFIG=/home/robert/.rclone.conf
-># export RCLONE_TRANSFERS=16
-># export RCLONE_CHECKERS=64
-># export RCLONE_ONE_FILE_SYSTEM=true
->#  rclone \
>   --size-only \
>  copy -v /data/Media1 /data/Media2

I actually thought the rclone may have been hung so I killed it but then I checked other things and it is copying data.

Human error. reclosed. :+1:


[stdlib] Fix AnyHashable's Equatable/Hashable conformance

AnyHashable has a few corner cases where it's not correctly conforming to Equatable and Hashable. This breaks Set and Dictionary invariants and can cause unexpected behavior and/or traps.
This change modifies AnyHashable to fix these cases. It is relatively important to fix this now, because to complete SE-0206, we need to switch integers to type-dependent hashing, and that exacerbates the issue. (The hashing change is also included in this PR.)

Fix transitivity of ==. Previously, comparisons involving AnyHashable values with Objective-C provenance were handled specially, breaking Equatable:
let a = (42 as Int as AnyHashable)
let b = (42 as NSNumber as AnyHashable)
let c = (42 as Double as AnyHashable)
a == b // true
b == c // true
a == c // was false(!), now true

let d = ("foo" as AnyHashable)
let e = ("foo" as NSString as AnyHashable)
let f = ("foo" as NSString as NSAttributedStringKey as AnyHashable)
d == e // true
e == f // true
d == f // was false(!), now true

Fix Hashable conformance for numeric types boxed into AnyHashable:
b == c // true
b.hashValue == c.hashValue // was false(!), now true

For numeric types, SE-0170 changed bridging so that conversion to NSNumber loses information about the original type. Therefore, to ensure AnyHashable correctly conforms to Equatable, it must follow/emulate NSNumber's rules for equality. The same goes for _SwiftNewtypeWrapper types and NSString. To fix these, this PR defines custom AnyHashable boxes for all numeric types and newtypes. The custom box canonicalizes values that bridge to the same Objective-C representation so that they compare equal and hash the same way, independent of the original Swift type.
SE-0143 added conditional Hashable conformances for arrays, sets and dictionaries, so values of these collection types may themselves be converted to AnyHashable. These collections are also bridged to NSArray, NSSet and NSDictionary, respectively. Therefore, their AnyHashable forms must follow the rules equality set out by Foundation. This adds a number of new cases where Equatable/Hashable conformance is currently broken -- for example:
let x: AnyHashable = [1, 2, 3] as [UInt8]
let y: AnyHashable = [1, 2, 3] as NSArray
let z: AnyHashable = [1, 2, 3] as [Double]

x == y // true
y == z // true
x == z // was false(!), now true

x.hashValue == y.hashValue // true
y.hashValue == z.hashValue // was false(!), now true
x.hashValue == z.hashValue // was false(!), now true

This PR fixes this by adding additional AnyHashable boxes for these three collection types. The boxes for sets and dictionaries aren't particularly efficient. However, they're internal to the stdlib, so we'll be able to change them later in a future release without breaking the ABI.
Additional changes:

Remove AnyHashable._usedCustomRepresentation. The provenance of a value should not affect its behavior.
Allow AnyHashable values to be downcasted into compatible types more often. Since equality implies that (1 as UInt8 as AnyHashable) must be substitutable with (1 as Double as AnyHashable), it follows that we should be able to successfully downcast either value to both types. (This is currently done by bridging to Objective-C and back, so it only works when the Objective-C runtime is available and Foundation is loaded.)
Forward _rawHashValue(seed:) to AnyHashable box. This fixes AnyHashable hashing for types that customize single-shot hashing.
Enable integer values of different types to produce different hashes, so that e.g. (1 as UInt8).hashValue will typically differ from (1 as UInt64).hashValue. This completes SE-0206 by feeding hasher only as many bytes as necessary to represent each type.(42 as UInt8).hashValue == (42 as Int).hashValue // was true, now false

https://bugs.swift.org/browse/SR-7496
rdar://problem/39648819

@swift-ci please test

@swift-ci please test source compatibility

Build failed
Swift Test OS X Platform
Git Sha - e081c32654efba762bcac3bb5bcb218de995188d

Some open issues:

AnyHashable downcasting must not involve bridging, because even if the ObjC runtime is available, Foundation may not have been imported. (The original code only tried round-trip bridging if the box came from Foundation, so it did not have this issue.)
Ensure that the critical NSDictionary -> [AnyHashable: Any] -> [String: Any] bridging+casting path does not rehash the dictionary more than once.

The second step need not do a full rehash, because "foo" as AnyHashable hashes exactly the same as "foo".
This is also true for newtypes wrapping Strings, so [AnyHashable: Any] -> [NSAttributedStringKey: Any] should not do a full rehash.

Investigate if we can guarantee the same for NSDictionary -> [AnyHashable: Any] -> [Int: Any].

Sadly, this seems unlikely, because hashing is different in all three steps.

We should allow downcasting from any AnyHashable holding a numeric value to any numeric type that can hold that value exactly, without requiring Foundation to be loaded

Ugly solution: We could manually switch on the target type

Verify that the AnyHashable representations of Array/Set/Dictionary values holding numeric types & newtypes (1) work correctly (e.g. AnyHashable-boxed sets holding equal numbers compare equal, even if the original types were different), and (2) bridge correctly with NSArray/NSSet/NSDictionary.
Find out how to implement downcasting from an AnyHashable holding some type to an "extensible enum" (newtype) that wraps that same type (e.g., NSAttributedStringKey)

These newtypes get bridged to their raw value (e.g. NSAttributedStringKey becomes NSString), so their AnyHashable representation should be interchangeable with their raw value. I.e., ideally, this should return non-nil: "hello" as AnyHashable as? NSAttributedStringKey.
For extra fun, the raw value can be a numeric type, with its own complications

Fix Float80's AnyHashable semantics. Float80 values should not compare equal to Double values unless they represent the exact same number.

@swift-ci test

Build failed
Swift Test Linux Platform
Git Sha - e081c32654efba762bcac3bb5bcb218de995188d

Build failed
Swift Test OS X Platform
Git Sha - e081c32654efba762bcac3bb5bcb218de995188d

@swift-ci test

Build failed
Swift Test Linux Platform
Git Sha - e04d05a62d4f41c1ce37c6b8e3870d0eb3010a89

Build failed
Swift Test OS X Platform
Git Sha - e04d05a62d4f41c1ce37c6b8e3870d0eb3010a89

@swift-ci please test

Build failed
Swift Test OS X Platform
Git Sha - 44cb54bce67a26d9496078b8b48346dc7ad205a2

Build failed
Swift Test Linux Platform
Git Sha - 44cb54bce67a26d9496078b8b48346dc7ad205a2

@swift-ci test

@swift-ci test source compatibility

@swift-ci please test

@swift-ci please test

@swift-ci please test linux platform

@swift-ci please test source compatibility

@swift-ci please test source compatibility

@DougGregor If you have a bit of time, could you take a quick look at this? I think you may be able to spot unintended consequences that I missed during testing.
After this lands, I intend to cherry-pick it to 4.2, too.

@swift-ci please test

@swift-ci please test

@swift-ci please test source compatibility


Ramiro color sensor plugin

Added functionality for the color sensor visuals

Implemented action server and client

Please review guys @eborghi10 @ernestmc @LucasJSch @stevendes

Ready for review @LucasJSch @stevendes @eborghi10 @ernestmc

If you want, merge it to the target branch.


Error in Waterfox downloading GIF

GIF download was not working in Waterfox (with tracking protection enabled). The error in browser console was:
"The resource at "https://video.twimg.com/tweet_video/..." was blocked because tracking protection is enabled."
Apparently, adding "https://video.twimg.com/" to the "permissions" of the addon on manifest.json was enough to fix it. Maybe waterfox doesn't accept wildcards to bypass tracking protection?

#9


Error: Building package sdl1:x64-windows failed with: BUILD_FAILED

Host Environment

Windows 8.1
Visual Studio Commity 2019

To Reproduce
Steps to reproduce the behavior:
`./vcpkg install sdl1:x64-windows
Failure logs
Error: Building package sdl1:x64-windows failed with: BUILD_FAILED
Please ensure you're using the latest portfiles with .\vcpkg update, then
submit an issue at https://github.com/Microsoft/vcpkg/issues including:
Package: sdl1:x64-windows
Vcpkg version: 2020.02.04-nohash
Additionally, attach any relevant sections from the log files above.
Additional context
Any Ideas?
Note that I have already updated vcpkg and git pull

In the log build-x64-windows-rel-out.log is written:
TRACKER : error TRK0005: Failed to locate: "CL.exe". The system cannot find the file specified.

Do you install Visual C++ https://docs.microsoft.com/en-us/cpp/overview/visual-cpp-in-visual-studio?view=vs-2019?

Pinging @Amine74

Thanks for posting this issue. Please open a new issue if this is still a problem for you.


Fix bug: enforce 1 argument in sort -i command

Codecov Report

Merging #185 (dfbf734) into master (5f4a4e4) will increase coverage by 0.08%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##             master     #185      +/-   ##
============================================
+ Coverage     54.49%   54.58%   +0.08%
- Complexity      528      530       +2
============================================
  Files           102      102
  Lines          2347     2347
  Branches        313      313
============================================
+ Hits           1279     1281       +2
+ Misses          947      945       -2
  Partials        121      121

Impacted Files
Coverage Δ

.../address/logic/commands/SortInternshipCommand.java
0.00% <ø> (ø)

...eedu/address/logic/commands/SortPersonCommand.java
61.90% <ø> (ø)

...ress/logic/parser/SortInternshipCommandParser.java
0.00% <0.00%> (ø)

...ava/seedu/address/model/internship/Internship.java
36.73% <0.00%> (+2.04%)
:arrow_up:

...seedu/address/model/internship/InternshipRole.java
70.00% <0.00%> (+10.00%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Weekly fixes for Prism

Closes #1106
Closes #1103
Closes #1104

@XVincentX I am still not able to execute this as per the documentation -
docker run --init --rm -it -v "$(pwd):/tmp" -P stoplight/prism:3 mock -h <IP_ADDRESS> /tmp/openapi.v1.json
[11:50:23 AM] › [CLI] …  awaiting  Starting Prism…
It just breaks, still not able to use the prism image

@sgrmgj-quotient We have not released yet a new Prism version. You would need to build the docker image by yourself first:

Clone the source code
docker built -t prism .

okay cool I will do that thanks @XVincentX


feat: support for custom dates

Summary
fix #25, and enhance it.
What kind of change does this PR introduce? (check at least one)

[X] Bugfix
[X] Feature
[ ] Code style update
[ ] Refactor
[X] Docs
[ ] Build-related changes
[ ] Other, please describe:

If changing the UI of default theme, please provide the before/after screenshot:
Does this PR introduce a breaking change? (check one)

[ ] Yes
[X] No

If yes, please describe the impact and migration path for existing applications:
The PR fulfills these requirements:

[X] When resolving a specific issue, it's referenced in the PR's title (e.g. fix #xxx[,#xxx], where "xxx" is the issue number)

You have tested in the following browsers: (Providing a detailed version will be better.)

[x] Chrome
[ ] Firefox
[ ] Safari
[ ] Edge
[ ] IE

If adding a new feature, the PR's description includes:

[X] A convincing reason for adding this feature
[ ] Related documents have been updated
[ ] Related tests have been updated

To avoid wasting your time, it's best to open a feature request issue first and wait for approval before working on it.
Other information:

can we add a linter such as eslint? IDE automatically remove useless spaces

@Himself65  I've just added the linter.👌

@Himself65 I've just added the linter.👌
that's great

It shows invalid date In Safari. Have you considered leveraging dayjs instead of moment? It's more light-weight and it might resolve the bug.

ok, I will try that


Kabanero operator to deploy kabanero eventing operator

This depends on #579.
The kabanero controller currently deploys kabanero-eventing if the Kabanero CR is configured for it.  This will be changing.  Kabanero should always deploy kabanero-eventing.  Kabanero eventing is now an operator (although it's going to look more or less like it does now).  In #579 we'll get the new YAMLs to deploy.  Eventing should no longer be optional.  It will always be deployed when a Kabaner CR instance is created, in the same way we deploy the CLI service, and landing page.

This delivered in #675.  There may be additional changes that can be handled in follow-on PRs.


Cannot download tinkey - The project to be billed is associated with a closed billing account.

Trying to download tinkey (as per https://developers.google.com/tink/install-tinkey) using the link https://storage.googleapis.com/tinkey/tinkey-1.9.0.tar.gz
Fails with following error message:
<Error>
UserProjectAccountProblem
<Message>The project to be billed is associated with a closed billing account.</Message>
The billing account for the owning project is disabled in state closed
</Error>

Thank you for the report. We will take a look.

This issue should be resolved.


Update s3, sdk-core, sqs to 2.16.35

Updates

software.amazon.awssdk:s3
software.amazon.awssdk:sdk-core
software.amazon.awssdk:sqs

from 2.16.26 to 2.16.35.
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "software.amazon.awssdk" } ]

labels: library-update, semver-patch

Superseded by #143.


Fixed .NET version name

Summary
Noticed a mistake in the docs calling .NET 6.0 ".NET Core 6.0"

:+1:


Conflict with Create Deco Stuff

When iam installing your mod the copper/ gold / netherite bars of create deco will loose all there specific textures
https://imgur.com/a/0z3nrVx

For the record, this bug also affects Tinkers Construct. The cause is you change the vanilla models for iron bars making them incompatible with mods (or resource packs) that extend them. I see little reason to override the vanilla models to add your own bars, however if you do wish to change them be kind to other modders by using the same texture names as vanilla.

For the record, this bug also affects Tinkers Construct. The cause is you change the vanilla models for iron bars making them incompatible with mods (or resource packs) that extend them. I see little reason to override the vanilla models to add your own bars, however if you do wish to change them be kind to other modders by using the same texture names as vanilla.

It will be fixed, but as I'm busy with work stuff it's not going to be soon


Cannot python serialize arbitrary classes that alter getattr behavior

Initial Checks

[X] I confirm that I'm using Pydantic V2

Description
I wish to serialize an arbitrary object that is an instance of a 3rd party class. However, the 3rd party class implements __getattr__(self, attr) that alters the normal behavior of getattr by wrapping attr in another class. This results in TypeError: 'function' object cannot be converted to 'SchemaSerializer'. However, all that matters is that the 3rd party class's getattr does not raise an error if __pydantic_serializer__ or __dataclass_fields__ are missing.
Example Code
In [1]: from pydantic import ConfigDict, BaseModel

In [2]: class A:
   ...:     def __getattr__(self, attr):
   ...:         return 5
   ...:

In [3]: class Schema(BaseModel):
   ...:     a: A
   ...:     model_config = ConfigDict(arbitrary_types_allowed=True)
   ...:

In [4]: Schema(a=A()).model_dump()
in BaseModel.model_dump(self, mode, include, exclude, context, by_alias, exclude_unset, exclude_defaults, exclude_none, round_trip, warnings, serialize_as_any)
    309 def model_dump(
    310     self,
    311     *,
   (...)
    322     serialize_as_any: bool = False,
    323 ) -> dict[str, Any]:
    324     """Usage docs: https://docs.pydantic.dev/2.7/concepts/serialization/#modelmodel_dump
    325
    326     Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.
   (...)
    345         A dictionary representation of the model.
    346     """
--> 347     return self.__pydantic_serializer__.to_python(
    348         self,
    349         mode=mode,
    350         by_alias=by_alias,
    351         include=include,
    352         exclude=exclude,
    353         context=context,
    354         exclude_unset=exclude_unset,
    355         exclude_defaults=exclude_defaults,
    356         exclude_none=exclude_none,
    357         round_trip=round_trip,
    358         warnings=warnings,
    359         serialize_as_any=serialize_as_any,
    360     )

TypeError: 'int' object cannot be converted to 'SchemaSerializer'

Fixed by:

In [5]: class A:
   ...:     def __getattr__(self, attr):
   ...:         if attr == "__pydantic_serializer__" or attr == "__dataclass_fields__":
   ...:             raise AttributeError
   ...:         return 5
   ...:

In [6]: class Schema(BaseModel):
   ...:     a: A
   ...:     model_config = ConfigDict(arbitrary_types_allowed=True)
   ...:

In [7]: Schema(a=A()).model_dump()
Out[7]: {'a': <__main__.A at 0x7f53d94a0400>}

Python, Pydantic & OS Version
pydantic version: 2.7.4
        pydantic-core version: 2.18.4
          pydantic-core build: profile=release pgo=true
               python version: 3.10.14 (main, May  6 2024, 19:42:50) [GCC 11.2.0]
                     platform: Linux-5.15.0-1062-gcp-x86_64-with-glibc2.31
             related packages: typing_extensions-4.12.2 pydantic-settings-2.3.3
                       commit: unknown

Hi @ro-joshi,
I wouldn't consider this a bug, but rather a consequence of the internal implementation of serialization for arbitrary types.
Perhaps you could subclass the types in question and wrap the parent's __getattr__ with said serializer check.
Closing as not planned for now, but happy to reconsider if someone has a compelling argument as to why this should be changed.


Assertion `mtu_floor <= mtu_ceiling' failed.

utp_internal.cpp:1301: void UTPSocket::mtu_search_update(): Assertion `mtu_floor <= mtu_ceiling' failed.

??

I believe this was a memory corruption issue, not a bug in libutp.


create internal document that maps public roadmap items to vault projects

related slack thread:
https://shopify.slack.com/archives/C01FT69928K/p1692214677566879


Update readme for Mac OS with alpha4 instructions

Updating instructions to

uninstall old kvm
updating brew script
installing new ones

@suhasj can this be closed? Looks like there is no further action on this as this is related to alpha4.

Readme is updated already. Closing this bug.


Multi-Part Challenges break a side track

At the moment, we cannot include multi-part challenges in a side track, see an example here:
https://thecodingtrain.com/tracks/games/46-asteroids
@lobertrand what do you think the best way to handle this is?
@kfahn22 do you think we should remove them temporarily until we have this resolved?

Hello, I guess one way to handle this could "spread" the multiple parts on the track, something like this :

I don't know how hard that would be to implement but I'm willing to try :smiley:

Yes I will remove

Once this is resolved we can revert #739

@lobertrand this seems like a great solution to me!

Hi @shiffman @kfahn22 :steam_locomotive:
I'm working on fixing this issue and I was wondering which title I should display in the timeline for multi-part challenges.
If we take this example :
{
  "title": "Agar.io",
  "parts": [
    { "title": "Agar.io - Part 1 (Basic Game Mechanics)" },
    { "title": "Agar.io - Part 2 (Networking with Socket.IO and Node.js)" }
  ]
}

Which way should we display the title?
A. video.parts[*].title

B. video.title - Part #

Option A works great with the above example but less with the following ones:
{
  "title": "Polar Noise Loops",
  "parts": [
    { "title": "Polar Perlin Noise Loops" },
    { "title": "Perlin Noise GIF Loops" }
  ]
}

{
  "title": "Traveling Salesperson",
  "parts": [
    { "title": "Part 1: Traveling Salesperson (TSP)" },
    { "title": "Part 2: Lexicographic Order" },
    { "title": "Part 3: TSP with Lexicographic Order" },
    { "title": "Part 4: TSP with Genetic Algorithm" },
    { "title": "Part 5: TSP with Genetic Algorithm and Crossover" }
  ]
}

Since the title of these parts don't include the name of the challenge itself, we cannot easily tell that they are parts of the same challenge.
I think we should homogenize the way part titles are formatted in the JSON files. The question is, should they include the name of the challenge? or just something like this (which eliminates data redundancy):
{
  "title": "Agar.io",
  "parts": [
    { "title": "Part 1 (Basic Game Mechanics)" },
    { "title": "Part 2 (Networking with Socket.IO and Node.js)" }
  ]
}

In my opinion, the part title should not contain the challenge title, only Part #, and/or the part-specific title.

I guess you could make a good argument either way.  Another consideration is that whatever is chosen will affect the YouTube description that is generated.

Another consideration is that whatever is chosen will affect the YouTube description that is generated.  (Or maybe not, is it always just Part 2, Part 3 regardless of what the title is?)

In the YouTube description, currently the part-specific title is used to link to other parts of the video.

@kfahn22 @dipamsen Thanks for the feeback!
@shiffman I'm gonna call PR #743 done, not using the part-specific titles for now
I'll open a new PR to homogenize the part-specific titles and adjust their usage accordingly :smiley:

Fantastic! Sounds good, I don't think it's a big deal if we don't have the part specific titles so am fine with leaving as is too!


Fail early in case problems with GObject import

Hi,
this PR fixes an issue, where pyperclip selects gtk-clipboard even if it's not really working, and then fails with an exception later (see below).
This problem occurred, when I used pyperclip together with pystray. It seems they both use conflicting ways to import GObjects. A simple call gtk.Clipboard() can detect that and pyperclip can switch to an alternative clipboard handler.
Thanks!
[...]
  File "/home/holger/coding/normcap/normcap/handlers/clipboard_handler.py", line 23, in handle
    pyperclip.copy(request.transformed)
  File "/home/holger/coding/normcap/.venv/lib/python3.8/site-packages/pyperclip/__init__.py", line 721, in lazy_load_stub_copy
    return copy(text)
  File "/home/holger/coding/normcap/.venv/lib/python3.8/site-packages/pyperclip/__init__.py", line 166, in copy_gtk
    cb = gtk.Clipboard()
  File "/home/holger/coding/normcap/.venv/lib/python3.8/site-packages/gi/__init__.py", line 69, in __getattr__
    raise AttributeError(_static_binding_error)
AttributeError: When using gi.repository you must not import static modules like "gobject". Please change all occurrences of "import gobject" to "from gi.repository import GObject". See: https://bugzilla.gnome.org/show_bug.cgi?id=709183

Related #225


Additionally except TypeError in try_date function

Codecov Report

Merging #67 (1a62d99) into stable (6129956) will not change coverage.
The diff coverage is 0.00%.

@@           Coverage Diff           @@
##           stable      #67   +/-   ##
=======================================
  Coverage   58.78%   58.78%
=======================================
  Files         233      233
  Lines        7551     7551
=======================================
  Hits         4439     4439
  Misses       3112     3112

Impacted Files
Coverage Δ

majora2/util.py
15.32% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6129956...1a62d99. Read the comment docs.


Implement R.collectBy

Follow up to #2706
I wanted to keep the implementation to a bare (perhaps naive) minimum and get some early feedback. I'm not too sure about a few things:

Compatibility with transducers.
Compatibility with any relevant fantasy land specs.
I'm not too familiar with the Hindley-Milner notation.
Is it something that the collective would consider a nice addition to the library? :)

Thank you

@CrossEye I'll finish this!

@customcommander:
I'd forgotten all about it, I'm afraid.

:herb:
This looks great to me.  Anyone else? @ramda/core


Update basic instructions for new users

For people not familiar with python, it's widespread to land on some errors. In this case, if you have a macOS machine that you never worked with python, you probably are still using the python2 version. In this case, you need to switch to python3 as your default python.

Closing since it's polluting the Pull Requests opened. Feel free to re-open and merge.


Batch action on Page loading screen fix

Fixes batch actions that were not triggering the SS loading overlay. Issue affects 4 as well so will need to be merged up.
Fixes https://github.com/silverstripe/silverstripe-cms/issues/441

Also, the unit test failure is pre-existing

Does this issue affect 3.5? if so can we target 3.5 then we can merge all the way up

this file has also likely moved to silverstripe-admin in 4.x so a new PR is likely neeeded

@robbieaverill Have rebased to 3.5 and updated this pr to merge into 3.5 as well. Please have a look. Cheers.

Reopened to re-trigger Travis builds

Thanks @rbarling!


Update TaskbarIcon.Declarations.cs

Added support for dynamic, in-memory icons to be assigned at the xaml level. This allows better support for MVVM design.
For a small wpf project I created, I wanted the ability to generate icons at runtime, while maintaining MVVM design pattern. The only way I could figure out how to assign dynamic in-memory icons was to break MVVM and access the TaskbarIcon object directly in the ViewModel code. So I added this ability which worked well for my needs, and thought I'd share if you are interested in it.
Usage in xaml:
        <tb:TaskbarIcon
            x:Name ="ToolbarIcon"
            **IconInMemory="{Binding TrayIcon}"**
           ... />

In ViewModel (where TrayIcon is set via dynamic icon generation):
public Icon TrayIcon
{
get
{
return _trayIcon;
}
set
{
_trayIcon = value;
NotifyPropertyChanged(nameof(TrayIcon));
}
}

Thank you for sharing this PR, sorry for the delay but I have a lot of my plate.
I see a bit of an issue with this change, an Icon is a IDisposable, so it needs to be disposed otherwise a memory leak will happen. The OnIconInMemoryPropertyChanged will overwrite it, and not allow it to be disposed.
The issue is that sometimes an icon needs to be disposed and sometimes not (as you will need it), and for these scenarios we might want to have a better solution. I know there were some other issues with icons, I need to consolidate all of that to come up with one... bear with me.


fix dataloss bug in batch Storage API sink.

Two issues fixed:

Retry behavior was incorrect. We created a new stream abandoning the old, but then only retried the current append.
Offset was inadvertently reset to 0 the first time we called append, which meant that the second append would always fail with invalid offset. This triggered the data-loss bug in 1.

We fixed the offset bug and no longer create a new stream on failure. In cases where the stream is known to be unusable (e.g. it's been deleted), we raise an exception which forces retry of the entire work item.

LGTM. This will break storage api writes for now, but we will no longer have false positive results

LGTM

fixes #26521 and #26520

Why isn't there a test for this?

Good question - there should be a unit test, and I believe this would be
fairly easy to reproduce in a unit test.
On Wed, May 3, 2023 at 1:20 PM Kenn Knowles @.***>
wrote:

Why isn't there a test for this?
—
Reply to this email directly, view it on GitHub
https://github.com/apache/beam/pull/26503#issuecomment-1533682287, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AFAYJVKPIYSAZ67MOZQOK5DXEK42HANCNFSM6AAAAAAXTISF4A
.
You are receiving this because you modified the open/close state.Message
ID: @.***>

We only have one test case that indirectly tests for multiple stream appends, and it uses streaming mode (while this error occurs in batch): https://github.com/apache/beam/blob/03ecb158f9a24a0e956d79c6d096d353d6b28a7b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOWriteTest.java#L1971-L1976

Yea let's add a batch test that reproduces the issue at the commit before the fix, and confirms it fixed. Keep the bug open until the test is in.

I'll write up the test and tag @reuvenlax to review


When using AlternateTypeRuleConvention ApiModelProperty annotation does not work

When using AlternateTypeRules as with the springfox-grails. ApiModelProperty has no effect.
See https://github.com/springfox/springfox-grails-integration/issues/20 for some more details.
What I would expect is:
When alternate type rule are added as DynamicTypes using AlternateTypeBuilder. The ModelPropertyBuilderPlugin should have enough information available in context, to know which original type this property belongs to so it can work accordingly, and process annotations or do whatever it needs for the given property based on original type or the new alternate type.

@snimavat again thanks for detailed report. It helps improve the library overall 👍


From cdnjs, about library request.

Hi @verlok ~
This is  cdnjs, and we desire to contain vanilla-lazyload.:smiley:
However, I must to make a check for the files that we need to added.
The files I would like to add are all the files under dist folder.
Is that complete for the user? Or other files required?
Thanks for your great work ✨
Piicksarn

Hi there,
thanks for publishing my script.
Yes, all that is inside the dist folder is good.
Now I cleaned it up :)

Closed issue by mistake.
Fell free to close it when you're done :)
Thanks

Thank you!


Fixing release CI failures and adding frozen 1.1 ROM

This run shows it solving all the test failures in the nightly. (The final publish step fails as expected since this is not the main branch).
https://github.com/chipsalliance/caliptra-sw/actions/runs/11976900101


Better handling of differences in modelling between schemas.

Sometimes the entity-relation models of different standards are very different.  For example, some standards for Competency Definitions show relationships directly between Competencies with different properties for different types of relationship, in others there is an intermediary object that defines the nature of the relationship. Another example is that in some standards the name-properties for a person (such as: given name, family name, display name) are directly attached to the Person object whereas in others there is a Name entity that can be repeated to provide all those properties for uses such as preferred name, legal name, former name etc. Finally it might be that what in one standard is a single class is represented by many related (sub-)classes in another.
Often the modelling is very different for graph-based and tree-based approaches.
Mapping between different models can difficult, but we could handle it better by

allowing a representation of a path to show how to get from the entity being modelled to another related entity, or vice versa.
a mapping predicate of transformed can very useful in showing when data from different parts of one model needs to be combined in order to map to another, but we should allow the transformations to be represented in a machine actionable notation (the different notations would be suitable for different modelling frameworks / execution environments).
capturing the relationship between the abstract class and mapped schema classes in the domain being mapped may be worth investigating.

Good analysis, @philbarker. I would add one point. I do not think that transformed should be a mapping predicate option for two reasons: (1) it does not align with the meaning and purpose of mapping predicates which are to express the level of comparability between the two terms being mapped; and (2) if the mapper selects transformed, there is no means of also expressing the level of comparability (e.g., picking transformed and also reworded). Possibly consider a clearly defined transformed option as a boolean toggle available to all individual mappings ... perhaps even triggering some mechanism for formally capturing the nature of the transformation (of which I have no immediate thoughts).

It's addressed in transformation's task.


Cube merge loads NetCDF deferred auxiliary coordinates.

Iris implements deferred loading of auxiliary coordinates payload via the NetCDF loader via a LazyArray.
This is in response to large multi-dimensional auxiliary coordinates causing a large memory footprint. Loading many such cubes can easily exhaust the available RAM for a process.
At the moment cube merge has no concept of such deferred auxiliary coordinates. As such, cube merge loads the payload of the deferred auxiliary coordinates, thus negating all benefit of such deferred loading.
The only workaround at the moment is to use iris.load_raw as this circumvents the merge process. This is a valid step to take in the majority of times, as NetCDF sourced cubes tend to be complete multi-dimensional entities i.e. non-mergeable.
We require to extend cube merge to recognise such deferred auxiliary coordinates and not realise the payload.

Pretty certain we have this capability now. Closing this out.


jets seems to be incompatible with Dynamite

Hello, first of all I want to congratulate you for the wonderful framework you've built. Jets is amazing!
I was playing around with it, running some tests and came with a really strange behaviour.
I was trying to publish a notification to an SNS queue, following the docs and examples and everything went smooth.
Then I decided to add some logic persisting data on a DynamoDB table, using Dynamite and everything started failing.
After digging and running several tests I came to the conclusion that the problem has to do with the following line on dynamite or something related to Aws.config.
I'll try to describe the failing situation:

I'm using DYNAMODB_ENDPOINT=https://dynamodb.eu-west-1.amazonaws.com in my .env files
I have a Dynamite::Item:

class Channel < ApplicationItem
  partition_key "name"
end

I have an Alert.rb similar to the docs one:

class Alert < Jets::Stack
  sns_topic(:delivery_completed, display_name: "cool topic")
end

And then a simple controller:

class ChannelsController < ApplicationController
  include Jets::AwsServices
  class_iam_policy("dynamodb", "sns")
  def index
    channels = Channel.scan
    topic_arn = Alert.lookup(:delivery_completed)
    Jets.logger.debug("About to publish to #{topic_arn}")
    sns.publish(topic_arn: topic_arn,  subject: "Test", message: "Just testing")
    render json: channels.to_json
  end
end

Using this scenario I am able to get the channels, but when I try to lookup for the arn I get the following exception:
{
  "stackTrace": [
    "/tmp/bundled/gems/ruby/2.5.0/gems/aws-sdk-core-3.36.0/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'",
    "/tmp/bundled/gems/ruby/2.5.0/gems/aws-sdk-core-3.36.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'",
    "/tmp/bundled/gems/ruby/2.5.0/gems/aws-sdk-core-3.36.0/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'",
    "/tmp/bundled/gems/ruby/2.5.0/gems/aws-sdk-core-3.36.0/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'",
    "/tmp/bundled/gems/ruby/2.5.0/gems/aws-sdk-core-3.36.0/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'",
    "/tmp/bundled/gems/ruby/2.5.0/gems/aws-sdk-core-3.36.0/lib/seahorse/client/plugins/response_target.rb:23:in `call'",
    "/tmp/bundled/gems/ruby/2.5.0/gems/aws-sdk-core-3.36.0/lib/seahorse/client/request.rb:70:in `send_request'",
    "/tmp/bundled/gems/ruby/2.5.0/gems/aws-sdk-sns-1.7.0/lib/aws-sdk-sns/client.rb:1204:in `publish'",
    "/var/task/app/controllers/channels_controller.rb:9:in `inde [TRUNCATED]"
  ]
}

The catchy thing is that if I lookup the arn first and then scan the dynamo model, the first time the function is invoked, it works as expected, the consequent calls fail.
class ChannelsController < ApplicationController
  include Jets::AwsServices
  class_iam_policy("dynamodb", "sns")
  def index
    topic_arn = Alert.lookup(:delivery_completed)
    Jets.logger.debug("About to publish to #{topic_arn}")
    sns.publish(topic_arn: topic_arn,  subject: "Test", message: "Just testing")
    channels = Channel.scan
    render json: channels.to_json
  end
end

I hope the description is clear enough.
RaY

@graimon Thanks for the nice words.
Interesting. Wasn't able to reproduce the issue.  😕
Here's the code based on your examples: https://github.com/tongueroo/dynamodb-demo
Ran the channels#index multiple times without fail 🧐

Manually subscribe to the SNS topic and got the emails:

Am unable to reproduce right now. Doesn't mean there's no issue, just am unable to reproduce on my setup. Maybe the full CloudWatch logs will reveal a more debugging info what you're seeing?   Here's a post on how to view the CloudWatch logs: Jets Tutorial Debugging Logs Part 3: AWS Lambda Ruby
Let me know. Leaving open for now in case there's more info.
P.S. Impressed that you were able to use Dynomite, would like to improve the docs on it, too busy with work though 😁

@graimon Glad that it worked when not setting DYNAMODB_ENDPOINT.  👍

RE: It may be worth changing the way the endpoint is passed to the Aws::DynamoDB::Client.new on Dynomite itself, what do you think?

Good point! The endpoint setting should be scoped to the DynamoDB client only. Can see how setting it globally like that can cause issues. Updated dynomite and released in v1.0.8. Dynomite Changelog  Updated the vendor/dynomite submodule reference in jets project master branch and this will go out in the next Jets release.
Thanks again for the report. Closing this out now.


feat: add notes.tpl & ingress to tier

Summary

This PR fixes the failed deployment because it doesn't have notes.tpl. It also adds ingress to tier.
Test Plan

Checklist

[ ]  No images hosted from dockerhub
[ ]  Are dashboards present to understand the health of the application.  There must be at least 1 of these

[ ]  all databases should have dashboards
[ ]  ideally also have at least cpu/mem utilization dashboards for webserver tier of the app
[ ]  you can use plural from-grafana to convert a grafana dashboard found via google to our CRD

[ ]  Are scaling runbooks present

[ ]  all databases must have scaling runbooks
[ ]  you can use the charts in pluralsh/module-library to accelerate this

[ ]  do you need to add config overlays?

[ ]  inputing secrets
[ ]  configuring autoscaling

[ ]  If there’s a web-facing component to the app, we need to support OIDC authentication and setting up private networks if no authentication option is viable
[ ]  All major clouds must be supported

[ ]  Azure
[ ]  AWS
[ ]  GCP

Also how does tier persist its data?  Does it need a database, I'd assume it has to be saved somewhere.

Also how does tier persist its data? Does it need a database, I'd assume it has to be saved somewhere.

All data is stored stripe.


ARM64 detection via asm

Added an initial implementation, detecting ARM64 through assembly instructions, thereby getting rid of OS binding, which can be used, for example, to detect ARM on Windows.
What do you think?

@gchatelet, @Mizux this pull request is ready for review.
In generally I tested it on Linux with proc/cpuinfo and new implementation through inline asm and checked behaviour with fully implementation in case only inline asm:
Linux proc/cpuinfo + MIDR

Linux DetectFeaturesBase + MIDR

As you can see detection features the same, except  evtstrm, since HWCAP_EVTSTRM is generic timer is configured to generate events at a frequency of approximately 100KHz, and we cannot get this info through cpuid like everyone else.
All other cpu features which we have in Aarch64Features struct was added. I think this should work for Unix, but not Windows yet. Implementation wasn't added, since msvc compiler on Aarch64 __asm keyword  is not supported, then I tried using armasm64 this is the same armasm but microsoft with some diference microsoft armasm directives
in GNU assembler is used .inst directive, and for armasm DCI, but it doesn't work, it is an unknown directive.

could you provide dev/aarch64 branch, where we can add CI/CD and if there are defects to fix them, improve tests and only then merge it into the master or is it unnecessary?

checked via qemu and it works on freebsd

@toor1245 does it work on aarch64-darwin by any chance? I tried running it, it hits an "illegal instruction".

@toor1245 does it work on aarch64-darwin by any chance? I tried running it, it hits an "illegal instruction".

Hi, thanks for the reply! I think no, for now. Right now I have no opportunity to test, and there is no CI to test yet. An issue has been created to add CI jobs for ios, https://github.com/google/cpu_features/issues/197, as a temporary solution can be done through sysctlbyname, but I would not like this since there will be a dependence on the OS, and lowers the possibility for further improving the information of a given architecture

Btw, if this gets merged, are we going to drop all the OS APIs (e.g., sysctl) for arm64 and switch to this version?

Btw, if this gets merged, are we going to drop all the OS APIs (e.g., sysctl) for arm64 and switch to this version?

for Linux / Adnroid, something that has been working for a long time and does not break the public api, I think definitely not now, but on other OS this api may become truncated, what's not good

Since a new pull request was introduced, I converted this pull request to draft until it was merged https://github.com/google/cpu_features/pull/204 to the master branch

Thx for the work here. I'm keeping an eye on the PR (even though I haven't commented lately). Just to make sure we're on the same page, there should be only one header for aarch64 eventually. I assume you have the two headers in parallel so to check that the results are consistent?

Thx for the work here. I'm keeping an eye on the PR (even though I haven't commented lately). Just to make sure we're on the same page, there should be only one header for aarch64 eventually. I assume you have the two headers in parallel so to check that the results are consistent?

@gchatelet at the moment, this PR has only one include/cpuinfo_aarch64.h header unchanged and added two headers to include/internal as include/internal/cpuid_x86.h, or do we need to move the implementation? Sorry, but I misunderstood the question of the consistency of the two headers.
I think we сan to create small pull-requests, since PR too big and hard to review, and leave this PR as a sample.
I see that we can create at least:

Add new inl file stringize.inl, since often operantion and dependency of define_introspection.inl .
Fix or add override function of ExctractBitRange uint64_t, as there is an overflow when extracting 64 bit info in aarch64.
Add include/internal/cpuid_aarch64.h, src/define_cpuid_aarch64.inl and impl_aarch64__base_implementation.inl where only MIDR_EL1(architecture, revision, part num, variant) will be implemented.
Provide new inline asm testing logic.
Provide PR for each ID_* separately.

And as I mentioned above, it will be good if we create a dev branch for implementation.
What do you think?

Update
I have completed research on aarch64 support for Windows. Access to system registers is limited from EL0 and we can't get information from user mode, to solve this problem we have only one way, we have to write software driver in kernel mode, where we have more access, and make client communicate with driver.
here is the result of the test driver of load and unload

to read processor info we can use _ReadStatusReg, thus we don't have to think about armasm64 integration
https://reviews.llvm.org/D53115
https://docs.microsoft.com/en-us/cpp/intrinsics/arm64-intrinsics?view=msvc-170
in lib we have to add something like this:
HANDLE hDevice = CreateFile(L"\\\\.\\CpuFeatures", GENERIC_READ,
          0, NULL, OPEN_EXISTING, 0, NULL);

 if (hDevice == INVALID_HANDLE_VALUE)
      // handle hDevice == INVALID_HANDLE_VALUE

 EL1Info info;
 DWORD bytes;
 BOOL ok = ReadFile(hDevice, &info, sizeof(EL1Info), &bytes, NULL);
 if (!ok)
     // handle !ok
 if (bytes != sizeof(EL1Info))
     // handle bytes != sizeof(EL1Info)
 CloseHandle(hDevice);

Linux allows reading information from EL1 for the following system registers:

MIDR_EL1
ID_AA64ISAR0_EL1
ID_AA64PFR0_EL1
ID_AA64PFR1_EL1
ID_AA64ISAR1_EL1
ID_AA64MMFR0_EL1
ID_AA64MMFR2_EL1
ID_AA64ZFR0_EL1
ID_AA64MMFR1_EL1
ID_AA64ISAR2_EL1
https://www.kernel.org/doc/html/latest/arm64/cpu-feature-registers.html

@gchatelet, @Mizux, what do you think?

hi @arkivm, could you please run this code on your M1 to make sure the behavior with EL on macOS is the same as it is on Windows
compiler clang.
https://reviews.llvm.org/D53115
#include <stdint.h>
#include <intrin.h>

#define ARM64_SYSREG(op0, op1, crn, crm, op2) \
        ( ((op0 & 1) << 14) | \
          ((op1 & 7) << 11) | \
          ((crn & 15) << 7) | \
          ((crm & 15) << 3) | \
          ((op2 & 7) << 0) )

#define ARM64_CNTVCT            ARM64_SYSREG(3, 3, 14, 0, 2)

int main() {
   uint64_t cntvct = _ReadStatusReg(ARM64_CNTVCT); // should work
   return 0;
}

#include <stdint.h>
#include <intrin.h>

#define ARM64_SYSREG(op0, op1, crn, crm, op2) \
        ( ((op0 & 1) << 14) | \
          ((op1 & 7) << 11) | \
          ((crn & 15) << 7) | \
          ((crm & 15) << 3) | \
          ((op2 & 7) << 0) )

#define ARM64_MIDR_EL1         ARM64_SYSREG(3, 0, 0, 0, 0)

int main() {
   uint64_t midr = _ReadStatusReg(ARM64_MIDR_EL1); // the probability that it will fail the way it is done in user mode
   return 0;
}

@toor1245
__asm__ volatile ("mrs %0, cntvct_el0" : "=r" (cntvct1)); // works
__asm__ volatile ("mrs %0, tpidrro_el0" : "=r" (tpidr));  // works
__asm__ volatile ("mrs %0, midr_el1" : "=r" (midr)); // fails - illegal instruction

I couldn't get the_ReadStatusReg intrinsic to work. When I add intrin.h, it was complaining
include/intrin.h:12:15: fatal error: 'intrin.h' file not found
#include_next <intrin.h>
              ^~~~~~~~~~

Let me know if you want me to test something else.

@arkivm, thanks for the help!


CRON-like trigger and ML pipeline

Evaluate existing technologies and choose a Python pipeline for triggering searches to FB and then running the resulting data against a model.
Additionally, take inferencing from a model and put the results in a pipeline.

Replaced by #15


[CodeCompletion][NFC] Move collectArgumentExpectation()

Now that, these functions are only used by CodeCompletionTypeContextAnalyzer. Move them into it.

@swift-ci Please smoke test

@swift-ci Please smoke test


Failed to build the newer version of UniDic dictionary

Overview
The UniDic builder (https://github.com/ikawaha/kagome-dict/blob/main/uni/tool/main.go) produced an error when generating a binary file from the latest UniDic dictionary.
What I have tried

Downloaded the unidic-cwj-3.1.1-full.zip, unidic-cwj-2.2.0.zip, and unidic-mecab-2.1.2_src.zip from the following site:
https://clrd.ninjal.ac.jp/unidic/back_number.html#unidic_bccwj
Tried to build a package from each of them like the following

❯ go run ./main.go -dict ../../../../../sandbox/tokenizer_dic_test/unidic-mecab-2.1.2_src -out ./dict/uni.dict_212
❯ go run ./main.go -dict ../../../../../sandbox/tokenizer_dic_test/unidic-cwj-2.2.0 -out ./dict/uni.dict_220
build failed: read error, "../../../../../sandbox/tokenizer_dic_test/unidic-cwj-2.2.0/lex_naist.csv", invalid format csv: ../../../../../sandbox/tokenizer_dic_test/unidic-cwj-2.2.0/lex_naist.csv, want col size 21, got 0, [/ 7062 8395 -1249 補助記号 一般 * * * * * / / * / * 記号 * * * * * * 補助 * * * * * * *<PHONE_NUMBER>6535296 294759]
exit status 1
❯ go run ./main.go -dict ../../../../../sandbox/tokenizer_dic_test/unidic-cwj-3.1.1-full -out ./dict/uni.dict_311
build failed: read error, "../../../../../sandbox/tokenizer_dic_test/unidic-cwj-3.1.1-full/lex_3_1.csv", invalid format csv: ../../../../../sandbox/tokenizer_dic_test/unidic-cwj-3.1.1-full/lex_3_1.csv, want col size 21, got 0, [/ 10643 4176 -1092 補助記号 一般 * * * * * / / * / * 記号 * * * * * * 補助 * * * * * * *<PHONE_NUMBER>6535296 294759]
exit status 1

2.1.2 can be successfully built, but newer versions are failed.
It seems that the format of lex.def has been changed since 2.2.0, which might be a reason for this issue. Please let me know If my operation needs to be corrected or if you need more information.
Reference
 『現代日本語書き言葉均衡コーパス』形態論情報データベースの設計と実装 改訂版

TL;DR
I was also experiencing the same issue, but after tinkering around I was able to build a unidic-cwj-202302_full dictionary
What I have tried
Note: First, I tried to build the non-full version dictionary

Cloned the repo and switched to the main branch
Downloaded unidic-cwj-202302.zip from https://clrd.ninjal.ac.jp/unidic_archive/2302/unidic-cwj-202302.zip
Unzipped unidic-cwj-202302.zip under <REPO_ROOT_DIR>/uni/tool/unidic-cwj-202302/
While in <REPO_ROOT_DIR>/uni/tool, ran

  $ go run ./main.go -dict ./unidic-cwj-202302 -out ./unidic-cwj-202302.dict

The output was:
go: downloading github.com/ikawaha/kagome-dict v1.0.1
go: downloading golang.org/x/text v0.3.8
build failed: open ./unidic-cwj-202302/matrix.def: no such file or directory

The above suggested that I must try building the dictionary from the full archive, as it contains the matrix.def

Thus, I downloaded unidic-cwj-202302_full.zip from https://clrd.ninjal.ac.jp/unidic_archive/2302/unidic-cwj-202302_full.zip
Unzipped unidic-cwj-202302_full.zip under <REPO_ROOT_DIR>/uni/tool/unidic-cwj-202302_full/
While in <REPO_ROOT_DIR>/uni/tool, ran

  $ go run ./main.go -dict ./unidic-cwj-202302_full -out ./unidic-cwj-202302_full.dict

The output was, which matches the output of @hkazuakey :
build failed: read error, "unidic-cwj-202302_full/lex.csv", invalid format csv: unidic-cwj-202302_full/lex.csv,
want col size 21, got 0, [/ 18820 21163 4142 補助記号 一般 * * * * * / / * / * 記号 * * * * * * 補助 * * * * * * *<PHONE_NUMBER>6535296 294759]
exit status 1

I decided to experiment and modify uni/tool/go.mod to (i.e.: upgraded the GoLang version and bumped up kagome-dict to v1.0.9):

module tool

go 1.19

require (
	github.com/ikawaha/kagome-dict v1.0.9
	golang.org/x/text v0.7.0
)

I ran go mod tidy to download/sync dependencies
While in <REPO_ROOT_DIR>/uni/tool, I ran the command again:

  $ go run ./main.go -dict ./unidic-cwj-202302_full -out ./unidic-cwj-202302_full.dict

Now the output was slightly different to before. Please note that the col size which the tool got is no longer zero, but 33:
build failed: read error, "unidic-cwj-202302_full/lex.csv", invalid format csv: unidic-cwj-202302_full/lex.csv,
want col size 21, got 33, [/ 18820 21163 4142 補助記号 一般 * * * * * / / * / * 記号 * * * * * * 補助 * * * * * * *<PHONE_NUMBER>6535296 294759]
exit status 1

I modified uni/tool/main.go#L25 from ColSize:                 21 to ColSize:                 33
While in <REPO_ROOT_DIR>/uni/tool, I re-ran the command:

  $ go run ./main.go -dict ./unidic-cwj-202302_full -out ./unidic-cwj-202302_full.dict

Summary
The file unidic-cwj-202302_full.dict was successfully generated.
The dictionary was also successfully loaded and used in Kagome v2 to tokenize text.

Thank you! @azagniotov

I am happy to issue a PR, if you are interested 🙇
Also, we could make the col size to be configurable via a command line arg.
For example, the "33" will be the default size. But, for older dictionaries
you could pass in the flag and set "21" as a column size (for the sake of
backwards compatibility). Anyways, this is just a random idea.
Please let me know your thoughts.
Thank you
On Fri, Jul 14, 2023 at 2:26 PM ikawaha @.***> wrote:

Thank you! @azagniotov https://github.com/azagniotov
—
Reply to this email directly, view it on GitHub
https://github.com/ikawaha/kagome-dict/issues/35#issuecomment-1635292966,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAHRVTFTVTBOMTLNP2WKKN3XQDJ65ANCNFSM6AAAAAAUNEBNVQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

I am happy to issue a PR, if you are interested 🙇
Also, we could make the col size to be configurable via a command line arg.
For example, the "33" will be the default size. But, for older dictionaries
you could pass in the flag and set "21" as a column size (for the sake of
backwards compatibility). Anyways, this is just a random idea.
Please let me know your thoughts.
Thank you
On Fri, Jul 14, 2023 at 2:26 PM ikawaha @.***> wrote:

Thank you! @azagniotov https://github.com/azagniotov

—

Reply to this email directly, view it on GitHub

https://github.com/ikawaha/kagome-dict/issues/35#issuecomment-1635292966,

or unsubscribe

https://github.com/notifications/unsubscribe-auth/AAHRVTFTVTBOMTLNP2WKKN3XQDJ65ANCNFSM6AAAAAAUNEBNVQ

.

You are receiving this because you were mentioned.Message ID:

@.***>

@ikawaha 🙇🏼♂

@azagniotov
Sorry for the late reply.
I want to handle the new unidic experimentally in the repository as follows :
https://github.com/ikawaha/kagome-dict-uni3
The new unidic is not suitable to embed because the adjacency matrix is too large (matrix.def is 3.5GB!).
In fact, if you try to embed dictionaries, the size of the repository will be too large and you will have problems.
see. https://github.com/ikawaha/kagome-dict-uni3/issues/2
Therefore, it may be necessary to build the dictionary before using it. Please use the dictionary build tool from the above repository.
Thank you for the comment.

@ikawaha thanks for pointing me to the newer build tool, I see it works with the col size 33.
I will attempt to rebuild the new unidic dictionary using the tool from the other repo.
Thank you!


fix spc upload problems

Hi @angelo-wf,
This pr fixes all of the spc transfer issues that I know of, the theory of why this is necessary is simple:
enix game jams data into the spc as fast as it possibly can, and running a single opcode at a time causes the following problem, example...  snes_catchupApu() needs to run 2 cycles, but the spc runs a whole opcode - of which would be 6 cycles long (0xfe).  The next spc-opcode is a mov from the port.
So, now the next write happens from cpu -> spc, and catching up apu will not run the spc at all since we are in debt by 4 cycles.  Now the spc "misses" the new data, and instead has the old data loaded from the port... uhoh :)
There is a new problem with "Tales of Phantasia".  The samples are no longer corrupted-sounding when they play individually, but when the game starts playing the music and the voice samples at the same time (during the fight) - everything gets garbled.  It sounds like there is a possible overrun in the sample buffer. (this is a guess).  I will investigate this soon,
best regards,

dink

a little investigation into the sound issue of "Tales of Phantasia".  I noticed that the cycles per frame count go absolutely berzerk 78 frames into bootup for a few frames, then stays normal until the music+voice starts playing when you start a game.
from bootup:
apu / cpu cycles ran:  17088		357362 ... normal!
apu / cpu cycles ran:  17088		357368
apu / cpu cycles ran:  17088		357376
... cut ...
apu / cpu cycles ran:  17089		357378
apu / cpu cycles ran:  17086		357340
apu / cpu cycles ran:  17094		357366
apu / cpu cycles ran:  19836		414930  frame 78, starts to get abnormal
apu / cpu cycles ran:  25856		540760  .. even more abnormal
apu / cpu cycles ran:  23601		493572
apu / cpu cycles ran:  55338		1157266  "1157266" cycles per frame on the main cpu? yikes!
apu / cpu cycles ran:  23635		494340
apu / cpu cycles ran:  25889		541302
apu / cpu cycles ran:  13807		288880
apu / cpu cycles ran:  17087		357342 ... back to normal...
apu / cpu cycles ran:  17092		357362
apu / cpu cycles ran:  17086		357374
Here's a log from when the game starts:
apu / cpu cycles ran:  17090		357370   .. just the voice is playing here, no problem
apu / cpu cycles ran:  17086		357360
apu / cpu cycles ran:  17088		357366
apu / cpu cycles ran:  17089		357368
apu / cpu cycles ran:  17086		357362
apu / cpu cycles ran:  17089		357376
apu / cpu cycles ran:  17087		357358
apu / cpu cycles ran:  17088		357368
apu / cpu cycles ran:  17088		357370
apu / cpu cycles ran:  17088		357368
apu / cpu cycles ran:  17088		357358
apu / cpu cycles ran:  17254		360828  .. the game-screen appears, the music starts playing - it gets "warbley"/bad sounding.
apu / cpu cycles ran:  17085		357328
apu / cpu cycles ran:  16924		353934 .. notice the cycles oscillate between over-shooting and under-shooting?
apu / cpu cycles ran:  17253		360820
apu / cpu cycles ran:  16923		353916
apu / cpu cycles ran:  17254		360828
apu / cpu cycles ran:  16923		353926
apu / cpu cycles ran:  17149		358636
apu / cpu cycles ran:  17026		356078
apu / cpu cycles ran:  17255		360744
apu / cpu cycles ran:  16922		354006
apu / cpu cycles ran:  17551		367058
apu / cpu cycles ran:  16624		347656
apu / cpu cycles ran:  17555		367048
apu / cpu cycles ran:  16621		347696
apu / cpu cycles ran:  17552		367074
apu / cpu cycles ran:  16623		347640
apu / cpu cycles ran:  17558		367072
apu / cpu cycles ran:  16618		347658
apu / cpu cycles ran:  17554		367094
apu / cpu cycles ran:  16624		347642
apu / cpu cycles ran:  17549		367052
apu / cpu cycles ran:  16625		347676
I'm not too familiar with snes yet, so there is the possibility that this is normal(?)  What are your thoughts?
best regards,

dink

to sum it up, the new sound-problem in "Tales of Phantasia" is caused by dma going past the end of frame.  I have some ideas on how to deal with this, will do some experimentation tonight!
best regards,

dink

@angelo-wf, been doing tests over the past weeks and this PR is very solid.  The issue with "Tales of Phantasia" is due to the dma-frame-boundary issue. (I shouldn't have mentioned it in this PR, but I was unsure at the time...)
On to some really great news - my cx4 processor core is up and running, passes all of the tests, and runs Megaman X2/X3!  After a bit of clean-up I'll make a pr for it.
best regards,

dink

I want to look through this PR soon, I just haven't had much time for it lately. Nice that you got the CX4 working, looking forward to seeing how you implemented it.

Hi,
Just a little update on my cx4 core, so far things are looking good - I've spent the past week or so polishing it up, but there are a few caveats, here's a list...
simple/non-issue type things
~~
1: the core code (cx4.cpp) uses tab=4space, because my editor will not play nicely with tab=2spc, (it's something I'd been using since the mid 90's, go figure)
2: it's single context only
3: probably should playtest both games to the end :)
slightly difficult thing
~~
running the cx4 at its rated 20mhz, and the attract mode gameplay demo desychs (megaman gets killed, instead of megaman kills the boss).  though, running at half-speed (10mhz) causes everything to align perfectly.  I spent a solid week just trying to get to the bottom of this one, but, for now it eludes me.  I'll keep trying, though!
I would really like to do a playtest and try to get the timing 100% before making a PR, but if you would like to give it a try before then, I could e-mail over the files, or something like that :)
best regards,

dink

Hi there, a little update on the CX4-core project:
...just fixing up some bugs found while play-testing, also took a short break to work on other things (hint, see new PR).  So far, things are looking good!
best regards,

dink

Update:
While playtesting X2, found 3 gfx-related bugs:
1: A flicker on "You Got ...." scene (the screen after defeating boss), was caused by a slight timing inconsistency in my cx4 core
2: ? (I forgot the bug), due to unimpl. h/v latch in ppu, very easy to implement/fix.  snes.cpp 0x4201 & ppu.cpp 0x37 (ppu_read)
3: in Bubble Crab's stage, the fish ship's radar beam would flicker (on/off every other frame), finding the cause of this one was quite soul-crushing - but - it did allow me to get proper acquainted with the snes ppu!  Basically, the fix involves cancelling the halfColor shift if the clipmode == 1 or 2 logic hits (in ppu_handlePixel)
best regards,

dink

@angelo-wf going forward, I will only be committing to my fork of your project, I recommend closing my pr's here and syncing with my fork to keep up to date.
https://github.com/dinkc64/LakeSnes
best regards,

dink

A bit of awesome news:
One thing that kinda bothered me is the insane amount of cpu this project uses.  I spent perhaps a solid month attempting to optimize things to no avail, but...  finally, a couple of good ideas lead to super results today!
cut the cpu usage down from 10-11% of a core (13 being fully maxed, equal to 100%) to 7-9%!
fastforward fps from 75-86 frames/per second to 92-104 frames/sec!! (ranges specified, as it fluxuates)
that's with the cx4 running (X2) :)
Hopefully I can get back to the playtesting so that I can finally wrap up and commit the CX4 project.
best regards,

dink

Hi @angelo-wf,
Hope you're well (and not getting tired of my messages.. :)
Using a profiler helped to find some hotspots in LakeSnes, a few stood out, like:
snes_runCycle(), ppu_handlePixel() and ppu_getPixelForBgLayer(), so as an experiment I did a little bit of caching of values and also implemented a latch in snes_runCycle to reduce the amount of logic to a minimum.  So far the results are great, cpu usage is way down now.    I'm curious what you think of this block of code:  I think the timing should be exactly the same with this (moving hPos+=2 to the top)..?
note: nextHoriEvent an int which gets set to 16 on reset, (its always 16 at the end-of-frame)

static void snes_runCycle(Snes* snes) {
  snes->cycles += 2;
  // increment position
  snes->hPos += 2;
  // check for h/v timer irq's
  bool condition = (
    (snes->vIrqEnabled || snes->hIrqEnabled) &&
    (snes->vPos == snes->vTimer || !snes->vIrqEnabled) &&
    (snes->hPos == snes->hTimer * 4 || !snes->hIrqEnabled)
  );
  if(!snes->irqCondition && condition) {
    snes->inIrq = true;
    cpu_setIrq(snes->cpu, true);
  }
  snes->irqCondition = condition;
  // handle positional stuff
  if (snes->hPos == nextHoriEvent) {
    switch (snes->hPos) {
      case 16: {
        nextHoriEvent = 512;
        if(snes->vPos == 0) snes->dma->hdmaInitRequested = true;
      } break;
      case 512: {
        nextHoriEvent = 1104;
        // render the line halfway of the screen for better compatibility
        if(!snes->inVblank && snes->vPos > 0) ppu_runLine(snes->vPos);
      } break;
      case 1104: {
        if(!snes->inVblank) snes->dma->hdmaRunRequested = true;
        if(!snes->palTiming) {
          // line 240 of odd frame with no interlace is 4 cycles shorter
          // if((snes->hPos == 1360 && snes->vPos == 240 && !ppu_evenFrame() && !ppu_frameInterlace()) || snes->hPos == 1364) {
          nextHoriEvent = (snes->vPos == 240 && !ppu_evenFrame() && !ppu_frameInterlace()) ? 1360 : 1364;
        } else {
          // line 311 of odd frame with interlace is 4 cycles longer
          // if((snes->hPos == 1364 && (snes->vPos != 311 || ppu_evenFrame() || !ppu_frameInterlace())) || snes->hPos == 1368)
          nextHoriEvent = (snes->vPos != 311 || ppu_evenFrame() || !ppu_frameInterlace()) ? 1364 : 1368;
        }
      } break;
      case 1360:
      case 1364:
      case 1368: { // this is the end (of the h-line)
        nextHoriEvent = 16;

        snes->hPos = 0;
        snes->vPos++;
        if(!snes->palTiming) {
          // even interlace frame is 263 lines
          if((snes->vPos == 262 && (!ppu_frameInterlace() || !ppu_evenFrame())) || snes->vPos == 263) {
            if (snes->cart->type == 4) cx4_run();
            snes->vPos = 0;
            snes->frames++;
          }
	    } else {
          // even interlace frame is 313 lines
          if((snes->vPos == 312 && (!ppu_frameInterlace() || !ppu_evenFrame())) || snes->vPos == 313) {
            snes->vPos = 0;
            snes->frames++;
          }
        }

        // end of hblank, do most vPos-tests
        bool startingVblank = false;
        if(snes->vPos == 0) {
          // end of vblank
          snes->inVblank = false;
          snes->inNmi = false;
          ppu_handleFrameStart();
        } else if(snes->vPos == 225) {
          // ask the ppu if we start vblank now or at vPos 240 (overscan)
          startingVblank = !ppu_checkOverscan();
        } else if(snes->vPos == 240){
          // if we are not yet in vblank, we had an overscan frame, set startingVblank
          if(!snes->inVblank) startingVblank = true;
        }
        if(startingVblank) {
          // if we are starting vblank
          ppu_handleVblank();
          snes->inVblank = true;
          snes->inNmi = true;
          if(snes->autoJoyRead) {
            // TODO: this starts a little after start of vblank
            snes->autoJoyTimer = 4224;
            snes_doAutoJoypad(snes);
          }
          if(snes->nmiEnabled) {
            cpu_nmi(snes->cpu);
          }
        }
      } break;
    }
  }
  // handle autoJoyRead-timer
  if(snes->autoJoyTimer > 0) snes->autoJoyTimer -= 2;
}

stats with the latest on my fork - https://github.com/dinkc64/LakeSnes
game: Megaman X2 w/cx4 processor
fps values are with fast-forward (no frame limiting)
before optimizations: 76 - 94fps
after optimizations: 106 - 133fps
after optimizations & not drawing skipped frames**: 208 - 276fps
after optimizations + de-contexting ppu.c*:  118 - 153fps
after optimizations + de-contexting ppu.c & not drawing skipped frames**:  217 - 320fps success!
** in ppu_runLine(), return after "if(!forcedBlank) ppu_evaluateSprites(line - 1);"

de-contexting ppu.c involves moving the entire ppu structure into the ppu.c's global space as static variables.

and **, being experimental are not (yet) on git.

best regards,

dink


Improve semantics for listing methods

This PR moves listing requests to an ListXXXInternal entity, which keeps all the low-level pagination request fields, like offset, limit, token, page. Original entity keeps all of the fields, except pagination-related. This allows to mitigate a common confusion, where users supply limit for the iterator and it acting only a result page limit, not the max number of results.
Follow-up Python SDK PR: https://github.com/databricks/databricks-sdk-py/pull/279

Is the idea that the limit field in the request is only used for the page size and not the total number of responses? I thought you would just be able to remove https://github.com/databricks/databricks-sdk-go/blob/main/.codegen/api.go.tmpl#L219-L221, then no global limit is enforced and this API behaves the same as in the other SDKs.
Rather than supporting a global limit, we should instead switch to the lazy iterator approach and allow users to pull as many resources as they need.

@mgyucht  https://github.com/databricks/databricks-sdk-go/blob/main/.codegen/api.go.tmpl#L219-L221 i think we need to change this to max_results "virtual field", that would do a "global limit". Or we can keep the limit name, but redefine the semantics for when they're used in iterators. It's necessary for things like CLI. e.g. get the latest XX jobs or show the latest XX queries.

Once I get around to https://github.com/databricks/databricks-sdk-go/pull/543, users can define their own iterators and filter however they like. IMO I don't think it needs to be built into the Go SDK, and it isn't in any other SDK we publish today.

@mgyucht it's orthogonal to #543 (we should hold on that because of the upcoming language changes - https://github.com/golang/go/issues/61405).
this change is about cross-SDK semantics on removing low-level fields to paginated methods, so it should be merged ;)

Codecov Report
Patch coverage: 8.57% and project coverage change: -0.04% :warning:

Comparison is base (a3c5a3c) 17.97% compared to head (e2f4cee) 17.93%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #569      +/-   ##
==========================================
- Coverage   17.97%   17.93%   -0.04%
==========================================
  Files          85       85
  Lines        9805     9840      +35
==========================================
+ Hits         1762     1765       +3
- Misses       7889     7919      +30
- Partials      154      156       +2

Files Changed
Coverage Δ

openapi/code/method.go
17.74% <ø> (ø)

openapi/code/service.go
40.18% <8.57%> (-6.19%)
:arrow_down:

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Can we prepare the changes in the other SDKs at the same time, since we're making changes to the template logic?


Duplicate elements in split_between_processes

System Info
accelerate==git+https://github.com/huggingface/accelerate.git

Information

[ ] The official example scripts
[ ] My own modified scripts

Tasks

[ ] One of the scripts in the examples/ folder of Accelerate or an officially supported no_trainer script in the examples folder of the transformers repo (such as run_no_trainer_glue.py)
[ ] My own task or dataset (give details below)

Reproduction

Test example test_accelerate.py

from accelerate import PartialState  # Can also be Accelerator or AcceleratorState

state = PartialState()
input_list = list(range(17))

with state.split_between_processes(input_list) as splitted_input_list:
    print(f"{state.device}, {splitted_input_list}")

Run accelerate launch --num_processes 8 test_accelerate.py
Output

cuda:2, [6, 7, 8]
cuda:7, [16]
cuda:1, [3, 4, 5]
cuda:4, [12, 13, 14]
cuda:3, [9, 10, 11]
cuda:6, [16]
cuda:0, [0, 1, 2]
cuda:5, [15, 16]

As we can see, there are three 16 among three different processes. It is caused by https://github.com/huggingface/accelerate/blob/7ac153f4040ce04c3be0c26736577783d14e42b3/src/accelerate/state.py#L440-L442
We should modify above codes as following to get the expected output
num_samples_per_process = length // self.num_processes
num_extras = length % self.num_processes
start_index = self.process_index * num_samples_per_process + min(self.process_index, num_extras)
end_index = start_index + num_samples_per_process + (1 if self.process_index < num_extras else 0)

Expected behavior

Expected output:

cuda:2, [5, 6]
cuda:7, [15, 16]
cuda:1, [3, 4]
cuda:4, [9, 10]
cuda:3, [7, 8]
cuda:6, [13, 14]
cuda:0, [0, 1, 2]
cuda:5, [11, 12]

@muellerzr , Could you check it out?

Thanks for the flag! Would you like to make a PR with your fix? :)

Thanks for the flag! Would you like to make a PR with your fix? :)

Sure!


Tags in mcr.microsoft.com should be sorted semantically not lexicographically

https://mcr.microsoft.com/v2/azure-cli/tags/list
The tags are currently presented in ascending lexicographic order, so for example 2.19.1 sorts before 2.2.0. This makes it more cumbersome to see what the latest tag actually is.
Instead, they should be presented in descending semantic order, with newer tags up top, like the DockerHub UI does.
"latest"
"2.35.0"
"2.34.1"
...
"0.10.0"

@jiasli for awareness