[Core] Support disaggregated prefill with Mooncake Transfer Engine

We really appreciate @KuntaiDu for his remarkable work in supporting the disaggregated prefill feature in vLLM. Since PR https://github.com/vllm-project/vllm/pull/10502 has been merged. After rebase, we switch the mooncake integration from PR https://github.com/vllm-project/vllm/pull/10728 to here.
This PR is related to #10727, as well as a continuation of PR #10502, which uses Mooncake's Transfer Engine for KVCache transfer instead of NCCL.
Mooncake is a KVCache-centric disaggregated architecture for LLM serving. Transfer Engine is the core component of Mooncake, see documentations for its design & API list.
Compared with NCCL, Mooncake Transfer Engine has the following features:

a unified programming interface for data transfers between DRAM-to-DRAM (both local and remote), DRAM-to-GPU VRAM (both local and remote), and DRAM-to-remote NVMe devices
support for TCP, RDMA, and NVMe-of protocols
topology-aware path selection (link to our English doc, transfer_engine.md), aggregating bandwidth from multiple NICs

Like the current implementation of PR #10502, there are two roles: KV provider (e.g. prefill vLLM instance) and KV consumer (e.g. decode vLLM instance)

Provider side implements insert: insert a KV cache into a buffer, so that it can be transferred upon request
Consumer side implements drop_select: select a KV cache based on tokens, transfer the selected KV, and drop this KV out from the buffer

Both roles are run on different machines.
Integration guide: https://github.com/kvcache-ai/mooncake/blob/main/doc/en/vllm-integration-v0.2.md
Benchmark result: https://github.com/kvcache-ai/mooncake/blob/main/doc/en/vllm-benchmark-results-v0.2.md
More benchmark results will be added in the future.
Test files will be added to align with the future test CI pipeline for PR #10502.
CC List.
@KuntaiDu @youkaichao @alogfans @stmatengss @james0zan

Now working on OSDI submission, will review after Dec 10.

This is a great demonstration to adopt mooncake to current disaggregation implementation. Could you share some benchmark data and best practice here? transfer engine's primary feature like more protocols support, topology-aware path selection would be beneficial in larger scale clusters. I am just curious how mooncake perform in 1P1D simple case or isomorphic environments.

This is a great demonstration to adopt mooncake to current disaggregation implementation. Could you share some benchmark data and best practice here? transfer engine's primary feature like more protocols support, topology-aware path selection would be beneficial in larger scale clusters. I am just curious how mooncake perform in 1P1D simple case or isomorphic environments.

Here are some preview mooncake benchmark results on A10 with up to 2 RDMA NICs. I am currently having some trouble benchmarking PyNcclConnector now. For some unknown reasons, it crashes a lot for inter-node disaggregated scenarios. And I am digging into the lookup_buffer and connector to try to identify the root cause. But I haven't found it. So the benchmark results haven't included the PyNcclConnector yet.
 But if you want to test the mooncake transfer engine, you can follow the guidance doc to reproduce the results.
In addition, we are also coordinating resources to integrate some machines with more RDMA NICs and more advanced GPUs. The official benchmark results will be released in due time.

Use latest mooncake code, when I test with tp=1, num_rdma_nic=2, qps=2, input_len=200, output_len=100 in a single machine, which prefill instance num is 1 and decode instance num also is 1.
My mooncake_config.json is shown as below:
{
"prefill_url": "<IP_ADDRESS>:8144",
"decode_url": "<IP_ADDRESS>:8149",
"metadata_server": "<IP_ADDRESS>:2333",
"metadata_backend": "etcd",
"protocol": "rdma",
"device_name": "mlx5_0,mlx5_1"
}
There will occur an error in transfer_engine:
E1213 02:57:10.528410  5811 worker_pool.cpp:274] Worker: Process failed for slice (opcode: 0, source_addr: 0x7efdf3ffd010, length: 404, dest_addr:<PHONE_NUMBER>81264, local_nic: mlx5_1, peer_nic: <IP_ADDRESS>:8149@mlx5_0, dest_rkey: 2105088, retry_cnt: 0): transport retry counter exceeded
E1213 02:57:14.286239  5811 worker_pool.cpp:274] Worker: Process failed for slice (opcode: 0, source_addr: 0x7efdf3ffd010, length: 404, dest_addr:<PHONE_NUMBER>81264, local_nic: mlx5_1, peer_nic: <IP_ADDRESS>:8149@mlx5_0, dest_rkey: 2105088, retry_cnt: 1): transport retry counter exceeded
E1213 02:57:18.044381  5811 worker_pool.cpp:274] Worker: Process failed for slice (opcode: 0, source_addr: 0x7efdf3ffd010, length: 1975, dest_addr:<PHONE_NUMBER>73072, local_nic: mlx5_1, peer_nic: <IP_ADDRESS>:8149@mlx5_0, dest_rkey: 2105088, retry_cnt: 0): transport retry counter exceeded
E1213 02:57:21.802461  5811 worker_pool.cpp:274] Worker: Process failed for slice (opcode: 0, source_addr: 0x7efdf3ffd010, length: 1975, dest_addr:<PHONE_NUMBER>73072, local_nic: mlx5_1, peer_nic: <IP_ADDRESS>:8149@mlx5_0, dest_rkey: 2105088, retry_cnt: 1): transport retry counter exceeded
And with one rdma device(mlx5_0 or mlx5_1) is ok

@junna2016 I think the errors are reported from the underlying transfer engine, please open an issue in the Mooncake repo and we will get someone to help identify the root cause.

@junna2016 I think the errors are reported from the underlying transfer engine, please open an issue in the Mooncake repo and we will get someone to help identify the root cause.

https://github.com/kvcache-ai/Mooncake/issues/35

这将 Mooncake 实现目前的一个很好的例子。您可以分享一些基准数据和最佳实践吗？传输引擎的主要功能（如更多协议支持、拓扑拓扑路径选择）在更大规模的我只是好奇月饼在1P1D简单情况或同构环境中的表现如何。

以下是在最多配备 2 个 RDMA NIC 的 A10 上进行的一些月饼基准测试结果。我现在在基准测试中遇到了麻烦PyNcclConnector。由于某些未知原因，它在节点间分割场景中经常崩溃。我正在研究研究lookup_buffer并connector试图找出根本原因。但我还没有找到它。所以基准测试结果PyNcclConnector还不包括。
但如果你想测试月饼轮转引擎，你可以按照指导文档来替换结果。
另外我们同时协调资源，整合一些搭载更多RDMA中断和更高级GPU的机器，官方的基准测试结果也适时发布。

This is a great demonstration to adopt mooncake to current disaggregation implementation. Could you share some benchmark data and best practice here? transfer engine's primary feature like more protocols support, topology-aware path selection would be beneficial in larger scale clusters. I am just curious how mooncake perform in 1P1D simple case or isomorphic environments.

Here are some preview mooncake benchmark results on A10 with up to 2 RDMA NICs. I am currently having some trouble benchmarking PyNcclConnector now. For some unknown reasons, it crashes a lot for inter-node disaggregated scenarios. And I am digging into the lookup_buffer and connector to try to identify the root cause. But I haven't found it. So the benchmark results haven't included the PyNcclConnector yet.
 But if you want to test the mooncake transfer engine, you can follow the guidance doc to reproduce the results.
In addition, we are also coordinating resources to integrate some machines with more RDMA NICs and more advanced GPUs. The official benchmark results will be released in due time.

Hello, could you please provide the PCIe bus bandwidth of the benchmark machine? This will help us make a horizontal comparison. Currently, the machine I am using is configured with 8*A100-SXM4-80GB GPUs, and the bus bandwidth for each slot is Speed 16GT/s, Width x16.

@liweiqing1997  Hello. Here we use 8x GA102GL [A10] , the PCIe Phylink Speed is 16GT/s (x16).

@liweiqing1997 Hello. Here we use 8x GA102GL [A10] , the PCIe Phylink Speed is 16GT/s (x16).

Hello, may I ask if you're using GPUDirect RDMA? I want to know what the transfer path for KvCache is . is it VRAM->DRAM->DRAM->VRAM? or  VRAM->VRAM?

@liweiqing1997 Hello. Here we use 8x GA102GL [A10], the PCIe Phylink Speed is 16GT/s (x16).

Hello, may I ask if you're using GPUDirect RDMA? I want to know what the transfer path for KvCache is . is it VRAM->DRAM->DRAM->VRAM? or VRAM->VRAM?

@liweiqing1997
Currently, it is "VRAM->DRAM->DRAM->VRAM." Mooncake supports GPUDirect though. We still have some work to do to implement zero-copy, it is on our roadmap.

Hello, I try MooncakeConnector in vllm, choose {"protocol": "tcp"}  and {"kv_buffer_device":"cpu"}. When need to send tensor in prefill node, I will convert to cpu from gpu. But when the input_token of request is greater than 853, there is an error on the decode node:
ERROR 12-26 00:15:20 engine.py:135]   File "/workspace/vllm/vllm/worker/model_runner.py", line 1668, in execute_model ERROR 12-26 00:15:20 engine.py:135]     get_kv_transfer_group().recv_kv_caches_and_hidden_states( ERROR 12-26 00:15:20 engine.py:135]   File "/workspace/vllm/vllm/distributed/kv_transfer/kv_transfer_agent.py", line 74, in recv_kv_caches_and_hidden_states ERROR 12-26 00:15:20 engine.py:135]     return self.connector.recv_kv_caches_and_hidden_states( ERROR 12-26 00:15:20 engine.py:135]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 12-26 00:15:20 engine.py:135]   File "/workspace/vllm/vllm/distributed/kv_transfer/kv_connector/simple_connector.py", line 236, in recv_kv_caches_and_hidden_states ERROR 12-26 00:15:20 engine.py:135]     ret = self.select(current_tokens, ERROR 12-26 00:15:20 engine.py:135]           ^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 12-26 00:15:20 engine.py:135]   File "/workspace/vllm/vllm/distributed/kv_transfer/kv_connector/simple_connector.py", line 137, in select ERROR 12-26 00:15:20 engine.py:135]     return self.consumer_buffer.drop_select(input_tokens, roi) ERROR 12-26 00:15:20 engine.py:135]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 12-26 00:15:20 engine.py:135]   File "/workspace/vllm/vllm/distributed/kv_transfer/kv_lookup_buffer/simple_buffer.py", line 207, in drop_select ERROR 12-26 00:15:20 engine.py:135]     key = self.data_pipe.recv_tensor().to(ori_input_device) ERROR 12-26 00:15:20 engine.py:135]           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 12-26 00:15:20 engine.py:135]   File "/workspace/vllm/vllm/distributed/kv_transfer/kv_pipe/mooncake_pipe.py", line 268, in recv_tensor ERROR 12-26 00:15:20 engine.py:135]     tensor = self.transport_thread.submit(self._recv_impl).result() ERROR 12-26 00:15:20 engine.py:135]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 12-26 00:15:20 engine.py:135]   File "/usr/lib/python3.12/concurrent/futures/_base.py", line 456, in result ERROR 12-26 00:15:20 engine.py:135]     return self.__get_result() ERROR 12-26 00:15:20 engine.py:135]            ^^^^^^^^^^^^^^^^^^^ ERROR 12-26 00:15:20 engine.py:135]   File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result ERROR 12-26 00:15:20 engine.py:135]     raise self._exception ERROR 12-26 00:15:20 engine.py:135]   File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58, in run ERROR 12-26 00:15:20 engine.py:135]     result = self.fn(*self.args, **self.kwargs) ERROR 12-26 00:15:20 engine.py:135]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 12-26 00:15:20 engine.py:135]   File "/workspace/vllm/vllm/distributed/kv_transfer/kv_pipe/mooncake_pipe.py", line 254, in _recv_impl ERROR 12-26 00:15:20 engine.py:135]     return pickle.loads(data) ERROR 12-26 00:15:20 engine.py:135]            ^^^^^^^^^^^^^^^^^^ ERROR 12-26 00:15:20 engine.py:135] _pickle.UnpicklingError: invalid load key, '\xb9'. ERROR 12-26 00:15:20 engine.py:135]
Do you happen to know what the reason might be?
It seems that the error is in the place where the k cache is received. I compare the received src pointer with the data length. and they are the same as the prefill node transmits. I do not know whether it is because the amount of k cache data transmitted by tcp is too large, and my network speed is relatively slow, resulting in the loss of data transmission.

Hello, I try MooncakeConnector in vllm, choose {"protocol": "tcp"} and {"kv_buffer_device":"cpu"}. When prefill node need to send tensor, I move the tensor from the gpu to the cpu. But when the input_token of request is greater than 800, there is an error on the decode node:
ERROR 12-26 00:15:20 engine.py:135]   File "/workspace/vllm/vllm/worker/model_runner.py", line 1668, in execute_model ERROR 12-26 00:15:20 engine.py:135]     get_kv_transfer_group().recv_kv_caches_and_hidden_states( ERROR 12-26 00:15:20 engine.py:135]   File "/workspace/vllm/vllm/distributed/kv_transfer/kv_transfer_agent.py", line 74, in recv_kv_caches_and_hidden_states
ERROR 12-26 00:15:20 engine.py:135]     return self.connector.recv_kv_caches_and_hidden_states(
ERROR 12-26 00:15:20 engine.py:135]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 12-26 00:15:20 engine.py:135]   File "/workspace/vllm/vllm/distributed/kv_transfer/kv_connector/simple_connector.py", line 236, in recv_kv_caches_and_hidden_states
ERROR 12-26 00:15:20 engine.py:135]     ret = self.select(current_tokens, ERROR 12-26 00:15:20 engine.py:135]           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 12-26 00:15:20 engine.py:135]   File "/workspace/vllm/vllm/distributed/kv_transfer/kv_connector/simple_connector.py", line 137, in select
ERROR 12-26 00:15:20 engine.py:135]     return self.consumer_buffer.drop_select(input_tokens, roi)
ERROR 12-26 00:15:20 engine.py:135]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 12-26 00:15:20 engine.py:135]   File "/workspace/vllm/vllm/distributed/kv_transfer/kv_lookup_buffer/simple_buffer.py", line 207, in drop_select
ERROR 12-26 00:15:20 engine.py:135]     key = self.data_pipe.recv_tensor().to(ori_input_device) ERROR 12-26 00:15:20 engine.py:135]           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 12-26 00:15:20 engine.py:135]   File "/workspace/vllm/vllm/distributed/kv_transfer/kv_pipe/mooncake_pipe.py", line 268, in recv_tensor ERROR 12-26 00:15:20 engine.py:135]     tensor = self.transport_thread.submit(self._recv_impl).result() ERROR 12-26 00:15:20 engine.py:135]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 12-26 00:15:20 engine.py:135]   File "/usr/lib/python3.12/concurrent/futures/_base.py", line 456, in result
ERROR 12-26 00:15:20 engine.py:135]     return self.__get_result()
ERROR 12-26 00:15:20 engine.py:135]            ^^^^^^^^^^^^^^^^^^^
ERROR 12-26 00:15:20 engine.py:135]   File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result ERROR 12-26 00:15:20 engine.py:135]     raise self._exception
ERROR 12-26 00:15:20 engine.py:135]   File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58, in run
ERROR 12-26 00:15:20 engine.py:135]     result = self.fn(*self.args, **self.kwargs)
ERROR 12-26 00:15:20 engine.py:135]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 12-26 00:15:20 engine.py:135]   File "/workspace/vllm/vllm/distributed/kv_transfer/kv_pipe/mooncake_pipe.py", line 254, in _recv_impl
ERROR 12-26 00:15:20 engine.py:135]     return pickle.loads(data)
ERROR 12-26 00:15:20 engine.py:135]            ^^^^^^^^^^^^^^^^^^
ERROR 12-26 00:15:20 engine.py:135] _pickle.UnpicklingError: invalid load key, '\xb9'.
ERROR 12-26 00:15:20 engine.py:135]
Do you happen to know what the reason might be?
It seems that the error is in the place where the k cache is received. I compare the received src pointer with the data length. and they are the same as the prefill node transmits. I do not know whether it is because the amount of k cache data transmitted by tcp is too large, and my network speed is relatively slow, resulting in the loss of data transmission.

Hello, I try MooncakeConnector in vllm, choose {"protocol": "tcp"} and {"kv_buffer_device":"cpu"}. When prefill node need to send tensor, I move the tensor from the gpu to the cpu. But when the input_token of request is greater than 800, there is an error on the decode node: ERROR 12-26 00:15:20 engine.py:135] File "/workspace/vllm/vllm/worker/model_runner.py", line 1668, in execute_model ERROR 12-26 00:15:20 engine.py:135] get_kv_transfer_group().recv_kv_caches_and_hidden_states( ERROR 12-26 00:15:20 engine.py:135] File

@liuyumoye You should not set kv_buffer_device to "cpu", currently kv_buffer_device only supports "cuda".

@ShangmingCai Sorry, I don't understand why kv_buffer_device only supports "cuda".
I noticed that both the transmit and receive kv cache call mva.mooncake_vllm_adaptor.allocateManagedBuffer to open up space on the cpu, and then read the serialized kv cache through pickle.dumps into the buffer, so  this transmission chain is "tensor in gpu"-> "cpu buffer" -> "cpu buffer" -> "tensor in gpu". If the initial tensor is on the cpu, there should be no impact on the transfer chain. Am I understanding this correctly？
By the way, I am also curious, if the initial tensor is on the gpu, where is the VRAM->DRAM and DRAM->VRAM in the transfer chain implemented?

@ShangmingCai Sorry, I don't understand why kv_buffer_device only supports "cuda". I noticed that both the transmit and receive kv cache call mva.mooncake_vllm_adaptor.allocateManagedBuffer to open up space on the cpu, and then read the serialized kv cache through pickle.dumps into the buffer, so this transmission chain is "tensor in gpu"-> "cpu buffer" -> "cpu buffer" -> "tensor in GPU".

That is correct. But kv_buffer_device this parameter is used to config the buffer location of vllm/distributed/kv_transfer/kv_lookup_buffer/simple_buffer.py, not for mooncake. And the lookup buffer only supports "cuda" now. So the tensor needs to be sent to the Vram eventually for now.
See the code: https://github.com/vllm-project/vllm/blob/81b979f2a8f7ec91c262dac7dcbf30ed577ebafd/vllm/config.py#L2511C1-L2513C45
    # The device used by kv connector to buffer the KV cache.
    # Currently only support 'cuda'.
    kv_buffer_device: Optional[str] = "cuda"

By the way, I am also curious, if the initial tensor is on the gpu, where is the VRAM->DRAM and DRAM->VRAM in the transfer chain implemented?

You can check the implementation of simple_buffer and simple_connector. Mooncake only helps with the data transfer, the other logic has not been touched yet.

I think this comment is because vllm uses pynccl for transmitting kv cache, but this is not the case for mooncake.

Nope, either pynccl or mooncake, they both need to send data to the kv_buffer in VRAM. That's why we need to set up kv_buffer_size because it is not in the paged kvcache.

There is no transfer from VRAM to DRAM in simple_buffer and simple_connector implementation. So won't Mooncake convert from VRAM to DRAM when it transfers VRAM data?

BTW, tensor objects have device info. If you are not sure where the transfers happen, you can print the device info of tensors, it might help.


Use Michelson interferometer for measurement

Use Michelson interferometer for measurement. Currently using MZI

This should be grouped with adding optical self-heterodyne issue


Drop TType class in favor of cdef-ed enum

Most of this is just s/thriftrw.wire.TType/thriftrw.wire.ttype/. See ttype.pxd and ttype.pyx.

:ship: :it:


Unsupported flag "--platform", multi arch support

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind feature
Description

Steps to reproduce the issue:

docker run -it --name archlinux-cupsd --platform linux/amd64 archlinux

Describe the results you received:
Unsupported flag "--platform"
Describe the results you expected:
Emulate target platform using Qemu. This seems to be an experimental feature on Docker but I would like to use it on Podman.
Additional information you deem important (e.g. issue happens only occasionally):
Output of podman version:
ERRO[0000] cannot find mappings for user picloud: open /etc/subuid: no such file or directory
Version:            1.9.2
RemoteAPI Version:  1
Go Version:         go1.14.2
Git Commit:         de2379a75863bdaf3ce8d414045d0b52c544f49a
Built:              Thu May 14 00:47:07 2020
OS/Arch:            linux/arm64

Output of podman info --debug:
ERRO[0000] cannot find mappings for user picloud: open /etc/subuid: no such file or directory
debug:
  compiler: gc
  gitCommit: de2379a75863bdaf3ce8d414045d0b52c544f49a
  goVersion: go1.14.2
  podmanVersion: 1.9.2
host:
  arch: arm64
  buildahVersion: 1.14.8
  cgroupVersion: v1
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.0.16, commit: a97780984207f29652af90fa3dc4e8e7576548e7'
  cpus: 4
  distribution:
    distribution: archarm
    version: unknown
  eventLogger: file
  hostname: picloud
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1002
      size: 1
    uidmap:
    - container_id: 0
      host_id: 1002
      size: 1
  kernel: 5.0.11-1-ARCH
  memFree:<PHONE_NUMBER>
  memTotal:<PHONE_NUMBER>
  ociRuntime:
   name: runc
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc10
      commit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
      spec: 1.0.1-dev
  os: linux
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.3.0
  swapFree: 0
  swapTotal: 0
  uptime: 3h 20m 45.52s (Approximately 0.12 days)
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  configFile: /home/picloud/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: Unknown
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.0.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/picloud/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /run/user/1002/containers
  volumePath: /home/picloud/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):
picloud@picloud ~> pacman -Q podman
podman 1.9.2-1

Additional environment details (AWS, VirtualBox, physical, etc.):

Not going to be high on our priority list, but would welcome community to add patches.

Seems like a good project for an intrepid intern.

@lsm5 are you interested in tackling the issue?
We can handle it entirely on the client side. Currently, podman run first checks if the image exists and pulls in case not. We shouldn't perform this check if --platform is specified and always pull in this case (image exists checks don't cover platform etc.). --platform is effectively a combination of pull --override-arch ... --override-os.


Document new getViteConfig() option for Vitest

Description (required)
Updates the Vitest testing docs to mention a new config option.
For Astro version: 4.8. See astro PR https://github.com/withastro/astro/pull/10963.
I did not use <Since /> in this case because it doesn’t really work well inline in the way I integrated these new docs. Any thoughts on whether this should be added somehow? There could be a separate heading to fit it under, but that felt a little artificial.

Lunaria Status Overview
🌕 This pull request will trigger status changes.

Learn more
By default, every PR changing files present in the Lunaria configuration's files property will be considered and trigger status changes accordingly.
You can change this by adding one of the keywords present in the ignoreKeywords property in your Lunaria configuration file in the PR's title (ignoring all files) or by including a tracker directive in the merged commit's description.

Tracked Files

Locale
File
Note

en
guides/testing.mdx
Source changed, localizations will be marked as outdated.

Warnings reference

Icon
Description

🔄
The source for this localization has been updated since the creation of this pull request, make sure all changes in the source have been applied.

updating branch and pointing to 4.8.0 branch!


Display correct name for student

@kuberti, is this enabled in production? It looks like the changes you made here got reverted by #474.

yeah, it's not working for esanders on production


Broken checkbox at list

Issue Example
If add checkbox at list and remove space inside '[ ]' it breaks

Here without removing the spaces

Test Test
Test Test
-[ ] Test

Gist Example
In this example, it is breaking even without removing the spaces
https://gist.github.com/marcosrjjunior/aa3cd1239050876193dd

This issue was moved to deckar01/task_list#6


Turbo experiment: see your colleague's actions

Experiment
Turbo experiment: see your colleague's actions.
Value: know the world has changed live, don't land on dead pages by accident
Ideas:
"As a librarian, whenever I am logged in, I can see what my colleagues do live inside folders and with files:"
Folders:

[x] When I am in the folder overview, I can see new folders being created
Copy: Your colleague [Colleague Name] just created the folder [FolderName].
Toaster type:
[x] When I am in the folder overview, I can see folders being renamed
Copy: Your colleague [Colleague Name] just renamed the folder [FolderName] to [NewfolderName].
Toaster type:
[x] When I am in the folder overview, I can see folders being deleted
Copy: Your colleague [Colleague Name] just deleted the folder [Folder Name] and all its files.
Toaster type:

Files:

[x] When I am inside the same folder as a colleague, I can see when files are being added by my colleague
Copy: Your colleague [Colleague Name] just added the file [FileName] to this folder.
Toaster type:
[x] When I am inside the same folder as a colleague, I can see when files are being deleted by my colleague
Copy: Your colleague [Colleague Name] just deleted the file [FileName] to this folder.
Toaster type:

Example

Success toaster

Your colleague [Colleague Name] just [action] ...

        <button class="close" type="button" data-dismiss="toast" aria-label="Close">

        </button>

Alert toaster

Your colleague [Colleague Name] just [action] ...

        <button class="close" type="button" data-dismiss="toast" aria-label="Close">

        </button>

Positioning

Component
Toaster: https://biblio-ugent-prototype-emily-v2.vercel.app/styleguide/b-toasts.html#component-04-b-toast-simple

@nics here's an idea, please yell if this needs clarification / extra info / simplification

What works:

Seeing a message when a folder is deleted

In the folder
In the folder overview

I do fear people are going to miss the message if they are looking away briefly.

I can make an improvement issue that it shows as a static alert + interaction to move to the folder overview.
Or we can make the toaster not disappear (only to be removed with the X).

Bugs:
Folders

not sure if in scope Seeing the folder disappear when your colleague deletes it
not sure if in scope I cannot see the folder name change when my colleague edits it live (not in the overview and not in the folder)
not sure if in scope I can not see folders appear when a colleague adds it

Files

I cannot see files added by my colleague live
I cannot see files deleted by my colleague live

Downloads was not in scope, but documenting here

Seeing the download counter go up (see video)

When person A downloads a file, person B sees the counter go up
When person A downloads a file, person A can not see the counter go up

https://user-images.githubusercontent.com/1572885/234586991-975b47d4-1c31-4c3b-9687-9f0c9933527b.mov

Only download counter and alert if folder deleted are implemented now

Okay, then the only thing that needs another look is the download counter, optionally the alert duration. The rest is out of scope.

@mietcls I'll remove the autohide from that alert


Add pre-commit to github action

I have seen you are using ruff (yay) you can use https://github.com/pre-commit/action to have the same pre-commit version running in the ci instead of using https://github.com/s3alfisc/pyfixest/blob/master/.github/workflows/linting.yml

Would you maybe be up to prepare a PR for this @juanitorduz?

@contributor-bot add @juanitorduz for infra and code


Proposal:  Pluggable templating engines for Helm

Author(s):
Nima Kaviani @nimakaviani (IBM) , Dmitriy Kalinin @cppforlife (Pivotal)
Helm Version: 3.0+
Motivation
We have recently open sourced ytt (https://get-ytt.io) - the YAML templating tool - which we believe improves on current ways to template YAML. Helm currently uses Go's text templating (text/template package) for templating configuration. Given the rising popularity of different templating languages, we would like to propose an optional mechanism for Helm to offload its templating functionality. Such mechanism would allow Helm users to use ytt, jsonnet, and jinja (to name a few) instead of default templating engine without having to abandon other Helm features.
In this proposal we have also accounted for thoughts and conversations made in the earlier discussions on making the HELM templating engine customizable:

https://github.com/helm/helm/issues/5826
https://github.com/helm/helm/issues/5084
https://github.com/helm/helm/issues/4718
https://github.com/k14s/ytt/issues/40

Requirements / Assumptions

each chart can use a different engine (including subcharts)
engine provides a script that allows for running the engine
engine can be independently versioned from helm itself
engine is trusted to keep template content secure
engine should be kept external to the core of HELM

User Experience
* Installing an Engine
You should be able to install an engine exactly as a plugin with the same structural format, with the same top level directory and the plugin.yaml file
helm plugin install https://github.com/k14s/ytt-helm-engine
The engine is however slightly different than a regular plugin in that the plugin.yml file provides the additional field type:
name: "ytt"
version: "0.1.0"
usage: "Integrate ytt templating engine with Helm"
description: |-
  This plugin provides ytt templating engine to Helm.
ignoreFlags: false
useTunnel: false
type: engine
command: "$HELM_PLUGIN_DIR/ytt.sh"

For a templating engine, the type field needs to be strictly set to engine, informing HELM that the plugin should be used as a templating engine.
The rest of the fields are the same as in HELM Plugins.
Having the engine listed as a subcommand under helm allows for the engine to offer additional functionality (e.g. debugging) of the charts templates in the target engine’s templating language. As a result, something like the following should be doable:
helm ytt -f templates/ -f values.yml --debug
* Creating a HELM Chart with a Templating Engine
Users should be able to choose an engine when creating their Helm chart. This can be done by passing in a --engine flag when creating the chart:
helm create [chart-name] --engine [engine-name]
The created Chart.yaml file will have an extra field with a reference to the engine.
apiVersion: v2
appVersion: "1.0"
description: A Helm chart for Kubernetes
name: chart
version: 0.1.0
engine:
  name: [engine-name]
  version: [engine-version]

The create command is expected to provide the necessary scaffolding for the HELM chart in the given templating language.
The engine.version in Chart.yaml should match the version of installed engine from doing the earlier: helm plugin install.
Attempting a dry run of the generated chart should result in valid YAML configuration deployable to kubernetes:
helm install [chart-name] [chart-path] --dry-run
A mismatch between the version of the installed engine and the engine-version listed under Chart.yaml should result in an error when installing / updating the chart.
* Runtime Contract Between Helm and the Templating Engine
v1 input to engine would look like this:
{
  "version": "v1",
  "chart": {
    "templates": [
	{
		"name": "template/something.yml",
		"path": "abs-path/template/something.yml"
	},
        {
		"name": "template/something2.yml",
		"path": "abs-path/template/something2.yml"
	}
    ]
  },
  "values": {
    "Release": {
      ...
    },
    ...
  }
}

version is currently set to v1 and may change with future versions
templates is a list of all template files in a chart (i.e. chart.Templates). Note that file paths are resolved to their absolute paths to avoid any mistakes on engine's part.
values is based on contents of chartutil.Values.

for v1 input, the engine should produce the following output:
{
  "results": [
	{
		"name": "template/something.yml",
		"content": "<output>"
	},
        {
		"name": "template/something2.yml",
		"content": "<output>"
	}
  ],
}

It's important that all of the output is not saved anywhere to disk (kept in memory) as it most likely will contain sensitive values.

Hi @nimakaviani, thanks for writing this up. Do you have a reference implementation of this available somewhere?
I'm a little concerned about the user needing to install a plugin for helm install to work. This seems detrimental to the end-user: If I as a user want to install a chart, I now need to

determine what engine is being used, and at what version
determine how to retrieve that engine
install the engine using helm plugin install
install as usual

This is a big disconnect from the existing use case where the user does not need to understand the working internals of a chart in order to install it. How do you imagine we could resolve that?

very good question @bacongobbler ...
For the purpose of this proposal, we meant to limit the scope and have the user go through the steps you described above in order to download and install an engine. Helm can help with providing information on where to find the plugin and how to download it if it realizes an engine is missing.
If we want to extend it further and make it more seamless to the users, then our idea has been to follow a model similar to terraform providers and how they are installed.
We can have Helm offer a list of blessed engines which it knows how to install. Upon a helm install it would fetch the appropriate version of the engine and make it available to the user. For third party engines it will ask users to do a manual installation at their own risk.
Quoting below what terraform does for its providers:

Provider initialization is one of the actions of terraform init. Running this command will download and initialize any providers that are not already initialized.
Providers downloaded by terraform init are only installed for the current working directory; other working directories can have their own installed provider versions.
Note that terraform init cannot automatically download providers that are not distributed by HashiCorp. See Third-party Plugins below for installation instructions.

This looks to me like it would just lead to fragmentation of which helm charts can be installed by what tools. This is why the Helm 3 proposal called for the TPL engine to be embedded in the chart using Lua. And that feature is delayed until after Helm 3.0 is released.

@nimakaviani and @cppforlife thanks for the proposal.
There are a few things going here.

There is a potential for dependency problems here. Imagine you have chart A depend on chart B depend on chart C and depend on chart D (i.e., A -> B -> C -> D). Now, imagine that each of the charts uses a different engine. Now Helm would need to warn people about all the missing plugins and a user has to install each of those. Does this sound like a good user experience?
In Helm v2 there is the ability to add engines to Helm. Yet, no one used it or added any engines new engines. Note, funny easter egg... the internal type for a list of the engines is called EngineYard. This code was written why folks worked at Deis which was a division of EngineYard. How much call is there for this functionality? When we polled for it in the past it was minimal. We do prioritize application operators and application distributors over other rolls. How much do they want this?
For Helm v3 we are nearing a feature freeze to focus on getting bugs fixed and Helm v3 released. I'm not sure there is enough time to get something like this written in time for that. The way new engines are handled in Helm v3 may have to wait for a feature release due to timing.
Our original thought was that other engines could be embedded in charts or library charts. The idea was to use Lua but that appears to not be ready for 3.0. I wonder if Lua or WASM embedded in a chart or subchart that handles the engine would be a better way to go. What do y'all think. Then the engine doesn't require a plugin to run.

Thanks for the comments @technosophos and @mattfarina
@mattfarina,
On your first point, sure chart dependency can become complicated. that's why in my next comment I suggested the idea of trusted engines for which helm acts automatically, potentially pulling the engines before templating the charts. The idea with the proposal was to minimize user facing changes particularly for the operators and distributors (hence piggy backing on plugins)
On your second point, correct me if I am wrong but I think in v2 EngineYard isn't really pluggable. An engine needs to implement the interface and for a wider adoption it has to be compiled with the rest of the Helm code. Despite that, it appears that some people managed to build their custom engines like in https://github.com/helm/helm/issues/5826 but didnt go as far as sharing it with others. For other operators and users, the ytt community is pretty new but we already see some demands for it. like questions on our slack channel or issues like https://github.com/k14s/ytt/issues/40
As for points 3 and 4, fair enough. We don't intend to rush a half baked feature into Helm but it would be good to allow some way of  even experimentally supporting other templating engines at some point in v3. It doesn't have to be through this proposal but may potentially adhere to these requirements that we have identified:

each chart can use a different engine (including subcharts)
engine can be independently versioned from helm itself
engine is trusted to keep template content secure
engine should be kept external to the core of HELM

@technosophos to your point,

This looks to me like it would just lead to fragmentation of which helm charts can be installed by what tools.

I wonder if this is really a concern if the tool used to install the chart is kept hidden from the users? This can be either through some Lua extension point or some other abstraction that Helm provides. I am assuming you dont mean Lua to implement the entire templating engine but rather have a way to run the templating engine in a sandbox in which case how we run the engine (which seems to be through bash scripts in current implementation of plugins) is orthogonal to the pluggability idea.

@nimakaviani I have some more questions...

I suggested the idea of trusted engines for which helm acts automatically, potentially pulling the engines before templating the charts. The idea with the proposal was to minimize user facing changes particularly for the operators and distributors (hence piggy backing on plugins)

There are two situations I'm aware of people encountering and I'm curious how a system like this would work with them. The situations are:

Environments cut off from the public Internet. The chart, images, and helm are on on the network and work.
Security conscious environments where management doesn't want things pulled down from the Internet that are non-obvious. Some of the changes to Helm v3 enable it to run without privileged access to the cluster (no need for an admin to install something like a CRD or Tiller).

How would these situations work? For example, in charts you can specify an alternative image location.
This also opens up the question, what is trusted and how does Helm verify trust? For example, an engine could become trusted and then loose it if new maintainers do something bad. On npm some modules were handed over to maintainers who then injected malicious code into the packages. They lost trust. What kind of trust system would the Helm project need to run in order to handle this?

On your second point, correct me if I am wrong but I think in v2 EngineYard isn't really pluggable. An engine needs to implement the interface and for a wider adoption it has to be compiled with the rest of the Helm code. Despite that, it appears that some people managed to build their custom engines like in #5826 but didnt go as far as sharing it with others.

You are right about it being compiled in. I wasn't aware of #5826. Thanks for pointing it out.
I appreciate the desire to have pluggable systems here. I've proposed a couple different ways to achieve this in the past. The idea of using something like Lua or WASM was interesting because everything could be packaged as part of a chart or a common library chart. This proposal would work for 3 of your 4 bullets.
The one bullet that concerns me is:

engine is trusted to keep template content secure

What do you mean by this? If an engine is a binary it could make network calls and send content over the network to someone else. How do we make sure it is secure? Do we need to review each engine at each release version to make sure it's secure? How is this security verified and by whom?
If we expect enterprises to trust statements on keeping content secure they should be able to audit what we are doing and agree with it, right?
Lua interested us in this way because a script would need to list the libraries it used (e.g., networking) and the end user would need to opt-in to using it. This could allow network access to be discovered and looked at.

I wonder if this is really a concern if the tool used to install the chart is kept hidden from the users?

Will users want it hidden? What kind of bad things might someone do if it was too hidden? Wouldn't it be better to be automatic and in the logs rather than hidden?

I am assuming you dont mean Lua to implement the entire templating engine but rather have a way to run the templating engine in a sandbox in which case how we run the engine (which seems to be through bash scripts in current implementation of plugins) is orthogonal to the pluggability idea.

You could implement an entire engine in Lua (or WASM) or you could call out to something on the file system. The person doing the implementation is free to take either route.
A complexity to this that shows up with bash scripts is Windows users who are not using windows subsystem for linux. Bash scripts don't work there and A LOT of developers are on Windows. If we want them as a market we need a better way to do plugins and engines.
Even take YTT as an example. It is written in Go. So, if a user is on Mac or Linux how does the system know how to handle the right binary for that environment?

good questions @mattfarina

Environments cut off from the public Internet. The chart, images, and helm are on on the network and work.
Security conscious environments where management doesn't want things pulled down from the Internet that are non-obvious. Some of the changes to Helm v3 enable it to run without privileged access to the cluster (no need for an admin to install something like a CRD or Tiller).

How would these situations work? For example, in charts you can specify an alternative image location.

I think this is similar to the situation where they want to use some stable charts internally. or to use some plugins internally. The probably end up cloning them and storing them in a repo local to their networks. They should be able to do the same for the engine too. It certainly won't be as smooth but it won't be impossible.

This also opens up the question, what is trusted and how does Helm verify trust? For example, an engine could become trusted and then loose it if new maintainers do something bad. On npm some modules were handed over to maintainers who then injected malicious code into the packages. They lost trust. What kind of trust system would the Helm project need to run in order to handle this?

yeah trust is a difficult thing to achieve. But I suppose these questions apply to Helm itself at a higher level as well. People still trust Helm with their secrets and believe it doesn't do evil (they probably achieve it through detailed auditing or just by assuming good virtue). Similarly, if Helm chooses to bless some engines (e.g. even if it is only gotpl) it is believed to have been reviewed and approved by the Helm maintainers. From there on whoever trusts Helm will trust the templating engine too (whichever way the reach this level of trust). Like I said, I believe this is already happening.

The one bullet that concerns me is:

  engine is trusted to keep template content secure

What do you mean by this? If an engine is a binary it could make network calls and send content over the network to someone else. How do we make sure it is secure? Do we need to review each engine at each release version to make sure it's secure? How is this security verified and by whom?

Right. If Helm chooses to trust an engine then it needs to be reviewed and approved. Like any other library or tool that gets pulled in. Organizations independently will have to do their auditing too.

You could implement an entire engine in Lua (or WASM) or you could call out to something on the file system. The person doing the implementation is free to take either route.

I haven't played with Lua enough to know all the quirks of it, but I assume the richness of libraries in other languages and developer's familiarity with a certain language are enough motivations for a subset of engine authors to want to try other languages.

Will users want it hidden? What kind of bad things might someone do if it was too hidden? Wouldn't it be better to be automatic and in the logs rather than hidden?

Right, so by hidden I meant transparent. In the sense that the platform takes care of it for them but obviously it should leave the audit trail behind.

Even take YTT as an example. It is written in Go. So, if a user is on Mac or Linux how does the system know how to handle the right binary for that environment?

Right, so ytt is already released for all platforms. This is not in the initial proposal but I can see in a manual install of the engine, devs would choose the target distro they want to install. As you mentioned, this is already an issue with plugins too. So if Helm chooses to solve it at a higher level, the engine would benefit from the same solution too.

@nimakaviani to continue this...

yeah trust is a difficult thing to achieve. But I suppose these questions apply to Helm itself at a higher level as well. People still trust Helm with their secrets and believe it doesn't do evil (they probably achieve it through detailed auditing or just by assuming good virtue).

As I think about this I'm finding several things come to my mind...

Who does the work of certifying engines and keeping up on this? As in, which people. Open source is about contributions and people need to do this leg work. If it were merged into core, the way a new addition to the EngineYard would be, it would follow along with the way gotpl is
Part of the trust for Helm comes from the parent organization. The CNCF which is really a sub-foundation of the linux foundation. For CNCF project graduation projects now get security audits (Helm's is coming soon). This is an example of things in place to do verification on the old saying, "trust but verify". Trust is earned as well. How does this work for engines that are not part of the CNCF? Enterprises are pretty strong on trust. Look at the models used by the Linux distributions.
assuming good virtue, as you put it, is something we can't do. There are simply too many bad actors. So, who writes the auditing code and maintains that? Where do the people and time for that come from?

If Helm chooses to trust an engine then it needs to be reviewed and approved.

I don't think Helm can trust an engine in general. There are two parts to this.

An engine at a version would need to be trusted. What if maintainer-ship is changed and a new maintainer does something bad? This has recently happened with some high profile dependencies in some ecosystems.
If an engine can be moved between locations, such as to a private network, how do we verify the engine? Would it need to be signed or otherwise checked (hash?)? How does that system work?

I raise these kinds of questions to illustrate the complexity and behind executing on a system like this. Not to shoot it down.
In a chat with @technosophos on this topic he wondered what it would like to wrap engines in containers. Then you would just need a runtime. This isn't even a half baked idea just yet but it has promise. Calling to the engine could use CRI so many could be used. It could default to docker and work with docker for mac, etc. Container images already have signing tools (e.g., Notary and TUF which are part of the CNCF). The container trust model is something organizations already know.
After the release of v3 this is something I would like to take a look at because it might solve some of the problems from previous attempts to make engines pluggable. What do you think?

@mattfarina, very valid points.

I raise these kinds of questions to illustrate the complexity and behind executing on a system like this. Not to shoot it down.

totally understood.
Complexities of dealing with trust was one of the reasons we went down the path of the already existing plugins model. The idea was that Helm leaves it to the users to decide whether or not they want to trust an engine. similar to how plugins are treated currently.

In a chat with @technosophos on this topic he wondered what it would like to wrap engines in containers. Then you would just need a runtime. This isn't even a half baked idea just yet but it has promise. Calling to the engine could use CRI so many could be used. It could default to docker and work with docker for mac, etc. Container images already have signing tools (e.g., Notary and TUF which are part of the CNCF). The container trust model is something organizations already know.

Running templating engines / plugins in a container I would say solves half of the problem, ensuring that the engine or the plugin will not do evil on the host machine where they run. In fact ytt already maintains strong sandbox boundaries between the templating language and the external environment.
On the other hand however, even if an engine runs inside a container it can still push secrets out to a remote server (unless you are thinking of modifying iptable rules or blocking outbound connections?). Even then, the engine can still inject containers into a deployment, mount secrets, and then send them away upon deployment to a cluster. Who is responsible to verify that won't happen? the operators or the Helm community?
Every way we look at it, unless some serious auditing process is in place, it would be hard to assume a fully trusted solution from a technology standpoint.
I suggest we take a step back and see what happens in the absence of Helm assuming responsibility for engines or having to offer a trusted set of engines.
The problems we discussed are the followings:

the user needs to trust the plugin (based on their own discretion)
the user needs to manually install the plugin
In case of some chained dependencies among a number of charts, the user needs to deal with resolving engine constrains manually.

Anything else you can think of?
As for the above problems, the biggest one is probably the last one, in which case I believe it is up to the chart authors to ensure smooth installation of their charts given the subchart dependencies. Maybe less of a concern for Helm itself? Basically Helm just fails if an engine of the right version is not found. This may even encourage users who want to use charts with custom engines to look into what engine they are installing and why.
As for the other two problems, I wonder if users who choose to use engines other than what helm is shipped with by default are willing to live with those constraints?

Hi, @nimakaviani - Thank you for your ideas to improve Helm. We now have a defined process so that if you'd like, you can submit a Helm Improvement Proposal so as to work in the community to make these ideas a reality: https://github.com/helm/community/blob/master/hips/hip-0001.md - meanwhile, I will close this issue. Thanks!


Make a new release of libsv on pypi

A lot has been done since Oct 2021 so make a new pypi release

Closing this issue. Released v0.2.0 on PyPI


Improve UG and DG based on feedback

Removed contributor names from UG and DG as well.
Added callouts to UG.

Codecov Report

Merging #92 into master will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master      #92   +/-   ##
=========================================
  Coverage     44.15%   44.15%
  Complexity      510      510
=========================================
  Files           147      147
  Lines          2702     2702
  Branches        349      349
=========================================
  Hits           1193     1193
  Misses         1429     1429
  Partials         80       80

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3ded61e...63233be. Read the comment docs.


Ch  store cards bar update css

Description
Describe change or new feature here.
Type of change
Please delete options that are not relevant.

[ ] New feature (non-breaking change which adds functionality)

Checklist
Remove any items which are not applicable.

[ ] I have performed a self-review of my own code
[ ] My code follows the style guidelines of this project
[ ] My changes generate no new warnings

Link to Trello card
https://trello.com/c/c0rj6HtR/86-style-cards-category-picker

I need a description before I go review this.


Dynamic sparse feature allocation for flexible incremental training

PR for issue #8232
Combined version of the following PRs: #8771 , #8795 , #8796
Proposed changes:

New sparse features will be added dynamically during incremental training instead of creating
an extra buffer in advance
No need to specify additional_vocabulary_size for CountVectorsFeaturizer or
number_additional_patterns for RegexFeaturizer.
Load and store sparse feature sizes with the model
DenseForSparse layers will be dynamically adjusted according to old and current sparse feature sizes

Status (please check what you already did):

[x] added some tests for the functionality
[x] updated the documentation
[x] updated the changelog (please check changelog for instructions)
[x] reformat files using black (please check Readme for instructions)

/modeltest
include:
  - dataset: ["all"]
    config: ["all"]


Bump version to 0.3.2 and sync Cargo.lock

Hi, I swear I'm not stalking this repo ( :sweat_smile: ), I just happened to update my system, and using the nix flake the package no longer builds because of this. I think something similar happened during one of the 0.2 releases so I just did the same as the person did back then.

Thanks!

Could you also please tag a release when you have the time?

👍 done


Add experimental/incomplete Bootstrap 5 support to existing components

This change pretends that BS5 components require the same markup as BS4. This should be true for many components, but not all. Follow up PRs will be created to address the latter cases, see #1494.
But at least this is better than what we had before, as the macros checked for only 3 or 4 and would leave the markup for BS5 broken basically everywhere. And this allows us to iterate step by step on fixing the remaining issues, with BS5 already enjoying full CI coverage.
@jelhan This was mostly a find-and-replace job, so no line by line review required. But would like to hear if you approve this approach here!

I like the approach. Wondering if we should name it isNotBS3. This would highlight that we could remove when dropping support for Bootstrap 3. And might be easier to read than isBS45. But not having strong opinion about.

Wondering if we should name it isNotBS3

I like, updated it!
CI seems to be green now finally (except for #1498)


feat: Delete rows and columns infrastructure

Reviewers
Functional:
@seve
Readability:
@MDunitz

Changes

add

src/views/WheresMyGene/common/store for useReducer to centralize heat map state, since many values change together in various events
The reducer/actions pattern follows closely to how Redux Toolkit structures the code to reduce boilerplate. I didn't dynamically generate actions like RTK does because the TS overhead involved to type things properly would take a lot more time for me to figure out 😆 And I don't think the app is global/heavy enough to justify adding RTK, so hopefully the current approach is a good balance
Add useDeleteGenesAndCellTypes() hook to encapsulate business logic, which internally uses the reducer and dispatch relevant actions as needed
Add ehcarts click events for row and column labels. Also update its style to highlight labels when selected

remove
modify

Move selectedGenes to use the store state

Definition of Done (from ticket)
QA steps (optional)

Load some genes
You can now select/deselect rows and columns! NOTE: I looked into highlighting rows and columns on selection as shown in Figma, but couldn't find a quick way to do it, so decided to just highlight the labels for now
I haven't added delete key press detection yet 😆
If you delete a gene, it will deselect the labels

Known Issues

Fantastic!! Thanks so much, both 🤩 🙏 !!


CFFL

CFFL

Linter check found the following problems:
The following problems have been found:
ERROR: /tmp/abs_1cdaszzu1i/clone/recipe/meta.yaml:22: missing_wheel: For pypi packages, wheel should be present in the host section
ERROR: /tmp/abs_1cdaszzu1i/clone/recipe/meta.yaml:22: host_section_needs_exact_pinnings: Packages in host must have exact version pinnings, except python build tools.
ERROR: /tmp/abs_1cdaszzu1i/clone/recipe/meta.yaml:36: missing_pip_check: For pypi packages, pip check should be present in the test commands
ERROR: /tmp/abs_1cdaszzu1i/clone/recipe/meta.yaml:57: missing_documentation: The recipe is missing a doc_url or doc_source_url
Errors were found

Linter check found the following problems:
The following problems have been found:
ERROR: /tmp/abs_9dby4vk1t4/clone/recipe/meta.yaml:22: host_section_needs_exact_pinnings: Packages in host must have exact version pinnings, except python build tools.
ERROR: /tmp/abs_9dby4vk1t4/clone/recipe/meta.yaml:22: missing_wheel: For pypi packages, wheel should be present in the host section
ERROR: /tmp/abs_9dby4vk1t4/clone/recipe/meta.yaml:36: missing_pip_check: For pypi packages, pip check should be present in the test commands
ERROR: /tmp/abs_9dby4vk1t4/clone/recipe/meta.yaml:57: missing_documentation: The recipe is missing a doc_url or doc_source_url
Errors were found


Some clarifications on the OmniView functionality

Is this a correct understanding?

Streams of data called chunks are coming from devices and consumed by the Epoch
Chunks are pointers to void
Devices are the exported API/functions of a shared library
Epoch allocates memory for the pointer then hands it back to the device and the device will fill it in turn, e.g., using iota in 256-char ranges - each range analogous to a time domain. Afterwards, the consumer (Epoch) picks the data and forwards it to other parts to be viewed/analyzed.

a) Streams get chopped into chunks by the UaDI. Empty chunks get allocated by the consumer. Empty chunks get handed to the library at claim and via push_chunk. Chunks get handed back from the Library via the callback.
b) chunks are the things themselves. A chunk_ptr is implemented as unsigned char* according to @odinthenerd s advise. See: https://github.com/skunkforce/unified-abstract-dataproducer-template/blob/4b1d17670bc2fd8f5ff7fe798d1facd9396a0d00/src/UaDI_template.h#L77
c) A Device is every thing that produces data, fills a chunk and calls the callback.
d)

epoch allocates memory: yes
hands it to the device: yes
the device fills it: yes
when done filling one chunk, the device calls the callback with the chunk_ptr
the iota simulates an oscilloscope measuring a saw-tooth function
the epoch doesn't pick the data up, its callback get's triggered by the device
the epoch forwards it (or saves it): yes

I hope I make sense. Please ask for more clarification if needed. This explaination shall be added to the product vision.

Please explain the following terms too. The code doesn't seem complicated but at times the terminology makes things a little dark.

What is or what parts of the code (the DLL and the source using it) do stand for "the consumer"? Is it the init(...) and deinit(...) functions?
Many times the term library is used in the comments. Do you refer to the DLL by that, please?
The uadi_status function with 9 parameters may be hard to use and reason about I suppose.

A consumer is what that consumes the DLL. Now things are clearer. Thanks.
I guess if you add a line at the beginning of the comments at the top saying: "By library we mean the shared library (.so/DLL)", and then only use the name library, no confusion remains regarding that.
As for the function name, sorry, yes, I meant claim. I suggest taking its functionality and giving it to several other functions each with up to three parameters. Many number of functions makes more sense than a function with many number of parameters, I suppose.

Well, I understand your concern about claim, but: if you would call those three functions in the same order anyways, it can also be one function. Without all there parameters, the claimed device would be in an illegal state.
The trade-off here is:
I will have to explain, that when a device is being claimed, 3 functions with 3 paramters each have to be called in the exact order.
Or
Call this one function that takes 9 parameters.
First approach is what Qt does, but this doesn't make any sense. If you don't use the one function correctly, you'll get a compiler-error. If you don't use the three functions correctly, you could end up having a program that compiles, but crashes at run-time.
I'd love to here @odinthenerd opinion on that.


fix: Manage horizontal margin when apply in tileStyle props.

@Krunal-K-SimformSolutions With there many changes in the file, I am not able to understand what exactly are we trying to fix?
An issue first followed by a PR would have helped :)

@Karthik-B-06: Here I am added dynamic margin because you have applied static margin 4 but when a user provides margin on 'tileStyle' attribute that causes the issue of active segment indicator margin.
Check line number 35, 36 in
const marginHorizontal = (props.tileStyle.marginHorizontal || 2) * 2;
const translateValue = (width - marginHorizontal) / props?.tabs?.length;

And 84 in
width: (width - marginHorizontal) / props?.tabs?.length

Other changes for eslint and prettier format change.

@Krunal-K-SimformSolutions Can you change the PR base branch to development, that way I can fix those eslint/prettier issues and go ahead and merge this fix.

@Krunal-K-SimformSolutions
Adding a dynamic margin other than the static value breaks the style.
      <SegmentedControl
        tabs={["Label", "Label"]}
        onChange={() => {}}
        paddingVertical={6}
        containerStyle={{
          marginVertical: 20,
          borderRadius: 15,
        }}
        tileStyle={{
          borderRadius: 15,
          marginHorizontal: 16
        }}
        currentIndex={tabIndex}
        onChange={handleTabsChange}
        theme={theme}
      />

This is the result after your changes are in effect.


Please add the ability to append Array values through --set.

Please add the ability to append Array values through --set.
For example, for adding a flag to increase verbosity:
flags:
  - "--zap-devel=true"

Instead of doing --set flags[1]="--zap-log-level=5", I need to append to the list of flags. If someone adds another flag, having to go see the index of the last flag in values.yaml kills the use of set.
If this is already possible, please do tell. And update the docs to show how.

Have a look at my answer here: https://github.com/helm/helm/issues/12675#issuecomment-1878088767

@gjenkins8 Sorry. I didn't understand. I understand concat and merge work in helm charts. Can I also use them with --set flag?


Globally rename "csfr" to "csrf"

I noticed a misspelling in the acronym currently used to denote Cross Site Request Forgery tokens.
Related files:

https://github.com/agentejo/cockpit/search?q=csrf
https://github.com/agentejo/cockpit/search?q=csfr

Related commits:

add CSRF protection to login form
more work on csfr token support

Have a nice Day,
Raruto

Thanks! 👍

Thanks! 👍


Implement Network Switching based on WAN Connectivity

Implement some code so that it switches the network interface based on WAN Connectivity.
Currently it'll switch based on if the interface is up such as if Wi-Fi drops out and an ethernet cable is connected. However if for example the unit has an ethernet connection but the ISP is down, it won't fall back to 4G automatically.
Possibly similar code to
https://github.com/PiSupply/iot-lora-gw-docker/blob/master/iot-lora-switcher.py

transferred from https://github.com/NebraLtd/helium-miner-software/issues/21
relates to https://github.com/NebraLtd/helium-miner-software/issues/56 and #131 and #132

ref https://superuser.com/questions/501406/make-networkmanager-prefer-ethernet-over-wifi-if-both-present


Performance Improvement: deduplicate in-flight requests

Hi just wanted to open up the discussion to an idea that I have. The problem I notice is that with many concurrent users arriving at the same time, a lot of duplicate requests can get issued to the database. For small requests, that's usually not a problem as they have very low latency. However, with large layers at low zoom levels, this can lead to a lot of different postgres processes doing full table scans.
Our current solution (and presumably a common solution) is HTTP caching. But caching opens up a huge amount of potential complexity for cache invalidation for layers that change frequently. I think my proposal could reduce this database burden without actually introducing a tile caching layer. It would do this with a small in-flight db request cache, keyed by a struct containing the Source,xyz, and Query values. This request cache will be active as long as the request is in flight. All requests issued which have requests in flight would receive async channel futures. As soon as the database responds, it would broadcast the result to the channels awaiting it, then clean up its' entry in the cache.
A minimal hello-world implementation of this idea is below:
use std::time::Duration;
use std::borrow::Borrow;
use std::collections::HashMap;
use std::sync::Arc;
use std::thread::sleep;

use actix_web::{App, get, HttpServer, Responder, web};
use dashmap::DashMap;
use futures::join;
use tokio;
use tokio::sync::broadcast;
use tokio::sync::broadcast::{Receiver, Sender};
use tokio::sync::broadcast::error::{RecvError, SendError};

async fn get_tile(id: i32) -> String {
    sleep(Duration::from_secs(id as u64));
    println!("running_now");
    return format!("Hello {id}!");
}
type RequestCache = Arc<DashMap<i32,Receiver<String>>>;

#[get("/hello/{secs}")]
async fn greet(cache: web::Data<RequestCache>, secs: web::Path<i32>) -> impl Responder {
    let key = secs.into_inner();
    if(!cache.contains_key(&key)){
        let (s, mut r) = broadcast::channel(1);
        cache.insert(key,r);
        let ret_val = get_tile(key).await;
        let _ = s.send(ret_val.clone());
        cache.remove(&key);
        return ret_val;
    }
    return cache.get(&key).unwrap().resubscribe().recv().await.unwrap();
}

#[actix_web::main] // or #[tokio::main]
async fn main() -> std::io::Result<()> {
    let cache: RequestCache = Arc::new(DashMap::new());
    HttpServer::new(move || {
        App::new()
            .app_data(web::Data::new(cache.clone()))
            .route("/hello", web::get().to(|| async { "Hello World!" }))
            .service(greet)
    })
    .bind(("<IP_ADDRESS>", 8080))?
    .run()
    .await
}

You can run the above code, and then in 2 (or more) different terminals, call curl http://localhost:8080/hello/10. The first request that you send will take a full 10 seconds to fulfill, but if you send another within that 10 second window, it will fulfill at the same exact time as the first request. Once the in-flight requests are fulfilled, the next request would take the full 10 seconds.
This has the drawback of a little additional complexity for requests, but it could dramatically reduce database load for uncached large tiles. If this is something the maintainers could want, I would be willing to work on a pull request to implement it. But if it is not, I'd rather not spend the time, so I'd much prefer a little discussion about it first.

@saosebastiao I must say that my first reaction was - why not just use a reverse proxy, e.g. my personal favorite Varnish, with its grace mode:

When several clients are requesting the same page Varnish will send one request to the backend and place the others on hold while fetching one copy from the backend. In some products this is called request coalescing and Varnish does this automatically.

That said, if the cost of implementing and maintaining it is relatively low, and performance impact is negligible, I do think we should have this feature available as well.  There are several aspects to consider:

I think this should be an opt-in compiler feature, at least at first.  In the long term, we may make this as a user-configurable option (?)
we should have very clear measurable evaluation of the performance impact, especially for
there is currently a lot of significant changes being done to the code (e.g. #440), but it seems this will be relatively simple to merge.

I agree that my first response would probably be to use a cache. It seems like different caches use different names for this idea...for example, nginx calls this a cache lock, which is what I currently use for some of our sources.
However, there are some use cases where setting up a cache is not what I would want. For example:

If I don't want to deal with the performance or operational overhead of an extra process, container, etc.
If I don't want to deal with complex cache invalidation rules
If I don't want cached values at all, just the most recent data within reason

I agree that it should be an opt in feature. It should also be turned off for all function sources that might be marked volatile, which means we should get that information into our function sources at startup.
Since there are significant pending structural changes to the code, I think I'll hold off for a bit. Specifically, I'm gonna have to implement Hash + Eq for the components that make up a unique request (Source, Xyz, QueryParams), I think I'll wait until the Source implementation is more stable.
And yes, you were right, there is a race condition. I'll make sure to remedy that.

@saosebastiao if still interested, there is now rudimentary LRU cache support in Martin, but a lot more can be done with it.


Fix end-2-end test for the flare

What does this PR do?
The migration to component has changed the filenames for metadata in flares. This also highlighted some missing unit-tests for the inventory providers.
This PR fix both the end-2-end tests and add unit-tests.
Describe how to test/QA your changes
generate a flare and check that the metadata files are present:

metadata/inventory/host.json
metadata/inventory/agent.json
metadata/inventory/checks.json
metadata/host.json

Reviewer's Checklist

[ ] If known, an appropriate milestone has been selected; otherwise the Triage milestone is set.
[ ] Use the major_change label if your change either has a major impact on the code base, is impacting multiple teams or is changing important well-established internals of the Agent. This label will be use during QA to make sure each team pay extra attention to the changed behavior. For any customer facing change use a releasenote.
[ ] A release note has been added or the changelog/no-changelog label has been applied.
[ ] Changed code has automated tests for its functionality.
[ ] Adequate QA/testing plan information is provided if the qa/skip-qa label is not applied.
[ ] At least one team/.. label has been applied, indicating the team(s) that should QA this change.
[ ] If applicable, docs team has been notified or an issue has been opened on the documentation repo.
[ ] If applicable, the need-change/operator and need-change/helm labels have been applied.
[ ] If applicable, the k8s/<min-version> label, indicating the lowest Kubernetes version compatible with this feature.
[ ] If applicable, the config template has been updated.

/merge


No example for Custom ListViewItemPresenter

This paragraph mentions doing 'something' for a custom ListViewItemPresenter:
> For a custom ListViewItemPresenter, use the name of your class as the string within the object element syntax for XAML. You'll probably have to map an xmlns definition with a prefix to refer to your class definition. For more info, see XAML namespaces and namespace mapping.
However, it's not very clear what the intent for the developer to do is.  It's be great if this could be clarified with a more complete example with code and XAML provided to show how to do this and what the benefits of a custom ListViewItemPresenter are.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: e61659b2-ba24-5f14-c375-6c6497b06bc4
Version Independent ID: 40496455-80e3-fdb8-f18c-be3bc3fbf136
Content: ListViewItemPresenter Class (Windows.UI.Xaml.Controls.Primitives) - Windows UWP applications
Content Source: https://cpubwin.visualstudio.com/DefaultCollection/windows-uwp/_git/winrt-api-build?path=%2fwinrt-api-build%2fwindows.ui.xaml.controls.primitives.listviewitempresenter.yml&version=GBlive&_a=contents
Product: uwp
Git Login: jwmsft
Microsoft Alias: jimwalk

On @jwmsft 's radar.


"time" vs "date" field and variable consistency

The django-timescaledb module requires the model fields to be called 'time' for time series data. It would be good to go through and check that we are not flicking between "time" and "date" for field and variable names elsewhere in the code to keep things consistent and reduce the risk of errors.

This has been largely sorted.


Loading sigma.js and plugins using amd

I am looking through the examples and see that all the examples (even the quick start example on the homepage) load the needed libraries through script tags.
I would need to load sigma and the json parser plugin using amd - specifically - require.js:
like so:
define([
    'jquery',
    'underscore',
    'backbone',
    '../lib/sigma/sigma.min.js',
    '../lib/sigma/plugins/sigma.parsers.json.min.js'
], function($, _, Backbone, sigma, sigmaParser ){
...

render: function(){
          sigma.parsers.json('../conf/sigma_data.json', {
            container: this.el,
            settings: {
              defaultNodeColor: '#ec5148'
            }
          });
        },
...
}

but when I try to reference sigma or sigmaParser - I get a null reference.  However, in the debugger I see that the js files were properly fetched.  Is there a naming convention that I am missing here?
Can sigma be loaded via AMD - specifically requirejs?

Actually, it is really cleaner to use the "shim" option of requirejs (if you use require) :
require.config({
    paths: {
        'sigma': '../lib/sigma-1.0.3.require'
    },
    shim: {
        'sigma': {
            exports: 'sigma'
        }
    }
});

I'd be really good for sigmajs to support AMD loading better.
For tips on how how achieve AMD support with minimal changes, have a look at UMD -- specifically their returnExports.js example.

+1


Add node 6 to CI.

Coverage remained the same at 100.0% when pulling 4243f2873ca4e5c1ab59d530c9d0b5e4806ba6d1 on pwnall:ci_node6 into 719437775ebb88c3a33d2864c9bd7524372bd558 on heap:master.


Rendering style

Hello,
The rendering styles are different before the update (2~3 month ago?) and the current version.
After the update, the brain looks more flat.
Is there any settings that I can render the brain in the previous style?
The first image is from the current version
the second is from previous version
Thanks!

Hi,
Sorry just to check, which version of brainrender are you currently using?
And what code did you use for this plot?
The way to change the looks is to use
import brainrender
brainrender.SHADER_STYLE = "XXX"

From the figure it looks like you have cartoon in the top figure  but the default should be plastic.
Hope this helps,
Federico

Hi Federico,
Thanks for your help!
I'm using version: <IP_ADDRESS>
I changed the style to plastic but the rendered brain still looks like the top image I posted.
Here's my code. Could you take a look?
We begin by adding the current path to sys.path to make sure that the imports work correctly
import sys
sys.path.append('../')
import os
import pandas as pd
import numpy as np
from vtkplotter import *
Import variables
import brainrender
brainrender.SHADER_STYLE = 'plastic'
brainrender.ROOT_ALPHA = .1
Import brainrender classes and useful functions
from brainrender.scene import Scene
from brainrender.Utils.data_io import listdir
scene = Scene()
actors = scene.add_brain_regions(['XII'], colors="ivory", alpha= 0.2, use_original_color=False)
scene.add_cells(file_names[1], color='DarkViolet', radius = 10, res = 15); # aqua forestgreen
vp = Plotter(axes=0)
vp.show(scene.get_actors())
Thanks!

Hi,
Okay, that's curious.
A thing to try to make sure everything is working correctly is set brainrender.ROOT_ALPHA = 1, if that doesn't change anything then there's something wrong.
If that's the case, then you might want to replace:
vp = Plotter(axes=0)
vp.show(scene.get_actors())

with
scene.render()

If changing ROOT_ALPHA is working, then you can play around with SHADER_STYLE values. The allowed values are:
[default,metallic,plastic,shiny,glossy,ambient,off]

shiny might be what you're looking for.

shiny, glossy, default give me the 3D-ish brain.
I changed multiple alpha, lighting parameters via "plotter" interactive window.
But somehow the rendered brain always looks more gray compared to the brain I previously plotted.
Do you think where I can change?

sorry I'm not sure I understand, are those parameters changing the way the render looks, or are they not changing anything?
If it's the color that you want to change, you can try playing around with brainrender.ROOT_COLOR and increase it to something like [.8, .8, .8] for instance. It's [.4, .4, .4] by default.

Sorry. I didn't explain well. Those parameters do change the way the render looks. The root alpha/opacity around 0.2 ~ 0.3 with 'plastic' gives me good 3D effect/shading I want. But with this setting, inside the brain is dark gray even if I use brighter brainrender.ROOT_COLOR. I would like to get the look similar to the image in my first comment or your examples like this. In those rendered brains, the color inside the brain is light gray, and the surface still maintains the shading.

mmm, I see. I think you can then leave SHADER_STYLE to its default value and just play around with ROOT_ALPHA and ROOT_COLOR. That's the only two parameters that affect that.
I don't remember exactly what values I've used to generate that image, but these seem to get results similar to it:
brainrender.ROOT_COLOR = [.8, .8, .8]
brainrender.ROOT_ALPHA = .2

I still don't get the same look.

Do you think that's the reason?

But I don't understand why you're using
vp = Plotter(axes=0)
vp.show(scene.get_actors())

instead of
scene.render()

?
Are you using a jupyter notebook?
If so, I'd suggest not doing that, it does make things a bit more difficult with brainrender at this stage unfortunately.
The black background might be due to brainrender.SCREENSHOT_TRANSPARENT_BACKGROUND, you could try setting it to False, by default it's True but that might not work well with the backend used in the notebooks.

Yes, I'm using jupyter. I will try spyder.
When I use "scene.render()"
I get this error. Could you tell me how to avoid this?
Thanks!
AttributeError                            Traceback (most recent call last)
 in
----> 1 scene.render()
/opt/anaconda3/lib/python3.7/site-packages/brainrender/scene.py in render(self, interactive, video, camera, zoom, **kwargs)
1095                                         camera = check_camera_param(camera)
1096
-> 1097                                 set_camera(self, camera)
1098
1099                         if interactive:
/opt/anaconda3/lib/python3.7/site-packages/brainrender/Utils/camera.py in set_camera(scene, camera)
114
115         # set params
--> 116         set_camera_params(scene.plotter.camera, camera)
117     else:
118         scene.plotter.camera = camera
/opt/anaconda3/lib/python3.7/site-packages/brainrender/Utils/camera.py in set_camera_params(camera, params)
91 def set_camera_params(camera, params):
92     # Apply camera parameters
---> 93     camera.SetPosition(params['position'])
94     camera.SetFocalPoint(params['focal'])
95     camera.SetViewUp(params['viewup'])
AttributeError: 'NoneType' object has no attribute 'SetPosition'

Hi,
No sorry with the version of brainrender you're using scene.render() won't work in a notebook, it will be fixed in a future release.
I would suggest trying to not use notebooks if you can, but if you really want to then it might be a better idea to use another backend.
You can try adding these two lines before the scene creation:
from vedo import embedWindow
embedWindow(False)

see here for more details.

I will close for now, but let me know if there's something you're not happy with.

Hi Federico,
Sorry to get back to you late.
Now I get the good rendering in Jupyter with
from vedo import embedWindow
embedWindow(False)
Thanks for your help as always!

excellent! thanks, and enjoy!


ci(dspy): Add tests to CI

Create a new action to run the tests that @thomasahle implemented. Fix #456
Fix the pre-commit checks workflow
Adjusted the pyproject.toml to have pytest as a dependency.

Note: databricks integration does not pass tests. Commented out as that is the only integration to do so. To be added back at a later date

The databricks integration is very new so it doesn’t have users yet (merged today). Is unmerging it temporarily useful for this

Yes unmerging would unblock this

Also how do you feel about the pre-commit checks? A lot of the code isn't up to style, and it seems very strict. But this could be the start of actually requiring people's code to meet the style guide

Imo let’s not worry about style checks just yet

And 100% feel free to temporarily remove the databricks integration if you like. We’ll add it back soon

I’d like to merge this when you think it’s ready! No rush though

Oh looks like import pydantic is failing here. Btw so far I've been relying on setup.py so it's very possible the *.toml is not completely correct, someone created it a while back and it may have some mistakes

Yeah weird i already manually added pydantic to the .toml file

Ready to merge whenever @okhat
We do have 3 different build systems currently and should probably get rid of 2 of them, but that can be further discussion

Wait don't merge setup.py is broken now

Ok we good now, setup.py build works

@isaacbmiller , this is great.
I did not go through databrick file since it is commented out, but others looks good to me.

@okhat I am locking pydantic dependency to 2.5.0 because there is a test that specifically relies on the version of pydantic. I would rather merge this than rewrite that test, so I will rewrite the test later

@okhat Can I merge?

Hi @isaacbmiller , I just patched the databricks.py file in this latest PR to move the OpenAI dependency within a local function. feel free to check it out if that passes the tests now so we can merge that alongside this PR!

Thanks @isaacbmiller ! ready to merge on @okhat 's go :)

Ok running some checks!

Unclear why the first test is failing... I tinkered with poetry but it didn't help


EcdsaVerify throws UnsupportedAlgorithm exception on .Net Framework

Based on the FIDO2 example placed in this repository I created a client which creates and verifies a credential with YubiKey.
Under .NET Core 6.0 it works well. Under .NET Framework 4.7.2 and 4.8 the Yubico.YubiKey.Cryptography.EcdsaVerify class throws UnsupportedAlgorithm exception from the CheckECDsa method.
It happens after creating a credential and trying to verify the attestation and verifying the assertion during authentication as well.
Making it impossible to properly verify the results.
The problem occurs not just with a Fido2Session but with an U2fSession as well.
(On .NET Framework the U2fSession related code worked with the 1.4.1 YubiKey SDK version.)
According to my investigation, when the CheckECDsa is called the ECDsa object has a proper OID (the ECDsa.Curve.Oid.Value is properly filled with the 1.2.840.10<IP_ADDRESS> value).
But when the CheckECDsa method calls the toCheck.ExportParameters(false); method, the result
(ECParameters) eccParams.Curve.Oid.Value property is 'null'. The eccParams.Curve.Oid will still exist and it still carries the name of the algoritm. It seems only the Value property is lost.
(Yubico.YubiKey.Cryptography.EcdsaVerify, line: 404)
This is why the method throws UnsupportedAlgorithm exception.

Hi @GregoryMajor - Version 1.6.1 has been released and contains a fix for this issue. Please let us know if you still encounter it after upgrading, or if you run into any similar sort of errors.
Closing this issue, but feel free to re-open if issue persists.


Adding version to an existing file record from the client

Is there a way to upload a new version to an existing file record? I only see information to insert which then allows the server to add multiple versions, but I would like to add a version to the existing item from the client
For instance if I had a user that uploaded an image, but he wanted to upload the same image but in a different language, I would like it to be all contained in a single collection item and then deciding on the server which one to show, rather than adding a whole new collection item.

Hello @schlaegerz,
At this moment library has no such functionality.
Current implementation targeted different usage. The best example would be — upload an image and generate its multiple versions of different sizes on a Server.
Feel free to close it in case if the issue is solved on your end.

Closed due to silence at issue owner end.
Feel free to reopen it in case if the issue still persists on your end.


Remove the middleware addon component from all language servers

Fixes #23897

If I'm remembering correctly this was necessary when Pylance did it's own concatenation of notebook cells (it was necessary to skip intellisense for notebook cells so we wouldn't get dupes?)
I believe it's not used/necessary anymore too.


Cron:Entries() may return nil when called concurrently

This is a theoretical issue I have not yet encountered in practice (but think could actually happen) that I came across while thinking about the source code, in particular these bits:
// Entries returns a snapshot of the cron entries.
func (c *Cron) Entries() []*Entry {
	if c.running {
		c.snapshot <- nil // A
		x := <-c.snapshot // B
		return x
	}
	return c.entrySnapshot()
}

		case <-c.snapshot: // C
			c.snapshot <- c.entrySnapshot() // D

Let's say Goroutine 1 is executing the run() loop containing C and D. Now Goroutine 2 executes Entries() until A, where it blocks. Goroutine 1 executes C, but before it gets a chance to execute D the scheduler decides to preempt it in favour of Goroutine 3, which executes Entries() until A, where it puts nil into the snapshot channel, leading Goroutine 2 to receive nil in B!
Since entrySnapshot() is a non-trivial function I feel like there's a good chance the Goroutine might get preempted during its execution.
The issue lies in using the same channel for communication in both directions. You could instead have a channel of channels over which you send a result channel created in Entries().

go func() {
	<-c
	c <- 3
}()

go func() {
	c <- 0
	t.Log("2:", <-c)
}()

c <- 0
t.Log("1:", <-c)

Is this issue like this code snap?   it could print different value for channel c.
I think the cron object must be used in just one Goroutine.  the Start() function is not Thread safety obviously

I agree with you, good find! I think using the same channel for requests and responses is not a good design either. I believe I can fix this easily, as you say by using a channel of channels.

Fixed on v3


use of IO.html

what is the use of IO.html in canopennode

what is the use of IO.eds in canopennode

please have a look on CANopen specifications for eds files.


Introduces super simple and basic (property) migrations with a passin…

…g test.
We will have to flesh this out during the Configuration API.  Which is up next now that this is in place.  Hopefully in another week or so we'll finally be feature complete with v2. 👍

thank


Add echolevel option

It would be nice to be able to reduce the echolevel and only show errors when opening files. I am happy trusting that the plugin works and get annoyed by the press enter to continue prompt on every file. Right now I have simply edited the line https://github.com/luochen1990/indent-detector.vim/blob/master/plugin/indent_detector.vim#L71, but this is not portable. Adding a let g:??? = 2 to my vimrc would be a much better alternative.

It's sad panda to see this issue open for more than a year.
I had to switch to https://github.com/Khouba/indent-detector.vim which works great and gives available options to silence useless output.
If you are not able to continue project development - update your readme.


Conditional Autoparking

I need to program a condition to trigger autoparking; if a certain event occurs, the system should initiate autoparking, otherwise, it should remain in its default state.
thanks for helping.

You need to wrap up everything in a Manager, then in the def step() function, you can tell if some conditions are satisfied are not and then you can trigger corresponding events.
A good example is: https://github.com/metadriverse/metadrive/blob/233a3a1698be7038ec3dd050ca10b547b4b3324c/metadrive/manager/traffic_manager.py#L31 It triggers traffic vehicles only when ego car is approaching them to maximize interaction.


com.google.analytics.enhanced-ecommerce required fields

Do you know why the fields in this schema doesn't allow null https://github.com/snowplow/iglu-central/blob/master/schemas/com.google.analytics.enhanced-ecommerce/productFieldObject/jsonschema/1-0-0 and this schema does? https://github.com/snowplow/iglu-central/blob/master/schemas/com.google.analytics.measurement-protocol/product/jsonschema/1-0-0
When calling
window.snowplow('addEnhancedEcommerceProductContext',
  'P12345', // The ID
  'Android Warhol T-Shirt', // The name
  'Search Results', // The list
  'Google', // The brand
  'Apparel/T-Shirts', // The category
  'Black', // The variant
  1 // The quantity
);

you may pass null for a parameter and therefore won't pass validation.

hey, I think this is a blunder on the enhanced-ecommerce's side, wdyt @dilyand ?

Yes, I agree. All the product fields are optional in the Measurement Protocol's parameter reference docs, which makes https://github.com/snowplow/iglu-central/blob/master/schemas/com.google.analytics.enhanced-ecommerce/productFieldObject/jsonschema/1-0-0 too restrictive.

Cool thanks, @elliotlings feel free to open a PR for a more relaxed schema :+1:

I looked into this further and it may not be too much of a problem as I first thought. It turns out if you pass undefined instead of null they are stripped out by JSON.stringify and therefore won't be sent up. And as the schema doesn't specify any required properties, missing values won't cause an issue.


no event for message received in channel

The problem
I don't receive messages from channel in the python script (dump.py as reference)
Your environment
Fill out your used versions:

Component
Version

python
2.7.13

pytg
0.4.10

OS
Last Ubuntu

Expected behaviour
I should receive an event with the message posted in the channel.
I m no admin of the channel by the way.
Actual behaviour
No event are received. Then when I run ./telegram-cli it shows me directly all messages received from the channel that are unread.
Thanks for this great library :)

Please specify which is the Last Ubuntu ($ uname -a), as that will change in the future.
Can you check if it works with the --json flag set?
Best to try starting the CLI with the same parameters as pytg does, they get printed in the debug output.

Hello, did you manage to fix your issue? I have exactly the same error and lagging since several hours now...
Thanks

Hi.
I read somewhere that for big groups (like more than 500 people.. or more, don't remember exactly), the library doesn't work correctly.
I finally switch to https://github.com/vysheng/tg which has fixed this bug.
I finally used telethon because it was more adapted to my project. Sorry ^^


Timer API inconsistency with Duration

What problem does this solve or what need does it fill?
Duration.as_secs() returns u64. But Timer.elapsed_secs() and Stopwatch.elapsed_secs() return f32. It's confusing.
What solution would you like?
I would suggest to return u64 from these functions and add additional helpers elapsed_secs_f32 and elapsed_secs_f64 as in Duration. Should be an easy change.

The suggestion to make those methods return u64 is strange. That creates quite a significant footgun, since that will in many cases just return 0, likely leading to incorrect behaviour.
I guess if we want this consistency, we should rename the methods to elapsed_secs_f32, but I firmly oppose adding the methods which return u64 as the default name.

I would expect seconds in unsigned integer. Probably this is why Duration returns u64 in standard Rust library.

From my point of view, Bevy exposes an api that is often better for games, with seconds as a f32 being a common expectation.
To have a more general Rust like api, you can access Timer.elapsed.as_secs()which will give you the u64.

Okay, makes sense.

To have a more general Rust like api, you can access Timer.elapsed.as_secs()which will give you the u64.

Yes, this is how I use it when I need only seconds.


clang compile warning dlib 19.13

Compile warning using Xcode Version 9.4.1 (9F2000); clang Apple LLVM version 9.1.0 (clang-9<IP_ADDRESS>) and -std=gnu++1y:

dlib/cuda/tensor_tools.cpp:121: warning: comparison of integers of different signs: 'long long' and 'size_t' (aka 'unsigned long') [-Wsign-compare]:
DLIB_CASSERT(m.num_samples() == v.size());
dlib/cuda/tensor_tools.cpp:143: warning: comparison of integers of different signs: 'size_t' (aka 'unsigned long') and 'long long' [-Wsign-compare]:
DLIB_CASSERT(v1.size() == m1.num_samples());
dlib/image_transforms/interpolation.h:2165: warning: suggest braces around initialization of subobject [-Wmissing-braces]:
const std::array<dpoint,4> corners = {bounding_box.tl_corner(), bounding_box.tr_corner(), bounding_box.bl_corner(), bounding_box.br_corner()};
dlib/geometry/line.h:225: warning: suggest braces around initialization of subobject [-Wmissing-braces]:
return {v03, v31, v12, v20};
dlib/geometry/line.h:224: warning: suggest braces around initialization of subobject [-Wmissing-braces]:
if (is_convex_quadrilateral({v03, v31, v12, v20}))
dlib/geometry/line.h:223: warning: suggest braces around initialization of subobject [-Wmissing-braces]:
return {v03, v32, v21, v10};
dlib/geometry/line.h:222: warning: suggest braces around initialization of subobject [-Wmissing-braces]:
if (is_convex_quadrilateral({v03, v32, v21, v10}))
dlib/geometry/line.h:220: warning: suggest braces around initialization of subobject [-Wmissing-braces]:
return {v02, v21, v13, v30};
dlib/geometry/line.h:219: warning: suggest braces around initialization of subobject [-Wmissing-braces]:
if (is_convex_quadrilateral({v02, v21, v13, v30}))
dlib/geometry/line.h:218: warning: suggest braces around initialization of subobject [-Wmissing-braces]:
return {v02, v23, v31, v10};
dlib/geometry/line.h:217: warning: suggest braces around initialization of subobject [-Wmissing-braces]:
if (is_convex_quadrilateral({v02, v23, v31, v10}))
dlib/geometry/line.h:215: warning: suggest braces around initialization of subobject [-Wmissing-braces]:
return {v01, v13, v32, v20};
dlib/geometry/line.h:214: warning: suggest braces around initialization of subobject [-Wmissing-braces]:
if (is_convex_quadrilateral({v01, v13, v32, v20}))
dlib/geometry/line.h:213: warning: suggest braces around initialization of subobject [-Wmissing-braces]:
return {v01, v12, v23, v30};
dlib/geometry/line.h:212: warning: suggest braces around initialization of subobject [-Wmissing-braces]:
if (is_convex_quadrilateral({v01, v12, v23, v30}))

Version: 19.13
Where did you get dlib: https://github.com/davisking/dlib/archive/v19.13.tar.gz
Platform: macOS High Sierra 10.13.4 (17E199)
Compiler: Apple LLVM version 9.1.0 (clang-9<IP_ADDRESS>)

Can you submit a PR that adds the appropriate casts?

Yep, I'll do it later.


10193 Lots of cleanup + UX fixes

Issue #: /bcgov/entity#10193
Description of changes:

replaced hex colours with colour variables
cleaned up misc HTML layouts/containers/etc
cleaned up misc code formatting
updated some text + fixed typos
fixed lint issues
implemented smooth scrolling for fragment links
replaced favicon with clear background one
updated Node to v16 (readme + minor changes to lockfile)
fixed some icons (temporary until proper SVGs are available)
updated some layouts (still WIP)
updated v-divider styles
moved hero-banner CSS to index.vue
changed "route to PPR" to "link to PPR" (so it opens in new tab as expected)
deleted some obsolete components
deleted obsolete/unneeded CSS
logged app name and version to console on startup
fixed Contact Info link colour
app version = 1.0.2

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the business-filings-ui license (Apache 2.0).

Hi everyone.

Could someone please assign me this PR? I may need "read" permissions in this repo first. (Who can do this?)

You will find it easier to review the side by side diffs if you click the ⚙ icon at the top of the page and then "hide whitespace".

Please resolve your conversations above if you are satisfied with the changes or explanations.

Please review the updates I made. (You can review each commit separately if that helps.) I think this project is in a good place now for UX to review again and provide new feedback, and for me to make more functional/tactical changes (per ticket 10193 and the UX design document). Please approve your reviews or comment why not.

I'd like to get this into Dev as soon as possible so I can commit further changes without making THIS PR a moving target and harder to review. Who can do this?


Remove the pylint test!!

Hi, @afred-khan will you give more description on this ? so I can help on that!

@hardik-dadhich The lint_python test often fails for every PR causing proplem.

@hardik-dadhich are you taking this issue? If not, please assign this issue to me.

@afred-khan oh it's already assigned :(

@afred-khan oh it's already assigned :(

I'm sorry :((( I thought you were gone