It has RDMA net device in continer,but " Init RDMA failed!Create rdma server failed!" Why?

rpc_server.cc:112] Init RDMA failed!Create rdma server failed!
Describe your problem
A clear and concise description of what your problem is. It might be a bug,
a feature request, or just a problem that need support from the vineyard team.

If is is a bug report, to help us reproducing this bug, please provide information below:

Your Operation System version (uname -a):
The version of vineyard you use (vineyard.__version__):
Versions of crucial packages, such as gcc, numpy, pandas, etc.:
Full stack of the error (if there are a crash):
Minimized code to reproduce the error:

If it is a feature request, please provides a clear and concise description of what you want to happen:
What is the problem:
The behaviour that you expect to work:
Additional context
Add any other context about the problem here.

Hi @hsh258, could you please use something like ib_write_bw or lib-fabric to check whether the rdma dev can work.

Hi @hsh258, could you please use something like ib_write_bw or lib-fabric to check whether the rdma dev can work.

Hi,there are some details:
scene：in container
fi_getinfo: return -FI_ENODATA
find / -name 'librdmacm*' 2>/dev/null
/var/lib/dpkg/info/librdmacm1:amd64.shlibs
/var/lib/dpkg/info/librdmacm1:amd64.triggers
/var/lib/dpkg/info/librdmacm1:amd64.symbols
/var/lib/dpkg/info/librdmacm1:amd64.md5sums
/var/lib/dpkg/info/librdmacm1:amd64.list
/var/cache/apt/archives/librdmacm1_28.0-1ubuntu1_amd64.deb
/usr/lib/x86_64-linux-gnu/librdmacm.so
/usr/lib/x86_64-linux-gnu/librdmacm.so.<IP_ADDRESS>
/usr/lib/x86_64-linux-gnu/librdmacm.so.1
/usr/share/doc/librdmacm1
find / -name 'libibverbs*' 2>/dev/null
/etc/libibverbs.d
/var/lib/dpkg/info/libibverbs1:amd64.md5sums
/var/lib/dpkg/info/libibverbs1:amd64.shlibs
/var/lib/dpkg/info/libibverbs-dev:amd64.list
/var/lib/dpkg/info/libibverbs1:amd64.list
/var/lib/dpkg/info/libibverbs-dev:amd64.md5sums
/var/lib/dpkg/info/libibverbs1:amd64.postinst
/var/lib/dpkg/info/libibverbs1:amd64.symbols
/var/lib/dpkg/info/libibverbs1:amd64.triggers
/var/cache/apt/archives/libibverbs1_28.0-1ubuntu1_amd64.deb
/var/cache/apt/archives/libibverbs-dev_28.0-1ubuntu1_amd64.deb
/usr/lib/x86_64-linux-gnu/pkgconfig/libibverbs.pc
/usr/lib/x86_64-linux-gnu/libibverbs.so
/usr/lib/x86_64-linux-gnu/libibverbs.a
/usr/lib/x86_64-linux-gnu/libibverbs
/usr/lib/x86_64-linux-gnu/libibverbs.so.<IP_ADDRESS>
/usr/lib/x86_64-linux-gnu/libibverbs.so.1
/usr/share/doc/libibverbs1
/usr/share/doc/libibverbs-dev
apt-cache search libfabric
libfabric-bin - Diagnosis programs for the libfabric communication library
libfabric-dev - Development files for libfabric1
libfabric1 - libfabric communication library
dpkg -l | grep libfabric
ii  libfabric1                             1.6.2-3ubuntu0.1                  amd64        libfabric communication library

apt-cache search libfabric

Hi！Could you give me more details? For example, specific error messages like this:

And could you give me your command to run vineyardd? Thanks.
By the way, you can install the fabtest to run fi_getinfo. Please make sure the fabtest version is compatible with libfabric.

apt-cache search libfabric

Hi！Could you give me more details? For example, specific error messages like this:
And could you give me your command to run vineyardd? Thanks.
By the way, you can install the fabtest to run fi_info. Please make sure the fabtest version is compatible with libfabric.

Hi，
Here is command
./vineyardd --rdma_endpoint fd00:80:2200:3205::1207:b02
fi_info -p verbs
fi_getinfo: -61 (No data available)
lrwxrwxrwx. 1 root root       19 Oct 26 09:17 libfabric.so -> libfabric.so.1.24.0
lrwxrwxrwx. 1 root root       19 Oct 26 09:17 libfabric.so.1 -> libfabric.so.1.24.0
-rwxr-xr-x. 1 root root  1187520 Oct 26 09:17 libfabric.so.1.24.0

apt-cache search libfabric

Hi！Could you give me more details? For example, specific error messages like this:
And could you give me your command to run vineyardd? Thanks.
By the way, you can install the fabtest to run fi_info. Please make sure the fabtest version is compatible with libfabric.

Hi， Here is command ./vineyardd --rdma_endpoint fd00:80:2200:3205::1207:b02
fi_info -p verbs fi_getinfo: -61 (No data available) lrwxrwxrwx. 1 root root 19 Oct 26 09:17 libfabric.so -> libfabric.so.1.24.0 lrwxrwxrwx. 1 root root 19 Oct 26 09:17 libfabric.so.1 -> libfabric.so.1.24.0 -rwxr-xr-x. 1 root root 1187520 Oct 26 09:17 libfabric.so.1.24.0

Does "fd00:80:2200:3205::1207:b02" is an ipv6 address? Currently vineyard does not support ipv6 address resolution, please try it again with ipv4 address. Additionally, rdma devices requires root privileges. Are you doing this as root?
By the way, RDMA module of vineyard is based on libfabric, so if the fabric component "fi_info" can't see the information of RDMA device, vineyard can't get it either.

In addition, the param of rdma_endpoint need to point out the port. For example:
./vineyardd --rdma_endpoint=ipv4:port

In addition, the param of "--rdma_endpoint" needs to specify port information for the address. Such as:
./vineyardd --rdma_endpoint=ipv4_addr:port

apt-cache search libfabric

Hi！Could you give me more details? For example, specific error messages like this:
And could you give me your command to run vineyardd? Thanks.
By the way, you can install the fabtest to run fi_info. Please make sure the fabtest version is compatible with libfabric.

Hi， Here is command ./vineyardd --rdma_endpoint fd00:80:2200:3205::1207:b02
fi_info -p verbs fi_getinfo: -61 (No data available) lrwxrwxrwx. 1 root root 19 Oct 26 09:17 libfabric.so -> libfabric.so.1.24.0 lrwxrwxrwx. 1 root root 19 Oct 26 09:17 libfabric.so.1 -> libfabric.so.1.24.0 -rwxr-xr-x. 1 root root 1187520 Oct 26 09:17 libfabric.so.1.24.0

Does "fd00:80:2200:3205::1207:b02" is an ipv6 address? Currently vineyard does not support ipv6 address resolution, please try it again with ipv4 address. Additionally, rdma devices requires root privileges. Are you doing this as root?
By the way, RDMA module of vineyard is based on libfabric, so if the fabric component "fi_info" can't see the information of RDMA device, vineyard can't get it either.

Hi, it is ip6 address。as root login
detail error info:
libfabric:2795:1730110628::core:core:fi_param_get_():372 variable perf_cntr=
libfabric:2795:1730110628::core:core:fi_param_get_():372 variable hook=
libfabric:2795:1730110628::core:core:fi_param_get_():372 variable hmem=
libfabric:2795:1730110628::core:core:ofi_hmem_init():658 Hmem iface FI_HMEM_CUDA not supported
libfabric:2795:1730110628::core:core:ofi_hmem_init():658 Hmem iface FI_HMEM_ROCR not supported
libfabric:2795:1730110628::core:core:ofi_hmem_init():658 Hmem iface FI_HMEM_ZE not supported
libfabric:2795:1730110628::core:core:ofi_hmem_init():658 Hmem iface FI_HMEM_NEURON not supported
libfabric:2795:1730110628::core:core:ofi_hmem_init():658 Hmem iface FI_HMEM_SYNAPSEAI not supported
libfabric:2795:1730110628::core:core:fi_param_get_():372 variable hmem_disable_p2p=
libfabric:2795:1730110628::core:mr:ofi_monitors_init():222 Initializing memory monitor uffd
libfabric:2795:1730110628::core:mr:ofi_monitors_init():222 Initializing memory monitor memhooks
libfabric:2795:1730110628::core:mr:ofi_monitors_init():222 Initializing memory monitor cuda
libfabric:2795:1730110628::core:mr:ofi_monitors_init():222 Initializing memory monitor cuda_ipc
libfabric:2795:1730110628::core:mr:ofi_monitors_init():222 Initializing memory monitor rocr
libfabric:2795:1730110628::core:mr:ofi_monitors_init():222 Initializing memory monitor rocr_ipc
libfabric:2795:1730110628::core:mr:ofi_monitors_init():222 Initializing memory monitor xpmem
libfabric:2795:1730110628::core:mr:ofi_monitors_init():222 Initializing memory monitor ze
libfabric:2795:1730110628::core:mr:ofi_monitors_init():222 Initializing memory monitor import
libfabric:2795:1730110628::core:core:fi_param_get_():372 variable mr_cache_max_size=
libfabric:2795:1730110628::core:core:fi_param_get_():372 variable mr_cache_max_count=
libfabric:2795:1730110628::core:core:fi_param_get_():372 variable mr_cache_monitor=
libfabric:2795:1730110628::core:core:fi_param_get_():372 variable mr_cuda_cache_monitor_enabled=
libfabric:2795:1730110628::core:core:fi_param_get_():372 variable mr_rocr_cache_monitor_enabled=
libfabric:2795:1730110628::core:core:fi_param_get_():372 variable mr_ze_cache_monitor_enabled=
libfabric:2795:1730110628::core:mr:ofi_default_cache_size():83 default cache size=5633248768
libfabric:2795:1730110628::core:mr:ofi_monitors_init():306 Default memory monitor is: memhooks
libfabric:2795:1730110628::core:core:fi_param_get_():372 variable provider=
libfabric:2795:1730110628::core:core:fi_param_get_():372 variable universe_size=
libfabric:2795:1730110628::core:core:fi_param_get_():372 variable av_remove_cleanup=
libfabric:2795:1730110628::core:core:fi_param_get_():372 variable offload_coll_provider=
libfabric:2795:1730110628::core:core:fi_param_get_():372 variable provider_path=
libfabric:2795:1730110628::core:core:ofi_register_provider():518 registering provider: udp (121.0)
libfabric:2795:1730110628::core:core:ofi_register_provider():518 registering provider: sockets (121.0)
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable prov_name=
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable port_high_range=
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable port_low_range=
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable tx_size=
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable rx_size=
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable max_inject=
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable max_saved=
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable max_saved_size=
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable max_rx_size=
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable nodelay=
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable staging_sbuf_size=
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable prefetch_rbuf_size=
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable zerocopy_size=
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable trace_msg=
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable disable_auto_progress=
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable io_uring=
libfabric:2795:1730110628::core:core:ofi_register_provider():518 registering provider: tcp (121.0)
libfabric:2795:1730110628::core:core:ofi_register_provider():518 registering provider: ofi_hook_noop (121.0)
libfabric:2795:1730110628::core:core:ofi_register_provider():518 registering provider: off_coll (121.0)
By the way,what time to support ip6?

I think one of the reasons vineyard was unable to create an RDMA server was because of the ipv6 address. But I don't know why fi_info can't get device information. If fi_info does not get device information, vineyard theoretically cannot get device information even if it is using ipv4.
And ipv6 support is not in our short-term plans at the moment.

I think one of the reasons vineyard was unable to create an RDMA server was because of the ipv6 address. But I don't know why fi_info can't get device information. If fi_info does not get device information, vineyard theoretically cannot get device information even if it is using ipv4.
And ipv6 support is not in our short-term plans at the moment. You can open a new issue about the ipv6 support and we may support ipv6 in the future. Thanks.

whether or not to install other package besides librdmacm.so and libibverbs.so in container scene ？for example ofed，and so on。
In container，use rdma link command，can look up rdma dev：
rdma link
link mlx5_2/1 state DOWN physical_state DISABLED
link mlx5_3/1 state DOWN physical_state DISABLED
link mlx5_4/1 state ACTIVE physical_state LINK_UP
link mlx5_5/1 state ACTIVE physical_state LINK_UP
link mlx5_6/1 state DOWN physical_state DISABLED
link mlx5_7/1 state DOWN physical_state DISABLED
link mlx5_8/1 state ACTIVE physical_state LINK_UP
link mlx5_9/1 state ACTIVE physical_state LINK_UP
link mlx5_10/1 state DOWN physical_state DISABLED
link mlx5_11/1 state DOWN physical_state DISABLED
link mlx5_12/1 state ACTIVE physical_state LINK_UP
link mlx5_13/1 state ACTIVE physical_state LINK_UP
link mlx5_14/1 state DOWN physical_state DISABLED
link mlx5_15/1 state DOWN physical_state DISABLED
link mlx5_16/1 state ACTIVE physical_state LINK_UP
link mlx5_17/1 state ACTIVE physical_state LINK_UP
link mlx5_bond_1/1 state ACTIVE physical_state LINK_UP
link mlx5_0/1 state ACTIVE physical_state LINK_UP
link mlx5_1/1 state ACTIVE physical_state LINK_UP
link mlx5_18/1 state DOWN physical_state DISABLED
link mlx5_19/1 state DOWN physical_state DISABLED
link mlx5_20/1 state DOWN physical_state DISABLED
link mlx5_21/1 state DOWN physical_state DISABLED
link mlx5_22/1 state DOWN physical_state DISABLED
link mlx5_23/1 state DOWN physical_state DISABLED
link mlx5_24/1 state DOWN physical_state DISABLED
link mlx5_25/1 state DOWN physical_state DISABLED
link mlx5_26/1 state DOWN physical_state DISABLED
link mlx5_27/1 state DOWN physical_state DISABLED
link mlx5_28/1 state DOWN physical_state DISABLED
link mlx5_29/1 state DOWN physical_state DISABLED
link mlx5_30/1 state DOWN physical_state DISABLED
link mlx5_31/1 state DOWN physical_state DISABLED
link mlx5_32/1 state DOWN physical_state DISABLED
link mlx5_33/1 state DOWN physical_state DISABLED
link mlx5_34/1 state DOWN physical_state DISABLED
link mlx5_35/1 state DOWN physical_state DISABLED
link mlx5_36/1 state DOWN physical_state DISABLED
link mlx5_37/1 state DOWN physical_state DISABLED
link mlx5_38/1 state DOWN physical_state DISABLED
link mlx5_39/1 state DOWN physical_state DISABLED
link mlx5_40/1 state DOWN physical_state DISABLED
link mlx5_41/1 state DOWN physical_state DISABLED
link mlx5_42/1 state DOWN physical_state DISABLED
link mlx5_43/1 state DOWN physical_state DISABLED
link mlx5_44/1 state DOWN physical_state DISABLED
link mlx5_45/1 state DOWN physical_state DISABLED
link mlx5_46/1 state DOWN physical_state DISABLED
link mlx5_47/1 state DOWN physical_state DISABLED
link mlx5_48/1 state ACTIVE physical_state LINK_UP netdev net_101
link mlx5_49/1 state DOWN physical_state DISABLED

Fabric depends on libibverbs. So libibverbs is necessery.
I suggests that you should install the libfabric and fabtest to use fi_info. Refer to the script below:
For fabric dependencies(CentOS):
yum -y install rdma-core libibverbs libibverbs-devel

Install fabric and fabtests
cd /tmp
wget https://github.com/ofiwg/libfabric/releases/download/v1.22.0/libfabric-1.22.0.tar.bz2
tar xf ./libfabric-1.22.0.tar.bz2
cd libfabric-1.22.0/
./configure --disable-usnic \
            --disable-psm3 \
            --disable-opx \
            --disable-dmabuf_peer_mem \
            --disable-hook_hmem \
            --disable-hook_debug \
            --disable-trace \
            --disable-rxm \
            --disable-psm2 \
            --disable-xpmem \
            --disable-shm \
            --disable-rxd \
            --disable-perf \
            --disable-efa \
            --disable-mrail \
            --enable-verbs \
            --with-cuda=no
make -j
make install

cd /tmp
wget https://github.com/ofiwg/libfabric/releases/download/v1.22.0/fabtests-1.22.0.tar.bz2
tar xf ./fabtests-1.22.0.tar.bz2
cd fabtests-1.22.0
./configure
make -j
make install

Again, vineyard compiles the fabric in the submodule itself, so in theory you only need the ibverbs library to use vineyard RDMA support (The premise is that fabric can also work alone).  You can install the fabtests according to the script above and see if the fabtests works.(Such as fi_rma_bw / fi_info)

./vineyardd --rdma_endpoint=ipv4_addr:port

Hi ./vineyardd --rdma_endpoint=ipv4_addr:port
The command ipv4_addr has or not mask？ format is <IP_ADDRESS>/24:1234 or <IP_ADDRESS>:1234  ?

./vineyardd --rdma_endpoint=ipv4_addr:port

Hi ./vineyardd --rdma_endpoint=ipv4_addr:port The command ipv4_addr has or not mask？ format is <IP_ADDRESS>/24:1234 or <IP_ADDRESS>:1234 ?

Without mask. It should be the ipv4 address of RDMA device.

./vineyardd --rdma_endpoint=ipv4_addr:port

Hi ./vineyardd --rdma_endpoint=ipv4_addr:port The command ipv4_addr has or not mask？ format is <IP_ADDRESS>/24:1234 or <IP_ADDRESS>:1234 ?

The format of "ipv4:port" only affects the parsing of the port. The reason that it cannot use IPv6 is the same, as it parses the content after the first ":" as the port. The currently specified RDMA IPv4 address will not take effect; instead, it will automatically look for the first suitable RDMA device.
Refer tohttps://github.com/v6d-io/v6d/issues/2005

Therefore, it is a priority to ensure that fi_info can get the RDMA device information. If fi_info can retrieve the RDMA device information, then vineyard should initialize successfully. If fi_info cannot get the RDMA device information, then vineyard will not be able to initialize either.

Whether or not is ok when i parses ipv6:port to get port?

No.  Because vineyard will automatically look for the first suitable RDMA device.

By the way，as rdma client，read_env("VINEYARD_RDMA_ENDPOINT"), Who write VINEYARD_RDMA_ENDPOINT?

If you provide rdma_endpoint when trying to connect to vineyardd, this environment variable will not be read. If you don't give it, it will try to read it. Environment variables are also set by the user.

If RPCServer has tcp server and RDMAServer with different port at the same time, Whether or not to conflict？

They won't conflict.

Whether or not is ok when i parses ipv6:port to get port?

No.  Because vineyard will automatically look for the first suitable RDMA device. As I said above, specifying NIC by address will be supported in the next pr.

By the way，as rdma client，read_env("VINEYARD_RDMA_ENDPOINT"), Who write VINEYARD_RDMA_ENDPOINT?

If you provide rdma_endpoint when trying to connect to vineyardd, this environment variable will not be read. If you don't give it, it will try to read it. Environment variables are also set by the user.

If RPCServer has tcp server and RDMAServer with different port at the same time, Whether or not to conflict？

They won't conflict.

Hi
as client,usr set VINEYARD_RDMA_ENDPOINT,for example <IP_ADDRESS>:1234,is it client self address or server address ? tks

as client,usr set VINEYARD_RDMA_ENDPOINT,for example <IP_ADDRESS>:1234,is it client self address or server address ? tks

The client should use the exact ipv4 address of vineyard server. Suppose the NIC of the server is at the address <IP_ADDRESS>, and vineyard RDMA server use port of 1234. You should use <IP_ADDRESS>:1234 as the VINEYARD_RDMA_ENDPOINT of client. It is also currently not possible for client to specify the NIC used to send the data, so this field means the address of the server.

As I said above, specifying NIC by address will be supported in the next pr.
These include the NIC used by the server to receive data and the NIC used by the client to send data.

To summarize, there is currently no way for the server to specify the NIC, and the server will automatically select the appropriate NIC to listen to RDMA messages. The rdma endpoint on the client side is the address of the server. It is also currently not possible for client to specify the NIC used for sending data. The feature to specify NIC will be supported in the pr mentioned above, but cannot currently be incorporated into the main branch.

I20241031 15:04:05.911379 7 rpc_server.cc:105] Vineyard will listen on <IP_ADDRESS>:9600 for RPC
I20241031 15:04:08.535791 7 rpc_server.cc:109] Vineyard will listen on <IP_ADDRESS>:9600 for RDMA

Do not make RDMA and RPC work on the same port.

As client, if VINEYARD_RDMA_ENDPOINT set to server address <IP_ADDRESS>:9600,show
Connect RDMA server failed! Fall back to RPC mode. Error:fi_getinfo failed.
Why?client can't find server?
As client, if VINEYARD_RDMA_ENDPOINT set to itself address <IP_ADDRESS>:9600,show
rpc_client = vineyard.connect('vineyardd-sample-rpc.admin',9600)

Are the client and server in the same container? If not, can the client's container connect to the server?
You can start the server and client in the same container to test if the vineyard RDMA works.

As client, if VINEYARD_RDMA_ENDPOINT set to server address <IP_ADDRESS>:9600,show
Connect RDMA server failed! Fall back to RPC mode. Error:fi_getinfo failed.
Why?client can't find server?
As client, if VINEYARD_RDMA_ENDPOINT set to itself address <IP_ADDRESS>:9600,show
rpc_client = vineyard.connect('vineyardd-sample-rpc.admin',9600)

Are the client and server in the same container? If not, can the client's container connect to the server?
You can start the server and client in the same container to test if the vineyard RDMA works.

Hi
Now ,when client put data,then it has issue,Why?

import numpy as np
import vineyard
rpc_client = vineyard.connect('vineyardd-sample-rpc.admin',9600)
Connected to RPC server: vineyardd-sample-rpc.admin:9600, RDMA server: <IP_ADDRESS>:9600
objid = rpc_client.put(np.zeros(8))
mlx5: vineyard-python-client-847f8b8b86-s9t55: got completion with error:
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00008813 08000241 0001fdd2
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python3.8/dist-packages/vineyard/core/client.py", line 823, in put
return put(self, value, builder, persist, name, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/vineyard/core/builder.py", line 197, in put
meta = get_current_builders().run(client, value, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/vineyard/core/builder.py", line 100, in run
return self.factory[ty](client, value, **kw)
File "/usr/local/lib/python3.8/dist-packages/vineyard/data/tensor.py", line 90, in numpy_ndarray_builder
meta.add_member('buffer', build_numpy_buffer(client, value))
File "/usr/local/lib/python3.8/dist-packages/vineyard/data/utils.py", line 178, in build_numpy_buffer
return build_buffer(client, address, array.nbytes)
File "/usr/local/lib/python3.8/dist-packages/vineyard/data/utils.py", line 157, in build_buffer
return client.create_remote_blob(buffer)
File "/usr/local/lib/python3.8/dist-packages/vineyard/core/client.py", line 575, in create_remote_blob
return self.rpc_client.create_remote_blob(blob_builder)
vineyard._C.InvalidException: Invalid: GetTXCompletion failed:-5

Hi. Could you please show me the complete instructions to start vineyardd and the code of putting object on the client side? Let me test it locally.

Hi. Could you please show me the complete instructions to start vineyardd and the code of putting object on the client side? Let me test it locally.

Hi, instructions are there:
use python client, login
export VINEYARD_RDMA_ENDPOINT=<IP_ADDRESS>:9600   // <IP_ADDRESS> is the rdma server address
python3

import numpy as np
import vineyard
rpc_client = vineyard.connect('vineyardd-sample-rpc.admin',9600)   //it needs adout 20s success
objid = rpc_client.put(np.zeros(8))     // it is sometimes possible to put success, but sometimes fail.

Hi. Could you please show me the complete instructions to start vineyardd and the code of putting object on the client side? Let me test it locally.

Hi, instructions are there: use python client, login export VINEYARD_RDMA_ENDPOINT=<IP_ADDRESS>:9600 // <IP_ADDRESS> is the rdma server address python3

import numpy as np
import vineyard
rpc_client = vineyard.connect('vineyardd-sample-rpc.admin',9600)   //it needs adout 20s success
objid = rpc_client.put(np.zeros(8))     // it is sometimes possible to put success, but sometimes fail.

And the command of starting a vineyardd?

Hi. Could you please show me the complete instructions to start vineyardd and the code of putting object on the client side? Let me test it locally.

Hi, instructions are there: use python client, login export VINEYARD_RDMA_ENDPOINT=<IP_ADDRESS>:9600 // <IP_ADDRESS> is the rdma server address python3

import numpy as np
import vineyard
rpc_client = vineyard.connect('vineyardd-sample-rpc.admin',9600)   //it needs adout 20s success
objid = rpc_client.put(np.zeros(8))     // it is sometimes possible to put success, but sometimes fail.

And the command of starting a vineyardd?
as server, set:
json RpcSpecResolver::resolve() const {
json spec;
spec["rpc"] = FLAGS_rpc;
spec["port"] = FLAGS_rpc_socket_port;
spec["rdma_endpoint"] = "<IP_ADDRESS>:9600";
return spec;
}
then deployment vineyard use helm chart

And If registering memory fails, try increasing the vineyard's available memory.

And If registering memory fails, try increasing the vineyard's available memory.
Hi
as 2Gi memory,it is same, sometimes possible to put success, but sometimes fail.
cat <<EOF | kubectl apply -f -
apiVersion: k8s.v6d.io/v1alpha1
kind: Vineyardd
metadata:
name: vineyardd-sample
namespace: admin
spec:
replicas: 1
metaServiceReplicas: 1
service:
type: ClusterIP
port: 9600
vineyard:
image: test:test/admin/vineyardd:v1
imagePullPolicy: IfNotPresent
cpu: "2"
memory: "2Gi"
securityContext:
privileged: true
EOF

Hi @hsh258. The 9600 is the default port for RPC, you should define another unique port for RDMA endpoint such as "<IP_ADDRESS>:9601"

Hi @hsh258. The 9600 is the default port for RPC, you should define another unique port for RDMA endpoint such as "<IP_ADDRESS>:9601"

Hi ,
If set other port, connect fail,show:
export VINEYARD_RDMA_ENDPOINT=<IP_ADDRESS>:19601
python3
Python 3.8.10 (default, Sep 11 2024, 16:02:53)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import numpy as np
import vineyard
rpc_client = vineyard.connect('vineyardd-sample-rpc.admin',19601)
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python3.8/dist-packages/vineyard/init.py", line 418, in connect
return Client(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/vineyard/core/client.py", line 296, in init
raise ConnectionError(
ConnectionError: Failed to connect to vineyard via both IPC and RPC connection. Arguments, environment variables VINEYARD_IPC_SOCKET and VINEYARD_RPC_ENDPOINT, as well as the configuration file, are all unavailable.
rpc_client = vineyard.connect('<IP_ADDRESS>',19601)
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python3.8/dist-packages/vineyard/init.py", line 418, in connect
return Client(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/vineyard/core/client.py", line 296, in init
raise ConnectionError(
ConnectionError: Failed to connect to vineyard via both IPC and RPC connection. Arguments, environment variables VINEYARD_IPC_SOCKET and VINEYARD_RPC_ENDPOINT, as well as the configuration file, are all unavailable.

The rpc is must be connected at first, you can try the following code.
export VINEYARD_RDMA_ENDPOINT=<IP_ADDRESS>:19601
import numpy as np
import vineyard
rpc_client = vineyard.connect('vineyardd-sample-rpc.admin',9600)

The rpc must be connected at first while using the rdma, you can try the following code.
export VINEYARD_RDMA_ENDPOINT=<IP_ADDRESS>:19601
import numpy as np
import vineyard
rpc_client = vineyard.connect('vineyardd-sample-rpc.admin',9600)

Hi, accord to above order,
The issue is same,especially when server and client are in the different node.
In server:
rpc_server.cc:105] Vineyard will listen on <IP_ADDRESS>:9600 for RPC
rpc_server.cc:109] Vineyard will listen on <IP_ADDRESS>:19800 for RDMA
rpc_server.cc:203] Receive vineyard request mem!
rpc_server.cc:208] Receive remote request address: 0x7fe39bfff040 size: 31457280
rpc_server.cc:241] Failed to register mem. size 31457280
rpc_server.cc:392] Connection error!Client crashed.
rpc_server.cc:334] Receive close msg!
rpc_server.cc:400] Get RX completion failed! Error:Client crashed.
In client:
mlx5: vineyard-python-client-65dfffb656-ghc72: got completion with error:
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00008813 080004bc 000106d2
Traceback (most recent call last):
File "", line 1, in
File "", line 8, in process_blocks
File "/usr/local/lib/python3.8/dist-packages/vineyard/core/client.py", line 823, in put
return put(self, value, builder, persist, name, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/vineyard/core/builder.py", line 197, in put
meta = get_current_builders().run(client, value, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/vineyard/core/builder.py", line 100, in run
return self.factory[ty](client, value, **kw)
File "/usr/local/lib/python3.8/dist-packages/vineyard/data/tensor.py", line 90, in numpy_ndarray_builder
meta.add_member('buffer', build_numpy_buffer(client, value))
File "/usr/local/lib/python3.8/dist-packages/vineyard/data/utils.py", line 178, in build_numpy_buffer
return build_buffer(client, address, array.nbytes)
File "/usr/local/lib/python3.8/dist-packages/vineyard/data/utils.py", line 157, in build_buffer
return client.create_remote_blob(buffer)
File "/usr/local/lib/python3.8/dist-packages/vineyard/core/client.py", line 575, in create_remote_blob
return self.rpc_client.create_remote_blob(blob_builder)
vineyard._C.InvalidException: Invalid: GetTXCompletion failed:-5

Could you please add an option （size：1024Mi） to the vineyard yaml as follows and try again？

Hi
as 2Gi memory,it is same, sometimes possible to put success, but sometimes fail.
cat <<EOF | kubectl apply -f -
apiVersion: k8s.v6d.io/v1alpha1
kind: Vineyardd
metadata:
name: vineyardd-sample
namespace: admin
spec:
replicas: 1
metaServiceReplicas: 1
service:
type: ClusterIP

port: 9600

vineyard:
size：1024Mi
image: test:test/admin/vineyardd:v1
imagePullPolicy: IfNotPresent

cpu: "2"

memory: "2Gi"

securityContext:
privileged: true

EOF

size：1024Mi

Hi
deploy has issue in "size" ,
size：1024Mi
error: error parsing STDIN: error converting YAML to JSON: yaml: line 16: mapping values are not allowed in this context
size："1024Mi"
error: error parsing STDIN: error converting YAML to JSON: yaml: line 16: mapping values are not allowed in this context

Sorry for the misleading indentation. You could try the following command.
cat <<EOF | kubectl apply -f -
apiVersion: k8s.v6d.io/v1alpha1
kind: Vineyardd
metadata:
  name: vineyardd-sample
  namespace: admin
spec:
  replicas: 1
  service:
    type: ClusterIP
    port: 9600
  vineyard:
    size: 1024Mi
    image: test:test/admin/vineyardd:v1
    imagePullPolicy: IfNotPresent
    cpu: "2"
    memory: "2Gi"
  securityContext:
    privileged: true
EOF

Sorry for the misleading indentation. You could try the following command.
cat <<EOF | kubectl apply -f -
apiVersion: k8s.v6d.io/v1alpha1
kind: Vineyardd
metadata:
  name: vineyardd-sample
  namespace: admin
spec:
  replicas: 1
  service:
    type: ClusterIP
    port: 9600
  vineyard:
    size: 1024Mi
    image: test:test/admin/vineyardd:v1
    imagePullPolicy: IfNotPresent
    cpu: "2"
    memory: "2Gi"
  securityContext:
    privileged: true
EOF

Hi, above issue,
error: error parsing STDIN: error converting YAML to JSON: yaml: line 16: mapping values are not allowed in this context
As deply:
cat <<EOF | kubectl apply -f -
apiVersion: k8s.v6d.io/v1alpha1
kind: Vineyardd
metadata:
name: vineyardd-sample
namespace: admin
spec:
replicas: 1
service:
type: ClusterIP
port: 9600
vineyard:
size: 1024Mi
image: test:test/admin/vineyardd:v1
imagePullPolicy: IfNotPresent
cpu: "2"
memory: "2Gi"
securityContext:
privileged: true
EOF

The rpc must be connected at first while using the rdma, you can try the following code.
export VINEYARD_RDMA_ENDPOINT=<IP_ADDRESS>:19601
import numpy as np
import vineyard
rpc_client = vineyard.connect('vineyardd-sample-rpc.admin',9600)

Hi,
As exec "rpc_client = vineyard.connect('vineyardd-sample-rpc.admin',9600)",it cost long time,from a few seconds to over 20 seconds.
add debug info,it focus the func
CHECK_ERROR(!fi_getinfo(VINEYARD_FIVERSION, server_address.c_str(),
std::to_string(port).c_str(), 0, hints,
reinterpret_cast<fi_info**>(&(info.fi)))
Could you tell me how to shorten the time? tks

The rpc must be connected at first while using the rdma, you can try the following code.
export VINEYARD_RDMA_ENDPOINT=<IP_ADDRESS>:19601
import numpy as np
import vineyard
rpc_client = vineyard.connect('vineyardd-sample-rpc.admin',9600)

Hi, As exec "rpc_client = vineyard.connect('vineyardd-sample-rpc.admin',9600)",it cost long time,from a few seconds to over 20 seconds. add debug info,it focus the func CHECK_ERROR(!fi_getinfo(VINEYARD_FIVERSION, server_address.c_str(), std::to_string(port).c_str(), 0, hints, reinterpret_cast<fi_info**>(&(info.fi))) Could you tell me how to shorten the time? tks

It shouldn't be very slow, what's your k8s environment (ack/aws/...)?

The rpc must be connected at first while using the rdma, you can try the following code.
export VINEYARD_RDMA_ENDPOINT=<IP_ADDRESS>:19601
import numpy as np
import vineyard
rpc_client = vineyard.connect('vineyardd-sample-rpc.admin',9600)

Hi, As exec "rpc_client = vineyard.connect('vineyardd-sample-rpc.admin',9600)",it cost long time,from a few seconds to over 20 seconds. add debug info,it focus the func CHECK_ERROR(!fi_getinfo(VINEYARD_FIVERSION, server_address.c_str(), std::to_string(port).c_str(), 0, hints, reinterpret_cast<fi_info**>(&(info.fi))) Could you tell me how to shorten the time? tks

It shouldn't be very slow, what's your k8s environment (ack/aws/...) and machine environment ?

Hi,
Version: v1.28.3

cat <<EOF | kubectl apply -f -
apiVersion: k8s.v6d.io/v1alpha1
kind: Vineyardd
metadata:
name: vineyardd-sample
namespace: admin
spec:
replicas: 1
service:
type: ClusterIP
port: 9600
vineyard:
size: 1024Mi
image: test:test/admin/vineyardd:v1
imagePullPolicy: IfNotPresent
cpu: "2"
memory: "2Gi"
securityContext:
privileged: true
EOF

How do you install the vineyard operator? Besides, can you copy the code to the shell and try again, it's better to show the failed screenshot so that we can check where is wrong.

cat <<EOF | kubectl apply -f -
apiVersion: k8s.v6d.io/v1alpha1
kind: Vineyardd
metadata:
name: vineyardd-sample
namespace: admin
spec:
replicas: 1
service:
type: ClusterIP
port: 9600
vineyard:
size: 1024Mi
image: test:test/admin/vineyardd:v1
imagePullPolicy: IfNotPresent
cpu: "2"
memory: "2Gi"
securityContext:
privileged: true
EOF

How do you install the vineyard operator? Besides, can you copy the code to the shell and try again, it's better to show the failed screenshot so that we can check where is wrong.

Hi,
Use helm chart to install the vineyard operator

error: error parsing STDIN: error converting YAML to JSON: yaml: line 16: mapping values are not allowed in this context
This error shouldn't happen. In my test environment, it can work fine as follows.

$ helm repo update
$ kubectl create namespace vineyard-system
$ helm install vineyard-operator vineyard/vineyard-operator -n vineyard-system
$ cat <<EOF | kubectl apply -f -
apiVersion: k8s.v6d.io/v1alpha1
kind: Vineyardd
metadata:
  name: vineyardd-sample
  namespace: admin
spec:
  replicas: 1
  service:
    type: ClusterIP
    port: 9600
  vineyard:
    size: 1024Mi
    image: test:test/admin/vineyardd:v1
    imagePullPolicy: IfNotPresent
    cpu: "2"
    memory: "2Gi"
  securityContext:
    privileged: true
EOF

error: error parsing STDIN: error converting YAML to JSON: yaml: line 16: mapping values are not allowed in this context
This error shouldn't happen. In my test environment, it can work fine as follows.

$ helm repo update
$ kubectl create namespace vineyard-system
$ helm install vineyard-operator vineyard/vineyard-operator -n vineyard-system
$ cat <<EOF | kubectl apply -f -
apiVersion: k8s.v6d.io/v1alpha1
kind: Vineyardd
metadata:
  name: vineyardd-sample
  namespace: admin
spec:
  replicas: 1
  service:
    type: ClusterIP
    port: 9600
  vineyard:
    size: 1024Mi
    image: test:test/admin/vineyardd:v1
    imagePullPolicy: IfNotPresent
    cpu: "2"
    memory: "2Gi"
  securityContext:
    privileged: true
EOF

Hi,
Try it, and can deploy,tks.
By the way, connect cost long time still.

I think it is likely to be caused by your environmental factors.
Can RDMA work now？

I think it is likely to be caused by your environmental factors. Can RDMA work now？

Hi，
I want to try to set server IP by fi_getinfo.Is the method feasible？If ok，how to set? tks.

I think it is likely to be caused by your environmental factors. Can RDMA work now？

Hi， I want to try to set server IP by fi_getinfo.Is the method feasible？If feasible，how to set? tks.

Refer to src/common/rdma/rdma_client.cc, src/common/rdma/rdma_server.cc and https://ofiwg.github.io/libfabric/
Vineyard client get the server RDMA device info by calling fi_getinfo with param of server ip address.

I think it is likely to be caused by your environmental factors. Can RDMA work now？

Hi, Now RDMA doesn't work normal. It connect very slowly,and put and get data very slow,too.

Hi,
About speed of rdma write and read is slow，I find that  the transmition is stucked for over 100 milliseconds every time hundreds or 1000 packages。
I want to try to change tx_ctx_cnt and rx_ctx_cntof verbs_domain_attr. Is the method feasible？Or other way? tks.

Since there is no way to replicate your current environment, and no ipv6 support for vineyard, it may be difficult to give appropriate advice.

I want to try to change tx_ctx_cnt and rx_ctx_cntof verbs_domain_attr. Is the method feasible？Or other way? tks.
We haven't tried to change this field, so I suggest you ask in the fabric community.

Since there is no way to replicate your current environment, and no ipv6 support for vineyard, it may be difficult to give appropriate advice.

I want to try to change tx_ctx_cnt and rx_ctx_cntof verbs_domain_attr. Is the method feasible？Or other way? tks.

We haven't tried to change this field, so I suggest you ask in the fabric community.

Hi，
the environment is ipv4，the speed of rdma write and read is slow。

Since there is no way to replicate your current environment, and no ipv6 support for vineyard, it may be difficult to give appropriate advice.

I want to try to change tx_ctx_cnt and rx_ctx_cntof verbs_domain_attr. Is the method feasible？Or other way? tks.

We haven't tried to change this field, so I suggest you ask in the fabric community.

Hi， the environment is ipv4，the speed of rdma write and read is slow。

Hi, what is the size of the blob you are testing now? At present, Vineyard RDMA module needs to adapt to some working conditions, so blob has advantages over TCP only when it is larger than 4M.

Hi， the environment is ipv4，the speed of rdma write and read is slow。

Hi, what is the size of the blob you are testing now? At present, Vineyard RDMA module needs to adapt to some working conditions, so blob has advantages over TCP only when it is larger than 4M.

Hi，
Blob is 30M，and I find the mainly reason  is that  recv_message of io.cc cost about 60ms between 30M request.
How to short the time ? tks.


[HOLD for payment 2022-11-09] [$250] Bug: old message not loading while switching from offline to online reported by @gadhiyamanan

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

go any chat which has lots of messages
go to offline
go back to online try to load old messages

Expected Result:
messages should load
Actual Result:
old message not loading
Workaround:
unknown
Platform:

Where is this issue occurring?

Web
iOS
Android
Desktop App
Mobile Web

Version Number: 1.2.12-3
Reproducible in staging?: y
Reproducible in production?: y
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos:
https://user-images.githubusercontent.com/43996225/195211614-553a84b6-d8c3-4415-9a72-6a51f36585ed.mp4
https://user-images.githubusercontent.com/43996225/195211618-48ca49dc-ca98-4d7f-9955-264b5b5f3ac5.mov
Expensify/Expensify Issue URL:
Issue reported by: @gadhiyamanan
Slack conversation: https://expensify.slack.com/archives/C01GTK53T8Q/p1665499460172419
View all open jobs on GitHub

I actually can't even scroll at all - regardless of being offline > online again when I test this in my own account on STG.

But I can still scroll when I test this on PROD:

Are we tackling the general issue with scrolling more broadly? I can't see it in Opensource or GH yet. Thanks eng team, this one is ready for your eyes!

Yep looks like when we get back online we stop calling the command to get more chats

I can repro this on my account in staging

Proposal
Problem

Currently, in https://github.com/Expensify/App/blob/7044ab2b96067081c8f67191e51fa859dba66d91/src/pages/home/report/ReportActionsView.js#L263  when user scroll to get old message, in function loadMoreChats, ReadOldestAction will be called and update isLoadingMoreReportActions=true. After that when API is successfully called or error return by BE, will update isLoadingMoreReportActions=false.

  API.read('ReadOldestAction',
        {
            reportID,
            reportActionsOffset: oldestActionSequenceNumber,
        },
        {
            optimisticData: [{
                onyxMethod: CONST.ONYX.METHOD.MERGE,
                key: `${ONYXKEYS.COLLECTION.REPORT}${reportID}`,
                value: {
                    isLoadingMoreReportActions: true,
                },
            }],
            successData: [{
                onyxMethod: CONST.ONYX.METHOD.MERGE,
                key: `${ONYXKEYS.COLLECTION.REPORT}${reportID}`,
                value: {
                    isLoadingMoreReportActions: false,
                },
            }],
            failureData: [{
                onyxMethod: CONST.ONYX.METHOD.MERGE,
                key: `${ONYXKEYS.COLLECTION.REPORT}${reportID}`,
                value: {
                    isLoadingMoreReportActions: false,
                },
            }],
        });

In offline mode, ReadOldestAction will be called first and update isLoadingMoreReportActions=true, but API not called => responseData =undefined and in function SaveResponseInOnyx will return void without update Onyx. So isLoadingMoreReportActions still being true, at this case in function loadMoreChats ReadOldestAction won't be called when user back to online and scroll to get old message.

function SaveResponseInOnyx(response, request) {
    return response
        .then((responseData) => {
            const onyxUpdates = [];
            if (!responseData) {
+           we dont't have responseData
                return;
            }
            if (responseData.jsonCode === 200 && request.successData) {
                onyxUpdates.push(...request.successData);
            } else if (responseData.jsonCode !== 200 && request.failureData) {
                onyxUpdates.push(...request.failureData);
            }
           ...
        });
}

Solution
In https://github.com/Expensify/App/blob/7044ab2b96067081c8f67191e51fa859dba66d91/src/libs/actions/Report.js add function resetLoadMoreReportActions to set isLoadingMoreReportActions in this report to false
+ function resetLoadMoreReportActions(reportID) {
+    Onyx.merge(`${ONYXKEYS.COLLECTION.REPORT}${reportID}`, { isLoadingMoreReportActions: false })
+ }

export {
    ...
+   resetLoadMoreReportActions
}

In https://github.com/Expensify/App/blob/7044ab2b96067081c8f67191e51fa859dba66d91/src/pages/home/report/ReportActionsView.js in shouldUpdateComponent lifecycle, when user come back online mode, we'll check this.props.report.isLoadingMoreReportActions, if true we'll change isLoadingMoreReportActions in this report to false.
if (lodashGet(nextProps.network, 'isOffline') !== lodashGet(this.props.network, 'isOffline')) {
+            if (!lodashGet(nextProps.network, 'isOffline') && this.props.report.isLoadingMoreReportActions) {
+                Report.resetLoadMoreReportActions(this.props.report.reportID)
+            }
            return true;
        }

https://user-images.githubusercontent.com/113963320/196002715-acf6db4f-5d44-45ee-9f68-0674d2ea04bb.mp4

@tienifr That's a great investigation - thanks for that! That's a well described issue, so we optimistically set isLoadingMoreReportActions to true. ReadOldestAction is not requeued when the user is back online since it's a read request, but when we reconnect, we should be calling Report.reconnect here which should reset our isLoadingReportActions to false after a successful/failed request. At that point, isLoadingMoreReportActions would be updated and then we wouldn't need to it override it here.  Could you try to investigate why that isn't happening?

I am preoccupied with some other stuff. @thienlnam Please assign this to someone else. Thanks.

@thienlnam In here, isLoadingReportActions will be updated but isLoadingMoreReportActions is not updated in Report.openReport and Report.reconnect
https://github.com/Expensify/App/blob/e38df100d62d197689bb16888a3ac7859f157681/src/libs/actions/Report.js#L695-L725
https://github.com/Expensify/App/blob/e38df100d62d197689bb16888a3ac7859f157681/src/libs/actions/Report.js#L732-L758
Solution 2
If you want to update isLoadingMoreReportActions  in https://github.com/Expensify/App/blob/e38df100d62d197689bb16888a3ac7859f157681/src/pages/home/report/ReportActionsView.js#L163-L175, we must pass isLoadingMoreReportActions: false to value of optimisticData in function  openReport and reconnect.
function openReport(reportID) {
    API.write('openReport',
            ...
             optimisticData: [{
                ...
                value: {
                    isLoadingReportActions: true,
+                  isLoadingMoreReportActions: false,
                },
            }],
            ...
}

function reconnect(reportID) {
    API.write('ReconnectToReport',
             ...
             optimisticData: [{
                ...
                value: {
                    isLoadingReportActions: true,
+                  isLoadingMoreReportActions: false,
                },
            }],
            ...
        });
}

Solution 3
A simpler way, https://github.com/Expensify/App/blob/7044ab2b96067081c8f67191e51fa859dba66d91/src/pages/home/report/ReportActionsView.js#L263 in function loadMoreChats if user in offline mode we'll immediately return.
loadMoreChats(){
       if (this.props.report.isLoadingMoreReportActions || lodashGet(nextProps.network, 'isOffline')) {
            return;
        }
        ...
}

@thienlnam Could you please send the Upwork link that I can apply to? thanks

@tienifr  - here ya go!  https://www.upwork.com/ab/applicants/1582472084181204992/suggested

Thanks @slafortune, I applied to the job and the PR will be ready by October 21

@slafortune Upwork job missing for reporting bonus

hi @slafortune I applied to the job a while ago as in https://github.com/Expensify/App/issues/11737#issuecomment-1285231203 but haven't received the offer for the job yet, can you please help check?
thanks

cc @slafortune Looks like we're missing some payments here

Sorry for my delay on this - looking for some help here and will get this done by tomorrow!

@thesahindia @gadhiyamanan I've sent you both an offer for this, let me know once you've accepted that and I'll get this paid out - sorry for my delay!

@slafortune applied 🎉

Accepted, thanks!

@thesahindia  I’m trying to wrap up this GH and am looking for the original PR that caused the issue, can you help me out with that?

@thesahindia I’m trying to wrap up this GH and am looking for the original PR that caused the issue, can you help me out with that?

Oh sorry, I didn't see the comment. Let me check.

I think it was missed in the refactors as it was an edge case. Also it's hard to tell which specific PR should have handled this
cc: @thienlnam

Yeah that's correct, this was likely missed during an internal refactor as an edge case.  It's been a while but it was probably this one? https://github.com/Expensify/App/pull/11439


/etc/environment not working

hello sir! first of all my humble thanks to you for creating this wonder full tool for saving my time .but, small error is still arising .when i try to use internet in firefox ,its not working.please see this issue
Thanks in advance

Firefox by default seems to obey system proxy. Please check in Firefox settings (preferences), what the proxy settings are.

If your issue is specific to /etc/environment, move on to #45.


Add support for SPI bus

This pull request adds SPI bus support.
What has been tested: The example/spi.rs returns plausible values in 4-wire-SPI-mode.
What has not been tested: 3-wire-SPI-mode.
I created a private trait that is not exported from the crate to hold methods (mostly) common between busses.
Open question: How should we handle the Bus and SlaveSelect types?

Should we stay with the current integer to enum mapping? (Note the commented out mappings for newer rppal version).
Should we re-export the types from rppal?
Any other ideas?

Please share your thoughts. What other aspects should be improved?
Thanks a lot.

Do you have any comments for improvements?
Thanks :)

Sorry taking so long for me to respond. I have very limited internet access right now. The code looks very good just probably need to rename the xxPrivate to something better but I'll to think about it. Give me a couple more days to switch from thinking in Typescript react frontend code to Rust hardware driver.

Thanks for your reply.
No need to hurry. Please take your time to think this through.
In the mean time I tested this quite a bit and everything seems to work fine.
One thing I was thinking about after sending the pull:
Should we even bother with 3-wire-SPI? I'm running 4-wire-SPI. I'm not even sure whether rppal and the RPI hardware supports 3-wire-SPI mode. Therefore I'm unsure whether we should expose this option here.
What about renaming Adxl345Private -> Adxl345BusCommon?

There is support for 3-wire on the RP hardware side so should leave that in with comments about not test etc. Instead of Adxl345BusCommon was think more like Adxl345Init but still think on that. The phone I was using for internet died and had to wait for replace to show up so I've spent less time on this than I like 😒

Hi. Are there any news here? Any blocking things?
I'd currently like to go on and port this over to embedded-hal to make it usable on devices other than RPi.
But I'd like to base that work on this PR.
Thanks for your time.

I'm very sorry I never got back to this like I was going to. After partially getting phone/ Internet issues fixed and TS project out of the way I took some time off from programming as I was starting to feel some burnout then watched some YT videos which got me started on a open hardware project which became 3 and counting projects 😉😳 and learning about CAD based PCB design. Anyway think I've decided Adxl345Init is my preference for renaming Adxl345Private so if you can make that change on PR I'll be happy to merge it. Think there was something else I had a question on but was minor and can't remember what it was but know it wasn't anything blocking. BTW if your looking at porting to anything like RPi pico or WeMOS D1 mini I'd be very interested as I've get a idea for a project where I was thinking this device would be useful which will use one of those as a microcontroller.

Cool. Thanks. I'll make that change.
I'm currently porting it to embedded-hal. My current changes already work on Raspi with embedded-hal + rppal (without direct rppal dependency in adxl345_driver) . That would make it usable on ESP32 (which is my goal) and possibly also Raspi Pico (I didn't check whether the pico drivers support embedded-hal. But if not, such support shall be added).
I also think this crate should be no_std. I also plan to work on that.

Done. I pushed the rename of the trait.


[NGINX proxy] Add article to "Read more"

The config.yaml seemed to be the only place to put this.. Thanks!

If you agree with this suggestion, perhaps it would be prudent to add the same article to config.yaml below as well?
https://github.com/smallstep/hello-mtls/blob/master/docs/nginx/config.yaml


Escaping characters in environment variables in MP

Environment Details

Helidon Version: 2.0.0 snapshot
Helidon MP

In SE  (and in MP for older Helidon version) we support the possibility to escape dashes in environment variables.
Consider the following properties:
app.data-size
app.data.size
These are two distinct properties in config, yet would get replaced by the same environment variable.
This could be mitigated by supporting escape sequences for dash and dot in environment variable names, such as:
APP_DATA_dash_SIZE for the first property
APP_DATA_dot_SIZE for the second property
APP_DATA_SIZE would replace both properties unless one of the specifics above is defined as well.
Escaping dash is not sufficient, as there would be no way how to only override the second property (app.data.size).

Until this is resolved in the specification, we will keep current behavior.


per 2 Per communication

Hello I would like to know if this library can be used for p2p coms. if it is possible could someone make an example this would really help me out. I have been trying to get p2p working with the RFM95 for over a week now.

No this library is for LoRaWAN. It a port of an outdated version of LoRaMac-Node and seems to be abandoned. You could try akshayabali/pico-lora. Or more recently, radiolib can do both LORaWAN and LoRa P2P as well as a bunch of other stuff.


Unable to switch to using kubeip v2, returning region-related error

Running the latest version of KubeIP v2's DaemonSet on GKE, pods start in the correct place but then immediately throw this error:
func: "main.assignAddress"
msg: "failed to assign static public IP address to node (node id)"
error: "check if static public IP is already assigned to instance (node id): failed to list assigned addresses: failed to list available addresses: googleapi: Error 400: Invalid value for field 'region': 'us-central1-a'. Unknown region., invalid"

(It should fail as it's got a static IP - it's currently using Kubeip v1 which we want to upgrade from as it seems to wipe other Kubernetes labels when setting its own - but I'm suspecting this region error is different, as the region presumably should be us-central1? No region is defined in the user-facing KubeIP config so I'm unclear what could need tweaking here).
Any assistance appreciated!

Current config, nothing particularly non-standard here:
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kubeip

spec:
  selector:
    matchLabels:
      app: kubeip
  template:
    metadata:
      labels:
        app: kubeip
    spec:
      tolerations:
      - effect: NoSchedule
        key: app
        operator: Equal
        value: appname
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: app
                operator: In
                values:
                - appname
      serviceAccountName: kubeip-service-account
      terminationGracePeriodSeconds: 30
      priorityClassName: system-node-critical
#      nodeSelector:
#        kubeip.com/public: "true"
      containers:
        - name: kubeip
          image: doitintl/kubeip-agent
          resources:
            requests:
              cpu: 100m
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: FILTER
              value: labels.kubeip=nodepoolname
            - name: LOG_LEVEL
              value: debug
            - name: LOG_JSON
              value: "true"

Current config, nothing particularly non-standard here:
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kubeip

spec:
  selector:
    matchLabels:
      app: kubeip
  template:
    metadata:
      labels:
        app: kubeip
    spec:
      tolerations:
      - effect: NoSchedule
        key: app
        operator: Equal
        value: appname
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: app
                operator: In
                values:
                - appname
      serviceAccountName: kubeip-service-account
      terminationGracePeriodSeconds: 30
      priorityClassName: system-node-critical
#      nodeSelector:
#        kubeip.com/public: "true"
      containers:
        - name: kubeip
          image: doitintl/kubeip-agent
          resources:
            requests:
              cpu: 100m
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: FILTER
              value: labels.kubeip=nodepoolname
            - name: LOG_LEVEL
              value: debug
            - name: LOG_JSON
              value: "true"

I had the same issue so added an env entry for region myself which seems to have resolved this problem. E.g.
        - name: REGION
          value: "us-central1"


Added two methods to klarnaOrderManagement

Called getOrder and updateReferences.
reason for pull against master is that development is behind.

Thinking about it - the changes requires the lastest es7. Would you like it changed into es6?


[Bug report]: "Chrysalis database migration failed" Error message

Platform
Desktop
Version
2.0.1
Description
After Upgrading from 1.7.5 to 2.0.1 when i try to open a Profile it says:
Wed, 04 Oct 2023 16:36:36 GMT
wallet: Chrysalis database migration failed: migration failed CipherError { alg: "XCHACHA20-POLY1305" }
I recovered for a test my old Ledger Firefly Folder from an old Backup. Uninstalled Firefly 2.0.1 and installed Firefly 1.7.5 --> Profile works like a charm. Upgrading it again to 2.0.1 --> Same Error see above.
In my Firefly are two profiles --> one based on Ledger and one from Stronghold. --> No one is accessible and after PIN-Entry the Error appears on both Profiles.
Expected behaviour
A Clean Migration including all Transactions where in the Profile.
Actual behaviour
See description
Can the issue reliably be reproduced?
Yes
Steps to reproduce the issue
See Description
Operating System
Windows
Diagnostics
App Name: Firefly
Version 2.0.1
Sprache: de
Plattform: win32
Plattformversion: 10.0.22621
Plattform-Architektur: x64
CPU-Anzahl: 24
Gesamtspeicher: 32488.7 MB
Freier Speicher: 17552.8 MB

Developer Profile
No
Profile Type
Hardware (Ledger)
Staking
None
Crash Status
None of the above
Error Messages
wallet: Chrysalis database migration failed: migration failed CipherError { alg: "XCHACHA20-POLY1305" }
Error Logs
wallet: Chrysalis database migration failed: migration failed CipherError { alg: "XCHACHA20-POLY1305" }

Duplicate declaration

[X] I have searched the issues tracker this issue and there is none

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Hey @CrashOverride-lab we have released a new version 2.0.2, can you give it a try and let us know if this is fixed for you? :smile:
Thank you!

Hi Marc,
unfortunatelly Version 2.0.2 only helps with the Problem on Stronghold Profile.
The Ledger Profile displays now:

But i was in Contact with Thoralf about this and he means, that in the Next Release this should be fixed. Can you estimate when the next Version will be available?

Hi Marc,
unfortunatelly Version 2.0.2 only helps with the Problem on Stronghold Profile.
The Ledger Profile displays now:

But i was in Contact with Thoralf about this and he means, that in the Next Release this should be fixed. Can you estimate when the next Version will be available?

We found and fixed the issue you describe in this message too, we will release a new version as soon as possible

Fixed by https://github.com/iotaledger/firefly/pull/7600 & https://github.com/iotaledger/firefly/pull/7572, we will release a new production version very soon


Move Astro vs JSX comparison to an existing page

We currently have our Astro vs JSX chart on our deprecated “Astro vs X” page which could be deleted soon 😱
We’re linking this in the components guide so should probably move this somewhere more reliable.

Done in #1188!


HTML Build and FunkinLua

Describe your bug here.
Build for html.
Pain
I didnt download from releases, if that makes a difference, i used it as a template on git hub then downloaded it from github.
I have also uninstalled and reinstalled linc_luajit MULTIPLE TIMES. Windows Build works fine but this doesnt. Other than that psych engine is pretty good so far :). Please help me if you can.
Command Prompt/Terminal logs (if existing)
C:\fnf>echo BUILDING GAME
BUILDING GAME

C:\fnf>lime test html5 -release
source/FunkinLua.hx:3061: characters 23-28 : Type not found : State
source/CoolUtil.hx:18: lines 18-141 : ... Defined in this class
source/PlayState.hx:3752: characters 40-68 : Class<FunkinLua> has no field getPropertyLoopThingWhatever
source/PlayState.hx:4895: characters 8-12 : FunkinLua has no field call
source/PlayState.hx:4896: characters 8-12 : FunkinLua has no field stop

Are you modding a build from source or with Lua?
Source
What is your build target?
HTML5
Did you edit anything in this build? If so, mention or summarize your changes.
In the referenced files in the log, i did not edit anything.


Problems after Update

function containsObs(obs, aftersList) {
    for (var i in aftersList) {
>      if (aftersList[i][0].id == obs.id) return true;
    }
    return false;
  }

It throws with
   TypeError: Cannot read property 'id' of undefined at containsObs (main.generated.js:15853)
    at afterTransaction (main.generated.js:15836)
    at Object.sink (main.generated.js:15978)
    at PropertyDispatcher.Dispatcher.pushToSubscriptions (main.generated.js:16740)
    at PropertyDispatcher.Dispatcher.pushIt (main.generated.js:16762)
    at Object.inTransaction (main.generated.js:15947)
    at PropertyDispatcher.Dispatcher.push (main.generated.js:16731)
    at PropertyDispatcher.push (main.generated.js:17172)
    at PropertyDispatcher.Dispatcher.handleEvent

Trace ends there.
I've come so far, that I can say that there is a function in the aftersList, that doesn't have an id.
How to fix this? Gotta roll back to older Bacon for now.
Rollback to 0.7.59 solves the issues (it' the only version I'm trying now though)

Interesting and yet unseen. Any chance of producing an isolated repeatable scenario in webpackbin, jsfiddle or such?

I cannot isolate this as I don't know exactly what actually triggers it. The trace is too short. Sadly bacon Code is often difficult to debug.
Can you find out anything reviewing Code and changes?

I understand. I'll take a look at this soon.

These Bacon statements should be relevant for the problem - not sure though.
      function deviceStream(id) {
        return Bacon.fromPromise(getDevice(id).catch(restoreOffline));
      }

      deviceBus = new Bacon.Bus();
      deviceProperty = deviceBus.toProperty();

      deviceProperty.init = function() {
        if (!store.get('deviceID')) { // first device initialization
          // expose the creating promise instance
          // so other domain entity busses can await it
          deviceProperty.creating = newDevice().then(function(device) {
            deviceBus.push(device);
            return device;
          });
        } else {
          deviceBus.plug(deviceStream(store.get('deviceID')));
        }
        return deviceBus;
      };

I added a couple of extra assertions and an console.error message to debug this. Haven't figured out how this happens yet. If you care to try with this build it might help to sort this out.

So far I haven't figured out a way how an entry [obs, f] with obs==null or an empty entry [] could be in the aftersStack, unless an undefined observable is pushed into the stack. But that should also be impossible. If you run your app with the bacon.js build linked above and report errors here, it might help us to have a clue. Just replace the Bacon.js file with the linked one under your node_modules/baconjs/dist directory and repeat the problem.

When running, make sure you see the message "Debugging issue #693" in your console, to make sure you're actually running with this special build :)

Mighst also make sense to try with Bacon.js version 0.7.92 to narrow down the problem to the new "after transaction processing logic" introduced in 0.7.93. I'll do my best to fix this, but currently can't think of a way to reproduce the problem.
Another pair of eyes and a brain might help :) It seems that the state in the aftersStack variable in updatebarrier.js gets somehow corrupted. Also, maybe it's time to move to TypeScript?

Oh, an even easier way to test the application with this special build is by modifying your package.json to contain
  "dependencies": {
    "baconjs": "baconjs/bacon.js#693"
  }

This will get you the branch "693" from Github.

error: "observable missing from at index remove in array Bacon.Bus().toProperty(true).take(1),function (){if(!n){var e=t(r);if(e===w.noMore)return o()}}"id:

Sorry again with better traces:

// internal-domain.js:29
return isOnlineBus.toProperty(ionic.Platform.isWebView() ? $cordovaNetwork.isOnline() : navigator.onLine).log('isOnline');

Problems with this Boolean Bus?

Well it surely looks like there's an Observable where there's supposed to be an array (the aftersList appears to be an Observable).

hmm.. in fact I think this is the classic problem with the for (var i in array): when there are additional methods added to the Array prototype, they'll appear as indices in the for-loop. Patch coming up!

We had added junk into the Array prototype in Bacon.js tests but unfortunately our "junk" was a string so it didn't make tests fail. I added more junk (functions, objects, null ...) to the prototype and was able to reproduce this bug. The fix was simple. Will be included in 0.7.94 as soon as possible.

..and it's out. Pls test with Bacon.js 0.7.94 and report if it works for you!
Sorry for the inconvenience and thanks a lot for your help! With the improved prototype junk in the tests, there will be no more errors of this kind in future releases.

Okay nice :)
Question: why only minor version bumps with The latest reworks and additions?

That's a fair question! I think we should stop this madness and promote the current version as 1.0. Then do semantic versioning.

See #694 :)

Indeed fixed in the latest release! Good job!


Dependency deletions with destroy: true occur in the wrong order

tf-controller 0.16.0-rc.3
When you want to remove all the managed Terraform resources, you would run a Terraform destroy on all the resources.
If you have multiple Terraform CRs with dependencies set between them and you set:
destroyResourcesOnDeletion: true
then when you delete the Terraform CRs, tf-controller correctly deletes resources in the reverse order and blocks on a reverse dependency with the message:
Deletion in progress, but blocked, please delete XXX to continue
However, if you set the value:
destroy: true
on all Terraform CRs instead of deleting them, then tf-controller tries to destroy the resources in the wrong order, i.e. the same ordering as when running an apply and not the reverse ordering required for a destroy.
For example, resulting in attempting to delete the VPC prior to deleting the resources inside that VPC, which blocks the deletion of the VPC and the destroy failing to complete.

Thank you @darrylweaver
Currently we're still relying on K8s asynchronous reconciliation to do deletion.
To make everything deleted in the right order, we would need to adopt an algorithm from, for example, StatefulSets.
An idea would be:

Compute DAG
Sort the DAG with topology sorting
Block deletion of all objects in the sorted resources
Start the deletion process from in the reverse order

@madAndroid to try and reproduce. Q: is this a feature or a bug?

Currently ignores the dependencies and the expected behavior is that it would not.


Change temperature delta label to degrees Celsius

Problem
iOS Temperature Sensor screen shows Delta field with unit as F. Related to #9295.
Change overview
Change iOS Temperature Sensor screen Delta field label to degrees Celsius.
Testing
How was this tested?

Built with Xcode 12.5.1 and tested on iPhone with iOS 15 beta.

@chrisdecenzo @Damian-Nordic @hawk248 @jepenven-silabs @msandstedt ?


Feature request / help needed : Setting a gap between ticks for Y

I would like to put any gap I want for Y ticks. Here is an example with a 42 gap :

I managed to do this but I'm not happy with how I have done it and it leads to bugs, particulary with zooming. What I did is that, when rendering, I get the previous domain for the Y axis (which I know is the root of the problem):
let yScale = self.refs[myChartRef].yScale();
get Min / max value:
let top = yScale.domain()[1];
let bottom = yScale.domain()[0];
then, the short of it, is that I construct an array of values for my ticks, feeding them to YAxis tickValues.
Of course, when zooming and resetting zoom, it doesn't work, because the domain is limited to the chart when it was zoomed. The problem is that I can't predict the bottom / top values inside render() function (I guess browsing all my data points would be really overkill just for this)
I was thinking to make a callback to setState which would get the yScale domain, then setState with the correct top & bottom values. Is that the correct approach?

Well that seems to do it, simply doing a callback on setState which update the Y tickValues (which belong to state) when adding / removing points, and using my state' tickValues on YAxis.

in 0.4, there will be a yTicks={...} in the Chart.
yTicks would accept either an array or a function(domain, plotData) { /* calculate ticks */ return [tick1, tick2]; }
The function seems to solve your problem natively, no?

sorry. I take it back I like your solution better.
how about changing the ticks on XAxis and YAxis to accept a function or array?

Yes, a props of XAxis / YAxis that, given the domain, would return all the ticks would be enough.
I don't know about changing the ticks signature though,  even if d3 doesn't always obey you when specifying  a precise tick number, the users might not want to bother ?

yeah not ticks I meant to say tickValues edited the comment after


Improve room instance, continue songs, bind to API's

Rebased.

 So,you should set the new song.


index.d.ts missing in npm package

Version
2.9.3
Steps to reproduce
install version 2.9.3
Expected Behavior
index.d.ts is in downloaded package
Actual Behavior
index.d.ts is not in downloaded package. Because the file is in the src directory which is not included in the file's section of package.json file.

When you stopped including the src directory in the npm package, you broke the package for TypeScript users. 😭

Bad robot!

I made a PR to fix this: #238

🎉


Notifications

The bot tend to notify people whenever someone is muting their microphone/headphones, it dosnt do it all the time but for the most part it does, which is odd, any fixes?

Thanks for reporting.
When the bot restarts to a connection instability, it loses the tracking of who are in the channel.
I deployed a fix for this today

Thanks for reporting.
When the bot restarts to a connection instability, it loses the tracking of who are in the channel.
I deployed a fix for this today


nwg-dock-hyprland is not launching after updating hyprland to V42.0

after updating to hyprland v42 , it did as expected , got independent from wlroots and no more using it .
Everything is working flawlessly but I now nwg-dock-hyprland is no more launching , if I run from terminal , then it gives me this output :
─ankur@ankur in ~
 ╰─λ nwg-dock-hyprland
INFO[0000] Using auto-detected launcher command: 'nwg-drawer'
INFO[0000] Using style: /home/ankur/.config/nwg-dock-hyprland/style.css
FATA[0000] Couldn't list clients: json: cannot unmarshal number into Go struct field client.fullscreen of type bool

 ╭─ankur@ankur in ~ took 81ms
[🔴] ×

Expected behavior
it should launch when I run the command on terminal
Desktop (please complete the following information):

Distro : Gaurda Linux
go version : was never installed in my system, still it was working
gtk-layer-shell version :  0.8.2-1

Additional context
it can be due to hyprland dropping it's dependency from wlroots completely  , can have a look here

I haven't yet tested my stuff with Hyprland 0.42.0. It's still in Extra-Testing. If it introduces breaking changes, it's not my  fault. I'm leaving for a week on Monday. When I get back, I'll be deciding whether to adapt all my software or drop support for Hyprland.

Thanks for your reply
Will be waiting to know what is your final decision

LOL, most likely I won't drop Hyprland support. I just need a reset, so I'll be fishing and drinking beer all the next week. Will try and fix when I'm back.

Thanks for keeping up the support!!! 😃

Got it working :)
The problem is in the changed json data being returned from the hyprctl j/clients command.
There have been some changes made with the fullscreen so in the client struct (hypr.go) I removed FakeFullscreen and changed fullscreenMode into FullscreenClient. Oh and changed the bool of Fullscreen into an int (not sure why that doesn't just return a true or false)

Got it working :)
The problem is in the changed json data being returned from the hyprctl j/clients command. There have been some changes made with the fullscreen so in the client struct (hypr.go) I removed FakeFullscreen and changed fullscreenMode into FullscreenClient. Oh and changed the bool of Fullscreen into an int (not sure why that doesn't just return a true or false)

Could you show me how to fix it?))

Could you show me how to fix it?))

Sure.

git clone https://github.com/nwg-piotr/nwg-dock-hyprland.git
open the hypr.go file in some editor
in line 62, replace bool with int
save file
cd nwg-dock-hyprland
make build
sudo make install

Got it working :)

Thanks a lot man !!! ❤


Removing secrets doesn't check if secret in use

Description
If you delete a secret, that is still in use, you will get the message, that it was removed successfully instead of still in use.
Steps to reproduce the issue:

Create a secret
Use the secret in a container
Remove the secret
Refresh and the secret is still there

Any other info e.g. Why do you consider this to be a bug? What did you expect to happen instead?
Technical details:

Portainer version: 1.13.2
Target Docker version: 17.03.0-ce
Platform (windows/linux): linux
Command used to start Portainer (docker run -p 9000:9000 portainer/portainer): docker image + -H unix:///var/run/docker.sock
Target Swarm version (if applicable): -
Browser: Safari

@Duvel I was not able to reproduce this, each time I try to delete a secret that is used, an error message is displayed. See the screenshots below.

I tried it today again in 1.13.2 and it still fails. I will try next week if 1.13.4 helps.
The only difference that I can think off, is the different name for a secret.

I tried now in 1.13.4 and it is still failing. Is there any debugging that I could do?

@Duvel you could try to open the developer console of your browser and inspect the requests.
Recording your screen and posting it here would be really helpful too.

Interesting, the response of the server is:
{
"message": "rpc error: code = 3 desc = secret '003456_ts001984_v4.secrets' is in use by the following service: ts001984"
}
So the interface is thinking, it succeeded.

Thanks for the screencast. Indeed, what is the status code of the http request?

The http response is 200.
Just tried in Chrome and the same, I think I also did it in Firefox.

Ah ! Gotta love Docker API. HTTP status 200 + error message. I've tested on docker >= 17.05-ce so they probably fixed that (meaning that they set the status code to > 500). I'll add a workaround that for previous versions.

Glad you found it, I was already expecting something Docker specific.
Thanks for a great tool!

This is pretty strange as I can't even reproduce that in a Swarm cluster using Docker 17.03-ce...
I've created an image based on the fix in #984, can you give a try to portainer/portainer:pr984 (Linux amd64 build)?

I give it a try and it works!


Blocks db

get blocks array to appear on page via index.js

Need to figure out how to get the blocks living in the game object :-/


Project clone

Users should be able to create a copy of a project to which they have at least READER permission. Users should be able to edit all of the fields from workspace creation.
In particular, most of the time, the user is going to change the billing project.
The default name for the clone should be the original workspace name with the prefix "Copy of ..."
Clone should be initiated from within a project via the following icon (already in our UI):

Requires visual design based on project creation


Create management cluster using controller bootstrap task

Issue #, if available:
Description of changes:
Create management cluster using controller bootstrap task
The original https://github.com/aws/eks-anywhere/pull/7110 is being split into smaller PRs. This PR is the first one.
Testing (if applicable):

The whole workflow has been tested end to end for Docker, vSphere, and Cloudstack providers.
Unit tests

Documentation added/planned (if applicable):
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/test eks-anywhere-generate-files-presubmits

/retest


The Kernel appears to have died. Always when I am loading a large file

I am using Jupyter notebooks for creating aML Model with TuriCreate.
The steps that I am following are the above.
I have download a .csv and .json (same file) from https://www.kaggle.com/zynicide/wine-reviews 1
The file has 51 MB size.
I have made an environment turienv From Anaconda Navigation and the following steps work perfect for smaller CSV / JSON files.

source activate turienv
pip install turicreate=5.0
Jupiter notebook

----inside Notebook----
> import turicreate as tc
>
> wine_data = tc.SFrame.read_json('winemag-data-130k-v2.json', orient='records')
>
> wine_data.head() <-- I see that everything is loaded properly
>
> wine_model = tc.text_classifier.create(wine_data,'title',features=['description'])
>
> PROGRESS: Creating a validation set from 5 percent of training data. This may take a while.
>           You can set ``validation_set=None`` to disable validation tracking.
>
> Logistic regression:
> --------------------------------------------------------
> Number of examples          : 123481
> Number of classes           : 113404
> Number of feature columns   : 1
> Number of unpacked features : 21030
> Number of coefficients      :<PHONE_NUMBER>
> Starting L-BFGS
> --------------------------------------------------------
> +-----------+----------+-----------+--------------+-------------------+---------------------+
> | Iteration | Passes   | Step size | Elapsed Time | Training Accuracy | Validation Accuracy |
> +-----------+----------+-----------+--------------+-------------------+---------------------+

And then in about 3-4 minutes I get error msg = The Kernel appears to have died.

Could anyone help?? I am new in Python and Jupyter is only env I have worked with. If there is an other env that I could run the same things with some guidance to have a more solid error msg that I could debug please let me know.
Edited: I am running the above on a 2018 MacBook Pro 16GB 512GB. I see on Activity Monitor Memory for python goes to 130GB and CPU 83%
Thanks in Advance

Hi @grNorway - by default text_classifier.create will train a logistic regression model with a coefficient for each word that is in your dataset at least twice. Sounds like your dataset is too large for this approach.
Take a look at the word_count_threshold in our API documentation. Try setting that to a value larger than two.

I still don't manage to classify the .csv file. Is there a limitation on the size of the file?
I use also drop_stop_words and still I get a huge memory 140GB on python. Any solution how to Classify a csv file in that size ?

@grNorway - What values of word_count_threshold have you tried? I would start by trying a fairly large values (say 100).
Taking another look at the output you shared, it looks like you have 113,404 classes but only 123,481 examples. That's is less than 1.09 examples on average per class. It is very difficult for a model to correctly predict a class, if it only has a small number of examples for that class. Having a large number of classes also increases training time.
I recommend you explore how often classes appear in your dataset. You probably want to remove examples from your training data if their classes only appear a small number of times.


worker.js file should be placed into the output directory when targeting Emscripten with pthread support

Problem
When targeting wasm32-unknown-emscripten with Pthread support, the *.worker.js file is not copied to the output directory from the deps folder, even though the Emscripten compiler generates that file. The worker.js is required by multithreaded wasm apps and is a mandatory output artefact.
Steps
# Create a test package
cargo new --bin issue && cd issue
# Compile with Pthreads support
cargo +nightly \
   -Z build-std=panic_abort,std \
   --config='build.rustflags = ["-C","target-feature=+atomics,+bulk-memory", "-C", "link-args=-pthread"]' \
   build \
      --bin issue \
      --target wasm32-unknown-emscripten
# List output files
find target/wasm32-unknown-emscripten/debug -maxdepth 1 -regex ".*issue.*"

Expected output:
target/wasm32-unknown-emscripten/debug/issue.d
target/wasm32-unknown-emscripten/debug/issue.js
target/wasm32-unknown-emscripten/debug/issue.wasm
target/wasm32-unknown-emscripten/debug/issue.worker.js

Actual output:
target/wasm32-unknown-emscripten/debug/issue.d
target/wasm32-unknown-emscripten/debug/issue.js
target/wasm32-unknown-emscripten/debug/issue.wasm

Note that issue.worker.js is missing in the output directory. However, it is successfully created by the compiler and can be found in the deps folder:
$ find target/wasm32-unknown-emscripten/debug/deps -maxdepth 1 -regex ".*issue.*"

target/wasm32-unknown-emscripten/debug/deps/issue.d
target/wasm32-unknown-emscripten/debug/deps/issue.js
target/wasm32-unknown-emscripten/debug/deps/issue.wasm
target/wasm32-unknown-emscripten/debug/deps/issue.worker.js

Possible Solution(s)
A solution would be to adjust the list of file types returned by file_types() for wasm32 target to include *.worker.js:
https://github.com/rust-lang/cargo/blob/5da28587846d6cb2694e5bb1db1b5ca327285bf7/src/cargo/core/compiler/build_context/target_info.rs#L405
Notes
No response
Version
cargo 1.77.1 (e52e36006 2024-03-26)
release: 1.77.1
commit-hash: e52e360061cacbbeac79f7f1215a7a90b6f08442
commit-date: 2024-03-26
host: aarch64-apple-darwin
libgit2: 1.7.2 (sys:0.18.2 vendored)
libcurl: 8.4.0 (sys:0.4.70+curl-8.5.0 system ssl:(SecureTransport) LibreSSL/3.3.6)
ssl: OpenSSL 1.1.1w  11 Sep 2023
os: Mac OS 14.3.1 [64-bit]

Thanks for the report!
Could you share some more specifications and documentations of that output behavior of emscripten? I found one here, but not sure if it always does that. worker.js could possibly be considered as an auxiliary file and uplifted for empscripten target, though we might need to understand the stability before moving forward. We might also bump into issues like https://github.com/rust-lang/cargo/issues/13355 and https://github.com/rust-lang/cargo/issues/13358 when dealing with it.

Hi @weihanglo
Thanks for a quick response! In the Special Considerations sections it mentions that the worker.js will be always generated when compiling with -pthread flag and the file must be deployed:

Also note that when compiling code that uses pthreads, an additional JavaScript file NAME.worker.js is generated alongside the output .js file (where NAME is the basename of the main file being emitted). That file must be deployed with the rest of the generated code files.

In addition, -pthread is not the only case when Emscripten generates a worker.js. According to the emcc compiler manual , when linking with --proxy-to-worker flag it will also generate worker.js when emitting JavaScript. Cargo always emits JavaScript and never targets HTML when building for  wasm32.

--proxy-to-worker
[link] Runs the main application code in a worker, proxying events to it and output from it. If emitting HTML, this emits a .html file, and a separate .js file containing the JavaScript to be run in a worker. If emitting JavaScript, the target file name contains the part to be run on the main thread, while a second .js file with suffix “.worker.js” will contain the worker portion.

Emscripten choses what to emit based on the suffix of the output file: .js for JavaScript and .html for HTML. The following condition in cargo guarantees that Emscripten will emit JavaScript (due to .js suffix) and therefore we get a worker.js file.
https://github.com/rust-lang/cargo/blob/5da28587846d6cb2694e5bb1db1b5ca327285bf7/src/cargo/core/compiler/build_context/target_info.rs#L405
Additional files

BTW, are there any more files for emscripten that we want to uplift to the target directory?

Yes indeed. Those files are: .data, .data.js and .mem. See Build Files. data related files are the important ones. They contain preloaded files and directories that will populate the virtual file system. See Packaging Files. It is difficult to envision an app without asset files. Please consider outputting them as well.
Cheers

Hey. Sorry for the delayed response.
The Cargo team discussed this during today's meeting. The general idea here is that we are okay for adding worker.js as an additional auxiliary files, since we already have special cases for something like .wasm.map.. For any other files at this moment, I'll encourage using --message-format=json to access artifact messages and process what you need if possible.
Just note that, due to the fact that wasm32-unknown-emscripten is a tier-2 target, as well as we lack of CI infra to test it, Cargo would provide its best-effort for the proposed behavior. There is no strong guarantee like tier-1.
In a better future, we'd like to see rustc informing Cargo about the files it emits, so that Cargo doesn't need to hard-code all of them. Thank you.

A general note for the future, keep in mind of this PR that recently was merged in Emscripten.
With the master branch Emcscripten, *.worker.js file generates the following,
// This file is no longer used by emscripten and has been created as a placeholder
// to allow build systems to transition away from depending on it.
//
// Future versions of emscripten will likely stop generating this file at all.

Thanks @charmitro for the updated info!
Given *.worker.js is no longer useful, I propose to close this issue.

I generally agree. However, is there a way to check which version is installed and, based on that, generate *.worker.js, at least until Emscripten completely removes this feature?

To me, I won't bother that. Emscripten is not a tier 1 platform and user can always download the latest version of it. The maintenance cost might not be worthy for a deprecated-soon-removed feature. And we don't have proper CI infra to test that. Also, I don't think there is a safe and stable way to get the version info. Invoking emscripnt is probably the straightforward one, and Cargo should invoke external binaries as less as possible.

Makes sense, you're right. I completely agree with not moving forward with this.

Hi
What a timing of the emscripten change! Thanks @charmitro for digging it up. I think we an close the issue 👍
Cheers


GC overhead limit when mining wikipedia and extracting anchor text

Hi
I am following the steps provided here to train my model.
I have pre-processed the datapack. But when I am trying to "Build Data Structures and extract anchor text", I am having this GC overhead issue.

I have even increased the MAPRED and HADOOP memory to 15G and even provided opts for
Dmapreduce.reduce.java.opts and Dmapreduce.reduce.memory.mb
My system has 8 cores 32 GB, using java 8.
Pardon me as I am novice to hadoop and java

@aasish Could you please comment as to how should I resolve this?

FYI, I solved the issue with this shell script. README needs to be updated.


Resolve problem with sqlite3_win32_set_directory for wasm

See dotnet/aspnetcore#39825 for more info.
The problem is that sqlite3_win32_set_directory() only appears in the native SQLite library when compiled for Windows.
In the case of a typical DllImport, this doesn't cause much trouble, because it results in a failure at runtime.
In AOT scenarios, the situation is worse, since the symbol is missing, so it fails at build time.
In provider_internal, which we assume is iOS-specific, and is therefore an AOT scenario, we have a special flag we use when generating that provider to make sure the actual native function is not referenced, so we don't get a build-time failure.
Wasm is a similar situation.
There are basically two ways we could fix this:

Use something similar to what we do with provider_internal, creating a special build of provider_e_sqlite3 and provider_e_sqlcipher (any others) which do what provider_internal does.  This approach would require us to handle RID-specific builds of those providers.
Use something similar to what we do for sqlite3_key and friends, putting instances of those functions in a stubs.c file in the native build (over in the cb repo).

The unfortunate thing about RID-specific builds of provider_e_sqlite3 is that I recently just changed all the provider builds to use multi-targeting and to no longer need a separate nuspec file.  This simplified the tree quite a bit, and it seems like it would be SO SAD to undo all that.
What we would need is to say "build this assembly twice for the same TFM (which multi-targeting can't do), and when creating the nuspec for pack, put one of them in the usual place in lib/TFM, and put the other one in the RID-specific area".  There is, AFAICT, no way to express what we want in csproj properties for use with dotnet pack without a separate nuspec.

@bricelam Over in the other issue, you said:

The RID-specific assembly is more work, but would probably be the better option.

Initially I agreed with that, but now I want to push on it a little bit.
So, er, WHY would it probably be the better option?
I wish to highlight the fact that both approaches accomplish the same result, which is to make sure that sqlite3_win32_set_directory() returns an error code.  Either way, that function is a member of ISQLite3Provider.  We're just deciding whether we want SQLITE_ERROR to be returned in C# or C.
And right now, doing the work in stubs.c is a heckuva lot simpler.
One big unknown here is the question of whether I will eventually need RID-specific assemblies anyway.  If the pain is coming no matter what, there may be less incentive to avoid it now.  OTOH, I suppose maybe stalling might pay off if "dotnet pack without nuspec" later gains the ability handle more advanced scenarios.

The only real advantage I see is that users may want to bring their own custom build of SQLite on Wasm. If you go with the stubs.c approach, they'll just have to know to include that in their custom builds. But I can't imagine that being very common, so it's probably OK to take that approach too.
I also see a world where AoT becomes more and more popular. For example, creating an AoT version of your ASP.NET Core application to deploy and run inside Docker. But that's a bit forward thinking, and we can always solve any problems with it as they arrise.

Good observations.  I'll think about this a bit more.

To get it work pls. add this line to your project:
-s ERROR_ON_UNDEFINED_SYMBOLS=0

To borrow another quote from @bricelam over in the issue where this started:

We could even reverse it and make the one with sqlite3_win32_set_directory specific to windows.

I do think the best way forward is likely to go back to treating Windows as the special case here.  I mean, not only is this API call specific to Windows, the most common use case is UWP, which is an even more specialized case.
I recently changed the providers to build with multi-targeting, and I hope to preserve that.
In that transition I also changed to use dotnet pack without a side nuspec file, and I'd like to keep that too.  Earlier today I made a commit which demonstrates a way to use an advanced feature of dotnet pack to put an assembly in runtimes/rid/etc/.
For the 4 main DllImport providers (e_sqlite3, e_sqlcipher, sqlite3, sqlcipher,), there are basically two switches we can turn on and off when we generate the C# code for that provider:

Whether to use (funcptrs) C# 9 function pointers or (prenet5) the older way, delegates with UnmanagedFunctionPointerAttribute.
Whether sqlite3_win32_set_directory should (win32dir) try to call the native code or (notwin) just return an error.

(funcptrs) requires net5.0 or higher.
(win32dir) is only useful on Windows.  On non-Windows, non-AOT platforms, it has apparently been harmless, simply returning an error at runtime if something happens to call it.  But it actually causes problems on any non-Windows platform that uses AOT, because the missing function will cause a linker failure.
As part of the recent changes, I am generating two versions for each of those 4 providers:

A version called "most", which has (prenet5) and (win32dir), and which is built for the TFM netstandard2.0
A version call "funcptrs", which has (funcptrs) and (win32dir), and which is built for the TFM net6.0

In other words, almost all providers include (win32dir).  The only provider currently compiled for (notwin) is the __Internal provider, since it is for iOS which is AOT.
So the approach I am investigating is to change all these providers to (notwin), and then add back special cases with (win32dir) where needed.
Hopefully while keeping multi-targeting and dotnet pack without side nuspec, and hopefully not mixing up UWP and other kinds of Windows like I did in 2.0.7 (see discussion in #465).
Details yet to be figured out.

It is worth noting that the winsqlite3 provider is separate, and it is generated with support for sqlite3_win32_set_directory.

Now it works without the "-s ERROR_ON_UNDEFINED_SYMBOLS=0" switch, thank you. One final warning remains:
Warning : Found a native function (sqlite3_config) with varargs in e_sqlcipher. Calling such functions is not supported, and will fail at runtime. Managed DllImports:
    System.Int32 sqlite3_config_none(System.Int32) (in [SQLitePCLRaw.provider.e_sqlcipher] SQLitePCL.SQLite3Provider_e_sqlcipher+NativeMethods)
    System.Int32 sqlite3_config_int(System.Int32, System.Int32) (in [SQLitePCLRaw.provider.e_sqlcipher] SQLitePCL.SQLite3Provider_e_sqlcipher+NativeMethods)
    System.Int32 sqlite3_config_int_arm64cc(System.Int32, System.IntPtr, System.IntPtr, System.IntPtr, System.IntPtr, System.IntPtr, System.IntPtr, System.IntPtr, System.Int32) (in [SQLitePCLRaw.provider.e_sqlcipher] SQLitePCL.SQLite3Provider_e_sqlcipher+NativeMethods)
    System.Int32 sqlite3_config_log(System.Int32, System.IntPtr, SQLitePCL.hook_handle) (in [SQLitePCLRaw.provider.e_sqlcipher] SQLitePCL.SQLite3Provider_e_sqlcipher+NativeMethods)
    System.Int32 sqlite3_config_log_arm64cc(System.Int32, System.IntPtr, System.IntPtr, System.IntPtr, System.IntPtr, System.IntPtr, System.IntPtr, System.IntPtr, System.IntPtr, SQLitePCL.hook_handle) (in [SQLitePCLRaw.provider.e_sqlcipher] SQLitePCL.SQLite3Provider_e_sqlcipher+NativeMethods)

The fix for this is in 2.1.0, which has now been released.


Repair and Repair All Issue

The player repair button resets the level of the player and the pals he carries with him to the first level. Is it problem?

Thats because u have duplicate player uid, u can use delete player with instance id on the interactive mode to remove duplicated player first


Fix peek function of accountQueue in txpool

Fix peek function of accountQueue in txpool
Description
this check isn't required in account queue as it will be done in min nonce queue
https://github.com/0xPolygon/polygon-edge/blob/33e09676d95bf2a2d714bfa02a4a17235f56ed68/txpool/queue_account.go#L81

Could you @sarvalabs-karthik please clarify what issues it causes?

its just a performance issue. it may be negligible. but a redundant check.