Distributed training with synchronized SGD using 'grpc+verbs' sometimes hangs indefinitely

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes.
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04.2 LTS
TensorFlow installed from (source or binary): Source
TensorFlow version (use command below): v1.2.0-1755-gee4259a 1.2.1
Python version: Python 3.5.2
Bazel version (if compiling from source): 0.5.1
CUDA/cuDNN version: 8.0/5.1
GPU model and memory: GeForce GTX 1080 Ti
Exact command to reproduce: Sorry, the command is not available since this is custom code
RDMA driver version: libibverbs-dev (1.2.1mlnx1-OFED.<IP_ADDRESS>.3.40200)

Describe the problem
I am training a seq2seq model with synchronized SGD (using tf.train.SyncReplicasOptimizer and tf.train.Supervisor), over RDMA (using grpc+verbs protocol), on 2 workers and 1 parameter server. Each worker has 8 GPUs, and the model on each GPU is the same.
I can train this model fine with the default grpc protocol using the same setting. When I switched to grpc+verbs, the behavior becomes unpredictable. Most of the times both workers hang (for at least 12 hours, so not because I didn't wait long enough). Sometimes the chief worker (worker 0) starts training, but worker 1 hangs. (This should not happen since I am programming with synchronized SGD. Worker 0 should be blocked if worker 1 is not ready.) In rare case it can go through and start training. I believe the RDMA driver is correctly installed.
When worker hangs, the CPU utilization stays low but not zero, and the GPU utilization is 0.
Source code / logs
I attached below the log from worker 1 worker1.txt. In this case, worker 1 hangs, and worker 0 starts training. (There are some custom logs printed.) In the log you can find that worker 1 sent a RDMA_MESSAGE_TENSOR_REQUEST to the parameter server, but never got an ACK, and I have checked the log from the parameter server, which showed that it never received a RDMA_MESSAGE_TENSOR_REQUEST from worker 1.
I also attached the full trace gdb_worker1.txt collected on worker 1 in gdb with thread apply all bt, during worker 1 hanging.
Thanks.

@junshi15 Could you shed some light on this? Thanks!

Have you tried your code in loopback mode? It helps to make sure that the problem doesn't lie in your network fabrics configuration.

@byronyi Thanks for the quick reply. I have tried my code over Ethernet, and it worked fine. I think that should indicate the network fabrics is fine. Also, one experiment I tried over RDMA did go through as I described in the issue. So RDMA fabric seems fine. The loopback mode won't traverse the network stack, right?

Note that RoCE requires a lossless fabric, while gRPC does not. If you could reproduce the error in loopback mode, it would be easier for us to identify the possible issue.

@on-the-run, can u verify the RoCE configuration with ib_write_bw between the 2 nodes ?
As @byronyi said, you may need to configure lossless (PFC) or ECN to prevent the packet loss.
ib_write_bw is a good stress test to see if RoCE is configured good.
Can u attach the full logs of the PS and worker ? I'm missing a lot of events here in the logs.

@shamoya, the RDMA I am using is Infiniband, instead of RoCE. ib_write_bw runs fine. The workers and the parameter sever can talk to each other via RDMA. Posting the output from running ib_write_bw between one worker and one parameter server below.
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF          Device         : mlx4_0
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 CQ Moderation   : 100
 Mtu             : 2048[B]
 Link type       : IB
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x15 QPN 0x32da PSN 0xebb028 RKey 0xa0010194 VAddr 0x007f11bbfc2000
 remote address: LID 0x12 QPN 0x0337 PSN 0x198251 RKey 0xd001018f VAddr 0x007f728f89f000
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 65536      5000             6071.23            6071.03            0.097136
---------------------------------------------------------------------------------------

@shamoya, about the logs of PS and worker, I didn't turn on VLOG for the log file I posted earlier. will a log with VLOG(2) enabled useful?

If it's IB then we shouldn't have any packet loss.
VLOG(2) is maybe too much verbose, maybe just change all VLOG in rdma.cc to 0, so we can just see the Rendezvous protocol messages.
How many model parameters are there ?

@on-the-run Did you even run an example successfully, say InceptionV3 training on ImageNet dataset?

@junshi15, Yes. The same code did go through once on RDMA, and it worked well at that time. I didn't try InceptionV3, though.

that's strange. I have not seen issues like that, Your problem probably depends on the ordering of events, which is random. I am wondering if there is memory corruption somewhere.

@junshi15, one thing that I observed to happen repeatedly was that after the RDMA channels have been built, worker 1 sent a RDMA_MESSAGE_TENSOR_REQUEST  to PS, and never got an ACK. Then worker 1 was blocked there. And, PS never received that RDMA_MESSAGE_TENSOR_REQUEST . Does this ring a bell?

@shamoya, there are about 70 million parameters. Here are full logs:
ps.txt,
worker0.txt,
worker1.txt
In this case, worker 1 hanged, but worker 0 didn't. Not sure if the log of worker 1 gives enough information. Let me know if you need me to turn on other logs. Thanks.

does distributed tensorflow using 'grpc+verbs' have unit tests? I didn't find any. I meet 2017-07-11 09:46:46.591411: F tensorflow/contrib/verbs/rdma.cc:95] Check failed: cq_ Failed to create completion queue  when using 'grpc+verbs' in an ib network.

@on-the-run RDMA_MESSAGE_TENSOR_REQUEST is a fairly simple message. Here are some possibilities why PS did not receive it.

the message was not sent by worker. Note we only enqueue the message. The actual RDMA write happens here, by logging more in that function, we will know if the message is indeed written.

the message was not received at PS. Here is how we get notification of events. I do not expect and missed events, but maybe there are corner cases...
My focus would be on (1) since it is easier to verify. If in doubt, You can also check if the worker's tx_message_buf and PS' rx_message_buf addresses are paired, so that messages go to the right buffer.

@suiyuan2009 It's a good idea to add unit tests. right now there is none. Your error is likely due to hardware config. For example, the IB adapter is down. One situation I have seen is some people have multiple IB adapters, let's say the first one is down, the second is up. The current code opens the first device, which is down, hence the failure.

@junshi15 , it's seems user and root have different limit in my vm, I use root to run tf then it works.

@junshi15, Thanks for the suggestion. For 1, I am pretty sure that the RDMA write from worker succeeded, or at least didn't return non-zero value. The evidence is that this line went through, and I did see the code in ibv_post_send() (defined in infiniband/verbs.h) was executed. I will try to look at 2 and 3.

I meet stuck problem twice today when using grpc not grpc+verbs.

@junshi15, some updates about (2) and (3). I am attaching new logs, which give more information on RDMA related events, but fewer lines. In this case, both workers hang.
ps.txt
worker0.txt
worker1.txt
For (2), I believe the message is not received by PS, but don't know why. In the logs you can find the buffers are correctly paired.
For (3), I don't think there is a hashing collision, since not even one tensor had been successfully sent yet in the logs attached.
Could you shed some light on the following questions?
(1) Which part of the code send the RDMA_MESSAGE_ACK? Why did I see RDMA_MESSAGE_ACK received in rx_message_buffer?
(2) What will happen if all the outstanding SendNextItem() end up with doing nothing (which seems to be the case shown in the logs)?
(3) At line 87 and 88 of ps.txt, PS received a RDMA_MESSAGE_ACK from worker 1. This doesn't make sense since PS had never sent any message to worker 1, nor worker 1 had received any message from PS.

@on-the-run, Looking at the ps log, something doesn't make sense to me.
The second call of RecvFromRemoteAsync is for Tensor<PHONE_NUMBER>48928.
The thread (coming from RecvFromRemoteAsync) then calls EnqueItem (with a lock) and then SendNextItem which doesn't do Write since local and remove are busy.
But this tensor should stay at the top of the queue, and the next SendNextItem should send it, but that doesn't happen (same for the third tensor -<PHONE_NUMBER>69888)
After the RDMA_MESSAGE_ACK is received (which set remote = idle and calls SendNextItem) the SendNextItem takes Tensor<PHONE_NUMBER>64448, and I don't understand why ?
Is it possible your print change the locking scheme of the queue ?

@shamoya, My tid actually means thread id. Sorry for the confusion. I think PS finally sent three RDMA_MESSAGE_TENSOR_REQUEST, which matches the number of calls of RecvFromRemoteAsync . But I did added some VLOG inside lock surrounded code. Will that break something?

@shamoya, I just noticed this odd behavior, too. It seems that somehow the code got a RDMA_MESSAGE_TENSOR_REQUEST  but thought it got a RDMA_MESSAGE_ACK . Note that in the log PS also received a RDMA_MESSAGE_ACK  from worker 1, but worker 1 only sent a RDMA_MESSAGE_TENSOR_REQUEST .
I am running it on x86_64. How can I check the endian issue? Is endian handled by the Infiniband driver?

I don't have an idea of the acutal issue, but as far as I know the immediate data is in network byte-order. You should use wr.imm_data = htonl(val) and imm_data = ntohl(wc.imm_data) to store and load the immediate data.

@byronyi @shamoya, Does that mean the code here should be converted with htonl(), before sending to the RDMA hardware?

See here and there for an example.
And yes, but you need to change save/load together.

@shamoya @byronyi, Yes. It sounds like we found something! Let me try the change and get back. Thanks!

@on-the-run
some high-level stuff:
there are three types of buffers, message/tensor/ack, each of them has tx and rx version. "tx" is for sending and rx is for receiving. tensorbuf is for tensor exchange, messagebuf is for message and all acks are on ackbuf. Since there is one one pair of message buffer and one pair of ack buffer, all the messages needs to be sent one by one. So are all the packs.
When a message is sent from A to B, in most cases, B needs to send an ack, hence if you look at A's log, you have "sent RDMA_MESSAGE_XXXX", then "received RDMA_MESSAGE_ACK". In B's log, you should have "received RDMA_MESSAGE_XXX", then "sent RDMA_MESSAGE_ACK"
it seems all nodes sent "RDMA_MESSAGE_TENSOR_REQUEST" and received "RDMA_MESSAGE_ACK". I don't see any log from B's perspective. this is odd. A logging issue?
To answer your questions:

Mostly in Process_CQ(), e.g. https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/verbs/rdma.cc#L154

buffers are busy. Remember all the messages has to be sent sequentially. A message can be sent only if both local tx-buf and remote rx-buf are idle. I think worker1 was stalled there.

I don't see any "sent RDMA_MESSAGE_ACK". logging issue?

Actually I'm not sure this is the cause, the imm_data is the same one sent from the PS and received by worker0 (986591585). Nothing would have it this was the case.
Maybe the NameHash function acts different between the 2 nodes ?
in PS this hash is for "rx_message_buffer" (RdmaMessageBuffer::SendNextItem) but somehow it worker0 the FindBuffer acts differently ?

Thanks @junshi15
In (1) he meant if the type is recv RDMA_MESSAGE_ACK, then rb (derived from imm_data) must rx_ack_buffer (which isn't the case here for some reason).

@on-the-run - same TF version / dependent python packages on all sides ?
Important to mention I don't see this issue.

@shamoya, Yes, everything is the same on PS and workers: TF version, CUDA drivers, IB drivers, etc. I launch them with the same docker image.

@shamoya, I have checked that behaviors of NameHash() and FindBuffer() are consistent across PS and workers.

I see.
Then what's happening here is very strange.
I see also my previous claim is wrong since the imm_data is correct and also the RDMABuffer (rx_message_buffer). so the only think I can think of is the buffer of rx_message_buffer is corrupted/not read well by ParseMessage.

@on-the-run, It is strange I don't see any log about "RDMA_MESSAGE_ACK sent" and "RDMA_MESSAGE_TENSOR_REQUEST received".
Did you log anything inside RdmaAckBuffer::SendNextItem()?
Also, I see many occurrence of recv RDMA message: RDMA_MESSAGE_ACK, imm_data recevied = 986591585. I assume 986591585 is NameHash('rx_ack_buffer'), can you confirm?

@junshi15, I did log inside RdmaAckBuffer::SendNextItem(). It was jut not triggered. 986591585 is the hash of "rx_message_buffer" instead of "rx_ack_buffer". That's why I can't understand why the RDMA_MESSAGE_ACK was sent to rx_message_buffer. Is it possible that somehow the code misunderstands RDMA_MESSAGE_TENSOR_REQUEST  to be RDMA_MESSAGE_ACK? Since that assumption can explain the strange log events.

@shamoya, RDMA driver version: libibverbs-dev (1.2.1mlnx1-OFED.<IP_ADDRESS>.3.40200).

@on-the-run - it's just the version VERBS library, I would like to know also the kernel drivers of the HCA and also the FW version. please attach the output of:

ofed_info -s
ibv_devinfo

Also what do u think of my guess (all received buffer is actually 0 - and then it's parsed as RDMA_MESSAGE_ACK) ?

@shamoya, sorry, I am not so familiar with these stuff.
Output of ofed_info -s:
MLNX_OFED_LINUX-4.0-<IP_ADDRESS>:

Output of ibv_devinfo:
hca_id: mlx4_0
        transport:                      InfiniBand (0)
        fw_ver:                         2.40.7000
        node_guid:                      248a:0703:00bd:5740
        sys_image_guid:                 248a:0703:00bd:5743
        vendor_id:                      0x02c9
        vendor_part_id:                 4099
        hw_ver:                         0x1
        board_id:                       MT_1100120019
        phys_port_cnt:                  1
        Device ports:
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 1
                        port_lid:               18
                        port_lmc:               0x00
                        link_layer:             InfiniBand

I think your guess makes sense. I am trying to log all the values of each work completion event. Will get back.

We are using mlx5_0, if it matters.
hca_id:	mlx5_0
transport:			InfiniBand (0)
fw_ver:				12.1100.6630
node_guid:			e41d:2d03:0067:c9fc
sys_image_guid:			e41d:2d03:0067:c9fc
vendor_id:			0x02c9
vendor_part_id:			4115
hw_ver:				0x0
board_id:			MT_2180110032
phys_port_cnt:			1
Device ports:
port:	1
state:			PORT_ACTIVE (4)
max_mtu:		4096 (5)
active_mtu:		4096 (5)
sm_lid:			2
port_lid:		3
port_lmc:		0x00
link_layer:		InfiniBand

@byronyi Thanks for raising the endian-ness issue. It will probably improve the portability.
If one of the nodes in the cluster is linux (little endian) and another is hp-ux (big endian), for example, then there may be problems. If the cluster is homogeneous, then there should be no issue.

@junshi15, when parsing a RdmaMessage with ParseMessage(), does it need to be protected by lock?

The message and tensor rx_buffers are read-only locally and protected  at the remote end by "remote_status_". I don't see same protection on the ack rx_buffer.
I need to think about the reason why ack buffers are treated differently.

@shamoya, I have checked the work completion events. It looks fine to me. Most importantly, the imm_data is correctly sent. So it is not an endian issue. But the endian should be considered as it suggests here in infiniband/verbs.h:
 struct ibv_wc {
     uint64_t        wr_id;
     enum ibv_wc_status  status;
     enum ibv_wc_opcode  opcode;
     uint32_t        vendor_err;
     uint32_t        byte_len;
     /* When (wc_flags & IBV_WC_WITH_IMM): Immediate data in network byte order.
      * When (wc_flags & IBV_WC_WITH_INV): Stores the invalidated rkey.
      */
     uint32_t        imm_data;
     uint32_t        qp_num;
     uint32_t        src_qp;
     int         wc_flags;
     uint16_t        pkey_index;
     uint16_t        slid;
     uint8_t         sl;
     uint8_t         dlid_path_bits;
};

It looks to me now a memory corruption issue.

@on-the-run
I refreshed my memory. Here is how the buffers are protected. I will focus on rx buffer which is protected by its remote side (tx side).
Rx_TensorBuffer:
When we send a tensor via TensorBuffer, we need to make sure both the Tx_TensorBuffer (at the sender) and the Rx_TensorBuffer (at the receiver) are idle. This is achieved by checking [local_status_ and remote_status_] (https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/verbs/rdma.cc#L778) at the sender under a lock. When the tensor is received, the receiver send a RDMA_MESSAGE_BUFFER_IDLE message to release remote_status_, while local_status_ is released at the sender.
Rx_MessageBuffer:
Similar to Rx_TensorBuffer, both local_status_ and remote_status_ flags are checked. They are turned to busy once transmission starts. When the receiver reads the buffer, it sends an ACK to notify the sender to release remote_status_.
Rx_AckBuffer:
No protection at all. The sender does not check AckBuffer before sending an Ack. This is OK because (1) all the acks are sent (ab->SendNextItem()) by one thread inside Process_CQ().
(2) messages and acks are interleaved. Let's say A wants to send multiple messages to B. After the first message is sent, A has to wait for the ack before sending the next message, i.e. this is the only place tx_message_buffer_.remote_status_ becomes idle. Since A can not send multiple messages consecutively, B can not ack consecutively. No risk of multiple ACKs stepping on each other.
But I afraid I may have missed corner cases. This messaging system is very fragile. If one message, even an ACK, is missing, you will likely get a hang.
To further debug, I suggest [the following change]:(https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/verbs/rdma.h#L55-L60). Switch RDMA_MESSAGE_ACK  with RDMA_MESSAGE_TENSOR_WRITE. If indeed all message types are zero due to memory corruption, you should see all RDMA_MESSAGE_TENSOR_WRITE in your log.

@junshi15, Thanks for the explanation. I think memory corruption is confirmed. I switched the order of RDMA_MESSAGE_ACK  and RDMA_MESSAGE_TENSOR_WRITE. Now the first message received by worker 0 is indeed RDMA_MESSAGE_TENSOR_WRITE, which should be the RDMA_MESSAGE_TENSOR_REQUEST sent by PS. Worker 0 later failed at the CHECK here. Need to figure out why memory is corrupted.

@on-the-run Thanks for the info. I did not see any obvious reason. it may require a serious debug.

Hi @on-the-run - can u print the buffer (not the completion buffer) ?
I want to see if it's all 0's.
I'm not sure why we don't see the issues u r seeing .. looks like something severe in your case.

Hello @shamoya, I am working with @on-the-run on this issue. One important fact we observed is that with only one GPU per server, it will run without a problem. However, with increasing number of GPUs per server, the more likely it will get stuck. With eight GPU, it rarely runs.
Since our logs show that 1. the (virtual) memory addresses are correctly mapped and exchanged, 2. the NIC reports RDMA WRITE is successful, 3. userspace sees the target buffer remains unchanged (we try initializing rx_message_buffer with 0 or different values, any values remain unchanged), we suspect that the problem is because the mapping from virtual addr to physical addr has changed.
We suspect that this is related to we are using CUDA 8 and Pascal GPU, with which Nvidia has a new memory management feature (https://devblogs.nvidia.com/parallelforall/cuda-8-features-revealed/)
This is what may have happened (our speculation):

ibv_reg_mr --> NIC remembers the physical addr
CUDA maps the eight GPUs' memory to virtual addr, somehow changed the userspace virtual addr to physical addr mapping.
RDMA transmission starts, NIC thinks it has written to the physical addr, but when the tensorflow process checks the memory contents, the process is actually looking at a different physical addr.

Our current workaround is to call mlock() right after malloc in rdma.cc. It tells the OS to pin the physical memory. We have run our program successfully for a few times since the change. We'll continue checking whether it really solves the problem.
I am not sure which is the right side to blame -- we don't expect CUDA to change the virtual addr mapping, however it seems that it does, especially when it works aggressively (with eight GPU). We expect ibv_reg_mr to pin the memory, however we searched through the ibv_reg_mr implementation in OFED and didn't see it did that.

Maybe I have some fundamentally wrong assumptions about how memory registration works in verbs, but shouldn't the IB driver pin the memory pages in kernel space, as verbs user space library will send a command in ibv_cmd_reg_mr?
The user space library is not supposed to pin the user pages in user space API, like mlock(2).

@byronyi, If I am looking at the correct place, I didn't find any memory pinning done inside ibv_cmd_reg_mr().

It's inside the providers directory, e.g. mlx4_reg_mr

@byronyi, I believe both mlx5_reg_mr() and mlx4_reg_mr() call ibv_cmd_reg_mr(), but there seems no memory pinning done.

@byronyi, Personally, I used to believe that too. However, @on-the-run and I both searched in the code and didn't see the memory pinning. In the past, I have done very heavy hacking in the verbs libs, as heavy as rewriting most of the verbs code. Still, I cannot find it.
However, I am not sure what happens after ibv_cmd_reg_mr (I believe this is the last step in userspace. mlx4_* calls ibv_cmd_*) passes the cmd to underlying driver. Mellanox @shamoya should be able to tell us more about this.

All the buffers are allocated in system memory, so CUDA8 should not play a role here. My impression is that ibv_reg_mr does pin the memory, although I am not familiar the source code. @shamoya may be able to explain it.

@junshi15 I understand that you only allocate main memory. Nevertheless, CUDA starts after your code, and with Pascal GPUs, it maps GPU memory to the same virtual memory address space as the process (see Unified Memory in https://devblogs.nvidia.com/parallelforall/cuda-8-features-revealed/). We do not know what it does with the existing virtual addr and physical addr mapping.
Otherwise, I don't know how to explain why it works with one GPU, while starts to break with more GPUs..

Maybe I didn't make it clear, I mean the verbs library is not doing nor supposed to do memory pinning in user space: that job is solely done by the in kernel part of your IB driver. Thus you should search it in the kernel source tree and kernel memory pinning API, something like get_user_pages, not the syscall nor user space API like mlock.

@bobzhuyb And I think I have checked with TF people before, and they have made it pretty clear that TF is not designed to utilize UVM nor support this feature. See relevant discussion in here. I also tend to believe neither IB driver or CUDA driver should alter memory mappings managed by the other one, as we are not doing peer memory access for NIC and GPU.
On the other hand I have worked with TF memory allocation subsystem and they are doing a pretty good job on keeping track of CUDA host/device memory allocation. See relevant details for host and device memory. And if you adjust those parameters (like the initial memory allocation or allow_growth=False), you could force it to allocate all CUDA memory at very early stage (before the whole distributed runtime is even initialized). After that, the virtual memory mappings of these memory regions should not be altered. If you do that, the step 2 of your speculation is rather unlikely to happen.
Being said that, the is a possibility that something is wrong in either CUDA or IB driver, but our chances are small. cc @zheng-xq @poxvoculi any comments on this?

@on-the-run @bobzhuyb, the assumption doesn't make sense to me.
The memory is being pinned in https://github.com/torvalds/linux/blob/master/drivers/infiniband/core/umem.c#L83 , which is executed in the ibv_reg_mr flow, so mlcok shouldn't have any effect.
Also as @byronyi (this is what I know as well) said, TF doesn't support UVM.
Are u sure this is the only thing u changed that fixed the bug ?
I'm having a similar issue with 8 P100 per node (4 nodes), happens with only certain conditions (tf_cnn_script benchmark with "real" data) and the pinning issue is not the case.
I think maybe found the issue (checking now and will report soon).

@byronyi @shamoya Thanks for the clarification. Yes, mlock() was the thing that made it stable in our environment. Good to know that you can reproduce the issue. Please keep us posted.

Hi @bobzhuyb,
The mlock solution still doesn't make sense to me.
it shouldn't have any effect, unless you a severe driver issues.
Can u involve RDMA community in this issue in https://github.com/linux-rdma/rdma-core ?
Anyway my issue still not solved, I opened an issue in https://github.com/tensorflow/tensorflow/issues/11725.
Any help is appreciated, if u can take a look.
@junshi15 @byronyi if u have time I would be delighted as well.

Hi @on-the-run,
Is there any way to know if the CUDA remaps the memory mapping of the process ?
As said, I feel we see a similar issue, but the mlock workaround doesn't help in our case.
We also feel it's something to do with Pascal memory management (can't say exactly what).
Any clues to what we can check ?
Thanks !

I was using K80 and CUDA-7.5, and was able to reproduce the other issue (#11725). So at least that issue is not only limited to Pascal.

We don't have a concrete evidence. However, we did a basic test -- when we malloc the receiver buffer, we init the buffer with a special value as a signature (say every byte initialized as 0xab). After the successful receive event triggered by WRITE_WITH_IMM, we immediate check the receiver buffer value. It remains unchanged in TF process.
If you use ibdump at the receiver side, you should see the special signature in the packets. So the data must have arrived. Would you do the same as above to confirm that we are seeing the same (or similar) problem?
There are only two potential explanations,

Receiver NIC didn't DMA the data into the receiver buffer -- we don't see why this may happen
NIC did write the data to physical memory, but the vaddr mapping has changed

Getting a concrete evidence requires some kernel hacking to get the vaddr to physical addr mapping at different stages.

@shamoya, unfortunately I am not an expert on CUDA. Also, my colleague @bobzhuyb has taken this issue over. @bobzhuyb, could you give some input when you have time? Thanks!

Hi @bobzhuyb, I made a separate RDMA patch #11392 that supports checksum with VLOG=2. Maybe you could give it a try and see wether the problem persists.


thanks

helped a bunch!

Yes, thanks a bunch!  I was trying to make my own and found it so difficult to track down all of the old aliases, so I'm very glad I found this!


fix: don't fail empty test suite with passWithNoTests (fix #3779)

Fix #3779

Current behavior is intended.


Audio from Webcam under Windows

I try to stream my laptop webcam under Windows so I use a config as the following:
streams:
  windows_webcam:
    - ffmpeg:device?video=0#video=h264
http:
    listen: ":1984"

The video is streamed well but no sound. What should I do to hear audio?
And tell me please if it's possible to record the streamed video/audio?
Sorry for the beginner's questions and thanks for the amazing project.

Web UI > Add > FFmpeg Devices


404

https://github.com/aiogram/bot#how-this-bot-is-deployed-now
here we have the wrong link

I know it. And that OK. Thanks for bothering.
Home page of https://illemius.xyz is placeholder and temporary removed.
All other services specified in list works perfectly on subdomains in the same cluster.


[INTERNAL] Replace use of whitelist term

Coverage remained the same at 91.432% when pulling 7959d091010baecb0e90cc3c885726ef69f91956 on replace-whitelist-term into 7917d4a27768a5f0759fa71cabd7c87056205a8f on master.


[Feature] Admin can remove user's 2 factor

Proposal
This allows the admin to see if the user requesting two factor authentication removal has it, and if they do, remove it. ADM-5
Repercussions
"Power is of two kinds. One is obtained by the fear of punishment and the other by acts of love. Power based on love is a thousand times more effective and permanent then the one derived from fear of punishment." Mahatma Ghandi

Two factor can be disabled via user settings page, I'm uncertain of the purpose for this PR. Is it for when  users lock themselves out by misplacing mobile devices?

The ticket for this PR has a bulleted list of requirements, duplicating that here as a task list:

[X]     User accounts include a button to remove 2-factor authentication
[X]     OSF staff member receives a warning after selecting: "You are about to remove 2-factor authentication from [user name's] account. Do you want to continue?" Yes No
[X]     Clicking the Remove 2-Factor button removes 2-factor for this user
[ ]     OSF staff member receives verification message: "You have successfully removed 2-factor authentication from [user name's] account. (the refreshed display will show this.)

(CR Note: Unsure if this is necessary, view updates and indicates that 2F has been removed)

[ ]     User record now has an ADD 2-Factor button (ie: this action can be reversed) (removing two factor deletes the addon, but the user can add it back when they log in.)

(CR Note: This feature appears to have been abandoned.)

[ ]     The removal of 2-factor is logged (date, time, and which staff member did it)

(CR Note: Is this included in the Log Admin Actions PR, or will it be added in a follow-up to both?)

[ ]     OSF user receives an email telling them about the setting change. (manual)

(CR Note: No email appears to be sent out, and the Email button beside user emails is disabled.)

Pass complete @Ghalko, there are some discrepancies between ticket description and functionality, please address :basketball:

To respond (further) to the not checked off items.

The change is indicated
Refer to reversing comment above.
Logging actions will be subsequent tickets to #5001
Sending a mail will be in subsequent tickets to #5056 Basically the desk.com integration will allow the admin to send an Email through desk, keeping the desk ticket up-to-date. Implementing a whole new Email function in support of the user might very well confuse them if an Email is sent from admin (without the admin knowing) and then receiving one from desk as well.

In short:
We have plans for better tools and really the ticket is outlining an en total approach rather than a 'make Sara's and Steve's lives easier' approach.

Tickets created for unmet, potentially difficult, updates to the actions. ADM-5, ADM-79, ADM-80, ADM-81

Tests look good, but will also need to be changed because it's a GET request sent to remove_2_factor.
:ox:  Back to you @Ghalko

Please merge #5013 before this can proceed.


{Network} az network nic ip-config update: All IP configurations must belong to the same ASGs

Related command

az network nic ip-config update
Description

It's a potential bug introduced by my previous migration: https://github.com/Azure/azure-cli/pull/25498.
Testing Guide

History Notes

[Component Name 1] BREAKING CHANGE: az command a: Make some customer-facing breaking change
[Component Name 2] az command b: Add some customer-facing feature

This checklist is used to make sure that common guidelines for a pull request are followed.

[ ] The PR title and description has followed the guideline in Submitting Pull Requests.

[ ] I adhere to the Command Guidelines.

[ ] I adhere to the Error Handling Guidelines.

Network


Browser crashes when Stripe dashboard prompts to confirm password when navigating to FocusView

The browser crashes when, after a period of user inactivity in the Stripe dashboard, an attempt is made to update a Stripe object, thus presenting a "Confirm your password" dialog box in the Stripe dashboard.  This happens when the request to update an invoice is called from within a Focus view in a MemoryRouter.
To Reproduce

Create a FocusView within a MemoryRouter, then make any "update" call on the Stripe API, for example:

import { MemoryRouter, Routes, Route, useNavigate, useParams } from 'react-router-dom'
import Stripe from 'stripe';
import { useEffect, useCallback } from 'react';
import { Button, ContextView, FocusView } from '@stripe/ui-extension-sdk/ui';
import { ExtensionContextValue } from '@stripe/ui-extension-sdk/context';
import { STRIPE_API_KEY, createHttpClient } from "@stripe/ui-extension-sdk/utils/httpClient";

const stripeClient = new Stripe(STRIPE_API_KEY, {
    httpClient: createHttpClient(),
    apiVersion: '2023-08-16',
});

const InvoiceDetailView = ({ userContext, environment, appContext }: ExtensionContextValue) => {

    const context = {
        userContext,
        environment,
        appContext
    };

    return (
        <MemoryRouter>
            <Routes>
                <Route path='/' element={<InvoiceHomeView context={context} />} />
                <Route path='/invoice/:id' element={<BrowserCrashTest />} />
            </Routes>
        </MemoryRouter>
    )
}

const InvoiceHomeView = ({ context }: { context: ExtensionContextValue }) => {

    const navigate = useNavigate();

    const navigateToInvoice = () => {
        if (context.environment && context.environment.objectContext?.id) {
            navigate(`/invoice/${context.environment.objectContext.id}`);
        }
    };

    return (
    <ContextView title="Browser crash test">
    <Button type="primary" onPress={navigateToInvoice}>Test for browser crash</Button>
     </ContextView>
    )
}

const BrowserCrashTest = () => {
    const { id } = useParams<"id">();
    const navigate = useNavigate();

    const artificiallyTriggerPasswordPrompt = useCallback(async (id) => {
        if (id) {
            await stripeClient.invoices.update(id, { metadata: { preship: "" } })
        }
    }, [])

    useEffect(() => {
        if (id?.includes('in_')) {
            artificiallyTriggerPasswordPrompt(id);
        }
    }, [id]);

    return (
        <FocusView
            title="Trigger Stripe password prompt"
            shown={true}
            onClose={() => { navigate("/") }}>
            Triggering password prompt  (This test will only trigger an error if enough time has passed to prompt the Stripe user to confirm their password after a long period of inactivity in the Stripe dashboard)
        </FocusView>
    );
};

export default InvoiceDetailView

Screenshots

Desktop

OS: Windows 11
Browser: Chrome v119

I've flagged this to the team internally and we'll investigate. Thanks for the report!

I was able to resolve this issue by switching to a ContextView as required by the docs. But this error as reported in the example I gave can be repeated in multiple environments.

Hey there, we haven't tested using Stripe Apps with React Router. Can you tell me more about your use case? Are you trying to set up navigation within your app that alters the URL? Thanks.

No, I'm not attempting to alter the URL.  The memory router doesn't behave this way, the URL in the memory router is more of an internal route used to distribute chunks of pages so a very large app can load efficiently; without loading unnecessary components.  To reproduce the error, I did leave-out one key detail: The InvoiceDetailView component in my example should be referenced in your stripe-app.json views object as:
 {
        "viewport": "stripe.dashboard.invoice.detail",
        "component": "InvoiceDetailView "
 }

You would then need to navigate to a individual invoice view in Stripe (after a period of inactivity sufficient to trigger a password prompt).  At that point, it's a guaranteed browser crash (according to my tests on multiple environments).
That said, I have resolved the error by following the documentation indicating that a FocusView should never be the root component.  Rather, a FocusView should always live as a child of a ContextView.
You're welcome to close this issue.

Great, thanks Benson.


500 Internal Server error

Hello,
i did a fresh install witht the SampleScript and get a 500 Internal Server error when i try to open the webgui
i double checked the correct locales are installed.

have tested the docker container and wont use openxpki in my project


link to "classic" collection details page from collection modal

Need to first know how to format links to pages in the Wing system

The URL pattern is: '/classic/basicCollectionSearch?collection_no=' + collection_no
When extracting the numeric part from the collection identifier, I would suggest using a regular expression to extract (\d+) rather than trying to delete characters. This approach will be more robust in case we need to change the identifier syntax at any point.


Fix some typos

I understand if extract_instrinsics_from_llvm change is too intrusive a change... let me know!

No it's fine, thanks for the fix!


extracted files corrupted

hi,
we are using node-unzip and have recently found a reproducible case where one of the files extracted from the archive is corrupted, both the file size and the shasums of the decompressed file are different when compared to the same archive extracted on the command line.
the archive is sourced from the US Census Bureau and is valid (as far as I can tell using unzip -t)
I've written up the issue in more detail here:
https://github.com/pelias/interpolation/issues/162

@missinglink was this closed because the problem was related to something else, or is this still an issue?


XamCode sample : unable to reference a dll

It appears that winmd is the only option. Is this permanent?

Hi Jack. For a UWP app, you are restricted to winmd references. If you have other projects in the solution for WinRT components, you should be able to reference those projects as well.
If you want to reference a Win32 dll, I recommend starting here. This gives background on linking to your Win32 dll and instructions for including it in the app package.

Kevin
For a  C# UWP  app, a dll can be referenced if it is portable. So my question is, will C++/winrt be able to access a portable dll

I see. Thanks for the clarification.
Portable DLLs are .NET assemblies, but are not WinRT type libraries. Hence no available winmd to  reference. In a C# UWP app, this isn't a problem. But in a C++ application, you need to enable the /clr option to add .NET assemblies (portable or otherwise) as project references. C++/winrt uses compiler flags and syntax that are not compatible with the /clr option.

Kevin,
Bottom line, it looks like I will be unable to convert my apps from C# to C++/winrt...
Thanks for your help
Jack

From: Kevin Welton<EMAIL_ADDRESS>Sent: Friday, January 13, 2017 11:51:55 AM
To: Microsoft/cppwinrt
Cc: JackDetrick; Author
Subject: Re: [Microsoft/cppwinrt] XamCode sample : unable to reference a dll (#76)
I see. Thanks for the clarification.
Portable DLLs are .NET assemblies, but are not WinRT type libraries. Hence no available winmd to reference. In a C# UWP app, this isn't a problem. But in a C++ application, you need to enable the /clr option to add .NET assemblies (portable or otherwise) as project references. C++/winrt uses compiler flags and syntax that are not compatible with the /clr option.

You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/Microsoft/cppwinrt/issues/76#issuecomment-272531387, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ADSGiYOKRP-wWxzGotsHau5yh9oVrFFIks5rR9XbgaJpZM4Li2Gj.

You can convert apps from C# to C++/WinRT. Basically, this means converting the C# app code to C++ code (especially as C++/WinRT is simply standard C++ code).
But standard C++ code cannot link to C# libraries, portable or not. Therefore, you'd need to convert the libraries from C# to C++ as well.
Nothing here is particular to C++/WinRT.

Brent
I have converted my C# code into C++/winrt   (the non XAML portion).The issue, is I need to reference a portable DLL from my C++/winRT code (and then let the certification process validate it).

From: Brent Rector<EMAIL_ADDRESS>Sent: Friday, January 13, 2017 12:35:04 PM
To: Microsoft/cppwinrt
Cc: JackDetrick; Author
Subject: Re: [Microsoft/cppwinrt] XamCode sample : unable to reference a dll (#76)
You can convert apps from C# to C++/WinRT. Basically, this means converting the C# app code to C++ code (especially as C++/WinRT is simply standard C++ code).
But standard C++ code cannot link to C# libraries, portable or not. Therefore, you'd need to convert the libraries from C# to C++ as well.
Nothing here is particular to C++/WinRT.

You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/Microsoft/cppwinrt/issues/76#issuecomment-272541223, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ADSGibanIb41Ivi0GAsPHuE5sLla2Rkxks5rR9_4gaJpZM4Li2Gj.

Brent
The DLL in question was written in C#,. Does that mean it must be rewitten in C++/CX ?

There's no need to rewrite the C# code into C++/CX.
Standard C++ cannot use/call .NET types, and C++/WinRT is simply standard C++. However, you can use C++/CLI to interoperate with C++ and .NET types.
You could write an interop component using C++/CLI. That component would expose a C++ layer that your client code would call. In turn the C++/CLI component could map the incoming C++ calls to outgoing .NET calls to your C# DLL and vice-versa.
Note that this applies to any interop between C++ and .NET components. It's not specific to C++/WinRT.

Thanks


Runtime ModuleNotFoundError for training and basecalling

I get a ModuleNotFoundError when trying to run the train_original or basecall_original scripts with the required packages and demo data:
(venv3) s@bwt:~/mobile_workspace/original/basecalling_architectures$ python ./scripts/basecall_original.py \
--model bonito \
--fast5-dir ./demo_data/fast5 \
--checkpoint ./demo_data/properly_trained_bonito/bonito_checkpoint.pt \
--output-file ./demo_data/demo_basecalls.fastq
Traceback (most recent call last):
  File "/home/shishir/mobile_workspace/original/basecalling_architectures/./scripts/basecall_original.py", line 9, in <module>
    from classes import BasecallerImpl, BaseFast5Dataset
  File "/home/shishir/mobile_workspace/original/basecalling_architectures/src/classes.py", line 15, in <module>
    from normalization import normalize_signal_from_read_data, med_mad
  File "/home/shishir/mobile_workspace/original/basecalling_architectures/src/normalization.py", line 5, in <module>
    from scipy.signal import find_peaks
  File "/home/shishir/mobile_workspace/original/basecalling_architectures/venv3/lib/python3.9/site-packages/scipy/signal/__init__.py", line 310, in <module>
    from ._peak_finding import *
  File "/home/shishir/mobile_workspace/original/basecalling_architectures/venv3/lib/python3.9/site-packages/scipy/signal/_peak_finding.py", line 8, in <module>
    from scipy.stats import scoreatpercentile
  File "/home/shishir/mobile_workspace/original/basecalling_architectures/venv3/lib/python3.9/site-packages/scipy/stats/__init__.py", line 441, in <module>
    from .stats import *
  File "/home/shishir/mobile_workspace/original/basecalling_architectures/venv3/lib/python3.9/site-packages/scipy/stats/stats.py", line 43, in <module>
    from . import distributions
  File "/home/shishir/mobile_workspace/original/basecalling_architectures/venv3/lib/python3.9/site-packages/scipy/stats/distributions.py", line 11, in <module>
    from . import _discrete_distns
  File "/home/shishir/mobile_workspace/original/basecalling_architectures/venv3/lib/python3.9/site-packages/scipy/stats/_discrete_distns.py", line 19, in <module>
    from .biasedurn import (_PyFishersNCHypergeometric,
  File "biasedurn.pyx", line 1, in init scipy.stats.biasedurn
ModuleNotFoundError: No module named 'numpy.random.bit_generator'

This error seems consistent with a scipy issue here: https://github.com/scipy/scipy/issues/14541
After following the instructions in this reply, the error disappears, but the command hangs on importing BasecallerImpl and BaseFast5Dataset instead.
I am using cuda 10.2, cudnn v8.7.0 with python packages and versions corresponding to the requirements.txt file. Please let me know if there is a different configuration of packages I should be using or a workaround here, thanks!

I have added the numpy version I used on the requirements (it was missing for some reason), it is numpy==1.21.5. I also used python3.7, as I see you use 3.9. My suggestion would be that you create a new environment on python3.7 and re-install the requirements with the new numpy version constraint.

I have created a new environment with python3.7 and the updated requirements.txt file, but I am still seeing some dependency conflicts. It looks like there is a conflict where seqdist 0.0.3 depends on numpy <= 1.18.5:
(venv3) s@bwt:~/mobile_workspace/original/basecalling_architectures$ pip install -r requirements.txt
Collecting numpy==1.21.5 (from -r requirements.txt (line 1))
  Using cached numpy-1.21.5-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)
Collecting cupy-cuda102==9.4.0 (from -r requirements.txt (line 2))
  Using cached cupy_cuda102-9.4.0-cp37-cp37m-manylinux1_x86_64.whl (60.7 MB)
Collecting fast_ctc_decode==0.3.0 (from -r requirements.txt (line 3))
  Using cached fast_ctc_decode-0.3.0-cp37-cp37m-manylinux2010_x86_64.whl (508 kB)
Collecting h5py==3.7.0 (from -r requirements.txt (line 4))
  Using cached h5py-3.7.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (4.1 MB)
Collecting koi-cuda102==0.0.5 (from -r requirements.txt (line 5))
  Using cached koi_cuda102-0.0.5-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (350 kB)
Collecting ont-fast5-api==4.0.2 (from -r requirements.txt (line 6))
  Using cached ont_fast5_api-4.0.2-py3-none-any.whl (2.0 MB)
Collecting pandas==1.3.5 (from -r requirements.txt (line 7))
  Using cached pandas-1.3.5-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.3 MB)
Collecting parasail==1.2.4 (from -r requirements.txt (line 8))
  Using cached parasail-1.2.4-py2.py3-none-manylinux2010_x86_64.whl (14.1 MB)
Collecting psutil==5.8.0 (from -r requirements.txt (line 9))
  Using cached psutil-5.8.0-cp37-cp37m-manylinux2010_x86_64.whl (296 kB)
Collecting scipy==1.7.1 (from -r requirements.txt (line 10))
  Using cached scipy-1.7.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (28.5 MB)
Collecting seqdist==0.0.3 (from -r requirements.txt (line 11))
  Using cached seqdist-0.0.3-py3-none-any.whl (21 kB)
Collecting torch==1.9.0 (from -r requirements.txt (line 12))
  Using cached torch-1.9.0-cp37-cp37m-manylinux1_x86_64.whl (831.4 MB)
Collecting torchaudio==0.9.0 (from -r requirements.txt (line 13))
  Using cached torchaudio-0.9.0-cp37-cp37m-manylinux1_x86_64.whl (1.9 MB)
Collecting torchvision==0.10.0 (from -r requirements.txt (line 14))
  Using cached torchvision-0.10.0-cp37-cp37m-manylinux1_x86_64.whl (22.1 MB)
Collecting tqdm==4.62.0 (from -r requirements.txt (line 15))
  Using cached tqdm-4.62.0-py2.py3-none-any.whl (76 kB)
Collecting fastrlock>=0.5 (from cupy-cuda102==9.4.0->-r requirements.txt (line 2))
  Using cached fastrlock-0.8.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_24_x86_64.whl (46 kB)
Collecting cffi (from koi-cuda102==0.0.5->-r requirements.txt (line 5))
  Using cached cffi-1.15.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (427 kB)
Collecting packaging (from ont-fast5-api==4.0.2->-r requirements.txt (line 6))
  Using cached packaging-23.1-py3-none-any.whl (48 kB)
Collecting progressbar33>=2.3.1 (from ont-fast5-api==4.0.2->-r requirements.txt (line 6))
  Using cached progressbar33-2.4.tar.gz (10 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting python-dateutil>=2.7.3 (from pandas==1.3.5->-r requirements.txt (line 7))
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting pytz>=2017.3 (from pandas==1.3.5->-r requirements.txt (line 7))
  Using cached pytz-2023.3-py2.py3-none-any.whl (502 kB)
INFO: pip is looking at multiple versions of seqdist to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install -r requirements.txt (line 10), -r requirements.txt (line 11), -r requirements.txt (line 2), -r requirements.txt (line 4), -r requirements.txt (line 6), -r requirements.txt (line 7), -r requirements.txt (line 8) and numpy==1.21.5 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested numpy==1.21.5
    cupy-cuda102 9.4.0 depends on numpy<1.24 and >=1.17
    h5py 3.7.0 depends on numpy>=1.14.5
    ont-fast5-api 4.0.2 depends on numpy>=1.16
    pandas 1.3.5 depends on numpy>=1.17.3; platform_machine != "aarch64" and platform_machine != "arm64" and python_version < "3.10"
    parasail 1.2.4 depends on numpy
    scipy 1.7.1 depends on numpy<1.23.0 and >=1.16.5
    seqdist 0.0.3 depends on numpy<=1.18.5

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

After installing seqdist 0.0.3 separately and upgrading numpy to 1.21.5, both training and basecalling work! Thanks for the help

Thanks for letting me know! I will add a note on the readme in case someone faces the same issue.


How to start Nunustudio fullscreen

Hello ,
I start nunustudio via the command line  on linux
nw source
Now  I wonder how to make it appear full screen (maximized).
Thanks a lot ! Long live Nunu !

Hi
I have made some changes recently that allow to run nunu using nw from npm. (npm install, npm star run)
To run nunuStudio in fullscreen change the source/package.json file and add "fullscreen": true inside of window:{...}.
Thaks a lot for your suppot i hope this helps.
Cheers

Perfect thanks ! it is working fine


Your request to provision "Microsoft 365 learning pathways"  template failed!

Correlation ID: c95419ec-d775-4454-8257-00896d5f74a6

Hi, Thanks for your feedback. I'm closing this item because it is now outdated.

Thank you so much for your timely assistance


fix: Update vendor lookup method and dependency for Debian packaging

Replaced import OuiLookup with from ouilookup import OuiLookup due to compatibility issues with ouilookup 0.3.1. This change ensures the project works correctly.
Additionally, introduced ieee-data for CSV access, enabling the software to function independently of the ouilookup package, crucial for Debian packaging.

Hi @ndejong, I hope you're doing well! I've been diligently working on packaging arpwitch for Debian and wanted to provide an update on the recent developments. In the process, I encountered a hiccup regarding the ouilookup package, which isn't packaged in Debian.
To ensure arpwitch's functionality within the Debian environment, I've implemented a crucial change by introducing a dependency on ieee-data. This addition is aimed at circumventing the unavailability of the ouilookup package and enabling arpwitch to seamlessly access the necessary CSV data from ieee-data.
These changes are pivotal to arpwitch's successful integration into the Debian ecosystem.

Hi @aquilamacedo thanks for the PR.
Arpwitch does need some love to get it up-to-scratch before being Debian package repo worthy in my opinion.
I'm not super excited about the hard-coding of this in your PR
oui_path = "/usr/share/ieee-data/oui.csv"

Give me a few days (or more) to rework Arpwitch, among other things it needs some async worker magic to keep up with traffic.
I'd still value your assistance and efforts in packaging for Debian though - perhaps we can pick that up as an issue - https://github.com/ndejong/arpwitch/issues

Hi @ndejong. Nice, I'll keep an eye out for any updates on the Arpwitch rework.. If there's anything I can do to lend a hand, just let me know. I'm here and ready to help however I can :)


Skip ambient light for standard shader

Hi guys,
I might be terribly missing something, but it looks like ambient light always gets added to meshes rendered with Standard Shader, even if the lightning is turned off. The suspect is https://github.com/playcanvas/engine/blob/master/src/graphics/program-lib/standard.js#L664
As standard shader is used in various instances, including UI, I believe there should be an option to skip the ambient light completely?

Ambient can be useful, the useLighting param only affects lights. We should add separate useAmbient maybe?

Yeah, that's what I thought

I missed this control as well few days ago.
A workaround was material.diffuse = new pc.Color(0, 0, 0);, but that's not ideal


Migrate Documentation

Our documentation is currently in README.md. As our codebase expands, we will eventually outgrow this. The transition may be very difficult with a large codebase. I would like to explore various options to host our documentation before this happens.

Hi @andyluo03 !
I am interested in this.
I think readthedocs would be a good option for documentation.
Would love to work on this

@adityakode This is definitely an option! How would we go about linking this to version control and hosting our web API? Also, feel free to join our contributing discord

@adityakode I created docs using mkdocs. If you are still interested in helping contribute documentation as the project grows, please let me know!


Read the Docs (RTD) build failing

Facing same issue as in: https://github.com/ga4gh/schemas/issues/819

Looks like an RTD issue: https://github.com/rtfd/readthedocs.org/issues/2651

I suppose one option would be to get RTD's commercial support for this issue: https://readthedocs.com/services/#open-source-support  Of course, both RTD yaml config and conda support are both beta features...

Another option, hosting them off of a docs.ga4gh.org domain and deploying static HTML instead of recreating the sphinx environment.

Their code is open source so we could always fix the bug for them, heh.  Although I suspect it's more of a configuration issue on their side than a problem with the code per se.

@dcolligan did you have another approach using pip?

@dcolligan yeah I just emailed https://github.com/ericholscher out of desperation. They probably need to update some local docker registry or some such thing.
@kozbo I believe we can make pip work for the release in all repos since we are now including compiled proto everywhere. But I don't think it will last if we accept the proposed change at https://github.com/ga4gh/schemas/issues/828 so we'll be thrashing back as soon as we want to build docs in RTD since we are going to require protoc to compile from source during development.
Another option: use a deploy script at travis to deploy a version of the schemas that includes the pb2 files on merged pull request. This would allow the downstream repos to use a "released" version that doesn't have the protoc requirement. If you tried to pip install git//ga4gh/schemas it would fail if you don't have protoc, but pip install ga4gh-schemas --pre will install a version with the proto installed.
I don't think our internal dependencies should NOT have to build the proto from source. If we can do continuous release with a line or two of travis, I think that's the best option and solves https://github.com/ga4gh/schemas/issues/828 as well.

This should now be fixed. New issue for discussing taking out RTD.

Still not working, but it seems like a problem with our code now (importing bigwig), which is good.
https://readthedocs.org/projects/ga4gh-reference-implementation/builds/5120676/

Ok, made some progress (added bigwig to the pip requirements), but now stuck on this error:
https://readthedocs.org/projects/ga4gh-reference-implementation/builds/5120992/
(I pointed the RTD builds to my repo for the purposes of testing: https://github.com/dcolligan/server )
Even though curl is listed as a dependency, it doesn't seem to be installed before conda attempts to setup bigwig and it fails because it can't find curl-config.
@david4096 Do you have any suggestions for fixing this?  Also, is there any guide to the environment.yml file format syntax?

Create Environment File By Hand
That binary is provided by the libcurl-openssl-dev package in the server. I did a quick search of conda channels for it and didn't find anything.
I think we should be able to use PyBigWig without curl, but I'm not quite sure how. https://github.com/dpryan79/pyBigWig/issues/29

Thanks for filing that issue with pyBigWig.
Another route would be creating a conda package ourselves that included curl-config... I'm not sure how feasible this is, knowing little about conda.
Yet another thing to do would be to comment out portions of the code that include bigwig if we detect that we are on RTD: http://docs.readthedocs.io/en/latest/faq.html#how-do-i-change-behavior-for-read-the-docs
That seems the most feasible -- although cumbersome and hacky -- option at this point, since ditching RTD, the conda build, or BigWig all seem unacceptable... thoughts?

If it's a sphinx feature it's not weird or hacky as far as they're concernred.
The next section let's you mock C-modules. Can we do it that way?
There's no reason to actually have pybigwig around for doc generation since we just generate docs for CLI portions, that don't depend on it. We don't really need to have pybigwig around (or a lot of other things). If the conditional approach is pretty easy, we can use that.

I've posted instructions on how to mock dependencies on the issue @david4096 posted on pyBigWig. I'll add that you could just install it through conda too, since that's what you're using.

Thanks @dpryan79 https://github.com/dpryan79/pyBigWig/issues/29#issuecomment-285497043

Fixed in #1614


deprecated masternodes/list : how to get a masternode list?

Hi devs,
At first : thank you all for you commitment to the Dash network!
I'm playing with the libs and I'm looking for a way to have a masternode list with insight-api.
For the moment at insight-api/masternodes/list we can find:
"This endpoint has been deprecated and will be replaced with a new endpoint compatible with the deterministic masternode list introduced in v0.13"
Although when you look at https://insight.dash.org/api/masternodes/list we still have a JSON answer (a good answer?).
Am I using the wrong branch? is the dash.org insight an old version?
Thank you in advance for your answers!

Hi @med4u!
The old endpoint is not valid anymore since the introduction of deterministic mn list couple of month ago. We didn't introduce a new method in insight yet, but we will do soon

Added to the backlog.


Validate input circuits don't exceed maximum allowed

What is the expected enhancement?
If a list of circuits is passed to backend.run(), make sure the number of circuits don't exceed what the backend allows.

We can use backend.max_circuits

@jyu00 - I am wondering if we need to make the same fix in qiskit-ibm-runtime. I tried to run more than max_circuits on a real device, and it ran, without giving an exception. Should I limit the number there as in the provider?


add all of the SwiftNIO project

Ah, sorry I didn't see your force push. I re-sorted it and merged manually.

So your original commit made it onto master. I'll close this.

Thank you and apologies for first committing it unsorted


refactor: add dragonfly-client-config crate

Description

Related Issue

Motivation and Context

Screenshots (if appropriate)

@yxxhero


unison-ucm 1.0.M1l-alpha -> 1.0.M1m-alpha

See title.  Also, add missing less to buildInputs.

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[ ] NixOS
[X] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[X] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[X] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 104809 run on x86_64-linux 1

1 package built:

unison-ucm


Two albums with the same name

Hi. If two albums have the same name, even by two different artists, Audiofy lists only one album, with all the tracks of the two artists inside! Now, i'm not an experienced coder, but i could see that the function:
suspend fun ContentResolver.getAlbum(-----)
gets the list of the audios of an album by the album name. Why not by id?
Thx

Thanks for letting us know about the issue! We'll investigate it thoroughly and aim to include a fix in the next update.

Hi iZakir. I tested the same thing with other players and it works correctly. The two albums, with the same name, are listed two times, with different songs and covers. Hope it can help. By.

Hi iZakir. I tested the same thing with other players and it works correctly. The two albums, with the same name, are listed two times, with different songs and covers. Hope it can help. By.

In the next update, I will try to fix this issue.


fix: ethers signMessage test message

Description

Ethers was sending "Hello Web3Modal!" instead of "Hello AppKit!"

Type of change

[ ] Chore (non-breaking change that addresses non-functional tasks, maintenance, or code quality improvements)
[X] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist

[X] Code in this PR is covered by automated tests (Unit tests, E2E tests)
[X] My changes generate no new warnings
[X] I have reviewed my own code
[X] I have filled out all required sections

Fixing this on multi-chain directly


use up instead of table in migration

This PR bugfixes the stub of migrations to use Schema::create instead of Schema::table.

We reverted this, as it is wrong 🙄


Change log level to show port conflict error

Background
This PR change log level to show error message including port conflict. Note that, this PR doesn't not terminate the server, it just allow error message goes through. I think it's sufficient enough and generic.
Resolves https://github.com/Shopify/checkout-web/issues/5067
Solution
Change log level
🎩

Create new checkout extension shopify create extension
Run the extension via shopify serve, note the port
In this PR run, yarn && yarn build && cp -R ./packages/* ../your-extension-prodject/node_modules/@shopify/
Run the extension in different terminal using same port yarn start --port <some-port-number>
Verify that the error message about port conflict appear

yarn start --port 8910
yarn run v1.22.4
$ argo-run dev --port 8910
🔭 > Starting dev server on http://localhost:8910
✖ ｢wds｣:  Error: listen EADDRINUSE: address already in use <IP_ADDRESS>:8910
    at Server.setupListenHandle [as _listen2] (net.js:1316:16)
    at listenInCluster (net.js:1364:12)
    at doListen (net.js:1501:7)
    at processTicksAndRejections (internal/process/task_queues.js:81:21) {
  code: 'EADDRINUSE',
  errno: -48,
  syscall: 'listen',
  address: '<IP_ADDRESS>',
  port: 8910
}

Checklist

[ ] I have :tophat:'d these changes

I feel like an error is an error, though, so probably not.

I think so too. Based on my test, I don't see any other errors when testing both cases. I think it's fine. It will output future errors if any which is good to have I believe.

Can you paste what it looks like in a port conflict?

The port conflict message looks great. If there is an easy way to exit the process non-zero that would be a bonus but that's definitely not a blocker here.


[BUG] RWX volume is hang on Photon Real Time OS

Describe the bug
When there are multiple pods on the same node access a RWX volume and doing IOs. The mount point of the RWX volume inside the pod is frozen and pod cannot read/write
To Reproduce

Install Photon OS real time flavor https://packages.vmware.com/photon/5.0/GA/iso/photon-rt-5.0-dde71ec57.x86_64.iso
Install K3s and Longhorn 1.6.0 on it
Deploy this workload:# Long-running test with prometheus metric exporter to monitor the data
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-1
spec:
  volumeMode: Filesystem
  storageClassName: longhorn  # replace with your storage class
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 2Gi
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: test-workload
  namespace: default
spec:
  serviceName: test-workload
  replicas: 1
  selector:
    matchLabels:
      app: test-workload
  podManagementPolicy: Parallel
  template:
    metadata:
      labels:
        app: test-workload
    spec:
      nodeName: photon-02 # Replace this one by one of the node in your cluster to force all pods of this workload on that node
      containers:
        - name: kbench-1
          image: phanle1010/kbench:broadcom
          imagePullPolicy: Always
          env:
            - name: MODE
              value: "random-write-iops"
            - name: OUTPUT
              value: /test-result/pvc1
            - name: FILE_NAME
              value: "/pvc1/test.img"
            - name: SIZE
              value: "1638M" # 80% of the pvc-1's size
            - name: CPU_IDLE_PROF
              value: "disabled"
            - name: SKIP_PARSE
              value: "true"
            - name: LONG_RUN
              value: "true"
            - name: RATE_IOPS
              value: "1167"
          volumeMounts:
            - name: pvc1
              mountPath: /pvc1/
            - name: shared-data
              mountPath: /test-result
        - name: metric-exporter
          image: phanle1010/kbench:broadcom
          imagePullPolicy: Always
          command:
            - metric-exporter
            - -d
            - start
          env:
            - name: DATA_DIR
              value: /test-result
            - name: VOLUME_ACCESS_MODE
              value: rwx
            - name: TEST_MODE
              value: write-only
            - name: RATE_LIMIT_TYPE
              value: rate-limit
          ports:
            - containerPort: 8080
              name: metrics
          volumeMounts:
            - name: shared-data
              mountPath: /test-result
      volumes:
        - name: shared-data
          emptyDir: {}
        - name: pvc1
          persistentVolumeClaim:
            claimName: pvc-1

Scale up the stateful set to 6
After 2-6 mins, exec into the pods and check that the mount point of RWX is frozen. I.e., running ls -l /pvc1 is hang

Expected behavior
Should not hang
Environment

Longhorn version: Longhorn v1.6.0
Impacted volume (PV):
Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Helm
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: v1.28.7+k3s1

Number of control plane nodes in the cluster: 1
Number of worker nodes in the cluster: 4

Node config

OS type and version: Photon RT OS https://github.com/vmware/photon/wiki/Downloading-Photon-OS#downloading-photon-os-50-ga
Kernel version: 6.1.10-10.ph5-rt
CPU per node: 64
Memory per node: 503GB
Disk type (e.g. SSD/NVMe/HDD): NVMe
Network bandwidth between the nodes (Gbps): 20Gbps

Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): Baremetal
Number of Longhorn volumes in the cluster: 1

Additional context
When I added an Ubuntu node to the cluster and deploy the sts on to ubuntu node, RWX volume is not frozen

Support bundle and dmesg when the mount point in the pods are hang:

The problematic volume is pvc-b706aa95-d4b2-42ff-b6c1-b844f3af3409
6 workload pods and the share-manager is on node photon-03
The global mount of the RWX volume is also stuckroot@photon-03 [ ~ ]# ls -l /var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/1a1eb22a8c78f39ae84d894c5d483b7ac44c482a67816b146a742d16f03001c8/globalmount
<stuck forever>

supportbundle_f9c6d852-b61d-470a-8147-009b614febce_2024-03-26T05-15-14Z.zip
dmesg.log

@shuo-wu @james-munson I followed your recommendation of manually mounting the nfs server on same node / use Ubuntu:

Manually mounting the nfs server on same node still work even though the mount points inside pods are stuck
I install Ubuntu 22.04 on another baremetal node and join that node to this cluster. Then force all pod of the sts on the ubuntu node. Result: no hanging

How about launching a new workload pod with the same image then doing a manual mount in the CSI plugin for the pod?

How about launching a new workload pod with the same image then doing a manual mount in the CSI plugin for the pod?

Sorry Shuo, I don't quite understand this step. Do you mean:

launch a new pod on same node without the RWX PVC attached
Then exec into the pod and mount nfs manually

Simpler reproduce steps:

Deploy this simple workload# Long-running test with prometheus metric exporter to monitor the data
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-1
spec:
  volumeMode: Filesystem
  storageClassName: longhorn  # replace with your storage class
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 2Gi
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: test-workload
  namespace: default
spec:
  serviceName: test-workload
  replicas: 1
  selector:
    matchLabels:
      app: test-workload
  podManagementPolicy: Parallel
  template:
    metadata:
      labels:
        app: test-workload
    spec:
      nodeName: photon-03 # Replace this one by one of the node in your cluster to force all pods of this workload on that node
      containers:
        - name: kbench-1
          image: ubuntu:xenial
          imagePullPolicy: IfNotPresent
          command: ["/bin/bash", "-c", "while : ;do dd if=/dev/urandom of=/pvc1/test.img bs=4096 count=102400 conv=notrunc; sleep 1; done"]
          volumeMounts:
            - name: pvc1
              mountPath: /pvc1/
      volumes:
        - name: pvc1
          persistentVolumeClaim:
            claimName: pvc-1

Check that read/write to /pvc1 in the pod is ok
Scale it up to 6 pods
Wait for 5 mins. check that read/write to /pvc1 in the pods are frozen

Does this Only happen in this specific OS? @PhanLe1010
Is it possible due to nfs host package issue?

Does this Only happen in this specific OS? @PhanLe1010
Is it possible due to nfs host package issue?

@innobead it happen in Photon OS RT but doesn't happen on the Ubuntu 22.04 cluster. I am going to test photon OS on RT to see if it can happens

Is it possible due to nfs host package issue?

Good idea, I will check this

A few things can also be checked:

if reduce the replica of the workload, will the stuck IO be resolved at runtime?
during IO stuck, any logs happening in the share manager pod? Are you able to manually mount it in another workload and also encounter the IO stuck issue?

if reduce the replica of the workload, will the stuck IO be resolved at runtime?

@innobead No, when scaling down the sts, the pods stuck in pending terminating since the nfs mount cannot be unmount. I have to reboot the node OR kill the share-manager pod to make the workload pod finish terminating.

during IO stuck, any logs happening in the share manager pod?

There is nothing special in share-manager pod's logs. The logs is just like this after the NFS server start:
26/03/2024 22:42:35 : epoch 66034f5b : share-manager-pvc-b706aa95-d4b2-42ff-b6c1-b844f3af3409 : nfs-ganesha-37[main] nfs_start :NFS STARTUP :EVENT :             NFS SERVER INITIALIZED
2024-03-26T22:42:35.882456693Z 26/03/2024 22:42:35 : epoch 66034f5b : share-manager-pvc-b706aa95-d4b2-42ff-b6c1-b844f3af3409 : nfs-ganesha-37[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
26/03/2024 22:42:42 : epoch 66034f5b : share-manager-pvc-b706aa95-d4b2-42ff-b6c1-b844f3af3409 : nfs-ganesha-37[svc_5] longhorn_add_clid :CLIENT ID :EVENT :Add client '23:Linux NFSv4.1 photon-03' to recovery backend share-manager-pvc-b706aa95-d4b2-42ff-b6c1-b844f3af3409

Are you able to manually mount it in another workload and also encounter the IO stuck issue?

Deploying a pod using the RWX volume on the same node will stuck in containerCreating as mentioned at https://github.com/longhorn/longhorn/issues/8253#issuecomment-2021240433.
However, deploying a pod using the RWX on a new node is OK

Is it possible due to nfs host package issue?

Looks like the Unbuntu one is using 2.6.1 while Photon one is using 2.6.2?
root@ubuntu-01:/home/ubuntu# dpkg -l | grep nfs
ii  libnfsidmap1:amd64                     1:2.6.1-1ubuntu1.2                      amd64        NFS idmapping library
ii  nfs-common                             1:2.6.1-1ubuntu1.2                      amd64        NFS support files common to client and server
==============
root@photon-03 [ ~ ]# yum list installed | grep nfs
libnfsidmap.x86_64                           2.6.2-9.ph5               @System
nfs-utils.x86_64                             2.6.2-9.ph5               @System

Update:
I did some test with different distribution of workload and share-manager pods. Result:

share-manager node
workload pods' node
state

Photon RT OS
Photon RT OS
nfs mounts are frozen

Photon RT OS
Ubtunu 22.04
nfs mounts are frozen

Ubtunu 22.04
Ubtunu 22.04
nfs mounts are NOT frozen

Ubtunu 22.04
Photon RT OS
nfs mounts are NOT frozen

Looks like the location of the share-manager is what matter

Update:
I did some tests with different distributions of workload pods and share-manager pod. Result:

share-manager pod's node
workload pods' node
state

Photon RT OS
Photon RT OS
nfs mounts are frozen

Photon RT OS
Ubtunu 22.04
nfs mounts are frozen

Ubtunu 22.04
Photon RT OS
nfs mounts are frozen but it takes longer time than the above 2 cases

Ubtunu 22.04
Ubtunu 22.04
nfs mounts are NOT frozen

Looks like the location of the share-manager pod is what matter

Do we know about the nfs kernel module? But doubt we are able to check it.

Do we know about the nfs kernel module? But doubt we are able to check it.

Is it this one?
root@photon-03 [ ~ ]# modinfo nfs
filename:       /lib/modules/6.1.10-10.ph5-rt/kernel/fs/nfs/nfs.ko.xz
license:        GPL
author:         Olaf Kirch<EMAIL_ADDRESS>alias:          nfs4
alias:          fs-nfs4
alias:          fs-nfs
depends:        sunrpc,fscache,lockd
retpoline:      Y
intree:         Y
name:           nfs
vermagic:       6.1.10-10.ph5-rt SMP preempt_rt mod_unload modversions
sig_id:         PKCS#7
signer:         Build time autogenerated kernel key
sig_key:        7D:F5:20:BC:BD:79:8F:D8:46:EF:B2:DB:45:C4:2F:FC:55:E1:22:EE
sig_hashalgo:   sha512
=========
ubuntu@ubuntu-01:~$ modinfo nfs
filename:       /lib/modules/5.15.0-101-generic/kernel/fs/nfs/nfs.ko
license:        GPL
author:         Olaf Kirch<EMAIL_ADDRESS>alias:          nfs4
alias:          fs-nfs4
alias:          fs-nfs
srcversion:     E46B121D3C0A67D41B92D03
depends:        fscache,sunrpc,lockd
retpoline:      Y
intree:         Y
name:           nfs
vermagic:       5.15.0-101-generic SMP mod_unload modversions
sig_id:         PKCS#7
signer:         Build time autogenerated kernel key
sig_key:        7D:31:86:13:F2:76:80:28:1A:E7:90:1A:C6:12:9B:85:D8:E4:8E:E9
sig_hashalgo:   sha512

Looks like the one on Photon RT OS is newer

The module version seem to be related to the kernel version. I am wondering if there will be a similar issue when Ubuntu kernel version is the same as that of Photon RT OS.

Update:

Since each machine has 2 NICs and I saw that some time NFS conntion is going to the 2nd NICs while I specify K3s to use the 1st NIC only. I disable the 2nd NIC on all machines --> NFS mounts are still stuck on Phonton RT node
Trying to add a Photon non RT node and test it ...
Trying to upgrade kernel and NFS module ...

Update:
Sorry! After repeatedly stress testing, the Photon non-RT OS with latest kernel 6.1.10-11.ph5 still suffers the NFS mounts frozen 😔

As discussed, @PhanLe1010 will also check if nfs mount has problems w/o Longhorn.

Update:
On Photon non-RT OS with latest kernel 6.1.81-2.ph5, we performed the following tests:
Case 1: Test with Kubernetes external NFS provisioner + hostPath (without any Longhorn component) ✅

This is the yaml to deploy this setup

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-provisioner
---
kind: Service
apiVersion: v1
metadata:
  name: nfs-provisioner
  labels:
    app: nfs-provisioner
spec:
  ports:
    - name: nfs
      port: 2049
    - name: nfs-udp
      port: 2049
      protocol: UDP
    - name: nlockmgr
      port: 32803
    - name: nlockmgr-udp
      port: 32803
      protocol: UDP
    - name: mountd
      port: 20048
    - name: mountd-udp
      port: 20048
      protocol: UDP
    - name: rquotad
      port: 875
    - name: rquotad-udp
      port: 875
      protocol: UDP
    - name: rpcbind
      port: 111
    - name: rpcbind-udp
      port: 111
      protocol: UDP
    - name: statd
      port: 662
    - name: statd-udp
      port: 662
      protocol: UDP
  selector:
    app: nfs-provisioner
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: nfs-provisioner
spec:
  selector:
    matchLabels:
      app: nfs-provisioner
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: nfs-provisioner
    spec:
      nodeName: photon-02 # Replace this one by one of the node in your cluster to force all pods of this workload on that node
      serviceAccount: nfs-provisioner
      containers:
        - name: nfs-provisioner
          image: registry.k8s.io/sig-storage/nfs-provisioner:v4.0.8
          ports:
            - name: nfs
              containerPort: 2049
            - name: nfs-udp
              containerPort: 2049
              protocol: UDP
            - name: nlockmgr
              containerPort: 32803
            - name: nlockmgr-udp
              containerPort: 32803
              protocol: UDP
            - name: mountd
              containerPort: 20048
            - name: mountd-udp
              containerPort: 20048
              protocol: UDP
            - name: rquotad
              containerPort: 875
            - name: rquotad-udp
              containerPort: 875
              protocol: UDP
            - name: rpcbind
              containerPort: 111
            - name: rpcbind-udp
              containerPort: 111
              protocol: UDP
            - name: statd
              containerPort: 662
            - name: statd-udp
              containerPort: 662
              protocol: UDP
          securityContext:
            capabilities:
              add:
                - DAC_READ_SEARCH
                - SYS_RESOURCE
          args:
            - "-provisioner=example.com/nfs"
          env:
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: SERVICE_NAME
              value: nfs-provisioner
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          imagePullPolicy: "IfNotPresent"
          volumeMounts:
            - name: export-volume
              mountPath: /export
      volumes:
        - name: export-volume
          hostPath:
            path: /tmp/nfs-provisioner
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
  - apiGroups: [""]
    resources: ["services", "endpoints"]
    verbs: ["get"]
  - apiGroups: ["extensions"]
    resources: ["podsecuritypolicies"]
    resourceNames: ["nfs-provisioner"]
    verbs: ["use"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-provisioner
     # replace with namespace where provisioner is deployed
    namespace: default
roleRef:
  kind: ClusterRole
  name: nfs-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-provisioner
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-provisioner
    # replace with namespace where provisioner is deployed
    namespace: default
roleRef:
  kind: Role
  name: leader-locking-nfs-provisioner
  apiGroup: rbac.authorization.k8s.io
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: example-nfs
provisioner: example.com/nfs
mountOptions:
  - vers=4.1
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nfs-pvc
  annotations:
    volume.beta.kubernetes.io/storage-class: "example-nfs"
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 8Gi
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: test-workload
  namespace: default
spec:
  serviceName: test-workload
  replicas: 1
  selector:
    matchLabels:
      app: test-workload
  podManagementPolicy: Parallel
  template:
    metadata:
      labels:
        app: test-workload
    spec:
      nodeName: photon-02 # Replace this one by one of the node in your cluster to force all pods of this workload on that node
      containers:
        - name: kbench-1
          image: phanle1010/kbench:broadcom
          imagePullPolicy: IfNotPresent
          env:
            - name: MODE
              value: "random-write-iops"
            - name: OUTPUT
              value: /tmp/pvc1
            - name: FILE_NAME
              value: "/pvc1/test.img"
            - name: SIZE
              value: "1638M" # 80% of the pvc-1's size
            - name: CPU_IDLE_PROF
              value: "disabled"
            - name: SKIP_PARSE
              value: "true"
            - name: LONG_RUN
              value: "true"
            - name: RATE_IOPS
              value: "1000"
          volumeMounts:
            - name: pvc1
              mountPath: /pvc1/
      volumes:
        - name: pvc1
          persistentVolumeClaim:
            claimName: nfs-pvc

Result: NFS mounts are NOT frozen
Case 2: Test with Kubernetes external NFS provisioner + Longhorn RWO PVC ✅

This is the yaml to deploy this setup

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs-provisioner-storage # longhorn backing pvc
spec:
  storageClassName: longhorn
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: "10Gi" # make this 10% bigger then the workload pvc
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-provisioner
---
kind: Service
apiVersion: v1
metadata:
  name: nfs-provisioner
  labels:
    app: nfs-provisioner
spec:
  ports:
    - name: nfs
      port: 2049
    - name: nfs-udp
      port: 2049
      protocol: UDP
    - name: nlockmgr
      port: 32803
    - name: nlockmgr-udp
      port: 32803
      protocol: UDP
    - name: mountd
      port: 20048
    - name: mountd-udp
      port: 20048
      protocol: UDP
    - name: rquotad
      port: 875
    - name: rquotad-udp
      port: 875
      protocol: UDP
    - name: rpcbind
      port: 111
    - name: rpcbind-udp
      port: 111
      protocol: UDP
    - name: statd
      port: 662
    - name: statd-udp
      port: 662
      protocol: UDP
  selector:
    app: nfs-provisioner
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: nfs-provisioner
spec:
  selector:
    matchLabels:
      app: nfs-provisioner
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: nfs-provisioner
    spec:
      nodeName: photon-02 # Replace this one by one of the node in your cluster to force all pods of this workload on that node
      serviceAccount: nfs-provisioner
      containers:
        - name: nfs-provisioner
          image: registry.k8s.io/sig-storage/nfs-provisioner:v4.0.8
          ports:
            - name: nfs
              containerPort: 2049
            - name: nfs-udp
              containerPort: 2049
              protocol: UDP
            - name: nlockmgr
              containerPort: 32803
            - name: nlockmgr-udp
              containerPort: 32803
              protocol: UDP
            - name: mountd
              containerPort: 20048
            - name: mountd-udp
              containerPort: 20048
              protocol: UDP
            - name: rquotad
              containerPort: 875
            - name: rquotad-udp
              containerPort: 875
              protocol: UDP
            - name: rpcbind
              containerPort: 111
            - name: rpcbind-udp
              containerPort: 111
              protocol: UDP
            - name: statd
              containerPort: 662
            - name: statd-udp
              containerPort: 662
              protocol: UDP
          securityContext:
            capabilities:
              add:
                - DAC_READ_SEARCH
                - SYS_RESOURCE
          args:
            - "-provisioner=example.com/nfs"
          env:
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: SERVICE_NAME
              value: nfs-provisioner
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          imagePullPolicy: "IfNotPresent"
          volumeMounts:
            - name: export-volume
              mountPath: /export
      volumes:
        # - name: export-volume
        #   hostPath:
        #     path: /tmp/nfs-provisioner
        - name: export-volume
          persistentVolumeClaim:
            claimName: nfs-provisioner-storage
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
  - apiGroups: [""]
    resources: ["services", "endpoints"]
    verbs: ["get"]
  - apiGroups: ["extensions"]
    resources: ["podsecuritypolicies"]
    resourceNames: ["nfs-provisioner"]
    verbs: ["use"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-provisioner
     # replace with namespace where provisioner is deployed
    namespace: default
roleRef:
  kind: ClusterRole
  name: nfs-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-provisioner
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-provisioner
    # replace with namespace where provisioner is deployed
    namespace: default
roleRef:
  kind: Role
  name: leader-locking-nfs-provisioner
  apiGroup: rbac.authorization.k8s.io
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: example-nfs
provisioner: example.com/nfs
mountOptions:
  - vers=4.1
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nfs-pvc
  annotations:
    volume.beta.kubernetes.io/storage-class: "example-nfs"
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 8Gi
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: test-workload
  namespace: default
spec:
  serviceName: test-workload
  replicas: 1
  selector:
    matchLabels:
      app: test-workload
  podManagementPolicy: Parallel
  template:
    metadata:
      labels:
        app: test-workload
    spec:
      nodeName: photon-02 # Replace this one by one of the node in your cluster to force all pods of this workload on that node
      containers:
        - name: kbench-1
          image: phanle1010/kbench:broadcom
          imagePullPolicy: IfNotPresent
          env:
            - name: MODE
              value: "random-write-iops"
            - name: OUTPUT
              value: /tmp/pvc1
            - name: FILE_NAME
              value: "/pvc1/test.img"
            - name: SIZE
              value: "1638M" # 80% of the pvc-1's size
            - name: CPU_IDLE_PROF
              value: "disabled"
            - name: SKIP_PARSE
              value: "true"
            - name: LONG_RUN
              value: "true"
            - name: RATE_IOPS
              value: "1000"
          volumeMounts:
            - name: pvc1
              mountPath: /pvc1/
      volumes:
        - name: pvc1
          persistentVolumeClaim:
            claimName: nfs-pvc

Result: NFS mounts are NOT frozen
Case 3: Test with Longhorn RWX PVC ❌

Result: NFS mounts are frozen
Analsys:

From the test results, we can see that the difference between frozen case (case 3) and non frozen cases (case 1 and case 2) is at Longhorn CSI/Longhorn share manager VS Kubernetes external NFS provisioner + nfs-ganesha.
Since the workload are mounting the volume OK at the beginning, the Longhorn CSI plugin might not be the culprit. So Is it possible that something is wrong in our Longhorn share-manager?
The main component of Longhorn share-manager (frozen case) and Kubernetes external NFS provisioner (non-frozen cases)  is the nfs-ganesha. I see that Kubernetes external NFS provisioner is using nfs-ganesha: V4.0.8 while Longhorn share manager is using nfs-ganesha: V5.7:# Kubernetes external NFS provisioner
bash-5.2# ./usr/local/bin/ganesha.nfsd -v
NFS-Ganesha Release = V4.0.8

# Kubernetes external NFS provisioner
share-manager-pvc-0bcdb279-04f7-4a8c-a83b-d301c0e37237:/ # ./usr/local/bin/ganesha.nfsd -v
NFS-Ganesha Release = V5.7

In order to confirm whether nfs-ganesha version difference is the culprit, I revert the share-manager to use nfs-ganesha: V4.2.3:share-manager-pvc-adf07290-9d3b-4683-8637-da6cedf8fa6b:/ # ./usr/local/bin/ganesha.nfsd -v
NFS-Ganesha Release = V4.2.3

Result: NFS mounts are NOT frozen
I am doing more investigating but I suspect that the root cause might be related to:

https://github.com/nfs-ganesha/nfs-ganesha/issues/1063
https://github.com/nfs-ganesha/nfs-ganesha/issues/1063

What's NFS server used in NFS provisioned? Is it nfs ganesha? If yes, is the used version the same as us?

What's NFS server used in NFS provisioned? Is it nfs ganesha? If yes, is the used version the same as us?

Hi @innobead! I see that Kubernetes external NFS provisioner is using nfs-ganesha: V4.0.8 while Longhorn share manager is using nfs-ganesha: V5.7:
# Kubernetes external NFS provisioner
bash-5.2# ./usr/local/bin/ganesha.nfsd -v
NFS-Ganesha Release = V4.0.8

# Kubernetes external NFS provisioner
share-manager-pvc-0bcdb279-04f7-4a8c-a83b-d301c0e37237:/ # ./usr/local/bin/ganesha.nfsd -v
NFS-Ganesha Release = V5.7

The latest version of nfs-ganesha is v5.7 hitting the fronzen issue. The fix will be included in v6.0.
Downgranding our nfs-ganesha temporarily is on choice. IIRC, v4.x we used before didn't introduce issues after fixing the crashing issue. cc @james-munson
Or, backport the fixed in the thread to v5.7 by ourselves is feasible as well.

Thanks @derekbit , Yeah, I was also looking at the ticket https://github.com/nfs-ganesha/nfs-ganesha/issues/1102 in the above analysis as well! Agree that we either need to pump the  nfs-ganesha version or revert it to avoid the hanging issue

Let's do this if reasonable:

backport the patch to our downstream 5.x for our existing Longhorn releases like 1.6, 1.5
upgrade to 6 in Longhorn 1.7.0

@PhanLe1010 After patching, I think we can continue using the RT-enabled distro to continue the testing.

@PhanLe1010 I've patched the fixes to nfs-ganesha v5.7 and created a share-image v1.6.0 with it. You can test it, derekbit/longhorn-share-manager:v1.6.0-fix-hang, to see if the frozen issue is gone.

I think we need to be careful because we are cherry picking from upstream PRs but the upstream fixed PRs have not been merged:

https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/1188187
https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/1188188
They are actively reviewing it so maybe better to wait for them to merge first before we add this to our stable releases. Hope that they will merge in the next few days though.

In the meantime, yes, I will do the report with the upstream patches.

Update:

The hanging issue actually also happens in Ubuntu. It just takes a lot longer time to stuck. In my case, it is stuck after letting the test run for a night.
After applying the upstream patch, Photon RT/ Photon non-RT/Ubuntu no longer have the frozen issue

Update:

The hanging issue actually also happens in Ubuntu. It just takes a lot longer time to stuck. In my case, it is stuck after letting the test run for a night.

So, the information in https://github.com/longhorn/longhorn/issues/8253#issuecomment-2021632406 is not correct, right? RWX volumes in Ubuntu are frozen as well?

After applying the (pending, not merged yet) upstream patches, Photon RT/ Photon non-RT/Ubuntu no longer have the frozen issue

So, the information in #8253 (comment) is not correct, right? RWX volumes in Ubuntu are frozen as well?

Yeah, when I run the test on Ubuntu over night, nfs mounts are also frozen. It takes much longer time on Ubuntu. I am not sure why. Photon OS hit it after 1-3 minutes.

Update:
Upstream PR changed its status to ready to submit but has not merged yet. Perhaps, hope that it will be merged in 1 or 2 more days

Remove backport 1.5.x, because v1.5.x is using nfs-ganesh v4.2.3 which is not impacted by the frozen issue.
Ref: https://github.com/longhorn/longhorn/issues/8253#issuecomment-2027851926


Create a Dashboard & drilldown links to see Policies attached to a Gateway/HTTPRoute and vice/versa

Use Cases

As an MGC Admin, I want to view all managed gateways and the associated policies
As an MGC Admin, I want a view of Policies to see what Gateways or HTTPRoutes they are attached

Details
There are options to solve this for the MGC use cases above, however they will likely be very specific to MGC.
New metrics will be required, and a set of new dashboards.
To provide drill down links, existing dashboards provided by #438 would need to be modified/forked from the upstream, which isn't ideal.
It would be best to see the outcome of https://github.com/david-martin/gateway-api-state-metrics/issues/4 first before taking on this issue as there may be a more upstream/generic friendly way of doing this without having to modify what's pulled in from upstream.

This issue is stale because it has been open for 60 days with no activity.

This issue was closed because it has been inactive for 30 days since being marked as stale.


Git guide

Nice git guide which we could include/re-write in here https://zulip.readthedocs.io/en/latest/git/overview.html

closed in favour of #14