Not stable Thread network topology in a big > 100 nodes network

Hi, first I must say it is very critical issue for us and I would need any urgent help if possible.
The Setup:
We have a hardware setup of 50x nRF5280 + one NCP and a border router.
Currently we test them in a line topology:
BR (NCP) --- NODE --- NODE --- NODE --- ... --- NODE
Each link to a nearest neighbour is between 70 and 80 dbm, Nodes can see around 3-4 neighbour nodes (RSSI: 80-95).
All nodes send keep-alive confirmable coap to the NCP Mesh Local address every 1 minute.
Observations:
Thread network topology becomes stable quite quickly (within single minutes),
Keep alives from all nodes reach border router which is fine.
Communication can be stable for hours with rare keep alives being lost.
When we push some messages into the network ( Mesh multicast + confirmable command to every node Mesh local address ) everything goes crazy.
Keep alives are all gone or almost all gone, we observed NCP parent router to change its RLOC (d000 ->bc00), looks like thread topology is changing/partitioning?.
Short higher traffic can break the network within seconds but it takes around 10 minutes to recover from that situation. After several minutes everything gets back to normal.
Notes
We didn't observe similar issue when topology wasn't linear (less hops).
We will provide more details ASAP like sniffer pcap, some test results, charts and anything that could help.
Questions
Do you have any thoughts why this is happening?
Are there any timeouts we could tune to make it better?
Have you worked with similar hardware setups with many nodes and many hops?
@jwhui I would be greatful if you could look into the issue

@mwilniewiec, did you get a chance to test with a larger value for OPENTHREAD_CONFIG_TMF_ADDRESS_CACHE_ENTRIES? Any other observations you can share?

Yes, we tried that its hard to say if bumping OPENTHREAD_CONFIG_TMF_ADDRESS_CACHE_ENTRIES from 20 to 200 did anything good but we left it that way.
We did some investigation with the sniffer and we found that our multicast, actually multicast answers were causing too big load on the network. Every node upon receiving multicast message was responding with multiple packets on NCP Mesh Local address. Additionally packets sent to Mesh Local were lost causing sending a/aq and a/an and causing even bigger mess in the network.
I believe traffic was so big that thread network couldn't handle that and it takes a few minutes to recover.
Currently we removed the multicast and use 50x unicasts to every node with 200ms delay for each and we try to reduce answers to the minimum. That helped and everything is more stable and reliable. We also plan to get back to sending messages to NCP ALOC instead of Mesh Local when we confirm that #4472 #4424 are fixed.
Still if its just a matter of heavy network load I believe there could be some improvements in openthread to favor MLE or other critical messages over application coap.
We are still investigating some issues and with such setup strange things happen from time to time. I will provide you with more info/issues when I'll be able to describe them properly.

@mwilniewiec, any new information you can share?

I can share some interesting information not directly related with that issue:
Currently we have a test setup of 140 hardware nodes and a Border router with NCP.
Every node can see only a few neighbors (RSSI ~75dBm) so we have more linear physical topology.
When we find anything that can be described we create a separate issue for that, there were some already :)
Network is mostly stable. I still must sniff a bit more to verify how many topology changes we have.
Our application CoAP messages work fine as long as we use them carefully without flooding the network.
The number of 140+ seems to be a number where we start having problems with wpantund and NCP. Hangups, crashes, permanent death. This is under investigation now and some fixes were already done: buffer size tuning, #4641 .
I am quite surprised with the fact that we observe high packet loss (even on smaller setup like 5-10pcs). If we set them in a linear topology we lose lots of CoAP messages on the application layer even when those packets are very small and fit into 1 frame. Instead of receiving an answer immediately we often must wait for multiple retries.
This issue is most striking during thread joining process. There are multiple DTLS handshake messages which are quite big and with high packet loss its almost impossible for a node to join over multiple hops. This can take forever or at least hours of join tries.
I must do some more investigation to find out where the problem really lies. Is the signal too weak, are there too many collisions or is it something totally different.
Please share any of your thoughts.

Please also note, that this environment is using the long routes extension, allowing us to have 32 hops in the partition, up from standard 4.

@jwhui I did a bit more investigation on network stability.
In such a 140 nodes network we have huge numbers in parent changes, better partition attempts.
It  isn't helping at all when nodes change their RLOCs.
Scale of problem:

MLE Link, Parent, Child req/res/accept are even 33% of all sniffed packets while MLE Advertisement is 3%
Nodes change their RLOC every few minutes (noticed while working with single nodes CLI etc)
Nodes get into detached state from 1 to 5 times per hour (taken from thread counters)
Better partition attempts occur up to 100 times per hour

Can you see anything that can be checked, changed to limit the problem somehow?

I am quite surprised with the fact that we observe high packet loss (even on smaller setup like 5-10pcs). If we set them in a linear topology we lose lots of CoAP messages on the application layer even when those packets are very small and fit into 1 frame. Instead of receiving an answer immediately we often must wait for multiple retries.

Have you tried increasing OPENTHREAD_CONFIG_MAC_MAX_FRAME_RETRIES_DIRECT to a larger value? In Google Nest products, we set this to 15.

In such a 140 nodes network we have huge numbers in parent changes, better partition attempts.
It isn't helping at all when nodes change their RLOCs.
Scale of problem:

MLE Link, Parent, Child req/res/accept are even 33% of all sniffed packets while MLE Advertisement is 3% and Radio ACK is 30%
Nodes change their RLOC every few minutes (more personal feeling while working with single nodes CLI)
Nodes get into detached state from 1 to 5 times per hour (taken from thread counters)
Better partition attempts occur up to 100 times per hour

Can you see anything that can be checked, changed to limit the problem somehow?

The above seems to indicate that the network is not converging to a single partition. Some things to look for:

Device reboot: normally the device should try to restore its previous state after a reboot. If the device cannot communicate with its neighbors, it will establish its own partition. Note that we recently fixed an issue that could address some poor performance here: #4753.

Failure to receive sequence number updates from Leader: Thread routers determine connectivity to the leader by periodically receiving sequence number updates from the leader. If the Thread router is not receiving sequence number updates, it will detach and could form a new partition. Failure to receive updates generally due to lost MLE Advertisement messages. Sometimes the issue is more subtle. For example, in #2755, we had to decrease kAdvertiseIntervalMax to ensure that the information propagates with a more deterministic period, especially in large networks.

Failure to become a router after attaching as a end device: a router-capable device must communicate a CoAP exchange with the Leader to obtain a valid router id. If the device is not able to communicate with the leader, any neighboring device trying to attach will not be able to and may form their own partitions.

Not sure if any of the above helps, just trying to brainstorm around what may be going on.

@jwhui thanks for your ideas. I did proposed changes and will test the behavior soon. Our kAdvertiseIntervalMax is already 5 as we use Long routes in our setup.
From what I read there is a 120 timeout for receiving MLE over hops. What woud you say to increase it to 180 or 240 in our setup? I believe such changes could make the network more stable too.

@jwhui I did some more investigation on devices which stay in a different partition unable to merge.
Im not sure yet what is the reason of initial disconnection but we have such case:

Device sends a Parent Request for some reason (already in different partiton, lost his parent, lost a leader?)
bla bla
fdfdsfds

@jwhui I did some more investigation on devices which stay in a different partition unable to merge.
Im not sure yet what is the reason of initial disconnection but we have such case:

We have already 32 routers in our 140 nodes network, routers are not downgrading possibly due to not having 7 neighbors with (link quality 2)
Device sends a Parent Request for some reason (already in different partiton, lost his parent, lost a leader?)
Initial parent request asks only routers for an answer
Node is collecting parent responses and none of the routers has link quality 3
Node sends second parent request to routers and REEDs
We have lots of answers and the best match (link 3) is some child
We ask that child to be our parent but it fails with ADDR_SOL because of max routers in the network
We get no answer for Child ID Request
The same path can be replied forever and node is always trying to connect to the same child (creating own partition in the meanwhile?)
Eventually node will find some link 3 router or would not receive a parent response from a link 3 child successfully attaching to some router.

This issue is causing very long disconnections and big radio traffic due to Parent Responses (even 25% of the whole traffic).
Im planning to patch OT allowing to attach to a router with link quality < 3 when there are already 32 routers in the network. Its better for us to have link 2 parent than no parent.
Additionally Im thinking about decreasing kMinDowngradeNeighbors from 7 to lets say 3.
Please let me know what you think.

Additionally Im thinking about decreasing kMinDowngradeNeighbors from 7 to lets say 3.

We have observed a similar situation in OTNS simulation that two or more partitions can not merge because there are already 32 routers in the major partition and routers do not downgrade for a very long duration causing significant traffic. @librasungirl
The script for simulating partition merging is form_partition.py.
I simulated 8x8 nodes for 10 times, and got following result:

8x8 nodes, kMinDowngradeNeighbors=7:

Simulation
Duration (s)
Partition Num

1#
1800
4

2#
20105
7

3#
27
0

4#
1610
5

5#
19
0

6#
263
3

7#
607
5

8#
7533
9

9#
>10 hours
take too long, seems never end

10#
604
3

The first value of each column is the duration it takes for 64 nodes to form one partition.
The second value of each column is the number of different partitions that showed up in the simulation after the initial 30 seconds. This value can be an indication of the stability of the network. Partitions in the initial 30 seconds are ignored because lots of nodes become Leader at the same time at the beginning of simulation.

There are 3 cases that took hours before merging to one partition. In these cases, the nodes at the boundary of the major partition (the partiton containing most nodes) are all REEDs that unable to become Routers.
I changed kMinDowngradeNeighbors to 3 and simulated again:

8x8 nodes, kMinDowngradeNeighbors=3:

Simulation
Duration (s)
Partition Num

1#
539
26

2#
480
17

3#
158
6

4#
310
7

5#
2606
116

6#
510
35

7#
893
48

8#
254
10

9#
649
49

10#
531
44

My intuitional feeling is that the major partition is much less stable with much more partitions appeared during the simulation. However, there is no case that take extreme long time to form one partition like case 2#, 8#, 9# of kMinDowngradeNeighbors == 7. The data above also confirmed that.
I would suggest us analyze more on the impact of different kMinDowngradeNeighbors values (maybe using OTNS) and explore other possible solutions.
@jwhui @mwilniewiec

I got coredumps on macOS when simulating 8x8 nodes with kMinDowngradeNeighbors == 4 and OPENTHREAD_CONFIG_MLE_LONG_ROUTES_ENABLE=1
* thread #1, stop reason = signal SIGSTOP
  * frame #0: 0x00007fff5ecf82c2 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff5edb3bf1 libsystem_pthread.dylib`pthread_kill + 284
    frame #2: 0x00007fff5ec626a6 libsystem_c.dylib`abort + 127
    frame #3: 0x00007fff5ec2b20d libsystem_c.dylib`__assert_rtn + 324
    frame #4: 0x0000000108bc9fc8 ot-cli-ftd`ot::Mle::MleRouter::HandleParentRequest(this=<unavailable>, aMessage=<unavailable>, aMessageInfo=<unavailable>) at mle_router.cpp:1622:5 [opt]
    frame #5: 0x0000000108bc4221 ot-cli-ftd`ot::Mle::Mle::HandleUdpReceive(this=0x0000000108c19cc8, aMessage=0x0000000108c16790, aMessageInfo=0x00007ffee7086b90) at mle.cpp:2889:26 [opt]
    frame #6: 0x0000000108bb3c49 ot-cli-ftd`ot::Ip6::Udp::HandleMessage(this=0x0000000108c18ff0, aMessage=0x0000000108c16790, aMessageInfo=0x00007ffee7086b90) at udp6.cpp:336:5 [opt]
    frame #7: 0x0000000108bae163 ot-cli-ftd`ot::Ip6::Ip6::HandleDatagram(ot::Message&, ot::Ip6::Netif*, void const*, bool) [inlined] ot::Ip6::Ip6::HandlePayload(this=0x0000000108c18f98, aMessage=<unavailable>, aMessageInfo=0x00000000000002ff, aIpProto='\x11') at ip6.cpp:1002:22 [opt]
    frame #8: 0x0000000108bae13d ot-cli-ftd`ot::Ip6::Ip6::HandleDatagram(this=0x0000000108c18f98, aMessage=0x0000000108c16790, aNetif=0x0000000108c19108, aLinkMessageInfo=<unavailable>, aFromNcpHost=false) at ip6.cpp:1253 [opt]
    frame #9: 0x0000000108bbd4cd ot-cli-ftd`ot::MeshForwarder::HandleLowpanHC(unsigned char const*, unsigned short, ot::Mac::Address const&, ot::Mac::Address const&, otThreadLinkInfo const&) [inlined] ot::MeshForwarder::HandleDatagram(this=<unavailable>, aMessage=<unavailable>, aLinkInfo=0x00007ffee7086cb0, aMacSource=<unavailable>) at mesh_forwarder.cpp:1337:28 [opt]
    frame #10: 0x0000000108bbd4a0 ot-cli-ftd`ot::MeshForwarder::HandleLowpanHC(this=0x0000000108c19b70, aFrame="\x7f;\x02?MLML?3", aFrameLength=<unavailable>, aMacSource=0x00007ffee7086ca0, aMacDest=0x00007ffee7086c90, aLinkInfo=0x00007ffee7086cb0) at mesh_forwarder.cpp:1311 [opt]
    frame #11: 0x0000000108bbd0d7 ot-cli-ftd`ot::MeshForwarder::HandleReceivedFrame(this=<unavailable>, aFrame=<unavailable>) at mesh_forwarder.cpp:1041:13 [opt]
    frame #12: 0x0000000108b99655 ot-cli-ftd`ot::Mac::Mac::HandleReceivedFrame(this=0x0000000108c198c8, aFrame=<unavailable>, aError=<unavailable>) at mac.cpp:1801:26 [opt]
    frame #13: 0x0000000108bdd7dc ot-cli-ftd`platformRadioReceive [inlined] radioProcessFrame(aInstance=<unavailable>) at radio.c:770:13 [opt]
    frame #14: 0x0000000108bdd79e ot-cli-ftd`platformRadioReceive [inlined] radioReceive(aInstance=0x0000000108c10320) at radio.c:500 [opt]
    frame #15: 0x0000000108bdd6b9 ot-cli-ftd`platformRadioReceive(aInstance=<unavailable>, aBuf=<unavailable>, aBufLength=<unavailable>) at radio.c:596 [opt]
    frame #16: 0x0000000108bded78 ot-cli-ftd`otSysProcessDrivers [inlined] receiveEvent(aInstance=<unavailable>) at platform-sim.c:111:9 [opt]
    frame #17: 0x0000000108bded18 ot-cli-ftd`otSysProcessDrivers(aInstance=<unavailable>) at platform-sim.c:304 [opt]
    frame #18: 0x0000000108b78b20 ot-cli-ftd`main(argc=2, argv=0x00007ffee7087808) at main.c:114:9 [opt]
    frame #19: 0x00007fff5ebbd3d5 libdyld.dylib`start + 1
    frame #20: 0x00007fff5ebbd3d5 libdyld.dylib`start + 1

This simulation never coredumped without LONG_ROUTES. So, I would expect it has something to do with the LONG_ROUTES extension.
Will investigate later.

@simonlingoogle, thanks for sharing your findings.
@mwilniewiec, can you confirm again what your topology is like? Your original post indicated a long linear topology where devices only had 3-4 neighbors. In that case, I can certainly see why kMinDowngradeNeighbors = 7 is too large. But with 140 nodes, maybe the density is higher. How many neighbors per device and total hops do you expect in your topology?

We are using a simulation setup, trying to keep RSSI of closest nodes ~75dbm. We did tests with different topologies with different number of devices. We started from 50 linear but as we were growing to 140 it was a bit hard to keep them linear as it took a lot of space.
Currently Im doing tests on 140 hardware devices where currently topology is more or less something like:
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNN       BR NNNNNNNNNNNNNNNNNNNNNNNNNN
                            NNNNNNNNNNNNNNNNNNNNNN
                              NNNNNNNNNNNNNNNNNNNNN

N is a node and BR is a border router with NCP. We track their status from the cloud. When device is in a different partition than NCP it is disconnected for us.
Example neighbor table and scan results from 2 different, random nodes:
> neighbor table
| Role | RLOC16 | Age | Avg RSSI | Last RSSI |R|S|D|N| Extended MAC     |
+------+--------+-----+----------+-----------+-+-+-+-+------------------+
|   R  | 0x1800 |   0 |      -91 |       -93 |1|0|1|1| da9c8f0f52b3b369 |
|   R  | 0x5c00 |   3 |      -77 |       -78 |1|0|1|1| 8e79af0446e57223 |
|   R  | 0x9000 |   3 |      -75 |       -75 |1|0|1|1| eef4fbd8888d4d91 |
|   R  | 0xec00 |   1 |      -67 |       -67 |1|0|1|1| fe4c44b7a4cb5c28 |

> scan
| J | Network Name     | Extended PAN     | PAN  | MAC Address      | Ch | dBm | LQI |
+---+------------------+------------------+------+------------------+----+-----+-----+
> | 1 | biotcloud-40     | 3c9b17163421fa72 | 6acb | 0a07c52fdc1b10f4 | 22 | -82 |  48 |
| 1 | biotcloud-40     | 3c9b17163421fa72 | 6acb | 021ac9f6b665dca2 | 22 | -70 |  92 |
| 1 | biotcloud-40     | 3c9b17163421fa72 | 6acb | ea0680f5124bb38e | 22 | -87 |  20 |
| 1 | biotcloud-40     | 3c9b17163421fa72 | 6acb | 1abe48e43c442031 | 22 | -84 |  32 |
| 1 | biotcloud-40     | 3c9b17163421fa72 | 6acb | fe4c44b7a4cb5c28 | 22 | -92 |   4 |
| 1 | biotcloud-40     | 3c9b17163421fa72 | 6acb | be25a412dc0fcf91 | 22 | -87 |  20 |

In general I see that devices link with 2-4 neighbors in every direction.
Ive also seen cases where node was a child of a parent 10 devices away from it with rssi 90

Hi again,
I have some more news and more questions to ask.
I created a testing machinery for 140 nodes farm, did some improvements and before and after tests.
My general idea was to sniff packets that appear too often, understand and reduce them + increase default radio retries for better overall communication.
Changes that were done:

Changed OPENTHREAD_CONFIG_MAC_MAX_FRAME_RETRIES_DIRECT from 3 to 8
integrated #3358 and set OPENTHREAD_CONFIG_MLE_CHILD_ROUTER_LINKS = 1
intergrated #4753
Changed HandleParentResponse passing through a linkQuality < 3 router when it has kMaxRouters in connectivity TLV
Changed kMinDowngradeNeighbors from 7 to 4
Limited and randomized application packets
Added COAP re-transmissions on border router br #396

With those changes I can see much less packets sent over network, partitions are much more stable with no nodes leaving partitions for longer period of time (seen before adding point 4).
General packets delivery BR -> nodes is around 3 times better and reliable than before.
There is still room for improvement and I have some ideas like:

Stop using border router ML-EID address (must transfer RLOC packets from ncp->linux which can be tricky) + nodes will address border router by its service ALOC
Go higher with radio retries, @jwhui suggested 15
Upfront precaching of EID-RLOC entries on NCP not to flood network with a/aq multicasts when suddenly a command is sent to multiple or all nodes.

My newest finding is that I see lot of Link Requests from multiple routers to a router sending an advertisement. 13% of all traffic now are Link Requests (10 000) with almost no Link Accepts (70).
Any thoughts why this could happen?

With those changes I can see much less packets sent over network, partitions are much more stable with no nodes leaving partitions for longer period of time (seen before adding point 4).
General packets delivery BR -> nodes is around 3 times better and reliable than before.

@mwilniewiec,  great to hear that your network performance is improving!

There is still room for improvement and I have some ideas like:

Stop using border router ML-EID address (must transfer RLOC packets from ncp->linux which can be tricky) + nodes will address border router by its service ALOC

Makes sense. This would avoid the EID-to-RLOC lookup.
The issue with RLOC addresses becomes much simpler when using the RCP model (which puts OpenThread on the host side). ot-br-posix master now defaults to RCP mode and we have updated the Border Router guide on openthread.io to match.

Go higher with radio retries, @jwhui suggested 15

This depends on the loss rates you are experiencing.

Upfront precaching of EID-RLOC entries on NCP not to flood network with a/aq multicasts when suddenly a command is sent to multiple or all nodes.

Last year, we added the ability to update the EID-to-RLOC cache upon receiving a message (#4091). This allows devices to update a receiver's cache simply by sending a message.

My newest finding is that I see lot of Link Requests from multiple routers to a router sending an advertisement. 13% of all traffic now are Link Requests (10 000) with almost no Link Accepts (70).
Any thoughts why this could happen?

I wonder if this is due to asymmetric link qualities. Are the Link Requests mostly coming from active routers, router-eligible end devices (REEDs), or a mix? Looking at the code, active routers only send Link Requests if the received Advertisement is above some RSSI threshold. However, REEDs do not apply the same - we could look into apply the same RSSI filter if we think this will improve things.

@jwhui there were lot of Link requests from REEDs before I applied point 2 from the list.
Now its only active routers that send link requests from what I see. I also dont know why I can't see radio ACKs for the link requests in the sniff. Wireshark screen attached.

@jwhui I tried to investigate those strange Link Requests without a response a bit more.
From what I see that are a few routers in my setup that are getting those link requests on every Advertisement (mostly every 5 sec) and not responding to it (also radio ACKS are missing). This is causing quite a lot of unnecessary traffic.
From the application perspective those devices work normally as expected (sending application packets properly, acking them). I also checked commands from CLI, bufferinfo, router table, everything looked fine. Routers that are requesting for a link are in the router table but are not in the neighbor table. Neighbor table has some other entries.
In the same sniff I also see other routers that work as expected. Sending Adverisement, responding to a Link request (if they get any) with a radio ACK and Link Accept.
All devices have the same software.
I also tried to reboot such a router to see if its persistent but after a reboot it acted a bit strage ...
Within 30 ms sent:

Link request to ff02:2
Two parent reqests (one only for routers, the other one for REEDs)
Data Response  to ff02:1 (with a new partition data)
Advertisement (saying its a new leader)
After that it started receiving Parent Responses, picked one parent and became a child

Do you know any fixes that were done in these areas?
Any idea what could be the reason of a router not responding to link requests?

Thanks for continuing to investigate and keeping us up to date on your findings!

@jwhui I tried to investigate those strange Link Requests without a response a bit more.
From what I see that are a few routers in my setup that are getting those link requests on every Advertisement (mostly every 5 sec) and not responding to it (also radio ACKS are missing). This is causing quite a lot of unnecessary traffic.

By "getting those link requests" can you tell that the device is actually receiving those (e.g. via logs)?

From the application perspective those devices work normally as expected (sending application packets properly, acking them). I also checked commands from CLI, bufferinfo, router table, everything looked fine. Routers that are requesting for a link are in the router table but are not in the neighbor table. Neighbor table has some other entries.

In the case of A sending a Link Request to B. Can you share whether A is also in B's router table? A and B should have each other in their router table.

In the same sniff I also see other routers that work as expected. Sending Adverisement, responding to a Link request (if they get any) with a radio ACK and Link Accept.
All devices have the same software.
I also tried to reboot such a router to see if its persistent but after a reboot it acted a bit strage ...

In the above example, are you rebooting A (the sender of Link Requests not receiving ACKs) or B (the device not responding to Link Requests) or both?

Within 30 ms sent:

Link request to ff02:2
Two parent reqests (one only for routers, the other one for REEDs)
Data Response  to ff02:1 (with a new partition data)
Advertisement (saying its a new leader)
After that it started receiving Parent Responses, picked one parent and became a child

The fact that Parent Responses are received only after becoming a Leader and sending Advertisement messages is not normal.

Were there any additional Parent Request messages sent between steps 2 and 4 above?

Did the Parent Response message have a Response TLV value matching the Challenge TLV value in the Parent Request from step 2?

What are the time delays between each of the messages? I'm wondering if the neighboring device is taking too long to respond to the request for some reason.

If possible, could you share the pcap file that demonstrates the above behavior?

Do you know any fixes that were done in these areas?
Any idea what could be the reason of a router not responding to link requests?

Unfortunately, nothing comes to the top of my mind. I think having logging information could help determine whether neighboring devices are receiving the Link Request and, if so, whether they are trying to send Link Accept messages in response.

I made more investigation today and I think I know where the problem with Link Requests lies.
As you said "asymmetric link qualities" can be an issue.
My setup is using different antennas and it seems that:

Router A is sendng Advertisement
Routers R1, R2, R2 are receiving advertisement with link margin > 10
Routers R1, R2, R3 are sending Link Request to A (with OPENTHREAD_CONFIG_MAC_MAX_FRAME_RETRIES_DIRECT retries)
Router A can't hear those requests and isn't sending even a radio ACK
Such traffic persists on every Advertisement sent by A

I tried to use a better antenna for any of R1, R2, R3 and suddenly A could hear a Link Request, responding with "Link Accept and Request". I'm not sure how this issue would look like in a real environment where antennas are the same for all devices.
I have a plan to introduce an improvement:
"Router A is not responding to Link Requests, do not send another Link Request for 5 minutes for that router (possibly with increasing backoff)"
This should reduce unnecessary Link requests from ~20% to almost 0. What do you think about it?
PS. When I look at one longer sniff with low application layer traffic it is showing:

47% - MLE

27% - Link Request
10 % - Advertisement
5% - Parent response

25% - Radio ACK
21% - Coap (mostly application ~19%)
5% Beacon

It looks like this Link Request is currently a biggest unnecessary traffic here.
According to the second topic:

Were there any additional Parent Request messages sent between steps 2 and 4 above?

No, 2 parent request one after another (Time: 28.098 & 28.104), more parent request a bit later after parent responses (28.70 for routers only and 29.45 for REEDs)

2 .Did the Parent Response message have a Response TLV value matching the Challenge TLV value in the Parent Request from step 2?

Yes, both parent requests have corresponding responses (matching Response TLV)

What are the time delays between each of the messages? I'm wondering if the neighboring device is taking too long to respond to the request for some reason.

Attached screen from the sniff, I can send full sniffs & keys over private channel if you'd like to look deeper. It can be interesting as there are 140 devices.

@mwilniewiec regarding the timings, I wonder if you can get logs with timestamps from the device which sends MLE Link Request, Parent Requests and Data Request right after the other without respecting the timings. One of the causes could be that there is a problem with alarm implementation, but I wonder if it can be caused by the application itself. Is it possible that the main loop is not executed constantly in your case but it is blocked for some time? That would explain why timers fired in this short time.
Some more questions..

Did you modify the CSMA/CA parameters?
Do you use any operating system e.g. FreeRTOS?
Can you see this timing problem on any boards, or it may be related to the particular hardware board?

@LuDuda thanks, good catch with blocked main task. I found that our code called otThreadSetEnabled(true), blocked from 0 to 4 seconds and started feeding thread in regular loop. It looks like this was the root-cause of strage packets during wakeup.
To answer your question:

Did you modify the CSMA/CA parameters? Currently we are using NRF_RADIO_CCA_MODE_CARRIER_AND_ED instead of NRF_RADIO_CCA_MODE_ED (planning to get back with some regression testing), we changed NRF_802154_CCA_ED_THRESHOLD_DEFAULT and NRF_802154_CCA_CORR_THRESHOLD_DEFAULT both from 20 to 10
Do you use any operating system e.g. FreeRTOS?, Yes we are using FreeRTOS

I would like to summarize where we are now with our big network testing.
We are still testing 140 Nodes with 1 NCP, we try to keep RSSI between nearest neighbors ~75dBm. Our test process is as follows:

All nodes send application ping to NCPs Mesh Local Address every 2 minutes (using their MeshLocal as a source address). This way NCP can have 140 entries in his EID-RLOC Cache (#4091 applied)
Border Router sends Confirmable COAP to all nodes Mesh Local addresses in 200ms intervals and passes the response to the cloud. Nodes response is quite big and fragmented into 3 frames.
Request is retried for all nodes that did not respond with increasing backoff staring from 30 sec.
Such test (sending requests to all nodes) is run multiple times every 10 minutes.

Improvements that were done, mainly reducing overal unnecessary traffic in order to have more room for application message retransmissions:

Changed OPENTHREAD_CONFIG_MAC_MAX_FRAME_RETRIES_DIRECT from 3 to 10
integrated #3358 and set OPENTHREAD_CONFIG_MLE_CHILD_ROUTER_LINKS = 1
intergrated #4753
Changed HandleParentResponse passing through a linkQuality < 3 router when it has kMaxRouters in connectivity TLV
Changed kMinDowngradeNeighbors from 7 to 4
Limited and randomized application packets
Added COAP re-transmissions on border router https://github.com/openthread/ot-br-posix/issues/396
Applied #4091 so that nodes and NCP shouldn't need to use a/aq multicast at all
Added backoff (up to 5 min) for link request to nodes that did not respond previously, see https://github.com/openthread/openthread/issues/4508#issuecomment-638268332

After all those improvements sending req-resp to all nodes works much better.
Previously we had around 40 from 140 responses in 10 minutes and many tests with 0 responses as NCP become it's own leader in a different partition unable to merge because of point 4 from the improvements list. We could also see that application pings were lost (node not seeing NCP).
Now we have mostly 140/140 responses in around 5 minutes with average response time ~2 min. When network is "idle" with only thread traffic + application pings everything works flawlessly, during req-resp tests nodes sometimes lose their connection for a few miniutes and comes back.
Some tests show us that we are able to get all 140 responses in 2 minutes and some fail with only 80 of 140 nodes responding in 10 minutes with many nodes losing connection to NCP.
I also did some testing with multiple regular antennas and part of the nodes & NCP seeing one another as neighbours. Such test works great with immediate 140 answers ~ 1 minute.
Im trying to understand where the problem lies when results are bad and what is different between tests. For now Im not assuming that radio environment can be so different in our test setup.
Im still observing that nodes are changing their RLOCs a lot during testing (every 10 minutes we have different RLOCs on several nodes) and this is not helping for sure. @jwhui any hints what to check, how to investigate RLOC changes?
Future plans:

Use muticast to Realm local as a first request with randomly distributed responses to reduce traffic.
Investigate RLOC changes and network partitioning during req-resp tests.
Investigate why we observe huge differences in results on the same software.
Put Thread NCP on border router host according to new OT architecture and recommendations.

@mwilniewiec Just one note, you mentioned that:

1. `Did you modify the CSMA/CA parameters?` Currently we are using NRF_RADIO_CCA_MODE_CARRIER_AND_ED instead of NRF_RADIO_CCA_MODE_ED (planning to get back with some regression testing), we changed NRF_802154_CCA_ED_THRESHOLD_DEFAULT and NRF_802154_CCA_CORR_THRESHOLD_DEFAULT both from 20 to 10

Changing the default value from 20 to 10 changes the CCA threshold from -72 dBm to -82 dBm which can significantly reduce the device's transmission opportunity (every ED result above -82 dBm blocks the transmission). Please ensure that it's expected behavior.

@konradderda our RSSI between devices in more 80tish than 70tish. If I understand that correctly having CCA enabled from -72dBm threshold would almost disable it totally. I believe that our setups can be noisy and it seems resonable to block transmission instead of of filling the ether fully. Still thanks for the note, we should actually compare both values.
PS. Counters in our test setup shows that blocking transmission due to CCA isn't a big issue:
TxTotal: 17617
    TxUnicast: 3900
    TxBroadcast: 13717
    TxAckRequested: 3709
    TxAcked: 3441
    TxNoAckRequested: 13908
    TxData: 17235
    TxDataPoll: 0
    TxBeacon: 191
    TxBeaconRequest: 191
    TxOther: 0
    TxRetry: 4176
    TxErrCca: 302
    TxErrBusyChannel: 8

@mwilniewiec , do you have any updates on this issue?

Lately we didn't do much testing as we achieved let's say "satisfying performance" and focused on different issues. One thing that I mentioned earlier is that we did a few tests that went almost perfect but lately its getting only worse and worse (even tests on the same software were showing generally worse results).
I think that we also recently added some bigger regression (for example we added a mutex for all OT API calls, this could be a reason) and  we are getting many disconnections during tests.
Additionally we successfully merged Nordic SDK 4.1 with newer OT and we are ready to make tests on this SW.
I'm planning to do more investigations very soon, why do we see a regression + test on SDK 4.1 if it improves anything and fixes bugs that we are still experiencing.
If it's OK for you I would keep the issue opened and we should all try to achieve better responsiveness in huge > 100 nodes OT networks. In the past we were expecting a lot more from OT and hardware setup as still we are struggling with the fact that lots of packets are lost + OT network breaks apart and needs to recover when there ale lots of packets moving around.

@mwilniewiec , thanks for the update. We should work together on improving performance whenever we can.

@jwhui another updade from my side. I did more investigation on frequently changing RLOCS and I did some improvements on OT constants.
From what Ive seen from logs and sniffs the flow was as folows:

There was a lot of traffic in our network due to sending messages
Child is waiting 4 minutes - 4 seconds to send Child Update Request to his parent with 3 retries (every 1 second)
While sending it fails on a NO_ACK because the ether is noisy. Sometimes it fails even after 1 or 0 retries because previously it sent other packet to the parent with NO_ACK.

I changed:
-    kMaxChildKeepAliveAttempts = 4, ///< Maximum keep alive attempts before attempting to reattach to a new Parent
+    kMaxChildKeepAliveAttempts = 6, ///< Maximum keep alive attempts before attempting to reattach to a new Parent

-    kUnicastRetransmissionDelay     = 1000,  ///< Base delay before retransmitting an MLE unicast.
+    kUnicastRetransmissionDelay     = 2000,  ///< Base delay before retransmitting an MLE unicast.

-    kMaxTransmissionCount           = 3,     ///< Maximum number of times an MLE message may be transmitted.
+    kMaxTransmissionCount           = 5,     ///< Maximum number of times an MLE message may be transmitted.

-    kFailedRouterTransmissions = 4,   ///< FAILED_ROUTER_TRANSMISSIONS
+    kFailedRouterTransmissions = 10,   ///< FAILED_ROUTER_TRANSMISSIONS

Where kFailedRouterTransmissions is the most valuable from what I understand.
Can you see any risk related with applying such configuration?
With those settings Network works better, isn't disconnecting so often.
Previously we had a snowball effect:

Lot of application traffic caused child to detach from parent
Detach caused lots of parent request/response
Detach caused eidcache being lost and needed a/aq a/an to get eid mapping for border router
RLOC change caused another a/aq a/an to get EID->RLOC for that node on our NCP
Too many packets and detached state caused app packets being lost and retried making more packets in the network
As a result we had a snowball where OT packets are fighting with application packets to reach the target.

This is a step forward but still more to be done. We are waiting for Nordic to analyse SDK 4.1 upgradability problem and we want to switch on that. Regards & a happy new year :)

Closing as I have no further progress for now. If I do so Ill open new issues with more detail.
Summary: Tested 140 hardware devices in a single network + 1 border router.
When devices send packets hourly or once every several minutes everything looks stable and works fine.
When we do stuff like sending many messages to different ML-EIDs at once or multicast with lots of responses or device messages are not evenly distributed and all want to send packet exactly 10 sec after startup things can go wrong.
I would expect OT topology to be a bit more stable as I see that RLOCs change too often and when everything reorganizes auto-healing can take a few minutes or more.
Still ... when it works - it works :)

Obviously we have some valuable configuration changes that work better in large-scale networks as in https://github.com/openthread/openthread/issues/4508#issuecomment-748853553
How do we adopt these changes to OT code that other may reuse them (without having to figure them out again) ?
Maybe we can add a document for recommended OT configurations in large-scale networks?
Thoughts? @mwilniewiec @jwhui

I pointed out multiple other improvements that I've done here:
https://github.com/openthread/openthread/issues/4508#issuecomment-652328664
Some changes from the list were applied only as local patches and never contributed to OT. Its hard to say if its still valid but it was helpful to reduce unnecessary traffic & get satisfying network performance.
For example:
4: https://github.com/openthread/openthread/issues/4508#issuecomment-615873005
9: https://github.com/openthread/openthread/issues/4508#issuecomment-638268332
Answering to your question I think that some configuration + good practices for large-scale networks would be helpful because people wanting to use it this way have no idea where to start. It also depends if it's your target at all.
In general Thread is presented as:

best way to connect and control products in the home and buildings

But ...We decided to use it for city infrastructure where number of devices and distances between them drastically change.
And... it works :)

@mwilniewiec Just one note, you mentioned that:

1. `Did you modify the CSMA/CA parameters?` Currently we are using NRF_RADIO_CCA_MODE_CARRIER_AND_ED instead of NRF_RADIO_CCA_MODE_ED (planning to get back with some regression testing), we changed NRF_802154_CCA_ED_THRESHOLD_DEFAULT and NRF_802154_CCA_CORR_THRESHOLD_DEFAULT both from 20 to 10

Changing the default value from 20 to 10 changes the CCA threshold from -72 dBm to -82 dBm which can significantly reduce the device's transmission opportunity (every ED result above -82 dBm blocks the transmission). Please ensure that it's expected behavior.

Hello I would like to point out that, according to NRF documentation, CCA register to dbm conversion is:
PRF[dBm] = ED_RSSIOFFS + ED_RSSISCALE x VALHARDWARE
For NRF52840, ED_RSSIOFFS = -92, ED_RSSISCALE=4
Value of 20 results in -12dBm, value 10 results in -52dBm.
Please correct me if I was wrong.


Added RRAM Cells 1T1R, 4T1R and 4T1R MUX2

Added 1T1R, 4T1R, and 4T1R-MUX2 in cells
Each cell has GDS, Spice file, and Schematic in SVG format

Thanks for the contribution! Could you tell me a little more about the origin of these files? Are they original works, or extractions from another source?

Hello, These are the primitive elements (Multiplexer) of RRAM-based FPGA design. This is research work from our lab Laboratory for NanoIntegrated Systems (LNIS) from the University of Utah. You can refer to our publications below for more information.
TCAS16
TETC17
JETCAS18

Hello @QuantamHD, are there any further questions on this PR?


[$250] [HOLD for payment 2024-05-06] Group split scan - Error appears for Merchant when clicking Split expense for the second time

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: 1.4.64-0
Reproducible in staging?: y
Reproducible in production?: n
If this was caught during regression testing, add the test name, ID and link from TestRail: n/a
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by: Applause internal team
Slack conversation:
Action Performed:

Go to staging.new.expensify.com
Go to FAB > Split expense > Scan.
Create split scan expense with a few users.
In group chat, click on the split preview.
Click Split expense.
Click Split expense again.

Expected Result:
There will be no error under Merchant field because Merchant is not a requirement for non-workspace expense.
Actual Result:
Error shows up under Merchant field when clicking Split expense for the second time.
Workaround:
unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

Screenshots/Videos
Add any screenshot/video evidence
https://github.com/Expensify/App/assets/43996225/d77d2a68-3f03-4eb1-aed1-5d43f837c459

View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~013158f9aa7567f244
Upwork Job ID:<PHONE_NUMBER>570207744
Last Price Increase: 2024-05-13

Issue OwnerCurrent Issue Owner: @

@johncschuster  FYI I haven't added the External label as I wasn't 100% sure about this issue. Please take a look and add the label if you agree it's a bug and can be handled by external contributors.

We think this bug might be related to #vip-split

@GandalfGwaihir
We should modify the condition to ensure that the error is set to true only if the merchant is required. So, an additional && check for isMerchantRequired needs to be added. This adjustment ensures that the error is set only when the merchant is both required and empty:
`diff --git a/src/components/MoneyRequestConfirmationList.tsx b/src/components/MoneyRequestConfirmationList.tsx
index 99901dd261..87d605a7a5 100755
@@ -577,7 +577,7 @@ function MoneyRequestConfirmationList({
if (selectedParticipants.length === 0) {
return;
}

       if ((isMerchantRequired && isMerchantEmpty) || (shouldDisplayFieldError && TransactionUtils.isMerchantMissing(transaction ?? null))) {

       if ((isMerchantRequired && isMerchantEmpty) || (isMerchantRequired && shouldDisplayFieldError && TransactionUtils.isMerchantMissing(transaction ?? null))) {
           setMerchantError(true);
           return;
       }

`

This is minor UX issue, demoting @koko57 seems like its coming from your PR

@mountiny As the follow-up PR wasn't merged yet I can fix it there

@mountiny PR opened: https://github.com/Expensify/App/pull/40868

Reviewing payment now

I think the only payment here is to pay $250 to @Ollyws for their review and testing

Applying the External label to create the job. @Ollyws, Upwork isn't returning you when I search for you because... Upwork 🙄 . Can you apply to the job and ping me? I'll get payment issued ASAP!

@Ollyws Aha! I found you. Your offer is here. Can you accept that?

@Ollyws, I see you've just been added to the list of contributors who can accept payment via NewDot. I'll go ahead and post a payment summary instead so you can be paid via that method.

Payment Summary
Contributor+: @Ollyws - $250 via Manual Request

@robertjchen can you complete the BZ checklist above?

$250 approved for @Ollyws

Working through checklist

Bumping this for Melvin. How's the checklist going, Robert?

@johncschuster Made my assessment, and left the last item for you 👍

Thanks @robertjchen! Looks like we can close this up!


Add option for encode-marcXml and encode-xml for escaping unicode characters

https://metafacture.org/playground/?flux=inputFile
|open-file
|as-lines
|decode-json
|encode-xml(rootTag%3D"collection")
|print
%3B&data={"name"%3A+"Open+Educational+Resources+-+\u000Beine+kritische+Einführung"}
When special unicode characters are part of the transformed metadata encode-xml and encode-marcXml might transform the metadata to invalid xml.
There should be the possibility to create valid xml at least by adding an option for  escping unicode characters.

The behaviour regarding unicode characters seems to have changed in 844aa74. Previously, StringEscapeUtils.escapeXml() used to escape unicode characters to their numerical \u equivalent.

I had a similar issue with Catmandu some years ago.

Two things are mixed up here, i think:

What makes you think that? Syntax markers (1) are still escaped. AFAIUI, we're only talking about escaping invalid characters (2) here.

What makes you think that?

The docs of StringEscapeUtils.escapeXml() say it does 1. and besides of that:

Note that unicode characters greater than 0x7f are currently escaped to their numerical \u equivalent. This may change in future releases.

That's why i think it does not 2. (remove or replace invalid characters).
As a side note, from the logs of QA catalogue:
Apr 30, 2024 8:55:35 AM de.gwdg.metadataqa.marc.cli.utils.RecordIterator processFile
INFO: processing: oersiMarcDump.xml.tar.gz
[main] INFO org.reflections.Reflections - Reflections took 300 ms to scan 1 urls, producing 3 keys and 453 values
[Fatal Error] :3319174:57: An invalid XML character (Unicode: 0xb) was found in the element content of the document.

0xb is below 0x7f according to https://www.utf8-zeichentabelle.de/unicode-utf8-table.pl so this control character won't be handled by escapeXml() at all, i think.

I think the title of this issue is misleading, too.

Now I see what you mean, thanks. I haven't actually looked at the character values in question ;)
This implies that the behaviour in Metafacture hasn't changed, but rather that it's always been this way (with regard to invalid XML characters below 0x7f).
This also means that simply passing escapeUnicode=true to XmlUtil.escape() won't affect all invalid characters (namely, those below 0x7f).

Yep.
See https://www.w3.org/TR/xml/#charsets for all invalid and "discouraged" characters in XML.

The behaviour regarding unicode characters seems to have changed in https://github.com/metafacture/metafacture-core/commit/844aa74334d672a7a19b3a98ce8b979ef58d5a1a.

Looking at SimpleXmlWriter.java it seems to me that XML is being written "manually" (eg. builder.append("<");) and not by a XML library which would capture those invalid characters.


public folder and static files

How to tell iron router to ignore them and serve them accordingly?

@cmather Just tried updating to 1.0.7 but the issue isn't fixed for me.
@hellogerard Thanks for the workaround. It works in Chrome but not in Safari.
More details with my code here.

@francisbyrne Odd. It works for me in Safari on Mac OS.

Can there just be a ?iron-router-bypass=1 query flag? I'm trying to load static JS with script tags, the _blank workaround isn't going to work.

I still have this problem with 1.0.7.

@chicagogrooves you don't need the _blank_ workaround, the latest version is working fine for me.

On Meteor <IP_ADDRESS> my image files that are located in /public are not served correctly.

This still not working for sub directory.
I put images on /public/images/favicons/favicon.ico, but when I tried to access it, I got 404.
If I put it directly on images (/public/images/favicon.icon), it works, but I need to put it inside sub folder to organize it.
Is this also bug, or is this the intended behaviour?

@rajit this problem still exists in latest Meteor version <IP_ADDRESS>.
A quick fix is  to add a target="_self" attribute. It worked for me.

I still have this problem. Does anyone know how to fix this router issue?

@ryanacme i can take a look if you can give me a reproduction

@chrisbutler thanks a lot. it would be a great help. Here is my repo:
https://github.com/ryanacme/publicFolder_temp

@ryanacme so your issue is not this, the symptom just seems similar because your code has other issues
your issue is with relative/vs absolute paths on the images... you need to put forward slashes before your {{img_src}} otherwise the images won't load correctly if you aren't on the root path

Thanks heaps @chrisbutler . it was a great help. Now by having this forward slash my static images are working perfect but once I import images with absolute path (e.g. from web) then I face the same issue with those images. Since I know the issue now I think it would be easy to find a workaround.

@chrisbutler I found it. I just add "forward slash" when I insert my static images into my collection on server code.
Thanks again.

@ryanacme happy to help!


some code problems in model.py

Appreciate to your work first!
I'm working on coqa now and when I try to use your model I find that there is a strange package the interpreter can't find. It's on line 455 to 460 in model.py as follows:
           if not cands:                 # try getting from globals? (multiprocessing in pipeline mode)                 from ..pipeline.wrmcqa import PROCESS_CANDS                 cands = PROCESS_CANDS             if not cands:                 raise RuntimeError('No candidates given.')
I can't find the pipeline.wrmcqa on line so could you help me ?
Thank you very much!

hi spark, thanks for your question. This repo is only one part of my final year project, I do not release all code of that project. Therefore, code of this repos may import some another code. But core functions of machine comprehension are all included. You can delete those unnecessary code.
Thanks.

:) OK i've got it.
Thank you very much !


Change the translation of frequency in ntpc to semantically appropriate translation

Different words are used in Japanese for the frequency of radio waves and  the frequency of repeated work.
And I add 1 translation.
Signed-off-by: Satoru Yoshida<EMAIL_ADDRESS>

Please add/change the I18N translation with weblat
Please have a look at the pinned issue https://github.com/openwrt/luci/issues/3183


Add post

Requesting a change @HonestPretzels @liushuyu, when we add a moodEvent to firebase, can we generate the key locally (instead of having firebase do it) and add that to the event as a field called eventId or something? This would assist us when we're implementing delete events etc. Same for adding a user, I think we should generate the key locally (might as well make it the username)


[Feature Request]  not able to use element ids into webviews

Is your feature request related to a problem? Please describe.
Today, maestro seems to only accept to use a label to identify the element the action is related to when this element is located into a webview.

the tapOn action does not work in this case
Describe the solution you'd like
To be able to put element id as an identifier even for webviews element.
Additional context
maestro studio does not identify the id too.

If the WebView/Browser doesn't expose HTML ids as native accessibility IDs, then there's unfortunately not much we (as Maestro) can do.

Understood. I'll check with the platform team who made the page.  Thanks


Change your default security verification method

Steps are incorrect,  looks like these steps are valid for older version.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 5a5d93c5-5135-b9e9-e536-eb0830918b2e
Version Independent ID: 49be1539-737b-8874-eca6-1cdbc5263d73
Content: Change your two-factor verification method and settings - Azure AD
Content Source: articles/active-directory/user-help/multi-factor-authentication-end-user-manage-settings.md
Service: active-directory
Sub-service: user-help
GitHub Login: @eross-msft
Microsoft Alias: lizross

@slsourabh18 Thank you for pointing this out. We will engage the authors to review this article and update this thread .

@eross-msft Could you please review this article for the changes suggested by the customer ?

@slsourabh18  This actually is content for the older way to do things. The newer way is still in preview and you can see the instructions for end-users, here: https://docs.microsoft.com/azure/active-directory/user-help/user-help-security-info-overview. Thanks!
#please-close


Debian:stable python venv tests fail with non-bash shells

Currently test is disabled.
only debian:stable, not debian:oldstable or debian:testing fails with venv only with shells that are not bash. reason seems to be that reinsturmentation is not apply and pip path is therefore incorrect

looks like hash -r is never called, but still its assumed that the shell realizes the PATH changes. it looks like we shouldnt instrument hash to reinstrument, but rather PATH=. but since, this looks impossible, maybe it makes sense to instead instrument export
# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands.  Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH:-}" -o -n "${ZSH_VERSION:-}" ] ; then
    hash -r 2> /dev/null
fi


Can't plot anything other than evenly spaced out data points?

From everything I've seen, the data prop only accepts a 1 dimensional array of Y values, (or an array of 1 dimensional values in the case of multiple lines).
The implication then is that this chart cannot plot anything that's not uniformly spaced out data. What if wanted to plot something like a weight graph, where the X axis is the date, and Y the weight. The user isn't going to chart something every single day.
[[date: 1, weight 100], [date:15, weight: 102], [date:17, weight: 95]...]
is this pretty much not possible with this library?

https://www.youtube.com/watch?v=m8gyy7JItmU https://www.youtube.com/watch?v=1WZjeESH4a0


Improve the docker image

Follow-on from #837
Currently, we install our own server.xml and our own server, then delete the defaultServer.
However, now that we're based on OpenLiberty, we should follow the guidance at https://github.com/OpenLiberty/ci.docker#building-an-application-image
Here's at least 3 things that I think we could clean up/take advantage of:

The ibm-fhir-server container prints this warning during startup:

[AUDIT   ] CWWKG0102I: Found conflicting settings for defaultKeyStore instance of keyStore configuration.
  Property password has conflicting values:
    Secure value is set in file:/opt/ol/wlp/usr/servers/fhir-server/configDropins/defaults/keystore.xml.
    Secure value is set in file:/opt/ol/wlp/usr/servers/fhir-server/server.xml.
  Property password will be set to the value defined in file:/opt/ol/wlp/usr/servers/fhir-server/server.xml.

We might be able to workaround that by convincing Liberty not to mixin their own keystore.xml, but a better option would be to just let liberty generate its own keystore instead of pre-packaging ours.
For more info, see https://github.com/OpenLiberty/ci.docker/blob/master/SECURITY.md#providing-a-custom-keystore

Annotation caching defined at https://developers.redhat.com/blog/2020/01/28/use-red-hat-openshifts-built-in-oauth-server-as-an-authentication-provider-in-open-liberty/

This should help us with the startup performance, which I feel is noticably slower since we moved to OpenLiberty <IP_ADDRESS> (not sure if its about OpenLiberty vs WebSphere Liberty or about <IP_ADDRESS> vs <IP_ADDRESS>)

Class caching (when using OpenJ9) as defined at https://github.com/OpenLiberty/ci.docker#openj9-shared-class-cache-scc

Currently we use the "full" image.
There's a open issue with startup time on this image:
https://github.com/OpenLiberty/ci.docker/issues/174
However, it looks like this should be improved in future versions: https://github.com/OpenLiberty/ci.docker/pull/175
To move to the "kernel" version, we need to move our server name from fhir-server to defaultServer (or ask liberty to make their configure.sh script more flexible).

#2196 addresses points number 2 and 3 from the description by extending kernel-slim instead of full and using the Liberty-managed features.sh and configure.sh to install the minimal set of required features and set up the shared class cache (respectively).
On the first point, I don't think we see this warning any more.  However, we continue to use our own self-signed example certificate (rather than using a liberty-generated one on the fly)...mainly because we have all the trust relationships set up between our client and server certs and this would require significant release automation to replicate.

I also moved us onto the java 11 version of liberty (from java 8)


Lynda organization session has expired

Please follow the guide below

You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your issue (like that [x])
Use Preview tab to see how your issue will actually look like

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2016.04.06. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

[x] I've verified and I assure that I'm running youtube-dl 2016.04.06

Before submitting an issue make sure you have:

[x] At least skimmed through README and most notably FAQ and BUGS sections
[x] Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

[x] Bug report (encountered problems with youtube-dl)
[ ] Site support request (request for adding support for a new site)
[ ] Feature request (request for a new functionality)
[ ] Question
[ ] Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:
Add -v flag to your command line you run youtube-dl with, copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):
$ youtube-dl --cookies cookies.txt https://www.lynda.com/Unity-tutorials/Add-player-positions/427923/479695-4.html? --verbose
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--cookies', u'cookies.txt', u'https://www.lynda.com/Unity-tutorials/Add-player-positions/427923/479695-4.html?', u'--verbose']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2016.04.06
[debug] Python version 2.7.10 - Windows-8-6.2.9200
[debug] exe versions: none
[debug] Proxy map: {}
[lynda] 479695: Downloading video JSON
ERROR: lynda returned error: Session has expired
Traceback (most recent call last):
  File "youtube_dl\YoutubeDL.pyo", line 669, in extract_info
  File "youtube_dl\extractor\common.pyo", line 323, in extract
  File "youtube_dl\extractor\lynda.pyo", line 124, in _real_extract
ExtractorError: lynda returned error: Session has expired
...
<end of log>

Description of your issue, suggested solution and other information
I'm having problems downloading videos from Lynda.com using my login cookies.txt file, it displays an error requiring me to use username and password which are not available in organization case

Most likely your cookiefile has expired.

Can you watch it in browser?
How long ago have you exported your cookies? Re-export them and try again.
Post the output of youtube-dl --dump-pages --print-traffic --cookies cookies.txt https://www.lynda.com/Unity-tutorials/Add-player-positions/427923/479695-4.html.

I can watch in browser.
2 or 3 minutes
It worked for some time after I re-exported the cookies and it seems working for some time and I must re-export the cookies again.

Some cookies worked just fine and some other didn't.


fix: add reason indicating pseudorandom split

Adding const for SPLIT reason.

Codecov Report

Merging #76 (e7d5ce1) into main (aa59b35) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main      #76   +/-   ##
=======================================
  Coverage   84.30%   84.30%
=======================================
  Files           6        6
  Lines         376      376
=======================================
  Hits          317      317
  Misses         54       54
  Partials        5        5

Impacted Files
Coverage Δ

pkg/openfeature/provider.go
100.00% <ø> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more