ACK lost with high network traffic

This is issue is reported by @nkvaratskhelia  at https://github.com/eclipse/leshan/issues/491#issuecomment-534544892.
We're running a modified version of leshan-client-demo, the modifications being the support for running multiple clients. We've implemented a number of thread related optimizations (ideas being similar to the ones described in this issue, plus different or additional changes for the 1.0.0-M12 version).
System description: 2 vms, 1 gateway vm and 1 device simulator vm. The gateway can be considered as a "server", where all devices are registered. On the device simulator vm we run our modified leshan-client-demo. After starting up and registering with the gateway, each device receives an observe command for 3 different resources (eg: temperature, timer, addressable text display) and establishes observe relations.
Traffic pattern: excluding the initial start-up time, there are three types of messages that are exchanged between the devices and the gateway.

non-confirmable udp messages (NON) -- these are the notify messages for each resource sent by each device to the gateway every five minutes, no reply from the gateway expected.
confirmable udp messages (CON) -- every 10th NON message is a CON (we have the NOTIFICATION_CHECK_INTERVAL_COUNT Californium parameter set to 10), meaning that after sending 9 NON messages, each device will send a CON message to the gateway. An ACK (acknowledgement) reply is expected from the gateway for each CON request. These are 75 bytes each.
keepalive udp messages -- these messages are sent every 90 seconds by the gateway to each device, so if we have a large number of devices, there's basically a constant flow of keepalive messages from the gateway to the device simulator vm. These are 42 bytes each.

Problem: we have noticed that after running for some period of time (this depends on the number of devices we start, eg: ~6 hours for 10k devices with the described traffic pattern or ~50 hours for 1k devices) the devices are no longer able to receive the ACK type messages, retry a number of times and then cancel all observe relations.
After investigating the problem, we discovered that there are lots of udp packet receive errors in the OS (we're using Ubuntu 16.04 and you can monitor this with the netstat -su command). We checked tcpdump on both sides (gateway and the device simulator vm) and everything is as expected, ACK packets are sent by the gateway and received on the device simulator VM. At first we though this is some OS network buffer related issue (due to the heavy traffic between gateway and devices) and tried a number of optimizations recommended by network experts but the issue remains the same. In the end, we came to the conclusion that the kernel fails to send the packages to the jvm process and the issue lies inside the jvm process (meaning that it's no longer accepting packets from the kernel).
My question to you is why this might be happening. Is there some counter or limitation (after this limit is hit, no more udp messages are accepted or something else breaks) in leshan/californium that we are not aware of?
The only other Californium parameter we have overriden (apart from NOTIFICATION_CHECK_INTERVAL_COUNT) is MAX_PEER_INACTIVITY_PERIOD. We set this to 30 hours.
I'm not sure if this is the right place for this issue (whether the problem lies in the leshan, Californium or Scandium layer) so please let me know if I need to take it up with the Californium github.

As you said that you see ACKs in tcpdump capture, what does it means ? What exactly makes you think that you didn't receive ACK ?
I didn't get the rate of notification sending ?
Which kind of udp keep alive are you using ? Why are you using this ? are you trying to deal with NAT ?
Are you using DTLS ? (if  yes, did you face same issue without DTLS)

(@boaks , maybe you could have an idea about that)

We took tcpdumps on the gateway vm and on the device simulator vm. We can see that the gateway replies with an ACK packet to every 10th CON (including the last problematic ones) and that the device simulator vm receives these ACKS packets (including the last problematic ones).
When the udp receive errors start occurring, and a device sends a CON, we can see that it doesn't "process" the recieved ACK, then times out waiting for the ACK, retries this process a few times (I think it was 4 times, probably the default value for the MAX_RETRANSMIT param) and then cancels the observe relation.
The rate of notification sending is 5 mins (for each resource) in this particular traffic model that I described.
Not sure what you mean by the kind of udp keep alives (how can I check this?) but we're using zero-length udp keep alive packets and yes, NAT is the reason for that.
We are using DTLS and we haven't tried without it.

We took tcpdumps

Maybe you share them :-).
As I understand:

the gateway send ACKs to CON notifies (which are probes for the client aliveness).
these ACKs reaches the device client, but "sometimes" these are not processed by the client.

If you can provide the captures for both sides, I will have a look on it.

A question about your modification:
Did you modify the DatagramSocket to asyn one? if so, do you have java-logs with message about receiving the raw-dtls-records from that socket?

Do you run the demo-device-clients in one java-vm? Or did you start "some" java-vm's?
If you use one java-vm, maybe you can split the test to 2-3 java-vms for the demo-device-clients.
That would help. If only the devices of one java-vm hangs and the time of the hangups of the other java-vm seems to be not related, it may be a rare-race-condition.

As @boaks says :

the wireshark capture seems to show that the packet is received at client side.
the CON retransmission seems to indicate that this ACK is not processed by the client.

If this is correct, we should search where the ACK is lost/not processed.
You could try to activate logs and share with us if you see anything "suspect".

the devices are no longer able to receive the ACK type messages, retry a number of times and then cancel all observe relations.

Is it all devices or just a small part ?

Maybe you share them :-).

Attached the trace for both sides. You can decode the packets with the preshared key: 4567
traces.zip

the gateway send ACKs to CON notifies (which are probes for the client aliveness).

It's rather a probe for the gateway (whether it's still interested in observe notifications) than the client.

these ACKs reaches the device client, but "sometimes" these are not processed by the client.

Once the problem starts, these ACKs reach the device simulator VM (OS), not the device client per se.

Did you modify the DatagramSocket to asyn one? if so, do you have java-logs with log-messages
about receiving the raw-dtls-records from that socket?

We haven't done this modification.

Do you run the demo-device-clients in one java-vm?

Yes, for all runs so far we've used a single jvm process.

maybe you can split the test to 2-3 java-vms for the demo-device-clients

We can try this.

Is it all devices or just a small part ?

All devices.

You could try to activate logs

Do you mean the step described here: https://github.com/eclipse/leshan/issues/566#issuecomment-422337631 ?

All devices.

:open_mouth: !  I would not expected this answer.

Do you mean the step described here: #566 (comment) ?

Yes something like this but this will generate a lot logs regarding your traffic so maybe just begin to activate logs for "org.eclipse.californium.scandium.DTLSConnector" (@boaks do you see any other good class to watch ?)
If it's easy for you to test, could you see if you are able to reproduce it without DTLS ?

The traffic look good for me, so I wonder, why these ACK are not processed.
I'll start that experiment for coap-only in the evening (hope, nothing holds me up today :-) ). But it will take some time, as you mentioned.

It's rather a probe for the gateway (whether it's still interested in observe notifications) than the client.

DTLS, CoAP, and LwM2M have all roles, but they don't need to match :-).
The lwm2m-device-client is the coap-server for the observe, and the gateway seems to be something as a lwm2m-server, which in that case is a coap-client. Therefore the CON notify probes the aliveness/interest of the coap-client (gateway)

You could try to activate logs

Do you mean the step described here: #566 (comment) ?

I think so. I'm mainly interested in the output of:
org.eclipse.californium.scandium.DTLSConnector - level DEBUG

to see the message written in receiveNextDatagramFromNetwork by
LOGGER.debug("Received {} DTLS records from {} using a {} byte datagram buffer",
				records.size(), peerAddress, inboundDatagramBufferSize);

That should give a hint, if java still reads the datagrams.

The interesting thing seems to be, that the ACK is not processed, but the CON notifies are still send.

If it's easy for you to test, could you see if you are able to reproduce it without DTLS ?

Not that easy or quick, but we can do it if needed.

Therefore the CON notify probes the aliveness/interest of the coap-client (gateway)

I see what you mean. :+1:

Yes something like this but this will generate a lot logs regarding your traffic so maybe just begin to > activate logs for...

For the next run we will have the root level INFO and org.eclipse.californium.scandium.DTLSConnector - level DEBUG. Is this logging setup ok?

The interesting thing seems to be, that the ACK is not processed, but the CON notifies are still
send.

Yes, it seems the issue is only with receiving packets, not sending.

Are you device vm's using DHCP?
The retransmission works, so the timer and protocol thread pool seems alive.
Message are sent, so the serial execution thread pool seems alive.
The only left threads are the receiver threads, may be the interface thrown an exception,
which killed the receiver thread, and DHCP may have such an effect.
But that's only a guess.

Are you device vm's using DHCP?

No, the device vm has a set ip.

OK.  If you enabled the logs, please check for LOGGER.info "

Worker thread [{}] has terminated"

I started a long run test using the cf-extplugtest-client/demo (slightly modified for the 5 minutes). I use 3 java processes with each 2000 simulated device clients. The server the observes these simulated device clients. All run on the same host using the loopback interface. It runs now for 10h with 700000 notifies without issues and we will see, if in long terms the issue could be reproduced.

Here's a log file for the latest run (1 jvm instance, 15k devices, 5min notify interval).
log.gz

2019-09-26 12:46:45,378 DEBUG [DTLS-Receiver-0-<IP_ADDRESS>/<IP_ADDRESS>:24887] o.e.c.s.DTLSConnector Received 1 DTLS records from /<IP_ADDRESS>:5684 using a 16490 byte datagram buffer
2019-09-26 12:46:45,378 DEBUG [SharedExecutor-#11] o.e.c.s.DTLSConnector Received APPLICATION_DATA for DTLS(eshaiis-444-08.westeurope.cloudapp.azure.com:5684,ID:5D8C70)
2019-09-26 12:46:45,381 DEBUG [SharedExecutor-#11] o.e.c.s.DTLSConnector Sending application layer message to [DTLS(eshaiis-444-08.westeurope.cloudapp.azure.com:5684,ID:5D8C6F)]
2019-09-26 12:46:45,393 DEBUG [SharedExecutor-#11] o.e.c.s.DTLSConnector Sending application layer message to [DTLS(eshaiis-444-08.westeurope.cloudapp.azure.com:5684,ID:5D8C6F)]

After that, I can't see, that the socket receives data, nor that the receiver thread reports an error.
I can add some "health-logging", then we may see, if the receiver-threads are still working, but the DatagramSocket got broken somehow. Are you able to build leshan on your own? Maybe with a californium SNAPSHOT? if so, I would try to implement that health check soon. If you then repeat the test, may be you can setup the logging with split/rolling logs

Are you able to build leshan on your own?

Yes, I can build the leshan-1.0.0-M12 tag with the californium snapshot that you'll provide.

Just to let you know:
My test is up now for 1549641 notifies and all device-clients are well.

Please checkout the Californium PR1081 . Try to adjust your logback.xml according the "demo-apps/cf-extplugtest-client/src/main/resources/logback.xml" contained in the PR.
I restarted my longrun tests with that. But with 35 hours, my device-clients are still available.
If your device client stops, do you have a chance to send a udp message to the dtls port within the vm it runs on? And you don't use something as docker, it runs "plain" on the vm?

Please checkout the Californium PR1081 . Try to adjust your logback.xml according the
"demo-apps/cf-extplugtest-client/src/main/resources/logback.xml" contained in the PR.

:+1:

If your device client stops, do you have a chance to send a udp message to the dtls port within the vm > it runs on?

We can send a read command for example.

And you don't use something as docker, it runs "plain" on the vm?

Device clients run "plain" on the vm.

We can send a read command for example.

Any data, but within the same vm as the device client runs.
I would guess, the read command would be sent from the gateway vm.

if you setup the health logger similar to my example, then the alive logging should be written to "alive.log" files. It would be very interesting, what they contain after the "event".

I would guess, the read command would be sent from the gateway vm.

Aah right, I meant the read command from the gateway.

Any data, but within the same vm as the device client runs.

Yeah, we can use the Scapy tool for example to send a udp message from the same VM.

That would be great! I still have no hints, indications or similar issues, that the DatagramSocket stops receiving data while it is able to send.

Please checkout the Californium PR1081 . Try to adjust your logback.xml according the
"demo-apps/cf-extplugtest-client/src/main/resources/logback.xml" contained in the PR.

I built the new client and we'll make another run on Monday. @alexnorth will follow up from now on.

Maybe you can also add a memory/cpu statistic as in
JVM statistics

After 3 days and 5327215 notifes, I still can't reproduce this.
Thought the device-client wireshark capture indicates, that the messages are arriving, I'm not sure, why they a dropped during receiving them with the java DatagramSocket. There may be many reasons, one would be some "unreported internal out of memory errors in the upd stack", but that's just a wild guess.
To narrow down the root cause, I would recommend, that you start with 2000 device-clients simulated by one java vm. If you start the java-vms with an interval of 1h, we will see, if they also stop together or with 1h. Please also check, how much heap/ram the java-vm consumes (maybe using visualvm to check, if it's increasing or also got cyclic released).

Here are the results of the run that includes the healthcheck branch logging (had to strip out for 1 device to attach): 10006_alive.txt
Client log: appiot-client.log.gz
Then we sent some data via UDP to one of the devices (the one filtered in the alive log) using scrap like this: send(IP(dst="<IP_ADDRESS>")/UDP(dport=10006)/Raw(load="abc"))
Resulting tcpdump: eshaiis_111_00.pcap.zip
Also added the stats in as requested, this was dumped upon quit:
org.eclipse.californium.extplugtests.statistics 16 processors
org.eclipse.californium.extplugtests.statistics cpu-time: 10712554 ms (per-processor: 669534 ms)
org.eclipse.californium.extplugtests.statistics gc: 1685 ms, 40 calls
org.eclipse.californium.extplugtests.statistics average load: 0.27
Couldn't see that anything shows up why we have a failing case.
Next, we will try running smaller counts in per JVM and start new JVMs hourly.

I'm not sure, how to related the "times" of the logs.
alive.txt shows no issues (nor indicators for errors) from 00:00 to 05:57
But I would have expected, that it contains some

worker thread [DTLS-Receiver-0-<IP_ADDRESS>/<IP_ADDRESS>:10006] received {} bytes.

message. Are they filtered out? They would indicate, when data was received, potentially last received.
appiot-client shows the shutdown of the device-clients from 06:45 to 06:53.
(I miss the values for the memory usage in the statistic, but they are also just missing in my snippet :-) .)
the wireshark capture shows the ping (from gateway to the device) and 3 messages from the device host <IP_ADDRESS>:53 to <IP_ADDRESS>:10006.
My idea was to proof, that californium either receives the messages or not, when it reaches the state, when the ACKs are ignored (to narrow down the root cause).
Therefore my intention was to send some data from the same host to the scandium connector, after the ACKs are ignored. I would not use

send(IP(dst="<IP_ADDRESS>")/UDP(dport=10006)/Raw(load="abc"))

and instead specify the dst:

send(IP(dst="<IP_ADDRESS>")/UDP(dport=10006)/Raw(load="abc"))

for that.
And then it would be important to have the alive log from that:
Does it contain receive messages for those send? Or just alive messages?

Sorry, the 10006_alive.log was grepped before we shutdown.  After shutdown there is 1 'received' line per client (but not the sum of all clients for some reason).
With the complete alive.log:
grep received alive.log | grep 10006
2019-10-01 06:45:33,487 DEBUG [DTLS-Receiver-0-<IP_ADDRESS>/<IP_ADDRESS>:10006] org.eclipse.californium.scandium.DTLSConnector.health worker thread [DTLS-Receiver-0-<IP_ADDRESS>/<IP_ADDRESS>:10006] received 41 bytes.
We did have 15000 devices registered with the gateway however not all seemed to report the received line to alive.log
grep received alive.log | wc -l
14583
All devices were alive at some point ...
grep -o 'DTLS-Receiver-0-<IP_ADDRESS>/<IP_ADDRESS>:.....] alive' alive.log | sort | uniq | wc -l
15000
As we are now on a new run, we can redo the scapy command as suggested when the situation repeats. We will also have separate logs per batch too so easier to supply whole files here without filtering.

Sounds good.
Just to mention:
15000 seems to be quite a lot, especially, though we don't use the async io and just use sync io with many threads. Unfortunately, we are not too many working in californium.
If you want to add asnyc io as alternative, contributions are very welcome.

We started a JVM with 2000 devices every hour, up to 15k devices still.
All devices stopped working at the same time (about 5.5 hours in)
We sent the UDP packet like this:
sendp(Ether(dst="00:0d:3a:44:13:95")/IP(dst="<IP_ADDRESS>")/UDP(dport=10006)/Raw(load="abc"), iface="eth0")
tcpdump of that send: tcpdump.pcap.gz
Here are the logs, which are taken after the UDP packet send but we haven't shutdown the clients yet.
alive-1.log.gz
appiot-client-1.log.gz
As an observation, from our previous runs with a single JVM and 15k devices, all devices seemed ok and alive and communicating just fine also until about 5.5 hours in.

All devices stopped working at the same time (about 5.5 hours in)

For me this smells more for an "environment" issue, than for a Californium isssue.
I still can't see any "worker thread [DTLS-Receiver-0-<IP_ADDRESS>/<IP_ADDRESS>:10006] received {} bytes."
so, when was the udp packet send? does you "none filter" alive log shows a "receive" at that time?

Is your "device-client" public available?

All devices stopped working at the same time (about 5.5 hours in)

For me this smells more for an "environment" issue, than for a Californium isssue.

Would you have any suggestions as to what could be doing this?

I still can't see any "worker thread [DTLS-Receiver-0-<IP_ADDRESS>/<IP_ADDRESS>:10006] received {} bytes."
so, when was the udp packet "abc" send? Does your "none filter" alive log shows a "receive" at that time?

There isn't any.  The clients have not been shutdown yet.  They did not appear until the client was quit last time either.  This attached log is the full alive.log from the first JVM with 2k devices, it is not filtered.

Is your "device-client" public available?

Not at this point. Is the one you used for 2k devices over 3 days?
Perhaps we could run that one ourselves in the same environment.

Would you have any suggestions as to what could be doing this?

DHCP, but this was already denied.

There isn't any. The clients have not been shutdown yet. They did not appear until the client was quit last time either. This attached log is the full alive.log from the first JVM with 2k devices, it is not filtered.
The UDP packet was sent before taking the attached log file.

Now I'm mixed up! Without "received" messages, I'm not sure, what happens on your site.
Can we reduce your setup to say 10 devices? Clean the log directory before starting the tests. The just send me the alive.log from the first 10 minutes. There must be some "received" messages ...

Is the one you used for 2k devices over 3 days available?

Yes, either checkout the PR 1081 from the https://github.com/eclipse/californium repo, or the branch healthcheck from the https://github.com/bsinno/californium repo.
Then mvn clean install.
The "cf-extplugtest-client" module in demo-apps contains the client, "cf-extplugtest-server" the related server. In the client you find a benchmark.sh, which contains the instructions. You should mainly adjust the java heapsize "-Xmx6g" according you system.
On the client:
I use to start the different java-vms with "sh benchmark.sh" in different directories. Therefore I copy the "benchmark.sh" and the "cf-extplugtest-client-2.0.0-SNAPSHOT.jar" into that folders (I have also a updated benchmark.sh, which obsoletes the copy of the jar, but right now not pushed)
If you try to setup, please check after a couple of minutes, if you see received messages in the alive log.

If you have any question related to the californium benchmark stuff, maybe you can use the issues in californium?

As we hadn't stopped the JVMs earlier, we stopped the first one.  At this point the 8th JVM started to make successful registrations and form observe relationships. This JVM (8th) had previously started the clients but as it was started up after this issue had kicked so it wasn't successful registering then.
After a short period, we started to see the packet loss again and the current observes were cancelled.
So my interpretation of this is that it does appear to be linked to some resource usage/limitations on that machine. Would you agree, or do you still see any other path worth following?
We initially started off at investigating why the packets were being lost and tried a number configuration settings, as Nik said earlier ...

At first we though this is some OS network buffer related issue (due to the heavy traffic between gateway and devices) and tried a number of optimizations recommended by network experts but the issue remains the same. In the end, we came to the conclusion that the kernel fails to send the packages to the jvm process and the issue lies inside the jvm process (meaning that it's no longer accepting packets from the kernel).

Perhaps we need to revisit this issue from this angle again.

From the initial comment:

Problem: we have noticed that after running for some period of time (this depends on the number of devices we start, eg: ~6 hours for 10k devices with the described traffic pattern or ~50 hours for 1k devices) the devices are no longer able to receive the ACK type messages, retry a number of times and then cancel all observe relations.

That says, it also fails with 1k devices.
Maybe you retest, if it works reliable for e.g. 2000 clients.

I didn't follow this discussion in detail, so I will maybe be off track but did you try to look at the state of jvm when you begin to get stuck, using jstak to find if there are deadlock ?

deadlock

What we saw at the beginning:

CON  messages are sent and resent.
ACK seems to be ignored.

FMPOV, that indicates, the timer and protocol threads are well (retransmission), the thread-pool in scandium is well (sending at all). For the receiver threads I added "alive" loggin messages, and they seems to be well also.
But, yes, maybe I oversee something.
I would appreciate, if the californium longterm benchmark is executed in the "target environment" e.g. with 2x2000 clients. then we will see, if that stops or runs. On my environment, 3x2000 device have been running for 3 days wihtou any issue.

I'm pretty sure there is no deadlock too, but having a quick look with jstack should be pretty easy.

I would also try to check the memory/ram (e.g. top).
Searching for UDP message drops with java DatagramSocket results in some hints, that on memory shortage, the OS may cause that drop.

Which JVM is used? I mainly use java 8.

We started a JVM with 2000 devices every hour, up to 15k devices still.
All devices stopped working at the same time (about 5.5 hours in)

Maybe I misunderstood this:
Does every jvm run for 5.5 hours? Means they stop at different timepoints?
I interpreted it, as all on the same timepoint, but now I think it may be after the same timespan (at different timepoints).

Maybe I misunderstood this:
Does every jvm run for 5.5 hours? Means they stop at different timepoints?
I interpreted it, as all on the same timepoint, but now I think it may be after the same timespan (at different timepoints).

Your initial intepretation is correct.

I would also try to check the memory/ram (e.g. top).

We have monitored this in the past and haven't found any high memory consumption in machine nor within JVM.  I don't have any results but can obtain if needed.  The cpu usage is low and is spread pretty evenly across all cores. Machine is 16 core @ 2.3GHz with 32 GB RAM.

Which JVM is used? I mainly use java 8.

openjdk version "1.8.0_222"
OpenJDK Runtime Environment (build 1.8.0_222-8u222-b10-1ubuntu1~16.04.1-b10)
OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
As suggested above, I used JStack to make a thread dump of one of the JVMs while this issue is happening, no apparent deadlocks: threaddump_jvm2.log.gz

I would appreciate, if the californium longterm benchmark (cf-extplugtest-client from the healthcheck branch) is executed in the "target environment" e.g. with 2x2000 clients. then we will see, if that stops or runs. On my environment, 3x2000 device have been running for 3 days without any issue.

This is what we will do next.

I'm currently updating (including some more docu) the PR #1081 ... if not already started please wait for a hour (or two) ... I will ping, when I'm ready.

I'm currently updating (including some more docu) the PR #1081 ... if not already started please wait for a hour (or two) ... I will ping, when I'm ready.

No problem

I pushed it.
The "benchmark.sh" is setup for longterm run.
I adjusted it to run the client multiple times from subfolders without copy the jar.
To setup that test:
I would prefer, if you can start with clients and server on same host (that's how I tested it).

checkout the branch from bsinno or the PR from eclipse.
build it with mvn clean install
start the server in "demo-apps/run" with java -d64 -Xmx6g -XX:+UseG1GC -jar cf-extplugtest-server-2.0.0-SNAPSHOT.jar -onlyLoopback -noPlugtest
cleanup the clients config in "demo-apps/cf-extplugtest-client" by deleting the "Californium???.properties" files.
start the clients 1. time in "demo-apps/cf-extplugtest-client" by sh benchmark.sh. Check the alive logs (in folder logs), if the contain alive and receive messages. If that is successful, then do the next steps.
create subfolders for the other client-groups in "demo-apps/cf-extplugtest-client". start them from that subfolders by "sh ../benchmark.sh".

If you have any trouble, don't hesitate to report it.

If the test runs with client and server on the same host, switching to different host is done by:

start the server java -d64 -Xmx6g -XX:+UseG1GC -jar cf-extplugtest-server-2.0.0-SNAPSHOT.jar -noLoopback -noPlugtest

edit the benchmark.sh and replace the "localhost" in CF_HOST=localhost with your server address.

start the benchmarks again.

The jvm, cpu and ram seems to should not be the root-cause.
Did you succeed in try out the californium longterm run?
Maybe you check also in the meantime some IP/UDP OS settings.
Ubuntu and UDP

Did you succeed in try out the californium longterm run?
Yes, it runs and was going from yesterday afternoon until this morning just fine, when I accidentally killed the server process. I started it again so it can run over the weekend to get the full 3 days just to be sure.

Maybe you check also in the meantime some IP/UDP OS settings.
Ubuntu and UDP
Yes, this is the type of things we have been looking at too, thanks.

We've been varying different parameters such as data sizes, client counts, etc to try to find out what could be causing this.  It's a long test cycle though so going is slow and we don't know enough about what is wrong to try to influence it to speed up the failure.

It's a long test cycle though so going is slow

very true.
My test uses only 1 notification, your device-client sends 3. I use a small message, one of the 3 of your device-client is a larger one.
For other test within the californium benchmark suite I use blockwise transfer and therefore a blocksize of 64. If you want to check with larger notifies, I can change the setup of that long term run. Maybe also switch to always CON notifies and notify every minute may cause that issue earlier.

My test uses only 1 notification, your device-client sends 3. I use a small message, one of the 3 of your device-client is a larger one.

Last night, we did re-run the same 2k every hour up to 15k test again but with smaller data (100 bytes instead of 700) on that 3rd object but result was that it failed at the pretty much the same time as before.

For other test within the californium benchmark suite I use blockwise transfer and therefore a blocksize of 64. If you want to check with larger notifies, I can change the setup of that long term run. Maybe also switch to always CON notifies and notify every minute may cause that issue earlier.

Always CON is something we haven't tried yet. I think we did try more frequent notifies but don't have any result recorded so will do both of these again with our client, thanks.

Really mystic.
My be I'm repeating to often:
The "added health check" in californium should show both "alive" and "receive" messages in the "alive.log". Could you check/verify that? If that works, then check/verify, that you can sent message with "scapy" to one of the device-clients and that this is shown as "receive" in the alive log (I would go for both, outside and inside the device-client host). If that works, wait for the "clients dying". Then repeat the scapy test (outside and inside) and I'm really looking forward, if that shows "receive" in the alive log, at least from inside the device-client host.

The CF benchmark (with 3 x 2k devices) is still happily running, so that's over 72 hours now. alive.log for each batch has 'alive' and 'received' in it.
With a test of 10k devices that increased the notification period from 5 mins to 1 min and NOTIFICATION_CHECK_INTERVAL_COUNT=1, the test failed in just over 4 hours, so that's a much quicker failure, we will try faster notifications still.
As for the alive/received messages,  we are still not seeing (for our own client test) any 'received' still, just 'alive'.  What does the received log line show up?

The test with 10k devices is your own test? Or the cf benchmark 5x2k devices)?
And please check, if the scapy test works when the system is well, and then use it, if the system gets broken to see, if messages from the same host as the devices are still processed.

The test with 10k devices is your own test? Or the cf benchmark 5x2k devices)?

10k was with our own devices.
We did the cf benchmark with 3 lots of 2k devices btw.

And please check, if the scapy test works when the system is well, and then use it, if the system gets broken to see, if messages from the same host as the devices are still processed.

Even in a good/clean state, it seems that we are not able to see any received activity in the alive.log when we use scapy on local machine.  tcpdump shows packet received but nothing in log.
Sending the packet from another machine works ok and we see the received line in the alive.log.

Do you use a specified NetworkInterface to build the DTLSConnector?
DtlsConnectorConfig.Builder.setAddress(InetSocketAddress address) maybe used with new InetSocketAddress(int port) or InetSocketAddress(InetAddress addr, int port).
The first bind the connector to "all NetworkInterfaces" the other only to the specified one.
if scapy doesn't work local, maybe you used the specific one. If possible, please try the unspecific one for that test. The idea behind the test is, to find, if the java DatagramSocket can't generally receive data, or just no data from other host. With the second one, I guess the effect is something, which the supplier of the vm knows.

Do you use a specified NetworkInterface to build the DTLSConnector?

No, InetSocketAddress(int port) is being used

scapy on localhost seems to require special setup:
from scapy.all import *
conf.L3socket=L3RawSocket
send(IP(dst='<IP_ADDRESS>')/UDP(dport=5684,sport=9999)/"abcdefgh")

shows:
16:49:56.401 DEBUG [health]: worker thread [DTLS-Receiver-0-/<IP_ADDRESS>:5684] received record of 8 bytes.

Any updates? New insights?

Any updates? New insights?

Sorry, no.  We had to shift focus a bit and have some other things going on. Can try to get back to this investigation next week.
So latest bit of info was that we got the issue to happen faster when putting CON-ACK always on, which is great. Still a matter of hours, but half the previous amount.
Will try the above scapy info for localhost when we are able to run these tests again and get it to break.

Good luck! I will be out of office next week, so take the time you need for your investigations.

send(IP(dst='<IP_ADDRESS>')/UDP(dport=10000,sport=5684)/"abcdefgh")
This worked ok with a freshly started client, "received" came to alive log.  Did not work once the packet loss was present.
Outgoing messages (notifications in this case) from the client to the gateway were still going without problems.

Did not work once the packet loss was present.

but the messages:
Worker thread [xxxxx] is alive!

are in the alive log (at least, earlier comments shows that)? That's really mystic to me. It seems to be then an issue deeper in the java-os-udp-stack. I searched again, and maybe this is a hint
UDP freezing

The weird thing is that , when the sender uses a large window size, i.e. sends a large number of packets to the receiver in a short time, the Datagram Socket of the receiver just stops receiving packets after a while. More explicitly, the receiver program is stuck at the DatagramSocket.receive() method, It seems that the receive buffer of the Datagram Socket has been overflowed, and cause it to block in the DatagramSocket.receive() method. To confirm this suspicion, I set the receive buffer size much larger, then the trouble is gone.

Unfortunately, I'm still not able to reproduce this failure. If you're interested, you may patch the DTLSConnector.start(final InetSocketAddress bindAddress) first to log the recv buffer size and then try larger values.

but the messages:
Worker thread [xxxxx] is alive!

are in the alive log (at least, earlier comments shows that)?

Correct

UDP freezing

The weird thing is that , when the sender uses a large window size, i.e. sends a large number of packets to the receiver in a short time, the Datagram Socket of the receiver just stops receiving packets after a while. More explicitly, the receiver program is stuck at the DatagramSocket.receive() method, It seems that the receive buffer of the Datagram Socket has been overflowed, and cause it to block in the DatagramSocket.receive() method. To confirm this suspicion, I set the receive buffer size much larger, then the trouble is gone.

This sounds very much like the behaviour we are experiencing.

Unfortunately, I'm still not able to reproduce this failure. If you're interested, you may patch the DTLSConnector.start(final InetSocketAddress bindAddress) first to log the recv buffer size and then try larger values.

If I understood what you were describing, I see that datagram buffer size is already logged and shows:
2019-10-21 10:06:06,009 INFO  [pool-6-thread-2] org.eclipse.californium.scandium.DTLSConnector DTLS connector listening on [<IP_ADDRESS>/<IP_ADDRESS>:64020] with MTU [1500] using (inbound) datagram buffer size [16490 bytes]
I added 1024 to the inboundDatagramBufferSize before the receivedThreadCount loop.  This didn't seem to affect in any way. Too small perhaps?  Redoing with +10000
Additionally, when running some tests now, we get a failure around the 2 hour mark, we noticed from the netstat -su output that no matter if there are 10k or 20k clients, it will fail at about 8 000 000 packets received.
This is with config setting:
networkConfig.setInt(NetworkConfig.Keys.NOTIFICATION_CHECK_INTERVAL_COUNT, 1);
However, if we change these settings to try to make it stay alive as long as possible, then it lasts for some more hours and we get way past that same packets received count (was at 20 000 000 when it broke):
networkConfig.setInt(NetworkConfig.Keys.MAX_PEER_INACTIVITY_PERIOD, 108000); networkConfig.setString(NetworkConfig.Keys.RESPONSE_MATCHING, "RELAXED"); networkConfig.setInt(NetworkConfig.Keys.NOTIFICATION_CHECK_INTERVAL_COUNT, 10000); networkConfig.setInt(NetworkConfig.Keys.NOTIFICATION_CHECK_INTERVAL_TIME,<PHONE_NUMBER>);
We then hit memory issues as whole system is running on the same machine now, so kind of expected.  The point was that the failure point is influenced by these settings. I just used the same settings that have been used in the past for longer running tests, I could whittle it down to which one(s) affect if needed.

My feeling it's not about the "datagram buffer size", which is passed into DatagramSocket.receive, it's about DatagramSocket.setReceiveBufferSize(int).

In the fast "8 000 000" case, is there also an OOM error?
Or only for the "20 000 000"?
(And why is there an OOM error at all? why do you expect that?)
If both are related to OOM, maybe try to fix #757 helps. @sbernard31 already created a PR for that, so I hopefully will be able to update the "healthcheck" branch tomorrow or on Wednesday.

My feeling it's not about the "datagram buffer size", which is passed into DatagramSocket.receive, it's about DatagramSocket.setReceiveBufferSize(int).

Right, now I see what you mean, thanks.

In the fast "8 000 000" case, is there also an OOM error?

No, packet loss is the stopper in this case.

(And why is there an OOM error at all? why do you expect that?)

We are currently running a lot in memory from one machine, leshan server, undertow, a tool for exercising our backend, redis, and 10k+ clients with 20 sec notification rate per client.  And with notification CONs off for this case, the messages build up. "Expect" is maybe not the right word, but not suprised that happened - no tuning or monitoring of all of this together has been done.  I'd be happy getting to the point where OOM was the actual problem :)

Ok, I just want to ensure, that you're not blocked by that #757 .

Let's wait for the next try with larger recv buffer enabled ...

The receive buffer was 52428800
This was down to a setting in net.core.rmem_default of double that.
As this was rather large to start with, we dropped the value to 8388608, giving a socket receive buffer of 4194304.
Problem still occurred at about the same time/packets received count (just under 2 hours / ~8 million packets received).

Assuming, that the DatagramSocket.setReceiveBufferSize(int) affects the buffer per client, I would try to use just 8k (8192). That should be enough for 1 client to receive the ACK, but not too much, assuming that this potentially allocates "8388608 x 10000 clients" bytes of memory.

That one gave a different result.
This time we got some packet loss but it also said it was due to 'RcvbufErrors' in the netstat output, which it has never done before (packet receive errors had same count as RcvbufErrors). The application kept on running as normal too, reads happening just fine.
Is this a memory issue after all, we were giving possibility of grabbing too much memory from the system (ours has 32GB RAM) as a whole by having too large a value for net.core.rmem_max ?  But then how does it explain 10k and 20k clients causing the packet loss at about the same time?  If it was at JVM side, where is the OOM?

I'm still not sure, if we consider the same configuration.
You use 8192 byte per socket recv buffer per 20000 clients (should result in about 160MByte).
You use "net.core.rmem_default" at the initial default? What was that? Does the 160MByte (+ send buffers) fit into that?
When I did my first websearch for that some weeks ago, one result was, that there are many settings for many levels. And potentially the defaults are not intended for 20000 active udp sockets. So the painful work is to check all these settings on all levels ...

Is this a memory issue after all, we were giving possibility of grabbing too much memory from the system (ours has 32GB RAM) as a whole by having too large a value for net.core.rmem_max ?

Maybe something bumps into some limits, but which limit is still not fully clear.
32GB seems to be larger then 160MB. so there is still a chance to find the right settings :-).

But then how does it explain 10k and 20k clients causing the packet loss at about the same time?

Same time or same number of packets?

Does using DatagramSocket.setReceiveBufferSize(int)differs in the resulting behaviour from setting net.core.rmem_default?
If yes, maybe  DatagramSocket.setSendBufferSize(int) to 8192 changes  also the behaviour.

I'm still not sure, if we consider the same configuration.
You use 8192 byte per socket recv buffer per 20000 clients (should result in about 160MByte).
You use "net.core.rmem_default" at the initial default? What was that? Does the 160MByte (+ send buffers) fit into that?

Both net.core.rmem_default and net.core.rmem_max were set to 8192.  I put a currentSocket.getReceiveBufferSize() output in the DTLSConnector and it shows half of whatever I set so 4096 in this case.

When I did my first websearch for that some weeks ago, one result was, that there are many settings for many levels. And potentially the defaults are not intended for 20000 active udp sockets. So the painful work is to check all these settings on all levels ...

Yes, we have been (ab)using these settings: kernel.pid_max, vm.max_map_count, net.core.somaxconn, net.core.netdev_max_backlog, net.core.rmem_default, net.core.rmem_max, net.core.wmem_default, net.core.wmem_default, net.ipv4.ip_local_port_range at various stages with varying values.  One previous misunderstanding for us was that the rmem_default/max was for the whole system, hence the way too large allocation.
Finally it feels like we are getting somewhere.

Maybe something bumps into some limits, but which limit is still not fully clear.
32GB seems to be larger then 160MB. so there is still a chance to find the right settings :-).

I think the situation with the receiving stopping is because we have setup that we have too large receive buffer memory size per socket and as it eats it up we get to a point where we exceed the actual memory availability of the machine.
With the RcvbufErrors it is because the receive buffer is too small.  This is when we set an 8kb max.  Now we can tweak this along with our device counts and find a level that works - at least this is what I am hoping :)

But then how does it explain 10k and 20k clients causing the packet loss at about the same time?

Same time or same number of packets?

About same for both.

Thanks, that are really good news!
FMPOV, I would like to focus on release Californium 2.0.0. Therefore I will try the make these socket buffers configurable within californium, I hope it will be available begin next week.
For simulating more clients, I still think, that we should try out async io, but after that 2.0.0 release.
To analyse what exactly happens, I would try out other setups, e.g. iperf3 and see, if the "blocking" could be also reproduced without java. But that would be something for "bad weather winter days". If it looks like being related with java, I would try out running it with java 11.
But I will not be able to test that in the next days/weeks.

This is a very long thread and I'm a bit lost.
Could you give a short resume ?
You said :

At this point we could attribute it to misconfigured kernel settings.

But it's not clear to me :

the default configuration was an issue ?
or you tweak this and finally this cause the issue ? (and go back to default config fix the issue)

If 2. what was the initial reason to change the config ?

My values (defaults for 8GB) :
sudo sysctl -n net.core.rmem_default
212992
sudo sysctl net.ipv4.udp_mem
net.ipv4.udp_mem = 179829	239773	359658
sudo sysctl net.ipv4.udp_rmem_min
net.ipv4.udp_rmem_min = 4096

That results in 4096*359658 bytes buffers at all.
With 212992 bytes per socket, about 6900 sockets are possible (neglecting the send buffers).
I guess, if I try it on that machine with 10000 clients, I will run in to the issue.
But the first comments reported the issue also for less clients.

To analyse what exactly happens, I would try out other setups, e.g. iperf3 and see, if the "blocking" could be also reproduced without java. But that would be something for "bad weather winter days". If it looks like being related with java, I would try out running it with java 11.
But I will not be able to test that in the next days/weeks.

I can try the "bad" kernel settings and Java 11, it was on my TODO list anyway - just to see how things behave.

At this point we could attribute it to misconfigured kernel settings.

But it's not clear to me :

the default configuration was an issue ?
or you tweak this and finally this cause the issue ? (and go back to default config fix the issue)

If 2. what was the initial reason to change the config ?

By configuration I am actually referring to the kernel net.core.rmem_default/max setting.
We used some settings from a previously known "good" setting and that had rmem set to 8MB.
This was a mistake to start with (we didn't know this at that time though).  This led to some strange behaviours when confirmable notifications were triggered, which led us here.
As you said, it's been a long thread and there also has been some amount of experimentation, learnings, and better understanding.
The end result is that if we set the kernels net.core.rmem_default/max settings to a number which will eventually lead to wanting to use up more memory than the system has available then we get a problem with packet loss (client receives stop working).  When setting too small, or tweaking to a more reasonable value for the amount of clients and data we have then everything is ok.
Not sure if that helps clarify, but I could probably setup some info and code to be able to reproduce this in conjunction with some kernel settings if needed.  It is still not clear where the packet loss is occurring - at JVM level or somewhere else.

That results in 4096*359658 bytes buffers at all.
With 212992 bytes per socket, about 6900 sockets are possible (neglecting the send buffers).
I guess, if I try it on that machine with 10000 clients, I will run in to the issue.
But the first comments reported the issue also for less clients.

We might have had a -Mmx6g at that point, despite it being on 32GB machine.  So 8MB rmem setting with 3000 clients would exceed that.  As I said above, I can make some instructions for reproducing this now that we are (hopefully) wiser as to what is going on ... but next week.  Would be good to clarify from own point of view as well and limit all the different settings and code down to exactly what is needed to reproduce this issue.

The latest Californium "master" branch contains now a version, which offers to set the socket buffers using DtlsConnectorConfig. "cf-extplugtest-client" setup these socket buffers from the "Californium???.properties" values
"UDP_CONNECTOR_SEND_BUFFER=8192" and "UDP_CONNECTOR_RECEIVE_BUFFER=8192". Delete the "Californium???.properties" to prepare them with the new defaults.
The startup of the BenchmarkClient is also improved and much faster.
Try it out. Maybe with a "unmodified linux", except the required enlarging the number of files (socket) and threads (pid), see Setup ubuntu 18.04.

@alexnorth, @nkvaratskhelia  what should we do with this issue ?

@sbernard31 sorry, it's has slipped off the radar as we got things working and then busy (excuses, sorry) ... it's up to you how you would like to process this from the projects perspective as we have a working solution.  However, if wanted we can take some clean VMs and try the same by configuring via Californium.properties rather than kernel. I see this more to confirm that that this suggested way of configuring works rather than it being a fix for anything as the original problem was down to bad kernel settings for us. This will take a little time to get going again, first trying to reproduce the problem and then 'fixing'.  I remember that the test cycles were long ... hands free but long :P

@alexnorth,
The problem is not clear to me but I understand after a while there was some "communication issue".
It seems you finally find a solution tweaking to buffer size ?
So the questions is "clarifying/sumerazing the issue and sharing advice with leshan/californium community" does make sense to you ?
To me it's hard to say as I don't get the issue enough. :)

So the questions is "clarifying/sumerazing the issue and sharing advice with leshan/californium community" does make sense to you ?

Yes - and we will attempt to do so.

@alexnorth Thanks :pray:

@alexnorth I will close this issue but do not hesitate to comment this issue anyway to share with us your advices.


mvision edr

Hi Iakobi,
the commits contain tokens that are unique to each customer / partner tenant and are variable inputs.
I won't merge the pull request for now. If you are facing any issues please open an issue on the repo.
All the best,
Martin


Feature/timeslots

This feature add timeslots which is include in React big calendar.

Released on v4.15.0


[DO NOT MERGE UNTIL 5.6] Fix re-exported defaults in ExportInfoMap

Fixes #58590

@typescript-bot test tsserver top400
@typescript-bot perf test

I don’t think this is very risky, but the RC is already out and the prevalence of the issue seems very low, so I’m going to play it safe and hold off for merging until after the 5.5 stable branch is cut.

I’ve just been informed that main is already 5.6!

Thank you, @andrewbranch!


License?

Could you add a LICENSE file to the repository?
I would like to add functionality in a fork you will probably not be interested in, but I would update the project to the current bevy version in the process and give you a PR for that part.

Just pushed a commit with the two files (Apache / MIT).
It would be awesome if you can make this work with the latest Bevy.


Prepare release 0.12.2

Prepare release 0.12.2.
It's needed to release https://github.com/elastic/terraform-provider-ec/pull/866.

The tests fail due to some changes to API.

Hello @gigerdo
Do you have an estimation about when v0.12.2 will be released?
We really could use PR #866.
Thanks!

@GeorgeGkinis , sorry for the delay - we faced with a number of failed acceptance tests which was caused by old DTs/ICs deprecation in API and also discovered another defect along the way. Addressing these findings took some time.
TL/DR, we will proceed with the release.

Thanks for the feedback!
I figured it out that the interface changes could prove to be time consuming.
Good to hear they are resolved!
Cheers.

There are 3 flaky acceptance tests which failed:

TestAccDatasource_trafficfilter failed  with Failed to read deployment traffic filter ruleset after create;
TestAccDeployment_basic_defaults_first and TestAccDeployment_integrationsServer failed because plan which adds IntegrationsServer was completed before IntergrationServer's endpoints became available (we have a ticket for this).

https://github.com/elastic/terraform-provider-ec/releases/tag/v0.12.2


Add MustParsePath

Another tiny one.
I find it useful for addressing into some kind of data structure to use a path, like
p := MustParsePath(fmt.Sprintf(`.accounts.byEmail["%s"]`, email))


Global config dir

Could ipydeps look for its configuration files in a global location (e.g. /etc) as well as ~/.config?  We have a multi-user environment but would like to set the same dependencies.link for everyone.

Done: https://github.com/nbgallery/ipydeps/blame/ae64d0c8550e2f69c17bb543df79b072822948e9/ipydeps/config.py#L27


[ERROR]: vectorsService.upsertVector - Error: No relative links found

Why would it give me that error? Where could be the issue?

I fixed the issue!!
Just check for Relative Links, verify that the page at the URL contains relative links.


Inefficient layer sizes when changing permissions or adding directories to existing directories

I have a Rails app that I am trying to place into a container. A subset of our Dockerfile looks something like this:
WORKDIR /srv/www
RUN useradd rails

ADD Gemfile /srv/www/
ADD Gemfile.lock /srv/www/
RUN chown -R rails /srv/www
RUN sudo -u rails bundle install

ADD . /srv/www
RUN chown -R rails /srv/www

It's crafted like this because our Gemfile is expected to change less frequently than the code, and various tools (bundler, bower) warn about running as root, so I wanted to run them as they expect.
However, running docker history has (abridged) output like this:
CREATED BY                                                                                SIZE
/bin/sh -c chown -R rails /srv/www                                                                     204.1 MB
/bin/sh -c #(nop) ADD dir:1ba5e606b64673a429a6e41f22dd80c96f6b9fafb5e9988d813202ac9fd17ef0 in /srv/www 204.1 MB
/bin/sh -c sudo -u rails bundle install                                                                203.2 MB

The first step makes sense - my gems take 200 MB of space. However, my app is only 5.5MB (including .git), so I would expect the second image to only be ~5.5 MB, not 200MB. The chmod takes another 200MB.
My uninformed guess is that both of these operations touch /srv/www, so AUFS snapshots them all the way. Perhaps there is some workaround that I can use to avoid bloating my containers?
If there was a way to ADD as a particular user, I could avoid the chown. I'd also expect that adding to an existing directory would only require space proportial to what is added.
Thanks for the awesome work!
# OS X 10.9.2
# boot2docker: 0.8.0
$ docker version
Client version: 0.10.0
Client API version: 1.10
Go version (client): go1.2.1
Git commit (client): dc9c28f
Server version: 0.10.0
Server API version: 1.10
Git commit (server): dc9c28f
Go version (server): go1.2.1
Last stable version: 0.10.0

@shepmaster  Afaik, chown commond does not introduce additional disk usage when the storage driver is chosen as devicemapper. #6119


fix: Allow memory sizes between 2gb and 4gb by using wasm allocation

Context
Browsers don't allow allocation of more than 2gb memory using the standard buffer allocators.  This PR adds the ability to allocate using the WebAssembly.Memory allocator, which seems to work fine up to 4 gb, although the GPU does complain about memory texture size limits.
Changes & Results
Added a test example, using local data for now (and so not in the shared examples), plus changed the  createFloat32SharedArray to use WebAssembly.Memory
Testing
Deploy the test data to localhost:5000/dicomweb
Run
yarn example volumeLarge
View the page, should display (may still be browser dependent).
Checklist
PR

[] My Pull Request title is descriptive, accurate and follows the
semantic-release format and guidelines.

Code

[] My code has been well-documented (function documentation, inline comments,
etc.)

[] I have run the yarn build:update-api to update the API documentation, and have
committed the changes to this PR. (Read more here https://www.cornerstonejs.org/docs/contribute/update-api)

Public Documentation Updates

[] The documentation page has been updated as necessary for any public API
additions or removals.

Tested Environment

[] "OS:
[] "Node version:
[] "Browser:

works great on orthanc but seems there's some issues when using P10 dicom from local


Fixed broken link in show_species_list.

No tests or review, just a quick bug fix.

Can you also please fix the User who complained about this? :-(

Way beyond the scope!  He was right in this case, though.
On Sat, Oct 3, 2020 at 10:37 PM Joseph D. Cohen<EMAIL_ADDRESS>wrote:

Can you also please fix the User who complained about this? :-(
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/MushroomObserver/mushroom-observer/pull/672#issuecomment-703191540,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAYTNNJXCR4X4RREMMJMLELSI7NWLANCNFSM4SDM4DTQ
.

He is right occasionally. Thanks for the fix!
On Sat, Oct 3, 2020 at 7:40 PM Jason Hollinger<EMAIL_ADDRESS>wrote:

Way beyond the scope! He was right in this case, though.
On Sat, Oct 3, 2020 at 10:37 PM Joseph D. Cohen<EMAIL_ADDRESS>wrote:

Can you also please fix the User who complained about this? :-(
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
<
https://github.com/MushroomObserver/mushroom-observer/pull/672#issuecomment-703191540
,
or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/AAYTNNJXCR4X4RREMMJMLELSI7NWLANCNFSM4SDM4DTQ
.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/MushroomObserver/mushroom-observer/pull/672#issuecomment-703191787,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAALDFG3CZI6XGTGONK62YLSI7OCXANCNFSM4SDM4DTQ
.


Ensure the listen addresses are consistent with the transport

This PR improves the error message AND return early with a failure if the network configuration is inconsistent.
The only checks that are done are:

if the TransportConfig is MemoryOnly but a listening address is not Memory
if the TransportConfig is not MemoryOnly but a listening address is Memory

Example during test before:

2020-06-19 14:22:16 Can't listen on /memory/10872111545232357078 because: MultiaddrNotSupported("/memory/10872111545232357078")
(continue running the done)

Example during test after:

thread 'ensure_test_service_build_blocks' panicked at 'called Result::unwrap() on an Err value: Network(The network address /memory/18303963195922760223 is invalid because the transport is not set on MemoryOnly)', polkadot-test-service/tests/build-blocks.rs:31:21

In cli there will be a change of behavior as the node won't start at all but the process will stop.
:thinking: it also detected something wrong in the test service::tests::notifications_state_consistent that I fixed.
Related to #6409

Well, if we do that, we also have to check external_addresses, boot_nodes, and reserved_nodes

@tomaka I can't do it for external addresses as this is populated during runtime. I don't think it's necessary to check anyway since it's not user/developer input anyway.


How to configure Ruff to disallow double assignments in single lines in Python?

I'm looking for a way to configure Ruff to disallow double assignments in single lines, like this:
name, position = files.name, pos

Instead, I want the linter to enforce separate assignments on separate lines, like this:
name = files.name
position = pos

However, I still want to allow tuple unpacking, like this:
name, position = extract_details()

Is there a way to configure Ruff to achieve this behavior?

I don't think such a rule exists today. I just tried to see if Ruff flags the assignment when enabling all rules but it doesn't.
https://play.ruff.rs/e1feb7f2-7c32-4bfd-ae87-b12102a382a4

Hi @MichaReiser , thanks for replying. Do you think it would be possible to add this rule? :slightly_smiling_face:

+1

What would the logic for a rule like this. E.g would the following be allowed?
b = [1, 2]
name, position, b = *b, 2

I think that would be allowed.


Support for customized built-ins

Created CustomizedElement.js where I've been playing around with these.
Ideally use a decorator (with @tag) or as a mixin to allow dynamic sub-classing via element type, and/or interface, but type/name seems easier, i.e.
@tag({name: 'cool-button', extends: 'button'})
or
class CoolButton extends Element({name: 'cool-button', extends: 'button' }) {}
https://www.chromestatus.com/feature/4670146924773376

https://html.spec.whatwg.org/multipage/custom-elements.html#custom-elements-customized-builtin-example


[WIP] Architecture documentation

In this PR I will create an elaborate set of documentation regarding the internal workings and architecture of phpDocumentor. Due to the size of this effort it will probably take some time to be finished but I want to offer it as a WIP PR for everyone to see and review.
Important This PR contains information of changes found on the develop branch for phpDocumentor; which means that some of the features mentioned in this documentation will not be applicable for phpDocumentor2 but become available with phpDocumentor3.
TODO:

[ ] Change plantuml sphinx integration to include external UML files
[ ] Write a listing of available events and what they do
[ ] Expand conventions document
[ ] Review and expand doc on how descriptors are built
[ ] Add missing chapters described in outline
[ ] Review complete set of documentation

Closing this PR because the phpDocumentor 3 architecture is documented along the way and it supercedes this effort and information


Implement signature verification of Terraform binaries instead of blindly trusting CDN

This is a feature request, I consider this to be a mandatory requirement personally.
I am doing the below instead for now. Hope this helps anyone looking for a solution to this.
curl https://apt.releases.hashicorp.com/gpg -L --output /tmp/hashicorp.gpg
hashicorp_gpg_hash=$(sha256sum /tmp/hashicorp.gpg | awk '{print $1}')
if [[ "$hashicorp_gpg_hash" != "cafb01beac341bf2a9ba89793e6dd2468110291adfbb6c62ed11a0cde6c09029" ]]; then
  echo "File does not match pinned SHA256 hash. Potential supply chain attack detected on Hashicorp repo."
  exit 1
fi
cat /tmp/hashicorp.gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update
sudo apt install -y terraform

Hi, thanks for you feature request. We will take a look and see when we can get this implemented. Br Manuel


feat: getAllDependents & getAllDependencies

Purpose

Describe the problems, issues, or needs driving this feature/fix and include links to related issues.

Fixes #36113
Approach
In the modified code, I've added two new methods: getAllDependents and getAllDependencies. These methods use recursion to traverse the graph and collect all direct and indirect dependents/dependencies of a given node. The helper methods getAllDependentsRecursive and getAllDependenciesRecursive are used for the recursive traversal.
Check List

[ ] Read the Contributing Guide
[ ] Updated Change Log
[ ] Checked Tooling Support (#)
[ ] Added necessary tests

[ ] Unit Tests
[ ] Spec Conformance Tests
[ ] Integration Tests
[ ] Ballerina By Example Tests

[ ] Increased Test Coverage
[ ] Added necessary documentation

[ ] API documentation
[ ] Module documentation in Module.md files
[ ] Ballerina By Examples

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

I created the tests as requested

@sameerajayasoma could you also label the PR as hacktoberfest-accepted before 31 october? Thank you


kernel: spinlock: k_spin_is_locked introduction

Currently spinlock internals are directly accessed from the tests. This way the test becomes bound to the particular spinlock implementation. To remove this unnecessary dependency the distinct API to check if spinlock is locked is introduced.
k_spin_is_locked should be used for the spinlock testing only, so the scope of this API is intentionally restricted.

Hi @evgeniy-paltsev, sorry for the disturbance!
I've made a small clean-up for the comment, so your approval was lost.

@andyross , @nashif , @npitre , @evgeniy-paltsev , guys, I'm not authorized to merge this PR. Could you please help?

Suggestion: put the comment inside the #ifdef and remove the /** marker
by using only /*. This is for internal testing purposes so this doesn't
have to be part of the auto-generated documentation.

Suggestion: put the comment inside the #ifdef and remove the /** marker by using only /*. This is for internal testing purposes so this doesn't have to be part of the auto-generated documentation.

FWIW, Doxygen already ignores anything prefixed with z_, but good call nevertheless :)
https://github.com/zephyrproject-rtos/zephyr/blob/1eab46e36dc21e3eed53335f9b1cfda8189fe2c0/doc/zephyr.doxyfile.in#L1015-L1018

@dcpleung , @galak , @carlescufi , @nashif , @MaureenHelm , guys, could you please take a look at k_spin_is_locked API ? Thanks!

@amr-sc I was waiting for @andyross' approval to merge this. Will ping him on Discord.

@amr-sc I was waiting for @andyross' approval to merge this. Will ping him on Discord.

Heh.  My position remains a very mildly passive-aggressive "I don't see the need, but it's harmless".   So no +1, but go right ahead.


Account recovery

Purpose
Adding recovery of accounts and credentials, using only the ledger as a backup.
Changes

Updated NoIdentitiesView to link to the recovery page.
Added Path to Recovery in the settings menu.
Updated import to replace "placeholder" identities with imported identities that are real.
Added the recovery algorithm, and the view to run it.

Checklist

[x] My code follows the style of this project.
[x] The code compiles without warnings.
[x] I have performed a self-review of the changes.
[x] I have documented my code, in particular the intent of the
hard-to-understand areas.
[x] (If necessary) I have updated the CHANGELOG.

If I stop recovery early and go back to the account page, then the accounts are not there. If I close the desktop wallet and open it again, then they are there. So it appears there are some issues with the state handling.

After having found the accounts for index 0 it says:
Identity Index 1:
Looking for accounts...

even though I have not submitted anything to the Ledger yet.

If I recover for index 0 it says (after recovering)
Identity index 0:
Done: Found 4 accounts.

If I press Stop recovery, I found all my accounts and then Go back and look for more, then it shows as:
Identity index 0: (Recovered: Index 0)
Done: Found 4 additional accounts.


Remove tab/TabPage usage where it's not needed

tab query argument and TabPage in context is used only in profile so remove it from all other places where it's not used anymore.

tab query argument and TabPage in context is used only in profile so remove it from all other places where it's not used anymore.

Do you mean TabName? I can see it set in:
// SetDefaultParams sets common pagination params that are often used
func (p *Pagination) SetDefaultParams(ctx *Context) {
	p.AddParam(ctx, "sort", "SortType")
	p.AddParam(ctx, "q", "Keyword")
	p.AddParam(ctx, "tab", "TabName")
	// do not add any more uncommon params here!
	p.AddParam(ctx, "t", "queryType")
}

and used by:

Not sure whether it's used or not by explore (I haven't tested yet 😂)

Sorry for bothering, but the code symbol is TabName, not TabPage, what does TabPage refer to? Did I misunderstand something ....


Keep filename extension in rotation

Insert file index before filename extension (if exist) and get
filename.0.log
filename.1.log
filename.2.log

instead of
filename.log.0
filename.log.1
filename.log.2

On my FreeBSD 12 system, the index comes at the end of the rotated name
(freebsd-var-log.txt). logrotate appears to allow preserving the extension if you specify it.
@inikulshin is there a specific problem you're trying to solve?

Excuse me, I should have mentioned, that I'm working with Windows, which gives special attention to extensions.
E. g. I can define automatic opening of  *.log files with some text editor. But .0, .1, etc. - are unknown extensions and double clicking on them (in explorer) opens dialog with programs list, where I should choose the desired text editor. At least once for each number.

OK, thanks for the explanation. The fastest way to get this implemented would be to submit a pull request with tests.

A workaround on Windows would be to set your text editor as the application for files ending in .0, .1, etc

Sure, that's why I wrote "At least once for each number" and not "Every time" :)


allow disabling of Google Container Registry credential checks

This is a continuation of #35; I added a commit addressing my own feedback to be able to merge and release this quicker.

I released version 1.3.3 of the plugin a few minutes ago with this change in it.


Give each item in opening hours and menu items dropdown more space

Fixes #160

Does rundhallen need to be displayed in opening hours? That would remove one item from the list


[Question] What about the layers

The template file references lib folder which is absent from the code.
So what is going on with the layers here?

:+1: looks like the packages folder is missing in the blank-python example

These examples include a script that downloads dependencies and stages them in the layer folder-
https://github.com/awsdocs/aws-lambda-developer-guide/blob/main/sample-apps/blank-nodejs/2-build-layer.sh
https://github.com/awsdocs/aws-lambda-developer-guide/blob/main/sample-apps/blank-python/2-build-layer.sh


Create release

Hi there,
Is there a standard periodic release process for this package or are releases initiated manually by the maintainers?
I had a PR merged in https://github.com/aws/aws-lambda-go/issues/452 which I'd like to use. I see the last release was created 18 days ago.
Looking at the history it doesn't seem like there's a regular cadence, could you please create a release so that the above changes can be used?

done! https://github.com/aws/aws-lambda-go/releases/tag/v1.33.0

Much appreciated.


Advanced Properties and Parameters for Bands

Band and BandEmitter still has a bunch of magic numbers, especially with the timeline animations that should be exposed as parameters and properties.

fixed with this merge request
https://github.com/Birdasaur/LitFX/pull/13


choose build pod by name AND namespace

This will ensure that the ContainerBuildPod associated with a ContainerGroup is from the same namespace.
Tests are in https://github.com/ManageIQ/manageiq-providers-openshift/pull/36 due to the inexistance of tests for this file in this repo.

@cben @moolitayer PTAL at this one liner
@miq-bot add_label bug

@enoodle is the a ticket associated with this bug?

@moolitayer This is a bug only in the backend that doesn't have an effect on the ui user experience.

@blomquisg @Fryguy PTAL

LGTM :+1:
@miq-bot assign blomquisg
cc @blomquisg

@miq-bot add_label fine/yes
@enoodle we want to backport this to fine, we need a BZ.

BZ added

ping @blomquisg

Fine backport details:
$ git log -1
commit a0f7105f213d1429c6f9a12bda54350e8cb64074
Author: Greg Blomquist<EMAIL_ADDRESS>Date:   Thu Jul 27 09:09:39 2017 -0400

    Merge pull request #15575 from enoodle/namespace_container_build_pod_identification

    choose build pod by name AND namespace
    (cherry picked from commit ed83f8b65addc09c4f8f729716181c558ecb3863)

    https://bugzilla.redhat.com/show_bug.cgi?id=1478568


How to embed tab-view inside a bigger view

I have a requirement where the tab-view is created as per the button clicked in the footer. So basically there are 3 buttons at the footer and clicking any of these should create specific tab-view with a footer (so there will be two footers) and the scene will be generated on the basis of the tab selected in tab-view. So the cumulative effect is that a the scene is determined by both the footer's selected value.
To implement this I am trying to define TabViewAnimated inside View so that I can add a sibling to it. Doing this results in empty screen. How to fix this?

Try setting style={{flex: 1}} on your TabView - sounds to me like it currently has 0 height. Check the flex styling on any containing views as well - somewhere in your view hierarchy you're likely missing a flex style that is causing the view to be 0 height.

Hey, I just released a new alpha 2.0.0-alpha.0 of the library. It's rewritten using react-native-gesture-handler and react-native-reanimated addresses a many platform specific bugs and performance problems.
Please try the new version and see if it addresses your issue. If not, please open a new issue following the issue template.

Hey @avratmic  Did you finally solve this this?


Use relative time instead of expiration time when forwarding all local sessions

Description

Avoid problems with clock inconsistencies between servers.

@nicelulu we will add it to next release and pls add it to milestone v2.0.2


Issue with German translation?

We received this question from a German user:

nice website with great functionality. I tried the german version out of
curiosity and came upon the word
"Fällung" for precipitation. I have never heard it used in this
context before. I would suggest that "Niederschlag"
is a better translation and more commonly understood in this context.
Faellung is both a term used in chemistry for precipitation or has to do
with chopping trees.
(If you look up precipation on Wikipedia and than look for that page in
other languages and select german it will take you to a page about
Niederschlag and talk about the weather.)

Oops, this applies to the website, not to the API!


Config

This isn't actually a feature request so I apologize for that but I've tried the CKEditor site and can't get anywhere with my question.
I'm not sure how to configure the toolbar in my code. I just have the default toolbar but I need at least font colors (background/foreground) and would like to add some more. The Toolbar Configuration tool from CKEditor is nice but I'm not sure what to do with the code.
Sorry, I'm still a n00b and just looking for some direction. Thanks in advance.

Sample toolbar you can reference here.
You should use Full Package of Ckeditor for full toolbar

I'm using the following code to specify config but it's being ignored. Just below the data() { return ... } function is the components: { VueCkeditor }, by the way.

What am I doing wrong that the config isn't working?
Thanks in advance.

Ignore the last post/question. I had inadvertently omitted the :config="config" attribute.


Text in sidebar ?

Good morning,
What would be great is to be able to put text in the sidebar for example after a separation.
Would it be possible to add it in the next update?
Great job!

You can add permanent/non-changing text via Templates but I'm not sure if you're referring to the ability to add a Heading?

I put counts in my sidebar, and I would have liked to put a title on each count...

Ah, a title/name option for timers to distinguish them, that's reasonable


Fixes #8640: Update to lift 2.6.3 / scala 2.11.8 (security)

https://www.rudder-project.org/redmine/issues/8640

Tested, and works without any problem.

OK, merging this PR


Implement caller/publisher options

I have added support for call options. It came in with #51

Came across this while looking for provide_options for procedures.
How to specify provide_options while registering new method providers? I would like to use round-robin mechanism etc. to avoid the 'provider already registered' errors.

Also, the invoke values here define roundrobing as one of the possible values. - is it a typo? if not what is the g stand for?

Tried this code. But still getting wamp.error.procedure_already_exists error
autobahn::provide_options options;
options.insert_or_assign("invoke", msgpack::object("last"));

for (auto&& rpcMethod : rpcMethods)
{
	session->provide(rpcMethod.uri, rpcMethod.handler, options).then([&](boost::future<autobahn::wamp_registration> registration) {
		std::cerr << "[" << getTimestamp() << "] registered procedure: " << rpcMethod.uri << " , " << registration.get().id() << std::endl;
	});
}

Didn't want to hijack this thread. Created new issue: https://github.com/crossbario/autobahn-cpp/issues/139


Add text at the footer?

Hey! I'd like to add text (Copyright text, etc.) to the bottom of my jekyll-now site. Any way I can do this? What file should I edit?

the default.html layout in the _layouts folder is where you can change this.
add © {{ site.name }} 2015 | All Rights Reserved.  to the    or what ever content you'd like. I did that to my site, claytonerrington.com

@averyburnettmiller a better copyright would be
<p>© {{ site.name }} {{ site.time | date: '%Y' }} | All Rights Reserved. </p>

the ruby  {{ site.time | date: '%Y' }} gets the time and outputs the whole year, 2015, and wont need to change this each year.

Ooo, that's an especially nice fix @cjerrington. Closing this :smiley:

Thanks!

@cjerrington  Thank You
<p>© {{ site.name }} {{ site.time | date: '%Y' }} | All Rights Reserved. </p>
Worked Perfectly 💕


Install section of README should point to docs

Need to remove the inline install section and point over to Installation and Support

/assign

@sbtaylor15 : Could you please guide where this change needs to be made?

@divya-mohan0209 its the section starting with Ortelius as a SaaS option up to Support in the https://github.com/ortelius/ortelius README.  @jiakalita maybe working on this.  (Its hard to assign things since not everyone is a member of the project but an external contributor)

Fixed


Use PhantomPinned to get !Unpin on ring types

I've started a prototype, using this crate and eventfd, to try an implement efficent async file-backed Stream and Sink types.  In review, I noticed this comment on IoUring:
https://github.com/tokio-rs/io-uring/blob/56531043ad78e38b1f27986d175b8d961112d3a4/src/lib.rs#L72
Given the self-referential nature, should SubmissionQueue, CompletionQueue, and IoUring use PhantomPinnned? As far as I can tell at present they all implement Unpin, automatically.  Or is this not relevant because these types don't own memory buffers, just contain pointers to memory mapped regions that won't be moved?   If the latter, then it might be possible to say something more assuring in place of the current comment? Thanks!

I think it is latter, and I will add some comment. Thank you.


HHH-12106 - Database name not quoted at schema update

https://hibernate.atlassian.net/browse/HHH-12106
an alternative solution to https://github.com/hibernate/hibernate-orm/pull/2074

@dreab8 I leave it up to you which to use.  IIUC however, both result in the generation of a QualifiedTableName

@sebersole I have double checked and my solution may have some issues related with schema and/or catalog. I'm dropping this PR.


Update ITK Superbuild version along the 5.1 development branch

This should address the valgrind defect on the dashboard along with many warnings.

The new test failure related to the ScaleSkewVersorTransfrom are related to InsightSoftwareConsortium/ITK#1523.


Add "Clear Sans" Typeface

This is the typeface I've been using in the design so far. Can we add the webfont to our project to start adding font styles? I'm mostly using Regular, Medium, and Bold, but it would be nice to have all of the weights.
I think this makes it simple, but also happy to add them via font-face:
https://github.com/resir014/Clear-Sans-Webfont

Oh nice, I wondered which font you had in your mockups!

I'll pull this into #10.

@cheshire137 yay! thank you!

Lol, maybe I'll link to the CDN version of the font...


Ability to refine the score

Hello, when I see the 4 first pwas in the score page :
https://pwa-directory.appspot.com/pwas/score
I notice that there are some apps that do not always give a score of 100 results for the lighthouse score.
Furthermore, score for other audits ( performance, accessibility , ...) are in some cases not so good with warnings.
For example, the first one ( https://web-performance-dot-pwa-directory.appspot.com/lighthouse-viewer/?pwadir=4658345284206592 )
says : Includes front-end JavaScript libraries with known security vulnerabilities.
Could you refine the score page so that it reflects a really good global score.
Thanks.

Hi @hugues-koolsol
You are right, probably we need to change our score's names, Lighthouse has evolve to have more weight in multiple tests and we still call Lighthouse the PWA test. We will look into it.
For the second part, probably we could surface warnings into our PWA page with some annotations in the graph.
Thank you,
Julian


[Tou.tv] UnicodeDecodeError: 'utf8' codec can't decode byte 0x96 in position 17: invalid start byte

[Tou.tv] UnicodeDecodeError: 'utf8' codec can't decode byte 0x96 in position 17: invalid start byte
Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2016.09.08. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

[x] I've verified and I assure that I'm running youtube-dl 2016.09.08

Before submitting an issue make sure you have:

[x] At least skimmed through README and most notably FAQ and BUGS sections
[x] Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

[x] Bug report (encountered problems with youtube-dl)
[ ] Site support request (request for adding support for a new site)
[ ] Feature request (request for a new functionality)
[ ] Question
[ ] Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:
Add -v flag to your command line you run youtube-dl with, copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):
$  youtube-dl -v http://ici.tou.tv/vox-pop/S01E02
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'http://ici.tou.tv/vox-pop/S01E02']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.09.08
[debug] Python version 2.7.12 - Linux-4.4.0-36-generic-x86_64-with-Ubuntu-16.04-xenial
[debug] exe versions: ffmpeg 2.8.6-1ubuntu2, ffprobe 2.8.6-1ubuntu2
[debug] Proxy map: {}
[tou.tv] vox-pop/S01E02: Downloading JSON metadata
[radiocanada] 122195: Downloading ipad XML
[radiocanada] 122195: Downloading m3u8 information
[radiocanada] 122195: Downloading android XML
[radiocanada] 122195: Downloading m3u8 information
[radiocanada] 122195: Downloading f4m manifest
[radiocanada] 122195: Downloading m3u8 information
[radiocanada] 122195: Downloading f4m manifest
[radiocanada] 122195: Downloading m3u8 information
[radiocanada] 122195: Downloading f4m manifest
[radiocanada] 122195: Downloading m3u8 information
[radiocanada] 122195: Downloading f4m manifest
[radiocanada] 122195: Downloading m3u8 information
[radiocanada] 122195: Downloading f4m manifest
[radiocanada] 122195: Downloading metadata XML
[debug] Invoking downloader on u'http://medias-rtsp.tou.tv/vodtoutv/_definst_/mp4:012/mp4/v/2016-04-26_19_30_00_voxpop_0013_3000.mp4/chunklist.m3u8?wowzasessionid=1817449880'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 130
[download] Destination: Épisode 2-122195.mp4
[download]  62.3% of ~503.38MiB at  2.46MiB/s ETA 01:35Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/local/bin/youtube-dl/__main__.py", line 19, in <module>
  File "/usr/local/bin/youtube-dl/youtube_dl/__init__.py", line 434, in main
  File "/usr/local/bin/youtube-dl/youtube_dl/__init__.py", line 424, in _real_main
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1788, in download
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 702, in extract_info
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 773, in process_ie_result
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 748, in process_ie_result
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1432, in process_video_result
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1698, in process_info
  File "/usr/local/bin/youtube-dl/youtube_dl/utils.py", line 2133, in error_to_compat_str
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x96 in position 17: invalid start byte

<end of log>

Looks like a network issue. Could you try again? Also, if the problem persists, could you try youtube-dl on Python 3?


scheduler "error persisting job" due to 404 from controller

Not sure what causes this, I've seen it in most Flynn clusters that have anything interesting running on them (look at CI logs for lots of examples).
lvl=eror msg="error persisting job" component=scheduler fn=RunPutJobs job.id=f124815b-b30e102b-c00c-42b3-a38b-1a14d6863c9f job.state=down err="controller: resource not found"

This could happen when deleting an app, which scales down related formations but then persisting the job state will get a 404 as the app has gone.

We should probably not scope jobs by app for PutJob as we need to mark the job as down regardless of whether the app is deleted or not.


[balsa] Verify URL.

This fixes Http1ServerConnectionImplTest.Http11InvalidRequest, and adds
extensive testing to the URL validator that both http-parser and the new
isUrlValid in BalsaParser pass.
Tested by patching in #22039 and running
bazel test //test/common/http/http1:codec_impl_test
--test_arg=--runtime-feature-override-for-tests=envoy.reloadable_features.http1_use_balsa_parser
Tracking issue: #21245
Signed-off-by: Bence Béky<EMAIL_ADDRESS>
Commit Message:
Additional Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Fixes commit #PR or SHA]
[Optional Deprecated:]
[Optional API Considerations:]

/assign @diannahu @birenroy
Dianna and Biren: PTAL.

/assign @yanavlasov

@bencebeky needs merge and fixing merge conflicts.

/wait

Thank you for the review.  I resolved the merge conflicts, and all checks have passed.  This PR seems to be ready to be merged.


Classifier-free guidance

Hi,
Just going over the code line-by-line and checking things ;)
this line
I think
noise_pred = noise_pred_uncond + guidance_scale * (noise_pred_text - noise_pred_uncond)

should be
noise_pred = noise_pred_text + guidance_scale * (noise_pred_text - noise_pred_uncond)

to match the equation of the paper
Happy to do a PR if you confirm it needs to be change.
Thanks,

@ThibaultGROUEIX Hi, thanks for the checking! I simply followed diffusers' tutorial here. You can find it in the last part.

indeed ! Seems like either the paper or the tuto has a typo. I a not sure which. It may not matter in this case since the guidance parameter is so high.

@ThibaultGROUEIX Did you make this change work?

There are two forms to express classifier-free guidance:

epsilon = epsilon_uncond + s * (epsilon - epsilon_uncond), which is used in GLIDE (Nichol et al., 2022), LDM and Stable Diffusion (Rombach et al., 2022)
epsilon = (1+w) * epsilon + w * epsilon_uncond, which is used in the classifier-free guidance original paper (Ho and Salimans, 2021) and DreamFusion (Poole et al., 2022)

Both of them are correct. But for the first case, you should set s>1 to enable classifier-free guidance, and set w>0 instead in the second case.


Get MXNetError: Check failed: to.IsDefaultData() when using validation data in WaveNet

Description
When trying to use validation data in WaveNet, I get the following error:
MXNetError: Traceback (most recent call last):
  File "../src/ndarray/ndarray.cc", line 1146
MXNetError: Check failed: to.IsDefaultData():

To Reproduce
The following code produces an error (see below).
from gluonts.mx.model.wavenet import WaveNetEstimator
from gluonts.mx.trainer import Trainer
from gluonts.dataset.repository.datasets import get_dataset

dataset = get_dataset("m4_hourly")
estim = WaveNetEstimator(
    freq = 'H',
    prediction_length=2,
    trainer=Trainer(epochs = 4, num_batches_per_epoch = 10)
)
estim.train(dataset.train, dataset.test)

Note that the same code without the validation set does NOT produce an error:
from gluonts.mx.model.wavenet import WaveNetEstimator
from gluonts.mx.trainer import Trainer
from gluonts.dataset.repository.datasets import get_dataset

dataset = get_dataset("m4_hourly")
estim = WaveNetEstimator(
    freq = 'H',
    prediction_length=2,
    trainer=Trainer(epochs = 4, num_batches_per_epoch = 10)
)
estim.train(dataset.train)

Error message or code output
---------------------------------------------------------------------------
MXNetError                                Traceback (most recent call last)
<ipython-input-1-24de9e6a8a0f> in <module>
      9     trainer=Trainer(epochs = 10, num_batches_per_epoch = 20)
     10 )
---> 11 estim.train(dataset.train, dataset.test)

~/anaconda3/lib/python3.8/site-packages/gluonts/mx/model/estimator.py in train(self, training_data, validation_data, shuffle_buffer_length, cache_data, **kwargs)
    236         **kwargs,
    237     ) -> Predictor:
--> 238         return self.train_model(
    239             training_data=training_data,
    240             validation_data=validation_data,

~/anaconda3/lib/python3.8/site-packages/gluonts/mx/model/estimator.py in train_model(self, training_data, validation_data, from_predictor, shuffle_buffer_length, cache_data)
    220
    221         with self.trainer.ctx:
--> 222             predictor = self.create_predictor(transformation, training_network)
    223
    224         return TrainOutput(

~/anaconda3/lib/python3.8/site-packages/gluonts/mx/model/wavenet/_estimator.py in create_predictor(self, transformation, trained_network)
    388         prediction_network.initialize()
    389
--> 390         copy_parameters(
    391             net_source=trained_network,
    392             net_dest=prediction_network,

~/anaconda3/lib/python3.8/site-packages/gluonts/mx/util.py in copy_parameters(net_source, net_dest, ignore_extra, allow_missing)
    112     ) as model_dir:
    113         model_dir_path = str(Path(model_dir) / "tmp_model")
--> 114         net_source.save_parameters(model_dir_path)
    115         net_dest.load_parameters(
    116             model_dir_path,

~/anaconda3/lib/python3.8/site-packages/mxnet/gluon/block.py in save_parameters(self, filename, deduplicate)
    450         arg_dict = {key: val._reduce() for key, val in params.items()}
    451         save_fn = _mx_npx.save if is_np_array() else ndarray.save
--> 452         save_fn(filename, arg_dict)
    453
    454     def save_params(self, filename):

~/anaconda3/lib/python3.8/site-packages/mxnet/ndarray/utils.py in save(fname, data)
    275         raise ValueError("data needs to either be a NDArray, dict of str, NDArray pairs "
    276                          "or a list of NDarrays.")
--> 277     check_call(_LIB.MXNDArraySave(c_str(fname),
    278                                   mx_uint(len(handles)),
    279                                   handles,

~/anaconda3/lib/python3.8/site-packages/mxnet/base.py in check_call(ret)
    244     """
    245     if ret != 0:
--> 246         raise get_last_ffi_error()
    247
    248

MXNetError: Traceback (most recent call last):
  File "../src/ndarray/ndarray.cc", line 1146
MXNetError: Check failed: to.IsDefaultData():

Environment

Operating system: Linux
Python version: 3.8.8
GluonTS version: 0.11.1
MXNet version: 1.9.1

@VictorJouault this appears to be a bug related to the ModelAveraging callback, which is used by default in the Trainer class. Turning off default callbacks with add_default_callbacks=False when constructing the Trainer gets rid of the issue.
On to finding what's actually causing it.

Something in the network changes with this line: https://github.com/awslabs/gluonts/blob/d5d897cd02b3aaa499f3c3632d3a18863b1a24e7/src/gluonts/mx/trainer/model_averaging.py#L310
Trying to save the network parameters (with training_network.save_parameters(SOME_PATH)) right before that line is fine; doing the same right after that line raises the exception as in the OP.

Thanks for looking into this. I can confirm that on my side, using add_default_callbacks=False solves the issue.
However, I was using the LR Scheduler. Removing the ModelAveraging callback by itself doesn't seem to take care of the issue, as even with only the LearningRateReduction callback, the same exception is raised. Could it be linked?

Could it be linked?

I’m not sure, I actually saw it working if just comment out the ModelAveraging callback from the default ones (and leave learning rate reduction in).
What I could find so far: the issue arises if the network is loaded back from a checkpoint after running in “prediction mode”: the next save will trigger the exception. This can be seen by adding the following lines in the training loop here
if not is_training:
    net.save_parameters(SOME_FILE_PATH)
    net.load_parameters(SOME_FILE_PATH)
    net.save_parameters(SOME_FILE_PATH)

The first save is just to store the parameters so we can load them back; the second save triggers the issue. This is rather strange.


Downloading only a subset of the data

When the "show" tab is not set to "All", only a subset of the data is downloaded.

Hi! I discovered that this is actually a known issue with DataTables that occurs with a recent update where they changed a default setting. It's described in this forum question of theirs: https://datatables.net/forums/discussion/78627/select-all-checkbox-not-working-for-non-dom-sourced-data#Comment_230262. After changing the said setting back to the old value JATOS' result tables work again as expected. This fix will be part of the next release. I can't say for sure when it will be released but it shouldn't take too long.

Nice :) Thanks Kristian!


Update SpaceService to allow editing of spaces

What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

Feature. Fixes #91 to allow consumers of SpaceService to perform editing of spaces.

What is the current behavior? (You can also link to an open issue here)

SpaceService does not support updates to space instances.

What is the new behavior (if this is a feature change)?

A new method to support PATCH requests for space instance objects is created under SpaceService.

Other information:

Builds on top of #90

Can one of the admins verify this patch?

[test]

Waiting for almighty-core PR to get merged.

[test]

Closing this as this work has been subsumed into ngx-fabric8-wit entirely.


depo curation call to sumstats service for 6130d5e6088ad000014c5e72 needs to pass

this is due to the outstanding issue #67 where depo curation should not call the sumstats service.
We need to make this pass.

I believe this is already sorted , please review & close


Helm Charts don't work with helm 2.15.0

Bug description
helm 2.15.0 fails to properly enumerate .Values.global.arch in _affinity.tpl when the values are integers. This results in deployments missing key elements and values for deployments, for example:
          name: istio
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
          preferredDuringSchedulingIgnoredDuringExecution:

Quoting the values in values.yaml like
  #   0 - Never scheduled
  #   1 - Least preferred
  #   2 - No preference
  #   3 - Most preferred
  arch:
    amd64: "3"
    s390x: "2"
    ppc64le: "2"

produces the desired yaml.
Expected behavior
Istio helm charts should produce useable charts with the latest stable Helm version (2.15.0)
Steps to reproduce the bug
helm fetch --untar --untardir charts istio/istio
helm template charts/istio --namespace istio-system  --output-dir  .

Version (include the output of istioctl version --remote and kubectl version)
$ kubectl version --client=true
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T19:18:23Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
$ helm version
Client: &version.Version{SemVer:"v2.15.0", GitCommit:"c2440264ca6c078a06e088a838b0476d2fc14750", GitTreeState:"clean"}

How was Istio installed?
This is an installation error with the Helm charts.
Environment where bug was observed (cloud vendor, OS, etc)
This is pre-deployment.

It is a bug in helm 2.15, workaround in  https://github.com/istio/istio/pull/18170.


[PrintAsObjC] Use 'unsafe_unretained' to print 'unowned', not 'assign'

The ObjC generator previously preserved a subtle difference between unowned and unowned(unsafe) / Unmanaged by printing the former as assign and the latter as unsafe_unretained. Upstream Clang, however, has gotten a new warning to discourage the use of assign with reference-countable types at all. Since it was always a subtle distinction, just go with the new convention and print unsafe_unretained for unowned properties as well.

@swift-ci Please test source compatibility

@swift-ci Please smoke test Linux

@swift-ci Please test macOS

Good suggestion, spotted another reasonable cleanup while I was doing that.
@swift-ci Please smoke test


Fixing errors for CM6 code input.

Fixing error where once a CM6 field has been opened it can't be opened again without a page refresh, as documented in #296.

Thank you for looking into this! So this fixed it?
I'll have to check what this piece of code was doing first.


Add diagnostics plots

Improve plotting capabilities by adding more diagnostics plots.

Cell size distribution
Histograms for markers

Addressed in #69
Syntax is: img.pl.histogram("area")

Wolfgang noted that it would be better to just show users how to export the data and use pyplot on it themselves. So this method should be removed

Addressed in #79


Syntax highlighting for escape sequences in strings

I have added a new semantic token type ESCAPE_SEQUENCE as the LSP specification does not seem to have an appropriate token type. This may actually be a regression for some users, as the TextMate Rust grammar has a scope constant.character.escape.rust which highlights escape sequences (which caused problems with semantic highlighting, see #4138).
Fixes #2604.

bors r+