Set reset coords

@rsignell-usgs @martindurant I have implemented set/reset coords and plot generation according to our discussion here . There were few points, which I did a bit differently. I would soon add a detailed description of the functionality implemented in this dashboard. In the mean time, please have a look at the interface.

In case of DataArray, since there is only single variable, Set coord has not been implemented. For a DataSet, it has been implemented.
In the Set Coords tab, Cross selector displays all the variables preset in the DataSet. User selects the variable from left side and move them to right side by >> button. After making selection, click on the Set Coords button, which  converts the vars to coords. Reverse operation (coord to var) is also possible.
We cannot reset the indexed_coords.
Each time user clicks on the Set Coord:

Display section is updated to annotate a coord with ‘📈’
Description section updates.
Selectors in Fields panel update

For each of dashboard,control,display,describe,fields,coord_setter, a method set_coords has been implemented which performs necessary updates required in that particular panel.
In the fields panel, x displays all the options (var_dims+non_indexed_coords), while y is selective.

If var_dim has been selected in x, y also displays var_dim.
Same is the case for non_indexed_coords.
In case of non_indexed_coords, dimensionality of two vars need to be same, so only the coords having same ndim as x are displayed in y.

If dims have been selected, data.var.sel method is used for plotting.
If coord is selected, data.hvplot.quadmesh(x = 'coord1', y='coord2') method has been used.
Please have a look by playing around with the interface.

@martindurant @rsignell-usgs Please review the changes.

@rsignell-usgs @martindurant Unlike dims, in case of creating graph for a variable with respect to a coord, Aggregations would be empty. Also there doesn't seem any generalisable way to  know beforehand, along what coords/vars index selectors would be generated. So is it fine if we are displaying both selector and player in the graph output?

@hdsingh, I was just about to comment on this.   If coords like lon and lat are selected, it seems that aggregation along other dimensions like sigma and time should still work.

Yes,  it seems that we can have aggregation for a variable along a dimension, even when x and y Select has coord as options. I will check and implement this. But in this case Aggregation options will not include Select and Animate as option because we don't know beforehand for sure, if that dim would be present in the generated index selectors.

We can have aggregation for a variable along a dimension, when x and y Select has coord as options.
When a coord has been selected in x and y, we can only aggregate it along the dims which are not present in x or y. For example :
Var: temp
x: lat
y: lon
So now we can't aggregate along dims nx and ny since they are already present in lat and lon. Only aggregation only along remaining dims: time and sigma is possible.

@rsignell-usgs @martindurant Everything is working as expected. Please have a look.

With this latest commit, I'm getting:
(xrviz) C:\Users\rsignell\Documents\GitHub\xrviz\examples>python control_section.py
Traceback (most recent call last):
  File "control_section.py", line 7, in <module>
    dash = Dashboard(ds)
  File "c:\users\rsignell\documents\github\xrviz\xrviz\dashboard.py", line 34, in __init__
    self.control = Control(self.data)
AttributeError: 'Dashboard' object has no attribute 'data'

Now it is working. I will soon simplify the code using data = xr.Dataset({data.name: data}, attrs=data.attrs) as discussed above.

With this latest commit

I will make a PR, perhaps after Wednesday's meeting, implementing Travis CI. We are probably starting to have enough functionality which is concrete that thorough testing is warranted.

@hdsingh, when I select lon and lat as coordinates, I get x=lat, y=lon by default.   Is it possible to get x=lon and y=lat as default, or is it impossible to tell what should go where?

Yes correct defaults need to be set. I will look in detail how this could be done from this answer and will implement this in one of upcoming PRs. As of now options are being displayed according to their alphabetical order.

Now there is no separate handling for DataArrays due to conversion while initialisation by data = xr.Dataset({data.name: data}, attrs=data.attrs).

@martindurant I have started writing tests. Please have a look and let me know if this looks fine. My aim here is to replicate the user behaviour and check if the interface is functioning as expected.

Yes, these tests look like a good start. I expect they are all passing.
I'll merge this now, so we can discuss next steps tomorrow.


[lostfilm] bypass captcha

issue: current login form with captcha

solutions:

resolve captcha

can help  https://github.com/sirfz/tesserocr

use OAuth 2.0

thanks!


fix nb-sticky css

fix: #189195
fixes styling for hc-light and hc-black themes in vscode. Adds background color and bottom border to sticky-container.

screenshot of fix:

Upper left -- current insiders
Lower right/middle -- OSS with fix.


Split Peerset into reputation store & ProtocolControllers

This is a draft PR that should resolve https://github.com/paritytech/substrate/issues/13531.
This is essentially connection logic (connect/drop/accept/reject) extracted from Peerset into ProtocolController and PeerStore. Each ProtocolController instance is responsible for a specific notifications protocol, and should allow customization of connection logic by protocols in the future. PeerStore is responsible for connection candidates and peer reputations.
Work in progress.

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable-int
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2528668

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2528663

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable-int
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2533674

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2533669

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2533668

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2533667

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable-int
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2534452

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2534446

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2534445

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2534447

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2537651

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2537650

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2537652

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2539223

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2539224

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2539222

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable-int
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2545755

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2545749

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2545748

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2545750

@altonen I've reworked peer management using ideas from your prototype. Apart from a couple of places where I had to call self.peerset_handle.is_banned() instead of self.is_banned(), borrowing issues are gone. Implementation of alloc_slots() feels sub-optimal, but that's likely an inevitable consequence of separating peer reputation values and connection handling.

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable-int
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2545809

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2545804

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2545803

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2545802

... could you write some tests to confirm this implementation works as expected?
Sure, that was my plan.

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2551473

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable-int
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2551479

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2551675

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2551673

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2552636

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2553727

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2553726

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable-int
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2554300

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2554294

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2554293

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable-int
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2555730

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2555723

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2555725

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable-int
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2555922

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2555916

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2555917

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2555915

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2558663

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2558717

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2563004

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2563006

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2563204

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable-int
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2566851

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2571805

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2571804

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable-int
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2583468

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable-int
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2584341

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable-int
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2585530

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable-int
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2586647

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2586641

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2586640

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2627040

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable-int
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2627121

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2627114

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-each-crate
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2638615

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2638599

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-each-crate
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2640708

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-each-crate
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2647189

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2647831

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2648022

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2656540

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-each-crate
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2656556

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2656617

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-each-crate
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2656633

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2660960

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2661010

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2661664

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2661715

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2666898

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2666896

It's looking very good.
Two questions:
1. could you test running this with a regular Polkadot full/validator node on your computer and see how it works it with. Especially what kind of load there is on the channels that this PR introduced

2. what was the issue with fuzz testing?

3. There is still some commented out code in `lib.rs` is that staying or going?

Sure, but the fuzz test still fails, which means the contract between Peerset and Notifications is broken. I'd like to fix that first and then run the real node.
Fuzz test fails because it thinks (acting on behalf of Notifications) that peers' state is different to what it observes through messages from Peerset (PeerStore and ProtocolControllers). Either it receives a command to connect a node which it thinks is already connected, or accept a node that was connected before by Peerset. That might be caused by the fact that now both Peerset -> Notifications and Notifications -> Peerset communications are asynchronous, and with old Peerset only Peerset -> Notifications was async and Notifications -> Peerset communication was done by Notifications directly calling Peerset methods. I'm not sure that it's possible to retain the old contract with both directions being async. May be, we'll need to handle such invalid commands in Notifications.
The commented code in lib.rs is tests of Peerset. I'm going to keep them for now with minimal adjustments to the new Peerset API, but in the future they should test the bundle of PeerStore and ProtocolController directly.

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable-int
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2678698

2. Fuzz test fails because it thinks (acting on behalf of Notifications) that peers' state is different to what it observes through messages from Peerset (PeerStore and ProtocolControllers). Either it receives a command to connect a node which it thinks is already connected, or accept a node that was connected before by Peerset. That might be caused by the fact that now both Peerset -> Notifications and Notifications -> Peerset communications are asynchronous, and with old Peerset only Peerset -> Notifications was async and Notifications -> Peerset communication was done by Notifications directly calling Peerset methods. I'm not sure that it's possible to retain the old contract with both directions being async. May be, we'll need to handle such invalid commands in Notifications.

I see. I have trouble understanding something though: the response to, e.g., incoming connection was received asynchronously so I don't understand why making the call to, e.g., ProtocolController::incoming_connection(), asynchronous makes any difference? Could you elaborate or am I talking about some other issue?

I see. I have trouble understanding something though: the response to, e.g., incoming connection was received asynchronously so I don't understand why making the call to, e.g., ProtocolController::incoming_connection(), asynchronous makes any difference? Could you elaborate or am I talking about some other issue?

Previously, Peerset knew the state of Notifications synchronously, and only Peerset commands sent to Notifications were async. There were some "fixes" in place to account for this delay (like ignoring some messages), but at least Peerset state was in sync with Notifications state. Now both are out of sync, and I have a feeling that this might be the reason of the failing test. To be honest, I wasn't able to understand so far why the test didn't fail previously and why just ignoring some messages were enough (considering a lot of commands can queue up and the states can be out of sync significantly).

Ok that mismatch may be an actual problem we need to address. FWIW, the eviction fix will make Peerset even busier than what it now is because peers are constantly getting evicted which causes, for example, GRANDPA and transactions to update their reserved peers accordingly. So when you're testing this code on a live node, it might be a good idea to rebase your work on top of those changes so see if something breaks.
As for this current issue, having both halves of the connection between Peerset and Notifications sounds a little risky if the tests are failing. Have you looked what kind of changes would it take Notifications to make the link between it and Peerset asynchronous?
I have the same plan with NotificationService, i.e., Notifications sends an inbound substream to protocol which then either accepts or rejects it and in the mean time this substream could've been closed or while it was sending the inbound info, the protocol might've asked to open a substream to the same peer. I think these cases need to be handled by Notifications and I've understood they're handled for the most part. In other words, having the connection between protocols and Notifications must be asynchronous so we can do these necessary changes. Notifications just has to accommodate us and handle these race conditions in the substream state gracefully.

As for this current issue, having both halves of the connection between Peerset and Notifications sounds a little risky if the tests are failing. Have you looked what kind of changes would it take Notifications to make the link between it and Peerset asynchronous?

Yes, we can try making Notifications -> Peerset channel async with old Peerset and seeing if it breaks the test. If it doesn't, than the problem is in some other place. I will try twisting my head around the logs of the failing test a little more, and if it doesn't help, implement this async channel.

I have made commands from Notifications to Peerset handled asynchronously with old Peerset, and this broke the fuzz test. Here is the code: https://github.com/paritytech/substrate/pull/13909. So we need to somehow workaround these async issues.

I have made commands from Notifications to Peerset handled asynchronously with old Peerset, and this broke the fuzz test. Here is the code: #13909. So we need to somehow workaround these async issues.

Alright that has to be fixed first. It could be that it's not an actual issue with substreams but that the test is just written in a way that assumes that connection is synchronous.

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable-int
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2683548

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2683543

The fuzz test didn't fail with old Peerset due to the combination of the following factors:

"incoming" and "dropped" commands from Notifications (fuzz test) to Peerset were sent synchronously via function calls.
Answers to "incoming" command (i.e., "accept"/"reject") were collected in a queue and dispatched as the first action during the polling of Peerset by 'Notifications` / fuzz test. This means that "incoming" processing was very close to synchronous.
All other commands from PeersetHandle were processed only after all "accepts" and "rejects" were processed by Notifications / fuzz test.

After refactoring, both commands from Notifications to "Peerset" and from "Peerset" to Notifications are sent via channels. This leads to previously (almost) synchronous "incoming" requests and responses being interleaved by other events from the fuzz test sent via handles. And "accepts" and "rejects" interleaved by these other commands break the assumptions of Notifications / fuzz test about what it can hear from Peerset.
So, may be, we need to adapt Notifications to such previously impossible orders of events. Another idea that comes into my mind is to make ProtocolControllers receive and process "incoming" & "dropped" requests synchronously, prioritizing them over all other commands. It's not clear though how to make ProtocolController allocate slots synchronously and if it is needed to avoid race conditions between slot allocation and incoming connections processing (previously it was done almost synchronously when Notifications polled Peerset).

I was also thinking of adding a separate, higher-priority queue for events coming from Notifications but I don't think it's a good idea long-term. The vision is to make Notifications work independently from protocol implementations so the link between them has to be asynchronous. Furthermore, having two levels of prioritization for events sounds quite complicated to get right and if we managed to get it working, it's just added complexity to the already-complicated implementation that we have to keep maintaining.
That being said, if we forget about the fuzz testing for now, how hard is it to make the link fully async? I'm looking at the code and since response to the validation request (incoming()) is already asynchronously received, it looks like we're already halfway there. I can't really see anything in incoming() that would require it to be synchronous. Same for dropped(), Notifications doesn't event expect a response from Peerset for that function. Could you make it fully async and test on a live node how it works and if there are any concerning logs from either Peerset or Notifications?

I've been running polkadot node for a few of hours with semi-async link between Notifications and Peerset (the code from https://github.com/paritytech/substrate/pull/13909). I.e., "incoming" and "dropped" commands for Peerset are delivered asynchronously, and the commands to Notifications are delivered by polling. No errors in the log so far.
But the problem is, even with the PR changes the fuzz test is passing if we prioritize processing of the commands to Notifications over other actions. And we need to run it really long for it to fail. So, I'm afraid we might hit problems on quite rare occasions and won't be able to detect them just by running a node for some time.
The real issue with Notifications and Peerset/ProtocolControllers communicating asynchronously is that both of them try to maintain the state of the peers, and this state is synchronized using async messages. So they don't really know the real state of each other, just the local representation of it. So, it's essentially a consensus problem, unless we find some dirty hacks to simplify it.

I think I'm not understanding something. Could you explain why the following would not work/what are the corner cases I'm missing:

Notifications receives an inbound substream and it sends the substream information for the protocol (ProtocolController) for validation. The result is polled asynchronously from FuturesUnordered or something.
While awaiting the validation result, the substream may close at which point it will read a stale response from the validation queue. This can be ignored silently as the validation and open events are separate events so for the first event the protocol only checks if the peer can be accepted and for the second event it actually accepts the connections.
When the validation result is received and if the substream is accepted, the substream handshake is concluded and the protocol is notified of a new open substream. If it's rejected, the substream is closed without informing the protocol about it any further.
The substream might be abruptly closed after the peer has been accepted at which point the protocol is notified of this and the substream is state is cleaned up by Notifications. If the protocol closes the substream, it calls whatever function to do that and Notifications destroys the substream context.

As long as events between the protocol and Notifications are in order (as they should be since it's a stream of events), I can't see what is the issue with them communicating over asynchronous channels. Since we're talking about network events, some defensive mechanisms has to be used either way because we're not in control of whatever happens on the network (substream being opened and closed at any point) and I believe Notifications accommodates with this quite well already. Could you elaborate on what is the issue with making it fully async and/or is there something I missed from the above description?

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable-int
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2691932

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2691926

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable-int
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2709505

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable-int
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2736496

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable-int
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2736640

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2736633

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2742230

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2742228

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2742229

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable-int
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2742779

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2742772

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2742773

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2750371

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable-int
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2750376

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2750370

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2750369

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable-int
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2753945

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2753939

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2753938

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2753940

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2754401

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable-int
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2754408

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2754402

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2754403

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable-int
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2754461

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2754455

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2754454

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2754456

The PR seems to be in a working state now. The things left are:

Replace rendezvous channels to Notifications with ACKs to not pause other peers processing by ProtocolControllers while waiting for ACK for one peer from Notifications. The idea is to introduce HashMap<PeerId, Outstanding>, where Outstanding is:

struct Outstanding {
    pending_ack: Option<oneshot::Receiver<()>>,
    actions: VecDeque<Action>,
    events: VecDeque<Event>,
}

When we are waiting for an ACK, do not include the peer into slot allocation and collect actions and events into buffers instead of instantly processing them. Once we received the ACK — process all outstanding commands.
2. Disconnect peers when their reputation is below the BANNED_THRESHOLD. Unfortunately, missed this one. This would likely require registering ProtocolControllers in the PeerStore for receiving a callback (and reverting Remove unused ProtocolHandle::disconnect_peer).
3. I worry a little about protocols (like BABE) working in not-reserved-only mode. Originally, once a peer was discovered, we added it only to the default (sync/block-announcements) peer set. Now all the protocols fetch peers from PeerStore. Not sure if this is going to be a problem. If so, we will need to follow peer sets explicitly via ProtocolHandles.

P.S. I've tested the PR in the warp-sync & major sync modes. Seems to work so far.

I worry a little about protocols (like BABE) working in non-reserved-only mode. Originally, once a peer was discovered, we added it only to the default (sync/block-announcements) peer set. Now all the protocols fetch peers from PeerStore. Not sure if this is going to be a problem. If so, we will need to follow peer sets explicitly via ProtocolHandles.

I don't understand this point. Any protocol following the syncing protocol's peer set should still be able to follow it the same was as before this PR. Both GRANDPA and transactions use the SyncEventStream to listen to accepted sync peers and add these peers as reserved peers. Why does querying peers from Peerstore change this behavior?

I worry a little about protocols (like BABE) working in non-reserved-only mode. Originally, once a peer was discovered, we added it only to the default (sync/block-announcements) peer set. Now all the protocols fetch peers from PeerStore. Not sure if this is going to be a problem. If so, we will need to follow peer sets explicitly via ProtocolHandles.

I don't understand this point. Any protocol following the syncing protocol's peer set should still be able to follow it the same was as before this PR. Both GRANDPA and transactions use the SyncEventStream to listen to accepted sync peers and add these peers as reserved peers. Why does querying peers from Peerstore change this behavior?

BABE was a bad example (does it actually work over sync protocol?), I probably should have written BEEFY. Following sync peers via reserved peers still works the same. What changed is, before this PR, only the sync protocol peer set was populated by the nodes discovered via DHT and by Identify (https://github.com/paritytech/substrate/blob/master/client/network/src/service.rs#L1457-L1467). This PR changed it to adding discovered nodes to PeerStore, which is the common source of peers for all protocols working in non-reserved peers mode. So I worry that some protocols may receive extra peers from PeerStore, which they did not try to connect to in the past.

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2796910

I worry a little about protocols (like BABE) working in non-reserved-only mode. Originally, once a peer was discovered, we added it only to the default (sync/block-announcements) peer set. Now all the protocols fetch peers from PeerStore. Not sure if this is going to be a problem. If so, we will need to follow peer sets explicitly via ProtocolHandles.

I don't understand this point. Any protocol following the syncing protocol's peer set should still be able to follow it the same was as before this PR. Both GRANDPA and transactions use the SyncEventStream to listen to accepted sync peers and add these peers as reserved peers. Why does querying peers from Peerstore change this behavior?

BABE was a bad example (does it actually work over sync protocol?), I probably should have written BEEFY. Following sync peers via reserved peers still works the same. What changed is, before this PR, only the sync protocol peer set was populated by the nodes discovered via DHT and by Identify (https://github.com/paritytech/substrate/blob/master/client/network/src/service.rs#L1457-L1467). This PR changed it to adding discovered nodes to PeerStore, which is the common source of peers for all protocols working in non-reserved peers mode. So I worry that some protocols may receive extra peers from PeerStore, which they did not try to connect to in the past.

I don't think it is a problem. As long as a protocol can follow other protocol's set I think it's fine. But we can try running this in a testnet first to see if that causes any observable issues.

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2802832

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable-int
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2803245

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2803464

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2804407

I was working on the integration and while the idea was that NotificationService would do all of the communication on behalf of ProtocolController and that the latter would be converted to an event-based approach (NotificationService would poll events from it and send them to Notifications), I think that change will have to wait a little bit. Otherwise the PR keeps growing and growing and it'll be hard to understand all the implications of the changes.
Maybe the best approach for now is to keep them separate and have substream acknowledgements happen in two phases: first ProtocolController will receive the substream and either reject or accept it and if it's accepted, the substream is forwarded to NotificationService for validation which can then either reject or accept it. If the substream is rejected, ProtocolController would have to be notified of this somehow. It's not ideal because it's two back-and-forth between Notifications and the protocol but it can be refactored once the initial changes are merged and verified to be working.
Any comments @dmitry-markin?

To be honest, I'm starting to think that may be splitting connection logic between Notifications (NotificationsService?) and ProtocolController, communicating asynchronously, was not the best idea. All kinds of issues we are having are because the peer/substream states are distributed between the components trying to sync this views using asynchronous messages. The implementation already relies on the hacks that should allow imperfect synchronization (like different order of perceived events) to be correctly handled. What is going to follow, is another hack that should sometimes queue events dedicated to individual peers, and sometimes postpone global operations (SetReservedOnly). Moreover, some global operations like SetReservedPeers should be dissected into events dedicated to individual peers, and some of these events should be processes immediately, and some queued on a per-peer basis (for peers awaiting ACK from Notifications). This is going to break the order of processing of events sent by outer API to ProtocolController, and it's hard to imagine the implications of this.
All this can be implemented (and I already started doing it), but I would feel better if we had a vision of how to simplify this in the future (merge ProtocolController with protocols? with Notifications?). Otherwise the ProtocolController code is going to be quite difficult to comprehend and maintain.
Having said all this, I'm not against increasing the complexity of communication, including two back-and-forths, but only if we are sure that it's a temporary measure we can't avoid, and we have a clear picture of how to simplify it in subsequent PRs (and are really determined to simplify).
@altonen

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2814919

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2814920

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2818479

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2818480

Recapping what was discussed in the call with @dmitry-markin
Making the link fully asynchronous was the initial idea and the main idea behind that was to add more decoupling between Notifications and the protocol implementations. That turned out to be difficult to achieve due to many factors that have been discussed at length in this PR.
The new idea is try to and keep the link between Notifications and ProtocolController synchronous and offer some way for the protocol to asynchronously validate the inbound protocol. This would still allow to implement the revised notification system we've been planning but also keep the complexity of the implementation as low as currently possible. This will still allow to split the global Peerset into ProtocolControllers and in the future we can revisit the issue of the asynchronous link.

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2827002

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2827001

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2827102

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2827103

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2827426

The implementation of ACKing in ProtocolController is ready. What is left is to cover postponed events/actions processing with unit tests.
Unfortunately, even with ACKs the fuzz test fails on runs with 250000 iterations (the default value is 2500). Basically, ProtocolController emits a "connect" message for an already connected peer, and I'm not eager to remove the debug assertions from Notifications for such cases. I'm going to look into it a little more in case I overlooked something simple.
CC @altonen

Why are these necessary to implement? I was under the impression that ACKing and postponing events were needed because of the channel half being asynchronous. If it's converted back to synchronous channel, I have trouble understanding why it doesn't work as it did before.

Why are these necessary to implement? I was under the impression that ACKing and postponing events were needed because of the channel half being asynchronous. If it's converted back to synchronous channel, I have trouble understanding why it doesn't work as it did before.

My plan was to use ACKing as a temporary solution that would simplify initial integration of ProtocolController with NotificationService. Once we got it working, we could convert everything to synchronous. May be you are right and this is an unneeded extra step.

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2836809

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2837908

The PR is ready for review. Relaxing requirements on possible messages in Notifications a little more allowed to get rid of ACKing, preserving the eventual consistency of peers' views in ProtocolController and Notifications.

We definitely need a burn-in for this PR.

We definitely need a burn-in for this PR.

Versi burn-in would be nice


Remove unnecessary deletion - update readme - up versions

We fixed an issue where widget wouldn't use the context data which has been introduced in the last release

PR fails on deploying app, which is an infra problem, thus i am merging since one of the builds passed


add Intro to Natual Language processing to paid courses

According to the site, the course is listed at a duration of 12 weeks, but I don't know what the "Effort" amount should be.

@andy5995 thanks for your help! This course actually is in the main curriculum so I'll close this PR.

You're welcome.
I see it's on the main curriculum, but it's one of those that need updating. Instead of a PR, I've documented it here: https://github.com/open-source-society/ossu-client#57
Cheers! :)


image_pipeline: 1.12.20-0 in 'lunar/distribution.yaml' [bloom]

Increasing version of package(s) in repository image_pipeline to 1.12.20-0:

upstream repository: https://github.com/ros-perception/image_pipeline.git
release repository: https://github.com/ros-gbp/image_pipeline-release.git
distro file: lunar/distribution.yaml
bloom version: 0.5.26
previous version for package: 1.12.19-0

camera_calibration
* properly save bytes buffer as such
  This is useful for Python 3 and fixes #256 <https://github.com/ros-perception/image_pipeline/issues/256>.

  OpenCV 3.2 gives slightly different results apparently.
* Use floor division where necessary. (#247 <https://github.com/ros-perception/image_pipeline/issues/247>)
* Fix and Improve Camera Calibration Checker Node (#254 <https://github.com/ros-perception/image_pipeline/issues/254>)
  * Fix according to calibrator.py API
  * Add approximate to cameracheck
* Force first corner off chessboard to be uppler left.
  Fixes #140 <https://github.com/ros-perception/image_pipeline/issues/140>
* fix doc jobs
  This is a proper fix for #233 <https://github.com/ros-perception/image_pipeline/issues/233>
* During stereo calibration check that the number of corners detected in the left and right images are the same. This fixes ros-perception/image_pipeline#225 <https://github.com/ros-perception/image_pipeline/issues/225>
* Contributors: Léonard Gérard, Martin Peris, Vincent Rabaud, hgaiser

depth_image_proc
* Fix CMake warnings about Eigen.
* Convert depth image metric from [m] to [mm]
* address gcc6 build error
  With gcc6, compiling fails with stdlib.h: No such file or directory,
  as including '-isystem /usr/include' breaks with gcc6, cf.,
  https://gcc.gnu.org/bugzilla/show_bug.cgi?id=70129.
  This commit addresses this issue for this package in the same way
  it was addressed in various other ROS packages. A list of related
  commits and pull requests is at:
  https://github.com/ros/rosdistro/issues/12783
  Signed-off-by: Lukas Bulwahn<EMAIL_ADDRESS>* Contributors: Kentaro Wada, Lukas Bulwahn, Vincent Rabaud

image_pipeline
* Update package.xml (#263 <https://github.com/ros-perception/image_pipeline/issues/263>)
* Contributors: Kei Okada

image_proc
* Add nodelet to resize image and camera_info (#273 <https://github.com/ros-perception/image_pipeline/issues/273>)
  * Add nodelet to resize image and camera_info
  * Depends on nodelet_topic_tools
  * Use recursive_mutex for mutex guard for dynamic reconfiguring
* Fix nodelet name: crop_nonZero ->  crop_non_zero (#269 <https://github.com/ros-perception/image_pipeline/issues/269>)
  Fix https://github.com/ros-perception/image_pipeline/issues/217
* Fix permission of executable files unexpectedly (#260 <https://github.com/ros-perception/image_pipeline/issues/260>)
* address gcc6 build error
  With gcc6, compiling fails with stdlib.h: No such file or directory,
  as including '-isystem /usr/include' breaks with gcc6, cf.,
  https://gcc.gnu.org/bugzilla/show_bug.cgi?id=70129.
  This commit addresses this issue for this package in the same way
  it was addressed in various other ROS packages. A list of related
  commits and pull requests is at:
  https://github.com/ros/rosdistro/issues/12783
  Signed-off-by: Lukas Bulwahn<EMAIL_ADDRESS>* Contributors: Kentaro Wada, Lukas Bulwahn

image_publisher
* explicitly cast to std::vector<double> to make gcc6 happy
  With gcc6, compiling image_publisher fails with this error:

/[...]/image_publisher/src/nodelet/image_publisher_nodelet.cpp: In member function 'virtual void image_publisher::ImagePublisherNodelet::onInit()':
/[...]/image_publisher/src/nodelet/image_publisher_nodelet.cpp:180:43: error: ambiguous overload for 'operator=' (operand types are 'sensor_msgs::CameraInfo_<std::allocator >::_D_type {aka std::vector}' and 'boost::assign_detail::generic_list')
camera_info_.D = list_of(0)(0)(0)(0)(0);
After adding an initial explicit type cast for the assignment,
compiling fails further with:

| /[...]/image_publisher/src/nodelet/image_publisher_nodelet.cpp: In member function 'virtual void image_publisher::ImagePublisherNodelet::onInit()':
| /[...]/image_publisher/src/nodelet/image_publisher_nodelet.cpp:180:65: error: call of overloaded 'vector(boost::assign_detail::generic_list&)' is ambiguous
|      camera_info_.D = std::vector (list_of(0)(0)(0)(0)(0));
Various sources on the internet [1, 2, 3] point to use the
convert_to_container method; hence, this commit follows those
suggestions and with that image_publisher compiles with gcc6.
[1] http://stackoverflow.com/questions/16211410/ambiguity-when-using-boostassignlist-of-to-construct-a-stdvector
[2] http://stackoverflow.com/questions/12352692/ambiguous-call-with-list-of-in-vs2010/12362548#12362548 <https://github.com/ambiguous-call-with-list-of-in-vs2010/12362548/issues/12362548>
[3] http://stackoverflow.com/questions/13285272/using-boostassignlist-of?rq=1
Signed-off-by: Lukas Bulwahn<EMAIL_ADDRESS>* address gcc6 build error
With gcc6, compiling fails with stdlib.h: No such file or directory,
as including '-isystem /usr/include' breaks with gcc6, cf.,
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=70129.
This commit addresses this issue for this package in the same way
it was addressed in various other ROS packages. A list of related
commits and pull requests is at:
https://github.com/ros/rosdistro/issues/12783
Signed-off-by: Lukas Bulwahn<EMAIL_ADDRESS>* Contributors: Lukas Bulwahn

image_rotate
* Fix CMake warnings about Eigen.
* address gcc6 build error
  With gcc6, compiling fails with stdlib.h: No such file or directory,
  as including '-isystem /usr/include' breaks with gcc6, cf.,
  https://gcc.gnu.org/bugzilla/show_bug.cgi?id=70129.
  This commit addresses this issue for this package in the same way
  it was addressed in various other ROS packages. A list of related
  commits and pull requests is at:
  https://github.com/ros/rosdistro/issues/12783
  Signed-off-by: Lukas Bulwahn<EMAIL_ADDRESS>* Contributors: Lukas Bulwahn, Vincent Rabaud

image_view
* DisparityViewNodelet: fixed freeze (#244 <https://github.com/ros-perception/image_pipeline/issues/244>)
* launch image view with a predefined window size (#257 <https://github.com/ros-perception/image_pipeline/issues/257>)
* Remove python-opencv run_depend for image_view (#270 <https://github.com/ros-perception/image_pipeline/issues/270>)
  The python-opencv dependency pulls in the system OpenCV v2.4 which is
  not required since the image_view package depends on cv_bridge which
  pulls in opencv3 and opencv3 provides the python library that
  image_view can use.
* Fix encoding error message (#253 <https://github.com/ros-perception/image_pipeline/issues/253>)
  * Fix encoding error message
  * Update image_saver.cpp
  Allow compilation on older compilers
* Including stereo_msgs dep fixes #248 <https://github.com/ros-perception/image_pipeline/issues/248> (#249 <https://github.com/ros-perception/image_pipeline/issues/249>)
* Add no gui mode to just visualize & publish with image_view (#241 <https://github.com/ros-perception/image_pipeline/issues/241>)
* stere_view: fixed empty left, right, disparity windows with opencv3
* Apply value scaling to depth/float image with min/max image value
  If min/max image value is specified we just use it, and if not,
  - 32FC1: we assume depth image with meter metric, and 10[m] as the max range.
  - 16UC1: we assume depth image with milimeter metric, and 10 * 1000[mm] as the max range.
* Depends on cv_bridge 1.11.13 for CvtColorForDisplayOptions
  Close #238 <https://github.com/ros-perception/image_pipeline/issues/238>
* fix doc jobs
  This is a proper fix for #233 <https://github.com/ros-perception/image_pipeline/issues/233>
* address gcc6 build error
  With gcc6, compiling fails with stdlib.h: No such file or directory,
  as including '-isystem /usr/include' breaks with gcc6, cf.,
  https://gcc.gnu.org/bugzilla/show_bug.cgi?id=70129.
  This commit addresses this issue for this package in the same way
  it was addressed in various other ROS packages. A list of related
  commits and pull requests is at:
  https://github.com/ros/rosdistro/issues/12783
  Signed-off-by: Lukas Bulwahn<EMAIL_ADDRESS>* Contributors: Christopher Wecht, Kartik Mohta, Kei Okada, Kentaro Wada, Lukas Bulwahn, Léonard Gérard, Vincent Rabaud, cwecht, mryellow

stereo_image_proc
* fix doc jobs
  This is a proper fix for #233 <https://github.com/ros-perception/image_pipeline/issues/233>
* address gcc6 build error
  With gcc6, compiling fails with stdlib.h: No such file or directory,
  as including '-isystem /usr/include' breaks with gcc6, cf.,
  https://gcc.gnu.org/bugzilla/show_bug.cgi?id=70129.
  This commit addresses this issue for this package in the same way
  it was addressed in various other ROS packages. A list of related
  commits and pull requests is at:
  https://github.com/ros/rosdistro/issues/12783
  Signed-off-by: Lukas Bulwahn<EMAIL_ADDRESS>* Contributors: Lukas Bulwahn, Vincent Rabaud

thanks @vrabaud !


Incompatible with transformers 4.38

Hi,
I came across an incompatibility with pip package transformers >=4.38.
See https://github.com/huggingface/transformers/issues/29763
I could resolve by restricting version of transformers in pyproject.toml file:
sed -i "s/\^4.11.3/~4.37.2/" pyproject.toml
python3 -m pip install .

This will probably be fixed upstream, but one should maybe add exempt transformers>=4.38 from the versions. Maybe something like:
sed 's/transformers = "\^4.11.3/transformers = "^4.11.3, !=4.38/' pyproject.toml

Hi,
Thank you for bringing this to my attention.
I'll think about how to best handle this, as restricting it might affect others that run Transformers 4.38 with a compatible PyTorch version (2.2+ seems to work). I personally haven't tested TMbed with anything more recent than PyTorch 2.1.1 but I would assume that it runs just fine with PyTorch 2.2 and up, too.


fix: patch with fake_generate_content instead of replacing the original

Replacing the original functions led to fails on consecutive tests. This avoids it, patching the function in the context of the test.

Thanks for fixing my mess 🙏🏻 will learn from it


[$500] [Android] Toolbar has excess padding when opening closed app

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: 1.4.42-0
Reproducible in staging?: Yes
Reproducible in production?: Yes
Issue reported by: Me
Action Performed:

Navigate to the Settings > About page
Kill the app my swiping it away
Reopen the app
Splash screen shows...

Expected Result:

The About page is reopened, but the < About Component now has a huge gap above
The status bar area is black

Actual Result:

There should not be a huge gap above the < About Component
The status bar area should match the page background colour

Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android: Native
[ ] Android: mWeb Chrome
[ ] iOS: Native
[ ] iOS: mWeb Safari
[ ] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos

Before
After

https://github.com/Expensify/App/assets/10736861/3e129199-b7f8-4207-bf49-c7a9ed0e42ab

View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~01708ae4fe1601b62d
Upwork Job ID:<PHONE_NUMBER>654196224
Last Price Increase: 2024-02-20

Not overdue, waiting on proposals

I don't think the issue can be reproduced.
I was able to reproduce on version 1.4.42-0:

You can see a view with paddingTop: 24 that seems related to SafeAreaInsets
I was not able to reproduce on version 1.4.43-18:

Looks like this issue was resolved in later versions.
I'm new to Expensify contributing. I guess this means I don't make a proposal, and don't get paid for this diagnostic work?

Contributor details
Your Expensify account email: 181673
Upwork Profile Link: https://www.upwork.com/freelancers/~014cfd325ccf67045b

Contributor details
Your Expensify account email<EMAIL_ADDRESS>Upwork Profile Link: https://www.upwork.com/freelancers/~014cfd325ccf67045b

Yep confirmed this has been fixed elsewhere. Thanks @louiechristie, I'll close the issue.

Cheers Jules, I've removed the Upworks job.


Metrics per-timestamp Distribution Plot

🚀 Feature Request
Create method to plot metrics distribution over the timestamps
Motivation
Tool for model quality analysis
Proposal
In etna.analysis.plotters create method:
def metric_per_timestamp_distribution_plot(
    forecasts_df: pd.DataFrame,
    ts: "TSDataset",
    metric: Metric,
    plot_type: Union[Literal["hist"], Literal["box"], Literal["violin"]] = "hist",
    segments: Optional[List[str]] = None,
    columns_num: int = 2,
    figsize: Tuple[int, int] = (10, 5),
    **seaborn_kwargs
):

Where:
forecast_df - forecasts from backtest
ts - dataset that was used for backtest
metric - metric to plot
plot_type - type of the plot
For each segment

count the metric values on each timestamp
plot the metrics values according to plot_type

Use hist/box/violin methods for each plot_type
Add picture with visualization example to PR
Alternatives
No response
Additional context
No response
Checklist

[ ] I discussed this issue with ETNA Team

This plot seems to be not enough popular


Switch to the new Docker Github action in Meshery-maesh repo

Description
Since last Monday (7th October 2019) the default Docker GitHub action stopped working because the repo was deleted. We have migrated to a different docker GitHub action in the Meshery repo.
We need to replace the entire Docker block in Meshery-maesh repo's https://github.com/layer5io/meshery-maesh/blob/master/.github/workflows/ci.yml file with the one from the Meshery repo (https://github.com/layer5io/meshery/blob/master/.github/workflows/ci.yml).
Here is the snippet from the Meshery repo which we want copied over:
https://github.com/layer5io/meshery/blob/82018c54539f3d79eb639109b7c7cf6f67e41b2b/.github/workflows/ci.yml#L139-L173
Here is the snippet which needs to be replaced:
https://github.com/layer5io/meshery-maesh/blob/aabf9be36ac63f34be500d24c956d8a640db2aea/.github/workflows/ci.yml#L106-L158

@ashiscs I think you are addressing this one. Will you ensure that your PR has a reference to this issue?
@girishranganathan to date, we've been opening adapter-specific issues within their respective repos.

https://github.com/layer5io/meshery-maesh/pull/14
Fixed #362


general Issues on examples with used hardware ( tlk110....)

hi guys
a please to you developers at espressif who build example code for the idf:
can you please refer to the used hardware in your example
with shematic and used hardware, pins and used ESP32 Board?
there are more ESP32 Boards on the market and each has other hardware reference.
this would help in future us to test and build the same environment on hardware
and test your examples like you have wrote and thinked in code
cause this is a general theme it will post the same in esp32 forum
https://esp32.com/viewtopic.php?f=2&t=1259
thank you for your understanding!
best wishes
rudi ;-)

Regarding ethernet exmaples & hardware, there will be a reference design available soon (final revision are being done now). Once the reference design is final, the schematic will be published and the examples will refer to this design.
Regarding the other examples, if there are hardware details that are missing from a particular example please let us know and we will add them to the README.


full_bin_index.7dbe22a9 frequent loading failed

The visit area is outside the United States, such as the Chinese mainland, Vietnam, Malaysia.
full_bin_index.7dbe22a9 file is relatively large, in the loading process, very vulnerable to the impact of the network environment, resulting in loading failed.

Expect SDK to add support and customize proxy transit?
Expect SDK to support customized cache valid period?

Or is there another better solution?

Hey @hurrykee! :wave:  Thanks for the report!
I think we could definitely retry to fetch those resources if they failed to load at a minimum :+1:
In terms of proxying, we use Cloudflare and here it responded with CF-Cache-Status: DYNAMIC which means that it was in its cache. They have data centers in all the locations you mentioned. Looking at the status page it also seems that their datacenter in Vietnam is being re-routed but not for China nor Malaysia at this time.
Could you run:
curl -vv https://w-corp-staticblitz.com/cdn-cgi/trace

and share what you get for colo ? This will tell us which data center you're being connected to. Maybe there's something strange happening there.
Regarding caching, this resource, full_bin_index.7dbe22a9 is cached for a year (as you can see by the Cache-Control header) but it is dependent on the webcontainer version which is updated frequently. We've been talking about improvement on that front.

Thanks for the reply:
At present, Guangdong Province in the Chinese mainland has repeatedly requested the resource, but it is still timed out and cannot successfully obtain it completely.
The caching time of the CDN is 1 year, I have learned.
I'm sorry I didn't make my request clear.
What I expect is whether the SDK can support setting the cache time of a local browser. During this valid period, the network is not requested repeatedly, or the network is requested less. In order to avoid service instability due to network problems.
Even more, I expect that the relevant basic resources can be directly integrated into the local project. Developers regularly update these resources locally according to official updates.
*   Trying <IP_ADDRESS>...
* TCP_NODELAY set
* Connected to w-corp-staticblitz.com (<IP_ADDRESS>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=w-corp-staticblitz.com
*  start date: Apr 13 00:42:33 2024 GMT
*  expire date: Jul 12 00:42:32 2024 GMT
*  subjectAltName: host "w-corp-staticblitz.com" matched cert's "w-corp-staticblitz.com"
*  issuer: C=US; O=Let's Encrypt; CN=E1
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7f8da300dc00)
> GET /cdn-cgi/trace HTTP/2
> Host: w-corp-staticblitz.com
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200
< date: Sat, 20 Apr 2024 10:02:49 GMT
< content-type: text/plain
< access-control-allow-origin: *
< server: cloudflare
< cf-ray: 87744d0cafa7dc67-LHR
< x-frame-options: DENY
< x-content-type-options: nosniff
< expires: Thu, 01 Jan 1970 00:00:01 GMT
< cache-control: no-cache
<
fl=378f21
h=w-corp-staticblitz.com
ip=119.***.***.202
ts=1713607369.71
visit_scheme=https
uag=curl/7.64.1
colo=LHR
sliver=010-tier1
http=http/2
loc=CN
tls=TLSv1.2
sni=plaintext
warp=off
gateway=off
rbi=off
kex=X25519
* Connection #0 to host w-corp-staticblitz.com left intact
* Closing connection 0

Woo it looks like you're being routed to the LHR data center which is based in london! :exploding_head:
That sounds like our main issue here.
Thanks for the info! :smiley:

What I expect is whether the SDK can support setting the cache time of a local browser. During this valid period, the network is not requested repeatedly, or the network is requested less. In order to avoid service instability due to network problems.

The fact that resources are fetched again means that you don't have them in your local cache. All resources have a Cache-Control set to maximize the use of your local cache.
Note that if your cache is used, the resources will still appear in the network tab of the devtools but it will be served from your cache.

I think the issue here is that you can't even get it in the first place?

The disk cache does take effect after the initial request is successful. But it is still very unstable.
Although the webcontainer version is updated frequently, in the actual production environment, we expect more stability.
So, why not package these resources into a project that uses webcontainers? This way, you can make full use of the local network and closer CDN nodes.
Of course, hypothesis we can configuration customized proxy server in the SDK, we can choose a better quality and local conditions of CDN acceleration line. Compared to Cloudflare this speed will be faster, the failure rate will be lower.

Hey @hurrykee! I wanted to let you know that there's been some changes made to our infra. I'm wondering how it's looking for you nowadays. Do you still see similar issues as before or is it a bit better now?


Jest not working on fresh install

Description
npm run test fails on a new project (ejected)
Expected Behavior
I'd expect Jest to be installed and to run when npm run test is called.
Observed Behavior
I receive an error saying that jest/bin/jest.js can't be found. (I anonymized some data)
npm run test

><EMAIL_ADDRESS>test /Users/---/projects/---
> node node_modules/jest/bin/jest.js

module.js:327
    throw err;
    ^

Error: Cannot find module '/Users/---/projects/---/node_modules/jest/bin/jest.js'
    at Function.Module._resolveFilename (module.js:325:15)
    at Function.Module._load (module.js:276:25)
    at Function.Module.runMain (module.js:441:10)
    at startup (node.js:140:18)
    at node.js:1043:3
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>test: `node node_modules/jest/bin/jest.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>test script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR!      /Users/---/.npm/_logs/2018-02-01T14_59_32_916Z-debug.log

Environment
Please run these commands in the project folder and fill in their results:

npm ls react-native-scripts:

<EMAIL_ADDRESS>/Users/---/projects/---
└── (empty)

npm ls react-native:

<EMAIL_ADDRESS>/Users/---/projects/---
└──<EMAIL_ADDRESS>

npm ls expo:

<EMAIL_ADDRESS>/Users/---/projects/---
└── (empty)

node -v: v6.12.3
npm -v: 3.10.10
yarn --version: zsh: command not found: yarn
watchman version: zsh: command not found: watchman

Also specify:

Operating system: macOS High Sierra (10.13.3)
Phone/emulator/simulator & version: Simulator 10.0

Reproducible Demo
N/A (it's a new, ejected, install)

It's because jest is not included among the dev dependencies. I'll submit a PR shortly.

Should this still be open?

Nope, was merged in!


No color temp support with Tradfri RGB via hue hub

Home Assistant release with the issue:

Last working Home Assistant release (if known):
N/A
Operating environment (Hass.io/Docker/Windows/etc.):
Raspbian

Component/platform:
https://www.home-assistant.io/components/hue

Description of problem:
The IKEA Tradfri RGB bulbs don't seem to support to set their color using kelvins. I can only provide color using the color wheel.

Link a IKEA Tradfri RGB bulb to a Hue bridge 2.0
Link the Hue bridge to Home Assistant (using auto discovery is fine)
Click on the bulb in the Home Assistant UI
Notice only the color wheel is available, no temperature bar is visible

Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):

Traceback (if applicable):

Additional information:
I'm not sure if it's a bug or if this kind of bulb doesn't really support colors expressed in temperatures. But even if it doesn't, couldn't Hass convert the kelvin to an RGB value?

This is related to https://github.com/home-assistant/home-assistant/issues/14525.

@pvizeli I can still reproduce this issue with 0.72.1

This issue shouldn't be related to #14887 at all, since in my case the bulb is being controlled by the Hue bridge rather than the Ikea Gateway.
It must be something in the Hue bridge component or in the way the bulb is handled in Home Assistant, since I can control the bulb temperature just fine using the Hue app or API.

Please provide a step by step instruction to reproduce the issue. Also fill in your relevant config section. Eg the automation that is used. Then we can reopen.

I added the steps. There's no relevant configuration, everything can be reproduced by simply using the auto discovery tool.

This was limited in pytradfri:
https://github.com/ggravlingen/pytradfri/pull/162
We base the supported features in the tradfri home assistant light platform om what pytradfri reports.
Please take the discussion with the pytradfri team in that repo. They probably have a good reason to limit this?

I already did open an issue on that repo:
https://github.com/ggravlingen/pytradfri/issues/177
He said that pytradfri only communicates with Ikea gateway, so my problem must be related to some other code.

I'm not using the Tradfri component I guess, I'm using the Hue component. I'll update the issue.

Here's a picture of the RGB bulb interface:

And these are the info from the entities list page, if they can be useful:

This was last changed in the hue light platform in this big light refactor PR:
https://github.com/home-assistant/home-assistant/pull/11288
Maybe @armills knows about the background?

Can you add some logging to determine what type is being reported for this light?
https://github.com/home-assistant/home-assistant/blob/2ff5b4ce9547e1d0b15904ee1eb74906eb8f7608/homeassistant/components/light/hue.py#L281
Also, can you please provide the full model number for this bulb? Not all RGB bulbs support setting color temperature.

Sorry I'm not so used to Python, what's the right way to log it?
Regarding the support, the bulb does support it because I can control its color temperature from the Hue app just fine.

The Hue app says the bulb is a "TRADFRI bulb E27 CWS opal 6... (something)" the full model is cut off 😓

Ok so I have to open a new issue.
Thanks :smile:

This issue is for non Ikea gateway issue

The issue is still there

The issue is still there. As I told on my previous reply

I can confirm the same issue on my end, no color temperature is shown but I can see colors.
I use an Ikea Hub and Philips Hue emulation that is not exposing lights and Tradfri hub is enabled in my config.

Not stale

Why is this issue labeled with waiting-for-reply? Reply from who? About what?

The waiting-for-reply is an older label, let me just remove that. Easy.
Please mind your words. Thanks 👍

It's since October that this bot is constantly trying to close this issue, and nobody from the project members did anything to address this problem.
I'm sure this is not the only issue that has been almost closed (or closed all together) because of this useless and rude bot, what's the purpose of it?

Its purpose is explained by the bot itself, which is not a rude message IMHO. It is just asking if the issue still persists or not.
For more information about stalebot: https://github.com/probot/stale#is-closing-stale-issues-really-a-good-idea

The bot is not the rude one here.

Same problem here: No color temp for the RGB bulb.

Just discovered a workaround to control the color temp of trådfri bulbs: use a light group, the color temp slider is visible there and the bulbs will follow suit.
I'm on deconz/conbee but I understand that this issue happens for all that do not use the trådfri gateway.

Hi @ReekyMarko !
I am using the Ikea Gateway but the problem is the same.
The same identical issue is shown in Alexa (without using HA).

Aah okay so it's even worse. Can you try if you are able to control the temperature using a group, like I am?

@ReekyMarko What I have found is this:
The light identify as 'color temperature lights' with a CT value on startup. If you touch the CT value it will change the color of the light, not update it and start reporting as a XY light.
If you change CT with ct_inc instead it not only works, but restores the CT value, until you change it again. Happens to all three types if IKEA lights that I have.
Groups sounds like a workaround for a while.
https://github.com/dresden-elektronik/deconz-rest-plugin/issues/1861

Just discovered a workaround to control the color temp of trådfri bulbs: use a light group, the color temp slider is visible there and the bulbs will follow suit.
I'm on deconz/conbee but I understand that this issue happens for all that do not use the trådfri gateway.
EDIT: using a service call directly to set a kelvin value on the bulb directly works too, so the only issue here is the actual popup in the web UI. I have no familiarity with the HASS codebase but I suspect that the way it checks for light capabilities is not working. We could just override that on all the affected bulbs (if there is a way to reliably detect them) and show the color temp slider.
It seems that the deconz webui has the same issue. I will have a look on their issue tracker later today if they have made any progress on this, so we can share our findings.

any chance that i can see you config for that @ReekyMarko  thanks for the workaround!

I didn't write about it here, since I thought the fix was going to be in deconz already.
The bug is in  deconz and I know where. I told the devs some weeks ago.
If you compile the plugin yourself you can just comment out one line. I can supply a library that is fixed for 2.0.69 if you have access to your filesystem. Best if you complile yourself as described in the README.
Comment out this line in : zcl_tasks.cpp (line 384)
useXy = useXy || task.lightNode->manufacturerCode() == VENDOR_IKEA;

That is make it look like:
//useXy = useXy || task.lightNode->manufacturerCode() == VENDOR_IKEA;

I didn't write about it here, since I thought the fix was going to be in deconz already.
The bug is in deconz and I know where. I told the devs some weeks ago.
If you compile the plugin yourself you can just comment out one line. I can supply a library that is fixed for 2.0.69 if you have access to your filesystem. Best if you complile yourself as described in the README.
Comment out this line in : zcl_tasks.cpp (line 384)
useXy = useXy || task.lightNode->manufacturerCode() == VENDOR_IKEA;

That is make it look like:
//useXy = useXy || task.lightNode->manufacturerCode() == VENDOR_IKEA;

thx for the reply @Flurkmark i can see that they haven't updated the zcl_tasks.cpp file in the lastest version, my second problem is that i'm using the docker version https://github.com/marthoc/docker-deconz and i have no luck to locate the file, and even if i do, i somehow need to update the dockerfile to get the new version of the file - my docker and linux knowledge might be a bit limited to figure it out :)
nice you found the reason for the problem, ill properly just wait for a new release with the fix :)

@Flurkmark I have created a PR for this change. Hopefully it will be merged.
https://github.com/dresden-elektronik/deconz-rest-plugin/pull/2507

@Flurkmark  I compiled rest-plugin with your solution but Im having trouble figuring out which file I need to replace from my compile folder and my marthoc/deconz docker. Any hint would be much appreciated :)

@Flurkmark I compiled rest-plugin with your solution but Im having trouble figuring out which file I need to replace from my build folder to my marthoc/deconz docker. Any hint would be much appreciated :)
I tried finding zcl_tasks.o in my container but no luck

I don't use doicker, just a .deb install. The file:
/usr/share/deCONZ/plugins/libde_rest_plugin.so needs replacing.

🤬🤬🤬🤬🤬🤬🤬🤬🤬🤬🤬

This is still very much an issue for me. :(


issue-1272: support for disabling logging of a specific HTTP request

https://github.com/softwaremill/sttp/issues/1272

Thanks for implementing this feature.
Unfortunately there are problems with binary compatibility - look at failed mima job - you can try to make it in binary compatibility.
I am not sure if it is possible in reasonable fashion so it will probably have to wait before being merged.

@Kobenko @Pask423 while we cannot modify RequestOptions in a backwards-compatible way (it should have been a trait, but well, too late ;) ), there is the .tag method, which allows attaching arbitrary meta-data to a request. The signature is type-unsafe def tag(k: String, v: Any): RequestT[U, T, R] (again, type-safe attributes as in tapir would be better), but we could make this nicer via an extension methods added e.g. by an import from the logging package or companion object

thanks for the help


EL-1756: Create new field in the completed user journeys table to record EE result

Jira ticket
What changed and why

amend completed user journey table for early_result_type

Guidance to review

n/a

Checklist
Before you ask people to review this PR:

Tests and rubocop should be passing
Branch is generally up to date with main Github - definitely no conflicts
No unnecessary whitespace changes. These make diffs harder to read and conflicts more likely.
PR description says what changed and why, with a link to the JIRA story.
Diff has been checked for unexpected changes being included.
Commit messages say why the change was made.

Closing this as having issues with pods and migrations.


Build errors - lts >= 7

I can build with stack lts <= 6. I will try to paste the build errors here eventually. It might be a minor change to fix?
[ 8 of 27] Compiling Database.HaskellDB.BoundedList ( src/Database/HaskellDB/BoundedList.hs, .stack-work/dist/x86_64-linux/Cabal-<IP_ADDRESS>/build/Database/HaskellDB/BoundedList.o )

/home/mackey/mackeyhackage/haskelldb/src/Database/HaskellDB/BoundedList.hs:104:10: error:
    • Could not deduce (Size a)
        arising from the superclasses of an instance declaration
      from the context: Less a N1
        bound by the instance declaration
        at src/Database/HaskellDB/BoundedList.hs:104:10-31
      Possible fix:
        add (Size a) to the context of the instance declaration
    • In the instance declaration for ‘Less a N2’

I'm glad to accept a pull request but I probably won't be able to fix
myself.
On Tue, Mar 28, 2017, 6:49 PM Kanishka Azimi<EMAIL_ADDRESS>wrote:

I can build with stack lts <= 6. I will try to paste the build errors here
eventually. It might be a minor change to fix?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/m4dc4p/haskelldb/issues/27, or mute the thread
https://github.com/notifications/unsubscribe-auth/AABJAXunuoQlVmWsURirONsPVzvJnX8gks5rqbiygaJpZM4MsZxV
.

@kanishka-azimi HaskellDB has been bitrotting for years.  Is there a reason Opaleye.

I was going to switch to opaleye's Haskelldb soon. I thought I would note
this, to see if we can get it building for others before I switch.
On Mar 29, 2017 2:32 AM, "tomjaguarpaw"<EMAIL_ADDRESS>wrote:

@kanishka-azimi https://github.com/kanishka-azimi HaskellDB has been
bitrotting for years. Is there a reason Opaleye
https://github.com/tomjaguarpaw/haskell-opaleye.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/m4dc4p/haskelldb/issues/27#issuecomment-289996246,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABUlShy3kvtgqFjeHeL9DdfWGJnN_-57ks5rqfrogaJpZM4MsZxV
.

I don't mind what happens with respect to this issue, but leaving this as documentation for future developers who might attempt the same.

I'm not familiar with stack's LTS packages; could you paste the .cabal file used, so I can see all versions that caused the problem? (I assume it's got something to do with a newer GHC and the version of base should tell me that).
Thanks!

https://www.stackage.org/nightly-2017-03-31/cabal.config?global=true
That's the long answer. I don't know the fastest way to get stack to dump
the subset of library versions for a particular build. I know people have
made helpers like Jenga recently.
On Mar 31, 2017 2:15 PM, "m4dc4p"<EMAIL_ADDRESS>wrote:

I'm not familiar with stack's LTS packages; could you paste the .cabal
file used, so I can see all versions that caused the problem? (I assume
it's got something to do with a newer GHC and the version of base should
tell me that).
Thanks!
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/m4dc4p/haskelldb/issues/27#issuecomment-290788160,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABUlSq6dGdNwJw6vCWKkNAJSbqVU1cDpks5rrUK4gaJpZM4MsZxV
.

More detailed information:
$ stack list-dependencies
array <IP_ADDRESS>
base <IP_ADDRESS>
bytestring <IP_ADDRESS>
containers <IP_ADDRESS>
deepseq <IP_ADDRESS>
directory <IP_ADDRESS>
filepath <IP_ADDRESS>
ghc-prim <IP_ADDRESS>
haskelldb 2.2.4
integer-gmp <IP_ADDRESS>
mtl 2.2.1
old-locale <IP_ADDRESS>
old-time <IP_ADDRESS>
pretty <IP_ADDRESS>
time <IP_ADDRESS>
transformers <IP_ADDRESS>
unix <IP_ADDRESS>
```

You can reopen when this is active or if another developer attempts to build this project.


Auto label workflow automation 🚀

Description: Create a GitHub Actions workflow that automatically labels new issues based on their content. This will help categorize and prioritize tasks for better project management.
note it will only add two labels hacktoberfest and gssoc ext
please @apu52  assign this with level 3 label

@amiya-cyber Please try to follow the issue templates which are already provided in the repo to raise your issue , otherwise it will be difficult for me/mentors to assign u the issue without much info.
This issue is good just try to raise a new issue on same topic using the issue templates already given
Sorry to close this issue for now .


Compile Error

Describe the bug
I am getting compile error with the latest clone:
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/Users/anaconda3/lib/python3.11/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
^^^^^^^^^^^^^^^^^^^
File "/Users/anaconda3/lib/python3.11/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
^^^^^^^^^^^^^^^^
File "/Users/anaconda3/lib/python3.11/site-packages/Cython/Build/Dependencies.py", line 1325, in cythonize_one_helper
return cythonize_one(*m)
^^^^^^^^^^^^^^^^^
File "/Users/anaconda3/lib/python3.11/site-packages/Cython/Build/Dependencies.py", line 1301, in cythonize_one
raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: hummingbot/connector/exchange/bitfinex/bitfinex_exchange.pyx
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/Documents/hummingbot/setup.py", line 161, in
main()
File "/Users/Documents/hummingbot/setup.py", line 148, in main
ext_modules=cythonize(cython_sources, compiler_directives=compiler_directives, **cython_kwargs),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/anaconda3/lib/python3.11/site-packages/Cython/Build/Dependencies.py", line 1125, in cythonize
result.get(99999)  # seconds
^^^^^^^^^^^^^^^^^
File "/Users/anaconda3/lib/python3.11/multiprocessing/pool.py", line 774, in get
raise self._value
Cython.Compiler.Errors.CompileError: hummingbot/connector/exchange/bitfinex/bitfinex_exchange.pyx
Steps to reproduce
./install
then
./compile
This is being compile on Mac M2
Cython version 3.0.2
Release version
clone of the latest version
Type of installation
Source
Attach required files
No response

Another round of compilation and the error was more clear:
Error compiling Cython file:
...
async def _api_do_request(self,
                          http_method: str,
                          url,
                          headers,
                          data_str: Optional[str, list] = None) -> list:
                                            ^

hummingbot/connector/exchange/bitfinex/bitfinex_exchange.pyx:478:48: 'object' takes exactly one template argument.
I fixed it by aligning the data type to what the caller method were using:
    async def _api_do_request(self,
                              http_method: str,
                              url,
                              headers,
                              data_str: Optional[Dict[str, Any]] = None) -> list:

Hi @rgb1380 good day
Thanks for the update, unfortunately I have no M2 mbp to reproduce this issue atm. Would it be possible if you submit a PR to fix? 🙇🏼


mosquitto: 1.4.5 -> 1.5

Motivation for this change
Things done

[X] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[X] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[X] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[X] Fits CONTRIBUTING.md.

@GrahamcOfBorg  build mosquitto

Success on x86_64-linux (full log)
Attempted: mosquitto
Partial log (click to expand)
shrinking /nix/store/8qlfd5fld1sjmmlfp5d6kf9qmsm1jswd-mosquitto-1.5/lib/libmosquitto.so.1
shrinking /nix/store/8qlfd5fld1sjmmlfp5d6kf9qmsm1jswd-mosquitto-1.5/lib/libmosquittopp.so.1
shrinking /nix/store/8qlfd5fld1sjmmlfp5d6kf9qmsm1jswd-mosquitto-1.5/sbin/mosquitto
gzipping man pages under /nix/store/8qlfd5fld1sjmmlfp5d6kf9qmsm1jswd-mosquitto-1.5/share/man/
strip is /nix/store/4qvrxzxa535y8304mk195x50b6p9607d-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/8qlfd5fld1sjmmlfp5d6kf9qmsm1jswd-mosquitto-1.5/lib  /nix/store/8qlfd5fld1sjmmlfp5d6kf9qmsm1jswd-mosquitto-1.5/bin  /nix/store/8qlfd5fld1sjmmlfp5d6kf9qmsm1jswd-mosquitto-1.5/sbin
patching script interpreter paths in /nix/store/8qlfd5fld1sjmmlfp5d6kf9qmsm1jswd-mosquitto-1.5
checking for references to /build in /nix/store/8qlfd5fld1sjmmlfp5d6kf9qmsm1jswd-mosquitto-1.5...
moving /nix/store/8qlfd5fld1sjmmlfp5d6kf9qmsm1jswd-mosquitto-1.5/sbin/* to /nix/store/8qlfd5fld1sjmmlfp5d6kf9qmsm1jswd-mosquitto-1.5/bin
/nix/store/8qlfd5fld1sjmmlfp5d6kf9qmsm1jswd-mosquitto-1.5

Failure on x86_64-darwin (full log)
Attempted: mosquitto
Partial log (click to expand)
           ^
/tmp/nix-build-mosquitto-1.5.drv-0/mosquitto-1.5/client/sub_client_output.c:52:19: error: use of undeclared identifier 'CLOCK_REALTIME'
        if(clock_gettime(CLOCK_REALTIME, &ts) != 0){
                         ^
1 warning and 1 error generated.
make[2]: *** [client/CMakeFiles/mosquitto_sub.dir/build.make:76: client/CMakeFiles/mosquitto_sub.dir/sub_client_output.c.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:232: client/CMakeFiles/mosquitto_sub.dir/all] Error 2
make: *** [Makefile:130: all] Error 2
builder for '/nix/store/vn8i3pnf6gz8rcxvqs9vvarkg4af7ab5-mosquitto-1.5.drv' failed with exit code 2
error: build of '/nix/store/vn8i3pnf6gz8rcxvqs9vvarkg4af7ab5-mosquitto-1.5.drv' failed

Success on aarch64-linux (full log)
Attempted: mosquitto
Partial log (click to expand)
shrinking /nix/store/qnsnw7vkz0dhj2yypa4bp3xwx7sgnic0-mosquitto-1.5/bin/mosquitto_pub
shrinking /nix/store/qnsnw7vkz0dhj2yypa4bp3xwx7sgnic0-mosquitto-1.5/lib/libmosquittopp.so.1
shrinking /nix/store/qnsnw7vkz0dhj2yypa4bp3xwx7sgnic0-mosquitto-1.5/lib/libmosquitto.so.1
gzipping man pages under /nix/store/qnsnw7vkz0dhj2yypa4bp3xwx7sgnic0-mosquitto-1.5/share/man/
strip is /nix/store/0pjsgkxz0rp5baycq5sp2s72lrr5q9sg-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/qnsnw7vkz0dhj2yypa4bp3xwx7sgnic0-mosquitto-1.5/lib  /nix/store/qnsnw7vkz0dhj2yypa4bp3xwx7sgnic0-mosquitto-1.5/bin  /nix/store/qnsnw7vkz0dhj2yypa4bp3xwx7sgnic0-mosquitto-1.5/sbin
patching script interpreter paths in /nix/store/qnsnw7vkz0dhj2yypa4bp3xwx7sgnic0-mosquitto-1.5
checking for references to /build in /nix/store/qnsnw7vkz0dhj2yypa4bp3xwx7sgnic0-mosquitto-1.5...
moving /nix/store/qnsnw7vkz0dhj2yypa4bp3xwx7sgnic0-mosquitto-1.5/sbin/* to /nix/store/qnsnw7vkz0dhj2yypa4bp3xwx7sgnic0-mosquitto-1.5/bin
/nix/store/qnsnw7vkz0dhj2yypa4bp3xwx7sgnic0-mosquitto-1.5

Darwin failure is new. Current version (1.4.15, btw, there's a typo in your commit message) builds on darwin.

closing as duplicate of #40084, see discussion there.


Added support for oci oss in cloud storage sdk

Upgraded guava version to 31.1-jre
Upgrade jclouds-all version to 2.5.0 which supports aws s3 v4 sigining

@ddevadat can you please provide the tested

java version
mvn version
scala version

Tested on MacOS  with the below
java --version
openjdk 11.0.17 2022-10-18
OpenJDK Runtime Environment Homebrew (build 11.0.17+0)
OpenJDK 64-Bit Server VM Homebrew (build 11.0.17+0, mixed mode)

mvn --version
Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
Maven home: /Users/ddevadat/Oracle_Mac/Deepak/training/MicroServices/apache-maven-3.8.4
Java version: 11.0.17, vendor: Homebrew, runtime<EMAIL_ADDRESS>Default locale: en_AU, platform encoding: UTF-8
OS name: "mac os x", version: "12.6", arch: "x86_64", family: "mac"

scala version : 2.12.11 ( from pom.xml )

Tested on ubuntu
 java --version
openjdk 11 2018-09-25
OpenJDK Runtime Environment 18.9 (build 11+28)
OpenJDK 64-Bit Server VM 18.9 (build 11+28, mixed mode)

mvn --version
Apache Maven 3.6.3
Maven home: /usr/share/maven
Java version: 11, vendor: Oracle Corporation, runtime: /home/ubuntu/jdk-11
Default locale: en, platform encoding: UTF-8
OS name: "linux", version: "5.15.0-1016-oracle", arch: "amd64", family: "unix"