CFP: Bandwidth Manager with fq_codel?

Hi!
(First of all, a disclaimer: I am not trying to start an endless debate like https://github.com/systemd/systemd/issues/9725 or a flamewar 🙂 I am mostly interested in understanding the technical reasons why the Cilium Bandwidth Manager is using fq under the hood and if it would make sense to give users an option to use fq_codel instead)
In the Bandwidth Manager code, I can see that it sets net.core.default_qdisc to fq and then forces the fq qdisc on all relevant devices.
Based on the docs, it looks like the choice of fq was originally made in order to have an option to use BBR? I don't think that in 2023 it's a requirement anymore: according to this thread, fq was required for BBR because it supports TCP pacing but it seems that fq_codel can also work now as well:

Yes, it is fine to use BBR with fq_codel on recent kernels.
For kernels v4.20 and later, BBR will use the Linux TCP-layer pacing if the connection notices that there is no qdisc on the sending host implementing pacing.

I was also curious about the performance impact of fq vs fq_codel when Cilium Bandwidth Manager rate-limiting was in place. In order to test this, I ran a set of benchmarks. My benchmark setup was as follows:

I ran the standard 60 second "Realtime Response Under Load" test from the Flent test suite 1.3.2. The test creates 4 TCP flows doing downloads and 4 TCP flows doing uploads and tries to saturate the links. Latency is measured with ICMP and UDP pings.
I did 4 benchmarks: fq vs fq_codel with Cilium Bandwidth Manager egress limit enforced vs unenforced. When the egress limit was set, it was limiting the Flent client bandwidth to 100Mbit/s with kubernetes.io/egress-bandwidth: "100M" (You can see its effect on the "Upload" graphs below). In order to try out the Bandwidth Manager + fq_codel scenarios, I manually replaced the qdiscs on all devices with tc qdisc del / tc qdisc add.
I benchmarked 2 instances in AWS and 2 in GCP. In each Cloud Provider I chose one "small" instance with 2 vCPU and 2 NIC driver queues and one bigger instance with at least 8 vCPU and exactly 8 NIC driver queues.
I ensured that client & server instances are in the same AZ and that the Flent clients and servers are the only workload pods running on these instances.
The TCP Congestion Control algorithm was CUBIC in all tests

More details about the instance types which were used

Cloud Provider
Instance Type
vCPU
Egress bandwidth
NIC queues

AWS
m5.8xlarge
32
10 Gbps
8

GCP
n2-standard-8
8
16 Gbps
8

AWS
m5.large
2
10 Gbps [up to]
2

GCP
n2-standard-2
2
10 Gbps
2

The results are generally consistent across Cloud Providers and instance types:

When the Cilium Bandwidth Manager rate-limiting is off, both fq and fq_codel are in the same latency ballpark and can easily attain the instances' max throughput and maintain it consistently (beware that you need to multiply all throughput numbers by 4 on all graphs because they show throughput by flow)
When the Cilium Bandwidth Manager rate-limiting is on, fq_codel is able to maintain low latency (< 2 ms) and is able to keep the same max download throughput as with Cilium Bandwidth Manager rate-limiting turned off. When it comes to fq, the situation seems to be much worse: the latency skyrockets (>400ms in worst cases, so ~200x times worse that fq_codel) and the download throughput suffers significantly despite only the egress bandwidth being limited. The only advantage of fq is that the numbers are much more uniform compared to fq_codel which is much more jittery.

Here are all the benchmark results 👇

So the main question is whether it would make sense to give users the option to run the Bandwidth Manager with fq_codel?
I am also curious to hear your thoughts about the benchmarks: is there anything I'm missing here? Is there any other metric to look at?

Thank you for doing these benchmarks! I was very concerned when I heard of this new design that they had not taken a hard look at in-flow latency.

Several suggestions, if you have time:

Try cake with it's integral shaper.
Try enabling ecn on both sides of the container, and repeat the series. This will make fq_codel and cake less jittery. sudo sysctl -w net.ipv4.tcp_ecn=1
(you can do this on a route specific basis also)

The designers of this cilium subsystem were claiming that it was bbr that was doing the magic, but it looks to my eye that it is merely excessive latency. I am under the impression however that at least kernel 6.1 was needed for cilium + fq to work "correctly"?
One fq_codel/cake trick few DC users apply is you can for short paths actually tune fq_codel's target down quite a bit, to achieve the same bandwidth at much lower within stream latency. I regularly run fq_codel on bare metal with a target of 250us and interval of 5ms in the one in-a-dc-only app I have (the cross dc latency is about 4ms tops). I have often wondered how low you could go across containers with ecn enabled.
Lastly, as much as I love the rrul test, most container workloads are pretty unidirectional, and I might start by taking packet captures of your application in deployment and trying to model that, otherwise my next test in your test series would be more like --step-size=.05 -x --test-parameter=upload_streams=4 --socket-stats tcp_nup
to just look at uploads through the shaper

https://blog.cerowrt.org/post/juniper/ has a simple script to sweep from 1-64 flows in it.

Hmm, so I have a hunch about what's the cause of this behaviour, but I don't have time to dig into the details right now. But just to see if I'm completely off base, could you try running a UDP-based bandwidth test as well? Just using iperf2 with iperf -u -s on one end and iperf -u -c HOSTNAME -b 200M on the other end and seeing what actual bandwidth it reports at the end of the test should be sufficient (the traffic goes from the client to the server, so the client should be inside the rate-limited container). Don't worry about measuring latency for that test, I just want to see how effective the bandwidth enforcement is for UDP traffic with each qdisc.

Thank you both for your comments! I took some time to look into them.
First of all, regarding the "UDP-based bandwidth test", I tried it out on a pair of n2-standard-8 nodes in GCP. I left the default fq setup and rate-limited the client to 100Mbps and this is the result I got:
# iperf --udp --client <IP_ADDRESS> --bandwidth 200M --enhancedreports
------------------------------------------------------------
Client connecting to <IP_ADDRESS>, UDP port 5001 with pid 101
Sending 1470 byte datagrams, IPG target: 56.08 us (kalman adjust)
UDP buffer size: 16.0 MByte (default)
------------------------------------------------------------
[  3] local <IP_ADDRESS> port 48567 connected with <IP_ADDRESS> port 5001
[ ID] Interval            Transfer     Bandwidth      Write/Err  PPS
[  3] 0.0000-10.0001 sec   250 MBytes   210 Mbits/sec  178330/0    17832 pps
[  3] Sent 178330 datagrams
[  3] Server Report:
[  3]  0.0-12.0 sec  3.81 MBytes  2.66 Mbits/sec   0.077 ms 175618/178334 (98%)
[  3] 0.0000-11.9995 sec  1 datagrams received out-of-order

Interestingly enough there was 98% packet loss (this is the 175618/178334 (98%) number reported by the server)
Once I removed the kubernetes.io/egress-bandwidth annotation to disable rate-limiting, the packet loss was gone:
# iperf --udp --client <IP_ADDRESS> --bandwidth 200M --enhancedreports
------------------------------------------------------------
Client connecting to <IP_ADDRESS>, UDP port 5001 with pid 104
Sending 1470 byte datagrams, IPG target: 56.08 us (kalman adjust)
UDP buffer size: 16.0 MByte (default)
------------------------------------------------------------
[  3] local <IP_ADDRESS> port 48935 connected with <IP_ADDRESS> port 5001
[ ID] Interval            Transfer     Bandwidth      Write/Err  PPS
[  3] 0.0000-10.0001 sec   250 MBytes   210 Mbits/sec  178329/0    17832 pps
[  3] Sent 178329 datagrams
[  3] Server Report:
[  3]  0.0-10.0 sec   250 MBytes   210 Mbits/sec   0.018 ms    0/178329 (0%)
[  3] 0.0000-9.9989 sec  6 datagrams received out-of-order

Regarding

Try cake with it's integral shaper in "besteffort" mode

To give a bit more context, we are mostly interested in fq_codel specifically. The reason is that we are currently running Cilium + fq_codel on 10000s of nodes and we want to enable the Bandwidth Manager on all these nodes but at the same time want to minimize the number of changes to our infrastructure.
This is why the forced switch to fq when the Bandwidth Manager is enabled is problematic because it increases the delta from our current configuration. So using cake does sound like an interesting avenue to explore but ideally we would switch things one at a time: first enable the Bandwidth Manager and then potentially look into changing the qdiscs on all of our nodes.

Try enabling ecn and tcp cubic on both sides of the container, and repeat the series

I tried this out by setting
  securityContext:
    sysctls:
      - name: net.ipv4.tcp_ecn
        value: "1"

on both pods and running sysctl -w net.ipv4.tcp_ecn=1 on the hosts (it used to be set to 2 in my previous tests). CUBIC was already set-up everywhere. Unfortunately I didn't observe any noticeable difference, there is still a fair amount of jitter:
Here are the graphs
AWS m5.8xlarge benchmark with ECN on:

GCP n2-standard-8 benchmark with ECN on:

I might start by taking packet captures of your application in deployment and trying to model that

Yes, this is definitely on my to-do list, we have 100s of applications running in our infra but I do have some in mind which are quite sensitive to latency and throughput, and which would be good candidates for more benchmarking. It's a bit of a hassle to benchmark them though because I will need to sync with application teams first, this is why I wanted to get as many insights as possible from synthetic benchmarks and only then move to real applications 😄

otherwise my next test in your test series would be more like --step-size=.05 -x --test-parameter=upload_streams=4 --socket-stats tcp_nup
to just look at uploads through the shaper. --socket-stats on the upload portion of many tests will give you plots of the underlying tcp rtts.

Added this to my to-do list as well, will try to get to it this week.

On 15 November 2023 18:22:58 CET, Anton Ippolitov @.***> wrote:

Thank you both for your comments! I took some time to look into them.
First of all, regarding the "UDP-based bandwidth test", I tried it out on a pair of n2-standard-8 nodes in GCP. I left the default fq setup and rate-limited the client to 100Mbps and this is the result I got:
# iperf --udp --client <IP_ADDRESS> --bandwidth 200M --enhancedreports
------------------------------------------------------------
Client connecting to <IP_ADDRESS>, UDP port 5001 with pid 101
Sending 1470 byte datagrams, IPG target: 56.08 us (kalman adjust)
UDP buffer size: 16.0 MByte (default)
------------------------------------------------------------
[  3] local <IP_ADDRESS> port 48567 connected with <IP_ADDRESS> port 5001
[ ID] Interval            Transfer     Bandwidth      Write/Err  PPS
[  3] 0.0000-10.0001 sec   250 MBytes   210 Mbits/sec  178330/0    17832 pps
[  3] Sent 178330 datagrams
[  3] Server Report:
[  3]  0.0-12.0 sec  3.81 MBytes  2.66 Mbits/sec   0.077 ms 175618/178334 (98%)
[  3] 0.0000-11.9995 sec  1 datagrams received out-of-order

Interestingly enough there was 98% packet loss (this is the 175618/178334 (98%) number reported by the server)
Once I removed the kubernetes.io/egress-bandwidth annotation to disable rate-limiting, the packet loss was gone:
# iperf --udp --client <IP_ADDRESS> --bandwidth 200M --enhancedreports
------------------------------------------------------------
Client connecting to <IP_ADDRESS>, UDP port 5001 with pid 104
Sending 1470 byte datagrams, IPG target: 56.08 us (kalman adjust)
UDP buffer size: 16.0 MByte (default)
------------------------------------------------------------
[  3] local <IP_ADDRESS> port 48935 connected with <IP_ADDRESS> port 5001
[ ID] Interval            Transfer     Bandwidth      Write/Err  PPS
[  3] 0.0000-10.0001 sec   250 MBytes   210 Mbits/sec  178329/0    17832 pps
[  3] Sent 178329 datagrams
[  3] Server Report:
[  3]  0.0-10.0 sec   250 MBytes   210 Mbits/sec   0.018 ms    0/178329 (0%)
[  3] 0.0000-9.9989 sec  6 datagrams received out-of-order

Okay, that's very interesting. What about with egress-bandwidth set, but fq_codel as the qdisc?

I am pleased you are running fq_codel in the first place and would love a dump from prod of, say 100 "boxes", of
tc -s qdisc show
so as to see packets, drops, marks, backlogs, and reschedules. reschedules = FQ is working (but how often compared to the total packets?). Backlogs implies queues are forming within fq_codel, as well as lower layers. marks and drops are of the aqm actually needing to "do stuff", also relative to the number of packets.
In the non-rate limited k8 case I have generally assumed it was mq doing a goodly percentage of the FQ in the first place.
I usually post process this stuff with awk. I am obsolete.
If you are a JSON wizard tc -s -j qdisc show and go to town.

quantum 1474? Not 1514? why?

Lastly, I think we have a differing interpretation of "jitter" on the rrul and I would love you to describe how you were thinking about it before I launch into my std lecture so I can maybe lecture sanely to others in the future? I note that I used the word wrong myself in my initial comment.
I am happy you are delving into this. It seemed 99.9999% of the k8 crowd thinks tcp is a function call. I was once able to save a cloudy user 80% of their bandwidth bill by cutting tcp_notsent_lowat down to something reasonable.... there are other things worth tuning, like initcwnd...
anyway in this topology:
(internet) <- web proxy <- local containers the local container interfaces can be configged down to target 250us interval 2.5ms on bare metal, esp with ECN enabled.
As for the "jitter" thing, you can possibly see the difference I was expecting via a cdf plot comparing the before/after ECN, on the tcp_nup tcp rtt. I also note the raw JSON files in flent contain very little sensitive info (IP addresses, qdiscs) and toke and I really good at flipping through them. The -x option however gathers more than most corps would like.

Hello, sorry for the delay due to conference travel.. digesting this thread a bit:
[...]

In the Bandwidth Manager code, I can see that it sets net.core.default_qdisc to fq and then forces the fq qdisc on all relevant devices.

Yep.

Based on the docs, it looks like the choice of fq was originally made in order to have an option to use BBR? I don't think that in 2023 it's a requirement anymore: according to this thread, fq was required for BBR because it supports TCP pacing but it seems that fq_codel can also work now as well:

The initial motivation for fq was to do EDT for the Cilium Bandwidth manager (see also https://netdevconf.info/0x14/session.html?talk-replacing-HTB-with-EDT-and-BPF). As far as I know fq_codel does not support EDT. Compare https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/sched/sch_fq.c#n530 vs https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/sched/sch_fq_codel.c#n184 . See also<EMAIL_ADDRESS>. The EDT we make use of for the kubernetes.io/egress-bandwidth annotation.
The BBR for Pods etc needs v5.18+ kernel, otherwise pacing is broken given the skb->tstamp is cleared upon netns traversal and the rates are fluctuating. The relevant fixes which are part from these kernel onwards are<EMAIL_ADDRESS>.

Yes, it is fine to use BBR with fq_codel on recent kernels.
For kernels v4.20 and later, BBR will use the Linux TCP-layer pacing if the connection notices that there is no qdisc on the sending host implementing pacing.

I was also curious about the performance impact of fq vs fq_codel when Cilium Bandwidth Manager rate-limiting was in place. In order to test this, I ran a set of benchmarks. My benchmark setup was as follows:

Again, fq_codel does not support EDT, hence the fq_codel measurement is "broken" here given skb->tstamps that were set by the Cilium Bandwidth Manager's BPF code are ignored.

I ran the standard 60 second "Realtime Response Under Load" test from the Flent test suite 1.3.2. The test creates 4 TCP flows doing downloads and 4 TCP flows doing uploads and tries to saturate the links. Latency is measured with ICMP and UDP pings.
I did 4 benchmarks: fq vs fq_codel with Cilium Bandwidth Manager egress limit enforced vs unenforced. When the egress limit was set, it was limiting the Flent client bandwidth to 100Mbit/s with kubernetes.io/egress-bandwidth: "100M" (You can see its effect on the "Upload" graphs below). In order to try out the Bandwidth Manager + fq_codel scenarios, I manually replaced the qdiscs on all devices with tc qdisc del / tc qdisc add.
I benchmarked 2 instances in AWS and 2 in GCP. In each Cloud Provider I chose one "small" instance with 2 vCPU and 2 NIC driver queues and one bigger instance with at least 8 vCPU and exactly 8 NIC driver queues.
I ensured that client & server instances are in the same AZ and that the Flent clients and servers are the only workload pods running on these instances.
The TCP Congestion Control algorithm was CUBIC in all tests

Btw, semi-related, did you also try to run a 6.7-rc2 kernel? These ones got merged there:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b49a948568dcbb5f38cbf5356ea0fb9c9c6f6953

Thank you @borkmann !
I looked into your main point

fq_codel does not support EDT, hence the fq_codel measurement is "broken"

and I believe you are right, my fq_codel benchmark results seem to have been flawed: the egress rate-limiting that I saw in these benchmarks seems to have been achieved as a side-effect of horizon drops here, meaning that so many packets were dropped that the bandwidth was effectively capped at ~100Mbps.
I removed these two lines, then re-ran a benchmark with fq_codel and the bandwidth was essentially uncapped:

So it looks like using fq_codel with the Bandwidth Manager is not an option after all (unless we somehow add EDT support there)

did you also try to run a 6.7-rc2 kernel?

I haven't had the chance to try it out yet, we are still mainly running 5.15 everywhere. I will give it a try.
I am still curious to understand the performance issues I saw with fq when the Bandwidth Manager is enabled since we are not going to migrate to 6.7 any time soon. What would explain the latency + download throughput degradations?
Also thank you @dtaht for the suggestions.

I am pleased you are running fq_codel in the first place and would love a dump from prod of, say 100 "boxes", of
tc -s qdisc show so as to see packets, drops, marks, backlogs, and reschedules

It would indeed be interesting to look into better tuning fq_codel on our instances (and I'm sure we have room to improve) but it's slightly unrelated to the work I'm doing right now, I will try to come back to this later on.

quantum 1474? Not 1514? why?

This is a default value set by Google Cloud, probably their MTU of 1460 bytes + hardware header length of 14 bytes = 1474.

Lastly, I think we have a differing interpretation of "jitter" on the rrul and I would love you to describe how you were thinking about it

Yeah, I am not sure that "jitter" is the exact right term, I was trying to refer to the sharp oscillations looking like this:

As compared to smoother curves like this one:

I removed these two lines, then re-ran a benchmark with fq_codel and the bandwidth was essentially uncapped:

Ah! This is the behaviour I was expecting, and couldn't for the life of me figuring out why this wasn't the case; totally missed the horizon drop thing in the BPF code, that explains it :)

I am still curious to understand the performance issues I saw with fq when the Bandwidth Manager is enabled since we are not going to migrate to 6.7 any time soon. What would explain the latency + download throughput degradations?

I wouldn't expect 6.7 to help with the latency issues you were seeing. This is caused by the way the bandwidth manager is implemented, AFAICT; basically it creates a virtual FIFO queue without implementing any kind of AQM or flow queueing, so the terrible latency is totally expected.
Specifically, the shaper logic here (right above the horizon drop thing you linked above), does a lookup into the rate config map using the previously set queue mapping as the key[0], finds the rate and the last timestamp, and sets a timestamp for the packet based on the rate and the length of the packet. So packets will be delayed in the fq qdisc until their virtual TX time, and since there's only a single running timestamp for each queue mapping ID, this in effect becomes a virtual FIFO, with the only limit being the horizon timestamp, as you found.
From a queueing algorithm behaviour PoV this is obviously not great. And doing better at this level is not so straight forward either: since the dequeue time is computed before each packet is queued, tail dropping is the only action possible to control the queue. I've seen a CoDel implementation that works in this mode at one point, so that would be possible, I guess; but flow queueing is not, really, at least not without temporarily going above the bandwidth limit after the fact. Also just straight round-robin scheduling between becomes quite challenging to do in this "virtual queue of future transmission times" mode.
So based on the above, my recommendation would simply be "don't use the bandwidth manager unless you don't care about latency at all". I'm frankly a little puzzled that no one noticed this behaviour before; I guess it's the ever-present curse of "only benchmark TCP throughput"? IDK.
Or who knows, maybe I'm missing something fundamental here, and there's some reason why things are not as dire as I paint them above? If so, I would love to know what that reason is! :)
[0] Not quite sure how that is initially set, but seems to be coming from inside the pod? so veth queue id, I guess? anyway, doesn't look like there is more than at most a few IDs per pod.

I note that I do this sort of tuning for a living, and am between contracts at the moment.
Anyway, thank you all for getting into the methods and claims behind how this cilium subsystem works without me having to poke into it much. I would like to know how much the rtt grows before shuddering overmuch.
A) Packet captures would be nice. Seeing the rtt in both slow start and congestion avoidance.
B) BBR + EDT + short horizon is, I think, a means to do bandwidth shaping multicore?
In general the rrul test is a partial emulation of a BitTorrent workload, and has been a good proxy for creating "good" residential and small corporate network behaviors in the general case. Regrettably it has next to no bearing on the actual behaviors of a container workload, IMHO.
Measure those.
Most of these I have seen (with the exception of movie streaming) have been totally dominated by slow start, where tuning initcwnd and tcp_notsent_lowwat matter most, and on the places I have been called in, leveraging fq_codel + ECN + cubic to manage it, further, without packet drop. I never bothered to try and rate shape anything before now, the problems being always more dominated by the behaviors of the flows from the web proxy outwards.

The "jitter" you observe on the rrul test is the conflation of two things: sampling error and actual bandwidth usage during the sample interval defined. It is normal for it to occilate somewhat because that is how the tcp sawtooth works as a function of the rtt, to go deep see https://ee.lbl.gov/papers/congavoid.pdf and really deep see https://en.wikipedia.org/wiki/Lyapunov_stability
It is necessary for it to bounce around a bit in order for the internet to not collapse. rfc970. The height is related to the rtt, but flent sampling error so huge on this sub-1us RTT that you are not getting a real picture of the carnage underneath at all.
Your second plot shows a classic example of slow convergence, where the first flows to start hog all the processor and bandwidth and the later flows - due in part to the massively inflating rtt, takes a longer time to achieve equality.
In both these cases, honestly, the underlying behavior of the container's real workload looks nothing like this and the only way I can ever convince someone of this is for them to take 5 minutes or so of packet capture and tear it, rather than iperf/netperf test traffic, apart.

B) BBR + EDT + short horizon is, I think, a means to do bandwidth shaping multicore?

Well, no, not really, at least not in itself, for the reason I outlined above...

I would like a packet capture. And a glass of scotch.

I am still curious to understand the performance issues I saw with fq when the Bandwidth Manager is enabled since we are not going to migrate to 6.7 any time soon. What would explain the latency + download throughput degradations?

I wouldn't expect 6.7 to help with the latency issues you were seeing. This is caused by the way the bandwidth manager is implemented, AFAICT; basically it creates a virtual FIFO queue without implementing any kind of AQM or flow queueing, so the terrible latency is totally expected.
Specifically, the shaper logic here (right above the horizon drop thing you linked above), does a lookup into the rate config map using the previously set queue mapping as the key[0], finds the rate and the last timestamp, and sets a timestamp for the packet based on the rate and the length of the packet. So packets will be delayed in the fq qdisc until their virtual TX time, and since there's only a single running timestamp for each queue mapping ID, this in effect becomes a virtual FIFO, with the only limit being the horizon timestamp, as you found.

The idea was to transfer https://netdevconf.info//0x14/pub/slides/55/slides.pdf / https://netdevconf.info//0x14/pub/papers/55/0x14-paper55-talk-paper.pdf for K8s and utilise the Pod egress rate annotations there.
[...]

Or who knows, maybe I'm missing something fundamental here, and there's some reason why things are not as dire as I paint them above? If so, I would love to know what that reason is! :)

Hm, I wonder also if we're hitting other things such as sch->limit with the defaults, I've heard about issues like these due to defaults being too low.
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/sched/sch_fq.c#n530

[0] Not quite sure how that is initially set, but seems to be coming from inside the pod? so veth queue id, I guess? anyway, doesn't look like there is more than at most a few IDs per pod.

Given all other skb fields such as mark etc are already used up, it basically stores the endpoint ID into queue_mapping which is preserved all the way through the stack and later resets it, so kernel picks queue via flow hash. Not great, but seems to function at least.

I am still curious to understand the performance issues I saw with fq when the Bandwidth Manager is enabled since we are not going to migrate to 6.7 any time soon. What would explain the latency + download throughput degradations?

I wouldn't expect 6.7 to help with the latency issues you were seeing. This is caused by the way the bandwidth manager is implemented, AFAICT; basically it creates a virtual FIFO queue without implementing any kind of AQM or flow queueing, so the terrible latency is totally expected.
Specifically, the shaper logic here (right above the horizon drop thing you linked above), does a lookup into the rate config map using the previously set queue mapping as the key[0], finds the rate and the last timestamp, and sets a timestamp for the packet based on the rate and the length of the packet. So packets will be delayed in the fq qdisc until their virtual TX time, and since there's only a single running timestamp for each queue mapping ID, this in effect becomes a virtual FIFO, with the only limit being the horizon timestamp, as you found.

The idea was to transfer https://netdevconf.info//0x14/pub/slides/55/slides.pdf / https://netdevconf.info//0x14/pub/papers/55/0x14-paper55-talk-paper.pdf for K8s and utilise the Pod egress rate annotations there.
[...]

Yup, I did recognise the approach from there. I'm also wondering why the Google folks didn't see these latency spikes. My best guess is that it's BBR throttling that kicks in early enough to mostly mask the FIFO behaviour. That, combined with some workload specific traffic properties may put you into "don't care about latency" territory for some deployments (especially if you're coming from a highly contended global HTB lock scenario)?

Or who knows, maybe I'm missing something fundamental here, and there's some reason why things are not as dire as I paint them above? If so, I would love to know what that reason is! :)

Hm, I wonder also if we're hitting other things such as sch->limit with the defaults, I've heard about issues like these due to defaults being too low. Would be good to trace kfree_skb to double check.
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/sched/sch_fq.c#n530

Those limit drops should be visible in the qdisc stats, then (tc -s qdisc). The default limit for fq is 10k packets, which is around 1.2 seconds at 100Mbit (for 1500-MTU packets), so yeah, I guess you could hit that limit before you hit the horizon drop. However, TCP backpressure should also kick in before that (TSQ should still work all the way into the container, right?), which I think is supported by the results in the graphs above - we're a fair way from 1.2 seconds.
Hmm, or maybe we're not, on the AWS instances, at least? There will be some smaller packets (ACKs) in the queue as well, so the ~400 ms could well be the queue overflow point? Maybe that's also the reason for the difference between the GCP and AWS results - i.e., differences in TCP stack backpressure effectiveness?

[0] Not quite sure how that is initially set, but seems to be coming from inside the pod? so veth queue id, I guess? anyway, doesn't look like there is more than at most a few IDs per pod.

Given all other skb fields such as mark etc are already used up, it basically stores the endpoint ID into queue_mapping which is preserved all the way through the stack and later resets it, so kernel picks queue via flow hash. Not great, but seems to function at least.

Right, OK, so it's basically one ID per container/pod? That's what I was assuming (as you'd want the limit to be global for that entity), I just didn't manage to trace the code back far enough to figure out where those IDs were coming from :)

@borkmann I am not sure if we are talking past each other or not? In the 100mbit FIFO case sch->limit (and/or sch_fq) should be knocked way down, not up. 100 packets tops. In the 10gbit case, internal to a container, no more than 2ms is needed. It's worse than that, in that with gso present a "packet" can get bulked up by 42x, which is why byte limits are better than packet limits.

@tohojo the results so far are from the rrul test which on a packet limit floods the queue with short acks and starves the data path somewhat while taking 1/15th the time to transmit. So why it is merely 400ms, not 1.2 seconds is explained by that. A pure upload test may well hit 1.2s (but due to exhausting other limits might take multiple flows to hit. See also: https://www.duo.uio.no/bitstream/handle/10852/45274/1/thesis.pdf
I do not know much about sch_fq, I thought it naively used 100 packets per flow? I recall Eric dumazet suggesting putting any form of ECN with a 5ms brick-wall limit over the whole qdisc before EDT came out. BBR internal to Google has an ECN response.
tsq does regulate simple things fairly well, but going back to the debate: https://github.com/systemd/systemd/issues/9725#issuecomment-413369212
I won that debate, then. :) I was hoping this new facility truly extended containers to be doing more of the right thing.
A good simple test would be 4 saturating flows + a typical request/response payload for the application (sized to fit initcwnd) or tcp_rr (which will understate, being only 5 packets). My guess is that the tps (transactions per second) at fq 100mbit, 10k packets would be 1000x worse than fq_codel.

Just following-up on a couple of things: I indeed ran the benchmarks with Legacy Host Routing. I ran some more with eBPF Host Routing and the results are much better: the download throughput is now maxed out and the latency doesn't spike as much. Here are the results for comparison (all of these are from n2-standard-8 instances on GCP):
Legacy Host Routing

eBPF Host Routing

I also gathered the qdisc stats on client nodes after running these benchmarks (I booted the node, ran 1 benchmark, gathered the stats and then killed the node)
Here are the results:
Stats after running the benchmark with Legacy Host Routing
$ sudo tc -s qdisc show dev ens4
qdisc mq 8002: root
 Sent 783370773 bytes 1006231 pkt (dropped 584, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq 0: parent 8002:8 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 2948b initial_quantum 14740b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop
 Sent 171368969 bytes 176640 pkt (dropped 104, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  flows 163 (inactive 156 throttled 0)
  gc 0 highprio 0 throttled 87232 latency 26.6us flows_plimit 104
qdisc fq 0: parent 8002:7 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 2948b initial_quantum 14740b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop
 Sent 205563021 bytes 201232 pkt (dropped 37, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  flows 170 (inactive 167 throttled 0)
  gc 0 highprio 0 throttled 95733 latency 23.2us flows_plimit 37
qdisc fq 0: parent 8002:6 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 2948b initial_quantum 14740b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop
 Sent 5867951 bytes 79383 pkt (dropped 185, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  flows 165 (inactive 162 throttled 0)
  gc 0 highprio 0 throttled 37829 latency 26.1us flows_plimit 185
qdisc fq 0: parent 8002:5 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 2948b initial_quantum 14740b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop
 Sent 4932330 bytes 56204 pkt (dropped 18, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  flows 173 (inactive 172 throttled 0)
  gc 0 highprio 0 throttled 31996 latency 27.6us flows_plimit 18
qdisc fq 0: parent 8002:4 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 2948b initial_quantum 14740b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop
 Sent 194851808 bytes 181687 pkt (dropped 40, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  flows 176 (inactive 166 throttled 0)
  gc 0 highprio 0 throttled 93956 latency 24.5us flows_plimit 40
qdisc fq 0: parent 8002:3 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 2948b initial_quantum 14740b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop
 Sent 4472102 bytes 61329 pkt (dropped 14, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  flows 165 (inactive 157 throttled 0)
  gc 0 highprio 0 throttled 33655 latency 29.1us flows_plimit 14
qdisc fq 0: parent 8002:2 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 2948b initial_quantum 14740b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop
 Sent 5385404 bytes 70527 pkt (dropped 107, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  flows 159 (inactive 155 throttled 0)
  gc 0 highprio 0 throttled 33977 latency 36.6us flows_plimit 107
qdisc fq 0: parent 8002:1 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 2948b initial_quantum 14740b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop
 Sent 190929188 bytes 179229 pkt (dropped 79, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  flows 173 (inactive 171 throttled 0)
  gc 0 highprio 0 throttled 88435 latency 27.7us flows_plimit 79
qdisc clsact ffff: parent ffff:fff1
 Sent<PHONE_NUMBER>9 bytes 1427707 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

Stats after running the benchmark with eBPF Host Routing
$ sudo tc -s qdisc show dev ens4
qdisc mq 8002: root
 Sent 768138725 bytes 2279589 pkt (dropped 13619, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq 0: parent 8002:8 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 2948b initial_quantum 14740b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop
 Sent 286859943 bytes 392418 pkt (dropped 1743, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  flows 93 (inactive 85 throttled 0)
  gc 0 highprio 0 throttled 110950 latency 18.3us flows_plimit 1743
qdisc fq 0: parent 8002:7 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 2948b initial_quantum 14740b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop
 Sent 274331527 bytes 415580 pkt (dropped 2506, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  flows 85 (inactive 79 throttled 0)
  gc 0 highprio 0 throttled 122406 latency 25.3us flows_plimit 2506
qdisc fq 0: parent 8002:6 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 2948b initial_quantum 14740b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop
 Sent 69129154 bytes 243111 pkt (dropped 1803, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  flows 103 (inactive 93 throttled 0)
  gc 0 highprio 0 throttled 116332 latency 17.6us flows_plimit 1803
qdisc fq 0: parent 8002:5 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 2948b initial_quantum 14740b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop
 Sent 63169165 bytes 250551 pkt (dropped 1382, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  flows 103 (inactive 97 throttled 0)
  gc 0 highprio 0 throttled 122248 latency 18us flows_plimit 1382
qdisc fq 0: parent 8002:4 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 2948b initial_quantum 14740b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop
 Sent 16547541 bytes 255166 pkt (dropped 1791, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  flows 87 (inactive 84 throttled 0)
  gc 0 highprio 0 throttled 124166 latency 21.1us flows_plimit 1791
qdisc fq 0: parent 8002:3 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 2948b initial_quantum 14740b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop
 Sent 15222457 bytes 235888 pkt (dropped 1523, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  flows 90 (inactive 85 throttled 0)
  gc 0 highprio 0 throttled 117827 latency 21.4us flows_plimit 1523
qdisc fq 0: parent 8002:2 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 2948b initial_quantum 14740b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop
 Sent 27700561 bytes 252835 pkt (dropped 1191, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  flows 88 (inactive 84 throttled 0)
  gc 0 highprio 0 throttled 120161 latency 23.5us flows_plimit 1191
qdisc fq 0: parent 8002:1 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 2948b initial_quantum 14740b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop
 Sent 15178377 bytes 234040 pkt (dropped 1680, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  flows 86 (inactive 82 throttled 0)
  gc 0 highprio 0 throttled 112652 latency 19.2us flows_plimit 1680
qdisc clsact ffff: parent ffff:fff1
 Sent<PHONE_NUMBER>90 bytes 2519853 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

And I also agree that the RRUL test is not very representative of a real container workload. As I mentioned, we have many hundreds of very different applications so it's a bit hard to find something representative tbh... I might try out

A better test would be to start, say two saturating flows and then measure transactions per second for a zillion other flows, netperf's tcp_rr test for example.

And also look for a couple of candidate applications to capture packets from as well.

I used iperf to test cilium bandwidth manager recently, and found the similar issue about its UDP throughput.
My limit is 10Mbps(kubernetes.io/egress-bandwidth: "100M"), And I tested it using 10Mbps UDP traffic.
# case 1: 10Mbps
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  11.9 MBytes  10.0 Mbits/sec  0.000 ms  0/8633 (0%)  sender
[  5]   0.00-10.00  sec  9.10 MBytes  7.63 Mbits/sec  0.304 ms  1786/8376 (21%)  receiver

Those limit drops should be visible in the qdisc stats, then (tc -s qdisc). The default limit for fq is 10k packets, which is around 1.2 seconds at 100Mbit (for 1500-MTU packets), so yeah, I guess you could hit that limit before you hit the horizon drop. However, TCP backpressure should also kick in before that (TSQ should still work all the way into the container, right?), which I think is supported by the results in the graphs above - we're a fair way from 1.2 seconds.

I can see there is no backpressure for UDP from @tohojo 's explanation and it causes the drop. In my case, when sending at 10Mbps, the limit is also 10Mbps, while the received throughput is smaller(7Mbps) . I can see flow_plimit increases from tc -s qdisc, could you help me understand why this still hits the flow_limit 100 since the sender's not exceeding the limit.

Any progress here?

A quick status update on our end: we've paused our work on the Bandwidth Manager for now, while we migrate our fleet to eBPF Host Routing which will take us a bit of time.
Also thank you @tohojo for the very well written blog post! FWIW, I am also planning on sharing some of our findings in a short presentation at Cilium + eBPF day in Paris in March (not sure how much I'll be able to fit in a 5 minute window but we'll see 😄)

Anton Ippolitov @.***> writes:

Any progress here?

A quick status update on our end: we've paused our work on the Bandwidth Manager for now, while we migrate our fleet to eBPF Host Routing which will take us a bit of time.
Also thank you @tohojo for the very well written blog post! FWIW, I am
also planning on sharing some of our findings in a short
presentation
at Cilium + eBPF day in Paris in March (not sure how much I'll be able
to fit in a 5 minute window but we'll see 😄)

Thanks for the update! Glad you enjoyed the blog post! :)

I would be tickled if you tried cake instead, and enabled ecn sender side.
tc qdisc add the_interface bandwidth 10Mbit rtt 5ms
you can add ecn support via a ip route, or via the global sysctl -w net.ipv4.tcp_ecn=1
The above rtt is sized more or less correctly for codel and the target bandwidth within containers. (250us target)

ping?

ping?

Hi Dave, I have it on my roadmap to work on this likely for Cilium 1.17.

2 seconds for within containers on the same device is kind of nuts...

@dtaht we are a happy user of tc-cake. I wish you a happy Thanksgiving and a terrific holiday season!
tc-cake has been used to throttle the ingress bandwidth of trino  clusters for several months . Peak bandwidth of all the trino containers is larger than 1.5 Tb/s.

Thanks to tc-cake, those services are able to running on the same nodes of the trino workers.
Let me share the stats of one of our k8s node.
A note for those who are not familiar with tc-cake: 'marks' means we have enabled TCP ECN.
# tc -s qdisc show dev ifb4eth0
qdisc mq 1: root
 Sent<PHONE_NUMBER>461062 bytes<PHONE_NUMBER> pkt (dropped 2454154, overlimits<PHONE_NUMBER> requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8001: parent 1:1 bandwidth 8Gbit diffserv3 dual-dsthost nonat nowash ingress no-ack-filter no-split-gso rtt 1ms noatm overhead 38 mpu 84
 Sent<PHONE_NUMBER>01650 bytes<PHONE_NUMBER> pkt (dropped 1192093, overlimits<PHONE_NUMBER> requeues 0)
 backlog 0b 0p requeues 0
 memory used: 4281088b of 4Mb
 capacity estimate: 8Gbit
 min/max network layer size:           46 /    1500
 min/max overhead-adjusted size:       84 /    1538
 average network hdr offset:           14

                   Bulk  Best Effort        Voice
  thresh        500Mbit        8Gbit        2Gbit
  target           50us         50us         50us
  interval          1ms          1ms          1ms
  pk_delay         91us         12us          0us
  av_delay         20us          2us          0us
  sp_delay          0us          1us          0us
  backlog            0b           0b           0b
  pkts        182803950<PHONE_NUMBER>            0
  bytes<PHONE_NUMBER>49999<PHONE_NUMBER>26351            0
  way_inds    225131238    599585639            0
  way_miss    533872792<PHONE_NUMBER>            0
  way_cols            0          943            0
  drops          826120       365973            0
  marks        51784581     46454061            0
  ack_drop            0            0            0
  sp_flows            1            2            0
  bk_flows            0            0            0
  un_flows            0            0            0
  max_len         68519        68519            0
  quantum          3028         3028         3028

qdisc cake 8002: parent 1:2 bandwidth 8Gbit diffserv3 dual-dsthost nonat nowash ingress no-ack-filter no-split-gso rtt 1ms noatm overhead 38 mpu 84
 Sent<PHONE_NUMBER>59412 bytes<PHONE_NUMBER> pkt (dropped 1262061, overlimits<PHONE_NUMBER> requeues 0)
 backlog 0b 0p requeues 0
 memory used: 4281472b of 4Mb
 capacity estimate: 8Gbit
 min/max network layer size:           46 /    1500
 min/max overhead-adjusted size:       84 /    1538
 average network hdr offset:           14

                   Bulk  Best Effort        Voice
  thresh        500Mbit        8Gbit        2Gbit
  target           50us         50us         50us
  interval          1ms          1ms          1ms
  pk_delay         22us          3us          0us
  av_delay          2us          1us          0us
  sp_delay          0us          1us          0us
  backlog            0b           0b           0b
  pkts        193810664<PHONE_NUMBER>            0
  bytes<PHONE_NUMBER>80545<PHONE_NUMBER>36187            0
  way_inds    225356579    604529018            0
  way_miss    533784418<PHONE_NUMBER>            0
  way_cols            0          707            0
  drops          839621       422440            0
  marks        51804336     46385740            0
  ack_drop            0            0            0
  sp_flows            1            1            0
  bk_flows            0            1            0
  un_flows            0            0            0
  max_len         68519        68519            0
  quantum          3028         3028         3028

# tc -s qdisc show dev ifb4eth1
qdisc mq 1: root
 Sent<PHONE_NUMBER>717826 bytes<PHONE_NUMBER> pkt (dropped 2501391, overlimits<PHONE_NUMBER> requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8003: parent 1:1 bandwidth 8Gbit diffserv3 dual-dsthost nonat nowash ingress no-ack-filter no-split-gso rtt 1ms noatm overhead 38 mpu 84
 Sent<PHONE_NUMBER>48699 bytes<PHONE_NUMBER> pkt (dropped 1223367, overlimits<PHONE_NUMBER> requeues 0)
 backlog 0b 0p requeues 0
 memory used: 4281216b of 4Mb
 capacity estimate: 8Gbit
 min/max network layer size:           46 /    1500
 min/max overhead-adjusted size:       84 /    1538
 average network hdr offset:           14

                   Bulk  Best Effort        Voice
  thresh        500Mbit        8Gbit        2Gbit
  target           50us         50us         50us
  interval          1ms          1ms          1ms
  pk_delay         58us         34us          0us
  av_delay          7us          2us          0us
  sp_delay          0us          0us          0us
  backlog            0b           0b           0b
  pkts        242564561<PHONE_NUMBER>            0
  bytes<PHONE_NUMBER>38520<PHONE_NUMBER>59171            0
  way_inds    225554613    606753461            0
  way_miss    533905978<PHONE_NUMBER>            0
  way_cols            0         3117            0
  drops          841871       381496            0
  marks        52162961     46514679            0
  ack_drop            0            0            0
  sp_flows            0           22            0
  bk_flows            1            0            0
  un_flows            0            0            0
  max_len         68519        68519            0
  quantum          3028         3028         3028

qdisc cake 8004: parent 1:2 bandwidth 8Gbit diffserv3 dual-dsthost nonat nowash ingress no-ack-filter no-split-gso rtt 1ms noatm overhead 38 mpu 84
 Sent<PHONE_NUMBER>69127 bytes<PHONE_NUMBER> pkt (dropped 1278024, overlimits<PHONE_NUMBER> requeues 0)
 backlog 0b 0p requeues 0
 memory used: 4181Kb of 4Mb
 capacity estimate: 8Gbit
 min/max network layer size:           46 /    1500
 min/max overhead-adjusted size:       84 /    1538
 average network hdr offset:           14

                   Bulk  Best Effort        Voice
  thresh        500Mbit        8Gbit        2Gbit
  target           50us         50us         50us
  interval          1ms          1ms          1ms
  pk_delay         50us          5us          0us
  av_delay         12us          1us          0us
  sp_delay          0us          0us          0us
  backlog            0b           0b           0b
  pkts        238720122<PHONE_NUMBER>            0
  bytes<PHONE_NUMBER>21517<PHONE_NUMBER>51471            0
  way_inds    221314827    605251303            0
  way_miss    533859584<PHONE_NUMBER>            0
  way_cols            0          480            0
  drops          848823       429201            0
  marks        52048251     46370197            0
  ack_drop            0            0            0
  sp_flows            0           12            0
  bk_flows            0            0            0
  un_flows            0            0            0
  max_len         68519        68519            0
  quantum          3028         3028         3028

50us is mind-blowing. One other thing that stands out for me is the very low hash collision rate - the theory said an 8 way set associative hash would work, and seeing it work is great!

@wenjianhn you might drop less packets if you hand cake a bit more memory via the memlimit parameter. Say 8Mbytes rather than 4. What that will do to your p99 would have to be measured, though!

If you're interested in jitter, there's a nice video here
https://youtu.be/I_TtMk5z0O0?si=EUdasVZg-nt2YmMV
regards,
Dave


Git clone depth 1

I'd like to clone a very "minimalistic" version of my repo.
I tried this using the following command:
git clone http://gitServer/Repo.git --branch Branch1 --single-branch --depth 1 Branch1

The command never finishes.
When I start the same command (without using Bonobo) it works:
git clone "C:\PathToRepo\Repo" "C:\Dest\Repo" --branch Branch1 --single-branch --depth 1

Am i doing something wrong or has Bonobo a problem with --depth 1?

Depth --1 is automatically added by bower v>=1.4.0 due to feature detection done here. Bonobo is saying it support shallowing cloning but then hangs when requested to do so. Please get this fixed as we cannot use the latest bower version with bonobo.

Is this still an issue with 5.1.1?
if it is please reopen the issue.


Triage view text is not selectable or copyable

Version and Platform (required):

Binary Ninja Version: 2.4.3149-dev
OS: Windows
OS Version: 11

Bug Description:
Text in triage view is not selectable, and therefor not copyable.
Steps To Reproduce:
Please provide all steps required to reproduce the behavior:

Go to triage view for any binary
Attempt to select any of the displayed text
See error

Expected Behavior:
I expect that all text is selectable and copyable.

2:22 PM <jon> could you make an issue? this should be an easy task

This is going to be a little more complicated since a lot of the fields in triage view are actually token-based (and custom-painted) rather than just QLabel instances with monospace fonts.

This could be done a bit more simply by adding a right-click->copy  which would copy everything, rather than making them individually selectable.


Autorest Regen Preview to 3.0.0-alpha.20240307.10 by Mingzhe Huang from refs/pull/4363/head

Triggered from https://github.com/Azure/autorest.csharp/pull/4363

API change check
API changes are not detected in this pull request.


Cookbook: add SignalXY recipe demonstrating DateTime horizontal axis

https://scottplot.net/cookbook/5.0/SignalXY/ could use a recipe like https://scottplot.net/cookbook/5.0/Signal/SignalDateTime/
Edit this file:
https://github.com/ScottPlot/ScottPlot/blob/main/src/ScottPlot5/ScottPlot5 Cookbook/Recipes/PlotTypes/SignalXY.cs
To add a block similar to:
https://github.com/ScottPlot/ScottPlot/blob/5936757cffe2d15923c8f1dd769f9af7d4997d06/src/ScottPlot5/ScottPlot5 Cookbook/Recipes/PlotTypes/Signal.cs#L167-L184

DateTime start = new(2024, 01, 01);
DateTime[] xs = Generate.ConsecutiveDays(100, start);
double[] ys = Generate.RandomWalk(100);
myPlot.Add.SignalXY(xs, ys);
myPlot.Axes.DateTimeTicksBottom();


brew cask uninstall teamviewerleaves artifacts in /Applications

The app is still there but it doesn't work after 'uninstall'.

Fixed.


add flutter plugin

Can you add a basic README documenting what it does and what aliases it defines? The npm plugin is a good example.

@mcornella readme added

Good job thanks!


BAMF - NNUnet Kidney CT

@rahul99 didn't this model delineate kidney cysts too?
We're about to add an entry to SegDB for kidney cysts after consultation with @dclunie.

I extended our SegDB with the morphologically abnormal structure CYST.
Now, you can use LEFT_KIDNEY+CYST to encode a cyst in the left kidney or KIDNEY+CYST in case your nnUNet algorithm does not distinguish between cysts in the left or right kidney.

@rahul99 for testing the model, please provide the following for a sample CT series that is known to produce acceptable results with this model:
*SeriesInstanceUID

aws_url for the bucket folder containing the series files
version of IDC data that you used for getting the above

If you pick one of the cases that are already accompanied by a segmentation produced by this model in IDC (they have just been added yesterday in IDC v17), you can also include the items above for the SEG series.

/test
sample:
  idcv: 17.0
  data:
    - SeriesInstanceUID: <IP_ADDRESS>.4.1.145<IP_ADDRESS>.9203.4004.385483163880959624431394063795
      aws: 43b672b6-dd41-4a00-a1ad-dd5a8cf407df
      path: dicom

reference:
  url: https://www.dropbox.com/scl/fi/v517ibmkdggsz3kgmfd9g/bamf_nnunet_ct_kidney.zip?rlkey=4ecm7zd23g1vqwh444dytyxen&dl=0

aws: 43b672b6-dd41-4a00-a1ad-dd5a8cf407df

@rahul99 I believe you need to update this to follow the updated schema (attribute names changed, full S3 URL is required), see here for the example: https://github.com/MHubAI/models/pull/71#issuecomment-1977987878.

/test
sample:
  idc_version: 17.0
  data:
    - SeriesInstanceUID: <IP_ADDRESS>.4.1.145<IP_ADDRESS>.9203.4004.385483163880959624431394063795
      aws_url: s3://43b672b6-dd41-4a00-a1ad-dd5a8cf407df/*
      path: dicom

reference:
  url: https://www.dropbox.com/scl/fi/v517ibmkdggsz3kgmfd9g/bamf_nnunet_ct_kidney.zip?rlkey=4ecm7zd23g1vqwh444dytyxen&dl=0

aws_url: s3://43b672b6-dd41-4a00-a1ad-dd5a8cf407df/*

This is not a valid URL. To look up the URL you can use the following query:
SELECT
  DISTINCT(series_aws_url)
FROM
  `bigquery-public-data.idc_current.dicom_all`
WHERE
  SeriesInstanceUID = '<IP_ADDRESS>.4.1.145<IP_ADDRESS>.9203.4004.385483163880959624431394063795'

Which returns s3://idc-open-data/43b672b6-dd41-4a00-a1ad-dd5a8cf407df/.

/test
sample:
  idc_version: 17.0
  data:
    - SeriesInstanceUID: <IP_ADDRESS>.4.1.145<IP_ADDRESS>.9203.4004.385483163880959624431394063795
      aws_url: s3://idc-open-data/43b672b6-dd41-4a00-a1ad-dd5a8cf407df/*
      path: dicom

reference:
  url: https://www.dropbox.com/scl/fi/v517ibmkdggsz3kgmfd9g/bamf_nnunet_ct_kidney.zip?rlkey=4ecm7zd23g1vqwh444dytyxen&dl=0

Thanks @fedorov , I was just verifying the url and was about to update it. I see you already did that, thanks :)

/test
sample:
  idc_version: 17.0
  data:
  - SeriesInstanceUID: <IP_ADDRESS>.4.1.145<IP_ADDRESS>.9203.4004.385483163880959624431394063795
    aws_url: s3://idc-open-data/43b672b6-dd41-4a00-a1ad-dd5a8cf407df/*
    path: dicom

reference:
  url: https://www.dropbox.com/scl/fi/v517ibmkdggsz3kgmfd9g/bamf_nnunet_ct_kidney.zip?rlkey=4ecm7zd23g1vqwh444dytyxen&dl=0

Hey @LennyN95 , I have update meta with whatever information I could find in the source paper. Do I need to comment the test case again or will the last one be picked up automatically since there is no change in the test?

Perfect! The passed test is still valid then, you won't need to redo it. I'll review in a bit and then continue to merge and push. Great work @rahul99, it's so great to see the platform growing now!

@rahul99 could you upload 1-5 images for the model card that show the model in action? This is not a hard requirement, thus I moved on with the push. However, from a UX perspective, visuals add a great value to the model page and also help readers to quickly recognize a model (especially as the model card is made to look similar across all models, images can be a great help here).


add y-axis to dotplot

Currently the dot plot doesn't have any y-axis labels and marker values.  Can we add those in the plot..

Sorry this is not possible, it is a limitation of ggplot2 :(


No space left on device

Every second run has an issue with space on device as below:
failed to register layer: Error processing tar file(exit status 1): write /usr/local/opt/gridlabd/4.3.1-230317-develop-Linux-x86_64/lib/python3.9/site-packages/pyproj/proj_dir/share/proj/proj.db: no space left on device

Command returned nonzero code: 1

This issue is rooted within the system's caching. It will require an intensive fix, as when gridlabd:latest is cached, it fills up all available space and prevents untar for gridlabd:develop.
We will need to fix how images are cached and used, or potentially how traffic to a specific runner is managed, to fix this. Short term solution is to use a local openfido deployment, and tear down/rebuild it when you need to switch between gridlabd images.


Fix issue with generating component initiatilze_signature from attributes

Issue
ComponentGenerator creates component with broken parameters leaving off a : on the last attribute. For example rails g component Example hello world generates:
class ButtonComponent < ActionView::Component::Base

  def initialize(hello:, world)
.....

Solution
@dylnclrk and I altered the #map method in ComponentGenerator#initialize_signature to append the : and then join the array with , .
NOTE: We did not add a test for this case since there are no established patterns for testing the generators.

Thanks for the fix!
Do you have any recommendations/references for testing generators? It sounds like something we should do here soon.

@joelhawksley I did some searching into testing rails generators and looked at the source code of other gems that use generators and didn't find anything concrete for minitest. Most of the references I found related to rspec.


"Feature" request: Better documentation and code examples

I know that this is still mostly an early WIP project with an unstable API, but I think it would still be nice to have some more code examples in the wiki to illustrate, incrementally, how certain features are implemented using this library. Yes, one could look in the code demos and the documentation, but this would still be nice to have. For example, see how this unrelated library does it in their wiki.
P.S. I quite like this project, and it's much less clunky than CDK, let alone directly using ncurses.

Ignore if this is already something that is already being addressed, or will be addressed once the API has stabilzed.

Not something that warrants an issue or feature request.

Hey, I think some documentation could really help out this project, at the moment it is mostly kept in my head, which is less than ideal. I don't have a whole lot of time at the moment to work on this, but it is definitely a high priority.
And thanks for the link, that's a great example. I've been trying to design a generic Table widget API so it might help me with that too.

Hello, a-n-t-h-o-n-y! I opened this "issue" because I quite like this project (much better than something like CDK), but didn't think that opening an issue would be appropriate, since you probably had this in mind anyway. But, in the meantime, I think piece-by-piece code samples could work, too. For example, a sample that demonstrates the use of one or two things instead of a full control gallery or toy program, for those who are lazy to read larger samples, and/or for people to learn CPPurses control by control.

Something like Qt’s widget samples? Small, independent samples illustrating
a few controls and usage of the library, which can then be included or
referenced in the docs, perhaps?
Jon
On Thu, Nov 21, 2019 at 8:51 AM Deneuki<EMAIL_ADDRESS>wrote:

Hello, a-n-t-h-o-n-y! I opened this "issue" because I quite like this
project (much better than something like CDK), but didn't think that
opening an issue would be appropriate, since you probably had this in mind
anyway. But, in the meantime, I think piece-by-piece code samples could
work, too. For example, a sample that demonstrates the use of one or two
things instead of a full control gallery or toy program, for those who are
lazy to read larger samples, and/or for people to learn CPPurses control by
control.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/a-n-t-h-o-n-y/CPPurses/issues/30?email_source=notifications&email_token=AC2NUY7OZTIKJ3VVKWEUVNTQU2N53A5CNFSM4JE22ZO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEE2PKQA#issuecomment-557118784,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AC2NUY4CGPE5VISDOKBU373QU2N53ANCNFSM4JE22ZOQ
.

Yes, I was thinking of something like that. It's not entirely necessary, as some complete control galleries are pretty simply written, or written in a modular, control-by-control (or category-by-category) way, but I think this would be both easier to read and write, in terms of self-documenting code.


Error when querying for current attestation policy

Describe the bug
Command Name
az attestation policy show Extension Name: attestation. Version: 0.2.0.
Errors:
The command failed with an unexpected error. Here is the traceback:
It is required that you pass in a value for the "algorithms" argument when calling decode().
Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/knack/cli.py", line 231, in invoke
    cmd_result = self.invocation.execute(args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 657, in execute
    raise ex
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 720, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 691, in _run_job
    result = cmd_copy(params)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 328, in __call__
    return self.handler(*args, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/command_operation.py", line 363, in handler
    show_exception_handler(ex)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/arm.py", line 429, in show_exception_handler
    raise ex
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/command_operation.py", line 361, in handler
    return op(**command_args)
  File "/home/csegarra/.azure/cliextensions/attestation/azext_attestation/manual/custom.py", line 191, in get_policy
    policy = jwt.decode(token, verify=False).get('x-ms-policy', '')
  File "/opt/az/lib/python3.6/site-packages/jwt/api_jwt.py", line 119, in decode
    decoded = self.decode_complete(jwt, key, algorithms, options, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/jwt/api_jwt.py", line 87, in decode_complete
    'It is required that you pass in a value for the "algorithms" argument when calling decode().'
jwt.exceptions.DecodeError: It is required that you pass in a value for the "algorithms" argument when calling decode().

To Reproduce:
Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

az attestation policy show --resource-group {} --name {} --attestation-type {}

Expected Behavior
Environment Summary
Linux-4.15.0-96-generic-x86_64-with-debian-buster-sid
Python 3.6.10
Installer: DEB

azure-cli 2.27.2

Extensions:
attestation 0.2.0

Additional Context

route to service team

@csegarragonz, could you please check the version of pyjwt on your machine? There are some breaking changes since pyjwt 2.0.0.


login logic

how to create google login.

[x] Research and reading documentation

[x] Creating code for login logic

How to Sign In with Google
Google's How to get started documentation
1. Google Cloud Platform Setup
Create a Project: Go to the Google Cloud Console, create a new project or select an existing one.
Enable APIs: Navigate to "APIs & Services > Dashboard", search for "Google Identity Services", and enable it.
Configure OAuth Consent Screen: Under "APIs & Services", go to "OAuth consent screen" and provide the necessary details about your application.
Create OAuth 2.0 Client IDs: In "Credentials", click "+ CREATE CREDENTIALS" and select "OAuth 2.0 Client IDs". Fill in the application type, authorized JavaScript origins, and redirect URIs.
2. Frontend Implementation
Load Google Identity Services Library: Include  in your HTML.
Add a Google Sign-In Button: Use the div id="g_id_onload" data-client_id="YOUR_CLIENT_ID" data-callback="handleCredentialResponse">div" to add a sign-in button. Replace "YOUR_CLIENT_ID" with your actual client ID.
Handle Sign-In Response: Implement a handleCredentialResponse function in your JavaScript to handle the sign-in response, which includes the user's ID token.
Parse User Information: Optionally, parse the ID token (JWT) to extract user information like name, email, and profile picture.
3. Backend Verification (Optional but Recommended)
Send ID Token to Your Server: After successful sign-in, send the ID token to your server using an HTTP request.
Verify ID Token: Use Google's library (available in multiple languages) on the server to verify the token's integrity, ensuring it's valid and intended for your application.
Create/Update User Session: Based on the verified token, create or update the user session in your application.
4. Usage
Access Protected Resources: Use the access token obtained through this process to call Google APIs or access protected resources in your application on behalf of the signed-in user.
Sign Out: Implement a sign-out mechanism by using google.accounts.id.disableAutoSelect() and clearing the application's session.


webpack_require why is slow?

i find  webpack_require is slow in chrome than requirejs, they all use AMD. but why???
and how make it faster??

Have you measured the slowness? If so, what are the results?

@gagayang I'm going to close this for now as this not really a question worth answering unless you have merit-able performance data. If you have a better solution, I would recommend a proposal or PR along with reasonings and performance data to back it up.

I have experienced this too - something weird is happening on chrome where it's really slow. I don't have performance measures as all of a sudden it started happening. I am on beta.20

If you could provide the information in the github issue template, then this will allow us to help you!!

Or create a new issue with useful information for us.

It is true that __webpack_require__ will be the top of any flame chart and profile, since it's the function that actually runs literally everything else in the bundle, but the function itself should have negligible run time internally...

Not sure what is best to describe this issue to be honest :S
But this is my best stab at it so far:

Notice, after the javascript is loaded, there seems to be 20-30s gap between that and when addStyles kicks in to add the style blobs to the head. Not sure why - it only happens on chrome and not on safari.

Can you provide a minimal working example?

Sorry, it's part of work app :(


UE 5.1.1 Map data doesn't appear in viewport just anchor point.

I get it to compile on UE5.1.1 but when I Import an OSM file I get no output in the view port just the anchor. I figured I'll downgrade and migrate but that didn't work.

Yeah, I can reproduce this as well in 5.2.  The map is there but only renders if its selected.  You can select it in the World Outliner.  If you use the "Generate Mesh" button in the details panel, you can at least make a static mesh representation of the map.  I'll investigate further when I have a chance!

I also got the same results but when I generate the mesh it creates it like if its wrapped in a bound node in Houdini to large for the world map even with streaming enabled.

Latest commit should fix it!  Thanks for letting me know.
For the generated mesh being too big, I guess that would depend on the metrics of your OSM data.  But in any case, the built-in rendering for the map and also the mesh generator are really just simplistic examples.  For a real city with streaming good visuals, you'd probably want to interpret the loaded OSM map data offline to generate nice looking worlds.

Thanks, I'll compile it now.


Version 0.6.17 breaks the behavior

It should have been a minor release.
I'm not sure what's the issue now and I cannot show the code, but I'll tell when I figure out.

@antoyo Any update on this?

Nope. Didn't have the time to investigate.


Type parameters missing from typed_data.dart.

patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_ExternalUint8ClampedArray::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_ExternalUint8ClampedArray::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_ExternalUint8ClampedArray::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_Int8List::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_Int8List::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_Int8List::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_Uint8List::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_Uint8List::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_Uint8List::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_Uint8ClampedList::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_Uint8ClampedList::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_Uint8ClampedList::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_Int16List::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_Int16List::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_Int16List::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_Uint16List::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_Uint16List::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_Uint16List::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_Int32List::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_Int32List::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_Int32List::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_Uint32List::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_Uint32List::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_Uint32List::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_Int64List::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_Int64List::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_Int64List::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_Uint64List::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_Uint64List::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_Uint64List::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_Float32List::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_Float32List::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_Float32List::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_Float64List::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_Float64List::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_Float64List::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_Float32x4List::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_Float32x4List::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_Float32x4List::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_Int32x4List::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_Int32x4List::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_Int32x4List::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_Float64x2List::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_Float64x2List::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_Float64x2List::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_ExternalInt8Array::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_ExternalInt8Array::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_ExternalInt8Array::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_ExternalUint8Array::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_ExternalUint8Array::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_ExternalUint8Array::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_ExternalInt16Array::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_ExternalInt16Array::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_ExternalInt16Array::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_ExternalUint16Array::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_ExternalUint16Array::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_ExternalUint16Array::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_ExternalInt32Array::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_ExternalInt32Array::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_ExternalInt32Array::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_ExternalUint32Array::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_ExternalUint32Array::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_ExternalUint32Array::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_ExternalInt64Array::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_ExternalInt64Array::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_ExternalInt64Array::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_ExternalUint64Array::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_ExternalUint64Array::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_ExternalUint64Array::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_ExternalFloat32Array::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_ExternalFloat32Array::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_ExternalFloat32Array::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_ExternalFloat64Array::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_ExternalFloat64Array::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_ExternalFloat64Array::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_ExternalFloat32x4Array::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_ExternalFloat32x4Array::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_ExternalFloat32x4Array::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_ExternalInt32x4Array::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_ExternalInt32x4Array::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_ExternalInt32x4Array::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_Int8ArrayView::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_Int8ArrayView::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_Int8ArrayView::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_Uint8ArrayView::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_Uint8ArrayView::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_Uint8ArrayView::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_Uint8ClampedArrayView::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_Uint8ClampedArrayView::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_Uint8ClampedArrayView::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_Int16ArrayView::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_Int16ArrayView::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_Int16ArrayView::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_Uint16ArrayView::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_Uint16ArrayView::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_Uint16ArrayView::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_Int32ArrayView::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_Int32ArrayView::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_Int32ArrayView::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_Uint32ArrayView::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_Uint32ArrayView::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_Uint32ArrayView::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_Uint64ArrayView::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_Uint64ArrayView::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_Uint64ArrayView::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_Float32ArrayView::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_Float32ArrayView::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_Float32ArrayView::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_Float64ArrayView::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_Float64ArrayView::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_Float64ArrayView::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_Float32x4ArrayView::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_Float32x4ArrayView::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_Float32x4ArrayView::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_Int32x4ArrayView::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_Int32x4ArrayView::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_Int32x4ArrayView::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_Float64x2ArrayView::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_Float64x2ArrayView::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_Float64x2ArrayView::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_Int64ArrayView::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_Int64ArrayView::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_Int64ArrayView::expand' doesn't match those on overridden method 'Iterable::expand'.
patched_sdk/lib/typed_data/typed_data.dart:3237:3: Warning: Declared type variables of '_ExternalFloat64x2Array::map' doesn't match those on overridden method 'Iterable::map'.
patched_sdk/lib/typed_data/typed_data.dart:3228:3: Warning: Declared type variables of '_ExternalFloat64x2Array::fold' doesn't match those on overridden method 'Iterable::fold'.
patched_sdk/lib/typed_data/typed_data.dart:3239:3: Warning: Declared type variables of '_ExternalFloat64x2Array::expand' doesn't match those on overridden method 'Iterable::expand'.

To reproduce these warnings:

patch in CL<PHONE_NUMBER>.
Remove the TODO in pkg/front_end/lib/src/fasta/kernel/kernel_class_builder.dart marked with this bug.
Build the target runtime_kernel.


rest api separate certificate, and client verification

You can file this under enhancement.
I have recently been working with the rest api and would like empire to have the ability to define a separate certificate to use with rest.
I also think there should be a option to enable client certificate checking, this would greatly increase the security especially if the rest port is internet facing.
We could also add a little helper script to generate the needed certificates, just like the one we currently have.
Thank you.

We won't be able to implement this feature request at this time. Thanks.


Objective Evaluation

Hello author， I adopt “https://github.com/slSeanWU/MusDr” to evaluate the generated MIDI.
But after I process the midi file to get the pickle file, the MusDr warehouse also needs the corresponding csv file to get the objective evaluation value. How do you convert the processed pickle file to csv?

The quick answer is I am not sure since it's been a long time. We may not have converted midi to csv. We might measure pitch entropy and grooving similarity using Muspy, which does not require conversion. As for Structure Indicators, I can only remember we used some public Matlab code. I am not sure about the source.
I will try to find our evaluation code, but I am unsure when and if I can find it. I suggest you check Muspy, MusDR, and The Jazz Transformer.

Thank you for your reply, if you can find the evaluation code, that would be great, looking forward to your reply


[enhancement] Capability of applying different inset or item/line spacing to every section controller of IGListStackedSectionController.

New issue checklist

[x] I have searched existing issues and this is not a duplicate
[x] I have reviewed the README and documentation

General information

IGListKit version: 2.1.0
iOS version(s): 10.2
CocoaPods/Carthage version: 1.1.1
Xcode version: 8.2
Devices/Simulators affected: both
Reproducible in the demo project?: Yes
Related issues: None

===============
Let's say I have a stackedSectionController managing three section controllers:
...
let stackedSectionController = IGListStackedSectionController(sectionControllers: [
	SectionControllerAAA(),
	SectionControllerBBB(),
	SectionControllerCCC(),
	])
...

If I want to only give inset and line spacing to SectionControllerAAA.
The problem is: setting inset and line spacing in SectionControllerAAA's initializer does't work. Because it is managed by stackedSectionController (I assumed all cells of all three sub section controllers are combined into a single section controller).
So I set inset and line spacing to the stackedSectionController:
...
stackedSectionController?.inset = UIEdgeInsets(top: 5, left: 5, bottom: 5, right: 5)
stackedSectionController?.minimumLineSpacing = 5
...

But another problem would be:
Setting inset and line spacing on stackedSectionController would effects all the three section controllers (and top/bottom inset doesn't apply between different sub section controllers) which is obviously not what I want in this case.
So I suggest to enhance IGListKit with the capability of applying different inset or item/line spacing to every section controller of IGListStackedSectionController.
I haven't got a chance look into it yet, so that I don't have the idea how complicated this enhancement would be. If this suggestion makes sense to you guys, I will go ahead try to do this.

I love this idea, but I think that w/out a heavily customized UICollectionViewLayout that is designed specifically to integrate w/ IGListKit, it'll be almost impossible. UICollectionViewFlowLayout only applies insets to the entire section. The new layout we're introducing in #449 also uses that logic.
We could look into extending the new layout w/ this, but atm I think it's going to be too complicated.
IMO if the sections need to behave that differently, I wonder if they belong in the stack in the first place?
@ShaneQi mind if we close this?

@rnystrom
I looked into source code a little bit earlier today, seems too complicated, not worthy.
Also, my model is too complicated which makes layout complicated. I may need to simplify my model in the first place.
I agree we close this issue.
Thanks for the time!

Hello guys. Want to up this topic.
Is there any way to make space between SectionControllers inside IGListStackedSectionController ?

Unfortunately we're planning on deprecating IGListStackedSectionController (see #1099), but even if we weren't this would require a custom UICollectionViewLayout.


changed: packaging using rye and use file relative path instead of pw…

…d relative.
Pull Request Checklist

[x] Target branch: Pull requests should target the dev branch.
[x] Description: Briefly describe the changes in this pull request.
[x] Changelog: Ensure a changelog entry following the format of Keep a Changelog is added at the bottom of the PR description.
[x] Documentation: Have you updated relevant documentation Open WebUI Docs, or other documentation sources?
[x] Dependencies: Are there any new dependencies? Have you updated the dependency versions in the documentation?
[x] Testing: Have you written and run sufficient tests for the changes?

Description
Packaging open-webui as a Python package with minimum modification. It is not rye-only, any package support PEP 517 can build and package at the root of this project.
I also noticed that openwebui is currently not token in pypi.org, maybe it is a better choice?
after packaging and publishing to pypi, we can declare namespace package as plugin system, then langchain agent support could be easily implemented.

Changelog Entry
Added

Python packaging

Changed

Use file relative (relative path to backend/config.py) instead of pwd relative path in backend/config.py, so it can start backend at the root directory of this project.

Additional Information

now we can produce the wheel file and sdist for publishing

$ rye build   # file will place at dist/
$ rye publish  # after you config pypi account, you can upload the backend file to Pypi.org

backend/requirements.txt, backend/dev.sh, backend/start.sh and backend/start_windows.bat now can be removed, but weather delete them or not is depend on you.

related to #1885

@tcztzy Are you on our discord server? I'd prefer to setup a high bandwidth communication channel to configure this.

@tcztzy Are you on our discord server? I'd prefer to setup a high bandwidth communication channel to configure this.

OK, now I login in discord as tcztzy

Now I realize following features:

pip install open-webui && open-webui serve
Add a ci.yml github workflow, every time main branch has pull event, it will release to pypi.org.

You may want to claim both open-webui and openwebui just in case someone decides to post malware to the package.


pathfind: Add new Python API functions to C API and update dependencies

Adds the Python API functions added by @devw4r to the C API as well.
Also updates the dependency submodules since we might as well keep up to date.
I think the pybind update is the most necessary since it adds support for newer Python versions.
If you don't want the dependency updates let me know and I'll remove them.

Sorry for the force pushed. I goofed up and made pathfind_has_adts return a bool instead of a PathfindResultType.