RUDY and global route map differ for testcase with centered die area

Describe the bug
More fallout from  https://github.com/The-OpenROAD-Project/OpenROAD/issues/5284
The RUDY map looks different then the global_route map.

And then there are ERROR's from detailed placement and manually started global_route ...
[INFO DPL-0034] Detailed placement failed on the following 62 instances:
[INFO DPL-0035]  lx1/lx0/IRAM/MUX_INSTW_UP/U27
[INFO DPL-0035]  lx1/lx0/IRAM/MUX_INSTW_UP/U28
...
[INFO DPL-0035]  lx1/lx0/IRAM/U40
[INFO DPL-0035]  lx1/lx0/IRAM/U41
[ERROR DPL-0036] Detailed placement failed.

 global_route
[ERROR GRT-0118] Routing congestion too high. Check the congestion heatmap in the GUI.
[ERROR GUI-0070] GRT-0118

I'm assuming that this testcase was P&R'able in 2004, when init floorpan would always center the diearea.
Expected Behavior
matching rudy and global route map
Environment
OpenROAD v2.0-14335-g0a6aa3b92 (Jun 27 2024)
Features included (+) or not (-): +Charts +GPU +GUI +Python

To Reproduce
read_lef lef/risc2.lef.gz
read_def def/risc2.def.gz
#rtl_macro_placer
macro_placement
global_placement -density 0.95
detailed_placement
global_route

Relevant log output
No response
Screenshots
No response
Additional Context
No response

I was referring to the sedsm commands used to run this iccad04 testcase back in the days.

This design has 90+% placement density so it seems unlikely it was every routable.
I don't know what sedsm commands are but I don't think we've ever had this in our regression

sedsm stands for "silicon ensemble deep sub micron", the name of cadence's P&R tool at the time.
This is a 6 layer testcase where the ram's are only blocked up to M3, no power routing.
Yes, density looks high by today's standards, but back then this must have been routable.
It was part of the iccad04 testcases. I didn't make this thing up.
There is a huge discrepancy between RUDY and global route heat maps.
The ram placement could very likely be improved upon, macro_placement needs to use halo's.
There is a general question whether heat map's should display the usage of "overall" routing resources or "available" routing resources. When you have ram's blocking M1/M2/M3 but M4/M5/M6 are available, yes, then only have 2 layers less resources then the stdcell area, but if M4/M5/M6 routing resources are only used sparingly, the ram area's shouldn't show up in dark red, no ?

Using the OpenRoad gui to visualize the routing tracks, it seems as if metal2/metal3 tracks are missing.
Looking at def/risc2.def.gz, they're there. Maybe OpenRoad doesn't read this "old" notation ?
   285 TRACKS Y -479400 DO 2399 STEP 400 LAYER metal1 metal2 ;
   286 TRACKS Y -479400 DO 2399 STEP 400 LAYER metal2 metal3 metal4 ;
   287 TRACKS Y -479400 DO 1200 STEP 800 LAYER metal4 metal5 metal6 ;
   288 TRACKS X -479400 DO 2399 STEP 400 LAYER metal1 metal2 metal3 ;
   289 TRACKS X -479400 DO 2399 STEP 400 LAYER metal3 metal4 metal5 ;
   290 TRACKS X -479400 DO 1200 STEP 800 LAYER metal5 metal6 ;
   291 GCELLGRID Y -479600 DO 240 STEP 4000 ;
   292 GCELLGRID Y 480000 DO 1 STEP 0 ;
   293 GCELLGRID X -479600 DO 240 STEP 4000 ;
   294 GCELLGRID X 480000 DO 1 STEP 0 ;

I stand corrected.
a) OpenRoad reads the tracks just fine.
I just didn't zoom in far enough to see them.
The M2/M3 are not visible unless I zoom in to less than the height of a single stdcell whereas M1/M4/M5/M6 are brighter and visible at larger zoom factors.
b) The 4 large ram's are blocked up to M5, so only one layer routing over them.
Which means that the RUDY map looks as if it might correctly show the usage of available resources.
But the discrepancy between RUDY and groute map is still suspicious to me.

global_route --allow_congestion
yupp, global_route is correct, > 150% on all layers, this won't route at all.
But then why does RUDY shows the stdcell area's as being routable ?
[WARNING GRT-0300] Timing is not available, setting critical nets percentage to 0.
[INFO GRT-0020] Min routing layer: metal1
[INFO GRT-0021] Max routing layer: metal6
[INFO GRT-0022] Global adjustment: 0%
[INFO GRT-0023] Grid origin: (0, 0)
[INFO GRT-0043] No OR_DEFAULT vias defined.
[INFO GRT-0088] Layer metal1  Track-Pitch = 0.4000  line-2-Via Pitch: 2.1200
[INFO GRT-0088] Layer metal2  Track-Pitch = 0.4000  line-2-Via Pitch: 2.1200
[INFO GRT-0088] Layer metal3  Track-Pitch = 0.4000  line-2-Via Pitch: 2.1200
[INFO GRT-0088] Layer metal4  Track-Pitch = 0.4000  line-2-Via Pitch: 2.2200
[INFO GRT-0088] Layer metal5  Track-Pitch = 0.8000  line-2-Via Pitch: 2.3200
[INFO GRT-0088] Layer metal6  Track-Pitch = 0.8000  line-2-Via Pitch: 2.3200
[INFO GRT-0019] Found 0 clock nets.
[INFO GRT-0001] Minimum degree: 2
[INFO GRT-0002] Maximum degree: 6963
[INFO GRT-0003] Macros: 7
[INFO GRT-0043] No OR_DEFAULT vias defined.
[INFO GRT-0004] Blockages: 776492

[INFO GRT-0053] Routing resources analysis:
          Routing      Original      Derated      Resource
Layer     Direction    Resources     Resources    Reduction (%)
---------------------------------------------------------------
metal1     Horizontal      54855          5081          90.74%
metal2     Vertical        54855         34481          37.14%
metal3     Horizontal      54855         34710          36.72%
metal4     Vertical        54855         35947          34.47%
metal5     Horizontal      52470         35388          32.56%
metal6     Vertical        52470         35170          32.97%
---------------------------------------------------------------

[INFO GRT-0101] Running extra iterations to remove overflow.
[WARNING GRT-0170] Net lx1/lbc1/LBC1/data/n85: Invalid index for position (-362600, 129400). Net degree: 104.
[WARNING GRT-0153] Net lx1/lbc1/LBC1/data/n85 has errors during updateRouteType2.
[INFO GRT-0103] Extra Run for hard benchmark.
[INFO GRT-0197] Via related to pin nodes: 331871
[INFO GRT-0198] Via related Steiner nodes: 17164
[INFO GRT-0199] Via filling finished.
[INFO GRT-0111] Final number of vias: 511783
[INFO GRT-0112] Final usage 3D: 1886470
[WARNING GRT-0115] Global routing finished with overflow.

[INFO GRT-0096] Final congestion report:
Layer         Resource        Demand        Usage (%)    Max H / Max V / Total Overflow
---------------------------------------------------------------------------------------
metal1            5081         53033         1043.75%            12 /  5 / 49443
metal2           34481         84001          243.62%             4 / 12 / 51518
metal3           34710         52834          152.22%            10 /  1 / 19799
metal4           35947         57522          160.02%             2 /  6 / 23561
metal5           35388         55738          157.51%             7 /  2 / 21266
metal6           35170         47993          136.46%             2 /  6 / 14417
---------------------------------------------------------------------------------------
Total           180777        351121          194.23%            37 / 32 / 180004

[INFO GRT-0018] Total wirelength: 3024996 um
[INFO GRT-0014] Routed nets: 33760

Let's start at the beginning.  If I just load the lef & def provided I see things are already place somewhat strangely:

Is this intended?  If the design is already placed, why run placement again?

The web pages with regard to the 2004 iccad contest seem to be lost in time.
All I can say is that this testcase came with lef/def and a sedsm command file.
No result/log files, so I can only assume that the diearea/pin placement worked.

90% utilization for a 9 track M1 rail stdcell lib with 6 layers of metal is definitely pushing what’s doable, or just not possible. In that case the def is bogus and one needs a different diesize.

The def stdcell placement is obviously illegal and the default macro placement clearly not good either. An mpl placement puts the large M5 OBS ram on the right edge, but they need a halo as to not block the M3 IO access and avoid stdcells underneath their power ring. And a lot of spacing to allow access to the pins on their bottom.
My main question is why even with bad macro placement RUDY is looking routable, but completely different from the groute map, which is showing that the placement is completely unroutable.
The sedsm script does run
read lef
read def
automatic macro placement
stdcell placement
routing
defout

Where did you get this data?

I agree the rudy diff needs looking at.

I probably downloaded the data at the time of the contest.

sedsm stands for "silicon ensemble deep sub micron", the name of cadence's P&R tool at the time. The iccad04 testcase contains a script that run's qplace and wroute.
This is a 6 layer testcase where the ram's are only blocked up to M3, and there is no power routing in the diearea.
Yes, density looks high by today's standards, but back then this must have been routable. It was part of the iccad04 testcases. I didn't make this thing up.
There is a huge discrepancy between RUDY and global route heat maps. The ram placement could very likely be improved upon, macro_placement needs to use halo's.
There is a general question whether heat map's should display the usage of "overall" routing resources or "available" routing resources. When you have ram's blocking M1/M2/M3 but M4/M5/M6 are available, there are 2 layers less resources then in the stdcell areas, but if M4/M5/M6 routing resources are only used sparingly, the ram areas shouldn't show up in dark red, no ?

Hi @stefanottili. Could explain how exactly you generated this images? You read an ODB file?
If I run the design from scratch I get a different heatmap.

Hi @gudeh,
I had to modify the lef, changing the 3 ram's CLASS RING to CLASS BLOCK.
15531:  CLASS BLOCK ;
19167:  CLASS BLOCK ;
21165:  CLASS BLOCK ;

The first pictures are from "macro_placement", which moved the ram's to the right.
global_placement will move the ram's too, so I've changed their status to + FIXED in the def to keep their original placement. This is shown with the second set of pictures.
32912:-	ICACHE_INST0/SRAM	tsyncram_512x32	+	FIXED	(	-304400	148000	)	N	;
32913:-	ICACHE_TAG0/SRAM	rf_128x22	+	FIXED	(	-119240	-42800	)	N	;
32914:-	DCACHE_DATA/SRAM	tsyncram_512x32	+	FIXED	(	-169000	-467600	)	N	;
32915:-	DCACHE_TAG/SRAM	rf_128x22	+	FIXED	(	-15720	-42800	)	N	;
32916:-	DRAM_DATA/SRAM	tsyncram_512x32	+	FIXED	(	87800	148000	)	N	;
32917:-	IRAM_DATA/SRAM	tsyncram_512x32	+	FIXED	(	87800	-39200	)	N	;
32918:-	IRAM_VALID/SRAM	rf_8x32	+	FIXED	(	-201220	32800	)	N	;

read_lef lef/risc2.lef.gz
read_def def/risc2.def.gz
#rtl_macro_placer
#macro_placement
global_placement -density 0.95
#detailed_placement
#global_route -verbose
#detailed_route

I started detailed route 10 hours ago with the + FIXED original ram placement and it's now at the 58th iteration with 42 Metal1/Metal2 violations. So it looks as if RUDY is more indicative of the routeability of this design than groute.
Is there any way to interrupt this (ctrl-c) and keep the routing/error markers ?
If not this leads to two more feature requests. I'll file them if you agree with them ;-)

update the view after each detailed route iteration with error markers to show the progress
create a big "stop" button in the gui to be able to stop detailed routing and keep the progress made so far.

With the original + PLACED macro's I've also tried
a) rtl_macro_placer, which coredumps (likely due to missing verilog netlist) and
b) macro_placement, which places all rams to the right (first two pics).
It looks to me as if global_placement might actually provide the best hint at macro placement, even though it placed one of the large rams at the bottom with the pins facing down, should flip the ram to get pins to the top.
To be tried next.

detailed_route has -drc_report_iter_step to report periodically.  ORFS sets it to 5 by default.
Stop is the GUI is something I've planned to do for a long time so feel free to open an issue.

Well, the groute map is completely wrong, the rudy map looks valid.
Looking at where detailed route will fail, I limited the # of iteration to 5,
Looking at the errors after the 5th iteration, groute rather goes thru M4 OBS both with horizontal and vertical routing then using the available M6 (the large ram's have OBS from M1-M5).
If the global router see's everything as equally congested, it has limited choices of what to do.

The global placer must have some notion of routing congestion, is there any way to visualize that ?
Is there a way to increase the macro OBS violation cost for groute ?

Keep in mind that after ~10 hours of detailed route, the # of M4 shorts was down to ~1000.
global_route -allow_congestion
detailed_route -drc_report_iter_step 1 -droute_end_iter 5

[INFO GRT-0096] Final congestion report:
Layer         Resource        Demand        Usage (%)    Max H / Max V / Total Overflow
---------------------------------------------------------------------------------------
metal1            4954         61139         1234.13%            15 /  4 / 57518
metal2           34480         92459          268.15%             4 / 12 / 59581
metal3           34704         53684          154.69%             8 /  2 / 20594
metal4           35947         59531          165.61%             2 / 10 / 25335
metal5           35388         57378          162.14%             7 /  2 / 22762
metal6           35170         48978          139.26%             2 /  5 / 15318
---------------------------------------------------------------------------------------
Total           180643        373169          206.58%            38 / 35 / 201108

[INFO DRT-0195] Start 5th optimization iteration.
...
[INFO DRT-0199]   Number of violations = 33443.
Viol/Layer      metal1     V1 metal2     V2 metal3     V3 metal4     VL metal5     VQ metal6
Cut Spacing          0    223      0     38      0     13      0     22      0      0      0
Metal Spacing       15      0    443      0     35      0     71      0     15      0     13
NS Metal             0      0      0      0      1      0      4      0      0      0      0
Recheck              0      0      0      0      0      0     10      0      0      0      6
Short              106      1   1424      8    320      3  29499     98    329     10    729
SpacingRange         0      0      2      0      2      0      0      0      3      0      0
[INFO DRT-0267] cpu time = 00:26:09, elapsed time = 00:05:30, memory = 4918.47 (MB), peak = 4998.53 (MB)
Total wire length = 2334478 um.
Total wire length on LAYER metal1 = 33168 um.
Total wire length on LAYER metal2 = 546802 um.
Total wire length on LAYER metal3 = 671212 um.
Total wire length on LAYER metal4 = 592058 um.
Total wire length on LAYER metal5 = 269319 um.
Total wire length on LAYER metal6 = 221917 um.
Total number of vias = 367893.

This starts with a bad macro placement.
Which in turn gives the global placement problems. There doesn't seem to be a way to visualize it's view of congestion.
The global router congestion calculation seems off, especially with regard to large rams obstructing M1-M5.
And the detailed router then gives it all to fix the mess the previous steps caused.

Ok, ... let's take the macro's out of the equation. (It's still a good mpl/global_placement testcase).
This way isn't a mostly "why is the groute off by so much" ?
Let's also move the metal2 pins to metal4, since the global placement puts stdcells right beside the pins and this can't be routed later. (I still would want gpl's view of congestion to be visualized, a "placement congestion routing map")
risc2.noram.def.gz
[ERROR GRT-0118] Routing congestion too high. Check the congestion heatmap in the GUI.
[ERROR GUI-0070] Error: risc2.or, 9 GRT-0118

Rudy doesn't seem to correctly display congestion at the top three rows and on the right at a similar distance.
[INFO DRT-0195] Start 4th optimization iteration.
    Completing 10% with 275 violations.
...
[INFO DRT-0199]   Number of violations = 160.
Viol/Layer          V1 metal2
Cut Spacing        159      0
Short                0      1

it took detailed route another 30 iterations to get rid of these V1 cut spacing errors, half of the time iterating on the remaining 8 violations. Better pin access should be able to avoid all these iterations.

So groute map is wrong, without macros this should be easily routable.

The difference between grt and rudy still look awkward. @eder-matheus, do you think this might be because of the die coordinates origin with negative values, from #5284 ? I see nothing extraordinary during rudy calculations.

risc2.noram.def.gz
The RUDY has a gap of 2 "blocks" on the right/top, whereas on left/bottom it goes right up to the die area.

risc2.noram.def.gz
The RUDY map has a gap of 2 "blocks" on the right/top, whereas on left/bottom it goes right up to the die area.

The last line/row of the grid is greater than the other ones for die areas that aren't multiples of the grid size. So the gap you see is because these regions have more resources than the others, leading to less congestion.

I don’t quite buy that argument.
or doesn’t the grid extend all the way to the pins ?
Next to the pins on the top there is a pretty uniform stdcell density and you have the additional resource requirement of the pin connection, but you go from a rows of “green, yellow, orange and red” congestion indicators to two rows of “no congestion to see here at all”.
Doesn’t add up to my eye.

if anything should be read then it should be the area right below the pins on the top.
Way more routing in that are then the red area's right below.

Hmm, something doesn't quite look right to me.
Next to the pins on the top there is a pretty uniform stdcell density and you have the additional resource requirement of the metal4 pin connection, but the rudy map goes from a rows of “green, yellow, orange and red” congestion indicators to a by the looks of it two row hight of “no congestion to see here at all”.
DIEAREA ( -479600 -479600 ) (<PHONE_NUMBER>00 ) ; GCELLGRID Y -479600 DO 240 STEP 4000 ;
240 * 4000 - 479600 = 480400, so the last Y GCELL is slightly too tall for the top diearea. Is it coorect to assume that there GCELL's on top is of size 4000x7600 ?
So this box is bigger then the one below, but it will have "the same density of routing requirements"
Is there a way to dump the rudy box + it's density info into a text file to be able check the coordinates and requirements ?

I understand your point, and I think RUDY might have a bug. You can check the density value in the GUI; double-click on the Congestion Map (RUDY) option and it will show some options. "Show Numbers" will draw the density values in each grid tile, and in these larger tiles, the density is zero. Some further investigation of the RUDY code will be required to understand why this is happening.
Here's a figure with the density data. The zeros there are definitely wrong.

Thanks for confirming.
By the looks of it, groute suffers from something similar.
And now one more datapoint with regard to groute/rudy maps:
I've saved the routed.def, started a new openroad session, read_lef/read_def routed.def/global_route and then display both groute and rudy maps. They look very different from the groute/rudy maps at the end of routing ...

and they also look very different when reading routed.db and displaying groute/rudy map.

and for the last experiment of the day, I've move the centered floorpan to have 0,0 at the lower left.
risc2.first_quadrant.noram.def.gz

the rudy map gaps on the right/top don't look so pronounced any more but are still clearly visible
the weird vertical rudy congestion below the top pins is gone
Global route spreads out the routing much more, but it's congestion map is still dark red
Detail route still wastes 30 iteration on fixing V1 cut spacing errors

a) Both global route and rudy behave differently when the floorpan is centered vs. the left/bottom is at 0,0
b) Both rudy and global route map violently disagree about the amount of congestion.

risc2.first_quadrant.noram.def.gz took out the issue of macro placement + 95% utilization and the issue of OR not handling centered floorplans.
As of the 10th Aug 2024, running the RISC2 FARADAY_ICCAD04Bench led to a couple of bug fixes, but the main problems remain.
This TSMC 180 4+2 layer testcase routes in 10 iterations, despite the GRT's resources computation being completely off and showing an unroutable design.
ispd24 with a metal stack using different layer pitches doesn't show this behavior, so why is OR grt struggling with this tech ?
 131 [INFO GRT-0088] Layer metal1  Track-Pitch = 0.4000  line-2-Via Pitch: 2.1200
 132 [INFO GRT-0088] Layer metal2  Track-Pitch = 0.4000  line-2-Via Pitch: 2.1200
 133 [INFO GRT-0088] Layer metal3  Track-Pitch = 0.4000  line-2-Via Pitch: 2.1200
 134 [INFO GRT-0088] Layer metal4  Track-Pitch = 0.4000  line-2-Via Pitch: 2.2200
 135 [INFO GRT-0088] Layer metal5  Track-Pitch = 0.8000  line-2-Via Pitch: 2.3200
 136 [INFO GRT-0088] Layer metal6  Track-Pitch = 0.8000  line-2-Via Pitch: 2.3200

Using only 5 layer, placement densitiy to 0.8 and reducing the routing resources by 30% routes this design much quicker in just 6 drt iterations. The global route resource map remains off.
 164 [INFO GRT-0096] Final congestion report:
 165 Layer         Resource        Demand        Usage (%)    Max H / Max V / Total Overflow
 166 ---------------------------------------------------------------------------------------
 167 metal1           10338         60369          583.95%            12 /  2 / 54780
 168 metal2           28124         96723          343.92%             2 / 13 / 71862
 169 metal3           28124         36713          130.54%             6 /  2 / 12886
 170 metal4           28124         41323          146.93%             2 /  6 / 16726
 171 metal5           26386         31055          117.69%             4 /  1 / 8855
 172 ---------------------------------------------------------------------------------------
 173 Total           121096        266183          219.81%            26 / 24 / 165109

read_lef lef/risc2.lef.gz
#read_def def/risc2.def.gz
read_def def/risc2.first_quadrant.noram.def.gz
set_routing_layers -signal metal1-metal5
set_global_routing_layer_adjustment * 0.3
#macro_placement
#global_placement -density 0.95
global_placement -density 0.8
detailed_placement
global_route -allow_congestion -verbose
detailed_route -drc_report_iter_step 1

The centered die use case is a very low priority as OR never generates such.

this is not about the centered case any more, the last pictures/groute overflow numbers are with
"...moved the centered floorpan to have 0,0 at the lower left"

It's hard to follow the new problem with many comments here. Could you attach a new reproducible with all the files and scripts needed to run the problem? If the problem is not about the centered case, perhaps creating a separate issue would make sense.

see https://github.com/The-OpenROAD-Project/OpenROAD/issues/5548 for the "first_quadrant.noram" testcase.

closed and replaced by https://github.com/The-OpenROAD-Project/OpenROAD/issues/5557


Rate limit is sometimes ignored

Notice: Please don't file a bug report for questions on how to use ZMap, etc. Check out our Resources section for FAQ and feel free to chime in on Github Discussions if you're still unclear and we'll be happy to assist!

Describe the bug
ZMap intermittently ignores the specified --rate limit during scans. This issue can occur at any point during a scan and persists until the scan is completed.
Below is a screenshot illustrating this behavior.

All 10 scanners in this example were invoked with --rate=24000 (See the full command used below).
Unfortunately, no logs have been identified that indicate an issue.
CLI Arguments
Please paste your ZMap invocation below
/usr/local/sbin/zmap --quiet --target-port=XXX --rate=24000 --seed=9 --allowlist-file=/mnt/scanning/allowlist.conf --blocklist-file=/mnt/scanning/blocklist.conf --metadata-file=/mnt/scanning/scan.metadata.json --status-updates-file=/mnt/scanning/scan.status.json --probe-module=tcp_synscan --shards=10 --shard=3 --user-metadata={"scan_id":"xx", "scanner_id":"xx"} --sender-threads=1 --output-module=json --output-fields=timestamp_str,daddr,dport,saddr,sport,success --output-filter=success=1 && repeat=0

Example Target IP
Please list any IPs that can be used for testing behavior.
This bug seems to occur when running
Expected behavior
The --rate limit should be respected for the entire duration of the scan.
Environment:

OS: Debian 12.8
Version: 4.2.0, also tested to build from HEAD (ae3fcbc) with same result.

Additional context
It may not be related but on occasion (approximately 5% of the time), scans fail to complete, and ZMap does not exit as expected. When this happens, logs repeatedly display the message: "Dropping X packets in the last second" roughly once every 5–10 seconds. If helpful, I can file a separate bug report for this issue. It seems similar to #868.

Hey @huxcrux, thanks for taking to the time to file this detailed issue. Let me take a look and see what's going on here.

Hey @huxcrux, took a look at this today and wasn't able to reproduce. I wanted to describe my setup and discuss how yours may be different and ask a few questions to help me troubleshoot.
On a single Ubuntu 24.04 VM, I ran 10 ZMap shards in parallel with:
sudo ./src/zmap --target-port=13579 --rate=24000 --seed=9 --metadata-file=meta.json --output-file=out --probe-module=tcp_synscan --shards=10 --shard=X --cores=X --user-metadata='{"scan_id":"xx", "scanner_id":"xx"}' --sender-threads=1 --output-module=json --output-fields=timestamp_str,daddr,dport,saddr,sport,success --output-filter=success=1 --status-updates-file=zmap-status-4.txt && repeat=0

I ran these for an hour and logged per-second the number of packets sent/received by the VM. Nothing else was running so ZMap's traffic should dominate the utilized traffic.
My logs show the overall mean traffic volume of the sum of the 10 invocations stayed very close to 240k (as it should). Additionally, the max tx packets is just 5,700 over the --rate, and this could be explained by the OS doing some background activity.
$ head Downloads/pps.log
2024-11-18 19:37:38;tx 241670 pkts/s;rx 11202 pkts/s; on interface enp0s8
2024-11-18 19:37:39;tx 241343 pkts/s;rx 10978 pkts/s; on interface enp0s8
...

~ ⌚ 22:28:23
$ cat Downloads/pps.log | cut -d ';' -f 2 | cut -d ' ' -f 2 | st
N	min	max	sum	mean	stddev
3610	3	245721	8.70426e+08	241115	7415.29

So had a few questions:

Do you see this behavior every time you run this scan/how easily reproducible is it on your machine?
Are your ZMap shards on separate VMs or all on the same one?
How are you gathering your per-shard PPS?
Do you see the same behavior with v3.0.0

I think my next step will be to try to reproduce with Debian 12.8 and see if OS makes a difference.

Hey @huxcrux, took a look at this today and wasn't able to reproduce. I wanted to describe my setup and discuss how yours may be different and ask a few questions to help me troubleshoot.
On a single Ubuntu 24.04 VM, I ran 10 ZMap shards in parallel with:
sudo ./src/zmap --target-port=13579 --rate=24000 --seed=9 --metadata-file=meta.json --output-file=out --probe-module=tcp_synscan --shards=10 --shard=X --cores=X --user-metadata='{"scan_id":"xx", "scanner_id":"xx"}' --sender-threads=1 --output-module=json --output-fields=timestamp_str,daddr,dport,saddr,sport,success --output-filter=success=1 --status-updates-file=zmap-status-4.txt && repeat=0

I ran these for an hour and logged per-second the number of packets sent/received by the VM. Nothing else was running so ZMap's traffic should dominate the utilized traffic.
My logs show the overall mean traffic volume of the sum of the 10 invocations stayed very close to 240k (as it should). Additionally, the max tx packets is just 5,700 over the --rate, and this could be explained by the OS doing some background activity.
$ head Downloads/pps.log
2024-11-18 19:37:38;tx 241670 pkts/s;rx 11202 pkts/s; on interface enp0s8
2024-11-18 19:37:39;tx 241343 pkts/s;rx 10978 pkts/s; on interface enp0s8
...

~ ⌚ 22:28:23
$ cat Downloads/pps.log | cut -d ';' -f 2 | cut -d ' ' -f 2 | st
N	min	max	sum	mean	stddev
3610	3	245721	8.70426e+08	241115	7415.29

So had a few questions:
* Do you see this behavior every time you run this scan/how easily reproducible is it on your machine?

* Are your ZMap shards on separate VMs or all on the same one?

* How are you gathering your per-shard PPS?

* Do you see the same behavior with [v3.0.0](https://github.com/zmap/zmap/releases/tag/v3.0.0)

I think my next step will be to try to reproduce with Debian 12.8 and see if OS makes a difference.

So had a few questions:
* Do you see this behavior every time you run this scan/how easily reproducible is it on your machine?
- Around 50-60% of my scans result in rate being ignored. Most of the times it occurs within the first 30 minutes however it could occur later in as well.

* Are your ZMap shards on separate VMs or all on the same one?
- I run ZMap on 10 different VMs, All VMs are KVM based and have an public IP on one of its interface.

* How are you gathering your per-shard PPS?
- I use Prometheus node-exporter for metrics on all scanner meaning the PPS reported includes other system services, however since the machines are dedicated scanners other traffic isn't a significant amount (when ZMap is not running the machines report around 5PPS with spikes to ~20).

* Do you see the same behavior with [v3.0.0](https://github.com/zmap/zmap/releases/tag/v3.0.0)
- I will test running ZMAP3 and keep you posted on the result.

I also have an update regarding the dropped logs. This was my scanners just being to slow writing to disk. I buffer the output in memory prior to writing to disk and I no longer see dropped packages meaning the only Issue we currently se is the rate being ignored.
Thanks alot for taking this bug report seriously and just let me know in case there is anything I can do helping the investigation.

Another update.
We tested downgrade and to isolate the zmap process (we usually run it together with our log shipper and zgrab). I was able to still trigger the issue every time. However when running version 3.0.0 we had around 10-15% of scans ignoring the rate limit. And when running 4.2.0 we have around 50%.
I found that the metrics written by ZMap during a run contains an average rate and duration I can confirm this is ZMap sending more packages than instructed.
We collect the files once a run is completed and therefore i do not have any from yesterdays testing (test where canceled mid run to cram in more tests). I do however have this data stored for all runs we have done during the last couple of months.
Example from a run on the 18th (2 days ago): https://gist.github.com/huxcrux/1c32c1831863513ee71d0706f24324c0
We will run some tests on Ubuntu 24.04 today since you mentioned it worked during your testing (we are currently reinstalling half of our scanner VMs and I will share our findings in a few hours.

This is great info, thank you. Very useful that in reproducing I can just trust the ZMap metadata. I can also try having more shards at once to give better chance at reproducing on my end. I'm a bit overloaded on another project so might be a few days before I can get back to this. One thing you can try as well to unblock yourself is using -B to set bandwidth limitations if you're just trying to limit bandwidth utilization by each scanner, in case that's useful to your use case.

I tested with Ubuntu 24.04 and see the same behavior there.
We have however seen a change in behavior when running on "dedicated" cores. Meaning we pin VMs to specific cores. I need to look more into this and will share more details during the upcoming days.
I can do a few tests tomorrow using the bandwidth limiter to se if we get another behavoiur.

Hey @huxcrux, wanted to give you an update on this.
I took a look at this today and after trying several things, just wasn't able to reproduce. Bit of a disclaimer, my setup is a little different: I'm using a single Ubuntu VM and pinning multiple ZMap invocations to different cores.

I was wondering if either the -B bandwidth option or core pinning had uncovered any new behavior for you?
Also, wondering about the specs on these VMs/Hosts you're using (cores, RAM, etc.). Without core-pinning, it'd be reasonable if you were seeing less send rate if there was contention for running on a given core. But I can't explain seeing more.

Thanks and really appreciate being so helpful in this process of getting to the bottom of this.

I've also added a couple debugging logs to this branch: phillip/903-debugging-logs. This will add logs around the adaptive rate-limiting and print variables + exit if our rate spikes above 10x the specified --rate. Apologies for asking you to run custom code, but not sure how to reproduce on my side so this would be really helpful.
If you could, run to compile: git pull && git checkout phillip/903-debugging-logs && cmake . && make -j4
Run with - sudo ./src/zmap -p 80 -r 24000 --verbosity=5 --log-file=log -T 1 --output-file=/dev/null --seed=9 --probe-module=tcp_synscan --shards=10 --shard=3.
And then send me the log file of the machine that has a rate spike. I ran this test locally on my machine and the log file accumulated about 220 kB/min so the log file shouldn't take up much space on your machine.
Thanks again!

Hello again,
I tested a couple of runs with your test branch and every time the rate limit is ignored the debug output is stopped.
For simplicity I removed --log-file=log to get all output to stdout. Here is the output from one of my runs: https://gist.github.com/huxcrux/87cc7ccbc52c2fc441db0ba50961af69
The same result seems to be the case when running with bandwidth limit instead of rate: https://gist.github.com/huxcrux/7a960ea11c4850a228af24a7ab2be268
It may be related or not however when dedicating cores to my VMs I se the problem less frequently. It goes from ignoring the rate-limit around 50% of times to maybe 1%.
I have tested a bit more with different kernels and Switching between Debian and Ubuntu and the change in behavior between different combinations seems to be withing error margin.
Also regarding the hosts we are currently running KVM based VMs. The scanner machines have 4 cores and 8 Gib of ram (I have tested to bump this to 8/16 just to make sure). We use virtio based networking and virtio scsi controller for storage. The underlying hosts are running REHL OS with Intel Xeon Platinum 81** CPUs. VMs are always assigned cores from the same CPU to avoid potential NUMA related problems.
When pinning cores meaning each machine gets "dedicated" cores the failure rate seems to drop to around 1-2% while running without CPU pinning we get around 50%.

Hey @huxcrux, thanks for being honestly amazing in assisting with the debugging process, those logs were invaluable.
I'm almost positive I've found the bug and implemented a fix in phillip/903-force-delay-nonzero. PR is here.
I attached details of reproducing the same issue in the PR and after our internal review we can get this fix merged into main.


[FEATURE]: Make user aware that some link redirects to another identical GitHub repository

Describe the solution you'd like
The solution should include the following:

Links to other repositories in each of the karavi observability repositories open those links in another tab
Links to external sites from the karavi observability repositories open those links in another tab
Links within the same repository continue to open within the same tab
Links on Karavi Observability repositories with external links should be updated to open in a new tab.

Describe alternatives you've considered
I tried right-clicking and selecting the "Open link in new tab" option but  this is time-consuming and nonintuitive within Github
Additional context
Since the design and look of all the karavi observability repositories are exactly the same, It's very confusing to click a link and be get taken to another repo.

At the moment, GitHub Repos uses  github-flavored-markdown as the markdown engine. For security concerns, This engine automatically strips out target="_blank"  in the anchor during HTML sanitization. Nonetheless, Github pages use kramdown  which has the following syntax go{:target="_blank"}  that will allow us to automatically open links on new tabs. Hopefully, we hope GitHub Repos adopts the same engine soon.
Unless Github Repo updates its markdown engine to one that supports the target, we may not be able to add this feature. Thus this request will be CLOSED pending this GitHub update!

At the moment, GitHub Repos uses  github-flavored-markdown as the markdown engine. For security concerns, This engine automatically strips out target="_blank"  in the anchor during HTML sanitization. Nonetheless, Github pages use kramdown  which has the following syntax go{:target="_blank"}  that will allow us to automatically open links on new tabs. Hopefully, we hope GitHub Repos adopts the same engine soon.
Unless Github Repo updates its markdown engine to one that supports the target, we may not be able to add this feature. Thus this request will be CLOSED pending this GitHub update!


[enhancement] Utilize the capability of distributed tracing to support traffic recording

We currently have an idea to implement traffic recording functionality using a tracing system. Why can traffic recording be achieved using tracing?
Firstly, the technical solution for traffic recording involves logging the incoming and outgoing parameters of requests, as well as request headers, into log files within the business containers. This information is then collected by a log agent. This approach bears significant resemblance to the current architecture of tracing, and they share certain commonalities, indicating that some capabilities of tracing can be reused.
 This will enable Opentelemetry to utilize Nacos configuration functionality. In the trac-etl-manager, capabilities will be provided to push configurations to Nacos.

Traffic interception involves how to intercept and collect traffics. Generally, available interception locations include：

NIC driver
protocol stack
syscall
language standard library
communication framework (eg: rpc framework)
business code
For the above interception positions, the further down, the more traffic characteristics there are, and the more flexible the interception methods are.

Trace's probe's existing burying points and code enhancement capabilities can implement flexible traffic interception at the communication framework layer and business code layer, and reduce the impact on business logic in a certain flow chart, making it easier to implement record strategies. Taking RPC traffic as an example, an independent recording filter is implemented through the code enhancement of the Trace mark, and it can be conveniently determined through configuration whether to record this request or not


Use --config in ecosystem checks

Our ecosystem checks manually overwrite the pyproject.toml file to perform some overrides but now that we have --config support we should use that instead.

Yeah basically I think we can replace this whole blurb
https://github.com/astral-sh/ruff/blob/c269c1a7063bb27e75dc81c2a598eab13aa8b754/python/ruff-ecosystem/ruff_ecosystem/projects.py#L89-L165
with appending a --config option at
https://github.com/astral-sh/ruff/blob/main/python/ruff-ecosystem/ruff_ecosystem/projects.py#L204-L225

@zanieb I started looking into this, and I have a few things I would like to ask:
1. How to unset / restore to default using --config
One of the overrides is required-version = null. Currently this just removes the entry from the toml file, however I don't know how to do that from the CLI with --config (since toml does not have null).
2. Where to make the changes
In order to keep CheckOptions and ConfigOverrides separated, I'm thinking of adding the overrides to the list returned by CheckOptions.to_ruff_args():
    ruff_args = options.to_ruff_args()
    ruff_args.append(overrides)

But I could also do it inside of the to_ruff_args method by changing its signature to the one below.

    def to_ruff_args(
        self,
        config_overrides: ConfigOverrides | None = None,
    ) -> list[str]:

What do you think about it ?

I'm not sure, I'd have to check. @AlexWaygood added that feature, I presume there's a way to unset keys.
I don't have a strong preference. options.to_ruff_args() + overrides.to_ruff_args() seems reasonable to start.


Add test_create_user_backend().

This tests provides full line coverage on the
create_user_backend view.

Automated message from Dropbox CLA bot
@showell, it looks like you've already signed the Dropbox CLA. Thanks!

Looks great!  I'll merge assuming this passes tests.  Thanks @showell!

Merged!


Posting and deleting a message in an old topic leaves the topic higher than it should be in the stream topic list

If you post a message in a topic that's long been buried in the topic list to the left, and then delete your new message, the topic stays on top of the list, even if the latest message is months old.

This is a tricky issue; the problem is that we order things in the sidebars based on the latest message ID, but for performance reasons don't store every single message ID in every topic in the frontend.  As a result, when the latest message gets deleted, the frontend doesn't have an easy thing it can do to compute the second-latest message ID.
I had thought our solution was generally to just delete the topic's data and recompute, however, so something is wrong here.
@amanagr can you investigate?

Yes, on it. @zulipbot claim


Create application that allows to display spectrum

In this project we have an 'App'
https://github.com/zakodium-oss/analysis-ui-components/blob/main/src/app/App.tsx
That can be started using npm run vite
The goal is to slowly improve this example to implement various features.
In the future we are expecting to have 3 Contexts:

DataContext containing all the results of the experiments
LayoutContext containing the arrangement of the different modules
TempContext some information that will never be saved but are useful when working with the application

Currently we will add a DataContext
analyses: {
  ir: [{
    data
    info: {
      filename: '',
      dirname: '',
    }
  }]
}

The goal is to be able to drag and drop the following file.
50453.jdx.zip
This file will then be parsed by jcampconverter
import {convert} from 'jcampconverter';

const data = convert(arrayBuffer of the jcamp).flatten[0];

And we will push the result to the state in analysis.ir

could you please explain:
analyses: {
  ir: [{
    data
    info: {
      filename: '',
      dirname: '',
    }
  }]
}

is this context structure?

i understand that i have to use the dropzone to get data from file and put it in dataContext, right?

I edited part of the issue and renamed some properties as well.
Yes after drag / drop the json file you should add an entry in 'measurements.irs' and the DataState should be
measurements: {
  irs: [{
    data: {x: [...], y:[...]},
    info: {
      filename: 'ir-spectra.jdx',
      path: '', // the content of [File.webkitRelativePath](https://developer.mozilla.org/en-US/docs/Web/API/File/webkitRelativePath)
    }
  }]
}


107 libraries downgrade

https://github.com/Visual-Regression-Tracker/Visual-Regression-Tracker/issues/107

Codecov Report

Merging #23 into master will not change coverage.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##              master       #23   +/-   ##
===========================================
  Coverage     100.00%   100.00%
  Complexity        19        19
===========================================
  Files              3         3
  Lines             86        86
  Branches           5         5
===========================================
  Hits              86        86

Impacted Files
Coverage Δ
Complexity Δ

...sion_tracker/sdk_java/VisualRegressionTracker.java
100.00% <100.00%> (ø)
17.00 <0.00> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b83a4ed...158da8c. Read the comment docs.


Bind Trap::i32_exit_status in C API

This allows embedders to take a different action when a WASI program
exits depending on whether it exited with an exit status or exited with
a trap.
cc bytecodealliance/wasmtime-py#30

This looks good to me. In the future we'll have other kinds of exit values, so this API may need to be generalized, but this works for now.


Edit recyclerview items

Hello! How can access list mapped to Carpaccio and notifiydatasetchanged??? Thanks

Hi, you can use carpaccio.getMappedList("user")if you used mapList("user")
I'm doin the wiki, so don't hesitate to send me questions, if it's implemented I'll put the answer here, else I'll implement it

if you want to notify dataset changed `carpaccio.getAdapter("user"). notifiydatasetchanged``

Ok, thanks! Another problem: i have edited an user of an arraylist that i have passed to maplist. Before doing carpaccio.getAdapter("user").notifyDataSetChanged(), i have to set the edited user in the correct position of the list. Is there a method to do that o i have to cycling entire list and do a comparison by POJO attribute to find the correct position? Thanks

You can use list.indexOf
Le 4 août 2015 09:07, "kegita"<EMAIL_ADDRESS>a écrit :

Ok, thanks! Another problem: i have edited an user of an arraylist that i
have passed to maplist. Before doing
carpaccio.getAdapter("user").notifyDataSetChanged(), i have to set the
edited user in the correct position of the list. Is there a method to do
that o i have to cycling entire list and do a comparison by POJO attribute
to find the correct position? Thanks
—
Reply to this email directly or view it on GitHub
https://github.com/florent37/Carpaccio/issues/5#issuecomment-127503727.


folder creating

when running on the same folder twice, the software remove everything from the existing folder in the second run, even though there are no files with the same name.

What commands are you using?

I use ```
python auto-editor.py "media_folder_path"

I tested running that command twice and I can't replicate that error. See if the error still is there on the latest version of auto-editor.

I'll close this in a few days if no more context is provided. I can't recreate this bug, nor seeming anyone else.


Creates better way or returning data in a familiar format

Supports pagination but returns one item if using get and an array of items if using find even if it has only one item.

@codingfriend1 thanks! Can you resolve conflicts and provide a brief description of what this is doing?

Merge conflicts fixed!

Please merge and describe changes :)

Done.

Whoop! Whoop! Thanks @codingfriend1!

Awesome guys :)


BootGrid Command Buttons, Custom button bootstrap

How can I put custom button (Style Bootstrap) in the row "Command"
Commands

< button type="button" class="btn btn-default" aria-label="Left Align" >

< /button >

When Run the Application don't show me the button in that row. any suggestion?

You should look at this example on the homepage of the plugin: http://www.jquery-bootgrid.com/Examples#command-buttons
The short story is that you need to provide a "Formatter" as a parameter when instantiating the bootgrid class and then using the "onload" event of bootgrid to bind the even handler of those button handle to the appropriate functions.

Do you mean in this part of the JS
formatters: {
"commands": function(column, row)
{
return "< button type="button" class="btn btn-xs btn-default command-edit" data-row-id="" + row.id + ""> " +
"< button type="button" class="btn btn-xs btn-default command-delete" data-row-id="" + row.id + "">";
}
}

Yes. Also, remember to chain the onload handler as follows:
.on("loaded.rs.jquery.bootgrid", function()
{
/* Executes after data is loaded and rendered */
grid.find(".command-edit").on("click", function(e)
{
alert("You pressed edit on row: " + $(this).data("row-id"));
}).end().find(".command-delete").on("click", function(e)
{
alert("You pressed delete on row: " + $(this).data("row-id"));
});
});

@ksidibe @rstaib  don't appears... I new of using bootgrid I have this:
in the HTML

commands

< button type="button" class="btn btn-xs btn-default command-edit" data-row-id=""> < /button>
 < /button>

In the JS:
var grid = $("#grid").bootgrid({
ajax: true,
post: function () {
return {
id: "b0df282a-0d67-40e5-8558-c9e93b7befed"
};
},
url: "/currencies/",
formatters: {
"commands": function (column, row) {
return "<button type="button" class="btn btn-xs btn-default command-edit" data-row-id="" + row.id + ""> " +
"<button type="button" class="btn btn-xs btn-default command-delete" data-row-id="" + row.id + "">";
}
}
}).on("loaded.rs.jquery.bootgrid", function () {
/* Executes after data is loaded and rendered */
grid.find(".command-edit").on("click", function (e) {
alert("You pressed edit on row: " + $(this).data("row-id"));
}).end().find(".command-delete").on("click", function (e) {
alert("You pressed delete on row: " + $(this).data("row-id"));
});
});

You are instantiating bootgrid on the same element twice. Try deleting  data-toggle="bootgrid" data-url="/currencies/" from the html table definition.

@ksidibe  if I remove  data-toggle="bootgrid"  don't show me the records that I get from the database and show me the message "No results found"

When I had that problem, it was because of the route definition. I was using url:"/mysuperurl/". Then I changed it to url:"/mysuperurl" (without the trailing /) and that worked.
Also, you should delete this line:
post: function () {
return {
id: "b0df282a-0d67-40e5-8558-c9e93b7befed"
};
},

@ksidibe  Doesn't work, that frustrating... I do a application in ASP.NET MVC 4, my code is like this:
cshtml:

                    Create Date
                    Commands

                @if (Model != null)
                {

                    foreach (var item in Model)
                    {

                        <<EMAIL_ADDRESS>
                            <button type="button" class="btn btn-xs btn-default command-edit" ></button>
                            <button type="button" class="btn btn-xs btn-default command-delete" ></button>

                    }
                }

JS:
var grid = $("#grid").bootgrid({
ajax: true,
url: "/currencies",
formatters: {
"commands": function (column, row) {
return "<button type="button" class="btn btn-xs btn-default command-edit" data-row-id="" + row.id + ""> " +
"<button type="button" class="btn btn-xs btn-default command-delete" data-row-id="" + row.id + "">";
}
}
}).on("loaded.rs.jquery.bootgrid", function () {
/* Executes after data is loaded and rendered */
grid.find(".command-edit").on("click", function (e) {
alert("You pressed edit on row: " + $(this).data("row-id"));
}).end().find(".command-delete").on("click", function (e) {
alert("You pressed delete on row: " + $(this).data("row-id"));
});
});

There are two issues I can see with you code. First, when using bootgrid, you do not include the  tags in you table. So, all you need is:

Create Date
Commands

So, in you cshtml file, you need to delete:

                      <<EMAIL_ADDRESS>
                            <button type="button" class="btn btn-xs btn-default command-edit" ></button>
                            <button type="button" class="btn btn-xs btn-default command-delete" ></button>

2- Whatever url you are referring to when instantiating the bootgring class, that url must return json. NO HTML. I see that in your csHtml, you are iterating through your model an manually creating the
 elements. You can't do that. Here's what I recommend.
 Option A (if you want to use the same URL to service both requests for html and JSON): In your controller, check to see if the Request is an Ajax Request the you should serialize the model into JSON and return that. If the request is not Ajax then you can fetch the model and pass it the view.
An example of how that can be done. place this at the top of you controller: using System.Web.Script.Serialization;

then add this to the method handling the request:
public ActionResult SomeActionMethod()
{
.
.
.
.
.
   if (Request.IsAjaxRequest())
   {
     var jsonSerialiser = new JavaScriptSerializer();
     var json = jsonSerialiser.Serialize(myListOfModels);
     return json;
   }
.
.
.
.
.

}

Option B: create a completely separate route that only returns JSON, and call that URL instead
 i Here's an example(from the plugin's page) of the data that must be returned by the URL: {
  "current": 1,
  "rowCount": 10,
  "rows": [
    {
      "id": 19,
      "sender"<EMAIL_ADDRESS>      "received": "2014-05-30T22:15:00"
    },
    {
      "id": 14,
      "sender"<EMAIL_ADDRESS>      "received": "2014-05-30T20:15:00"
    },
    ...
  ],
  "total": 1123
}

By the way, one small update. You should set the json key rows to the array of models. You should also set rowCount to modelList.count() and finally properly set the value of total.

@ksidibe  In my controller have this method with the option A
public ActionResult CurrencyIndex()
{
        var currenlist = (from r in context.currency select r);

        if (Request.IsAjaxRequest())
        {
            var jsonSerialiser = new JavaScriptSerializer();
            var json = jsonSerialiser.Serialize(currenlist.ToList());
            return View(json);
        }

        return View(currenlist.ToList());
    }

how in the view get the values model without iterating the model and manually creating the  elements

When servicing a request from bootgrid, you are not supposed to return a view. You are supposed to return a JSON.
Also, I noticed that in your Javascript, your command formatter returns an empty string. You should have:
formatters: {
        "commands": function(column, row)
        {
            return "<button type=\"button\" class=\"btn btn-xs btn-default command-edit\" data-row-id=\"" + row.id + "\"></button> " +
                "<button type=\"button\" class=\"btn btn-xs btn-default command-delete\" data-row-id=\"" + row.id + "\"></button>";
        }
    }

Finally, here's what I would do if I were you. Create a URL that returns this static JSON object:
{
  "current": 1,
  "rowCount": 10,
  "rows": [
    {
      "id": 19,
      "sender"<EMAIL_ADDRESS>      "received": "2014-05-30T22:15:00"
    },
    {
      "id": 14,
      "sender"<EMAIL_ADDRESS>      "received": "2014-05-30T20:15:00"
    },
    ...
  ],
  "total": 1123
}

Make the table work with that static dummy data. Then you can rewrite your controller properly.

Achieve That worked with static dummy data, it is advance. now I did this method
public ActionResult listCurrencyJson() {
var json = "";
if (Request.IsAjaxRequest())
{
var jsonSerialiser = new JavaScriptSerializer();
json = jsonSerialiser.Serialize(context.currency.ToList());
return Json(json, JsonRequestBehavior.AllowGet);
}
return Json(json, JsonRequestBehavior.AllowGet);
}
And in the JS call that method
var grid = $("#grid").bootgrid({
ajax: true,
url: "/Currency/listCurrencyJson",
formatters: {
"commands": function(column, row)
{
return "< button type="button" class="btn btn-default btn-lg delete" data-row-id="" + row.id  + ""> Star" +
"<button type="button" class="btn btn-xs btn-default command-edit" data-row-id="" +  row.id + ""> " +
"<button type="button" class="btn btn-xs btn-default command-delete" data-row-id="" +  row.id + "">< /button>";
}
}
}).on("loaded.rs.jquery.bootgrid", function () {
/* Executes after data is loaded and rendered */
grid.find(".command-edit").on("click", function (e) {
alert("You pressed edit on row: " + $(this).data("row-id"));
}).end().find(".delete").on("click", function (e) {
alert("You pressed delete on row: " + $(this).data("row-id"));
});
});
But when run, give a error, Unhandled exception at line 565, column 9 in http://localhost:58969  /Scripts/jquery.bootgrid.js
0x800a138f - JavaScript runtime error: Unable to get property 'length' of undefined or null reference

Like I mentioned in my previous post, the keys of the should include "rows", "current", "rowCount".
the code should be:
public ActionResult listCurrencyJson() {
var json = "";
if (Request.IsAjaxRequest())
{
var keyValues = new Dictionary<string, Currency>
               {
                   { "rows", context.currency.ToList()},
                   { "rowCount", context.currency.count() },
                   {"current", Request.params.getValue("current")}
               };

JavaScriptSerializer js = new JavaScriptSerializer();
string json = js.Serialize(keyValues);
return Json(json, JsonRequestBehavior.AllowGet);
}
return Json(json, JsonRequestBehavior.AllowGet);
}

Finally, since the issue you are having is not related to the bootgrid plugin, and in order to keep this forrum as useful as possible, I recommend that you post a question on Stackoverflow. You can post the link here and I can try an help you there instead.

@ksidibe  I have error with the dictionary but I open a post in the Stackoverflow for your help
http://stackoverflow.com/questions/28346446/info-from-database-and-transform-in-json-to-show-in-bootgrid-in-the-view

@ksidibe when I serializer the keyValues in the row have DateTime Column and show me in the gird /Date(1422943200000)/ and not the normal format 2/03/2015 12.00.00 PM

Can you post the skeleton of the Currency class?

@ksidibe in the currency class don't have the column of the database that is a DateTime. I get the data of that specific column directly of the context of entity framework

Let's step back for a minute. What data are you trying to display?

@ksidibe  I get data from the context of entity framework, table currency. I want to display the date of the column "CreateDate" that type of that column is DateTime
in keyValues get this result and show me a normal format
CREATE_DATE	{2/3/2015 12:00:00 AM}
but when string json = js.Serialize(keyValues);  the result of the Json is this
"CREATE_DATE":"/Date(1422943200000)/
And the bootgrid show me that result, I want to show the normal format

@pabloso18 Please ask here only questions regarding bootgrid not ASP.Net MVC specific questions and don't mixup too many questions so it is easier for people to help you. For questions regarding ASP.Net MVC please go to Stackoverflow. Thanks!
@ksidibe Thanks for helping @pabloso18

how to make entire row using font-weight:bold ????????? i tired to get search... remember just one row.???


Publish results as job summary

Fixes #264
I took the liberty of enhancing the tests to use absolute paths for files, it's easier to run tests in my IDE.

Well, I think pretty much all comments have been covered 😌
Hope it's ok now 🤞

PR #267 is merged, please rebase and I will review again tomorrow.

Look at this (scroll down to the job summaries): https://github.com/EnricoMi/publish-unit-test-result-action/actions/runs/2306018943

What about this markdown for the job summary (reference comment and check)?
Currently:
4 parse errors, 6 errors, 25 fail, 20 skipped, 228 pass in 39m 21s
26 files      4 errors  23 suites   39m 21s :stopwatch:
279 tests 228 :heavy_check_mark: 20 :zzz: 25 :x: 6 :fire:
450 runs  358 :heavy_check_mark: 57 :zzz: 28 :x: 7 :fire:
Results for commit 9eac4d8f.

With title and one-line summary:
Unit Test Results (Test Files)
4 parse errors, 6 errors, 25 fail, 20 skipped, 228 pass in 39m 21s
26 files      4 errors  23 suites   39m 21s :stopwatch:
279 tests 228 :heavy_check_mark: 20 :zzz: 25 :x: 6 :fire:
450 runs  358 :heavy_check_mark: 57 :zzz: 28 :x: 7 :fire:
Results for commit 9eac4d8f.

Like the comment, not the check:
Unit Test Results (Test Files)
26 files      4 errors  23 suites   39m 21s :stopwatch:
279 tests 228 :heavy_check_mark: 20 :zzz: 25 :x: 6 :fire:
450 runs  358 :heavy_check_mark: 57 :zzz: 28 :x: 7 :fire:
Results for commit 9eac4d8f.

Which one do you prefer? I assume it's about not passing the title to get to the last one (my personal favorite)

Yes, I prefer that one too.

Thanks for the contribution!

As those environment files are a GitHub Actions feature, I have added them to GithubAction class, including tests: #269
I am going to migrate publish_summary to it soon.

This has been released, enjoy!


pad should pad Promise rejections as well

Otherwise rejections can sneak through before the specified delay.

Hmm, this might be tough, as the underlying Promise.all also rejects the moment a single rejection comes through, so there might be an expectation that this only works for resolving not rejecting 🤔


Make sure datetime object arrays are converted to datetime64

[x] Closes #2512 (remove if there is no corresponding issue, which should only be the case for minor changes)
[x] Tests added (for all bug fixes or enhancements)

Hello @dcherian! Thanks for submitting the PR.

There are no PEP8 issues in the file xarray/core/variable.py !

There are no PEP8 issues in the file xarray/tests/test_variable.py !


Dockerfile use the Makefile to build tikv

What have you changed? (mandatory)
Replace RUN cargo build --no-default-features --release --features "jemalloc portable sse no-fail" with RUN make build in script scripts/gen-dockerfile.sh  and execute the script to generate a new Dockerfile.
What are the type of the changes? (mandatory)

Improvement

How has this PR been tested? (mandatory)
N/A
Does this PR affect documentation (docs) or release note? (mandatory)
No
Does this PR affect tidb-ansible update? (mandatory)
No
Refer to a related PR or issue link (optional)
#4845

thanks @xiamengyu


Add refresh event for tablesaw-stack

Currently a refresh() event won't trigger an update to the table labels when new rows where added after initialization or the labels have changed.
The attached patch modifies Stack.prototype.init() so that column labels are only added if missing, and they will be updated. Additionally, the "tablesawrefresh" event is intercepted and will reexecute init().
tablesaw.stackonly.js_add-stack-refresh.patch.txt

Updated the patch to modify the 's HTML content only when changed. Otherwise refresh() couldn't be triggered by an MutationObserver because each update to the labels content would trigger another mutation event.
tablesaw.stackonly.js_add-stack-refresh.v2.patch.txt

@Cybso Can you file this as a PR instead?


Multiple read command in a single I2C command link times out.  (IDFGH-8147)

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.
master
Operating System used.
macOS
How did you build your project?
Command line with idf.py
If you are using Windows, please specify command line type.
No response
Development Kit.
ESP32 WROVER
Power Supply used.
USB
What is the expected behavior?
I'm creating a static i2c_cmd_buffer where I'm preparing a transaction with 2 successive reads, like this:
#define FilterError(x) ((x) == ESP_OK)

uint8 buffer[I2C_LINK_RECOMMENDED_SIZE(4)]; // Because 2 read transaction as documented
i2c_cmd_handle_t cmd = i2c_cmd_link_create_static(buffer, I2C_LINK_RECOMMENDED_SIZE(4));
uint8 devAddr = 0x76;  // For BMP210
uint8 regAddr[] = { 0xFA, 0xF7 };
uint8 dataBuffer[6] = {};

for (size_t index = 0; index < 2; i++) {
    if (!FilterError(i2c_master_start(cmd))) return false;
    if (!FilterError(i2c_master_write_byte(cmd, (uint8)((devAddr << 1) | I2C_MASTER_WRITE), true))) return false;
    // Set the register we are writing to
    if (!FilterError(i2c_master_write(cmd, (uint8*)&regAddr[index], 1, true))) return false;
    // Repeated start
    if (!FilterError(i2c_master_start(cmd))) return false;
    if (!FilterError(i2c_master_write_byte(cmd, (uint8)((devAddr << 1) | I2C_MASTER_READ), true))) return false;
    if (!FilterError(i2c_master_read(cmd, &dataBuffer[i * 3], 3, I2C_MASTER_LAST_NACK))) return false;
    // Tried with and without the following line (repeated start or not), but it doesn't work
    if (!FilterError(i2c_master_stop(cmd))) return false;
 }

// Later
if (!FilterError(i2c_master_cmd_begin(port, cmd, 1000 / portTICK_PERIOD_MS))) return false;

What is the actual behavior?
If there's only one read, the above code works. But if there are multiple read (like above) it fails with timeout, even with the stop part.
If I create 2 cmd link, it works. But I want to create a complete transaction with 2 read and (re)execute it whenever required.
What's the purpose of I2C_LINK_RECOMMENDED_SIZE if it only support one transaction only ?
As a side note, the documentation state:
For example, if one wants to perform a read on an I2C device register, TRANSACTIONS must be at least 2, because the commands required are the following:
    write device register
    read register content

but the actual code does this in i2c_master_write_read_device does this:
{
    esp_err_t err = ESP_OK;
    uint8_t buffer[I2C_TRANS_BUF_MINIMUM_SIZE] = { 0 };

(That's I2C_LINK_RECOMMENDED_SIZE(1) and not 2)
Steps to reproduce.
Please copy the above (simple) code and connect on any I2C device where you can read 2 register with a 8 bit address.
Debug Logs.
Error is: !ESP_ERR_TIMEOUT(263)

More Information.
No response

Few progress here:
I have similar issue without using a static buffer (typically, by using i2c_cmd_link_create) so it's not related to the way the command link handle is created.
Also, by merging the first 2 writes in a single write, that is, by doing:
uint8 skipWrite[2] = {(uint8)((devAddr << 1) | I2C_MASTER_WRITE), 0xFA };
if (!FilterError(i2c_master_write(cmd, skipWrite, 2, true))) return false;

Then no timeout error is reported, but the second read returns 0xFF (that's worse, IMHO, since it feels like it worked but returned garbish).
Also, by reading the I2C code, there's a race condition going on here in i2c_master_cmd_begin_static:
            i2c_hal_write_txfifo(&(i2c_context[i2c_num].hal), write_pr, fifo_fill);
            i2c_hal_write_cmd_reg(&(i2c_context[i2c_num].hal), hw_cmd, p_i2c->cmd_idx);
            i2c_hal_write_cmd_reg(&(i2c_context[i2c_num].hal), hw_end_cmd, p_i2c->cmd_idx + 1);
            i2c_hal_enable_master_tx_it(&(i2c_context[i2c_num].hal));
            p_i2c->cmd_idx = 0;
            if (i2c_cmd_is_single_byte(cmd) || cmd->total_bytes == cmd->bytes_used) {
                p_i2c->cmd_link.head = p_i2c->cmd_link.head->next;
                if(p_i2c->cmd_link.head) {
                    p_i2c->cmd_link.head->cmd.bytes_used = 0;
                }
            }
            p_i2c->status = I2C_STATUS_WRITE;

Here you enable the hardware interrupt (line 1421), but you later modify the p_i2c->cmd_link.head. So if the CPU reach line 1421 (enable_master_tx_it), and the interrupt fires, it'll call the interrupt handler, that'll, in turn, call i2c_master_cmd_begin_static (so you're re-entering this function before the effect on the linked list is applied and it's wrong).
Typical example, for this corruption:

You have a sequence of "read / write / read" command and are actually processing the write command.
The cpu process to line 1424 (cmd_link.head = head->next) but not bytes_used = 0 and also not the status = WRITE line
I2C interrupt triggers (cpu enters i2c_isr_handler_default)
Write event is processed, it calls back i2c_master_cmd_begin_static
Because the status wasn't updated yet, the line 1354 that read

if (p_i2c->cmd_link.head != NULL && p_i2c->status == I2C_STATUS_READ) {

pass the test (the previous operation was a read, and the link head isn't NULL). It shouldn't pass this test, since we are executing a write not a read.
7. Then, the code tries to copy the I2C's read buffer while it was executing a write command
I didn't follow what happens next, but it explains why I'm seeing garbage in the read buffer.

I've tried to bitbang the I2C command flow above and it works, with or without the stop condition (in a repeated start condition) between both read. So it's definitively an issue with this HW implementation not the device  itself.
Then I've added logs in the driver and it times out in the command that's created for the 3rd write_byte. The interrupt handler never receive an event for this write's command, so the timeout is actually a real timeout. This could happen if the device does ACK the write, but in that case, I guess we would get a HW's event with TOUT right ?
I'm trying to capture the output of the actual bus traffic with a logic analyzer, so I can see what's happening and if the device is ACKing the third write or not.

Ok, so here's the actual result on the console:
E i2c: I2C prog 0: cmd idx: 0
E i2c: I2C prog write: 2/2/2, cmd idx: 1
E i2c: I2C isr event type 4: cmd idx: 0
E i2c: I2C prog 0: cmd idx: 0
E i2c: I2C prog write: 1/0/1, cmd idx: 1
E i2c: I2C isr event type 4: cmd idx: 0
E i2c: I2C prog read: 2/0/2, cmd idx: 0
E i2c: I2C isr event type 4: cmd idx: 0
E i2c: I2C read int cb: 2/0/2, cmd idx: 0
E i2c: I2C prog read: 1/0/1, cmd idx: 0
E i2c: I2C isr event type 4: cmd idx: 0
E i2c: I2C read int cb: 1/0/1, cmd idx: 0
E i2c: I2C prog 0: cmd idx: 0
E i2c: I2C prog write: 2/2/2, cmd idx: 1
!ESP_ERR_TIMEOUT(263) in main.cpp:86

I've added logging with ESP_DRAM_LOGE in multiple places (typically, in the generic command prog, write command prog write, read cmd prog read, interrupt handler isr, read callback read int cb in cmd_begin_static).
So it means that what I expected above doesn't happen, yet the second write doesn't trigger the ISR and thus a timeout is received.
On the bus, with a logical analyzer, I'm getting this:

The first 4 communications are correct (a write of the address to read, then 2 reads with a NACK for the first one and a ACK for the last one), then it goes crazy, it sends a read and never finish (while it should send a write transaction instead). It repeats reading undefinitively (well, until the end of the capture, like 2 thousand reads).
If you look carefully, it seems the last NACK for the read command only happens inside the bad, infinite command (so waaaayyyy later). I would have expected SCL to go high after the last bit in the read command, but it doesn't:

I've tried with a stop before the 2nd read and while the sequence seems correct on the terminal (the values read are wrong for the 2nd read), a physical capture show that the HW is still struck/crashed:
E i2c: I2C prog 0: cmd idx: 0
E i2c: I2C prog write: 2/2/2, cmd idx: 1
E i2c: I2C isr event type 4: cmd idx: 0
E i2c: I2C prog 0: cmd idx: 0
E i2c: I2C prog write: 1/0/1, cmd idx: 1
E i2c: I2C isr event type 4: cmd idx: 0
E i2c: I2C prog read: 2/0/2, cmd idx: 0
E i2c: I2C isr event type 4: cmd idx: 0
E i2c: I2C read int cb: 2/0/2, cmd idx: 0
E i2c: I2C prog read: 1/0/1, cmd idx: 0
E i2c: I2C isr event type 4: cmd idx: 0
E i2c: I2C read int cb: 1/0/1, cmd idx: 0
E i2c: I2C prog 3: cmd idx: 0 // This is stop
E i2c: I2C prog 0: cmd idx: 1 // This is start for 2nd read request
E i2c: I2C prog write: 2/2/2, cmd idx: 2
E i2c: I2C isr event type 5: cmd idx: 0
E i2c: I2C prog 0: cmd idx: 0
E i2c: I2C prog write: 1/0/1, cmd idx: 1
E i2c: I2C isr event type 4: cmd idx: 0
E i2c: I2C prog read: 2/0/2, cmd idx: 0
E i2c: I2C isr event type 4: cmd idx: 0
E i2c: I2C read int cb: 2/0/2, cmd idx: 0
E i2c: I2C prog read: 1/0/1, cmd idx: 0
E i2c: I2C isr event type 4: cmd idx: 0
E i2c: I2C read int cb: 1/0/1, cmd idx: 0
E i2c: I2C prog 3: cmd idx: 0 // This is stop
E i2c: I2C isr event type 5: cmd idx: 0
Trans: 80 00 00 FF FF FF

The capture is the same as the above except for the last stuff (last image in my previous message):

Here, there isn't any write of the register address, and no read whatsoever. In fact, it's even worse, since nothing's said it has failed.

Just a remark in the logic in cmd_begin, when you parse a read request, you're making a read command and you append a end command (in cmd index 1). However, when the read complete (cmd index 0), the ISR is invoked, and you resume in cmd_begin_static, where you reset the cmd index to 0, so the end command is never executed. That's probably why we see the ACK / NACK happening in the later command and not at end of the ISR's context. Maybe you should execute the end command, then and only then, resetting cmd index to 0 and then continue appending the later commands.
Also, there's a huge temporal distance/delay between commands, so it's very far from optimal (since every step requires a ISR processing and a FreeRTOS queueing/scheduling). It would be better to enqueue as much commands as possible since you have 14 slots available if I understand the TRM correctly), to avoid these delay and performance issues.

Hi @X-Ryl669 ,
Thanks for providing that many details! I tested on my side and I see the same issue, few things here to note however:

I2C commands where not meant to represent multiple successive transactions. This is why we now have must simpler functions (i2c_master_write_to_device, i2c_master_read_from_device, i2c_master_write_read_device) that should reply to almost all the common use-case of I2C devices. Is there any particular reason that let you chain two transactions in the same command instead of making two transactions?
You are right about the fact that the way the software commands are written to the hardware could be highly optimized (because the hardware buffer has 16 entries). This is something that should be done in the NG (new-generation) driver implementation.
The race condition you talked about makes the assumption that as soon as you write to the tx FIFO and enable TX interrupt, an interrupt could occur. But in practice that is not the case. The transaction won't start until the function i2c_ll_trans_start is called. This is done before leaving the function, outside the loop where the previous function calls happened, and thus, after modifying any command link head or field.
The I2C hardware IP doesn't seem to support a stop following a restart. However, there is a way to force it to put a pause in between to prevent the error you are seeing. I have just created a small patch to overcome this issue:
i2c_patch.txt
(The file is a .diff but Github won't let me upload files with such extension)

Overall, the best solution would be to split up your code to actually perform two transactions thanks to i2c_master_write_read_device. If you still want to chain two transactions within one I2C command link, you can apply the patch above and have a try.

Is there any particular reason that let you chain two transactions in the same command instead of making two transactions?

I was seeing very sub-optimal I2C communication time, where a read of 6 bytes (at 2 different address) took 1ms where it should have taken 270µs (108 bits@400kHz). I thought it was due to task scheduling (and it's probably the case), so instead of executing multiple transactions that were allocating buffer and freeing them, I thought that executing a single static transaction should be a lot faster. And since the new release of esp-idf introduced the I2C_LINK_RECOMMENDED_SIZE macro, I thought it was the way to go.
Then I've looked into the I2C code and I've understood that even a single read transaction is in fact split in multiple sub-transaction, each with FreeRTOS's scheduling latencies issue in them. I don't know why the driver is so dependent on FreeRTOS here. Honestly I don't understand why the ISR had to post to a queue while executing a single command of a larger transaction, since this creates huge delays between commands.

This is something that should be done in the NG (new-generation) driver implementation.

Can I test it ? Is it available in beta somewhere ?

The race condition you talked about makes the assumption that as soon as you write to the tx FIFO and enable TX interrupt, an interrupt could occur. But in practice that is not the case. The transaction won't start until the function i2c_ll_trans_start is called. This is done before leaving the function, outside the loop where the previous function calls happened, and thus, after modifying any command link head or field.

You're right. I've figured this out later when I read again the TRM. I've still a question how to understand this line:
**Note**: When there are more than three segments, the address of an END command in the cmd should not be
altered into another command by the next segment.

If you know what it mean.
I'll test your patch and come back to you as soon as possible.
I was making 2 transactions before, but it's so slow that it's useless for my application (I have a very busy I2C bus with one current sensor that I must read at very regular interval of 250µs to accumulate the power consumption like a coulomb sensor). Here I can't afford a 5x longer I2C transaction time.

I've tested your patch and it's working to prevent the HW from crashing the bus so I guess it's a good workaround.
But it's not faster than splitting transactions, so I'm back to point 0 now, how to use the HW implementation at full speed?
Also, it's obviously not working in the Start/Write/Start/Read/Start/Write/Start/Read/Stop case, and I still don't understand why.
I guess it's due to the ACK bit of a read appearing in the next command (not the current read command) so the next Start fails because the master is likely holding the SDA line high for the NACK ?
I'm wondering why the current driver is architectured like this? Why doesn't it convert the command to fill the HW's command fifo and tries to execute all of them in one shot, instead of splitting each command by its own small subcommand ?

Hi @X-Ryl669 ,
As you saw, I2C_LINK_RECOMMENDED_SIZE and these related static functions won't influence the transaction time-to-finish once started. It is making the creation of the i2c link faster ad they won't perform any alloc.
I guess the driver was made like this (one operation at a time) because it was simpler to implement, less edge cases. Imagine we have two WRITEs and the first one has more bytes than the hardware FIFO, then we would still need to break the transaction down into two.
Anyways, I have a question regarding your requirements, you said you have to read the sensor at a regular interval of 270µs, but you also said that you have to transfer 6 bytes? Like every time?
If that's the case, it means that in theory each transaction will take about 180µs, if we add the software overhead, you will spend more than 200µs easily for a single transaction.
Moreover, I am convinced you have other things to process in other tasks so the task that performs the I2C is not the only task, FreeRTOS needs to schedule the other tasks to let them run. So the delay you will see between transaction is very long.
What delay are you seeing between each transaction?
On a very small example that has only a single app task that performs:
/* 4 bytes in total */
while (1) {
    i2c_master_write_read_device(I2C_MASTER_NUM, 0x20, wr, 2, rd, 2, 100);
}

on an ESP32, with a CPU frequency of 160MHz and an I2C bus at 400KHz with 2KOhm external pull-up resistors, I get 50µs between each transaction.
One way to optimize this a bit is to prepare a static command link, like you did, with i2c_cmd_link_create_static and i2c_master_stop, etc, that will be re-use across all the transactions. you don't need to free and re-create one every time you want to perform a transaction.
For example, you can imagine having:
    i2c_cmd_handle_t cmd = i2c_cmd_link_create_static(buffer, I2C_LINK_RECOMMENDED_SIZE(2));
    uint8 devAddr = 0x76;
    uint8 write_buffer[3] = { devAddr, 0 };

    i2c_master_start(cmd);
    i2c_master_write(cmd, write_buffer, 3, true);
    i2c_master_stop(cmd);

    /* Later, we can alter write_buffer, to generate a new operation */
    for (int i = 0; i < 10; i++) {
        write_buffer[1] = i;
        write_buffer[2] = i + 1;
        i2c_master_cmd_begin(port, cmd, 1000 / portTICK_PERIOD_MS);
    }

That's perfectly valid, maybe this is already what youa re doing.
With the same hardware setup as I mentioned above, by using this method of re-using a cmd link, I drop from 50µs to 40µs between each transaction.
Finally, if that's not fast enough, you can create a FreeRTOS task dedicated to that, pinned to the second core, with a high priority (higher than your other tasks).

Anyways, I have a question regarding your requirements, you said you have to read the sensor at a regular interval of 270µs, but you also said that you have to transfer 6 bytes? Like every time?

Yes, and to be honest it's a bit more complex than that on my system, I have 3 INA3221 sensor + 1 BMP280 sensor + 1 ICM20948 sensor on the same bus. Only one of the current sensor needs to be read at this interval, so I've a I2C scheduling task with a high priority, and I'm doing a ping/pong reading "battery current sensor / other sensor (one of env. sensor, or attitude sensor or other power sensor) / current sensor / other sensor / etc...). So, yes, I've prepared static transactions and I expected to be able to schedule them with the lowest overhead possible.

If that's the case, it means that in theory each transaction will take about 180µs, if we add the software overhead, you will spend more than 200µs easily for a single transaction.

I'm observing 1ms for a theoretical 270µs transaction with long pauses between the internal I2C subcommands (like between write and read transactions, between read and stop transaction etc...). It's clearly linked with the isr's handler time and looking at the code, I guess it's due to the alive event being queued for each sub-command transaction (I guess its used to deal with software timeout from the caller).
If the HW queue could be maximized (with the minimum ISR time used to refill/empty the queue and FIFO), I think it should be possible to avoid this overhead (so instead of having a END interrupt after each subcommand, maybe it's worth only dealing with the transaction complete interrupt (or, only have a END interrupt after a READ and before a WRITE, since the READ command overwrite the WRITE's RAM buffer).  TLDR: Reducing the interrupt overhead and trying to remove/reduce the FreeRTOS's overhead too.

What delay are you seeing between each transaction?

My ESP32 isn't too much loaded, since doing the I2C processing is 80% of the tasks it must perform. The other 2 task are sending UART communication (which is HW based too, validated and it works), and some SPI code for capturing a weight sensor sampling (HW based too, validated and it works). There is exceptional relay control via GPIO that doesn't count in the time budget.
So, yes the I2C communication is central for the system to work. I'm currently seeing around  ~100µs to 200µs between subcommands (on average). I'm currently cleaning my code to remove all the traces I've added to be able to measure this more precisely.
Also, even if I set the I2C speed to X Hz, its clock is going slower (for example, I measure 88kHz for a 100kHz target, ~350kHz for a 400kHz target). I don't know if it's expected or not.

That's perfectly valid, maybe this is already what you are doing.
With the same hardware setup as I mentioned above, by using this method of re-using a cmd link, I drop from 50µs to 40µs between each transaction

Yes that's what I was doing. But I wasn't seeing these delays (those are completely acceptable for my application). I'll try again now that the transaction works and report here with captures. I'm not seeing such low latency as soon as FreeRTOS is involved either, so maybe I haven't configured FreeRTOS the right way ?
When you said a new generation driver is coming, is it already written or just planned?

Hi @X-Ryl669 ,
Sorry, I forgot to reply to that question: the new generation driver is planned, not written yet.
Having an effective I2C clock speed slower than the theoretical one is expected, but the gap for you is not negligeable. Are you using the internal pull-up resistors? If yes, you should definitely add external pull up resistors to both SCL and SDA lines. Try with 4.7Kohms resistors first and if you still see that the effective SCL is far from 400KHz, use 2Kohm resistors instead.
Is the hardware still modifiable? If yes, you could install the high priority sensor on one I2C port and the other non-critical sensors on the I2C port, since the ESP32 has two hardware I2C port.
1ms is indeed very long for such transaction. What you can do to test whether this is due to the ISR handling is to pin your I2C task (the high priority one) on the core 1 and keep the other tasks, UART- and SPI-related, on core 0. As such, the I2C interrupt handler will always be called from the core 1, check the function xTaskCreatePinnedToCore
@Dazza0 any inputs related to FreeRTOS that could cause such delays?

I'll plug an oscilloscope to look at the slew rate and shape of the I2C bits. There are external pull up on the bus, on each sensor (they are all optional but one) so there should be an N parallel pull-up resistor (I'll measure the actual value and make some test).
You make me think about this, thanks, I'm pretty sure I've enabled pull up on the GPIO configuration for the I2C line, I'll disable them, should be useless anyway.
I'm out of pins on the ESP32 module we are using, I can't find 2 other free pins, they are all used for the sensors and the relays control and the signal leds. That's the main default of the ESP32, there isn't enough pins ;-)
I don't want to use an I2C IO extender.

Using several pull-up in parallel is not necessary, it will make the total pull-up resistor value of the bus lower.
Using an I2C I/O expander will make things worse as you will also need to program it using I2C the bus, so it means more bytes to send every time.
Do you mean that you have used up all the 26 available pins on the ESP32 module? Are the connected modules also time critical? If not, you could imagine reducing a amount of pins used by adding chips to manage the LEDs or other components?
For example, if you have 8 LEDs, instead of having 1 GPIO per LED, you could imagine adding a serial-in/parallel-out shift register (74HC595) to your circuit that will let you control 8 LEDs thanks to 2 GPIOs. (clock and serial data in)
Anyways, what you can try at the moment is "reserving" one core for your high priority I2C as I said before :smile:

Ok, it took too long because I had to port all platformio stuff to work with the master branch (BTW, espressif hasn't published the 11.2 version of the toolchain to platformio's registry).
So I'm measuring the time it takes to complete both transactions (not including the time to allocate/delete the command link), removed all logs and here are my results:

With this patch, i2C running at 400kHz, no other task than app_main, both read transaction in a single command link: 562µs
With this patch, i2C running at 400kHz, no other task than app_main, each transactions in its own command link: 608µs
Disabling "pullup enable" option: no change in timing

Hi @X-Ryl669 ,
562µs is more than expected, do you have a screenshot of the logic analyzer capture you have made?
Can you please check the resistance value between SCL-3.3V and SDA-3.3V?

So here are the captures:
Sampling of the analyzer is 2MHz, so a sample duration is 0.5µs.
I2C Bus was set to 200KHz (else, it failed with timeout when I plugged the logic analyzer, probably too high capacitance on the line)
The whole capture:

The whole duration is 1457 samples, which is 728µs (reported as 890µs in the software, I guess due to notifying the main task that the transfer is done)
There are very few pauses in the clock line (the largest is between the stop and start: 33 samples / 16.5µs see below)
Between ACK and repeated start condition:

6 samples = 3µs
Between stop and restart condition:

20 samples : 10µs
I guess there's much to gain here, it seems that the I2C bus is quite busy with no dead time.
When I set the bus to 400kHz, I'm not getting half the time here (it would be good enough) but 562µs instead of ~450µs.
I'll try to fix why I can't capture it with my analyzer.
As for the line's pull up resistance:

If I only plug the BMP280 sensor, I read 10kOhm between SDA and VCC (same with SCL).
If I plug all the sensors, I read 2.5kOhm on those lines.

Thanks @X-Ryl669 for the details here!
Honestly, the timings I see in your capture are correct for a 200KHz I2C Bus. The timing between a stop and a (re)start is expected.
However, what you said about the pull-up resistors confirm what I was thinking. You have 10kOhm pull-up resistors on each sensors (as you said), and as you have 5 sensors, the total resistance between SDA/SCL and 3.3V will be 2kOhm in theory (1/R = 1/R1 + 1/R2 + ... + 1/RN, in your case, R = 1/(5/10000) = 2000)
This should be enough to reach 400KHz, but in that case you need ALL the sensors to be plugged, else you won't be able to reach this speed.
I would advise you to try connecting your SDA and SCL lines to 3.3V with a 2kOhm (or 1kOhm) on the main board, so that they doesn't rely on the connected sensors' resistors, and retry your test.
For info, on ESP modules, with a 10kOhm pull-up resistor, you can only reach about 340/350KHz. (Of course the circuit capacitance is also an important factor but this is why I think you should try both 1kOhm and 2kOhm)

Sure, you're right. While doing my tests, I've removed all other sensors since initially the SW reported timeout errors. Since the pull up resistors are on the sensor boards, that's why I only have 10kOhm on the line. I need to tinker with the harness to plug the logic analyzer. I'll solder wires on the I2C line so I can make proper captures with the logic analyzer.
I'll do like you recommended and try to free 2 pins for using the additional I2C bus by removing 2 leds.
As a side note, I've found other cases of the system failing depending on how to set up the transaction (with your patch applied):

If you append Start / Write(dev address + W) / Write (reg address) / Rep Start / Write(dev addr + R) / Read N-1 with ACK / Read 1 NACK / Stop, it works
If you append Start / Write(dev address + W) / Write (reg address) / Rep Start / Write(dev addr + R) / Read N-1 with ACK / Read 1 NACK / Stop / Start / Write(dev address + W) / Write (reg address) / Rep Start / Write(dev addr + R) / Read N-1 with ACK / Read 1 NACK / Stop, it fails
If you append Start / Write 2 bytes (dev address + W, reg address) / Rep Start / Write(dev addr + R) / Read N-1 with ACK / Read 1 NACK / Stop / Start / Write(dev address + W, reg address) / Rep Start / Write(dev addr + R) / Read N-1 with ACK / Read 1 NACK / Stop, it works
If you append Start / Write 2 bytes (dev address + W, reg address) / Rep Start / Write(dev addr + R) / Read N-1 with ACK / Read 1 NACK / Rep Start / Write(dev address + W, reg address) / Rep Start / Write(dev addr + R) / Read N-1 with ACK / Read 1 NACK / Stop, it fails

So it seems to depend on the transaction operations' length, or since the driver is splitting the transaction is a small subset of command with END command in-between, it should work. Yet, it doesn't. I guess it's due to the position of the end command in the HW command's buffer, and I've spot a note in the TRM that seems linked to this: "When there are more than three segments, the address of an END command in the cmd should not be altered into another command by the next segment.". I don't understand what it means.

If I understand the driver code correctly, any write operation is converted to a "write + end" command. Same for the read operation that's converted to "read + end" command. However, if you follow the read callback code in i2c_master_cmd_begin_static, around line 1360, if the read operation hasn't finished reading all bytes, the command index is set to 0 (as expected). But else, it isn't reset to 0, so the next "command splitting" will happen after the executed read+end operation, so the HW command buffer will be like this:
Before interrupt:
 [ Read 1 NACK | End ]
/\ HW pointer

In Interrupt:
 [ Read 1 NACK | End ]
                /\ HW pointer
At line 1441 (after executing the code around 1360, total_bytes == bytes_used):
 [ Read 1 NACK | End | Stop ]
                /\ HW pointer

It works if the read is the last operation in the transaction because at line 1457 `if (p_i2c->cmd_link.head == NULL ... => cmd_idx = 0`

However, as soon as there are additional operations after the read operation (for example a rep start) line 1457 isn't executed and the cmd index keeps incrementing while it's already processed until it's set back to 0 on the next write, like this:
 [ Read 1 NACK | End | Start | Write | End ]
/\ HW pointer

That might explain why I'm seeing infinite read in the first capture, and also why your patch works with a stop&start successive operations, since you're triggering a new END interrupt that's acting like a write, that it, bypassing the cmd_index non reset.
I think the cmd_idx should be reset to 0 after a read operation also when total_bytes == bytes_used so that progress can be made in the HW command buffer.

Hi @X-Ryl669,
Thank you for all these details and the tests you have performed!
I can confirm what you have found in the previous message after finally understanding the sentence When there are more than three segments, the address of an END command in the cmd should not be altered into another command by the next segment.
The TRM specifies that a segment is a transaction starting from command index 0 and ending with an END (or STOP, but not important here). When we have more than 3 segments, the 4th+ segments must not modify the command operation that contains an END operation.
Which means that if during a segment, the END command is the second one (command index 1), in the next one, END must NOT be altered, so the command index 1 should still be an END.
I will take back your example 2. from your previous message. Let's perform 2 successive read transactions.
The driver will first perform:
1 - START + WRITE + END (Segment 1)
2 - WRITE + END (Segment 2)
3 - RSTART + WRITE + END (Segment 3)
4 - READ + END (Segment 4)
[...]
Starting from the fourth segment, the second command (index 1) shall always be an END, until the transaciton is finished with a STOP.
Moreover, the hardware doesn't support a succession of STOP/START commands because it only supports a single transaction within the commands table. In fact, as soon as the STOP command is encountered, the HW internal command index is reset to 0. So, the START command that was located after the STOP won't be executed.
Thinking about the patch back, inserting the END command may not be necessary, but it is still necessary to tell the software that after the end of the current transfer, there is another one waiting (managed by the added I2C_STATUS_BREAK status)

Thanks for the explanation. Unfortunately, it raises more question than it solves. I don't understand why the HW is so complex for following I2C specifications...
Anyway:

When the HW finds a END command, and the END_INT is enabled, it triggers an interrupt. But it also reset its HW's command pointer to zero, right ? (I thought that the HW was checking the "command done" bit for each command and skipping those that were done). Else, that doesn't match what we are observing, that is: the command 0 is overwritten with new work to do and work is resumed from here, it seems.
How can the HW knows it's executing the 3rd transaction (or the 4th...) ? Since each previous transaction starts with HW's command at position 0 ?  Is there a counter we can reset somehow ?
Can we store a END at the 16th command item, and fill the empty slots with 0 sized byte length's WRITE ? I've tried to add one in the command list and it seems to be ignored, as expected.

I might be wrong, but as I've understood, the HW stop condition doesn't need to be in the command buffer for the stop condition interruption to be triggered. In fact, I've done another project in the past with multiple master on the bus, and the ESP32 HW was generating the interrupt when another master generated the stop condition (not a big deal per se). So, it seems the logic to generate interrupts is based on observing the line, not on the command being executed (except for END, right ?), whatever the hardware which is manipulating the lines.
I've tried to set the cmd_index to 0 in the READ finished callback, but it doesn't work either for solving the failing cases.


[DNM]try more group.

What problem does this PR solve?

Issue Number: close #xxx
What is changed and how it works?
Check List
Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Questions
Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?
Release note
Please refer to [Release Notes Language Style Guide](https://pingcap.github.io/tidb-dev-guide/contribute-to-tidb/release-notes-style-guide.html) to write a quality release note.

If you don't think this PR needs a release note then fill it with `None`.

Observing the existing build history, it is found that except for group11, the execution time of other group tests is generally around 4 minutes. It is suggested to consider dispersing the tests in group11 among the existing groups instead of adding more groups.

/retest