...finished in 0:05:26.380400

--- Merging RPKI and IRR data ---

Parse base file to numpy arrays
Parse extra file to Pandas DataFrame
Filtering extra prefixes that were already included in the base file:

 100.00% :::::::::::::::::::::::::::::::::::::::: |   308569 /   308569 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   308568 /   308568 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   308568 /   308568 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   308568 /   308568 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   308568 /   308568 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   308568 /   308568 |

Finished filtering! Originally 1851409 entries filtered down to 523116
Merging base file with filtered extra file
...finished in 2:12:40.567283

--- Parsing Routeviews pfx2as ---

Unzipping /home/b10c/kartograf/data/1703167200/collectors/routeviews_pfx2asn_ip4.txt.gz
Formatting /home/b10c/kartograf/out/1703167200/collectors/routeviews_pfx2asn_ip4.txt
Unzipping /home/b10c/kartograf/data/1703167200/collectors/routeviews_pfx2asn_ip6.txt.gz
Formatting /home/b10c/kartograf/out/1703167200/collectors/routeviews_pfx2asn_ip6.txt
Cleaning /home/b10c/kartograf/out/1703167200/collectors/pfx2asn.txt
...finished in 0:03:10.729591

--- Merging Routeviews and base data ---

Parse base file to numpy arrays
Parse extra file to Pandas DataFrame
Filtering extra prefixes that were already included in the base file:

 100.00% :::::::::::::::::::::::::::::::::::::::: |   198314 /   198314 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   198314 /   198314 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   198314 /   198314 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   198314 /   198314 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   198314 /   198314 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   198314 /   198314 |

Finished filtering! Originally 1189884 entries filtered down to 363874
Merging base file with filtered extra file
...finished in 3:19:28.471326

--- Sorting results ---

...finished in 0:00:12.626709

--- Finishing Kartograf ---

The SHA-256 hash of the result file is: 28039075f34ef65f9faff36a5141ce35a9b322600a263d4cf4cbb5d3e45390fe
Total runtime: 6:03:28.581414

Matched as well :triumph:
The SHA-256 hash of the result file is: 28039075f34ef65f9faff36a5141ce35a9b322600a263d4cf4cbb5d3e45390fe
Total runtime: 9:34:04.141037

--- Start Kartograf ---
Using rpki-client version 8.6.
Coordinated launch mode: Waiting until<PHONE_NUMBER> (2023-12-21 14:00:00 WET) to launch mapping process.
The epoch for this run is:<PHONE_NUMBER> (2023-12-21 14:00:00 UTC, local: 2023-12-21 14:00:00 WET)
--- Fetching RPKI ---
Downloaded TAL for AFRINIC to /home/base/code/kartograf/data/1703167200/rpki/tals/afrinic.tal
Downloaded TAL for APNIC to /home/base/code/kartograf/data/1703167200/rpki/tals/apnic.tal
Downloaded TAL for ARIN to /home/base/code/kartograf/data/1703167200/rpki/tals/arin.tal
Downloaded TAL for LACNIC to /home/base/code/kartograf/data/1703167200/rpki/tals/lacnic.tal
Downloaded TAL for RIPE to /home/base/code/kartograf/data/1703167200/rpki/tals/ripe.tal
Downloading RPKI Data
...finished in 0:03:20.407919
--- Fetching IRR ---
Downloading afrinic.db.gz
Downloading apnic.db.route.gz
Downloading apnic.db.route6.gz
Downloading arin.db.gz
Downloading lacnic.db.gz
Downloading ripe.db.route.gz
Downloading ripe.db.route6.gz
...finished in 0:01:19.785957
--- Fetching Routeviews pfx2as ---
Downloading https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2023/12/routeviews-rv2-20231219-1200.pfx2as.gz
Downloading https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2023/12/routeviews-rv2-20231219-1200.pfx2as.gz
Downloading https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2023/12/routeviews-rv6-20231220-1200.pfx2as.gz
...finished in 0:00:12.090912
--- Validating RPKI ---
Validating RPKI ROAs
209048 raw RKPI ROA files found.
209048 RKPI ROAs validated and saved to /home/base/code/kartograf/out/1703167200/rpki/rpki_raw.json
...finished in 0:23:37.554676
--- Parsing RPKI ---
Parsing 209048 ROAs
Result entries written: 450887
Duplicates found: 55917
Invalids found: 20535
Incompletes: 0
Non-ROA files: 0
...finished in 0:01:37.959571
--- Parsing IRR ---
Extracting afrinic.db.gz
Extracting apnic.db.route.gz
Extracting apnic.db.route6.gz
Extracting arin.db.gz
Extracting lacnic.db.gz
Extracting ripe.db.route.gz
Extracting ripe.db.route6.gz
Parsing /home/base/code/kartograf/out/1703167200/irr/afrinic.db
Found valid entries: 112325
Parsing /home/base/code/kartograf/out/1703167200/irr/arin.db
Found valid entries: 102296
Parsing /home/base/code/kartograf/out/1703167200/irr/ripe.db.route6
Found valid entries: 197126
Parsing /home/base/code/kartograf/out/1703167200/irr/ripe.db.route6                                                                                [35/1947]
Found valid entries: 197126
Parsing /home/base/code/kartograf/out/1703167200/irr/apnic.db.route6
Found valid entries: 329103
Parsing /home/base/code/kartograf/out/1703167200/irr/lacnic.db
Found valid entries: 12224
Parsing /home/base/code/kartograf/out/1703167200/irr/ripe.db.route
Found valid entries: 405374
Parsing /home/base/code/kartograf/out/1703167200/irr/apnic.db.route
Found valid entries: 693049
...finished in 0:06:14.558009
--- Merging RPKI and IRR data ---
Parse base file to numpy arrays
Parse extra file to Pandas DataFrame
Filtering extra prefixes that were already included in the base file:
0.00%                                          |        0 /   231427 |    0.00%                                          |        0 /   231427 |
93  93.27% :::::::::::::::::::::::::::::::::::::    |   215846 /   231427 |
93  93.27% :::::::::::::::::::::::::::::::::::::    |   215852 /   231427 |
100.00% ::::::::::::::::::::::::::::::::::::::::     |   231427 /   231427 |
100.00% ::::::::::::::::::::::::::::::::::::::::     |   231426 /   231426 |
100.00% ::::::::::::::::::::::::::::::::::::::::     |   231426 /   231426 |
100.00% ::::::::::::::::::::::::::::::::::::::::     |   231426 /   231426 |
100.00% ::::::::::::::::::::::::::::::::::::::::     |   231426 /   231426 |                                                                                ^
100.00% ::::::::::::::::::::::::::::::::::::::::     |   231426 /   231426 |                                                                                ^
100.00% ::::::::::::::::::::::::::::::::::::::::     |   231426 /   231426 |
100.00% ::::::::::::::::::::::::::::::::::::::::     |   231426 /   231426 |
Finished filtering! Originally 1851409 entries filtered down to 523115
Merging base file with filtered extra file
...finished in 3:40:25.841542
--- Parsing Routeviews pfx2as ---
Unzipping /home/base/code/kartograf/data/1703167200/collectors/routeviews_pfx2asn_ip4.txt.gz
Unzipping /home/base/code/kartograf/data/1703167200/collectors/routeviews_pfx2asn_ip4.txt.gz
Formatting /home/base/code/kartograf/out/1703167200/collectors/routeviews_pfx2asn_ip4.txt
Unzipping /home/base/code/kartograf/data/1703167200/collectors/routeviews_pfx2asn_ip6.txt.gz
Formatting /home/base/code/kartograf/out/1703167200/collectors/routeviews_pfx2asn_ip6.txt
Cleaning /home/base/code/kartograf/out/1703167200/collectors/pfx2asn.txt
...finished in 0:03:56.049441
--- Merging Routeviews and base data ---
Parse base file to numpy arrays
Parse extra file to Pandas DataFrame
Filtering extra prefixes that were already included in the base file:
100.00% :::::::::::::::::::::::::::::::::::::::: |   148736 /   148736 |
100.00% :::::::::::::::::::::::::::::::::::::::: |   148736 /   148736 |
100.00% :::::::::::::::::::::::::::::::::::::::: |   148736 /   148736 |
100.00% :::::::::::::::::::::::::::::::::::::::: |   148736 /   148736 |
100.00% :::::::::::::::::::::::::::::::::::::::: |   148735 /   148735 |
100.00% :::::::::::::::::::::::::::::::::::::::: |   148735 /   148735 |
100.00% :::::::::::::::::::::::::::::::::::::::: |   148735 /   148735 |
100.00% :::::::::::::::::::::::::::::::::::::::: |   148735 /   148735 |
Finished filtering! Originally 1189884 entries filtered down to 363874
Merging base file with filtered extra file
...finished in 5:13:03.231756
--- Sorting results ---
...finished in 0:00:14.493410
--- Finishing Kartograf ---
The SHA-256 hash of the result file is: 28039075f34ef65f9faff36a5141ce35a9b322600a263d4cf4cbb5d3e45390fe
Total runtime: 9:34:04.141037

Not a match for me but very happy the fix for the previous bug now works and we are getting matches! This looks already much better than I expect a few weeks ago when I had the idea for this feature :)
My hash is b118079898b62383f13a81269367e45adc24a015e5f8a83799fc14517411c637 FWIW.

My logs
$ ./run map -w=1703167200 -irr -rv

--- Start Kartograf ---

Using rpki-client version 8.5.
Coordinated launch mode: Waiting until<PHONE_NUMBER> (2023-12-21 15:00:00 CET) to launch mapping process.
The epoch for this run is:<PHONE_NUMBER> (2023-12-21 14:00:00 UTC, local: 2023-12-21 15:00:00 CET)

--- Fetching RPKI ---

Downloaded TAL for AFRINIC to /Users/FJ/projects/python/kartograf/data/1703167200/rpki/tals/afrinic.tal
Downloaded TAL for APNIC to /Users/FJ/projects/python/kartograf/data/1703167200/rpki/tals/apnic.tal
Downloaded TAL for ARIN to /Users/FJ/projects/python/kartograf/data/1703167200/rpki/tals/arin.tal
Downloaded TAL for LACNIC to /Users/FJ/projects/python/kartograf/data/1703167200/rpki/tals/lacnic.tal
Downloaded TAL for RIPE to /Users/FJ/projects/python/kartograf/data/1703167200/rpki/tals/ripe.tal
Downloading RPKI Data
...finished in 0:08:13.241378

--- Fetching IRR ---

Downloading afrinic.db.gz
Downloading apnic.db.route.gz
Downloading apnic.db.route6.gz
Downloading arin.db.gz
Downloading lacnic.db.gz
Downloading ripe.db.route.gz
Downloading ripe.db.route6.gz
...finished in 0:00:56.004794

--- Fetching Routeviews pfx2as ---

Downloading https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2023/12/routeviews-rv2-20231219-1200.pfx2as.gz
Downloading https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2023/12/routeviews-rv6-20231220-1200.pfx2as.gz
...finished in 0:00:05.761229

--- Validating RPKI ---

Validating RPKI ROAs
209048 raw RKPI ROA files found.
209048 RKPI ROAs validated and saved to /Users/FJ/projects/python/kartograf/out/1703167200/rpki/rpki_raw.json
...finished in 0:36:42.780071

--- Parsing RPKI ---

Parsing 209048 ROAs
Result entries written: 450877
Duplicates found: 55916
Invalids found: 20546
Incompletes: 0
Non-ROA files: 0
...finished in 0:00:51.757210

--- Parsing IRR ---

Extracting afrinic.db.gz
Extracting apnic.db.route.gz
Extracting apnic.db.route6.gz
Extracting arin.db.gz
Extracting lacnic.db.gz
Extracting ripe.db.route.gz
Extracting ripe.db.route6.gz
Parsing /Users/FJ/projects/python/kartograf/out/1703167200/irr/lacnic.db
Found valid entries: 12224
Parsing /Users/FJ/projects/python/kartograf/out/1703167200/irr/ripe.db.route
Found valid entries: 405374
Parsing /Users/FJ/projects/python/kartograf/out/1703167200/irr/apnic.db.route6
Found valid entries: 329103
Parsing /Users/FJ/projects/python/kartograf/out/1703167200/irr/apnic.db.route
Found valid entries: 693049
Parsing /Users/FJ/projects/python/kartograf/out/1703167200/irr/arin.db
Found valid entries: 102296
Parsing /Users/FJ/projects/python/kartograf/out/1703167200/irr/afrinic.db
Found valid entries: 112325
Parsing /Users/FJ/projects/python/kartograf/out/1703167200/irr/ripe.db.route6
Found valid entries: 197126
...finished in 0:03:15.496587

--- Merging RPKI and IRR data ---

Parse base file to numpy arrays
Parse extra file to Pandas DataFrame
Filtering extra prefixes that were already included in the base file:

 100.00% :::::::::::::::::::::::::::::::::::::::: |   185141 /   185141 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   185141 /   185141 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   185141 /   185141 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   185141 /   185141 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   185141 /   185141 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   185141 /   185141 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   185141 /   185141 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   185141 /   185141 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   185141 /   185141 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   185140 /   185140 |

Finished filtering! Originally 1851409 entries filtered down to 523146
Merging base file with filtered extra file
...finished in 1:07:13.947766

--- Parsing Routeviews pfx2as ---

Unzipping /Users/FJ/projects/python/kartograf/data/1703167200/collectors/routeviews_pfx2asn_ip4.txt.gz
Formatting /Users/FJ/projects/python/kartograf/out/1703167200/collectors/routeviews_pfx2asn_ip4.txt
Unzipping /Users/FJ/projects/python/kartograf/data/1703167200/collectors/routeviews_pfx2asn_ip6.txt.gz
Formatting /Users/FJ/projects/python/kartograf/out/1703167200/collectors/routeviews_pfx2asn_ip6.txt
Cleaning /Users/FJ/projects/python/kartograf/out/1703167200/collectors/pfx2asn.txt
...finished in 0:01:57.510614

--- Merging Routeviews and base data ---

Parse base file to numpy arrays
Parse extra file to Pandas DataFrame
Filtering extra prefixes that were already included in the base file:

 100.00% :::::::::::::::::::::::::::::::::::::::: |   118989 /   118989 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   118989 /   118989 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   118989 /   118989 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   118989 /   118989 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   118988 /   118988 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   118988 /   118988 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   118988 /   118988 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   118988 /   118988 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   118988 /   118988 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   118988 /   118988 |

Finished filtering! Originally 1189884 entries filtered down to 363875
Merging base file with filtered extra file
...finished in 1:58:06.471503

--- Sorting results ---

...finished in 0:00:07.708851

--- Finishing Kartograf ---

The SHA-256 hash of the result file is: b118079898b62383f13a81269367e45adc24a015e5f8a83799fc14517411c637
Total runtime: 6:03:34.475947

So, comparing the logs it looks to me like there is probably a slight different from the RPKI data. Comparing to @Sjors I found one duplicate less and 11 invalids more. That's probably where diff is coming from but I expect it to be small. I'm sharing my result here if someone wants do compare but I would also like to receive the result from one of you so I can look into it.
final_result.txt.zip
Additionally, I would be happy if someone can share their out/rpki/rpki_raw.json, there we can see what caused the actual validation differences. It's a pretty big file so it probably can't be shared here directly even after compression.

@fjahr I'll make a WeTransfer link

I've noticed that the final_results.txt file contains AS37963, but also aS30844, and As51528, and as200890. Is the final_results.txt file relevant for next steps in the process? Does it make sense to standardize this to uppercase(e.g., always "AS1234").?

Here's my result file @fjahr .
(q: how are you uploading zip files to this repo directly?)

(q: how are you uploading zip files to this repo directly?)

You can just drag-and-drop it into the comment field here or use the paper clip icon on the top right.

I've noticed that the final_results.txt file contains AS37963, but also aS30844, and As51528, and as200890. Is the final_results.txt file relevant for next steps in the asmap process? Does it make sense to standardize this to uppercase (e.g., always "AS1234")?

Oh, very interesting. I didn't notice that yet, whenever I write it explicitly in kartograf I do 'AS' but I will look into it. In the back of my mind I always had the idea that writing AS explicitly was redundant anyway, so when should just drop it and save those bytes. I will test if that even breaks Sipa's compression code and if not, I will simply do that instead :)

Keeping the AS prefix might it marginally more human readable and easier to copy-paste into google. These extra bytes don't make into the final bitcoin binary I assume, so it's probably fine to just keep them.

Looking through the file, I found it interesting that there are duplicates: e.g., 126 entries for 2001:503:eea3::30/128 with different ASNs. Not sure if that's to be expected. I probably need to look into the whole asmap process a bit more.
FWIW here's the diff -patch final_result_0xb10c.txt final_result_fjahr.txt between fjahr's and my file. No difference between sjors and my file - I guess that's to be expected when we get the same hash.

fjahr's results have (multiple) ASNs for 45.132.[188-191].0/24 which I'm missing
<IP_ADDRESS>/23 AS58367 and <IP_ADDRESS>/24 AS58367  <IP_ADDRESS>/24 AS58367 are the same result, just differently written. A .. 8.0/23 net is the same as a .. 8.0/24 + a .. 9.0/24 net. Not sure how to solve this? Maybe always trying to build the biggest supernet for an ASN? This might also reduce the file size a bit?

*** final_result_0xb10c.txt	2023-12-21 22:15:19 +0100
--- final_result_fjahr.txt	2023-12-21 21:03:35 +0100
***************
*** 113800,113806 ****
--- 113800,113827 ----
  <IP_ADDRESS>/24 AS35830
  <IP_ADDRESS>/24 AS35830
  <IP_ADDRESS>/24 AS35830
+ <IP_ADDRESS>/24 AS3970
+ <IP_ADDRESS>/24 AS47065
+ <IP_ADDRESS>/24 AS61574
+ <IP_ADDRESS>/24 AS61575
+ <IP_ADDRESS>/24 AS61576
+ <IP_ADDRESS>/22 AS3130
  <IP_ADDRESS>/22 AS3970
+ <IP_ADDRESS>/24 AS3970
+ <IP_ADDRESS>/24 AS47065
+ <IP_ADDRESS>/24 AS61574
+ <IP_ADDRESS>/24 AS61575
+ <IP_ADDRESS>/24 AS61576
+ <IP_ADDRESS>/24 AS3970
+ <IP_ADDRESS>/24 AS47065
+ <IP_ADDRESS>/24 AS61574
+ <IP_ADDRESS>/24 AS61575
+ <IP_ADDRESS>/24 AS61576
+ <IP_ADDRESS>/24 AS3970
+ <IP_ADDRESS>/24 AS47065
+ <IP_ADDRESS>/24 AS61574
+ <IP_ADDRESS>/24 AS61575
+ <IP_ADDRESS>/24 AS61576
  <IP_ADDRESS>/24 AS210636
  <IP_ADDRESS>/24 AS39351
  <IP_ADDRESS>/24 AS61272
***************
*** 560369,560375 ****
  <IP_ADDRESS>/24 AS61575
  <IP_ADDRESS>/24 AS61576
  <IP_ADDRESS>/24 AS3130
! <IP_ADDRESS>/23 AS58367
  <IP_ADDRESS>/24 AS47065
  <IP_ADDRESS>/24 AS61574
  <IP_ADDRESS>/24 AS61575
--- 560390,560397 ----
  <IP_ADDRESS>/24 AS61575
  <IP_ADDRESS>/24 AS61576
  <IP_ADDRESS>/24 AS3130
! <IP_ADDRESS>/24 AS58367
! <IP_ADDRESS>/24 AS58367
  <IP_ADDRESS>/24 AS47065
  <IP_ADDRESS>/24 AS61574
  <IP_ADDRESS>/24 AS61575

You could also have a two-round process. First round like we did above, where some people share more details to figure out the differences. Then in a second round, which skips the download step, everyone applies a patch to reconcile or get rid of the conflicting entries. And then re-run the steps.

Keeping the AS prefix makes it marginally more human readable and easier to copy-paste into google. These extra bytes don't make into the final bitcoin binary I assume, so it's probably fine to just keep them.

I am keeping the AS in but explicitly format it to be all-caps now every time I write it.

Maybe always trying to build the biggest supernet for an ASN? This might also reduce the file size a bit?

I am currently not in favor of this since it would make debugging/analyzing the result harder and in terms of the file size I think in the compressed state this doesn't have much impact, if it's not resolved completely. It would potentially also mess with the coverage statistics of the network, depending on how it's implemented.

I found it interesting that there are duplicate assignments for IP-net to ASN

These shouldn't have been there. I found they are coming from IRR and from my understanding, this isn't normal usage. I checked some entries and it seems more like these are there to not block some experiments. Either way, I have resolved this in the same way I am doing it in PRKI, where duplicates are very common.
Example:
route:          <IP_ADDRESS>/24
descr:          RPKI Experiment
origin:         AS61575
notify<EMAIL_ADDRESS>mnt-by:         MAINT-RGNET
created:        2020-11-28T19:56:21Z
last-modified:  2020-11-28T19:56:21Z
source:         RIPE

I also found another issue where a handful of IPv6 prefixes had leading zeros, i.e. 2401:0001::/32. These only disappeared later in the process when the IP is converted to an int and so this influenced results.
All the issues are resolved in master and I will do some more testing before we can to another test. Thanks a lot everyone again for joining and testing!

You could also have a two-round process. First round like we did above, where some people share more details to figure out the differences. Then in a second round, which skips the download step, everyone applies a patch to reconcile or get rid of the conflicting entries. And then re-run the steps.

I think this is a bit too complicated for starting out. I think the regular process should be very quick by default so that we can be sure it isn't blocked. It seems pretty promising that we can do a collaborative launch with a group and if the majority has the same result, it's good to merge. If we don't get a majority we simply try again.
Figuring out the differences may be easy sometimes but often I think this could create a lot of research work and/or the result may not be satisfactory for everyone. We would need some kind of definition for what is ok and not ok and then come to a rough consensus agreement on which way to go for each line of the diff. I am a bit afraid we may not have everyone's attention all of the time and so if a diff analysis gets drawn out do we still have enough of the original runners engaged when it's resolved? Or would it be ok if someone else comes in who was not part of the original group but then reviews the diff and says that it is alright? I see a lot of unanswered questions so I would like us to learn a bit more along the way before we commit to figuring out the diffs every time. Of course, it's always encouraged to look at the diffs and figure out where they came from and what they mean, if there is something to be concerned or a sign for improvement. I just would like to bank the  whole process on that part right now.
It also makes the process a bit more vulnerable to attacks, I think. If we try to reconcile all differences the process can be blocked by a single person submitting a bogus result that is impossible to resolve because it's not based on real data. With the "federated" approach above there would at least be a significant part of the group that submits such results and may be more obvious what is happening if 3-5 never seen before github nyms join the process rather than just one.

Hi and happy new year! 🎆
I have tagged release 0.4.0 which has the fixes I mentioned previously and a few other nice usability improvements, like an added progress bar during RPKI validation, showing a countdown during the wait, printing the current Kartograf version etc.
I want to do (yet another) test run with this version. The previous run already was a success IMO since it showed that except for me everyone else agreed on the result and the differences between me and the rest were in the "explainable" category. But doing another run would be great because we can't test too much and the fixes I have introduced should lead to an easier time drilling down more into the diff.
I propose tomorrow 2:00 pm GMT<PHONE_NUMBER>) and I hope it's not too short notice. Thanks again! 🙏 🙏 🙏
./run map -w=1704463200 -irr -rv

@brunoerg did you end up having a result hash for the previous run BTW?

@fjahr no, I had a problem on my laptop, fortunately I just bought a better one and will have the runs faster 🚀

@fjahr no, I had a problem on my laptop, fortunately I just bought a better one and will have the runs faster 🚀

Ok, thanks for the heads-up!

I propose tomorrow 2:00 pm GMT<PHONE_NUMBER>) and I hope it's not too short notice. Thanks again! 🙏 🙏 🙏
./run map -w=1704463200 -irr -rv

Feel free to use 0.4.1 for this one instead, where I have improved the formatting of the countdown, something I forgot to include in the release previously: https://github.com/fjahr/kartograf/releases/tag/0.4.1

I propose tomorrow 2:00 pm GMT<PHONE_NUMBER>) and I hope it's not too short notice. Thanks again! 🙏 🙏 🙏

Reminder, this is in 1h and 30 min. I'll be joining.

I'll be joining!
The countdown is great, nice job!

My result hash is4aba172195b8dc375c2bb1b39c3508f48c212cedc02983a7dc794a373820653b

Detailed logs
$ ./run map -w=1704463200 -irr -rv

--- Start Kartograf ---

Kartograf version: 0.4.1
Using rpki-client version 8.5.
Coordinated launch mode: Waiting until<PHONE_NUMBER> (2024-01-05 15:00:00 CET) to launch mapping process.
Countdown: 0 second(s)
Starting...
The epoch for this run is:<PHONE_NUMBER> (2024-01-05 14:00:00 UTC, local: 2024-01-05 15:00:00 CET)

--- Fetching RPKI ---

Downloaded TAL for AFRINIC to /Users/FJ/projects/python/kartograf/data/1704463200/rpki/tals/afrinic.tal
Downloaded TAL for APNIC to /Users/FJ/projects/python/kartograf/data/1704463200/rpki/tals/apnic.tal
Downloaded TAL for ARIN to /Users/FJ/projects/python/kartograf/data/1704463200/rpki/tals/arin.tal
Downloaded TAL for LACNIC to /Users/FJ/projects/python/kartograf/data/1704463200/rpki/tals/lacnic.tal
Downloaded TAL for RIPE to /Users/FJ/projects/python/kartograf/data/1704463200/rpki/tals/ripe.tal
Downloading RPKI Data
...finished in 0:08:59.532504

--- Fetching IRR ---

Downloading afrinic.db.gz
Downloading apnic.db.route.gz
Downloading apnic.db.route6.gz
Downloading arin.db.gz
Downloading lacnic.db.gz
Downloading ripe.db.route.gz
Downloading ripe.db.route6.gz
...finished in 0:01:05.592947

--- Fetching Routeviews pfx2as ---

Downloading https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2024/01/routeviews-rv2-20240103-1200.pfx2as.gz
Downloading https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2024/01/routeviews-rv6-20240104-1200.pfx2as.gz
...finished in 0:00:07.467450

--- Validating RPKI ---

Validating RPKI ROAs
212003 raw RKPI ROA files found.
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 212003/212003 [36:11<00:00, 97.65it/s]
212003 RKPI ROAs validated and saved to /Users/FJ/projects/python/kartograf/out/1704463200/rpki/rpki_raw.json
...finished in 0:36:33.467654

--- Parsing RPKI ---

Parsing 212003 ROAs
Result entries written: 451864
Duplicates found: 56623
Invalids found: 23066
Incompletes: 0
Non-ROA files: 0
...finished in 0:00:53.797292

--- Parsing IRR ---

Extracting afrinic.db.gz
Extracting apnic.db.route.gz
Extracting apnic.db.route6.gz
Extracting arin.db.gz
Extracting lacnic.db.gz
Extracting ripe.db.route.gz
Extracting ripe.db.route6.gz
Parsing /Users/FJ/projects/python/kartograf/out/1704463200/irr/lacnic.db
Parsing /Users/FJ/projects/python/kartograf/out/1704463200/irr/ripe.db.route
Parsing /Users/FJ/projects/python/kartograf/out/1704463200/irr/apnic.db.route6
Parsing /Users/FJ/projects/python/kartograf/out/1704463200/irr/apnic.db.route
Parsing /Users/FJ/projects/python/kartograf/out/1704463200/irr/arin.db
Parsing /Users/FJ/projects/python/kartograf/out/1704463200/irr/afrinic.db
Parsing /Users/FJ/projects/python/kartograf/out/1704463200/irr/ripe.db.route6
Found valid, unique entries: 1419403
...finished in 0:03:10.402381

--- Merging RPKI and IRR data ---

Parse base file to numpy arrays
Parse extra file to Pandas DataFrame
Filtering extra prefixes that were already included in the base file:

 100.00% :::::::::::::::::::::::::::::::::::::::: |   141941 /   141941 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   141941 /   141941 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   141941 /   141941 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   141940 /   141940 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   141940 /   141940 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   141940 /   141940 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   141940 /   141940 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   141940 /   141940 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   141940 /   141940 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   141940 /   141940 |

Finished filtering! Originally 1419403 entries filtered down to 394036
Merging base file with filtered extra file
...finished in 0:59:16.045177

--- Parsing Routeviews pfx2as ---

Unzipping /Users/FJ/projects/python/kartograf/data/1704463200/collectors/routeviews_pfx2asn_ip4.txt.gz
Formatting /Users/FJ/projects/python/kartograf/out/1704463200/collectors/routeviews_pfx2asn_ip4.txt
Unzipping /Users/FJ/projects/python/kartograf/data/1704463200/collectors/routeviews_pfx2asn_ip6.txt.gz
Formatting /Users/FJ/projects/python/kartograf/out/1704463200/collectors/routeviews_pfx2asn_ip6.txt
Cleaning /Users/FJ/projects/python/kartograf/out/1704463200/collectors/pfx2asn.txt
...finished in 0:02:00.596561

--- Merging Routeviews and base data ---

Parse base file to numpy arrays
Parse extra file to Pandas DataFrame
Filtering extra prefixes that were already included in the base file:

 100.00% :::::::::::::::::::::::::::::::::::::::: |   118929 /   118929 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   118929 /   118929 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   118929 /   118929 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   118929 /   118929 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   118929 /   118929 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   118929 /   118929 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   118929 /   118929 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   118929 /   118929 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   118929 /   118929 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   118928 /   118928 |

Finished filtering! Originally 1189289 entries filtered down to 410310
Merging base file with filtered extra file
...finished in 1:27:57.625737

--- Sorting results ---

...finished in 0:00:11.438770

--- Finishing Kartograf ---

The SHA-256 hash of the result file is: 4aba172195b8dc375c2bb1b39c3508f48c212cedc02983a7dc794a373820653b
Total runtime: 3:20:17.761217

I've got a match on 4aba172195b8dc375c2bb1b39c3508f48c212cedc02983a7dc794a373820653b!

--- Start Kartograf ---

Kartograf version: 0.4.1
Using rpki-client version 8.6.
Coordinated launch mode: Waiting until<PHONE_NUMBER> (2024-01-05 15:00:00 CET) to launch mapping process.
Countdown: 0 second(s)
Starting...
The epoch for this run is:<PHONE_NUMBER> (2024-01-05 14:00:00 UTC, local: 2024-01-05 15:00:00 CET)

--- Fetching RPKI ---

Downloaded TAL for AFRINIC to /home/b10c/kartograf/data/1704463200/rpki/tals/afrinic.tal
Downloaded TAL for APNIC to /home/b10c/kartograf/data/1704463200/rpki/tals/apnic.tal
Downloaded TAL for ARIN to /home/b10c/kartograf/data/1704463200/rpki/tals/arin.tal
Downloaded TAL for LACNIC to /home/b10c/kartograf/data/1704463200/rpki/tals/lacnic.tal
Downloaded TAL for RIPE to /home/b10c/kartograf/data/1704463200/rpki/tals/ripe.tal
Downloading RPKI Data
...finished in 0:03:09.905992

--- Fetching IRR ---

Downloading afrinic.db.gz
Downloading apnic.db.route.gz
Downloading apnic.db.route6.gz
Downloading arin.db.gz
Downloading lacnic.db.gz
Downloading ripe.db.route.gz
Downloading ripe.db.route6.gz
...finished in 0:00:36.541110

--- Fetching Routeviews pfx2as ---

Downloading https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2024/01/routeviews-rv2-20240103-1200.pfx2as.gz
Downloading https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2024/01/routeviews-rv6-20240104-1200.pfx2as.gz
...finished in 0:00:07.256778

--- Validating RPKI ---

Validating RPKI ROAs
212003 raw RKPI ROA files found.
100%|██████████████████████████████████████████████████████████████████████████| 212003/212003 [18:46<00:00, 188.22it/s]
212003 RKPI ROAs validated and saved to /home/b10c/kartograf/out/1704463200/rpki/rpki_raw.json
...finished in 0:18:57.690965

--- Parsing RPKI ---

Parsing 212003 ROAs
Result entries written: 451864
Duplicates found: 56623
Invalids found: 23066
Incompletes: 0
Non-ROA files: 0
...finished in 0:01:30.792939

--- Parsing IRR ---

Extracting afrinic.db.gz
Extracting apnic.db.route.gz
Extracting apnic.db.route6.gz
Extracting arin.db.gz
Extracting lacnic.db.gz
Extracting ripe.db.route.gz
Extracting ripe.db.route6.gz
Parsing /home/b10c/kartograf/out/1704463200/irr/afrinic.db
Parsing /home/b10c/kartograf/out/1704463200/irr/apnic.db.route
Parsing /home/b10c/kartograf/out/1704463200/irr/apnic.db.route6
Parsing /home/b10c/kartograf/out/1704463200/irr/arin.db
Parsing /home/b10c/kartograf/out/1704463200/irr/lacnic.db
Parsing /home/b10c/kartograf/out/1704463200/irr/ripe.db.route
Parsing /home/b10c/kartograf/out/1704463200/irr/ripe.db.route6
Found valid, unique entries: 1419403
...finished in 0:05:35.192046

--- Merging RPKI and IRR data ---

Parse base file to numpy arrays
Parse extra file to Pandas DataFrame
Filtering extra prefixes that were already included in the base file:

 100.00% :::::::::::::::::::::::::::::::::::::::: |   236568 /   236568 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   236567 /   236567 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   236567 /   236567 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   236567 /   236567 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   236567 /   236567 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   236567 /   236567 |

Finished filtering! Originally 1419403 entries filtered down to 394036
Merging base file with filtered extra file
...finished in 1:45:43.123389

--- Parsing Routeviews pfx2as ---

Unzipping /home/b10c/kartograf/data/1704463200/collectors/routeviews_pfx2asn_ip4.txt.gz
Formatting /home/b10c/kartograf/out/1704463200/collectors/routeviews_pfx2asn_ip4.txt
Unzipping /home/b10c/kartograf/data/1704463200/collectors/routeviews_pfx2asn_ip6.txt.gz
Formatting /home/b10c/kartograf/out/1704463200/collectors/routeviews_pfx2asn_ip6.txt
Cleaning /home/b10c/kartograf/out/1704463200/collectors/pfx2asn.txt
...finished in 0:03:21.305320

--- Merging Routeviews and base data ---

Parse base file to numpy arrays
Parse extra file to Pandas DataFrame
Filtering extra prefixes that were already included in the base file:

 100.00% :::::::::::::::::::::::::::::::::::::::: |   198215 /   198215 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   198215 /   198215 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   198215 /   198215 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   198215 /   198215 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   198215 /   198215 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   198214 /   198214 |

Finished filtering! Originally 1189289 entries filtered down to 410310
Merging base file with filtered extra file
...finished in 2:47:23.942468

--- Sorting results ---

...finished in 0:00:18.885259

--- Finishing Kartograf ---

The SHA-256 hash of the result file is: 4aba172195b8dc375c2bb1b39c3508f48c212cedc02983a7dc794a373820653b
Total runtime: 5:06:46.250158

I got 3d42c9c376ebcbe2e21ddf1ba9951e7940dbbf7e6e1f7df9bc182b57d760d768.

--- Start Kartograf ---
Kartograf version: 0.4.1
Using rpki-client version 8.6.
Coordinated launch mode: Waiting until<PHONE_NUMBER> (2024-01-05 14:00:00 WET) to launch mapping process.
Countdown: 0 second(s)
Starting...
The epoch for this run is:<PHONE_NUMBER> (2024-01-05 14:00:00 UTC, local: 2024-01-05 14:00:00 WET)
--- Fetching RPKI ---
Downloaded TAL for AFRINIC to /home/base/code/kartograf/data/1704463200/rpki/tals/afrinic.tal
Downloaded TAL for APNIC to /home/base/code/kartograf/data/1704463200/rpki/tals/apnic.tal
Downloaded TAL for ARIN to /home/base/code/kartograf/data/1704463200/rpki/tals/arin.tal
Downloaded TAL for LACNIC to /home/base/code/kartograf/data/1704463200/rpki/tals/lacnic.tal
Downloaded TAL for RIPE to /home/base/code/kartograf/data/1704463200/rpki/tals/ripe.tal
Downloading RPKI Data
...finished in 0:03:37.514900
--- Fetching IRR ---
Downloading afrinic.db.gz
Downloading apnic.db.route.gz
Downloading apnic.db.route6.gz                                                                                [70/112]
Downloading arin.db.gz
Downloading lacnic.db.gz
Downloading ripe.db.route.gz
Downloading ripe.db.route6.gz
...finished in 0:01:48.509536
--- Fetching Routeviews pfx2as ---
Downloading https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2024/01/routeviews-rv2-20240103-1200.pfx2as.gz
Downloading https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2024/01/routeviews-rv6-20240104-1200.pfx2as.gz
...finished in 0:00:19.151505
--- Validating RPKI ---
Validating RPKI ROAs
211735 raw RKPI ROA files found.
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 211735/211735 [24:50<00:00, 142.02it/s]
211735 RKPI ROAs validated and saved to /home/base/code/kartograf/out/1704463200/rpki/rpki_raw.json
...finished in 0:25:05.523637
--- Parsing RPKI ---
Parsing 211735 ROAs
Result entries written: 451209
Duplicates found: 56435
Invalids found: 23060
Incompletes: 0
Non-ROA files: 0
...finished in 0:01:27.220209
--- Parsing IRR ---
Extracting afrinic.db.gz
Extracting apnic.db.route.gz
Extracting apnic.db.route6.gz                                                                                [35/112]
Extracting arin.db.gz
Extracting lacnic.db.gz
Extracting ripe.db.route.gz
Extracting ripe.db.route6.gz
Parsing /home/base/code/kartograf/out/1704463200/irr/afrinic.db
Parsing /home/base/code/kartograf/out/1704463200/irr/arin.db
Parsing /home/base/code/kartograf/out/1704463200/irr/ripe.db.route6
Parsing /home/base/code/kartograf/out/1704463200/irr/apnic.db.route6
Parsing /home/base/code/kartograf/out/1704463200/irr/lacnic.db
Parsing /home/base/code/kartograf/out/1704463200/irr/ripe.db.route
Parsing /home/base/code/kartograf/out/1704463200/irr/apnic.db.route
Found valid, unique entries: 1419403
...finished in 0:05:21.492319
--- Merging RPKI and IRR data ---
Parse base file to numpy arrays
Parse extra file to Pandas DataFrame
Filtering extra prefixes that were already included in the base file:
100.00% :::::::::::::::::::::::::::::::::::::::: |   177426 /   177426 |
100.00% :::::::::::::::::::::::::::::::::::::::: |   177426 /   177426 |
100.00% :::::::::::::::::::::::::::::::::::::::: |   177426 /   177426 |
100.00% :::::::::::::::::::::::::::::::::::::::: |   177425 /   177425 |
100.00% :::::::::::::::::::::::::::::::::::::::: |   177425 /   177425 |
100.00% :::::::::::::::::::::::::::::::::::::::: |   177425 /   177425 |
100.00% :::::::::::::::::::::::::::::::::::::::: |   177425 /   177425 |
100.00% :::::::::::::::::::::::::::::::::::::::: |   177425 /   177425 |
Finished filtering! Originally 1419403 entries filtered down to 395754
Merging base file with filtered extra file
...finished in 2:40:31.195831
--- Parsing Routeviews pfx2as ---
Unzipping /home/base/code/kartograf/data/1704463200/collectors/routeviews_pfx2asn_ip4.txt.gz
Formatting /home/base/code/kartograf/out/1704463200/collectors/routeviews_pfx2asn_ip4.txt
Unzipping /home/base/code/kartograf/data/1704463200/collectors/routeviews_pfx2asn_ip6.txt.gz
Formatting /home/base/code/kartograf/out/1704463200/collectors/routeviews_pfx2asn_ip6.txt
Cleaning /home/base/code/kartograf/out/1704463200/collectors/pfx2asn.txt
...finished in 0:03:14.562903
--- Merging Routeviews and base data ---
Parse base file to numpy arrays
Parse extra file to Pandas DataFrame
Filtering extra prefixes that were already included in the base file:
100.00% :::::::::::::::::::::::::::::::::::::::: |   148662 /   148662 |
100.00% :::::::::::::::::::::::::::::::::::::::: |   148661 /   148661 |
100.00% :::::::::::::::::::::::::::::::::::::::: |   148661 /   148661 |
100.00% :::::::::::::::::::::::::::::::::::::::: |   148661 /   148661 |
100.00% :::::::::::::::::::::::::::::::::::::::: |   148661 /   148661 |
100.00% :::::::::::::::::::::::::::::::::::::::: |   148661 /   148661 |
100.00% :::::::::::::::::::::::::::::::::::::::: |   148661 /   148661 |
100.00% :::::::::::::::::::::::::::::::::::::::: |   148661 /   148661 |
Finished filtering! Originally 1189289 entries filtered down to 410773
Merging base file with filtered extra file
...finished in 4:31:07.391481
--- Sorting results ---
...finished in 0:00:19.157710
--- Finishing Kartograf ---
The SHA-256 hash of the result file is: 3d42c9c376ebcbe2e21ddf1ba9951e7940dbbf7e6e1f7df9bc182b57d760d768
Total runtime: 7:52:53.590652

@jurraca can you upload your final_results.txt here?
Mine is here final_result.txt.gz

I've got a match on: 4aba172195b8dc375c2bb1b39c3508f48c212cedc02983a7dc794a373820653b
Total runtime: 14:56:59.428796 (slow i3 thin client)

I also got 4aba172195b8dc375c2bb1b39c3508f48c212cedc02983a7dc794a373820653b.
Total runtime: 2:38:01.512775 which is 30 minutes faster than last time.

@jurraca can you upload your final_results.txt here?

here it is final_result.txt.gz

@jurraca can you upload your final_results.txt here?

here it is final_result.txt.gz

Alright. Yeah, that's a bigger diff. Don't have the time to step through it in detail.
I've noticed that there are overlapping mappings to the same AS in my final_results.txt file. For example:
<IP_ADDRESS>/16 AS49666
<IP_ADDRESS>/24 AS49666

The /16 should already cover the /24 mapping.

I've noticed that there are overlapping mappings to the same AS in my final_results.txt file. For example:
<IP_ADDRESS>/16 AS49666
<IP_ADDRESS>/24 AS49666

The /16 should already cover the /24 mapping.

Yeah, this is an interesting observation. We can look into doing this as a final step but we need to think about an algorithm to do this efficiently. One edge case that we would need to handle as well is if the file had an additional entry that is from a different AS and is between the two entries in terms of specificity, then we would not be able to do this, i.e. we would also need to check that no entry like <IP_ADDRESS>/20 AS1337 exists. We can probably do this reasonably easily after the file result file is sorted. I will check how often we see candidates to decide if this is worth the effort. Another small downside is that this new final result file becomes a bit harder to debug.

Alright, nice, so we had 4/5 for the 4aba172195b8dc375c2bb1b39c3508f48c212cedc02983a7dc794a373820653b hash. @brunoerg did you get a result?
@jurraca was the dissenting hash and as @0xB10C said it's a pretty big diff this time. I could see in the logs that there are certainly differences coming out of RPKI again. So this is kind of the situation that I described above: we have a majority for one result and I would personally be fine with recommending the 4/5 hash to run it in their node based on this. It would still be a great idea to develop tools to check what input files caused the diff explicitly and what we can learn from that but actually figuring out which of these 1000+ differing lines would be preferred will take unreasonably long with the manpower available to us.
I am still very happy with these results :) These are two runs in a row were we just had one dissenting result which makes me more confident that this process is a viable solution. It was also not the same person that dissented so it doesn't seem to be a systemic issue afaict.
I will spend some time documenting the findings now. I will also open a PR here to demo what it would look like to make a compressed result part of the repo. Please let me know if you have any further feedback on the tools, process etc.!

Here is the demo PR for the finalized asmap.dat file: https://github.com/fjahr/asmap-data/pull/6
Would be great to get some feedback from 1-2 of you whether you receive the same result and the process makes sense like this. Thanks you!

I've noticed that there are overlapping mappings to the same AS in my final_results.txt file. For example:
<IP_ADDRESS>/16 AS49666
<IP_ADDRESS>/24 AS49666

The /16 should already cover the /24 mapping.

Yeah, this is an interesting observation. We can look into doing this as a final step but we need to think about an algorithm to do this efficiently. One edge case that we would need to handle as well is if the file had an additional entry that is from a different AS and is between the two entries in terms of specificity, then we would not be able to do this, i.e. we would also need to check that no entry like <IP_ADDRESS>/20 AS1337 exists. We can probably do this reasonably easily after the file result file is sorted. I will check how often we see candidates to decide if this is worth the effort. Another small downside is that this new final result file becomes a bit harder to debug.

I took a quick look. Radix trees are commonly used as data structures in network routing for use-cases where you have entries with different network prefixes. There's a python package py-radix implementing a radix tree for python network addresses.
When building a radix tree from the addresses in final_results.txt, I check each prefix if there is already a prefix covering it. This seems to work well. However, I noticed there are a lot of cases where the smaller subnets have a different prefix than the subnet covering it.
import ipaddress
import radix

with open("final_result.txt", 'r') as f:
    rtree = radix.Radix()

    for line in f.readlines():
        ip_str, asn = line.strip().split(" ")
        #ip = ipaddress.ip_network(ip_str)

        # check if there is already a prefix covering the current ip in the RadixTree
        for rnode in rtree.search_covering(ip_str):
            if asn != rnode.data:
                print(f"{rnode.prefix} already covers {ip_str}, but {ip_str} has a different ASN of {asn} compared to {rnode.data}")
        if len(rtree.search_covering(ip_str)) == 0:
            rnode = rtree.add(ip_str)
            rnode.data = asn

This prints 293720 lines similar to the following. The final_results.txt contains 1256210 entries. So about 23% are overlapping in some way.
<IP_ADDRESS>/19 (AS9737) already covers <IP_ADDRESS>/24 (AS23969), but <IP_ADDRESS>/24 has a different ASN!
<IP_ADDRESS>/18 (AS9737) already covers <IP_ADDRESS>/24 (AS23969), but <IP_ADDRESS>/24 has a different ASN!
<IP_ADDRESS>/17 (AS9737) already covers <IP_ADDRESS>/24 (AS23969), but <IP_ADDRESS>/24 has a different ASN!
<IP_ADDRESS>/22 (AS9583) already covers <IP_ADDRESS>/24 (AS132215), but <IP_ADDRESS>/24 has a different ASN!
<IP_ADDRESS>/22 (AS9583) already covers <IP_ADDRESS>/24 (AS132215), but <IP_ADDRESS>/24 has a different ASN!
<IP_ADDRESS>/22 (AS9583) already covers <IP_ADDRESS>/24 (AS4755), but <IP_ADDRESS>/24 has a different ASN!
<IP_ADDRESS>/22 (AS9583) already covers <IP_ADDRESS>/24 (AS132215), but <IP_ADDRESS>/24 has a different ASN!
<IP_ADDRESS>/22 (AS9583) already covers <IP_ADDRESS>/24 (AS132215), but <IP_ADDRESS>/24 has a different ASN!
<IP_ADDRESS>/22 (AS9583) already covers <IP_ADDRESS>/24 (AS132215), but <IP_ADDRESS>/24 has a different ASN!
<IP_ADDRESS>/22 (AS9583) already covers <IP_ADDRESS>/24 (AS132215), but <IP_ADDRESS>/24 has a different ASN!
...

I don't know (as I'm still not familiar enough with asmap) if we want to prefer the larger prefixes or the smaller ones.

When building a radix tree from the addresses in final_results.txt, I check each prefix if there is already a prefix covering it. This seems to work well. However, I noticed there are a lot of cases where the smaller subnets have a different ASN than the subnet covering it.

AFAIK the more specific (smaller) subnet is always used for routing.

After taking a look at #6 and playing around with the asmap-tool, it seems that the tool takes care of it. After encoding and decoding the file, only the smaller (fewer IPs) prefixes seems to remain. E.g.:
... (from above)
<IP_ADDRESS>/22 (AS9583) already covers <IP_ADDRESS>/24 (AS132215), but <IP_ADDRESS>/24 has a different ASN!
<IP_ADDRESS>/22 (AS9583) already covers <IP_ADDRESS>/24 (AS132215), but <IP_ADDRESS>/24 has a different ASN!

# in asmap.txt (decoded asmap.dat)
<IP_ADDRESS>/24 AS132215
<IP_ADDRESS>/24 AS132215

I don't think you'd need to implement any deduplication or de-overlapping here then.

Alright, nice, so we had 4/5 for the 4aba172195b8dc375c2bb1b39c3508f48c212cedc02983a7dc794a373820653b hash. @brunoerg did you get a result?

Same hash: 4aba172195b8dc375c2bb1b39c3508f48c212cedc02983a7dc794a373820653b

Thanks everyone for participating! This has been a great success! The collaborative launch feature has worked far better than I expected and we found and fixed several issues along the way.
I am closing this now but would like to do another one tomorrow in about 24 hours: https://github.com/fjahr/asmap-data/issues/7 I would be very happy if as many of you as possible could join again. My goal with this one is to have a 'clean' demo that makes it easier to grasp what the process will look like in the future when not as many issues arise as within this one. Upon success, I want use it as an example to show what we have achieved here :)
@brunoerg @Sjors @0xB10C @Emzy @jurraca


Brokers MUST support deleting instances with active bindings

Following #127 and per this week's call, brokers MUST support deletion of service instances that have active bindings by doing a cascade delete.
Since this change is backwards incompatible, we will defer it to v3.

Added this to the Ideas for v3 wiki page. Closing here.


Flutter driver integration test unable to scroll DropdownButton items

I'm unable to find the Scrollable of the DropdownButton after using widgetTester.tap() using find.byType(Scrollable);. I'm following this guide.
Steps to Reproduce

Test
testWidgets("Test DropdownButton", (WidgetTester widgetTester) async {
  await widgetTester.pumpWidget(MyApp());
  final dropdownButtonFinder = find.byKey(const ValueKey('DropdownButton'));
  final dropdownItemFinder = find.textContaining('Item 10');

  // Tap on the DropdownButton
  await widgetTester.tap(dropdownButtonFinder);
  final dropdownListFinder = find.byType(Scrollable);  // Doesn't find any Scrollable

  // Scroll until the item to be found appears. Doesn't work for DropdownButton
  await widgetTester.scrollUntilVisible(dropdownItemFinder, 500.0,
     scrollable: dropdownListFinder);
  await widgetTester.tap(dropdownItemFinder);

  // Verify that the item contains the correct text.
  expect(dropdownItemFinder, findsOneWidget);
});

Run the test for the sample app with flutter driver
Error from the test finding the Scrollable

══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞═════════════════
The following StateError was thrown running a test:
Bad state: No element

When the exception was thrown, this was the stack:
#0      Iterable.single (dart:core/iterable.dart:520:25)
#1      WidgetController.widget (package:flutter_test/src/controller.dart:69:30)
#2      WidgetController.scrollUntilVisible.<anonymous closure> (package:flutter_test/src/controller.dart:1192:15)
#3      WidgetController.scrollUntilVisible.<anonymous closure> (package:flutter_test/src/controller.dart:1190:39)
#6      TestAsyncUtils.guard (package:flutter_test/src/test_async_utils.dart:71:41)
#7      WidgetController.scrollUntilVisible (package:flutter_test/src/controller.dart:1190:27)
#8      main.<anonymous closure> (file:///Users/Downloads/dev/flutter/sampletest/integration_test/sample_test.dart:74:24)
<asynchronous suspension>
#9      testWidgets.<anonymous closure>.<anonymous closure> (package:flutter_test/src/widget_tester.dart:176:15)
<asynchronous suspension>
#10     TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:817:5)
<asynchronous suspension>
(elided 2 frames from dart:async)

Expected results:
Flutter test should be able to find the Scrollable from DropdownButton after widgetTester.tap()
Actual results:
Integration test returns Bad state: No element error looking for the Scrollable. Skipping the part of looking for a Scrollable and just using find.textContaining(String) to find the Dropdown item works fine though.

Code sample

import 'package:flutter/material.dart';

void main() {
  runApp(MyApp());
}

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Demo',
      theme: ThemeData(
        primarySwatch: Colors.blue,
      ),
      home: MyHomePage(title: 'Flutter Demo Home Page'),
    );
  }
}

class MyHomePage extends StatefulWidget {
  MyHomePage({Key? key, required this.title}) : super(key: key);

  final String title;

  @override
  _MyHomePageState createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  List<String> mockList() => List<String>.generate(100, (i) => 'Item $i');
  String? dropdownValue;

  @override
  Widget build(BuildContext context) {
    // debugPrint('${foo!}');
    return Scaffold(
      appBar: AppBar(
        title: Text(widget.title),
        key: Key('Widget1'),
      ),
      body: Center(
        child: DropdownButton(
          key: Key('DropdownButton'),
          value: dropdownValue,
          onChanged: (String? newValue) {
            setState(() {
              dropdownValue = newValue!;
            });
          },
          items: mockList()
              .map<DropdownMenuItem<String>>(
                (String value) => DropdownMenuItem<String>(
                  value: value,
                  child: Text(value),
                  key: Key('item$value'),
                ),
              )
              .toList(),
      )
    );
  }
}

Logs

[✓] Flutter (Channel stable, 2.5.3, on macOS 12.0.1 21A559 darwin-x64)
    • Flutter version 2.5.3
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 18116933e7 (9 weeks ago), 2021-10-15 10:46:35 -0700
    • Engine revision d3ea636dc5
    • Dart version 2.14.4

[!] Android toolchain - develop for Android devices (Android SDK version 30.0.2)
    • Android SDK at /Android/sdk
    ✗ cmdline-tools component is missing
      Run `path/to/sdkmanager --install "cmdline-tools;latest"`
      See https://developer.android.com/studio/command-line for more details.
    ✗ Android license status unknown.
      Run `flutter doctor --android-licenses` to accept the SDK licenses.
      See https://flutter.dev/docs/get-started/install/macos#android-setup for more details.

[✓] Xcode - develop for iOS and macOS
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 13.1, Build version 13A1030d
    • CocoaPods version 1.10.0

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2020.3)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.10+0-b96-7281165)

[✓] VS Code (version 1.63.0)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.27.0

! Doctor found issues in 1 category.
Process finished with exit code 0

Hi @omatt, this doesn't seem like the right way to test a DropdownButton, see links below for more information
https://stackoverflow.com/a/64496868
https://github.com/flutter/flutter/blob/master/packages/flutter/test/material/dropdown_test.dart
https://github.com/flutter/flutter/issues/89905
Please let me know if this works for you.
Thank you.

Hi @mamuseferha thanks for providing these references. However, upon checking, the issues on the references you've shared doesn't have the scrolling issue I've mentioned. Let me know on how I can apply the references you've shared to solve this issue.
Note that find.textContaining('Item 10'); works fine if I skipped find.byType(Scrollable); after widgetTester.tap(dropdownButtonFinder);. So I can seem to find DropdownButton items, just not the scrolling part.

Hi @omatt, I tested this on the latest stable and master channel and it is reproducible. Dropdown button test is unable to find a scrollable when tapped.

logs

══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following StateError was thrown running a test:
Bad state: No element

When the exception was thrown, this was the stack:
#0      Iterable.single (dart:core/iterable.dart:654:25)
#1      WidgetController.widget (package:flutter_test/src/controller.dart:69:30)
#2      WidgetController.scrollUntilVisible.<anonymous closure> (package:flutter_test/src/controller.dart:1190:15)
#3      WidgetController.scrollUntilVisible.<anonymous closure> (package:flutter_test/src/controller.dart:1188:39)
#6      TestAsyncUtils.guard (package:flutter_test/src/test_async_utils.dart:71:41)
#7      WidgetController.scrollUntilVisible (package:flutter_test/src/controller.dart:1188:27)
#8      main.<anonymous closure> (file:///Users/flo/projects/oos/flutter_master/test/main_test.dart:17:24)
<asynchronous suspension>
<asynchronous suspension>
(elided 3 frames from dart:async and package:stack_trace)

The test description was:
  Test DropdownButton
════════════════════════════════════════════════════════════════════════════════════════════════════
Test failed. See exception logs above.
The test description was: Test DropdownButton

✖ Test DropdownButton
Exited (1)

code sample

import 'package:flutter/material.dart';

void main() {
  runApp(MyApp());
}

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Demo',
      theme: ThemeData(
        primarySwatch: Colors.blue,
      ),
      home: MyHomePage(title: 'Flutter Demo Home Page'),
    );
  }
}

class MyHomePage extends StatefulWidget {
  MyHomePage({Key? key, required this.title}) : super(key: key);

  final String title;

  @override
  _MyHomePageState createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  List<String> mockList() => List<String>.generate(100, (i) => 'Item $i');
  String? dropdownValue;

  @override
  Widget build(BuildContext context) {
    // debugPrint('${foo!}');
    return Scaffold(
        appBar: AppBar(
          title: Text(widget.title),
          key: Key('Widget1'),
        ),
        body: Center(
            child: DropdownButton(
          key: Key('DropdownButton'),
          value: dropdownValue,
          onChanged: (String? newValue) {
            setState(() {
              dropdownValue = newValue!;
            });
          },
          items: mockList()
              .map<DropdownMenuItem<String>>(
                (String value) => DropdownMenuItem<String>(
                  value: value,
                  child: Text(value),
                  key: Key('item$value'),
                ),
              )
              .toList(),
        )));
  }
}

flutter doctor -v
[✓] Flutter (Channel stable, 2.8.1, on macOS 11.6.1 20G224 darwin-x64, locale en-US)
    • Flutter version 2.8.1 at /Users/flo/development/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 77d935af4d (18 hours ago), 2021-12-16 08:37:33 -0800
    • Engine revision 890a5fca2e
    • Dart version 2.15.1

[✓] Android toolchain - develop for Android devices (Android SDK version 31.0.0)
    • Android SDK at /Users/flo/Library/Android/sdk
    • Platform android-31, build-tools 31.0.0
    • Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.10+0-b96-7281165)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 13.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • CocoaPods version 1.11.0

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2020.3)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.10+0-b96-7281165)

[✓] VS Code (version 1.62.3)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.29.0

[✓] Connected device (4 available)
    • Android SDK built for x86 (mobile) • emulator-5554                        • android-x86    • Android 11 (API 30)
      (emulator)
    • iPhone 13 (mobile)                 • EF1251D3-C9C4-475E-9F91-58CD010FDA48 • ios            •
      com.apple.CoreSimulator.SimRuntime.iOS-15-0 (simulator)
    • macOS (desktop)                    • macos                                • darwin-x64     • macOS 11.6.1 20G224
      darwin-x64
    • Chrome (web)                       • chrome                               • web-javascript • Google Chrome 93.0.4577.63

• No issues found!

[✓] Flutter (Channel master, 2.9.0-1.0.pre.123, on macOS 11.6.1 20G224 darwin-x64, locale en-US)
    • Flutter version 2.9.0-1.0.pre.123 at /Users/flo/fvm/versions/master
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 5087f2de4c (5 hours ago), 2021-12-17 00:29:09 -0500
    • Engine revision 9177b32b75
    • Dart version 2.16.0 (build 2.16.0-118.0.dev)
    • DevTools version 2.9.1

[✓] Android toolchain - develop for Android devices (Android SDK version 31.0.0)
    • Android SDK at /Users/flo/Library/Android/sdk
    • Platform android-31, build-tools 31.0.0
    • Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.10+0-b96-7281165)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 13.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • CocoaPods version 1.11.0

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2020.3)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.10+0-b96-7281165)

[✓] VS Code (version 1.62.3)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.29.0

[✓] Connected device (4 available)
    • Android SDK built for x86 (mobile) • emulator-5554                        • android-x86    • Android 11 (API 30) (emulator)
    • iPhone 13 (mobile)                 • EF1251D3-C9C4-475E-9F91-58CD010FDA48 • ios            • com.apple.CoreSimulator.SimRuntime.iOS-15-0 (simulator)
    • macOS (desktop)                    • macos                                • darwin-x64     • macOS 11.6.1 20G224 darwin-x64
    • Chrome (web)                       • chrome                               • web-javascript • Google Chrome 93.0.4577.63

[✓] HTTP Host Availability
    • All required HTTP hosts are available

• No issues found!

Thank you

Hi @omatt. This is working as intended. The popup list, and hence the Scrollable is not a part of the tree unless the DropdownButton is tapped and the animation completes. To test it, you have to wait after tapping it for the animation to complete, and then you'll be able to locate the Scrollable. Lemme know if this resolves your issue.
testWidgets("Test DropdownButton", (WidgetTester widgetTester) async {
  await widgetTester.pumpWidget(MyApp());
  final dropdownButtonFinder = find.byKey(const ValueKey('DropdownButton'));
  final dropdownItemFinder = find.textContaining('Item 10');

  // Tap on the DropdownButton
  await widgetTester.tap(dropdownButtonFinder);

  await tester.pumpAndSettle(); // wait for the dropdown animation to complete
  expect(find.byType(Scrollable), findsOneWidget); // expectation is met

  final dropdownListFinder = find.byType(Scrollable);

  // Scroll until the item to be found appears. Doesn't work for DropdownButton
  await widgetTester.scrollUntilVisible(dropdownItemFinder, 500.0,
     scrollable: dropdownListFinder);
  await widgetTester.tap(dropdownItemFinder);

  // Verify that the item contains the correct text.
  expect(dropdownItemFinder, findsOneWidget);
});

Thanks! Adding widgetTester.pumpAndSettle() worked for me and pointed me to the right direction. I'm now able to scroll on the List popup from the DropdownButton. Though oddly find.textContaining(String) doesn't seem to work for Text on List Items. I had to use find.widgetWithText(InkWell, String).
testWidgets("Test DropdownButton", (WidgetTester widgetTester) async {
    await widgetTester.pumpWidget(MyApp());
    final dropdownButtonFinder = find.byKey(const ValueKey('DropdownButton'));
    final dropdownItemFinder = find.widgetWithText(InkWell, 'Item 50'); // find.textContaining() doesn't seem to work

    // Tap on the DropdownButton
    await widgetTester.tap(dropdownButtonFinder);
    await widgetTester.pumpAndSettle(const Duration(seconds: 2));
    final dropdownListFinder = find.byType(Scrollable);
    expect(dropdownListFinder, findsOneWidget); // Finds Scrollable from tapping DropDownButton

    // Scroll until the item to be found appears.
    await widgetTester.scrollUntilVisible(dropdownItemFinder, 500.0,
        scrollable: dropdownListFinder);
    await widgetTester.tap(dropdownItemFinder);
    await widgetTester.pumpAndSettle(const Duration(seconds: 2));

    // Verify that the item contains the correct text.
    expect(find.textContaining('Item 50'), findsOneWidget);
  });


⚠ Processors has degraded performance

In 7810cc9, Processors ($STATUS_URL) experienced degraded performance:

HTTP code: 200
Response time: 46 ms

Resolved: Processors performance has improved in 43f0ede.


[Question]: if I have setup windows once ; how can i ignore setup ,direct start windows

Is your question not already answered in the FAQ?

[X] I made sure the question is not listed in the FAQ.

Is this a general question and not a technical issue?

[X] I am sure my question is not about a technical issue.

Question
if I have setup windows once ; how can i ignore setup ,direct start windows and auto run install.bat

very thank you , If I configure a startup program to run install.bat, can  install.bat run when windows up ? or how can I to achieve the same effect as I did with entrypoint.sh inside the linux container.

@kroese What if you what to run it in rke2 cluster? Is it possible to build a docker image that will directly start windows on each run ?

@anglersking  "The installed Windows and all programs/settings are stored in a container in the /storage folder. You can fully customize Windows as needed and then copy the /storage folder from the container to your local host (docker cp container_id:/storage /local_storage). Subsequently, you can simply mount the /local_storage volume to a 'clean' container (docker run -it -p 8006:8006 -p 3389:3389 -v /local_storage:/storage --cap-add NET_ADMIN --device=/dev/kvm dockurr/windows), and you'll have a fully configured Windows environment with your changes, without needing a new installation."

can I not use -v , I try dockerfile  Copy local storage to docker /storage but not work , download windows in /staroage/tmp again

can I not use -v , I try dockerfile Copy local storage to docker /storage but not work  @evyaroshevich

我不能使用 -v 吗，我尝试将 dockerfile 本地存储复制到 docker /storage 但失效 @evyaroshevich
哥们你解决了 我也遇到了这个问题 求帮助

要重新build 一个镜像  ， 把原始的dockerfile 里面的 这个去掉 VOLUME /storage  ，就生效了，  VOLUME /storage 是个挂载点 ，每次run docker 会清空 。 但实测还有另一个问题，就是没有-v 启动的块，除了启动时间其他效果是和-v一样的 。

要重新build 一个镜像 ， 把原始的dockerfile 里面的 这个去掉 VOLUME /storage ，就生效了， VOLUME /storage 是个挂载点 ，每次run docker 会清空 。 但实测还有另一个问题，就是没有-v 启动的块，除了启动时间其他效果是和-v一样的 。

我把VOLUME /storage去掉后，启动容器，在容器里执行./entry.sh后报错


Problems encountered when using multiple Gpus for training

Dear author, I encountered this problem when using two gpu. How to solve this problem?
(zq) omnisky@node01:/data01/zq/CaDDN/tools$ python -m torch.distributed.launch --nproc_per_node=2 train.py --launcher pytorch --batch_size 2 --cfg_file cfgs/kitti_models/CaDDN.yaml

Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.

Traceback (most recent call last):
File "train.py", line 197, in
main()
File "train.py", line 72, in main
assert args.batch_size % total_gpus == 0, 'Batch size should match the number of gpus'
AssertionError: Batch size should match the number of gpus
Traceback (most recent call last):
File "train.py", line 197, in
main()
File "train.py", line 72, in main
assert args.batch_size % total_gpus == 0, 'Batch size should match the number of gpus'
AssertionError: Batch size should match the number of gpus
Traceback (most recent call last):
File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/omnisky/zq/lib/python3.6/site-packages/torch/distributed/launch.py", line 263, in
main()
File "/home/omnisky/zq/lib/python3.6/site-packages/torch/distributed/launch.py", line 259, in main
cmd=cmd)
subprocess.CalledProcessError: Command '['/home/omnisky/zq/bin/python3', '-u', 'train.py', '--local_rank=1', '--launcher', 'pytorch', '--batch_size', '2', '--cfg_file', 'cfgs/kitti_models/CaDDN.yaml']' returned non-zero exit status 1.
(zq) omnisky@node01:/data01/zq/CaDDN/tools$ python -m torch.distributed.launch --nproc_per_node=2 train.py --launcher pytorch --batch_size 2 --cfg_file cfgs/kitti_models/CaDDN.yaml^C

i MET IT TOO


Basler pypylon with detectnet

Hi,
Does anyone know how to connect a balser camera to detectnet?
I found this on the nvidia forum but their github page doesnt exist anymore
https://forums.developer.nvidia.com/t/how-make-inference-with-basler-camera/111803/8
Do i need to somehow convert the image from pypylon to a cudaImage?

Hi there, any solutions here?


Merge TLS 1.1 deprecation message to live

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

sysinternals/Announce/TLSDeprecation.md
:white_check_mark:Succeeded
View

sysinternals/index.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Added emacs note to editor setup

Fixes #1505

This is great.
But: This is not an official editor supported by us, so we should probably mention that this is a community thing that might work or not. This also means, the other editors do not have the same features. It should definitely not be listed on the same level as VSCode which we developer and spend a lot of effort on.

You know what, i was actually going to ask exactly this question. I can add a community supported section and list this there

We btw. have language server working on Emacs, thanks to @pantharshit00. So, it seems Emacs is now BETTER than VSCode :)

If we start adding community extensions here, we also might want to include the one for Jetbrains IDEs: https://plugins.jetbrains.com/plugin/14240-prisma

Looks good to me. :) Left 2 comments, otherwise SHIIIIPP IIIITTT


MDX endpoint is inconsistent

The /mdx endpoint sends application/json Content-type but returns text. Backbone can't read the returned data.


Updates wifi to not require password, allows security setting

Fixes #373
Fixes #393

@HipsterBrown see comments but this PR looks great!

@hipsterbrown I've also reviewed as promised, but I can't sign off until I've done a smoke test. I will tackle that this morning and post my findings here.

@rwaldron Thanks. I'll be going through @johnnyman727's comments tonight and push another update then.

Reality check/smoke tests (using an open network that setup with a wifi extender):

Connect to a wifi network which has no security

$ t2 wifi --ssid NETGEAR07
INFO Looking for your Tessel...
INFO Connected to bishop.
INFO Wifi Enabled.
INFO Wifi connection successful. SSID: NETGEAR07

On successful connection:
1. SSH to Tessel
ssh<EMAIL_ADDRESS>-i ~/.tessel/id_rsa

1. Confirm connectivity

  ```
  Tessel 2  /  Built on OpenWrt
  root@bishop:~# ping google.com
  PING google.com (<IP_ADDRESS>): 56 data bytes
  64 bytes from <IP_ADDRESS>: seq=0 ttl=251 time=16.320 ms
  64 bytes from <IP_ADDRESS>: seq=1 ttl=251 time=14.481 ms
  64 bytes from <IP_ADDRESS>: seq=2 ttl=251 time=17.040 ms
  64 bytes from <IP_ADDRESS>: seq=3 ttl=251 time=15.660 ms
  64 bytes from <IP_ADDRESS>: seq=4 ttl=251 time=16.740 ms
  ^C
  --- google.com ping statistics ---
  5 packets transmitted, 5 packets received, 0% packet loss
  round-trip min/avg/max = 14.481/16.048/17.040 ms
  ```
1. Disconnect

  ```
  root@bishop:~# exit
  Connection to bishop.local closed.
  ```

On failure, report to PR/issue
Connect to a wifi network which requires security (password, mode)

$ t2 wifi -n JLY4Q_EXT -p XCCGM3C43SC7JVDX
INFO Looking for your Tessel...
INFO Connected to bishop.
INFO Wifi Enabled.
INFO Wifi connection successful. SSID: JLY4Q_EXT, password XCCGM3C43SC7JVDX, security mode: psk2

On successful connection:
1. SSH to Tessel
ssh<EMAIL_ADDRESS>-i ~/.tessel/id_rsa

1. Confirm connectivity

  ```
  Tessel 2  /  Built on OpenWrt
  root@bishop:~# ping google.com
  PING google.com (<IP_ADDRESS>): 56 data bytes
  64 bytes from <IP_ADDRESS>: seq=0 ttl=251 time=14.480 ms
  64 bytes from <IP_ADDRESS>: seq=1 ttl=251 time=15.220 ms
  64 bytes from <IP_ADDRESS>: seq=2 ttl=251 time=14.641 ms
  64 bytes from <IP_ADDRESS>: seq=3 ttl=251 time=16.061 ms
  64 bytes from <IP_ADDRESS>: seq=4 ttl=251 time=13.720 ms
  ^C
  --- google.com ping statistics ---
  5 packets transmitted, 5 packets received, 0% packet loss
  round-trip min/avg/max = 13.720/14.824/16.061 ms
  ```
1. Disconnect

  ```
  root@bishop:~# exit
  Connection to bishop.local closed.
  ```

On failure, report to PR/issue

As you can see, there were no issues encountered—great work!

Reality check/smoke tests (using a secure network setup with a wifi extender):

Setup wifi using appropriate security mode:

rwaldron at new-host-6 in ~/clonez/t2-cli on HipsterBrown-wifi-no-pass
$ t2 wifi --ssid NETGEAR07 --password coolcanoe323 --security psk2
INFO Looking for your Tessel...
INFO Connected to bishop.
INFO Wifi Enabled.
INFO Wifi connection successful. SSID: NETGEAR07, password coolcanoe323, security mode: psk2

On successful connection:
1. Check wifi:
rwaldron at new-host-6 in ~/clonez/t2-cli on HipsterBrown-wifi-no-pass
$ t2 wifi
INFO Looking for your Tessel...
INFO Connected to bishop.
INFO Connected to "NETGEAR07"
INFO IP Address: <IP_ADDRESS>
INFO Signal Strength: (70/70)
INFO Bitrate: 58mbps

1. Confirm connectivity; SSH & Re-confirm

  ```
  rwaldron at new-host-6 in ~/clonez/t2-cli on HipsterBrown-wifi-no-pass
  $ ping <IP_ADDRESS>
  PING <IP_ADDRESS> (<IP_ADDRESS>): 56 data bytes
  64 bytes from <IP_ADDRESS>: icmp_seq=0 ttl=64 time=12.919 ms
  64 bytes from <IP_ADDRESS>: icmp_seq=1 ttl=64 time=17.741 ms
  64 bytes from <IP_ADDRESS>: icmp_seq=2 ttl=64 time=23.953 ms
  64 bytes from <IP_ADDRESS>: icmp_seq=3 ttl=64 time=12.022 ms
  64 bytes from <IP_ADDRESS>: icmp_seq=4 ttl=64 time=6.467 ms
  ^C
  --- <IP_ADDRESS> ping statistics ---
  5 packets transmitted, 5 packets received, 0.0% packet loss
  round-trip min/avg/max/stddev = 6.467/14.620/23.953/5.881 ms
  rwaldron at new-host-6 in ~/clonez/t2-cli on HipsterBrown-wifi-no-pass
  $ ssh<EMAIL_ADDRESS>-i ~/.tessel/id_rsa

  BusyBox v1.23.2 (2015-10-27 13:38:31 UTC) built-in shell (ash)

  Tessel 2  /  Built on OpenWrt
  root@bishop:~# ping google.com
  PING google.com (<IP_ADDRESS>): 56 data bytes
  64 bytes from <IP_ADDRESS>: seq=0 ttl=251 time=9.900 ms
  64 bytes from <IP_ADDRESS>: seq=1 ttl=251 time=20.880 ms
  64 bytes from <IP_ADDRESS>: seq=2 ttl=251 time=13.440 ms
  64 bytes from <IP_ADDRESS>: seq=3 ttl=251 time=20.660 ms
  64 bytes from <IP_ADDRESS>: seq=4 ttl=251 time=13.980 ms
  ^C
  --- google.com ping statistics ---
  5 packets transmitted, 5 packets received, 0% packet loss
  round-trip min/avg/max = 9.900/15.772/20.880 ms
  root@bishop:~# exit
  Connection to bishop.local closed.
  ```

On failure, report to PR/issue

Setup wifi using another appropriate security mode:

rwaldron at new-host-6 in ~/clonez/t2-cli on HipsterBrown-wifi-no-pass
$ t2 wifi --ssid NETGEAR07 --password coolcanoe323 --security psk
INFO Looking for your Tessel...
INFO Connected to bishop.
INFO Wifi Enabled.
INFO Wifi connection successful. SSID: NETGEAR07, password coolcanoe323, security mode: psk

On successful connection:
1. Check wifi:
rwaldron at new-host-6 in ~/clonez/t2-cli on HipsterBrown-wifi-no-pass
$ t2 wifi
INFO Looking for your Tessel...
INFO Connected to bishop.
INFO Connected to "NETGEAR07"
INFO IP Address: <IP_ADDRESS>
INFO Signal Strength: (70/70)
INFO Bitrate: 52mbps

1. Confirm connectivity; SSH & Re-confirm

  ```
  rwaldron at new-host-6 in ~/clonez/t2-cli on HipsterBrown-wifi-no-pass
  $ ping <IP_ADDRESS>
  PING <IP_ADDRESS> (<IP_ADDRESS>): 56 data bytes
  64 bytes from <IP_ADDRESS>: icmp_seq=0 ttl=64 time=93.706 ms
  64 bytes from <IP_ADDRESS>: icmp_seq=1 ttl=64 time=8.907 ms
  64 bytes from <IP_ADDRESS>: icmp_seq=2 ttl=64 time=26.671 ms
  64 bytes from <IP_ADDRESS>: icmp_seq=3 ttl=64 time=24.951 ms
  64 bytes from <IP_ADDRESS>: icmp_seq=4 ttl=64 time=11.153 ms
  ^C
  --- <IP_ADDRESS> ping statistics ---
  5 packets transmitted, 5 packets received, 0.0% packet loss
  round-trip min/avg/max/stddev = 8.907/33.078/93.706/31.138 ms
  rwaldron at new-host-6 in ~/clonez/t2-cli on HipsterBrown-wifi-no-pass
  $ ssh<EMAIL_ADDRESS>-i ~/.tessel/id_rsa

  BusyBox v1.23.2 (2015-10-27 13:38:31 UTC) built-in shell (ash)

  Tessel 2  /  Built on OpenWrt
  root@bishop:~# ping google.com
  PING google.com (<IP_ADDRESS>): 56 data bytes
  64 bytes from <IP_ADDRESS>: seq=0 ttl=251 time=18.300 ms
  64 bytes from <IP_ADDRESS>: seq=1 ttl=251 time=24.661 ms
  64 bytes from <IP_ADDRESS>: seq=2 ttl=251 time=17.900 ms
  64 bytes from <IP_ADDRESS>: seq=3 ttl=251 time=11.480 ms
  64 bytes from <IP_ADDRESS>: seq=4 ttl=251 time=15.340 ms
  ^C
  --- google.com ping statistics ---
  5 packets transmitted, 5 packets received, 0% packet loss
  round-trip min/avg/max = 11.480/17.536/24.661 ms
  root@bishop:~# exit
  Connection to bishop.local closed.
  ```

On failure, report to PR/issue

@HipsterBrown looking good! Ping me when you've finished up smoke tests :)

@rwaldron @johnnyman727 I ran into some issues last night while testing this on my Tessel. ubus listen never received any data, so the process would hang even though the Tessel had successfully connected.
I tried replacing commands.ubusListen() with commands.getWifiInfo() and listening for ssid in the output because that means the connection was successful. Unfortunately there are some side effects to this:

OpenWRT is very forgiving about incorrect wifi settings and still outputs through stdout
After a bad connection, the wifi needs to be disconnected before attempting to reconnect with new credentials

For the bad connection issue, I've tried recursively running setWifiState to disable the connection rather than just resolving with info that the credentials are bad and telling the user to run t2 wifi --off before trying new credentials. I will push what I have and would love some feedback on how to proceed.

Is that an issue with "connecting to wifi without password", or with "confirming wifi connection state in general"?

@rwaldron Confirming wifi connection state in general. The Tessel will still connect or disconnect, just the check I'm having trouble with.

ubus listen never received any data, so the process would hang even though the Tessel had successfully connected.

In what scenarios does this happen? Can you write up steps-to-reproduce? I'm using your branch right now and it appears to be working correctly?

Also, could you just use ping? All you'd need is the t2's name, then: ping t2-name.local. That appears to work in the quick tests I just did.

Unrelated, can we make this:
t2 wifi --off
t2 wifi --on

Match t2 ap on, t2 ap off? In another issue, of course.

@rwaldron I just pushed a new update that resolved the issues I was running into when using commands.getWifiInfo() instead of ubusListen().
I thought about trying ping as well but wanted to see if I could use the existing commands first.

Oh, in testing ping, I think I found a way to reproduce:
t2 wifi --off

The light goes out, but the operation just hangs:
rwaldron at nova in ~/clonez/particle-io on master*
$ t2 wifi --off
INFO Looking for your Tessel...
INFO Connected to bishop.
^CINFO Wifi Disabled.

Notice I had to manually sigint

Just tested with latest changes and this works perfectly.

@HipsterBrown can you update the docs for t2 wifi? Thanks and awesome work!

Sure thing. I'll do that and the ap docs + tutorial. :D

:clap: :+1:


Generate documentation for the ansible test-playbooks in e2e

The ansible based CI automation code is growing, with multiple YAMLs getting added into the repo. This necessitates a mechanism to generate some basic usage docs for potential users and contributors.
Identify some available open-source options to achieve this, like :
Ansible-docgen is an opensource tool which generates documentation from annotated ansible playbooks - https://github.com/starboarder2001/ansible-docgen

Experiment docs are now available via docs.litmuschaos.io


Is it possible to add own commands?

I'm wanting to add custom commands to this bot. Which file do I need to edit to do so? Is it even possible?

Hey! There is currently no official support for this. Feel free to edit main.js or one of the files under modules to edit/add commands.


Do not run browserstack for external users (security reasons)

See https://docs.travis-ci.com/user/pull-requests/#Pull-Requests-and-Security-Restrictions

This change is

Review status: 0 of 3 files reviewed at latest revision, all discussions resolved, some commit checks failed.

Comments from Reviewable


Submit to Godot offical asset resources

Submit!
This looks amazing.

Thanks for your positive feedback & suggestion.
I have just now submitted this to the Godot Asset Library so should hopefully eventually appear here: https://godotengine.org/asset-library/asset?user=rancidbacon
Although I just realised maybe you meant: https://github.com/godotengine/godot-demo-projects?


fix(scheduler): apply healthcheck to the right application container instead of assuming

Right now there is an assumption that the first container in a Pod is the Deis application container. While not wrong it is better to find the right container and only apply to it
Fixes #518

Current coverage is 82.45%

Merging #564 into master will not affect coverage as of 31ad180

@@            master    #564   diff @@
======================================
  Files           36      36
  Stmts         2303    2308     +5
  Branches       331     333     +2
  Methods          0       0
======================================
+ Hit           1899    1903     +4
- Partial        115     116     +1
  Missed         289     289

Review entire Coverage Diff as of 31ad180

Powered by Codecov. Updated on successful CI builds.

Aren't the healthchecks done over HTTP?  All containers in a pod share a common IP as if they are one logical machine, so this shouldn't matter... unless you are also exposing a different healthcheck option, like running some certain command.

Right now it is HTTP but later it may not. It was an easy change to tack this on to the right container

I like the defensiveness of this change :)


Gracefully handle errors when fetching data

Create a modal, or a popup to inform a user that something went wrong while fetching the data.

This was fixed in this commit.


Add button Add Dataset in list page

fixes #188
For now, the UI is not very good but I will improve it while working on UI tickets

works as expected, with all tests passing


Fallback for missing register-level

In the SVD for stm32f103[1] there are registers where the <access> is declared per-field rather than for the whole register.
This caused nil instead of Register to be emitted:
pub const cr = nil(cr_val).init(0x40021000 + 0x0);

This change makes those register be treated like read-write even though some of the contained fields are read-only.
[1] https://www.st.com/resource/en/svd/stm32f1_svd.zip

Thanks for the patch! I think it'd be better to see if we can come up with an API that more honestly represents what the SVD tells us re: read/write access on field-by-field basis. I once I had to debug a bad register API and don't want to risk inflicting that on anyone who might be writing to a read-only register and then wondering why nothing is happening =D
Let's discuss in https://github.com/lynaghk/svd2zig/issues/2


Debugger skips breakpoints in C# projects

Environment data
dotnet --info output:
.NET Core SDK (reflecting any global.json):
Version:   3.1.201
Commit:    b1768b4ae7
Runtime Environment:
OS Name:     Windows
OS Version:  10.0.18363
OS Platform: Windows
RID:         win10-x86
Base Path:   C:\Program Files (x86)\dotnet\sdk\3.1.201\
Host (useful for support):
Version: 3.1.3
Commit:  4a9f85e9f8
.NET Core SDKs installed:
3.1.201 [C:\Program Files (x86)\dotnet\sdk]
.NET Core runtimes installed:
Microsoft.AspNetCore.App 3.1.3 [C:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 3.1.3 [C:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 3.1.3 [C:\Program Files (x86)\dotnet\shared\Microsoft.WindowsDesktop.App]
To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download
VS Code version:
1.45.1 x64
C# Extension version:
1.21.18
OmniSharp log
Starting OmniSharp server at 5/28/2020, 5:16:46 PM
Target: ***\HelloWorld
OmniSharp server started.
Path: ***.vscode\extensions\ms-dotnettools.csharp-1.21.18.omnisharp\1.35.1\OmniSharp.exe
PID: 3772
    Starting OmniSharp on Windows 6.2.9200.0 (x64)

    DotNetPath set to dotnet

    Located 1 MSBuild instance(s)
        1: StandAlone 16.4 - "***\.vscode\extensions\ms-dotnettools.csharp-1.21.18\.omnisharp\1.35.1\.msbuild\Current\Bin"

    MSBUILD_EXE_PATH environment variable set to '***\.vscode\extensions\ms-dotnettools.csharp-1.21.18\.omnisharp\1.35.1\.msbuild\Current\Bin\MSBuild.exe'

    Registered MSBuild instance: StandAlone 16.4 - "***\.vscode\extensions\ms-dotnettools.csharp-1.21.18\.omnisharp\1.35.1\.msbuild\Current\Bin"
        CscToolExe = csc.exe
        CscToolPath = ***\.vscode\extensions\ms-dotnettools.csharp-1.21.18\.omnisharp\1.35.1\.msbuild\Current\Bin\Roslyn
        MSBuildExtensionsPath = ***\.vscode\extensions\ms-dotnettools.csharp-1.21.18\.omnisharp\1.35.1\.msbuild
        MSBuildToolsPath = ***\.vscode\extensions\ms-dotnettools.csharp-1.21.18\.omnisharp\1.35.1\.msbuild\Current\Bin

    Detecting Cake files in '***\HelloWorld'.

    Could not find any Cake files

    No solution files found in '***\HelloWorld'

    Queue project update for '***\HelloWorld\HelloWorld.csproj'

    Detecting CSX files in '***\HelloWorld'.

    Could not find any CSX files

    Invoking Workspace Options Provider: OmniSharp.Roslyn.CSharp.Services.CSharpFormattingWorkspaceOptionsProvider, Order: 0

    Loading project: d:\programming\HelloWorld\HelloWorld.csproj

    Invoking Workspace Options Provider: OmniSharp.Roslyn.CSharp.Services.RenameWorkspaceOptionsProvider, Order: 100

    Invoking Workspace Options Provider: OmniSharp.Roslyn.CSharp.Services.ImplementTypeWorkspaceOptionsProvider, Order: 110

    Invoking Workspace Options Provider: OmniSharp.Roslyn.CSharp.Services.BlockStructureWorkspaceOptionsProvider, Order: 140

    Configuration finished.

    Omnisharp server running using Stdio at location '***\HelloWorld' on host 8256.

    Successfully loaded project file '***\HelloWorld\HelloWorld.csproj'.

    Adding project '***\HelloWorld\HelloWorld.csproj'

    Update project: HelloWorld

Steps to reproduce
Open cmd
mkdir HelloWorld
cd HelloWorld
dotnet new console
code .
put breakpoint on line 10
hit F5
Expected  behavior
VSCode should stop at the line
Actual behavior
DEBUG CONSOLE tab shows
"You may only use the Microsoft .NET Core Debugger (vsdbg) with
Visual Studio Code, Visual Studio or Visual Studio for Mac software
to help you develop and test your applications."

Unfortunately, VS Code has a bug where it doesn't display error messages. It will be fixed in their next release. You can turn on debugger logging to see what error is being returned.
I believe the problem is that you have the x86 .NET Core SDK installed on your machine and earlier in the path than the x64 .NET Core SDK (assuming you have both). This extension only supports debugging x64 processes on Windows.

Here is the debugger's log:
`-> (C) {"command":"initialize","arguments":{"clientID":"vscode","clientName":"Visual Studio Code","adapterID":"coreclr","pathFormat":"path","linesStartAt1":true,"columnsStartAt1":true,"supportsVariableType":true,"supportsVariablePaging":true,"supportsRunInTerminalRequest":true,"locale":"en-us","supportsProgressReporting":true},"type":"request","seq":1}
-> (C) {"command":"launch","arguments":{"name":".NET Core Launch (console)","type":"coreclr","request":"launch","preLaunchTask":"build","program":"\HelloWorld/bin/Debug/netcoreapp3.1/HelloWorld.dll","args":[],"cwd":"\HelloWorld","console":"internalConsole","stopAtEntry":false,"logging":{"engineLogging":false},"internalConsoleOptions":"openOnSessionStart","__sessionId":"5e32a8c9-097b-4d28-bdfc-77a9a380d448"},"type":"request","seq":2}
<- (E) {"seq":3,"type":"event","event":"output","body":{"category":"console","output":"-------------------------------------------------------------------\nYou may only use the Microsoft .NET Core Debugger (vsdbg) with\nVisual Studio Code, Visual Studio or Visual Studio for Mac software\nto help you develop and test your applications.\n-------------------------------------------------------------------\n"}}
You may only use the Microsoft .NET Core Debugger (vsdbg) with
Visual Studio Code, Visual Studio or Visual Studio for Mac software
to help you develop and test your applications.
<- (R) {"seq":4,"type":"response","request_seq":2,"success":true,"command":"launch"}
<- (E) {"seq":5,"type":"event","event":"initialized","body":{}}
-> (C) {"command":"setBreakpoints","arguments":{"source":{"name":"Program.cs","path":"***\HelloWorld\Program.cs","checksums":[{"algorithm":"SHA1","checksum":"3fb673958d99b05299acc895a52b30de6fde8dbe"},{"algorithm":"SHA256","checksum":"5344d406136a1fff8ddae8b327736d9beac318a4f9b67cbdeb48cba43465efde"},{"algorithm":"SHA1","checksum":"b438947097df5c09d068f6368eecdd41e4d0940b"},{"algorithm":"SHA256","checksum":"cda8030706f1ab035ae682fd437995b93c15160925e598cba88aaa915f726514"}]},"lines":[10],"breakpoints":[{"line":10}],"sourceModified":false},"type":"request","seq":3}
<- (R) {"seq":6,"type":"response","request_seq":3,"success":true,"command":"setBreakpoints","message":"","body":{"breakpoints":[{"id":1,"verified":false,"message":"The breakpoint is pending and will be resolved when debugging starts.","line":10}]}}
-> (C) {"command":"setExceptionBreakpoints","arguments":{"filters":["user-unhandled"]},"type":"request","seq":4}
<- (R) {"seq":7,"type":"response","request_seq":4,"success":true,"command":"setExceptionBreakpoints"}
-> (C) {"command":"configurationDone","type":"request","seq":5}
<- (E) {"seq":8,"type":"event","event":"output","body":{"category":"telemetry","output":"VS/Diagnostics/Debugger/vsdbg/LaunchFailed","data":{"VS.Diagnostics.Debugger.vsdbg.OSFamily":"Windows","VS.Diagnostics.Debugger.vsdbg.Version":"16.6.20415.1 commit:0184bb70e1ed40656c0dfa53125805036207007b","VS.Diagnostics.Debugger.vsdbg.WindowsVersion":"10.0.18363","VS.Diagnostics.Debugger.vsdbg.ErrorCode":-2147220965}}}
<- (R) {"seq":9,"type":"response","request_seq":5,"success":false,"command":"configurationDone","message":"Only 64-bit processes can be debugged."}
-> (C) {"command":"disconnect","arguments":{"restart":false},"type":"request","seq":6}
-> (C) {"command":"threads","type":"request","seq":7}
<- (E) {"seq":10,"type":"event","event":"output","body":{"category":"telemetry","output":"VS/Diagnostics/Debugger/vsdbg/DebugCompleted","data":{"VS.Diagnostics.Debugger.vsdbg.OSFamily":"Windows","VS.Diagnostics.Debugger.vsdbg.Version":"16.6.20415.1 commit:0184bb70e1ed40656c0dfa53125805036207007b","VS.Diagnostics.Debugger.vsdbg.WindowsVersion":"10.0.18363","VS.Diagnostics.Debugger.vsdbg.DebugCompleted.BreakCounter":0,"VS.Diagnostics.Debugger.vsdbg.AdapterId":"coreclr"}}}
<- (R) {"seq":11,"type":"response","request_seq":6,"success":true,"command":"disconnect"}
`
Thank you for the help. Much appreciated. The problem was the x86 SDK.
Two questions:

Why there is not information that only x64 processes can be debugged?
Is there a plan for support of x86 processes?

1: This is the VS Code bug that I alluded to. You should have seen this message:
<- (R) {"seq":9,"type":"response","request_seq":5,"success":false,"command":"configurationDone","message":"Only 64-bit processes can be debugged."}
2: There are no current plans.


Use dnspython library instead of dig command

Then this hook can be used where dig is not available, like on a synology server.

Thanks for your contribution ;)


[Hold #11768] mWeb- Connect Bank Account - Pressing back button at android in the middle of a step takes you back to the workspace page instead of going back to the previous step @Puneet-here

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

Go to staging.new.expensify.com and log in
Go to bank account > add info > company step
Press back button of mobile

Expected Result:
ou should be navigated back to the previous step
Actual Result:
You will be navigated to the workspace page
Workaround:
Unknow
Platform:

Where is this issue occurring?

Web
iOS
Android
Desktop App
Mobile Web

Version Number: <IP_ADDRESS>
Reproducible in staging?: Yes
Reproducible in production?: Yes
Email or phone of affected tester (no customers): any
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation
https://user-images.githubusercontent.com/93399543/193102567-8d812de1-7932-4f37-9cfb-d59b118b642c.mov
Expensify/Expensify Issue URL:
Issue reported by: @Puneet-here
Slack conversation: https://expensify.slack.com/archives/C01GTK53T8Q/p1663792588717789
View all open jobs on GitHub

I don't have an android phone, but worked with @pecanoro to reproduce this in an android device.

Discussing ideal behavior for how this should work here.

In the meantime, going to pass to engineering to review. Also, based on this post, whatever solution we do decide on is going to be on hold pending completion of this issue.

Demoting priority and adding to project: https://github.com/Expensify/App/projects/2

I'm not able to reproduce this on web staging (v1.2.18-2) or on my iPhone app (v1.2.15-3). I am successfully taken back to the previous step. Are we sure this is still broken?

Hi @tjferriss, it's still reproducible. You have to repro this on Android and you will need to press native back button.

@MariaHCD I'm looking for your advice here based on your previous comments. If this is still reproducible should I toss it back into the engineering pool?

@tjferriss I see that it is reproducible if you use the native back button on an Android device. However, I don't believe we have a solution for this yet and should continue to remain on hold as part of https://github.com/Expensify/App/issues/11768. I think there's no need for another engineer to take a look at this.

Ok we'll hold on for now :)

Still on hold

Snagging this now that I'm on the tracking issue.

Still on hold for the tracking issue.

Thanks Melvin, though we're still on hold as noted above. Adding the weekly label instead.

Still on hold.

Still held as we go through the design process.

On hold per the larger react navigation initiative.

Still on hold for the larger navigation re-write.

Still on hold for navigation.

Held on navigation revamp.

Switching out weekly for monthly given that the navigation project is large and has no defined ETA.

Still on hold for the navigation reboot.

Same same

Still on hold.

Still on hold.

Coming off hold very soon!

Working on testing this week.

Still trying to make time. Low priority.

@Puneet-here do you mind re-testing this one and confirming that it's no longer an issue? At that point we can pay you for reporting the issue.

It's still reproducible.

Thanks! @mountiny is the BBA flow potentially a special case that we didn't cover for the react-navigation project? Perhaps just as relevant: What's the expected behavior for a multi-step flow in a modal? Just thinking generally on this, I'm kind of doubting that each step has it's own route, and thus back would correctly close the modal.

Yeah this is because each step is most likely not its own screen. I think however we can close this since this project will probably change this completely https://docs.google.com/document/d/1t8_Fn93LSSU4gG4z7WRt8IFcpSGK4TIe4QamdcTXjxE/edit?usp=sharing

Ok cool. I'll still go ahead and pay out @Puneet-here for reporting, since we will fix this.

Offer sent. Please accept and we can get this closed out @Puneet-here!

Accepted!

All paid out! Thank you, have a great weekend.


It will crash while touched the androidview with three fingers

Steps to Reproduce
Device: Android 8.1

Touch on AndroidView with 3 fingers
AndroidView crashed in background

Logs
E/flutter (26622): [ERROR:flutter/lib/ui/ui_dart_state.cc(148)] Unhandled Exception: NoSuchMethodError: The method '_asList' was called on null.

Yes, similar with #25137, but crash trace are not same, reproduced while phone three-finger screenshot enabled, and unable to reproduce while disabled three-finger screenshot.

Closing as dupe of #20517.


add correlation id

Is your feature request related to a problem? Please describe.
It is a new behaviour
Describe the solution you'd like
add an id share in all logs
Describe alternatives you've considered
We could use the uuid from the play resource
Additional context
N/A

done in #45