How many REFINE

All of them (hopefully).
$ rg --no-ignore -U '^(mp-)?(im|ex)port:.*(\n[+\s]+.*)*\bREFINE\b.*(\n[+\s]+.*)*'
priority/altdb.db
21993:mp-import:  afi any.unicast  from AS13911 accept ANY AND NOT {<IP_ADDRESS>/0, ::0/0};
21994:            REFINE
21995:            afi ipv4.unicast from AS13911 action pref = 200; accept <^AS13911 AS6327+$>;
22003:mp-import:  afi ipv4.unicast {
22004:            from AS14595:AS-CUSTOMER action pref = 200; accept PeerAS;
22005:            } REFINE afi ipv4.unicast {
22006:            from AS20161 action pref = 200; accept AS20161 AS19915;
22007:            from AS30500 accept {<IP_ADDRESS>/24,<IP_ADDRESS>/23};
22008:            }

backup/altdb.db
22024:mp-import:  afi any.unicast  from AS13911 accept ANY AND NOT {<IP_ADDRESS>/0, ::0/0};
22025:            REFINE
22026:            afi ipv4.unicast from AS13911 action pref = 200; accept <^AS13911 AS6327+$>;
22034:mp-import:  afi ipv4.unicast {
22035:            from AS14595:AS-CUSTOMER action pref = 200; accept PeerAS;
22036:            } REFINE afi ipv4.unicast {
22037:            from AS20161 action pref = 200; accept AS20161 AS19915;
22038:            from AS30500 accept {<IP_ADDRESS>/24,<IP_ADDRESS>/23};
22039:            }

priority/ripe.db
69197:import:         from AS20965 <IP_ADDRESS> action community.append(5408:1001); accept NOT community.contains(5408:1002) AND NOT community.contains(5408:1003) AND NOT fltr-martian; REFINE { from AS20965 action aspath.prepend(AS20965,AS20965,AS20965); accept community.contains(20965:7777); from AS20965 accept ANY; }
69198:import:         from AS20965 <IP_ADDRESS> action community.append(5408:1001); med=20; accept NOT community.contains(5408:1002) AND NOT community.contains(5408:1003) AND NOT fltr-martian; REFINE { from AS20965 action aspath.prepend(AS20965,AS20965,AS20965); accept community.contains(20965:7777); from AS20965 accept ANY; }
69201:export:         to AS20965 announce AS5408:RS-ROUTES; REFINE { to AS20965 <IP_ADDRESS> action community.append(20965:7030); announce ANY; to AS20965 <IP_ADDRESS> action community.append(20965:7030); announce ANY; to AS20965 <IP_ADDRESS> announce ANY; to AS20965 <IP_ADDRESS> announce ANY; }
69202:export:         to AS20965 announce community.contains(5408:1003) AND NOT community.contains(5408:4005); REFINE { to AS20965 announce (AS5408:AS-TO-GEANT AND NOT AS5408:RS-ROUTES^0-32); to AS20965 announce AS5408:RS-ROUTES^0-32 AND community.contains(5408:2000); } REFINE { to AS20965 <IP_ADDRESS> action community.append(20965:7030); announce ANY; to AS20965 <IP_ADDRESS> action community.append(20965:7030); announce ANY; to AS20965 <IP_ADDRESS> announce ANY; to AS20965 <IP_ADDRESS> announce ANY; }
69219:import:         from prng-as5408-grix-peers action pref=105; community.append(5408:1002); accept NOT community.contains(5408:1001) AND NOT community.contains(5408:1003) AND <^PeerAS+AS5408:AS-GRIX:PeerAS*$>; REFINE { from AS-GRIX at <IP_ADDRESS> action med=0; accept ANY; from AS-GRIX at <IP_ADDRESS> action med=20; accept ANY; } # AND (PeerAS^0-27 OR AS5408:AS-GRIX:PeerAS^0-27 OR AS5408:AS-GRIX:PeerAS^0-27); Primary-Backup
69221:export:         to prng-as5408-grix-peers announce community.contains(5408:1003) AND NOT community.contains(5408:4001); REFINE { to prng-as5408-grix-peers announce (AS5408:AS-TO-AIX AND NOT AS5408:RS-ROUTES^0-32); to prng-as5408-grix-peers announce AS5408:RS-ROUTES^0-32 AND community.contains(5408:2000); }
69231:import:         from AS199399 action pref=105; community.append(5408:1002); accept AS199399; REFINE { from AS-GRIX at <IP_ADDRESS> action med=0; accept ANY; from AS-GRIX at <IP_ADDRESS> action med=20; accept ANY; } # Primary-Backup
69240:import:         from AS25152 action pref=105; community.append(5408:1002); community.append(5408:4001); community.append(5408:4005); accept {<IP_ADDRESS>/24, <IP_ADDRESS>/28} AND NOT community.contains(5408:1001) AND NOT community.contains(5408:1003); REFINE { from AS-GRIX at <IP_ADDRESS> action med=0; accept ANY; from AS-GRIX at <IP_ADDRESS> action med=20; accept ANY; } # Primary-Backup
69266:import:         from prng-as5408-grnet-customers action community.append(5408:1003); accept NOT community.contains(5408:1001) AND NOT community.contains(5408:1002) AND community.contains(NO_EXPORT) AND (PeerAS^0-32 OR PeerAS:AS-TO-GRNET^0-32) AND <^PeerAS+PeerAS:AS-TO-GRNET*$>; REFINE { from AS-ANY action pref=110; accept community.contains(5408:110); from AS-ANY action pref=100; accept ANY; }
69267:import:         from prng-as5408-grnet-customers action community.append(5408:1003); accept NOT community.contains(5408:1001) AND NOT community.contains(5408:1002) AND (PeerAS OR PeerAS:AS-TO-GRNET) AND <^PeerAS+PeerAS:AS-TO-GRNET*$>; REFINE { from AS-ANY action aspath.prepend(AS5408,AS5408,AS5408,AS5408,AS5408); accept community.contains(5408:2005); from AS-ANY accept ANY; } REFINE { from AS-ANY action pref=110; accept community.contains(5408:110); from AS-ANY action pref=100; accept ANY; }
69282:import:         from prng-as5408-customers-eu-academic-internet action community.append(5408:1003); community.append(5408:2000); community.append(5408:4001); community.append(20965:0000); community.append(20965:0011); community.append(20965:0012); community.append(20965:0013); community.append(20965:7000); accept NOT community.contains(5408:1001) AND NOT community.contains(5408:1002) AND community.contains(NO_EXPORT) AND (PeerAS^0-32 OR PeerAS:AS-TO-GRNET^0-32) AND <^PeerAS+PeerAS:AS-TO-GRNET*$>; REFINE { from AS-ANY action pref=110; accept community.contains(5408:110); from AS-ANY action pref=100; accept ANY; }
69283:import:         from prng-as5408-customers-eu-academic-internet action community.append(5408:1003); community.append(5408:2000); community.append(5408:4001); community.append(20965:0000); community.append(20965:0011); community.append(20965:0012); community.append(20965:0013); community.append(20965:7000); accept NOT community.contains(5408:1001) AND NOT community.contains(5408:1002) AND (PeerAS OR PeerAS:AS-TO-GRNET) AND <^PeerAS+PeerAS:AS-TO-GRNET*$>; REFINE { from AS-ANY action aspath.prepend(AS5408,AS5408,AS5408,AS5408,AS5408); accept community.contains(5408:2005); from AS-ANY accept ANY; } REFINE { from AS-ANY action pref=110; accept community.contains(5408:110); from AS-ANY action pref=100; accept ANY; }
69320:export:         to AS65535 announce community.contains(5408:1003); REFINE { to AS65535 announce (AS5408:AS-GRNET-CUSTOMERS-PUBLIC-AS AND NOT AS5408:RS-ROUTES^0-32); to AS65535 announce (AS5408:AS-GRNET-CUSTOMERS-PRIVATE-AS AND NOT AS5408:RS-ROUTES^0-32); }
349333:mp-import:      afi ipv6.unicast { from AS2914 action community.delete(15562:1100, 15562:1101,15562:1102, 15562:1200,15562:1201, 15562:1300,15562:1301,15562:1302, 15562:2100,15562:2101,15562:2102); community.append(15562:1300); accept fltr-iana-allocated-v6 AND NOT fltr-martian-v6 AND NOT fltr-bogons-v6 AND NOT AS-SNIJDERS^+; } REFINE afi ipv6.unicast { from AS2914 2001:9e0:810:1::10 at 2a02:898:52:ffff::204 action community.append(15562:1301); accept ANY; from AS2914 2001:9e0:411:1::10 at 2a02:898:52:ffff::204 action community.append(15562:1302); accept ANY; }
349334:mp-export:      afi ipv6.unicast { to AS2914 announce AS-SNIJDERS AND NOT community(NO_EXPORT); } REFINE afi ipv6.unicast { to AS2914 2001:9e0:810:1::10 at 2a02:898:52:ffff::204 action community.append(15562:1301); announce ANY; to AS2914 2001:9e0:411:1::10 at 2a02:898:52:ffff::204 action community.append(15562:1302); announce ANY; }
7468537:mp-export:      afi ipv4.unicast to AS6799 announce NOT fltr-martian;
7468538:                REFINE {
7468539:                to AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { 6799:4080 }; announce AS29247:RS-ROUTES-n;
7468540:                to AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { 6799:4150 }; announce AS29247:RS-ROUTES-r;
7468541:                to AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { 6799:4150 }; announce AS29247:RS-ROUTES-a;
7468542:                to AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { no_export }; announce AS29247:RS-ROUTES-nne;
7468543:                to AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { 6799:4080 }; announce AS29247:RS-ROUTES-n;
7468544:                to AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { 6799:4150 }; announce AS29247:RS-ROUTES-r;
7468545:                to AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { 6799:4150 }; announce AS29247:RS-ROUTES-a;
7468546:                to AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { no_export }; announce AS29247:RS-ROUTES-nne;
7468547:                }
7468550:mp-import:      afi ipv4.unicast from AS6799 action med=100 ;
7468551:                accept NOT fltr-martian AND NOT { <IP_ADDRESS>/0 } AND NOT { <IP_ADDRESS>/0^25-32 } ;
7468552:                REFINE {
7468553:                from AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { 29247:50073 , no_export }; accept ANY;
7468554:                from AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { 29247:50073 , no_export }; accept ANY;
7468555:                }
7468559:mp-export:      afi ipv4.unicast to AS6799 announce NOT fltr-martian;
7468560:                REFINE {
7468561:                to AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { 6799:4150 }; announce AS29247:RS-ROUTES-n;
7468562:                to AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { 6799:4080 }; announce AS29247:RS-ROUTES-r;
7468563:                to AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { 6799:4150 }; announce AS29247:RS-ROUTES-a;
7468564:                to AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { no_export }; announce AS29247:RS-ROUTES-rne;
7468565:                to AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { 6799:4150 }; announce AS29247:RS-ROUTES-n;
7468566:                to AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { 6799:4080 }; announce AS29247:RS-ROUTES-r;
7468567:                to AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { 6799:4150 }; announce AS29247:RS-ROUTES-a;
7468568:                to AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { no_export }; announce AS29247:RS-ROUTES-rne;
7468569:                }
7468572:mp-import:      afi ipv4.unicast from AS6799 action med=100 ;
7468573:                accept NOT fltr-martian AND NOT { <IP_ADDRESS>/0 } AND NOT { <IP_ADDRESS>/0^25-32 } ;
7468574:                REFINE {
7468575:                from AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { 29247:50035 , no_export }; accept any;
7468576:                from AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { 29247:50035 , no_export }; accept any;
7468577:                }
7468581:mp-export:      afi ipv4.unicast to AS6799 announce NOT fltr-martian;
7468582:                REFINE {
7468583:                to AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { 6799:4150 }; announce AS29247:RS-ROUTES-n;
7468584:                to AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { 6799:4150 }; announce AS29247:RS-ROUTES-r;
7468585:                to AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { 6799:4080 }; announce AS29247:RS-ROUTES-a;
7468586:                to AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { no_export }; announce AS29247:RS-ROUTES-ane;
7468587:                to AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { 6799:4150 }; announce AS29247:RS-ROUTES-n;
7468588:                to AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { 6799:4150 }; announce AS29247:RS-ROUTES-r;
7468589:                to AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { 6799:4080 }; announce AS29247:RS-ROUTES-a;
7468590:                to AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { no_export }; announce AS29247:RS-ROUTES-ane;
7468591:                }
7468594:mp-import:      afi ipv4.unicast from AS6799 action med=100 ;
7468595:                accept NOT fltr-martian AND NOT { <IP_ADDRESS>/0 } AND NOT { <IP_ADDRESS>/0^25-32 } ;
7468596:                REFINE {
7468597:                from AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { 29247:51010 , no_export }; accept any;
7468598:                from AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { 29247:51010 , no_export }; accept any;
7468599:                }
7468606:mp-export:      afi ipv4.unicast to AS6799 announce NOT fltr-martian; REFINE { to AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { 12713:8600 }; announce AS29247:RS-ROUTES-n; to AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { 6799:4150 }; announce AS29247:RS-ROUTES-r; to AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { 12713:8600 }; announce AS29247:RS-ROUTES-a; }
7468609:mp-import:      afi ipv4.unicast from AS6799 action med=2 ; accept NOT fltr-martian AND NOT { <IP_ADDRESS>/0 } AND NOT { <IP_ADDRESS>/0^25-32} ; REFINE { from as6799 <IP_ADDRESS> at <IP_ADDRESS> action community ={29247:40073 , no_export}; accept any; }
7468613:mp-export:      afi ipv4.unicast to AS6799 announce NOT fltr-martian; REFINE { to AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { 6799:4150 }; announce AS29247:RS-ROUTES-n; to AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { 12713:8600 }; announce AS29247:RS-ROUTES-r; to AS6799 <IP_ADDRESS> at <IP_ADDRESS> action community = { 12713:8600 }; announce AS29247:RS-ROUTES-a; }
58143435:mp-import:      afi ipv4.unicast { from AS199284 action community = { 64628:10 }; accept AS199284 AND NOT AS199284^-; from AS199284 action community = { 64628:11 }; accept AS199284; from AS199284 action community = { 64628:12 }; accept AS199284^0-24; from AS199284 action community = { 64628:12 }; accept AS199284:FLTR-LegacyCustomers; from AS199284 action community = { 64628:13 }; accept AS199284:FLTR-CGN-Inside; from AS199284 action community = { 64628:14 }; accept AS199284:FLTR-CGN-Outside; from AS199284 action community = { 64628:15 }; accept AS199284:FLTR-LegacyPrivate; } REFINE afi ipv6.unicast { from AS199284 action community = { 64628:10 }; accept AS199284 AND NOT AS199284^-; from AS199284 action community = { 64628:11 }; accept AS199284; from AS199284 action community = { 64628:12 }; accept AS199284^- AND { 2000::/3^4-48 }; }
58143443:mp-import:      afi any { from AS-ANY action community.delete(64628:10, 64628:11, 64628:12, 64628:13, 64628:14, 64628:15, 64628:20, 64628:21, 64628:22); accept ANY; } REFINE afi any { from AS-ANY action pref = 65535; accept community(65535:0); from AS-ANY action pref = 65435; accept ANY; } REFINE afi any { from AS-ANY accept NOT AS199284^+; } REFINE afi ipv4 { from AS-ANY accept NOT fltr-martian; } REFINE afi ipv4 { from AS-ANY accept { <IP_ADDRESS>/0^0-24 } AND NOT community(65535:666); from AS-ANY accept { <IP_ADDRESS>/0^24-32 } AND community(65535:666); } REFINE afi ipv6 { from AS-ANY accept { 2000::/3^4-48 } AND NOT community(65535:666); from AS-ANY accept { 2000::/3^64-128 } AND community(65535:666); } REFINE afi any { from AS15725 action community .= { 64628:20 }; accept AS-IKS AND <^AS-IKS+$>; from AS196714 action community .= { 64628:20 }; accept AS-TNETKOM AND <^AS-TNETKOM+$>; from AS199284:AS-UP action community .= { 64628:21 }; accept ANY; from AS35366 action community .= { 64628:22 }; accept AS-ISPPRO AND <^AS-ISPPRO+$>; from AS20940 action community .= { 64628:22 }; accept <^AS-AKAMAI+$>; from AS16509 action community .= { 64628:22 }; accept <^AS-AMAZON+$>; from AS32934 action community .= { 64628:22 }; accept <^AS-FACEBOOK+$>; from AS2906 action community .= { 64628:22 }; accept <^AS-NFLX+$>; from AS46489 action community .= { 64628:22 }; accept <^AS-TWITCH+$>; from AS714 action community .= { 64628:22 }; accept <^AS-APPLE+$>; from AS26415 action community .= { 64628:22 }; accept <^AS-GTLD+$>; from AS13335 action community .= { 64628:22 }; accept <^AS-CLOUDFLARE+$>; from AS-ANY action community .= { 64628:22 }; accept PeerAS and <^PeerAS+$>; } REFINE afi any { from AS-ANY EXCEPT (AS40027 OR AS63293 OR AS65535) accept ANY; }
58143455:mp-export:      afi any.unicast { to AS-ANY announce community(64628:10, 64628:11, 64628:20); to AS-ANY action community .= { NO_EXPORT }; announce community(64628:12); to AS199284:AS-DOWN announce community(64628:21, 64628:22); to AS15725 announce community(64628:13, 64628:15); to AS65535 action community .= { 15169:12100 }; announce community(64628:13); to AS65535 action community .= { 15169:12000 }; announce community(64628:14); } REFINE afi any { to AS35366 or AS6939 action community .= { NO_EXPORT }; to AS-ANY announce ANY; }

29 in total.
$ rg --no-ignore -U -c '^(mp-)?(im|ex)port:.*(\n[+\s]+.*)*\bREFINE\b.*(\n[+\s]+.*)*'
priority/altdb.db:2
backup/altdb.db:2
priority/ripe.db:27

That was wrong, Ripgrep was case-sensitive.
It is 145 in total.
All of them:
all_rules_w_refine.txt
$ rg --no-ignore -Uic '^(mp-)?(im|ex)port:.*(\n[+\s]+.*)*\bREFINE\b.*(\n[+\s]+.*)*'                                   internet_route_verification/data/irrs
priority/apnic.db.aut-num:6
backup/altdb.db:2
backup/radb.db:39
priority/ripe.db:98


Python 3

Coverage increased (+0.04%) to 97.938% when pulling 19d143279d259e85d3e9d3a87a38e33325a86cea on python-3 into f647cea103bb098ea1a80287eb8e42d089255f47 on master.

Coverage increased (+0.09%) to 97.98% when pulling 20c44c97bb79e69ed91a125b9e550ebd29818a64 on python-3 into f647cea103bb098ea1a80287eb8e42d089255f47 on master.


Adding gitlab-ci logs to logrotate

Ci logs can be quite chatty and tend to fill the disk quite fast.
This enable logrotate for  gitlab-ci logs by default

Thanks @frconil !


for adding predicted features: add a flag for exists or not

inside transform for AddPredictedFeature, check to see if we already have this feature present in the dataset. if we do, don't do anything! If we don't, add it in.
and maybe add it in every time for a single row, just to be sure we're doing that right at prediction time.
Doing this will let us get predicted features once for our training set, and once for our test set, rather than having to re-compute them each and every new model we train up.

way too much complexity. removing.


Add SimTrackerHit

First Iteration: Taken from PLCIO, but with one long long int member for cellID instead of two ints, and without Extracode.

One question: For FCC this class was one of the bottlenecks and took a lot of diskspace. The MCParticle linked to the Hit also has infomation on "momentum", but not at the point where the Hit was created. Furthermore some Digitization algorithms depend on the angle of the particle path crossing the sensor, which this also provides, so I think this should be kept as is, no?

@vvolkl I had tried to make your PR compatible for rebasing but seem to have made a mistake.
I will simply create a new PR with Sim hits that we can include and then iterate on.

Thanks for the comments, should be ok now except that it needs some changes in podio (https://github.com/AIDASoft/podio/pull/60) but of course you can also create a new one @gaede.

Superseded by #11


[Bug]: vllm server 部署base和lora模型后，请求lora模型失败

Your current environment
部署指令：
vllm serve /root/autodl-fs/llm_models/qwen/Qwen2-7B-Instruct --enable-lora --lora-modules bi-lora=/root/autodl-fs/saves/Qwen2-7B-Instruct/lora/sft/checkpoint-1500/ --port 6006
请求体：
{
"model": "bi-lora",
"prompt": "你是谁",
"temperature": 0.7,
"top_p": 0.8,
"repetition_penalty": 1.05,
"max_tokens": 512
}
报错信息：RuntimeError: Loading lora /root/autodl-fs/saves/Qwen2-7B-Instruct/lora/sft/checkpoint-1500/ failed
如果 访问base模型是没问题的（model改成  /root/autodl-fs/llm_models/qwen/Qwen2-7B-Instruct）
🐛 Describe the bug
PyTorch version: 2.3.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A
OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.22.1
Libc version: glibc-2.35
Python version: 3.12.3 | packaged by Anaconda, Inc. | (main, May  6 2024, 19:46:43) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-91-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.1.105
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA L20
Nvidia driver version: 550.54.14
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      52 bits physical, 57 bits virtual
Byte Order:                         Little Endian
CPU(s):                             180
On-line CPU(s) list:                0-179
Vendor ID:                          GenuineIntel
Model name:                         Intel(R) Xeon(R) Platinum 8457C
CPU family:                         6
Model:                              143
Thread(s) per core:                 2
Core(s) per socket:                 45
Socket(s):                          2
Stepping:                           8
BogoMIPS:                           5200.00
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd arat avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid cldemote movdiri movdir64b fsrm md_clear serialize tsxldtrk arch_lbr amx_bf16 avx512_fp16 amx_tile amx_int8 arch_capabilities
Hypervisor vendor:                  KVM
Virtualization type:                full
L1d cache:                          4.2 MiB (90 instances)
L1i cache:                          2.8 MiB (90 instances)
L2 cache:                           180 MiB (90 instances)
L3 cache:                           195 MiB (2 instances)
NUMA node(s):                       2
NUMA node0 CPU(s):                  0-89
NUMA node1 CPU(s):                  90-179
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Unknown: No mitigations
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Mitigation; TSX disabled
Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] pyzmq==26.0.3
[pip3] torch==2.3.0+cu121
[pip3] torchvision==0.18.0+cu121
[pip3] transformers==4.43.4
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] nvidia-nccl-cu12          2.20.5                   pypi_0    pypi
[conda] pyzmq                     26.0.3                   pypi_0    pypi
[conda] torch                     2.3.0+cu121              pypi_0    pypi
[conda] torchvision               0.18.0+cu121             pypi_0    pypi
[conda] transformers              4.43.4                   pypi_0    pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: N/A
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    NIC0    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      SYS     90-179  1               N/A
NIC0    SYS      X
Legend:
X    = Self
SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX  = Connection traversing at most a single PCIe bridge
NV#  = Connection traversing a bonded set of # NVLinks
NIC Legend:
NIC0: mlx5_0

Which version of vLLM are you using? Additionally, could you provide more detailed information about the error you're encountering?

Which version of vLLM are you using? Additionally, could you provide more detailed information about the error you're encountering?
vllm=0.5.4
具体信息：我用llama-factory这个项目，lora训练了 qwen2-7b-instruct，借鉴与 longlora，训练参数里还有norm和embed。之后用vllm部署 模型 和 lora（包含norm 和 embed）
vllm serve /root/autodl-fs/llm_models/qwen/Qwen2-7B-Instruct --enable-lora --lora-modules bi-lora=/root/autodl-fs/saves/Qwen2-7B-Instruct/lora/sft/checkpoint-1500/ --port 6006
上面的命令里 /root/autodl-fs/saves/Qwen2-7B-Instruct/lora/sft/checkpoint-1500/ 是lora参数（包含norm和embed）所在的目录。
访问的时候，下面的请求体会报错：RuntimeError: Loading lora /root/autodl-fs/saves/Qwen2-7B-Instruct/lora/sft/checkpoint-1500/ failed 。但访问 chat模型是没问题的（model参数改成 /root/autodl-fs/llm_models/qwen/Qwen2-7B-Instruct）
{
"model": "bi-lora",
"prompt": "你是谁",
"temperature": 0.7,
"top_p": 0.8,
"repetition_penalty": 1.05,
"max_tokens": 512
}

The cause of this error is that it caught an execution error from the previous code, so there should be error messages preceding this one. You should check the log information again.


Switch back to previous application after "code --wait"

When using code --wait in, for example, Terminal.app on Mac, it would be convenient if it would switch back to Terminal.app after closing the file in VS Code.
The subl command-line tool of Sublime Text does this, and it's awesome. It has an option to disable this behavior ("-s or --stay: Keep the application activated after closing the file"); I suppose for code it could also be the other way round, with an opt-in parameter in order to not change existing behavior.
For now I'm approximating the behavior on Mac with a small wrapper script:
#/bin/sh

pid="$(osascript -e 'tell application "System Events" to get unix id of first process whose frontmost is true')"
code -w "$@"
osascript -e "tell application \"System Events\" to set frontmost of processes whose unix id is $pid to true"

Yes, I would like this as well but never got to how to actually implement this across all platforms. If someone has suggestions, I am open to hear them.

Not if it can help but TextMate does that too, this is a reference to the CLI accepting --wait https://github.com/textmate/textmate/blob/2624e59e2c51354a8282f7fdfbc9dff350bd8ca7/Applications/mate/src/mate.mm#L281


Training code

Can anyone please share their training code?

Could you please tell me how you solve this issue？Thanks, looking forward to your reply.

Same here!


Version 1.8.1 and 1.9

Because we don't yet have an official communication channel, I am opening this issue to lay down what happened in the last few weeks:

@nvie transferred maintainership to the jazzband organization.
I started to cleanup the repo of old issues and started merging some bugfix only PRs in the 1.8.1 release
I want to let the 1.8.1rc3 live for one week in order to be sure that it is safe to release.
The next step would be to make a feature release 1.9.

Don't hesitate to comment, this is what it's all about !

Very very glad to see this repo getting action again, thanks a bunch for diving in!  Is this an appropriate channel for general questions about all the goings on, or are you trying to keep it focused on 1.8.1 and 1.9 plans?

We can focus on 1.8.1 and 1.9 for now, maybe open another ticket for more general ideas and directions we would like to take.
1.8.1 should bring a bit more stability in the resolver. I do not want to add other non bug fixing to this release. I think the rc3 will probably materialize next week if I don't hear anything more preventing it's release.
1.9 should bring small changes and features, there are a lot of good PRs, but some lack tests, and because of the new collaborative maintainership, I think it is a deal breaker for those. I am not sure what PRs are more important than others, I will survey the number of involvement and issue/PR participation and probably merge the best ones (with tests :-)).
Anybody wanting to be involved should open a request with jazzband, that will let them add Labels and close old tickets. There are really old issues that don't even apply anymore.
I am on vacation for the next 8 days, so I will not be able to participate that much.

Any chance of a 1.9.0 release soon? The merged https://github.com/jazzband/pip-tools/pull/441 would be super useful for us (and seemingly a lot of others, too).
There's this issue in the latest pip release, https://github.com/pypa/pip/issues/4264, which causes pip to fail when uninstalling appdirs (a setuptools dependency.) It takes quite a bit of duct tape to keep using pip-tools like this.

@underyx I'm all for a 1.9 release, but I do not have alot of metrics for finalizing. 1.9 has a lot of new features and I wouldn't want to break anybody. I will create a rc1 for you to test.
In the meantime, does installing on master fix your use cases ?

@davidovich yep, I can confirm that it's fixed on master. The RC will not be necessary — I can just keep using the commit from the latest master. It'd be a bit more inconvenient however to get everyone at the company to update each of their virtualenvs to a prerelease to make sure nobody overwrites requirements.txts with versions that have appdirs in them.

@davidovich Since you've mentioned it's a bit tricky to gather a sense of what will and won't break things when you're taking on an existing project like this, maybe the next release, currently called (1.9) should be 2.0? then people will know they have a risk of breaking when they upgrade, however low it may be, and do so a little more gingerly? it has a lot of new features already. Specifically, the one I'm interested in (pulling from setup.py) is a game-changer, in that it lets you use pip-tools in an existing project without the overhead of managing a separate requirements.in file. I wouldn't mind upgrading to 2.0 for the sake of having that feature.
Setting the next release at 2.0 lets you make the decision around finalization more easily—namely, in feature-space, not in issue-space. Once 2.0 is set, you can then start making smaller releases for the remaining features, adding them in one or two at a time, in minor version bumps, minimizing the chance of breakage.

@myw Thanks for your suggestion, but I still feel this as being a 1.9, and as you say, the major feature is to support setup.py install_requires as a requirements.in.
No new code entered without tests, so this brings confidence a bit up. I will be making a 1.9rc1 today (if time permits), which should also marinate a little while.

1.9 is out, will close this. Thanks everyone!


[5] Docker container that has algod and indexer

One quick-start docker container with algod and algorand-indexer and postgres

Need to re-vet this with Rotem. Some of this may already be covered by Will's work with docker-compose, so need to see if this is a real ticket, and what the use cases are.

We at Reach have built this if anyone needs it --- https://github.com/reach-sh/reach-lang/tree/master/scripts/algorand-devnet --- the image doesn't contain postgres, but we use a compose file that connects to postgres. I'm happy to chat with anyone who wants to work on adapting what we have to your uses.

More related work
https://github.com/algorand/algorand-sdk-testing/
https://github.com/algorand/sandbox/issues/14


Adding a new member with the same name and URL as a removed member, fails

One of the member of my cluster lost all its persisted data. So, I want to make it be recognized as a new member in the cluster, and sync all the data from scratch from the other members.
As far as I know, the way to do it is to remove the old member from the cluster (the one we lost its data), and etcdctl member add the new member. The only particularity here is that the 'new' member name and URL are the same as the old member name and URL (because it's actually the same machine, it's just that we lost all of its data...).
After using etcdctl to add the 'new' member and starting etcd again with initial-state=existing, I'm seeing this in the logs of the new member. I've changed the IDs to what they actually represent to make it clearer. The member that lost its data is etcd-cluster-1.etcd-cluster, and I have three nodes in total (0 to 2) in the cluster:
 I | rafthttp: started streaming with peer [MASTER_ID] (stream MsgApp v2 reader)
 I | rafthttp: started streaming with peer [MASTER_ID] (stream Message reader)
 I | etcdserver/membership: added member [OTHER_MEMBER_ID] [https://etcd-cluster-2.etcd-cluster:2380] to cluster [CLUSTER_ID]
 I | etcdserver/membership: added member [OLD_REMOVED_MEMBER_ID] [https://etcd-cluster-1.etcd-cluster:2380] to cluster [CLUSTER_ID]
 I | etcdserver/membership: added member [MASTER_ID] [https://etcd-cluster-0.etcd-cluster:2380] to cluster [CLUSTER_ID]
 E | rafthttp: streaming request ignored (ID mismatch got [NEW_ADDED_MEMBER_ID] want [OLD_REMOVED_MEMBER_ID])
 I | raft: [OLD_REMOVED_MEMBER_ID] [term: 1] received a MsgHeartbeat message with higher term from [MASTER_ID] [term: 24]
 E | rafthttp: streaming request ignored (ID mismatch got [NEW_ADDED_MEMBER_ID] want [OLD_REMOVED_MEMBER_ID])
 I | raft: [OLD_REMOVED_MEMBER_ID] became follower at term 24
 I | raft: raft.node: [OLD_REMOVED_MEMBER_ID] elected leader [MASTER_ID] at term 24
 E | rafthttp: streaming request ignored (ID mismatch got [NEW_ADDED_MEMBER_ID] want [OLD_REMOVED_MEMBER_ID])
 E | rafthttp: streaming request ignored (ID mismatch got [NEW_ADDED_MEMBER_ID] want [OLD_REMOVED_MEMBER_ID])
 E | etcdserver: the member has been permanently removed from the cluster
 I | etcdserver: the data-dir used by this member must be removed.
 I | etcdserver: aborting publish because server is stopped

After this, there is some more uninteresting output and etcd dies. The data directory gets populated with some files, even though it doesn't seem to actually have synced the previous data.

@albertvaka did you remove all data associated with the old member before starting the new member?
sleep 5s
kill -9 $victim
export ETCDCTL_API=3
members=$(bin/etcdctl member list)
echo "$members"
id=$(echo "$members" | grep infra3 | cut -f1 -d',')
echo revmoing id $id
bin/etcdctl member remove $id
sleep 5s
bin/etcdctl member list
echo adding member
env=$(bin/etcdctl member add --peer-urls=http://<IP_ADDRESS>:32380 infra3)
echo "$env"

# REMOVE OLD DATA
rm -rf *infra3*

bin/etcd        \
         --name infra3  \
         --listen-client-urls http://<IP_ADDRESS>:32379    \
         --advertise-client-urls http://<IP_ADDRESS>:32379 \
         --listen-peer-urls http://<IP_ADDRESS>:32380      \
         --initial-advertise-peer-urls http://<IP_ADDRESS>:32380   \
         --initial-cluster-token etcd-cluster-1 \
         --initial-cluster "infra3=http://<IP_ADDRESS>:32380,infra1=http://<IP_ADDRESS>:12380,infra2=http://<IP_ADDRESS>:22380"  \
         --initial-cluster-state existing

Closing, as it was indeed my fault... I'm setting the config as environment variables and I was missing an export (ie: ETCD_INITIAL_CLUSTER_STATE="existing" instead of export  ETCD_INITIAL_CLUSTER_STATE="existing"). The symptoms were really unexpected to my, though, since the old ID was appearing in the logs (where does it come from? :confused:).
Thanks a lot for your time @heyitsanthony, your snippet helped me reproduce the problem :) Maybe it would be a good idea to post it somewhere in a "Troubleshooting etcd locally" section.

it's already documented under https://github.com/coreos/etcd/blob/master/Documentation/faq.md#what-does-the-etcd-warning-request-ignored-cluster-id-mismatch-mean but maybe it belongs in https://github.com/coreos/etcd/blob/master/Documentation/op-guide/failures.md


add process in ci

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@4581d22). Click here to learn what that means.
The diff coverage is n/a.

:exclamation: Current head 7b470f0 differs from pull request most recent head c612dea. Consider uploading reports for the commit c612dea to get more accurate results

@@            Coverage Diff            @@
##             master     #337   +/-   ##
=========================================
  Coverage          ?   47.05%
=========================================
  Files             ?       25
  Lines             ?     2733
  Branches          ?        0
=========================================
  Hits              ?     1286
  Misses            ?     1356
  Partials          ?       91

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4581d22...c612dea. Read the comment docs.


Deleted Default Category Unable to add weather and clock in new one

Environment
Docker
Version
0.11.2
Describe the problem
I Deleted the Default Category and now in a new category I'm unable to add weather and clock.

Additional info
No response
Please tick the boxes

[X] You've read the docs
[X] You've checked for duplicate issues
[X] You've tried to debug yourself

Duplicate of #630 , linked pull requests in that issue will fix this. @ajnart , can you please review, so we can merge?


Getting Tailable cursor.on error from the library in our Application Logs. How can i resolve that issue ?

This is how we are creating the pubsub.
import { databaseName, connectionString } from '../config.js';
import { MongoClient } from 'mongodb';
import { MongodbPubSub } from 'graphql-mongodb-subscriptions';
import { supplyOrderCollection } from '../config.js';
export const client = new MongoClient(connectionString);
export const db = client.db(databaseName);
const PubSubMongoDbOptions = {
connectionDb: db,
channelName: databaseName,
};
export const pubsub = new MongodbPubSub(PubSubMongoDbOptions);
export const watchSupplyOrderCollection = db.collection(supplyOrderCollection);

Please review this other issue and see if that helps with your setup
https://github.com/mjwheatley/graphql-mongodb-subscriptions/issues/43

This is a different issue. The setup works in production for about 6-10 days and then we are getting tailableCursor.on('error') in all the pods and subscription stops working.
We can see the subscribe logs but not the publish logs after this error.
This is something to do with capped collection that is being created by graphql-mongodb-subscriptions

@mjwheatley Do you have any suggestions for me ?
This error is happening due to the mubsub capped collection created by the module.

This is the dependency package that actually creates the tailable cursor.
https://github.com/mjwheatley/mongopubsub/blob/master/src/channel.ts
Line 134 is the console.error() messages that you may be seeing.  Does it print out the error object and provide any additional information about the error?

We really need to find out what the actual error is.  Not sure how your application is being packaged and ran, but you could potentially go into the node_modules folder and find the error message and modify t so that it shows more info in your logs.

You could could change it to something like
console.error(`tailableCursor.on('error') Error Message: ` + error.message);

It seems like it might be a connection issue.  Perhaps setting timeout or noCursorTimeout on the FindOptions to false.

Are you experiencing any connectivity issues with the Mongo connection itself or is it just the tailable cursor?


Update MD file on new branch 2017-25-05_21-12-27

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

split/updateTestRef.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


F+A03/04: strong overlap

It is unclear to me what the difference between F+A03 and F+A04 is. If an appropriate community standard is used, this should already define data structure incl types, constraints etc. In light of this, F+A04 seems to be superfluous, as long as F+A03 is updated to clarify that the use of an accepted community standard, if available, is encouraged over the definition of a new in-house standard.

I would split them in "subindicators"  like F+S08, since they are related. But i would not remove F+A04, beacuse you can fulfill F+A04 without really fulfilling F+A03.


[Request] Nix pkg

I've already requested Nix pkgs since Nix doesn't support certain executables

NixOS/nixpkgs#347781 should close this.


Stop commiting doc changes when you havent actually changed it

Close this when you have read it, i would comment on a commit but its too long

Good idea. Each commit is too long to read. Most of it is doc changes which is not the main attention point. The better is not to have doc committed at all polluting code changes.
Please allow me to give possible solutions here.
-Use gh-pages branch of Github to post only doc content to it so the project can still have free document hosted online. It could be of use to use ghp-import tool to automate thing further.

Document content could be generate on-demand i.e. only when doing push on Github, or just do it every time the project has a new release either manually or through automated-script.

PS. JRakNet is great project.

I will definitely look into this. I can see this annoying people now that I think about it. @TheDiamondYT1 Could you tell me what you were originally going to comment and on what commit you were going to post it on?

it was the most recent code update. Now i realise i couldve commented on the Update README commits, but oh well


Drawing a full-window fragment shader

Sorry if it's a noob question, I'm new into graphics programming and I didn't find how to do what I want in the documentation...
I created a fragment shader which works exactly as I need when I open it on programs like glslViewer so now I need to display it in my application.
I'm trying to do so with glow but I'm stuck in the vertex shader's triangle of the example project. No matter what I do, all fragment shaders that I try works only inside this triangle and I can't change him.
So I'm seeking some advince to get rid of this vertex shader and use only use my fragment shader, like I do in glslViewer. Or at least use a vertex shader that covers all screen.
Another question is how can I get some data (like textures and positions) from my rust code into my fragment shader code. glslViewer has some special variable names for that like u_tex0, u_tex1, u_resolution, u_time... but it seems to not works for glow.

You'll need both a vertex shader and fragment shader for this, but you can have a vertex shader just create a single full screen quad (two triangles) or a triangle that covers the full screen.
For uniforms like u_resolution, you'd use any of the functions prefixed with uniform_ to set data for the uniform. These uniforms need to be declared in the fragment shader and be used somewhere in the shader (otherwise they'll be automatically removed from the shader).
If you're interested, the #games-and-graphics channel on the unofficial Rust Community Discord might be a helpful resource to get started with Rust and GL.

Thank you @grovesNL!
Your tips worked like a charm.


tests failing with postgresql

Details
We're trying to run tests against postgres to match the database we use in production.
Expected Result
tests pass
Actual Result
With the following changes required to run tests against postgres https://github.com/rtfd/readthedocs.org/compare/master...italia:testspostgres?expand=1 tests fails with this https://travis-ci.org/italia/docs.italia.it/jobs/367583533

This are logs against postgresql 9.6: https://travis-ci.org/italia/docs.italia.it/jobs/367631789

For the other 50% not fixed by the PR above i've tried this but that's not it:
https://github.com/italia/docs.italia.it/commit/fa3983666a786f8ca511e706c78955717090e636
Any help would be much appreciated

As I know, RTD uses Postgres in production.


System.Text.RegularExpressions.RegexMatchTimeoutException

Seems to be affecting Macs again: https://github.com/thomhurst/TUnit/actions/runs/12029859325/job/33536030398
Happening since upgrading to 1.5.0-preview.24575.7
Error: Unhandled exception. System.Text.RegularExpressions.RegexMatchTimeoutException: The Regex engine has timed out while trying to match a pattern to an input string. This can occur for many reasons, including very large inputs or excessive backtracking caused by nested quantifiers, back-references and other factors.
   at System.Text.RegularExpressions.RegexRunner.<CheckTimeout>g__ThrowRegexTimeout|25_0()
   at System.Text.RegularExpressions.Generated.<RegexGenerator_g>F06D33C3F8C8C3FD257C1A1967E3A3BAC4BE9C8EC41CC9366C764C2205C68F0CE__GetFrameRegex_1.RunnerFactory.Runner.TryMatchAtCurrentPosition(ReadOnlySpan`1) in /_/artifacts/obj/Microsoft.Testing.Platform/Release/net8.0/System.Text.RegularExpressions.Generator/System.Text.RegularExpressions.Generator.RegexGenerator/RegexGenerator.g.cs:line 639
   at System.Text.RegularExpressions.Generated.<RegexGenerator_g>F06D33C3F8C8C3FD257C1A1967E3A3BAC4BE9C8EC41CC9366C764C2205C68F0CE__GetFrameRegex_1.RunnerFactory.Runner.Scan(ReadOnlySpan`1) in /_/artifacts/obj/Microsoft.Testing.Platform/Release/net8.0/System.Text.RegularExpressions.Generator/System.Text.RegularExpressions.Generator.RegexGenerator/RegexGenerator.g.cs:line 537
   at System.Text.RegularExpressions.Regex.ScanInternal(RegexRunnerMode, Boolean, String, Int32, RegexRunner, ReadOnlySpan`1, Boolean)
   at System.Text.RegularExpressions.Regex.RunSingleMatch(RegexRunnerMode, Int32, String, Int32, Int32, Int32)
   at System.Text.RegularExpressions.Regex.Match(String)
   at Microsoft.Testing.Platform.OutputDevice.Terminal.TerminalTestReporter.AppendStackFrame(ITerminal, String) in /_/src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/TerminalTestReporter.cs:line 650
   at Microsoft.Testing.Platform.OutputDevice.Terminal.TerminalTestReporter.FormatStackTrace(ITerminal, FlatException[], Int32) in /_/src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/TerminalTestReporter.cs:line 601
   at Microsoft.Testing.Platform.OutputDevice.Terminal.TerminalTestReporter.RenderTestCompleted(ITerminal , String , String, String, String , TestOutcome, TimeSpan, FlatException[] , String, String, String, String) in /_/src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/TerminalTestReporter.cs:line 517
   at Microsoft.Testing.Platform.OutputDevice.Terminal.TerminalTestReporter.<>c__DisplayClass27_0.<TestCompleted>b__0(ITerminal terminal) in /_/src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/TerminalTestReporter.cs:line 439
   at Microsoft.Testing.Platform.OutputDevice.Terminal.TestProgressStateAwareTerminal.WriteToTerminal(Action`1) in /_/src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/TestProgressStateAwareTerminal.cs:line 129
   at Microsoft.Testing.Platform.OutputDevice.Terminal.TerminalTestReporter.TestCompleted(String , String, String, String, String , TestOutcome, TimeSpan, FlatException[] , String, String, String, String) in /_/src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/TerminalTestReporter.cs:line 439
   at Microsoft.Testing.Platform.OutputDevice.Terminal.TerminalTestReporter.TestCompleted(String , String, String, String, String , TestOutcome, TimeSpan, String, Exception, String, String, String, String) in /_/src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/TerminalTestReporter.cs:line 386
   at Microsoft.Testing.Platform.OutputDevice.TerminalOutputDevice.ConsumeAsync(IDataProducer, IData, CancellationToken) in /_/src/Platform/Microsoft.Testing.Platform/OutputDevice/TerminalOutputDevice.cs:line 458
   at Microsoft.Testing.Platform.Messages.AsyncConsumerDataProcessor.ConsumeAsync() in /_/src/Platform/Microsoft.Testing.Platform/Messages/ChannelConsumerDataProcessor.cs:line 74
   at Microsoft.Testing.Platform.Messages.AsyncConsumerDataProcessor.DrainDataAsync() in /_/src/Platform/Microsoft.Testing.Platform/Messages/ChannelConsumerDataProcessor.cs:line 146
   at Microsoft.Testing.Platform.Messages.AsynchronousMessageBus.DrainDataAsync() in /_/src/Platform/Microsoft.Testing.Platform/Messages/AsynchronousMessageBus.cs:line 177
   at Microsoft.Testing.Platform.Messages.MessageBusProxy.DrainDataAsync() in /_/src/Platform/Microsoft.Testing.Platform/Messages/MessageBusProxy.cs:line 39
   at Microsoft.Testing.Platform.Hosts.CommonTestHost.NotifyTestSessionEndAsync(SessionUid, BaseMessageBus, ServiceProvider, CancellationToken) in /_/src/Platform/Microsoft.Testing.Platform/Hosts/CommonTestHost.cs:line 192
   at Microsoft.Testing.Platform.Hosts.CommonTestHost.ExecuteRequestAsync(IPlatformOutputDevice, ITestSessionContext, ServiceProvider, BaseMessageBus, ITestFramework, ClientInfo) in /_/src/Platform/Microsoft.Testing.Platform/Hosts/CommonTestHost.cs:line 133
   at Microsoft.Testing.Platform.Hosts.ConsoleTestHost.InternalRunAsync() in /_/src/Platform/Microsoft.Testing.Platform/Hosts/ConsoleTestHost.cs:line 85
   at Microsoft.Testing.Platform.Hosts.ConsoleTestHost.InternalRunAsync() in /_/src/Platform/Microsoft.Testing.Platform/Hosts/ConsoleTestHost.cs:line 117
   at Microsoft.Testing.Platform.Hosts.CommonTestHost.RunTestAppAsync(CancellationToken) in /_/src/Platform/Microsoft.Testing.Platform/Hosts/CommonTestHost.cs:line 106
   at Microsoft.Testing.Platform.Hosts.CommonTestHost.RunAsync() in /_/src/Platform/Microsoft.Testing.Platform/Hosts/CommonTestHost.cs:line 34
   at Microsoft.Testing.Platform.Hosts.CommonTestHost.RunAsync() in /_/src/Platform/Microsoft.Testing.Platform/Hosts/CommonTestHost.cs:line 72
   at Microsoft.Testing.Platform.Builder.TestApplication.RunAsync() in /_/src/Platform/Microsoft.Testing.Platform/Builder/TestApplication.cs:line 244
   at TestingPlatformEntryPoint.Main(String[]) in /_/TUnit.TestProject/obj/Release/net8.0/osx-x64/TestPlatformEntryPoint.cs:line 16
   at TestingPlatformEntryPoint.<Main>(String[])

Exceptions within Console/Display classes should probably be logged but swallowed - Currently they break test execution.
Offending line is here: https://github.com/Microsoft/testfx/blob/eaf26c33a6e9e4e7ff37b3447ef0ca856e6b4239/src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/TerminalTestReporter.cs#L54
It's got a 1 second timeout on it. I guess due to constrained system resources this is being hit.

Exceptions within Console/Display classes should probably be logged but swallowed - Currently they break test execution.

That is unfortunate, but imho the correct way to do this. Otherwise you would not see it and we have no other way of receiving that kind of information.

I've added 1000ms timeout because that seemed like lightyears in regex matching, especially on the source generated.  I would start by wrapping the match into try catch, and report the string that is being matched.
Great that you are already consuning preview, so we can debug this with a simpler loop.

There should be new package in the preview channel in about half an hour, please update when convenient for you. Hopefully we will see what the source of the timeout is.

@nohwnd

  Error: Unhandled Exception:
  System.Text.RegularExpressions.RegexMatchTimeoutException: Parsing stack trace
  line with regex timed out. Input: '   at
  System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe'
   ---> System.Text.RegularExpressions.RegexMatchTimeoutException: The Regex
  engine has timed out while trying to match a pattern to an input string. This
  can occur for many reasons, including very large inputs or excessive
  backtracking caused by nested quantifiers, back-references and other factors.
     at
  System.Text.RegularExpressions.RegexRunner.g__ThrowRegexTimeout|25
  _0() + 0x5f
     at
  System.Text.RegularExpressions.Generated.F06D33C3F8C8C3FD257C1
  A1967E3A3BAC4BE9C8EC41CC9366C764C2205C68F0CE__GetFrameRegex_1.RunnerFactory.Runn
  er.TryMatchAtCurrentPosition(ReadOnlySpan`1) + 0xbbd
     at
  System.Text.RegularExpressions.Generated.F06D33C3F8C8C3FD257C1
  A1967E3A3BAC4BE9C8EC41CC9366C764C2205C68F0CE__GetFrameRegex_1.RunnerFactory.Runn
  er.Scan(ReadOnlySpan`1) + 0x35
     at System.Text.RegularExpressions.Regex.ScanInternal(RegexRunnerMode,
  Boolean, String, Int32, RegexRunner, ReadOnlySpan`1, Boolean) + 0x36
     at System.Text.RegularExpressions.Regex.RunSingleMatch(RegexRunnerMode,
  Int32, String, Int32, Int32, Int32) + 0x157
     at System.Text.RegularExpressions.Regex.Match(String) + 0x3e
     at
  Microsoft.Testing.Platform.OutputDevice.Terminal.TerminalTestReporter.AppendStac
  kFrame(ITerminal, String) + 0x115
     --- End of inner exception stack trace ---
     at
  System.Text.RegularExpressions.RegexRunner.g__ThrowRegexTimeout|25
  _0() + 0x5f
     at
  System.Text.RegularExpressions.Generated.F06D33C3F8C8C3FD257C1
  A1967E3A3BAC4BE9C8EC41CC9366C764C2205C68F0CE__GetFrameRegex_1.RunnerFactory.Runn
  er.TryMatchAtCurrentPosition(ReadOnlySpan`1) + 0xbbd
     at
  System.Text.RegularExpressions.Generated.F06D33C3F8C8C3FD257C1
  A1967E3A3BAC4BE9C8EC41CC9366C764C2205C68F0CE__GetFrameRegex_1.RunnerFactory.Runn
  er.Scan(ReadOnlySpan`1) + 0x35
     at System.Text.RegularExpressions.Regex.ScanInternal(RegexRunnerMode,
  Boolean, String, Int32, RegexRunner, ReadOnlySpan`1, Boolean) + 0x36
     at System.Text.RegularExpressions.Regex.RunSingleMatch(RegexRunnerMode,
  Int32, String, Int32, Int32, Int32) + 0x157
     at System.Text.RegularExpressions.Regex.Match(String) + 0x3e
     at
  Microsoft.Testing.Platform.OutputDevice.Terminal.TerminalTestReporter.AppendStac
  kFrame(ITerminal, String) + 0x115
  --- End of stack trace from previous location ---
     at
  Microsoft.Testing.Platform.OutputDevice.Terminal.TerminalTestReporter.AppendStac
  kFrame(ITerminal, String) + 0x9e8
     at
  Microsoft.Testing.Platform.OutputDevice.Terminal.TerminalTestReporter.FormatStac
  kTrace(ITerminal, FlatException[], Int32) + 0x11a
     at
  Microsoft.Testing.Platform.OutputDevice.Terminal.TerminalTestReporter.RenderTest
  Completed(ITerminal, String, String, String, String, TestOutcome, TimeSpan,
  FlatException[], String, String, String, String) + 0x783
     at
  Microsoft.Testing.Platform.OutputDevice.Terminal.TerminalTestReporter.<>c__Displ
  ayClass27_0.b__0(ITerminal terminal) + 0x63
     at
  Microsoft.Testing.Platform.OutputDevice.Terminal.TestProgressStateAwareTerminal.
  WriteToTerminal(Action`1) + 0x94
     at
  Microsoft.Testing.Platform.OutputDevice.Terminal.TerminalTestReporter.TestComple
  ted(String, String, String, String, String, TestOutcome, TimeSpan, FlatException
  [], String, String, String, String) + 0x3d2
     at
  Microsoft.Testing.Platform.OutputDevice.TerminalOutputDevice.d__57
  .MoveNext() + 0x407
  --- End of stack trace from previous location ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) +
  0xbe
     at
  System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotificat
  ion(Task, ConfigureAwaitOptions) + 0x4e
     at
  Microsoft.Testing.Platform.Messages.AsyncConsumerDataProcessor.d__
  12.MoveNext() + 0x2ca
  --- End of stack trace from previous location ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) +
  0xbe
     at
  System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotificat
  ion(Task, ConfigureAwaitOptions) + 0x4e
     at
  Microsoft.Testing.Platform.Messages.AsyncConsumerDataProcessor.d
  __14.MoveNext() + 0x1f0
  --- End of stack trace from previous location ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) +
  0xbe
     at
  System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotificat
  ion(Task, ConfigureAwaitOptions) + 0x4e
     at
  Microsoft.Testing.Platform.Messages.AsynchronousMessageBus.d__17
  .MoveNext() + 0x775
  --- End of stack trace from previous location ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) +
  0xbe
     at
  System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotificat
  ion(Task, ConfigureAwaitOptions) + 0x4e
     at
  Microsoft.Testing.Platform.Messages.MessageBusProxy.d__6.MoveNex
  t() + 0xaa
  --- End of stack trace from previous location ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) +
  0xbe
     at
  System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotificat
  ion(Task, ConfigureAwaitOptions) + 0x4e
     at
  Microsoft.Testing.Platform.Hosts.CommonTestHost.d__16
  .MoveNext() + 0xd2
  --- End of stack trace from previous location ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) +
  0xbe
     at
  System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotificat
  ion(Task, ConfigureAwaitOptions) + 0x4e
     at
  Microsoft.Testing.Platform.Hosts.CommonTestHost.d__12.MoveN
  ext() + 0x318
  --- End of stack trace from previous location ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) +
  0xbe
     at
  System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotificat
  ion(Task, ConfigureAwaitOptions) + 0x4e
     at
  Microsoft.Testing.Platform.Hosts.ConsoleTestHost.d__10.MoveNex
  t() + 0x867
  --- End of stack trace from previous location ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
     at
  Microsoft.Testing.Platform.Hosts.ConsoleTestHost.d__10.MoveNex
  t() + 0x10b3
  --- End of stack trace from previous location ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) +
  0xbe
     at
  System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotificat
  ion(Task, ConfigureAwaitOptions) + 0x4e
     at
  Microsoft.Testing.Platform.Hosts.CommonTestHost.d__10.MoveNext(
  ) + 0x233
  --- End of stack trace from previous location ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) +
  0xbe
     at
  System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotificat
  ion(Task, ConfigureAwaitOptions) + 0x4e
     at Microsoft.Testing.Platform.Hosts.CommonTestHost.d__8.MoveNext()
  + 0x18e
  --- End of stack trace from previous location ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
     at Microsoft.Testing.Platform.Hosts.CommonTestHost.d__8.MoveNext()
  + 0x722
  --- End of stack trace from previous location ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) +
  0xbe
     at
  System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotificat
  ion(Task, ConfigureAwaitOptions) + 0x4e
     at
  Microsoft.Testing.Platform.Builder.TestApplication.d__17.MoveNext() +
  0xaf
  --- End of stack trace from previous location ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) +
  0xbe
     at
  System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotificat
  ion(Task, ConfigureAwaitOptions) + 0x4e
     at TestingPlatformEntryPoint.d__0.MoveNext() + 0x224
  --- End of stack trace from previous location ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) +
  0xbe
     at
  System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotificat
  ion(Task, ConfigureAwaitOptions) + 0x4e
     at TestingPlatformEntryPoint.(String[] args) + 0x2e
     at TUnit!+0xed2b6c

outsider here, hoping to not distract -
How constrained does this mean?  Is the system just totally trashed (swapping, etc.)?

I guess due to constrained system resources this is being hit.

It skips a test after 6 seconds and then takes over 3 minutes to pass one test.  Is that expected?
  [+0/x0/?0] TUnit.Engine.Tests.dll (net8.0|arm64)(3s)
  [+0/x0/?0] TUnit.Engine.Tests.dll (net8.0|arm64)(6s)
  Skipping JsonOutputTests.Test...
  skipped Test (0ms)
  [+0/x0/?1] TUnit.Engine.Tests.dll (net8.0|arm64)(9s)
...
  [+0/x0/?1] TUnit.Engine.Tests.dll (net8.0|arm64)(3m 27s)
  [+1/x0/?1] TUnit.Engine.Tests.dll (net8.0|arm64)(3m 30s)

Yeah I was looking at how long the tests run as well, but there was no complain from OP about the testing being slow, and I don't know their tests.
I've discussed the timeout issue internally, and the implementation of the timeout is not super helpful, because it uses wallclock time rather than how much time the regex actually spent executing on cpu. So if the thread / runner gets suspended, the regex will still time out. In all our tries we could not get the regex itself to run over 1ms, which is still more than I would expect on matching a single line. And the regex is linear, so it should be okay, to simply remove the timeout.

Removing the timeout sounds sensible to me


feat: support specific framework & plugin in scanner

此 PR 旨在允许通过 scanner 扫描时指定框架和插件列表来启动应用，此时应用自身的插件或者框架配置不管存在与否都不再生效。
close #150

Codecov Report

Merging #158 (31b76c2) into master (df834fc) will increase coverage by 0.02%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #158      +/-   ##
==========================================
+ Coverage   89.25%   89.28%   +0.02%
==========================================
  Files          51       51
  Lines        1089     1092       +3
  Branches      178      179       +1
==========================================
+ Hits          972      975       +3
  Misses        117      117

Impacted Files
Coverage Δ

src/scanner/scan.ts
94.96% <100.00%> (+0.11%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update df834fc...31b76c2. Read the comment docs.


[Conversion Problem] Conversion fails for asset with multiple input files

Describe the problem
At some point in the last 2 months our method of converting certain models (which contain multiple files referenced by a single .gltf) has stopped working. We don't convert models like this that often, so we hadn't noticed the breakage and can't really narrow down exactly when it occurred.
Files that use a layout as simple as the following exhibit the problem:
<storagecontainer>/<dirname>/scene.gltf
<storagecontainer>/<dirname>/scene.bin
And produce output .results.json files with contents like the following:
{"error":"2007","title":"The input asset is not valid.","details":{"import error":"GLTF: could not open referenced file \"scene.bin\""}}
The code on our side which uploads the files to an input storage container and constructs + issues the REST request to start the conversion hasn't changed during the time since this was working.
We've also recreated the issue using the Conversion.ps1 powershell script in this repo and via manually constructing and issuing REST requests. Our accounts use and always have used linked storage, but just in case we've tried both with and without providing the inputLocation.storageContainerReadListSas argument to the create request. We're seeing the same results in all cases.
The one method that doesn't seem to generate this issue, and which does work, is the latest version of the ARRT tool. Performing the equivalent operations to kick off a conversion with the tool produces a successful result.
Provide additional information
Below I'll provide a bunch of conversionIds, how they got generated, and the error/success results for each. Unless otherwise specified, they used this file layout, which is just this sketchfab file, downloaded using sketchfab's 'autoconvert GLTF' download option:
.
├── license.txt
├── scene.bin
└── scene.gltf

Direct REST submission using linked storage account d33167a4-f7a1-4b97-8d16-6ebcd278d855
Failed: {"error":"2007","title":"The input asset is not valid.","details":{"import error":"GLTF: could not open referenced file \"scene.bin\""}},
Direct REST submission using SAS URI 602d96b6-7719-4257-bcd4-8e2020967711
Failed: {"error":"2007","title":"The input asset is not valid.","details":{"import error":"GLTF: could not open referenced file \"scene.bin\""}}
ARRT submission 0b89cd8c-deeb-49b7-8649-129cd8643420
Succeeded!

Additional context
In addition to those above, here are the conversion IDs for an old conversion of similar assets (i.e. a multi-file model obtained from sketchfab with identical file layout). These were both submitted for conversion to the same ARR resource, with the same storage account, using the same code and the same original assets (a gltf zip of this sketchfab asset).
One succeeded, one failed. Unfortunately these conversion IDs, despite coming from the API originally and being stored in our database, no longer seem to be available for lookup in the conversion API (or something else has gone wrong on our end and those conversions were somehow created in a different region/account, but I feel like that's unlikely looking at our code + config changelog). Do finished conversions have some kind of maximum lifetime after completion that they're available for query?

May 4th (success) 29aca865-5cd4-431f-87c5-564ebffa1bd7
July 18th (failure) 5f00fc8b-1182-483f-8334-633102179cd0

Hey @robbbbbb thank you for the very detailed description. This gives us a lot of information to try to reproduce and get to the bottom of the problem, and we'll come back to you as soon as possible.
Generally this kind of conversion is pretty standard for GLTF files and we are currently not aware of any issues. The only thing that comes to my mind right now, is that the ARRT tool generates an access token for the entire container, so that the conversion service has full access to everything. Code is here. Maybe your token is more restrictive? Of course our ps1 scripts should do the right thing regardless, so we'll investigate that use-case.
To answer your last question: Yes, the conversion IDs are only kept in the database for a limited time (either a week or a month, can't remember the details). So it's expected that you don't get data about them anymore.
Sorry for the inceonvenience!

Thanks for the quick response @jakrams, our production code has always just used linked storage accounts rather than SAS URIs. The test I did using a SAS URI was was just a URI (container level, Read + List) I manually generated in the azure portal.
By the way, looking at the ARRT code you linked I noticed that it generates the token with Read + Write + List + Create privileges (see this line) rather than just the Read + List privileges specified here.
The Conversion.ps1 powershell script is similar: it generates a SAS token with full container privileges, including Delete (see this line)
Just to double check, I generated a SAS token with the same privileges used by the ARRT and submitted it via the REST API, but the conversion still failed, so I'm guessing Read + List are still valid and the issue is elsewhere.

Just to let you know: we can reproduce the issue and are currently investigating what's going wrong. I'll keep you posted.

It appears that a recent optimization broke multi-file conversions under certain conditions. We have a fix that we are currently testing and if all goes well, we will deploy it in the next few days. I'll keep this ticket open until then.
Thanks a lot for the very detailed report, it helped us find the issue relatively quickly.

Awesome, thanks for the quick work @jakrams, I'll keep an eye on this issue 👍

Hey @robbbbbb the fix for the conversion service has been deployed. Our tests have been successful, but it would be great, if you can give it a shot and let us know whether it works as expected for you now as well. You do not need to update your SDK version or anything, just run a new asset conversion.

@jakrams I can confirm multiple file conversions are now working again for us, thanks for the fast resolution 🙏


hi

I copied the link on my chrome browser and it's not opening at all:
http://octanebenchmark.
googlecode.com/svn/latest/index.html

That is the old link, please refer to the README file for the new link.


Refactor settings

Tasks

Merges the two settings tabs
Update settings wording
Change recording fps control to a select
Add looping option and connect to export

Preview

Exporting with looping off:

Exporting with looping on:

Questions

Wasn't sure if looping should be enabled by default (for now I enabled it)
Wasn't sure if we are adding the Kaptures dir (wasn't in the mockup) and what are we doing about the path thing

versus

I agree, looping should be on by default for all formats that support it.
Previously we ended up shortening the path to .../Kaptures, I believe the reasoning for that is we never know how long the path is and that could cause issues in the UI. What are your thoughts on it?

I agree, looping should be on by default for all formats that support it.

👍

Previously we ended up shortening the path to .../Kaptures, I believe the reasoning for that is we never know how long the path is and that could cause issues in the UI. What are your thoughts on it?

The ../Kaptures always confused me. It's not clear where it actually is. I think it would be better to have the path as ~/Movies/Kaptures and if it's too long, we can just shorten it with text-overflow: ellipsis.

I think it would be better to have the path as ~/Movies/Kaptures and if it's too long, we can just shorten it with text-overflow: ellipsis.

Sounds good @sindresorhus, we can also make the path clickable to open the specified folder!

@skllcrn @sindresorhus
Sounds good.

What should we use instead of "Kaptures" for looping. "Recordings"?
If the path is clickable (open in finder) should it be styled differently to appear clickable?
Is there an easy way to get the path as ~/... instead of /Users/username/...?

What should we use instead of "Kaptures" for looping. "Recordings"?

GIF is the only format with support for looping right? Then maybe "Loop gifs"

What should we use instead of "Kaptures" for looping. "Recordings"?

👍, but @skllcrn should weigh in here too.

If the path is clickable (open in finder) should it be styled differently to appear clickable?

Style it as a link?

Is there an easy way to get the path as ~/... instead of /Users/username/...?

https://github.com/sindresorhus/tildify

I think it's a nice touch to call them "Kaptures" in the UI, we do that for the cursor settings in Preferences too, as well as the default filename of exports.

Style it as a link?

👍

When attempting to export using a Plugin with a missing or incorrect configuration, show a dialog with options to cancel or open the configuration in question:

Once a Plugin has been installed, send a notification (banner) to indicate that it's ready for use. In the case of a Plugin that requires configuration there's an action associated with the notification (alert) and it does not dismiss automatically:

Review your plugin configuration to continue exporting

Should mention the name of the plugin.

The alert dialog should also use a more descriptive button than OK. I would use Review.

Those alerts and notifications will probably not be part of this PR, but the next one I'm working on for the plugin configuration

Awesome @karaggeorge, want me to create a separate issue for it?

@skllcrn Yes, do it so it's not forgotten.


The Workflows is Not Working

when i try to run this workflow its show me error that nodejs version is Old update it

Closing this issue due to no response. Feel free to re-open this again if your getting the issue


Added support for explicitly setting the dpi(xres,yres) on TIFF images.

In reference to #823 .
This PR exposes the xres and yres properties from libvips.
http://jcupitt.github.io/libvips/API/current/VipsForeignSave.html#vips-tiffsave

Coverage increased (+0.009%) to 99.319% when pulling 76ee84ecadda85c92bc2b07cfe11596eb0c79918 on YvesBos:master into 361ed98353a67ecdf2e68a218b677bf2de25665c on lovell:master.

Coverage increased (+0.009%) to 99.319% when pulling 721242d9e8856609678621ba97194b23e5b3a06b on YvesBos:master into 361ed98353a67ecdf2e68a218b677bf2de25665c on lovell:master.

Coverage increased (+0.009%) to 99.319% when pulling c01a445e495841f2eccfb14ee6e768d4e5dcc592 on YvesBos:master into 361ed98353a67ecdf2e68a218b677bf2de25665c on lovell:master.

Coverage increased (+0.009%) to 99.319% when pulling 01b47db4dcea911064c77153fb882b8ef57d08c8 on YvesBos:master into 361ed98353a67ecdf2e68a218b677bf2de25665c on lovell:master.

Coverage increased (+0.009%) to 99.319% when pulling 01b47db4dcea911064c77153fb882b8ef57d08c8 on YvesBos:master into 361ed98353a67ecdf2e68a218b677bf2de25665c on lovell:master.

Thanks for the updates, looking good.

"I also added xres, and yres when writing to buffer. However, these values do not get saved when calling the ToBuffer method on a tiff image."

Based on the code in thie PR, it sounds like a possible bug in libvips or libtiff itself. I should have a chance to investigate this further at the weekend if you haven't been able to work it out in the meantime.

I had a quick look at this branch and added your proposed toBuffer based scenario and the test passed. This was on Linux, using the pre-compiled libvips v8.5.5 binaries.

Coverage increased (+0.009%) to 99.319% when pulling 4c2b0e17dd65ca1c0bbb640e2ce5353932cb00d4 on YvesBos:master into 361ed98353a67ecdf2e68a218b677bf2de25665c on lovell:master.

A reinstall of sharp did the trick, pretty weird as I was also on libvips v8.5.5.
Glad it's working though, thanks for your help.

A reinstall of sharp did the trick, pretty weird as I was also on libvips v8.5.5.
Glad it's working though, thanks for your help.

Fantastic, thank you very much for this Yves.


[BUG] rossmann nvt+tf inference on triton crashes after receiving a query

Describe the bug
After starting the triton server for rossmann ensemble model (nvt + tf) with the python backend, if you send a query, it crashes the triton and gives a segfault.
Below is the gdb output:
Thread 177 "tritonserver" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fee57fff000 (LWP 6844)]
0x00007ff7ad8aa44f in triton::backend::python::Tensor::Tensor(triton::backend::python::Tensor const&) () from /opt/tritonserver/backends/python/libtriton_python.so
Steps/Code to reproduce bug
If you run the integration tests in this PR https://github.com/NVIDIA/NVTabular/pull/848 with order defined in this PR https://github.com/NVIDIA/NVTabular/pull/863#event-5028551967, with the main branch pulled down in the containers, you can reproduce the bug.

Ran the same integration test with NVTabular C++ backend got the following error (btw, C++ backend did't crash the triton :) );

/usr/local/lib/python3.8/dist-packages/tritonclient/grpc/__init__.py:61: InferenceServerException```

Looks like, we are missing one of the outputs here.

After printing the transformed data columns in model.py, noticed that not all of the required outputs are being returned. And, this is the reason why we get that error. Below is the list of the columns that are included in the transformed data;
Store
DayOfWeek
Year
Month
Day
StateHoliday
CompetitionMonthsOpen
Promo2Weeks
StoreType
Assortment
PromoInterval
CompetitionOpenSinceYear
Promo2SinceYear
State
Week
Events
Promo_fw
Promo_bw
StateHoliday_fw
StateHoliday_bw
SchoolHoliday_fw
SchoolHoliday_bw```

And we are missing many columns here.

I have a fix in #972 -  I didn't see this when I was testing myself because this doesn't happen with #926


fix jwt_token_test puzzling exp param

fix confused jwt_token_test exp variable in claim struct.

Hi @jinglina. Thanks for your PR.
I'm waiting for a istio member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.
Once the patch is verified, the new status will be reflected by the ok-to-test label.
I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@googlebot I signed it!

/ok-to-test

@jinglina: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

release-notes_istio
330eaedb22cee29f3ea8d63d470683082ebeef05
link
/test release-notes_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@jinglina: The following tests failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

release-notes_istio
330eaedb22cee29f3ea8d63d470683082ebeef05
link
/test release-notes_istio

unit-tests_istio
330eaedb22cee29f3ea8d63d470683082ebeef05
link
/test unit-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


Visual Studio and maybe other programs won't get detected

I clicked "Get Installed" but visual studio is not ticked thought it is installed but not on my C: drive but on my D:

I had winutil install vscode for me to C: drive, but it wasn't detected with "Get Installed".  "Github Desktop" wasn't detected the same way either.

Thanks. I just tried the latest version, and now "VS Code" is detected as expected.  Unfortunately, now "VS Codium" is detected, but I never installed it manually nor from winutil.


[Bug]: No more possible to install new theme at portal level

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?
I've tried to install new theme throught Settings=>Extensions and Manage=>Themes but in both way the result is the same:

Install from PB menu Settings => Extensions: Theme has been installed to \Portals\_default\Skins\
Install from PB menu Manage => Themes: Theme has been installed to \Portals\_default\Skins\

It is a issue because throught Settings => Extensions it should install the theme in the path \Portals\PortalName-System\Skins\
Now, to install a new theme at Portal level I have to move it manually to the right folders.
Steps to reproduce?

New DNN 9.12.0 installation
Install new theme from PB menu Manage => Themes

Current Behavior
The new theme is installed under \Portals\_default\Skins\
Expected Behavior
The new theme sould be installed under \Portals\PortalName-System\Skins\
Relevant log output
No response
Anything else?
No response
Affected Versions
9.12.0 (latest release)
What browsers are you seeing the problem on?
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

@dnnsoftware/approvers correct me if I'm wrong, but I'm 99.9% sure that this was an intentional change years back to no-longer support portal based themes.

@mitchelsellers that was a conscious decision (for security reasons) to remove the portal skins feature as we knew it.  However, I am pretty sure it is possible again as of DNN 9.1.1 for a super user to install a site-level theme.  I have never used or tried this, but I am pretty sure it is possible again.  If my memory serves me correctly, those site themes live in a different folder structure than before.

My apologies for not fully reading the original issue post @Scippy - I think you are correct in what is expected.  That rings a bell to me.

I don't think it's possible that it's intentional.
If it were no longer supported, installing the theme at the global level would be a huge limitation. It would mean that if that skin is used for multiple portals, its customization would affect all the portals for which that skin is in use.
Furthermore, the skins are also sold with specific licenses for uses in multiple portals on the same DNN installation, what would be the point if the same skin is shared among all the portals?

Without divulging too much here in the open forum, the conversation/concern about the existing "Site Themes" functionality was that a non-privledged user could upload/update a theme package.  Theme packages contain executable code and are not validated.
I see the concern desire here with multiple themes, but there isn't al imitation that you cannot have many in the global level.
I'm not sure that the theme/configuration process even supports site-level themes anymore.  This warrants a deeper dive for those with "Super User" permissions to possibly drop into the /Portals/{PORTALHOME}/skins directory

The limitation at global level is that I'm no able to install the same skin multiple times. Then using the same skin on multiple portal don't allow me to customize the skin for each portal but any customization it would reverberate in all portal.
For design work it is unthinkable that you cannot customize the theme because it is shared among several portals. otherwise we have to inform all skin vendors that the professional license is useless!

IMO customizing a skin directly is a bad practice, one should make a copy and customize the copy... The copy would have a different name.

Now, to install a new theme at Portal level I have to move it manually to the right folders.
I believe this is what you should do. DNN still recognizes the appearance of a theme in /Portals/{...}/skins (and /containers), so in your situation, my suggestion is that you have your solution (workaround).
If there is nothing special that the installer used to do that is somehow hard or not obvious when doing it manually (unzip the installer, copy files to the right place(s), read the .dnn manifest for clues), then this seems like a good solution.
@mitchelsellers, @david-poindexter, @valadas, @bdukes - are there plans that DNN v10 or v11 will prohibit (prevent or ignore) the use of themes at the Site (Portal) level?

@Scippy typically, for multi-site themes, we simply update our theme build automation to package up each theme installation package separately (but still install them globally - the packages just have unique names to distinguish usage).  We share some code between themes, but for those items that are unique to a specific portal, we use strict naming conventions that are used by our build automation to package them accordingly.  That's for more complex scenarios, but most of the time the differences between sites are minimal and do not require this level of effort.  For example, it may just be some CSS differences, so we just use specificity within the CSS to handle the differences.

Thanks everyone for the extensive explanations. yes ok, I have my workaround but it seems to me that the problem is much more danger than a bug. As I understand it, globally installed skins are safe while locally installed skins are not. Right? If so, it means that now I find myself with dozens of unsafe portals!😱 To solve it, I have to move all the graphics under the _default folder by renaming the folders with specific names?
Is this security issue there even if i am the only host and admin user above all portals?

As I understand it then globally installed skins are safe while locally installed skins are not. Right? If so, it means that now I find myself with dozens of unsafe portals!

This isn't the actual issue.  The issue was that the older functionality allowed NON Super Users to upload Themes (Skins) to the portal specific elements.  THIS introduced a risk.
If you are doing it as a host user, it isn't introducing any more risk than normal


Add forward and reverse link type description

This lets us have a forward and reverse description in link types to give helpful text inside the drop down for link types in the UI.

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@7e2a8fd). Click here to learn what that means.
The diff coverage is 100%.

@@            Coverage Diff            @@
##             master    #2231   +/-   ##
=========================================
  Coverage          ?   69.79%
=========================================
  Files             ?      170
  Lines             ?    15877
  Branches          ?        0
=========================================
  Hits              ?    11082
  Misses            ?     3779
  Partials          ?     1016

Impacted Files
Coverage Δ

migration/migration.go
61.16% <100%> (ø)

workitem/link/type.go
92.72% <100%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7e2a8fd...3cef9e9. Read the comment docs.


Impeller: Strange issue with path fills

Using the new Impeller interop toolkit, and I've run into a fairly elusive issue that causes path fills to not be drawn when they should.
The issue seems to involve interactions between transforms, clip regions, and the DisplayListBuilder save stack. After adding lots of logging, I've reduced the issue down to a fairly simple example:  https://gist.github.com/lyceel/de6d80caffb3c209d3dadbdfd10447de
With the code as it is, the path is not filled at all:

If either the rounded rectangle's transform (Option 1 comment), or the unused clip (Option 2) are commented out, the path is filled properly with red:

I recognize that the simple solution would be to remove the useless save/clip/restore block in the middle, but this is much more difficult to prevent in the real-world use case where this issue was discovered.

This issue started with the switch to the experimental canvas.
Possibly this is similar to the issue seen in https://github.com/flutter/flutter/issues/153504?

Thanks for the breadcrumb. I'll compare different backends and attempt a fix if thats the case.

Converted the example to a DisplayListBuilder based test case:
https://gist.github.com/jason-simmons/f9e450e3f8379a14a1de7381299e16c8
Using this, the issue reproduces on the GLES back end but not Metal.

The intersect clip prep call seems to be doing some funky stuff to the depth buffer on OpenGL. The Metal backend comparison is

This will be fixed in https://github.com/flutter/engine/pull/56310

Fixed in https://github.com/flutter/engine/pull/56310

Thanks for the quick turnaround!


Proxy Protocol Support

What?
I’d like to implement Proxy Protocol Support to help balance WebSocket connections served by Caddy in production. It’s supported by both NGINX (docs) and HAProxy (docs).
Why?
It allows us to know the origin of requests behind external load balancers which route traffic to a cluster of Caddy instances.
How?
The existing middleware API can’t be used to implement this feature because the PROXY packet would be sent before Caddy even receives the HTTP method.
For example, a raw request might look like this:
PROXY TCP4 <IP_ADDRESS> <IP_ADDRESS> 56324 443\r\n
GET / HTTP/1.1\r\n
Host: <IP_ADDRESS>\r\n
\r\n

As such, this is something that would have to be parsed, handled and made available at a much lower level. We can write a custom net.Listener to parse the data, then inject it in the HTTP headers in the server’s ServeHTTP method (i.e. X-Forwarded-For)
For end users, there’d be something that could be added in the Caddyfile to enable the use of this listener. For instance a proxy_header directive.

I’d appreciate any feedback. I’ll gladly implement support for this myself after discussion with you beautiful guys and gals :)

So this takes place below the HTTP layer? Caddy is an HTTP server... but this still might be possible as an add-on, even if that isn't possible until 0.9 (which I'm writing now).
Would this custom listener also serve regular HTTP requests?

@mholt Yes, this would have to take place below the HTTP layer.
The custom listener would still serve regular HTTP requests; it would work similarly to the integration between server.Server and server.gracefulListener.
This listener would essentially read only the first line, parse it, and return a net.Conn variant to server.Server which would have the contextual information. If only the first line is read, then the remaining data within the net.Conn buffer should be the expected HTTP request.

Okay, so maybe it would be possible to just wrap the underlying code so that they still see the HTTP request as expected.
I'm not opposed to this. I'm just not sure how it would hook into Caddy, but now's the time to figure that out, since I'm working on the 0.9 rewrite, which tries to answer questions like this. Take a look at what I've got so far, although I need to update that projection with some recent changes. I can do that hopefully early next week.
So basically your plugin would need a way to wrap the listener of the server it's plugging into. Right?

Yeah, my goal would be to make this completely seamless; HTTP requests should appear unmodified except for an additional/explicit tag: X-Forwarded-For.
This plugin would only need a way to wrap the listener of the server, but in order to relay this information via an HTTP header, I’d imagine we’d need to implicitly enable a bundled middleware component.

The middleware component shouldn't be a problem; that's already possible. Hmm, well, if you take a gander in that 0.9 branch inside the caddyhttp/httpserver folder, you'll find server.go and plugin.go, two relevant files for plugging into the HTTP server. If you have any proposals/suggestions how to go about that (even if you're not sure it'll work), let me know! That will help me get a better idea what is needed going forward.

Ok, I’ll have a look early next week and see how it might work in your 0.9 branch. Thanks for the quick response!

@mikepulaski I'm a little ahead of schedule and just updated it with my current progress. Take a gander when you get the chance. It might be as simple as a function hook or callback to wrap a listener.

@mholt At first pass, it seems this feature could be implemented by adding an attribute to SiteConfig, let’s say SiteConfig.AcceptsProxyHeader. If enabled, a custom listener will be either returned by Server.Listen or privately wrapped inside of Server.Serve. Either would hide any unnecessary implementation details from callers, but I’m not sure what the implications are by having it returned by Listen. Are there any use cases where a listener would be used outside of Server.Serve?
The benefit of having it as an attribute within httpserver.SiteConfig is that we can implicitly prepend a middleware component within httpserver.NewServer  to handle the parsed PROXY line so we can inject X-Forwarded-For headers.

@mikepulaski Thanks for the thoughts!

Are there any use cases where a listener would be used outside of Server.Serve?

As of right now, no, however I am not sure I would guarantee that in the future.

it seems this feature could be implemented by adding an attribute to SiteConfig, let’s say SiteConfig.EnableProxyProtocol. If enabled, a custom listener will be either returned by Server.Listen or privately wrapped inside of Server.Serve.

If you would like, you are welcome to draft up a pull request with what the change would look like. I can't guarantee it'll be merged right away, but if it doesn't take you too long I think it could definitely be worth a review.
One tricky thing might be in the MakeServers() function where site configs are grouped by their listeners. Sites that share a listener are of course served with the same listener. This change will muddle slightly what it means to "share" a listener (for example, TLS and plaintext sites can't share a listener because the protocols are different). Whenever possible, I like the state that's contained within a SiteConfig to belong to a site, not the whole listener, because as you will see, it makes things harder.
(Also take note of occurrences of "X-Forwarded-For" in the caddy source code, make sure it wouldn't break.)

The benefit of having it as an attribute within httpserver.SiteConfig is that we can implicitly prepend a middleware component within httpserver.NewServer to handle the parsed PROXY line so we can inject X-Forwarded-For headers.

I'm a little nervous about melding listeners and middlewares as I see those on separate levels, separated by yet another which is the HTTP server itself (and its root ServeHTTP method). But that's why a PR might help, it will help me see what you are thinking of. :)

Now 0.9 is out ;-).
Congratulation very good job.
Where should we consider to add such a protocol in the new 0.9 infrastructure?

@git001 Start by implementing a listener that wraps an underlying net.Listener. Then we'll figure out a good way to wire it up into Caddy once you have a simple http.Serve() example working.

Here's an implementation of the v1 protocol (as supported by AWS ELB) that we're currently using in production: https://github.com/SpruceHealth/go-proxy-protocol
To use it with the http server need to create a custom listener and then wrap it and hand it to http.(*Server).Serve(&proxyproto.Listener{Listener: listener})

I've been using this in production: https://github.com/armon/go-proxyproto

Links that could help:

http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/enable-proxy-protocol.html
http://www.haproxy.org/download/1.8/doc/proxy-protocol.txt

@scalp42 It's not on my TODO list, but somebody is welcome to implement it, as long as it overcomes the technical/perf issues discussed above.

@scalp42 It's not on my TODO list, but somebody is welcome to implement it, as long as it overcomes the technical/perf issues discussed above.

I may be able to take a shot at this in the coming week or two.
@mholt do you know that it would cause an issue for the wrapped listener to return a wrapped Conn struct? If not, the RemoteAddr method could just return the source address instead of the underlying address.
Something like this:
type proxyConn struct {
    net.Conn
    remote net.Addr
}
func (p *proxyConn) RemoteAddr() net.Addr {
    return p.remote
}

Thus keeping it separate from any middleware/header stuff if possible -- I think this could actually be fairly clean.

@mastercactapus I haven't looked into that possibility much, and I haven't looked at the base server code very much in a while, either. You can definitely try that approach, I like cleanness. :grin: I'll recommend that you take care to implement any interfaces that the type you're wrapping implements, so that we don't break anything...

Just an update: I threw something together today. I want to test it a bit more before making a PR, though.
There are 2 new flags: -proxy-protocol-subnets and -proxy-protocol-timeout
I would have added directives, but alas, 1 vhost != 1 net.Listener. So I followed what QUIC and HTTP2 were doing and added flags (since those two also happen before we know what virtual server is being requested)
Also, I implemented subnet filters instead of a boolean, to limit who could send the PROXY.... header. It's really only for traffic from the ELB or what-have-you. (one could set it to <IP_ADDRESS>/0,::/0 if they wanted, however)
@mholt let me know what your thoughts are if you get the chance. I know adding more flags isn't ideal, but I couldn't think of a more appropriate place for what needs to happen. And I called the flags generically what they were, but the names kind of stick out -- maybe there's a better name for them that matches the other flags in caddy better.
@mwpastore @samuel I chose the https://github.com/armon/go-proxyproto package, as it made it easy to make the subnet-filtering-listener.
Last but not least, it falls back cleanly if the PROXY header is missing (but not invalid). I'm not sure if nginx and/or haproxy do this, or if they are strict and require the header. It was pretty useful in testing to not have things fail without it though. In either case, if we want it to be strict, the github.com/armon/go-proxyproto package will have to be updated.
mastercactapus/caddy@b022e2cbed5ef33b8ab3898c2cbfaf2da6b053b7
https://github.com/mastercactapus/caddy/tree/proxy_protocol

This looks great! Thanks @mastercactapus. The more difficult part is likely to be implementing PROXY protocol upstream, as it doesn't appear as though there is any existing prior art. Not in Go at least.

Since Caddy now supports listener middleware, Proxy protocol can be implemented as a plugin. I think we're done here!


Fix binder

I switched to docker in order to use a base image containing already FreeCAD, it helps reduce the image build time on Binder.

I don't see the JupyterCAD extension working on Binder, though it seems to be enabled:

That's weird, it worked till yesterday before changing the base image.