Parameterized and multiple argument functionality enabled

Purpose
$title
Goals

Parameterization of variables in the code base
Supporting multiple arguments in a ballerina function
Resolving bugs of supporting multiple function declaration

Approach
New parameters were defined in the xml file to enable the multiple arguments. Singleton introduced for connector and the xml generation will be done at the LifeCycle phase. New parameters and methods introduced to connector to parameterize the variables.
User stories

Summary of user stories addressed by this change>

Release note

Brief description of the new feature or bug fix as it will appear in the release notes

Documentation

Link(s) to product documentation that addresses the changes of this PR. If no doc impact, enter “N/A” plus brief explanation of why there’s no doc impact

Training

Link to the PR for changes to the training content in https://github.com/wso2/WSO2-Training, if applicable

Certification

Type “Sent” when you have provided new/updated certification questions, plus four answers for each question (correct answer highlighted in bold), based on this change. Certification questions/answers should be sent to<EMAIL_ADDRESS>and NOT pasted in this PR. If there is no impact on certification exams, type “N/A” and explain why.

Marketing

Link to drafts of marketing content that will describe and promote this feature, including product page changes, technical articles, blog posts, videos, etc., if applicable

Automation tests

Unit tests

Code coverage information

Integration tests

Details about the test cases and coverage

Security checks

Followed secure coding standards in http://wso2.com/technical-reports/wso2-secure-engineering-guidelines? no
Ran FindSecurityBugs plugin and verified report? no
Confirmed that this PR doesn't commit any keys, passwords, tokens, usernames, or other secrets? yes

Samples

Provide high-level details about the samples related to this feature

Related PRs

List any other related PRs

Migrations (if applicable)

Describe migration steps and platforms on which migration has been tested

Test environment

List all JDK versions, operating systems, databases, and browser/versions on which this feature/fix was tested

Learning

Describe the research phase and any blog posts, patterns, libraries, or add-ons you used to solve the problem.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.samithkavishke seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


why i get a '\r' not a '\n'>? when i use the text.input

why i get a '\r' not a '\n'>? when i use the text.input.
how can i get a '\n' when i enter?
thanks .
pub async fn test_input(st: SystemTable<Boot>) {
    loop {
        let events = &mut [st.stdin().wait_for_key_event()];
        &st.boot_services().wait_for_event(events).unwrap();
        match st.stdin().read_key().unwrap().unwrap() {
            Some(key) => {
                match key {
                    uefi::proto::console::text::Key::Printable(char16) => {
                        // 为什么这里的回车\r 不是 \n 呢？
// here !!!!
                        write!(st.stdout(), "{}", char16).unwrap();
                    }
                    uefi::proto::console::text::Key::Special(scanCode) => {
                        info!("{:?}", key)
                    }
                }
            }
            None => {}
        };
    }
}

Hello, I've tested and reproduced this locally, but I don't think there's something we can fix.
This is just the way UEFI implementation works. UEFI is heavily inspired by Windows, and an enter on Windows would mean \r\n (I'm guessing UEFI returns only the first \r).

Thank you for your response.
I understand. By the way, is there a good way to determine what the input value is? I find that what is returned is a structure called char16. which I can't easily access. Or can I just convert the data to u8 and let something like pc-keyboard cargo handle it? Do you have any good suggestions?

The Char16 structure can be converted with .into() to a u16 or a char, which you can then use normally.

Closing as it sounds like the behavior is expected.


The first request to our app delivers an timeout

Hi,
first of all very cool stuff, that probot framework. I'm started to work with it a few days ago. The current status is, that we deploy an App to Azure functions. It works in general very smooth and stable.
The only problem I have so far is, that after some time, the event which is send to the app could not delivered. I got a timeout for it.
After a short time, the events can then be delivered without any problems and the app works as expected.
I assume it is because the Azure function needs a short time until it is online and can handle the events. Now the question is, what are the possibilities to work around the timeout?
Unfortunately I don't have direct access to the Azure function myself. This was set up by another department
We are not using gh.com. We are running GHES 3.0.6

After a short time, the events can then be delivered without any problems and the app works as expected.
I assume it is because the Azure function needs a short time until it is online and can handle the events. Now the question is, what are the possibilities to work around the timeout?

GitHub has a timeout of 10s, if your app does not respond by that time to a webhook request, they consider it as unsuccessful. You can make sure that your app responds as quick as possible by handling the events asynchronously, maybe use some kind of queue system, so all the event handlers do is checking if you care about the event and write to the queue, then probot can respond to the request.
Then have a separate system were you act upon the requests?
It sounds like you your app might become dormant and if an event occurs after some time, your app or function needs to be restarted, which seems to be taking more than 1s? Probot has a built-in timeout of 9 seconds after which it always responds, but it seems that in your case that is still too long.
I'll have a look if we can be smart about probot's own timeout. Instead of hardcoding it to 9s we can take the time of the event into consideration and then add 9s to that


Flaky agent spec: Kontena::Observer For a supervised observer that observes a supervised actor by name restarts after the observable crashes and waits for it to update

  1) Kontena::Observer For a supervised observer that observes a supervised actor by name restarts after the observable crashes and waits for it to update

     Failure/Error: expect(@observer_actor.values).to eq [1]

       expected: [1]

            got: nil

       (compared using ==)

     # ./spec/lib/kontena/observer_spec.rb:545:in `block (3 levels) in <top (required)>'

Also for the same spec:
  1) Kontena::Observer For a supervised observer that observes a supervised actor by name restarts after the observable crashes and waits for it to update

     Failure/Error<EMAIL_ADDRESS>raise_error(Celluloid::DeadActorError)

     Timeout::Error:

       execution expired

     # ./vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/mailbox.rb:63:in `sleep'

     # ./vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/mailbox.rb:63:in `wait'

     # ./vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/mailbox.rb:63:in `block in check'

     # ./vendor/bundle/ruby/2.4.0/gems/timers-4.1.1/lib/timers/wait.rb:14:in `for'

     # ./vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/mailbox.rb:58:in `check'

     # ./vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/mailbox.rb:77:in `block in receive'

     # ./vendor/bundle/ruby/2.4.0/gems/timers-4.1.1/lib/timers/wait.rb:14:in `for'

     # ./vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/mailbox.rb:76:in `receive'

     # ./vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/call/sync.rb:50:in `wait'

     # ./vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid.rb:141:in `suspend'

     # ./vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/call/sync.rb:41:in `response'

     # ./vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/call/sync.rb:45:in `value'

     # ./vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/proxy/sync.rb:22:in `method_missing'

     # ./spec/lib/kontena/observer_spec.rb:568:in `block (4 levels) in <top (required)>'

     # ./spec/lib/kontena/observer_spec.rb:568:in `block (3 levels) in <top (required)>'

     # ./spec/spec_helper.rb:86:in `block (3 levels) in <top (required)>'

     # ./spec/spec_helper.rb:85:in `block (2 levels) in <top (required)>'

     # ./spec/spec_helper.rb:78:in `block (2 levels) in <top (required)>'

     # ./spec/spec_helper.rb:69:in `block (2 levels) in <top (required)>'


Illegal instruction on Opteron (detects SSE3 when no sse3 present)

I run MLC@Home, and recently recompiled the client to use OpenBLAS instead of MKL.  However, volunteers running on older Opterons reported crashes with SIGILL.  This is easy to reproduce, simply launch a new VM (virt-manager/kvm includes a CPU profile for an opteron 240 (gen 1) ) . Here's the lscpu output for this VM:
$ lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
Address sizes:       40 bits physical, 48 bits virtual
CPU(s):              4
On-line CPU(s) list: 0-3
Thread(s) per core:  1
Core(s) per socket:  1
Socket(s):           4
NUMA node(s):        1
Vendor ID:           AuthenticAMD
CPU family:          15
Model:               6
Model name:          AMD Opteron 240 (Gen 1 Class Opteron)
Stepping:            1
CPU MHz:             3593.248
BogoMIPS:            7186.49
Hypervisor vendor:   KVM
Virtualization type: full
L1d cache:           64K
L1i cache:           64K
L2 cache:            512K
L3 cache:            16384K
NUMA node0 CPU(s):   0-3
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid extd_apicid tsc_known_freq pni x2apic hypervisor 3dnowprefetch vmmcall

Then you run my client with libopenblas compiled into pytorch under GDB, here's the output:
Thread 1 "mlds" received signal SIGILL, Illegal instruction.
0x00007ffff4d9fe04 in sgemm_oncopy_OPTERON_SSE3 () from /home/build/squashfs-root/usr/bin/../lib/libtorch_cpu.so

Obviously, there's no SSE3 in this generation of opteron.  Setting OPENBLAS_CORETYPE=GENERIC allows it to run fine. However, setting OPENBLAS_CORETYPE=OPTERON, still crashes with in the same OPTERON_SSE3 function. So something is very messed up with opteron detection.
OpenBLAS master is compiled with BINARY=64 TARGET=GENERIC USE_THREAD=1 USE_OPENMP=1 DYNAMIC_ARCH=1 DYNAMIC_OLDER=1 MAX_THREADS=64 NO_AFFFINITY=1 NO_WARMUP=1 NO_SHARED=1.

Probably not with detection as such - at the very least it is consistent between compile-time (cpuid_x86.c) and runtime (driver/others/dynamic.c), both using ecx&1 as the flag for SSE3 capability. More likely an SSE3 instruction crept into an OPTERON-only BLAS kernel, or the same SSE3-using kernel is configured for both OPTERON and OPTERON-SSE3. Does
the backtrace for the crash with OPENBLAS_CORETYPE=OPTERON actually lead back to the exact same sgemm_oncopy_OPTERON_SSE3 function (then perhaps "only" the string matching for that particular option is broken) ?

Hmm. WRT detection, it would not be the first time a VM environment does not emulate all details of cpuid correctly.
Also both OPTERON and OPTERON_SSE3 use the same gemm_ncopy_4_opteron.S from the original GotoBLAS, and I see nothing that looks like SSE3 in there. (In fact the only difference in kernels appears to be in ZGEMV, using the MOVDDUP instruction on newer Opterons).

Ok, this is weird..... I could have sworn it did, but its the reverse...?
OPENBLAS_CORETYPE=OPTERON_SSE3 : SIGILL in sgemm_oncopy_OPTERON ()
OPENBLAS_CORETYPE=OPTERON : SIGILL in sgemm_oncopy_OPTERON_SSE3 ()
OPENBLAS_CORETYPE=PENRYN : Works fine.
GENERIC, of course, also works fine.
So you're right, its correctly interpreting the system as opteron, but running sse3 code regardless... why the reverse still gives a sigill, I don't know.   Opteron isn't the only machine having SIGILL issues after we switched from MKL to OpenBLAS, Phenom and Core2 Duos also fail, but I don't have time or hardware to get backtraces on those.  There's a thread over in our forums with the full list which includes a DB dump of all the CPUs I've seen errors on:  https://www.mlcathome.org/mlcathome/forum_thread.php?id=63 .

The systems listed on the forum from our internal DB were from a client linked against the libopenblas.so shipped with Ubuntu 14.04 (so very old), so perhaps those issues are fixed in a newer release.  What you see above is with the opteron testing is a new client I'm testing which compiles in the latest libopenblas (from git) statically into the binary in the hopes of fixing these issues.  This opteron issue in this is the only one I can test and verify its still an issue.

Thanks. Seems the logic (if one can even call it that) which maps the environment variable to the list of supported core types simply has the two Opteron models reversed. This does not explain the SIGILL though, as the sgemm_oncopy maps to the same code in both. An unmet alignment requirement seems more likely than an actual unsupported instruction, I will set up a qemu VM as I do not have the actual hardware. (Same goes for Phenom, I may be able to revive a Core2Duo later - chances are that the OpenBLAS code for these is unchanged from the original GotoBLAS of ten years ago while compilers have advanced to expose formerly harmless mistakes.

An alignment issue would make sense. I'm building with gcc9 and gfortran9 . Good luck and let me know if I can help.
The code is open source https://gitlab.com/clemej/mlds , the embedded openblas build is in theopenblas-build branch, but be prepared because it downloads and compiles all of pytorch so its a pain to build. Also, I'd need to post/send you a data file.
Thanks again for looking into it.

CPUID is from 2nd generation
Authentic SSE3-less is here:
https://www.linuxjournal.com/article/6711

/usr/share/libviert/cpu_map/x86_Opteron_G1.xml
Should have
model="5"
and
feature="3dnow"
feature="3dnowext"

@brada4 not sure if any of this is actually relevant when (a) it does not look like the SIGILL involves SSE3 instructions at all and (b) as I understand it the problem was/is observed on actual hardware.

Just an observation that emulator has wrong bits and may not reflect the real hardware, though opteron and crash is heard in both sides.
CPUID and backtrace from the real thing is needed.

Actually toplevel CPUID never detects OPTERON_SSE3
So no idea where is the truth.

The only non-portable instruction in that code is an mmx or 3dnow "emms/femms" instruction towards the end of sgemm_ncopy_4_opteron.S - Ubuntu happens to have a stale ticket with something similar  https://answers.launchpad.net/ubuntu/+question/284671 where disassembling the crash address in gdb pointed at just that statement.
Unfortunately I cannot reproduce the SIGILL in qemu with the BLAS-Tester testsuite, which would have made things much easier.

I'm posting the binary and instructions on how to run it if that would help. It's a multi-step process since it's an appimage bundle, but in case its helpful:

Download the client appimage executable (~50MB) from here: https://gitlab.com/clemej/mlds/-/jobs/703431871/artifacts/raw/build/mlds.appimage
Download example dataset (6MB) from here: https://www.mlcathome.org/analysis/dataset.hdf5
chmod +x mlds.appimage
If you run mlds.appimage in the same directory as dataset.hdf5 you should see if fail when starting training.

To debug under gdb, run:

./mlds.appimage --appimage-extract to extract the files into the directory "squashfs-root".
cd /squashfs-root/usr/lib
copy or link dataset.hdf5 to this directory with all the libs .
gdb ../bin/mlds

NOTE: This binary includes libopenblas 0.2.18 as a shared library, not the latest git embedded version compiled statically. Since you said the code hadn't changed, I hope that's not an issue. Library is in the same lib runtime directory.
I'm now experiencing another issue where I the build sometimes(?) fails to compile with a segv when compiling openblas master as part of my full build, but I haven't determined if that's pilot error or an actual openblas issue (binutils in ubuntu 16.04 might be too old).

Depends - is it a segv in the compiler, or in one of the tests that are run at the end of the build ? (also this might just be semantics, but "master" branch is outdated, you'll want "develop").
I'll see later if getting gdb to disassemble the highlighted address in your libopenblas tells me anything - ideally it will also point to the femms instruction.

The segv is in the tests I'm pretty sure, but I'm too distracted to run that down at the moment. I'll open another issue if that turns out to be a problem.  And yes, I meant (and have been using) the default develop branch.

zen 2300U with Opteron_G1 VM - no crash to get backtrace from, and detects prescott

@clemej - can you run mlds inside gdb i.e gdb usr/bin/mlds
then when it crashes capture following
t a a bt
t a a disa

Might take a day or so to get back to you on this. day job needs attention at the moment.

No rush, if there is a bug, it is 15+ years old.

@clemej if/when you get back to this and you are in gdb, could you please run disassemble /r 0x00007ffff4d9fe04,+32 similar to what was done in the Ubuntu ticket I linked ? This should tell which assembly instruction inside sgemm_ncopy failed. (My qemu experiment has not turned up anything useful so far, and unfortunately the old distribution I picked to get something that still boots on Opteron is too old to run your appimage so I will have to redo this)

Hi, sorry fro the delay. Yes, the instruction points directly to the femms instruction.
[2020-08-29 12:46:29                    main:448]       :       INFO    :       Starting Training

Thread 1 "mlds" received signal SIGILL, Illegal instruction.
0x00007ffff4da0004 in sgemm_oncopy_OPTERON_SSE3 () from /home/john/squashfs-root/usr/bin/../lib/libtorch_cpu.so
(gdb) disassemble /r 0x00007ffff4da0004,+32
Dump of assembler code from 0x7ffff4da0004 to 0x7ffff4da0024:
=> 0x00007ffff4da0004 <sgemm_oncopy_OPTERON_SSE3+4>:    0f 0e   femms
   0x00007ffff4da0006 <sgemm_oncopy_OPTERON_SSE3+6>:    48 8d 0c 8d 00 00 00 00 lea    0x0(,%rcx,4),%rcx
   0x00007ffff4da000e <sgemm_oncopy_OPTERON_SSE3+14>:   49 89 f2        mov    %rsi,%r10
   0x00007ffff4da0011 <sgemm_oncopy_OPTERON_SSE3+17>:   49 c1 fa 02     sar    $0x2,%r10
   0x00007ffff4da0015 <sgemm_oncopy_OPTERON_SSE3+21>:   0f 8e 65 01 00 00       jle    0x7ffff4da0180 <sgemm_oncopy_OPTERON_SSE3+384>
   0x00007ffff4da001b <sgemm_oncopy_OPTERON_SSE3+27>:   0f 1f 44 00 00  nopl   0x0(%rax,%rax,1)
   0x00007ffff4da0020 <sgemm_oncopy_OPTERON_SSE3+32>:   49 89 d3        mov    %rdx,%r11
   0x00007ffff4da0023 <sgemm_oncopy_OPTERON_SSE3+35>:   4c 8d 24 0a     lea    (%rdx,%rcx,1),%r12
End of assembler dump.

This leads to an interesting (if frustrating) question:

I get a report that users are getting crashes (illegal instruction) when running openblas on their opteron
I load up a KVM VM in virt manager and set the cpu to emulate an opteron (yet it is still running on a Ryzen CPU)
I run my code and get an illegal instruction, and assume it is the same sigill the user sees on real hardware.
Is this sigill a result my emulation environment, or is it the same one they see on real hardware?

Note that I pushed out an update to the client that sets OPENBLAS_CORETYPE=generic, and users are still reporting sigill issues, this time with CORE2-based systems.
I'm still swamped with my dayjob, so I'll be sporadic in responding over the next few days, but thank you again for looking into it.

I've dug a little deeper and we might be chasing a bit of a red herring here.  Here's a list of the CPUs from my database that have shown SIGILL instructions, note the opterons I see are not G1 based.  Also, obviously not all are cause by this issue.  Perhaps these systems are too new for 3dnow?
+------+---------------------------------------------------------------------------------+
| id   | p_model                                                                         |
+------+---------------------------------------------------------------------------------+
|  293 | Six-Core AMD Opteron(tm) Processor 2431 [Family 16 Model 8 Stepping 0]          |
|  345 | Intel(R) Core(TM)2 Duo CPU T5870 @ 2.00GHz [Family 6 Model 15 Stepping 13]      |
|  349 | AMD Phenom(tm) II X6 1090T Processor [Family 16 Model 10 Stepping 0]            |
|  477 | AMD Opteron(tm) Processor 6128 HE [Family 16 Model 9 Stepping 1]                |
|  573 | AMD A8-3820 APU with Radeon(tm) HD Graphics [Family 18 Model 1 Stepping 0]      |
| 1359 | AMD Phenom(tm) II X6 1090T Processor [Family 16 Model 10 Stepping 0]            |
| 1473 | Intel(R) Core(TM)2 CPU 6400 @ 2.13GHz [Family 6 Model 15 Stepping 6]            |
| 1588 | Intel(R) Pentium(R) 4 CPU 3.00GHz [Family 15 Model 4 Stepping 9]                |
| 1646 | Six-Core AMD Opteron(tm) Processor 8425 HE [Family 16 Model 8 Stepping 0]       |
| 1988 | Intel(R) Core(TM)2 Duo CPU T7500 @ 2.20GHz [Family 6 Model 15 Stepping 11]      |
| 2032 | Six-Core AMD Opteron(tm) Processor 2431 [Family 16 Model 8 Stepping 0]          |
| 2066 | Intel(R) Core(TM)2 Duo CPU     T7500  @ 2.20GHz [Family 6 Model 15 Stepping 11] |
| 2169 | AMD Phenom(tm) II X6 1055T Processor [Family 16 Model 10 Stepping 0]            |
| 2170 | AMD Phenom(tm) II X6 1055T Processor [Family 16 Model 10 Stepping 0]            |
| 2171 | AMD Phenom(tm) II X6 1055T Processor [Family 16 Model 10 Stepping 0]            |
| 2172 | AMD Phenom(tm) II X6 1055T Processor [Family 16 Model 10 Stepping 0]            |
| 2173 | AMD Phenom(tm) II X6 1055T Processor [Family 16 Model 10 Stepping 0]            |
| 2190 | Intel(R) Core(TM)2 Duo CPU     T7300  @ 2.00GHz [Family 6 Model 15 Stepping 11] |
| 2214 | Intel(R) Core(TM)2 Duo CPU     T7300  @ 2.00GHz [Family 6 Model 15 Stepping 10] |
| 2221 | AMD Athlon(tm) 64 X2 Dual Core Processor 4000+ [Family 15 Model 107 Stepping 1] |
| 2230 | Intel(R) Core(TM)2 Duo CPU T5870 @ 2.00GHz [Family 6 Model 15 Stepping 13]      |
| 2232 | Intel(R) Core(TM)2 Duo CPU     T7500  @ 2.20GHz [Family 6 Model 15 Stepping 11] |
+------+---------------------------------------------------------------------------------+

I apologize for not looking this deep sooner.. I just assumed the sigill I was seeing was the same.  Now I suspect the one i see if a matter of using KVM to emulate an old opteron on a system without 3dnow.

Yes it is frustrating - I cannot answer that question as I do not have old Opteron hardware. Ideally "we" would coerce one of your users into running your software under gdb to see if they get the same location in their traceback. or you could supply them a modified build with the #define EMMS in common_x86_64.h changed to return either emms or an empty string and see what happens.
Of course the SIGILL on Core2 is likely to have a different cause (but I notice that while KERNEL.CORE2 names a different source file for the sgemm_oncopy operation, the copy.S used for scopy references the EMMS macro as well.)

Agreed. without a stack trace on hardware its likely impossible.  I own a core2 system I can try on (thanks thinkpad t400!), and I may have access to one of the newer opterons to get a stack trace and experiment a little on real hardware.  Give me another day or so.

The Opteron 2431 in your list looks new enough to expect it to handle all of MMX, 3DNOW and SSE3, in fact it has its separate ISTANBUL target in OpenBLAS. The gcc compile farm has an Opteron 2212 system that would probably correspond to OpenBLAS' OPTERON_SSE3, I can see if I can repeat the SIGILL there (hope their  OS is new enough to have glibc >=2.14 as required by your appimage).

Also related: https://community.amd.com/thread/159993

Not that it matters for this bug, but I can't reproduce an issue on my penryn-based Core2 system. However, I note that all the core2 systems in that list are Merom, one (tiny) generation behind.

FEMMS is 3dnow instruction. It works on authentic old opteron, but not on emulator. So what instruction fails on that old opteron?

@clemej interesting find but 3DNOW is probed via cpuid instructions in a dedicated build, and only assumed to be available in Athlon and Opteron cpus in DYNAMIC_ARCH builds.

Unfortunately one of the two Opteron-2212 hosts in the gcc compile farm runs ancient debian5 (glibc 2.7, no problems seen there with BLAS-Tester though), the other currently does not accept my login.

Update - user error w.r.t. my gcc login on the other Opteron. However mlds.appimage --extract fails with a SIGILL in _static_initialization_and_destruction_0 from its libtorch.so.0, without any apparent involvement of OpenBLAS:
/proc/cpuinfo on that system claims sse, sse2, 3dnow, 3dnowext 3dnowprefetch capabilities, the OpenBLAS build system detects it as "OPTERON".

you mean mlds-appimage --appimage-extract, right?

Right, just caught this mistake.. However trying to run the mlds binary from gdb after successful extraction, with dataset.hdf5 placed in squash-fs/usr/lib, again results in a SIGILL in the same location.
0x00007ffff1da3420 in __static_initialization_and_destruction_0 () from /home/<myusername>/squashfs-root/usr/bin/../lib/libtorch_cpu.so

Alright, then this might not actually be an libopenblas bug. If you would be so kind as to post the disassembler output and then close this as not-a-bug, I would appreciate it.  And my sincerest apologies leading you on a wild goose chase.

Full backtrace (same in single-threaded i..e OMP_NUM_THREADS=1, and with OPENBLAS_CORETYPE=GENERIC)
Program received signal SIGILL, Illegal instruction.
0x00007ffff1da3420 in __static_initialization_and_destruction_0 () from /home/<username>/squashfs-root/usr/bin/../lib/libtorch_cpu.so
(gdb) bt
#0  0x00007ffff1da3420 in __static_initialization_and_destruction_0 () from /home/<username>/squashfs-root/usr/bin/../lib/libtorch_cpu.so
#1  0x00007ffff7de879a in call_init (l=<optimized out>, argc=argc@entry=1, argv=argv@entry=0x7fffffffe558, env=env@entry=0x7fffffffe568) at dl-init.c:72
#2  0x00007ffff7de88ab in call_init (env=0x7fffffffe568, argv=0x7fffffffe558, argc=1, l=<optimized out>) at dl-init.c:30
#3  _dl_init (main_map=0x7ffff7ffe170, argc=1, argv=0x7fffffffe558, env=0x7fffffffe568) at dl-init.c:120
#4  0x00007ffff7dd9c5a in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#5  0x0000000000000001 in ?? ()
#6  0x00007fffffffe781 in ?? ()
#7  0x0000000000000000 in ?? ()
(gdb) quit

gdb disassemble /r at this point gives
Dump of assembler code for function _Z41__static_initialization_and_destruction_0ii.constprop.0:
   0x00007ffff1da33c0 <+0>:     41 57   push   %r15
   0x00007ffff1da33c2 <+2>:     48 8d 0d e7 d6 39 03    lea    0x339d6e7(%rip),%rcx        # 0x7ffff5140ab0 <_ZN4dnnl4impl3cpu16simple_reorder_tIL16dnnl_data_type_t3EL17dnnl_format_tag_t1ELS3_2ELS4_52ELb1EvE4pd_t6createEPPNS0_12reorder_pd_tEP11dnnl_enginePK19dnnl_primitive_attrSB_PK18dnnl_memory_desc_tSB_SH_>
   0x00007ffff1da33c9 <+9>:     48 8d 15 a0 bf 39 03    lea    0x339bfa0(%rip),%rdx        # 0x7ffff513f370 <_ZN4dnnl4impl3cpu16simple_reorder_tIL16dnnl_data_type_t3EL17dnnl_format_tag_t5ELS3_2ELS4_68ELb1EvE4pd_t6createEPPNS0_12reorder_pd_tEP11dnnl_enginePK19dnnl_primitive_attrSB_PK18dnnl_memory_desc_tSB_SH_>
   0x00007ffff1da33d0 <+16>:    41 56   push   %r14
   0x00007ffff1da33d2 <+18>:    41 55   push   %r13
   0x00007ffff1da33d4 <+20>:    41 54   push   %r12
   0x00007ffff1da33d6 <+22>:    55      push   %rbp
   0x00007ffff1da33d7 <+23>:    53      push   %rbx
   0x00007ffff1da33d8 <+24>:    48 8d 1d d1 c7 39 03    lea    0x339c7d1(%rip),%rbx        # 0x7ffff513fbb0 <_ZN4dnnl4impl3cpu16simple_reorder_tIL16dnnl_data_type_t3EL17dnnl_format_tag_t5ELS3_2ELS4_74ELb1EvE4pd_t6createEPPNS0_12reorder_pd_tEP11dnnl_enginePK19dnnl_primitive_attrSB_PK18dnnl_memory_desc_tSB_SH_>
   0x00007ffff1da33df <+31>:    48 81 ec 68 0e 00 00    sub    $0xe68,%rsp
   0x00007ffff1da33e6 <+38>:    64 48 8b 04 25 28 00 00 00      mov    %fs:0x28,%rax
   0x00007ffff1da33ef <+47>:    48 89 84 24 58 0e 00 00 mov    %rax,0xe58(%rsp)
   0x00007ffff1da33f7 <+55>:    31 c0   xor    %eax,%eax
   0x00007ffff1da33f9 <+57>:    48 8d 05 50 da 39 03    lea    0x339da50(%rip),%rax        # 0x7ffff5140e50 <_ZN4dnnl4impl3cpu21rnn_weights_reorder_tIL16dnnl_data_type_t3ELS3_2EE4pd_t6createEPPNS0_12reorder_pd_tEP11dnnl_enginePK19dnnl_primitive_attrSA_PK18dnnl_memory_desc_tSA_SG_>
   0x00007ffff1da3400 <+64>:    48 8d ac 24 70 02 00 00 lea    0x270(%rsp),%rbp
   0x00007ffff1da3408 <+72>:    48 c7 84 24 08 05 00 00 00 00 00 00     movq   $0x0,0x508(%rsp)
   0x00007ffff1da3414 <+84>:    66 48 0f 6e c0  movq   %rax,%xmm0
   0x00007ffff1da3419 <+89>:    48 8d 05 f0 d2 39 03    lea    0x339d2f0(%rip),%rax        # 0x7ffff5140710 <_ZN4dnnl4impl3cpu16simple_reorder_tIL16dnnl_data_type_t3EL17dnnl_format_tag_t1ELS3_2ELS4_52ELb0EvE4pd_t6createEPPNS0_12reorder_pd_tEP11dnnl_enginePK19dnnl_primitive_attrSB_PK18dnnl_memory_desc_tSB_SH_>
=> 0x00007ffff1da3420 <+96>:    66 48 0f 3a 22 05 1d 1c d6 05 01        pinsrq $0x1,0x5d61c1d(%rip),%xmm0        # 0x7ffff7b05048
   0x00007ffff1da342b <+107>:   48 89 ef        mov    %rbp,%rdi
   0x00007ffff1da342e <+110>:   4c 8d a4 24 90 04 00 00 lea    0x490(%rsp),%r12
   0x00007ffff1da3436 <+118>:   4c 8d ac 24 10 0c 00 00 lea    0xc10(%rsp),%r13
   0x00007ffff1da343e <+126>:   0f 29 84 24 90 04 00 00 movaps %xmm0,0x490(%rsp)
   0x00007ffff1da3446 <+134>:   66 48 0f 6e c1  movq   %rcx,%xmm0
   0x00007ffff1da344b <+139>:   4c 89 e6        mov    %r12,%rsi
   0x00007ffff1da344e <+142>:   48 8d 0d 1b cf 39 03    lea    0x339cf1b(%rip),%rcx        # 0x7ffff5140370 <_ZN4dnnl4impl3cpu16simple_reorder_tIL16dnnl_data_type_t3EL17dnnl_format_tag_t1ELS3_2ELS4_91ELb1EvE4pd_t6createEPPNS0_12reorder_pd_tEP11dnnl_enginePK19dnnl_primitive_attrSB_PK18dnnl_memory_desc_tSB_SH_>

no idea what is going wrong, but I do notice that you ship a lot of libraries but rely on the system libpthread.so.0
Just let me know if and when you want me to try any modified appimage on that system.

that's an sse4_1 instruction.  that is being emitted by my compiler, not openblas. :(. thank you for the help, i''ll take it from here.

Ugh. gcc assuming -march=native or something ? Did not really expect that so did not bother to look up the mnemonic.

It's from the intel DNNL library, which is now opeAPI.. which is embedded in the pytorch build by default.  I'm gonna assume they just don't bother to a) check and b) support CPUs that don't have SSE4.1 or higher.. which was introduces with... drumrolll... penryn core2duos! which would explain all the merom and p4 failures in the above list, and probably most of the K10 amd failures too.

Ah ok. Assuming oneDNN is basically the same thing, the build options section of its documentation mentions a build option DNNL_ARCH_OPT_FLAGS which indeed defaults to requiring sse4.1
https://oneapi-src.github.io/oneDNN/dev_guide_build_options.html (see Warning just above headline for "Runtime CPU dispatcher control" in the middle of that page)


Chat History for new members

I have two questions:
1-how to check and change chat history for new members in tdlib?
2-Is there any way to show the chat history to a member of the group that has already been added to the group in hide (chat history is hide) mode?

In TDLib it is called is_all_history_available. You can find all related fields and methods in https://github.com/tdlib/td/blob/master/td/generate/scheme/td_api.tl.
No.


Fix template path for check-cla

Description
In https://github.com/CesiumGS/cesium/pull/11996, we adjusted the working directory for the check-cla check. This adjust the template path for account for this.
Issue number and link
N/A
Testing plan
Difficult to test locally due to the required environment variables. CI should pass after this check, but this won't work until this PR is merged into main due to the trigger type.
Author checklist

[x] I have submitted a Contributor License Agreement
[x] I have added my name to CONTRIBUTORS.md
[ ] ~I have updated CHANGES.md with a short summary of my change~
[ ] ~I have added or updated unit tests to ensure consistent code coverage~
[ ] ~I have update the inline documentation, and included code examples where relevant~
[x] I have performed a self-review of my code

LGTM, thanks @ggetz
CI failed the first time to what looks like a random error? Re-running it passed but probably should keep an eye out in main


liferay-cli will use a previously built localdev image, leading to errors

As I was looking into errors found with latest localdev PR https://github.com/liferay/liferay-localdev/pull/83, I found that the latest localdev next image fails to build.  As it turns out,  if CLI fails to build new localdev image, it will fall back to using previously built localdev image.
I cleared all my image ccaches and my localdev-server tags and then I re-ran a command I got this error
 ---> Running in eb6200041fdc
groupmod: GID '20' already exists
2022/11/05 16:11:45 Failed to create container localdev-status: Error response from daemon: No such image: localdev-server:latest```

So the issue with CLI is that it does not fail the CLI command if the localdev-image fails to build.

See https://issues.liferay.com/browse/LXC-157


cmd/cover: extend coverage testing to include applications

Proposal: extend code coverage testing to include applications
Author(s): Than McIntosh
Last updated: 2022-03-02
Detailed design document: markdown, CL 388857
Abstract
This document contains a proposal for improving/revamping the system used in Go for code coverage testing.
Background
Current support for coverage testing
The Go toolchain currently includes support for collecting and reporting
coverage data for Golang unit tests; this facility is made available via the "go
test -cover" and "go tool cover" commands.
The current workflow for collecting coverage data is baked into "go test"
command; the assumption is that the source code of interest is a Go package
or set of packages with associated tests.
To request coverage data for a package test run, a user can invoke the test(s)
via:
  go test -coverprofile=<filename> [package target(s)]

This command will build the specified packages with coverage instrumentation,
execute the package tests, and write an output file to "filename" with the
coverage results of the run.
The resulting output file can be viewed/examined using commands such as
  go tool cover -func=<covdatafile>
  go tool cover -html=<covdatafile>

Under the hood, the implementation works by source rewriting: when "go test" is
building the specified set of package tests, it runs each package source file
of interest through a source-to-source translation tool that produces an
instrumented/augmented equivalent, with instrumentation that records which
portions of the code execute as the test runs.
A function such as
  func ABC(x int) {
    if x < 0 {
      bar()
    }
  }

is rewritten to something like
  func ABC(x int) {GoCover_0_343662613637653164643337.Count[9] = 1;
    if x < 0 {GoCover_0_343662613637653164643337.Count[10] = 1;
      bar()
    }
  }

where "GoCover_0_343662613637653164643337" is a tool-generated structure with
execution counters and source position information.
The "go test" command also emits boilerplate code into the generated
"_testmain.go" to register each instrumented source file and unpack the coverage
data structures into something that can be easily accessed at runtime.
Finally, the modified "_testmain.go" has code to call runtime routines that
emit the coverage output file when the test completes.
Strengths and weaknesses of what we currently provide
The current implementation is simple and easy to use, and provides a good user
experience for the use case of collecting coverage data for package unit tests.
Since "go test" is performing both the build and the invocation/execution of the
test, it can provide a nice seamless "single command" user experience.
A key weakness of the current implementation is that it does not scale well-- it
is difficult or impossible to gather coverage data for applications as opposed
to collections of packages, and for testing scenarios involving multiple
runs/executions.
For example, consider a medium-sized application such as the Go compiler ("gc").
While the various packages in the compiler source tree have unit tests, and one
can use "go test" to obtain coverage data for those tests, the unit tests by
themselves only exercise a small fraction of the code paths in the compiler that
one would get from actually running the compiler binary itself on a large
collection of Go source files.
For such applications, one would like to build a coverage-instrumented copy of
the entire application ("gc"), then run that instrumented application over many
inputs (say, all the Go source files compiled as part of a "make.bash" run for
multiple GOARCH values), producing a collection of coverage data output files,
and finally merge together the results to produce a report or provide a
visualization.
Many folks in the Golang community have run into this problem; there are large
numbers of blog posts and other pages describing the issue, and recommending
workarounds (or providing add-on tools that help); doing a web search for
"golang integration code coverage" will turn up many pages of links.
An additional weakness in the current Go toolchain offering relates to the way
in which coverage data is presented to the user from the "go tool cover")
commands. The reports produced are "flat" and not hierarchical (e.g. a flat list of
functions, or a flat list of source files within the instrumented packages).
This way of structuring a report works well when the number of instrumented
packages is small, but becomes less attractive if there are hundreds or
thousands of source files being instrumented. For larger applications, it would make
sense to create reports with a more hierarchical structure: first a summary by module,
then package within module, then source file within package, and so on.
Finally, there are a number of long-standing problems that arise due to the use
of source-to-source rewriting used by cmd/cover and the go command, including
#23883
"cmd/go: -coverpkg=all gives different coverage value when run on a
package list vs ./..."
#23910
"cmd/go: -coverpkg packages imported by all tests, even ones that
otherwise do not use it"
#27336
"cmd/go: test coverpkg panics when defining the same flag in
multiple packages"
Most of these problems arise because of the introduction of additional imports
in the _testmain.go shim created by the Go command when carrying out a coverage
test run in combination with the "-coverpkg" option.
Proposed changes
Building for coverage
While the existing "go test" based coverage workflow will continue to be
supported, the proposal is to add coverage as a new build mode for "go build".
In the same way that users can build a race-detector instrumented executable
using "go build -race", it will be possible to build a coverage-instrumented
executable using "go build -cover".
To support this goal, the plan will be to migrate the support for coverage
instrumentation into the compiler, moving away from the source-to-source
translation approach.
Running instrumented applications
Applications are deployed and run in many different ways, ranging from very
simple (direct invocation of a single executable) to very complex (e.g. gangs of
cooperating processes involving multiple distinct executables). To allow for more
complex execution/invocation scenarios, it doesn't make sense
to try to serialize updates to a single coverage output data file during the
run, since this would require introducing synchronization or some other
mechanism to ensure mutually exclusive access.
For non-test applications built for coverage, users will instead select an
output directory as opposed to a single file; each run of the instrumented
executable will emit data files within that directory. Example:
$ go build -o myapp.exe -cover ...
$ mkdir /tmp/mycovdata
$ export GOCOVERDIR=/tmp/mycovdata
$ <run test suite, resulting in multiple invocations of myapp.exe>
$ go tool cover -html=/tmp/mycovdata
$

For coverage runs in the context of "go test", the default will continue to be
emitting a single named output file when the test is run.
File names within the output directory will be chosen at runtime so as to
minimize the possibility of collisions, e.g. possibly something to the effect of
  covdata.<metafilehash>.<processid>.<nanotimevalue>.out

When invoked for reporting, the coverage tool itself will test its input
argument to see whether it is a file or a directory; in the latter case, it will
read and process all of the files in the specified directory.
Programs that call os.Exit(), or never terminate
With the current coverage tooling, if a Go unit test invokes os.Exit() passing a
non-zero exit status, the instrumented test binary will terminate immediately
without writing an output data file. If a test invokes os.Exit() passing a zero exit
status, this will result in a panic.
For unit tests, this is perfectly acceptable-- people writing tests generally
have no incentive or need to call os.Exit, it simply would not add anything in
terms of test functionality. Real applications routinely finish by calling os.Exit,
however, including cases where a non-zero exit status is reported.
Integration test suites nearly always include tests that ensure an application
fails properly (e.g. returns with non-zero exit status) if the application
encounters an invalid input. The Go project's all.bash test suite has many of these sorts of tests,
including test cases that are expected to cause compiler or linker errors (and
to ensure that the proper error paths in the tool are covered).
To support collecting coverage data from such programs, the Go runtime will need
to be extended to detect os.Exit calls from instrumented programs and ensure (in
some form) that coverage data is written out before the program terminates.
This could be accomplished either by introducing new hooks into the os.Exit
code, or possibly by opening and mmap'ing the coverage output file earlier in
the run, then letting writes to counter variables go directly to an mmap'd
region, which would eliminated the need to close the file on exit (credit to
Austin for this idea).
To handle server programs (which in many cases run forever and may not call
exit), APIs will be provided for writing out a coverage profile under user
control, e.g. something along the lines of
  import "<someOfficialPath>/cover"

  var *coverageoutdir flag.String(...)

  func server() {
    ...
    if *coverageoutdir != "" {
        f, err := cover.OpenCoverageOutputFile(...)
        if err != nil {
            log.Fatal("...")
	   }
    }
    for {
      ...
      if <received signal to emit coverage data> {
        err := f.Emit()
        if err != nil {
            log.Fatalf("error %v emitting ...", err)
        }
      }
    }

In addition to OpenCoverageOutputFile() and Emit() as above, an Emit() function
will be provided that accepts an io.Writer (to allow coverage profiles to be
written to a network connection or pipe, in case writing to a file is not
possible).
Coverage and modules
Most modern Go programs make extensive use of dependent third-party packages;
with the advent of Go modules, we now have systems in place to explicitly
identify and track these dependencies.
When application writers add a third-party dependency, in most cases the authors
will not be interested in having that dependency's code count towards the
"percent of lines covered" metric for their application (there will definitely
be exceptions to this rule, but it should hold in most cases).
It makes sense to leverage information from the Go module system when collecting
code coverage data. Within the context of the module system, a given package feeding
into the build of an application will have one of the three following dispositions (relative to
the main module):

Contained: package is part of the module itself (not a dependency)
Dependent: package is a direct or indirect dependency of the module (appearing in go.mod)
Stdlib: package is part of the Go standard library / runtime

With this in mind, the proposal when building an application for coverage will
be to instrument every package that feeds into the build, but record the
disposition for each package (as above), then allow the user to select the
proper granularity or treatment of dependencies when viewing or reporting.
As an example, consider the Delve debugger
(a Go application). One entry in the Delve V1.8 go.mod file is:
    github.com/cosiner/argv v0.1.0

This package ("argv") has about 500 lines of Go code and a couple dozen Go
functions; Delve uses only a single exported function. For a developer trying to
generate a coverage report for Delve, it seems unlikely that they would want to
include "argv" as part of the coverage statistics (percent lines/functions executed),
given the secondary and very modest role that the dependency plays.
On the other hand, it's possible to imagine scenarios in which a specific
dependency plays an integral or important role for a given application, meaning
that a developer might want to include the package in the applications coverage
statistics.
Merging coverage data output files
As part of this work the proposal to enhance the "go tool cover" command to
provide a profile merging facility, so that collection of coverage data files
(emitted from multiple runs of an instrumented executable) can be merged into a
single summary output file. Example usage:
  $ go tool cover -merge -coveragedir=/tmp/mycovdata -o finalprofile.out
  $

The merge tool will be capable of writing files in the existing (legacy)
coverage output file format, if requested by the user.
In addition to a "merge" facility, it may also be interesting to support other
operations such as intersect and subtract (more on this later).
Differential coverage
When fixing a bug in an application, it is common practice to add a new unit
test in addition to the code change that comprises the actual fix.
When using code coverage, users may want to learn how many of the changed lines
in their code are actually covered when the new test runs.
Assuming we have a set of N coverage data output files (corresponding to those
generated when running the existing set of tests for a package) and a new
coverage data file generated from a new testpoint, it would be useful to provide
a tool to "subtract" out the coverage information from the first set from the
second file. This would leave just the set of new lines / regions that the new test causes to
be covered above and beyond what is already there.
This feature (profile subtraction) would make it much easier to write tooling
that would provide feedback to developers on whether newly written unit tests
are covering new code in the way that the developer intended.
Design details
Please see the design document for details on proposed changes to the compiler, etc.
Implementation timetable
Plan is for thanm@ to implement this in go 1.19 time frame.
Prerequisite Changes
N/A
Preliminary Results
No data available yet.

In the past I've had the need to merge coverage profiles from the same application executed in different OSs, therefore different binaries, in order to report a single coverage metric and have an OS-agnostic report.
I see no explicit reference to this use case in the design, would it be covered (pun intended)?

Merging coverage profiles produced in different GOOS/GOARCH environments: yes, this will absolutely be supported.
One of the interesting (and IMHO slightly weird) aspects of the current coverage system is that its definition of "all source code in the package" is limited by what's picked up by the build tags in effect for the "go test" run.
So it's possible to do "go test -cover" and see "100%" statement coverage on linux, even if there may be a 500-line function in a file foo_windows.go in the package (with build tags to select only for GOOS=windows) that is effectively invisible to the coverage tooling.

(#31007 is also somewhat related.)

I really like the direction this proposal would take code coverage for Go. It looks like this proposal would lay the foundation to bring Go's code coverage story to the next level. From a developer's perspective I really liked the capabilities of the EMMA tool for Java code coverage I used to use years ago. I felt I had lost something with the code coverage tools available to Go when it became my main programming language roughly ten years ago. Consider taking a look at it as prior art if you haven't already.
In particular I would like to advocate for the "Intra-line coverage" feature mentioned in the "Possible extensions" section of the proposal. That was a feature of EMMA that I got a lot of value out of when I used it (EMMA calls it fractional line coverage).
But also, the ability to measure code coverage for a whole application and gather coverage data from integration tests are two use cases my current project at work needs a solution for. This was a discussion that came up just last week, so the timeliness of this proposal is amazing.

Nice idea. This would bring the designs for coverage used by the go build tool and Blaze/Bazel into convergence. (The latter already implements coverage as a build mode, and supports whole-program coverage.) Moving the instrumentation step from a source-based transformation to a compiler pass should also simplify both go and Bazel, and may permit finer-grained coverage of short-circuit conjunctions such as a || b && c than line granularity.

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

CC @robpike

While I understand the weaknesses of the current approach as well as anyone, moving the heavy lifting into the compiler seems like a mistake to me. There are advantages to the extant decoupling, and the ease of maintenance of the tool and tweaks to the build chain.
Wouldn't it be simpler and more flexible to keep the source-to-source translator and just work on go build? The problems of compilation and coverage can be kept separate, with all the advantages of simplicity and portability that result.

Wouldn't it be simpler and more flexible to keep the source-to-source translator and just work on go build?

This is a fair question, and something I've been thinking about a good bit lately.
My initial instinct was to use an entirely compiler-based approach because it seemed to me to offer the most control/freedom in terms of the implementation, but also (to be honest) because the compiler + linker are where I have the most expertise, e.g. my "comfort zone".
I've written CLs to do things in the compiler, and this approach seems to work well overall there are definitely some headaches.
One headache is that if we move things into the main GC compiler, we also need something that will work with the other compilers (gccgo, gollvm).
A second problem is that the compiler at the moment doesn't capture all of the source position information that you need. For example, consider this function:
func addStr(x, y string) string {	// line 19
	return x + y 			// line 20
}					// line 21

Here's the AST dump from the compiler before the 'walk' phase:
before walk addStr
.   RETURN tc(1) # p.go:20:2
.   RETURN-Results
.   .   ADDSTR esc(h) string tc(1) # p.go:20:11
.   .   ADDSTR-List
.   .   .   NAME-p.x esc(no) Class:PPARAM Offset:0 string tc(1) # p.go:19:13
.   .   .   NAME-p.y esc(no) Class:PPARAM Offset:0 string tc(1) # p.go:19:16

The compiler doesn't capture any source position info for the open and close brackets on lines 19 and 21 (why bother, we don't need them for debugging), and if you look at the references to "x" and "y" on line 20, their source posisitions correspond to their definitions, not their uses.
This doesn't create any issues when reporting coverage basic metrics (e.g. percent of statements covered) but it's a real problem when generating HTML reports, since you want to be able to "paint" chunks of the code red or green (depending on whether they executed or not)-- you can't paint it correctly if you didn't capture any source position info for it.
Of course, I could change the compiler to work in a mode where it captures/records more of this info (if building with "-cover"), but the risk there is that it might slow down the "fast path" for compilation even when the extra info collection is turned off.
Given that I've already written a compiler-based implementation, I think what I am going to do now is try prototyping a source-to-source based alternative as well (but moving to emitting the new output format). That might be a better middle ground.

Is there any synergy between this proposal and the new go test -fuzz feature? Could fuzz testing reuse the coverage tooling from this proposal to any benefit?

Is there any synergy between this proposal and the new go test -fuzz feature? Could fuzz testing reuse the coverage tooling from this proposal to any benefit?

This is a reasonable question (I might add that this also came up previously during an internal design review).
Although coverage testing and fuzzing both incorporate some notion of "coverage data", the two things are sufficiently different in terms of how they use the data means that it probably isn't worth trying to share implementations.
In both cases the compiler (or tool) is adding instrumentation code (counter increment or modification) on control flow edges, but beyond that things diverge in a big way.
For fuzzing there is no need to capture source position information for counters at all (there would be no point), and the values of the coverage counters are are used only internally / on-line within the test, never written out or stored. The only thing the fuzzer wants to know is whether coverage "changed" as a result of a mutation (AIUI).
For coverage testing on the other hand, source position info is critical (it's arguably the key piece), and unlike fuzzing we don't just want to determine that coverage has changed, we need to be able to write it out and report on it.
So with that in mind, my guess is that there isn't going to be a lot of useful overlap.  Which I think is probably OK -- the existing fuzzer support for coverage instrumentation is pretty simple, e.g.
https://go.googlesource.com/go/+/f1dce319ffd9d3663f522141abfb9c1ec9d92e04/src/cmd/compile/internal/walk/order.go#444

problem:

Lets say you have a codebase and the whole test suite takes a considerable time to run.
And you need to make a change to a particular section of the codebase(maybe one function).
What I always want to know is; which particular testcase/s covers the function that I'm about to modify?
That way, once I'm done modifying that function: I can run just the particular testcase/s that cover that function without having to run the whole test suite. Of course the whole test suite would still be ran in the builders/CI, but on my laptop I just want to run the subset of tests that cover the function I was working on.

The above is a situation I always find myself in and something I had hoped go tool cover would help with.
I don't know if this proposal will make it possible to answer the question; what testcase/s cover func Bar?,
but I'm sharing my usecase just in case.

The discussion seems to have trailed off. Is there anything left to discuss? Does anyone object to this plan?

Rob's point about compiler complexity/portability is valid, and Than's "middle ground"---retaining source-to-source translation but standardizing the build and runtime interfaces of coverage---sounds like a good compromise.

I am not sure we want to end up in a world where there are two different coverage mechanisms we have to maintain, so I am not sure about the middle ground of keeping both - what would use the old source-to-source translator, and why would we maintain it?
LLVM and gccgo do not support cmd/cover right now; they just use the coverage built in to those compilers. (It might be nice to have some kind of adapter to generate the files that cmd/cover builds nice HTML displays from though.)
Is there anything more we need to know from the implementation side in order to decide here? That is, do we want to wait for any more CLs?

My understanding of Than's "middle ground" approach is that he intended to proceed with the changes to the run-time interface (the Go packages used within the target process, and the contract between those packages and the build tool regarding where files are written), but to back away from compiler-based instrumentation and keep the existing source-to-source translation algorithm.

@thanm, what do you think the status of this proposal is? Are there any details in flux that we still need to work out?
And can you confirm @adonovan's comment that the plan is to stick with source-to-source and not do compiler instrumentation?

what do you think the status of this proposal is? Are there any details in flux that we still need to work out?

hi, thanks for the "ping".
In terms of the design, I think things are mostly settled overall. There are a couple smallish items relating to command line API that need hashing out; I am in the process of adding more details in these areas to the design doc, and when I am done I will post an update here on the issue.
In terms of the implementation, I have "go build -cover" working and all.bash passing with the new stuff turned on (new design does everything that the old one did).  I have not actually landed any of my CL stack yet however.
The test "origin" feature and the intra-line coverage featureare not yet implemented; I think at this point (given that the release freeze in in four weeks) I'll need to postpone features until after Go 1.19.

can you confirm @adonovan's comment that the plan is to stick with source-to-source and not do compiler instrumentation?

Confirmed, there will still be source-to-source rewriting, then there will be a small amount of additional "special sauce" done by the compiler when building the rewritten source. Doing things this way (IMO) provides the best overall solution.
Thanks.

@thanm Please make sure that the source-to-source translator can still be used by other compiler suites.

@thanm Please make sure that the source-to-source translator can still be used by other compiler suites.

Yes, that is definitely "in plan".

FYI: progress on my implementation CLs is moving along, but at this point in the game (less than 1 week to go before release freeze) it seems unlikely that this feature will be included in 1.19-- wanted to share that with folks who are following along on this issue.  Slipping to Go 1.20 is probably a good thing in the long run; will allow for more thorough review of the trickier parts of the implementation.
I've also posted an update to the detailed design to bring it into alignment with my current implementation (this is primarily to reflect the fact that the new scheme will use a mix of source-to-source rewriting and compiler support).
I've also tagged my implementation CLs with this issue (now that they have stabilized for the most part).

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

Would this new design/revamp also support coverage of programs that re-execute themselves? On Linux, applications and packages that deal with Limux namespaces to some extend does re-execution in order to switch into namespaces that require the process to be single-threaded at the time of switching. Docker does re-execution, podman I've seen, other container engines too...
In the past, I've developed a workaround https://github.com/thediveo/gons#gonsreexectesting that wraps a testing.M in order to write multiple cov files for re-execution and then aggregatint it. I would be a happy gopher to be able to retire this ugly hack!

@thediveo yes you should be able to do away with these sorts of wrappers entirely.
The new usage model will be
  $ go build -cover -o myapp.exe ...
  $ mkdir /path/to/coverageprofiledir
  $ GOCOVERDIR=/path/to/coverageprofiledir ./myapp.exe ...
  <fork, re-exec to whatever degree wanted>
  $ go tool covdata textfmt -i=/path/to/coverageprofiledir -o output.cov.txt
  $

The GOCOVERDIR=... "run" step above can be a simple execution, or a script that does repeated runs of the application, or whatever other scenario you need as part of your integration test.

The error messages in https://go.dev/cl/432757 use still the old "clunky" function names.

@thediveo Thanks, I'll fix that up.

On the other hand, it's possible to imagine scenarios in which a specific
dependency plays an integral or important role for a given application, meaning
that a developer might want to include the package in the applications coverage
statistics.

FYI: I'm exactly waiting for this option. My use-case is simply. I develop a version Go library that, as you may expect, relays on build process. So I have a bunch of e2e tests where I have just example "programs" that are using this lib and are built in different ways. I would be happy to get a clear way to see what lib functionally is already well tested. Currently, I'm a bit blind here.
Hope that this will be shipped too 🤞

Great proposal, this is definitely a big step forward for the go coverage toolchain, looking forwarding to it being release ready.
In addition, i have a question that does the new solution support get coverage data remotely?
The problem is that in some complex scenario like system or e2e testing, the main entity under test is usually a complete remote cluster composed by a lot of applications running in many different machines or containers. so, it will be convenient for users to obtain the coverage data of remote services locally.
For the current (or old) go's coverage ability, this didn't seem to be supported, so we built a workaround named goc to address it. While, based on my understanding this proposal should solve this problem natively, but after going through the latest code, i didn't find such ways. please help.

From the existing description I would say that you might run a bunch of tests and upload the coverage data directory. But I expect this to be very basic so you'll have to add your more specific handling of the generated data anyway.

Hi, I just tried this in Go 1.20rc1 and it looks really great. It really improves a lot of things.
I have a questions though. How can we merge coverage data generated by integration tests and coverage data generated by unit tests? So far, I haven't found a way to tell go test to output coverage data in the new format. I can surely convert the integration tests coverage into the old format using go1.20rc1 tool covdata textfmt ... and then merge (manually) the two coverage reports together.
I also see cases where someone would want to run unit tests on multiple platforms and simply merge the coverage to get a single coverage report. Right now it doesn't seems to be natively possible?
Thanks!

@JeanChristopheMorinPerso this is a feature that didn't quite make it into the release. It is on my short list of follow-on features to add, however.  I have a CL 456595 with a prototype implementation, but it didn't make it in before the freeze.
There is one hack/workaround you can use in the mean time, which is to build a test executable and then run it with an arg to pass in the GOCOVERDIR value. Example:
$ rm -f t.exe
$ go test -cover -c -o t.exe .
$ mkdir somedata
$ ./t.exe -test.count=1 -test.gocoverdir=somedata
...
$ ls somedata
covcounters.5e4ee7d09d99e0afcf93eafc6af2dcfd.2577901.1670614336696882176
covmeta.5e4ee7d09d99e0afcf93eafc6af2dcfd
$ go tool covdata percent -i=somedata
	cov.example/p	coverage: 63.8% of statements
$

At the moment -test.gocoverdir is an unexpected test flag, but in theory you can hijack it for this purpose.

Great, thanks @thanm . I look like -test.gocoverdir does excatly what I was looking for!
go1.20rc1 test -cover ./ -args -test.gocoverdir=mydir

I'm really exited about these improvements. Thanks a lot @thanm and also great work!

Hello all,
I would like to raise a discussion about a potential issue on how the go tool covdata displays the coverage coming from the main file:
For the sake of this discussion, I created a reproducer repository also hosting a CI execution that displays my point.
It seems that, unlike all the other files which get prefixed by their respective package/module path, the main.go  file enters the report prefixed with the absolute path to it:
here you can see that after merging the results of unit-tests coverage reports and binary coverage reports using go tool covdata you get something like this:
$ cat coverage/merged/coverage_percent.out
mode: set
/usr/src/myapp/main.go:10.13,12.2 1 1
github.com/rbroggi/binarycov/greetings/greetings.go:3.23,5.2 1 1

the first line of this report breaks tools like gocover-cobertura which is the default way to allow repositories hosted on gitlab to feature MR coverage visualizations.
Was there a reason why prefixing main.go with the absolute path instead of the package path like all the other files?

Thanks for the feedback. Just as a meta-remark, it might make sense to have this conversation in the context of another (new) issue, since this issue is really more about the decision to extend the coverage tooling as opposed to the nitty gritty on how the tools work.
What's happening here has to do with how the Go command classifies packages; you can actually see similar behavior in 1.19 prior to the changes in this proposal. Here is an example:
$ cat go.mod
module repro.test
go 1.19
$ cat main/main.go
package main
func main() {
	println("himom")
}
$ cat main/main_test.go
package main

import "testing"

func TestMumble(t *testing.T) {
	if 42 != 42 {
		t.Errorf("bad")
	}
}
$ go test -coverprofile=firstcover.txt repro.test/main
ok  	repro.test/main	0.014s	coverage: 0.0% of statements
$ cat firstcover.txt
mode: set
repro.test/main/main.go:3.13,5.2 1 0
$ cd main
$ go test -coverprofile=secondcover.txt ./main.go ./main_test.go
ok  	command-line-arguments	0.014s	coverage: 0.0% of statements
$ cat secondcover.txt
mode: set
/tmp/repro/main/main.go:3.13,5.2 1 0
$

The convention is that for "local" imports of this sort (e.g. where the package is selected via a relative path) we use an absolute path for the filename so as to make HTML reporting easier -- see this code in the Go command.
This is what's triggering the behavior you are seeing in your example. To work around it, try changing your Makefile from
build_coverage: coverage_dir
	@go build -cover  -v -o $(BINARY_NAME) .

to
build_coverage: coverage_dir
	@go build -cover  -v -o $(BINARY_NAME) github.com/rbroggi/binarycov

That should give you a regular import path instead of a local full path.
There are arguments to be made both ways in terms of whether the Go command should be doing this, but for the moment it's the convention.

Dear @thanm thank you a lot for the spot-on observation. I feared so indeed that there was a good reason for that to happen. As far as I'm concerned, the fact that we can achieve both output formats depending on how the inputs are provided is satisfactory and indeed it solves the issue I was facing.

At the moment -test.gocoverdir is an unexported test flag, but in theory you can hijack it for this purpose.

Any news on making -test.gocoverdir more prominently visible? As it is, I use it to run my unit tests once as root and once as non-root and as of Go 1.20 I don't need an external merging tool anymore, as the Go toolchain now supports this out of the box. At least with the unexported flag.
What's in for -test.gocoverdir=  in the near future?

What's in for -test.gocoverdir= in the near future?

I will be working on this during the 1.21 timeframe (this sort of change is not the kind of thing we back-port to minor releases, so no changes planned for 1.20.X).

I'm wondering if these docs should be extended to at least briefly mention runtime/coverage functions, which are quite handy for long running binaries. I only discovered these functions by introspecting the commit history here. I got it working without these by killing my process and collecting the profiles (which was fun since I was running these as PID 1), but I suppose this programmatic way is a bit more ergonomic.

@kokes I agree, this makes sense.  This is something that I've considered but hasn't made it to the top of my "to do" list (too many other pots on the stove).

I have a question/suggestion for https://go.dev/doc/build-cover#panicprof .
I want to collect coverage data of a server application. And I want coverage data be written without terminating the server process so that I can determine what request should I try for not yet covered statements.
Is something like below possible (are coverage write functions goroutine safe)?
package main

import (
	"net/http"
	"os"
	"runtime/coverage"
)

func main() {
	if dir := os.Getenv(`GOCOVERDIR`); dir != `` && coverage.WriteMetaDir(dir) == nil {
		http.HandleFunc(`/save-coverage`, func(w http.ResponseWriter, r *http.Request) {
			if err := coverage.WriteCountersDir(dir); err != nil {
				w.WriteHeader(http.StatusInternalServerError)
				w.Write([]byte(err.Error()))
			} else {
				w.WriteHeader(http.StatusNoContent)
			}
		})
	}
	http.ListenAndServe(`<IP_ADDRESS>:1080`, nil)
}

If yes, please consider appending something like "But you can save profile data using coverage.WriteMetaDir() and coverage.WriteCountersDir() as needed, in order to minimize amount of profile data lost by crash." after "profile data from statements executed during the run will be lost." .

are coverage write functions goroutine safe

Not sure quite what you mean by "goroutine safe" in this context.
Your code looks as though it ought to run in theory, have you run it and tried it out? By the way, the coverage.WriteMetaDir() call in your code is unnecessary; if a program is built with "-cover" and GOCOVERDIR is set, meta-data will be written on program startup.

please consider appending something like

I'm not sure that is really a good idea; most applications just don't panic or crash often enough to justify that sort of extra hackery and additional overhead.

Oops, I missed "Programs that call os.Exit(), or never terminate" already answered my question.

To handle server programs (which in many cases run forever and may not call exit), APIs will be provided for writing out a coverage profile under user control

Anyway, replying to your comment.

Not sure quite what you mean by "goroutine safe" in this context.

src/runtime/coverage/emit.go says

This file contains functions that support the writing of data files emitted at the end of code coverage testing runs, from instrumented executables.

which might imply that it is not safe to call functions in emit.go outside of process termination event. But WriteCountersDir() says

The counter data written will be a snapshot taken at the point of the call.

which might imply that it is safe to call WriteCountersDir() outside of process termination event.

Your code looks as though it ought to run in theory, have you run it and tried it out?

I wanted to know if we are allowed to call WriteCountersDir() outside of process termination event.

By the way, the coverage.WriteMetaDir() call in your code is unnecessary; if a program is built with "-cover" and GOCOVERDIR is set, meta-data will be written on program startup.

OK, I see.

I'm not sure that is really a good idea; most applications just don't panic or crash often enough to justify that sort of extra hackery and additional overhead.

My case was

Can I collect coverage data of a server application which won't terminate?

and the answer is

Yes, call coverage.WriteCountersDir(os.Getenv(GOCOVERDIR)) to collect a snapshot of coverage data.

.
Thank you.

I wanted to know if we are allowed to call WriteCountersDir() outside of process termination event.

Yes, this is allowed/supported.

Was this proposal meant to be left open? As far as I can remember, the original objectives landed in https://go.dev/doc/go1.20#cover, and the last related work mentioned here was for 1.21.

I'm ok with closing it out, just as long as folks can still add comments. There has been an unpleasantly long tail of coverage-related bugs due to fallout from the redesign, so I think it's useful to have this issue as an avenue of discussion still.

Let's close as to not confuse users thinking open == unimplemented.
Having not seen any comments here for the past year, I think most bugs have been filed as new issues.

I think documentation should still be updated.


Handling of Commander Packets

This is a discussion to decide on how to properly handle packets on ports CRTP_PORT_SETPOINT and CRTP_PORT_SETPOINT_GENERIC. Messages to those ports are real-time commands (such es desired roll/pitch angles). As such, they should not be repeated even if they don't make it to the CF. Ideally, we would also only keep a queue of size 1, i.e., we should never send "old" commands, just in order to flush out our queue. Options:

Add a new single-CF address that works similar to the current broadcasts. This requires a (NRF) firmware change, a small change here, and minor changes for users of crazyflie-link-cpp. We could also add an option in this library to keep the queue size to 1 and to give all broadcasts highest priority.
Add a flag in the Packet data structure to disable retry, i.e., we don't need any firmware changes. The downside is that we still use safelink in that case, which could reject a packet of host and CF are out-of-sync. To keep the queue size to 1, we would need special handling for the two ports mentioned above.

Test case: a client application that sends setpoints at a very high rate (e.g., 1 kHz), should still be able to fly the Crazyflie well.
@ataffanel: It would be nice to get your thoughts on this.

hi! Arnaud will be gone for a while so I'm looking through the isues from the repo. I still need to get more familiar with the code.
It is interesting though. So just to send setpoints to the crazyflie without expecting an ack. So a broadcast with an CF2 address attached.
But I guess we still need to have the occasional ping from the crazyflie in question, just to check if it is still receiving or not? I guess this ping needs to be a seperate package.

Yep, exactly! It is true that the ping is still needed. This is currently done automatically in this library, whenever no other packet is in the queue.

I accidentally pushed a change for option 2 already in 1f9d89362853adb99d682ca717176f0c05524d77 without a PR/review. Sorry about that. Post-review would be welcome, or I can revert the change and open a proper PR.

Ah woops 😄
that is okay, it seemed to be all fine. So let's keep it like this for now.
The only thing is that if we want to do a release, the github automatic changes list will not note this in particular, so that why PRs are usually better, just to let you know for next time.


Kryo class resistration through reflection does not take in consideration Interface fields or Parameterized type field

It is useless to register interfaces for Kryo, because it is the concrete class that needs to be serialized to transfer over network. Also, there is no easy way to know all specific implementation of a class with a parameterized type. Therefore, the automatic registration of classes for Kyro fails to identify some cases with some dtos. As such interfaces and generics type wont have their resolved type registered automatically.
If we really wish to use 100 % automatic registration, we should find a solution for that.
It is not a big blocker because if we know the resolved type in those case, we can register those classes manually

As a reference, see Register and MarvelousBobProperties classes in the common module, for contextual code.

A possible solution could be to add an annotation to interface fields of the dots, something like @Implementations(classes = {ArrayList.class, LinkedList.class}) or @Register({ArrayList.class, LinkedList.Class}) to specify right at the problematic field location, which implementations of the Interface, or generic field, needs to be Register for that field. It then becomes esay to check for that annotation during registration process and also register classes specified in the annotation argument.


Add annapurna workflows and dependencies (from fork)

It would be nice to check in this PR:

[x] All actions has their name correctly set in action.yaml and it is the same as the directory
[x] All actions and arguments have descriptions
[x] All shell actions use ::group:: at the beginning and ::end group:: at the end
[x] There are no mix actions (using shell and other actions) or they are well documented
[x] No relative paths are used along the arguments

BEFORE MERGE

[x] Change all action links jparisu/eProsima-CI/...@main to eProsima/eProsima-CI/...@main

All actions references changed in commit https://github.com/eProsima/eProsima-CI/pull/17/commits/bd7451d2740262b69077e105f6049185782d233d


AccessToken is null

I have created a new Mobile App and enabled the authentication option for Azure AD. When I use the GetAppServiceIdentityAsync to get the AzureActiveDirectoryCredentials the AccessToken is null.
How do I get the AccessToken, I need this value to request a new AccessToken for a different url?
My old App Service used a gateway where it was possible to get the AccessToken.

This looks like a bug. It's a pretty easy fix; i should have a PR out shortly.
The problem is that we're copying the 'access_token' value of the response to the token store, which is null. Instead, we should be looking at 'id_token' for AAD.
If you need a workaround before we get this released, here's one way you can get this working right now -- this intercepts the response from the token store and copies 'id_token' over to 'access_token':
First, create a DelegatingHandler to copy over the value:
public class AadAccessTokenMessageHanlder : DelegatingHandler
{
    public AadAccessTokenMessageHanlder(HttpMessageHandler innerHandler) :
        base(innerHandler)
    {
    }

    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        HttpResponseMessage response = await base.SendAsync(request, cancellationToken);
        JObject result = await response.Content.ReadAsAsync<JObject>();
        if (result.HasValues && result["id_token"] != null && result["access_token"] == null)
        {
            result["access_token"] = result["id_token"];
            response.Content = new StringContent(result.ToString(), Encoding.UTF8, "application/json");
        }
        return response;
    }
}

Then create your own HttpClient and pass it to GetAppServiceIdentityAsync:
HttpClient httpClient = new HttpClient(new AadAccessTokenMessageHanlder(new HttpClientHandler()));
AzureActiveDirectoryCredentials aadCreds = await this.User.GetAppServiceIdentityAsync<AzureActiveDirectoryCredentials>(this.Request, httpClient);

I just upgraded my NuGet packages from version <IP_ADDRESS> to <IP_ADDRESS> and the problem is back.

Were you using the workaround above?

No, I wasn't. I have to use a reverse workaround:
if (result.HasValues && result["id_token"] == null && result["access_token"] != null)         {             **result["id_token"] = result["access_token"];**             response.Content = new StringContent(result.ToString(), Encoding.UTF8, "application/json");         }

Okay thanks -- I'm going to re-activate this issue until I can figure out what's going on.

If you want me to send you the json I receive, just let me know,

Yeah, could you? That would help.
When I call var creds = await this.User.GetAppServiceIdentityAsync<AzureActiveDirectoryCredentials>(this.Request);, my creds has a valid AccessToken.
When I capture the raw JSON coming back from the request to .auth/me, I see: {"id_token":"eyJ0...","provider_name":"aad","user_claims":[...],"user_id":"..."}, which is what I would expect.
In case you aren't sure how (or for others stumbling onto this issue) -- you can issue a request to https://{yoursite}.azurewebsites.net/.auth/me and attach your X-ZUMO-AUTH token to see the data directly.

@mlankamp - this is not happening for me with the latest SDKs.  Are you still having an issue?
Ref: to close 12/1/2016

CLosing per 1-month closure above.


problem building OpenBlas with CMAke GUI and MS Visual studion 2017

Hello
I have used CMAKE 3.14.x GUI to configure and build OpenBlas with MS Visual Studio 2017
It generates a workspace and then I can compile all of the libraries
when building the openblas_utest project I get two linking errors
2>test_potrs.obj : error LNK2019: unresolved external symbol zpotrs_ referenced in function _ctest_potrf_bug_695_run
2>openblas.lib(lapacke_dgesvd_work.obj) : error LNK2019: unresolved external symbol dgesvd referenced in function LAPACKE_dgesvd_work
2>C:\software\openblas\build\utest\openblas_utest.exe : fatal error LNK1120: 2 unresolved externals
2>Done building project "openblas_utest.vcxproj" -- FAILED.
Digging a little bit into it I see that the workspace does not build the fortran files , even if intel fortran is installed in my computer.
Any help is welcome
best
jac

Did you specify any options in the cmake configure step ? (I must admit I am not familiar with the windows cmake gui, there are some instructions floating around that suggest -DNOFORTRAN=1 which obviously would not be what you want but if you cut&pasted from somewhere they could have slipped in). Perhaps try setting CMAKE_Fortran_COMPILER to ifc instead of letting cmake autodetect compilers.

Perhaps try setting CMAKE_Fortran_COMPILER to ifc

Actually a poor suggestion, should have written ifort - "ifc" on Windows appears to be an unrelated component of VS on Windows (while on Linux it used to be a synonym for ifort). As far as I can tell https://gitlab.kitware.com/cmake/cmake/issues/17752  was fixed in cmake 3.12 though

Hi
this resolved the problem. The CMakelists does not include the path to the fortran compiler nor it includes a function to detect it.
I just put the following entry
CMAKE_fortran_Compiler=C:/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/bin/intel64/ifort.exe
which is the path to the fortran compiler in my computer
best
jac

Thanks for the feedback. In theory, CMAKE_Fortran_Compiler should have been figured out by cmake itself, possibly this is not done right. (cmake/f_check.cmake has enable_language(Fortran) but maybe there should be an explicit check_language(Fortran) before it and the current code only works "by accident" when FC happens to be predefined or something...)

Is it cmake-gui.exe ?
Could you show options you selected?

As far as I can tell, cmake relies on predefined environment variables to select the compilers. Hardcoding a path in CMakelists.txt clearly makes no sense when it may vary from one installation to another, but the CMake GUI should be offering you an option to specify the system compilers (on the second screen, after hitting Configure )

Hello
I have now another issue. CMAKE generates a number of workspaces. One of them is openblas, which includes a mix of fortran and c files. MSVC does not allow this mixing and therefore does not compile the fortran files. Then during linking the lapack files are missing (as they are in fortran).
In order to resolve the problem, there is a need to extract the fortran files and compile them into a separate lib, and we must provide the way to enable calling the routines from c.
another way would be to use the clapack the c version of lapack.
any idea and or suggestion ?
thanks
jac

You need a fortran compiler to compile LAPACK. You can find outdated binaries around the web if compiler does not serve you.

I have the intel visual fortran and it is recognized and tested by cmake
but still during the build of the openblas lib the fortran files are not compiled
jac

I have not tested this combination yet. Maybe something in this thread helps:
https://discourse.cmake.org/t/how-to-set-up-compilation-with-intel-fortran-compiler-on-windows-correctly/532/8

Reproduced the problem now (with the Ninja setup described in the Wiki), though I do see the fortran files getting compiled with ifort, it is "only" at link time that the symbols cannot be found. Strangely, dumpbin appears to confirm they are present as External in the library, possibly there is something wrong with f_check.cmake setting BUNDERSCORE to expect one trailing underscore on the symbol name.

hello
back to this thanks
will try now

Hello
I tried and failed. The lapack fortran is not compiled and therefore the lib is not complete.
I tried building with conda3 and while the process is obscure it worked.
I was able to create a static library in debug and release but not a dll
is it possible to create a dll this way ?
thanks
jac

It should be possible to create a dll by setting -DBUILD_SHARED_LIBS=ON in cmake.

Hi
I tried that but I got an error claiming that the script was not build for dll

Could you elaborate with accurate error text/messagebox ?

Hi
I shall try rebuilding again and fetch the error message

You can rerun last build command that produces the error, no need to go through full build.

Hello
tried to build the dll and I got the following error message
ninja: error: '/FORCE:MULTIPLE', needed by 'lib/openblas.dll', missing and no known rule to make it

Are you trying to build with ReLAPACK included ? This is the only case I can think of where we use target_link_libraries(${OpenBLAS_LIBNAME} "/FORCE:MULTIPLE") as ReLAPACK overloads some generic LAPACK functions.

yes I have set to build ReLAPACK
will try now
best
jac

Hello ,
I am getting below problem in theano. I am working with python3.6 on CentOS in SSH.

hi tried
did not work

actually it is working (after I deleted the content of the build directory
thanks
jac

@jbensabat what did you try? Building without ReLAPACK, or changing the line in CMakeLists.txt ?
@preksha12 please do not just dump totally unrelated questions into a random topic. All I can tell from that screenshot is that your theano installation cannot find a BLAS module, please review the installation instructions for theano and ask there.

I tried without RELAPACK and after cleaning the build directory
jac


Always says [no files]

Hey, I'm quite new to Grunt, I'm not sure why this isn't working. It seems pretty obviously looking in the correct spot yet it can't find the file I want to obfuscate.
Here is the output
Running "jsObfuscate:files" (jsObfuscate) task
Verifying property jsObfuscate.files exists in config...OK
File: [no files]
Options: concurrency=2, keepLinefeeds=false, keepIndentations=false, encodeStrings, encodeNumbers, moveStrings, replaceNames, variableExclusions=["^_get_","^_set_","^_mtd_"]

Yet, if I run uglify it finds the files fine:
    ...

    uglify: {
      dist: {
        files: {
          '<%= yeoman.dist %>/public/test.js': [
            'public/test.js'
          ]
        }
      }
    },

    jsObfuscate: {
      options: {
        concurrency: 2,
        keepLinefeeds: false,
        keepIndentations: false,
        encodeStrings: true,
        encodeNumbers: true,
        moveStrings: true,
        replaceNames: true,
        variableExclusions: ['^_get_', '^_set_', '^_mtd_']
      },
      files: {
        '<%= yeoman.dist %>/public/test.js': [
          'public/test.js'
        ]
      }
    },

    ...

Here's where I call them
//  'uglify', // always uglify test.js before passing into obfuscator serice
    'jsObfuscate',

Currently if I uncomment uglify and re-comment obfuscate, uglify is properly minifying the files, so I know for a fact the file exists where files: key is pointing to... but uglify it can see the file and obfuscate can not. I'm quite confused... thanks!

argh, its because i needed an extra 'key' inside the {}. It originally had test: {} but I delete that when I saw another example in an issue in this repo.


[Bug] openwrt tun mixed stack unable to route local traffic.

Verify steps

[X] I have read the documentation and understand the meaning of all configuration items I have written, avoiding a large number of seemingly useful options or default values.
[ ] I have not reviewed the documentation and resolve this issue.
[ ] I have not searched the Issue Tracker  for the problem I am going to raise.
[X] I have tested with the latest Alpha branch version, and the issue still persists.
[X] I have provided server and client configuration files and processes that can reproduce the issue locally, rather than a desensitized complex client configuration file.
[X] I have provided the simplest configuration that can reproduce the error I reported, rather than relying on remote servers, TUN, graphical client interfaces, or other closed-source software.
[X] I have provided complete configuration files and logs, rather than providing only parts that I believe are useful due to confidence in my own intelligence.

Operating System
Linux
System Version
Openwrt 23.05.3
Mihomo Version
Mihomo Meta alpha-619f341 linux arm64 with go1.22.2 Sat May 11 16:12:32 UTC 2024
Use tags: with_gvisor
Configuration File
allow-lan: true
bind-address: "*"
mode: rule
log-level: warning
find-process-mode: 'off'
unified-delay: true
keep-alive-interval: 1800

external-controller: <IP_ADDRESS>:9090
external-ui: ./ui

dns:
  enable: true
  listen: <IP_ADDRESS>:1053
  enhanced-mode: redir-host
  fake-ip-range: <IP_ADDRESS>/16
  nameserver:
    # - <IP_ADDRESS>
    - "https://<IP_ADDRESS>/dns-query"
  default-nameserver:
    # - system
    - <IP_ADDRESS>
  proxy-server-nameserver:
    # - system
    # - <IP_ADDRESS>
    - <IP_ADDRESS>
proxies: []

proxy-groups:
  - name: 🌝Proxy
    type: select
    use:
      - myproxies

proxy-providers:
  myproxies:
    type: file
    path: ./proxy_provider/myproxies.yaml
    health-check:
      enable: true
      url: http://www.gstatic.com/generate_204
      interval: 1800

rules:
  - IP-CIDR,<IP_ADDRESS>/8,DIRECT,no-resolve
  - IP-CIDR,<IP_ADDRESS>/16,DIRECT,no-resolve
  - MATCH,🌝Proxy

sniffer:
  enable: true
  force-dns-mapping: false
  parse-pure-ip: true
  override-destination: false
  sniff: # TLS 和 QUIC 默认如果不配置 ports 默认嗅探 443
    QUIC:
      ports: [443]
    TLS:
      ports: [443, 8443, 5222-5228]
    # 默认嗅探 80
    HTTP: # 需要嗅探的端口
      ports: [80, 8080-8880]
  sniffing:
    - tls
    - http
  port-whitelist:
    - 80
    - 443

tun:
  enable: true
  mtu: 9000
  device: utun
  stack: mixed
  dns-hijack:
    - <IP_ADDRESS>:53
  auto-route: true
  strict-route: false
  auto-detect-interface: true

Description
When using tun mixed stack local traffic didn't appear in connection log. It seems there's no local traffic routed to mihomo tun. Checked with curl https://ip.sb/ip it's indeed not enter mihomo.
Changed to system or gvisor there's no problem. It able to route local traffic normally.
I also tried sing-box. it has the same issue. I suspected  it's something to do with sing library. I opened the issue few weeks ago in sing project it's immediately closed without reply as expected.
Reproduction Steps
Enable tun with mixed stack.
Try to use internet within the router with ssh. Eg curl https://ip.sb/ip
Observe
Logs
No response

It works fine on my device. I suggest checking if it's an issue with the local device/system.

It works fine on my device. I suggest checking if it's an issue with the local device/system.

What to check?
I tried openclash it works but it use manual route and some services don't get routed like local dns and other net service.
ip a | grep utun
8: utun: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast state UNKNOWN group default qlen 500
    inet <IP_ADDRESS>/30 brd <IP_ADDRESS> scope global utun

@lux5am  Check the resolv.conf file to ensure it does not contain private addresses, as private addresses cannot be hijacked.

@xishang0128
I always checked ignore resolv file in DHCP setting tho since it's connected to my isp router/modem and provide DHCP with its own DNS (<IP_ADDRESS>) so indeed it can't be hijacked.

I tried with kdig @<IP_ADDRESS> or any random server it also timeout. But sometimes it works. Which is strange.

Switched to gvisor everything is fine.

The same with system.


Duplicate conditional expression in Validations.php

!is_string($options['with']) is repeated.

LGTM 👍
Thanks for the contribution!

👌🏼


Try to retrieve more details for Maven dependencies

Maven can output project dependencies in different formats, but in all formats, the information about a dependency only contains its name, though full-qualified. We should try to ask Maven for more information or at least for the resolved dependency location (normally located in the local repo somewhere under the ".m2" folder ) as in this case we could just open the jar and read the info on our own.

The mvn dependency:build-classpath command seems to give paths to the local repository where the dependencies are stored. Next to the .jar files are also the .pom files which we could read for metadata.


Support slave node auth with private key instead of username password.

Does the plugin support ssh to slave node with private key instead of username password? I went thru the code but could not find it. Am i missing something here?

It does, the exact credential form is abstracted away using ssh-slaves plugin. To use private key authentication you need to configure the key in credentials configuration and instruct openstack plugin to use that credential.


Explicitly encode data passed to securesystemslib.keys

Note: this change breaks compatibility with ssl v0.11.3 and thus should not be merged until a new release of ssl has been made and is available on PyPI.
PR https://github.com/secure-systems-lab/securesystemslib/pull/162 removed implicit encoding of data to bytes in securesystemslib.keys.[create_signature|verify_signature]
Fixes issue #: #913
Description of the changes being introduced by the pull request: explicitly encode data we pass to securesystemslib.keys.create_signature and securesystemslib.keys.verify_signature
Please verify and check that the pull request fulfills the following
requirements:

[x] The code follows the Code Style Guidelines
[ ] Tests have been added for the bug fix or new feature
[ ] Docs have been added for the bug fix or new feature

Will be merged with #934.


Notebook does not open as Json, when explicitly asked

Bug: Notebook Editor, Interactive Window, Editor cells
When notebook is opened with Text editor (idea - as a json), it shows for a second, then switches to notebook view.
Steps to cause the bug to occur

have some existing notebook
click open with via right button click in Vscode file navigator
select "text editor"

Actual behavior
json shows for a second (if the notebook  is not opened already), then switch to notebook view
Expected behavior
shows notebook file as a json
Your Jupyter and/or Python environment
Please provide as much info as you readily know

Jupyter server running: Local
Extension version: 2020.9.111407
VS Code version: 1.49.2
Setting python.jediEnabled: false
Setting python.languageServer: PyLance
Python and/or Anaconda version: 3.7
OS: MacOs Catalina
Virtual environment: conda

Python Output
None output
Microsoft Data Science for VS Code Engineering Team: @rchiodo, @IanMatthewHuff, @DavidKutu, @DonJayamanne, @greazer, @joyceerhl

Hey @Casyfill, thanks for the bug report. Unfortunately I cannot reproduce this issue on macOS + local Jupyter + Anaconda 3.7 + 2020.9.111407. Would you mind trying to disable all extensions besides the Python extension and checking if the issue persists?

@Casyfill, we have a fix in the works to address this, but in the meantime, you can turn off automatically opening ipynb as a notebook by adjusting this setting:
"python.dataScience.useNotebookEditor": false

Thank you!
On Thu, Oct 1, 2020 at 4:28 PM Jim Griesmer<EMAIL_ADDRESS>wrote:

Closed #14130 https://github.com/microsoft/vscode-python/issues/14130.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/microsoft/vscode-python/issues/14130#event-3831320389,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAUNHJJ2XHGZAXD6OWD23CDSITQ6LANCNFSM4R4SXEIA
.


Updated /raspberry_video/LeptonThread to work for Lepton 3.

Edited LeptonThread to correctly assemble the four segments given by Lepton 3 into a single image. Increased SPI speed to 32 MHz. Works on Rasberry Pi 3, but SPI speed is too fast for Raspberry Pi 2 (a SPI speed of 24 MHz is best for RBPi 2, but is still jumpy). Skips entire frames when not synced, rather than rendering half the image and creating tears.
-Andrew Myers
Novacoast, Inc.

Thanks for the PR. Great to see code contributed for L3.
The clock change seems a bit dangerous to me, and I think unnecessary. The Lepton 3 datasheet (preliminary one that I have at least) calls out 20 MHz as the maximum SPI clock rate; has this changed? I think we should adhere to the published limit, or if it's higher now, at least make sure that the faster clock does not break Lepton 2 by providing a means for backward compatibility in this application.
That being said, I was able to correctly download frames in Python from the L3 (and stay in sync) with an 18 MHz clock. Rather than increasing the clock speed, I was able to lower overhead by batching a bunch of VOSPI transfers in a single read ioctl after initial sync was established. The default spidev bufsiz is 4096 bytes so that's 24 VOSPI frames (4096 / VOSPI_FRAME_SIZE_BYTES = 24.97... ) that can be downloaded in a single ioctl. This technique would be worth trying for this application.
Also, in this PR, you've broken the spi_device switch for people who don't have the Lepton on RPI CS1 (see: software/raspberrypi_video/SPI.cpp:40)

Closing to track in #33

Hi,
I used the codebase for Lepton3 with raspberry pi3, but I cannot get rid of the red square. Are there any other changes to be made?


Feature/Staking

Summary
Done Staking Feature.
Checklist

[x] I removed all commented or unnecessary code.
[ ] Provide the screenshots due to UI changed or bug fixed
[ ] My code has covered these test cases (please list down your tested cases):

a
I got the error when click confirm delegate in view mode. Expected to show no signer message

a I got the error when click confirm delegate in view mode. Expected to show no signer message

Please run script env:mainnet and env:testnet to update AUCTION_HASH variables
This bug occurred because AUCTION_HASH is undefined.

LGTM