Allowing NixOS VM's to be run on macOS

People using macOS currently can't interactively run NixOS VM's as described e.g. here, even with a remote Linux builder. In this issue I'm describing some ways in how that could be made to work. Note that speed is important as well, so kvm/hvf and co. should be used if possible.
Relevant issue is https://github.com/NixOS/nixpkgs/issues/64578. Ping @zupo @matthewbauer @nmattia @roberth. Any pointers/help with this is appreciated.
This issue is sponsored by Niteo :)
Using qemu directly (doesn't work)
With this change it's possible to build a NixOS VM run script that is executable on macOS (note that this requires a remote Linux builder)
$ nix-build '<nixpkgs/nixos>' \
  --argstr system x86_64-linux \
  --arg configuration '{ virtualisation.qemu.pkgs = import <nixpkgs> { system = "x86_64-darwin"; }; }' \
  -A vm \
  -I nixpkgs=https://github.com/infinisil/nixpkgs/archive/4d244410ee0f3e3ece5494533217bbafbd95d9b3.tar.gz
/nix/store/2a7cbyp9xp12ddc4lxb4h93dxa5yfndy-nixos-vm

However running the VM doesn't actually work:
$ result/bin/run-nixos-vm
qemu-system-x86_64: -virtfs local,path=/nix/store,security_model=none,mount_tag=store: There is no option group 'virtfs'
qemu-system-x86_64: -virtfs local,path=/nix/store,security_model=none,mount_tag=store: virtfs support is disabled

This is because qemu doesn't support virtfs on macOS.
It was suggested that this patch could be used to make it support virtfs. This was attempted here, however the build doesn't succeed:
$ nix-build https://github.com/infinisil/nixpkgs/archive/57562282ab34fc44a492f32a13939d77e29d0d9b.tar.gz -A qemu
[...]
fsdev/virtfs-proxy-helper.c:16:10: fatal error: 'sys/fsuid.h' file not found
#include <sys/fsuid.h>
         ^~~~~~~~~~~~~
1 error generated.
make: *** [/private/var/folders/l6/7v9ppmg12q90382m246y3g5c0000gn/T/nix-build-qemu-5.1.0.drv-0/qemu-5.1.0/rules.mak:69: fsdev/virtfs-proxy-helper.o] Error 1
make: *** Waiting for unfinished jobs....
builder for '/nix/store/z8mkhjvaz3r7dixv94c1jprwx8gvl0gh-qemu-5.1.0.drv' failed with exit code 2
error: build of '/nix/store/z8mkhjvaz3r7dixv94c1jprwx8gvl0gh-qemu-5.1.0.drv' failed

Attempting to remove all the missing header files from virtfs-proxy-helper.c, including <sys/fsuid.h>, <sys/vfs.h>, <linux/fs.h> and <cap-ng.h> just leads to compilation errors, seemingly indicating that this is a Linux-only functionality.
This page however mentions that:

FSDRIVER: Either "local", "proxy" or "synth". This option specifies the filesystem driver backend to use. In short: you want to use "local". In detail:

local: Simply lets QEMU call the individual VFS functions (more or less) directly on host.
proxy: this driver was supposed to dispatch the VFS functions to be called from a separate process (by virtfs-proxy-helper), however the "proxy" driver is currently not considered to be production grade.

And indeed, the run-nixos-vm script only uses local, quoting the script:
-virtfs local,path=/nix/store,security_model=none,mount_tag=store \
-virtfs local,path=$TMPDIR/xchg,security_model=none,mount_tag=xchg \
-virtfs local,path=${SHARED_DIR:-$TMPDIR/xchg},security_model=none,mount_tag=shared \

Attempting to not compile that tool with this commit however also doesn't work, failing again on seemingly Linux-specific headers:
$ nix-build https://github.com/infinisil/nixpkgs/archive/5ed493ed6ae957aa510d5292eb373b8b1f4a3db1.tar.gz -A qemu
[...]
/private/var/folders/l6/7v9ppmg12q90382m246y3g5c0000gn/T/nix-build-qemu-5.1.0.drv-0/qemu-5.1.0/fsdev/file-op-9p.h:19:10: fatal error: 'sys/vfs.h' file not found
#include <sys/vfs.h>
         ^~~~~~~~~~~
1 error generated.
make: *** [/private/var/folders/l6/7v9ppmg12q90382m246y3g5c0000gn/T/nix-build-qemu-5.1.0.drv-0/qemu-5.1.0/rules.mak:69: fsdev/qemu-fsdev.o] Error 1
builder for '/nix/store/zfnmaj3jh16x2qlryzyf3m2gy3zcww53-qemu-5.1.0.drv' failed with exit code 2
error: build of '/nix/store/zfnmaj3jh16x2qlryzyf3m2gy3zcww53-qemu-5.1.0.drv' failed

So it seems that qemu just doesn't support virtfs on macOS.
Using libvirt (might work)
Above qemu documentation also has this section, which describes how the same can be achieved with libvirt. While I believe that underneath it just uses qemu as well, there might be some additional libvirt magic happening. And it seems that macOS Mojave supports virtio-9p, in which post the author also uses libvirt successfully.
Currently the NixOS VM runner just passes arguments to qemu. In order to use libvirt we'll have to transform all these arguments to the libvirt equivalent in its XML configuration. There even is a libvirt page describing the equivalent of qemu arguments, which will be very useful. There is also the virsh domxml-from-native qemu-argv command which can supposedly do this transformation automatically, though I didn't have any success with it yet.
In above-linked document describing qemu argument equivalents, the -virtfs (or the -fsdev and -device options it's a shorthand for) option is notably missing. This should be replaced with a <filesystem> section as described in the qemu documentation link above.
It would be a good idea to first manually create a libvirt configuration and verifying that it works on a NixOS VM. libvirt has a graphical interface which could also be used.
More relevant links:

libvirt supports virtio-9p since version 6.9.0
haxm might be needed for the end result to be fast
virtio-9p can be slow. There is a faster replacement virtio-fs. However as far as I know, macOS doesn't have a driver for that.
A blogpost describing how to use libvirt on macOS to start an Ubuntu VM
The NixOS Wiki page on libvirt

Removing the need for filesystem mapping (might work)
The main reason this -virtfs argument is used at all is so that the guest machine in the VM can access the host machines /nix/store, which is where the whole system that the guest runs resides in.
An alternate approach however could be to create a /nix/store image that can be used by qemu as the /nix/store directly, therefore not depending on this filesystem mapping to the host.
See the various make-* files in https://github.com/NixOS/nixpkgs/tree/master/nixos/lib, which could be useful for this

Another benefit if we get MacOS + QEMU story working: ATM, running NixOS tests on various CI providers such as CircleCI and GitHub Actions is very slow since these services do not provide /dev/kvm/ on their Linux runners. But they do provide MacOS runners, that come with Mac's KVM alternative, so if MacOS + QEMu story is improved, even people that don't really care about MacOS, but do run CI in the cloud (looking at @domenkozar), stand to gain lots of speed improvements.

Another benefit if we get MacOS + QEMU story working: ATM, running NixOS tests on various CI providers such as CircleCI and GitHub Actions is very slow since these services do not provide /dev/kvm/ on their Linux runners. But they do provide MacOS runners, that come with Mac's KVM alternative, so if MacOS + QEMu story is improved, even people that don't really care about MacOS, but do run CI in the cloud (looking at @domenkozar), stand to gain lots of speed improvements.

I don't have a lot of time right now, but here's a basic skeleton that could be used to create a libvirt xml config for seeing if that approach is viable:
let
  pkgs = import <nixpkgs> {};
  config = (import <nixpkgs/nixos> {
    system = "x86_64-linux";
    configuration = <nixpkgs/nixos/modules/virtualisation/qemu-vm.nix>;
  }).config;
in pkgs.writeText "nixos.xml" ''
  <domain type='qemu' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
    ...
    <os>
      ...
      <kernel>${config.system.build.toplevel}/kernel</kernel>
      ...
    </os>
    ...
    <filesystem type='mount' accessmode='$security_model'>
      <source dir='/nix/store'/>
      <target dir='store'/>
    </filesystem>
    ...
  </domain>
''

I don't have a lot of time right now, but here's a basic skeleton that could be used to create a libvirt xml config for seeing if that approach is viable:
let
  pkgs = import <nixpkgs> {};
  config = (import <nixpkgs/nixos> {
    system = "x86_64-linux";
    configuration = <nixpkgs/nixos/modules/virtualisation/qemu-vm.nix>;
  }).config;
in pkgs.writeText "nixos.xml" ''
  <domain type='qemu' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
    ...
    <os>
      ...
      <kernel>${config.system.build.toplevel}/kernel</kernel>
      ...
    </os>
    ...
    <filesystem type='mount' accessmode='$security_model'>
      <source dir='/nix/store'/>
      <target dir='store'/>
    </filesystem>
    ...
  </domain>
''

Refs https://github.com/NixOS/nixpkgs/issues/5241

Refs https://github.com/NixOS/nixpkgs/issues/5241

virtio-9p can be slow. There is a faster replacement virtio-fs. However as far as I know, macOS doesn't have a driver for that.

Did you see https://passthroughpo.st/mac-os-adds-early-support-for-virtio-qemu/

virtio-9p can be slow. There is a faster replacement virtio-fs. However as far as I know, macOS doesn't have a driver for that.

Did you see https://passthroughpo.st/mac-os-adds-early-support-for-virtio-qemu/

@domenkozar Yeah, click the "uncovered by Qemu developer" link there, it leads to a blogpost I linked above, which only mentions virtio-9p, not virtio-fs. I'm now just seeing that that news article mistakenly used virtio-fs in its picture..

@domenkozar Yeah, click the "uncovered by Qemu developer" link there, it leads to a blogpost I linked above, which only mentions virtio-9p, not virtio-fs. I'm now just seeing that that news article mistakenly used virtio-fs in its picture..

Ah! That was confused me.

Ah! That was confused me.

Benchmarks show that should also speed up the NixOS tests https://matrix.org/_matrix/media/r0/download/johnguant.com/qgAVstnDkCydsABgKmOXRPGW
cc @JJJollyjim do you have a branch for this?

Benchmarks show that should also speed up the NixOS tests https://matrix.org/_matrix/media/r0/download/johnguant.com/qgAVstnDkCydsABgKmOXRPGW
cc @JJJollyjim do you have a branch for this?

The patch you linked seems to be only one of a series - presumably you need all of them for it to work. The series is visible here<EMAIL_ADDRESS>Not sure if there's a way to get the whole series from Patchwork.

Oh hey, just catching up here, I'm a little confused - the stuff where apple has added support for something is all about osx guests, not osx hosts which is what we care about here right?
I don't believe the current virtiofsd (host component) is able to run on osx (e.g. it uses namespaces to sandbox itself), but I don't think there is any technical limitation stopping it being ported?
I recommend patchew if you want a nice way to search for qemu patches and their status<EMAIL_ADDRESS>(wish the kernel had something like this... -_-).
Anyway, let me know if you find that switching to virtio-fs will make OSX support work:
I have a branch, but haven't worked on cleaning it up and posting it because for some reason the DAX patches still haven't been submitted to qemu (last I checked), and without DAX the tiny performance improvement isn't worth the complexity. That calculus changes if it will fix OSX though :)

On second thoughts, I patch out the namespace stuff (so it works in the nix sandbox), so it's possible it will run on osx without any further changes. I don't have an osx machine to test on :(

I have a macOS machine I'm happy to test stuff on if it'd b helpful for oyu.

Oh hey, just catching up here, I'm a little confused - the stuff where apple has added support for something is all about osx guests, not osx hosts which is what we care about here right?

Not quite sure what you're referring to, but here's a quick run-down of Apple's recent-ish features for macOS hosts:

Hypervisor.framework: 3-5 years old. Provides low-level userspace access to the CPU's virtualization features. I think QEMU has a backend for this, called HVF.
Virtualization.framework: New with Big Sur, this year's release. Much higher-level - you give it a kernel, initrd, and disk, and it goes from there. Pretty limited - only emulates a few devices (the virtio devices for disk, network, serial, RNG and memory balloons), and only supports raw images (though APFS's sparse files negate the need for qcow2 a bit). vftool provides a thing command-line wrapper over this.

(As a side note, I've been experimenting with using Virtualization.framework to implement a simple Linux VM, using a tiny linux kernel with u-root as a simple "bootloader" to find and exec the real kernel image off a Linux filesystem; I've got it working with Ubuntu, but not NixOS; I suspect there's some issue with virtio-console support on the install ISO.)

@Gaelan I was referring to Domen's Passthru Post link about virtio :)

Ah, my bad.

I ported the original 9p patchset (mentioned earlier in this thread) to QEMU 5.2.0. In initial tests, 9p support on Darwin appears to be working.
But this is a large change that should probably go to upstream and not to Nixpkgs. If you want to try anyways: https://github.com/mroi/nixpkgs/commit/839559b486d4c4dbd911a0f13a3b2d319b942cd5

@mroi Would be help to at least submit the patch to Homebrew. Thanks for the work! I'll look on giving it a try.

I would really love the test driver to run on MacOS with HAXM enabled. (-enable-hax)
I tried to merge both patches from @mroi and @Infinisil and ran the same command as above :
$ nix-build '<nixpkgs/nixos>' \
            --argstr system x86_64-linux \
            --arg configuration '{ virtualisation.qemu.pkgs = import <nixpkgs> { system = "x86_64-darwin"; };
}' \
            -A vm \
            -I nixpkgs=http://github.com/SCOTT-HAMILTON/NixPkgs/archive/230f823cd985e499ff2fd450b419bc5b44c6db
f8.tar.gz

But I get this error :
error: a 'x86_64-linux' with features {} is required to build '/nix/store/qbr4wjfz9lzrqha4gl6mjp2l0imwqdgh-append-initrd-secrets.drv', but I am a 'x86_64-darwin' with features {benchmark, big-parallel, nixos-test}

For testing patches you'll need a remote builder to build the Linux bits. The quickest way is to follow https://nix.dev/tutorials/continuous-integration-github-actions.html and get them from your binary cache.

Steps left:

make a PR for https://github.com/Infinisil/nixpkgs/commit/4d244410ee0f3e3ece5494533217bbafbd95d9b3
make a PR for https://github.com/mroi/nixpkgs/commit/839559b486d4c4dbd911a0f13a3b2d319b942cd5
build on linux & test on macOS

make a PR for mroi@839559b

You mean as a PR toward Nixpkgs or toward upstream QEMU? I can easily do the former. For the latter I would need to familiarise myself with their e-mail-based workflow. (Anyone here with experience in that?)

Former, although upstream patch would be really nice.

make a PR for mroi/nixpkgs@839559b

You mean as a PR toward Nixpkgs or toward upstream QEMU? I can easily do the former. For the latter I would need to familiarise myself with their e-mail-based workflow. (Anyone here with experience in that?)

The linux kernel has a bot explaining their email workflow which you might find similar with QEMU
https://github.com/torvalds/linux/pull/803#issuecomment-592644193

@r2r-dev did you manage to get it working?

Yup. I've prepared 2 test branches:

Based mostly on stuff mentioned in this issue, as well as my fix for virtfs.

https://github.com/r2r-dev/nixpkgs/commits/qemu-darwin-fixes
can be tested via:
https://github.com/r2r-dev/nixos-shell/commit/95b586bd5aef938a64ce2a9e9b61c99c9da936a2

With additional patches for M1, and vmnet device adapter:

https://github.com/r2r-dev/nixpkgs/commits/wip-qemu-darwin-fixes
can be tested via:
https://github.com/r2r-dev/nixos-shell/commit/1f53a820f1570f9e75263e6a69103465e0cf2c46

Obviously, both of these branches aren't pretty and surely will need some cleanup.
Some remarks:

Qemu's v9fs requires additional patching (see 1st branch). Without it, any getdents or getdents64 syscalls on shared filesystem will result in a deadlock.
overlayfs won't work on top of v9fs mounts. It seems that v9fs cannot handle xattrs querying. This can be fixed by mounting lower-dir with -o version=9p2000.u. However, even though I've managed to boot into VM, it was quite unstable. In order to mitigate that I've used squashfs-based /nix/store instead of one mounted from host.

@r2r-dev This is far beyond cool, works well for my 2018 15' MBP! Kudos.

This is superb.

This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/nix-macos-monthly/12330/4

@r2r-dev Hi, I'm giving this a try on a M1 machine and I'm getting the error:
# ./result/bin/nixos-shell examples/darwin.nix -- -- --argstr system aarch64-linux -I nixpkgs=http://github.com/r2r-dev/nixpkgs/archive/f68911645209742013dd6abe570a23548d9a5780.tar.gz --option builders-use-substitutes true

[...]

error: a 'x86_64-darwin' with features {} is required to build '/nix/store/zq54yx4gydijq5hryk60qs660pw84s5h-MacOSX-SDK-11.0.0.drv', but I am a 'aarch64-darwin' with features {benchmark, big-parallel, nixos-test, recursive-nix}

If I change virtualisation.qemu.pkgs.system = "aarch64-darwin" I now get:
error: anonymous function at /nix/store/5s210gnpc6f7nn2jhnaakq8zm47p2dlz-source/pkgs/build-support/bintools-wrapper/default.nix:8:1 called without required argument 'lib'

       at /nix/store/5s210gnpc6f7nn2jhnaakq8zm47p2dlz-source/pkgs/stdenv/native/default.nix:133:18:

          132|       inherit lib nativePrefix;
          133|       bintools = import ../../build-support/bintools-wrapper {
             |                  ^
          134|         name = "bintools";
(use '--show-trace' to show detailed location information)

Any ideas on how to get this working on aarch64-darwin?  Thanks!

See https://github.com/NixOS/nixpkgs/pull/162243 for an updated PR with the upcoming QEMU functionality backported to v6.2.0.

Looking back over this thread, what remains for nixpkgs? At this point, is it only the qemu-vm.nix stuff?

I've just merged #162243.  Is there anything else to be done here, or can this be closed now?

Perhaps a test that can, for example, be run on ofborg should be created?

I'm trying to run NixOS VM on my macOS machine with M1. What's missing so far:

virtfs still doesn't work (see below)
we need to backport commit by @Infinisil that allows to build VM script for macOS (it applies quite cleanly)
still missing ability to detect whether to use kvm or hvf in nixos/lib/qemu-common.nix

Here's what I'm trying to build:
flake.nix:

I'm trying to run NixOS VM on my macOS machine with M1. What's missing so far:

either QEMU 6.2 or virtfs doesn't work (see below)
we need to backport commit by @Infinisil that allows to build VM script for macOS (it applies quite cleanly)
still missing ability to detect whether to use kvm or hvf in nixos/lib/qemu-common.nix

Here's what I'm trying to build:
flake.nix:
{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
  };

  outputs = { self, nixpkgs }: {
    nixosConfigurations = {
      qemu = nixpkgs-aarch.lib.nixosSystem {
        system = "aarch64-linux";
        modules = [
          (import ./qemu.nix { inherit nixpkgs; })
        ];
      };
    };
  };
}

qemu.nix:
{ nixpkgs }:
{ config, pkgs, lib, modulesPath, ... }:

{
  imports = [
    "${modulesPath}/virtualisation/qemu-vm.nix"
  ];

  system.stateVersion = "21.11";

  networking.useDHCP = false;
  networking.interfaces.eth0.useDHCP = true;

  services.getty.autologinUser = "test";
  users.users.test = {
    isNormalUser = true;
    extraGroups = [ "wheel" ]; # Enable ‘sudo’ for the user.
    hashedPassword = "$5$ZjxV1oXdKi61yoVw$iBoQwq3URyEeN56FZWEouw4G1UJM0eGQMLm54hmvd/8"; # "test"
  };
  security.sudo.wheelNeedsPassword = false;

  virtualisation = {
    qemu.pkgs = nixpkgs.legacyPackages.aarch64-darwin;
    # These two disable 9p/virtfs entirely
    useNixStoreImage = true;
    sharedDirectories = lib.mkForce { };
    graphics = false;
  };
}

I build it like this:

point nixpkgs input at a repo with cherry-picked qemu.pkgs commit
nix build .#nixosConfigurations.qemu.config.system.build.vm
copy result/bin/run-nixos-vm and replace -enable-kvm -machine virt,gic-version=host -cpu host in qemu arguments with -machine virt,accel=hvf,highmem=off -cpu cortex-a72

I was trying this with previous version of nixpkgs with the qemu.pkgs patch, and it worked fine, I got autologin on console. After I've rebased that commit on top of latest nixpkgs master, I don't get any output on both text and graphical consoles, both with useNixStoreImage and sharedDirectories lines and without them.
I'll post both rebased qemu.pkgs commit and my example somewhere a bit later, and will keep trying.

@YorikSar Looks like qemu 7.0 enables 9pfs for macOS: https://wiki.qemu.org/ChangeLog/7.0#9pfs
That should fix the virtfs stuff, right?

@ElvishJerricco the feature you’re noting for 7.0 is what was backported to 6.2 via #162243!

@YorikSar: I have an Intel-based MacBook 13" that is just sitting on my desk. I'd love to help testing this! I'm a bit of a noob so need some handholding though. Alternatively, I can give you SSH access?

nix run on an intel mac with this flake fails with:
error: a 'x86_64-linux' with features {} is required to build '/nix/store/89zr2gg8dd8mw8x5nq33fyy65imsc2qv-append-initrd-secrets.drv', but I am a 'x86_64-darwin' with features {benchmark, big-parallel, nixos-test}.
It earlier also failed with the same error but for stdenv-linux, but rerunning nix run a few times settled on initrd-secrets.

Of course, duh. I unfortunately don't have a linux builder set up! If no one else is able to find one I can get one set up but it probably won't be until next week...

I've tested it on Intel, it seems like I've messed up QEMU arguments and also there's some problem with HVF there. I wonder if old macOS SDK on Intel is to blame. I will keep figuring this out.
By the way, one of the outcomes of this project that I see would be some way to have a local Linux builder on the Mac without any hassle. Something like a VM that is automatically spun up when the builder is needed.

type=q35 instead of virt did the trick! Now it booted into the system, just like on M1 :)
I'll check out what does q35 actually mean later. For now I've updated the commit and uploading (x86 virtualised build is painfully slow) my build artefacts to Cachix. You should be able to run nix run nixpkgs#cachix -- use yoriksar-gh to pull them during nix build.

@YorikSar Tried to give it a go. Got a hash mismatch on the 9p patchfile:
error: hash mismatch in fixed-output derivation '/nix/store/5l7yvn3kgsinggwpn6bb40wxxbvb7d0g-f5643914a9e8f79c606a76e6a9d7ea82a3fc3e65.patch.drv':
         specified: sha256-VBcvy+t6UAQeUe4hqytBJIORKoe9auAmiVur9ZLUIhs=
            got:    sha256-8i13wU135h+YxoXFtkXweBN3hMslpWoNoeQ7Ydmn3V4=

@ElvishJerricco I've fixed the hash and rebased the branch on the current master. Please check again, don't forget to update flake inputs.

By the way, one of the outcomes of this project that I see would be some way to have a local Linux builder on the Mac without any hassle. Something like a VM that is automatically spun up when the builder is needed.

@YorikSar Do you have an outline of how this could work?  Today it seems like you need a remote Linux builder to build the builder image itself, unless it has already been built and cached somewhere...

unless it has already been built and cached somewhere...

That would be hydra.nixos.org. If we add a system configuration to the relevant channels, as channel blockers, we can pretty much guarantee an "image" to be available.
Scare quotes because we could use 9p for the system store, just like we do in the tests. The work of configuring the VM is best done through a nix-darwin module. Or perhaps two modules. One for bootstrapping a lightweight builder, and a second one for configuring a VM with arbitrary configuration, that can also serve as a linux builder.

Updates:

I've rebased the branch on the latest nixpkgs master, applied fixes for #178954
Rebuilt the example from https://github.com/NixOS/nixpkgs/issues/108984#issuecomment-1150149700 and pushed all rebuilt packages to Cachix
Created pull request for qemu 9p patch: #180221
Created pull request for NixOS modules changes: #180222

Next steps:

Get those pull requests merged
Add a test to Hydra that would run this basic configuration on Darwin platforms for unstable.

@YorikSar I have to disable the Nix sandbox, otherwise I get errors trying to build python3.10-requests-2.28.0.drv on darwin. Something about sockets in the test suite. Other than that, it appears to work!

@ElvishJerricco That's great! Do you get the same errors if you rebuild python310Packages.requests from master with sandbox? If so, maybe file an issue (or even a PR) about that?

@willcohen Oh, yeah, to build this you need to have a Linux builder configured for your Nix daemon.

Ah, that's unfortunate -- please could you edit your comment "So far on my M1 MacBook Pro this flake:" to indicate this requirement, since I've hit this and did not understand the subsequent comments until I had the problem.
Requiring a Linux builder to be configured to allow building the Linux VM from Darwin scuppers my plans to use this. What are new users meant to do if you don't have a Linux builder available? I was hoping to have a configuration I could share with anyone, instruct someone to install NixOS, and all they'd have to do was build a flake and they'd get a VM. But it seems this isn't possible yet, is that right?

@YorikSar I have to disable the Nix sandbox, otherwise I get errors trying to build python3.10-requests-2.28.0.drv on darwin. Something about sockets in the test suite. Other than that, it appears to work!

Works with sandbox turned off for me too!

I've created a repository with instructions on how to use results of merged PRs: https://github.com/YorikSar/nixos-vm-on-macos - feel free to check that out and see if it works for you.

@peterwaller-arm: Correct me if I'm wrong, but I believe if cache.nixos.org can serve enough Linux build products to deploy a blank NixOS VM then you can bootstrap things on a Darwin system without a remote Linux builder by:

Deploying a blank NixOS VM for use as a local Linux builder
Use that to then build other NixOS systems

I had to add highmem=off to the QEMU run script to get this to work on my M1 MacBook Pro due to https://github.com/utmapp/UTM/issues/3946.  Without that fix this causes a kernel protection failure.
Specifically, I made the following change:
@@ -29,7 +29,7 @@

 # Start QEMU.
-exec /nix/store/h5yrqjziml4vwqq7azph6y9r2r267aq6-qemu-host-cpu-only-7.0.0/bin/qemu-system-aarch64 -machine virt,gic-version=2,accel=hvf:tcg -cpu max \
+exec /nix/store/h5yrqjziml4vwqq7azph6y9r2r267aq6-qemu-host-cpu-only-7.0.0/bin/qemu-system-aarch64 -machine virt,gic-version=2,accel=hvf:tcg,highmem=off -cpu max \
     -name nixos \
     -m 1024 \
     -smp 1 \

I'll try to put up a PR to fix this.

Actually, after investigating this further it looks like the real issue was due to me being on an old version of macOS (12.0.1) and setting highmem=off is an undesirable solution (because it limits you to a smaller address space).  Upgrading macOS to 12.4 or newer should fix the kernel protection failure.

The main issue I see is that networking does not appear to work within the guest VM when I tested the instructions from https://github.com/NixOS/nixpkgs/issues/108984#issuecomment-1234408582 (and several variations, including the default networking options)
For example, these commands both fail when run from inside the VM:
$ host google.com
;; communications error to <IP_ADDRESS>#53: timed out
;; communications error to <IP_ADDRESS>#53: timed out
;; communications error to fec0::3#53: timed out
;; no servers could be reached

$ ping <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>) 56(84) bytes of data.
64 bytes from <IP_ADDRESS>: icmp_seq=0 ttl=255 time=1886095099620737 ms
wrong data byte #16 should be 0x10 but was 0x0
#16	0 1 0 1 98 43 26 63 0 0 0 0 34 2b 1 0 0 0 0 0 10 11 12 13 14 15 16 17 18 19 1a 1b
#48	1c 1d 1e 1f 20 21 22 23
64 bytes from <IP_ADDRESS>: icmp_seq=0 ttl=255 time=2630877887998121 ms (DUP!)
wrong data byte #16 should be 0x10 but was 0x0
#16	0 1 0 2 99 43 26 63 0 0 0 0 89 34 1 0 0 0 0 0 10 11 12 13 14 15 16 17 18 19 1a 1b
#48	1c 1d 1e 1f 20 21 22 23
64 bytes from <IP_ADDRESS>: icmp_seq=0 ttl=255 time=3259692985970728 ms (DUP!)
wrong data byte #16 should be 0x10 but was 0x0
#16	0 1 0 3 9a 43 26 63 0 0 0 0 42 3c 1 0 0 0 0 0 10 11 12 13 14 15 16 17 18 19 1a 1b
#48	1c 1d 1e 1f 20 21 22 23
^C
--- <IP_ADDRESS> ping statistics ---
3 packets transmitted, 1 received, +2 duplicates, 66.6667% packet loss, time 2004ms
rtt min/avg/max/mdev =<PHONE_NUMBER>620736.537/2592221991196528.640/3259692985970727.795/-6642957192704464.-896 ms, pipe 4

It looks like there is another issue for qemu VMs on macOS: the default DNS server (<IP_ADDRESS>) created by qemu as part of user-mode networking (SLiRP) does not work.  See:
https://github.com/utmapp/UTM/issues/2353
You have to override it with the IP address of a working DNS server (e.g. networking.nameServers = [ "<IP_ADDRESS>" ];) in order for DNS to work on the guest VM.

Alright, I managed to get the idea I proposed in https://github.com/NixOS/nixpkgs/issues/108984#issuecomment-1250066968 working.  Specifically, I created a repository that lets you bootstrap a Linux guest of the same architecture on a macOS host, relying on a pre-seeded cache instead of a separate Linux builder (similar to https://github.com/NixOS/nixpkgs/issues/108984#issuecomment-1234408582):
https://github.com/Gabriella439/macos-builder
You can then use that Linux guest to build and run additional NixOS VMs on macOS.
As far as I can tell, the only remaining improvements I can think of are:

cache.nixos.org should probably host those cached build products instead of using my own cache so that there's a better chain of trust

Maybe we should wait until https://github.com/utmapp/UTM/issues/2353 is closed before marking this done?
I'm not sure if this should be a blocker, though, because the work-around (hard-coding a public DNS server) is pretty easy

Other than that, I don't see any work remaining here.

cache.nixos.org should probably host those cached build products instead of using my own cache so that there's a better chain of trust

We should add the default config toplevel to the nixpkgs-darwin jobsets, so that we can guarantee its availability in the cache.
Perhaps the keys could be loaded through a 9p mount instead of baking them into the config?
Maybe enable automatic gc? That will save some annoyance and confusion. Relevant paths will be gc-rooted on the host anyway, and copying between the host and vm should be fairly quick.
Having this in a nix-darwin module would be awesome.

Having this in a nix-darwin module would be awesome.

+100 to this idea!

How does this compare to https://github.com/nix-community/linuxkit-nix, which also seems to provide a virtual machine that can be used as a remote linux builder for a macOS machine?

@anka-213 that is orthogonal to this issue which aims to allow building interactive VMs (per the first sentence of the description). Looks like it could be used to bootstrap a NixOS VM build though.

So is it now possible to run github actions nixos vm tests on macos runners to speed up the CIs ?

Unfortunately, not yet. Current state is that you can build a VM from NixOS configuration that will run on macOS.
This PR by @roberth provides a way to run NixOS tests on non-Linux: https://github.com/NixOS/nixpkgs/pull/193336, and it worked for me on macOS, but it is still a draft.

@roberth: Alright, I added automatic GC and also removed the hardcoded keys like you suggested.  The next step is that I'll work on upstreaming this into Nixpkgs so that it can be built and cached as a nixpkgs-darwin jobset

@Gabriella439 @domenkozar very excited to see this being pushed over the finish line!
Wondering whether there is any tutorial, blog post or similar that explains this further?

@schickling The PR added a Nixpkgs manual section: https://nixos.org/manual/nixpkgs/unstable/#sec-darwin-builder

@schickling: I'm also going to announce this via a blog post, but that will focus more on the background work and motivation for this

% nix run nixpkgs#darwin.builder
error: a 'aarch64-linux' with features {} is required to build '/nix/store/6i2m765ifgzihprpzix5f5ghghpi2sk3-install-credentials.drv', but I am a 'aarch64-darwin' with features {benchmark, big-parallel, nixos-test}

Am I doing something wrong?

I think the issue is that it's not being built by hydra.nixos.org because it's missing a platforms attribute.  I'm looking into this

This should hopefully fix it: https://github.com/NixOS/nixpkgs/pull/207459

It's still not working 😔
% nix run github:NixOS/nixpkgs/nixpkgs-unstable#darwin.builder
error: a 'aarch64-linux' with features {} is required to build '/nix/store/6i2m765ifgzihprpzix5f5ghghpi2sk3-install-credentials.drv', but I am a 'aarch64-darwin' with features {benchmark, big-parallel, nixos-test}
% nix build --dry-run github:NixOS/nixpkgs/nixpkgs-unstable#darwin.builder
these 2 derivations will be built:
  /nix/store/6i2m765ifgzihprpzix5f5ghghpi2sk3-install-credentials.drv
  /nix/store/yl0r0qgdrx50ymbrwi66wpxf4krwkrxy-create-builder.drv
these 223 paths will be fetched (220.65 MiB download, 983.89 MiB unpacked):
<..................>

Yeah, this should hopefully be the last fix: https://github.com/NixOS/nixpkgs/pull/207577

This was working for me, but started failing just now:
% nix run github:NixOS/nixpkgs/nixpkgs-unstable#darwin.builder
cannot build on 'ssh-ng://builder@localhost': error: cannot open connection to remote store 'ssh-ng://builder@localhost': error: unexpected end-of-file
Failed to find a machine for remote build!
derivation: g43xrrmfr80sza9fshg99l1yc80ny1il-nixos-disk-image.drv
required (system, features): (aarch64-linux, )
1 available machines:
(systems, maxjobs, supportedFeatures, mandatoryFeatures)
(aarch64-linux, 10, , )
error: a 'aarch64-linux' with features {} is required to build '/nix/store/g43xrrmfr80sza9fshg99l1yc80ny1il-nixos-disk-image.drv', but I am a 'aarch64-darwin' with features {benchmark, big-parallel, nixos-test}

% nix build --dry-run github:NixOS/nixpkgs/nixpkgs-unstable#darwin.builder
these 4 derivations will be built:
  /nix/store/g43xrrmfr80sza9fshg99l1yc80ny1il-nixos-disk-image.drv
  /nix/store/1lhnpvb9n70gz7sf3yapwybl476cd9rd-run-nixos-vm.drv
  /nix/store/hzy3zqwb795fai8zla07hf8c04gb1cvj-nixos-vm.drv
  /nix/store/vcps33v4bra88pv8h00v41mqs3vka3sd-create-builder.drv

% nix flake metadata github:NixOS/nixpkgs/nixpkgs-unstable
Resolved URL:  github:NixOS/nixpkgs/nixpkgs-unstable
Locked URL:    github:NixOS/nixpkgs/2d38b664b4400335086a713a0036aafaa002c003
Description:   A collection of packages for the Nix package manager
Path:          /nix/store/ld350gbq2w5654yw4q95kxhxp4569zi6-source
Revision:      2d38b664b4400335086a713a0036aafaa002c003
Last modified: 2023-01-17 10:21:52

Trying a slightly older commit, and it works again:
% nix run github:NixOS/nixpkgs/6fecd5a257327db6116b28acdfb45561e65e3dfa#darwin.builder
Formatting '/Users/tobias/nixos.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=21474836480 lazy_refcounts=off refcount_bits=16
Formatting '/tmp/nix-vm.qudT2qcgrQ/store.img', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=1198522368 backing_file=/nix/store/9rk814dr9dr1cxz698iq22g3c5713hfy-nixos-disk-image/nixos.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
...
<<< Welcome to NixOS 23.05pre-git (aarch64) - ttyAMA0 >>>

Run 'nixos-help' for the NixOS manual.

nixos login: builder (automatic login)

[builder@nixos:~]$

@bergkvist (I'm an interested user, non-authoritative comment) One tricky thing to consider is whether the packages are in the cache yet for the commit you're building. If not, your machine would try to build it and fail.
I think the nixos/nixpkgs/nixpkgs-unstable branch might run ahead of the cache being populated. Try nixos/nixpkgs/nixos-unstable instead, which I think is updated as the cache is updated if I understand correctly.

When I try to build github:NixOS/nixpkgs/594b94b4c3038f5c2cfb2f5d9c10ef30c7070a4c#darwin.builder inside of my builder (nix run github:NixOS/nixpkgs/af89d3a2be6f70edb187dd817377d6c4360134fa#darwin.builder)
% nix build github:NixOS/nixpkgs/594b94b4c3038f5c2cfb2f5d9c10ef30c7070a4c#darwin.builder
error: build of '/nix/store/rnwbjdkq9wz43nlsmggr7ysn7k9h9z2w-nixos-disk-image.drv' on 'ssh-ng://builder@localhost' failed: builder for '/nix/store/rnwbjdkq9wz43nlsmggr7ysn7k9h9z2w-nixos-disk-image.drv' failed with exit code 32;
       last 10 log lines:
       > copying path '/nix/store/65cvxfd36l9cawzj1gkx3zpzj0bdfg0b-unit-serial-getty-.service' to 'local'...
       > copying path '/nix/store/8izgxj2nzpwkd0hpm3r38zkiklm1jkzr-unit-nixos-activation.service' to 'local'...
       > copying path '/nix/store/82nc5jrzri2w4hw6rhffis9m6n25xnbv-unit-systemd-fsck-.service' to 'local'...
       > copying path '/nix/store/r9h4h2j299iz32gnnr2qksz2hsi6i8cw-unit-systemd-udevd.service' to 'local'...
       > copying path '/nix/store/klkbaapdbcs35bxyp0l97ic3lpp0njnb-user-units' to 'local'...
       > copying path '/nix/store/c04f008fdyqmarjl6nf35fdrzfm9h6lk-system-units' to 'local'...
       > copying path '/nix/store/786ah3bs0n5lcvjq2qhp4laiqkg60wr0-etc' to 'local'...
       > copying path '/nix/store/mj26i1zj0bjv1b732bafrg2ffly34adj-nixos-system-nixos-23.05pre-git' to 'local'...
       > mount: /build/root/build/root: must be superuser to use mount.
       >        dmesg(1) may have more information after failed mount system call.
       For full logs, run 'nix log /nix/store/rnwbjdkq9wz43nlsmggr7ysn7k9h9z2w-nixos-disk-image.drv'.
error: builder for '/nix/store/rnwbjdkq9wz43nlsmggr7ysn7k9h9z2w-nixos-disk-image.drv' failed with exit code 1
error: 1 dependencies of derivation '/nix/store/wr12pkszc3gj021fy08cch9syhqvwyj2-run-nixos-vm.drv' failed to build
error: 1 dependencies of derivation '/nix/store/p5pa1imyb5yc6l3wy1cda5bz38dc430f-nixos-vm.drv' failed to build
error: 1 dependencies of derivation '/nix/store/ld4rlr7sf599qi53bmd4d2xrjsm53n11-create-builder.drv' failed to build

https://github.com/NixOS/nixpkgs/pull/210812 also seems to have broken other builds - which triggered this emergency fix: https://github.com/NixOS/nixpkgs/pull/211218. Doesn't seem to have made it to the unstable branch yet though.

Is it possible to add NixOS modules to the darwin.builder's config? If so, it's not clear how.
I ask because I'd like to run this on our aarch64-darwin CI machines, but enable the NixOS Tailscale module to make each aarch64-darwin's darwin.builder available to our Hydra as a remote aarch64-linux builder, as well. (I believe this would also get around any host port 22 and/or macOS firewall issues.)

@dhess: Yes.  If you copy the code from here:
https://github.com/NixOS/nixpkgs/blob/0cf78f562607f75256703af568814d3a62ded6de/pkgs/top-level/darwin-packages.nix#L215-L233
… then you can add in additional modules of your own

In case you need more than 3GB of memory for your builder
# Starts darwin.builder VM with 8GB RAM
QEMU_OPTS="-m 8192" nix run github:NixOS/nixpkgs#darwin.builder

Does qemu support passing rosetta binary to guest system? Would be awesome to enable https://github.com/NixOS/nixpkgs/pull/202847 to support building for x86_64-linux too.

@misuzu Rosetta "drive" is mounted via higher-level Virtualisation.framework while QEMU uses lower-level Hypervisor.framework. The later is more like kvm while the former is more like QEMU itself, so I doubt its support will happen in QEMU. I think, #5241 is relevant to bringing virtualisation backends other than QEMU for running NixOS VMs.

UTM does support it, is supposedly based on QEMU, is packaged, and we have a NixOS module to support it on its guests https://github.com/NixOS/nixpkgs/pull/202847. If it has a qemu compatible command line it might be close to a drop-in replacement.

#5241

If it's like qemu, you can set qemu.package at the test level if you use the documented entrypoint.

@roberth UTM docs clearly state https://docs.getutm.app/advanced/rosetta/:

Rosetta allows you to run Intel Linux executables in an Apple Silicon Linux virtual machine (using Apple Virtualization backend).

QEMU VMs don't have this setting in UTM.

@roberth IIUC, UTM only supports rosetta if you tell it to use Apple Virtualisation instead of QEMU.

I love QEMU, but this is awesome. I still don't quiiite have copy/paste working this way (not sure if spice + wayland is a thing), but using Apple Virtualization solves the other major pain point of having normal-ish resolutions relative to my laptop and external monitor working with wayland/sway on a graphical NixOS VM. I feel like there should be a flashing exclamation point somewhere that this works so well.

supposedly based on

This was outdated information then. Thanks for correcting me.
I was thinking about the VM tests in my last comment. It'd be far easier to add Apple Virtualisation support if it's just for the darwin.builder, as it that's a simple entrypoint. I guess you'd have to figure out

how to invoke Apple Virtualization from the command line. Some apple command? UTM? Custom program?
create a new module with a new option that holds the Apple Virtualization invocation

figure out which NixOS options hold the image and kernel

change the darwin.builder package to get that option instead of the qemu one

how to invoke Apple Virtualization from the command line. Some apple command? UTM? Custom program?

Maybe this could be adapted to our use-case:
https://github.com/Code-Hex/vz/tree/main/example/linux
https://github.com/lima-vm/lima/pull/1155

I do wish the kernel had support for enabling TSO mode per process. I don't even know if that's possible in a VM, but it'd be very nice for asahi linux to be able to use the rosetta binary via hacks and with TSO enabled.

There's a vmcli project: https://github.com/gyf304/vmcli#vmcli-1 - it allows to run VMs with Virtualisation.framework from CLI. We could use it to start VM and add Rosetta support to it.

There is also Tart that has Rosetta support. One just need to:
brew install cirruslabs/cli/tart
tart clone ghcr.io/cirruslabs/ubuntu:20.04 ubuntu
tart run --rosetta="rosetta" ubuntu

And make sure Rosetta is configured inside the VM according to these docs.

I've gotten pretty far with Tart. Thanks to @fkorotkov for making me aware of it in the above comment!

Create a new, empty Virtualization.Framework VM named nixos with tart create nixos --linux
Build an aarch64-linux NixOS configuration as a raw-efi image using https://github.com/nix-community/nixos-generators. For bonus points, set virtualisation.rosetta.enable = true and nix.settings.extra-platforms = [ "x86_64-linux" ] in the VM's config, so that the VM will be able to run x86_64 Linux binaries, as well.
Overwrite the Tart VM's disk.img with the nixos.img file from step 2.
Run the VM with tart run nixos --rosetta rosetta

I've spent most of my time on this so far just learning how Virtualization.Framework works, and not much time experimenting with the running VM, but everything I've tried so far works great, including building and running x86_64-linux derivations. Additionally, I think it would be straightforward to automate steps 1-3 in a Nix derivation.
Even better would be to use the Mac's /nix/store in the VM. To experiment with this, I did the following:

Run the VM with tart run nixos --rosetta rosetta --dir="store:/nix/store"
In the running VM, run mount -t virtiofs com.apple.virtio-fs.automount /mnt. This mounts the Mac's /nix/store on /mnt/store in the running VM. I can see all the files there, though performance is a bit slow. Maybe mounting it with some cache= options would help.
Bind mount the virtiofs-mounted Mac Nix store to the VM's /nix/store via mount --bind /mnt/store /nix/store. (I think this extra step is necessary due to the way that Virtualization.Framework makes all shared filesystems available under a single mount point, but I might be missing something.) Confirm that everything still works: running commands still works, etc.

So far, so good. However, my particular use case for this VM is to use it as an aarch64-linux remote builder, so I want to be able to build derivations in it, while re-using the Mac's own /nix/store for disk and network savings. Now I try to build something that's not in the store: nix run nixpkgs#hello. This fails with error: cannot open connection to remote store 'daemon': error: reading from file: Connection reset by peer
Looking at the logs, the problem is apparent: unexpected Nix daemon error: error: changing ownership of path '/nix/store': Operation not permitted
This makes sense: Nix is running in multi-user mode on my Mac, so my Mac's /nix/store is owned by root and managed by the local nix-daemon running as root. I'm running tart as my local macOS user, not root, so tart doesn't have permissions to change ownership on the Mac's /nix/store.
Anyway, besides that small catch, this route looks very promising. I think the main issue for our use case is that Virtualization.Framework doesn't support nested virtualization, so we won't be able to use this virtualized remote aarch64-linux builder to run NixOS tests.

@dhess: You probably don't want to share the host's /nix/store with the builder anyway, for the reason outlined in this comment:
https://github.com/NixOS/nixpkgs/blob/99563190d585896d7ddb43ea448a95574dfa2373/nixos/modules/profiles/macos-builder.nix#L159-L168

@Gabriella439 Thanks, I did see that comment linked from elsewhere, either in this discussion, or in a related one.
However, if I understand the comment correctly, I don't think it applies in our use case. We have several dedicated macOS remote builders (let's call them mac1, mac2, etc.), which appear in our x86_64-linux NixOS dedicated builder's /etc/nix/machines file; let's call that machine nixos1. mac1 etc. are not for interactive use: they only build derivations that are delegated by nixos1.
nixos1 cannot build aarch64-linux derivations, but virtualized aarch64-linux NixOS VMs running on mac1 etc. could. So what we'd like to do is have mac1 etc. run these aarch64-linux VMs (let's call them arm1 etc.), list those VMs in nixos1's /etc/nix/machines file, and then have nixos1 delegate aarch64-linux jobs to those VMs. Therefore, unless I'm missing something, the comment does not apply in our case, because the host is not delegating any jobs to the guest, only nixos1 is.
Perhaps it's possible that nixos1 could ask arm1 to build platform-independent derivation foo at the same time as it's asking mac1 to build it? But even in this case, I assume that whichever remote builder wins the race will go first while the other waits for the lock, then proceeds (and presumably sees that the derivation has magically appeared in the store in the meantime). In any case, I wouldn't expect a deadlock to occur in this scenario.

Over in this podman issue, I see that virtiofs in MacOS doesn't bypass open file limits, which leads to unexpected behavoir in the guest OS: https://github.com/containers/podman/issues/16106
I'm addressing the issue directly in podman, but I was hoping someone from this issue might have an opinion on if qemu should be bumping it's own ulimits up when it is acting as the virtiofs daemon?

You can now run NixOS tests on macOS, too.  See: https://github.com/NixOS/nixpkgs/pull/282401/
Note that you still need a Linux builder to build the test VM, though


The spec did not pass validation

The spec did not pass validation
-> SuperRecord (1.3)
    - WARN  |  SuperRecord/SuperRecord/SuperFetchedResultsControllerDelegate.swift:120:10: warning: parameters of 'controller(_:didChangeObject:atIndexPath:forChangeType:newIndexPath:)' have different optionality than expected by protocol 'NSFetchedResultsControllerDelegate'
    - NOTE  |  CoreData.NSFetchedResultsControllerDelegate:2:23: note: requirement 'controller(_:didChangeObject:atIndexPath:forChangeType:newIndexPath:)' declared here
    - WARN  |  SuperRecord/SuperRecord/SuperFetchedResultsControllerDelegate.swift:96:10: warning: parameters of 'controller(_:didChangeSection:atIndex:forChangeType:)' have different optionality than expected by protocol 'NSFetchedResultsControllerDelegate'
    - NOTE  |  CoreData.NSFetchedResultsControllerDelegate:3:23: note: requirement 'controller(_:didChangeSection:atIndex:forChangeType:)' declared here
    - WARN  |  SuperRecord/SuperRecord/SuperFetchedResultsControllerDelegate.swift:87:10: warning: parameter of 'controllerWillChangeContent' has different optionality than expected by protocol 'NSFetchedResultsControllerDelegate'
    - NOTE  |  CoreData.NSFetchedResultsControllerDelegate:4:23: note: requirement 'controllerWillChangeContent' declared here
    - WARN  |  SuperRecord/SuperRecord/SuperFetchedResultsControllerDelegate.swift:160:10: warning: parameter of 'controllerDidChangeContent' has different optionality than expected by protocol 'NSFetchedResultsControllerDelegate'
    - NOTE  |  CoreData.NSFetchedResultsControllerDelegate:5:23: note: requirement 'controllerDidChangeContent' declared here

Analyzed 1 podspec.

Fixed!


Render direct to screen option turns Viewport to black

Godot version
v3.5.2.stable.official [170ba337a]
System information
Ubuntu 22.10 LTS, GTX1060, GLES2
Issue description
here is two same viewports with render_direct_to_screen enabled and disabled.

Steps to reproduce

open attachment
toggle render_direct_to_screen option

Minimal reproduction project
New Game Project.zip

Related to https://github.com/godotengine/godot/issues/58499.

Please take a look at the description for render_direct_to screen
In particular "If used, reading from the Viewport or from SCREEN_TEXTURE becomes unavailable". In your MRP you are trying to read from the Viewport even though you enabled render_direct_to_screen. Those two things are incompatible.

so, what use case of this option then?

It's used when you want to render the Viewport directly to the screen without using a ViewportContainer or a Sprite to read from it. This can be much more efficient on certain hardware. As the description says it is a niche optimization that shouldn't be used in most cases.


Can I access the phpmyadmin on wocker?

I want to see if some options were put on the sql, but how can I access the mysql?
tks

Hi @pedrofilho20,
I recommend you Sequel Pro or Workbench.
Host: wocker.dev or <IP_ADDRESS>
Username: wordpress
Password: wordpress
Database: wordpress
Port: 3306


What is the purpose of the num_stages parameter?

LGTM. Thanks
Just wanted to add that pipelining is not limited to matmul. Conceptually, it's applicable to overlap any expensive compute and memory instructions.

@Jokeren @vesuppi  Thanks very much. I get it.


chore(FDS-339): Proposal and POC for selecting rows from Table

Proposal and POC for selecting rows from Table
Resolves FDS-339

Adds support for row selection in Table

Dependencies

Adds ramda at v0.27.1

Testing

yarn cosmos
navigate to Table -> Fixtures -> Selection and play with the different fixtures

Fails

:no_entry_sign:
Please add a '### Dependencies' section to explain the reason we are changing dependencies.

  Generated by :no_entry_sign: dangerJS against ee8172d1354804fb5ee517277bbe18f92e558bb3

Hi @lordpixel :
Just a question regarding the status ALL selection
I can select all by clicking the checkbox at the top, but I can't select any of the children independently. Is this expected?

Hi @lordpixel :
Just a question regarding the status ALL selection
I can select all by clicking the checkbox at the top, but I can't select any of the children independently. Is this expected?

Good catch, no! they should be intependently selectable, let me check

Manu, I marked this PR to auto merge so once we have two approvals, and builds all pass, it will come in automatically.

I am setting this PR to merge as I have already approved it and it looks like Rocio did not get a chance to review PRs yesterday before getting her other PRs up.


AlertManager Default Repeat_Interval

So, I have disabled any type of repeat for alertmanager. The thing is that I still get alerts every 4 hours it seems. So is there a default repeat interval for alert manager?
I have also setup inhibit rules but that doesn't seem to work completely.
route:
      receiver: slack
      group_wait: 30s
      routes:
      - match:
          alertname: NodeLowDiskSpace
          severity: 'critical'
        receiver: slack
        group_wait: 2m
        group_by: ['alertname', 'node', 'kubernetes_io_hostname']
      - match:
          alertname: NodeLowDiskSpace
          severity: 'warning'
        receiver: slack
        group_wait: 2m
        group_by: ['alertname','node', 'kubernetes_io_hostname']
      - match:
          alertname: NodeNotReady
        receiver: 'slack_pagerduty'
        group_by: ['alertname', 'node']

    inhibit_rules:
      # Inhibit warnings for NowLowDiskSpace when critical exists
    - source_match:
        alertname: NodeLowDiskSpace
        severity: 'critical'
      target_match:
        alertname: NodeLowDiskSpace
        severity: 'warning'
      # Apply inhibition if the alertname is the same.
      equal: ['alertname', 'node', 'kubernetes_io_hostname', 'device']

This is a usage question that should be directed to the prometheus-users google mailing group

For the sake of others asking the same question via google, the relevant section is here:
https://prometheus.io/docs/alerting/configuration/
# How long to wait before sending a notification again if it has already
# been sent successfully for an alert. (Usually ~3h or more).
[ repeat_interval: <duration> | default = 4h ]


remove dev operator dockerfile.

resolve: https://github.com/stolostron/backlog/issues/25046
Signed-off-by: morvencao<EMAIL_ADDRESS>

/lgtm


align default resolveOptions with webpack

there're some default resolveOptions inconsistencies between Rspack and webpack, which we should align with, see https://github.com/webpack/webpack/blob/main/lib/config/defaults.js#L1441
https://github.com/web-infra-dev/rspack/blob/main/packages/rspack/src/config/defaults.ts#L820

@Boshen any update on this？

@SyMind this seems resolved?


Run save numpy array

Added functionality to save a pickleable object or file.
Added test to check that if a numpy array is passed the pickled file compares to the numpy array
Added test to check if an invalid file is passed an exception is raised

Update to client code for retrieving a pickled object: https://github.com/simvue-io/client/tree/get_pickled


WIP Switch from glog to klog

closes #694

Did you intend to vendor klog?
I would prefer to see imports for klog aliased to glog rather than touching every single reference to glog:
glog "k8s.io/klog"

I also encourage you to lay the groundwork for your efforts going forward by discussing changes you want to make with owners and/or reviewers of this repo to ensure that your efforts are in line with their expectations.  Even maintainers need to do the legwork of consensus building or risk wasted effort.

@marun

Did you intend to vendor klog?

yes, I indend to vendor and write a gate check for the same too, so that it doesn't creep in again.

I would prefer to see imports for klog aliased to glog rather than touching every single reference to glog:
glog "k8s.io/klog"

Ok, i'll change it accordingly.

I also encourage you to lay the groundwork for your efforts going forward by discussing changes you want to make with owners and/or reviewers of this repo to ensure that your efforts are in line with their expectations.
sure, i'll keep it in mind from now on.

thanks for reviewing this.

@poothia, Are you stuck on something with this pr?

Superceded by #857


Navbar not responding in Mobile mode

Its hamburger doesn't open up!!

@codewithvk May I work on it?

@Castella1313 Go ahead!


Adjust speed monitor

What does this PR do?
Adjusts speed monitor to start tracking at BEFORE_DATALOADER instead of BEFORE_BATCH. This resolves issues with dataloader bottlenecks being potentially masked. Note that currently it should still work since data transfer is done async, but to be safe, we should start the clock on BEFORE_DATALOADER.

Did you run a benchmark with it to see if it changed the numbers?


feat: added support for "exposedFunction" per #306

This is really important! I'm with the same problem/need.

Per comments in these threads, I can actually close this PR as duplicating functionality that already exists.
https://github.com/yujiosaka/headless-chrome-crawler/issues/306#issuecomment-431606370
https://github.com/yujiosaka/headless-chrome-crawler/pull/275


[resolves #630] Move module org.apache.cxf.ext to extras

[resolves #629] Move module org.bouncycastle.pgp to extras

Refer to this link for build results (access rights to CI server needed):
https://fabric8-ci.fusesource.com/job/wildfly-camel-pullreq/491/
Test PASSed.


Image link decoration it not shown

The decoration for the image link inside a kustomization file where there is an array of images is not shown

it looks like the Kustomization resources have their own logic for being processed which means we cannot use the RefMappers to create links to images

I can perhaps look into this - what exactly should we link here? the "newTag" property?

The first thing should be to check if the images from inside a kustomization are being parsed and added to the images list. If they are, then a link should be created from the kustomization ( with a decoration, similar to the screenshot )

I think we should link the image on image.*.name , not sure how to use newName and newTag , I'm guessing those are replaced when the kustomization is previewed.
I was thinking of also adding the following to the base outgoing refs for resources:
{
    type: 'image',
    source: {
      pathParts: ['images', '*', 'name'],
    },
    target: {
      kind: 'monokle://image',
    },
  },

But, I saw that we handle the image ref types with a custom logic so we can parse the image name and tag..
We could allow passing a function in the refMapper for parsing a RefTarget based on a scalar value, then it should work with the usual logic of finding links.

Allowing to customize the refMappers by passing some functions to handle the parsing in some places could make it more powerful, but it will make creating plugins more difficult.
Although Wito has already managed to make plugins that run JavaScript code, so it would probably be something similar.
Not sure if I should do any refactors at this point, it seems like this should move into a parser library in the monokle-core.

as long as we make sure to sandbox any code as we currently do when evaluating expressions in our templates we are good..
Unfortunately I don't think the validators do this currently since they can be running in a non-node context (i.e. your browser) where the vm2 sandbox we are using in Monokle Desktop doesn't apply...


Feature request implement "port status"

Implement "port status" to display the status of ports as shown by the main screen of the web interface. Output formats would be similar to the formats supported by the other commands (md, json).
Great work. Thank you.

@chris4474
I'm working on feature completeness for GS316 on a branch.
I added the fields 'Port Status' and 'Link Speed' in the port settings command,
because the port status and port settings have a lot of overlap.
You can try this branch on your own ... is this what you're looking for?
https://github.com/nitram509/ntgrrc/tree/feature/gs316-port-set
Example:
ntgrrc port settings --address=<IP_ADDRESS>

Considered 'done' with latest release https://github.com/nitram509/ntgrrc/releases/tag/v0.12.0


FIX Block types that are in-line editable will have links to their page on reports

Currently the link on reports that takes you to edit a block will take you to the GridFieldDetailForm view of the block. Given we now offer in-line editing on a block it makes more sense to take the user to the page where the block exists.
I don't know why we haven't changed the CMSEditLink API for in-line editable blocks but I'm happy to do that instead if we agree that makes more sense.

I think I'll leave CMSEditLink for #506 to fix. Until then I'm not sure we can just switch CMSEditLink to just take you to the page containing the block - seems a bit lacking in terms of functionality.

I've changed my mind again - I think we should update CMSEditLink because we're running into the same issue on HistoryViewer now.
I'll update this PR...


ImportError: libcudart.so.8.0: cannot open shared object file: No such file or directory

Here is what happens when I try to import tensorflow. I installed using the compile from source method for Linux GPU. This is on Ubuntu 14.04. The file that it says there is no such file or directory is in the path file that I have listed. I've tried searching for help to no avail.
(I apologize if this doesn't make a lot of sense. Inexperience, sleep-deprivation, and frustration are the main contributors to my inability to communicate clearly at the moment.)

import tensorflow
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/init.py", line 61, in
from tensorflow.python import pywrap_tensorflow
File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 28, in
_pywrap_tensorflow = swig_import_helper()
File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow', fp, pathname, description)
File "/usr/lib/python3.4/imp.py", line 243, in load_module
return load_dynamic(name, filename, file)
ImportError: libcudart.so.8.0: cannot open shared object file: No such file or directory
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<pyshell#1>", line 1, in
import tensorflow
File "/usr/local/lib/python3.4/dist-packages/tensorflow/init.py", line 24, in
from tensorflow.python import *
File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/init.py", line 72, in
raise ImportError(msg)
ImportError: Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/init.py", line 61, in
from tensorflow.python import pywrap_tensorflow
File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 28, in
_pywrap_tensorflow = swig_import_helper()
File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow', fp, pathname, description)
File "/usr/lib/python3.4/imp.py", line 243, in load_module
return load_dynamic(name, filename, file)
ImportError: libcudart.so.8.0: cannot open shared object file: No such file or directory
Failed to load the native TensorFlow runtime.
See https://github.com/tensorflow/tensorflow/blob/master/tensorflow/g3doc/get_started/os_setup.md#import_error
for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.

I encounter the same error, 64 bit machine and my solutions was :
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda/lib64"
sudo ldconfig
Hope my solution can give you some help.

This question is probably better for stackoverflow as we try to keep the github issues focused on bugs and feature requests.
See also  https://www.tensorflow.org/get_started/os_setup#optional_linux_enable_gpu_support which is consistent with what @MalcolmSun was suggesting


Update requirements-test.txt

Description
Please include a summary of the change. For large or complex changes please include enough information to introduce your change and explain motivation for it.
Changes proposed in this pull request:

@Mergifyio rebase


Uploading 'empty' blocks to page blobs is wasteful.

Re: PageBlobService#create_blob_from_stream
When a page blob is provisioned, the content is a stream of 0s. Therefore, when uploading a page blob, any block that is all zeros does not need to be uploaded, it can simply be skipped.
Most fixed-length VHD images are mostly empty blocks, so uploading all those zeros represents a huge waste of time and bandwidth.
Our optimum solution for dealing with this in azurectl was to load a block from /dev/zero and compare it to the current block in the stream. If they are equal... move along to the next block without uploading.
See https://github.com/SUSE/azurectl/blob/master/azurectl/storage/page_blob.py#L50 for example.
And, just for reference, a 30GB openSUSE Leap 42.3 image, contains < 300MB of full blocks. Uploading with the API convenience methods, or az, results in an upload of 29.7GB of zeros. o.0

@bear454 Thank you so much for the suggestion! We will consider it carefully and update this issue once I have any update. 👍

I have a working patch( https://github.com/bear454/azure-storage-python/commit/28318b2cb41fba75d4784fb80259da5f3792966e ) but there are still some outstanding issues, maybe you can help with...

I don't see anything in the tests that runs at a small enough scope to test the change. Suggestions on how to test?
There seems to be a code path for patching an existing VHD, and obviously you don't want to skip blocks in that case, so we need to differentiate that code path.

Hi @bear454, you should be patching _PageBlobChunkUploader, not _BlobChunkUploader, since we only want the change for page blobs.
And there should be plenty of tests covering this scenario, such as this one.

Hi @bear454, I spoke to my colleagues and we'd love to have this improvement in the library. Would you be interested in submitting a PR for it? If not, I'll add this item to our backlog. Thanks! 😄

@bear454, I have added this item to our backlog, thank you for the suggestion!


UI glitches after exiting the test level

Once you exit the test level, the UI becomes invisible & unclickable
Example: https://streamable.com/76lm8l

Thanks for the report, it's been traced down to an issue with questui, which should be fixed in the core mods sometime soon


Add time test file

import CPython's time test
Reference:
https://github.com/python/cpython/blob/main/Lib/test/test_time.py

One issue: specify the CPython version in your commit message. That way we can keep track of which files might need updating in the future. (Usually checking out the 3.8 or 3.9 branch on CPython and copying things over)

Cpython version is 3.8.11
now it's added to commit message

Great. I triggered the actions again since I just merged #3009 that made some big changes to the time module. Now we will hopefully get a better picture of what errors appear.


Bind Ctrl-a and Ctrl-e on macOS

FIX: Add a workaround for a bug on macOS where Ctrl-a and
Ctrl-e getting stuck at the edge of inline nodes.
Issue https://discuss.prosemirror.net/t/shortcut-ctrl-a-ctrl-e-dont-work-on-macos-with-inline-node

I didn't bind these two commands to Home and End in pcBaseKeymap because the browsers on Windows don't seem to have this bug.
(Test environment: Windows Server 2019 version 1809, Chrome 97, Firefox 96)
I'm not sure if these commands should be exported as public APIs, so I only exported them for testing.

Merged this, with some changes to export the commands and make them do the right thing inside inline nodes.

Thanks!


Add support for image rendition prefetching

Let's say you are rendering a list of 20 EventPage objects on a responsive website, and each representation has three renditions of the event's listing_image, targeting multiple devices.
Without these changes
The optimal QuerySet looks something like:
EventPage.objects.select_related("listing_image")[:20]

Rendering this page for the first time will require 20 x 3 x 5 database operations just for card image renditions, because of the need to look up each rendition separately, and to create each new rendition with get_or_create() to avoid race conditions. If rendition caching is used, you can add to that 20 x 3 individual cache reads and 20 x 3 individual cache write operations.
Rendering the page a second time will require 20 x 3 database or cache lookup operations for card image generation (depending on whether rendition caching is being used)
With these changes
The optimal QuerySet looks something like:
EventPage.objects.select_related("listing_image").prefetch_related("listing_image__renditions")[:20]

Rendering this page for the first time will require (20 x 3 x 4) + 1 database queries for card image generation (a ~20% reduction). If rendition caching is used, you can add to that 20 x 3 individual cache write operations, but zero reads (a 50% reduction in cache operations overall).
Rendering the page a second time will require 1 database operation to fetch all renditions (whether rendition caching is enabled or not). In this example, that is at least a 5,000% improvement.
What else have I changed here?
The single get_rendtion() method was already a bit difficult to follow, and bringing prefetch support into the mix only made that situation worse. So, I thought it only right to break things down into several more dedicated methods. With the addition of docstrings and type hints, I'm sure you'll agree that it makes things much easier to follow overall.
Considerations
If all renditions are prefetched, and two or more EventPage objects shared the same listing_image, the performance of the first render will be negatively impacted, because renditions generated for each instance of listing_image would not be shared by the others... leading to the same thumbnail images being freshly generated multiple times.
We could sacrifice some of the performance gains elsewhere to reduce this impact - However, I think optimising for the more common scenario (each image being unique) is the better route.
We could also solve this problem a different way: By passing in the current HttpRequest (or some other arbitrary mutable object), and caching any newly created renditions on there - making them available practically anywhere within the same thread.

I haven't had a chance to look at the code yet, but this looks like a really great improvement, thanks!
On the face of it, it looks similar to a change I proposed here: https://github.com/wagtail/wagtail/pull/7225. The differences are 7255 allows selecting specific rendition sizes but doesn't support prefetch over a relationship (which is important for page listings).

Thanks @kaedroho. I must admin I had forgotten about #7225. I think you're right - this is more 'generic' solution to the problem. Most Wagtail sites I work on these days typically have around 7-15 renditions for each image, and the amount of extra memory required to load all of those renditions is minimal, so just being able to do a standard prefetch_related() without specifying the renditions you need (which may be in-flux, or difficult to pin down) is pretty much an 'instant win' in most cases.

@kaedroho I'm totally bias, but I would say this PR probably solves the more immediate problem: Making prefetched renditions a possibility for those who probably expect that they already are :) I feel like most dev understand the models involved, and the basic select_related/prefetch_related options well enough - then become frustrated when the prefetched data isn't used (which has been case on a couple of projects I've helped with).

@kaedroho would you be kind enough to do a full review for this, or assign to another core team member for review? It's unlikely anyone else will review this if it looks like you already have interest.

@kaedroho The changes in functionality here are actually very slight. It's the more contentious changes (splitting up get_rendition into separate methods, the method names used) that I feel you're input would be particularly valuable.

@zerolab I've made some documentation additions here, and am in the process of writing something for the release notes.

@gasman this is ready to merge too. I am away from computers for a few hours

@zerolab / @gasman Have added docs and release notes for you to do with as you please :) If you'd like to keep the release notes simple, that's absolutely fine with me.


pulsar broker feature request

Pulsar cluster has three brokers: broker 1:6650, broker 2:6650 and broker 3:6650
Now I replace the original broker with three new broker nodes. The new brokers are broker 4:6650, broker 5:6650 and broker 6:6650
My server config the new broker (4 ~ 6),  but when I started server, I found that the old broker (1 ~ 3)  was connected either. Why? Should the broker be connected with what I configed ?

broker.conf  advertisedAddress whether there is a set value？

broker.conf advertisedAddress whether there is a set value？

new broke node config right value to advertisedAddress in broker.conf.

Closed as stale. Please open a new issue if it's still relevant in maintained versions.
This seems not a feature request. If you'd like to ask a question, you can go to https://github.com/apache/pulsar/discussions/categories/q-a.


Enhance UI to bubble up errors that occur during registration

Right now the site blindly redirects and swallows exceptions. These get logged, but there is no UI indication if an error occurs.

Hopefully that works. Covered by 9fbdc1e227bfd261ed49e5fdd22055bc5bed0fe9 and ef2966f3f6a9f6446bb031f408ee0ccf7d19ee5a


fix JS extraction; upgrade Tower

Our javascript.pot file has been empty for a long time and this was on my list to investigate.  Looks like it got left out of the media -> static move.  Upgraded Tower while I was in there.

r+

thanks


Database connection error in dbeaver

When I create a new database connection in dbeaver, I receive an error message:
(base) C:\Users\thinkpad>python -m buenavista.examples.duckdb_postgres D:\abc.db
Using DuckDB database at D:\abc.db
Listening on <IP_ADDRESS>:5433
Catalog Error: Type with name REGCLASS does not exist!
Did you mean "real"?
Catalog Error: Table Function with name pg_get_keywords does not exist!
Did you mean "main.duckdb_keywords"?
LINE 1: SELECT GROUP_CONCAT(word, ',') FROM pg_catalog.PG_GET_KEYWORDS() WHERE word...
^
Catalog Error: Type with name REGCLASS does not exist!
Did you mean "real"?

Please explain why this is and how it needs to be resolved? Thank you.

This sounds like an issue in Buenavista, not DuckDB

Indeed it's an issue with Buenavista-- we hack some of the postgres catalog to make it work with DuckDB, but there is still plenty of stuff that different clients want that is missing-- e.g., the regclass types. I unfortunately do not have time right now to add it in, so I'm just going to post a note that Buena Vista can be used with e.g. psql and leave it at that.

How can I establish a connection correctly in dbeaver now?

Using navicat also fails to connect properly, with the following error prompt:
Binder Error: Referenced column "datlastsysoid" not found in FROM clause!
Candidate bindings: "pg_database.oid"
LINE 1: SELECT DISTINCT datlastsysoid FROM pg_database
^
Binder Error: Referenced column "datlastsysoid" not found in FROM clause!
Candidate bindings: "pg_database.oid"
LINE 1: SELECT DISTINCT datlastsysoid FROM pg_database

This sounds like an issue in Buenavista, not DuckDB

yes,thank you

@e20999 https://github.com/jwills/buenavista/commit/d99f8fcf236a1cad3bab38308feaff59866912d6
fix that

I have a fix too, similar to @e20999's, except it removes just the cast instead of the entire AND clause: https://github.com/jwills/buenavista/pull/42


Crash on editable install in requirements.txt from pip-compile

Bug description
Here is my requirements.txt, output from pip-compile, containing one editable install:
#
# This file is autogenerated by pip-compile with python 3.10
# To update, run:
#
#    pip-compile --no-emit-index-url requirements.in
#
-e file:instrument

When I run:
pip-audit -r requirements.txt

It crashes with an AttributeError: AttributeError: 'NoneType' object has no attribute 'name'
Expected behavior
I expect it not to crash.
Screenshots and logs
DEBUG:pip_audit._cli:parsed arguments: Namespace(local=False, requirements=[<_io.TextIOWrapper name='requirements.txt' mode='r' encoding='UTF-8'>], project_path=None, format=<OutputFormatChoice.Columns: 'columns'>, vulnerability_service=<VulnerabilityServiceChoice.Pypi: 'pypi'>, dry_run=False, strict=False, desc=<VulnerabilityDescriptionChoice.Auto: 'auto'>, cache_dir=None, progress_spinner=<ProgressSpinnerChoice.On: 'on'>, timeout=15, paths=[], verbose=True, fix=False, require_hashes=False, index_url='https://pypi.org/simple', extra_index_urls=[], skip_editable=False, no_deps=False, output=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, ignore_vulns=[])
Traceback (most recent call last):
  File "/path/to/venv/bin/pip-audit", line 8, in <module>
    sys.exit(audit())
  File "/path/to/venv/lib/python3.10/site-packages/pip_audit/_cli.py", line 428, in audit
    for (spec, vulns) in auditor.audit(source):
  File "/path/to/venv/lib/python3.10/site-packages/pip_audit/_audit.py", line 66, in audit
    for dep, vulns in self._service.query_all(specs):
  File "/path/to/lib/python3.10/site-packages/pip_audit/_service/interface.py", line 148, in query_all
    for spec in specs:
  File "/path/to/lib/python3.10/site-packages/pip_audit/_dependency_source/requirement.py", line 101, in collect
    for _, dep in self._collect_cached_deps(filename, reqs):
  File "/path/to/venv/lib/python3.10/site-packages/pip_audit/_dependency_source/requirement.py", line 299, in _collect_cached_deps
    for req, resolved_deps in self._resolver.resolve_all(iter(req_values)):
  File "/path/to/venv/lib/python3.10/site-packages/pip_audit/_dependency_source/interface.py", line 87, in resolve_all
    yield (req, self.resolve(req))
  File "/path/to/venv/lib/python3.10/site-packages/pip_audit/_dependency_source/resolvelib/resolvelib.py", line 75, in resolve
    result = self.resolver.resolve([req])
  File "/path/to/venv/lib/python3.10/site-packages/resolvelib/resolvers.py", line 481, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/path/to/venv/lib/python3.10/site-packages/resolvelib/resolvers.py", line 348, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/path/to/venv/lib/python3.10/site-packages/resolvelib/resolvers.py", line 140, in _add_to_criteria
    identifier = self._p.identify(requirement_or_candidate=requirement)
  File "/path/to/venv/lib/python3.10/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 290, in identify

Platform information

OS name and version: macOS Big Sur v11.5.1
pip-audit version (pip-audit -V): 2.4.2
Python version (python -V or python3 -V): 3.10.3
pip version (pip -V or pip3 -V): 22.2

Additional context
Add any other context about the problem here.

Are we missing some of your stacktrace here?

@jamesbraza Can you try the fix in #328 and see if it works for you? You can install that locally with:
$ pip install git+https://github.com/trailofbits/pip-audit.git@fix/327

@di I have updated the description, seems I did cut off the stack trace!  The fix in #328 did not fix my issue sadly :/

Thanks for the debugging help @jamesbraza! I've cut 2.4.3 with the fixes in #331; should be available momentarily.


"Absolute Layout Issue"

Hi, I spotted an issue when I try to implement "AbsoluteLayout" In any page...
My code looks kind of like this:

<rxui:ReactiveContentPage
x:Class="SmartDemo.View.InicioV"
xmlns="http://xamarin.com/schemas/2014/forms"
xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml"
xmlns:rxui="clr-namespace:ReactiveUI.XamForms"
xmlns:vm="clr-namespace:SmartDemo.ViewModel"
Title="Inicio"
x:TypeArguments="vm:InicioVM">
<ContentPage.Content>

        <!--  Status  -->
        <Button
            x:Name="Btn_status"
            BackgroundColor="{StaticResource GENERICButtonColor}"
            Font="35, Bold"
            HeightRequest="150"
            Image="off.png"
            Text="Desconectado."
            VerticalOptions="StartAndExpand"
            WidthRequest="200" />

    </StackLayout>
</ContentView>

I was trying to add an "Overlay" page to work as a "MasterDetail" or something...
First solution I tried: "Add  and do some mods:
1.- At my first stacklayout I added this
"StackLayout AbsoluteLayout.LayoutBounds="0, 0, 1, 1"
AbsoluteLayout.LayoutFlags="All">
[... this stacklayout content]
</StackLayout"
2.- Then I created a "ContentView" right after the end of the previous stacklayotu:
     <!-- Overlay -->
<ContentView x:Name="overlay"
             AbsoluteLayout.LayoutBounds="0, 0, 1, 1"
             AbsoluteLayout.LayoutFlags="All"
             IsVisible="False"
             BackgroundColor="#C0808080"
             Padding="10, 0">

  <StackLayout Orientation="Vertical"
               BackgroundColor="White"
               HeightRequest="175"
               WidthRequest="300"
               HorizontalOptions="Center"
               VerticalOptions="Start"
               Margin="0,20,0,0" >

    <Label BackgroundColor="Black" FontSize="18" TextColor="White" HorizontalOptions="Fill" Text="Im an Overlay" />
  </StackLayout>
</ContentView>

I tried to compile and run severaltimes the code and it broke everytime... So I decided to remove piece by piece of the code... until I reached the point where I had to remove the "" then everything went ok... but then the content on my page disappeared and I was able only to summon the "content view" trhoughout (whenever I keep the ContentView, If I delete the content view code everything goes ok)
So It seems that "AbsoluteLayout" gives problems whenever I try to work along with it...
Still maybe I'm doing something wrong with my code... I'll leave the full code here:

<rxui:ReactiveContentPage
x:Class="SmartDemo.View.InicioV"
xmlns="http://xamarin.com/schemas/2014/forms"
xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml"
xmlns:rxui="clr-namespace:ReactiveUI.XamForms"
xmlns:vm="clr-namespace:SmartDemo.ViewModel"
Title="Inicio"
x:TypeArguments="vm:InicioVM">

<ContentPage.Content>

        <!--  Status  -->
        <Button
            x:Name="Btn_status"
            BackgroundColor="{StaticResource GENERICButtonColor}"
            Font="35, Bold"
            HeightRequest="150"
            Image="off.png"
            Text="Desconectado."
            VerticalOptions="StartAndExpand"
            WidthRequest="200" />
       </StackLayout>

    <!--  OVERLAY  -->
    <ContentView
        x:Name="Beta"
        BackgroundColor="Red"
        IsVisible="False">
        <StackLayout Margin="0,0,50,0" BackgroundColor="Wheat">
            <Label
                BackgroundColor="Black"
                FontSize="18"
                Text="CAPITAL CITIES"
                TextColor="White" />
            <StackLayout HorizontalOptions="Center">
                <Button x:Name="Close_btn" FontSize="Large" />
            </StackLayout>
        </StackLayout>
    </ContentView>
</ContentPage.Content>

Alright forget it: I found the problem:
You can actually use "AbsoluteLayout" but you MUST remove the "ContenPage.Content" in order to AbsoluteLayout to work...
Now I can easily implement a menu that Mocks the "Master Detail" navigation menu...


Open source license?

Hi! I'm wondering if there is meant to be an open-source license for this repo, or if this was intentionally not included. Great job on the code. Thanks!

Hi,
The omission of an open-source license was not intentional. I have added a suitable open-source license to the repository shortly to ensure clear guidelines for usage and contribution.
Thank you for bringing this to my attention!
Best regards,
Haobo


feat: use pressable with press feedback

Details

Made a new component that uses the already existing withDelayToggleButtonState and makes the logic of showing a checkmark icon and different tooltip text once the pressable was pressed.

If inline prop is true, it uses Text as a wrapper instead of Pressable, since there are some limitation on RN side regarding that ([SO](https://stackoverflow.com/questions/66590167/vertically-align-pressable-inside-a-text-component, RN issue 1, RN issue 2).

Used new component in CodesPage buttons
Change the already existing CopyTextToClipboard component to use the new component, since it's the same logic

NOTE: the icon in Settings -> Profile -> Contact Method is not being displayed in Android. I checked in main and asked other people if they were experiencing the same, and it seems it's a problem. However, it's not related with this change.
Fixed Issues
$ https://github.com/Expensify/App/issues/19180
https://github.com/Expensify/App/issues/19180#issuecomment-1553127743
Tests

Go to Settings -> Security -> Two-factor authentication
Check that the buttons contain icons and change the icons when pressed
Go to Settings -> Profile -> Contact Method
Check that no regressions in the<EMAIL_ADDRESS>clipboard text was made
Go to a chat -> click on the user name
Check that the copy icon next to the email suffered no regression

[x] Verify that no errors appear in the JS console

Offline tests
Not relevant.
QA Steps

Go to Settings -> Security -> Two-factor authentication
Check that the buttons contain icons and change the icons when pressed
Go to Settings -> Profile -> Contact Method
Check that no regressions in the<EMAIL_ADDRESS>clipboard text was made
Go to a chat -> click on the user name
Check that the copy icon next to the email suffered no regression

[x] Verify that no errors appear in the JS console

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android / native
[x] Android / Chrome
[x] iOS / native
[x] iOS / Safari
[x] MacOS / Chrome / Safari
[x] MacOS / Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is approved by marketing by adding the Waiting for Copy label for a copy review on the original GH to get the correct copy.
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If a new component is created I verified that:

[x] A similar component doesn't exist in the codebase
[x] All props are defined accurately and each prop has a /** comment above it */
[x] The file is named correctly
[x] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[x] The only data being stored in the state is data necessary for rendering and nothing else
[x] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[x] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[x] All JSX used for rendering exists in the render method
[x] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG))

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.
[x] I have checked off every checkbox in the PR author checklist, including those that don't apply to this PR.

Screenshots/Videos

Web
https://github.com/Expensify/App/assets/25725586/fec5c040-4517-464e-8616-8084e0c8ce13

Mobile Web - Chrome
https://github.com/Expensify/App/assets/25725586/f853ac95-7d64-4f25-b71b-5f9f863acadd

Mobile Web - Safari
https://github.com/Expensify/App/assets/25725586/454e1011-c40e-41f8-adbc-4e6c72aa7a8d

Desktop
https://github.com/Expensify/App/assets/25725586/59f2776b-4284-4590-8d82-5e8407d662a8

iOS
https://github.com/Expensify/App/assets/25725586/1b9d874c-248a-4fc8-ab4f-fa4facdd880b

Android
https://github.com/Expensify/App/assets/25725586/68269d89-8e91-4a31-8b36-d31c4020f0b8

I'm currently OOO until 1st of June, but @thiagobrez will take it from me

Hey! Looking into this now

Hey @rushatgabhane!

This comment should describe when / why would we use this prop

There is a console error caused by this line -

These 2 are done, just pushed the fix ✅

On Android I'm getting this weird bug where the recovery codes overflow.
Device - Pixel 5

This one is being fixed by this PR: https://github.com/Expensify/App/pull/19260
Let me know what you think 🙌🏻

@rushatgabhane would you be able to take a look today, so we can merge this 🚀

@thiagobrez the app crashes when we go to workspace > any workspace > bills. Could you please check. Thanks

the app crashes when we go to workspace > any workspace > bills

@MonilBhavsar Should be fixed now!

Sorry for not noticing earlier. But while we're here, can we please also update the "Copy" button inside verify page, to be pressable with feedack.

@MonilBhavsar Updated!
https://github.com/Expensify/App/assets/25725586/3ab41395-9e05-4f0f-9f0a-f6d6aa55ecbf

🚀 Deployed to staging by https://github.com/MonilBhavsar in version: 1.3.24-0 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

🚀 Deployed to production by https://github.com/roryabraham in version: 1.3.24-5 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
failure ❌

🍎 iOS 🍎
failure ❌

🕸 web 🕸
success ✅

🚀 Deployed to production by https://github.com/roryabraham in version: 1.3.24-5 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
cancelled 🔪

🍎 iOS 🍎
cancelled 🔪

🕸 web 🕸
success ✅


Add version: hluk.CopyQ version 7.1.0

Pull request has been created with WinGet Automation 🪟📦🤖
Microsoft Reviewers: Open in CodeFlow

Automatic Validation ended with:

Hi @vedantmgoyal2009,
The installation isn't unattended. Is there an installer switch to have the package install silently?
(Automated response - build 575.)


nullable field does not accept None as default

in the fields.py the method get_prep_value contains the following code:
...
        if not value and self.null and self.default is not None:
            value = self.default
        else:
            value = self.to_python(value)

...

this generates an exception in the following scenario:

the field is a DateField
the field is marked as null=True and has default=None

but I would expect it to not generate any exception: I simply desire to have a datefield which is null if not specified in the csv file. at the moment I patched the code as follows:
        if not value and self.null:
            value = self.default
        else:
            value = self.to_python(value)

which behaves as I expect. I thought you may desire to have it in your code, as I didn't find any other way to specify that a DateField can be optional in the csv file.

arghhh! After hours of having this problem we just discovered the identical fix. Lol.
So I second this!
Is this project maintained?

So I made Pull Request #21. @anthony-tresontani, can you please consider and pull it?


feat: reintroduce auto-impl

Auto-impl should be no_std compatible since https://github.com/auto-impl-rs/auto_impl/pull/73, ref https://github.com/rust-blockchain/evm/pull/71 and https://github.com/rust-blockchain/evm/pull/62

Given approval, how do we merge? I cannot apparently

@sorpaas hey - could we merge this PR and cut a new release?


Add martini lunch layer

Work with map, add layer for sit down restaurants, edit existing locations, make sure they all have the same info (websites, etc.)

DONE.


New message issue (Debian)

This happens when I try to start a new conversation. No available contacts are showing up in the populated list.
Here is the output from command-line:
Starting Yakyak v1.5.12-beta...
  using hangupsjs v1.3.10
--------
hangupsjs:: connecting
reconnecting 0
resolving proxy http://plus.google.com
resolving proxy https://plus.google.com
resolved proxy DIRECT
resolved proxy DIRECT
connected 0
Error on YakYak window:
 {
  msg: "Uncaught TypeError: Cannot read property 'id' of undefined",
  url: 'file:///usr/lib/yakyak/resources/app.asar/ui/app.js',
  lineNo: 242,
  columnNo: 27,
  error: TypeError: Cannot read property 'id' of undefined
      at file:///usr/lib/yakyak/resources/app.asar/ui/app.js:242:27
      at EventEmitter.<anonymous> (file:///usr/lib/yakyak/resources/app.asar/ui/app.js:104:16)
      at EventEmitter.emit (events.js:315:20)
      at Object.onMessage (electron/js2c/renderer_init.js:91:925)
}
--- End of error message in YakYak window.

And a screenshot (blurred for privacy):

This should now be working properly in the latest beta release!

Confirmed this is working in the current 1.5.12-beta release!


multiple types support

Consider this example:
size:
  description: Property size
  example: 45
  types:
    - number
    - string

It is valid json schema, but prmd renders only first type. Currently prmd renders only null type additionally.

Good call, would you be up for a pull request implementing this? Thanks!


Fix atl08 classification presets

"we should default the ATL08 Classifications (when selected) to be just Ground, Canopy, and Top of Canopy."

fixed