Disposing any WriteableBitmap causes NRE in WriteableBitmap.Lock() for new WriteableBitmaps

I'm not sure if I could create a consistent repro, but I will explain what happens.
Step 1: I have a Window created (not the parent Window) which has WriteableBitmaps that I write to often. I close this Window (which disposes the WriteableBitmaps)
Step 2: I open a new instance of of this Window which will then create new WriteableBitmaps
Step 3: First WriteableBitmap.Lock() will throw NRE
Stacktrace is simply:
   at Avalonia.Media.Imaging.WriteableBitmap.Lock()

Unfortunately for me, my entire program revolves around WriteableBitmaps, and this will occur extremely often for all users

Happens with both .UseSkia() and .UseDirect2D1()

I really cannot create a repro for this for some reason. I tried for hours to make something really simple and it didn't crash, then I tried to mimic what my project does and it still didn't crash. The best I can do is link you to my faulty project and you can clone/download it and follow the below steps, sorry for the inconvenience. It's currently lightweight since I'm in the very early stages.
Step 1: Download/clone from https://github.com/Kermalis/PokemonGameEngine/tree/c7d12078cafaf6586a61a952746e6da403d35a8a
Step 2: Run the "MapEditor" project (debug and release both have the issue)
Step 3: In the main window that appears, click "Open Block Editor"
Step 4: Click "Clear Block" in the new window
Step 5: Close the new window
Step 6: Click "Open Block Editor" again
Step 7: Click "Clear Block" again (will crash at this point)
The only thing going on in the background of the program is that WriteableBitmaps are getting redrawn when you click "Clear Block" (you can visually see it update the images). When you close the window it will Dispose() the WriteableBitmaps. When you open it again, there are now new WriteableBitmaps, and clicking "Clear Block" again will queue a redraw and the crash will occur on the first WriteableBitmap.Lock() call.
It's really simple, and I really don't know why I wasn't able to reproduce it, but these steps are simple enough to follow in my opinion. As I said before, this is a very necessary workflow for this program; opening the block editor and closing it again over and over. As it stands currently, the user must restart the program each time the block editor is closed in order to avoid this crash.
If you're skeptical that it's my code, I don't see how that is possible. The crash is in Avalonia, and it's upon accessing PlatformImpl which I have no access to in my code. My code doesn't do anything funky, it just uses unsafe code (pointers) to directly write to the pixels of the WriteableBitmap, and yes, there are checks to make sure it's not writing outside of the bitmap, so this should be acceptable behavior. Also, it's not that specific button, it's any Lock() call that comes after opening the new window. I chose that button for the step-by-step to make it as easy as possible to reproduce.
Also, I'm on Windows 7 x64 if that makes any difference.

@kekekeks I updated to 0.9.0-preview7 as per your request: https://github.com/Kermalis/PokemonGameEngine/tree/ca204fa735aaab655b8f603c398ae38a42fa96bf

It appears this line https://github.com/Kermalis/PokemonGameEngine/blob/ca204fa735aaab655b8f603c398ae38a42fa96bf/MapEditor/Core/Blockset.cs#L297
is calling the event on the previous editor instance and hence working with deleted bitmaps.

@ahopper ah, thank you so much for pointing that out, it turns out it was me after all. Perhaps WriteableBitmap should throw an ObjectDisposedException so it is easier to figure out what was happening. Anyway, I will close this, as it was not a bug in Avalonia, my apologies to everyone.


USB Support

This issue is tracking the implementation of USB.

[ ] USB Device crate atomics don't compile on thumbv6m https://github.com/mvirkkunen/usb-device/issues/2
[ ] Can we do better than Mutex around the inner portion of the UsbBus impl?  This currently satisfies the 'Sync' requirement but may to too coarse
[x] Move buffer allocation off the stack.  Either do something custom as a static or look at an existing heap allocation crate for this purpose
[ ] Prune out some of the ep! helper macro invocations.  Likely don't need so many of these now
[x] implement the is_stalled method
[ ] Can we do better than all these globals? https://github.com/wez/atsamd21-rs/blob/usb/metro_m0/examples/usb_serial.rs#L23-L25

FWIW I spent some time getting the USB stuff to work with RTFM (which could help with eliminating the globals) but found it hard to get both the inner references (i.e. device needs a ref to the serial object) and then return those same objects from init (because the objects constructed in init did not have static lifetimes). I am sure there is some magic here that would allow RTFM to work that I am just missing though I am not sure whether RTFM counts as solving the globals issue.

Not sure what the official status is on this, I just want to report that I've tried porting the usb_serial example from metro_m0 to a trinket_m0. To do so, I've un-commented the usb feature from hal and enabled it from trinket_m0. I was expecting compilation issues with atomics (?) but the build went fine - using nightly. Loading the resulting binary resulted in my Linux host reporting cdc_acm 5-2:1.0: failed to set dtr/rts and the trinket immediately locking up.
I've switched clock init to use the internal oscillator (trinket_m0 has no external osc?) and am now getting EILSEQ(-84) and EPROTO(-71) errors on dmesg log upon trinket reset. I'm assuming some form of miscommunication is now occuring. Also the board no longer locks up. Yay, progress?
[1490808.087147] usb 5-2: new full-speed USB device number 22 using uhci_hcd
[1490808.251255] usb 5-2: device descriptor read/all, error -84
[1490808.373836] usb 5-2: new full-speed USB device number 23 using uhci_hcd
[1490808.536213] usb 5-2: device descriptor read/all, error -71
[1490808.537160] usb usb5-port2: attempt power cycle
[1490808.973926] usb 5-2: new full-speed USB device number 24 using uhci_hcd
[1490809.011214] usb 5-2: device descriptor read/all, error -71
[1490809.340593] usb 5-2: new full-speed USB device number 25 using uhci_hcd
[1490809.371206] usb 5-2: device descriptor read/8, error -71
[1490809.504211] usb 5-2: device descriptor read/8, error -71
[1490809.610493] usb usb5-port2: unable to enumerate USB device

Anyway, not having a debugger (yet), using LED blinks as a heartbeat and being otherwise way out of my league I'm kind of stuck there for now but I thought I would just drop by and leave my cool bro story for others to read while I'm sadly going back to regular Arduino ino coding to achieve some ordinary success with my USB MIDI project.
As a general note, I feel embedded Rust is shaping up nicely and I'm still quite excited about it. Good work!

I'm interested in working on this, but as a newbie to rust its best if I'm told what to do. How can I help?

This is one of those thing's that's left over from when @wez was running the project and frankly I'm scared to touch it lol.

I've been a bit too busy to pick this up in a while.  Things have matured a bit more in the upstream usb-device crate so what I think needs to be done in this one now is:

Update to the latest usb-device code; there have been a few API changes that will require some changes
Revise https://github.com/atsamd-rs/atsamd/blob/master/boards/metro_m0/examples/usb_serial.rs to use the now standalone CDC-ACM serial implementation crate https://github.com/mvirkkunen/usbd-serial

With that done, we'll have a relatively low level support for USB on this class of devices and can consider actually enabling that code when publishing to crates.io, but there is still more to be done to make practical USB device implementations:

USB HID support (near and dear to my heart for keyboards)
USB storage device support to expose eg: on-board SPI based flash storage, or perhaps UF2 bootloader

I see these as requiring some work in the upstream usb device crate with appropriate atsamd-specific support happening in this crate.
There's also a distinction between low-level support (eg: it basically works) and ergonomic high-level support (eg: there's a nice way to respond to USB HID events and embed support in a consuming application crate).
Beyond the bus implementation in this crate, most of the work to be done is generic USB stuff that should happen in the usb-device project.

I'll take a look at updating the samd21 crate, but I don't have hardware so it will need someone else to test.
I'll also try and get it working for samd51, which I do have hardware for.

Looking at the samd21 code, it looks very similar to what is required for samd51. I expect there's a good amount we can put in the common crate.

This is slowly coming along here. Some notes/changes for comment:

Instead of only exposing an implementation of the usb_device::bus::UsbBus device, I'm going to expose a lower-level (no state, no memory management) API UsbController, which the UsbBus implementation will be built off.
I'm refactoring the ep! macro to collect all the registers for a given endpoint into a single structure, EndpointRegs, which will provide a nicer internal API over mutating the state of the endpoint registers.
Aside from HAL layers, move away from using the term Bank to describe one endpoint in an endpoint pair, as I couldnt find any reference to it outside of datasheets.
I'm using a single lifetime across all objects from UsbController downwards, and instantiating them all at the same time. This allows us to avoid Ref* and Mutex boxing.

Heya @wez, how did you get the dbgprint! macro to dance your tune?
I made the mistake of writing the samd51 driver all at once, so it attaches but doesnt enumerate. I'm thinking it would be nice to have a printf out a SERCOM UART, but if thats something youve already done I'll gladly copypasta :)

@twitchyliquid64 Did you make any progress on the samd21 port? I've got some devices I'd be willing to test with.

@BenBergman no - haven't ported samd21 USB support over to the 0.3 version of usb-device. The original code (which ran from an earlier version) should still be usable.
In terms of working on the USB peripheral, I've been side-tracked by other projects as of late, but hope to get back to this + fixing the other bug soon.

I tried compiling the Metro-M0 usb-serial example but it is failing to compile for me, so I'm guessing some other change broke something. I poked at it a bit but low level USB is a bit beyond me and the limited time I have to focus on it.
Let me know when you get back to it and I can test on a few Adafruit M0 boards I have lying around.


Filter by path in project finder

Did I get it right that the ctrl-cmd-p filter just searches the main directory name, but not the project's path?
Because, most of my projects have the same root folder and thus, look like that in the project finder:

live
~/Websites/clientA/web/live
live
~/Websites/clientB/web/live
live
~/Websites/clientC/web/live

So if I type "clientC", the filtering is not applied.
And filtering by "live" would make no sense.
Is there a way to make the project finder search the entire path?

Currently the Project Finder only filters by name. I'd suggest first saving, then editing the titles of those projects to include the client's name: Project Plus: Save then Project Plus: Edit Projects in the command palette. It is something that could be added though if you still need it. Thoughts, @mehcode?

I seem to remember a TODO comment indicating this somewhere in the code base at one time (it's not there now). Anyway.. definitely in favor of this. We should rank an exact name match higher than a path match.
Relevant code to modify:
https://github.com/mehcode/atom-project-plus/blob/master/lib/project-finder-view.js#L124-L129

Thanks for considering the feedback, great!


Confusing print statement if openshiftVerifyDeployment fails

If the container errors out and the deployment fails, the logs inside Jenkins for the "verify deployment" step are unclear. It shows this same error message regardless of why the pod failed:
[Pipeline] openshiftVerifyDeployment

Starting "Verify OpenShift Deployment" with deployment config "dials-gateway" from the project "vibe-develop".
  Waiting on the latest deployment for "dials-gateway" to complete and scale to "1" replica(s) ...
Operation will timeout after 240000 milliseconds
java.util.MissingFormatArgumentException: Format specifier '%s'
	at java.util.Formatter.format(Unknown Source)
	at java.util.Formatter.format(Unknown Source)
	at java.lang.String.format(Unknown Source)
	at com.openshift.jenkins.plugins.pipeline.model.IOpenShiftDeploymentVerification.coreLogic(IOpenShiftDeploymentVerification.java:140)
	at com.openshift.jenkins.plugins.pipeline.model.IOpenShiftPlugin.doItCore(IOpenShiftPlugin.java:308)
	at com.openshift.jenkins.plugins.pipeline.dsl.OpenShiftDeploymentVerifierExecution.run(OpenShiftDeploymentVerifierExecution.java:42)
	at com.openshift.jenkins.plugins.pipeline.dsl.OpenShiftDeploymentVerifierExecution.run(OpenShiftDeploymentVerifierExecution.java:18)
	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
	at hudson.security.ACL.impersonate(ACL.java:221)
	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
[Pipeline] echo
hudson.AbortException: "Verify OpenShift Deployment" failed

It would be great if it printed logs from the failed pod or a link to the deployment in the console instead!

hey @eleanordare which version of the plugin were you running at?  A bug in the logging statement at the key point in your stack trace, com.openshift.jenkins.plugins.pipeline.model.IOpenShiftDeploymentVerification.coreLogic(IOpenShiftDeploymentVerification.java:140),  was fixed with commit 7c4ac3b1c5bc3c144973d47bed08caf67db783a3
The fix made v1.0.43 of the plugin.
Pretty sure that is what is going on here.  But please reopen if you are still seeing this at v1.0.43 or later.
thanks


[rmlui] Update to version 6.0

https://github.com/mikke89/RmlUi/releases/tag/6.0

[x] Changes comply with the maintainer guide.
[x] SHA512s are updated for each updated download.
[x] The "supports" clause reflects platforms that may be fixed by this new version.
[x] Any fixed CI baseline entries are removed from that file.
[x] Any patches that are no longer applied are deleted from the port's directory.
[x] The version database is fixed by rerunning ./vcpkg x-add-version --all and committing the result.
[x] Only one version is added to each modified port's versions file.

All features passed with following triplets:
x86-windows
x64-windows
x64-windows-static

Usage test passed on x64-windows.


CoreOS identification fix

Description
CoreOS has a VERSION_ID identified in os-release that contains an epoch value since the coreos project started and hence will change every upgrade. This breaks the catchall os-release test.
Apologies for not catching this in the initial submission.
Features
Issues
Behaviour pre-patch:
$ distro="${NAME:-${DISTRIB_ID}} ${VERSION_ID:-${DISTRIB_RELEASE}}"; echo $distro
Container Linux by CoreOS 1409.1.0

Behaviour post-patch:
$ [[ $NAME == "Container Linux by CoreOS" ]] && distro="$ID"; echo $distro
coreos

TODO

Sorry for replying this way too late, but I don't see how this breaks the catch-all release detection just because VERSION_ID contains an epoch value that will change every upgrade.

I think the existing check matches distro="${NAME:-${DISTRIB_ID}} ${VERSION_ID:-${DISTRIB_RELEASE}}"from line 133 against the string Container Linux by CoreOS on line 3205, which results in Container Linux by CoreOS 1409.1.0 not matching Container Linux by CoreOS because of the presence of the epoch timestamp, in turn causing the ascii art to not be displayed.
Rather than adding the epoch timestamp to line 3205 (which would break every time CoreOS produced a new release), the proposed change matches on $NAME and uses $ID to be concise and have something that shouldn't change at each release.
Please let me know if that doesn't make sense and thanks for the reply!  :)

I don't think it needs to be changed.
The case statement in line 3205 goes like this: "Container Linux by CoreOS"*).
That means whenever $distro detects Container Linux by CoreOS 1409.1.0, when get_ascii does its job, it will check Container Linux by CoreOS and whatever comes after it (e.g. Container Linux by CoreOS 1409.1.0 and Container Linux by CoreOS 1408.1.0 will work just fine). It doesn't have to be exactly the same character.
This is what I got when I tested it with a temporary /etc/os-release for two different versions of CoreOS.

Cool - there must be something incorrect in my test instance - existing result was giving me this:

Given your test, please discard the PR.


protoc-gen-connect-go: 1.3.1 -> 1.4.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/bufbuild/connect-go/releases.
meta.description for protoc-gen-connect-go is: "library for building browser and gRPC-compatible HTTP APIs"
meta.homepage for protoc-gen-connect-go is: "https://github.com/bufbuild/connect-go"
meta.changelog for protoc-gen-connect-go is: "https://github.com/bufbuild/connect-go/releases/tag/v1.3.2"
Updates performed

Version update

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 1.3.2 with grep in /nix/store/z40nsych2k1cz76y8pc21vxa8qlnpxq1-protoc-gen-connect-go-1.3.2
found 1.3.2 in filename of file in /nix/store/z40nsych2k1cz76y8pc21vxa8qlnpxq1-protoc-gen-connect-go-1.3.2

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

First fifty rebuilds by attrpath
protoc-gen-connect-go

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/z40nsych2k1cz76y8pc21vxa8qlnpxq1-protoc-gen-connect-go-1.3.2 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A protoc-gen-connect-go https://github.com/r-ryantm/nixpkgs/archive/5c394a85eefce29225b1b034c2ad9a58505337be.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/z40nsych2k1cz76y8pc21vxa8qlnpxq1-protoc-gen-connect-go-1.3.2
ls -la /nix/store/z40nsych2k1cz76y8pc21vxa8qlnpxq1-protoc-gen-connect-go-1.3.2/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

protoc-gen-connect-go

Maintainer pings
cc @kilimnik for testing.

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/bufbuild/connect-go/releases.
meta.description for protoc-gen-connect-go is: "library for building browser and gRPC-compatible HTTP APIs"
meta.homepage for protoc-gen-connect-go is: "https://github.com/bufbuild/connect-go"
meta.changelog for protoc-gen-connect-go is: "https://github.com/bufbuild/connect-go/releases/tag/v1.4.0"
Updates performed

Version update

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 1.4.0 with grep in /nix/store/n1g3nalm9vij5qg0bm523wssxgwmrfgi-protoc-gen-connect-go-1.4.0
found 1.4.0 in filename of file in /nix/store/n1g3nalm9vij5qg0bm523wssxgwmrfgi-protoc-gen-connect-go-1.4.0

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

First fifty rebuilds by attrpath
protoc-gen-connect-go

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/n1g3nalm9vij5qg0bm523wssxgwmrfgi-protoc-gen-connect-go-1.4.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A protoc-gen-connect-go https://github.com/r-ryantm/nixpkgs/archive/157f606165a66771260e5e908b9f3b781b8c17f9.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/n1g3nalm9vij5qg0bm523wssxgwmrfgi-protoc-gen-connect-go-1.4.0
ls -la /nix/store/n1g3nalm9vij5qg0bm523wssxgwmrfgi-protoc-gen-connect-go-1.4.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

protoc-gen-connect-go

Maintainer pings
cc @kilimnik for testing.

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/bufbuild/connect-go/releases.
meta.description for protoc-gen-connect-go is: "library for building browser and gRPC-compatible HTTP APIs"
meta.homepage for protoc-gen-connect-go is: "https://github.com/bufbuild/connect-go"
meta.changelog for protoc-gen-connect-go is: "https://github.com/bufbuild/connect-go/releases/tag/v1.4.1"
Updates performed

Version update

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 1.4.1 with grep in /nix/store/6wb4yp93sw30ifq0y07z77pphk9ldr4h-protoc-gen-connect-go-1.4.1
found 1.4.1 in filename of file in /nix/store/6wb4yp93sw30ifq0y07z77pphk9ldr4h-protoc-gen-connect-go-1.4.1

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

First fifty rebuilds by attrpath
protoc-gen-connect-go

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/6wb4yp93sw30ifq0y07z77pphk9ldr4h-protoc-gen-connect-go-1.4.1 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A protoc-gen-connect-go https://github.com/r-ryantm/nixpkgs/archive/853fbd6e4e7416df924abb29aabd0a3fb9922009.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/6wb4yp93sw30ifq0y07z77pphk9ldr4h-protoc-gen-connect-go-1.4.1
ls -la /nix/store/6wb4yp93sw30ifq0y07z77pphk9ldr4h-protoc-gen-connect-go-1.4.1/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

protoc-gen-connect-go

Maintainer pings
cc @kilimnik for testing.


Fix transition validation and remove verified staged ledger

I would like to wait until @deepthiskumar has a chance to review the changes to the staged ledger before I merge. In particular, I am unsure if I implemented apply_diff_unchecked correctly.


Better PowerCharge menu wrapping (retry)

About the PR
See #33656

Fixed and slightly edited the PowerCharge menu. Now on the RU localization, no text will climb where it is not needed.

They've granted permissions (Russian channel):
https://discord.com/channels/310555209753690112/1278342990305300491/1320747429712363521
So I'll finish this.
Why / Balance
Looks better
Media
Before:

After:

Requirements

[X] I have read and am following the Pull Request and Changelog Guidelines.
[X] I have added media to this PR or it does not require an ingame showcase.

Breaking changes
Changelog
:cl: VlaDOS1408

fix: Fixed Gravity gen menu resizing! Now it looks like it should.

Why is the wattage under "charge" instead of power? That doesn't make sense.

Why is the wattage under "charge" instead of power? That doesn't make sense.

Should I then add a "Consumption" field?

Why is the wattage under "charge" instead of power? That doesn't make sense.

Should I then add a "Consumption" field?

Why can't it be left as-is?

Why can't it be left as-is?

Personally, I find it confusing. Is it consumption? Or is it capacity? You can't tell for sure on the first look.
Anyway, I corrected it.

Is it consumption? Or is it capacity?

Watts is a unit of power, not energy. It cannot be capacity by definition.


Mandatory option --cores is breaking peoples workflows

Also see #283
Snakemake version
5.13
Describe the bug
The option --cores is mandatory, which is inconsistent with the documentation printed by --help.
Minimal example
See #283
Desired behavior
Users should not be required to set mandatory options. Please pick reasonable defaults.
The default for --cores should be "all", like in ninja. This make snakemake run in parallel by default, which is what most users naively expect anyway. The need to restrict the number of cores only occurs when people run many things in parallel on the same computer, which is much less common.

I am seeing the same issue with version 5.13.0
$ snakemake
Error: you need to specify the maximum number of CPU cores to be used at the same time with --cores.

However, I disagree with @HDembinski 's statement

The default for --cores should be "all", like in ninja.

The default should always be 1 for any program. Programs which silently steal all threads have caused myself and other users many problems in the past. If I want more than one thread I should have to ask for it.

For what it's worth, I'd like to say that I agree with @mbhall88 that I think the old default of j=1 was most sensible. Automatically claiming all cores on the system (Linux is a shared computing environment after all) is dangerous, and adding new mandatory arguments to stable interfaces is problematic.
One issue I see here is that you can get away with some shortcuts when j=1, like having a temporary file called tempfile or printing results directly in the console. Once j>1 then the contents of tempfile and the console output can get scrambled. Now you could just wag your finger and say "write your rules properly!" "use shadow rules!" but here we're trying to help people who haven't even twigged that they need to use the --cores setting to get parallel execution. So I fear that the questions from people asking "why is my workflow not running in parallel" will now be replaced with more complex problems from people who don't understand that parallel execution is tricky, even with Snakemake to help. I'll bet that many people have workflows that work OK with one thread, and now they'll see that --cores is mandatory so they will give a number >1 and then the workflow will, as they see it, suddenly break.

It is difficult to predict how people use a tool, but why make a DAG of a workflow in the first place, unless you want to do as much as possible in parallel? snakemake imitated make, for which the default is 1 thread, but make only has this default, because make is ancient and was developed before multi-core computers were common.
I see the problem of changing the default. A reasonable way forward is to return to cores=1 in the next version and add a configuration flag for the Snakefile, where I can set core to 1,2,...,all on a per-project basis.

For a slightly different argument - I use snakemake all the time and it would just be annoying to have to type --cores N for every single invocation.  Please do not make this required - it feels like the kind of design decision I'd have to apologize for if recommending this tool to a colleague.  Or if it must be required, allow for some specification in the snakefile or config file to override this.

Piling on here...
A lot of people work in environments where they share servers with other people. A 1-core default is the safe behavior. If the user wants more cores than that, then they ought to make the decision consciously. It shouldn't be a required argument, either.
I don't see how people are confused by the 1-core default. If they bother looking at the documentation or --help for five minutes, they'll see the --cores option.
I love Snakemake and have really enjoyed using it. It's been a huge boost to my productivity. But, frankly, I'm flabbergasted by these design choices in the recent releases.

Defaulting to one core is the same behaviour as make. Please revert.

IMHO any default would be better than an explicit requirement to define it. Since the default was -j1 in the past, I also think a return to that would be ideal.

Let me cite myself:

People not coming from make frequently didn't anticipate the fact that it uses 1 core by default, wondering why it does not parallelize. Hence, we decided to make the specification of cores explicit. I know that it causes additional noise for simple cases, but not for the majority. There is always the possibility to define your own default via a profile.

I think it does not hurt to think about the amount of paralelization before starting a workflow.

For what it's worth, I'd like to say that I agree with @mbhall88 that I think the old default of j=1 was most sensible. Automatically claiming all cores on the system (Linux is a shared computing environment after all) is dangerous, and adding new mandatory arguments to stable interfaces is problematic.
One issue I see here is that you can get away with some shortcuts when j=1, like having a temporary file called tempfile or printing results directly in the console. Once j>1 then the contents of tempfile and the console output can get scrambled. Now you could just wag your finger and say "write your rules properly!" "use shadow rules!" but here we're trying to help people who haven't even twigged that they need to use the --cores setting to get parallel execution. So I fear that the questions from people asking "why is my workflow not running in parallel" will now be replaced with more complex problems from people who don't understand that parallel execution is tricky, even with Snakemake to help. I'll bet that many people have workflows that work OK with one thread, and now they'll see that --cores is mandatory so they will give a number >1 and then the workflow will, as they see it, suddenly break.

We can improve the error message, saying that --cores 1 is very well possible.

Ok, reopening the issue after having read all your comments.

I hear you, and can understand your feelings about it. I am unsure what to do now. Be assured that I made that decision after talking with several people that did not like the current behavior. So basically, we have two groups now, which are kind of unable to be satisfied at the same time.

Not sure yet though why exactly the new behavior hurts.

So, you say this makes people accidentally use paralelism. But why is that, they need to explicitly specify the number of cores.
Yes, you need to type a bit more, you can type -j1 for a single core, which is just three key presses. You can even define yourself an alias for single core snakemake.
I know that it is nice to be consistent with make, but one also has to consider that many users nowadays (like the new generation of scientists) don't know about make at all anymore. So the value of being consistent with make is less important than it was when Snakemake was new.

However, looking at the error message again, it indeed sounds like people should just use --cores (without arg). This is of course not what I meant. I want them to think about the number if necessary.

This is what I ended up using. Just in case anyone wants a shorthand solution/workaround for now.
# .bashrc
# snakemake use all cores by default
sm() {
    if [[ "$*" == *"-j"* || "$*" == *"--jobs"* || "$*" == *"--cores"* ]]; then
        snakemake $@
    else
        snakemake -j all $@
    fi
 }

I think it does not hurt to think about the amount of paralelization before starting a workflow.

Wouldn't the same be accomplished by a warning if the number of cores is not specified? I agree wholeheartedly with all the arguments above for default -j1.

We had a warning, but apparently people missed it.

The issue is that it gets quickly out of sight when stuff is executed.

I think this is one of those age-old divides in the computing community that is unsolvable. Some tools implicitly take all cores available, others use only 1 - some use something in-between. I don't know a lot of tools that require you to specify the number of jobs/cores, but as you say, making people have to think about it upfront isn't necessarily a bad thing.
The only thing I would say (and sorry to sound like a broken record) is it might be a good idea to bump the major version of snakemake. I know this change came in quite a few minor versions ago, but there have been other API-breaking changes. It does make it difficult when I jump between old and new workflows to remember which minor version is valid for the given project.

I think this is one of those age-old divides in the computing community that is unsolvable. Some tools implicitly take all cores available, others use only 1 - some use something in-between.

I disagree. Can anyone name a (widely used) tool that implicitly takes all cores?

it might be a good idea to bump the major version of snakemake.

I certainly agree, if the API has changed.

I disagree. Can anyone name a (widely used) tool that implicitly takes all cores?

E.g. Matlab is guilty of this. This default makes sense for users working on their own workstation if the workflow is compute heavy. In shared environments or for workflows heavy on IO or memory, single CPU is a safer bet.

ahem can anyone name a (widely used) tool that implicitly takes all cores, where that's a good design choice?

I mean for Matlab it is a good design choice imho -- its main use case is to run on a single-user workstation, using just one core would be wasteful. Of course if you try to run it on a cluster within a single core job, it will promptly get killed for using too many cores ;) Which happens often enough.

ahem can anyone name a (widely used) tool that implicitly takes all cores, where that's a good design choice?

My point was not that it is a good idea to take all cores, it was more to illustrate that whether it is a "good design choice" is really a matter of opinion (mine being 1 is ideal).
All of that aside, the requirement of the --cores parameter seems to be here to stay, so continuing discussions on what the default should be seems pointless. Given the points made by @johanneskoester in https://github.com/snakemake/snakemake/issues/308#issuecomment-648073021 I find it hard to provide a strong counter-argument to it staying required (provided there is a major version bump).

I find it hard to provide a strong counter-argument to it staying required (provided there is a major version bump).

I still think the users should be advised to start with -j,1 and only after everything is working correctly, try to get some speedup with parallelization. Race conditions https://github.com/snakemake/snakemake/issues/308#issuecomment-617692653 and performance issues https://github.com/snakemake/snakemake/issues/308#issuecomment-648481872 are, strong enough arguments, I think. Default value being a form of very clear advice (even for those who don't read --help or manual, which is apparently prevalent enough to warrant the change in the first place).

The question is do you want to be more like a traditional, well behaved Unix tool, which is quiet, runs as a single process by default and trusts users to read the manual to understand the full extent of its power, or more like monoliths like Matlab, the JVM, and Apache which think of the user's machine as more theirs than the user's. To me it's very obvious which school has good taste and which bad, but we are unfortunately in the territory of taste and this could be bikeshedded indefinitely.

The question is do you want to be more like a traditional, well behaved Unix tool, which is quiet, runs as a single process by default and trusts users to read the manual to understand the full extent of its power, or more like monoliths like Matlab, the JVM, and Apache which think of the user's machine as more theirs than the user's? To me it's very obvious which school has good taste and which bad, but we are unfortunately in the territory of taste and this could be bikeshedded indefinitely.

In the current version, Snakemake is not like that at all. It does not take all cores by default. It just requires people to specify the number they want.

A major version bump is maybe indeed a good idea. When implementing this change, I did not see the command line interface as an API. But I now acknowledge that Snakemake is used as a backend in some contexts, so it is indeed appropriate to make a major version bump. I will probably combine that with some other breaking changes that can be used to simplify the codebase (removing dynamic:, version:, and singularity: (the latter can be written as container: now, since the runtime is pretty arbitrary nowadays)).

First and foremost:  I think Snakemake is absolutely wonderful and easily the best tool of its kind that I've used.
I'm just so confused about that mandatory --cores design choice.  I think of it this way:

One core:  Safe and universal.  If you want more performance the #1 FAQ item clearly shows you what to do.  (Or it should.)
All cores:  Hmmmm.  Probably good under most circumstances but could be embarrassingly wrong in some cases.
No cores:  Print an error message and fall over dead on the floor.

One of these is less useful than the others.  :-)

ahem can anyone name a (widely used) tool that implicitly takes all cores, where that's a good design choice?

I think I mentioned ninja a few times already, which takes all cores by default. At it is a good design choice.

Well, what we can easily see from this discussion is that there is no consensus about what behavior is best. Hence, forcing a decision from the user by default seems like a reasonable choice to me.
And then, there are profiles, which allow you to configure the default behavior of Snakemake on your system. Moreover, I am currently crafting the ability to have a default profile in PR #467. So, you could simply have a profile .config/snakemake/default/config.yaml with
cores: 1

the behavior will be the same GNU Make again (1 core by default). Similarly, with
cores: all

in the same place, you get ninja behavior (all cores). And without a profile, Snakemake will just ask, as it is now.

OK, I'm convinced. And this reminds me - I should start replacing my collection of Snakemake wrapper shell scripts with profiles.

From reading this, I'd agree that there is no consensus, but I'd say that disagreement has mostly been about whether they default should be 1 or All cores.
However, my sense is that most people would prefer there to be a default rather than for this option to always need to be specified.
I think there is an elegant simplicity in typing snakemake <foo> and having that file be built.  And when extra options are required, we lose some of that.  I would suggest that the appeal of this simplicity has been a large contributor to the widespread adoption of this great tool.  I've given tutorials showing other bioinformaticians how to use snakemake and I'd prefer not to have to start having the conversation about 'what are cores' and parallelism when just introducing the Hello World example.
So even though I would prefer a default of 1 core, I strongly advocate for the use of any default over making this a required parameter.

I just don't buy the story that "there is no consensus". I see the majority of users on one side of this discussion.
As far as I can tell, the situation looks like this:

I've seen many users on this thread (and other threads) express a preference for the --cores 1 default. This includes me, @deto, @i-Zaak , @frankier , @mbhall88 , @danijoo , @tbooth , @jjarmagost. Most of us base our argument around overwhelmingly common conventions for command line tools.
In all the related threads, I've only seen two people advocate something else. In #283 they refer to "thousands of beginners" as a reason for a mandatory --cores argument. They also mention the ninja build system as an argument for default --cores all behavior.

Do we really want to let the design of Snakemake be driven by the experience of beginners, rather than well-established conventions?
Do we really want to let the design of Snakemake be driven by people who haven't read the (very good) online documentation? or the --help message?
Consensus matters, but consensus among the right group matters more.
All of that said -- I like Snakemake a lot and will continue using it regardless of the decision. @johanneskoester makes the very fair point that we can define our own profiles.

@deto, a default profile can even reside in the workdir, allowing the tutorial to simplify this away. I would personally say though that nobody should leave a snakemake tutorial without knowing about parallelization anyway.

@dpmerrell we don't have a representative picture. Indeed, here, more people don't like the change. Which is natural, because that is kind of the topic of the thread. My only point is that deciding about parallelization is a good idea before starting a workflow. Yes, using a single core by default is command line convention, and it is a good convention if there is a need for a default. If I would choose a new default for snakemake now, I would also choose a single core. But I don't think we need such a default. I cannot see anything bad in enforcing a decision about the cores from the user. And if the user has made his or her choice, he can easily define a default in his profile if he or she wants to type less (at least once the PR is merged).

Hey, brand new to snakemake.  As a brand new user / beginner having to specify --cores 1 each time is just tedious and frustrating. I would argue there really should be a default and ideally that default should be 1 to be conservative. The documentation and support for parallisation is amazing so I really don't see an issue with setting such a default.

It's impossible to determine if a default of "one" or "all" is most appropriate.  Everyone has different needs.  It would have been best just to leave it at "all" like it originally was, and to avoid this whole discussion.
I really love snakemake but this idea that "new users should really be forced to think about cores so the argument will become mandatory" reaches cosmic levels of silliness.

Maybe I'm being naive here but I honestly have no real understanding of why "all" would be a good default option. There is no guarantee that the machine is well configured and that using all cores won't drown out a server and prevent others from working or that it may spawn more processes than the user is allowed to use and result in throttling or job cancelations.
Rephrasing, "all" has edge cases which could be seriously problematic. "One" is never a problem it just might be a bit slower than expected (which is easily resolved after looking at the manual or help page for no less than 5 minutes). A default of 1 just seems intuitive, safe, well behaved and consistent with other similar tools. Still regardless of "one" or "all" having no default is just crazy.

Sorry final comment - With regards to the profile "solution" is it possible to have a system wide default as well as a user/workdir default so that we can configure this for all users in advance.  This is inline with @deto's comment of allowing us to have tutorials or get new users experiment with the basics without having to introduce complexities associated with parallelisation. Its never a good look having to say "you need to type -j1 but don't worry just ignore this for now"

You can find examples of preexisting, industry standard tools (that everyone is perfectly happy with) that fall into either category.  No matter how much you like "default one" and have good arguments for it, someone else likes "default all" and has good arguments for it.  It's an unwinnable argument.
The egregious offense is that the interface changed under some pretense of helping new users understand how computers work, and broke preexisting flows.

having no default is just crazy.

We agree completely on that part.
Disclaimer:  Despite my comments above, I think Snakemake is exceptionally well done and, for me, it's the best tool of its kind.  Kudos @johanneskoester!

@jjarmagost -  Genuine question (not trying to be antagonist am just genuinely curious for my own understanding/knowledge) what would be the arguments for a default of "all" and how do you mitigate issues relating to shared servers ?
Also echo your comments that this is fundamentally an amazing tool and hope that none of my critiques of the -j1 give any impression to the contrary.

@gowerc Some tools are mentioned by others previously in this thread.  My example of a tool that takes all cores by default (actually it takes "most" cores) will probably not be helpful to you because it has nothing to do with bioinformatics.  I'm using Snakemake in an FPGA build environment.
The compute-heavy steps in the flow take the approach of grabbing "most" of the cores.  For example, on my 10 core machine the tool usually takes 6 cores.  To be honest, I'm not sure if it is trying to be "polite" or if grabbing more cores wouldn't actually speed up the computations.  (There's usually a limit to how much parallelism can speed up a given compute task.)  Anyway, using a default of one core is absolutely, positively not what I want the tool to do.  The machine is my personal build machine and I'm not doing any other compute-heavy tasks while the build is running.  By default, I want it to grab as many cores as it wants.
As far as remote compute farms, if I ever expand my build flow to make use of them I would probably have to do some adjustments with respect to the number of cores/amount of resources, but I would not consider that as evidence that "default of one is best".

The issue is that it gets quickly out of sight when stuff is executed.

I think the issue here wasn't technical. If a user runs a program and expects it to automatically parallelize, does not look up in the options what a possible argument could be and ignores warnings, we do have a very different kind of problem! That's not the user which should be targeted in terms of required arguments, that's the user group which should be targeted with "read a tutorial and get some basic computing understanding first".
We have to be very careful and sometimes say simply "no" and maybe blame the user. In fact, a warning is already quite generous to me!
Definitely undo and restate the default. I also think one core is the most sensible. While I understand the argument for all cores, I think the usecases are:

you run it in parallel, efficient. Then you know that you gonna do that and need to take care of it: you think about which machine, is it shared, bottlenecks etc (for example, I often scale down the number of cores because of I/O heavy scripts (100 GBs of (ROOT) files). Since you think about it, it's not a problem to think about the number of cores.
you test things out or do not understand parallelism (the latter is dangerous!): in this case, the default 1 is perfect.

Actually, it gets especially ridiculous when running with -n (dry run) and it requires me to specify the number of cores: this is not a design choice but a design mistake, since the number of cores is ignored anyway.

@johanneskoester let's maybe get the facts speaking: how many users did ever complain that they expected it to run "automatically" in parallel, did not read the warnings that the code printed and also did not look up the docs?
We have in this thread here a consensus among probably more experienced developers: have a default. We have an adult discussion if one or all should be the default with many (including me) leaning towards 1. But even - so I read from the posts - "all" supporters would prefer a default.
So let's get some facts straight (and I say that because I love snakemake and find the "continued degredation" through questionable design choices just difficult to watch):

how many people really are surprised by a default of 1 core?
which usecase does it really serve? Like what's the user experience that they need this to be mandatory? I.e. that they cannot just see that it "doesn't parallelize" and then Google it and find the answer after 2 minutes? This 2 minutes that it "saves them" (and introduces potentially more problems in case of concurrent I/O), is that what the change is for?

I just really can't make up my mind why such a great tool with many good design decisions ends up incorporating such poor decisions too.

I concur that the number of cores should have a default value of 1. Anyone who needs extra cores will have the option to add more cores as required.

I was also surprised by this change due to a recent update.
Single-core is a sane default as it works whenever the multi-core doesn't.
Having to explicitly type -c1 is inconvenient - especially from the background that this change was done deliberately for no apparent benefit.
Make has single-core as its default, not because it's "ancient" but as it's tried and tested: If I want more performance and I'm positive that my Makefile works and enough resources are available, I can consciously specify -j. If I don't specify anything, single-core will be the slower but safer choice. That helps both experienced and non-experienced people. Also to me, that's the least surpising behavior (POLS!).
If people were surprised that snakemake did not parallelize properly, they clearly lack experience with command line tools. They should read --help, the man page acquire basic linux proficiency or thoroughly work through the tutorial. None of this is in particular related to snakemake. For me, snakemake (or automation, for that matter) is not a beginner's tool: IMHO, good design should be more important than convenience. Snakemake is already way easier than make (and that's a good thing), let's not water it down just for the sake of simplicity if there are no other good reasons.
Also, I don't really want people that don't know what they are doing to over-parallelize by default and crash shared compute servers - especially considering that a lot of bioinformatics tools are RAM-hungry.
Hence, ninja's default of #cores+2 might be alright for compilation - the problem is well studied and regular and the memory footprint of compilers is usually not that high when compared to general workflows that make and snakemake aim at. That doesn't automatically make #cores a sensible choice for snakemake, too.

@jjarmagost can you explain a bit further? This issue is open as the discussion is definitely not resolved: we have a clear consensus among people here, @Mitmischer just summarized again all the points brought up, yet it doesn't seem that any of the maintainers is seriously picking this up or bringing an argument.
As users, we rely on a package, build things around, and expect a reasonable development in good faith. This issue is questioning that.
So yes, this is rather an open wound than an open thread ;) Let's please leave it open for new people to see and for developers to be able to respond, either by providing a rational or by implementing the changes.

This whole conversation started two years ago because a breaking change was introduced (making the --cores argument mandatory) that broke flows.  This is generally considered a bad thing to do, especially when the justification is weak.  And in many of our opinions the justification was very weak.
What followed was a lengthy discussion about whether one or all cores is the correct default.  It became apparent pretty quickly that no consensus could be reached.  There are excellent arguments and examples on both sides.
Then the conversation effectively ended here back in 2022:  https://github.com/snakemake/snakemake/issues/308#issuecomment-1336430955
At that point, in my opinion, the topic should have been closed.  The only thing that's really clear (at least to me) is that introducing another breaking change that breaks flows would be really dumb.  There are no new compelling arguments.  Nothing has changed.
Just leave the behavior how it is.

This whole conversation started two years ago because a breaking change was introduced (making the --cores argument mandatory) that broke flows.  This is generally considered a bad thing to do, especially when the justification is weak.  And in many of our opinions the justification was very weak.
What followed was a lengthy discussion about whether one or all cores is the correct default.  It became apparent pretty quickly that no consensus could be reached.  There are excellent arguments and examples on both sides.
Then the conversation effectively ended here back in 2022:  https://github.com/snakemake/snakemake/issues/308#issuecomment-1336430955
At that point, in my opinion, the topic should have been closed.  The only thing that's really clear (at least to me) is that introducing another breaking change that breaks flows would be really dumb.  There are no new compelling arguments.  Nothing has changed.
Just leave the behavior how it is.

It is too late. That much I don't deny, however adding a new default would be backwards compatible. Also it's clear that there is consensus that either default would be better than no default. This is a classic case of a split vote. This shows how defective this approach to decision making is.

It became apparent pretty quickly that no consensus could be reached.

I would suggest you to reread the conversation: the consensus was pretty clear

a default is better that no default (unanimous!)
a large majority favored one core over all core

There is not a single (!) disagreement or argument (!!) against 1, and if you ask, "should there be 1 core as default or no default", everybody agrees.

introducing another breaking change that breaks flows would be really dumb

Can you explain how introducing a default of one core breaks even a single workflow?

It became apparent pretty quickly that no consensus could be reached.

I would suggest you to reread the conversation: the consensus was pretty clear

a default is better that no default (unanimous!)
a large majority favored one core over all core

There is not a single (!) disagreement or argument (!!) against 1, and if you ask, "should there be 1 core as default or no default", everybody agrees.

introducing another breaking change that breaks flows would be really dumb

Can you explain how introducing a default of one core breaks even a single workflow?
Why this thread is so important I am working in a large scientific collaboration (~1K scientists). We want to use workflow tools and make recommendations on which to build our analyses on (we're talking years of work, millions and millions of line of code). This issue is my number one argument, why we should not use snakemake: because it's not governed by rational arguments. No matter the consensus.

This whole conversation started two years ago because a breaking change was introduced (making the --cores argument mandatory) that broke flows.  This is generally considered a bad thing to do, especially when the justification is weak.  And in many of our opinions the justification was very weak.
What followed was a lengthy discussion about whether one or all cores is the correct default.  It became apparent pretty quickly that no consensus could be reached.  There are excellent arguments and examples on both sides.
Then the conversation effectively ended here back in 2022:  #308 (comment)
At that point, in my opinion, the topic should have been closed.  The only thing that's really clear (at least to me) is that introducing another breaking change that breaks flows would be really dumb.  There are no new compelling arguments.  Nothing has changed.
Just leave the behavior how it is.

It is too late. That much I don't deny, however adding a new default would be backwards compatible. Also it's clear that there is consensus that either default would be better than no default. This is a classic case of a split vote. This shows how defective this approach to decision making is.

Aha!!  I somehow missed the point that any changes would be backwards compatible.  That erases my entire argument.  Please ignore that part of my post.
However, I still don't think there is anything approaching consensus on the "one vs. all" debate.  Good luck with that...

It became apparent pretty quickly that no consensus could be reached.

I would suggest you to reread the conversation: the consensus was pretty clear

a default is better that no default (unanimous!)
a large majority favored one core over all core

There is not a single (!) disagreement or argument (!!) against 1, and if you ask, "should there be 1 core as default or no default", everybody agrees.

introducing another breaking change that breaks flows would be really dumb

Can you explain how introducing a default of one core breaks even a single workflow?
Why this thread is so important I am working in a large scientific collaboration (~1K scientists). We want to use workflow tools and make recommendations on which to build our analyses on (we're talking years of work, millions and millions of line of code). This issue is my number one argument, why we should not use snakemake: because it's not governed by rational arguments. No matter the consensus.

Saying that Snakemake is not governed by rational arguments is a bit extreme.  Even if the wrong default is chosen (in your opinion) how does that put your googolplex lines of code and gazillion hours of work at risk?  If that's really true then you have structured your project wrong.

This issue is my number one argument, why we should not use snakemake: because it's not governed by rational arguments. No matter the consensus.

Look, I'm on the side of "having a default makes sense" regardless of whether its 1 or All.  But from the thread it became clear that the developers preferred to make this explicit.
I can live with that.  I've accepted that in using free software developed by others there might be design decisions that I don't agree with that I just have to live with.  Overall snakemake has been an incredibly useful tool for me and works extremely well.  Clearly considerable thought has gone into this project and thousands of sensible design decisions have been made over the years.  It's unfair to characterize this tool as 'not being governed by rational arguments' over this one decision you don't agree with.
That being said, I'm sure the developers of this tool want feedback from their users, so I think commenting on things like this should be encouraged (even if the issue is 'closed').  Let's not blow things out of proportion and
be mindful that this is a passion project by people who are donating their free time and expertise.

This issue is my number one argument, why we should not use snakemake: because it's not governed by rational arguments. No matter the consensus.

Look, I'm on the side of "having a default makes sense" regardless of whether its 1 or All. But from the thread it became clear that the developers preferred to make this explicit.
I can live with that. I've accepted that in using free software developed by others there might be design decisions that I don't agree with that I just have to live with. Overall snakemake has been an incredibly useful tool for me and works extremely well. Clearly considerable thought has gone into this project and thousands of sensible design decisions have been made over the years. It's unfair to characterize this tool as 'not being governed by rational arguments' over this one decision you don't agree with.
That being said, I'm sure the developers of this tool want feedback from their users, so I think commenting on things like this should be encouraged (even if the issue is 'closed'). Let's just make sure to not blow things out of proportion and be mindful that this is a passion project by people who are donating their free time and expertise.

^^^This.

I do agree with you, and snakemake is an incredibly useful tool! However, when you think about building a lot of code around it, you want to double check how the project is handled.

I can live with that. I've accepted that in using free software developed by others there might be design decisions that I don't agree with that I just have to live with. Overall snakemake has been an incredibly useful tool for me and works extremely well. Clearly considerable thought has gone into this project and thousands of sensible design decisions have been made over the years. It's unfair to characterize this tool as 'not being governed by rational arguments' over this one decision you don't agree with.

It's not that this is the only thing that is a concerning design decision, but it's the easiest to argue about: it's not my personal preference but that of everyone else in this thread. Furthermore, no rational has been given, not a single real use-case. That's the difference. Individuals are failable, decisions are maybe bad. But this can be recognized. Or be ignored.
The issue is, what other design decisions will maybe come out of it? For example the rerunning behavior, which is now a lot more inconsistent than before (that updated files sometimes rerun the workflow, don't find the thread now).
It all adds up. Once may seems harmless, but it adds up. That being said, snakemake is great and design decisions can be made, but this here stands out as an extreme to so many other, well made choices. In the end, it influences where you want to build your code around.
But enough said, the thread is clear: everybody agrees that a default is better than none and yet, the decision was made to introduce a (code breaking!) mandatory argument, all because one user once was surprised that snakemake doesn't run in parallel.


Fork and use our nova kilo branch

This is needed to fix a nova <-> cinder interaction with availability
zones.

retest

retest

retest


django-jinja 1.3.1 + django-pipeline

Hello,
I'm just upgraded django-jinja 1.1.0 to django-jinja 1.3.1 and got some errors with pipeline extension:
Encountered unknown tag 'stylesheet'.

My settings for django-jinja:
DEFAULT_JINJA2_TEMPLATE_EXTENSION = '.jinja'
JINJA2_BYTECODE_CACHE_ENABLE = True
JINJA2_ENVIRONMENT_OPTIONS = {
    'trim_blocks': True,
    'autoescape': True
}

from django_jinja.base import DEFAULT_EXTENSIONS
JINJA2_EXTENSIONS = DEFAULT_EXTENSIONS + ['pipeline.jinja2.ext.PipelineExtension']

I've solved this issue by replace this settings as next:
DEFAULT_JINJA2_TEMPLATE_EXTENSION = '.jinja'
JINJA2_BYTECODE_CACHE_ENABLE = True
JINJA2_ENVIRONMENT_OPTIONS = {
    'trim_blocks': True,
    'autoescape': True,
    'extensions': ['pipeline.jinja2.ext.PipelineExtension']
}

Seems like there is a bug with JINJA2_EXTENSIONS setting.
Version of django-pipeline is 1.4.7

Thanks for reporting it! Seems clearly a bug.

This is mainly related to from django_jinja.base import DEFAULT_EXTENSIONS import, because it causes some circular imports. You should specify the explicit list of default extensions without importing it.

Thanks!


Evaluate butteraugli on TID-2013 and publish results

http://www.ponomarenko.info/tid2013.htm
Metrics are ranked by their Spearman correlation with Mean Opinion Score.
Someone who has hardware powerful enough to make it practical could do this. It would be a real proof (or maybe disproof) that this metric is superior to others.

I've done that with an earlier version of the librarry #15

oops, I somehow overlooked that…


Docs: Show how to compare two different traces in Tracy

Objective
Fixes #5199.
Solution
Mention how to compare two different saved tracy traces in the profiling section.

bors r+


[Bug]: CUDA out of memory for llama3.1 70gb gptq, while in llama3 70gb gptq doesn't

Your current environment

The output of `python collect_env.py`
PyTorch version: 2.4.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.4 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.30.2
Libc version: glibc-2.35

Python version: 3.10.14 (main, May  6 2024, 19:42:50) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-6.5.0-41-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.1.105
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA GeForce GT 1030
GPU 1: NVIDIA RTX A6000

Nvidia driver version: 535.183.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.1.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      43 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             64
On-line CPU(s) list:                0-63
Vendor ID:                          AuthenticAMD
Model name:                         AMD Ryzen Threadripper 3970X 32-Core Processor
CPU family:                         23
Model:                              49
Thread(s) per core:                 2
Core(s) per socket:                 32
Socket(s):                          1
Stepping:                           0
Frequency boost:                    enabled
CPU max MHz:                        3700.0000
CPU min MHz:                        2200.0000
BogoMIPS:                           7386.22
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sev sev_es
Virtualization:                     AMD-V
L1d cache:                          1 MiB (32 instances)
L1i cache:                          1 MiB (32 instances)
L2 cache:                           16 MiB (32 instances)
L3 cache:                           128 MiB (8 instances)
NUMA node(s):                       1
NUMA node0 CPU(s):                  0-63
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Mitigation; untrained return thunk; SMT enabled with STIBP protection
Vulnerability Spec rstack overflow: Mitigation; Safe RET
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

Versions of relevant libraries:
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.26.4
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] pyzmq==26.1.0
[pip3] sentence-transformers==3.0.1
[pip3] torch==2.4.0
[pip3] torchvision==0.19.0
[pip3] transformers==4.44.0
[pip3] triton==3.0.0
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] nvidia-nccl-cu12          2.20.5                   pypi_0    pypi
[conda] pyzmq                     26.1.0                   pypi_0    pypi
[conda] sentence-transformers     3.0.1                    pypi_0    pypi
[conda] torch                     2.4.0                    pypi_0    pypi
[conda] torchvision               0.19.0                   pypi_0    pypi
[conda] transformers              4.44.0                   pypi_0    pypi
[conda] triton                    3.0.0                    pypi_0    pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.5.4
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    GPU1    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      SYS     0-63    0               N/A
GPU1    SYS      X      0-63    0               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

🐛 Describe the bug
Trying to load LLAMA3.1 70B GPTQ and get cuda out of memory on A6000 48GB, when LLAMA3 70B GPTQ is working great. They both have 39.8gb  safetensors, and 37.096gb when loading into memory, almost identical sizes. Tried a range of values of gpu_memory_utilization from 0.1 to 0.9 (I knew it won't change anything, as the regular info message of 'try reduce gpu utilization..' hasn't been shown, which means it didn't get to this stage, while in llama3 I am using 0.9 and it works great.
I avoided using marlin kernels, to avoid additional problems with the tests.
This is the llama3.1 70 gptq one
https://huggingface.co/shuyuej/Meta-Llama-3.1-70B-Instruct-GPTQ/tree/main
This is the llama3 70 gptq one
https://huggingface.co/TechxGenus/Meta-Llama-3-70B-Instruct-GPTQ/tree/main
import os
from vllm import LLM, SamplingParams

os.environ["CUDA_VISIBLE_DEVICES"] = "0" # to be sure using only A6000 and not GT1030

prompts = [
    "Hello, my name is"
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

# llm = LLM(model="shuyuej/Meta-Llama-3.1-70B-Instruct-GPTQ", gpu_memory_utilization=0.9, dtype='float16', quantization='gptq') # does not load
 llm = LLM(model="TechxGenus/Meta-Llama-3-70B-Instruct-GPTQ", gpu_memory_utilization=0.9, dtype='float16', quantization='gptq') # works great
outputs = llm.generate(prompts, sampling_params)

# Print the outputs.
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

getting CUDA out of memory
CUDA out of memory. Tried to allocate 2.00 GiB. GPU 0 has a total capacity of 47.54 GiB of which 1.58 GiB is free. Including non-PyTorch memory, this process has 45.94 GiB memory in use. Of the allocated memory 45.61 GiB is allocated by PyTorch, and 6.10 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

@orellavie1212 Llama 3.1 has a much larger context length than Llama 3, it increased from 8192 to 131072. You probably don't have enough space for the 128k length, so please set max_model_len to a context length that is smaller and usable for you.

@orellavie1212 Llama 3.1 has a much larger context length than Llama 3, it increased from 8192 to 131072. You probably don't have enough space for the 128k length, so please set max_model_len to a context length that is smaller and usable for you.

lol, forgot about that, you were right
thanks, working


Docs: New Data Masking utitlity

What were you searching in the docs?
How to use the new Data Masking utility
Is this related to an existing documentation section?
No response
How can we improve?
Creating the documentation
Got a suggestion in mind?
No response
Acknowledgment

[X] I understand the final update might be different from my proposed suggestion, or refused.

Python layer:
Compressed:
Current size: 5MB
Building with aws-encryption-sdk + cryptography: 9-10MB
Uncompressed
Current size: 12MB
Building with aws-encryption-sdk + cryptography: ~ 29MB
We have little difference in ColdStart using the layer with and without aws-encryption-sdk, just few ms, even importing the AWSEncryptionSDKProvider + DataMasking. But of course, the first execution of encrypt or decrypt takes 3x times than the second using the same Lambda. Most of this time is calling KMS + Creating cache + caching python libraries.
Executions:
First encrypt execution (coldstart) - 128MB - 3.5s
Second and subsequent encryption executions (warm) - 128MB - 1.0s (edited)
First encrypt execution (coldstart) - 1024MB - 450ms
Second and subsequent encryption executions (warm) - 1024MB - 140ms (edited)


Add write method to python WriteToBigQuery display data

Add write method to python WriteToBigQuery display data
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

we spoke about this offline. Display data is not well supported on composites. We can merge this but it won't have an effect for now. Is that okay?


Missing type annotations for AsyncLocalStorage?

VSCode (& presumably tsc) complains with:
Cannot find module 'node:async_hooks' or its corresponding type declarations.ts(2307)

I have the compatibility flag set in wrangler.
I don't see any mention of AsyncLocalStorage in node_modules/@cloudflare/workers-types. Is the expectation that I just add @types/node to my tsconfig.json? If so, that could use some documentation on https://developers.cloudflare.com/workers/platform/nodejs-compatibility/ or somewhere in the docs.

node:events is also missing
Cannot find module 'node:events' or its corresponding type declarations.


chore: add c++ autoformatter

The CI failure for rustfmt is expected as all the big changes aren't in tree yet, so it should be failing.

You should either make a new CI workflow or rename "Rustfmt" to just "Check formatting"

Yet again rebase merge, cuz it makes more sense as both commits stand on their own


softethervpn-android ，I really want everyone to develop and develop.

“github.com/lfasmpao/softethervpn-android”
“https://play.google.com/store/apps/details?id=io.github.lfasmpao.softethervpn&hl=zh”
At present, some enthusiasts have developed a test version, but they must have ROOT permissions. In the past, he also released the source code, but unfortunately the new mobile phone can not make ROOT permissions, can not use his test client, I hope to have a team development The client that comes out of the mobile phone link, don't try to use the protocol that comes with the phone to link to the server, because this is completely impossible in some countries, such as China.
The benefit of this is that it can increase the range of customers using softether, and maybe this project is really difficult. But someone has tried it, but I think this is a way that software must go. There must be a dedicated Android mobile client.

This software is licensed open source, so they removed the digital signature part. As for how to increase the digital signature, you have to ask SOFTETHER, the version they released is signed, and the open source is not signed.

Superseded by #1019.


Activate and Deactive methods work. But activate_user and deactivate_user do not.

The deactivate and activate methods work well. The deactivate_user method does not, as shown above!

This is expected (and also a duplicate report).  @reneklacan explained the problem well in the original report https://github.com/fetlife/rollout/issues/87#issuecomment-237365901


Improve the VB.Net Scripting Facility

The vMix VB.Net scripting facility is the only possibility to extend vMix from itself. Unfortunately, it has many limitations (see "Issue References"). It should be improved by adding script autostarting, add script arguments, add functions, provide full input state/information, allow meta-information be attached to inputs, provide more global information exchange variables,
Issue Reference: #10 #12 #18 #19 #20 #21
Vendor Response: "The Scripting feature in vMix is designed to support simple scripts and execute simple functions in vMix, it is not meant to accommodate advanced scripts or highly sophisticated and/or specialized usages."

Vendor Response: "The Scripting feature in vMix is designed to support simple scripts and execute simple functions in vMix, it is not meant to accommodate advanced scripts or highly sophisticated and/or specialized usages."

I'm only addressing the request to pass a parameter to a script via the web api.
Adding the ability to pass a parameter to a script is not particular sophisticated; it is the basis of the web API. More importantly, it reduces the number of shortcuts that must be maintained in vMix. I send MIDI commands to utilize shortcuts so that an action may be taken upon selecting an input. Each MIDI command triggers two shortcuts; the first sets a dynamic value (essentially the input) and the next calls the script that requires the value of the input. I'm not even sure if multiple shortcuts using the same MIDI sequence is supported, or if it's just luck that the shortcuts are triggered sequentially.
I'm using many PTZ presets in vMix, so I need to set up and maintain 81 shortcuts, which requires a lot of effort. If I could pass a single parameter to a script, I could reduce my shortcuts to 2. It would also allows me to utilize the web API vs MIDI, which is far more clear and maintainable.
In my case, the application calling the web API does not have the ability to make multiple API calls in a single action. Even if it could, a vMix script is the proper place for this to reside, because access to vMix states is available there. At this point, I can only use MIDI and a ton of shortcuts to meet my need.  I see no way to replicate this using the API functionality, but perhaps, I am overlooking functionality that would make this possible. My current option is to build middle-ware, which seems quite excessive to achieve what would otherwise be a very simple task.
From my perspective, allowing a parameter to be passed maintains the goal of keeping vmix scripting simple by eliminating the need for more advanced programming.


admin panel based on gentelella UI

Added admin UI based on gentelella
Currently supports basic functionality of editing / adding objects to Camps / Users using an extensible framework which allows to easily add admin pages for more models / modify the behaviors
removing the gentelella dependency causes ~3000 files to be deleted, you can only the relevant changes in the preview PR: https://github.com/OriHoch/Spark/pull/2
fixes #36

:ok: merged after linting.


Move to using interface for better mocking to graphql server

Proposed changes
Following up on the last PR, I've converted all packages to the interface pattern for better mocking in graphql server. I made this PR based on this layered architecture.
Issue No. #3931
Prev PR. #3933
Types of changes
What types of changes does your code introduce to Litmus? Put an x in the boxes that apply

[ ] New feature (non-breaking change which adds functionality)
[x] Bugfix (non-breaking change which fixes an issue)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation Update (if none of the other choices applies)

Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

[x] I have read the CONTRIBUTING doc
[x] I have signed the commit for DCO to be passed.
[x] Lint and unit tests pass locally with my changes
[x] I have added tests that prove my fix is effective or that my feature works (if appropriate)
[x] I have added necessary documentation (if appropriate)

Dependency

Please add the links to the dependent PR need to be merged before this (if any).

Special notes for your reviewer: @S-ayanide @amityt

I fixed all codes using interface. So no more global variable mongo operator now 😄

A is current Changes, and B is my proposals.
I made this PR based on [this layered architecture](https://github.com/bxcodec/go-clean-arch).
For example in the chaoshub package, As of now A, When I make a unit test in the ChaosHubService(service layer), I cannot create ChaosHubOperator Mock(repository layer) because ChaosHubOperator is not an interface but a struct. So, For better testing, I propose B. I want to know your opinion. @imrajdas @gdsoumya @amityt @S-ayanide

@namkyu1999 currently the A approach needs you to mock the dboperator instead of the service specific operator so you mock every db method that exists. Imo that should work fine if we can create a generic in-memory mock impl of the db but if that's too complex we can go with B but imo we should avoid going with B if we can as the service specific operator might still have some business logic that we need to test. Also if we go with B then there's no point in having a db interface in the first place as we can just always mock the service specific operator.

@gdsoumya Thanks for your reply. I agree with your opinion. So No more architectural changes here. And I will make the unit test code in the ChaosHub package this week and raise a different PR.

Here's my sample unit test code PR
#3946

Hi @namkyu1999, can we update the following terminologies in all the logs :
cluster/agent -> chaos infrastructure
workflow -> chaos experiment
experiment -> chaos faults

Hi @namkyu1999, can we update the following terminologies in all the logs : cluster/agent -> chaos infrastructure workflow -> chaos experiment experiment -> chaos faults

@Saranya-jena,
Do I just need to change the terminology in the logs, or do I need to change the terminology in the entire codebase, including package names? And whatever it is, it doesn't seem to be very relevant to this PR, so is it okay to create a new issue and raise a PR?

Hi @namkyu1999, can we update the following terminologies in all the logs : cluster/agent -> chaos infrastructure workflow -> chaos experiment experiment -> chaos faults

@Saranya-jena, Do I just need to change the terminology in the logs, or do I need to change the terminology in the entire codebase, including package names? And whatever it is, it doesn't seem to be very relevant to this PR, so is it okay to create a new issue and raise a PR?

You can only change the logs as I am making the changes in the codebase(currently in a separate branch). Yes you can raise a separate PR for this, posted here to track this.


Optimize the code to avoid useless casts to swift arrays.

The swift Array and Dictionary are transparently bridged to and from Objective-C NSArray and NSDictionary. Unfortunately, this bridge is not really toll free. As explained into the documentation, it involves a cost that often is very high. This pull request remove some of the costs associated to this conversion.
The base idea is that in most of the cases, the data hold by JSON instances is produced through parsing and therefore the most of the embedded data is NSArray or NSDictionary instances. This means that most of the casts to [AnyObject] or [String:AnyObject] are expensive and useless. Since the code producing the data is completely beyond the control of developers (unless Apple rewrites NSData, etc. in Swift), for those concerned by the speed, the only possibility to reduce this cost is avoiding the casts. In our case, using the changes proposed by this pull request we noticed huge performance gains, the time spent during get methods being reduced by more than 95%.
Of course, if the data is constructed using pure Swift objects (for example for serialization), the casts are performed in the opposite direction. Since the creation of these objects is under complete control of the developer, if the data is created from the beginning using NSArray or NSDictionary, these potential performance problems are largely diminished. Also, the effective serialization will be much faster because no superfluous conversions will be performed.
This pull request also disables a wrong test case that is checking a value against the 1st entry into the allKeys array produced by a NSDictionary. The documentation clearly states that the order of these keys is not defined, so checking the value at 1st index doesn't really make sense.

:+1:

JSON's object is [String : AnyObject] not  NSDictionary.

There is a misunderstanding here. As I said into the pull request, the object that is received from cocoa is a NSDictionary. Systematically casting it to [String : AnyObject] is very expensive and useless. Please take a deeper look at the patch or even better, profile the code to better understand the implications.
Your code is great, but it has this speed penalty. It would be a pity to leave it in this state.


Copy functionality causes build failure on iOS

Because of the NSPasteboard usage the package cannot be used in the iOS environment. This section (roughly) should be in the #if os(macOS) block or something like that.

Fix is done in v3.05


Added an overview of Pipelines interfaces

Created a page introducing the fact that there are various Pipelines interfaces: SDK, UI, and API. The page gives an overview of when you'd use each interface, and links to other guides for further info.
Fixes https://github.com/kubeflow/website/issues/574

This change is

Preview: https://deploy-preview-829--competent-brattain-de2d6d.netlify.com/docs/pipelines/overview/interfaces/

/assign @IronPan

Thanks for the review @IronPan - PTAL.

/lgtm
/approve


Storage fail safe

When storage account is not valid, instead of returning an error the storage panics.
This PR returns an Base64Decode error instead.

Nice! Thank you!


Uppercase extension support and .oga support

Video and audio players weren't present when the original filename had an uppercase extension. This PR fixes this and adds media players for .oga files

Released as v1.2.1. Thank you.


Use existential any in generated code

Motivation
The generated code doesn't use existential any which disables the user targets of this package to enable the .enableUpcomingFeature("ExistentialAny") swift settings.
Modifications
Always use existential any in the generated code. An exhaustive list of the protocols I found which now use existential any:

any Decoder
any Encoder
any ClientTransport
any ServerTransport
any ClientMiddleware
any ServerMiddleware

Result
Both compatibility with users who wish to enable the ExistentialAny upcoming-feature flag on their targets, as well as being more ready for Swift 6.
Test Plan
Modified the tests to reflect this change.
Considerations

Moving to existential any should not break any user codes.
I did not include the some keyword as that might be removed for Swift 6 and become the default when no related keywords are used, from what i've read/remember in the forums.

@swift-server-bot add to allowlist

Hi @MahdiBM, thanks for helping us prepare for Swift 6!
I'm generally in favor of this change, and I'd actually like it done even more holistically:

first, can you confirm that this feature is based on an accepted Swift Evolution proposal, to make sure it's not something that might still get backed out (as that'd cause a breaking change in this project)?
can you confirm what the lowest supported version of Swift allows explicitly specifying any for existentials?
if this explicit opt-in to existential any can be set in Package.swift, could you actually enable it in the integration test package as well, to verify that the generated code actually compiles with this option?
regarding the implementation, since the existential any could appear more than in just function parameters, but also properties, I think the any  part should be thought of as part of the type name (as wrapping it happens as Wrap<any Foo> rather than any Wrap<Foo>, right?). That way, we don't need the extra property in the structured Swift representation, we just change e.g. "Encoder" to "any Encoder" when providing the type to generate.

Thanks!

Generally, as i understand, the "upcoming feature" flags are all accepted and confirmed for Swift 6.
There are another type of flags called "experimental" which are not confirmed.

The proposal: https://github.com/apple/swift-evolution/blob/main/proposals/0335-existential-any.md
👍

The proposal mentions Swift 5.6 as the lowest version.

Great, so we can adopt it right away, as we require 5.8.

We could use it in the whole generator as well, so the code of the generator itself is forced to use existential anys. That's not 100% related to the generated code though so i will still need to look into enabling the flag for integration tests as well.

I don't want to feature-creep your PR too much, but if you try to enable it across the board in the project, even for the generator code itself, as it doesn't require too many changes, let's do it all here. If it requires any non-trivial changes, we'll break it out into a separate issue, and just focus this PR on adding any to generated code.

I agree. Does that mean i should create a new "TypeDescription" struct instead and use that instead of just a simple string for type names in  ParameterDescription?
In that case should i move everything to this new TypeDescription, or should i just use it where it's needed for now?

Let's just use a simple string for now. After re-reading SE-335, the any keyword is very much part of the type name, so I don't think we need to break it out just yet. If we need to, we can always do it in the future, but let's do the simplest thing that gets the job done for now.

it doesn't require too many changes, let's do it all here

Yeah it's not a problem

Let's just use a simple string for now.

👍

@czechboy0 I can add the upcoming flag to the swift build command, but i think you should just add it to the swift settings of targets of the test package in Package.swift (like in this PR)... Do you want to do that or should i just add it to the swift build command?

I think Package.swift is the right place. Would you also be open to adding this to swift-openapi-runtime, and possibly swift-openapi-urlsession and swift-openapi-async-http-client? Feel free to say no, and I'll file it as a separate issue, I just like the change and would like it to have across the board 🙂

Sure that's no problem 🙂

@czechboy0
After noticing the docker compose files i see i can add the "existential any" flag to those. The other way, as i mentioned right above, is to add the flag to the that test-project the CI seems to be pulling which i don't think is public? (please correct me if i'm wrong)
I think adding the flag to CI might be better but i'm not sure at all so feel free to disagree.
One question, is this PR waiting on me adding existential-any to the other projects as well? Because i don't think this PR should need any of those for it to be merged (it doesn't make any of those fail to build/compile or anything. It's completely "backward compatible" as we're already using it with no-existential-any openapi-runtime and async-http-client-generator)

Thanks, @MahdiBM! 👏


Version Detection is not working

Hi I have put configuration as below:
required:{e:0,f:0,o:0,s:0,c:0},
According to API, alert should appear except latest version.
but for chrome and firefox does not appear.
Can you please help me here.

This should work. Only the version before the latest is also not notified
for some weeks as a grace period. Otherwise a lot of people would be
notified because the automatic updates are usually rolling out slowly.
Am Fr., 27. Juli 2018 um 17:18 Uhr schrieb dev2214<EMAIL_ADDRESS>
:

Hi I have put configuration as below:
required:{e:0,f:0,o:0,s:0,c:0},
According to API, alert should appear except latest version.
but for chrome and firefox does not appear.
Can you please help me here.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/browser-update/browser-update/issues/408, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AIErKjEbTgTDKXGy3WVwoqIj_To1w9-cks5uKy9IgaJpZM4Vjrev
.

Hi
I have tried with latest version but still no luck.
I noticed that when we keep 0 it avoids and it takes what is in JS file.
one more issue i found that after appearing alert box if i do refresh the browser it calls the ignore action and 2nd time the box does not appear.
My configuration is as below.
var $buoop = {required:{e:-0,f:0,o:0,s:0,c:0},insecure:true,api:2018.08,
reminderClosed:336,
text:{
msg:'Note: You are using a browser designed for older applications. This site works best with current versions of Chrome, Edge, Firefox, or Safari.',
bignore: 'Ignore',
bupdate: '',
msgmore:'',
remind: 'You will be reminded in {days} days.',
bnever: 'Never Show Again'
} };
function $buo_f(){
var e = document.createElement("script");
e.src = "//browser-update.org/update.min.js";
document.body.appendChild(e);
};
try {document.addEventListener("DOMContentLoaded", $buo_f,false)}
catch(e){window.attachEvent("onload", $buo_f)}

I am pretty much never seeing the banner unless I go 10 versions back in Firefox... never seen it in IE unless I put test: true – how is the latest version even calculated?


add video file-loader

Codecov Report

Merging #1224 into dev will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##              dev    #1224   +/-   ##
=======================================
  Coverage   98.39%   98.39%
=======================================
  Files          12       12
  Lines         684      684
=======================================
  Hits          673      673
  Misses         11       11

Impacted Files
Coverage Δ

lib/builder/webpack/base.config.js
80% <ø> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4c0db59...9f5886c. Read the comment docs.


Support instance names

Currently the support for instance names is somewhat incomplete. At the very least we need to:

persist in the database
populate the openstack compute api when getting the instances
support setting the name on the cli
show the name in the cli
support setting the name when creating instances (controller/API)
handle clashes - names should be tenant unique
create a useful name if user does not supply one

Do we need to enforce "handle clashes - names should be tenant unique"?  I don't feel like that's a requirement enforced in other clouds and it would would annoy me as a user but I'm also quite possibly not
a typical user.

So I want to enforce that they are tenant unique so that you can reliably use them as DNS entries. To me it makes sense than the tenancy is a single DNS namespace. In the future we might add some kind of instance grouping that would allow us to relax this requirement.
Also I want to be able to say ciao-cli instance delete  and allowing clashes would increase the complexity of that somewhat. In my patch set if you request multiple instances with the same name i.e ciao-cli instance add -name foobar --instances 100 .... then the instances will be created as foobar-0, foobar-1,....foobar-99

OK the DNS rationale makes a lot of sense.
What do you think of doing like AWS with name-XXXXXXXX eg: "ubuntuvm-51f7ced1"?  The fixed length is easier on the eyes in cli and we views and an ordering 0..N will get random probably over time as some long used instances hang around and other shorter lived ones come and go.