Scratchpad like functionality for single window

Just some thoughts, design not final..
It would be nice to be able to bring  a specific window to the front of the currently active space, do something with the window, and just get rid of it when done, all with a single bind.
This would likely be set up using window rules, assigning a unique identifier, and optionally its desired size. Windows that are given such an identifier would automatically be marked as floating, and hidden in some way.
It would then be possible to just use a window --toggle identifier to toggle visibility. Syntax might look like:
Rule property: scratchpad=<LABEL>.
Window command: yabai -m window --toggle <LABEL>
# set identifier, set frame, and hide window
yabai -m rule --add app="kitty" title="dropdown" scratchpad="terminal" grid="6:6:1:1:4:4"

# show window on active space and make it focused, or hide if already visible
yabai -m window --toggle terminal

Maybe some visual effects would be nice:
yabai -m config window_scratchpad_transition off|fade|slide|dropdown

This would be a pretty awesome addition. +1
While I've got something working with yabai and hammerspoon [1] (calling quake-mode.sh WezTerm via a keybinding), a "native" solution would definitely be preferred.
[1] https://github.com/matthewtraughber/dots-etc/blob/master/yabai/quake-mode.sh

# mark window as scratchpad using rule and set size (scratchpad windows are manage=off automatically)
yabai -m rule --add app="^Spotify$" scratchpad=spotify grid=1:1:0:0:1:1

# add window to scratchpad manually (manage=off automatically)
yabai -m window --scratchpad <LABEL>

# remove window from scratchpad
yabai -m window --scratchpad

# toggle scratchpad window visibility
cmd + alt - s : yabai -m window --toggle spotify

Window commands

yabai -m window [<WINDOW_SEL>] --toggle float|sticky|pip|shadow|split|zoom-parent|zoom-fullscreen|native-fullscreen|expose|

Toggle the given property of the selected window.
The following properties require System Integrity Protection to be partially disabled: sticky, pip, shadow, LABEL (scratchpad identifier) .

yabai -m window [<WINDOW_SEL>] --scratchpad []

Unique identifier used to identify a window scratchpad.
An identifier may only be assigned to a single window at any given time.
A scratchpad window will automatically be treated as a (manage=off) floating window.
If the scratchpad is already taken by another window, this assignment will fail.
If the scratchpad is re-assigned, the previous identifier will become available.
If no value is given, the window will seize to be a scratchpad window.
System Integrity Protection must be partially disabled.

Rule property

scratchpad=

Unique identifier used to identify a window scratchpad.
An identifier may only be assigned to a single window at any given time.
A scratchpad window will automatically be treated as a (manage=off) floating window.
If this rule matches multiple windows, only the first window that matched will be assigned this scratchpad identifier.
System Integrity Protection must be partially disabled.

Haha, that was fast. I haven't built yabai from source yet, so it'll be a little bit before I can test this out.
Was recently messing around with hyprland's special workspaces (scratchpads) - any technical reason why we couldn't assign multiple windows to a yabai "scratchpad"?

I also use hyprland special workspace on my desktop machines and I love it. I never really put more than one window in it, while it could be handy I don't find it a necessity and I don't even think it would fit the idea of it.
Taking the chance to tank you for the amazing work you've been doing with yabai recently, it's almost perfect from a feature perspective.

Cheers. I think it is pretty solid right now, but there are still a thousand things I'd like to do still. Unfortunately there is only so much time I can allocate for this project.
Regarding multiple windows per scratchpad; it is possible to toggle multiple scratchpads simultaneously, giving the illusion that there are multiple windows: yabai -m window --toggle spotify --toggle discord.

This is super rad - I too have a uberjank hammerspoon/yabai mashup making scratchpads work, looking forward to trying this out.
You've been on a mission recently, massive thanks from me as well!

Might as well use rules to assign multiple windows to the same macOS space at that point and label the space instead. Use the 'space --switch ..' command to activate that space.

Yeah, makes sense.

Regarding multiple windows per scratchpad; it is possible to toggle multiple scratchpads simultaneously, giving the illusion that there are multiple windows: yabai -m window --toggle spotify --toggle discord.
They are in essence just floating windows, so can be moved and resized using window --grid, window --move, window --resize at any time.

Oh nice!
Gonna reiterate what everyone's been saying here - massive thanks for everything you're doing (and have already done). I would have left macOS years ago if I hadn't found yabai.

I agree with @matthewtraughber, thanks so much. If yabai doesn't start up even for 10 minutes, I feel like my laptop is a completely different, almost unusable computer.
Scratchpads are also integral to my workflow from i3 so I've been using @arpandaze 's very nice  https://github.com/arpandaze/scratchpad-rs until seeing this. It could be an alternative if someone wants to customize a 3rd party software instead - I found the code very readable and hackable as needed (rust).
Thanks!

Question about this change (and other new changes in general): since my memory only stores things for 3 seconds 😆 , I was wondering if the docs are updated to reflect these new features.
For example, I just check the wiki (the window commands section); I tried searching for scratchpad, and I didn't find anything

Wiki is only a small sample of available commands. I don't have the bandwidth to keep that much details up to date. Use the man page for full docs: (which is always up to date for head and you can check the same file for the version that that you are interested in): https://github.com/koekeishiya/yabai/blob/master/doc/yabai.asciidoc

Nice! Thank you for taking the time to reply. That's all I wanted to know.
Thank you for the amazing work you're doing

@koekeishiya thank you for this feature!! I stumbled into a small problem I hope you can help me figure out. Whenever I use the script (via Raycast) to toggle let's say Messages scratchpad:
yabai -m window --toggle Messages || open -a Messages
and if Messages isn't running at the time, the following happens:

Yabai reports unknown value 'Messages' given to command '--toggle' for domain 'window'
App launches, window shows floating for split second and disappears.

If I trigger the scratchpad again, it shows up and works normally until I quit the app. I went through my yabairc a million times, looking for something that might cause this and I have no idea where to go from here. Is there something wrong with my setup / syntax or…?
Thanks a ton!
yabai -V output:

 yabai -V
process_create: xpc service 'ViewBridgeAuxiliary' detected! ignoring..
process_create: xpc service 'CursorUIViewService' detected! ignoring..
process_create: xpc service 'Mail Networking' detected! ignoring..
process_create: xpc service 'Mail Web Content' detected! ignoring..
process_create: xpc service 'Mail Graphics and Media' detected! ignoring..
process_create: xpc service 'Mail Web Content' detected! ignoring..
process_create: xpc service 'Mail Web Content' detected! ignoring..
process_manager_add_running_processes: Finder (78439) was found! caching psn..
process_create: xpc service 'Adobe Content Synchronizer Finder Extension' detected! ignoring..
process_create: xpc service 'ShareSheetUI (Finder)' detected! ignoring..
process_create: xpc service 'Mail Web Content' detected! ignoring..
process_create: xpc service 'Mail Web Content' detected! ignoring..
process_create: xpc service 'Mail Web Content' detected! ignoring..
process_create: xpc service 'Mail Web Content' detected! ignoring..
process_create: xpc service 'Open and Save Panel Service (Adobe After Effects)' detected! ignoring..
process_create: xpc service 'QuickLookUIService (Open and Save Panel Service (Adobe After Effects))' detected! ignoring..
process_create: xpc service 'Dock Extra' detected! ignoring..
process_create: xpc service 'Dock Extra' detected! ignoring..
process_create: xpc service 'nsattributedstringagent' detected! ignoring..
process_create: xpc service 'com.apple.CoreSimulator.CoreSimulatorService' detected! ignoring..
process_create: xpc service 'Simulator' detected! ignoring..
process_create: xpc service 'ThumbnailExtension_macOS' detected! ignoring..
process_create: xpc service 'ExternalQuickLookSatellite-x86_64' detected! ignoring..
window_manager_add_existing_application_windows: Finder has windows that are not yet resolved
window_manager_create_and_add_window:63706 iTerm2 - -zsh (AXWindow:AXStandardWindow:1)
window_manager_begin: AppLocker (1513) is not observable, subscribing to activationPolicy changes
window_manager_begin: Spotlight (745) is not observable, subscribing to activationPolicy changes
window_manager_add_existing_application_windows: Adobe After Effects has windows that are not yet resolved
window_manager_begin: nbagent (87232) is not observable, subscribing to activationPolicy changes
window_manager_begin: calaccessd (2145) is not observable, subscribing to activationPolicy changes
window_manager_begin: CoreServicesUIAgent (75356) is not observable, subscribing to activationPolicy changes
window_manager_begin: Raycast (82796) is not observable, subscribing to activationPolicy changes
window_manager_begin: Universal Control (1327) is not observable, subscribing to activationPolicy changes
window_manager_begin: Keychain Circle Notification (7264) is not observable, subscribing to activationPolicy changes
window_manager_begin: BackgroundTaskManagementAgent (78979) is not observable, subscribing to activationPolicy changes
window_manager_begin: loginwindow (380) is not observable, subscribing to activationPolicy changes
window_manager_add_existing_application_windows: Things has windows that are not yet resolved
window_manager_add_existing_application_windows: Mail has windows that are not yet resolved
window_manager_begin: Private Internet Access (83864) is not observable, subscribing to activationPolicy changes
window_manager_begin: avconferenced (1270) is not observable, subscribing to activationPolicy changes
window_manager_begin: Creative Cloud Core Service (84037) is not observable, subscribing to activationPolicy changes
window_manager_begin: TextInputMenuAgent (797) is not observable, subscribing to activationPolicy changes
window_manager_begin: universalAccessAuthWarn (91872) is not observable, subscribing to activationPolicy changes
window_manager_begin: familycircled (731) is not observable, subscribing to activationPolicy changes
window_manager_begin: karabiner_console_user_server (75771) is not observable, subscribing to activationPolicy changes
window_manager_add_existing_application_windows: Music has windows that are not yet resolved
window_manager_add_existing_application_windows: Notes has windows that are not yet resolved
window_manager_begin: CoreLocationAgent (7202) is not observable, subscribing to activationPolicy changes
window_manager_begin: Wi-Fi (676) is not observable, subscribing to activationPolicy changes
window_manager_begin: Menuwhere (701) is not observable, subscribing to activationPolicy changes
window_manager_begin: PowerChime (31404) is not observable, subscribing to activationPolicy changes
window_manager_begin: Raycast Helper (Extensions) (36717) is not observable, subscribing to activationPolicy changes
window_manager_begin: photolibraryd (34140) is not observable, subscribing to activationPolicy changes
window_manager_begin: Family (52591) is not observable, subscribing to activationPolicy changes
window_manager_begin: AirPlayUIAgent (791) is not observable, subscribing to activationPolicy changes
window_manager_begin: Adobe Content Synchronizer (84035) is not observable, subscribing to activationPolicy changes
window_manager_begin: TeamProjectsLocalHub (28533) is not observable, subscribing to activationPolicy changes
window_manager_begin: next-server (80014) is not observable, subscribing to activationPolicy changes
window_manager_begin: Syncthing (708) is not observable, subscribing to activationPolicy changes
window_manager_begin: Wallpaper (643) is not observable, subscribing to activationPolicy changes
window_manager_begin: universalaccessd (624) is not observable, subscribing to activationPolicy changes
window_manager_begin: FigmaAgent (83145) is not observable, subscribing to activationPolicy changes
window_manager_begin: Wallet (75308) is not observable, subscribing to activationPolicy changes
window_manager_begin: Creative Cloud Libraries Synchronizer (13328) is not observable, subscribing to activationPolicy changes
window_manager_begin: chronod (714) is not observable, subscribing to activationPolicy changes
window_manager_begin: Notification Center (695) is not observable, subscribing to activationPolicy changes
window_manager_begin: Control Center (631) is not observable, subscribing to activationPolicy changes
window_manager_add_existing_application_windows: Figma has windows that are not yet resolved
window_manager_begin: com.apple.PressAndHold (766) is not observable, subscribing to activationPolicy changes
window_manager_begin: UserNotificationCenter (83198) is not observable, subscribing to activationPolicy changes
window_manager_begin: UIKitSystem (746) is not observable, subscribing to activationPolicy changes
window_manager_create_and_add_window:62344 VSCodium - yabairc — .dotfiles (AXWindow:AXStandardWindow:1)
window_manager_begin: Things Helper (703) is not observable, subscribing to activationPolicy changes
window_manager_add_existing_application_windows: Arc has windows that are not yet resolved
window_manager_begin: yabai (92243) is not observable, subscribing to activationPolicy changes
window_manager_begin: Shortcuts Events (35171) is not observable, subscribing to activationPolicy changes
window_manager_begin: Karabiner-NotificationWindow (75918) is not observable, subscribing to activationPolicy changes
window_manager_begin: Dock (30858) is not observable, subscribing to activationPolicy changes
window_manager_begin: AdGuard for Safari (1116) is not observable, subscribing to activationPolicy changes
window_manager_begin: SystemUIServer (632) is not observable, subscribing to activationPolicy changes
window_manager_begin: Creative Cloud Interprocess Service (44085) is not observable, subscribing to activationPolicy changes
EVENT_HANDLER_WINDOW_TITLE_CHANGED: iTerm2 63706
EVENT_HANDLER_DAEMON_MESSAGE: signal --add event=dock_did_restart action=sudo yabai --load-sa
EVENT_HANDLER_DAEMON_MESSAGE: signal --add event=window_created action=sketchybar --trigger space_windows_change
EVENT_HANDLER_DAEMON_MESSAGE: signal --add event=window_destroyed action=sketchybar --trigger space_windows_change
EVENT_HANDLER_DAEMON_MESSAGE: config window_placement second_child focus_follows_mouse autofocus menubar_opacity 0.0 window_animation_duration 0.0 layout bsp auto_balance off top_padding 0 bottom_padding 8 left_padding 8 right_padding 8 window_gap 8
EVENT_HANDLER_WINDOW_TITLE_CHANGED: iTerm2 63706
setup space 1 : Web
EVENT_HANDLER_WINDOW_RESIZED: iTerm2 63706
EVENT_HANDLER_WINDOW_MOVED:DEBOUNCED iTerm2 63706
EVENT_HANDLER_WINDOW_RESIZED:DEBOUNCED VSCodium 62344
EVENT_HANDLER_WINDOW_MOVED:DEBOUNCED VSCodium 62344
EVENT_HANDLER_WINDOW_RESIZED:DEBOUNCED VSCodium 62344
EVENT_HANDLER_WINDOW_RESIZED:DEBOUNCED iTerm2 63706
EVENT_HANDLER_WINDOW_RESIZED:DEBOUNCED VSCodium 62344
EVENT_HANDLER_WINDOW_RESIZED:DEBOUNCED iTerm2 63706
EVENT_HANDLER_WINDOW_MOVED:DEBOUNCED iTerm2 63706
EVENT_HANDLER_WINDOW_RESIZED:DEBOUNCED VSCodium 62344
EVENT_HANDLER_WINDOW_MOVED:DEBOUNCED VSCodium 62344
EVENT_HANDLER_WINDOW_RESIZED:DEBOUNCED iTerm2 63706
EVENT_HANDLER_WINDOW_RESIZED:DEBOUNCED VSCodium 62344
EVENT_HANDLER_WINDOW_MOVED:DEBOUNCED VSCodium 62344
EVENT_HANDLER_WINDOW_RESIZED:DEBOUNCED iTerm2 63706
EVENT_HANDLER_WINDOW_RESIZED:DEBOUNCED VSCodium 62344
EVENT_HANDLER_WINDOW_MOVED:DEBOUNCED VSCodium 62344
EVENT_HANDLER_DAEMON_MESSAGE: query --spaces --space 1
EVENT_HANDLER_WINDOW_TITLE_CHANGED: iTerm2 63706
EVENT_HANDLER_DAEMON_MESSAGE: space 1 --label Web
yabai -m space 1 --label Web
setup space 2 : Mail
EVENT_HANDLER_DAEMON_MESSAGE: query --spaces --space 2
EVENT_HANDLER_DAEMON_MESSAGE: space 2 --label Mail
yabai -m space 2 --label Mail
setup space 3 : Todo
EVENT_HANDLER_DAEMON_MESSAGE: query --spaces --space 3
EVENT_HANDLER_DAEMON_MESSAGE: space 3 --label Todo
yabai -m space 3 --label Todo
setup space 4 : Code
EVENT_HANDLER_DAEMON_MESSAGE: query --spaces --space 4
EVENT_HANDLER_DAEMON_MESSAGE: space 4 --label Code
yabai -m space 4 --label Code
setup space 5 : Music
EVENT_HANDLER_DAEMON_MESSAGE: query --spaces --space 5
EVENT_HANDLER_DAEMON_MESSAGE: space 5 --label Music
yabai -m space 5 --label Music
setup space 6 : Design
EVENT_HANDLER_DAEMON_MESSAGE: query --spaces --space 6
EVENT_HANDLER_DAEMON_MESSAGE: space 6 --label Design
yabai -m space 6 --label Design
setup space 7 : Presentation
EVENT_HANDLER_DAEMON_MESSAGE: query --spaces --space 7
EVENT_HANDLER_DAEMON_MESSAGE: space 7 --label Presentation
yabai -m space 7 --label Presentation
EVENT_HANDLER_DAEMON_MESSAGE: rule --add app=^(Arc|Discord)$ space=^1
EVENT_HANDLER_DAEMON_MESSAGE: rule --add app=^(Mail)$ space=^2
EVENT_HANDLER_DAEMON_MESSAGE: rule --add app=^(Things|Notes)$ space=^3
EVENT_HANDLER_DAEMON_MESSAGE: rule --add app=^(VSCodium|iTerm)$ space=^4
EVENT_HANDLER_DAEMON_MESSAGE: rule --add app=^Music$ space=^5
EVENT_HANDLER_DAEMON_MESSAGE: rule --add app=^(Figma|Adobe After Effects 2024|Adobe Photoshop Lightroom Classic|Photoshop)$ space=^6
EVENT_HANDLER_DAEMON_MESSAGE: rule --add app=^(Keynote|Microsoft PowerPoint)$ space=^7
EVENT_HANDLER_DAEMON_MESSAGE: rule --add app=^(Steam Helper|Raycast|Calculator|Software Update|Dictionary|VLC|System Preferences|System Settings|zoom.us|Photo Booth|Archive Utility|App Store|Clock|Karabiner-Elements|Plexamp|Alfred Preferences|ImageOptim|The Unarchiver|Installer|Hosting AU|Numi)$ manage=off
EVENT_HANDLER_DAEMON_MESSAGE: rule --add label=Small windows title=(Co(py|nnect)|Move|Info|Pref) manage=off
EVENT_HANDLER_DAEMON_MESSAGE: rule --add label=GLMv4 app=^GLMv4$ title=(Preparations for Measurements) manage=off
EVENT_HANDLER_DAEMON_MESSAGE: rule --add label=Cyberduck app=^Cyberduck$ title=(Transfers) manage=off
EVENT_HANDLER_DAEMON_MESSAGE: rule --add label=Mail New Message app=^Mail$ title=(New Message) manage=off
EVENT_HANDLER_DAEMON_MESSAGE: rule --add label=About This Mac app=System Information title=About This Mac manage=off
EVENT_HANDLER_DAEMON_MESSAGE: rule --add label=Activity Windows title=^Activity$ manage=off
EVENT_HANDLER_DAEMON_MESSAGE: rule --add app=^Adobe (After Effects|Audition|Media Encoder|Lightroom)$ role=AXLayoutArea subrole=AXFloatingWindow manage=on
EVENT_HANDLER_DAEMON_MESSAGE: rule --add app=^IINA$ sticky=on manage=off
EVENT_HANDLER_DAEMON_MESSAGE: rule --add app=^Messages$ scratchpad=Messages grid=11:11:1:1:9:9
EVENT_HANDLER_DAEMON_MESSAGE: rule --add app=^Transmission$ scratchpad=Transmission grid=11:11:1:1:9:9
EVENT_HANDLER_DAEMON_MESSAGE: rule --apply
Yabai configuration loaded…

Eh, I think I accidentally left in code that automatically hides the window when assigned to a scratchpad through a rule.

Fixed on master.

Hi, thanks for the feature!
I'm having an issue:
to reproduce open 2 windows on a single space, fullscreen one of them (not native fullscreen)
then trigger the scratchpad window, then 3 windows are visible, instead of scratchpad one and fullscreened in the background

That's intentional. Scratchpad is not meant to replace spaces for separation of windows. The scratchpad is simply a window that can be toggled into focus at any time regardless of which display or space is active. If you have window opacity enabled and the scratchpad window doesn't cover the entire screen, you will see the other windows in the back.

Hey is there any intention of making scratchpad work with single instances of application? For example having one window of terminal in the scratchpad and other windows in tiling mode in a space? or like the ability to have the shortcut to launch an application in scratchpad without having to give it a window rule (I don't always need discord to be in a scratchpad)

For example having one window of terminal in the scratchpad and other windows in tiling mode in a space

I need to figure this out too. My terminal doesn't allow me to set title in a way that sticks such that Yabai can detect it.
Historically, I've had to script it so that after I launch a window, I get the latest window for that app, and then grab the ID.
launch_scratchpad() {
  ~/.scripts/term
  sleep 0.25
  scratchpad_id=$(yabai -m query --windows | jq 'map(select(.app == "WezTerm" or .app == "wezterm-gui")) | sort_by(.id) | last | .id')
  position_scratchpad "$scratchpad_id"
}

position_scratchpad() {
  scratchpad_id=$1
  yabai -m window --focus "$scratchpad_id"
  yabai -m window "$scratchpad_id" --toggle float
  yabai -m window "$scratchpad_id" --grid 11:11:1:1:9:9
}

Is it possible to dynamically add a rule based on window ID. I understand that's a hack.

You can do: yabai -m window window_id --scratchpad my_scratchad instead of using a rule.

Hey! Thanks for this feature. I started using this on my Linux Hyprland desktop and thought “I wonder if yabai has this?” And here it was. Awesome!
Wanted some clarification on how the workflow is supposed to go. Am I correct in thinking that assigning the window rules per your above examples and then the proper skhd toggle bind set up. That pressing it while the app is hidden/closed should:

Bring it to the front and float. Open if not already running.
Pressing the bind again should then hide the application?

Number two doesn’t seem to be happening and I have to press cmd + h to hide the app.
Thanks again.

For anyone trying to use this and getting weird partially-working behavior, I had missed that this requires partially disabling System Integrity Protection

On my setup toggling the scratchpad window off also changes the active space, i'm not sure if i'm doing something wrong or if this is expected behaviour..
# on yabairc
yabai -m rule --add app="^Spotify$" scratchpad="Spotify" opacity=".9" grid="11:11:1:1:9:9"

# on keypress
yabai -m window --toggle "$NAME" || open  -a "$NAME"

Reproducing steps, until after the step 3, yabai query shows space=1 for the app, on step 4 i see space=2

On space 1, trigger the key
See app open, everything looks normal
trigger the key again
app closes, but now i'm focused on the second space

Can I provide more info somehow?

You can do: yabai -m window window_id --scratchpad my_scratchad instead of using a rule.

I had missed this. Maybe worth documenting the scratchpad actions/options in the Wiki/man?


Configure Pod Initialization

The very first example for pod with init container is failing, pod goes to Init:CrashLoopBackOff, it supposed to be in Running Status.
Not sure what has changed !

I think you're reporting that you're having trouble following https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-initialization/ @SystemicEmotions
I think that's right, and I can replicate these symptoms. The busybox container's wget command does not provide TLS certificate verification.
I think that if we amend https://github.com/kubernetes/website/blob/29e13ed527659898c14d0d95acbceea1fcb1ca61/content/en/examples/pods/init-containers.yaml to fetch http://info.cern.ch/ rather than http://kubernetes.io/, then that will fix it.
/triage accepted
/language en
/kind bug
/priority backlog
Thanks @SystemicEmotions

/good-first-issue
(if anyone is interested in fixing this, but isn't sure where to start, see https://kubernetes.io/docs/contribute/ or reply with your question here)

Duplicated by https://github.com/kubernetes/website/issues/24529

Okay, I will replace it with the "http://info.cern.ch/".
Regards,

Hello, I would like to start contributing and I was thinking about solving this issue but I would like to know why this issue still open if after replacing the Url with "http://info.cern.ch" sample works again. Is there anything pending?
Thanks

This is fixed
/close


exaile: 4.1.1 -> 4.1.2

Description of changes
https://github.com/exaile/exaile/wiki/Exaile-4.1-release-notes
Things done

Built on platform(s)

[X] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[X] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[X] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 190870 run on x86_64-linux 1

1 package built:

exaile


CLI stuck on prompt when running in non-interactive mode

When executing a command that yields a prompt, such as spo app retract, when executed in non-interactive mode, the CLI doesn't render the prompt in the shell. Instead it hangs. If the prompt is suppressed using the --confirm option, the command works as expected. Verify if this is a limitation in Vorpal or a bug in the CLI implementation.

After a quick look it seems like an issue with Vorpal. There was an issue with PR logged for it in the past (https://github.com/dthree/vorpal/pull/166) but apparently there were some side-effects which is why it isn't available in the current build of Vorpal. Let's have a closer look and see if we can find a way to deal with this issue in the CLI.

I've got a workaround for the issue in Vorpal. The only side effect is an extra empty line printed before and after the command output, but they seem harmless and acceptable trade-off to be able to correctly display prompts in the non-interactive mode in the CLI.
I would appreciate it, if you could have a look at the fix guys, give it a test and tell me if I missed anything and if it would be acceptable to merge in the CLI.
See the fix in my fork https://github.com/waldekmastykarz/office365-cli/tree/fix-noninteractiveprompt.
/cc: @VelinGeorgiev @andrewconnell


feat: defer the database connection to when it's needed

Related Issues

fixes #762

Proposed Changes:

How did you test it?

Notes for the reviewer

Checklist

I have read the contributors guidelines and the code of conduct
I have updated the related issue with new insights and changes
I added unit tests and updated the docstrings
I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.

To my joy, all the tests are green locally using the same Docker image https://tableflipper.com


Revert react-docgen patch once downstream fix is released

In #526 we introduced a hotfix for a broken third party library, react-docgen. We should revert this patch when a new version of the library is released upstream.

May be able to remove it soon:
https://twitter.com/TschinderDaniel/status/1204897856995373063


engine: fix engine metrics not resetting

Signed-off-by: tabokie<EMAIL_ADDRESS>What problem does this PR solve?
Issue Number: close #10294
What is changed and how it works?
What's Changed: Revert metrics flusher changes brought by #10223.
Related changes
Check List
Release note
None

/merge


changed conector test cache param

Fix some issues with GenericHeaders that cause mail parsing crash.

Given that ParameterHeader has the method getValueFor the change is equivalent to the current implementation.


(#341) Add tests to verify that various hard-limits are enforced.

This commit adds checks in different places to validate that the documented page-, extent-, key-size limits are properly enforced.
Some asserts trapping these limits are converted to error messages. Add new test, limitations_test, to verify that different Splinter config / init / create interfaces fail (as expected) when used with illegal values.

NOTE: This is really not new code-dev, but mainly tightening error checks in some of the test-config interfaces, and in a few places in Splinter main code. This implements checks to verify that our supported page- / extent-size and max-key-size limits are properly enforced, with error messages, in most commonly seen interfaces. New test dev is done to verify these checks. Fixes issue #341 .

Here is a successful run of the limitations tests added in this PR:
Fusion-LocalVM:[492] $ ./bin/unit_test limitations
Running 1 CTests, suite name 'limitations', test case 'all'.
TEST 1/7 limitations:test_io_init_invalid_page_size [OK]
TEST 2/7 limitations:test_io_init_invalid_extent_size [OK]
TEST 3/7 limitations:test_splinterdb_create_invalid_page_size [OK]
TEST 4/7 limitations:test_splinterdb_create_invalid_extent_size [OK]
TEST 5/7 limitations:test_splinterdb_mount_invalid_key_size [OK]
TEST 6/7 limitations:test_trunk_create_invalid_key_size [OK]
TEST 7/7 limitations:test_trunk_mount_invalid_key_size [OK]
RESULTS: 7 tests (7 ok, 0 failed, 0 skipped) ran in 172 ms

HI, @rtjohnso , @ajhconway - Update on this test-dev to verify that hard-limits are properly enforced.
I have reworked the checks as discussed yesterday. Notably:

Add laio_config_valid() as single point validation function.
Relocate new page / extent size limits to laio.h, naming them LAIO_DEFAULT_PAGE_SIZE, LAIO_DEFAULT_EXTENT_SIZE etc.

Before I squash and push these changes, I wanted to give you a chance to re-look at the delta-diffs arising from just the changes as discussed yesterday. You can find them in my dev-branch agurajada/341-PgExt-size-limits-CI, in this commit, which I just pushed to git-hub.
Am planning to go ahead with this merge as the only outstanding item from previous reviews should have been addressed with above commit.
I have another PR in the CI-run way coming up shortly soon. Once that merges, I will rebase off of /main, and will get this PR's changes into the CI-mill.

FYI, @rtjohnso , @ajhconway - I have made the changes discussed to consolidate all IO-config parameter validation to new fn, laio_config_valid(), and related changes.
Have rebased v/s main, and squashed into one single commit.
This will be merged once CI-passes. Bombs away!


Release v9.3.0

may want to double-check the branch name. I think the release automation depends on it

may want to double-check the branch name. I think the release automation depends on it, but maybe I don't know what's permitted

I think the title only matters.


ImportError: No module named 'treedict.treedict'

I have installed the master branch with pip, cython was not used. I get this error with python 3.4 on windows:
from treedict import TreeDict
Traceback (most recent call last):
  File "C:\prs\main\test_2.py", line 165, in <module>
    from treedict import TreeDict
  File "C:\dev\python\lib\site-packages\treedict\__init__.py", line 1, in <module>
    from .treedict import TreeDict, getTree, treeExists, HashError
ImportError: No module named 'treedict.treedict'

I hit the same problem with python 2.6, 2.7, 3.3, 3.4 virtualenvs.  What was happening is the old version on pypi is treedict-0.14.tar.gz while the latest version with python 3 support is treedict-0.2.2.tar.gz.  Pip says 14 > 2 so installs the old.  Work around is to explicitly "pip install treedict==0.2.2".
HTH

Oh, oops!  It was supposed to be 0.1.4.  I'll try to do an update to 1.0
soon then.
-- Hoyt
++++++++++++++++++++++++++++++++++++++++++++++++

Hoyt Koepke
Data Scientist at GraphLab
<EMAIL_ADDRESS>++++++++++++++++++++++++++++++++++++++++++

On Thu, Jan 22, 2015 at 1:49 PM, Roy Wright<EMAIL_ADDRESS>wrote:

I hit the same problem with python 2.6, 2.7, 3.3, 3.4 virtualenvs. What
was happening is the old version on pypi is treedict-0.14.tar.gz while the
latest version with python 3 support is treedict-0.2.2.tar.gz. Pip says 14

2 so installs the old. Work around is to explicitly "pip install
treedict==0.2.2".
HTH

—
Reply to this email directly or view it on GitHub
https://github.com/hoytak/treedict/issues/8#issuecomment-71106006.

Had the same problem. Command "pip install treedict==0.2.2" helped. Any chance that this will be fixed soon?

I'm also having the same version trying to install via pip using the Github repo as my source, as well as building with "python ./setup.py install" in a directory - even when using the 0.2.2 .tar.gz file fetched off pypi.
Turns out this is a little weird.  If you build and 'setup.py install' then you can't "import treedict" while you're in that same directory.  change directory elsewhere, no worries.

Came across the same problem with python3 in Jupyter notebook, solved by "pip3 install treedict==0.2.2"

Personally, at this point I'm using python-box because it isn't an
abandoned project. I suggest you do as well.


Cannot get older home timeline (max_id issue?)

Response of GET /api/v1/timelines/home?max_id=123456789&limit=20 contains item id: 123456789 at last of result array.
Actually GET /api/v1/timelines/home?limit=20 has same response with GET /api/v1/timelines/home?max_id=123456789&limit=20.
Tested on: Pinafore, Tusky
Maybe related: #18

Maybe related?: https://github.com/videah/SkyBridge/issues/11#issuecomment-1632997087
Currently I'm running SkyBridge on my server btw

That's quite important, fixing it would make skybridge quite useful.


Write ATS for TG requirement 65

To support collation with other maps for both supported image formats (GIF and PNG), the transparency parameter (TRANSPARENT) of the WMS GetMap request shall be set to “true” and the background parameter (BGCOLOR) for all layers shall be set to the same colour.

I agree, not machine testable, with a visual check a user might be able to detect visual artifacts related to the case to be tested here


Fix spotlight position for tutorial

What

:up: Gecco supported version to greater than iOS 11.0
Fix calculate spotlight position for tutorial.

device
1
2
3
4

iPhone 11 Pro

iPhoneSE(2nd)


#212 form id renamer handles additional expressions

Added id renamer support for
item.canAddRow
item.canRemoveRow
valueSetEntry.when

@patope do you know if there are any other "new" expression attributes that I am missing here?


[BUG] Clockwork debugger won't output empty values

Wasted quite a bit of time until realized debugger doesn't work as expected. Some empty values are completely ignored and not displayed in log. Eg.:
    {{ dump('TRUE') }}
    {{ dump(true) }}
    {{ dump('FALSE') }}
    {{ dump(false) }}
    {{ dump(0) }}
    {{ dump('') }}
    {{ dump(null) }}
    {{ dump([]) }}
    {{ dump({}) }}

Debugger output:

So basically these three are never displayed:
    {{ dump(false) }}
    {{ dump(0) }}
    {{ dump('') }}

P. S. Also was always wondering why null is wrapped into an array 🤔

This is upstream issue.

I have no idea what that means 😄
Are you saying it's Clockwork related and nothing can be done on Grav side?

Oh, I meant to say that the issue is in clockwork. Debugbar is fine.

Created an issue there

Apparently with clock() it works just fine (and also null is logged normally) 😕


Gold adjustments

Updated gold formula by reducing high end creep values. Increased increment cost for upgrades. Lowered base cost for caster upgrade.
Reduced gold scaling factor from 4 -> 3.
Additional gold bounties:
Creeps over level 7 - 5g -> 3g
Creeps over 1500 - 7g -> 5g
Creeps over 2000 - 10g -> 8g
Creeps over 2500 - 15g -> 10g
E.g. total bonus for 2500hp mob 37g -> 26g
Upgrade changes:
Attack upgrade gold increment 55 -> 75
Attack upgrade time increment 25s -> 30s
Defense upgrade gold increment 50 -> 65
Defense upgrade time increment 25s -> 30s
Caster upgrade base gold cost 100 -> 75

I think this will have to wait until I can do those migration changes we talked about! There are so many files here that I can't review any of them.

Makes sense - do you know how long that might be? Otherwise might think of doing some other changes for some races that need it sooner rather than later, and maybe put the upgrade changes into another PR.

Makes sense - do you know how long that might be? Otherwise might think of doing some other changes for some races that need it sooner rather than later, and maybe put the upgrade changes into another PR.

It'll be a while I think, because I'll be putting those changes into dev rather than main. So yup I think you'll want to do that split.

Ok @vampirika, I made the change: level and gold bounty data is no longer stored in .json files, since it's calculated from your map migration anyway.
As you can see you now have a tonne of conflicts, but that's no big deal. I recommend just rolling back all of the changes in your the UnitData folder, and we can keep going with the rest.

@YakaryBovine not sure exactly how meant to do it, so just made a new pr with the changes in

Great, as I understand it we've just merged this in via the other branch so I'll close this one.


ZonedDateTime toString is broken after deserialization

ZonedDateTime serialization relies in the serialization of ZoneOffset. However, ZoneOffset has a transient id field that is not handled correctly: when deserializing a ZoneOffset, only the totalSeconds field is recovered. The id will be null and the toString method will return "null".

ZoneOffset serialization works fine, because it delegates the work to a serialization proxy (org.threeten.bp.Ser). The readResolve method on ZoneOffset emphasises this by throwing an exception. There are also tests for serialization of ZoneOffset.
I suspect you are using a tool that does not correctly implement the serialization specification, which isn't something I can help with.


package react-codemod with ncc

This would avoid issues like https://github.com/reactjs/react-codemod/issues/249

Hey @threepointone, I wanted to share CodeshiftCommunity, which could help side-step some of these issues.
It allows you to publish codemods as packages to NPM, the CLI will pull it down and run it in an isolated context.
$ npx @codeshift/cli -p react#error-boundaries path/to/src
Meaning:

You always run the latest codemods
You don't have to maintain your own CLI implementation
You can bundle your own dependencies with the mods

Codeshift allows developers to publish codemods as their own NPM packages or as part of an existing NPM package. We provide a standardized CLI tool that can download and run the latest codemods from any location. This approach has the added benefit of allowing codemods to be published with dependencies, which is not currently possible with most JSCodeshift CLI implementations.

To integrate with this repo, you just need to add a codeshift.config.js.
module.exports = {
  maintainers: ['threepointone'],
  targets: ['react'],
  presets: {
    'React-DOM-to-react-dom-factories': require.resolve('./transforms/React-DOM-to-react-dom-factories.js'),
    'React-PropTypes-to-prop-types': require.resolve('./transforms/React-PropTypes-to-prop-types.js'),
    'ReactNative-View-propTypes': require.resolve('./transforms/ReactNative-View-propTypes.js'),
    'create-element-to-jsx': require.resolve('./transforms/create-element-to-jsx.js'),
    'error-boundaries': require.resolve('./transforms/error-boundaries.js'),
    ...
  },
};

Obviously, no pressure here, thought it might help in the future 👍


Missing comma on the API configuration for both plans

[Enter feedback here]
Hi,
there's a comma missing in the API templates. It should be like this:
{
"properties": {
"pricingTier": "Standard",
"subPlan": "PerTransaction"
}
}
The comma missing is after the standard word within quotes. It should be added on both API templates.
Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 2b92c195-c670-2e27-2422-fd131586bdd7
Version Independent ID: 75ad901d-8064-bfbe-67f5-de507bede1bc
Content: Enable Microsoft Defender for Storage - Azure Storage
Content Source: articles/storage/common/azure-defender-storage-configure.md
Service: storage
Sub-service: common
GitHub Login: @bmansheim
Microsoft Alias: benmansheim

@acruz2021
Thanks for your feedback! We will investigate and update as appropriate.

@acruz2021
Thanks for pointing this out. Required changes have been made and visible in the document itself. I am going to close this thread now. Please feel free to reach out.


memory leak in backtrace mode

Hi, please have a look at the memory leak that occurs during conversion from K vectors to numpy arrays. The increasing reference count occurs for all date and time types.
>>> q('a:1000000?.z.D')
>>> q('-16!a')
k('1i')

>>> np.asarray(q.a)
array(['2008-02-04', '2015-01-02', '2003-07-05', ..., '2002-10-16',
       '2004-02-07', '2018-03-25'], dtype='datetime64[D]')
>>> q('-16!a')
k('2i')

>>> for i in range(1000):
...     _ = np.asarray(q.a)
...
>>> gc.collect()
0

>>> q('-16!a')
k('1002i')

The same thing happens when you press the tab key.
q.a=0
q('-16!a')
q.a.<tab>
q('-16!a')
q.a.<tab>
q('-16!a')

Is there an update on this?

@antipisa please feel free to provide pull request to address this issue.

@antipisa Can you provide any environment information on versions that you have seen this issue if possible, I can't replicate the issue locally and will look at adding a fix once I can replicate.
>>> q('a:1000000?.z.D')
k('::')
>>> q('-16!a')
k('1i')
>>> import numpy as np
>>> np.asarray(q.a)
array(['2008-02-04', '2015-01-02', '2003-07-05', ..., '2012-12-29',
       '2008-05-15', '2000-07-16'], dtype='datetime64[D]')
>>> q('-16!a')
k('1i')
>>> for i in range(1000):
...     _ = np.asarray(q.a)
...
>>> q('-16!a')
k('1i')

PyQ version 4.2.1, KDB version 3.6.2018.12.24, numpy 1.16.2, all are installed from conda.

@antipisa Thanks for the extra information, unfortunately I still haven't been able to replicate on macOS or Ubuntu Linux what architecture are you seeing the behaviour on?
@sashkab Have you been able to replicate the issue on your side?

@cmccarthy1 I don’t think this is a specific environment issue. I can reproduce it on CentOS, macOS and RedHat 6 and 7.

@cmccarthy1 Nope. Just tried again:
This is RHEL6, 64-bit.
$ pyq --versions
PyQ 4.2.1
NumPy 1.16.2
KDB+ 3.5 (2017.06.19) l64
Python 3.6.6 (default, Jul  9 2018, 09:23:37)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-17)]

$ pyq
Python 3.6.6 (default, Jul  9 2018, 09:23:37)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-17)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> q('a:1000000?.z.D')
k('::')
>>> q('-16!a')
k('1i')
>>> import numpy as np
>>> np.asarray(q.a)
array(['2008-02-04', '2015-01-02', '2003-07-05', ..., '2004-06-10',
       '2004-09-14', '2013-07-24'], dtype='datetime64[D]')
>>> q('-16!a')
k('1i')
>>> for i in range(1000):
...      _ = np.asarray(q.a)
...
>>> q('-16!a')
k('1i')
>>>

And this is 64-bit macOS:
$ pyq --versions
PyQ 4.2.1
NumPy 1.16.2
KDB+ 3.5 (2019.01.31) m64
Python 3.6.9 (default, Oct  4 2019, 22:26:04)
[GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1<IP_ADDRESS>)]

$ pyq
Python 3.6.9 (default, Oct  4 2019, 22:26:04)
[GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1<IP_ADDRESS>)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> q('a:1000000?.z.D')
k('::')
>>> q('-16!a')
k('1i')
>>> import numpy as np
>>> np.asarray(q.a)
array(['2008-02-04', '2015-01-02', '2003-07-05', ..., '2004-06-10',
       '2004-09-14', '2013-07-24'], dtype='datetime64[D]')
>>> q('-16!a')
k('1i')
>>> for i in range(1000):
...     _ = np.asarray(q.a)
...
>>> q('-16!a')
k('1i')

@sashkab @cmccarthy1  Can you please try:
Do a fresh install with conda
Try it with python 3.6.5 and kdb  3.5.2018.10.12, and with python 3.7.1 and kdb 3.6.2018.12.24

@antipisa I tried that as follows on Mac unfortunately with the same result on my side
(test_pyq) conors-Air:pyq conormccarthy$ pyq --versions
PyQ 4.2.1
NumPy 1.17.2
KDB+ 3.6 (2018.12.24) m64
Python 3.7.1 (default, Dec 14 2018, 13:33:11)
[Clang 4.0.1 (tags/RELEASE_401/final)]

(test_pyq) conors-Air:pyq conormccarthy$ pyq
Python 3.7.1 (default, Dec 14 2018, 13:33:11)
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> import gc
>>> q('a:1000000?.z.D')
k('::')
>>> q('-16!a')
k('1i')
>>> np.asarray(q.a)
array(['2008-02-04', '2015-01-02', '2003-07-05', ..., '2004-06-10',
       '2004-09-14', '2013-07-24'], dtype='datetime64[D]')
>>> q('-16!a')
k('1i')
>>> for i in range(1000):
...     _ = np.asarray(q.a)
...
>>> gc.collect()
253
>>> q('-16!a')
k('1i')

I won't be online for about a week but once I can replicate this on return I'll take a look into it again. Unfortunately as it stands until we can replicate locally it's not going to be easy to fix as I'm sure you can appreciate

@cmccarthy1, you are on the wrong version of numpy, could you try it with 1.16.2?

With downgraded numpy I'm seeing the same behaviour.
(test_pyq) conors-Air:pyq conormccarthy$ pyq --versions
PyQ 4.2.1
NumPy 1.16.2
KDB+ 3.6 (2018.12.24) m64
Python 3.7.1 (default, Dec 14 2018, 13:33:11)
[Clang 4.0.1 (tags/RELEASE_401/final)]

(test_pyq) conors-Air:pyq conormccarthy$ pyq
Python 3.7.1 (default, Dec 14 2018, 13:33:11)
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> import gc
>>> q('a:1000000?.z.D')
k('::')
>>> q('-16!a')
k('1i')
>>> np.asarray(q.a)
array(['2008-02-04', '2015-01-02', '2003-07-05', ..., '2004-06-10',
       '2004-09-14', '2013-07-24'], dtype='datetime64[D]')
>>> q('-16!a')
k('1i')
>>> for i in range(1000):
...     _ = np.asarray(q.a)
...
>>> q('-16!a')
k('1i')
>>> gc.collect()
253

Could this be a difference between regular and nosse builds of KDB?

@sashkab @cmccarthy1  Yes, I think the difference is the conda binary is gcc 7.2 compiled.

@cmccarthy1  is there any update on this?

Hi @antipisa,
Apologies for the delay in response this is being looked into now, we have only in the last few days been able to recreate the issue and I will update here once a fix for the issue has been found.

@cmccarthy1, @antipisa  - can someone comment on the status of this issue?  Is there a clear way to reproduce the problem?

@abalkin my understanding this is reproducible only when PyQ is compiled by conda with GCC 7.2. Not sure if it's downloaded pre-compiled from our repository, or compiled on end user's workstation.

@sashkab - can you point me to the comment reporting "conda with GCC 7.2"?

The standard miniconda image (continuumio/miniconda3) uses gcc 7.3:
# pyq --versions
PyQ 4.2.1
NumPy 1.17.4
KDB+ 3.6 (2019.08.20) l64
Python 3.7.4 (default, Aug 13 2019, 20:39:44)
[GCC 7.3.0]

and I don't see a reference leak there:
>>> import numpy as np
>>> q('a:1000000?.z.D')
k('::')
>>> q('-16!a')
k('1i')
>>> x = np.asarray(q.a)
>>> q('-16!a')
k('1i')

can you point me to the comment reporting "conda with GCC 7.2"?

https://github.com/KxSystems/pyq/issues/109#issuecomment-548814527

been able to recreate the issue

@cmccarthy1 Please provide steps you used to recreate the issue. We are unable.

Please note, this issue will be closed as unreproducable on or after January 31, 2020 if the exact steps to reproduce won't be posted.

@sashkab @abalkin @antipisa
We had been able to produce similar behaviour using a docker image which I can't really share here, however we are currently unsure that the behaviour we had been seeing was a memory leak and wasn't explainable by other effects relating to the interaction between kdb and Python at an interface level.
If we can reproduce consistently and in line with the above or have any further information of relevance to provide I will revert here, provide code to reproduce if feasible.

@cmccarthy1 - thanks for the update, but I think it is best to close this issue and open a new one once there is a problem that can be reproduced reliably.

Hi @sashkab + @abalkin We're getting a number of questions raised internally regarding this issue still, I see that there was a 5.0 tag placed on this issue and not sure if it has been fixed with this update?
We do have some more information regarding an environment which may show the potential cause of this which may allow you to replicate as the setup is internal to the client and as such required some changes to their setup to fit with their environmental requirements
// From the client anonymised for privacy reasons
The setup.py script:
As we can’t install to q_home so install to install_lib
-        self.set_undefined_options('config', ('q_home', 'install_dir'))
+        self.set_undefined_options('install', ('install_lib', 'install_dir'))

-        self.install_dir = os.path.join(self.q_home, self.q_arch)
+        self.set_undefined_options('install', ('install_lib', 'install_dir'))

pyq.c script:
-        args[1] = "python.q";
+        /*c code to allow us to use an env variable to point to the location of python.q*/

p.k script:
-\l pyq-config.q
-q)(`:./pyq 2:(`p_init;1)).pyq.python_dll
+// exact same thing but using an env variable to get the location and prepend it on the load and before the 2:

python.q script:
-\l p.k
+//get the location from an env variable and prepend it to the load

The way we load numpy is also different:
--- ./pyq-4.2.1/src/pyq/__init__.py.orig        2020-03-16 23:28:29.000000002 -0400
+++ ./pyq-4.2.1/src/pyq/__init__.py     2020-03-16 23:40:12.000000002 -0400
@@ -13,7 +13,11 @@
try:
     import numpy as _np
except ImportError:
-    _np = None
+    # code to add numpy 1.17.4 to sys.path so the below works
+    import numpy as _np

try:
     from ._k import K as _K, error as kerr, Q_VERSION, Q_DATE, Q_OS
@@ -637,7 +641,12 @@

def _genmethods(cls, obj):
-    q(r'\l pyq-operators.q')
+    pyqlib = os.environ.get('PYQLIB', '')
+    if len(pyqlib):
+        pyqlib += '/'
+    q(r'\l %spyq-operators.q' % pyqlib)
+
     cls._show = q('{` sv .Q.S[y;z;x]}')
     cls._sizeof = q('.pyq.sizeof')
     for spec, verb in [
@@ -839,7 +848,7 @@
def __import__(name, globals={}, locals={}, fromlist=[], level=[-1, 0][_PY3K],
                _imp=_imp, _c=converters, _lc=lazy_converters):
     m = _imp(name, globals, locals, fromlist, level)
-    pairs = _lc.get(name)
+    pairs = _lc.get(m.__name__)
     if pairs is not None:
         _c.update((getattr(m, cname), conv) for cname, conv in pairs)
     return m

Any clarity on potential causes and fixes for this would be greatly appreciated

I don't see anything in user configuration that may cause the described leak.
Note that rather than patching setup.py file we recommend generating setup.cfg as follows and edit it as needed
$ python setup.py config --write
running config
[config]
q_home = /Users/a/.virtualenvs/3.7/q
q_version = 4.0
q_arch = m64
python_dll = /Users/a/.virtualenvs/3.7/bin/../Python3
^^^ Written to setup.cfg.

@cmccarthy1 - you mentioned above that you were able to reproduce this in a docker container.  Would it be possible to share a dockerfile here?

And there aren't likely to be any side effects from the modification to the numpy setup within the clients environment?
The docker image that we had been using did not reproduce this issue, it highlighted some minor inconsistencies in reference count but not the issue that these users have been seeing consistently. I can try dig it up and send over as a zip but it's not likely to provide a mechanism for testing any changes to fix this.
Just for full clarity there haven't been changes to PyQ in 5.0 that may fix this issue as a side effect?

Further investigation from the client seems to suggest the following may, as a side effect, result in the exposure of the memory leak
According to new findings, "export PYQ_BACKTRACE=y" triggers the memory leak.

Confirm that it is the PYQ_BACKTRACE=y flag that causes this.
`
Python 3.7.4 (default, Aug 13 2019, 15:20:19)
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.

import numpy as np
from pyq import q
q('a:1000000?.z.D')
k('::')
for i in range(1000):
...    _ =np.asarray(q.a)
...

import gc
gc.collect()
226
q('-16!a')
k('1001i')

`

I was able to reproduce this as well.  I'll reopen this issue with an updated title.

Can you test the following patch:
diff --git a/src/pyq/__init__.py b/src/pyq/__init__.py
index af2391a..3472c1d 100755
--- a/src/pyq/__init__.py
+++ b/src/pyq/__init__.py
@@ -248,8 +248,10 @@ class K(_K):
         >>> q("([]a:1 2 3; b:10 20 30)").a
         k('1 2 3')
         """
+        if a.startswith('_'):
+            raise AttributeError(a)
         t = self._t
-        if t == 98 and not a.startswith('_'):
+        if t == 98:
             return self._k(0, '{x`%s}' % a, self)

This seems to fix it. What was causing the memory leak before?

This fixes the leak for symbols and timestamps, but not for numeric dtypes:
q$ PYQ_BACKTRACE=y pyq
Welcome to kdb+ 32bit edition
Python 3.7.4 (default, Aug 13 2019, 15:20:19)
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> from pyq import q
>>> q('a:1000000?til 100')
k('::')
>>> for i in range(1000):
...    _ =np.asarray(q.a)
...

>>> gc.collect()
>>> q('-16!a')
k('2i')

@antipisa - I don't see a memory leak in your example.  Two references are expected: one is held by the q global a and the other by the _ variable in python that is bound to a numpy array which in turn holds a reference to the same q object as q.a.
You can see that the reference count goes does when you delete _:
>>> del _
>>> q('-16!a')
k('1i')

@abalkin I'm confused then why this doesn't happen for date types? The refcount in that case is 1, even without deleting _:

>>> q('a:1000000?.z.D')
k('::')
>>> q('-16!a')
k('1i')
>>> import numpy as np
>>> np.asarray(q.a)
array(['2008-02-04', '2015-01-02', '2003-07-05', ..., '2004-06-10',
       '2004-09-14', '2013-07-24'], dtype='datetime64[D]')
>>> q('-16!a')
k('1i')
>>> for i in range(1000):
...     _ = np.asarray(q.a)
...
>>> q('-16!a')
k('1i')

@antipisa - the date case is different because numpy and k store dates differently (1970 vs. 2000 epoch) and we cannot reuse the k object by reference in numpy.  Instead asarray() creates an independent array and fills it with converted values.  This does
not require holding to a reference.

@abalkin  if the reference counts are correct, then something else is causing the memory leak. Every time a kdb table to dataframe conversion happens, the rss grows. See example:

import pyq
from pyq import q
import gc
import psutil
import pandas as pd
import numpy as np

def used_memory():
	process = psutil.Process(os.getpid())
	return (process.memory_info().rss)/2**30

def convert_to_dataframe(K_table):
	df = pd.DataFrame(dict(q("{flip x#y}", q.cols(K_table), K_table)))
	del K_table
	gc.collect()
	q(".Q.gc[]")

print("[mem:{:.2g}GB] start".format(used_memory()))

q('''
	N:10000000;
	t:([] symbol:n?`ibm.n`goog.q`fb.q; qty:"f"$n?1000000; date:2007.01.02; ts:`minute$n?700);
	t:update dt:date+ts from t;
	''')

df = convert_to_dataframe(q.t)
print("[mem:{:.2g}GB] post conversion".format(used_memory()))

df = ()
q("t:(); .Q.gc[]")
gc.collect()

print("[mem:{:.2g}GB] post deletion".format(used_memory()))

It's straightforward to verify that the memory is handled and cleaned up in pure python:
import numpy as np
import pandas as pd
from numpy.random import choice
import datetime

print("[mem:{:.2g}GB] start".format(used_memory()))
n = 50000000
d = datetime.date(2001,1,1)
df = pd.DataFrame(data=dict(x=np.random.randn(n), sym = choice(list("ABCD")), date=d))
print("[mem:{:.2g}GB] post creation".format(used_memory()))

del df
gc.collect()
print("[mem:{:.2g}GB] post deletion".format(used_memory()))

[mem:0.1GB] start
[mem:1.2GB] post creation
[mem:0.1GB] post deletion

Has the above with respect to the conversion between K vectors and arrays been handled or verified on this end @abalkin
@sashkab ?
Currently getting some internal queries regarding this issue and would be nice to know if there is any movement on this.

@antipisa - when you post snippets, please make sure they are complete and runnable.  If you suspect that there is a memory leak that shows up in regular (non-backtrace) mode – please open a separate issue.  If confirmed, it will get a higher priority.

So far my results are different from those reported by @antipisa above:
$ cat mem.py
import gc, os
import numpy as np
import pandas as pd
from numpy.random import choice
import datetime
import psutil

def used_memory():
	process = psutil.Process(os.getpid())
	return (process.memory_info().rss)/2**30

print("[mem:{:.2g}GB] start".format(used_memory()))
n = 50000000
d = datetime.date(2001,1,1)
df = pd.DataFrame(data=dict(x=np.random.randn(n), sym = choice(list("ABCD")), date=d))
print("[mem:{:.2g}GB] post creation".format(used_memory()))

del df
gc.collect()
print("[mem:{:.2g}GB] post deletion".format(used_memory()))
$ python mem.py
[mem:0.043GB] start
[mem:2.3GB] post creation
[mem:1.5GB] post deletion

my results do not agree with @abalkin  and furthermore do not show a memory leak in python, so it may be something in your environment that is giving us a red herring. In any case, we can confirm there is a memory leak in backtrace mode during K array conversion, so something pyq-related must be causing this. @cmccarthy1, can you please see if you can reproduce this in your end?
$ cat mem.py
import gc, os
import numpy as np
import pandas as pd
from numpy.random import choice
import datetime
import psutil

def used_memory():
	process = psutil.Process(os.getpid())
	return (process.memory_info().rss)/2**30

print("[mem:{:.2g}GB] start".format(used_memory()))
n = 50000000
d = datetime.date(2001,1,1)
df = pd.DataFrame(data=dict(x=np.random.randn(n), sym = choice(list("ABCD")), date=d))
print("[mem:{:.2g}GB] post creation".format(used_memory()))

del df
gc.collect()
print("[mem:{:.2g}GB] post deletion".format(used_memory()))

def convert_to_dataframe(K_table):
	df = pd.DataFrame(dict(q("{flip x#y}", q.cols(K_table), K_table)))
	del K_table
	gc.collect()
	q(".Q.gc[]")
	return df

print("[conversion][mem:{:.2g}GB] start".format(used_memory()))

q('''
	N:10000000;
	t:([] symbol:N?`ibm.n`goog.q`fb.q; qty:"f"$N?1000000; date:2007.01.02; ts:`minute$N?700);
	t:update dt:date+ts from t;
	''')

df = convert_to_dataframe(q.t)
print("[conversion][mem:{:.2g}GB] post conversion".format(used_memory()))

del df
q("t:(); .Q.gc[]")
gc.collect()

print("[conversion][mem:{:.2g}GB] post deletion".format(used_memory()))

$ python mem.py
[mem:0.091GB] start
[mem:1.2GB] post creation
[mem:0.092GB] post deletion
[conversion][mem:0.092GB] start
[conversion][mem:1.1GB] post creation
[conversion][mem:1.1GB] post deletion

I can confirm that my results of the above are broadly in line with the results @antipisa has seen in backtrace mode, this is using the provided script above mem.py with the different results in non backtrace mode as outlined by your analysis @abalkin
With backtrace
$ PYQ_BACKTRACE=y pyq mem.py
[mem:0.046GB] start
[mem:0.14GB] post creation
[mem:0.034GB] post deletion
[conversion][mem:0.034GB] start
[conversion][mem:0.83GB] post conversion
[conversion][mem:0.83GB] post deletion

Without backtrace
$pyq mem.py
[mem:0.046GB] start
[mem:0.032GB] post creation
[mem:0.039GB] post deletion
[conversion][mem:0.039GB] start
[conversion][mem:1GB] post conversion
[conversion][mem:0.57GB] post deletion

This is on a completely fresh environment with the newest version of pyq installed via pip

@cmccarthy1 could you try the first half of the script in pure python?
import gc, os
import numpy as np
import pandas as pd
from numpy.random import choice
import datetime
import psutil

def used_memory():
	process = psutil.Process(os.getpid())
	return (process.memory_info().rss)/2**30

print("[mem:{:.2g}GB] start".format(used_memory()))
n = 50000000
d = datetime.date(2001,1,1)
df = pd.DataFrame(data=dict(x=np.random.randn(n), sym = choice(list("ABCD")), date=d))
print("[mem:{:.2g}GB] post creation".format(used_memory()))

del df
gc.collect()
print("[mem:{:.2g}GB] post deletion".format(used_memory()))

$python mem.py

[mem:0.1GB] start
[mem:1.2GB] post creation
[mem:0.1GB] post deletion

With @antipisa latest script (renamed mem2.py) I see the same results with and without backtrace.
$ pyq mem2.py
[mem:0.046GB] start
[mem:2.3GB] post creation
[mem:1.5GB] post deletion
[conversion][mem:1.5GB] start
[conversion][mem:2.5GB] post conversion
[conversion][mem:2.2GB] post deletion

[mem:0.046GB] start
[mem:2.3GB] post creation
[mem:1.5GB] post deletion
[conversion][mem:1.5GB] start
[conversion][mem:2.5GB] post conversion
[conversion][mem:2.5GB] post deletion

I have modified the mem2.py script to show the k memory usage (from .Q.w[]) and python's count of allocated blocks (sys.getallocatedblocks()).  Here are the results:
$ pyq mem3.py
[start] prss=0.046GB kuse=0.00035GB blks=206036
[post creation] prss=2.3GB kuse=0.00035GB blks=206379
[post deletion] prss=1.5GB kuse=0.00035GB blks=206156
[conversion start] prss=1.5GB kuse=0.00035GB blks=206157
[conversion post conversion] prss=2.5GB kuse=0.5GB blks=206267
[conversion post deletion] prss=2.2GB kuse=0.00035GB blks=206204

As we've seen before the memory is held by python, not q so the next step is to identify the object that holds memory.
I also noticed that the rss number has changed, so I am not sure it can be used as a reliable indicator.  Kernel is allowed to do all kinds of magic when it comes to memory management.  On the other hand "blks" is a good system independent measure.

Moreover, looking at the blks counts for pyq and pure python portions of the script, I notice that behavior is very similar.
I strongly suspect that we are investigating the quirks of python/linux or python/macos memory management systems rather than anything related to pyq.

@albakin @cmccarthy1, I have tested this script in a for loop on a jupyter notebook and in plain pyq, whereupon several hundred conversions of kdb tables into pandas dataframes, the box memory blows up. This does not happen in python, so I can't imagine that it's purely an artifact of python's memory management or a quirk that happens outside of pyq.

@antipisa – did you try to simplify the problematic case?  Do you know which one out of the four table columns causes the trouble?
Why did you experiment in a Jupyter notebook?  Jupiter saves all computation results in history lists and this adds yet another layer of complexity.  Can you come up with a stand-alone script that shows how memory usage grows with repeated conversions?

@abalkin ok, let's forget about jupyter notebook and look at this in plain pyq. Running the modified mem.py script will result in memory growing from 0->n*2GB at each step. In fact for any desired RSS=x size you can modify the number of steps n so that RSS exceeds x at step n. Considering that the box will crash when it is out of memory I think that this is a reasonable indicator.
import gc, os
import numpy as np
import pandas as pd
from numpy.random import choice
import datetime
import psutil

def used_memory():
    process = psutil.Process(os.getpid())
    return (process.memory_info().rss)/2**30

def convert_to_dataframe(K_table):
    df = pd.DataFrame(dict(q("{flip x#y}", q.cols(K_table), K_table)))
    del K_table
    gc.collect()
    q(".Q.gc[]")
    return df

for i in range(100):

    print("[{}][conversion][mem:{:.2g}GB] start".format(i, used_memory()))

    q('''
        N:10000000;
        t:([] symbol:N?`ibm.n`goog.q`fb.q; qty:"f"$N?1000000; date:2007.01.02; ts:`minute$N?700);
        t:update dt:date+ts from t;
        ''')

    df = convert_to_dataframe(q.t)
    print("[{}][conversion][mem:{:.2g}GB] post conversion".format(i, used_memory()))

    del df
    q("t:(); .Q.gc[]")
    gc.collect()

    print("[{}][conversion][mem:{:.2g}GB] post deletion".format(i, used_memory()))

I added a loop around convert_to_dataframe(q.t) in your script:
for i in range(1, 10):
    df = convert_to_dataframe(q.t)
    used_memory(f"conversion post conversion {i}")

and I see that the memory stabilizes after a few iterations:
[start] prss=0.046GB kuse=0.00035GB blks=205820
[post creation] prss=2.3GB kuse=0.00035GB blks=206105
[post deletion] prss=1.5GB kuse=0.00035GB blks=206061
[conversion start] prss=1.5GB kuse=0.00035GB blks=206062
[conversion post conversion 1] prss=2.4GB kuse=0.38GB blks=206164
[conversion post conversion 2] prss=2.7GB kuse=0.38GB blks=206164
[conversion post conversion 3] prss=2.7GB kuse=0.38GB blks=206164
[conversion post conversion 4] prss=2.7GB kuse=0.38GB blks=206164
[conversion post conversion 5] prss=2.7GB kuse=0.38GB blks=206166
[conversion post conversion 6] prss=2.7GB kuse=0.38GB blks=206166
[conversion post conversion 7] prss=2.7GB kuse=0.38GB blks=206166
[conversion post conversion 8] prss=2.7GB kuse=0.38GB blks=206166
[conversion post conversion 9] prss=2.7GB kuse=0.38GB blks=206166
[conversion post deletion] prss=2.7GB kuse=0.38GB blks=206113

Sorry I missed your post – let me try your script.

I ran you script to completion and did not see any increase in memory usage:
$ pyq mem4.py
[0][conversion][mem:0.046GB] start
[0][conversion][mem:1GB] post conversion
[0][conversion][mem:0.73GB] post deletion
[1][conversion][mem:0.73GB] start
[1][conversion][mem:1GB] post conversion
[1][conversion][mem:0.73GB] post deletion
...
[96][conversion][mem:0.73GB] start
[96][conversion][mem:1GB] post conversion
[96][conversion][mem:0.73GB] post deletion
[97][conversion][mem:0.73GB] start
[97][conversion][mem:1GB] post conversion
[97][conversion][mem:0.73GB] post deletion
[98][conversion][mem:0.73GB] start
[98][conversion][mem:1GB] post conversion
[98][conversion][mem:0.73GB] post deletion
[99][conversion][mem:0.73GB] start
[99][conversion][mem:1GB] post conversion
[99][conversion][mem:0.73GB] post deletion

Can you run this in backtrace mode?

Once again, since I cannot reproduce your result, can you help me to narrow down the problem?  Can you reproduce it with a 1-column table rather than 4-column one?

Aha, so we are back talking about backtrace mode:
[0][conversion][mem:0.046GB] start
[0][conversion][mem:1GB] post conversion
[0][conversion][mem:1GB] post deletion
[1][conversion][mem:1GB] start
[1][conversion][mem:2GB] post conversion
[1][conversion][mem:2GB] post deletion
[2][conversion][mem:2GB] start
[2][conversion][mem:3GB] post conversion
[2][conversion][mem:3GB] post deletion
[3][conversion][mem:3GB] start
[3][conversion][mem:3.9GB] post conversion
[3][conversion][mem:3.9GB] post deletion
^C

and we don't have a reproducible issue in default mode, right?

Correct. I do not see this happening in default mode.

@albakin, here is a single date type column repro.

import gc, os
import numpy as np
import pandas as pd
from numpy.random import choice
import datetime
import psutil

def used_memory():
    process = psutil.Process(os.getpid())
    return (process.memory_info().rss)/2**30

def convert_to_dataframe(K_table):
    df = pd.DataFrame(dict(q("{flip x#y}", q.cols(K_table), K_table)))
    del K_table
    gc.collect()
    q(".Q.gc[]")
    return df

for i in range(100):

    print("[{}][conversion][mem:{:.2g}GB] start".format(i, used_memory()))

    q('''
        N:10000000;
        t:([] date:N#2007.01.02);
        ''')

    df = convert_to_dataframe(q.t)
    print("[{}][conversion][mem:{:.2g}GB] post conversion".format(i, used_memory()))

    del df
    q("t:(); .Q.gc[]")
    gc.collect()

    print("[{}][conversion][mem:{:.2g}GB] post deletion".format(i, used_memory()))

Apparently the problem shows up when date/time columns get converted to python:
q.N = 10000000
q.t = q('([]date:N#2007.01.02)')
for i in range(1, 10):
    df = convert_to_dataframe(q.t)
    used_memory(f"conversion post conversion {i}")

$ PYQ_BACKTRACE=y pyq mem5.py
[conversion start] prss=0.046GB kuse=0.00035GB blks=205836
[conversion post conversion 1] prss=0.24GB kuse=0.063GB blks=206067
[conversion post conversion 2] prss=0.39GB kuse=0.063GB blks=206167
[conversion post conversion 3] prss=0.54GB kuse=0.063GB blks=206267
[conversion post conversion 4] prss=0.69GB kuse=0.063GB blks=206367
[conversion post conversion 5] prss=0.84GB kuse=0.063GB blks=206467
[conversion post conversion 6] prss=0.99GB kuse=0.063GB blks=206567
[conversion post conversion 7] prss=1.1GB kuse=0.063GB blks=206667
[conversion post conversion 8] prss=1.3GB kuse=0.063GB blks=206767
[conversion post conversion 9] prss=1.4GB kuse=0.063GB blks=206867
[conversion post deletion] prss=1.4GB kuse=0.063GB blks=206865

It is notable that each conversion results in +100 blocks allocated by python.

Same thing for timestamps, just take
    q('''
        N:10000000;
        t:([] ts:`minute$N?700);
        ''')

Could you share your used_memory function that returns blks and prss?

GiB = 2**30
def used_memory(msg):
    sys._clear_type_cache()
    gc.collect(); q(".Q.gc[]")
    process = psutil.Process(os.getpid())
    prss = process.memory_info().rss/GiB
    kuse = q(".Q.w[]").used / GiB
    blks = sys.getallocatedblocks()
    print(f"[{msg}] {prss=:.2}GB {kuse=:.2}GB {blks=}")

Please note that I also moved gc cleanup to this function.

It also appears to be something specific to pandas because simple conversion of a table column to np.array does not cause  any leakage:
def convert_to_dataframe(K_table):
    return np.asarray(K_table.date)

[conversion start] prss=0.047GB kuse=0.00035GB blks=205836
[conversion post conversion 1] prss=0.17GB kuse=0.063GB blks=205853
[conversion post conversion 2] prss=0.24GB kuse=0.063GB blks=205853
[conversion post conversion 3] prss=0.24GB kuse=0.063GB blks=205853
[conversion post conversion 4] prss=0.24GB kuse=0.063GB blks=205853
[conversion post conversion 5] prss=0.24GB kuse=0.063GB blks=205853
[conversion post conversion 6] prss=0.24GB kuse=0.063GB blks=205853
[conversion post conversion 7] prss=0.24GB kuse=0.063GB blks=205853
[conversion post conversion 8] prss=0.24GB kuse=0.063GB blks=205853
[conversion post conversion 9] prss=0.24GB kuse=0.063GB blks=205853
[conversion post deletion] prss=0.24GB kuse=0.063GB blks=205849

BTW, while it does not fix anything, instead of
df = pd.DataFrame(dict(q("{flip x#y}", q.cols(K_table), K_table)))

you can simply write
df = pd.DataFrame(dict(K_table.flip))

Thanks. Given that it is pandas specific, do you have any idea why it may only happen in backtrace mode?

I don't know what exactly is happening yet, but it looks like I found a workaround. If you rewrite table to dataframe conversion code to explicitly convert columns to numpy arrays, you will eliminate the leaks in backtrace mode.
df = pd.DataFrame({k:np.asarray(K_table[k]) for k in K_table.cols})

This means that most likely pandas attempts some introspection on k objects which causes the leak.  I'll continue the investigation.

@albakin I am aware of this workaround, but unfortunately this method of converting to dataframe is slow for non-numeric datatypes. The fastest method of conversion is by separating types, i.e.
def convert_to_dataframe(K_table):
	all_cols = q.cols(K_table)
	numeric_cols = K_table.meta.exec("c", where='t in "feijhx"')
	df = pd.DataFrame(dict(q("{$[count x; flip x#y; ()!()]}", numeric_cols, K_table)))
	for col in q.except_(all_cols, numeric_cols):
		df[col] = K_table[col]
	return df

Please try the following patch
diff --git a/src/pyq/_k.c b/src/pyq/_k.c
index 5c3400d..9c2a470 100644
--- a/src/pyq/_k.c
+++ b/src/pyq/_k.c
@@ -274,6 +274,7 @@ K_trp(KObject *self, KObject *args)
             Py_DECREF(message);
             Py_DECREF(traceback);
             PyErr_SetObject(ErrorObject, exc_value);
+            Py_DECREF(exc_value);
             return NULL;
         }
     }

If it solves the problem we will prepare a bug-fix release.

@albakin it looks like this fixes it.

Fixed in internal MR 755/issue 1090.


FID error

RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory
How to solve it?

same issue here.
I'm trying to modify the code to windows like suggested here https://github.com/GaParmar/clean-fid/pull/14 and https://github.com/GaParmar/clean-fid/pull/13/commits/13dddbe63bf0cfd57ff8f02e23ab20a263dbeb28 ... It seems that torch is having trouble in finding the location of InceptionV3W file on Windows.
Traceback:
Traceback (most recent call last):
  File "fid.py", line 22, in <module>
    score = fid.compute_fid(fdir1, fdir2, mode="clean", num_workers=2,batch_size=16)
  File "C:\Users\Pichau\anaconda3\lib\site-packages\cleanfid\fid.py", line 384, in compute_fid
    feat_model = build_feature_extractor(mode, device)
  File "C:\Users\Pichau\anaconda3\lib\site-packages\cleanfid\features.py", line 43, in build_feature_extractor
    feat_model = feature_extractor(name="torchscript_inception", resize_inside=False, device=device)
  File "C:\Users\Pichau\anaconda3\lib\site-packages\cleanfid\features.py", line 22, in feature_extractor
    model = InceptionV3W(tmp_dir, download=True, resize_inside=resize_inside).to(device)
  File "C:\Users\Pichau\anaconda3\lib\site-packages\cleanfid\inception_torchscript.py", line 35, in __init__
    self.base = torch.jit.load(path).eval()
  File "C:\Users\Pichau\anaconda3\lib\site-packages\torch\jit\_serialization.py", line 161, in load
    cpp_module = torch._C.import_ir_module(cu, str(f), map_location, _extra_files)
RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory

@Yves907 Similarly on Issue https://github.com/GaParmar/clean-fid/issues/3 to make it work I needed to put the inception-2015-12-05.pt file of --> https://nvlabs-fi-cdn.nvidia.com/stylegan2-ada-pytorch/pretrained/metrics/inception-2015-12-05.pt inside the working directory... So it was something like this:
%cd C:\Users\Pichau\pytorch-CycleGAN-and-pix2pix
fdir1 = "./results/FINAL_TEST/images/_real_B"
fdir2 = "./results/FINAL_TEST/images/_fake_B"

score = fid.compute_fid(fdir1, fdir2, mode="clean", num_workers=0,batch_size=16)
print(f"clean-fid score: {score:.3f}")

Hi,
Thank you for taking a look at the issue and the corresponding pull request.
I have merged the PR and the new clean-fid release should have this issue addressed.
Regards,
Gaurav

I re-installed clean-FID, but this error still occurs. In fact, I ran this code to calculate FID without any errors three months ago, and recently started to get the error.

@GaParmar   I re-installed clean-FID, but this error still occurs. In fact, I ran this code to calculate FID without any errors three months ago, and recently started to get the error.

Hi @Yves907,
Thank you for pointing out the issue!
Can you show the contents of this file?

home/sll/miniconda3/envs/yanjie_torch/lib/python3.6/site-packages/cleanfid/features.py

This will better help me debug why this runtime error is occurring.
Regards,
Gaurav

Hi @GaParmar
These are contents of "feature.py" file
import os
import numpy as np
import torch
import cleanfid
from cleanfid.downloads_helper import check_download_url
from cleanfid.inception_pytorch import InceptionV3
from cleanfid.inception_torchscript import InceptionV3W
"""
returns a functions that takes an image in range [0,255]
and outputs a feature embedding vector
"""
def feature_extractor(name="torchscript_inception", device=torch.device("cuda"), resize_inside=False):
if name == "torchscript_inception":
model = InceptionV3W("/tmp", download=True, resize_inside=resize_inside).to(device)
model.eval()
def model_fn(x): return model(x)
elif name == "pytorch_inception":
model = InceptionV3(output_blocks=[3], resize_input=False).to(device)
model.eval()
def model_fn(x): return model(x/255)[0].squeeze(-1).squeeze(-1)
else:
raise ValueError(f"{name} feature extractor not implemented")
return model_fn
"""
Build a feature extractor for each of the modes
"""
def build_feature_extractor(mode, device=torch.device("cuda")):
if mode == "legacy_pytorch":
feat_model = feature_extractor(name="pytorch_inception", resize_inside=False, device=device)
elif mode == "legacy_tensorflow":
feat_model = feature_extractor(name="torchscript_inception", resize_inside=True, device=device)
elif mode == "clean":
feat_model = feature_extractor(name="torchscript_inception", resize_inside=False, device=device)
return feat_model
"""
Load precomputed reference statistics for commonly used datasets
"""
def get_reference_statistics(name, res, mode="clean", seed=0, split="test", metric="FID"):
base_url = "https://www.cs.cmu.edu/~clean-fid/stats/"
if split == "custom":
res = "na"
if metric == "FID":
rel_path = (f"{name}{mode}{split}{res}.npz").lower()
url = f"{base_url}/{rel_path}"
mod_path = os.path.dirname(cleanfid.file)
stats_folder = os.path.join(mod_path, "stats")
fpath = check_download_url(local_folder=stats_folder, url=url)
stats = np.load(fpath)
mu, sigma = stats["mu"], stats["sigma"]
return mu, sigma
elif metric == "KID":
rel_path = (f"{name}{mode}{split}{res}_kid.npz").lower()
url = f"{base_url}/{rel_path}"
mod_path = os.path.dirname(cleanfid.file)
stats_folder = os.path.join(mod_path, "stats")
fpath = check_download_url(local_folder=stats_folder, url=url)
stats = np.load(fpath)
return stats["feats"]
Thank you for your consideration.

It looks like the Inception model was not downloaded correctly.
Can you check if the file /tmp/inception-2015-12-05.pt exists?

Hi @GaParmar
The code runs successfully on my win10, but cannot work well on the linux. I have checked that the "inception-2015-12-05.pt" exists. Next, I will completely clear the file and reinstall it again, I think I can solve the problem this time.
Thank you!

@Yves907 Hi, I'm using ubuntu 20.04 LTS and CUDA 11.6. I suffered from same error, [RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory] although I ahve inception-2015-12-05.pt file in /tmp/
I fixed it by removing the file and run fid.compute_fid(dir1, dir2). I guess that I'm using other fid calculation packags so those packages already downloaded inception model in /tmp/, so that clean-fid package may suffer conflicts in downloading or may not recognize existing inception-2015-12-05.pt file because it's downloaded by other package (may be accessibility issue?).
I feel this package very useful, so hope that this error never happen again...
@GaParmar Thank you for nice project!

Thank you for pointing these details out!
I have noticed that sometime the download of the inception file gets corrupted if the internet connection is interrupted midway. I will add a check for this that will re-download the file in the next version of the library.

Gaurav

Removing the inception-2015-12-05.pt in the /tmp/ folder works for me.

In the Linux system, the same error is reported：
Traceback (most recent call last):
File "/home/genggeng/SDD_svd_contra-main/myeval_fid.py", line 64, in
f = fid.compute_fid(temp_dir1, temp_dir2)
File "/home/genggeng/.conda/envs/SDD/lib/python3.9/site-packages/cleanfid/fid.py", line 465, in compute_fid
feat_model = build_feature_extractor(mode, device, use_dataparallel=use_dataparallel)
File "/home/genggeng/.conda/envs/SDD/lib/python3.9/site-packages/cleanfid/features.py", line 46, in build_feature_extractor
feat_model = feature_extractor(name="torchscript_inception", resize_inside=False, device=device, use_dataparallel=use_dataparallel)
File "/home/genggeng/.conda/envs/SDD/lib/python3.9/site-packages/cleanfid/features.py", line 21, in feature_extractor
model = InceptionV3W(path, download=True, resize_inside=resize_inside).to(device)
File "/home/genggeng/.conda/envs/SDD/lib/python3.9/site-packages/cleanfid/inception_torchscript.py", line 35, in init
self.base = torch.jit.load(path).eval()
File "/home/genggeng/.conda/envs/SDD/lib/python3.9/site-packages/torch/jit/_serialization.py", line 162, in load
cpp_module = torch._C.import_ir_module(cu, str(f), map_location, _extra_files, _restore_shapes)  # type: ignore[call-arg]
RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory
and the file "inception-2015-12-05.pt" does not exist. How to deal with it?

在 Linux 系统中，报告了相同的错误：
回溯（最近一次调用）：文件“/home/genggeng/SDD_svd_contra-main/myeval_fid.py”，第 64 行，f = fid.compute_fid（temp_dir1，temp_dir2） 文件“/home/genggeng/.conda/envs/SDD/lib/python3.9/site-packages/cleanfid/fid.py”，第 465 行，在 compute_fid 中 feat_model = build_feature_extractor（mode， device， use_dataparallel=use_dataparallel）文件“/home/genggeng/.conda/envs/SDD/lib/python3.9/site-packages/cleanfid/features.py”，第 46 行，在build_feature_extractor feat_model = feature_extractor（name=“torchscript_inception”， resize_inside=False， device=device， use_dataparallel=use_dataparallel） 文件“/home/genggeng/.conda/envs/SDD/lib/python3.9/site-packages/cleanfid/features.py”，第 21 行，在 feature_extractor 模型中 = InceptionV3W（path， download=True， resize_inside=resize_inside）.to（device） 文件 “/home/genggeng/.conda/envs/SDD/lib/python3.9/site-packages/cleanfid/inception_torchscript.py”，第 35 行，init self.base = torch.jit.load（path）.eval（） 文件 “/home/genggeng/.conda/envs/SDD/lib/python3.9/site-packages/torch/jit/_serialization.py”，第 162 行，加载 cpp_module = torch._C.import_ir_module（cu， str（f）， map_location， _extra_files， _restore_shapes） # type： ignore[call-arg]运行时错误：PytorchStreamReader 读取 zip 存档失败：查找中央目录失败
并且文件“inception-2015-12-05.pt”不存在。如何处理？

maybe the xxx.pth is bad, you could delete it from the /tmp and then download it from scratch...


Mobile Responsiveness

fix aboutme section title in 320px-480px

updated directly to main branch. Looks GOOD ✨

updated directly to main branch. Looks GOOD ✨


[BUG]: On the eight-card A100, testing the 'examples/language/llama2' with the 'gemini_auto' plugin resulted in an 'out of memory' error."

🐛 Describe the bug
Here are my script,  it can run with hybrid_parallel plugin, but other plugins have the same error "out of memory"
torchrun --standalone --nproc_per_node 8 finetune.py
--plugin "gemini_auto"
--dataset "self_instruct"
--model_path "Llama2-Chinese-7b-Chat"
--task_name "finetuning"
--batch_size 2
--save_dir "output_test"
Environment
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB (GPU 4; 79.21 GiB total capacity; 75.40 GiB already allocated; 1.74 GiB free; 76.12 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Hi, how about trying to set offload_optim_frac and offload_param_frac to 1.0?


State Framework similar to Fish

Dear @o1298098 ,
I would like to ask you about fish-redux. I feel that their maintenance is very slow, and almost no feedback about the null safety issues posted there.

do you think that fish-redux is being abandoned ?
what do you think about an idea: to use https://pub.dev/packages/event_bus and manage the state in the same way fish redux is managing, keeping the state, effect, action pattern, but using the bus to send back and forward the communication between the state and view ?
what is your opinion about riverpod as a possible candidate to change fish-redux ?

regards
hh

Dear @o1298098 ,
I would like to ask you about fish-redux. I feel that their maintenance is very slow, and almost no feedback about the null safety issues posted there.

In the past six months, fish-redux has not supported null safety, and it has not been updated for a long time. I think fish-redux should be abandoned.

do you think that fish-redux is being abandoned ?
what do you think about an idea: to use https://pub.dev/packages/event_bus and manage the state in the same way fish redux is managing, keeping the state, effect, action pattern, but using the bus to send back and forward the communication between the state and view ?
what is your opinion about riverpod as a possible candidate to change fish-redux ?

I prefer to use Mobx combined Provider for state management.

@o1298098  I am adapting this approach: https://medium.com/flutter-community/flutter-state-management-for-minimalists-4c71a2f2f0c1 to be used in an almost identical approach used by fish, but.... without any framework, super clean, O(1) performance, only using plain dart. This is my first scratch: https://github.com/insurdao/flutter-boilerplate


Support PrivilegeChecker in new parser and fix create analyze fail bug

What type of PR is this：

[x] bug
[ ] feature
[ ] enhancement
[ ] others

Which issues of this PR fixes ：

Fixes #4099
Problem Summary(Required) ：

I don't get where fix the create analyze fail bug

run starrocks_fe_unittest

I don't get where fix the create analyze fail bug

In the 85th line of PrivilegeChecker, except for the ddl in the if, all will report the error Unsupported DdlStmt. This is not correct.

run starrocks_fe_unittest

run starrocks_fe_unittest

[FE PR Coverage check]
:disappointed: fail : 48 / 76 (63.16%)
file detail

path
covered line
new line
coverage

:large_blue_circle:
com/starrocks/analysis/DdlStmt.java
0
1
00.00%

:large_blue_circle:
com/starrocks/sql/ast/AstVisitor.java
3
6
50.00%

:large_blue_circle:
com/starrocks/sql/analyzer/PrivilegeChecker.java
25
47
53.19%

:large_blue_circle:
com/starrocks/sql/analyzer/AnalyzerUtils.java
11
13
84.62%

:large_blue_circle:
com/starrocks/qe/StmtExecutor.java
1
1
100.00%

:large_blue_circle:
com/starrocks/sql/StatementPlanner.java
1
1
100.00%

:large_blue_circle:
com/starrocks/analysis/CreateViewStmt.java
1
1
100.00%

:large_blue_circle:
com/starrocks/sql/analyzer/CTASAnalyzer.java
5
5
100.00%

:large_blue_circle:
com/starrocks/analysis/AlterViewStmt.java
1
1
100.00%


Service not functional after canary deleted

I have created a canary for my service, and after I delete it, the service still has the selector from when the canary was in place:
selector:
    app.kubernetes.io/name: basic-demo-primary

Flagger 0.17.0 with linkerd 2.4.0 and kube 1.12.10

When you delete a canary that service should also be deleted. You’ve created the ClusterIP on your own? For canaries Flagger creates the services for you, only the deployment is needed.

I see, so the recommended approach is to just not create a service at all?  I was attempting to add this to an already existing service with no downtime.  This means that a service already existed, as did an ingress.
My hope was that for customers of mine who want to add canary deployments, we could implement this and migrate them seamlessly.

I'll provide some more details of my observation since I may have been a little vague in my initial report.
When I applied the canary to the deployment that already had a service, it seems that Flagger modifies the existing service in addition to creating the new ones.
apiVersion: flagger.app/v1alpha3
kind: Canary
metadata:
  name: demo
  namespace: demo
spec:
  # deployment reference
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: basic-demo
  # HPA reference (optional)
  autoscalerRef:
    apiVersion: autoscaling/v1
    kind: HorizontalPodAutoscaler
    name: basic-demo
  ingressRef:
    apiVersion: extensions/v1beta1
    kind: Ingress
    name: basic-demo
  # the maximum time in seconds for the canary deployment
  # to make progress before it is rollback (default 600s)
  progressDeadlineSeconds: 600
  service:
    # container port
    port: 8080
  canaryAnalysis:
    # schedule interval (default 60s)
    interval: 60s
    # max number of failed metric checks before rollback
    threshold: 5
    # max traffic percentage routed to canary
    # percentage (0-100)
    maxWeight: 50
    # canary increment step
    # percentage (0-100)
    stepWeight: 5
    # Linkerd Prometheus checks
    metrics:
    - name: request-success-rate
      # minimum req success rate (non 5xx responses)
      # percentage (0-100)
      threshold: 99
      interval: 1m
    - name: request-duration
      # maximum req duration P99
      # milliseconds
      threshold: 500
      interval: 30s
    # testing (optional)
    webhooks:
      - name: acceptance-test
        type: pre-rollout
        url: http://loadtester.demo/
        timeout: 30s
        metadata:
          type: bash
          cmd: "curl -sd 'test' https://sudermanjr-demo.hillghost.com/"
      - name: load-test
        type: rollout
        url: http://loadtester.demo/
        metadata:
          cmd: "hey -z 2m -q 10 -c 2 https://sudermanjr-demo.hillghost.com/"

Application chart is here:
https://github.com/FairwindsOps/charts/tree/master/incubator/basic-demo
The values for that chart are as follows:
linkerd:
  serviceProfile: false
  enableRetry: false
demo:
  title: "Testing Demo"
  metadata: "Brought to you by Fairwinds"
  refreshInterval: 400
hpa:
  cpuTarget: 30
  customMetric:
    enabled: false
    metric: route_response_latency_ms_99th_get_ping
    target: 30
  max: 40
  min: 3
vpa:
  enabled: false
ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: "nginx-ingress"
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    certmanager.k8s.io/acme-challenge-type: dns01
    certmanager.k8s.io/acme-dns01-provider: route53
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:80;
      proxy_hide_header l5d-remote-ip;
      proxy_hide_header l5d-server-id;
  paths:
    - /
  hosts:
    - sudermanjr-demo.hillghost.com
  tls:
    - secretName: basic-demo-tls
      hosts:
        - sudermanjr-demo.hillghost.com
resources:
  limits:
    cpu: 3m
    memory: 131072k
  requests:
    cpu: 3m
    memory: 131072k

The service before the canary is deployed:
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: 2019-07-12T14:47:05Z
  labels:
    app.kubernetes.io/instance: basic-demo
    app.kubernetes.io/managed-by: Tiller
    app.kubernetes.io/name: basic-demo
    helm.sh/chart: basic-demo-0.2.1
  name: basic-demo
  namespace: demo
  resourceVersion: "6444810"
  selfLink: /api/v1/namespaces/demo/services/basic-demo
  uid: eae4af81-a4b3-11e9-a9b9-06d1b2b834b8
spec:
  clusterIP: <IP_ADDRESS>
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  selector:
    app.kubernetes.io/instance: basic-demo
    app.kubernetes.io/name: basic-demo
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

Service after the canary is applied (in addition to the two services created by Flagger)
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: 2019-07-12T14:47:05Z
  labels:
    app.kubernetes.io/instance: basic-demo
    app.kubernetes.io/managed-by: Tiller
    app.kubernetes.io/name: basic-demo
    helm.sh/chart: basic-demo-0.2.1
  name: basic-demo
  namespace: demo
  resourceVersion: "6445292"
  selfLink: /api/v1/namespaces/demo/services/basic-demo
  uid: eae4af81-a4b3-11e9-a9b9-06d1b2b834b8
spec:
  clusterIP: <IP_ADDRESS>
  ports:
  - name: http
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app.kubernetes.io/name: basic-demo-primary
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

These changes are not reverted after deleting the canary.
Looking at it now, I can see how this is intended to work without a pre-existing service.  However, I wonder if there is a way to make this work seamlessly with an existing service for migration/implementation purposes.

Flagger can take over ClusterIP service so I see no issues on migrating to Flagger without downtime. The canary deletion is different problem, when you delete a canary Kubernetes GC removes all objects owned by Flagger but since the ClusterIP service is not owned it doesn’t get deleted. I’ll try to fix this in a future release.

In the case of a service that Flagger takes over, I wouldn't expect deleting the canary to delete the service, but to put it back to it's original configuration.
Thanks for the response.  It's good to know that this is understood/expected behavior right now.


Paragraph indent as a single token

Is your feature request related to a problem? Please describe.
It's complicated and redundant to have an extra typography style everywhere just to have extra paragraph indent.
Describe the solution you'd like
A separate single token to apply like Paragraph Spacing or Text Decoration would be great.

We'll let you use dimension tokens for this in the next (major) release, but there won't be a dedicated token for paragraphIndent.


fix: show shanghai meetup events

Fixes: #464
Changes:
Add a link to meetup in the events list.

The error log is like this PR has only one non-merge commit and it's not semantic; add another commit before squashing.
It seems that this PR has only one commit.

seems that commit message is not very standard, so

failed, please try to update your commit message something like fix: xx @bzp2010

Fixed.


onDisable not executed

Just found out that the applications onDisable method is not being executed.

Please don't merge yet since I'm currently working on some more bugs involving this topic.


added new test to status-router-test

Coverage remained the same at 92.268% when pulling f8ca2832a90d7a10be61a60d6e4db295c8c42f5f on group-issue-99 into<PHONE_NUMBER>93a776c5b9c51ec96b187ca13d9074 on staging-branch.


updates NooBaa CRDs to latest available spec

runs make gen-latest-csv for update which pulls latest
available CRD for NooBaa v2.3.0
Signed-off-by: Umanga Chapagain<EMAIL_ADDRESS>

/test verify-latest-csv


Is it possible to probe event loop to see if anything is running?

In Node.js:
setTimeout(function () {
  console.log(process._getActiveHandles().length)
}, 5000)

Would output 1 as the setTimeout is still on the event loop. However, in the renderer process, this returns 0. Is there a way to tap into the event loop (presumably v8) and check if anything is pending?

If I understand this correctly, does it mean CLI tools like electron-spawn could use this to auto-quit?
If so :+1: this feature would be amazing.

Electron itself uses an async handle to implement Node.js integration so that's why process._getActiveHandles() is ways >=1 handle.
But even if we avoid creating active handles in Node.js, the Chromium itself still maintains a quite complex message loop, which means even if there is nothing active in Node.js we might still have lots of things running in Chromium.

I'm closing this as won't fix, because even though we can check whether there is things running in Node, we can not do it for Chromium's message loops. Many tasks of Chromium are designed to run forever, so the state of message loop can only be always running.


[Delete record] Add ability to remove records

Only accessible to creator of record and project managers.

Will also need to implement in local mutation queue and remote sync


Http Basic Authentication using port :80:80 and giving false positive results

Please Describe the issue or question and share your OS and Python version.

OS: Linux
OS Version: Linux Ubuntu
Python Version: 2.7.3

I did not consider into the possibililty when the mentioned host doesn't have the authentiation. Thanks for telling me a corner case. I will send a fix soon!! :smile:

I suggest you should use Owasp-DVWA for testing web modules.
Best Regards
Pradeep Jairamani

Hello,
Thanks for reporting this, I think to give the framework :80:80 is related to the framework to detect the false target since we have a topic related to this module with the bug, I'd close this and open another one regarding the framework target types. let me know if you have any ideas.
Best Regards.


Doctor warning about @vue/compiler-dom when vue-tsc is installed in 2.7 project.

I'm getting the following doctor warning because vue-tsc has @vue/compiler-dom as an indirect dependency, but the warning is telling me to remove compiler-dom from my package.json.
HelloWorld.vue Doctor
❗ Unnecessary @vue/compiler-dom
Vue 2.7 already includes JSX type definitions. You can remove the @vue/compiler-dom dependency from package.json.

vue: c:\src\MyProject\node_modules\vue\package.json
@vue/compiler-dom<EMAIL_ADDRESS>
> npm ls @vue/compiler-dom
<EMAIL_ADDRESS>C:\src\MyProject
└─┬<EMAIL_ADDRESS>  └─┬<EMAIL_ADDRESS>    ├──<EMAIL_ADDRESS>    └─┬<EMAIL_ADDRESS>      ├──<EMAIL_ADDRESS>deduped
      └─┬<EMAIL_ADDRESS>        └──<EMAIL_ADDRESS>deduped
``

this issue still exists 🥲

this issue still exists 🥲这个问题依然存在🥲

How did you solve the problem

@vue/runtime-dom is not part of vue-tsc anymore so this issue is fixed. If you have it in your project then it might be coming from some other package. Try running npm ls @vue/runtime-dom.


fix remove ShortNameResolver to solve thread race problem

What problem does this PR solve?
Problem Summary: remove ShortNameResolver to solve thread race problem
Release note
None
Check List (For Author)

Test

[ ] Regression test
[ ] Unit Test
[x] Manual test (add detailed scripts or steps below)
[ ] No need to test or manual test. Explain why:

[ ] This is a refactor/code format and no logic has been changed.
[ ] Previous test can cover this change.
[ ] No code files have been changed.
[ ] Other reason

Behavior changed:

[x] No.
[ ] Yes.

Does this need documentation?

[x] No.
[ ] Yes.

Check List (For Reviewer who merge this PR)

[ ] Confirm the release note
[ ] Confirm test cases
[ ] Confirm document
[ ] Add branch pick label

run buildall


Content negotation

We have some of the raw ingredients of content negotiation, but we lack helpers or idioms for it.  This is something a lot of people need.  If there's a widely agreeable approach, it should be in the library.  If there are lots of ideas, we might at least consider some sort of cookbook.
Here's a starting point, courtesy of @shajra.

I updated the gist.  I think the API is stabilizing, but it needs a ton of testing (getting the spec right is non-trivial).  I put a method in the companion object to help create ContentNegotiates from EntityEncoders.  I'm not approaching this work from the perspective of a pull request (which would be a larger overhaul --probably replacing EntityEncoder.  At my work, we're doing this work more as an extension of the version of HTTP4s that we're pinned to for a moment.  Mostly, I want to get some work out there as a reference so someone doesn't have to implement everything from scratch.
I think hooking in this into the DSL won't be too crazy.  I'll probably make my own version of EntityResponseGenerator that will either return a normal response or a 406 if the ContentNegotiate returns an empty Maybe.

Revived discussion on Gitter.