Azure AI ML: Creating a pipeline component with multiple inputs

Is your feature request related to a problem? Please describe.
Hello! I'm trying to use the Python SDK v2 for ML Studio and I need to run a pipeline on data from a blob storage location. My storage location is divided into folders and each folder contains multiple files. I want to run my pipeline on, say, 20 of these folders, and not the other 50. However, 20 is just an example, it's variable and could as easily be 50 or 100.
Describe the solution you'd like
I would like to pass multiple paths to a single Input object.
Describe alternatives you've considered
There may be some way of doing this by first registering a v1 dataset with multiple paths, then getting this dataset by name in v2. I haven't managed it yet. But even if that is possible, it's a horrible approach and requires mixing v1 and v2, which doesn't seem sensible. And I can't use v1 for the whole thing since it is missing some other requirements.
And in case it's suggested, I can't filter my files inside the script as I'm using a parallel component which automatically splits the data.

@azureml-github

Hi! One option you can try is using mltable. Please check this doc for reference: https://docs.microsoft.com/en-us/azure/machine-learning/how-to-create-register-data-assets?tabs=CLI#create-a-mltable-data-asset

Hi @yongjiaaaa, thanks for the suggestion! I thought of that too since the old Dataset object seems to be backed by MLTable now (I've actually been reading that page to try to find a solution). However, my files are image files, not parts of a tabular dataset, and I can't find a way of getting this to work. I imagine it must be possible somehow to get an MLTable dataset to act as a file dataset, since this is what v1 does, but I haven't managed.

Hey @davystrong , while MLTable has tabular in mind, the transformations section is optional. So to use multiple paths you can do something like:
MLTable:
paths:
  - file: ./file1.jpg
  - file: ./file2.png

cc @SturgeonMi for doc updates, adding the MLTable schema JSON and calling out the required/optional portions. Thanks!!

Hi @Man-MSFT, my main problem was actually slightly further on. I finally worked out today that the path when defining the input must point to a folder containing the MLTable file, not to the MLTable file itself, even though I don't actually include any local files in my MLTable. I understand why this is but it would be helpful to mention this specifically in the docs (it's the same for the YAML schema for a pipeline job, which I had been trying to get working before moving to Python). Unless, of course, I missed it!

@davystrong I see we added a purple note here that path must point to the folder. But that is good feedback and we will see if we can make it more obvious. Thanks again!
Let me know if there's anything else, otherwise will consider the matter resolved.

@Man-MSFT Well, not sure how I missed that! Not sure how you could make it more obvious. That's exactly what I needed. Thanks for your help!


Warn user about /opt/homebrew not being properly set in path

Because Homebrew was set on my PATH as /System/Volumes/Data/opt/homebrew and this actually /opt/homebrew, it was mostly working, installation and installed software execution were fine, but it wasn't able to compile some dependencies unless I manually created some dirs before, as I was always having this error:
Last 15 lines from /Users/antonioribeiro/Library/Logs/Homebrew/guile/02.make:
 .././build-aux/install-sh -c -d '/System/Volumes/Data/opt/homebrew/Cellar/guile/3.0.8_2/share/aclocal'
 .././build-aux/install-sh -c -d '/System/Volumes/Data/opt/homebrew/Cellar/guile/3.0.8_2/lib/pkgconfig'
mkdir: /System/Volumes/Data/opt/homebrew/Cellar/guile/3.0.8_2/lib: Operation not permitted
mkdir: /System/Volumes/Data/opt/homebrew/Cellar/guile/3.0.8_2/bin: Operation not permitted
mkdir: /System/Volumes/Data/opt/homebrew/Cellar/guile/3.0.8_2/share: Operation not permitted
mkdir: /System/Volumes/Data/opt/homebrew/Cellar/guile/3.0.8_2/lib: Operation not permitted
mkdir: /System/Volumes/Data/opt/homebrew/Cellar/guile/3.0.8_2/share: Operation not permitted
mkdir: /System/Volumes/Data/opt/homebrew/Cellar/guile/3.0.8_2/bin: Operation not permitted
make[3]: *** [install-pkgconfigDATA] Error 1
make[3]: *** Waiting for unfinished jobs....
make[3]: *** [install-aclocalDATA] Error 1
make[3]: *** [install-binSCRIPTS] Error 1
make[2]: *** [install-am] Error 2
make[1]: *** [install-recursive] Error 1
make: *** [install] Error 2

Do not report this issue to Homebrew/brew or Homebrew/core!

Possibly because, /System is writable by the root only, even if permissions on /opt/homebrew would allow mkdir from the user running brew install.
After running brew doctor I found this strange message saying Your Homebrew's prefix is not /opt/homebrew, as it was, and digging a little more issues here, found that prefix is being (somehow) computed by the PATH, probably because the PATH locating and running it from /System/Volumes/Data/opt/homebrew/bin/brew instead of /opt/homebrew/bin/brew.
I believe this extra warning message could also save other people some research time.

Thanks for the PR! Passing on this because I'm not sure the extra messaging is worth it for a single case where you've solved the problem and the previous messaging helped you figure this out. Sorry!


Support for ROIAlign Layer

Fixes and improvements to ROIAlign forward pass.
Working implementation and improvements of ROIAlign backwards for GPU & CPU.
Tests based on results from Caffe2 to verify correctness.

Support for autograd.gradcheck is still needed.

This is awesome, thanks!
There are some conflicts with this branch, can you rebase and add an autograd.gradcheck test, and the this is good to merge

@fmassa done and done!

Merged https://github.com/pytorch/vision/pull/632, so this is redundant, right?

Yup. Closing this.


The latest version starts with an error

time="2024-06-20T02:31:55.386Z" level=error msg="History: Skipping catchup: incompatible core version or invalid local state" pid=37 service=ingest subservice=stellar-core
time="2024-06-20T02:32:01.533Z" level=info msg="Herder: Quorum information for 52201745 : {"agree":23,"cost":30933344,"delayed":0,"disagree":0,"fail_at":6,"hash":"5c464e","lag_ms":905,"ledger":52201745,"missing":0,"phase":"EXTERNALIZE"}" pid=37 service=ingest subservice=stellar-core
time="2024-06-20T02:32:01.536Z" level=info msg="Ledger: Got consensus: [seq=52201747, prev=989c00, txs=531, ops=805, sv: [ SIGNED@Whalestack (Germany) txH: 5fab13, ct:<PHONE_NUMBER>, upgrades: [ ] ]]" pid=37 service=ingest subservice=stellar-core
time="2024-06-20T02:32:01.536Z" level=info msg="Ledger: Close of ledger 52201747 buffered. mSyncingLedgers has 20 ledgers" pid=37 service=ingest subservice=stellar-core
time="2024-06-20T02:32:01.536Z" level=error msg="History: Skipping catchup: incompatible core version or invalid local state" pid=37 service=ingest subservice=stellar-core
time="2024-06-20T02:32:07.164Z" level=info msg="Herder: Quorum information for 52201746 : {"agree":23,"cost":34158088,"delayed":0,"disagree":0,"fail_at":6,"hash":"5c464e","lag_ms":905,"ledger":52201746,"missing":0,"phase":"EXTERNALIZE"}" pid=37 service=ingest subservice=stellar-core
time="2024-06-20T02:32:07.167Z" level=info msg="Ledger: Got consensus: [seq=52201748, prev=1c5372, txs=277, ops=852, sv: [ SIGNED@Blockdaemon Validator 2 txH: f9bd1a, ct:<PHONE_NUMBER>, upgrades: [ ] ]]" pid=37 service=ingest subservice=stellar-core
time="2024-06-20T02:32:07.168Z" level=info msg="Ledger: Close of ledger 52201748 buffered. mSyncingLedgers has 21 ledgers" pid=37 service=ingest subservice=stellar-core
time="2024-06-20T02:32:07.168Z" level=error msg="History: Skipping catchup: incompatible core version or invalid local state" pid=37 service=ingest subservice=stellar-core

imaiops What versions of stellar-horizon and stellar-core are you running?


Create a Sort functionality

Is your feature request related to a problem? Please describe.
Create a sort option so that users can sort the products from low to high or high to low by price.
Describe the solution you'd like
Here is the sort option in amazon.in

Implement the same functionality in Freshbey. As of now we only need "low to high" and "high to low" by price options in the sort dropdown.
Describe alternatives you've considered
No response
Additional context
No response

@roopeshsn assign this to me

Hi, @kanavmittal Check the README.md for the styling frameworks used.

Got it

@roopeshsn should it be " low to high " or " high to low " or none by default

By default it should be none.

completed sorting, will create pr after approval

This issue is up for grabs!

Hi, @silvina-varela! Yeah, you can work on this!


Remove purge step from build workflow.

Task
Remove the following "purge" step from the build workflow:
https://github.com/jonathanknowles/haskell-example/blob/ed8660f6ef691559bd0e473eabfbe96e6cd9608e/.github/workflows/build.yml#L38-L44
Dependencies
This task can only be performed after https://github.com/actions/runner-images/pull/6972 has been merged.
Context
The "purge" step was introduced to work around issues caused by incomplete installations of GHC on certain GitHub runner images, where documentation for GHC core libraries (such as base) is completely missing:
https://github.com/haskell/actions/issues/176
The root cause is a section of code that deletes documentation for all GHC core libraries:
https://github.com/actions/runner-images/blob/c6393f1f231a4bce8aaeb2665d1312b14271169b/images/linux/scripts/installers/haskell.sh#L34-L39
The lack of documentation for GHC core libraries makes it impossible to build haddock documentation with working hyperlinks to those core libraries.

Fixed by #28.


no_cursor_warps prevents focusing window on another monitor

Hyprland Version
Hyprland, built from branch  at commit b08b72358ad549fd066e5be0fc3aa4c9df367607 dirty (). Tag: v0.27.2  flags: (if any)
Bug or Regression?
Bug
Description
When no_cursor_warps is enabled you can't switch to another monitor where the mouse isn't, as the focus is instantly given back to the window the cursor is hovering.
The expected behavior should be that a window on the other monitor stays focused when switching.
How to reproduce

Have 2 monitors
Open a window on each monitor
Hover over a window on one monitor, then change workspace to the one on the other monitor
Observe the border briefly changing to focused, before focus is given back to the other workspace

Crash reports, logs, images, videos
No response

cannot repro

I am experiencing the same issue.
Here is my hyprland.conf (pinned to when I was experiencing the issue).
Output of hyprctl version:
Hyprland, built from branch  at commit<PHONE_NUMBER>83f8e986f9fb2a37a13e9b3a232cc2  ().
Tag: ?

flags: (if any)
debug

Using the Nix flake version from revision<PHONE_NUMBER>83f8e986f9fb2a37a13e9b3a232cc2, with enableNvidiaPatches = true;

nop, nothing
I have no idea what I'm missing

I should note that I also have WLR_NO_HARDWARE_CURSORS set to 1 (not through hyprland.conf). I have some additional configuration appended to my config file, generated through Nix (though I'm doubtful it makes any difference). You can try it if you feel like testing this further (you should omit the first line):
exec-once=/nix/store/2g5287nbipd9xr1mqjp1m9pmg292nhal-dbus-1.14.8/bin/dbus-update-activation-environment --systemd DISPLAY WAYLAND_DISPLAY HYPRLAND_INSTANCE_SIGNATURE XDG_CURRENT_DESKTOP && systemctl --user start hyprland-session.target
input {
    kb_layout = ee
    kb_variant = us
    kb_options = caps:ctrl_modifier
}

layerrule = blur, (waybar|notifications|gtk-layer-shell|rofi)

general {
    gaps_out = 10
    gaps_in = 5

    border_size = 1
    col.active_border = rgb(f5bde6) rgb(c6a0f6) rgb(8bd5ca) rgb(91d7e3) rgb(7dc4e4) rgb(8aadf4) rgb(b7bdf8)
    col.inactive_border = rgb(494d64)
}

decoration {
    rounding = 1;
    blur {
        enabled = true
        size = 8
        passes = 3
    }
}

bezier = overshot, 0.05, 0.9, 0.1, 1.1
bezier = easeInOutQuint, 0.86, 0, 0.07, 1
bezier = easeOutExpo, 0.19, 1, 0.22, 1

animations {
    animation = workspaces, 1, 3, easeInOutQuint, slidevert
    animation = windows, 1, 4, easeOutExpo, popin
}

source = ~/.config/hypr/monitors.conf

binde = SUPER, apostrophe, exec, kitty

# Prefer high refresh rate
monitor=,highrr,auto,1

input {
    # 0 - Cursor movement will not change focus.
    # 1 - Cursor movement will always change focus to the window under the cursor.
    # 2 - Cursor focus will be detached from keyboard focus. Clicking on a window will move keyboard focus to that window.
    # 3 - Cursor focus will be completely separate from keyboard focus. Clicking on a window will not change keyboard focus.
    follow_mouse = 2
}

general {
    layout = master
}

master {
    mfact = 0.5
}

misc {
    animate_manual_resizes = true
    animate_mouse_windowdragging = true
    enable_swallow = true
    swallow_regex = ^(Alacritty)$
}

binde = SUPER, C, killactive,
binde = SUPER, V, togglefloating,
binde = SUPER, F, fullscreen, 0 # Fullscreen
binde = SUPER, U, fullscreen, 1 # Maximized

binde = SUPER, H, movefocus, l
binde = SUPER, J, layoutmsg, cyclenext
binde = SUPER, K, layoutmsg, cycleprev
binde = SUPER, L, movefocus, r

binde = SUPER SHIFT, H, resizeactive, -60 0
binde = SUPER SHIFT, J, resizeactive, 0 60
binde = SUPER SHIFT, K, resizeactive, 0 -60
binde = SUPER SHIFT, L, resizeactive, 60 0

binde = SUPER, return, layoutmsg, swapwithmaster
binde = SUPER SHIFT, return, layoutmsg, focusmaster

binde = SUPER, space, layoutmsg, swapnext
binde = SUPER SHIFT, space, layoutmsg, swapprev

binde = CTRL SUPER, H, moveactive, -60 0
binde = CTRL SUPER, J, moveactive, 0 60
binde = CTRL SUPER, K, moveactive, 0 -60
binde = CTRL SUPER, L, moveactive, 60 0

bind = SUPER ALT, W, layoutmsg, orientationtop
bind = SUPER ALT, A, layoutmsg, orientationleft
bind = SUPER ALT, S, layoutmsg, orientationbottom
bind = SUPER ALT, D, layoutmsg, orientationright
bind = SUPER ALT, C, layoutmsg, orientationcenter

bind = SUPER, 1, workspace, 1
bind = SUPER, 2, workspace, 2
bind = SUPER, 3, workspace, 3
bind = SUPER, 4, workspace, 4
bind = SUPER, 5, workspace, 5
bind = SUPER, 6, workspace, 6
bind = SUPER, 7, workspace, 7
bind = SUPER, 8, workspace, 8
bind = SUPER, 9, workspace, 9
bind = SUPER, 0, workspace, 10

binde = SUPER SHIFT, 1, movetoworkspacesilent, 1
binde = SUPER SHIFT, 2, movetoworkspacesilent, 2
binde = SUPER SHIFT, 3, movetoworkspacesilent, 3
binde = SUPER SHIFT, 4, movetoworkspacesilent, 4
binde = SUPER SHIFT, 5, movetoworkspacesilent, 5
binde = SUPER SHIFT, 6, movetoworkspacesilent, 6
binde = SUPER SHIFT, 7, movetoworkspacesilent, 7
binde = SUPER SHIFT, 8, movetoworkspacesilent, 8
binde = SUPER SHIFT, 9, movetoworkspacesilent, 9
binde = SUPER SHIFT, 0, movetoworkspacesilent, 10

bind = SUPER CTRL, H, movecurrentworkspacetomonitor, l
bind = SUPER CTRL, J, movecurrentworkspacetomonitor, d
bind = SUPER CTRL, K, movecurrentworkspacetomonitor, u
bind = SUPER CTRL, L, movecurrentworkspacetomonitor, r

# LMB
bindm = SUPER, mouse:272, movewindow
# RMB
bindm = SUPER, mouse:273, resizewindow
$kw = master:no_gaps_when_only
binde = SUPER, M, exec, hyprctl keyword $kw $(($(hyprctl getoption $kw -j | /nix/store/rp0bkxz69iry91676ymw64gw9x2av0ky-jq-1.6-bin/bin/jq '.int') ^ 1))
env = LIBVA_DRIVER_NAME,nvidia
env = XDG_SESSION_TYPE,wayland
env = GBM_BACKEND,nvidia-drm
env = __GLX_VENDOR_LIBRARY_NAME,nvidia

bind = SUPER, Q, exec, pkill wlogout || wlogout -p layer-shell

general {
  cursor_inactive_timeout = 5
}

bind = SUPER SHIFT, S, exec, grimblast copy area

bind = SUPER, P, exec, pkill rofi || rofi -show drun

bindle = ,XF86AudioRaiseVolume, exec, volumectl -u up
bindle = ,XF86AudioLowerVolume, exec, volumectl -u down
bindle = ,XF86AudioMute, exec,  volumectl toggle-mute
bindle = ,XF86AudioMicMute, exec, volumectl -m toggle-mute

bindle = ,XF86MonBrightnessUp, exec, lightctl up
bindle = ,XF86MonBrightnessDown, exec, lightctl down

Can you upload the logs after reproducing this issue?  /tmp/hypr/$HYPRLAND_INSTANCE_SIGNATURE/hyprland.log.

Interesting, I have exactly the same behavior but not with monitors (switching focus between monitors if fine for me), but with steam (from pacman) - if I have steam and some other window on the screen then when I switch focus between steam and other window then the focus jumps back to the window which has the cursor over. So I think this would have the same root cause - can open a separate issue if this turns our to be a separate issue though.
Focus works fine between many non-steam windows. The logic of the issues on my side seems to be: if focus changes between steam and any other window then move focus to the window under cursor. I can confirm that this was not an issue in the past, I don't remember when this regressed but it was at least a month ago, I think.
Can also confirm that the issue doesn't happen with Brave which is also Xwayland on my side as that was my first guess.
I have commented my entire config side from my keybindings and genera.no_cursor_warps = true and the issue is reproducible.
Minimal config:
general {
    no_cursor_warps = true
}
bind = SUPER     , h   , movefocus , l    # focus left window
bind = SUPER     , j   , movefocus , d    # focus below window
bind = SUPER     , k   , movefocus , u    # focus above window
bind = SUPER     , l   , movefocus , r    # focus right window

This is what is being logged when I have my cursor and focus on my terminal and movefocus to steam:
[LOG] Keybind triggered, calling dispatcher (64, , 104)
[LOG] Searching for matching rules for Alacritty (title: Alacritty)
[LOG] Set keyboard focus to surface 5ec8a769c600, with [Window 5ec8a769f070: title: "Steam"]
[LOG] Callback 5ec8a76e4d78 -> 5ec8a76e4d70, CTextInput removed.
[LOG] Callback 5ec8a76e4de0 -> 5ec8a76e4dd8, CTextInput removed.
[LOG] Searching for matching rules for steam (title: Steam)
[LOG] cursorImage request: surface 5ec8a75b1d80
[LOG] Registered signal for owner 5ec89ac48a78: 5ec8a75b2098 -> 5ec89ac48ac0 (owner: CWLSurface)
[LOG] Registered signal for owner 5ec89ac48a78: 5ec8a75b2058 -> 5ec89ac48b28 (owner: CWLSurface)
[LOG] CWLSurface 5ec89ac48a78 called init()
[LOG] Callback 5ec8a75e8228 -> 5ec8a75e8220, IME Keyboard Grab removed.
[LOG] IME TextInput Keyboard Grab destroy
[LOG] Searching for matching rules for steam (title: Steam)
[LOG] Set keyboard focus to surface 5ec8a58540e0, with [Window 5ec8a5542f50: title: "Alacritty"]
[LOG] Registered signal for owner 5ec8a76e4b90: 5ec8a58543d8 -> 5ec8a76e4d50 (owner: CTextInput)
[LOG] Registered signal for owner 5ec8a76e4b90: 5ec8a58543f8 -> 5ec8a76e4db8 (owner: CTextInput)
[LOG] Searching for matching rules for Alacritty (title: Alacritty)
[LOG] TI ENABLE
[LOG] cursorImage request: shape 9
[LOG] Callback 5ec89ac48ae8 -> 5ec89ac48ae0, CWLSurface removed.
[LOG] Callback 5ec89ac48b50 -> 5ec89ac48b48, CWLSurface removed.
[LOG] CWLSurface 5ec89ac48a78 called destroy()
[LOG] IME TextInput Keyboard Grab new
[LOG] Registered signal for owner 5ec8a576ead0: 5ec8a565b820 -> 5ec8a576eae0 (owner: IME Keyboard Grab)


[SPARK-5118][SQL] Fix: create table test stored as parquet as select ..

Can one of the admins verify this patch?

Can you add a test case?  Perhaps in HiveQuerySuite

i think i should remove the test case for stored as parquet  only can pass in hive-0.13

Just check the version in the hive shim in the test case.
On Jan 8, 2015 7:53 PM, "guowei2"<EMAIL_ADDRESS>wrote:

i think i should remove the test case for stored as parquet only can pass
in hive-0.13
—
Reply to this email directly or view it on GitHub
https://github.com/apache/spark/pull/3921#issuecomment-69290028.

Like this one.

check the version in the hive shim in the test case Done
@yhuai   thanks a lot

ok to test

Also can you please fix the merge conflict?

Test build #25364 has started for   PR 3921 at commit 833f438.

This patch does not merge cleanly.

Test build #25364 has finished for   PR 3921 at commit 833f438.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25364/
Test PASSed.

fix the merge conflict

Test build #25388 has started for   PR 3921 at commit 5405b84.

This patch merges cleanly.

Test build #25388 has finished for   PR 3921 at commit 5405b84.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25388/
Test PASSed.

LGTM.

@guowei2 Can you rebase it? Also, can we add a test to make sure we actually generate the correct Parquet file? Just reading back the data of created table cannot verify the file format. Thank you.

Test build #25879 has started for   PR 3921 at commit 9da56f8.

This patch merges cleanly.

@yhuai  How to check the correct Parquet file?

create a parquet table, and then insert into data
create table stored as parquet as select ..
and then checkAnswer with them
is it OK?

Test build #25879 has finished for   PR 3921 at commit 9da56f8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25879/
Test PASSed.

I think you can use describe formatted <table name> to get the information of a table. You should be able to find InputFormat and OutputFormat in the output. We can just double check the output of describe and then make sure the data is stored in the correct format.

Test build #25948 has started for   PR 3921 at commit b1ba3be.

This patch merges cleanly.

@yhuai  I use DESC EXTENDED to check the the stored file, Is it OK?

Test build #25948 has finished for   PR 3921 at commit b1ba3be.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25948/
Test PASSed.

Thanks for fixing this!  Merged to master and branch 1.3.


fix: CSS files are now applied and not just loaded.

Details of the problem:
In some cases, EEL simply loads all the files included in the initialized folder following an eel.init(). However, the missing .css extension prevents the CSS files from loading on the page for some configurations.

I also haven't had an issue with CSS loading, this that because HTML can load them?
Not saying this isn't a good fix!


wireguard: remove deprecated userspace fallback

The userspace fallback was deprecated in v1.16 with https://github.com/cilium/cilium/pull/31867. Let's remove it now.
Fixes: https://github.com/cilium/cilium/issues/33594

/test

@learnitall ping :pray:


Passing auth result (client identity) to backend

When mTLS is enabled, Sever side Envoy needs to send the client identity to the backend through HTTP header, so that backend can do its own check/processing based on the client identity.
Questions:
What HTTP header do we use?
How do we support this for non-HTTP requests (This is a stretch goal)?

@myidpt I'm not sure that this an Envoy specific question exactly. It seems like you need to write a filter to set some header based on some parameters of the client connection. Unless you are asking to specify some format that we would perform automatically for all requests? Can you expand please?

I think it would be good to automatically set the client identity for all requests that did mTLS. But it also works for us if we use a filter to achieve the goal.
I think this is a question for Envoy whether to automatically set the header for all mTLS requests.

I'm open to setting a header by default, but:

What would the header be.
What would be in it?

Can you propose something?

@lookuptable @lizan @wlu2016
I think we need more discussion on that.
Some options:
"from" header (https://en.wikipedia.org/wiki/List_of_HTTP_header_fields)
"x-ssl-" headers that some people used, probably not standardized.
Currently, we should have the client identity in it, resolved from client-cert SAN.

From is supposed to contain reachable email address, so we probably shouldn't use it, see RFC7231, sec. 5.5.1.

Thinking more about this, does it make more sense for us to have our own filter to do header injection? This allows us to do Istio specific sanitization (e.g. ignore non-URI identities in the SAN list) and use headers like "x-istio-service-account".

I am skeptical that we are going to find a common header to fit this use case, and will probably need to make one up like the other x-envoy headers. However, since you already have your mixer filter, you could probably just do it there if you want specific functionality.

we should give this a more generic name, x-service-account will be better than x-istio-service-account. We should also consider a consistent story passing other auth info to the backend like email address, group, issuer, etc. We have been using jwt token or json object to convey these info in the past. Since this is not an urgent feature, lets prepare a design to cover all these scenarios first.

I think there is utility in making this generic to Envoy. In advanced deployments having an edge proxy forward details about the TLS negotiation to backends can be useful x-forwarded-proto is an example of doing this but it doesn't help with mutual auth.
Theres some doc with HAProxy which is still pretty generic but it doesn't cover SANs
https://raymii.org/s/tutorials/haproxy_client_side_ssl_certificates.html#Sending_certificate_details_to_backend_application
and I wouldn't suggest we adopt their nomenclature. I think
x-forwarded-client-cert-****
e.g.
x-forwarded-client-cert-name -> common name
x-forwarded-client-cert-san -> SAN
etc..
fits better with whats already present. If Envoy was going to pick up generic header rewriting capabilities then I think we could use that.
Don't think we need to use x-envoy given that there's already a rough convention for x-forwarded-*.
@htuch - thoughts?

One issue to think about if we add fixed support into Envoy is how we handle multi-level proxying. XFF gets appended as you go, so you would have to be able to do something similar with the client cert info and make sure you keep all headers in sync (e.g. every proxy would have to insert all headers).
By generic header rewriting capabilities are you talking about something like https://support.citrix.com/article/CTX114461 (but obviously in the Envoy config and some way of getting at Envoy internal fields like the SSL cert name by label)?
BTW, looks like XFF and friends are now in the forwarded header covered by RFC https://tools.ietf.org/html/rfc7239.

I think side-car proxies might be the exception to XFF appending rules, there's not much use to the extra appending if the proxy is logically part of the receiving application.
Beyond that I think appending is the desired behavior though it clearly becomes tricky when were talking about appending details about things which are auth related. There is potential for the ultimate receiver to have confusion unless we're careful about the details. Maybe instead of separate headers for parts of the cert we have just one header and a format inside it? E.g.
X-Forwarded-Client-Cert: name=X; SAN=A; SAN=B; hash=1fd445e2
and then folks get to configure which of the fields in that format are filled out. This would allow normal header concatenation rules to apply while keeping the correlation with hops clear.

Envoy isn't just used as a side-car though, so we need to support a number of different topologies, including classical multi-level proxies.
I like the idea of combining into a single XFCC. If I understand your proposal correctly, there is an extensible list of key/values for the client cert which are semi-colon separated, and then each level of proxy is able to either append a header or use a comma separator (per RFC 7230) to add additional cert info.
It will also be necessary for the "frontline" Envoy instance to be configured to scrub, rather than append to, the XFCC header.

Thanks @louiscryan and @htuch.
I think according to https://tools.ietf.org/html/rfc7230#section-3.2.2, we should always concatenate the info of additional certs to the end of the XFCC field, separated by comma.
When conveying the authentication results such as cert info, we need to think about the security risks. What if a non-frontline Envoy instance receives traffic from a malicious user, and that user injects fake auth results into XFCC? Maybe to prevent this, we should only trust the XFCC from clients that is authenticated through mTLS. Otherwise, the client XFCC header should be scrubbed.

We should apply the same scrubbing that we use for other "trusted" headers already. If we do that, we won't have any issues.

What would be the practical use for multiple XFCC values? Even with tiered proxies and service mesh, backend is only interested in the client's SSL certificate and not that of middle proxies that handled the traffic.
IMHO, we should create XF* headers only at the "frontline".

This is mostly just future proofing. It's probably fine to just do XFCC for frontline in a typical service mesh or Envoy deployment today.
Maybe one day some backend wants to verify the certs used by intermediary proxies are all of a certain vintage or provenance in some very large deployment.
I think it makes sense though to think about how this could be extended if we're defining an implicit standard here and not box ourselves in if an Envoy configuration wants this one day, all things being equal. Earlier design iterations would imply only frontline XF* ever. We can just implement it for frontline today.

I agree with<EMAIL_ADDRESS>Maybe micro-services need to control quota, do logging or even authz based on the intermediary services? For example:
End user -> ... -> Service A -> ... -> Service C
End user -> ... -> Service B -> ... -> Service C
If Service C provisions high QPS for Service A, or only allows Service A to access some core data, XFCC chaining can be helpful.
Considering all use cases, IMHO, for now, we can implement a flag "forwarding Client Cert" in each upstream cluster, with four possible values:

"forward only" - Only forward all XF* headers in the request to the next hop.
"append and forward" - Append current hop results to the request's XF* headers and forward it to the next hop.
"sanitize and set" - Sanitize all XF* headers in the request and set the current hop result in.
"sanitize" - Do not send any XF* headers to the next hop.

Please keep in mind that the biggest issue with XF* headers (and the reason behind Forwarded header) is that there is no way to match XF* headers together, i.e. in both "forward only" and "append and forward" options, there is no guarantee that all hops will set all XF* headers, and receiving end might see different number of various XF* headers with no ability to infer which proxy sent which certificate (or even if a client sent one!).
It's a security nightmare, unless you control the whole perimeter.
IMHO, we should consider using Forwarded header if we want to forward multiple XFCC headers, even though there is virtually no support for it in other proxies and runtimes.

I don't have a strong preference on whether to make up a new header called XFCC or to use Forwarded, but I agree we should optionally support append like we currently do for XFF.
For reference, the current trusting/scrubbing behavior in the HTTP connection manager is controlled by the "use_remote_address" option and is used here:
https://github.com/lyft/envoy/blob/master/source/common/http/conn_manager_utility.cc#L46
I would suggest that we expand this into an enum of various scrub/append/set settings as per @myidpt  depending on what we need.
Once the remote address is determined, Envoy then decides whether a request is internal or external, and then does additional scrubbing if external. Note that the internal vs. external logic is currently fairly simplistic (what was needed for Lyft) and may need to be expanded a bit with options depending on other use cases.

@PiotrSikora (Please correct me if I'm wrong)
So as I understand, the advantage of using Forwarded is to be able to mark "by whom" for each forwarded header, using by. Other than that, forwarded header should have same security concerns as XFCC RFC 7239. However, I think the security concerns can be alleviated with mTLS chain between proxies, through which the header's authenticity can be guaranteed through the proxy trust chain.
So can we can add by to the XFCC, like:
X-Forwarded-Client-Cert: name=X; SAN=A; SAN=B; hash=1fd445e2; by=<SAN of the proxy's cert>, ...
(certificate SAN should always contain the identity of the proxy, when there are multiple, there should be a priority to pick one). In this way, it is clear who has set each of the header. When mutual trust is established between proxies, the headers should be trusted.
To me, XFCC makes sense only when it is conveyed through mTLS. I'm not sure if it makes sense for non-mTLS connections because if the receiver never authenticates the sender, how can it trust the auth result header? So I suggest that the receiver only trusts the XFCC header sent through mTLS connections. (PS: just to think aloud, if we want to trust the XFCC header through non-mTLS connections, maybe the XFCC can contain a special value digest, that is the hash value of the XFCC header using the proxy's private key, so that the receiver is able to verify the authenticity of the header using the proxy's certificate, but this seems to much for now).
And I think the proxy should be able to be configured with the 4 options I mentioned. It's orthogonal to the security concern, right?

Folks, just chiming in here with a use case as this sounds like a feature we also need for a couple of systems. Specifically what we would like is, when receiving a mTLS request, have the ability to extract the CN, the DN and, less importantly, the SANs on the inbound client certificate, and then remap them onto a header that is propagated to the downstream cluster.
Personally I don't much care if its Forwarded or a custom header, but right now we are constructing our own custom header, because we had a similar discussion to the one that has already taken place here and decided that we'd make our own thing to be clear on the semantics. With that experience having been around this little track before, I'd vote for the XFCC. I like @myidpt's suggestion; parsing the details out would be pretty trivial.
It might also be worth considering what nginx does here [1], as its pretty extensive how they break apart the myriad of things available in the client certificate. Does anyone need all that? It's not clear - perhaps a subset would be enough.
[1] http://nginx.org/en/docs/http/ngx_http_ssl_module.html#var_ssl_client_verify

@myidpt @PiotrSikora how should we move the ball forward on this one? The actual implementation here is not difficult. We just need to decide on which header to use and the various semantics. Can the two of you a final proposal and we can get some +1s on it and then implement?

@PiotrSikora as I haven't heard from you, please comment on the following proposal:
For the security nature of the client cert info, I think it is better to choose the dedicated header (XFCC) for forwarding auth result, so that it is decoupled from other forwarded info, thus can be controlled more strictly.
The definition of XFCC ("x-forwarded-client-cert") is very similar to the "Forwarded" header. The XFCC header value is a comma (",") separated string. Each substring an XFCC element, which holds information added by a single proxy. Each subsequent proxy may choose to append to the end of the request's XFCC header after a comma.
Each XFCC element is a semicolon ";" separated string. Each substring is a key-value pair, grouped together by an equals ("=") sign. The keys are case-insensitive, the values are case-sensitive.
In the implementation, Envoy will support the following keys:
BY - Mandatory. The SAN (CN if SAN is not available) of the proxy that writes the element.
Subject - Optional. Client cert's subject field.
SAN - Optional. Client cert's subject alt name field.
(Please comment if more fields are needed)
Envoy config will have a "forward_client_cert" field in HTTP connection manager.
This field has four possible values:

"forward_only" - Only forward the XFCC header in the request to the next hop.
"append_forward" - Append current hop results to the request's XFCC header and forward it to the next hop.
"sanitize_set" - Sanitize the XFCC header in the request and set the current hop result in.
"sanitize" - Do not send the XFCC header to the next hop.

The default value is "sanitize".
Before modifying the XFCC header, Envoy will look at the mTLS result to check if the client is authenticated. If the authentication was not successful, Envoy will fall back to the default "sanitize" option.

@myidpt +1 from me. @timperrett can you look at ^. @louiscryan @htuch @PiotrSikora ?

+1 from me with a few suggestions

Can we give folks control over which fields are forwarded. I suspect Envoy will be asked to support others over time and the header could get quite unwieldly without some contro, particularly when appending is enabledl. A list of the cert field names to include should suffice. We should also decide what the default set is. Suggested name "forward_client_cert_details"

Should be able to forward a hash of the client cert

w.r.t to the modes I wonder if we need all this control. forward_only seems odd as it would omit a locally used credential while forwarding a remote certs details.
I wonder if it's useful for Envoy instead to always append a marker value if no cert was present on the local hop even if a cert was used on a downstream hop. E.g
X-Forwarded-Client-Cert: SAN=....; none; none
would imply that the two most immediate proxy hops did not use a client cert

This all seems reasonable. I'm wondering for the v2 gRPC APIs whether we want to support this via https://github.com/lyft/envoy-api/issues/4?

I don't buy the argument that decoupling this from theForwarded header provides any security advantage. On the contrary, proxies and their operators need to sanitize multiple headers and keep the whole set in consistent state.

Right now, there is no concept of trusted downstreams, and forward_client_cert: forward_only and forward_client_cert: append_forward cannot work without adding that first.

In terms of forwarded values, do we really care about the whole Subject? Neither that or SAN are globally unique without corresponding CA, unless we want to restrict this feature to a single CA. I think something like SHA256 of the certificate or SPKI might be more useful.

Right now, there is no concept of trusted downstreams

@PiotrSikora what do you mean exactly? We do implicitly have this support today via the poorly named "use_remote_address" field, and in fact we implement trusted downstreams with Envoy in Lyft today via mTLS for our POP edge proxies. In this case, we trust the downtream proxy to have correctly set XFF, etc.

@mattklein123 ah, sorry, I've missed that... but unless I'm misreading it, listener either trusts all accepted connections or none of them, correct?

@louiscryan

We can add a field forward_client_cert_details, and currently it supports values: Subject, SAN (BY and Hash should be mandatory).
What is the purpose of the hash of client cert? Is it generated using the private key of the proxy, or it's the hash value in the cert?
I guess forward_only can keep the XFCC header cleaner, by omitting less-important mTLS conections (such as between cluster-ingress proxy and server's side-car proxy)?
You are right that XFCC may still be trusted even without TLS/mTLS (e.g., within a private cluster). But I don't know if the "none" place holder is meaningful vs. just do a forwarding. Shall we add options always_forward and always_append_forward to trust the headers even when the connection is not mTLS? This can be useful for downstream traffic from a private network.

@PiotrSikora

Envoy supports per-downstream ssl_context (TLS/mTLS) config. So I think envoy should trust a downstream connection iff. it is a mTLS connection.
Your point that "the identity in cert depends on the CA" is great. The BY value scopes the cert information on the specific connection. Also, normally, I suppose the trusted root CAs configured on each proxy should be well controlled, so that conflicts in Subject or SAN should not happen. Using "SHA256 of the certificate or SPKI" doesn't work because certs keeps being rotated, and it is very hard to keep everything in sync.

@myidpt SPKI value doesn't change when certs are rotated (at least, it shouldn't).

@PiotrSikora
I see. But in our case, the cert content may change, for example, a SAN may be added/removed in a cert. Does the SPKI change in this case?

@myidpt that, I don't know... It might not... But SAN will change, so SPKI isn't worse in this case.

ah, sorry, I've missed that... but unless I'm misreading it, listener either trusts all accepted connections or none of them, correct?

Yes, basically. How else could it work though? As @myidpt points out it really needs mTLS to be trusted. The current implementation doesn't enforce that, and IMO I don't think we should make that required since people have different types of deployments. I would rather get rid of "use_remote_address" and make it much more explicit what is done with XFF, etc. like we are discussing here for XFCC. I just put a note on the v2 API about this cc @htuch

@PiotrSikora
Sorry I'm not very familiar with SPKI. What does SPKI actually look like? I can only find this but it's not related to X.509 ...

@myidpt check:

https://www.owasp.org/index.php/Certificate_and_Public_Key_Pinning
https://www.imperialviolet.org/2011/05/04/pinning.html
https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning
https://tools.ietf.org/html/rfc7469

@PiotrSikora
Thanks for the pointers. The idea seems to be putting the CA's public key info in the XFCC elements so that the receiver is able to verify if the CA in the certification chain is really allowed to issue the client's identity.
I think this can be useful for enforcing strong security, and we can consider it as a TODO. I think we may need to solve some implementation issues here, such as how to configure passing the public key of which CA (there can be multiple CAs in the certificate chain), and how to pass the public key of the CA to the HTTP connection manager.

@myidpt @louiscryan @PiotrSikora I think we are pretty much converged here in terms of control options. As long as we have the ability to specify what fields get populated in XFCC we can always add more fields later. So I the only remaining question is whether to use XFCC or Forwarded as the header name? Can we resolve that today? I don't personally care that much, though I think I would vote for XFCC just to be very explicit about what we are doing.

I vote for XFCC.
The basic idea is to decouple the security-related headers from the non-security headers, so that special rules can be applied to them. Because we introduce BY into XFCC, XFCC can address the writer of each element, just like the Forwarded header.

@louiscryan @PiotrSikora @mattklein123 @htuch
I have also updated the proposal according to our feedback, by appending to the original one. Please review the appended part by searching for "Added according to feedback". Thanks!

@myidpt this looks good for the most part. One question: what's the rational for choosing the CN or the SAN(s)? I'm guessing the working assumption is that the latter is a superset of the former? Is that part of the spec? Whilst I don't think it would break our usage, just want to make sure we're making a valid assumption, as im not sure there is any requirement from a cert perspective for SANs to be a super set including the CN.

I am fine with proceeding with XFCC, we can always support config to use Forwarded later if/when it gets some traction.

@timperrett
SubjectAltName (SAN) is not a superset of CN. We use SAN because CN is deprecated in favor of SAN for identity checks: https://tools.ietf.org/html/rfc6125#section-2.3

@myidpt OK. I think we are closed on the design here. Feel free to implement. Looking forward to landing this! :)

Cool, thanks everyone! I will start implementing it :)

@myidpt just so i'm not polling and annoying you, what's a rough ETA for this feature? Many thanks in advance!

@timperrett
Yeah sorry I was distracted by the Istio launch and other stuff this week. Hopefully within next week.

@myidpt any news on this good sir?

Working on implementing integration tests. Will send the PR out today or tomorrow.


Add Login to Pico

Authorize through Kynetx and get root Pico ECI.

Current progress:
Replacing jqueryMobile ajax call with Angular http.

Authorization is being developed in a simple app first, and will be imported to main project after proven functionality.
Authorize through Kynetx is working. storing token in session storage is implemented. Redirect is still being tested but looks like its working with expiration date commented out.
Need to do:

do we need or want expiration of oauth tokens
getter for token, currently the token is stored as a json string with some other variables. we need a clean getter function.

This needs testing, but seems to be working.  https://picolab.github.io/manifold/ .
manifoldAuth.authenticatedSession() in the java console should return true after log in. manifoldAuth.defaultEci should not be null.


Add generated files

What does this PR change?
Adds generated files so OBS won't have to generate them.

:tada:


Downloader: Sparse checkout is (no longer) working

Possibly some code changes in https://github.com/oss-review-toolkit/ort/pull/8170 led to the effect that git sparse checkouts are no longer working.
To reproduce:
Let's say, from a gradle managed repository with many modules, I only want to check out an Angular web application subdirectory
ort --debug download --project-url=https://github.com/wkl3nk/fun-with-notes.git --vcs-path=frontend-angular16 --vcs-revision=main -o .
Looking into .git/info/sparse-checkout it looks good, everything is prepared, also .git/config is set to sparseCheckout = true
But the final step, a "git.reset() ..." in the code has no effect.
If I manually do a git reset --hard FETCH_HEAD in the target directory, then everything looks good.
As I said, this line here in the code of Git.kt just seems to have no effect.
git.reset().setMode(ResetCommand.ResetType.HARD).setRef(resolvedRevision).call()
The OSS Review Toolkit, version 15.0.0-051.sha.64fd9db.
Running 'download' under Java <IP_ADDRESS> on Mac OS X

JGit does not support sparse checkouts currently, that's a know limitation:
https://github.com/oss-review-toolkit/ort/blob/64fd9db7fd5c27e2a9d50692fea6def98e2a037b/plugins/version-control-systems/git/src/main/kotlin/Git.kt#L244-L246
I'm not sure what we can do about it without breaking #7725 again other than contributing sparse checkout support to upstream JGit...

JGit does not support sparse checkouts currently

I forgot about that part, that's unfortunate. I have put this on the agenda for the next community meeting so that we can discuss the options:

Remove support for sparse checkout completely. That would be bad when you need to scan only part of a large repo.
Revert the change and find another solution for #7225.
Go back to Git CLI. What benefits does JGit bring as long as we still depend on the CLI as well?
Provide two alternative Git implementations, one using JGit and one using the CLI. The former for users who do not want to depend on the CLI, the latter for users who need sparse checkout.
Contribute sparse-checkout support to JGit. What is the effort and are the maintainers even interested in the feature?

Go back to Git CLI. What benefits does JGit bring as long as we still depend on the CLI as well?

That's a valid question. My initial motivation to replace Git CLI with JGit was to reduce dependencies on external tools and make ORT as self-contained as possible. However, we've been struggling to get rid of Git CLI completely for a long time, mostly due to limitations in JGit.
Also, I had to accept that when trying to target Kotlin Multiplatform, it would actually be easier to call Git CLI on all platforms rather than finding native Git implementations on all platforms.

Provide two alternative Git implementations

I would not want to maintain those two implementations. We already are in the state of using Git CLI and JGit, but that should have been only a temporary solution.

Contribute sparse-checkout support to JGit. What is the effort and are the maintainers even interested in the feature?

That would be very nice from a community perspective. I believe the maintains would happily accept such a feature, but they're not actively looking into it, mostly because JGIt seems to be more and more about the Git server side (as it's used in Gerrit) than the Git client side.

I'm actually a bit surprised that the code path with
git.reset().setMode(ResetCommand.ResetType.HARD).setRef(resolvedRevision).call()

is even triggered for the given example. IIUC this code should only be necessary when updating a previously checked out non-fixed revision. That is, it should not be necessary when doing an initial sparse checkout in a freshly initialized working tree. So, avoid to call this on the first "update" call which actually populates the working tree initially should also fix the problem, or?

But the final step, a "git.reset() ..." in the code has no effect.

Debugging this actually shows that git.reset()... does have an effect, but an unwanted one: It turn the sparse checkout that's correctly done by run("checkout", ...) into a full checkout. A simple work-around is to only reset if in non-sparse mode. That should be rather easy to add and cover most of the cases. I'll make a proposal.

A simple work-around is to only reset if in non-sparse mode. That should be rather easy to add and cover most of the cases. I'll make a proposal.

I've found a better very simple solution, see https://github.com/oss-review-toolkit/ort/pull/8199.


Provide life cycle support of current pool implementation

Just paste chat context with @trustin in slack first, will provide better summary later
hi guys, what is the correct way to release / close a client (in detail, http client) when I finish using that (edited)
[4:47]
or there's no need to do that?
trustin [4:51 PM]
`ClientFactory.close()`?
[4:51]
If you are using `ClientFactory.DEFAULT`, the default factory will be closed when JVM shuts down.
[4:52]
If you prefer to close it explicitly, you can use `ClientFactory.closeDefault()`
huydx [4:52 PM]
so it will close all clients attach to that factory?
trustin [4:52 PM]
yes
huydx [4:53 PM]
what should I do if I just want to remove a single client only
trustin [4:53 PM]
You can't because the connections are shared across the clients that belong to the same factory.
[4:53]
If the connection is idle for about 10 seconds, it will be closed automatically.
huydx [4:54 PM]
I see, so in my case I just not need to do anything
trustin [4:54 PM]
Yes. If you prefer to get it closed sooner, you may want to decrease the idle timeout value.
huydx [4:54 PM]
thank you!
trustin [4:55 PM]
You're welcome. Do you think we need to provide tight connection life cycle support?
huydx [4:56 PM]
I don't know it's needed for common users or not, expose those life cycle may increase complexity of the framework
trustin [4:57 PM]
I think it's just a matter of pool implementation strategy, so it's doable without altering much of the core.
[4:58]
For example, a user may want to employ different pooling strategy (e.g. LIFO vs FIFO)
[4:58]
It's just that people didn't ask for it yet :slightly_smiling_face:
huydx [4:59 PM]
I see, so it may useful in some cases
[4:59]
vote :+1: for that
trustin [5:07 PM]
Cool. Care to create a new issue? :slightly_smiling_face:
huydx [5:12 PM]
sure

Related issue: #818


[ISSUE #339] add design doc for integrating OpenSchema

Codecov Report

Merging #502 (c09bbfd) into develop (a62bf65) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             develop     #502   +/-   ##
==========================================
  Coverage      10.42%   10.42%
  Complexity       328      328
==========================================
  Files            240      240
  Lines          11743    11743
  Branches         999      999
==========================================
  Hits            1224     1224
  Misses         10419    10419
  Partials         100      100

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a62bf65...c09bbfd. Read the comment docs.

LGTM, but there might one thing need to confirmed.
In this plan, the schema id will be added in each message. Does this will modify the current protocol? Maybe we should add the schema id in header or props?
I think the schema can be the metadata of a topic, then the producer doesn't need to send the schema id, and it also doesn't need to be notified when the scheme added/deleted in topics, since the validation and serialize is managed in eventmesh-runtime.

As for the format of schema id, sure it could be added in the header as long as an MQ supports user-defined headers. Anyway, It's only about how to organize schema id which doesn't affect the architecture or processes.
As for where to generate schema id, It might work to put it on eventmesh-runtime, but I have one question:
How to set the format of messages(JSON?/JDK?/Avro?/...) between client and runtime? Maybe set it to JSON? because the current format is JSON and other types of format will introduce more problems😂.

Mostly the design is ok, so i merge it frist, if there are still some detail discussion, please feel free to do it.


Hash for StructField should consider more than the name

Two StructFields with the same name but different data_types are considered to hash equivalently, which I think is incorrect.
I'm working around this for now, but filing for @roeap and I to discuss later

Agreed, in fact, name, data_type and nullability should match for a field to be considered equivalent.

take

@rtyler: I added PR to fix this.


make optional warning about long functions

Make optional existing warning about long functions.
Tested to work both on and off.

Merged. Thanks for your contribution.


improve print log when trigger memory limiter actually

Proposed changes
Reached the one tenth of load hard limit<PHONE_NUMBER>and process remaining allocator cache<PHONE_NUMBER>8reached process soft memory limit<PHONE_NUMBER>6, waited for flush, time_ns:15579

There is too many log in log file, but part of them do not trigger memtable flush and return directly.

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

Close it for memory limiter will be refactored.


GAMChanger fails to load data in some cases

I've found a bug where GAMChanger sometimes doesn't populate the 'metrics' / 'feature' / 'history' panel. It seems that when this happens, the GAMChanger interface has failed to load the validation samples, because the status bar says '0/0 validation samples selected.
This seems to occur sometimes based on the data that is provided, and might have something to do with missing data points, but I'm struggling to figure out exactly what the cause is.
Below is the smallest reproducing example I can come up with.

import pandas as pd
import gamchanger as gc
from interpret.glassbox import ExplainableBoostingRegressor

# Works
X = pd.read_csv('demo-X-succeed.csv')
y = pd.read_csv('demo-y-succeed.csv')['OrderedFractionOfEstate']

# Doesn't work
#X = pd.read_csv('demo-X-fail.csv')
#y = pd.read_csv('demo-y-fail.csv')['OrderedFractionOfEstate']

ebm = ExplainableBoostingRegressor(interactions=False)
ebm.fit(X, y)

gc.visualize(ebm, X, y)

I've attached the CSV files, which differ in that the 'succeed' files have a single extra data point. That is, when loading 'demo-[X|y]-fail.csv' the GamChanger interface loads, but the side panel doesn't populate (unexpected behaviour). When loading 'demo-[X|y]-succeed.csv', the GamChanger interface loads and the side panel populates the metrics as expected.
demo-X-fail.csv
demo-X-succeed.csv
demo-y-fail.csv
demo-y-succeed.csv

I have produced a more compact MWE;

import numpy as np
import pandas as pd
import gamchanger as gc

from interpret.glassbox import ExplainableBoostingRegressor

size = 5
x1 = np.linspace(0, 10, size)
y = -1.0 * x1.copy() + 3.0

# Introduce missing data
x1[1] = np.nan
x1[2] = np.nan

# With only two missing datapoints, the GAMChanger interface loads fine
# If we introduce a third missing feature valueby un-commenting the below
# line, the validation data fails to load
#x1[3] = np.nan

df = pd.DataFrame(
    data={
        'x1': x1,
        'y' : y
    }
)

X = df[['x1']]
y = df['y']

print(df)

# Train model
ebm = ExplainableBoostingRegressor(interactions=False)
ebm.fit(X, y)

gc.visualize(ebm, X, y)

...update... based on the above MWE, I have been able to narrow down the error to this javascript uncaught error in the Firefox JS console;

...which I believe is coming from the variable messenger_js_base64 at gamchanger.py#L528

Following the rabbit trail from the 'gamchangerData' event down, I can see that this event is intercepted at GAM.svelte:663, which calls initDataLoaded, which calls initGAMView theninitSidebar.

Wow @aaronsnoswell thank you so much for your detailed report and effort in debugging this issue!
I tried to reproduce this error using your example, but I got a ValueError when fitting an EBM model with missing values. I believe EBM does not support missing value yet? My interpret version is 0.2.7
     x1    y
0   0.0  3.0
1   NaN  0.5
2   NaN -2.0
3   7.5 -4.5
4  10.0 -7.0
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-2-13a640a7d74a> in <module>
     32 # Train model
     33 ebm = ExplainableBoostingRegressor(interactions=False)
---> 34 ebm.fit(X, y)

~/miniconda3/envs/gam/lib/python3.7/site-packages/interpret/glassbox/ebm/ebm.py in fit(self, X, y, sample_weight)
    822         #     AND add some tests for the X.dim == 1 scenario
    823
--> 824         # TODO PK write an efficient striping converter for X that replaces unify_data for EBMs
    825         # algorithm: grap N columns and convert them to rows then process those by sending them to C
    826

~/miniconda3/envs/gam/lib/python3.7/site-packages/interpret/utils/all.py in unify_data(data, labels, feature_names, feature_types, missing_data_allowed)
    340         msg = "Missing values are currently not supported."
    341         log.error(msg)
--> 342         raise ValueError(msg)
    343
    344     return new_data, new_labels, new_feature_names, new_feature_types

ValueError: Missing values are currently not supported.

Hi @xiaohk thanks for getting back to me!
The latest versions of Interpret have experimental support for missing values - I forgot to mention that I am using this experimental code. To enable it, you need to change a few places in the interpret source code. See this comment on interpretml/interpret#18 for the details.
So for instance, I checked pip show interpret to get the install location, then opened ebm.py, and changed all instances of missing_data_allowed=False to missing_data_allowed=True in ebm.py.
After doing that, the example should work.
Thanks again!

I see, thanks!
EBM's experimental support for missing values introduces a separate additive_scores[0] that is only used when computing the prediction score for missing values. To fully support this, GAM Changer has to visualize the missing value "bin" in the GAM Canvas — I will leave it for future work. For now, GAM Changer will remove all rows in X that contains any missing values in get_sample_data().
https://github.com/interpretml/gam-changer/commit/fb7ba181d00c20f0ba9018a8e6a6b6975c53556e#diff-de8698f459a11697fd2d6614444871f69e802ae1af354cd35aba32e62e6698bbR267-R277
I will close this issue for now. Let me know if it doesn't work for you @aaronsnoswell. Thanks for reaching out to me!

Thanks for looking into this, @xiaohk!
fb7ba181d00c20f0ba9018a8e6a6b6975c53556e seems like a good patch for now. Dropping all rows with missing values is pretty rough for users with real-world data though in the longer run :D
I'd be happy to take a stab at adding proper support for missing values if you can provide a little guidance for me. E.g. could you draw a sketch / doodle of what the GAMChanger interface should look like to show the missing value bin (where this would go in the interface?). Also, is there any documentation about setting up a development environment for GAMChanger?

Thank you so much for your interest! I believe supporting missing value will be super helpful. Adding this feature might sound straightforward, but I am sure it would require **A LOT of work. 😅
Some high-level steps:

Support missing value in the EBM inference in WebAssembly

Need to handle continuous, categorical features, and interaction terms (implementations are different)

Visualize the missing value

Continuous feature: a separate dot on the line chart
Categorical feature: a separate bin in the bar chart
Cont X Cont interaction: a new row and a new column in the matrix
Cont X Cat interaction: many new separate bars (when NA happens in cont) or a new bar (when NA happens in cat)
Cat X Cat interaction: a new row/column of dots

Interaction with the missing value

Integrate editing tools to support missing values: e.g., align/interpolate/monotonicity do not really make sense.
We need to specially handle/prevent users from selecting regular bins and NA bin all at once

Integration with other views

Feature panel
History panel: new event logging when editing missing values
Footer: new event name when editing missing values

Loading .gamchanger files

Need to load missing value related parameters

To set up a development environment for GAM Changer:
git clone git@github.com:interpretml/gam-changer.git

# Install the dependencies:
npm install

# Start a development server
npm run dev

You might have noticed that the EBM inference and isotonic regression WebAssembly code are shipped as binaries in this repo. Their source code is at xiaohk/ebm.js and xiaohk/isotonic.js, respectively.
If you are interested, I am happy to provide any sketches, feedback, and guides that can help you! It would be a hard and rewarding contribution to GAM Changer!

Wow :) That does sound like a lot of work.
A first point - interpretml doesn't currently support missing values for continuous features, and I don't believe they plan to - that would potentially reduce some of the interactions you mention above and simplify the workload.
Perhaps a good starting point is to figure out how stable the interpretml missing value support is. Assuming I can rely on it not changing too much, I could potentially bite off part of this work list in a new branch to get the ball rolling.
P.S. Thanks for the dev environment instructions.


Add networking.knative.dev/ingress-provider: contour label.

This will allow serving operator to have a way to pick and choose ingress controller for installation.

/cc @houshengbo

@tcnghia : Why do we need labels on rbac and other contour resources?

I just looked at net-istio repo. Looks like all resources under config gets the label. Got it 👍


Model Builder doesn't work for binary classification.

System Information (please complete the following information):

Model Builder Version: 16.0.1906.701
Visual Studio Version: 2019 Professional v 16.1.4

Describe the bug
Model Builder throws error on binary classification task for sentiment analysis scenario on wikipedia-detox-250-line-test.tsv
To Reproduce
Steps to reproduce the behavior:

https://dotnet.microsoft.com/learn/machinelearning-ai/ml-dotnet-get-started-tutorial/create
https://dotnet.microsoft.com/learn/machinelearning-ai/ml-dotnet-get-started-tutorial/scenario
https://dotnet.microsoft.com/learn/machinelearning-ai/ml-dotnet-get-started-tutorial/data'
https://dotnet.microsoft.com/learn/machinelearning-ai/ml-dotnet-get-started-tutorial/train
See error

Expected behavior
I expected the basic Get Started tutorial for the Model Builder to work successfully 😊
Screenshots

Additional context
I have had similar errors on other projects that I tried but this is the most simple case to reproduce the issue.

This is working for the customer. The wikipedia-detox-250-line-test.tsv was corrupted when it was downloaded from the web.


Error while installing application

This occured after downloading the setupfile, executing it and it downloaded stuff and got stuck at installing (0%) and finaly threw this error:

This is the log:
The following properties have been set:
Property: [AdminUser] = true {boolean}
Property: [InstallMode] = HomeSite {string}
Property: [NTProductType] = 1 {int}
Property: [ProcessorArchitecture] = AMD64 {string}
Property: [VersionNT] = 10.0.0 {version}
Running checks for package '.NET Desktop Runtime 7.0.4 (x64)', phase BuildList
Checking conditions before installer checks for command 'net7desktopruntime_x64\windowsdesktop-runtime-7.0.4-win-x64.exe'
Result of running operator 'ValueEqualTo' on property 'ProcessorArchitecture' and value 'Intel': false
Result of running operator 'ValueEqualTo' on property 'ProcessorArchitecture' and value 'Arm': false
Result of checking conditions before installer checks for command 'net7desktopruntime_x64\windowsdesktop-runtime-7.0.4-win-x64.exe' is: No action
Running external check with command 'C:\Users\Lorenz\AppData\Local\Temp\VSD8448.tmp\net7desktopruntime_x64\NetCoreCheck.exe' and parameters 'Microsoft.WindowsDesktop.App 7.0.4'
Process exited with code 12289
Setting value '12289 {int}' for property 'NetCoreCheck'
The following properties have been set for package '.NET Desktop Runtime 7.0.4 (x64)':
Property: [NetCoreCheck] = 12289 {int}
Running checks for command 'net7desktopruntime_x64\windowsdesktop-runtime-7.0.4-win-x64.exe'
Result of running operator 'ValueEqualTo' on property 'ProcessorArchitecture' and value 'Intel': false
Result of running operator 'ValueEqualTo' on property 'ProcessorArchitecture' and value 'Arm': false
Result of running operator 'VersionLessThan' on property 'VersionNT' and value '6.1.0': false
Result of running operator 'ValueEqualTo' on property 'NetCoreCheck' and value '0': false
Result of running operator 'ValueEqualTo' on property 'AdminUser' and value 'false': false
Result of checks for command 'net7desktopruntime_x64\windowsdesktop-runtime-7.0.4-win-x64.exe' is 'Install'
'.NET Desktop Runtime 7.0.4 (x64)' RunCheck result: Install Needed
Installation of components '.NET Desktop Runtime 7.0.4 (x64)' was accepted.
Copying files to temporary directory "C:\Users\Lorenz\AppData\Local\Temp\VSD8448.tmp\"
Downloading files to "C:\Users\Lorenz\AppData\Local\Temp\VSD8448.tmp\"
(29.03.2023 18:33:29) Downloading 'net7desktopruntime_x64\windowsdesktop-runtime-7.0.4-win-x64.exe' from 'https://go.microsoft.com/fwlink/?linkid=2226490' to 'C:\Users\Lorenz\AppData\Local\Temp\VSD8448.tmp\'
Download completed at 29.03.2023 18:33:30
Verifying file integrity of C:\Users\Lorenz\AppData\Local\Temp\VSD8448.tmp\net7desktopruntime_x64\windowsdesktop-runtime-7.0.4-win-x64.exe
WinVerifyTrust returned 0
File trusted
Microsoft trusted cert
Running checks for package '.NET Desktop Runtime 7.0.4 (x64)', phase BeforePackage
Checking conditions before installer checks for command 'net7desktopruntime_x64\windowsdesktop-runtime-7.0.4-win-x64.exe'
Result of running operator 'ValueEqualTo' on property 'ProcessorArchitecture' and value 'Intel': false
Result of running operator 'ValueEqualTo' on property 'ProcessorArchitecture' and value 'Arm': false
Result of checking conditions before installer checks for command 'net7desktopruntime_x64\windowsdesktop-runtime-7.0.4-win-x64.exe' is: No action
Running external check with command 'C:\Users\Lorenz\AppData\Local\Temp\VSD8448.tmp\net7desktopruntime_x64\NetCoreCheck.exe' and parameters 'Microsoft.WindowsDesktop.App 7.0.4'
Process exited with code 12289
Setting value '12289 {int}' for property 'NetCoreCheck'
The following properties have been set for package '.NET Desktop Runtime 7.0.4 (x64)':
Property: [NetCoreCheck] = 12289 {int}
Running checks for command 'net7desktopruntime_x64\windowsdesktop-runtime-7.0.4-win-x64.exe'
Result of running operator 'ValueEqualTo' on property 'ProcessorArchitecture' and value 'Intel': false
Result of running operator 'ValueEqualTo' on property 'ProcessorArchitecture' and value 'Arm': false
Result of running operator 'VersionLessThan' on property 'VersionNT' and value '6.1.0': false
Result of running operator 'ValueEqualTo' on property 'NetCoreCheck' and value '0': false
Result of running operator 'ValueEqualTo' on property 'AdminUser' and value 'false': false
Result of checks for command 'net7desktopruntime_x64\windowsdesktop-runtime-7.0.4-win-x64.exe' is 'Install'
'.NET Desktop Runtime 7.0.4 (x64)' RunCheck result: Install Needed
Verifying file integrity of C:\Users\Lorenz\AppData\Local\Temp\VSD8448.tmp\net7desktopruntime_x64\windowsdesktop-runtime-7.0.4-win-x64.exe
WinVerifyTrust returned 0
File trusted
Microsoft trusted cert
Installing using command 'C:\Users\Lorenz\AppData\Local\Temp\VSD8448.tmp\net7desktopruntime_x64\windowsdesktop-runtime-7.0.4-win-x64.exe' and parameters ' /q '
Process exited with code 1618
Status of package '.NET Desktop Runtime 7.0.4 (x64)' after install is 'InstallFailed'

This occured after downloading the setupfile, executing it and it downloaded stuff and got stuck at installing (0%) and finaly threw this error:

The error message in the log shows that there was an issue installing the .NET Desktop Runtime 7.0.4 (x64) package. The log also shows that the download was successful, and the file integrity was verified.
Here are some troubleshooting steps you can try:

Restart your computer and try installing the package again.
Disable your antivirus temporarily and try installing the package again.
Run the installation file with administrator privileges. Right-click on the setup file and select "Run as administrator."
Check if there is enough disk space available on your system drive (C:). If the drive is full, free up some space and try installing the package again.
Check if there is any other .NET runtime installed on your computer that might be conflicting with the installation. If so, uninstall it and try installing the package again.
Try downloading the .NET Desktop Runtime 7.0.4 (x64) package again from the official website and then install it.


If window is resized while swiper is in the background, slide positions occasionally scatter around

To reproduce, create a swiper with these settings and add multiple slides:
speed: 850,
autoplay: 3000,
loop: true,
slidesPerView: 1,
simulateTouch: false,
direction: "horizontal",
effect: "fade",
fade: { crossfade: false }

Open the page where the swiper is
Open a new tab and leave the other page in the background
Resize the window horizontally.

(You might need to repeat the resizing part multiple times as the bug won't always trigger. I guess it only happens while slide change is happening in the background)

One workaround:
Add event listener for window focusin, call destroy(true, true) and reinit slider if the window width has changed. The downside is that this resets the swiper position.
I'm also investigating if it still happend if you stop autoplay while swiper is in the background.
Example of scattered slides. Only one slide should be visible at a time:

Reproduced on multiple sites with multiple settings and multiple browsers.

It is a browser issue/designed behavior. Because it slows down/ignores timeouts. Look at the forum, there were few solutions for that


feat(docs): Updated docs for markdown package

PR desicription

Updated documentation for markdown compoonent
Added to markdown to component section in sidebar

Awesome!


Update energiefluss-erweitert to version 0.4.1- Hotfix

Hotfix: After uprading the Adapter, it could happen, that the first datasource was not updated properly

merged
will trigger an extra update after checks have completed so stable is updates asap

Thanks a lot!


Intellisense messes with docstrings when function is decorated via typing.overload

In this snippet:
from typing import overload

@overload
def foo(x: int) -> int: ...

@overload
def foo(x: str) -> str: ...

def foo(x):
    """Return identity"""
    return x

f = foo(5)

Intellisense should show docs for foo, but it doesn't.
Though, everything works when all overloads are moved to *.pyi.
Tested on:

Python: 3.7.5-conda
Python Language Server: <IP_ADDRESS>
VSCode: 1.42.0-insider

typing is not actually imported but rather specialized in code so overload is not doing anything. When you move it to a stub, then it is properly handled as a multiple overloads since LS has that capability. Related #535.


WireGuard externally-accessible nodes

A possible solution for hybrid clusters (ref #31902).

@brb @thebsdbox This actually works, so what external node has all peers addressed by external IPs, and regular nodes get an external node address set to the external address? I'm kind of surprised how easy it turned out. There is still plenty of refinement, testing and discussion.

One challenge I'm noticing with this is that ExternalIP is only read once upon startup, if the node object doesn't have it when Cilium starts up, the only way to make Cilium observe that change is by restarting it. In my case, the external IPs are added to node objects after these were created by a custom cloud controller.

This PR did solve my problem, however I am no longer working on this so won't have the time to see this through.


⚠ Jenkins Plugins Artifactory has degraded performance

In 7eeca1f, Jenkins Plugins Artifactory (https://get.jenkins.io/plugins/artifactory/) experienced degraded performance:

HTTP code: 200
Response time: 1079 ms

Resolved: Jenkins Plugins Artifactory performance has improved in cda6ae5 after 34 minutes.


Run CI job on 0.1.x branch

@externl, do you know how to upgrade the badge URL for referring to this branch?

@externl, do you know how to upgrade the badge URL for referring to this branch?

Don't we just need to update the link in the README. I think the branch is part of the URL.


Git Page in the Resource Gallery

I am planning to add some resources in the git pages like git documentation(documentation is the best place to learn something), git cheatsheet(for people to remember the commands), youtube resources to learn git easily and a Git for Beginners blog(I will write this blog).
The layout of the page will be similar to other pages like javascript and web development.

I want to work on this issue. Please assign it to me under gssoc23.

@Astrodevil I want to work on this issue under gssoc'23


Support MinBy and MaxBy for non-float ordering

Closes https://github.com/NVIDIA/spark-rapids/issues/10968
This PR supports MinBy and MaxBy aggregation for non-float ordering, which is not required by the customer.
The main idea is to pack the ordering and value columns to a struct column and do a min/max aggregation on them. When the ordering column is not unique this pr will always return the minimal value while spark's result is non-deterministic.
So this pr is not using the previous argmin + remove nulls + gather method in https://github.com/rapidsai/cudf/pull/16163 and https://github.com/NVIDIA/spark-rapids/pull/11123. It does not have cudf dependency pr now.
Spark has a special NaN handling which is different with cuDF, so it will be not straightforward to implement float ordering min/max_by with current method. But it is a rare use case I think we don't need to support for now.
Many works are from @firestarman
Ran a simple perf test for 3 times
// (with big data gen)
import org.apache.spark.sql.tests.datagen._
val dataTable = DBGen().addTable("data", "a byte, b long, c long", 500000000)
dataTable.toDF(spark).write.mode("OVERWRITE").parquet("PERF")

// spark-rapids
val df = spark.read.parquet("PERF")
spark.time(df.groupBy("a").agg(min_by(col("b"), col("c")), min_by(col("c"), col("b"))).show(300))

on 500000000 rows of (a: byte, b: long, c: long)
results:
gpu: 3,484 ms
cpu: 5,844 ms
speedup: 68%

Another round with a complex value and simple order:
val dataTable = DBGen().addTable("data", "a byte, b struct<string, array<long>>, c int", 20000000)
spark.time(df.groupBy("a").agg(min_by(col("b"), col("c"))).show(300, false))

gpu: 1,049 ms
#11123: 1,019 m s
cpu: 2,776 ms
speedup: 2.65x

build

Look good to me

build

I am also concerned with the performance numbers. I don't know how to reproduce those numbers because df is not included. A speedup of 36% over the CPU is not much, and I am very concerned that if the value of the aggregation is long and complicated while the ordering key is not we will lose to the CPU because the CPU just has to do ordering for a very simple operation.

Thanks for the review! Added data generation part in pr description, and another round test of "a byte, b struct<string, array<long>>, c int", it got 2.65x speedup.
I think in most use cases the ordering key is unique, so the compare function will almost always stop early before getting into the complicated value part, so it won't be slower.

build

@thirtiseven could you please file a follow on issue to look at ways to improve the performance of these operators. I did my own benchmarking and this is far from an ideal solution. CUDF does not deal well with min/max on a struct of two values. I ran the following benchmarks.
I have 4 different data sets and 4 separate queries on those data sets.  All of the tests were run on a 32 core thread ripper with spark configured to use 32 cores. The GPU is an a6000, but limited to 16 GiB of GPU memory.
Dataset 1: (100 group by keys)
spark.range(10000000000L).selectExpr("id div 100000000 as a", "id div 1000000 as b", "id as c").createOrReplaceTempView("tbl")

Dataset 2: (100 group by keys, but made into structs before agg) The 2048 partitions is to eliminate spilling on the sort for the CPU.  I also ran one with the GPU at 64 partitions to make it more comparable to other CPU jobs.
spark.range(1,<PHONE_NUMBER>0L, 1 , 2048).selectExpr("id div 100000000 as a", "named_struct('f', id div 1000000, 'o', id) as b", "named_struct('o', id, 'f', id div 1000000) as c").createOrReplaceTempView("tbl")

Dataset 3: (1 million group by keys)
spark.range(10000000000L).selectExpr("id div 10000 as a", "id div 100 as b", "id as c").createOrReplaceTempView("tbl")

Dataset 4: (1 million group by keys, but made into structs before agg) The 2048 partitions is to eliminate spilling on the sort for the CPU. I also ran one with the GPU at 64 partitions to make it more comparable to other CPU jobs.
spark.range(1,<PHONE_NUMBER>0L, 1 , 2048).selectExpr("id div 1000 as a", "named_struct('f', id div 100, 'o', id) as b", "named_struct('o', id, 'f', id div 100) as c").createOrReplaceTempView("tbl")

Query 1: (group by min_by and max_by)
spark.time(spark.sql("SELECT a, min_by(c,b), max_by(c,b), min_by(b, c), max_by(b, c) FROM tbl GROUP BY a ORDER BY a").show())

Query 2: (group by min and max)
spark.time(spark.sql("SELECT a, min(c), max(c), min(b), max(b) FROM tbl GROUP BY a ORDER BY a").show())

Query 3: (min_by and max_by reduction)
spark.time(spark.sql("SELECT min_by(c,b), max_by(c,b), min_by(b, c), max_by(b, c) FROM tbl").show())

Query 4: (min and max reduction)
spark.time(spark.sql("SELECT min(c), max(c), min(b), max(b) FROM tbl").show())

hardware
data set
Q1
Q2
Q3
Q4

GPU
DS1
42,988
7,508
4,689
952

CPU
DS1
11,602
6,549
2,822
1,540

GPU
DS2(2048)
56,256
44,122
23,232
7,053

CPU
DS2(2048)
482,006
272,889
418,851
222,792

GPU
DS2(64)
50,855
41,485
9,438
4,074

GPU
DS3
42,686
3,698
4,518
953

CPU
DS3
10,837
7,073
2,645
1,540

GPU
DS4(2048)
57,188
45,757
20,856
6,928

CPU
DS4(2048)
492,970
289,821
418,500
221,631

GPU
DS4(64)
52,555
42,694
9,199
3,896

A few things to note from the results.

A min/max on a long (DS1 - Q2) is about 5.7 times faster than a struct(long, long) (DS1 - Q1) (because in CUDF we are forced to fall back to a sort aggregate for the struct operation.
The CPU saw less than a 2x speedup for the same change.
For DS2(2048) and DS2(64)  the GPU saw very little speedup because now both are sorted.
For DS2(2048) the CPU saw a very similar speedup of a little less that 2x.
We know that GPU has some issue with a small number of group by keys in aggregations compared to reductions. The CPU shows no such problems.

In general I think if we went with the argmin and then a gather for many fixed width types we would see a huge performance improvement. If the they are not fixed width, then we would not likely see much performance improvement, but we might see a very small amount, just because there would be less data to look at.

@revans2 Thanks for the testing and analysis! filed https://github.com/NVIDIA/spark-rapids/issues/11412


cmd/atlas/internal/migrate: default values for all revision columns to keep backwards compatability

Also adding a test for this case.
Resolves #1093 once and for all.

The following image shows it is working:

Checkout v0.5.0
Apply migration files
Checkout bugged v0.6.0
See the error as desribed by #1093
Checkout this branch
See it works


Generate Typescript types based on where the filter supplied

Problem
The types are not currently able to change based on any filters used in a query.
Given a model like
model User {
  email String?
}

and requesting users in the client
const users = await prisma.users.findMany({
  where: {
    NOT: {
      email: {
        equals: null,
      },
    },
  },
})

The current return type is
{ email: string | null }[]
Suggested solution
It would be great if the generated type was able to know that a null email was impossible.
{ email: string }[]
Alternatives
A work around is
import { User } from '@prisma/client'

const users = (await prisma.users.findMany({
  where: {
    NOT: {
      email: {
        equals: null,
      },
    },
  },
})) as UsersWithEmail[]

type UsersWithEmail = User & { email: string }

Additional context
I saw this stack overflow question that got me thinking initially.
https://stackoverflow.com/questions/69364534/ts-assert-non-null-property-of-object?noredirect=1#comment122617485_69364534

Would love this feature. My guess is it's not currently being prioritized given the lack of activity but just wanted to add a very emphatic +1.

I'd go even further than the example given.  Narrowing Type | null to Type example is definitely by far the most common annoyance, but I think it would be incredible if Prisma did this for every filter.
Given the following schema:
enum Status {
  PENDING
  ACTIVE
}
model Post {
   id        String
   status Status?
}

and the following query:
const post = await prisma.post.findUniqueOrThrow({ where: { id, status: Post.ACTIVE } });

It would be amazing if we could infer that the type of status must be specifically 'ACTIVE', rather than Status | null.


Ih/demandtech/combat section

Description
I design the Combat Section, I imported IzzyButton  button component into my work;
Fixes #7
Type of change
Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)

[ ] New feature (non-breaking change which adds functionality)

[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

[ ] This change requires a documentation update

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

[ ] Test A
[ ] Test B

Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my code
[x] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[ ] Any dependent changes have been merged and published in downstream modules

@mrxshinji @aftabrehan all correction made

@aftabrehan can help to approve this so i can merge it in?
Thank you


[BEAM-2271] Remove files from source assembly

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[ ] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[ ] Make sure tests pass via mvn clean verify.
[ ] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[ ] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

@aaltay I can't test this as I don't have permissions to follow the release guide. Can you test this out?

Coverage increased (+0.01%) to 70.887% when pulling 6eff61afbefad63a7d9d9baeea92273030c68746 on sb2nov:BEAM-2271-fix-release-filter-1 into 16f87f49f20796e29d01ed363a9097ea5420583c on apache:master.

Coverage increased (+0.02%) to 70.643% when pulling 0cc830e9d7b123d7d06868cb72febb0d4109b3cf on sb2nov:BEAM-2271-fix-release-filter-1 into 7e4719cd01f412c000bab61f99c687ab5c5491cc on apache:master.

R: @jbonofre

cc @davorbonaci as he had concerns about not adding a descriptor file as we have been inheriting that from org.apache.apache so far.
I've been poking more at other ways to make this work so any help would be great.

Any progress on this?

@jbonofre said he'll take a look but I wasn't able to make much more progress beyond this.


VM app manifest needs updating for win 8.1 and later

The manifest built into the Dolphin exe doesn't specify any OS compatibility settings. This means Windows will default to assuming that the app wants to run on Windows 8.0, and indeed it reports the OS version 6.2 (which is 8.0). In order to take advantage of later OS features, the manifest needs updating.

`OSVERSIONINFO current {Ctrl+D}
an OSVERSIONINFO(dwMajorVersion=10 dwMinorVersion=0 dwBuildNumber=14260 dwPlatformId=2 szCSDVersion='' wServicePackMajor=0 wServicePackMinor=0 wSuiteMask=256 wProductType=1)`


Add support for Play 2 App

Play 2 App is a run configuration provided by the Scala plugin

I will look into this someday. Until then - contributions are welcome.

I would like to contribute and take time to add Play 2 App support.
But do you know where the code is located for something similar ?
Is it a good start to go from GradleEnvFileProvider in order to create a PlayAppEnvFileProvider ?

Ok. i saw it use the GradleRunConfiguration to attach to it.
I created a run configuration of a Play 2 App configuration and then stored it as xml project file.
type="Play2FrameworkRunConfigurations" factoryName="Play 2 App"
I searched everywhere a Play2FrameworkRunConfigurations on the web and couldn't find anything.
Even in the scala plugin it's not referenced. (https://github.com/JetBrains/intellij-scala)
If anyone can help me to find the missing link from Play2FrameworkRunConfigurations and the ConfigurationFactory that it must extends
I found a AbstractTestRunConfigurationFactory  but i'm not sure if it's helpfull for us.
Maybe it's made by typesafe in another library but i didn't see any clear dependency in the build.sbt file.
I saw a Resolver.typesafeIvyRepo("releases") used but i don't find the link to a lib from it.

I have created a PR with sbt support, it's not updated to last IntelliJ version but it's something #167
I usually compile and run my play application with sbt tasks so it does the trick.


LITE-27894 Add PPRVersion schema

Fix ConfigurationSchema - remove deployment

There are conflicts that need to be fixed :)


Call method is not logging the traceback on failure

I see the below call is not logging traceback on failure and throwing the error message instead.
Call method   ${obj}  ${method}  ${args}
It would be great to log the traceback incase of any failure, it will be helpful to identify the issue soon.
Need the fix in version 4.1

@pekkaklarck

Could you please clarify the problem?

Could you please clarify the problem?

if call_method is failing with an error then we are not getting trace back of the failure, we are just getting the failure message and it is hard to debug such failures

Do you mean BuiltIn.call_method or something you created? Have you tried running tests with --loglevel debug?

Do you mean BuiltIn.call_method or something you created? Have you tried running tests with --loglevel debug?

It is with call_method and I tried with --loglevel debug also but no use

call_method is raising the exception but  logging the traceback as None

It's true that if I have something like
*** Test Cases ***
Example
   Call method    string    upper    bad

the traceback is just this:
Traceback (most recent call last):
  None
RuntimeError: Calling method 'upper' failed: TypeError: upper() takes no arguments (1 given)

I don't think there could be any traceback to show in this case, though. You don't get anything more if you run same thing directly in Python:
>>> 'string'.upper('bad')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: str.upper() takes no arguments (1 given)

Do you have a concrete example where there should be a traceback but isn't?

Yes, Assume that we have one python module and I have called a method and want to see the at which line of the code the failure appeared?? how to see at which line it is failed

If you see the above example the traceback is given(if you execute python directly)

Yeah, there's problem if the exception occurs somewhere else. The reason is that the original traceback is lost because we raise a new RuntimeError if calling the method fails. Python 3 makes it possible to attach context information to the exception like raise NewError from old but it seems that our traceback formatting code needs to be enhanced slightly as well. Fixing this is so easy that I can still include it into RF 5.1.
Notice that the fix won't be backported to RF 4.1 or any other older version. If you need to use older versions and must have a fix, you need to patch your installation yourself. Alternatively, and probably better, you should change your code into a library and call those methods as keywords.

A trivial fix would be just reraising the original exception, but then the error message would change and that could cause backwards compatibility problems. Better enhance the code so that traceback is preserved.

It would actually be enough to fix only the traceback formatting code to get the original traceback included. The problem there is that if the exception has no normal traceback to show, traceback of the original exception isn't shown either. This was a mistake made when adding support for chained exceptions in general in RF 5.0 (#4039). I decided to add from err to call_method anyway because it makes the resulting error a bit better.

Thanks


get_rect() functions are getting really gnarly

We should either simplify them with refactoring things, make them easier to understand, or just document them really well :confused: . Getting pretty difficult to understand with imgui in the mix

@clewis7 thoughts?

nevermind I simplified this and I think there is a weird thing with some canvases where they sporadically initiate at a weird small size which is not the same as the size they were forced to be set to :shrug:


Doesn't work if multiple services share the same redis

From this page: https://github.com/mperham/sidekiq/wiki/Middleware

Obscure Scenario: If you share a single redis instance between multiple services, and use client-side middleware on your Sidekiq server, be warned that it might have to deal with workers from unexpected queues. Sidekiq uses a singe redis set to handle scheduled workers, and any server might requeue a worker that's due to be processed, even if it's not for a queue it was configured to process. When it does so, your middleware might be passed a worker name that does not exist in the current ruby context and you might see uninitialized constant exceptions or similar errors.

Since version 0.8.0 the middleware client is doing a Module.const_get without handling any potential NameError. This causes annoying breakage. I'd like to propose something like this:
  def call(worker_class, msg, queue, redis_pool=nil)

    # Determine the actual job class
    klass = msg["args"][0]["job_class"] || worker_class rescue worker_class

    job_class = if klass.is_a?(Class)
                  klass
                elsif Module.const_defined?(klass) # Ensure constant is defined before attempting access
                  Module.const_get(klass)
                else
                  nil
                end

    # Store data if the job is a Sidekiq::Status::Worker
    if job_class && job_class.ancestors.include?(Sidekiq::Status::Worker)
      initial_metadata = {
        jid: msg['jid'],
        status: :queued,
        worker: Sidekiq::Job.new(msg, queue).display_class,
        args: display_args(msg, queue)
      }
      store_for_id msg['jid'], initial_metadata, job_class.new.expiration || @expiration, redis_pool
    end

    yield

  end

end

+1

Fixed in v1.1.0


Make SeleniumHtmlReportPublisher a SimpleBuildStep

https://issues.jenkins-ci.org/browse/JENKINS-28388
I think this could work....
There are no existing tests, and I have no idea how testing works for jenkins plugin development, have to read first....

Thank you for a pull request! Please check this document for how the Jenkins project handles pull requests

I have no idea how testing works for jenkins plugin development

There is a learning curve I am afraid. If you are interested in trying, I can point you to tests added in various other PRs which add workflow compatibility for build steps.

There is a learning curve I am afraid. If you are interested in trying, I can point you to tests added in various other PRs which add workflow compatibility for build steps.

this would be nice... thx

So https://github.com/jenkinsci/config-file-provider-plugin/pull/9 is an example I wrote. (That was for a SimpleBuildWrapper, not a SimpleBuildStep, but for purposes of looking at test infrastructure it should not matter.)

Could perhaps benefit from making one or both parameters optional (as noted in tip), but overall :+1:

Hi,
Should I close this request or is there still something to do here?
Not really sure about the "pull request/merge" process...
Regards,
Stefan

Looks like @morca has done no releases for several years, so I think we can assume this is abandoned. Best to ask on jenkinsci-dev if you want to become a maintainer.

Oops, I left comments on the last commit by accident—meant to leave them on the PR diff. So they will not be automatically shown as “on an outdated diff” if you address them. Sorry.


ClassNotFound - Tesla (1.10)

Issue Description:
[USING SPONGE - Version included below]
https://github.com/SleepyTrousers/EnderIO/issues/3766
The issue above is very similar to the one I'm facing, however, I haven't enabled debug logging - not knowingly and I don't know how to? (or to disable).
What happens:
Huge console spam -> http://pastebin.com/BaL0pHwC
Steps to reproduce:

My plugin which spawns items on certain block breaks seems to be related?

Affected Versions (Do not use "latest"):

EnderIO: 1.10.2-3.1.164
EnderCore: 1.10.2-<IP_ADDRESS>-beta
Minecraft: 1.10
Forge: 2202
Sponge: spongeforge-1.10.2-2221-5.1.0-BETA-2109

Your most recent log file where the issue was present:
as above.
Apologies in advance if this a duplicate issue but I did look :/

@HenryLoenwind apologies. pasted wrong link

Two things:
First:
        if (Log.LOGGER.isDebugEnabled()) {
          e.printStackTrace();
        }

Second: You game is crashing. But some mod is preventing the crash from stopping the game or printing a crash report. That is bad. (Hint: It's not Ender IO...)

How do I disable the logger then?


Create a Sprint Calendar

I loved this idea from @JanNash so I put it up for discussion here
If that works what about putting our next 3 sprints in a calendar and reserve the days for code reviewing, issue picking, estimation meetings... ?

I guess this was fixed by #38 ?

You guessed well @JanNash


Tutorial

Slightly modify for Interspeech 2019 tutorial

Coverage remained the same at 89.328% when pulling 3bb3682a5f409cba6156e253060a8e6a7b952013 on tutorial into ebbb7ad8d7a1f7fc61f1c4a6ab8211a870804c6e on master.