Keycloak and Headlamp

Hello,
We are unable to get Headlamp working with Keycloak. Do you have an example on how to configure Keycloak/Headlamp?
Thanks.

Hi, what's the use-case you're trying to accomplish when you write "Headlamp working with Keycloak"?

We are running in K8s and we have a Keycloak server up and running, however we are unable to get it working with Keycloak and the Headlamp example on the webpage. We do have this config:
kind: Service
apiVersion: v1
metadata:
  name: headlamp
  namespace: kube-system
spec:
  ports:
    - port: 4466
      targetPort: 4466
  selector:
    k8s-app: headlamp
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: headlamp
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: headlamp
  template:
    metadata:
      labels:
        k8s-app: headlamp
    spec:
      containers:
      - name: headlamp
        image: quay.io/kinvolk/headlamp:latest
        args:
          - "-in-cluster"
          - "-plugins-dir=/headlamp/plugins"
          - "-oidc-client-id=kubernetes"
          - "-oidc-client-secret=2f887c20-85b0-43ea-9911-56a911a396b2"
          - "-oidc-idp-issuer-url=https://sso.mydomain.com/auth/realms/sso/protocol/openid-connect/token"
        ports:
        - containerPort: 4466
        livenessProbe:
          httpGet:
            scheme: HTTP
            path: /
            port: 4466
          initialDelaySeconds: 30
          timeoutSeconds: 30
      nodeSelector:
        'beta.kubernetes.io/os': linux

We do think that the -oidc-idp-issuer-url is the wrong one, but we do not know which value should be entered here.

Hi @ws-way , just a quick message to lete you know we're looking into this and will let you know once we have more news.

Hi @ws-way ,
I couldn't look as deep as I wanted due to time constraints, but assuming you have you have set up your Kubernetes API server with your KeyCloak's OIDC settings, you should be able to use Headlamp with the args you added but the issuer URL as you guessed doesn't look right.
Can you try with
https://sso.mydomain.com/auth/realms/sso
(sso is your realm)
Also, don't forget to add the oidc-callback to your redirect valid URLs, e.g. https://HEADLAMP_PREFIX.mydomain.com/oidc-callback
Please let me know how it goes after you try again (and if you find more "gotchas", please do share them).

Thanks, our Keycloak is working for all other Kubernetes 'stuff', except for Headlamp. We have tried https://sso.mydomain.com/auth/realms/sso already, but no luck.
We use a wildcard for the callback, this should be ok.
Any ideas? Can we see a log or debug it?

Thanks, our Keycloak is working for all other Kubernetes 'stuff', except for Headlamp. We have tried https://sso.mydomain.com/auth/realms/sso already, but no luck.
We use a wildcard for the callback, this should be ok.
Any ideas? Can we see a log or debug it?

@ws-way , what exactly happens when you access Headlamp? Do you see the sign-in button? What happens when you click on it?
In the problems we had before (using dex) we had good hints just the replies during the OIDC exchange.
As for the logs, you should be able to see them for the Headlamp pod in k8s.
We're a bit busy ATM but maybe we try setting up a K8s cluster with KeyCloak and try it out here by next week.
Let me know of any hints you may see in the logs or the pages during the OIDC attempts.

@joaquimrocha , thanks. Yes, we do see the sign-in button, but when clicking on it, we do get an error:

BTW, this is tested with https://sso.mydomain.com/auth/realms/sso.
Keycloak log:
8:59:31,346 ERROR [org.keycloak.services] (default task-193) KC-SERVICES0093: Invalid parameter value for: scope
18:59:31,347 INFO [org.jboss.aerogear.keycloak.metrics.MetricsEventListener] (default task-193) Received user event of type LOGIN_ERROR in realm sso
18:59:31,350 WARN [org.keycloak.events] (default task-193) type=LOGIN_ERROR, realmId=sso, clientId=kubernetes, userId=null, ipAddress=<IP_ADDRESS>, error=invalid_request, response_type=code, redirect_uri=https://headlamp.facility.mydomain.com/oidc-callback, response_mode=query
18:59:31,399 WARN [org.keycloak.protocol.oidc.utils.OAuth2CodeParser] (default task-193) Invalid format of the code
18:59:31,400 INFO [org.jboss.aerogear.keycloak.metrics.MetricsEventListener] (default task-193) Received user event of type CODE_TO_TOKEN_ERROR in realm sso
18:59:31,402 WARN [org.keycloak.events] (default task-193) type=CODE_TO_TOKEN_ERROR, realmId=sso, clientId=kubernetes, userId=null, ipAddress=<IP_ADDRESS>, error=invalid_code, grant_type=authorization_code, client_auth_method=client-secret
18:59:31,416 WARN [org.keycloak.protocol.oidc.utils.OAuth2CodeParser] (default task-193) Invalid format of the code
18:59:31,417 INFO [org.jboss.aerogear.keycloak.metrics.MetricsEventListener] (default task-193) Received user event of type CODE_TO_TOKEN_ERROR in realm sso
18:59:31,419 WARN [org.keycloak.events] (default task-193) type=CODE_TO_TOKEN_ERROR, realmId=sso, clientId=kubernetes, userId=null, ipAddress=<IP_ADDRESS>, error=invalid_code, grant_type=authorization_code, client_auth_method=client-secret

@ws-way Thanks! This was very helpful.
Seems like KeyCloak requires a scope value to be passed and Headlamp doesn't expose that option. I have opened #210 with more context for tracking this and we will prioritize it for the next days as it's a very important feature we're missing. I will keep this one up to date with that.

@joaquimrocha , thanks again. I hope you get it fixed soon.

Hi @ws-way,
I took another look today and tested with a local KeyCloak instance. The issue seems to be that Headlamp always passes openid,profile,email and groups as the scopes to OIDC providers, but KeyCloak is picky about these and fails because of groups.
I have created a branch with a couple of quick changes to expose the scopes as needed. I removed the groups from the default options for this test, but will likely add it to the final changes so we maintain backward compatibility for now (meaning you'd have to override the new scopes option).
I have pushed an image built from this branch to quay.io/kinvolk/headlamp:wip-oidc .
With this image however, you shouldn't need to override the scopes as I removed the groups one. You should just need to change the tag in your K8s deployment so it gets this version.
If it still fails and you want to check changing the scopes, you can pass the scopes option as -oidc-scopes profile,email,other-scopes.
Please let me know if this works for you so I can proceed to a cleaner solution.

@ws-way , we have a new beta release out but I decided to keep the groups out of the scopes by default. So you shouldn't need any changes on your side for using the beta and the next final release.
More details in https://kinvolk.io/docs/headlamp/latest/installation/in-cluster/#scopes

Hi,
Thanks, I got it working, however I found one issue:
If you have a trailing / with the -oidc-idp-issuer-url , there is a panic in the headlamp container.
Example:
Error: - "-oidc-idp-issuer-url=https://sso-test.test.mydomain.com/auth/realms/sso/"
Ok: - "-oidc-idp-issuer-url=https:/sso-test.test.mydomain.com/auth/realms/sso"

2021/03/17 18:31:35 Error while fetching the provider from https://sso-test.test.mydomain.com/auth/realms/sso/ error oidc: issuer did not match the issuer returned by provider, expected "https://sso-test.test.mydomain.com/auth/realms/sso/" got "https://sso-test.test.mydomain.com/auth/realms/sso"
2021/03/17 18:31:35 http: panic serving <IP_ADDRESS>:48000: runtime error: invalid memory address or nil pointer dereference

Kind regards,
WS

Hi @ws-way ! Thanks for trying it out once again!
That's a good catch.
From the logs you shared, it seems like the server aborts after this error, we should at least report but not let the server abort because of this (but we will try to see if we can prevent the failure for equivalent URLs of course).

I created a PR for not having the server crash on this issue, but since the OIDC plugin is performing a hard comparison on the issuer URL, I don't want ATM to go and work around this by having a repeated call with a trimmed URL.

Thanks!


I need help of white balance

Hi,
I have some encrypted jpeg but i repaired to viewer with wrong color, could you upload the compiled exe file?
I need to use it as batch on command line of Windows

sorry i can't help :(


DAC Fix for Win9x bootlogo progressbar

I don't know how, but I've randomly fixed it.
Before:

After:

QEMU:
static inline int c6_to_8(int v)
{
    int b;
    v &= 0x3f;
    b = v & 1;
    return (v << 2) | (b << 1) | b;
}

Halfix:
static uint8_t c6to8(uint8_t a)
{
    if (vga.vbe_enable & VBE_DISPI_8BIT_DAC)
        return a;
    uint8_t b = a & 1;
    return a << 2 | b << 1 | b;
}

QEMU:
Halfix:

Watch out, we can't use GPL code in v86 without tainting the license (I'll accept this one since the code is trivial).

QEMU:
Halfix:

Watch out, we can't use GPL code in v86 without tainting the license (I'll accept this one since the code is trivial).

Why don't you want GPL code?

Cheers!


Add CESM POP 1-degree

This is the start of a pipeline for CESM run v5_rel04_BC5_ne30_g16. I have followed steps 1-5 laid out by @cisaacstern in Issue #46.
@cisaacstern Could I get a bit more guidance on how to write the meta.yaml and pipeline.py files? It looks like the instructions at https://github.com/paigem/staged-recipes.git aren't quite finished yet.

@paigem, thanks for kicking off this PR!
The staged-recipes documentation is actually quite out of date. In response to your question, I've begun the process of updating the README in commit https://github.com/pangeo-forge/staged-recipes/pull/57/commits/08cff9692f3b3ba74dd0ee4b0f8da065f6ec0c98 of https://github.com/pangeo-forge/staged-recipes/pull/57. The big picture is that pipeline.py files are actually no longer needed, and have been replaced by the recipe.py file you've already created in this PR.
As for meta.yaml, that will be needed for automating recipe execution by a Bakery, but is not required to get started on your recipe. In fact, it is also possible to manually execute your recipe, thereby creating your Zarr store without a meta.yaml. We can revisit this once the recipe itself is where we want it. :smile:
For a next step, I've pushed an outline for you to fill in to your recipe.py. In order to see the outline locally, you'll need to fetch it. From within your local PR branch, run:
git fetch origin
git merge origin/cesm-pop-lowres-1deg

Now, when you open recipes/cesm-pop-lowres-1deg/recipe.py, it should look like this.
You will want to change the f-string returned from the make_full_path function here so that it reflects a real path for downloading the source files. Place {variable} in the part of that path where the variable names occur. I do recall we'll need credentials for these downloads, but you can ignore that for now, as those will be added in at the execution phase (either manual, or in a Bakery).
Some more detail on make_full_path functions is provided in the docs here. We do not have a time dimension in the make_full_path function for this recipe, because based on https://github.com/pangeo-forge/staged-recipes/issues/46#issue-910090895, it appears the data provider wraps all of the time steps for a given variable in a single netCDF file.
The merging of the variables is achieved by the MergeDim instance. To complete this, you'll just need to fill out the vars list with the names of your 14 variables as they appear in the source file paths.
Finally, we'll need a chunk size here.
Once you've made these changes to recipe.py (or as many of them as make sense at this time), you can commit them back to GitHub with:
git add -A
git commit -m "<some descriptive message here>"
git push origin cesm-pop-lowres-1deg

Feel free to ask any questions if any of the above doesn't make sense. It's still early days for us onboarding new recipe contributors, so if something doesn't make sense, it's likely my fault for not explaining it well!

Thanks @cisaacstern for the great instructions! I think I have mostly completed all of the above steps:

added the recipe.py file
added the download path for each variable (note: this path takes you to a sign-in page, so authentication is needed for this step)
added a list of strings with all 14 variables
put in an estimated chunk size, based only on the size of the full dataset and the number of time steps

currently, I put {time=20}, and estimate that 20 time steps should be roughly 70 MB
this should be double checked before the final conversion to Zarr

@cisaacstern just checking on the progress here! Let me know if there is something more I need to do in order to move this forward.

@paigem, thanks for following-up! I made a few commits to the recipe to get it ready to run.
I then made an OpenID account, but when I click "Download Options" here, I get the Authorization Required response copied below.
Rather than requesting and waiting for authorization on my account, perhaps it's easiest if we use your credentials to cache the netCDFs? If you're comfortable with that approach, you can securely message them to me at https://keybase.io/cstern. If not, I'll make a request for authorization on my account.
"""
Authorization RequiredYou do not have enough privileges to access the requested resource.Please request membership in the following group to gain access to the requested resource:

Group Name
Description

CCSM
CCSM (Community Climate System Model) users

"""

This recipe has been blocked by three issues, the first two of which I have (albeit provisional) solutions for:

We don't want @paigem's secret API token to end up as part of the names of a lot of public files on the GCS cache. Installing from https://github.com/pangeo-forge/pangeo-forge-recipes/pull/167 solves this.
How do we append the API token to the recipe urls? We don't want to push it to the public PR. https://github.com/pangeo-forge/staged-recipes/pull/56/commits/0548f8bd592a513be9acbfd4dc44f8a477af7af0 plus the usage pattern described in https://github.com/pangeo-forge/pangeo-forge-recipes/pull/167#issuecomment-881109187 resolves this for manual execution.

The third issue is as-yet unresolved, and I'll make it it's own comment because it's (I think) unrelated to these.

Thank you for these updates @cisaacstern!

The actively blocking issue for this recipe is what appear to be incorrect HTTP Content Length headers.

Unfortunately, the authentication issues referenced in https://github.com/pangeo-forge/staged-recipes/pull/56#issuecomment-881111273 mean I can't provide a fully reproducible example, but I will describe the situation in detail below.

For the first input url, curl reads the headers as:
curl -I https://tds.ucar.edu/thredds/fileServer/datazone/campaign/cesm/collections/ASD/v5_rel04_BC5_ne30_g16/ocn/proc/tseries/daily/v5_rel04_BC5_ne30_g16.pop.h.nday1.HMXL_2.00010101-01661231.nc?api-token=$TOKEN

HTTP/1.1 200
Date: Fri, 16 Jul 2021 01:44:31 GMT
Server: Apache
Last-Modified: Thu, 30 Oct 2014 01:38:26 GMT
Accept-Ranges: bytes
Content-Type: application/x-netcdf
Content-Length:<PHONE_NUMBER>
Set-Cookie: JSESSIONID=EA92736C6194748B575F70E199020F8E;path=/;Secure;HttpOnly
Via: 1.1 tds.ucar.edu

fsspec agrees on Content-Length, because bytes=1590000000-1598457681 is the last range it fetches when called from within _copy_btw_filesystems:
fsspec.http - DEBUG - Fetch range for <File-like object HTTPFileSystem, https://tds.ucar.edu/thredds/fileServer/datazone/campaign/cesm/collections/ASD/v5_rel04_BC5_ne30_g16/ocn/proc/tseries/daily/v5_rel04_BC5_ne30_g16.pop.h.nday1.HMXL_2.00010101-01661231.nc?api-token=<TOKEN>>:<PHONE_NUMBER>-1598457682
fsspec.http - DEBUG - https://tds.ucar.edu/thredds/fileServer/datazone/campaign/cesm/collections/ASD/v5_rel04_BC5_ne30_g16/ocn/proc/tseries/daily/v5_rel04_BC5_ne30_g16.pop.h.nday1.HMXL_2.00010101-01661231.nc?api-token=<TOKEN> : bytes=1590000000-1598457681
pangeo_forge_recipes.storage - DEBUG - _copy_btw_filesystems copying block of 8457682 bytes
pangeo_forge_recipes.storage - DEBUG - _copy_btw_filesystems reading data
pangeo_forge_recipes.storage - DEBUG - FSSpecTarget.open yielded
pangeo_forge_recipes.storage - DEBUG - _copy_btw_filesystems done

The problem is that this file is not<PHONE_NUMBER>/1e9 ≈ 1.6 GBs, but rather<PHONE_NUMBER>6 ≈ 18.8 GBs. So fsspec gives us an incomplete file. We know the actual file size because the wget script provided by NCAR (copied to gist here, with tokens removed) somehow determines the actual length (and downloads without issue):
Running wget_cmems.sh version:
wget command: wget -c --user-agent=wget/1.21.1/esg/3.0.34-20210630-162451/created/2021-07-12T18:29:21-06:00
v5_rel04_BC5_ne30_g16.pop.h.nday1.HMXL_2.00010101-01661231.nc ...Downloading
--2021-07-15 18:56:46--  https://tds.ucar.edu/thredds/fileServer/datazone/campaign/cesm/collections/ASD/v5_rel04_BC5_ne30_g16/ocn/proc/tseries/daily/v5_rel04_BC5_ne30_g16.pop.h.nday1.HMXL_2.00010101-01661231.nc?api-token=$TOKEN
Resolving tds.ucar.edu (tds.ucar.edu)... <IP_ADDRESS>
Connecting to tds.ucar.edu (tds.ucar.edu)|<IP_ADDRESS>|:443... connected.
HTTP request sent, awaiting response... 206
Length:<PHONE_NUMBER>6 (17G),<PHONE_NUMBER>7 (16G) remaining [application/x-netcdf]
Saving to: ‘v5_rel04_BC5_ne30_g16.pop.h.nday1.HMXL_2.00010101-01661231.nc’

@martindurant, any suggestions for how to move forward? I've tried setting fsspec_open_kwargs={"block_size": 0} to stream the file instead, which does start working, and then seems to reliably stall after a minute or two, before the transfer is complete. I've also tried (perhaps naively) to explicitly pass the known length as fsspec_open_kwargs={"size":<PHONE_NUMBER>6}, which turns out to not be a recognized kwarg.
The only thing I can think is that NCAR's wget script is doing some further authentication using the JSESSIONID in the Set-Cookie header, but not certain (a) if that's even technically plausible; or (b) how to test and/or replicate it with fsspec if so.
cc @rsignell-usgs b/c we discussed this yesterday on the Pangeo call, and perhaps your THREDDS expertise will yield some key insights

I've tried setting fsspec_open_kwargs={"block_size": 0} to stream the file instead, which does start working, and then seems to reliably stall after a minute or two,

Can you figure out why? This is with dask running? Is h5py or other library in on this download? You can try interrupting the coroutines or thread to find out at least where its waiting.

Can you figure out why?

I'm executing this recipe incrementally in Jupyter (no Dask).
Looks now like this isn't an fsspec issue. I believe that with {"block_size": 0} this (a) plus this (b)  logger.debug in pangeo_forge_recipes/storage.py just output so many lines to stdout that Jupyter can't keep up with rendering them all. At first, a bunch of debug lines appear, then the cell appears to stall. But in reality, only the output render has stalled; fsspec is actually cranking away in the background. Eventually, Jupyter catches up and provides a warning:

 IOPub message rate exceeded
IOPub message rate exceeded.
The notebook server will temporarily stop sending output
to the client in order to avoid crashing it.
To change this limit, set the config variable
`--NotebookApp.iopub_msg_rate_limit`.

Current values:
NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)
NotebookApp.rate_limit_window=3.0 (secs)

And a little while later, the output render resumes and the download is completed. I am about to submit a PR with a proposed solution to this issue.

@rabernat, now that I have this recipe's files cached to GCS, I'm encountering an issue where they take an unexpectedly long time to open with xr.open_dataset. This means prepare_target hangs for many minutes here.
As far as I can tell, these are valid netCDF files which have cached transferred in full, because I did get a bunch of intelligible debug information back from xarray when I re-installed it with all these added print statements. The logging just goes on for minutes upon minutes without returning the dataset, however, which is quite different from the "instantaneous" lazy loading I've come to expect.
This repo contains an notebook (binder link included) which demonstrates this behavior with an attempt to open the ssh_2 netCDF input for this recipe:
https://github.com/cisaacstern/cesm-open-dataset
NCAR provides a "netCDF header" for each of the input files, and I've also provided the ssh_2 header alongside the notebook as a text file. The only item there that seems particularly noteworthy to me is that it contains 60,590 timesteps, which is considerably more than for any other netCDF file I've worked with thus far.
Is there anything else in that header that points to why this might be taking so long to open? Are there any strategies we might employ to make xr.open_dataset run faster in this case? I assume if I just left my laptop open for a very long time, this recipe might work as-is, but I figured this was a good opportunity to make sure I wasn't overlooking something obvious.

As far as I can tell, these are valid netCDF files which have been transferred to the cache uncorrupted,

If the netCDF files are very large, it can indeed take a long time to initialize them. This is why we don't like to use netCDF on cloud storage! I would recommend playing around with this manually. Since this recipe has many variables, each one of the variable files needs to be opened up by prepare_target. Currently this happens in serial.
To get some intuition for this, I'd recommend trying to directly open one of the files from the cache, outside of pangeo_forge_recipes. Could you just do?
%time ds = xr.open_dataset(cache_path)

That will give us some valuable quantitative information. This is also a place where it matters a lot where you are running the recipe. Are you sure you are in the same cloud region as your cache bucket?
One way we might be able to make this go a bit faster would be to use the approach suggested in https://github.com/pangeo-forge/pangeo-forge-recipes/pull/82, where we parallelize the opening of the files required by prepare_target. But that would introduce a lot of complexity.
In general, the most important property of Pangeo Forge recipes is that they work, not that they are extremely fast. We only have to run the recipe once. It's acceptable if prepare_target takes an hour for certain recipes.

Could you just do?
%time ds = xr.open_dataset(cache_path)

Yes, I did try that in https://github.com/cisaacstern/cesm-open-dataset/blob/main/cesm_open_dataset.ipynb. But at the time I didn't know that ...

It's acceptable if prepare_target takes an hour for certain recipes.

... so now, I can move forward with this by just "letting it run".

This is also a place where it matters a lot where you are running the recipe. Are you sure you are in the same cloud region as your cache bucket?

In this case, I believe yes: opening from the pangeo-forge GCS cache (which I'm pretty sure is in us-central1), from a Pangeo Cloud notebook at us-central1-b.gcp.pangeo.io. But good to know to keep an eye out for this in the future.

@paigem, this data is now available as a zarr store on OSN:
import fsspec
import xarray as xr

path = "s3://Pangeo/pangeo-forge/cesm_pop_lowres_1deg/v5_rel04_BC5_ne30_g16.zarr"
mapper = fsspec.get_mapper(
    path, anon=True, client_kwargs={'endpoint_url': 'https://ncsa.osn.xsede.org'},
)
ds = xr.open_zarr(mapper, consolidated=True)
print(ds)

<xarray.Dataset>
Dimensions:             (nlat: 384, nlon: 320, time: 60590, z_t: 60, z_w: 60, d2: 2, z_t_150m: 15, z_w_bot: 60, z_w_top: 60)
Coordinates:
    TLAT                (nlat, nlon) float64 dask.array<chunksize=(384, 320), meta=np.ndarray>
    TLONG               (nlat, nlon) float64 dask.array<chunksize=(384, 320), meta=np.ndarray>
    ULAT                (nlat, nlon) float64 dask.array<chunksize=(384, 320), meta=np.ndarray>
    ULONG               (nlat, nlon) float64 dask.array<chunksize=(384, 320), meta=np.ndarray>
    day                 (time) int64 dask.array<chunksize=(200,), meta=np.ndarray>
  * time                (time) object 0001-01-02 00:00:00 ... 0167-01-01 00:0...
    time_counter        (time) int64 dask.array<chunksize=(200,), meta=np.ndarray>
  * z_t                 (z_t) float32 500.0 1.5e+03 ... 5.125e+05 5.375e+05
  * z_t_150m            (z_t_150m) float32 500.0 1.5e+03 ... 1.35e+04 1.45e+04
  * z_w                 (z_w) float32 0.0 1e+03 2e+03 ... 5e+05 5.25e+05
  * z_w_bot             (z_w_bot) float32 1e+03 2e+03 3e+03 ... 5.25e+05 5.5e+05
  * z_w_top             (z_w_top) float32 0.0 1e+03 2e+03 ... 5e+05 5.25e+05
Dimensions without coordinates: nlat, nlon, d2
Data variables: (12/63)
    ANGLE               (nlat, nlon) float64 dask.array<chunksize=(384, 320), meta=np.ndarray>
    ANGLET              (nlat, nlon) float64 dask.array<chunksize=(384, 320), meta=np.ndarray>
    DXT                 (nlat, nlon) float64 dask.array<chunksize=(384, 320), meta=np.ndarray>
    DXU                 (nlat, nlon) float64 dask.array<chunksize=(384, 320), meta=np.ndarray>
    DYT                 (nlat, nlon) float64 dask.array<chunksize=(384, 320), meta=np.ndarray>
    DYU                 (nlat, nlon) float64 dask.array<chunksize=(384, 320), meta=np.ndarray>
    ...                  ...
    sea_ice_salinity    float64 ...
    sflux_factor        float64 ...
    sound               float64 ...
    stefan_boltzmann    float64 ...
    time_bound          (time, d2) object dask.array<chunksize=(200, 2), meta=np.ndarray>
    vonkar              float64 ...
Attributes:
    Conventions:   CF-1.0; http://www.cgd.ucar.edu/cms/eaton/netcdf/CF-curren...
    calendar:      All years have exactly  365 days.
    cell_methods:  cell_methods = time: mean ==> the variable values are aver...
    contents:      Diagnostic and Prognostic Variables
    history:       none
    nsteps_total:  24
    revision:      $Id: tavg.F90 41939 2012-11-14 16:37:23Z<EMAIL_ADDRESS>$
    source:        CCSM POP2, the CCSM Ocean Component
    start_time:    This dataset was created on 2014-03-18 at 17:05:44.6
    tavg_sum:      64800.0
    title:         v5_rel04_BC5_ne30_g16

Do let me know if that works for you, and if that data looks as you expect it to.
Oh, and please feel free to ignore the added coordinates "day" and "time_counter". I added them simply to help me save out binned subset files using the netcdf_subsets module. I'll follow-up with an explanation of that module in a separate comment.

@rabernat, the usage pattern for the subsetting module I mentioned in https://github.com/pangeo-forge/staged-recipes/pull/56#issuecomment-899180590 is:
from subset_recipe import variables

fs_gcs =  # cache location
target_bins = 60  # number of bins required to bring each subset < 500 MB

for var in variables:
    subset_var = NetCDFSubsets(
        cache_fs=fs_gcs,
        cache_dir=f"{cache_base}/{recipe_name}",
        var_name=var,
        target_bins=target_bins,
        concat_dim_name="time",
        concat_dim_length=60590,
    )
    subset_var.subset_netcdf()

I ran this code for each of this recipe's 14 source files, saving out netCDF subsets (rather than using recipe.subset_inputs), because the source files from NCAR average about ~30 GB each, and take between 5-8 minutes for xarray to open from the GCS cache (even when running same-region/same-provider compute). Which means, of course, that the recipe.subset_inputs approach would take 14 files x 60 subsets x 5 mins = 70 hours in file-opening time.
Once I'd saved out the netCDF subsets (in 4 parallel batches, ~4 hrs total runtime), I ran the recipe from a new subset_recipe.py, and the execution took about 9 hours start to finish, even with copy_input_to_local_file=True (to avoid any risk of interference from the h5py hanging issues we've been seeing, e.g., in https://github.com/pangeo-forge/staged-recipes/pull/68#issuecomment-897866695 and https://github.com/pangeo-forge/pangeo-forge-recipes/issues/177).
Noting this approach for my own future reference, and in case it's useful for pushing through other recipes that face similar constraints, even though I imagine it's not an ideal long-term solution.

the source files from NCAR average about ~30 GB each, and take between 5-8 minutes for xarray to open from the GCS cache (even when running same-region/same-provider compute)

Just out of curiosity, what if you open the files with directly with netCDF4, rather than xarray? How long does it take to open them and list the variables? I'm also curious how long it would take to create a reference fs for these files (cc @martindurant).

Hooray thanks @cisaacstern!! It looks great so far, but for two comments/questions:

It looks like the SST variable did not get transferred here. I count 14 variables from the original source files and only 13 on the cloud, and it looks like SST is the missing variable. This is one of the key output variables that users will be interested in, so if it's not too much trouble, it should probably be added to the cloud.
Is this data sitting on AWS S3 storage? Perhaps it doesn't matter, but the other NCAR CESM models are (I believe) on Google Cloud Storage, and I have been using Pangeo's Google Cloud deployment. I just want to verify that I won't incur excess costs when accessing data stored on S3 from Google Cloud.

It looks like the SST variable did not get transferred here.

Thanks for catching this, Paige. This was indeed an oversight on my part. When I ran the NetCDF file subsetting described in https://github.com/pangeo-forge/staged-recipes/pull/56#issuecomment-899187300, I overlooked the fact that "SST" is a substring of "SST2", which meant that the method _fn_from_var grabbed the wrong file to subset from the cache. Referring back the my logs confirms that the subsets for SST were incorrectly created from the SST2 source file:
Filename for SST is pangeo-forge-us-central1/pangeo-forge-cache/cesm_pop_lowres_1deg/v5_rel04_BC5_ne30_g16/80554384345dbfeb88651f95e0ef7065-https_tds.ucar.edu_thredds_fileserver_datazone_campaign_cesm_collections_asd_v5_rel04_bc5_ne30_g16_ocn_proc_tseries_daily_v5_rel04_bc5_ne30_g16.pop.h.nday1.sst2.00010101-01661231.nc

I'm now re-subsetting the SST source file with the correction I just made in https://github.com/pangeo-forge/staged-recipes/pull/56/commits/f6062f360ce5b96835fc381893326ab68198b0be, and will fix the zarr store once that's complete. I'll notify you on this thread when the corrected zarr is available.

Is this data sitting on AWS S3 storage? I just want to verify that I won't incur excess costs when accessing data stored on S3 from Google Cloud.

Yes, this data is on Open Storage Network (OSN), which is an AWS S3 instance. OSN is not a "requester pays" bucket, however, so accessing it from Pangeo's Google Cloud deployment will not result in any egress costs for the requester. (I do all of my OSN bucket work from the Google Cloud as well.)

I'll notify you on this thread when the corrected zarr is available.

Rebuild is complete. The complete dataset should now be accessible via the code block provided above in https://github.com/pangeo-forge/staged-recipes/pull/56#issuecomment-899180590.

pre-commit.ci autofix


Postprocessing step np.nan_to_num

Hello thank you for providing us with this package. In the post processing step of acs.ACSIncome np.nan_to_num is applied with the second argument -1. Second argument of np.nan_to_num is [copy] (https://numpy.org/doc/stable/reference/generated/numpy.nan_to_num.html) which is a boolean variable. Does -1 implies copy=False or the goal was to replace missing values with -1?

-1 implies copy=True (can test this with assert bool(-1)). copy=True is the intended behavior for the postprocessing by the authors as you can see on folktables.py:101. This leads me to believe it is coincident that -1 as the second arg causes a different intended behavior (copying) and that the authors meant for nan's to be replaced by -1, which would be achieved with lambda x: np.nan_to_num(x, nan=-1) (this by default also has copying)
The same postprocess (lambda x: np.nan_to_num(x, 1) is applied to all BasicProblem's defined in acs.py which would changes the results of using the package. When you change the postprocessing of ACSEmployment to nan=-1 the equality of opportunity violation reported in the first README example for Alabama changes from 0.0871 to 0.0789.
Because the values of ACS variables are largely categorical I think there should maybe be more discussion around what to do with NaN values rather than just replace with -1. Perhaps entries with NaN values for some variable should be thrown out? If the answer is to just let the user decide then discussion of this question and alternative postprocessing functions with their implications is definitely warranted in the README.

Right, I just stumbled over this while reviewing https://github.com/socialfoundations/folktables/pull/41
If I had to guess what happened, I'd say that the way we called nan_to_num was unintended. It has the effect that NaNs were replaced with 0 and not with -1.
I'll have to think more about what's the best solution for dealing with nans. Given that this package has been used for quite some time, my inclination might be to continue to replace NaNs with 0 to ensure replicability of various results that might depend on it.

Personally I think it'd make sense to keep the previous unintended behavior (replace with 0s instead of -1s) to maintain reproducibility of results 😅
One potential issue with replacing NaNs with 0s is that 0 is a valid value that is present in multiple columns.
I looked into it and only ACSHealthInsurance seems to have this problem in a significant part of the dataset (17% NaNs and 10% 0s are mapped into the same 0 representation). Other tasks have plenty of NaNs but no 0s.
See the last cell of this notebook for a more detailed investigation of which tasks and columns are affected.
Since only one task seems to be affected by this I suggest keeping the previous behavior and adding a warning in the package repository. Also, the implicit NaN-to-0 mapping should perhaps be made explicit with nan_to_num(x, copy=True, nan=0.0) instead of hiding behind the default value of nan_to_num.

FWIW the teachability of this is awesome. If we could find some results that changed due to the treatment of the nans that would be so great for a teaching lesson eg on data processing decisions affecting results.

In my previous comment I provide an example of such a case! @ericvd-ucb

When you change the postprocessing of ACSEmployment to nan=-1 the equality of opportunity violation reported in the first README example for Alabama changes from 0.0871 to 0.0789

I just checked for the Texas example in the README too, and it also changes: from the reported 0.0397 to 0.03419


@vitejs/plugin-legacy support only output legacy bundle

Description
In vue-cli v5, vue-cli-service build --no-module will only output the legacy bundles that support all target browsers (loaded via plain <script>s). See Vue CLI --no-module
But @vitejs/plugin-legacy will output both modern and legacy codes.
Suggested solution
Can we add --no-module config in vite-plugin-legacy?
Alternative
No response
Additional context
No response
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

Hi @xiaoxiangmoe , could you try this feature? I used
legacy({ renderModernChunks: false }) then I will get the entry point is index-legacy-123456.js in manifest.json, but when I used it, I will encounter the issue System is not defined, do you know this issue?

@sangquangnguyen would you create a new issue with a minimal reproduction linking to the issue and PR? Thanks!

Ah, sorry for bothering you 😅 , I just want to have a quick check whether the author of this issue encountered like me. I might have some wrong configs, let me do more investigations and create a ticket with the details if needed, thanks @patak-dev for your time ❤


EventTimeLine: Expose a matrix filter to use for pagination

so that a SDK user can select which kind of events they want from paginations requests.
I think we already have such internal filter to manage lazy-loading. If yes, we just need to expose it and make it customisable.

Kamino cloned this issue to vector-im/riot-ios


Envelope

Added Sphere Envelope and Cube Envelope classes
Added test cases for newly added classes
Updated build.sbt to address test cases issues in IntelliJ

Great, I'm checking the new code!
You do not need to close and re-open a PR each time you add a commit. Once a PR is opened, as long as it is not merged, you can add as many commits as you want (the CI will be ran + code coverage).

Codecov Report

Merging #22 into envelopeMayur will decrease coverage by 0.48%.
The diff coverage is 97.01%.

@@                Coverage Diff                @@
##           envelopeMayur      #22      +/-   ##
=================================================
- Coverage          97.63%   97.15%   -0.49%
=================================================
  Files                 10       12       +2
  Lines                127      387     +260
  Branches              32      126      +94
=================================================
+ Hits                 124      376     +252
- Misses                 3       11       +8

Impacted Files
Coverage Δ

...in/scala/com/spark3d/geometryObjects/Point3D.scala
100% <100%> (ø)
:arrow_up:

...ain/scala/com/spark3d/geometry/ShellEnvelope.scala
96.55% <96.55%> (ø)

...main/scala/com/spark3d/geometry/CubeEnvelope.scala
96.98% <96.98%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 25c7fd6...c215c76. Read the comment docs.

yes, I didn't have a permission/couldn't find an option to add commit to a pull request. Can you please point that out, because this will have to be done over and over again. Thanks :)

For example, your last commit appeared in your last PR before you closed it.

no, I guess. I think it is because I am working on the forked repo, for
a branch in the original repo, simply making a commit works for sure, I
have done that before. I'll try making a dummy commit to this branch and
verify if it works.
On Mon, May 28, 2018 at 10:52 AM Julien<EMAIL_ADDRESS>wrote:

Once the PR is opened, just adding new commits from your branch as usual
is enough (git commit -m "..."). They will automatically appear here (under
the last message of this thread at the moment of commit). Is that not
working for you?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/JulienPeloton/spark3D/pull/22#issuecomment-392422135,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMMPXtOHfaW52HaZO3pL2Mhv9h1nSjqBks5t24mOgaJpZM4UPo3p
.

Oh yeah, my bad I was looking at commits from top to the bottom :p, that's
why was under the impression that PR is not getting updated. Thanks :)
On Mon, May 28, 2018 at 10:58 AM Mayur Bhosale<EMAIL_ADDRESS>wrote:

no, I guess. I think it is because I am working on the forked repo, for
a branch in the original repo, simply making a commit works for sure, I
have done that before. I'll try making a dummy commit to this branch and
verify if it works.
On Mon, May 28, 2018 at 10:52 AM Julien<EMAIL_ADDRESS>wrote:

Once the PR is opened, just adding new commits from your branch as usual
is enough (git commit -m "..."). They will automatically appear here (under
the last message of this thread at the moment of commit). Is that not
working for you?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/JulienPeloton/spark3D/pull/22#issuecomment-392422135,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMMPXtOHfaW52HaZO3pL2Mhv9h1nSjqBks5t24mOgaJpZM4UPo3p
.

He he great! It had to work - otherwise it would make collaborative work a nightmare for collaborators :-)
Anyway, the best usually is to open quickly a PR when you make a change, and add commit/push little-by-little such that the code review can be done progressively (instead of having one single large PR).
I'm reviewing the code. That's amazing you almost hit 97% coverage ;-)

CubeEnvelope here is CuboidEnvelope actually. I think we should rename the class accordingly?

By cuboid, you mean something which has 90 deg angles, but sides can have different length?

If yes, maybe "BoxEnvelope" sounds more familiar than "CuboidEnvelope" (which is valid word I reckon - just less used).

yes. I have defined cube's endpoints of the sides in the Envelope by a min and max coordinates along all three axes. Considering the diff between max and min in all three directions will be diff, the shape of Envelope would turn out to be Cuboid rather than Cube

sure, will rename accordingly :)

Just rename the class CubeEnvelope.scala to something like BoxEnvelope.scala, and then I will do the merging.

@JulienPeloton I was going through the onion partitioning code, and while placing an object inside the grid, we make a check like isPointInShell(lower_sphere, upper_sphere, center)) to get the containing spheres of the object. So if we want to use this code there, should I had helper methods in ShellEnvelope, i.e. something like,
object ShellEnvelope {
    def isPointInShell(lower_sphere: Sphere, upper_sphere: Sphere, center: Point3D)) {
     }
}

and also refactor the ShellEnvelopeCode to get defined by innner and outer Sphere instead of inner and outer radius?
class ShellEnvelope(
    var center: Point3D,
    var innerSphere: Sphere,
    var outerSphere: Sphere)
  extends Envelope {
}

Hum, I think I prefer to keep your code with innerRadius and outerRadius, and rather add a method isPointInShell in ShellEnvelope taking radii as inputs. The reason is that it's lighter to work with double (radius) rather than Sphere objects, and Sphere was a quick class that I add while waiting for your PR (it contains very few methods compared to your ShellEnvelope class).  It also means that I will use your class ShellEnvelope all over the place in the OnionPartitioning instead of Sphere.
I will do the modifications once your PR merged.

Great, thanks for the great PR!


Translate OTEL log's InstrumentationScope as logger name

InstrumentationScope usually contains the name of the instrumentation library. However, in the case of logs, it may also contain the name of the logger if the log source defines it [1]. An instrumentation library usually has a version set, so assume that the name without the version is the name of the logger.
[1] https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/api.md#get-a-logger

Signed the agreement


[21.05] hedgedoc: 1.8.2 -> 1.9.0, fixes CVE-2021-39175

Motivation for this change
Backport of #138468.
ChangeLog: https://github.com/hedgedoc/hedgedoc/releases/tag/1.9.0
As documented in the Nix expression, I unfortunately had to patch
yarn.lock manually (the yarn.nix result isn't affected by this). By
adding a git+https-prefix to
midi "https://github.com/paulrosen/MIDI.js.git#abcjs" in the lock-file
I ensured that yarn actually uses the MIDI.js from the offline-cache
from yarn2nix rather than trying to download a tarball from GitHub.
Also, this release contains a fix for CVE-2021-39175 which doesn't seem
to be backported to 1.8. To quote NVD[1]:

In versions prior to 1.9.0, an unauthenticated attacker can inject
arbitrary JavaScript into the speaker-notes of the slide-mode feature
by embedding an iframe hosting the malicious code into the slides or by
embedding the HedgeDoc instance into another page.

Even though it "only" has a medium rating by NVD (6.1), this seems
rather problematic to me (also, GitHub rates this as "High"), so it's
actually a candidate for a backport.
[1] https://nvd.nist.gov/vuln/detail/CVE-2021-39175
(cherry picked from commit 0a10c17c8d01e5f9fefa3d6dbb7802a3cbce7e23)
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
21.11 Release Notes (or backporting 21.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 139238 at 4eb0a2b3 run on aarch64-linux 1

1 package failed to build:

hedgedoc

1 package built successfully:

nixos-install-tools

Note that build failures may predate this PR, and could be nondeterministic or hardware dependent.
Please exercise your independent judgement. Does something look off? Please file an issue or reach out on IRC.

Meh. Gonna regenerate the lockfiles tomorrow.

@ofborg build hedgedoc
Rebuilding now. I could reproduce the hash-mismatch, but it's gone now.

Now this is weird. Can anybody please check the result of nix-build /nix/store/zr0dr0cczfgbdyl3y08mipv90wz8xl52-js-sequence-diagrams-bda0e49.drv --check here?

Yeah, as mentioned above, I'm aware of that (also, ofborg already failed with the same issue) :)
Will try to investigate what's wrong at some point.

OK, the overall cause is that the store-path with the new hash (i.e. the store-path that doesn't mismatch) is missing a submodule while the problematic store-path doesn't.
However, both stem from the same derivation (/nix/store/zr0dr0cczfgbdyl3y08mipv90wz8xl52-js-sequence-diagrams-bda0e49.drv). Not sure how that's possible so far, but I'll see.

OK, found the problem (and with this explanation, master is also affected, so I'll fix that first):

the first part is rather trivial: yarn2nix runs nix-prefetch-git without --fetch-submodules, but fetchgit fetches submodule sby default (I think that this behavior is really problematic btw). The easiest fix is to change the nix-prefetch-git of yarn2nix. COnsidering that noone else had this before, this just wasn't noticed until now.
It took me a while until I realized that I still had some output path with the wrong hash lying around which silenced the error. HOWEVER, nix-build --check didn't raise an error even though it should which seems like a Nix bug to me (in fact, I would've noticed way earlier otherwise), will file a patch for that later.

This now contains the changes from #139489 as well, so we should wait until this is also merged :)

I could reproduce the hash-mismatch, but it's gone now.

That sentence led me to believe the issue had been fixed at that point.

Oh I only gave an update that it wasn't gone in Matrix, but not here, sorry for that! :)


Inquiry about GPU

Thanks for your great work！
What is the memory size of your graphics card? Can I use a 3090 for training?

You can change the batch size to 6, and fit the memory size on 3090.


Why nest theme include nth lines?

Noty version: 3.2.0-beta
Browser: Chrome
OS: Windows 10

I want to use nest theme but it's position is realy bad because of nth-child usage. How can i disable it without editing original file?

related: #450
It's the nest theme's feature. You can create your own theme and use.


Flexible Legal Basis = True Default

❗ ~Contains migration; check downrev~
❗ ~Dependent on fideslang release of https://github.com/ethyca/fideslang/pull/184~
Closes #PROD-1264
Description Of Changes
Make "flexible legal basis for processing" non nullable and True by default in the backend.
Code Changes

Update PrivacyDeclaration.flexible_legal_basis_for_processing to get rid of the third state. Make it required, and set the server default to True, to match what should happen when systems are created outside of Compass.
Add a migration that first updates existing flexible_legal_basis_for_processing to False if it is currently null, then adds the new server default and makes it required.

Steps to Confirm

[ ] Creating a system without this field specified via the API (POST System) will set this value to True by default in the backend.

Pre-Merge Checklist

[ ] All CI Pipelines Succeeded
Documentation:

[ ] documentation complete, PR opened in fidesdocs
[ ] documentation issue created in fidesdocs

[ ] Issue Requirements are Met
[ ] Relevant Follow-Up Issues Created
[ ] Update CHANGELOG.md
[ ] For API changes, the Postman collection has been updated

@adamsachs I think this is ready to go, just need a 👍

@adamsachs I think this is ready to go, just need a 👍

Looks good - does changelog need to be updated?

thank you for the changelog catch, I thought I'd done it, but just in fideslang!


Use SO_EXCLUSIVEADDRUSE instead of SO_REUSEADDR on Windows

On Linux, SO_REUSEADDR is used so that a socket in TIME-WAIT state can be rebound after a listening process is restarted. It does not allow two processes to listen on the exact same (addr, port) combination. However, on Windows, it does, and SO_EXCLUSIVEADDRUSE is required to reproduce the Linux behavior.
Reticulum relies on an error being returned by bind() that reuses the same (addr, port) combination as another process to detect whether there is a shared instance already running. Setting SO_EXCLUSIVEADDRUSE makes this detection process work on Windows as well.

The combination of SO_REUSEADDR and SO_EXCLUSIVEADDRUSE on Windows has WSABind return a WSAEINVAL error. Ideally, Reticulum would probe for a running shared instance by attempting to connect to it rather than attempting to bind, but this PR is a much simpler change that fixes the immediate issue of not being able to use most of the functionality of RNS with RNodes (which relies on being able to use a shared instance).

Thank you very much for this contribution!


Add University

Official URL: https://newsummit.edu.np/
Long Term IT Course: https://newsummit.edu.np/course/bachelor-of-science-in-computer-science-information-technology
Image of University using the domain:
https://user-images.githubusercontent.com/12593145/210191884-2ecf7302-b896-4c12-8d1c-37459208ea53.png

@orphikkkk First stage of verification passed, official domain is newsummit.edu.np as far as I verified. This is just an informational note. Pull request is still in the review. I appreciate your patience while waiting for the final decision.


[State] Burrow panics without global permissions address

Please include in your bug report:

burrow version: master
go version: 1.10
what you expected to happen: Burrow to handle the global permissions address not existing
what actually happened: Burrow panicked without the global permission step
steps to reproduction: Run the tests in this PR: https://github.com/hyperledger/sawtooth-seth/pull/73

The panic in question:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x9b9cd1]

goroutine 20 [running]:
github.com/hyperledger/burrow/acm/state.GlobalAccountPermissions(0x7f9fbdc30230, 0xc4201b81b8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/project/sawtooth-seth/processor/src/github.com/hyperledger/burrow/acm/state/state.go:122 +0x91
github.com/hyperledger/burrow/execution/evm.HasPermission(0xd8c260, 0xc4201b81b8, 0xd8d160, 0xc4201b81d0, 0x8, 0xc4201b81c8)
	/project/sawtooth-seth/processor/src/github.com/hyperledger/burrow/execution/evm/vm.go:108 +0xee
seth_tp/handler.CreateContractAccount(0xc4201c1590, 0xc4201cd400, 0xc4201b81b8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/project/sawtooth-seth/processor/src/seth_tp/handler/transaction_handlers.go:245 +0x3f3
seth_tp/handler.(*BurrowEVMHandler).Apply(0x13980b0, 0xc4201e4ae0, 0xc4201bff00, 0xc4201c76c0, 0xd8a4e0)
	/project/sawtooth-seth/processor/src/seth_tp/handler/handler.go:110 +0x3ef
github.com/hyperledger/sawtooth-sdk-go/processor.worker(0xc4201beb80, 0xbdda48, 0x10, 0xc4201e4720, 0xc4202001c0, 0xc4201b6660, 0x1, 0x1)
	/project/sawtooth-seth/processor/src/github.com/hyperledger/sawtooth-sdk-go/processor/worker.go:67 +0x340
created by github.com/hyperledger/sawtooth-sdk-go/processor.(*TransactionProcessor).start
	/project/sawtooth-seth/processor/src/github.com/hyperledger/sawtooth-sdk-go/processor/processor.go:140 +0x4b0

The unit tests for Seth don't set up the global permissions account, which breaks Burrow. The version we were using (0.17) worked without it set up. The unit tests work with this diff applied:
115c115,123
< 	return GlobalPermissionsAccount(getter).Permissions()
---
>
> 	global := GlobalPermissionsAccount(getter)
> 	if global == nil {
> 		return permission.AccountPermissions{
> 			Roles: []string{},
> 		}
> 	}
>
> 	return global.Permissions()

Would it be possible to have this or a similar fix applied to Burrow to allow running without the global permissions account set up?

I think this fix would have some unintended consequences since the permissions codes expects the fallback permission account to have all bits in the mask 'set'. We should fallback to DefaultAccountPermissions to which the global account is initialised. However this would also need to be handled in the snatives.
I think it might be better to ensure that the global permissions account is always defined. I presume difficulty is you may need hack on your account/address model to get the zero address defined? We may want to handle the global permissions a distinguished piece of state rather than treating it as a pseudo account just because that sort of works....

Yeah, the difficulty arises in that if we attempt to set up the global permissions address for the unit tests, evm.HasPermission gets called and panics. Not sure how to hack our way around that unless burrow switches to returning an error instead of panicking.

Sorry about the delay on getting back on this I've been away a lot this month.
In your previous comment are you saying that in the process of setting the global permission address you are hitting the code that panics? Could you link some code? I would have thought you can set up some initial accounts without that being an issue.
Here's where we do that in Burrow: https://github.com/hyperledger/burrow/blob/develop/execution/state.go#L146-L151

I'm afraid we weren't able to get to the bottom of why Burrow ought to function without a global permissions address defined. Running Burrow without global permissions address is effectively running it partially initialised. I am open to better designs, but haven't been able to reach you @knkski about this so I'm closing for now.


Protobuf issue and API number is not compatible between Unity and python

If I put that folder in ML Agents folder I tried installing ml agents requirements and without installing the ml agents requirements. But it dont work
If I put it out of ml agents folder, it dont work either.

The API number is not compatible between Unity and python. Python API : API-4, Unity API : API-8.

I am using latest ml agents 0.8.1
unityagents and MLagents supports different versions for Protobuf. How do we solve this issue.

This repo was based on ml-agents 0.4. Unfortunately, I can't guarantee its compatibility with newer versions.


change to use objects instead of strings

Hi there! I saw this and love it. Naturally I wanted to see whether it works with glamorous and found that it almost does except it returns strings rather than objects. Turns out that styled-components can accept objects as well as strings! This is what polished does and you can use polished with styled-components or any other CSS-in-JS library.
Here's a start to making it work with objects. It's a work in progress. I've just updated the fontSize util. I've published these changes as @kentcdodds/styled-system and verified that it works with both glamorous and styled-components here.
If you're ok with this kind of change, I will be happy to update the pull request to apply to the other utils 👍

Generally I'm all 👍 on this.
But, wait, what!? does styled-components support passing plain JS objects in?? I don't know where I missed that part of the docs...

Sweet! I'll finish the PR then!

I figured out how the relab stuff worked 😅 and updated the examples for the new structure :+1:

does styled-components support passing plain JS objects in?? I don't know where I missed that part of the docs...

Don't worry you didn't miss anything, it's not documented because we don't want people to use styled-components that way 😆  we added it to make polished work for all CSS in JS libraries, not just return strings and only support styled-components. :heart:

it's not documented because we don't want people to use styled-components that way 😆

Maybe I shouldn't have tweeted about it then 😅  Or I should have been more clear that you don't want to do that. (though, I don't know why anyone would want to!)

@mxstbr is there any perf dent or disadvantage to using that then? I feel like the docs mention something about inline styles but didn't realize the stylis part could handle that?

Alright, I'm done with my snarky comments, @kentcdodds - this looks fantastic. Will give it another review in the morning before merging. Thanks! 😎

Feel free to change whatever. I probably won't get to it until later.

is there any perf dent or disadvantage to using that then?

Maybe? I haven't benchmarked it or anything but essentially we have to parse twice, once objects -> strings and once strings -> final CSS. Unsure how slow that first part is though, honestly 🤷♂

Great job @kentcdodds, looking forward to seeing this released! Just a quick note though:
When using props like fontSize or mt the generated styles raise the Warning: Unsupported style property margin-top. Did you mean marginTop? warning. It still works but you get that error in the console. I think you need to change everything to pascalCase as styled-components will parse that anyway. (tests will break though!)

Nice one! And as a bonus point, it also works with Fela:
https://codesandbox.io/s/k5roZ2Krx

@braposo I'm unable to reproduce locally, do you have an example anywhere?

@kentcdodds this is great - going to merge it in. thanks again for all your help! 💯

Thank you! Can't wait for the release! I'll add it to the glamorous integrations page! 🎉

@jxnblk it only raises the warning if you're using it with glamorous I think. @kentcdodds example in here has that problem, just open the console and you'll see the warning for font-size.

I've created another sandbox where that's more evident and it uses the latest version of styled-system. I'm happy to create a PR if you want.

@jxnblk I just went ahead and created a PR for this: https://github.com/jxnblk/styled-system/pull/3


292 feature add cli arguments for twitter benchmark

:tada: This PR is included in version 0.7.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Layers

In a typical game we don't want to have everything to collide with everything. This is especially true when using sensors.
Use cases

I want to create a "hurt-box" (sensor) that only collides only with "hit-boxes" (sensors).
I want to create a "finish" sensor that triggers a "victory" when the player enter the zone, but not when an ennemy enther it.
I want my character/ennemy to not be blocked by walls under some conditions.

Using physics layer to simplify/optimize game logic is a standard, useful and powerful technique in game-development.


Refactor Status class for hopefully increased legibility and reasoning

I tested this PR by pointing Argo at this branch, deploying Argo, and running the bulk metadata update integration test which we believe should test the code path changed here.

you may want to regen .rubocop_todo.yml as I expect some of the complexity numbers went down.

@ndushay 💬

you may want to regen .rubocop_todo.yml as I expect some of the complexity numbers went down.

Good call. I included that in the version bump commit I just pushed.


Baggage is lost when explicitly setting parent observation

Using: id("org.springframework.boot") version "3.3.5"
I would expect baggage to be carried over when setting a parent observation. Currently it is lost. I am using the otel tracing bridge.
Example:
    @GetMapping("observation")
    public Mono<String> observation() {
        String baggageBefore = tracer.getBaggage("key").get();
        Observation parent = observationRegistry.getCurrentObservation();
        Observation o = Observation.createNotStarted("child", observationRegistry).parentObservation(parent);
        try(Observation.Scope ignored = o.openScope()) {
            String baggageAfter = tracer.getBaggage("key").get(); // expect "val" but is null
            return Mono.just("baggagebefore="+baggageBefore+", baggageAfter="+baggageAfter);
        }
    }

Full minimal reproduction in this repo: https://github.com/jamesmoessis/baggage-bug-demo
Reproduce by running it then:
$ curl http://localhost:8080/observation -H 'baggage: key=val'
baggagebefore=val, baggageAfter=null%

Typically I would expect creating the child observation in the opened scope from the parent observation, which would allow this to work. Is there a reason in your code you can't do that?

@shakuzen thanks for clarifying. Given that https://github.com/micrometer-metrics/tracing/issues/455 was only recently closed, we can assume this feature is supported now. I assume it wasn't supported when I raised this issue initially.

Is there a reason in your code you can't do that?

I suppose there's many scenarios in which a context is not in current threadlocal scope, but yet you still want to use it as a parent context, but it's kind of beside the point. It's unclear why I would need parent context to be the current active context in order for this to work. It doesn't work like that for trace context, and as far as I understand this is not documented behaviour.
I oversee the usage of this library at our company for hundreds of service, and people use it in all kinds of ways. They come to me with their issues using this library. If the API supports it, it should work - and oftentimes the API just does not work and I come here to raise an issue. It's simply too brittle to use at this scale, and this is another example. It's a shame because I would like to recommend a single observability API that integrates with Spring well, but unfortunately it's too buggy so I just point people towards the OTel API which does what it says on the tin.

I oversee the usage of this library at our company for hundreds of service, and people use it in all kinds of ways. They come to me with their issues using this library. If the API supports it, it should work - and oftentimes the API just does not work and I come here to raise an issue. It's simply too brittle to use at this scale, and this is another example. It's a shame because I would like to recommend a single observability API that integrates with Spring well, but unfortunately it's too buggy so I just point people towards the OTel API which does what it says on the tin.

We're sorry that you find the API too brittle. You're providing the baggage being lost when using explicitly as one example and I can understand that. Are there any others that you can mention? We don't see too many issue tickets related to the API and bugs so that's quite some news to us.

Also with reactor we explain how to use the baggage over here https://docs.micrometer.io/tracing/reference/configuring.html#_context_propagation_with_micrometer_tracing


How to write deep image

Hi,
is there a way to write a deepscanline instead of scanlineImage ?
Cheers

No, unfortunately this is not supported.


Issue with transit agency description

Phone number and URL descriptions are in the wrong order

@lorenzopicoli Please take a moment to fill out this short survey. Thank you!
This is an automated message, feel free to ignore.

@lorenzopicoli Thank you for opening this issue. 🙏
Please check out these other resources that might be applicable:

Support Console - If you have a support contract.
Discord - chat with other developers
StackOverflow - use the google-maps tag
Google Maps Platform issue trackers

This is an automated message, feel free to ignore.


HGR2 malfunctions.

Consider this program:
10 HGR2 : HCOLOR= 5
20 FOR I = 1 TO 199 STEP 3
30 HPLOT 100,100 TO I,10
40 NEXT

When run without the card installed, it works as expected and remains in memory.
When run with the card installed, the HGR (not HGR2) screen is shown and nothing happens.
Blindly typing TEXT and then LIST reveals the program is gone, as if one had typed NEW.
Apple IIe, enhanced, with 64k 80 col card.
VGA card in slot 3, happens with 80 col firm ware on or off.

Just checked and it does NOT happen if the card is in slot 5.
I had thought slot 3 was OK, only limiting access to upload fonts and config settings - neither of which seem to work no matter what slot I am using.

On the IIe you shouldn't put any card in slot 3 (even this one) with very few exceptions. Can you also inspect the level shifters for shorts? There should not be any problem with soft switches being detected correctly. It may also be worth trying different firmware versions to see if the problem is specific to a certain version.  There can be slight timing differences between systems and even batches of the level shifters on the board. I've made adjustments to try and center up the bus sampling to the stable portions and maximize the time the RP2040 has to respond to reads from it's address space.  If the config utilities are not working, make sure you are using the latest release package from this project.  The google drive folder isn't updated as often and mainly has historical revisions.

Well, the issue does not occur if the card is in slot 4.
While I can poke as prescribed and get color changes, fonts don't seem to work via the config utils. I get garbled text like it switches to a font with no lower case letters (Apple ][ style) until I power cycle the computer.
Can you explicitly direct me to firmware and utils to try?

https://github.com/V2RetroComputing/analog/releases/tag/20230504 is the latest build. it is currently marked as a pre-release since it contains some experimental features regarding color rendering. there is a two-step process when coming from previous builds since the fonts are kept separate to avoid overwriting custom fonts when upgrading to future releases.  flash the preloader first, then the actual firmware.  The 4ns builds are recommended unless you run into issues with the Pico not booting.

Understood, I'll attempt a test in the next few days and report on my findings.


XbOne BT fix

Xbox One Wireless controllers in some circumstances fail the Windows XInput filter.
The old firmware presents a good state that will fall through to recieve a map via SDL db, but new firmware (in a bad state) presents no trigger ouput and a non-normative mapping.
This patch adds accommodations to make the best of new firmware when it's in a bad state. This is an XInput related issue and affects Windows only.

Idle thought: I feel like we should have some way of surfacing gamepad issues, either because a gamepad has been blacklisted or because we have some sort of hack that implements half-support. No idea what that would look like.


Refactor metadata retry logic and improve logging

This doesn't change the behaviour, but it makes the implementation easier to follow. I also improved logging.
@Shopify/kafka

Rebased, and address all comments.

LGTM, but it's another thing that will need stress testing before 1.4.0 is released

Merging this.


[1.10.2] Immersive Engineering silos

Issue description:
When a silo is connected to the system via "External Storage" only 65 items are shown in the grid and you can't take more than 4 at a time. Second bug is that if you connect a second silo to the system, the items from both silos are shown, but you can only take items from one of the silos (again, not more than 4 items at a time)
Steps to reproduce:

Place a controller (can be creative too) and a grid.
Create two Immersive Engineering silos and fill them with more than 65 of whatever item.
Connect one of the silos with an "External Storage" and check what's shown in the grid, try to take a stack.
Connect the second silo and again check the grid and try to take items from both silos, you won't be able to take anything from one of them.

Version (Make sure you are on the latest version before reporting):

Minecraft: 1.10.2
Forge: <IP_ADDRESS>5
Refined Storage: 1.2.21
Server[yes/no]: No

Part of this is due to bad code in the silo's IItemhandler (I've fixed most (all?) of that, I'll push it soon), part is probably due to expectations of RS's caching system and part is just due to how things work:
The silo has 2 slots in code, one for input and one for output. The first slot is always empty (Stack of 1 in the current version, that is causing the 65). The second slot is for output, it contains up to a full stack of the stored item (all stored items if there is less than one stack). This causes external systems to see no more than one stack of the stored item.
The limited output is our (IE's) fault, I fixed that.
Connecting more than one silo to the network: This is probably on RS's side, may be related to some sort of caching. I've seen the same thing happen with a single silo: Put in 32 something into the silo using RS (shows 64 in RS), put in another 64 (shows 96, should be 64 since the silo shows no more than 64).  The system will not show the remaining 32 something. Breaking the connection from the controller to the ext. storage (not the ext. storage itself) fixes this.
I glanced over the code for external storage and I don't see a way to make all stored items visible from IE's side without either making gigantic ItemStack's or adding a lot of slots. The silo currently isn't accessible using the API, but we can make it accessible if you need that (you probably will if you add support for it).

Personally I don't want all the items shown.  I actually quite like that my silo only appears as one stack in RS.  So long as I can pull a stack out at a time that's good.
My reasoning is if I have both RS and IE, then RS Silo's are for the 'junk' items like cobble or dirt.  So long as I can get it a stack at a time I don't need to know how much there is because the answer is "Too Much" and if I really care I can go look at the Silo, which feels much more in keeping with IE.
Obviously others may feel differently, but I'd leave this one as 'won't fix' if it was me.

I glanced over the code for external storage and I don't see a way to
make all stored items visible from IE's side...

StorageDrawers does it somehow.  You might get inspiration from that
code, or maybe you've already looked and don't like the way it's done,
in which case pretend I said nothing.

No, StorageDrawers doesn't do it. RS has explicit compatibility as far as I can see.

@BluSunrize maybe you might have a solution to this?

Well Malte already replied, didn't he?
He addressed compat as far as was possible. Unless RS adds specific compat, that's how far we get.

Same goes to Thermal Expansion caches in 1.12

Unfortunately, there is no easy way to fix this. Just the nature of the item handler.


422 Unprocessable Entity during Github release

I get an exception when releasing a project using JReleaser Maven Plugin (running in Github Actions) configured like so:
  <plugin>
    <groupId>org.jreleaser</groupId>
    <artifactId>jreleaser-maven-plugin</artifactId>
    <version>1.6.0</version>
    <inherited>false</inherited>
    <configuration>
      <jreleaser>
        <signing>
          <active>ALWAYS</active>
          <armored>true</armored>
        </signing>
        <release>
          <github>
            <draft>true</draft>
            <overwrite>true</overwrite>
            <sign>true</sign>
            <issues>
              <enabled>true</enabled>
            </issues>
            <changelog>
              <formatted>ALWAYS</formatted>
              <preset>conventional-commits</preset>
              <contentTemplate>.github/workflows/release-changelog-template.md</contentTemplate>
              <contributors>
                <format>- {{contributorName}}{{#contributorUsernameAsLink}} ({{.}}){{/contributorUsernameAsLink}}</format>
              </contributors>
            </changelog>
          </github>
        </release>
        <deploy>
          <maven>
            <nexus2>
              <maven-central>
                <active>ALWAYS</active>
                <!-- TODO Remove verifyPom tag, hack for https://github.com/jreleaser/jreleaser/issues/1397 -->
                <verifyPom>false</verifyPom>
                <url>https://s01.oss.sonatype.org/service/local</url>
                <closeRepository>true</closeRepository>
                <releaseRepository>false</releaseRepository>
                <stagingRepositories>target/staging-deploy</stagingRepositories>
                <artifactOverrides>
                  <artifactOverride>
                    <groupId>ai.timefold.solver</groupId>
                    <artifactId>timefold-solver-core</artifactId>
                    <sourceJar>false</sourceJar>
                    <javadocJar>false</javadocJar>
                  </artifactOverride>
                  <artifactOverride>
                    <groupId>ai.timefold.solver</groupId>
                    <artifactId>timefold-solver-spring-boot-starter</artifactId>
                    <sourceJar>false</sourceJar>
                    <javadocJar>false</javadocJar>
                  </artifactOverride>
                  <artifactOverride>
                    <groupId>ai.timefold.solver</groupId>
                    <artifactId>timefold-solver-ide-config</artifactId>
                    <sourceJar>true</sourceJar>
                    <javadocJar>false</javadocJar>
                  </artifactOverride>
                  <artifactOverride>
                    <groupId>ai.timefold.solver</groupId>
                    <artifactId>timefold-solver-webui</artifactId>
                    <sourceJar>true</sourceJar>
                    <javadocJar>false</javadocJar>
                  </artifactOverride>
                </artifactOverrides>
              </maven-central>
            </nexus2>
          </maven>
        </deploy>
      </jreleaser>
    </configuration>
  </plugin>

This is the last thing I get in the trace log:
[DEBUG] fetching all branches on TimefoldAI/timefold-solver
[DEBUG] looking up release with tag v0.10.0 at repository TimefoldAI/timefold-solver
[DEBUG] fetching release on TimefoldAI/timefold-solver with tag v0.10.0
[DEBUG] release v0.10.0 does not exist
[DEBUG] creating release v0.10.0
[DEBUG] tagging local repository with v0.10.0
[DEBUG] creating release on TimefoldAI/timefold-solver with tag v0.10.0
422 Unprocessable Entity
org.jreleaser.sdk.commons.RestAPIException: 422: Unprocessable Entity
at org.jreleaser.sdk.commons.ClientUtils.lambda$builder$1(ClientUtils.java:113)
    ...
Caused by: org.jreleaser.sdk.commons.RestAPIException: 422: Unprocessable Entity
at org.jreleaser.sdk.commons.ClientUtils.lambda$builder$1(ClientUtils.java:113)
at feign.ResponseHandler.decodeError(ResponseHandler.java:136)
at feign.ResponseHandler.handleResponse(ResponseHandler.java:70)
at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:114)
at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:70)
at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:96)
at jdk.proxy3/jdk.proxy3.$Proxy31.createRelease(Unknown Source)
at org.jreleaser.sdk.github.Github.createRelease(Github.java:334)
at org.jreleaser.sdk.github.GithubReleaser.createRelease(GithubReleaser.java:352)
at org.jreleaser.sdk.github.GithubReleaser.createRelease(GithubReleaser.java:235)
... 35 more

This is the branch of our project that exhibits the issue:
https://github.com/TimefoldAI/timefold-solver/tree/triceo-releasetest
This was the run that failed:
https://github.com/TimefoldAI/timefold-solver/actions/runs/5332409307

Could it be that, while this new release is v0.10.0, the previous release was 0.9.38 (without the v), and therefore nothing could be found?

I believe I’ve seen this 422 error before at least once. I think it’s not related to a change in tag pattern as that affects changelog generation, not release publication.

I have just confirmed it is related to the tag.
When I specified previousTagName, the issue went away.
(Or it went away for a different environmental reason.)

Alright, thanks for verifying. I suppose there's a recent change that requires the tag pattern also during release 😅

FWIW 422 is returned when validation fails or the endpoint has been spammed -> https://docs.github.com/en/rest/releases/releases?apiVersion=2022-11-28#create-a-release
The error occurs after setting initial parameters to create a release https://github.com/jreleaser/jreleaser/blob/c64c90f7bdacc9918f647198cf33ce5e97f6dbb8/sdks/jreleaser-github-java-sdk/src/main/java/org/jreleaser/sdk/github/GithubReleaser.java#L333-L346
Do you have access to the trace.log file when the release failed? Would like to find out what were the resolved values for the release section.

trace.log
I've attached the trace log from that run.

I'm going on a limb here but I think setting the release to draft may be the cause. I have to test this theory. We used to rely on hub4j (external dependency) for interacting with the GitHub REST API. Recently we moved away from it and provided our own implementation. It's possible we missed something during that migration.

Looks like one probable cause for this error is performing a release targeting a branch that does not yet exist on the remote.
JReleaser must ensure that the target branch exists before creating a release.

🎉 This issue has been resolved in v1.16.0 (Release Notes)


feature/api

Change the API a little bit so that it is easier to use and understand.
Also remove some things that are not necessary for the user of the API.

Even though this is not the largest change I think it can be marked as "ready for review".


fix: update requests

:tada: This PR is included in version 3.0.1 :tada:
The release is available on:

v3.0.1
GitHub release

Your semantic-release bot :package::rocket:


Not able to create partial index on json column

It’s not possible to create index on key, on json column.
Say you have a json column, and want create index on only parts of this object. Throws error now when trying to save.
You can index the whole object (column), but there might be reasons (size for one) you don’t want everything indexed.

Could you provide an example index statement that is failing?
Both indexes on expression and partial indexes should be supported.

To define an index on json column expression you have to use the bult-in SQLite json functions, aka. something like:
CREATE INDEX `idx_example` on `example` (JSON_EXTRACT(`json_test_field`, "$.variant"))

Thanks!

The Create Index dialog looks different in the public build. The textfield where you can preview/edit your SQL that creates the index, is missing on my end.

I'm not able to reproduce it (both in Chrome and Firefox). Could you check the browser inspect tool if there is any JS error? If that's the case, feel free to let me know and I'll investigate it further.

Restarted the instance locally, and now it shows the textfield. Weird. I'll take note if I see this behaviour some time later on..


Remove unnecessary constructors

A constructor that redefines the super constructor but only calls the super constructor with the same parameters is unnecessary and can be removed
https://github.com/abaplint/abaplint/issues/1417

open an issue in abaplint? this is something that can be discovered automatically

I linked to it already above ;-)

thanks 👍


[RHOAIENG-5073] - Routing and Headless Service Support in KServe Raw …

…Mode Deployment
What this PR does / why we need it:
Raw Deployments need to be fronted by a routing component. Currently, the FMaaS/Rust router (and Caikit) client-side load balance and proxy requests across a model deployment's pods/replica's. To do so they utilizes a Headless service that sits in between itself and the replica's, queries addresses to the physical pods, and round robins requests.
Type of changes
Please delete options that are not relevant.

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

Release note:

NONE

/retest e2e-raw

/test e2e-raw


Enable uploading of code coverage data to https://app.codecov.io/

Signed-off-by: Alan Jowett<EMAIL_ADDRESS>Description
Run the existing tests and measure code coverage numbers.
Testing
CI/CD pipeline covers this.
Documentation
Example output:
https://codecov.io/gh/Alan-Jowett/ebpf-for-windows/tree/12c1054cd99ce3170c7bc1c4748fcb0a15fafbc2
Resolves: #760

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@ee098c0). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #862   +/-   ##
=======================================
  Coverage        ?   84.83%
=======================================
  Files           ?       78
  Lines           ?    12495
  Branches        ?        0
=======================================
  Hits            ?    10600
  Misses          ?     1895
  Partials        ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ee098c0...87c2737. Read the comment docs.

I would keep the changes in build.yml and drop the new YML. The change in time taken to complete the msbuild job doesnt look significantly longer.

Could we also add a coverage badge to the top of the readme by the other 2 badges there?
See https://shields.io/category/coverage


Fix normalization, VRAM usage, and new story feature

Removed normalization from save_image -> images appear correctly now and are much brighter
Every train_step now returns the generated image to be more flexible and it is reused in save_image
If an image and a text is used in the Image() instantiation, the average embedding of the two is created -> no need for beginners to play with clip encodings directly
Big: added new option create_story. This will separate a given input text into words and then move over the words in a sliding-window fashion to generate a story of the text - this allows the visualization of stories much longer than 77 chars (although there is no memory beyond the 77 chars) - save_progress should be turned on for this feature such that a GIF can be created in the end

Please give it a try, especially the create_story feature. If you turn on save_progress, nice movies can be made. I set it up such that it optimizes for 1 episode on three words, then for the next episode it adds 3 more words etc. (old words are kicked out if the CLIP context length is reached).
I'm generating some dream stories at the moment. I could update the README too and put a story in there, along with explanations of the new img feature.
Also, as the VRAM issue is now fixed I can (with my 8GB RTX 2060 Super) run a 44 layer net with a batch size of 96 on a resolution of 256. For a 512 resolution I have not yet found a good setup.

Looks great! Merging! :D


Removing all whitespaces from our Lua files

Used a dedicated script to remove all whitespaces in Lua files

Great improve Good job ! :)


Generate led color functions for the EV3 platform

Use the updated specification to generate functions managing EV3 leds.

Thanks!


[RFC] Precompute top taxons for heatmap

Description
This a request for comment on a change that would help short and long-term performance and scalability.
The heatmap (and report) page is fundamentally slow because there are too many taxons per sample. Generating a heatmap for 50 samples typical results in 1M rows being fetched from taxon_counts that are filtered, sorted, then filtered again down to the top 30 per sample and per count type, 3000 rows total.
The fully-indexed mysql query that does this typically executes in 5s, which seems pretty good for querying a 100M row table and processing 1M rows. However, 5+ secs is not a good experience when loading a heatmap for the first time, and we want the heatmap to be fast for more than 50 samples.
Given the above, the obvious optimization is to precompute the top N taxons per sample needed for a heatmap. Currently, the max N allowed in the UI is 100. So instead of taxon_counts we could query a table that is >100x smaller without any visible change.
Thus, my design proposal is:

Add the new table included in this PR
In results monitor, after all rows are inserted into taxon_counts, insert into top_taxon_counts, using an adapted version of the query in HeatmapHelper
In HeatmapHelper, query top_taxon_counts if rpm is the selected metric else do the old query
Profit

Notes

We could get performance gains for zscore as well by either

assuming that the top N taxons by zscore are in the top 100 taxons by rpm
precomputing zscores for the top backgrounds (similar to precaching)

Question: why would someone want to rank by reads instead of rpm?

With our analytics, we could assess how good the current "30 to 100" taxons limit is on the heatmap page

Partitioning taxon_counts is not a good solution because we are using an index already, not doing a full table scan

The perf problem of taxon_counts will only get worse over time as the table grows, and as the taxon database grows

The current TTI for the heatmap page as a whole is around 25s for 50 samples. I have another optimization planned separately that should take another 5s off that.

This table could also speed up the report page if we make the default view there show only the top 100 taxons.

Currently, after a heatmap loads once, it is cached in Redis so the slowness described above is mitigated

If we don't care about loading NR info with NT (or vice versa), we could cut the data needed in half again. The only place I see in the heatmap where the NR counts appear is in the hover tooltips, which could be loaded by AJAX on hover

We could add a foreign key constraint to top_taxon_counts on taxon_counts which would guarantee some consistency

Starting 1M rows for a typical 50 sample heatmap:

MySQL [idseq_prod]> select count(*) from taxon_counts where pipeline_run_id IN (14433, 14424, 14429, 14420, 14427, 14425, 14423, 14428, 14422, 14421, 14426, 14432, 14430, 14435, 14438, 14431, 14437, 14436, 14440, 14443, 14441, 14439, 14444, 14434, 14442, 14446, 14445, 14453, 14455, 14447, 14452, 14448, 14451, 14457, 14449, 14459, 14460, 14456, 14458, 14450, 14462, 14454, 14464, 14461, 14474, 14470, 14481, 14469, 14465, 14478, 14484, 14477, 14807)
    -> ;
+----------+
| count(*) |
+----------+
|   982917 |
+----------+
1 row in set (0.29 sec)

Top taxons query
SELECT *
  FROM (
    SELECT
      @rank := IF(@current_id = pipeline_run_id, @rank + 1, 1) AS rank,
      @current_id := pipeline_run_id AS current_id,
      a.*
    FROM (

      SELECT
        pr.sample_id,
        pipeline_run_id,
        taxon_counts.tax_id,
        taxon_counts.count_type,
        taxon_counts.tax_level,
        genus_taxid,
        family_taxid,
        taxon_counts.name,
        superkingdom_taxid,
        is_phage,
        count               AS  r,
        stdev,
        mean,
        count / (
            (total_reads - total_ercc_reads) *
            COALESCE(fraction_subsampled, 1.0)
          ) * 1000 * 1000 AS rpm

        ,
        COALESCE(
          GREATEST(-99, LEAST(99,
            (count / (
            (total_reads - total_ercc_reads) *
            COALESCE(fraction_subsampled, 1.0)
          ) * 1000 * 1000 - mean) / stdev
          )),
          100) AS zscore
      FROM taxon_counts
      LEFT OUTER JOIN pipeline_runs pr ON pipeline_run_id = pr.id
      LEFT OUTER JOIN taxon_summaries ON
        26   = taxon_summaries.background_id   AND
        taxon_counts.count_type = taxon_summaries.count_type      AND
        taxon_counts.tax_level  = taxon_summaries.tax_level       AND
        taxon_counts.tax_id     = taxon_summaries.tax_id
      WHERE
        pipeline_run_id IN (14231, 14433, 14424, 14429, 14420, 14427, 14425, 14423, 14428, 14422, 14421, 14426, 14432, 14430, 14435, 14438, 14431, 14437, 14436, 14440, 14443, 14441, 14439, 14444, 14434, 14442, 14446, 14445, 14453, 14455, 14447, 14452, 14448, 14451, 14457, 14449, 14459, 14460, 14456, 14458, 14450, 14462, 14454, 14464, 14461, 14474, 14470, 14481, 14469, 14465, 14478, 14484, 14477, 14807)
        AND genus_taxid != -201
        AND count >= 5
        -- We need both types of counts for threshold filters
        AND taxon_counts.count_type IN ('NT', 'NR')

         AND taxon_counts.tax_id > 0

    ) a
    ORDER BY
      pipeline_run_id,
      count_type = 'NT' DESC,
      rpm ASC
  ) b
  -- Overfetch by a factor of 4 to allow for a) both count types, and
  -- b) any post-SQL filtering
  WHERE rank <= 121;

EXPLAIN
+----+-------------+-----------------+--------+--------------------------------------------------------+---------------------------+---------+-----------------------------------------------------------------------------------------------------------+---------+------------------------------------+
| id | select_type | table           | type   | possible_keys                                          | key                       | key_len | ref                                                                                | rows    | Extra                              |
+----+-------------+-----------------+--------+--------------------------------------------------------+---------------------------+---------+-----------------------------------------------------------------------------------------------------------+---------+------------------------------------+
|  1 | PRIMARY     | <derived2>      | ALL    | NULL                                                   | NULL                      | NULL    | NULL                                                                                | 1763951 | Using where                        |
|  2 | DERIVED     | <derived3>      | ALL    | NULL                                                   | NULL                      | NULL    | NULL                                                                                | 1763951 | Using filesort                     |
|  3 | DERIVED     | taxon_counts    | range  | index_pr_tax_hit_level_tc,index_taxon_counts_on_tax_id | index_pr_tax_hit_level_tc | 14      | NULL                                                                                | 1763951 | Using index condition; Using where |
|  3 | DERIVED     | pr              | eq_ref | PRIMARY                                                | PRIMARY                   | 8       | idseq_prod.taxon_counts.pipeline_run_id                                                                   |       1 | NULL                               |
|  3 | DERIVED     | taxon_summaries | ref    | index_bg_tax_ct_level                                  | index_bg_tax_ct_level     | 787     | const,idseq_prod.taxon_counts.tax_id,idseq_prod.taxon_counts.count_type,idseq_prod.taxon_counts.tax_level |       1 | NULL                               |
+----+-------------+-----------------+--------+--------------------------------------------------------+---------------------------+---------+-----------------------------------------------------------------------------------------------------------+---------+------------------------------------+

My initial reaction:
This approach has several problems:

First is does not guarantee that user will be able to see relevant taxa (is not necessarily on the top 30) since we cannot guess all use cases
Will not allow computation of genus information (we could preload those, but each genus could have a lot of species that should be stored too)
Does not take into account filters (advanced thresholds, categories, etc.) * leading to missing important results
Does not take into account background which can drastically change the results
There will be a mismatch between the information we show on the report and what we make available on the heat map
It introduces complexity as it is one more table to maintain on all the code that deals with this table
From a UX point of view, I believe this an exploratory tool that should allow users to drill down and explore all data
IIUC, this top would only postpone the problem. With proper indices and limit options you can emulate the behavior of this separate table.

Other options to consider:

A longer wait time can be acceptable by the user, if it can slice and dice on the client side. So we can load more data and in different requests (sort of pagination);
Each sample can be loaded independently. This means we can parallelize heatmap data retrieval (still have a bottle neck on concurrent requests on the DB to consider
If we can make it performant enough with MySQL, we have been considering storing all report info a different way, e.g. noSQL or key/value DB.
If indeed we have a problem with a large table, we consider other type of shards that do not loose information (like per taxon ID)
Even if we consider a top that could help on more than x% of cases, I would not store in MySQL

Side notes:

We will invest some time on reducing the clutter/noise on the report. This is a better way to address too much data for the heatmap (will not get to only 30) but should help.
Backgrounds are a very particular problem that is hard to tackle here. I believe we need to redefine our strategy. If we “lock down” how we evaluate backgrounds we can think of ways to pre-compute tops for a lot of things

In line with the process wishlist we discussed, this may be a good avenue for a team-wide design discussion :) In the meantime, I almost think it might be higher-priority to knock out some of the frontend issues first -- what do you think?

Good points @tfrcarvalho. My responses below. First, as a general response, I agree with the weaknesses of the proposed but I think it was worth considering as something I could finish in the next week.

First is does not guarantee that user will be able to see relevant taxa (is not necessarily on the top 30) since we cannot guess all use cases

Yes, the precomputation is on a subset to taxon_counts. Any query that cannot be satisfied by that subset will be routed to the existing code path.

Will not allow computation of genus information (we could preload those, but each genus could have a lot of species that should be stored too)

Yes, I was assuming we would preload the genus-level counts as well, which I assume is a negligible amount of additional rows.

but each genus could have a lot of species that should be stored too)

Not sure what that means... would we need to store all species that go into a genus count? I don't see that information in the UI currently.

Does not take into account filters (advanced thresholds, categories, etc.) * leading to missing important results

See above comment about subset.

Does not take into account background which can drastically change the results

See note about precomputing zscores for top backgrounds.

There will be a mismatch between the information we show on the report and what we make available on the heat map

I don't think there should. Can you provide an example?

It introduces complexity as it is one more table to maintain on all the code that deals with this table

I think this is unavoidable if we keep taxon_counts as-is. We will need some smaller derived data structure for performance. It could be in a non-SQL database.

From a UX point of view, I believe this an exploratory tool that should allow users to drill down and explore all data
IIUC, this top would only postpone the problem.

With proper indices and limit options you can emulate the behavior of this separate table.

In SQL? I'd love to hear more.

The max N allowed in the UI is applied (or should be) only after filters and background subtraction

Yes, see first reply about subset.

Other options to consider:

A longer wait time can be acceptable by the user, if it can slice and dice on the client side. So we can load more data and in different requests (sort of pagination);

Yes I think this is what we need for the best UEX.

we have been considering storing all report info a different way, e.g. noSQL or key/value DB.

I think an in-memory database or a database designed for OLAP would allow a more complete solution to heatmap performance and the size of taxon_counts. Of course, this option would take more work upfront and a bigger commitment from the team to support another system.

@cdebourcy, yes my plan is to work on UI changes while we discuss re-design for performance. Can you discuss a good time to discuss this proposal as a team in person?

As another interesting fact, currently the heatmap only considers taxons that have more than 4 reads. There is no way to override this in the UI, even with threshold filters (which is certainly misleading). And 65M out of 114M rows in taxon_counts have 4 or fewer reads, more than half.

@cdebourcy, yes my plan is to work on UI changes while we discuss re-design for performance. Can you suggest a good time to discuss this proposal as a team?

Maybe Wednesday? Or a different day, without me :) You should value the opinion of @davidrissato / @tfrcarvalho / others on the team much more than mine anyway ha

I synched briefly with @tfrcarvalho and we decided to invest our energy in a more comprehensive solution when we have time.