Running "docker-compose up" fails to compile successfully (Error on importing STR_NA_VALUES from pandas)

Running docker-compose up on version 1.3.2 (and 1.4.0rc3) fails to compile.
I understand that we are meant to use the non-dev build (i.e. docker-compose -f docker-compose-non-dev.yml up) in order to use the superset app when running via docker, but there are also times where we may want to make modifications which would require the docker-compose up command to be run?
How to reproduce the bug

Go to release on the github, download the 1.3.2 zip (https://github.com/apache/superset/archive/refs/tags/1.3.2.zip) (edit: also happens on 1.4.0rc3, you can try it with that too)
Extract the folder
Enter the superset-1.3.2 folder in a terminal
Run docker-compose up

Expected results
I would expect to be presented with the Superset login screen / welcome page when viewing on browser.

I would expect it to compile successfully without error.
Actual results
Presented with stack trace of error when viewing in browser.

The docker logs say this, or similar in most of the containers (app, worker, worker-beat):
ImportError: cannot import name 'STR_NA_VALUES' from 'pandas.io.parsers' (/usr/local/lib/python3.8/site-packages/pandas/io/parsers/__init__.py) in most containers.
More full logs here:
https://0bin.net/paste/QaIy4jLo#n1g3eqjy3UCcxVj6OZ2VLYge32s-nN2aWqLNy7XCTNE
Environment
(please complete the following information):

docker: Docker version 20.10.12, build e91ed57
browser type and version: N/A (any / all), tested on ChromeVersion 97.0.4692.71 (Official Build) (64-bit)
superset version: 1.3.2 / Superset 0.0.0dev (according to running command in docker container)
python version: Python 3.8.12 (according to running command in docker container)
node.js version: OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "node": executable file not found in $PATH: unknown (according to running command in docker container)
any feature flags active: Unsure / Only default

Checklist
Make sure to follow these steps before submitting your issue - thank you!

[x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
[x] I have reproduced the issue with at least the latest released version of superset.
[x] I have checked the issue tracker for the same issue and I haven't found one similar.

Additional context
Here is the pip freeze on the superset app docker container:
https://0bin.net/paste/aKKgpHJ3#uAUB7QBQyt7wpPXn6d3CPtjoFsnMZI0+oF7VymnAwbO
I see that pandas is on 1.3.4 but in the requirements base.txt it is 1.2.2, so I am not sure how it upgraded itself ( I haven't checked if other packages are correct or not)
Might be related to this? https://github.com/apache/superset/pull/16400
Other user had an error here similar: https://github.com/apache/superset/issues/17333 but I didn't understand how they fixed it.

The pull request from @ravimakhija https://github.com/apache/superset/pull/16400 does seem to fix it, if I set it to use pandas._libs.parsers instead of pandas.io.parsers in the superset/config.py

Just ran into this trying to develop against 1.3, but I was able to resolve it by running docker build -t apache/superset:latest-dev ., which made sure I had the right versions of things in the docker image. An alternative workaround might be to find a suitable version here and pull it.
docker-compose.yml is configured to use apache/superset:latest-dev, which presumably is built from the master branch. So by building locally you can ensure you have the right versions of things for your branch.

@jenseng thanks man! It helps.

Cause: Git clone from any branch other than master, doesn’t build & get all the required files/folders for docker to run in development mode.
Solution: The below command should be ran first to get all the required files/folders for docker.
docker build -t apache/superset:latest-dev .
Order of execution:

Remove all the node modules in superset-frontend folder.
delete all volumes in docker related to the existing container. ( docker-compose down -v )
docker build -t apache/superset:latest-dev .
docker-compose up
Wait until superset-init & superset-node complete.

The pull request from @ravimakhija #16400 does seem to fix it, if I set it to use pandas._libs.parsers instead of pandas.io.parsers in the superset/config.py

No need of this, docker build -t apache/superset:latest-dev . takes care of it too.

The pull request from @ravimakhija #16400 does seem to fix it, if I set it to use pandas._libs.parsers instead of pandas.io.parsers in the superset/config.py

No need of this, docker build -t apache/superset:latest-dev . takes care of it too.

Oh wow that seems like a better option, thank you :)

Cause: Git clone from any branch other than master, doesn’t build & get all the required files/folders for docker to run in development mode.
Solution: The below command should be ran first to get all the required files/folders for docker. docker build -t apache/superset:latest-dev .
Order of execution:

Remove all the node modules in superset-frontend folder.
delete all volumes in docker related to the existing container. ( docker-compose down -v )
docker build -t apache/superset:latest-dev .
docker-compose up
Wait until superset-init & superset-node complete.

I did the same as you and tried other attempts and didn't get any error (in logs) but the page is not showing

Cause: Git clone from any branch other than master, doesn’t build & get all the required files/folders for docker to run in development mode.
Solution: The below command should be ran first to get all the required files/folders for docker. docker build -t apache/superset:latest-dev .
Order of execution:

Remove all the node modules in superset-frontend folder.
delete all volumes in docker related to the existing container. ( docker-compose down -v )
docker build -t apache/superset:latest-dev .
docker-compose up
Wait until superset-init & superset-node complete.

 I did the same as you and tried other attempts and didn't get any error (in logs) but the page is not showing

You need to wait for more than 30 mins to let all the examples & modules load. It'll work.

Cause: Git clone from any branch other than master, doesn’t build & get all the required files/folders for docker to run in development mode.
Solution: The below command should be ran first to get all the required files/folders for docker. docker build -t apache/superset:latest-dev .
Order of execution:

Remove all the node modules in superset-frontend folder.
delete all volumes in docker related to the existing container. ( docker-compose down -v )
docker build -t apache/superset:latest-dev .
docker-compose up
Wait until superset-init & superset-node complete.

 I did the same as you and tried other attempts and didn't get any error (in logs) but the page is not showing

You need to wait for more than 30 mins to let all the examples & modules load. It'll work. (Since it is the first time, it takes time to load all the examples, tables, modules etc..)

hmm, I've been waiting near by 2 hours and without changes :/

Cause: Git clone from any branch other than master, doesn’t build & get all the required files/folders for docker to run in development mode.
Solution: The below command should be ran first to get all the required files/folders for docker. docker build -t apache/superset:latest-dev .
Order of execution:

Remove all the node modules in superset-frontend folder.
delete all volumes in docker related to the existing container. ( docker-compose down -v )
docker build -t apache/superset:latest-dev .
docker-compose up
Wait until superset-init & superset-node complete.

 I did the same as you and tried other attempts and didn't get any error (in logs) but the page is not showing

You need to wait for more than 30 mins to let all the examples & modules load. It'll work. (Since it is the first time, it takes time to load all the examples, tables, modules etc..)

hmm, I've been waiting near by 2 hours and without changes :/ any errors, any visible results on the page

It should work. Is the image got sucessfully build? can you share screenshots for more understanding?

Cause: Git clone from any branch other than master, doesn’t build & get all the required files/folders for docker to run in development mode.
Solution: The below command should be ran first to get all the required files/folders for docker. docker build -t apache/superset:latest-dev .
Order of execution:

Remove all the node modules in superset-frontend folder.
delete all volumes in docker related to the existing container. ( docker-compose down -v )
docker build -t apache/superset:latest-dev .
docker-compose up
Wait until superset-init & superset-node complete.

 I did the same as you and tried other attempts and didn't get any error (in logs) but the page is not showing

You need to wait for more than 30 mins to let all the examples & modules load. It'll work. (Since it is the first time, it takes time to load all the examples, tables, modules etc..)

hmm, I've been waiting near by 2 hours and without changes :/ any errors, any visible results on the page

It should work. Is the image got sucessfully build? can you share screenshots for more understanding?

Yes, image builded successfuly  (Successfully tagged apache/superset:latest-dev), everything is the same as in apache/superset repo on branch 1.4.1cr1 except docker-compose.yml, because I need to have prodaction version of database seted locally (in docker-compose-non-dev.yml) and dev db in docker container for my experements, I already had this configuration on superset version 1.3.2
x-superset-image: &superset-image apache/superset:latest-dev
x-superset-user: &superset-user root
x-superset-depends-on: &superset-depends-on
  - db
  - redis
x-superset-volumes: &superset-volumes
  # /app/pythonpath_docker will be appended to the PYTHONPATH in the final container
  - ./docker:/app/docker
  - ./superset:/app/superset
  - ./superset-frontend:/app/superset-frontend
  - superset_home:/app/superset_home
  - ./tests:/app/tests

version: "3.7"
services:
  redis:
    image: redis:latest
    container_name: d_superset_cache
    restart: unless-stopped
    # ports:
    #   - "<IP_ADDRESS>:6379:6379"
    volumes:
      - redis:/data

  db:
    env_file: docker/.env
    image: postgres:10
    container_name: d_superset_db
    restart: unless-stopped
    # ports:
    #   - "<IP_ADDRESS>:5432:5432"
    volumes:
      - db_home:/var/lib/postgresql/data
      - /home/dbondarenko/Dev/superset/extensions:/dev/db_extensions

  d_superset:
    env_file: docker/.env
    image: *superset-image
    container_name: d_superset_app
    command: ["/app/docker/docker-bootstrap.sh", "app"]
    restart: unless-stopped
    ports:
      - 8099:8088
    user: *superset-user
    depends_on: *superset-depends-on
    volumes: *superset-volumes
    environment:
      CYPRESS_CONFIG: "${CYPRESS_CONFIG}"

  d_superset-websocket:
    container_name: d_superset_websocket
    build: ./superset-websocket
    image: superset-websocket
    ports:
      - 8080:8080
    depends_on:
      - redis
    # Mount everything in superset-websocket into container and
    # then exclude node_modules and dist with bogus volume mount.
    # This is necessary because host and container need to have
    # their own, separate versions of these files. .dockerignore
    # does not seem to work when starting the service through
    # docker-compose.
    #
    # For example, node_modules may contain libs with native bindings.
    # Those bindings need to be compiled for each OS and the container
    # OS is not necessarily the same as host OS.
    volumes:
      - ./superset-websocket:/home/superset-websocket
      - /home/superset-websocket/node_modules
      - /home/superset-websocket/dist
    environment:
      - PORT=8080
      - REDIS_HOST=redis
      - REDIS_PORT=6379
      - REDIS_SSL=false

  d_superset-init:
    image: *superset-image
    container_name: d_superset_init
    command: ["/app/docker/docker-init.sh"]
    env_file: docker/.env
    depends_on: *superset-depends-on
    user: *superset-user
    volumes: *superset-volumes
    environment:
      CYPRESS_CONFIG: "${CYPRESS_CONFIG}"

  d_superset-node:
    image: node:16
    container_name: d_superset_node
    command: ["/app/docker/docker-frontend.sh"]
    env_file: docker/.env
    depends_on: *superset-depends-on
    volumes: *superset-volumes

  d_superset-worker:
    image: *superset-image
    container_name: d_superset_worker
    command: ["/app/docker/docker-bootstrap.sh", "worker"]
    env_file: docker/.env
    restart: unless-stopped
    depends_on: *superset-depends-on
    user: *superset-user
    volumes: *superset-volumes
    # Bump memory limit if processing selenium / thumbails on superset-worker
    # mem_limit: 2038m
    # mem_reservation: 128M

  d_superset-worker-beat:
    image: *superset-image
    container_name: d_superset_worker_beat
    command: ["/app/docker/docker-bootstrap.sh", "beat"]
    env_file: docker/.env
    restart: unless-stopped
    depends_on: *superset-depends-on
    user: *superset-user
    volumes: *superset-volumes

  d_superset-tests-worker:
    image: *superset-image
    container_name: d_superset_tests_worker
    command: ["/app/docker/docker-bootstrap.sh", "worker"]
    env_file: docker/.env
    environment:
      DATABASE_HOST: localhost
      DATABASE_DB: test
      REDIS_CELERY_DB: 2
      REDIS_RESULTS_DB: 3
      REDIS_HOST: localhost
    network_mode: host
    depends_on: *superset-depends-on
    user: *superset-user
    volumes: *superset-volumes

volumes:
  superset_home:
    external: false
  db_home:
    external: false
  db_extensions:
    external: false
  redis:
    external: false

and few added variables in superset_config.py:
SQLLAB_CTAS_NO_LIMIT = True
SUPERSET_WEBSERVER_TIMEOUT = 60 * 30

Cause: Git clone from any branch other than master, doesn’t build & get all the required files/folders for docker to run in development mode.
Solution: The below command should be ran first to get all the required files/folders for docker. docker build -t apache/superset:latest-dev .
Order of execution:

Remove all the node modules in superset-frontend folder.
delete all volumes in docker related to the existing container. ( docker-compose down -v )
docker build -t apache/superset:latest-dev .
docker-compose up
Wait until superset-init & superset-node complete.

 I did the same as you and tried other attempts and didn't get any error (in logs) but the page is not showing

You need to wait for more than 30 mins to let all the examples & modules load. It'll work. (Since it is the first time, it takes time to load all the examples, tables, modules etc..)

hmm, I've been waiting near by 2 hours and without changes :/ any errors, any visible results on the page

It should work. Is the image got sucessfully build? can you share screenshots for more understanding?

Yes, image builded successfuly (Successfully tagged apache/superset:latest-dev), everything is the same as in apache/superset repo on branch 1.4.1cr1 except docker-compose.yml, because I need to have prodaction version of database seted locally (in docker-compose-non-dev.yml) and dev db in docker container for my experiments, I already had this configuration on superset version 1.3.2
x-superset-image: &superset-image apache/superset:latest-dev
x-superset-user: &superset-user root
x-superset-depends-on: &superset-depends-on
  - db
  - redis
x-superset-volumes: &superset-volumes
  # /app/pythonpath_docker will be appended to the PYTHONPATH in the final container
  - ./docker:/app/docker
  - ./superset:/app/superset
  - ./superset-frontend:/app/superset-frontend
  - superset_home:/app/superset_home
  - ./tests:/app/tests

version: "3.7"
services:
  redis:
    image: redis:latest
    container_name: d_superset_cache
    restart: unless-stopped
    # ports:
    #   - "<IP_ADDRESS>:6379:6379"
    volumes:
      - redis:/data

  db:
    env_file: docker/.env
    image: postgres:10
    container_name: d_superset_db
    restart: unless-stopped
    # ports:
    #   - "<IP_ADDRESS>:5432:5432"
    volumes:
      - db_home:/var/lib/postgresql/data
      - /home/dbondarenko/Dev/superset/extensions:/dev/db_extensions

  d_superset:
    env_file: docker/.env
    image: *superset-image
    container_name: d_superset_app
    command: ["/app/docker/docker-bootstrap.sh", "app"]
    restart: unless-stopped
    ports:
      - 8099:8088
    user: *superset-user
    depends_on: *superset-depends-on
    volumes: *superset-volumes
    environment:
      CYPRESS_CONFIG: "${CYPRESS_CONFIG}"

  d_superset-websocket:
    container_name: d_superset_websocket
    build: ./superset-websocket
    image: superset-websocket
    ports:
      - 8080:8080
    depends_on:
      - redis
    # Mount everything in superset-websocket into container and
    # then exclude node_modules and dist with bogus volume mount.
    # This is necessary because host and container need to have
    # their own, separate versions of these files. .dockerignore
    # does not seem to work when starting the service through
    # docker-compose.
    #
    # For example, node_modules may contain libs with native bindings.
    # Those bindings need to be compiled for each OS and the container
    # OS is not necessarily the same as host OS.
    volumes:
      - ./superset-websocket:/home/superset-websocket
      - /home/superset-websocket/node_modules
      - /home/superset-websocket/dist
    environment:
      - PORT=8080
      - REDIS_HOST=redis
      - REDIS_PORT=6379
      - REDIS_SSL=false

  d_superset-init:
    image: *superset-image
    container_name: d_superset_init
    command: ["/app/docker/docker-init.sh"]
    env_file: docker/.env
    depends_on: *superset-depends-on
    user: *superset-user
    volumes: *superset-volumes
    environment:
      CYPRESS_CONFIG: "${CYPRESS_CONFIG}"

  d_superset-node:
    image: node:16
    container_name: d_superset_node
    command: ["/app/docker/docker-frontend.sh"]
    env_file: docker/.env
    depends_on: *superset-depends-on
    volumes: *superset-volumes

  d_superset-worker:
    image: *superset-image
    container_name: d_superset_worker
    command: ["/app/docker/docker-bootstrap.sh", "worker"]
    env_file: docker/.env
    restart: unless-stopped
    depends_on: *superset-depends-on
    user: *superset-user
    volumes: *superset-volumes
    # Bump memory limit if processing selenium / thumbails on superset-worker
    # mem_limit: 2038m
    # mem_reservation: 128M

  d_superset-worker-beat:
    image: *superset-image
    container_name: d_superset_worker_beat
    command: ["/app/docker/docker-bootstrap.sh", "beat"]
    env_file: docker/.env
    restart: unless-stopped
    depends_on: *superset-depends-on
    user: *superset-user
    volumes: *superset-volumes

  d_superset-tests-worker:
    image: *superset-image
    container_name: d_superset_tests_worker
    command: ["/app/docker/docker-bootstrap.sh", "worker"]
    env_file: docker/.env
    environment:
      DATABASE_HOST: localhost
      DATABASE_DB: test
      REDIS_CELERY_DB: 2
      REDIS_RESULTS_DB: 3
      REDIS_HOST: localhost
    network_mode: host
    depends_on: *superset-depends-on
    user: *superset-user
    volumes: *superset-volumes

volumes:
  superset_home:
    external: false
  db_home:
    external: false
  db_extensions:
    external: false
  redis:
    external: false

and few added variables in superset_config.py:
SQLLAB_CTAS_NO_LIMIT = True
SUPERSET_WEBSERVER_TIMEOUT = 60 * 30

can you remove SUPERSET_WEBSERVER_TIMEOUT configuration & try all the procedure from the beginning.

docker build -t apache/superset:latest-dev .

after switched up to 1.4.2 branch I did the same with equal config and it's working in dev mode. Currently I can't check if the prod version works too because it's used daily by other departments. But I really appreciate your help, thank's a lot.

The solution proposed by @pdakkata-quot solved the problem. The white screen after the build is due to the webpack still being processing the frontend. You should wait for the message <s> [webpack.Progress] 100% to access the interface. In my case, it took around 20 minutes (!!!!) with 4 cores and 8gb allocated to docker for mac. They should consider switching Webpack to Esbuild.

Thank you to all who helped in this thread! It sounds like the situation was resolved and this just didn't get closed. I'm also happy to say that nowadays in 2.0.1, I do docker-compose up almost daily to tweak on things. Seems to be working just fine :)


Got Access token is unavailable Error when use githubToken option

**Description of the bug

When adding the githubToken option according to documentation, Access token is unavailable Error occurs.
If the option is removed, the Action succeeds.

**How to reproduce

Create a Github personal access token with full access and no expiration date.
Pass the PAT to the githubToken option of game-ci/unity-test-runner@v2.
Action failed.

expected behavior.

Action succeeded.

Additional details

 My workflow
name: Test, Build, and Release My Demo

on:
  workflow_dispatch: {}
  # schedule:
  #   - cron: '0 19 * * 1-5'

jobs:
  tests:
    name: Test Code Quality
    runs-on: ubuntu-latest
    timeout-minutes: 60
    steps:
      - name: Checkout Repository
        uses: actions/checkout@v2
        with:
          fetch-depth: 0
      - name: Access to private repo
        uses<EMAIL_ADDRESS>        with:
          ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }}
      - name: Cache Library
        uses: actions/cache@v2
        with:
          path: Library
          key: Library
      - name: Run EditMode and PlayMode Tests
        id: tests
        uses: game-ci/unity-test-runner@v2
        env:
          UNITY_EMAIL: ${{ secrets.UNITY_EMAIL }}
          UNITY_PASSWORD: ${{ secrets.UNITY_PASSWORD }}
          UNITY_SERIAL: ${{ secrets.UNITY_SERIAL }}
        with:
          sshAgent: ${{ env.SSH_AUTH_SOCK }}
      - uses: actions/upload-artifact@v2
        if: always()
        with:
          name: Test results for ${{ matrix.testMode }}
          path: ${{ steps.tests.outputs.artifactsPath }}

 Error message
###########################
#    Play Mode Results    #
###########################

<?xml version="1.0" encoding="utf-8"?>
<test-run id="2" testcasecount="0" result="Passed" total="0" passed="0" failed="0" inconclusive="0" skipped="0" asserts="0" engine-version="<IP_ADDRESS>" clr-version="4.0.30319.42000" start-time="2022-01-21 09:33:34Z" end-time="2022-01-21 09:33:34Z" duration="0.0027389">
  <test-suite type="TestSuite" id="1000" name="My Demo" fullname="MOOOS Demo" runstate="Runnable" testcasecount="0" result="Passed" start-time="2022-01-21 09:33:34Z" end-time="2022-01-21 09:33:34Z" duration="0.002739" total="0" passed="0" failed="0" inconclusive="0" skipped="0" asserts="0">
    <properties />
  </test-suite>
</test-run><test-run id="2" testcasecount="0" result="Passed" total="0" passed="0" failed="0" inconclusive="0" skipped="0" asserts="0" engine-version="<IP_ADDRESS>" clr-version="4.0.30319.42000" start-time="2022-01-21 09:33:34Z" end-time="2022-01-21 09:33:34Z" duration="0.0027389">
Changing to "/github/workspace/_activate-license" directory.
/github/workspace/_activate-license /github/workspace
Unity Editor version:    2021.2.6f1 (8c4e826ba445)
Branch:                  2021.2/staging
Build type:              Release
Batch mode:              YES
System name:             Linux
Distro version:          #27~20.04.1-Ubuntu SMP Fri Jan 7 15:02:06 UTC 2022
Kernel version:          5.11.0-1025-azure
Architecture:            x86_64
Available memory:        6951 MB
[Licensing::Module] Channel doesn't exist: "LicenseClient-root"
[Licensing::Module] Successfully launched the LicensingClient (PId: 2362)
[SignatureVerifier] Application signature verification not supported on this platform.
[LicensingClient] Handshaking with LicensingClient (version: 1.9.0+249add7)
[Licensing::Module] Successfully connected to LicensingClient on channel: "LicenseClient-root" (connect: 0.45s, validation: 0.09s, handshake: 0.00s)
[Licensing::Module] Connected to LicensingClient (PId: 2362, launch time: 0.00, total connection time: 0.54s)
Entitlement-based licensing initiated
[Licensing::Module] Error: Access token is unavailable
[Licensing::Module] Error: Failed to return entitlement license

The githubToken option is intended to help with displaying the test results in the GitHub UI.
However, I see you are instead using the sshAgent option in your workflow, which is intended to allow using private repos in your dependencies.
What is your actual intention?

Closing due to lack of information.


docs(versioning): update versioning guide with css custom property changes

Saw this scenario come up recently and wanted to make sure it was documented with respect to corresponding semver change 👀
Changelog

New

Changed

Add section to versioning guide around changing CSS Custom Properties for a component

Removed

Rollout strategy

[x] None; if selected, include a brief description as to why

This is a change to project documentation
Testing & Reviewing

I would love to get your feedback on this entry in the versioning doc. Here are some questions if none come to mind to get things going 👀

What do you think of this decision? Is it too broad? Too specific?
Are there any cases to changing a CSS Custom Property that this entry misses?
Do you agree or disagree with the decision for the semver bump annotation for this type of change?

Wanted to get your thoughts on this @langermank @mperrotti if you have the time! Let me know if I can provide any additional context for this.
Also wanted to include you @lukasoppermann as an FYI in case this was of interest due to overlap with primitives 👀 The work you did with Banner prompted this and I wanted to make sure this was captured in our versioning guide 👍


Small copy change to 'Add Funds' Screen

AC:
Figma

Add funds => Add [currency] in the header
[Currency] next to the total on the right

The cUSD next to the total amount on the right seems repetitive to me, I'll bring up to Nitya

For now, let's keep the cUSD next to the amount so that its clear to the user the units. Even though we have cUSD at the top of the page, users will be drawn to the conversion they see in the last line. Being explicit about units helps build user trust.

Hey @MuckT we have provided our observations in PR #2309
Thanks..!
Cc: @lisaqyli


Table rfc

Fixed Issues?
N/A
Patch: Bug Fix?
N/A
Major: Breaking Change?
N/A
Minor: New Feature?
N/A
Tests Added + Pass?
N/A
Documentation PR Link
[skip ci]
Any Dependency Changes?
License
MIT
This is a technical proposal for baseUI regarding the API design of a basic table, which handles the canonical table format of one header row and many rows of data. Any other table formats (e.g. having rows or columns that may span multiple rows or columns can be handled using a renderProp)
Also, there will be support for sorting.
I had a question about what was the best way to determine how many rows are displayed initially in the table. Let's say the user has 5000 rows to load into a table. I was thinking about arbitrarily setting a pageSize of 10, such that 10 rows are displayed initially, and then on user scroll or click of a pagination element, more elements are displayed. This lends the question: should a table come with a scrollbar and/or pagination element out of the box? I currently don't have this documented in the proposal but needed further discussion.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

closing this to continue conversation in https://github.com/uber-web/baseui/pull/632


Add a way to supply a custom ParagraphBuilder

Use case
ParagraphBuilder is the abstraction for deciding how paragraphs should be layed out. But widgets like Text.rich don't support supplying a custom ParagraphBuilder. As described in #154946, it would be useful if one could supply a custom ParagraphBuilder to build a chat bubble.
This would also enable packages to provide custom layout algorithms.
Proposal
I'd suggest adding a ParagraphBuilderFactory parameter to the relevant widgets that can be called to receive a custom ParagraphBuilder. This parameter could be nullable and fallback to the default _NativeParagraphBuilder.

I've started implementing this here. But I run into #154958 while using my implementation.

The Native types are not overridable, as their implementation is in the engine and the engine needs to make assumptions about the objects it interacts with. I would recommend pursuing a different strategy.


Displaying error message when error

This change is to control the error handling of the GUI endpoints. It's assuming any error coming from the endpoint should be considered as an error page. If we want to change to have specific error pages per error (404, 500) we can easily change it.

@EricDube Please rebase on master to get new CI Pipeline

@CalvinRodo @newcitywave @will0684 @MarcoGoC  I have removed front-end logging, could I get a few reviews? Let me know if theres any changes to be made.

Eric, it's breaking some linting...

@P0NDER0SA

Eric, it's breaking some linting...

I'm looking into it, I believe some code in Master was merged and broke something.


Beta

PR-Codex overview
This PR focuses on major version updates, new exports, adapters, scripts, and dependencies in the thirdweb package.


[Bug]: tab Field Values at Dimension lookup/update step not saving values

Apache Hop version?
2.3.0
Java version?
OpenJDK11U 11.0.18_10
Operating system
Windows
What happened?
At the Dimension lookup/update step

Even after choosing one type for every field

Once saved (pressing Ok) it's not saved successfuly and is set to None again once I reopen the transform. Causing an error on the execution.
I checked the hpl file and it looks to be missing a tag at

Tried adding this tag. As I saw other transformations having type features used it.
<type>String</type>
However it didn't work and it was overwritten to its original state.
I tried to replicate this case in the samples project with new connections and the error is happening there too. In case it helps and to be more precise  this happens in our Windows Server 2019 Datacenter
Issue Priority
Priority: 1
Issue Component
Component: Hop Gui

.take-issue

which version of Hop is this because your dialog is wrong and does not match 2.3
your dialog should look like this on the fields tab.

Unless something is blocking you from doing an upgrade I would update at least to 2.4

Ah, you disabled the update dimension option... that part was removed from your screenshots.
let me look a bit further


WIP: Honor UUID interceptor in ElasticSearch sink

ElasticSearch provides support for deduplication of events using the
_id field when inserting documents using the bulk insert facility.
When using the UUID interceptor with headerName = _id for this
purpose, the logstash serializer will rename this as @fields._id,
which is stored as though it were any other field.
In order to enable deduplication of events in the ElasticSearch sink, we
need to ensure the header containing the document ID is sent as _id,
no matter what its original name is.
The dynamic serializer probably does not have the same limitation, but
it currently counts on the UUID interceptor's headerName property to
be set to _id.  This is not the default and it is unlikely to work in
complex Flume networks that replicate events to multiple sinks.
Therefore, I fixed it to support the same logic as the logstash
serializer.

Support for Elasticsearch has been removed from the main Flume build since it is so out of date and its new license is problematic. Please work against the flume-search repo.


Sync version to 17.3.2

Sync version to 17.3.2

Would it be possible to merge this already and publish this version?
Since it seems to patch a critical CVE.

Anyways, thank you for providing this image ❤


Features: startPaused, getCurrentFrame and  onFrame Event

Related to #4, Some useful tools for multiplayer and recording/playback would be related to knowing the frame.

{ startPaused: true } starting paused would allow a group of players to prepare the games and start at relatively the same time.
getCurrentFrame() getting the current frame would be useful for knowing when a player did an action. Good for broadcasting actions to happen in the future or for recording the action to be played later
on("frame", ()=>{}) knowing what frame the player is on would be good commanding prepared actions. For my examples, the event happens before the buttons are read.

This might be more work than it's worth but it seems interesting
// Not working Example Multiplayer Client

const controller = new OnscreenController(document.querySelector("#OnscreenController"))

const connection = new WebSocket("/game-room/12345");

const nostalgist = await Nostalgist.launch({
  startPaused: true, // Needed so that all players start around the same time
  core: 'fceumm',
  rom: 'flappybird.nes',
})

await syncWithRoom(connection);
const frameOffset = await getFrameOffset(connection)

var startListener = (event)=>{
  const data = JSON.parse(event.data)
  if(data.action !== "start") return;
  connection.removeEventListener("message", startListener)
  nostalgist.resume();
}
connection.addEventListener("message", startListener)

const ACTIONS_TODO = {};

controller.addEventListener("button", (event)=>{
  const { button, toggleValue } = event.data
  const targetFrame = nostalgist.getFrame() + frameOffset;
  connection.send({
    targetFrame, button, toggleValue
  })
})

// Doesn't have to be event target. just simpler to use it for the example
nostalgist.addEventListener("frame", (event)=>{
  const currentFrame = event.frame;
  const todos = ACTIONS_TODO[currentFrame];
  if(todos === void 0) return;
  delete ACTIONS_TODO[currentFrame]
  todos.forEach((todo)=>{
    nostalgist.press(todo.player, todo.button, todo.toggleValue)
  });
})

connection.addEventListener("message", (event)=>{
  // Handle Input
  const data = JSON.parse(event.data);
  if(data.action !== "input") return;

  const { targetFrame, player, button, toggleValue } = data
  const currentFrame = nostalgist.getFrame()
  if(targetFrame <= currentFrame){
    nostalgist.exit();
    connection.send(JSON.stringify({
      action: "desync"
    }));
    alert("You are out of sync with the room")
    return;
  }
  if(!(targetFrame in ACTIONS_TODO)){
    ACTIONS_TODO[targetFrame] = []
  }
  ACTIONS_TODO[targetFrame].push({ player, button, toggleValue })
})
connection.addEventListener("message", (event)=>{
  const data = JSON.parse(event.data);
  if(data.action !== "desync") return;
  nostalgist.removeController(data.player)
})

// Not working Example Input Recording

const PLAYER = 0;

// Recording

function runRecorder(){
  const controller = new OnscreenController(document.querySelector("#OnscreenController"))

  const nostalgist = await Nostalgist.launch({
    startPaused: true, // Needed so that all players start around the same time
    core: 'fceumm',
    rom: 'flappybird.nes',
  })

  const transcript = [];

  controller.addEventListener("button", (event)=>{
    const { button, toggleValue } = event.data
    transcript.push({ frame: nostalgist.getFrame(), type: "button", button, toggleValue });
    nostalgist.press(PLAYER, button, toggleValue)
  })

  return new Promise((res)=>{
    nostalgist.addEventListener("exit", ()=>{
      transcript.push({ frame: nostalgist.getFrame(), type: "off" })
      res(transcript)
    })
  });
}

// Playback

function playTranscript(transcript){
  var transcriptOffset = 0;
  nostalgist.addEventListener("frame", (event)=>{
    const currentFrame = event.frame;
    for(transcriptOffset; transcriptOffset < transcript.length; transcriptOffset++){
      const action = transcript[transcriptOffset]
      if(action.frame > currentFrame) break;
      switch(action.type){
        case "button": {
          nostalgist.press(PLAYER, action.button, action.toggleValue);
          break;
        }
        case "exit":{
          nostalgist.pause();
          alert("recording finished");
          break;
        }
      }
    }
  })
}

Though these are great ideas, unfortunately, I'm afraid they can't be implemented very easily 🥲  ! Due to the fact Nostalgist.js being built on up of RetroArch, a native program written in C, what can we do at JS level is quite limited. Here are some "commands" and "functions" exposed by RetroArch. Nostalgist.js uses some of them under the hood.
Basically, we can only interact with RetroArch through three methods:

specifying its configuration files
invoking its commands/functions(including some undocumented ones like that dynCall_iiii mentioned in https://github.com/arianrhodsandlot/nostalgist/issues/4#issuecomment-1761485219)
reading its file system

Additionally, do players in a netplay room need to start a game at the same time? Maybe the host can start a game first. When a guest joins the room, he can synchronize his state with the host's. Some arcade games even allow 2P to join the game while 1P is playing in a game scene.

Thats a great point, people should be able to enter and leave on demand.
Just figured id see if something like this is possible, regardless great job on the project!


Question for about more custom waves

The wave param list is statically placed in the SceShell data segment and currently only one is available. This plugin makes use of that space. A simple workaround is to overwrite the SCE original waveparams.

Can you clarify how this is done? Does it mean that i can rewrite EVERY Vita original Wave? I have bunch of cool looking mutated waves and i want to use them. Can you please explain how to do it?
I love this plugin so much, it does have future, just letting you know :)

That's amazing!

It's was released as pre-release. https://github.com/Princess-of-Sleeping/psp2wpp/releases/tag/v1.1
Copy the txt with the name in list to the same place as waveparam.txt, or rename your waveparam to the name in list and copy it to the same place as waveparam.txt. and report the results.

Results: It still works only as v1.0 (waveparam.txt + waveparam.bin)
Things i've tried:

"iboot.txt" - does nothing
"effects_default_0512_01.txt" - does nothing
"effects_default_0512_01.txt" + "waveparam.txt" - does nothing
"effects_default_0512_01.txt" + "waveparam.txt" + "waveparam.bin" - works, but shows only "waveparam" wave
"effects_default_0512_01.txt" + "effects_default_0512_01.bin" + "waveparam.txt" + "waveparam.bin" - works, but shows only "waveparam" wave

I have tried everytime other waves (in case if it targets differene wave) and yes, switching to screen with different wave and back.

I tried renaming wave to "waveparam_effects_default_0512_01.txt", restarted system and it does not work too.

Okay finally got it working, i've just totally misunderstood your directions. Now i have "waveparam.bin", "waveparam_effects_default_0512_01.txt" and "waveparam_iboot.txt" in "data" folder, and first two waves are my custom ones! Thank you!

By the way, "psp2wpp-reload" crashes when is "waveparam.txt" present in "data" folder.

Thank you for the detailed information. There is something that fixes them, so please test them with waveparam.txt
psp2wpp.zip

You're welcome.
I tried the plugin you sent and everything works as it should.
I have a small suggestion for improvement: if it would be possible for the plugin to take waveparams from a subfolder (probably "ux0:data/waveparams/") and for the waveparams names to be simplified (0.txt, 1.txt, 2.txt, 3.txt. ....).

Thank you for testting.
And the subfolder tests are in the plugin below.
psp2wpp.zip
ux0:data/waveparams/%d.txt. %d is from 0 to 30.

Here is one of my waveparams https://webshare.cz/#/file/dJxToJlJBo/waveparam-txt

Surprisingly I like that wave. Well, I waiting for the report of subfolder.

I have just ended testing it and it works perfectly. Thank you !

You are welcome.

Sorry to interrupt after closing issue, but I was just adding more waveparams to the plugin, and I find that some of them are mis-referencing. I thoroughly tested it, and I am sending the result: https://webshare.cz/#/file/IuGAK21Vhp/test-waveparams2-bmp

Do some of your waveparams have unfilled values?

Try this one.
psp2wpp.zip

I've tried 8 waves, "0.txt" to "6.txt" is ok, but "7.txt" (which should be dark-blue) is now found on fourth wave from end (yellow-ish color), in older .suprx it was on fifth wave from end (light blue color)

Soory, the value was off by one.
psp2wpp.zip

This is should work now.
psp2wpp.zip

Yes, it works. I will test other 24 waves, i'll report back after while.

All the way from "0.txt" to "30.txt" it works as supposed to. Thank you very much!
By the way when i rename/delete ".txt"
file and launch "psp2wpp-reload", it does not refresh corresponding wave to default one, until vita restarts.
I have little question; wave icons in "select window" can be changed so they match reality? If yes then please how to do that?

The default waveparam is literally overwritten, so it stays there until you reboot.
There is already a select window color change, but this would require an offset per fw. Please wait a moment to add them.
https://github.com/Princess-of-Sleeping/psp2wpp/blob/12ac51384ce6392b9a1bc4fc865959580b32a99f/src/main.c#L51-L59

Allright, thank you.

We have released v1.1.2 with these fixes. And if you want to apply a color to the select add the following parameter to your waveparam.txt.
If you have any problems, open iusse again. Thank you.
type Select
set_value selecter[0].r           0.500000
set_value selecter[0].g           0.000000
set_value selecter[0].b           0.000000
set_value selecter[0].a           1.000000
set_value selecter[1].r           0.9
set_value selecter[1].g           0.2
set_value selecter[1].b           0.1
set_value selecter[1].a           1.000000

i've discovered that those parameters does not work in "waveparam.txt", but they work in other waves. But they work half-way somewhat. selecter[0] changes bottom line with according color, but selecter[1] seems to have twisted values. If i set selecter[1].r, selecter[1].g, selecter[1].b all to 1, i see torquise color

*i see torquise color on top, instead of white. Sorry, message got sent before i ended it.

Thanks again for finding the bug. It seems the value of select[1].r is not applied.
Wrong reference indeed.
https://github.com/Princess-of-Sleeping/psp2wpp/blob/3506d6daebc2f6ab22a90ce82cca5b7c19b67c84/src/wave_config.c#L41-L44
I will soon release v1.1.3 with this fixed.

selecter colors in "waveparam.txt" still doesn't work, i've restarted vita multiple times.
selecter in other waves works.

We have released psp2wpp v1.1.4 which fixes that bug. In v1.1.3, the contents of waveparam.bin were applied.

It works! Thank you!

you are welcome. And Thank you.


IndexError: list index out of range in mcmc_step with JAX v0.4.36

When I've tried to verify if FermiNet is now operational after fix 9a1deec8222c1838c5aafb65830fc41372c48677, I've hit another error:
$ ferminet --config ferminet/configs/atom.py --config.system.atom Li --config.batch_size 256 --config.pretrain.iterations 100
<...snip...>
I1208 17:34:32.893374<PHONE_NUMBER>21664 train.py:584] No checkpoint found. Training new model.
Traceback (most recent call last):
  File "/home/ktokarev/venv/ferminet/bin/ferminet", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/mnt/storage/src/git/ferminet/bin/ferminet", line 39, in <module>
    app.run(main)
  File "/usr/lib/python3.11/site-packages/absl/app.py", line 308, in run
    _run_main(main, args)
  File "/usr/lib/python3.11/site-packages/absl/app.py", line 254, in _run_main
    sys.exit(main(argv))
             ^^^^^^^^^^
  File "/mnt/storage/src/git/ferminet/bin/ferminet", line 35, in main
    train.train(cfg)
  File "/mnt/storage/src/git/ferminet/ferminet/train.py", line 680, in train
    params, data.positions = pretrain.pretrain_hartree_fock(
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/storage/src/git/ferminet/ferminet/pretrain.py", line 342, in pretrain_hartree_fock
    data, params, opt_state_pt, loss, pmove = pretrain_step(
                                              ^^^^^^^^^^^^^^
  File "/mnt/storage/src/git/ferminet/ferminet/pretrain.py", line 255, in pretrain_step
    data, pmove = mcmc_step(full_params, data, key, width=0.02)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/storage/src/git/ferminet/ferminet/mcmc.py", line 277, in mcmc_step
    new_data, key, _, num_accepts = lax.fori_loop(
                                    ^^^^^^^^^^^^^^
IndexError: list index out of range

This error disappears and pretraining starts fine if I roll back jax to v0.4.35, however, in the latter case a different error occurs after the end of the pretraining iterations:
$ ferminet --config ferminet/configs/atom.py --config.system.atom Li --config.batch_size 256 --config.pretrain.iterations 100
<...snip...?
I1208 17:41:02.053053<PHONE_NUMBER>13984 pretrain.py:344] Pretrain iter 00099: 0.000968613 0.914062
I1208 17:41:02.849053<PHONE_NUMBER>13984 tag_graph_matcher.py:1722] ==================================================
I1208 17:41:02.849577<PHONE_NUMBER>13984 tag_graph_matcher.py:1723] Graph parameter registrations:
{'envelope': [{'pi': 'Auto[tag_variant=scale_and_shift(1)|match_type=scale_only_broadcast_1]',
               'sigma': 'Auto[tag_variant=scale_and_shift(0)|match_type=scale_only_broadcast_1]'},
              {'pi': 'Auto[tag_variant=scale_and_shift(3)|match_type=scale_only_broadcast_1]',
               'sigma': 'Auto[tag_variant=scale_and_shift(2)|match_type=scale_only_broadcast_1]'}],
 'layers': {'input': {},
            'streams': [{'double': {'b': 'Auto[tag_variant=repeated_dense(1)|match_type=repeated_dense2_with_bias]',
                                    'w': 'Auto[tag_variant=repeated_dense(1)|match_type=repeated_dense2_with_bias]'},
                         'single': {'b': 'Auto[tag_variant=repeated_dense(0)|match_type=repeated_dense1_with_bias]',
                                    'w': 'Auto[tag_variant=repeated_dense(0)|match_type=repeated_dense1_with_bias]'}},
                        {'double': {'b': 'Auto[tag_variant=repeated_dense(3)|match_type=repeated_dense2_with_bias]',
                                    'w': 'Auto[tag_variant=repeated_dense(3)|match_type=repeated_dense2_with_bias]'},
                         'single': {'b': 'Auto[tag_variant=repeated_dense(2)|match_type=repeated_dense1_with_bias]',
                                    'w': 'Auto[tag_variant=repeated_dense(2)|match_type=repeated_dense1_with_bias]'}},
                        {'double': {'b': 'Auto[tag_variant=repeated_dense(5)|match_type=repeated_dense2_with_bias]',
                                    'w': 'Auto[tag_variant=repeated_dense(5)|match_type=repeated_dense2_with_bias]'},
                         'single': {'b': 'Auto[tag_variant=repeated_dense(4)|match_type=repeated_dense1_with_bias]',
                                    'w': 'Auto[tag_variant=repeated_dense(4)|match_type=repeated_dense1_with_bias]'}},
                        {'single': {'b': 'Auto[tag_variant=repeated_dense(6)|match_type=repeated_dense1_with_bias]',
                                    'w': 'Auto[tag_variant=repeated_dense(6)|match_type=repeated_dense1_with_bias]'}}]},
 'orbital': [{'w': 'Auto[tag_variant=repeated_dense(7)|match_type=repeated_dense1_no_bias]'},
             {'w': 'Auto[tag_variant=repeated_dense(8)|match_type=repeated_dense1_no_bias]'}]}
I1208 17:41:02.849617<PHONE_NUMBER>13984 tag_graph_matcher.py:1729] ==================================================
Traceback (most recent call last):
  File "/home/ktokarev/venv/ferminet/bin/ferminet", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/mnt/storage/src/git/ferminet/bin/ferminet", line 39, in <module>
    app.run(main)
  File "/usr/lib/python3.11/site-packages/absl/app.py", line 308, in run
    _run_main(main, args)
  File "/usr/lib/python3.11/site-packages/absl/app.py", line 254, in _run_main
    sys.exit(main(argv))
  File "/mnt/storage/src/git/ferminet/bin/ferminet", line 35, in main
    train.train(cfg)
  File "/mnt/storage/src/git/ferminet/ferminet/train.py", line 852, in train
    opt_state = optimizer.init(params, subkeys, data)
  File "/home/ktokarev/venv/ferminet/lib/python3.11/site-packages/kfac_jax/_src/optimizer.py", line 1031, in init
    return self._init(params, rng, batch, func_state)
  File "/home/ktokarev/venv/ferminet/lib/python3.11/site-packages/kfac_jax/_src/utils/staging.py", line 297, in decorated
    outs = func(instance, *args, **kwargs)
  File "/home/ktokarev/venv/ferminet/lib/python3.11/site-packages/kfac_jax/_src/optimizer.py", line 999, in _init
    estimator_state=self.estimator.init(
  File "/home/ktokarev/venv/ferminet/lib/python3.11/site-packages/kfac_jax/_src/utils/misc.py", line 342, in wrapped
    return method(instance, *args, **kwargs)
  File "/home/ktokarev/venv/ferminet/lib/python3.11/site-packages/kfac_jax/_src/curvature_estimator/block_diagonal.py", line 390, in init
    self.finalize(func_args)
  File "/home/ktokarev/venv/ferminet/lib/python3.11/site-packages/kfac_jax/_src/utils/misc.py", line 315, in finalize
    self._finalize(*args, **kwargs)
  File "/home/ktokarev/venv/ferminet/lib/python3.11/site-packages/kfac_jax/_src/curvature_estimator/block_diagonal.py", line 375, in _finalize
    self._jaxpr = self._jaxpr_extractor(func_args)
  File "/home/ktokarev/venv/ferminet/lib/python3.11/site-packages/kfac_jax/_src/tracer.py", line 466, in get_processed_jaxpr
    closed_jaxpr, _ = retrieve(func_args)
  File "/home/ktokarev/venv/ferminet/lib/python3.11/site-packages/kfac_jax/_src/tracer.py", line 432, in retrieve
    processed_jaxpr = ProcessedJaxpr.make_from_func(
  File "/home/ktokarev/venv/ferminet/lib/python3.11/site-packages/kfac_jax/_src/tracer.py", line 321, in make_from_func
    func = tgm.auto_register_tags(
  File "/home/ktokarev/venv/ferminet/lib/python3.11/site-packages/kfac_jax/_src/tag_graph_matcher.py", line 2067, in auto_register_tags
    graph = make_jax_graph(
  File "/home/ktokarev/venv/ferminet/lib/python3.11/site-packages/kfac_jax/_src/tag_graph_matcher.py", line 355, in make_jax_graph
    closed_jaxpr, out_shapes = jax.make_jaxpr(func, return_shape=True)(*func_args)
  File "/home/ktokarev/venv/ferminet/lib/python3.11/site-packages/kfac_jax/_src/optimizer.py", line 1682, in value_func
    out, _ = value_and_grad_func(*args, **kwargs)
  File "/mnt/storage/src/git/ferminet/ferminet/loss.py", line 267, in total_energy_jvp
    psi_primal, psi_tangent = jax.jvp(batch_network, primals, tangents)
  File "/mnt/storage/src/git/ferminet/ferminet/train.py", line 525, in <lambda>
    logabs_network = lambda *args, **kwargs: signed_network(*args, **kwargs)[1]
  File "/mnt/storage/src/git/ferminet/ferminet/networks.py", line 1507, in apply
    orbitals = orbitals_apply(params, pos, spins, atoms, charges)
  File "/mnt/storage/src/git/ferminet/ferminet/networks.py", line 1172, in apply
    ae, ee, r_ae, r_ee = construct_input_features(pos, atoms, ndim=options.ndim)
  File "/mnt/storage/src/git/ferminet/ferminet/networks.py", line 477, in construct_input_features
    jnp.linalg.norm(ee + jnp.eye(n)[..., None], axis=-1) * (1.0 - jnp.eye(n)))
  File "/home/ktokarev/venv/ferminet/lib/python3.11/site-packages/jax/_src/numpy/array_methods.py", line 1050, in op
    return getattr(self.aval, f"_{name}")(self, *args)
  File "/home/ktokarev/venv/ferminet/lib/python3.11/site-packages/jax/_src/numpy/array_methods.py", line 646, in _getitem
    return lax_numpy._rewriting_take(self, item)
  File "/home/ktokarev/venv/ferminet/lib/python3.11/site-packages/jax/_src/numpy/lax_numpy.py", line 11411, in _rewriting_take
    return _gather(arr, treedef, static_idx, dynamic_idx, indices_are_sorted,
  File "/home/ktokarev/venv/ferminet/lib/python3.11/site-packages/jax/_src/numpy/lax_numpy.py", line 11452, in _gather
    return expand_dims(y, indexer.newaxis_dims)
  File "/home/ktokarev/venv/ferminet/lib/python3.11/site-packages/jax/_src/numpy/lax_numpy.py", line 2521, in expand_dims
    return lax.expand_dims(a, axis)
jax._src.source_info_util.JaxStackTraceBeforeTransformation: TypeError: _broadcast_in_dim_staging_rule() got an unexpected keyword argument 'sharding'

The preceding stack trace is the source of the JAX operation that, once transformed by JAX, triggered the following exception.

--------------------

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ktokarev/venv/ferminet/bin/ferminet", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/mnt/storage/src/git/ferminet/bin/ferminet", line 39, in <module>
    app.run(main)
  File "/usr/lib/python3.11/site-packages/absl/app.py", line 308, in run
    _run_main(main, args)
  File "/usr/lib/python3.11/site-packages/absl/app.py", line 254, in _run_main
    sys.exit(main(argv))
             ^^^^^^^^^^
  File "/mnt/storage/src/git/ferminet/bin/ferminet", line 35, in main
    train.train(cfg)
  File "/mnt/storage/src/git/ferminet/ferminet/train.py", line 852, in train
    opt_state = optimizer.init(params, subkeys, data)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ktokarev/venv/ferminet/lib/python3.11/site-packages/kfac_jax/_src/optimizer.py", line 1031, in init
    return self._init(params, rng, batch, func_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ktokarev/venv/ferminet/lib/python3.11/site-packages/kfac_jax/_src/utils/staging.py", line 297, in decorated
    outs = func(instance, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ktokarev/venv/ferminet/lib/python3.11/site-packages/kfac_jax/_src/optimizer.py", line 999, in _init
    estimator_state=self.estimator.init(
                    ^^^^^^^^^^^^^^^^^^^^
  File "/home/ktokarev/venv/ferminet/lib/python3.11/site-packages/kfac_jax/_src/utils/misc.py", line 342, in wrapped
    return method(instance, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ktokarev/venv/ferminet/lib/python3.11/site-packages/kfac_jax/_src/curvature_estimator/block_diagonal.py", line 390, in init
    self.finalize(func_args)
  File "/home/ktokarev/venv/ferminet/lib/python3.11/site-packages/kfac_jax/_src/utils/misc.py", line 315, in finalize
    self._finalize(*args, **kwargs)
  File "/home/ktokarev/venv/ferminet/lib/python3.11/site-packages/kfac_jax/_src/curvature_estimator/block_diagonal.py", line 375, in _finalize
    self._jaxpr = self._jaxpr_extractor(func_args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ktokarev/venv/ferminet/lib/python3.11/site-packages/kfac_jax/_src/tracer.py", line 466, in get_processed_jaxpr
    closed_jaxpr, _ = retrieve(func_args)
                      ^^^^^^^^^^^^^^^^^^^
  File "/home/ktokarev/venv/ferminet/lib/python3.11/site-packages/kfac_jax/_src/tracer.py", line 432, in retrieve
    processed_jaxpr = ProcessedJaxpr.make_from_func(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ktokarev/venv/ferminet/lib/python3.11/site-packages/kfac_jax/_src/tracer.py", line 328, in make_from_func
    typed_jaxpr = jax.make_jaxpr(func)(*func_args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ktokarev/venv/ferminet/lib/python3.11/site-packages/kfac_jax/_src/tag_graph_matcher.py", line 1696, in __call__
    flat_output = self._flat_func(*flat_args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: _broadcast_in_dim_staging_rule() got an unexpected keyword argument 'sharding'

Side note: wouldn't it be more productive to pin down certain dependencies (e.g., all stuff related to jax and kfac) to the versions which are know to work, and update them deliberately when certain feature is required, or when person doing such update has the time to verify that the new set of dependencies actually works when installing everything from scratch? I guess it would save a lot of time for everyone: no need to guess working revisions, no need to fight a fallback of unexpected updates in 3rd party code in the middle of other things.

The same as #88
Do you have any workaround at this point? 👀

Side note: wouldn't it be more productive to pin down certain dependencies (e.g., all stuff related to jax and kfac) to the versions which are know to work, and update them deliberately when certain feature is required, or when person doing such update has the time to verify that the new set of dependencies actually works when installing everything from scratch? I guess it would save a lot of time for everyone: no need to guess working revisions, no need to fight a fallback of unexpected updates in 3rd party code in the middle of other things.

folx and kfac-jax use JAX internal APIs and so HEAD of this repository typically tracks HEAD of those and jax. This matches what we use in our own work. [Note that folx doesn't have any releases yet.]
I will take a look at pinning versions when I get time, but the ecosystem moves quickly that this has caused problems when we've had pinned versions in the past.

This is now fixed -- ensure you are using jax 0.4.37 or chex 0.1.88 (or both). Sorry for the breakage.

Thanks!
I wonder if it would be helpful to set up nightly CI builds for ferminet like it's done e.g. in chex: https://github.com/google-deepmind/chex/blob/master/.github/workflows/ci.yml#L9. But it would require keeping CI green, otherwise it would be useless. If you are interested I can submit PR for that.

I already have a change for fixing the github CI, which should be pushed soon.
[We also have internal CI which is always green, but that is testing against HEAD of the various dependencies]


How can I specify a List to be parsed from the command line?

Hi,
I have the following option :
 [Option('c', "countries", Required = true)]
 public IEnumerable<string> Countries { get; set; }

But I don't know how to fill the command from the command line arguments, I have tied :
-c ES;CO
-c ES -c CO
-c ES CO
Any help here, please :) ?
Thanks you!

Specifying a Separator worked for me, e.g.
[Option('c', "format", Required = true, Separator = ',')]
public IEnumerable Countries { get; set; }
and then:
-c ES,CO


(feat) reenable benches/end_to_end.rs

This benches were disabled becuase of the deprecation of Client and Server,
this pr is to reenable them with the new conn aproach

@seanmonstar @oddgrd  Ready for review !

Hey, that's cool that you figured it out! I can have a look at this in the coming week, but I'm a new contributor myself so you'll have to wait for Sean to sign off on this whenever he has time. :smile:

Hey Robertoskr. I had a look at this, and when I tried to run the benchmarks on your branch it seems some of the HTTP2 ones take quite long to finish. On my computer your branch took ~290 seconds to complete, but on the commit before these benchmarks were disabled they completed in ~80s. I left a couple of comments, but unfortunately I wasn't able to solve this myself so I can't be of much help here.
If you'd like you can have a look at my attempt, maybe it will help you. I was able to get it down to ~100s, but some of the HTTP2 benches are unstable. I might keep looking into this, if I figure it out I'll let you know. Good luck!

In my computer it runs the same as the previous version of them in 65 seconds

That's odd, because on my computer the http2 ones take a long time (the http1 ones run great):

Thank you for tackling this issue! #3109 was just merged, which re-enables most of these benchmarks, so I'll close this one.


all_gather operation fails in multi GPU setting

Bug description
I can gather tensors with all_gather in the test_epcoh_end. Then I can also compute things like similarity scores. But if I have an operation like     target = torch.eye(similarity_scores.shape[0]) it fails. What could be the reason ?
How to reproduce the bug
from torchmetrics.functional import pairwise_cosine_similarity
def test_epoch_end(self, outputs):

        A_emb = torch.cat([x["A_emb"] for x in outputs],dim=0)
        B_emb = torch.cat([x["B_emb"] for x in outputs],dim=0)

        all_A_emb = self.all_gather(A_emb).view(-1,item_emb.shape[1])
        all_B_emb = self.all_gather(B_emb).view(-1,user_emb.shape[1])

        similarity_scores = pairwise_cosine_similarity(all_A_emb, all_B_emb)

        target = torch.eye(similarity_scores.shape[0])

Code works fine without the pairwise_cosine_similarity part.

Error messages and logs
ProcessExitedException: process 0 terminated with signal SIGSEGV
Environment
Name: pytorch-lightning
Version: 1.9.4
Summary: PyTorch Lightning is the lightweight PyTorch wrapper for ML researchers. Scale your models. Write less boilerplate.
Home-page: https://github.com/Lightning-AI/lightning
Author: Lightning AI et al.
Author-email<EMAIL_ADDRESS>License: Apache-2.0
Location: /local_disk0/.ephemeral_nfs/envs/pythonEnv-8d432daf-ab0f-41af-9b09-5c6128436d80/lib/python3.9/site-packages
Requires: fsspec, tqdm, typing-extensions, numpy, lightning-utilities, torchmetrics, packaging, PyYAML, torch
Required-by:
More info
No response

Hi @shamanez
I took your snippet and integrated it into a minimal running example. Could you please change it to show how it fails (since you didn't post all code)?
import torch
from torch.utils.data import DataLoader, Dataset

from pytorch_lightning import LightningModule, Trainer

from torchmetrics.functional import pairwise_cosine_similarity

class RandomDataset(Dataset):
    def __init__(self, size, length):
        self.len = length
        self.data = torch.randn(length, size)

    def __getitem__(self, index):
        return self.data[index]

    def __len__(self):
        return self.len

class BoringModel(LightningModule):
    def __init__(self):
        super().__init__()
        self.layer = torch.nn.Linear(32, 2)

    def forward(self, x):
        return self.layer(x)

    def test_step(self, batch, batch_idx):
        loss = self(batch).sum()
        self.log("test_loss", loss)

    def test_epoch_end(self, _):
        A_emb = torch.rand(3, 5)
        B_emb = torch.rand(3, 5)
        all_A_emb = self.all_gather(A_emb).view(-1, 5)
        all_B_emb = self.all_gather(B_emb).view(-1, 5)

        similarity_scores = pairwise_cosine_similarity(all_A_emb, all_B_emb)
        target = torch.eye(similarity_scores.shape[0])
        print(target)

    def configure_optimizers(self):
        return torch.optim.SGD(self.layer.parameters(), lr=0.1)

def run():
    test_data = DataLoader(RandomDataset(32, 64), batch_size=2)
    model = BoringModel()
    trainer = Trainer(devices=2, accelerator="cpu")
    trainer.test(model, dataloaders=test_data)

if __name__ == "__main__":
    run()

Thanks a lot.
The only difference is

def run():
    test_data = DataLoader(RandomDataset(32, 64), batch_size=2)
    model = BoringModel()
    trainer = Trainer(devices=2, accelerator="ddp_notebook")
    trainer.test(model, dataloaders=test_data)

Update on this: I executed this on a databricks notebook. But when I executed this from the terminal changing the accelerator="ddp" everything worked.

Hey @shamanez
I actually don't believe you 😄
You reported that the issue was with Lightning version 1.9.3 in your original post, but here you are saying that you set accelerator="ddp", which is not a valid setting in this version. Have you mixed up something? I am glad that you found a solution, but it can't be this :))
Let me know if we should close the ticket or if there is anything left to do.

No. I said all gather fails in multi-GPU settings. It was in a databricks notebook. But when I executed it as a python script, it worked by changing the ddp_notebook. Maybe my lightning version was incorrect. Actually, it is the latest version.

If it is the latest version, then accelerator="ddp" or accelerator="ddp_notebook" will rase a ValueError, that the accelerator is not supported. You should set instead strategy="ddp", which is the right Trainer argument name.
What was the error? Can you post the full stack trace please?

oh my bad. It was Strategy I am using :).  Just in the minimal example, I
was using the accelerator. Sorry about that.  But yeah error happens when
only I run the code in a notebook.
On Tue, Mar 14, 2023 at 2:59 PM Adrian Wälchli @.***>
wrote:

If it is the latest version, then accelerator="ddp" or
accelerator="ddp_notebook" will rase a ValueError, that the accelerator
is not supported. You should set instead strategy="ddp", which is the
right Trainer argument name.
What was the error? Can you post the full stack trace please?
—
Reply to this email directly, view it on GitHub
https://github.com/Lightning-AI/lightning/issues/17042#issuecomment-1467228197,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEA4FGTGZN5E6CXFHH7LQ73W37GJLANCNFSM6AAAAAAVX5ZCOM
.
You are receiving this because you were mentioned.Message ID:
@.***>

Closing the issue now. DDP in notebooks is just very hit and miss. It's very difficult from the PyTorch side to support this reliably.
I suggest to do prototyping in notebooks but then move the code to a script. Or use the notebook to instrument a Python script.


How to make the package type stable?

Not sure how much of a concern this is but for example, the mean sea-level pressure is always a Float64
https://github.com/CliMA/CLIMAParameters.jl/blob/f779db6fa86f0bfce9de4ee31757fed1fe8b75ed/src/Planet/planet_parameters.jl#L47
while the total solar irradiance is always an Int
https://github.com/CliMA/CLIMAParameters.jl/blob/f779db6fa86f0bfce9de4ee31757fed1fe8b75ed/src/Planet/planet_parameters.jl#L46
This might introduce lots of type conversions if you're running a Float32 model, potentially impacting performance. Type conversions can completely kill performance during automatic differentiation and backpropagation.
So I'm guessing the onus is on the user, e.g. ClimateMachine.jl or Oceananigans.jl, to always perform the appropriate conversions?
One solution might be to add a parametric type (default is Float64) to the parameter sets that then takes care of all conversions:
abstract type AbstractParameterSet{FT} end

abstract type AbstractEarthParameterSet{FT} <: AbstractParameterSet{FT} end

AbstractEarthParameterSet(FT=Float64) = AbstractEarthParameterSet{FT}()

Planet.TSI(ps::AbstractEarthParameterSet{FT}) where FT  = FT(1362)
Planet.MSLP(ps::AbstractEarthParameterSet{FT}) where FT = FT(1.01325e5)

@simonbyrne and I had some discussions about this back when I was moving clima parameters from ClimateMachine.jl into CLIMAParameters.jl. We used to have FT as a type parameter, but there were several considerations that lead us to remove it. The big one that comes to mind is that many of the tests/drivers/experiments, especially those with multiple float types, using clima parameters got really messy. Not having FT in the parameter type allows us to have a single param_set defined at the top of the file--and that is not the case if FT is included. The trade-off is that we need to locally enforce type-stability, which is something we need to be cognizant of anyway.

Not having FT in the parameter type allows us to have a single param_set defined at the top of the file--and that is not the case if FT is included.

I don't quite understand the reasoning --- can you explain @charleskawczynski ? Users have to specify a floating point type for their experiment at some point in their script. What's the downside to requiring this statement when building parameters, as opposed to some other place?

I found the relevant PR where we made this decision.

I remember really struggling with the decision to remove it--I did not want to, and that's why I looped in @simonbyrne.
Maybe things have changed since then that could make including FT easier now. Or maybe I was just fatigued by the long process of extracting all of the parameters, and maybe it'd be easier to include FT now that they've all been extracted (from the original PlanetParameters).
It'd probably still be a significant effort, though. Personally, I don't feel too motivated as those PRs were some of the most difficult ones to decompose and get across the finish line.

Now that I'm thinking more about it, I think one of the reasons those PRs were difficult to merge was because of the way CI was configured at the time (all serially). Since many of the changes were across the test/experiment/driver files, it took seemingly forever to find out that the nth experiment failed. And it seemed like by the time I reached the last experiment, other PRs merged and I'd have to rebase (and re-apply changes to newly added code). It might be easier now with buildkite running most experiments in parallel, but we have even more experiments/tests now.

Thanks for the explanation @charleskawczynski. Didn't realize a decision has already been made to remove FT.
I guess I saw my suggestion as non-breaking since the default FT is Float64 so I think adding it wouldn't break anything in ClimateMachine.jl? Although I'm not familiar with ClimateMachine.jl so I can't say I suppose.

The big one that comes to mind is that many of the tests/drivers/experiments, especially those with multiple float types, using clima parameters got really messy.

Do you have a link to an example of this? I'm not sure why there would be many float types. Seeing some of the messy code would help me understand some of the reasons you posted in PR https://github.com/CliMA/ClimateMachine.jl/pull/893.

I guess I saw my suggestion as non-breaking since the default FT is Float64 so I think adding it wouldn't break anything in ClimateMachine.jl?

We mostly use FT-generic interfaces in ClimateMachine.jl. In several unit tests, we test multiple float types, which I believe is an open issue in ClimateMachine.jl.
Here are a few places where we'd need param_set = ... in just one file:

https://github.com/CliMA/ClimateMachine.jl/blob/master/test/Common/Thermodynamics/runtests.jl#L48
https://github.com/CliMA/ClimateMachine.jl/blob/master/test/Common/Thermodynamics/runtests.jl#L104
https://github.com/CliMA/ClimateMachine.jl/blob/master/test/Common/Thermodynamics/runtests.jl#L458
https://github.com/CliMA/ClimateMachine.jl/blob/master/test/Common/Thermodynamics/runtests.jl#L642
https://github.com/CliMA/ClimateMachine.jl/blob/master/test/Common/Thermodynamics/runtests.jl#L727
https://github.com/CliMA/ClimateMachine.jl/blob/master/test/Common/Thermodynamics/runtests.jl#L975
https://github.com/CliMA/ClimateMachine.jl/blob/master/test/Common/Thermodynamics/runtests.jl#L1094

With the existing approach, all we need is https://github.com/CliMA/ClimateMachine.jl/blob/master/test/Common/Thermodynamics/runtests.jl#L18.

We mostly use FT-generic interfaces in ClimateMachine.jl. In several unit tests, we test multiple float types, which I believe is an open issue in ClimateMachine.jl.

@charleskawczynski are you confirming that the suggested feature is non-breaking, or saying that it is breaking?
As far as I can tell, adding a default conversion to Float64 would primarily have the effect of promoting the few Int parameters to Float64 (leaving parameters already defined as Float64 unchanged).
This would then be a feature that package develops could choose to take advantage of --- or not.

@charleskawczynski are you confirming that the suggested feature is non-breaking, or saying that it is breaking?

If we added FT to a type parameter in CLIMAParameters.jl, we would need to make changes across ClimateMachine. So, in that sense, yes.

This would then be a feature that package developers could choose to take advantage of --- or not.

I guess I'd need to see an example of your suggestion. Feel free to open a tracer bullet PR 🚀

@charleskawczynski @ali-ramadhan has provided example code showing the implementation. The proposal is to add a type parameter FT to CLIMAParameters types. Parameters are converted to FT when the parameter functions are called; for example:
Planet.TSI(ps::AbstractEarthParameterSet{FT}) where FT  = FT(1362)
Planet.MSLP(ps::AbstractEarthParameterSet{FT}) where FT = FT(1.01325e5)

For packages that depend on CLIMAParameters like ClimateMachine, the effect of a default FT=Float64 is hopefully clear. For the parameters that are currently Float64 (like "MSLP"), there is no effect. For parameters that are currently Int (like "TSI") the addition of this default has the effect of promoting Int to Float64.
I don't think it's likely this is a breaking change. A type parameter FT with default Float64 is an optional feature. If a package like ClimateMachine doesn't want to use it because they would rather convert numbers internally, that options remains available. The only possibility of breaking behavior is if some part of ClimateMachine specifically assumes that Int parameters (like "TSI") are integers.

A type parameter FT with default Float64 is an optional feature.

How will users optionally create instances of AbstractEarthParameterSet{FT} subtypes?
julia> abstract type AbstractParameterSet{FT} end

julia> struct EarthParameterSet <: AbstractParameterSet{FT} end
ERROR: UndefVarError: FT not defined
Stacktrace:
 [1] top-level scope at REPL[2]:1

Hmm I see. Something like
julia> abstract type AbstractTypedParameterSet{FT} end

julia> const AbstractParameterSet = AbstractTypedParameterSet{Float64}
AbstractTypedParameterSet{Float64}

julia> struct EarthParameterSet <: AbstractParameterSet end

might work.

why does the abstract type need to be parameterized? any fall through is undefined (if there is no parameter one doesn't exist)

Something like .... might work

That might work, but it may also cause confusion. You could turn it around and create your own instances and locally overload methods:
import CLIMAParameters
abstract type AbstractParameterSet end
struct MyEarthParameterSet{FT} <: AbstractParameterSet end

CLIMAParameters.Planet.planet_radius(::MyEarthParameterSet{FT}) where {FT} = FT(10^3)

You just wouldn't be able to impose the type restriction on methods in CLIMAParameters.

I guess i don't know why there is an abstract earth and not just a concrete type earth

I think we wanted to be able to create custom Earth instances:
struct SmallEarth <: AbstractEarthParameterSet end
CLIMAParameters.Planet.planet_radius(::SmallEarth) = 10^3

so that it still inherits all other Earth parameters.

Closed by #79. (we no longer ship these types, we only define the values and users must define their own types).


[RFC] OpenSearch Data Format

User Pain point

Not Open Format Dataset: Currently, all the data indexed in the OpenSearch service. Users rely on the OpenSearch service to access and retrieve the dataset.
Lacking segregation of read and write operations : using single cluster serve Index traffic and query traffic.
Complex data management workflow: usually, user need to setup pipeline to (1) ingest data to S3, and then (2) ingest to OpenSearch and (3) then manage index lifecycle with customized rules.

Proposed Solution

The index themselves are encoded with Lucene format of each shard.
The metadata is in object store. The metadata also include skipping index such as mix/max for each shard.
No server need to be running on to maintain OpenSearch Index. Transactions are achieve using optimistic concurrency protocol.
User only need to launch server when run queries, and benefits of separately scaling compute and storage.

Technical Challenge

OpenSearch Data Format on Object Store

OpenSearch Data Format structure on object store
Metadata specification. Using transaction log to record actions. The actions include (1) Add/Remove (2) Metadata Change
Access Protocols

Optimistic concurrency protocols
Serializable isolation

Implement OpenSearch Data Format Writer/Reader as library
Implement Virtual Index in OpenSearch which attached to OpenSearch Data Format
Implement DSL query rewrite logic with skipping index.

Demo setup

Demo Streaming application

Write 1 shard to fs every 5s
Write skipping index for each shard.

Reuse searchable snapshot interface

restore unassigned shard from fs every 10s.

Rewrite DSL query WITH skipping index

Here is the demo video which covers the following topic:

OpenSearch Data Format proposed in this issue that remove hard dependency on OpenSearch cluster and separate read and write path
Virtual / External Index that makes data set on object store accessible to OpenSearch. Please find more details in https://github.com/opensearch-project/sql/issues/1080
Skipping Index that skip unnecessary shard load/read. Please find more details in https://github.com/opensearch-project/opensearch-spark/issues/2

https://github.com/opensearch-project/OpenSearch/assets/46505291/b2b71f27-3f55-4f31-815c-ede2df6e5aa2

this is very cool!  has any progress been made on the spark sql execution datasources side?


Resource entry activity_horizontal_margin is already defined.

activity_horizontal_margin
Error:(2) Resource entry activity_horizontal_margin is already defined.
S:\android_studio\workspace\MyApp\app\build\intermediates\exploded-aar\com.github.mrengineer13\snackbar\1.2.0\res\values-sw600dp-v13\values.xml
Error:(2, 2) Originally defined here.```

Sounds like you imported the wrong module. You need to import the snackbar module, not the sample.


feat: :sparkles: You can now edit the current workflow step

Description
Please include a summary of the change and which issue is fixed.
List any external dependencies that are required for this change.
Completes: AB#FILL_IN_YOUR_ISSUE_ID
Checklist:

[ ] I have performed a self-review of my own code.
[ ] I have linked my DevOps task using the AB# tag.
[ ] My code is easy to read, and comments are added where needed.
[ ] My code is covered by tests.

Great work Idris


How do you add validation for header fields?

How do you add validation for header fields?
Looking through the code it looks like this isn't supported.  If you can confirm this, I can work on a PR to add support for header scope validation.

Thx you for this new feature :)


Update .env

Thanks @alkanarslan! Looks like the tests are green 👍
Before we merge this, could you please also update the README.md and docker-stack.yaml files accordingly? See an example here: Update to 7.6.0.
You can point the badge in the README to this PR, like:
- [![Elastic Stack version](https://img.shields.io/badge/ELK-7.6.0-blue.svg?style=flat)](https://github.com/deviantony/docker-elk/issues/473)
+ [![Elastic Stack version](https://img.shields.io/badge/ELK-7.6.2-blue.svg?style=flat)](https://github.com/deviantony/docker-elk/issues/482)

@deviantony I think the integration with Travis CI may be broken, tests failures are not reported inside PRs anymore.

@antoineco RIP Travis. Something changed recently in our travis config? I'll investigate the repo settings.

Right, the CI status is back, cheers!


Delete Default Badge with D-Rating on Frontend Repo README.md

Action Required
Edit README.md and remove link pointing to the default badge (with Red D-rating)

let me quickly get on it


DSP-7062: Open Visibility on RowReader

Implementing some DSE only functionality requires this to be more
publically accessible. See DSP-7062

+1

Build against Scala 2.10 finished with success

Build against Scala 2.11 finished with success

Build against Scala 2.10 finished with success

Build against Scala 2.11 finished with success


Profit calculation is incorrect because the gas used varies on even and odd oracle blocks.

for slot five the gas varies
1,671,890 for odd oracle blocks
and
1,776,264 for even oracle blocks.
It seems that we need to adjust the profit calc to save the gas used in a different way.

Wait, seriously? I have no idea why that happens

yep , look at the gas usage for these 2 transactions
https://etherscan.io/tx/0x938a71c9c02a4a1fd561ff7f682e3a7f66e3b8efad04c3d7487f430e39f339be
https://etherscan.io/tx/0x8e5aeca09d9eb9e0679e2d68dbb1cfe5f61b8e2b1c637819843a8bcee1133be4
It is actually quite random. some use 100-110k more gas.
Do you think this is normal or some extra variable causing this?

aah could it be because the values for some data Ids have more numbers in them? like the TVL


Wrong link and alt text in the MITx Online footer image of edX LMS

The MITx Online logo image in the footer has the wrong link and alt text (example page)
Please:

[ ] Change the alt text to MITx Online
[ ] Change the href link to https://mitxonline.mit.edu/ or https://rc.mitxonline.mit.edu/ depending on platform

Current HTML

Image

I just realized, this isn't in the theme. It's part of the MFE.
Here's an example URL, which is served through the MFE: https://courses-qa.mitxonline.mit.edu/learn/course/course-v1:MITx+14.100x+3T2021/block-v1:MITx+14.100x+3T2021+type@sequential+block@b7cd91c5366f4fb1b577d660a151205f/block-v1:MITx+14.100x+3T2021+type@vertical+block@15501c3d139148cb98b70fdbb43051d8
I'd hope this is a configuration that can be overridden, but I'm not sure.


More Precise Phase Impact levels

Foundry darkness is in 0.05 level increments, but phase impact is in 0.1 increments. It would be nice if phase impact was adjusted at the same variable level as foundry lighting so you could have more control in how variable the phase increment is.

The Phase Impact value is a percentage applied as a multiplier to the effect the moon's phase has on the Darkness level. I think 10% increments will be fine here.


Have docs always display examples with the latest version

We made this a bit better by switching to using @^1.X

I just switched it to use the latest published build so should be good now.


Add multiple clustering resolution functionality

If you haven't already, please read through our contributing guidelines before opening your PR
What is the purpose of this PR?
Closes #1070.
How did you implement your changes
Add helper function to create a new clustering column in the cell table. Include in notebook 4 with a description of the necessary inputs.
Remaining issues
NA

The ark tests are still failing, but can you just review the newest changes so it's ready to go once we get that fixed?


mouse hover and keyboard select design conflict

What happened?
Working with your example.
There are 2 selections highlighted on the frontend , one with the keyboard selection and other with mouse hover
Affected version
Latest
Reproduction URL
https://codepen.io/niketpathak/pen/vYRKdNq
How to reproduce?

Type the letter 'b'
hover mouse over blue
now use down arrow key

Browsers
Firefox
Relevant log output
No response
Additional context

Hello, this is not a bug. This is by choice to reuse the styling.
The default css on hover can be overridden or rewritten as per your choice.
For example
.tt-suggestion:hover {
   background-color: yellow;
}

/* You can also override the tt-selected element to distinguish between the hovered and the selected suggestion
Don't forget to use higher specificity to ensure the desired styles are applied correctly.
*/


Broken text display

Description

DBeaver Version
CE 24.0.1
Operating System
win 10
Database and driver
No response
Steps to reproduce
No response
Additional context
No response

Hello @sequencerr,
Please provide more information regarding your issue.
Is this the first time you installed DBeaver, or were you using an older version and decided to update? If so, what version did you use before that? Was the issue present there?
Additionally, please try resetting UI settings to see if it helps. You can find related information here: https://github.com/dbeaver/dbeaver/wiki/Reset-UI-settings#reset-settings

Hi. I've just installed it for very first time using zip. Resetting doesn't help.

Hello @sequencerr
Maybe this tip will help you: https://github.com/dbeaver/dbeaver/issues/20123#issuecomment-1572614592

Dupe of #20123, #20190


Cameron/ers simple

Code Review Processes
New Feature Review
Before submitting a pull request for new review, make sure the following is done:

Design doc is created and posted here: [Insert Link]

README Checks

[] README has proper context for the reviewer to understand what the code includes, any important design considerations, and areas to pay more attention to

Code Checks

[] Add explanatory comments. If there is complex code that requires specific context or understanding, note that in a comment
[] Remove unncessary comments. Any comments that do not add additional context, information, etc. should be removed
[] Add natspec.
[] Scrub through the code for inconsistencies (e.g. removing extra spaces)
[] Ensure there are not any .onlys in spec files

Broader Considerations

[] Ensure variable, function and event naming is clear, consistent, and reflective for the scope of the code.
[] Consider if certain pieces of logic should be placed in a different library, module

coverage: 52.97% (-47.0%) from 100.0%
when pulling 951d799feacb30d5ef692b77dfbc8a0b08e64485 on cameron/ers-simple
into 02f88abf6dfc3954c2aa433c197835ff7a7a14b5 on master.


Add AtkListItemData and fix entirely unrelated struct size

As a note:

We change the size of AtkEventData to 0x28, because it copies the data inside AtkEventDispatcher_DispatchEvent using OWORDs, not QWORDs.
AtkListItemData is prepared and dispatched inside AtkComponentList_vf44.


Unable to Add Drop Shadow as Filter ignored

Description
I am using FFImageLoading to populate an imagebutton on a Xamarin forms project.
            var svgImageSource = SvgImageSource.FromResource("Listi.asjit.com.Svg.chat.svg", vectorHeight:1000, vectorWidth: 1000);
            Btn.Source = svgImageSource;

This loads the image fine.  But, I am trying to use the imagebutton as a floating button so would like a drop shadow.  Unfortunately, the drop shadow filter is being ignored
The SVG is :
<svg xmlns="http://www.w3.org/2000/svg"  version="1.1"  width="240" height="240" >
  <defs>
    <filter id="f3" x="0" y="0" width="200%" height="200%">
      <feOffset result="offOut" in="SourceAlpha" dx="20" dy="20" />
      <feGaussianBlur result="blurOut" in="offOut" stdDeviation="10" />
      <feBlend in="SourceGraphic" in2="blurOut" mode="normal" />
    </filter>
  </defs>
	<g fill="none" fill-rule="nonzero" stroke="none" stroke-width="1" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="10" stroke-dasharray="" stroke-dashoffset="0" font-family="none" font-weight="none" font-size="none" text-anchor="none" style="mix-blend-mode: normal" >
		<path d="M0,192v-192h192v192z" fill="none"/>
		<path d="M96,192c-53.01934,0 -96,-42.98066 -96,-96v0c0,-53.01934 42.98066,-96 96,-96v0c53.01934,0 96,42.98066 96,96v0c0,53.01934 -42.98066,96 -96,96z" fill="#00798c" filter="url(#f3)"  />
		<g id="original-icon" fill="#dbc2cf">
			<path d="M134.4,56h-76.8c-5.28,0 -9.552,4.32 -9.552,9.6l-0.048,86.4l19.2,-19.2h67.2c5.28,0 9.6,-4.32 9.6,-9.6v-57.6c0,-5.28 -4.32,-9.6 -9.6,-9.6zM91.2,75.2h9.6v9.6h-9.6zM91.2,94.4h9.6v19.2h-9.6z"/>
		</g>
	</g>
</svg>

Steps to Reproduce
Load SVG into image button;
Expected Behavior
Image should be displayed with a drop shadow
Actual Behavior
The image is displayed without the drop shadow.
Basic Information

Version with issue: <IP_ADDRESS>2
Platform: Xamarin/IOS

Screenshots

Reproduction Link / Code

Not yet implemented, duplicate of https://github.com/luberda-molinet/FFImageLoading/issues/1205


Past sponsors page

Yes - for past sponsors we don't even need the sponsorship type so far - we only had one sponsor per edition.
If we could have a logotype + short text it would be Perfect. Kind of like here:
https://www.ecommerceawards.london/sponsors/

Yes - for past sponsors we don't even need the sponsorship type so far - we only had one sponsor per edition.
If we could have a logotype + short text it would be Perfect. Kind of like here:
https://www.ecommerceawards.london/sponsors/

resolved via https://github.com/ecommerceberlin/ecommercegermanyawards.com/issues/79


fix a fetchgit problem

fetchgit does not only remove .git directory, but also .gitignore file

I'm pretty sure that's intentional, it removes everything git-related (also .gitattributes). Just set leaveDotGit = true if you need them.

I want to contribute a package that called ostinato, but I met some trouble.
src = fetchgit {
  url = "https://github.com/pstavirs/ostinato.git";
  rev = "414d89860de0987843295d149bcabeac7c6fd9e5";
  sha256 = "9c667844d2ef983e51f9dfbba3c41daf06ee66825ffbbf57cb434d9daf0c9acc";
};

When I tried to get source code through github, the error messages was appeared:
these derivations will be built:
  /nix/store/bir9aak05ifngs4spr586in3grp97q0m-ostinato-414d898.drv
  /nix/store/hj28sxs5202v4fdgqj7i49h21xbaawx6-ostinato-2015-12-24.drv
building path(s) ‘/nix/store/g2cv38xkz9wd3c1g3pp5wm6rdad8rq9q-ostinato-414d898’
exporting https://github.com/pstavirs/ostinato.git (rev 414d89860de0987843295d149bcabeac7c6fd9e5) into /nix/store/g2cv38xkz9wd3c1g3pp5wm6rdad8rq9q-ostinato-414d898
Initialized empty Git repository in /nix/store/g2cv38xkz9wd3c1g3pp5wm6rdad8rq9q-ostinato-414d898/.git/
remote: Counting objects: 405, done.
remote: Compressing objects: 100% (311/311), done.
remote: Total 405 (delta 189), reused 236 (delta 91), pack-reused 0
Receiving objects: 100% (405/405), 438.35 KiB | 324.00 KiB/s, done.
Resolving deltas: 100% (189/189), done.
From https://github.com/pstavirs/ostinato
 * branch            HEAD       -> FETCH_HEAD
Switched to a new branch 'fetchgit'
git revision is 414d89860de0987843295d149bcabeac7c6fd9e5
git human-readable version is -- none --
Commit date is 2015-12-24 14:16:35 +0530
removing `.git'...
output path ‘/nix/store/g2cv38xkz9wd3c1g3pp5wm6rdad8rq9q-ostinato-414d898’ should have r:sha256 hash ‘1k4s1jprska3rdbvzyszh9kfw1mg3p2a7fyzz58kx67gs927hrlw’, instead has ‘0hb78bq51r93p0yr4l1z5xlf1i666v5pa3zkdj7jmpb879kj05dx’
cannot build derivation ‘/nix/store/hj28sxs5202v4fdgqj7i49h21xbaawx6-ostinato-2015-12-24.drv’: 1 dependencies couldn't be built
error: build of ‘/nix/store/hj28sxs5202v4fdgqj7i49h21xbaawx6-ostinato-2015-12-24.drv’ failed

Do you have any idea?

Set sha256 = "0hb78bq51r93p0yr4l1z5xlf1i666v5pa3zkdj7jmpb879kj05dx"? I'm not sure if I understand the question..

Thank you very much, you solved my problem. :)


Could swift enums + index be made possible with @Persisted?

!!! MANDATORY TO FILL OUT !!!

Goals
It would be great to make it possible to add index to a swift enum property. (Or if someone can confirm we have no plan to support this)
Expected Results

Make it possible to use @Persisted(indexed: true) var property: EnumType with enum EnumType: String/Int/..., PersistableEnum.
Actual Results

An compile error was produced:
Referencing initializer 'init(wrappedValue:indexed:)' on 'Persisted' requires that 'EnumType' conform to '_Indexable'
Steps for others to Reproduce

Code Sample
class Entity: Object {
    @Persisted(indexed: true) var category: Category
    enum Category: String, PersistableEnum {
        case a, b, c, d
    }
}

Version of Realm and Tooling

Realm framework version: 10.12
Xcode version: 12.5.1
iOS/OSX version: iOS 14.7
Dependency manager + version: SwiftPackageManager

This should be pretty trivial to allow and not supporting it was just an oversight.

Thanks for your super quick response, that's what I thought.


Problem with the relational terminology

Hi! I'm looking for a Backbone plugin that adds a method to work with relations. And I was surprised and confused when reading your project documentation. Could you please help me to understand how this works?
The project home page says:

Backbone plugin to specify 1:1 & 1:N associations between Backbone Models.

You provide two types of relations: Backbone.One and Backbone.Many, so I assume they are 1:1 and 1:N respectively. Let's see the examples:
var Employee = Backbone.AssociatedModel.extend({
    relations: [
        {
            type: Backbone.One, //nature of the relationship
            key: 'manager', // attribute of Employee
            relatedModel: 'Employee' //AssociatedModel for attribute key
        }
    ],

This probably means each employee is associated with one manager. This is a kind of foreign key in relational database, and corresponds to one-to-many relation in the relational model, because each manager is associated with many employees. Is this what you mean?
Let's see the next example:
var Project = Backbone.AssociatedModel.extend({
  relations: [
      {
        type: Backbone.Many,//nature of the relation
        key: 'locations', //attribute of Project
        relatedModel:Location //AssociatedModel for attribute key
      }
  ],

This probably means each project is associated with many locations. I'm not sure why your project has many locations and what use case you demonstrate here, but this is probably what you mean because the key is locations, not location, and you also do this:
project1.get("locations").add(loc1);

This probably means we can add more locations to the project. While we also probably can add same locations to another projects. If this is true then this means each project can be associated with many locations, and each locations can be associated with many projects. This is called many-to-many in the relational model.
If I understand correctly, Backbone.One is one-to-many (or many-to-one to be more correct) relation, and Backbone.Many is many-to-many relation. This is usually specified as 1:N and N:M.
Am I right or wrong regarding to the meaning of your examples? If I'm right then don't you think your naming is a bit confusing? Why not to use traditional notation?

@artemrizhov  : Please look at the ER diagram provided with the help for the relationships http://dhruvaray.github.io/backbone-associations/docs/img/cd_example.png

@dhruvaray Thanks, the diagram. It is helpful. If so, I have to say that imo the example provided is very confusing. I'm still in doubt.

Project may have many locations, but location can not have many projects. How do you imagine this? What kind of projects the example is about? What is the location?
How do you locate a department in few locations?
Individual manager for each employee? Are you kidding? :) Let's better provide many dedicated managers to each employee ;)
Each department has one employee, really? What is your business about?
Who is those dependents? I thought employee is that man who is dependent. Is there even more dependent people?

I think you are reading the diagram wrong...


Limit Messages per Conversation

The current storage model of conversations require to limit the number of messages per conversation to ensure respecting the document size limit of MongoDB of 16 megabytes.
Assuming ~200 chars average per message and ~200 chars required for additional information a limit of 5000 messages seams to be a reasonable limit.

Just as explanation:
The Smarti conversation was meant to represent a section within a communication channel. This lead to the decision to store the messages with the conversation.
However when integrations do establish a 1:1 mapping between communication channels and Smarti conversations their is a risk that the amount of messages in those channels do overrun the storage limit of MongoDB.
To avoid this situations this issue will limit the number of messages/conversation and only store the 5000 most recent messages in Smarti
Depending on the goals future versions might change the storage model to directly represent Channels and their messages. With messages stored as single documents in an own collections. Conversations would than be an additional class that just refer their the channel and the messages they contain

The number of messages per conversation is configureable via smarti.storage.mongodb.maxConvMsg. The default is set to 5000

NOTE: This feature is also useful to avoid the memory issue while indexing public channels with manny messages (see #299 and #302)


[EPIC][CPU] Enable direct pack/unpack vectorization in IREE

This collects the tasks to enable direct pack/unpack vectorization by default.
### Tasks
- [ ] Land https://github.com/openxla/iree/pull/16692
- [ ] Finish vector input sizes inference from IR for pack op.
- [ ] Finish vector input sizes inference from IR for unpack op.
- [ ] Add support for static pack op vectorization without providing input vector size.
- [ ] Add support for static unpack op vectorization without providing input vector size.
- [ ] Deprecate DecomposePackUnPackOps pass.

The two prototypes I have for pack/unpack shape inference are https://github.com/openxla/iree/pull/16629 and https://github.com/openxla/iree/pull/16664