Transformer image approx 7GB in size, transformer pod takes a lot of time to start

/kind bug
What steps did you take and what happened:
Its not exactly a bug, just a functionally inefficiency.
I followed steps given here to package and push a transformer docker image (Dockerfile)
The image comes out to be ~7.5GB in size. This results in transformer pod taking a huge amount of time to start and hence inference service is not ready even after 15 min. This is not ideal for a production environment and a use case that I am trying to build.
What did you expect to happen:
Lightweight image for the transformer code which is fast to push to a registry and pull from a registry
What's the InferenceService yaml:
N/A
Anything else you would like to add:
pip install kserve takes the most amount of time. It installs a large number of packages. Maybe we could separate packages into smaller modules instead of installing everything at once?
I might be missing something. Help is greatly appreciated
Environment:

Istio Version: 1.14.6
Knative Version: 1.4.0
KServe Version: 0.10
Kubeflow version:
Minikube/Kind version:
Kubernetes version: (use kubectl version): 1.24.9
OS (e.g. from /etc/os-release): Ubuntu 22.04

@pallav0894 There has been many changes since 0.10 release. Kserve storage dependency is optional now #2700. Also there is a WIP PR which makes ray as optional dependency #2854. Please try the new custrom transformer dockerfile  which has much reduced size around ~1.5GB. Also, the custom transformer is used for demonstration purposes only.

Thanks for the quick answer @sivanantha321 . Let me try the new dockerfile

@pallav0894 another suggestion is to test docker history $name-of-the-image to figure out what layer is the biggest, maybe it is not kserve but something else.

Thanks a lot! It's working much better for me now.

Could you also point me to the Dockerfile used to build that image? It already has a model.py and an entrypoint which I would like to customize according to my use case.

 For others reading; I just did
FROM kserve/image-transformer:latest

COPY my-own-transformer.py custom_transformer/

ENTRYPOINT ["python", "-m", "custom_transformer.my-own-transformer.py"]


Refresh artist data if out of date

Fixes #42
A few notes:

I have gone with a 90 day threshold initially, hardcoded into constants so straightforward to change in one place
This should work transparently with existing cache data - if there is no last updated timestamp, it simply assumes it needs a refresh. This will naturally cause some slowness in the live environment because every request will have to happen in realtime when a user next hits each artist - so for those festivals still remaining during 2022 I would recommend running the warm cache script if that is a concern
I did end up recording the last updated timestamp separately for top/newest/setlist, in part because they come from different sources (Spotify vs Setlist.fm), but also because in some cases the data never seems to be available and it's probably useful for debugging
I did go to implement the requested flag you mentioned on #42 to force a refresh when running the warm cache script, but then found I had to touch quite a lot more code, so in the interests of keeping this PR easy to review I didn't include it here. Happy to follow up with that
I added 3x log entries per artist by logging out the last updated timestamps per artist. Whilst this adds a bit to the noise, I thought while the feature is new it's probably helpful to see.

Happy to make any tweaks required to improve. I've done quite a bit of testing locally but it's probably worth checking three key scenarios once deployed: a) that pre cache warm, any given festival still works properly albeit a little slower b) that running the cache warm script for a given festival works as expected and does a load of refreshing and c) that re-running the cache warm script immediately after for the same festival doesn't then bother trying to refresh data.

Got a more detailed review incoming, but as an overall design comment:

This will naturally cause some slowness in the live environment because every request will have to happen in realtime when a user next hits each artist
unfortunately we can't leave it like this. It's brutally slow for the fetches to happen real-time when someone is generating a lineup, especially since it's going to be every artist initially. Theoretically it could be left like it is right now, but that would require running through and refreshing every lineup/year once it was deployed which is kind of a large headache (people make playlists for past festivals more than you'd expect). And even if I did that, in 90 days some poor soul is going to be the first person to generate a festival lineup and since all of those artists will be last updated 90 days ago, the code will fetch new info for every single one.

I'm not sure the best way to differentiate the logic since I agree that it should be in the redis-helper layer. It's going to be kind of ugly, but you could add a bool flag into each of those gettop/getnew/getsetlisttracks functions that basically indicates whether or not it's 'allowed' to fetch new ones off the last updated date too, or only based on if it exists in the cache like it was previously. Something like allowFetchBasedOnUpdatedDate: bool = false in the function param. Then only the call in the warm cache script would have to be updated to pass true as the last arg.
Very open to alternatives, but I'd like to keep the live frontend requests restricted to only what's in the cache unless there's nothing there at all

Thanks for the detailed review! So for starters, I've just rebased on to dev following your #45

And now re-pushed with all the requested changes, bar the flag requested in your comment

Got a more detailed review incoming, but as an overall design comment:

This will naturally cause some slowness in the live environment because every request will have to happen in realtime when a user next hits each artist
unfortunately we can't leave it like this. It's brutally slow for the fetches to happen real-time when someone is generating a lineup, especially since it's going to be every artist initially. Theoretically it could be left like it is right now, but that would require running through and refreshing every lineup/year once it was deployed which is kind of a large headache (people make playlists for past festivals more than you'd expect). And even if I did that, in 90 days some poor soul is going to be the first person to generate a festival lineup and since all of those artists will be last updated 90 days ago, the code will fetch new info for every single one.

I'm not sure the best way to differentiate the logic since I agree that it should be in the redis-helper layer. It's going to be kind of ugly, but you could add a bool flag into each of those gettop/getnew/getsetlisttracks functions that basically indicates whether or not it's 'allowed' to fetch new ones off the last updated date too, or only based on if it exists in the cache like it was previously. Something like allowFetchBasedOnUpdatedDate: bool = false in the function param. Then only the call in the warm cache script would have to be updated to pass true as the last arg.
Very open to alternatives, but I'd like to keep the live frontend requests restricted to only what's in the cache unless there's nothing there at all

Will implement the flag as requested. I thought that would probably be a necessary change, hence bringing it to your attention, but given it was a bit messy as it touched a lot of files wanted to give the opportunity in case you preferred a background task that just cycles through.

Umm. Actually your suggestion of using the optional function argument makes this...really easy. Pushed now.
I will need to re-test everything locally as there have been a fair few changes, so you might want to hold off merging until I give the all-clear.

Just tested your changes locally and it seems pretty solid to me.

Ran warm cache script on an existing festival and it refresh all three track types for every artist since each last_updated key was nonexistent
Ran it again on the festival and zero new tracks were fetched
manually changed the top tracks last_updated key value in the cache for an artist to over three months ago and re-ran warm cache script and verified only top tracks were re-fetched for that artist
manually changed the new tracks last_updated key value in the cache for an artist to over three months ago and re-ran warm cache script and verified only new tracks were re-fetched for that artist
manually changed the setlist tracks last_updated key value in the cache for an artist to over three months ago and re-ran warm cache script and verified only setlist tracks were re-fetched for that artist
re-ran it one more time immediately and verified zero tracks fetched again.

Going to leave one comment on the redis format of the date but logic looks good

Changes done and squashed, plus rebased over your other change on dev. Ready to merge!


Cleanup + Logo Widget <- WIP

Realised I was setting up an individual resize event on each component, that was silly must have been tired don't know why I did that :S
Was causing some performance issues making testing almost impossible (like 40-50s wait time in between hot reloads) so I ripped it out and fed the same event through App.tsx into the other widgets so they now have access to the height and width property this way.
Also started implementing the Logo widget, still need to get alignment right (it's weird working with Victory doing this stuff), and to add an alpha banner of some kind.
Will finish Logo Widget in next PR

Nice, will check it out :D


Wifi Shield?

Would it be possible to use a WiFi shield instead of an ethernet one?
http://www.amazon.com/Arduino-WiFi-Shield/dp/B009M8BU50
Thoughts?

What about ESP8266?

I can't comment specifically about different chipsets because I don't really have any experience with them; all I can sensibly say is that the hardware you use doesn't really matter, as long as it comes with Arduino-compatible libraries (if you want to try to re-use the existing sketch), or if you're comfortable adapting the code yourself.
Hope that helps!


Upgrade Selenium version

Currently AET makes use of Selenium 3.141.
Upgrade it to version 4 or newer once some stable version is provided.

AFAIK, AET uses Selenium 3.8:
https://github.com/wttech/aet/blob/master/osgi-dependencies/selenium/pom.xml#L37
3.141 is the latest 3.X (major) version.

AFAIK, AET uses Selenium 3.8:
https://github.com/wttech/aet/blob/master/osgi-dependencies/selenium/pom.xml#L37
3.141 is the latest 3.X (major) version.


Can't use S3 Sdk - SignatureDoesNotMatch

Bug report

[x] I confirm this is a bug with Supabase, not with my own application.
[x] I confirm I have searched the Docs, GitHub Discussions, and Discord.

Describe the bug
I'm trying to use S3 Sdk to perform operations on Supabase Storage. But every Command that I try simple doesn't works and result in 403: SignatureDoesNotMatch error. But if I call to Minio endpoint instead it works.
To Reproduce
Run docker-compose.s3.yml with the following:

code
services:

  minio:
    image: minio/minio
    ports:
      - '9000:9000'
      - '9001:9001'
    environment:
      MINIO_ROOT_USER: supa-storage
      MINIO_ROOT_PASSWORD: secret1234
    command: server --console-address ":9001" /data
    healthcheck:
      test: [ "CMD", "curl", "-f", "http://minio:9000/minio/health/live" ]
      interval: 2s
      timeout: 10s
      retries: 5
    volumes:
      - ./volumes/storage:/data:z

  minio-createbucket:
    image: minio/mc
    depends_on:
      minio:
        condition: service_healthy
    entrypoint: >
      /bin/sh -c "
      /usr/bin/mc alias set supa-minio http://minio:9000 supa-storage secret1234;
      /usr/bin/mc mb supa-minio/stub;
      exit 0;
      "

  storage:
    image: supabase/storage-api:v1.3.1 #supabase/storage-api:v0.43.11
    depends_on:
      db:
        # Disable this if you are using an external Postgres database
        condition: service_healthy
      rest:
        condition: service_started
      imgproxy:
        condition: service_started
      minio:
        condition: service_healthy
    healthcheck:
      test:
        [
          "CMD",
          "wget",
          "--no-verbose",
          "--tries=1",
          "--spider",
          "http://localhost:5000/status"
        ]
      timeout: 5s
      interval: 5s
      retries: 3
    restart: unless-stopped
    environment:
      ANON_KEY: ${ANON_KEY}
      SERVICE_KEY: ${SERVICE_ROLE_KEY}
      POSTGREST_URL: http://rest:3000
      PGRST_JWT_SECRET: ${JWT_SECRET}
      DATABASE_URL: postgres://supabase_storage_admin:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:${POSTGRES_PORT}/${POSTGRES_DB}
      FILE_SIZE_LIMIT: 52428800
      STORAGE_BACKEND: s3
      GLOBAL_S3_BUCKET: stub
      GLOBAL_S3_ENDPOINT: http://minio:9000
      GLOBAL_S3_PROTOCOL: http
      GLOBAL_S3_FORCE_PATH_STYLE: true
      S3_PROTOCOL_ACCESS_KEY_ID: supa-storage
      S3_PROTOCOL_ACCESS_KEY_SECRET: secret1234
      AWS_ACCESS_KEY_ID: supa-storage
      AWS_SECRET_ACCESS_KEY: secret1234
      AWS_DEFAULT_REGION: stub
      FILE_STORAGE_BACKEND_PATH: /var/lib/storage
      TUS_USE_FILE_VERSION_SEPARATOR: true
      TENANT_ID: stub
      IS_MULTITENANT: false
      # TODO: https://github.com/supabase/storage-api/issues/55
      REGION: stub
      STORAGE_S3_REGION: stub
      ENABLE_IMAGE_TRANSFORMATION: "true"
      IMGPROXY_URL: http://imgproxy:5001
    volumes:
      - ./supabase/volumes/storage:/var/lib/storage:z

  imgproxy:
    image: darthsim/imgproxy:v3.8.0
    healthcheck:
      test: [ "CMD", "imgproxy", "health" ]
      timeout: 5s
      interval: 5s
      retries: 3
    environment:
      IMGPROXY_BIND: ":5001"
      IMGPROXY_USE_ETAG: "true"
      IMGPROXY_ENABLE_WEBP_DETECTION: ${IMGPROXY_ENABLE_WEBP_DETECTION}

Then I copy the following code from supabase storage tests:
  const s3Client = new S3Client({
    logger: console,
    forcePathStyle: true,
    region: "stub",
    endpoint: "http://localhost:5000/storage/v1/s3", // -> Don't work
    // endpoint: "http://localhost:9000", // -> It Works
    credentials: {
      accessKeyId: "supa-storage",
      secretAccessKey: "secret1234",
    },
  });

  const createBucketRequest = new CreateBucketCommand({
    Bucket: randomUUID(),
    ACL: "public-read",
  });

  const { Location } = await s3Client.send(createBucketRequest);
  console.log("Location", Location);

Current behaviour

Application Logs:
endpoints Initial EndpointParams: {
  "ForcePathStyle": true,
  "UseArnRegion": false,
  "DisableMultiRegionAccessPoints": false,
  "Accelerate": false,
  "DisableS3ExpressSessionAuth": false,
  "UseGlobalEndpoint": false,
  "UseFIPS": false,
  "Endpoint": "http://localhost:5000/storage/v1/s3",
  "Region": "stub",
  "UseDualStack": false,
  "UseS3ExpressControlEndpoint": true,
  "DisableAccessPoints": true,
  "Bucket": "a25af482-0a92-4036-aef3-1a9125389070"
}
endpoints evaluateCondition: isSet($Region) = true
endpoints evaluateCondition: booleanEquals($Accelerate, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: isSet($Endpoint) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, true) = false
endpoints evaluateCondition: isSet($Endpoint) = true
endpoints evaluateCondition: booleanEquals($Accelerate, true) = false
endpoints evaluateCondition: booleanEquals($UseFIPS, true) = false
endpoints evaluateCondition: isSet($Bucket) = true
endpoints evaluateCondition: substring($Bucket, 0, 6, true) = 389070
endpoints assign: bucketSuffix := 389070
endpoints evaluateCondition: stringEquals($bucketSuffix, --x-s3) = false
endpoints evaluateCondition: not(isSet($Bucket)) = false
endpoints evaluateCondition: isSet($Bucket) = true
endpoints evaluateCondition: substring($Bucket, 49, 50, true) = null
endpoints evaluateCondition: isSet($Bucket) = true
endpoints evaluateCondition: isSet($Endpoint) = true
endpoints evaluateCondition: not(isSet(parseURL($Endpoint))) = false
endpoints evaluateCondition: booleanEquals($ForcePathStyle, false) = false
endpoints evaluateCondition: isSet($Endpoint) = true
endpoints evaluateCondition: parseURL($Endpoint) = {
  "scheme": "http",
  "authority": "localhost:5000",
  "path": "/storage/v1/s3",
  "normalizedPath": "/storage/v1/s3/",
  "isIp": false
}
endpoints assign: url := {
  "scheme": "http",
  "authority": "localhost:5000",
  "path": "/storage/v1/s3",
  "normalizedPath": "/storage/v1/s3/",
  "isIp": false
}
endpoints evaluateCondition: stringEquals(getAttr($url, scheme), http) = true
endpoints evaluateCondition: aws.isVirtualHostableS3Bucket($Bucket, true) = true
endpoints evaluateCondition: booleanEquals($ForcePathStyle, false) = false
endpoints evaluateCondition: booleanEquals($ForcePathStyle, false) = false
endpoints evaluateCondition: substring($Bucket, 0, 4, false) = a25a
endpoints assign: arnPrefix := a25a
endpoints evaluateCondition: stringEquals($arnPrefix, arn:) = false
endpoints evaluateCondition: booleanEquals($ForcePathStyle, true) = true
endpoints evaluateCondition: aws.parseArn($Bucket) = null
endpoints evaluateCondition: uriEncode($Bucket) = a25af482-0a92-4036-aef3-1a9125389070
endpoints assign: uri_encoded_bucket := a25af482-0a92-4036-aef3-1a9125389070
endpoints evaluateCondition: aws.partition($Region) = {
  "dnsSuffix": "amazonaws.com",
  "dualStackDnsSuffix": "api.aws",
  "implicitGlobalRegion": "us-east-1",
  "name": "aws",
  "supportsDualStack": true,
  "supportsFIPS": true
}
endpoints assign: partitionResult := {
  "dnsSuffix": "amazonaws.com",
  "dualStackDnsSuffix": "api.aws",
  "implicitGlobalRegion": "us-east-1",
  "name": "aws",
  "supportsDualStack": true,
  "supportsFIPS": true
}
endpoints evaluateCondition: booleanEquals($Accelerate, false) = true
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: not(isSet($Endpoint)) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: not(isSet($Endpoint)) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: not(isSet($Endpoint)) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: isSet($Endpoint) = true
endpoints evaluateCondition: parseURL($Endpoint) = {
  "scheme": "http",
  "authority": "localhost:5000",
  "path": "/storage/v1/s3",
  "normalizedPath": "/storage/v1/s3/",
  "isIp": false
}
endpoints assign: url := {
  "scheme": "http",
  "authority": "localhost:5000",
  "path": "/storage/v1/s3",
  "normalizedPath": "/storage/v1/s3/",
  "isIp": false
}
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: stringEquals($Region, aws-global) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: isSet($Endpoint) = true
endpoints evaluateCondition: parseURL($Endpoint) = {
  "scheme": "http",
  "authority": "localhost:5000",
  "path": "/storage/v1/s3",
  "normalizedPath": "/storage/v1/s3/",
  "isIp": false
}
endpoints assign: url := {
  "scheme": "http",
  "authority": "localhost:5000",
  "path": "/storage/v1/s3",
  "normalizedPath": "/storage/v1/s3/",
  "isIp": false
}
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: not(stringEquals($Region, aws-global)) = true
endpoints evaluateCondition: booleanEquals($UseGlobalEndpoint, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: isSet($Endpoint) = true
endpoints evaluateCondition: parseURL($Endpoint) = {
  "scheme": "http",
  "authority": "localhost:5000",
  "path": "/storage/v1/s3",
  "normalizedPath": "/storage/v1/s3/",
  "isIp": false
}
endpoints assign: url := {
  "scheme": "http",
  "authority": "localhost:5000",
  "path": "/storage/v1/s3",
  "normalizedPath": "/storage/v1/s3/",
  "isIp": false
}
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: not(stringEquals($Region, aws-global)) = true
endpoints evaluateCondition: booleanEquals($UseGlobalEndpoint, false) = true
endpoints Resolving endpoint from template: {
  "url": "{url#scheme}://{url#authority}{url#normalizedPath}{uri_encoded_bucket}",
  "properties": {
    "authSchemes": [
      {
        "disableDoubleEncoding": true,
        "name": "sigv4",
        "signingName": "s3",
        "signingRegion": "{Region}"
      }
    ]
  },
  "headers": {}
}
endpoints Resolved endpoint: {
  "headers": {},
  "properties": {
    "authSchemes": [
      {
        "disableDoubleEncoding": true,
        "name": "sigv4",
        "signingName": "s3",
        "signingRegion": "stub"
      }
    ]
  },
  "url": "http://localhost:5000/storage/v1/s3/a25af482-0a92-4036-aef3-1a9125389070"
}
{
  clientName: "S3Client",
  commandName: "CreateBucketCommand",
  input: {
    Bucket: "a25af482-0a92-4036-aef3-1a9125389070",
    ACL: "public-read"
  },
  error: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
    at throwDefaultError (file:///home/kalleby/.cache/deno/npm/registry.npmjs.org/@smithy/smithy-client/3.0.1/dist-cjs/index.js:838:20)
    at file:///home/kalleby/.cache/deno/npm/registry.npmjs.org/@smithy/smithy-client/3.0.1/dist-cjs/index.js:847:5
    at de_CommandError (file:///home/kalleby/.cache/deno/npm/registry.npmjs.org/@aws-sdk/client-s3/3.583.0_1/dist-cjs/index.js:4748:14)
    at Object.runMicrotasks (ext:core/01_core.js:642:26)
    at processTicksAndRejections (ext:deno_node/_next_tick.ts:39:10)
    at runNextTicks (ext:deno_node/_next_tick.ts:48:3)
    at eventLoopTick (ext:core/01_core.js:175:21)
    at async file:///home/kalleby/.cache/deno/npm/registry.npmjs.org/@smithy/middleware-serde/3.0.0/dist-cjs/index.js:35:20
    at async file:///home/kalleby/.cache/deno/npm/registry.npmjs.org/@aws-sdk/middleware-signing/3.577.0/dist-cjs/index.js:226:18
    at async file:///home/kalleby/.cache/deno/npm/registry.npmjs.org/@smithy/middleware-retry/3.0.1/dist-cjs/index.js:320:38 {
    name: "SignatureDoesNotMatch",
    "$fault": "client",
    "$metadata": {
      httpStatusCode: 403,
      requestId: undefined,
      extendedRequestId: undefined,
      cfId: undefined,
      attempts: 1,
      totalRetryDelay: 0
    },
    Resource: "a25af482-0a92-4036-aef3-1a9125389070",
    Code: "SignatureDoesNotMatch",
    message: "The request signature we calculated does not match the signature you provided. Check your key and si"... 13 more characters
  },
  metadata: {
    httpStatusCode: 403,
    requestId: undefined,
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  }
}
SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
    at throwDefaultError (file:///home/kalleby/.cache/deno/npm/registry.npmjs.org/@smithy/smithy-client/3.0.1/dist-cjs/index.js:838:20)
    at file:///home/kalleby/.cache/deno/npm/registry.npmjs.org/@smithy/smithy-client/3.0.1/dist-cjs/index.js:847:5
    at de_CommandError (file:///home/kalleby/.cache/deno/npm/registry.npmjs.org/@aws-sdk/client-s3/3.583.0_1/dist-cjs/index.js:4748:14)
    at Object.runMicrotasks (ext:core/01_core.js:642:26)
    at processTicksAndRejections (ext:deno_node/_next_tick.ts:39:10)
    at runNextTicks (ext:deno_node/_next_tick.ts:48:3)
    at eventLoopTick (ext:core/01_core.js:175:21)
    at async file:///home/kalleby/.cache/deno/npm/registry.npmjs.org/@smithy/middleware-serde/3.0.0/dist-cjs/index.js:35:20
    at async file:///home/kalleby/.cache/deno/npm/registry.npmjs.org/@aws-sdk/middleware-signing/3.577.0/dist-cjs/index.js:226:18
    at async file:///home/kalleby/.cache/deno/npm/registry.npmjs.org/@smithy/middleware-retry/3.0.1/dist-cjs/index.js:320:38 {
  name: "SignatureDoesNotMatch",
  "$fault": "client",
  "$metadata": {
    httpStatusCode: 403,
    requestId: undefined,
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  Resource: "a25af482-0a92-4036-aef3-1a9125389070",
  Code: "SignatureDoesNotMatch",
  message: "The request signature we calculated does not match the signature you provided. Check your key and si"... 13 more characters
}

Supabase Storage Logs:
{
  "level": 40,
  "time": "2024-05-31T09:22:39.510Z",
  "pid": 1,
  "hostname": "eb7efc6421e9",
  "reqId": "req-2",
  "tenantId": "stub",
  "project": "stub",
  "type": "request",
  "req": {
    "region": "stub",
    "traceId": "req-2",
    "method": "PUT",
    "url": "/s3/a25af482-0a92-4036-aef3-1a9125389070/",
    "headers": {
      "host": "storage:5000",
      "x_forwarded_proto": "http",
      "x_forwarded_host": "localhost",
      "x_forwarded_port": "8000",
      "x_real_ip": "<IP_ADDRESS>",
      "content_length": "186",
      "content_type": "application/xml",
      "user_agent": "aws-sdk-js/3.583.0 ua/2.0 os/linux#<IP_ADDRESS>-microsoft-standard-WSL2 lang/js md/nodejs#20.11.1 api/s3#3.583.0",
      "accept": "*/*"
    },
    "hostname": "storage:5000",
    "remoteAddress": "<IP_ADDRESS>",
    "remotePort": 55906
  },
  "res": {
    "statusCode": 403,
    "headers": {
      "content_type": "application/xml; charset=utf-8",
      "content_length": "293"
    }
  },
  "responseTime": 7.990687999874353,
  "error": {
    "raw": "{\"metadata\":{},\"code\":\"SignatureDoesNotMatch\",\"httpStatusCode\":403,\"userStatusCode\":400}",
    "name": "Error",
    "message": "The request signature we calculated does not match the signature you provided. Check your key and signing method.",
    "stack": "Error: The request signature we calculated does not match the signature you provided. Check your key and signing method.\n    at Object.SignatureDoesNotMatch (/app/dist/storage/errors.js:107:41)\n    at Object.<anonymous> (/app/dist/http/plugins/signature-v4.js:36:36)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
  },
  "resources": [
    "/a25af482-0a92-4036-aef3-1a9125389070"
  ],
  "msg": "stub | PUT | 403 | <IP_ADDRESS> | req-2 | /s3/a25af482-0a92-4036-aef3-1a9125389070/ | aws-sdk-js/3.583.0 ua/2.0 os/linux#<IP_ADDRESS>-microsoft-standard-WSL2 lang/js md/nodejs#20.11.1 api/s3#3.583.0"
}

Expected behavior - Calling :9000 (Minio)

Application Logs:
endpoints Initial EndpointParams: {
  "ForcePathStyle": true,
  "UseArnRegion": false,
  "DisableMultiRegionAccessPoints": false,
  "Accelerate": false,
  "DisableS3ExpressSessionAuth": false,
  "UseGlobalEndpoint": false,
  "UseFIPS": false,
  "Endpoint": "http://localhost:9000/",
  "Region": "stub",
  "UseDualStack": false,
  "UseS3ExpressControlEndpoint": true,
  "DisableAccessPoints": true,
  "Bucket": "a9766c7a-e0ab-4a25-a5fe-ad7279813895"
}
endpoints evaluateCondition: isSet($Region) = true
endpoints evaluateCondition: booleanEquals($Accelerate, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: isSet($Endpoint) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, true) = false
endpoints evaluateCondition: isSet($Endpoint) = true
endpoints evaluateCondition: booleanEquals($Accelerate, true) = false
endpoints evaluateCondition: booleanEquals($UseFIPS, true) = false
endpoints evaluateCondition: isSet($Bucket) = true
endpoints evaluateCondition: substring($Bucket, 0, 6, true) = 813895
endpoints assign: bucketSuffix := 813895
endpoints evaluateCondition: stringEquals($bucketSuffix, --x-s3) = false
endpoints evaluateCondition: not(isSet($Bucket)) = false
endpoints evaluateCondition: isSet($Bucket) = true
endpoints evaluateCondition: substring($Bucket, 49, 50, true) = null
endpoints evaluateCondition: isSet($Bucket) = true
endpoints evaluateCondition: isSet($Endpoint) = true
endpoints evaluateCondition: not(isSet(parseURL($Endpoint))) = false
endpoints evaluateCondition: booleanEquals($ForcePathStyle, false) = false
endpoints evaluateCondition: isSet($Endpoint) = true
endpoints evaluateCondition: parseURL($Endpoint) = {
  "scheme": "http",
  "authority": "localhost:9000",
  "path": "/",
  "normalizedPath": "/",
  "isIp": false
}
endpoints assign: url := {
  "scheme": "http",
  "authority": "localhost:9000",
  "path": "/",
  "normalizedPath": "/",
  "isIp": false
}
endpoints evaluateCondition: stringEquals(getAttr($url, scheme), http) = true
endpoints evaluateCondition: aws.isVirtualHostableS3Bucket($Bucket, true) = true
endpoints evaluateCondition: booleanEquals($ForcePathStyle, false) = false
endpoints evaluateCondition: booleanEquals($ForcePathStyle, false) = false
endpoints evaluateCondition: substring($Bucket, 0, 4, false) = a976
endpoints assign: arnPrefix := a976
endpoints evaluateCondition: stringEquals($arnPrefix, arn:) = false
endpoints evaluateCondition: booleanEquals($ForcePathStyle, true) = true
endpoints evaluateCondition: aws.parseArn($Bucket) = null
endpoints evaluateCondition: uriEncode($Bucket) = a9766c7a-e0ab-4a25-a5fe-ad7279813895
endpoints assign: uri_encoded_bucket := a9766c7a-e0ab-4a25-a5fe-ad7279813895
endpoints evaluateCondition: aws.partition($Region) = {
  "dnsSuffix": "amazonaws.com",
  "dualStackDnsSuffix": "api.aws",
  "implicitGlobalRegion": "us-east-1",
  "name": "aws",
  "supportsDualStack": true,
  "supportsFIPS": true
}
endpoints assign: partitionResult := {
  "dnsSuffix": "amazonaws.com",
  "dualStackDnsSuffix": "api.aws",
  "implicitGlobalRegion": "us-east-1",
  "name": "aws",
  "supportsDualStack": true,
  "supportsFIPS": true
}
endpoints evaluateCondition: booleanEquals($Accelerate, false) = true
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: not(isSet($Endpoint)) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: not(isSet($Endpoint)) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: not(isSet($Endpoint)) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: isSet($Endpoint) = true
endpoints evaluateCondition: parseURL($Endpoint) = {
  "scheme": "http",
  "authority": "localhost:9000",
  "path": "/",
  "normalizedPath": "/",
  "isIp": false
}
endpoints assign: url := {
  "scheme": "http",
  "authority": "localhost:9000",
  "path": "/",
  "normalizedPath": "/",
  "isIp": false
}
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: stringEquals($Region, aws-global) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: isSet($Endpoint) = true
endpoints evaluateCondition: parseURL($Endpoint) = {
  "scheme": "http",
  "authority": "localhost:9000",
  "path": "/",
  "normalizedPath": "/",
  "isIp": false
}
endpoints assign: url := {
  "scheme": "http",
  "authority": "localhost:9000",
  "path": "/",
  "normalizedPath": "/",
  "isIp": false
}
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: not(stringEquals($Region, aws-global)) = true
endpoints evaluateCondition: booleanEquals($UseGlobalEndpoint, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: isSet($Endpoint) = true
endpoints evaluateCondition: parseURL($Endpoint) = {
  "scheme": "http",
  "authority": "localhost:9000",
  "path": "/",
  "normalizedPath": "/",
  "isIp": false
}
endpoints assign: url := {
  "scheme": "http",
  "authority": "localhost:9000",
  "path": "/",
  "normalizedPath": "/",
  "isIp": false
}
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: not(stringEquals($Region, aws-global)) = true
endpoints evaluateCondition: booleanEquals($UseGlobalEndpoint, false) = true
endpoints Resolving endpoint from template: {
  "url": "{url#scheme}://{url#authority}{url#normalizedPath}{uri_encoded_bucket}",
  "properties": {
    "authSchemes": [
      {
        "disableDoubleEncoding": true,
        "name": "sigv4",
        "signingName": "s3",
        "signingRegion": "{Region}"
      }
    ]
  },
  "headers": {}
}
endpoints Resolved endpoint: {
  "headers": {},
  "properties": {
    "authSchemes": [
      {
        "disableDoubleEncoding": true,
        "name": "sigv4",
        "signingName": "s3",
        "signingRegion": "stub"
      }
    ]
  },
  "url": "http://localhost:9000/a9766c7a-e0ab-4a25-a5fe-ad7279813895"
}
{
  clientName: "S3Client",
  commandName: "CreateBucketCommand",
  input: {
    Bucket: "a9766c7a-e0ab-4a25-a5fe-ad7279813895",
    ACL: "public-read"
  },
  output: { Location: "/a9766c7a-e0ab-4a25-a5fe-ad7279813895" },
  metadata: {
    httpStatusCode: 200,
    requestId: "17D4884674C7FD41",
    extendedRequestId: "dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8",
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  }
}
Location /a9766c7a-e0ab-4a25-a5fe-ad7279813895

Minio dashboard:

System information

OS: Windows + WSL2

Additional context
In the example provide I use the CreateBucketCommand but the same occours when I trying to upload Multipart documents, like the following snippet copied from Supabase docs:

Uploading to Supabase Storage with S3Client
export const handler: Deno.ServeHandler = async (req: Request, _info) => {
        const s3Client = new S3Client({
          logger: console,
          forcePathStyle: true,
          region: "stub",
          endpoint: "http://localhost:5000/storage/v1/s3",
          // endpoint: "http://localhost:9000",
          credentials: {
            accessKeyId: "supa-storage",
            secretAccessKey: "secret1234",
          },
        });

      const file = await req.blob();

      const uploader = new Upload({
        client: s3Client,
        params: {
          Bucket: "stub",
          Key: "mypdf.pdf"`,
          ContentType: "application/pdf",
          Body: file.stream(),
        },
      });

      uploader.on("httpUploadProgress", (progress) => {
        console.log(progress);
      });

      await uploader.done();

      console.log("Uploaded");

      return new Response(null, { status: 201 });
};

I can upload from Supabase dashboard or changing the endpoint to Minio directly, but using S3 SDK on supabase storage, always returns me the Signature Error.  I'm also had try the older version supabase/storage-api:v0.43.11 but gives me 404 Errors

are you running storage behind a reverse proxy?

If you are not, don't set the S3_PREFIX_URL, and the URL you are using to connect to storage shouldn't contain the storage/v1 use http://localhost:5000/s3 for example
const s3Client = new S3Client({
    logger: console,
    forcePathStyle: true,
    region: "stub",
    endpoint: "http://localhost:5000/s3",
    // endpoint: "http://localhost:9000",
    credentials: {
      accessKeyId: "supa-storage",
      secretAccessKey: "secret1234",
    },
  });

are you running storage behind a reverse proxy?
If yes, you'd need to add the S3_PROTOCOL_PREFIX=/storage/v1 environment variable

I'm running the default self-host template from supabase repo.
That uses kong as API Gateway. I just cloned the repo and tried to call with @aws-sdk
Following your instructions I'd added the follwoing to my docker-compose.s3.yml file
# ...
S3_PROTOCOL_PREFIX: /storage/v1 # Added
# ...
S3_PROTOCOL_ACCESS_KEY_ID: supa-storage
S3_PROTOCOL_ACCESS_KEY_SECRET: secret1234
AWS_ACCESS_KEY_ID: supa-storage
AWS_SECRET_ACCESS_KEY: secret1234

and then request with:
  const s3Client = new S3Client({
    logger: console,
    forcePathStyle: true,
    region: "stub",
    endpoint: "http://localhost:5000/storage/v1/s3",
    credentials: {
      accessKeyId: "supa-storage",
      secretAccessKey: "secret1234",
    },
  });

  const createBucketRequest = new CreateBucketCommand({
    Bucket: "test-bucket",
    ACL: "public-read",
  });

But still got the same error.

Storage container logs
{
  "level": 40,
  "time": "2024-06-04T11:41:52.996Z",
  "pid": 1,
  "hostname": "6acfe7891859",
  "reqId": "req-1",
  "tenantId": "stub",
  "project": "stub",
  "type": "request",
  "req": {
    "region": "stub",
    "traceId": "req-1",
    "method": "PUT",
    "url": "/s3/test-bucket/",
    "headers": {
      "host": "storage:5000",
      "x_forwarded_proto": "http",
      "x_forwarded_host": "localhost",
      "x_forwarded_port": "8000",
      "x_real_ip": "<IP_ADDRESS>",
      "content_length": "186",
      "content_type": "application/xml",
      "user_agent": "aws-sdk-js/3.583.0 ua/2.0 os/linux#<IP_ADDRESS>-microsoft-standard-WSL2 lang/js md/nodejs#20.11.1 api/s3#3.583.0",
      "accept": "*/*"
    },
    "hostname": "storage:5000",
    "remoteAddress": "<IP_ADDRESS>",
    "remotePort": 39424
  },
  "res": {
    "statusCode": 403,
    "headers": {
      "content_type": "application/xml; charset=utf-8",
      "content_length": "268"
    }
  },
  "responseTime": 1094.2835690006614,
  "error": {
    "raw": "{\"metadata\":{},\"code\":\"SignatureDoesNotMatch\",\"httpStatusCode\":403,\"userStatusCode\":400}",
    "name": "Error",
    "message": "The request signature we calculated does not match the signature you provided. Check your key and signing method.",
    "stack": "Error: The request signature we calculated does not match the signature you provided. Check your key and signing method.\n    at Object.SignatureDoesNotMatch (/app/dist/storage/errors.js:107:41)\n    at Object.<anonymous> (/app/dist/http/plugins/signature-v4.js:36:36)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
  },
  "resources": [
    "/test-bucket"
  ],
  "msg": "stub | PUT | 403 | <IP_ADDRESS> | req-1 | /s3/test-bucket/ | aws-sdk-js/3.583.0 ua/2.0 os/linux#<IP_ADDRESS>-microsoft-standard-WSL2 lang/js md/nodejs#20.11.1 api/s3#3.583.0"
}

Right! if you use kong you'll need to add this snippet in the storage route:
https://github.com/supabase/cli/blob/develop/internal/start/templates/kong.yml#L111-L115

Just be more specific, you'll need to update this line on the supabase repo you have linked: https://github.com/supabase/supabase/blob/master/docker/volumes/api/kong.yml#L64
The host will be the host you are accessing Kong, in development will be localhost and the port will be the port you are accessing kong, ex: 5000
@kallebysantos let me know if this works for you, I'm quite confident it will since we use the same setup on the supabase cli

Just be more specific, you'll need to update this line on the supabase repo you have linked: https://github.com/supabase/supabase/blob/master/docker/volumes/api/kong.yml#L64
The host will be the host you are accessing Kong, in development will be localhost and the port will be the port you are accessing kong, ex: 5000
@kallebysantos let me know if this works for you, I'm quite confident it will since we use the same setup on the supabase cli

Thanks
I'll try to redo everything in a new fresh installation of supabase.

So I need to update both: storage and auth sections right?
only the storage one.

It will work since we use the same setup on the supabase cli.
I'll be closing this issue for now, however if you need need any more help on this, please feel free to comment below and I can re-open in case.

Hey @fenos
I'm self-hosting, and tried to follow the steps you wrote above. Am seeing the SignatureDoesNotMatch error:
{
    "raw": "{\"metadata\":{},\"code\":\"SignatureDoesNotMatch\",\"httpStatusCode\":403,\"userStatusCode\":400}",
    "name": "Error",
    "message": "The request signature we calculated does not match the signature you provided. Check your key and signing method.",
    "stack": "Error: The request signature we calculated does not match the signature you provided. Check your key and signing method.\n    at Object.SignatureDoesNotMatch (/app/dist/internal/errors/codes.js:140:39)\n    at Object.<anonymous> (/app/dist/http/plugins/signature-v4.js:72:34)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
  }

Standard and Resumable uploads work.
At one point I was getting the following error:

Missing S3 Protocol Access Key ID or Secret Key Environment variables

But that seems to have been my mistake with restarting the containers properly, because I'm not seeing this anymore. Now I get the SignatureDoesNotMatch error.
Any pointers on what I could be doing wrong?
Is there anything more I can share that could help?
Storage config
These are the environment variables for the storage container:
STORAGE_BACKEND: s3

# Removing this seems to have no effect
# S3_PROTOCOL_PREFIX: /storage/v1
S3_PROTOCOL_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
S3_PROTOCOL_ACCESS_KEY_SECRET: ${AWS_SECRET_ACCESS_KEY}

AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}

AWS_DEFAULT_REGION: ${S3_REGION}
REGION: ${S3_REGION}

GLOBAL_S3_BUCKET: ${S3_GLOBAL_S3_BUCKET}

TENANT_ID: ${TENANT_ENVIRONMENT}
IS_MULTITENANT: "false"

TUS_URL_PATH: /upload/resumable
TUS_URL_EXPIRY_MS: 86400000
UPLOAD_SIGNED_URL_EXPIRATION_TIME: 86400000

Kong config
My kong config for the storage-v1 path is as follows:
Notice the /storage/v1 in "Forwarded: host=$(headers.host)/storage/v1;proto=http".
## Storage routes: the storage server manages its own auth
- name: storage-v1
  _comment: "Storage: /storage/v1/* -> http://storage:5000/*"
  connect_timeout: 60
  write_timeout: 3600
  read_timeout: 3600
  url: http://storage:5000/
  routes:
    - name: storage-v1-all
      strip_path: true
      paths:
        - /storage/v1/
      request_buffering: false
      response_buffering: false
  plugins:
    - name: cors
    - name: request-transformer
      config:
        add:
          headers:
            - "Forwarded: host=$(headers.host)/storage/v1;proto=http"


Click handler and hover class?

Hello, thank you for the library. Do you have any plans to upgrade it to input element?

Hello, it's not really something I was planning on adding as I think there are other libraries that do it and maintaining it woulnd't be a priority, but happy to accept contributions :)


Fix

File 15. Decision_Tree.ipynb, regarding PSO, looks both lb and ub only need array of 3 element bcs we have 3 hyper parameters (max_leaf_nodes, min_samples_split, max_depth)

File 17. Neural Network.ipynb:

train and evaluation section, I have replaced targets.to(targets) with targets.to(device)
transfer model into GPU too.  model.to(device)
looks it is necessary to convert these arrays ( acc_train_hist, acc_valid_hist ) into CPU before use them in plot

Thanks for your response
I will consider it tomorrow. Thank you
On Wed, Jun 19, 2024, 11:26 PM Amirali Sahraei @.***>
wrote:

File 15. Decision_Tree.ipynb, regarding PSO, looks both lb and ub only
need array of 3 element bcs we have 3 hyper parameters (max_leaf_nodes,
min_samples_split, max_depth)
2.
File 17. Neural Network.ipynb:

train and evaluation section, I have replaced targets.to(targets)
with targets.to(device)
transfer model into GPU too. model.to(device)
looks it is necessary to convert these arrays ( acc_train_hist,
acc_valid_hist ) before use them in plot

You can view, comment on, or merge this pull request online at:
https://github.com/Ebimsv/Machine_Learning_Course/pull/2
Commit Summary

8f5d2a2
https://github.com/Ebimsv/Machine_Learning_Course/pull/2/commits/8f5d2a23d18b262c2ccb602ec9f271b8c3a76f0a
an array with 3 elements would be enough
de489c0
https://github.com/Ebimsv/Machine_Learning_Course/pull/2/commits/de489c0bae71df20b09a1c20f9a179b9e39bbafd
replace 'targets' with 'device' in devic(), also convert arrays from GPU to
CPU at the end to be able plot loss values

File Changes
(2 files https://github.com/Ebimsv/Machine_Learning_Course/pull/2/files)

M 15. Decision_Tree.ipynb
https://github.com/Ebimsv/Machine_Learning_Course/pull/2/files#diff-3df47af4d5aca0571a904221db673945cb1bb6dee30874c665b4bff09c021d7f
(4)
M 17. Neural Network.ipynb
https://github.com/Ebimsv/Machine_Learning_Course/pull/2/files#diff-98ba979767cade19518bcc8110a957922ff5388091ea6b6b1c58baa3fc1be26d
(27)

Patch Links:

https://github.com/Ebimsv/Machine_Learning_Course/pull/2.patch
https://github.com/Ebimsv/Machine_Learning_Course/pull/2.diff

—
Reply to this email directly, view it on GitHub
https://github.com/Ebimsv/Machine_Learning_Course/pull/2, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AMJG3NBE7XOPTX4Q6I3Z57DZIHPABAVCNFSM6AAAAABJSUAEJGVHI2DSMVQWIX3LMV43ASLTON2WKOZSGM3DGMBUGQYDEMY
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

Thank you for the contribution! Changes look good, merging now.


Clean up voting logs, and export to vote logger

Summary:

Clean up voting / election related logs. Moving duplicated ones to VLOG.
Add host:port to log statement for better readability
Add vote_logger.log() in three places (1) Election started, (2) Voting received, (3) Election decided.
Edit leader-election-test so that code can compile.

Test Plan:
Tested in D35510511

Looks good other than the ptr CHECK


grpc_cli: init at 1.46.6

Description of changes
The command line tool for interacting with grpc services.
https://github.com/grpc/grpc/blob/master/doc/command_line_tool.md
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Build is failing due to hash mismatch:
error: hash mismatch in fixed-output derivation '/nix/store/lc8niax8sggk6q3fxzpi01sf7j5h2d27-source.drv':
         specified: sha256-m55FKwqlAQ2Ge7y4RMFO4W1onu0NYARWrOygChJMxng=
            got:    sha256-UPenQh6+FBryQiOoeijsXkCZjlMzYljkg2aUtSFJFL4=

Still failing due to error: format not a string literal and no format arguments [-Werror=format-security], may have to disable some hardening.


Update conditionize.jquery.js

adding the ability to manage empty options, like placeholders

Thanks for contributing. :)


Failed to connect to static.verkstad.net port 80: Operation timed out

Downloading dependencies
Using InAppSettingsKit (2.4.4)
Installing OpenWebRTC (0.3.91)
[!] Error installing OpenWebRTC
[!] /usr/bin/curl -f -L -o /var/folders/l9/1pqqwd2161x02qtzjr3hp_5c0000gn/T/d20151204-21529-1fm76zh/file.zip http://static.verkstad.net/OpenWebRTC_0.3.91.zip
--create-dirs
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Dload  Upload   Total   Spent    Left  Speed
0     0    0     0    0     0      0      0 --:--:--  0:01:14 --:--:--     0curl: (7) Failed to connect to static.verkstad.net port 80: Operation timed out

The server is down at the moment, we are working on it.

Server is back up now. Closing.


Pin sqlalchemy<2 in CI environments

Looks like our test code is incompatible with API changes made in SQLAlchemy v2; this PR pins CI environments to sqlalchemy<2 to unblock CI. Since there aren't any immediate reasons to bump to using SQLAlchemy v2, I'm okay with maintaining this pinning versus updating the test code.
Closes https://github.com/dask-contrib/dask-sql/issues/1078

Codecov Report

Merging #1080 (0cdfa96) into main (6f0400c) will increase coverage by 0.11%.
The diff coverage is n/a.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@            Coverage Diff             @@
##             main    #1080      +/-   ##
==========================================
+ Coverage   81.04%   81.15%   +0.11%
==========================================
  Files          78       78
  Lines        4378     4378
  Branches      786      786
==========================================
+ Hits         3548     3553       +5
+ Misses        659      650       -9
- Partials      171      175       +4

see 1 file with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Rethink action mask API

Rethink action mask API
The current API for specifying actors is very confusing and poorly documented (I myself was very confused by it after not looking at it for a month after implementing it and forgetting how it works 😆). I think we can come up with something better.
Running example
Suppose we have an environment with two entity types, Worker and Resource, both with features x and y, a harvest action, and a move action. The observation and action spaces might look like this:
obs_space = {
  "resource": Entity(["x", "y"]),
  "worker": Entity(["x", "y"]),
}
action_space = {
  "harvest": CategoricalAction(["harvest", "don't harvest"]),
  "move": CategoricalAction(["up", "down", "left", "right"]),
}

Suppose that on a particular timestep, we have two resource entities r1 and r2, and three worker entities w1, w2, and w3. Only w1 and w3 can perform the harvest action on this timestep. All resources and workers can perform the move action, but w2 can't move down or left.
Observations
Observations are currently defined like this:
https://github.com/entity-neural-network/incubator/blob/c887f4a2f6601c67e8feb03ba9b16f27f74a223f/entity_gym/entity_gym/environment/environment.py#L82-L92
entities
The observation consists of a dictionary mapping each entity type to a numpy array with the features of all entities of that type:
entities = {
  "resource": [
    [10, 4], # r1
    [11, 6], # r2
  ],
  "worker": [
    [10, 4], # w1
    [0, 0],  # w2
    [11, 6], # w3
  ],
}

This seems fine, or at least I can't think of any obvious simplification or more ergonomic API.
ids
The ids are meant to allow the environment to designate an opaque identifier for each entity, which then makes it easy to later lookup that entity when executing actions chosen by the policy. For example, we could use the following list of strings to identify our entities:
ids = ["r1", "r2", "w1", "w2", "w3"],

In practice, this mostly adds runtime overhead and API complexity, and it's unclear whether it is actually particularly useful for most environments. One issue is also that the correspondence between the ids and entities is not obvious since it relies on the implicit ordering of entity types defined by the observation space.
action_masks
The environment must specify which entities can perform each action. This is done by returning an ActionMask for each action that specifies an actors list that specifies the indices of the entities that can perform the action:
action_masks = {
  "harvest": DenseCategoricalActionMask(actors=[2, 4]),
  "move": DenseCategoricalActionMask(
    actors=[0, 1, 2, 3, 4],
    mask=[
      [True, True, True, True],
      [True, True, True, True],
      [True, True, True, True],
      [True, False, False, True],
      [True, True, True, True],
    ],
  ),
}

The actors field is very unintuitive. In contrast to the observation, it's a single list rather than a list per entity, and it relies on the implicit ordering of entity types given by the observation space dictionary. The mask is also inconsistent with the ids, since it uses indices rather than the ids to specify actors.
action
Once we have sampled actions from the policy, we call the step method again and supply a action: Mapping[str, Action] with the choices made by the policy for each action. For categorical actions, this is represented by a actions: List[Tuple[EntityID, int]] which pairs each actor with the index of the chosen action:
actions = {
  "move": CategoricalAction([
    "r1": 0,
    "r2": 0,
    "w1": 2,
    "w2", 0,
    "w3", 3
  ]),
  "harvest": CategoricalAction([
    "w1": 0,
    "w3": 3,
  ]),
}

Solutions
The current API is pretty bad, not sure what the best solution is but here are some ideas:
Proposal 1: Group action_masks and ids by entity type
The indices would now be per entity, corresponding to the features in entities. This would make it a little more obvious what's going on. And in the common case of all entities of some type being able to execute an action, we wouldn't need to specify the actors field at all.
action_masks = {
  "worker": {
    "harvest": DenseCategoricalActionMask(actors=[0, 2]),
    "move": DenseCategoricalActionMask(
      mask=[
        [True, True, True, True],
        [True, False, False, True],
        [True, True, True, True],
      ],
    ),
  },
  "resource": {
    "move": DenseCategoricalActionMask(
      mask=[
        [True, True, True, True],
        [True, True, True, True],
      ],
    )
  },
}

ids = {
  "worker": ["w1", "w2", "w3"],
  "resource": ["r1", "r2"],
}

Proposal 2: Use EntityIDs to specify actors
Instead of indices, use EntiyIDs for actors:
action_masks = {
  "harvest": DenseCategoricalActionMask(actors=["w1", "w2"]),
  "move": DenseCategoricalActionMask(
    actors=["r1", "r2", "w1", "w2", "w3"],
    mask=[
      [True, True, True, True],
      [True, True, True, True],
      [True, True, True, True],
      [True, False, False, True],
      [True, True, True, True],
    ],
  ),
}

Proposal 3: Just get rid of ids
The EntityIDs haven't proved particularly useful so far. We could just get rid of the ids field and remove the EntityID from the actions passed to act. Something similar to the old interface that works with environment-supplied ids rather than indices could still be achieved with an env wrapper or helper methods.
actions = {
  "move": CategoricalAction([0, 0, 2, 0, 3]),
  "harvest": CategoricalAction([0, 3]),
}

If combined with (1):
actions = {
  "worker": {
    "move": CategoricalAction([2, 0, 3]),
    "harvest": CategoricalAction([0, 3]),
   },
  "resource": {
    "move": CategoricalAction([0, 0]),
  },
}

This seems fine, or at least I can't think of any obvious simplification or more ergonomic API.

Yeah, I agree. The observation scheme is perfectly fine to me :)

and it's unclear whether it is actually particularly useful for most environments

Would you mind elaborating on this?

One issue is also that the correspondence between the ids and entities is not obvious since it relies on the implicit ordering of entity types defined by the observation space.

I'd say the API should only rely on the unique IDs themselves. Each id exactly corresponds to a live entity. An edge case is if an entity is dead, we should still enforce different entity IDs because the dead and live entities sharing the same IDs may cause confusion to the training algorithm.
The implicit ordering seems problematic.

we wouldn't need to specify the actors field at all.

Did you mean
 action_masks = {
   "worker": {
-   "harvest": DenseCategoricalActionMask(actors=[0, 2]),
+   "harvest": DenseCategoricalActionMask(
+        mask=[
+            [True, True],
+            [True, True],
+        ]
+    ),
     "move": DenseCategoricalActionMask(
       mask=[
         [True, True, True, True],
         [True, False, False, True],
         [True, True, True, True],
       ],
     ),
   },
   "resource": {
     "move": DenseCategoricalActionMask(
       mask=[
         [True, True, True, True],
         [True, True, True, True],
       ],
     )
   },
 }

Proposal 3: Just get rid of ids

How does this handle the masking issues?

and it's unclear whether it is actually particularly useful for most environments

Would you mind elaborating on this?

I've found that in the environments I've implemented so far, entities can already naturally be iterated through in the same order as done to create features/action masks, so having the entity id for each action doesn't make the environment easier to implement since it's just ignored anyway, e.g.: https://github.com/entity-neural-network/incubator/blob/4001598e59e02276f398dd990d1f1ff42a180107/enn_zoo/enn_zoo/codecraft/vec_env.py#L433-L438

One issue is also that the correspondence between the ids and entities is not obvious since it relies on the implicit ordering of entity types defined by the observation space.

I'd say the API should only rely on the unique IDs themselves. Each id exactly corresponds to a live entity. An edge case is if an entity is dead, we should still enforce different entity IDs because the dead and live entities sharing the same IDs may cause confusion to the training algorithm.

Yeah, that's a good point I hadn't thought of before. Doesn't matter in the current implementation, if we want to implement something like per-entity memory, we need to be able to track identities across timesteps.

we wouldn't need to specify the actors field at all.

Did you mean
 action_masks = {
   "worker": {
-   "harvest": DenseCategoricalActionMask(actors=[0, 2]),
+   "harvest": DenseCategoricalActionMask(
+        mask=[
+            [True, True],
+            [True, True],
+        ]
+    ),
     "move": DenseCategoricalActionMask(
       mask=[
         [True, True, True, True],
         [True, False, False, True],
         [True, True, True, True],
       ],
     ),
   },
   "resource": {
     "move": DenseCategoricalActionMask(
       mask=[
         [True, True, True, True],
         [True, True, True, True],
       ],
     )
   },
 }

On the "move" action, the actors field can be omitted since all entities of that type are performing the action. On the "harvest" action, we set actors=[0, 2] to specify that only the first and third worker entity can perform the action, and the mask can be omitted since all choices are available.

Proposal 3: Just get rid of ids

How does this handle the masking issues?

Mostly it would just simplify the API for environments that don't care about getting an EntityID associated with each action. Selecting actors would be by index as done currently (but probably grouped by entity type).

My thoughts:
when I implemented the Griddly wrapper, I was very confused between EntityID and Index. It was unclear which one should be used where. I actually implemented my wrapper first using Proposal 2 (thinking this is how it was set up).
I then tried to make the API do the unwrapping back to IDs but this ended up being super problematic because keeping track of entity-to-id maps was complex especially in complex environments where the entities change per-frame. Essentially you would need to keep rebuilding this map.

I've found that in the environments I've implemented so far, entities can already naturally be iterated through in the same order as done to create features/action masks
Instead what I did was generate a unique hash for each entity and then always order by that hash. this way the index to unique ID is always consistent and it removes the requirement for there to be some mapping because the order of the mapping is the same as the index. I had to fix a bunch of stuff to get this to work (I'm fairly convinced this is bug free as i wrote a bunch of tests for it)

if we want to implement something like per-entity memory, we need to be able to track identities across timesteps.
Additionally we have to think about how this works when things get batched and have diffent numbers of entities in each observation. The indexes/masking still needs to be consistent across the whole batch using ragged buffers etc. I ended up writing some tests after fixing how this worked: https://github.com/entity-neural-network/incubator/blob/main/entity_gym/entity_gym/tests/test_environment.py

Proposal 1 does not fix the ordering issue within the groups.. so there still needs to be some way to match the order of masks to entities.
Proposal 2 has the issue of mapping that I came across above and while I'm sure this can be done, the environment still needs to provide a index mapping, or have one calculated per frame to be consistent with changable environments.
Proposal 3... again same issue with 1 and 2, the mapping still needs to be solved.
Proposal 4:
To make the environment super friendly and obvious (but probably bit less efficient) I would think and API for masks thats mapped per-entityID is the most intuitive to build as an environment integrator. It also relies on the mapping function in proposal 2... but means that the environments don't have to worry about the ordering or mapping function (because its hidden away internally):
a)
{
  "r1": {
    "move": [True, True, True, True]
  },
  "r2": {
    "move": [True, True, True, True]
  },
  "w1": {
    "move": [True, True, True, True],
    "harvest": [True, True]
  },
  "w2": {
    "move": [True, False, False, True]
  },
  "w3": {
    "move": [True, True, True, True],
    "harvest": [True, True]
  }
}

b)
{
  "move": {
    "r1": [True, True, True, True],
    "r2": [True, True, True, True],
    "w1": [True, True, True, True],
    "w2": [True, False, False, True],
    "w3": [True, True, True, True]
  },
  "harvest": {
    "w1": [True, True],
    "w3": [True, True]
  }
}

This

Enforces the requirement of the unique ID in the environment.
Makes it pretty impossible for the environment to supply incorrect indexed masks because it's all calculated underneath.
b) This matches the action API making things even easier to understand

Internally we can just order the unique ids every frame (this happens internally in Griddly) and then just keep that order consistent when we calculate the indexes.

Thanks for all the input! It sounds like we already have general consensus on two points:

We should keep EntityIDs and always use EntityIDs rather than indices
The ids should be grouped by entity type, same as entities

@Bam4d In Proposal 4, would you also group the entities by ID? So do something like this:
entities = {
  "resource": {
    "r1": [10, 4],
    "r2": [11, 6], # r2
  },
  "worker": {
    "w1": [10, 4],
    "w2": [0, 0],
    "w3": [11, 6],
  },
}

Then you wouldn't need the ids field since it's now merged with entities, and the entities work
I do think this would limit performance, especially for environments with many entities, but that's not necessarily a problem since we can still have a less ergonomic but more efficient alternative API.
The most ergonomic version I can think of that is still efficient would be to combine proposals 1 and 2:
ids = {
  "worker": ["w1", "w2", "w3"],
  "resource": ["r1", "r2"],
}

entities = {
  "resource": [
    [10, 4], # r1
    [11, 6], # r2
  ],
  "worker": [
    [10, 4], # w1
    [0, 0],  # w2
    [11, 6], # w3
  ],
}

action_masks = {
  "worker": {
    "harvest": DenseCategoricalActionMask(actors=["w1", "w3"]),
    "move": DenseCategoricalActionMask(
      # if actors is omitted, we implicitly have actors = ids["worker"] = ["w1", "w2", "w3"]
      mask=[
        [True, True, True, True],
        [True, False, False, True],
        [True, True, True, True],
      ],
    ),
  },
  "resource": {
    "move": DenseCategoricalActionMask(
      mask=[
        [True, True, True, True],
        [True, True, True, True],
      ],
    )
  },
}

This ought to be convenient for environments, e.g. I imagine in most cases you could do something like this:
for entity_type in entity_types:
  for id, entity in entities[entity_type]:
    for action_type in entity.action_types():
      action_masks[entity_type][action_type].append(entity.mask())
    ids[entity_type].append(id)
    features.append(entity.features())

Or this:
for entity_type in entity_types:
  for id, entity in entities[entity_type]:
    ids[entity_type].append(id)
    features.append(entity.features())
actors = []
masks = []
for id, entity in all_entities:
  if entity.can_act():
    actors.append(id)
    masks.append(entity.mask())

Quick nit: Why couldn't we just combine the DenseCategoricalActionMask(actors= and DenseCategoricalActionMask(mask= together?
The actors mask feels like a special case for mask. You would simply have an action space that look like
action_space = {
  "harvest": CategoricalAction(["noop", "harvest"]),
  "move": CategoricalAction(["noop", "up", "down", "left", "right"]),
}

and if the entity cannot perform an action, we simply mask out everything except noop. In gym-micrrorts, we adopted this paradigm.

That's a good point. If the actors are very sparse, there's might be some efficiency gains, but they're probably fairly minor, especially if we're going to partition the masks by entity type.
I'm still leaning towards keeping the actors since it might still be the more natural/efficient API in some cases, but make it so you can also specify just the mask (or neither) and have that be the default.

I've created a draft of a revised API, which I'm thinking of as a starting point for further discussion: https://github.com/entity-neural-network/incubator/pull/166
It includes a new tutorial on how to implement an Environment. Let me know what you think!
Some high-level comments:

The API doesn't stray too far from the previous structure, with the main difference being that Environment does not deal with indices anymore and everything uses EntityIDs instead. It would now be very straightforward though to add alternative constructors for Observation that expose an interface more similar to e.g. "Proposal 4" and just do a little bit of straightforward translation.
The Environment is now optimized for an ergonomic interface, and VecEnv is optimized for an efficient interface. I tried to find some middle ground for some time (e.g. similar to previous VecEnv where all features are combined in a ragged buffer, but we still use EntityIDs as opposed to indices) but what I was able to come up with was neither efficient nor ergonomic and also just kind of inconsistent. The clear separation between efficient and ergonomic interfaces we have now seems cleaner, and since most people won't have to implement VecEnv themselves it should be fine for that interface to be a bit more complicated.
There are still no provisions for tracking entities across timesteps. I think this will add quite a bit of additional complexity (for implementation and VecEnv interface, but not the Environment interface)  and the correct data structures aren't clear to me yet. This will likely be a substantial project of its own and is probably best tackled at a later time.

Hey, @cswinter this new API mask sense to me. The tutorial is super helpful. A couple of nit questions:

Do the ids have to be integers?
Do the ids have to be sequential (i.e. can it be 4, 7, 1, -4)

Do the ids have to be integers?

From the tests, this is not the case, the IDs in the tests are strings.

The IDs have to obey the following constraints (should probably be added to docs):

Serializable by numpy_msgpack
Hashable
Unique (on that timestep)
So e.g. something like Tuple[str, int] also works

Unique (on that timestep)

Not sure if this is true. what happens if several entities share the same ID across a batch, the indexing will become corrupted when merging/extending Observations/Batches? It definitely did before. How are we handling this case now?

There are still no provisions for tracking entities across timesteps. I think this will add quite a bit of additional complexity (for implementation and VecEnv interface, but not the Environment interface)  and the correct data structures aren't clear to me yet. This will likely be a substantial project of its own and is probably best tackled at a later time.

Is this not handled by the batching operations? As long as the IDS are unique for entities, things get ordered correctly. Maybe need to write a test to prove this.

The EntityID's are tracked separately for each environment/timestep. So say we have two environments, we might end up with something like this:
ids= [
    {"robot": [("Robot", 0)], "mine": [("Mine", 1)]},
    {"robot": [("Robot", 0), ("Robot", 1)], "mine": [("Mine", 1)]},
]
actor_ids = [
    {"move":  [("Robot", 0)]},
    {"move":  [("Robot", 0), ("Robot", 1)]},
]

Assuming that the obs space is ordered ["mine", "robot"], we then end up with the following mapping of IDs to indices:
id_to_idx = [
    {
        ("Mine", 0): 0,
        ("Robot", 0): 1,
    },
    {
        ("Mine", 4): 0,
        ("Robot", 0): 1,
        ("Robot", 1): 2,
    },
]
actor_idx = {
    "move": [
        [1],
        [1, 2],
    ]
}
# RaggedI64 for index of entity corresponding to `ids` above, this is not actually materialized by the ` VecEnv` since it's implicit in the shape of the features
idx = [
    [0, 1],
    [0, 1, 2],
]

The indices are unique per sequence but not unique per batch since we are still going to join/shuffle the sequences anyway. We do eventually calculate flat indices that are unique per batch during the network forward pass by adding the cumulative number of entities to each sequence (this happens in rogue_net). The VecEnv doesn't have to care about this since it all happens outside of the environment interface:
offsets = [0, 2]
flat_actor_idx = {
    "move": [1, 3, 4]
}
flat_idx = [0, 1, 2, 3, 4]

There are still no provisions for tracking entities across timesteps. I think this will add quite a bit of additional complexity (for implementation and VecEnv interface, but not the Environment interface)  and the correct data structures aren't clear to me yet. This will likely be a substantial project of its own and is probably best tackled at a later time.

Is this not handled by the batching operations? As long as the IDS are unique for entities, things get ordered correctly. Maybe need to write a test to prove this.

So at the level of Environment, we do have the information to construct the right data structures, but the information is erased by VecEnv, which only retains indices and not IDs. We'd probably need to construct a second set of indices here that maps each entity to a slot in a "memory bank" that persists information across timesteps, but until we know how this would be implemented by the network architecture, it's a little unclear as to how exactly that should look like.

Right ok makes more sense now. we don't actually know which entities from which environment when we are at the batch level... but can assume that the indexes acrosss the sequence is consistent.

Most of the ideas discussed here have been incorporated into https://github.com/entity-neural-network/incubator/pull/166. If we want to make any additional changes, it will be easier to keep track of those in new issues.


bump version to take this prod

smartcitiesdata/discovery_ui#177
co-authored-by: Kenny Drobnack<EMAIL_ADDRESS>

Used for hotfix, closing now


Dedicated Service account / Cluster Role for Tiller Deploy

Currently Tiller Deploy is running as a cluster admin through the default service account.
This PR gives the tiller deploy its own service account binded to a cluster role with only the correct permission scope (taken from here).
Resolves #878
For the future, it would be worth checking if anything else is running with the default service account binded to a role with more permissions than it needs.

@mumoshu
The old PR's were a mess due to the wrong email being assigned to a commit.
I've fixed the issue and opened a fresh PR. Apologies for the messing around

Codecov Report

Merging #1202 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master   #1202   +/-   ##
======================================
  Coverage    36.2%   36.2%
======================================
  Files          63      63
  Lines        3823    3823
======================================
  Hits         1384    1384
  Misses       2224    2224
  Partials      215     215

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 62f93cd...f707d01. Read the comment docs.


chore: changeset version

Motivation
Need the latest release to start regenerating casts with the new number of mentions, as well as byte-based mentionsPositions.
Change Summary
Ran yarn changeset version
Merge Checklist
Choose all relevant options below by adding an x now or at any time before submitting for review

[x] The title of this PR adheres to the conventional commits standard
[x] The PR has been tagged with the appropriate change type label(s) (i.e. documentation, feature, bugfix, or chore)

Successfully published:
⨠ yarn changeset publish
yarn run v1.22.19
$ ~/src/hubble/node_modules/.bin/changeset publish
🦋  info npm info eslint-config-custom
🦋  info npm info @farcaster/js
🦋  info npm info @farcaster/protobufs
🦋  info npm info @farcaster/utils
🦋  warn eslint-config-custom is not being published because version 0.0.0 is already published on npm
🦋  info @farcaster/js is being published because our local version (0.2.9) has not been published on npm
🦋  warn @farcaster/protobufs is not being published because version 0.1.7 is already published on npm
🦋  info @farcaster/utils is being published because our local version (0.2.9) has not been published on npm
🦋  info Publishing "@farcaster/js" at "0.2.9"
🦋  info Publishing "@farcaster/utils" at "0.2.9"
🦋  info This operation requires a one-time password from your authenticator.
🦋  Enter one-time password: · ******
🦋  success packages published successfully:
🦋<EMAIL_ADDRESS>🦋<EMAIL_ADDRESS>🦋  Creating git tags...
🦋  New tag<EMAIL_ADDRESS>🦋  New tag<EMAIL_ADDRESS>


Add more descriptive violation logging

Howdy! 👋
As we are about to use this add-on in a new application, the other developer and I thought it would be nice to have more descriptive logging in the console rather than sifting through the violations object output.
This both updates the default messages that is displayed inside the browser test runner, along with more precise console logging, rather than dumping the entire violations object.
RFC
It seems the default environment tests hang due to differences in Ember versions. I'm not entirely sure how to go about fixing that. Not sure if it has something to do with the WIP Ember upgrade or what the best way to resolve it now would be.
New test runner messaging

New console level information

I'm very excited to use this add-on, but would obviously like some improved logging such as outlined in one of the issues (https://github.com/trentmwillis/ember-axe/issues/3)
Travis seems to be hanging on...something, but there's no helpful error with it, nor are there any clear instructions on how to run the tests locally for the project. :(

Hey @aliciasedlock! I restarted the Travis job and reviewed. Code appears fine, I'm not sure what is going on with Travis. Just to be safe and hopefully keep things green we're going to land #6 first to bring this project up-to-date and then land this.
To run the tests locally you can just check out the project and run npm test.

@aliciasedlock Can you reopen this PR against ember-a11y-testing? We migrated this addon into that repository. After normal updates this PR should still merge. :)

@nathanhammond can do! Will likely do that in the next day or two.

@aliciasedlock Ping! We'd love to get this code in!

@nathanhammond just finished up! https://github.com/ember-a11y/ember-a11y-testing/pull/25


share visualisation code between dashboard and jupyter

probably using the _repr_javascript_ mentioned here: https://ipython.readthedocs.io/en/stable/config/integrating.html#custom-methods

done with 1c52095027fc3feaa1ef3df55e9a890c718c2dd5


Fix CI lint issue

See e.g. here: https://github.com/Skybox-Technologies/bjs-ecs/actions/runs/8737315090/job/23974024192#step:7:40
  /home/runner/work/bjs-ecs/bjs-ecs/libs/bjs-ecs/package.json
    11:3  error  The "bjs-ecs" project uses the following packages, but they are missing from "dependencies":
      - tinybench  @nx/dependency-checks

Fixed


VS Code freezes after pasting inside a file, its file name copied from the Save as windows

Issue Type: Bug
Steps to Reproduce:

Create a new file or open an existing one
Go to File->Save As...
Copy the name of the file ( then close the windows by canceling or saving the file ).
Past the name inside the file -> VS Code is freezing for something like 10s, and then nothing is pasted

If I past inside an other text editor, things works fine.
This problem only happen recently, it was working fine before.

VS Code Version: March 2021 (version 1.55) and also tried on Insider April 2021 (version 1.56)
OS Version: Linux Ubuntu 20.04.2

VS Code version: Code - Insiders 1.56.0-insider (0310f02dc5d834bb3a7cc421ea5374aec8d011f1, 2021-04-23T05:13:20.010Z)
OS version: Linux x64 5.8.0-50-generic snap

System Info

Item
Value

CPUs
Intel(R) Xeon(R) E-2176M  CPU @ 2.70GHz (12 x 4078)

GPU Status
2d_canvas: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onoop_rasterization: disabled_offopengl: enabled_onrasterization: disabled_softwareskia_renderer: enabled_onvideo_decode: disabled_softwarevulkan: disabled_offwebgl: enabledwebgl2: enabled

Load (avg)
1, 1, 1

Memory (System)
62.31GB (20.53GB free)

Process Argv
--force-user-env --no-sandbox --unity-launch --no-sandbox

Screen Reader
no

VM
0%

DESKTOP_SESSION
ubuntu

XDG_CURRENT_DESKTOP
Unity

XDG_SESSION_DESKTOP
ubuntu

XDG_SESSION_TYPE
x11

Extensions: none
A/B Experiments
vsliv695:30137379
vsins829:30139715
vsliv368cf:30146710
vsreu685:30147344
python383cf:30185419
pythonvspyt602:30291494
vspor879:30202332
vspor708:30202333
vspor363:30204092
vstry244:30244315
pythonvsdeb440:30224570
pythonvsded773:30223139
pythonvspyt875:30259475
pythonvspyt639:30291487
pythontb:30258533
vspre833:30267464
pythonptprofiler:30281269
vscnewfiletext:30288477
vshan820:30276952
pythondataviewer:30285072
vscus158cf:30286554
vscgsv2:30286872

Unable to repro with latest stable 1.73


Fish autocompletion bug

Fish appears to autocomplete the command in wrong places
For example after another git command
> git add pr
pr  (Create a new pull request)  prune-branches  (Delete merged branches)

I was trying to autocomplete git add p to the file package.json but instead it autocompleted to pr

This is intentional, at least for now.
Initially the autocomplete was more specific. But the full graph of which things can/should be autocompleted in which situation is surprisingly complex. So I erred on the simple side, autocomplete everything all the time, and leave it up to the user to pick the right things.

This is intentional, at least for now.
Initially the autocomplete was more specific. But the full graph of which things can/should be autocompleted in which situation is surprisingly complex. So I erred on the simple side, autocomplete everything all the time, and leave it up to the user to pick the right things.

Closing this for now since it works as intended, and there are no plans on fixing this on the roadmap. Please reopen if this changes.


Question: error using convert-di-to-inject

Discussed in https://github.com/ngxtension/ngxtension-platform/discussions/399

Originally posted by kscavenius May 31, 2024
Hi
I am experimenting with the great convert-di-to-inject functionality. When running ng g ngxtension:convert-di-to-inject i get the error:

My app compiles just fine, and it works great when specifying a component to migrate.
All the best
Kristoffer

Are you running the schematic on the root application path? And did you try it again with the newer versions? If yes, and the error persists, can you please give more details?

Hello @kscavenius
Can you provide some more information please. It's hard to find the root cause here. Also would be great if you could update to latest version and try again. This kind of error was solved once from what I remember.

Hi
Thanks for getting back. I have some major issues with Webstorm (freezing constantly) and just switched to VS Code today. So I will have to focus on that. I will not be able to look into it until the end of next week due to a conference attendance.
But I guess my package.json answers it, right? I am on core 17.3.10, and 18 is required. My apologies.
Here is my package.json
{
  "name": "osint-app",
  "version": "0.0.0",
  "scripts": {
    "ng": "ng",
    "start": "ng serve",
    "build": "ng build --base-href .",
    "watch": "ng build --watch --configuration development",
    "test": "jest",
    "find-dead-code": "ts-prune"
  },
  "jest": {
    "preset": "jest-preset-angular",
    "setupFilesAfterEnv": [
      "<rootDir>/setup-jest.ts"
    ]
  },
  "private": true,
  "dependencies": {
    "@angular/animations": "^17.3.10",
    "@angular/cdk": "^17.0.4",
    "@angular/common": "^17.3.10",
    "@angular/compiler": "^17.3.10",
    "@angular/core": "^17.3.10",
    "@angular/forms": "^17.3.10",
    "@angular/localize": "^17.3.10",
    "@angular/platform-browser": "^17.3.10",
    "@angular/platform-browser-dynamic": "^17.3.10",
    "@angular/router": "^17.3.10",
    "@total-typescript/ts-reset": "^0.5.1",
    "@videogular/ngx-videogular": "^8.0.0",
    "angular-highcharts": "^16.0.0",
    "bootstrap": "^5.3.2",
    "flag-icons": "^7.1.0",
    "highcharts": "^11.2.0",
    "highcharts-angular": "^4.0.0",
    "idb": "^8.0.0",
    "jspdf": "^2.5.1",
    "leaflet": "^1.9.4",
    "lodash": "^4.17.21",
    "ng2-google-charts": "^7.0.0",
    "ngx-clipboard": "^16.0.0",
    "ngxtension": "^3.4.0",
    "ol": "^8.2.0",
    "popper.js": "^1.16.1",
    "primeicons": "^6.0.1",
    "primeng": "^17.2.0",
    "proj4": "^2.9.2",
    "quill": "2.0.0-rc.0",
    "rxjs": "~7.8.0",
    "tslib": "^2.4.1",
    "tui-image-editor": "^3.15.3",
    "zone.js": "~0.14.2"
  },
  "devDependencies": {
    "@angular-devkit/build-angular": "^17.3.8",
    "@angular/cli": "^17.3.8",
    "@angular/compiler-cli": "^17.3.10",
    "@testing-library/angular": "^13.4.0",
    "@types/jasmine": "~5.1.0",
    "@types/jest": "^29.5.11",
    "@types/node": "^20.10.5",
    "jasmine-core": "~5.1.0",
    "jest": "^29.7.0",
    "karma": "~6.4.0",
    "karma-chrome-launcher": "~3.2.0",
    "karma-coverage": "~2.2.0",
    "karma-jasmine": "~5.1.0",
    "karma-jasmine-html-reporter": "~2.1.0",
    "ts-prune": "^0.10.3",
    "typescript": "~5.2.2"
  }
}

Best regards
Kristoffer

Hello @kscavenius
In this release https://github.com/angular/angular/releases/tag/18.2.0-next.2 Angular added official support for inject migration. I'd recommend to try to run that migration too in your project and if there are issues, please open them in the Angular repo.
I will be closing the issues regarding migrations in ngxtension which are now directly supported in the fwk.


JWT audience claim

Upstream https://github.com/eclipse-dataspaceconnector/DataSpaceConnector/issues/1176
Implement audience claim check per upstream issue
Acceptance criteria

[ ] Client generates JWT for IDS requests containing destination web hook HTTP address as audience claim
[ ] Server rejects requests with wrong audience
[ ] Implementation is not dependent on DID and would support any other identity methods

Blocked by upstream issues


Element Flows Delete Vertices

When I draw a crazy line, I want to be able to delete vertices that are created that are wild. Current implementation, while much better, still produces lots of weird loops and out of control lines.

@velkymx I believe this issue was addressed in https://github.com/ProcessMaker/spark-modeler/pull/405. While not obvious to the user you can delete a vertices by double clicking the dot on the vertices. I think a tooltip or a helper text when highlighting the sequence flow would make for a better user experience.

Fix of the weird loops in spark
https://drive.google.com/open?id=1xDwMNcsoMoBplt3Wh-uNYQXsr1JyP5EC

OK closing with note: Double click to remove vertices


Fixing directory permissions (octal instead than decimal)

Hi again! I have been trying to generate your book's HTML code on my machine but I got problems with the directory permission. Please note that mkdir (and other functions like umask & co) use octal permission, while the strings that you've used, '0755' and '755' get interpreted as decimal numbers (you can see it by yourself by evaluating '0755' * 1 or something like that).

Merged, thanks!


CI Build on ubuntu-latest aarch64 is broken

Install version / commit hash
v4.0.6 ~ v4.2.3
Expected Behavior vs Actual Behavior
Expected Bahaviour
CI suceeds
Actual Behaviour
CI failed
https://github.com/spotDL/spotify-downloader/actions/workflows/python-publish.yml

Other details

https://github.com/spotDL/spotify-downloader/actions/runs/7071645050/job/19249566681
$ grep -i err 'Build on ubuntu-latest aarch64/3_Run commands.txt' | grep -v transferr | grep -v dockerR
2023-12-02T17:38:20.0551331Z Error response from daemon: manifest unknown
2023-12-02T17:38:21.0564736Z #8 ERROR: failed to configure registry cache importer: ghcr.io/spotdl/spotify-downloader/run-on-arch-spotdl-spotify-downloader-upload-python-package-aarch64-ubuntu-latest:latest: not found
2023-12-02T17:41:21.2982479Z #10 ERROR: process "/bin/sh -c chmod +x /root/run-on-arch-install.sh && /root/run-on-arch-install.sh" did not complete successfully: exit code: 2
2023-12-02T17:41:21.3063023Z ERROR: failed to solve: process "/bin/sh -c chmod +x /root/run-on-arch-install.sh && /root/run-on-arch-install.sh" did not complete successfully: exit code: 2
2023-12-02T17:41:21.3124279Z ##[error]The process '/home/runner/work/_actions/uraimo/run-on-arch-action/v2/src/run-on-arch.sh' failed with exit code 1

Yeah I know about this, I've tried fixing it multiple times but nothing worked. The best thing is that between version 4.0.5 and 4.0.6 nothing changed in the build workflow so yeah....
I've reverted the build-arm action to v4.0.5, maybe it will start working again...

fixed!

Thank you!


ci: ensure color is valid length #rrggbb

See https://github.com/nvim-tree/nvim-web-devicons/pull/351#issuecomment-1866285143
Checks that #rrggbb is correct length

Didn't test it, but it should just work. :)

We'll find out soon enough...


[windows] Consolidate dependency infos in build.ps1 (NFC)

The recent move away from host Python, additional modules and the upcoming LLDB test dependencies are adding 3rd-party dependencies to the build script and their URLs/hashes/version kept spreading throughout the code. This patch is an attempt to consolidate the required data in a $Components map that is indexed with an abstract name. Each item has the fields URL and Hash but isn't limited to that. The PinnedSwift items for example have an extra Version field.
Major refactors in this patch are:

DownloadAndVerify was changed. It takes an abstract component name now instead of URL/Hash.
Ensure-PythonModule was added. It takes a Python module name and checks/installs that.
Ensure-PythonModules was dropped
$PinnedBuild/SHA256/Version remain bare parameter values and don't get overwritten anymore.
$PinnedSwift contains the component name that selects which pinned toolchain is used.

Closing this as the above PR contains these patches now.


Use a go.work for examples instead of go.mod's replace directive

Go workspaces are easier to override.

Potentially my understanding is wrong here, but I was under the impression that it's not generally advisable to commit a go.work file.

I wouldn't do it a project's root but I think it is acceptable for an examples/ sub folder.

Sorry about the delayed response on this one.
I dislike that the examples have a replace directive in them, because it means you can't copy/paste the folder outside of the directory structure and use it, which I think is confusing when you're trying to build on top of an example.
It sounds like a go.work file is a good way to avoid this, but also that the general idea is that they shouldn't be checked in.
I'm not an expert on Go workspaces, and don't recall using them myself. I guess they allow you to open multiple Go projects at once or something?
So, the reasons we haven't merged this seem to be that @joerdav and I don't currently have the knowledge and experience to make a call on this, so we need to do that before we can make a call on it, hence the delay.

The use case I've seen for workspaces is very different than this, like for using a local version of a dependency without editing the go.mod
And most projects I've seen have it gitignored, but if this works then maybe it's a correct use case too!

If you want an example of checked in go.work you have https://github.com/kubernetes-sigs/kustomize/.


feat: Show code snipped in diagram cache dialog

In order for users to better understand how a personal access token can be helpful the codesnippet from the docs is displayed in the frontend.
closes #598

Currently looks like

I used CSS in order to style it as I thought would be nicer than javascript - nevertheless the copied code still has a lot of white spaces in front even though they do not get displayed...


txt2img.py doesn't work.

I ran txt2img.py on Google Colab.
!python scripts/txt2img.py --prompt "prompt" --ckpt /content/trinart2_step60000.ckpt --config configs/stable-diffusion/v2-inference-v.yaml
But it stop in the middle of process with the '^C'. Why is that?

Global seed set to 42
Loading model from /content/trinart2_step60000.ckpt
Global Step: 60000
No module 'xformers'. Proceeding without it.
/usr/local/lib/python3.10/dist-packages/pytorch_lightning/utilities/distributed.py:258: LightningDeprecationWarning: pytorch_lightning.utilities.distributed.rank_zero_only has been deprecated in v1.8.1 and will be removed in v2.0.0. You can import it from pytorch_lightning.utilities instead.
rank_zero_deprecation(
LatentDiffusion: Running in v-prediction mode
DiffusionWrapper has 865.91 M params.
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
open_clip_pytorch_model.bin: 100% 3.94G/3.94G [02:02<00:00, 32.1MB/s]
^C

As my experience, it is caused due to RAM out of memory. Try to run in a device with larger RAM