JWT plugin does not find credential by key when key is URL

Summary
Hello, we use kong along with keycloak and we protect every api publication with JWT plugin on the route. keycloak generate JWT with iss claim with value of keycloak realm url. When I setup jwt plugin to use iss claim as a key for the secret the plugin can't find the secret by this key. When I use another custom claim in JWT to identify secret everything works correctly. But when I use admin API to find secret/jwt with key the result is correct in both cases.
Steps To Reproduce

Setup JWT with the 'iss' claim as secret key

apiVersion: v1
kind: Secret
metadata:
  name: jwt-nghis-dev
  namespace: ng-apps
type: Opaque
data:
  kongCredType: and0
  key: aHR0cHM6Ly9hdXRoLmRldi5wcm9zb2Z0a2Uuc2svYXV0aC9yZWFsbXMvaGlz
  algorithm: UlMyNTY=
  rsa_public_key: LS0tLS1CRUdJTiBQVUJMSUMgS0VZLS0tLS0KTUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUFqc3AzVGR5ODFKU0VSN2NZdW9ZcgoxVmhqSlB4WXR0Nk1ad1hQSUJsRlhpZ21nYlZnTzNiQmhyR2x1Q2lnVlUrVkJ1YkZnM29VektVS0pWV2ZCRzZJCmgwcXUvcTFqSXRZdGI1bDJINXhHcTIwU3RDK1NCSU4rUFY2eW4zMFhPeWVIN1dmSCtUOGQxcVlKS243aVlhQWkKaWlKbUFKOHR5bHk2eHlqT01ZbnJJRXM1VEU4STlkaitrUlJTQzJTVGpFN2xoR0hOTUVheVZKeFJkejliN00yQgpKWmtmZ09NMEhKUnlzaXNiZXVwYnRidGdvMWQycUV6eVkxTTg3VTk2ejJsRWtwQllvREpuMFFqdGE3MWl3bzdkCmoyamFEVDlPZDhtdzdzWXNxeUxQK2NTTE9QQjRoNC9ncjBjeWl6M05hNGJHbVJrYU9Qd0tlM2hMOGdQMFJjWFYKd1FJREFRQUIKLS0tLS1FTkQgUFVCTElDIEtFWS0tLS0t

---

apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
  name: nghis-dev-consumer
  namespace: ng-apps
username: nghis-dev
credentials:
- jwt-nghis-dev

---

apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: nghis-jwt-administration-service
  namespace: ng-apps
config:
  key_claim_name: iss
  claims_to_verify:
    - exp
plugin: jwt

---

apiVersion: configuration.konghq.com/v1
kind: KongIngress
metadata:
  name: administration-service-test-ingress
  namespace:  ng-apps
route:
  strip_path: true

---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: administration-test-service
  namespace: ng-apps
  labels:
    app: administration-test-service
    layer: frontend
  annotations:
    plugins.konghq.com: nghis-jwt-administration-service
    configuration.konghq.com: administration-service-test-ingress
spec:
  rules:
  - host: "api.dev.prosoftke.sk"
    http:
      paths:
      - path: /admin_test
        backend:
          serviceName: administration-service
          servicePort: 8080

admin API works as expceted:
curl http://kong-admin:8001/jwts/https%3A%2F%2Fauth.dev.prosoftke.sk%2Fauth%2Frealms%2Fhis

{"rsa_public_key":"-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAjsp3Tdy81JSER7cYuoYr\n1VhjJPxYtt6MZwXPIBlFXigmgbVgO3bBhrGluCigVU+VBubFg3oUzKUKJVWfBG6I\nh0qu\/q1jItYtb5l2H5xGq20StC+SBIN+PV6yn30XOyeH7WfH+T8d1qYJKn7iYaAi\niiJmAJ8tyly6xyjOMYnrIEs5TE8I9dj+kRRSC2STjE7lhGHNMEayVJxRdz9b7M2B\nJZkfgOM0HJRysisbeupbtbtgo1d2qEzyY1M87U96z2lEkpBYoDJn0Qjta71iwo7d\nj2jaDT9Od8mw7sYsqyLP+cSLOPB4h4\/gr0cyiz3Na4bGmRkaOPwKe3hL8gP0RcXV\nwQIDAQAB\n-----END PUBLIC KEY-----","created_at":1582725445,"consumer":{"id":"2af3d7cc-5590-5c3b-a904-dcb32e16fae9"},"id":"7207410d-15b5-513a-8400-c6ec638a0da8","algorithm":"RS256","tags":null,"key":"https:\/\/auth.dev.prosoftke.sk\/auth\/realms\/his","secret":"SgSNLGREhLChx9rR4rxda03TKFTw70Oe"}

The call on route end with the error:

{
  "message": "No credentials found for given 'iss'"
}

When I change key_claim_name: tenantName in the plugin definition and 'key: REVW' in the secret everything works fine.

Additional Details & Logs

Kong version 2.0.0; db-less
Kubernetes 1.15.6; kong:2.0.0-alpine; kong-ingress-controller:0.7.1
token used in tests eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJRWW9fLXlPRk1hQTFmX1RINXNSb3U0d0JsdDR4dWVSc01iRThZbG1IYUJjIn0.eyJqdGkiOiJkZWM4NDU2Zi1jZmJjLTQwYjYtYTM3Yi03NTUyYjEwODQ1YmEiLCJleHAiOjE1ODI3MjM0NzQsIm5iZiI6MCwiaWF0IjoxNTgyNzIzMTc0LCJpc3MiOiJodHRwczovL2F1dGguZGV2LnByb3NvZnRrZS5zay9hdXRoL3JlYWxtcy9oaXMiLCJhdWQiOiJhY2NvdW50Iiwic3ViIjoiYmFjMWExM2EtMWE5YS00NmZiLTg5YjYtODdiNzRmMmFkYjUyIiwidHlwIjoiQmVhcmVyIiwiYXpwIjoibmdoaXMiLCJub25jZSI6IjJmMjgyMTMxLWIxOGItNGFiZi05MzQ2LTFlZGM1YTRlNDkwNSIsImF1dGhfdGltZSI6MTU4MjcyMzE2Niwic2Vzc2lvbl9zdGF0ZSI6ImQ2MDczN2VjLTVkY2EtNGY2NS1iMWJmLTQ1OGFmOTk5OGUzMyIsImFjciI6IjAiLCJhbGxvd2VkLW9yaWdpbnMiOlsiaHR0cDovLzEyNy4wLjAuMTo4ODg4IiwiaHR0cDovL2xvY2FsaG9zdDo4ODg4IiwiaHR0cDovL2xvY2FsaG9zdDo4ODg3IiwiaHR0cDovL2xvY2FsaG9zdDo4ODg5IiwiaHR0cHM6Ly9uZ2hpcy5kZXYucHJvc29mdGtlLnNrIiwiaHR0cDovLzEyNy4wLjAuMTo4ODg5IiwiaHR0cHM6Ly9hZG1pbi5kZXYucHJvc29mdGtlLnNrIl0sInJlYWxtX2FjY2VzcyI6eyJyb2xlcyI6WyJvZmZsaW5lX2FjY2VzcyIsInVtYV9hdXRob3JpemF0aW9uIl19LCJyZXNvdXJjZV9hY2Nlc3MiOnsiYWNjb3VudCI6eyJyb2xlcyI6WyJtYW5hZ2UtYWNjb3VudCIsIm1hbmFnZS1hY2NvdW50LWxpbmtzIiwidmlldy1wcm9maWxlIl19fSwic2NvcGUiOiJvcGVuaWQgcHJvZmlsZSBlbWFpbCIsImVtYWlsX3ZlcmlmaWVkIjpmYWxzZSwidGVuYW50TmFtZSI6IkRFViIsInVzZXIiOnsibGFzdE5hbWUiOiJMZWthciIsImZpcnN0TmFtZSI6IkphbiIsInByZWZpeCI6IlBoZCIsImhlYWx0aGNhcmVQcm9mZXNzaW9uYWwiOjEsInBlcnNvbmFsTnVtYmVyIjoiMTExMSIsInBvc3RmaXgiOiJJbmciLCJsb2dpbiI6Imxla2FyIiwiZW1wbG95ZWVOdW1iZXIiOjF9LCJ0ZW5hbnQiOjk5OTl9.ew34kC-_hix_J054qiu2s4VLTfqyLxxNUuSLwuS7YLwn2jYXOSkTP9GSrWjs85zObGFxEacGarcWDDmjmxJDoQlxvOlpyNAZ8LuXVplsVBz5TQRMjQrTw5CSyA5uhVA8s7lBNdYzMKMh3LNEdp93wMALzsbsMFqHMX9SI2rFTISvAAknUgQmxAAUjNdOFap-lWdOnXESh4vXXRyXApc5Av9_JbH0v2ZG8ay2nHF3VEweWrFpkDOPeTy2Opza2Z08GAkJDWDfgq_TixylUK4vSBvyKZ4-uYbwORal4XLknZbD3mpNMl0W7v8eccGmwzaA4M6rKgfxuUMuYGittra4VQ

{
"message": "No credentials found for given 'iss'"
}
same response, after i plugin that jwt plugin

I can confirm as well.   K8S 1.17, kong helm chart 1.14.5.

NM I got it working.....   I had upgraded to my  helm chart from 1.5 5o 1.14 and which kong versions those contain...   After putting a large dent in my desk and reading docs and looking at examples, I finally figured out I had to update my KongConsumer def to have an annotation....   So in the example above...
apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
name: nghis-dev-consumer
namespace: ng-apps
username: nghis-dev
credentials:

jwt-nghis-dev

should  probably  be
apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
name: nghis-dev-consumer
namespace: ng-apps
annotations:
kubernetes.io/ingress.class: kong
username: nghis-dev
credentials:

jwt-nghis-dev

Works fantastic now...

I am hoping anyone would help me put here.
I am pretty much using this example. https://docs.konghq.com/kubernetes-ingress-controller/1.1.x/guides/configure-acl-plugin/
echo "
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: app-jwt
plugin: jwt
" | kubectl apply -f -

echo '
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: demo-get
  annotations:
    konghq.com/plugins: app-jwt
    konghq.com/strip-path: "false"
    kubernetes.io/ingress.class: kong
spec:
  rules:
  - http:
      paths:
      - path: /get
        backend:
          serviceName: httpbin
          servicePort: 80
' | kubectl apply -f -

$ echo '
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: demo-post
  annotations:
    konghq.com/plugins: app-jwt
    konghq.com/strip-path: "false"
    kubernetes.io/ingress.class: kong
spec:
  rules:
  - http:
      paths:
      - path: /post
        backend:
          serviceName: httpbin
          servicePort: 80
' | kubectl apply -f -

$ curl -i $PROXY_IP/get

HTTP/1.1 401 Unauthorized

echo "
apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
name: admin
annotations:
kubernetes.io/ingress.class: kong
username: admin
" | kubectl apply -f -
echo "
apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
  name: plain-user
  annotations:
    kubernetes.io/ingress.class: kong
username: plain-user
" | kubectl apply -f -

# create secret for jwt public key
$ kubectl create secret \
  generic app-admin-jwt  \
  --from-literal=kongCredType=jwt  \
  --from-literal=key="admin-issuer" \
  --from-literal=algorithm=RS256 \
  --from-literal=rsa_public_key="-----BEGIN PUBLIC KEY-----
  MIIBIjA....
  -----END PUBLIC KEY-----"

# create a second secret with a different key
$ kubectl create secret \
  generic app-user-jwt  \
  --from-literal=kongCredType=jwt  \
  --from-literal=key="user-issuer" \
  --from-literal=algorithm=RS256 \
  --from-literal=rsa_public_key="-----BEGIN PUBLIC KEY-----
  qwerlkjqer....
  -----END PUBLIC KEY-----"

Now for the above step, i use following to generate jwt
ssh-keygen -t rsa -P "" -b 4096 -m PEM -f jwtRS256.key
ssh-keygen -e -m PEM -f jwtRS256.key > jwtRS256.key.pub

import jwt
private_key = open('jwtRS256.key').read()
token = jwt.encode({'username': 'admin',  'iss': 'admin-issuer'}, private_key, algorithm='RS256').decode('utf-8')
print token

$ echo "
apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
  name: admin
  annotations:
    kubernetes.io/ingress.class: kong
username: admin
credentials:
  - app-admin-jwt
" | kubectl apply -f -

$ echo "
apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
  name: plain-user
  annotations:
    kubernetes.io/ingress.class: kong
username: plain-user
credentials:
  - app-user-jwt
" | kubectl apply -f -

curl -i -H "Authorization: Bearer ${ADMIN_JWT}" $PROXY_IP/get
{"message":"No credentials found for given 'iss'"}%
Any help on where i might be going wrong. it's pretty much using the same example.

Hello everybody.
I've set up my Keycloak inside K8S cluster, and Kong DB-less within the cluster too.
Keycloak installed via Keycloak Operator 0.17, but it's not really important, as long as you have a recent version:
PS C:\Users\ferdi> kubectl get csv NAME                        DISPLAY             VERSION   REPLACES                    PHASE keycloak-operator.v12.0.3   Keycloak Operator   12.0.3    keycloak-operator.v12.0.1   Succeeded
Kong via official Helm chart:
PS C:\Users\ferdi> helm list NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION kong    kong            1               2021-04-22 10:31:34.5898107 +0200 CEST  deployed        kong-2.0.0      2.3
I've created my Realm, my Client and my User within Keycloak.
Just remember to enable "Direct Grants" within Client configuration.
Below you can find my YAML:
`---
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: jwt-test
namespace: default
config:
secret_is_base64: false
run_on_preflight: true
plugin: jwt
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: kong-httpbin
namespace: default
annotations:
kubernetes.io/ingress.class: kong
konghq.com/plugins: jwt-test
spec:
rules:

host: httpbin2.localhost
http:
paths:

path: /
backend:
serviceName: httpbin2
servicePort: 80

apiVersion: v1
kind: Secret
metadata:
name: kong-credentials-httpbin
namespace: default
type: Opaque
stringData:
kongCredType: jwt
key: https://keycloak.localhost/auth/realms/myrealm
secret: 9080ed51-b61d-416d-bded-302ca1d62033
algorithm: RS256
rsa_public_key: |-
-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAmfTOZLJ9rP8iL4gWmlIe
xlrC7rnh2qPqvXT5GEHYoKt2wgOyCxOHiClTQB4sLVWld36OmiqzSnGgzPgnN6z6
iS8rE9DXE/LDiwFOU2Roq6CrVAUvpJGqwUwU5xkWxf4l7CFegVjli0x/qDywhPww
tBsR7yD5c9KM9A/OrBK/S5oaCa7a6zTJJRTujGR+o3nRUojpzphdendjx6hrjsot
rgj0ux1Mghc9AjLPxv/Bld4MTYIplmeVfardOsMdkJLluaOguGwjVkzA85bTDw7n
IiibiptsJhJ3qARbu3+oqYapti8h4b8TFN+7GixQeOQKBuKUsh1uMCRFXB/HFzMY
rQIDAQAB
-----END PUBLIC KEY-----
apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
name: kong-consumer-httpbin
annotations:
kubernetes.io/ingress.class: kong
username: whateveryouwant
credentials:

kong-credentials-httpbin`

I've just reviewed my previous contribution... and I missed the most important part: with the outlined setup, my authorization flow went fine! @SachinMaharana , as far as I can tell (I'm not building my JWT on my own, I'm relying on Keycloak), I can  spot only one difference between our setups: your KongPlugin definition lacks the "config" subsection.
config:
  secret_is_base64: false
  run_on_preflight: true

Appears solved - please comment if you're experiencing a similar problem.

{
"message": "No credentials found for given 'iss'"
}
same response, after i plugin that jwt plugin

Hi all, I had and then solved this problem.
Problem:
"message": "No credentials found for given 'iss'"
Solution:
I was using a Secret, a KongConsumer and a KongPlugin.
This problem was occurring because I believe the Plugin is not able to find the Consumer, or the Consumer is not able to find the Secret.
I resolved the issue by ensuring they were all in the same 'default' namespace. Here are the .yaml files I was using if that helps anyone:
`apiVersion: configuration.konghq.com/v1   #Plugin referenced in the Ingresses, tells kong we are using jwt to authenticate
kind: KongPlugin
metadata:
name: auth0-jwt-plugin
namespace: default
config:
secret_is_base64: false
run_on_preflight: true
plugin: jwt
apiVersion: configuration.konghq.com/v1   #Consumer is used as a proxy for the user. i.e. if someone logs in and satisfies jwt found in credentials, they are "username" certified
kind: KongConsumer                        #In this version they are all the same user
metadata:
name: auth0consumer
namespace: default
annotations:
kubernetes.io/ingress.class: kong
username: auth0user
credentials:

auth0jwtkey

apiVersion: v1                            #Public jwt key used to determine whether it was truly generated by our auth0 account
kind: Secret
metadata:
name: auth0jwtkey
namespace: default
type: Opaque
stringData:
kongCredType: jwt
key: https://companyname.auth0.com/
algorithm: RS256
rsa_public_key: |-
-----BEGIN PUBLIC KEY-----
asdasdasdasdasdasdasdasdasdasd
-----END PUBLIC KEY-----`

Dear contributor,
We're closing this issue as there hasn't been any update to it for a long time. If the issue is still relevant in the latest version, please feel free to reopen it. We're more than happy to revisit it again.
Please have a look at our pledge to the community for more information.
Sincerely,
Kong Gateway Team


Fixes  to TFJS Flask App.

The Flask app provided to showcase the TFJS model is not working. The audio is resampled to 44100Hz and there are CORS errors when the app tries to create the audioContext without any user interaction. This PR addresses these issues.

Thanks, works like a charm!


[SPARK-19999]: Fix for flakey tests due to java.nio.Bits.unaligned() …

java.nio.Bits.unaligned() does not return true for the ppc64le arch.
see https://bugs.openjdk.java.net/browse/JDK-8165231
What changes were proposed in this pull request?
check architecture
How was this patch tested?
unit test

Can one of the admins verify this patch?

@kiszk @srowen updated per your commets

@gatorsmile I think my Intellij has some annoying auto indenting changes. Does spark have some project wide settings I can auto import?

I do not think we have such a default steup. You can manually change the indentation of InteiiJ

ok @gatorsmile i set default indentation to 2, how does it look now?

@gatorsmile moved the comment per your suggestion, but to be honest if the comment is unclear surly the first thing someone will do is check that JIRA?

ok to test

Clear code comments can help code reading. : )

Test build #75372 has started for PR 17472 at commit bf7cc24.

Test build #75372 has finished for PR 17472 at commit bf7cc24.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75372/
Test FAILed.

Test build #75380 has started for PR 17472 at commit 77777e9.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75380/
Test FAILed.

jenkins test this please

Can someone guide me why the tests failed? is it another flakey test or? because the below message doesnt make sense to me :(

Traceback (most recent call last):
File "./dev/run-tests-jenkins.py", line 226, in
main()
File "./dev/run-tests-jenkins.py", line 213, in main
test_result_code, test_result_note = run_tests(tests_timeout)
File "./dev/run-tests-jenkins.py", line 140, in run_tests
test_result_note = ' * This patch fails %s.' % failure_note_by_errcode[test_result_code]
KeyError: -9

Test build #75386 has started for PR 17472 at commit 632161b.

Test build #75386 has finished for PR 17472 at commit 632161b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75386/
Test PASSed.

Merged build finished. Test PASSed.

@gatorsmile @srowen how does it look now?

LGTM

Merged to master


[Transformation] New transform to move linear ops past several ops

New transform MoveScalarLinearPastInvariants to move linear ops past "GlobalAveragePool" ,"Reshape" ,"Transpose" and "Flatten"
Notes

Lucian's code
It was needed when our use case used "GlobalAveragePool" instead of "QuantAvPool". Non needed for this case anymore.

Added calls to InferDataTypes and InferShapes before exiting transform if graph was modified. Note that this may still lead to incorrect DataType annotations around the transformed tensors, e.g. if a scalar Mul originally had a INT4 output annotated this transform will take that away.

Hopefully the output datatype issue is solved with this last commit. This should be checked in a future test, but since this PR is a dependency for other important PRs for upcoming work like #200 I am merging it as-is.


How to manage boards with many revision?

Hi
https://open-cmsis-pack.github.io/Open-CMSIS-Pack-Spec/main/html/pdsc_boards_pg.html#element_boards
If one board have many revision, but each revision impact only 1 of the feature,
it seems we need to duplicate all the board structure for the new revision (and new uuid).
Maybe we could find a way to avoid this and to create a board level with all common information,
and then a sub level for revision ?

Your request reminds me of device variants that we introduced in the specification a long time ago. So the concept by which common information is inherited rather than repeated has been recognized as a means of reducing redundancy. At the same time we find that people generating this information from other sources do not benefit much and it can be harder to read.
I am also worried about existing tools being unable to correctly process this new board description format. I could also imagine that it maybe favorable to release a separate new board support pack for each new board revision assuming that previous revisions won't require any update at all.

@avilei

It's an interesting proposal. Of course we would need to find the right trade-off between reducing redundancy on the one hand, and improving tools efficiency and back compatibility on the other hand.
I'm also adding @silviooliva to the discussion.

@avilei and @silviooliva I think we are awaiting a proposal from you.


Not able to run mergerfs.fsck under OMV3

Sorry, when this turns out to be an noob question, but I'm still a beginner on linux systems
Copied over the files to /usr/bin, which is part of my $PATH
Running mergerfs.fsck -h get "Permission denied" Error.
sudo mergerfs.fsck -h throws "Command not found"
running as root has the same effects.
python3 -V reads 3.4.2
Any ideas?
Big thx , Timm

is the file set to executable ? chmod ugo+x /usr/bin/mergerfs.fsck

Ooops, forgot about this... after all I AM a linux noob.
Thanks for pointing this out, now it seems to work.


Romfs compression

Adds support for LZSS compression of the ROMFS; saves ~50K of firmware space.

I'm retaining the branch, but we still were not able to swallow the risk yet. Thanks for doing this!


16 remove client

Changes the "Asset" helper method to no longer fall back to looking up data from Tessa.  If it's not in ActiveStorage, it doesn't exist.
Preserves other deprecated methods, but changes them to no-op with a simulated 503 service unavailable.  This should result in Tessa::Asset::Failure being returned any other places where an app directly calls Tessa.

This will be a major version bump.
fixes #16

Basically yes.
The upgrade process should be:

upgrade to major version 2
Run rake tessa:verify in bin/release
Track down any bugsnags where we still accidentally are hitting Tessa and getting Asset Failures.

On that note it's probably a good idea to call Bugsnag.notify inside the FakeConnection if bugsnag is defined.


STM32: Fixes on Pin configuration

This PR contains two fixes on STM32 pin configuration:

Return and error when attempting to configure a PIN sharing resources with already configured PIN.
Add missing GPIO ports in port_enable array

Codecov Report

Merging #11868 into master will decrease coverage by 0.13%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #11868      +/-   ##
==========================================
- Coverage   51.43%    51.3%   -0.14%
==========================================
  Files         231      231
  Lines       28751    28752       +1
  Branches     7152     7156       +4
==========================================
- Hits        14789    14751      -38
- Misses      11144    11188      +44
+ Partials     2818     2813       -5

Impacted Files
Coverage Δ

subsys/shell/shell_dummy.c
37.14% <0%> (-8.58%)
:arrow_down:

subsys/shell/shell.c
31.36% <0%> (-3.76%)
:arrow_down:

subsys/net/ip/udp.c
64.58% <0%> (-2.09%)
:arrow_down:

subsys/net/ip/ipv4.c
62.19% <0%> (-1.91%)
:arrow_down:

subsys/net/ip/connection.c
74.6% <0%> (-0.66%)
:arrow_down:

subsys/net/ip/tcp.c
56.98% <0%> (-0.4%)
:arrow_down:

subsys/net/ip/icmpv6.c
30.48% <0%> (-0.35%)
:arrow_down:

kernel/mutex.c
97.05% <0%> (ø)
:arrow_up:

subsys/net/ip/utils.c
79.37% <0%> (+1.41%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 459ad67...6b2c5fc. Read the comment docs.


[openjpeg] build failure x64-windows with features (jp3d,jpip,jpwl,mj2) on VS 2019 latest update 16.9.1

Host Environment

OS: [Microsoft Windows [Version 10.0.19042.867]]
Compiler:
Visual Studio 2019 Community Edition [Version 16.9.1]
Visual Studio Toolset [Version 14.28.29910]
Microsoft (R) C/C++ Optimizing Compiler [Version 19.28.29912 for x64]

To Reproduce
Steps to reproduce the behavior:

Default triplet VCPKG_DEFAULT_TRIPLET = x64-windows
PS D:\2019\test>vcpkg install openjpeg[core,jp3d,jpip,jpwl,mj2]

Failure logs

Cut and past the appropriate build messages from the console output.

Computing installation plan...
The following packages will be built and installed:
openjpeg[core,jp3d,jpip,jpwl,mj2]:x64-windows -> 2.3.1#2
Starting package 1/1: openjpeg:x64-windows
Building package openjpeg[core,jp3d,jpip,jpwl,mj2]:x64-windows...
-- Using cached D:/2019/test/downloads/uclouvain-openjpeg-v2.3.1.tar.gz
-- Cleaning sources at D:/2019/test/buildtrees/openjpeg/src/v2.3.1-ea4b63573b.clean. Use --editable to skip cleaning for the packages you specify.
-- Extracting source D:/2019/test/downloads/uclouvain-openjpeg-v2.3.1.tar.gz
-- Applying patch dll.location.patch
-- Using source at D:/2019/test/buildtrees/openjpeg/src/v2.3.1-ea4b63573b.clean
CMake Deprecation Warning at scripts/cmake/vcpkg_check_features.cmake:182 (message):
calling vcpkg_check_features without the FEATURES keyword has been
deprecated.
 Please add the `FEATURES` keyword to the call.

Call Stack (most recent call first):
ports/openjpeg/portfile.cmake:18 (vcpkg_check_features)
scripts/ports.cmake:139 (include)
-- Configuring x64-windows
-- Building x64-windows-dbg
-- Building x64-windows-rel
CMake Error at scripts/cmake/vcpkg_execute_build_process.cmake:144 (message):
Command failed: "C:/Program Files/CMake/bin/cmake.exe" --build . --config Release --target install -- -v -j8
Working Directory: D:/2019/test/buildtrees/openjpeg/x64-windows-rel
See logs for more information:
D:\2019\test\buildtrees\openjpeg\install-x64-windows-rel-out.log
Call Stack (most recent call first):
scripts/cmake/vcpkg_build_cmake.cmake:105 (vcpkg_execute_build_process)
scripts/cmake/vcpkg_install_cmake.cmake:45 (vcpkg_build_cmake)
ports/openjpeg/portfile.cmake:38 (vcpkg_install_cmake)
scripts/ports.cmake:139 (include)
Error: Building package openjpeg:x64-windows failed with: BUILD_FAILED
Please ensure you're using the latest portfiles with .\vcpkg update, then
submit an issue at [https://github.com/Microsoft/vcpkg/issues] including:
Package: openjpeg:x64-windows
Vcpkg version: 2021-01-13-8027bea90b47d50fd6090bbf5ded330f00f87570
Additionally, attach any relevant sections from the log files above.

Please attach any additional failure logs mentioned in the console output.
Several iterations of error within attached file:
D:\2019\test\buildtrees\openjpeg\src\v2.3.1-ea4b63573b.clean\src\lib\openmj2\opj_includes.h(97): error C2169: 'lrintf': intrinsic function, cannot be defined
D:\2019\test\buildtrees\openjpeg\install-x64-windows-rel-out.log

Additional context
Add any other context about the problem here, such as what you have already tried to resolve the issue.

Port openjpeg[core,jp3d,jpip,jpwl,mj2] works on latest update of VS 2017:
Visual Studio 2017 Community Edition [Version 15.9.34]
Visual Studio Toolset [Version 14.16.27023]
Microsoft (R) C/C++ Optimizing Compiler [Version 19.16.27045 for x64]
Port openjpeg will install without features [jp3d,jpip,jpwl,mj2] (but these features are needed)
This worked on VS 2019 before install of latest update of VS 2019 (do not know if it was 16.9.0 or 16.9.1 causing the break, as I upgraded from the most current 16.8.xx to 16.9.1 skipping 16.9.0)
The debug build portion of the port install appears to work as seen in the attached D:\2019\test\buildtrees\openjpeg\install-x64-windows-dbg-out.log

@StarGate-One: Please report upstream:
the code states:
/* MSVC and Borland C do not have lrintf */
#if defined(_MSC_VER) || defined(__BORLANDC__)

It seems that openjepg has dropped lrintf . We can see that upstream has renamed lrintf as opj_lrintf in openjp2 module.
But it still uses lrintf  in openmj2 module.
As @Neumann-A suggested, we should report this to upstream.
ITK also fixed the same issue for openjpeg embedded in itk. https://github.com/InsightSoftwareConsortium/ITK/issues/1967

@NancyLi1013 /cc @Neumann-A

So the openjpeg is only supporting openjp2.
I would agree we probably need to upgrade openjpeg to v3.4.0, with feature jp2 and leave out the other features jp3d, jpip, jpwl and mj2.
Unless we go through and make changes like mentioned in ITK PR https://github.com/InsightSoftwareConsortium/ITK/pull/2388
But I am confused, as the port with these features still build/compile/install under VS 2017, leading me to think either:
a. The port never really installed these features properly -or-
b. MS changed  how it supports lrintf in this latest update to VS 2019 as it did install prior to VS 2019 16.9.0?

Install on 2017:
config-x64-windows-out.log
install-x64-windows-dbg-out.log
install-x64-windows-rel-out.log

PS D:\2017\test> vcpkg install openjpeg[core,jp3d,jpip,jpwl,mj2]
Computing installation plan...
The following packages will be built and installed:
openjpeg[core,jp3d,jpip,jpwl,mj2]:x64-windows -> 2.3.1#2
Starting package 1/1: openjpeg:x64-windows
Building package openjpeg[core,jp3d,jpip,jpwl,mj2]:x64-windows...
-- Using cached D:/2017/test/downloads/uclouvain-openjpeg-v2.3.1.tar.gz
-- Extracting source D:/2017/test/downloads/uclouvain-openjpeg-v2.3.1.tar.gz
-- Applying patch dll.location.patch
-- Using source at D:/2017/test/buildtrees/openjpeg/src/v2.3.1-ea4b63573b.clean
CMake Deprecation Warning at scripts/cmake/vcpkg_check_features.cmake:182 (message):
calling vcpkg_check_features without the FEATURES keyword has been
deprecated.
 Please add the `FEATURES` keyword to the call.

Call Stack (most recent call first):
ports/openjpeg/portfile.cmake:18 (vcpkg_check_features)
scripts/ports.cmake:142 (include)
-- Configuring x64-windows
-- Building x64-windows-dbg
-- Building x64-windows-rel
-- Using msys root at D:/2017/test/downloads/tools/msys2/50864d9fae363d6e
-- Fixing pkgconfig file: D:/2017/test/packages/openjpeg_x64-windows/lib/pkgconfig/libopenjp2.pc
-- Fixing pkgconfig file: D:/2017/test/packages/openjpeg_x64-windows/lib/pkgconfig/libopenjp3d.pc
-- Fixing pkgconfig file: D:/2017/test/packages/openjpeg_x64-windows/lib/pkgconfig/libopenjpip.pc
-- Fixing pkgconfig file: D:/2017/test/packages/openjpeg_x64-windows/lib/pkgconfig/libopenjpwl.pc
-- Fixing pkgconfig file: D:/2017/test/packages/openjpeg_x64-windows/debug/lib/pkgconfig/libopenjp2.pc
-- Fixing pkgconfig file: D:/2017/test/packages/openjpeg_x64-windows/debug/lib/pkgconfig/libopenjp3d.pc
-- Fixing pkgconfig file: D:/2017/test/packages/openjpeg_x64-windows/debug/lib/pkgconfig/libopenjpip.pc
-- Fixing pkgconfig file: D:/2017/test/packages/openjpeg_x64-windows/debug/lib/pkgconfig/libopenjpwl.pc
-- Installing: D:/2017/test/packages/openjpeg_x64-windows/share/openjpeg/copyright
-- Performing post-build validation
-- Performing post-build validation done
Building package openjpeg[core,jp3d,jpip,jpwl,mj2]:x64-windows... done
Installing package openjpeg[core,jp3d,jpip,jpwl,mj2]:x64-windows...
Installing package openjpeg[core,jp3d,jpip,jpwl,mj2]:x64-windows... done
Elapsed time for package openjpeg:x64-windows: 45.34 s
Total elapsed time: 45.61 s
The package openjpeg:x64-windows provides CMake targets:
find_package(OpenJPEG CONFIG REQUIRED)
target_link_libraries(main PRIVATE openjp2 openjpip openjpwl)

MS changed how it supports lrintf in this latest update to VS 2019 as it did install prior to VS 2019 16.9.0?

seemingly yes. VS updates sometimes add missing intrinsics.
Personally I would just patch the #ifdef to include a _MSC_VER version check.

I created a patch by doing what ITK did:
diff --git a/src/lib/openmj2/opj_includes.h b/src/lib/openmj2/opj_includes.h
index 340e8a9..11bb0e4 100644
--- a/src/lib/openmj2/opj_includes.h
+++ b/src/lib/openmj2/opj_includes.h
@@ -93,7 +93,7 @@ Most compilers implement their own version of this keyword ...

 /* MSVC and Borland C do not have lrintf */
 #if defined(_MSC_VER) || defined(__BORLANDC__)
-static INLINE long lrintf(float f)
+static INLINE long opj_lrintf(float f)
 {
 #ifdef _M_X64
     return (long)((f > 0.0f) ? (f + 0.5f) : (f - 0.5f));

It seems all the features call this same opj_includes.h file, so it allows all features to be installed again.
For some reason the intrinsic function was changed from lrintf to opj_lrintf.
I just do not know why it works in 2017 but not 2019 with the lrintf name? (It just keeps me wondering)
Anyway, the patch I made works on both 2017 and 2019 x64-windows.
I will try to submit PR with patch shortly.


New package: browsh.browsh version 1.8.2

Description

Resolve #104234

Checks

[x] Have you signed the Contributor License Agreement?
[x] Have you checked that there aren't other open pull requests for the same manifest update/change?
[x] This PR only modifies one (1) manifest
[x] Have you validated your manifest locally with winget validate --manifest <path>?
[ ] Have you tested your manifest locally with winget install --manifest <path>?
[x] Does your manifest conform to the 1.5 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Microsoft Reviewers: Open in CodeFlow

@stephengillie - Please remove https://github.com/microsoft/winget-pkgs/labels/Blocking-Issue, as problems from external area is successfully solved.

Hi @Dragon1573,
Our automation readded the label before I could approve, and I lost track of it. Then it closed the PR. Sorry for this outcome. If you can un-delete the branch somehow, I would be happy to review to approve.

The branch issues/104234 is successfully restored.
Thanks for "re-tracking" this issue ❤

It seems the bot keeps violating your control 😄
Is there anything wrong with my contents? Or is they with the bot? Can I close and recreate a brand new pull request on the same base branch Dragon1573:issues/104234?


Missing files in sdist

It appears that the manifest is missing at least one file necessary to build from the sdist for version 1.1.0. You're in good company, about 5% of other projects updated in the last year are also missing files.
+ /tmp/venv/bin/pip3 wheel --no-binary ccy -w /tmp/ext ccy==1.1.0
Looking in indexes: http://<IP_ADDRESS>:9191/root/pypi/+simple/
Collecting ccy==1.1.0
  Downloading http://<IP_ADDRESS>:9191/root/pypi/%2Bf/40b/54135fb2edcf0/ccy-1.1.0.tar.gz (20 kB)
    ERROR: Command errored out with exit status 1:
     command: /tmp/venv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-wheel-edw6af7k/ccy/setup.py'"'"'; __file__='"'"'/tmp/pip-wheel-edw6af7k/ccy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-wheel-edw6af7k/ccy/pip-egg-info
         cwd: /tmp/pip-wheel-edw6af7k/ccy/
    Complete output (9 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-wheel-edw6af7k/ccy/setup.py", line 32, in <module>
        install_requires = requirements("dev/requirements.txt")[0]
      File "/tmp/pip-wheel-edw6af7k/ccy/setup.py", line 18, in requirements
        for line in read(name).split("\n"):
      File "/tmp/pip-wheel-edw6af7k/ccy/setup.py", line 10, in read
        with open(filename, encoding="utf8") as fp:
    FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-wheel-edw6af7k/ccy/dev/requirements.txt'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

This should have been fixed in version 1.2.0, thanks for reporting

Confirmed, thanks!


(PUP-9244) Skip test on non-Windows

The stub was incorrect, but the test passed because we don't typically
run tests as root. Just confine the test to only run on Windows.

CLA signed by all contributors.


Rename brkt_cli.jwt to brkt_cli.brkt_jwt

The brkt_cli.jwt module name was creating problems with our subcommand
loading code.  If the user didn't have pyjwt installed, module loading
silently failed and we wouldn't register the make-jwt command.  This is
because ImportError doesn't give us a way to know which jwt module was
the one that failed to load.
Rename brkt_cli.jwt to brkt_cli.brkt_jwt to avoid this ambiguity.  Also,
change "raise e" in the subcommand loading code to "raise", so that we
preserve the original ImportError traceback.

LGTM


Uncaught SyntaxError: Unexpected token C in JSON at position 0

[Enter steps to reproduce:]

Run auto-correct current file
this error happen. every file, this error happens.

Atom: 1.19.7 x64
Electron: 1.6.9
OS: Mac OS X 10.12.6
Thrown From: rubocop-auto-correct package 1.5.1
Stack Trace
Uncaught SyntaxError: Unexpected token C in JSON at position 0
At file:///Applications/Atom.app/Contents/Resources/app/static/index.html:1

SyntaxError: Unexpected token C in JSON at position 0
    at JSON.parse (<anonymous>)
    at /packages/rubocop-auto-correct/lib/rubocop-auto-correct.coffee:129:27
    at Socket.<anonymous> (/Applications/Atom.app/Contents/Resources/app/src/buffered-process.js:212:17)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at readableAddChunk (_stream_readable.js:176:18)
    at Socket.Readable.push (_stream_readable.js:134:10)
    at Pipe.onread (net.js:551:20)

Commands
  3x -1:07.5.0 core:move-up (input.hidden-input)
     -1:06.8.0 core:select-down (input.hidden-input)
     -1:06.6.0 core:backspace (input.hidden-input)
  2x -1:06.4.0 core:move-up (input.hidden-input)
     -1:05.8.0 core:select-up (input.hidden-input)
     -1:05.2.0 core:backspace (input.hidden-input)
  3x -1:04.9.0 core:move-up (input.hidden-input)
     -1:04.1.0 core:select-down (input.hidden-input)
     -1:03.7.0 core:select-left (input.hidden-input)
  2x -1:03.3.0 core:backspace (input.hidden-input)
     -1:02.6.0 core:move-right (input.hidden-input)
  4x -1:02.3.0 core:move-down (input.hidden-input)
     -1:01.5.0 core:save (input.hidden-input)
     -0:56.5.0 core:move-down (input.hidden-input)
     -0:56.1.0 core:backspace (input.hidden-input)
     -0:55.2.0 core:save (input.hidden-input)

Non-Core Packages
atom-beautify 0.30.5
busy-signal 1.4.3
color-picker 2.2.5
file-icons 2.1.11
git-plus 7.9.3
intentions 1.1.5
language-javascript-jsx 0.3.7
linter 2.0.0
linter-rubocop 2.1.1
linter-ui-default 1.6.8
markdown-preview-plus 2.4.10
pretty-json 1.6.4
rubocop-auto-correct 1.5.1
ruby-slim 0.2.0

Version
$ bundle exec rubocop -v
0.49.1

Hi @sumyapp,
How many times did this error occur? I want to know that this error has occurred every time or sometimes. And let me know where rubocop is installed (e.g. run where rubocop in terminal).
Thank you

Thanks for the reply. Every time.
rubocop installed here.
myname$ which rubocop
/Users/myname/.rbenv/shims/rubocop

And, this is my configure. linter-rubocop is run properly.
  "linter-rubocop":
    command: "/Users/myname/.rbenv/shims/bundle exec rubocop"
  "rubocop-auto-correct":
    autoRun: true
    correctFile: true
    rubocopCommandPath: "/Users/myname/.rbenv/shims/bundle exec rubocop"

@sumyapp Thank you for your investigation.
Quick fix is you change rubocopCommandPath  configuration  to /Users/myname/.rbenv/shims/rubocop.
Because the path of the rubocop-auto-correct command cannot contain arguments.
If you want to use bundle exec strongly, I will be able to apply it. Do you want?

Thank you for telling me.
I tried that. Unfortunately, it did not work in my environment. My environment may be strange. Sorry to trouble you.
The output errors are as follows.
rbenv: rubocop: command not found
The `rubocop 'command exists in these Ruby versions:
2.3.1

I am using multiple versions of Ruby. Also, I use multiple versions of RuboCop.

I got it.
If you launch Atom on terminal (this meaning is Atom knows PATH environment), You do not need to set rubocopCommandPath. I also use multiple ruby environments in rbenv. However, rbenv can automatically resolve path of rubocop command.

I can not make enough time to verify, so I give up once. I am very sorry.
my new config.cson
  "linter-rubocop":
    command: "rubocop"
  "rubocop-auto-correct":
    autoRun: false
    correctFile: true

launch command
$ atom .
# Open ruby file include rubocop warning codes
# linter-rubocop suggest warning
# rubocop-auto-correct appear error(attach the image)

rubocop is installed correctly.
$ bundle install | grep rubocop
Using rubocop 0.49.1
Using rubocop-checkstyle_formatter 0.4.0
Using rubocop-rspec 1.15.1
$ rubocop -v
0.49.1
$ which rubocop
/Users/myname/.rbenv/shims/rubocop

OMG!  Your rubocop installed path is same with me.
$ where rubocop
/Users/tomoya/.rbenv/shims/rubocop

Could you check process.env.PATH value on Atom's DevTools console?
View > Developer > Toggle Developer Tools (Alt + Cmd + i)
In my case, like this.

Is this it?
> process.env.PATH
< "/usr/local/sbin:/usr/local/opt/sqlite/bin:/usr/local/opt/imagemagick@6/bin:/usr/local/opt/openssl/bin:/Users/koichiro/.rbenv/shims:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin"

Hi @sumyapp,  Thank you for your response.
I believe that the cause is rbenv. process.env.PATH includes /Users/koichiro/.rbenv/shims. So, your environment variable is correct.
But your error messages are rbenv: rubocop: command not found and The rubocop command existsin these ruby version. Thus your calling ruby version cannot find rubocop command.
I guess that your environment has several ruby version, and your repository ruby version was not installed rubocop command.
In other words, your repository has .ruby-version file and selected other ruby versions, I guess.
Could you check your ruby version in your repository?  Like a below,
$ where ruby
/Users/tomoya/.rbenv/shims/ruby
/usr/bin/ruby
$ ruby -v
ruby 2.3.3p222 (2016-11-21 revision 56859) [x86_64-darwin15]

Thank you,

Sorry, I think, RuboCop is installed correctly.
Is there a way to check ruby version running rubocop-auto-correct?
$ which ruby
/Users/koichiro/.rbenv/shims/ruby
$ ruby -v
ruby 2.4.2p198 (2017-09-14 revision 59899) [x86_64-darwin17]
$ bundle list | grep rubocop
  * rubocop (0.49.1)
  * rubocop-checkstyle_formatter (0.4.0)
  * rubocop-rspec (1.15.1)
$ rubocop -v
0.49.1
$ rbenv which rubocop
/Users/koichiro/.rbenv/versions/2.4.2/bin/rubocop

I'm seeing this same issue.
Using the default rubocop command path (just rubocop) works OK, but uses the latest installed rubocop version for my current ruby version - therefore I'm seeing a bunch of errors in certain projects that aren't fully up-to-date.
Using a custom rubocop command path (bundle exec rubocop, or /Users/myname/.rbenv/shims/rubocop, or whatever) results in the error described above. I'm not sure how to diagnose the root cause.
As a temporary workaround, just to make this package behave nicely in my project, I'm manually setting the rubocop path to the one needed for this project:
/Users/tomlord/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/rubocop-0.52.1/bin/rubocop


Repackage to hatch/pyproject.toml

This set of changes brings gpio up to the latest standard (or at least the boilerplate I work with) of Python packaging, and includes conveniences like code spellcheck, import sorting and ruff linting in order to maintain code quality and catch bugs early.
TODO

[ ] - ~test in Python 2.7+ for regressions~
[x] - hope the tests pass
[x] - add file mode fixes from #26

Notes
It looks like Python 2.7 support is probably not feasible against newer packaging, we should probably add the last couple of bug fixes in #26 to the 2.7-compatible 1.x.x branch.
sysfs gpio has been obsolete since 2016 and removed from kernels since 2020, with it limping on in forks of Linux mainline -<EMAIL_ADDRESS>It's likely to be dropped from Raspberry Pi OS, too, so significant efforts to maintain this library will probably be redundant.
It's sensible to fixup the 1.0.0 branch for Python 2.7 and then update the packaging and release a 2.0.0 branch for Python 3.7+, with the implication that the 1.0.0 branch is a critical bugfix only legacy branch.
I appreciate legacy Python versions and sysfs gpio probably go hand-in-hand on more esoteric embedded devices, which is why I don't want to break the Python 2.7 library.

LGTM


Update python package

The python package isnt updated to include the  smart_unicode fix which is causing errors

Mind opening a PR?

Just a quick bump that I'd really like for pypy to be updated with the latest fixes. Manually patching the virtualenv stuff for a number of projects is horribly unmaintainable.
PR submitted

The latest version is now up on PyPI: https://pypi.python.org/pypi/django-html_sanitizer


feat: Support for SvelteKit Next 303

Summary
Update Adapter to support the latest version (next@303) of SvelteKit.
Fixes: #153
No support for fancy features like route splitting.
Tested deployment on a live site with Firebase
Other Information
Took some code from the other PR #157 so the author deserves credit.
Implementation details

Uses Server instead of App
Use the generated manifest of the builder instead of SvelteKit
New way of using installFetch
Remove the deprecated prerender call
Thought process in long live video on YouTube

@nielsvandermolen please see the Lint PR failure

Done

Any updates regarding merging this?

Maybe time to fork this project since @jthegedus seems to be permanently AFK. Hope he's ok

Is there anybody who would be willing, and has the time to maintain a fork?
I'd do it but then I've just done https://github.com/evdama/is-it-check which pretty much consumed my FLOSS time budget for this quarter so...

I could help in maintaining the fork as I am needing this library for a project we just started at work and I wasn't aware of all the build issues until just now since local emulation was working splendidly. It seems to me just having a fork of the existing repo with the 2 PRs open on this repo merged in with verification would be sufficient to begin with?

I could help in maintaining the fork as I am needing this library for a project we just started at work and I wasn't aware of all the build issues until just now since local emulation was working splendidly. It seems to me just having a fork of the existing repo with the 2 PRs open on this repo merged in with verification would be sufficient to begin with?

We're using svelte-adapter-firebase at work and am interested in helping co-maintain.
I'm using both PRs right now and have been using SvelteKit 315 without issues.

Not sure what 'best practice' is in these situations, but maybe the fork could make @jthegedus a collaborator, so if he comes back, we don't have an awkward situation where 2 forks are maintained in parallel?

I could help in maintaining the fork as I am needing this library for a project we just started at work and I wasn't aware of all the build issues until just now since local emulation was working splendidly. It seems to me just having a fork of the existing repo with the 2 PRs open on this repo merged in with verification would be sufficient to begin with?

We're using svelte-adapter-firebase at work and am interested in helping co-maintain.
I'm using both PRs right now and have been using SvelteKit 315 without issues.

You say "both" PRs, which other PRs are valid that I need to merge to unblock people?

Sorry all, my time is not my own right now. I was genuine in my prior responses, I was just wrong.
@Nushio I have invited you to collaborate on this repo to help as you seem to be more up to date with the state of things.
I am part of the Firebase Alpha program and there are a few new changes coming soon which may affect how this tool can integrate with peoples existing projects, so I will wait for those to release before reconsidering any of the more complex feature support like functions per route.
I have been considering the utility of a sveltekit-adapter-community GitHub org which might mitigate situations like this, however, I do encourage people to fork in these situations, use privately, and if there ends up being multiple solutions down the road, so be it.

Invite @nielsvandermolen as well? 🚀 🚀 🚀

I could help in maintaining the fork as I am needing this library for a project we just started at work and I wasn't aware of all the build issues until just now since local emulation was working splendidly. It seems to me just having a fork of the existing repo with the 2 PRs open on this repo merged in with verification would be sufficient to begin with?

We're using svelte-adapter-firebase at work and am interested in helping co-maintain.
I'm using both PRs right now and have been using SvelteKit 315 without issues.

You say "both" PRs, which other PRs are valid that I need to merge to unblock people?

Sorry, to be more specific, I tested both PRs, naabu's and co3k's.
Naabu's works all the way to sveltekit-320 (probably 324 if no recent breaking changes, but i'll test tomorrow).
Co3k's doesn't need to be integrated as Naabu's PR already includes fixes for that 'log' error.
I am flattered and really appreciate being a co-maintainer! I've been following your work, jthegus, throughout the github issues and commits on everything related to svelte kit.
I'm excited for whatever Firebase might come up with in the near future. I/O, maybe? Blink twice if yes.

@Nushio #157 is still an issue. Anyone else?


Not able to connect with api

copied files to my local machine everything works fine apart from the api not getting connected.

Failed to load resource: the server responded with a status of 404 (Not Found)

This repo in its current form is a few years old so yeoman and the other dependencies may opperate differently now. To debug this issue you'll need to look at how the  grunt file is including folders, such as the api folder.

I am actually new to angular+slim. Should I share the grunt file here? Could you please help me debug ?

This wouldn't much to do with either Angular or Slim and more to do with the http server, connect, watch and so on... all controlled by Grunt and by extension, node.js.
Follow the instructions in the README and where your grunt file differs from what I've documented, show those changes here.


start server failed

I tried to use wasmedge --env "DATABASE_URL=mysql://user:passwd@<IP_ADDRESS>:3306/mysql" order_demo_service.wasm  to run the program, but the above error occurred, how can I solve it ?

You already have another server listening at that port.

Thanks, successfully solved the problem.


Fix pthread

Currently pthread is using clone_syscall, which is implemented using Asyncify. There is an issue on current approach using Asyncify to implement pthread (i.e. unwind/rewind two wasm within same memory at the same time): The variable to control the unwind/rewind state is a global variable that is shared by parent and child threads, which means they will compete for the unwind/rewind state. I am actually pretty surprised that the pthread could work kind stably without triggering the issue previously, until I finally observed the issue.
To resolve the issue might be a little tricky, there are several ways I currently can think of to resolve the issue:

Try to syncronize the parent and child thread on asyncify state (i.e. use barrier to make parent and child always at the same asyncify step). This may add additional overhead on pthread creation and could still potentially have issues on other aspects.
Try to make the asyncify state global variable be non-shared for parent and child: create a non-shared memory region for threading that is used to store some per-thread global variables and make the asyncify state global variable sits in there. Since we now have vmmap, this approach shouldn't be hard to implement, but I am not sure if that aligns with our memory management architecture.
Fallback to wasi-thead implementation of pthread

Let's also fix epoch interruption to work with multi-processing here. Should be able to give us much more control for exiting.

Closing this issue and creating a separate issue for epoch_interruption.


dexie-cloud: experiment 1, verbose logs and some patches to try and isolate the issue

Didn't mean to open this here! Sorry for the noise. I was trying to create a diff over on my fork.

Didn't mean to open this here! Sorry for the noise. I was trying to create a diff over on my fork.


add back bundle polling

closing because this PR existed for testing purposes only; bundle polling should only be added back in when it's more fully-featured


[POC] Build Linux System package on aarch64

https://github.com/beeware/briefcase/pull/1360#issuecomment-1658571722

PR Checklist:

[ ] All new features have been tested
[ ] All new features have been documented
[X] I have read the CONTRIBUTING.md file
[X] I will abide by the code of conduct

Closing. GitHub has arm64 Linux runners in private beta; when that reaches GA, I'll revisit this.


Fix checkstyle windows newline detection

@junlong4321 Would appreciate if you could help us test on Windows platform.

@ypinhsuan can yours work? my one works fine

Yep works on mine too


Webserver Authentication

Enables optional web authentication specified in https://airflow.incubator.apache.org/security.html#web-authentication.
Can be passed via environment variables in compose files.
Thanks!

Found a less intrusive way to do this, i.e. spin up an nginx instance or spin up a separate service that depends on webserver (to call initdb) and add a user there.


Backtest data from Fxcm

In backtest mode the system  looks for historical data into  ..Data/Forex/Fxcm/  folder. How I  change it to use another broker?.   Thanks.

We break data into markets; e.g. you can have AAPL stock in the Mexican and the US exchanges as two separate tradable contracts. This is even more apparent with FX brokerages who often are market makers themselves and have their own contract pricing.
This is controlled in the Market.X enum; you specify which market you'd like when you AddForex("EURUSD", Resolution.Minute, Market.Oanda / Market.Fxcm); You could add your own brokerage and add an entry to the market enum here.
If you add a new brokerage implementation please submit it as a PR!
PS: We provide all the FXCM and Oanda data for download via our data library on quantconnect.com/data.


Incorrect detection of Vue version

Hi, I was trying to use @vueuse/core when I ran into an issue with this package. I'm using npm 7.0.3, node 15.0.1 and my package.json looks like this:
{
    "private": true,
    "scripts": {
        "development": "mix",
        "watch": "mix watch",
        "watch-poll": "mix watch -- --watch-options-poll=1000",
        "hot": "mix watch --hot",
        "production": "mix --production"
    },
    "devDependencies": {
        "@vue/compiler-sfc": "^3.0.2",
        "autoprefixer": "^10.0.1",
        "axios": "^0.19",
        "cross-env": "^7.0",
        "laravel-mix": "^6.0.0-beta.11",
        "lodash": "^4.17.19",
        "postcss-import": "^12.0.1",
        "postcss-nested": "^4.2.3",
        "postcss-simple-vars": "^5.0.2",
        "resolve-url-loader": "^3.1.0",
        "sass-resources-loader": "^2.1.1",
        "tailwindcss": "^1.9.6",
        "tailwindcss-filters": "^3.0.0",
        "vue": "^3.0.2",
        "vue-loader": "^16.0.0-beta.10",
        "vue-router": "^4.0.0-rc.1",
        "vuex": "^4.0.0-rc.1",
        "@vueuse/core": "^4.0.0-beta.38"
    }
}

I'm using Laravel Mix to compile all the assets (I'll be using Laravel as a backend). Everything was going fine until I tried to use any function from the vueuse package.
The console error looked like this:

Going thru the module code I see some lines are overwritten depending of the Vue version and it looks like the module thinks I'm on version 2 of Vue. Manually changing those lines to the 'v3' fixed the issue. I'm not sure this is Laravel Mix fault (webpack wrapper) or not. Let me know how can I help fixing this issue.

@gerardnll Which package manager you are using? Can you try to run npx vue-demi-fix to see if it fixes your issue?

I'm using NPM. where am I supposed to run 'npx vue-demi-fix'? because executing that line in my project root throws:
npm ERR! code E404
npm ERR! 404 Not Found - GET https://registry.npmjs.org/vue-demi-fix - Not found
npm ERR! 404
npm ERR! 404  'vue-demi-fix@latest' is not in the npm registry.
npm ERR! 404 You should bug the author to publish it (or use the name yourself!)
npm ERR! 404
npm ERR! 404 Note that you can also install from a
npm ERR! 404 tarball, folder, http url, or git url.

What was weird. Did you bump the version of Vue from 2 to 3 after installing @vueuse? Could you try to remove your lockfile and node_modules and reinstall again?

I started the project with those dependencies and I added vueuse the last one.
Anyway, now I just get a warning (files are still pointing to v2 and app is broken)

There's a whole bunch of warnings when 'npm i':

Can you share a minimal reproduce repo? It seems to work fine on my Vue 2 and Vue 3 projects.

OK. But first, please, correct me if I'm wrong but, when I do npm i I guess the postinstall command of vue-demi is executed, right? If so, I'm just installing these:
{
    "private": true,
    "devDependencies": {
        "@vueuse/core": "^4.0.0-beta.38",
        "vue": "^3.0.2"
    }
}

and the content of<EMAIL_ADDRESS>is:
export * from './v2/index.esm'
Is that how it's supposed to look?

Copied your code and did a clean install. It looks fine to me

Maybe try to upgrade your npm?

I had to downgrade npm. But now, this warning appears and there's no longer this file<EMAIL_ADDRESS>
I think NPM changed some things with version 7. Take a look at this document  in the section 'Breaking changes' I think there's some breaking things that this package is using, like 'require' and maybe even this affects.

@gerardnll I see. I have heard that npm 7 will install the peerDepenencies automatically. I will have a look at this to see if we can come up with a solution. Thanks.

Thanks! Ping me if you need some help testing.

I have released v0.4.3 and it should work for npm 7. Please give a try.

Yes. Looks like it's working on my main project (where I do the compiling). Great!


Don't compatible with 1.13.2

Hello, first of all i'm really interested in the good developpement of the plugin, good idea with good features, but sadly my server is on 1.13.2 and based on my console, the usage of the oak_sign is hardcoded, but it didn't exist in 1.13.2 so it crash.
Is it possible to put the material in the config, or at least using generic material and checking the version, thx for all.

https://github.com/7sat/SSDynamicShop/commit/458e66e7bb740c3efe1b86661904e38c3d0bb0af


[1pt] Error in identify_src_bankfull.py

I had some inundation TIFs in my FIM directory that was being overwritten and identify_src_bankfull.py thought it was a directory. Just add a check in there to make sure that it's a directory.
Estimating bankfull stage in SRCs
Writing progress to log file here: /logs/log_bankfull_indentify.log
This may take a few minutes...
Traceback (most recent call last):
  File "/foss_fim/src/identify_src_bankfull.py", line 436, in <module>
    run_prep(fim_dir, bankfull_flow_filepath, number_of_jobs, verbose, src_plot_option)
  File "/foss_fim/src/identify_src_bankfull.py", line 344, in run_prep
    for branch_id in os.listdir(huc_branches_dir):
NotADirectoryError: [Errno 20] Not a directory: '/outputs/hydrocond/01080107_2yr_fim.tif/branches'

I'm a regex fanatic, so I would change line 337 from this:
huc_list = os.listdir(fim_dir)
to this:
huc_list = [d for d in os.listdir(fim_dir) if re.match(r'\d{8}', d)]


Proposal for redesigning Boots:

Description

Boots is burning for clearer design and scope, organization and structure, and flexibility of operation.
Discrete libraries will provide clarity of scope and allow for Boots to focus on flexibility of operation.
See the README.md in this commit for more details.
rendered proposal: https://github.com/jacobweinstock/proposals/blob/0022-rearch-boots/proposals/0022/README.md
Why is this needed

Fixes: #
How Has This Been Tested?

How are existing users impacted? What migration steps/scripts do we need?

Checklist:
I have:

[ ] updated the documentation and/or roadmap (if required)
[ ] added unit or e2e tests
[ ] provided instructions on how to upgrade

View rendered proposals/0022/README.md

This is awesome! Thank you!


querying notebooks with SQL migration

Describe your changes
Migrated this doc to use ploomber-engine from ploomber.
Issue ticket number and link
Closes #206
Checklist before requesting a review

[x] I have performed a self-review of my code
[x] I have added thorough tests (when necessary).
[x] I have added the right documentation (when needed). Product update? If yes, write one line about this update.

:books: Documentation preview :books:: https://sklearn-evaluation--235.org.readthedocs.build/en/235/

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.06%) to 94.259%

Totals

Change from base Build<PHONE_NUMBER>:
0.06%

Covered Lines:
2660

Relevant Lines:
2822

💛  - Coveralls

@gtduncan I think it's missing some file?
FileNotFoundError: [Errno 2] No such file or directory: 'data.ipynb'```

Yeah, my approach was to have the user manually write .ipynb files in this example because with ploomber the code was able to run code and automatically generate the necessary dependencies. Here, to start, I have the user create notebooks and ploomber-engine executes them, but the CI doesn't seem to like that. I wasn't able to figure out how to automatically generate .ipynb files in a ploomber-engine example... is there a way to do this?

You can save the notebook as part of the code as it was previously, either as a separate notebook (comes built in):
curl https://raw.githubusercontent.com/ploomber/ploomber-engine/main/examples/display.ipynb \
    --output running-demo.ipynb```

or via a cell, output the code into `.ipynb`:

```python
Path("my-nb.ipynb").write_text("""
THIS IS MY RAW PYTHON NOTEBOOK
""")

@gtduncan also, check out jupytext, it can read the .py formatted strings that appear as examples in the documentation and export as JSON which is the format that .ipynb uses, then you can pass it to ploomber-engine

@gtduncan it still seems to fail on this?
FileNotFoundError: [Errno 2] No such file or directory: 'data.ipynb'

Now just failing as a result of the confusion matrix error

I see we have these blank cells which are weird. Is it possible to remove them?

this would imply changes to the internal implementation, I think we should address this but in a different PR

@gtduncan are the review issues fixed?
I think for sanitation, when a discussion is resolved click resolve so it's obvious what's done and what's pending.

CI working now— I implemented all the necessary changes I believe


istio-agent.deb, istio-auth-node-agent.deb and istio-proxy.deb not published for 0.5.*

I tried following the directions for Mesh Expansion, but it appears istio-agent.deb, istio-auth-node-agent.deb and istio-proxy.deb are not published:
$ # Detailed output for a single deb
$ curl https://storage.googleapis.com/istio-release/releases/0.5.1/deb/istio-agent.deb
<?xml version='1.0' encoding='UTF-8'?><Error>NoSuchKey<Message>The specified key does not exist.</Message></Error>

# All three packages for 0.5.1 are missing
$ for pkg in istio-agent.deb istio-auth-node-agent.deb istio-proxy.deb; do curl -s -L -o /dev/null -w "%{http_code}\n" https://storage.googleapis.com/istio-release/releases/0.5.1/deb/$pkg; done
404
404
404

# All three packages for 0.5.0 are missing
$ for pkg in istio-agent.deb istio-auth-node-agent.deb istio-proxy.deb; do curl -s -L -o /dev/null -w "%{http_code}\n" https://storage.googleapis.com/istio-release/releases/0.5.0/deb/$pkg; done
404
404
404

# However, all packages for 0.4.0 are present.
$ for pkg in istio-agent.deb istio-auth-node-agent.deb istio-proxy.deb; do curl -s -L -o /dev/null -w "%{http_code}\n" https://storage.googleapis.com/istio-release/releases/0.4.0/deb/$pkg; done
200
200
200

As of 0.5 there's only one debian (istio-sidecar.deb) rather than the 3.  It looks like the URLs in istio.VERSION wasn't consolidated but it looks like 0.6 on on-track to have this.  I don't know if a doc update is on anyone's radar but I suppose this issue could serve that role.

The 0.8 doc for sure had been updated.
https://istio.io/docs/setup/kubernetes/mesh-expansion/


csv-parse pipe() does not close file

I have version 1.2.3.
I am not sure if this is by design or not, but I recently ran out of file descriptors and traced it to this code snippet in my code:
...
	console.log('getCsvLines', filepath);
	var csv = fs.createReadStream(filepath).pipe(parse({from: 1, ltrim: true, rtrim:true, columns: true}));
	var lineNo=1; // skip header
	csv.on('data', (row) => {
	    /* do magic */
	});
	csv.on('end', () => { console.log('csv error'); resolve(out); });
	csv.on('error', (e) => { console.log('csv error'); reject(e); });
	csv.on('close', (e) => { console.log('csv close'); })
});

I was expecting that the file would have been automatically closed (autoClose is true) when the csv emitted end, because the file should also be at end and invoke the close call, but this does not happen. I have to split it:
var file = fs.createReadStream(filepath);
var csv = file.pipe(parse({from: 1, ltrim: true, rtrim:true, columns: true}));

then explicitly file.close() on csv.end, otherwise it never happens:
csv.on('end', () => { console.log('csv end'); file.close(); resolve(out); });
file.on('close', (e) => { console.log('file close'); })

Are my expectations wrong, or is this a bug?

I am running into the same issue. When piping parse() the finish event is never called but stringify() works. I commented the transformStream but that didn't make any difference.
The other strange thing that I noticed is that if I use your transform module instead of the native one, then the event is fired.
this.reader
				.getReadStream()
				.pipe(parse())
				.pipe(transformStream)
				.pipe(stringify())
				.pipe(this.reader.getWriteStream())
				.on('error', (e) => {
					reject(e);
				})
				.on('finish', () => {
					console.log('this gets finished');
					resolve(this.reader.getWritePath());
				});
		});

Unless I'm mistaken, this is agnostic to the parser and relate to Node.js internal stream feature. When only implement the transform function and have no control over firing the close event of the final stream writer. Feel free to re-open if you prove me wrong.


side effect of sum on dict values

meaningless example, real bug
x = IntVar(0,10, 2)
y = IntVar(0,10, 2)
cnt = dict()
for i in range(len(x)):
for j in range(len(y)):
cnt[(i,j)] = sum([ x[i] == x[j], x[i] == x[j]  ]) # just for the example...
print(cnt[(0,0)])
m = Model()
m.objective = sum(cnt.values())
print(cnt[(0,0)])

more generally,
v = intvar(0,9, shape=2)
w = intvar(0,9, shape=2)
sum_v = sum(v)
m = Model(5 == sum_v+sum(w),
sum_v < 5)
unsat, because 'sum_v' is chained with 'sum(w)' in the first constraint...
Using 'sum_v' should not change its expression, so we must rework 'sum' such that it creates a new array instead of appending to the previous sum if existing (with some performance overhead...).


make padding routines faster

Since i have nothing better to do on a sunday night, I figured i might as well look into why it takes so long to process data before we can train (inspired by a conversation with Mark on friday afternoon).
To start off, I refactored the pad_sequence_to_length function and made it faster by using list.extend versus repeated list.append. While the amortized runtime of list.append is O(1), you lose a ton of efficiency by going back and forth from the python / C levels. On the other hand, list.extend does all of the iteration in C, which makes it a lot faster.
I'll also look into refactoring how we make word + character paddings, because that's also torturously slow right now.
Benchmarks will come in a separate comment.

In [1]: from typing import Any, Callable, Dict, List

In [2]:     def old_pad_sequence_to_length(sequence: List,
   ...:                                desired_length: int,
   ...:                                default_value: Callable[[], Any]=lambda: 0,
   ...:                                truncate_from_right: bool=True) -> List:
   ...:
   ...:         padded_sequence = []
   ...:         for _ in range(desired_length):
   ...:             padded_sequence.append(default_value())
   ...:         sequence_length = min(len(sequence), desired_length)
   ...:         if sequence_length != 0:
   ...:             if truncate_from_right:
   ...:                 padded_sequence[-sequence_length:] = sequence[-sequence_length:]
   ...:             else:
   ...:                 padded_sequence[:sequence_length] = sequence[:sequence_length]
   ...:         return padded_sequence
   ...:

In [3]:     def new_pad_sequence_to_length(sequence: List,
   ...:                                    desired_length: int,
   ...:                                    default_value: Callable[[], Any]=lambda: 0,
   ...:                                    truncate_from_right: bool=True) -> List:
   ...:         if truncate_from_right:
   ...:             truncated_or_extended = sequence[:desired_length]
   ...:         else:
   ...:             truncated_or_extended = sequence[-desired_length:]
   ...:         if len(truncated_or_extended) < desired_length:
   ...:             truncated_or_extended.extend([default_value()] *
   ...:                                          (desired_length -
   ...:                                           len(truncated_or_extended)))
   ...:         return truncated_or_extended
   ...:

In [4]: %timeit old_pad_sequence_to_length([0]*50, 700)
1000 loops, best of 3: 239 µs per loop

In [5]: %timeit new_pad_sequence_to_length([0]*50, 700)
100000 loops, best of 3: 7.66 µs per loop

In [6]: %timeit old_pad_sequence_to_length([0]*300, 700)
1000 loops, best of 3: 241 µs per loop

In [7]: %timeit new_pad_sequence_to_length([0]*300, 700)
100000 loops, best of 3: 8.88 µs per loop

so, looks like it's ~30 times faster to do it this way...ran this on a p2.xlarge instance, since that's what we're running the models on anyway.

hmm, perhaps the method doesn't work as I thought it did. it's really odd to me that setting the truncation mode would affect the end to which padding is added --- is this intended?
(this is the current behavior):
In [4]: pad_sequence_to_length([1,2,3,4,5], 2)
Out[4]: [4, 5]

In [5]: pad_sequence_to_length([1,2,3,4,5], 8)
Out[5]: [0, 0, 0, 1, 2, 3, 4, 5]

In [6]: pad_sequence_to_length([1,2,3,4,5], 8, truncate_from_right=False)
Out[6]: [1, 2, 3, 4, 5, 0, 0, 0]

In [7]: pad_sequence_to_length([1,2,3,4,5], 2, truncate_from_right=False)
Out[7]: [1, 2]

I think it would make a lot more sense to either pad on the left or on the right always, and you could probably make this a constructor param. thoughts? If there's a willful reason behind this, I'd be interested in hearing it.
In addition, i would have thought that truncate_from_right=True would mean that you would start chopping off the array starting from the right, when in fact the opposite is true...
Regardless, i've modified this PR to have the current behavior. Running the benchmarks again on the new code yields results that are almost the same as before.

also, is there a reason why we wrote our own tokenization / string processing functions? Those are also a major bottleneck...why not use something like NLTK?

oh interesting, just saw this https://github.com/allenai/deep_qa/blob/master/deep_qa/data/tokenizers/word_splitter.py#L20

Does really simple tokenization. NLTK was too slow, so we wrote our own simple tokenizer instead.

looks like spaCy blows them both out of the water (this is on a 569 token string). Perhaps it would be worth trying out a spaCytokenizer? Should be fairly trivial to implement.
In [15]: %timeit [token for token in en_nlp.tokenizer(lorem.lower())]
1000 loops, best of 3: 522 µs per loop

In [16]: %timeit SimpleWordSplitter().split_words(lorem.lower())
100 loops, best of 3: 5.63 ms per loop

In [17]: %timeit NltkWordSplitter().split_words(lorem.lower())
100 loops, best of 3: 12.7 ms per loop

It's been on my TODO for a long time to look into this.  Thanks for doing it.  I haven't actually looked at the code yet, because it doesn't look like you want a review yet; just responding to your PR comments.
It's actually pretty important to keep the option to pad from either side.  Sometimes you expect the most important content to come at the end of a sequence, so if you have to shorten it, you should throw away the beginning (this is true of question text).  Other times you want to throw out the end (this is true of news articles in WDW).
And, yeah, feel free to add the spacy tokenizer; looks like it's a faster, better option than what I wrote.  I also thought I did a little better than just 2x NLTK, but I guess not... =)

I also think the location of the 0's might impact Keras' LSTM, though I'm not sure about that - Mark might know something more there.

This is ready for a review. SpaCy's tokenizer is a lot faster, but it also is very conservative. E.g. tokenizing Mr. John Smith will give [Mr., John, Smith] as expected, but tokenizing mr. john smith will give [mr, ., john, smith]. This means that the max lengths can be quite a bit larger than usual. I mentioned this on their repo, hoping to get a response. I think it's a worthy addition anyway, but I didn't set it as the default. if it had the same behavior as NLTK or something i would.

What if you tokenize, then lower?  Does it work alright?

yeah, that would work but i'm not sure if we'd lose the speed benefit (which is the reason we're using it in the first place) by going through and lowering everything again. i suppose ill run some benchmarks.

surprisingly, turns out to be almost the same. i suppose i'll do that.

on WDW, the max passage length is 3090 with NLTK / SimpleWordTokenizer and 3539 with SpaCy. hmm...

You might try actually inspecting that passage - could be some rare case that we don't really care about anyway, and we should just be truncating.


Introduce DecodeStreamer for Client.Do

This enables a client to stream values out of the HTTP response. It does
so by providing a result that implements DecodeStreamer with something
like the following.  Error handling has been omitted for clarity.
// DecodeStream decodes and processes each record of a JSON array
func (streamingDecoder) DecodeStream(r io.Reader) error {
    dec := json.NewDecoder(r)
    tok, _ := dec.Token() // want: json.Delim('[')
    for dec.More() {
        _ = dec.Decode(&record)
        // do something with record
    }
    tok, _ = dec.Token() // want: json.Delim(']')
    _, err = dec.Token() // want: io.EOF
    return nil
}

I'm open to name changes, or adding tests or docs if you point me to where they'd go. I have tested this with an implementation that looks like the above, and it allows me to receive records before the body is closed.

Hello @MichaelUrman
Thank you for bringing streaming to the table!
I've reviewed the changes, I was wondering if https://github.com/grafadruid/go-druid/blob/aef53699c90801d5224f7946c7dc1f03a46b60c8/druid.go#L144 could have an impact but most of all I'm wondering what should be the streaming api ?
Should:
var results interface{}
_, err = d.Query().Execute(qry, &results)

become something like:
var recordHandler func(record json.RawMessage)
_, err = d.Query().Stream(qry, recordHandler)

then decoder becomes part of the library, and then it opens the door to questions like "does each type of response from druid share the same structure? should we bring multiple decoders ? one per query type ?"

Hi @jbguerraz, thanks for reviewing!
Yes, that defer resp.Body.Close() prevents simply passing a nil result to Do, then decoding the returned response's body later. I didn't remove it because any existing callers of Do(r, nil), would stop closing Body. This approach was the least intrusive I could come up with, and required only the same knowledge as choosing a good type for result already required.
For your alternate d.Query.Stream entry point, are you proposing the func(json.RawMessage) be called once for each entry in the array, allowing the implementation to decode individual entries with json.Unmarshal([]byte(record), &result)? I think that would work as well.
As tradeoffs, the approach currently in the PR enables the caller to use any JSON parser, and makes it easy to exit early, but by the same token it requires understanding the format of Druid's response. I think your proposed approach requires some use of encoding/json (to declare the callback), but could partially abstract Druid's response formats.
A third approach might mix these: a new API calls a callback/interface with an interface { io.Reader; Next(result interface{}) }; that is the caller can use it like an io.Reader, or read individual records. I think this approach may have too much complexity for its added flexibility.
Let me know if you need me to prototype another approach. I'm fairly inexpert in Druid, so can't yet speak towards its other response formats, and whether they're compatible with a func(json.RawMessage) callback.

Hi @jbguerraz and any other members,
This is just a ping to see if there's anything I can do to further facilitate landing some sort of data streaming in go-druid. I'm willing to prototype other approaches, or to refine this one, if it will help suit your vision.

Hello @MichaelUrman
Didn't forget it. I need to find a moment to check the different response formats and so get a broader view.
Gonna give it a try in the weekend I believe. Thank you for your patience. If somehow this is critical for you we also can move ahead that way and iterate right after.

Thanks! While I strongly expect to need streaming, I'm not blocked on it. So I'd rather you have a chance to get it right, and not have to live with the remnants of the "wrong" implementation.
(If I need it before it's ready here, there's always go mod replace. 😄)

Hello @MichaelUrman
I've checked it out a bit more :)
Maybe we could have a Query().Stream(builder.Query, chan json.RawMessage) (instead of Query().Execute(builder.Query, interface{}) that use (a new) Client.Stream (instead of  existing Client.Do).
builder.Query would have a DecodeStream method or so (that would implement the row extraction logic, for each of the query https://github.com/grafadruid/go-druid/tree/master/builder/query , similar to https://github.com/grafadruid/druid-grafana/blob/master/pkg/druid.go#L425 ). That way the stream client would receive a json raw message per row, in a channel it provides.
We could start by providing the Client.Stream method which basically would be the same feature than this but without (ab)using execute / do methods. WDYT ?

Apologies for the length of these thoughts. I'm not sure how to condense it in the time I have available. I hope it's at least clear. As always, if you'd like me to prototype some of this, let me know.
I believe we're considering some implicit questions here, and would like to try to make them explicit:

How to request streaming?
Who implements the streaming?
If go-druid, how does it implement the streaming?

Regarding 1: The PR proposes an implicit approach by testing for an interface, and you've proposed an explicit separate call. I'm assuming some of the worry about implicit is a caller that already "accidentally" implements the interface, breaking compatibility.  Or that a typo in implementing the interface could result in hard to diagnose lack of its use. One might address these by introducing a new go-druid type to wrap the interface, check specifically for that type.
Regarding 2: This PR gave all responsibility to the caller; this is powerful and liberating, but potentially repeats a lot of complexity. You've proposed having go-druid implement the streaming which avoids this. However, having go-druid implement this requires choosing an API for it.
So regarding 3: An API for streaming leaves us with several suboptimal options, due to limitations in go's syntax.

Returning a helper: pros include ease of reading; cons include a requirement to close it
Returning a channel (a special case of returning a helper, or could be the internals of a helper)
Accepting a callback or interface: pros include lifetime management of Body; cons include indirectness and limitations on the callback code (The PR's approach is similar to this, but the caller controls the loop so can break early.)
Accepting a channel (similar to a callback, but likely requires the caller start a goroutine)

I'm skeptical of exposing channels at all, as they are easy to misuse, for instance by not consuming all rows from it. I'm strongly skeptical of having the API accept a channel, as the ownership is backwards: typically the producer creates, sends, and closes the channel; but here the consumer would create it and the producer would send and close.
With that framework in mind here's what I think about your proposed Query().Stream(builder.Query, chan json.RawMessage). I like that it's explicit, and go-druid owns the complexity of streaming and understanding the JSON. I don't like the channel in the API, especially with its unusual ownership semantics, and I don't like being tied to encoding/json. Also I'm too uninformed about what would be in this message (but that's just as true with the PR's approach).
By contrast, I don't love how this PR's approach is so implicit and requires both heavy lifting and knowledge of the underlying response format, but I do like its ownership semantics and ability to use other JSON parsers. (Though it seems go-druid already requires knowledge of the response format when creating a struct for result.)
(Note that the two proposals can be combined. An explicit go-druid interface-wrapping type can be the building block upon which your proposed additional API is built. Then if the interface-wrapping type is exported, it can be used by those with unusual needs.)


Bars going out of left Y axis in Grouped Bar chart with Brush when it is extended. Clip path not working.

I am trying to design a grouped bar chart with a course range selector(or brush feature)across the bottom. Brush is working fine but the bars move to left , crossing the left y-axis when it is extended. How can I avoid it. Same is the case if I use scatter plot.
I tried following the clip path code which is there and working for line chart in group bar and scatter charts, but see no fix.
Please help
KR

Thanks for the report! I will have a look

Hey sorry for the late reply!
Could you show me a code sample so that I can reproduce the problem?
I didn't have time to create one yet.
But I have an idea what could be the problem.
Again sorry for the late reply.


Hiding "Share" option in list command bar using JSON is not working

What Should Happen
I am trying to hide "Share" option in list command bar using JSON formatting. I have tried below JSON in view formatting but it is not hiding the "Share" option.
{
  "commandBarProps": {
    "commands": [
      {
        "key": "share",
        "hide": true
      }
    ]
  }
}

I have also tried this JSON specifying all selectionModes for testing.
{
  "commandBarProps": {
    "commands": [
      {
        "key": "share",
        "hide": true,
        "selectionModes": [
          "SingleSelection",
          "MultiSelection",
          "NoSelection"
        ]
      }
    ]
  }
}

What Actually Happens
As per the Command bar customization syntax reference documentation, "Share" option should be hidden from command bar when using any one of the above JSON codes.
Screenshot related to this issue:

I was able to hide Share in the command bar in my environment!

Possible issue hidding menu options -> @shagra-ms
I have the same issue as described by @ganesh-sanap.
Below associated gif.

Thanks @ganesh-sanap @tecchan1107 @aaclage for sharing more repro details. We have a local repro now.

@tecchan1107 Still not working for me. I tried it on 3 different tenants (2 standard release tenants and 1 targeted release tenants). Same issue on all tenants.

It works in the tenant I used for testing.

Hi, I have the same problem in my test tenant, in production the code works as expected.
The only difference is the external sharing configuration which is enabled in the test tenant

Tested in a production tenant and it didn't work as expected.

If you notice where this works and hides the Share button is where that button does not have a drop down.  In our tenant, we have under Share > Share Link , and Manage Access.  So Share now has a drop down, the "share" hides share link now instead of the entire button.  So you are left with Share and a drop down.
We need to identify how to hide the new Share button and Manage Access.  Tried all sorts of variations in JSON but nothing worked.

This also appears to stop working with the new Microsoft Lists interface


[vm/ffi] Constructing struct classes from load()/ref in new structs API

Struct classes are allowed to define constructors, so long as they don't contain field initializers.
However, when loading a struct reference from a pointer, we need to manufacture an instance of the struct without a reference to any constructor. The proposed design is to skip any user-defined constructors and initialize the object via a synthetic one-argument constructor (which takes the target address).
Other options include allowing or requiring the user to define a constructor (with the correct signature) which would be called instead.
Of course, the user can always invoke their own constructors directly, which would be responsible for calling the super constructor (of Struct) with a Pointer for the backing memory.

We have converged on a design taking finalizers and structs by value into account.


[BUG]: nanobind wrapping of Python-arrays slower than pybind11

Problem description
In DOLFINx, we are currently seeing some major performance regressions when using nanobind instead of pybind:
https://github.com/FEniCS/dolfinx/issues/2891
I've tried to dig into what is wrong, but haven't been able to explain the 2x slowdown with nanobind.
When trying to isolate the issue, I just compared wrapping a numpy array with nanobind and pybind, code available at:
https://github.com/jorgensd/nanobind_example/tree/dokken/nano_vs_pb
i.e
Pybinding
#include <pybind11/numpy.h>
#include <pybind11/pybind11.h>
#include <pybind11/pytypes.h>
#include
namespace py = pybind11;
namespace
{
    template <typename T>
    void spanned_function(std::span<T> b)
    {
        std::for_each(b.begin(), b.end(), [](auto &val)
                      { val *= -2; });
    }
}
PYBIND11_MODULE(pybind11_example_ext, m)
{
    m.def(
        "add_vector",
        [](py::array_t<double, py::array::c_style> b)
        {
            spanned_function<double>(std::span<double>(b.mutable_data(), b.size()));
        },
        py::arg("b"));
}

nanobinding
#include <nanobind/nanobind.h>
#include <nanobind/stl/complex.h>
#include <complex.h>
#include <nanobind/ndarray.h>
#include
namespace nb = nanobind;

using namespace nb::literals;

namespace
{
    template <typename T>
    void spanned_function(std::span<T> b)
    {
        std::for_each(b.begin(), b.end(), [](auto &val)
                      { val *= -2; });
    }
}
NB_MODULE(nanobind_example_ext, m)
{
    m.def(
        "add_vector",
        [](nb::ndarray<double, nb::ndim<1>, nb::c_contig> b)
        {
            spanned_function<double>(std::span<double>(b.data(), b.size()));
        },
        nb::arg("b").noconvert());
}

Running the code in the reproducible example below, I get the timings:
Nanobind: 5.960962e-02
Pybind: 5.023153e-02

for an array of size 100 million, and
Nanobind: 5.951924e-01
Pybind: 5.107849e-01

with a 1 billion array.
As I stated earlier, I know this is not the cause of the regression (but could be a part of it).
Do you have any insight for us?
The binding that we are currently timing is: https://github.com/FEniCS/dolfinx/commit/0edd19f3cbed5d9acc911610aef4703410123dad#diff-97f6b59b7db9ef802d0a9cff4b9a3780864c3aa8c5efe78a920a7a04c428714aL147-L162
Reproducible example code
import nanobind_example as m
import numpy as np
import time

arr = np.arange(10_000_000, dtype=np.float64)

def test_vector():
    arr_nb = arr.copy()
    start = time.perf_counter()
    m.add_vector(arr_nb)
    end = time.perf_counter()
    print(f"Nanobind: {end-start:5e}")

    arr_pb = arr.copy()
    start = time.perf_counter()
    m.add_vector_pb(arr_pb)
    end = time.perf_counter()
    print(f"Pybind: {end-start:5e}")
    assert np.allclose(arr_nb, arr_pb)

At least for the regression seen in DOLFINx, we have found a resolution: https://github.com/FEniCS/dolfinx/pull/2895#issuecomment-1817294687
I will test is for the case above today. Could you comment on https://github.com/FEniCS/dolfinx/pull/2895#issuecomment-1817294687?

If you are doing heavy computations in your bindings themselves rather than the more typical setup of forwarding calls to a that does the heavy lifting, then NOMINSIZE is probably an option you will want to add.

If you are doing heavy computations in your bindings themselves rather than the more typical setup of forwarding calls to another library that does the heavy lifting, then NOMINSIZE is probably an option you will want to add.

In the wrapper itself in DOLFINx, we do not do heavy lifting, it calls the DOLFINx C++ code that has been compiled.

Maybe the issue is that some of the functions we wrap from dolfinx are templated, so they are compiled together with the bindings.

In any case, as long as your performance regression disappears with NOMINSIZE, it seems to me that this issue can be closed.