Github Enterprise SSH handshake failing

Describe the bug
We use flux2 with our github enterprise instance. This worked fine until the 0.28.3 Release. Since the Update we cannot fetch our repos anymore and get the following error message:
k get gitrepositories -n flux-system

NAME         URL                                                              AGE   READY   STATUS
flux2-sync   ssh://git@foo/abc/def   64d   False   failed to checkout and determine revision: unable to clone ' ssh://git@foo/abc/def': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

With version 0.27.0 everything is fine again, without any other change to the environment (same keys and so on):
k get gitrepositories -n flux-system

NAME         URL                                                              AGE   READY   STATUS
flux2-sync   ssh://git@foo/abc/def   64d   True    Fetched revision: main/4afa5937b48b12a42c0739c13870c282826b3979

If you need more information, please let me know. Unfortunately the source-controller log also doesnt give me more than the above error message
Steps to reproduce
We installed flux with the community helm charts without any relevant change to the values.yaml
Expected behavior
It is working again ^^
Screenshots and recordings
No response
OS / Distro
N/A
Flux version
N/A
Flux check
► checking prerequisites
✗ flux 0.24.1 <0.28.5 (new version is available, please upgrade) //cli is not used
✔ Kubernetes 1.21.5-eks-bc4871b >=1.19.0-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.18.2
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.21.1
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.17.1
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.22.2
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.23.1
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.22.3
✔ all checks passed
Git provider
GitHub Enterprise
Container Registry provider
No response
Additional context
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

@steveizzle thank you for reporting this. Do you mind sharing what gitImplementation are you using?

Can you also confirm whether this is consistent and every single reconcile yields the error mentioned, or is intermittent and happens every so often?
Would you mind running the latest version of source-controller and enabling the experimental transport to see whether the issue persists? Here are more information about it: https://github.com/fluxcd/source-controller/issues/636#issuecomment-1080789920

@steveizzle thank you for reporting this. Do you mind sharing what gitImplementation are you using?

I used go-git and tried now libgit2. With libgit2 it is working.

Can you also confirm whether this is consistent and every single reconcile yields the error mentioned, or is intermittent and happens every so often?
Would you mind running the latest version of source-controller and enabling the experimental transport to see whether the issue persists? Here are more information about it: #636 (comment)

Sure, i will give it a try and report back

Can you also confirm whether this is consistent and every single reconcile yields the error mentioned, or is intermittent and happens every so often?
Would you mind running the latest version of source-controller and enabling the experimental transport to see whether the issue persists? Here are more information about it: #636 (comment)

The error seems to be consistent.
As soon as i enable the experimental transport neither libgit2 nor go-git is working:
        env:
        - name: EXPERIMENTAL_GIT_TRANSPORT
          value: "true"
        - name: RUNTIME_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: ghcr.io/fluxcd/source-controller:v0.22.5

We found out that one repository is using an ED25519 Key which seems to be still working. Normally we use RSA keys.

Thanks for that. Do you mind sharing the errors you get when using libgit2 and the experimental transport?

We are getting the same error suddenly after redeploying a test cluster that we deploy often (almost every day) and doesn't give any errors with Flux.
Is there any resolution for this, or just keep trying until it works?

I currently pinned an old version, but will spent some time on getting more debug Infos the next days

@steveizzle @campanellaf this may have to do with some of the crypto level settings that may have changed upstream.
So to confirm, you are using RSA for your authentication keypair, right? What is the algorithm used for the known_host you are providing?
We have just released a RC with some improvements specially around the libgit2 implementation. Would you mind to also checking whether it fixes your problem?
ghcr.io/fluxcd/source-controller:rc-4b3e0f9a

The issue still exists on v0.30.2.
@pjbgf I tried your RC with the go-git implementation, but no luck.
The algorithm we're using for known_hosts is ecdsa-sha2-nistp256

@pjbgf Regarding the rc, Unfortunately the rc didnt fix the problem:
This is a example of a git repository in my test cluster:
k get deploy -n flux-system  -o wide

helm-controller               1/1     1            1           132d   manager      ghcr.io/fluxcd/helm-controller:v0.21.0                   app=helm-controller
image-automation-controller   1/1     1            1           132d   manager      ghcr.io/fluxcd/image-automation-controller:rc-48bcca59   app=image-automation-controller
image-reflector-controller    1/1     1            1           132d   manager      ghcr.io/fluxcd/image-reflector-controller:v0.18.0        app=image-reflector-controller
kustomize-controller          1/1     1            1           132d   manager      ghcr.io/fluxcd/kustomize-controller:v0.25.0              app=kustomize-controller
notification-controller       1/1     1            1           132d   manager      ghcr.io/fluxcd/notification-controller:v0.23.5           app=notification-controller
source-controller             1/1     1            1           132d   manager      ghcr.io/fluxcd/source-controller:rc-4b3e0f9a             app=source-controller

k get gitrepositories.source.toolkit.fluxcd.io -n flux-system flux2-sync -o yaml

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
  annotations:
    meta.helm.sh/release-name: flux2-sync
    meta.helm.sh/release-namespace: flux-system
    reconcile.fluxcd.io/requestedAt: "2022-06-02T17:34:40.013251+02:00"
  creationTimestamp: "2022-01-21T12:20:41Z"
  finalizers:
  - finalizers.fluxcd.io
  generation: 9
  labels:
    app.kubernetes.io/instance: flux-system
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: flux
    helm.sh/chart: flux2-sync-0.4.0
  name: flux2-sync
  namespace: flux-system
  resourceVersion: "556712592"
  uid: 8f76a6b4-cf42-49e1-af14-9608a1dc25c2
spec:
  gitImplementation: libgit2
  interval: 5m0s
  ref:
    branch: main
  secretRef:
    name: flux2-secret
  timeout: 20s
  url: ssh://git@SECRET_ENTERPRISE_URL
status:
  conditions:
  - lastTransitionTime: "2022-06-02T15:16:18Z"
    message: no artifact for resource in storage
    observedGeneration: 9
    reason: NoArtifact
    status: "True"
    type: Reconciling
  - lastTransitionTime: "2022-06-02T15:32:34Z"
    message: 'failed to checkout and determine revision: unable to fetch-connect to
      remote ''ssh://SECRET_ENTERPRISE_URL'': ssh:
      handshake failed: ssh: unable to authenticate, attempted methods [none publickey],
      no supported methods remain'
    observedGeneration: 9
    reason: GitOperationFailed
    status: "False"
    type: Ready
  - lastTransitionTime: "2022-06-02T15:32:34Z"
    message: 'failed to checkout and determine revision: unable to fetch-connect to
      remote ''ssh://SECRET_ENTERPRISE_URL'': ssh:
      handshake failed: ssh: unable to authenticate, attempted methods [none publickey],
      no supported methods remain'
    observedGeneration: 9
    reason: GitOperationFailed
    status: "True"
    type: FetchFailed
  lastHandledReconcileAt: "2022-06-02T17:34:40.013251+02:00"
  observedGeneration: 8

I also checked out the algorithms for the known_hosts and got the following:
Working:

gitrepo1: 2 entries in known_hosts: keyA with ssh-rsa and keyB with ecdsa-sha2-nistp256
gitrepo2: 1 entry in known_hosts: keyB with ecdsa-sha2-nistp256

Not Working:

gitrepo3:  1 entry in known_hosts: keyB with ecdsa-sha2-nistp256
gitrepo4:  2 entries in known_hosts: keyA with ssh-rsa and keyB with ecdsa-sha2-nistp256

So i would say its not related to the known_hosts.. And still everything is working when i switch the source controller back to v0.21.2

I did not set the Experimental env Variable in the test with the rc.  Is this still necessary with the rc?

@steveizzle we have released source-controller yesterday with managed transport enabled by default. The official image is here: https://github.com/fluxcd/source-controller/pkgs/container/source-controller/23808902?tag=v0.25.0
Please test it with go-git and libgit2, no env var needed.

Unfortunately we have got the same problem with v0.25.0 :(
k get deploy -n flux-system source-controller -o yaml | grep -i image:
        image: ghcr.io/fluxcd/source-controller:v0.25.0

k patch gitrepositories.source.toolkit.fluxcd.io -n flux-system flux2-sync --type='json' -p='[{"op": "replace", "path":"/spec/gitImplementation", "value":"libgit2"}]'
gitrepository.source.toolkit.fluxcd.io/flux2-sync patched (no change)

flux reconcile source git -n flux-system flux2-sync
► annotating GitRepository flux2-sync in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✗ GitRepository reconciliation failed: 'failed to checkout and determine revision: unable to fetch-connect to remote 'ssh://git@SECRET_ENTERPRISE_URL': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain'

k patch gitrepositories.source.toolkit.fluxcd.io -n flux-system flux2-sync --type='json' -p='[{"op": "replace", "path":"/spec/gitImplementation", "value":"go-git"}]'
gitrepository.source.toolkit.fluxcd.io/flux2-sync patched

flux reconcile source git -n flux-system flux2-sync
► annotating GitRepository flux2-sync in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✗ GitRepository reconciliation failed: 'failed to checkout and determine revision: unable to clone 'ssh://git@SECRET_ENTERPRISE_URL': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain'

I also let it ran with loglevel debug, but i think stacktrace doesnt help much:
{"level":"error","ts":"2022-06-09T12:45:48.671Z","logger":"controller.gitrepository","msg":"failed to checkout and determine revision: unable to clone<EMAIL_ADDRESS>ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"GitRepository","name":"flux-app-test","namespace":"testing-cmo-flux","error":"failed to checkout and determine revision: unable to clone<EMAIL_ADDRESS>ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain","stacktrace":"github.com/fluxcd/source-controller/internal/reconcile/summarize.ErrorActionHandler\n\tgithub.com/fluxcd/source-controller/internal/reconcile/summarize/processor.go:78\ngithub.com/fluxcd/source-controller/internal/reconcile/summarize.(*Helper).SummarizeAndPatch\n\tgithub.com/fluxcd/source-controller/internal/reconcile/summarize/summary.go:182\ngithub.com/fluxcd/source-controller/controllers.(*GitRepositoryReconciler).Reconcile.func1\n\tgithub.com/fluxcd/source-controller/controllers/gitrepository_controller.go:204\ngithub.com/fluxcd/source-controller/controllers.(*GitRepositoryReconciler).Reconcile\n\tgithub.com/fluxcd/source-controller/controllers/gitrepository_controller.go:233\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:227"}

im facing same problem, but is a timeoud instead.
here is my GitRepository CRD:
 ~  k get gitrepositories.source.toolkit.fluxcd.io flux-system flux-system -o yaml

apiVersion: v1
items:
- apiVersion: source.toolkit.fluxcd.io/v1beta2
  kind: GitRepository
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"source.toolkit.fluxcd.io/v1beta1","kind":"GitRepository","metadata":{"annotations":{},"labels":{"aws.cdk.eks/prune-c81a02c7111cf5a6f15fc6117762957d9e76218ab0":""},"name":"flux-system","namespace":"flux-system"},"spec":{"interval":"10m0s","ref":{"branch":"main"},"secretRef":{"name":"github-keypair"},"url":"ssh://git@SECRET_GITHUB_URL"}}
      reconcile.fluxcd.io/requestedAt: "2022-06-09T19:23:55.150431-03:00"
    creationTimestamp: "2022-06-09T20:53:42Z"
    finalizers:
    - finalizers.fluxcd.io
    generation: 1
    labels:
      aws.cdk.eks/prune-c81a02c7111cf5a6f15fc6117762957d9e76218ab0: ""
    name: flux-system
    namespace: flux-system
    resourceVersion: "57598"
    selfLink: /apis/source.toolkit.fluxcd.io/v1beta2/namespaces/flux-system/gitrepositories/flux-system
    uid: e1f352f5-93bc-4863-99d9-c43a6c0fc7c7
  spec:
    gitImplementation: go-git
    interval: 10m0s
    ref:
      branch: main
    secretRef:
      name: github-keypair
    timeout: 60s
    url: ssh://git@SECRET_GITHUB_URL
  status:
    conditions:
    - lastTransitionTime: "2022-06-09T20:53:42Z"
      message: no artifact for resource in storage
      observedGeneration: 1
      reason: NoArtifact
      status: "True"
      type: Reconciling
    - lastTransitionTime: "2022-06-09T23:18:49Z"
      message: 'failed to checkout and determine revision: unable to clone ''ssh://git@SECRET_GITHUB_URL'':
        dial tcp INTERNAL_IP: connect: connection timed out'
      observedGeneration: 1
      reason: GitOperationFailed
      status: "False"
      type: Ready
    - lastTransitionTime: "2022-06-09T23:18:49Z"
      message: 'failed to checkout and determine revision: unable to clone ''ssh://git@SECRET_GITHUB_URL'':
        dial tcp INTERNAL_IP: connect: connection timed out'
      observedGeneration: 1
      reason: GitOperationFailed
      status: "True"
      type: FetchFailed
    lastHandledReconcileAt: "2022-06-09T19:23:55.150431-03:00"
    observedGeneration: -1
- apiVersion: source.toolkit.fluxcd.io/v1beta2
  kind: GitRepository
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"source.toolkit.fluxcd.io/v1beta1","kind":"GitRepository","metadata":{"annotations":{},"labels":{"aws.cdk.eks/prune-c81a02c7111cf5a6f15fc6117762957d9e76218ab0":""},"name":"flux-system","namespace":"flux-system"},"spec":{"interval":"10m0s","ref":{"branch":"main"},"secretRef":{"name":"github-keypair"},"url":"ssh://git@SECRET_GITHUB_URL"}}
      reconcile.fluxcd.io/requestedAt: "2022-06-09T19:23:55.150431-03:00"
    creationTimestamp: "2022-06-09T20:53:42Z"
    finalizers:
    - finalizers.fluxcd.io
    generation: 1
    labels:
      aws.cdk.eks/prune-c81a02c7111cf5a6f15fc6117762957d9e76218ab0: ""
    name: flux-system
    namespace: flux-system
    resourceVersion: "57598"
    selfLink: /apis/source.toolkit.fluxcd.io/v1beta2/namespaces/flux-system/gitrepositories/flux-system
    uid: e1f352f5-93bc-4863-99d9-c43a6c0fc7c7
  spec:
    gitImplementation: go-git
    interval: 10m0s
    ref:
      branch: main
    secretRef:
      name: github-keypair
    timeout: 60s
    url: ssh://git@SECRET_GITHUB_URL
  status:
    conditions:
    - lastTransitionTime: "2022-06-09T20:53:42Z"
      message: no artifact for resource in storage
      observedGeneration: 1
      reason: NoArtifact
      status: "True"
      type: Reconciling
    - lastTransitionTime: "2022-06-09T23:18:49Z"
      message: 'failed to checkout and determine revision: unable to clone ''ssh://git@SECRET_GITHUB_URL'':
        dial tcp INTERNAL_IP: connect: connection timed out'
      observedGeneration: 1
      reason: GitOperationFailed
      status: "False"
      type: Ready
    - lastTransitionTime: "2022-06-09T23:18:49Z"
      message: 'failed to checkout and determine revision: unable to clone ''ssh://git@SECRET_GITHUB_URL'':
        dial tcp INTERNAL_IP: connect: connection timed out'
      observedGeneration: 1
      reason: GitOperationFailed
      status: "True"
      type: FetchFailed
    lastHandledReconcileAt: "2022-06-09T19:23:55.150431-03:00"
    observedGeneration: -1
kind: List
metadata:
  resourceVersion: ""

Im using version v0.25.5, with a ssh-ed25519 ssh key.
I try to do some connections tests inside the docker container of SourceController, but the container doesn't allow turn into sudo to perform apk operations and install ssh, in order to perform ssh -v git@HOST.
Even i standing in corporate network, im out from VPN and i can perform ssh -v git@HOST, even with my personal computer.

@steveizzle the repositories that are not working, are all of them on the same Git server?
Also, I noticed that the failing repos are using ssh-rsa for Auth-N, can you confirm that is enabled on the server?
We have released another release candidate that further improves on the connection management and observability.
Can you please give it a try using --log-level=trace?
ghcr.io/fluxcd/source-controller:rc-b877bc21

About the auth-N:
ssh -fNvv steveizzle@SECRET_ENTERPRISE_URL
.....
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp256,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
...

That should be a yes. And the rsa key is still working with the old version of the source controller..
Regarding the new rc:
k get deploy -n flux-system source-controller -o wide
source-controller   1/1     1            1           142d   manager      ghcr.io/fluxcd/source-controller:rc-b877bc21   app=source-controller

# ecdsa-sha2-nistp384:
k apply -f flux-secret-new.yaml
flux reconcile source git -n testing-cmo-flux flux-app-test

► annotating GitRepository flux-app-test in testing-cmo-flux namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ fetched revision master/bd3e6d2d8b24c97aa57dd8cb66db6fa9c113d022

# ssh-rsa 4096:
k delete -f flux-secret-new.yaml
k apply -f flux-secret-new-rsa.yaml

flux reconcile source git -n testing-cmo-flux flux-app-test
► annotating GitRepository flux-app-test in testing-cmo-flux namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✗ GitRepository reconciliation failed: 'failed to checkout and determine revision: unable to list remote for 'ssh://git@'SECRET_ENTERPRISE_URL: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain'

# Relevant Reconcilliation Loop errors:
k logs -n flux-system source-controller-67f4567c-7bz5z
{"level":"error","ts":"2022-06-13T08:36:38.301Z","logger":"controller.gitrepository","msg":"failed to checkout and determine revision: unable to list remote for<EMAIL_ADDRESS>ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"GitRepository","name":"flux-app-test","namespace":"testing-cmo-flux","cid":"83666aeb-96e2-4a90-aaf1-6b3abe5f5bb1","error":"failed to checkout and determine revision: unable to list remote for<EMAIL_ADDRESS>ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain","stacktrace":"github.com/fluxcd/source-controller/internal/reconcile/summarize.ErrorActionHandler\n\tgithub.com/fluxcd/source-controller/internal/reconcile/summarize/processor.go:78\ngithub.com/fluxcd/source-controller/internal/reconcile/summarize.(*Helper).SummarizeAndPatch\n\tgithub.com/fluxcd/source-controller/internal/reconcile/summarize/summary.go:182\ngithub.com/fluxcd/source-controller/controllers.(*GitRepositoryReconciler).Reconcile.func1\n\tgithub.com/fluxcd/source-controller/controllers/gitrepository_controller.go:211\ngithub.com/fluxcd/source-controller/controllers.(*GitRepositoryReconciler).Reconcile\n\tgithub.com/fluxcd/source-controller/controllers/gitrepository_controller.go:240\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2022-06-13T08:36:38.301Z","msg":"failed to checkout and determine revision: unable to list remote for<EMAIL_ADDRESS>ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain","name":"flux-app-test","namespace":"testing-cmo-flux","reconciler kind":"GitRepository","annotations":null,"error":"GitOperationFailed","stacktrace":"github.com/fluxcd/pkg/runtime/events.(*Recorder).Eventf\n\tgithub.com/fluxcd/pkg/runtime@v0.16.2/events/recorder.go:113\ngithub.com/fluxcd/source-controller/internal/reconcile/summarize.recordEvent\n\tgithub.com/fluxcd/source-controller/internal/reconcile/summarize/processor.go:123\ngithub.com/fluxcd/source-controller/internal/reconcile/summarize.ErrorActionHandler\n\tgithub.com/fluxcd/source-controller/internal/reconcile/summarize/processor.go:80\ngithub.com/fluxcd/source-controller/internal/reconcile/summarize.(*Helper).SummarizeAndPatch\n\tgithub.com/fluxcd/source-controller/internal/reconcile/summarize/summary.go:182\ngithub.com/fluxcd/source-controller/controllers.(*GitRepositoryReconciler).Reconcile.func1\n\tgithub.com/fluxcd/source-controller/controllers/gitrepository_controller.go:211\ngithub.com/fluxcd/source-controller/controllers.(*GitRepositoryReconciler).Reconcile\n\tgithub.com/fluxcd/source-controller/controllers/gitrepository_controller.go:240\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:227"}
{"level":"debug","ts":"2022-06-13T08:36:38.302Z","logger":"events","msg":"Warning","object":{"kind":"GitRepository","namespace":"testing-cmo-flux","name":"flux-app-test","uid":"7e4d321f-b7b9-4370-bd92-70f825d88f42","apiVersion":"source.toolkit.fluxcd.io/v1beta2","resourceVersion":"567995259"},"reason":"GitOperationFailed","message":"failed to checkout and determine revision: unable to list remote for<EMAIL_ADDRESS>ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain"}
{"level":"error","ts":"2022-06-13T08:36:38.306Z","logger":"controller.gitrepository","msg":"Reconciler error","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"GitRepository","name":"flux-app-test","namespace":"testing-cmo-flux","error":"failed to checkout and determine revision: unable to list remote for<EMAIL_ADDRESS>ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:227"}

i meet same issue. My flux env :
flux check
► checking prerequisites
✗ flux 0.24.1 <0.31.1 (new version is available, please upgrade)
✔ Kubernetes 1.21.9 >=1.19.0-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.12.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.15.5
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.17.1
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.16.0
✔ all checks passed

@Kevinwoolworth @steveizzle would it be possible for you to post the description of the secret that the GitRepository object refers to by running kubectl describe -n <your-namespace> secrets <your-secret>? Thanks

@aryan9600 Sure, we switched to working authentications, but i surely will help to assist on this issue:
Name:         flux-app-test
Namespace:    testing-cmo-flux
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
identity:      2602 bytes
identity.pub:  571 bytes
known_hosts:   581 bytes

Hi is there any update or potential timeline for this issue?

Having this issue on a private GitLab instance as well. I am running source-controller v0.25.11 and still having this issue:
@aryan9600 @stefanprodan
❯ k -n flux-system describe secret flux-system
Name:         flux-system
Namespace:    flux-system
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
identity:      306 bytes
identity.pub:  205 bytes
known_hosts:   171 bytes

❯ k get gitrepository -A
NAMESPACE     NAME          URL                                        AGE    READY   STATUS
flux-system   flux-system<EMAIL_ADDRESS>  139d   False   failed to checkout and determine revision: unable to list remote for<EMAIL_ADDRESS>ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
flux-system   gilogging     https://gitlab.example.com/gi/apps/gilogging.git   139d   True    stored artifact for revision 'main/adb4d0932cf6366e435b5b540c48414b143c40cf'

❯ flux check
► checking prerequisites
✗ flux 0.31.5 <0.32.0 (new version is available, please upgrade)
✔ Kubernetes 1.23.7-eks-4721010 >=1.20.6-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.22.2
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.26.3
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.24.1
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.25.11
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta1
✔ buckets.source.toolkit.fluxcd.io/v1beta1
✔ gitrepositories.source.toolkit.fluxcd.io/v1beta1
✔ helmcharts.source.toolkit.fluxcd.io/v1beta1
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta1
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta1
✔ receivers.notification.toolkit.fluxcd.io/v1beta1
✔ all checks passed
❯ flux get sources git
NAME       	REVISION                                     	SUSPENDED	READY	MESSAGE
flux-system	main/5160fe3801b485575424d53616dff76fcc74e478	False    	False	failed to checkout and determine revision: unable to list remote for<EMAIL_ADDRESS>ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
gilogging  	main/adb4d09                                 	False    	True 	stored artifact for revision 'main/adb4d0932cf6366e435b5b540c48414b143c40cf'
❯ flux logs --level=error
❯ kubectl describe gitrepository flux-system -n flux-system
Name:         flux-system
Namespace:    flux-system
Labels:       kustomize.toolkit.fluxcd.io/name=flux-system
              kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations:  reconcile.fluxcd.io/requestedAt: 2022-08-17T10:52:38.057356-04:00
API Version:  source.toolkit.fluxcd.io/v1beta2
Kind:         GitRepository
Metadata:
  Creation Timestamp:  2022-03-30T18:50:51Z
  Finalizers:
    finalizers.fluxcd.io
  Generation:        1
  Resource Version:  90866210
  UID:               6e766830-bebd-4f26-a1e2-07232736d405
Spec:
  Git Implementation:  go-git
  Interval:            1m0s
  Ref:
    Branch:  main
  Secret Ref:
    Name:   flux-system
  Timeout:  20s
  URL<EMAIL_ADDRESS>Status:
  Artifact:
    Checksum:          0bf60b79888c89ee8ba48d42a296f8fafb1a27fb8b5b8d319e253c1fc981299e
    Last Update Time:  2022-08-15T18:49:12Z
    Path:              gitrepository/flux-system/flux-system/5160fe3801b485575424d53616dff76fcc74e478.tar.gz
    Revision:          main/5160fe3801b485575424d53616dff76fcc74e478
    Size:              37517
    URL:               http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/5160fe3801b485575424d53616dff76fcc74e478.tar.gz
  Conditions:
    Last Transition Time:     2022-08-16T16:40:35Z
    Message:                  failed to checkout and determine revision: unable to list remote for<EMAIL_ADDRESS>ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
    Observed Generation:      1
    Reason:                   GitOperationFailed
    Status:                   False
    Type:                     Ready
    Last Transition Time:     2022-08-15T16:56:08Z
    Message:                  stored artifact for revision 'main/5160fe3801b485575424d53616dff76fcc74e478'
    Observed Generation:      1
    Reason:                   Succeeded
    Status:                   True
    Type:                     ArtifactInStorage
    Last Transition Time:     2022-08-16T16:40:35Z
    Message:                  failed to checkout and determine revision: unable to list remote for<EMAIL_ADDRESS>ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
    Observed Generation:      1
    Reason:                   GitOperationFailed
    Status:                   True
    Type:                     FetchFailed
  Content Config Checksum:    sha256:fcbcf165908dd18a9e49f7ff27810176db8e9f63b4352213741664245224f8aa
  Last Handled Reconcile At:  2022-08-17T10:52:38.057356-04:00
  Observed Generation:        1
  URL:                        http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/latest.tar.gz
Events:
  Type     Reason              Age                  From               Message
  ----     ------              ----                 ----               -------
  Warning  GitOperationFailed  12m (x106 over 23h)  source-controller  failed to checkout and determine revision: unable to list remote for<EMAIL_ADDRESS>ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

Having this issue on a private GitLab instance as well. I am running source-controller v0.25.11 and still having this issue

Can you share more details about your GitLab instance? I'm running GitLab 15.2.2-ee and it's working fine.

I did these steps, using AWS EKS and CodeCommit, from the scratch:
eksctl create cluster --name flux-demo --nodegroup-name ng-flux

git clone ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/flux-demo

cd flux-demo

mkdir -p ./clusters/flux-demo/flux-system

flux install \
  --components-extra=image-reflector-controller,image-automation-controller \
  --export > ./clusters/flux-demo/flux-system/gotk-components.yaml

git add -A && git commit -m "add components" && git push

kubectl apply -f ./clusters/flux-demo/flux-system/gotk-components.yaml

flux check

aws iam create-user --user-name fluxserviceaccount

aws iam attach-user-policy --policy-arn arn:aws:iam::aws:policy/AWSCodeCommitFullAccess --user-name fluxserviceaccount

PUBLIC_KEY=$(cat ~/.ssh/flux_rsa.pub)
aws iam upload-ssh-public-key --user-name fluxserviceaccount --ssh-public-key-body $PUBLIC_KEY

flux create source git flux-system \
  --git-implementation=libgit2 \
  --url=ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/flux-demo \
  --branch=master \
  --private-key-file=flux_rsa \
  --interval=1m

and I got this error:
✗ failed to checkout and determine revision: unable to clone 'ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/flux-demo': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

I am using:

flux version 0.32.0
source-controller:v0.26.1
eksctl 0.109.0-dev+78243b4c.2022-08-19T12:49:29Z
kubernetes 1.22

@anetogit are you sure that your ssh key-pair isn't protected by a password? if it is, then you could create a secret using flux create secret git (ref: https://fluxcd.io/docs/cmd/flux_create_secret_git/) and then use the flux create source --secret-ref instead of --private-key-file to authenticate.

@anetogit this issue is about GitHub not AWS… For AWS CodeCommit please read the docs: https://fluxcd.io/docs/use-cases/aws-codecommit/

@anetogit are you sure that your ssh key-pair isn't protected by a password? if it is, then you could create a secret using flux create secret git (ref: https://fluxcd.io/docs/cmd/flux_create_secret_git/) and then use the flux create source --secret-ref instead of --private-key-file to authenticate.

@aryan9600 No, my key is not protected with a password, there is no passphrase. I was using private key because of the CodeCommit of AWS.

@anetogit this issue is about GitHub not AWS… For AWS CodeCommit please read the docs: https://fluxcd.io/docs/use-cases/aws-codecommit/

@stefanprodan Yes. The tutorial works with the HTTPS:
flux create source git flux-system \
  --git-implementation=libgit2 \
  --url=https://git-codecommit.<region>.amazonaws.com/v1/repos/<my-repository> \
 ...

but not with SSH, because o the issue:
flux create source git flux-system \
  --git-implementation=libgit2 \
<EMAIL_ADDRESS>\
  ...

thanks.

I hope this is placed in the correct thread. I am facing a similar issue with Github Organizations where I can successfully connect to private repositories via ssh, but after roughly 5-10 minutes 'flux get source git' will show failure with error message:  failed to checkout and determine revision: unable to fetch-connect to remote 'https://github.com/{org}/{repo}': unhandled HTTP error 401 Authorization Required. I can run 'flux create source {flags}' right after and it will re-establish the connection without issue, then repeat failure.

This is still happening, on public Github, non-enterprise:
kubectl get deploy -n flux-system  -o wide
NAME                          READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                                               SELECTOR
helm-controller               1/1     1            1           48d   manager      ghcr.io/fluxcd/helm-controller:v0.32.1               app=helm-controller
image-automation-controller   1/1     1            1           48d   manager      ghcr.io/fluxcd/image-automation-controller:v0.32.0   app=image-automation-controller
image-reflector-controller    1/1     1            1           48d   manager      ghcr.io/fluxcd/image-reflector-controller:v0.27.0    app=image-reflector-controller
kustomize-controller          1/1     1            1           48d   manager      ghcr.io/fluxcd/kustomize-controller:v1.0.0-rc.1      app=kustomize-controller
notification-controller       1/1     1            1           48d   manager      ghcr.io/fluxcd/notification-controller:v1.0.0-rc.1   app=notification-controller
source-controller             1/1     1            1           48d   manager      ghcr.io/fluxcd/source-controller:v1.0.0-rc.1         app=source-controller

kubectl -n flux-system describe secret flux-system
Name:         flux-system
Namespace:    flux-system
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
identity:      306 bytes
identity.pub:  205 bytes
known_hosts:   171 bytes

flux get source git flux-system

NAME            REVISION                SUSPENDED       READY   MESSAGE

flux-system     main@sha1:098b74a4      False           False   failed to checkout and determine revision: unable to list remote for<EMAIL_ADDRESS>ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

This is still happening, on public Github, non-enterprise:
kubectl get deploy -n flux-system  -o wide
NAME                          READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                                               SELECTOR
helm-controller               1/1     1            1           48d   manager      ghcr.io/fluxcd/helm-controller:v0.32.1               app=helm-controller
image-automation-controller   1/1     1            1           48d   manager      ghcr.io/fluxcd/image-automation-controller:v0.32.0   app=image-automation-controller
image-reflector-controller    1/1     1            1           48d   manager      ghcr.io/fluxcd/image-reflector-controller:v0.27.0    app=image-reflector-controller
kustomize-controller          1/1     1            1           48d   manager      ghcr.io/fluxcd/kustomize-controller:v1.0.0-rc.1      app=kustomize-controller
notification-controller       1/1     1            1           48d   manager      ghcr.io/fluxcd/notification-controller:v1.0.0-rc.1   app=notification-controller
source-controller             1/1     1            1           48d   manager      ghcr.io/fluxcd/source-controller:v1.0.0-rc.1         app=source-controller

kubectl -n flux-system describe secret flux-system
Name:         flux-system
Namespace:    flux-system
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
identity:      306 bytes
identity.pub:  205 bytes
known_hosts:   171 bytes

flux get source git flux-system

NAME            REVISION                SUSPENDED       READY   MESSAGE

flux-system     main@sha1:098b74a4      False           False   failed to checkout and determine revision: unable to list remote for<EMAIL_ADDRESS>ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

@LockedThread given this issue is kind of stale, can you please share your source-controller version?

@LockedThread given this issue is kind of stale, can you please share your source-controller version?

The version does show in my comment but here is the output of flux version
helm-controller: v0.32.1
image-automation-controller: v0.32.0
image-reflector-controller: v0.27.0
kustomize-controller: v1.0.0-rc.1
notification-controller: v1.0.0-rc.1
source-controller: v1.0.0-rc.1

I also just decided to update to see if the issue fixed itself, still broken:
flux: v2.0.0-rc.5
helm-controller: v0.34.1
image-automation-controller: v0.34.1
image-reflector-controller: v0.28.0
kustomize-controller: v1.0.0-rc.4
notification-controller: v1.0.0-rc.4
source-controller: v1.0.0-rc.5

This is the bootstrap command I am using:
GITHUB_TOKEN=REDACTED flux bootstrap github --owner=REDACTED --repository=REDACTED --path=REDACTED --read-write-key --private --components-extra=image-reflector-controller,image-automation-controller

Here's more context from flux events:
4m43s (x25 over 6m22s)  Warning error                   ImageUpdateAutomation/flux-system       unable to clone<EMAIL_ADDRESS>ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
3m7s (x9 over 6m24s)    Warning GitOperationFailed      GitRepository/flux-system               failed to checkout and determine revision: unable to clone<EMAIL_ADDRESS>ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

Can you please try to rotate the credentials (and things like known_hosts)? The instructions for this can be found here: https://fluxcd.io/flux/installation/#deploy-key-rotation

Can you please try to rotate the credentials (and things like known_hosts)? The instructions for this can be found here: https://fluxcd.io/flux/installation/#deploy-key-rotation

I really appreciate the quick response. That worked. You're really saving me right now.

If that worked (and based on the error), I suspect the deploy key got deleted from the repository. As from the top of my head, a known_hosts entry mismatch would yield another error (and the last change made by GitHub there was at the end of March). Given this, you may want to figure out who and/or what's to blame for this :-).

Did your PAT expired 6 days ago? The deploy key expires when the GH token used for bootstrap expires. If you use PATs with an expiry date, you need to make sure to rotate the key before they expire.

Did your PAT expired 6 days ago? The deploy key expires when the GH token used for bootstrap expires. If you use PATs with an expiry date, you need to make sure to rotate the key before they expire.

None of the keys or PATs expired

I had the same issue was driving me crazy. Turned out that I specified the URL wrong for SSH:
-  url<EMAIL_ADDRESS>+  url: ssh://git@github.com:22/MY-ORG/MY-REPO


Transparent backgrounds

First of all, great crate, I'm using it in a Discord bot.
Discord has 2 themes, dark and light. I would want to use a module color that fits both dark and light, but in order to do that, I need transparent backgrounds.
In your opinion, how hard would that be to make? I looked into the SVG this crate worked with, but didn't see any transparency options at first glance.
If in your opinion it shouldn't be so hard, I might want to take a stab at it :)

Hello @jorian it is indeed possible i believe and I overlooked it, I will put some time to make the feature, you can also check how it is done internally, it is not so complex !

I believe d0300c944ba6e7d112fffae881c415c6bdb823ff fixes what you want, I'll push a version :)

fast_qr 0.7.1 is now available at crates.io with transparancy

Awesome, I was already fiddling with that 4th number as I assumed it would be opacity, but it just wasn't implemented yet.
Thanks a ton!

No problem 👍


ISM files that have duplicate audio stream tags create duplicate stream in output

We have several ISM files that look like this:
?xml version="1.0" encoding="UTF-8" standalone="no"?>
<smil xmlns="http://www.w3.org/2001/SMIL20/Language">
   <head>
      <meta content="mp4" name="formats"/>
   </head>
   <body>
      <switch>
         <video src="160917_1280x720_1000.mp4"/>
         <video src="160917_1280x720_2000.mp4"/>
         <video src="160917_1280x720_3000.mp4"/>
         <audio src="160917_1280x720_1000.mp4" systemBitrate="128000" title="AAC_eng_ch2_128kbps"/>
      </switch>
   </body>
</smil>

The generated mpd/m3u8 files will contain the three video streams correctly and it looks like the audio signal is processed correctly. But it will also contain a fourth video stream which is the same than the first stream.
<?xml version="1.0" encoding="UTF-8"?>
<!--Generated with https://github.com/google/shaka-packager version v2.6.1-634af65-release-->
<MPD xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:mpeg:dash:schema:mpd:2011 DASH-MPD.xsd" profiles="urn:mpeg:dash:profile:isoff-on-demand:2011" minBufferTime="PT2S" type="static" mediaPresentationDuration="PT6743.52001953125S">
  <Period id="0">
    <AdaptationSet id="0" contentType="video" width="1280" height="720" frameRate="25/1" subsegmentAlignment="true" par="16:9">
      <Representation id="0" bandwidth="2086456" codecs="avc1.4d401f" mimeType="video/mp4" sar="1:1">
        <BaseURL>160917_manifest_1.mp4</BaseURL>
        <SegmentBase indexRange="872-14391" timescale="25">
          <Initialization range="0-871"/>
        </SegmentBase>
      </Representation>
      <Representation id="1" bandwidth="3126380" codecs="avc1.4d401f" mimeType="video/mp4" sar="1:1">
        <BaseURL>160917_manifest_2.mp4</BaseURL>
        <SegmentBase indexRange="872-14391" timescale="25">
          <Initialization range="0-871"/>
        </SegmentBase>
      </Representation>
      <Representation id="2" bandwidth="1147506" codecs="avc1.4d401f" mimeType="video/mp4" sar="1:1">
        <BaseURL>160917_manifest_0.mp4</BaseURL>
        <SegmentBase indexRange="872-14391" timescale="25">
          <Initialization range="0-871"/>
        </SegmentBase>
      </Representation>
      <Representation id="3" bandwidth="1147506" codecs="avc1.4d401f" mimeType="video/mp4" sar="1:1">
        <BaseURL>160917_manifest_3.mp4</BaseURL>
        <SegmentBase indexRange="872-14391" timescale="25">
          <Initialization range="0-871"/>
        </SegmentBase>
      </Representation>
    </AdaptationSet>
  </Period>
</MPD>

I am not sure if the source files are "invalid" (they were created by a AMS premium encoder workflow several years ago) or if the migration tool needs some code to fix such issues. For example, if the audio src value is the same as an existing video src value, it can be filtered out?

@duggaraju Let me know if you need sample assets.

@schuettecarsten  yes that would be helpful

@schuettecarsten yes that would be helpful

@duggaraju no problem, but I cannot post a SAS link here, so I ask you to drop me an email at schuettecarsten at googlemail.com or search for c.schuette at neusta.de at Teams and I can send you details.

@schuettecarsten will follow up with you on gmail


Effect checker unpacking breaks if unpacked value is not used

In this example where unpacked value y is not used:
module test {
  pure val foo = tuples(Set(1, 2, 3), Set(4, 5, 6)).map((x, y) => x > 1)
}

We get the following error:
error: [QNT000] Can only unpack effects if they are all concrete or all variable
Trying to unpack effects: Pure and Read[v20] & Temporal[v21],e4
Trying to unify (Read[v22] & Temporal[v23], (Read[v22] & Temporal[v23]) => Read[v24] & Temporal[v25]) => Read[v22, v24] & Temporal[v23, v25] and (Pure, (Read[v20] & Temporal[v21], e4) => Read[v20] & Temporal[v21]) => e5
Trying to infer effect for operator application in map(tuples(Set(1, 2, 3), Set(4, 5, 6)), (x, y => igt(x, 1)))

I remember I had a good reason for enforcing that restriction, but have to revisit to remember what it was, and perhaps find a workaround at least when the value is not used.

Stale, we removed unpacking like this


Disable style.min.css loading on the plugin

Hello everyone,
We are currently in process of developing a Divi Extension.
By default, Divi Extension loads the stylesheet style.min.css and it is not enqueued by plugin itself. I also tried the wp_dequeue_style method but this style still got loaded.
The reason I am try to disable its loading so that we can only load CSS for enabled modules, which is already in place but style.min.css still which makes our method useless.
How can we disable the styles/style.min.css loading from the plugin?

@Robustur Could you please attach a screenshot?

I just posted this as a feature request in the slack channel. Actually Divi enqueues scripts directly in the Divi_Extension class. Instead it should register the scripts there and enqueue them in a public method we can overwrite in our extension subclasses to not enqueue the scripts/styles and rather enqueue them ourselves when needed, e. g. in render()

Hi @Robustur, my apologies for the late response.
I just tried it myself and the wp_dequeue_style works well on my end. For example, I have an extension named Sample Extension with slug sample-extension. To dequeue the styles/style.min.css, I use wp_dequeue_style as you can check on the code below.
function smpl_deregister_styles() {
	wp_dequeue_style( 'sample-extension-styles' );
}
add_action( 'wp_enqueue_scripts', 'smpl_deregister_styles', 11 );

Please note that since the styles/style.min.css is enqueued on priority 10, the wp_enqueue_scripts priority I use above should be higher like 11 to ensure the styles/style.min.css is already registered before we can remove it.
Could you please share the code on how did you run wp_dequeue_style?

@ayubadiputra Ah, sometimes the obvious doesn't come to mind. Thanks for the hint with wp_dequeue, I think that should work just fine.

@ayubadiputra Ah, sometimes the obvious doesn't come to mind. Thanks for the hint with wp_dequeue, I think that should work just fine.


Continue after server crash

As a hacker with a short fuse
I want the Juice Shop to save my challenge progress
so that I'm not frustrated when crashing the server.
Reasoning: Some attacks such as SQLI or weird JSON payloads can actually kill the nodejs server. The same goes for massive overload by automatic attack modes of pentesting tools.
Problem: Today the challenges are reset together with the entire DB on server launch. So all progress is gone.
Idea: Introduce either a "continue code" (like in Nintendo GameBoy classic times) or an automated backup of the DB after each challenge is solved. The code is somewhat retro-cool and corruption-safe, the DB copy might be corrupted but is more convenient. Would offer option to restore on Score Board. Continue code I could just store on disk on completion of a challenge and display with push notification and on score board. Might be some crazy hash of the DB ids of the challenges. Plus over DB backup: Would also work for containerized environment, where DB backups would die together with a crashed container.

Hello!
Ok here's one way I was able to crash the instance.  First I fired up a sqlmap command like:

root@kali:~/Desktop/js/sqlmap# sqlmap -u 'http://<IP_ADDRESS>:3000/rest/product/search?q=searchy' --level=3 --union-cols=8 -p 'q' --dbms='sqlite' --tables --cleanup --flush-session --fresh-queries --purge-output --cookie='token=%22eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdGF0dXMiOiJzdWNjZXNzIiwiZGF0YSI6eyJpZCI6MSwiZW1haWwiOiJhZG1pbkBqdWljZS1zaC5vcCIsInBhc3N3b3JkIjoiMDE5MjAyM2E3YmJkNzMyNTA1MTZmMDY5ZGYxOGI1MDAiLCJjcmVhdGVkQXQiOiIyMDE2LTA5LTI3IDAzOjEwOjQ0LjAwMCArMDA6MDAiLCJ1cGRhdGVkQXQiOiIyMDE2LTA5LTI3IDAzOjEwOjQ0LjAwMCArMDA6MDAifSwiaWF0IjoxNDc1MDg1NTUzLCJleHAiOjE0NzUxMDM1NTN9.5QfGJtYiaJAvVMwhh73zIIavyqsoIEdepLXguD1DELc%2;'

At this point everything continued as normal:

[*] starting at 15:51:07
[15:51:07] [INFO] purging content of directory '/root/.sqlmap/output'...
[15:51:07] [INFO] testing connection to the target URL
[15:51:07] [INFO] checking if the target is protected by some kind of WAF/IPS/IDS
[15:51:07] [INFO] testing if the target URL is stable
[15:51:08] [INFO] target URL is stable
[15:51:08] [WARNING] heuristic (basic) test shows that GET parameter 'q' might not be injectable
[15:51:08] [INFO] testing for SQL injection on GET parameter 'q'
[15:51:08] [INFO] testing 'AND boolean-based blind - WHERE or HAVING clause'
[15:51:08] [WARNING] reflective value(s) found and filtering out
[15:51:09] [INFO] testing 'AND boolean-based blind - WHERE or HAVING clause (Generic comment)'
[15:51:09] [INFO] testing 'Boolean-based blind - Parameter replace (DUAL)'
[15:51:09] [INFO] testing 'Boolean-based blind - Parameter replace (DUAL) (original value)'
[15:51:09] [INFO] testing 'SQLite inline queries'
[15:51:09] [INFO] testing 'Generic UNION query (NULL) - 8 to 8 columns (custom)'
injection not exploitable with NULL values. Do you want to try with a random integer value for option '--union-char'? [Y/n]
[15:51:18] [INFO] GET parameter 'q' is 'Generic UNION query (NULL) - 8 to 8 columns (custom)' injectable
[15:51:18] [INFO] checking if the injection point on GET parameter 'q' is a false positive
GET parameter 'q' is vulnerable. Do you want to keep testing the others (if any)? [y/N]
sqlmap identified the following injection point(s) with a total of 379 HTTP(s) requests:

What's important here is as soon as I hit Enter for no, I can watch the server go down (I had a docker stats going on via SSH)

Parameter: q (GET)
Type: UNION query
Title: Generic UNION query (NULL) - 8 columns (custom)
Payload: q=searchy')) UNION ALL SELECT 'qkbxq'||'oiTPbdquzeTYdWCgUZgiYuvpOVeMdxMfQzdPHLNf'||'qvzpq',NULL,NULL,NULL,NULL,NULL,NULL,NULL-- jVso
[15:51:21] [INFO] the back-end DBMS is SQLite
back-end DBMS: SQLite
[15:51:21] [INFO] fetching tables for database: 'SQLite_masterdb'
[15:51:21] [CRITICAL] connection dropped or unknown HTTP status code received. Try to force the HTTP User-Agent header with option '--user-agent' or switch '--random-agent'. sqlmap is going to retry the request(s)
[15:51:21] [WARNING] something went wrong with full UNION technique (could be because of limitation on retrieved number of entries). Falling back to partial UNION technique
[15:51:21] [CRITICAL] unable to connect to the target URL ('Connection refused'). sqlmap is going to retry the request(s)
[15:51:21] [CRITICAL] unable to connect to the target URL ('Connection refused')
[15:51:21] [WARNING] HTTP error codes detected during run:
500 (Internal Server Error) - 284 times
[*] shutting down at 15:51:21
A continue code or save/load scoreboard option would be flippin' sweet :-)
Thanks for all your work on this!
Brian

Any convenient save/load option would only work in a simple way for the local node.js run-option. For Docker, Vagrant, Heroku or EC2 the backup would just die with a butchered container. So, I guess displaying a "continue code" on screen for people to write down (if they want to play it safe) seems like the best way to make it work on every run-option.
Now, any ideas on a two-way operation that would

create the shortest & simplest possible string
for representing a unique subset of the numbers 1-50?
(Because, I wouldn't want people to have to write down Base64 strings...)

@braimee Could you please check if your crash from sqlmap looks similar to #155? If so, please append your logs there and I'll reopen that problem... but, I don't see much hope to make a node.js app such as Juice Shop resistent against massive scripted attacks easily.
Maybe https://github.com/rook2pawn/node-ddos would be worth a try?

Possible two-way function to use: https://github.com/ivanakimov/hashids.js (exactly made for arrays of numbers)


UriResolver::generate does not properly reconstruct URLs with a query fragment.

The UriResolver appends the fragment to the path without a leading ?  causing an incorrect URL.
Ex: http://example.org/?rest=/route becomes http://example.org/rest=/route.
https://github.com/justinrainbow/json-schema/blob/master/src/JsonSchema/Uri/UriResolver.php#L65
Happy to submit a patch.

Thanks for the bug report! This is fixed in #425. I will backport the fix for 5.2.2 once it's merged.

Fantastic, thank you!

Closed with #425


upgrade sqlx to 0.6

Any chance this will get done soon? This is a big blocker on upgrading.

any updates on this? I'm afraid we'll need to move to another orm as looks like this one is not maintained anymore :(

Ill publish a new version in the coming days!

Maintaining an OS project is a lot of work, and I've got plenty of things on my plate right now. However, I don't want to just abandon this, so I'll do my best to keep it in a usable state!

hi @NyxCode , sorry to bug you. https://crates.io/crates/ormx/0.10.0/dependencies is still showing sqlx being ^ 0.5. any chance of updating?

This still isn't fixed; the issue should at least be re-opened until a new crate version is published.
@NyxCode , it doesn't look like a new version has been released. Is there anything special that needs to be done to release a new version? I'd certainly be happy to help.
The issue I'm having is I need the sqlx::test macro, which was introduced in 0.6, and I'm also depending on ormx.
Let me know if you need any help getting the new version up.

For those who are having issues with compatibility with sqlx 0.6, the good news is the updated code can be found in this repository. If you specify ormx as a git dependency with this repo's URL, you should be fine.


Add "Auto-Identify" checkbox to "New Recording" page that, if checked, submits recording to a web service that automatically identifies the name and composer

This means that, if this is checked, the user will not have to fill in these details manually by themselves.

Another potential service is https://www.musipedia.org/.


Gkorland 107

Codecov Report

Merging #36 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #36   +/-   ##
=======================================
  Coverage   94.91%   94.91%
=======================================
  Files           6        6
  Lines         118      118
  Branches       17       17
=======================================
  Hits          112      112
  Misses          1        1
  Partials        5        5

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5b7d416...42f0d15. Read the comment docs.


cli throwing error "TypeError: undefined is not a function"

I installed javascript-obfuscator as:
$ npm install --save-dev javascript-obfuscator
I cd into node_modules/javascript-obfuscator/
I ran the following command:
$ ./bin/javascript-obfuscator webpack.config.js --output webpack.config.test
I throws an error :: TypeError: undefined is not a function

If you using obfuscator as CLI - install it globally with -g flag.
Also show me your node version, obfuscator version and full error log (line above Type error)

1/How do I install gloablly?
2/Please see the SS

3/I am using this tool outside of node.(frankly I am not using node, I only used it to fetch the package, that's it)
Still here is what you asked::
$node --version
v0.12.7
I  did javascript-obfuscator --version I got the same error, as attached in screen shot.
Now I did this to fetch the version::
$ cat package.json (inside ~.../node_modules/javascript-obfuscator)
{
...
"version": "0.14.3",
...
}

You using very old version of the node.js that not supported by obfuscator.
Install latest LTS version of the node.js.
You can install obfuscator globally with following command:
npm i -g javascript-obfuscator


handle cross origin * with webview

It is not good to provide access origin * to access backend APIs from any short of source.
Is there any good way to bypass and restrict to hits from my app only.
I dont want to use native http it is hard to debug in web app and takes more time of dev to finish the project
Thank you

If you have control over API server then you can implement this by having a list of allowed origins. The Access-Control-Allow-Origin header only allows one value. So on your server you need to take the origin of request, check if it is allowed and then set the header to that origin.
For instance if request came from origin http://localhost (webview by default), you return header Access-Control-Allow-Origin: http://localhost. That's how i will be implementing it on the API server.

Hi @ljudbane , there is iOS scheme too that also need to whitelist ionic://localhost, I don't know custom scheme is allowed to whitelist, have you tried with iOS?

@indraraj26 I just started testing on iOS and had to add ionic://localhost to the list of allowed origins on API server. So my allowed list now looks like this:
'http://localhost',     // webview on Android device
'ionic://localhost',    // webview on iOS device
'http://localhost:8100' // ionic serve


Titles get capped on import via WoS

Bug
Titles get capped on import
Context
Import with WoS file:
https://www.webofscience.com/wos/woscc/full-record/WOS:000332386100024
Title in WoS txt file:

See import:

See full title on copernicus:
https://acp.copernicus.org/articles/14/2155/2014/

@netsensei Ha, I found out why – it is not importing it via DOI, but importing via WoS.
When I download the record via here https://www.webofscience.com/wos/woscc/full-record/WOS:000332386100024
I can see that in the txt file the title is on multiple lines. This causes the field to break I think.

Tested with multiple types of downloads, it all behaves the same.

I'll check which ones we need researchers to use


Test and migrate Scale example to 1.9.

Can't self assign this issue, but I'm on it.

This change to update Scale was merged, thanks Jonathan!


Allow backups to restore into cluster defined storage

In order for the api/ui to generate a secure download link, it needs to know the s3-compatible location and access/secret keys. Currently the api only knows about a single, global, endpoint which means all restores for all clusters have to get sent to a central storage.
In order to allow per-cluster restore storage:

[ ] Update the api so that the Kubernetes type has it's own restore s3 configs
[ ] Write a migration to default all Kubernetes to the current global s3 configs
[ ] Update the api restoreLocation to use Kubernetes specific configs
[ ] Ensure k8up can be configured at the cluster level to send restores to it's own storage

Probably related to #2985

Having to put the restore credentials into the API seems like a bad idea, but could be a good stop gap until we look to move to NATS for messaging.
At least having them configurable at a cluster level would reduce the need to add custom location variables in some cases
When we look to move to using NATS for messages though, we would be able to have the API reach out to the remote to generate the download URL using the credentials in the remote and this request is essentially an API call (unlike how rabbitmq would do it), this way the credentials wouldn't need to be in the API at all.


[Network] az network dns export: export FQDN for MX, PTR, NS and SRV type instead of relative path

Description

Fix #14412
Testing Guide

History Notes

[Component Name 1] BREAKING CHANGE: az command a: Make some customer-facing breaking change.
[Component Name 2] az command b: Add some customer-facing feature.

This checklist is used to make sure that common guidelines for a pull request are followed.

[ ] The PR title and description has followed the guideline in Submitting Pull Requests.

[ ] I adhere to the Command Guidelines.

Network


Release version 0.11.2

@fnothaft I'll do this today unless I hear otherwise

@heuermh Please hold until I update https://github.com/bigdatagenomics/adam/pull/1684 to point at 0.11.2-SNAPSHOT and CI passes.

OK, tests passed on bigdatagenomics/adam#1684, feel free to proceed. Thanks @heuermh!

Fixed by commit 44db97841f5b478a0e9ee98e68f66dd8cbe9a486


Tracking: Implement sock_accept and basic networking

With the addition of sock_accept() in wasi-0.11.0, wasmtime can now implement basic networking for pre-opened sockets.
Todo

[x] https://github.com/bytecodealliance/wasmtime/pull/3711
[x] https://github.com/bytecodealliance/wasmtime/pull/3729
[x] Rust: https://github.com/rust-lang/rust/pull/93158
[ ] Wasi definition of Preopentype for sockets (listener and stream, tcp and unix) to make fd_prestat_get() work for sockets
[ ] Non-busy polling on Windows
[ ] Fix get_fdflags() for sockets on Windows
[ ] More networking test cases
[x] Discussion of SocketAddr return value for ::accept() in the rustsrc PR

        Ok((
            TcpStream::from_inner(unsafe { Socket::from_raw_fd(fd as _) }),
            // WASI has no concept of SocketAddr yet
            // return an unspecified IPv4Addr
            SocketAddr::new(Ipv4Addr::UNSPECIFIED.into(), 0),
        ))

How is this going on? I try to use just  std::net::TcpListener to handle TcpStream and it raises errors that wasi is not supported yet.

Is there a way to use sock_accept (and reading / writing) without having a busy wait loop?
With wasmtime I couldn't see that poll_oneoff could be used to wait on the preopened fd to wait for a new connection.

How is this going on? I try to use just std::net::TcpListener to handle TcpStream and it raises errors that wasi is not supported yet.

Any success in this direction?

How is this going on? I try to use just std::net::TcpListener to handle TcpStream and it raises errors that wasi is not supported yet.
passing --tcplisten=<IP_ADDRESS>:10000  to wasmtime and then

let mut server = {
let s = unsafe { std::net::TcpStream::from_raw_fd(3) };
  s.set_nonblocking(true).context("failed to set non-blocking flag on socket")?;
  TcpListener::from_std(stdlistener)
}

makes the passed socket available to use in server app.
btw I dont know how to perform connect when writing a client app.
passing --tcplisten=<IP_ADDRESS>:10000 to client wasm app in case server is running is not possible. (socket already in use)
but enarx example seems to achieve this functionality https://github.com/enarx/codex/blob/main/demos/chat-client/rust/src/main.rs#L32
I still haven't figured out whether this functionality implemented in wasi .. If anyone can provide any info about how pass a socket to wasm client via wasmtime to perform a connection.. would be great

I'm actually going to go ahead and close this as more-or-less not planned at this time. The wasip1 proposal's implementation of sockets was never fully completed, however the wasip2 implementation does have a full suite of socket-based APIs for use. If it works I'd recommend using the wasip2-based APIs.

I've actually seen the mio Wasm tcp listener just work with the latest wasmtime, once I used the -S preview2=n -S tcplisten=... options to run. I had to modify some lines within the mio src too since it is now treating warnings as errors and the wasi build target of mio was causing all sorts of warnings (well at least two, my memory of what all the failures were two days ago is fuzzy).
So the mio Wasm example is able to listen to a socket opened up by wasmtime and interact with nc processes I opened from other terminals. It's cool and I actually didn't expect it to work, so I may dig a little deeper to understand how it works with a Wasm module (a preview1 image).

I commented in another issue as well but if it aligns with your goals one thing that might be good is to implement mio's support for WASI in terms of the wasip2 target with wasi-sockets. Otherwise though I'm glad you were able to find the -Spreview2=n flag for local testing for now (sorry again for the bad name...)


Fix for nmcli syntax issue in getCurrentConnections() (Issue #27)

See https://developer.gnome.org/NetworkManager/stable/nmcli.html for nmcli syntax reference.
https://github.com/friedrith/node-wifi/issues/27

Thx for the PR. It is quite weird I didn't notice this bug before. In order to avoid conflict because I was improving the code in this file and also becaues your PR was not to the good branch. I didn't use your PR. However, I made your changes so everything should be ok now.


over[left/right]arrow commands not recognized

Describe the bug
\overleftarrow{a} and \overrightarrow{a} are not recognized as commands.
To Reproduce

Steps to reproduce the behavior:

Call LaTeXParser.MathListFromLaTeX("\\overrightarrow{f}(x,y) = xy^2")
Observe that the result has an Error with the following text:

Error: Invalid command \overrightarrow\n\overrightarrow{f}(x,y) = xy^2\n↑ (pos 1)

Expected behavior
The parsing to be successful. If displayed in an SKCanvas, it should look something like this:
Environment (please complete the following information):

Platform: CSharpMath.SkiaSharp
Package Version or Commit: 0.5.0-beta
Device: Windows 10 laptop (but should be applicable to all devices)

This is just not implemented yet.


VSTS Linux Agent is missing .NET Core Runtime 2.0.6

We updated our ASP.NET Core App to use the .NET Core Runtime 2.0.6. Currently the VSTS Linux Agent fails when running tests with the following error message:
2018-03-15T12:18:27.3974218Z Starting test execution, please wait...
2018-03-15T12:19:27.6545141Z Testhost process exited with error: It was not possible to find any compatible framework version
2018-03-15T12:19:27.6561495Z The specified framework 'Microsoft.NETCore.App', version '2.0.6' was not found.
2018-03-15T12:19:27.6575815Z   - Check application dependencies and target a framework version installed at:
2018-03-15T12:19:27.6589266Z       /
2018-03-15T12:19:27.6602676Z   - Alternatively, install the framework version '2.0.6'.

Please update the agent so it is compatible with the latest .NET Core SDK/Runtime.

The updated images are being pushed to Docker Hub; should be complete within the hour.

All the images have been pushed.


Build system usage statistics

Designers of a build system often need to optimise it for the most common use-case, to make life easier for as many people as possible. There is currently no easy way to collect usage statistics in Shake.
What kind of statistics am I talking about? Average build time. Total build time over a particular period. Frequency of using particular command line flags. Targets that get rebuilt most often in real development. How frequently the build system fails. And others.
Can we come up with a non-intrusive, fast, easy-to-implement solution? What comes to mind: a file that is persistently updated after each build: e.g., the command line + the runtime + a short build digest.
If there is such a file then designers of a build system could sometimes ask their users to email them the file for analysis. (Automating this further is probably a bad idea as people may not want their build system to send out any data, even with their explicit consent.)
What do you think?

So, I'm skeptical this will be of any real use... It would be fantastic information to have for all users of Hadrian to get these stats, and given the complete aggregated/anonymous data you could really figure out very useful information and improve things. But, the only way you get lots of data is to automatically upload it to a server, and as you say, I don't think anyone will want that. If you are relying on explicit "people sending a file" you're not going to get much data, and the data you get will be atypical and usually associated with a precise bug, in which case profiling output is probably more useful to track it down.
However, if you still want to do that, you can do about 90% of it without any involvement from Shake. Just add try $ duration $ shakeWithArgs ... or similar, and then you'll be capturing success/failure and duration. Add in getArgs and you have command line info. You can then appendFile it to some log.
If I'm wrong and it does turn out to be useful in ways I haven't foreseen adding a shakeLogFile option and storing a one-line summary of each build would be a reasonable next step.

It would be fantastic information to have for all users of Hadrian to get these stats, and given the complete aggregated/anonymous data you could really figure out very useful information and improve things. But, the only way you get lots of data is to automatically upload it to a server, and as you say, I don't think anyone will want that.

Yep, and that's sad :-(

However, if you still want to do that, you can do about 90% of it without any involvement from Shake. Just add try $ duration $ shakeWithArgs ... or similar, and then you'll be capturing success/failure and duration. Add in getArgs and you have command line info. You can then appendFile it to some log.

I was thinking about something like this too. Anyway, I wanted to write this down here; perhaps you can come up with something better at some point.
I guess automated sending of aggregated/anonymous data to a server may work well within a company, or in a distributed/cloud build system where some data is being sent anyway.

Yep, I am now very tempted to do this collection internally. Would be great data!

Closing on the basis it can be done outside the build system just as effectively, and probably more aggressively in internal company settings.

BTW, if anyone ever does do this, I'd be keen to know what they did exactly, and how valuable the data was.


fix a python 3.x test problem

the order of the result in Payload.dict() changed after python upgrade，which caused a failure in test in python 3.6. And now it has been fixed.

please test it again. I am new to Github, and do not know how to pull request, so maybe I have made some mistake. for example, indent problem. Thanks.


JSON file in expansions ,tweet-fields , user-field ,

I think there is a problem with the JSON file when try to merge multiple files together  such as :

attachments.media_keys
2)entities.mentions.username
3)eferenced_tweets.id
4)referenced_tweets.id.author_id

Please see this
https://stackoverflow.com/questions/64456339/how-to-flat-large-data-of-tweets-from-multiped-json-file
https://stackoverflow.com/questions/64431731/merge-multiple-json-files-into-one-file-by-using-python-stream-twitter

Hi there,
I would like to better understand the issue you are having. Perhaps we can make some updates to how the client writes out search responses to make it even more flexible.
Currently, when writing responses... If the response has the following form:
{"data": [
{tweet-object},{tweet-object},{tweet-object}
],
"includes":{
"media": [
{media-object},{media-object},{media-object}
],
"users": [
{user-object},{user-object}
],
"tweets": [
{tweet-object}
]
},
"meta": {meta-object}
}
The client will write out the following content, as "atomic" JSON entities (each line below represents a whole, valid JSON object):
{tweet-object}
{tweet-object}
{tweet-object}
{"media":[ {media-object},{media-object},{media-object}]}
{"users":[ {user-object},{user-object}]}
{"tweets":[ {tweet-object}}
{meta-object}
When taken as together as a whole set, then these objects are not 'packaged' as one valid JSON object.
As an alternative, there could be an option to write out the entire response as the single JSON object that is received from the endpoint...?

Seems to be a duplicate of a related issue. Closing.

Seems to be a duplicate of a related issue. Closing.


Issue 26 / FUGR determination of include

Finally, the correct Pull Request created. time to go to bed.

@ed-holland did this work out? the code looks okay, however I've not tried it
in all cases its better than before

It works nicely for the includes of a fuba (type 3). For type 2, the include is also determined and passed, but it is not used. Nothing happens in this case. Debugging shows that it is cleared but i have no idea how this happened.

if you set the PR to "Ready for review" then we can merge it

Now it works as it should be...

thanks 👍


Use web3 transactions timeout properly

The ZWeb3 wrapper implements a custom flow to deal with transactions timeout, we should use Web3's native one instead (see transactionBlockTimeout).
Additionally, note that we are using users' session timeout to set that value. Those are different concepts, so please evaluate if it's necessary to introduce a new way to specify different values.

Closing in favour of #1065


Update cromwell instance to  n1-standard-2 and update cromwell version to 38

Adding fixes to address gsutil bugs and memory bugs in wdl_runner

This will also require an updated Docker image from the Broad

hey @kdbinder just to confirm, do you want a new docker image to update broad-dsde-outreach/wdl_runner:2018_11_28 (rather than your placeholder gcr.io/google.com/kbinder-sandbox/wdl-runner-rev?)

@cjllanwarne thats correct. We need the docker container in the broad-dsde-outreach repo to be rebuilt with the new Dockerfile.  @ruchim did this last time.

@kdbinder gcr.io/broad-dsde-outreach/wdl_runner:2019-04-15, let me know if that works!

Thanks @ruchim that works.  Updated

@aednichols are you able to review this to merge?


Issue using a Custom Cast

Larastan Version: 1.0.2
--level used: 5

Description
Maybe this issue would be better off in a discussion, but those are not active here.
To the topic:
I am using https://github.com/mewebstudio/Purifier to make sure that certain user inputs, which are stored in the database, do not contain any unwanted tags (e.g. <script>).
To do so, I added the provided CleanHtmlOutput-cast to my model's $casts-array:
/**
 * @var string[]
 */
protected $casts = [
    'uuid'          => 'string',
    'lastname'      => CleanHtmlOutput::class,
    'firstname'     => CleanHtmlOutput::class,
    'created_at'    => 'datetime',
    'updated_at'    => 'datetime',
];

This works fine, but Larastan throws a lot of warnings, as I am using this CleanHtmlOutput::class custom cast quite a lot.
Larastan reports:

Part $this->name (Mews\Purifier\Casts\CleanHtmlOutput) of encapsed string cannot be cast to string.

Laravel code where the issue was found
Larastan refers to a line like this:
/**
 * @return string
 */
public function getFullNameAttribute(): string
{
    return "{$this->firstname} {$this->lastname}";
}

这是来自QQ邮箱的假期自动回复邮件。

您好，我最近正在休假中，无法亲自回复您的邮件。我将在假期结束后，尽快给您回复。

Hi,
There is already a feature request for that: https://github.com/nunomaduro/larastan/issues/890

Did not see that, I am very sorry.


Images are not displayed

Ok, I found the images by just coping/pasting the links - from getData() method of the code - to a browser. But the project created in VS2017 shows the public\image sub-folder with no such sub-folder created in File explorer. And after I created this sub-folder and copy the images there and ran the app again, the images still do not show-up in the app.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 7aa65a08-a9e2-721d-e00c-1f28afffe2be
Version Independent ID: babf20c3-c235-d839-7340-fb9c11c449dc
Content: Create a Node.js and Express app - Visual Studio
Content Source: docs/javascript/tutorial-nodejs.md
Product: visual-studio-dev15
GitHub Login: @Mikejo5000
Microsoft Alias: mikejo

@saf-itpro Hi, it sounds like you are trying to do something different than the steps in the tutorial? Like copying images locally and then showing them in the app? The tutorial is only showing you how to load the images as a web resource. The images should show up when you click the buttons, if you use the code described in the article.

Possibly you typed getElementByID (cap "ID") when it should be getElementById (lower "d" in Id) - see the index.pug file in this tutorial. That is what I did. Hope that helps.

Hi,
I checked getElementById and it is as it should be in the code.  The problem still persists.
Any Ideas??
Thanks.


Prevent bot from logging individual conversations

We want a parameter where each group has a unique number. This is to prevent random people from starting conversations with it.

I'm not sure I follow on the issue here. Our logger is specifically only logging messages on a public TG group. It does not log individual/private conversations.
Please clarify what you are referring to.

Fixed by having CHAT_IDS var setting chats that bot pays attention to.
https://github.com/OriginProtocol/telegram-moderator/commit/b9b1663db3104ee006d11f1dd289c344cc303ba5


Allow to set parent Channel when constructing EmbeddedChannel

Motivation:
Sometimes it is beneficial to be able to set a parent Channel in EmbeddedChannel if the handler that should be tested depend on the parent.
Modifications:

Add another constructor which allows to specify a parent
Add unit tests

Result:
Fixes https://github.com/netty/netty/issues/9228.

@ryanoneill PTAL


Designate channels as limited messages

To be fleshed out further. General idea is, we use slow mode to subtly indicate to users not to discuss in some channels. This doesn't work well.
Ideal flow:
If not a mod, then delete their message and respond inline or direct privately with the contents of the message and explain why it was deleted.
Are there existing bots that can achieve as close as possible to this and are simple?
Consider different variations of limited messages.

What about making the worst offender (bugs) a channel that has threads, like-experiences - so that folks will need to post a thread if they found a bug.
I often see "bugs" (which are not bugs at all) that I know an answer to and then, because of how the channel is designed, I click "Reply" before I think to make a thread for that person.


Corrected repo link and added README

Hello,
I like the theme I kinda wanted to help out.
In these two commits you will find
A change in the link on the package.json so it reflects on the vscode marketplace and shows the correct repo
Added link to version svg and connected to theme marketplace link. Also added more information on the installation

Thanks!


Helm install via insecureRegistry=true

In my environment, I have an nginx ingress controller that exposes two node ports one forwarding to 80 and another one to 443.
Goal: To be able to reach harbor using the kubernetes ingress IP  via nodeport for 443, let ingress handle SSL termination. Internally, harbor should run on insecure registry and the docker client should only need the certificate for the kubernetes ingress and not the one generated by harbor.
Installing harbor using the helm chart with insecureRegistry=true, and externalPort=<nodePort for 80> I can reach the UI just fine. However, docker login fails with the following
Error response from daemon: Get http://harbor.my.domain:30314/v2/: Get https://harbor.my.domain:30314/service/token?account=admin&client_id=docker&offline_token=true&service=harbor-registry: http: server gave HTTP response to HTTPS client

I have added insecure-registries harbor.my.domain:30314 to my docker daemon.
Note: installing harbor with insecureRegistry=false with externalPort=<nodePort for 443> works fine as I can reach the UI, and docker login works with FQDN:<nodePort for 443>. However, this goes against my design goal where I want the docker client to only need the certificate for kubernetes ingress.

Is my design goal possible with harbor ? If so, what changes do I need to make to values.yaml or helm charts ?
If insecureRegistry is not supported, is there a way to switch off SSL termination in harbor so kubernetes ingress controller handles this ? In other words, docker client only needs certificates to my nginx ingress.

Hi,
The helm chart by default leverages the ingress for SSL termination.

@reasonerjt Thanks for the reply.
Are you suggesting to install harbor with generateCertificates=false and harbor FQDN to the certificate I'm using for the nginx ingress controller ?

Ye, I think that's a better way to solve your problem.

Closing, if there's further issue please re-open or open new ones.


Fixed Cloud VCR sync Helix cluster failure.

The VCR replication failure is due to Helix update didn't work.
We picked the wrong source Helix Zk connection string. For Azure VCR, we couldn't use local DC Zk String to find the Helix resource. It didn't exist.

Patched on Azure VCR node and tested the fix. It replication works now.
./gradlew build locally failed with 24 unit test case. But the existing code has the same issue. Shouldn't be related to the code change. PR test run passed.

Codecov Report

Merging #2136 (c9f0dcc) into master (e27ffc9) will decrease coverage by 3.92%.
The diff coverage is 9.37%.

@@             Coverage Diff              @@
##             master    #2136      +/-   ##
============================================
- Coverage     73.75%   69.82%   -3.93%
+ Complexity    10161     9668     -493
============================================
  Files           722      722
  Lines         55244    55271      +27
  Branches       6835     6839       +4
============================================
- Hits          40744    38594    -2150
- Misses        12326    14525    +2199
+ Partials       2174     2152      -22

Impacted Files
Coverage Δ

...m/github/ambry/clustermap/HelixClusterManager.java
84.02% <0.00%> (-1.74%)
:arrow_down:

.../com/github/ambry/cloud/VcrReplicationManager.java
35.13% <11.11%> (-32.87%)
:arrow_down:

...java/com/github/ambry/notification/UpdateType.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...thub/ambry/notification/BlobReplicaSourceType.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../cloud/LeaderStandbyHelixVcrStateModelFactory.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...om/github/ambry/named/MySqlNamedBlobDbFactory.java
0.00% <0.00%> (-94.74%)
:arrow_down:

...ry/accountstats/AccountStatsMySqlStoreFactory.java
0.00% <0.00%> (-93.34%)
:arrow_down:

...ava/com/github/ambry/network/http2/Http2Utils.java
0.00% <0.00%> (-90.00%)
:arrow_down:

.../github/ambry/account/mysql/MySqlAccountStore.java
0.00% <0.00%> (-87.50%)
:arrow_down:

...y/network/http2/Http2ClientStreamStatsHandler.java
0.00% <0.00%> (-86.37%)
:arrow_down:

... and 109 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e27ffc9...c9f0dcc. Read the comment docs.

Discard the PR.


Add support to ignore the HostedCluster and Nodepool resource.

Signed-off-by: Joshua Packer<EMAIL_ADDRESS>
What this PR does / why we need it:

We are using an ObjectRef in ACM to point to HostedCluster and NodePool CR's for certain deployment scenarios. To make sure the CR's are not reconciled by the controller, we are looking to put an early escape in the reconcile loop.

Which issue(s) this PR fixes (optional, use fixes #<issue_number>(, fixes #<issue_number>, ...) format, where issue_number might be a GitHub issue, or a Jira story:
Related to https://github.com/stolostron/hypershift-deployment-controller/pull/86
Checklist

[x] Subject and description added to both, commit and PR.
[x] Relevant issues have been referenced.
[ ] This change includes docs.
[x] This change includes unit tests.

/hold
Thanks @jnpacker

We are using an ObjectRef in ACM to point to HostedCluster and NodePool CR's for certain deployment scenarios. To make sure the CR's are not reconciled by the controller, we are looking to put an early escape in the reconcile loop.

For CRs to avoid reconciliation there're two different design possibilities:

Introduce support for the controller to watch only particular namespaces which is a common pattern we can introduce to the HO.
We already support pausing CRs today via API that you could use at your convenience https://github.com/openshift/hypershift/blob/main/api/v1alpha1/hostedcluster_types.go#L271

We wouldn't want to introduce a hidden API as an annotation to achieve the same behaviour/semantic with arbitrary nuances. Lets instead let the consumer leverage and design atop the generic building blocks described above.
If that's not enough for the scenario you are considering let's discuss the problem trying to be solved in a doc or so.

I'm thinking that maybe we can get away with the Pause feature.
ie. We have both ObjectRef and the HostedClusterSpec in our HypershiftDeployment.
So after thinking about it, with ObjectRef, they set that with Pause YES.  When our control reads it from ObjectRef we disable the setting and place the resource into the ManifestWork.
When we read HostedClusterSpec, we don't make any modifications so the feature is still usable.
This way we get the best of both worlds and we don't need this change.  @enxebre what do you think?

Had a discussion in the Hypershift slack channel, abandoning this merge.


[CT-2294] [Feature] Seed JSON and Parquet

Is this your first time submitting a feature request?

[X] I have read the expectations for open source contributors
[X] I have searched the existing issues, and I could not find an existing issue for this feature
[X] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature
Right now (as far as I am aware) seed files have to be CSVs.
Using something like duckdb (which critically is dependency-less) seeds could easily be extended to json or parquet as well, by having something like:
seeds/
  foo.csv
  bar.csv
  foobar.json

->
seeds/
  foo.csv
  bar.csv
  foobar.json
  __dbt_foobar.csv (temporary file, created by duckdb)

From which the usual seed method could be created. Finally dbt could delete the temp file.
Possibly it's silly to widen the net of things that could be seeded 🤷
Describe alternatives you've considered
No response
Who will this benefit?
No response
Are you interested in contributing this feature?
Yup, if i'm pointed in the right direction! (I think it's probably core/dbt/task/seed.py
Anything else?
No response

Hey @jpmmcneill, always good to see you again!
JSON seeds
There's actually a pre-existing issue for the JSON part: https://github.com/dbt-labs/dbt-core/issues/2365. Specifically, the NDJSON format was proposed with one valid JSON value per line (which will most commonly be an object or array).
Since the issue you submitted is primarily concerned with JSON seeds, I'm going to close this as a duplicate of https://github.com/dbt-labs/dbt-core/issues/2365
Parquet seeds
But it sounds like you might have a stand-alone (but related) request to be able to seed Parquet files too.
My initial (and not fully refined thoughts) related to adding official support for Parquet as a seed format is: not at this time*.
The primary reasons:

Supporting Parquet would beg the question about support other formats like ORC, Avro, etc.
There are warehouse specific workarounds to ingest Parquet today (see below for dbt-duckdb)

*With that being said, I would absolutely welcome and encourage you to open a new feature request for Parquet seeds if you want to discuss further!
Parquet in DuckDB
Let's assume you have a valid parquet file located at seeds/my_parquet_seed.parquet. For DuckDB users (using dbt-duckdb), I think parquet "seeds" are possible as-is via syntax like this:
models/my_test.sql
select * from 'seeds/my_parquet_seed.parquet'

This syntax obviously isn't a true seed, because it wouldn't support references like ref("my_parquet_seed"). But you could just do ref("my_test") instead when that is needed/desired.

Hey @dbeatty10, thanks for this! No problem to close.
I agree that supporting n != 1 always begs the question of "can you support X". I agree that there are warehouse specific workarounds but they're complete pains sometimes (I'm thinking specifically of snowflake here). Really what attracted me to duckdb for this issue is that it provides a quite headless way to turn other data types into csv's in a quite lightweight method.
So I wasn't really talking about the duckdb adapter specifically! Indeed, adding duckdb as a requirement for dbt core would give josh a tonne of dependency headaches most likely! 😂
Nice that JSON stuff exists. Sorry that I missed that issue, I'll give it a read.

Yeah, that's a clever idea of using DuckDB as lightweight format converter 🧠
Two things that would improve that approach:

CSV seeds in dbt rely on combination of type inference + column_types configuration. It would be nice if converting from Parquet to a CSV seed also emitted the appropriate column_types configuration (either database specific or database agnostic).
Currently, Parquet is one of the few file formats supported for importing by DuckDB. This approach would benefit if there were more export options.


It would be convenient if LLVM_COMPILER defaulted to clang

I always forget to set it.

Try out this puppy.

@ianamason nice!


Daisy Pod/Patch/etc schematics

I've currently been combing through the other examples for the different configurations and there are some examples that I'd like to use with the Daisy Seed.
I've currently been going to the header and class file for the Patch configuration to figure out what potentiometers are wired to which pins to try and create the configuration on a bread board.
Will the schematics or pin-outs be available for each configuration so we can attempt to wire one up ourselves?

@monkeyharhar
We have added a Hardware repo that will be home to schematics, etc. for all of the breakout boards and more.
For now, only the Daisy Seed, and the Daisy Patch schematics are up, but we'll be adding the others soon.
https://github.com/electro-smith/Hardware

closing since the repo is up. Other schematics coming soon.


wip! feat: support for multiple files/folders

Closes #472

Broken right now, as the folder structure isn't being retained.


Authenticate user from access token?

This is more of a question than an issue so please feel free to close it.
I know during a Password Grant the database stores a chain of info from oauth_access_tokens to oauth_sessions connecting the access token to a user (owner_type and owner_id). This can be used to authenticate a user but my question is is this something that's expected to be done? If not, what do you recommend for authentication?
I had half expected the oauth filter to do this automatically but I guess it's not a good practice?
Edit: To clarify: the idea is to do this in a auth middleware so Auth::user() can be used in the rest of the codebase.

Auth::user() can't be used with oauth because the authenticated client can be both accessing data on user's behalf and on it's own. Plus it could go ahead and set some cookies, and APIs should be stateless in this regard. Just stick to Authorizer::getOwnerId() and Authorizer::getOwnerType()

I.. did not know about Authorizer::getOwnerId() and Authorizer::getOwnerType()! Thanks that helps.
The idea behind using Auth::user() is just to make it easy to retrieve the owner information for access control purposes inside controllers and middleware so you can use the existing Auth stuff from Laravel. The API itself is stateless and requires no session information so it doesn't authenticate a session this way, per se. Thanks!


TaskbarX seems very offset on the right

I set primary taskbar offset to zero and it didn't work

me too i have the same thing

I set primary taskbar offset to zero and it didn't work

Open the configurator -> Position settings -> Disable "Compensate Tray Clock etc..."

This doesn't seem to work! any other way @im-csm ?

The only way I can reproduce this for myself is by setting an offset of 500 but I don't think thats the issue you're having. I thought maybe it was the task view and search button in the taskbar but that didn't even do it for me. Try opening the configurator and clicking stop taskbarx and see where that places your icons. @whoibrar


Bazel error

Hey man, first of all to start, LOVE the videos!
I have found a problem with utils/utf8/... when I run the bazel command I am thrown with this:
ERROR: no targets found beneath 'utils/utf8'.
INFO: Elapsed time: 0.102s
ERROR: Couldn't start the build. Unable to run tests.

could this be an issue as to why I cannot run the python script test.p? To point out my error looks like the following:
syntaxnet/demo.sh: line 31: bazel-bin/syntaxnet/parser_eval: No such file or directory
syntaxnet/demo.sh: line 43: bazel-bin/syntaxnet/parser_eval: No such file or directory
syntaxnet/demo.sh: line 55: bazel-bin/syntaxnet/conll2tree: No such file or directory
Traceback (most recent call last):
  File "test.py", line 10, in <module>
    root = demo_lib.get_root_word(dep_tree)
  File "/root/Desktop/AI_Reader-master/syntaxnet/demo_lib.py", line 21, in get_root_word
    root = dependency_tree[2].split()[0]
IndexError: list index out of range

Thanks in advance!

Update on the issue. I just realized it is util/utf8 by accident I added a s but now I have another issue:
ERROR: com.google.devtools.build.lib.packages.BuildFileContainsErrorsException: error loading package '': Extension file not found. Unable to load package for '//third_party/gpus:cuda_configure.bzl': BUILD file not found on package path.

Problem is I do not have a graphics card like nVidia on my laptop and while building Tensorflow I did not add Cuda support, what should I do? Would I need to use something like AWS to compile this or can the code be modified to work without GPU?

I have the same prolem, what could i do ?
I`m using a virtual machine

@baditaflorin Unfortunately have not found a solution yet.


New version: Google.Chrome.Canary version 118.0.5983.0

Pull request has been created with WinGet Automation :rocket:
Microsoft Reviewers: Open in CodeFlow

Reset PR


Not respecting subfolder file placement

Hey there!
The tool works great for my use case minus one glaring issue.  When I run my command ./node_modules/.bin/simple-scaffold text-field -t scaffolds/new-package/**/* -o packages, all of the contents get created properly but the files that were in the subdirectories just get put in the root of the directory.
Also, MDX files don't seem to work for the handlebars variable replacements.
Thanks for any help you can provide!

Hey @shaneajeffery, any update? Did one of the solutions help you?


Proficient saving throws not working

Found one issue since the last patch:
Monsters with proficient saving throws show the link when hovered over, but do not generate a saving throw roll when clicked.

Are you saying that clicking of these is not rolling a save?

Are you saying that clicking one of these is not rolling a save?

That is correct. The skills do work, it is just the saving throws for some reason. Since you cannot reproduce, I will continue to look into what may be happening.  Thank you!

Are any errors generated in the console? Are you using Better Rolls or default rolls?
Can you verify that this issue persists for you when Monster Blocks is the only module enabled?

Ok, I found the issue.
It is an incompatibility with the MESS (Moerills enhancing super-suite) add-on, "Alternative rolling and targetting" function:
Uncaught TypeError: Cannot read property 'dataset' of null
    at HTMLSpanElement.<anonymous> (modules/mess/scripts/index.js:57)
foundry.js:3855 Foundry VTT | Rendering a
modify-rolling.js:297 Uncaught TypeError: Cannot read property 'dataset' of null
    at HTMLSpanElement.<anonymous> (modify-rolling.js:297)

I have confirmed that disabling the Alternative rolling and targetting option in that add-on allows saves to work normally.