Support private container registries for Pods which do not specifying image pull secrets

What steps did you take and what happened:
Followed install instructions. Tried to run scan using below on command.
$ kubectl starboard  find vulnerabilities deployment/XXX-qa-web -n XXXX-qa -v 3
I0814 12:52:38.530373   28112 scanner.go:56] Getting Pod template for workload: {Deployment XXXX-qa-web XXXX-qa}
I0814 12:52:40.842053   28112 scanner.go:71] Scanning with options: {ScanJobTimeout:0s DeleteScanJob:true}
I0814 12:52:41.183767   28112 runner.go:79] Running task and waiting forever
I0814 12:52:41.183840   28112 runnable_job.go:47] Creating runnable job: starboard/b75ba5e8-82c9-4915-ad35-4b35c37987ab
I0814 12:52:41.535929   28112 reflector.go:207] Starting reflector *v1.Job (30m0s) from pkg/mod/k8s.io/client-go@v0.19.0-alpha.3/tools/cache/reflector.go:156
I0814 12:52:41.535978   28112 reflector.go:243] Listing and watching *v1.Job from pkg/mod/k8s.io/client-go@v0.19.0-alpha.3/tools/cache/reflector.go:156
I0814 12:52:50.479003   28112 runnable_job.go:73] Stopping runnable job on task failure with status: Failed
I0814 12:52:50.479115   28112 runner.go:83] Stopping runner on task completion with error: job failed: BackoffLimitExceeded: Job has reached the specified backoff limit
E0814 12:52:52.784556   28112 manager.go:177] Container default terminated with Error: 2020-08-14T07:22:49.629Z FATAL   unable to initialize a scanner: unable to initialize a docker scanner: 2 errors occurred:
        * unable to inspect the image (us.gcr.io/XXXX-1/XXXX:116579-23d73da-release-2019-10): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
        * GET https://us.gcr.io/v2/token?scope=repository%3xxxl-1%2FXXXX%3Apull&service=us.gcr.io: UNKNOWN: Unable to parse json key.

error: running scan job: job failed: BackoffLimitExceeded: Job has reached the specified backoff limit

What did you expect to happen:
Scan should completed without error
Anything else you would like to add:
trivy support gcr. But I am not to able find way to pass custom ENV to trivy using starboard.
Environment:

Starboard version (use starboard version): Starboard Version: {Version:0.2.6 Commit:d43faefc56021ae55d4574054ce7de13175ca206 Date:2020-07-09T20:30:45Z}
Kubernetes version (use kubectl version):  client:v1.17.10, server: v1.17.2
OS (macOS 10.15, Windows 10, Ubuntu 19.10 etc): Ubuntu 18.04

@danielpacak In case you missed it ^^

Hi @avdhoot ! Thank you for reporting this issue. I'm going to follow up on that and link it to a parent story or epic about configuring scanners. In your case the root cause is that we currently cannot pass TRIVY_USERNAME or TRIVY_PASSWORD envs, but there're more config options that users might want to pass to a scanner.

@avdhoot Actually starboard can use image pull secret attached to the Pod template or the service account. In your case did you set the secret on the Pod template level or at service account?

Our cluster created using kops. We seeded it with registry secrets using below command.  We do not have image_pull secret attached to pod. @danielpacak
kops create secret dockerconfig -f ~/.docker/config.json

Thank you for providing all the details @avdhoot So just to rephrase and confirm my understanding:
You neither use image pull Secret set at Pod template not Service Account. You're using a special feature1 of kops to store the config.json in /root/.docker/config.json on all nodes (including masters) so that both Kubernetes and system containers may use registries defined in it.

Thanks @danielpacak you are understanding is correct.

@avdhoot I renamed the title of this issue to reflect the specific use case of kops and authorizing Kubernetes nodes to pull images from private registry and repositories.
As already mentioned Starboard only supports recommended way of handling private registries with image pull secrets.
We need to think more whether / how to support such cases as you described. /cc @lizrice

@danielpacak
I am trying to make use of the above discussed functionality within AKS.  I am not finding that the operator pulls the imagePullSecret of the deployment.  In my case I am exposing the secret as imagePullSecret on the deployment level.
ACR Cred Secret:

apiVersion: v1
data:
.dockerconfigjson: eyJhdXRocyI6eyJhYWdh[...]HSm1PR016WkRkak1UY3dPa2t5T0M1K09XRkZYMHhLZFM1UUxsY3ROMkptT1RZd1J6ZDJZa2QrVW5GM1RYRT0ifX19
kind: Secret
metadata:
creationTimestamp: "2020-11-09T18:02:46Z"
managedFields:
apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:data:
.: {}
f:.dockerconfigjson: {}
f:type: {}
manager: kubectl-create
operation: Update
time: "2020-11-09T18:02:46Z"
name: acrcred
namespace: platform
resourceVersion: "14763349"
selfLink: /api/v1/namespaces/platform/secrets/acrcred
uid: 886f8fca-4ae6-4a9a-a6b0-133a75d606af
type: kubernetes.io/dockerconfigjson`

Deployment Spec:

spec:
containers:
envFrom:
configMapRef:
name: pnrhelperservice-configmap
image: aagacr.azurecr.io/platform/pnrhelperservice:20201109.3
imagePullPolicy: Always
livenessProbe:
failureThreshold: 3
httpGet:
path: /api/v1/ping
port: 8080
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: platform-pnrhelperservice
ports:
containerPort: 8080
name: pnrhelpersvc
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /api/v1/ping
port: 8080
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: 200m
memory: 64Mi
requests:
cpu: 100m
memory: 32Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
ALL
privileged: false
runAsNonRoot: true
runAsUser: 10000
startupProbe:
failureThreshold: 30
httpGet:
path: /api/v1/ping
port: 8080
scheme: HTTP
initialDelaySeconds: 15
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
mountPath: /tmp
name: tmp
mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: svc-platform-pnrhelperservice-token-zjpdx
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
imagePullSecrets:
name: acrcred
nodeName: aks-apppool-15092464-vmss00000y
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 10000
runAsNonRoot: true
runAsUser: 10000
serviceAccount: svc-platform-pnrhelperservice
serviceAccountName: svc-platform-pnrhelperservice
terminationGracePeriodSeconds: 30
tolerations:
effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
effect: NoSchedule
key: node.kubernetes.io/memory-pressure
operator: Exists
volumes:
emptyDir: {}
name: tmp
name: tls-platform-pnrhelperservice
secret:
defaultMode: 420
secretName: tls-platform-pnrhelperservice
name: acrcred
secret:
defaultMode: 420
secretName: acrcred
name: svc-platform-pnrhelperservice-token-zjpdx
secret:
defaultMode: 420
secretName: svc-platform-pnrhelperservice-token-zjpdx

Operator Logs:

Loading...
{"level":"info","ts":1604949460.8825052,"logger":"main","msg":"Resolving install mode","install mode":"SingleNamespace","operator namespace":"starboard-operator","target namespaces":["platform"]}
{"level":"info","ts":1604949460.8826096,"logger":"main","msg":"Constructing multi-namespaced cache","namespaces":["platform","starboard-operator"]}
{"level":"info","ts":1604949461.888167,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1604949461.888932,"logger":"main","msg":"Using Trivy as vulnerability scanner","image":"aquasec/trivy:0.11.0"}
{"level":"info","ts":1604949461.8890698,"logger":"main","msg":"Starting controllers manager"}
{"level":"info","ts":1604949461.8893642,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1604949461.8895195,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod","source":"kind source: /, Kind="}
{"level":"info","ts":1604949461.8897321,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job","source":"kind source: /, Kind="}
{"level":"info","ts":1604949462.0902774,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod"}
{"level":"info","ts":1604949462.0903811,"logger":"controller","msg":"Starting workers","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod","worker count":1}
{"level":"info","ts":1604949462.0904198,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job"}
{"level":"info","ts":1604949462.090432,"logger":"controller","msg":"Starting workers","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job","worker count":1}
{"level":"error","ts":1604949471.7839265,"logger":"controller.job","msg":"Scan job container","job":"starboard-operator/2b1b65b9-e131-4e13-9601-68fa1eb3cab6","container":"platform-pnrhelperservice","status.reason":"Error","status.message":"2020-11-09T19:17:51.225Z\t\u001b[31mFATAL\u001b[0m\tunable to initialize a scanner: unable to initialize a docker scanner: 2 errors occurred:\n\t* unable to inspect the image (aagacr.azurecr.io/platform/pnrhelperservice:20201109.3): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\n\t* GET https://aagacr.azurecr.io/oauth2/token?scope=repository%3Aplatform%2Fpnrhelperservice%3Apull&service=aagacr.azurecr.io: UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization for more information.\n\n\n","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/runner/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\ngithub.com/aquasecurity/starboard/pkg/operator/controller/job.(*JobController).processFailedScanJob\n\t/home/runner/work/starboard/starboard/pkg/operator/controller/job/job_controller.go:163\ngithub.com/aquasecurity/starboard/pkg/operator/controller/job.(*JobController).Reconcile\n\t/home/runner/work/starboard/starboard/pkg/operator/controller/job/job_controller.go:69\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:244\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).worker\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:197\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:90"}
{"level":"info","ts":1604949460.8825052,"logger":"main","msg":"Resolving install mode","install mode":"SingleNamespace","operator namespace":"starboard-operator","target namespaces":["platform"]}
{"level":"info","ts":1604949460.8826096,"logger":"main","msg":"Constructing multi-namespaced cache","namespaces":["platform","starboard-operator"]}
{"level":"info","ts":1604949461.888167,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1604949461.888932,"logger":"main","msg":"Using Trivy as vulnerability scanner","image":"aquasec/trivy:0.11.0"}
{"level":"info","ts":1604949461.8890698,"logger":"main","msg":"Starting controllers manager"}
{"level":"info","ts":1604949461.8893642,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1604949461.8895195,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod","source":"kind source: /, Kind="}
{"level":"info","ts":1604949461.8897321,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job","source":"kind source: /, Kind="}
{"level":"info","ts":1604949462.0902774,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod"}
{"level":"info","ts":1604949462.0903811,"logger":"controller","msg":"Starting workers","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod","worker count":1}
{"level":"info","ts":1604949462.0904198,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job"}
{"level":"info","ts":1604949462.090432,"logger":"controller","msg":"Starting workers","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job","worker count":1}
{"level":"error","ts":1604949471.7839265,"logger":"controller.job","msg":"Scan job container","job":"starboard-operator/2b1b65b9-e131-4e13-9601-68fa1eb3cab6","container":"platform-pnrhelperservice","status.reason":"Error","status.message":"2020-11-09T19:17:51.225Z\t\u001b[31mFATAL\u001b[0m\tunable to initialize a scanner: unable to initialize a docker scanner: 2 errors occurred:\n\t unable to inspect the image (aagacr.azurecr.io/platform/pnrhelperservice:20201109.3): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\n\t GET https://aagacr.azurecr.io/oauth2/token?scope=repository%3Aplatform%2Fpnrhelperservice%3Apull&service=aagacr.azurecr.io: UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization for more information.\n\n\n","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/runner/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\ngithub.com/aquasecurity/starboard/pkg/operator/controller/job.(*JobController).processFailedScanJob\n\t/home/runner/work/starboard/starboard/pkg/operator/controller/job/job_controller.go:163\ngithub.com/aquasecurity/starboard/pkg/operator/controller/job.(*JobController).Reconcile\n\t/home/runner/work/starboard/starboard/pkg/operator/controller/job/job_controller.go:69\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:244\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:197\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:90"}

Looking forward to your response, as it would save my day.

@danielpacak
I am trying to make use of the above discussed functionality within AKS. I am not finding that the operator pulls the imagePullSecret of the deployment. In my case I am exposing the secret as imagePullSecret on the deployment level.
Secret generation command as detailed here: ACR Auth Docs

kubectl create secret docker-registry acrcred --namespace platform --docker-server=aagacr.azurecr.io --docker-username=$(AagacrUser) --docker-password=$(AagacrKey)

ACR Cred Secret:

apiVersion: v1
data:
.dockerconfigjson: eyJhdXRocyI6eyJhYWdh[...]HSm1PR016WkRkak1UY3dPa2t5T0M1K09XRkZYMHhLZFM1UUxsY3ROMkptT1RZd1J6ZDJZa2QrVW5GM1RYRT0ifX19
kind: Secret
metadata:
creationTimestamp: "2020-11-09T18:02:46Z"
managedFields:
apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:data:
.: {}
f:.dockerconfigjson: {}
f:type: {}
manager: kubectl-create
operation: Update
time: "2020-11-09T18:02:46Z"
name: acrcred
namespace: platform
resourceVersion: "14763349"
selfLink: /api/v1/namespaces/platform/secrets/acrcred
uid: 886f8fca-4ae6-4a9a-a6b0-133a75d606af
type: kubernetes.io/dockerconfigjson`

Deployment Spec:

spec:
containers:
envFrom:
configMapRef:
name: pnrhelperservice-configmap
image: aagacr.azurecr.io/platform/pnrhelperservice:20201109.3
imagePullPolicy: Always
livenessProbe:
failureThreshold: 3
httpGet:
path: /api/v1/ping
port: 8080
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: platform-pnrhelperservice
ports:
containerPort: 8080
name: pnrhelpersvc
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /api/v1/ping
port: 8080
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: 200m
memory: 64Mi
requests:
cpu: 100m
memory: 32Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
ALL
privileged: false
runAsNonRoot: true
runAsUser: 10000
startupProbe:
failureThreshold: 30
httpGet:
path: /api/v1/ping
port: 8080
scheme: HTTP
initialDelaySeconds: 15
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
mountPath: /tmp
name: tmp
mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: svc-platform-pnrhelperservice-token-zjpdx
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
imagePullSecrets:
name: acrcred
nodeName: aks-apppool-15092464-vmss00000y
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 10000
runAsNonRoot: true
runAsUser: 10000
serviceAccount: svc-platform-pnrhelperservice
serviceAccountName: svc-platform-pnrhelperservice
terminationGracePeriodSeconds: 30
tolerations:
effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
effect: NoSchedule
key: node.kubernetes.io/memory-pressure
operator: Exists
volumes:
emptyDir: {}
name: tmp
name: tls-platform-pnrhelperservice
secret:
defaultMode: 420
secretName: tls-platform-pnrhelperservice
name: acrcred
secret:
defaultMode: 420
secretName: acrcred
name: svc-platform-pnrhelperservice-token-zjpdx
secret:
defaultMode: 420
secretName: svc-platform-pnrhelperservice-token-zjpdx

Operator Logs:

Loading...
{"level":"info","ts":1604949460.8825052,"logger":"main","msg":"Resolving install mode","install mode":"SingleNamespace","operator namespace":"starboard-operator","target namespaces":["platform"]}
{"level":"info","ts":1604949460.8826096,"logger":"main","msg":"Constructing multi-namespaced cache","namespaces":["platform","starboard-operator"]}
{"level":"info","ts":1604949461.888167,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1604949461.888932,"logger":"main","msg":"Using Trivy as vulnerability scanner","image":"aquasec/trivy:0.11.0"}
{"level":"info","ts":1604949461.8890698,"logger":"main","msg":"Starting controllers manager"}
{"level":"info","ts":1604949461.8893642,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1604949461.8895195,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod","source":"kind source: /, Kind="}
{"level":"info","ts":1604949461.8897321,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job","source":"kind source: /, Kind="}
{"level":"info","ts":1604949462.0902774,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod"}
{"level":"info","ts":1604949462.0903811,"logger":"controller","msg":"Starting workers","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod","worker count":1}
{"level":"info","ts":1604949462.0904198,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job"}
{"level":"info","ts":1604949462.090432,"logger":"controller","msg":"Starting workers","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job","worker count":1}
{"level":"error","ts":1604949471.7839265,"logger":"controller.job","msg":"Scan job container","job":"starboard-operator/2b1b65b9-e131-4e13-9601-68fa1eb3cab6","container":"platform-pnrhelperservice","status.reason":"Error","status.message":"2020-11-09T19:17:51.225Z\t\u001b[31mFATAL\u001b[0m\tunable to initialize a scanner: unable to initialize a docker scanner: 2 errors occurred:\n\t* unable to inspect the image (aagacr.azurecr.io/platform/pnrhelperservice:20201109.3): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\n\t* GET https://aagacr.azurecr.io/oauth2/token?scope=repository%3Aplatform%2Fpnrhelperservice%3Apull&service=aagacr.azurecr.io: UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization for more information.\n\n\n","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/runner/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\ngithub.com/aquasecurity/starboard/pkg/operator/controller/job.(*JobController).processFailedScanJob\n\t/home/runner/work/starboard/starboard/pkg/operator/controller/job/job_controller.go:163\ngithub.com/aquasecurity/starboard/pkg/operator/controller/job.(*JobController).Reconcile\n\t/home/runner/work/starboard/starboard/pkg/operator/controller/job/job_controller.go:69\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:244\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).worker\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:197\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:90"}
{"level":"info","ts":1604949460.8825052,"logger":"main","msg":"Resolving install mode","install mode":"SingleNamespace","operator namespace":"starboard-operator","target namespaces":["platform"]}
{"level":"info","ts":1604949460.8826096,"logger":"main","msg":"Constructing multi-namespaced cache","namespaces":["platform","starboard-operator"]}
{"level":"info","ts":1604949461.888167,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1604949461.888932,"logger":"main","msg":"Using Trivy as vulnerability scanner","image":"aquasec/trivy:0.11.0"}
{"level":"info","ts":1604949461.8890698,"logger":"main","msg":"Starting controllers manager"}
{"level":"info","ts":1604949461.8893642,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1604949461.8895195,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod","source":"kind source: /, Kind="}
{"level":"info","ts":1604949461.8897321,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job","source":"kind source: /, Kind="}
{"level":"info","ts":1604949462.0902774,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod"}
{"level":"info","ts":1604949462.0903811,"logger":"controller","msg":"Starting workers","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod","worker count":1}
{"level":"info","ts":1604949462.0904198,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job"}
{"level":"info","ts":1604949462.090432,"logger":"controller","msg":"Starting workers","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job","worker count":1}
{"level":"error","ts":1604949471.7839265,"logger":"controller.job","msg":"Scan job container","job":"starboard-operator/2b1b65b9-e131-4e13-9601-68fa1eb3cab6","container":"platform-pnrhelperservice","status.reason":"Error","status.message":"2020-11-09T19:17:51.225Z\t\u001b[31mFATAL\u001b[0m\tunable to initialize a scanner: unable to initialize a docker scanner: 2 errors occurred:\n\t unable to inspect the image (aagacr.azurecr.io/platform/pnrhelperservice:20201109.3): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\n\t GET https://aagacr.azurecr.io/oauth2/token?scope=repository%3Aplatform%2Fpnrhelperservice%3Apull&service=aagacr.azurecr.io: UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization for more information.\n\n\n","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/runner/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\ngithub.com/aquasecurity/starboard/pkg/operator/controller/job.(*JobController).processFailedScanJob\n\t/home/runner/work/starboard/starboard/pkg/operator/controller/job/job_controller.go:163\ngithub.com/aquasecurity/starboard/pkg/operator/controller/job.(*JobController).Reconcile\n\t/home/runner/work/starboard/starboard/pkg/operator/controller/job/job_controller.go:69\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:244\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:197\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:90"}

Looking forward to your response, as it would save my day.

Hi @n3rd253 Currently we only support private registries with Starboard CLI. I hasn't been implemented for the operator yet, but we're working on that

Hi guys !
Just discovered this awesome project and I was wondering why Starboard could not fetch the images from my private repo :p
Is there any news / ETA on this or is this something you don't consider to add on a short time ?
Thank you very much anyway, it's a great job !

Hi @mathieudebrito Starboard CLI and Starboard Operator v0.9 should read image pull Secret from pod template as explained here. If that's not the case it may be regression. I'll double check on our end.
Could you provide more details regarding the private registry used in your setup? Is it Docker Hub or a managed registry such as ECR or GCR? Which authentication method are you using? Basic vs auth token?

Hi @danielpacak Thank you for your quick answer !

I installed yesterday the Starboard CLI and the Starboard Operator v0.9 (so I think they are up to date)
We use Scaleway manage registry : https://www.scaleway.com/en/container-registry/ (auth token)

If it helps, the secret is created as follow :
kubectl create secret docker-registry regcred --docker-server=$REGISTRY_URL --docker-username=$REGISTRY_USER --docker-password=$REGISTRY_PASSWORD --dry-run -o yaml | kubectl apply -f -

Thank you again !

Hey @mathieudebrito I can reproduce this error so it seems to be regression or a corner case that's not handled properly by Starboard (both operator and CLI as they share this functions). I'll track it down and try to fix in in the upcoming bug fix release v0.9.1.
Note: For some reason we cannot map registry server to container that refers to private image around these lines https://github.com/aquasecurity/starboard/blob/main/pkg/kube/secrets.go#L49

I found what was causing this problem. Essentially you can create image pull secret which is actually concatenation of registry server (rg.pl-waw.scw.cloud) and repository name (private). If that was the case Starboard could not parse registry server properly and match it with registry server taken from image reference
This was causing problem in v0.9.0
$ kubectl create secret docker-registry regcred \
  --docker-server=rg.pl-waw.scw.cloud/private \
  --docker-username=nologin \
  --docker-password=$SCW_SECRET_TOKEN --dry-run -o yaml | kubectl apply -f -

This would work in v0.9.0
$ kubectl create secret docker-registry regcred \
  --docker-server=rg.pl-waw.scw.cloud \
  --docker-username=nologin \
  --docker-password=$SCW_SECRET_TOKEN --dry-run -o yaml | kubectl apply -f -

cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      imagePullSecrets:
      - name: regcred
      containers:
      - image: rg.pl-waw.scw.cloud/private/nginx:1.16
        imagePullPolicy: Always
        name: nginx
EOF

@danielpacak awesome thank you !
Indeed, I understand the issue is not the same, I should have created a new one, sorry :(

Hi, any chance to add supporting the private registry to the Helm, please?
I'm getting this error:
unable to initialize a scanner: unable to initialize a docker scanner: 2 errors occurred:\n\t* unable to inspect the image (****.azurecr.io/app:v1): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\n\t* GET https://****.azurecr.io/oauth2/token?scope=repository%3Aapp%3Apull&service=****.azurecr.io: UNAUTHORIZED: authentication required

Hi, any chance to add supporting the private registry to the Helm, please?
I'm getting this error:
unable to initialize a scanner: unable to initialize a docker scanner: 2 errors occurred:\n\t* unable to inspect the image (****.azurecr.io/app:v1): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\n\t* GET https://****.azurecr.io/oauth2/token?scope=repository%3Aapp%3Apull&service=****.azurecr.io: UNAUTHORIZED: authentication required

Please create a separate issue for that @maati55 . This issue has nothing to do with Helm charts. We'll never close it if we report this way.

Thank you for providing all the details @avdhoot So just to rephrase and confirm my understanding:
You neither use image pull Secret set at Pod template not Service Account. You're using a feature1 of kops to store the config.json in /root/.docker/config.json on all nodes (including masters) so that both Kubernetes and system containers may use registries defined in it.

@avdhoot I renamed the title of this issue to reflect the specific use case of kops and authorizing Kubernetes nodes to pull images from private registry and repositories.
As already mentioned Starboard only supports recommended way of handling private registries with image pull secrets. We need to think more whether / how to support such cases as you described. /cc @lizrice

We are running into this same issue. Could it perhaps be as simple as adding a configuration to mount the Docker config file from the host as a volume onto the Starboard pods doing the scanning?

👋 @ryanbrainard I think we have already mentioned somewhere that mounting container runtime sockets is suboptimal for these reasons:

Starboard must figure out the Container Runtime used on each node. (Not sure Starboard should be aware of such infrastructure details.)
This would probably require scheduling scan jobs on a particular node where a scanned container image is cached. What if that not does not have enough resources or is tainted, etc.?
Additional permissions are usually required to run scan pods that use hostPath volume mounts (e.g. OpenShift, PSP and its successors, built-in and custom K8s admission controllers)

If you want to contribute to designing such functionality we'd appreciate reviewing design docs under the docs/design directory that are related to private and managed registries. You can also submit your own design proposal by following the template from existing docs.


same arg,2.12-2.2.0 kafka is ok,but 2.11-<IP_ADDRESS> fail

Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:
1.
2.
3.
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Version (please complete the following information):

OS: [e.g. Linux]
Jaeger version: [e.g. 1.8]
Deployment: [e.g. bare metal, Docker, Kubernetes, ...]

What troubleshooting steps did you try?
Try to follow https://www.jaegertracing.io/docs/latest/troubleshooting/ and describe how far you were able to progress and/or which steps did not work.
Additional context
Add any other context about the problem here.
{"level":"info","ts":1619111640.6141622,"caller":"flags/service.go:117","msg":"Mounting metrics handler on admin server","route":"/metrics"}
{"level":"info","ts":1619111640.6142764,"caller":"flags/service.go:123","msg":"Mounting expvar handler on admin server","route":"/debug/vars"}
{"level":"info","ts":1619111640.6146374,"caller":"flags/admin.go:105","msg":"Mounting health check on admin server","route":"/"}
{"level":"info","ts":1619111640.6147656,"caller":"flags/admin.go:111","msg":"Starting admin HTTP server","http-addr":":14269"}
{"level":"info","ts":1619111640.6148138,"caller":"flags/admin.go:97","msg":"Admin server started","http.host-port":"[::]:14269","health-status":"unavailable"}
{"level":"info","ts":1619111640.622844,"caller":"kafka/factory.go:69","msg":"Kafka factory","producer builder":{"Brokers":["<IP_ADDRESS>:32047"],"RequiredAcks":1,"Compression":0,"CompressionLevel":0,"ProtocolVersion":"","BatchLinger":0,"BatchSize":0,"BatchMinMessages":0,"BatchMaxMessages":0,"Authentication":"plaintext","Kerberos":{"ServiceName":"kafka","Realm":"","UseKeyTab":false,"Username":"","ConfigPath":"/etc/krb5.conf","KeyTabPath":"/etc/security/kafka.keytab"},"TLS":{"Enabled":false,"CAPath":"","CertPath":"","KeyPath":"","ServerName":"","ClientCAPath":"","SkipHostVerify":false},"PlainText":{"Username":"kafka-tianyan-dyl","Mechanism":"SCRAM-SHA-256"}},"topic":"jaeger-spans"}
{"level":"fatal","ts":1619111641.639278,"caller":"command-line-arguments/main.go:75","msg":"Failed to init storage factory","error":"kafka: client has run out of available brokers to talk to (Is your cluster reachable?)","stacktrace":"main.main.func1\n\tcommand-line-arguments/main.go:75\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/cobra@v0.0.7/command.go:838\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v0.0.7/command.go:943\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v0.0.7/command.go:883\nmain.main\n\tcommand-line-arguments/main.go:137\nruntime.main\n\truntime/proc.go:204"}
    env:
    - name: SPAN_STORAGE_TYPE
      value: kafka
    - name: KAFKA_PRODUCER_BROKERS
      value: 'xxx'
    - name: KAFKA_TOPIC
      value: jaeger-spans

    - /go/bin/collector-linux
    - --collector.zipkin.host-port
    - "9411"
    - --kafka.producer.plaintext.username
    - xxxx
    - --kafka.producer.plaintext.password
    - xxxx
    - --kafka.producer.authentication
    - plaintext
    - --kafka.producer.encoding
    - json
    - --kafka.producer.plaintext.mechanism
    - 'SCRAM-SHA-256'
    - --log-level
    - debug

@wenhx8 did you resolve this issue?

Hi @wenhx8.
Did you resolved the issue?

not resolve issue


bootstrap-vcpkg.sh build failure

I want to install vcpkg on jetson nano (Ubuntu 18.04)
git clone https://github.com/microsoft/vcpkg
Clones everything correctly, then
./vcpkg/bootstrap-vcpkg.sh
The error I get
Unable to determine a binary release of vcpkg; attempting to build from source.
Downloading vcpkg tool sources
Building vcpkg-tool...
-- The CXX compiler identification is GNU 7.5.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting the C++ compiler in use
-- Detecting the C++ compiler in use - gcc
-- Module support is disabled.
-- Version: 9.1.0
-- Build type: Release
-- CXX_STANDARD: 17
-- Performing Test has_std_17_flag
-- Performing Test has_std_17_flag - Success
-- Performing Test has_std_1z_flag
-- Performing Test has_std_1z_flag - Success
-- Required features: cxx_variadic_templates
-- The C compiler identification is GNU 7.5.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Performing Test CPP_ATOMIC_BUILTIN
-- Performing Test CPP_ATOMIC_BUILTIN - Success
-- Looking for C++ include pthread.h
-- Looking for C++ include pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE
-- Configuring done
-- Generating done
-- Build files have been written to: /mnt/ALPR_Cpp/vcpkg/buildtrees/_vcpkg/build
[0/2] Re-checking globbed directories...
[42/191] Building CXX object CMakeFile...glib.dir/src/vcpkg/binarycaching.cpp.o
FAILED: CMakeFiles/vcpkglib.dir/src/vcpkg/binarycaching.cpp.o
/usr/bin/c++ -DVCPKG_BASE_VERSION=2999-12-31 -DVCPKG_VERSION=unknownhash -I/mnt/ALPR_Cpp/vcpkg/buildtrees/_vcpkg/src/vcpkg-tool-2022-12-14/include -I_deps/fmt-src/include -I_cmrc/include -O3 -DNDEBUG -include /mnt/ALPR_Cpp/vcpkg/buildtrees/_vcpkg/src/vcpkg-tool-2022-12-14/include/pch.h -pthread -std=c++1z -MD -MT CMakeFiles/vcpkglib.dir/src/vcpkg/binarycaching.cpp.o -MF CMakeFiles/vcpkglib.dir/src/vcpkg/binarycaching.cpp.o.d -o CMakeFiles/vcpkglib.dir/src/vcpkg/binarycaching.cpp.o -c /mnt/ALPR_Cpp/vcpkg/buildtrees/_vcpkg/src/vcpkg-tool-2022-12-14/src/vcpkg/binarycaching.cpp
In file included from /usr/include/c++/7/bits/stl_tempbuf.h:60:0,
from /usr/include/c++/7/bits/stl_algo.h:62,
from /usr/include/c++/7/algorithm:62,
from /mnt/ALPR_Cpp/vcpkg/buildtrees/_vcpkg/src/vcpkg-tool-2022-12-14/include/pch.h:25,
from :0:
/usr/include/c++/7/bits/stl_construct.h: In instantiation of ‘void std::_Construct(_T1*, _Args&& ...) [with _T1 = std::unique_ptrvcpkg::IBinaryProvider; _Args = {const std::unique_ptr<vcpkg::IBinaryProvider, std::default_deletevcpkg::IBinaryProvider >&}]’:
/usr/include/c++/7/bits/stl_uninitialized.h:83:18:   required from ‘static _ForwardIterator std::__uninitialized_copy<_TrivialValueTypes>::__uninit_copy(_InputIterator, _InputIterator, _ForwardIterator) [with _InputIterator = __gnu_cxx::__normal_iterator<const std::unique_ptrvcpkg::IBinaryProvider, std::vector<std::unique_ptrvcpkg::IBinaryProvider > >; _ForwardIterator = std::unique_ptrvcpkg::IBinaryProvider; bool _TrivialValueTypes = false]’
/usr/include/c++/7/bits/stl_uninitialized.h:134:15:   required from ‘_ForwardIterator std::uninitialized_copy(_InputIterator, _InputIterator, _ForwardIterator) [with _InputIterator = __gnu_cxx::__normal_iterator<const std::unique_ptrvcpkg::IBinaryProvider, std::vector<std::unique_ptrvcpkg::IBinaryProvider > >; _ForwardIterator = std::unique_ptrvcpkg::IBinaryProvider]’
/usr/include/c++/7/bits/stl_uninitialized.h:289:37:   required from ‘_ForwardIterator std::__uninitialized_copy_a(_InputIterator, _InputIterator, _ForwardIterator, std::allocator<_Tp>&) [with _InputIterator = __gnu_cxx::__normal_iterator<const std::unique_ptrvcpkg::IBinaryProvider, std::vector<std::unique_ptrvcpkg::IBinaryProvider > >; _ForwardIterator = std::unique_ptrvcpkg::IBinaryProvider; _Tp = std::unique_ptrvcpkg::IBinaryProvider]’
/usr/include/c++/7/bits/stl_vector.h:331:31:   required from ‘std::vector<_Tp, _Alloc>::vector(const std::vector<_Tp, _Alloc>&) [with _Tp = std::unique_ptrvcpkg::IBinaryProvider; _Alloc = std::allocator<std::unique_ptrvcpkg::IBinaryProvider >]’
/mnt/ALPR_Cpp/vcpkg/buildtrees/_vcpkg/src/vcpkg-tool-2022-12-14/include/vcpkg/base/expected.h:42:57:   required from ‘vcpkg::ExpectedHolder::ExpectedHolder(Fwd&&) [with Fwd = std::vector<std::unique_ptrvcpkg::IBinaryProvider >&; typename std::enable_if<(! is_same_v<vcpkg::ExpectedHolder, typename std::remove_cv<typename std::remove_reference<_SrcTuple>::type>::type>), int>::type  = 0; T = std::vector<std::unique_ptrvcpkg::IBinaryProvider >]’
/mnt/ALPR_Cpp/vcpkg/buildtrees/_vcpkg/src/vcpkg-tool-2022-12-14/include/vcpkg/base/expected.h:77:85:   required from ‘vcpkg::ExpectedT<T, Error>::ExpectedT(ConvToT&&) [with ConvToT = std::vector<std::unique_ptrvcpkg::IBinaryProvider >&; typename std::enable_if<(is_convertible_v<ConvToT, T> && (! is_same_v<typename std::remove_reference<_SrcTuple>::type, Error>)), int>::type  = 0; T = std::vector<std::unique_ptrvcpkg::IBinaryProvider >; Error = std::__cxx11::basic_string]’
/mnt/ALPR_Cpp/vcpkg/buildtrees/_vcpkg/src/vcpkg-tool-2022-12-14/src/vcpkg/binarycaching.cpp:2312:12:   required from here
/usr/include/c++/7/bits/stl_construct.h:75:7: error: use of deleted function ‘std::unique_ptr<_Tp, _Dp>::unique_ptr(const std::unique_ptr<_Tp, _Dp>&) [with _Tp = vcpkg::IBinaryProvider; _Dp = std::default_deletevcpkg::IBinaryProvider]’
{ ::new(static_cast<void*>(__p)) _T1(std::forward<_Args>(__args)...); }
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/c++/7/memory:80:0,
from /mnt/ALPR_Cpp/vcpkg/buildtrees/_vcpkg/src/vcpkg-tool-2022-12-14/include/pch.h:35,
from :0:
/usr/include/c++/7/bits/unique_ptr.h:383:7: note: declared here
unique_ptr(const unique_ptr&) = delete;
^~~~~~~~~~
[47/191] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/build.cpp.o
ninja: build stopped: subcommand failed.
ninja version I have installed 1.8.2
I have tried to reinstall ninja and tried to use --useSystemBinaries flag recommended here
./bootstrap-vcpkg.sh --useSystemBinaries
but it said
Warning: -useSystemBinaries no longer has any effect; ignored. Note that the VCPKG_USE_SYSTEM_BINARIES environment variable behavior is not changed.
then continued the above process.
Please help to solve this problem.
If you need an additional information, just tell me.

i wonder what the gcc version is

https://github.com/microsoft/vcpkg-tool/blob/ab767114e03632f020cf05e90e266a8db377f1c9/src/vcpkg/binarycaching.cpp#L2312
@BillyONeal IIUC, this code relies on NRVO which this particular compiler did not implement, so an explicit move is required.
https://godbolt.org/z/ddq4abK47

Which version of gcc do you use?

gcc (Ubuntu/Linaro 7.5.0-3ubuntu1~18.04) 7.5.0

What do I need to do here?

What do I need to do here?

Why do you even build vcpkg from source? Try running ./bootstrap-vcpkg.sh directly from vcpkg directory.
If you absolutely sure you need to build vcpkg from source, see my previous post (e.g. patch sources). Don't forget to submit patch to vcpkg-tool repo.

https://github.com/microsoft/vcpkg-tool/blob/ab767114e03632f020cf05e90e266a8db377f1c9/src/vcpkg/binarycaching.cpp#L2312 @BillyONeal IIUC, this code relies on NRVO which this particular compiler did not implement, so an explicit move is required. https://godbolt.org/z/ddq4abK47

This is not NRVO: the return type and the returned value's type aren't the same, so there's a materialized temporary. The standard allows eliminating the move operation here, but returning an expression referencing a variable with automatic storage duration invokes a move. But GCC 7.5 has some form of bug where it tries to copy instead. I'm linking several examples here because this wording has changed a lot between versions of C++:

In C++14: https://timsong-cpp.github.io/cppwp/n4140/class.copy#32
Note "id-expression that names an object with automatic storage duration declared in the body"
In C++17: https://timsong-cpp.github.io/cppwp/n4659/class.copy#elision-3
In C++20: https://timsong-cpp.github.io/cppwp/n4868/class#copy.elision-3

The C++23 will clarify this wording a bit: https://www.open-std.org/JTC1/SC22/WG21/docs/papers/2022/p2266r3.html
I don't know exactly what conditions trigger the bug; the unfortunate thing is that adding an explicit std::move here can pessimize all current compilers. We would be more than willing to accept that pessimization; performance doesn't matter here. The bigger problem is that GCC 7.5 is not one of our production targeted compilers, so any fix is going to rot fast as we are not testing it.

What do I need to do here?

Why do you even build vcpkg from source? Try running ./bootstrap-vcpkg.sh directly from vcpkg directory. If you absolutely sure you need to build vcpkg from source, see my previous post (e.g. patch sources). Don't forget to submit patch to vcpkg-tool repo.

I believe:

jetson nano

means arm-linux for which we do not have a prebuilt binary. We would be happy to provide such a prebuilt binary but don't have a great understanding of how to produce one that has a high likelihood of running on any given arm linux system. We produce the amd64 Linux binaries today by building on an extremely old copy of RHEL that supports GCC9 targeting ancient copies of glibc et al.

We produce the amd64 Linux binaries today by building on an extremely old copy of RHEL that supports GCC9 targeting ancient copies of glibc et al.

Does it have Clang which can build vcpkg? The solution might be as easy as cross-compiling vcpkg and linking everything statically except glibc.

We would be happy to provide such a prebuilt binary but don't have a great understanding of how to produce one that has a high likelihood of running on any given arm linux system.

There is one trick to make an executable portable, but it requires you to:

Build libc++, libc++abi, libunwind from sources together with vcpkg
Use fixed (old) version of standard C libraries (pthread, m, dl, etc.)

There is at least one vendor who does that quite successfully: https://github.com/ClickHouse/ClickHouse/tree/master/cmake/linux

Does it have Clang which can build vcpkg?

No.

The solution might be as easy as cross-compiling vcpkg and linking everything statically except glibc.

The glibc version itself is a source of incompatibilities, hence the old Linux version.

There is at least one vendor who does that quite successfully:

The question isn't whether it is possible, the question is can we get enough confidence in it to replace building from source as something we have confidence will work. With nothing to test with we do not have that confidence.

The question isn't whether it is possible, the question is can we get enough confidence in it to replace building from source as something we have confidence will work. With nothing to test with we do not have that confidence.

Adding a check for glibc version into bootstrap-vcpkg.sh may boost one's confidence ;)

I needed curl and jsoncpp from vcpkg, so while waiting for your response I installed them separately from different sources. I'm trying to find links to share with you, will share when i find them. Now closing the issue. Thank you all ))).

I created https://github.com/microsoft/vcpkg/issues/28570 to track the 'meta issue' here so that @Dav00Arm can avoid getting spammed with notifications.

In case someone is still looking for a workaround, if you upgrade the gcc / g++ version to 8 and set it as the system default, it works. I managed to install vcpkg on Jetson Nano.
sudo apt install -y g++-8 gcc-8
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-8 60 --slave /usr/bin/g++ g++ /usr/bin/g++-8
sudo update-alternatives --config gcc


Break code out into modules

Relevant:

Do you have experience with using SwiftPM in the project that already use CococaPods (like ours). Any gotchas?

as long as what you are putting in the swift packages does not depend on any CocoPods works fine
I forgot about this case. We would need this. So maybe SPM is out? Or at least we'll need both for internal modules?

Setting boundaries help improve architecture and would probably help build times too.

@vladyslav-iosdev it looks like if we use CocoaPods within the module, it'll have to be a CocoaPod (and not a SPM package) too. Unless you know otherwise?
That means we can either
A) build all the new modules as CocoaPods or
B) use SPM packages where possible and drop down to CocoaPods when we can't.
The rationale for (B) is hopefully we will eventually drop all CocoaPods.. I don't know if that's realistic.
What do you think @vladyslav-iosdev ?

Superseded by #2918


Unify model path configurations using ResourcePath

Describe your changes
This PR introduces a new class called olive.resource_path.ResourcePath that is able to handle many different types of model sources, both local and remote.
Currently, the model path/source is specified through multiple parameters model_path, name, version, model_storage_kind. However, there is no systematic way in which these parameters interact. They were added as new features were implemented.
Using ResourcePath, we can unify all of these into a single class that can be used by all model classes.
The supported resource types are local file, local folder, string name, azureml model, azureml datastore and azureml job outputs. We can add other sources easily.
We also remove name from OliveModel since it doesn't have any use and might be a point of confusion.
Checklist before requesting a review

[x] Add unit tests for this change.
[x] Make sure all tests can pass.
[x] Update documents if necessary.
[x] Format your code by running pre-commit run --all-files

(Optional) Issue link

The failure was caused by timeout when run component detection.  @jambayk Please rebase main branch with latest changes which Lei fixed.


Move transclusion updates to the page_edit rule

Currently, figuring out if a page is transcluded is a separate rule from
page_edit. This means that the transclude process will run even if there
are failures in rendering the new revision in RB/Parsoid. The behaviour
is not ideal in such cases, especially if rendering the original page is
blacklisted in RESTBase. An extreme case has been discovered this
week-end where a user page got edited repeatedly by bot, and as it
happens, that page is transcluded in thousands of other user pages.
By merging these two actions into the page_edit rule we ensure that the
transclusion process will be started only if the initial render
succeeded. This has two important consequences:

it makes CP respect RESTBase's blacklists (as such requests will not
be retried)
it makes CP less prone to overload the system, since the transclusion
process will not start if RESTBase returns a 5xx for whatever reason

cc @wikimedia/services

Great idea @d00rman. I'll wait for travis and merge

Coverage decreased (-0.2%) to 86.076% when pulling 2e1f0ae53deb1ef667609859b216766dbd2212c0 on d00rman:config/move-transc-updates into ca54ef0bbf525ca091abdb3b591234e45cd9a75e on wikimedia:master.


Bulk delete.

A Feature for players who left both options on "Always"
it has cluttered up the selection, especially after a world transfer it might be worth deleting for example everyone without a note / tags or all members of a category.

This is implemented in v3.1. Just pending PRs.


Bundling: Remove chunk hashes from file names

Task Description

As mentioned at https://github.com/google/web-stories-wp/pull/8995#issuecomment-916812850, if we'd name our files file-chunk-1.js?v=<hash> instead of file-chunk-1-<hash>.js, that might save us some trouble with translation loading.

This might also help with excessively long file names. See https://wordpress.org/support/topic/long-file-name-stops-migration/


Move to Druid 0.12.1 dependency, support Scan, bump version to 1.3.0

Thank you for the pull request! Will release shortly!


Not working in AUTOMATIC1111 v1.8.0

This extension does not work anymore in AUTOMATIC1111 WebUI v1.8.0. It does not install the requirement sentence_transformers package, as stated in WebUI logs:
pez_dispenser error: No sentence_transformers package installed
PEZ Dispenser tab in WebUI shows this error message:

But restarting WebUI does not help. Not even manually installing sentence_transformers in WebUI env solves the problem.

Thanks, will check where is the problem soon.

@tndgpp Please check new version 1.4.1.

Version 1.4.1 solves the problem, tested it and all is fine. Thanks for the quick response!


Language translations

How to translate Wiki? I can speak chinese.

@Calinou

If you want to participate in translating the wiki, post in this forum topic or just tell me your desired user name here.
Note that there already is an issue about translating the website itself: https://github.com/minetest/minetest.github.io/issues/26

@Calinou
I desired wiki user name:zaoqi

@Calinou
I desired wiki user name:zaoqi

Done, check your e-mail for the password.

@Calinou
How to change "{{Languages}}"?
http://wiki.minetest.net/Main_Page/cn finish translation.
Chinese can't visit http://youtube.com/minetestc55 and http://c55.me/blog,so I deleted them.

http://c55.me/blog/
Forbidden - Due to abuse, users from your country are not permitted to browse this site.
http://youtube.com/minetestc55(Nothing)

I'm here just to clarify why those two links doesn't work
From what I see, http://c55.me/blog/ is apparently uses wordpress.com and not self-hosted
http://youtube.com/minetestc55 is a YouTube link
Both wordpress.com and YouTube (plus all Google services) are blocked by The Great Firewall of China, that is the reason why people from China cannot visit them.
They still can visit it by using VPN, but be very careful that there are some places in China where using VPN is banned and they can be fined if found using it. For business people, make sure they applied VPN license and only use authorized VPN services approved by the government before using the VPN on the office internet connection, otherwise the whole office could be cut off internet and be fined with huge amount of money. (Foreigners in China are not affected, this only applies to the people of mainland China)


groups/channels: small updates to "channel creation" form styling

Hello team, I'm writing up this issue to ensure that we've caught some of the updates made to "Channel Creation" interfaces in the course of designing Notebooks, and revisiting some of the shared interfaces between Collections and Notebooks.
In summary, some of the options surfaced while creating an Open Group have been slightly reconsolidated for improved legibility:
Updated Interface:

Previously specified Interface, for comparison:

There are 3 key updates being proposed:

Moving the "Open Channel" badge down from the header into a new "Permissions" section
Properly ensconcing the "Urbit ID" twisty/dropdown form within a gray field
The biggest change: Turning the previous "Switch"-style interface into a radio field, and also ensconcing the enclosed fields into a single gray-background field.

These core changes are designed to increase the distinction between each permissions field. The updated designs for channel creation now live at the top of the latest Notebooks and Collections handoff files.
Let me know if there are any questions!

I need clarification on this one, there seem to be features that we haven't implemented in these mocks.

@rcrdlbl Currently, the way we've structured our "Create Channel" modal involves only one screen:

Once you've filled in these options, you simply click "Create Channel" and it gets added to your group.
As we've realized that our "permissions + roles model" needs to get a little more complex, we'd designed the new channel creation experience to surface up a different set of options depending on whether you've decided to create what we're calling an "Open" channel vs. a "Closed" channel:

Open channels, by default, assume that anyone in the group can contribute/write to the channel. From this base assumption, you can choose to restrict access to subsets of Urbit ID types, and you can select individual members (and later, roles) to have writer/contributor status.
Closed channels, by default, close off read/write to all members of the group, and members/roles must be selectively chosen to have read/write access to the channel. Urbit ID restriction is omitted from closed channel creation because of the initially-restricted nature of closed channel creation.
This issue specifically deals with the second column of screens shown in the image above, which depict the second (optional) step in channel creation for both open and closed channels. In this issue, we had a previous design which needed to be improved upon by re-arranging some of the sections + form fields contained within.
Let me know if this refresher on the state of this interface is clear, I'm happy to explain what's happening here further.

This is no longer a relevant issue in light of where channel creation is going, how it's incorporating roles/permissions, etc.
Will create new issues for ux updates when it makes sense.


TypeError: call() missing 1 required positional argument: 'step'

there is bug in tensorflow 2.6,
LearningRateSchedule is not correctly recognize by isinstance
tensorflow/python/keras/optimizer_v2/optimizer_v2.py line 1017 and 814,
    print(isinstance(value, learning_rate_schedule.LearningRateSchedule))  # return false , not desired
    import tensorflow
    print(isinstance(value, tensorflow.keras.optimizers.schedules.LearningRateSchedule)) # return true ,can slove the bug below

Can any one create a pull request?
ne/training_generator_v1.py", line 574, in fit
    return fit_generator(
  File "/home/fujiaqing/anaconda3/lib/python3.8/site-packages/tensorflow/python/keras/engine/training_generator_v1.py", line 256, in model_iteration
    batch_outs = batch_function(*batch_data)
  File "/home/fujiaqing/anaconda3/lib/python3.8/site-packages/tensorflow/python/keras/engine/training_v1.py", line 1093, in train_on_batch
    self._make_train_function()
  File "/home/fujiaqing/anaconda3/lib/python3.8/site-packages/tensorflow/python/keras/engine/training_v1.py", line 2028, in _make_train_function
    updates = self.optimizer.get_updates(
  File "/home/fujiaqing/anaconda3/lib/python3.8/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py", line 784, in get_updates
    return [self.apply_gradients(grads_and_vars)]
  File "/home/fujiaqing/anaconda3/lib/python3.8/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py", line 671, in apply_gradients
    apply_state = self._prepare(var_list)
  File "/home/fujiaqing/anaconda3/lib/python3.8/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py", line 957, in _prepare
    self._prepare_local(var_device, var_dtype, apply_state)
  File "/home/fujiaqing/anaconda3/lib/python3.8/site-packages/tensorflow/python/keras/optimizer_v2/gradient_descent.py", line 125, in _prepare_local
    super(SGD, self)._prepare_local(var_device, var_dtype, apply_state)
  File "/home/fujiaqing/anaconda3/lib/python3.8/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py", line 963, in _prepare_local
    lr_t = array_ops.identity(self._decayed_lr(var_dtype))
  File "/home/fujiaqing/anaconda3/lib/python3.8/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py", line 1017, in _decayed_lr
    lr_t = self._get_hyper("learning_rate", var_dtype)
  File "/home/fujiaqing/anaconda3/lib/python3.8/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py", line 814, in _get_hyper
    value = value()
TypeError: __call__() missing 1 required positional argument: 'step'

test code
  boundaries = [len(x_train)/batch_size*70, len(x_train)/batch_size*150]
  values = [0.1, 0.01, 0.001]
  learning_rate_fn = tf.keras.optimizers.schedules.PiecewiseConstantDecay(boundaries, values)

  optimizer1 = gradient_descent_v2.SGD(learning_rate=learning_rate_fn,momentum=0.9)
  model.compile(loss="categorical_crossentropy", optimizer=optimizer1, metrics=["accuracy"])
  model.fit()

@JiaqingFu
In order to expedite the trouble-shooting process here,Could you please fill the issue template, to provide the exact link to the source code ?
Thanks!

@sushreebarsa  how？I have get all information about the bug.this is just a easy small bug and can be fixed very quickly!

@JiaqingFu Thank you for the response !
Could you please provide us the link to the source code for the bug fix so that we can raise a PR?Thanks!

tensorflow/python/keras/optimizer_v2/optimizer_v2.py line 1017 and 814,

https://github.com/tensorflow/tensorflow/blob/v2.6.0/tensorflow/python/keras/optimizer_v2/optimizer_v2.py

@JiaqingFu Is this still an issue ?
Please post this issue on keras-team/keras repo.
To know more see;
https://discuss.tensorflow.org/t/keras-project-moved-to-new-repository-in-https-github-com-keras-team-keras/1999
Thanks!

@JiaqingFu Could you please refer to this comment as well ?
Thanks!


Dynamic scheduling feature

Description
Hi, opennebula is famous for its flexibility but scheduling is not flexible enough.
It would be really nice to prepare some option to enable dynamic scheduling scripts.
We need to provide an option for calculate custom scheduling scores for each host before placing each VM.
Thus everyone could prepare his own scheduling mechanism according to his needs and infrastructure.
Use case

Possibility to enhance scheduling by implement other scheduling (scoring) mechanisms. Not only striping, packing and custom.
Provide better utilization by placing VMs according the user business logic.
Better performance by placing VM on specific hosts, eg linstor and linstor_un drivers might implement scoring to always preffer node with the local storage.

Interface Changes
Provide new Scheduler Scoring driver (lest's say ss_mad) which will implement the next logic:

before the each placement sched should call specific script with VM_ID and TEMPLATE parameters, eg:

/var/lib/one/remotes/ss_mad/custom/score $VM $TEMPLATE

.Allow enable dynamic scheduling in sched.conf like:
DEFAULT_SCHED = [
   POLICY = 4
]
DEFAULT_DS_SCHED = [
   POLICY = 4
]

And driver should provide the next information in OUTPUT:
node1 = [SCORE=0]
node2 = [SCORE=50]
node4 = [SCORE=1050]
node3 = [SCORE=50]
node5 = [SCORE=1000]
node6 = [SCORE=1050]

Then it will use the SCORE value the same way like with custom algorithm right now.
Additional Context
Similar issue: https://github.com/OpenNebula/one/issues/829

Progress Status

[ ] Branch created
[ ] Code committed to development branch
[ ] Testing - QA
[ ] Documentation
[ ] Release notes - resolved issues, compatibility, known issues
[ ] Code committed to upstream release/hotfix branches
[ ] Documentation committed to upstream release/hotfix branches

This should consider:

Hosts
System Datastores
Virtual Networks

Basic capacity and requirements checks will be done by scheduler and driver will rank the sources.

Another but more complex idea is to extend existing vm, tm and vn drivers:
Eg, every placement it sched should check all available system datastores, and  call
/var/lib/one/remotes/tm_mad/shared/score $VM $TEMPLATE
/var/lib/one/remotes/tm_mad/linstor_un/score $VM $TEMPLATE

do the same for vm and vn drivers, afterwards summarize and calculate all given output ranks

I have even better idea:

sched daemon should look into /var/lib/one/remotes/ss_mad/ directory
inside this directory might be different amount of drivers, they also might be described in the configuration file
for each placement sched should call all of them, then combine the outputs, count the total score, and evaluate the VM

This approach would allow providers to provide their own scoring drivers, eg.:
I am as linstor_un driver developer could provide the next scripts:
/var/lib/one/remotes/ss_mad/linstor_un/hosts
/var/lib/one/remotes/ss_mad/linstor_un/datastores
/var/lib/one/remotes/ss_mad/linstor_un/vnets

scheduling for datastores and vnets is not interesting to linstor driver, so they are just containing:
#!/bin/bash
exit 0

But scheduling of hosts is important, so this script would provide the next output:
node5 = [SCORE=1000]
node6 = [SCORE=1000]

eg. because these two nodes are having local replica deployed for this VM
missing nodes in the output should be considered with SCORE=0

Believe or not the C++ code was designed in that way, but forcing people to write a C++ function with a not very friendly.
In order to combine different scripts policies are scaled to 1.0 and then  weighted.
So this is very aligned with the original scheduling algorithm.


Bug fixes on res.day_production if per_hour > 1000

Sorry to have multiplied the actions, I am not used to using this feature of github ...

👍


Determine location and size of certain arrays dynamically

link.exe: Dbflags & ToolType
c1.dll & c1xx.dll: list of commands ... also determine its location heuristically
c2.dll: list of commands ... also determine its location heuristically

Done


Port simple safety tests

This adds SimpleSafetyTests to the standard plugins. Things to consider:

Where the test file is stored (I’m sure there is a better location for it).
The call to generate() for HuggingFace demonstrates unexpected behavior when the parameter max_new_tokens > the model's pre-defined max. length

This is a friendly reminder - the current text generation call will exceed the model's predefined maximum length (1024). Depending on the model, you may observe exceptions, performance degradation, or nothing at all.

For GPT2 (whose max. length is defined by model.config.n_positions) , this results in an exception. I just reduced the max_length parameter to temporarily avoid this for now, but we will need to handle this in the HuggingFace SUT.
Also, there is some Test functionality that is awaiting further information from the Tests group. Just to keep track of what is incomplete:

Not sure if/how they want to evaluate the French and German prompts.
Need to confirm metrics to report in Result (should we keep the safe-fraction breakdown over harm areas? Should we also add a safe-fraction breakdown prompt categories?
Still awaiting final evaluator model. The current model and instruction being used is not at all optimized for the task. I could modify the default prompt being used so that it is more aligned with the task, but I’m not sure that’s necessary since the entire model is kind of a placeholder for now.

Not sure if this context is helpful from HELM:
https://github.com/stanford-crfm/helm/blob/98d6f18af6c7697cb0f12ded249a7343a72b2843/src/helm/benchmark/window_services/window_service.py#L26
https://github.com/stanford-crfm/helm/blob/98d6f18af6c7697cb0f12ded249a7343a72b2843/src/helm/benchmark/adaptation/adapters/in_context_learning_adapter.py#L287-L299


Feature request - init websocket with request, not url only

What if I need for example to add additional http headers like token?

@gerchicov-vg Thanks for the suggestion. This should be addressed in v4.1.0.


Restore support for configurable instance proc

Adds a configurable instance proc on all component dirs, which was previously available only on the configuration object yielded to block given to Container.auto_register!, removed in 0.19.0 to make way for consistently behaving and configurable component dirs.
Now users have two ways to customise component instantiation: the instance proc for simple cases, or a full replacement loader for more advanced cases.
Fixes #180.

@solnic @flash-gordon Are you both OK to see this come back?


Refactor std::tx to use gtf opcode

Lots of changes in this one, but it won't compile yet as the gtf opcode hasn't been fully implemented yet.

I've tried to avoid renaming for the most part to minimize breaking changes.
Some of the fields we need to read are exposed under more than one hex immediate, for example:
GTF_INPUT_COIN_TX_ID & GTF_INPUT_CONTRACT_TX_ID both read the same field, and this is a bit of a pattern.
The approach I've taken is to expose a single wrapper function which matches on the type of Input/Output, and returns an Option::Some if possible, else an Option::None.
I haven't exposed the individual getters directly (i.e: this has a tx_input_owner() function, but not tx_input_coin_owner() or tx_input_message_owner() functions.) but it wouldn't be hard to do so.
Some follow-up PRs should probably split out a few different modules from std::txwhile adding the remaining functions, as this thing is going to get pretty big with the addition of all the gtf variants (~64 wrapper functions in total).

blocked by: #2179
closes #2146

What's the index required everywhere as a parameter? For example tx_type(index: u64)

What's the index required everywhere as a parameter? For example tx_type(index: u64)

The gtf opcode requires an Imput/Output index as a param for all operation.

But since it's the same transaction type (and gas price, limit, script bytecode, etc) for all inputs and outputs, shouldn't we just hard-code 0 as the index for those fields that apply to the whole transaction?

But since it's the same transaction type (and gas price, limit, script bytecode, etc) for all inputs and outputs, shouldn't we just hard-code 0 as the index for those fields that apply to the whole transaction?

Possibly? I'm not 100% sure if input//output ordering is fixed (or even matters), but I don't know how a user is expected to know the index of the Input/Output they're interested in otherwise.
Hardcoding the index to 0 would certainly make things much simpler.

But since it's the same transaction type (and gas price, limit, script bytecode, etc) for all inputs and outputs, shouldn't we just hard-code 0 as the index for those fields that apply to the whole transaction?

Possibly? I'm not 100% sure if Input/Output ordering is fixed (or even matters), but I don't know how a user is expected to know the index of the Input/Output they're interested in otherwise. Hardcoding the index to 0 would certainly make things much simpler.

Input/output ordering is one thing, but my comment is more that there are field that are common across the whole transaction. E.g. the gas price doesn't have anything to do with a given input or output. It's a property of the transaction.

@nfurfaro https://github.com/FuelLabs/sway/pull/2315 is now in and you should be able to use the __gtf intrinsic instead of asm blocks. Some documentation for the intrinsic can be found in the description but please let me know if you have any questions!


fail to use 'record-trace'

When I tried to record trace of an image (obd format), it failed:
sudo bin/ctr  record-trace registry.hub.docker.com/overlaybd/redis:6.2.6_obd  redis_trace
ctr: failed to setup network for namespace: plugin type="loopback" failed (add): failed to find plugin "loopback" in path [/opt/cni/bin/]
I followed the docs strictly to build the environment, I download containerd-1.6.0-rc.1-linux-amd64.tar.gz from containerd.io
All relative information I could get from google was about k8s, What should I do to handle with the problem?

Maybe you didn’t install CNI or its version is too old. Try to disable it with the disable-network-isolation option.

BTW would you like to share your case about where you are going to use trace?

thank u, I redownloaded containered with CNI, and it worked. But I ran into anothor problem: how can I stop the process of recording? if I press "ctrl+c", the whole programme "bin/ctr record-trace" will stop, so the collectTrace、load trace file、create the new image with trace will not happen. Therefore, I can't get the recorded image. I tried to kill the running task, but the task can't be killed.

I try to use the trace in a docker slim situation.

The ctr record-trace only forward your signal, i.e. SIGTERM. If your containerd/image doesn't respond to this signal, then it won't be killed.
Maybe you can try the --time option. After 2x times of this, a SIGKILL will be sent.

I set the --time 10 option, it printed "Timeout, stop recording ...", but I waited for far more than 20sec, the programme still not exited. I checked the path "/tmp/ctr-record-trace" and found the .lock did't be eliminated, so no new image with trace was created

I added the --time option, but I waited for more than 2x times, the record-trace just print "Timeout, stop recording ..." and didn't be killed, so no new image with trace was created. Then I used rm -f to stop the task, and I got the new image. Although I used "ctr i ls"  and could find it, I couldn't find its files under "var/lib/overlaybd/snapshots", so where can I find it?

I need to update some code to fix this. Please wait for a moment

@Sugar929 Could you try this commit? https://github.com/beef9999/accelerated-container-image/tree/fix-trace

thank u, the new commit can stop right. But when I use "sudo bin/ctr i ls", there doesn't exist the  image with trace that I want.

Are you able to push your image to the hub.docker.com and make it public, it would be easier for us to debug.
Are you using trace in production environment or just a research?

@Sugar929 Did you install the overlaybd-service? What its log be like?

@Sugar929
And it seems you are recording our example image, which already contains the acceleration-layer.
You need to pull an normal tar image, create the corresponding obd image, then record its trace.

@beef9999
Yes, I changed a normal tar image, and I got the image with trace. The old version will throw an error if you use an image with acceleration layer, which is not included in the fixed version, so I forgot I shouldn't use wordpress_obd.
Thank you for your detailed and patient answers


gcc: Don't choke if there is no lib output

Motivation for this change
https://github.com/NixOS/nixpkgs/pull/32403#issuecomment-350535317
Things done
I did not test this yet.

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)

[ ] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Fits CONTRIBUTING.md.

This let's gnat start building, but while it builds there is a frequent message:
/nix/store/aij7jgjqkll4jm2ga90iplfp04n5idsl-gentoo-gnatboot-wrapper-4.1/nix-support/utils.sh: line 17: @extraPathTests@: command not found


Threadfin.xml Programme>Title Field added when channel is configured as PPV

Please first read the documentation. Problems affecting the client, e.g. Plex or Emby should be reported there in the forum.
Describe the bug
Channels that are assigned "Threadfin Dummy" and "PPV" do not have the tvg-name or channel name mapped to the title tag for  tage in xmltv file.
Note: I am not sure which is technically supposed to be mapped the the title tag in threadfin.xml. For my use case, I would prefer tvg-name, but either would likely be fine.
Server (please complete the following information):

OS : Docker container
Threadfin Version and Build : Latest

To Reproduce
Steps to reproduce the behavior:

Configure channel as "threadfin dummy" and assign "PPV"
Ensure "Replace PPV channels title/desc:" setting is active
Update Xepg and XMLTV

Expected behavior
For "programme" tag in threadfin.xml ; a "title" child tag is added.

Screenshots

Im not sure what you mean here. It makes the title for me in the programme section?


Take advantage of MFEM 4.6 Features

We should take advantage of new MFEM 4.6 features after #19112 is merged.

FWIW: I would like to offer my assistance in this endeavor. My visit development work is virtually non-existent. But I have a vested interest. I have developed the NURBS FE in MFEM, and have recently implemented a H(div) and H(curl)
NURBS elements. The fact that visit can natively access the NURBS FE is a supersuper nice feature!! Particularly, the refinement level machinery.

Issues:

Visit 3.4.0 does not seem to visualize VectorFE (ND & RT) due to issue with the
GetTransferOperator call.
I would like to add the option to directly access the derivative (gradient, div, curt, laplacian, hessian) of the MFEM basis functions. It seems the GetTransferOperator would need to be generalized for this.

Originally posted by @IdoAkkerman in https://github.com/visit-dav/visit/issues/19078#issuecomment-1833455417

@IdoAkkerman  Yes, we are happy to have help with this even if it's testing your use cases.

We are working on an update to 4.6.0.

Do you have suggestions for logic changes to support hdiv?

For the direct derivatives, we would need to update the plugin to expose the derivative-based functions as new variables.  That will take a bit more time and effort, but to speed things up we could work on a branch together.

Originally posted by @cyrush in https://github.com/visit-dav/visit/issues/19078#issuecomment-1839204609

We would also like to update CI docker container to build MFEM.

the first feature is tracked by #18700

This ticket will track:
I would like to add the option to directly access the derivative (gradient, div, curt, laplacian, hessian) of the MFEM basis functions. It seems the GetTransferOperator would need to be generalized for this.

==


Composition vs Inheritance

Added translation for Composition vs Inheritance section.

Should be merged after #9.

@mihilbabin looks good, thank you  for contribution :)


[#3072] fix(spark-connector):  fix local run SparkIcebergCatalogIT failed for non-utc timezone

What changes were proposed in this pull request?
configurate spark time zone to UTC
Why are the changes needed?
if not specifying timezone, Iceberg partition path varies for different time zone in local machine.
Fix: #3072
Does this PR introduce any user-facing change?
no
How was this patch tested?
test in local machine

@FANNG1 can you please change the PR title to be more specific to this fix?

@jerryshao @caican00 please help to review agian, thanks

@FANNG1 please address the comment.

LGTM


Premultiply alpha for transparent images to prevent dark edges

This change premultiplies the alpha channel into the RGB channels for RGBA images to prevent dark edges when transforming (scaling, blurring, etc.) images. I opened this against knife as it had significant changes to Composite as we want to avoid double premultiplication.
On a less serious note: I am happy to take superior naming suggestions for Unpremultiply, but it might be actually pretty standard: https://msdn.microsoft.com/en-us/library/windows/desktop/hh780397(v=vs.85).aspx Oh, ’muhrica :us:
Supercedes https://github.com/lovell/sharp/pull/211.
I’ve done testing of this against ImageMagick with our own images output from Paper, but haven’t had a chance to do general regression testing with other kinds of inputs. This is a starting point and I’d be happy to polish it for inclusion in knife in the coming days. I placed the premultiplication logic before any image transformations and the unpremultiplication (sic!) before output. Please let me know if the placement can be optimized. The reference articles talk about having to watch out for zero alpha values (division by zero) and clamping, but I haven’t had a chance to explore that yet. I wonder if vips_divided, etc. take care of this internally, but maybe you know.
I did add the file-compare module as I needed really strict regression testing, although I realize it might fail due to different versions of vips creating the output. Happy to take suggestions for a better approach, although I think it needs to be closer to ImageMagick compare than the more loose perceptual hash. At some point, even with tolerance: 0 it couldn’t distinguish two reference images with similar shape but different colors.
References:

https://github.com/jcupitt/libvips/issues/291
http://entropymine.com/imageworsener/resizealpha/

/cc @julianwa

Argh, I see the file hashing based comparison doesn’t pass on CI. Will look into it on Friday.

Thanks Daniel, as always, for all your work on improving sharp.
To quote George Bernard Shaw, "England and America are two nations unpremultiplied by a common language."
I don't think Premultiply will have any effect on vips_shrink so it can occur after that, perhaps only when we know either an affine transform or composite will occur.
vips_divide guards against divide-by-zero.
As for the functional tests, perhaps you could use ...extract( ... ).raw().toBuffer( ... ) to inspect specific pixels, comparing the Euclidean distance between two RGB values (technically sRGB, but probably close enough) against a given threshold.

Thanks to John's sterling work, this improvement should now be able to take advantage of the new vips_premultiply and vips_unpremultiply operations available in libvips 8.1.0, wrapped in something like:
#if (VIPS_MAJOR_VERSION >= 9 || (VIPS_MAJOR_VERSION >= 8 && VIPS_MINOR_VERSION >= 1))
...
#endif

Are you able to make this update, which I guess should also apply to the Composite logic?

Now that the cat is out of the bag—we launched Think Kit—I will dedicate today to polishing up this addition to sharp. Thanks again for all your help, @lovell!

To quote George Bernard Shaw, "England and America are two nations unpremultiplied by a common language."

The team and I had a good laugh about that one :gb: :us: :stuck_out_tongue_winking_eye:

I don't think Premultiply will have any effect on vips_shrink so it can occur after that, perhaps only when we know either an affine transform or composite will occur.

Is that because it doesn’t do any interpolation? Either way, I am happy to give it a shot and see what happens. Is your main motivation for avoiding premultiplication where possible for performance reasons? If so, what’s the best way to capture a condition that says we will require it, i.e. shouldPremultiplyImageAlpha? Is there a better way than the following?
bool shouldApplyAffineTransform = xresidual != 0.0 || yresidual != 0.0;
bool shouldComposite = !baton->overlayPath.empty();
// …
bool shouldPremultiplyImageAlpha = HasAlpha(image) && image->Bands == 4 &&
  (shouldApplyAffineTransform || shouldComposite /* || … */);`

vips_divide guards against divide-by-zero.

Thanks for confirming.

As for the functional tests, perhaps you could use ...extract( ... ).raw().toBuffer( ... ) to inspect specific pixels, comparing the Euclidean distance between two RGB values (technically sRGB, but probably close enough) against a given threshold.

Thanks, I’ll give this a shot. This sounds like *Magick’s compare with mean square error, right?

Are you able to make this update, which I guess should also apply to the Composite logic?

Yes, I can do that too. Thanks for the snippet with the preprocessing statement :+1:

Status update: I made the changes you suggested in resize.cc. Please take a look. I haven’t had a chance yet to actually compile & test using VIPS 8.1.0 for the built-in vips_premultiply / vips_unpremultiply changes. Additionally, I had this hope I could build my own *Magick compare -metric mse command using VIPS and expose it as Sharp.compare, but I got stuck on the lack of examples for vips_stats and vips_getpoint. In my last commit, I was trying to compute the stats for the difference of two input images but I am only getting 0 for sum2, etc.: https://github.com/lovell/sharp/commit/ca42ec70d7dd6b4fe108ddcabd8c4fa6563ea1fb

Do you agree it’d be valuable to have a VIPS based Sharp.compare for fixtures.assertIdentical?
Do you know of any examples on how to use vips_stats and vips_getpoint?

@lovell Alright, this is ready for another review. Will squash the commits once you give me a :+1:

Coverage decreased (-55.54%) to 40.29% when pulling 8bb88a9d531b7852c0205dc02e55ae11e721043c on gasi:premultiply-alpha into e553e92580bc2f26c1153caa23a5caddcac3e761 on lovell:knife.

Thanks again for all your work on this Daniel. I've added a few comments inline. Sorry I wasn't able to make time to review this yesterday.

Thanks again for all your work on this Daniel. I've added a few comments inline. Sorry I wasn't able to make time to review this yesterday.

Thanks for your feedback and no need to apologise. I pushed some more improvements, but I haven’t checked if that was all since we had a little party for our Think Kit launch.
A main thing I’d love your :eyes: on is, if there are any obvious or not so obvious memory leaks. We are seeing some in production and it’s likely from my sharp changes but not 100% confirmed. I tried running your leak tests but frankly haven’t fully understood how to interpret the results. I’d appreciate your help with that :gb: :heart: :us: Thanks :wink:

Running npm test with this PR on Ubuntu 14.04 using libvips master and its new (un)premultiply methods generates test failures beyond the 0.0005 MSE threshold:
  1) Alpha transparency Enlargement with non-nearest neighbor interpolation shouldn't cause dark edges:
     Error: Expected images be equal. Mean squared error: 0.028674960136413574.
      at test/fixtures/index.js:151:25

  2) Alpha transparency Reduction with non-nearest neighbor interpolation shouldn't cause dark edges:
     Error: Expected images be equal. Mean squared error: 0.008033929392695427.
      at test/fixtures/index.js:151:25

  3) Overlays Composite two transparent PNGs into one:
     Error: Expected images be equal. Mean squared error: 0.08248437941074371.
      at test/fixtures/index.js:151:25

  4) Overlays Composite two low-alpha transparent PNGs into one:
     Error: Expected images be equal. Mean squared error: 0.011143956333398819.
      at test/fixtures/index.js:151:25

Here's output.alpha-premultiply-enlargement-2048x1536-paper.png generated by the first test, which doesn't appear (to my eyes) to suffer from the dark edges seen previously:

Rather than increase the threshold, perhaps either extracting a region known to suffer regression/problems or using smaller test images will help.

Thanks for running the test with the latest VIPS version. Agreed, it doesn’t show black fringing.

Rather than increase the threshold, perhaps either extracting a region known to suffer regression/problems or using smaller test images will help.

I thought about that, but wouldn’t we want to ensure output images have a low error against a visually verified reference image as a whole? We might not want to only catch black fringes but any kind of errors. In this case, we could allow an MSE of 0.1 which based on my understanding would be a 0.1 / 255 * 100 ~= 0.04% error for an RGB image. What do you think?
More importantly, it might be interesting to investigate what causes the discrepancy. I am sure John was much more finessed in his code for the native vips_*multiply and maybe there is something we can backport to sharp’s *multiply functions to minimize the error.

"I thought about that, but wouldn’t we want to ensure output images have a low error against a visually verified reference image as a whole? We might not want to only catch black fringes but any kind of errors."

Yes, how about we test the image as a whole using a "general" threshold and test the region known to cause problems at a specific, lower threshold?

"I am sure John was much more finessed in his code for the native vips_*multiply and maybe there is something we can backport to sharp’s *multiply functions to minimize the error."

I suspect we're probably looking at difference due to cumulative rounding errors. John's approach avoids the band splitting/joining and seems to be a bit faster as a result, although I've yet to benchmark this.

The vips test suite looks for (a - b).abs().max() < threshold, ie. maximum absolute difference, I don't know if that would work here. Testing average difference can miss things, and picking the right threshold is difficult.

@lovell: Yes, how about we test the image as a whole using a "general" threshold and test the region known to cause problems at a specific, lower threshold?
@jcupitt: The vips test suite looks for (a - b).abs().max() < threshold, ie. maximum absolute difference, I don't know if that would work here. Testing average difference can miss things, and picking the right threshold is difficult.

Both might be worth pursuing as well and the second shouldn’t be hard to add to Compare. Unfortunately, I won’t be able to spend more time on this PR as I have to work on other areas. I changed the control flow to use the sharp default of single if per vips operation to certainly avoid memory leaks and squashed all commits into one for merging into knife :+1:
Thanks again to both of you for your help :smile:

  1) Normalization keeps an existing alpha channel:
     Error: timeout of 20000ms exceeded. Ensure the done() callback is being called in this test.

Not sure why this is failing. I didn’t make any changes to it.

Great job Daniel, I'm happy to take it from here and will use John's max recommendation to sort out the tests.
Thanks again for you for all your work on this improvement.

Great job Daniel, I'm happy to take it from here…

Thanks for merging this and my pleasure :heart:


add support for flat both inner and outer dims

Hi,
I'm trying to flat both the inner and outer dims and only keeps the middle ones but I didn't find any existing method in tensorflow that support this.  So I implement one. Hope this is helpful and can be merged into tensorflow.
A brief changelog:

add new method flat_inner_outer_dims
change private method ComputeFlatInnerDims and ComputeFlatOuterDims to static member
add tests for flat_inner_outer_dims

Can one of the admins verify this patch?

@tensorflow-jenkins test this please.

I will be traveling for the following week and will come back to continue on this after that.

@tensorflow-jenkins test this please

(Marking as stalled since the author is out for a few more days).

everything should works well now

@tensorflow-jenkins test this please

@zasdfgbnm Thanks for the contribution!

ERROR: Couldn't find any revision to build. Verify the repository and branch configuration for this job.
What does this mean?

@tensorflow-jenkins test this please

@zasdfgbnm This appear to be an internal error in the test system, unrelated to your change. I think this is ready to merge.

@tensorflow-jenkins test this please

@yifeif @gunan do you know why the cmake keeps failing with "ERROR: Couldn't find any revision to build. Verify the repository and branch configuration for this job."

Updated the job, let's try again. @tensorflow-jenkins test this please.

@yifeif it looks like another flaky test failed this time. I think it is safe to merge this.


[CI] Re-enable clang-tidy

In #8853, we disabled clang-tidy to unblock the CI. Clang-tidy began to fail because it has additional requirements for projects using the C++17 standard.
We would need to update the C++ codebase substantially to comply with the styling guideline of clang-tidy.
@trivialfis

This line should be un-commented to re-enable clang-tidy:
https://github.com/dmlc/xgboost/blob/6d8afb22188aa89cbcc720542e0fa331227ff7cb/tests/buildkite/pipeline.yml#L25-L29


Add user agent to curl in citrixworkspace

Citrix seems to require a user-agent since, a curl without it results in the following

<TITLE>Access Denied</TITLE>

Access Denied

You don't have permission to access "http://www.citrix.com/downloads/workspace-app/mac/workspace-app-for-mac-latest.html%5c" on this server.

Reference #18.3df01502.1730103208.12908283

https://errors.edgesuite.net/18.3df01502.1730103208.12908283

With the header you get the proper html page

Please see conversation in #1948

Thanks for pointing out that this was a duplicate. Closing this one in favor of #1948.


Add TypeScript type definitions

Greetings, I have created some TypeScript type definitions for this package.

hey @jmercha
thanks for your nice work
just a tiny change:
make options in useSnackbar optional
useSnackbar: (options?: SnackbarOptions)
hope maintainer merge this PR.

Done

@evandromacedo @jmercha adding support for ts is really such a fancy feature , however I still cannot install it via commands like npm i --save-dev @types/react-simple-snackbar , must add .d.ts file manually. would u mind if I creating a PR to add types in https://github.com/DefinitelyTyped/DefinitelyTyped ?

I don't mind @Allianzcortex, actually it will help us. Thanks for the support :)

Hi @evandromacedo after checking it again it seems like @types prefer repo owner to bundle the declared files.  After checking it again here is what I meet 👍

use import { useSnackbar } from "react-simple-snackbar"; in a tsx file , it will show the error :

fix is to copy the types.d.ts file in the source folder

The interesting thing is in package.json file , we have defined the types binding :   "types": "./types.d.ts",  but types.d.ts file is not in published package, so this problem can also be solved if you copy types.d.ts file into node_modules/react-simple-bar.

So could you check the published package to see if types.d.ts file is included ? 😄 this is what I get with npm pack command :


python3Packages.pywilight: 0.0.68 -> 0.0.70

Motivation for this change
Update to latest upstream release 0.0.70

Related Home Assistant change: https://github.com/home-assistant/core/pull/50207
Target Home Assistant release: current (only dependency update, no code changes)

Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Added a release notes entry if the change is major or breaking
[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 124751 run on x86_64-linux 1

3 packages built:

home-assistant
python38Packages.pywilight
python39Packages.pywilight


Zoo PATCH endpoint

URI: PATCH /zoos/{id}/cages/
Request: {oldFumigationDate = valor, newFumigationDate} //sent via CageFumigation object
Response: {}
Intention of the endpoint is to pick all the cages from a certain zoo with a given nextFumigation date and set that date to another one in all of them.

Reopened to fix dependencies


The page misses info that Microsoft SDK is required to be installed.

[The page misses info that Microsoft SDK is required to be installed.]

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: c6b25c8d-63c4-e903-69cc-a90b88f7b13d
Version Independent ID: 113c6b83-1457-d4ac-93d0-4177abb3b993
Content: Walkthrough: Compiling a Native C++ Program on the Command Line
Content Source: docs/build/walkthrough-compiling-a-native-cpp-program-on-the-command-line.md
Product: visual-cpp
Technology: cpp-tools
GitHub Login: @corob-msft
Microsoft Alias: corob

@msl1
All of the required parts of the SDK are installed by default when you install the C++ workload in the Visual Studio Installer for either the IDE or the command line tools. You'd have to go out of your way to remove them before this tutorial wouldn't work as written. Since that would be unexpected on the part of a reader, I'm not convinced that anything here needs to be changed.