Manager failing to find runner v0.11.2

Hello again :)
Thanks for fixing 0.11.0 so quickly! I'm getting 1 more error now.
tag v0.11.2
Scenario Deployed the stack from scratch, I removed Helm from the equation in case that was somehow causing issues. My deployment also includes a service account and some config for using role based auth in EKS on AWS, this shouldn't be related, including for prosperity's sake
Steps to Reproduce

Update manifest to use the namespace sre-actions-runner-system (I deploy to a custom namespace, including for prosperity's sake)
kubectl apply -k ./default -n sre-actions-runner-system
kubectl apply -f asg-org-runner.yaml -n sre-actions-runner-system (see bottom of issue for exact yaml)

Error
callum.tait@PBXUK-HH-05772 ~/git/work/actions-runner-controller/resources/runners$ kubectl logs controller-manager-777fb76ddb-4x7sf -c manager
2020-11-11T04:53:16.799Z	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": "<IP_ADDRESS>:8080"}
2020-11-11T04:53:16.799Z	INFO	controller-runtime.builder	Registering a mutating webhook	{"GVK": "actions.summerwind.dev/v1alpha1, Kind=Runner", "path": "/mutate-actions-summerwind-dev-v1alpha1-runner"}
2020-11-11T04:53:16.799Z	INFO	controller-runtime.webhook	registering webhook	{"path": "/mutate-actions-summerwind-dev-v1alpha1-runner"}
2020-11-11T04:53:16.799Z	INFO	controller-runtime.builder	Registering a validating webhook	{"GVK": "actions.summerwind.dev/v1alpha1, Kind=Runner", "path": "/validate-actions-summerwind-dev-v1alpha1-runner"}
2020-11-11T04:53:16.799Z	INFO	controller-runtime.webhook	registering webhook	{"path": "/validate-actions-summerwind-dev-v1alpha1-runner"}
2020-11-11T04:53:16.799Z	INFO	controller-runtime.builder	Registering a mutating webhook	{"GVK": "actions.summerwind.dev/v1alpha1, Kind=RunnerDeployment", "path": "/mutate-actions-summerwind-dev-v1alpha1-runnerdeployment"}
2020-11-11T04:53:16.799Z	INFO	controller-runtime.webhook	registering webhook	{"path": "/mutate-actions-summerwind-dev-v1alpha1-runnerdeployment"}
2020-11-11T04:53:16.799Z	INFO	controller-runtime.builder	Registering a validating webhook	{"GVK": "actions.summerwind.dev/v1alpha1, Kind=RunnerDeployment", "path": "/validate-actions-summerwind-dev-v1alpha1-runnerdeployment"}
2020-11-11T04:53:16.799Z	INFO	controller-runtime.webhook	registering webhook	{"path": "/validate-actions-summerwind-dev-v1alpha1-runnerdeployment"}
2020-11-11T04:53:16.799Z	INFO	controller-runtime.builder	Registering a mutating webhook	{"GVK": "actions.summerwind.dev/v1alpha1, Kind=RunnerReplicaSet", "path": "/mutate-actions-summerwind-dev-v1alpha1-runnerreplicaset"}
2020-11-11T04:53:16.799Z	INFO	controller-runtime.webhook	registering webhook	{"path": "/mutate-actions-summerwind-dev-v1alpha1-runnerreplicaset"}
2020-11-11T04:53:16.799Z	INFO	controller-runtime.builder	Registering a validating webhook	{"GVK": "actions.summerwind.dev/v1alpha1, Kind=RunnerReplicaSet", "path": "/validate-actions-summerwind-dev-v1alpha1-runnerreplicaset"}
2020-11-11T04:53:16.799Z	INFO	controller-runtime.webhook	registering webhook	{"path": "/validate-actions-summerwind-dev-v1alpha1-runnerreplicaset"}
2020-11-11T04:53:16.799Z	INFO	setup	starting manager
2020-11-11T04:53:16.800Z	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}
2020-11-11T04:53:16.900Z	INFO	controller-runtime.webhook.webhooks	starting webhook server
2020-11-11T04:53:16.900Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
2020-11-11T04:53:16.900Z	INFO	controller-runtime.certwatcher	Starting certificate watcher
2020-11-11T04:53:16.992Z	INFO	controller-runtime.webhook	serving webhook server	{"host": "", "port": 9443}
2020-11-11T04:53:33.597Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"ConfigMap","namespace":"sre-actions-runner-system","name":"controller-leader-election-helper","uid":"3d7bd549-6d72-42aa-af83-a9c342281c0b","apiVersion":"v1","resourceVersion":"124929696"}, "reason": "LeaderElection", "message": "controller-manager-777fb76ddb-4x7sf_c1357a9b-0449-401a-b5f0-e8833138f70e became leader"}
2020-11-11T04:53:33.598Z	INFO	controller-runtime.controller	Starting EventSource	{"controller": "horizontalrunnerautoscaler", "source": "kind source: /, Kind="}
2020-11-11T04:53:33.598Z	INFO	controller-runtime.controller	Starting EventSource	{"controller": "runnerreplicaset", "source": "kind source: /, Kind="}
2020-11-11T04:53:33.598Z	INFO	controller-runtime.controller	Starting EventSource	{"controller": "runnerreplicaset", "source": "kind source: /, Kind="}
2020-11-11T04:53:33.598Z	INFO	controller-runtime.controller	Starting EventSource	{"controller": "runner", "source": "kind source: /, Kind="}
2020-11-11T04:53:33.598Z	INFO	controller-runtime.controller	Starting EventSource	{"controller": "runnerdeployment", "source": "kind source: /, Kind="}
2020-11-11T04:53:33.699Z	INFO	controller-runtime.controller	Starting EventSource	{"controller": "runner", "source": "kind source: /, Kind="}
2020-11-11T04:53:33.699Z	INFO	controller-runtime.controller	Starting Controller	{"controller": "runnerreplicaset"}
2020-11-11T04:53:33.699Z	INFO	controller-runtime.controller	Starting Controller	{"controller": "horizontalrunnerautoscaler"}
2020-11-11T04:53:33.799Z	INFO	controller-runtime.controller	Starting EventSource	{"controller": "runnerdeployment", "source": "kind source: /, Kind="}
2020-11-11T04:53:33.799Z	INFO	controller-runtime.controller	Starting Controller	{"controller": "runnerdeployment"}
2020-11-11T04:53:33.999Z	INFO	controller-runtime.controller	Starting workers	{"controller": "runnerdeployment", "worker count": 1}
2020-11-11T04:53:33.999Z	INFO	controller-runtime.controller	Starting workers	{"controller": "horizontalrunnerautoscaler", "worker count": 1}
2020-11-11T04:53:33.999Z	INFO	controller-runtime.controller	Starting Controller	{"controller": "runner"}
2020-11-11T04:53:33.999Z	INFO	controller-runtime.controller	Starting workers	{"controller": "runnerreplicaset", "worker count": 1}
2020-11-11T04:53:34.099Z	INFO	controller-runtime.controller	Starting workers	{"controller": "runner", "worker count": 1}
2020-11-11T04:54:03.631Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runnerdeployment", "UID": "c3344dff-da24-471e-b936-ad28bfbcb19a", "kind": "actions.summerwind.dev/v1alpha1, Kind=RunnerDeployment", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runnerdeployments"}}
2020-11-11T04:54:03.692Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runnerdeployment", "UID": "c3344dff-da24-471e-b936-ad28bfbcb19a", "allowed": true, "result": {}, "resultError": "got runtime.Object without object metadata: &Status{ListMeta:ListMeta{SelfLink:,ResourceVersion:,Continue:,RemainingItemCount:nil,},Status:,Message:,Reason:,Details:nil,Code:200,}"}
2020-11-11T04:54:03.792Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runnerdeployment", "UID": "29a36d97-e024-4e0c-8ccb-9b6f6af2ca4f", "kind": "actions.summerwind.dev/v1alpha1, Kind=RunnerDeployment", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runnerdeployments"}}
2020-11-11T04:54:03.792Z	INFO	runnerdeployment-resource	validate resource to be created	{"name": "sre-github-runners-runner-deployment"}
2020-11-11T04:54:03.792Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runnerdeployment", "UID": "29a36d97-e024-4e0c-8ccb-9b6f6af2ca4f", "allowed": true, "result": {}, "resultError": "got runtime.Object without object metadata: &Status{ListMeta:ListMeta{SelfLink:,ResourceVersion:,Continue:,RemainingItemCount:nil,},Status:,Message:,Reason:,Details:nil,Code:200,}"}
2020-11-11T04:54:03.808Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runnerreplicaset", "UID": "41a373d2-9e18-4356-a61a-e5b2a9901eb4", "kind": "actions.summerwind.dev/v1alpha1, Kind=RunnerReplicaSet", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runnerreplicasets"}}
2020-11-11T04:54:03.811Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runnerreplicaset", "UID": "41a373d2-9e18-4356-a61a-e5b2a9901eb4", "allowed": true, "result": {}, "resultError": "got runtime.Object without object metadata: &Status{ListMeta:ListMeta{SelfLink:,ResourceVersion:,Continue:,RemainingItemCount:nil,},Status:,Message:,Reason:,Details:nil,Code:200,}"}
2020-11-11T04:54:03.813Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runnerreplicaset", "UID": "edabadcb-c8a1-41c6-8f5a-af823eeb7759", "kind": "actions.summerwind.dev/v1alpha1, Kind=RunnerReplicaSet", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runnerreplicasets"}}
2020-11-11T04:54:03.813Z	INFO	runnerreplicaset-resource	validate resource to be created	{"name": "sre-github-runners-runner-deployment-mpr47"}
2020-11-11T04:54:03.813Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runnerreplicaset", "UID": "edabadcb-c8a1-41c6-8f5a-af823eeb7759", "allowed": true, "result": {}, "resultError": "got runtime.Object without object metadata: &Status{ListMeta:ListMeta{SelfLink:,ResourceVersion:,Continue:,RemainingItemCount:nil,},Status:,Message:,Reason:,Details:nil,Code:200,}"}
2020-11-11T04:54:03.825Z	INFO	controllers.RunnerReplicaSet	debug	{"runner": "sre-actions-runner-system/sre-github-runners-runner-deployment-mpr47", "desired": 1, "available": 0}
2020-11-11T04:54:03.825Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "runnerdeployment", "request": "sre-actions-runner-system/sre-github-runners-runner-deployment"}
2020-11-11T04:54:03.825Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "runnerdeployment", "request": "sre-actions-runner-system/sre-github-runners-runner-deployment"}
2020-11-11T04:54:03.827Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runner", "UID": "9f4dc1e3-dede-485b-943b-abf0b709e069", "kind": "actions.summerwind.dev/v1alpha1, Kind=Runner", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runners"}}
2020-11-11T04:54:03.892Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runner", "UID": "9f4dc1e3-dede-485b-943b-abf0b709e069", "allowed": true, "result": {}, "resultError": "got runtime.Object without object metadata: &Status{ListMeta:ListMeta{SelfLink:,ResourceVersion:,Continue:,RemainingItemCount:nil,},Status:,Message:,Reason:,Details:nil,Code:200,}"}
2020-11-11T04:54:03.894Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runner", "UID": "f3542094-cfbb-4356-94ba-70ac48f1b3c4", "kind": "actions.summerwind.dev/v1alpha1, Kind=Runner", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runners"}}
2020-11-11T04:54:03.894Z	INFO	runner-resource	validate resource to be created	{"name": "sre-github-runners-runner-deployment-mpr47-xxlnk"}
2020-11-11T04:54:03.894Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runner", "UID": "f3542094-cfbb-4356-94ba-70ac48f1b3c4", "allowed": true, "result": {}, "resultError": "got runtime.Object without object metadata: &Status{ListMeta:ListMeta{SelfLink:,ResourceVersion:,Continue:,RemainingItemCount:nil,},Status:,Message:,Reason:,Details:nil,Code:200,}"}
2020-11-11T04:54:03.899Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "runnerreplicaset", "request": "sre-actions-runner-system/sre-github-runners-runner-deployment-mpr47"}
2020-11-11T04:54:03.899Z	INFO	controllers.RunnerReplicaSet	debug	{"runner": "sre-actions-runner-system/sre-github-runners-runner-deployment-mpr47", "desired": 1, "available": 1}
2020-11-11T04:54:03.902Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runner", "UID": "275439cf-0577-4aa3-8a69-eb951f4786ac", "kind": "actions.summerwind.dev/v1alpha1, Kind=Runner", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runners"}}
2020-11-11T04:54:03.902Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runner", "UID": "275439cf-0577-4aa3-8a69-eb951f4786ac", "allowed": true, "result": {}, "resultError": "got runtime.Object without object metadata: &Status{ListMeta:ListMeta{SelfLink:,ResourceVersion:,Continue:,RemainingItemCount:nil,},Status:,Message:,Reason:,Details:nil,Code:200,}"}
2020-11-11T04:54:03.904Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runner", "UID": "b518c167-2e36-4aab-a4a3-c25a63d842ec", "kind": "actions.summerwind.dev/v1alpha1, Kind=Runner", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runners"}}
2020-11-11T04:54:03.904Z	INFO	runner-resource	validate resource to be updated	{"name": "sre-github-runners-runner-deployment-mpr47-xxlnk"}
2020-11-11T04:54:03.904Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runner", "UID": "b518c167-2e36-4aab-a4a3-c25a63d842ec", "allowed": true, "result": {}, "resultError": "got runtime.Object without object metadata: &Status{ListMeta:ListMeta{SelfLink:,ResourceVersion:,Continue:,RemainingItemCount:nil,},Status:,Message:,Reason:,Details:nil,Code:200,}"}
2020-11-11T04:54:03.908Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "runnerdeployment", "request": "sre-actions-runner-system/sre-github-runners-runner-deployment"}
2020-11-11T04:54:03.908Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "runnerreplicaset", "request": "sre-actions-runner-system/sre-github-runners-runner-deployment-mpr47"}
2020-11-11T04:54:03.908Z	INFO	controllers.RunnerReplicaSet	debug	{"runner": "sre-actions-runner-system/sre-github-runners-runner-deployment-mpr47", "desired": 1, "available": 1}
2020-11-11T04:54:03.908Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "runnerreplicaset", "request": "sre-actions-runner-system/sre-github-runners-runner-deployment-mpr47"}
2020-11-11T04:54:03.909Z	INFO	controllers.RunnerReplicaSet	debug	{"runner": "sre-actions-runner-system/sre-github-runners-runner-deployment-mpr47", "desired": 1, "available": 1}
2020-11-11T04:54:03.909Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "runnerreplicaset", "request": "sre-actions-runner-system/sre-github-runners-runner-deployment-mpr47"}
2020-11-11T04:54:03.909Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "runner", "request": "sre-actions-runner-system/sre-github-runners-runner-deployment-mpr47-xxlnk"}
2020-11-11T04:54:04.792Z	INFO	controllers.Runner	Updated registration token	{"runner": "sre-github-runners-runner-deployment-mpr47-xxlnk", "repository": ""}
2020-11-11T04:54:04.792Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"Runner","namespace":"sre-actions-runner-system","name":"sre-github-runners-runner-deployment-mpr47-xxlnk","uid":"20d90a92-4467-45cc-9515-90c3773483ce","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"124929899"}, "reason": "RegistrationTokenUpdated", "message": "Successfully update registration token"}
2020-11-11T04:54:04.792Z	INFO	controllers.RunnerReplicaSet	debug	{"runner": "sre-actions-runner-system/sre-github-runners-runner-deployment-mpr47", "desired": 1, "available": 1}
2020-11-11T04:54:04.792Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "runnerreplicaset", "request": "sre-actions-runner-system/sre-github-runners-runner-deployment-mpr47"}
2020-11-11T04:54:04.828Z	INFO	controllers.Runner	Created runner pod	{"runner": "sre-actions-runner-system/sre-github-runners-runner-deployment-mpr47-xxlnk", "repository": ""}
2020-11-11T04:54:04.828Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "runner", "request": "sre-actions-runner-system/sre-github-runners-runner-deployment-mpr47-xxlnk"}
2020-11-11T04:54:04.828Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"Runner","namespace":"sre-actions-runner-system","name":"sre-github-runners-runner-deployment-mpr47-xxlnk","uid":"20d90a92-4467-45cc-9515-90c3773483ce","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"124929906"}, "reason": "PodCreated", "message": "Created pod 'sre-github-runners-runner-deployment-mpr47-xxlnk'"}
2020-11-11T04:54:04.893Z	ERROR	controllers.Runner	Failed to create pod resource	{"runner": "sre-actions-runner-system/sre-github-runners-runner-deployment-mpr47-xxlnk", "error": "pods \"sre-github-runners-runner-deployment-mpr47-xxlnk\" already exists"}
github.com/go-logr/zapr.(*zapLogger).Error
	/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
github.com/summerwind/actions-runner-controller/controllers.(*RunnerReconciler).Reconcile
	/workspace/controllers/runner_controller.go:142
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:256
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:232
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:88
2020-11-11T04:54:04.893Z	ERROR	controller-runtime.controller	Reconciler error	{"controller": "runner", "request": "sre-actions-runner-system/sre-github-runners-runner-deployment-mpr47-xxlnk", "error": "pods \"sre-github-runners-runner-deployment-mpr47-xxlnk\" already exists"}
github.com/go-logr/zapr.(*zapLogger).Error
	/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:258
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:232
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:88
2020-11-11T04:54:05.616Z	DEBUG	controllers.HorizontalRunnerAutoscaler	Calculated desired replicas	{"computed_replicas_desired": 1, "spec_replicas_min": 1, "spec_replicas_max": 10, "workflow_runs_completed": 33, "workflow_runs_in_progress": 0, "workflow_runs_queued": 0, "workflow_runs_unknown": 0}
2020-11-11T04:54:05.622Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "horizontalrunnerautoscaler", "request": "sre-actions-runner-system/sre-github-runners-autoscaler"}
2020-11-11T04:54:05.900Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "runner", "request": "sre-actions-runner-system/sre-github-runners-runner-deployment-mpr47-xxlnk"}
2020-11-11T04:54:05.900Z	INFO	controllers.RunnerReplicaSet	debug	{"runner": "sre-actions-runner-system/sre-github-runners-runner-deployment-mpr47", "desired": 1, "available": 1}
2020-11-11T04:54:05.900Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "runnerreplicaset", "request": "sre-actions-runner-system/sre-github-runners-runner-deployment-mpr47"}
2020-11-11T04:54:06.171Z	ERROR	controllers.Runner	Failed to check if runner is busy	{"runner": "sre-actions-runner-system/sre-github-runners-runner-deployment-mpr47-xxlnk", "error": "runner not found"}
github.com/go-logr/zapr.(*zapLogger).Error
	/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
github.com/summerwind/actions-runner-controller/controllers.(*RunnerReconciler).Reconcile
	/workspace/controllers/runner_controller.go:197
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:256
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:232
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:88
2020-11-11T04:54:06.171Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "runner", "request": "sre-actions-runner-system/sre-github-runners-runner-deployment-mpr47-xxlnk"}
2020-11-11T04:54:06.466Z	DEBUG	controllers.HorizontalRunnerAutoscaler	Calculated desired replicas	{"computed_replicas_desired": 1, "spec_replicas_min": 1, "spec_replicas_max": 10, "workflow_runs_completed": 33, "workflow_runs_in_progress": 0, "workflow_runs_queued": 0, "workflow_runs_unknown": 0}
2020-11-11T04:54:06.466Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "horizontalrunnerautoscaler", "request": "sre-actions-runner-system/sre-github-runners-autoscaler"}
2020-11-11T04:54:07.696Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "runner", "request": "sre-actions-runner-system/sre-github-runners-runner-deployment-mpr47-xxlnk"}
2020-11-11T04:54:07.696Z	INFO	controllers.RunnerReplicaSet	debug	{"runner": "sre-actions-runner-system/sre-github-runners-runner-deployment-mpr47", "desired": 1, "available": 1}
2020-11-11T04:54:07.704Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "runnerreplicaset", "request": "sre-actions-runner-system/sre-github-runners-runner-deployment-mpr47"}
2020-11-11T04:54:07.704Z	INFO	controllers.RunnerReplicaSet	debug	{"runner": "sre-actions-runner-system/sre-github-runners-runner-deployment-mpr47", "desired": 1, "available": 1}
2020-11-11T04:54:07.704Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "runnerreplicaset", "request": "sre-actions-runner-system/sre-github-runners-runner-deployment-mpr47"}
2020-11-11T04:54:07.704Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "runnerdeployment", "request": "sre-actions-runner-system/sre-github-runners-runner-deployment"}
2020-11-11T04:54:07.980Z	ERROR	controllers.Runner	Failed to check if runner is busy	{"runner": "sre-actions-runner-system/sre-github-runners-runner-deployment-mpr47-xxlnk", "error": "runner not found"}
github.com/go-logr/zapr.(*zapLogger).Error
	/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
github.com/summerwind/actions-runner-controller/controllers.(*RunnerReconciler).Reconcile
	/workspace/controllers/runner_controller.go:197
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:256
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:232
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:88
2020-11-11T04:54:07.980Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "runner", "request": "sre-actions-runner-system/sre-github-runners-runner-deployment-mpr47-xxlnk"}

Runner YAML
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sre-github-actions-service-account
  namespace: sre-actions-runner-system
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::373378159773:role/eksctl-sre-eks-test-addon-iamserviceaccount-Role1-1244USHBUKLEY
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: sre-github-runners-runner-deployment
  namespace: sre-actions-runner-system
spec:
  replicas: 1
  template:
    spec:
      organization: my-org
      serviceAccountName: sre-github-actions-service-account
      env:
      - name: AWS_ROLE_ARN
        value: arn:aws:iam::373378159773:role/eksctl-sre-eks-test-addon-iamserviceaccount-Role1-1244USHBUKLEY
      - name: AWS_WEB_IDENTITY_TOKEN_FILE
        value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
      volumeMounts:
      - mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
        name: aws-iam-token
        readOnly: true
      volumes:
      - name: aws-iam-token
        projected:
          defaultMode: 420
          sources:
          - serviceAccountToken:
              audience: sts.amazonaws.com
              expirationSeconds: 86400
              path: token
      securityContext:
        fsGroup: 27
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: sre-github-runners-autoscaler
  namespace: sre-actions-runner-system
spec:
  scaleDownDelaySecondsAfterScaleUp: 10
  scaleTargetRef:
    name: sre-github-runners-runner-deployment
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: TotalNumberOfQueuedAndInProgressWorkflowRuns
    repositoryNames:
    - callum-test-repo
    - callum-test-repo-2

@callum-tait-pbx Thanks for reporting! The error following error:
Failed to create pod resource	{"runner": "sre-actions-runner-system/sre-github-runners-runner-deployment-mpr47-xxlnk", "error": "pods \"sre-github-runners-runner-deployment-mpr47-xxlnk\" already exists"}

is super interesting, as the controller is coded to create the pod only when the pod doesn't exist in the cluster:
https://github.com/summerwind/actions-runner-controller/blob/e613219a8980c88977f8231bedbe691b5caa3d9c/controllers/runner_controller.go#L124-L144
Out of curiosity - does it work if you run kubectl apply -f https://github.com/summerwind/actions-runner-controller/releases/latest/download/actions-runner-controller.yaml without modifying the namespace or anything else, and used namespace other than sre-actions-runner-system for the RunnerDeployment resource? (I usually use default for RunnerDeployment and actions-runner-system for the controller)

I'll give it a test now, give a few mins to set up the namespace. It's curious as I've been testing this solution ina  custom namespace for a while. We have a policy within my organisaiton where namespaces need to be prefixed by a team to track ownership etc. This isn't something I'm going to be able to convince the team to drop so it's critical that I can deploy the solution in a custom namespace.

@callum-tait-pbx I hear you. As long as the root cause is isolated, I'll try my best to fix it so that it works with the custom namespace for you 👍

That would be amazing if possible, I think this solution is fantastic and has lots of potential so I'm really excited to adopt it within my organisation but this would be a shop stopped for us :(. The other requirement I'm battling atm is the need to apply some custom labels for resources, I'm confident sanity will prevail though and I can convince the team to only require them for pods / runners :D
Currently getting auth issues with the github app when trying the solution in the standard namespace, not had that before but we might have policies causing issues or something. This might take a while to figure out, for your information the error is:
2020-11-11T05:52:15.312Z	ERROR	controller-runtime.controller	Reconciler error	{"controller": "horizontalrunnerautoscaler", "request": "sre-actions-runner-system/sre-github-runners-autoscaler", "error": "Get \"https://api.github.com/repos/photobox/callum-test-repo/actions/runs\": could not refresh installation id 73707's token: request &{Method:POST URL:https://api.github.com/app/installations/73707/access_tokens Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept:[application/vnd.github.machine-man-preview+json application/vnd.github.machine-man-preview+json] Authorization:[Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2MDUwNzM5OTUsImlhdCI6MTYwNTA3MzkzNSwiaXNzIjoiMTA2ODU0NTYifQ.sbcUURCui5ocW7tPaDwpOC5w0jMqCxpBKsozevDo_3rbHYfVUEHxmx9RYJ4Ak0cJKXhEY0XW1-oSHuF6YgxfhoApSfNgfbwVtEpgHbiYfMJDbsdHng_s1OejBBDN7vOPLpvbri5iwqbaBdrWLvG-kQ4_-53tlVVNgb7cH2LgA9-nAjrbTKJTvvAN5E5LHuR_sY79RtXIizj4u_kLwcuaxmhh9Xr5IxZ-KaLGcIjPn7pkmTUNXwIzdhXpKOm2YLMDdstrZ7mjKIIvMZh6O4QNE-VQSaduu8-hIbsFmkK6Lkdq0ZNsE5GJEUvXgCJtJL3KyG3USDAIdrTepxGl4M9hRQ] Content-Type:[application/json]] Body:{Reader:} GetBody:0x7fab00 ContentLength:5 TransferEncoding:[] Close:false Host:api.github.com Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil> Cancel:<nil> Response:<nil> ctx:0xc0001a6240} received non 2xx response status &{[] %!q(*http.bodyEOFSignal=&{0xc00059d880 {0 0} false <nil> 0x7fc8e0 0x7fc860}) %!q(*gzip.Reader=<nil>) <nil>} with body &{Method:POST URL:https://api.github.com/app/installations/73707/access_tokens Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept:[application/vnd.github.machine-man-preview+json application/vnd.github.machine-man-preview+json] Authorization:[Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2MDUwNzM5OTUsImlhdCI6MTYwNTA3MzkzNSwiaXNzIjoiMTA2ODU0NTYifQ.sbcUURCui5ocW7tPaDwpOC5w0jMqCxpBKsozevDo_3rbHYfVUEHxmx9RYJ4Ak0cJKXhEY0XW1-oSHuF6YgxfhoApSfNgfbwVtEpgHbiYfMJDbsdHng_s1OejBBDN7vOPLpvbri5iwqbaBdrWLvG-kQ4_-53tlVVNgb7cH2LgA9-nAjrbTKJTvvAN5E5LHuR_sY79RtXIizj4u_kLwcuaxmhh9Xr5IxZ-KaLGcIjPn7pkmTUNXwIzdhXpKOm2YLMDdstrZ7mjKIIvMZh6O4QNE-VQSaduu8-hIbsFmkK6Lkdq0ZNsE5GJEUvXgCJtJL3KyG3USDAIdrTepxGl4M9hRQ] Content-Type:[application/json]] Body:{Reader:} GetBody:0x7fab00 ContentLength:5 TransferEncoding:[] Close:false Host:api.github.com Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil> Cancel:<nil> Response:<nil> ctx:0xc0001a6240} and TLS &{Version:772 HandshakeComplete:true DidResume:false CipherSuite:4865 NegotiatedProtocol:http/1.1 NegotiatedProtocolIsMutual:true ServerName:api.github.com PeerCertificates:[0xc000ca4b00 0xc000ca5080] VerifiedChains:[[0xc000ca4b00 0xc000ca5080 0xc00063cb00]] SignedCertificateTimestamps:[] OCSPResponse:[] TLSUnique:[] ekm:0x750be0}"}
github.com/go-logr/zapr.(*zapLogger).Error
	/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:258
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:232
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:88

some custom labels for resources

@callum-tait-pbx Sounds like a valid feature request. Would you mind writing issues to track those? That would allow me to keep remembering what I need to work on :)

Currently getting auth issues with the github app when trying the solution in the standard namespace

Probably this might be an issue with your K8s secret containing the token as it should now reside within the actions-runner-system namespace, that should have been created by running:
$ kubectl create secret generic controller-manager \
    -n actions-runner-system \
    --from-literal=github_app_id=${APP_ID} \
    --from-literal=github_app_installation_id=${INSTALLATION_ID} \
    --from-file=github_app_private_key=${PRIVATE_KEY_FILE_PATH}

Did that :(, I'm quite confused now. When I posted previously I realised I had the secret in both my custom namespace as well as the solutions default namespace. I deleted it all and redeployed again with just the 1 version of the secret in the solutions default namespace. I'm going to delete the app too and start fresh at this point.

I was wondering if running kubecto get -o yaml -n actions-runner-system controller-manager would reveal the issue if that's some redundant/missing character in your token contained in the secret, or insufficient permissions on the github app 🤔

Curious, I've:

Deleted everthing
Deployed the namespace and secret
Deployed the latest manager kubectl apply -f https://github.com/summerwind/actions-runner-controller/releases/latest/download/actions-runner-controller.yaml

error: the server doesn't have a resource type "controller-manager"```

Thanks. The deployment seems fine.
The possible problem should only reside in your secret containing github_app_id, github_app_installation_id, and github_app_private_key. I thought I have once failed due to redundant new line after the private key content. So hopefully something like that is the cause.
Also, I'll follow and confirm the whole setup process with github app and report back to you, so that becomes clear if it's really happening only for you or not.

Thanks! Once my co-workers are online I'll nuke the namespace I was working under and deploy again so I'm confident the node and namespace hasn't got additional junk on it which could be causing the issue. It's super odd as this previously was working fine in a custom namespace :(

sorry for hijacking this issue, but i'm now experiencing the same situation while trying a brand new installation with 0.11.2. before that, we were successfully running 0.7.1, if that matters. in my case, I'm using the actions-runner-system namespace for both controller and RunnerDeployments and personal access token instead of Github Application, the token has org admin permissions...
logs can be found here: https://gist.github.com/hatemosphere/15cae780ac4a889b33ead2905a779a7d

Oh, there was another comment a few hours ago from hatemosphere saying they were getting the same issue with a new installion on 0.11.2 having previously had a working installtion of 0.7. something. Gist they provided was here https://gist.github.com/hatemosphere/15cae780ac4a889b33ead2905a779a7d

@callum-tait-pbx Thanks for sharing! I do see messages like Failed to check if runner is busy	{"runner": "actions-runner-system/docker-runner-rz885-9lttl", "error": "runner not found"} on my env, too.
But neither the hatemosphere's logs and my logs contain could not refresh installation id %MY_INSTALLTION_ID%'s token nor Failed to create pod resource. And mine seems to successfully create runner pods despite the Failed to check if runner is busy errors.
Would you mind confirming if hatemosphere's runner controller instance is really not working, or it's just that the controller emits the verbose Failed to check if runner is busy messages?

I am also seeing these Failed to check if runner is busy {"runner": "actions-runner-system/docker-runner-rz885-9lttl", "error": "runner not found"} messages on the controller.
I often find them paired with:
28m         Warning   FailedScheduling           pod/actions-runner-deployment-g6rkj-g5djg      0/23 nodes are available: 1 Insufficient memory, 20 Insufficient cpu, 3 node(s) had taints that the pod didn't tolerate.
29m         Normal    NotTriggerScaleUp          pod/actions-runner-deployment-g6rkj-g5djg      pod didn't trigger scale-up (it wouldn't fit if a new node is added): 3 node(s) had taints that the pod didn't tolerate, 1 max node group size reached
34m         Normal    NotTriggerScaleUp          pod/actions-runner-deployment-g6rkj-g5djg      pod didn't trigger scale-up (it wouldn't fit if a new node is added): 1 max node group size reached, 3 node(s) had taints that the pod didn't tolerate
2m35s       Warning   FailedScheduling           pod/actions-runner-deployment-g6rkj-g5djg      0/23 nodes are available: 1 Insufficient memory, 20 Insufficient cpu, 3 node(s) had taints that the pod didn't tolerate.
2m33s       Normal    NotTriggerScaleUp          pod/actions-runner-deployment-g6rkj-g5djg      pod didn't trigger scale-up (it wouldn't fit if a new node is added): 3 node(s) had taints that the pod didn't tolerate, 1 max node group size reached

are either of you seeing scale up issues tied to this?

@ZacharyBenamram Yep. Are you sure you have available node resources on your cluster?
Reading 3 node(s) had taints that the pod didn't tolerate, it seems like you have too restrictive node taints that actions-runner-controller can't tolerate.
If that's the case, you have to remove the node taints, or modify actions-runner-controller deployment to give pods necessary tolerations.

@mumoshu - yes! I'll add the tolerations which should resolve pods being created, but failing to schedule causing the "runner not found" side effect.

@ZacharyBenamram Thanks for confirming! Please see my updated comment above:

~ you have to remove the node taints, or modify the runnerdeployment to give runner pods necessary tolerations

As you may already know, it's the runner pods that are failing to schedule, and that prevents runners from being registered. If it's unregistered, it's not found via GitHub API, hence the "runner not found" error.
That being said, I think the root cause is the node taints/runner pods tolerations matching for your case. Once that's fixed, runner pods should be scheduled normally, and "runner not found" errors will disappear.

That would be amazing if possible, I think this solution is fantastic and has lots of potential so I'm really excited to adopt it within my organisation but this would be a show stopper for us :(. The other requirement I'm battling atm is the need to apply some custom labels for resources, I'm confident sanity will prevail though and I can convince the team to only require them for pods / runners :D
Currently getting auth issues with the github app when trying the solution in the standard namespace, not had that before but we might have policies causing issues or something. This might take a while to figure out, for your information the error is:
2020-11-11T05:52:15.312Z	ERROR	controller-runtime.controller	Reconciler error	{"controller": "horizontalrunnerautoscaler", "request": "sre-actions-runner-system/sre-github-runners-autoscaler", "error": "Get \"https://api.github.com/repos/%MY_ORG%/callum-test-repo/actions/runs\": could not refresh installation id %MY_INSTALLTION_ID%'s token: request &{Method:POST URL:https://api.github.com/app/installations/%MY_INSTALLTION_ID%/access_tokens Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept:[application/vnd.github.machine-man-preview+json application/vnd.github.machine-man-preview+json] Authorization:[Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2MDUwNzM5OTUsImlhdCI6MTYwNTA3MzkzNSwiaXNzIjoiMTA2ODU0NTYifQ.sbcUURCui5ocW7tPaDwpOC5w0jMqCxpBKsozevDo_3rbHYfVUEHxmx9RYJ4Ak0cJKXhEY0XW1-oSHuF6YgxfhoApSfNgfbwVtEpgHbiYfMJDbsdHng_s1OejBBDN7vOPLpvbri5iwqbaBdrWLvG-kQ4_-53tlVVNgb7cH2LgA9-nAjrbTKJTvvAN5E5LHuR_sY79RtXIizj4u_kLwcuaxmhh9Xr5IxZ-KaLGcIjPn7pkmTUNXwIzdhXpKOm2YLMDdstrZ7mjKIIvMZh6O4QNE-VQSaduu8-hIbsFmkK6Lkdq0ZNsE5GJEUvXgCJtJL3KyG3USDAIdrTepxGl4M9hRQ] Content-Type:[application/json]] Body:{Reader:} GetBody:0x7fab00 ContentLength:5 TransferEncoding:[] Close:false Host:api.github.com Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil> Cancel:<nil> Response:<nil> ctx:0xc0001a6240} received non 2xx response status &{[] %!q(*http.bodyEOFSignal=&{0xc00059d880 {0 0} false <nil> 0x7fc8e0 0x7fc860}) %!q(*gzip.Reader=<nil>) <nil>} with body &{Method:POST URL:https://api.github.com/app/installations/%MY_INSTALLTION_ID%/access_tokens Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept:[application/vnd.github.machine-man-preview+json application/vnd.github.machine-man-preview+json] Authorization:[Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2MDUwNzM5OTUsImlhdCI6MTYwNTA3MzkzNSwiaXNzIjoiMTA2ODU0NTYifQ.sbcUURCui5ocW7tPaDwpOC5w0jMqCxpBKsozevDo_3rbHYfVUEHxmx9RYJ4Ak0cJKXhEY0XW1-oSHuF6YgxfhoApSfNgfbwVtEpgHbiYfMJDbsdHng_s1OejBBDN7vOPLpvbri5iwqbaBdrWLvG-kQ4_-53tlVVNgb7cH2LgA9-nAjrbTKJTvvAN5E5LHuR_sY79RtXIizj4u_kLwcuaxmhh9Xr5IxZ-KaLGcIjPn7pkmTUNXwIzdhXpKOm2YLMDdstrZ7mjKIIvMZh6O4QNE-VQSaduu8-hIbsFmkK6Lkdq0ZNsE5GJEUvXgCJtJL3KyG3USDAIdrTepxGl4M9hRQ] Content-Type:[application/json]] Body:{Reader:} GetBody:0x7fab00 ContentLength:5 TransferEncoding:[] Close:false Host:api.github.com Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil> Cancel:<nil> Response:<nil> ctx:0xc0001a6240} and TLS &{Version:772 HandshakeComplete:true DidResume:false CipherSuite:4865 NegotiatedProtocol:http/1.1 NegotiatedProtocolIsMutual:true ServerName:api.github.com PeerCertificates:[0xc000ca4b00 0xc000ca5080] VerifiedChains:[[0xc000ca4b00 0xc000ca5080 0xc00063cb00]] SignedCertificateTimestamps:[] OCSPResponse:[] TLSUnique:[] ekm:0x750be0}"}
github.com/go-logr/zapr.(*zapLogger).Error
	/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:258
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:232
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:88

I've raised an issue as I am getting GitHub API auth issues with a PAT token now, hopefully I'm just being dumb and it's config on my end 🤞  https://github.com/summerwind/actions-runner-controller/issues/181

As you might expect, this error appears to break the scaling feature. I triggered 2 builds, and whilst my 1st runner was busy, a second never scaled up to deal with the second. If this is a bug in the current version of the code this is a major issue.

hey guys, sorry for misleading you with my message - in my case i had a different issue on my side, so Failed to check if runner is busy wasn't causing any issues and runners are being created and registered successfully. that's why i removed my message :)

No worries @hatemosphere! Out of interest did you resolve the error in your env or is it still there but a non-issue I've raised an issue for it as it would be good to know why it happens and how to fix it regardless https://github.com/summerwind/actions-runner-controller/issues/183

I'm going to close this issue as I have raised a new issue specifically for the "error": "runner not found". @hatemosphere if you could confirm if you are still having that error and it's not a problem or you no longer have that error and what the fix was on the new issue that would be awesome. New issue here https://github.com/summerwind/actions-runner-controller/issues/183


The check dosn't accept max lenght of row.

 But when I run --check I received messsage that  the files would be reformated. It looks that the check doesn't accept max lenght of row.
Operating system:
Python version:
Black version: black::18.6b4
Does also happen on master:
black /app --pyi -l 120 --exclude venv
black /app --check -l 120 --exclude venv

Your problem is that you missed --pyi in your --check call.

I'm not sure why you'd use --pyi in the first place though. Black will reconize .pyi files just fine by default. Using --pyi means ALL found Python files are reformatted like .pyi files. Is this really what you want?

For my case is not import  --pyi. I wanted to check files with 120 max length. It is the point. But the command doesn't accept parameter "-l".

It does, which is why I'm asking what you're trying to do. Look at the code, --check simply means we don't write the file back and return 1 if there are changes. It's the same code path otherwise.

Can you re-run without --pyi anywhere but with --verbose?

I'll close this for now but @Pajinek let us know if you manage to reproduce it without using --pyi anywhere


Audio system stops working when unplugging the network

I was doing some network recovery testing with one of my niche bots when I discovered that the voice system simply stopped being able to send or leave a channel after being unplugged for a few seconds.

AudioManager#getConnectedChannel() appears to return null
AudioManager#closeAudioConnection() does nothing.
It doesn't seem that the sending handler is getting any calls.
The bot can move to a different voice channel, but the above still occurs.
I am on 3.1.0_204

Wow, what a coincidence, I ran into the very same problem a week ago. Had an in-depth talk with @MinnDevelopment about it. Discord appears to not resume/terminate a lost audio connection unless you change voice regions, or login again (e.g. reviving a shard).

I had some luck getting the connection audio connection restored by following this reference http://knowledgebase.progress.com/articles/Article/000044970
and setting the timeout of the machine the bot is running on to rather aggressive values of
echo 60 > /proc/sys/net/ipv4/tcp_keepalive_time
echo 1 > /proc/sys/net/ipv4/tcp_keepalive_probes

Setting the timeout to 120 had lavaplayer collect the track with AudioTrackEndReason.CLEANUP.
I'm not sure if editing the timeout values like this is a good idea.


DeID Handbook update - two/multi stage process

IHE_Profile_Proposal DeID-handbook-update-process-enhancement-20241212.docx

Can possibly combine with #216


Nullables Still Acting Up

Try to run copyHelper.SaveAll(connection, entities);
Where entities is a list of Student Objects with a string Name field and nullable int field for age.
If the first student in the entities list has a null age and the second has an age defined, it freaks out and fails.

Weird. 🤔 I will look into it, but it will take some time. Would you mind writing a small Unit Test to reproduce fhe problem?

Working on that now

Without exposing my code too much (working in an Enterprise project), the test should look like this:
        [Test]
        public void Test_NullableSmallInt_FirstNull()
        {

            var entity0 = new TestEntity()
            {
                SmallInt = null
            };

            var entity1 = new TestEntity()
            {
                SmallInt = Int16.MaxValue
            };

            subject.SaveAll(connection, new[] { entity0, entity1 });

            var result = GetAll();

            // Check if we have the amount of rows:
            Assert.AreEqual(2, result.Count);

            Assert.IsNotNull(result[0][0]);
            Assert.IsNotNull(result[1][0]);

            Assert.AreEqual(Int16.MinValue, (Int16)result[0][0]);
            Assert.AreEqual(Int16.MaxValue, (Int16)result[1][0]);
        }

Unfortunately running the test against my RDS actually fails :(

This is obviously an example but should apply to all types, Integer, Numeric, etc.

Looking at the test it has to fail, because of this line:
 Assert.AreEqual(Int16.MinValue, (Int16)result[0][0]);

The result should be null.

Fails before it gets to that line unfortunately. But yea I will patch that in the PR.
   at Npgsql.NpgsqlParameter`1.ValidateAndGetLength() in C:\projects\npgsql\src\Npgsql\NpgsqlParameter`.cs:line 90
   at Npgsql.NpgsqlBinaryImporter.Write[T](T value, NpgsqlParameter param) in C:\projects\npgsql\src\Npgsql\NpgsqlBinaryImporter.cs:line 244
   at PostgreSQLCopyHelper.PostgreSQLCopyHelper`1.<>c__DisplayClass12_0`1.<MapNullable>b__0(NpgsqlBinaryImporter writer, TEntity entity) in D:\work\projects\PostgreSQLCopyHelper\PostgreSQLCopyHelper\PostgreSQLCopyHelper\PostgreSQLCopyHelper\PostgreSQLCopyHelper.cs:line 61
   at PostgreSQLCopyHelper.PostgreSQLCopyHelper`1.WriteToStream(NpgsqlBinaryImporter writer, IEnumerable`1 entities) in D:\work\projects\PostgreSQLCopyHelper\PostgreSQLCopyHelper\PostgreSQLCopyHelper\PostgreSQLCopyHelper\PostgreSQLCopyHelper.cs:line 74
   at PostgreSQLCopyHelper.PostgreSQLCopyHelper`1.SaveAll(NpgsqlConnection connection, IEnumerable`1 entities) in D:\work\projects\PostgreSQLCopyHelper\PostgreSQLCopyHelper\PostgreSQLCopyHelper\PostgreSQLCopyHelper\PostgreSQLCopyHelper.cs:line 39
   at PostgreSQLCopyHelper.Test.NullableBasicDataTypesTest.Test_NullableSmallInt_NullFirst() in D:\work\projects\PostgreSQLCopyHelper\PostgreSQLCopyHelper\PostgreSQLCopyHelper\PostgreSQLCopyHelper.Test\NullableBasicDataTypesTest.cs:line 76

Here is the stack FYI

I wonder why it acts up. Could you paste the error message?

Meanwhile, please stay on Version 1.3.0 until there is a fix in place.

This is how Nullables are written. Are you able to spot an obvious error there?

https://github.com/bytefish/PostgreSQLCopyHelper/blob/4ecb2aa0bdfc7042838eb902223e096bd1b641a1/PostgreSQLCopyHelper/PostgreSQLCopyHelper/PostgreSQLCopyHelper/PostgreSQLCopyHelper.cs#L52

I looked there first. It should basically use the writer.Write(DBNull.Value, type); the first time and then writer.Write(value, type); the second time. Looks like it should work but... :/

The Npgsql Binary writer has a WriteNull() method. Could you try if replacing this with my current DBNull.Value writing fixes the issue?

That does fix the issue 😎

💪💪💪
I will fix it and release a new version to NuGet. Thanks for your help!

No problem thanks for your responsiveness :D

I have released a new Version 2.2.0 to NuGet, which should fix the issue. If the problem still persists, please reopen the ticket. And thanks again! 👍

Sorry I was too quick to close! I will add your tests later today in a separate merge.


Catching ECONNRESET

Hello,
I am using WAL-E to backup to SoftLayer's object store using swift.  I'm not sure why, but there's some connection issues to the object store in the morning and I get connection reset errors for some reason during an upload.
It looks like the python openssl is throwing an exception when this happens (as it should) but this type of exception isn't caught in the blobstore upload. Shouldn't we catch this exception and retry? It seems silly to stop the whole database backup because one storage attempt failed.
requests.packages.urllib3.connectionpool INFO     Starting new HTTPS connection (1): dal05.objectstorage.service.networklayer.com
requests.packages.urllib3.connectionpool INFO     Starting new HTTPS connection (1): dal05.objectstorage.service.networklayer.com
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/gevent/greenlet.py", line 327, in run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/wal_e/worker/upload.py", line 144, in __call__
    k = put_file_helper()
  File "/usr/local/lib/python2.7/dist-packages/wal_e/retries.py", line 72, in shim
    exc_processor_cxt=exc_processor_cxt)
  File "/usr/local/lib/python2.7/dist-packages/wal_e/retries.py", line 104, in retry_with_count_internal
    side_effect_func(exc_tup, exc_processor_cxt)
  File "/usr/local/lib/python2.7/dist-packages/wal_e/retries.py", line 62, in shim
    return f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/wal_e/worker/upload.py", line 139, in put_file_helper
    return self.blobstore.uri_put_file(self.creds, url, tf)
  File "/usr/local/lib/python2.7/dist-packages/wal_e/blobstore/swift/utils.py", line 37, in uri_put_file
    container_name, url_tup.path, fp, content_type=content_encoding
  File "/usr/local/lib/python2.7/dist-packages/swiftclient/client.py", line 1369, in put_object
    response_dict=response_dict)
  File "/usr/local/lib/python2.7/dist-packages/swiftclient/client.py", line 1243, in _retry
    rv = func(self.url, self.token, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/swiftclient/client.py", line 993, in put_object
    conn.putrequest(path, headers=headers, data=chunk_reader())
  File "/usr/local/lib/python2.7/dist-packages/swiftclient/client.py", line 227, in putrequest
    return self.request('PUT', full_path, data, headers, files)
  File "/usr/local/lib/python2.7/dist-packages/swiftclient/client.py", line 217, in request
    files=files, **self.requests_args)
  File "/usr/local/lib/python2.7/dist-packages/swiftclient/client.py", line 185, in _request
    return self.request_session.request(*arg, **kwarg)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 408, in send
    low_conn.close()
  File "/usr/lib/python2.7/httplib.py", line 765, in close
    self.sock.close()   # close it manually... there may be other refs
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 213, in close
    return self.connection.shutdown()
  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 1522, in shutdown
    self._raise_ssl_error(self._ssl, result)
  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 1178, in _raise_ssl_error
    raise SysCallError(errno, errorcode.get(errno))
SysCallError: (104, 'ECONNRESET')
<Greenlet at 0x2068370: <wal_e.worker.upload.PartitionUploader object at 0x205e550>([ExtendedTarInfo(submitted_path='/ssd/pgsqldata/ba)> failed with SysCallError

wal_e.operator.backup WARNING  MSG: blocking on sending WAL segments
        DETAIL: The backup was not completed successfully, but we have to wait anyway.  See README: TODO about pg_cancel_backup
        STRUCTURED: time=2015-11-08T08:08:50.246090-00 pid=23170
NOTICE:  pg_stop_backup complete, all required WAL segments have been archived
wal_e.main   CRITICAL MSG: An unprocessed exception has avoided all error handling
        DETAIL: Traceback (most recent call last):
          File "/usr/local/lib/python2.7/dist-packages/wal_e/cmd.py", line 572, in main
            pool_size=args.pool_size)
          File "/usr/local/lib/python2.7/dist-packages/wal_e/operator/backup.py", line 197, in database_backup
            **kwargs)
          File "/usr/local/lib/python2.7/dist-packages/wal_e/operator/backup.py", line 502, in _upload_pg_cluster_dir
            pool.put(tpart)
          File "/usr/local/lib/python2.7/dist-packages/wal_e/worker/upload_pool.py", line 108, in put
            self._wait()
          File "/usr/local/lib/python2.7/dist-packages/wal_e/worker/upload_pool.py", line 65, in _wait
            raise val
        SysCallError: (104, 'ECONNRESET')

        STRUCTURED: time=2015-11-08T08:08:52.326030-00 pid=23170

Going by the git-blame, @alex may be the right person to ping to get help/feedback as he seems to be the original author.

@alex, do you have any thoughts on this?

I've never seen this error before, so I can't claim to have any particular insight.


Prep for v0.1.1 release

Summary

Adds entry to change log for v0.1.1

Should #26 be mentioned (type hinting added)


How to use the events? Any example?

Hello,
I didn't find any example on the doc. I want to fire the selected event. What is the syntax please?
Regards

swapnilmishra@ this works fine


Implement multiprocess for reconstructions

Extending the issue solved by #322. It might be ideal in some cases to do multi processed reconstructions at least in the Stokes calculation.
The scatter-gather method here given that I'm just throwing silicon (64 cores) at it we can do 40slices(stokes+bire) in 9 secs with each process taking 0.8s for stokes and .4 for bire vs 111stokes and 7s bire.

I agree. All of the voxel-wise operations in recOrder/waveorder would benefit from multiprocessing.

Closing. See #352.


disabled add new team member functionality

Targets #1624

Description

disable add new team member functionality in inactive and halted project
Checklist:

[x] I have performed a self-review of my own code.

Closing it because no one is working on it.


api: add PreferredAllocator interface

This adds a PreferredAllocator interface so that plugins can
optionally implement the API.
Signed-off-by: Ukri Niemimuukko<EMAIL_ADDRESS>

Codecov Report

Merging #535 (6eaf51f) into master (5d80f2d) will decrease coverage by 0.06%.
The diff coverage is 71.42%.

@@            Coverage Diff             @@
##           master     #535      +/-   ##
==========================================
- Coverage   57.00%   56.93%   -0.07%
==========================================
  Files          31       31
  Lines        2114     2120       +6
==========================================
+ Hits         1205     1207       +2
- Misses        839      841       +2
- Partials       70       72       +2

Impacted Files
Coverage Δ

pkg/deviceplugin/api.go
27.77% <ø> (ø)

pkg/deviceplugin/manager.go
79.59% <50.00%> (-3.02%)
:arrow_down:

pkg/deviceplugin/server.go
82.97% <80.00%> (-1.08%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5d80f2d...6eaf51f. Read the comment docs.


Ninja claimed the wrong player chose a color

I suspect that this is related to notification for the demander getting confused with the person being demanded in the transfer logic.

Link to bug: https://boardgamearena.com/bug?id=93980

This is confirmed.

It looks like the names are reversed for the color selection. (the opposing players show the wrong player making the choice.)

It might be related to the effect being a demand effect and the demander is making the choice and not the main player

public function handleSpecialChoice(int $color): void
{
$this->notifications->notifyColorChoice($color, self::getPlayerId());
self::setAuxiliaryValue($color);
}


Script fails to operate if password contains ampersand

Example - ./telnet-enable2.py <IP_ADDRESS> A0:40:A0:69:B6:30 admin “yourpass”
If “yourpass” contains an “&” then python cannot correctly parse the attributes and throws an error, I.e., for instance, if the password was 12345&67890 then bash would output the following
 -bash: 67890: command not found

Just wrap your password with double quotes so it's treated as a string and that should do it (my pwd had a parenthesis and that solved it for me).

Will add that to the readme, thanks for the hint ;)


Errors while doing upgrade

I get the following error message when I try to setup tables using ECS. The easiest way I found was to do the dispatch database upgrade using a separate container and have the web and scheduler containers run only after that migrate container has finished execution.
Towards doing the database upgrade, I get the following errors:
Traceback (most recent call last):
--
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
return fn()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 364, in connect
return _ConnectionFairy._checkout(self)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
rec = pool._do_get()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/impl.py", line 140, in _do_get
self._dec_overflow()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.raise_(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/impl.py", line 137, in _do_get
return self._create_connection()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
return _ConnectionRecord(self)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 440, in __init__
self.__connect(first_connect_check=True)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
pool.logger.debug("Error on connect(): %s", e)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.raise_(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
connection = pool._invoke_creator(self)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
return dialect.connect(*cargs, **cparams)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 493, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/usr/local/lib/python3.8/site-packages/psycopg2/__init__.py", line 127, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: FATAL:  database "dispatch_user" does not exist
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/bin/dispatch", line 8, in <module>
sys.exit(entrypoint())
File "/usr/local/lib/python3.8/site-packages/dispatch/cli.py", line 569, in entrypoint
dispatch_cli()
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/dispatch/cli.py", line 302, in upgrade_database
if not database_exists(str(config.SQLALCHEMY_DATABASE_URI)):
File "/usr/local/lib/python3.8/site-packages/sqlalchemy_utils/functions/database.py", line 466, in database_exists
return bool(get_scalar_result(engine, text))
File "/usr/local/lib/python3.8/site-packages/sqlalchemy_utils/functions/database.py", line 445, in get_scalar_result
result_proxy = engine.execute(sql)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2234, in execute
connection = self._contextual_connect(close_with_result=True)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2302, in _contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2339, in _wrap_pool_connect
Connection._handle_dbapi_exception_noconnection(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1583, in _handle_dbapi_exception_noconnection
util.raise_(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
return fn()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 364, in connect
return _ConnectionFairy._checkout(self)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
rec = pool._do_get()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/impl.py", line 140, in _do_get
self._dec_overflow()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.raise_(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/impl.py", line 137, in _do_get
return self._create_connection()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
return _ConnectionRecord(self)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 440, in __init__
self.__connect(first_connect_check=True)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
pool.logger.debug("Error on connect(): %s", e)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.raise_(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
connection = pool._invoke_creator(self)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
return dialect.connect(*cargs, **cparams)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 493, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/usr/local/lib/python3.8/site-packages/psycopg2/__init__.py", line 127, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL:  database "dispatch_user" does not exist
(Background on this error at: http://sqlalche.me/e/13/e3q8)

Do you think this is a connection issue or is it a table setup issue? Are you familiar with this error?

Never mind.  Silly password inconsistency mistake.

Issue persists. Not sure why the database username that is passed is being used for the database name?

Is dispatch_user defined in your .env? I don't see it defined anywhere else so I'm not sure where that value is coming from.

dispatch_user is passed is via POSTGRES_USER and also via DATABASE_CREDENTIALS. But for some reason it says database "dispatch_user" doesn't exist

Seems like a case of this. Let me try this and report back.

@kevgliss I believe this is an issue due to this. This issue might have to be moved to the dispatch repository.
As the sqlalchemy-utils 0.36.8 library onwards seems to have this error. There is a pull to fix it here. But until then the fix is to pin the library version SQLAlchemy-Utils==0.36.7 in requirements-base.txt.
Towards that, I was wondering is SQLAlchemy-Utils pulled in transitively? I don't see it in requirements-dev.txt file.
Hope this helps someone else facing the same error. Two days I'll never get back! 😅

I see it's pulled in transitively due in this library. I'll pin it while building our container for now and that should tape the issue for now.

@kevgliss @mvilanova
So I was able to run an ECS container with the fix above just to run the dispatch database upgrade command and I got the following logs:
The upgrade itself was successful, but it had a bunch of endpoint related errors that I wanted to understand. Would you know how to resolve them?

INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running stamp_revision  -> 1c3f808aa00c
Success.

That issue with sqlalchemy-utils is a good catch (we've always had the database and the user match so we've never encountered it).
The second error regarding  ContextualVersionConflict is typically due to some dependency deciding to pin their version and another library requiring a different incompatible version.  In this case it looks like aiohttp is the culprit: https://github.com/aio-libs/aiohttp/pull/5334
In the near term we can pin aiohttp, or attempt to figure out which library is requiring <chardet4.0 and see if they have an upgrade path.

Oh yeah, also wanted to mention, RDS uses dispatch as a keyword and hence can't have a database with that name. ☹
So regarding the ContextualVersionConflict, Is it normal for dispatch.common.utils.cli's install_plugins to run during database upgrade?

Running install_plugins is expected to run during any CLI function. This really only registers the plugins in an internal repository so they can be used. Are you facing issues with it?

@kevgliss @mvilanova  Yes. I was able to pin the version and then dispatch database upgrade worked like a charm, however I'm unable to register a new user. This was the sequence of events.

I downloaded the sample data dump and add the files to the database
I then did the equivalent of docker-compose up -d on ECS which was to spin up the ECS service and that is setup to run the dispatch database upgrade, upto this point everything worked without any errors.
Now I visited the http:///register and I get the following error:

Somehow it seems like the /register endpoint is not setup.

Ah, we recently moved the register url to /auth/register I've updated the install.sh here: https://github.com/Netflix/dispatch-docker/pull/96
Sorry about the confusion!

@kevgliss Ah. That makes sense. It is auth related. I'm happy to report that I was able to get as deep as setting up the data dump and got the endpoint up and running using ECS and RDS and was going to test integrations next.
I did get a few errors while doing the data dump, I'm not sure how critical they are, but I thought I'd let you know that I'm looking into them and any insights from you will be super useful.
psql:./dispatch-sample-data.dump:548: ERROR:  unrecognized configuration parameter "default_table_access_method"

psql:./dispatch-sample-data.dump:2457: ERROR:  invalid input syntax for integer: "Post Incident Review Template"
CONTEXT:  COPY document, line 4, column id: "Post Incident Review Template"

psql:./dispatch-sample-data.dump:4229: ERROR:  insert or update on table "incident_type" violates foreign key constraint "incident_type_template_document_id_fkey"
DETAIL:  Key (template_document_id)=(1) is not present in table "document".

@kevgliss The first one seems to be based on the postgres version so is an easy fix. But the 2nd and 3rd errors seem like they could break something.

@kevgliss Like I suspected. The 1st one doesn't occur on local (macOS) run of the install.sh. But the 2nd and 3rd error happen even on localhost.

I haven't been able to reproduce these errors on the latest example data dump (released today). Closing for now, please create a new issue if you encounter them again.


unable to access device tokens in firebase cloud messaging

I was able to access the device tokens for sending a notification when they are in the main collection. But unable to access them when they are inside a subcollection can anyone help me with that?
it's working fine for a collection

But getting issue when it is in sub collection

Hi @girish4474
From what I can see, the issue is related to a FlutterFire plugin rather than to Flutter itself. Please open the issue in the dedicated repository.
Closing, as this isn't an issue with Flutter itself. If you disagree, please write in the comments, possibly providing a minimal reproducible code sample that does not use flutter fire or other 3rd party plugins, and I will reopen it.
Thank you


Issues loading dataset

I'm having some issues loading in my own training dataset into the various models. I added the directories of the training data into dataset.yaml with the following code:
DATA:
train: /home/users/user/dataset/dataHR/Train/
test: /home/users/user/dataset/dataHR/Val/
train_pair: /home/users/user/dataset/dataLR/Train/
test_pair: /home/users/user/dataset/dataLR/Val/
param:
parser: custom_pairs
Inside of the data the folders have the structure:
0001:
0001_00.tif
0001_01.tif
...
0002:
0002_00.tif
0002_01.tif
...
...
When I run VESPCN in Pytorch the model is able to train but the testing images are not loaded in:
| 2019-05-02 14:18:59 | Epoch: 5/5 | LR: 0.0001 |
100%|#######################################| 200/200 [01:03<00:00,  3.53batch/s, image=00.00097, flow=00.00035, tv=01.90621]
| Epoch average image = 0.001059 |
| Epoch average flow = 0.000375 |
| Epoch average tv = 2.140414 |
2019-05-02 14:20:02,334 WARNING: frames is empty. [size=10]
Test: 0it [00:00, ?it/s]
When I run VESPCN in tensorflow the same thing happens:
INFO:tensorflow:Fitting: VESPCN
| 2019-05-02 14:26:59 | Epoch: 1/10 | LR: 0.0001 |
100%|###################################| 200/200 [00:42<00:00,  4.79batch/s, l2=23615.77148, loss=24443.99414, me=828.22260]
| Epoch average l2 = 35973.230469 |
| Epoch average loss = 36285.953125 |
| Epoch average me = 312.723450 |
frames is empty. [size=10]
Test: 0it [00:00, ?it/s]
When I run FRSVR I get the following errors:
2019-05-02 14:21:51,380 WARNING: frames is empty. [size=3200]
| 2019-05-02 14:21:51 | Epoch: 8/10 | LR: 0.0001 |
0batch [00:00, ?batch/s]
2019-05-02 14:21:51,387 WARNING: frames is empty. [size=10]
Test: 0it [00:00, ?it/s]
Then when I run FRSVR without custom pairs it seems to train but the testing data does not load:
| 2019-05-02 14:44:21 | Epoch: 1/11 | LR: 0.0001 |
100%|#####################| 200/200 [01:23<00:00,  3.14batch/s, total_loss=00.00239, image_loss=00.00236, flow_loss=00.00006]
| Epoch average total_loss = 0.004055 |
| Epoch average image_loss = 0.003931 |
| Epoch average flow_loss = 0.000247 |
2019-05-02 14:45:44,950 WARNING: frames is empty. [size=10]
Any type of advice or fix would be appreciated.

@SandLeg Validation during training should be:
val: ...
val_pair: ...

@SandLeg Validation during training should be:
val: ...
val_pair: ...

Thanks for the reply. That is the way I have it in my file I just did not list it that way in the post. Just to be sure I ran it again with the same format (ie. train, train_pair, val, val_pair,test_test_pair) and I am still getting "WARNING: frames is empty."

@SandLeg Since I'm not able to reproduce your issue, could you fetch the latest 'dev' branch and do the following tests?
# pwd >VSRTorch
python train.py vespcn --cuda --dataset videopair --data_config ../UTest/data/fake_datasets.yml -v
# pwd >Train
python run.py --model vespcn --data_config ../UTest/data/fake_datasets.yml --dataset videopair -v

It seems your test is OK. So come back to the original issue, can it be related to mistake data URL?

I went back into datasets.yaml and edited my URL to make it similar to what you had in fake_datasets.yaml. I used quotes, used the root directory tag which I wasn't using previously and removing the end of URLs. Nothing that I did stopped the "WARNING: frames is empty. [size=10]" error from showing up

Something enlight me! How large is your validation dataset? For now, I assume it's no more than 1GB. It's hardcoded. If your validation data (all files sum up) more than 1GB, it may cause that problem.

I recall that I assume that the validation data is no more than 1GB. If the size of them (all data sums up) is more than 1GB, may cause that problem.
1GB is hardcoded in Trainer.py. You can change it as you wish.

Delete some out-of-order reply

I recall that I assume that the validation data is no more than 1GB. If the size of them (all data sums up) is more than 1GB, may cause that problem.
1GB is hardcoded in Trainer.py. You can change it as you wish.

I recall that I assume that the validation data is no more than 1GB. If the size of them (all data sums up) is more than 1GB, may cause that problem.
1GB is hardcoded in Trainer.py. You can change it as you wish.
The last line of tensorflow's execution is expected.

Yeah my training and validation set is fairly large. The validation set has around 200 images and comes in about 4.5 GB. The training set has over 400 images so it is probably much larger than that.

For training, you can use --memory_limit=???GB to constrain memory usage during training.
For validation, I thought too many files could slow down the overall speed. If you have enough runtime memory, you could raise that value as I mentioned.

For training, you can use --memory_limit=???GB to constrain memory usage during training.
For validation, I thought too many files could slow down the overall speed. If you have enough runtime memory, you could raise that value as I mentioned.

Yeah, that still did not do anything. So I tried running CARN and got a fatal file error with a specific line in the code. Maybe this will help you diagnose the problem:
2019-05-08 12:35:18,164 INFO: Fitting: [CARN]
2019-05-08 12:35:18,175 WARNING: frames is empty. [size=3200]
| 2019-05-08 12:35:18 | Epoch: 1/13 | LR: 0.0001 |
0batch [00:00, ?batch/s]
Exception in thread fetch_thread_7:
Traceback (most recent call last):
File "/user/miniconda3/envs/py36/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/user/miniconda3/envs/py36/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "/user/VideoSuperResolution/VSR/DataLoader/Loader.py", line 261, in _prefetch
frames = self.parser[index * interval:]
File "/user/VideoSuperResolution/VSR/DataLoader/Parser/custom_pairs.py", line 43, in getitem
ret += self.gen_frames(copy.deepcopy(vf), vf.frames)
File "/user/VideoSuperResolution/VSR/DataLoader/Parser/custom_pairs.py", line 72, in gen_frames
lr = [img for img in vf.pair.read_frame(depth)]
File "/user/VideoSuperResolution/VSR/DataLoader/VirtualFile.py", line 352, in read_frame
image_bytes = [BytesIO(self.read()) for _ in range(frames)]
File "/user/VideoSuperResolution/VSR/DataLoader/VirtualFile.py", line 352, in
image_bytes = [BytesIO(self.read()) for _ in range(frames)]
File "/user/VideoSuperResolution/VSR/DataLoader/VirtualFile.py", line 126, in read
raise FileNotFoundError('No frames in File')
FileNotFoundError: No frames in File
Test: 100%|###############################################################| 10/10 [00:00<00:00, 30.97it/s]
psnr: 11.185015,

Still not able to reproduce the problem. Have you pulled to the latest master or dev branch? What's your data exactly, I notice they are TIFF, can you send me a sample image?

custom_pairs parer needs to support video well. In developing...

Still not able to reproduce the problem. Have you pulled to the latest master or dev branch? What's your data exactly, I notice they are TIFF, can you send me a sample image?

I started on the master branch but I have been on the dev branch since you told me to run the tests. I'm not sure if I am able to share the images directly but they come from here http://xviewdataset.org/ . The frames come from a single image that is downsampled x amount of times using a convolutional filter. So the video would look like a sliding window through the original image if you looked at them in order. The data that I am trying to test only has four frames but I need to be able to work to up to 64 frames for other data that I have yet to create.

@SandLeg Fix related issue. Try at my latest dev branch, and tell me anything goes wrong.

@SandLeg Fix related issue. Try at my latest dev branch, and tell me anything goes wrong.
It's caused by a memory leak. You will need to use --memory_limit to avoid consuming too much memory.
As for hardcoded 1GB for validation, you don't need to change that.

I figured out part of the issue but I am still not getting it to run. The last folder in my validation and training set was empty which is probably why I was getting the "No File in Frame" issue. Now all the models hang up at the same point. I get the "WARNING: frames is empty. [size=3200]" message then it hangs for a couple of seconds and runs the test which was never done before.
Do you have any suggestions on dealing with the memory issue? Should I try to decrease the size of my dataset and see if that will get it to run?

That's odd. You can try to start with a subset of your dataset, for example, 10 videos with 10 frames each.

I tried it with 10 folders with 4 frames each and it's still doing the same thing. Something must not be loading right. I really don't know. I'll try some more investigation on Monday.

Much beyond my expectation. What if using another dataset? For example, let's use VID4 to train.

Much beyond my expectation. What if using another dataset? For example, let's use VID4 to train.

Sorry for the late reply but I have been busy and I am just getting back to this today. I fixed the issue. My path in dataset.yaml was wrong, which combined with the previous issue with the missing data is probably the main source of the issue. Thanks for all of your help.
And while your reading this do you know how to configure training for multiple GPUs?

For the existing models, they don't support multi-GPUs. But if you're familiar with multi-GPU programming, you can code as usual.
The entire training process is encapsulated in Train function, so you can manipulate your training procedure inside it. Moreover, you can overwrite Trainer to control the data path as you like.
Close this thread since the problem is solved.

Tacking on to this, .tif files support 32 bit float values. In the case that the training set was composed of float arrays, rather than image-like integer arrays, .tif seems like a good choice. Would your code need substantial modifications to run with 32 bit float .tif files? Changing the .png output, mentioned in #57, would be one issue, I suspect.


[Footer]: Responsiveness on container width

Description
Footer does not adjust on the parent container width and it breaks out. Expecting it to switch to mobile version when the space does not allow it to fit.
It can be tested on Storybook: https://www.ibm.com/standards/carbon/web-components/?path=/story/components-footer--micro-language-only
Add width to a parent container to less than 910px.

This gets worse in other languages (e.g. German) that have longer text for links.

Using as web component: "@carbon/ibmdotcom-web-components": "^v2.2.0-rc.1"
Component(s) impacted
Footer
https://www.ibm.com/standards/carbon/web-components/?path=/story/components-footer--micro-language-only
Browser
No response
Carbon for IBM.com version
v2.2.0-rc.1
Severity
Severity 3 = The problem is visible or noticeable to users but does not impede the usability or functionality. Affects minor functionality, has a workaround.
Application/website
IBM Documentation
Package
@carbon/ibmdotcom-web-components
CodeSandbox example
https://www.ibm.com/standards/carbon/web-components/?path=/story/components-footer--micro-language-only
Steps to reproduce the issue (if applicable)

Open https://www.ibm.com/standards/carbon/web-components/?path=/story/components-footer--micro-language-only
Change the footer's parent container to something like 620px

Release date (if applicable)
(asap, we are waiting on masthead fixes as well)
Code of Conduct

[X] I agree to follow this project's Code of Conduct
[X] I checked the current issues for duplicate issues

Hi @ndtx, can you give some more context why you are adding a width to the parent container? We usually do not recommend adding specific styles, since it could go against the designs

Hi, maybe you could provide some more details? Looking at those websites I am not seeing the issue you mentioned above and trying it on storybook

I also notice in your screenshots you have two "Accessibility" links. It seems like there is some customization going on in these footers which could be causing the errors you are seeing.

Trying it out in a stackblitz example I do see the small error of the line going slightly above: https://stackblitz.com/edit/github-nbzfxs-hgwwxr?file=package.json,index.html

but in your screenshot it seems to be breaking where you add extra code for the "IBM Documentation Help". I would suggest joining the design office hours and discussing this change with our design team. The micro footer does not currently support the adjunct link. You could also try using the "short" footer instead, which does support the adjunct link

ahh okay i see where it's breaking now. Thank you! i

We went round and round with you guys in 2020 on why we need the footer as it is. Asking us to change it with this version changes some of our user interactions:

Our left nav closes and opens - is there an impact?
we have scroll to the bottom where we have PDF and local version downloads? How will this affect that?
most of our left navs require scrolling. Having the black bar across the entire bottom gives customer the idea that they are see the entire left nav and does not account for the fact that our left nav scrolls because we have a ton of content.

Our side nav iincludes content from other tech content types
Again, i understand what you are asking but we had specific use cases we ran by you so now  that you expected so trying to understand why you are now changing that position?
We will have to test all of our scenarios to comply

Hi @kcbuchanan
Our team has had people leave or change roles since 2020, so most of us did not have the background knowledge of your discussions from 2020. Thank you for sharing the additional context that was not included in the original issue, as that always helps set everyone’s understanding of the product or experience on the same level.
The Dotcom components were designed to respond to the overall browser width, rather than a container like on the Docs site. I was able to chat with some members from my team that remember your use case and that we allowed your team to customize the footer on your own to work within this space.
With any new version release there are going to be breaking changes and we have tried our best to minimize those for our adopters with Carbon for IBM.com v2, and we are only supporting Web Components this year, which locks down styles for more consistency across experiences.
All that being said, we do want to help reach your user requirements as best we can, so here are the options:

Our team can look into adding some additional styles to the Footer to accommodate the container layout of the Docs experience. I can share some examples of what we would anticipate the footer to look like beforehand. We release every 2 weeks, so once we reach a design decision it would likely release sometime in mid or end of March.
Your team could create a customized footer that works exactly as you need it to utilizing the translation files. I would love if it could match our v2 designs as closely as possible, while behaving in a way that works with your experience.
Continue utilizing the customized footer that you currently have that is working in your experience. I believe that our v1 and v2 packages should work together, and yours is a customized component already.

Again, thank you for sharing the additional context as it helps our team better understand your user experience, especially when we haven’t chatted in a while! 🙂

@kcbuchanan @ndtx here are some examples of what I would expect Option 1 that I outlined in the above comment to look like. These are rough mock ups to show:

The Download PDF pinned to the bottom of the left navigation/table of contents
How the footer would sit between the left nav/table of contents and the right side panel

Please note the footer will still be at the bottom of the page when the user scrolls, this is just to show the relationship between the panel layouts and demo that the English and German translation can fit at different widths

Breakpoint Examples
English
German

Wide example

Narrowest with right side panel

Almost mobile, when right rail side panel collapses


Support with Local LLM Models

Hi, Came Across TaskGen and strictjson
Particularly with strictjson, is it possible to use Local LLM Models via a local llm provider like Ollama ?
If it is feasible, then would love to give it a shot. Plus, I do not see any example snippet around it as well, so would love to contribute that as well.

Definitely, you can simply assign your own llm function (which includes local llms) with the llm parameter.
Check out the Tutorial for more information!

For those stumbling upon this issue, you can start with something as easy as:
import ollama
from strictjson import strict_json

def llm(system_prompt: str, user_prompt: str) -> str:
    response = ollama.chat(model="llama3.2:1b", messages=[
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": user_prompt}
    ])
    return response['message']['content']

res = strict_json(system_prompt='You are a classifier',
                  user_prompt='It is a beautiful and sunny day',
                  output_format={'Sentiment': 'Type of Sentiment',
                                 'Adjectives': 'Array of adjectives',
                                 'Words': 'Number of words'},
                  llm=llm)

print(res)