Synchronization mutex or semaphore is not working properly

Summary
What happened/what you expected to happen?
Synchronization mutex or semaphore is not working when trying to prevent running more that one workflow in the same namespace.
Maybe related to #6110?
First tried with:
semaphore:
 configMapKeyRef:
   name: ci-config
   key: workflow

and
apiVersion: v1
kind: ConfigMap
metadata:
  name: ci-config
data:
  workflow: "1"

I triggered 3 workflows from a argo-events webhook, at first it looked like it worked because only one was running and the other two were "waiting". After a while once the first workflow had finished I noticed that the other 2 workflows were both running.

Then I tried with:
synchronization:
  mutex:
    name:  test

I triggered 4 workflows and got the same issue

I've been trying to reproduce since then but no luck, the only difference is that now the workflow takes 6 minutes to run instead of 15m and there are no errors in any step. Before I had errors in one step handled with:
continueOn:
  failed: true

Diagnostics
workflow-controller logs
time="2021-07-13T12:07:19Z" level=info msg="index config" indexWorkflowSemaphoreKeys=true
time="2021-07-13T12:07:19Z" level=info msg="cron config" cronSyncPeriod=10s
time="2021-07-13T12:07:19.540Z" level=info msg="config map" name=workflow-controller-configmap
time="2021-07-13T12:07:19.550Z" level=info msg="Get configmaps 200"
time="2021-07-13T12:07:19.557Z" level=info msg="Configuration:\nartifactRepository: {}\ninitialDelay: 0s\nmetricsConfig: {}\nnodeEvents: {}\npodSpecLogStrategy: {}\ntelemetryConfig: {}\n"
time="2021-07-13T12:07:19.557Z" level=info msg="Persistence configuration disabled"
time="2021-07-13T12:07:19.558Z" level=info msg="Starting Workflow Controller" version=v3.1.1
time="2021-07-13T12:07:19.558Z" level=info msg="Workers: workflow: 32, pod: 32, pod cleanup: 4"
time="2021-07-13T12:07:19.561Z" level=info msg="List configmaps 200"
time="2021-07-13T12:07:19.563Z" level=info msg="Watch configmaps 200"
time="2021-07-13T12:07:19.563Z" level=info msg="List workflowtemplates 200"
time="2021-07-13T12:07:19.564Z" level=info msg="List pods 200"
time="2021-07-13T12:07:19.564Z" level=info msg="Watch configmaps 200"
time="2021-07-13T12:07:19.566Z" level=info msg="Watch workflowtemplates 200"
time="2021-07-13T12:07:19.566Z" level=info msg="List workflows 200"
time="2021-07-13T12:07:19.583Z" level=info msg="Watch workflows 200"
time="2021-07-13T12:07:19.586Z" level=info msg="Watch pods 200"
time="2021-07-13T12:07:19.661Z" level=info msg="Create selfsubjectaccessreviews 201"
time="2021-07-13T12:07:19.663Z" level=info msg="Create selfsubjectaccessreviews 201"
time="2021-07-13T12:07:19.664Z" level=info msg="Create selfsubjectaccessreviews 201"
time="2021-07-13T12:07:19.666Z" level=info msg="List clusterworkflowtemplates 200"
time="2021-07-13T12:07:19.668Z" level=info msg="List workflows 200"
time="2021-07-13T12:07:19.669Z" level=info msg="Watch clusterworkflowtemplates 200"
time="2021-07-13T12:07:19.681Z" level=info msg="Manager initialized successfully"
I0713 12:07:19.682229       1 leaderelection.go:243] attempting to acquire leader lease  argo/workflow-controller...
time="2021-07-13T12:07:19.682Z" level=info msg="Starting prometheus metrics server at localhost:9090/metrics"
time="2021-07-13T12:07:19.684Z" level=info msg="Get leases 200"
time="2021-07-13T12:07:19.715Z" level=info msg="Update leases 200"
I0713 12:07:19.715393       1 leaderelection.go:253] successfully acquired lease argo/workflow-controller
time="2021-07-13T12:07:19.715Z" level=info msg="new leader" id=workflow-controller-5487447d-jvw48 leader=workflow-controller-5487447d-jvw48
time="2021-07-13T12:07:19.715Z" level=info msg="started leading" id=workflow-controller-5487447d-jvw48
time="2021-07-13T12:07:19.716Z" level=info msg="Persistence disabled - so archived workflow GC disabled - you must restart the controller if you enable this"
time="2021-07-13T12:07:19.716Z" level=info msg="Performing periodic GC every 5m0s"
time="2021-07-13T12:07:19.716Z" level=info msg="Starting workflow TTL controller (workflowTTLWorkers 4)"
time="2021-07-13T12:07:19.716Z" level=info msg="Started workflow TTL worker"
time="2021-07-13T12:07:19.718Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-sptdr
time="2021-07-13T12:07:19.718Z" level=info msg="Starting CronWorkflow controller"
time="2021-07-13T12:07:19.719Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-84pqf
time="2021-07-13T12:07:19.720Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:19.720Z" level=info msg="List cronworkflows 200"
time="2021-07-13T12:07:19.720Z" level=info msg="List workflows 200"
time="2021-07-13T12:07:19.721Z" level=info msg="Watch cronworkflows 200"
time="2021-07-13T12:07:19.722Z" level=info msg="Watch workflows 200"
time="2021-07-13T12:07:19.722Z" level=info msg="List workflows 200"
time="2021-07-13T12:07:19.731Z" level=info msg="Get leases 200"
time="2021-07-13T12:07:19.731Z" level=info msg="Watch workflows 200"
time="2021-07-13T12:07:19.755Z" level=info msg="Create events 201"
time="2021-07-13T12:07:19.762Z" level=info msg="Get configmaps 200"
time="2021-07-13T12:07:19.763Z" level=info msg="Get configmaps 200"
time="2021-07-13T12:07:19.763Z" level=info msg="Get configmaps 200"
time="2021-07-13T12:07:19.763Z" level=info msg="remote-tests-runner/Mutex/test acquired by remote-tests-runner/ci-lzlhr " mutex=remote-tests-runner/Mutex/test
time="2021-07-13T12:07:19.763Z" level=info msg="Pod failed: Error (exit code 1)" displayName=check-and-tests namespace=remote-tests-runner pod=ci-lzlhr-414183890 templateName=run-client-checks workflow=ci-lzlhr
time="2021-07-13T12:07:19.763Z" level=info msg="Updating node ci-lzlhr-414183890 exit code 1" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:19.763Z" level=info msg="Updating node ci-lzlhr-414183890 status Running -> Failed" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:19.763Z" level=info msg="Updating node ci-lzlhr-414183890 message: Error (exit code 1)" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:19.763Z" level=warning msg="Workflow processing has been postponed due to concurrency limit" namespace=remote-tests-runner workflow=ci-sptdr
time="2021-07-13T12:07:19.763Z" level=warning msg="Workflow processing has been postponed due to concurrency limit" namespace=remote-tests-runner workflow=ci-84pqf
time="2021-07-13T12:07:19.764Z" level=info msg="Update leases 200"
time="2021-07-13T12:07:19.764Z" level=info msg="SG Outbound nodes of ci-lzlhr-2997997659 are [ci-lzlhr-2997997659]" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:19.764Z" level=info msg="SG Outbound nodes of ci-lzlhr-2816619268 are [ci-lzlhr-2816619268]" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:19.765Z" level=info msg="Step group node ci-lzlhr-2057234380 successful" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:19.765Z" level=info msg="node ci-lzlhr-2057234380 phase Running -> Succeeded" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:19.765Z" level=info msg="node ci-lzlhr-2057234380 finished: 2021-07-13 12:07:19.765156183 +0000 UTC" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:19.765Z" level=info msg="StepGroup node ci-lzlhr-2124197761 initialized Running" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:19.765Z" level=info msg="SG Outbound nodes of ci-lzlhr-414183890 are [ci-lzlhr-414183890]" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:19.766Z" level=info msg="Skipping ci-lzlhr[3].set-status-success: when '1 == 0' evaluated false" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:19.766Z" level=info msg="Skipped node ci-lzlhr-723745886 initialized Skipped (message: when '1 == 0' evaluated false)" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:19.766Z" level=info msg="Pod node ci-lzlhr-1343933299 initialized Pending" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:19.785Z" level=info msg="Update workflows 200"
time="2021-07-13T12:07:19.789Z" level=info msg="Create pods 201"
time="2021-07-13T12:07:19.802Z" level=info msg="Workflow update successful" namespace=remote-tests-runner phase= resourceVersion=102972631 workflow=ci-84pqf
time="2021-07-13T12:07:19.806Z" level=info msg="Created pod: ci-lzlhr[3].set-status-failure (ci-lzlhr-1343933299)" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:19.806Z" level=info msg="Workflow step group node ci-lzlhr-2124197761 not yet completed" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:19.817Z" level=info msg="Update workflows 200"
time="2021-07-13T12:07:19.819Z" level=info msg="Workflow update successful" namespace=remote-tests-runner phase=Running resourceVersion=102972635 workflow=ci-lzlhr
time="2021-07-13T12:07:19.826Z" level=info msg="Create events 201"
time="2021-07-13T12:07:19.829Z" level=info msg="cleaning up pod" action=labelPodCompleted key=remote-tests-runner/ci-lzlhr-414183890/labelPodCompleted
time="2021-07-13T12:07:19.831Z" level=info msg="Create events 201"
time="2021-07-13T12:07:19.835Z" level=info msg="Create events 201"
time="2021-07-13T12:07:19.839Z" level=info msg="Patch pods 200"
time="2021-07-13T12:07:24.768Z" level=info msg="Get leases 200"
time="2021-07-13T12:07:24.787Z" level=info msg="Update leases 200"
time="2021-07-13T12:07:29.718Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:29.718Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-sptdr
time="2021-07-13T12:07:29.722Z" level=info msg="Get configmaps 200"
time="2021-07-13T12:07:29.722Z" level=info msg="Get configmaps 200"
time="2021-07-13T12:07:29.723Z" level=warning msg="Workflow processing has been postponed due to concurrency limit" namespace=remote-tests-runner workflow=ci-sptdr
time="2021-07-13T12:07:29.723Z" level=info msg="Updating node ci-lzlhr-1343933299 exit code 0" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:29.723Z" level=info msg="Updating node ci-lzlhr-1343933299 status Pending -> Succeeded" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:29.724Z" level=info msg="SG Outbound nodes of ci-lzlhr-2997997659 are [ci-lzlhr-2997997659]" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:29.724Z" level=info msg="SG Outbound nodes of ci-lzlhr-2816619268 are [ci-lzlhr-2816619268]" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:29.724Z" level=info msg="SG Outbound nodes of ci-lzlhr-414183890 are [ci-lzlhr-414183890]" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:29.724Z" level=info msg="Step group node ci-lzlhr-2124197761 successful" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:29.724Z" level=info msg="node ci-lzlhr-2124197761 phase Running -> Succeeded" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:29.724Z" level=info msg="node ci-lzlhr-2124197761 finished: 2021-07-13 12:07:29.72488628 +0000 UTC" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:29.724Z" level=info msg="StepGroup node ci-lzlhr-3198406662 initialized Running" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:29.724Z" level=info msg="SG Outbound nodes of ci-lzlhr-723745886 are [ci-lzlhr-723745886]" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:29.725Z" level=info msg="SG Outbound nodes of ci-lzlhr-1343933299 are [ci-lzlhr-1343933299]" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:29.725Z" level=info msg="Skipping ci-lzlhr[4].slack-notify-success: when '1 == 0' evaluated false" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:29.725Z" level=info msg="Skipped node ci-lzlhr-1247890768 initialized Skipped (message: when '1 == 0' evaluated false)" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:29.725Z" level=info msg="Pod node ci-lzlhr-957369457 initialized Pending" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:29.732Z" level=info msg="Create pods 201"
time="2021-07-13T12:07:29.735Z" level=info msg="Created pod: ci-lzlhr[4].slack-notify-failure (ci-lzlhr-957369457)" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:29.735Z" level=info msg="Workflow step group node ci-lzlhr-3198406662 not yet completed" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:29.764Z" level=info msg="Update workflows 200"
time="2021-07-13T12:07:29.765Z" level=info msg="Workflow update successful" namespace=remote-tests-runner phase=Running resourceVersion=102972706 workflow=ci-lzlhr
time="2021-07-13T12:07:29.770Z" level=info msg="Create events 201"
time="2021-07-13T12:07:29.775Z" level=info msg="Create events 201"
time="2021-07-13T12:07:29.778Z" level=info msg="cleaning up pod" action=labelPodCompleted key=remote-tests-runner/ci-lzlhr-1343933299/labelPodCompleted
time="2021-07-13T12:07:29.783Z" level=info msg="Create events 201"
time="2021-07-13T12:07:29.785Z" level=info msg="Patch pods 200"
time="2021-07-13T12:07:29.794Z" level=info msg="Get leases 200"
time="2021-07-13T12:07:29.795Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-84pqf
time="2021-07-13T12:07:29.795Z" level=info msg="Create events 201"
time="2021-07-13T12:07:29.799Z" level=info msg="Update leases 200"
time="2021-07-13T12:07:29.803Z" level=info msg="Get configmaps 200"
time="2021-07-13T12:07:29.804Z" level=warning msg="Workflow processing has been postponed due to concurrency limit" namespace=remote-tests-runner workflow=ci-84pqf
time="2021-07-13T12:07:34.803Z" level=info msg="Get leases 200"
time="2021-07-13T12:07:34.811Z" level=info msg="Update leases 200"
time="2021-07-13T12:07:39.736Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:39.741Z" level=info msg="Get configmaps 200"
time="2021-07-13T12:07:39.741Z" level=info msg="Updating node ci-lzlhr-957369457 exit code 0" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:39.741Z" level=info msg="Updating node ci-lzlhr-957369457 status Pending -> Succeeded" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:39.742Z" level=info msg="SG Outbound nodes of ci-lzlhr-2997997659 are [ci-lzlhr-2997997659]" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:39.742Z" level=info msg="SG Outbound nodes of ci-lzlhr-2816619268 are [ci-lzlhr-2816619268]" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:39.742Z" level=info msg="SG Outbound nodes of ci-lzlhr-414183890 are [ci-lzlhr-414183890]" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:39.742Z" level=info msg="SG Outbound nodes of ci-lzlhr-723745886 are [ci-lzlhr-723745886]" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:39.742Z" level=info msg="SG Outbound nodes of ci-lzlhr-1343933299 are [ci-lzlhr-1343933299]" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:39.743Z" level=info msg="Step group node ci-lzlhr-3198406662 successful" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:39.743Z" level=info msg="node ci-lzlhr-3198406662 phase Running -> Succeeded" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:39.743Z" level=info msg="node ci-lzlhr-3198406662 finished: 2021-07-13 12:07:39.743501395 +0000 UTC" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:39.743Z" level=info msg="Outbound nodes of ci-lzlhr-1247890768 is [ci-lzlhr-1247890768]" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:39.743Z" level=info msg="Outbound nodes of ci-lzlhr-957369457 is [ci-lzlhr-957369457]" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:39.743Z" level=info msg="Outbound nodes of ci-lzlhr is [ci-lzlhr-1247890768 ci-lzlhr-957369457]" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:39.743Z" level=info msg="node ci-lzlhr phase Running -> Succeeded" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:39.743Z" level=info msg="node ci-lzlhr finished: 2021-07-13 12:07:39.743720569 +0000 UTC" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:39.743Z" level=info msg="Checking daemoned children of ci-lzlhr" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:39.743Z" level=info msg="Updated phase Running -> Succeeded" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:39.743Z" level=info msg="Updated message Waiting for remote-tests-runner/Mutex/test lock. Lock status: 0/1  -> " namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:39.743Z" level=info msg="Marking workflow completed" namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:39.743Z" level=info msg="Checking daemoned children of " namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:39.744Z" level=info msg="Lock has been released by remote-tests-runner/ci-lzlhr. Available locks: 1" mutex=remote-tests-runner/Mutex/test
time="2021-07-13T12:07:39.744Z" level=info msg="remote-tests-runner/ci-lzlhr released a lock from remote-tests-runner/Mutex/test"
time="2021-07-13T12:07:39.744Z" level=info msg="Released all acquired locks" key=ci-lzlhr namespace=remote-tests-runner workflow=ci-lzlhr
time="2021-07-13T12:07:39.747Z" level=info msg="Create events 201"
time="2021-07-13T12:07:39.755Z" level=info msg="Update workflows 200"
time="2021-07-13T12:07:39.756Z" level=info msg="Workflow update successful" namespace=remote-tests-runner phase=Succeeded resourceVersion=102972782 workflow=ci-lzlhr
time="2021-07-13T12:07:39.760Z" level=info msg="Create events 201"
time="2021-07-13T12:07:39.763Z" level=info msg="Create events 201"
time="2021-07-13T12:07:39.763Z" level=info msg="remote-tests-runner/ci-lzlhr released a lock from remote-tests-runner/Mutex/test"
time="2021-07-13T12:07:39.764Z" level=info msg="Queueing Succeeded workflow remote-tests-runner/ci-lzlhr for delete in 5m0s"
time="2021-07-13T12:07:39.765Z" level=info msg="cleaning up pod" action=labelPodCompleted key=remote-tests-runner/ci-lzlhr-957369457/labelPodCompleted
time="2021-07-13T12:07:39.766Z" level=info msg="Queueing Succeeded workflow remote-tests-runner/ci-lzlhr for delete in 5m0s"
time="2021-07-13T12:07:39.767Z" level=info msg="Create events 201"
time="2021-07-13T12:07:39.771Z" level=info msg="Create events 201"
time="2021-07-13T12:07:39.776Z" level=info msg="Patch pods 200"
time="2021-07-13T12:07:39.815Z" level=info msg="Get leases 200"
time="2021-07-13T12:07:39.820Z" level=info msg="Update leases 200"
time="2021-07-13T12:07:44.826Z" level=info msg="Get leases 200"
time="2021-07-13T12:07:44.833Z" level=info msg="Update leases 200"
time="2021-07-13T12:07:49.745Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-84pqf
time="2021-07-13T12:07:49.772Z" level=info msg="Get configmaps 200"
time="2021-07-13T12:07:49.773Z" level=info msg="remote-tests-runner/Mutex/test acquired by remote-tests-runner/ci-84pqf " mutex=remote-tests-runner/Mutex/test
time="2021-07-13T12:07:49.773Z" level=info msg="Updated phase  -> Running" namespace=remote-tests-runner workflow=ci-84pqf
time="2021-07-13T12:07:49.773Z" level=info msg="Updated message Waiting for remote-tests-runner/Mutex/test lock.
Self hosted
What version of Argo Workflows are you running?
v3.1.1
What executor are you running? Docker/K8SAPI/Kubelet/PNS/Emissary
(default: docker)
Did this work in a previous version? I.e. is it a regression?
Didn't test in the previous version

Message from the maintainers:
Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

time="2021-07-13T12:07:39.763Z" level=info msg="remote-tests-runner/ci-lzlhr released a lock from remote-tests-runner/Mutex/test"
time="2021-07-13T12:07:39.764Z" level=info msg="Queueing Succeeded workflow remote-tests-runner/ci-lzlhr for delete in 5m0s"
time="2021-07-13T12:07:39.765Z" level=info msg="cleaning up pod" action=labelPodCompleted key=remote-tests-runner/ci-lzlhr-957369457/labelPodCompleted
time="2021-07-13T12:07:39.766Z" level=info msg="Queueing Succeeded workflow remote-tests-runner/ci-lzlhr for delete in 5m0s"
time="2021-07-13T12:07:39.767Z" level=info msg="Create events 201"
time="2021-07-13T12:07:39.771Z" level=info msg="Create events 201"
time="2021-07-13T12:07:39.776Z" level=info msg="Patch pods 200"
time="2021-07-13T12:07:39.815Z" level=info msg="Get leases 200"
time="2021-07-13T12:07:39.820Z" level=info msg="Update leases 200"
time="2021-07-13T12:07:44.826Z" level=info msg="Get leases 200"
time="2021-07-13T12:07:44.833Z" level=info msg="Update leases 200"
time="2021-07-13T12:07:49.745Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-84pqf
time="2021-07-13T12:07:49.772Z" level=info msg="Get configmaps 200"
time="2021-07-13T12:07:49.773Z" level=info msg="remote-tests-runner/Mutex/test acquired by remote-tests-runner/ci-84pqf " mutex=remote-tests-runner/Mutex/test
time="2021-07-13T12:07:49.773Z" level=info msg="Updated phase  -> Running" namespace=remote-tests-runner workflow=ci-84pqf

I can see ci-lzlhr releasing and within 10s ci-84pqf is acquired.
Can you provide the full from workflow start to end for ci-lzlhr?

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: ci-
  namespace: remote-tests-runner
spec:
  entrypoint: ci-client
  serviceAccountName: remote-tests-runner-sa
  synchronization:
    mutex:
      name:  test
  ttlStrategy:
    secondsAfterSuccess: 300
  imagePullSecrets:
  - name: docker-registry-credentials
  arguments:
    parameters:
    - name: repo_owner
      value: jbaez
    - name: repo_name
      value: repo-test
    - name: commit_sha
      value: ""
  templates:
    - name: ci-client
      steps:
      - - name: set-status-pending
          template: status-change
          arguments:
            parameters:
            - name: description
              value: check started
            - name: state
              value: pending
      - - name: get-client-src
          template: setup-source
          arguments:
            artifacts:
            - name: repo-src
              git:
                repo: 'git@git.repo.com:{{workflow.parameters.repo_owner}}/{{workflow.parameters.repo_name}}.git'
                revision: "{{workflow.parameters.commit_sha}}"
                depth: 4
                sshPrivateKeySecret:
                  name: git-ssh
                  key: id_rsa
                insecureIgnoreHostKey: true
      - - name: check-and-tests
          template: run-client-checks
          arguments:
            artifacts:
              - name: client-src
                from: "{{steps.get-client-src.outputs.artifacts.src}}"
          continueOn:
            failed: true
      - - name: set-status-success
          template: status-change
          arguments:
            parameters:
            - name: description
              value: check finshed
            - name: state
              value: "success"
          when: "{{steps.check-and-tests.exitCode}} == 0"
        - name: set-status-failure
          template: status-change
          arguments:
            parameters:
            - name: description
              value: check finshed
            - name: state
              value: "failure"
          when: "{{steps.check-and-tests.exitCode}} != 0"
      - - name: slack-notify-success
          template: slack-message
          arguments:
            parameters:
            - name: message
              value: CI - JS checks passed
          when: "{{steps.check-and-tests.exitCode}} == 0"
        - name: slack-notify-failure
          template: slack-message
          arguments:
            parameters:
            - name: message
              value: CI - JS checks failed
          when: "{{steps.check-and-tests.exitCode}} != 0"
    - name: setup-source
      container:
        image: 'registry.repo.com/client-tools:1.0.5'
        command:
        - sh
        - '-c'
        args:
        - npm install
        workingDir: /client/src/repo
      inputs:
        artifacts:
          - name: repo-src
            path: /client
      outputs:
        artifacts:
          - name: src
            path: /client/src/repo

    - name: run-client-checks
      inputs:
        artifacts:
        - name: client-src
          path: /client/src/repo
      metadata:
        labels:
          app: karma
      container:
        image: 'registry.repo.com/client-tools:1.0.5'
        command:
        - sh
        - '-c'
        args:
        -  npx gulp check-code && npx gulp test --ci=true
        workingDir: /client/src/repo

    - name: status-change
      inputs:
        parameters:
        - name: description
        - name: state
      container:
        image: 'registry.repo.com/client-tools:1.0.5'
        command:
          - sh
          - '-c'
        args:
        - |
          #redacted API stuff to change status
          http_code=201

          echo "HTTP response code: $http_code"
          if [ ${http_code} -eq 201 ]
          then
            exit 0
          else
            exit 1
          fi
        env:
          - name: ACCESS_TOKEN
            valueFrom:
              secretKeyRef:
                name: git-access
                key: token

    - name: slack-message
      inputs:
        parameters:
        - name: message
      container:
        image: 'registry.repo.com/client-tools:1.0.5'
        command:
          - sh
          - '-c'
        args:
        - >
          curl -X POST -d '{ "text": "{{inputs.parameters.message}}" }'
          "https://hooks.slack.com/services/$TEAM/$SERVICE/$TOKEN"
        env:
          - name: TEAM
            valueFrom:
              secretKeyRef:
                name: slack-api
                key: team
          - name: SERVICE
            valueFrom:
              secretKeyRef:
                name: slack-api
                key: service
          - name: TOKEN
            valueFrom:
              secretKeyRef:
                name: slack-api
                key: token

Sure, here is the manifest, I've changed URLs and redacted some private info.

I just had the issue again, and found something interesting. When I took the screenshot (09:06:46 GMT) ci-544g4 was running for 5 minutes already and the logs say at 09:05:19 Workflow processing has been postponed due to concurrency limit. (when I downloaded the logs they start at 09:05:19, so can't see what happened 5 minutes before)

I downloaded the logs again when there was only one worflow running and at 09:19:31 it says remote-tests-runner/Mutex/test acquired by remote-tests-runner/ci-544g4 🤔when it was running for a long time already.
workfow-controller logs:
First part:
time="2021-07-14T09:05:19Z" level=info msg="index config" indexWorkflowSemaphoreKeys=true
time="2021-07-14T09:05:19Z" level=info msg="cron config" cronSyncPeriod=10s
time="2021-07-14T09:05:19.473Z" level=info msg="config map" name=workflow-controller-configmap
time="2021-07-14T09:05:19.488Z" level=info msg="Get configmaps 200"
time="2021-07-14T09:05:19.493Z" level=info msg="Configuration:\nartifactRepository: {}\ninitialDelay: 0s\nmetricsConfig: {}\nnodeEvents: {}\npodSpecLogStrategy: {}\ntelemetryConfig: {}\n"
time="2021-07-14T09:05:19.493Z" level=info msg="Persistence configuration disabled"
time="2021-07-14T09:05:19.494Z" level=info msg="Starting Workflow Controller" version=v3.1.1
time="2021-07-14T09:05:19.494Z" level=info msg="Workers: workflow: 32, pod: 32, pod cleanup: 4"
time="2021-07-14T09:05:19.497Z" level=info msg="Watch configmaps 200"
time="2021-07-14T09:05:19.500Z" level=info msg="List pods 200"
time="2021-07-14T09:05:19.500Z" level=info msg="List configmaps 200"
time="2021-07-14T09:05:19.500Z" level=info msg="List workflowtemplates 200"
time="2021-07-14T09:05:19.506Z" level=info msg="Watch configmaps 200"
time="2021-07-14T09:05:19.506Z" level=info msg="List workflows 200"
time="2021-07-14T09:05:19.507Z" level=info msg="Watch workflowtemplates 200"
time="2021-07-14T09:05:19.515Z" level=info msg="Watch pods 200"
time="2021-07-14T09:05:19.519Z" level=info msg="Watch workflows 200"
time="2021-07-14T09:05:19.602Z" level=info msg="Create selfsubjectaccessreviews 201"
time="2021-07-14T09:05:19.605Z" level=info msg="Create selfsubjectaccessreviews 201"
time="2021-07-14T09:05:19.606Z" level=info msg="Create selfsubjectaccessreviews 201"
time="2021-07-14T09:05:19.610Z" level=info msg="List clusterworkflowtemplates 200"
time="2021-07-14T09:05:19.613Z" level=info msg="List workflows 200"
time="2021-07-14T09:05:19.614Z" level=info msg="Watch clusterworkflowtemplates 200"
time="2021-07-14T09:05:19.625Z" level=info msg="Manager initialized successfully"
I0714 09:05:19.625360       1 leaderelection.go:243] attempting to acquire leader lease  argo/workflow-controller...
time="2021-07-14T09:05:19.625Z" level=info msg="Starting prometheus metrics server at localhost:9090/metrics"
time="2021-07-14T09:05:19.635Z" level=info msg="Get leases 200"
time="2021-07-14T09:05:19.641Z" level=info msg="Update leases 200"
I0714 09:05:19.641644       1 leaderelection.go:253] successfully acquired lease argo/workflow-controller
time="2021-07-14T09:05:19.641Z" level=info msg="new leader" id=workflow-controller-5487447d-jvw48 leader=workflow-controller-5487447d-jvw48
time="2021-07-14T09:05:19.641Z" level=info msg="started leading" id=workflow-controller-5487447d-jvw48
time="2021-07-14T09:05:19.642Z" level=info msg="Performing periodic GC every 5m0s"
time="2021-07-14T09:05:19.642Z" level=info msg="Persistence disabled - so archived workflow GC disabled - you must restart the controller if you enable this"
time="2021-07-14T09:05:19.644Z" level=info msg="Get leases 200"
time="2021-07-14T09:05:19.646Z" level=info msg="Starting workflow TTL controller (workflowTTLWorkers 4)"
time="2021-07-14T09:05:19.646Z" level=info msg="Started workflow TTL worker"
time="2021-07-14T09:05:19.648Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-94b4q
time="2021-07-14T09:05:19.648Z" level=info msg="Create events 201"
time="2021-07-14T09:05:19.649Z" level=info msg="Starting CronWorkflow controller"
time="2021-07-14T09:05:19.649Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:19.649Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-544g4
time="2021-07-14T09:05:19.649Z" level=info msg="Update leases 200"
time="2021-07-14T09:05:19.650Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:05:19.651Z" level=info msg="List workflows 200"
time="2021-07-14T09:05:19.652Z" level=info msg="List cronworkflows 200"
time="2021-07-14T09:05:19.653Z" level=info msg="List workflows 200"
time="2021-07-14T09:05:19.654Z" level=info msg="Watch cronworkflows 200"
time="2021-07-14T09:05:19.654Z" level=info msg="Watch workflows 200"
time="2021-07-14T09:05:19.655Z" level=info msg="Get configmaps 200"
time="2021-07-14T09:05:19.655Z" level=info msg="remote-tests-runner/Mutex/test acquired by remote-tests-runner/ci-pd6nm " mutex=remote-tests-runner/Mutex/test
time="2021-07-14T09:05:19.655Z" level=info msg="Pod failed: Error (exit code 1)" displayName=check-and-tests namespace=remote-tests-runner pod=ci-pd6nm-3684625071 templateName=run-client-checks workflow=ci-pd6nm
time="2021-07-14T09:05:19.655Z" level=info msg="Updating node ci-pd6nm-3684625071 exit code 1" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:19.655Z" level=info msg="Updating node ci-pd6nm-3684625071 status Running -> Failed" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:19.655Z" level=info msg="Updating node ci-pd6nm-3684625071 message: Error (exit code 1)" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:19.656Z" level=info msg="Get configmaps 200"
time="2021-07-14T09:05:19.656Z" level=warning msg="Workflow processing has been postponed due to concurrency limit" namespace=remote-tests-runner workflow=ci-544g4
time="2021-07-14T09:05:19.656Z" level=info msg="SG Outbound nodes of ci-pd6nm-1125991168 are [ci-pd6nm-1125991168]" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:19.657Z" level=info msg="SG Outbound nodes of ci-pd6nm-2888078899 are [ci-pd6nm-2888078899]" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:19.657Z" level=info msg="Step group node ci-pd6nm-282325193 successful" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:19.657Z" level=info msg="node ci-pd6nm-282325193 phase Running -> Succeeded" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:19.657Z" level=info msg="node ci-pd6nm-282325193 finished: 2021-07-14 09:05:19.657593786 +0000 UTC" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:19.657Z" level=info msg="StepGroup node ci-pd6nm-3436664660 initialized Running" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:19.657Z" level=info msg="SG Outbound nodes of ci-pd6nm-3684625071 are [ci-pd6nm-3684625071]" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:19.657Z" level=info msg="Get configmaps 200"
time="2021-07-14T09:05:19.658Z" level=info msg="Skipping ci-pd6nm[3].set-status-success: when '1 == 0' evaluated false" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:19.658Z" level=info msg="Skipped node ci-pd6nm-2088383165 initialized Skipped (message: when '1 == 0' evaluated false)" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:19.658Z" level=warning msg="Workflow processing has been postponed due to concurrency limit" namespace=remote-tests-runner workflow=ci-94b4q
time="2021-07-14T09:05:19.658Z" level=info msg="Pod node ci-pd6nm-1807515160 initialized Pending" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:19.665Z" level=info msg="Watch workflows 200"
time="2021-07-14T09:05:19.670Z" level=info msg="Get configmaps 200"
time="2021-07-14T09:05:19.671Z" level=warning msg="Workflow processing has been postponed due to concurrency limit" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:05:19.676Z" level=info msg="Update workflows 200"
time="2021-07-14T09:05:19.682Z" level=info msg="Create pods 201"
time="2021-07-14T09:05:19.683Z" level=info msg="Update workflows 200"
time="2021-07-14T09:05:19.686Z" level=info msg="Workflow update successful" namespace=remote-tests-runner phase= resourceVersion=103365388 workflow=ci-94b4q
time="2021-07-14T09:05:19.697Z" level=info msg="Created pod: ci-pd6nm[3].set-status-failure (ci-pd6nm-1807515160)" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:19.697Z" level=info msg="Workflow step group node ci-pd6nm-3436664660 not yet completed" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:19.703Z" level=info msg="Workflow update successful" namespace=remote-tests-runner phase= resourceVersion=103365390 workflow=ci-f7kw7
time="2021-07-14T09:05:19.712Z" level=info msg="Update workflows 200"
time="2021-07-14T09:05:19.714Z" level=info msg="Workflow update successful" namespace=remote-tests-runner phase=Running resourceVersion=103365393 workflow=ci-pd6nm
time="2021-07-14T09:05:19.719Z" level=info msg="Create events 201"
time="2021-07-14T09:05:19.722Z" level=info msg="cleaning up pod" action=labelPodCompleted key=remote-tests-runner/ci-pd6nm-3684625071/labelPodCompleted
time="2021-07-14T09:05:19.727Z" level=info msg="Create events 201"
time="2021-07-14T09:05:19.732Z" level=info msg="Patch pods 200"
time="2021-07-14T09:05:19.736Z" level=info msg="Create events 201"
time="2021-07-14T09:05:24.655Z" level=info msg="Get leases 200"
time="2021-07-14T09:05:24.663Z" level=info msg="Update leases 200"
time="2021-07-14T09:05:29.643Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-544g4
time="2021-07-14T09:05:29.644Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:29.649Z" level=info msg="Get configmaps 200"
time="2021-07-14T09:05:29.650Z" level=warning msg="Workflow processing has been postponed due to concurrency limit" namespace=remote-tests-runner workflow=ci-544g4
time="2021-07-14T09:05:29.650Z" level=info msg="Get configmaps 200"
time="2021-07-14T09:05:29.650Z" level=info msg="Updating node ci-pd6nm-1807515160 exit code 0" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:29.650Z" level=info msg="Updating node ci-pd6nm-1807515160 status Pending -> Succeeded" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:29.651Z" level=info msg="SG Outbound nodes of ci-pd6nm-1125991168 are [ci-pd6nm-1125991168]" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:29.652Z" level=info msg="SG Outbound nodes of ci-pd6nm-2888078899 are [ci-pd6nm-2888078899]" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:29.652Z" level=info msg="SG Outbound nodes of ci-pd6nm-3684625071 are [ci-pd6nm-3684625071]" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:29.652Z" level=info msg="Step group node ci-pd6nm-3436664660 successful" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:29.652Z" level=info msg="node ci-pd6nm-3436664660 phase Running -> Succeeded" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:29.652Z" level=info msg="node ci-pd6nm-3436664660 finished: 2021-07-14 09:05:29.652925715 +0000 UTC" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:29.652Z" level=info msg="StepGroup node ci-pd6nm-3436811755 initialized Running" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:29.653Z" level=info msg="SG Outbound nodes of ci-pd6nm-2088383165 are [ci-pd6nm-2088383165]" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:29.653Z" level=info msg="SG Outbound nodes of ci-pd6nm-1807515160 are [ci-pd6nm-1807515160]" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:29.653Z" level=info msg="Skipping ci-pd6nm[4].slack-notify-success: when '1 == 0' evaluated false" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:29.653Z" level=info msg="Skipped node ci-pd6nm-2098516475 initialized Skipped (message: when '1 == 0' evaluated false)" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:29.653Z" level=info msg="Pod node ci-pd6nm-2507481642 initialized Pending" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:29.664Z" level=info msg="Create pods 201"
time="2021-07-14T09:05:29.665Z" level=info msg="Created pod: ci-pd6nm[4].slack-notify-failure (ci-pd6nm-2507481642)" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:29.665Z" level=info msg="Workflow step group node ci-pd6nm-3436811755 not yet completed" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:29.672Z" level=info msg="Get leases 200"
time="2021-07-14T09:05:29.678Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-94b4q
time="2021-07-14T09:05:29.683Z" level=info msg="Update workflows 200"
time="2021-07-14T09:05:29.684Z" level=info msg="Update leases 200"
time="2021-07-14T09:05:29.687Z" level=info msg="Workflow update successful" namespace=remote-tests-runner phase=Running resourceVersion=103365469 workflow=ci-pd6nm
time="2021-07-14T09:05:29.692Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:05:29.692Z" level=info msg="Create events 201"
time="2021-07-14T09:05:29.694Z" level=info msg="Get configmaps 200"
time="2021-07-14T09:05:29.695Z" level=warning msg="Workflow processing has been postponed due to concurrency limit" namespace=remote-tests-runner workflow=ci-94b4q
time="2021-07-14T09:05:29.698Z" level=info msg="cleaning up pod" action=labelPodCompleted key=remote-tests-runner/ci-pd6nm-1807515160/labelPodCompleted
time="2021-07-14T09:05:29.702Z" level=info msg="Create events 201"
time="2021-07-14T09:05:29.707Z" level=info msg="Patch pods 200"
time="2021-07-14T09:05:29.710Z" level=info msg="Create events 201"
time="2021-07-14T09:05:29.711Z" level=info msg="Get configmaps 200"
time="2021-07-14T09:05:29.711Z" level=warning msg="Workflow processing has been postponed due to concurrency limit" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:05:29.715Z" level=info msg="Create events 201"
time="2021-07-14T09:05:34.689Z" level=info msg="Get leases 200"
time="2021-07-14T09:05:34.711Z" level=info msg="Update leases 200"
time="2021-07-14T09:05:39.667Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:39.673Z" level=info msg="Get configmaps 200"
time="2021-07-14T09:05:39.673Z" level=info msg="Updating node ci-pd6nm-2507481642 exit code 0" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:39.673Z" level=info msg="Updating node ci-pd6nm-2507481642 status Pending -> Succeeded" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:39.674Z" level=info msg="SG Outbound nodes of ci-pd6nm-1125991168 are [ci-pd6nm-1125991168]" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:39.674Z" level=info msg="SG Outbound nodes of ci-pd6nm-2888078899 are [ci-pd6nm-2888078899]" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:39.675Z" level=info msg="SG Outbound nodes of ci-pd6nm-3684625071 are [ci-pd6nm-3684625071]" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:39.675Z" level=info msg="SG Outbound nodes of ci-pd6nm-2088383165 are [ci-pd6nm-2088383165]" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:39.675Z" level=info msg="SG Outbound nodes of ci-pd6nm-1807515160 are [ci-pd6nm-1807515160]" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:39.675Z" level=info msg="Step group node ci-pd6nm-3436811755 successful" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:39.675Z" level=info msg="node ci-pd6nm-3436811755 phase Running -> Succeeded" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:39.675Z" level=info msg="node ci-pd6nm-3436811755 finished: 2021-07-14 09:05:39.675917354 +0000 UTC" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:39.675Z" level=info msg="Outbound nodes of ci-pd6nm-2098516475 is [ci-pd6nm-2098516475]" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:39.676Z" level=info msg="Outbound nodes of ci-pd6nm-2507481642 is [ci-pd6nm-2507481642]" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:39.676Z" level=info msg="Outbound nodes of ci-pd6nm is [ci-pd6nm-2098516475 ci-pd6nm-2507481642]" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:39.676Z" level=info msg="node ci-pd6nm phase Running -> Succeeded" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:39.676Z" level=info msg="node ci-pd6nm finished: 2021-07-14 09:05:39.676147329 +0000 UTC" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:39.676Z" level=info msg="Checking daemoned children of ci-pd6nm" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:39.676Z" level=info msg="Updated phase Running -> Succeeded" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:39.676Z" level=info msg="Updated message Waiting for remote-tests-runner/Mutex/test lock. Lock status: 0/1  -> " namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:39.676Z" level=info msg="Marking workflow completed" namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:39.676Z" level=info msg="Checking daemoned children of " namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:39.677Z" level=info msg="Lock has been released by remote-tests-runner/ci-pd6nm. Available locks: 1" mutex=remote-tests-runner/Mutex/test
time="2021-07-14T09:05:39.677Z" level=info msg="remote-tests-runner/ci-pd6nm released a lock from remote-tests-runner/Mutex/test"
time="2021-07-14T09:05:39.677Z" level=info msg="Released all acquired locks" key=ci-pd6nm namespace=remote-tests-runner workflow=ci-pd6nm
time="2021-07-14T09:05:39.694Z" level=info msg="Create events 201"
time="2021-07-14T09:05:39.705Z" level=info msg="Update workflows 200"
time="2021-07-14T09:05:39.707Z" level=info msg="Workflow update successful" namespace=remote-tests-runner phase=Succeeded resourceVersion=103365542 workflow=ci-pd6nm
time="2021-07-14T09:05:39.717Z" level=info msg="Queueing Succeeded workflow remote-tests-runner/ci-pd6nm for delete in 5m0s"
time="2021-07-14T09:05:39.717Z" level=info msg="cleaning up pod" action=labelPodCompleted key=remote-tests-runner/ci-pd6nm-2507481642/labelPodCompleted
time="2021-07-14T09:05:39.717Z" level=info msg="Queueing Succeeded workflow remote-tests-runner/ci-pd6nm for delete in 5m0s"
time="2021-07-14T09:05:39.753Z" level=info msg="Create events 201"
time="2021-07-14T09:05:39.792Z" level=info msg="Get leases 200"
time="2021-07-14T09:05:39.793Z" level=info msg="Patch pods 200"
time="2021-07-14T09:05:39.803Z" level=info msg="Create events 201"
time="2021-07-14T09:05:39.817Z" level=info msg="Update leases 200"
time="2021-07-14T09:05:39.820Z" level=info msg="Create events 201"
time="2021-07-14T09:05:39.902Z" level=info msg="Create events 201"
time="2021-07-14T09:05:44.822Z" level=info msg="Get leases 200"
time="2021-07-14T09:05:44.830Z" level=info msg="Update leases 200"
time="2021-07-14T09:05:49.678Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:05:49.689Z" level=info msg="Get configmaps 200"
time="2021-07-14T09:05:49.689Z" level=info msg="remote-tests-runner/Mutex/test acquired by remote-tests-runner/ci-f7kw7 " mutex=remote-tests-runner/Mutex/test
time="2021-07-14T09:05:49.689Z" level=info msg="Updated phase  -> Running" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:05:49.689Z" level=info msg="Updated message Waiting for remote-tests-runner/Mutex/test lock. Lock status: 0/1  -> " namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:05:49.690Z" level=info msg="Steps node ci-f7kw7 initialized Running" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:05:49.690Z" level=info msg="StepGroup node ci-f7kw7-1242180384 initialized Running" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:05:49.691Z" level=info msg="Pod node ci-f7kw7-134148425 initialized Pending" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:05:49.700Z" level=info msg="Create events 201"
time="2021-07-14T09:05:49.704Z" level=info msg="Create pods 201"
time="2021-07-14T09:05:49.705Z" level=info msg="Created pod: ci-f7kw7[0].set-status-pending (ci-f7kw7-134148425)" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:05:49.705Z" level=info msg="Workflow step group node ci-f7kw7-1242180384 not yet completed" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:05:49.718Z" level=info msg="Update workflows 200"
time="2021-07-14T09:05:49.719Z" level=info msg="Workflow update successful" namespace=remote-tests-runner phase=Running resourceVersion=103365601 workflow=ci-f7kw7
time="2021-07-14T09:05:49.724Z" level=info msg="Create events 201"
time="2021-07-14T09:05:49.731Z" level=info msg="Create events 201"
time="2021-07-14T09:05:49.836Z" level=info msg="Get leases 200"
time="2021-07-14T09:05:49.843Z" level=info msg="Update leases 200"
time="2021-07-14T09:05:54.849Z" level=info msg="Get leases 200"
time="2021-07-14T09:05:54.870Z" level=info msg="Update leases 200"
time="2021-07-14T09:05:59.709Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:05:59.777Z" level=info msg="Get configmaps 200"
time="2021-07-14T09:05:59.779Z" level=info msg="Updating node ci-f7kw7-134148425 exit code 0" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:05:59.779Z" level=info msg="Updating node ci-f7kw7-134148425 status Pending -> Succeeded" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:05:59.780Z" level=info msg="Step group node ci-f7kw7-1242180384 successful" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:05:59.780Z" level=info msg="node ci-f7kw7-1242180384 phase Running -> Succeeded" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:05:59.780Z" level=info msg="node ci-f7kw7-1242180384 finished: 2021-07-14 09:05:59.780861102 +0000 UTC" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:05:59.780Z" level=info msg="StepGroup node ci-f7kw7-1309143765 initialized Running" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:05:59.780Z" level=info msg="SG Outbound nodes of ci-f7kw7-134148425 are [ci-f7kw7-134148425]" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:05:59.781Z" level=info msg="Pod node ci-f7kw7-1059688890 initialized Pending" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:05:59.844Z" level=info msg="Create pods 201"
time="2021-07-14T09:05:59.845Z" level=info msg="Created pod: ci-f7kw7[1].get-client-src (ci-f7kw7-1059688890)" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:05:59.845Z" level=info msg="Workflow step group node ci-f7kw7-1309143765 not yet completed" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:05:59.946Z" level=info msg="Update workflows 200"
time="2021-07-14T09:05:59.949Z" level=info msg="Workflow update successful" namespace=remote-tests-runner phase=Running resourceVersion=103365674 workflow=ci-f7kw7
time="2021-07-14T09:05:59.959Z" level=info msg="cleaning up pod" action=labelPodCompleted key=remote-tests-runner/ci-f7kw7-134148425/labelPodCompleted
time="2021-07-14T09:05:59.970Z" level=info msg="Get leases 200"
time="2021-07-14T09:06:00.046Z" level=info msg="Create events 201"
time="2021-07-14T09:06:00.050Z" level=info msg="Update leases 200"
time="2021-07-14T09:06:00.050Z" level=info msg="Patch pods 200"
time="2021-07-14T09:06:00.092Z" level=info msg="Create events 201"
time="2021-07-14T09:06:00.127Z" level=info msg="Create events 201"
time="2021-07-14T09:06:00.144Z" level=info msg="Create events 201"
time="2021-07-14T09:06:05.055Z" level=info msg="Get leases 200"
time="2021-07-14T09:06:05.061Z" level=info msg="Update leases 200"
time="2021-07-14T09:06:09.848Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:06:09.854Z" level=info msg="Get configmaps 200"
time="2021-07-14T09:06:09.854Z" level=info msg="Updating node ci-f7kw7-1059688890 message: PodInitializing" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:06:09.855Z" level=info msg="SG Outbound nodes of ci-f7kw7-134148425 are [ci-f7kw7-134148425]" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:06:09.855Z" level=info msg="Workflow step group node ci-f7kw7-1309143765 not yet completed" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:06:09.871Z" level=info msg="Update workflows 200"
time="2021-07-14T09:06:09.872Z" level=info msg="Workflow update successful" namespace=remote-tests-runner phase=Running resourceVersion=103365737 workflow=ci-f7kw7
time="2021-07-14T09:06:10.071Z" level=info msg="Get leases 200"
time="2021-07-14T09:06:10.087Z" level=info msg="Update leases 200"
time="2021-07-14T09:06:15.097Z" level=info msg="Get leases 200"
time="2021-07-14T09:06:15.107Z" level=info msg="Update leases 200"
time="2021-07-14T09:06:19.876Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:06:19.882Z" level=info msg="Get configmaps 200"
time="2021-07-14T09:06:19.883Z" level=info msg="SG Outbound nodes of ci-f7kw7-134148425 are [ci-f7kw7-134148425]" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:06:19.883Z" level=info msg="Workflow step group node ci-f7kw7-1309143765 not yet completed" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:06:20.122Z" level=info msg="Get leases 200"
time="2021-07-14T09:06:20.137Z" level=info msg="Update leases 200"

Second part logs:
time="2021-07-14T09:09:19Z" level=info msg="index config" indexWorkflowSemaphoreKeys=true
time="2021-07-14T09:09:19Z" level=info msg="cron config" cronSyncPeriod=10s
time="2021-07-14T09:09:19.418Z" level=info msg="config map" name=workflow-controller-configmap
time="2021-07-14T09:09:19.435Z" level=info msg="Get configmaps 200"
time="2021-07-14T09:09:19.442Z" level=info msg="Configuration:\nartifactRepository: {}\ninitialDelay: 0s\nmetricsConfig: {}\nnodeEvents: {}\npodSpecLogStrategy: {}\ntelemetryConfig: {}\n"
time="2021-07-14T09:09:19.442Z" level=info msg="Persistence configuration disabled"
time="2021-07-14T09:09:19.443Z" level=info msg="Starting Workflow Controller" version=v3.1.1
time="2021-07-14T09:09:19.443Z" level=info msg="Workers: workflow: 32, pod: 32, pod cleanup: 4"
time="2021-07-14T09:09:19.445Z" level=info msg="Watch configmaps 200"
time="2021-07-14T09:09:19.446Z" level=info msg="List configmaps 200"
time="2021-07-14T09:09:19.446Z" level=info msg="List workflowtemplates 200"
time="2021-07-14T09:09:19.448Z" level=info msg="List pods 200"
time="2021-07-14T09:09:19.452Z" level=info msg="List workflows 200"
time="2021-07-14T09:09:19.453Z" level=info msg="Watch configmaps 200"
time="2021-07-14T09:09:19.456Z" level=info msg="Watch workflowtemplates 200"
time="2021-07-14T09:09:19.464Z" level=info msg="Watch workflows 200"
time="2021-07-14T09:09:19.473Z" level=info msg="Watch pods 200"
time="2021-07-14T09:09:19.548Z" level=info msg="Create selfsubjectaccessreviews 201"
time="2021-07-14T09:09:19.551Z" level=info msg="Create selfsubjectaccessreviews 201"
time="2021-07-14T09:09:19.556Z" level=info msg="Create selfsubjectaccessreviews 201"
time="2021-07-14T09:09:19.559Z" level=info msg="List clusterworkflowtemplates 200"
time="2021-07-14T09:09:19.565Z" level=info msg="List workflows 200"
time="2021-07-14T09:09:19.565Z" level=info msg="Watch clusterworkflowtemplates 200"
time="2021-07-14T09:09:19.580Z" level=info msg="Manager initialized successfully"
I0714 09:09:19.581032       1 leaderelection.go:243] attempting to acquire leader lease  argo/workflow-controller...
time="2021-07-14T09:09:19.581Z" level=info msg="Starting prometheus metrics server at localhost:9090/metrics"
time="2021-07-14T09:09:19.584Z" level=info msg="Get leases 200"
time="2021-07-14T09:09:19.595Z" level=info msg="Update leases 200"
I0714 09:09:19.596047       1 leaderelection.go:253] successfully acquired lease argo/workflow-controller
time="2021-07-14T09:09:19.596Z" level=info msg="started leading" id=workflow-controller-5487447d-jvw48
time="2021-07-14T09:09:19.596Z" level=info msg="new leader" id=workflow-controller-5487447d-jvw48 leader=workflow-controller-5487447d-jvw48
time="2021-07-14T09:09:19.597Z" level=info msg="Performing periodic GC every 5m0s"
time="2021-07-14T09:09:19.597Z" level=info msg="Persistence disabled - so archived workflow GC disabled - you must restart the controller if you enable this"
time="2021-07-14T09:09:19.597Z" level=info msg="Starting workflow TTL controller (workflowTTLWorkers 4)"
time="2021-07-14T09:09:19.597Z" level=info msg="Started workflow TTL worker"
time="2021-07-14T09:09:19.597Z" level=info msg="Starting CronWorkflow controller"
time="2021-07-14T09:09:19.598Z" level=info msg="Get leases 200"
time="2021-07-14T09:09:19.598Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:09:19.600Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-94b4q
time="2021-07-14T09:09:19.600Z" level=info msg="Create events 201"
time="2021-07-14T09:09:19.602Z" level=info msg="List cronworkflows 200"
time="2021-07-14T09:09:19.603Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-544g4
time="2021-07-14T09:09:19.603Z" level=info msg="List workflows 200"
time="2021-07-14T09:09:19.604Z" level=info msg="Queueing Succeeded workflow remote-tests-runner/ci-pd6nm for delete in 1m20s"
time="2021-07-14T09:09:19.606Z" level=info msg="List workflows 200"
time="2021-07-14T09:09:19.606Z" level=info msg="Get configmaps 200"
time="2021-07-14T09:09:19.609Z" level=info msg="remote-tests-runner/Mutex/test acquired by remote-tests-runner/ci-f7kw7 " mutex=remote-tests-runner/Mutex/test
time="2021-07-14T09:09:19.606Z" level=info msg="Watch workflows 200"
time="2021-07-14T09:09:19.607Z" level=info msg="Get configmaps 200"
time="2021-07-14T09:09:19.610Z" level=warning msg="Workflow processing has been postponed due to concurrency limit" namespace=remote-tests-runner workflow=ci-544g4
time="2021-07-14T09:09:19.610Z" level=info msg="SG Outbound nodes of ci-f7kw7-134148425 are [ci-f7kw7-134148425]" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:09:19.610Z" level=info msg="SG Outbound nodes of ci-f7kw7-1059688890 are [ci-f7kw7-1059688890]" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:09:19.607Z" level=info msg="Update leases 200"
time="2021-07-14T09:09:19.607Z" level=info msg="Watch cronworkflows 200"
time="2021-07-14T09:09:19.611Z" level=info msg="Workflow step group node ci-f7kw7-235229054 not yet completed" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:09:19.616Z" level=info msg="Watch workflows 200"
time="2021-07-14T09:09:19.619Z" level=info msg="Queueing Succeeded workflow remote-tests-runner/ci-pd6nm for delete in 1m20s"
time="2021-07-14T09:09:19.622Z" level=info msg="Get configmaps 200"
time="2021-07-14T09:09:19.622Z" level=warning msg="Workflow processing has been postponed due to concurrency limit" namespace=remote-tests-runner workflow=ci-94b4q
time="2021-07-14T09:09:19.623Z" level=info msg="Update workflows 200"
time="2021-07-14T09:09:19.635Z" level=info msg="Workflow update successful" namespace=remote-tests-runner phase=Running resourceVersion=103366775 workflow=ci-544g4
time="2021-07-14T09:09:19.653Z" level=info msg="Update workflows 200"
time="2021-07-14T09:09:19.654Z" level=info msg="Workflow update successful" namespace=remote-tests-runner phase= resourceVersion=103366776 workflow=ci-94b4q
time="2021-07-14T09:09:24.615Z" level=info msg="Get leases 200"
time="2021-07-14T09:09:24.630Z" level=info msg="Update leases 200"
time="2021-07-14T09:09:29.598Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-544g4
time="2021-07-14T09:09:29.598Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:09:29.602Z" level=info msg="Get configmaps 200"
time="2021-07-14T09:09:29.603Z" level=info msg="Get configmaps 200"
time="2021-07-14T09:09:29.603Z" level=warning msg="Workflow processing has been postponed due to concurrency limit" namespace=remote-tests-runner workflow=ci-544g4
time="2021-07-14T09:09:29.604Z" level=info msg="SG Outbound nodes of ci-f7kw7-134148425 are [ci-f7kw7-134148425]" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:09:29.604Z" level=info msg="SG Outbound nodes of ci-f7kw7-1059688890 are [ci-f7kw7-1059688890]" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:09:29.604Z" level=info msg="Workflow step group node ci-f7kw7-235229054 not yet completed" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:09:29.633Z" level=info msg="Get leases 200"
time="2021-07-14T09:09:29.638Z" level=info msg="Update leases 200"
time="2021-07-14T09:09:29.657Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-94b4q
time="2021-07-14T09:09:29.664Z" level=info msg="Get configmaps 200"
time="2021-07-14T09:09:29.665Z" level=warning msg="Workflow processing has been postponed due to concurrency limit" namespace=remote-tests-runner workflow=ci-94b4q
time="2021-07-14T09:09:34.642Z" level=info msg="Get leases 200"
time="2021-07-14T09:09:34.649Z" level=info msg="Update leases 200"
time="2021-07-14T09:09:39.653Z" level=info msg="Get leases 200"
time="2021-07-14T09:09:39.681Z" level=info msg="Update leases 200"
time="2021-07-14T09:09:44.685Z" level=info msg="Get leases 200"
time="2021-07-14T09:09:44.691Z" level=info msg="Update leases 200"
time="2021-07-14T09:09:49.695Z" level=info msg="Get leases 200"
time="2021-07-14T09:09:49.702Z" level=info msg="Update leases 200"
time="2021-07-14T09:09:54.707Z" level=info msg="Get leases 200"
time="2021-07-14T09:09:54.720Z" level=info msg="Update leases 200"
time="2021-07-14T09:09:59.724Z" level=info msg="Get leases 200"
time="2021-07-14T09:09:59.734Z" level=info msg="Update leases 200"
time="2021-07-14T09:10:04.739Z" level=info msg="Get leases 200"
time="2021-07-14T09:10:04.767Z" level=info msg="Update leases 200"
time="2021-07-14T09:10:09.770Z" level=info msg="Get leases 200"
time="2021-07-14T09:10:09.779Z" level=info msg="Update leases 200"
time="2021-07-14T09:10:14.784Z" level=info msg="Get leases 200"
time="2021-07-14T09:10:14.797Z" level=info msg="Update leases 200"
time="2021-07-14T09:10:19.800Z" level=info msg="Get leases 200"
time="2021-07-14T09:10:19.807Z" level=info msg="Update leases 200"
time="2021-07-14T09:10:24.812Z" level=info msg="Get leases 200"
time="2021-07-14T09:10:24.817Z" level=info msg="Update leases 200"
time="2021-07-14T09:10:29.820Z" level=info msg="Get leases 200"
time="2021-07-14T09:10:29.826Z" level=info msg="Update leases 200"
time="2021-07-14T09:10:34.830Z" level=info msg="Get leases 200"
time="2021-07-14T09:10:34.836Z" level=info msg="Update leases 200"
time="2021-07-14T09:10:39.840Z" level=info msg="Get leases 200"
time="2021-07-14T09:10:39.863Z" level=info msg="Update leases 200"
time="2021-07-14T09:10:40.000Z" level=info msg="Deleting TTL expired workflow 'remote-tests-runner/ci-pd6nm'"
time="2021-07-14T09:10:40.012Z" level=info msg="Delete workflows 200"
time="2021-07-14T09:10:40.012Z" level=info msg="Successfully deleted 'remote-tests-runner/ci-pd6nm'"
time="2021-07-14T09:10:44.867Z" level=info msg="Get leases 200"
time="2021-07-14T09:10:44.884Z" level=info msg="Update leases 200"
time="2021-07-14T09:10:49.888Z" level=info msg="Get leases 200"
time="2021-07-14T09:10:49.896Z" level=info msg="Update leases 200"
time="2021-07-14T09:10:54.901Z" level=info msg="Get leases 200"
time="2021-07-14T09:10:54.907Z" level=info msg="Update leases 200"
time="2021-07-14T09:10:59.911Z" level=info msg="Get leases 200"
time="2021-07-14T09:10:59.917Z" level=info msg="Update leases 200"
time="2021-07-14T09:11:04.921Z" level=info msg="Get leases 200"
time="2021-07-14T09:11:04.927Z" level=info msg="Update leases 200"
time="2021-07-14T09:11:09.932Z" level=info msg="Get leases 200"
time="2021-07-14T09:11:09.955Z" level=info msg="Update leases 200"
time="2021-07-14T09:11:14.960Z" level=info msg="Get leases 200"
time="2021-07-14T09:11:14.966Z" level=info msg="Update leases 200"
time="2021-07-14T09:11:19.026Z" level=info msg="List workflows 200"
time="2021-07-14T09:11:19.027Z" level=info msg=healthz age=5m0s err="workflow never reconciled: ci-94b4q" instanceID= labelSelector="!workflows.argoproj.io/phase,!workflows.argoproj.io/controller-instanceid" managedNamespace=
time="2021-07-14T09:11:19.971Z" level=info msg="Get leases 200"
time="2021-07-14T09:11:19.981Z" level=info msg="Update leases 200"
time="2021-07-14T09:11:24.985Z" level=info msg="Get leases 200"
time="2021-07-14T09:11:24.994Z" level=info msg="Update leases 200"
time="2021-07-14T09:11:29.998Z" level=info msg="Get leases 200"
time="2021-07-14T09:11:30.004Z" level=info msg="Update leases 200"
time="2021-07-14T09:11:35.010Z" level=info msg="Get leases 200"
time="2021-07-14T09:11:35.018Z" level=info msg="Update leases 200"
time="2021-07-14T09:11:40.022Z" level=info msg="Get leases 200"
time="2021-07-14T09:11:40.031Z" level=info msg="Update leases 200"
time="2021-07-14T09:11:45.035Z" level=info msg="Get leases 200"
time="2021-07-14T09:11:45.040Z" level=info msg="Update leases 200"
time="2021-07-14T09:11:50.044Z" level=info msg="Get leases 200"
time="2021-07-14T09:11:50.054Z" level=info msg="Update leases 200"
time="2021-07-14T09:11:55.059Z" level=info msg="Get leases 200"
time="2021-07-14T09:11:55.079Z" level=info msg="Update leases 200"
time="2021-07-14T09:12:00.083Z" level=info msg="Get leases 200"
time="2021-07-14T09:12:00.090Z" level=info msg="Update leases 200"
time="2021-07-14T09:12:05.094Z" level=info msg="Get leases 200"
time="2021-07-14T09:12:05.100Z" level=info msg="Update leases 200"
time="2021-07-14T09:12:10.105Z" level=info msg="Get leases 200"
time="2021-07-14T09:12:10.122Z" level=info msg="Update leases 200"
time="2021-07-14T09:12:13.453Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:13.458Z" level=info msg="Get configmaps 200"
time="2021-07-14T09:12:13.458Z" level=info msg="Pod failed: Error (exit code 1)" displayName=check-and-tests namespace=remote-tests-runner pod=ci-f7kw7-884495176 templateName=run-client-checks workflow=ci-f7kw7
time="2021-07-14T09:12:13.458Z" level=info msg="Updating node ci-f7kw7-884495176 exit code 1" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:13.458Z" level=info msg="Updating node ci-f7kw7-884495176 status Running -> Failed" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:13.458Z" level=info msg="Updating node ci-f7kw7-884495176 message: Error (exit code 1)" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:13.460Z" level=info msg="SG Outbound nodes of ci-f7kw7-134148425 are [ci-f7kw7-134148425]" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:13.461Z" level=info msg="SG Outbound nodes of ci-f7kw7-1059688890 are [ci-f7kw7-1059688890]" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:13.461Z" level=info msg="Step group node ci-f7kw7-235229054 successful" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:13.461Z" level=info msg="node ci-f7kw7-235229054 phase Running -> Succeeded" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:13.461Z" level=info msg="node ci-f7kw7-235229054 finished: 2021-07-14 09:12:13.461477151 +0000 UTC" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:13.461Z" level=info msg="StepGroup node ci-f7kw7-167971483 initialized Running" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:13.461Z" level=info msg="SG Outbound nodes of ci-f7kw7-884495176 are [ci-f7kw7-884495176]" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:13.461Z" level=info msg="Skipping ci-f7kw7[3].set-status-success: when '1 == 0' evaluated false" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:13.462Z" level=info msg="Skipped node ci-f7kw7-1334290768 initialized Skipped (message: when '1 == 0' evaluated false)" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:13.462Z" level=info msg="Pod node ci-f7kw7-1043769457 initialized Pending" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:13.480Z" level=info msg="Create pods 201"
time="2021-07-14T09:12:13.480Z" level=info msg="Created pod: ci-f7kw7[3].set-status-failure (ci-f7kw7-1043769457)" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:13.480Z" level=info msg="Workflow step group node ci-f7kw7-167971483 not yet completed" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:13.500Z" level=info msg="Update workflows 200"
time="2021-07-14T09:12:13.503Z" level=info msg="Workflow update successful" namespace=remote-tests-runner phase=Running resourceVersion=103367693 workflow=ci-f7kw7
time="2021-07-14T09:12:13.509Z" level=info msg="Create events 201"
time="2021-07-14T09:12:13.512Z" level=info msg="cleaning up pod" action=labelPodCompleted key=remote-tests-runner/ci-f7kw7-884495176/labelPodCompleted
time="2021-07-14T09:12:13.520Z" level=info msg="Create events 201"
time="2021-07-14T09:12:13.524Z" level=info msg="Patch pods 200"
time="2021-07-14T09:12:13.532Z" level=info msg="Create events 201"
time="2021-07-14T09:12:15.126Z" level=info msg="Get leases 200"
time="2021-07-14T09:12:15.160Z" level=info msg="Update leases 200"
time="2021-07-14T09:12:19.026Z" level=info msg="List workflows 200"
time="2021-07-14T09:12:19.027Z" level=info msg=healthz age=5m0s err="workflow never reconciled: ci-94b4q" instanceID= labelSelector="!workflows.argoproj.io/phase,!workflows.argoproj.io/controller-instanceid" managedNamespace=
time="2021-07-14T09:12:20.164Z" level=info msg="Get leases 200"
time="2021-07-14T09:12:20.171Z" level=info msg="Update leases 200"
time="2021-07-14T09:12:23.489Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:23.494Z" level=info msg="Get configmaps 200"
time="2021-07-14T09:12:23.494Z" level=info msg="Updating node ci-f7kw7-1043769457 exit code 0" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:23.494Z" level=info msg="Updating node ci-f7kw7-1043769457 status Pending -> Succeeded" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:23.495Z" level=info msg="SG Outbound nodes of ci-f7kw7-134148425 are [ci-f7kw7-134148425]" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:23.495Z" level=info msg="SG Outbound nodes of ci-f7kw7-1059688890 are [ci-f7kw7-1059688890]" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:23.495Z" level=info msg="SG Outbound nodes of ci-f7kw7-884495176 are [ci-f7kw7-884495176]" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:23.496Z" level=info msg="Step group node ci-f7kw7-167971483 successful" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:23.496Z" level=info msg="node ci-f7kw7-167971483 phase Running -> Succeeded" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:23.496Z" level=info msg="node ci-f7kw7-167971483 finished: 2021-07-14 09:12:23.496437733 +0000 UTC" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:23.496Z" level=info msg="StepGroup node ci-f7kw7-1241488836 initialized Running" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:23.496Z" level=info msg="SG Outbound nodes of ci-f7kw7-1334290768 are [ci-f7kw7-1334290768]" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:23.496Z" level=info msg="SG Outbound nodes of ci-f7kw7-1043769457 are [ci-f7kw7-1043769457]" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:23.496Z" level=info msg="Skipping ci-f7kw7[4].slack-notify-success: when '1 == 0' evaluated false" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:23.496Z" level=info msg="Skipped node ci-f7kw7-1443980226 initialized Skipped (message: when '1 == 0' evaluated false)" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:23.497Z" level=info msg="Pod node ci-f7kw7-2365774311 initialized Pending" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:23.506Z" level=info msg="Create pods 201"
time="2021-07-14T09:12:23.507Z" level=info msg="Created pod: ci-f7kw7[4].slack-notify-failure (ci-f7kw7-2365774311)" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:23.507Z" level=info msg="Workflow step group node ci-f7kw7-1241488836 not yet completed" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:23.520Z" level=info msg="Update workflows 200"
time="2021-07-14T09:12:23.522Z" level=info msg="Workflow update successful" namespace=remote-tests-runner phase=Running resourceVersion=103367767 workflow=ci-f7kw7
time="2021-07-14T09:12:23.532Z" level=info msg="cleaning up pod" action=labelPodCompleted key=remote-tests-runner/ci-f7kw7-1043769457/labelPodCompleted
time="2021-07-14T09:12:23.545Z" level=info msg="Create events 201"
time="2021-07-14T09:12:23.582Z" level=info msg="Patch pods 200"
time="2021-07-14T09:12:23.582Z" level=info msg="Create events 201"
time="2021-07-14T09:12:23.588Z" level=info msg="Create events 201"
time="2021-07-14T09:12:23.591Z" level=info msg="Create events 201"
time="2021-07-14T09:12:25.175Z" level=info msg="Get leases 200"
time="2021-07-14T09:12:25.180Z" level=info msg="Update leases 200"
time="2021-07-14T09:12:30.184Z" level=info msg="Get leases 200"
time="2021-07-14T09:12:30.190Z" level=info msg="Update leases 200"
time="2021-07-14T09:12:33.510Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:33.514Z" level=info msg="Get configmaps 200"
time="2021-07-14T09:12:33.515Z" level=info msg="Updating node ci-f7kw7-2365774311 exit code 0" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:33.515Z" level=info msg="Updating node ci-f7kw7-2365774311 status Pending -> Succeeded" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:33.515Z" level=info msg="SG Outbound nodes of ci-f7kw7-134148425 are [ci-f7kw7-134148425]" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:33.516Z" level=info msg="SG Outbound nodes of ci-f7kw7-1059688890 are [ci-f7kw7-1059688890]" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:33.516Z" level=info msg="SG Outbound nodes of ci-f7kw7-884495176 are [ci-f7kw7-884495176]" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:33.516Z" level=info msg="SG Outbound nodes of ci-f7kw7-1334290768 are [ci-f7kw7-1334290768]" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:33.516Z" level=info msg="SG Outbound nodes of ci-f7kw7-1043769457 are [ci-f7kw7-1043769457]" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:33.516Z" level=info msg="Step group node ci-f7kw7-1241488836 successful" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:33.516Z" level=info msg="node ci-f7kw7-1241488836 phase Running -> Succeeded" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:33.516Z" level=info msg="node ci-f7kw7-1241488836 finished: 2021-07-14 09:12:33.516878283 +0000 UTC" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:33.516Z" level=info msg="Outbound nodes of ci-f7kw7-1443980226 is [ci-f7kw7-1443980226]" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:33.516Z" level=info msg="Outbound nodes of ci-f7kw7-2365774311 is [ci-f7kw7-2365774311]" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:33.516Z" level=info msg="Outbound nodes of ci-f7kw7 is [ci-f7kw7-1443980226 ci-f7kw7-2365774311]" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:33.516Z" level=info msg="node ci-f7kw7 phase Running -> Succeeded" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:33.516Z" level=info msg="node ci-f7kw7 finished: 2021-07-14 09:12:33.516992415 +0000 UTC" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:33.517Z" level=info msg="Checking daemoned children of ci-f7kw7" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:33.517Z" level=info msg="Updated phase Running -> Succeeded" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:33.517Z" level=info msg="Marking workflow completed" namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:33.517Z" level=info msg="Checking daemoned children of " namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:33.517Z" level=info msg="Lock has been released by remote-tests-runner/ci-f7kw7. Available locks: 1" mutex=remote-tests-runner/Mutex/test
time="2021-07-14T09:12:33.517Z" level=info msg="remote-tests-runner/ci-f7kw7 released a lock from remote-tests-runner/Mutex/test"
time="2021-07-14T09:12:33.517Z" level=info msg="Released all acquired locks" key=ci-f7kw7 namespace=remote-tests-runner workflow=ci-f7kw7
time="2021-07-14T09:12:33.521Z" level=info msg="Create events 201"
time="2021-07-14T09:12:33.531Z" level=info msg="Update workflows 200"
time="2021-07-14T09:12:33.534Z" level=info msg="Workflow update successful" namespace=remote-tests-runner phase=Succeeded resourceVersion=103367841 workflow=ci-f7kw7
time="2021-07-14T09:12:33.537Z" level=info msg="Queueing Succeeded workflow remote-tests-runner/ci-f7kw7 for delete in 5m0s"
time="2021-07-14T09:12:33.538Z" level=info msg="remote-tests-runner/ci-f7kw7 released a lock from remote-tests-runner/Mutex/test"
time="2021-07-14T09:12:33.539Z" level=info msg="Create events 201"
time="2021-07-14T09:12:33.541Z" level=info msg="Queueing Succeeded workflow remote-tests-runner/ci-f7kw7 for delete in 5m0s"
time="2021-07-14T09:12:33.545Z" level=info msg="Create events 201"
time="2021-07-14T09:12:33.548Z" level=info msg="cleaning up pod" action=labelPodCompleted key=remote-tests-runner/ci-f7kw7-2365774311/labelPodCompleted
time="2021-07-14T09:12:33.549Z" level=info msg="Create events 201"
time="2021-07-14T09:12:33.553Z" level=info msg="Create events 201"
time="2021-07-14T09:12:33.555Z" level=info msg="Patch pods 200"
time="2021-07-14T09:12:35.194Z" level=info msg="Get leases 200"
time="2021-07-14T09:12:35.214Z" level=info msg="Update leases 200"
time="2021-07-14T09:12:40.219Z" level=info msg="Get leases 200"
time="2021-07-14T09:12:40.225Z" level=info msg="Update leases 200"
time="2021-07-14T09:12:43.518Z" level=info msg="Processing workflow" namespace=remote-tests-runner workflow=ci-94b4q
time="2021-07-14T09:12:43.527Z" level=info msg="Get configmaps 200"
time="2021-07-14T09:12:43.528Z" level=info msg="remote-tests-runner/Mutex/test acquired by remote-tests-runner/ci-94b4q " mutex=remote-tests-runner/Mutex/test
time="2021-07-14T09:12:43.528Z" level=info msg="Updated phase  -> Running" namespace=remote-tests-runner workflow=ci-94b4q
time="2021-07-14T09:12:43.528Z" level=info msg="Updated message Waiting for remote-tests-runner/Mutex/test lock.   Thanks!
I'll close the issue.


fix for GitHub Issue #501 on adding propagation clock

I have updated or scripts to write sdc file at CTS stage to use the updated sdc file for timing check post CTS with propagated delay. This is part of #501 fix. I checked #make fastest_test_set flow completed without fail.

It seems ok to me but I'll leave it to @donn to approve as I don't normally approve OL changes.


Release 0.5.120

We somehow skipped changelog update for 0.5.118 and 0.5.119; this PR also adds a late check that the changelog at least contains the tagged version somewhere prior to release. It's a weak check and happens late in the process. If it fails, we'll need to add a new semver tag and update the changelog to use that. Unfortunately there is nowhere earlier in the process we can detect this reliably since adding a semver tag is enough to trigger a binary release (first I considered using Danger, but that only applies to PRs not tags). Once an annotated tag is pushed it should not be changed, although I am not hardline on that in the case of failed builds, if it is changed quickly (since this is not a library we would be unlikely to break dependencies). Once any binary exists for a version in github releases, the tag should definitely not be modified under any circumstances.
In future it might be helpful to open PRs like this that update the changelog prior to tagging releases.
NOTE: This PR can be safely merged, but the merge commit should be tagged 0.5.120, which will trigger binary builds. ~That tagging is not safe to do yet, as changes here will have an impact on users so we need to communicate that first.~ EDIT: That's not true, this should be safe to tag, I was thinking #800 had been merged but not yet...

LGTM!
No errors, no warnings. :shipit:

  Generated by :no_entry_sign: Danger


Can not connect to local rest api

Describe the bug
A clear and concise description of what the bug is.
can not connect to local rest api, time out occurs.
To Reproduce
Steps to reproduce the behavior:

Code/Link to Repo:

pc init (in <IP_ADDRESS> server)
pc run (in <IP_ADDRESS> server)
import pynecone as pc
from pydantic import BaseModel
from typing import List
from typing import Dict
import requests
import json

'''
{"data": []}
'''
class Payload(BaseModel):
    data: List[Dict]

async def hello_api(payload: Payload):
    for datum in payload.data:
        #print(datum)
        pass
    return {"data": [{"message": "hello"}]}

class BaseState(pc.State):
    async def hello(self):
        print("test")
        url = "http://localhost:8000/hello_api" #can not connet to localhost rest api (blocked in requests.post line)
        #url = "http://<IP_ADDRESS>:8000/hello_api" #can not connet to localhost rest api by server's extermal address, too (blocked in requests.post line)
        payload = {"data": []}
        response = requests.post(url, json=payload) #blocked
        outputs = response.json()
        print(outputs)

def contents():
    return pc.button("hello", on_click=BaseState.hello)

app = pc.App(state=BaseState)
app.add_page(contents, route="/")
app.api.add_api_route("/hello_api", hello_api, methods=["POST"])
app.compile()

Expected behavior
A clear and concise description of what you expected to happen.
Should print {"data": [{"message": "hello"}]} message in browser.
Screenshots
If applicable, add screenshots to help explain your problem.
** Specifics (please complete the following information):**

Python Version: Python 3.9.13
Pynecone Version: 0.1.27
OS: Ubuntu 20.04
Browser (Optional): Chrome

Additional context
Add any other context about the problem here.
Thanks for supporting!

Do other apis work just your local is not connecting?

Do other apis work just your local is not connecting?

hi, can connet to external rest api well (not blockede)
see following example code.
Thank you your supporting.
url = "https://jsonplaceholder.typicode.com/posts/1" #can connet to external rest api well (not blockede)
response = requests.get(url) #not blocked
outputs = response.json()
print(outputs)

Another information:
Rest api works when using http://<IP_ADDRESS>:8000/docs with browser via fastapi rest api tool.

In summary:
local client (pynecone) to local rest api (pynecone) => not work
local client (pynecone) to remote rest api (pynecone) => work

This is happening to me as well.
Python 3.10 // Pynecone 0.1.28
def index() -> pc.Component:
    return pc.center(
        pc.vstack(
            pc.heading("Welcome to Hughmungus!", font_size="2em"),
            pc.box("Get started by editing ", pc.code(filename, font_size="1em")),
            pc.link(
                "Check out our docs!",
                href=docs_url,
                border="0.1em solid",
                padding="0.5em",
                border_radius="0.5em",
                _hover={
                    "color": "rgb(107,99,246)",
                },
            ),
            spacing="1.5em",
            font_size="2em",
        ),
        padding_top="10%",
    )

async def hello():
    return {"message": "Hello World!"}

# Add state and page to the app.
app = pc.App(state=State)
app.add_page(index, title="Hughmungus")
app.api.add_api_route("/api/hello", hello)
app.compile()

Running curl http://localhost:3000/api/hello returns a 404.

The api runs on the 8000 port - can you try curl http://localhost:8000/api/hello. We should make this more clear in the docs.

welp that worked, is there a reason why it the frontend and api run on two different ports? I'm assuming security/CORS related?

The frontend 3000 port is for the NextJS app and the backend 8000 port runs the FastAPI Python app

Blocking happens when in python not when external environment (curl ... etc)

Ran into this issue when making this example : https://github.com/reflex-dev/reflex-examples/pull/156
The core of the issue is that when the backend call itself over HTTP, it need to do so in async mode.
Without async, the backend softlock itself (the backend part doing the query doesn't return the control to let the backend part that answer the query run).


AllowedInstanceRoles

There is no reference to creating the  AllowedInstanceRoles  in the documentation.
I am using the sample app from the tutorial.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 8602bfd5-a60d-86fe-82ef-6c5fef624bf7
Version Independent ID: 6062ee0a-be91-0b72-44c7-98dfcaa899f4
Content: Using applications in Azure Blockchain Workbench
Content Source: articles/blockchain-workbench/blockchain-workbench-use.md
Service: azure-blockchain
GitHub Login: @PatAltimore
Microsoft Alias: patricka

@HowardEdidin Thanks for the feedback! I have assigned the issue to the content author to evaluate and update as appropriate.

@HowardEdidin Thanks for the feedback. I'm working on post build changes. I'll add this to the list.
#in-progress

@HowardEdidin Oh, I forgot...  Check out other samples here: https://github.com/Azure-Samples/blockchain

I am waiting for the IoT samples
Get Outlook for iOShttps://aka.ms/o0ukef

From: Pat Altimore<EMAIL_ADDRESS>Sent: Tuesday, May 8, 2018 4:03:47 PM
To: MicrosoftDocs/azure-docs
Cc: Howard Edidin; Mention
Subject: Re: [MicrosoftDocs/azure-docs] AllowedInstanceRoles (#8254)
@HowardEdidinhttps://github.com/HowardEdidin Oh, I forgot... Check out other samples here: https://github.com/Azure-Samples/blockchain
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/MicrosoftDocs/azure-docs/issues/8254#issuecomment-387541694, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ATB82dksV-reIp_YmqOeLgR5YxhpwSBNks5twggzgaJpZM4T25TT.

@HowardEdidin I've made a few updates to the "Using" article.  The issue you mentioned was an error. You actually need to be a member of Initiators to create a contract. Instance role discussion moved down into "take action". I explain a little bit and reference the configuration document. Also updated the configuration document to help clarify the difference between application roles and instance roles in the transition section.  Thanks for the feedback.  I'm going to request this issue be closed.
#please-close

@HowardEdidin
We will now close this issue. If there are further questions regarding this matter, please reopen it and we will gladly continue the discussion.


'NoneType' object has no attribute 'get'

Version of the custom_component

Configuration
config not used for the moment

Add your logs here.
Enregistreur: homeassistant.setup
Source: setup.py:404
S'est produit pour la première fois: 10:12:28 (1 occurrences)
Dernier enregistrement: 10:12:28

Error during setup of component solar_optimizer
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/setup.py", line 404, in _async_setup_component
    result = await task
             ^^^^^^^^^^
  File "/config/custom_components/solar_optimizer/__init__.py", line 97, in async_setup
    hass.data[DOMAIN]["coordinator"] = coordinator = SolarOptimizerCoordinator(
                                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/solar_optimizer/coordinator.py", line 64, in __init__
    raise err
  File "/config/custom_components/solar_optimizer/coordinator.py", line 56, in __init__
    for _, device in enumerate(config.get("devices")):
                               ^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'get'

Describe the bug
A clear and concise description of what the bug is.
impossible to get the plugin work
Debug log

done before

Add your logs here.
Enregistreur: homeassistant.setup
Source: setup.py:404
S'est produit pour la première fois: 10:12:28 (1 occurrences)
Dernier enregistrement: 10:12:28

Error during setup of component solar_optimizer
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/setup.py", line 404, in _async_setup_component
    result = await task
             ^^^^^^^^^^
  File "/config/custom_components/solar_optimizer/__init__.py", line 97, in async_setup
    hass.data[DOMAIN]["coordinator"] = coordinator = SolarOptimizerCoordinator(
                                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/solar_optimizer/coordinator.py", line 64, in __init__
    raise err
  File "/config/custom_components/solar_optimizer/coordinator.py", line 56, in __init__
    for _, device in enumerate(config.get("devices")):
                               ^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'get'

Hello,
Your configuration is not good. Please add a configuration or deactivate the integration.

Thanks for your answer.
I finally went through your solar optimizer integration . (I had not
created the yaml file, thought that the integration had to be run before)

I created the yaml file with the device I currently have and can manage.
specified the solar optimizer yam file in the config file
reconfigured the service integration and relaunched HA
incorporated the decluttering and expander cards in HACS.
I can now see the device and the associated power .
Next I may incorporate a new device to handle the swimming pool water pump
! ( this one is already managed by some automations and calendar so i have
to check whether it is a good idea)
Anyhow , thanks for your integration.

Le lun. 13 mai 2024 à 07:20, Jean-Marc Collin @.***> a
écrit :

Hello,
Your configuration is not good. Please add a configuration or deactivate
the integration.
—
Reply to this email directly, view it on GitHub
https://github.com/jmcollin78/solar_optimizer/issues/31#issuecomment-2106676442,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AG5WRE6OM7UZ7IKTHN33SILZCBERFAVCNFSM6AAAAABHR6CYOSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBWGY3TMNBUGI
.
You are receiving this because you authored the thread.Message ID:
@.***>

--
Cordialement


Issue 2324 setup script

Initial version of a Powershell setup script to handle some of the allReady prerequisite setup:

Installs Chocolatey
Installs and does initial config of Git (name, mail, long path)
Installs NodeJS LTS
Gives user option of configuring proxies for system and NPM
Installs Gulp and Bower
Reminds user they need VS 2017 or higher
Installs .NET Core

@jimholmes - I like what you started here and agree with some of the feedback from @dchristensen.  Are you planning to work on this further?

Sorry, I lost track of this last year. I'll work to get the suggestions incorporated. Apologies for going dark on it. Thank you for the poke!

@jimholmes No need for apologies.  My motives are selfish.  I manually rebuilt the development environment for AllReady in a VM multiple times in order to diagnose some build issues.  Your script would definitely simplify things and act as informal documentation.


Cannot install plugins on Windows

I installed Vim 8.0 on Windows 10 but can't get vundle to install the plugins listed in .vimrc, each time I run :PluginInstall I get this error:
C:\Program' is not recognized as an internal or external command,

and a lot of other errors, but I think this is the main one.
Vim was installed to C:\Vim.

@sryze can you post a minimal vimrc ?
Have you looked into the Windows Setup?

@sryze I'm experiencing this with current 64 bit nightly VIM - Vi IMproved 8.0 (2016 Sep 12, compiled Apr 23 2017 22:02:16:
set nocompatible              " be iMproved, required
filetype off                  " required
set rtp+=$HOME/vimfiles/bundle/Vundle.vim/
call vundle#begin('$HOME/vimfiles/bundle/')
Plugin 'VundleVim/Vundle.vim' " let Vundle manage Vundle, required
"Plugin 'Valloric/YouCompleteMe'  " see installation instructions at https://github.com/Valloric/YouCompleteMe
Plugin 'fatih/vim-go'
call vundle#end()            " required
filetype plugin indent on    " required

I have Vundle.Vim and YouCompleteMe folders in bundle/, I do not have a bundle/vim-go folder. Vundle.vim folder contains up-to-date clone of this repo.
:PluginList shows Vundle.Vim and vim-go. I have tried running :PluginClean, it warns that it will remove YouCompleteMe, flashes an error for an instant, then shows error 'C:\Program' is not recognized as an internal or external command, operable program or batch file.
The instantaneous error is
Processing 'YouCompleteMe'
Error detected while processing function vundle#installer#clean[25]..<SNR>20_process[11]..vundle#installer#run[8]..vundle#installer#delete[9]..<SNR>20_system:
line    1:
E484: Can't open file C:/Users/there/AppData/Local/Temp/VIoDAA3.tmp
Error detected while processing function vundle#installer#clean[25]..<SNR>20_process:
line   13:
E121: Undefined variable: s:last_status
E15: Invalid expression: 'error' == s:last_status
line   17:
E121: Undefined variable: s:last_status
E15: Invalid expression: 'updated' == s:last_status && empty(msg)

I have manually deleted some folders from bundle/ while trying to start my config fresh. But now I'm stuck because I can't find where Vundle.Vim is storing the state.

The YouCompleteMe errors and incorrect :PluginList output persists after removing everything in $VIM/bundle and re-cloning Vundle.Vim

I failed to mention I'm on windows 10 running git bash in hyper, with recent vim binaries

attn @sryze , xposting workarounds from vim-plug#539:

:PlugInstall works fine if i start gvim from the windows start menu.

I discovered i'm able to get it working by changing &shell in .vimrc to 'cmd.exe'. I'm not sure how that effects other things i may try to do, but it makes PlugInstall work!

"let &shell='/usr/bin/bash'
let &shell='cmd.exe'

The second workaround fixed it for me

perhaps @janlazo's fix is portable to vundle? https://github.com/junegunn/vim-plug/pull/676

Vim picks up SHELL. Windows Vim does not work well with shells other than cmd.exe.
For any command requiring the shell, after set shell=cmd.exe, there are at least 2 options:

Revert all shell-related options to work with cmd.exe (best done by the user in their vimrc)
Put the command in a temporary batchfile (make sure this batchfile has the bat or cmd extension)

junegunn/vim-plug#676 opts for the batchfile because it is already tested in the Vim plugin of fzf and fzf.vim. Note that the escape code required for 1 and 2 has slight differences regarding variables and literal escaping is used in vim-plug.

Is there any way to keep using bash while having Vundle work? Can I instruct vundle to use cmd.exe but keep using bash myself?


Wrong PropertyGroups in readme

The README.md specifies that AssemblyName, Description, and Version will generate the PLUGIN_GUID, PLUGIN_NAME, and PLUGIN_VERSION respectively. Quoted below:
Define the following properties in your csproj:
<AssemblyName>Example.Plugin</AssemblyName>
<Description>My first plugin</Description>
<Version>1.0.0</Version>

this will generate the following class:
using System;

internal static class MyPluginInfo
{
    public const string PLUGIN_GUID = "Example.Plugin";
    public const string PLUGIN_NAME = "My first plugin";
    public const string PLUGIN_VERSION = "1.0.0";
}

However, the props file is actually using AssemblyName, Product, and Version.
<BepInExPluginGuid Condition="'$(BepInExPluginGuid)' == ''">$(AssemblyName)</BepInExPluginGuid>
<BepInExPluginName Condition="'$(BepInExPluginName)' == ''">$(Product)</BepInExPluginName>
<BepInExPluginVersion Condition="'$(BepInExPluginVersion)' == ''">$(Version)</BepInExPluginVersion>

This would not be immediately obvious to somebody that is using the plugin but not cloning/using the props file manually, as they'd have to dig into the nuget reference to see the difference.
Edit: @ghorsington  I don't mind submitting a PR, but wasn't sure which one you'd like to be the correct one, the docs or the props?

I have to imagine that the readme should be updated, as to avoid breaking existing usage of this. Additionally, it probably makes more sense to use the "Product" property, since it represents a name more than the "Description" property should.
Product attribute: https://learn.microsoft.com/en-us/dotnet/api/system.reflection.assemblyproductattribute.product?view=net-7.0#system-reflection-assemblyproductattribute-product


fix: Fix InferGetStaticPropsType example

What kind of changes does this PR include?

Minor content fixes (broken links, typos, etc.)

Description
Our tooling does not automatically uses type Params to type Astro.params. It never has, actually. Additionally, the two slash commands were in the wrong spot

LGTM! Will we ever auto-type Astro.params? I guess it’s less important and also using params in a component would still need the as so maybe best just to leave it.

Maybe! We still have this old accepted RFC to implement eventually https://github.com/withastro/roadmap/blob/main/proposals/0026-component-interface.md


Specify HTTP 1.1 version

It seems that with new bridge version on main branch (maybe related to a Vert.x bump), the test-clients try to upgrade from protocol HTTP 1.1 to HTTP 2.0 which is not supported by the bridge and it closes the connection (the clients will raise and EOF exception on its side).

This PR specifies the HTTP 1.1 protocol in the HTTP request to fix this.

Works as expected, thanks :)