fix: ArtifactGC Fails for Stopped Workflows. Fixes #11879

Fixes #11879
Motivation

Currently, stopped workflows will happily save artifacts to a storage bucket but the artifacts won't get garbage collected after workflow deletion. The changes make it so that a stopped workflow's artifacts are properly garbage collected upon workflow deletion.
Modifications

The operator.go's call to woc.taskResultReconciliation() previously occurred after Artifact Garbage Collection (ArtifactGC) and an early return due to if woc.wf.Labels[common.LabelKeyCompleted] == "true", leading to a scenerio in which garbage collection would fail upon workflow deletion. This in turn would cause workflow deletion to get blocked by the ArtifactGC finalizer. woc.taskResultReconciliation() is responsible for updating the node status in the workflow based on the artifact outputs saved during workflow execution with reportOutputs() in executor.go.

Verification

I tested the changes by adding an e2e test called TestStoppedWorkflow which uses a custom workflow with the following logic.

Ensure no artifact exists in the bucket.
Start the workflow.
Workflow runs and creates a test artifact.
Verifiy the artifact exists in the bucket.
The workflow submits a CLI-based stop event for itself.
The workflow is deleted by the test.
Artifacts are verified to have been deleted.
Finalizer is removed incase the test failed.

I made some additional minor tweaks to other files, most notably, timeout = v - 30*time.Second + defaultTimeout in when.go which would cause custom timeout settings to be incorrect when using the WaitForWorkflow() e2e testing utility.

@juliev0 Could you help review when you get a chance?

Yes. It may be more than a couple of weeks from now, though, as I have a vacation followed by a Hackathon week at work. (It looks like this is still in "Draft" state as well.)

Failing test is RetryTestSuite/TestRetryBackoff

Failing test is RetryTestSuite/TestRetryBackoff

This was fixed by adding the 30s. Good catch @juliev0

Sadly my test (TestStoppedWorkflow) ran into the intermittent error I was receiving earlier. I'll try and debug it.

Sadly my test (TestStoppedWorkflow) ran into the intermittent error I was receiving earlier. I'll try and debug it.

Sorry to hear. Thank you!

@agilgur5 @juliev0
I'm having a hard time debugging this. The failure rate is like 1/30+ test runs on my local machine. The artifacts are being saved, but the task reconciliation never receives the on-deletion-wf-stopped artifact for some reason. Task reconciliation does, however, receive the main-logs artifact. My best guess is that somehow the artifact gets saved in the wait container but fails to be properly written in reportOutputs / `reportResult.
Is there an easy way to write the wait container logs to a file in the same way that the controller logs are written to a file in the logs directory? Without this, It's basically impossible for me to debug it.

@agilgur5 @juliev0
I'm having a hard time debugging this. The failure rate is like 1/30+ test runs on my local machine. The artifacts are being saved, but the task reconciliation never receives the on-deletion-wf-stopped artifact for some reason. Task reconciliation does, however, receive the main-logs artifact. My best guess is that somehow the artifact gets saved in the wait container but fails to be properly written in reportOutputs / reportResult.
Is there an easy way to write the wait container logs to a file in the same way that the controller logs are written to a file in the logs directory? Without this, It's basically impossible for me to debug it.

Interesting. So, is the difference between state of "latest" branch and this one that basically it fails for stopped workflows 1/30 times (at least in this unit test) vs 100%? Are you seeing it fail more often than that in the CI, though?
As far as the wait container logs, no, I don't know of a wait container log artifact unfortunately. But maybe you could run something like: kubectl logs --follow -lworkflows.argoproj.io/workflow=<name> --container wait. Based on this post, it seems like that should follow newly created pods after you start running.

Interesting. So, is the difference between state of "latest" branch and this one that basically it fails for stopped workflows 1/30 times (at least in this unit test) vs 100%?

Yeah, it should fail 100% of the time on the latest branch. I'll double check tomorrow by undoing my changes locally and run the test a few times.

Are you seeing it fail more often than that in the CI, though?

I will resubmit the CI job tomorrow to see. If it fails again, then yes.

kubectl logs --follow -lworkflows.argoproj.io/workflow= --container wait

Thanks for this, I will test it out.

Interesting. So, is the difference between state of "latest" branch and this one that basically it fails for stopped workflows 1/30 times (at least in this unit test) vs 100%? Are you seeing it fail more often than that in the CI, though?

I've confirmed the test fails 100% of the time on latest.

But maybe you could run something like: kubectl logs --follow -lworkflows.argoproj.io/workflow= --container=wait. Based on this post, it seems like that should follow newly created pods after you start running.

I ended up using the following command to watch the logs between multiple test runs so that I can rapid fire the tests until I hit a failure. Note that I added the template=artgc-dag-artifact-creator label to the artgc-dag-artifact-creator template locally.
kubectl get pods -l 'template=artgc-dag-artifact-creator' --watch -o name | while read -r pod; do kubectl logs -f -c wait "$pod"; done

As a result of the above, I was able to confirm that the workflowtaskresults are being properly created (Create workflowtaskresults 201).
Maybe a fresh set of eyes can help.
Here is an excerpt of the controller logs of a failure (this is from near the end of the test):
time="2023-10-11T20:02:54.540Z" level=info msg="Processing workflow" namespace=argo workflow=artgc-dag-wf-stopped-hllgt
time="2023-10-11T20:02:54.541Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=artgc-dag-wf-stopped-hllgt
time="2023-10-11T20:02:54.541Z" level=debug msg="woc wf status nodes: map[artgc-dag-wf-stopped-hllgt:{artgc-dag-wf-stopped-hllgt artgc-dag-wf-stopped-hllgt artgc-dag-wf-stopped-hllgt DAG artgc-dag-wf-stopped-main nil local/artgc-dag-wf-stopped-hllgt Failed   2023-10-11 20:02:25 +0000 UTC 2023-10-11 20:02:48 +0000 UTC %!s(v1alpha1.EstimatedDuration=0) 1/3 6s*(1 cpu),4s*(100Mi memory)  %!s(*bool=<nil>) nil nil [artgc-dag-wf-stopped-hllgt-3193530808 artgc-dag-wf-stopped-hllgt-2203728917] [artgc-dag-wf-stopped-hllgt-3193530808 artgc-dag-wf-stopped-hllgt-3113225067]  nil nil} artgc-dag-wf-stopped-hllgt-2203728917:{artgc-dag-wf-stopped-hllgt-2203728917 artgc-dag-wf-stopped-hllgt.delay-stop-workflow delay-stop-workflow Pod artgc-dag-delay-stop nil local/artgc-dag-wf-stopped-hllgt Succeeded artgc-dag-wf-stopped-hllgt  2023-10-11 20:02:25 +0000 UTC 2023-10-11 20:02:41 +0000 UTC %!s(v1alpha1.EstimatedDuration=0) 1/1 6s*(1 cpu),4s*(100Mi memory)  %!s(*bool=<nil>) nil &Outputs{Parameters:[]Parameter{},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:&S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:artgc-dag-wf-stopped-hllgt/artgc-dag-wf-stopped-hllgt-artgc-dag-delay-stop-2203728917/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:*0,} [artgc-dag-wf-stopped-hllgt-3113225067] [] k3d-k3s-default-server-0 nil nil} artgc-dag-wf-stopped-hllgt-3113225067:{artgc-dag-wf-stopped-hllgt-3113225067 artgc-dag-wf-stopped-hllgt.stop-workflow stop-workflow Pod artgc-dag-workflow-stopper nil local/artgc-dag-wf-stopped-hllgt Failed artgc-dag-wf-stopped-hllgt workflow shutdown with strategy:  Stop 2023-10-11 20:02:43 +0000 UTC 2023-10-11 20:02:48 +0000 UTC %!s(v1alpha1.EstimatedDuration=0) 0/1   %!s(*bool=<nil>) nil &Outputs{Parameters:[]Parameter{},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:&S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:artgc-dag-wf-stopped-hllgt/artgc-dag-wf-stopped-hllgt-artgc-dag-workflow-stopper-3113225067/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,} [] [] k3d-k3s-default-server-0 nil nil} artgc-dag-wf-stopped-hllgt-3193530808:{artgc-dag-wf-stopped-hllgt-3193530808 artgc-dag-wf-stopped-hllgt.create-artifact create-artifact Pod artgc-dag-artifact-creator nil local/artgc-dag-wf-stopped-hllgt Failed artgc-dag-wf-stopped-hllgt workflow shutdown with strategy:  Stop 2023-10-11 20:02:25 +0000 UTC 2023-10-11 20:02:48 +0000 UTC %!s(v1alpha1.EstimatedDuration=0) 0/1   %!s(*bool=<nil>) nil nil [] [] k3d-k3s-default-server-0 nil nil}]" namespace=argo workflow=artgc-dag-wf-stopped-hllgt
time="2023-10-11T20:02:54.541Z" level=debug msg="woc wf status outputs: nil" namespace=argo workflow=artgc-dag-wf-stopped-hllgt
time="2023-10-11T20:02:54.541Z" level=debug msg="woc wf artifactgc status: &ArtGCStatus{StrategiesProcessed:map[ArtifactGCStrategy]bool{OnWorkflowCompletion: true,OnWorkflowDeletion: true,},PodsRecouped:map[string]bool{},NotSpecified:false,}" namespace=argo workflow=artgc-dag-wf-stopped-hllgt
time="2023-10-11T20:02:54.541Z" level=info msg="reconciling artifact-gc pod" message= namespace=argo phase=Succeeded pod=artgc-dag-wf-stopped-hllgt-artgc-wfdel-2470140894 workflow=artgc-dag-wf-stopped-hllgt
time="2023-10-11T20:02:54.541Z" level=info msg="processing completed Artifact GC Pod \"artgc-dag-wf-stopped-hllgt-artgc-wfdel-2470140894\"" namespace=argo workflow=artgc-dag-wf-stopped-hllgt
time="2023-10-11T20:02:54.545Z" level=debug msg="List workflowartifactgctasks 200"
time="2023-10-11T20:02:54.546Z" level=debug msg="processing WorkflowArtifactGCTask artgc-dag-wf-stopped-hllgt-artgc-wfdel-2470140894-0" namespace=argo workflow=artgc-dag-wf-stopped-hllgt
time="2023-10-11T20:02:54.546Z" level=debug msg="deleting WorkflowArtifactGCTask: artgc-dag-wf-stopped-hllgt-artgc-wfdel-2470140894-0" namespace=argo workflow=artgc-dag-wf-stopped-hllgt
time="2023-10-11T20:02:54.556Z" level=debug msg="Delete workflowartifactgctasks 200"
time="2023-10-11T20:02:54.556Z" level=info msg="no remaining artifacts to GC, removing artifact GC finalizer (forceFinalizerRemoval=false)" namespace=argo workflow=artgc-dag-wf-stopped-hllgt
time="2023-10-11T20:02:54.557Z" level=debug msg="Log changes patch: {\"metadata\":{\"finalizers\":null},\"status\":{\"artifactGCStatus\":{\"podsRecouped\":{\"artgc-dag-wf-stopped-hllgt-artgc-wfdel-2470140894\":true}},\"nodes\":{\"artgc-dag-wf-stopped-hllgt-2203728917\":{\"outputs\":{\"artifacts\":[{\"deleted\":true,\"name\":\"main-logs\",\"s3\":{\"key\":\"artgc-dag-wf-stopped-hllgt/artgc-dag-wf-stopped-hllgt-artgc-dag-delay-stop-2203728917/main.log\"}}]}},\"artgc-dag-wf-stopped-hllgt-3113225067\":{\"outputs\":{\"artifacts\":[{\"deleted\":true,\"name\":\"main-logs\",\"s3\":{\"key\":\"artgc-dag-wf-stopped-hllgt/artgc-dag-wf-stopped-hllgt-artgc-dag-workflow-stopper-3113225067/main.log\"}}]}}}}}"
time="2023-10-11T20:02:54.557Z" level=info msg="Workflow to be dehydrated" Workflow Size=4597
time="2023-10-11T20:02:54.573Z" level=debug msg="Update workflows 200"
time="2023-10-11T20:02:54.573Z" level=info msg="Workflow update successful" namespace=argo phase=Failed resourceVersion=392281 workflow=artgc-dag-wf-stopped-hllgt
time="2023-10-11T20:02:54.590Z" level=debug msg="DeleteCollection workflowtaskresults 200"

Here is an excerpt of the controller logs of a success (this is from near the end of the test):
time="2023-10-11T20:02:00.286Z" level=info msg="Processing workflow" namespace=argo workflow=artgc-dag-wf-stopped-7zxmc
time="2023-10-11T20:02:00.287Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=artgc-dag-wf-stopped-7zxmc
time="2023-10-11T20:02:00.287Z" level=debug msg="woc wf status nodes: map[artgc-dag-wf-stopped-7zxmc:{artgc-dag-wf-stopped-7zxmc artgc-dag-wf-stopped-7zxmc artgc-dag-wf-stopped-7zxmc DAG artgc-dag-wf-stopped-main nil local/artgc-dag-wf-stopped-7zxmc Failed   2023-10-11 20:01:29 +0000 UTC 2023-10-11 20:01:53 +0000 UTC %!s(v1alpha1.EstimatedDuration=0) 1/3 6s*(1 cpu),4s*(100Mi memory)  %!s(*bool=<nil>) nil nil [artgc-dag-wf-stopped-7zxmc-141800804 artgc-dag-wf-stopped-7zxmc-1982811905] [artgc-dag-wf-stopped-7zxmc-141800804 artgc-dag-wf-stopped-7zxmc-2210815367]  nil nil} artgc-dag-wf-stopped-7zxmc-141800804:{artgc-dag-wf-stopped-7zxmc-141800804 artgc-dag-wf-stopped-7zxmc.create-artifact create-artifact Pod artgc-dag-artifact-creator nil local/artgc-dag-wf-stopped-7zxmc Failed artgc-dag-wf-stopped-7zxmc workflow shutdown with strategy:  Stop 2023-10-11 20:01:29 +0000 UTC 2023-10-11 20:01:53 +0000 UTC %!s(v1alpha1.EstimatedDuration=0) 0/1   %!s(*bool=<nil>) nil &Outputs{Parameters:[]Parameter{},Artifacts:[]Artifact{Artifact{Name:on-deletion-wf-stopped,Path:/mnt/vol/test.txt,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:&S3Artifact{S3Bucket:S3Bucket{Endpoint:minio:9000,Bucket:my-bucket-3,Region:,Insecure:*true,AccessKeySecret:&v1.SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:my-minio-cred,},Key:accesskey,Optional:nil,},SecretKeySecret:&v1.SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:my-minio-cred,},Key:secretkey,Optional:nil,},RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:on-deletion-wf-stopped,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:&ArtifactGC{Strategy:OnWorkflowDeletion,PodMetadata:nil,ServiceAccountName:,},Deleted:false,},Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:&S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:artgc-dag-wf-stopped-7zxmc/artgc-dag-wf-stopped-7zxmc-artgc-dag-artifact-creator-141800804/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,} [] [] k3d-k3s-default-server-0 nil nil} artgc-dag-wf-stopped-7zxmc-1982811905:{artgc-dag-wf-stopped-7zxmc-1982811905 artgc-dag-wf-stopped-7zxmc.delay-stop-workflow delay-stop-workflow Pod artgc-dag-delay-stop nil local/artgc-dag-wf-stopped-7zxmc Succeeded artgc-dag-wf-stopped-7zxmc  2023-10-11 20:01:29 +0000 UTC 2023-10-11 20:01:44 +0000 UTC %!s(v1alpha1.EstimatedDuration=0) 1/1 4s*(100Mi memory),6s*(1 cpu)  %!s(*bool=<nil>) nil &Outputs{Parameters:[]Parameter{},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:&S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:artgc-dag-wf-stopped-7zxmc/artgc-dag-wf-stopped-7zxmc-artgc-dag-delay-stop-1982811905/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:*0,} [artgc-dag-wf-stopped-7zxmc-2210815367] [] k3d-k3s-default-server-0 nil nil} artgc-dag-wf-stopped-7zxmc-2210815367:{artgc-dag-wf-stopped-7zxmc-2210815367 artgc-dag-wf-stopped-7zxmc.stop-workflow stop-workflow Pod artgc-dag-workflow-stopper nil local/artgc-dag-wf-stopped-7zxmc Failed artgc-dag-wf-stopped-7zxmc workflow shutdown with strategy:  Stop 2023-10-11 20:01:47 +0000 UTC 2023-10-11 20:01:53 +0000 UTC %!s(v1alpha1.EstimatedDuration=0) 0/1   %!s(*bool=<nil>) nil &Outputs{Parameters:[]Parameter{},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:&S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:artgc-dag-wf-stopped-7zxmc/artgc-dag-wf-stopped-7zxmc-artgc-dag-workflow-stopper-2210815367/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,} [] [] k3d-k3s-default-server-0 nil nil}]" namespace=argo workflow=artgc-dag-wf-stopped-7zxmc
time="2023-10-11T20:02:00.287Z" level=debug msg="woc wf status outputs: nil" namespace=argo workflow=artgc-dag-wf-stopped-7zxmc
time="2023-10-11T20:02:00.287Z" level=debug msg="woc wf artifactgc status: &ArtGCStatus{StrategiesProcessed:map[ArtifactGCStrategy]bool{OnWorkflowCompletion: true,OnWorkflowDeletion: true,},PodsRecouped:map[string]bool{},NotSpecified:false,}" namespace=argo workflow=artgc-dag-wf-stopped-7zxmc
time="2023-10-11T20:02:00.287Z" level=info msg="reconciling artifact-gc pod" message= namespace=argo phase=Succeeded pod=artgc-dag-wf-stopped-7zxmc-artgc-wfdel-2470140894 workflow=artgc-dag-wf-stopped-7zxmc
time="2023-10-11T20:02:00.287Z" level=info msg="processing completed Artifact GC Pod \"artgc-dag-wf-stopped-7zxmc-artgc-wfdel-2470140894\"" namespace=argo workflow=artgc-dag-wf-stopped-7zxmc
time="2023-10-11T20:02:00.292Z" level=debug msg="List workflowartifactgctasks 200"
time="2023-10-11T20:02:00.292Z" level=debug msg="processing WorkflowArtifactGCTask artgc-dag-wf-stopped-7zxmc-artgc-wfdel-2470140894-0" namespace=argo workflow=artgc-dag-wf-stopped-7zxmc
time="2023-10-11T20:02:00.292Z" level=debug msg="deleting WorkflowArtifactGCTask: artgc-dag-wf-stopped-7zxmc-artgc-wfdel-2470140894-0" namespace=argo workflow=artgc-dag-wf-stopped-7zxmc
time="2023-10-11T20:02:00.364Z" level=debug msg="Delete workflowartifactgctasks 200"
time="2023-10-11T20:02:00.364Z" level=info msg="no remaining artifacts to GC, removing artifact GC finalizer (forceFinalizerRemoval=false)" namespace=argo workflow=artgc-dag-wf-stopped-7zxmc
time="2023-10-11T20:02:00.365Z" level=debug msg="Log changes patch: {\"metadata\":{\"finalizers\":null},\"status\":{\"artifactGCStatus\":{\"podsRecouped\":{\"artgc-dag-wf-stopped-7zxmc-artgc-wfdel-2470140894\":true}},\"nodes\":{\"artgc-dag-wf-stopped-7zxmc-141800804\":{\"outputs\":{\"artifacts\":[{\"artifactGC\":{\"strategy\":\"OnWorkflowDeletion\"},\"deleted\":true,\"name\":\"on-deletion-wf-stopped\",\"path\":\"/mnt/vol/test.txt\",\"s3\":{\"accessKeySecret\":{\"key\":\"accesskey\",\"name\":\"my-minio-cred\"},\"bucket\":\"my-bucket-3\",\"endpoint\":\"minio:9000\",\"insecure\":true,\"key\":\"on-deletion-wf-stopped\",\"secretKeySecret\":{\"key\":\"secretkey\",\"name\":\"my-minio-cred\"}}},{\"deleted\":true,\"name\":\"main-logs\",\"s3\":{\"key\":\"artgc-dag-wf-stopped-7zxmc/artgc-dag-wf-stopped-7zxmc-artgc-dag-artifact-creator-141800804/main.log\"}}]}},\"artgc-dag-wf-stopped-7zxmc-1982811905\":{\"outputs\":{\"artifacts\":[{\"deleted\":true,\"name\":\"main-logs\",\"s3\":{\"key\":\"artgc-dag-wf-stopped-7zxmc/artgc-dag-wf-stopped-7zxmc-artgc-dag-delay-stop-1982811905/main.log\"}}]}},\"artgc-dag-wf-stopped-7zxmc-2210815367\":{\"outputs\":{\"artifacts\":[{\"deleted\":true,\"name\":\"main-logs\",\"s3\":{\"key\":\"artgc-dag-wf-stopped-7zxmc/artgc-dag-wf-stopped-7zxmc-artgc-dag-workflow-stopper-2210815367/main.log\"}}]}}}}}"
time="2023-10-11T20:02:00.365Z" level=info msg="Workflow to be dehydrated" Workflow Size=4950
time="2023-10-11T20:02:00.384Z" level=debug msg="Update workflows 200"
time="2023-10-11T20:02:00.385Z" level=info msg="Workflow update successful" namespace=argo phase=Failed resourceVersion=392104 workflow=artgc-dag-wf-stopped-7zxmc
time="2023-10-11T20:02:00.407Z" level=debug msg="DeleteCollection workflowtaskresults 200"
time="2023-10-11T20:02:01.388Z" level=debug msg="Won't process Workflow since it's completed" key=argo/artgc-dag-wf-stopped-7zxmc

The woc wf status nodes and woc wf artifactgc status logs are after woc.taskResultReconciliation in operator.go. Notice how the on-deletion-wf-stopped is present in the success case and not present in the failure case. I've verified the result passed in reportResult of executor.go includes both the on-deletion-wf-stopped and main-logs artifacts.
For example, here is a result logged from reportResult for a failure case:
{  &Outputs{Parameters:[]Parameter{},Artifacts:[]Artifact{Artifact{Name:on-deletion-wf-stopped,Path:/mnt/vol/test.txt,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:&S3Artifact{S3Bucket:S3Bucket{Endpoint:minio:9000,Bucket:my-bucket-3,Region:,Insecure:*true,AccessKeySecret:&v1.SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:my-minio-cred,},Key:accesskey,Optional:nil,},SecretKeySecret:&v1.SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:my-minio-cred,},Key:secretkey,Optional:nil,},RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:on-deletion-wf-stopped,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:&ArtifactGC{Strategy:OnWorkflowDeletion,PodMetadata:nil,ServiceAccountName:,},Deleted:false,},Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:&S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:artgc-dag-wf-stopped-hllgt/artgc-dag-wf-stopped-hllgt-artgc-dag-artifact-creator-3193530808/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,} }"

At this point, I'm peppering logs in taskResultReconciliation to see if there may be an issue there. Perhaps there is a bug in the reconcoliation logic? Will follow up shortly.

on-deletion-wf-stopped is not referenced a single time during taskResultReconciliation() in the failure case. I'm lost.
Here's the logs:
time="2023-10-11T22:19:51.054Z" level=info msg="Processing workflow" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:51.055Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main"
time="2023-10-11T22:19:51.055Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:51.055Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:51.055Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:51.055Z" level=debug msg="Getting the template by name: artgc-dag-artifact-creator"
time="2023-10-11T22:19:51.055Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:19:51.055Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:19:51.055Z" level=debug msg="Getting the template by name: artgc-dag-artifact-creator" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:19:51.055Z" level=debug msg="Getting the template by name: artgc-dag-delay-stop" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:19:51.055Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:19:51.055Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:19:51.055Z" level=debug msg="Getting the template by name: artgc-dag-delay-stop" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:19:51.055Z" level=debug msg="Getting the template by name: artgc-dag-workflow-stopper" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:19:51.055Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-workflow-stopper)"
time="2023-10-11T22:19:51.055Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-workflow-stopper)"
time="2023-10-11T22:19:51.055Z" level=debug msg="Getting the template by name: artgc-dag-workflow-stopper" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-workflow-stopper)"
time="2023-10-11T22:19:51.055Z" level=debug msg="Getting the template by name: artgc-dag-artifact-creator" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-workflow-stopper)"
time="2023-10-11T22:19:51.055Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:19:51.055Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:19:51.055Z" level=debug msg="Getting the template by name: artgc-dag-artifact-creator" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:19:51.056Z" level=debug msg="Getting the template by name: artgc-dag-delay-stop" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:19:51.056Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:19:51.056Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:19:51.056Z" level=debug msg="Getting the template by name: artgc-dag-delay-stop" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:19:51.056Z" level=debug msg="Getting the template by name: artgc-dag-workflow-stopper" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:19:51.056Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-workflow-stopper)"
time="2023-10-11T22:19:51.056Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-workflow-stopper)"
time="2023-10-11T22:19:51.056Z" level=debug msg="Getting the template by name: artgc-dag-workflow-stopper" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-workflow-stopper)"
time="2023-10-11T22:19:51.056Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:51.056Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:51.056Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:51.056Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:51.056Z" level=debug msg="Getting the template by name: artgc-dag-delay-stop" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:51.056Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-delay-stop)"
time="2023-10-11T22:19:51.056Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-delay-stop)"
time="2023-10-11T22:19:51.056Z" level=debug msg="Getting the template by name: artgc-dag-delay-stop" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-delay-stop)"
time="2023-10-11T22:19:51.056Z" level=debug msg="Getting the template by name: artgc-dag-workflow-stopper" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-delay-stop)"
time="2023-10-11T22:19:51.056Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-workflow-stopper)"
time="2023-10-11T22:19:51.056Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-workflow-stopper)"
time="2023-10-11T22:19:51.056Z" level=debug msg="Getting the template by name: artgc-dag-workflow-stopper" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-workflow-stopper)"
time="2023-10-11T22:19:51.056Z" level=debug msg="Getting the template by name: artgc-dag-artifact-creator" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-workflow-stopper)"
time="2023-10-11T22:19:51.056Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-artifact-creator)"
time="2023-10-11T22:19:51.056Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-artifact-creator)"
time="2023-10-11T22:19:51.056Z" level=debug msg="Getting the template by name: artgc-dag-artifact-creator" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-artifact-creator)"
time="2023-10-11T22:19:51.062Z" level=debug msg="Get configmaps 200"
time="2023-10-11T22:19:51.062Z" level=info msg="resolved artifact repository" artifactRepositoryRef="argo/#"
time="2023-10-11T22:19:51.062Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:51.062Z" level=info msg="adding artifact GC finalizer" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:51.062Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:51.062Z" level=info msg="Creating pvc artgc-dag-wf-stopped-kg8tk-artifacts" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:51.063Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"argo\", Name:\"artgc-dag-wf-stopped-kg8tk\", UID:\"a79f56c0-9c76-444e-b499-749078a78c6a\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"405097\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowRunning' Workflow Running"
time="2023-10-11T22:19:51.071Z" level=debug msg="Create events 201"
time="2023-10-11T22:19:51.076Z" level=debug msg="Create persistentvolumeclaims 201"
time="2023-10-11T22:19:51.077Z" level=debug msg="Evaluating node artgc-dag-wf-stopped-kg8tk: template: *v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main), boundaryID: " namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:51.077Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:51.077Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:51.077Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:51.077Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:51.077Z" level=debug msg="Initializing node artgc-dag-wf-stopped-kg8tk: template: *v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main), boundaryID: " namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:51.077Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:51.077Z" level=info msg="DAG node artgc-dag-wf-stopped-kg8tk initialized Running" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:51.077Z" level=warning msg="was unable to obtain the node for artgc-dag-wf-stopped-kg8tk-3846913764, taskName create-artifact"
time="2023-10-11T22:19:51.077Z" level=warning msg="was unable to obtain the node for artgc-dag-wf-stopped-kg8tk-3846913764, taskName create-artifact"
time="2023-10-11T22:19:51.077Z" level=info msg="All of node artgc-dag-wf-stopped-kg8tk.create-artifact dependencies [] completed" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:51.077Z" level=debug msg="Evaluating node artgc-dag-wf-stopped-kg8tk.create-artifact: template: *v1alpha1.DAGTask (artgc-dag-artifact-creator), boundaryID: artgc-dag-wf-stopped-kg8tk" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:51.077Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:51.077Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:19:51.077Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:19:51.077Z" level=debug msg="Getting the template by name: artgc-dag-artifact-creator" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:19:51.078Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:51.078Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:51.078Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:51.078Z" level=debug msg="Initializing node artgc-dag-wf-stopped-kg8tk.create-artifact: template: *v1alpha1.DAGTask (artgc-dag-artifact-creator), boundaryID: artgc-dag-wf-stopped-kg8tk" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:51.078Z" level=info msg="Pod node artgc-dag-wf-stopped-kg8tk-3846913764 initialized Pending" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:51.078Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:51.078Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:51.078Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:51.078Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.create-artifact with container template: artgc-dag-artifact-creator\n" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:51.078Z" level=debug namespace=argo needLocation=true workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:51.086Z" level=debug msg="Get serviceaccounts 200"
time="2023-10-11T22:19:51.089Z" level=debug msg="Creating Pod: artgc-dag-wf-stopped-kg8tk.create-artifact (artgc-dag-wf-stopped-kg8tk-artgc-dag-artifact-creator-3846913764)" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:51.098Z" level=debug msg="Create pods 201"
W1011 22:19:51.098305   22635 warnings.go:70] metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must be no more than 63 characters]
time="2023-10-11T22:19:51.099Z" level=info msg="Created pod: artgc-dag-wf-stopped-kg8tk.create-artifact (artgc-dag-wf-stopped-kg8tk-artgc-dag-artifact-creator-3846913764)" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:51.099Z" level=warning msg="was unable to obtain the node for artgc-dag-wf-stopped-kg8tk-1504799495, taskName stop-workflow"
time="2023-10-11T22:19:51.099Z" level=warning msg="was unable to obtain the node for artgc-dag-wf-stopped-kg8tk-3444961409, taskName delay-stop-workflow"
time="2023-10-11T22:19:51.099Z" level=warning msg="was unable to obtain the node for artgc-dag-wf-stopped-kg8tk-3444961409, taskName delay-stop-workflow"
time="2023-10-11T22:19:51.099Z" level=info msg="All of node artgc-dag-wf-stopped-kg8tk.delay-stop-workflow dependencies [] completed" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:51.099Z" level=debug msg="Evaluating node artgc-dag-wf-stopped-kg8tk.delay-stop-workflow: template: *v1alpha1.DAGTask (artgc-dag-delay-stop), boundaryID: artgc-dag-wf-stopped-kg8tk" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:51.099Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:51.099Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:19:51.099Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:19:51.099Z" level=debug msg="Getting the template by name: artgc-dag-delay-stop" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:19:51.099Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:51.099Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:51.099Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:51.099Z" level=debug msg="Initializing node artgc-dag-wf-stopped-kg8tk.delay-stop-workflow: template: *v1alpha1.DAGTask (artgc-dag-delay-stop), boundaryID: artgc-dag-wf-stopped-kg8tk" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:51.099Z" level=info msg="Pod node artgc-dag-wf-stopped-kg8tk-3444961409 initialized Pending" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:51.099Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:51.099Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:51.099Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:51.099Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.delay-stop-workflow with container template: artgc-dag-delay-stop\n" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:51.099Z" level=debug namespace=argo needLocation=true workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:51.104Z" level=debug msg="Get serviceaccounts 200"
time="2023-10-11T22:19:51.106Z" level=debug msg="Creating Pod: artgc-dag-wf-stopped-kg8tk.delay-stop-workflow (artgc-dag-wf-stopped-kg8tk-artgc-dag-delay-stop-3444961409)" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:51.114Z" level=debug msg="Create pods 201"
time="2023-10-11T22:19:51.115Z" level=info msg="Created pod: artgc-dag-wf-stopped-kg8tk.delay-stop-workflow (artgc-dag-wf-stopped-kg8tk-artgc-dag-delay-stop-3444961409)" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:51.115Z" level=warning msg="was unable to obtain the node for artgc-dag-wf-stopped-kg8tk-1504799495, taskName stop-workflow"
time="2023-10-11T22:19:51.115Z" level=warning msg="was unable to obtain the node for artgc-dag-wf-stopped-kg8tk-1504799495, taskName stop-workflow"
time="2023-10-11T22:19:51.115Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:51.115Z" level=info msg=reconcileAgentPod namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:51.116Z" level=debug msg="Log changes patch: {\"metadata\":{\"annotations\":{\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"finalizers\":[\"workflows.argoproj.io/artifact-gc\"],\"labels\":{\"workflows.argoproj.io/phase\":\"Running\"}},\"spec\":{\"activeDeadlineSeconds\":300,\"podSpecPatch\":\"terminationGracePeriodSeconds: 3\\n\"},\"status\":{\"artifactGCStatus\":{},\"artifactRepositoryRef\":{\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"accessKeySecret\":{\"key\":\"accesskey\",\"name\":\"my-minio-cred\"},\"bucket\":\"my-bucket\",\"endpoint\":\"minio:9000\",\"insecure\":true,\"secretKeySecret\":{\"key\":\"secretkey\",\"name\":\"my-minio-cred\"}}},\"configMap\":\"artifact-repositories\",\"key\":\"default-v1\",\"namespace\":\"argo\"},\"nodes\":{\"artgc-dag-wf-stopped-kg8tk\":{\"children\":[\"artgc-dag-wf-stopped-kg8tk-3846913764\",\"artgc-dag-wf-stopped-kg8tk-3444961409\"],\"displayName\":\"artgc-dag-wf-stopped-kg8tk\",\"finishedAt\":null,\"id\":\"artgc-dag-wf-stopped-kg8tk\",\"name\":\"artgc-dag-wf-stopped-kg8tk\",\"phase\":\"Running\",\"startedAt\":\"2023-10-11T22:19:51Z\",\"templateName\":\"artgc-dag-wf-stopped-main\",\"templateScope\":\"local/artgc-dag-wf-stopped-kg8tk\",\"type\":\"DAG\"},\"artgc-dag-wf-stopped-kg8tk-3444961409\":{\"boundaryID\":\"artgc-dag-wf-stopped-kg8tk\",\"displayName\":\"delay-stop-workflow\",\"finishedAt\":null,\"id\":\"artgc-dag-wf-stopped-kg8tk-3444961409\",\"name\":\"artgc-dag-wf-stopped-kg8tk.delay-stop-workflow\",\"phase\":\"Pending\",\"startedAt\":\"2023-10-11T22:19:51Z\",\"templateName\":\"artgc-dag-delay-stop\",\"templateScope\":\"local/artgc-dag-wf-stopped-kg8tk\",\"type\":\"Pod\"},\"artgc-dag-wf-stopped-kg8tk-3846913764\":{\"boundaryID\":\"artgc-dag-wf-stopped-kg8tk\",\"displayName\":\"create-artifact\",\"finishedAt\":null,\"id\":\"artgc-dag-wf-stopped-kg8tk-3846913764\",\"name\":\"artgc-dag-wf-stopped-kg8tk.create-artifact\",\"phase\":\"Pending\",\"startedAt\":\"2023-10-11T22:19:51Z\",\"templateName\":\"artgc-dag-artifact-creator\",\"templateScope\":\"local/artgc-dag-wf-stopped-kg8tk\",\"type\":\"Pod\"}},\"persistentVolumeClaims\":[{\"name\":\"artifacts\",\"persistentVolumeClaim\":{\"claimName\":\"artgc-dag-wf-stopped-kg8tk-artifacts\"}}],\"phase\":\"Running\",\"startedAt\":\"2023-10-11T22:19:51Z\"}}"
time="2023-10-11T22:19:51.116Z" level=info msg="Workflow to be dehydrated" Workflow Size=3063
time="2023-10-11T22:19:51.125Z" level=debug msg="Update workflows 200"
time="2023-10-11T22:19:51.126Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=405106 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:51.128Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"argo\", Name:\"artgc-dag-wf-stopped-kg8tk\", UID:\"a79f56c0-9c76-444e-b499-749078a78c6a\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"405106\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeRunning' Running node artgc-dag-wf-stopped-kg8tk"
time="2023-10-11T22:19:51.135Z" level=debug msg="Create events 201"
time="2023-10-11T22:19:52.055Z" level=info msg="Processing workflow" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:52.056Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:52.056Z" level=info msg="node unchanged" namespace=argo nodeID=artgc-dag-wf-stopped-kg8tk-3444961409 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:52.056Z" level=info msg="node unchanged" namespace=argo nodeID=artgc-dag-wf-stopped-kg8tk-3846913764 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:52.056Z" level=debug msg="Evaluating node artgc-dag-wf-stopped-kg8tk: template: *v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main), boundaryID: " namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:52.056Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:52.056Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:52.056Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:52.056Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk of DAG is Running" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:52.056Z" level=debug msg="Evaluating node artgc-dag-wf-stopped-kg8tk.create-artifact: template: *v1alpha1.DAGTask (artgc-dag-artifact-creator), boundaryID: artgc-dag-wf-stopped-kg8tk" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:52.057Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:19:52.057Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:19:52.057Z" level=debug msg="Getting the template by name: artgc-dag-artifact-creator" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:19:52.057Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.create-artifact of Pod is Pending" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:52.057Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:52.057Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:52.057Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:52.057Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.create-artifact with container template: artgc-dag-artifact-creator\n" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:52.057Z" level=debug msg="Skipped pod artgc-dag-wf-stopped-kg8tk.create-artifact (artgc-dag-wf-stopped-kg8tk-3846913764) creation: already exists" namespace=argo podPhase=Pending workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:52.057Z" level=warning msg="was unable to obtain the node for artgc-dag-wf-stopped-kg8tk-1504799495, taskName stop-workflow"
time="2023-10-11T22:19:52.057Z" level=debug msg="Evaluating node artgc-dag-wf-stopped-kg8tk.delay-stop-workflow: template: *v1alpha1.DAGTask (artgc-dag-delay-stop), boundaryID: artgc-dag-wf-stopped-kg8tk" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:52.057Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:19:52.057Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:19:52.057Z" level=debug msg="Getting the template by name: artgc-dag-delay-stop" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:19:52.057Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.delay-stop-workflow of Pod is Pending" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:52.057Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:52.057Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:52.057Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:52.057Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.delay-stop-workflow with container template: artgc-dag-delay-stop\n" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:52.057Z" level=debug msg="Skipped pod artgc-dag-wf-stopped-kg8tk.delay-stop-workflow (artgc-dag-wf-stopped-kg8tk-3444961409) creation: already exists" namespace=argo podPhase=Pending workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:52.057Z" level=warning msg="was unable to obtain the node for artgc-dag-wf-stopped-kg8tk-1504799495, taskName stop-workflow"
time="2023-10-11T22:19:52.057Z" level=warning msg="was unable to obtain the node for artgc-dag-wf-stopped-kg8tk-1504799495, taskName stop-workflow"
time="2023-10-11T22:19:52.057Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:52.057Z" level=info msg=reconcileAgentPod namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:52.540Z" level=debug msg="Get leases 200"
time="2023-10-11T22:19:52.545Z" level=debug msg="Update leases 200"
time="2023-10-11T22:19:54.341Z" level=debug msg="Syncing all CronWorkflows"
time="2023-10-11T22:19:56.114Z" level=info msg="Processing workflow" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:56.115Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:56.116Z" level=info msg="node changed" namespace=argo new.message=PodInitializing new.phase=Pending new.progress=0/1 nodeID=artgc-dag-wf-stopped-kg8tk-3444961409 old.message= old.phase=Pending old.progress=0/1 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:56.116Z" level=info msg="node changed" namespace=argo new.message=PodInitializing new.phase=Pending new.progress=0/1 nodeID=artgc-dag-wf-stopped-kg8tk-3846913764 old.message= old.phase=Pending old.progress=0/1 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:56.116Z" level=debug msg="Evaluating node artgc-dag-wf-stopped-kg8tk: template: *v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main), boundaryID: " namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:56.116Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:56.116Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:56.116Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:56.116Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk of DAG is Running" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:56.116Z" level=debug msg="Evaluating node artgc-dag-wf-stopped-kg8tk.create-artifact: template: *v1alpha1.DAGTask (artgc-dag-artifact-creator), boundaryID: artgc-dag-wf-stopped-kg8tk" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:56.116Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:19:56.116Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:19:56.116Z" level=debug msg="Getting the template by name: artgc-dag-artifact-creator" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:19:56.116Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.create-artifact of Pod is Pending" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:56.116Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:56.116Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:56.116Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:56.116Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.create-artifact with container template: artgc-dag-artifact-creator\n" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:56.116Z" level=debug msg="Skipped pod artgc-dag-wf-stopped-kg8tk.create-artifact (artgc-dag-wf-stopped-kg8tk-3846913764) creation: already exists" namespace=argo podPhase=Pending workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:56.116Z" level=warning msg="was unable to obtain the node for artgc-dag-wf-stopped-kg8tk-1504799495, taskName stop-workflow"
time="2023-10-11T22:19:56.117Z" level=debug msg="Evaluating node artgc-dag-wf-stopped-kg8tk.delay-stop-workflow: template: *v1alpha1.DAGTask (artgc-dag-delay-stop), boundaryID: artgc-dag-wf-stopped-kg8tk" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:56.117Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:19:56.117Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:19:56.117Z" level=debug msg="Getting the template by name: artgc-dag-delay-stop" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:19:56.117Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.delay-stop-workflow of Pod is Pending" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:56.117Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:56.117Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:56.117Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:56.117Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.delay-stop-workflow with container template: artgc-dag-delay-stop\n" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:56.117Z" level=debug msg="Skipped pod artgc-dag-wf-stopped-kg8tk.delay-stop-workflow (artgc-dag-wf-stopped-kg8tk-3444961409) creation: already exists" namespace=argo podPhase=Pending workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:56.117Z" level=warning msg="was unable to obtain the node for artgc-dag-wf-stopped-kg8tk-1504799495, taskName stop-workflow"
time="2023-10-11T22:19:56.117Z" level=warning msg="was unable to obtain the node for artgc-dag-wf-stopped-kg8tk-1504799495, taskName stop-workflow"
time="2023-10-11T22:19:56.117Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:56.117Z" level=info msg=reconcileAgentPod namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:56.118Z" level=debug msg="Log changes patch: {\"status\":{\"conditions\":[{\"status\":\"False\",\"type\":\"PodRunning\"}],\"nodes\":{\"artgc-dag-wf-stopped-kg8tk-3444961409\":{\"hostNodeName\":\"k3d-k3s-default-server-0\",\"message\":\"PodInitializing\"},\"artgc-dag-wf-stopped-kg8tk-3846913764\":{\"hostNodeName\":\"k3d-k3s-default-server-0\",\"message\":\"PodInitializing\"}}}}"
time="2023-10-11T22:19:56.118Z" level=info msg="Workflow to be dehydrated" Workflow Size=3450
time="2023-10-11T22:19:56.143Z" level=debug msg="Update workflows 200"
time="2023-10-11T22:19:56.143Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=405136 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:57.549Z" level=debug msg="Get leases 200"
time="2023-10-11T22:19:57.554Z" level=debug msg="Update leases 200"
time="2023-10-11T22:19:57.936Z" level=info msg="Processing workflow" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:57.937Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:57.937Z" level=info msg="node unchanged" namespace=argo nodeID=artgc-dag-wf-stopped-kg8tk-3444961409 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:57.937Z" level=info msg="node unchanged" namespace=argo nodeID=artgc-dag-wf-stopped-kg8tk-3846913764 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:57.937Z" level=debug msg="Evaluating node artgc-dag-wf-stopped-kg8tk: template: *v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main), boundaryID: " namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:57.937Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:57.937Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:57.937Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:57.939Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk of DAG is Running" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:57.939Z" level=debug msg="Evaluating node artgc-dag-wf-stopped-kg8tk.create-artifact: template: *v1alpha1.DAGTask (artgc-dag-artifact-creator), boundaryID: artgc-dag-wf-stopped-kg8tk" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:57.939Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:19:57.939Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:19:57.939Z" level=debug msg="Getting the template by name: artgc-dag-artifact-creator" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:19:57.940Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.create-artifact of Pod is Pending" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:57.940Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:57.940Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:57.940Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:57.940Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.create-artifact with container template: artgc-dag-artifact-creator\n" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:57.940Z" level=debug msg="Skipped pod artgc-dag-wf-stopped-kg8tk.create-artifact (artgc-dag-wf-stopped-kg8tk-3846913764) creation: already exists" namespace=argo podPhase=Pending workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:57.940Z" level=warning msg="was unable to obtain the node for artgc-dag-wf-stopped-kg8tk-1504799495, taskName stop-workflow"
time="2023-10-11T22:19:57.940Z" level=debug msg="Evaluating node artgc-dag-wf-stopped-kg8tk.delay-stop-workflow: template: *v1alpha1.DAGTask (artgc-dag-delay-stop), boundaryID: artgc-dag-wf-stopped-kg8tk" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:57.940Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:19:57.940Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:19:57.940Z" level=debug msg="Getting the template by name: artgc-dag-delay-stop" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:19:57.940Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.delay-stop-workflow of Pod is Pending" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:57.940Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:57.940Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:57.940Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:19:57.940Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.delay-stop-workflow with container template: artgc-dag-delay-stop\n" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:57.940Z" level=debug msg="Skipped pod artgc-dag-wf-stopped-kg8tk.delay-stop-workflow (artgc-dag-wf-stopped-kg8tk-3444961409) creation: already exists" namespace=argo podPhase=Pending workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:57.940Z" level=warning msg="was unable to obtain the node for artgc-dag-wf-stopped-kg8tk-1504799495, taskName stop-workflow"
time="2023-10-11T22:19:57.940Z" level=warning msg="was unable to obtain the node for artgc-dag-wf-stopped-kg8tk-1504799495, taskName stop-workflow"
time="2023-10-11T22:19:57.940Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:19:57.940Z" level=info msg=reconcileAgentPod namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:02.559Z" level=debug msg="Get leases 200"
time="2023-10-11T22:20:02.566Z" level=debug msg="Update leases 200"
time="2023-10-11T22:20:02.902Z" level=info msg="Processing workflow" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:02.902Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:02.902Z" level=info msg="node unchanged" namespace=argo nodeID=artgc-dag-wf-stopped-kg8tk-3444961409 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:02.903Z" level=info msg="node unchanged" namespace=argo nodeID=artgc-dag-wf-stopped-kg8tk-3846913764 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:02.903Z" level=debug msg="Evaluating node artgc-dag-wf-stopped-kg8tk: template: *v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main), boundaryID: " namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:02.903Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:02.903Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:02.903Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:02.903Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk of DAG is Running" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:02.903Z" level=debug msg="Evaluating node artgc-dag-wf-stopped-kg8tk.create-artifact: template: *v1alpha1.DAGTask (artgc-dag-artifact-creator), boundaryID: artgc-dag-wf-stopped-kg8tk" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:02.903Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:20:02.903Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:20:02.903Z" level=debug msg="Getting the template by name: artgc-dag-artifact-creator" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:20:02.903Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.create-artifact of Pod is Pending" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:02.903Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:02.903Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:02.903Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:02.903Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.create-artifact with container template: artgc-dag-artifact-creator\n" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:02.903Z" level=debug msg="Skipped pod artgc-dag-wf-stopped-kg8tk.create-artifact (artgc-dag-wf-stopped-kg8tk-3846913764) creation: already exists" namespace=argo podPhase=Pending workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:02.903Z" level=warning msg="was unable to obtain the node for artgc-dag-wf-stopped-kg8tk-1504799495, taskName stop-workflow"
time="2023-10-11T22:20:02.903Z" level=debug msg="Evaluating node artgc-dag-wf-stopped-kg8tk.delay-stop-workflow: template: *v1alpha1.DAGTask (artgc-dag-delay-stop), boundaryID: artgc-dag-wf-stopped-kg8tk" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:02.903Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:20:02.903Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:20:02.903Z" level=debug msg="Getting the template by name: artgc-dag-delay-stop" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:20:02.903Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.delay-stop-workflow of Pod is Pending" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:02.903Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:02.903Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:02.903Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:02.904Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.delay-stop-workflow with container template: artgc-dag-delay-stop\n" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:02.904Z" level=debug msg="Skipped pod artgc-dag-wf-stopped-kg8tk.delay-stop-workflow (artgc-dag-wf-stopped-kg8tk-3444961409) creation: already exists" namespace=argo podPhase=Pending workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:02.904Z" level=warning msg="was unable to obtain the node for artgc-dag-wf-stopped-kg8tk-1504799495, taskName stop-workflow"
time="2023-10-11T22:20:02.904Z" level=warning msg="was unable to obtain the node for artgc-dag-wf-stopped-kg8tk-1504799495, taskName stop-workflow"
time="2023-10-11T22:20:02.904Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:02.904Z" level=info msg=reconcileAgentPod namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:04.342Z" level=debug msg="Syncing all CronWorkflows"
time="2023-10-11T22:20:04.907Z" level=info msg="Processing workflow" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:04.907Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:04.908Z" level=info msg="node changed" namespace=argo new.message= new.phase=Running new.progress=0/1 nodeID=artgc-dag-wf-stopped-kg8tk-3444961409 old.message=PodInitializing old.phase=Pending old.progress=0/1 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:04.908Z" level=info msg="node changed" namespace=argo new.message= new.phase=Running new.progress=0/1 nodeID=artgc-dag-wf-stopped-kg8tk-3846913764 old.message=PodInitializing old.phase=Pending old.progress=0/1 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:04.908Z" level=debug msg="Evaluating node artgc-dag-wf-stopped-kg8tk: template: *v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main), boundaryID: " namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:04.908Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:04.908Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:04.908Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:04.908Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk of DAG is Running" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:04.908Z" level=debug msg="Evaluating node artgc-dag-wf-stopped-kg8tk.create-artifact: template: *v1alpha1.DAGTask (artgc-dag-artifact-creator), boundaryID: artgc-dag-wf-stopped-kg8tk" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:04.908Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:20:04.908Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:20:04.908Z" level=debug msg="Getting the template by name: artgc-dag-artifact-creator" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:20:04.908Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.create-artifact of Pod is Running" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:04.908Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:04.908Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:04.908Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:04.908Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.create-artifact with container template: artgc-dag-artifact-creator\n" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:04.908Z" level=debug msg="Skipped pod artgc-dag-wf-stopped-kg8tk.create-artifact (artgc-dag-wf-stopped-kg8tk-3846913764) creation: already exists" namespace=argo podPhase=Running workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:04.908Z" level=warning msg="was unable to obtain the node for artgc-dag-wf-stopped-kg8tk-1504799495, taskName stop-workflow"
time="2023-10-11T22:20:04.908Z" level=debug msg="Evaluating node artgc-dag-wf-stopped-kg8tk.delay-stop-workflow: template: *v1alpha1.DAGTask (artgc-dag-delay-stop), boundaryID: artgc-dag-wf-stopped-kg8tk" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:04.908Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:20:04.908Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:20:04.908Z" level=debug msg="Getting the template by name: artgc-dag-delay-stop" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:20:04.909Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.delay-stop-workflow of Pod is Running" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:04.909Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:04.909Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:04.909Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:04.909Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.delay-stop-workflow with container template: artgc-dag-delay-stop\n" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:04.909Z" level=debug msg="Skipped pod artgc-dag-wf-stopped-kg8tk.delay-stop-workflow (artgc-dag-wf-stopped-kg8tk-3444961409) creation: already exists" namespace=argo podPhase=Running workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:04.909Z" level=warning msg="was unable to obtain the node for artgc-dag-wf-stopped-kg8tk-1504799495, taskName stop-workflow"
time="2023-10-11T22:20:04.909Z" level=warning msg="was unable to obtain the node for artgc-dag-wf-stopped-kg8tk-1504799495, taskName stop-workflow"
time="2023-10-11T22:20:04.909Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:04.909Z" level=info msg=reconcileAgentPod namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:04.909Z" level=debug msg="Log changes patch: {\"status\":{\"conditions\":[{\"status\":\"True\",\"type\":\"PodRunning\"}],\"nodes\":{\"artgc-dag-wf-stopped-kg8tk-3444961409\":{\"message\":null,\"phase\":\"Running\"},\"artgc-dag-wf-stopped-kg8tk-3846913764\":{\"message\":null,\"phase\":\"Running\"}}}}"
time="2023-10-11T22:20:04.909Z" level=info msg="Workflow to be dehydrated" Workflow Size=3419
time="2023-10-11T22:20:04.917Z" level=debug msg="Update workflows 200"
time="2023-10-11T22:20:04.918Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=405164 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:04.918Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"argo\", Name:\"artgc-dag-wf-stopped-kg8tk\", UID:\"a79f56c0-9c76-444e-b499-749078a78c6a\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"405164\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeRunning' Running node artgc-dag-wf-stopped-kg8tk.delay-stop-workflow"
time="2023-10-11T22:20:04.919Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"argo\", Name:\"artgc-dag-wf-stopped-kg8tk\", UID:\"a79f56c0-9c76-444e-b499-749078a78c6a\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"405164\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeRunning' Running node artgc-dag-wf-stopped-kg8tk.create-artifact"
time="2023-10-11T22:20:04.924Z" level=debug msg="Create events 201"
time="2023-10-11T22:20:04.930Z" level=debug msg="Create events 201"
time="2023-10-11T22:20:06.919Z" level=info msg="Processing workflow" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:06.920Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:06.920Z" level=debug msg="result: &WorkflowTaskResult{ObjectMeta:{artgc-dag-wf-stopped-kg8tk-3444961409  argo  574562ae-7d3e-444e-962b-ae0a52e6f226 405168 1 2023-10-11 22:20:06 +0000 UTC <nil> <nil> map[workflows.argoproj.io/workflow:artgc-dag-wf-stopped-kg8tk] map[] [{v1 Pod artgc-dag-wf-stopped-kg8tk-artgc-dag-delay-stop-3444961409 f496561d-e39a-43b8-b930-15b8c2ee14c2 <nil> <nil>}] []  [{argoexec Update argoproj.io/v1alpha1 2023-10-11 22:20:06 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"f496561d-e39a-43b8-b930-15b8c2ee14c2\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:&Outputs{Parameters:[]Parameter{},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:&S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:artgc-dag-wf-stopped-kg8tk/artgc-dag-wf-stopped-kg8tk-artgc-dag-delay-stop-3444961409/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:06.920Z" level=debug msg="woc.wf.Status.Nodes.Get err: %!s(<nil>)" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:06.920Z" level=debug msg="old: &NodeStatus{ID:artgc-dag-wf-stopped-kg8tk-3444961409,Name:artgc-dag-wf-stopped-kg8tk.delay-stop-workflow,DisplayName:delay-stop-workflow,Type:Pod,TemplateName:artgc-dag-delay-stop,TemplateRef:nil,Phase:Running,BoundaryID:artgc-dag-wf-stopped-kg8tk,Message:,StartedAt:2023-10-11 22:19:51 +0000 UTC,FinishedAt:0001-01-01 00:00:00 +0000 UTC,PodIP:,Daemoned:nil,Inputs:nil,Outputs:nil,Children:[],OutboundNodes:[],TemplateScope:local/artgc-dag-wf-stopped-kg8tk,ResourcesDuration:ResourcesDuration{},HostNodeName:k3d-k3s-default-server-0,MemoizationStatus:nil,EstimatedDuration:0,SynchronizationStatus:nil,Progress:0/1,}" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:06.920Z" level=debug msg="newNode: &NodeStatus{ID:artgc-dag-wf-stopped-kg8tk-3444961409,Name:artgc-dag-wf-stopped-kg8tk.delay-stop-workflow,DisplayName:delay-stop-workflow,Type:Pod,TemplateName:artgc-dag-delay-stop,TemplateRef:nil,Phase:Running,BoundaryID:artgc-dag-wf-stopped-kg8tk,Message:,StartedAt:2023-10-11 22:19:51 +0000 UTC,FinishedAt:0001-01-01 00:00:00 +0000 UTC,PodIP:,Daemoned:nil,Inputs:nil,Outputs:&Outputs{Parameters:[]Parameter{},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:&S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:artgc-dag-wf-stopped-kg8tk/artgc-dag-wf-stopped-kg8tk-artgc-dag-delay-stop-3444961409/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Children:[],OutboundNodes:[],TemplateScope:local/artgc-dag-wf-stopped-kg8tk,ResourcesDuration:ResourcesDuration{},HostNodeName:k3d-k3s-default-server-0,MemoizationStatus:nil,EstimatedDuration:0,SynchronizationStatus:nil,Progress:0/1,}" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:06.920Z" level=info msg="task-result changed" namespace=argo nodeID=artgc-dag-wf-stopped-kg8tk-3444961409 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:06.920Z" level=info msg="node changed" namespace=argo new.message= new.phase=Running new.progress=0/1 nodeID=artgc-dag-wf-stopped-kg8tk-3444961409 old.message= old.phase=Running old.progress=0/1 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:06.921Z" level=info msg="node unchanged" namespace=argo nodeID=artgc-dag-wf-stopped-kg8tk-3846913764 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:06.921Z" level=debug msg="Evaluating node artgc-dag-wf-stopped-kg8tk: template: *v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main), boundaryID: " namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:06.921Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:06.921Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:06.921Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:06.921Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk of DAG is Running" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:06.921Z" level=debug msg="Evaluating node artgc-dag-wf-stopped-kg8tk.create-artifact: template: *v1alpha1.DAGTask (artgc-dag-artifact-creator), boundaryID: artgc-dag-wf-stopped-kg8tk" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:06.921Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:20:06.921Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:20:06.921Z" level=debug msg="Getting the template by name: artgc-dag-artifact-creator" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:20:06.921Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.create-artifact of Pod is Running" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:06.921Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:06.921Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:06.921Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:06.921Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.create-artifact with container template: artgc-dag-artifact-creator\n" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:06.921Z" level=debug msg="Skipped pod artgc-dag-wf-stopped-kg8tk.create-artifact (artgc-dag-wf-stopped-kg8tk-3846913764) creation: already exists" namespace=argo podPhase=Running workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:06.921Z" level=warning msg="was unable to obtain the node for artgc-dag-wf-stopped-kg8tk-1504799495, taskName stop-workflow"
time="2023-10-11T22:20:06.921Z" level=debug msg="Evaluating node artgc-dag-wf-stopped-kg8tk.delay-stop-workflow: template: *v1alpha1.DAGTask (artgc-dag-delay-stop), boundaryID: artgc-dag-wf-stopped-kg8tk" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:06.921Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:20:06.921Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:20:06.921Z" level=debug msg="Getting the template by name: artgc-dag-delay-stop" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:20:06.922Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.delay-stop-workflow of Pod is Running" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:06.922Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:06.922Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:06.922Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:06.922Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.delay-stop-workflow with container template: artgc-dag-delay-stop\n" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:06.922Z" level=debug msg="Skipped pod artgc-dag-wf-stopped-kg8tk.delay-stop-workflow (artgc-dag-wf-stopped-kg8tk-3444961409) creation: already exists" namespace=argo podPhase=Running workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:06.922Z" level=warning msg="was unable to obtain the node for artgc-dag-wf-stopped-kg8tk-1504799495, taskName stop-workflow"
time="2023-10-11T22:20:06.922Z" level=warning msg="was unable to obtain the node for artgc-dag-wf-stopped-kg8tk-1504799495, taskName stop-workflow"
time="2023-10-11T22:20:06.922Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:06.922Z" level=info msg=reconcileAgentPod namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:06.922Z" level=debug msg="Log changes patch: {\"status\":{\"nodes\":{\"artgc-dag-wf-stopped-kg8tk-3444961409\":{\"outputs\":{\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artgc-dag-wf-stopped-kg8tk/artgc-dag-wf-stopped-kg8tk-artgc-dag-delay-stop-3444961409/main.log\"}}],\"exitCode\":\"0\"}}}}}"
time="2023-10-11T22:20:06.922Z" level=info msg="Workflow to be dehydrated" Workflow Size=3567
time="2023-10-11T22:20:06.926Z" level=info msg="cleaning up pod" action=terminateContainers key=argo/artgc-dag-wf-stopped-kg8tk-artgc-dag-delay-stop-3444961409/terminateContainers
time="2023-10-11T22:20:06.926Z" level=info msg="https://<IP_ADDRESS>:42429/api/v1/namespaces/argo/pods/artgc-dag-wf-stopped-kg8tk-artgc-dag-delay-stop-3444961409/exec?command=%2Fvar%2Frun%2Fargo%2Fargoexec&command=kill&command=15&command=1&container=wait&stderr=true&stdout=true&tty=false"
time="2023-10-11T22:20:06.926Z" level=debug msg="exec container command" command="[/var/run/argo/argoexec kill 15 1]" container=wait error="<nil>" namespace=argo pod=artgc-dag-wf-stopped-kg8tk-artgc-dag-delay-stop-3444961409
time="2023-10-11T22:20:06.934Z" level=debug msg="Update workflows 200"
time="2023-10-11T22:20:06.934Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=405170 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:06.951Z" level=debug msg="Create pods/exec 500"
time="2023-10-11T22:20:06.951Z" level=info msg="signaled container" container=wait error="unable to upgrade connection: container not found (\"wait\")" namespace=argo pod=artgc-dag-wf-stopped-kg8tk-artgc-dag-delay-stop-3444961409 stderr="<nil>" stdout="<nil>"
time="2023-10-11T22:20:07.572Z" level=debug msg="Get leases 200"
time="2023-10-11T22:20:07.578Z" level=debug msg="Update leases 200"
time="2023-10-11T22:20:07.932Z" level=info msg="Processing workflow" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:07.933Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:07.933Z" level=debug msg="result: &WorkflowTaskResult{ObjectMeta:{artgc-dag-wf-stopped-kg8tk-3444961409  argo  574562ae-7d3e-444e-962b-ae0a52e6f226 405168 1 2023-10-11 22:20:06 +0000 UTC <nil> <nil> map[workflows.argoproj.io/workflow:artgc-dag-wf-stopped-kg8tk] map[] [{v1 Pod artgc-dag-wf-stopped-kg8tk-artgc-dag-delay-stop-3444961409 f496561d-e39a-43b8-b930-15b8c2ee14c2 <nil> <nil>}] []  [{argoexec Update argoproj.io/v1alpha1 2023-10-11 22:20:06 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"f496561d-e39a-43b8-b930-15b8c2ee14c2\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:&Outputs{Parameters:[]Parameter{},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:&S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:artgc-dag-wf-stopped-kg8tk/artgc-dag-wf-stopped-kg8tk-artgc-dag-delay-stop-3444961409/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:07.933Z" level=debug msg="woc.wf.Status.Nodes.Get err: %!s(<nil>)" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:07.933Z" level=debug msg="old: &NodeStatus{ID:artgc-dag-wf-stopped-kg8tk-3444961409,Name:artgc-dag-wf-stopped-kg8tk.delay-stop-workflow,DisplayName:delay-stop-workflow,Type:Pod,TemplateName:artgc-dag-delay-stop,TemplateRef:nil,Phase:Running,BoundaryID:artgc-dag-wf-stopped-kg8tk,Message:,StartedAt:2023-10-11 22:19:51 +0000 UTC,FinishedAt:0001-01-01 00:00:00 +0000 UTC,PodIP:,Daemoned:nil,Inputs:nil,Outputs:&Outputs{Parameters:[]Parameter{},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:&S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:artgc-dag-wf-stopped-kg8tk/artgc-dag-wf-stopped-kg8tk-artgc-dag-delay-stop-3444961409/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:*0,},Children:[],OutboundNodes:[],TemplateScope:local/artgc-dag-wf-stopped-kg8tk,ResourcesDuration:ResourcesDuration{},HostNodeName:k3d-k3s-default-server-0,MemoizationStatus:nil,EstimatedDuration:0,SynchronizationStatus:nil,Progress:0/1,}" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:07.933Z" level=debug msg="newNode: &NodeStatus{ID:artgc-dag-wf-stopped-kg8tk-3444961409,Name:artgc-dag-wf-stopped-kg8tk.delay-stop-workflow,DisplayName:delay-stop-workflow,Type:Pod,TemplateName:artgc-dag-delay-stop,TemplateRef:nil,Phase:Running,BoundaryID:artgc-dag-wf-stopped-kg8tk,Message:,StartedAt:2023-10-11 22:19:51 +0000 UTC,FinishedAt:0001-01-01 00:00:00 +0000 UTC,PodIP:,Daemoned:nil,Inputs:nil,Outputs:&Outputs{Parameters:[]Parameter{},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:&S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:artgc-dag-wf-stopped-kg8tk/artgc-dag-wf-stopped-kg8tk-artgc-dag-delay-stop-3444961409/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:*0,},Children:[],OutboundNodes:[],TemplateScope:local/artgc-dag-wf-stopped-kg8tk,ResourcesDuration:ResourcesDuration{},HostNodeName:k3d-k3s-default-server-0,MemoizationStatus:nil,EstimatedDuration:0,SynchronizationStatus:nil,Progress:0/1,}" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:07.933Z" level=info msg="task-result changed" namespace=argo nodeID=artgc-dag-wf-stopped-kg8tk-3444961409 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:07.933Z" level=info msg="node unchanged" namespace=argo nodeID=artgc-dag-wf-stopped-kg8tk-3444961409 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:07.933Z" level=info msg="node unchanged" namespace=argo nodeID=artgc-dag-wf-stopped-kg8tk-3846913764 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:07.934Z" level=debug msg="Evaluating node artgc-dag-wf-stopped-kg8tk: template: *v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main), boundaryID: " namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:07.934Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:07.934Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:07.934Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:07.934Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk of DAG is Running" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:07.934Z" level=debug msg="Evaluating node artgc-dag-wf-stopped-kg8tk.create-artifact: template: *v1alpha1.DAGTask (artgc-dag-artifact-creator), boundaryID: artgc-dag-wf-stopped-kg8tk" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:07.934Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:20:07.934Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:20:07.934Z" level=debug msg="Getting the template by name: artgc-dag-artifact-creator" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:20:07.934Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.create-artifact of Pod is Running" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:07.934Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:07.934Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:07.934Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:07.934Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.create-artifact with container template: artgc-dag-artifact-creator\n" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:07.934Z" level=debug msg="Skipped pod artgc-dag-wf-stopped-kg8tk.create-artifact (artgc-dag-wf-stopped-kg8tk-3846913764) creation: already exists" namespace=argo podPhase=Running workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:07.934Z" level=warning msg="was unable to obtain the node for artgc-dag-wf-stopped-kg8tk-1504799495, taskName stop-workflow"
time="2023-10-11T22:20:07.934Z" level=debug msg="Evaluating node artgc-dag-wf-stopped-kg8tk.delay-stop-workflow: template: *v1alpha1.DAGTask (artgc-dag-delay-stop), boundaryID: artgc-dag-wf-stopped-kg8tk" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:07.935Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:20:07.935Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:20:07.935Z" level=debug msg="Getting the template by name: artgc-dag-delay-stop" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:20:07.935Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.delay-stop-workflow of Pod is Running" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:07.935Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:07.935Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:07.935Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:07.935Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.delay-stop-workflow with container template: artgc-dag-delay-stop\n" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:07.935Z" level=debug msg="Skipped pod artgc-dag-wf-stopped-kg8tk.delay-stop-workflow (artgc-dag-wf-stopped-kg8tk-3444961409) creation: already exists" namespace=argo podPhase=Running workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:07.935Z" level=warning msg="was unable to obtain the node for artgc-dag-wf-stopped-kg8tk-1504799495, taskName stop-workflow"
time="2023-10-11T22:20:07.935Z" level=warning msg="was unable to obtain the node for artgc-dag-wf-stopped-kg8tk-1504799495, taskName stop-workflow"
time="2023-10-11T22:20:07.935Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:07.935Z" level=info msg=reconcileAgentPod namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:07.935Z" level=debug msg="Log changes patch: {\"status\":{\"conditions\":[{\"status\":\"False\",\"type\":\"PodRunning\"}]}}"
time="2023-10-11T22:20:07.935Z" level=info msg="Workflow to be dehydrated" Workflow Size=3568
time="2023-10-11T22:20:07.939Z" level=info msg="cleaning up pod" action=terminateContainers key=argo/artgc-dag-wf-stopped-kg8tk-artgc-dag-delay-stop-3444961409/terminateContainers
time="2023-10-11T22:20:07.943Z" level=debug msg="Update workflows 200"
time="2023-10-11T22:20:07.944Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=405173 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:08.946Z" level=info msg="Processing workflow" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:08.947Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:08.947Z" level=debug msg="result: &WorkflowTaskResult{ObjectMeta:{artgc-dag-wf-stopped-kg8tk-3444961409  argo  574562ae-7d3e-444e-962b-ae0a52e6f226 405168 1 2023-10-11 22:20:06 +0000 UTC <nil> <nil> map[workflows.argoproj.io/workflow:artgc-dag-wf-stopped-kg8tk] map[] [{v1 Pod artgc-dag-wf-stopped-kg8tk-artgc-dag-delay-stop-3444961409 f496561d-e39a-43b8-b930-15b8c2ee14c2 <nil> <nil>}] []  [{argoexec Update argoproj.io/v1alpha1 2023-10-11 22:20:06 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"f496561d-e39a-43b8-b930-15b8c2ee14c2\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:&Outputs{Parameters:[]Parameter{},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:&S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:artgc-dag-wf-stopped-kg8tk/artgc-dag-wf-stopped-kg8tk-artgc-dag-delay-stop-3444961409/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:08.947Z" level=debug msg="woc.wf.Status.Nodes.Get err: %!s(<nil>)" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:08.947Z" level=debug msg="old: &NodeStatus{ID:artgc-dag-wf-stopped-kg8tk-3444961409,Name:artgc-dag-wf-stopped-kg8tk.delay-stop-workflow,DisplayName:delay-stop-workflow,Type:Pod,TemplateName:artgc-dag-delay-stop,TemplateRef:nil,Phase:Running,BoundaryID:artgc-dag-wf-stopped-kg8tk,Message:,StartedAt:2023-10-11 22:19:51 +0000 UTC,FinishedAt:0001-01-01 00:00:00 +0000 UTC,PodIP:,Daemoned:nil,Inputs:nil,Outputs:&Outputs{Parameters:[]Parameter{},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:&S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:artgc-dag-wf-stopped-kg8tk/artgc-dag-wf-stopped-kg8tk-artgc-dag-delay-stop-3444961409/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:*0,},Children:[],OutboundNodes:[],TemplateScope:local/artgc-dag-wf-stopped-kg8tk,ResourcesDuration:ResourcesDuration{},HostNodeName:k3d-k3s-default-server-0,MemoizationStatus:nil,EstimatedDuration:0,SynchronizationStatus:nil,Progress:0/1,}" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:08.947Z" level=debug msg="newNode: &NodeStatus{ID:artgc-dag-wf-stopped-kg8tk-3444961409,Name:artgc-dag-wf-stopped-kg8tk.delay-stop-workflow,DisplayName:delay-stop-workflow,Type:Pod,TemplateName:artgc-dag-delay-stop,TemplateRef:nil,Phase:Running,BoundaryID:artgc-dag-wf-stopped-kg8tk,Message:,StartedAt:2023-10-11 22:19:51 +0000 UTC,FinishedAt:0001-01-01 00:00:00 +0000 UTC,PodIP:,Daemoned:nil,Inputs:nil,Outputs:&Outputs{Parameters:[]Parameter{},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:&S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:artgc-dag-wf-stopped-kg8tk/artgc-dag-wf-stopped-kg8tk-artgc-dag-delay-stop-3444961409/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:*0,},Children:[],OutboundNodes:[],TemplateScope:local/artgc-dag-wf-stopped-kg8tk,ResourcesDuration:ResourcesDuration{},HostNodeName:k3d-k3s-default-server-0,MemoizationStatus:nil,EstimatedDuration:0,SynchronizationStatus:nil,Progress:0/1,}" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:08.947Z" level=info msg="task-result changed" namespace=argo nodeID=artgc-dag-wf-stopped-kg8tk-3444961409 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:08.947Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=artgc-dag-wf-stopped-kg8tk-3444961409 old.message= old.phase=Running old.progress=0/1 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:08.947Z" level=info msg="node unchanged" namespace=argo nodeID=artgc-dag-wf-stopped-kg8tk-3846913764 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:08.947Z" level=debug msg="Evaluating node artgc-dag-wf-stopped-kg8tk: template: *v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main), boundaryID: " namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:08.947Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:08.947Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:08.947Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.WorkflowStep (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:08.947Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk of DAG is Running" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:08.947Z" level=debug msg="Evaluating node artgc-dag-wf-stopped-kg8tk.create-artifact: template: *v1alpha1.DAGTask (artgc-dag-artifact-creator), boundaryID: artgc-dag-wf-stopped-kg8tk" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:08.947Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:20:08.947Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:20:08.947Z" level=debug msg="Getting the template by name: artgc-dag-artifact-creator" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-artifact-creator)"
time="2023-10-11T22:20:08.948Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.create-artifact of Pod is Running" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:08.948Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:08.948Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:08.948Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:08.948Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.create-artifact with container template: artgc-dag-artifact-creator\n" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:08.948Z" level=debug msg="Skipped pod artgc-dag-wf-stopped-kg8tk.create-artifact (artgc-dag-wf-stopped-kg8tk-3846913764) creation: already exists" namespace=argo podPhase=Running workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:08.948Z" level=warning msg="was unable to obtain the node for artgc-dag-wf-stopped-kg8tk-1504799495, taskName stop-workflow"
time="2023-10-11T22:20:08.948Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:20:08.948Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:20:08.948Z" level=debug msg="Getting the template by name: artgc-dag-delay-stop" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-delay-stop)"
time="2023-10-11T22:20:08.948Z" level=warning msg="was unable to obtain the node for artgc-dag-wf-stopped-kg8tk-1504799495, taskName stop-workflow"
time="2023-10-11T22:20:08.948Z" level=warning msg="was unable to obtain the node for artgc-dag-wf-stopped-kg8tk-1504799495, taskName stop-workflow"
time="2023-10-11T22:20:08.948Z" level=info msg="All of node artgc-dag-wf-stopped-kg8tk.stop-workflow dependencies [delay-stop-workflow] completed" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:08.948Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-delay-stop)"
time="2023-10-11T22:20:08.948Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-delay-stop)"
time="2023-10-11T22:20:08.948Z" level=debug msg="Getting the template by name: artgc-dag-delay-stop" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-delay-stop)"
time="2023-10-11T22:20:08.948Z" level=debug msg="Evaluating node artgc-dag-wf-stopped-kg8tk.stop-workflow: template: *v1alpha1.DAGTask (artgc-dag-workflow-stopper), boundaryID: artgc-dag-wf-stopped-kg8tk" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:08.948Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:08.948Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-workflow-stopper)"
time="2023-10-11T22:20:08.948Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-workflow-stopper)"
time="2023-10-11T22:20:08.948Z" level=debug msg="Getting the template by name: artgc-dag-workflow-stopper" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.DAGTask (artgc-dag-workflow-stopper)"
time="2023-10-11T22:20:08.948Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:08.948Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:08.948Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:08.948Z" level=debug msg="Initializing node artgc-dag-wf-stopped-kg8tk.stop-workflow: template: *v1alpha1.DAGTask (artgc-dag-workflow-stopper), boundaryID: artgc-dag-wf-stopped-kg8tk" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:08.948Z" level=info msg="Pod node artgc-dag-wf-stopped-kg8tk-1504799495 initialized Pending" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:08.948Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:08.948Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:08.948Z" level=debug msg="Getting the template by name: artgc-dag-wf-stopped-main" base="*v1alpha1.Workflow (namespace=argo,name=artgc-dag-wf-stopped-kg8tk)" tmpl="*v1alpha1.NodeStatus (artgc-dag-wf-stopped-main)"
time="2023-10-11T22:20:08.948Z" level=debug msg="Executing node artgc-dag-wf-stopped-kg8tk.stop-workflow with container template: artgc-dag-workflow-stopper\n" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:08.948Z" level=debug namespace=argo needLocation=true workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:08.952Z" level=debug msg="Get serviceaccounts 200"
time="2023-10-11T22:20:08.954Z" level=debug msg="Cache hit" cmd="&{[argo] []}" image="argoproj/argocli:latest"
time="2023-10-11T22:20:08.954Z" level=debug msg="Creating Pod: artgc-dag-wf-stopped-kg8tk.stop-workflow (artgc-dag-wf-stopped-kg8tk-artgc-dag-workflow-stopper-1504799495)" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:08.961Z" level=debug msg="Create pods 201"
W1011 22:20:08.961687   22635 warnings.go:70] metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must be no more than 63 characters]
time="2023-10-11T22:20:08.962Z" level=info msg="Created pod: artgc-dag-wf-stopped-kg8tk.stop-workflow (artgc-dag-wf-stopped-kg8tk-artgc-dag-workflow-stopper-1504799495)" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:08.962Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:08.962Z" level=info msg=reconcileAgentPod namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:08.963Z" level=debug msg="Log changes patch: {\"status\":{\"nodes\":{\"artgc-dag-wf-stopped-kg8tk-1504799495\":{\"boundaryID\":\"artgc-dag-wf-stopped-kg8tk\",\"displayName\":\"stop-workflow\",\"finishedAt\":null,\"id\":\"artgc-dag-wf-stopped-kg8tk-1504799495\",\"name\":\"artgc-dag-wf-stopped-kg8tk.stop-workflow\",\"phase\":\"Pending\",\"startedAt\":\"2023-10-11T22:20:08Z\",\"templateName\":\"artgc-dag-workflow-stopper\",\"templateScope\":\"local/artgc-dag-wf-stopped-kg8tk\",\"type\":\"Pod\"},\"artgc-dag-wf-stopped-kg8tk-3444961409\":{\"children\":[\"artgc-dag-wf-stopped-kg8tk-1504799495\"],\"finishedAt\":\"2023-10-11T22:20:06Z\",\"phase\":\"Succeeded\",\"resourcesDuration\":{\"cpu\":6,\"memory\":4}}}}}"
time="2023-10-11T22:20:08.963Z" level=info msg="Workflow to be dehydrated" Workflow Size=3936
time="2023-10-11T22:20:08.972Z" level=debug msg="Update workflows 200"
time="2023-10-11T22:20:08.973Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=405178 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:08.973Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"argo\", Name:\"artgc-dag-wf-stopped-kg8tk\", UID:\"a79f56c0-9c76-444e-b499-749078a78c6a\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"405178\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeSucceeded' Succeeded node artgc-dag-wf-stopped-kg8tk.delay-stop-workflow"
time="2023-10-11T22:20:08.978Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/artgc-dag-wf-stopped-kg8tk-artgc-dag-delay-stop-3444961409/labelPodCompleted
time="2023-10-11T22:20:08.994Z" level=debug msg="Patch pods 200"
time="2023-10-11T22:20:09.013Z" level=debug msg="Create events 201"
time="2023-10-11T22:20:09.952Z" level=info msg="cleaning up pod" action=killContainers key=argo/artgc-dag-wf-stopped-kg8tk-artgc-dag-delay-stop-3444961409/killContainers
time="2023-10-11T22:20:09.964Z" level=info msg="Processing workflow" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:09.966Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:09.966Z" level=debug msg="result: &WorkflowTaskResult{ObjectMeta:{artgc-dag-wf-stopped-kg8tk-3444961409  argo  574562ae-7d3e-444e-962b-ae0a52e6f226 405168 1 2023-10-11 22:20:06 +0000 UTC <nil> <nil> map[workflows.argoproj.io/workflow:artgc-dag-wf-stopped-kg8tk] map[] [{v1 Pod artgc-dag-wf-stopped-kg8tk-artgc-dag-delay-stop-3444961409 f496561d-e39a-43b8-b930-15b8c2ee14c2 <nil> <nil>}] []  [{argoexec Update argoproj.io/v1alpha1 2023-10-11 22:20:06 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"f496561d-e39a-43b8-b930-15b8c2ee14c2\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:&Outputs{Parameters:[]Parameter{},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:&S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:artgc-dag-wf-stopped-kg8tk/artgc-dag-wf-stopped-kg8tk-artgc-dag-delay-stop-3444961409/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk
time="2023-10-11T22:20:09.966Z" level=debug msg="woc.wf.Status.Nodes.Get err: %!s(<nil>)" namespace=argo workflow=artgc-dag-wf-stopped-kg8tk