3.4 log propagation with emissary and Vault sidecar

Pre-requisites

[X] I have double-checked my configuration
[X] I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
[X] I have searched existing issues and could not find a match for this bug
[X] I'd like to contribute the fix myself (see contributing guide)

What happened/what did you expect to happen?
We are using argo workflow with v3.4.0(0.18.0 - Chart version). Our kubernetes version is 1.25.1.
For our applications, We have Cronworkflow, Workflowtemplate object.
In Workflowtemplate, We are using annotations to send logs to fluentbit and also Vault is running as side car container.
When we upgrade from 3.3.2 to 3.4.0, We observed the strange issue that logs are not getting visible on elasticsearch/kibana. Previously it was working.
There must be some change w.r.t to emissary which is causing this trouble.
Also when I stop vault as side car container then logs are coming to kibana but it is not possible for us to stop using vault.
I shared you my cronworkflow, workflowtemplate for reference.
Please check on priority because this is bug which is there on all versions after > 3.4
Version
v3.4.0
Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.
# Workflowtemplate -

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  annotations:
    meta.helm.sh/release-name: test-migration-job  # Updated release-name
    meta.helm.sh/release-namespace: dev
  labels:
    app.kubernetes.io/managed-by: Helm
  name: test-migration-v1-template  # Updated name
  namespace: dev
spec:
  activeDeadlineSeconds: 28800
  arguments:
    parameters:
    - name: script
  templates:
  - container:
      command:
      - /bin/bash
      - '{{`{{inputs.parameters.script}}`}}'
      image: 123456789.dkr.ecr.ap-south-1.amazonaws.com/test-migration:1.0.97886  # Updated image
      name: test-migration-v1  # Updated container name
      resources: {}
      volumeMounts:
      - mountPath: /data
        name: shared-data
    initContainers:
    - env:
      - name: APP_NAME
        value: test-migration  # Updated environment variable
      - name: PROFILE
        value: nonprod-dev
      - name: VERSION
        value: master
      - name: MY_SERVER_URL
        value: http://my-internal-service.dev.svc.cluster.local/  # Updated service URL
      image: 123456789.dkr.ecr.ap-south-1.amazonaws.com/my-internal-client:latest  # Updated image
      name: my-internal-client  # Updated initContainer name
      resources: {}
      volumeMounts:
      - mountPath: /data
        name: shared-data
    inputs:
      parameters:
      - name: script
    metadata:
      annotations:
        co.elastic.logs/enabled: "true"
        co.elastic.logs/json.add_error_key: "true"
        co.elastic.logs/json.keys_under_root: "true"
        co.elastic.logs/json.message_key: log
        fluentbit.job.name: job-test-migration-v1  # Updated fluentbit job name
        vault.hashicorp.com/agent-image: 123456789.dkr.ecr.ap-south-1.amazonaws.com/vault:1.6.2  # Updated image
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-secret-secrets.txt: kv/apps/dev/test-migration  # Updated secret path
        vault.hashicorp.com/agent-inject-template-secrets.txt: |
          {{ with secret "kv/apps/dev/test-migration" }}{{ range $k, $v := .Data.data }}
          {{ $k }}={{ $v }}{{ end }}{{ end }}
        vault.hashicorp.com/role: dev
    name: test-migration-v1-template  # Updated template name
    outputs: {}
    securityContext:
      fsGroup: 65534
    serviceAccountName: test-migration-v1  # Updated service account name
    volumes:
    - emptyDir: {}
      name: shared-data
  ttlStrategy:
    secondsAfterCompletion: 86400

---
# Cronworkflow -

apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
  annotations:
    meta.helm.sh/release-name: test-migration-job  # Updated release-name
    meta.helm.sh/release-namespace: dev
  labels:
    app.kubernetes.io/instance: test-migration-job  # Updated instance label
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: argo-jobs
    app.kubernetes.io/version: 1.16.0
    helm.sh/chart: argo-jobs-0.1.0
  name: argo-job-test-migration-v1  # Updated job name
  namespace: dev
spec:
  activeDeadlineSeconds: 28800
  concurrencyPolicy: Forbid
  podGC:
    strategy: Never
  schedule: '*/15 0-2 * * *'
  selector:
    matchLabels:
      app.kubernetes.io/instance: test-migration-job  # Updated instance label
      app.kubernetes.io/name: argo-jobs
  ttlStrategy:
    secondsAfterCompletion: 86400
  workflowSpec:
    activeDeadlineSeconds: 43200000
    entrypoint: dag
    synchronization:
      mutex:
        name: test-migration-v1  # Updated mutex name
    templates:
    - dag:
        tasks:
        - arguments:
            parameters:
            - name: script
              value: /run-test.sh  # Updated script value
          name: test-details-store
          templateRef:
            name: test-migration-v1-template  # Updated templateRef name
            template: test-migration-v1-template  # Updated template name
      name: dag
---
# Workflow-controller-configmap -

apiVersion: v1
data:
  config: |
    artifactRepository:
      s3: {}
    persistence: {}
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: argo
    meta.helm.sh/release-namespace: admin
  creationTimestamp: "2024-07-06T10:04:01Z"
  labels:
    app.kubernetes.io/component: workflow-controller
    app.kubernetes.io/instance: argo
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: argo-workflows-cm
    app.kubernetes.io/part-of: argo-workflows
    helm.sh/chart: argo-workflows-0.18.0
  name: argo-argo-workflows-workflow-controller-configmap

Logs from the workflow controller
kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

No issues in logs

Logs from in your workflow's wait container
kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

No issues observed in logs

I am also facing same issue.

@agilgur5 Thanks for responding.
We also tested with 3.5.8 version, it is having the same behaviour as we are not getting logs.We tired all versions starting from > 2.12.9 to check where it stopped sending logs. We observed argo is working fine and also we are able to get logs until v3.4.0.
It means something has been added from 3.4.0 and later version onwards which is causing issues.
And Emissary is the major change I observed which starts from v3.4.0.
We already tested on latest version (3.5.8) but due this strange behaviour we have role it back from our production env to 3.3.2
I agree most of users are using elastic. In my scenario, I am using elastic as well as vault as side car container. You can see the annotations.
Let me know if you want more info on the same.

We also tested with 3.5.8 version, it is having the same behaviour as we are not getting logs.We tired all versions starting from > 2.12.9 to check where it stopped sending logs.

Ok, can you reference that in the "Version" section? Or elsewhere in your description.

It means something has been added from 3.4.0 and later version onwards which is causing issues.
And Emissary is the major change I observed which starts from v3.4.0.

Yes, your observation is correct, and Emissary runs an initContainer, a sidecar, and as a parent process to your container as well. So it certainly can affect something like this in theory.
Given that this hasn't been reported previously though (despite 3.4 being out for some time), I imagine it is some specific combination of your services that causes this issue.
To be clear though, Emissary was available since Argo 3.1, was the default in 3.3, and then was the only executor in 3.4+. I assume you were using another executor in your config prior to the 3.4 upgrade?

Images are private but you can take any image.

Can you please provide a minimal reproduction with a public image, as the issue template asks?

Only prerequisite is to have EFK(Fluentbit/kibana) and vault setup. So that you can use that annotations like I did.

I don't think Elastic or Kibana are relevant here, as those are not impacted by Argo and are post-processors after fluentbit. I would imagine that your logs are not getting parsed by fluentbit or aren't even making it to stdout. Can you please debug and confirm which one it is? Again that is important for a minimal reproduction.
You also did not provide the logs of the main or wait containers, although you said there were "no issues" in the logs. That would suggest that Argo passed the logs to stdout fine and the issue is actually somewhere in your other services. For instance, if your config is not handling Argo's initContainer or wait container.

On 3.3.2 , We are using container runtime executor is k8s api.
I will try to give workflow and logs , allow me some time, I will share you evidence of I tried 3.5.8
Still following Questions in minds coming.

In version 3.3.2, I have elastic and vault agent as side car. How everything working as expected like logs are coming?

If I moved to v3.5.8 with the same setup of vault and elastic. Vault is doing there job not an issue but why logs stop?

In the same 3.5.8, If we say issue with fluentbit, then when I intentionally stop my vault injection (not allowing vault side car to run) then how logs are coming?

I have number of jobs using workflow but why only the jobs which uses workflowtemplate gets affected? mean to say jobs which uses cronworkflow only that is working fine , able to get logs and vault is also doing its jobs.

Something is happening between emissary and sidecars. Mistery!

3. then when I intentionally stop my vault injection (not allowing vault side car to run) then how logs are coming?

Yea I don't know, that's odd that it affects it. If I had to guess, maybe the wait container or something didn't get properly added to the Pod, since it is also a sidecar? i.e. that the problem is not with the Emissary process itself, but with the Pod being created. It might also be timing-related since Vault uses sidecar injection?

4. mean to say jobs which uses cronworkflow only that is working fine , able to get logs and vault is also doing its jobs.

That's bizarre... Are you sure they're otherwise exactly the same? Using a WorkflowTemplate has no affect on the Pod created itself (it only affects the DAG logic in the Controller, which occurs prior to Pod creation). Maybe your fluentbit config excludes certain labels that your WorkflowTemplate adds?

If same elastic, same vault agent and same workflow templates works in 3.3 version , it should not cause issue if we are upgrading to 3.5 but it is causing issue after upgrade only.
One more thing I observed is if we remove initcontainer, logs are visible in kibana.
I think if same setof my services working fine in 3.3 and not working in latest version means  in upgraded version something is causing issues to side car or init container.
Check the document here it is saying something about emissary. and attaching same snap for reference.
https://blog.argoproj.io/sunsetting-docker-executor-support-in-argo-workflows-77d352d601c

it should not cause issue if we are upgrading to 3.5 but it is causing issue after upgrade only.

I'm not sure why you're repeating yourself. Please answer all my previous questions and provide the output, logs, and reproductions asked for, which are necessary to debug. Without those essential debugging pieces, there is nothing actionable we can do.
Please note that I am an unpaid volunteer maintainer and that reading and responding to all your comments takes time. Please try to make that an effective and efficient use of time.

something is causing issues to side car or init container

Yes, I said that may be the case above as well. I also said twice before that it could very well be your fluentbit config not picking up certain Pods or containers, since you appear to have logs in stdout fine. Given that other fluentbit users have not faced this before, that would be my prime suspicion.

https://blog.argoproj.io/sunsetting-docker-executor-support-in-argo-workflows-77d352d601c

I know how Emissary works, and this repo contains its source code, so I'm not sure why you're linking a 3 year old notice saying to upgrade to it.

Here's a checklist of things needed and questions unanswered:

[ ] "Can you please provide a minimal reproduction with a public image, as the issue template asks?"
[ ] Were you "using another executor in your config prior to the 3.4 upgrade?"
[ ] "I would imagine that your logs are not getting parsed by fluentbit or aren't even making it to stdout. Can you please debug and confirm which one it is? Again that is important for a minimal reproduction."
[ ] "The output of kubectl get pod <my-pod> -o yaml might be helpful in checking that"
[ ] "Maybe your fluentbit config excludes certain labels that your WorkflowTemplate adds?"

I appreciate your time on this.
Earlier, We are using k8sapi executor. When when swith to 3.5.8 we removed it from configmap as emissary is default one.
 Learn how to enable transport layer security: https://argo-workflows.readthedocs.io/en/release-3.5/tls/"
time="2024-07-11T08:36:44.303Z" level=warning msg="You are running without client authentication. Learn how to enable client authentication: https://argo-workflows.readthedocs.io/en/release-3.5/argo-server-auth-mode/"
time="2024-07-11T08:36:44.303Z" level=info msg="SSO disabled"
time="2024-07-11T08:36:44.315Z" level=info msg="Starting Argo Server" instanceID= version=v3.5.8
time="2024-07-11T08:36:44.315Z" level=info msg="Creating event controller" asyncDispatch=false operationQueueSize=16 workerCount=4
time="2024-07-11T08:36:44.321Z" level=info msg="GRPC Server Max Message Size, MaxGRPCMessageSize, is set" GRPC_MESSAGE_SIZE=104857600
time="2024-07-11T08:36:44.321Z" level=info msg="Argo Server started successfully on http://localhost:2746" url="http://localhost:2746"
time="2024-07-11T08:37:04.061Z" level=info duration="129.005µs" method=GET path=index.html size=487 status=0
time="2024-07-11T08:37:10.759Z" level=debug msg="Piping events to channel"
time="2024-07-11T08:37:24.061Z" level=info duration="103.608µs" method=GET path=index.html size=487 status=0
time="2024-07-11T08:37:44.061Z" level=info duration="99.201µs" method=GET path=index.html size=487 status=0
time="2024-07-11T08:38:04.061Z" level=info duration="65.947µs" method=GET path=index.html size=487 status=0
time="2024-07-11T08:38:13.656Z" level=debug msg="Result channel done"
time="2024-07-11T08:38:13.656Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=WatchWorkflows grpc.service=workflow.WorkflowService grpc.start_time="2024-07-11T08:37:10Z" grpc.time_ms=62900.21 span.kind=server system=grpc
2024/07/11 08:38:13 http: superfluous response.WriteHeader call from github.com/argoproj/argo-workflows/v3/server/apiserver/accesslog.(*resultCapturingWriter).WriteHeader (result_capturing_writer.go:23)
time="2024-07-11T08:38:13.656Z" level=info duration=1m2.900850898s method=GET path=/api/v1/workflow-events/perf size=19 status=408
time="2024-07-11T08:38:13.660Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=ListCronWorkflows grpc.service=cronworkflow.CronWorkflowService grpc.start_time="2024-07-11T08:38:13Z" grpc.time_ms=6.969 span.kind=server system=grpc
time="2024-07-11T08:38:13.664Z" level=info duration=10.67679ms method=GET path=/api/v1/cron-workflows/perf size=1720 status=0
time="2024-07-11T08:38:16.365Z" level=debug msg=CanI name= namespace=perf resource=workflows verb=list
time="2024-07-11T08:38:16.367Z" level=debug msg=CanI name= namespace=perf resource=workflows status="{true false RBAC: allowed by ClusterRoleBinding \"argo-server-binding\" of ClusterRole \"argo-server-cluster-role\" to ServiceAccount \"argo-server/admin\" }" verb=list
time="2024-07-11T08:38:16.367Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=ListWorkflows grpc.service=workflow.WorkflowService grpc.start_time="2024-07-11T08:38:16Z" grpc.time_ms=2.559 span.kind=server system=grpc
time="2024-07-11T08:38:16.368Z" level=info duration=3.159236ms method=GET path=/api/v1/workflows/perf size=57 status=0
time="2024-07-11T08:38:16.369Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetCronWorkflow grpc.service=cronworkflow.CronWorkflowService grpc.start_time="2024-07-11T08:38:16Z" grpc.time_ms=5.046 span.kind=server system=grpc
time="2024-07-11T08:38:16.369Z" level=info duration=5.626759ms method=GET path=/api/v1/cron-workflows/perf/argo-job-test-migration-v1 size=1665 status=0
time="2024-07-11T08:38:19.097Z" level=debug msg="Getting the template by name: dag"
time="2024-07-11T08:38:19.097Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.WorkflowStep (dag)"
time="2024-07-11T08:38:19.097Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.WorkflowStep (dag)"
time="2024-07-11T08:38:19.097Z" level=debug msg="Getting the template by name: dag" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.WorkflowStep (dag)"
time="2024-07-11T08:38:19.098Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.DAGTask (test-migration-v1-template/test-migration-v1-template#false)"
time="2024-07-11T08:38:19.098Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.DAGTask (test-migration-v1-template/test-migration-v1-template#false)"
time="2024-07-11T08:38:19.098Z" level=debug msg="Getting the template from ref" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.DAGTask (test-migration-v1-template/test-migration-v1-template#false)"
time="2024-07-11T08:38:19.105Z" level=debug msg="Stored the template" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.DAGTask (test-migration-v1-template/test-migration-v1-template#false)"
time="2024-07-11T08:38:19.108Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-07-11T08:38:19.108Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-07-11T08:38:19.108Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-07-11T08:38:19.108Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-07-11T08:38:19.108Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-07-11T08:38:19.108Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-07-11T08:38:19.108Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-07-11T08:38:19.108Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-07-11T08:38:19.108Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-07-11T08:38:19.108Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-07-11T08:38:19.108Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-07-11T08:38:19.108Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-07-11T08:38:19.108Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-07-11T08:38:19.108Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.DAGTask (test-migration-v1-template/test-migration-v1-template#false)"
time="2024-07-11T08:38:19.109Z" level=debug msg="Found stored template" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.DAGTask (test-migration-v1-template/test-migration-v1-template#false)"
time="2024-07-11T08:38:19.113Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-07-11T08:38:19.113Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-07-11T08:38:19.113Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-07-11T08:38:19.113Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-07-11T08:38:19.113Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-07-11T08:38:19.113Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-07-11T08:38:19.113Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-07-11T08:38:19.113Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-07-11T08:38:19.113Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-07-11T08:38:19.113Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-07-11T08:38:19.113Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-07-11T08:38:19.113Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-07-11T08:38:19.113Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-07-11T08:38:19.113Z" level=debug msg="Getting the template by name: dag" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.WorkflowStep (dag)"
time="2024-07-11T08:38:19.113Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.WorkflowStep (dag)"
time="2024-07-11T08:38:19.113Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.WorkflowStep (dag)"
time="2024-07-11T08:38:19.113Z" level=debug msg="Getting the template by name: dag" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.WorkflowStep (dag)"
time="2024-07-11T08:38:19.124Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=SubmitWorkflow grpc.service=workflow.WorkflowService grpc.start_time="2024-07-11T08:38:19Z" grpc.time_ms=29.911 span.kind=server system=grpc
time="2024-07-11T08:38:19.124Z" level=info duration=30.737748ms method=POST path=/api/v1/workflows/perf/submit size=2969 status=0
time="2024-07-11T08:38:19.179Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetWorkflow grpc.service=workflow.WorkflowService grpc.start_time="2024-07-11T08:38:19Z" grpc.time_ms=7.935 span.kind=server system=grpc
time="2024-07-11T08:38:19.179Z" level=info duration=8.452007ms method=GET path=/api/v1/workflows/perf/argo-job-test-migration-v1-dg5rj size=2969 status=0
time="2024-07-11T08:38:19.226Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetWorkflow grpc.service=workflow.WorkflowService grpc.start_time="2024-07-11T08:38:19Z" grpc.time_ms=11.466 span.kind=server system=grpc
time="2024-07-11T08:38:19.226Z" level=info duration=11.974651ms method=GET path=/api/v1/workflows/perf/argo-job-test-migration-v1-dg5rj size=2969 status=0
time="2024-07-11T08:38:19.236Z" level=debug msg="Piping events to channel"
time="2024-07-11T08:38:19.237Z" level=debug msg="Received workflow event"
time="2024-07-11T08:38:19.237Z" level=debug msg="Sending workflow event" phase=Running type=ADDED workflow=argo-job-test-migration-v1-dg5rj
time="2024-07-11T08:38:19.272Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetWorkflow grpc.service=workflow.WorkflowService grpc.start_time="2024-07-11T08:38:19Z" grpc.time_ms=12.397 span.kind=server system=grpc
time="2024-07-11T08:38:19.273Z" level=info duration=13.065806ms method=GET path=/api/v1/workflows/perf/argo-job-test-migration-v1-dg5rj size=5072 status=0
time="2024-07-11T08:38:19.545Z" level=debug msg="Received workflow event"
time="2024-07-11T08:38:19.545Z" level=debug msg="Sending workflow event" phase=Running type=MODIFIED workflow=argo-job-test-migration-v1-dg5rj
time="2024-07-11T08:38:24.061Z" level=info duration="73.656µs" method=GET path=index.html size=487 status=0
time="2024-07-11T08:38:29.232Z" level=debug msg="Received workflow event"
time="2024-07-11T08:38:29.233Z" level=debug msg="Sending workflow event" phase=Running type=MODIFIED workflow=argo-job-test-migration-v1-dg5rj
time="2024-07-11T08:38:44.061Z" level=info duration="106.593µs" method=GET path=index.html size=487 status=0
time="2024-07-11T08:39:04.061Z" level=info duration="107.957µs" method=GET path=index.html size=487 status=0
time="2024-07-11T08:39:24.061Z" level=info duration="109.535µs" method=GET path=index.html size=487 status=0
time="2024-07-11T08:39:44.061Z" level=info duration="82.925µs" method=GET path=index.html size=487 status=0
time="2024-07-11T08:40:02.693Z" level=info duration=1m43.468569554s method=GET path=/api/v1/workflow-events/perf size=15971 status=0
time="2024-07-11T08:40:02.693Z" level=debug msg="Result channel done"
time="2024-07-11T08:40:02.693Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=WatchWorkflows grpc.service=workflow.WorkflowService grpc.start_time="2024-07-11T08:38:19Z" grpc.time_ms=103468.36 span.kind=server system=grpc
time="2024-07-11T08:40:02.695Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=ListWorkflowTemplates grpc.service=workflowtemplate.WorkflowTemplateService grpc.start_time="2024-07-11T08:40:02Z" grpc.time_ms=7.043 span.kind=server system=grpc
time="2024-07-11T08:40:02.696Z" level=info duration=7.903142ms method=GET path=/api/v1/workflow-templates/perf size=2326 status=0
time="2024-07-11T08:40:04.061Z" level=info duration="102.507µs" method=GET path=index.html size=487 status=0
time="2024-07-11T08:40:04.590Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetWorkflowTemplate grpc.service=workflowtemplate.WorkflowTemplateService grpc.start_time="2024-07-11T08:40:04Z" grpc.time_ms=4.445 span.kind=server system=grpc
time="2024-07-11T08:40:04.590Z" level=info duration=5.129671ms method=GET path=/api/v1/workflow-templates/perf/test-migration-v1-template size=2271 status=0
time="2024-07-11T08:40:24.061Z" level=info duration="103.768µs" method=GET path=index.html size=487 status=0
time="2024-07-11T08:40:44.061Z" level=info duration="86.675µs" method=GET path=index.html size=487 status=0
time="2024-07-11T08:41:04.061Z" level=info duration="95.86µs" method=GET path=index.html size=487 status=0
time="2024-07-11T08:41:24.060Z" level=info duration="81.517µs" method=GET path=index.html size=487 status=0
time="2024-07-11T08:41:44.061Z" level=info duration="89.283µs" method=GET path=index.html size=487 status=0
time="2024-07-11T08:41:44.302Z" level=info msg="Alloc=10816 TotalAlloc=18987 Sys=34149 NumGC=6 Goroutines=100"
time="2024-07-11T08:42:04.061Z" level=info duration="100.999µs" method=GET path=index.html size=487 status=0
time="2024-07-11T08:42:24.061Z" level=info duration="85.946µs" method=GET path=index.html size=487 status=0

argo job pod - manifest
Name:         argo-job-test-migration-v1-zgpnz-test-migration-v1-template-2007578379
Namespace:    perf
Priority:     0
Node:         ip-10-10-9-4-compute.internal
Start Time:   Thu, 11 Jul 2024 09:30:13 +0000
Labels:       workflows.argoproj.io/completed=false
              workflows.argoproj.io/workflow=argo-job-test-migration-v1-zgpnz
Annotations:  co.elastic.logs/enabled: true
              co.elastic.logs/json.add_error_key: true
              co.elastic.logs/json.keys_under_root: true
              co.elastic.logs/json.message_key: log
              fluentbit.job.name: job-test-migration-v1
              kubectl.kubernetes.io/default-container: main
              vault.hashicorp.com/agent-image: hashicorp/vault:1.6.2
              vault.hashicorp.com/agent-inject: true
              vault.hashicorp.com/agent-inject-secret-secrets.txt: kv/apps/perf/test-migration
              vault.hashicorp.com/agent-inject-status: injected
              vault.hashicorp.com/agent-inject-template-secrets.txt:
                {{ with secret "kv/apps/dev/test-migration" }}{{ range $k, $v := .Data.data }} {{ $k }}={{ $v }}{{ end }}{{ end }}
              vault.hashicorp.com/role: perf
              workflows.argoproj.io/node-id: argo-job-test-migration-v1-zgpnz-2007578379
              workflows.argoproj.io/node-name: argo-job-test-migration-v1-zgpnz.test-details-store

Controlled By:  Workflow/argo-job-test-migration-v1-zgpnz
Init Containers:
  init:
    Container ID:  containerd://35b1df68e7c4c62206afa067ab86ca24641f8f07d1c639652213fcf7d9c16ae0
    Image:         quay.io/argoproj/argoexec:v3.5.8
    Image ID:      quay.io/argoproj/argoexec@sha256:a9e8906183c3c738573b8e5d8a2a67eee428dd7a52eb8c7b77c9916ba8f4685b
    Port:          <none>
    Host Port:     <none>
    Command:
      argoexec
      init
      --loglevel
      debug
      --log-format
      text
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 11 Jul 2024 09:30:21 +0000
      Finished:     Thu, 11 Jul 2024 09:30:21 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      ARGO_POD_NAME:                      argo-job-test-migration-v1-zgpnz-test-migration-v1-template-2007578379 (v1:metadata.name)
      ARGO_POD_UID:                        (v1:metadata.uid)
      GODEBUG:                            x509ignoreCN=0
      ARGO_WORKFLOW_NAME:                 argo-job-test-migration-v1-zgpnz
      ARGO_WORKFLOW_UID:                  df1aaf50-a11b-45db-9c8e-20b3fd69ca9e
      ARGO_CONTAINER_NAME:                init
      ARGO_TEMPLATE:                      {"name":"test-migration-v1-template","inputs":{"parameters":[{"name":"script","value":"echo hi"}]},"outputs":{},"metadata":{"annotations":{"co.elastic.logs/enabled":"true","co.elastic.logs/json.add_error_key":"true","co.elastic.logs/json.keys_under_root":"true","co.elastic.logs/json.message_key":"log","fluentbit.job.name":"job-test-migration-v1","vault.hashicorp.com/agent-image":"hashicorp/vault:1.6.2","vault.hashicorp.com/agent-inject":"true","vault.hashicorp.com/agent-inject-secret-secrets.txt":"kv/apps/perf/test-migration","vault.hashicorp.com/agent-inject-template-secrets.txt":"{{ with secret \"kv/apps/dev/test-migration\" }}{{ range $k, $v := .Data.data }} {{ $k }}={{ $v }}{{ end }}{{ end }}\n","vault.hashicorp.com/role":"perf"}},"container":{"name":"test-migration-v1","image":"nginx","command":["echo hi"],"resources":{},"volumeMounts":[{"name":"shared-data","mountPath":"/data"}]},"volumes":[{"name":"shared-data","emptyDir":{}}],"initContainers":[{"name":"test-config-client","image":"busybox:latest","env":[{"name":"APP_NAME","value":"test-migration"},{"name":"VERSION","value":"master"}],"resources":{},"volumeMounts":[{"name":"shared-data","mountPath":"/data"}]}],:false,"accessKeySecret":{"name":"argo-artifact-secret","key":"awsAccessKey"},"secretKeySecret":{"name":"argo-artifact-secret","key":"awsSecretAccessKey"},"useSDKCreds":true,"key":"argo-job-test-migration-v1-zgpnz/argo-job-test-migration-v1-zgpnz-test-migration-v1-template-2007578379"}},"serviceAccountName":"test-migration","securityContext":{"fsGroup":65534}}
      ARGO_NODE_ID:                       argo-job-test-migration-v1-zgpnz-2007578379
      ARGO_INCLUDE_SCRIPT_OUTPUT:         false
      ARGO_DEADLINE:                      2025-11-23T09:30:13Z
      ARGO_PROGRESS_FILE:                 /var/run/argo/progress
      ARGO_PROGRESS_PATCH_TICK_DURATION:  1m0s
      ARGO_PROGRESS_FILE_TICK_DURATION:   3s

    Mounts:
      /argo/secret/argo-artifact-secret from argo-artifact-secret (ro)
      /var/run/argo from var-run-argo (rw)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6xgxr (ro)
      /vault/secrets from vault-secrets (rw)
  test-config-client:
    Container ID:   containerd://119245520cdd21aa00c6b87d73bdd5210677b6ae71f6336bab1c42f003613ea7
    Image:          busybox:latest
    Image ID:       docker.io/library/busybox@sha256:9ae97d36d26566ff84e8893c64a6dc4fe8ca6d1144bf5b87b2b85a32def253c7
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 11 Jul 2024 09:30:26 +0000
      Finished:     Thu, 11 Jul 2024 09:30:26 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      APP_NAME:                           test-migration
      VERSION:                            master
      ARGO_CONTAINER_NAME:                test-config-client
      ARGO_TEMPLATE:                      {"name":"test-migration-v1-template","inputs":{"parameters":[{"name":"script","value":"echo hi"}]},"outputs":{},"metadata":{"annotations":{"co.elastic.logs/enabled":"true","co.elastic.logs/json.add_error_key":"true","co.elastic.logs/json.keys_under_root":"true","co.elastic.logs/json.message_key":"log","fluentbit.job.name":"job-test-migration-v1","vault.hashicorp.com/agent-image":"hashicorp/vault:1.6.2","vault.hashicorp.com/agent-inject":"true","vault.hashicorp.com/agent-inject-secret-secrets.txt":"kv/apps/perf/test-migration","vault.hashicorp.com/agent-inject-template-secrets.txt":"{{ with secret \"kv/apps/dev/test-migration\" }}{{ range $k, $v := .Data.data }} {{ $k }}={{ $v }}{{ end }}{{ end }}\n","vault.hashicorp.com/role":"perf"}},"container":{"name":"test-migration-v1","image":"nginx","command":["echo hi"],"resources":{},"volumeMounts":[{"name":"shared-data","mountPath":"/data"}]},"volumes":[{"name":"shared-data","emptyDir":{}}],"initContainers":[{"name":"test-config-client","image":"busybox:latest","env":[{"name":"APP_NAME","value":"test-migration"},{"name":"VERSION","value":"master"}],"resources":{},"volumeMounts":[{"name":"shared-data","mountPath":"/data"}]}],"insecure":false,"accessKeySecret":{"name":"argo-artifact-secret","key":"awsAccessKey"},"secretKeySecret":{"name":"argo-artifact-secret","key":"awsSecretAccessKey"},"useSDKCreds":true,"key":"argo-job-test-migration-v1-zgpnz/argo-job-test-migration-v1-zgpnz-test-migration-v1-template-2007578379"}},"serviceAccountName":"test-migration","securityContext":{"fsGroup":65534}}
      ARGO_NODE_ID:                       argo-job-test-migration-v1-zgpnz-2007578379
      ARGO_INCLUDE_SCRIPT_OUTPUT:         false
      ARGO_DEADLINE:                      2025-11-23T09:30:13Z
      ARGO_PROGRESS_FILE:                 /var/run/argo/progress
      ARGO_PROGRESS_PATCH_TICK_DURATION:  1m0s
      ARGO_PROGRESS_FILE_TICK_DURATION:   3s
    Mounts:
      /data from shared-data (rw)
      /var/run/argo from var-run-argo (rw)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6xgxr (ro)
      /vault/secrets from vault-secrets (rw)
  vault-agent-init:
    Container ID:  containerd://42b7df5472e7c869cf0d572491163156499739479617d8262e1843f63d1db6a3
    Image:         hashicorp/vault:1.6.2
    Image ID:      docker.io/hashicorp/vault@sha256:0d9048f209d2b754a5a92512e3d96fd010cc2888f162e68df515f26e93347156
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -ec
    Args:
      echo ${VAULT_CONFIG?} | base64 -d > /home/vault/config.json && vault agent -config=/home/vault/config.json
    State:          Running
      Started:      Thu, 11 Jul 2024 09:30:27 +0000
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  128Mi
    Requests:
      cpu:     250m
      memory:  64Mi
    Environment:
      NAMESPACE:                    perf (v1:metadata.namespace)
      HOST_IP:                       (v1:status.hostIP)
      POD_IP:                        (v1:status.podIP)
      VAULT_LOG_LEVEL:              info
      VAULT_LOG_FORMAT:             standard
      VAULT_CONFIG:                 eyJhdXRvX2F1dGgiOnsibWV0aG9kIjp7InR5cGUiOiJrdWJlcm5ldGVzIiwibW91bnRfcGF0aCI6ImF1dGgva3ViZXJuZXRlcyIsImNvbmZpZyI6eyJyb2xlIjoicGVyZiIsInRva2VuX3BhdGgiOiIvdmFyL3J1bi9zZWNyZXRzL2t1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvdG9rZW4ifX0sInNpbmsiOlt7InR5cGUiOiJmaWxlIiwiY29uZmlnIjp7InBhdGgiOiIvaG9tZS92YXVsdC8udmF1bHQtdG9rZW4ifX1dfSwiZXhpdF9hZnRlcl9hdXRoIjp0cnVlLCJwaWRfZmlsZSI6Ii9ob21lL3ZhdWx0Ly5waWQiLCJ2YXVsdCI6eyJhZGRyZXNzIjoiaHR0cDovL3ZhdWx0LmFkbWluLnN2Yzo4MjAwIn0sInRlbXBsYXRlIjpbeyJkZXN0aW5hdGlvbiI6Ii92YXVsdC9zZWNyZXRzL3NlY3JldHMudHh0IiwiY29udGVudHMiOiJ7eyB3aXRoIHNlY3JldCBcImt2L2FwcHMvZGV2L3Rlc3QtbWlncmF0aW9uXCIgfX17eyByYW5nZSAkaywgJHYgOj0gLkRhdGEuZGF0YSB9fSB7eyAkayB9fT17eyAkdiB9fXt7IGVuZCB9fXt7IGVuZCB9fVxuIiwibGVmdF9kZWxpbWl0ZXIiOiJ7eyIsInJpZ2h0X2RlbGltaXRlciI6In19In1dLCJ0ZW1wbGF0ZV9jb25maWciOnsiZXhpdF9vbl9yZXRyeV9mYWlsdXJlIjp0cnVlfX0=

    Mounts:
      /home/vault from home-init (rw)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6xgxr (ro)
      /vault/secrets from vault-secrets (rw)
Containers:
  wait:
    Container ID:
    Image:         quay.io/argoproj/argoexec:v3.5.8
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      argoexec
      wait
      --loglevel
      debug
      --log-format
      text
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      ARGO_POD_NAME:                      argo-job-test-migration-v1-zgpnz-test-migration-v1-template-2007578379 (v1:metadata.name)
      ARGO_POD_UID:                        (v1:metadata.uid)
      GODEBUG:                            x509ignoreCN=0
      ARGO_WORKFLOW_NAME:                 argo-job-test-migration-v1-zgpnz
      ARGO_WORKFLOW_UID:                  df1aaf50-a11b-45db-9c8e-20b3fd69ca9e
      ARGO_CONTAINER_NAME:                wait
      ARGO_TEMPLATE:                      {"name":"test-migration-v1-template","inputs":{"parameters":[{"name":"script","value":"echo hi"}]},"outputs":{},"metadata":{"annotations":{"co.elastic.logs/enabled":"true","co.elastic.logs/json.add_error_key":"true","co.elastic.logs/json.keys_under_root":"true","co.elastic.logs/json.message_key":"log","fluentbit.job.name":"job-test-migration-v1","vault.hashicorp.com/agent-image":"hashicorp/vault:1.6.2","vault.hashicorp.com/agent-inject":"true","vault.hashicorp.com/agent-inject-secret-secrets.txt":"kv/apps/perf/test-migration","vault.hashicorp.com/agent-inject-template-secrets.txt":"{{ with secret \"kv/apps/dev/test-migration\" }}{{ range $k, $v := .Data.data }} {{ $k }}={{ $v }}{{ end }}{{ end }}\n","vault.hashicorp.com/role":"perf"}},"container":{"name":"test-migration-v1","image":"nginx","command":["echo hi"],"resources":{},"volumeMounts":[{"name":"shared-data","mountPath":"/data"}]},"volumes":[{"name":"shared-data","emptyDir":{}}],"initContainers":[{"name":"test-config-client","image":"busybox:latest","env":[{"name":"APP_NAME","value":"test-migration"},{"name":"VERSION","value":"master"}],"resources":{},"volumeMounts":[{"name":"shared-data","mountPath":"/data"}]}],:false,"accessKeySecret":{"name":"argo-artifact-secret","key":"awsAccessKey"},"secretKeySecret":{"name":"argo-artifact-secret","key":"awsSecretAccessKey"},,"useSDKCreds":true,"key":"argo-job-test-migration-v1-zgpnz/argo-job-test-migration-v1-zgpnz-test-migration-v1-template-2007578379"}},"serviceAccountName":"test-migration","securityContext":{"fsGroup":65534}}
      ARGO_NODE_ID:                       argo-job-test-migration-v1-zgpnz-2007578379
      ARGO_INCLUDE_SCRIPT_OUTPUT:         false
      ARGO_DEADLINE:                      2025-11-23T09:30:13Z
      ARGO_PROGRESS_FILE:                 /var/run/argo/progress
      ARGO_PROGRESS_PATCH_TICK_DURATION:  1m0s
      ARGO_PROGRESS_FILE_TICK_DURATION:   3s
    Mounts:
      /argo/secret/argo-artifact-secret from argo-artifact-secret (ro)
      /mainctrfs/data from shared-data (rw)
      /tmp from tmp-dir-argo (rw,path="0")
      /var/run/argo from var-run-argo (rw)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6xgxr (ro)
      /vault/secrets from vault-secrets (rw)
  main:
    Container ID:
    Image:         nginx
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /var/run/argo/argoexec
      emissary
      --loglevel
      debug
      --log-format
      text
      --
      echo hi
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      ARGO_CONTAINER_NAME:                main
      ARGO_TEMPLATE:                      {"name":"test-migration-v1-template","inputs":{"parameters":[{"name":"script","value":"echo hi"}]},"outputs":{},"metadata":{"annotations":{"co.elastic.logs/enabled":"true","co.elastic.logs/json.add_error_key":"true","co.elastic.logs/json.keys_under_root":"true","co.elastic.logs/json.message_key":"log","fluentbit.job.name":"job-test-migration-v1","vault.hashicorp.com/agent-image":"hashicorp/vault:1.6.2","vault.hashicorp.com/agent-inject":"true","vault.hashicorp.com/agent-inject-secret-secrets.txt":"kv/apps/perf/test-migration","vault.hashicorp.com/agent-inject-template-secrets.txt":"{{ with secret \"kv/apps/dev/test-migration\" }}{{ range $k, $v := .Data.data }} {{ $k }}={{ $v }}{{ end }}{{ end }}\n","vault.hashicorp.com/role":"perf"}},"container":{"name":"test-migration-v1","image":"nginx","command":["echo hi"],"resources":{},"volumeMounts":[{"name":"shared-data","mountPath":"/data"}]},"volumes":[{"name":"shared-data","emptyDir":{}}],"initContainers":[{"name":"test-config-client","image":"busybox:latest","env":[{"name":"APP_NAME","value":"test-migration"},{"name":"VERSION","value":"master"}],"resources":{},"volumeMounts":[{"name":"shared-data","mountPath":"/data"}]}],:false,"accessKeySecret":{"name":"argo-artifact-secret","key":"awsAccessKey"},"secretKeySecret":{"name":"argo-artifact-secret","key":"awsSecretAccessKey"},,"useSDKCreds":true,"key":"argo-job-test-migration-v1-zgpnz/argo-job-test-migration-v1-zgpnz-test-migration-v1-template-2007578379"}},"serviceAccountName":"test-migration","securityContext":{"fsGroup":65534}}
      ARGO_NODE_ID:                       argo-job-test-migration-v1-zgpnz-2007578379
      ARGO_INCLUDE_SCRIPT_OUTPUT:         false
      ARGO_DEADLINE:                      2025-11-23T09:30:13Z
      ARGO_PROGRESS_FILE:                 /var/run/argo/progress
      ARGO_PROGRESS_PATCH_TICK_DURATION:  1m0s
      ARGO_PROGRESS_FILE_TICK_DURATION:   3s
    Mounts:
      /data from shared-data (rw)
      /var/run/argo from var-run-argo (rw)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6xgxr (ro)
      /vault/secrets from vault-secrets (rw)
  vault-agent:
    Container ID:
    Image:         hashicorp/vault:1.6.2
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -ec
    Args:
      echo ${VAULT_CONFIG?} | base64 -d > /home/vault/config.json && vault agent -config=/home/vault/config.json
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  128Mi
    Requests:
      cpu:     250m
      memory:  64Mi
    Environment:
      NAMESPACE:                    perf (v1:metadata.namespace)
      HOST_IP:                       (v1:status.hostIP)
      POD_IP:                        (v1:status.podIP)
      VAULT_LOG_LEVEL:              info
      VAULT_LOG_FORMAT:             standard
      AWS_STS_REGIONAL_ENDPOINTS:   regional
    Mounts:
      /home/vault from home-sidecar (rw)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6xgxr (ro)
      /vault/secrets from vault-secrets (rw)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  aws-iam-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  86400
  var-run-argo:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  tmp-dir-argo:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  shared-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  argo-artifact-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  argo-artifact-secret
    Optional:    false
  kube-api-access-6xgxr:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
  home-init:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  home-sidecar:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  vault-secrets:
    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:      Memory
    SizeLimit:   <unset>
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  2m3s  default-scheduler  Successfully assigned perf/argo-job-test-migration-v1-zgpnz-test-migration-v1-template-2007578379 to ip-<IP_ADDRESS>
  Normal  Pulling    2m2s  kubelet            Pulling image "quay.io/argoproj/argoexec:v3.5.8"
  Normal  Pulled     115s  kubelet            Successfully pulled image "quay.io/argoproj/argoexec:v3.5.8" in 6.787386635s (6.787401053s including waiting)
  Normal  Created    115s  kubelet            Created container init
  Normal  Started    115s  kubelet            Started container init
  Normal  Pulling    114s  kubelet            Pulling image "busybox:latest"
  Normal  Pulled     110s  kubelet            Successfully pulled image "busybox:latest" in 4.327784116s (4.327797931s including waiting)
  Normal  Created    110s  kubelet            Created container test-config-client
  Normal  Started    110s  kubelet            Started container test-config-client
  Normal  Pulled     109s  kubelet            Container image "hashicorp/vault:1.6.2" already present on machine
  Normal  Created    109s  kubelet            Created container vault-agent-init
  Normal  Started    109s  kubelet            Started container vault-agent-init

  Note- It doesn't matter if pod running or not , I am expecting logs to come even if pods failed.
  Also When stop the init container, I starts to get logs for the pod even when it is failed.
  Regenerating issue simple, You need some init container, side car container and Rest template can be created around it. I already shared sample templates from which I regenerate shared with you.

That is better, but you only completed a single one of the checkboxes I listed out for you above.
This reproduction is better, but still not minimal:

you have generated fields in your manifests such as creationTimestamp, generation, resourceVersion, uid, labels, annotation, and status. Some of these will make it impossible to apply directly.
you have many other unused fields, including fields with values of {}, unused volumes + volumeMounts (they do nothing in your manifest), unused env
you have other fields that are unnecessary to the reproduction, such as synchronization, concurrencyPolicy, activeDeadlineSeconds, ttlStrategy, and an input that can just be hard-coded
you provided an SA but none of its RBAC. As far as I can tell, you would only need RBAC for Argo (workflowtaskresults) and Vault (depending on your set-up)

Minimal means as small as possible. The minimum necessary to reproduce, with nothing extra.
I made several edits to your comment, please read them and note the differences. Please note how they substantially shrink the reproduction.
You also had some poor formatting, please see the multiple significant edits I made to your comment.
Those edits take a lot of time and are things you could have done yourself initially. In the future, when asking for assistance or reporting an issue to any OSS repo, please make those yourself.
You also did not answer my two fluentbit-related questions, which I suspect are critical to the root cause of this issue. If your output makes it to stdout, then it is likely not Argo-related, but specific your fluentbit config.

argo job pod - manifest

This isn't a manifest, this is the output of kubectl describe pod, whereas I had asked for kubectl get pod -o yaml.
This may suffice though. The init, main, and wait containers are all there, as is your additional initContainer and your vault-agent-init and vault-agent containers.
Although this seems to be a running Pod, not a completed one.

Also When stop the init container, I starts to get logs for the pod even when it is failed.

Does the init container not run to completion on its own? Is it always stuck?
From the events, it looks like it does not finish, since it started 109s ago.
That would be the issue if so (not log propagation); there are no logs because the Pod is stuck and all containers haven't run. The question would then be why is the vault-agent-init container stuck. fluentbit and logs would be entirely unrelated.

That is better, but you only completed a single one of the checkboxes I listed out for you above.
This reproduction is better, but still not minimal:

you have generated fields in your manifests such as creationTimestamp, generation, resourceVersion, uid, labels, annotation, and status. Some of these will make it impossible to apply directly.
you have many other unused fields, including fields with values of {}, unused volumes + volumeMounts (they do nothing in your manifest), unused env
you have other fields that are unnecessary to the reproduction, such as synchronization, concurrencyPolicy, activeDeadlineSeconds, ttlStrategy, and an input that can just be hard-coded
you provided an SA but none of its RBAC. As far as I can tell, you would only need RBAC for Argo (workflowtaskresults) and Vault (depending on your set-up)
I also don't have any of your Vault config -- ideally you can make a reproduction where you don't need any secrets and inject a Vault agent for the sole purpose of reproduction. Alternatively you can make simple, fake secrets, but the former would be simpler and therefore preferable.

Minimal means as small as possible. The minimum necessary to reproduce, with nothing extra.
I made several edits to your comment, please read them and note the differences. Please note how they substantially shrink the reproduction. You also had some poor formatting, please see the multiple significant edits I made to your comment. Those edits take a lot of time and are things you could have done yourself initially. In the future, when asking for assistance or reporting an issue to any OSS repo, please make those yourself.
Thank You. It looks good. I noted this.
You also did not answer my two fluentbit-related questions, which I suspect are critical to the root cause of this issue. If your output makes it to stdout, then it is likely not Argo-related, but specific your fluentbit config.
In my day time I will share you my config for fluent bit.

argo job pod - manifest

This isn't a manifest, this is the output of kubectl describe pod, whereas I had asked for kubectl get pod -o yaml. This may suffice though. The init, main, and wait containers are all there, as is your additional initContainer and your vault-agent-init and vault-agent containers.
Although this seems to be a running Pod, not a completed one.
Yes it is not completed one but I am expecting that failure logs also. I can able to get the failure logs after remove initcontainer section.

Also When stop the init container, I starts to get logs for the pod even when it is failed.

Does the init container not run to completion on its own? Is it always stuck? That would be a very important, critical detail that you did not previously mention. From the events, it looks like it does not finish, since it started 109s ago.
I mean to say, if I removed init container section from template, I am getting logs.
Even if my initcontainer completed, unable to get logs. I regenerated issue by doing this with public images.
That would be the issue if so (not log propagation); there are no logs because the Pod is stuck and all containers haven't run. The question would then be why is the vault-agent-init container stuck. fluentbit and logs as a whole would be entirely unrelated. That container is not running Emissary as a parent process though, so it's purely running Vault code.
Vault agent is stuck because it is not having that secret. Just for regenerate I have not created secret. I was able to regenerate without creating secret in vault also.
I am aware my pod will not complete because of this. But I am seeing that failure logs also in kibana if remove initcontainer section.

Thanks Anton. Its strange! When I removed this annotations co.elastic.logs/json.keys_under_root: "true" I am getting logs.
But I am confused here, how the same set of annotations works in 3.3 version.
Any logging related changes do I need to make in this version, please let me know if you have any idea.
Fluent Bit Conf -
apiVersion: v1
data:
 custom_parsers.conf: |
  [PARSER]
    Name docker_no_time
    Format json
    Time_Keep Off
    Time_Key time
    Time_Format %Y-%m-%dT%H:%M:%S.%L
  [PARSER]
    Name cri
    Format regex
    Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
    Time_Key time
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z
 fluent-bit.conf: |
  [SERVICE]
    Flush 1
    Daemon Off
    Parsers_File parsers.conf
    Parsers_File custom_parsers.conf
    HTTP_Server On
    HTTP_Listen <IP_ADDRESS>
    HTTP_Port 2020
     # Log_Level debug
    storage.metrics on

  [INPUT]
    Name tail
    Path /var/log/containers/*.log
    Exclude_Path /var/log/containers/argo-job-*.log,/var/log/containers/argo-workflow-*.log
    Refresh_Interval 10
    multiline.parser docker, cri
    Tag jio_service.*
    Mem_Buf_Limit 50MB
    Skip_Long_Lines On
  [INPUT]
    Name tail
    Path /var/log/containers/argo-job-*.log
    Exclude_Path /var/log/containers/argo-workflow-*.log
    Refresh_Interval 10
    multiline.parser docker, cri
    Tag argo_job.*
    Mem_Buf_Limit 50MB
    Skip_Long_Lines On
[INPUT]
    Name tail
    Path /var/log/containers/ingress-*.log
    Exclude_Path /var/log/containers/argo-workflow-*.log
    Refresh_Interval 10
    multiline.parser docker, cri
    Tag ingress.*
    Mem_Buf_Limit 100MB
    Buffer_Chunk_Size 1MB
    Buffer_Max_Size 5MB
    Skip_Long_Lines Off

  [FILTER]
    Name kubernetes
    Match jiao_service.*
    Labels On
    Annotations Off
    Merge_Log On
    Keep_Log On
    K8S-Logging.Parser On
    K8S-Logging.Exclude On
    Kube_Tag_Prefix neo_service.var.log.containers.
  [FILTER]
    Name kubernetes
    Match argo_job.*
    Labels On
    Annotations On
    Merge_Log On
    Keep_Log On
    K8S-Logging.Parser On
    K8S-Logging.Exclude Off
    Kube_Tag_Prefix argo_job.var.log.containers.
  [FILTER]
    Name kubernetes
    Match ingress.*
    Labels On
    Annotations On
    Merge_Log On
    Keep_Log On
    K8S-Logging.Parser On
    K8S-Logging.Exclude On
    Kube_Tag_Prefix ingress.var.log.containers.
[OUTPUT]
    Name es
    Match jiao_service.*
    Host multi-client
    Port 9200
    HTTP_User elastic
    HTTP_Passwd test@123
    Logstash_Format On
    logstash_prefix_key $kubernetes['namespace_name'].$kubernetes['container_name']
    Retry_Limit 5
    tls off
    tls.verify off
    Suppress_Type_Name On
    Replace_Dots On
    Trace_Error on
    Generate_ID On
  [OUTPUT]
    Name es
    Match argo_job.*
    Host multi-client
    Port 9200
    HTTP_User elastic
    HTTP_Passwd test@123
    Logstash_Format On
    logstash_prefix_key $kubernetes['namespace_name'].$kubernetes['annotations']['fluentbit.job.name']
    Retry_Limit 5
    tls off
    tls.verify off
    Replace_Dots On
    Suppress_Type_Name On
    Generate_ID On
  [OUTPUT]
    Name es
    Alias Ingress
    Match ingress.*
    Host multi-client
    Port 9200
    HTTP_User elastic
    HTTP_Passwd test@123
    Logstash_Format On
    logstash_prefix_key $kubernetes['namespace_name'].$kubernetes['labels']['app.kubernetes.io/name']
    Retry_Limit 5
    tls off
    tls.verify off
    net.keepalive Off
    Suppress_Type_Name On
    Replace_Dots On
    Generate_ID On

Thanks Anton. Its strange! When I removed this annotations co.elastic.logs/json.keys_under_root: "true" I am getting logs.🙂

This is therefore a problem in fluentbit, and not in argo workflows.

Any logging related changes do I need to make in this version, please let me know if you have any idea. Because not sure if removing this annotations can cause any other issue.

I suggest you research or ask around in the elastic/fluentbit community.

Still one question comes in my mind, if possible for you can answer, if same annotations works in 3.3 version of argo, what is that in 3.4+ version of argo workflow stopping it to send log and interestingly without removing mentioned elastic annotations also it work if I stop/remove sidecar or init container in the pod.
Thanks!

I don't know.

Upon further investigation, I discovered that there has been a change in metadata(annotations) for Workflowtemplate CRDs in version 3.4+ of Argo. This change has affected fluentbit's ability to process logs for jobs that specifically utilize Workflowtemplate. Additionally, due to increased metadata for pods, fluentbit now requires a larger buffer size (default 32KB) to temporarily store log data. Solution for the users who faced the issue is just go to fluent bit config and add this in kubernetes filter. Buffer_Size 5MB
[FILTER]
      Name kubernetes
      Match argo_job.*
      Labels On
      Annotations On
      Merge_Log On
      Keep_Log On
      Buffer_Size 5MB
      K8S-Logging.Parser On
      K8S-Logging.Exclude Off
      Kube_Tag_Prefix argo_job.var.log.containers.

In my day time I will share you my config for fluent bit.

Note that I did not ask for your fluentbit config, I asked you to tick off the checkboxes I wrote out for you above.

Note - Its neither fluent bit issue nor argo issue It is where one should fine tune the configuration with respect to such changes.

Specifically, this answers the 3rd one.
In the future, I would please ask, once again, that you answer questions from maintainers and follow directions on minimal repros and such. Doing so will also make you a better debugger and perhaps even able to root cause and resolve such issues on your own.


Look ahead binary sensor - will price be below x in next n hours

Just sketching this out at the moment, but I plan to add the following:
Add a binary sensor that will be set to true if the price will go below a certain point, n, within the next x hours, up to the maximum look ahead that Octopus Energy provide price data for.
e.g.
binary_sensor.octopus_below_n
Use case example:
If Octopus price is going below n, at some point in the near future, turn off gas fired hot water program, as immersion heater will be used when price goes below n instead.
Current sensors don't cover this, they give current price or cheapest price for a certain length of time.
Things to consider:
When time period has passed that price was below low point, set back to false, but not if it will drop below low point again within the next x hours.

Awesome idea, it would also be helpful to be able to say "below a avg price for the next x hours" as maybe you need a full hour to make it worth using. So this would not give you the cheapest time but in the next 4 hours you should use the immersion heater in 2 hours time for 1 hour as your boost.


fix(eslint): add package_json

Added the path to package.json.
Please check it.

@mjlbach
I did some research on eslint.
In vscode-eslint, there were many differences in the configuration values from the server.
Those that are in the package.json configuration and are also in the server definition:
"eslint.codeAction.disableRuleComment"
"eslint.codeAction.showDocumentation"
"eslint.codeActionsOnSave.mode"
"eslint.codeActionsOnSave.rules"
"eslint.nodePath"
"eslint.onIgnoredFiles"
"eslint.options"
"eslint.packageManager"
"eslint.quiet"
"eslint.run"
"eslint.useESLintClass"
"eslint.validate"

Not in the package.json setting, but in the server definition (Value not set in VSCode?):
"eslint.format"
"eslint.rulesCustomizations"
"eslint.workspaceFolder"

A value that is in the package.json setting but not in the server definition (VSCode's own setting?):
"eslint.alwaysShowStatus"
"eslint.autoFixOnSave"
"eslint.debug"
"eslint.enable"
"eslint.execArgv"
"eslint.format.enable"
"eslint.lintTask.enable"
"eslint.lintTask.options"
"eslint.migration.2_x"
"eslint.nodeEnv"
"eslint.probe"
"eslint.provideLintTask"
"eslint.rules.customizations"
"eslint.runtime"
"eslint.trace.server"
"eslint.workingDirectories"

The more you look into these the more I am tempted to just remove all package.json :/ I didn't add the feature, and I think that as the coc plugins show it's better to just curate your own settings list from the server documentation.

Right, it's probably not a good idea to generate documentation based on the configuration list in package.json.
However, it would be very hard to create all the lists manually.
I can't think of a good idea.
Or maybe a script that adds what is missing and removes what is not needed based on the package.json?
This would also be a lot of work, though.


Added the bun installation guide to README.md

This is to close out #15.

thank you for the fixes! I will merge this into the main branch!


Implicitly create a virtual environment using the Python version specified in pyproject.toml

Issue
In the absence of an activated virtual environment (I insist), the poetry add, poetry build, poetry install, poetry lock, poetry remove, poetry run, poetry shell, poetry show and poetry update commands implicitly create a Poetry built-in virtual environment. However currently, the Python interpreter that they include in this virtual environment is the one called by the #!/usr/bin/env python shebang line of the ~/.poetry/bin/poetry script. The problem is that on MacOS and most Linux distributions, /usr/bin/env python resolves to the Python 2.7 interpreter.
So if for instance a user specified in his pyproject.toml file a dependency on the Python 3.7 interpreter:
[tool.poetry.dependencies]
python = "^3.7"

when he runs one of the poetry add, poetry build, poetry install, poetry lock, poetry remove, poetry run, poetry shell, poetry show and poetry update commands for the first time, a RuntimeError is raised:
[RuntimeError]
The current Python version (2.7.14) is not supported by the project (^3.7)
Please activate a compatible Python version.

This is a terrible user experience. Many users have been repeatedly complaining about this since Poetry's inception, so this issue cannot be ignored: https://github.com/sdispater/poetry/issues/77, https://github.com/sdispater/poetry/issues/305, https://github.com/sdispater/poetry/issues/522, https://github.com/sdispater/poetry/issues/655, https://github.com/sdispater/poetry/issues/721, https://github.com/sdispater/poetry/issues/952, https://github.com/sdispater/poetry/issues/988.
Solution
In https://github.com/sdispater/poetry/issues/988, @Suor made an excellent suggestion that could address this issue:

add a command that allows the explicit creation of Poetry built-in virtual environments with an option to specify the Python interpreter;
use the Python interpreter specified in the tool.poetry.dependencies.python property of the pyproject.toml file for the implicit creation of Poetry built-in virtual environments.

In PR https://github.com/sdispater/poetry/pull/731 that will be part of Poetry 1.0.0, @sdispater added a new poetry env command that implements the first point, which is great. Thank you for that @sdispater. But that only addresses half of the problem, the explicit virtual environment creation problem. That does not address the implicit virtual environment creation problem that still raises a RuntimeError.
If we also implemented the second point I think it would greatly improve user experience.

That's something that I already mentioned before but, for instance, ^3.4 means >=3.4,<4.0 so which Python version should Poetry choose? The highest one available? The lowest? There is no clear answer and letting Poetry guess might not be the best approach here.
Now, maybe we could improve the error message by mentioning the env command.

Agree on mentioning the env command.
Recommend hard-setting the Python version rather than making it a requirement.

@sdispater

That's something that I already mentioned before but, for instance, ^3.4 means >=3.4,<4.0 so which Python version should Poetry choose?

Contrary to any other dependencies, the lowest version available I guess. Because otherwise I am not sure how we would do the lookup. For instance with the >=3.4,<4.0 constraint, where would we start? python3.9? What if there is a python3.10 or python3.11?
And even if we look up only by the major version we will have the same issue for constraints without upper limits. For instance with the >=3.7 constraint, where would we start? python3? What if there is a python4 or python5? So unless there is a reliable way to get the highest version available, I think we have to choose the lowest version.

What do you think?

Now, maybe we could improve the error message by mentioning the env command.

That is a reasonable alternative.
But I think I prefer the implicit lookup described above. Maybe the best approach is to do both: if the implicit lookup fails, we fallback on the error message suggesting the use of the poetry env command?

Here's one approach to consider:
Attempt to run python -V, python3 -V, and pythonA.B -V, where A and B are each possible python version. Parse stdout and stderr - note that old versions of Python output this to stderr. Use this to compile a list of Python versions on the PATH, and their aliases. Match the compatible-with-pyproject.toml subset's length: 0 means abort with an error. 1 Means use that alias. Multiple means ask the user.
Note that this won't pick up on Python installations not on the PATH.

That's something that I already mentioned before but, for instance, ^3.4 means >=3.4,<4.0 so which Python version should Poetry choose?

Contrary to any other dependencies, the lowest version available I guess. Because otherwise I am not sure how Poetry would do the lookup. For instance with the >=3.4,<4.0 constraint, where would Poetry start? python3.9? What if there is a python3.10 or python3.11?

If you want to use the lowest version possible, does it mean that you would prefer 3.6.0 to 3.6.1 if both are installed?

@stinovlas

If you want to use the lowest version possible, does it mean that you would prefer 3.6.0 to 3.6.1 if both are installed?

If you have the ^3.4 (>=3.4,<4.0) constraint in your pyproject.toml file, Poetry would use that lookup order: [python3.4, python3, python]. So the Python version that would be called depends on which Python version python3 and python point to on your machine.

@maggyero

If you have the ^3.4 (>=3.4,<4.0) constraint in your pyproject.toml file, Poetry would use that lookup order: [python3.4, python3, python]. So the Python version that would be called depends on which Python version python3, and if it does not exist, python, point to on your machine (assuming python3.4 does not exist).

Oh, i see, so you are just trying different binaries in PATH, from the most specific to the least specific.
So, let's say that I have python >= 3.6 specified in my pyproject.toml. According to you, the resolution order would be [python3.6, python3, python]. Now, let's assume that I have python2.7, python3.5 and python3.7 installed on my system. Binary python3 is pointing to python3.5 and binary python is pointing to python2.7.

The first option (python3.6) does not exist, so we skip it.
The second option (python3) points to python3.5, so we should not create virtualenv with it (project requires python >= 3.6)
The third option (python) points to python2.7, so we should not create virtualenv with it (same as above)

It seems that there is no option left. But there is! We also have python3.7 installed on our system, which satisfies the project requirement.

@David-OConnor I like the resolution you suggested before (letting the user choose when there are multiple versions available), I'm just disputing the "use the minimal version" attitude :-).

Contrary to any other dependencies, the lowest version available I guess.

This seems very bad to me. I think the interpretation of ^3.4 should be the same for python as it is for any other dependency.
I don’t see why poetry shouldn’t treat python the same as any other dependency in this regard — if python 3.9, 3.10, etc are available on the path then use it, the same as you would if dependency X was pinned in such a way that v3.9 was the most recent compatible version available on PyPI.
If you want to impose a cap on the python version then just mark it as such.
I think it could make sense to be able to specify the desired version of python for a development environment as part of pyproject.toml, and to use it if possible (allowing and using compatible fallbacks), but I don’t think there should be an implicit python version selection process that differs from the standard dependency version selection process.

@stinovlas

So, let's say that I have python >= 3.6 specified in my pyproject.toml. According to you, the resolution order would be [python3.6, python3, python]. Now, let's assume that I have python2.7, python3.5 and python3.7 installed on my system. Binary python3 is pointing to python3.5 and binary python is pointing to python2.7.

The first option (python3.6) does not exist, so we skip it.
The second option (python3) points to python3.5, so we should not create virtualenv with it (project requires python >= 3.6)
The third option (python) points to python2.7, so we should not create virtualenv with it (same as above)

It seems that there is no option left. But there is! We also have python3.7 installed on our system, which satisfies the project requirement.

Yes of course, but how could Poetry know that python3.7 is installed? If you expect Poetry to increment the minor version number until it finds a Python interpreter (so [python3.6, python3.7, python3.8, …]), where should it stop if there is no Python 3 interpreter installed on the system? That is why I said that Poetry should only look for the version specified in the pyproject.toml file, from the most specific to the least specific. So for the python = ">=3.6" constraint, as you said, the lookup order would be: [python3.6, python3, python]. If these files do not exist in the PATH environment variable, Poetry should display the error message suggesting to specify the Python interpreter explicitly with the poetry env command.

Contrary to any other dependencies, the lowest version available I guess.

This seems very bad to me. I think the interpretation of ^3.4 should be the same for python as it is for any other dependency.
I don’t see why poetry shouldn’t treat python the same as any other dependency in this regard — if python 3.9, 3.10, etc are available on the path then use it, the same as you would if dependency X was pinned in such a way that v3.9 was the most recent compatible version available on PyPI.

I would also prefer Poetry to find the highest Python version available in the PATH environment variable instead of the lowest for consistency with other dependencies. But I would you do that reliably? If the constraint is python = ">=3.6", do you start at python3.9? python3.24?  python5.3? For library dependencies that is different, PyPI API can give you the last version, but here we don't use PyPI API, we use the PATH environment variable of the OS.

@dmontagu

Contrary to any other dependencies, the lowest version available I guess.

This seems very bad to me. I think the interpretation of ^3.4 should be the same for python as it is for any other dependency.
I don’t see why poetry shouldn’t treat python the same as any other dependency in this regard — if python 3.9, 3.10, etc are available on the path then use it, the same as you would if dependency X was pinned in such a way that v3.9 was the most recent compatible version available on PyPI.

I would also prefer Poetry to find the highest Python version available in the PATH environment variable instead of the lowest for consistency with other dependencies. But I would you do that reliably? If the constraint is python = ">=3.6", do you start at python3.9? python3.24?  python5.3? For library dependencies that is different, PyPI API can give you the last version, but here we don't use PyPI API, we use the PATH environment variable of the OS.

@David-OConnor I like the resolution you suggested before (letting the user choose when there are multiple versions available), I'm just disputing the "use the minimal version" attitude :-).

I agree - lowest version is inconsistent with what a user expects from semvar, and how dependency versions are chosen.

Yes of course, but how could Poetry know that python3.7 is installed? If you expect Poetry to increment the minor version number until it finds a Python interpreter (so [python3.6, python3.7, python3.8, …]), where should it stop if there is no Python 3 interpreter installed on the system?

3.10 or so. It's computationally quick to check these.

Yeah, my point wasn't that I think poetry needs to support finding exactly the right version in the PATH according to semver; it was just that if poetry is going to support this, I don't think it should be using different logic. (In particular, I would prefer there be no implicit python selection over choosing the minimum compatible.)
From a practical perspective, I personally would be fine with @David-OConnor 's suggestion of checking for a limited list of future versions (if it were sufficiently fast). I would also be fine with requiring some other form of explicit specification for the desired default environments (e.g., an extra key in the poetry config with value listing out the python executables to look for).

This has been discussed before, and other people have linked to this comment in other issues:

It has been discussed before and the decision remains: Poetry is not and never will be a Python management tool. This is something that tools like pyenv do really well and is not something a package/dependency manager should do.

I don't think this is going to happen, and in my personal experience, working with pyenv works very well for this, especially if you just add a .python-version file to your project.

@David-OConnor

3.12 or so, so we won't have to change it for a while. It's computationally quick to check these. Re 3.24 and 5.3: Could go up to 3.24, but no point. 5.3 isn't possible.

5.3 is not currently possible.

We know what versions of Python are avail, and expected to be released soon.

Yes, but that means we have to hard code this into Poetry and update it frequently. It creates a maintenance burden. I would prefer a future-proof algorithm, that is why I suggested to use the lowest version in the first place. But this is not ideal.
The more I think about it, the more I think we should stick to PEP 394:

Distributors may choose to set the behavior of the python command as follows:

python2,
python3,
not provide python command,
allow python to be configurable by an end user or a system administrator.

That means we should only look for python and pythonX in the OS PATH (not pythonX.Y nor pythonX.Y.Z), that is to say we should not be more specific that the major version. And since we want to look for the highest Python version for consistency with the dependency lookup mechanism, we should hard code in Poetry that the current highest Python major version is 3. And with the relatively slow Python release cycle, we will not update this number frequently so there is actually no maintenance.
So with the following pyproject.toml requirements, Poetry should use the following PATH lookup orders:

python = ">=3.4.1": [python3, python];
python = ">=2.6.3": [python3, python2, python];
python = ">=2.6.3,<3.0.0": [python2, python].

And when Python 4 will be available, we will update the current highest Python major version to 4 in Poetry so the PATH lookup orders will be:

python = ">=3.4.1": [python4, python3, python];
python = ">=2.6.3": [python4, python3, python2, python];
python = ">=2.6.3,<3.0.0": [python2, python].

In all cases, if no pythonX or python is found in the PATH, or one is found but its version (given by pythonX -V or python -V) does not match the pyproject.toml requirements, the following message error should be raised, prompting the user to specify a Python version with the poetry env command:

[RuntimeError]
The current Python version (2.7.14) is not supported by the project (^3.7)
Please activate a compatible Python version explicitly with poetry env use <python> or another virtual environment manager.

@David-OConnor

3.12 or so, so we won't have to change it for a while. It's computationally quick to check these. Re 3.24 and 5.3: Could go up to 3.24, but no point. 5.3 isn't possible.

5.3 is not currently possible.

We know what versions of Python are avail, and expected to be released soon.

Yes, but that means we have to hard code this into Poetry and update it frequently. It creates a maintenance burden. I would prefer a future-proof algorithm, that is why I suggested to use the lowest version in the first place. But this is not ideal.
@dmontagu

Yeah, my point wasn't that I think poetry needs to support finding exactly the right version in the PATH according to semver; it was just that if poetry is going to support this, I don't think it should be using different logic.

I agree.
The more I think about it, the more I think we should stick to PEP 394:

Distributors may choose to set the behavior of the python command as follows:

python2,
python3,
not provide python command,
allow python to be configurable by an end user or a system administrator.

That means we should only look for python and pythonX in the OS PATH (not pythonX.Y nor pythonX.Y.Z), that is to say we should not be more specific that the major version. And since we want to look for the highest Python version for consistency with the dependency lookup mechanism, we should hard code in Poetry that the current highest Python major version is 3. And with the relatively slow Python release cycle, we will not update this number frequently so there is actually no maintenance.
So with the following pyproject.toml requirements, Poetry should use the following PATH lookup orders:

python = ">=3.4.1": [python3, python];
python = ">=2.6.3": [python3, python2, python];
python = ">=2.6.3,<3.0.0": [python2, python].

And when Python 4 will be available, we will update the current highest Python major version to 4 in Poetry so the PATH lookup orders will be:

python = ">=3.4.1": [python4, python3, python];
python = ">=2.6.3": [python4, python3, python2, python];
python = ">=2.6.3,<3.0.0": [python2, python].

In all cases, if no pythonX or python is found in the PATH, or one is found but its version (given by pythonX -V or python -V) does not match the pyproject.toml requirements, the following message error should be raised, prompting the user to specify a Python version with the poetry env command:

[RuntimeError]
The current Python version (2.7.14) is not supported by the project (^3.7)
Please activate a compatible Python version explicitly with poetry env use <python> or another virtual environment manager.

@David-OConnor

3.12 or so, so we won't have to change it for a while. It's computationally quick to check these. Re 3.24 and 5.3: Could go up to 3.24, but no point. 5.3 isn't possible.

5.3 is not currently possible.

We know what versions of Python are avail, and expected to be released soon.

Yes, but that means we have to hard code this into Poetry and update it frequently (new Python minor versions are released every year). It creates a maintenance burden. I would prefer a future-proof algorithm, that is why I suggested to look up the lowest Python version instead. But this is not ideal.
@dmontagu

Yeah, my point wasn't that I think poetry needs to support finding exactly the right version in the PATH according to semver; it was just that if poetry is going to support this, I don't think it should be using different logic.

I agree.
Solution
The more I think about it, the more I think we should just stick to PEP 394:

Distributors may choose to set the behavior of the python command as follows:

python2,
python3,
not provide python command,
allow python to be configurable by an end user or a system administrator.

This means that Poetry should only look up python and pythonX in the OS PATH (not pythonX.Y nor pythonX.Y.Z), that is to say we should not be more specific that the major Python version. And since we want to look up the highest Python version for consistency with the dependency lookup mechanism, we should hard code in Poetry the current highest Python major version released (3). With the relatively slow Python release cycle, there will be practically no maintenance: just a number to update every 10 years or so (Python 3 was released in 2008 and Python 4 will not be released before 2020, and possibly never).
So with the following pyproject.toml requirements, Poetry will use the following PATH lookup orders:

python = ">=3.4.1": [python3, python];
python = ">=2.6.3": [python3, python2, python];
python = ">=2.6.3,<3.0.0": [python2, python].

When Python 4 is available, we will update the current highest Python major version to 4 in Poetry, so the PATH lookup orders of Poetry will become:

python = ">=3.4.1": [python4, python3, python];
python = ">=2.6.3": [python4, python3, python2, python];
python = ">=2.6.3,<3.0.0": [python2, python].

In all cases, if no pythonX or python is found in the PATH, or one is found but its version (given by pythonX -V or python -V) does not match the pyproject.toml requirements, the following error message will be raised:

[RuntimeError]
The current Python version (2.7.14) is not supported by the project (^3.7)
Please activate a compatible Python version explicitly with poetry env use <python> or another virtual environment manager.

What do you think @sdispater?

@David-OConnor

3.12 or so, so we won't have to change it for a while. It's computationally quick to check these. Re 3.24 and 5.3: Could go up to 3.24, but no point. 5.3 isn't possible.

5.3 is not currently possible.

We know what versions of Python are avail, and expected to be released soon.

Yes, but that means we have to hard code this into Poetry and update it frequently (new Python minor versions are released every year). It creates a maintenance burden. I would prefer a future-proof algorithm, that is why I suggested to look up the lowest Python version instead of the highest one. But this is not ideal.

I think we should be practical here: It will likely be decades before we encounter a python 5.3. It will be years before we encounter a Python 3.11. When we hit the latter, it'll be a trivial update.

@David-OConnor

Popular linux distros like Ubuntu distinguish using python3.6, python3.7 etc. Ignoring this will not meet the intent, which is to distinguish minor versions.

I think it is not Poetry's responsibility to find minor versions. Poetry should only look for python, python2 and python3 as required by PEP 394. If you want a specific Python version you should tell it explicitly with poetry env use <python>. And minor versions change every year, and the community plans to accelerate the cycle, so hard coding the latest minor version is a maintenance burden that is better avoided in my opinion. PEP 394 should be our reference.

I understood your original issue as wanting to specify the environment version using pyproject.toml. Is that not the case?

@David-OConnor

I understood your original issue as wanting to specify the environment's Python version version using pyproject.toml.

My concern was that Poetry looks up python in the PATH, but not python3 or python2, which is contrary to PEP 394 recommendations (that I realized it afterwards).

current Python version is a trap, when you can (and commonly do on Linux) have multiple versions of Python installed. I'm in favor of specifying a version or semvar requirement in pyproject.toml, using that version if available, and displaying an error if not.

I am against this as Poetry provides an explicit way to do it: poetry env use <python>. That <python> argument is OS specific (it can be /usr/bin/python3.7, python3.7, 3.7) so it should not go into the pyproject.toml. Environment parameters should be outside the pyproject.toml.

That  argument is OS specific (it can be /usr/bin/python3.7, python3.7, 3.7) so it should not go into the pyproject.toml. Environment parameters should be outside the pyproject.toml.

I think the python-version should be project specific - If a project contains something like dataclasses or f-strings, you can specify 3.7 or ^3.7 in pyproject.toml.

It seems this issue has been addressed in #1477, so maybe this issue can be closed?

Hi @sdispater I have just tested your PR #1477 with the latest Poetry (1.0.0b5) on MacOS Catalina (10.15.1). (Sorry for not providing any feedback earlier.) Poetry now correctly creates virtual environments based on the requirements of tool.poetry.dependencies.python in pyproject.toml! However it does not follow the specifications in my initial post (https://github.com/sdispater/poetry/issues/1386#issue-495076159, section "[Edit] Specification for the second point").
Consequently, with the value ">=2.7" for tool.poetry.dependencies.python in pyproject.toml, I get two different Python versions installed, according to the Python running the poetry script (note: on my system, python and python2 points to Python 2.7 and python3 on Python 3.7):

Python 2.7 is installed if poetry is run with Python 2.7:

[maggyero@MacBookPro~/Desktop/foo]$ python ~/.poetry/bin/poetry install
Creating virtualenv foo-1XhTtlMR-py2.7 in /Users/maggyero/Library/Caches/pypoetry/virtualenvs
Installing dependencies from lock file
Warning: The lock file is not up to date with the latest changes in pyproject.toml. You may be getting outdated dependencies. Run update to update them.

Package operations: 1 install, 0 updates, 0 removals

  - Installing pika (1.1.0)
  - Installing foo (0.1.0)

Python 3.7 is installed if poetry is run with Python 3.7:

[maggyero@MacBookPro~/Desktop/foo]$ python3 ~/.poetry/bin/poetry install
Creating virtualenv foo-1XhTtlMR-py3.7 in /Users/maggyero/Library/Caches/pypoetry/virtualenvs
Installing dependencies from lock file
Warning: The lock file is not up to date with the latest changes in pyproject.toml. You may be getting outdated dependencies. Run update to update them.

Package operations: 1 install, 0 updates, 0 removals

  - Installing pika (1.1.0)
  - Installing foo (0.1.0)

Both installed Python respect the ">=2.7" requirements but I would expect Poetry to always install the same Python: the highest version available (Python 3.7 in this case), regardless of the Python running poetry.

@maggyero This is the expected behavior. Poetry will always prioritize the currently used Python version. The reason for that is to support the use case of managing Python environments with pyenv for instance, like I do.

@sdispater That makes sense. I am closing this issue. Thanks a lot for you hard work!


[Parser] Make Lexer::getStateForBeginningOfToken() to work with Trivia

CC: @nkcsgexi
Resolve a problem where BacktrackingScope does not restore Trivia state.

@swift-ci Please smoke test

@swift-ci Please smoke test


Improve gist embed FOUC style while loading

Related to #870
The stylesheet for the embed loads after the first content paint, so the Flash Of Unstyled Content overflows and overlaps until that completes. This PR avoid the overflow and makes it look decent during the FUOC.
Before

After

Much better now :)


fix: update script to support custom appimage name

I have read the CLA Document and I hereby sign the CLA