HTTP01 challenge fails, solver pod short-lived and all its traces wiped out

Describe the bug:
The certificate is not being issued, with order and challenge being in a bad state
➜  accounting-service git:(xxxxxxxxx) ✗ kubectl get challenges                                                                                aws-platform-staging-eu-west-1
NAME                                                 STATE     DOMAIN                                            AGE
accounting-service-tls-lccsr-2824239067-3765429214   expired   accounting-service-public.stag.aws.worksome.net   52m
➜  accounting-service git:(xxxxxxxx) ✗ kubectl get orders                                                                                aws-platform-staging-eu-west-1
NAME                                      STATE     AGE
accounting-service-tls-lccsr-2824239067   invalid   52m

➜  accounting-service git:(xxxxxxxxxx) ✗ kubectl describe order accounting-service-tls-lccsr-2824239067                                                                                aws-platform-staging-eu-west-1
Name:         accounting-service-tls-lccsr-2824239067
Namespace:    default
Labels:       <none>
Annotations:  cert-manager.io/certificate-name: accounting-service-tls
              cert-manager.io/certificate-revision: 1
              cert-manager.io/private-key-secret-name: accounting-service-tls-ff7pk
API Version:  acme.cert-manager.io/v1
Kind:         Order
Metadata:
  Creation Timestamp:  2022-01-11T13:32:39Z
  Generation:          1
  Managed Fields:
    API Version:  acme.cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:cert-manager.io/certificate-name:
          f:cert-manager.io/certificate-revision:
          f:cert-manager.io/private-key-secret-name:
        f:ownerReferences:
          .:
          k:{"uid":"c6e54947-8ce7-48c5-92c8-8ec7331d1273"}:
            .:
            f:apiVersion:
            f:blockOwnerDeletion:
            f:controller:
            f:kind:
            f:name:
            f:uid:
      f:spec:
        .:
        f:dnsNames:
        f:issuerRef:
          .:
          f:group:
          f:kind:
          f:name:
        f:request:
      f:status:
        .:
        f:authorizations:
        f:failureTime:
        f:finalizeURL:
        f:state:
        f:url:
    Manager:    controller
    Operation:  Update
    Time:       2022-01-11T13:34:25Z
  Owner References:
    API Version:           cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  CertificateRequest
    Name:                  accounting-service-tls-lccsr
    UID:                   c6e54947-8ce7-48c5-92c8-8ec7331d1273
  Resource Version:        17805529
  UID:                     71323017-b594-4020-ae74-de30a4f607d4
Spec:
  Dns Names:
    accounting-service-public.stag.aws.worksome.net
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   Issuer
    Name:   letsencrypt-staging
  Request:  <THE CERTIFICATE REQUEST BASE64-ENCODED GOES HERE>
Status:
  Authorizations:
    Challenges:
      Token:        HDU7Jy0sqG4bgp1ADI6nACKYYVs0g_5cVfdUsXcVOgg
      Type:         http-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/1401516058/Oc1iHA
      Token:        HDU7Jy0sqG4bgp1ADI6nACKYYVs0g_5cVfdUsXcVOgg
      Type:         dns-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/1401516058/Ax0QhQ
      Token:        HDU7Jy0sqG4bgp1ADI6nACKYYVs0g_5cVfdUsXcVOgg
      Type:         tls-alpn-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/1401516058/qtKlag
    Identifier:     accounting-service-public.stag.aws.worksome.net
    Initial State:  pending
    URL:            https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1401516058
    Wildcard:       false
  Failure Time:     2022-01-11T13:34:25Z
  Finalize URL:     https://acme-staging-v02.api.letsencrypt.org/acme/finalize/39574498/1503494108
  State:            invalid
  URL:              https://acme-staging-v02.api.letsencrypt.org/acme/order/39574498/1503494108
Events:
  Type    Reason   Age   From          Message
  ----    ------   ----  ----          -------
  Normal  Created  53m   cert-manager  Created Challenge resource "accounting-service-tls-lccsr-2824239067-3765429214" for domain "accounting-service-public.stag.aws.worksome.net"

➜  accounting-service git:(xxxxxxxxxxxxxx) ✗ kubectl describe challenge accounting-service-tls-lccsr-2824239067-3765429214                                                                                aws-platform-staging-eu-west-1
Name:         accounting-service-tls-lccsr-2824239067-3765429214
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  acme.cert-manager.io/v1
Kind:         Challenge
Metadata:
  Creation Timestamp:  2022-01-11T13:32:41Z
  Finalizers:
    finalizer.acme.cert-manager.io
  Generation:  1
  Managed Fields:
    API Version:  acme.cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"finalizer.acme.cert-manager.io":
        f:ownerReferences:
          .:
          k:{"uid":"71323017-b594-4020-ae74-de30a4f607d4"}:
            .:
            f:apiVersion:
            f:blockOwnerDeletion:
            f:controller:
            f:kind:
            f:name:
            f:uid:
      f:spec:
        .:
        f:authorizationURL:
        f:dnsName:
        f:issuerRef:
          .:
          f:group:
          f:kind:
          f:name:
        f:key:
        f:solver:
          .:
          f:http01:
            .:
            f:ingress:
              .:
              f:class:
        f:token:
        f:type:
        f:url:
        f:wildcard:
      f:status:
        .:
        f:presented:
        f:processing:
        f:reason:
        f:state:
    Manager:    controller
    Operation:  Update
    Time:       2022-01-11T13:32:44Z
  Owner References:
    API Version:           acme.cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Order
    Name:                  accounting-service-tls-lccsr-2824239067
    UID:                   71323017-b594-4020-ae74-de30a4f607d4
  Resource Version:        17805528
  UID:                     4b30bb88-72b4-4049-8ac4-a2ebc7d2d9fa
Spec:
  Authorization URL:  https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1401516058
  Dns Name:           accounting-service-public.stag.aws.worksome.net
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   Issuer
    Name:   letsencrypt-staging
  Key:      HDU7Jy0sqG4bgp1ADI6nACKYYVs0g_5cVfdUsXcVOgg.-yuHWS8lDxJT_DIqqUoRVEU3PIZV3RT-ln1_oYBMf0A
  Solver:
    http01:
      Ingress:
        Class:  nginx
  Token:        HDU7Jy0sqG4bgp1ADI6nACKYYVs0g_5cVfdUsXcVOgg
  Type:         HTTP-01
  URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/1401516058/Oc1iHA
  Wildcard:     false
Status:
  Presented:   false
  Processing:  false
  Reason:      Error accepting challenge: 400 urn:ietf:params:acme:error:malformed: Unable to update challenge :: authorization must be pending
  State:       expired
Events:
  Type    Reason     Age   From          Message
  ----    ------     ----  ----          -------
  Normal  Started    55m   cert-manager  Challenge scheduled for processing
  Normal  Presented  55m   cert-manager  Presented challenge using HTTP-01 challenge mechanism

The cert-manager controller pod created "cm-acme-http-solver" pod+service+ingress but they might not work as expected - it seems to return 503 for a minute, then it seems to get into the "Error accepting challenge: 400" error above and the controller seems to remove the setup. The logs from a loop of the cert-manager controller:
I0111 11:32:43.082104       1 pod.go:71] cert-manager/controller/challenges/http01/ensurePod "msg"="creating HTTP01 challenge solver pod" "dnsName"="accounting-service-public.stag.aws.worksome.net" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0111 11:32:43.344491       1 pod.go:59] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="accounting-service-public.stag.aws.worksome.net" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-ng88s" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0111 11:32:43.344598       1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="accounting-service-public.stag.aws.worksome.net" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-shxmm" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0111 11:32:43.344653       1 ingress.go:90] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="accounting-service-public.stag.aws.worksome.net" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-4wrgx" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
E0111 11:32:48.404069       1 sync.go:186] cert-manager/controller/challenges "msg"="propagation check failed" "error"="wrong status code '503', expected '200'" "dnsName"="accounting-service-public.stag.aws.worksome.net" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0111 11:32:48.427112       1 pod.go:59] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="accounting-service-public.stag.aws.worksome.net" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-ng88s" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0111 11:32:48.427186       1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="accounting-service-public.stag.aws.worksome.net" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-shxmm" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0111 11:32:48.427279       1 ingress.go:90] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="accounting-service-public.stag.aws.worksome.net" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-4wrgx" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
E0111 11:32:48.440863       1 sync.go:186] cert-manager/controller/challenges "msg"="propagation check failed" "error"="wrong status code '503', expected '200'" "dnsName"="accounting-service-public.stag.aws.worksome.net" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0111 11:32:58.404531       1 pod.go:59] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="accounting-service-public.stag.aws.worksome.net" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-ng88s" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0111 11:32:58.404635       1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="accounting-service-public.stag.aws.worksome.net" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-shxmm" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0111 11:32:58.404689       1 ingress.go:90] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="accounting-service-public.stag.aws.worksome.net" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-4wrgx" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
E0111 11:32:58.423602       1 sync.go:186] cert-manager/controller/challenges "msg"="propagation check failed" "error"="wrong status code '503', expected '200'" "dnsName"="accounting-service-public.stag.aws.worksome.net" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0111 11:33:08.423918       1 pod.go:59] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="accounting-service-public.stag.aws.worksome.net" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-ng88s" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0111 11:33:08.424030       1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="accounting-service-public.stag.aws.worksome.net" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-shxmm" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0111 11:33:08.424086       1 ingress.go:90] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="accounting-service-public.stag.aws.worksome.net" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-4wrgx" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
E0111 11:33:08.433494       1 sync.go:186] cert-manager/controller/challenges "msg"="propagation check failed" "error"="wrong status code '503', expected '200'" "dnsName"="accounting-service-public.stag.aws.worksome.net" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0111 11:33:18.434851       1 pod.go:59] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="accounting-service-public.stag.aws.worksome.net" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-ng88s" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0111 11:33:18.434940       1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="accounting-service-public.stag.aws.worksome.net" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-shxmm" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0111 11:33:18.435001       1 ingress.go:90] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="accounting-service-public.stag.aws.worksome.net" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-4wrgx" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
E0111 11:33:23.444981       1 sync.go:186] cert-manager/controller/challenges "msg"="propagation check failed" "error"="wrong status code '503', expected '200'" "dnsName"="accounting-service-public.stag.aws.worksome.net" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0111 11:33:33.445392       1 pod.go:59] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="accounting-service-public.stag.aws.worksome.net" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-ng88s" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0111 11:33:33.445478       1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="accounting-service-public.stag.aws.worksome.net" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-shxmm" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0111 11:33:33.445536       1 ingress.go:90] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="accounting-service-public.stag.aws.worksome.net" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-4wrgx" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
E0111 11:33:38.452673       1 sync.go:186] cert-manager/controller/challenges "msg"="propagation check failed" "error"="wrong status code '503', expected '200'" "dnsName"="accounting-service-public.stag.aws.worksome.net" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0111 11:33:48.453060       1 pod.go:59] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="accounting-service-public.stag.aws.worksome.net" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-ng88s" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0111 11:33:48.453144       1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="accounting-service-public.stag.aws.worksome.net" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-shxmm" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0111 11:33:48.453209       1 ingress.go:90] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="accounting-service-public.stag.aws.worksome.net" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-4wrgx" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
E0111 11:33:53.462773       1 sync.go:186] cert-manager/controller/challenges "msg"="propagation check failed" "error"="wrong status code '503', expected '200'" "dnsName"="accounting-service-public.stag.aws.worksome.net" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0111 11:34:03.463584       1 pod.go:59] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="accounting-service-public.stag.aws.worksome.net" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-ng88s" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0111 11:34:03.463669       1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="accounting-service-public.stag.aws.worksome.net" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-shxmm" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0111 11:34:03.463734       1 ingress.go:90] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="accounting-service-public.stag.aws.worksome.net" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-4wrgx" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
E0111 11:34:39.004532       1 sync.go:386] cert-manager/controller/challenges/acceptChallenge "msg"="error waiting for authorization" "error"="context deadline exceeded" "dnsName"="accounting-service-public.stag.aws.worksome.net" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
E0111 11:34:39.004807       1 controller.go:163] cert-manager/controller/challenges "msg"="re-queuing item due to error processing" "error"="context deadline exceeded" "key"="default/accounting-service-tls-2bnht-2824239067-1373156968"
I0111 11:34:44.006058       1 pod.go:59] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="accounting-service-public.stag.aws.worksome.net" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-ng88s" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0111 11:34:44.006177       1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="accounting-service-public.stag.aws.worksome.net" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-shxmm" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0111 11:34:44.006238       1 ingress.go:90] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="accounting-service-public.stag.aws.worksome.net" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-4wrgx" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
E0111 11:34:59.233881       1 sync.go:378] cert-manager/controller/challenges/acceptChallenge "msg"="error accepting challenge" "error"="400 urn:ietf:params:acme:error:malformed: Unable to update challenge :: authorization must be pending" "dnsName"="accounting-service-public.stag.aws.worksome.net" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0111 11:34:59.321416       1 pod.go:119] cert-manager/controller/challenges/cleanupPods "msg"="deleting pod resource" "dnsName"="accounting-service-public.stag.aws.worksome.net" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-ng88s" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0111 11:34:59.335495       1 pod.go:127] cert-manager/controller/challenges/cleanupPods "msg"="successfully deleted pod resource" "dnsName"="accounting-service-public.stag.aws.worksome.net" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-ng88s" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
E0111 11:34:59.440884       1 controller.go:102] ingress 'default/cm-acme-http-solver-4wrgx' in work queue no longer exists
I0111 11:34:59.727133       1 trigger_controller.go:160] cert-manager/controller/certificates-trigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="default/accounting-service-tls" "retry_delay"=3459272891786
E0111 11:35:00.030063       1 sync.go:70] cert-manager/controller/orders "msg"="failed to update status" "error"=null "resource_kind"="Order" "resource_name"="accounting-service-tls-2bnht-2824239067" "resource_namespace"="default" "resource_version"="v1"
I0111 11:35:00.030144       1 controller.go:161] cert-manager/controller/orders "msg"="re-queuing item due to optimistic locking on resource" "key"="default/accounting-service-tls-2bnht-2824239067" "error"="Operation cannot be fulfilled on orders.acme.cert-manager.io \"accounting-service-tls-2bnht-2824239067\": the object has been modified; please apply your changes to the latest version and try again"

I managed to quickly describe the pod-service-ingress that the cert-manager controller creates, sometime during their short lifespan:
➜  ~ kubectl describe pod cm-acme-http-solver-jnm6f; kubectl describe ingress cm-acme-http-solver-2j9fm; kubectl describe service cm-acme-http-solver-8vzb7
Name:                 cm-acme-http-solver-jnm6f
Namespace:            default
Priority:<PHONE_NUMBER>
Priority Class Name:  system-node-critical
Node:                 <none>
Labels:               acme.cert-manager.io/http-domain=3159757414
                      acme.cert-manager.io/http-token=607653218
                      acme.cert-manager.io/http01-solver=true
                      eks.amazonaws.com/fargate-profile=platform-staging-fargate-pod-profile
Annotations:          CapacityProvisioned: 0.25vCPU 0.5GB
                      Logging: LoggingDisabled: LOGGING_CONFIGMAP_NOT_FOUND
                      kubernetes.io/psp: eks.privileged
                      sidecar.istio.io/inject: false
Status:               Pending
IP:
IPs:                  <none>
Controlled By:        Challenge/accounting-service-tls-hprbm-2824239067-3707414063
NominatedNodeName:    1c983fda60-0ef78df90139477ab992d594d1188b1b
Containers:
  acmesolver:
    Image:      quay.io/jetstack/cert-manager-acmesolver:v1.6.1
    Port:       8089/TCP
    Host Port:  0/TCP
    Args:
      --listen-port=8089
      --domain=accounting-service-public.stag.aws.worksome.net
      --token=K6SJKwJBSPDYjW5MI4ICRuHMpBN8lDwU1NLhOX2chN4
      --key=K6SJKwJBSPDYjW5MI4ICRuHMpBN8lDwU1NLhOX2chN4.-yuHWS8lDxJT_DIqqUoRVEU3PIZV3RT-ln1_oYBMf0A
    Limits:
      cpu:     100m
      memory:  64Mi
    Requests:
      cpu:        10m
      memory:     64Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rbjd6 (ro)
Volumes:
  kube-api-access-rbjd6:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason           Age   From               Message
  ----     ------           ----  ----               -------
  Warning  LoggingDisabled  63s   fargate-scheduler  Disabled logging because aws-logging configmap was not found. configmap "aws-logging" not found
Name:             cm-acme-http-solver-2j9fm
Labels:           acme.cert-manager.io/http-domain=3159757414
                  acme.cert-manager.io/http-token=607653218
                  acme.cert-manager.io/http01-solver=true
Namespace:        default
Address:          k8s-default-ingressn-5f8fde044d-bcdcaea9f98c3614.elb.eu-west-1.amazonaws.com
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host                                             Path  Backends
  ----                                             ----  --------
  accounting-service-public.stag.aws.worksome.net
                                                   /.well-known/acme-challenge/K6SJKwJBSPDYjW5MI4ICRuHMpBN8lDwU1NLhOX2chN4   cm-acme-http-solver-8vzb7:8089 (<none>)
Annotations:                                       nginx.ingress.kubernetes.io/whitelist-source-range: <IP_ADDRESS>/0,::/0
Events:
  Type    Reason  Age                From                      Message
  ----    ------  ----               ----                      -------
  Normal  Sync    50s (x2 over 64s)  nginx-ingress-controller  Scheduled for sync
Name:                     cm-acme-http-solver-8vzb7
Namespace:                default
Labels:                   acme.cert-manager.io/http-domain=3159757414
                          acme.cert-manager.io/http-token=607653218
                          acme.cert-manager.io/http01-solver=true
Annotations:              auth.istio.io/8089: NONE
Selector:                 acme.cert-manager.io/http-domain=3159757414,acme.cert-manager.io/http-token=607653218,acme.cert-manager.io/http01-solver=true
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       <IP_ADDRESS>
IPs:                      <IP_ADDRESS>
Port:                     http  8089/TCP
TargetPort:               8089/TCP
NodePort:                 http  30593/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

And I also managed to get some of the logs of this spawned solver pod in its short lifespan:
Logs from acmesolver in cm-acme-http-solver-jnm6f

I0111 12:33:58.754880       1 solver.go:39] cert-manager/acmesolver "msg"="starting listener"  "expected_domain"="accounting-service-public.stag.aws.worksome.net" "expected_key"="K6SJKwJBSPDYjW5MI4ICRuHMpBN8lDwU1NLhOX2chN4.-yuHWS8lDxJT_DIqqUoRVEU3PIZV3RT-ln1_oYBMf0A" "expected_token"="K6SJKwJBSPDYjW5MI4ICRuHMpBN8lDwU1NLhOX2chN4" "listen_port"=8089
I0111 12:34:09.444659       1 solver.go:64] cert-manager/acmesolver "msg"="validating request" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/K6SJKwJBSPDYjW5MI4ICRuHMpBN8lDwU1NLhOX2chN4" "token"="K6SJKwJBSPDYjW5MI4ICRuHMpBN8lDwU1NLhOX2chN4"
I0111 12:34:09.444701       1 solver.go:72] cert-manager/acmesolver "msg"="comparing host" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/K6SJKwJBSPDYjW5MI4ICRuHMpBN8lDwU1NLhOX2chN4" "token"="K6SJKwJBSPDYjW5MI4ICRuHMpBN8lDwU1NLhOX2chN4" "expected_host"="accounting-service-public.stag.aws.worksome.net"
I0111 12:34:09.444734       1 solver.go:79] cert-manager/acmesolver "msg"="comparing token" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/K6SJKwJBSPDYjW5MI4ICRuHMpBN8lDwU1NLhOX2chN4" "token"="K6SJKwJBSPDYjW5MI4ICRuHMpBN8lDwU1NLhOX2chN4" "expected_token"="K6SJKwJBSPDYjW5MI4ICRuHMpBN8lDwU1NLhOX2chN4"
I0111 12:34:09.444753       1 solver.go:87] cert-manager/acmesolver "msg"="got successful challenge request, writing key" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/K6SJKwJBSPDYjW5MI4ICRuHMpBN8lDwU1NLhOX2chN4" "token"="K6SJKwJBSPDYjW5MI4ICRuHMpBN8lDwU1NLhOX2chN4"

Issuer used:
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
       # The ACME server URL
    server: https://acme-staging-v02.api.letsencrypt.org/directory
       # Email address used for ACME registration
    email<EMAIL_ADDRESS>       # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-staging.issuer.private-key
       # Enable the HTTP-01 challenge provider
    solvers:
    - http01:
        ingress:
          class: nginx

Expected behaviour:
challenge is successful, resulting in "valid" order state and certificate being issued
Steps to reproduce the bug:

create an EKS cluster and set it up to work with Fargate
set-up external-dns, ingress-nginx, aws-load-balancer-controller
deploy a sample application served behind the ingress-nginx, with a HOST record configured in e.g. Route53 via external-dns
check that it can be reached via http curl http://$HOST
deploy Issuers: letsencrypt HTTP01 issuers for both staging and production
amend Ingress, add cert-manager.io/issuer: "letsencrypt-staging" annotation to the corresponding service Ingress
(expect certificate issuance and the endpoint responding successfully over https)

Anything else we need to know?:

I tried also setting acme.cert-manager.io/http01-ingress-class: nginx and acme.cert-manager.io/http01-edit-in-place: "false" annotations on the service ingress, but to no avail.
I also went through the troubleshooting info - https://cert-manager.io/docs/faq/acme/, but no luck.
curl http://$HOST feels slow at times, I wonder if it's because of the DNS (and I therefore wonder if it somehow impacts the cert-manager solver or the overall process here)
This is a detailed bug report of my input in the comments of https://github.com/jetstack/cert-manager/issues/4709

Environment details::

Kubernetes version: v1.21.2-eks-06eac09, or more specifically

➜  cert-manager git:(xxxxxxxxxxxxx) ✗ kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2021-12-16T11:33:37Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.2-eks-06eac09", GitCommit:"5f6d83fe4cb7febb5f4f4e39b3b2b64ebbbe3e97", GitTreeState:"clean", BuildDate:"2021-09-13T14:20:15Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.23) and server (1.21) exceeds the supported minor version skew of +/-1

Cloud-provider/provisioner: AWS / eks with fargate
cert-manager version: 1.6.1
Install method: installed via helm template cert-manager jetstack/cert-manager --version 1.6.1 -f values.yaml + kubectl apply of the resulting yaml (values.yaml is obtained by helm show values jetstack/cert-manager --version 1.6.1 and the only value changed in it is webhook.securePort: 10260) and of the corresponding CRDs

I can't seem to debug further, as the solver pod+service+ingress live for just a minute or so.
/kind bug

full logs of the solver pod:
➜  ~ kubectl logs -f pod/cm-acme-http-solver-zsbvb
I0111 15:33:40.757651       1 solver.go:39] cert-manager/acmesolver "msg"="starting listener"  "expected_domain"="accounting-service-public.stag.aws.worksome.net" "expected_key"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8.-yuHWS8lDxJT_DIqqUoRVEU3PIZV3RT-ln1_oYBMf0A" "expected_token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "listen_port"=8089
I0111 15:33:51.152388       1 solver.go:64] cert-manager/acmesolver "msg"="validating request" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:33:51.152428       1 solver.go:72] cert-manager/acmesolver "msg"="comparing host" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "expected_host"="accounting-service-public.stag.aws.worksome.net"
I0111 15:33:51.152547       1 solver.go:79] cert-manager/acmesolver "msg"="comparing token" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "expected_token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:33:51.152573       1 solver.go:87] cert-manager/acmesolver "msg"="got successful challenge request, writing key" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:33:53.163170       1 solver.go:64] cert-manager/acmesolver "msg"="validating request" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:33:53.163212       1 solver.go:72] cert-manager/acmesolver "msg"="comparing host" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "expected_host"="accounting-service-public.stag.aws.worksome.net"
I0111 15:33:53.163245       1 solver.go:79] cert-manager/acmesolver "msg"="comparing token" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "expected_token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:33:53.163271       1 solver.go:87] cert-manager/acmesolver "msg"="got successful challenge request, writing key" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:33:55.180318       1 solver.go:64] cert-manager/acmesolver "msg"="validating request" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:33:55.180363       1 solver.go:72] cert-manager/acmesolver "msg"="comparing host" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "expected_host"="accounting-service-public.stag.aws.worksome.net"
I0111 15:33:55.180392       1 solver.go:79] cert-manager/acmesolver "msg"="comparing token" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "expected_token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:33:55.180422       1 solver.go:87] cert-manager/acmesolver "msg"="got successful challenge request, writing key" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:34:02.192734       1 solver.go:64] cert-manager/acmesolver "msg"="validating request" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:34:02.192773       1 solver.go:72] cert-manager/acmesolver "msg"="comparing host" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "expected_host"="accounting-service-public.stag.aws.worksome.net"
I0111 15:34:02.192797       1 solver.go:79] cert-manager/acmesolver "msg"="comparing token" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "expected_token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:34:02.192837       1 solver.go:87] cert-manager/acmesolver "msg"="got successful challenge request, writing key" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:34:04.212469       1 solver.go:64] cert-manager/acmesolver "msg"="validating request" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:34:04.212507       1 solver.go:72] cert-manager/acmesolver "msg"="comparing host" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "expected_host"="accounting-service-public.stag.aws.worksome.net"
I0111 15:34:04.212529       1 solver.go:79] cert-manager/acmesolver "msg"="comparing token" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "expected_token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:34:04.212555       1 solver.go:87] cert-manager/acmesolver "msg"="got successful challenge request, writing key" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:34:06.652075       1 solver.go:64] cert-manager/acmesolver "msg"="validating request" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:34:06.652114       1 solver.go:72] cert-manager/acmesolver "msg"="comparing host" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "expected_host"="accounting-service-public.stag.aws.worksome.net"
I0111 15:34:06.652144       1 solver.go:79] cert-manager/acmesolver "msg"="comparing token" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "expected_token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:34:06.652169       1 solver.go:87] cert-manager/acmesolver "msg"="got successful challenge request, writing key" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:34:26.363561       1 solver.go:64] cert-manager/acmesolver "msg"="validating request" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:34:26.363629       1 solver.go:72] cert-manager/acmesolver "msg"="comparing host" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "expected_host"="accounting-service-public.stag.aws.worksome.net"
I0111 15:34:26.363670       1 solver.go:79] cert-manager/acmesolver "msg"="comparing token" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "expected_token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:34:26.363698       1 solver.go:87] cert-manager/acmesolver "msg"="got successful challenge request, writing key" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:34:28.378051       1 solver.go:64] cert-manager/acmesolver "msg"="validating request" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:34:28.378090       1 solver.go:72] cert-manager/acmesolver "msg"="comparing host" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "expected_host"="accounting-service-public.stag.aws.worksome.net"
I0111 15:34:28.378186       1 solver.go:79] cert-manager/acmesolver "msg"="comparing token" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "expected_token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:34:28.378214       1 solver.go:87] cert-manager/acmesolver "msg"="got successful challenge request, writing key" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:34:35.386120       1 solver.go:64] cert-manager/acmesolver "msg"="validating request" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:34:35.386157       1 solver.go:72] cert-manager/acmesolver "msg"="comparing host" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "expected_host"="accounting-service-public.stag.aws.worksome.net"
I0111 15:34:35.386180       1 solver.go:79] cert-manager/acmesolver "msg"="comparing token" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "expected_token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:34:35.386204       1 solver.go:87] cert-manager/acmesolver "msg"="got successful challenge request, writing key" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:34:42.393468       1 solver.go:64] cert-manager/acmesolver "msg"="validating request" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:34:42.393512       1 solver.go:72] cert-manager/acmesolver "msg"="comparing host" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "expected_host"="accounting-service-public.stag.aws.worksome.net"
I0111 15:34:42.393538       1 solver.go:79] cert-manager/acmesolver "msg"="comparing token" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "expected_token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:34:42.393562       1 solver.go:87] cert-manager/acmesolver "msg"="got successful challenge request, writing key" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:34:49.402319       1 solver.go:64] cert-manager/acmesolver "msg"="validating request" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:34:49.402354       1 solver.go:72] cert-manager/acmesolver "msg"="comparing host" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "expected_host"="accounting-service-public.stag.aws.worksome.net"
I0111 15:34:49.402380       1 solver.go:79] cert-manager/acmesolver "msg"="comparing token" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "expected_token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
I0111 15:34:49.402407       1 solver.go:87] cert-manager/acmesolver "msg"="got successful challenge request, writing key" "base_path"="/.well-known/acme-challenge" "host"="accounting-service-public.stag.aws.worksome.net" "path"="/.well-known/acme-challenge/7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8" "token"="7-uc4teVwb8XRBKNQ4sTUbrNJiksFSoSxYPENX9WwZ8"
Error: http: Server closed
Usage:
  acmesolver [flags]

Flags:
      --domain string     the domain name to verify
  -h, --help              help for acmesolver
      --key string        the challenge key to respond with
      --listen-port int   the port number to listen on for connections (default 8089)
      --token string      the challenge token to verify against

http: Server closed
➜  ~

public records (crt.sh) suggests that there is no certificate issued for the domain - not sure if it's meant to show only issued certs or requests as well(?)

mentioned it on letsencrypt community as well
https://community.letsencrypt.org/t/cert-manager-letsencrypt-certificate-request-fails-with-authorization-must-be-pending/169496

Thank you for the great issue description, I am still to read through the logs you posted

I can't seem to debug further, as the solver pod+service+ingress live for just a minute or so.

I have previously done this when debugging by modifying RBAC, so that cert-manager doesn't have permissions to delete pods, services, ingresses and challenges- see https://github.com/jetstack/cert-manager/issues/4676#issuecomment-1003355941 although the user said it didn't work for them, but it should be achievable by modifying RBAC.
The solver pod appears to be functioning as expected, the error at the end of its log output is it getting killed after cert-manager deleted the invalid Challenge.

public records (crt.sh) suggests that there is no certificate issued for the domain - not sure if it's meant to show only issued certs or requests as well(?) MikeMcQ on community.letsencrypt.org says it displays only issued production certs

As you say, there will be no certs on crt.sh as it does not appear that ACME was able to successfully validate the challenge
Looking at the last part of logs from controller:

E0111 11:34:39.004532       1 sync.go:386] cert-manager/controller/challenges/acceptChallenge "msg"="error waiting for authorization" "error"="context deadline exceeded" "dnsName"="accounting-service-public.stag.aws.worksome.net" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"

This is cert-manager waiting for ACME to accept the authorization, so at this point the self check must have succeeded, the challenge has been accepted with ACME and cert-manager waits for ACME to validate the challenge, but that times out presumably because the ACME request for the token that the solver pod serves fails.

E0111 11:34:59.233881       1 sync.go:378] cert-manager/controller/challenges/acceptChallenge "msg"="error accepting challenge" "error"="400 urn:ietf:params:acme:error:malformed: Unable to update challenge :: authorization must be pending" "dnsName"="accounting-service-public.stag.aws.worksome.net" "resource_kind"="Challenge" "resource_name"="accounting-service-tls-2bnht-2824239067-1373156968" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"

This is cert-manager again attempting to accept the challenge and wait for ACME to validate it, but it gets back authorization must be pending. This could actually mean that the authorization was set to some other state than 'pending' (i.e 'invalid') in ACME as a result of the previous attempt to validate the challenge failing (see the conversation on https://github.com/jetstack/cert-manager/issues/4676 for context) so I think the actual issue is the timeout from ACME that happened before.
As you suggest on the other issue, I think it is likely that the error is DNS/ingress setup related so that the ACME server query for the challenge URL is failing

I will dig some more. I assume there's no (public) way to see what ACME is up to? (logs, or something)

I will dig some more. I assume there's no (public) way to see what ACME is up to? (logs, or something)

Not directly logs as such. You can get some information about the state of authorizations, orders etc by looking at the URLs  on the cert-manager resources. You can also increase log level on cert-manager controller with --v=5 flag to controller which will, between other, make it log what calls in makes to ACME

I didn't change anything about what I previously wrote, especially since I also wrote to AWS support and wanted to keep the setup as I described, for them to look - but...
... I just got back to the problem and it seems like the cert got issues successfully sometime in between now and yesterday, at least against the staging letsencrypt.
At this point I am (was) suspecting that it might have been a DNS problem.
I am now trying it against production, but for now the problem persists. I will let it retry as it usually does every hour - and see if the same happens.
At this point I don't know what to suspect, as it's probably not a supposed DNS problem, given that just changing the issuer (from letsencrypt staging to letsencrypt production) leads to the same problem. Unless there's also some ban I got on the production server for trying previously - and the ban/ACME-state needs to timeout.

ok, now I see what the problem is
https://acme-v02.api.letsencrypt.org/acme/chall-v3/67498740770/sxoV8g
"error": {
    "type": "urn:ietf:params:acme:error:connection",
    "detail": "During secondary validation: Fetching http://accounting-service-public.stag.aws.worksome.net/.well-known/acme-challenge/uGVpLZsTxGEafUE5UQgHS7vSBRVUxgETvS6OPyh-AY0: Timeout during connect (likely firewall problem)",
    "status": 400
  },

wouldn't be surprised if the SOA configuration is at play here somehow...
However, if you ever erroneously delete a valid record and later recreate it, DNS resolvers will cache the negative response (this record doesn't exist) for a longer period. This lengthens the amount of time that your customers or users can't reach the corresponding resource, such as a web server for acme.example.com.

Unless there's also some ban I got on the production server for trying previously - and the ban/ACME-state needs to timeout.

There is rate limiting for ACME production certs, but as far as I am aware in case of any of those rate limits having been hit, cert-manager will log that error and write it to Order's status (it would be returned by ACME when a new order is being created, which cert-manager will do every hour when it's retrying to a failed issuance)

DNS resolvers will cache the negative response (this record doesn't exist) for a longer period

The challenge URL will be different for every new issuance though

I suspect a TTL on a SOA record to be the culprit, as it looks like the certificate got issued successfully on the first run of the challenge after what I calculated the TTL expire time (or at least it looks that way):
Every 5.0s: kubectl get certificaterequest,order,challenge           Worksomes-MacBook-Pro.local: Wed Jan 12 18:25:49 2022

NAME                                                                                APPROVED   DENIED   READY   ISSUER
               REQUESTOR                                    AGE
certificaterequest.cert-manager.io/accounting-service-public-production-tls-v6gjx   True                True    letsencrypt-production   system:serviceaccount:default:cert-manager   61m
certificaterequest.cert-manager.io/applebanana-prod-tls-8v42s                       True                True    letsencrypt-production   system:serviceaccount:default:cert-manager   6d1h

NAME                                                                                STATE   AGE
order.acme.cert-manager.io/accounting-service-public-production-tls-v6gjx-3211555173   valid   61m
order.acme.cert-manager.io/applebanana-prod-tls-8v42s-795040331                        valid   6d1h

However, it doesn't look like the certificate gets successfully installed in ingress-nginx, as the certificate is still the dummy one. Is there some way to force retry of certificate installing in ingress-nginx (not sure how the two controllers, cert-manager and ingress-nginx interact)?

not sure how the two controllers, cert-manager and ingress-nginx interact

cert-manager just stores the TLS cert in a Kubernetes Secret, it is then up to the ingress controller to load it (and reload if the contents have changed)
For a cert-manager certificate created via ingress annotations, it should use the ingress.spec.tls.secretName Secret, you could check if the contents have been updated with the LetsEncrypt cert

Hi
I'm facing the same problem and the logs say the pods are not staying upto the required time.
The situation is extremely dangerous because we are at the edge of expiring all our certs and suddenly these certs have stopped renewing automatically. all the challanges/orders/certs are invliad and not working.
SetUp:
We are using MetalLB at the top of the kubeadm based cluster, which assigns LB ip to ingress-nginx and then the ingress-nginx assignes these certs via cert-manager.
my cert-manager version is
NAME                    NAMESPACE       REVISION        UPDATED                                         STATUS          CHART                   APP VERSION
cert-manager            cert-manager    1               2024-05-10 17:36:30.755087379 +0000 UTC         deployed        cert-manager-1.1.5      1.14.5

all the acme-pods are showing logs like below and the certificate never gets renewed, although the authentication is done successfully.
I0510 19:08:10.535274       1 solver.go:99] "got successful challenge request, writing key" logger="cert-manager.acmesolver" host="dv.ourdomain.com" path="/.well-known/acme-challenge/4HbSMnekTl3awOwlPzQu6u2DO3KXvir8zkV65y5B_V0" base_path="/.well-known/acme-challenge" token="4HbSMnekTl3awOwlPzQu6u2DO3KXvir8zkV65y5B_V0"
Error: http: Server closed
Usage:
  acmesolver [flags]

Flags:
      --domain string     the domain name to verify
  -h, --help              help for acmesolver
      --key string        the challenge key to respond with
      --listen-port int   the port number to listen on for connections (default 8089)
      --token string      the challenge token to verify against

E0510 19:08:20.940748       1 main.go:39] "error executing command" err="http: Server closed" logger="cert-manager"
rpc error: code = NotFound desc = an error occurred when try to find container "df6dac07cc4581b7fcf39085e61ec8bbb7a2c159604b6a657c670238b77d4823": not found

list of certificates
$ k get certificate
NAME                                  READY   SECRET                                AGE
acme-crt-app3-production-secret        True    acme-crt-app3-production-secret        15h
acme-crt-app2-production-secret   True    acme-crt-app2-production-secret   15h
acme-crt-app1-production-secret     False   acme-crt-app1-production-secret     15h
acme-crt-app4-production-secret   False   acme-crt-app4-production-secret   13h

list of challenegs
$ k get challenges
NAME                                                          STATE     DOMAIN                   AGE
acme-crt-app1-production-secret-1-1307169492-1604380830        invalid   our.maindomain.com        12m
acme-crt-app2-production-secret-1-794814838-4274302761      invalid   our1.maindomain.com        12m
acme-crt-app3-production-secret-1-4028356611-1929410386   invalid   dev.somedomain.com               6h31m
acme-crt-app3-production-secret-1-4028356611-347009463    invalid   dev.someotherdomain.com            6h31m
acme-crt-app3-production-secret-1-4028356611-98421249     invalid   skinny.maindomain.com   6h31m

last challenge desciption says
  Reason:      Error accepting authorization: acme: authorization error for skinny.maindomain.com: 400 urn:ietf:params:acme:error:connection: During secondary validation: 12.1x.x.x: Fetching http://skinny.maindomain.com/.well-known/acme-challenge/wEmSReq6mzrbv4PWWk7vdVjucM9IJ7pLIUCLMD6O2-o: Timeout during connect (likely firewall problem)

list of orders
~$ k get orders
NAME                                               STATE     AGE
acme-crt-app1-production-secret-1-1307169492        invalid   14m
acme-crt-app2-production-secret-1-794814838      invalid   14m
acme-crt-app3-production-secret-1-4028356611   invalid   6h33m

 kubectl describe challenges  acme-crt-shorturl-production-secret-1-4028356611-1977896871
says like below:
Events:
  Type     Reason     Age    From                     Message
  ----     ------     ----   ----                     -------
  Normal   Started    2m14s  cert-manager-challenges  Challenge scheduled for processing
  Normal   Presented  2m14s  cert-manager-challenges  Presented challenge using HTTP-01 challenge mechanism
  Warning  Failed     105s   cert-manager-challenges  Accepting challenge authorization failed: acme: authorization error for dv.ourdomain.ink: 400 urn:ietf:params:acme:error:connection: During secondary validation: 12.1x.1xx.x: Fetching http://dv.ourdomain.ink/.well-known/acme-challenge/OohPlmaBWaSM5FRn4L33Kt-jKggMnRKhjzPQX3je3uk: Timeout during connect (likely firewall problem)

However, I have tried to fetch an HTTP only page from this service which can be accessed without any delay.
Can someone guide, do we need to make the external IP 12.1x.1.x also accessible from inside as well.
not sure what's happening except that the IT folks are enabling some  compliance and some protocols are denied as a default which were working fine earlier.

We have a similar issue, lately, where the solver pod fails way to quickly, while it is perfectly possible to curl the challenge from a remote location. It is easy to reproduce, but very hard to understand what is wrong.
We upgrade cert-manager tot the latest version (1.15.0) to no avail, whichever version we deploy the issue remains the same.
The strangest thing was we had 2 subdomains with an identical ingress object; one was provisioned correctly, the other was not.
Our setup is HAProxy -> Ingress NGINX -> Application
When I apply the ingress, the certificaterequest is created, the solver pod starts and then exits with
E0704 23:49:47.770372       1 main.go:42] "error executing command" err="http: Server closed" logger="cert-manager"

In the challenge we get
Error accepting authorization: acme: authorization error for xyz.com : 400 urn:ietf:params:acme:error:connection: xyz : Error getting validation data

In the issuer configuration we added:
    solvers:
    - http01:
        ingress:
          ingressClassName: nginx
          ingressTemplate:
            metadata:
              annotations:
                nginx.ingress.kubernetes.io/ssl-redirect: "false"

But the problem remains the same :(


Adjust meditation value to have a better impact in test

User story
As a tester tests car, I want to be able to see the car significantly accelerated or decelerated.
Acceptable criteria
As meditation value drops quickly for tester, it's harder to keep the car accelerating in a short distance.

The meditation value on headset changes randomly and it works very bad when doing test.


update dependencies on 6.4.x

I want to update spring-boot, mainly b/c some CVEs for netty are showing up when I build CAS with latest 6.4.x and those dependencies come from spring boot. I mainly updated spring dependencies but I updated a few others without a good reason other than they seemed fairly safe.

This is not something we can merge. The boot version does affect overlays and could potentially break, and will require everyone to upgrade and know about the change. That is a no go. Other dependencies are also frozen in time, and will only change if there is good serious reason to do. scanners picking up CVEs for things that are irrelevant do not qualify.

Some notes here: https://apereo.github.io/cas/developer/Sec-Vuln-Response.html#dependency-upgrades


shadow on top

by adding elevation, and stroke on "top" drawable , can u add better solution for shadow?

+1


Document refresh max jitter

Refresh API option refresh_jitter_max is not documented. It should be documented.

Documentation is available, added PR to enhance
https://docs.spiceai.org/components/data-accelerators/data-refresh#refresh-jitter

https://docs.spiceai.org/reference/spicepod/datasets#accelerationrefresh_jitter_max


Type 'string' is not assignable to type 'never'

Bug description
For some reason i'm getting the Type 'string' is not assignable to type 'never' error, but the variables are with same tipe (string)
How to reproduce
prisma.< database >.<create || update>({ data:  })
Prisma information

generator client {
  provider = "prisma-client-js"
}

datasource db {
  provider = "mysql"
  url      = env("DATABASE_URL")
}

model Users {
  id           String         @id @unique
  firstName    String
  lastName     String
  email        String         @unique
  password     String
  number       Int
  curriculo    String?
  createdAt    DateTime       @default(now())
  updatedAt    DateTime       @updatedAt
  anuncios     Anuncios[]
  candidaturas Candidaturas[]
}

model Anuncios {
  id           String         @id @unique
  ownerID      String
  companyName  String
  local        String
  description  String         @db.VarChar(500)
  salary       String?
  role         String
  typeJob      String // Remoto, presencial ou híbrido
  requirements String?        @db.VarChar(500)
  contractType String
  createdAt    DateTime       @default(now())
  updatedAt    DateTime       @updatedAt
  owner        Users          @relation(fields: [ownerID], references: [id])
  candidaturas Candidaturas[]
}

model Candidaturas {
  id         String   @id @unique
  ownerID    String
  anuncioID  String
  status     Int      @default(0) //0 - nenhuma atualização, 1 - aceito, 2 - recusado, 3 - Em analise
  createdAt  DateTime @default(now())
  updatedAt  DateTime @updatedAt
  owner      Users    @relation(fields: [ownerID], references: [id])
  anuncio    Anuncios @relation(fields: [anuncioID], references: [id])
}

async create(candidatura: Candidaturas): Promise<Candidaturas> {
    return this.prisma.candidaturas.create({ data: candidatura });
  }

  Candidaturas.ts:
import { v4 as uuid } from 'uuid';

import Users from './Users';
import Anuncios from './Anuncios';

class Candidaturas {
  public readonly id: string;

  public readonly ownerID: string;

  public readonly anuncioID: string;

  public status?: number;

  public createdAt: Date;

  public updatedAt: Date;

  public readonly owner?: Users;

  public readonly anuncio?: Anuncios;

  constructor(props: Omit<Candidaturas, 'id' | 'createdAt' | 'updatedAt'>, id?: string) {
    Object.assign(this, props);

    if (!id) {
      this.id = uuid();
    }
  }
}

export default Candidaturas;

Environment & setup

OS: Windows11
Database: mysql
Node.js version: v16.16.0

Prisma Version
prisma                  : 4.5.0
@prisma/client          : 4.5.0
Current platform        : windows
Query Engine (Node-API) : libquery-engine 0362da9eebca54d94c8ef5edd3b2e90af99ba452 (at<EMAIL_ADDRESS>Migration Engine        : migration-engine-cli 0362da9eebca54d94c8ef5edd3b2e90af99ba452 (at<EMAIL_ADDRESS>Introspection Engine    : introspection-core 0362da9eebca54d94c8ef5edd3b2e90af99ba452 (at<EMAIL_ADDRESS>Format Binary           : prisma-fmt 0362da9eebca54d94c8ef5edd3b2e90af99ba452 (at<EMAIL_ADDRESS>Format Wasm             : @prisma/prisma-fmt-wasm 4.5.0-43.0362da9eebca54d94c8ef5edd3b2e90af99ba452
Default Engines Hash    : 0362da9eebca54d94c8ef5edd3b2e90af99ba452
Studio                  : 0.476.0

@Jolg42 any soluction ? Or you need more infos ?

Someone would need to reproduce first, so if you can give us more information, that would help.
For which field do you get Type 'string' is not assignable to type 'never'? Is there any extra info about this error? For example, where it comes from?
Could you give us an example of the data passed in prisma.candidaturas.create({ data: candidatura }); (candidatura)

The camp with Type 'string' is not assignable to type 'never' is the ownerID, and the type of candidatura is the class of the first message of this issues, and sorry for the delay to response

THanks @DiogoMarques2003
It would help if you can give us more details, like a full example of the query that error (with the data, feel free to replace sensitive information of course)

prisma.< database >.<create || update>({ data: })
Also does this happen for both create and update?

Note: Issue with same error https://github.com/prisma/prisma/issues/8797

THanks @DiogoMarques2003
It would help if you can give us more details, like a full example of the query that error (with the data, feel free to replace sensitive information of course)

prisma.< database >.<create || update>({ data: })
Also does this happen for both create and update?

Note: Issue with same error #8797

the example of query is this:
async create(candidatura: Candidaturas): Promise<Candidaturas> {
    return this.prisma.candidaturas.create({ data: candidatura });
  }

@Jolg42 any update ?

@DiogoMarques2003 It would help if you can provide more information like:

a full example of the query that errors, with the data, (feel free to replace sensitive information of course)
In your previous comment, you didn't provide an example of candidatura, a full example would help, so we can try a reproduction using your schema and some TypeScript code running a query.

@Jolg42 on this issues you already have the all info, you have the candidatura class file on the first message.
And you have the example of request on my last message.

@DiogoMarques2003 We do have the class file, but what I'm asking for is a full example with example data.
The class only gives us the fields and their type, they do not help to figure out what the data actually looks like, and this would be helpful, so we can easily run a reproduction.

@Jolg42 this ? or you need in other format?

@Jolg42 any update ?

We will look into this when we have time. We do not have an ETA for when this might be the case, but it looks like the information you provided should be enough.
ALthough: Can you maybe paste the values of your table? Otherwise, we will need to type them by hand and that might go wrong.

@janpio the data of table is:
id, status, createdAt, updatedAt, ownerID, anuncioID
'dc667d1b-796a-4821-b2ef-7581326e2edf', '0', '2022-11-19 17:09:32.933', '2022-11-19 17:09:32.933', '548095e2-bd83-4028-89aa-e6f016031e53', 'ae7cb90d-db65-460c-bcd3-fde0bbef9a7c'

@janpio any update ?

No. We have the data now we need for someone to take a look.

I'm facing a similar issue with my crud app, the error is throwing with the following function:

Function
async createWithSubOperations(userId: number, dto: CreateOperationDto) {
      const subOperations = this.nestedCreateMany(userId, dto);
   	  const status = this.setOperationStatus(subOperations);

      return await this.prisma.$transaction(async (tx) => {
         const operation = await tx.operation.create({
            data: {
               userId,
               children: { createMany: { data: subOperations } },
               status
            },
            include: {
               packages: true,
               children: true,
            },
         });

		// more code...

         return operation;

      })
}

This function throws this error:
Type '{ userId: number; children: { createMany: { data: CreateSubOperationDto[]; }; }; status: string; }' is not assignable to type 'Without<OperationUncheckedCreateInput, OperationCreateInput>'.
Types of property 'userId' are incompatible.
Type 'number' is not assignable to type 'never'.ts(2322)
index.d.ts(2959, 5): The expected type comes from property 'data' which is declared here on type '{ select?: OperationSelect; include?: OperationInclude; data: (Without<OperationCreateInput, OperationUncheckedCreateInput> & OperationUncheckedCreateInput) | (Without<...> & OperationCreateInput); }'
and when I access my the index.d.ts to check the type it has the following type:

I ran the npx prisma generate to check if my client wasn't outdated with my schema.prisma, but didn't work.
I've already had this problem, with the same relationated userId field, and I just could solve it recreating my github codespace environment, where I write my code.
And I don't know why but I have another function, in the same file, that do the same thing, and don't throws the error.

Function

   async create(userId: number, dto: CreateOperationDto) {
      if (dto.value > maxOperationValue) return this.createWithSubOperations(userId, dto)

      const packages = this.packageService.nestedCreateMany({ ...dto });

      const status = this.setOperationStatus(packages);

      return await this.prisma.operation.create({
         data: {
            userId,
            ...dto,
            packages: { createMany : { data: packages } },
            status
         },
         include: {
            packages: true,
            children: true,
         },
      });
   }

this doesn't throw error.

any update @janpio ?
If you need the source code with this problem is one this repository: https://github.com/DiogoMarques2003/portugal-jobs/tree/main/Backend

@janpio  any update

same issue -- with upsert

If you made changes to your schema run "npx prisma generate"

Had the exact issue! Resolved it by running the following
npm uninstall prisma @prisma-client
npm install -d prisma
npm install @prisma/client

While I was getting this error I was receiving a warning while generating the client that it was unable to install @prisma/client and requested me to run
npm i @prisma/client

I did so and ran the generate command again but it showed me the same warning so I choose to remove and reinstall the dependencies.
Hope this helps!

Got rid for the error by removing the tsconfig.json file.
Not really a solution !!
Can someone look into it ?

Managed to solve it
Just add "strict": true, to your tsconfig.json as below
{
     "compilerOptions": {
         "strict": true,
         ...
     },
     ...
}

Managed to solve it Just add "strict": true, to your tsconfig.json as below
{
     "compilerOptions": {
         "strict": true,
         ...
     },
     ...
}

This does not resolve the error in @DiogoMarques2003's project. It simply changes the error message from:
Type 'Candidaturas' is not assignable to type '(Without<CandidaturasCreateInput, CandidaturasUncheckedCreateInput> & CandidaturasUncheckedCreateInput) | (Without<...> & CandidaturasCreateInput)'.
  Type 'Candidaturas' is not assignable to type 'Without<CandidaturasUncheckedCreateInput, CandidaturasCreateInput> & CandidaturasCreateInput'.
    Type 'Candidaturas' is not assignable to type 'Without<CandidaturasUncheckedCreateInput, CandidaturasCreateInput>'.
      Types of property 'ownerID' are incompatible.
        Type 'string' is not assignable to type 'never'.ts(2322)

to
Type 'Candidaturas' is not assignable to type '(Without<CandidaturasCreateInput, CandidaturasUncheckedCreateInput> & CandidaturasUncheckedCreateInput) | (Without<...> & CandidaturasCreateInput)'.
  Type 'Candidaturas' is not assignable to type 'Without<CandidaturasUncheckedCreateInput, CandidaturasCreateInput> & CandidaturasCreateInput'.
    Type 'Candidaturas' is not assignable to type 'Without<CandidaturasUncheckedCreateInput, CandidaturasCreateInput>'.
      Types of property 'ownerID' are incompatible.
        Type 'string' is not assignable to type 'undefined'.ts(2322)

I'm afraid I am also in the same boat, I have a model such as:
model Answer {
  id String @id @default(cuid())
  question Question @relation(fields: [questionId], references: [id], onDelete: Cascade)
  questionId String
}

but then trying to create a record in this model I like:
create(questionId: string, input: AnswerInput) {
  return this.prisma.answer.create({
     data: {
      ...input,
      questionId,
    },
  });
}

My create function complains about Type 'string' is not assignable to type 'never', I have multiple other models that follow the same pattern but this is the only model where this problem exits, also fwiw is the final model in my schema not sure if that's somehow related.
I am running a nestjs with latest prisma, is there something I can provide to demistify this? I did try adding strict to tsconfig but that did not change anything, the tsconfig file provided by default by nest has already some configs that may be in conflict I'm not sure.

@borjapazr I just managed to resolve this with the help of someone in the nestjs discord, the problem was a little tricky to figure out as I never thoguht to check the generated types of converting my graphql SDL into typescript, it turns out that while my input type was written like:
input AnswerInput {
  text: String!
  correct: Boolean = False
}

I was expecting this to be converted to:
export class AnswerInput {
  text: string;
  correct: boolean | false;
}

it was in reality converted to:
export class AnswerInput {
  text: string;
  correct?: Nullable<boolean>;
}

The combination of which with questionId: string made TypeScript complain, the fix for me was to change my service method to:
create(questionId: string, input: AnswerInput) {
  const { text, correct = false } = input;

  return this.prisma.answer.create({
    data: {
      correct,
      text,
      questionId,
    },
  });
}

If it wasn't for @xxninjabunnyxx (hope I tagged the correct person) at discord that actually tried by inlining the type I would have never tought to check the generated type for AnswerInput I hope the issue you're having is this simple to fix as well, cheers.

@EnriqueVidal , can you explain what you have changed? I didn't understand the difference since you haven't changed the questioned definition.

I have this module:
model StockOperation {
  id        Int      @id @default(autoincrement())
  createdAt DateTime @default(now())
  updatedAt DateTime @updatedAt

  stockCode     String
  quantity      Int
  unitValue     Float
  operationType OperationType @default(BUY)
  operationDate DateTime

  userId Int
  user   User @relation(fields: [userId], references: [id])

  @@map("stockOperations")
}

and this service:
async createStokeOperation(userId: number, dto: CreateStokeOperation) {
    const operation = await this.prisma.stockOperation.create({
      data: {
        userId,
        ...dto,
      },
    });
    return operation;
  }

I am having a similar problem to the ones described above:
Types of property 'userId' are incompatible. Type 'number' is not assignable to type 'never'.
However, userId was defined as a number (INT) on the schema.

Yes, you can consider it fixed. I will add the explanation here as soon as I have some free time. Thanks, @EnriqueVidal

Yes, you can consider it fixed. I will add the explanation here as soon as I have some free time. Thanks, @EnriqueVidal

Great 👏 It would be great to know what your solution is. Thanks to both @heitortessaro and @EnriqueVidal 🙏

i have the  issue did any find any solution

@borjapazr and @DiogoMarques2003 I just managed to resolve this with the help of someone in the nestjs discord, the problem was a little tricky to figure out as I never thoguht to check the generated types of converting my graphql SDL into typescript, it turns out that while my input type was written like:
input AnswerInput {
  text: String!
  correct: Boolean = False
}

I was expecting this to be converted to:
export class AnswerInput {
  text: string;
  correct: boolean | false;
}

it was in reality converted to:
export class AnswerInput {
  text: string;
  correct?: Nullable<boolean>;
}

The combination of which with questionId: string made TypeScript complain, the fix for me was to change my service method to:
create(questionId: string, input: AnswerInput) {
  const { text, correct = false } = input;

  return this.prisma.answer.create({
    data: {
      correct,
      text,
      questionId,
    },
  });
}

If it wasn't for @xxninjabunnyxx (hope I tagged the correct person) at discord that actually tried by inlining the type I would have never tought to check the generated type for AnswerInput I hope the issue you're having is this simple to fix as well, cheers.

@TalebBahan maybe this go help you, I didn't try but for the other people this work

Hello everyone, I am facing the same problem but with different data
Model:
model Job {
  id Int @id @default(autoincrement())
  type JobType
  createdAt DateTime @default(now())
  date DateTime
  price Float
  priceType PriceType @default(WHOLE)
  currency Currency @default(RSD)
  userId Int
  user   User @relation(fields: [userId], references: [id])

  @@map("jobs")
}

DTO:
export class CreateJobDto {
  @IsString()
  @IsNotEmpty()
  name: string;

  @IsEnum(JobType)
  @IsNotEmpty()
  type: string;

  @IsDate()
  @IsNotEmpty()
  date: Date;

  @IsNotEmpty()
  @IsNumber()
  price: number;

  @IsEnum(Currency)
  @IsNotEmpty()
  currency: string;

  @IsNotEmpty()
  @IsEnum(PriceType)
  priceType: string;
}

Where the error is happening:
 async createJob(userId: number, jobDto: CreateJobDto) {
    const job = await this.prisma.job.create({
      data: { ...jobDto, userId},
    });
  }

It is not letting me assign userId because it says that the types are incompatible, more accurately:
Types of property 'userId' are incompatible.
        Type 'number' is not assignable to type 'never'

Hello,
For my part, when I got this error message, the problem was actually that I forgot to make a Datetime field optional in my model.
When I tried to use the create function without this field, I got this wrong error pointing to another field instead of the Datetime one :
Types of property 'user_id' are incompatible.         Type 'number' is not assignable to type 'never'
Hope it will help a few people !

Perhaps it's due to the editor's TS cache, try clearing the TS cache

It's possible that some or most of these issues occur because of TypeScript and is not an issue in Prisma. For example, the Prisma schema may contain fields that are required, because they are not specified as optional (especially relations). While it intuitively makes sense that we're always passing data to these fields, class-validator may not always be properly implemented in a DTO class to guarantee the existence of those fields.
In certain cases, you also need the help of class-transformer (e.g. plainToClass()) to convince TypeScript that your input data is compatible with types generated by Prisma.

for me, I have added strict:true in the tsconfig.json under the compilation option then deleted and recreate the primsa model in which the error occurred and it has worked

Same problem, I was missing a required field, but the error pointed to a different one.
await prisma.minuteBars.upsert({
                where: {
                    symbol_symbolType_date: {
                        symbol: symbol,
                        symbolType: type,
                        date: new Date(bar.t)
                    }
                },
                create: {
                    close: bar.c,
                    open: bar.o,
                    high: bar.h,
                    low: bar.l,
                    volume: bar.v,
                    symbol: symbol, // error was saying that this field was "not assignable to type never", but this was right
                    symbolType: type,
                    date: new Date(bar.t), // I was missing this field which is not optional
                },
                update: {}
            })
           ```

@veljkocukic how you fixed that?


Add School to ma/ac

@scorpionTaj I am sorry, the domain usms.ac.ma is removed from our trusted domains list because it was abused in the past. If you are the student of this school, please enroll to GitHub Student Developer Program. Then apply for JetBrains educational subscription with your GitHub account (GitHub tab at https://www.jetbrains.com/shop/eform/students).


Secure environment variables on docsv2 branch need to be regenerated in the travis.yml

The travis build task now errors out with:
remote: Permission to k8sdocs/k8sdocs.github.io.git denied to johndmulhausen.
fatal: unable to access<EMAIL_ADDRESS>The requested URL returned error: 403
The command "git push" exited with 128.
Done. Your build exited with 1.

The .travis.yml file on docsv2 contains some secure environment variables that need to be regenerated. The method is explained here: https://docs.travis-ci.com/user/environment-variables/#Encrypted-Variables
I think this time, it would be good to tie them to a bot account.
cc @pwittrock

@jeffmendoza @jaredbhatti Is this issue fixed with the docsv2 branch?

Looks like it is. Moving to close this.


How does this differ from the time played value inside the stat screen?

Don't get me wrong, I like the idea and also congrats on the first mod. Keep it up!
As it currently stands though, this mod seems... redundant.
A suggestion though, if you are willing to hear:

Maybe use that data to show total time played within an instance?

All servers and single player are added together and showed on home-screen

For sure, i would love the suggestions, add-ons are being made for admins to be able to track playtime for each player, and afk time, also about the playtime in an instance -- awesome idea, I'll surely look into it, thank you

Sounds cool man! Look forward to seeing it


Add Google Maps support for showing satellite reports

All satellite reports with a grid square included can now be seen on a map.

Need to get all CI builds passing before this can be merged.


[Unification] Refactor ALBEF multimodal encoder with MultiHeadAttention

Stack from ghstack (oldest at bottom):

#191
-> #190
#189
#170
#169
#168

Codecov Report

:exclamation: No coverage uploaded for pull request base (gh/RdoubleA/17/base@4d1a9ca). Click here to learn what that means.
The diff coverage is n/a.

@@                  Coverage Diff                   @@
##             gh/RdoubleA/17/base     #190   +/-   ##
======================================================
  Coverage                       ?   92.02%
======================================================
  Files                          ?       47
  Lines                          ?     2796
  Branches                       ?        0
======================================================
  Hits                           ?     2573
  Misses                         ?      223
  Partials                       ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4d1a9ca...d1e2028. Read the comment docs.

@RdoubleA has imported this pull request.  If you are a Facebook employee, you can view this diff on Phabricator.


Add support for trusted types to lit-next's test system

In order to add Trusted Types support to lit next we need to be able to test it, and it looks like our current test setup has issues.
We're using @web/test-runner with @web/test-runner-mocha, which depends on mocha, which does some .innerHTML assignment. I sent over https://github.com/mochajs/mocha/pull/4447 to fix.
Patching that in, I think that's all that we need here. If that change doesn't get reviewed in a timely fashion though, we can look into alternatives like jasmine.

If the amount of tests you need to write for this are somewhat manageable, you could also write tests without a test framework. You just need to set up some boilerplate to ping back results to the test runner.
You can write them in JS, but that requires running a separate test runner instance. If you write a HTML test, you can set this up within the current test runner. See an example here at the bottom of the page.

You may also be able to bypass the this issue by not using the HTML reporter in mocha.

Ideally we'd run all of our tests both with and without trusted types enabled.
Running without the HTML reporter would work I bet. The only other place that needed changing was some bit of code highlighting.

@rictic how is this looking? Looks like you got a ping on https://github.com/mochajs/mocha/pull/4447

@rictic can we close this out?


test: add end-to-end testing setup

Fixes #35

ok @maninak this is now in a good place.
I'll leave the one failing lint check to you.
The endToEnd tests:

install radicle
create a new radicle profile
create a new radicle repo
open that radicle repo with VS Code

at that point, you can write your tests to start verifying whatever scenarios you feel are the most important. I'd recommend just starting with 1 scenario.
For example that could be, listing the patches and opening a patch (that you have previously created in the radicle-init step), and verifying you can view the patch.
As you will also see in the commit message, I have followed the guide in: https://code.visualstudio.com/api/working-with-extensions/testing-extension#the-test-runner-script .
If it helps, I'm happy to go over these changes together as well once you're back.
🖖🏼

Closing in favor of #158


React controlled Dropdowns

First, this rearranges the Dropdown internals to hopefully make them a bit clearer, then it adds the various bits and pieces to make our Dropdowns React powered rather than bootstrap/jQuery.
For dropdowns created via the Dropdown props, this uses react-popper for positioning of DropdownMenus near their DropdownTrigger and state built into the Dropdown for handling showing/hiding of the menu.
Resolves #151

@AaronCoplan have made those amends.
On react-popper, does quite a lot for us under the hood, it handles the positioning of the Dropdown menu, takes into account scrolling, viewport, containing elements etc and re-adjusts the positioning to make sure it's where we want it and visible.

Awesome! Good stuff.  I was just wondering and didn't have any context - thats great!