Upgrading from 1.13 to 1.14 leaves triggers in broken state

Describe the bug
Upgrading from knative=1.13.8 and kafka-extensions=1.13.11.
knative-eventing related resources update and restart as expected.
eventing-kafka extension related resource upgrade fails:

kafka-broker-dispatcher statefulset set starts at 3 replicas and eventually scales up to 15 replicas in our dev environment. Only 3/15 replicas end up with data populated to the corresponding kafka-broker-dispatcher-X configmap resource. I see a continued process of the pods going through container creating state as it waits for the CM to get created and then throws an error when it finally starts up

{"@timestamp":"2024-08-30T21:12:19.847Z","@version":"1","message":"failed to parse from JSON","logger_name":"dev.knative.eventing.kafka.broker.core.eventbus.ContractPublisher","thread_name":"contract-file-watcher","level":"ERROR","level_value":40000,"stack_trace":"com.google.protobuf.InvalidProtocolBufferException: Expect message object but got: null\n\tat com.google.protobuf.util.JsonFormat$ParserImpl.mergeMessage(JsonFormat.java:1481)\n\tat com.google.protobuf.util.JsonFormat$ParserImpl.merge(JsonFormat.java:1458)\n\tat com.google.protobuf.util.JsonFormat$ParserImpl.merge(JsonFormat.java:1322)\n\tat com.google.protobuf.util.JsonFormat$Parser.merge(JsonFormat.java:486)\n\tat dev.knative.eventing.kafka.broker.core.eventbus.ContractPublisher.parseFromJson(ContractPublisher.java:98)\n\tat dev.knative.eventing.kafka.broker.core.eventbus.ContractPublisher.updateContract(ContractPublisher.java:74)\n\tat dev.knative.eventing.kafka.broker.dispatcher.main.Main.lambda$start$0(Main.java:137)\n\tat dev.knative.eventing.kafka.broker.core.file.FileWatcher.run(FileWatcher.java:124)\n\tat java.base/java.lang.Thread.run(Thread.java:1595)\n"}

The kafka-controller-post-install job completes and outputs no logs.

If (2) is replaced with a manual step to cleanup the old dispatcher deployment, all event traffic ceases and triggers go into a failed state:
dev020-ke.a.a.v1.training-complete               knative-wa-a-broker                     107d   False   Schedule
dev020-ke.a.a.v1.training-failed                 knative-wa-a-broker                     107d   False   Schedule
dev020-ke.a.a.v2.training-start                  knative-wa-a-broker                     462d   False   Schedule
dev020-ke.a.c-c.v1.training-complete          knative-wa-c-broker                           386d   False   Schedule
dev020-ke.a.c-c.v1.training-failed            knative-wa-c-broker                           386d   False   PropagateSubscriberURI

Recreation of any trigger does not help. Existing broker resources continue to report ready but if any are recreated will also go into a failed state. A describe on any trigger shows a failure on old/no-longer-existing dispatcher deployment
  - lastTransitionTime: "2024-08-30T20:23:29Z"
    message: 'failed to schedule consumers: failed to get data plane volume kafka-resources
      in pod knative-eventing/kafka-broker-dispatcher-6b6855d6c5-7nhtw'
    reason: Schedule
    status: "False"
    type: Ready

Expected behavior
To Reproduce
Steps to reproduce the behavior.
Knative release version
knative=1.13.8 and kafka-extensions=1.13.11
Additional context
Add any other context about the problem here such as proposed priority

Hi @mdwhitley, thanks for reporting, can you describe what is present in your knative-eventing namespace (deployments, pods, statefulsets)?
Did you run the post-install job?
How many Kafka brokers and triggers do you have in your cluster?

Our 1.13 environment knative-eventing namespace
pods:
NAME                                          READY   STATUS    RESTARTS   AGE
eventing-controller-785f6cfbcf-9jzb5          1/1     Running   0          9d
eventing-controller-785f6cfbcf-nnn8j          1/1     Running   0          10d
eventing-controller-785f6cfbcf-nxb4c          1/1     Running   0          19h
eventing-webhook-7bc98f9577-6v6z2             1/1     Running   0          5d22h
eventing-webhook-7bc98f9577-w9xkp             1/1     Running   0          9d
kafka-broker-dispatcher-6b6855d6c5-ccxdx      1/1     Running   0          10d
kafka-broker-dispatcher-6b6855d6c5-lxhhd      1/1     Running   0          3d14h
kafka-broker-receiver-5bccbdbdcf-2zggd        1/1     Running   0          10d
kafka-broker-receiver-5bccbdbdcf-r8g4c        1/1     Running   0          9d
kafka-controller-5674695d4-bw9b8              1/1     Running   0          10d
kafka-controller-5674695d4-jrgkr              1/1     Running   0          5d8h
kafka-controller-5674695d4-ww2zx              1/1     Running   0          10d
kafka-webhook-eventing-7544bd5884-8gncc       1/1     Running   0          10d
kafka-webhook-eventing-7544bd5884-hg4ll       1/1     Running   0          10h
kafka-webhook-eventing-7544bd5884-t92st       1/1     Running   0          10d
ke-sample-api-service-9d5f7c7d9-drpnd         1/1     Running   0          13d
ke-sample-training-service-5bcc4bc485-9wgrx   1/1     Running   0          13d
knative-sandbox-dlq-6fb6bc87f8-qnl2k          1/1     Running   0          13d

deployments:
NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
event-display-mock-clu-training   0/0     0            0           601d
eventing-controller               3/3     3            3           653d
eventing-webhook                  2/2     2            2           653d
kafka-broker-dispatcher           2/2     2            2           10d
kafka-broker-receiver             2/2     2            2           653d
kafka-controller                  3/3     3            3           653d
kafka-webhook-eventing            3/3     3            3           653d
ke-sample-api-service             1/1     1            1           652d
ke-sample-training-service        1/1     1            1           652d
knative-sandbox-dlq               1/1     1            1           652d
pingsource-mt-adapter             0/0     0            0           586d

No statefulsets
3 brokers, 266 triggers (prod will have 2-3x as many triggers in large clusters)

I mean during/after the upgrade process

Environment after 1.14 upgrade. This runs all knative templates+post-install followed by eventing-kafka templates+post-install.
pods:
NAME                                           READY   STATUS      RESTARTS   AGE
eventing-controller-575bc94c49-75rx5           1/1     Running     0          3m34s
eventing-controller-575bc94c49-c7x5q           1/1     Running     0          4m36s
eventing-controller-575bc94c49-ckdlx           1/1     Running     0          4m5s
eventing-webhook-65b4948885-4xrgf              1/1     Running     0          4m29s
eventing-webhook-65b4948885-6wx8q              1/1     Running     0          4m36s
kafka-broker-dispatcher-0                      1/1     Running     0          2m47s
kafka-broker-dispatcher-1                      1/1     Running     0          2m47s
kafka-broker-dispatcher-6b6855d6c5-ccxdx       1/1     Running     0          10d
kafka-broker-dispatcher-6b6855d6c5-lxhhd       1/1     Running     0          3d14h
kafka-broker-receiver-54d9dcc8cd-9qv78         1/1     Running     0          2m13s
kafka-broker-receiver-54d9dcc8cd-m9bdl         1/1     Running     0          2m47s
kafka-controller-6f889fcc6b-crjrh              1/1     Running     0          2m38s
kafka-controller-6f889fcc6b-lf55k              1/1     Running     0          2m43s
kafka-controller-6f889fcc6b-r62wv              1/1     Running     0          2m48s
kafka-controller-post-install-fdxkc            0/1     Completed   0          2m47s
kafka-webhook-eventing-8686d549db-78m6x        1/1     Running     0          2m36s
kafka-webhook-eventing-8686d549db-9ztlk        1/1     Running     0          2m41s
kafka-webhook-eventing-8686d549db-zhn6x        1/1     Running     0          2m47s
ke-sample-api-service-9d5f7c7d9-drpnd          1/1     Running     0          13d
ke-sample-training-service-5bcc4bc485-9wgrx    1/1     Running     0          13d
knative-kafka-storage-version-migrator-62cxr   0/1     Completed   0          2m47s
knative-sandbox-dlq-6fb6bc87f8-qnl2k           1/1     Running     0          13d
storage-version-migration-eventing-9t9pv       0/1     Completed   0          4m34s

Note controlplane post-install job completed successfully prior to the 2 new dispatcher pods becoming available.
Deployments:
NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
event-display-mock-clu-training   0/0     0            0           601d
eventing-controller               3/3     3            3           653d
eventing-webhook                  2/2     2            2           653d
kafka-broker-dispatcher           2/2     2            2           10d
kafka-broker-receiver             2/2     2            2           653d
kafka-controller                  3/3     3            3           653d
kafka-webhook-eventing            3/3     3            3           653d
ke-sample-api-service             1/1     1            1           652d
ke-sample-training-service        1/1     1            1           652d
knative-sandbox-dlq               1/1     1            1           652d
pingsource-mt-adapter             0/0     0            0           586d

replicasets:
NAME                      READY   AGE
kafka-broker-dispatcher   2/2     4m30s

After completion all 266 triggers have gone unavailable and eventing cluster is down.

Post-install job is supposed to delete the old dispatcher deployment, a manual deletion should help
kubectl delete deployments -n knative-eventing -l=app.kubernetes.io/component=kafka-broker-dispatcher

I've been manually deleting the deployments in previous attempts as well.
After deleting deployment:
NAME                                           READY   STATUS      RESTARTS   AGE
eventing-controller-76f978475-ctb67            1/1     Running     0          100s
eventing-controller-76f978475-l8p98            1/1     Running     0          39s
eventing-controller-76f978475-wlpcv            1/1     Running     0          70s
eventing-webhook-79f898797-ltzwh               1/1     Running     0          100s
eventing-webhook-79f898797-rtw4r               1/1     Running     0          90s
kafka-broker-dispatcher-0                      1/1     Running     0          5m54s
kafka-broker-dispatcher-1                      1/1     Running     0          5m54s
kafka-broker-receiver-5c6f978479-brnmj         1/1     Running     0          100s
kafka-broker-receiver-5c6f978479-kk7gj         1/1     Running     0          60s
kafka-controller-77b6997bc8-29wnn              1/1     Running     0          88s
kafka-controller-77b6997bc8-m26tt              1/1     Running     0          91s
kafka-controller-77b6997bc8-pgwxx              1/1     Running     0          100s
kafka-controller-post-install-6dtnw            0/1     Completed   0          5m53s
kafka-webhook-eventing-5f55558b96-jlllw        1/1     Running     0          87s
kafka-webhook-eventing-5f55558b96-rfhwg        1/1     Running     0          91s
kafka-webhook-eventing-5f55558b96-rmtpf        1/1     Running     0          100s
ke-sample-api-service-ffd8dc566-7lgxh          1/1     Running     0          100s
ke-sample-training-service-67fc74d7bd-r5pxk    1/1     Running     0          100s
knative-kafka-storage-version-migrator-9d6mn   0/1     Completed   0          5m53s
knative-sandbox-dlq-cc6fd94c5-7sfg7            1/1     Running     0          99s
storage-version-migration-eventing-nj2tn       0/1     Completed   0          7m34s

Eventing cluster is still completely down. This is the 2nd attempt today and now triggers are all failed with
  - lastTransitionTime: "2024-09-10T16:33:09Z"
    message: "failed to initialize consumer group offset: cannot obtain Kafka cluster
      admin, kafka: client has run out of available brokers to talk to: 3 errors occurred:\n\t*
      EOF\n\t* EOF\n\t* EOF\n"
    reason: InitializeOffset
    status: "False"
    type: DependencyReady

I also observe on the kafka side a large chunk  of consumers (presumably knative) have gone inactive.

\n\t* EOF\n\t* EOF\n\t* EOF\n

That is an error we get from the Kafka cluster,
    message: 'failed to schedule consumers: insufficient running pods replicas for
      StatefulSet knative-eventing/kafka-broker-dispatcher to schedule resource replicas
      (left: 1): retry requeue after: 5s'

the autoscaler should scale up the statefulset replicas, are you setting a specific replicas number for the statefulset kafka-broker-dispatcher?

While pulling additional logs there was a burst of activity scaling up the statefulset. I had observed this happen previously as well:
NAME                                          READY   STATUS    RESTARTS   AGE
eventing-controller-76f978475-ctb67           1/1     Running   0          10m
eventing-controller-76f978475-l8p98           1/1     Running   0          9m39s
eventing-controller-76f978475-wlpcv           1/1     Running   0          10m
eventing-webhook-79f898797-ltzwh              1/1     Running   0          10m
eventing-webhook-79f898797-rtw4r              1/1     Running   0          10m
kafka-broker-dispatcher-0                     1/1     Running   0          14m
kafka-broker-dispatcher-1                     1/1     Running   0          14m
kafka-broker-dispatcher-10                    1/1     Running   0          2m8s
kafka-broker-dispatcher-11                    1/1     Running   0          2m8s
kafka-broker-dispatcher-12                    1/1     Running   0          2m8s
kafka-broker-dispatcher-13                    1/1     Running   0          2m8s
kafka-broker-dispatcher-14                    1/1     Running   0          2m8s
kafka-broker-dispatcher-2                     1/1     Running   0          2m8s
kafka-broker-dispatcher-3                     1/1     Running   0          2m8s
kafka-broker-dispatcher-4                     1/1     Running   0          2m8s
kafka-broker-dispatcher-5                     1/1     Running   0          2m8s
kafka-broker-dispatcher-6                     1/1     Running   0          2m8s
kafka-broker-dispatcher-7                     1/1     Running   0          2m8s
kafka-broker-dispatcher-8                     1/1     Running   0          2m8s
kafka-broker-dispatcher-9                     1/1     Running   0          2m8s
kafka-broker-receiver-5c6f978479-brnmj        1/1     Running   0          10m
kafka-broker-receiver-5c6f978479-kk7gj        1/1     Running   0          10m
kafka-controller-77b6997bc8-29wnn             1/1     Running   0          10m
kafka-controller-77b6997bc8-m26tt             1/1     Running   0          10m
kafka-controller-77b6997bc8-pgwxx             1/1     Running   0          10m
kafka-webhook-eventing-5f55558b96-jlllw       1/1     Running   0          10m
kafka-webhook-eventing-5f55558b96-rfhwg       1/1     Running   0          10m
kafka-webhook-eventing-5f55558b96-rmtpf       1/1     Running   0          10m
ke-sample-api-service-ffd8dc566-7lgxh         1/1     Running   0          10m
ke-sample-training-service-67fc74d7bd-r5pxk   1/1     Running   0          10m
knative-sandbox-dlq-cc6fd94c5-7sfg7           1/1     Running   0          10m

The primary difference is now I see our triggers becoming ready again. The new dispatcher pods actually came up completely this time vs the error reported in my original issue here.
This is the result of:

Deploy 1.14 -> failed eventing cluster
Redeploy 1.13, delete 1.14 statefulset -> back working
Deploy 1.14 -> failed eventing cluster
Manually delete dispatcher deployment -> still not working
~10 minutes additional dispatcher pods were created and everything came back online.

That leaves questions on why the post-install job that should cleanup the old dispatcher deployment is silently failing instead of waiting until the statefulset is completely available, and why after 1.14 is installed + old dispatcher deployment removed we have a period of time where our eventing cluster is down.

are you setting a specific replicas number for the statefulset kafka-broker-dispatcher?

Our default defines 2 replicas for the statefulset but it appears it auto-scales itself up to 15. When this happened earlier it blew up our k8 cluster quota so we had to increase resources. I am interested why it decided to scale up because increasing # deployments x7 is steep and we likely need a way to control that for production.

The number of replicas depends on the configured POD_CAPACITY, we default it to 20 but it can be increased
https://github.com/knative-extensions/eventing-kafka-broker/blob/4cf38f1acae9cfc0b3cc6747d5e1f6140b9c8e1b/control-plane/config/eventing-kafka-broker/200-controller/500-controller.yaml#L151-L153

Over night our 1.14 dev cluster went back to unavailable. All triggers flipped back to not ready with:
  - lastTransitionTime: "2024-09-10T22:14:41Z"
    message: "failed to initialize consumer group offset: cannot obtain Kafka cluster
      admin, kafka: client has run out of available brokers to talk to: 3 errors occurred:\n\t*
      EOF\n\t* EOF\n\t* EOF\n"
    reason: InitializeOffset
    status: "False"
    type: DependencyReady

Nothing has changed with our kafka cluster and we have several thousand other kafka consumers running off this same cluster without issue. Is there anything I can look into further to see any error logs from a running pod?

Can you share / look at the kafka-controller-* pod logs?

controller logs are filled with
{"level":"error","ts":"2024-09-11T13:07:05.344Z","logger":"kafka-broker-controller","caller":"controller/controller.go:566","msg":"Reconcile error","commit":"ff0b991-dirty","knative.dev/pod":"kafka-controller-77b6997bc8-2f6cd","knative.dev/controller":"knative.dev.eventing-kafka-broker.control-plane.pkg.reconciler.consumergroup.Reconciler","knative.dev/kind":"internal.kafka.eventing.knative.dev.ConsumerGroup","knative.dev/traceid":"ecfa7a7e-3e3a-493c-a596-f4e693c1283a","knative.dev/key":"conversation/knative-trigger-conversation-dev007-ke.assistant.clu-controller.v1.training-failed","duration":0.842944322,"error":"failed to initialize consumer group offset: cannot obtain Kafka cluster admin, kafka: client has run out of available brokers to talk to: 3 errors occurred:\n\t* EOF\n\t* EOF\n\t* EOF\n","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20240416145024-0f34a8815650/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20240416145024-0f34a8815650/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20240416145024-0f34a8815650/controller/controller.go:491"}
{"level":"info","ts":"2024-09-11T13:07:05.345Z","logger":"kafka-broker-controller.event-broadcaster","caller":"record/event.go:364","msg":"Event(v1.ObjectReference{Kind:\"ConsumerGroup\", Namespace:\"conversation\", Name:\"knative-trigger-conversation-dev007-ke.assistant.clu-controller.v1.training-failed\", UID:\"f8acb1bb-8025-433c-9002-4d0c7c4f0443\", APIVersion:\"internal.kafka.eventing.knative.dev/v1alpha1\", ResourceVersion:\"5909367411\", FieldPath:\"\"}): type: 'Warning' reason: 'InternalError' failed to initialize consumer group offset: cannot obtain Kafka cluster admin, kafka: client has run out of available brokers to talk to: 3 errors occurred:\n\t* EOF\n\t* EOF\n\t* EOF\n","commit":"ff0b991-dirty","knative.dev/pod":"kafka-controller-77b6997bc8-2f6cd"}
{"level":"error","ts":"2024-09-11T13:07:05.411Z","logger":"kafka-broker-controller","caller":"consumergroup/reconciler.go:303","msg":"Returned an error","commit":"ff0b991-dirty","knative.dev/pod":"kafka-controller-77b6997bc8-2f6cd","knative.dev/controller":"knative.dev.eventing-kafka-broker.control-plane.pkg.reconciler.consumergroup.Reconciler","knative.dev/kind":"internal.kafka.eventing.knative.dev.ConsumerGroup","knative.dev/traceid":"dbda47c0-28db-4c19-bdbf-ce9b7472ace6","knative.dev/key":"conversation/knative-trigger-conversation-dev016-ke.assistant.clu-controller.v1.training-start","targetMethod":"ReconcileKind","error":"failed to initialize consumer group offset: cannot obtain Kafka cluster admin, kafka: client has run out of available brokers to talk to: 3 errors occurred:\n\t* EOF\n\t* EOF\n\t* EOF\n","stacktrace":"knative.dev/eventing-kafka-broker/control-plane/pkg/client/internals/kafka/injection/reconciler/eventing/v1alpha1/consumergroup.(*reconcilerImpl).Reconcile\n\tknative.dev/eventing-kafka-broker/control-plane/pkg/client/internals/kafka/injection/reconciler/eventing/v1alpha1/consumergroup/reconciler.go:303\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20240416145024-0f34a8815650/controller/controller.go:542\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20240416145024-0f34a8815650/controller/controller.go:491"}
{"level":"error","ts":"2024-09-11T13:07:05.411Z","logger":"kafka-broker-controller","caller":"controller/controller.go:566","msg":"Reconcile error","commit":"ff0b991-dirty","knative.dev/pod":"kafka-controller-77b6997bc8-2f6cd","knative.dev/controller":"knative.dev.eventing-kafka-broker.control-plane.pkg.reconciler.consumergroup.Reconciler","knative.dev/kind":"internal.kafka.eventing.knative.dev.ConsumerGroup","knative.dev/traceid":"dbda47c0-28db-4c19-bdbf-ce9b7472ace6","knative.dev/key":"conversation/knative-trigger-conversation-dev016-ke.assistant.clu-controller.v1.training-start","duration":0.912392363,"error":"failed to initialize consumer group offset: cannot obtain Kafka cluster admin, kafka: client has run out of available brokers to talk to: 3 errors occurred:\n\t* EOF\n\t* EOF\n\t* EOF\n","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20240416145024-0f34a8815650/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20240416145024-0f34a8815650/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20240416145024-0f34a8815650/controller/controller.go:491"}
{"level":"info","ts":"2024-09-11T13:07:05.412Z","logger":"kafka-broker-controller.event-broadcaster","caller":"record/event.go:364","msg":"Event(v1.ObjectReference{Kind:\"ConsumerGroup\", Namespace:\"conversation\", Name:\"knative-trigger-conversation-dev016-ke.assistant.clu-controller.v1.training-start\", UID:\"c8ca0592-a9d8-4cbd-9494-fd2f79bc5632\", APIVersion:\"internal.kafka.eventing.knative.dev/v1alpha1\", ResourceVersion:\"5909367245\", FieldPath:\"\"}): type: 'Warning' reason: 'InternalError' failed to initialize consumer group offset: cannot obtain Kafka cluster admin, kafka: client has run out of available brokers to talk to: 3 errors occurred:\n\t* EOF\n\t* EOF\n\t* EOF\n","commit":"ff0b991-dirty","knative.dev/pod":"kafka-controller-77b6997bc8-2f6cd"}

I've tried setting the kafka-controller POD_CAPACITY back to 2 (replicas we ran with in 1.13) and now:
$ k get statefulsets -n knative-eventing
NAME                      READY    AGE
kafka-broker-dispatcher   15/141   20h

I am not sure what is trying to scale dispatchers, but it is out of control.

you need to increase POD_CAPACITY if you want less dispatcher replicas, here is the document describing how that works https://docs.google.com/document/d/1UktwiDyqq07MtA7pUlahEpux5CCdAsyI6k3nkQeeqXw/edit

Yep, that did it. I increased POD_CAPACITY to 40. I had to manually scale down the statefulset to be able to edit the resources and for now it looks like the system is sticking with 15 replicas.

The cluster is not stable, no. Over the course of the past hour we've now had a handful of triggers become unready with BindInProgress, currently at 5/266 triggers unready.
  - lastTransitionTime: "2024-09-11T14:22:07Z"
    reason: BindInProgress
    status: "False"
    type: DependencyReady

Another 10 minutes and ~18 ddditional triggers are now also becoming unready with InitializeOffset same EOF errors as before.

I spent some more time trying to track down those EOL errors and have discovered they have been happening since 8/31 in our cluster. 8/31 is when I attempted our most recent 1.14 upgrade and was what prompted opening this issue. Since that time, even after 1.13 downgrades, kafka-controller pods appear to start throwing these errors. On 1.13 during this time all triggers register as available and all events I have tested are continuing to flow.
I've engaged our Kafka service provider and their initial analysis of our cluster shows no issues. At this point the issue is directly linked to just Knative and began as part of the 1.14 upgrade attempt. So either Knative 1.14 bug, error with our configuration of this upgrade, or a combination of both. I've handled most all of our upgrades since 1.8.x, so this is very much an outlier in that we can't get anything to work/stay stable for more than a few hours.

Does it ever recovers from EOF errors? How long it takes to recover from EOF errors?

As far as I can tell, it does not automatically recover. I believe what I saw yesterday morning, with things initially working after deploying only to slowly have trigger readiness fail, is what happened over an extended period of time the previous night. Eventually no triggers are available and there are no events being received/sent by brokers.
I am in the process of setting up a new separate dev cluster to install 1.13 on and attempt the 1.14 upgrade there for a baseline comparison.

I have reproduced the EOF error on a new k8 cluster. Initially installed our prod 1.13 configuration with our tests passing and then attempted the first 1.14 upgrade on that cluster. It immediately began failing with the same errors
$ k get triggers
NAME                                     BROKER                   SUBSCRIBER_URI                                                    AGE   READY   REASON
sample-api-service-training-complete     knative-sandbox-broker                                                                     76m   False   InitializeOffset
sample-api-service-training-failed       knative-sandbox-broker   http://ke-sample-api-service.knative-eventing.svc.cluster.local   76m   False   InitializeOffset
sample-training-service-training-start   knative-sandbox-broker                                                                     76m   False   InitializeOffset

  - lastTransitionTime: "2024-09-12T19:59:44Z"
    message: "failed to initialize consumer group offset: cannot obtain Kafka cluster
      admin, kafka: client has run out of available brokers to talk to: 3 errors occurred:\n\t*
      EOF\n\t* EOF\n\t* EOF\n"
    reason: InitializeOffset
    status: "False"
    type: DependencyReady

I do still observe events being able to be sent to both sample-training-service-training-start and sample-training-service-training-complete triggers despite them not being in a Ready state. I can leave this on the cluster again over night to see what happens, but we still need to resolve the latent errors and issue with triggers readiness.

After 12 hours the triggers have not changed and are stilled marked as not ready due to the InitializeOffset / EOF error. The kafka instance used for this deployment is the same as our primary dev cluster. During the 12 hours we had around 15-20,000 non-knative events successfully flowing through kafka in addition to ~200 knative events via our main dev 1.13 cluster.
I have also experimented with pointing this isolated cluster to use our staging kafka instance. The same error shows up immediately in the kafka-controller and eventing is down. The dispatcher statefulsets, after switching kafka, will not restart (gave up waiting after 30 minutes) and are just hung on ContainerCreating because of
MountVolume.SetUp failed for volume "contract-resources" : configmap "kafka-broker-dispatcher-1" not found

Our staging kafka instance has less total topics/partitions, but is much larger and drives throughput closer to our production clusters (around 20,000 non-knative events/minute).

Backport to speed up mount volume https://github.com/knative-extensions/eventing-kafka-broker/pull/4101, I think that will help on the latest issue MountVolume.SetUp failed for volume "contract-resources" : configmap "kafka-broker-dispatcher-1" not found

With https://github.com/knative-extensions/eventing-kafka-broker/pull/4103, once backported to 1.14, we can try to get more debug logs about the Kafka client connections or disable the new client pooling
ENABLE_SARAMA_LOGGER -> "true"
ENABLE_SARAMA_DEBUG_LOGGER -> "true"
ENABLE_SARAMA_CLIENT_POOL -> "false"

New version with https://github.com/knative-extensions/eventing-kafka-broker/issues/4091#issuecomment-2354737144 is out https://github.com/knative-extensions/eventing-kafka-broker/releases/tag/knative-v1.14.11

I tried out 1.14.11 on my test cluster and have promising results. I pulled resource updates to our k8 templates from the release and things immediately began failing with EOF again, but I noticed that the yaml resources didn't have any of the new ENVs. I added them into the controller deployment:
            - name: ENABLE_SARAMA_LOGGER
              value: "false"
            - name: ENABLE_SARAMA_DEBUG_LOGGER
              value: "false"
            - name: ENABLE_SARAMA_CLIENT_POOL
              value: "true"

which has resulted in a stable cluster so far. Deployment of our test broker/trigger/services all came up and eventing is flowing with none of the previous EOF errors. I do see a few of
E0918 15:56:38.288958       1 leaderelection.go:308] Failed to release lock: Put "https://<IP_ADDRESS>:443/apis/coordination.k8s.io/v1/namespaces/knative-eventing/leases/kafka-broker-controller.knative.dev.eventing-kafka-broker.control-plane.pkg.reconciler.broker.reconciler.00-of-01": unexpected EOF

but thus far the triggers and eventing have remained available. I will continue to let this sit for a day in this test environment before I wipe it out and start back with our base 1.13 and try an upgrade path. Is there anything I could get/try for you in the meantime?

After around 1-2 hours triggers became not ready. Same old EOF errors spewing in kafka-controller pods.
I modified kafka-controller to set:
ENABLE_SARAMA_LOGGER -> "true"
ENABLE_SARAMA_DEBUG_LOGGER -> "true"
ENABLE_SARAMA_CLIENT_POOL -> "false"

When everything came back up, triggers went back to ready state. Another hour or so later and triggers are again not ready. I've captured the latest logs and sent them along via email.

The EOF on the leaderelection is a little odd, are we sure the environment is stable?

I modified kafka-controller to set:
ENABLE_SARAMA_LOGGER -> "true"
ENABLE_SARAMA_DEBUG_LOGGER -> "true"
ENABLE_SARAMA_CLIENT_POOL -> "false"

This is not yet released and merged, in the previous reply I was referring to https://github.com/knative-extensions/eventing-kafka-broker/issues/4091#issuecomment-2354737144

Backport to speed up mount volume https://github.com/knative-extensions/eventing-kafka-broker/pull/4101, I think that will help on the latest issue MountVolume.SetUp failed for volume "contract-resources" : configmap "kafka-broker-dispatcher-1" not found

Same problem is happening in 1.15

Id like to point out that my problem stemmed from running knative kafka across two different clusters. Each knative instance resulted in duplicate consumergroups for two different topics. This then resulted in consumers never committing offsets or polling messages. Eventually leading to all triggers being taken down.
To fix this I changed the template from: knative-trigger-{{ .Namespace }}-{{ .Name }} -> knative-trigger-{{ .Namespace }}-{{ .Name }}-{{ .Environment }} which results in unique consumer groups.
I then changed the offset to earliest to make sure we didnt lose any messages and then re ran our cicd pipelines to re-create the triggers after I deleted them.
This has resulted in no longer seeing:
failed to initialize consumer group offset: cannot obtain Kafka cluster admin, kafka: client has run out of available brokers to talk to: EOF

What lead me to this path was this log line:
{"@timestamp":"2024-09-20T16:33:34.693Z","@version":"1","message":"[Consumer clientId=REDACTED, groupId=REDACTED] Request joining group due to: encountered UNKNOWN_MEMBER_ID from JOIN_GROUP response","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"Thread-21","level":"INFO","level_value":20000}

Our dev configurations are similar to what @elebiodaslingshot described, but we do not have any JOIN_GROUP errors. The closest I have seen is
Request joining group due to: encountered UNKNOWN_MEMBER_ID from HEARTBEAT response

showed up once over the past week and does not appear to correlate with our trigger readiness.
That said, the scenario makes a lot of sense. We have, starting very recently, begun running multiple dev knative clusters off the same Kafka instance. Those were on 1.13 and seem to tolerate this issue as there is no evidence of it impacting eventing. While some of our triggers are ancient and still use GUID consumer groups, the majority would use the triggers-consumergroup-template which could impact us. We encode service/tenant information into trigger names, so the majority of trigger consumer groups are unique by default, but there are a handful per cluster that overlap and would logically hit weird issues like this.
I've made similar adjustments to include environment details in triggers-consumergroup-template and I managed to get things up and running. Over the past hour I did see a batch of EOF errors but no indication that triggers went down. I plan to let this bake over the weekend and if things go well try it in our primary dev cluster.

I've being trying to reproduce for a while with varying level of "scale", the only thing I found was a rather slow time to recover in the scheduler / autoscaler (see here for more details: https://github.com/knative/eventing/pull/8200) but I could reproduce the EOF errors

I've tried a as-vanilla-as-possible install of 1.14 release and still see the issue presenting. Today the EOF errors are occurring at 1 hour intervals with just 1 broker / 3 triggers created

While the 1.14 Knative install hits this issue and the triggers go unavailable, our 1.13 Knative cluster (on the same Kafka instance) continues to work fine along with a handful of other service stacks (altogether creating a couple hundred consumer groups). I've spoken with some of those teams, and at least one runs Go/Sarama as well. During the times where Knative is throwing EOF errors, their consumers are all up with no sign of the errors in their stack.
Looking through sarama, that library has had several issues opened related to this type of error with no resolution by that team.

@pierDipi is there an eta on new 1.14 release containing https://github.com/knative-extensions/eventing-kafka-broker/pull/4103?
I've continued to test with 1.14 without much change. Initially I did find some different errors when running kafka-controller with debug logging. Instead of EOF received connect: no route to host errors. Triaged that down to a subnet missing on some worker nodes in the new k8 cluster I am using, but once fixed I'm back to just the EOF errors with nothing else jumping out from debug logging. Double verified that when EOF errors are getting thrown, connectivity to the same Kafka IPs in the kafka-controller pod itself is still working fine.
I've attempted to jump to 1.15 to get access to the sarama logging ENVs but that fairs even worse. I am able to get services, brokers and triggers all up but no events are flowing through KE stack. I haven't spent much time digging through that release, but it looks like changes are sufficient that our existing configs are broken.
I've also had some discussions with other internal teams. We have a component that also interfaces with Kafka written in Go and uses the Sarama kafka client. They have taken a look at the periods where KE is receiving EOF errors and have been able to confirm that there were no interruptions within their stack during the same period. This component stack handles ~200x the kafka throughput that the KE stack sees, so it is a good canary in this case.

Note also that there is the template for the topic name that is based solely on namespace and name https://github.com/knative-extensions/eventing-kafka-broker/blob/b02817c50b9f700d1540ca878e7a79b0342d5fad/control-plane/config/eventing-kafka-broker/200-controller/100-config-kafka-features.yaml#L48-L50 and maybe using the same Kafka cluster for 2 different Knative installations could cause issues

@pierDipi is there an eta on new 1.14 release containing #4103?

It will be released shortly once one of the jobs here succeeds https://prow.knative.dev/?type=periodic&job=release_eventing-kafka-broker_release-1.14_periodic

I deployed 1.14.12 with sarama debug logging turned on and observed the error when triggers go down after working for a while:
[sarama][debug] 2024/09/26 18:09:47 github.com/IBM/sarama@v1.43.1/client.go:1027: client/metadata fetching metadata for all topics from broker kafka-0:9093
[sarama][debug] 2024/09/26 18:19:47 github.com/IBM/sarama@v1.43.1/client.go:1027: client/metadata fetching metadata for all topics from broker kafka-2:9093
[sarama] 2024/09/26 18:19:47 github.com/IBM/sarama@v1.43.1/client.go:1073: client/metadata got error from broker 2 while fetching metadata: write tcp x.x.x.x:52230->x.x.x.x:9093: write: broken pipe
[sarama] 2024/09/26 18:19:47 github.com/IBM/sarama@v1.43.1/broker.go:344: Error while closing connection to broker kafka-2:9093: tls: failed to send closeNotify alert (but connection was closed anyway): write tcp x.x.x.x:52230->x.x.x.x:9093: write: broken pipe
[sarama] 2024/09/26 18:19:47 github.com/IBM/sarama@v1.43.1/client.go:774: client/brokers deregistered broker #2 at kafka-2:9093
[sarama][debug] 2024/09/26 18:19:47 github.com/IBM/sarama@v1.43.1/client.go:1027: client/metadata fetching metadata for all topics from broker kafka-0:9093
[sarama] 2024/09/26 18:19:47 github.com/IBM/sarama@v1.43.1/client.go:1073: client/metadata got error from broker 0 while fetching metadata: EOF
[sarama][debug] 2024/09/26 18:19:47 github.com/IBM/sarama@v1.43.1/broker.go:342: Closed connection to broker kafka-0:9093
[sarama] 2024/09/26 18:19:47 github.com/IBM/sarama@v1.43.1/client.go:774: client/brokers deregistered broker #0 at kafka-0:9093
[sarama][debug] 2024/09/26 18:19:47 github.com/IBM/sarama@v1.43.1/client.go:1027: client/metadata fetching metadata for all topics from broker kafka-1:9093
[sarama] 2024/09/26 18:19:47 github.com/IBM/sarama@v1.43.1/client.go:1073: client/metadata got error from broker 1 while fetching metadata: write tcp x.x.x.x:51990->x.x.x.x:9093: write: broken pipe
[sarama] 2024/09/26 18:19:47 github.com/IBM/sarama@v1.43.1/broker.go:344: Error while closing connection to broker kafka-1:9093: tls: failed to send closeNotify alert (but connection was closed anyway): write tcp x.x.x.x:51990->x.x.x.x:9093: write: broken pipe
[sarama] 2024/09/26 18:19:47 github.com/IBM/sarama@v1.43.1/client.go:774: client/brokers deregistered broker #1 at kafka-1:9093
[sarama][debug] 2024/09/26 18:19:47 github.com/IBM/sarama@v1.43.1/client.go:1027: client/metadata fetching metadata for all topics from broker kafka-0:9093
[sarama] 2024/09/26 18:19:47 github.com/IBM/sarama@v1.43.1/client.go:1073: client/metadata got error from broker -1 while fetching metadata: write tcp x.x.x.x:41544->x.x.x.x:9093: write: broken pipe
[sarama] 2024/09/26 18:19:47 github.com/IBM/sarama@v1.43.1/broker.go:344: Error while closing connection to broker kafka-0:9093: tls: failed to send closeNotify alert (but connection was closed anyway): write tcp x.x.x.x:41544->x.x.x.x:9093: write: broken pipe

which seems to indicate Knative/Sarama client is not properly closing/refreshing connections.
I have also observed the same EOF error but on start up of kafka-controller pods. We run with 3 replicas and during this error, when the pod come up at least 1 out of the 3 pods will immediately hit the EOF errors. This brings the triggers down and as far as I have seen the pod never recovers. The other 2 pods are up and communicate to kafka fine. The impacted pod has full network connectivity to the kafka instance. Most importantly all 3 pods have the same kafka config... so 2/3 connecting with one failing SASL handshake is supicious.
[sarama] 2024/09/26 18:58:52 github.com/IBM/sarama@v1.43.1/config.go:618: ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama][debug] 2024/09/26 18:58:52 github.com/IBM/sarama@v1.43.1/client.go:1027: client/metadata fetching metadata for all topics from broker kafka-1:9093
[sarama] 2024/09/26 18:58:52 github.com/IBM/sarama@v1.43.1/broker.go:1250: Error while performing SASL handshake kafka-1:9093
[sarama][debug] 2024/09/26 18:58:52 github.com/IBM/sarama@v1.43.1/broker.go:267: Closed connection to broker kafka-1:9093
[sarama] 2024/09/26 18:58:52 github.com/IBM/sarama@v1.43.1/client.go:1073: client/metadata got error from broker -1 while fetching metadata: EOF
[sarama] 2024/09/26 18:58:52 github.com/IBM/sarama@v1.43.1/client.go:1086: client/metadata no available broker to send metadata request to
[sarama] 2024/09/26 18:58:52 github.com/IBM/sarama@v1.43.1/client.go:788: client/brokers resurrecting 3 dead seed brokers

I also tried with
      ENABLE_SARAMA_LOGGER:                       true
      ENABLE_SARAMA_DEBUG_LOGGER:                 true
      ENABLE_SARAMA_CLIENT_POOL:                  false

which resulted in the latter example of pods coming up with immediate EOF errors and down eventing.

I have also observed the same EOF error but on start up of kafka-controller pods. We run with 3 replicas and during this error, when the pod come up at least 1 out of the 3 pods will immediately hit the EOF errors. This brings the triggers down and as far as I have seen the pod never recovers. The other 2 pods are up and communicate to kafka fine

Can you share the leases in the knative-eventing namespace to identify the leader pod we need to focus on?
kubectl get leases -n knative-eventing -oyaml

A few other people are encountering EOF here with sarama 1.43 and Go 1.122 https://github.com/IBM/sarama/issues/2938

The issue I'm having is that sarama is never logging the actual error associated with "Error while performing SASL handshake kafka-1:9093"
https://github.com/IBM/sarama/blob/893978c87fe7af13a2b6849ba62b003493f97f25/broker.go#L1243-L1252
https://github.com/IBM/sarama/blob/893978c87fe7af13a2b6849ba62b003493f97f25/broker.go#L260-L275

I also tried with
      ENABLE_SARAMA_LOGGER:                       true
      ENABLE_SARAMA_DEBUG_LOGGER:                 true
      ENABLE_SARAMA_CLIENT_POOL:                  false

which resulted in the latter example of pods coming up with immediate EOF errors and down eventing.

For now, let's keep this configuration and continue gather logs and insights as it helps with reducing the moving parts

Can you share the leases in the knative-eventing namespace to identify the leader pod we need to focus on?

Snapshot of kubectl get leases -n knative-eventing -oyaml: leases.txt

Can you share the Kafka cluster version you're connecting to?

Our Kafka clusters are version 3.6.
I've tried deploying 1.14 with increased req/lim CPU resources for kafka-controller, but I do not believe that is realistically the issue. Our monitoring shows the kafka-controller pods using < 0.1m CPU during these EOF issues with no CPU throttling.
The EOF error is regularly associated with client/metadata sarama calls and looks like when the issue is hit no attempt at reconnecting is made (or at least not logged). eg:
[sarama] 2024/09/27 15:46:51 github.com/IBM/sarama@v1.43.1/client.go:1073: client/metadata got error from broker -1 while fetching metadata: EOF
[sarama] 2024/09/27 15:46:51 github.com/IBM/sarama@v1.43.1/client.go:1073: client/metadata got error from broker -1 while fetching metadata: EOF
[sarama] 2024/09/27 15:46:51 github.com/IBM/sarama@v1.43.1/client.go:1073: client/metadata got error from broker -1 while fetching metadata: EOF
[sarama] 2024/09/27 15:46:51 github.com/IBM/sarama@v1.43.1/client.go:1073: client/metadata got error from broker -1 while fetching metadata: EOF
[sarama] 2024/09/27 15:46:51 github.com/IBM/sarama@v1.43.1/client.go:1073: client/metadata got error from broker -1 while fetching metadata: EOF
[sarama] 2024/09/27 15:46:51 github.com/IBM/sarama@v1.43.1/client.go:1073: client/metadata got error from broker -1 while fetching metadata: EOF
{"level":"error","ts":"2024-09-27T15:46:51.412Z","logger":"kafka-broker-controller","caller":"consumergroup/reconciler.go:303","msg":"Returned an error","commit":"1754676-dirty","knative.dev/pod":"kafka-controller-775b4d995b-xg8vj","knative.dev/controller":"knative.dev.eventing-kafka-broker.control-plane.pkg.reconciler.consumergroup.Reconciler","knative.dev/kind":"internal.kafka.eventing.knative.dev.ConsumerGroup","knative.dev/traceid":"4fc6748b-0928-4dec-9e60-9545a79fafcf","knative.dev/key":"knative-eventing/knative-trigger-some-k8-cluster-knative-eventing-sample-api-service-training-complete","targetMethod":"ReconcileKind","error":"failed to initialize consumer group offset: cannot obtain Kafka cluster admin, kafka: client has run out of available brokers to talk to: 3 errors occurred:\n\t* EOF\n\t* EOF\n\t* EOF\n","stacktrace":"knative.dev/eventing-kafka-broker/control-plane/pkg/client/internals/kafka/injection/reconciler/eventing/v1alpha1/consumergroup.(*reconcilerImpl).Reconcile\n\tknative.dev/eventing-kafka-broker/control-plane/pkg/client/internals/kafka/injection/reconciler/eventing/v1alpha1/consumergroup/reconciler.go:303\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20240416145024-0f34a8815650/controller/controller.go:542\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20240416145024-0f34a8815650/controller/controller.go:491"}
{"level":"error","ts":"2024-09-27T15:46:51.412Z","logger":"kafka-broker-controller","caller":"controller/controller.go:566","msg":"Reconcile error","commit":"1754676-dirty","knative.dev/pod":"kafka-controller-775b4d995b-xg8vj","knative.dev/controller":"knative.dev.eventing-kafka-broker.control-plane.pkg.reconciler.consumergroup.Reconciler","knative.dev/kind":"internal.kafka.eventing.knative.dev.ConsumerGroup","knative.dev/traceid":"4fc6748b-0928-4dec-9e60-9545a79fafcf","knative.dev/key":"knative-eventing/knative-trigger-some-k8-cluster-knative-eventing-sample-api-service-training-complete","duration":0.850037078,"error":"failed to initialize consumer group offset: cannot obtain Kafka cluster admin, kafka: client has run out of available brokers to talk to: 3 errors occurred:\n\t* EOF\n\t* EOF\n\t* EOF\n","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20240416145024-0f34a8815650/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20240416145024-0f34a8815650/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20240416145024-0f34a8815650/controller/controller.go:491"}
{"level":"info","ts":"2024-09-27T15:46:51.412Z","logger":"kafka-broker-controller.event-broadcaster","caller":"record/event.go:364","msg":"Event(v1.ObjectReference{Kind:\"ConsumerGroup\", Namespace:\"knative-eventing\", Name:\"knative-trigger-some-k8-cluster-knative-eventing-sample-api-service-training-complete\", UID:\"13af28dd-68a3-4259-af1c-477f01688b5c\", APIVersion:\"internal.kafka.eventing.knative.dev/v1alpha1\", ResourceVersion:\"1412480808\", FieldPath:\"\"}): type: 'Warning' reason: 'InternalError' failed to initialize consumer group offset: cannot obtain Kafka cluster admin, kafka: client has run out of available brokers to talk to: 3 errors occurred:\n\t* EOF\n\t* EOF\n\t* EOF\n","commit":"1754676-dirty","knative.dev/pod":"kafka-controller-775b4d995b-xg8vj"}

In cases where the connection is failed, the client should be destroyed and recreated and in this case it looks like it just continues to try and reuse a dead connection. My understanding of our internal components using Go/Sarama is that they required additional resiliency added to support production level scale. Additionally, there may need to be tweaks to Sarama client configs (timeouts and whatnot) to handle large scale clusters with hundreds of topics/thousands of partitions like what we run.
On the subject of large kafka clusters, we provisioned a brand new Kafka 3.6 cluster today and deployed 1.14.12 using it. EOF errors continue to be thrown despite using a kafka cluster with 1 topic and 2 partitions and only this one KE cluster using it.

@mdwhitley did you disable the client pool https://github.com/knative-extensions/eventing-kafka-broker/issues/4091#issuecomment-2378640608?

Yes. I have tried with ENABLE_SARAMA_CLIENT_POOL=false, and I see the same problems.
Over the weekend I left the test cluster on 1.14 w/ client pool disabled and all triggers are currently marked as Not Ready/InitializeOffset. 1/3 kafka-controller pods is throwing constant EOF errors while the other 2 pods have no errors. Despite triggers reporting not ready, I am able to send events to them and they get routed to our receivers. Example error logs:
[sarama] 2024/09/30 16:37:29 github.com/IBM/sarama@v1.43.1/client.go:1012: client/metadata retrying after 250ms... (0 attempts remaining)
[sarama] 2024/09/30 16:37:29 github.com/IBM/sarama@v1.43.1/config.go:618: ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama][debug] 2024/09/30 16:37:29 github.com/IBM/sarama@v1.43.1/client.go:1027: client/metadata fetching metadata for all topics from broker kafka-2:9093
[sarama] 2024/09/30 16:37:29 github.com/IBM/sarama@v1.43.1/broker.go:1250: Error while performing SASL handshake kafka-2:9093
[sarama][debug] 2024/09/30 16:37:29 github.com/IBM/sarama@v1.43.1/broker.go:267: Closed connection to broker kafka-2:9093
[sarama] 2024/09/30 16:37:29 github.com/IBM/sarama@v1.43.1/client.go:1073: client/metadata got error from broker -1 while fetching metadata: EOF
[sarama] 2024/09/30 16:37:29 github.com/IBM/sarama@v1.43.1/config.go:618: ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama][debug] 2024/09/30 16:37:29 github.com/IBM/sarama@v1.43.1/client.go:1027: client/metadata fetching metadata for all topics from broker kafka-0:9093
[sarama] 2024/09/30 16:37:29 github.com/IBM/sarama@v1.43.1/broker.go:1250: Error while performing SASL handshake kafka-0:9093
[sarama][debug] 2024/09/30 16:37:29 github.com/IBM/sarama@v1.43.1/broker.go:267: Closed connection to broker kafka-0:9093
[sarama] 2024/09/30 16:37:29 github.com/IBM/sarama@v1.43.1/client.go:1073: client/metadata got error from broker -1 while fetching metadata: EOF
[sarama] 2024/09/30 16:37:29 github.com/IBM/sarama@v1.43.1/config.go:618: ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama][debug] 2024/09/30 16:37:29 github.com/IBM/sarama@v1.43.1/client.go:1027: client/metadata fetching metadata for all topics from broker kafka-1:9093
[sarama] 2024/09/30 16:37:30 github.com/IBM/sarama@v1.43.1/broker.go:1250: Error while performing SASL handshake kafka-1:9093
[sarama][debug] 2024/09/30 16:37:30 github.com/IBM/sarama@v1.43.1/broker.go:267: Closed connection to broker kafka-1:9093
[sarama] 2024/09/30 16:37:30 github.com/IBM/sarama@v1.43.1/client.go:1073: client/metadata got error from broker -1 while fetching metadata: EOF
[sarama] 2024/09/30 16:37:30 github.com/IBM/sarama@v1.43.1/client.go:1086: client/metadata no available broker to send metadata request to
[sarama] 2024/09/30 16:37:30 github.com/IBM/sarama@v1.43.1/client.go:788: client/brokers resurrecting 3 dead seed brokers
[sarama][debug] 2024/09/30 16:37:30 github.com/IBM/sarama@v1.43.1/client.go:309: Closing Client

Sarama error rehttps://github.com/IBM/sarama/issues/2994

My understanding of our internal components using Go/Sarama is that they required additional resiliency added to support production level scale. Additionally, there may need to be tweaks to Sarama client configs (timeouts and whatnot) to handle large scale clusters with hundreds of topics/thousands of partitions like what we run.

Thanks for the hints, contributions or specific suggestions are really welcome in this area

In cases where the connection is failed, the client should be destroyed and recreated and in this case it looks like it just continues to try and reuse a dead connection

If you disable, client pool, we create a new client each time we reconcile an object

From https://github.com/IBM/sarama/issues/2981, it seems that EOF is often the error got when there is a protocol mismatch

From https://github.com/IBM/sarama/issues/2981, it seems that EOF is the error got when there is a protocol mismatch

That and SASL configuration error. If there was a misconfiguration, then I would expect no eventing to ever work which is simply not the case in our environments. Same config is used everywhere and works except for knative 1.14. Exact same config works on 1.13 without any issues with triggers going unready.
Beyond the Sarama pool/logging ENV, are there any ways to influence the kafka properties used for the sarama client?

Release 1.14.13 which includes https://github.com/knative-extensions/eventing-kafka-broker/pull/4125 resolves the issues we have been experiencing. All existing triggers continue to be available/ready and new triggers can be created successfully.


Usage with volt?

Just spitballing ideas here - but if there were a way to;

Use this with volt functional style Livewire
A vite plugin to extract ts/js from a blade file itself

Then you could conceivably make a single file Livewire react/vue component that defines and exposes it's own API layer/data requirements
With folio this would be really similar to next/remix style page components
It'd  be really cool to be able to write volt functions in single files regardless of whether they're Livewire/alpine/react/vue/whatever
Can you think of any reason this wouldn't work?

Hey Dean!
Only reason I can think of is my limited knowledge 😄
But I do love the boldness and thinking out of the box ❤
Is this something you would like to put your hands on and contribute?

I had a play with this today - only gotchas so far is I still haven't got
vite playing ball perfectly (it's kinda working but breaks when I import
react components via js), and render on Volt/Component is final, so I
just removed the final blocker locally to see if it'd work and it looks
like it would 🤷♂
On the upside vscode was happy to treat a script tag in a blade file as
typescript, so it looks like it'd be somewhat usable if it did work
On Mon, 20 May 2024, 4:58 pm Joao Patricio, @.***>
wrote:

Hey Dean!
Only reason I can think of is my limited knowledge 😄
But I do love the boldness and thinking out of the box ❤
Is this something you would like to put your hands on and contribute?
—
Reply to this email directly, view it on GitHub
https://github.com/ijpatricio/mingle/issues/11#issuecomment-2119797763,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAVFHPX5OWELQBATIJASO6TZDGNIHAVCNFSM6AAAAABH6XHNQWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJZG44TONZWGM
.
You are receiving this because you authored the thread.Message ID:
@.***>

OK, I got this working;
Source: https://github.com/deanmcpherson/react-in-livewire-example
Demo: https://scenic-star-slvqcmdwqyh7.vapor-farm-f1.com
It essentially allows you to call Mingle::volt() in a volt blade file, and it will automatically compile out the first script tag.
E.g.
<?php

use function Livewire\Volt\{state};

Mingle::volt();

state([
    'hello' => 'world',
    'count' => 1,
]);

$makeItGoBoom = fn() => dd('This is dd');
$doubleIt = function() {
    $this->count *= 2;
};
?>

Thanks!
I think there's a few pieces in here, I'm not sure if mingle is even the right place the auto-extracting-inline-component voodoo I've got running
On the simpler side, you can also just call Mingle::volt("resources/js/blah.jsx"); and have it render like a standard mingle component without any plugin shenanigans

I made it with Vuejs and everything works normal. I used Volt class based anonymous component and just put empty div inside blade directive. Something like this
`use Ijpatricio\Mingle\Concerns\InteractsWithMingles;
use Ijpatricio\Mingle\Contracts\HasMingles;
use Livewire\Volt\Component;
new class () extends Component implements HasMingles {
public $count = 0;
use InteractsWithMingles;
public function component(): string
{
    return 'resources/js/Counter/index.js';
}

public function mingleData()
{
    return [
        'count' => $this->count
    ];
}

public function increment()
{
    $this->count++;
}

} `
Now just typical blade

 @volt('counter')

 @endvolt

Use Vuejs component like you would normaly do
There is just small issue that you should handle. HasMingle trait doesnt have return type on render function like Volt has (mixed). Youll get and error but it is easy fix, just set return type. It would be good for author to do that.

Hello everyone @deanmcpherson @cryptoforu
This seemed perfectly doable, so I went ahead and added that return type

To this day, I haven't used Volt 😅  Regular Livewire just hits me better, I guess I'm used to it over the years.
So I could feel more confident, would it be possible to see a reproduction repo of this?

Volt functional api requires the Vite plugin to extract, right? Unless we call Mingle::volt("resources/js/blah.jsx"); ??
How would that work? php file with the functions, and the jsx would live in another file?

Volt class api -> it seems it's ready already?

Like i said, class api works without problem with vuejs SFC. I even made it working with plugins, like VUE rtk query i vue laravel 18n. Sorry on my mobile now


View backup codes from account page 👀

This PR changes things up so that we are able to view our backup codes from account page, and then generate new codes on the /backup-codes page.
Only when we have no backup codes do we offer to generate them from the account page.
Screenshots
account page

before
after

"Get Security Codes" will generate 5 new codes
"View security codes" will show you your existing codes.

backup code page

before
after

"Visit portal" button at the bottom
"Get new security codes" button at the bottom

I went ahead and fixed two entries that weren't translating on the backup_list view.  The two buttons on the page still had fuzzy entries.

Thanks!


time format inconsistency

When the API returns the following error:
{"error":{"code":"accessDenied","message":"Self-service operations are currently not supported for user XXX.","innerError":{"message":"Self-service operations are currently not supported for user XXX.","date":"2023-07-07T13:06:51","request-id":"YYY","client-request-id":"ZZZ"}}}

The SDK fails to parse the response body because the date field is not properly formatted according to:
parsed, err := time.Parse(time.RFC3339, *v)

Note the missing "Z" or some other time zone designation at the end of the date field.

Hi @crewjam Thanks for trying this SDK. Are you able to share which endpoint is resulting to the response?

@rkodev The error is when the sdk is trying to parse the date in an error. You can try something simple like the following with an invalid userID to repro the same.
	client := msgraphsdk.NewGraphServiceClient(adapter)
	u, err := client.Users().
		ByUserId(userID).
		Drive().
		Get(context.Background(), nil)

Thanks for reply, all. Yes, @meain 's example does reproduce. The exact request I was doing was:
graphClient.Users().ByUserId(directoryEntry.ExternalID).Authentication().PasswordMethods().Get(ctx, nil)


Template the tomcat/carbon/WEB-INF/web.xml.j2 file

Purpose
Template the tomcat/carbon/WEB-INF/web.xml.j2 file and add default values for the configurable properties.
Part of the fix for the issue https://github.com/wso2/product-is/issues/10380

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/1409155208

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/1409155208
Status: failure

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/1409221303

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/1409221303
Status: failure

Changes included with https://github.com/wso2/carbon-kernel/pull/3116


Add TO Go /user/id/deliveryservices/available

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-trafficcontrol-PR/1817/
Test PASSed.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/1942/
Test PASSed.

Duplicate of https://github.com/apache/trafficcontrol/blob/2cf3457989a525493a03429a23912d7d01137d8c/traffic_ops/traffic_ops_golang/user/deliveryservices.go#L63


Remind me later doesn't honor reminder time

For users with schedules to transfer, setting a reminder doesn't seem to honor the reminder date.
For instance, I configured a reminder for 2 weeks, but it still appears again on refresh.

I can confirm this bug. However I can't seem to debug whats going wrong :-1:


Backup & restore docker swarm

RancherOS Version: (ros os version)
RancherOS v1.5.1
Where are you running RancherOS? (docker-machine, AWS, GCE, baremetal, etc.)
Running in VMs, docker swarm
As described here we want to backup and restore a docker swarm.
How can this be done if using RancherOS on the nodes?
Especially point 3 "stopping docker engine" seems not possible 🤔
Can /var/lib/docker/swarm be mounted directly on RancherOS as NFS share per node on boot or wouldn't you recommend it?

@x-jokay
Using system-docker stop docker to stop user-scope docker engine in ROS.
Mounting '/var/lib/docker/swarm' per node.  I wouldn't do it if I were you.

@Jason-ZW thank you, this works and is exactly what we need 👍


Fix #676, reference to deprecated CFE_SPACECRAFT_ID

Describe the contribution
Change to CFE_MISSION_SPACECRAFT_ID, which is the non-deprecated symbol.
Fixes #676
Testing performed
Build with extended header and OMIT_DEPRECATED and confirm success.
Expected behavior changes
Build now works with both extended headers and OMIT_DEPRECATED options set.
System(s) tested on
Ubuntu 20.04
Contributor Info - All information REQUIRED for consideration of pull request
Joseph Hickey, Vantage Systems, Inc.

CCB 20200506 - Approved


Size based rollover logging still broken in 1.0.2

It looks maxFileSize handling was not fixed in #1691. Since upgrading past 0.9.3 the following configuration results in no request log at all:
server:
  type: default
  applicationConnectors:
    - type: http
      port: 80
  adminConnectors:
    - type: http
      port: 8081
  registerDefaultExceptionMappers: false
  requestLog:
    appenders:
      - type: console
      - type: file
        currentLogFilename: /var/log/service_name/request.log
        archivedLogFilenamePattern: /var/log/service_name/request.log.%i
        maxFileSize: 50MB

The following configuration results in a working request log at /var/log/service_name/request.log:
server:
  type: default
  applicationConnectors:
    - type: http
      port: 80
  adminConnectors:
    - type: http
      port: 8081
  registerDefaultExceptionMappers: false
  requestLog:
    appenders:
      - type: console
      - type: file
        currentLogFilename: /var/log/service_name/request.log
        archivedLogFilenamePattern: /var/log/service_name/request.log.%d

I tested these using 1.0.2, and can confirm that the full configuration (only slightly more complex) works in 0.9.2 and 0.9.3, and fails in 1.0.0, 1.0.1 and 1.0.2.
Size based log rotation is important for my project because we're working with limited disk space.

Hello, @looppair.
Thanks for pointing this out! I've backported ab65a32 (the commit which fixes the size based rollover) to the 1.0.x branch and released Dropwizard 1.0.3. It should be available in the Maven Central repository in a couple of hours. Let me know, if the issue is fixed in this version.

Hello, @looppair!
Could you please verify whether Dropwizard 1.0.3 solves your issue or not. Thanks.

hello @arteam
It seems that you are familiar with the logging part. Could you please help me in https://github.com/dropwizard/dropwizard/issues/1827 ?

Closing as this is fixed.


Filtering siblings (two or more siblings, without parents)

Is your feature request related to a problem? Please describe.
When having two siblings in a family without parents, one is assigned index, while the other is not associated to the index (N/A)

This turns out to be an issue when performing comp het filtering:
In a scenario where both siblings are affected by the same disorder one would like to fiter for variants that are shared between both individuals and are in comp het state in both.
However, when setting comp het filter for index in genotype, the other sibling is automatically set to  and it is not possible to change it.
Likewise it is not possible to set the manually the parameters for in genotype in order to filter for comphet variants for both siblings.
Describe the solution you'd like
A potential solution would be to set comp het filter setting for each affected individual manually.
It would start the filter procedure as following:

find comphet variants in index
find comphet variants in sibling
display only shared variants.

Describe alternatives you've considered
there is no alternative other than filtering for comp het variants in index and checking manually whether they are in comphet state in sibling

another potential solution would be a similar filtering behavior as in cohorts, where each sample in genotpye filter can be set as "comphet index"
The results would be independent comphet variants for each sibling.
I think that ideally there should be two options in a duo sibling case when filtering comp het:

set index "c/h index" --> sibling is set to "any" --> index comphet filter results, sibling any.
set index "c/h index" --> sibling is set to "c/h sibling"  --> independent comp het filter results for index and sibling as in "c/h index"


pd.tslib._Timestamp should not be capable of crashing python.exe

Code Sample, a copy-pastable example if possible
python -c "import pandas as pd; print(pd.__version__); pd.tslib._Timestamp(42, 7, 13)"

Crashes python.exe on Windows 7, 10. Tested with  Python 3.3, 3.4, 3.5, pandas 0.16.2,  0.17.1, 0.18.0. ipython.exe crashes on exit when this code is executed.
Expected Output
Raise a suitable exception instead.

It should maybe not crash, but is there a reason you are using a private class?

this is not a public API and needs special construction inside the public class

Let me find an example of public API crashing python.exe. The above was the easiest way to point out that you may not always get an exception.

generally when you mess with cython things its pretty easy to crash python. If you find something in the public API that segfaults, pls reopen / new issue.


error in gulpfile.ts while install ng2-seed-advanced generator - "Unexpected Token import"

I'm submitting a ...  (check one with "x")
[ ] bug report => search github for a similar issue or PR before submitting
[ ] feature request
[ ] support request => Please do not submit support request here, instead see use [gitter](https://gitter.im/mgechev/angular2-seed) or [stackoverflow](https://stackoverflow.com/questions/tagged/angular2)

Current behavior
[11:04:42] Failed to load external module ts-node/register
[11:04:42] Failed to load external module typescript-node/register
[11:04:42] Failed to load external module typescript-register
[11:04:42] Failed to load external module typescript-require
/usr/local/lib/node_modules/generator-ng2-seed-advanced/node_modules/angular2-seed-advanced/gulpfile.ts:1
(function (exports, require, module, __filename, __dirname) { import * as gulp from 'gulp';
^^^^^^
SyntaxError: Unexpected token import
at Object.exports.runInThisContext (vm.js:78:16)
at Module._compile (module.js:543:28)
at Object.Module._extensions..js (module.js:580:10)
at Module.load (module.js:488:32)
at tryModuleLoad (module.js:447:12)
at Function.Module._load (module.js:439:3)
at Module.require (module.js:498:17)
at require (internal/module.js:20:19)
at Liftoff.handleArguments (/usr/local/Cellar/node/6.4.0/libexec/npm/lib/node_modules/gulp/bin/gulp.js:116:3)
at Liftoff. (/usr/local/Cellar/node/6.4.0/libexec/npm/lib/node_modules/gulp/node_modules/liftoff/index.js:198:16)
/usr/local/lib
└── (empty)
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none was installed.
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none was installed.
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none was installed.
npm ERR! Darwin 16.0.0
npm ERR! argv "/usr/local/Cellar/node/7.2.0/bin/node" "/usr/local/bin/npm" "i" "generator-ng2-seed-advanced" "-g"
npm ERR! node v7.2.0
npm ERR! npm  v3.10.9
npm ERR! code ELIFECYCLE
npm ERR<EMAIL_ADDRESS>postinstall: typings install && gulp check.versions && npm prune && gulp webdriver && node tools/install.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>postinstall script 'typings install && gulp check.versions && npm prune && gulp webdriver && node tools/install.js'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the angular2-seed-advanced package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     typings install && gulp check.versions && npm prune && gulp webdriver && node tools/install.js
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs angular2-seed-advanced
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls angular2-seed-advanced
npm ERR! There is likely additional logging output above.
npm ERR! Please include the following file with any support request:
npm ERR!     /Users/alokkulkarni/Documents/JavaSamples/angular/angularElectronTest/electronYo/npm-debug.log
npm ERR! code 1
Expected behavior
post install to complete for ng2-seed-advanced
Minimal reproduction of the problem with instructions

What is the motivation / use case for changing the behavior?

Please tell us about your environment:

Angular Seed Version: aaaaf75

Node: node --version = 7.2.0

[11:04:42] Failed to load external module ts-node/register
[11:04:42] Failed to load external module typescript-node/register
[11:04:42] Failed to load external module typescript-register
[11:04:42] Failed to load external module typescript-require
/usr/local/lib/node_modules/generator-ng2-seed-advanced/node_modules/angular2-seed-advanced/gulpfile.ts:1
(function (exports, require, module, __filename, __dirname) { import * as gulp from 'gulp';
^^^^^^
SyntaxError: Unexpected token import
at Object.exports.runInThisContext (vm.js:78:16)
at Module._compile (module.js:543:28)
at Object.Module._extensions..js (module.js:580:10)
at Module.load (module.js:488:32)
at tryModuleLoad (module.js:447:12)
at Function.Module._load (module.js:439:3)
at Module.require (module.js:498:17)
at require (internal/module.js:20:19)
at Liftoff.handleArguments (/usr/local/Cellar/node/6.4.0/libexec/npm/lib/node_modules/gulp/bin/gulp.js:116:3)
at Liftoff. (/usr/local/Cellar/node/6.4.0/libexec/npm/lib/node_modules/gulp/node_modules/liftoff/index.js:198:16)
/usr/local/lib
└── (empty)
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none was installed.
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none was installed.
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none was installed.
npm ERR! Darwin 16.0.0
npm ERR! argv "/usr/local/Cellar/node/7.2.0/bin/node" "/usr/local/bin/npm" "i" "generator-ng2-seed-advanced" "-g"
npm ERR! node v7.2.0
npm ERR! npm  v3.10.9
npm ERR! code ELIFECYCLE
npm ERR<EMAIL_ADDRESS>postinstall: typings install && gulp check.versions && npm prune && gulp webdriver && node tools/install.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>postinstall script 'typings install && gulp check.versions && npm prune && gulp webdriver && node tools/install.js'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the angular2-seed-advanced package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     typings install && gulp check.versions && npm prune && gulp webdriver && node tools/install.js
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs angular2-seed-advanced
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls angular2-seed-advanced
npm ERR! There is likely additional logging output above.
npm ERR! Please include the following file with any support request:
npm ERR!     /Users/alokkulkarni/Documents/JavaSamples/angular/angularElectronTest/electronYo/npm-debug.log
npm ERR! code 1

Hi @alokkulkarni it appears you are using the generator generator-ng2-seed-advanced?
That is actually not supported however you may want to open your issue on this repo instead:
https://github.com/peoplewareDo/generator-ng2-seed-advanced


test(auth): add unit test for auth selectors, fixes #336

What:

Add unit test for auth selectors

Issue number: #336

Hi @perjerz3434 , I'm sorry but there is already a PR open for this issue - https://github.com/tomastrajan/angular-ngrx-material-starter/pull/367;
I'm closing this one to prevent any confusion.


API - Docs: contributions points - schema property: lacking a lot of properties and documentation.

the api documentation about contributions points schema property is lacking a lot of properties and documentation.
For instance, this is a list of all the properties the LSP is showing containing the letter "a" in it.

A lot of them are not listed anywhere in the api documentation about contributions points schema property
Does this issue occur when all extensions are disabled?: No (Because that LSP feature it comes from a built-in extension (I didn't find which))

VS Code Version: 1.85.2 - 8b3775030ed1a69b13e4f4c628c612102e30a681 - x64

As explained in the documentation, this configuration keys are defined using a superset of JSON Schema.
In the VS Code documentation, we only list the properties we added on top.
https://json-schema.org/understanding-json-schema is a good documentation of JSON Schema.

Wdym @aeschli?
Are there not supposed to be any documentation for them?

It would take up a lot of space in the documentation and we would be repeating what's much better explained in the JSON schema documentation. We also don't go into the details of the JSON format. The target audience are extension developers, so I think it's ok to assume that they can look up the necessary information.

@aeschli, where's the schema, though? You haven't linked it yet and we need it understand how we should define our configurations

You can use everything from JSON schema (draft-07)
https://json-schema.org/understanding-json-schema describes the properties of JSON schemas and also has a line to the meta schema, if you need that? https://json-schema.org/draft-07/schema


SAK-43466 - Sections tool JSF upgrade

Hey Chris thanks for working on this it's really needed!
One suggestion instead of including this manual code to write a CSV file, Sakai already includes OpenCSV lib which makes it far more simple and less error prone. There is even a Bean pattern which basically writes a POJO as a CSV far more intuitive.
Examples https://www.baeldung.com/opencsv

@ern Sections only exports as xls so won't use the newly added opencsv export option.  Unless there is a desire to support xls and csv (which probably wouldn't be that hard - either add a second button or have it be a dropdown where either option could be selected.


Adding CSV output over time and fixing long-time typo

Summary:
Response times change over the course of a test, given that the CSV log continuously overwrites itself with stats based on a snapshot of the last 10 seconds, valuable data is lost of previous intervals. I have therefore added the option to allow writing an appended log of response times. While not set as default, I think this should be the default behaviour of the --csv flag.
Changes

First commit: All across the project every occurrence was missing the second r, this has been bugging me for a long time.

Second commit: Adds flag (--csv-append) for writing out a CSV log over time, essentially giving you the ability to track changes in the statistics as a function of test duration. Also added a column for total test duration at reporting interval.

Normal output (response.csv):
"Method","Name","Total elapsed time","# requests","# failures","Median response time","Average response time","Min response time","Max response time","Average Content Size","Requests/s"
"GET","/",25,172,0,13,13,10,55,12722,6.64
"GET","/about-me/",25,162,0,13,14,11,78,11484,6.25
"None","Total",25,334,0,13,14,10,78,12121,12.88

Appended output (response.csv):
"Method","Name","Total elapsed time","# requests","# failures","Median response time","Average response time","Min response time","Max response time","Average Content Size","Requests/s"
"GET","/",5,33,0,13,15,11,55,12722,5.57
"GET","/about-me/",5,38,0,13,14,12,70,11484,6.42
"GET","/",15,100,0,13,13,10,55,12722,6.70
"GET","/about-me/",15,95,0,13,15,11,78,11484,6.37
"GET","/",25,167,0,13,13,10,55,12722,6.44
"GET","/about-me/",25,158,0,13,14,11,78,11484,6.10
"GET","/",25,172,0,13,13,10,55,12722,6.64
"GET","/about-me/",25,162,0,13,14,11,78,11484,6.25
"None","Total",25,334,0,13,14,10,78,12121,12.88

same behaviour for the distribution.csv
/CC @heyman

@heyman any feedback? I think this is a critical feature for locust

Hi! This PR has been untouched for a very long time and has conflicts so I am closing it (I think this has already been solved, see https://docs.locust.io/en/stable/retrieving-stats.html.
Let me know if this was a mistake and feel free to open it again.


Right and left menu at the same time

Hello.
I am using your library. I added both right and left menu at the same time and identificated 2 different "SlidingRootNav" for them so I can reach the same menu by sliding the screen to both right and left but there is a problem.
When I open the left menu and click to a button, it opens right menu as it should be, but an graphical error occurs and the menu comes forward as half screen. I included an video so you can watch it. How can I solve that? By the way, I think you may develop this library as both right and left menu like I made. Thank you.
https://www.cbcapp.net/dosyalar/SlidingRootNav.mp4

Good work. But, a messy problem. :+1:


Add limit reach error

Need a error to indicate Request limit reached

@nj  Can I just merge this PR , as I need it for continue development

I close this PR , because it is from my account repo,   and I would like to move the code to extend_api branch too


Update: Amazon.AWSCLI version <IP_ADDRESS>

- [X] Have you signed the Contributor License Agreement?

[ ] Have you checked that there aren't other open pull requests for the same manifest update/change?

[ ] Have you tested your manifest locally with winget install --manifest <path>?
[ ] Does your manifest conform to the 1.0 schema?

Microsoft Reviewers: Open in CodeFlow

Leaving for #42616

https://github.com/microsoft/winget-pkgs/pull/42826#issuecomment-1023363375

/azp run


Added hovering effect to page #147 resolved

#147 issue is resolved

Thank you @SuhainaFathimaM that's looks nice


Build: Add Documentation PR type

Description

In order to keep things nicely organized
I was personally using the enhancement tag since it seemed to fit, but it had to be manually changed to docs so it would spam the changelog.
This will cover all XML documentation and website changes.
How Has This Been Tested?

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist

[X] The PR is submitted to the correct branch (dev).
[X] My code follows the code style of this project.
[ ] I've added relevant tests.

I have a branch with XML documentation changes I haven't PR'ed yet, if you need a PR to test this with.

I have a branch with XML documentation changes I haven't PR'ed yet, if you need a PR to test this with.

That would be great, thank you!

Thanks @danielchalmers

I have a branch with XML documentation changes I haven't PR'ed yet, if you need a PR to test this with.

@jperson2000 Thanks that would be great


Added support for schedules

Coverage decreased (-0.21%) to 80.77% when pulling a22569a276d0c91b9baca08ac5d09e638c5847e1 on jakub-plichta:master into c6beb654a437f56315e653272cd87f4051e97e05 on martiner:master.

It would be nice, if you separate the paging related things to independent commit

Coverage decreased (-0.13%) to 80.85% when pulling d1ab05c78ef0a7a1800f8cdc26de8b70fb9fd304 on jakub-plichta:master into c6beb654a437f56315e653272cd87f4051e97e05 on martiner:master.

[ ] acceptance tests in ShowcaseAT
[ ] update readme

Coverage decreased (-0.07%) to 79.41% when pulling f0b4c8fb992340fd08ca4b97fadfcb9562139827 on jakub-plichta:master into f1fa84bdad6253530108a4069ec9329bf87afcc4 on martiner:master.

Coverage decreased (-0.08%) to 79.62% when pulling e15b6f69d7b813dbb601e4185f9388fac11f189a on jakub-plichta:master into b9712b04e9089d589492462469c247cfd60526f1 on martiner:master.


Credit-card-validator.py

Hi, there this is my third PR in Hacktoberfest2022 in this is have created my project named Credit-card-validator in python.

@Dhruvgoyal21 are this PR/MR getting counted in Hacktoberfest 2022 see your profile weather it got excluded or got counted


SubMenu positioning

Hello!
I have a use case where I need to align a submenu (relative to the trigger MenuItem) starting from top when it fits into the screen (the default behaviour) and from the bottom when it doesn't, rather than pushing it a few pixels upwards (the behaviour I'm looking for). Is this implemented and if not, can it be added as a feature?
This is the behaviour right now:

This is what I'm looking  for:

*For clarification, in the second picture for each case, the bottom of the screenshot is where the bottom of the window was at.
React/React-dom version: 17.0.2
React-Menu version: 3.2.1

Hi @liviuclement , it was not implemented. The currently behaviour is the same as many submenu implementations.
For example, here is the native macOS menu:

When it cannot fit into screen from the bottom:

I'm curious to understand the rationale you wish to have a different behaviour.

You may try to set the shift prop which adjusts alignment of menu with its anchor element. https://szhsin.github.io/react-menu#menu-placement
Not sure your exact use case, but eventually there could be some JavaScript required in the user space to apply relevant props under different conditions.

Closing this issue as the current behaviour aligns with native OS menus, and there are no real benefits or use cases to implement it differently.


operator project-quay (3.9.0-nightly.20230525)

Add 3.9.0-nightly.20230525 to community operators

/merge possible

/merge possible


visualize labeled points in point cloud with different colors

Added a small piece of code to visualize points with different labels in the point cloud with different colors (10 for now, can be increased).
To use it, in the PointCloud2 plugin in RVIZ, set ColorTransformer to Intensity and then change Channel Name to label. Note this requires that label is added as an entry in the PointField[] in the sensor_msgs/PointCloud2 entry. All the points with the same label are assigned the same color. This is useful to visualize point clusters representing objects, ground plane with the same color.

I'll review this asap, but in the meantime I wanted to suggest some alternatives you could do without modification.
I believe that you could color by label already if it were a scalar index (an int) rather than a string by using the "rainbow coloring" feature. I'd have to try it with your specific point cloud, but it might be possible with the current features. However, it might not give as much control over the visualization as your patch.

yes, the label is an int.
Yeah, I tried using the rainbow coloring feature (although I didn't fully understand its implementation) and here are the two pictures showing point cloud using rainbow coloring feature and the new patch.
Using rainbow feature:

Using the patch:

After looking at the code and looking at your point cloud data, I think it is because the function getRainbowColor expects the channel's value to be within [0.0 - 1.0], see:
https://github.com/NamanCMU/rviz/blob/950af39b87e2e30eaa5f669b4dc5981d59a974f7/src/rviz/default_plugin/point_cloud_transformers.cpp#L53-L54
The other thing that I noticed is that if I select your pointcloud's points in rviz and look at the value for label, a few are normal integers, but others are large floating point numbers:

So, I don't know if it is your data or the fact that rainbow coloring expects the channel (in this case "label") to be a floating point number between 0 and 1. Either way, if you have control over your data's creation I'd advise trying out some different values for label and ensure you're putting the correct information into that channel for each point.
As for this patch, I don't think looking for a channel called "label" explicitly is very flexible. I'd rather see additional options for how rviz maps values in a channel to point colors than what's proposed here.

@wjwwood i needed something similar but w/ ability to name/alias labels + set colors so did that @ https://github.com/dispatch-ai/rviz-labeled-point-cloud. if you are interested i can submit it for merge.

@daizr pr's are always welcome. We'll do our best to merge it at some point, but at the very least others can see it and perhaps iterate on it.

I don't have access to your specific point cloud, @NamanCMU, but from what wjwwood mentioned it must be using UINT_MAX for some of its labels.
I suspect this is the reason why rainbow colouring wasn't sufficient, because that colouring scheme by default will auto-compute the range of intensity values, which it uses for the mapping of intensities onto the 0-to-1 rainbow range. Your range would have been very large, and so all labels around 0, e.g. 0-20, would have mapped to roughly the same colour the rainbow scale (red). If instead you deselect "Autocompute Intensity Bounds" in rviz and set the max intensity to the number of labels-1, you will be able to utilise the full rainbow range for your labels.
However, if the range of the labels is still large, then there won't be much colour distinction between similarly-numbered labels. At that point we can introduce additional properties for the Intensity colour transformer (such as modulo like you have incorporated in this PR), but that won't guarantee you colour distinction in space either (if labels 1 and 11 are next to each other), so we are probably better off considering an approach to colouring the label data that allows user modification, e.g. https://github.com/ros-visualization/rviz/pull/1162, which has since been proposed. You are welcome to try out that PR and give feedback, I suspect it will fit your use case nicely especially if you have high range in your labels.

I imagine that you have already resolved/worked around your issue, @NamanCMU, so I am going to close this PR. If other users encounter the same situation, I think it's possible to resolve without this PR. For data labeled integers, rainbow colouring should be sufficient, provided that "autocompute intensity bounds" is deselected.
If a use case comes up where the modulo approach to colouring is more appropriate than rainbow, this PR can be revived. Thanks again for proposal!


fix(@angular-devkit/schematics-cli): dryRun is not set by default when using a local collection

Fixes #12815

I added a story to move the parsing to core and use that in our other schematics.

@alan-agius4 There's a merge conflict with patch branch 7.0.x, can you please fix?

@kyliau, actually this should only target master. Updated the label


Update learning-administrator.md

@DaveBeasley
#sign-off

Closing this PR after verifying with Margo.


Fix suffix "+" for GKE cluster

Thanks for the fix, I just happened to be a bit faster than you :)
I already added the fix after testing it to the helper in commit


[BUG] Funcaptcha solving using 2Captcha

2Captcha Funcaptcha badly integrated
Whenever i try to solve funcaptcha using twocaptcha and openbullet am getting error of
[Executing block Solve Fun Captcha] TaskCreationException: ERROR_PAGEURL
To Reproduce
Steps to reproduce the behavior:

Go to 'stacker'
Get make funcaptcha function
Make sure u using twocaptcha
See error

Screenshots

Are you sure you're writing
https://urlwithcaptcha.com/

and not
https:/urlwithcaptcha.com/

?

Am not

I removed / accidentaly when making this bug report, it has nothin to do with it, it happens on new ob and also on old one

Am not

I don't understand this reply, you sent a screenshot with a malformed URL which is missing a / character. Can you add it and try again?

Am not

I don't understand this reply, you sent a screenshot with a malformed URL which is missing a / character. Can you add it and try again?

i fixed the screen, as i said it has nothin to do with it, it happens on old ob and also on this one, idk what can be the issue of this

Its weird, when i try using their api in py or smthing with site where am trying to solve it it works, but tru ob it doesnt, anyway i tried another domain in ob and it worked so idk what can be the issue of this

If the issue is only present on a specific site then there is no way I can debug this without having the site, maybe consider sending me an email.

sure, goin to do that

Email sent, thanks !

If you remove the www and make it https://sitewithcaptcha.com it works in our tests. I don't know what the issue may be, but it's coming from 2captcha so this is not an OB2 related issue. OB2 is just sending exactly what you put, and 2captcha is saying it's invalid.


The code

When will the codes be uploaded?

can you please upload the code

When will the codes be uploaded?

When will the codes be uploaded?

Thanks for your interest. The code has been uploaded and please let me know if you have further questions.

Thank you for uploading the code.


Crashing/Aimbot issue

I guess the author of this is not into making changes to this anymore, as it's open source and a little old, but here are some problems that I can't fix as I never studied coding.
 I noticed that the crash is more likely with aimbot on, sometimes I barely use it and it crashes.
Suddenly: after some time of gameplay, it just closes. But I noticed that it tends to happen more towards the first minutes of a match, and if you get to play like 60% of the match, it's less likely that it's going to crash before the end. But for the next match, the chances are again increased until mid-game. (These are just some observations regarding 'when' it happens)

The log files are attached, and as you'll notice, one of then contains no error message, but it is too the log file for a crashed session, this makes me think that there's an unknown error going on and the program is not prepared to call it.

 So I will point some failures and things to add or improve (regarding the aimbot), for those who know programming and haven't played enough to notice these problems.

The correction based on the bullet drop is faulty, it points above the target in situations where it shouldn't. In practice, this could be corrected by selecting a bone that sits lower in the body, but then it doesn't work properly, as those bones are covered by obstacles more often. In conclusion -generated with 0 knowledge and just guessing, I think the problem is associated with the math/poly_solver/w2s files. I changed the resolution from 1440p to 1080p (i dont remember where it is anymore) in the source code, but didn't correct the problem, although, with the possibly of placebo effect, it seems more consistent. Maybe it's calculating the bullet speed wrong?! Idk
I think the aimbot deals badly with obstacles in practice, small boxes and windows are the nightmare for the functionality.
edit: forgot to mention that the aimbot also don't work well if your altitude is too different from the enemy, so, high ground kills are rarely going to happen with the aimbot enabled

This cheat is intended for legit gameplay, the aimbot code is not mine. I've added it just so people stop asking for it.
Please look at other cheats with aimbot.
Also if you looked at default aimbot settings, it is configured to act like aim assist since most servers will get you banned for obvious aimbotting.


Update setup.md

Update setup.md

Anything in particular you find lacking, maybe replace vagrant with Docker? I've kept the conda instructions up to date.


Fixes DEVTOOLING-851 and DEVTOOLING-848 for genesyscloud_telephony_providers_edges_site_outbound_route

Fixes:

DEVTOOLING-851 - genesyscloud_telephony_providers_edges_site_outbound_routes resource should be handling individual routes
DEVTOOLING-848 - Adds a data resource for genesyscloud_telephony_providers_edges_site_outbound_route and adds unit tests

I'm going to update this PR with some fixes required for restructuring the genesyscloud_telephony_providers_edges_site_outbound_route resource as discussed earlier with @dginty4 .

@charliecon @carnellj-genesys @dginty4 this PR is ready for review with all tests passing


Ast, Parser, and Lexer for DSL (v2)

This branch has the development environment set up properly. Please look at the code here from now on and disregard #24. Currently, dune build works and I have a way to test small programs. I'm starting with a bare bones version of the DSL with only policies, trying to get that to work, and then I'll add on more features. To run, first type
dune build

To test a small sample program, first cd to dsl_lang folder, then run
dune utop lib

In utop, run
open Dsl_core;;

This last command must be run each time after opening utop. Finally, to test a sample program, run
Dsl_core.Util.parse_string "<my prog>";;

Also credit to @KabirSamsi for helping me to set up the environment and writing this method of testing!

Exciting stuff Cassandra! In case you wonder, I just went over and added a .gitignore so that we weren't tracking hundreds of meta files that live under _build. I am trying to recreate your test but I am unsure about the last line,
Dsl_core.Util.parse_string "<my prog>";;

While in the directory dsl_lang, I tried:
utop # Dsl_core.Util.parse_string "../dsl/progs/now/drop_a_class.sched";;
Error at line 1
Unrecognized character: [.]

and some variants like that. I always just get told that the first character of the name I passed is unrecognized.
Could you please tell us the exact command that you are running?

Hi, I probably should have been more specific that  is not a path to a file, but just a line from a program like "fifo[A, B]", (which currently does not work). Once I get some functionality, I want to be able to pass in a file, but that's currently not supported.

Update, I think I've gotten policies to parse, such as

But I cannot push because I'm facing a nasty merge conflict, so if you try this string now it will fail.

Understood, and makes a lot of sense! Sorry to jump the gun and assume the file thing was working haha. This string version is a fantastic start. I’d be happy to work through the merge conflict with you on Monday!

@csziklai re. our discussion on Thursday – I have code for lexing & parsing files to ASTs. I can help you set that up on Monday if you'd like!

@KabirSamsi Yes that'd be great! Also @anshumanmohan I think I figured out the merge conflict hopefully!

@csziklai there is a silly little problem: there is a directory called _build where a lot of temporary files are stored. Its contents will change very noisily every time you run dune build and similar commands. The right move is to add the _build directory to the .gitignore, which I have done. However, since you have been tracking that directory since before the directory was in the .gitignore, your local git client is still tracking lots of files and pushing them every time.
 But please make sure that you have fully committed and pushed before you do this! Like, you should see the following when you do git status:
$ git status
On branch test_env
Your branch is up to date with 'origin/test_env'.

nothing to commit, working tree clean

Only then should you do what the SO answer says.

@csziklai I have a suggestion re: assignments. For this PR, I suggest you just drop them.
That is, the user will not be able to write
p = Strict(A,B),
return RR(p, C)

Instead, the user will be forced to write
return RR(Strict(A,B), C)

I'm not sure the version you have in the AST is going to work, and I will make a separate issue where I'll suggest the fix! Happy to also discuss in person when things are feeling a little more stable with the simple version!

I've gotten declaring classes, policies, and return expressions to work. So now we can parse programs such as
classes A,B; return rr[A, B]

and
classes CU, CV, CW, CX; return rr[rr[CU, CV], strict[CW, CX]]

The output of parsing the above examples is screenshotted below.

I'm going to go ahead and take this from draft to regular PR and create a new branch to add variable assignment.