sometimes consumer have no partition assigned to it after rebalance

i'm try latest 1.12.1 and older 1.4.2 both have the same effect
kafka 2.4 and i cant upgrade it

i have 48-60 partitions in topic and all pods consumes messages.
i have 48-60 pods
i'm run helm upgrade to all pods that consumes from this topic
most of the pods consumes successful but some of them have no partitions assigned
picture looks like 40 partitions have consumers, and some partitions does not have any consumers.
my debug on says that this consumer id does not have any partition assigments
if i'm restart such pods - all works fine

im add debug to OnAssigned
and saw
{"level":"debug","@timestamp":"2023-02-21T15:17:57.574+0300","caller":"broker-kgo@v0.2.8/util.go:85","msg":"[kgo] assigned map[string][]int32{}"}
so consumer have no partitions...
logs from leader contains this messages
prod-3ec4a5c0-8dc5-417d-ae63-a3d3e33dbb8d{notify.v3[6]}, prod-5a8523a0-e0ff-4086-af95-78f680707a11{notify.v3[16]}, prod-833b1385-4433-43e6-95fd-5d22c53fe36a{notify.v3[24 25]}, prod-bec4095c-8fff-406e-bab0-888b274b7599{notify.v3[13 40]}, prod-dc5f85f6-5f52-4a21-9cfd-ef8229a539ea{notify.v3[8 37]}, prod-08cdbfb5-901d-4585-b6c6-5a3cdad6dd7d{notify.v3[1]}, prod-377b682e-335c-4bab-974f-25b2467bd45f{notify.v3[19]}, prod-ba944249-fbc5-46c3-a57c-7edb9dcbe25a{}, prod-c038aab5-46da-4d0b-be62-78476ce5f2f4{}, prod-d94cdb0d-77ce-4673-81b8-6d1417ebb99f{notify.v3[15 34]}, prod-fcd97f8e-d213-4d0e-b593-79db0bfb1204{}, prod-21324d70-b1fc-44b5-91e2-d493d618b02a{}, prod-4b988e95-184f-413e-b6db-a1222cfa0a21{}, prod-6fc9619a-bd9f-4c1b-9043-f19e0d63a6b5{notify.v3[20 42]}, prod-d00841bb-2e4e-4a8c-a6f4-1d0b5eb7003d{}, prod-ded4db59-deb9-49b9-97ab-b04f459acde2{notify.v3[10 39]}, prod-49321a12-bd30-484d-9d35-159d693e95d7{notify.v3[4]}, prod-e8b6365b-75c4-4c3c-90be-259a9eac449d{}, prod-575c17bc-d78a-4ad7-aef9-1a38b332a916{}, prod-7e0c4c97-15b7-4847-873e-e4eafa964ec6{notify.v3[11 12]}, prod-83dfaaec-e9af-4a6c-bff4-c28c3a47b628{notify.v3[3 31]}, prod-b8fb26fc-ecdb-4d17-a5b0-b6ba971be975{}, prod-cfe31e81-952a-42be-b1e0-ea5a31d238fc{}, prod-8c2abbc6-8f0b-464a-9268-fd977eb6d7af{notify.v3[7 30]}, prod-e65fbff2-bee1-44bd-b651-e1b47bbed3a9{notify.v3[17 44]}, prod-f9b53533-0bff-403f-b3ed-8c12711a17d4{notify.v3[21 47]}, prod-26e5bbe3-645e-4fa2-bcd5-82f93ccd7c40{}, prod-274d1555-e49c-42d3-bbfd-8c3ef5886d56{notify.v3[2]}, prod-35ee4b9e-1cf5-4415-b5fa-941fdea56158{notify.v3[9]}, prod-3ad691d8-1311-4cfe-8705-7488c62db865{notify.v3[23]}, prod-7de0b455-683e-4c79-9e62-9f93cd5a8170{notify.v3[22 46]}, prod-02ece2b4-28d5-43bc-8921-746b089c0128{notify.v3[0]}, prod-0b294f5c-9f78-4566-856b-39bc8e255912{notify.v3[18]}, prod-58698f86-8bff-4d24-a4fd-e9cc8049771b{notify.v3[14]}, prod-9508e2aa-96c5-491c-9ae1-b0a92bc0dab9{notify.v3[5 35]}

as you see many consumers does not have any assigned partitions....
between successful case then all consumers have assigned partitions and logs from above - spent only 1 minute.

i can create some workaround for such case like OnAssigned check how much partitions assigned to consumer and if no - restart pod, but this is looks very ugly.

What is the full log from the leader?

And the few logs before that -- particularly, log lines that contain "interests".

i dont have any lines contains word interests

Feb 21, 2023 @ 15:14:54.270	{"level":"info","@timestamp":"2023-02-21T15:14:54.270+0300","caller":"logger/logger.go:108","msg":"[kgo] connected to to kafka cluster version unknown custom version at least v2.7"}
Feb 21, 2023 @ 15:14:54.270	{"level":"info","ts":1676981694.2704318,"caller":"logger/zap.go:138","msg":"Broker [kgo] Connected to vm-kafka-ump04p.mbrd.ru:9092,vm-kafka-ump02p.mbrd.ru:9092,vm-kafka-ump05p.mbrd.ru:9092,vm-kafka-ump03p.mbrd.ru:9092,vm-kafka-ump01p.mbrd.ru:9092,vm-kafka-ump06p.mbrd.ru:9092"}
Feb 21, 2023 @ 15:14:54.270	{"level":"info","ts":1676981694.270906,"caller":"logger/zap.go:138","msg":"Register [memory] Registering node: pullers-553328c9-1c7e-4fa7-a503-e88474d619cd"}
Feb 21, 2023 @ 15:14:54.271	{"level":"info","ts":1676981694.2715344,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"beginning autocommit loop","group":"pullstandby"}
Feb 21, 2023 @ 15:14:54.271	{"level":"info","ts":1676981694.271024,"caller":"logger/zap.go:138","msg":"Subscribing to topic: notify.v3"}
Feb 21, 2023 @ 15:14:54.271	{"level":"info","ts":1676981694.2717695,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"immediate metadata update triggered","why":"querying metadata for consumer initialization"}
Feb 21, 2023 @ 15:14:54.333	{"level":"info","ts":1676981694.3337176,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"beginning to manage the group lifecycle","group":"pullstandby"}
Feb 21, 2023 @ 15:14:54.333	{"level":"info","ts":1676981694.333772,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"joining group","group":"pullstandby"}
Feb 21, 2023 @ 15:14:54.365	{"level":"info","ts":1676981694.3655846,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"join returned MemberIDRequired, rejoining with response's MemberID","group":"pullstandby"}
Feb 21, 2023 @ 15:14:57.257	{"level":"info","ts":1676981697.2577465,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"joined","group":"pullstandby","member_id":"prod-08cdbfb5-901d-4585-b6c6-5a3cdad6dd7d","instance_id":"<nil>"}
Feb 21, 2023 @ 15:14:57.257	{"level":"info","ts":1676981697.257798,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"syncing","group":"pullstandby","protocol_type":"consumer"}
Feb 21, 2023 @ 15:14:57.267	{"level":"info","ts":1676981697.2673867,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"synced","group":"pullstandby"}
Feb 21, 2023 @ 15:14:57.267	{"level":"info","ts":1676981697.2675006,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"beginning heartbeat loop","group":"pullstandby"}
Feb 21, 2023 @ 15:15:33.271	{"level":"info","ts":1676981733.2709644,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"assigning partitions","why":"revoking all assignments because we are not cooperative","how":"unassigning everything"}
Feb 21, 2023 @ 15:15:33.271	{"level":"info","ts":1676981733.2710292,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"immediate metadata update triggered","why":"waitmeta after heartbeat error"}
Feb 21, 2023 @ 15:15:33.271	{"level":"info","ts":1676981733.2708793,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"heartbeat errored","group":"pullstandby"}
Feb 21, 2023 @ 15:15:33.271	{"level":"info","ts":1676981733.270993,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"eager consumer revoking prior assigned partitions","group":"pullstandby"}
Feb 21, 2023 @ 15:15:33.281	{"level":"info","ts":1676981733.2810307,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"joining group","group":"pullstandby"}
Feb 21, 2023 @ 15:16:12.285	{"level":"info","ts":1676981772.284892,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"syncing","group":"pullstandby","protocol_type":"consumer"}
Feb 21, 2023 @ 15:16:12.285	{"level":"info","ts":1676981772.28486,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"joined","member_id":"prod-08cdbfb5-901d-4585-b6c6-5a3cdad6dd7d","instance_id":"<nil>","group":"pullstandby"}
Feb 21, 2023 @ 15:16:57.284	{"level":"info","ts":1676981817.2846646,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"sync failed with RebalanceInProgress, rejoining","group":"pullstandby"}
Feb 21, 2023 @ 15:16:57.304	{"level":"info","ts":1676981817.30417,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-35ee4b9e-1cf5-4415-b5fa-941fdea56158"}
Feb 21, 2023 @ 15:16:57.304	{"level":"info","ts":1676981817.3042185,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-58698f86-8bff-4d24-a4fd-e9cc8049771b"}
Feb 21, 2023 @ 15:16:57.304	{"level":"info","ts":1676981817.304335,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-e65fbff2-bee1-44bd-b651-e1b47bbed3a9"}
Feb 21, 2023 @ 15:16:57.304	{"level":"info","ts":1676981817.304017,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"joined, balancing group","group":"pullstandby","member_id":"prod-08cdbfb5-901d-4585-b6c6-5a3cdad6dd7d","instance_id":"<nil>"}
Feb 21, 2023 @ 15:16:57.304	{"level":"info","ts":1676981817.3042026,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-3ec4a5c0-8dc5-417d-ae63-a3d3e33dbb8d"}
Feb 21, 2023 @ 15:16:57.304	{"level":"info","ts":1676981817.3043272,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-ded4db59-deb9-49b9-97ab-b04f459acde2"}
Feb 21, 2023 @ 15:16:57.304	{"level":"info","ts":1676981817.3041825,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-377b682e-335c-4bab-974f-25b2467bd45f"}
Feb 21, 2023 @ 15:16:57.304	{"level":"info","ts":1676981817.3044817,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"syncing","protocol_type":"consumer","group":"pullstandby"}
Feb 21, 2023 @ 15:16:57.304	{"level":"info","ts":1676981817.3041124,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-02ece2b4-28d5-43bc-8921-746b089c0128"}
Feb 21, 2023 @ 15:16:57.304	{"level":"info","ts":1676981817.3040483,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balancing group as leader"}
Feb 21, 2023 @ 15:16:57.304	{"level":"info","ts":1676981817.3041382,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-0b294f5c-9f78-4566-856b-39bc8e255912"}
Feb 21, 2023 @ 15:16:57.304	{"level":"info","ts":1676981817.3042254,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-5a8523a0-e0ff-4086-af95-78f680707a11"}
Feb 21, 2023 @ 15:16:57.304	{"level":"info","ts":1676981817.304345,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-f9b53533-0bff-403f-b3ed-8c12711a17d4"}
Feb 21, 2023 @ 15:16:57.304	{"level":"info","ts":1676981817.3044434,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balanced","plan":"prod-08cdbfb5-901d-4585-b6c6-5a3cdad6dd7d{notify.v3[1 32]}, prod-274d1555-e49c-42d3-bbfd-8c3ef5886d56{notify.v3[2 33]}, prod-49321a12-bd30-484d-9d35-159d693e95d7{notify.v3[4 27]}, prod-ded4db59-deb9-49b9-97ab-b04f459acde2{notify.v3[10 39]}, prod-bec4095c-8fff-406e-bab0-888b274b7599{notify.v3[13 40]}, prod-35ee4b9e-1cf5-4415-b5fa-941fdea56158{notify.v3[9 36]}, prod-3ad691d8-1311-4cfe-8705-7488c62db865{notify.v3[23 41]}, prod-58698f86-8bff-4d24-a4fd-e9cc8049771b{notify.v3[14 38]}, prod-7de0b455-683e-4c79-9e62-9f93cd5a8170{notify.v3[22 46]}, prod-7e0c4c97-15b7-4847-873e-e4eafa964ec6{notify.v3[11 12]}, prod-833b1385-4433-43e6-95fd-5d22c53fe36a{notify.v3[24 25]}, prod-0b294f5c-9f78-4566-856b-39bc8e255912{notify.v3[18 43]}, prod-377b682e-335c-4bab-974f-25b2467bd45f{notify.v3[19 45]}, prod-3ec4a5c0-8dc5-417d-ae63-a3d3e33dbb8d{notify.v3[6 29]}, prod-6fc9619a-bd9f-4c1b-9043-f19e0d63a6b5{notify.v3[20 42]}, prod-9508e2aa-96c5-491c-9ae1-b0a92bc0dab9{notify.v3[5 35]}, prod-f9b53533-0bff-403f-b3ed-8c12711a17d4{notify.v3[21 47]}, prod-e65fbff2-bee1-44bd-b651-e1b47bbed3a9{notify.v3[17 44]}, prod-02ece2b4-28d5-43bc-8921-746b089c0128{notify.v3[0 28]}, prod-5a8523a0-e0ff-4086-af95-78f680707a11{notify.v3[16 26]}, prod-83dfaaec-e9af-4a6c-bff4-c28c3a47b628{notify.v3[3 31]}, prod-8c2abbc6-8f0b-464a-9268-fd977eb6d7af{notify.v3[7 30]}, prod-d94cdb0d-77ce-4673-81b8-6d1417ebb99f{notify.v3[15 34]}, prod-dc5f85f6-5f52-4a21-9cfd-ef8229a539ea{notify.v3[8 37]}"}
Feb 21, 2023 @ 15:16:57.304	{"level":"info","ts":1676981817.3041494,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-274d1555-e49c-42d3-bbfd-8c3ef5886d56"}
Feb 21, 2023 @ 15:16:57.304	{"level":"info","ts":1676981817.3042834,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-8c2abbc6-8f0b-464a-9268-fd977eb6d7af"}
Feb 21, 2023 @ 15:16:57.304	{"level":"info","ts":1676981817.30429,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-9508e2aa-96c5-491c-9ae1-b0a92bc0dab9"}
Feb 21, 2023 @ 15:16:57.304	{"level":"info","ts":1676981817.3042755,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-83dfaaec-e9af-4a6c-bff4-c28c3a47b628"}
Feb 21, 2023 @ 15:16:57.304	{"level":"info","ts":1676981817.3042355,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-6fc9619a-bd9f-4c1b-9043-f19e0d63a6b5"}
Feb 21, 2023 @ 15:16:57.304	{"level":"info","ts":1676981817.304317,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-dc5f85f6-5f52-4a21-9cfd-ef8229a539ea"}
Feb 21, 2023 @ 15:16:57.304	{"level":"info","ts":1676981817.30419,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-3ad691d8-1311-4cfe-8705-7488c62db865"}
Feb 21, 2023 @ 15:16:57.304	{"level":"info","ts":1676981817.3043098,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-d94cdb0d-77ce-4673-81b8-6d1417ebb99f"}
Feb 21, 2023 @ 15:16:57.304	{"level":"info","ts":1676981817.3042095,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-49321a12-bd30-484d-9d35-159d693e95d7"}
Feb 21, 2023 @ 15:16:57.304	{"level":"info","ts":1676981817.3041267,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-08cdbfb5-901d-4585-b6c6-5a3cdad6dd7d"}
Feb 21, 2023 @ 15:16:57.304	{"level":"info","ts":1676981817.304266,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-833b1385-4433-43e6-95fd-5d22c53fe36a"}
Feb 21, 2023 @ 15:16:57.304	{"level":"info","ts":1676981817.3042986,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-bec4095c-8fff-406e-bab0-888b274b7599"}
Feb 21, 2023 @ 15:16:57.304	{"level":"info","ts":1676981817.3042552,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-7e0c4c97-15b7-4847-873e-e4eafa964ec6"}
Feb 21, 2023 @ 15:16:57.304	{"level":"info","ts":1676981817.3042426,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-7de0b455-683e-4c79-9e62-9f93cd5a8170"}
Feb 21, 2023 @ 15:16:57.347	{"level":"info","ts":1676981817.3467789,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"synced","group":"pullstandby"}
Feb 21, 2023 @ 15:16:57.347	{"level":"info","ts":1676981817.3469324,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"beginning heartbeat loop","group":"pullstandby"}
Feb 21, 2023 @ 15:16:57.359	{"level":"info","ts":1676981817.3595927,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"assigning partitions","why":"newly fetched offsets for group pullstandby","how":"assigning everything new, keeping current assignment"}
Feb 21, 2023 @ 15:16:57.359	{"level":"info","ts":1676981817.359679,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"metadata update triggered","why":"loading offsets in new session from assign"}
Feb 21, 2023 @ 15:16:57.450	{"level":"info","ts":1676981817.4502654,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"producing to a new topic for the first time, fetching metadata to learn its partitions","topic":"deposit.v3"}
Feb 21, 2023 @ 15:16:57.450	{"level":"info","ts":1676981817.4502497,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"immediate metadata update triggered","why":"forced load because we are producing to a new topic for the first time"}
Feb 21, 2023 @ 15:16:57.465	{"level":"info","ts":1676981817.4653776,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"done waiting for metadata for new topic","topic":"deposit.v3"}
Feb 21, 2023 @ 15:16:57.485	{"level":"info","ts":1676981817.4654515,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"skipping producer id initialization because the client was configured to disable idempotent writes"}
Feb 21, 2023 @ 15:16:57.526	{"level":"info","ts":1676981817.526676,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"producing to a new topic for the first time, fetching metadata to learn its partitions","topic":"loanind.v3"}
Feb 21, 2023 @ 15:16:57.526	{"level":"info","ts":1676981817.5263426,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"immediate metadata update triggered","why":"forced load because we are producing to a new topic for the first time"}
Feb 21, 2023 @ 15:16:57.535	{"level":"info","ts":1676981817.5350637,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"done waiting for metadata for new topic","topic":"loanind.v3"}
Feb 21, 2023 @ 15:17:51.349	{"level":"info","ts":1676981871.349017,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"assigning partitions","why":"revoking all assignments because we are not cooperative","how":"unassigning everything"}
Feb 21, 2023 @ 15:17:51.349	{"level":"info","ts":1676981871.3489091,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"heartbeat errored","group":"pullstandby"}
Feb 21, 2023 @ 15:17:51.349	{"level":"info","ts":1676981871.3494205,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"eager consumer revoking prior assigned partitions","group":"pullstandby"}
Feb 21, 2023 @ 15:17:51.349	{"level":"info","ts":1676981871.349047,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"immediate metadata update triggered","why":"waitmeta after heartbeat error"}
Feb 21, 2023 @ 15:17:51.354	{"level":"info","ts":1676981871.3544266,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"joining group","group":"pullstandby"}
Feb 21, 2023 @ 15:17:51.385	{"level":"info","ts":1676981871.3858058,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"joined, balancing group","group":"pullstandby","member_id":"prod-08cdbfb5-901d-4585-b6c6-5a3cdad6dd7d","instance_id":"<nil>"}
Feb 21, 2023 @ 15:17:51.385	{"level":"info","ts":1676981871.385856,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balancing group as leader"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.3860447,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-35ee4b9e-1cf5-4415-b5fa-941fdea56158"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.3860624,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-377b682e-335c-4bab-974f-25b2467bd45f"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.3861804,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-7de0b455-683e-4c79-9e62-9f93cd5a8170"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.3863373,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-d94cdb0d-77ce-4673-81b8-6d1417ebb99f"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.3859587,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-02ece2b4-28d5-43bc-8921-746b089c0128"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.386253,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-8c2abbc6-8f0b-464a-9268-fd977eb6d7af"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.3862703,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-9508e2aa-96c5-491c-9ae1-b0a92bc0dab9"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.386307,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-bec4095c-8fff-406e-bab0-888b274b7599"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.3863227,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-cfe31e81-952a-42be-b1e0-ea5a31d238fc"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.3863907,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-e65fbff2-bee1-44bd-b651-e1b47bbed3a9"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.3864048,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-f9b53533-0bff-403f-b3ed-8c12711a17d4"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.3859935,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-08cdbfb5-901d-4585-b6c6-5a3cdad6dd7d"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.3860722,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-3ad691d8-1311-4cfe-8705-7488c62db865"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.3860955,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-49321a12-bd30-484d-9d35-159d693e95d7"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.3861215,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-575c17bc-d78a-4ad7-aef9-1a38b332a916"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.38616,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-5a8523a0-e0ff-4086-af95-78f680707a11"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.3862925,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-b8fb26fc-ecdb-4d17-a5b0-b6ba971be975"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.3862998,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-ba944249-fbc5-46c3-a57c-7edb9dcbe25a"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.3863294,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-d00841bb-2e4e-4a8c-a6f4-1d0b5eb7003d"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.3863976,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-e8b6365b-75c4-4c3c-90be-259a9eac449d"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.3865516,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balanced","plan":"prod-3ec4a5c0-8dc5-417d-ae63-a3d3e33dbb8d{notify.v3[6]}, prod-5a8523a0-e0ff-4086-af95-78f680707a11{notify.v3[16]}, prod-833b1385-4433-43e6-95fd-5d22c53fe36a{notify.v3[24 25]}, prod-bec4095c-8fff-406e-bab0-888b274b7599{notify.v3[13 40]}, prod-dc5f85f6-5f52-4a21-9cfd-ef8229a539ea{notify.v3[8 37]}, prod-08cdbfb5-901d-4585-b6c6-5a3cdad6dd7d{notify.v3[1]}, prod-377b682e-335c-4bab-974f-25b2467bd45f{notify.v3[19]}, prod-ba944249-fbc5-46c3-a57c-7edb9dcbe25a{}, prod-c038aab5-46da-4d0b-be62-78476ce5f2f4{}, prod-d94cdb0d-77ce-4673-81b8-6d1417ebb99f{notify.v3[15 34]}, prod-fcd97f8e-d213-4d0e-b593-79db0bfb1204{}, prod-21324d70-b1fc-44b5-91e2-d493d618b02a{}, prod-4b988e95-184f-413e-b6db-a1222cfa0a21{}, prod-6fc9619a-bd9f-4c1b-9043-f19e0d63a6b5{notify.v3[20 42]}, prod-d00841bb-2e4e-4a8c-a6f4-1d0b5eb7003d{}, prod-ded4db59-deb9-49b9-97ab-b04f459acde2{notify.v3[10 39]}, prod-49321a12-bd30-484d-9d35-159d693e95d7{notify.v3[4]}, prod-e8b6365b-75c4-4c3c-90be-259a9eac449d{}, prod-575c17bc-d78a-4ad7-aef9-1a38b332a916{}, prod-7e0c4c97-15b7-4847-873e-e4eafa964ec6{notify.v3[11 12]}, prod-83dfaaec-e9af-4a6c-bff4-c28c3a47b628{notify.v3[3 31]}, prod-b8fb26fc-ecdb-4d17-a5b0-b6ba971be975{}, prod-cfe31e81-952a-42be-b1e0-ea5a31d238fc{}, prod-8c2abbc6-8f0b-464a-9268-fd977eb6d7af{notify.v3[7 30]}, prod-e65fbff2-bee1-44bd-b651-e1b47bbed3a9{notify.v3[17 44]}, prod-f9b53533-0bff-403f-b3ed-8c12711a17d4{notify.v3[21 47]}, prod-26e5bbe3-645e-4fa2-bcd5-82f93ccd7c40{}, prod-274d1555-e49c-42d3-bbfd-8c3ef5886d56{notify.v3[2]}, prod-35ee4b9e-1cf5-4415-b5fa-941fdea56158{notify.v3[9]}, prod-3ad691d8-1311-4cfe-8705-7488c62db865{notify.v3[23]}, prod-7de0b455-683e-4c79-9e62-9f93cd5a8170{notify.v3[22 46]}, prod-02ece2b4-28d5-43bc-8921-746b089c0128{notify.v3[0]}, prod-0b294f5c-9f78-4566-856b-39bc8e255912{notify.v3[18]}, prod-58698f86-8bff-4d24-a4fd-e9cc8049771b{notify.v3[14]}, prod-9508e2aa-96c5-491c-9ae1-b0a92bc0dab9{notify.v3[5 35]}"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.3866107,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"syncing","group":"pullstandby","protocol_type":"consumer"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.386002,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-0b294f5c-9f78-4566-856b-39bc8e255912"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.3860116,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-21324d70-b1fc-44b5-91e2-d493d618b02a"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.3860838,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-3ec4a5c0-8dc5-417d-ae63-a3d3e33dbb8d"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.386146,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-58698f86-8bff-4d24-a4fd-e9cc8049771b"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.3861706,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-6fc9619a-bd9f-4c1b-9043-f19e0d63a6b5"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.3861935,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-7e0c4c97-15b7-4847-873e-e4eafa964ec6"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.3862157,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-833b1385-4433-43e6-95fd-5d22c53fe36a"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.3864126,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-fcd97f8e-d213-4d0e-b593-79db0bfb1204"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.3860204,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-26e5bbe3-645e-4fa2-bcd5-82f93ccd7c40"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.3860323,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-274d1555-e49c-42d3-bbfd-8c3ef5886d56"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.3861074,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-4b988e95-184f-413e-b6db-a1222cfa0a21"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.3862393,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-83dfaaec-e9af-4a6c-bff4-c28c3a47b628"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.386315,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-c038aab5-46da-4d0b-be62-78476ce5f2f4"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.3863697,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-dc5f85f6-5f52-4a21-9cfd-ef8229a539ea"}
Feb 21, 2023 @ 15:17:51.386	{"level":"info","ts":1676981871.386383,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-ded4db59-deb9-49b9-97ab-b04f459acde2"}
Feb 21, 2023 @ 15:17:51.397	{"level":"info","ts":1676981871.3976204,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"synced","group":"pullstandby"}
Feb 21, 2023 @ 15:17:51.398	{"level":"info","ts":1676981871.397773,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"beginning heartbeat loop","group":"pullstandby"}
Feb 21, 2023 @ 15:17:51.400	{"level":"info","ts":1676981871.400127,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"assigning partitions","how":"assigning everything new, keeping current assignment","why":"newly fetched offsets for group pullstandby"}
Feb 21, 2023 @ 15:17:54.400	{"level":"info","ts":1676981874.4004445,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"heartbeat errored","group":"pullstandby"}
Feb 21, 2023 @ 15:17:54.400	{"level":"info","ts":1676981874.400545,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"assigning partitions","why":"revoking all assignments because we are not cooperative","how":"unassigning everything"}
Feb 21, 2023 @ 15:17:54.400	{"level":"info","ts":1676981874.4005287,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"immediate metadata update triggered","why":"waitmeta after heartbeat error"}
Feb 21, 2023 @ 15:17:54.400	{"level":"info","ts":1676981874.4006095,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"eager consumer revoking prior assigned partitions","group":"pullstandby"}
Feb 21, 2023 @ 15:17:54.404	{"level":"info","ts":1676981874.404544,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"joining group","group":"pullstandby"}
Feb 21, 2023 @ 15:17:54.460	{"level":"info","ts":1676981874.4604058,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"joined, balancing group","group":"pullstandby","member_id":"prod-08cdbfb5-901d-4585-b6c6-5a3cdad6dd7d","instance_id":"<nil>"}
Feb 21, 2023 @ 15:17:54.460	{"level":"info","ts":1676981874.4604352,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balancing group as leader"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4614685,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-e072e17c-73be-4501-8c0d-18d08c0767b3"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.460966,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-274d1555-e49c-42d3-bbfd-8c3ef5886d56"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4611008,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-5a8523a0-e0ff-4086-af95-78f680707a11"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.461164,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-833b1385-4433-43e6-95fd-5d22c53fe36a"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4612005,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-83dfaaec-e9af-4a6c-bff4-c28c3a47b628"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4613202,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-bec4095c-8fff-406e-bab0-888b274b7599"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4609036,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-08cdbfb5-901d-4585-b6c6-5a3cdad6dd7d"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4609542,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-26e5bbe3-645e-4fa2-bcd5-82f93ccd7c40"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4613624,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-cfe31e81-952a-42be-b1e0-ea5a31d238fc"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4611216,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-7de0b455-683e-4c79-9e62-9f93cd5a8170"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4617505,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"syncing","group":"pullstandby","protocol_type":"consumer"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4612508,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-8c2abbc6-8f0b-464a-9268-fd977eb6d7af"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4609933,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-35ee4b9e-1cf5-4415-b5fa-941fdea56158"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4611537,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-7e0c4c97-15b7-4847-873e-e4eafa964ec6"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4613745,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-d00841bb-2e4e-4a8c-a6f4-1d0b5eb7003d"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4610183,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-3ec4a5c0-8dc5-417d-ae63-a3d3e33dbb8d"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4612792,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-b8fb26fc-ecdb-4d17-a5b0-b6ba971be975"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4610915,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-58698f86-8bff-4d24-a4fd-e9cc8049771b"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4615252,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-fcd97f8e-d213-4d0e-b593-79db0bfb1204"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4610007,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-377b682e-335c-4bab-974f-25b2467bd45f"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.461484,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-e8b6365b-75c4-4c3c-90be-259a9eac449d"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4610837,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-575c17bc-d78a-4ad7-aef9-1a38b332a916"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4616852,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balanced","plan":"prod-377b682e-335c-4bab-974f-25b2467bd45f{notify.v3[19]}, prod-833b1385-4433-43e6-95fd-5d22c53fe36a{notify.v3[24 25]}, prod-83dfaaec-e9af-4a6c-bff4-c28c3a47b628{notify.v3[3 31]}, prod-c038aab5-46da-4d0b-be62-78476ce5f2f4{notify.v3[43]}, prod-cfe31e81-952a-42be-b1e0-ea5a31d238fc{notify.v3[45]}, prod-d00841bb-2e4e-4a8c-a6f4-1d0b5eb7003d{}, prod-26e5bbe3-645e-4fa2-bcd5-82f93ccd7c40{notify.v3[32]}, prod-21324d70-b1fc-44b5-91e2-d493d618b02a{notify.v3[26]}, prod-5a8523a0-e0ff-4086-af95-78f680707a11{notify.v3[16]}, prod-6fc9619a-bd9f-4c1b-9043-f19e0d63a6b5{notify.v3[20]}, prod-02ece2b4-28d5-43bc-8921-746b089c0128{notify.v3[0]}, prod-49321a12-bd30-484d-9d35-159d693e95d7{notify.v3[4]}, prod-4b988e95-184f-413e-b6db-a1222cfa0a21{notify.v3[36]}, prod-ded4db59-deb9-49b9-97ab-b04f459acde2{notify.v3[10 39]}, prod-3ad691d8-1311-4cfe-8705-7488c62db865{notify.v3[23]}, prod-bec4095c-8fff-406e-bab0-888b274b7599{notify.v3[13 40]}, prod-d94cdb0d-77ce-4673-81b8-6d1417ebb99f{notify.v3[15 34]}, prod-e8b6365b-75c4-4c3c-90be-259a9eac449d{notify.v3[28]}, prod-3ec4a5c0-8dc5-417d-ae63-a3d3e33dbb8d{notify.v3[6]}, prod-dc5f85f6-5f52-4a21-9cfd-ef8229a539ea{notify.v3[8 37]}, prod-8c2abbc6-8f0b-464a-9268-fd977eb6d7af{notify.v3[7 30]}, prod-7de0b455-683e-4c79-9e62-9f93cd5a8170{notify.v3[22 46]}, prod-9508e2aa-96c5-491c-9ae1-b0a92bc0dab9{notify.v3[5 35]}, prod-ba944249-fbc5-46c3-a57c-7edb9dcbe25a{notify.v3[38]}, prod-e072e17c-73be-4501-8c0d-18d08c0767b3{notify.v3[27]}, prod-08cdbfb5-901d-4585-b6c6-5a3cdad6dd7d{notify.v3[1]}, prod-7e0c4c97-15b7-4847-873e-e4eafa964ec6{notify.v3[11 12]}, prod-b8fb26fc-ecdb-4d17-a5b0-b6ba971be975{notify.v3[33]}, prod-f9b53533-0bff-403f-b3ed-8c12711a17d4{notify.v3[21 47]}, prod-575c17bc-d78a-4ad7-aef9-1a38b332a916{notify.v3[41]}, prod-274d1555-e49c-42d3-bbfd-8c3ef5886d56{notify.v3[2]}, prod-35ee4b9e-1cf5-4415-b5fa-941fdea56158{notify.v3[9]}, prod-58698f86-8bff-4d24-a4fd-e9cc8049771b{notify.v3[14]}, prod-e65fbff2-bee1-44bd-b651-e1b47bbed3a9{notify.v3[17 44]}, prod-fcd97f8e-d213-4d0e-b593-79db0bfb1204{notify.v3[29]}, prod-0b294f5c-9f78-4566-856b-39bc8e255912{notify.v3[18]}"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4608798,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-02ece2b4-28d5-43bc-8921-746b089c0128"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4609122,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-0b294f5c-9f78-4566-856b-39bc8e255912"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.461267,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-9508e2aa-96c5-491c-9ae1-b0a92bc0dab9"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4610326,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-49321a12-bd30-484d-9d35-159d693e95d7"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.461065,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-4b988e95-184f-413e-b6db-a1222cfa0a21"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4611106,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-6fc9619a-bd9f-4c1b-9043-f19e0d63a6b5"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4614923,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-f9b53533-0bff-403f-b3ed-8c12711a17d4"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4610107,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-3ad691d8-1311-4cfe-8705-7488c62db865"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4613965,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-ded4db59-deb9-49b9-97ab-b04f459acde2"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4609416,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-21324d70-b1fc-44b5-91e2-d493d618b02a"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4613461,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-c038aab5-46da-4d0b-be62-78476ce5f2f4"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4613822,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-d94cdb0d-77ce-4673-81b8-6d1417ebb99f"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4612858,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-ba944249-fbc5-46c3-a57c-7edb9dcbe25a"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4614766,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-e65fbff2-bee1-44bd-b651-e1b47bbed3a9"}
Feb 21, 2023 @ 15:17:54.461	{"level":"info","ts":1676981874.4613893,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-dc5f85f6-5f52-4a21-9cfd-ef8229a539ea"}
Feb 21, 2023 @ 15:17:54.468	{"level":"info","ts":1676981874.468329,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"synced","group":"pullstandby"}
Feb 21, 2023 @ 15:17:54.468	{"level":"info","ts":1676981874.4688773,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"beginning heartbeat loop","group":"pullstandby"}
Feb 21, 2023 @ 15:17:54.470	{"level":"info","ts":1676981874.4707584,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"assigning partitions","why":"newly fetched offsets for group pullstandby","how":"assigning everything new, keeping current assignment"}
Feb 21, 2023 @ 15:17:57.471	{"level":"info","ts":1676981877.4710164,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"heartbeat errored","group":"pullstandby"}
Feb 21, 2023 @ 15:17:57.471	{"level":"info","ts":1676981877.471338,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"eager consumer revoking prior assigned partitions","group":"pullstandby"}
Feb 21, 2023 @ 15:17:57.471	{"level":"info","ts":1676981877.4712567,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"immediate metadata update triggered","why":"waitmeta after heartbeat error"}
Feb 21, 2023 @ 15:17:57.471	{"level":"info","ts":1676981877.4710898,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"assigning partitions","why":"revoking all assignments because we are not cooperative","how":"unassigning everything"}
Feb 21, 2023 @ 15:17:57.472	{"level":"info","ts":1676981877.4728072,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"joining group","group":"pullstandby"}
Feb 21, 2023 @ 15:17:57.564	{"level":"info","ts":1676981877.5646768,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-21324d70-b1fc-44b5-91e2-d493d618b02a"}
Feb 21, 2023 @ 15:17:57.564	{"level":"info","ts":1676981877.5646863,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-26e5bbe3-645e-4fa2-bcd5-82f93ccd7c40"}
Feb 21, 2023 @ 15:17:57.564	{"level":"info","ts":1676981877.5647068,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-35ee4b9e-1cf5-4415-b5fa-941fdea56158"}
Feb 21, 2023 @ 15:17:57.564	{"level":"info","ts":1676981877.5647333,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-3ec4a5c0-8dc5-417d-ae63-a3d3e33dbb8d"}
Feb 21, 2023 @ 15:17:57.564	{"level":"info","ts":1676981877.5647428,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-49321a12-bd30-484d-9d35-159d693e95d7"}
Feb 21, 2023 @ 15:17:57.564	{"level":"info","ts":1676981877.564715,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-377b682e-335c-4bab-974f-25b2467bd45f"}
Feb 21, 2023 @ 15:17:57.564	{"level":"info","ts":1676981877.564474,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"joined, balancing group","group":"pullstandby","member_id":"prod-08cdbfb5-901d-4585-b6c6-5a3cdad6dd7d","instance_id":"<nil>"}
Feb 21, 2023 @ 15:17:57.564	{"level":"info","ts":1676981877.5646613,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-08cdbfb5-901d-4585-b6c6-5a3cdad6dd7d"}
Feb 21, 2023 @ 15:17:57.564	{"level":"info","ts":1676981877.5646691,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-0b294f5c-9f78-4566-856b-39bc8e255912"}
Feb 21, 2023 @ 15:17:57.564	{"level":"info","ts":1676981877.5646963,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-274d1555-e49c-42d3-bbfd-8c3ef5886d56"}
Feb 21, 2023 @ 15:17:57.564	{"level":"info","ts":1676981877.5645642,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balancing group as leader"}
Feb 21, 2023 @ 15:17:57.564	{"level":"info","ts":1676981877.564648,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-02ece2b4-28d5-43bc-8921-746b089c0128"}
Feb 21, 2023 @ 15:17:57.564	{"level":"info","ts":1676981877.5647254,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-3ad691d8-1311-4cfe-8705-7488c62db865"}
Feb 21, 2023 @ 15:17:57.565	{"level":"info","ts":1676981877.5647702,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-58698f86-8bff-4d24-a4fd-e9cc8049771b"}
Feb 21, 2023 @ 15:17:57.565	{"level":"info","ts":1676981877.5649824,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-e8b6365b-75c4-4c3c-90be-259a9eac449d"}
Feb 21, 2023 @ 15:17:57.565	{"level":"info","ts":1676981877.5647533,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-4b988e95-184f-413e-b6db-a1222cfa0a21"}
Feb 21, 2023 @ 15:17:57.565	{"level":"info","ts":1676981877.564763,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-575c17bc-d78a-4ad7-aef9-1a38b332a916"}
Feb 21, 2023 @ 15:17:57.565	{"level":"info","ts":1676981877.5648007,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-7de0b455-683e-4c79-9e62-9f93cd5a8170"}
Feb 21, 2023 @ 15:17:57.565	{"level":"info","ts":1676981877.5648587,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-a29736b3-86cf-46f8-b2e5-9a07857f580c"}
Feb 21, 2023 @ 15:17:57.565	{"level":"info","ts":1676981877.5648696,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-b8fb26fc-ecdb-4d17-a5b0-b6ba971be975"}
Feb 21, 2023 @ 15:17:57.565	{"level":"info","ts":1676981877.5648909,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-bec4095c-8fff-406e-bab0-888b274b7599"}
Feb 21, 2023 @ 15:17:57.565	{"level":"info","ts":1676981877.5649104,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-cfe31e81-952a-42be-b1e0-ea5a31d238fc"}
Feb 21, 2023 @ 15:17:57.565	{"level":"info","ts":1676981877.564933,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-dc5f85f6-5f52-4a21-9cfd-ef8229a539ea"}
Feb 21, 2023 @ 15:17:57.565	{"level":"info","ts":1676981877.5649538,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-ded4db59-deb9-49b9-97ab-b04f459acde2"}
Feb 21, 2023 @ 15:17:57.565	{"level":"info","ts":1676981877.565163,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"syncing","group":"pullstandby","protocol_type":"consumer"}
Feb 21, 2023 @ 15:17:57.565	{"level":"info","ts":1676981877.56478,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-5a8523a0-e0ff-4086-af95-78f680707a11"}
Feb 21, 2023 @ 15:17:57.565	{"level":"info","ts":1676981877.5648222,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-833b1385-4433-43e6-95fd-5d22c53fe36a"}
Feb 21, 2023 @ 15:17:57.565	{"level":"info","ts":1676981877.5648518,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-9508e2aa-96c5-491c-9ae1-b0a92bc0dab9"}
Feb 21, 2023 @ 15:17:57.565	{"level":"info","ts":1676981877.5649624,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-e072e17c-73be-4501-8c0d-18d08c0767b3"}
Feb 21, 2023 @ 15:17:57.565	{"level":"info","ts":1676981877.5650022,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-fcd97f8e-d213-4d0e-b593-79db0bfb1204"}
Feb 21, 2023 @ 15:17:57.565	{"level":"info","ts":1676981877.5647905,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-6fc9619a-bd9f-4c1b-9043-f19e0d63a6b5"}
Feb 21, 2023 @ 15:17:57.565	{"level":"info","ts":1676981877.564832,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-83dfaaec-e9af-4a6c-bff4-c28c3a47b628"}
Feb 21, 2023 @ 15:17:57.565	{"level":"info","ts":1676981877.564842,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-8c2abbc6-8f0b-464a-9268-fd977eb6d7af"}
Feb 21, 2023 @ 15:17:57.565	{"level":"info","ts":1676981877.564881,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-ba944249-fbc5-46c3-a57c-7edb9dcbe25a"}
Feb 21, 2023 @ 15:17:57.565	{"level":"info","ts":1676981877.5649004,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-c038aab5-46da-4d0b-be62-78476ce5f2f4"}
Feb 21, 2023 @ 15:17:57.565	{"level":"info","ts":1676981877.5649168,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-d00841bb-2e4e-4a8c-a6f4-1d0b5eb7003d"}
Feb 21, 2023 @ 15:17:57.565	{"level":"info","ts":1676981877.5651197,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balanced","plan":"prod-21324d70-b1fc-44b5-91e2-d493d618b02a{notify.v3[26]}, prod-35ee4b9e-1cf5-4415-b5fa-941fdea56158{notify.v3[9]}, prod-6fc9619a-bd9f-4c1b-9043-f19e0d63a6b5{notify.v3[20]}, prod-cfe31e81-952a-42be-b1e0-ea5a31d238fc{notify.v3[45]}, prod-e072e17c-73be-4501-8c0d-18d08c0767b3{notify.v3[27]}, prod-02ece2b4-28d5-43bc-8921-746b089c0128{notify.v3[0]}, prod-4b988e95-184f-413e-b6db-a1222cfa0a21{notify.v3[36]}, prod-83dfaaec-e9af-4a6c-bff4-c28c3a47b628{notify.v3[3 31]}, prod-a29736b3-86cf-46f8-b2e5-9a07857f580c{notify.v3[42]}, prod-fcd97f8e-d213-4d0e-b593-79db0bfb1204{notify.v3[29]}, prod-575c17bc-d78a-4ad7-aef9-1a38b332a916{notify.v3[41]}, prod-7e0c4c97-15b7-4847-873e-e4eafa964ec6{notify.v3[11 12]}, prod-e8b6365b-75c4-4c3c-90be-259a9eac449d{notify.v3[28]}, prod-f9b53533-0bff-403f-b3ed-8c12711a17d4{notify.v3[21 47]}, prod-0b294f5c-9f78-4566-856b-39bc8e255912{notify.v3[18]}, prod-274d1555-e49c-42d3-bbfd-8c3ef5886d56{notify.v3[2]}, prod-3ad691d8-1311-4cfe-8705-7488c62db865{notify.v3[23]}, prod-7de0b455-683e-4c79-9e62-9f93cd5a8170{notify.v3[22]}, prod-9508e2aa-96c5-491c-9ae1-b0a92bc0dab9{notify.v3[5 35]}, prod-b8fb26fc-ecdb-4d17-a5b0-b6ba971be975{notify.v3[33]}, prod-08cdbfb5-901d-4585-b6c6-5a3cdad6dd7d{notify.v3[1]}, prod-377b682e-335c-4bab-974f-25b2467bd45f{notify.v3[19]}, prod-49321a12-bd30-484d-9d35-159d693e95d7{notify.v3[4]}, prod-5a8523a0-e0ff-4086-af95-78f680707a11{notify.v3[16]}, prod-833b1385-4433-43e6-95fd-5d22c53fe36a{notify.v3[24 25]}, prod-bec4095c-8fff-406e-bab0-888b274b7599{notify.v3[13 40]}, prod-26e5bbe3-645e-4fa2-bcd5-82f93ccd7c40{notify.v3[32]}, prod-c038aab5-46da-4d0b-be62-78476ce5f2f4{notify.v3[43]}, prod-8c2abbc6-8f0b-464a-9268-fd977eb6d7af{notify.v3[7 30]}, prod-d94cdb0d-77ce-4673-81b8-6d1417ebb99f{notify.v3[15 34]}, prod-ded4db59-deb9-49b9-97ab-b04f459acde2{notify.v3[10 39]}, prod-e65fbff2-bee1-44bd-b651-e1b47bbed3a9{notify.v3[17 44]}, prod-3ec4a5c0-8dc5-417d-ae63-a3d3e33dbb8d{notify.v3[6]}, prod-58698f86-8bff-4d24-a4fd-e9cc8049771b{notify.v3[14]}, prod-ba944249-fbc5-46c3-a57c-7edb9dcbe25a{notify.v3[38]}, prod-d00841bb-2e4e-4a8c-a6f4-1d0b5eb7003d{}, prod-dc5f85f6-5f52-4a21-9cfd-ef8229a539ea{notify.v3[8 37]}"}
Feb 21, 2023 @ 15:17:57.565	{"level":"info","ts":1676981877.564812,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-7e0c4c97-15b7-4847-873e-e4eafa964ec6"}
Feb 21, 2023 @ 15:17:57.565	{"level":"info","ts":1676981877.5649261,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-d94cdb0d-77ce-4673-81b8-6d1417ebb99f"}
Feb 21, 2023 @ 15:17:57.565	{"level":"info","ts":1676981877.5649729,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-e65fbff2-bee1-44bd-b651-e1b47bbed3a9"}
Feb 21, 2023 @ 15:17:57.565	{"level":"info","ts":1676981877.5649927,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-f9b53533-0bff-403f-b3ed-8c12711a17d4"}
Feb 21, 2023 @ 15:17:57.573	{"level":"info","ts":1676981877.5731091,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"synced","group":"pullstandby"}
Feb 21, 2023 @ 15:17:57.573	{"level":"info","ts":1676981877.5733993,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"beginning heartbeat loop","group":"pullstandby"}
Feb 21, 2023 @ 15:17:57.575	{"level":"info","ts":1676981877.575206,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"assigning partitions","how":"assigning everything new, keeping current assignment","why":"newly fetched offsets for group pullstandby"}

that's all logs from pod

There are multiple things confusing here.
First, the log itself includes an "interests" field,
https://github.com/twmb/franz-go/blob/3322f311fd833129f91b4ea76581150274bdb457/pkg/kgo/group_balancer.go#L427
this is key to seeing what all members are interested in consuming -- without these logs, I can't recreate the inputs to the balance plan. In your logs, I see this:
Feb 21, 2023 @ 15:17:54.461     {"level":"info","ts":1676981874.4609933,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-35ee4b9e-1cf5-4415-b5fa-941fdea56158"}
Feb 21, 2023 @ 15:17:54.461     {"level":"info","ts":1676981874.4611537,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-7e0c4c97-15b7-4847-873e-e4eafa964ec6"}
Feb 21, 2023 @ 15:17:54.461     {"level":"info","ts":1676981874.4613745,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-d00841bb-2e4e-4a8c-a6f4-1d0b5eb7003d"}
Feb 21, 2023 @ 15:17:54.461     {"level":"info","ts":1676981874.4610183,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-3ec4a5c0-8dc5-417d-ae63-a3d3e33dbb8d"}
Feb 21, 2023 @ 15:17:54.461     {"level":"info","ts":1676981874.4612792,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-b8fb26fc-ecdb-4d17-a5b0-b6ba971be975"}
Feb 21, 2023 @ 15:17:54.461     {"level":"info","ts":1676981874.4610915,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-58698f86-8bff-4d24-a4fd-e9cc8049771b"}
Feb 21, 2023 @ 15:17:54.461     {"level":"info","ts":1676981874.4615252,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"balance group member","id":"prod-fcd97f8e-d213-4d0e-b593-79db0bfb1204"}

Also, in your code, I see that you're using the balancers CooperativeSticky and Sticky. If all group members are cooperative-sticky, then I shouldn't see the following log:
Feb 21, 2023 @ 15:15:33.271     {"level":"info","ts":1676981733.2709644,"caller":"broker-kgo@v0.2.8/kgo.go:317","msg":"assigning partitions","why":"revoking all assignments because we are not cooperative","how":"unassigning everything"}

Something in your group is different -- if everything is cooperative, then that log wouldn't exist.

can you provide next steps for me?

does interests string exists in proper place in  1.4.2  ?

That "interests" string exists since the log line was introduced.
Please upgrade to 1.12.1, but be aware of this (the only) behavior change in 1.5.1: https://github.com/twmb/franz-go/blob/master/CHANGELOG.md#v151

i'm use 1.12.1 and slightly modified goroutine per partition example, what does it mean in logs:
heartbeat errored again while waiting for user revoke to finish
and this
in default commit: unable to commit offsets for topic partition","group":"TestAssign","topic":"mytopic","stacktrace":"github.com/twmb/franz-go/pkg/kgo.(*wrappedLogger).Log\n\t/Users/vtolstov/go/pkg/mod/github.com/twmb/franz-go@v1.12.1/pkg/kgo/logger.go:123\ngithub.com/twmb/franz-go/pkg/kgo.(*groupConsumer).defaultCommitCallback\n\t/Users/vtolstov/go/pkg/mod/github.com/twmb/franz-go@v1.12.1/pkg/kgo/consumer_group.go:1999\ngithub.com/twmb/franz-go/pkg/kgo.(*groupConsumer).commitAcrossRebalance.func4\n\t/Users/vtolstov/go/pkg/mod/github.com/twmb/franz-go@v1.12.1/pkg/kgo/consumer_group.go:274
my tests contains 48 partitions in topic, about 200000 messages per partition and i'm run 8 instances of service to handle partitions. when i'm start in single instance all works fine.

heartbeat errored again while waiting for user revoke to finish -- what's the rest of the log line? The log line is this:
915:			g.cfg.logger.Log(LogLevelInfo, "heartbeat errored again while waiting for user revoke to finish", "group", g.cfg.group, "err", err)

What's the rest of the unable to commit message?
1999                                 g.cfg.logger.Log(LogLevelError, "in default commit: unable to commit offsets for topic partition",
2000                                         "group", g.cfg.group,
2001                                         "topic", topic.Topic,
2002                                         "partition", partition.Partition,
2003                                         "error", err)

Both of these are missing the error field. It's impossible to help if the output is truncated.

sorry my logger eats messages:
[ERROR] in default commit: unable to commit offsets for topic partition; group: TestAssign, topic: rbo.interestrate.deposit, partition: 1, error: UNKNOWN_MEMBER_ID: The coordinator is not aware of this member. [ERROR] in default commit: unable to commit offsets for topic partition; group: TestAssign, topic: rbo.interestrate.deposit, partition: 5, error: UNKNOWN_MEMBER_ID: The coordinator is not aware of this member. [ERROR] in default commit: unable to commit offsets for topic partition; group: TestAssign, topic: rbo.interestrate.deposit, partition: 13, error: UNKNOWN_MEMBER_ID: The coordinator is not aware of this member. [ERROR] in default commit: unable to commit offsets for topic partition; group: TestAssign, topic: rbo.interestrate.deposit, partition: 21, error: UNKNOWN_MEMBER_ID: The coordinator is not aware of this member. [ERROR] in default commit: unable to commit offsets for topic partition; group: TestAssign, topic: rbo.interestrate.deposit, partition: 25, error: UNKNOWN_MEMBER_ID: The coordinator is not aware of this member. [ERROR] in default commit: unable to commit offsets for topic partition; group: TestAssign, topic: rbo.interestrate.deposit, partition: 29, error: UNKNOWN_MEMBER_ID: The coordinator is not aware of this member. [ERROR] in default commit: unable to commit offsets for topic partition; group: TestAssign, topic: rbo.interestrate.deposit, partition: 33, error: UNKNOWN_MEMBER_ID: The coordinator is not aware of this member. [ERROR] in default commit: unable to commit offsets for topic partition; group: TestAssign, topic: rbo.interestrate.deposit, partition: 37, error: UNKNOWN_MEMBER_ID: The coordinator is not aware of this member
and
[DEBUG] heartbeat complete; group: TestAssign, err: UNKNOWN_MEMBER_ID: The coordinator is not aware of this member. [INFO] heartbeat errored again while waiting for user revoke to finish; group: TestAssign, err: UNKNOWN_MEMBER_ID: The coordinator is not aware of this member.

but i'm not restart old consumer , why coordinator is not aware of it

what is the recommended size of num for PollRecords ?
I'm test - if i'm specify 10 - all works fine, if i'm specify 1000 - i have a problems described above

also what channel size to specify?

func (s *subscriber) assigned(_ context.Context, c *kgo.Client, assigned map[string][]int32) {
	for topic, partitions := range assigned {
		for _, partition := range partitions {
			pc := &consumer{
				c:         c,
				topic:     topic,
				partition: partition,

				quit:    make(chan struct{}),
				done:    make(chan struct{}),
				recs:    make(chan kgo.FetchTopicPartition, len(partitions)),
				handler: s.handler,
				kopts:   s.kopts,
				opts:    s.opts,
			}
			s.consumers[tp{topic, partition}] = pc
			go pc.consume()
		}
	}
}

after sometime i'm start another consumer and on assign it receives no partitions
Feb 27 10:25:59.005[DEBUG] read JoinGroup v7; broker: 3, bytes_read: 219, read_wait: 16.25µs, time_to_read: 5.018675375s, err: <nil>
Feb 27 10:25:59.005[INFO] joined; group: TestAssign, member_id: TestAssign-ea3c2858-0fbb-481a-8dea-bb7d959fe6ab-36c282b1-f006-48b1-b35e-f8521a7669d1, instance_id: <nil>, generation: 350, leader: false
Feb 27 10:25:59.005[INFO] syncing; group: TestAssign, protocol_type: consumer, protocol: cooperative-sticky
Feb 27 10:25:59.005[DEBUG] wrote SyncGroup v5; broker: 3, bytes_written: 193, write_wait: 19µs, time_to_write: 10.875µs, err: <nil>
Feb 27 10:25:59.005[DEBUG] read JoinGroup v7; broker: 3, bytes_read: 219, read_wait: 18.458µs, time_to_read: 5.023210458s, err: <nil>
Feb 27 10:25:59.005[INFO] joined; group: TestAssign, member_id: TestAssign-f1aa291d-e025-452f-9c08-00308e2ca773-0a756746-d3c7-4d6c-9924-e57aef39ed63, instance_id: <nil>, generation: 350, leader: false
Feb 27 10:25:59.005[INFO] syncing; group: TestAssign, protocol_type: consumer, protocol: cooperative-sticky
Feb 27 10:25:59.005[DEBUG] wrote SyncGroup v5; broker: 3, bytes_written: 193, write_wait: 10.375µs, time_to_write: 8.083µs, err: <nil>
Feb 27 10:25:59.005[DEBUG] read JoinGroup v7; broker: 3, bytes_read: 219, read_wait: 15.125µs, time_to_read: 5.065116333s, err: <nil>
Feb 27 10:25:59.005[INFO] joined; group: TestAssign, member_id: TestAssign-8b609f3b-9886-4e1f-8806-1ddc01d8313b-f85fd36a-ea89-4f9e-9986-71907a6753ce, instance_id: <nil>, generation: 350, leader: false
Feb 27 10:25:59.005[INFO] syncing; group: TestAssign, protocol_type: consumer, protocol: cooperative-sticky
Feb 27 10:25:59.005[DEBUG] wrote SyncGroup v5; broker: 3, bytes_written: 193, write_wait: 8µs, time_to_write: 10.208µs, err: <nil>
Feb 27 10:25:59.012[DEBUG] read JoinGroup v7; broker: 3, bytes_read: 219, read_wait: 18.875µs, time_to_read: 4.998113333s, err: <nil>
Feb 27 10:25:59.012[INFO] joined; group: TestAssign, member_id: TestAssign-de8effce-c33d-46ff-8646-eb072d8ba118-3cf3aff1-f31b-4f50-a35a-19c40b07444c, instance_id: <nil>, generation: 350, leader: false
Feb 27 10:25:59.012[INFO] syncing; group: TestAssign, protocol_type: consumer, protocol: cooperative-sticky
Feb 27 10:25:59.012[DEBUG] wrote SyncGroup v5; broker: 3, bytes_written: 193, write_wait: 12.75µs, time_to_write: 13.125µs, err: <nil>
Feb 27 10:25:59.102[DEBUG] autocommitting; group: TestAssign
Feb 27 10:25:59.102[DEBUG] autocommitting; group: TestAssign
Feb 27 10:25:59.102[DEBUG] autocommitting; group: TestAssign
Feb 27 10:25:59.102[DEBUG] autocommitting; group: TestAssign
Feb 27 10:25:59.110[DEBUG] read SyncGroup v5; broker: 3, bytes_read: 55, read_wait: 15.75µs, time_to_read: 104.542542ms, err: <nil>
Feb 27 10:25:59.110[INFO] synced; group: TestAssign, assigned:
Feb 27 10:25:59.110[INFO] new group session begun; group: TestAssign, added: , lost:
Feb 27 10:25:59.110[DEBUG] entering OnPartitionsAssigned; with: map[]
Feb 27 10:25:59.110[INFO] beginning heartbeat loop; group: TestAssign
Feb 27 10:25:59.112[DEBUG] read SyncGroup v5; broker: 3, bytes_read: 55, read_wait: 9.417µs, time_to_read: 106.340375ms, err: <nil>
Feb 27 10:25:59.112[INFO] synced; group: TestAssign, assigned:
Feb 27 10:25:59.112[INFO] new group session begun; group: TestAssign, added: , lost:
Feb 27 10:25:59.112[DEBUG] entering OnPartitionsAssigned; with: map[]
Feb 27 10:25:59.112[INFO] beginning heartbeat loop; group: TestAssign
Feb 27 10:25:59.112[DEBUG] read SyncGroup v5; broker: 3, bytes_read: 55, read_wait: 15.25µs, time_to_read: 99.546625ms, err: <nil>
Feb 27 10:25:59.112[INFO] synced; group: TestAssign, assigned:
Feb 27 10:25:59.112[INFO] new group session begun; group: TestAssign, added: , lost:
Feb 27 10:25:59.112[DEBUG] entering OnPartitionsAssigned; with: map[]
Feb 27 10:25:59.112[INFO] beginning heartbeat loop; group: TestAssign
Feb 27 10:25:59.112[DEBUG] read SyncGroup v5; broker: 3, bytes_read: 55, read_wait: 9.625µs, time_to_read: 106.520458ms, err: <nil>
Feb 27 10:25:59.112[INFO] synced; group: TestAssign, assigned:
Feb 27 10:25:59.112[INFO] new group session begun; group: TestAssign, added: , lost:
Feb 27 10:25:59.112[DEBUG] entering OnPartitionsAssigned; with: map[]
Feb 27 10:25:59.112[INFO] beginning heartbeat loop; group: TestAssign

but admin command says that this member id not present in group!
client:TestAssign-0509ceed-c471-4ddd-b33f-fad23f61cb95	partitions: 0
client:TestAssign-08c1a231-684c-410c-a999-5b1a5b2cc81a	partitions: 1
client:TestAssign-4d9ae2ea-5f61-42e5-8423-50b0fde08736	partitions: 2
client:TestAssign-afba9077-0d30-4f5f-a4ba-6c5ac4a4b9f0	partitions: 3
client:TestAssign-b7886f1d-2305-4199-8582-6ab2d4dc96e2	partitions: 4
client:TestAssign-6b47444c-69c9-4568-9009-7c33f334327a	partitions: 5
client:TestAssign-bf25186c-072b-4a7d-a075-85508ca2b603	partitions: 6
client:TestAssign-e656f59b-3122-47b1-9fe0-170644a82855	partitions: 7
client:TestAssign-09e22162-6154-4cf8-b96a-856059178fe5	partitions: 8
client:TestAssign-85f0c6ec-e97a-43f5-a77c-5ac1c944ef40	partitions: 9
client:TestAssign-ef7d7e80-8862-4661-886a-ddde3e37a268	partitions: 10
client:TestAssign-997f027c-ad7b-4c29-a239-ca9eee6e134a	partitions: 11
client:TestAssign-997f027c-ad7b-4c29-a239-ca9eee6e134a	partitions: 12
client:TestAssign-bf25186c-072b-4a7d-a075-85508ca2b603	partitions: 13
client:TestAssign-4d9ae2ea-5f61-42e5-8423-50b0fde08736	partitions: 14
client:TestAssign-0509ceed-c471-4ddd-b33f-fad23f61cb95	partitions: 15
client:TestAssign-08c1a231-684c-410c-a999-5b1a5b2cc81a	partitions: 16
client:TestAssign-6b47444c-69c9-4568-9009-7c33f334327a	partitions: 17
client:TestAssign-afba9077-0d30-4f5f-a4ba-6c5ac4a4b9f0	partitions: 18
client:TestAssign-b7886f1d-2305-4199-8582-6ab2d4dc96e2	partitions: 19
client:TestAssign-ef7d7e80-8862-4661-886a-ddde3e37a268	partitions: 20
client:TestAssign-09e22162-6154-4cf8-b96a-856059178fe5	partitions: 21
client:TestAssign-e656f59b-3122-47b1-9fe0-170644a82855	partitions: 22
client:TestAssign-85f0c6ec-e97a-43f5-a77c-5ac1c944ef40	partitions: 23
client:TestAssign-85f0c6ec-e97a-43f5-a77c-5ac1c944ef40	partitions: 24
client:TestAssign-afba9077-0d30-4f5f-a4ba-6c5ac4a4b9f0	partitions: 25
client:TestAssign-4d9ae2ea-5f61-42e5-8423-50b0fde08736	partitions: 26
client:TestAssign-997f027c-ad7b-4c29-a239-ca9eee6e134a	partitions: 27
client:TestAssign-bf25186c-072b-4a7d-a075-85508ca2b603	partitions: 28
client:TestAssign-6b47444c-69c9-4568-9009-7c33f334327a	partitions: 29
client:TestAssign-0509ceed-c471-4ddd-b33f-fad23f61cb95	partitions: 30
client:TestAssign-08c1a231-684c-410c-a999-5b1a5b2cc81a	partitions: 31
client:TestAssign-e656f59b-3122-47b1-9fe0-170644a82855	partitions: 32
client:TestAssign-ef7d7e80-8862-4661-886a-ddde3e37a268	partitions: 33
client:TestAssign-b7886f1d-2305-4199-8582-6ab2d4dc96e2	partitions: 34
client:TestAssign-09e22162-6154-4cf8-b96a-856059178fe5	partitions: 35

The first group of logs doesn't start early enough, but I'm assuming currently that your revoke callback is taking too long. The first log lines already indicate a rebalance is in progress at 9:17:05, then at 9:17:11, you start getting UNKNOWN_MEMBER_ID, which is due to your revoke callback not being done yet. In fact, your revoke still isn't done in that entire set of logs. Maybe it's a deadlock on your side.

what is the recommended size of num for PollRecords

Up to you

after sometime i'm start another consumer and on assign it receives no partitions

Not sure, but given the other issues above, it's not worth looking into this issue because it could just be a fallout of the other problems.

ok if i'm run goroutine per partition example with mark commit offsets - does it enough to reproduce problems ?

Not sure, this looks like an application side problem.

i'm start two instance of consumers each subscribe 4 times (so each must handle 6 partitions)
assignment before i'm stop tests:
client:TestAssign-7a601878-dcac-4bd0-abf0-947d282d925c	partitions: 0
client:TestAssign-bc87faa1-289d-4441-bd90-c7aa14adc13e	partitions: 1
client:TestAssign-f7b22fe6-d9e0-4bfc-a8c5-b2bec5826b4c	partitions: 2
client:TestAssign-c0d9f00e-297e-4673-9042-12b7e7a5bdc8	partitions: 3
client:TestAssign-c0d9f00e-297e-4673-9042-12b7e7a5bdc8	partitions: 4
client:TestAssign-bc87faa1-289d-4441-bd90-c7aa14adc13e	partitions: 5
client:TestAssign-7a601878-dcac-4bd0-abf0-947d282d925c	partitions: 6
client:TestAssign-f7b22fe6-d9e0-4bfc-a8c5-b2bec5826b4c	partitions: 7
client:TestAssign-f7b22fe6-d9e0-4bfc-a8c5-b2bec5826b4c	partitions: 8
client:TestAssign-bc87faa1-289d-4441-bd90-c7aa14adc13e	partitions: 9
client:TestAssign-c0d9f00e-297e-4673-9042-12b7e7a5bdc8	partitions: 10
client:TestAssign-7a601878-dcac-4bd0-abf0-947d282d925c	partitions: 11
client:TestAssign-7a601878-dcac-4bd0-abf0-947d282d925c	partitions: 12
client:TestAssign-bc87faa1-289d-4441-bd90-c7aa14adc13e	partitions: 13
client:TestAssign-f7b22fe6-d9e0-4bfc-a8c5-b2bec5826b4c	partitions: 14
client:TestAssign-c0d9f00e-297e-4673-9042-12b7e7a5bdc8	partitions: 15
client:TestAssign-c0d9f00e-297e-4673-9042-12b7e7a5bdc8	partitions: 16
client:TestAssign-bc87faa1-289d-4441-bd90-c7aa14adc13e	partitions: 17
client:TestAssign-7a601878-dcac-4bd0-abf0-947d282d925c	partitions: 18
client:TestAssign-f7b22fe6-d9e0-4bfc-a8c5-b2bec5826b4c	partitions: 19
client:TestAssign-f7b22fe6-d9e0-4bfc-a8c5-b2bec5826b4c	partitions: 20
client:TestAssign-bc87faa1-289d-4441-bd90-c7aa14adc13e	partitions: 21
client:TestAssign-c0d9f00e-297e-4673-9042-12b7e7a5bdc8	partitions: 22
client:TestAssign-7a601878-dcac-4bd0-abf0-947d282d925c	partitions: 23

logs from consumer1 and consumer2 instances in attached files

Those full logs help -- I see a bug -- can you remove Sticky() from your balancers, to make it just CooperativeSticky? On this line: https://github.com/unistack-org/micro-broker-kgo/blob/adb5c9bfc9fe9c100e6f090a0e5ea73344ed2ece/kgo.go#L385
The bug is that some logic in the client is thinking the group is eager, because there is an eager balancer option. The group is rebalancing as coopartive (i.e. only getting partial assignments) while the client is revoking all partitions.

thanks, you suggestion works fine for me. now all consumers always get assignment and process messages.


Added Italian translation.

Simple Italian translation.

Funny, the translations editor doesn't play well with the plurals (it doesn't show them at all).
I've added the missing strings.

Added some missing strings, added copyright notice and removed translation_description.

Thank you again! =D


Allow definition of first message field by parameter value

Context and Problem Statement
With the introduction of parameterized messages (#609) there is no strict need for a definite first field anymore, as the first field can depend on the value of a message parameter. Currently, the model is checked to have exactly one link to the first message field.
Considered Options
O1 Allow definition of first message field by parameter value
+ More flexible
O2 Keep current restriction
+ No changes needed
Decision Outcome
O1

We decided to support multiple initial links to allow the first message field to be defined by parameter values (O1).


feat: add default descriptions to cfn templates for metrics tracking

Add default descriptions to root and sub-stacks at create time.
SampleRoot-stack description:
{"createdOn":"Mac","createdBy":"Amplify","createdWith":"6.3.1","stackType":"root","metadata":{}}
Sample category sub-stack description:
{"createdOn":"Mac","createdBy":"Amplify","createdWith":"6.3.1","stackType":"storage-S3","metadata":{}}

Codecov Report

Merging #8702 (17da32c) into extOverrides3 (61fbf7c) will increase coverage by 0.00%.
The diff coverage is 63.04%.

@@              Coverage Diff               @@
##           extOverrides3    #8702   +/-   ##
==============================================
  Coverage          56.31%   56.32%
==============================================
  Files                834      835    +1
  Lines              47122    47165   +43
  Branches            9435     9441    +6
==============================================
+ Hits               26537    26565   +28
- Misses             19767    19782   +15
  Partials             818      818

Impacted Files
Coverage Δ

...y-provider-awscloudformation/src/push-resources.ts
18.29% <33.33%> (+0.11%)
:arrow_up:

...wscloudformation/src/template-description-utils.ts
63.41% <63.41%> (ø)

...lify-provider-awscloudformation/src/initializer.ts
47.85% <100.00%> (+0.61%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 61fbf7c...17da32c. Read the comment docs.


Says incorrect API key provided even after switching

Changed my API key and put in my new one when asked for an API key, yet it still uses my old API key and throws an error, and won't pop up the authentication window.

Debug Data

Property
Value

Name
"Wolfram/Chatbook"

Version
"1.0.0"

EvaluationEnvironment
"Session"

FrontEndVersion
"Desktop: 13.3 for Microsoft Windows (64-bit) (June 3, 2023)"

KernelVersion
"13.3.0 for Microsoft Windows (64-bit) (June 3, 2023)"

SystemID
"Windows-x86-64"

Notebooks
True

DynamicEvaluation
False

SynchronousEvaluation
False

TaskEvaluation
False

Settings
Inherited

Stack Data
throwFailure
trimURL

Failure Data
<|
  "Evaluation" :> Wolfram`Chatbook`Actions`Private`writeChunk[Dynamic[Wolfram`Chatbook`Actions`Private`container$56114], CellObject["584f0699-2334-4c42-b12c-b65d8704bea1", "c1874b0e-a856-4026-b4c2-81a138ff8155"], "W6xKpbfxzpgWp3i4ctUpJj\",\"object\":\"chat.completion.chunk\",\"created\":1702229641,\"model\":\"gpt-3.5-turbo-0613\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"\"},\"finish_reason\":null}]}\n\n"]
  "Arguments" -> {}
|>

I noticed you're on an older first, first please update to a recent version using PacletInstall["Wolfram/Chatbook", UpdatePacletSites -> True]
Trigger the service connection to prompt for the key with ServiceConnect["OpenAI", "New"] Checkmark save connection if you want to keep using it for future evaluations
Afterwards you can manage the connections by viewing the saved connections with ServiceConnections`SavedConnections[]
If you want to delete a saved connection ServiceConnections`DeleteConnection[ServiceObject] where ServiceObject is one of the objects printed by SavedConnections[]

This error was fixed in #379


CAMEL-10339: Allow date offsets and timezones with Simple language

https://issues.apache.org/jira/browse/CAMEL-10339

Can you add documentation to the adoc file also
https://github.com/apache/camel/blob/master/camel-core/src/main/docs/simple-language.adoc

Running the tests of camel-core shows some failures
Failed tests:
org.apache.camel.component.file.FileConsumerFilterFileTest.testFilterFilesWithDate(org.apache.camel.component.file.FileConsumerFilterFileTest)
  Run 1: FileConsumerFilterFileTest>TestSupport.runBare:58->testFilterFilesWithDate:68 mock://result2 Received message count. Expected: <1> but was: <0>
  Run 2: FileConsumerFilterFileTest>TestSupport.runBare:58->testFilterFilesWithDate:68 mock://result2 Expected minimum 1500 millis waiting on the result, but was faster with 1004 millis.
  Run 3: FileConsumerFilterFileTest>TestSupport.runBare:58->testFilterFilesWithDate:68 mock://result2 Received message count. Expected: <1> but was: <0>

org.apache.camel.language.simple.SimpleTest.testDateWithTimezone(org.apache.camel.language.simple.SimpleTest)
  Run 1: SimpleTest>TestSupport.runBare:58->testDateWithTimezone:495->LanguageTestSupport.assertExpression:69->LanguageTestSupport.assertExpression:62->ContextTestSupport.assertExpression:314->TestSupport.assertExpression:260 Expression: date(header.birthday:yyyy-MM-dd'T'HH:mm:ss:SSS:null) on Exchange: Exchange[] expected:<1974-04-20T08:55:47:123> but was:<1974-04-20T07:55:47:123>
  Run 2: SimpleTest>TestSupport.runBare:58->testDateWithTimezone:495->LanguageTestSupport.assertExpression:69->LanguageTestSupport.assertExpression:62->ContextTestSupport.assertExpression:314->TestSupport.assertExpression:260 Expression: date(header.birthday:yyyy-MM-dd'T'HH:mm:ss:SSS:null) on Exchange: Exchange[] expected:<1974-04-20T08:55:47:123> but was:<1974-04-20T07:55:47:123>
  Run 3: SimpleTest>TestSupport.runBare:58->testDateWithTimezone:495->LanguageTestSupport.assertExpression:69->LanguageTestSupport.assertExpression:62->ContextTestSupport.assertExpression:314->TestSupport.assertExpression:260 Expression: date(header.birthday:yyyy-MM-dd'T'HH:mm:ss:SSS:null) on Exchange: Exchange[] expected:<1974-04-20T08:55:47:123> but was:<1974-04-20T07:55:47:123>

Tests run: 5755, Failures: 2, Errors: 0, Skipped: 4

Thanks for the PR it has been merged. Do you mind closing this?

Cheers!


protect the DNS cache from entries that lack an Answer

Two changes in this PR:

protect the DNS cache from entries that lack an Answer (explanation below)
add .gitignore to ignore vendor/ folder

In testing Nuclei as an embedded library in an AWS lambda (even worse, in a lambda being run in localstack container in docker-compose), with a bit of fmt.Print* to print the value of b here:
b, _ := data.Marshal()
err = d.hm.Set(hostname, b)

I observed that when passing both https://something and http://something into the executor with Nuclei configured for BulkSize greater than 1, the value of b occasionally lacked an Answer, which would populate the cache with a bad entry.
I honestly don't know if it's due to odd behavior of docker-compose internal DNS, or a go bug or what, but the change in this PR adds some protection for the cache.
Related to this discussion:  https://github.com/projectdiscovery/nuclei/discussions/2277
Detail:
I added code like:
if b != nil {
    fmt.Printf("MIKE: b is %v\n", b)  // not string(b) because for some reason the lambda panicked
}

Then inspected the output to observe there was no Answer in the b passed to d.hm.Set in some cases.  I could not reliably reproduce the problem, but it was frequent enough to warrant the investigation and this PR.

Oops I picked the wrong source branch.


Subctl --image-override failed on x509: certificate signed by unknown authority

Running join on a cluster with access to registry-proxy.engineering.redhat.com:
Failed to pull image<EMAIL_ADDRESS>rpc error: code = Unknown desc = error pinging docker registry registry-proxy.engineering.redhat.com: Get https://registry-proxy.engineering.redhat.com/v2/: x509: certificate signed by unknown authority
subctl join   ./broker-info.subm --cable-driver libreswan   --ikeport 501 --nattport 4501 --enable-pod-debugging --ipsec-debug --health-check --image-override submariner-operator=registry-proxy.engineering.redhat.com/rh-osbs/rhacm2-tech-preview-submariner-rhel8-operator@sha256:c47d6c0fdb1a3be6d67014785ad2712fa890fcb02f02a7e0b36b5d08c862931f
14:21:53 * ./broker-info.subm says broker is at: https://api.nmanos-cluster-a.devcluster.openshift.com:6443
14:21:53  ��� Discovering network details  ...
14:21:53 * There are 1 labeled nodes in the cluster:
14:21:53   - default-cl1-l5mpb-worker-8d6gh
14:21:53  ��� Discovering network details
14:21:53     Discovered network details:
14:21:53         Network plugin:  OpenShiftSDN
14:21:53         Service CIDRs:   [<IP_ADDRESS>/16]
14:21:53         Cluster CIDRs:   [<IP_ADDRESS>/14]
14:21:57  ��� Discovering multi cluster details  ...
14:21:57  ��� Validating Globalnet configurations  ...
14:21:57  ��� Validating Globalnet configurations
14:21:57  ��� Assigning Globalnet IPs  ...
14:21:57  ��� Assigning Globalnet IPs
14:21:57  ��� Allocated GlobalCIDR: <IP_ADDRESS>/19
14:21:57  ��� Discovering multi cluster details
14:21:57  ��� Deploying the Submariner operator  ...
14:31:59  ��� Deploying the Submariner operator
14:31:59  ��� Created operator CRDs
14:31:59  ��� Created operator namespace: submariner-operator
14:31:59  ��� Created operator service account and role
14:31:59  ��� Created lighthouse service account and role
14:31:59  ��� Created Lighthouse service accounts and roles
14:31:59 Error deploying the operator: timed out waiting for the condition
14:31:59
14:31:59 subctl version: v0.8.0-rc1

Pod logs shows:
14:36:37 Events:
14:36:37   Type     Reason     Age                   From               Message
14:36:37   ----     ------     ----                  ----               -------
14:36:37   Normal   Scheduled  15m                   default-scheduler  Successfully assigned submariner-operator/submariner-operator-844678c6c6-rth5j to default-cl1-l5mpb-worker-8d6gh
14:36:37   Normal   Pulling    13m (x4 over 15m)     kubelet            Pulling image "registry-proxy.engineering.redhat.com/rh-osbs/rhacm2-tech-preview-submariner-rhel8-operator@sha256:c47d6c0fdb1a3be6d67014785ad2712fa890fcb02f02a7e0b36b5d08c862931f"
14:36:37   Warning  Failed     13m (x4 over 15m)     kubelet            Failed to pull image "registry-proxy.engineering.redhat.com/rh-osbs/rhacm2-tech-preview-submariner-rhel8-operator@sha256:c47d6c0fdb1a3be6d67014785ad2712fa890fcb02f02a7e0b36b5d08c862931f": rpc error: code = Unknown desc = error pinging docker registry registry-proxy.engineering.redhat.com: Get https://registry-proxy.engineering.redhat.com/v2/: x509: certificate signed by unknown authority
14:36:37   Warning  Failed     13m (x4 over 15m)     kubelet            Error: ErrImagePull
14:36:37   Normal   BackOff    10m (x21 over 15m)    kubelet            Back-off pulling image "registry-proxy.engineering.redhat.com/rh-osbs/rhacm2-tech-preview-submariner-rhel8-operator@sha256:c47d6c0fdb1a3be6d67014785ad2712fa890fcb02f02a7e0b36b5d08c862931f"
14:36:37   Warning  Failed     4m56s (x44 over 15m)  kubelet            Error: ImagePullBackOff

Environment:
subctl version: v0.8.0-rc1

@manosnoam we are well aware of the fact that you are blocked, but this is not a Submariner bug. @SteveMattar is on it, and will help you set up the environment properly. There is a bit of a learning curve for us also, as this is the first time you are testing with non-upstream images.


For Ping (Or HTTPs) Add a minimum response time - After that, warn, then service alert

⚠ Please verify that this feature request has NOT been suggested before.

[X] I checked and didn't find similar feature request

🏷 Feature Request Type
New Monitor
🔖 Feature description
there are people suggesting Ping TTL etc, but what I am suggesting is setting a minimum and maximum bar on response times.
For some failing services, their response times go through the roof, but still work. Its useful to know this before the inevitable crash.
✔ Solution
Add a minimum and maximum response time capability.
If response times (averaged over the number of times this happens - set by the user) fall within these bounds, alert with a WARNING
If response times fall outside of the Maximum, or service Fails, Alert with FAIL
❓ Alternatives
No response
📝 Additional Context
No response

Is this a duplicate of #459 and #1813? If it is, could you please close this issue to try and avoid cluttering the issue list too much.

Oops, sorry, didn't see those. Yes, will close.