Weedfs won't work without any reason

Hi,
Suddenly today I dont know  , my weedfs cluster became unhealthy,
There is some Errors that I would attach but I dont know how should I fix this issue.
I would appreciate if you could help me to solve the problem: @chrislusf
I have restarted master but it didn't have any effect
weed version:
version 8000GB 2.40 e983f91 linux amd64
Jun 22 08:53:51 weed-master-1 seaweedfs-master[21217]: I0622 08:53:51 21217 common.go:53] response method:GET URL:/dir/lookup?volumeId=286 with httpStatus:404 and JSON:{"volumeId":"286","error":"volume id 286 not found"} Jun 22 08:53:51 weed-master-1 seaweedfs-master[21217]: I0622 08:53:51 21217 common.go:53] response method:GET URL:/dir/lookup?volumeId=926 with httpStatus:404 and JSON:{"volumeId":"926","error":"volume id 926 not found"} Jun 22 08:53:51 weed-master-1 seaweedfs-master[21217]: I0622 08:53:51 21217 common.go:53] response method:GET URL:/dir/lookup?volumeId=22 with httpStatus:404 and JSON:{"volumeId":"22","error":"volume id 22 not found"}

{"volumeId":"286","error":"volume id 286 not found"}

seems like the volume file is not present on the filer directory.
how did you start your weedfs instance?

{"volumeId":"286","error":"volume id 286 not found"}

seems like the volume file is not present on the filer directory.
how did you start your weedfs instance?

There were working correctly for 3 monthm today suddenly weed baceame unhealthy,
and even I cant see weed-ui

{"volumeId":"286","error":"volume id 286 not found"}

seems like the volume file is not present on the filer directory.
how did you start your weedfs instance?

They were working correctly for 3 months, today suddenly weed baceame unhealthy,
and even I cant see weed-ui

Its okay don't worry be patient.
how did you start your seaweedfs instance?  the command?
are you in docker? are you in systemctl job?
can you see the logs?

restart the volume server(s)?

@chrislusf
These are my log:
Master continuously changed over and over till I restart all masters and then weed cluster became healthy,
I think there is a bug and problem with master election in weed:
Jun 22 06:25:02 weed-master-1 seaweedfs-master[1157]: I0622 06:25:02  1157 master_grpc_server.go:265] - client master@<IP_ADDRESS>:50400 Jun 22 06:25:02 weed-master-1 seaweedfs-master[1157]: I0622 06:25:02  1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:50400 Jun 22 06:25:20 weed-master-1 seaweedfs-master[1157]: I0622 06:25:20  1157 master_server.go:141] leader change event:  => weed-master-1.company.local:9333 Jun 22 06:25:20 weed-master-1 seaweedfs-master[1157]: I0622 06:25:20  1157 master_server.go:143] [ weed-master-1.company.local:9333 ] weed-master-1.company.local:9333 becomes leader. Jun 22 06:25:22 weed-master-1 seaweedfs-master[1157]: I0622 06:25:22  1157 master_grpc_server.go:265] - client master@[::1]:5294 Jun 22 06:25:22 weed-master-1 seaweedfs-master[1157]: I0622 06:25:22  1157 masterclient.go:126] redirected to leader weed-master-1.company.local:9333 Jun 22 06:25:22 weed-master-1 seaweedfs-master[1157]: I0622 06:25:22  1157 master_grpc_server.go:249] + client master@[::1]:5294 Jun 22 06:25:23 weed-master-1 seaweedfs-master[1157]: I0622 06:25:23  1157 master_grpc_server.go:265] - client master@<IP_ADDRESS>:50400 Jun 22 06:25:23 weed-master-1 seaweedfs-master[1157]: I0622 06:25:23  1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:50400 Jun 22 06:25:26 weed-master-1 seaweedfs-master[1157]: I0622 06:25:26  1157 master_grpc_server.go:265] - client master@<IP_ADDRESS>:46026 Jun 22 06:25:26 weed-master-1 seaweedfs-master[1157]: I0622 06:25:26  1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:46026 Jun 22 06:25:48 weed-master-1 seaweedfs-master[1157]: I0622 06:25:48  1157 master_server.go:141] leader change event:  => weed-master-1.company.local:9333 Jun 22 06:25:48 weed-master-1 seaweedfs-master[1157]: I0622 06:25:48  1157 master_server.go:143] [ weed-master-1.company.local:9333 ] weed-master-1.company.local:9333 becomes leader. Jun 22 06:25:49 weed-master-1 seaweedfs-master[1157]: I0622 06:25:49  1157 master_grpc_server.go:265] - client master@<IP_ADDRESS>:46026 Jun 22 06:25:50 weed-master-1 seaweedfs-master[1157]: I0622 06:25:50  1157 master_grpc_server.go:265] - client master@[::1]:5294 Jun 22 06:25:50 weed-master-1 seaweedfs-master[1157]: I0622 06:25:50  1157 masterclient.go:120] master masterClient failed to receive from weed-master-1.company.local:9333: EOF Jun 22 06:25:50 weed-master-1 seaweedfs-master[1157]: I0622 06:25:50  1157 masterclient.go:126] redirected to leader weed-master-1.company.local:9333 Jun 22 06:25:50 weed-master-1 seaweedfs-master[1157]: I0622 06:25:50  1157 master_grpc_server.go:249] + client master@[::1]:5294 Jun 22 06:25:50 weed-master-1 seaweedfs-master[1157]: I0622 06:25:50  1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:46026 Jun 22 06:25:51 weed-master-1 seaweedfs-master[1157]: I0622 06:25:51  1157 master_grpc_server.go:265] - client master@<IP_ADDRESS>:50400 Jun 22 06:25:52 weed-master-1 seaweedfs-master[1157]: I0622 06:25:52  1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:50400 Jun 22 06:26:17 weed-master-1 seaweedfs-master[1157]: I0622 06:26:17  1157 master_server.go:141] leader change event:  => weed-master-1.company.local:9333 Jun 22 06:26:17 weed-master-1 seaweedfs-master[1157]: I0622 06:26:17  1157 master_server.go:143] [ weed-master-1.company.local:9333 ] weed-master-1.company.local:9333 becomes leader. Jun 22 06:26:18 weed-master-1 seaweedfs-master[1157]: I0622 06:26:18  1157 master_grpc_server.go:265] - client master@[::1]:5294 Jun 22 06:26:18 weed-master-1 seaweedfs-master[1157]: I0622 06:26:18  1157 masterclient.go:120] master masterClient failed to receive from weed-master-1.company.local:9333: EOF Jun 22 06:26:18 weed-master-1 seaweedfs-master[1157]: I0622 06:26:18  1157 masterclient.go:126] redirected to leader weed-master-1.company.local:9333 Jun 22 06:26:18 weed-master-1 seaweedfs-master[1157]: I0622 06:26:18  1157 master_grpc_server.go:249] + client master@[::1]:5294 Jun 22 06:26:18 weed-master-1 seaweedfs-master[1157]: I0622 06:26:18  1157 master_grpc_server.go:265] - client master@<IP_ADDRESS>:46026 Jun 22 06:26:18 weed-master-1 seaweedfs-master[1157]: I0622 06:26:18  1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:46026 Jun 22 06:26:20 weed-master-1 seaweedfs-master[1157]: I0622 06:26:20  1157 master_grpc_server.go:265] - client master@<IP_ADDRESS>:50400 Jun 22 06:26:20 weed-master-1 seaweedfs-master[1157]: I0622 06:26:20  1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:50400 Jun 22 06:26:38 weed-master-1 seaweedfs-master[1157]: I0622 06:26:38  1157 master_server.go:141] leader change event:  => weed-master-1.company.local:9333 Jun 22 06:26:38 weed-master-1 seaweedfs-master[1157]: I0622 06:26:38  1157 master_server.go:143] [ weed-master-1.company.local:9333 ] weed-master-1.company.local:9333 becomes leader. Jun 22 06:26:39 weed-master-1 seaweedfs-master[1157]: I0622 06:26:39  1157 master_grpc_server.go:265] - client master@[::1]:5294 Jun 22 06:26:39 weed-master-1 seaweedfs-master[1157]: I0622 06:26:39  1157 masterclient.go:126] redirected to leader weed-master-1.company.local:9333 Jun 22 06:26:39 weed-master-1 seaweedfs-master[1157]: I0622 06:26:39  1157 master_grpc_server.go:249] + client master@[::1]:5294 Jun 22 06:26:39 weed-master-1 seaweedfs-master[1157]: I0622 06:26:39  1157 master_grpc_server.go:265] - client master@<IP_ADDRESS>:46026 Jun 22 06:26:39 weed-master-1 seaweedfs-master[1157]: I0622 06:26:39  1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:46026 Jun 22 06:26:41 weed-master-1 seaweedfs-master[1157]: I0622 06:26:41  1157 master_grpc_server.go:265] - client master@<IP_ADDRESS>:50400 Jun 22 06:26:41 weed-master-1 seaweedfs-master[1157]: I0622 06:26:41  1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:50400 Jun 22 06:27:06 weed-master-1 seaweedfs-master[1157]: I0622 06:27:06  1157 master_server.go:141] leader change event:  => weed-master-1.company.local:9333 Jun 22 06:27:06 weed-master-1 seaweedfs-master[1157]: I0622 06:27:06  1157 master_server.go:143] [ weed-master-1.company.local:9333 ] weed-master-1.company.local:9333 becomes leader. Jun 22 06:27:07 weed-master-1 seaweedfs-master[1157]: I0622 06:27:07  1157 master_grpc_server.go:265] - client master@[::1]:5294 Jun 22 06:27:07 weed-master-1 seaweedfs-master[1157]: I0622 06:27:07  1157 masterclient.go:120] master masterClient failed to receive from weed-master-1.company.local:9333: EOF Jun 22 06:27:07 weed-master-1 seaweedfs-master[1157]: I0622 06:27:07  1157 master_grpc_server.go:265] - client master@<IP_ADDRESS>:46026
and
`
Jun 22 06:29:17 weed-master-1 seaweedfs-master[1157]: I0622 06:29:17  1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:46026
Jun 22 06:29:19 weed-master-1 seaweedfs-master[1157]: I0622 06:29:19  1157 master_grpc_server.go:265] - client master@<IP_ADDRESS>:50400
Jun 22 06:29:19 weed-master-1 seaweedfs-master[1157]: I0622 06:29:19  1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:50400
Jun 22 06:29:24 weed-master-1 seaweedfs-master[1157]: I0622 06:29:24  1157 master_grpc_server.go:265] - client master@[::1]:5294
Jun 22 06:29:24 weed-master-1 seaweedfs-master[1157]: I0622 06:29:24  1157 masterclient.go:120] master masterClient failed to receive from weed-master-1.company.local:9333: EOF
Jun 22 06:29:24 weed-master-1 seaweedfs-master[1157]: I0622 06:29:24  1157 masterclient.go:126] redirected to leader weed-master-1.company.local:9333
Jun 22 06:29:24 weed-master-1 seaweedfs-master[1157]: I0622 06:29:24  1157 master_grpc_server.go:249] + client master@[::1]:5294
Jun 22 06:29:42 weed-master-1 seaweedfs-master[1157]: I0622 06:29:42  1157 master_server.go:141] leader change event:  => weed-master-1.company.local:9333
Jun 22 06:29:42 weed-master-1 seaweedfs-master[1157]: I0622 06:29:42  1157 master_server.go:143] [ weed-master-1.company.local:9333 ] weed-master-1.company.local:9333 becomes leader.
Jun 22 06:29:45 weed-master-1 seaweedfs-master[1157]: I0622 06:29:45  1157 master_grpc_server.go:265] - client master@<IP_ADDRESS>:46026
Jun 22 06:29:45 weed-master-1 seaweedfs-master[1157]: I0622 06:29:45  1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:46026
Jun 22 06:29:47 weed-master-1 seaweedfs-master[1157]: I0622 06:29:47  1157 master_grpc_server.go:265] - client master@[::1]:5294
Jun 22 06:29:47 weed-master-1 seaweedfs-master[1157]: I0622 06:29:47  1157 masterclient.go:120] master masterClient failed to receive from weed-master-1.company.local:9333: EOF
Jun 22 06:29:47 weed-master-1 seaweedfs-master[1157]: I0622 06:29:47  1157 master_grpc_server.go:265] - client master@<IP_ADDRESS>:50400
Jun 22 06:29:47 weed-master-1 seaweedfs-master[1157]: I0622 06:29:47  1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:50400
Jun 22 06:29:48 weed-master-1 seaweedfs-master[1157]: I0622 06:29:48  1157 master_grpc_server.go:249] + client master@[::1]:5294
Jun 22 06:30:06 weed-master-1 seaweedfs-master[1157]: I0622 06:30:06  1157 master_server.go:141] leader change event:  => weed-master-1.company.local:9333
Jun 22 06:30:06 weed-master-1 seaweedfs-master[1157]: I0622 06:30:06  1157 master_server.go:143] [ weed-master-1.company.local:9333 ] weed-master-1.company.local:9333 becomes leader.
Jun 22 06:30:10 weed-master-1 seaweedfs-master[1157]: I0622 06:30:10  1157 master_grpc_server.go:265] - client master@<IP_ADDRESS>:50400
Jun 22 06:30:10 weed-master-1 seaweedfs-master[1157]: I0622 06:30:10  1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:50400
Jun 22 06:30:11 weed-master-1 seaweedfs-master[1157]: I0622 06:30:11  1157 master_grpc_server.go:265] - client master@[::1]:5294
Jun 22 06:30:11 weed-master-1 seaweedfs-master[1157]: I0622 06:30:11  1157 masterclient.go:120] master masterClient failed to receive from weed-master-1.company.local:9333: EOF
Jun 22 06:30:11 weed-master-1 seaweedfs-master[1157]: I0622 06:30:11  1157 masterclient.go:126] redirected to leader weed-master-1.company.local:9333
`
and some part of logs said: response method:GET :
`
Jun 22 08:40:42 weed-master-1 seaweedfs-master[1157]: I0622 08:40:42  1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:46026
Jun 22 08:40:42 weed-master-1 seaweedfs-master[1157]: I0622 08:40:42  1157 master_grpc_server.go:265] - client master@<IP_ADDRESS>:50960
Jun 22 08:40:42 weed-master-1 seaweedfs-master[1157]: I0622 08:40:42  1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:50960
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04  1157 master_grpc_server.go:265] - client master@[::1]:5294
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04  1157 masterclient.go:120] master masterClient failed to receive from weed-master-1.company.local:9333: EOF
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04  1157 masterclient.go:126] redirected to leader weed-master-1.company.local:9333
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04  1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=830 with httpStatus:404 and JSON:{"volumeId":"830","error":"volume id 830 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04  1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=21 with httpStatus:404 and JSON:{"volumeId":"21","error":"volume id 21 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04  1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=867 with httpStatus:404 and JSON:{"volumeId":"867","error":"volume id 867 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04  1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=869 with httpStatus:404 and JSON:{"volumeId":"869","error":"volume id 869 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04  1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=12 with httpStatus:404 and JSON:{"volumeId":"12","error":"volume id 12 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04  1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=12 with httpStatus:404 and JSON:{"volumeId":"12","error":"volume id 12 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04  1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=927 with httpStatus:404 and JSON:{"volumeId":"927","error":"volume id 927 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04  1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=35 with httpStatus:404 and JSON:{"volumeId":"35","error":"volume id 35 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04  1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=740 with httpStatus:404 and JSON:{"volumeId":"740","error":"volume id 740 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04  1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=328 with httpStatus:404 and JSON:{"volumeId":"328","error":"volume id 328 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04  1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=84 with httpStatus:404 and JSON:{"volumeId":"84","error":"volume id 84 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04  1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=237 with httpStatus:404 and JSON:{"volumeId":"237","error":"volume id 237 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04  1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=562 with httpStatus:404 and JSON:{"volumeId":"562","error":"volume id 562 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04  1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=19 with httpStatus:404 and JSON:{"volumeId":"19","error":"volume id 19 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04  1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=20 with httpStatus:404 and JSON:{"volumeId":"20","error":"volume id 20 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04  1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=80 with httpStatus:404 and JSON:{"volumeId":"80","error":"volume id 80 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04  1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=960 with httpStatus:404 and JSON:{"volumeId":"960","error":"volume id 960 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04  1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=960 with httpStatus:404 and JSON:{"volumeId":"960","error":"volume id 960 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04  1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=956 with httpStatus:404 and JSON:{"volumeId":"956","error":"volume id 956 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04  1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=19 with httpStatus:404 and JSON:{"volumeId":"19","error":"volume id 19 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04  1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=236 with httpStatus:404 and JSON:{"volumeId":"236","error":"volume id 236 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04  1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=371 with httpStatus:404 and JSON:{"volumeId":"371","error":"volume id 371 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04  1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=960 with httpStatus:404 and JSON:{"volumeId":"960","error":"volume id 960 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04  1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=328 with httpStatus:404 and JSON:{"volumeId":"328","error":"volume id 328 not found"}
`

restart the volume server(s)?

No I restart all Masters

{"volumeId":"286","error":"volume id 286 not found"}

seems like the volume file is not present on the filer directory.
how did you start your weedfs instance?

They were working correctly for 3 months, today suddenly weed baceame unhealthy,
and even I cant see weed-ui

Its okay don't worry be patient.
how did you start your seaweedfs instance? the command?
are you in docker? are you in systemctl job?
can you see the logs?

I'm using systemctl job
Command to run weed cluster in systemd unit file:
ExecStart=/usr/local/bin/weed master -mdir="/etc/master" -ip=weed-master-1.company.local -port=9333 -peers="weed-master-1.company.local:9333,weed-master-2.company.local:9333,weed-master-3.company.local:9333" -defaultReplication=010 -volumeSizeLimitMB=102400 -metrics.address=weed-monitoring:9091

@chrislusf
These are my log:
Master continuously changed over and over till I restart all masters and then weed cluster became healthy,
I think there is a bug and problem with master election in weed:
Jun 22 06:25:02 weed-master-1 seaweedfs-master[1157]: I0622 06:25:02 1157 master_grpc_server.go:265] - client master@<IP_ADDRESS>:50400 Jun 22 06:25:02 weed-master-1 seaweedfs-master[1157]: I0622 06:25:02 1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:50400 Jun 22 06:25:20 weed-master-1 seaweedfs-master[1157]: I0622 06:25:20 1157 master_server.go:141] leader change event: => weed-master-1.company.local:9333 Jun 22 06:25:20 weed-master-1 seaweedfs-master[1157]: I0622 06:25:20 1157 master_server.go:143] [ weed-master-1.company.local:9333 ] weed-master-1.company.local:9333 becomes leader. Jun 22 06:25:22 weed-master-1 seaweedfs-master[1157]: I0622 06:25:22 1157 master_grpc_server.go:265] - client master@[::1]:5294 Jun 22 06:25:22 weed-master-1 seaweedfs-master[1157]: I0622 06:25:22 1157 masterclient.go:126] redirected to leader weed-master-1.company.local:9333 Jun 22 06:25:22 weed-master-1 seaweedfs-master[1157]: I0622 06:25:22 1157 master_grpc_server.go:249] + client master@[::1]:5294 Jun 22 06:25:23 weed-master-1 seaweedfs-master[1157]: I0622 06:25:23 1157 master_grpc_server.go:265] - client master@<IP_ADDRESS>:50400 Jun 22 06:25:23 weed-master-1 seaweedfs-master[1157]: I0622 06:25:23 1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:50400 Jun 22 06:25:26 weed-master-1 seaweedfs-master[1157]: I0622 06:25:26 1157 master_grpc_server.go:265] - client master@<IP_ADDRESS>:46026 Jun 22 06:25:26 weed-master-1 seaweedfs-master[1157]: I0622 06:25:26 1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:46026 Jun 22 06:25:48 weed-master-1 seaweedfs-master[1157]: I0622 06:25:48 1157 master_server.go:141] leader change event: => weed-master-1.company.local:9333 Jun 22 06:25:48 weed-master-1 seaweedfs-master[1157]: I0622 06:25:48 1157 master_server.go:143] [ weed-master-1.company.local:9333 ] weed-master-1.company.local:9333 becomes leader. Jun 22 06:25:49 weed-master-1 seaweedfs-master[1157]: I0622 06:25:49 1157 master_grpc_server.go:265] - client master@<IP_ADDRESS>:46026 Jun 22 06:25:50 weed-master-1 seaweedfs-master[1157]: I0622 06:25:50 1157 master_grpc_server.go:265] - client master@[::1]:5294 Jun 22 06:25:50 weed-master-1 seaweedfs-master[1157]: I0622 06:25:50 1157 masterclient.go:120] master masterClient failed to receive from weed-master-1.company.local:9333: EOF Jun 22 06:25:50 weed-master-1 seaweedfs-master[1157]: I0622 06:25:50 1157 masterclient.go:126] redirected to leader weed-master-1.company.local:9333 Jun 22 06:25:50 weed-master-1 seaweedfs-master[1157]: I0622 06:25:50 1157 master_grpc_server.go:249] + client master@[::1]:5294 Jun 22 06:25:50 weed-master-1 seaweedfs-master[1157]: I0622 06:25:50 1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:46026 Jun 22 06:25:51 weed-master-1 seaweedfs-master[1157]: I0622 06:25:51 1157 master_grpc_server.go:265] - client master@<IP_ADDRESS>:50400 Jun 22 06:25:52 weed-master-1 seaweedfs-master[1157]: I0622 06:25:52 1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:50400 Jun 22 06:26:17 weed-master-1 seaweedfs-master[1157]: I0622 06:26:17 1157 master_server.go:141] leader change event: => weed-master-1.company.local:9333 Jun 22 06:26:17 weed-master-1 seaweedfs-master[1157]: I0622 06:26:17 1157 master_server.go:143] [ weed-master-1.company.local:9333 ] weed-master-1.company.local:9333 becomes leader. Jun 22 06:26:18 weed-master-1 seaweedfs-master[1157]: I0622 06:26:18 1157 master_grpc_server.go:265] - client master@[::1]:5294 Jun 22 06:26:18 weed-master-1 seaweedfs-master[1157]: I0622 06:26:18 1157 masterclient.go:120] master masterClient failed to receive from weed-master-1.company.local:9333: EOF Jun 22 06:26:18 weed-master-1 seaweedfs-master[1157]: I0622 06:26:18 1157 masterclient.go:126] redirected to leader weed-master-1.company.local:9333 Jun 22 06:26:18 weed-master-1 seaweedfs-master[1157]: I0622 06:26:18 1157 master_grpc_server.go:249] + client master@[::1]:5294 Jun 22 06:26:18 weed-master-1 seaweedfs-master[1157]: I0622 06:26:18 1157 master_grpc_server.go:265] - client master@<IP_ADDRESS>:46026 Jun 22 06:26:18 weed-master-1 seaweedfs-master[1157]: I0622 06:26:18 1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:46026 Jun 22 06:26:20 weed-master-1 seaweedfs-master[1157]: I0622 06:26:20 1157 master_grpc_server.go:265] - client master@<IP_ADDRESS>:50400 Jun 22 06:26:20 weed-master-1 seaweedfs-master[1157]: I0622 06:26:20 1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:50400 Jun 22 06:26:38 weed-master-1 seaweedfs-master[1157]: I0622 06:26:38 1157 master_server.go:141] leader change event: => weed-master-1.company.local:9333 Jun 22 06:26:38 weed-master-1 seaweedfs-master[1157]: I0622 06:26:38 1157 master_server.go:143] [ weed-master-1.company.local:9333 ] weed-master-1.company.local:9333 becomes leader. Jun 22 06:26:39 weed-master-1 seaweedfs-master[1157]: I0622 06:26:39 1157 master_grpc_server.go:265] - client master@[::1]:5294 Jun 22 06:26:39 weed-master-1 seaweedfs-master[1157]: I0622 06:26:39 1157 masterclient.go:126] redirected to leader weed-master-1.company.local:9333 Jun 22 06:26:39 weed-master-1 seaweedfs-master[1157]: I0622 06:26:39 1157 master_grpc_server.go:249] + client master@[::1]:5294 Jun 22 06:26:39 weed-master-1 seaweedfs-master[1157]: I0622 06:26:39 1157 master_grpc_server.go:265] - client master@<IP_ADDRESS>:46026 Jun 22 06:26:39 weed-master-1 seaweedfs-master[1157]: I0622 06:26:39 1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:46026 Jun 22 06:26:41 weed-master-1 seaweedfs-master[1157]: I0622 06:26:41 1157 master_grpc_server.go:265] - client master@<IP_ADDRESS>:50400 Jun 22 06:26:41 weed-master-1 seaweedfs-master[1157]: I0622 06:26:41 1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:50400 Jun 22 06:27:06 weed-master-1 seaweedfs-master[1157]: I0622 06:27:06 1157 master_server.go:141] leader change event: => weed-master-1.company.local:9333 Jun 22 06:27:06 weed-master-1 seaweedfs-master[1157]: I0622 06:27:06 1157 master_server.go:143] [ weed-master-1.company.local:9333 ] weed-master-1.company.local:9333 becomes leader. Jun 22 06:27:07 weed-master-1 seaweedfs-master[1157]: I0622 06:27:07 1157 master_grpc_server.go:265] - client master@[::1]:5294 Jun 22 06:27:07 weed-master-1 seaweedfs-master[1157]: I0622 06:27:07 1157 masterclient.go:120] master masterClient failed to receive from weed-master-1.company.local:9333: EOF Jun 22 06:27:07 weed-master-1 seaweedfs-master[1157]: I0622 06:27:07 1157 master_grpc_server.go:265] - client master@<IP_ADDRESS>:46026
and
`
Jun 22 06:29:17 weed-master-1 seaweedfs-master[1157]: I0622 06:29:17 1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:46026
Jun 22 06:29:19 weed-master-1 seaweedfs-master[1157]: I0622 06:29:19 1157 master_grpc_server.go:265] - client master@<IP_ADDRESS>:50400
Jun 22 06:29:19 weed-master-1 seaweedfs-master[1157]: I0622 06:29:19 1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:50400
Jun 22 06:29:24 weed-master-1 seaweedfs-master[1157]: I0622 06:29:24 1157 master_grpc_server.go:265] - client master@[::1]:5294
Jun 22 06:29:24 weed-master-1 seaweedfs-master[1157]: I0622 06:29:24 1157 masterclient.go:120] master masterClient failed to receive from weed-master-1.company.local:9333: EOF
Jun 22 06:29:24 weed-master-1 seaweedfs-master[1157]: I0622 06:29:24 1157 masterclient.go:126] redirected to leader weed-master-1.company.local:9333
Jun 22 06:29:24 weed-master-1 seaweedfs-master[1157]: I0622 06:29:24 1157 master_grpc_server.go:249] + client master@[::1]:5294
Jun 22 06:29:42 weed-master-1 seaweedfs-master[1157]: I0622 06:29:42 1157 master_server.go:141] leader change event: => weed-master-1.company.local:9333
Jun 22 06:29:42 weed-master-1 seaweedfs-master[1157]: I0622 06:29:42 1157 master_server.go:143] [ weed-master-1.company.local:9333 ] weed-master-1.company.local:9333 becomes leader.
Jun 22 06:29:45 weed-master-1 seaweedfs-master[1157]: I0622 06:29:45 1157 master_grpc_server.go:265] - client master@<IP_ADDRESS>:46026
Jun 22 06:29:45 weed-master-1 seaweedfs-master[1157]: I0622 06:29:45 1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:46026
Jun 22 06:29:47 weed-master-1 seaweedfs-master[1157]: I0622 06:29:47 1157 master_grpc_server.go:265] - client master@[::1]:5294
Jun 22 06:29:47 weed-master-1 seaweedfs-master[1157]: I0622 06:29:47 1157 masterclient.go:120] master masterClient failed to receive from weed-master-1.company.local:9333: EOF
Jun 22 06:29:47 weed-master-1 seaweedfs-master[1157]: I0622 06:29:47 1157 master_grpc_server.go:265] - client master@<IP_ADDRESS>:50400
Jun 22 06:29:47 weed-master-1 seaweedfs-master[1157]: I0622 06:29:47 1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:50400
Jun 22 06:29:48 weed-master-1 seaweedfs-master[1157]: I0622 06:29:48 1157 master_grpc_server.go:249] + client master@[::1]:5294
Jun 22 06:30:06 weed-master-1 seaweedfs-master[1157]: I0622 06:30:06 1157 master_server.go:141] leader change event: => weed-master-1.company.local:9333
Jun 22 06:30:06 weed-master-1 seaweedfs-master[1157]: I0622 06:30:06 1157 master_server.go:143] [ weed-master-1.company.local:9333 ] weed-master-1.company.local:9333 becomes leader.
Jun 22 06:30:10 weed-master-1 seaweedfs-master[1157]: I0622 06:30:10 1157 master_grpc_server.go:265] - client master@<IP_ADDRESS>:50400
Jun 22 06:30:10 weed-master-1 seaweedfs-master[1157]: I0622 06:30:10 1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:50400
Jun 22 06:30:11 weed-master-1 seaweedfs-master[1157]: I0622 06:30:11 1157 master_grpc_server.go:265] - client master@[::1]:5294
Jun 22 06:30:11 weed-master-1 seaweedfs-master[1157]: I0622 06:30:11 1157 masterclient.go:120] master masterClient failed to receive from weed-master-1.company.local:9333: EOF
Jun 22 06:30:11 weed-master-1 seaweedfs-master[1157]: I0622 06:30:11 1157 masterclient.go:126] redirected to leader weed-master-1.company.local:9333
`
and some part of logs said: response method:GET :
`
Jun 22 08:40:42 weed-master-1 seaweedfs-master[1157]: I0622 08:40:42 1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:46026
Jun 22 08:40:42 weed-master-1 seaweedfs-master[1157]: I0622 08:40:42 1157 master_grpc_server.go:265] - client master@<IP_ADDRESS>:50960
Jun 22 08:40:42 weed-master-1 seaweedfs-master[1157]: I0622 08:40:42 1157 master_grpc_server.go:249] + client master@<IP_ADDRESS>:50960
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 master_grpc_server.go:265] - client master@[::1]:5294
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 masterclient.go:120] master masterClient failed to receive from weed-master-1.company.local:9333: EOF
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 masterclient.go:126] redirected to leader weed-master-1.company.local:9333
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=830 with httpStatus:404 and JSON:{"volumeId":"830","error":"volume id 830 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=21 with httpStatus:404 and JSON:{"volumeId":"21","error":"volume id 21 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=867 with httpStatus:404 and JSON:{"volumeId":"867","error":"volume id 867 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=869 with httpStatus:404 and JSON:{"volumeId":"869","error":"volume id 869 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=12 with httpStatus:404 and JSON:{"volumeId":"12","error":"volume id 12 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=12 with httpStatus:404 and JSON:{"volumeId":"12","error":"volume id 12 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=927 with httpStatus:404 and JSON:{"volumeId":"927","error":"volume id 927 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=35 with httpStatus:404 and JSON:{"volumeId":"35","error":"volume id 35 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=740 with httpStatus:404 and JSON:{"volumeId":"740","error":"volume id 740 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=328 with httpStatus:404 and JSON:{"volumeId":"328","error":"volume id 328 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=84 with httpStatus:404 and JSON:{"volumeId":"84","error":"volume id 84 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=237 with httpStatus:404 and JSON:{"volumeId":"237","error":"volume id 237 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=562 with httpStatus:404 and JSON:{"volumeId":"562","error":"volume id 562 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=19 with httpStatus:404 and JSON:{"volumeId":"19","error":"volume id 19 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=20 with httpStatus:404 and JSON:{"volumeId":"20","error":"volume id 20 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=80 with httpStatus:404 and JSON:{"volumeId":"80","error":"volume id 80 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=960 with httpStatus:404 and JSON:{"volumeId":"960","error":"volume id 960 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=960 with httpStatus:404 and JSON:{"volumeId":"960","error":"volume id 960 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=956 with httpStatus:404 and JSON:{"volumeId":"956","error":"volume id 956 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=19 with httpStatus:404 and JSON:{"volumeId":"19","error":"volume id 19 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=236 with httpStatus:404 and JSON:{"volumeId":"236","error":"volume id 236 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=371 with httpStatus:404 and JSON:{"volumeId":"371","error":"volume id 371 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=960 with httpStatus:404 and JSON:{"volumeId":"960","error":"volume id 960 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=328 with httpStatus:404 and JSON:{"volumeId":"328","error":"volume id 328 not found"}
`

These are just part of my weed cluster logs , because they were repetitive so I just copied the main part

Added a possible fix.


Version 2.0.2 has insufficient privileges for service account

Description
When installing version 2.0.2 the operator-controller-manager container crashes continuously. It seems that the service account is not authorised to list the various different resources.
To Reproduce

Install version 2.0.2 operator: helm install opensearch-operator opensearch-operator/opensearch-operator -n opensearch-operator-issue --version 2.0.2
View the logs for the operator-controller-manager container: k logs -l control-plane=controller-manager -c operator-controller-manager

Logs from container

I0816 11:07:34.977462       1 request.go:665] Waited for 1.023143129s due to client-side throttling, not priority and fairness, request: GET:https://<IP_ADDRESS>:443/apis/kafka.strimzi.io/v1beta1?timeout=32s
1.6606480551341822e+09	INFO	controller-runtime.metrics	Metrics server is starting to listen	{"addr": "<IP_ADDRESS>:8080"}
1.6606480551345303e+09	INFO	setup	starting manager
1.660648055134819e+09	INFO	Starting server	{"kind": "health probe", "addr": "[::]:8081"}
1.660648055134827e+09	INFO	Starting server	{"path": "/metrics", "kind": "metrics", "addr": "<IP_ADDRESS>:8080"}
I0816 11:07:35.134942       1 leaderelection.go:248] attempting to acquire leader lease opensearch-operator/a867c7dc.opensearch.opster.io...
I0816 11:07:50.847194       1 leaderelection.go:258] successfully acquired lease opensearch-operator/a867c7dc.opensearch.opster.io
1.6606480708473072e+09	DEBUG	events	Normal	{"object": {"kind":"ConfigMap","namespace":"opensearch-operator","name":"a867c7dc.opensearch.opster.io","uid":"a087bb5b-2d04-4a6d-8d86-c909deb71ffc","apiVersion":"v1","resourceVersion":"953403"}, "reason": "LeaderElection", "message": "opensearch-operator-controller-manager-749cbc65fd-qlzjc_55afdda9-abcd-4f78-b95c-82bb4d2348ad became leader"}
1.6606480708475082e+09	DEBUG	events	Normal	{"object": {"kind":"Lease","namespace":"opensearch-operator","name":"a867c7dc.opensearch.opster.io","uid":"6f28024a-eebc-42ef-beb3-91fe041a84f3","apiVersion":"coordination.k8s.io/v1","resourceVersion":"953404"}, "reason": "LeaderElection", "message": "opensearch-operator-controller-manager-749cbc65fd-qlzjc_55afdda9-abcd-4f78-b95c-82bb4d2348ad became leader"}
1.6606480708474424e+09	INFO	controller.opensearchrole	Starting EventSource	{"reconciler group": "opensearch.opster.io", "reconciler kind": "OpensearchRole", "source": "kind source: *v1.OpensearchRole"}
1.6606480708475385e+09	INFO	controller.opensearchrole	Starting EventSource	{"reconciler group": "opensearch.opster.io", "reconciler kind": "OpensearchRole", "source": "kind source: *v1.OpenSearchCluster"}
1.6606480708475494e+09	INFO	controller.opensearchrole	Starting Controller	{"reconciler group": "opensearch.opster.io", "reconciler kind": "OpensearchRole"}
1.6606480708474584e+09	INFO	controller.opensearchuser	Starting EventSource	{"reconciler group": "opensearch.opster.io", "reconciler kind": "OpensearchUser", "source": "kind source: *v1.OpensearchUser"}
1.6606480708476105e+09	INFO	controller.opensearchuserrolebinding	Starting EventSource	{"reconciler group": "opensearch.opster.io", "reconciler kind": "OpensearchUserRoleBinding", "source": "kind source: *v1.OpensearchUserRoleBinding"}
1.6606480708474417e+09	INFO	controller.opensearchcluster	Starting EventSource	{"reconciler group": "opensearch.opster.io", "reconciler kind": "OpenSearchCluster", "source": "kind source: *v1.OpenSearchCluster"}
1.660648070847642e+09	INFO	controller.opensearchuser	Starting EventSource	{"reconciler group": "opensearch.opster.io", "reconciler kind": "OpensearchUser", "source": "kind source: *v1.OpenSearchCluster"}
1.6606480708476424e+09	INFO	controller.opensearchuserrolebinding	Starting EventSource	{"reconciler group": "opensearch.opster.io", "reconciler kind": "OpensearchUserRoleBinding", "source": "kind source: *v1.OpenSearchCluster"}
1.6606480708476527e+09	INFO	controller.opensearchuser	Starting Controller	{"reconciler group": "opensearch.opster.io", "reconciler kind": "OpensearchUser"}
1.6606480708476562e+09	INFO	controller.opensearchuserrolebinding	Starting Controller	{"reconciler group": "opensearch.opster.io", "reconciler kind": "OpensearchUserRoleBinding"}
1.660648070847673e+09	INFO	controller.opensearchcluster	Starting EventSource	{"reconciler group": "opensearch.opster.io", "reconciler kind": "OpenSearchCluster", "source": "kind source: *v1.Pod"}
1.6606480708476892e+09	INFO	controller.opensearchcluster	Starting EventSource	{"reconciler group": "opensearch.opster.io", "reconciler kind": "OpenSearchCluster", "source": "kind source: *v1.Secret"}
1.6606480708476994e+09	INFO	controller.opensearchcluster	Starting EventSource	{"reconciler group": "opensearch.opster.io", "reconciler kind": "OpenSearchCluster", "source": "kind source: *v1.ConfigMap"}
1.660648070847708e+09	INFO	controller.opensearchcluster	Starting EventSource	{"reconciler group": "opensearch.opster.io", "reconciler kind": "OpenSearchCluster", "source": "kind source: *v1.Service"}
1.660648070847736e+09	INFO	controller.opensearchcluster	Starting EventSource	{"reconciler group": "opensearch.opster.io", "reconciler kind": "OpenSearchCluster", "source": "kind source: *v1.Deployment"}
1.660648070847746e+09	INFO	controller.opensearchcluster	Starting EventSource	{"reconciler group": "opensearch.opster.io", "reconciler kind": "OpenSearchCluster", "source": "kind source: *v1.StatefulSet"}
1.6606480708477461e+09	INFO	controller.opensearchcluster	Starting Controller	{"reconciler group": "opensearch.opster.io", "reconciler kind": "OpenSearchCluster"}
W0816 11:07:50.854783       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: failed to list *v1.OpensearchRole: opensearchroles.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchroles" in API group "opensearch.opster.io" at the cluster scope
E0816 11:07:50.854822       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: Failed to watch *v1.OpensearchRole: failed to list *v1.OpensearchRole: opensearchroles.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchroles" in API group "opensearch.opster.io" at the cluster scope
W0816 11:07:50.868590       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: failed to list *v1.OpensearchUserRoleBinding: opensearchuserrolebindings.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchuserrolebindings" in API group "opensearch.opster.io" at the cluster scope
E0816 11:07:50.868612       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: Failed to watch *v1.OpensearchUserRoleBinding: failed to list *v1.OpensearchUserRoleBinding: opensearchuserrolebindings.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchuserrolebindings" in API group "opensearch.opster.io" at the cluster scope
W0816 11:07:50.870149       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: failed to list *v1.OpensearchUser: opensearchusers.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchusers" in API group "opensearch.opster.io" at the cluster scope
E0816 11:07:50.870217       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: Failed to watch *v1.OpensearchUser: failed to list *v1.OpensearchUser: opensearchusers.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchusers" in API group "opensearch.opster.io" at the cluster scope
W0816 11:07:51.776898       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: failed to list *v1.OpensearchUserRoleBinding: opensearchuserrolebindings.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchuserrolebindings" in API group "opensearch.opster.io" at the cluster scope
E0816 11:07:51.776932       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: Failed to watch *v1.OpensearchUserRoleBinding: failed to list *v1.OpensearchUserRoleBinding: opensearchuserrolebindings.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchuserrolebindings" in API group "opensearch.opster.io" at the cluster scope
W0816 11:07:52.215575       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: failed to list *v1.OpensearchUser: opensearchusers.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchusers" in API group "opensearch.opster.io" at the cluster scope
E0816 11:07:52.215688       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: Failed to watch *v1.OpensearchUser: failed to list *v1.OpensearchUser: opensearchusers.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchusers" in API group "opensearch.opster.io" at the cluster scope
W0816 11:07:52.324137       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: failed to list *v1.OpensearchRole: opensearchroles.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchroles" in API group "opensearch.opster.io" at the cluster scope
E0816 11:07:52.324171       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: Failed to watch *v1.OpensearchRole: failed to list *v1.OpensearchRole: opensearchroles.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchroles" in API group "opensearch.opster.io" at the cluster scope
W0816 11:07:54.350316       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: failed to list *v1.OpensearchUser: opensearchusers.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchusers" in API group "opensearch.opster.io" at the cluster scope
E0816 11:07:54.350363       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: Failed to watch *v1.OpensearchUser: failed to list *v1.OpensearchUser: opensearchusers.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchusers" in API group "opensearch.opster.io" at the cluster scope
W0816 11:07:54.351623       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: failed to list *v1.OpensearchRole: opensearchroles.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchroles" in API group "opensearch.opster.io" at the cluster scope
E0816 11:07:54.351653       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: Failed to watch *v1.OpensearchRole: failed to list *v1.OpensearchRole: opensearchroles.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchroles" in API group "opensearch.opster.io" at the cluster scope
W0816 11:07:54.663603       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: failed to list *v1.OpensearchUserRoleBinding: opensearchuserrolebindings.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchuserrolebindings" in API group "opensearch.opster.io" at the cluster scope
E0816 11:07:54.663636       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: Failed to watch *v1.OpensearchUserRoleBinding: failed to list *v1.OpensearchUserRoleBinding: opensearchuserrolebindings.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchuserrolebindings" in API group "opensearch.opster.io" at the cluster scope
W0816 11:07:59.334585       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: failed to list *v1.OpensearchUser: opensearchusers.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchusers" in API group "opensearch.opster.io" at the cluster scope
E0816 11:07:59.334616       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: Failed to watch *v1.OpensearchUser: failed to list *v1.OpensearchUser: opensearchusers.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchusers" in API group "opensearch.opster.io" at the cluster scope
W0816 11:07:59.652854       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: failed to list *v1.OpensearchUserRoleBinding: opensearchuserrolebindings.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchuserrolebindings" in API group "opensearch.opster.io" at the cluster scope
E0816 11:07:59.652871       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: Failed to watch *v1.OpensearchUserRoleBinding: failed to list *v1.OpensearchUserRoleBinding: opensearchuserrolebindings.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchuserrolebindings" in API group "opensearch.opster.io" at the cluster scope
W0816 11:08:00.382839       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: failed to list *v1.OpensearchRole: opensearchroles.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchroles" in API group "opensearch.opster.io" at the cluster scope
E0816 11:08:00.382858       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: Failed to watch *v1.OpensearchRole: failed to list *v1.OpensearchRole: opensearchroles.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchroles" in API group "opensearch.opster.io" at the cluster scope
W0816 11:08:06.566381       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: failed to list *v1.OpensearchUser: opensearchusers.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchusers" in API group "opensearch.opster.io" at the cluster scope
E0816 11:08:06.566414       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: Failed to watch *v1.OpensearchUser: failed to list *v1.OpensearchUser: opensearchusers.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchusers" in API group "opensearch.opster.io" at the cluster scope
W0816 11:08:06.988674       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: failed to list *v1.OpensearchRole: opensearchroles.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchroles" in API group "opensearch.opster.io" at the cluster scope
E0816 11:08:06.988717       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: Failed to watch *v1.OpensearchRole: failed to list *v1.OpensearchRole: opensearchroles.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchroles" in API group "opensearch.opster.io" at the cluster scope
W0816 11:08:07.250944       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: failed to list *v1.OpensearchUserRoleBinding: opensearchuserrolebindings.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchuserrolebindings" in API group "opensearch.opster.io" at the cluster scope
E0816 11:08:07.250984       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: Failed to watch *v1.OpensearchUserRoleBinding: failed to list *v1.OpensearchUserRoleBinding: opensearchuserrolebindings.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchuserrolebindings" in API group "opensearch.opster.io" at the cluster scope
W0816 11:08:22.841977       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: failed to list *v1.OpensearchUserRoleBinding: opensearchuserrolebindings.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchuserrolebindings" in API group "opensearch.opster.io" at the cluster scope
E0816 11:08:22.842018       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: Failed to watch *v1.OpensearchUserRoleBinding: failed to list *v1.OpensearchUserRoleBinding: opensearchuserrolebindings.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchuserrolebindings" in API group "opensearch.opster.io" at the cluster scope
W0816 11:08:29.160502       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: failed to list *v1.OpensearchUser: opensearchusers.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchusers" in API group "opensearch.opster.io" at the cluster scope
E0816 11:08:29.160520       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: Failed to watch *v1.OpensearchUser: failed to list *v1.OpensearchUser: opensearchusers.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchusers" in API group "opensearch.opster.io" at the cluster scope
W0816 11:08:30.485528       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: failed to list *v1.OpensearchRole: opensearchroles.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchroles" in API group "opensearch.opster.io" at the cluster scope
E0816 11:08:30.485570       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: Failed to watch *v1.OpensearchRole: failed to list *v1.OpensearchRole: opensearchroles.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchroles" in API group "opensearch.opster.io" at the cluster scope
W0816 11:09:08.102664       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: failed to list *v1.OpensearchUser: opensearchusers.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchusers" in API group "opensearch.opster.io" at the cluster scope
E0816 11:09:08.102701       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: Failed to watch *v1.OpensearchUser: failed to list *v1.OpensearchUser: opensearchusers.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchusers" in API group "opensearch.opster.io" at the cluster scope
W0816 11:09:13.136275       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: failed to list *v1.OpensearchUserRoleBinding: opensearchuserrolebindings.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchuserrolebindings" in API group "opensearch.opster.io" at the cluster scope
E0816 11:09:13.136325       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: Failed to watch *v1.OpensearchUserRoleBinding: failed to list *v1.OpensearchUserRoleBinding: opensearchuserrolebindings.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchuserrolebindings" in API group "opensearch.opster.io" at the cluster scope
W0816 11:09:20.969079       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: failed to list *v1.OpensearchRole: opensearchroles.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchroles" in API group "opensearch.opster.io" at the cluster scope
E0816 11:09:20.969113       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: Failed to watch *v1.OpensearchRole: failed to list *v1.OpensearchRole: opensearchroles.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchroles" in API group "opensearch.opster.io" at the cluster scope
W0816 11:09:47.427315       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: failed to list *v1.OpensearchUserRoleBinding: opensearchuserrolebindings.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchuserrolebindings" in API group "opensearch.opster.io" at the cluster scope
E0816 11:09:47.427347       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.1/tools/cache/reflector.go:167: Failed to watch *v1.OpensearchUserRoleBinding: failed to list *v1.OpensearchUserRoleBinding: opensearchuserrolebindings.opensearch.opster.io is forbidden: User "system:serviceaccount:opensearch-operator:opensearch-operator-controller-manager" cannot list resource "opensearchuserrolebindings" in API group "opensearch.opster.io" at the cluster scope
1.6606481908477452e+09	ERROR	controller.opensearchuser	Could not wait for Cache to sync	{"reconciler group": "opensearch.opster.io", "reconciler kind": "OpensearchUser", "error": "failed to wait for opensearchuser caches to sync: timed out waiting for cache to be synced"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:208
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:234
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/manager/runnable_group.go:218
1.6606481908478312e+09	INFO	Stopping and waiting for non leader election runnables
1.6606481908478434e+09	INFO	Stopping and waiting for leader election runnables
1.6606481908479605e+09	ERROR	controller.opensearchrole	Could not wait for Cache to sync	{"reconciler group": "opensearch.opster.io", "reconciler kind": "OpensearchRole", "error": "failed to wait for opensearchrole caches to sync: timed out waiting for cache to be synced"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:208
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:234
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/manager/runnable_group.go:218
1.6606481908479881e+09	ERROR	controller.opensearchuserrolebinding	Could not wait for Cache to sync	{"reconciler group": "opensearch.opster.io", "reconciler kind": "OpensearchUserRoleBinding", "error": "failed to wait for opensearchuserrolebinding caches to sync: timed out waiting for cache to be synced"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:208
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:234
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/manager/runnable_group.go:218
1.6606481908480115e+09	ERROR	error received after stop sequence was engaged	{"error": "failed to wait for opensearchrole caches to sync: timed out waiting for cache to be synced"}
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).engageStopProcedure.func1
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/manager/internal.go:541
1.6606481908480253e+09	ERROR	error received after stop sequence was engaged	{"error": "failed to wait for opensearchuserrolebinding caches to sync: timed out waiting for cache to be synced"}
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).engageStopProcedure.func1
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/manager/internal.go:541
1.660648190848003e+09	ERROR	controller.opensearchcluster	Could not wait for Cache to sync	{"reconciler group": "opensearch.opster.io", "reconciler kind": "OpenSearchCluster", "error": "failed to wait for opensearchcluster caches to sync: timed out waiting for cache to be synced"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:208
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:234
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/manager/runnable_group.go:218
1.6606481908480515e+09	INFO	Stopping and waiting for caches
1.6606481908480752e+09	ERROR	error received after stop sequence was engaged	{"error": "failed to wait for opensearchcluster caches to sync: timed out waiting for cache to be synced"}
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).engageStopProcedure.func1
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/manager/internal.go:541
1.6606481908482618e+09	INFO	Stopping and waiting for webhooks
1.6606481908482788e+09	INFO	Wait completed, proceeding to shutdown the manager
1.6606481908482988e+09	ERROR	setup	problem running manager	{"error": "failed to wait for opensearchuser caches to sync: timed out waiting for cache to be synced"}
main.main
/workspace/main.go:132
runtime.main
/usr/local/go/src/runtime/proc.go:255
1.6606481908483193e+09	ERROR	error received after stop sequence was engaged	{"error": "leader election lost"}
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).engageStopProcedure.func1
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/manager/internal.go:541

Hi @dobharweim. Can you please try again with the newest helm chart release (2.0.4)? The newest version should have the needed permissions.

Hi @swoehrl-mw yes I can confirm the fix in 2.0.4 resolves this issue. Thanks.


immutable fields

We would like to have immutable fields on our form. Which means once the form is submitted those fields cannot be edited. How do we accomplish such a thing when form is loaded for editing and we want to mark some fields readonly? Thanks.

@randallknutson thanks. We still want to display them, but in readonly mode

We currently only have a hideComponents list. It makes sense to have a "disableComponents" list as well. Another way to do this is to set the disabled flag in the builder if they are always disabled. If they need to be dynamic, you can also iterate over the form components and set the disabled flag on any fields.

@randallknutson , I tried [hideComponents]="['submit']" to hide submit button. It does not hide the button. Any idea why, should open another issue?
<formio #formView [form]="template" [hideComponents]="['submit']" [readOnly]="true"></formio>

Same for me:
<formio
  src="https://examples.form.io/example"
  [hideComponents]="['firstName', 'phoneNumber']">
</formio>

Everything in the example form is visible

Looks like @randallknutson 's comment about hideComponents refers to the AngularJs version. There is no hideComponents in the Angular 2 version. (https://github.com/formio/angular-formio/wiki/Data-Table)

This may be a bug. We will investigate.

@dmenne Thank you for the report. We have a created a ticket for the issue and will review during our planning meeting next week. Please note, we welcome pull requests in this library. If you need this item expedited, please feel free to contact<EMAIL_ADDRESS>for pricing.

We're currently addressing a backlog of GitHub issues, and as part of this effort, some inactive issues may be marked as closed. This isn't a dismissal, but a step toward more efficient tracking.
If you feel the issue is still relevant, please re-open and we'll ensure it gets the attention it deserves. Your understanding is appreciated as we work to enhance our open-source responsiveness.


[Explore] wrap %s around selected value to implicitly indicate LIKE clause

As a Superset user, when I use the LIKE filter, it would be helpful the ad hoc filter can automatically apply % % on each end of the value input to implicitly indicate the LIKE clause.
Cursor can be placed in between %%, please make sure the query is not being affected after adding them.

Currently, like or ilike operator in filter, I think there are some flaws.
The query does not contain % in where clause. For this query, it is basically equivalent to = operator.

In addition to show % in the UI, I suggest putting % in the query.

In addition to show % in the UI, I suggest putting % in the query.

definitely!

@zhaoyongjie to eliminate confusion to business users, let's only add %% to the query, and not do anything on UI.

Closing this out since it's not a bug. The like/ilike option has been brought up a few times, but gets shot down as a default for a number of reasons, but mainly performance. I think we can add additional configuration options to the filter to support this, but we should take it up in an Ideas thread on Discussions if anyone is interested in doing so.


Non private async client

Currently, as far as I could understand by looking at issues, the async client is used with
from supabase._async.client import AsyncClient as SupabaseClient, create_client

However, in python, we are not supposed to access private members of a module (starting with _ makes it private). Thus my question is whether there is another official way of importing it, and if not if it would be possible to change this.

Ok I read a bit more some prs here, and understood how it works (using acreate_client). However, how am I supposed to import AsyncClient for type hinting?


Provide service-id-string for service ID parameters

We can provide custom PHPStan PHPDoc types. This could help us replace the dynamic return type extension DrupalServiceDynamicReturnTypeExtension. This can allow us to validate known service names as arguments.
@phpstan-param service-name-string $id

Then, maybe, we can tie it into templates for returning the class object type.
Such as the following in a stub of \Drupal::service
@phpstan-param service-name-string<T> $id
@return T

If our extension to provide service-name-string is able to provide the value of T for the template, it might just work.

Okay, so this is not yet implemented :)
(I have just had a need for something similar.)

I started working on it and dead-ended. I may try to pick it back up. It's been a while


Updated Google API Policies Causing Authorization Error

Attempting to sign in today, I received the following error page:
Error 400: invalid_request
You can't sign in to this app because it doesn't comply with Google's OAuth 2.0 policy for keeping apps secure.
You can let the app developer know that this app doesn't comply with one or more Google validation rules.
Request Details
The content in this section has been provided by the app developer. This content has not been reviewed or verified by Google.
If you’re the app developer, make sure that these request details comply with Google policies.
redirect_uri: urn:ietf:wg:oauth:2.0:oob
Digging into it, Google recently made a change to their API Policies:
https://developers.googleblog.com/2022/02/making-oauth-flows-safer.html#dates-oob
I'm not sure if there's a work around. I am running 3.9.7.

Same issue on revision 3.9.9


Kml draw order

This adds support for the drawOrder element in KML which was originally omitted.
@samthor PTAL

LGTM


Make NoiseLearnerResult JSON serializable

What is the expected feature or enhancement?
Unlike other result types, NoiseLearnerResult is not JSON serializable. I get an error when trying to save the job using this guide: https://docs.quantum.ibm.com/guides/save-jobs#save-results-to-disk.
import json
from qiskit_ibm_runtime import RuntimeEncoder
with open("result.json", "w") as file:
   json.dump(retrieved_job.result(), file, cls=RuntimeEncoder)

TypeError: Object of type NoiseLearnerResult is not JSON serializable

Full traceback.
Acceptance criteria
NoiseLearnerResult can be serialized with RuntimeEncoder and RuntimeDecoder.

Thank you!


chore(table): remove isSortable prop where inappropriate

Closes #2962
Summary

removes extraneous uses of isSortable in stories/tests where not appropriate.

Change List (commits, features, bugs, etc)

updated stories, tests, snapshots

Acceptance Test (how to verify the PR)

isSortable should not appear as knobs in stories
or in tests except where necessary

Regression Test (how to make sure this PR doesn't break old functionality)

n/a

Things to look for during review

[ ] Make sure all references to iot or bx class prefix is using the prefix variable
[ ] (React) All major areas have a data-testid attribute. New test ids should have test written to ensure they are not changed or removed.
[ ] UI should be checked in RTL mode to ensure the proper handling of layout and text.
[ ] All strings should be translatable.
[ ] The code should pass a11y scans (The storybook a11y knob should show no violations). New components should have a11y test written.
[ ] Unit test should be written and should have a coverage of 90% or higher in all areas.
[ ] All components should be passing visual regression test. For new styles or components either a visual regression test should be written for all permutations or the base image updated.
[ ] Changes or new components should either write new or update existing documentation.
[ ] PR should link and close out an existing issue

Nice, I think you missed one though. isSortable is also used in packages/react/src/components/Table/StatefulTable.test.jsx

Good catch. tough to sort through and catch all of them. fixed.


Add Discord Component Documentation

Description:
This pull request updates the documentation for several Discord components, enhancing our existing instructions and adding new guides for specific functionalities. The updates include:

New Instructions Added:

Instructions for adding emoji reactions and setting up reactions.
Guidelines for retrieving audio messages and listening to text messages.
Additional details for Discord message sending functionalities.

Existing Documentation Enhancements:

Updated instructions for file sending, channel message retrieval, guild information retrieval (channels and users), and image processing with Discord.
Improved guidelines for mention responses and thread interactions.

These documentation updates ensure users have access to comprehensive and clear instructions for integrating and utilizing Discord components within their projects. Each component guide is designed to help users quickly understand and implement the functionalities in their workflows.

@ogabrielluiz  review it, please.

Hey @berrytern
Could you please make sure the Docs builds so that we get a Vercel Preview?
Go to ./docs and run npm run build to see the problems.

I revised it with @joaoguilhermeS, and it seems fine now. Please take another look, @ogabrielluiz.


Report and ignore UCASMatch records with no applications

Context
We are getting many sidekiq failures on QA related to bad UCASMatch data. Many tasks still enqueued in Retries, which will never succeed because they have nil arguments. Obviously we could clear Retries, but the moment we switch on ucas matching again, other similar tasks will probably be enqueued. I haven't got to the bottom of the data issue, but the failures are caused by UCASMatch records with next_action of initial_emails_sent which return no application_for_the_same_course_in_progress_on_both_services? records. These UCASMatch records are not actionable, but will always be considered during ucas matching, which is not ideal. Maybe we need an additional enum/action_taken along the lines of :no_application_choice which is set so that the process forgets about them. But we should also raise an error, lest this masks data issues on production.
Changes proposed in this pull request
Introduce a no_application_choice enum/action_taken, and include it in the resolved? states.
Set this as action_taken for UCASMatch records that return no application choices.
Raise a meaningful error so that we hear about such cases from Sentry.
Guidance to review
Should be easy. Does the approach make sense?
Link to Trello card
No Trello card
Things to check

[x] This code does not rely on migrations in the same Pull Request
[x] If this code includes a migration adding or changing columns, it also backfills existing records for consistency
[x] API release notes have been updated if necessary
[x] This code does not rely on the addition/removal of Azure config environment variables in the same Pull Request
[x] New environment variables have been added to the Azure config

Database-level enum changes detected
Please include a data migration for these attributes and values:

No data migration needed as we are adding a new value which has never been set before.

@mnacos Hello, any update/blockers on this PR?


New API to support saving local only resources

Is your feature request related to a problem? Please describe.
Sometimes we need to persist resources on the device's database without creating local changes to be synced to the server. Such resources may be downloaded from the server or perhaps generated by the client itself.
Currently the only way to persist a resource is to use the FhirEngine.create API method which flags the resources for sync.
Describe the solution you'd like
This could be solved by exposing the Database.insertRemote API via the FhirEngine API by adding the following interface and implementation.
//FhirEngine.kt
suspend fun createRemote(vararg resource: Resource)

//FhirEngineImpl.kt
  override suspend fun createRemote(vararg resource: Resource){
    return database.insertRemote(*resource)
  }

Describe alternatives you've considered
An alternative approach could be to enhance the current FhirEngine.create API and pass a flag to determine whether it should be marked for sync or is a local only resource.
//FhirEngine.kt
suspend fun create(val isLocalOnlyChange:Boolean = false, vararg resource: Resource): List<String>

Might be better to name the flag something like isSyncable or syncUp and set its default to true to avoid any confusion with the existing concept of local changes.
Additional context
N/A
Would you like to work on the issue?
Yes

Appears in Kotlin an overriding function is not allowed to specify default values for its parameters. I will therefore make a PR with the original solution and we can iterate on that if need be.

what does it mean for a non-local only resource to depend on a local only resource? this shouldn't really work as we'd be potentially breaking data integrity on the server if the non-local only resource is uploaded to the server. to what extent is this a problem in our case and how should we handle it?

ping @ndegwamartin

@ndegwamartin Is this change still required or we can close it?

@aditya-07 yeah we could do that since it is superseded by this discussion here - https://github.com/google/android-fhir/discussions/2625
Note however, we do still use the related draft PR internally until that discussion is resolved.


Proposal: Volume refactor and external volume plugins

Extend the volume backend to accept configurations from an external process, aka plugin. It uses a pseudo rpc to get information from the plugin. Plugin discovery is performed by scanning the /usr/share/docker/plugins directory, looking for either a socket named like the volume driver specified or a file that contains the uri to the connection point.
This PR also includes a slim version of #13025 and https://github.com/crosbymichael/docker/pull/36.
TODO:

[x] More unit tests
[x] Integration tests
[x] Documentation

Can you please sign your commits following these rules:
https://github.com/docker/docker/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:
$ git clone -b "plugin_discovery" git@github.com:calavera/docker.git somewhere
$ cd somewhere
$ git rebase -i HEAD~12
editor opens
change each 'pick' to 'edit'
save the file and quit
$ git commit --amend -s --no-edit
$ git rebase --continue # and repeat the amend for each commit
$ git push -f

This will update the existing PR, so DO NOT open a new one.

Would it make sense to extend the Create function to take in a set of opaque labels to be interpreted by the VolumeDriver. This would allow storage attributes (such as those mentioned by @wallnerryan)  to be associated with the volume.
Create(name string, labels map[string]string) (err error)

@jvinod I think it would be great if we can see a real example why you need those options for. I agree with @LK4D4 that volume configurations should be separated from usage, but I can change my mind if I understand better why you need them.

@calavera e.g. requesting a volume of a specific size, minimum IOPS, etc.
These I would see as runtime requirements, and not driver init.

@calavera   the volume driver needs some context to configure perf and data mgmt (ex: snapshots) of the volumes.  Having this information will help the driver make appropriate volume placements decisions to meet the SLOs.  ex: choosing between  ssd/sas/sata tiers for volume placement.

Brace yourself: rebase is coming

@calavera +1 to @cpuguy83 & @nagendersoma. drivers may themselves understand these optioned differently, thus being able to pass this context allows the driver to make the decision during provision/request time for a volume.

There may indeed be use cases for driver-specific configurations, but I don't think this should be a blocker for this PR. One thing with this kind of features is that we don't know how people will use it: let's aim for the minimum, and build up from there.
Tentatively moving to code review.

It looks like the implementation is missing a few things.  Overall the framework looks to be coming together well.
When sending requests to the remote API via the proxy volume driver there should be some sort of context about the requester and information that will assist the volume management service to properly get a device to the host.  If the requirement is only NFS, then this can likely be handled already.  But otherwise, a unique identifier that helps the volume management service remotely understand who is sending the request would be helpful.  For example, if I am going to get an EBS volume from EC2 attached, I would need to send my underlying instance identifier.  This would be achieved via a form of local introspection.  This information could be hinted at inside of the SPEC file for now but should be passed via the API call.  This also likely assumes that a structured format exits for the SPEC file.
Currently the creation arguments only specify a name.  This would make sense if a volume has already been created from a volume management service, that the creation request is to simply get the existing volume attached to the guest or via NFS where size may not matter.  I would assume for composability that the ability to create a volume with parameters like size would exist which leads towards a volume being a 1st class citizen and some of @cpuguy83's work.  This also leads towards having separate volume API calls that occur outside of the RUN and -V parameters.
Last thing. The current iteration doesn't seem to include any authentication or encryption.  A SPEC file as YAML may include a pre-shared key for this node.  In addition the SSL based stuff will be important for real world usage.

Thanks for you comment @clintonskitson. As @icecrime mentions, we're aware that there are several limitations with this approach right now. We decided to start with the most simple case and iterate over it. This implementation will probably be marked as experimental for the next 1.7 and we'll keep working on it to make it stable and fully featured for 1.8.
We definitely have all your recommendations in mind, thanks for taking time reviewing this.

Can you please sign your commits following these rules:
https://github.com/docker/docker/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:
$ git clone -b "plugin_discovery" git@github.com:calavera/docker.git somewhere
$ cd somewhere
$ git rebase -i HEAD~21
editor opens
change each 'pick' to 'edit'
save the file and quit
$ git commit --amend -s --no-edit
$ git rebase --continue # and repeat the amend for each commit
$ git push -f

This will update the existing PR, so DO NOT open a new one.

Another note here.  Mount/Unmount are represented as volumeDriverProxy methods expecting a remote plugin, but the mount/unmount process are more likely to be methods that need to be managed locally.  Issuing a call to something external that in most cases doesn't have access to actually do a mount in the guest limits the usefulness of the plugin.  IMO things like mount/unmount would more likely be handled via the mount package internally by Docker and something more in line with attach/detach would be handled via the plugin.
As is, the volumeDriverProxy endpoint would likely need to be ran as a local service to each Docker instance versus something distributed or targeted.  Some of how the graphDriver functions would be helpful here versus offloading everything to a plugin.
Attach Volume to Docker instance (remote call)
Create File System (local call)
Mount Storage (local call)

Some test failures
Also ran compose's test suite against this, seems OK there.

For my understanding; is this a replacement for https://github.com/docker/docker/issues/13025? And will this deprecate --volumes-from for bind-mounts, as mentioned on that PR, or is the jury still out on that?

@thaJeztah yes, this is the replacement of that PR.
From #13025:

Another behavior change is that bind mounts cannot be inherited from volumes-from. I doubt this was ever expected and should have never worked.

I really do not know if this was expected or not, but this PR respects this behavior to be backwards compatible.

I really do not know if this was expected or not, but this PR respects this behavior to be backwards compatible.

It would make me really happy to keep the behavior as it is (for now). I realise there's some controversy around it (apparently, it wasn't really "by design"). Still, it's a feature that is now part of the "API" and I think it would be good to further discuss that and look for replacements / alternatives before it's deprecated, keeping various use-cases in mind.

btw, to get this to build we had to apply the following patch to hack/make.sh and run the build command with -e DOCKER_EXPERIMENTAL=1.
luke@slim:~/Projects/docker$ git diff
diff --git a/hack/make.sh b/hack/make.sh
index b9c2aef..a5a83ab 100755
--- a/hack/make.sh
+++ b/hack/make.sh
@@ -93,6 +93,12 @@ if [ ! "$GOPATH" ]; then
        exit 1
 fi

+if [ "$DOCKER_EXPERIMENTAL" ]; then
+    echo >&2 '# WARNING! DOCKER_EXPERIMENTAL is set: building experimental features'
+    echo >&2
+    DOCKER_BUILDTAGS+=" experimental"
+fi
+
 if [ -z "$DOCKER_CLIENTONLY" ]; then
        DOCKER_BUILDTAGS+=" daemon"
 fi

Good catch @lukemarsden, I think I rebased the code yesterday evening but never pushed it. Doing it now.

Latest build is giving me this on running bundles/1.7.0-dev/binary/docker-1.7.0-dev run -v foo:/data --volume-driver=flocker busybox sh. Granted I haven't actually got a flocker.sock on this machine but I'm pretty sure I haven't seen this before.
https://gist.github.com/lukemarsden/1b0d9088ca6d086ed815

I wiped out /var/lib/docker and tried this again, and got:
luke@slim:~/Projects/docker$ bundles/1.7.0-dev/binary/docker-1.7.0-dev run -v foo:/data --volume-driver=flocker busybox sh
WARN[0000] Running experimental build
Error response from daemon: cannot bind mount volume: foo:/data volume paths must be absolute.

This appears to be a regression due to @icecrime's recent refactoring. @icecrime or @calavera - could you take a look?

looking

In 5d717a8 I'm getting past docker run -v name:path --volume-driver=flocker but now not seeing any requests at all to volume plugins any more.
cc @calavera @icecrime - thanks in advance for investigating!

Yay, green build! http://build.labs.clusterhq.com/builders/flocker%2Facceptance%2Fvagrant%2Fubuntu-14.04/builds/136
Thanks @calavera :sparkles:

All my tests are good: code LGTM.

4345a23 is green here: http://build.labs.clusterhq.com/builders/flocker%2Facceptance%2Fvagrant%2Fubuntu-14.04/builds/138
LGTM.

I think this is good to go, granted it is a lot of changes but I have reviewed and LGTM

nice work everyone :)

Thanks @jfrazelle, pushing to docs review.

Ping @moxiegirl: I think we can take the PR as-is and have a followup PR to move in a dedicated experimental section when ready. Ideally, we'd like to avoid holding up on it for too long to avoid difficult rebases.

Docs LGTM - I think thats enough info for me to write one. - @moxiegirl @jamtur01 @thaJeztah ?

We've done some final testing on our side and uncovered one issue: external volume drivers don't seem to get called with Unmount or Destroy events at all. We are seeing Create, Mount & Path events though as expected.
If this is an easy fix it would be great to get this in today (and I'm happy to test it), but otherwise it could be a follow-up PR.
cc @calavera @crosbymichael @cpuguy83 @progrium @icecrime

Finally I'm not sure if the panic: assignment to entry in nil map issue on daemon upgrade got fixed -- did it?
cc @calavera @cpuguy83

When running docker run --rm -ti -v volumename:/data --volume-driver=test busybox sh without any such plugin socket, the error log looks like it could contain more details:
DEBU[0147] Calling POST /containers/create
INFO[0147] POST /v1.19/containers/create
ERRO[0147] Error: Plugin not found
ERRO[0147] Handler for POST /containers/create returned error: Plugin not found
ERRO[0147] HTTP Error

Update on missing Unmount/Remove events:
@cpuguy83 helped me figure out that I was never seeing Remove events because I wasn't doing docker rm -v. We should add that this only happens when the user explicitly types rm -v to the API docs.
We are still missing Unmount events though!
cc @calavera @crosbymichael

Implementing a volume ext I see a couple of issues.
Right now Mount is sent when the container is started and Unmount is sent (or at least would be according to current design but is not actually for some reason...) when the container stops.
Mount/Unmount are presumably supposed to be used for ref counting, however if all containers using a volume are stopped and someone docker rm -v's a container, the ext would think there's nothing using the volume so it's ok to remove.
This could be mitigated by only ref-counting based on Create/Remove requests, however this would yield Mount/Unmount pointless (and maybe it is?).
Also need to think about daemon restart (and presumably system restart) and making sure that all filesystems are indeed mounted in place so they can be used.

Also moving back to code-review because as is ext points are broken as unmount is never called.

Thanks for this @cpuguy83!
At the moment I'm seeing the following behaviour (ignoring Path for now):

docker create => Create
docker start => Mount
docker stop => ???
docker rm -v => Remove

So apart from the missing Unmount call (which should correspond to docker stop), this seems sensible. My view is that if we can get Unmount fixed, plugins should have enough information to implement the refcounting semantics they want.

From an API perspective, it would likely make sense to include the mapping that @lukemarsden included so that someone developing the plugin can easily understand how their implementation needs to react in certain situations.
For example, if a Create method is called the implementor has the ability to either respond with an error if the volume already exists, or proceed assuming the user wants to attach an already created volume to a container.
In the case of mounting it is similar.  If it already mounted then proceed, if not then continue with mount.
For removing, it is expected that a full process of unmounting and removing would need to take place.
The more implicit workflows would be called out something similar to the following.
docker run -v => Create,Mount,Unmount
docker run --rm -v => Create,Mount,Remove

Thanks for the review @cpuguy83. I just pushed some changes that should address your comments.

Thanks @calavera!
I'm seeing Unmount events when I explicitly stop a running container now, but there's no Unmount event when the container dies naturally. (ex: docker run [...] busybox sh -c "sleep 1").
That would be nice to have to be able to accurately refcount container stop/start, but if it's too much work we could just go with this for now ;)

@calavera now Unmount is indeed called.
However with --volumes-from Mount gets called, but Unmount does not.

@calavera also - Unmount doesn't seem to be called when doing docker rm -f foo while foo is running even though that implicitly stops the container.

Future bug to fix here.  When running docker run -ti -v testj22:/test/test busybox without specifying a proper plugin/name combination before the volume and missing the --volume-driver param, it should error out with a testj22 volume paths must be absolute as usual.  Right now it just mounts /test/test to the root fs.

@clintonskitson That's actually expected: it introduced a way to name volumes (which is required for a distributed volume backend to be able to identify volumes on different hosts to actually be the same). The overloaded -v syntax doesn't make it obvious, things will get better with a top-level docker volume command.

@icecrime Yup I am following that.  But if it is the --volume-driver=local or default, wouldn't you expect that a qualified path is given?

@clintonskitson Volumes managed by the local driver can be named like any other volumes.

@cpuguy83, @lukemarsden I think both your issues where the same, I forgot a call to unmount when the container is cleaned up :sweat_smile: It should be fixed now.

@calavera ok, better, but I'm getting some duplicate events now!
with named volume

docker run => 1 mount call
docker stop => 2 unmount calls (also seems to take a while to stop, might be unrelated)
docker start => 1 mount call
docker rm -f while running => 2 unmount calls
docker rm -f while not running =>  1 unmount call

with anonymous volume

docker run => 2 mount calls
everything else as above

I would expect:

docker run => 1 mount
docker start (when stopped) => 1 mount
docker stop (when running) => 1 unmount
docker rm -f (when running) => 1 unmount
docker rm -f (when stopped) => no unmounts

@icecrime but the volumes managed by local don't have names, they have absolute paths.  In general, it is going to be problematic if you miss the --volume-driver flag and don't get an error when you specify something like '-v name:/here'.  Seems like a simple safety net to ensure that if the volume driver is local that a proper path is specified.
I just checked the opposite and it is showing the likewise behavior.  ie. a volume-driver is specified but '/name:/here' is used and it mounts to root.  The behavior in this case may be to assume if a '/' is specified in the name that it is always a local mount regardless of the volume-driver specified.

@icecrime Regarding your previous comment.  I do get what you're saying around the future and having even local volumes named.  It probably needs some more attention though to ensure the -v flag doesn't present unexpected behavior in the shorter term.

I'll write up a test volume driver to include in integration-cli to test exactly these events

@cpuguy83 there is already one that I'm working on it right now to add more tests around events.

@calavera LGTM! http://build.labs.clusterhq.com/builders/flocker%2Facceptance%2Fvagrant%2Fubuntu-14.04/builds/143 is green and my smoke tests are looking good.

@cpuguy83 Looks good for you to push back to 3-docs-review?
@moxiegirl We moved the docs to dedicated "experimental" section, but I'm not sure how to link it with the rest. I hope you'll be find merging as-is and how we can figure out the proper links later. Thanks!

Yep code works, I still have concerns with regards to the semantics of mount/unmount  and being able to remove a remove a volume simply because all the containers that are using it are stopped... but it doesn't seem like anyone wants to address this.

@icecrime Add a section in experimental.md file --- there is a template at the top.  Link from that section to the details in the doc.  Slack me if you need more help.

LGTM

As seen with @moxiegirl: docs content is there but needs to be properly "linked" from some experimental land page: we'll deal with this in a followup PR.

#BOOM

I am looking for documentation as it may be addressed somewhere. The issue of garbage collection i.e. as how storage is cleaned up
a) In situation of errors, docker container crashed etc.
b) if volume clean up api returned error during docker going down. How can be the storage be addressed again for any cleanups.

@vaibhavkhanduja If yours is a general question, you should ask it in the Google Group or on IRC.  You'll find information on how to get help with questions here: https://docs.docker.com/project/get-help/
This page in the documentation mentions garbage collection as it applies to volumes: https://docs.docker.com/userguide/dockervolumes/#data-volumes

@moxiegirl .. my question is not a general ... in situations of failure or crashes are their any constructs to reclaim allocated storage? or it is responsibility of the volume driver to take care of gc?

it is responsibility of the volume driver to take care of gc?

@vaibhavkhanduja, yes, for now, it's responsibility of the volume driver, but we're open to suggestions.

Volume driver seems to be the best place to implement this but I am wondering as how would it come to know state of the container? The volume api create seems to be passing just a name
+```
+{

"Name": "volume_name"
+}

this is suppose is not the docker container name considering there could be multiple volumes hooked into a container. Please correct me if I am wrong here?
Traditionally each mount point or file-system is associated with a unique number/entity name etc. The driver map of container run id and volume names could help in identifying which ones to be removed. Or a callback from docker to initiate GC or reclaim space when the container is removed (not stopped).

@vaibhavkhanduja volume drivers get events for Create, Mount, Unmount, and Remove for each container that is using a given volume.
The name of a volume is globally addressable.

@cpuguy83  not clear, can you elaborate more on the name? If I have a docker container with uuid how would know volumes associated with it? Does create volume api is called in context of a container passing id of the container? BTW I am talking about a GC, failure scenario ..

@vaibhavkhanduja Please ask questions in docker-dev ML or #docker-dev on IRC, will be happy to address there.
Thanks.

https://groups.google.com/forum/#!topic/docker-dev/swvOe6Qmrfw


Sales/Purchase Blocked for G/L Accounts

For Items the new fields "Sales Blocked" and "Purchase Blocked" were invented with BC130 and on Table 37/39 the field "No." has a modified table relation to allow the refering items only for sales or purchasing.
We have the same idea on items and on G/l accounts, too. So if thos two fields would be added to table 15 and the table realtion in T37/39 would be enhanced similar to items we could delete our Extension with this implementation.

I'm closing the issue for two reasons:

it's not extensibility request but new functionality request
there are no plans to extends functionlaity for GLs. If advanced functionality is needed: attributes, prices, UoM, undo, etc (including extended blocking) - use Items (normal ones or of type Service).


Issue bash on ubuntu on windows

I got a problem with installing/setting it up on Bash on Ubuntu on Windows. It installs fine, but I get an error when running:
procyon@Raccoonbook:~$ pxl /mnt/c/Users/Procyon/Pictures/chattobuysupport.png
panic: invalid argument
goroutine 1 [running]:
panic(0x57f480, 0xc82000e4f0)
/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
main.main()
/home/procyon/work/src/github.com/mattn/pxl/main.go:62 +0x2eb
procyon@Raccoonbook:~$
It won't run.

Hi, did you manage to solve the issue? I would suggest to use the latest version of Go and try again.

Using the latest version of go:
`panic: invalid argument
goroutine 1 [running]:
main.main()
/home/procyon/go/src/github.com/ichinaski/pxl/main.go:79 +0x1fa`