k8s helm 方式部署 ByConity  POD 无法启动

values.yaml
use_existing_fdb: true
fdb_cluster_file: clusterdsc:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx@<IP_ADDRESS>:4500,<IP_ADDRESS>:4500,<IP_ADDRESS>:4500
kubectl get pods -n xxxx |grep byconity
byconity-daemon-manager-85c979f478-cxc8v                     0/1     Running     0               7m25s
byconity-resource-manager-9d755fdc4-f94b6                    0/1     Running     0               7m25s
byconity-server-0                                            0/1     Running     0               7m25s
byconity-tso-57bbb558fb-b5x2q                                0/1     Running     0               7m25s
byconity-vw-vw-default-0                                     0/1     Running     0               7m25s
byconity-vw-vw-write-0                                       0/1     Running     0               7m25s

kubectl logs byconity-server-0   -n xxxx
rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
rsyslogd: activation of module imklog failed [v8.1901.0 try https://www.rsyslog.com/e/2145 ]
jemalloc.background_thread was requested, however ClickHouse uses percpu_arena and background_thread most likely will not give any benefits, and also background_thread is not compatible with ClickHouse watchdog (that can be disabled with CLICKHOUSE_WATCHDOG_ENABLE=0)
Processing configuration file '/etc/byconity/server.yaml'.
Logging trace to /var/log/byconity/byconity.log
Logging errors to /var/log/byconity/byconity.err.log
Logging trace to console
2023.07.17 05:56:12.122020 [ 16 ] {} <Information> Application: Will watch for the process with pid 52
2023.07.17 05:56:12.122386 [ 52 ] {} <Information> Application: Forked a child process to watch
2023.07.17 05:56:12.122543 [ 52 ] {} <Information> SentryWriter: Sending crash reports is disabled
2023.07.17 05:56:12.122670 [ 52 ] {} <Trace> Pipe: Pipe capacity is 1.00 MiB
2023.07.17 05:56:12.240843 [ 52 ] {} <Information> : Starting ClickHouse <IP_ADDRESS> scm <IP_ADDRESS> with revision 54453, build id: E74C4B9AFA78CDA9, PID 52
2023.07.17 05:56:12.240972 [ 52 ] {} <Information> Application: starting up
2023.07.17 05:56:12.241001 [ 52 ] {} <Information> Application: OS name: Linux, version: 5.4.119-1-tlinux4-0009-public-eks, architecture: x86_64
2023.07.17 05:56:12.313420 [ 52 ] {} <Debug> registerStorageCnchHive: registerStorageCnchHive
Processing configuration file '/etc/byconity/cnch-config.yaml'.
Processing configuration file '/etc/byconity/cnch-config.yaml'.
2023.07.17 05:56:12.323133 [ 52 ] {} <Debug> Configuration: Config element service_discovery.vw.psm is recommended to set in config.xml. You'd better customize it.
2023.07.17 05:56:12.323527 [ 52 ] {} <Information> BrpcGflagsConfigHolder: Set brpc gflags [ max_body_size : 671088640 ]
2023.07.17 05:56:12.323597 [ 52 ] {} <Information> BrpcGflagsConfigHolder: Set brpc gflags [ socket_max_unwritten_bytes :<PHONE_NUMBER> ]
2023.07.17 05:56:12.323619 [ 52 ] {} <Information> BrpcGflagsConfigHolder: Set brpc gflags [ defer_close_second : 60 ]
2023.07.17 05:56:12.323639 [ 52 ] {} <Information> BrpcGflagsConfigHolder: Set brpc gflags [ event_dispatcher_num : 2 ]
2023.07.17 05:56:12.323 <Info> BrpcChannelPoolConfigHolder::afterinit Init brpc client channel pool with config: PoolOptions@0x7fd7170eb828 {pool_name:stream_default, max_connections:4, load_balancer:rr, ChannelOptions:{connect_timout_ms:1000, timeout_ms:3000, backup_request_ms:-1, max_retry:3, enable_circuit_breaker:false, succeed_without_server:true, log_succeed_without_server:true, connection_group:}}
PoolOptions@0x7fd7170eb728 {pool_name:rpc_default, max_connections:2, load_balancer:rr, ChannelOptions:{connect_timout_ms:1000, timeout_ms:3000, backup_request_ms:-1, max_retry:3, enable_circuit_breaker:false, succeed_without_server:true, log_succeed_without_server:true, connection_group:}}

2023.07.17 05:56:12.323 <Info> BrpcApplication::initialize Brpc is initialized with {} config holders2 config holders
2023.07.17 05:56:12.373673 [ 52 ] {} <Debug> Context: Initialising Resource Manager Client
2023.07.17 05:56:12.380393 [ 52 ] {} <Debug> ServiceDiscoveryDNS: byconity-resource-manager-headless error message:domain name does not exist
2023.07.17 05:56:12.380432 [ 52 ] {} <Debug> ServiceDiscoveryDNS: lookup RESOURCE_MANAGER [byconity-resource-manager][] returns empty result
2023.07.17 05:56:12.384049 [ 52 ] {} <Error> Context::initResourceManagerClient: void DB::Context::initResourceManagerClient(): Code: 906, e.displayText() = DB::Exception: No RM instance found with psm: byconity-resource-manager SQLSTATE: HY000, Stack trace (when copying this message, always include the lines below):

0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x190133f2 in /opt/byconity/bin/clickhouse
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb2e45a0 in /opt/byconity/bin/clickhouse
2. DB::ResourceManagement::fetchRMAddressByPSM(std::__1::shared_ptr<DB::Context const>) @ 0x1372157b in /opt/byconity/bin/clickhouse
3. DB::ResourceManagement::ResourceManagerClient::fetchRMAddress() const @ 0x1372115e in /opt/byconity/bin/clickhouse
4. DB::ResourceManagement::ResourceManagerClient::ResourceManagerClient(std::__1::shared_ptr<DB::Context const>) @ 0x13720e79 in /opt/byconity/bin/clickhouse
5. DB::Context::initResourceManagerClient() @ 0x13e94391 in /opt/byconity/bin/clickhouse
6. DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0xb36bb38 in /opt/byconity/bin/clickhouse
7. Poco::Util::Application::run() @ 0x18f9eaca in /opt/byconity/bin/clickhouse
8. DB::Server::run() @ 0xb36a124 in /opt/byconity/bin/clickhouse
9. Poco::Util::ServerApplication::run(int, char**) @ 0x18fb093b in /opt/byconity/bin/clickhouse
10. mainEntryClickHouseServer(int, char**) @ 0xb368ad2 in /opt/byconity/bin/clickhouse
11. main @ 0xb2de828 in /opt/byconity/bin/clickhouse
12. __libc_start_main @ 0x2409b in /lib/x86_64-linux-gnu/libc-2.28.so
13. _start @ 0xb2de02a in /opt/byconity/bin/clickhouse
 (version <IP_ADDRESS>)
2023.07.17 05:56:15.385201 [ 52 ] {} <Debug> ServiceDiscoveryDNS: byconity-resource-manager-headless error message:domain name does not exist
2023.07.17 05:56:15.385265 [ 52 ] {} <Debug> ServiceDiscoveryDNS: lookup RESOURCE_MANAGER [byconity-resource-manager][] returns empty result
2023.07.17 05:56:15.385394 [ 52 ] {} <Error> Context::initResourceManagerClient: void DB::Context::initResourceManagerClient(): Code: 906, e.displayText() = DB::Exception: No RM instance found with psm: byconity-resource-manager SQLSTATE: HY000, Stack trace (when copying this message, always include the lines below):

0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x190133f2 in /opt/byconity/bin/clickhouse
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb2e45a0 in /opt/byconity/bin/clickhouse
2. DB::ResourceManagement::fetchRMAddressByPSM(std::__1::shared_ptr<DB::Context const>) @ 0x1372157b in /opt/byconity/bin/clickhouse
3. DB::ResourceManagement::ResourceManagerClient::fetchRMAddress() const @ 0x1372115e in /opt/byconity/bin/clickhouse
4. DB::ResourceManagement::ResourceManagerClient::ResourceManagerClient(std::__1::shared_ptr<DB::Context const>) @ 0x13720e79 in /opt/byconity/bin/clickhouse
5. DB::Context::initResourceManagerClient() @ 0x13e94391 in /opt/byconity/bin/clickhouse
6. DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0xb36bb38 in /opt/byconity/bin/clickhouse
7. Poco::Util::Application::run() @ 0x18f9eaca in /opt/byconity/bin/clickhouse
8. DB::Server::run() @ 0xb36a124 in /opt/byconity/bin/clickhouse
9. Poco::Util::ServerApplication::run(int, char**) @ 0x18fb093b in /opt/byconity/bin/clickhouse
10. mainEntryClickHouseServer(int, char**) @ 0xb368ad2 in /opt/byconity/bin/clickhouse
11. main @ 0xb2de828 in /opt/byconity/bin/clickhouse
12. __libc_start_main @ 0x2409b in /lib/x86_64-linux-gnu/libc-2.28.so
13. _start @ 0xb2de02a in /opt/byconity/bin/clickhouse
 (version <IP_ADDRESS>)
2023.07.17 05:56:18.386539 [ 52 ] {} <Debug> ServiceDiscoveryDNS: byconity-resource-manager-headless error message:domain name does not exist
2023.07.17 05:56:18.386578 [ 52 ] {} <Debug> ServiceDiscoveryDNS: lookup RESOURCE_MANAGER [byconity-resource-manager][] returns empty result
2023.07.17 05:56:18.386684 [ 52 ] {} <Error> Context::initResourceManagerClient: void DB::Context::initResourceManagerClient(): Code: 906, e.displayText() = DB::Exception: No RM instance found with psm: byconity-resource-manager SQLSTATE: HY000, Stack trace (when copying this message, always include the lines below):

0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x190133f2 in /opt/byconity/bin/clickhouse
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb2e45a0 in /opt/byconity/bin/clickhouse
2. DB::ResourceManagement::fetchRMAddressByPSM(std::__1::shared_ptr<DB::Context const>) @ 0x1372157b in /opt/byconity/bin/clickhouse
3. DB::ResourceManagement::ResourceManagerClient::fetchRMAddress() const @ 0x1372115e in /opt/byconity/bin/clickhouse
4. DB::ResourceManagement::ResourceManagerClient::ResourceManagerClient(std::__1::shared_ptr<DB::Context const>) @ 0x13720e79 in /opt/byconity/bin/clickhouse
5. DB::Context::initResourceManagerClient() @ 0x13e94391 in /opt/byconity/bin/clickhouse
6. DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0xb36bb38 in /opt/byconity/bin/clickhouse
7. Poco::Util::Application::run() @ 0x18f9eaca in /opt/byconity/bin/clickhouse
8. DB::Server::run() @ 0xb36a124 in /opt/byconity/bin/clickhouse
9. Poco::Util::ServerApplication::run(int, char**) @ 0x18fb093b in /opt/byconity/bin/clickhouse
10. mainEntryClickHouseServer(int, char**) @ 0xb368ad2 in /opt/byconity/bin/clickhouse
11. main @ 0xb2de828 in /opt/byconity/bin/clickhouse
12. __libc_start_main @ 0x2409b in /lib/x86_64-linux-gnu/libc-2.28.so
13. _start @ 0xb2de02a in /opt/byconity/bin/clickhouse
 (version <IP_ADDRESS>)
2023.07.17 05:56:21.387729 [ 52 ] {} <Debug> ServiceDiscoveryDNS: byconity-resource-manager-headless error message:domain name does not exist
2023.07.17 05:56:21.387768 [ 52 ] {} <Debug> ServiceDiscoveryDNS: lookup RESOURCE_MANAGER [byconity-resource-manager][] returns empty result
2023.07.17 05:56:21.387874 [ 52 ] {} <Error> Context::initResourceManagerClient: void DB::Context::initResourceManagerClient(): Code: 906, e.displayText() = DB::Exception: No RM instance found with psm: byconity-resource-manager SQLSTATE: HY000, Stack trace (when copying this message, always include the lines below):

0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x190133f2 in /opt/byconity/bin/clickhouse
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb2e45a0 in /opt/byconity/bin/clickhouse
2. DB::ResourceManagement::fetchRMAddressByPSM(std::__1::shared_ptr<DB::Context const>) @ 0x1372157b in /opt/byconity/bin/clickhouse
3. DB::ResourceManagement::ResourceManagerClient::fetchRMAddress() const @ 0x1372115e in /opt/byconity/bin/clickhouse
4. DB::ResourceManagement::ResourceManagerClient::ResourceManagerClient(std::__1::shared_ptr<DB::Context const>) @ 0x13720e79 in /opt/byconity/bin/clickhouse
5. DB::Context::initResourceManagerClient() @ 0x13e94391 in /opt/byconity/bin/clickhouse
6. DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0xb36bb38 in /opt/byconity/bin/clickhouse
7. Poco::Util::Application::run() @ 0x18f9eaca in /opt/byconity/bin/clickhouse
8. DB::Server::run() @ 0xb36a124 in /opt/byconity/bin/clickhouse
9. Poco::Util::ServerApplication::run(int, char**) @ 0x18fb093b in /opt/byconity/bin/clickhouse
10. mainEntryClickHouseServer(int, char**) @ 0xb368ad2 in /opt/byconity/bin/clickhouse
11. main @ 0xb2de828 in /opt/byconity/bin/clickhouse
12. __libc_start_main @ 0x2409b in /lib/x86_64-linux-gnu/libc-2.28.so
13. _start @ 0xb2de02a in /opt/byconity/bin/clickhouse
 (version <IP_ADDRESS>)
2023.07.17 05:56:24.390261 [ 52 ] {} <Error> Application: DB::Exception: Unable to initialise Resource Manager Client SQLSTATE: HY000
2023.07.17 05:56:24.390304 [ 52 ] {} <Information> Application: shutting down
2023.07.17 05:56:24.390311 [ 52 ] {} <Debug> Application: Uninitializing subsystem: Logging Subsystem
2023.07.17 05:56:24.390372 [ 54 ] {} <Trace> BaseDaemon: Received signal -2
2023.07.17 05:56:24.390388 [ 54 ] {} <Information> BaseDaemon: Stop SignalListener thread

clusterdsc:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx@<IP_ADDRESS>:4500,<IP_ADDRESS>:4500,<IP_ADDRESS>:4500
这个内容是fdb.cluster文件获取的

clusterdsc:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx@<IP_ADDRESS>:4500,<IP_ADDRESS>:4500,<IP_ADDRESS>:4500
这个内容是fdb.cluster文件获取的

byconity-resource-manager-headless error message:domain name does not exist

``

clusterdsc:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx@<IP_ADDRESS>:4500,<IP_ADDRESS>:4500,<IP_ADDRESS>:4500
这个内容是fdb.cluster文件获取的

byconity-resource-manager-headless error message:domain name does not exist

values.yaml
use_existing_fdb: true fdb_cluster_file: clusterdsc:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx@<IP_ADDRESS>:4500,<IP_ADDRESS>:4500,<IP_ADDRESS>:4500
kubectl get pods -n xxxx |grep byconity
byconity-daemon-manager-85c979f478-cxc8v                     0/1     Running     0               7m25s
byconity-resource-manager-9d755fdc4-f94b6                    0/1     Running     0               7m25s
byconity-server-0                                            0/1     Running     0               7m25s
byconity-tso-57bbb558fb-b5x2q                                0/1     Running     0               7m25s
byconity-vw-vw-default-0                                     0/1     Running     0               7m25s
byconity-vw-vw-write-0                                       0/1     Running     0               7m25s

kubectl logs byconity-server-0   -n xxxx
rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
rsyslogd: activation of module imklog failed [v8.1901.0 try https://www.rsyslog.com/e/2145 ]
jemalloc.background_thread was requested, however ClickHouse uses percpu_arena and background_thread most likely will not give any benefits, and also background_thread is not compatible with ClickHouse watchdog (that can be disabled with CLICKHOUSE_WATCHDOG_ENABLE=0)
Processing configuration file '/etc/byconity/server.yaml'.
Logging trace to /var/log/byconity/byconity.log
Logging errors to /var/log/byconity/byconity.err.log
Logging trace to console
2023.07.17 05:56:12.122020 [ 16 ] {} <Information> Application: Will watch for the process with pid 52
2023.07.17 05:56:12.122386 [ 52 ] {} <Information> Application: Forked a child process to watch
2023.07.17 05:56:12.122543 [ 52 ] {} <Information> SentryWriter: Sending crash reports is disabled
2023.07.17 05:56:12.122670 [ 52 ] {} <Trace> Pipe: Pipe capacity is 1.00 MiB
2023.07.17 05:56:12.240843 [ 52 ] {} <Information> : Starting ClickHouse <IP_ADDRESS> scm <IP_ADDRESS> with revision 54453, build id: E74C4B9AFA78CDA9, PID 52
2023.07.17 05:56:12.240972 [ 52 ] {} <Information> Application: starting up
2023.07.17 05:56:12.241001 [ 52 ] {} <Information> Application: OS name: Linux, version: 5.4.119-1-tlinux4-0009-public-eks, architecture: x86_64
2023.07.17 05:56:12.313420 [ 52 ] {} <Debug> registerStorageCnchHive: registerStorageCnchHive
Processing configuration file '/etc/byconity/cnch-config.yaml'.
Processing configuration file '/etc/byconity/cnch-config.yaml'.
2023.07.17 05:56:12.323133 [ 52 ] {} <Debug> Configuration: Config element service_discovery.vw.psm is recommended to set in config.xml. You'd better customize it.
2023.07.17 05:56:12.323527 [ 52 ] {} <Information> BrpcGflagsConfigHolder: Set brpc gflags [ max_body_size : 671088640 ]
2023.07.17 05:56:12.323597 [ 52 ] {} <Information> BrpcGflagsConfigHolder: Set brpc gflags [ socket_max_unwritten_bytes :<PHONE_NUMBER> ]
2023.07.17 05:56:12.323619 [ 52 ] {} <Information> BrpcGflagsConfigHolder: Set brpc gflags [ defer_close_second : 60 ]
2023.07.17 05:56:12.323639 [ 52 ] {} <Information> BrpcGflagsConfigHolder: Set brpc gflags [ event_dispatcher_num : 2 ]
2023.07.17 05:56:12.323 <Info> BrpcChannelPoolConfigHolder::afterinit Init brpc client channel pool with config: PoolOptions@0x7fd7170eb828 {pool_name:stream_default, max_connections:4, load_balancer:rr, ChannelOptions:{connect_timout_ms:1000, timeout_ms:3000, backup_request_ms:-1, max_retry:3, enable_circuit_breaker:false, succeed_without_server:true, log_succeed_without_server:true, connection_group:}}
PoolOptions@0x7fd7170eb728 {pool_name:rpc_default, max_connections:2, load_balancer:rr, ChannelOptions:{connect_timout_ms:1000, timeout_ms:3000, backup_request_ms:-1, max_retry:3, enable_circuit_breaker:false, succeed_without_server:true, log_succeed_without_server:true, connection_group:}}

2023.07.17 05:56:12.323 <Info> BrpcApplication::initialize Brpc is initialized with {} config holders2 config holders
2023.07.17 05:56:12.373673 [ 52 ] {} <Debug> Context: Initialising Resource Manager Client
2023.07.17 05:56:12.380393 [ 52 ] {} <Debug> ServiceDiscoveryDNS: byconity-resource-manager-headless error message:domain name does not exist
2023.07.17 05:56:12.380432 [ 52 ] {} <Debug> ServiceDiscoveryDNS: lookup RESOURCE_MANAGER [byconity-resource-manager][] returns empty result
2023.07.17 05:56:12.384049 [ 52 ] {} <Error> Context::initResourceManagerClient: void DB::Context::initResourceManagerClient(): Code: 906, e.displayText() = DB::Exception: No RM instance found with psm: byconity-resource-manager SQLSTATE: HY000, Stack trace (when copying this message, always include the lines below):

0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x190133f2 in /opt/byconity/bin/clickhouse
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb2e45a0 in /opt/byconity/bin/clickhouse
2. DB::ResourceManagement::fetchRMAddressByPSM(std::__1::shared_ptr<DB::Context const>) @ 0x1372157b in /opt/byconity/bin/clickhouse
3. DB::ResourceManagement::ResourceManagerClient::fetchRMAddress() const @ 0x1372115e in /opt/byconity/bin/clickhouse
4. DB::ResourceManagement::ResourceManagerClient::ResourceManagerClient(std::__1::shared_ptr<DB::Context const>) @ 0x13720e79 in /opt/byconity/bin/clickhouse
5. DB::Context::initResourceManagerClient() @ 0x13e94391 in /opt/byconity/bin/clickhouse
6. DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0xb36bb38 in /opt/byconity/bin/clickhouse
7. Poco::Util::Application::run() @ 0x18f9eaca in /opt/byconity/bin/clickhouse
8. DB::Server::run() @ 0xb36a124 in /opt/byconity/bin/clickhouse
9. Poco::Util::ServerApplication::run(int, char**) @ 0x18fb093b in /opt/byconity/bin/clickhouse
10. mainEntryClickHouseServer(int, char**) @ 0xb368ad2 in /opt/byconity/bin/clickhouse
11. main @ 0xb2de828 in /opt/byconity/bin/clickhouse
12. __libc_start_main @ 0x2409b in /lib/x86_64-linux-gnu/libc-2.28.so
13. _start @ 0xb2de02a in /opt/byconity/bin/clickhouse
 (version <IP_ADDRESS>)
2023.07.17 05:56:15.385201 [ 52 ] {} <Debug> ServiceDiscoveryDNS: byconity-resource-manager-headless error message:domain name does not exist
2023.07.17 05:56:15.385265 [ 52 ] {} <Debug> ServiceDiscoveryDNS: lookup RESOURCE_MANAGER [byconity-resource-manager][] returns empty result
2023.07.17 05:56:15.385394 [ 52 ] {} <Error> Context::initResourceManagerClient: void DB::Context::initResourceManagerClient(): Code: 906, e.displayText() = DB::Exception: No RM instance found with psm: byconity-resource-manager SQLSTATE: HY000, Stack trace (when copying this message, always include the lines below):

0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x190133f2 in /opt/byconity/bin/clickhouse
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb2e45a0 in /opt/byconity/bin/clickhouse
2. DB::ResourceManagement::fetchRMAddressByPSM(std::__1::shared_ptr<DB::Context const>) @ 0x1372157b in /opt/byconity/bin/clickhouse
3. DB::ResourceManagement::ResourceManagerClient::fetchRMAddress() const @ 0x1372115e in /opt/byconity/bin/clickhouse
4. DB::ResourceManagement::ResourceManagerClient::ResourceManagerClient(std::__1::shared_ptr<DB::Context const>) @ 0x13720e79 in /opt/byconity/bin/clickhouse
5. DB::Context::initResourceManagerClient() @ 0x13e94391 in /opt/byconity/bin/clickhouse
6. DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0xb36bb38 in /opt/byconity/bin/clickhouse
7. Poco::Util::Application::run() @ 0x18f9eaca in /opt/byconity/bin/clickhouse
8. DB::Server::run() @ 0xb36a124 in /opt/byconity/bin/clickhouse
9. Poco::Util::ServerApplication::run(int, char**) @ 0x18fb093b in /opt/byconity/bin/clickhouse
10. mainEntryClickHouseServer(int, char**) @ 0xb368ad2 in /opt/byconity/bin/clickhouse
11. main @ 0xb2de828 in /opt/byconity/bin/clickhouse
12. __libc_start_main @ 0x2409b in /lib/x86_64-linux-gnu/libc-2.28.so
13. _start @ 0xb2de02a in /opt/byconity/bin/clickhouse
 (version <IP_ADDRESS>)
2023.07.17 05:56:18.386539 [ 52 ] {} <Debug> ServiceDiscoveryDNS: byconity-resource-manager-headless error message:domain name does not exist
2023.07.17 05:56:18.386578 [ 52 ] {} <Debug> ServiceDiscoveryDNS: lookup RESOURCE_MANAGER [byconity-resource-manager][] returns empty result
2023.07.17 05:56:18.386684 [ 52 ] {} <Error> Context::initResourceManagerClient: void DB::Context::initResourceManagerClient(): Code: 906, e.displayText() = DB::Exception: No RM instance found with psm: byconity-resource-manager SQLSTATE: HY000, Stack trace (when copying this message, always include the lines below):

0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x190133f2 in /opt/byconity/bin/clickhouse
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb2e45a0 in /opt/byconity/bin/clickhouse
2. DB::ResourceManagement::fetchRMAddressByPSM(std::__1::shared_ptr<DB::Context const>) @ 0x1372157b in /opt/byconity/bin/clickhouse
3. DB::ResourceManagement::ResourceManagerClient::fetchRMAddress() const @ 0x1372115e in /opt/byconity/bin/clickhouse
4. DB::ResourceManagement::ResourceManagerClient::ResourceManagerClient(std::__1::shared_ptr<DB::Context const>) @ 0x13720e79 in /opt/byconity/bin/clickhouse
5. DB::Context::initResourceManagerClient() @ 0x13e94391 in /opt/byconity/bin/clickhouse
6. DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0xb36bb38 in /opt/byconity/bin/clickhouse
7. Poco::Util::Application::run() @ 0x18f9eaca in /opt/byconity/bin/clickhouse
8. DB::Server::run() @ 0xb36a124 in /opt/byconity/bin/clickhouse
9. Poco::Util::ServerApplication::run(int, char**) @ 0x18fb093b in /opt/byconity/bin/clickhouse
10. mainEntryClickHouseServer(int, char**) @ 0xb368ad2 in /opt/byconity/bin/clickhouse
11. main @ 0xb2de828 in /opt/byconity/bin/clickhouse
12. __libc_start_main @ 0x2409b in /lib/x86_64-linux-gnu/libc-2.28.so
13. _start @ 0xb2de02a in /opt/byconity/bin/clickhouse
 (version <IP_ADDRESS>)
2023.07.17 05:56:21.387729 [ 52 ] {} <Debug> ServiceDiscoveryDNS: byconity-resource-manager-headless error message:domain name does not exist
2023.07.17 05:56:21.387768 [ 52 ] {} <Debug> ServiceDiscoveryDNS: lookup RESOURCE_MANAGER [byconity-resource-manager][] returns empty result
2023.07.17 05:56:21.387874 [ 52 ] {} <Error> Context::initResourceManagerClient: void DB::Context::initResourceManagerClient(): Code: 906, e.displayText() = DB::Exception: No RM instance found with psm: byconity-resource-manager SQLSTATE: HY000, Stack trace (when copying this message, always include the lines below):

0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x190133f2 in /opt/byconity/bin/clickhouse
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb2e45a0 in /opt/byconity/bin/clickhouse
2. DB::ResourceManagement::fetchRMAddressByPSM(std::__1::shared_ptr<DB::Context const>) @ 0x1372157b in /opt/byconity/bin/clickhouse
3. DB::ResourceManagement::ResourceManagerClient::fetchRMAddress() const @ 0x1372115e in /opt/byconity/bin/clickhouse
4. DB::ResourceManagement::ResourceManagerClient::ResourceManagerClient(std::__1::shared_ptr<DB::Context const>) @ 0x13720e79 in /opt/byconity/bin/clickhouse
5. DB::Context::initResourceManagerClient() @ 0x13e94391 in /opt/byconity/bin/clickhouse
6. DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0xb36bb38 in /opt/byconity/bin/clickhouse
7. Poco::Util::Application::run() @ 0x18f9eaca in /opt/byconity/bin/clickhouse
8. DB::Server::run() @ 0xb36a124 in /opt/byconity/bin/clickhouse
9. Poco::Util::ServerApplication::run(int, char**) @ 0x18fb093b in /opt/byconity/bin/clickhouse
10. mainEntryClickHouseServer(int, char**) @ 0xb368ad2 in /opt/byconity/bin/clickhouse
11. main @ 0xb2de828 in /opt/byconity/bin/clickhouse
12. __libc_start_main @ 0x2409b in /lib/x86_64-linux-gnu/libc-2.28.so
13. _start @ 0xb2de02a in /opt/byconity/bin/clickhouse
 (version <IP_ADDRESS>)
2023.07.17 05:56:24.390261 [ 52 ] {} <Error> Application: DB::Exception: Unable to initialise Resource Manager Client SQLSTATE: HY000
2023.07.17 05:56:24.390304 [ 52 ] {} <Information> Application: shutting down
2023.07.17 05:56:24.390311 [ 52 ] {} <Debug> Application: Uninitializing subsystem: Logging Subsystem
2023.07.17 05:56:24.390372 [ 54 ] {} <Trace> BaseDaemon: Received signal -2
2023.07.17 05:56:24.390388 [ 54 ] {} <Information> BaseDaemon: Stop SignalListener thread

byconity-resource-manager-headless error message:domain name does not exist
The resource manager service was not found. Can you check the log of the resouce manager?

kubectl logs byconity-resource-manager-9d755fdc4-f94b6   -n xxxxxx
rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
rsyslogd: activation of module imklog failed [v8.1901.0 try https://www.rsyslog.com/e/2145 ]
Processing configuration file '/etc/byconity/resource-manager.yaml'.
Couldn't save preprocessed config to /etc/byconity/resource-manager-preprocessed.yaml: File is read-only: /etc/byconity/resource-manager-preprocessed.yaml
Logging trace to /var/log/byconity/resource-manager.log
Logging errors to /var/log/byconity/resource-manager.err.log
Logging trace to console
2023.07.17 05:55:58.934223 [ 13 ] {} <Information> SentryWriter: Sending crash reports is disabled
2023.07.17 05:55:58.948211 [ 13 ] {} <Trace> Pipe: Pipe capacity is 1.00 MiB
2023.07.17 05:55:59.067624 [ 13 ] {} <Information> : Starting ClickHouse <IP_ADDRESS> scm <IP_ADDRESS> with revision 54453, build id: E74C4B9AFA78CDA9, PID 13
2023.07.17 05:55:59.069397 [ 13 ] {} <Information> Application: Resource Manager is starting up...
Processing configuration file '/etc/byconity/cnch-config.yaml'.
2023.07.17 05:55:59.075345 [ 13 ] {} <Debug> Configuration: Config element tcp_port is recommended to set in config.xml. You'd better customize it.
2023.07.17 05:55:59.075381 [ 13 ] {} <Debug> Configuration: Config element http_port is recommended to set in config.xml. You'd better customize it.
2023.07.17 05:55:59.075390 [ 13 ] {} <Debug> Configuration: Config element rpc_port is recommended to set in config.xml. You'd better customize it.
2023.07.17 05:55:59.075402 [ 13 ] {} <Debug> Configuration: Config element cnch_type is recommended to set in config.xml. You'd better customize it.
Processing configuration file '/etc/byconity/cnch-config.yaml'.
2023.07.17 05:55:59.077626 [ 13 ] {} <Debug> Configuration: Config element service_discovery.vw.psm is recommended to set in config.xml. You'd better customize it.
2023.07.17 05:55:59.129022 [ 13 ] {} <Information> BackgroundSchedulePool/BgSchPool: Create BackgroundSchedulePool with 128 threads
2023.07.17 05:55:59.136416 [ 13 ] {} <Information> ElectionController: RM is running in Single instance mode as ZooKeeper is not enabled. The current RM will become leader and should be the only instance.
2023.07.17 05:55:59.136473 [ 13 ] {} <Information> ElectionController: Starting leader callback for <IP_ADDRESS>:9000
2023.07.17 05:56:00.136566 [ 13 ] {} <Debug> VirtualWarehouseManager: Loading virtual warehouses...
2023.07.17 05:56:05.145136 [ 13 ] {} <Error> ElectionController: Code: 8067, e.displayText() = DB::Exception: Error whiling scanning data through FBDItrator. message: Operation aborted because the transaction timed out SQLSTATE: HY000, Stack trace (when copying this message, always include the lines below):

0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x190133f2 in /opt/byconity/bin/clickhouse
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb2e45a0 in /opt/byconity/bin/clickhouse
2. DB::Catalog::MetastoreFDBImpl::FDBIterator::next() @ 0x1362f755 in /opt/byconity/bin/clickhouse
3. DB::Catalog::MetastoreProxy::scanVirtualWarehouses(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x1364f66a in /opt/byconity/bin/clickhouse
4. ? @ 0x13612ea7 in /opt/byconity/bin/clickhouse
5. DB::Catalog::Catalog::scanVirtualWarehouses() @ 0x135e821d in /opt/byconity/bin/clickhouse
6. DB::ResourceManagement::VirtualWarehouseManager::loadVirtualWarehousesImpl(std::__1::lock_guard<std::__1::mutex>*) @ 0x13754d7a in /opt/byconity/bin/clickhouse
7. DB::ResourceManagement::VirtualWarehouseManager::loadVirtualWarehouses() @ 0x13754b61 in /opt/byconity/bin/clickhouse
8. DB::ResourceManagement::ResourceManagerController::initialize() @ 0x13737460 in /opt/byconity/bin/clickhouse
9. DB::ResourceManagement::ElectionController::pullState() @ 0x1373d2a0 in /opt/byconity/bin/clickhouse
10. DB::ResourceManagement::ElectionController::onLeader() @ 0x1373cc31 in /opt/byconity/bin/clickhouse
11. DB::ResourceManagement::ElectionController::ElectionController(DB::ResourceManagement::ResourceManagerController&) @ 0x1373c4ff in /opt/byconity/bin/clickhouse
12. DB::ResourceManagement::ResourceManagerController::ResourceManagerController(std::__1::shared_ptr<DB::Context const>) @ 0x13730cf8 in /opt/byconity/bin/clickhouse
13. DB::ResourceManager::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0x1371eb36 in /opt/byconity/bin/clickhouse
14. Poco::Util::Application::run() @ 0x18f9eaca in /opt/byconity/bin/clickhouse
15. DB::ResourceManager::run() @ 0x1371e0f5 in /opt/byconity/bin/clickhouse
16. Poco::Util::ServerApplication::run(int, char**) @ 0x18fb093b in /opt/byconity/bin/clickhouse
17. ? @ 0x137204ab in /opt/byconity/bin/clickhouse
18. main @ 0xb2de828 in /opt/byconity/bin/clickhouse
19. __libc_start_main @ 0x2409b in /lib/x86_64-linux-gnu/libc-2.28.so
20. _start @ 0xb2de02a in /opt/byconity/bin/clickhouse
 (version <IP_ADDRESS>)
2023.07.17 05:56:05.145213 [ 13 ] {} <Debug> VirtualWarehouseManager: Loading virtual warehouses...
2023.07.17 05:56:10.145856 [ 13 ] {} <Error> ElectionController: Code: 8067, e.displayText() = DB::Exception: Error whiling scanning data through FBDItrator. message: Operation aborted because the transaction timed out SQLSTATE: HY000, Stack trace (when copying this message, always include the lines below):

0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x190133f2 in /opt/byconity/bin/clickhouse
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb2e45a0 in /opt/byconity/bin/clickhouse
2. DB::Catalog::MetastoreFDBImpl::FDBIterator::next() @ 0x1362f755 in /opt/byconity/bin/clickhouse
3. DB::Catalog::MetastoreProxy::scanVirtualWarehouses(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x1364f66a in /opt/byconity/bin/clickhouse
4. ? @ 0x13612ea7 in /opt/byconity/bin/clickhouse
5. DB::Catalog::Catalog::scanVirtualWarehouses() @ 0x135e821d in /opt/byconity/bin/clickhouse
6. DB::ResourceManagement::VirtualWarehouseManager::loadVirtualWarehousesImpl(std::__1::lock_guard<std::__1::mutex>*) @ 0x13754d7a in /opt/byconity/bin/clickhouse
7. DB::ResourceManagement::VirtualWarehouseManager::loadVirtualWarehouses() @ 0x13754b61 in /opt/byconity/bin/clickhouse
8. DB::ResourceManagement::ResourceManagerController::initialize() @ 0x13737460 in /opt/byconity/bin/clickhouse
9. DB::ResourceManagement::ElectionController::pullState() @ 0x1373d329 in /opt/byconity/bin/clickhouse
10. DB::ResourceManagement::ElectionController::onLeader() @ 0x1373cc31 in /opt/byconity/bin/clickhouse
11. DB::ResourceManagement::ElectionController::ElectionController(DB::ResourceManagement::ResourceManagerController&) @ 0x1373c4ff in /opt/byconity/bin/clickhouse
12. DB::ResourceManagement::ResourceManagerController::ResourceManagerController(std::__1::shared_ptr<DB::Context const>) @ 0x13730cf8 in /opt/byconity/bin/clickhouse
13. DB::ResourceManager::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0x1371eb36 in /opt/byconity/bin/clickhouse
14. Poco::Util::Application::run() @ 0x18f9eaca in /opt/byconity/bin/clickhouse
15. DB::ResourceManager::run() @ 0x1371e0f5 in /opt/byconity/bin/clickhouse
16. Poco::Util::ServerApplication::run(int, char**) @ 0x18fb093b in /opt/byconity/bin/clickhouse
17. ? @ 0x137204ab in /opt/byconity/bin/clickhouse
18. main @ 0xb2de828 in /opt/byconity/bin/clickhouse
19. __libc_start_main @ 0x2409b in /lib/x86_64-linux-gnu/libc-2.28.so
20. _start @ 0xb2de02a in /opt/byconity/bin/clickhouse
 (version <IP_ADDRESS>)
2023.07.17 05:56:10.145932 [ 13 ] {} <Debug> VirtualWarehouseManager: Loading virtual warehouses...
2023.07.17 05:56:15.146418 [ 13 ] {} <Error> ElectionController: Code: 8067, e.displayText() = DB::Exception: Error whiling scanning data through FBDItrator. message: Operation aborted because the transaction timed out SQLSTATE: HY000, Stack trace (when copying this message, always include the lines below):

0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x190133f2 in /opt/byconity/bin/clickhouse
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb2e45a0 in /opt/byconity/bin/clickhouse
2. DB::Catalog::MetastoreFDBImpl::FDBIterator::next() @ 0x1362f755 in /opt/byconity/bin/clickhouse
3. DB::Catalog::MetastoreProxy::scanVirtualWarehouses(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x1364f66a in /opt/byconity/bin/clickhouse
4. ? @ 0x13612ea7 in /opt/byconity/bin/clickhouse
5. DB::Catalog::Catalog::scanVirtualWarehouses() @ 0x135e821d in /opt/byconity/bin/clickhouse
6. DB::ResourceManagement::VirtualWarehouseManager::loadVirtualWarehousesImpl(std::__1::lock_guard<std::__1::mutex>*) @ 0x13754d7a in /opt/byconity/bin/clickhouse
7. DB::ResourceManagement::VirtualWarehouseManager::loadVirtualWarehouses() @ 0x13754b61 in /opt/byconity/bin/clickhouse
8. DB::ResourceManagement::ResourceManagerController::initialize() @ 0x13737460 in /opt/byconity/bin/clickhouse
9. DB::ResourceManagement::ElectionController::pullState() @ 0x1373d3ac in /opt/byconity/bin/clickhouse
10. DB::ResourceManagement::ElectionController::onLeader() @ 0x1373cc31 in /opt/byconity/bin/clickhouse
11. DB::ResourceManagement::ElectionController::ElectionController(DB::ResourceManagement::ResourceManagerController&) @ 0x1373c4ff in /opt/byconity/bin/clickhouse
12. DB::ResourceManagement::ResourceManagerController::ResourceManagerController(std::__1::shared_ptr<DB::Context const>) @ 0x13730cf8 in /opt/byconity/bin/clickhouse
13. DB::ResourceManager::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0x1371eb36 in /opt/byconity/bin/clickhouse
14. Poco::Util::Application::run() @ 0x18f9eaca in /opt/byconity/bin/clickhouse
15. DB::ResourceManager::run() @ 0x1371e0f5 in /opt/byconity/bin/clickhouse
16. Poco::Util::ServerApplication::run(int, char**) @ 0x18fb093b in /opt/byconity/bin/clickhouse
17. ? @ 0x137204ab in /opt/byconity/bin/clickhouse
18. main @ 0xb2de828 in /opt/byconity/bin/clickhouse
19. __libc_start_main @ 0x2409b in /lib/x86_64-linux-gnu/libc-2.28.so
20. _start @ 0xb2de02a in /opt/byconity/bin/clickhouse
 (version <IP_ADDRESS>)
2023.07.17 05:56:15.147422 [ 13 ] {} <Trace> WorkerGroupResourceCoordinator: Stopping WorkerGroupResourceCoordinator
2023.07.17 05:56:15.147436 [ 13 ] {} <Debug> WorkerGroupResourceCoordinator: Dectivating auto-sharing background task
2023.07.17 05:56:15.147445 [ 13 ] {} <Trace> ResourceTracker: Stopping ResourceTrackerTask
2023.07.17 05:56:15.148864 [ 13 ] {} <Trace> BackgroundSchedulePool/BgSchPool: Waiting for threads to finish.
2023.07.17 05:56:15.151833 [ 13 ] {} <Error> Application: DB::Exception: Failed to pull metadata from KV store in Single instance mode, please check the metadata service and restart RM. SQLSTATE: HY000
2023.07.17 05:56:15.151863 [ 13 ] {} <Debug> Application: Uninitializing subsystem: Logging Subsystem
2023.07.17 05:56:15.151925 [ 38 ] {} <Trace> BaseDaemon: Received signal -2
2023.07.17 05:56:15.151951 [ 38 ] {} <Information> BaseDaemon: Stop SignalListener thread

byconity/byconity:0.1.0-GA
这个镜像版本

请问您是不是在公司内部有相关的需求，可以通过微信群加下小助手联系我们试试看。

@serenallttyy    已加

我配置为  byconity/byconity:latest
kubectl logs byconity-resource-manager-6c97c5dffc-w9pv6      -n xxxx
rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
rsyslogd: activation of module imklog failed [v8.1901.0 try https://www.rsyslog.com/e/2145 ]
Processing configuration file '/etc/byconity/resource-manager.yaml'.
Couldn't save preprocessed config to /etc/byconity/resource-manager-preprocessed.yaml: File is read-only: /etc/byconity/resource-manager-preprocessed.yaml
Logging trace to /var/log/byconity/resource-manager.log
Logging errors to /var/log/byconity/resource-manager.err.log
Logging trace to console
2023.07.17 07:57:14.296940 [ 12 ] {} <Information> Application: jeprof status: false
2023.07.17 07:57:14.310674 [ 12 ] {} <Information> SentryWriter: Sending crash reports is disabled
2023.07.17 07:57:14.310721 [ 12 ] {} <Trace> Pipe: Pipe capacity is 1.00 MiB
2023.07.17 07:57:14.417761 [ 12 ] {} <Information> : Starting ClickHouse <IP_ADDRESS> scm <IP_ADDRESS> with revision 54453, build id: D89888E231F8AF1F, PID 12
2023.07.17 07:57:14.419199 [ 12 ] {} <Information> Application: Resource Manager is starting up...
Processing configuration file '/etc/byconity/cnch-config.yaml'.
2023.07.17 07:57:14.426104 [ 12 ] {} <Debug> Configuration: Config element tcp_port is recommended to set in config.xml. You'd better customize it.
2023.07.17 07:57:14.426147 [ 12 ] {} <Debug> Configuration: Config element http_port is recommended to set in config.xml. You'd better customize it.
2023.07.17 07:57:14.426156 [ 12 ] {} <Debug> Configuration: Config element rpc_port is recommended to set in config.xml. You'd better customize it.
2023.07.17 07:57:14.426169 [ 12 ] {} <Debug> Configuration: Config element cnch_type is recommended to set in config.xml. You'd better customize it.
Processing configuration file '/etc/byconity/cnch-config.yaml'.
2023.07.17 07:57:14.428481 [ 12 ] {} <Debug> Configuration: Config element service_discovery.vw.psm is recommended to set in config.xml. You'd better customize it.
2023.07.17 07:57:14.474923 [ 12 ] {} <Information> BackgroundSchedulePool/BgSchPool: Create BackgroundSchedulePool with 128 threads
2023.07.17 07:57:14.482176 [ 12 ] {} <Information> ElectionController: RM is running in Single instance mode as ZooKeeper is not enabled. The current RM will become leader and should be the only instance.
2023.07.17 07:57:14.482237 [ 12 ] {} <Information> ElectionController: Starting leader callback for <IP_ADDRESS>:9000
2023.07.17 07:57:15.482326 [ 12 ] {} <Debug> VirtualWarehouseManager: Loading virtual warehouses...
2023.07.17 07:57:20.485340 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:57:25.485647 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:57:30.485985 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:57:35.486255 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:57:40.486534 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:57:45.487190 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:57:50.487489 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:57:55.487763 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:58:00.488022 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:58:05.488275 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:58:10.488541 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:58:15.488860 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:58:20.489180 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:58:25.489465 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:58:30.489722 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:58:35.489981 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:58:40.490257 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:58:45.490517 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
root@kaya-vpn:~/devops/byconity-deploy#

麻烦确认一下。

我配置为 byconity/byconity:latest
kubectl logs byconity-resource-manager-6c97c5dffc-w9pv6      -n xxxx
rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
rsyslogd: activation of module imklog failed [v8.1901.0 try https://www.rsyslog.com/e/2145 ]
Processing configuration file '/etc/byconity/resource-manager.yaml'.
Couldn't save preprocessed config to /etc/byconity/resource-manager-preprocessed.yaml: File is read-only: /etc/byconity/resource-manager-preprocessed.yaml
Logging trace to /var/log/byconity/resource-manager.log
Logging errors to /var/log/byconity/resource-manager.err.log
Logging trace to console
2023.07.17 07:57:14.296940 [ 12 ] {} <Information> Application: jeprof status: false
2023.07.17 07:57:14.310674 [ 12 ] {} <Information> SentryWriter: Sending crash reports is disabled
2023.07.17 07:57:14.310721 [ 12 ] {} <Trace> Pipe: Pipe capacity is 1.00 MiB
2023.07.17 07:57:14.417761 [ 12 ] {} <Information> : Starting ClickHouse <IP_ADDRESS> scm <IP_ADDRESS> with revision 54453, build id: D89888E231F8AF1F, PID 12
2023.07.17 07:57:14.419199 [ 12 ] {} <Information> Application: Resource Manager is starting up...
Processing configuration file '/etc/byconity/cnch-config.yaml'.
2023.07.17 07:57:14.426104 [ 12 ] {} <Debug> Configuration: Config element tcp_port is recommended to set in config.xml. You'd better customize it.
2023.07.17 07:57:14.426147 [ 12 ] {} <Debug> Configuration: Config element http_port is recommended to set in config.xml. You'd better customize it.
2023.07.17 07:57:14.426156 [ 12 ] {} <Debug> Configuration: Config element rpc_port is recommended to set in config.xml. You'd better customize it.
2023.07.17 07:57:14.426169 [ 12 ] {} <Debug> Configuration: Config element cnch_type is recommended to set in config.xml. You'd better customize it.
Processing configuration file '/etc/byconity/cnch-config.yaml'.
2023.07.17 07:57:14.428481 [ 12 ] {} <Debug> Configuration: Config element service_discovery.vw.psm is recommended to set in config.xml. You'd better customize it.
2023.07.17 07:57:14.474923 [ 12 ] {} <Information> BackgroundSchedulePool/BgSchPool: Create BackgroundSchedulePool with 128 threads
2023.07.17 07:57:14.482176 [ 12 ] {} <Information> ElectionController: RM is running in Single instance mode as ZooKeeper is not enabled. The current RM will become leader and should be the only instance.
2023.07.17 07:57:14.482237 [ 12 ] {} <Information> ElectionController: Starting leader callback for <IP_ADDRESS>:9000
2023.07.17 07:57:15.482326 [ 12 ] {} <Debug> VirtualWarehouseManager: Loading virtual warehouses...
2023.07.17 07:57:20.485340 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:57:25.485647 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:57:30.485985 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:57:35.486255 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:57:40.486534 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:57:45.487190 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:57:50.487489 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:57:55.487763 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:58:00.488022 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:58:05.488275 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:58:10.488541 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:58:15.488860 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:58:20.489180 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:58:25.489465 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:58:30.489722 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:58:35.489981 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:58:40.490257 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
2023.07.17 07:58:45.490517 [ 12 ] {} <Debug> FDBIterator: Transaction timeout, create new transaction
root@kaya-vpn:~/devops/byconity-deploy#

请问下换了镜像是ok了吗？
按照你上面的报错resource manager请求fdb失败，请帮忙检查

用fdb-cli访问之前配置的fdb.cluster文件能正常访问
2.将fdb.enbale 和fdb.operator.enable设置为false
检查一下byconity-fdb-config  这个configMap里的内容是否与你提供的fdb.cluster内容一致

用  bin/fdbcli -C fdb_runtime/config/fdb.cluster 可以正常访问

values 文件

fdb:
  enabled: false

fdb-operator:
  enabled: false

byconity-fdb-config 这个configMap里的内容与fdb.cluster内容一致， 而且网络，端口是可以访问的

foundationdb 版本是  7.3.7

@weigangchen1     请问有解决排查的思路吗？

@hillbun  是否可以提供 你的 values.yaml 文件的内容

# Default values for byconity.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

nameOverride: ""
fullnameOverride: ""

byconity:
  image: byconity/byconity:latest
  imagePullPolicy: IfNotPresent
  #hdfs_address: hdfs://<IP_ADDRESS>:9000  #default hdfs
  #  if you want to use your own fdb please add the following 2 configurations
  #  use_existing_fdb: true
  #  fdb_cluster_file: byconity_fdb:Is0hBgl6iICdHuspBmhAODmD5WISXKzI@<IP_ADDRESS>:4501,<IP_ADDRESS>:4501,<IP_ADDRESS>:4501
  use_existing_fdb: true
  fdb_cluster_file: clusterdsc:xxxxxxxxxx@10.1.x.x:4500,10.1.x.x:4500,10.1.x.x:4500

  ports:
    tcp: 9000
    http: 8123
    rpc: 8124
    tcpSecure: 9100
    https: 9123
    exchange: 9410
    exchangeStatus: 9510

  usersOverwrite:
    users:
      default:
        password: ""
      probe:
        password: probe

  server:
    replicas: 1
    image: ""
    podAnnotations: {}
    resources: {}
    hostNetwork: false
    nodeSelector: {}
    tolerations: []
    affinity: {}
    imagePullSecrets: []
    securityContext: {}
    storage:
      localDisk:
        pvcSpec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 50Gi
          storageClassName: cbs-customer
      log:
        pvcSpec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 50Gi
          storageClassName: cbs-customer
    configOverwrite:
      logger:
        level: trace
      disk_cache_strategies:
        simple:
          lru_max_size:<PHONE_NUMBER>00 # 400Gi
      # timezone: Etc/UTC
    hdfs3Config:
      dfs.client.metrics.enable: true
      dfs.client.metrics.use_domain_socket: true
      dfs.default.blocksize: "268435456"
      dfs.read.hedge_read.interval_ms: 120
      input.connect.timeout: 10000
      input.read.max.retry: 3
      input.read.timeout: 6000
      input.write.timeout: 10000
      output.close.timeout: 256000
      output.connect.timeout: 10000
      output.default.write.retry: 3
      output.read.timeout: 120000
      output.write.timeout: 256000
      rpc.client.connect.retry: 3
      rpc.client.connect.timeout: 10000
      rpc.client.read.timeout: 20000
      rpc.client.timeout: 20000
      rpc.client.write.timeout: 20000

  tso:
    replicas: 1
    image: ""
    podAnnotations: {}
    resources: {}
    hostNetwork: false
    nodeSelector: {}
    tolerations: []
    affinity: {}
    imagePullSecrets: []
    securityContext: {}
    configOverwrite: {}

  daemonManager:
    replicas: 1 # Please keep single instance now, daemon manager HA is WIP
    image: ""
    podAnnotations: {}
    resources: {}
    hostNetwork: false
    nodeSelector: {}
    tolerations: []
    affinity: {}
    imagePullSecrets: []
    securityContext: {}
    configOverwrite: {}

  resourceManager:
    replicas: 1
    image: ""
    podAnnotations: {}
    resources: {}
    hostNetwork: false
    nodeSelector: {}
    tolerations: []
    affinity: {}
    imagePullSecrets: []
    securityContext: {}
    configOverwrite: {}

  defaultWorker: &defaultWorker
    replicas: 1
    image: ""
    podAnnotations: {}
    resources: {}
    hostNetwork: false
    nodeSelector: {}
    tolerations: []
    affinity: {}
    imagePullSecrets: []
    securityContext: {}
    storage:
      localDisk:
        pvcSpec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 50Gi
          storageClassName: cbs-customer
      log:
        pvcSpec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 50Gi
          storageClassName: cbs-customer
    configOverwrite:
      logger:
        level: trace
      disk_cache_strategies:
        simple:
          lru_max_size:<PHONE_NUMBER>00 # 400Gi
      # timezone: Etc/UTC
    hdfs3Config:
      dfs.client.metrics.enable: true
      dfs.client.metrics.use_domain_socket: true
      dfs.default.blocksize: "268435456"
      dfs.read.hedge_read.interval_ms: 120
      input.connect.timeout: 10000
      input.read.max.retry: 3
      input.read.timeout: 6000
      input.write.timeout: 10000
      output.close.timeout: 256000
      output.connect.timeout: 10000
      output.default.write.retry: 3
      output.read.timeout: 120000
      output.write.timeout: 256000
      rpc.client.connect.retry: 3
      rpc.client.connect.timeout: 10000
      rpc.client.read.timeout: 20000
      rpc.client.timeout: 20000
      rpc.client.write.timeout: 20000

  virtualWarehouses:
    - <<: *defaultWorker
      name: vw_default
      replicas: 1
    - <<: *defaultWorker
      name: vw_write
      replicas: 1

  commonEnvs:
    - name: MY_POD_NAMESPACE
      valueFrom:
        fieldRef:
          fieldPath: "metadata.namespace"
    - name: MY_POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: "metadata.name"
    - name: MY_UID
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: "metadata.uid"
    - name: MY_POD_IP
      valueFrom:
        fieldRef:
          fieldPath: "status.podIP"
    - name: MY_HOST_IP
      valueFrom:
        fieldRef:
          # fieldPath: "status.hostIP"
          fieldPath: "status.podIP"
    - name: CONSUL_HTTP_HOST
      valueFrom:
        fieldRef:
          fieldPath: "status.hostIP"

  additionalEnvs: []

  additionalVolumes:
    volumes: []
    volumeMounts: []

  postStart: ""
  preStop: ""
  readiness: ""
  liveness: ""

  ingress:
    enabled: false
    className: ""
    annotations: {}
      # kubernetes.io/ingress.class: nginx
      # kubernetes.io/tls-acme: "true"
    hosts:
      - host: byconity-example.local
        paths:
          - path: /
            pathType: ImplementationSpecific
    tls: []
    #  - secretName: byconity-example-tls
    #    hosts:
    #      - byconity-example.local

# For more detailed usage, please check fdb-kubernetes-operator API doc: https://github.com/FoundationDB/fdb-kubernetes-operator/blob/main/docs/cluster_spec.md
fdb:
  enabled: false
  enableCliPod: true
  version: 7.1.15
  clusterSpec: {}

fdb-operator:
  enabled: false
  resources:
    limits:
      cpu: 1
      memory: 512Mi
    requests:
      cpu: 1
      memory: 512Mi
  image:
    repository: foundationdb/fdb-kubernetes-operator
    tag: v1.9.0
    pullPolicy: IfNotPresent

hdfs:
  enabled: false
  namenode:
    repository: gchq/hdfs
    tag: 3.2.2 # managed version
    resources: {}
    nodeSelector: {}
    dataVolumes:
      count: 1
      pvcTemplateSpec:
        storageClassName: cbs-customer
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
    extraEnvVars: {}
    # Example usage:
    # HADOOP_HEAPSIZE_MIN: 128m
    # hadoop.heapsize.min: 128m
  datanode:
    nodeCount: 3
    repository: gchq/hdfs
    tag: 3.2.2 # managed version
    resources: {}
    nodeSelector: {}
    dataVolumes:
      count: 1
      pvcTemplateSpec:
        storageClassName: cbs-customer
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
    extraEnvVars: {}
    # Example usage:
    # HADOOP_HEAPSIZE_MIN: 128m
    # hadoop.heapsize.min: 128m
  config:
    hdfsSite:
      dfs.permissions.enabled: "false"
      dfs.namenode.datanode.registration.ip-hostname-check: "false"
  shell:
    enabled: true
    repository: gchq/hdfs
    tag: 3.2.2 # managed version
    imagePullPolicy: IfNotPresent
    imagePullSecrets: []
    resources: {}
    nodeSelector: {}
    tolerations: []
    affinity: {}
  postInstallCommands: []
  # - hadoop fs -mkdir -p /accumulo
  # - hadoop fs -chown accumulo /accumulo
  # - hadoop fs -chmod 700 /accumulo
  # - hadoop fs -ls /

@hillbun  你是不使用HDFS来做存储么？ 我看你把hdfs的路径注释掉了，fdb 使用的是外部的fdb 对吧。

我是打算不用 HDFS  ， 改用 S3 的
我恢复回去看看

@hillbun  S3也是可以的，可以不用恢复回去，目前看来是 你的 resource manager 无法连接到 你的 fdb， 你这个fdb  数据库的地址很奇怪呢？是你在k8s 上搭建的么？你可以试试看 登陆到 你的resource manager , 看是否可以 连接上你的fdb数据库么?
'telnet ip地址 端口'

我更换了foundationdb版本为文档中的 foundationdb  7.1.25， 之前安装的是最新版本 7.3.7
镜像用  byconity/byconity:0.1.0-GA
hdfs_address和fdb_cluster_file用外部组件
byconity:
  image: byconity/byconity:0.1.0-GA
  imagePullPolicy: IfNotPresent
  hdfs_address: hdfs://10.1.x.x:9000  #default hdfs
  use_existing_fdb: true
  fdb_cluster_file: clusterdsc:test@10.1.x.x:4500

kubectl get pods -n kaya-dev |grep byconity
byconity-daemon-manager-85c979f478-h6hgf                     1/1     Running     0             3m40s
byconity-resource-manager-9d755fdc4-w76dr                    1/1     Running     0             3m40s
byconity-server-0                                            0/1     Running     0             2m3s
byconity-tso-57bbb558fb-w8fd9                                1/1     Running     0             3m40s
byconity-vw-vw-default-0                                     0/1     Running     0             2m3s
byconity-vw-vw-write-0                                       0/1     Running     0             2m4s

kubectl logs byconity-server-0    -n kaya-dev
rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
rsyslogd: activation of module imklog failed [v8.1901.0 try https://www.rsyslog.com/e/2145 ]
jemalloc.background_thread was requested, however ClickHouse uses percpu_arena and background_thread most likely will not give any benefits, and also background_thread is not compatible with ClickHouse watchdog (that can be disabled with CLICKHOUSE_WATCHDOG_ENABLE=0)
Processing configuration file '/etc/byconity/server.yaml'.
Logging trace to /var/log/byconity/byconity.log
Logging errors to /var/log/byconity/byconity.err.log
Logging trace to console
2023.07.18 03:32:27.030012 [ 15 ] {} <Information> Application: Will watch for the process with pid 39
2023.07.18 03:32:27.030242 [ 39 ] {} <Information> Application: Forked a child process to watch
2023.07.18 03:32:27.030469 [ 39 ] {} <Information> SentryWriter: Sending crash reports is disabled
2023.07.18 03:32:27.030599 [ 39 ] {} <Trace> Pipe: Pipe capacity is 1.00 MiB
2023.07.18 03:32:27.142736 [ 39 ] {} <Information> : Starting ClickHouse <IP_ADDRESS> scm <IP_ADDRESS> with revision 54453, build id: E74C4B9AFA78CDA9, PID 39
2023.07.18 03:32:27.144179 [ 39 ] {} <Information> Application: starting up
2023.07.18 03:32:27.145841 [ 39 ] {} <Information> Application: OS name: Linux, version: 5.4.119-1-tlinux4-0009-public-eks, architecture: x86_64
2023.07.18 03:32:27.324404 [ 39 ] {} <Debug> registerStorageCnchHive: registerStorageCnchHive
Processing configuration file '/etc/byconity/cnch-config.yaml'.
Processing configuration file '/etc/byconity/cnch-config.yaml'.
2023.07.18 03:32:27.341019 [ 39 ] {} <Debug> Configuration: Config element service_discovery.vw.psm is recommended to set in config.xml. You'd better customize it.
2023.07.18 03:32:27.341459 [ 39 ] {} <Information> BrpcGflagsConfigHolder: Set brpc gflags [ max_body_size : 671088640 ]
2023.07.18 03:32:27.341533 [ 39 ] {} <Information> BrpcGflagsConfigHolder: Set brpc gflags [ socket_max_unwritten_bytes :<PHONE_NUMBER> ]
2023.07.18 03:32:27.341558 [ 39 ] {} <Information> BrpcGflagsConfigHolder: Set brpc gflags [ defer_close_second : 60 ]
2023.07.18 03:32:27.341579 [ 39 ] {} <Information> BrpcGflagsConfigHolder: Set brpc gflags [ event_dispatcher_num : 2 ]
2023.07.18 03:32:27.341 <Info> BrpcChannelPoolConfigHolder::afterinit Init brpc client channel pool with config: PoolOptions@0x7f3d89aeb828 {pool_name:stream_default, max_connections:4, load_balancer:rr, ChannelOptions:{connect_timout_ms:1000, timeout_ms:3000, backup_request_ms:-1, max_retry:3, enable_circuit_breaker:false, succeed_without_server:true, log_succeed_without_server:true, connection_group:}}
PoolOptions@0x7f3d89aeb728 {pool_name:rpc_default, max_connections:2, load_balancer:rr, ChannelOptions:{connect_timout_ms:1000, timeout_ms:3000, backup_request_ms:-1, max_retry:3, enable_circuit_breaker:false, succeed_without_server:true, log_succeed_without_server:true, connection_group:}}

2023.07.18 03:32:27.341 <Info> BrpcApplication::initialize Brpc is initialized with {} config holders2 config holders
2023.07.18 03:32:27.390568 [ 39 ] {} <Debug> Context: Initialising Resource Manager Client
2023.07.18 03:32:27.394928 [ 39 ] {} <Debug> ResourceManagerClient[{<IP_ADDRESS>  rpc/9000}]: Initialized rpc channel to server with address: <IP_ADDRESS>:9000
2023.07.18 03:32:27.394973 [ 39 ] {} <Debug> Context: Initialised Resource Manager Client on try: 0
2023.07.18 03:32:27.394997 [ 39 ] {} <Information> CnchWorkerClientPools: Added new vw: vw_read
2023.07.18 03:32:27.395009 [ 39 ] {} <Information> CnchWorkerClientPools: Added new vw: vw_write
2023.07.18 03:32:27.395018 [ 39 ] {} <Information> CnchWorkerClientPools: Added new vw: vw_task
2023.07.18 03:32:27.395026 [ 39 ] {} <Information> CnchWorkerClientPools: Added new vw: vw_default
2023.07.18 03:32:28.286780 [ 39 ] {} <Warning> Application: Calculated checksum of the binary: D1EB286171264FAA33033121FE59C8DD. There is no information about the reference checksum.
2023.07.18 03:32:28.287737 [ 39 ] {} <Information> StatusFile: Status file /var/byconity/status already exists - unclean restart. Contents:
PID: 39
Started at: 2023-07-18 03:30:27
Revision: 54453

2023.07.18 03:32:28.287829 [ 39 ] {} <Debug> Application: rlimit on number of file descriptors is 1048576
2023.07.18 03:32:28.287850 [ 39 ] {} <Debug> Application: Initializing DateLUT.
2023.07.18 03:32:28.287859 [ 39 ] {} <Trace> Application: Initialized DateLUT with time zone 'Etc/UTC'.
2023.07.18 03:32:28.287878 [ 39 ] {} <Debug> Application: Setting up /var/byconity/tmp/ to store temporary data in it
2023.07.18 03:32:28.295047 [ 39 ] {} <Debug> Application: Initiailizing interserver credentials.
2023.07.18 03:32:28.297589 [ 39 ] {} <Debug> ConfigReloader: Loading config '/etc/byconity/server.yaml'
Processing configuration file '/etc/byconity/server.yaml'.
Saved preprocessed configuration to '/var/byconity/preprocessed_configs/server.yaml'.
2023.07.18 03:32:28.298856 [ 39 ] {} <Debug> ConfigReloader: Loaded config '/etc/byconity/server.yaml', performing update on configuration
2023.07.18 03:32:28.299174 [ 39 ] {} <Information> Application: Setting max_server_memory_usage was set to 3.44 GiB (3.83 GiB available * 0.90 max_server_memory_usage_to_ram_ratio)
2023.07.18 03:32:28.299 <Info> Reload brpc config with 2 config holders
2023.07.18 03:32:28.299406 [ 39 ] {} <Debug> ConfigReloader: Loaded config '/etc/byconity/server.yaml', performed update on configuration
Processing configuration file '/etc/byconity/users.yaml'.
2023.07.18 03:32:28.303267 [ 39 ] {} <Debug> Access(user directories): Added users.xml access storage 'users.xml', path:
2023.07.18 03:32:28.303295 [ 39 ] {} <Debug> void DB::Context::initResourceGroupManager(const DB::Context::ConfigurationPtr &): Skip initialize resource group
2023.07.18 03:32:28.303857 [ 39 ] {} <Debug> ConfigReloader: Loading config '/etc/byconity/users.yaml'
Processing configuration file '/etc/byconity/users.yaml'.
Saved preprocessed configuration to '/var/byconity/preprocessed_configs/users.yaml'.
2023.07.18 03:32:28.304311 [ 39 ] {} <Debug> ConfigReloader: Loaded config '/etc/byconity/users.yaml', performing update on configuration
2023.07.18 03:32:28.304592 [ 39 ] {} <Debug> ConfigReloader: Loaded config '/etc/byconity/users.yaml', performed update on configuration
2023.07.18 03:32:28.304942 [ 39 ] {} <Debug> Access(user directories): Added users.xml access storage 'users.xml', path: /etc/byconity/users.yaml
2023.07.18 03:32:28.306254 [ 39 ] {} <Information> Application: Mark cache size was lowered to 1.91 GiB because the system has low amount of memory
2023.07.18 03:32:28.306299 [ 39 ] {} <Information> KeyIndexFileCache: Initializing KeyIndexFileCache...
2023.07.18 03:32:28.307387 [ 39 ] {} <Information> KeyIndexFileCache: Initialized KeyIndexFileCache with 0 entries in 0 ms
2023.07.18 03:32:28.308322 [ 39 ] {} <Trace> HDFS3: Hdfs opertaion DefaultConfig::ctor takes 0 ms
2023.07.18 03:32:28.312347 [ 39 ] {} <Trace> HDFS3: FsImpl Connect to <hdfs>://<<IP_ADDRESS>>:9000
2023.07.18 03:32:28.312392 [ 39 ] {} <Trace> HDFS3: Hdfs opertaion hdfsBuilderConnect takes 4 ms
2023.07.18 03:32:28.317705 [ 39 ] {} <Trace> HDFS3: Hdfs opertaion DefaultConfig::ctor takes 0 ms
2023.07.18 03:32:28.317905 [ 39 ] {} <Trace> HDFS3: FsImpl Connect to <hdfs>://<<IP_ADDRESS>>:9000
2023.07.18 03:32:28.317929 [ 39 ] {} <Trace> HDFS3: Hdfs opertaion hdfsBuilderConnect takes 0 ms
2023.07.18 03:32:28.318010 [ 39 ] {} <Information> StoragePolicySelector: Storage policy `default` loaded
2023.07.18 03:32:28.318491 [ 39 ] {} <Information> BackgroundSchedulePool/BgSchPool: Create BackgroundSchedulePool with 128 threads
2023.07.18 03:32:28.328733 [ 146 ] {} <Information> DiskCacheLoader: Loaded 0 segs from disk /var/byconity/data/
2023.07.18 03:32:28.328962 [ 39 ] {} <Information> BackgroundSchedulePool/CNCHTopoPol: Create BackgroundSchedulePool with 4 threads
2023.07.18 03:32:28.330347 [ 39 ] {} <Information> Application: Init cnch storage cache.
2023.07.18 03:32:28.330376 [ 39 ] {} <Error> CnchServerManager: There is no zookeeper, skip start background task for serverManager
2023.07.18 03:32:28.331066 [ 146 ] {} <Information> DiskCacheMigrator: Migrated 0 segs from disk /var/byconity/data/
2023.07.18 03:32:28.331179 [ 60 ] {} <Debug> ThreadPool: clear thread for finalize : [146, ]
2023.07.18 03:32:28.331204 [ 60 ] {} <Information> DiskCacheLRU: load thread takes 3 ms
2023.07.18 03:32:28.335109 [ 64 ] {} <Debug> PartCacheManager: Reloading 1 active tables.
2023.07.18 03:32:28.335520 [ 176 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:28.432355 [ 39 ] {} <Debug> CnchServerManager: Successfully set leader status.
2023.07.18 03:32:28.432405 [ 39 ] {} <Debug> CnchServerManager: Current node <IP_ADDRESS>:8124 become leader
2023.07.18 03:32:28.432416 [ 39 ] {} <Information> Application: Loading metadata from /var/byconity/
2023.07.18 03:32:28.434192 [ 105 ] {} <Debug> TSOClient[{<IP_ADDRESS>  rpc/9000}]: Initialized rpc channel to server with address: <IP_ADDRESS>:9000
2023.07.18 03:32:28.435999 [ 105 ] {} <Trace> Catalog: Updating topologies : []
2023.07.18 03:32:28.440480 [ 39 ] {} <Debug> ThreadPool: clear thread for finalize : [190, 183, ]
2023.07.18 03:32:28.440508 [ 39 ] {} <Information> DatabaseAtomic (system): Total 2 tables and 0 dictionaries.
2023.07.18 03:32:28.444609 [ 183 ] {} <Debug> StorageFactory:  engine name MergeTree
2023.07.18 03:32:28.446850 [ 183 ] {} <Trace> system.query_log (6c7d4514-4e93-4863-aa35-e4537211bea0)(MetaStore): Initializing storage metastore.
2023.07.18 03:32:28.447483 [ 177 ] {} <Debug> ServiceDiscoveryDNS: byconity-server-headless error message:domain name does not exist
2023.07.18 03:32:28.447509 [ 177 ] {} <Debug> ServiceDiscoveryDNS: lookup SERVER [byconity-server][] returns empty result
2023.07.18 03:32:28.447530 [ 177 ] {} <Error> CnchServerManager: Failed to get any server from service discovery, psm: byconity-server
2023.07.18 03:32:28.477194 [ 183 ] {} <Debug> system.query_log (6c7d4514-4e93-4863-aa35-e4537211bea0): Start loading data parts from filesystem.
2023.07.18 03:32:28.512641 [ 226 ] {} <Error> auto DB::MergeTreeData::loadPartsFromFileSystem(DB::MergeTreeData::PartNamesWithDisks, DB::MergeTreeData::PartNamesWithDisks, bool, DB::MergeTreeMetaBase::DataPartsLock &)::(anonymous class)::operator()() const: Code: 246, e.displayText() = DB::Exception: While loading part /var/byconity/data/store/6c7/6c7d4514-4e93-4863-aa35-e4537211bea0/20230718_20230718_1_1_0/: calculated partition ID: 19850809 differs from partition ID in part name: 202307 SQLSTATE: HY000, Stack trace (when copying this message, always include the lines below):

0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x190133f2 in /opt/byconity/bin/clickhouse
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb2e45a0 in /opt/byconity/bin/clickhouse
2. DB::IMergeTreeDataPart::loadPartitionAndMinMaxIndex() @ 0x14ab5609 in /opt/byconity/bin/clickhouse
3. DB::IMergeTreeDataPart::loadColumnsChecksumsIndexes(bool, bool) @ 0x14ab2360 in /opt/byconity/bin/clickhouse
4. ? @ 0x14b42818 in /opt/byconity/bin/clickhouse
5. ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0xb31eb5e in /opt/byconity/bin/clickhouse
6. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()&&...)::'lambda'()::operator()() @ 0xb320a64 in /opt/byconity/bin/clickhouse
7. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0xb31b7ff in /opt/byconity/bin/clickhouse
8. ? @ 0xb31fada in /opt/byconity/bin/clickhouse
9. start_thread @ 0x7fa3 in /lib/x86_64-linux-gnu/libpthread-2.28.so
10. clone @ 0xf906f in /lib/x86_64-linux-gnu/libc-2.28.so
 (version <IP_ADDRESS>)
2023.07.18 03:32:28.512642 [ 225 ] {} <Error> auto DB::MergeTreeData::loadPartsFromFileSystem(DB::MergeTreeData::PartNamesWithDisks, DB::MergeTreeData::PartNamesWithDisks, bool, DB::MergeTreeMetaBase::DataPartsLock &)::(anonymous class)::operator()() const: Code: 246, e.displayText() = DB::Exception: While loading part /var/byconity/data/store/6c7/6c7d4514-4e93-4863-aa35-e4537211bea0/20230718_20230718_3_3_0/: calculated partition ID: 19850809 differs from partition ID in part name: 202307 SQLSTATE: HY000, Stack trace (when copying this message, always include the lines below):

0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x190133f2 in /opt/byconity/bin/clickhouse
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb2e45a0 in /opt/byconity/bin/clickhouse
2. DB::IMergeTreeDataPart::loadPartitionAndMinMaxIndex() @ 0x14ab5609 in /opt/byconity/bin/clickhouse
3. DB::IMergeTreeDataPart::loadColumnsChecksumsIndexes(bool, bool) @ 0x14ab2360 in /opt/byconity/bin/clickhouse
4. ? @ 0x14b42818 in /opt/byconity/bin/clickhouse
5. ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0xb31eb5e in /opt/byconity/bin/clickhouse
6. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()&&...)::'lambda'()::operator()() @ 0xb320a64 in /opt/byconity/bin/clickhouse
7. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0xb31b7ff in /opt/byconity/bin/clickhouse
8. ? @ 0xb31fada in /opt/byconity/bin/clickhouse
9. start_thread @ 0x7fa3 in /lib/x86_64-linux-gnu/libpthread-2.28.so
10. clone @ 0xf906f in /lib/x86_64-linux-gnu/libc-2.28.so
 (version <IP_ADDRESS>)
2023.07.18 03:32:28.512716 [ 226 ] {} <Error> system.query_log (6c7d4514-4e93-4863-aa35-e4537211bea0): Detaching broken part /var/byconity/data/store/6c7/6c7d4514-4e93-4863-aa35-e4537211bea0/20230718_20230718_1_1_0. If it happened after update, it is likely because of backward incompability. You need to resolve this manually
2023.07.18 03:32:28.512720 [ 225 ] {} <Error> system.query_log (6c7d4514-4e93-4863-aa35-e4537211bea0): Detaching broken part /var/byconity/data/store/6c7/6c7d4514-4e93-4863-aa35-e4537211bea0/20230718_20230718_3_3_0. If it happened after update, it is likely because of backward incompability. You need to resolve this manually
2023.07.18 03:32:28.516925 [ 226 ] {} <Error> auto DB::MergeTreeData::loadPartsFromFileSystem(DB::MergeTreeData::PartNamesWithDisks, DB::MergeTreeData::PartNamesWithDisks, bool, DB::MergeTreeMetaBase::DataPartsLock &)::(anonymous class)::operator()() const: Code: 246, e.displayText() = DB::Exception: While loading part /var/byconity/data/store/6c7/6c7d4514-4e93-4863-aa35-e4537211bea0/20230718_20230718_2_2_0/: calculated partition ID: 19850809 differs from partition ID in part name: 202307 SQLSTATE: HY000, Stack trace (when copying this message, always include the lines below):

0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x190133f2 in /opt/byconity/bin/clickhouse
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb2e45a0 in /opt/byconity/bin/clickhouse
2. DB::IMergeTreeDataPart::loadPartitionAndMinMaxIndex() @ 0x14ab5609 in /opt/byconity/bin/clickhouse
3. DB::IMergeTreeDataPart::loadColumnsChecksumsIndexes(bool, bool) @ 0x14ab2360 in /opt/byconity/bin/clickhouse
4. ? @ 0x14b42818 in /opt/byconity/bin/clickhouse
5. ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0xb31eb5e in /opt/byconity/bin/clickhouse
6. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()&&...)::'lambda'()::operator()() @ 0xb320a64 in /opt/byconity/bin/clickhouse
7. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0xb31b7ff in /opt/byconity/bin/clickhouse
8. ? @ 0xb31fada in /opt/byconity/bin/clickhouse
9. start_thread @ 0x7fa3 in /lib/x86_64-linux-gnu/libpthread-2.28.so
10. clone @ 0xf906f in /lib/x86_64-linux-gnu/libc-2.28.so
 (version <IP_ADDRESS>)
2023.07.18 03:32:28.516961 [ 226 ] {} <Error> system.query_log (6c7d4514-4e93-4863-aa35-e4537211bea0): Detaching broken part /var/byconity/data/store/6c7/6c7d4514-4e93-4863-aa35-e4537211bea0/20230718_20230718_2_2_0. If it happened after update, it is likely because of backward incompability. You need to resolve this manually
2023.07.18 03:32:28.517405 [ 183 ] {} <Debug> system.query_log (6c7d4514-4e93-4863-aa35-e4537211bea0): Loaded data parts (0 items)
2023.07.18 03:32:28.517508 [ 183 ] {} <Debug> ThreadPool: clear thread for finalize : [226, 225, ]
2023.07.18 03:32:28.521196 [ 183 ] {} <Debug> system.query_log (6c7d4514-4e93-4863-aa35-e4537211bea0): Takes 43ms to load parts from file system.
2023.07.18 03:32:28.521231 [ 183 ] {} <Debug> system.query_log (6c7d4514-4e93-4863-aa35-e4537211bea0): Loaded data parts (0 items)
2023.07.18 03:32:28.521348 [ 39 ] {} <Information> DatabaseAtomic (system): Starting up tables.
2023.07.18 03:32:28.521466 [ 39 ] {} <Debug> ThreadPool: clear thread for finalize : [183, 190, ]
2023.07.18 03:32:28.529409 [ 39 ] {} <Information> DatabaseAtomic (default): Total 0 tables and 0 dictionaries.
2023.07.18 03:32:28.529433 [ 39 ] {} <Information> DatabaseAtomic (default): Starting up tables.
2023.07.18 03:32:28.529765 [ 39 ] {} <Information> DatabaseCatalog: Found 0 partially dropped tables. Will load them and retry removal.
2023.07.18 03:32:28.531527 [ 39 ] {} <Debug> Application: Loaded metadata.
2023.07.18 03:32:28.531558 [ 39 ] {} <Information> BackgroundSchedulePool/BgLocalPool: Create BackgroundSchedulePool with 16 threads
2023.07.18 03:32:28.532361 [ 39 ] {} <Information> Application: It looks like the process has no CAP_SYS_NICE capability, the setting 'os_thread_priority' will have no effect. It could happen due to incorrect ClickHouse package installation. You could resolve the problem manually with 'sudo setcap cap_sys_nice=+ep /opt/byconity/bin/clickhouse'. Note that it will not work on 'nosuid' mounted filesystems.
2023.07.18 03:32:28.535987 [ 39 ] {} <Debug> Application: Requested any available port (port == 0), actual port is 45629
2023.07.18 03:32:28.536079 [ 39 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 12.75 MiB, peak 20.45 MiB, will set to 320.24 MiB (RSS), difference: 307.50 MiB
2023.07.18 03:32:28.538104 [ 39 ] {} <Information> DNSCacheUpdater: Update period 15 seconds
2023.07.18 03:32:28.538135 [ 39 ] {} <Information> Application: Available RAM: 3.83 GiB; physical cores: 2; logical cores: 2.
2023.07.18 03:32:28.538578 [ 67 ] {} <Debug> DNSResolver: Updating DNS cache
2023.07.18 03:32:28.538604 [ 67 ] {} <Debug> DNSResolver: Updated DNS cache
2023.07.18 03:32:28.544011 [ 39 ] {} <Information> brpc: [../contrib/incubator-brpc/src/brpc/server.cpp:1101]Server[DB::BrpcExchangeReceiverRegistryService] is serving on [::]:9410 with builtin service.

2023.07.18 03:32:28.545124 [ 39 ] {} <Information> brpc: [../contrib/incubator-brpc/src/brpc/server.cpp:1101]Server[DB::RuntimeFilterService+DB::PlanSegmentManagerRpcService] is serving on [::]:9510 with builtin service.

2023.07.18 03:32:28.545148 [ 39 ] {} <Information> Application: start BrpcExchangeReceiverRegistryService listening :: [::]:9410
2023.07.18 03:32:28.545155 [ 39 ] {} <Information> Application: start PlanSegmentManagerRpcService listening :: [::]:9510
2023.07.18 03:32:28.545249 [ 39 ] {} <Information> Application: Listening for http://[::]:8123
2023.07.18 03:32:28.545324 [ 39 ] {} <Information> Application: Listening for native protocol (tcp): [::]:9000
2023.07.18 03:32:28.545395 [ 39 ] {} <Information> Application: Listening for Prometheus: http://[::]:45629
2023.07.18 03:32:28.545854 [ 39 ] {} <Debug> GlobalGCManager: init thread pool with max_threads: 3 max_free_threads: 1 queue_size: 60
2023.07.18 03:32:28.548296 [ 39 ] {} <Information> brpc: [../contrib/incubator-brpc/src/brpc/server.cpp:1101]Server[DB::CnchServerServiceImpl] is serving on [::]:8124 with builtin service.

2023.07.18 03:32:28.548323 [ 39 ] {} <Information> Application: Listening rpc: [::]:8124
2023.07.18 03:32:28.548331 [ 39 ] {} <Information> Application: Ready for connections.
2023.07.18 03:32:28.837567 [ 176 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:28.949128 [ 169 ] {} <Debug> ServiceDiscoveryDNS: byconity-server-headless error message:domain name does not exist
2023.07.18 03:32:28.949175 [ 169 ] {} <Debug> ServiceDiscoveryDNS: lookup SERVER [byconity-server][] returns empty result
2023.07.18 03:32:28.949215 [ 169 ] {} <Error> CnchServerManager: Failed to get any server from service discovery, psm: byconity-server
2023.07.18 03:32:29.000146 [ 246 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 326.19 MiB, peak 326.19 MiB, will set to 328.75 MiB (RSS), difference: 2.57 MiB
2023.07.18 03:32:29.339962 [ 105 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:29.436598 [ 177 ] {} <Trace> Catalog: Updating topologies : []
2023.07.18 03:32:29.450447 [ 176 ] {} <Debug> ServiceDiscoveryDNS: byconity-server-headless error message:domain name does not exist
2023.07.18 03:32:29.450493 [ 176 ] {} <Debug> ServiceDiscoveryDNS: lookup SERVER [byconity-server][] returns empty result
2023.07.18 03:32:29.450519 [ 176 ] {} <Error> CnchServerManager: Failed to get any server from service discovery, psm: byconity-server
2023.07.18 03:32:29.842421 [ 177 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:29.951742 [ 105 ] {} <Debug> ServiceDiscoveryDNS: byconity-server-headless error message:domain name does not exist
2023.07.18 03:32:29.951786 [ 105 ] {} <Debug> ServiceDiscoveryDNS: lookup SERVER [byconity-server][] returns empty result
2023.07.18 03:32:29.951807 [ 105 ] {} <Error> CnchServerManager: Failed to get any server from service discovery, psm: byconity-server
2023.07.18 03:32:30.000115 [ 246 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 328.75 MiB, peak 328.75 MiB, will set to 332.45 MiB (RSS), difference: 3.69 MiB
2023.07.18 03:32:30.344933 [ 169 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:30.437261 [ 176 ] {} <Trace> Catalog: Updating topologies : []
2023.07.18 03:32:30.452907 [ 177 ] {} <Debug> ServiceDiscoveryDNS: byconity-server-headless error message:domain name does not exist
2023.07.18 03:32:30.452945 [ 177 ] {} <Debug> ServiceDiscoveryDNS: lookup SERVER [byconity-server][] returns empty result
2023.07.18 03:32:30.452966 [ 177 ] {} <Error> CnchServerManager: Failed to get any server from service discovery, psm: byconity-server
2023.07.18 03:32:30.847308 [ 176 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:30.954113 [ 169 ] {} <Debug> ServiceDiscoveryDNS: byconity-server-headless error message:domain name does not exist
2023.07.18 03:32:30.954161 [ 169 ] {} <Debug> ServiceDiscoveryDNS: lookup SERVER [byconity-server][] returns empty result
2023.07.18 03:32:30.954181 [ 169 ] {} <Error> CnchServerManager: Failed to get any server from service discovery, psm: byconity-server
2023.07.18 03:32:31.349613 [ 105 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:31.437886 [ 177 ] {} <Trace> Catalog: Updating topologies : []
2023.07.18 03:32:31.455466 [ 176 ] {} <Debug> ServiceDiscoveryDNS: byconity-server-headless error message:domain name does not exist
2023.07.18 03:32:31.455508 [ 176 ] {} <Debug> ServiceDiscoveryDNS: lookup SERVER [byconity-server][] returns empty result
2023.07.18 03:32:31.455531 [ 176 ] {} <Error> CnchServerManager: Failed to get any server from service discovery, psm: byconity-server
2023.07.18 03:32:31.851839 [ 177 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:31.956579 [ 105 ] {} <Debug> ServiceDiscoveryDNS: byconity-server-headless error message:domain name does not exist
2023.07.18 03:32:31.956623 [ 105 ] {} <Debug> ServiceDiscoveryDNS: lookup SERVER [byconity-server][] returns empty result
2023.07.18 03:32:31.956663 [ 105 ] {} <Error> CnchServerManager: Failed to get any server from service discovery, psm: byconity-server
2023.07.18 03:32:32.354065 [ 169 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:32.438408 [ 176 ] {} <Trace> Catalog: Updating topologies : []
2023.07.18 03:32:32.457754 [ 105 ] {} <Debug> ServiceDiscoveryDNS: byconity-server-headless error message:domain name does not exist
2023.07.18 03:32:32.457793 [ 105 ] {} <Debug> ServiceDiscoveryDNS: lookup SERVER [byconity-server][] returns empty result
2023.07.18 03:32:32.457813 [ 105 ] {} <Error> CnchServerManager: Failed to get any server from service discovery, psm: byconity-server
2023.07.18 03:32:32.770822 [ 241 ] {} <Trace> TCPHandlerFactory: TCP Request. Address: [::ffff:<IP_ADDRESS>]:56718
2023.07.18 03:32:32.770996 [ 241 ] {} <Debug> TCPHandler: Connected ClickHouse client version 21.8.0, revision: 54450, user: probe.
2023.07.18 03:32:32.771114 [ 241 ] {} <Trace> ContextAccess (probe): Settings: readonly=0, allow_ddl=true, allow_introspection_functions=false
2023.07.18 03:32:32.771145 [ 241 ] {} <Trace> ContextAccess (probe): List of all grants: GRANT SHOW, SELECT, INSERT, ALTER, CREATE, DROP, TRUNCATE, OPTIMIZE, KILL QUERY, MOVE PARTITION BETWEEN SHARDS, SYSTEM, dictGet, INTROSPECTION, SOURCES ON *.*
2023.07.18 03:32:32.771160 [ 241 ] {} <Trace> ContextAccess (probe): List of all grants including implicit: GRANT SHOW, SELECT, INSERT, ALTER, CREATE, DROP, TRUNCATE, OPTIMIZE, KILL QUERY, MOVE PARTITION BETWEEN SHARDS, SYSTEM, dictGet, INTROSPECTION, SOURCES ON *.*
2023.07.18 03:32:32.772375 [ 241 ] {f46287a7-cb93-4479-910b-eaa8bbc162ee} <Debug> TransactionCoordinator: Created txn txn_id:<PHONE_NUMBER>71954944 commit_ts: 0 status: Running priority: high location: "<IP_ADDRESS>:8124" initiator: "Server" type: Implicit

2023.07.18 03:32:32.772426 [ 241 ] {f46287a7-cb93-4479-910b-eaa8bbc162ee} <Debug> trySetVirtualWarehouse: Trying to set virtual warehouse for query `SELECT 1` ...
2023.07.18 03:32:32.772438 [ 241 ] {f46287a7-cb93-4479-910b-eaa8bbc162ee} <Debug> trySetVirtualWarehouse: Trying to set virtual warehouse for expression `SELECT 1` ...
2023.07.18 03:32:32.772448 [ 241 ] {f46287a7-cb93-4479-910b-eaa8bbc162ee} <Debug> trySetVirtualWarehouse: Trying to set virtual warehouse for expression `SELECT 1` ...
2023.07.18 03:32:32.772458 [ 241 ] {f46287a7-cb93-4479-910b-eaa8bbc162ee} <Debug> trySetVirtualWarehouse: Trying to set virtual warehouse for expression `SELECT 1` ...
2023.07.18 03:32:32.772465 [ 241 ] {f46287a7-cb93-4479-910b-eaa8bbc162ee} <Debug> trySetVirtualWarehouse: Trying to set virtual warehouse for expression `1` ...
2023.07.18 03:32:32.772472 [ 241 ] {f46287a7-cb93-4479-910b-eaa8bbc162ee} <Debug> trySetVirtualWarehouse: Trying to set virtual warehouse for expression `1` ...
2023.07.18 03:32:32.772503 [ 241 ] {f46287a7-cb93-4479-910b-eaa8bbc162ee} <Debug> executeQuery: (from [::ffff:<IP_ADDRESS>]:56718, user: probe) select 1
2023.07.18 03:32:32.775107 [ 241 ] {f46287a7-cb93-4479-910b-eaa8bbc162ee} <Trace> ContextAccess (probe): Access granted: SELECT(dummy) ON system.one
2023.07.18 03:32:32.775155 [ 241 ] {f46287a7-cb93-4479-910b-eaa8bbc162ee} <Trace> InterpreterSelectQuery: query: SELECT 1
2023.07.18 03:32:32.775210 [ 241 ] {f46287a7-cb93-4479-910b-eaa8bbc162ee} <Trace> InterpreterSelectQuery: FetchColumns -> Complete
2023.07.18 03:32:32.775301 [ 241 ] {f46287a7-cb93-4479-910b-eaa8bbc162ee} <Debug> QueryPlan: Build pipeline takes:0
2023.07.18 03:32:32.775580 [ 253 ] {f46287a7-cb93-4479-910b-eaa8bbc162ee} <Trace> ThreadStatus: node SourceWithProgress checkCpuTimeLimit thread cpu secs = 0.0, total cpu secs = 0.0, max = 0
2023.07.18 03:32:32.775672 [ 253 ] {f46287a7-cb93-4479-910b-eaa8bbc162ee} <Trace> PipelineExecutor: Pipeline: digraph
{
  rankdir="LR";
  { node [shape = record]
    n139900308022552[label="{SourceFromSingleChunk(2 jobs) (Finished)}"];
    n139900467864344[label="{ExpressionTransform(1 jobs) (Finished)}"];
    n139900468216856[label="{LimitsCheckingTransform(1 jobs) (Finished)}"];
    n139900468003800[label="{NullSource(1 jobs) (Finished)}"];
    n139900468003160[label="{NullSource(1 jobs) (Finished)}"];
    n139900467990552[label="{LazyOutputFormat(2 jobs) (Finished)}"];
  }
  n139900308022552 -> n139900467864344;
  n139900467864344 -> n139900468216856;
  n139900468216856 -> n139900467990552;
  n139900468003800 -> n139900467990552;
  n139900468003160 -> n139900467990552;
}

2023.07.18 03:32:32.775823 [ 241 ] {f46287a7-cb93-4479-910b-eaa8bbc162ee} <Debug> TransactionCoordinator: Deleted txn<PHONE_NUMBER>71954944

2023.07.18 03:32:32.775839 [ 241 ] {f46287a7-cb93-4479-910b-eaa8bbc162ee} <Trace> TransactionCoordinator: No need to erase active timestamp for txn<PHONE_NUMBER>71954944

2023.07.18 03:32:32.775878 [ 241 ] {f46287a7-cb93-4479-910b-eaa8bbc162ee} <Information> executeQuery: Read 1 rows, 1.00 B in 0.003337556 sec., 299 rows/sec., 299.62 B/sec.
2023.07.18 03:32:32.776008 [ 241 ] {f46287a7-cb93-4479-910b-eaa8bbc162ee} <Debug> MemoryTracker: Peak memory usage (for query): 0.00 B.
2023.07.18 03:32:32.776095 [ 241 ] {} <Debug> TCPHandler: Processed in 0.004346869 sec.
2023.07.18 03:32:32.777432 [ 241 ] {} <Debug> TCPHandler: Done processing connection.
2023.07.18 03:32:32.856248 [ 176 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:32.958960 [ 177 ] {} <Debug> ServiceDiscoveryDNS: byconity-server-headless error message:domain name does not exist
2023.07.18 03:32:32.959004 [ 177 ] {} <Debug> ServiceDiscoveryDNS: lookup SERVER [byconity-server][] returns empty result
2023.07.18 03:32:32.959037 [ 177 ] {} <Error> CnchServerManager: Failed to get any server from service discovery, psm: byconity-server
2023.07.18 03:32:33.358524 [ 105 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:33.438912 [ 177 ] {} <Trace> Catalog: Updating topologies : []
2023.07.18 03:32:33.460304 [ 169 ] {} <Debug> ServiceDiscoveryDNS: byconity-server-headless error message:domain name does not exist
2023.07.18 03:32:33.460345 [ 169 ] {} <Debug> ServiceDiscoveryDNS: lookup SERVER [byconity-server][] returns empty result
2023.07.18 03:32:33.460365 [ 169 ] {} <Error> CnchServerManager: Failed to get any server from service discovery, psm: byconity-server
2023.07.18 03:32:33.860817 [ 177 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:33.961449 [ 169 ] {} <Debug> ServiceDiscoveryDNS: byconity-server-headless error message:domain name does not exist
2023.07.18 03:32:33.961495 [ 169 ] {} <Debug> ServiceDiscoveryDNS: lookup SERVER [byconity-server][] returns empty result
2023.07.18 03:32:33.961514 [ 169 ] {} <Error> CnchServerManager: Failed to get any server from service discovery, psm: byconity-server
2023.07.18 03:32:34.000125 [ 246 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 333.76 MiB, peak 335.20 MiB, will set to 335.11 MiB (RSS), difference: 1.34 MiB
2023.07.18 03:32:34.363023 [ 169 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:34.439402 [ 176 ] {} <Trace> Catalog: Updating topologies : []

kubectl logs byconity-vw-vw-default-0    -n kaya-dev
rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
rsyslogd: activation of module imklog failed [v8.1901.0 try https://www.rsyslog.com/e/2145 ]
jemalloc.background_thread was requested, however ClickHouse uses percpu_arena and background_thread most likely will not give any benefits, and also background_thread is not compatible with ClickHouse watchdog (that can be disabled with CLICKHOUSE_WATCHDOG_ENABLE=0)
Processing configuration file '/etc/byconity/worker.yaml'.
Logging trace to /var/log/byconity/byconity.log
Logging errors to /var/log/byconity/byconity.err.log
Logging trace to console
2023.07.18 03:32:20.013134 [ 12 ] {} <Information> Application: Will watch for the process with pid 38
2023.07.18 03:32:20.013350 [ 38 ] {} <Information> Application: Forked a child process to watch
2023.07.18 03:32:20.013754 [ 38 ] {} <Information> SentryWriter: Sending crash reports is disabled
2023.07.18 03:32:20.013887 [ 38 ] {} <Trace> Pipe: Pipe capacity is 1.00 MiB
2023.07.18 03:32:20.154672 [ 38 ] {} <Information> : Starting ClickHouse <IP_ADDRESS> scm <IP_ADDRESS> with revision 54453, build id: E74C4B9AFA78CDA9, PID 38
2023.07.18 03:32:20.156670 [ 38 ] {} <Information> Application: starting up
2023.07.18 03:32:20.158094 [ 38 ] {} <Information> Application: OS name: Linux, version: 5.4.119-1-tlinux4-0009-public-eks, architecture: x86_64
2023.07.18 03:32:20.396428 [ 38 ] {} <Debug> registerStorageCnchHive: registerStorageCnchHive
Processing configuration file '/etc/byconity/cnch-config.yaml'.
Processing configuration file '/etc/byconity/cnch-config.yaml'.
2023.07.18 03:32:20.420189 [ 38 ] {} <Debug> Configuration: Config element service_discovery.vw.psm is recommended to set in config.xml. You'd better customize it.
2023.07.18 03:32:20.420640 [ 38 ] {} <Information> BrpcGflagsConfigHolder: Set brpc gflags [ max_body_size : 671088640 ]
2023.07.18 03:32:20.420707 [ 38 ] {} <Information> BrpcGflagsConfigHolder: Set brpc gflags [ socket_max_unwritten_bytes :<PHONE_NUMBER> ]
2023.07.18 03:32:20.420729 [ 38 ] {} <Information> BrpcGflagsConfigHolder: Set brpc gflags [ defer_close_second : 60 ]
2023.07.18 03:32:20.420751 [ 38 ] {} <Information> BrpcGflagsConfigHolder: Set brpc gflags [ event_dispatcher_num : 2 ]
2023.07.18 03:32:20.420 <Info> BrpcChannelPoolConfigHolder::afterinit Init brpc client channel pool with config: PoolOptions@0x7f2a3a6cb828 {pool_name:stream_default, max_connections:4, load_balancer:rr, ChannelOptions:{connect_timout_ms:1000, timeout_ms:3000, backup_request_ms:-1, max_retry:3, enable_circuit_breaker:false, succeed_without_server:true, log_succeed_without_server:true, connection_group:}}
PoolOptions@0x7f2a3a6cb728 {pool_name:rpc_default, max_connections:2, load_balancer:rr, ChannelOptions:{connect_timout_ms:1000, timeout_ms:3000, backup_request_ms:-1, max_retry:3, enable_circuit_breaker:false, succeed_without_server:true, log_succeed_without_server:true, connection_group:}}

2023.07.18 03:32:20.420 <Info> BrpcApplication::initialize Brpc is initialized with {} config holders2 config holders
2023.07.18 03:32:20.496242 [ 38 ] {} <Debug> Context: Initialising Resource Manager Client
2023.07.18 03:32:20.502363 [ 38 ] {} <Debug> ResourceManagerClient[{<IP_ADDRESS>  rpc/9000}]: Initialized rpc channel to server with address: <IP_ADDRESS>:9000
2023.07.18 03:32:20.502411 [ 38 ] {} <Debug> Context: Initialised Resource Manager Client on try: 0
2023.07.18 03:32:20.502435 [ 38 ] {} <Information> CnchWorkerClientPools: Added new vw: vw_read
2023.07.18 03:32:20.502446 [ 38 ] {} <Information> CnchWorkerClientPools: Added new vw: vw_write
2023.07.18 03:32:20.502456 [ 38 ] {} <Information> CnchWorkerClientPools: Added new vw: vw_task
2023.07.18 03:32:20.502465 [ 38 ] {} <Information> CnchWorkerClientPools: Added new vw: vw_default
2023.07.18 03:32:21.662477 [ 38 ] {} <Warning> Application: Calculated checksum of the binary: D1EB286171264FAA33033121FE59C8DD. There is no information about the reference checksum.
2023.07.18 03:32:21.663574 [ 38 ] {} <Information> StatusFile: Status file /var/byconity/status already exists - unclean restart. Contents:
PID: 39
Started at: 2023-07-18 03:30:27
Revision: 54453

2023.07.18 03:32:21.664145 [ 38 ] {} <Debug> Application: rlimit on number of file descriptors is 1048576
2023.07.18 03:32:21.664180 [ 38 ] {} <Debug> Application: Initializing DateLUT.
2023.07.18 03:32:21.664189 [ 38 ] {} <Trace> Application: Initialized DateLUT with time zone 'Etc/UTC'.
2023.07.18 03:32:21.664215 [ 38 ] {} <Debug> Application: Setting up /var/byconity/tmp/ to store temporary data in it
2023.07.18 03:32:21.668900 [ 38 ] {} <Debug> Application: Initiailizing interserver credentials.
2023.07.18 03:32:21.671552 [ 38 ] {} <Debug> ConfigReloader: Loading config '/etc/byconity/worker.yaml'
Processing configuration file '/etc/byconity/worker.yaml'.
Saved preprocessed configuration to '/var/byconity/preprocessed_configs/worker.yaml'.
2023.07.18 03:32:21.672821 [ 38 ] {} <Debug> ConfigReloader: Loaded config '/etc/byconity/worker.yaml', performing update on configuration
2023.07.18 03:32:21.673150 [ 38 ] {} <Information> Application: Setting max_server_memory_usage was set to 3.44 GiB (3.83 GiB available * 0.90 max_server_memory_usage_to_ram_ratio)
2023.07.18 03:32:21.673 <Info> Reload brpc config with 2 config holders
2023.07.18 03:32:21.673365 [ 38 ] {} <Debug> ConfigReloader: Loaded config '/etc/byconity/worker.yaml', performed update on configuration
Processing configuration file '/etc/byconity/users.yaml'.
2023.07.18 03:32:21.676902 [ 38 ] {} <Debug> Access(user directories): Added users.xml access storage 'users.xml', path:
2023.07.18 03:32:21.677438 [ 38 ] {} <Debug> ConfigReloader: Loading config '/etc/byconity/users.yaml'
Processing configuration file '/etc/byconity/users.yaml'.
Saved preprocessed configuration to '/var/byconity/preprocessed_configs/users.yaml'.
2023.07.18 03:32:21.677896 [ 38 ] {} <Debug> ConfigReloader: Loaded config '/etc/byconity/users.yaml', performing update on configuration
2023.07.18 03:32:21.678182 [ 38 ] {} <Debug> ConfigReloader: Loaded config '/etc/byconity/users.yaml', performed update on configuration
2023.07.18 03:32:21.678492 [ 38 ] {} <Debug> Access(user directories): Added users.xml access storage 'users.xml', path: /etc/byconity/users.yaml
2023.07.18 03:32:21.679710 [ 38 ] {} <Information> Application: Mark cache size was lowered to 1.91 GiB because the system has low amount of memory
2023.07.18 03:32:21.679752 [ 38 ] {} <Information> KeyIndexFileCache: Initializing KeyIndexFileCache...
2023.07.18 03:32:21.680685 [ 38 ] {} <Information> KeyIndexFileCache: Initialized KeyIndexFileCache with 0 entries in 0 ms
2023.07.18 03:32:21.684839 [ 38 ] {} <Trace> HDFS3: Hdfs opertaion DefaultConfig::ctor takes 3 ms
2023.07.18 03:32:21.689067 [ 38 ] {} <Trace> HDFS3: FsImpl Connect to <hdfs>://<<IP_ADDRESS>>:9000
2023.07.18 03:32:21.689112 [ 38 ] {} <Trace> HDFS3: Hdfs opertaion hdfsBuilderConnect takes 4 ms
2023.07.18 03:32:21.698246 [ 38 ] {} <Trace> HDFS3: Hdfs opertaion DefaultConfig::ctor takes 0 ms
2023.07.18 03:32:21.698388 [ 38 ] {} <Trace> HDFS3: FsImpl Connect to <hdfs>://<<IP_ADDRESS>>:9000
2023.07.18 03:32:21.698432 [ 38 ] {} <Trace> HDFS3: Hdfs opertaion hdfsBuilderConnect takes 0 ms
2023.07.18 03:32:21.698526 [ 38 ] {} <Information> StoragePolicySelector: Storage policy `default` loaded
2023.07.18 03:32:21.698686 [ 38 ] {} <Information> BackgroundSchedulePool/BgSchPool: Create BackgroundSchedulePool with 128 threads
2023.07.18 03:32:21.710184 [ 38 ] {} <Information> BackgroundSchedulePool/CNCHTopoPol: Create BackgroundSchedulePool with 4 threads
2023.07.18 03:32:21.710962 [ 38 ] {} <Information> Application: Loading metadata from /var/byconity/
2023.07.18 03:32:21.711803 [ 193 ] {} <Information> DiskCacheLoader: Loaded 0 segs from disk /var/byconity/data/
2023.07.18 03:32:21.713513 [ 193 ] {} <Information> DiskCacheMigrator: Migrated 0 segs from disk /var/byconity/data/
2023.07.18 03:32:21.713913 [ 66 ] {} <Debug> ThreadPool: clear thread for finalize : [193, ]
2023.07.18 03:32:21.713947 [ 66 ] {} <Information> DiskCacheLRU: load thread takes 3 ms
2023.07.18 03:32:21.718218 [ 151 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:21.718213 [ 38 ] {} <Debug> ThreadPool: clear thread for finalize : [193, ]
2023.07.18 03:32:21.718262 [ 38 ] {} <Information> DatabaseAtomic (system): Total 1 tables and 0 dictionaries.
2023.07.18 03:32:21.723610 [ 193 ] {} <Debug> StorageFactory:  engine name MergeTree
2023.07.18 03:32:21.725469 [ 193 ] {} <Trace> system.query_log (70999914-6aa1-4c0c-b33d-f9a75294d36f)(MetaStore): Initializing storage metastore.
2023.07.18 03:32:21.762126 [ 193 ] {} <Debug> system.query_log (70999914-6aa1-4c0c-b33d-f9a75294d36f): Start loading data parts from filesystem.
2023.07.18 03:32:21.804325 [ 229 ] {} <Error> auto DB::MergeTreeData::loadPartsFromFileSystem(DB::MergeTreeData::PartNamesWithDisks, DB::MergeTreeData::PartNamesWithDisks, bool, DB::MergeTreeMetaBase::DataPartsLock &)::(anonymous class)::operator()() const: Code: 246, e.displayText() = DB::Exception: While loading part /var/byconity/data/store/709/70999914-6aa1-4c0c-b33d-f9a75294d36f/20230718_20230718_2_2_0/: calculated partition ID: 19850809 differs from partition ID in part name: 202307 SQLSTATE: HY000, Stack trace (when copying this message, always include the lines below):

0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x190133f2 in /opt/byconity/bin/clickhouse
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb2e45a0 in /opt/byconity/bin/clickhouse
2. DB::IMergeTreeDataPart::loadPartitionAndMinMaxIndex() @ 0x14ab5609 in /opt/byconity/bin/clickhouse
3. DB::IMergeTreeDataPart::loadColumnsChecksumsIndexes(bool, bool) @ 0x14ab2360 in /opt/byconity/bin/clickhouse
4. ? @ 0x14b42818 in /opt/byconity/bin/clickhouse
5. ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0xb31eb5e in /opt/byconity/bin/clickhouse
6. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()&&...)::'lambda'()::operator()() @ 0xb320a64 in /opt/byconity/bin/clickhouse
7. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0xb31b7ff in /opt/byconity/bin/clickhouse
8. ? @ 0xb31fada in /opt/byconity/bin/clickhouse
9. start_thread @ 0x7fa3 in /lib/x86_64-linux-gnu/libpthread-2.28.so
10. clone @ 0xf906f in /lib/x86_64-linux-gnu/libc-2.28.so
 (version <IP_ADDRESS>)
2023.07.18 03:32:21.804338 [ 230 ] {} <Error> auto DB::MergeTreeData::loadPartsFromFileSystem(DB::MergeTreeData::PartNamesWithDisks, DB::MergeTreeData::PartNamesWithDisks, bool, DB::MergeTreeMetaBase::DataPartsLock &)::(anonymous class)::operator()() const: Code: 246, e.displayText() = DB::Exception: While loading part /var/byconity/data/store/709/70999914-6aa1-4c0c-b33d-f9a75294d36f/20230718_20230718_3_3_0/: calculated partition ID: 19850809 differs from partition ID in part name: 202307 SQLSTATE: HY000, Stack trace (when copying this message, always include the lines below):

0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x190133f2 in /opt/byconity/bin/clickhouse
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb2e45a0 in /opt/byconity/bin/clickhouse
2. DB::IMergeTreeDataPart::loadPartitionAndMinMaxIndex() @ 0x14ab5609 in /opt/byconity/bin/clickhouse
3. DB::IMergeTreeDataPart::loadColumnsChecksumsIndexes(bool, bool) @ 0x14ab2360 in /opt/byconity/bin/clickhouse
4. ? @ 0x14b42818 in /opt/byconity/bin/clickhouse
5. ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0xb31eb5e in /opt/byconity/bin/clickhouse
6. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()&&...)::'lambda'()::operator()() @ 0xb320a64 in /opt/byconity/bin/clickhouse
7. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0xb31b7ff in /opt/byconity/bin/clickhouse
8. ? @ 0xb31fada in /opt/byconity/bin/clickhouse
9. start_thread @ 0x7fa3 in /lib/x86_64-linux-gnu/libpthread-2.28.so
10. clone @ 0xf906f in /lib/x86_64-linux-gnu/libc-2.28.so
 (version <IP_ADDRESS>)
2023.07.18 03:32:21.804376 [ 229 ] {} <Error> system.query_log (70999914-6aa1-4c0c-b33d-f9a75294d36f): Detaching broken part /var/byconity/data/store/709/70999914-6aa1-4c0c-b33d-f9a75294d36f/20230718_20230718_2_2_0. If it happened after update, it is likely because of backward incompability. You need to resolve this manually
2023.07.18 03:32:21.804381 [ 230 ] {} <Error> system.query_log (70999914-6aa1-4c0c-b33d-f9a75294d36f): Detaching broken part /var/byconity/data/store/709/70999914-6aa1-4c0c-b33d-f9a75294d36f/20230718_20230718_3_3_0. If it happened after update, it is likely because of backward incompability. You need to resolve this manually
2023.07.18 03:32:21.807814 [ 229 ] {} <Error> auto DB::MergeTreeData::loadPartsFromFileSystem(DB::MergeTreeData::PartNamesWithDisks, DB::MergeTreeData::PartNamesWithDisks, bool, DB::MergeTreeMetaBase::DataPartsLock &)::(anonymous class)::operator()() const: Code: 246, e.displayText() = DB::Exception: While loading part /var/byconity/data/store/709/70999914-6aa1-4c0c-b33d-f9a75294d36f/20230718_20230718_1_1_0/: calculated partition ID: 19850809 differs from partition ID in part name: 202307 SQLSTATE: HY000, Stack trace (when copying this message, always include the lines below):

0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x190133f2 in /opt/byconity/bin/clickhouse
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb2e45a0 in /opt/byconity/bin/clickhouse
2. DB::IMergeTreeDataPart::loadPartitionAndMinMaxIndex() @ 0x14ab5609 in /opt/byconity/bin/clickhouse
3. DB::IMergeTreeDataPart::loadColumnsChecksumsIndexes(bool, bool) @ 0x14ab2360 in /opt/byconity/bin/clickhouse
4. ? @ 0x14b42818 in /opt/byconity/bin/clickhouse
5. ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0xb31eb5e in /opt/byconity/bin/clickhouse
6. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()&&...)::'lambda'()::operator()() @ 0xb320a64 in /opt/byconity/bin/clickhouse
7. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0xb31b7ff in /opt/byconity/bin/clickhouse
8. ? @ 0xb31fada in /opt/byconity/bin/clickhouse
9. start_thread @ 0x7fa3 in /lib/x86_64-linux-gnu/libpthread-2.28.so
10. clone @ 0xf906f in /lib/x86_64-linux-gnu/libc-2.28.so
 (version <IP_ADDRESS>)
2023.07.18 03:32:21.807847 [ 229 ] {} <Error> system.query_log (70999914-6aa1-4c0c-b33d-f9a75294d36f): Detaching broken part /var/byconity/data/store/709/70999914-6aa1-4c0c-b33d-f9a75294d36f/20230718_20230718_1_1_0. If it happened after update, it is likely because of backward incompability. You need to resolve this manually
2023.07.18 03:32:21.808292 [ 193 ] {} <Debug> system.query_log (70999914-6aa1-4c0c-b33d-f9a75294d36f): Loaded data parts (0 items)
2023.07.18 03:32:21.808377 [ 193 ] {} <Debug> ThreadPool: clear thread for finalize : [230, 229, ]
2023.07.18 03:32:21.812737 [ 193 ] {} <Debug> system.query_log (70999914-6aa1-4c0c-b33d-f9a75294d36f): Takes 50ms to load parts from file system.
2023.07.18 03:32:21.812756 [ 193 ] {} <Debug> system.query_log (70999914-6aa1-4c0c-b33d-f9a75294d36f): Loaded data parts (0 items)
2023.07.18 03:32:21.812875 [ 38 ] {} <Information> DatabaseAtomic (system): Starting up tables.
2023.07.18 03:32:21.812987 [ 38 ] {} <Debug> ThreadPool: clear thread for finalize : [193, ]
2023.07.18 03:32:21.817126 [ 38 ] {} <Information> DatabaseAtomic (default): Total 0 tables and 0 dictionaries.
2023.07.18 03:32:21.817146 [ 38 ] {} <Information> DatabaseAtomic (default): Starting up tables.
2023.07.18 03:32:21.817539 [ 38 ] {} <Information> DatabaseCatalog: Found 0 partially dropped tables. Will load them and retry removal.
2023.07.18 03:32:21.817560 [ 38 ] {} <Debug> Application: Loaded metadata.
2023.07.18 03:32:21.817573 [ 38 ] {} <Information> BackgroundSchedulePool/BgLocalPool: Create BackgroundSchedulePool with 16 threads
2023.07.18 03:32:21.818352 [ 38 ] {} <Information> Application: It looks like the process has no CAP_SYS_NICE capability, the setting 'os_thread_priority' will have no effect. It could happen due to incorrect ClickHouse package installation. You could resolve the problem manually with 'sudo setcap cap_sys_nice=+ep /opt/byconity/bin/clickhouse'. Note that it will not work on 'nosuid' mounted filesystems.
2023.07.18 03:32:21.820148 [ 38 ] {} <Debug> Application: Requested any available port (port == 0), actual port is 34179
2023.07.18 03:32:21.820248 [ 38 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 12.01 MiB, peak 19.75 MiB, will set to 320.80 MiB (RSS), difference: 308.79 MiB
2023.07.18 03:32:21.822160 [ 38 ] {} <Information> DNSCacheUpdater: Update period 15 seconds
2023.07.18 03:32:21.822194 [ 38 ] {} <Information> Application: Available RAM: 3.83 GiB; physical cores: 2; logical cores: 2.
2023.07.18 03:32:21.822236 [ 68 ] {} <Debug> DNSResolver: Updating DNS cache
2023.07.18 03:32:21.822264 [ 68 ] {} <Debug> DNSResolver: Updated DNS cache
2023.07.18 03:32:21.828089 [ 38 ] {} <Information> brpc: [../contrib/incubator-brpc/src/brpc/server.cpp:1101]Server[DB::BrpcExchangeReceiverRegistryService] is serving on [::]:9410 with builtin service.

2023.07.18 03:32:21.829122 [ 38 ] {} <Information> brpc: [../contrib/incubator-brpc/src/brpc/server.cpp:1101]Server[DB::RuntimeFilterService+DB::PlanSegmentManagerRpcService] is serving on [::]:9510 with builtin service.

2023.07.18 03:32:21.829143 [ 38 ] {} <Information> Application: start BrpcExchangeReceiverRegistryService listening :: [::]:9410
2023.07.18 03:32:21.829150 [ 38 ] {} <Information> Application: start PlanSegmentManagerRpcService listening :: [::]:9510
2023.07.18 03:32:21.829175 [ 38 ] {} <Trace> ResourceReporterTask: Create ResourceReporterTask.
2023.07.18 03:32:21.829269 [ 38 ] {} <Information> Application: Listening for http://[::]:8123
2023.07.18 03:32:21.829341 [ 38 ] {} <Information> Application: Listening for native protocol (tcp): [::]:9000
2023.07.18 03:32:21.829415 [ 38 ] {} <Information> Application: Listening for Prometheus: http://[::]:34179
2023.07.18 03:32:21.830831 [ 65 ] {} <Trace> ResourceReporterTask: Register Node in RM:  self: {byconity-vw-vw-default-0 <IP_ADDRESS>  rpc/8124 tcp/9000 exc/9410 exs/9510}
2023.07.18 03:32:21.832641 [ 38 ] {} <Information> brpc: [../contrib/incubator-brpc/src/brpc/server.cpp:1101]Server[DB::CnchWorkerServiceImpl] is serving on [::]:8124 with builtin service.

2023.07.18 03:32:21.832665 [ 38 ] {} <Information> Application: Listening rpc: [::]:8124
2023.07.18 03:32:21.832672 [ 38 ] {} <Information> Application: Ready for connections.
2023.07.18 03:32:22.000145 [ 251 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 327.10 MiB, peak 327.10 MiB, will set to 330.68 MiB (RSS), difference: 3.58 MiB
2023.07.18 03:32:22.220114 [ 151 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:22.722368 [ 154 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:22.834596 [ 65 ] {} <Trace> ResourceReporterTask: Send heartbeat to RM:  self: {byconity-vw-vw-default-0 <IP_ADDRESS>  rpc/8124 tcp/9000 exc/9410 exs/9510}
2023.07.18 03:32:23.000102 [ 251 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 330.68 MiB, peak 330.68 MiB, will set to 332.23 MiB (RSS), difference: 1.56 MiB
2023.07.18 03:32:23.224702 [ 189 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:23.726715 [ 189 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:23.835217 [ 78 ] {} <Trace> ResourceReporterTask: Send heartbeat to RM:  self: {byconity-vw-vw-default-0 <IP_ADDRESS>  rpc/8124 tcp/9000 exc/9410 exs/9510}
2023.07.18 03:32:24.229113 [ 194 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:24.731311 [ 151 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:24.835967 [ 78 ] {} <Trace> ResourceReporterTask: Send heartbeat to RM:  self: {byconity-vw-vw-default-0 <IP_ADDRESS>  rpc/8124 tcp/9000 exc/9410 exs/9510}
2023.07.18 03:32:25.233406 [ 154 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:25.735471 [ 189 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:25.836575 [ 78 ] {} <Trace> ResourceReporterTask: Send heartbeat to RM:  self: {byconity-vw-vw-default-0 <IP_ADDRESS>  rpc/8124 tcp/9000 exc/9410 exs/9510}
2023.07.18 03:32:26.237554 [ 194 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:26.739601 [ 151 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:26.837191 [ 78 ] {} <Trace> ResourceReporterTask: Send heartbeat to RM:  self: {byconity-vw-vw-default-0 <IP_ADDRESS>  rpc/8124 tcp/9000 exc/9410 exs/9510}
2023.07.18 03:32:27.241785 [ 154 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:27.524036 [ 246 ] {} <Trace> TCPHandlerFactory: TCP Request. Address: [::ffff:<IP_ADDRESS>]:57838
2023.07.18 03:32:27.524227 [ 246 ] {} <Debug> TCPHandler: Connected ClickHouse client version 21.8.0, revision: 54450, user: probe.
2023.07.18 03:32:27.524375 [ 246 ] {} <Trace> ContextAccess (probe): Settings: readonly=0, allow_ddl=true, allow_introspection_functions=false
2023.07.18 03:32:27.524406 [ 246 ] {} <Trace> ContextAccess (probe): List of all grants: GRANT SHOW, SELECT, INSERT, ALTER, CREATE, DROP, TRUNCATE, OPTIMIZE, KILL QUERY, MOVE PARTITION BETWEEN SHARDS, SYSTEM, dictGet, INTROSPECTION, SOURCES ON *.*
2023.07.18 03:32:27.524420 [ 246 ] {} <Trace> ContextAccess (probe): List of all grants including implicit: GRANT SHOW, SELECT, INSERT, ALTER, CREATE, DROP, TRUNCATE, OPTIMIZE, KILL QUERY, MOVE PARTITION BETWEEN SHARDS, SYSTEM, dictGet, INTROSPECTION, SOURCES ON *.*
2023.07.18 03:32:27.525093 [ 246 ] {9d02f288-c425-42dd-9437-543f69dcb1f4} <Debug> executeQuery: (from [::ffff:<IP_ADDRESS>]:57838, user: probe) select 1
2023.07.18 03:32:27.525375 [ 246 ] {9d02f288-c425-42dd-9437-543f69dcb1f4} <Trace> ContextAccess (probe): Access granted: SELECT(dummy) ON system.one
2023.07.18 03:32:27.525417 [ 246 ] {9d02f288-c425-42dd-9437-543f69dcb1f4} <Trace> InterpreterSelectQuery: query: SELECT 1
2023.07.18 03:32:27.525469 [ 246 ] {9d02f288-c425-42dd-9437-543f69dcb1f4} <Trace> InterpreterSelectQuery: FetchColumns -> Complete
2023.07.18 03:32:27.525545 [ 246 ] {9d02f288-c425-42dd-9437-543f69dcb1f4} <Debug> QueryPlan: Build pipeline takes:0
2023.07.18 03:32:27.525846 [ 258 ] {9d02f288-c425-42dd-9437-543f69dcb1f4} <Trace> ThreadStatus: node SourceWithProgress checkCpuTimeLimit thread cpu secs = 0.0, total cpu secs = 0.0, max = 0
2023.07.18 03:32:27.525931 [ 258 ] {9d02f288-c425-42dd-9437-543f69dcb1f4} <Trace> PipelineExecutor: Pipeline: digraph
{
  rankdir="LR";
  { node [shape = record]
    n139817535960344[label="{SourceFromSingleChunk(2 jobs) (Finished)}"];
    n139817535701400[label="{ExpressionTransform(1 jobs) (Finished)}"];
    n139817536021016[label="{LimitsCheckingTransform(1 jobs) (Finished)}"];
    n139817535849432[label="{NullSource(1 jobs) (Finished)}"];
    n139817535848792[label="{NullSource(1 jobs) (Finished)}"];
    n139817535836184[label="{LazyOutputFormat(2 jobs) (Finished)}"];
  }
  n139817535960344 -> n139817535701400;
  n139817535701400 -> n139817536021016;
  n139817536021016 -> n139817535836184;
  n139817535849432 -> n139817535836184;
  n139817535848792 -> n139817535836184;
}

2023.07.18 03:32:27.526103 [ 246 ] {9d02f288-c425-42dd-9437-543f69dcb1f4} <Information> executeQuery: Read 1 rows, 1.00 B in 0.000945693 sec., 1057 rows/sec., 1.03 KiB/sec.
2023.07.18 03:32:27.526229 [ 246 ] {9d02f288-c425-42dd-9437-543f69dcb1f4} <Debug> MemoryTracker: Peak memory usage (for query): 0.00 B.
2023.07.18 03:32:27.526305 [ 246 ] {} <Debug> TCPHandler: Processed in 0.001411054 sec.
2023.07.18 03:32:27.527889 [ 246 ] {} <Debug> TCPHandler: Done processing connection.
2023.07.18 03:32:27.743923 [ 189 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:27.837857 [ 78 ] {} <Trace> ResourceReporterTask: Send heartbeat to RM:  self: {byconity-vw-vw-default-0 <IP_ADDRESS>  rpc/8124 tcp/9000 exc/9410 exs/9510}
2023.07.18 03:32:28.246043 [ 194 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:28.748219 [ 151 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:28.838461 [ 78 ] {} <Trace> ResourceReporterTask: Send heartbeat to RM:  self: {byconity-vw-vw-default-0 <IP_ADDRESS>  rpc/8124 tcp/9000 exc/9410 exs/9510}
2023.07.18 03:32:29.250402 [ 154 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:29.752550 [ 189 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:29.839049 [ 78 ] {} <Trace> ResourceReporterTask: Send heartbeat to RM:  self: {byconity-vw-vw-default-0 <IP_ADDRESS>  rpc/8124 tcp/9000 exc/9410 exs/9510}
2023.07.18 03:32:30.254971 [ 154 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:30.757096 [ 194 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:30.839645 [ 78 ] {} <Trace> ResourceReporterTask: Send heartbeat to RM:  self: {byconity-vw-vw-default-0 <IP_ADDRESS>  rpc/8124 tcp/9000 exc/9410 exs/9510}
2023.07.18 03:32:31.259376 [ 151 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:31.761512 [ 189 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:31.813352 [ 65 ] {} <Debug> CnchSystemLogs: Initialised CNCH System log on try: 1
2023.07.18 03:32:31.817737 [ 65 ] {} <Debug> StorageFactory:  engine name CnchMergeTree
2023.07.18 03:32:31.820596 [ 65 ] {} <Debug> StorageFactory:  engine name CloudMergeTree
2023.07.18 03:32:31.821083 [ 65 ] {} <Information> CnchSystemLogs: Initializing CNCH System log on server for cnch_system.cnch_kafka_log successfully
2023.07.18 03:32:31.840238 [ 78 ] {} <Trace> ResourceReporterTask: Send heartbeat to RM:  self: {byconity-vw-vw-default-0 <IP_ADDRESS>  rpc/8124 tcp/9000 exc/9410 exs/9510}
2023.07.18 03:32:32.263679 [ 154 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:32.554845 [ 246 ] {} <Trace> TCPHandlerFactory: TCP Request. Address: [::ffff:<IP_ADDRESS>]:57840
2023.07.18 03:32:32.554951 [ 246 ] {} <Debug> TCPHandler: Connected ClickHouse client version 21.8.0, revision: 54450, user: probe.
2023.07.18 03:32:32.555630 [ 246 ] {25efb865-85e7-4986-9759-f1a4418d1440} <Debug> executeQuery: (from [::ffff:<IP_ADDRESS>]:57840, user: probe) select 1
2023.07.18 03:32:32.555853 [ 246 ] {25efb865-85e7-4986-9759-f1a4418d1440} <Trace> ContextAccess (probe): Access granted: SELECT(dummy) ON system.one
2023.07.18 03:32:32.555883 [ 246 ] {25efb865-85e7-4986-9759-f1a4418d1440} <Trace> InterpreterSelectQuery: query: SELECT 1
2023.07.18 03:32:32.555928 [ 246 ] {25efb865-85e7-4986-9759-f1a4418d1440} <Trace> InterpreterSelectQuery: FetchColumns -> Complete
2023.07.18 03:32:32.555992 [ 246 ] {25efb865-85e7-4986-9759-f1a4418d1440} <Debug> QueryPlan: Build pipeline takes:0
2023.07.18 03:32:32.556412 [ 299 ] {25efb865-85e7-4986-9759-f1a4418d1440} <Trace> ThreadStatus: node SourceWithProgress checkCpuTimeLimit thread cpu secs = 0.0, total cpu secs = 0.0, max = 0
2023.07.18 03:32:32.556503 [ 299 ] {25efb865-85e7-4986-9759-f1a4418d1440} <Trace> PipelineExecutor: Pipeline: digraph
{
  rankdir="LR";
  { node [shape = record]
    n139817535959704[label="{SourceFromSingleChunk(2 jobs) (Finished)}"];
    n139817535701400[label="{ExpressionTransform(1 jobs) (Finished)}"];
    n139817536020504[label="{LimitsCheckingTransform(1 jobs) (Finished)}"];
    n139817535848792[label="{NullSource(1 jobs) (Finished)}"];
    n139817535849752[label="{NullSource(1 jobs) (Finished)}"];
    n139817525248024[label="{LazyOutputFormat(2 jobs) (Finished)}"];
  }
  n139817535959704 -> n139817535701400;
  n139817535701400 -> n139817536020504;
  n139817536020504 -> n139817525248024;
  n139817535848792 -> n139817525248024;
  n139817535849752 -> n139817525248024;
}

2023.07.18 03:32:32.556712 [ 246 ] {25efb865-85e7-4986-9759-f1a4418d1440} <Information> executeQuery: Read 1 rows, 1.00 B in 0.001020733 sec., 979 rows/sec., 979.69 B/sec.
2023.07.18 03:32:32.556745 [ 246 ] {25efb865-85e7-4986-9759-f1a4418d1440} <Debug> MemoryTracker: Peak memory usage (for query): 0.00 B.
2023.07.18 03:32:32.556818 [ 246 ] {} <Debug> TCPHandler: Processed in 0.001381534 sec.
2023.07.18 03:32:32.557134 [ 246 ] {} <Debug> TCPHandler: Done processing connection.
2023.07.18 03:32:32.765794 [ 194 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:32.840809 [ 78 ] {} <Trace> ResourceReporterTask: Send heartbeat to RM:  self: {byconity-vw-vw-default-0 <IP_ADDRESS>  rpc/8124 tcp/9000 exc/9410 exs/9510}
2023.07.18 03:32:33.267960 [ 151 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:33.770115 [ 189 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:33.841386 [ 78 ] {} <Trace> ResourceReporterTask: Send heartbeat to RM:  self: {byconity-vw-vw-default-0 <IP_ADDRESS>  rpc/8124 tcp/9000 exc/9410 exs/9510}
2023.07.18 03:32:34.272373 [ 189 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:34.774563 [ 189 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:34.841966 [ 78 ] {} <Trace> ResourceReporterTask: Send heartbeat to RM:  self: {byconity-vw-vw-default-0 <IP_ADDRESS>  rpc/8124 tcp/9000 exc/9410 exs/9510}
2023.07.18 03:32:35.276706 [ 154 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:35.778852 [ 194 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:35.842588 [ 79 ] {} <Trace> ResourceReporterTask: Send heartbeat to RM:  self: {byconity-vw-vw-default-0 <IP_ADDRESS>  rpc/8124 tcp/9000 exc/9410 exs/9510}
2023.07.18 03:32:36.281006 [ 151 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:36.783126 [ 189 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:36.813285 [ 229 ] {} <Trace> SystemLog (system.query_log): Flushing system log, 4 entries to flush up to offset 4
2023.07.18 03:32:36.814144 [ 229 ] {} <Debug> SystemLog (system.query_log): Will use existing table system.query_log for QueryLog
2023.07.18 03:32:36.815399 [ 229 ] {} <Debug> DiskLocal: Reserving 1.00 MiB on disk `local_disk0`, having unreserved 48.82 GiB.
2023.07.18 03:32:36.815452 [ 229 ] {} <Debug> system.query_log (70999914-6aa1-4c0c-b33d-f9a75294d36f) (Writer): Writing temp part to store/709/70999914-6aa1-4c0c-b33d-f9a75294d36f/tmp_insert_20230718_20230718_1_1_0/...

2023.07.18 03:32:36.822878 [ 78 ] {} <Debug> DNSResolver: Updating DNS cache
2023.07.18 03:32:36.822907 [ 78 ] {} <Debug> DNSResolver: Updated DNS cache
2023.07.18 03:32:36.823976 [ 229 ] {} <Debug> system.query_log (70999914-6aa1-4c0c-b33d-f9a75294d36f): Renaming temporary part tmp_insert_20230718_20230718_1_1_0 to 20230718_20230718_1_1_0.
2023.07.18 03:32:36.824312 [ 229 ] {} <Trace> SystemLog (system.query_log): Flushed system log up to offset 4
2023.07.18 03:32:36.843276 [ 67 ] {} <Trace> ResourceReporterTask: Send heartbeat to RM:  self: {byconity-vw-vw-default-0 <IP_ADDRESS>  rpc/8124 tcp/9000 exc/9410 exs/9510}
2023.07.18 03:32:37.000181 [ 251 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 344.00 MiB, peak 609.66 MiB, will set to 339.61 MiB (RSS), difference: -4.39 MiB
2023.07.18 03:32:37.285331 [ 154 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:37.598527 [ 246 ] {} <Trace> TCPHandlerFactory: TCP Request. Address: [::ffff:<IP_ADDRESS>]:57842
2023.07.18 03:32:37.598673 [ 246 ] {} <Debug> TCPHandler: Connected ClickHouse client version 21.8.0, revision: 54450, user: probe.
2023.07.18 03:32:37.599331 [ 246 ] {91f857a1-9292-4f03-863a-0156fcea50c2} <Debug> executeQuery: (from [::ffff:<IP_ADDRESS>]:57842, user: probe) select 1
2023.07.18 03:32:37.599532 [ 246 ] {91f857a1-9292-4f03-863a-0156fcea50c2} <Trace> ContextAccess (probe): Access granted: SELECT(dummy) ON system.one
2023.07.18 03:32:37.599562 [ 246 ] {91f857a1-9292-4f03-863a-0156fcea50c2} <Trace> InterpreterSelectQuery: query: SELECT 1
2023.07.18 03:32:37.599608 [ 246 ] {91f857a1-9292-4f03-863a-0156fcea50c2} <Trace> InterpreterSelectQuery: FetchColumns -> Complete
2023.07.18 03:32:37.599701 [ 246 ] {91f857a1-9292-4f03-863a-0156fcea50c2} <Debug> QueryPlan: Build pipeline takes:0
2023.07.18 03:32:37.599938 [ 299 ] {91f857a1-9292-4f03-863a-0156fcea50c2} <Trace> ThreadStatus: node SourceWithProgress checkCpuTimeLimit thread cpu secs = 0.0, total cpu secs = 0.0, max = 0
2023.07.18 03:32:37.600002 [ 299 ] {91f857a1-9292-4f03-863a-0156fcea50c2} <Trace> PipelineExecutor: Pipeline: digraph
{
  rankdir="LR";
  { node [shape = record]
    n139817535959064[label="{SourceFromSingleChunk(2 jobs) (Finished)}"];
    n139817535701400[label="{ExpressionTransform(1 jobs) (Finished)}"];
    n139817536021016[label="{LimitsCheckingTransform(1 jobs) (Finished)}"];
    n139817535849752[label="{NullSource(1 jobs) (Finished)}"];
    n139817535849112[label="{NullSource(1 jobs) (Finished)}"];
    n139819307457304[label="{LazyOutputFormat(2 jobs) (Finished)}"];
  }
  n139817535959064 -> n139817535701400;
  n139817535701400 -> n139817536021016;
  n139817536021016 -> n139819307457304;
  n139817535849752 -> n139819307457304;
  n139817535849112 -> n139819307457304;
}

2023.07.18 03:32:37.600169 [ 246 ] {91f857a1-9292-4f03-863a-0156fcea50c2} <Information> executeQuery: Read 1 rows, 1.00 B in 0.000779564 sec., 1282 rows/sec., 1.25 KiB/sec.
2023.07.18 03:32:37.600198 [ 246 ] {91f857a1-9292-4f03-863a-0156fcea50c2} <Debug> MemoryTracker: Peak memory usage (for query): 0.00 B.
2023.07.18 03:32:37.600262 [ 246 ] {} <Debug> TCPHandler: Processed in 0.00111641 sec.
2023.07.18 03:32:37.600858 [ 246 ] {} <Debug> TCPHandler: Done processing connection.
2023.07.18 03:32:37.787436 [ 194 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:37.843937 [ 67 ] {} <Trace> ResourceReporterTask: Send heartbeat to RM:  self: {byconity-vw-vw-default-0 <IP_ADDRESS>  rpc/8124 tcp/9000 exc/9410 exs/9510}
2023.07.18 03:32:38.289695 [ 151 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:38.791880 [ 189 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:38.844545 [ 67 ] {} <Trace> ResourceReporterTask: Send heartbeat to RM:  self: {byconity-vw-vw-default-0 <IP_ADDRESS>  rpc/8124 tcp/9000 exc/9410 exs/9510}
2023.07.18 03:32:39.294080 [ 154 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:39.796324 [ 194 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:39.845156 [ 67 ] {} <Trace> ResourceReporterTask: Send heartbeat to RM:  self: {byconity-vw-vw-default-0 <IP_ADDRESS>  rpc/8124 tcp/9000 exc/9410 exs/9510}
2023.07.18 03:32:40.298488 [ 151 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.
2023.07.18 03:32:40.800560 [ 189 ] {} <Trace> CnchTopologyMaster: Cannot fetch topology from remote.