refactor(dynamic_scaling_config): refactor dynamic scaling config

Description

This allows to:

move some configuration knobs (enable scale up/down) onto the config to allow switching node creation on / off during trial runs in kas-installer or dev purposes like https://issues.redhat.com/browse/MGDSTRM-3363 or during migration from manual to auto,to allow for the initial reconciliation of dynamic capacity info without causing unintented creation of new clusters
allows to customize the machine type per cloud provider and workload type. This prepares room for https://issues.redhat.com/browse/MGDSTRM-9802 where the Kafka machine pool will have a different machine type from the cluster wide workload machine pool (the default machine pool which might still be). Now the two types of pools can be configured separetely.
move the openshift version knob onto the dynamic scaling configuration so that it can be controlled along side other dynamic scaling configuration knobs

Verification Steps

Unit & integration tests (both against emulate and real environment) should pass
Checklist (Definition of Done)

[ ] All acceptance criteria specified in JIRA have been completed
[ ] Unit and integration tests added that prove the fix is effective or the feature works (tested against emulated and non-emulated OCM environment)
[ ] Documentation added for the feature
[ ] CI and all relevant tests are passing
[ ] Code Review completed
[ ] Verified independently by reviewer
[ ] All PR comments are resolved either by addressing them or creating follow up tasks
[ ] Required metrics/dashboards/alerts have been added (or PR created).
[ ] Required Standard Operating Procedure (SOP) is added.
[ ] JIRA has been created for changes required on the client side

The new config now has this shape
https://github.com/bf2fc6cc711aee1a0c2a/kas-fleet-manager/blob/e1a0a4884b63f8cf16e5664dc930f1fbc3392d2b/config/dynamic-scaling-configuration.yaml#L1-L30

Unit & integration tests (both against emulate and real environment) should pass

I am running integration tests against a real OCM environment. Will report back once they finish.
It is the reason why this PR is still in draft, but it can be reviewed in its current state.

Unit & integration tests (both against emulate and real environment) should pass

I am running integration tests against a real OCM environment. Will report back once they finish. It is the reason why this PR is still in draft, but it can be reviewed in its current state.

The test have finished.
OCM_ENV=development TEST_SUMMARY_FORMAT=standard-verbose  make test/integration/kafka
gotestsum --junitfile data/results/kas-fleet-manager-integration-tests.xml --format standard-verbose -- -p 1 -ldflags -s -v -timeout 5h -count=1 \
			./internal/kafka/test/integration/... \

=== RUN   TestAdminKafka_Get
=== RUN   TestAdminKafka_Get/should_fail_authentication_when_there_is_no_role_defined_in_the_request
=== RUN   TestAdminKafka_Get/should_fail_when_the_role_defined_in_the_request_is_not_any_of_read,_write_or_full
=== RUN   TestAdminKafka_Get/should_success_when_the_role_defined_in_the_request_is_kas-fleet-manager-admin-read
=== RUN   TestAdminKafka_Get/should_success_when_the_role_defined_in_the_request_is_kas-fleet-manager-admin-write
=== RUN   TestAdminKafka_Get/should_success_when_the_role_defined_in_the_request_is_kas-fleet-manager-admin-full
=== RUN   TestAdminKafka_Get/should_fail_when_the_requested_kafka_does_not_exist
=== RUN   TestAdminKafka_Get/should_fail_when_the_request_does_not_contain_a_valid_issuer
--- PASS: TestAdminKafka_Get (3.13s)
    --- PASS: TestAdminKafka_Get/should_fail_authentication_when_there_is_no_role_defined_in_the_request (0.78s)
    --- PASS: TestAdminKafka_Get/should_fail_when_the_role_defined_in_the_request_is_not_any_of_read,_write_or_full (0.02s)
    --- PASS: TestAdminKafka_Get/should_success_when_the_role_defined_in_the_request_is_kas-fleet-manager-admin-read (0.99s)
    --- PASS: TestAdminKafka_Get/should_success_when_the_role_defined_in_the_request_is_kas-fleet-manager-admin-write (0.13s)
    --- PASS: TestAdminKafka_Get/should_success_when_the_role_defined_in_the_request_is_kas-fleet-manager-admin-full (0.13s)
    --- PASS: TestAdminKafka_Get/should_fail_when_the_requested_kafka_does_not_exist (0.01s)
    --- PASS: TestAdminKafka_Get/should_fail_when_the_request_does_not_contain_a_valid_issuer (0.01s)
=== RUN   TestAdminKafka_Delete
=== RUN   TestAdminKafka_Delete/should_fail_authentication_when_there_is_no_role_defined_in_the_request
=== RUN   TestAdminKafka_Delete/should_fail_when_the_role_defined_in_the_request_is_not_kas-fleet-manager-admin-full
=== RUN   TestAdminKafka_Delete/should_success_when_the_role_defined_in_the_request_is_kas-fleet-manager-admin-full
--- PASS: TestAdminKafka_Delete (1.96s)
    --- PASS: TestAdminKafka_Delete/should_fail_authentication_when_there_is_no_role_defined_in_the_request (0.91s)
    --- PASS: TestAdminKafka_Delete/should_fail_when_the_role_defined_in_the_request_is_not_kas-fleet-manager-admin-full (0.01s)
    --- PASS: TestAdminKafka_Delete/should_success_when_the_role_defined_in_the_request_is_kas-fleet-manager-admin-full (0.03s)
=== RUN   TestAdminKafka_List
=== RUN   TestAdminKafka_List/should_fail_authentication_when_there_is_no_role_defined_in_the_request
=== RUN   TestAdminKafka_List/should_success_when_the_role_defined_in_the_request_is_kas-fleet-manager-admin-full
=== RUN   TestAdminKafka_List/should_success_when_the_role_defined_in_the_request_is_kas-fleet-manager-admin-write
=== RUN   TestAdminKafka_List/should_success_when_the_role_defined_in_the_request_is_kas-fleet-manager-admin-read
--- PASS: TestAdminKafka_List (2.89s)
    --- PASS: TestAdminKafka_List/should_fail_authentication_when_there_is_no_role_defined_in_the_request (0.68s)
    --- PASS: TestAdminKafka_List/should_success_when_the_role_defined_in_the_request_is_kas-fleet-manager-admin-full (0.87s)
    --- PASS: TestAdminKafka_List/should_success_when_the_role_defined_in_the_request_is_kas-fleet-manager-admin-write (0.38s)
    --- PASS: TestAdminKafka_List/should_success_when_the_role_defined_in_the_request_is_kas-fleet-manager-admin-read (0.33s)
=== RUN   TestAdminKafka_Update
=== RUN   TestAdminKafka_Update/should_fail_when_kafkaUpdateRequest_is_empty
=== RUN   TestAdminKafka_Update/should_fail_when_kafkaUpdateRequest_request_params_contain_only_strings_with_whitespaces
=== RUN   TestAdminKafka_Update/should_fail_when_the_requested_kafka_does_not_exist
=== RUN   TestAdminKafka_Update/should_succeed_when_upgrading_all_possible_values
=== RUN   TestAdminKafka_Update/should_fail_authentication_when_there_is_no_role_defined_in_the_request
=== RUN   TestAdminKafka_Update/should_fail_when_the_role_defined_in_the_request_is_not_any_of_read,_write_or_full
=== RUN   TestAdminKafka_Update/should_fail_when_the_role_defined_in_the_request_is_kas-fleet-manager-admin-read
=== RUN   TestAdminKafka_Update/should_succeed_when_the_role_defined_in_the_request_is_kas-fleet-manager-admin-write
=== RUN   TestAdminKafka_Update/should_succeed_when_the_role_defined_in_the_request_is_kas-fleet-manager-admin-full
=== RUN   TestAdminKafka_Update/should_fail_when_the_request_does_not_contain_a_valid_issuer
=== RUN   TestAdminKafka_Update/should_fail_when_downgrading_ibp_version
=== RUN   TestAdminKafka_Update/should_fail_when_upgrading_ibp_version_to_version_higher_than_kafka_version
=== RUN   TestAdminKafka_Update/should_fail_when_upgrading_ibp_version_when_already_upgrade_in_progress
=== RUN   TestAdminKafka_Update/should_succeed_when_upgrading_ibp_version_to_lower_than_kafka_version
=== RUN   TestAdminKafka_Update/should_fail_when_downgrading_to_lower_minor_kafka_version
=== RUN   TestAdminKafka_Update/should_fail_when_upgrading_to_higher_minor_kafka_version_when_not_in_status
=== RUN   TestAdminKafka_Update/should_fail_when_downgrading_to_lower_patch_kafka_version_smaller_than_ibp_version
=== RUN   TestAdminKafka_Update/should_fail_when_downgrading_to_lower_major_kafka_version
=== RUN   TestAdminKafka_Update/should_fail_when_upgrading_kafka_version_when_already_upgrade_in_progress
=== RUN   TestAdminKafka_Update/should_succeed_when_downgrading_to_lower_patch_kafka_version
=== RUN   TestAdminKafka_Update/should_succeed_when_upgrading_to_higher_minor_kafka_version_when_in_status
=== RUN   TestAdminKafka_Update/should_fail_when_upgrading_strimzi_version_when_already_upgrade_in_progress
=== RUN   TestAdminKafka_Update/should_fail_when_upgrading_strimzi_version_to_a_version_not_in_the_status
=== RUN   TestAdminKafka_Update/should_fail_when_upgrading_simultaneously_kafka,_ibp_and_strimzi_version_when_any_of_those_not_in_status
=== RUN   TestAdminKafka_Update/should_fail_when_upgrading_strimzi_version_to_a_version_that_is_not_ready
=== RUN   TestAdminKafka_Update/should_succeed_when_upgrading_simultaneously_kafka,_ibp_and_strimzi_version
=== RUN   TestAdminKafka_Update/should_succeed_when_attempting_to_update_to_the_same_storage_size_using_kafka_storage_size_field
=== RUN   TestAdminKafka_Update/should_fail_when_attempting_to_update_to_smaller_storage_size_using_kafka_storage_size_field
=== RUN   TestAdminKafka_Update/should_fail_when_attempting_to_update_to_smaller_storage_size_in_different_format_using_kafka_storage_size_field
=== RUN   TestAdminKafka_Update/should_fail_when_attempting_to_update_to_smaller_storage_size_in_the_wrong_format_using_kafka_storage_size_field
=== RUN   TestAdminKafka_Update/should_succeed_when_updating_to_bigger_storage_size_using_kafka_storage_size_field
=== RUN   TestAdminKafka_Update/should_fail_when_attempting_to_update_storage_size_to_a_random_string_using_kafka_storage_size_field
=== RUN   TestAdminKafka_Update/should_succeed_when_attempting_to_update_to_the_same_storage_size_using_max_data_retention_size_field
=== RUN   TestAdminKafka_Update/should_fail_when_attempting_to_upgrade_to_smaller_storage_size_using_max_data_retention_size_field
=== RUN   TestAdminKafka_Update/should_fail_when_attempting_to_update_to_smaller_storage_size_in_different_format_using_kafka_storage_size_field#01
=== RUN   TestAdminKafka_Update/should_fail_when_attempting_to_update_storage_using_an_invalid_format_using_max_data_retention_size_field
=== RUN   TestAdminKafka_Update/should_succeed_when_upgrading_to_bigger_storage_size_using_max_data_retention_size_field
=== RUN   TestAdminKafka_Update/should_fail_when_attempting_to_update_storage_size_to_a_random_string_using_max_data_retention_size_field
=== RUN   TestAdminKafka_Update/should_fail_when_attempting_to_update_storage_when_current_storage_size_not_set
=== RUN   TestAdminKafka_Update/should_fail_when_attempting_to_update_storage_when_current_storage_size_is_set_to_some_incorrect_value
=== RUN   TestAdminKafka_Update/should_use_max_data_retention_size_over_kafka_storage_size_if_both_are_specified
--- PASS: TestAdminKafka_Update (4.45s)
    --- PASS: TestAdminKafka_Update/should_fail_when_kafkaUpdateRequest_is_empty (0.71s)
    --- PASS: TestAdminKafka_Update/should_fail_when_kafkaUpdateRequest_request_params_contain_only_strings_with_whitespaces (0.01s)
    --- PASS: TestAdminKafka_Update/should_fail_when_the_requested_kafka_does_not_exist (0.01s)
    --- PASS: TestAdminKafka_Update/should_succeed_when_upgrading_all_possible_values (0.61s)
    --- PASS: TestAdminKafka_Update/should_fail_authentication_when_there_is_no_role_defined_in_the_request (0.01s)
    --- PASS: TestAdminKafka_Update/should_fail_when_the_role_defined_in_the_request_is_not_any_of_read,_write_or_full (0.00s)
    --- PASS: TestAdminKafka_Update/should_fail_when_the_role_defined_in_the_request_is_kas-fleet-manager-admin-read (0.00s)
    --- PASS: TestAdminKafka_Update/should_succeed_when_the_role_defined_in_the_request_is_kas-fleet-manager-admin-write (0.12s)
    --- PASS: TestAdminKafka_Update/should_succeed_when_the_role_defined_in_the_request_is_kas-fleet-manager-admin-full (0.11s)
    --- PASS: TestAdminKafka_Update/should_fail_when_the_request_does_not_contain_a_valid_issuer (0.00s)
    --- PASS: TestAdminKafka_Update/should_fail_when_downgrading_ibp_version (0.00s)
    --- PASS: TestAdminKafka_Update/should_fail_when_upgrading_ibp_version_to_version_higher_than_kafka_version (0.00s)
    --- PASS: TestAdminKafka_Update/should_fail_when_upgrading_ibp_version_when_already_upgrade_in_progress (0.00s)
    --- PASS: TestAdminKafka_Update/should_succeed_when_upgrading_ibp_version_to_lower_than_kafka_version (0.14s)
    --- PASS: TestAdminKafka_Update/should_fail_when_downgrading_to_lower_minor_kafka_version (0.00s)
    --- PASS: TestAdminKafka_Update/should_fail_when_upgrading_to_higher_minor_kafka_version_when_not_in_status (0.00s)
    --- PASS: TestAdminKafka_Update/should_fail_when_downgrading_to_lower_patch_kafka_version_smaller_than_ibp_version (0.00s)
    --- PASS: TestAdminKafka_Update/should_fail_when_downgrading_to_lower_major_kafka_version (0.00s)
    --- PASS: TestAdminKafka_Update/should_fail_when_upgrading_kafka_version_when_already_upgrade_in_progress (0.00s)
    --- PASS: TestAdminKafka_Update/should_succeed_when_downgrading_to_lower_patch_kafka_version (0.14s)
    --- PASS: TestAdminKafka_Update/should_succeed_when_upgrading_to_higher_minor_kafka_version_when_in_status (0.16s)
    --- PASS: TestAdminKafka_Update/should_fail_when_upgrading_strimzi_version_when_already_upgrade_in_progress (0.01s)
    --- PASS: TestAdminKafka_Update/should_fail_when_upgrading_strimzi_version_to_a_version_not_in_the_status (0.01s)
    --- PASS: TestAdminKafka_Update/should_fail_when_upgrading_simultaneously_kafka,_ibp_and_strimzi_version_when_any_of_those_not_in_status (0.02s)
    --- PASS: TestAdminKafka_Update/should_fail_when_upgrading_strimzi_version_to_a_version_that_is_not_ready (0.01s)
    --- PASS: TestAdminKafka_Update/should_succeed_when_upgrading_simultaneously_kafka,_ibp_and_strimzi_version (0.18s)
    --- PASS: TestAdminKafka_Update/should_succeed_when_attempting_to_update_to_the_same_storage_size_using_kafka_storage_size_field (0.20s)
    --- PASS: TestAdminKafka_Update/should_fail_when_attempting_to_update_to_smaller_storage_size_using_kafka_storage_size_field (0.01s)
    --- PASS: TestAdminKafka_Update/should_fail_when_attempting_to_update_to_smaller_storage_size_in_different_format_using_kafka_storage_size_field (0.00s)
    --- PASS: TestAdminKafka_Update/should_fail_when_attempting_to_update_to_smaller_storage_size_in_the_wrong_format_using_kafka_storage_size_field (0.00s)
    --- PASS: TestAdminKafka_Update/should_succeed_when_updating_to_bigger_storage_size_using_kafka_storage_size_field (0.19s)
    --- PASS: TestAdminKafka_Update/should_fail_when_attempting_to_update_storage_size_to_a_random_string_using_kafka_storage_size_field (0.01s)
    --- PASS: TestAdminKafka_Update/should_succeed_when_attempting_to_update_to_the_same_storage_size_using_max_data_retention_size_field (0.20s)
    --- PASS: TestAdminKafka_Update/should_fail_when_attempting_to_upgrade_to_smaller_storage_size_using_max_data_retention_size_field (0.00s)
    --- PASS: TestAdminKafka_Update/should_fail_when_attempting_to_update_to_smaller_storage_size_in_different_format_using_kafka_storage_size_field#01 (0.00s)
    --- PASS: TestAdminKafka_Update/should_fail_when_attempting_to_update_storage_using_an_invalid_format_using_max_data_retention_size_field (0.00s)
    --- PASS: TestAdminKafka_Update/should_succeed_when_upgrading_to_bigger_storage_size_using_max_data_retention_size_field (0.19s)
    --- PASS: TestAdminKafka_Update/should_fail_when_attempting_to_update_storage_size_to_a_random_string_using_max_data_retention_size_field (0.01s)
    --- PASS: TestAdminKafka_Update/should_fail_when_attempting_to_update_storage_when_current_storage_size_not_set (0.01s)
    --- PASS: TestAdminKafka_Update/should_fail_when_attempting_to_update_storage_when_current_storage_size_is_set_to_some_incorrect_value (0.01s)
    --- PASS: TestAdminKafka_Update/should_use_max_data_retention_size_over_kafka_storage_size_if_both_are_specified (0.17s)
=== RUN   TestAuth_success
--- PASS: TestAuth_success (1.38s)
=== RUN   TestAuthSucess_publicUrls
--- PASS: TestAuthSucess_publicUrls (1.29s)
=== RUN   TestAuthSuccess_usingSSORHToken
--- PASS: TestAuthSuccess_usingSSORHToken (1.46s)
=== RUN   TestAuthFailure_withoutToken
--- PASS: TestAuthFailure_withoutToken (1.52s)
=== RUN   TestAuthFailure_invalidTokenWithInvalidTyp
--- PASS: TestAuthFailure_invalidTokenWithInvalidTyp (1.30s)
=== RUN   TestAuthFailure_ExpiredToken
--- PASS: TestAuthFailure_ExpiredToken (1.24s)
=== RUN   TestAuthFailure_invalidTokenMissingIat
--- PASS: TestAuthFailure_invalidTokenMissingIat (1.32s)
=== RUN   TestAuthFailure_invalidTokenMissingAlgHeader
--- PASS: TestAuthFailure_invalidTokenMissingAlgHeader (1.77s)
=== RUN   TestAuthFailure_invalidTokenUnsigned
--- PASS: TestAuthFailure_invalidTokenUnsigned (1.40s)
=== RUN   TestAuthFailure_usingMasSsoTokenOnKafkasGet
--- PASS: TestAuthFailure_usingMasSsoTokenOnKafkasGet (1.43s)
=== RUN   TestCloudProviderRegions
E0928 10:42:47.694434 1479062 logger.go:210] failed to reconcile ready cluster ccq0hqnfnec9rckq2vv0: failed to reconcile ready cluster resources ccq0hqnfnec9rckq2vv0 : failed to apply resources for cluster ccq0hqnfnec9rckq2vv0: KAFKAS-MGMT-9: failed to apply resources ext-managedservice-cluster-mgr
 caused by: failed to create syncset for cluster ccq0hqnfnec9rckq2vv0: KAFKAS-MGMT-7: ocm client failed to create syncset: status is 404, identifier is '404', code is 'CLUSTERS-MGMT-404' and operation identifier is '50cb6866-e5a4-4ed7-9c42-e69636a47cf4': Cluster 'ccq0hqnfnec9rckq2vv0' not found
--- PASS: TestCloudProviderRegions (3.03s)
=== RUN   TestCachedCloudProviderRegions
E0928 10:42:50.339379 1479062 logger.go:210] failed to reconcile ready cluster ccq0hqnfnec9rckq2vv0: failed to reconcile ready cluster resources ccq0hqnfnec9rckq2vv0 : failed to apply resources for cluster ccq0hqnfnec9rckq2vv0: KAFKAS-MGMT-9: failed to apply resources ext-managedservice-cluster-mgr
 caused by: failed to create syncset for cluster ccq0hqnfnec9rckq2vv0: KAFKAS-MGMT-7: ocm client failed to create syncset: status is 404, identifier is '404', code is 'CLUSTERS-MGMT-404' and operation identifier is 'e4b5ce6d-584a-4917-8556-9d54849fcecc': Cluster 'ccq0hqnfnec9rckq2vv0' not found
--- PASS: TestCachedCloudProviderRegions (2.94s)
=== RUN   TestListCloudProviders
E0928 10:42:53.289317 1479062 logger.go:210] failed to reconcile ready cluster ccq0hqnfnec9rckq2vv0: failed to reconcile ready cluster resources ccq0hqnfnec9rckq2vv0 : failed to apply resources for cluster ccq0hqnfnec9rckq2vv0: KAFKAS-MGMT-9: failed to apply resources ext-managedservice-cluster-mgr
 caused by: failed to create syncset for cluster ccq0hqnfnec9rckq2vv0: KAFKAS-MGMT-7: ocm client failed to create syncset: status is 404, identifier is '404', code is 'CLUSTERS-MGMT-404' and operation identifier is '0bb0d9c5-c961-41b2-b226-1e339d05f23f': Cluster 'ccq0hqnfnec9rckq2vv0' not found
--- PASS: TestListCloudProviders (2.48s)
=== RUN   TestListCloudProviderRegions
E0928 10:42:53.814396 1479062 logger.go:210] failed to get instance limit for af-east-1 on gcp and instance type standard: KAFKAS-MGMT-30: cloud provider 'gcp' is unsupported
DUMPING 'clusters' with filter 'cluster_id = 'ccq0i3ffnec9rckq39bg''
{{ EMPTY }}
1/7200 [0s] - Waiting for cluster 'ccq0i3ffnec9rckq39bg' to reach status 'ready'
2/7200 [1s] - Polling finished
DUMPING 'clusters' with filter 'cluster_id = 'ccq0i3ffnec9rckq39c0''
+----------------------+--------+--------------------------------+
|      CLUSTER ID      | STATUS |           UPDATED AT           |
+----------------------+--------+--------------------------------+
| ccq0i3ffnec9rckq39c0 | ready  | 2022-09-28 10:42:54.562932     |
|                      |        | +0200 CEST                     |
+----------------------+--------+--------------------------------+
1/7200 [0s] - Waiting for cluster 'ccq0i3ffnec9rckq39c0' to reach status 'ready'
1/7200 [0s] - Polling finished
E0928 10:42:55.100838 1479062 logger.go:210] failed to reconcile ready cluster ccq0i3ffnec9rckq39bg: failed to reconcile ready cluster resources ccq0i3ffnec9rckq39bg : failed to apply resources for cluster ccq0i3ffnec9rckq39bg: KAFKAS-MGMT-9: failed to apply resources ext-managedservice-cluster-mgr
 caused by: failed to create syncset for cluster ccq0i3ffnec9rckq39bg: KAFKAS-MGMT-7: ocm client failed to create syncset: status is 404, identifier is '404', code is 'CLUSTERS-MGMT-404' and operation identifier is 'b6f95f24-7000-4308-942f-c892b746e6b8': Cluster 'ccq0i3ffnec9rckq39bg' not found
E0928 10:42:55.102030 1479062 logger.go:210] failed to reconcile ready cluster ccq0i3ffnec9rckq39c0: failed to reconcile ready cluster resources ccq0i3ffnec9rckq39c0 : failed to apply resources for cluster ccq0i3ffnec9rckq39c0: KAFKAS-MGMT-9: failed to apply resources ext-managedservice-cluster-mgr
 caused by: failed to create syncset for cluster ccq0i3ffnec9rckq39c0: KAFKAS-MGMT-7: ocm client failed to create syncset: status is 404, identifier is '404', code is 'CLUSTERS-MGMT-404' and operation identifier is 'f77222d4-f8ba-47ff-b7b9-3e29d80e29ae': Cluster 'ccq0i3ffnec9rckq39c0' not found
--- PASS: TestListCloudProviderRegions (7.53s)
=== RUN   TestClusterCreate_InvalidAwsCredentials
--- PASS: TestClusterCreate_InvalidAwsCredentials (1.62s)
=== RUN   TestClusterCapacityUsedMetric
1/60 [0s] - Waiting for metric 'cluster_status_capacity_used' to contain values '[1 standard some-cluster-id us-west-1 aws]
E0928 10:43:03.571011 1479062 logger.go:210] failed to list provisioned clusters: KAFKAS-MGMT-9: failed to query by status
 caused by: sql: Scan error on column index 15, name "provider_spec": failed to unmarshal json value: "{}"
2/60 [1s] - Waiting for metric 'cluster_status_capacity_used' to contain values '[1 standard some-cluster-id us-west-1 aws]
2/60 [1s] - Polling finished
1/60 [0s] - Waiting for metric 'cluster_status_capacity_used' to contain values '[0 standard some-cluster-id us-west-1 aws]
E0928 10:43:04.271045 1479062 logger.go:210] failed to set kafka per cluster count metrics: KAFKAS-MGMT-9: failed to find cluster with id: some-cluster-id
 caused by: sql: Scan error on column index 15, name "provider_spec": failed to unmarshal json value: "{}"
E0928 10:43:04.271909 1479062 logger.go:210] failed to list provisioned clusters: KAFKAS-MGMT-9: failed to query by status
 caused by: sql: Scan error on column index 15, name "provider_spec": failed to unmarshal json value: "{}"
2/60 [1s] - Waiting for metric 'cluster_status_capacity_used' to contain values '[0 standard some-cluster-id us-west-1 aws]
2/60 [1s] - Polling finished
E0928 10:43:05.160855 1479062 logger.go:210] failed to list provisioned clusters: KAFKAS-MGMT-9: failed to query by status
 caused by: sql: Scan error on column index 15, name "provider_spec": failed to unmarshal json value: "{}"
--- PASS: TestClusterCapacityUsedMetric (2.79s)
=== RUN   TestOCMClusterResourceQuotaMetrics
--- SKIP: TestOCMClusterResourceQuotaMetrics (1.68s)
=== RUN   TestOCMClusterResourceQuotaMetricsNoQuotaAvailable
--- SKIP: TestOCMClusterResourceQuotaMetricsNoQuotaAvailable (1.79s)
=== RUN   TestClusterManager_SuccessfulReconcile
    kas-fleetshard-sync.go:212: Starting mock fleetshard sync
1/120 [0s] - Waiting for an ID to be assigned to the cluster (&{Provider:aws Region:us-east-1 MultiAZ:true Status: SupportedInstanceType:standard})
2/120 [1s] - Waiting for an ID to be assigned to the cluster (&{Provider:aws Region:us-east-1 MultiAZ:true Status: SupportedInstanceType:standard})
4/120 [3s] - Waiting for an ID to be assigned to the cluster (&{Provider:aws Region:us-east-1 MultiAZ:true Status: SupportedInstanceType:standard})
4/120 [3s] - Polling finished
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+----------------------+--------------------------------+
|            CLUSTER ID            |        STATUS        |           UPDATED AT           |
+----------------------------------+----------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioning | 2022-09-28 10:43:12.246033     |
|                                  |                      | +0200 CEST                     |
+----------------------------------+----------------------+--------------------------------+
1/7200 [0s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'cluster_provisioning'
1/7200 [0s] - Polling finished
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+----------------------+--------------------------------+
|            CLUSTER ID            |        STATUS        |           UPDATED AT           |
+----------------------------------+----------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioning | 2022-09-28 10:43:12.246033     |
|                                  |                      | +0200 CEST                     |
+----------------------------------+----------------------+--------------------------------+
1/7200 [0s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready'
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+----------------------+--------------------------------+
|            CLUSTER ID            |        STATUS        |           UPDATED AT           |
+----------------------------------+----------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioning | 2022-09-28 10:45:09.573421     |
|                                  |                      | +0200 CEST                     |
+----------------------------------+----------------------+--------------------------------+
144/7200 [2m23s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready' (current status: 'cluster_provisioning')
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+----------------------+--------------------------------+
|            CLUSTER ID            |        STATUS        |           UPDATED AT           |
+----------------------------------+----------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioning | 2022-09-28 10:47:39.645578     |
|                                  |                      | +0200 CEST                     |
+----------------------------------+----------------------+--------------------------------+
288/7200 [4m47s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready' (current status: 'cluster_provisioning')
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+----------------------+--------------------------------+
|            CLUSTER ID            |        STATUS        |           UPDATED AT           |
+----------------------------------+----------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioning | 2022-09-28 10:50:09.579768     |
|                                  |                      | +0200 CEST                     |
+----------------------------------+----------------------+--------------------------------+
432/7200 [7m11s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready' (current status: 'cluster_provisioning')
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+----------------------+--------------------------------+
|            CLUSTER ID            |        STATUS        |           UPDATED AT           |
+----------------------------------+----------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioning | 2022-09-28 10:52:39.66876      |
|                                  |                      | +0200 CEST                     |
+----------------------------------+----------------------+--------------------------------+
576/7200 [9m35s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready' (current status: 'cluster_provisioning')
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+----------------------+--------------------------------+
|            CLUSTER ID            |        STATUS        |           UPDATED AT           |
+----------------------------------+----------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioning | 2022-09-28 10:55:09.679363     |
|                                  |                      | +0200 CEST                     |
+----------------------------------+----------------------+--------------------------------+
720/7200 [11m59s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready' (current status: 'cluster_provisioning')
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+----------------------+--------------------------------+
|            CLUSTER ID            |        STATUS        |           UPDATED AT           |
+----------------------------------+----------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioning | 2022-09-28 10:57:10.008771     |
|                                  |                      | +0200 CEST                     |
+----------------------------------+----------------------+--------------------------------+
864/7200 [14m23s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready' (current status: 'cluster_provisioning')
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+----------------------+--------------------------------+
|            CLUSTER ID            |        STATUS        |           UPDATED AT           |
+----------------------------------+----------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioning | 2022-09-28 10:59:39.626824     |
|                                  |                      | +0200 CEST                     |
+----------------------------------+----------------------+--------------------------------+
1008/7200 [16m47s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready' (current status: 'cluster_provisioning')
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+----------------------+--------------------------------+
|            CLUSTER ID            |        STATUS        |           UPDATED AT           |
+----------------------------------+----------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioning | 2022-09-28 11:02:09.587636     |
|                                  |                      | +0200 CEST                     |
+----------------------------------+----------------------+--------------------------------+
1152/7200 [19m11s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready' (current status: 'cluster_provisioning')
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+----------------------+--------------------------------+
|            CLUSTER ID            |        STATUS        |           UPDATED AT           |
+----------------------------------+----------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioning | 2022-09-28 11:04:39.587181     |
|                                  |                      | +0200 CEST                     |
+----------------------------------+----------------------+--------------------------------+
1296/7200 [21m35s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready' (current status: 'cluster_provisioning')
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+----------------------+--------------------------------+
|            CLUSTER ID            |        STATUS        |           UPDATED AT           |
+----------------------------------+----------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioning | 2022-09-28 11:07:09.58971      |
|                                  |                      | +0200 CEST                     |
+----------------------------------+----------------------+--------------------------------+
1440/7200 [23m59s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready' (current status: 'cluster_provisioning')
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+----------------------+--------------------------------+
|            CLUSTER ID            |        STATUS        |           UPDATED AT           |
+----------------------------------+----------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioning | 2022-09-28 11:09:09.682719     |
|                                  |                      | +0200 CEST                     |
+----------------------------------+----------------------+--------------------------------+
1584/7200 [26m23s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready' (current status: 'cluster_provisioning')
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+----------------------+--------------------------------+
|            CLUSTER ID            |        STATUS        |           UPDATED AT           |
+----------------------------------+----------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioning | 2022-09-28 11:11:39.581829     |
|                                  |                      | +0200 CEST                     |
+----------------------------------+----------------------+--------------------------------+
1728/7200 [28m47s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready' (current status: 'cluster_provisioning')
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+----------------------+--------------------------------+
|            CLUSTER ID            |        STATUS        |           UPDATED AT           |
+----------------------------------+----------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioning | 2022-09-28 11:14:09.587337     |
|                                  |                      | +0200 CEST                     |
+----------------------------------+----------------------+--------------------------------+
1872/7200 [31m11s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready' (current status: 'cluster_provisioning')
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+----------------------+--------------------------------+
|            CLUSTER ID            |        STATUS        |           UPDATED AT           |
+----------------------------------+----------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioning | 2022-09-28 11:16:39.863313     |
|                                  |                      | +0200 CEST                     |
+----------------------------------+----------------------+--------------------------------+
2016/7200 [33m35s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready' (current status: 'cluster_provisioning')
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+----------------------+--------------------------------+
|            CLUSTER ID            |        STATUS        |           UPDATED AT           |
+----------------------------------+----------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioning | 2022-09-28 11:19:09.646311     |
|                                  |                      | +0200 CEST                     |
+----------------------------------+----------------------+--------------------------------+
2160/7200 [35m59s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready' (current status: 'cluster_provisioning')
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+----------------------+--------------------------------+
|            CLUSTER ID            |        STATUS        |           UPDATED AT           |
+----------------------------------+----------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioning | 2022-09-28 11:21:09.609299     |
|                                  |                      | +0200 CEST                     |
+----------------------------------+----------------------+--------------------------------+
2304/7200 [38m23s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready' (current status: 'cluster_provisioning')
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+----------------------+--------------------------------+
|            CLUSTER ID            |        STATUS        |           UPDATED AT           |
+----------------------------------+----------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioning | 2022-09-28 11:23:39.564739     |
|                                  |                      | +0200 CEST                     |
+----------------------------------+----------------------+--------------------------------+
2448/7200 [40m47s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready' (current status: 'cluster_provisioning')
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+----------------------+--------------------------------+
|            CLUSTER ID            |        STATUS        |           UPDATED AT           |
+----------------------------------+----------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioning | 2022-09-28 11:26:09.619262     |
|                                  |                      | +0200 CEST                     |
+----------------------------------+----------------------+--------------------------------+
2592/7200 [43m11s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready' (current status: 'cluster_provisioning')
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+----------------------+--------------------------------+
|            CLUSTER ID            |        STATUS        |           UPDATED AT           |
+----------------------------------+----------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioning | 2022-09-28 11:28:40.130205     |
|                                  |                      | +0200 CEST                     |
+----------------------------------+----------------------+--------------------------------+
2736/7200 [45m35s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready' (current status: 'cluster_provisioning')
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+----------------------+--------------------------------+
|            CLUSTER ID            |        STATUS        |           UPDATED AT           |
+----------------------------------+----------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioning | 2022-09-28 11:31:09.651318     |
|                                  |                      | +0200 CEST                     |
+----------------------------------+----------------------+--------------------------------+
2880/7200 [47m59s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready' (current status: 'cluster_provisioning')
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:33:09.755019     |
|                                  |                     | +0200 CEST                     |
+----------------------------------+---------------------+--------------------------------+
3024/7200 [50m23s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready' (current status: 'cluster_provisioned')
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:35:39.61755      |
|                                  |                     | +0200 CEST                     |
+----------------------------------+---------------------+--------------------------------+
3168/7200 [52m47s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready' (current status: 'cluster_provisioned')
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:38:09.69006      |
|                                  |                     | +0200 CEST                     |
+----------------------------------+---------------------+--------------------------------+
3312/7200 [55m11s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready' (current status: 'cluster_provisioned')
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:40:39.622078     |
|                                  |                     | +0200 CEST                     |
+----------------------------------+---------------------+--------------------------------+
3456/7200 [57m35s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready' (current status: 'cluster_provisioned')
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:43:09.7176 +0200 |
|                                  |                     | CEST                           |
+----------------------------------+---------------------+--------------------------------+
3600/7200 [59m59s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready' (current status: 'cluster_provisioned')
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:45:09.731837     |
|                                  |                     | +0200 CEST                     |
+----------------------------------+---------------------+--------------------------------+
3744/7200 [1h2m23s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready' (current status: 'cluster_provisioned')
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:47:39.859602     |
|                                  |                     | +0200 CEST                     |
+----------------------------------+---------------------+--------------------------------+
3888/7200 [1h4m47s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready' (current status: 'cluster_provisioned')
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:50:09.65199      |
|                                  |                     | +0200 CEST                     |
+----------------------------------+---------------------+--------------------------------+
4032/7200 [1h7m11s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready' (current status: 'cluster_provisioned')
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:52:39.581521     |
|                                  |                     | +0200 CEST                     |
+----------------------------------+---------------------+--------------------------------+
4176/7200 [1h9m35s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready' (current status: 'cluster_provisioned')
4232/7200 [1h10m31s] - Polling finished
1/60 [0s] - Waiting for metric 'cluster_status_capacity_used' to contain values '[0 standard 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t us-east-1 aws]
1/60 [0s] - Polling finished
1/60 [0s] - Waiting for metric 'cluster_status_capacity_max' to contain values '[5 standard 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t us-east-1 aws]
1/60 [0s] - Polling finished
1/60 [0s] - Waiting for metric 'cluster_status_capacity_available' to contain values '[5 standard 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t us-east-1 aws]
1/60 [0s] - Polling finished
--- PASS: TestClusterManager_SuccessfulReconcile (4235.34s)
=== RUN   TestClusterManager_SuccessfulReconcileDeprovisionCluster
    kas-fleetshard-sync.go:212: Starting mock fleetshard sync
DUMPING 'clusters' with filter 'cluster_id = 'some-cluster-id''
+-----------------+----------------------+--------------------------------+
|   CLUSTER ID    |        STATUS        |           UPDATED AT           |
+-----------------+----------------------+--------------------------------+
| some-cluster-id | cluster_provisioning | 2022-09-28 11:53:44.744539     |
|                 |                      | +0200 CEST                     |
+-----------------+----------------------+--------------------------------+
1/7200 [0s] - Waiting for cluster 'some-cluster-id' to reach status 'ready'
5/7200 [4s] - Polling finished
1/900 [0s] - Waiting for cluster 'test-cluster-to-be-deleted' to be deleted
18/900 [17s] - Waiting for cluster 'test-cluster-to-be-deleted' to be deleted
36/900 [35s] - Waiting for cluster 'test-cluster-to-be-deleted' to be deleted
54/900 [53s] - Waiting for cluster 'test-cluster-to-be-deleted' to be deleted
72/900 [1m11s] - Waiting for cluster 'test-cluster-to-be-deleted' to be deleted
88/900 [1m27s] - Polling finished
--- PASS: TestClusterManager_SuccessfulReconcileDeprovisionCluster (91.76s)
=== RUN   TestClusterService_FindKafkaInstanceCount
    kas-fleetshard-sync.go:212: Starting mock fleetshard sync
    kas-fleetshard-sync.go:228: Unable to update dataplane cluster status: Unable to retrieve a list of clusters in a 'waiting_for_kas_fleetshard_operator' state: ERROR: column clusters.client_id does not exist (SQLSTATE 42703); ERROR: column clusters.client_id does not exist (SQLSTATE 42703)
    kas-fleetshard-sync.go:234: Failed to update Kafka cluster status: Unable to retrieve a list of clusters in a 'ready' state: ERROR: column clusters.client_id does not exist (SQLSTATE 42703); ERROR: column clusters.client_id does not exist (SQLSTATE 42703)
--- PASS: TestClusterService_FindKafkaInstanceCount (1.53s)
=== RUN   TestClusterService_FindStreamingUnitCountByClusterAndInstanceType
    kas-fleetshard-sync.go:212: Starting mock fleetshard sync
E0928 11:55:19.685162 1479062 logger.go:210] KAFKAS-MGMT-9: failed to generate reserved managed kafkas for clusterID test-cluster: no ready strimzi versions found
    kas-fleetshard-sync.go:234: Failed to update Kafka cluster status: 500 Internal Server Error
    kas-fleetshard-sync.go:234: Failed to update Kafka cluster status: Get "http://localhost:8000/api/kafkas_mgmt/v1/agent-clusters/test-cluster/kafkas": dial tcp <IP_ADDRESS>:8000: connect: connection refused
--- PASS: TestClusterService_FindStreamingUnitCountByClusterAndInstanceType (2.90s)
=== RUN   TestDataPlaneCluster_GetManagedKafkaAgentCRAndObserveCapacityInfo
1/5 [0s] - Waiting for cluster dynamic capacity to show up in ManagedKafkaAgentCR
1/5 [1s] - Polling finished
1/5 [0s] - Waiting for cluster dynamic capacity to disappear from ManagedKafkaAgentCR
1/5 [0s] - Polling finished
--- PASS: TestDataPlaneCluster_GetManagedKafkaAgentCRAndObserveCapacityInfo (1.42s)
=== RUN   TestDataPlaneCluster_ClusterStatusTransitionsToReadySuccessfully
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:53:42.519256     |
|                                  |                     | +0200 CEST                     |
+----------------------------------+---------------------+--------------------------------+
1/7200 [0s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'waiting_for_kas_fleetshard_operator'
3/7200 [2s] - Polling finished
--- PASS: TestDataPlaneCluster_ClusterStatusTransitionsToReadySuccessfully (3.63s)
=== RUN   TestDataPlaneCluster_BadRequestWhenNonexistingCluster
--- PASS: TestDataPlaneCluster_BadRequestWhenNonexistingCluster (1.73s)
=== RUN   TestDataPlaneCluster_UnauthorizedWhenNoAuthProvided
--- PASS: TestDataPlaneCluster_UnauthorizedWhenNoAuthProvided (1.94s)
=== RUN   TestDataPlaneCluster_NotFoundWhenNoProperAuthRole
--- PASS: TestDataPlaneCluster_NotFoundWhenNoProperAuthRole (1.42s)
=== RUN   TestDataPlaneCluster_NotFoundWhenNotAllowedClusterID
--- PASS: TestDataPlaneCluster_NotFoundWhenNotAllowedClusterID (1.77s)
=== RUN   TestDataPlaneCluster_GetManagedKafkaAgentCRSuccess
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:53:42.519256     |
|                                  |                     | +0200 CEST                     |
+----------------------------------+---------------------+--------------------------------+
1/7200 [0s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'waiting_for_kas_fleetshard_operator'
3/7200 [2s] - Polling finished
--- PASS: TestDataPlaneCluster_GetManagedKafkaAgentCRSuccess (3.67s)
=== RUN   TestDataPlaneCluster_ClusterStatusTransitionsToWaitingForKASFleetOperatorWhenOperatorIsNotReady
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:53:42.519256     |
|                                  |                     | +0200 CEST                     |
+----------------------------------+---------------------+--------------------------------+
1/7200 [0s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'waiting_for_kas_fleetshard_operator'
3/7200 [2s] - Polling finished
--- PASS: TestDataPlaneCluster_ClusterStatusTransitionsToWaitingForKASFleetOperatorWhenOperatorIsNotReady (4.61s)
=== RUN   TestDataPlaneCluster_WhenReportedStrimziVersionsIsEmptyAndClusterStrimziVersionsIsEmptyItRemainsEmpty
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:53:42.519256     |
|                                  |                     | +0200 CEST                     |
+----------------------------------+---------------------+--------------------------------+
1/7200 [0s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'waiting_for_kas_fleetshard_operator'
3/7200 [2s] - Polling finished
--- PASS: TestDataPlaneCluster_WhenReportedStrimziVersionsIsEmptyAndClusterStrimziVersionsIsEmptyItRemainsEmpty (3.31s)
=== RUN   TestDataPlaneCluster_WhenReportedStrimziVersionsIsNilAndClusterStrimziVersionsIsEmptyItRemainsEmpty
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:53:42.519256     |
|                                  |                     | +0200 CEST                     |
+----------------------------------+---------------------+--------------------------------+
1/7200 [0s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'waiting_for_kas_fleetshard_operator'
3/7200 [2s] - Polling finished
--- PASS: TestDataPlaneCluster_WhenReportedStrimziVersionsIsNilAndClusterStrimziVersionsIsEmptyItRemainsEmpty (3.64s)
=== RUN   TestDataPlaneCluster_WhenReportedStrimziVersionsIsEmptyAndClusterStrimziVersionsIsNotEmptyItRemainsUnchanged
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:53:42.519256     |
|                                  |                     | +0200 CEST                     |
+----------------------------------+---------------------+--------------------------------+
1/7200 [0s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'waiting_for_kas_fleetshard_operator'
3/7200 [2s] - Polling finished
--- PASS: TestDataPlaneCluster_WhenReportedStrimziVersionsIsEmptyAndClusterStrimziVersionsIsNotEmptyItRemainsUnchanged (3.58s)
=== RUN   TestDataPlaneCluster_WhenReportedStrimziVersionsIsNilAndClusterStrimziVersionsIsNotEmptyItRemainsUnchanged
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:53:42.519256     |
|                                  |                     | +0200 CEST                     |
+----------------------------------+---------------------+--------------------------------+
1/7200 [0s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'waiting_for_kas_fleetshard_operator'
3/7200 [2s] - Polling finished
--- PASS: TestDataPlaneCluster_WhenReportedStrimziVersionsIsNilAndClusterStrimziVersionsIsNotEmptyItRemainsUnchanged (3.54s)
=== RUN   TestDataPlaneCluster_WhenReportedStrimziVersionsAreDifferentClusterStrimziVersionsIsUpdated
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:53:42.519256     |
|                                  |                     | +0200 CEST                     |
+----------------------------------+---------------------+--------------------------------+
1/7200 [0s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'waiting_for_kas_fleetshard_operator'
3/7200 [2s] - Polling finished
--- PASS: TestDataPlaneCluster_WhenReportedStrimziVersionsAreDifferentClusterStrimziVersionsIsUpdated (3.72s)
=== RUN   TestDataPlaneEndpoints_AuthzSuccess
--- PASS: TestDataPlaneEndpoints_AuthzSuccess (2.32s)
=== RUN   TestDataPlaneEndpoints_AuthzFailWhenNoRealmRole
--- PASS: TestDataPlaneEndpoints_AuthzFailWhenNoRealmRole (1.90s)
=== RUN   TestDataPlaneEndpoints_AuthzFailWhenClusterIdNotMatch
--- PASS: TestDataPlaneEndpoints_AuthzFailWhenClusterIdNotMatch (1.49s)
=== RUN   TestDataPlaneEndpoints_GetManagedKafkas
--- PASS: TestDataPlaneEndpoints_GetManagedKafkas (1.48s)
=== RUN   TestDataPlaneEndpoints_UpdateManagedKafkas
1/60 [0s] - waiting for Kafka routes to be created
2/60 [1s] - waiting for Kafka routes to be created
3/60 [2s] - waiting for Kafka routes to be created
4/60 [3s] - waiting for Kafka routes to be created
5/60 [4s] - waiting for Kafka routes to be created
6/60 [5s] - waiting for Kafka routes to be created
7/60 [6s] - waiting for Kafka routes to be created
8/60 [7s] - waiting for Kafka routes to be created
9/60 [8s] - waiting for Kafka routes to be created
10/60 [9s] - waiting for Kafka routes to be created
11/60 [10s] - waiting for Kafka routes to be created
12/60 [11s] - waiting for Kafka routes to be created
13/60 [12s] - waiting for Kafka routes to be created
14/60 [13s] - waiting for Kafka routes to be created
15/60 [14s] - waiting for Kafka routes to be created
16/60 [15s] - waiting for Kafka routes to be created
17/60 [16s] - waiting for Kafka routes to be created
18/60 [17s] - waiting for Kafka routes to be created
19/60 [18s] - waiting for Kafka routes to be created
20/60 [19s] - waiting for Kafka routes to be created
21/60 [20s] - waiting for Kafka routes to be created
22/60 [21s] - waiting for Kafka routes to be created
23/60 [22s] - waiting for Kafka routes to be created
24/60 [23s] - waiting for Kafka routes to be created
25/60 [24s] - waiting for Kafka routes to be created
26/60 [25s] - waiting for Kafka routes to be created
27/60 [26s] - waiting for Kafka routes to be created
28/60 [27s] - waiting for Kafka routes to be created
29/60 [28s] - waiting for Kafka routes to be created
30/60 [29s] - waiting for Kafka routes to be created
30/60 [29s] - Polling finished
E0928 11:56:37.026535 1479062 data_plane_kafka.go:146] failed to get kafka cluster by id ccq1kdnfnec9rckq3qg0: KAFKAS-MGMT-7: KafkaResource with id='ccq1kdnfnec9rckq3qg0' not found
--- PASS: TestDataPlaneEndpoints_UpdateManagedKafkas (31.81s)
=== RUN   TestDataPlaneEndpoints_GetAndUpdateManagedKafkasWithTlsCerts
--- PASS: TestDataPlaneEndpoints_GetAndUpdateManagedKafkasWithTlsCerts (1.44s)
=== RUN   TestDataPlaneEndpoints_GetAndUpdateManagedKafkasWithServiceAccounts
--- PASS: TestDataPlaneEndpoints_GetAndUpdateManagedKafkasWithServiceAccounts (1.62s)
=== RUN   TestDataPlaneEndpoints_GetManagedKafkasWithoutOAuthTLSCert
--- PASS: TestDataPlaneEndpoints_GetManagedKafkasWithoutOAuthTLSCert (1.56s)
=== RUN   TestDataPlaneEndpoints_GetManagedKafkasWithOauthMaximumSessionLifetime
--- PASS: TestDataPlaneEndpoints_GetManagedKafkasWithOauthMaximumSessionLifetime (1.39s)
=== RUN   TestDataPlaneEndpoints_UpdateManagedKafkasWithRoutesAndAdminApiServerUrl
1/60 [0s] - waiting for Kafka routes to be created
2/60 [1s] - waiting for Kafka routes to be created
3/60 [2s] - waiting for Kafka routes to be created
4/60 [3s] - waiting for Kafka routes to be created
5/60 [4s] - waiting for Kafka routes to be created
6/60 [5s] - waiting for Kafka routes to be created
7/60 [6s] - waiting for Kafka routes to be created
8/60 [7s] - waiting for Kafka routes to be created
9/60 [8s] - waiting for Kafka routes to be created
10/60 [9s] - waiting for Kafka routes to be created
11/60 [10s] - waiting for Kafka routes to be created
12/60 [11s] - waiting for Kafka routes to be created
13/60 [12s] - waiting for Kafka routes to be created
14/60 [13s] - waiting for Kafka routes to be created
15/60 [14s] - waiting for Kafka routes to be created
16/60 [15s] - waiting for Kafka routes to be created
17/60 [16s] - waiting for Kafka routes to be created
18/60 [17s] - waiting for Kafka routes to be created
19/60 [18s] - waiting for Kafka routes to be created
20/60 [19s] - waiting for Kafka routes to be created
21/60 [20s] - waiting for Kafka routes to be created
22/60 [21s] - waiting for Kafka routes to be created
23/60 [22s] - waiting for Kafka routes to be created
24/60 [23s] - waiting for Kafka routes to be created
25/60 [24s] - waiting for Kafka routes to be created
26/60 [25s] - waiting for Kafka routes to be created
27/60 [26s] - waiting for Kafka routes to be created
28/60 [27s] - waiting for Kafka routes to be created
29/60 [28s] - waiting for Kafka routes to be created
30/60 [29s] - waiting for Kafka routes to be created
31/60 [30s] - waiting for Kafka routes to be created
31/60 [30s] - Polling finished
--- PASS: TestDataPlaneEndpoints_UpdateManagedKafkasWithRoutesAndAdminApiServerUrl (32.51s)
=== RUN   TestDataPlaneEndpoints_GetManagedKafkasWithOAuthTLSCert
--- PASS: TestDataPlaneEndpoints_GetManagedKafkasWithOAuthTLSCert (1.90s)
=== RUN   TestDataPlaneEndpoints_UpdateManagedKafkaWithErrorStatus
E0928 11:57:18.812341 1479062 logger.go:205] Kafka status for Kafka ID 'ccq1kvnfnec9rckq3u6g' in ClusterID 'ccq0hqnfnec9rckq3010' reported as failed by KAS Fleet Shard Operator: 'test-err-message'
--- PASS: TestDataPlaneEndpoints_UpdateManagedKafkaWithErrorStatus (1.73s)
=== RUN   TestDataPlaneEndpoints_UpdateManagedKafka_RemoveFailedReason
--- PASS: TestDataPlaneEndpoints_UpdateManagedKafka_RemoveFailedReason (1.30s)
=== RUN   TestDataPlaneEndpoints_UpdateManagedKafka_PrewarmingStatusInfoMetrics
1/60 [0s] - Waiting for metric 'prewarmed_kafka_instances' to contain values '[0 standard ccq0hqnfnec9rckq3010 rejectedClusterFull]
1/60 [0s] - Polling finished
1/60 [0s] - Waiting for metric 'prewarmed_kafka_instances' to contain values '[1 standard ccq0hqnfnec9rckq3010 error]
1/60 [0s] - Polling finished
1/60 [0s] - Waiting for metric 'prewarmed_kafka_instances' to contain values '[1 standard ccq0hqnfnec9rckq3010 ready]
1/60 [0s] - Polling finished
1/60 [0s] - Waiting for metric 'prewarmed_kafka_instances' to contain values '[0 standard ccq0hqnfnec9rckq3010 unknown]
1/60 [0s] - Polling finished
1/60 [0s] - Waiting for metric 'prewarmed_kafka_instances' to contain values '[0 standard ccq0hqnfnec9rckq3010 deleted]
1/60 [0s] - Polling finished
1/60 [0s] - Waiting for metric 'prewarmed_kafka_instances' to contain values '[0 standard ccq0hqnfnec9rckq3010 rejected]
1/60 [0s] - Polling finished
1/60 [0s] - Waiting for metric 'prewarmed_kafka_instances' to contain values '[0 standard ccq0hqnfnec9rckq3010 installing]
1/60 [0s] - Polling finished
1/60 [0s] - Waiting for metric 'prewarmed_kafka_instances' to contain values '[0 developer ccq0hqnfnec9rckq3010 deleted]
1/60 [0s] - Polling finished
1/60 [0s] - Waiting for metric 'prewarmed_kafka_instances' to contain values '[1 developer ccq0hqnfnec9rckq3010 unknown]
1/60 [0s] - Polling finished
1/60 [0s] - Waiting for metric 'prewarmed_kafka_instances' to contain values '[0 developer ccq0hqnfnec9rckq3010 rejected]
1/60 [0s] - Polling finished
1/60 [0s] - Waiting for metric 'prewarmed_kafka_instances' to contain values '[1 developer ccq0hqnfnec9rckq3010 installing]
1/60 [0s] - Polling finished
1/60 [0s] - Waiting for metric 'prewarmed_kafka_instances' to contain values '[0 developer ccq0hqnfnec9rckq3010 rejectedClusterFull]
1/60 [0s] - Polling finished
1/60 [0s] - Waiting for metric 'prewarmed_kafka_instances' to contain values '[1 developer ccq0hqnfnec9rckq3010 error]
1/60 [0s] - Polling finished
1/60 [0s] - Waiting for metric 'prewarmed_kafka_instances' to contain values '[0 developer ccq0hqnfnec9rckq3010 ready]
1/60 [0s] - Polling finished
--- PASS: TestDataPlaneEndpoints_UpdateManagedKafka_PrewarmingStatusInfoMetrics (1.21s)
=== RUN   TestDataPlaneEndpoints_ReassignRejectedKafkaDueToInsufficientResources
    kas-fleetshard-sync.go:212: Starting mock fleetshard sync
1/120 [0s] - Waiting for kafka named 'test-kafka1' to have a ClusterID
E0928 11:57:23.375534 1479062 logger.go:210] failed to reconcile provisioning kafka ccq1l0nfnec9rckq3vn0: Region us-east-1 cannot accept instance type: developer at this moment for kafka ccq1l0nfnec9rckq3vn0
E0928 11:57:23.375676 1479062 logger.go:210] failed to reconcile provisioning kafka ccq1l0nfnec9rckq3vng: Region us-east-1 cannot accept instance type: standard at this moment for kafka ccq1l0nfnec9rckq3vng
2/120 [1s] - Waiting for kafka named 'test-kafka1' to have a ClusterID
E0928 11:57:24.395427 1479062 logger.go:210] failed to reconcile provisioning kafka ccq1l0nfnec9rckq3vng: Region us-east-1 cannot accept instance type: standard at this moment for kafka ccq1l0nfnec9rckq3vng
3/120 [2s] - Polling finished
1/120 [0s] - Waiting for kafka named 'a-kafka1' to have a ClusterID
E0928 11:57:25.373401 1479062 logger.go:210] failed to reconcile provisioning kafka ccq1l0nfnec9rckq3vng: Region us-east-1 cannot accept instance type: standard at this moment for kafka ccq1l0nfnec9rckq3vng
2/120 [1s] - Waiting for kafka named 'a-kafka1' to have a ClusterID
3/120 [2s] - Polling finished
--- PASS: TestDataPlaneEndpoints_ReassignRejectedKafkaDueToInsufficientResources (6.82s)
=== RUN   TestFederation_GetFederatedMetrics
    kas-fleetshard-sync.go:212: Starting mock fleetshard sync
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:53:42.519256     |
|                                  |                     | +0200 CEST                     |
+----------------------------------+---------------------+--------------------------------+
1/7200 [0s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready'
5/7200 [4s] - Polling finished
--- PASS: TestFederation_GetFederatedMetrics (4.61s)
=== RUN   TestFederation_GetFederatedMetricsUsingMasSsoToken
    kas-fleetshard-sync.go:212: Starting mock fleetshard sync
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:53:42.519256     |
|                                  |                     | +0200 CEST                     |
+----------------------------------+---------------------+--------------------------------+
1/7200 [0s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready'
5/7200 [4s] - Polling finished
--- PASS: TestFederation_GetFederatedMetricsUsingMasSsoToken (4.60s)
=== RUN   TestKafkaCreate_ManualScaling
    kas-fleetshard-sync.go:212: Starting mock fleetshard sync
DUMPING 'clusters' with filter 'cluster_id = 'test-cluster-id''
{{ EMPTY }}
1/7200 [0s] - Waiting for cluster 'test-cluster-id' to reach status 'ready'
5/7200 [4s] - Polling finished
=== RUN   TestKafkaCreate_ManualScaling/should_successfully_create_kafka
1/1200 [0s] - Waiting for kafka creation to be accepted
1/1200 [0s] - Polling finished
1/1800 [0s] - Waiting for kafka 'ccq1l5nfnec9rckq41q0' to reach status 'ready'
36/1800 [35s] - Waiting for kafka 'ccq1l5nfnec9rckq41q0' to reach status 'ready' (current status provisioning)
57/1800 [56s] - Polling finished
=== CONT  TestKafkaCreate_ManualScaling
    poll.go:168: 1/600 [0s] - Waiting for kafka 'ccq1l5nfnec9rckq41q0' to be deleted
    poll.go:168: 12/600 [11s] - Waiting for kafka 'ccq1l5nfnec9rckq41q0' to be deleted
    poll.go:168: 24/600 [23s] - Waiting for kafka 'ccq1l5nfnec9rckq41q0' to be deleted
    poll.go:110: 31/600 [30s] - Polling finished
=== RUN   TestKafkaCreate_ManualScaling/should_reject_kafka_request_if_there_is_no_capacity_left
1/1200 [0s] - Waiting for kafka creation to be accepted
Error ocurred when polling (will be ignored): 403 Forbidden
1/1200 [0s] - Polling finished
=== CONT  TestKafkaCreate_ManualScaling
    poll.go:168: 1/600 [0s] - Waiting for kafka 'ccq1lr7fnec9rckq45eg' to be deleted
    poll.go:168: 12/600 [11s] - Waiting for kafka 'ccq1lr7fnec9rckq45eg' to be deleted
    poll.go:168: 24/600 [23s] - Waiting for kafka 'ccq1lr7fnec9rckq45eg' to be deleted
    poll.go:110: 31/600 [30s] - Polling finished
    poll.go:168: 1/600 [0s] - Waiting for kafka 'ccq1lr7fnec9rckq45f0' to be deleted
    poll.go:168: 12/600 [11s] - Waiting for kafka 'ccq1lr7fnec9rckq45f0' to be deleted
    poll.go:168: 24/600 [23s] - Waiting for kafka 'ccq1lr7fnec9rckq45f0' to be deleted
    poll.go:110: 31/600 [30s] - Polling finished
--- PASS: TestKafkaCreate_ManualScaling (151.48s)
    --- PASS: TestKafkaCreate_ManualScaling/should_successfully_create_kafka (86.06s)
    --- PASS: TestKafkaCreate_ManualScaling/should_reject_kafka_request_if_there_is_no_capacity_left (60.06s)
=== RUN   TestKafkaCreate_DynamicScaling
    kas-fleetshard-sync.go:212: Starting mock fleetshard sync
=== RUN   TestKafkaCreate_DynamicScaling/should_successfully_create_kafka
1/1200 [0s] - Waiting for kafka creation to be accepted
1/1200 [1s] - Polling finished
1/1800 [0s] - Waiting for kafka 'ccq1mb7fnec9rckq47u0' to reach status 'ready'
36/1800 [35s] - Waiting for kafka 'ccq1mb7fnec9rckq47u0' to reach status 'ready' (current status accepted)
72/1800 [1m11s] - Waiting for kafka 'ccq1mb7fnec9rckq47u0' to reach status 'ready' (current status preparing)
108/1800 [1m47s] - Waiting for kafka 'ccq1mb7fnec9rckq47u0' to reach status 'ready' (current status provisioning)
122/1800 [2m1s] - Polling finished
=== CONT  TestKafkaCreate_DynamicScaling
    poll.go:168: 1/600 [0s] - Waiting for kafka 'ccq1mb7fnec9rckq47u0' to be deleted
    poll.go:168: 12/600 [11s] - Waiting for kafka 'ccq1mb7fnec9rckq47u0' to be deleted
    poll.go:168: 24/600 [23s] - Waiting for kafka 'ccq1mb7fnec9rckq47u0' to be deleted
    poll.go:110: 30/600 [29s] - Polling finished
=== RUN   TestKafkaCreate_DynamicScaling/should_reject_kafka_request_if_there_is_no_capacity_left
1/1200 [0s] - Waiting for kafka creation to be accepted
Error ocurred when polling (will be ignored): 403 Forbidden
1/1200 [0s] - Polling finished
=== CONT  TestKafkaCreate_DynamicScaling
    poll.go:168: 1/600 [0s] - Waiting for kafka 'ccq1ngvfnec9rckq4eig' to be deleted
    poll.go:168: 12/600 [11s] - Waiting for kafka 'ccq1ngvfnec9rckq4eig' to be deleted
    poll.go:168: 24/600 [23s] - Waiting for kafka 'ccq1ngvfnec9rckq4eig' to be deleted
    poll.go:110: 31/600 [30s] - Polling finished
    poll.go:168: 1/600 [0s] - Waiting for kafka 'ccq1ngvfnec9rckq4ej0' to be deleted
    poll.go:168: 12/600 [11s] - Waiting for kafka 'ccq1ngvfnec9rckq4ej0' to be deleted
    poll.go:168: 24/600 [23s] - Waiting for kafka 'ccq1ngvfnec9rckq4ej0' to be deleted
    poll.go:110: 31/600 [30s] - Polling finished
--- PASS: TestKafkaCreate_DynamicScaling (214.21s)
    --- PASS: TestKafkaCreate_DynamicScaling/should_successfully_create_kafka (150.84s)
    --- PASS: TestKafkaCreate_DynamicScaling/should_reject_kafka_request_if_there_is_no_capacity_left (60.04s)
=== RUN   TestKafkaCreate_ValidatePlanParam
    kas-fleetshard-sync.go:212: Starting mock fleetshard sync
--- PASS: TestKafkaCreate_ValidatePlanParam (2.69s)
=== RUN   Test_KafkaListSearchAndOrderBy
    kas-fleetshard-sync.go:212: Starting mock fleetshard sync
=== RUN   Test_KafkaListSearchAndOrderBy/Order_By_Name_Asc
=== RUN   Test_KafkaListSearchAndOrderBy/Order_By_Name_Desc
=== RUN   Test_KafkaListSearchAndOrderBy/Filter_By_Name
=== RUN   Test_KafkaListSearchAndOrderBy/Filter_By_Name_Not_In
=== RUN   Test_KafkaListSearchAndOrderBy/Filter_By_Name_Not_Equal
=== RUN   Test_KafkaListSearchAndOrderBy/Filter_By_Name_Not_Equal_Non_Existent_Name
=== RUN   Test_KafkaListSearchAndOrderBy/Filter_By_Owner_With_Special_Characters
=== RUN   Test_KafkaListSearchAndOrderBy/Filter_By_Invalid_Column_Name
=== RUN   Test_KafkaListSearchAndOrderBy/Filter_By_Incomplete_Query
=== RUN   Test_KafkaListSearchAndOrderBy/Filter_By_Incomplete_Composed_Query
=== RUN   Test_KafkaListSearchAndOrderBy/Filter_By_Invalid_Join
=== RUN   Test_KafkaListSearchAndOrderBy/Filter_By_Or_Join
=== RUN   Test_KafkaListSearchAndOrderBy/Filter_By_And_Join
=== RUN   Test_KafkaListSearchAndOrderBy/Filter_By_Two_Joins
=== RUN   Test_KafkaListSearchAndOrderBy/Filter_By_Too_Many_Joins
=== RUN   Test_KafkaListSearchAndOrderBy/Filter_By_Like_(%xxx%)
=== RUN   Test_KafkaListSearchAndOrderBy/MGDSTRM-2956_-_Order_By_Invalid_Fields
=== CONT  Test_KafkaListSearchAndOrderBy
    kas-fleetshard-sync.go:228: Unable to update dataplane cluster status: Unable to retrieve a list of clusters in a 'waiting_for_kas_fleetshard_operator' state: ERROR: column clusters.cluster_dns does not exist (SQLSTATE 42703); ERROR: column clusters.cluster_dns does not exist (SQLSTATE 42703)
    kas-fleetshard-sync.go:234: Failed to update Kafka cluster status: Unable to retrieve a list of clusters in a 'ready' state: ERROR: column clusters.cluster_dns does not exist (SQLSTATE 42703); ERROR: column clusters.cluster_dns does not exist (SQLSTATE 42703)
--- PASS: Test_KafkaListSearchAndOrderBy (1.35s)
    --- PASS: Test_KafkaListSearchAndOrderBy/Order_By_Name_Asc (0.72s)
    --- PASS: Test_KafkaListSearchAndOrderBy/Order_By_Name_Desc (0.01s)
    --- PASS: Test_KafkaListSearchAndOrderBy/Filter_By_Name (0.01s)
    --- PASS: Test_KafkaListSearchAndOrderBy/Filter_By_Name_Not_In (0.00s)
    --- PASS: Test_KafkaListSearchAndOrderBy/Filter_By_Name_Not_Equal (0.00s)
    --- PASS: Test_KafkaListSearchAndOrderBy/Filter_By_Name_Not_Equal_Non_Existent_Name (0.00s)
    --- PASS: Test_KafkaListSearchAndOrderBy/Filter_By_Owner_With_Special_Characters (0.00s)
    --- PASS: Test_KafkaListSearchAndOrderBy/Filter_By_Invalid_Column_Name (0.00s)
    --- PASS: Test_KafkaListSearchAndOrderBy/Filter_By_Incomplete_Query (0.00s)
    --- PASS: Test_KafkaListSearchAndOrderBy/Filter_By_Incomplete_Composed_Query (0.00s)
    --- PASS: Test_KafkaListSearchAndOrderBy/Filter_By_Invalid_Join (0.00s)
    --- PASS: Test_KafkaListSearchAndOrderBy/Filter_By_Or_Join (0.00s)
    --- PASS: Test_KafkaListSearchAndOrderBy/Filter_By_And_Join (0.00s)
    --- PASS: Test_KafkaListSearchAndOrderBy/Filter_By_Two_Joins (0.00s)
    --- PASS: Test_KafkaListSearchAndOrderBy/Filter_By_Too_Many_Joins (0.01s)
    --- PASS: Test_KafkaListSearchAndOrderBy/Filter_By_Like_(%xxx%) (0.00s)
    --- PASS: Test_KafkaListSearchAndOrderBy/MGDSTRM-2956_-_Order_By_Invalid_Fields (0.00s)
=== RUN   TestKafka_InstanceTypeCapacity
E0928 12:03:50.618187 1479062 logger.go:210] failed to reconcile ready cluster first: failed to reconcile ready cluster resources first : failed to apply resources for cluster first: KAFKAS-MGMT-9: failed to apply resources ext-managedservice-cluster-mgr
 caused by: failed to create syncset for cluster first: KAFKAS-MGMT-7: ocm client failed to create syncset: status is 404, identifier is '404', code is 'CLUSTERS-MGMT-404' and operation identifier is '66d17fe6-d0b5-4f85-805f-fab92f42a39b': Cluster 'first' not found
E0928 12:03:50.618886 1479062 logger.go:210] failed to reconcile ready cluster second: failed to reconcile ready cluster resources second : failed to apply resources for cluster second: KAFKAS-MGMT-9: failed to apply resources ext-managedservice-cluster-mgr
 caused by: failed to create syncset for cluster second: KAFKAS-MGMT-7: ocm client failed to create syncset: status is 404, identifier is '404', code is 'CLUSTERS-MGMT-404' and operation identifier is '9dd0389c-3d98-4def-97b5-aa1d7e4edf22': Cluster 'second' not found
E0928 12:03:50.619242 1479062 logger.go:210] failed to reconcile ready cluster third: failed to reconcile ready cluster resources third : failed to apply resources for cluster third: KAFKAS-MGMT-9: failed to apply resources ext-managedservice-cluster-mgr
 caused by: failed to create syncset for cluster third: KAFKAS-MGMT-7: ocm client failed to create syncset: status is 404, identifier is '404', code is 'CLUSTERS-MGMT-404' and operation identifier is '4cc407ad-5fa9-4e9f-835f-33d62ebf3f0c': Cluster 'third' not found
E0928 12:03:50.619607 1479062 logger.go:210] failed to reconcile ready cluster fourth: failed to reconcile ready cluster resources fourth : failed to apply resources for cluster fourth: KAFKAS-MGMT-9: failed to apply resources ext-managedservice-cluster-mgr
 caused by: failed to create syncset for cluster fourth: KAFKAS-MGMT-7: ocm client failed to create syncset: status is 404, identifier is '404', code is 'CLUSTERS-MGMT-404' and operation identifier is '8268ec7f-c50e-43bf-a4a9-b9e45fe0cb6b': Cluster 'fourth' not found
E0928 12:03:50.619922 1479062 logger.go:210] failed to reconcile ready cluster fifth: failed to reconcile ready cluster resources fifth : failed to apply resources for cluster fifth: KAFKAS-MGMT-9: failed to apply resources ext-managedservice-cluster-mgr
 caused by: failed to create syncset for cluster fifth: KAFKAS-MGMT-7: ocm client failed to create syncset: status is 404, identifier is '404', code is 'CLUSTERS-MGMT-404' and operation identifier is '00235e78-b130-47e7-86be-8b8174358ba4': Cluster 'fifth' not found
E0928 12:03:50.620269 1479062 logger.go:210] failed to reconcile ready cluster sixth: failed to reconcile ready cluster resources sixth : failed to apply resources for cluster sixth: KAFKAS-MGMT-9: failed to apply resources ext-managedservice-cluster-mgr
 caused by: failed to create syncset for cluster sixth: KAFKAS-MGMT-7: ocm client failed to create syncset: status is 404, identifier is '404', code is 'CLUSTERS-MGMT-404' and operation identifier is '020f97fb-1e03-465c-8063-652e90d09841': Cluster 'sixth' not found
--- SKIP: TestKafka_InstanceTypeCapacity (3.04s)
=== RUN   TestKafka_Update
=== RUN   TestKafka_Update/update_with_an_empty_body_should_not_fail
=== RUN   TestKafka_Update/should_fail_if_owner_is_empty
=== RUN   TestKafka_Update/should_fail_if_trying_to_update_owner_as_not_an_owner/_org_admin
=== RUN   TestKafka_Update/should_fail_if_trying_to_update_with_an_empty_body_as_a_non_owner_or_org_admin
=== RUN   TestKafka_Update/should_fail_when_trying_to_update_non-existent_kafka
=== RUN   TestKafka_Update/should_succeed_changing_reauthentication_enabled_value_to_true
=== RUN   TestKafka_Update/should_succeed_changing_reauthentication_enabled_value_to_false
=== RUN   TestKafka_Update/should_succeed_when_passing_a_valid_new_username_as_an_owner
--- PASS: TestKafka_Update (2.80s)
    --- PASS: TestKafka_Update/update_with_an_empty_body_should_not_fail (0.65s)
    --- PASS: TestKafka_Update/should_fail_if_owner_is_empty (0.01s)
    --- PASS: TestKafka_Update/should_fail_if_trying_to_update_owner_as_not_an_owner/_org_admin (0.01s)
    --- PASS: TestKafka_Update/should_fail_if_trying_to_update_with_an_empty_body_as_a_non_owner_or_org_admin (0.00s)
    --- PASS: TestKafka_Update/should_fail_when_trying_to_update_non-existent_kafka (0.00s)
    --- PASS: TestKafka_Update/should_succeed_changing_reauthentication_enabled_value_to_true (0.01s)
    --- PASS: TestKafka_Update/should_succeed_changing_reauthentication_enabled_value_to_false (0.00s)
    --- PASS: TestKafka_Update/should_succeed_when_passing_a_valid_new_username_as_an_owner (0.93s)
=== RUN   TestKafkaCreate_TooManyKafkas
DUMPING 'clusters' with filter 'cluster_id = 'test01''
{{ EMPTY }}
1/7200 [0s] - Waiting for cluster 'test01' to reach status 'ready'
2/7200 [1s] - Polling finished
E0928 12:03:55.122537 1479062 logger.go:210] failed to reconcile ready cluster test01: failed to reconcile ready cluster resources test01 : failed to apply resources for cluster test01: KAFKAS-MGMT-9: failed to apply resources ext-managedservice-cluster-mgr
 caused by: failed to create syncset for cluster test01: KAFKAS-MGMT-7: ocm client failed to create syncset: status is 404, identifier is '404', code is 'CLUSTERS-MGMT-404' and operation identifier is '8e5ada13-0275-4a5f-b180-e79f9394bcae': Cluster 'test01' not found
--- SKIP: TestKafkaCreate_TooManyKafkas (1.71s)
=== RUN   TestKafkaPost_Validations
=== RUN   TestKafkaPost_Validations/HTTP_400_when_region_not_supported
=== RUN   TestKafkaPost_Validations/HTTP_400_when_provider_not_supported
=== RUN   TestKafkaPost_Validations/HTTP_400_when_name_not_provided
=== RUN   TestKafkaPost_Validations/HTTP_400_when_name_is_not_valid
=== RUN   TestKafkaPost_Validations/HTTP_400_when_name_is_too_long
--- PASS: TestKafkaPost_Validations (1.44s)
    --- PASS: TestKafkaPost_Validations/HTTP_400_when_region_not_supported (0.80s)
    --- PASS: TestKafkaPost_Validations/HTTP_400_when_provider_not_supported (0.01s)
    --- PASS: TestKafkaPost_Validations/HTTP_400_when_name_not_provided (0.00s)
    --- PASS: TestKafkaPost_Validations/HTTP_400_when_name_is_not_valid (0.00s)
    --- PASS: TestKafkaPost_Validations/HTTP_400_when_name_is_too_long (0.00s)
=== RUN   TestKafkaPost_NameUniquenessValidations
    kas-fleetshard-sync.go:212: Starting mock fleetshard sync
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:53:42.519256     |
|                                  |                     | +0200 CEST                     |
+----------------------------------+---------------------+--------------------------------+
1/7200 [0s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready'
5/7200 [4s] - Polling finished
--- PASS: TestKafkaPost_NameUniquenessValidations (4.57s)
=== RUN   TestKafkaDenyList_UnauthorizedValidation
=== RUN   TestKafkaDenyList_UnauthorizedValidation/HTTP_403_when_listing_kafkas
=== RUN   TestKafkaDenyList_UnauthorizedValidation/HTTP_403_when_creating_a_new_kafka_request
=== RUN   TestKafkaDenyList_UnauthorizedValidation/HTTP_403_when_deleting_new_kafka_request
=== RUN   TestKafkaDenyList_UnauthorizedValidation/HTTP_403_when_getting_a_new_kafka_request
--- PASS: TestKafkaDenyList_UnauthorizedValidation (1.49s)
    --- PASS: TestKafkaDenyList_UnauthorizedValidation/HTTP_403_when_listing_kafkas (0.68s)
    --- PASS: TestKafkaDenyList_UnauthorizedValidation/HTTP_403_when_creating_a_new_kafka_request (0.00s)
    --- PASS: TestKafkaDenyList_UnauthorizedValidation/HTTP_403_when_deleting_new_kafka_request (0.00s)
    --- PASS: TestKafkaDenyList_UnauthorizedValidation/HTTP_403_when_getting_a_new_kafka_request (0.00s)
=== RUN   TestKafka_AccessList_UnauthorizedValidation
=== RUN   TestKafka_AccessList_UnauthorizedValidation/HTTP_403_when_listing_kafkas
=== RUN   TestKafka_AccessList_UnauthorizedValidation/HTTP_403_when_creating_a_new_kafka_request
=== RUN   TestKafka_AccessList_UnauthorizedValidation/HTTP_403_when_deleting_new_kafka_request
=== RUN   TestKafka_AccessList_UnauthorizedValidation/HTTP_403_when_getting_a_new_kafka_request
--- PASS: TestKafka_AccessList_UnauthorizedValidation (1.38s)
    --- PASS: TestKafka_AccessList_UnauthorizedValidation/HTTP_403_when_listing_kafkas (0.65s)
    --- PASS: TestKafka_AccessList_UnauthorizedValidation/HTTP_403_when_creating_a_new_kafka_request (0.00s)
    --- PASS: TestKafka_AccessList_UnauthorizedValidation/HTTP_403_when_deleting_new_kafka_request (0.00s)
    --- PASS: TestKafka_AccessList_UnauthorizedValidation/HTTP_403_when_getting_a_new_kafka_request (0.00s)
=== RUN   TestKafkaDenyList_RemovingKafkaForDeniedOwners
    kas-fleetshard-sync.go:212: Starting mock fleetshard sync
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:53:42.519256     |
|                                  |                     | +0200 CEST                     |
+----------------------------------+---------------------+--------------------------------+
1/7200 [0s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready'
5/7200 [4s] - Polling finished
1/1200 [0s] - Waiting for kafkas count to become 1
24/1200 [23s] - Waiting for kafkas count to become 1 (current 5)
48/1200 [47s] - Waiting for kafkas count to become 1 (current 5)
57/1200 [56s] - Polling finished
--- PASS: TestKafkaDenyList_RemovingKafkaForDeniedOwners (61.64s)
=== RUN   TestKafkaQuotaManagementList_MaxAllowedInstances
    kas-fleetshard-sync.go:212: Starting mock fleetshard sync
--- PASS: TestKafkaQuotaManagementList_MaxAllowedInstances (2.96s)
=== RUN   TestKafkaGet
    kas-fleetshard-sync.go:212: Starting mock fleetshard sync
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:53:42.519256     |
|                                  |                     | +0200 CEST                     |
+----------------------------------+---------------------+--------------------------------+
1/7200 [0s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready'
4/7200 [3s] - Polling finished
--- PASS: TestKafkaGet (3.82s)
=== RUN   TestKafka_Delete
=== RUN   TestKafka_Delete/should_fail_when_deleting_kafka_without_async_set_to_true
=== RUN   TestKafka_Delete/should_fail_when_deleting_kafka_with_empty_id
=== RUN   TestKafka_Delete/should_succeed_when_deleting_kafka_with_valid_id_and_context
--- PASS: TestKafka_Delete (3.27s)
    --- PASS: TestKafka_Delete/should_fail_when_deleting_kafka_without_async_set_to_true (0.69s)
    --- PASS: TestKafka_Delete/should_fail_when_deleting_kafka_with_empty_id (0.00s)
    --- PASS: TestKafka_Delete/should_succeed_when_deleting_kafka_with_valid_id_and_context (0.02s)
=== RUN   TestKafkaDelete_DeleteDuringCreation
    kas-fleetshard-sync.go:212: Starting mock fleetshard sync
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:53:42.519256     |
|                                  |                     | +0200 CEST                     |
+----------------------------------+---------------------+--------------------------------+
1/7200 [0s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready'
5/7200 [4s] - Polling finished
1/1200 [0s] - Waiting for kafka creation to be accepted
1/1200 [0s] - Polling finished
1/1800 [0s] - Waiting for kafka 'ccq1oo7fnec9rckq4rvg' to be deleted
27/1800 [26s] - Polling finished
1/1200 [0s] - Waiting for kafka creation to be accepted
1/1200 [0s] - Polling finished
1/1800 [0s] - Waiting for kafka 'ccq1ounfnec9rckq4t90' to reach status 'preparing'
31/1800 [30s] - Polling finished
1/1800 [0s] - Waiting for kafka 'ccq1ounfnec9rckq4t90' to be deleted
31/1800 [30s] - Polling finished
1/1200 [0s] - Waiting for kafka creation to be accepted
1/1200 [0s] - Polling finished
1/1800 [0s] - Waiting for kafka 'ccq1pdnfnec9rckq506g' to reach status 'provisioning'
31/1800 [30s] - Polling finished
1/1800 [0s] - Waiting for kafka 'ccq1pdnfnec9rckq506g' to be deleted
31/1800 [30s] - Polling finished
--- PASS: TestKafkaDelete_DeleteDuringCreation (151.31s)
=== RUN   TestKafkaDelete_Fail
--- PASS: TestKafkaDelete_Fail (1.49s)
=== RUN   TestKafka_DeleteAdminNonOwner
=== RUN   TestKafka_DeleteAdminNonOwner/should_fail_when_deleting_kafka_by_other_user_within_the_same_org_(not_an_owner)
=== RUN   TestKafka_DeleteAdminNonOwner/should_not_fail_when_deleting_kafka_by_other_user_within_the_same_org_(not_an_owner_but_org_admin)
--- PASS: TestKafka_DeleteAdminNonOwner (2.63s)
    --- PASS: TestKafka_DeleteAdminNonOwner/should_fail_when_deleting_kafka_by_other_user_within_the_same_org_(not_an_owner) (0.65s)
    --- PASS: TestKafka_DeleteAdminNonOwner/should_not_fail_when_deleting_kafka_by_other_user_within_the_same_org_(not_an_owner_but_org_admin) (0.02s)
=== RUN   TestKafkaList_Success
    kas-fleetshard-sync.go:212: Starting mock fleetshard sync
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:53:42.519256     |
|                                  |                     | +0200 CEST                     |
+----------------------------------+---------------------+--------------------------------+
1/7200 [0s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready'
3/7200 [2s] - Polling finished
--- PASS: TestKafkaList_Success (3.53s)
=== RUN   TestKafkaList_UnauthUser
--- PASS: TestKafkaList_UnauthUser (1.66s)
=== RUN   TestKafkaList_IncorrectOCMIssuer_AuthzFailure
--- PASS: TestKafkaList_IncorrectOCMIssuer_AuthzFailure (1.63s)
=== RUN   TestKafkaList_CorrectTokenIssuer_AuthzSuccess
--- PASS: TestKafkaList_CorrectTokenIssuer_AuthzSuccess (1.34s)
=== RUN   TestKafka_RemovingExpiredKafkas_NoStandardInstances
    kas-fleetshard-sync.go:212: Starting mock fleetshard sync
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:53:42.519256     |
|                                  |                     | +0200 CEST                     |
+----------------------------------+---------------------+--------------------------------+
1/7200 [0s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready'
5/7200 [4s] - Polling finished
1/1200 [0s] - Waiting for kafkas count to become 1
24/1200 [23s] - Waiting for kafkas count to become 1 (current 3)
48/1200 [47s] - Waiting for kafkas count to become 1 (current 3)
57/1200 [56s] - Polling finished
--- PASS: TestKafka_RemovingExpiredKafkas_NoStandardInstances (61.36s)
=== RUN   TestKafka_RemovingExpiredKafkas_WithStandardInstances
    kas-fleetshard-sync.go:212: Starting mock fleetshard sync
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:53:42.519256     |
|                                  |                     | +0200 CEST                     |
+----------------------------------+---------------------+--------------------------------+
1/7200 [0s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready'
4/7200 [3s] - Polling finished
1/1200 [0s] - Waiting for kafkas count to become 2
24/1200 [23s] - Waiting for kafkas count to become 2 (current 3)
48/1200 [47s] - Waiting for kafkas count to become 2 (current 3)
58/1200 [57s] - Polling finished
--- PASS: TestKafka_RemovingExpiredKafkas_WithStandardInstances (61.41s)
=== RUN   TestLeaderElection_StartedAllWorkersAndDropThenUp
--- PASS: TestLeaderElection_StartedAllWorkersAndDropThenUp (1.65s)
=== RUN   TestObservatorium_ResourceStateMetric
--- PASS: TestObservatorium_ResourceStateMetric (1.37s)
=== RUN   TestObservatorium_GetMetrics
    kas-fleetshard-sync.go:212: Starting mock fleetshard sync
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:53:42.519256     |
|                                  |                     | +0200 CEST                     |
+----------------------------------+---------------------+--------------------------------+
1/7200 [0s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready'
5/7200 [4s] - Polling finished
    poll.go:168: 1/600 [0s] - Waiting for kafka 'ccq1r0ffnec9rckq5eeg' to be deleted
    poll.go:168: 12/600 [11s] - Waiting for kafka 'ccq1r0ffnec9rckq5eeg' to be deleted
    poll.go:168: 24/600 [23s] - Waiting for kafka 'ccq1r0ffnec9rckq5eeg' to be deleted
    poll.go:110: 27/600 [26s] - Polling finished
--- PASS: TestObservatorium_GetMetrics (31.40s)
=== RUN   TestObservatorium_GetMetricsByQueryRange
    kas-fleetshard-sync.go:212: Starting mock fleetshard sync
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:53:42.519256     |
|                                  |                     | +0200 CEST                     |
+----------------------------------+---------------------+--------------------------------+
1/7200 [0s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready'
4/7200 [3s] - Polling finished
1/1800 [0s] - Waiting for kafka 'ccq1r87fnec9rckq5fs0' to reach status 'ready'
36/1800 [35s] - Waiting for kafka 'ccq1r87fnec9rckq5fs0' to reach status 'ready' (current status preparing)
72/1800 [1m11s] - Waiting for kafka 'ccq1r87fnec9rckq5fs0' to reach status 'ready' (current status provisioning)
88/1800 [1m27s] - Polling finished
    poll.go:168: 1/600 [0s] - Waiting for kafka 'ccq1r87fnec9rckq5fs0' to be deleted
    poll.go:168: 12/600 [11s] - Waiting for kafka 'ccq1r87fnec9rckq5fs0' to be deleted
    poll.go:168: 24/600 [23s] - Waiting for kafka 'ccq1r87fnec9rckq5fs0' to be deleted
    poll.go:110: 31/600 [30s] - Polling finished
--- PASS: TestObservatorium_GetMetricsByQueryRange (121.66s)
=== RUN   TestObservatorium_GetMetricsByQueryInstant
    kas-fleetshard-sync.go:212: Starting mock fleetshard sync
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:53:42.519256     |
|                                  |                     | +0200 CEST                     |
+----------------------------------+---------------------+--------------------------------+
1/7200 [0s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready'
5/7200 [4s] - Polling finished
1/1800 [0s] - Waiting for kafka 'ccq1s6nfnec9rckq5lkg' to reach status 'ready'
36/1800 [35s] - Waiting for kafka 'ccq1s6nfnec9rckq5lkg' to reach status 'ready' (current status preparing)
72/1800 [1m11s] - Waiting for kafka 'ccq1s6nfnec9rckq5lkg' to reach status 'ready' (current status provisioning)
88/1800 [1m27s] - Polling finished
    poll.go:168: 1/600 [0s] - Waiting for kafka 'ccq1s6nfnec9rckq5lkg' to be deleted
    poll.go:168: 12/600 [11s] - Waiting for kafka 'ccq1s6nfnec9rckq5lkg' to be deleted
    poll.go:168: 24/600 [23s] - Waiting for kafka 'ccq1s6nfnec9rckq5lkg' to be deleted
    poll.go:110: 30/600 [29s] - Polling finished
--- PASS: TestObservatorium_GetMetricsByQueryInstant (121.46s)
=== RUN   TestOCMClientGetQuotaCosts
--- SKIP: TestOCMClientGetQuotaCosts (1.42s)
=== RUN   TestTermsRequired_CreateKafkaTermsRequired
    kas-fleetshard-sync.go:212: Starting mock fleetshard sync
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:53:42.519256     |
|                                  |                     | +0200 CEST                     |
+----------------------------------+---------------------+--------------------------------+
1/7200 [0s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready'
5/7200 [4s] - Polling finished
    kas-fleetshard-sync.go:234: Failed to update Kafka cluster status: Put "http://localhost:8000/api/kafkas_mgmt/v1/agent-clusters/1v0rm7lg4n47ej5e1trdatvd4ih4ee7t/kafkas/status": dial tcp <IP_ADDRESS>:8000: connect: connection refused
--- SKIP: TestTermsRequired_CreateKafkaTermsRequired (4.92s)
=== RUN   TestTermsRequired_CreateKafka_TermsNotRequired
    kas-fleetshard-sync.go:212: Starting mock fleetshard sync
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:53:42.519256     |
|                                  |                     | +0200 CEST                     |
+----------------------------------+---------------------+--------------------------------+
1/7200 [0s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready'
5/7200 [4s] - Polling finished
    kas-fleetshard-sync.go:234: Failed to update Kafka cluster status: Put "http://localhost:8000/api/kafkas_mgmt/v1/agent-clusters/1v0rm7lg4n47ej5e1trdatvd4ih4ee7t/kafkas/status": dial tcp <IP_ADDRESS>:8000: connect: connection refused
--- SKIP: TestTermsRequired_CreateKafka_TermsNotRequired (4.85s)
=== RUN   TestTermsRequired_ListKafkaTermsRequired
    kas-fleetshard-sync.go:212: Starting mock fleetshard sync
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+---------------------+--------------------------------+
|            CLUSTER ID            |       STATUS        |           UPDATED AT           |
+----------------------------------+---------------------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | cluster_provisioned | 2022-09-28 11:53:42.519256     |
|                                  |                     | +0200 CEST                     |
+----------------------------------+---------------------+--------------------------------+
1/7200 [0s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready'
5/7200 [4s] - Polling finished
DUMPING 'clusters' with filter 'cluster_id = '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t''
+----------------------------------+--------+--------------------------------+
|            CLUSTER ID            | STATUS |           UPDATED AT           |
+----------------------------------+--------+--------------------------------+
| 1v0rm7lg4n47ej5e1trdatvd4ih4ee7t | ready  | 2022-09-28 12:14:54.414715     |
|                                  |        | +0200 CEST                     |
+----------------------------------+--------+--------------------------------+
1/7200 [0s] - Waiting for cluster '1v0rm7lg4n47ej5e1trdatvd4ih4ee7t' to reach status 'ready'
1/7200 [0s] - Polling finished
--- PASS: TestTermsRequired_ListKafkaTermsRequired (6.15s)
=== RUN   TestServiceAccounts_GetByClientID
--- PASS: TestServiceAccounts_GetByClientID (5.16s)
=== RUN   TestServiceAccounts_Success
--- PASS: TestServiceAccounts_Success (5.87s)
=== RUN   TestServiceAccounts_IncorrectOCMIssuer_AuthzFailure
--- PASS: TestServiceAccounts_IncorrectOCMIssuer_AuthzFailure (1.71s)
=== RUN   TestServiceAccounts_CorrectTokenIssuer_AuthzSuccess
--- PASS: TestServiceAccounts_CorrectTokenIssuer_AuthzSuccess (2.93s)
=== RUN   TestServiceAccounts_InputValidation
--- PASS: TestServiceAccounts_InputValidation (6.92s)
=== RUN   TestServiceAccounts_SsoProvider_MAS_SSO
--- PASS: TestServiceAccounts_SsoProvider_MAS_SSO (1.34s)
=== RUN   TestServiceAccounts_SsoProvider_SSO
--- PASS: TestServiceAccounts_SsoProvider_SSO (1.51s)
PASS
ok  	github.com/bf2fc6cc711aee1a0c2a/kas-fleet-manager/internal/kafka/test/integration	5584.167s

@machi1990 - I have looked at the code. it looks good to me, but would like other reviewers to comment/ approve as well

@miguelsorianod can you've another look at this? I'll like to get this in to progress with the work on https://issues.redhat.com/browse/MGDSTRM-3363

I've reworked the configuration.
The current structure of the configuration: https://github.com/bf2fc6cc711aee1a0c2a/kas-fleet-manager/blob/708a1e134837be1dbb41061698652caa88ff514c/config/dynamic-scaling-configuration.yaml#L1-L59
@miguelsorianod can you have a look at this? Thanks

I've reworked the configuration. The current structure of the configuration:
https://github.com/bf2fc6cc711aee1a0c2a/kas-fleet-manager/blob/708a1e134837be1dbb41061698652caa88ff514c/config/dynamic-scaling-configuration.yaml#L1-L59
@miguelsorianod can you have a look at this? Thanks

@miguelsorianod friendly ping ;-)

Thanks for this round of review @miguelsorianod
I've addressed the comments. Can you have another look?

Thanks @miguelsorianod and @pawelpaszki for the review. Let's get this one in.