Successful response validation actually returns 'skipped'?

When turning on the validator, I noticed the following debug output:
swagger-tools:middleware:validator     Validation: skipped (No response definition) 500 +0ms
However, I do have a response defined for 500. So looking through swagger-validator.js, I'm wondering if the code below is incorrect (lines 281-290 in swagger-validator.js):
  if (_.isUndefined(schema)) {
    sendData(swaggerVersion, res, val, encoding, true);
  } else {
    validateValue(req, schema, vPath, val, 'body', function (err) {
      if (err) {
        throw err;
      }
      sendData(swaggerVersion, res, val, encoding, false);
    });
  }

If err isn't thrown, shouldn't the last parameter to the subsequent sendData call be 'true'?

Oh yeah, did a debug print of 'schema', and it's an honest-to-goodness object. 😉


Custom msbuild sdk results in "Package restore was successful but a package with the ID of "xyz" was not installed"

Steps to reproduce
I am trying to build a custom msbuild sdk, and have published this successfully to a TFS 2018Update3 PackageManagement feed.
However I run into the issue that upon first time use of a new version of this sdk I get a "Package restore was successful but a package with the ID of "xxx" was not installed" error.
I can confirm that the NuGetSDKResolver has downloaded the nuget correctly and placed it in the global nuget cache with the correct version.
If I run the test project through msbuild a second time it succeeds to import sdk.props/targets from the custom sdk.
I have also tried the same with a simple example and "Microsoft.Build.CentralPackageVersions" and this always goes ok on the first try (so I can not replicate the issue with that NuGet sdk).
I have tried comparing contents of these Sdks (and nuget spec files) to see if I can understand why it is going wrong, but I cannot figure out what the exact cause is.
Any information on how to further analysis & fix this issue?
Project file sample2.proj goes wrong (with custom SDK)
  <Sdk Name="xyz" Version="0.3.32-gb626b61888"/>
  <PropertyGroup>
    <TargetFramework>netstandard2.0</TargetFramework>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Newtonsoft.Json" />
  </ItemGroup>
</Project>

Project sample.proj goes ok (with Microsoft.Build.CentralPackageVersions)
  <Sdk Name="Microsoft.Build.CentralPackageVersions" Version="2.0.1" />

  <PropertyGroup>
    <TargetFramework>netstandard2.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Newtonsoft.Json" />
  </ItemGroup>
</Project>

The custom SDK has the following relevant MSBuild information in the nuspec:
<tags>MSBuild MSBuildSdk</tags>
<packageTypes>
      <packageType name="MSBuildSdk" />
</packageTypes>

Environment data
msbuild /version output:
Microsoft (R) Build Engine version 15.8.166+gd4e8d81a88 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.
<IP_ADDRESS>604
OS info:
Windows 10 1703 (15063.1387)

After restore, the SDK resolver looks for your SDK based on the restore results:
https://github.com/NuGet/NuGet.Client/blob/dev/src/NuGet.Core/Microsoft.Build.NuGetSdkResolver/NuGetSdkResolver.cs#L150
Is the version and ID of the package exactly the same?  Maybe the case is different?  Can you attach a NUPKG of an SDK that doesn't work?

Yes indeed, casing is the issue here.
I can reproduce the problem by:

remove C:\Users\japj.nuget\packages\microsoft.build.centralpackageversions
run msbuild test.proj with the following content in test.proj:

<Project>
  <Sdk Name="Microsoft.Build.CENTRALPACKAGEVERSIONS" Version="2.0.1" />
</Project>

Thanks for the quick pointer, not sure if there is still an issue that needs to be solved though.
I would have expected it to either "always fail" or "always succeed".
Now it fails the first time, but succeeds after it has previously been downloaded and you run it the 2nd time.

I think it succeeds the second time because its already downloaded and there's different logic for locating an on disk package versus one that was just fetched.
I think the fix is to just add a StringComparison to that Equals() call so that its not case sensitive.  I'm going to move this issue to the NuGet repo since thats where the NuGetSdkResolver lives now.
Thanks for reporting this!

This issue was moved to NuGet/Home#7817


Implement semantic versioning.

Hight time we do this :D
I'm thinking of doing releases regularly, using github's Release-feature for this seems to be the most straightforward way.
@leiserfg WDYT, any ideas to make this as easy as possible?

Maybe something like this is helpful https://github.com/semantic-release/semantic-release

(At least I think I'll start out with that, maybe it becomes annoying as time goes on :D)

There also is https://github.com/marketplace/actions/release-please-action.

Oh, the release-PR seems like a nice way to keep track of changes.. I'll definitely keep that in mind :+1:


Save cloud not able to get an debug info in case, when test suite name contain whitespaces

That's not working for case when suite name have whitespace

May be fixed by #453 ?

Yes, tested, #453 fixed it


Add :remote_address_reader PlugContext option

When working on masking user IP from Sentry at work, after scrubbing the IP from headers I noticed that user's IP is also visible under Environment's REMOTE_ADDR. One way to hide/mask the IP I thought about was to have additional plugs before and after PlugContext, to change and re-set the conn.remote_ip value. But I think a better solution would be to make reading of this value configurable.
When working on the patch I noticed the logic on the master branch has changed to read the x-forwarded-for header when present, so I made this the new default remote address reader.

@mitchellhenke bump


Implement device.get_mac_address()

Connects-to: https://github.com/balena-io/balena-sdk/pull/914
Change-type: minor
Signed-off-by: Trong Nghia Nguyen<EMAIL_ADDRESS>

HQ: https://github.com/balena-io/balena-io/issues/2195


Chapter 2 - Hodor

Hodor looked at them bellowing, "which road you should be home."
I think a special case needs handled here ;-D
Awesome job

Haha, maybe it was a flashback scene from before Bran traveled in time.


Add integration tests

Blocked by #8, as mocking spring security is a little bit tricky before Spring Security 4


Include parsed to string instead of array when a single include is used

Request URL like GET "/my-resource?include%5B%5D=something" which decoded is GET "/my-resource?include[]=something" is being parsed to:
{
  "include": "something"
}

by Lux, which causes an error because it expects an array of strings.
When more include's are requested like GET "/my-resource?include[]=something&include[]=else" the include qp is correctly parsed to an array.

I've tracked this down to node's url.parse method which generates query: { 'include[]': 'myModel' } when a single include is used and query: { 'include[]': [ 'myModel', 'myOtherModel' } when multiple includes are used. Lux doesn't do anything with the brackets next to the include (apart from stripping them) which is why this isn't working correctly. I think we should detect the type based on those brackets instead of relying on url.parse handling for arrays like this.

This might not be a problem after all.... I had mistakenly interpreted include wrong in my client. It is a comma separated string instead of an array. So this fix might not be necessary!
cc @zacharygolba

Your right @nickschot. The way Lux is parsing the query string is correct. See #640 for more information.


Error Normalizing train_features

This Error is displayed when running code on line 298 of 01-data-preparation.R
ERROR: The sample data set for the analysis has no variables.
Caught exception in file: CxAnalysis.cpp, line: 3756. ThreadID: 12080 Rethrowing.
Caught exception in file: CxAnalysis.cpp, line: 5249. ThreadID: 12080 Rethrowing.
Error in doTryCatch(return(expr), name, parentenv, handler) :
ERROR: The sample data set for the analysis has no variables.
I suppose this error occurs due to the NULL value on the sd features for id=1 . An i really don't know how to remove this NULL values. I kindly need your help/assistance in removing this NULL values if that is the problem leading to this Error. Else what could be the reason for it to lead to an Error while everything works out for the normalization of test_features? And also in the test_features after being Normalized the columns setting3, s1, a1 and sd1 are missing.

Although this is an old issue, please note that this repo is for samples only. If you have issues with RServices, please use our forums.


Always use the latest Flutter available

Right now cirrusci/flutter:latest tag points to the latest released version of Flutter. For example, 0.2.0 right now. To make sure we always test agains the latest Flutter available I've added a script to always run flutter upgrade.
Another possibility will be to incorporate Cirrus CI's Container Builder into https://github.com/flutter/flutter so successful master builds will publish a Docker container.
to: @Hixie @yjbanov

Thanks!


gRPC call hanging even with timeout set

We are looking to implement timeouts on every gRPC call we are making to the GoogleAds API. Following the documentation, we are now using the callable endpoints like in the snippet bellow :
                response =
                        googleAdsServiceClient.searchPagedCallable()
                                .call(
                                        SearchGoogleAdsRequest.newBuilder()
                                                .setQuery(query)
                                                .setCustomerId(Long.toString(loginCustomerId))
                                                .build(),
                                        GrpcCallContext.createDefault()
                                                .withTimeout(Duration.ofMillis(30000)));

However, it seems that when the refresh token is invalid, the call hangs for much longer than the timeout. Is there a way to enforce the timeout in that case or maybe test it with another call? This is problematic because it locks a thread for too long and compromise the server ability to answer other calls.

One simple way to do this is, before you make any Google API request, refresh the Credentials on your GoogleAdsClient.
For example, if you have a GoogleAdsClient instance named googleAdsClient, put this before your call to searchPagedCallable():
    ((com.google.auth.oauth2.OAuth2Credentials)googleAdsClient.getCredentials()).refreshIfExpired();

That will throw a com.google.auth.oauth2.GoogleAuthException if the refresh token is invalid, like this:
Exception in thread "main" com.google.auth.oauth2.GoogleAuthException: com.google.api.client.http.HttpResponseException: 400 Bad Request
POST https://oauth2.googleapis.com/token
{
  "error": "invalid_grant",
  "error_description": "Token has been expired or revoked."
}

The difference with this approach is that you are explicitly verifying the refresh token is valid instead of relying on the GoogleAdsServiceClient to do so.
Hope that helps!
Josh, Google Ads API Team


Torch distribution not found

Thank you for your great work!
I am getting the following error when trying to install the package
"ERROR: No matching distribution found for torch==1.1.0 (from finbert-embedding==0.1.4)"
I already installed the pytorch package version 1.1.0. Do you have any ideas on how to proceed?
Thanks!

There were two versions of pytorch installed on my system. Uninstalling the other version,  fixed the problem.


[WIP] cdc: add metrics

Signed-off-by: 5kbpers<EMAIL_ADDRESS>
What problem does this PR solve?
Issue Number: close #xxx
Problem Summary:
What is changed and how it works?
Proposal: xxx
What's Changed:
Related changes

PR to update pingcap/docs/pingcap/docs-cn:
PR to update pingcap/tidb-ansible:
Need to cherry-pick to the release branch

Check List
Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression

Consumes more CPU
Consumes more MEM

Breaking backward compatibility

Release note

/release

/release

/release

/release

/release

/test

/release

/release

/run-build-image


Adding callbacks on validation end, test end and option to log by step or epoch

Create data monitor base

[ ] Callback base for vision
[ ] Callback base for language

Extend callbacks base

[ ] Adding hook on validation end
[ ] Adding hook on test end
[ ] Adding hook on (train, validation) epoch end

#19


Ctrl+Backspace in terminal window removes one character at a time instead of whole words

From @clboles on July 28, 2017 14:0
Copied from original issue: dgriffen/WhackWhackTerminal#11

Scheduling this for after I achieve parity with VSCode features since the terminal in VSCode currently does not implement this.

From @Tyriar on August 3, 2017 5:47
This may not be possible reliably on Windows, some more details here: https://github.com/Microsoft/vscode/issues/30276

Is there plan to fix this issue on windows?

Unfortunately I don't think this can be fixed, at least not on my end. xterm.js is not able to handle the key combo on windows and the same issue affects vscode today.

I think the problem here is within https://github.com/rprichard/winpty not doing the right thing on ctrl+backspace.


TAJO-1260: Add ALTER TABLE ADD/DROP PARTITION statement to parser

I plan to implement ALTER TABLE ADD/DROP PARTITION statement stage by stage. For the first time, I'll add the statement to parser.

Syntax

ALTER TABLE table_name [IF NOT EXISTS] ADD PARTITION (key1 = 'val2', key2 = 'val2', ...) [LOCATION 'partition_path_of_directory'];
ALTER TABLE table_name [IF NOT EXISTS] DROP PARTITION (key1 = 'val2', key2 = 'val2', ...);

Hi @hyunsik
Thanks for your quick review. I used FileUtil.readTextFileFromResource instead of direct paths and I added more unit test cases.

Could you trigger the travisCI job?

Hi @hyunsik
I've just merged the master branch to TAJO-1260 branch.

+1
The patch looks good to me.
BTW, you need to consider inequality expression in DROP PARTITION. But, the current patch is good starting point. We can improve it later in another jira.
ALTER TABLE table_name [IF EXISTS] DROP PARTITION  (key1 [=|<|<=|>|>=|!=] 'val1', key2 ...,);

ALTER TABLE .. [ADD | DROP] PARTITION statement will cause error if this statement is given in JDBC driver or tsql. So, could you add some exception handling code to prevent ugly errors?

Hi @hyunsik
I added the exception handling code. If possible, could you check the pathch again?

Sure, I'll review it soon.

@blrunner Could you trigger TravisCI?

@hyunsik
Sure, I've just triggered it. :)

+1 ship it. Thanks.
It should be committed to only master branch.


Fix check-rules-duplicates recipe indentation

Makefile recipes should be indented with a leading tab, this target had
tabs vs spaces in the wrong order.

👍

:+1:

@scop Thanks for your contribution.


[docs] Best Practices and Guidelines ("This / Not This") examples improvements

Break up our "This / Not This" best practices and guidelines images and examples into separate images and columns (or cards), similar to what we do on our Brand site.
This will make it easier to scan as well as improve non-visual accessibility.
"This" and "Not this" should be HTML text paired with an icon.
Current situation

What we're looking for

The Color page's Best Practices section uses this format.
This is a sample of what the Dos and Don'ts currently look like in code:

Do

Use the color variants already available for elements and patterns.

Don't

Do not create your own colors, design tokens, or change the colors of existing elements and patterns.

It would be good to add this format as a shortcode.

Also, is there a preference for using "This"/"Not this" vs "Do"/"Don't"?

Also, is there a preference for using "This"/"Not this" vs "Do"/"Don't"?

@marionnegp not really. I'm ok with using Do/Don't instead. Might be more of a common practice anyway.

I did some initial work on this in #1465
https://deploy-preview-1465--red-hat-design-system.netlify.app/foundations/color/usage/#custom-themes
The variation looks a bit different then the examples above as I used a
 For the image.  It seemed appropriate given the linked description of the images as the

Here is what the HTML looks like:
<uxdot-best-practice>
  <uxdot-example slot="do" no-border="" variant="full" alignment="left" color-palette="lightest">

Use the color variants already available for elements and patterns.

  </uxdot-example>
  <uxdot-example slot="dont" no-border="" variant="full" alignment="left" color-palette="lightest">

Do not create your own colors, design tokens, or change the colors of existing elements and patterns.

  </uxdot-example>
</uxdot-best-practice>

This component can be tweaked however we deem necessary as I was just evaluating using a slotted
 in a uxdot-example for this use case.
Right now <uxdot-best-practice> has 2 slots do and dont and currently only only displays in a 2 up grid as that is the only use case I've ran across so far.

@zeroedin, using
 makes sense to me, though we might want to adjust the placement of the caption or "Do/Don't" heading.
There might be times we want to add more than one "do" or "don't" image (like if we split the two images in this audio player example) or have a best practice example be full-width.

<uxdot-best-practice do> and <uxdot-best-practice dont> which use a figcaption in the shadowroot.  It is available in #1465.  This PR does not update any content.
I believe this issue partially an ask to update all Best Practice sections across the site to use this new style as well so will leave it open.

@zeroedin I like the design below if we can change the old icons for new brand icons as well as adding padding in between the icon and text.
How do we move this forward?


Release Vumi 0.6.9

We need to:

[ ] bump the version number
[x] update the changelog
[x] add bumping the dockerfile version to the version bumper.

Ready for review (although it can only land once #1049 has been merged in).

:+1:


BREAKING CHANGE: xSQLServerMaxDop cannot be used on more than one instance on the same server

I suggest we change xSQLServerMaxDop so it also uses SQLInstanceName as a key to make it possible to use the resource on more than one instance on the same server.
This is being worked on in PR #64

Hi @johlju. PR #64 has been merged. I think this can be closed.

This has been resolved in PR #64 which has been merged.


"Crash" issues - clean-up and next steps

Coming from this discussion.

[ ] Only open a "crash" issue after it happens more than 10 times
[ ] For existing crashes logged, update the reopen logic to see if the crash count is less than 10 before reopening
[ ] Remove the $500 price cap on crash issues
[ ] Implement source maps to aid with investigating these issues (GH)
[ ] Close the loop on people running staging builds locally sending false positives (GH)
[ ] Improve the documentation for how to approach these issues, and link the readme in the crash issue creation template

CC: @AndrewGable @JmillsExpensify @michaelhaxhiu @roryabraham

Should we throw a Bug label on this, on someone assign themself? I'm thinking yes since some of those todos are BZ tasks.

I don't think it's a bug really, it's a process improvement.
I'll assign myself anyway and take on this one:

Remove the $500 price cap on crash issues

Ok cool. Mainly I think someone from BZ should assign themselves.

There's already an issue for the source maps https://github.com/Expensify/App/issues/9293
Not sure if we want to close that out or remove that part from this isssue.
FWIW I think adding the source map should be the first step, because without it, some crashes are impossible to diagnose.

It's linked to this issue in the OP.

Focused on the source maps as the first priority for this, then will get to this.

Closing per https://github.com/Expensify/Expensify/issues/243479, tl;dr we are turning off Firebase crashes for the time being.


GO:0080008 Cul4-RING ubiquitin ligase complex and children, update term/def

Could E3 be added to
GO:0080008 Cul4-RING ubiquitin ligase complex
and its children?
Thanks
Val
Reported by: ValWood
Original Ticket: geneontology/ontology-requests/10695

Fixed ages ago, but ticket never closed.  Closing now.


ui-select is not compatible with angular-loader.js

I use the angular-seed where I load some more dependencies with $script as follow:
$script([
    'https://ajax.googleapis.com/ajax/libs/angularjs/1.4.7/angular.min.js',
    'https://ajax.googleapis.com/ajax/libs/angularjs/1.4.7/angular-route.min.js',
    'https://ajax.googleapis.com/ajax/libs/angularjs/1.4.7/angular-resource.min.js',
    'https://ajax.googleapis.com/ajax/libs/angularjs/1.4.7/angular-animate.min.js',
    'https://ajax.googleapis.com/ajax/libs/angularjs/1.4.7/angular-sanitize.min.js',
    'https://cdnjs.cloudflare.com/ajax/libs/angular-ui-select/0.13.2/select.js',
    'js/app.js'
], function() {
...

I think ui-select tries to call angular.element.prototype.querySelectorAll before angular has been loaded. Does it sounds right?
Here is the console error from Chrome (46)
Uncaught TypeError: Cannot read property 'prototype' of undefined
    (anonymous function) @ select.js:70
    (anonymous function) @ select.js:1913

Can you post a reproduction in Plunker?

Closing as a usage issue. The library works fine when angular.js is included via <script> any other loading methods aren't really up to us to support. PRs welcome as always.

Is there any more insight besides including angular.js in the script tag? Like I get that you don't support more than that, but is there anything i can do to get this to play nice with webpack? I get the exact same error after following a variety of different tutorials. I also know this is well over a year old, but i can't find anything to solve


Add warning session is expiring

This package is focused around always keeping forms alive, a bit of a security problem imo that's why the whole expiry thing is default in Laravel.
It would be nice to have an option where the user is prompted (via a model alert) when the session is about to expire and give them a button to stay active (essentially sending the 'drip'). This is a similar same way the banking industry handle form/session expiration.
The way the alert is shown to the user of course will be subjective and need to be styled per app, but if caffeine made it relatively generic and exposed some options or api of events to easily show a custom alert it would be nice.
LaravelCaffeine.start({
	alertText: 'Your session is about to expire.',
	btnText: 'Click to Continue'
});

LaravelCaffeine.expiring(function(drip) {
	// Show custom model and call 'drip' if button was pressed.
});

Hi @garygreen, yes, I am familiar with that approach as well, and thought about something similar while developing this package. However, it's a slightly different use case, and as such don't feel it should be included in this package.
I have had need for the session-count-down option as well though, so perhaps I'll create a package for that sometime down the road if I come across a project that calls for it again.
Thanks for your feedback :-)


Container always restarts

Installed it by the command

docker run  -d
--publish 8080:8080
--restart always
--volume ~/magic_mirror/config:/opt/magic_mirror/config
--volume ~/magic_mirror/modules:/opt/magic_mirror/modules
--name magic_mirror
bastilimbach/docker-magicmirror

Modified the port 80 to 8080 because I've already have some listens on 80.
Actually it is restarting every time with the following message in docker logs magic_mirror

exec user process caused "exec format error"

Docker info:
Containers: 2
 Running: 1
 Paused: 0
 Stopped: 1
Images: 2
Server Version: 18.09.0
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.59-v7+
Operating System: Raspbian GNU/Linux 9 (stretch)
OSType: linux
Architecture: armv7l
CPUs: 4
Total Memory: 927.3MiB
Name: WohnzimmerPi3
ID: TQ2P:VVP7:KNUD:EZ5Y:EUOF:O5EZ:FPLF:3HOT:A4VT:VHBI:WI43:LM7K
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 <IP_ADDRESS>/8
Live Restore Enabled: false
Product License: Community Engine

WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No oom kill disable support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support

Need to pull the pi image as follows:

docker run -d
--publish 8080:8080
--restart always
--volume ~/magic_mirror/config:/opt/magic_mirror/config
--volume ~/magic_mirror/modules:/opt/magic_mirror/modules
--name magic_mirror
bastilimbach/docker-magicmirror:raspberry

Need to pull the pi image as follows:

docker run -d
--publish 8080:8080
--restart always
--volume ~/magic_mirror/config:/opt/magic_mirror/config
--volume ~/magic_mirror/modules:/opt/magic_mirror/modules
--name magic_mirror
bastilimbach/docker-magicmirror:raspberry

Note: It looks like this image is way behind the github repo so not sure you should use it other than testing

Yeah, thank you! Just saw that too, that I use a false image. The age of this is not really problematic, cause I can update mm. Thank you!


Add visual tree verification baseline

Added tests to verify controls' visual trees and captured current content as the baseline.
Failed visual tree tests will generate a new master file and it'll be included in HelixTestFailureDumps.html (use to be crashdumps.html) in the artifacts drop folder. HelixTestFailureDumps.html now contains screenshots, crash dumps, and new visual tree master files when they are available (screenshot below).

What criteria was used to determine which controls would get masters?

What criteria was used to determine which controls would get masters?

We brought in a lot of templates from OS repo for rounded corner and shadow changes, but the code and tests of those controls are still in OS, that means we have 0 test coverage for them in MUX. So I'm targeting those controls first.
I've also randomly added some other controls with complicated visuals or might be impacted by rounded corners. But I think ideally we should have master file for every control in our repo @jevansaks ?

This is ready to be re-reviewed now. Latest changes in the PR description @jevansaks @kmahone.

Param(

Re-name this to something that doesn't refer only to dumps.
Something like CreateLinksToHelixFiles or something like that.

Refers to: build/Helix/CreateHelixTestFailureDumps.ps1:1 in 0584c50. [](commit_id = 0584c50efd83e3a805b43b85818e09f0ed2f3a8d, deletion_comment = False)

[string]$AccessToken = $env:SYSTEM_ACCESSTOKEN,

Is this script working?
If I go to the build:
https://dev.azure.com/ms/microsoft-ui-xaml/_build/results?buildId=37541
and look at the html file, it is empty.

Refers to: build/Helix/CreateHelixTestFailureDumps.ps1:2 in 0584c50. [](commit_id = 0584c50efd83e3a805b43b85818e09f0ed2f3a8d, deletion_comment = False)

[string]$AccessToken = $env:SYSTEM_ACCESSTOKEN,

Is this script working?
If I go to the build:
https://dev.azure.com/ms/microsoft-ui-xaml/_build/results?buildId=37541
and look at the html file, it is empty.

If it's a successful run then there won't be anything. This one had some actual failures, https://dev.azure.com/ms/microsoft-ui-xaml/_build/results?buildId=37518.
We probably shouldn't create the html file at all if the run is successful, I'll update.


Unclutter TQVaultAE.GUI

It was messy

Thanks for this, the project is quite historical and contribution guidelines have always been lacking!
Since this is significant, I'll wait until we figure out the mew expansion to take this PR in, as not to mess up everyone working on the current code.

Yes i agree ! Expansion first 👍

We could go ahead with this

Any problem with the review ?

There are conflicts in this PR. could you please solve them?

Should be OK

Thanks for fixing the conflicts, I'll review over the course of next week.

if you finaly consider merging #133 it should be done before this one (to avoid headache) :)

Merge is Ok


fold questions

Leaving this for reference, feel free to close; it should not be too much work: an adaption of vim-markdown could fold every question together with its answer for easier navigation through long AI chats. (Navigation without folds has been taken care of here.)

Okay, sorry for the noise. The optional setup is now in the wiki


Support multiple languages

Please support ability to use custom dictionaries with bad words for different languages or add more languages

oh very interesting, let me add this feature

Done, adding new feature to add and remove dictionary https://github.com/jojoee/leo-profanity/releases/tag/v1.7.0


Re-Creating node from scratch does not copy tables for the Postgres and Kafka engines

We use your Operator to manage Clickhouse cluster. Thank you.
After some hardware failure we reset PVC (and zookeeper namespace) to re-create one clickhouse node.
Most of metadata like views, materialized views and tables with most engines (MergeTree, ReplicatedMergeTree etc.) was successfully re-created on the node and replication was started.
Meantime none of Postgres and Kafka based engines tables was recreated.
Is it a bug, or we need to use some commands or hacks to sync all metadata across the cluster?

@Hubbitus , have you used latest 0.23.6 or earlier release?

That was in older version. Now we have updated operator. What is a correct way to re-init node? Is it enough to just delete PVC of failed node and delete POD?

@Hubbitus , if you want to re-init the existing node, delete STS, PVC, PV and start re-concile. Do you have multiple replicas?

@alex-zaitsev, thank you for the reply.
I understand how to delete objects, but how to "start re-concile"?
I have two replicas chi-gid-gid-0-0-0 and chi-gid-gid-0-1-0. And now chi-gid-gid-0-0-0 is misfunction. I want to re-init it from the data in chi-gid-gid-0-1-0. And that should include sync all:

metadata (all type of objects like MergeTree tables, Postgres, kafka engines, materialized views, etc)
populate it with data from replica 1
Users and all permissions to the objects

@Hubbitus , we have released 0.23.7 that is more aggressive re-creating the schema. So you may try to delete PVC/PV completely, and let it to re-create the objects.

@Hubbitus, thank you very much!
Eventually I get it updated for our cluster.
And doing in ArgoCD:

Deleted PVC default-volume-claim-chi-gid-gid-0-0-0
Deleted pod chi-gid-gid-0-0-0

Then PVC had been re-created.
I see pod is up and running.

But there are a lot of errors like 2024.09.04 23:50:34.382651 [ 712 ] {} <Error> Access(user directories): from: <IP_ADDRESS>, user: data_quality: Authentication failed: Code: 192. DB::Exception: There is no user data_quality in local_directory. (UNKNOWN_USER).... So, users are not copied
Tables looks like also not synced:

SELECT hostname() as node, COUNT(*)
FROM clusterAllReplicas('{cluster}', system.tables)
WHERE database NOT IN ('INFORMATION_SCHEMA', 'information_schema', 'system')
GROUP BY node

node
count()

chi-gid-gid-0-1-0
620

And also error in log like: 2024.09.04 23:52:49.039132 [ 714 ] {bb628508-db8e-4cf9-8307-a13133a185c9} <Error> PredefinedQueryHandler: Code: 60. DB::Exception: Table system.operator_compatible_metrics does not exist. (UNKNOWN_TABLE) - so even in system database some tables missing...
So, I see only tables in information_schema for the 1-st node.

Notes:

Users are not replicated by operator since it can not access sensitive data (like passwords). Use CHI/XML user management or replicated user directory.

<clickhouse>
  <user_directories replace="replace">
    <users_xml>
      <path>/etc/clickhouse-server/users.xml</path>
    </users_xml>
    <replicated>
      <zookeeper_path>/clickhouse/access/</zookeeper_path>
    </replicated>
    <local_directory>
       <path>/var/lib/clickhouse/access/</path>
    </local_directory>
  </user_directories>
</clickhouse>

Note, the order is important, but local_directory may be skipped if you are not using it. Keep it, if there are users defined with CREATE USER already, otherwise they disappear at all.

Tables in system database are not replicated as well, since it is supposed there are no user tables in there.

Others should work, so operator log is needed to check what went wrong.
The correct PVC recovery sequence is:

Delete PVC (or PVC and STS)
Run reconcile adding taskID to CHI, for instance

Today I eventfully receive rights to see operator pod in kube-system namespace.
And just after deletion of PVC and pod I see errors in clickhouse-operator pod:
I0921 22:13:23.555553       1 worker.go:275] processReconcilePod():gidplatform-dev/chi-gid-gid-0-0-0:Delete Pod. gidplatform-dev/chi-gid-gid-0-0-0
I0921 22:13:23.686901       1 worker.go:266] processReconcilePod():gidplatform-dev/chi-gid-gid-0-0-0:Add Pod. gidplatform-dev/chi-gid-gid-0-0-0
I0921 22:13:32.391425       1 chi.go:38] prepareListOfTemplates():gidplatform-dev/gid:Found applicable templates num: 0
I0921 22:13:32.391446       1 chi.go:82] ApplyCHITemplates():gidplatform-dev/gid:Applied templates num: 0
E0921 22:13:32.394908       1 connection.go:194] Exec():FAILED Exec(http://test_operator:***@chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local:8123/) doRequest: transport failed to send a request to ClickHouse: dial tcp <IP_ADDRESS>:8123: connect: connection refused for
SQL: SYSTEM DROP DNS CACHE
W0921 22:13:32.394938       1 retry.go:52] exec():chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local:FAILED single try. No retries will be made for Applying sqls
I0921 22:13:32.414341       1 chi.go:38] prepareListOfTemplates():gidplatform-dev/gid:Found applicable templates num: 0
I0921 22:13:32.414363       1 chi.go:82] ApplyCHITemplates():gidplatform-dev/gid:Applied templates num: 0
I0921 22:13:32.415447       1 worker.go:387] gidplatform-dev/gid/b22b39fe-b7d8-40e3-a510-e169d1ffab18:updating endpoints for CHI-1 gid
I0921 22:13:32.450485       1 worker.go:389] gidplatform-dev/gid/b22b39fe-b7d8-40e3-a510-e169d1ffab18:IPs of the CHI-1 update endpoints gidplatform-dev/gid: len: 2 [<IP_ADDRESS> <IP_ADDRESS>]
I0921 22:13:32.464127       1 chi.go:38] prepareListOfTemplates():gidplatform-dev/gid:Found applicable templates num: 0
I0921 22:13:32.464172       1 chi.go:82] ApplyCHITemplates():gidplatform-dev/gid:Applied templates num: 0
I0921 22:13:32.466517       1 worker.go:393] gidplatform-dev/gid/f2584b3a-a25a-4f22-8dfd-72f2a5166984:Update users IPS-1
I0921 22:13:32.481724       1 worker.go:1315] updateConfigMap():gidplatform-dev/gid/f2584b3a-a25a-4f22-8dfd-72f2a5166984:Update ConfigMap gidplatform-dev/chi-gid-common-usersd
I0921 22:13:42.168333       1 chi.go:38] prepareListOfTemplates():gidplatform-dev/gid:Found applicable templates num: 0
I0921 22:13:42.168355       1 chi.go:82] ApplyCHITemplates():gidplatform-dev/gid:Applied templates num: 0
I0921 22:13:42.190633       1 chi.go:38] prepareListOfTemplates():gidplatform-dev/gid:Found applicable templates num: 0
I0921 22:13:42.190651       1 chi.go:82] ApplyCHITemplates():gidplatform-dev/gid:Applied templates num: 0
I0921 22:13:42.191751       1 worker.go:387] gidplatform-dev/gid/ef8a0da7-09d3-4890-9a59-c760233aedb5:updating endpoints for CHI-1 gid
I0921 22:13:42.215106       1 worker.go:389] gidplatform-dev/gid/ef8a0da7-09d3-4890-9a59-c760233aedb5:IPs of the CHI-1 update endpoints gidplatform-dev/gid: len: 2 [<IP_ADDRESS> <IP_ADDRESS>]
I0921 22:13:42.224452       1 chi.go:38] prepareListOfTemplates():gidplatform-dev/gid:Found applicable templates num: 0
I0921 22:13:42.224470       1 chi.go:82] ApplyCHITemplates():gidplatform-dev/gid:Applied templates num: 0
I0921 22:13:42.225507       1 worker.go:393] gidplatform-dev/gid/d9105257-3cfe-4596-b3bf-0f6cd6935843:Update users IPS-1
I0921 22:13:42.235027       1 worker.go:1315] updateConfigMap():gidplatform-dev/gid/d9105257-3cfe-4596-b3bf-0f6cd6935843:Update ConfigMap gidplatform-dev/chi-gid-common-usersd

As we are speaking, I have tried to reconcile cluster by providing:
spec:
  taskID: "click-reconcile-1"

Indeed, that looks like triggering reconcile. Logs of operator pod:
kubectl -n kube-system logs --selector=app=clickhouse-operator --container=clickhouse-operator --tail=1000
I0929 11:54:59.076600       1 worker.go:574] ActionPlan start---------------------------------------------:
Diff start -------------------------
modified spec items num: 1
diff item [0]:'.TaskID' = '"click-reconcile-1"'
Diff end -------------------------

ActionPlan end---------------------------------------------
I0929 11:54:59.076655       1 worker-chi-reconciler.go:89] reconcileCHI():gidplatform-dev/gid/click-reconcile-1:ActionPlan has actions - continue reconcile
I0929 11:54:59.125555       1 worker.go:663] markReconcileStart():gidplatform-dev/gid/click-reconcile-1:reconcile started, task id: click-reconcile-1
I0929 11:54:59.681288       1 worker.go:820] FOUND host: ns:gidplatform-dev|chi:gid|clu:gid|sha:0|rep:0|host:0-0
I0929 11:54:59.681436       1 worker.go:820] FOUND host: ns:gidplatform-dev|chi:gid|clu:gid|sha:0|rep:1|host:0-1
I0929 11:54:59.681607       1 worker.go:844] RemoteServersGeneratorOptions: exclude hosts: [], attributes: status: , add: true, remove: false, modify: false, found: false, exclude: true
I0929 11:54:59.859367       1 worker.go:1315] updateConfigMap():gidplatform-dev/gid/click-reconcile-1:Update ConfigMap gidplatform-dev/chi-gid-common-configd
I0929 11:55:00.648852       1 worker.go:1315] updateConfigMap():gidplatform-dev/gid/click-reconcile-1:Update ConfigMap gidplatform-dev/chi-gid-common-usersd
I0929 11:55:01.284151       1 service.go:86] CreateServiceCluster():gidplatform-dev/gid/click-reconcile-1:gidplatform-dev/cluster-gid-gid
I0929 11:55:01.294688       1 worker-chi-reconciler.go:819] PDB updated: gidplatform-dev/gid-gid
I0929 11:55:01.294746       1 worker-chi-reconciler.go:554] not found ReconcileShardsAndHostsOptionsCtxKey, use empty opts
I0929 11:55:01.294769       1 worker-chi-reconciler.go:568] starting first shard separately
I0929 11:55:01.294967       1 cluster.go:84] Run query on: chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local of [chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local]
I0929 11:55:01.305993       1 worker-chi-reconciler.go:349] getHostClickHouseVersion():Get ClickHouse version on host: 0-0 version: <IP_ADDRESS>8
I0929 11:55:01.306072       1 worker-chi-reconciler.go:684] reconcileHost():Reconcile Host start. Host: 0-0 ClickHouse version running: <IP_ADDRESS>8
I0929 11:55:01.897135       1 worker.go:1565] getObjectStatusFromMetas():gidplatform-dev/chi-gid-gid-0-0:cur and new objects are equal based on object version label. Update of the object is not required. Object: gidplatform-dev/chi-gid-gid-0-0
I0929 11:55:01.897345       1 worker.go:1001] worker.go:1001:excludeHost():start:exclude host start
I0929 11:55:02.047624       1 worker.go:159] shouldForceRestartHost():Host restart is not required. Host: 0-0
I0929 11:55:02.047656       1 worker.go:1170] shouldExcludeHost():Host is the same, would not be updated, no need to exclude. Host/shard/cluster: 0/0/gid
I0929 11:55:02.047669       1 worker.go:1005] worker.go:1002:excludeHost():end:exclude host end
I0929 11:55:02.047693       1 worker.go:1020] worker.go:1020:completeQueries():start:complete queries start
I0929 11:55:02.047730       1 worker.go:1220] shouldWaitQueries():Will wait for queries to complete according to CHOp config 'reconcile.host.wait.queries' setting. Host is not yet in the cluster. Host/shard/cluster: 0/0/gid
I0929 11:55:02.047779       1 cluster.go:84] Run query on: chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local of [chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local]
I0929 11:55:02.087023       1 poller.go:138] Poll():gidplatform-dev/0-0:OK gidplatform-dev/0-0
I0929 11:55:02.087048       1 worker.go:1024] worker.go:1021:completeQueries():end:complete queries end
I0929 11:55:02.248789       1 worker.go:1315] updateConfigMap():gidplatform-dev/gid/click-reconcile-1:Update ConfigMap gidplatform-dev/chi-gid-deploy-confd-gid-0-0
I0929 11:55:02.884163       1 worker-chi-reconciler.go:716] reconcileHost():Reconcile PVCs and check possible data loss for host: 0-0
I0929 11:55:03.458635       1 worker-chi-reconciler.go:406] worker-chi-reconciler.go:406:reconcileHostStatefulSet():start:reconcile StatefulSet start
I0929 11:55:03.458764       1 cluster.go:84] Run query on: chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local of [chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local]
I0929 11:55:03.465752       1 worker-chi-reconciler.go:349] getHostClickHouseVersion():Get ClickHouse version on host: 0-0 version: <IP_ADDRESS>8
I0929 11:55:03.472628       1 worker-chi-reconciler.go:412] reconcileHostStatefulSet():Reconcile host: 0-0. ClickHouse version: <IP_ADDRESS>8
I0929 11:55:03.651853       1 worker.go:159] shouldForceRestartHost():Host restart is not required. Host: 0-0
I0929 11:55:03.651943       1 worker-chi-reconciler.go:425] reconcileHostStatefulSet():Reconcile host: 0-0. Reconcile StatefulSet
I0929 11:55:03.655273       1 worker.go:1565] getObjectStatusFromMetas():gidplatform-dev/chi-gid-gid-0-0:cur and new objects are equal based on object version label. Update of the object is not required. Object: gidplatform-dev/chi-gid-gid-0-0
I0929 11:55:04.097497       1 worker-chi-reconciler.go:445] worker-chi-reconciler.go:407:reconcileHostStatefulSet():end:reconcile StatefulSet end
I0929 11:55:04.654273       1 worker-chi-reconciler.go:900] reconcileService():gidplatform-dev/gid/click-reconcile-1:Service found: gidplatform-dev/chi-gid-gid-0-0. Will try to update
I0929 11:55:04.853666       1 worker.go:1459] updateService():gidplatform-dev/gid/click-reconcile-1:Update Service success: gidplatform-dev/chi-gid-gid-0-0
I0929 11:55:05.487521       1 worker-chi-reconciler.go:922] reconcileService():gidplatform-dev/gid/click-reconcile-1:Service reconcile successful: gidplatform-dev/chi-gid-gid-0-0
I0929 11:55:05.487592       1 worker-chi-reconciler.go:461] reconcileHostService():DONE Reconcile service of the host: 0-0
I0929 11:55:05.487682       1 cluster.go:84] Run query on: chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local of [chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local]
I0929 11:55:05.495665       1 worker-chi-reconciler.go:349] getHostClickHouseVersion():Get ClickHouse version on host: 0-0 version: <IP_ADDRESS>8
I0929 11:55:05.495739       1 poller.go:138] Poll():gidplatform-dev/0-0:OK gidplatform-dev/0-0
I0929 11:55:05.495824       1 worker-chi-reconciler.go:753] reconcileHost():Check host for ClickHouse availability before migrating tables. Host: 0-0 ClickHouse version running: <IP_ADDRESS>8
I0929 11:55:05.495957       1 worker.go:908] migrateTables():No need to add tables on host 0 to shard 0 in cluster gid
I0929 11:55:05.496005       1 worker.go:1057] includeHost():Include into cluster host 0 shard 0 cluster gid
I0929 11:55:05.496048       1 worker.go:1124] includeHostIntoClickHouseCluster():going to include host 0 shard 0 cluster gid
I0929 11:55:05.496070       1 worker.go:844] RemoteServersGeneratorOptions: exclude hosts: [], attributes: status: , add: true, remove: false, modify: false, found: false, exclude: true
I0929 11:55:05.648655       1 worker.go:1315] updateConfigMap():gidplatform-dev/gid/click-reconcile-1:Update ConfigMap gidplatform-dev/chi-gid-common-configd
I0929 11:55:06.449496       1 cluster.go:84] Run query on: chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local of [chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local]
I0929 11:55:06.463606       1 worker-chi-reconciler.go:349] getHostClickHouseVersion():Get ClickHouse version on host: 0-0 version: <IP_ADDRESS>8
I0929 11:55:06.463648       1 poller.go:138] Poll():gidplatform-dev/0-0:OK gidplatform-dev/0-0
I0929 11:55:06.463703       1 worker-chi-reconciler.go:776] reconcileHost():Reconcile Host completed. Host: 0-0 ClickHouse version running: <IP_ADDRESS>8
I0929 11:55:07.086061       1 worker-chi-reconciler.go:797] reconcileHost():[now: 2024-09-29 11:55:07.085979541 +0000 UTC m=+530555.182385088] ProgressHostsCompleted: 1 of 2
I0929 11:55:08.084486       1 worker-chi-reconciler.go:900] reconcileService():gidplatform-dev/gid/click-reconcile-1:Service found: gidplatform-dev/clickhouse-gid. Will try to update
I0929 11:55:08.253098       1 worker.go:1459] updateService():gidplatform-dev/gid/click-reconcile-1:Update Service success: gidplatform-dev/clickhouse-gid
I0929 11:55:08.883102       1 worker-chi-reconciler.go:922] reconcileService():gidplatform-dev/gid/click-reconcile-1:Service reconcile successful: gidplatform-dev/clickhouse-gid
I0929 11:55:08.883295       1 cluster.go:84] Run query on: chi-gid-gid-0-1.gidplatform-dev.svc.cluster.local of [chi-gid-gid-0-1.gidplatform-dev.svc.cluster.local]
I0929 11:55:08.889935       1 worker-chi-reconciler.go:349] getHostClickHouseVersion():Get ClickHouse version on host: 0-1 version: <IP_ADDRESS>8
I0929 11:55:08.890015       1 worker-chi-reconciler.go:684] reconcileHost():Reconcile Host start. Host: 0-1 ClickHouse version running: <IP_ADDRESS>8
I0929 11:55:09.524136       1 worker.go:1572] getObjectStatusFromMetas():gidplatform-dev/chi-gid-gid-0-1:cur and new objects ARE DIFFERENT based on object version label: Update of the object is required. Object: gidplatform-dev/chi-gid-gid-0-1
I0929 11:55:09.524219       1 worker.go:1001] worker.go:1001:excludeHost():start:exclude host start
I0929 11:55:09.647870       1 worker.go:159] shouldForceRestartHost():Host restart is not required. Host: 0-1
I0929 11:55:09.647935       1 worker.go:1177] shouldExcludeHost():Host should be excluded. Host/shard/cluster: 1/0/gid
I0929 11:55:09.647982       1 worker.go:1010] excludeHost():Exclude from cluster host 1 shard 0 cluster gid
I0929 11:55:10.090456       1 chi.go:38] prepareListOfTemplates():gidplatform-dev/gid/click-reconcile-1:Found applicable templates num: 0
I0929 11:55:10.090524       1 chi.go:82] ApplyCHITemplates():gidplatform-dev/gid/click-reconcile-1:Applied templates num: 0
I0929 11:55:10.132801       1 chi.go:38] prepareListOfTemplates():gidplatform-dev/gid/click-reconcile-1:Found applicable templates num: 0
I0929 11:55:10.132824       1 chi.go:82] ApplyCHITemplates():gidplatform-dev/gid/click-reconcile-1:Applied templates num: 0
I0929 11:55:10.134283       1 worker.go:387] gidplatform-dev/gid/click-reconcile-1:updating endpoints for CHI-1 gid
I0929 11:55:10.256392       1 worker.go:1099] excludeHostFromClickHouseCluster():going to exclude host 1 shard 0 cluster gid
I0929 11:55:10.256420       1 worker.go:844] RemoteServersGeneratorOptions: exclude hosts: [], attributes: status: , add: true, remove: false, modify: false, found: false, exclude: true
I0929 11:55:10.651725       1 worker.go:1315] updateConfigMap():gidplatform-dev/gid/click-reconcile-1:Update ConfigMap gidplatform-dev/chi-gid-common-configd
I0929 11:55:10.847886       1 worker.go:389] gidplatform-dev/gid/click-reconcile-1:IPs of the CHI-1 update endpoints gidplatform-dev/gid: len: 2 [<IP_ADDRESS> <IP_ADDRESS>]
I0929 11:55:10.859857       1 chi.go:38] prepareListOfTemplates():gidplatform-dev/gid/click-reconcile-1:Found applicable templates num: 0
I0929 11:55:10.859903       1 chi.go:82] ApplyCHITemplates():gidplatform-dev/gid/click-reconcile-1:Applied templates num: 0
I0929 11:55:10.862438       1 worker.go:393] gidplatform-dev/gid/click-reconcile-1:Update users IPS-1
I0929 11:55:11.249384       1 worker.go:1315] updateConfigMap():gidplatform-dev/gid/click-reconcile-1:Update ConfigMap gidplatform-dev/chi-gid-common-usersd
I0929 11:55:11.887237       1 worker.go:1203] shouldWaitExcludeHost():wait to exclude host fallback to operator's settings. host 1 shard 0 cluster gid
I0929 11:55:11.896425       1 schemer.go:134] IsHostInCluster():The host 0-1 is inside the cluster
I0929 11:55:16.902829       1 schemer.go:134] IsHostInCluster():The host 0-1 is inside the cluster
I0929 11:55:21.913913       1 schemer.go:134] IsHostInCluster():The host 0-1 is inside the cluster
I0929 11:55:26.921150       1 schemer.go:134] IsHostInCluster():The host 0-1 is inside the cluster
I0929 11:55:31.928701       1 schemer.go:134] IsHostInCluster():The host 0-1 is inside the cluster
I0929 11:55:36.936718       1 schemer.go:134] IsHostInCluster():The host 0-1 is inside the cluster
I0929 11:55:41.945459       1 schemer.go:134] IsHostInCluster():The host 0-1 is inside the cluster
I0929 11:55:46.954333       1 schemer.go:134] IsHostInCluster():The host 0-1 is inside the cluster
I0929 11:55:51.962841       1 schemer.go:134] IsHostInCluster():The host 0-1 is inside the cluster
I0929 11:55:56.971440       1 schemer.go:134] IsHostInCluster():The host 0-1 is inside the cluster
I0929 11:56:01.978083       1 schemer.go:134] IsHostInCluster():The host 0-1 is inside the cluster
I0929 11:56:06.984911       1 schemer.go:134] IsHostInCluster():The host 0-1 is inside the cluster
I0929 11:56:11.996098       1 schemer.go:134] IsHostInCluster():The host 0-1 is inside the cluster
I0929 11:56:11.996147       1 poller.go:170] Poll():gidplatform-dev/0-1:WAIT:gidplatform-dev/0-1
I0929 11:56:17.002241       1 schemer.go:134] IsHostInCluster():The host 0-1 is inside the cluster
I0929 11:56:17.002279       1 poller.go:170] Poll():gidplatform-dev/0-1:WAIT:gidplatform-dev/0-1
I0929 11:56:22.008717       1 schemer.go:134] IsHostInCluster():The host 0-1 is inside the cluster
I0929 11:56:22.008762       1 poller.go:170] Poll():gidplatform-dev/0-1:WAIT:gidplatform-dev/0-1
I0929 11:56:27.015747       1 schemer.go:134] IsHostInCluster():The host 0-1 is inside the cluster
I0929 11:56:27.015810       1 poller.go:170] Poll():gidplatform-dev/0-1:WAIT:gidplatform-dev/0-1
I0929 11:56:32.024632       1 schemer.go:134] IsHostInCluster():The host 0-1 is inside the cluster
I0929 11:56:32.024713       1 poller.go:170] Poll():gidplatform-dev/0-1:WAIT:gidplatform-dev/0-1
I0929 11:56:37.037036       1 schemer.go:137] IsHostInCluster():The host 0-1 is outside of the cluster
I0929 11:56:37.037107       1 poller.go:138] Poll():gidplatform-dev/0-1:OK gidplatform-dev/0-1
I0929 11:56:37.037132       1 worker.go:1015] worker.go:1002:excludeHost():end:exclude host end
I0929 11:56:37.037189       1 worker.go:1020] worker.go:1020:completeQueries():start:complete queries start
I0929 11:56:37.037281       1 worker.go:1220] shouldWaitQueries():Will wait for queries to complete according to CHOp config 'reconcile.host.wait.queries' setting. Host is not yet in the cluster. Host/shard/cluster: 1/0/gid
I0929 11:56:37.037353       1 cluster.go:84] Run query on: chi-gid-gid-0-1.gidplatform-dev.svc.cluster.local of [chi-gid-gid-0-1.gidplatform-dev.svc.cluster.local]
I0929 11:56:37.041809       1 poller.go:138] Poll():gidplatform-dev/0-1:OK gidplatform-dev/0-1
I0929 11:56:37.041827       1 worker.go:1024] worker.go:1021:completeQueries():end:complete queries end
I0929 11:56:37.048773       1 worker.go:1315] updateConfigMap():gidplatform-dev/gid/click-reconcile-1:Update ConfigMap gidplatform-dev/chi-gid-deploy-confd-gid-0-1
I0929 11:56:37.098510       1 worker-chi-reconciler.go:716] reconcileHost():Reconcile PVCs and check possible data loss for host: 0-1
I0929 11:56:37.119348       1 worker-chi-reconciler.go:406] worker-chi-reconciler.go:406:reconcileHostStatefulSet():start:reconcile StatefulSet start
I0929 11:56:37.119427       1 cluster.go:84] Run query on: chi-gid-gid-0-1.gidplatform-dev.svc.cluster.local of [chi-gid-gid-0-1.gidplatform-dev.svc.cluster.local]
I0929 11:56:37.123489       1 worker-chi-reconciler.go:349] getHostClickHouseVersion():Get ClickHouse version on host: 0-1 version: <IP_ADDRESS>8
I0929 11:56:37.127378       1 worker-chi-reconciler.go:412] reconcileHostStatefulSet():Reconcile host: 0-1. ClickHouse version: <IP_ADDRESS>8
I0929 11:56:37.131620       1 worker.go:159] shouldForceRestartHost():Host restart is not required. Host: 0-1
I0929 11:56:37.131650       1 worker-chi-reconciler.go:425] reconcileHostStatefulSet():Reconcile host: 0-1. Reconcile StatefulSet
I0929 11:56:37.133351       1 worker.go:1565] getObjectStatusFromMetas():gidplatform-dev/chi-gid-gid-0-1:cur and new objects are equal based on object version label. Update of the object is not required. Object: gidplatform-dev/chi-gid-gid-0-1
I0929 11:56:37.168247       1 worker-chi-reconciler.go:445] worker-chi-reconciler.go:407:reconcileHostStatefulSet():end:reconcile StatefulSet end
I0929 11:56:37.653395       1 worker-chi-reconciler.go:900] reconcileService():gidplatform-dev/gid/click-reconcile-1:Service found: gidplatform-dev/chi-gid-gid-0-1. Will try to update
I0929 11:56:37.849923       1 worker.go:1459] updateService():gidplatform-dev/gid/click-reconcile-1:Update Service success: gidplatform-dev/chi-gid-gid-0-1
I0929 11:56:38.491295       1 worker-chi-reconciler.go:922] reconcileService():gidplatform-dev/gid/click-reconcile-1:Service reconcile successful: gidplatform-dev/chi-gid-gid-0-1
I0929 11:56:38.491349       1 worker-chi-reconciler.go:461] reconcileHostService():DONE Reconcile service of the host: 0-1
I0929 11:56:38.491418       1 cluster.go:84] Run query on: chi-gid-gid-0-1.gidplatform-dev.svc.cluster.local of [chi-gid-gid-0-1.gidplatform-dev.svc.cluster.local]
I0929 11:56:38.495556       1 worker-chi-reconciler.go:349] getHostClickHouseVersion():Get ClickHouse version on host: 0-1 version: <IP_ADDRESS>8
I0929 11:56:38.495593       1 poller.go:138] Poll():gidplatform-dev/0-1:OK gidplatform-dev/0-1
I0929 11:56:38.495629       1 worker-chi-reconciler.go:753] reconcileHost():Check host for ClickHouse availability before migrating tables. Host: 0-1 ClickHouse version running: <IP_ADDRESS>8
I0929 11:56:38.495686       1 worker.go:908] migrateTables():No need to add tables on host 1 to shard 0 in cluster gid
I0929 11:56:38.495706       1 worker.go:1057] includeHost():Include into cluster host 1 shard 0 cluster gid
I0929 11:56:38.495726       1 worker.go:1124] includeHostIntoClickHouseCluster():going to include host 1 shard 0 cluster gid
I0929 11:56:38.495737       1 worker.go:844] RemoteServersGeneratorOptions: exclude hosts: [], attributes: status: , add: true, remove: false, modify: false, found: false, exclude: true
I0929 11:56:38.654056       1 worker.go:1315] updateConfigMap():gidplatform-dev/gid/click-reconcile-1:Update ConfigMap gidplatform-dev/chi-gid-common-configd
I0929 11:56:39.689499       1 chi.go:38] prepareListOfTemplates():gidplatform-dev/gid/click-reconcile-1:Found applicable templates num: 0
I0929 11:56:39.689543       1 chi.go:82] ApplyCHITemplates():gidplatform-dev/gid/click-reconcile-1:Applied templates num: 0
I0929 11:56:39.711932       1 chi.go:38] prepareListOfTemplates():gidplatform-dev/gid/click-reconcile-1:Found applicable templates num: 0
I0929 11:56:39.711952       1 chi.go:82] ApplyCHITemplates():gidplatform-dev/gid/click-reconcile-1:Applied templates num: 0
I0929 11:56:39.713061       1 worker.go:387] gidplatform-dev/gid/click-reconcile-1:updating endpoints for CHI-1 gid
I0929 11:56:39.851639       1 cluster.go:84] Run query on: chi-gid-gid-0-1.gidplatform-dev.svc.cluster.local of [chi-gid-gid-0-1.gidplatform-dev.svc.cluster.local]
I0929 11:56:39.853763       1 worker-chi-reconciler.go:349] getHostClickHouseVersion():Get ClickHouse version on host: 0-1 version: <IP_ADDRESS>8
I0929 11:56:39.853841       1 poller.go:138] Poll():gidplatform-dev/0-1:OK gidplatform-dev/0-1
I0929 11:56:39.853942       1 worker-chi-reconciler.go:776] reconcileHost():Reconcile Host completed. Host: 0-1 ClickHouse version running: <IP_ADDRESS>8
I0929 11:56:40.449305       1 worker.go:389] gidplatform-dev/gid/click-reconcile-1:IPs of the CHI-1 update endpoints gidplatform-dev/gid: len: 2 [<IP_ADDRESS> <IP_ADDRESS>]
I0929 11:56:40.460088       1 chi.go:38] prepareListOfTemplates():gidplatform-dev/gid/click-reconcile-1:Found applicable templates num: 0
I0929 11:56:40.460129       1 chi.go:82] ApplyCHITemplates():gidplatform-dev/gid/click-reconcile-1:Applied templates num: 0
I0929 11:56:40.462470       1 worker.go:393] gidplatform-dev/gid/click-reconcile-1:Update users IPS-1
I0929 11:56:40.849312       1 worker.go:1315] updateConfigMap():gidplatform-dev/gid/click-reconcile-1:Update ConfigMap gidplatform-dev/chi-gid-common-usersd
I0929 11:56:41.078096       1 worker-chi-reconciler.go:797] reconcileHost():[now: 2024-09-29 11:56:41.078003076 +0000 UTC m=+530649.174408624] ProgressHostsCompleted: 2 of 2
I0929 11:56:43.083018       1 worker-chi-reconciler.go:581] Starting rest of shards on workers: 1
I0929 11:56:43.249032       1 worker.go:1315] updateConfigMap():gidplatform-dev/gid/click-reconcile-1:Update ConfigMap gidplatform-dev/chi-gid-common-configd
I0929 11:56:43.885956       1 worker-deleter.go:43] clean():gidplatform-dev/gid/click-reconcile-1:remove items scheduled for deletion
I0929 11:56:44.481307       1 worker-deleter.go:46] clean():gidplatform-dev/gid/click-reconcile-1:List of objects which have failed to reconcile:
I0929 11:56:44.481378       1 worker-deleter.go:47] clean():gidplatform-dev/gid/click-reconcile-1:List of successfully reconciled objects:
PVC: gidplatform-dev/default-volume-claim-chi-gid-gid-0-0-0
PVC: gidplatform-dev/default-volume-claim-chi-gid-gid-0-1-0
StatefulSet: gidplatform-dev/chi-gid-gid-0-1
StatefulSet: gidplatform-dev/chi-gid-gid-0-0
Service: gidplatform-dev/chi-gid-gid-0-0
Service: gidplatform-dev/clickhouse-gid
Service: gidplatform-dev/chi-gid-gid-0-1
ConfigMap: gidplatform-dev/chi-gid-common-configd
ConfigMap: gidplatform-dev/chi-gid-common-usersd
ConfigMap: gidplatform-dev/chi-gid-deploy-confd-gid-0-0
ConfigMap: gidplatform-dev/chi-gid-deploy-confd-gid-0-1
PDB: gidplatform-dev/gid-gid
I0929 11:56:45.252969       1 worker-deleter.go:50] clean():gidplatform-dev/gid/click-reconcile-1:Existing objects:
PVC: gidplatform-dev/default-volume-claim-chi-gid-gid-0-0-0
PVC: gidplatform-dev/default-volume-claim-chi-gid-gid-0-1-0
PDB: gidplatform-dev/gid-gid
StatefulSet: gidplatform-dev/chi-gid-gid-0-0
StatefulSet: gidplatform-dev/chi-gid-gid-0-1
ConfigMap: gidplatform-dev/chi-gid-common-configd
ConfigMap: gidplatform-dev/chi-gid-common-usersd
ConfigMap: gidplatform-dev/chi-gid-deploy-confd-gid-0-0
ConfigMap: gidplatform-dev/chi-gid-deploy-confd-gid-0-1
Service: gidplatform-dev/chi-gid-gid-0-0
Service: gidplatform-dev/chi-gid-gid-0-1
Service: gidplatform-dev/clickhouse-gid
I0929 11:56:45.253123       1 worker-deleter.go:52] clean():gidplatform-dev/gid/click-reconcile-1:Non-reconciled objects:
I0929 11:56:45.253195       1 worker-deleter.go:68] worker-deleter.go:68:dropReplicas():start:gidplatform-dev/gid/click-reconcile-1:drop replicas based on AP
I0929 11:56:45.253260       1 worker-deleter.go:80] worker-deleter.go:80:dropReplicas():end:gidplatform-dev/gid/click-reconcile-1:processed replicas: 0
I0929 11:56:45.253308       1 worker.go:640] addCHIToMonitoring():gidplatform-dev/gid/click-reconcile-1:add CHI to monitoring
I0929 11:56:45.885652       1 worker.go:595] worker.go:595:waitForIPAddresses():start:gidplatform-dev/gid/click-reconcile-1:wait for IP addresses to be assigned to all pods
I0929 11:56:45.893820       1 worker.go:600] gidplatform-dev/gid/click-reconcile-1:all IP addresses are in place
I0929 11:56:45.893858       1 worker.go:673] worker.go:673:finalizeReconcileAndMarkCompleted():start:gidplatform-dev/gid/click-reconcile-1:finalize reconcile
I0929 11:56:45.904253       1 chi.go:38] prepareListOfTemplates():gidplatform-dev/gid/click-reconcile-1:Found applicable templates num: 0
I0929 11:56:45.904335       1 chi.go:82] ApplyCHITemplates():gidplatform-dev/gid/click-reconcile-1:Applied templates num: 0
I0929 11:56:45.904391       1 controller.go:617] OK update watch (gidplatform-dev/gid): {"namespace":"gidplatform-dev","name":"gid","labels":{"argocd.argoproj.io/instance":"bi-clickhouse-dev","k8slens-edit-resource-version":"v1"},"annotations":{},"clusters":[{"name":"gid","hosts":[{"name":"0-0","hostname":"chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local","tcpPort":9000,"httpPort":8123},{"name":"0-1","hostname":"chi-gid-gid-0-1.gidplatform-dev.svc.cluster.local","tcpPort":9000,"httpPort":8123}]}]}
I0929 11:56:45.906676       1 worker.go:677] gidplatform-dev/gid/click-reconcile-1:updating endpoints for CHI-2 gid
I0929 11:56:46.249853       1 worker.go:679] gidplatform-dev/gid/click-reconcile-1:IPs of the CHI-2 finalize reconcile gidplatform-dev/gid: len: 2 [<IP_ADDRESS> <IP_ADDRESS>]
I0929 11:56:46.261380       1 chi.go:38] prepareListOfTemplates():gidplatform-dev/gid/click-reconcile-1:Found applicable templates num: 0
I0929 11:56:46.261442       1 chi.go:82] ApplyCHITemplates():gidplatform-dev/gid/click-reconcile-1:Applied templates num: 0
I0929 11:56:46.263792       1 worker.go:683] gidplatform-dev/gid/click-reconcile-1:Update users IPS-2
I0929 11:56:46.449574       1 worker.go:1315] updateConfigMap():gidplatform-dev/gid/click-reconcile-1:Update ConfigMap gidplatform-dev/chi-gid-common-usersd
I0929 11:56:47.495545       1 worker.go:707] finalizeReconcileAndMarkCompleted():gidplatform-dev/gid/click-reconcile-1:reconcile completed successfully, task id: click-reconcile-1
I0929 11:56:48.077981       1 worker-chi-reconciler.go:134] worker-chi-reconciler.go:60:reconcileCHI():end:gidplatform-dev/gid/click-reconcile-1
I0929 11:56:48.078036       1 worker.go:469] worker.go:432:updateCHI():end:gidplatform-dev/gid/click-reconcile-1

Not sure what going wrong, but on host chi-gid-gid-0-0-0 even no databases copied. And still present only default.

@alex-zaitsev, could you please look on it?

I0929 11:55:05.495957 [worker.go:908] migrateTables():No need to add tables on host 0 to shard 0 in cluster gid

I0929 11:56:38.495686       1 worker.go:908] migrateTables():No need to add tables on host 1 to shard 0 in cluster gid

@Hubbitus is your cluster have 2 shards with only 1 replica inside shard?
Could you share:
kubectl get chi -n gidplatform-de gid -o yaml
without sensitive credentials?

@Slach, thanks to response.
We do not use sharding yet.
Output of kubectl get chi -n gidplatform-dev gid -o yaml:
chi.yaml.gz

Could you share result of following clickhouse-client query
SELECT database, table, engine_full, count() c FROM cluster('all-sharded',system.tables) WHERE database NOT IN ('system','INFORMATION_SCHEMA','information_schema') GROUP BY ALL HAVING c<2

Sure (limit to 10, total 269):

database
table
engine_full
c

datamart
appmarket__public__widget
PostgreSQL(appmarket_db, table = 'widget', schema = 'public')
1

datamart
bonus__public__promotion
PostgreSQL(bonus_db, table = 'promotion', schema = 'public')
1

sandbox
gid_mt_sessions
ReplicatedMergeTree('/clickhouse/tables/ad0a75c4-1aa7-4386-a542-c16c19f2b2c6/{shard}', '{replica}') ORDER BY tsEvent SETTINGS index_granularity = 8192
1

_source
scs__public__story__foreign
PostgreSQL(scs_db, table = 'story', schema = 'public')
1

datamart
loyalty__public__level
PostgreSQL(loyalty_db, table = 'level', schema = 'public')
1

_source
feed__public__reaction__foreign
PostgreSQL(feed_db, table = 'reaction', schema = 'public')
1

_loopback
nomail_account_register
ReplicatedMergeTree('/clickhouse/tables/{uuid}/{shard}', '{replica}') PARTITION BY tuple() ORDER BY time SETTINGS index_granularity = 8192
1

_source
questionnaires__public__anketa_access_group__foreign
PostgreSQL(questionnaire_db, table = 'anketa_access_group', schema = 'public')
1

sandbox
gid_mt_activities
ReplicatedMergeTree('/clickhouse/tables/{uuid}/{shard}', '{replica}') ORDER BY dtEvent SETTINGS index_granularity = 8192
1

_source
lms__public__lms_user_courses_progress__foreign
PostgreSQL(lms_db, table = 'lms_user_courses_progress', schema = 'public')
1

You shared logs for 29 sep 2024 since 11:55 UTC
is your node lost PVC data before this date, or after this date?

Hello.
Last shared logs was from 15 October. And that after one more attempt to recover by deleting PVC and STS.

@Hubbitus
https://github.com/Altinity/clickhouse-operator/issues/1455#issuecomment-2381334294
there is logs only for 29 sep 2024
i don't see logs from 15 Oct 2024
I need to ensure you tried to reconcile, after drop PVC and STS
did you change in CHI spec.taskID manually to trigger reconcile after delete PVC and STS?

share clickhouse-operator logs for 15 Oct related to your changes

operator.2024-10-01T16:39:32+03:00.obfuscated.log

Hello.
I do not have so old logs.
But I've switching on branch were set taskID: "click-reconcile-3". It looks like reconcile started automatically.
Relevant part of output (slightly obfuscated)
kubectl -n kube-system logs --selector=app=clickhouse-operator --container=clickhouse-operator

operator.2024-11-02T17:47:14+03:00.obfuscated.log
Output of
SELECT database, table, engine_full, count() c, hostname()
FROM
	cluster('{cluster}',system.tables)
WHERE
	database NOT IN ('system','INFORMATION_SCHEMA','information_schema')
GROUP BY ALL
HAVING c<2

Contains 515 rows. Heading of it:

database
table
engine_full
c
hostname()

datamart
v_subs__public__channel_requests

1
chi-gid-gid-0-1-0

cdc
api__public__reaction
ReplicatedReplacingMergeTree('/clickhouse/{cluster}/cdc/tables/api__public__reaction/{shard}', '{replica}') PRIMARY KEY id ORDER BY id SETTINGS index_granularity = 8192
1
chi-gid-gid-0-1-0

_source
bonus_to_gid__user_mappings
Kafka(kafka_integration, kafka_topic_list = 'dev__bonus_to_gid__user_mappings', kafka_group_name = 'dev__bonus_to_gid__user_mappings') SETTINGS format_avro_schema_registry_url = 'http://gid-integration-partner-kafka.gid.team:8081'
1
chi-gid-gid-0-1-0

datamart
v_calendar__public__event_type

1
chi-gid-gid-0-1-0

_raw
api__public__questionnaire_result__dbt_materialized
ReplicatedMergeTree('/clickhouse/tables/{uuid}/{shard}', '{replica}') ORDER BY id SETTINGS replicated_deduplication_window = 0, index_granularity = 8192
1
chi-gid-gid-0-1-0

datamart
v_jiradatabase__public__ao_54307e_slaauditlog

1
chi-gid-gid-0-1-0

datamart
v_appmarket__public__widget_notification

1
chi-gid-gid-0-1-0

_raw
feed__public__feed_comment__dbt_materialized
ReplicatedMergeTree('/clickhouse/tables/{uuid}/{shard}', '{replica}') ORDER BY id SETTINGS replicated_deduplication_window = 0, index_granularity = 8192
1
chi-gid-gid-0-1-0

_raw
lms__public__lms_courses_chapters__dbt_materialized
ReplicatedMergeTree('/clickhouse/tables/{uuid}/{shard}', '{replica}') ORDER BY id SETTINGS replicated_deduplication_window = 0, index_granularity = 8192
1
chi-gid-gid-0-1-0

datamart
tmp_gazprombonus_user_bonus_to_gid_mapping_inner
ReplicatedMergeTree('/clickhouse/tables/{uuid}/{shard}', '{replica}') ORDER BY id SETTINGS index_granularity = 1024
1
chi-gid-gid-0-1-0

datamart
api__poll_vote

1
chi-gid-gid-0-1-0

_source
loyalty__public__achievement__foreign
PostgreSQL(loyalty_db, table = 'achievement', schema = 'public')
1
chi-gid-gid-0-1-0

datamart
v_calendar__public__like

1
chi-gid-gid-0-1-0

_source
calendar__public__event_type__foreign
PostgreSQL(calendar_db, table = 'event_type', schema = 'public')
1
chi-gid-gid-0-1-0

according to logs you just triggered reconcile for -0-0-0 when sts is not deleted
try
kubectl delete sts -n gidplatform-dev chi-gid-gid-0-0
kubectl delete pvc -n gidplatform-dev -l clickhouse.altinity.com/cluster=gid,clickhouse.altinity.com/shard=0,clickhouse.altinity.com/replica=0

kubectl edit chi -n gidplatform-dev gid

edit spec.taskID to manual-4
watch reconciling process again, when sts and pvc not found during reconcile then operator shall propagate schema

Ok, thank you.
 Run sync in ArgoCD with prune.

I think relevants logs are:

I1102 16:33:32.889891       1 worker-chi-reconciler.go:89] reconcileCHI():gidplatform-dev/gid/click-reconcile-4:ActionPlan has actions - continue reconcile
I1102 16:33:32.934904       1 worker.go:663] markReconcileStart():gidplatform-dev/gid/click-reconcile-4:reconcile started, task id: click-reconcile-4
I1102 16:33:33.446722       1 worker.go:820] FOUND host: ns:gidplatform-dev|chi:gid|clu:gid|sha:0|rep:0|host:0-0
I1102 16:33:33.446764       1 worker.go:820] FOUND host: ns:gidplatform-dev|chi:gid|clu:gid|sha:0|rep:1|host:0-1
I1102 16:33:33.446914       1 worker.go:844] RemoteServersGeneratorOptions: exclude hosts: [], attributes: status: , add: true, remove: false, modify: false, found: false, exclude: true
I1102 16:33:33.642314       1 worker.go:1315] updateConfigMap():gidplatform-dev/gid/click-reconcile-4:Update ConfigMap gidplatform-dev/chi-gid-common-configd
I1102 16:33:34.443758       1 worker.go:1315] updateConfigMap():gidplatform-dev/gid/click-reconcile-4:Update ConfigMap gidplatform-dev/chi-gid-common-usersd
I1102 16:33:35.086985       1 service.go:86] CreateServiceCluster():gidplatform-dev/gid/click-reconcile-4:gidplatform-dev/cluster-gid-gid
I1102 16:33:35.104209       1 worker-chi-reconciler.go:819] PDB updated: gidplatform-dev/gid-gid
I1102 16:33:35.104304       1 worker-chi-reconciler.go:554] not found ReconcileShardsAndHostsOptionsCtxKey, use empty opts
I1102 16:33:35.104344       1 worker-chi-reconciler.go:568] starting first shard separately
I1102 16:33:35.104638       1 cluster.go:84] Run query on: chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local of [chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local]
E1102 16:33:35.112714       1 connection.go:145] QueryContext():FAILED Query(http://test_operator:***@chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local:8123/) doRequest: transport failed to send a request to ClickHouse: dial tcp: lookup chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local on <IP_ADDRESS>:53: no such host for SQL: SELECT version()
W1102 16:33:35.112777       1 cluster.go:91] QueryAny():FAILED to run query on: chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local of [chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local] skip to next. err: doRequest: transport failed to send a request to ClickHouse: dial tcp: lookup chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local on <IP_ADDRESS>:53: no such host
E1102 16:33:35.112846       1 cluster.go:95] QueryAny():FAILED to run query on all hosts [chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local]
W1102 16:33:35.112926       1 worker-chi-reconciler.go:345] getHostClickHouseVersion():Failed to get ClickHouse version on host: 0-0
W1102 16:33:35.112980       1 worker-chi-reconciler.go:690] reconcileHost():Reconcile Host start. Host: 0-0 Failed to get ClickHouse version: failed to query
W1102 16:33:35.692945       1 worker.go:1537] gidplatform-dev/chi-gid-gid-0-0:No cur StatefulSet available but host has an ancestor. Found deleted StatefulSet. for gidplatform-dev/chi-gid-gid-0-0

So operator can't resolve hostname of node:
doRequest: transport failed to send a request to ClickHouse: dial tcp: lookup chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local on <IP_ADDRESS>:53: no such host for SQL: SELECT version()

Indeed hostname chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local looks strange, clichouse known it in another way:
SELECT cluster, host_name
FROM system.clusters
WHERE cluster = 'gid'

cluster
host_name

gid
chi-gid-gid-0-0

gid
chi-gid-gid-0-1

You did not share full logs, just found first error message, error message is expected, because you deleted sts and kubernetes service name will not resolve
Hostname, contains SERVICE name, not pod
is sts chi-gid-gid-0-0-0 and pvc created?
could you share full operator logs?

Sure. I found much more errors in log:
operator.2024-11-02T19:40:16+03:00.obfuscated.log

@sunsingerus. according to shared logs
first reconcile was applied at 2024-11-02 14:44:41
and sts + pvs was exists

I1102 14:44:44.267254       1 cluster.go:84] Run query on: chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local of [chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local]
I1102 14:44:44.276002       1 worker-chi-reconciler.go:349] getHostClickHouseVersion():Get ClickHouse version on host: 0-0 version: <IP_ADDRESS>8
I1102 14:44:44.276073       1 worker-chi-reconciler.go:684] reconcileHost():Reconcile Host start. Host: 0-0 ClickHouse version running: <IP_ADDRESS>8

second try after sts + pvc deletion

I1102 16:30:39.627295       1 worker.go:275] processReconcilePod():gidplatform-dev/chi-gid-gid-0-0-0:Delete Pod. gidplatform-dev/chi-gid-gid-0-0-0
I1102 16:33:32.819212       1 controller.go:572] ENQUEUE new ReconcileCHI cmd=update for gidplatform-dev/gid

I1102 16:33:32.934904       1 worker.go:663] markReconcileStart():gidplatform-dev/gid/click-reconcile-4:reconcile started, task id: click-reconcile-4

STS and PVC was deleted

W1102 16:33:35.692945       1 worker.go:1537] gidplatform-dev/chi-gid-gid-0-0:No cur StatefulSet available but host has an ancestor. Found deleted StatefulSet. for gidplatform-dev/chi-gid-gid-0-0
I1102 16:33:35.839840       1 worker.go:1177] shouldExcludeHost():Host should be excluded. Host/shard/cluster: 0/0/gid
I1102 16:33:35.839914       1 worker.go:1010] excludeHost():Exclude from cluster host 0 shard 0 cluster gid
I1102 16:33:37.880392       1 worker-chi-reconciler.go:716] reconcileHost():Reconcile PVCs and check possible data loss for host: 0-0

PVC recreated

I1102 16:33:38.042697       1 worker-chi-reconciler.go:1251] PVC (gidplatform-dev/0-0/default-volume-claim/default-volume-claim-chi-gid-gid-0-0-0) not found and model will not be provided by the operator
W1102 16:33:38.042849       1 worker-chi-reconciler.go:1162] PVC is either newly added to the host or was lost earlier (gidplatform-dev/0-0/default-volume-claim/pvc-name-unknown-pvc-not-exist)

Migration force to be applied
Start creating Statefulset

I1102 16:33:38.043010       1 worker-chi-reconciler.go:730] reconcileHost():Data loss detected for host: 0-0. Will do force migrate
I1102 16:33:38.043073       1 worker-chi-reconciler.go:406] worker-chi-reconciler.go:406:reconcileHostStatefulSet():start:reconcile StatefulSet start

I1102 16:33:38.440090       1 worker.go:1596] createStatefulSet():Create StatefulSet gidplatform-dev/chi-gid-gid-0-0 - started
I1102 16:33:39.086823       1 creator.go:35] createStatefulSet()
I1102 16:33:39.086858       1 creator.go:44] Create StatefulSet gidplatform-dev/chi-gid-gid-0-0

I1102 16:34:09.634311       1 worker.go:1615] createStatefulSet():Create StatefulSet gidplatform-dev/chi-gid-gid-0-0 - completed

Prepare for table migration, try

I1102 16:34:11.228817       1 worker-chi-reconciler.go:753] reconcileHost():Check host for ClickHouse availability before migrating tables. Host: 0-0 ClickHouse version running: <IP_ADDRESS>8

Trying drop data from ZK
@sunsingerus
0-0-0 is empty and doesn't contains any table definitions,
so I think, SYSTEM DROP REPLICA 'chi-gid-gid-0-0-0' will do nothing, and this is root cause

I1102 16:34:11.228960       1 schemer.go:56] HostDropReplica():Drop replica: chi-gid-gid-0-0 at 0-0
I1102 16:34:11.236511       1 worker-deleter.go:414] dropReplica():Drop replica host: 0-0 in cluster: gid

Get SQL object definitions

I1102 16:34:12.415941       1 replicated.go:35] shouldCreateReplicatedObjects():SchemaPolicy.Shard says we need replicated objects. Should create replicated objects for the shard: [chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local chi-gid-gid-0-1.gidplatform-dev.svc.cluster.local]
I1102 16:34:12.416343       1 cluster.go:84] Run query on: chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local of [chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local chi-gid-gid-0-1.gidplatform-dev.svc.cluster.local]
I1102 16:34:12.437761       1 cluster.go:84] Run query on: chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local of [chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local chi-gid-gid-0-1.gidplatform-dev.svc.cluster.local]
I1102 16:34:12.717044       1 cluster.go:84] Run query on: chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local of [chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local chi-gid-gid-0-1.gidplatform-dev.svc.cluster.local]
I1102 16:34:12.756658       1 distributed.go:39] shouldCreateDistributedObjects():Should create distributed objects in the cluster: [chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local chi-gid-gid-0-1.gidplatform-dev.svc.cluster.local]
I1102 16:34:12.756850       1 cluster.go:84] Run query on: chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local of [chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local chi-gid-gid-0-1.gidplatform-dev.svc.cluster.local]
I1102 16:34:12.786098       1 cluster.go:84] Run query on: chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local of [chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local chi-gid-gid-0-1.gidplatform-dev.svc.cluster.local]
I1102 16:34:13.018954       1 cluster.go:84] Run query on: chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local of [chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local chi-gid-gid-0-1.gidplatform-dev.svc.cluster.local]

Trying to restore and failure cause ZK data still present

I1102 16:34:13.035413       1 schemer.go:98] HostCreateTables():Creating replicated objects at 0-0: [_loopback _raw service _source temp ....]
E1102 16:34:13.089822       1 connection.go:194] Exec():FAILED Exec(http://test_operator:***@chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local:8123/) Code: 253, Message: Replica /clickhouse/tables/edf41bd4-46aa-4341-bed7-2e19b838e9e1/0/replicas/chi-gid-gid-0-0 already exists for SQL: CREATE TABLE IF NOT EXISTS _loopback.nomail_account_register UUID 'edf41bd4-46aa-4341-bed7-2e19b838e9e1' ....
I1102 16:34:13.089918       1 cluster.go:160] func1():chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local:Replica is already in ZooKeeper. Trying ATTACH TABLE instead

We need to choose 0-1-0 for execution SYSTEM DROP REPLICA...

@Hubbitus after reconcile
most of tables shall restore (via ATTACH)
but some of tables, not restored with strange difference
`slo_value` Decimal(15, 5)	DEFAULT	0

in zookeeper
and
`slo_value` Decimal(15, 5)

in local SQL
E1102 16:36:08.836812       1 connection.go:194] Exec():FAILED Exec(http://test_operator:***@chi-gid-gid-0-0.gidplatform-dev.svc.cluster.local:8123/) Code: 122, Message: Table columns structure in ZooKeeper is different from local table structure. Local columns:
columns format version: 1
14 columns:
...
Zookeeper columns:
columns format version: 1
14 columns:
...
for SQL: CREATE TABLE IF NOT EXISTS _raw.victoriametrics__slo__metrics__airflow_hour_agg_old UUID '46a1c218-9274-446d-9300-e644bbd4cc0e' ....

@Hubbitus could you share
kubectl exec -n gidplatform-dev chi-gid-gid-0-1-0 -- clickhouse-client -q "SHOW CREATE TABLE _raw.victoriametrics__slo__metrics__airflow_hour_agg_old FORMA Vertical"
and
kubectl exec -n gidplatform-dev chi-gid-gid-0-0-0 -- clickhouse-client -q "SHOW CREATE TABLE _raw.victoriametrics__slo__metrics__airflow_hour_agg_old FORMA Vertical"

@Slach, sure (comments of columns and table stripped):
$ kubectl exec -n gidplatform-dev chi-gid-gid-0-1-0 -- clickhouse-client -q "SHOW CREATE TABLE _raw.victoriametrics__slo__metrics__airflow_hour_agg_old FORMAT Vertical"

Row 1:
──────
statement: CREATE TABLE _raw.victoriametrics__slo__metrics__airflow_hour_agg_old
(
    `slo_metric` LowCardinality(String),
    `slo_service` LowCardinality(String),
    `slo_namespace` LowCardinality(String),
    `slo_status` LowCardinality(String),
    `slo_method` LowCardinality(String),
    `slo_uri` LowCardinality(String),
    `slo_le` LowCardinality(String),
    `slo_event_ts` DateTime64(6, 'UTC'),
    `slo_orig_value` UInt64,
    `slo_value` UInt32,
    `slo_rec_num` UInt32,
    `slo_tags` Map(LowCardinality(String), LowCardinality(String)),
    `_row_hash_` UInt64 MATERIALIZED cityHash64(slo_metric, slo_service, slo_namespace, slo_status, slo_method, slo_uri, slo_event_ts, slo_value, slo_rec_num, slo_tags),
    `__insert_ts` DateTime64(6, 'UTC') DEFAULT now64(6, 'UTC')
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{uuid}/{shard}', '{replica}')
ORDER BY (slo_metric, slo_service, slo_namespace, slo_event_ts)
SETTINGS index_granularity = 8192

$ kubectl exec -n gidplatform-dev chi-gid-gid-0-0-0 -- clickhouse-client -q "SHOW CREATE TABLE _raw.victoriametrics__slo__metrics__airflow_hour_agg_old FORMA Vertical"
$ kubectl exec -n gidplatform-dev chi-gid-gid-0-0-0 -- clickhouse-client -q "SHOW CREATE TABLE _raw.victoriametrics__slo__metrics__airflow_hour_agg_old FORMAT Vertical"
Received exception from server (version 24.2.1):
Code: 390. DB::Exception: Received from localhost:9000. DB::Exception: Table `victoriametrics__slo__metrics__airflow_hour_agg_old` doesn't exist. (CANNOT_GET_CREATE_TABLE_QUERY)
(query: SHOW CREATE TABLE _raw.victoriametrics__slo__metrics__airflow_hour_agg_old FORMAT Vertical)
command terminated with exit code 134

add SETTINGS show_table_uuid_in_table_create_query_if_not_nil=1
kubectl exec -n gidplatform-dev chi-gid-gid-0-1-0 -- clickhouse-client -q "SHOW CREATE TABLE _raw.victoriametrics__slo__metrics__airflow_hour_agg_old SETTTINGS show_table_uuid_in_table_create_query_if_not_nil=1 FORMAT Vertical"

$ kubectl exec -n gidplatform-dev chi-gid-gid-0-1-0 -- clickhouse-client -q "SHOW CREATE TABLE _raw.victoriametrics__slo__metrics__airflow_hour_agg_old FORMAT Vertical SETTINGS show_table_uuid_in_table_create_query_if_not_nil=1"
Row 1:
──────
statement: CREATE TABLE _raw.victoriametrics__slo__metrics__airflow_hour_agg_old UUID '46a1c218-9274-446d-9300-e644bbd4cc0e'
(
    `slo_metric` LowCardinality(String),
    `slo_service` LowCardinality(String),
    `slo_namespace` LowCardinality(String),
    `slo_status` LowCardinality(String),
    `slo_method` LowCardinality(String),
    `slo_uri` LowCardinality(String),
    `slo_le` LowCardinality(String),
    `slo_event_ts` DateTime64(6, 'UTC'),
    `slo_orig_value` UInt64,
    `slo_value` UInt32,
    `slo_rec_num` UInt32,
    `slo_tags` Map(LowCardinality(String), LowCardinality(String)),
    `_row_hash_` UInt64 MATERIALIZED cityHash64(slo_metric, slo_service, slo_namespace, slo_status, slo_method, slo_uri, slo_event_ts, slo_value, slo_rec_num, slo_tags),
    `__insert_ts` DateTime64(6, 'UTC') DEFAULT now64(6, 'UTC')
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{uuid}/{shard}', '{replica}')
ORDER BY (slo_metric, slo_service, slo_namespace, slo_event_ts)
SETTINGS index_granularity = 8192

try to restore table
CREATE TABLE IF NOT EXISTS _raw.victoriametrics__slo__metrics__airflow_hour_agg_old UUID '46a1c218-9274-446d-9300-e644bbd4cc0e' ON CLUSTER '{cluster}'
(
    `slo_metric` LowCardinality(String),
    `slo_service` LowCardinality(String),
    `slo_namespace` LowCardinality(String),
    `slo_status` LowCardinality(String),
    `slo_method` LowCardinality(String),
    `slo_uri` LowCardinality(String),
    `slo_le` LowCardinality(String),
    `slo_event_ts` DateTime64(6, 'UTC'),
    `slo_orig_value` UInt64,
    `slo_value` UInt32,
    `slo_rec_num` UInt32,
    `slo_tags` Map(LowCardinality(String), LowCardinality(String)),
    `_row_hash_` UInt64 MATERIALIZED cityHash64(slo_metric, slo_service, slo_namespace, slo_status, slo_method, slo_uri, slo_event_ts, slo_value, slo_rec_num, slo_tags),
    `__insert_ts` DateTime64(6, 'UTC') DEFAULT now64(6, 'UTC')
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{uuid}/{shard}', '{replica}')
ORDER BY (slo_metric, slo_service, slo_namespace, slo_event_ts)
SETTINGS index_granularity = 8192

@Slach, why it is not restored automatically? Really, this table may be deleted. IS it the solution?

It tried to restore, and most of the tables were restored (am I right?)
but looks like you applied some mutations or something else and few tables contains different SQL structure which we get in operator from system.tables with current zookeeper /clickhouse/tables/{uuid}/{shard}/{replica}/columns key content

try to upgrade clickhouse-keeper to latest and clickhouse-server to 24.8 latest LTS

It tried to restore, and most of the tables were restored (am I right?)
No.

Query
SELECT database, table, engine_full, count() c, hostname()
FROM
	cluster('{cluster}',system.tables)
WHERE
	database NOT IN ('system','INFORMATION_SCHEMA','information_schema')
GROUP BY ALL
HAVING c<2

Still returns 729 tables present only on host chi-gid-gid-0-1-0.

try to upgrade clickhouse-keeper to latest and clickhouse-server to 24.8 latest LTS

It is not so fast. We will try to do it in the next week.
But we use zookeeper, and not clickhouse keeper. Is it important to switch (that may be the problem)?

ok. let's try again
# safe shutdown
kubectl exec -n gidplatform-dev chi-gid-gid-0-0-0 -- clickhouse-client -q "SYSTEM SHUTDOWN"

# different node
kubectl exec -n gidplatform-dev chi-gid-gid-0-1-0 -- clickhouse-client -q "SYSTEM DROP REPLICA 'chi-gid-gid-0-0'"

# delete sts to propogate schema during reconcile
kubectl delete sts -n gidplatform-dev chi-gid-gid-0-0

# change spec.taskID
kubectl edit chi -n gidplatform-dev gid

# wait when
watch -n 1 kubectl describe chi chi -n gidplatform-dev gid

# check tables
# share clickhouse-operator logs

kubectl delete sts -n gidplatform-dev chi-gid-gid-0-0
statefulset.apps "chi-gid-gid-0-0" deleted
Wasn't PVC should be deleted also?

I think we need to delete PVC+PV for dead 0-0-0, to complete erase data for -0-0-0
I found in logs
storageclassname: openebs-hostpath-dataplatform
it means data in clickhouse will lost if your pod will re-scheduled in different worker node
of if worker node will reinstall from scratch
did you have operations similar to described above?
Let another try
# safe shutdown
kubectl exec -n gidplatform-dev chi-gid-gid-0-0-0 -- clickhouse-client -q "SYSTEM SHUTDOWN"

# different node
kubectl exec -n gidplatform-dev chi-gid-gid-0-1-0 -- clickhouse-client -q "SYSTEM DROP REPLICA 'chi-gid-gid-0-0'"

# delete sts+PV+PVC to propogate schema during reconcile
kubectl delete sts -n gidplatform-dev chi-gid-gid-0-0
kubectl delete pvc -n gidplatform-dev -l clickhouse.altinity.com/chi=gid,clickhouse.altinity.com/replica=0,clickhouse.altinity.com/shard=0

kubectl get pv | grep 0-0-0
kubectl delete pv <name of pv from previous step>

# change spec.taskID
kubectl edit chi -n gidplatform-dev gid

# wait when reconcile completed and look to events, watch to Adding table
watch -n 1 bash -c "kubectl describe chi -n gidplatform-dev gid | grep -i table"

it means data in clickhouse will lost if your pod will re-scheduled in different worker node of if worker node will reinstall from scratch did you have operations similar to described above?

Yes, we use openebs and taint to node. Openebs is one of the fastest storage class. And I hope there was no pod migration to another node.

Let another try

Let's doing:
$ kubectl exec -n gidplatform-dev chi-gid-gid-0-0-0 -- clickhouse-client -q "SYSTEM SHUTDOWN"

$ kubectl exec -n gidplatform-dev chi-gid-gid-0-1-0 -- clickhouse-client -q "SYSTEM DROP REPLICA 'chi-gid-gid-0-0'"
Received exception from server (version 24.8.6):
Code: 305. DB::Exception: Received from localhost:9000. DB::Exception: Can't drop replica: chi-gid-gid-0-0, because it's active. (TABLE_WAS_NOT_DROPPED)
(query: SYSTEM DROP REPLICA 'chi-gid-gid-0-0')
command terminated with exit code 49

ok. change action sequence
# safe shutdown
kubectl exec -n gidplatform-dev chi-gid-gid-0-0-0 -- clickhouse-client -q "SYSTEM SHUTDOWN"
sleep 0.5
# delete sts+PV+PVC to propogate schema during reconcile
kubectl delete sts -n gidplatform-dev chi-gid-gid-0-0
kubectl delete pvc -n gidplatform-dev -l clickhouse.altinity.com/chi=gid,clickhouse.altinity.com/replica=0,clickhouse.altinity.com/shard=0

kubectl get pv | grep 0-0-0
kubectl delete pv <name of pv from previous step>

# clean ZK from different node
kubectl exec -n gidplatform-dev chi-gid-gid-0-1-0 -- clickhouse-client -q "SYSTEM DROP REPLICA 'chi-gid-gid-0-0'"

# change spec.taskID
kubectl edit chi -n gidplatform-dev gid

# wait when reconcile completed and look to events, watch to Adding table
watch -n 1 bash -c "kubectl describe chi -n gidplatform-dev gid | grep -i table"

+ kubectl exec -n gidplatform-dev chi-gid-gid-0-0-0 -- clickhouse-client -q 'SYSTEM SHUTDOWN'
+ read -p 'Press Enter' T
Press Enter
+ kubectl delete sts -n gidplatform-dev chi-gid-gid-0-0
statefulset.apps "chi-gid-gid-0-0" deleted
+ read -p 'Press Enter' T
Press Enter
+ kubectl delete pvc -n gidplatform-dev -l clickhouse.altinity.com/chi=gid,clickhouse.altinity.com/replica=0,clickhouse.altinity.com/shard=0
persistentvolumeclaim "default-volume-claim-chi-gid-gid-0-0-0" deleted
+ read -p 'Press Enter' T
Press Enter
+ kubectl -n gidplatform-dev delete pv pvc-b76f4aaf-c323-4c74-bbc5-682bdb35e607
Warning: deleting cluster-scoped resources, not scoped to the provided namespace
persistentvolume "pvc-b76f4aaf-c323-4c74-bbc5-682bdb35e607" deleted
+ read -p 'Press Enter' T
Press Enter
+ kubectl exec -n gidplatform-dev chi-gid-gid-0-1-0 -- clickhouse-client -q 'SYSTEM DROP REPLICA '\''chi-gid-gid-0-0'\'''
Received exception from server (version 24.8.6):
Code: 305. DB::Exception: Received from localhost:9000. DB::Exception: Can't drop replica: chi-gid-gid-0-0, because it's active. (TABLE_WAS_NOT_DROPPED)
(query: SYSTEM DROP REPLICA 'chi-gid-gid-0-0')
command terminated with exit code 49

Another try of kubectl exec -n gidplatform-dev chi-gid-gid-0-1-0 -- clickhouse-client -q "SYSTEM DROP REPLICA 'chi-gid-gid-0-0'" succeeded.
watch -n 1 -x bash -c "kubectl describe chi -n gidplatform-dev gid | grep -i table --color"

Last line appeared:
Line `Info     CreateStarted     3m8s   clickhouse-operator  Adding tables on shard/host:0/0 cluster:gid` app

Then:
  Error    CreateFailed            21s    clickhouse-operator  ERROR add tables added successfully on shard/host:0/0 cluster:gid err:Code: 415, Message: Table /clickhouse/tables/62324966-8fef-4f9e-908d-a990b880d20c/0 was suddenly removed

Reconcile done: I1106 15:02:21.382386       1 worker.go:469] worker.go:432:updateCHI():end:gidplatform-dev/gid/click-reconcile-6
Now 730 tables only on host chi-gid-gid-0-0-0 :(
Full log:
operator.2024-11-06T18:04:12+03:00.obfuscated.log

kubectl exec -n gidplatform-dev chi-gid-gid-0-1-0 -- clickhouse-client -q "SELECT * FROM system.tables WHERE uuid='62324966-8fef-4f9e-908d-a990b880d20c' FORMAT Vertical"

$ kubectl exec -n gidplatform-dev chi-gid-gid-0-1-0 -- clickhouse-client -q "SELECT * FROM system.tables WHERE uuid='62324966-8fef-4f9e-908d-a990b880d20c' FORMAT Vertical"
Row 1:
──────
database:                      _source
name:                          victoriametrics__slo__metrics_old
uuid:                          62324966-8fef-4f9e-908d-a990b880d20c
engine:                        ReplicatedReplacingMergeTree
is_temporary:                  0
data_paths:                    ['/var/lib/clickhouse/store/623/62324966-8fef-4f9e-908d-a990b880d20c/']
metadata_path:                 /var/lib/clickhouse/store/783/7834b023-e34a-4087-bba9-98c168870af6/victoriametrics__slo__metrics_old.sql
metadata_modification_time:    2024-09-10 08:22:16
metadata_version:              0
dependencies_database:         []
dependencies_table:            []
create_table_query:            CREATE TABLE _source.victoriametrics__slo__metrics_old (`metric` LowCardinality(String), `tags` Map(LowCardinality(String), String), `ts` UInt64, `dt` Nullable(DateTime64(6, 'UTC')) MATERIALIZED toDateTime64(ts, 6, 'UTC'), `value` Nullable(String), `_row_hash_` UInt64 MATERIALIZED cityHash64((metric, tags, ts, value)), `_insert_dt_` DateTime64(6, 'UTC') DEFAULT now64(6, 'UTC') ENGINE = ReplicatedReplacingMergeTree('/clickhouse/tables/{uuid}/{shard}', '{replica}') PRIMARY KEY (_row_hash_, ts) ORDER BY (_row_hash_, ts) SETTINGS index_granularity = 8192
engine_full:                   ReplicatedReplacingMergeTree('/clickhouse/tables/{uuid}/{shard}', '{replica}') PRIMARY KEY (_row_hash_, ts) ORDER BY (_row_hash_, ts) SETTINGS index_granularity = 8192
as_select:
partition_key:
sorting_key:                   _row_hash_, ts
primary_key:                   _row_hash_, ts
sampling_key:
storage_policy:                default
total_rows:                    .....
total_bytes:                   .....
total_bytes_uncompressed:      .....
parts:                         6
active_parts:                  6
total_marks:                   13301
lifetime_rows:                 ᴺᵁᴸᴸ
lifetime_bytes:                ᴺᵁᴸᴸ
has_own_data:                  1
loading_dependencies_database: []
loading_dependencies_table:    []
loading_dependent_database:    []
loading_dependent_table:       []

Comments stripped.
I can drop that table if it helps.  AND name NOT LIKE '.inner_id.%' AND engine LIKE '%MergeTree%' SETTINGS show_table_uuid_in_table_create_query_if_not_nil=1 FORMAT TSVRaw" | clickhouse-client -mn --echo"
Create Distributed tables
kubectl exec -n gidplatform-dev chi-gid-gid-0-0-0 -- bash -c " clickhouse-client -q \"SELECT DISTINCT replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW|DICTIONARY|LIVE VIEW|WINDOW VIEW)', 'CREATE \\1 IF NOT EXISTS') || ';' AS q FROM cluster('all-sharded', system.tables) WHERE database NOT IN ('system', 'information_schema', 'INFORMATION_SCHEMA') AND create_table_query != '' AND name NOT LIKE '.inner.%' AND name NOT LIKE '.inner_id.%' AND engine LIKE '%Distributed%' SETTINGS show_table_uuid_in_table_create_query_if_not_nil=1 FORMAT TSVRaw" | clickhouse-client -mn --echo"
Create Other tables, MV, Dictionaries
kubectl exec -n gidplatform-dev chi-gid-gid-0-0-0 -- bash -c " clickhouse-client -q \"SELECT DISTINCT replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW|DICTIONARY|LIVE VIEW|WINDOW VIEW)', 'CREATE \\1 IF NOT EXISTS') || ';' AS q FROM cluster('all-sharded', system.tables) WHERE database NOT IN ('system', 'information_schema', 'INFORMATION_SCHEMA') AND create_table_query != '' AND name NOT LIKE '.inner.%' AND name NOT LIKE '.inner_id.%' AND engine NOT LIKE '%Distributed%' AND engine NOT LIKE '%MergeTree%' SETTINGS show_table_uuid_in_table_create_query_if_not_nil=1 FORMAT TSVRaw" | clickhouse-client -mn --echo"

@Slach,
I've tried first:
+ kubectl exec -n gidplatform-dev chi-gid-gid-0-0-0 -- bash -c 'clickhouse-client -q "SELECT DISTINCT '\''CREATE DATABASE IF NOT EXISTS "'\'' || name || '\''" Engine = '\'' || engine_full || '\'';'\'' AS create_db_query FROM cluster('\''all-sharded'\'', system.databases) databases WHERE name NOT IN ('\''system'\'', '\''information_schema'\'', '\''INFORMATION_SCHEMA'\'') FORMAT TSVRaw" | clickhouse-client -mn --echo'
Code: 62. DB::Exception: Syntax error: failed at position 32 ('||'): || name ||  Engine = Atomic;
. Expected identifier. (SYNTAX_ERROR)

command terminated with exit code 62

My attempt to fix:
$ kubectl exec -n gidplatform-dev chi-gid-gid-0-0-0 -- bash -c "clickhouse-client -q \"SELECT DISTINCT 'CREATE DATABASE IF NOT EXISTS \"' || name || '\" Engine = ' || engine_full || ';' AS create_db_query FROM cluster('all-sharded', system.databases) databases WHERE name NOT IN ('system', 'information_schema', 'INFORMATION_SCHEMA') FORMAT TSVRaw\" "
CREATE DATABASE IF NOT EXISTS  || name ||  Engine = Atomic;

Correcting quotes probably will:
$ kubectl exec -n gidplatform-dev chi-gid-gid-0-0-0 -- bash -c "clickhouse-client -q \"SELECT DISTINCT 'CREATE DATABASE IF NOT EXISTS ' || name || ' Engine = ' || engine_full || ';' AS create_db_query FROM cluster('all-sharded', system.databases) databases WHERE name NOT IN ('system', 'information_schema', 'INFORMATION_SCHEMA') FORMAT TSVRaw\" "
CREATE DATABASE IF NOT EXISTS _loopback Engine = Atomic;
CREATE DATABASE IF NOT EXISTS _raw Engine = Atomic;
CREATE DATABASE IF NOT EXISTS _service_ Engine = Atomic;
CREATE DATABASE IF NOT EXISTS _source Engine = Atomic;
CREATE DATABASE IF NOT EXISTS _temp_ Engine = Atomic;
CREATE DATABASE IF NOT EXISTS _temporaldb Engine = Atomic;
CREATE DATABASE IF NOT EXISTS cdc Engine = Atomic;
CREATE DATABASE IF NOT EXISTS datamart Engine = Atomic;
CREATE DATABASE IF NOT EXISTS default Engine = Atomic;
CREATE DATABASE IF NOT EXISTS sandbox Engine = Atomic;
CREATE DATABASE IF NOT EXISTS tmp_incr Engine = Atomic;

That without last part | clickhouse-client -mn --echo.
But what the reason dong so on same 0 node? Dumping from it and applying there. So clause IF NOT EXISTS will lead to do nothing.
Is it intended to dump structure from 1 (working) node and apply to 0 (broken)?

yes

let's change approach
run shell in 0-0-0
kubectl exec -n gidplatform-dev chi-gid-gid-0-0-0 -- bash
Datbases
clickhouse-client -q "SELECT DISTINCT 'CREATE DATABASE IF NOT EXISTS \"' || name || '\" Engine = ' || engine_full || ';' AS create_db_query FROM cluster('all-sharded', system.databases) databases WHERE name NOT IN ('system', 'information_schema', 'INFORMATION_SCHEMA') FORMAT TSVRaw" | clickhouse-client -mn --echo
Create MergeTree tables
clickhouse-client -q "SELECT DISTINCT replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW|DICTIONARY|LIVE VIEW|WINDOW VIEW)', 'CREATE \\1 IF NOT EXISTS') || ';' AS q FROM cluster('all-sharded', system.tables) WHERE database NOT IN ('system', 'information_schema', 'INFORMATION_SCHEMA') AND create_table_query != '' AND name NOT LIKE '.inner.%' AND name NOT LIKE '.inner_id.%' AND engine LIKE '%MergeTree%' SETTINGS show_table_uuid_in_table_create_query_if_not_nil=1 FORMAT TSVRaw" | clickhouse-client -mn --echo
Create Distributed tables
clickhouse-client -q "SELECT DISTINCT replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW|DICTIONARY|LIVE VIEW|WINDOW VIEW)', 'CREATE \\1 IF NOT EXISTS') || ';' AS q FROM cluster('all-sharded', system.tables) WHERE database NOT IN ('system', 'information_schema', 'INFORMATION_SCHEMA') AND create_table_query != '' AND name NOT LIKE '.inner.%' AND name NOT LIKE '.inner_id.%' AND engine LIKE '%Distributed%' SETTINGS show_table_uuid_in_table_create_query_if_not_nil=1 FORMAT TSVRaw" | clickhouse-client -mn --echo
Create Other tables, MV, Dictionaries
clickhouse-client -q "SELECT DISTINCT replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW|DICTIONARY|LIVE VIEW|WINDOW VIEW)', 'CREATE \\1 IF NOT EXISTS') || ';' AS q FROM cluster('all-sharded', system.tables) WHERE database NOT IN ('system', 'information_schema', 'INFORMATION_SCHEMA') AND create_table_query != '' AND name NOT LIKE '.inner.%' AND name NOT LIKE '.inner_id.%' AND engine NOT LIKE '%Distributed%' AND engine NOT LIKE '%MergeTree%' SETTINGS show_table_uuid_in_table_create_query_if_not_nil=1 FORMAT TSVRaw" | clickhouse-client -mn --echo

Then it probably should be like:

no, i provided command with pipeline inside bash -c
last message https://github.com/Altinity/clickhouse-operator/issues/1455#issuecomment-2461786918
described command which just need to executed in interactive shell

Or we just want to confirm what manual way of copying objects works?

Yes, i'm looking into clickhouse-operator source code and trying to execute schemer.go commands manually

no, i provided command with pipeline inside bash -c
last message #1455 (comment)
described command which just need to executed in interactive shell

Yes, they piped, but inside an interactive shell of 0 node. There are no cross-nodes connections.

look to FROM cluster('all-sharded', system.tables)