[SPARK-39195][SQL] Spark OutputCommitCoordinator should abort stage when committed file not consistent with task status

What changes were proposed in this pull request?
There is a case for current code may cause data correctness issue:

One writing data task commit task success, but the container was killed after commit task, so this task failed, but the data file still remained under committedTaskPath .
DAGScheduler call handleCompletedTask and then call outputCommitCoordinator.taskCompleted,  outputCommitCoordinator will remove the lock of failed task's partition.
Then failed task's rerun with an new attempt,  the new attempt task call outputCommitCoordinator.canCommit() will return true since the lock of this partition had been removed, then it commit task success, also task final  succeed.
Two files remained under this job's attempt path for  same partition.
CommitJob commit both two committed task path's data.
Finally data duplicated.

In this pr, we do below since:

When commit task success, executor send an CommitOutputSuccess message to  outputCommitCoordinator.
When outputCommitCoordinator handle taskComplete, if task failed but commit success, means data duplicate will happen,  we should failed to job.

Why are the changes needed?
Fix data duplicated issue.
Does this PR introduce any user-facing change?
No
How was this patch tested?

gentle ping @dongjoon-hyun @cloud-fan @HyukjinKwon @srowen Could you take a view of this data correctness issue?

ping @cloud-fan Could you take a look?

Doc failed seems not related to this pr, any more suggestion?

LGTM if tests pass

GA failed not related to this PR
__w/spark/spark/docs/_plugins/copy_api_dirs.rb:130:in `<top (required)>': Python doc generation failed (RuntimeError)
2022-06-11T07:16:53.8252035Z 	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/external.rb:60:in `require'
2022-06-11T07:16:53.8252686Z 	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/external.rb:60:in `block in require_with_graceful_fail'
2022-06-11T07:16:53.8253306Z 	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/external.rb:57:in `each'
2022-06-11T07:16:53.8253947Z 	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/external.rb:57:in `require_with_graceful_fail'
2022-06-11T07:16:53.8254613Z 	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/plugin_manager.rb:89:in `block in require_plugin_files'
2022-06-11T07:16:53.8255235Z 	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/plugin_manager.rb:87:in `each'
2022-06-11T07:16:53.8255870Z 	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/plugin_manager.rb:87:in `require_plugin_files'
2022-06-11T07:16:53.8256520Z 	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/plugin_manager.rb:21:in `conscientious_require'
2022-06-11T07:16:53.8257116Z 	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/site.rb:131:in `setup'
2022-06-11T07:16:53.8257695Z 	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/site.rb:36:in `initialize'
2022-06-11T07:16:53.8258278Z 	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/commands/build.rb:30:in `new'
2022-06-11T07:16:53.8258888Z 	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/commands/build.rb:30:in `process'
2022-06-11T07:16:53.8259731Z 	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/command.rb:91:in `block in process_with_graceful_fail'
2022-06-11T07:16:53.8260335Z 	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/command.rb:91:in `each'
2022-06-11T07:16:53.8260952Z 	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/command.rb:91:in `process_with_graceful_fail'
2022-06-11T07:16:53.8261625Z 	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/commands/build.rb:18:in `block (2 levels) in init_with_program'
2022-06-11T07:16:53.8262274Z 	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/mercenary-0.4.0/lib/mercenary/command.rb:221:in `block in execute'
2022-06-11T07:16:53.8262921Z 	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/mercenary-0.4.0/lib/mercenary/command.rb:221:in `each'
2022-06-11T07:16:53.8263524Z 	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/mercenary-0.4.0/lib/mercenary/command.rb:221:in `execute'
2022-06-11T07:16:53.8264134Z 	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/mercenary-0.4.0/lib/mercenary/program.rb:44:in `go'
2022-06-11T07:16:53.8264718Z 	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/mercenary-0.4.0/lib/mercenary.rb:21:in `program'
2022-06-11T07:16:53.8265284Z 	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/exe/jekyll:15:in `<top (required)>'
2022-06-11T07:16:53.8265698Z 	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/bin/jekyll:23:in `load'
2022-06-11T07:16:53.8266159Z 	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/bin/jekyll:23:in `<main>'

ping @cloud-fan

I see the same test failed above https://github.com/AngersZhuuuu/spark/runs/6893999574?check_suite_focus=true#step:9:15747

@AngersZhuuuu Mind taking a look please?

Sure

@AngersZhuuuu mind taking a look https://issues.apache.org/jira/browse/SPARK-39622? I don't think there's any other committer release changes except this .. cc @HeartSaVioR FYI

Yea

Hi @AngersZhuuuu. I came across problems with this changes after updating to spark 3.4
I write data to iceberg table with S3 backend and the data upload happens in dataWriter.commit() after coordinator.canCommit was called. So if uploading to S3 fails for some reason, the task fails and the partition data can't be uploaded anymore event in the task's retries, because the failed task remains to be the assigned commiter.
Looks like usually data writing is idempotent, because each partition is written into a separate file and you always can do it again in retrying task without data duplication.

I'm surprised that iceberg does not overwrite https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/connector/write/BatchWrite.java#L63
cc @huaxingao @dongjoon-hyun

🤔 so the option useCommitCoordinator is for appending to the same destination, like inserting rows to a remote database and we don't need it in iceberg?

@cloud-fan Thanks for pinging me. It appears that Iceberg doesn't override this useCommitCoordinator. I will take a look at this issue.

@huaxingao @cloud-fan, could you confirm only a single WriterCommitMessage will be passed in case of speculative execution even without the commit coordinator? Based on what I see in V2TableWriteExec, if multiple task attempts succeed, the last one will win by replacing the commit message from the first one. Am I right?

@aokolnychyi In most cases, yes. However, we have BatchWrite#onDataWriterCommit, so implementations can decide how to deal with conflicting commit messages, maybe first-win.

Thanks for confirming, @cloud-fan!

To @AngersZhuuuu , @viirya , @cloud-fan .
Since this PR is too old, shall we file a JIRA issue for that comment, https://github.com/apache/spark/pull/36564#discussion_r1598660630 ?

To @AngersZhuuuu , @viirya , @cloud-fan . Since this PR is too old, shall we file a JIRA issue for that comment, #36564 (comment) ?

Yea, I think so.

I created https://issues.apache.org/jira/browse/SPARK-48292

Task commits MUST be atomic and a second attempt MUST be able to supercede the first.
Hadoop FileOutputCommitter v1 (on hdfs, localfs, abfs, not gcs) uses atomic rename for this, deleting task completed data path first. A retry will delete that dest path and rename their own work directory to it.
S3A committers PUT a manifest to the task path; relies on PUT being atomic.
Mapreduce Manifest committer does write of manifest and then rename; this is atomic on GCS so safe there (dir rename is non atomic)
The application MUST NOT constrain which of two attempts told to commit succeeds, only that the second one MUST report success.
Why so? Because if task attempt TA1 stops responding after being told to commit, then TA2 will be told to commit, and reports success. But if TA1 is somehow suspended and performs its atomic commit after TA2, then it will be the TA1 manifest which is processed.
If you are encountering problems with jobs where task failures are unrecoverable, that means there is something wrong with the task commit algorithm.
What were you seeing it with?

To @steveloughran , for the better context, you had better see and post on https://github.com/apache/spark/pull/46696 instead of here.
BTW, for me, I can say that I don't see any issue with both SPARK-39195 and its revert (SPARK-48292) with S3 environment.

Hi all
I am facing this issue after upgrading to Spark3.5.1 and wonder if this change is a root cause for it. Can anyone here confirm?
Many of our jobs are failing with "Authorized committer" errors and we might have to revert our whole system back to Spark3.3, which would be a lot of work. So I am trying to understand the root issue a bit more before I have to go back reverting my whole Spark+Hadoop system.
Thanks!

Hi, I have been facing a similar issue as @akki  after migrating to Spark 3.5.0


Add github actions status badge

Fixes #38

Thanks for the PR @hndoss! I left one change request. After that, this will be good to merge!


User-defined kinds

Is this possible? as in take some base kind, e.g. linux and define my custom kind over it. I thought this was what topology.kind is supposed to do, but looks like it's not?
e.g. this is what i want to do
name: user-defined

topology:
  kind:
    my-custom-kind:
      type: linux
     image: myimage:latest

Custom kinds are not something you can create in the clab file
I wonder what would be the use case? To have a special set of variables scoped for this kind specifically?

yes. so instead of listing all those kind params for every node, you define them once and then just reference that custom kind from the nodes section

we will take it as a feature enhancement, and add it in the subsequent release.

This would be very useful as it will allow each custom kind to have a specific version or interface mapping (to emulate different hardware) in the case of cEOS. I.e.:
topology:
  kinds:
    ceos-7050SX3-48YC8:
      startup-config: ../../ceos_lab_template/ceos.cfg.tpl
      image: ceosimage:4.28.1F
      binds:
        - 7050SX3-48YC8.json:/mnt/flash/EosIntfMapping.json:ro
    ceos-7050CX3-32S:
      startup-config: ../../ceos_lab_template/ceos.cfg.tpl
      image: ceosimage:4.28.0F
      binds:
        - 7050CX3-32S.json:/mnt/flash/EosIntfMapping.json:ro
    linux:
      image: alpine-host

In the example above the custom kind is denoted by not using a reserved name. Otherwise a "template" could be used.

@perimore do you need a custom kind for that though?
looks like your example can be written as
topology:
  nodes:
    ceos:
      startup-config: ../../ceos_lab_template/ceos.cfg.tpl
      image: ceosimage:4.28.1F
      binds:
        - 7050SX3-48YC8.json:/mnt/flash/EosIntfMapping.json:ro
    ceos:
      startup-config: ../../ceos_lab_template/ceos.cfg.tpl
      image: ceosimage:4.28.0F
      binds:
        - 7050CX3-32S.json:/mnt/flash/EosIntfMapping.json:ro

Yes, it can be "unfolded" and written explicitly for each node. But the idea is to define custom node types with predefined options and not to repeat them. Some sort of inheritance.


Reference and error docs for Embedded Readmes

Fixes: NuGet/Home#9834
Updated the nuspec and msbuild-targets files in docs\reference to reflect new readme support.
Added new error docs for Readmes.
NOTE: We have to wait for the Embedded Readme feature to be completed and merged end-to-end before merging these docs.

Docs Build status updates of commit 22be737:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/reference/nuspec.md
:bulb:Suggestion
View
Details

docs/reference/errors-and-warnings/NU5038.md
:white_check_mark:Succeeded
View

docs/reference/errors-and-warnings/NU5039.md
:white_check_mark:Succeeded
View

docs/reference/errors-and-warnings/NU5040.md
:white_check_mark:Succeeded
View

docs/reference/errors-and-warnings/NU5041.md
:white_check_mark:Succeeded
View

docs/reference/errors-and-warnings/NU5042.md
:white_check_mark:Succeeded
View

docs/reference/msbuild-targets.md
:white_check_mark:Succeeded
View

docs/reference/nuspec.md

Line 509, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 520, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 531, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 544, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 557, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 589, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 602, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 617, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 630, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 643, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 656, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 669, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 680, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 718, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 726, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 737, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit de651e3:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/reference/nuspec.md
:bulb:Suggestion
View
Details

docs/reference/errors-and-warnings/NU5038.md
:white_check_mark:Succeeded
View

docs/reference/errors-and-warnings/NU5039.md
:white_check_mark:Succeeded
View

docs/reference/errors-and-warnings/NU5040.md
:white_check_mark:Succeeded
View

docs/reference/errors-and-warnings/NU5041.md
:white_check_mark:Succeeded
View

docs/reference/errors-and-warnings/NU5042.md
:white_check_mark:Succeeded
View

docs/reference/msbuild-targets.md
:white_check_mark:Succeeded
View

docs/reference/nuspec.md

Line 509, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 520, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 531, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 544, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 557, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 589, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 602, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 617, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 630, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 643, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 656, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 669, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 680, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 718, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 726, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 737, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit 6718eb4:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/reference/nuspec.md
:bulb:Suggestion
View
Details

docs/reference/errors-and-warnings/NU5038.md
:white_check_mark:Succeeded
View

docs/reference/errors-and-warnings/NU5039.md
:white_check_mark:Succeeded
View

docs/reference/errors-and-warnings/NU5040.md
:white_check_mark:Succeeded
View

docs/reference/errors-and-warnings/NU5041.md
:white_check_mark:Succeeded
View

docs/reference/msbuild-targets.md
:white_check_mark:Succeeded
View

docs/reference/nuspec.md

Line 509, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 520, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 531, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 544, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 557, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 589, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 602, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 617, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 630, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 643, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 656, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 669, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 680, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 718, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 726, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 737, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit 0a3a0d9:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/reference/nuspec.md
:bulb:Suggestion
View
Details

docs/reference/errors-and-warnings/NU5038.md
:white_check_mark:Succeeded
View

docs/reference/errors-and-warnings/NU5039.md
:white_check_mark:Succeeded
View

docs/reference/errors-and-warnings/NU5040.md
:white_check_mark:Succeeded
View

docs/reference/errors-and-warnings/NU5041.md
:white_check_mark:Succeeded
View

docs/reference/msbuild-targets.md
:white_check_mark:Succeeded
View

docs/reference/nuspec.md

Line 552, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 563, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 574, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 587, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 600, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 632, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 645, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 660, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 673, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 686, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 699, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 712, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 723, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 761, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 769, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 780, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

@chgill-MSFT To keep PRs fresh as possible, can you convert this to a draft if this will not be worked on in the next month or so? Thanks!

Docs Build status updates of commit 80dfa96:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/reference/errors-and-warnings/NU5038.md
:bulb:Suggestion
View
Details

docs/reference/errors-and-warnings/NU5039.md
:bulb:Suggestion
View
Details

docs/reference/errors-and-warnings/NU5040.md
:bulb:Suggestion
View
Details

docs/reference/errors-and-warnings/NU5041.md
:bulb:Suggestion
View
Details

docs/reference/msbuild-targets.md
:white_check_mark:Succeeded
View

docs/reference/nuspec.md
:white_check_mark:Succeeded
View

docs/reference/errors-and-warnings/NU5038.md

Line 5, Column 12: [Suggestion-ms-author-invalid] Invalid value for 'ms.author', 't-adtand' is not a valid Microsoft alias.

docs/reference/errors-and-warnings/NU5039.md

Line 5, Column 12: [Suggestion-ms-author-invalid] Invalid value for 'ms.author', 't-adtand' is not a valid Microsoft alias.

docs/reference/errors-and-warnings/NU5040.md

Line 5, Column 12: [Suggestion-ms-author-invalid] Invalid value for 'ms.author', 't-adtand' is not a valid Microsoft alias.

docs/reference/errors-and-warnings/NU5041.md

Line 5, Column 12: [Suggestion-ms-author-invalid] Invalid value for 'ms.author', 't-adtand' is not a valid Microsoft alias.

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


compact: once it run out of disk space, it keeps running but is still broken

Thanos, Prometheus and Golang version used:
# thanos --version
thanos, version 0.34.1 (branch: HEAD, revision: 4cf1559998bf6d8db3f9ca0fde2a00d217d4e23e)
  build user:       root@61db75277a55
  build date:       20240219-17:13:48
  go version:       go1.21.7
  platform:         linux/amd64
  tags:             netgo

Object Storage Provider:
filesystem
What happened:
I run several times now into the problem, that compact ran out of local disk space, e.g. with an error like:
Mar 04 05:48:59 foo thanos@compact[2138629]: ts=2024-03-04T04:48:59.919722752Z caller=compact.go:491 level=error msg="critical error detected; halting" err="compaction: group 0@2951439482451486615: compact blocks [/data/main/thanos/compact/cache/compact/0@2951439482451486615/01HNPSBZE5B3W20MPYBB1ANBC9 /data/main/thanos/compact/cache/compact/0@2951439482451486615/01HNVY3CMAQGJ5FJVXFJ0HF8MX /data/main/thanos/compact/cache/compact/0@2951439482451486615/01HP12S5NVRYB1HGRRZF3WGHYS /data/main/thanos/compact/cache/compact/0@2951439482451486615/01HP67KTYPRMZ7REX5D5P7HBRH /data/main/thanos/compact/cache/compact/0@2951439482451486615/01HPBCDE60PGQT0NVJMAZN4REY /data/main/thanos/compact/cache/compact/0@2951439482451486615/01HPGH0QTWZG37HR1TVA3YWKHQ /data/main/thanos/compact/cache/compact/0@2951439482451486615/01HPNNYRYMC8F7PVP67THGF7JA]: 2 errors: populate block: write chunks: preallocate: no space left on device; sync<EMAIL_ADDRESS>file already closed"

The problem is, once this happened, it seems to be in a broken state and even if disk space gets free, it doesn't try again. It doesn't exit with a failure either, so there's no real chance for a service manager like systemd to notice the problem (and e.g. send mail or so).
What you expected to happen:
There should be an option for each component (including compact), which causes it to exit immediately with some non-zero exit status in case of an error.
Cheers,
Chris.

Options are:
/usr/bin/thanos compact --log.level=warn --wait --data-dir=/data/main/thanos/compact/cache --objstore.config-file=/etc/thanos/data-thanos.yml --consistency-delay=1h --hash-func=SHA256 --http-address=:10912 --web.external-prefix=/thanos/compact --web.route-prefix=/

But I just found --no-debug.halt-on-error which seems as if it may be causing the right behaviour.
I wonder why that isn't the default.
The description why not, is honestly not very convincing.
Yes, it would be bad to retry over and over if it just keeps failing, but there are much better ways to handle this.
a) Not retrying endlessly is anyway the service manager’s (i.e. systemd’s) responsibility, and there are plenty of proper ways in that to handle this and b) thanos could e.g. just set some flag on such error condition, that lets it fail immediately without retrying (which would still allow for the important part, namely failing - otherwise people in normal environments won't notice that anything's wrong).

I am seeing a similar behavior where it errors once (no space left on device), halts and never compacts again.
level=error ts=2024-03-19T16:41:39.30753731Z caller=compact.go:486 msg="critical error detected; halting" err="compaction: group 0@16225028991790494220: compact blocks [/data/compact/0@16225028991790494220/01F3EZ3ZAA58NSNX2EPRS03N5H /data/compact/0@16225028991790494220/01F3M3W8CDMFQP1ZTS72F07EGQ /data/compact/0@16225028991790494220/01F3S8RRSXCY18WNCS6NS6T1N4 /data/compact/0@16225028991790494220/01F3YDPP5RRD4FFERPTVTRY86T /data/compact/0@16225028991790494220/01F43J9M8VME00PDVN2FQ84SFH /data/compact/0@16225028991790494220/01F48Q59D5Z77DG93SGEYVZ8B1 /data/compact/0@16225028991790494220/01F4DW1XNESEQH4RBW5AZMN6P4]: 3 errors: populate block: write chunks: preallocate: no space left on device; sync<EMAIL_ADDRESS>file already closed; write<EMAIL_ADDRESS>no space left on device"
level=info ts=2024-03-19T16:42:04.648266954Z caller=fetcher.go:470 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=2.059553102s duration_ms=2059 cached=40630 returned=40630 partial=0
level=info ts=2024-03-19T16:42:32.306773124Z caller=fetcher.go:470 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=43.087680327s duration_ms=43087 cached=40630 returned=40610 partial=0
level=info ts=2024-03-19T16:43:15.869657891Z caller=fetcher.go:470 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=43.552571092s duration_ms=43552 cached=40630 returned=40610 partial=0
level=info ts=2024-03-19T16:43:15.88204689Z caller=clean.go:34 msg="started cleaning of aborted partial uploads"
level=info ts=2024-03-19T16:43:15.88207772Z caller=clean.go:61 msg="cleaning of aborted partial uploads done"
level=info ts=2024-03-19T16:43:15.882085075Z caller=blocks_cleaner.go:44 msg="started cleaning of blocks marked for deletion"
level=info ts=2024-03-19T16:43:15.882099343Z caller=blocks_cleaner.go:58 msg="cleaning of blocks marked for deletion done"

To be clear: you won't ever recover Compactor running out of disk space without giving it more disk. I agree with the suggested expected behavior from @calestyo though, the Compactor should exit when it detects it doesn't have enough disk space available.
To fix the Compactor:

scale Compactor to zero
expand the PVs if you can or give it bigger ones
scle it back up.

Finally when the Compactor catches up you can follow a similar procedure to shrink the PV.

you won't ever recover Compactor running out of disk space without giving it more disk.

The problem is however, even with more disk (respectively space freed) it never recovers and does anything useful again.
I think the default behaviour of daemons is to fail in such situation (so that the service manager, like systemd, can react)... a --keep-going mode might be nice for some people, but then only if it actually ever does something again (which seems not the case).

The problem is however, even with more disk (respectively space freed) it never recovers and does anything useful again.

You will need to give it more total disk space than it had before. The amount of extra space required is proportional to the time the Compactor had issues. I've seen setups where the Compactor had initially about 200 GB of disk and we had to give it 4x more.
As I said, I agree with you that the Compactor's default behavior is not ideal. We should improve it.

You will need to give it more total disk space than it had before.

I don't see why that should need to be the case. The filesystem could have had some other big occupier of space and with that being gone it could be just enough for compact, even with the total size being the same.
The other issue you're referring to, that it needs more and more space, is what I've described in #7198, which is probably a duplicate of #3405.
IMO the proper behaviour would be as follows:

Per default, if it runs out of space and cannot finish it's operation it should exit with a non-zero status
If some extra option is given, it may continue to run and continuously retry (perhaps with some delay in between or/and only if the free space is bigger), but without any extra conditions like the filesystem having gained more total space. But that mode only makes sense if some other system would be in place that automatically increases he space or frees up space - and if there's anyway such a system, it wouldn't cost much IMO, to then just restart compact... so in a way the whole mode of continuing to run makes IMO not that much sense.

@douglascamata Could you please remove needs-more-info? This is no longer the case.

Yet another case where compact runs into troubles, never fails, never recovers, no-one will ever notice that it's actually no longer doing anything:
For some reasons (don't ask me how that happens, perhaps another bug in Thanos?) overlapping blocks showed up:
Jul 04 05:59:11 lcg-lrz-monitor-ext thanos@compact[2524381]: ts=2024-07-04T03:59:11.880887578Z caller=main.go:161 level=error err="group 0@16428505770164304275: pre compaction overlap check: overlaps found while gathering blocks. [mint:<PHONE_NUMBER>708, maxt:<PHONE_NUMBER>000, range: 1h59m57s, blocks: 2]: <ulid: 01J0Z8WZBKK3G2VVVJN70WQN32, mint:<PHONE_NUMBER>085, maxt:<PHONE_NUMBER>000, range: 47h59m59s>, <ulid: 01J10W3EJT73RS0W5EPMFMKK82, mint:<PHONE_NUMBER>708, maxt:<PHONE_NUMBER>000, range: 1h59m57s>\ncompaction\nmain.runCompact.func7\n\t/app/cmd/thanos/compact.go:432\nmain.runCompact.func8.1\n\t/app/cmd/thanos/compact.go:517\ngithub.com/thanos-io/thanos/pkg/runutil.Repeat\n\t/app/pkg/runutil/runutil.go:74\nmain.runCompact.func8\n\t/app/cmd/thanos/compact.go:516\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650\ncritical error detected\nmain.runCompact.func8.1\n\t/app/cmd/thanos/compact.go:531\ngithub.com/thanos-io/thanos/pkg/runutil.Repeat\n\t/app/pkg/runutil/runutil.go:74\nmain.runCompact.func8\n\t/app/cmd/thanos/compact.go:516\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650\ncompact command failed\nmain.main\n\t/app/cmd/thanos/main.go:161\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"
Jul 04 05:59:19 lcg-lrz-monitor-ext thanos@compact[2524442]: ts=2024-07-04T03:59:19.844193328Z caller=intrumentation.go:67 level=warn msg="changing probe status" status=not-ready reason="critical error detected: compaction: group 0@16428505770164304275: pre compaction overlap check: overlaps found while gathering blocks. [mint:<PHONE_NUMBER>708, maxt:<PHONE_NUMBER>000, range: 1h59m57s, blocks: 2]: <ulid: 01J0Z8WZBKK3G2VVVJN70WQN32, mint:<PHONE_NUMBER>085, maxt:<PHONE_NUMBER>000, range: 47h59m59s>, <ulid: 01J10W3EJT73RS0W5EPMFMKK82, mint:<PHONE_NUMBER>708, maxt:<PHONE_NUMBER>000, range: 1h59m57s>"
Jul 04 05:59:19 lcg-lrz-monitor-ext thanos@compact[2524442]: ts=2024-07-04T03:59:19.844552963Z caller=main.go:161 level=error err="group 0@16428505770164304275: pre compaction overlap check: overlaps found while gathering blocks. [mint:<PHONE_NUMBER>708, maxt:<PHONE_NUMBER>000, range: 1h59m57s, blocks: 2]: <ulid: 01J0Z8WZBKK3G2VVVJN70WQN32, mint:<PHONE_NUMBER>085, maxt:<PHONE_NUMBER>000, range: 47h59m59s>, <ulid: 01J10W3EJT73RS0W5EPMFMKK82, mint:<PHONE_NUMBER>708, maxt:<PHONE_NUMBER>000, range: 1h59m57s>\ncompaction\nmain.runCompact.func7\n\t/app/cmd/thanos/compact.go:432\nmain.runCompact.func8.1\n\t/app/cmd/thanos/compact.go:517\ngithub.com/thanos-io/thanos/pkg/runutil.Repeat\n\t/app/pkg/runutil/runutil.go:74\nmain.runCompact.func8\n\t/app/cmd/thanos/compact.go:516\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650\ncritical error detected\nmain.runCompact.func8.1\n\t/app/cmd/thanos/compact.go:531\ngithub.com/thanos-io/thanos/pkg/runutil.Repeat\n\t/app/pkg/runutil/runutil.go:74\nmain.runCompact.func8\n\t/app/cmd/thanos/compact.go:516\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650\ncompact command failed\nmain.main\n\t/app/cmd/thanos/main.go:161\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"
Jul 04 05:59:27 lcg-lrz-monitor-ext thanos@compact[2524512]: ts=2024-07-04T03:59:27.839397302Z caller=intrumentation.go:67 level=warn msg="changing probe status" status=not-ready reason="critical error detected: compaction: group 0@16428505770164304275: pre compaction overlap check: overlaps found while gathering blocks. [mint:<PHONE_NUMBER>708, maxt:<PHONE_NUMBER>000, range: 1h59m57s, blocks: 2]: <ulid: 01J0Z8WZBKK3G2VVVJN70WQN32, mint:<PHONE_NUMBER>085, maxt:<PHONE_NUMBER>000, range: 47h59m59s>, <ulid: 01J10W3EJT73RS0W5EPMFMKK82, mint:<PHONE_NUMBER>708, maxt:<PHONE_NUMBER>000, range: 1h59m57s>"
Jul 04 05:59:27 lcg-lrz-monitor-ext thanos@compact[2524512]: ts=2024-07-04T03:59:27.840081544Z caller=main.go:161 level=error err="group 0@16428505770164304275: pre compaction overlap check: overlaps found while gathering blocks. [mint:<PHONE_NUMBER>708, maxt:<PHONE_NUMBER>000, range: 1h59m57s, blocks: 2]: <ulid: 01J0Z8WZBKK3G2VVVJN70WQN32, mint:<PHONE_NUMBER>085, maxt:<PHONE_NUMBER>000, range: 47h59m59s>, <ulid: 01J10W3EJT73RS0W5EPMFMKK82, mint:<PHONE_NUMBER>708, maxt:<PHONE_NUMBER>000, range: 1h59m57s>\ncompaction\nmain.runCompact.func7\n\t/app/cmd/thanos/compact.go:432\nmain.runCompact.func8.1\n\t/app/cmd/thanos/compact.go:517\ngithub.com/thanos-io/thanos/pkg/runutil.Repeat\n\t/app/pkg/runutil/runutil.go:74\nmain.runCompact.func8\n\t/app/cmd/thanos/compact.go:516\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650\ncritical error detected\nmain.runCompact.func8.1\n\t/app/cmd/thanos/compact.go:531\ngithub.com/thanos-io/thanos/pkg/runutil.Repeat\n\t/app/pkg/runutil/runutil.go:74\nmain.runCompact.func8\n\t/app/cmd/thanos/compact.go:516\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650\ncompact command failed\nmain.main\n\t/app/cmd/thanos/main.go:161\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"
Jul 04 05:59:33 lcg-lrz-monitor-ext thanos@compact[2524570]: ts=2024-07-04T03:59:33.281127456Z caller=intrumentation.go:67 level=warn msg="changing probe status" status=not-ready reason="critical error detected: compaction: group 0@16428505770164304275: pre compaction overlap check: overlaps found while gathering blocks. [mint:<PHONE_NUMBER>708, maxt:<PHONE_NUMBER>000, range: 1h59m57s, blocks: 2]: <ulid: 01J0Z8WZBKK3G2VVVJN70WQN32, mint:<PHONE_NUMBER>085, maxt:<PHONE_NUMBER>000, range: 47h59m59s>, <ulid: 01J10W3EJT73RS0W5EPMFMKK82, mint:<PHONE_NUMBER>708, maxt:<PHONE_NUMBER>000, range: 1h59m57s>"
Jul 04 05:59:33 lcg-lrz-monitor-ext thanos@compact[2524570]: ts=2024-07-04T03:59:33.285630641Z caller=compact.go:633 level=error msg="retriable error" err="BaseFetcher: iter bucket: context canceled"
Jul 04 05:59:33 lcg-lrz-monitor-ext thanos@compact[2524570]: ts=2024-07-04T03:59:33.286061605Z caller=main.go:161 level=error err="group 0@16428505770164304275: pre compaction overlap check: overlaps found while gathering blocks. [mint:<PHONE_NUMBER>708, maxt:<PHONE_NUMBER>000, range: 1h59m57s, blocks: 2]: <ulid: 01J0Z8WZBKK3G2VVVJN70WQN32, mint:<PHONE_NUMBER>085, maxt:<PHONE_NUMBER>000, range: 47h59m59s>, <ulid: 01J10W3EJT73RS0W5EPMFMKK82, mint:<PHONE_NUMBER>708, maxt:<PHONE_NUMBER>000, range: 1h59m57s>\ncompaction\nmain.runCompact.func7\n\t/app/cmd/thanos/compact.go:432\nmain.runCompact.func8.1\n\t/app/cmd/thanos/compact.go:517\ngithub.com/thanos-io/thanos/pkg/runutil.Repeat\n\t/app/pkg/runutil/runutil.go:74\nmain.runCompact.func8\n\t/app/cmd/thanos/compact.go:516\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650\ncritical error detected\nmain.runCompact.func8.1\n\t/app/cmd/thanos/compact.go:531\ngithub.com/thanos-io/thanos/pkg/runutil.Repeat\n\t/app/pkg/runutil/runutil.go:74\nmain.runCompact.func8\n\t/app/cmd/thanos/compact.go:516\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650\ncompact command failed\nmain.main\n\t/app/cmd/thanos/main.go:161\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"
Jul 04 05:59:38 lcg-lrz-monitor-ext thanos@compact[2524630]: ts=2024-07-04T03:59:38.626967738Z caller=intrumentation.go:67 level=warn msg="changing probe status" status=not-ready reason="critical error detected: compaction: group 0@16428505770164304275: pre compaction overlap check: overlaps found while gathering blocks. [mint:<PHONE_NUMBER>708, maxt:<PHONE_NUMBER>000, range: 1h59m57s, blocks: 2]: <ulid: 01J0Z8WZBKK3G2VVVJN70WQN32, mint:<PHONE_NUMBER>085, maxt:<PHONE_NUMBER>000, range: 47h59m59s>, <ulid: 01J10W3EJT73RS0W5EPMFMKK82, mint:<PHONE_NUMBER>708, maxt:<PHONE_NUMBER>000, range: 1h59m57s>"
Jul 04 05:59:38 lcg-lrz-monitor-ext thanos@compact[2524630]: ts=2024-07-04T03:59:38.630323903Z caller=compact.go:604 level=error msg="retriable error" err="syncing metas: BaseFetcher: iter bucket: context canceled"
Jul 04 05:59:38 lcg-lrz-monitor-ext thanos@compact[2524630]: ts=2024-07-04T03:59:38.630590996Z caller=main.go:161 level=error err="group 0@16428505770164304275: pre compaction overlap check: overlaps found while gathering blocks. [mint:<PHONE_NUMBER>708, maxt:<PHONE_NUMBER>000, range: 1h59m57s, blocks: 2]: <ulid: 01J0Z8WZBKK3G2VVVJN70WQN32, mint:<PHONE_NUMBER>085, maxt:<PHONE_NUMBER>000, range: 47h59m59s>, <ulid: 01J10W3EJT73RS0W5EPMFMKK82, mint:<PHONE_NUMBER>708, maxt:<PHONE_NUMBER>000, range: 1h59m57s>\ncompaction\nmain.runCompact.func7\n\t/app/cmd/thanos/compact.go:432\nmain.runCompact.func8.1\n\t/app/cmd/thanos/compact.go:517\ngithub.com/thanos-io/thanos/pkg/runutil.Repeat\n\t/app/pkg/runutil/runutil.go:74\nmain.runCompact.func8\n\t/app/cmd/thanos/compact.go:516\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650\ncritical error detected\nmain.runCompact.func8.1\n\t/app/cmd/thanos/compact.go:531\ngithub.com/thanos-io/thanos/pkg/runutil.Repeat\n\t/app/pkg/runutil/runutil.go:74\nmain.runCompact.func8\n\t/app/cmd/thanos/compact.go:516\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650\ncompact command failed\nmain.main\n\t/app/cmd/thanos/main.go:161\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"
Jul 04 05:59:45 lcg-lrz-monitor-ext thanos@compact[2524687]: ts=2024-07-04T03:59:45.697592546Z caller=intrumentation.go:67 level=warn msg="changing probe status" status=not-ready reason="critical error detected: compaction: group 0@16428505770164304275: pre compaction overlap check: overlaps found while gathering blocks. [mint:<PHONE_NUMBER>708, maxt:<PHONE_NUMBER>000, range: 1h59m57s, blocks: 2]: <ulid: 01J0Z8WZBKK3G2VVVJN70WQN32, mint:<PHONE_NUMBER>085, maxt:<PHONE_NUMBER>000, range: 47h59m59s>, <ulid: 01J10W3EJT73RS0W5EPMFMKK82, mint:<PHONE_NUMBER>708, maxt:<PHONE_NUMBER>000, range: 1h59m57s>"

Despite compact being invoked with --no-debug.halt-on-error it just mindlessly tries over and over. The reasons – I don't know, cause how should the overlapping blocks disappear if nobody notices them (unless by chance), because the service doesn't fail properly?

@calestyo do you have vertical compaction enabled run version 0.35 or above? There's literally a PR that intentionally made Compactor not halt on overlapping blocks errors: https://github.com/thanos-io/thanos/pull/7225.

@calestyo do you have vertical compaction enabled run version 0.35 or above? There's literally a PR that intentionally made Compactor not halt on overlapping blocks errors: #7225.

I don't think I have it enabled, in particular I don't use --compact.enable-vertical-compaction

We are not aware of any bug in Thanos that creates overlapping blocks. I think this is more about how you use the tool rather than a defect in the tool itself. I never saw Thanos create overlapping blocks in the deployments I oversee.

No idea. My setup is rather simple. I have two Prometheuses running, both scraping the exact same metrics/instances (well okay the prometheus and thanos metrics are not scraped by both).
Thanos get's their data, for one via sidecar and for the other via receive.
I still see the problem that it seems to require more and more space for compaction (despite the queried metrics and number of instances staying the same - and thus space requirements should also stay the same?!).

The blue one is a sshfs shared filesystem, which is my TSDB bucket store, that grows steadily (which is of course expected). Sometimes it goes a bit down (which I guess is when it cleans up after compaction, but might also be unrealted backups that are written to the same shared fs).
The red one is more interesting, it's the local storage used solely by receive (which uses up 16GB in my case, --tsdb.max-retention-bytes=16GB), store (which seems to never need much more than 100MB) and compact.
Most of the time, compact seems to behave well, and you see it also using a constance amount (around 20%) of the fs.
But ever now and then it fills the fs.
And you can also see that it does not happen every 14days (if it was that, I'd have said it would simply need more, space when I compact to the big chunks).
I can only recover from that be temporarily stopping receive, move it's data away and then let compact run. Or of course by adding more space, but I did so already 3 times, originally the fs had 40 GB, then 80GB, now 120 GB or so.

btw: I've also noticed, that every time when the problem occurs (i.e. fs is full, compact effectively broken, but doesn't fail properly as it should), then the scraping of the thanos_* metrics gets unstable (i.e. I loose single scrapes every know and then).


feat: remove course controller

Description
Remove course controller and menu in favor of https://github.com/Innoactive/Basic-UI-Component/pull/1
Type of change

[X] New feature (non-breaking change which adds functionality)
[X] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[X] This change requires a documentation update

How Has This Been Tested?
Get https://github.com/Innoactive/Basic-UI-Component/pull/1
and https://github.com/Innoactive/Creator-Pro/pull/13

Scenes that previously had a [COURSE_CONTROLLER] are now broken and no training course could be played without setting them up again.

Scenes that previously had a [COURSE_CONTROLLER] are now broken and no training course could be played (no play button) without setting them up again.

This does not happen when importing everything from scratch

Any ideas how to prevent/fix this @Gusinuhe ?

Any ideas how to prevent/fix this @Gusinuhe ?

I'm gonna test how critical this is by updating a project from an older version to this one hopefully, the issue is solved automatically.
The thing is that when it happened the first time, I cloned everything in develop, imported it to Unity, and then started to checkout individually each branch, this repo was the last one so I did have two [COURSE_CONTROLLER] in the project and that could actually be the reason why it was missing the first time.

tested this earlier on and it work so far, but my instance of the prefab was not linked earlier on. So the problem gustavo mentioned was not tested.

Tested again, the issue is not present when "upgrading" the creator, it only appears when switching manually the branches on a very specific combination. This should not be an issue that any user would face.


PR'ing that widen function I mentioned a while ago. :)

Fair point... that said, I ended up going even deeper down the rabbit hole which fundamentally shifted my approach to this whole "widen"ing thing.
But it makes the widen function more predictable.
That is, I realized that working in sRGB / HSL makes reasoning about accessible contrast a bit of a nightmare and that LCH just "fixes" it.
So I pulled the JS from https://css.land/lch/ and translated most of the JS to elm given most of it was pure functions anyways.
Basing my theme-ing primitives on LCH means I can adjust the Lightness of any hue according to this 10-step system.
Which means widen is just +/- 40 Lightness (for AA) and 50 (for AAA) for any hue of any color.
Ultimately I thought I could use elm-color-extra's Accessible module to do the lch conversion but it didn't have functions for converting True LCH back into the sRGB space which is why I had to adapt the css.land/lch functions.
ALL THAT said, not sure whether to make a new elm package just for LCH or add it to elm-color-extra or adapt the functions to work with this repo...
I'd love more eyes on the work and maybe some guidance / rubber 🦆'ing, I'm pretty much always available on Slack at @erlandsona if you're interested or want to talk more?
Feel free to do whatever with this.

Fair point... that said, I ended up going even deeper down the rabbit hole which fundamentally shifted my approach to this whole "widen"ing thing.
But it makes the widen function more predictable.
That is, I realized that working in sRGB / HSL makes reasoning about accessible contrast a bit of a nightmare and that LCH just "fixes" it.
So I pulled the JS from https://css.land/lch/ and translated most of the JS to elm given most of it was pure functions anyways.
Basing my theme-ing primitives on LCH means I can adjust the Lightness of any hue according to this 10-step system.
Which means widen is just +/- 40 Lightness (for AA) and 50 (for AAA) for any hue of any color.
Ultimately I thought I could use elm-color-extra's Accessible module to do the lch conversion but it didn't have functions for converting True LCH back into the sRGB space which is why I had to adapt the css.land/lch functions.
ALL THAT said, not sure whether to make a new elm package just for LCH or add it to elm-color-extra or adapt the functions to work with this repo...
I'd love more eyes on the work and maybe some guidance / rubber 🦆'ing, I'm pretty much always available on Slack at @erlandsona if you're interested or want to talk more?
Feel free to do whatever with this.


Slack kill switch

A command to make the bot exit immediately in case there is a problem but you can only access via slack.

I don't think I'm going to invest time in making Slack more than just a log channel unless more people express an interest in it. I haven't really needed it so far.


passes outside bounding box

I'm getting passes that are nowhere near the bounding box I'm inputting:
podaac-data-downloader -b="-117.5066681,32.6166649,-117.0066681,33.1166649" -c SWOT_L2_LR_SSH_UNSMOOTHED_2.0 -d ./raw -sd 2023-06-01T00:00:00Z -ed 2024-07-01T13:59:55Z

Hi! I’m fairly confident this is caused by a metadata bug that affects spatial queries for SSH products from SWOT calibration phase. Do you have the name of a file that was included erroneously? That would be good enough to confirm.
Please note there’s nothing wrong with the actual data in the files — you’re just getting more files than expected because of over-selection resulting from the metadata bug (that’s my strong suspicion, anyway). The issue should be fixed some time during the next couple weeks.

Hi @jjmcnelis here's a filename: SWOT_L2_LR_SSH_Expert_017_078_20240621T193629_20240621T202757_PIC0_01
Thanks!


Java EE 8 sample (IBM WebSphere Liberty is used as a Java EE 8 implementation)

The sample is a Java EE application exposing a RESTful web-service + an integration test for the application.
To run the integration test on VirtualBox on Windows (or Mac) do the following:
# minikube start

# gofabric8 docker-env

# @FOR /f "tokens=*" %i IN ('minikube docker-env') DO @%i

# cd fabric8-maven-plugin/samples/javaee8-webprofile-liberty

# mvn clean verify -DtestProxyHost=<IP_ADDRESS>

The sample is related to #1140 and demonstrates how to get the port number dynamically generated by Kubernetes for the exposed service. The org.codehaus.gmaven:groovy-maven-plugin is used.
P.S. If the integration test fails with error 404, just try to increase the value of the startPodPause property in the javaee8-webprofile-liberty-app-it/pom.xml file. The start goal is background and a pause is needed to let the Liberty profile start inside a docker container.
Closes #1140

Extremely sorry for the delay. I missed your reply in this thread, I would retest it.

@piyush1594 Could you please check on your side as well if you have some time? It has been hanging for quite a long time on our end. Things look good on code side, we just need to verify whether it's working on Linux.

@samolisov @rohanKanojia I tested it according to the steps mentioned, I increased the time as mentioned to 18000, but its getting failed for me.
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]   MeetingBookingServiceIT.testBaseline:57 » Processing java.net.ConnectException...

I am using minishift.

@piyush1594 could you share the full log, please?

Hey @samolisov
You can check the logs here https://pastebin.com/uP5T3YMc
Thanks

I've looked a bit about
[ERROR] 1/0 pod(s) ready for DeploymentConfig: javaee8-webprofile-liberty-app-it in namespace: fabric8  after waiting for 600 seconds so giving up

... and found the following issue of minishift on GitHub. Could you repeat, please, and share the output of minishift logs or something like oc logs -f javaee8-webprofile-liberty-app-it-1-s692r -n fabric8.

I had already approved this PR in the past but I was waiting on feedback for other members also. But I think I'm gonna try this myself now.
So right now on minishift, this doesn't seem to be working properly. Application pod does come in Ready state but tests fail. I tried it with minikube and this sample seems to be working here. Please take a look at my build logs.
~/work/repos/fabric8-maven-plugin/samples/javaee8-webprofile-liberty : $ oc get pods -w -n fabric8
NAME                                                READY     STATUS    RESTARTS   AGE
javaee8-webprofile-liberty-app-it-5f9866774-kqqhp   0/1       Pending   0          0s
javaee8-webprofile-liberty-app-it-5f9866774-kqqhp   0/1       Pending   0         0s
javaee8-webprofile-liberty-app-it-5f9866774-kqqhp   0/1       ContainerCreating   0         0s
javaee8-webprofile-liberty-app-it-5f9866774-kqqhp   1/1       Running   0         1s

~/work/repos/fabric8-maven-plugin/samples/javaee8-webprofile-liberty : $ kubectl logs javaee8-webprofile-liberty-app-it-5f9866774-kq
Launching defaultServer (WebSphere Application Server 2<IP_ADDRESS>/wlp-<IP_ADDRESS>80629-0500) on IBM J9 VM, version <IP_ADDRESS> - pxa6480sr5fp17-20180627_01(SR5 FP17) (en_US)
[AUDIT   ] CWWKE0001I: The server defaultServer has been launched.
[AUDIT   ] CWWKE0104I: This product is a beta and cannot be used in production. The full license terms can be viewed here: https://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/wasdev/license/early_access/ilar/2<IP_ADDRESS>/lafiles/en.html
[AUDIT   ] CWPKI0820A: The default keystore has been created using the 'keystore_password' environment variable.
[AUDIT   ] CWWKS4104A: LTPA keys created in 1.877 seconds. LTPA key file: /opt/ibm/wlp/output/defaultServer/resources/security/ltpa.keys
[AUDIT   ] CWPKI0803A: SSL certificate created in 2.084 seconds. SSL key file: /opt/ibm/wlp/output/defaultServer/resources/security/key.jks
[WARNING ] implicit.bean.scanning.disabled.CWOWB1009W
[AUDIT   ] CWWKT0016I: Web application available (default_host): http://javaee8-webprofile-liberty-app-it-5f9866774-kqqhp:9080/libertyapi/
[AUDIT   ] CWWKZ0001I: Application javaee8-webprofile-liberty-app started in 1.881 seconds.
[AUDIT   ] CWWKF0012I: The server installed the following features: [jsp-2.3, jsonb-1.0, ejbLite-3.2, managedBeans-1.0, beanValidation-2.0, servlet-4.0, jsf-2.3, ssl-1.0, jndi-1.0, cdi-2.0, jdbc-4.2, appSecurity-3.0, jsonp-1.1, appSecurity-2.0, jaxrsClient-2.1, el-3.0, jaxrs-2.1, jpaContainer-2.2, webProfile-8.0, jaspic-1.1, distributedMap-1.0, jpa-2.2, websocket-1.1].
[AUDIT   ] CWWKF0011I: The server defaultServer is ready to run a smarter planet.
~/work/repos/fabric8-maven-plugin/samples/javaee8-webprofile-liberty : $

I'm going to have this in since it is working for plain Kubernetes. Even if this doesn't work on Openshift, having an additional kubernetes sample should do no harm ;-)
@samolisov: Extremely sorry for the delay. This should have been merged a long time ago.


RDM-9370: Ignore un-known properties during json deserialisation

JIRA link
https://tools.hmcts.net/jira/browse/RDM-9370
Change description
Ignore un-known properties during json deserialisation
Work checklist

[ X] Unit tests added where applicable
[ ] Route tests added for new pages
[ ] New pages included in a11y tests
[ ] Task list and task completeness checks updated
[ ] Check and send page updated
[ ] Routes, page content and page flows of new features are toggled
[ ] UI changes look good on mobile
[ ] Required Google Analytics events are being sent

Developer self-QA run statement

[ ] I have clicked through the running application to see if all changes I made actually work.

Does this PR introduce a breaking change?

[ ] Yes
[ X] No

merged changes into master branch


Redirect to login page on IIS HTTPS problem

After I publish my application on IIS server with enabled HTTPS I have problem with login page after accessing to page that required authorization. For some reason it is redirected to the login page without the https protocol. As a result, I get a 404 error. If I allow http protocol on the IIS server, then there is no problem. I can make an authorization on the http protocol. Then I can again enter https address and continue to work normally. I do not understand why identity sends me on an unprotected protocol for authentication?
I use Microsoft.AspNetCore.Mvc 1.0.0-rc2-17051.
Thanks for the help.

@Tratcher Can you take a look at this?

When you're running Kestrel behind IIS, IIS forwards all requests to kestrel using http. If the request was originally https, IIS will include the x-forwarded-proto: https header. In RC1 the UseIISPlatformHandler middleware would take that header and update the request scheme. In RC2 this functionality has moved to UseForwardedHeaders. See https://github.com/aspnet/Announcements/issues/123

This code in Startup.cs solved my problem:
app.UseForwardedHeaders(new ForwardedHeadersOptions
{
    ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto
});
app.UseHttpMethodOverride();

This is a temporary solution or not?

That is the recommended pattern, though you don't need UseHttpMethodOverride.

 If the problem has been solved, do I have to close this issue or is it a global problem and does it require other solution?


Copy snippet code is not working

Works for me, tested on mac. Confirmation message is messing.

Lets move to js only approach https://clipboardjs.com/

replaced plugin, but I am not sure if it works on Mac Safari.

Doesn't work on safari, any solution?


[SPARK-47945][SQL] MsSQLServer: Document Mapping Spark SQL Data Types from Microsoft SQL Server and add tests

What changes were proposed in this pull request?

This PR added  Document Mapping Spark SQL Data Types from Microsoft SQL Server and added more tests
Why are the changes needed?

doc improvement
Does this PR introduce any user-facing change?

no
How was this patch tested?

new test and doc build

Was this patch authored or co-authored using generative AI tooling?

no

Thank you @dongjoon-hyun
Merged to master


Simple example notebook with 4 tiles.

Hi Marvin @m-albert ,
that's a fantastic tool you are developing here! May I ask for a little documentation improvement? I deal a lot with programming beginners, who seek to stitch images in Python. Do you think it would be possible to give them a very simple notebook as a starter? E.g. showing 2 or 4 images and stichting them? I have this notebook which obviously does not use your library, but I would be curious how this looks with the multiview-stitcher.
Let me know what you think!
Best,
Robert

Hi Robert @haesleinhuepf,
thanks for passing by :) I think you're completely right in that there's a notebook missing that shows the most basic stitching task.
Inspired by your notebook that you linked, here's an example of this: gist.
In fact it's so basic that it doesn't even contain any real alignment / registration. I should probably add that and then add the notebook to the example notebooks in this repo. I'm posting this gist here already as you were curious about this case 😃
Best,
Marvin

Or keep both examples separate and add both notebooks to your repository. That way you can make examples step by step more complicate and more versatile.

That's a good idea.

Maybe a naive question: the code in the notebook you linked is quite involved and likely deterrent for beginners. Do you think it would be possible to implement a convenience API e.g. like this:

Fair enough! I think you're right, having a simplified convenience API that works like the grid/collection stitching in Fiji will be useful. I also like the stitching.stitch entry point :)
Will implement this and leave this issue open for tracking. Thanks for the suggestion!


Problem installing on node-red v0.18.5\

The installation works fine but when I search the node I found this error. what can I do to solve? thanks

I think the problem is related to prerequisites of the "noble" package.
Check at the page https://github.com/noble/noble#prerequisites and install packages for your OS.

thank you so much for the help!


Unified docstring get behavior

I recommend unifying the docstring via inspect.getdoc.
I noticed that in addition to the built-in modules and dynamic link libraries,
the docstrings of other modules come directly from the __doc__ attribute.
Uniform behavior helps future expansion.

The language server avoids executing any Python code. Calling out to inspect to get docstrings would mean executing any code that needs to have its documentation examined, which is slow (and dangerous). We do it already, but only for compiled libraries where there is no other option.
Taking __doc__ if it can be statically determined may be possible, but it's unlikely that we're going to try and use the inspect library more than is strictly required.


Add telescope.show_previewer option

Implements #42 with true as default. I think that this option should be false by default.

Now to think about it, the previewer could be opened by default, but it would need to look like in nvim-possession plugin (plus cwd and maybe other things). This would make it more appealing to normal users and if adding your suggestion would make it the best solution for power users that would want more detailed/custom preview of their sessions.

@piotr-marendowski I created a draft of something more human-friendly as you suggested (PR: https://github.com/jedrzejboczar/possession.nvim/pull/44). Would this be ok, or do you have any suggestions?

@piotr-marendowski If you could apply the suggestion to change the name to previewer and use this vim.F.if_nil logic, than this could be merged before https://github.com/jedrzejboczar/possession.nvim/pull/44.

I updated this option, but don't know how to pass custom previewer, so I didn't write about it in docs.


Handler map factory should handle optimistic action type

According to #2, createHandlerMap should be capable to handle optimistic action creators.
Detailed Description
Yet, optimistic actions should be handled separately with multiple createHandlerMap call. So there would be a good point to reduce boilerplates.
Although, cancel is not part of observer protocol but this type practically is important and should be handled in many cases (isFetching entity, etc.) in the reducers.
Possible Implementation
createHandlerMap(addFoo, {
  next(state, action) {},
  error(state, action) {},
  complete(state, action) {},
  cancel(state, action) {}
})

hey @thebrodmann again!
How do you create action creator in this case?

Just by using createActionCreator in an object.
const addFoo = {
  next: createActionCreator('NEXT'),
  error: createActionCreator('ERROR'),
  cancel: createActionCreator('CANCEL'),
  complete: createActionCreator('COMPLETE'),
}


A Send-to Sub-menu for the Files App Context Menu

What's the Problem?
There's a Send-to SubMenu under the traditional Context Menu. I love the fact that Files seems to gradually have some of the traditional sub-menus linked already, but the send-to sub-menu is also just as vital, as it enables a user to be able to send files using Bluetooth and other archaic methods that are slowly fading away.

Solution/Idea
It'd be great if the Send to Sub-menu could be linked to the Files' App Context Menu.
Alternatives
Another idea I considered is that the send to menu could be Fused into a new share sub-menu that combines both the send to and Share features in one "exclusive-to-Files" pop-up window... I thought that'd be cool to see...

Priorities

Capability
Priority

This proposal will allow developers to accomplish W
Must

This proposal will allow end users to accomplish X
Should

This proposal will allow developers to accomplish Y
Could

This proposal will allow end users to accomplish Z
Won't

Files Version
<IP_ADDRESS>
Windows Version
Edition	Windows 11 Pro Version	21H2 Installed on	30/06/2022 OS build	22000.795 Experience	Windows Feature Experience Pack 1000.22000.795.0
Comments
I've linked screenshots in the pages above, hope they'll be sufficient to get the idea of my suggestion across... :)

Microsoft is working on improving the share menu so we don't plan to add the send to option.


Unable to download videos from kika.de

Checklist

[x] I'm reporting a broken site support
[x] I've verified that I'm running youtube-dl version 2021.12.17
[x] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bugtracker for similar issues including closed ones

Verbose log

C:\Users\xyz\Desktop\YOUTUBE DL>youtube-dl.exe -v "https://www.kika.de/echt/videos/echt-staffel-drei-darum-geht-s-100"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://www.kika.de/echt/videos/echt-staffel-drei-darum-geht-s-100']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.19041
[debug] exe versions: ffmpeg 4.2.1
[debug] Proxy map: {}
[generic] echt-staffel-drei-darum-geht-s-100: Requesting header
WARNING: Falling back on generic information extractor.
[generic] echt-staffel-drei-darum-geht-s-100: Downloading webpage
[generic] echt-staffel-drei-darum-geht-s-100: Extracting information
WARNING: [generic] echt-staffel-drei-darum-geht-s-100: Failed to parse JSON Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
WARNING: [generic] echt-staffel-drei-darum-geht-s-100: Failed to parse JSON Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
ERROR: Unsupported URL: https://www.kika.de/echt/videos/echt-staffel-drei-darum-geht-s-100
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\YoutubeDL.py", line 815, in wrapper
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\YoutubeDL.py", line 836, in __extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\common.py", line 534, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\generic.py", line 3489, in _real_extract
youtube_dl.utils.UnsupportedError: Unsupported URL: https://www.kika.de/echt/videos/echt-staffel-drei-darum-geht-s-100

Description

Since kika.de changed their website layout a few weeks ago it's no longer possible to download any video from it. It worked fine in the years before, and it's also on the list of supported sites.

Yes, the problem page is quite different from what the extractor is expecting. MDR, which is handled by the same extractor, is still serving the expected structure.
A new extractor is needed for Kika. It appears to be redirecting the media request to a ZDF video platform:
        'docType': 'externalVideo',
        'id': 'echt-staffel-drei-darum-geht-s-100',
        'uuid': '2f6a339f-cd3a-45e4-b05f-c4a1797fe565',
        'externalId': 'zdf-SCMS_1ad8eaaa-cacf-45ef-a469-33910b288bbf-default',

The previous episode of the series is available at https://www.zdf.de/kinder/echt/folge-10-s03-together-forever-100.html. Perhaps the problem episode will appear soon (currently just a trailer).

See https://github.com/yt-dlp/yt-dlp/pull/5788.

... but not very much:
$ python -m youtube_dl -F -v 'https://www.kika.de/echt/videos/echt-staffel-drei-darum-geht-s-100'
[debug] System config: [u'--prefer-ffmpeg']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-F', u'-v', u'https://www.kika.de/echt/videos/echt-staffel-drei-darum-geht-s-100']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: 774b346f9
[debug] Python version 2.7.18 (CPython) - Linux-4.4.0-210-generic-i686-with-Ubuntu-16.04-xenial
[debug] exe versions: avconv 4.3, avprobe 4.3, ffmpeg 4.3, ffprobe 4.3
[debug] Proxy map: {}
[debug] Using fake IP <IP_ADDRESS> (DE) as X-Forwarded-For.
[Kika] echt-staffel-drei-darum-geht-s-100: Downloading JSON metadata
[Kika] echt-staffel-drei-darum-geht-s-100: Downloading JSON metadata
[Kika] echt-staffel-drei-darum-geht-s-100: Downloading m3u8 information
[Kika] echt-staffel-drei-darum-geht-s-100: Checking mp4 video format URL
[Kika] echt-staffel-drei-darum-geht-s-100: Checking mp4 video format URL
[Kika] echt-staffel-drei-darum-geht-s-100: Checking mp4 video format URL
[Kika] echt-staffel-drei-darum-geht-s-100: Checking mp4 video format URL
[Kika] echt-staffel-drei-darum-geht-s-100: Checking mp4 video format URL
[Kika] echt-staffel-drei-darum-geht-s-100: Checking hls-414 video format URL
[Kika] echt-staffel-drei-darum-geht-s-100: Checking hls-636 video format URL
[Kika] echt-staffel-drei-darum-geht-s-100: Checking hls-1269 video format URL
[Kika] echt-staffel-drei-darum-geht-s-100: Checking hls-2564 video format URL
[info] Available formats for echt-staffel-drei-darum-geht-s-100:
format code  extension  resolution note
mp4-0        mp4        480x270      -2k video@  -1k, audio@ -1k, 0.00B
mp4-1        mp4        640x360      -2k video@  -1k, audio@ -1k, 0.00B
mp4-2        mp4        960x540      -2k video@  -1k, audio@ -1k, 0.00B
mp4-3        mp4        1024x576     -2k video@  -1k, audio@ -1k, 0.00B
mp4-4        mp4        1280x720     -2k video@  -1k, audio@ -1k, 0.00B
hls-414      mp4        480x270     414k , avc1.4d401f, 25.0fps, mp4a.40.2
hls-636      mp4        640x360     636k , avc1.4d401f, 25.0fps, mp4a.40.2
hls-1269     mp4        960x540    1269k , avc1.4d401f, 25.0fps, mp4a.40.2
hls-2564     mp4        1280x720   2564k , avc1.640028, 50.0fps, mp4a.40.2 (best)
$


Need Value List for band_name

From R. Seaman:
Is there a list of acceptable values for <band_name>?

Resolved with #16


Support for ABNF (Augmented Backus-Naur form)

It's just an idea to somehow support or parse things in accordance with the ABNF notation (https://en.wikipedia.org/wiki/Augmented_Backus–Naur_form). At present, a lot of RFCs contain definitions that can be implemented in accordance with ABNF. One example:
Macro definition (from SPF RFC7208):
The ABNF description for a macro is as follows:

   macro-string     = *( macro-expand / macro-literal )
   macro-expand     = ( "%{" macro-letter transformers *delimiter "}" )
                      / "%%" / "%_" / "%-"
   macro-literal    = %x21-24 / %x26-7E
                      ; visible characters except "%"
   macro-letter     = "s" / "l" / "o" / "d" / "i" / "p" / "h" /
                      "c" / "r" / "t" / "v"
   transformers     = *DIGIT [ "r" ]
   delimiter        = "." / "-" / "+" / "," / "/" / "_" / "="

At the moment, it would look more like this:
$delimiters = optional(atLeastOne(oneOfS('-\.+,/_=')));
$transformers = optional(atLeastOne(digitChar()))->append((optional(string('r'))));
$macroLetter = oneOfS('slodiphcrtv');
$macroLiteral = atLeastOne(anySingleBut('%'));
$macroExpand = string('%{')->append($macroLetter)->append($transformers)->append($delimiters)->append(string('}'))
    ->or(string('%%'))->or(string('%_'))->or(string('%-'));
$macroString = many($macroExpand->or($macroLiteral));

// test with data
$macroString->run('%{lr-}.lp.%{ir}.%{v}._spf.%{d2}');

Of course, you can work it out manually now, but I leave this idea here. Thanks for this excellent library :clap:

In fact, one could write a project that parses this notation with the help of this project :thinking:

That would be pretty sweet indeed :-)
One way to do ABNF->parsica  (or at least make it easier) would be to have a ABNF namespace in Parsica with its own set of parsers and combinators that strictly follow ABNF terminology and meaning.
Not a priority for me right now, too much other things to get to 1.0. But if you want to build this, I'd be happy to help or pair with you to get you started.


keras.layers.LSTM does not work with model.evaluate after training

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Debian 9.11 (GCE DeepLearning image)
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): v2.0.0-rc2-26-g64c3d38 2.0.0
Python version: 3.7.3 (anaconda3)
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version: V10.0.130 + <IP_ADDRESS>
GPU model and memory: V100 16GB

Describe the current behavior
model.evaluate raises this error after training
  File "lstm.py", line 76, in <module>
    model.evaluate([left, right], labels)
  File "/home/zhaohaozeng/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py", line 833, in evaluate
    use_multiprocessing=use_multiprocessing)
  File "/home/zhaohaozeng/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 456, in evaluate
import typing
    sample_weight=sample_weight, steps=steps, callbacks=callbacks, **kwargs)
  File "/home/zhaohaozeng/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 444, in _model_iteration
    total_epochs=1)
  File "/home/zhaohaozeng/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 123, in run_one_epoch
    batch_outs = execution_function(iterator)
  File "/home/zhaohaozeng/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 86, in execution_function
    distributed_function(input_fn))
  File "/home/zhaohaozeng/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 457, in __call__
    result = self._call(*args, **kwds)
  File "/home/zhaohaozeng/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 526, in _call
    return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds)  # pylint: disable=protected-access
  File "/home/zhaohaozeng/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 1141, in _filtered_call
    self.captured_inputs)
  File "/home/zhaohaozeng/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 1224, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager)
  File "/home/zhaohaozeng/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 511, in call
    ctx=ctx)
  File "/home/zhaohaozeng/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/execute.py", line 67, in quick_execute
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.NotFoundError: 2 root error(s) found.
  (0) Not found:  Resource AnonymousIterator/AnonymousIterator1/N10tensorflow4data16IteratorResourceE does not exist.
	 [[node IteratorGetNext (defined at /home/zhaohaozeng/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1751) ]]
	 [[IteratorGetNext/_45]]
  (1) Not found:  Resource AnonymousIterator/AnonymousIterator1/N10tensorflow4data16IteratorResourceE does not exist.
	 [[node IteratorGetNext (defined at /home/zhaohaozeng/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1751) ]]
0 successful operations.
0 derived errors ignored. [Op:__inference_distributed_function_11390]

Function call stack:
distributed_function -> distributed_function

Describe the expected behavior
model.evaluate should not raise this error after training.
Code to reproduce the issue
Code and log:  https://gist.github.com/matthew-z/e5848d545b60792dd84bfb9470ea541f
I tested with this machine on GCE:

n1-standard-4
1 x NVIDIA Tesla V100
image: c1-deeplearning-common-cu100-20191003 (Google Deep Learning VM, common-dl-gpu)
installed anaconda3, CUDNN <IP_ADDRESS> (got some errors with the original 7.4 CUDNN in the image), tensorflow-gpu 2.0

Other info / logs
The problem may not happen with CPU.

I have tried on colab with TF version 2.0  and was able to reproduce the issue.Please, find the gist here.Thanks!

I can reproduce this issue.
Somehow I don't think its a LSTM issue since the error log didn't indicate anything about LSTM. Also when I change the LSTM layer to SimpleRNN, the issue still persists.
The other experiment I did was change "experimental_run_tf_function=False" in model.compile(), which will let model run in func graph mode, and the issue goes away. So I think there is some issue for the training function in v2.
For now, you can set the "experimental_run_tf_function=False" to walk around the issue. We will dig deep to find the root cause.
Thanks.

Just tested tested same code again with the latest tf-nigthly and it runs fine now.


customLoadMoreItemView not set UltimateViewAdapter

customLoadMoreItemView not set in UltimateViewAdapter  alwasy null.
protected void revealDispatchLoadMoreView() {
        if (customLoadMoreItemView != null) {
            if (customLoadMoreItemView.getVisibility() != View.VISIBLE) {
                customLoadMoreItemView.setVisibility(View.VISIBLE);
            }
        }
    }

My code
mRecyclerView.hideEmptyView();
mRecyclerView.setLoadMoreView(R.layout._view_list_view_bottom_progressbar);
mRecyclerView.setOnLoadMoreListener(new UltimateRecyclerView.OnLoadMoreListener() {
	@Override
	public void loadMore(int itemsCount, final int maxLastVisiblePosition) {
		new Handler().postDelayed(new Runnable() {
			@Override
			public void run() {
				getData();
			}
		}, 500);
	}
});

getData() {

		if (mPageIndex == 0) {
			mRecyclerView.disableLoadmore();
		}
		else {
			mRecyclerView.reenableLoadmore();
		}

		mMainAllAdapter.insertInternal(data, mData);

		if (mData.size() >= mTotalRecordCount){
			mRecyclerView.disableLoadmore();
		}
		else {
			mRecyclerView.reenableLoadmore();
		}

		if (mMainAllAdapter.getItemCount() == 0) {
			mRecyclerView.showEmptyView();
		}

		mLoadData = true;

		mPageIndex++;
}

You need to explain more of ur issue. How does it produce this issue?

Exactly what do you want. I wrote the above code sample

I Think problem is here
  VH viewHolder = newFooterHolder(customLoadMoreView);
   customLoadMoreView is not null but
   viewHolder is null always

There should be a good mechanism to check for the null on customLoadMoreView. Yes, you need manually initialize the footer view holder in your implementation. With this architecture design to give developer alot more extendibility for their own and all other heavy lifting are done in this library.

Can you give example for Loadmore
easyRegularAdapter,Ultimate View Adapter,UltimateDifferentViewType Adapter
Thank you.

The same issue happens me


 Fixes #235
I still need to test if everything is working fine, so I'll leave this as a draft for now.
Another thing that I was thinking about was to automatically call this function when the program ends. I believe this could be achieved through atexit (source).

Coverage increased (+0.2%) to 43.222% when pulling 8bf425becbc6a69e1fbe4cb3898e046c74f5729b on igormp:kill-server into 646a490270c3f8e87cde20801a870d373203a9b9 on chrismattmann:master.

looking good as a start. Probably good to add a unit test to it thanks @igormp

Sure thing. I'm kinda busy this week, but will try to work on it as soon as I can.
If someone wants to get some work done on it before I do, feel free to do so.

hey @igormp let me know if you have time to make more progress on this thanks!

Hi, sorry for the long delay :sweat_smile:
Since we have shell=True, the actual Tika process is spawned as a child of the shell, which seems to die soon after the actual tika process starts, leaving the tika process without a parent and no information on its actual PID so we can kill it later on.
I'll try to find a way around that (maybe by looking for the shell's children before leaving the startServer function?) and report back as soon as I can.

Ok, finally got it working, now it's time to add some tests.

Dealt with another weird thing.
For context, I'm adding the following test in the test_tika.py file:
def test_server_killed(self):
    """parse some file then kills server"""
    file = os.path.join(os.path.dirname(__file__), 'files', 'rwservlet.pdf')
    with open(file, 'rb') as file_obj:
        tika.parser.from_file(file_obj)
    self.assertIsNone(tika.tika.killServer())

Simple enough, right? If I leave it be as it is, it works perfectly fine and kills the server without any problems.
However, if I let it run as the first test by naming it test_kill_server, then for some reason the remaining tests start to fail, no matter which test is the next one.
I guess that's because the server doesn't end immediately, so the next test still thinks the server is up, and then it exits during its execution. This is probably the reason, due to the fact that adding a 1 second sleep after killing the server solves our problems.
For now, I'll leave the 1 second sleep inside the function since it works, we can discuss a better solution if needed.

Everything seems fine. A couple points that we need to keep in mind for future discussion:

If the tika server was already running before our code execution, we won't be able to grab it's PID (since we do it when we start it ourselves), thus won't be able to kill it.
 On one hand, we could kill it after every parse, which would be simple enough, but would add tons of overhead in case someone was doing multiples parses one after another.

I would say @igormp that we don't kill after every parse. If it was running before our execution, I think we simply log that message and do a no op. Fair?

Sure thing. That's how I have it set up at the moment.

LGTM and time to merge thanks @igormp !

hey @igormp I'm getting periodic failures with this PR merged in 1.23.1, see:
test_ascii (tika.tests.tests_unpack.CreateTest) ... ok
test_utf8 (tika.tests.tests_unpack.CreateTest) ... ok

======================================================================
ERROR: test_kill_server (tika.tests.test_tika.CreateTest)
parse some file then kills server
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/mattmann/git/tika-python/tika/tests/test_tika.py", line 63, in test_kill_server
    self.assertIsNone(tika.tika.killServer())
  File "/Users/mattmann/git/tika-python/tika/tika.py", line 694, in killServer
    os.killpg(os.getpgid(TikaServerProcess.pid), signal.SIGTERM)
OSError: [Errno 3] No such process

----------------------------------------------------------------------
Ran 17 tests in 123.960s

FAILED (errors=1)
Test failed: <unittest.runner.TextTestResult run=17 errors=1 failures=0>
error: Test failed: <unittest.runner.TextTestResult run=17 errors=1 failures=0>
pomodoro:tika-python mattmann$

Any ideas for a fix?

looks like maybe adding your wait idea would help and we also have to catch that occasional OSError and print something sensible about it. @igormp

Hey, Chris! Sorry for the long delay once again :sweat_smile:
I've made another PR that should solve this problem.

thanks @igormp


Great Work !

Thank you for providing such innovative work, it's truly admirable! May I ask if there are any plans for open-sourcing the code?
:-)

Hi,
The code and data is going through internal review procedures, so I can't give you an exact timeline, sorry! But I will keep you updated.
Best,
Linyi


[Networking} Smart Out-of-Band Management and End-to-end network resilience for a Resilient SD-WAN

Proposed title of article
Smart Out-of-Band Management and End-to-end network resilience for a Resilient SD-WAN
Proposed article introduction
Now more than ever, the efficiency gained by SD-WAN will prove pivotal, but the ability to minimize downtime is equally important. Networks are becoming more geographically dispersed to support data-rich applications and Internet-connected devices. Additionally, as we are seeing with the current pandemic, unexpected issues can lead to new working conditions as well as network traffic surges of unprecedented duration and intensity.
This means networks must handle a lot of traffic economically and be resilient enough to withstand disruptions, security breaches, and behavioral changes in users. To handle this, organizations will need to implement a plan for end-to-end resilience into their overall enterprise networking ecosystem that will work in conjunction with SD-WAN to ensure efficiency and security.
Out-of-band management minimizes the need for truck rolls and mean time to recover (MTTR) from a disruption. And this dependability is available for a cost that is a small fraction of the savings when enterprises move from traditional to SD-WAN.
Key takeaways

The importance of SD-WAN.
Why network resilience matters to businesses.
How end-to-end network resilience ensures running of SD-WAN.
Out-of-Band management and SD-WAN.

Article quality
This article covers additional topics beyond what is available on other blogs.
References
N/A

Hello @jesmweni, we thank you for submitting your topic.
However, after some careful consideration, it struck us that this topic may be a bit over-saturated throughout other blog sites and official documentations.
We typically refrain from publishing content that is covered widely on the net or other blogs. As we're more interested in original, practitioner-focused content that takes a deeper dive into programming-centric concepts.
Please feel free to explore another more in-depth topic.


Setup repository for logos & design tokens

We should establish a "single source of truth" for core brand assets like logo image, colour palette, typography styles, etc.
Aside from aiding the website work, this will be a useful resource for Buildit in general.
Possible options are:

A git repo containing the image files and other data encoded as JSON. Using tools like Theo, we can automatically export things like SASS vars, Adobe colour swatches, etc. from those source files.
Purchase a subscription to InVision Design System Manager or similar which provides a more user-friendly front-end to manage those assets but can still export them in those kinds of formats

Checklist:

Choose an approach (DIY or commercial product)
Set it up
Populate it with current brand assets
Remove all other places where those brand assets currently live (gravity-ui-sass library, Dropbox folders, Paper docs, etc.)
Inform rest of Buildit to always fetch the logo from its new home!

I think this is "done", so I'm going to close it.
gravity-particles is now up-and-running and published to NPM. Once the color system stuff (see #147) gets merged, it'll become a dependency of gravity-ui-web.


How to get images with interesting patterns?

Thank you for supplying FISH-quant v2, which was a fantastic tool to analyze smFISH data.
However, after checking the document of sim-FISH, I just found the function to generate coordinations for simulating the patterns, but not end-to-end to images.
Can you teach me how to get images after applying “simulate_localization_pattern”?

Hi @TerminatorJ,
Unfortunately, Sim-FISH only allows to simulates 1) spots images and 2) point cloud of localization patterns. Simulating a cell image with localized spots would require to simulate the fluorescent background of the cell (in a realistic way), or reuse an existing cell image without spots and populate it with simulated spots.
The first option was not so successful so far, but the second option could be implemented in a future version. We have a library of real cells we use as template. We only exploit the extracted point cloud from these templates, but we could reuse the original cell images as well.
For now, you could simulate the patterns, initialize an empty image and use simfish.add_spots to generate the spots in your empty image. But you won't have the cell morphology.
Best,
Arthur

Hi Arthur, thank you for the quick response!
If you can solve the morphology question, that would be perfect!
To my understanding, if we need to generate spots with patterns by simfish.add_spots, we should use the "ground_truth" coordination. However, I tried to replace the "Coordinate along the y axis" and "Coordinate along the x axis" by using the results from "simulate_localization_pattern" function. But there is an error occur:"ValueError: operands could not be broadcast together with shapes (0,15) (15,15) (0,15) "
So, how can we generate spots with patterns using simfish.add_spots? If I want 7 patterns, what should I do? Should I generate a new "precomputed_gaussian"?

I have coded this Gist. Try to run it in a notebook!

Thank you! This coded document is very helpful.


feat: evalperf

run following command to generate stats and save into results/evalperf/COMBINED-RESULTS.json
python results/evalperf/stats.py

[x] winrate stats
[ ] heatmap of winrate
[x] docs @ganler

Let's leave the heatmap as another PR -- cc: @soryxie


Not keeping downloaded files

The function to keep downloaded files doesn't seem to work correctly.

Feature will be removed. Please use auto-h-encore for Windows systems.


Rewrite.

Rewrite
Refactor

[x] CI & Metrics
[x] Test Coverage
[x] Update deps
[x] MySQL engine support
[x] Better docs.

Rewrite
Refactor
Better docs
Now includes
CI & Metrics
Test Coverage
Update deps
MySQL engine support