LUCENE-9662: CheckIndex should be concurrent  - parallelizing index parts check within each segment

Description
This PR is currently WIP
Parallelize CheckIndex code
commit 1: parallelize checking each index part within each segment
Solution
Please provide a short description of the approach taken to implement your solution.
Tests
Passed  ./gradlew clean; ./gradlew check -Ptests.nightly=true -Pvalidation.git.failOnModified=false without nocommit comment

I am excited to see what happens to CheckIndex time in Lucene's nightly benchmarks after we push this!  But I agree we must also not crush the more common case of machines that don't have tons of cores ...

I am excited to see what happens to CheckIndex time in Lucene's nightly benchmarks after we push this! But I agree we must also not crush the more common case of machines that don't have tons of cores ...

Thanks Michael for the comment! Just curious, do you have any suggestion to my nocommit questions in this PR with regard to how to parallelize some of the section of code that currently depends on ordering (message printing inside each segment / index part, error handling etc)? I could also give it a try first and see how folks feel about the solution.

For now I've gone ahead and implemented the idea to add segment / part ids to logging messages so that they are still usable (with some log parsing) in concurrent setting. I also removed failFast support in each individual index test methods, as they seems to be primarily used for skipping (using exception) during sequential execution, and removing it can simplify exception handling in concurrent code a bit.
What do you think about these changes? @rmuir @dweiss @mikemccand

I love this change -- CheckIndex is often run in the "heat of the moment" so time is of the essence to recovering your Lucene index and getting things back online. Yet it is very slow today, even on highly concurrent boxes.
I left some small comments that I think are fine to do in a followon PR. This change is already massive enough (GitHub was at first not willing to even render the CheckIndex diffs!) and impactful enough that we can do the improvements after.

Thanks Michael for the review and feedback. I think as far as the original scope of the jira ticket goes, there's also the parallelization across segments that has not been implemented yet. But agree that this PR is already big and should already provide a good speed up on powerful concurrent boxes (up to 11 concurrent checks for each segment), so we can probably let it run for a while and see if parallelization across segments is still needed, which from my quick in-mind coding will definitely require much more changes for concurrency control to get it right.
One thing I'm still researching is that, it seems there's limited direct test coverage for this CheckIndex  class? I see there's TestCheckIndex, but it only has 4 tests, and the majority of its functionalities seems to be put under tests by other index testing utilities and test cases. Shall I still add a few more tests for these changes (and should I put them in TestCheckIndex)? On the other hand, I've been running ./gradlew check -Ptests.nightly=true -Pvalidation.git.failOnModified=false the whole time and all tests have been passing.

./gradlew check -Ptests.nightly=true -Pvalidation.git.failOnModified=false

Hi Zach. You can also "git add -A ." (stage your changes for commit); or just commit them in. Then there's  no need for the fail-on-modified flag to be turned off. :)

./gradlew check -Ptests.nightly=true -Pvalidation.git.failOnModified=false

Hi Zach. You can also "git add -A ." (stage your changes for commit); or just commit them in. Then there's  no need for the fail-on-modified flag to be turned off. :)

Ha yes I came to realize that also some time ago, but kinda formed the habit of passing it in by default now (mostly from past command search) so that I don't need to worry about where my changes are. But yeah that's good tip. Thanks Dawid!

One thing I'm still researching is that, it seems there's limited direct test coverage for this CheckIndex class? I see there's TestCheckIndex, but it only has 4 tests, and the majority of its functionalities seems to be put under tests by other index testing utilities and test cases. Shall I still add a few more tests for these changes (and should I put them in TestCheckIndex)? On the other hand, I've been running ./gradlew check -Ptests.nightly=true -Pvalidation.git.failOnModified=false the whole time and all tests have been passing.

  But I don't think that should block this change -- we can open a follow-on issue to get better direct unit tests for CheckIndex?  Those would be fun tests to write: make a healthy index, then make a random single bit change to one of its files, and then see if CheckIndex catches it.  Hmm I think we have such a test somewhere :)  But not apparently in BaseTestCheckIndex...

Our nightly benchmarks also exercise CheckIndex each night on a "real" index, and sometimes that finds interesting surprises ;)

OK I ran CheckIndex, with this PR, on last night's English Wikipedia benchmark index:
beast3:core[main]$ java -cp ../core/build/libs/lucene-core-9.0.0-SNAPSHOT.jar org.apache.lucene.index.CheckIndex /l/indices/trunk.nightly.index.prev/index

NOTE: testing will be more thorough if you run java with '-ea:org.apache.lucene...', so assertions are enabled

Opening index @ /l/indices/trunk.nightly.index.prev/index

Exception in thread "main" java.lang.IllegalArgumentException
        at java.base/java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1295)
        at java.base/java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1217)
        at java.base/java.util.concurrent.Executors.newFixedThreadPool(Executors.java:155)
        at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:527)
        at org.apache.lucene.index.CheckIndex.doCheck(CheckIndex.java:4036)
        at org.apache.lucene.index.CheckIndex.doMain(CheckIndex.java:3920)
        at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:3852)

I think the issue is that opts.threadCount is 0 if you don't explicitly set the thread count.  Can we fix it to default to number of cores on the box, maybe capped at a maximum (4?
8?), when CheckIndex is invoked interactively from the command-line?

Next I ran CheckIndex with -threadCount 1:
beast3:core[main]$ java -cp ../core/build/libs/lucene-core-9.0.0-SNAPSHOT.jar org.apache.lucene.index.CheckIndex /l/indices/trunk.nightly.index.prev/index -threadCount 1

NOTE: testing will be more thorough if you run java with '-ea:org.apache.lucene...', so assertions are enabled

Opening index @ /l/indices/trunk.nightly.index.prev/index

0.00% total deletions; 27625038 documents; 0 deleteions
Segments file=segments_2 numSegments=15 version=9.0.0 id=5kpcv7rf0icb64q1ern1xs4bp userData={userData=multi}
[Segment 110]   1 of 15: name=_32 maxDoc=4978031
[Segment 110]     version=9.0.0
[Segment 110]     id=5kpcv7rf0icb64q1ern1xs35l
[Segment 110]     codec=Lucene90
[Segment 110]     compound=false
[Segment 110]     numFiles=20
[Segment 110]     size (MB)=3,890.496
[Segment 110]     diagnostics = {java.version=15.0.1, java.vm.version=15.0.1+9-18, lucene.version=9.0.0, timestamp=1621919422678, os=Linux, java.runtime.version=15.0.1+9-18, mergeFactor=10\
, os.arch=amd64, source=merge, java.vendor=Oracle Corporation, os.version=5.11.2-arch1-1, mergeMaxNumSegments=-1}
[Segment 110]     no deletions
[Segment 110]    test: open reader.........[Segment 110] OK [took 0.166 sec]
[Segment 110]    test: check integrity.....[Segment 110] OK [took 0.580 sec]
[Segment 110][LiveDocs]    test: check live docs.....[Segment 110][LiveDocs] OK [took 0.001 sec]
[Segment 110][FieldInfos]    test: field infos.........[Segment 110][FieldInfos] OK [23 fields] [took 0.000 sec]
[Segment 110][FieldNorms]    test: field norms.........[Segment 110][FieldNorms] OK [2 fields] [took 0.031 sec]
[Segment 110][Postings]    test: terms, freq, prox...OK [16412141 terms; 525829996 terms/docs pairs; 765284459 tokens] [took 35.202 sec]
[Segment 110][StoredFields]    test: stored fields.......[Segment 110][StoredFields] OK [14934093 total field count; avg 3.0 fields per doc] [took 0.836 sec]
[Segment 110][TermVectors]    test: term vectors........[Segment 110][TermVectors] OK [0 total term vector count; avg 0.0 term/freq vector fields per doc] [took 0.000 sec]
[Segment 110][DocValues]    test: docvalues...........[Segment 110][DocValues] OK [15 docvalues fields; 4 BINARY; 2 NUMERIC; 7 SORTED; 0 SORTED_NUMERIC; 2 SORTED_SET] [took 3.205 sec]
[Segment 110][Points]    test: points..............[Segment 110][Points] OK [4 fields, 19912124 points] [took 0.636 sec]
[Segment 110][Vectors]    test: vectors..............[Segment 110][Vectors] OK [1 fields, 4978031 vectors] [took 0.169 sec]
[Segment 110]
[Segment 221]   2 of 15: name=_65 maxDoc=4978032
[Segment 221]     version=9.0.0
[Segment 221]     id=5kpcv7rf0icb64q1ern1xs3eu
[Segment 221]     codec=Lucene90
[Segment 221]     compound=false
[Segment 221]     numFiles=20
[Segment 221]     size (MB)=3,873.589
[Segment 221]     diagnostics = {java.version=15.0.1, java.vm.version=15.0.1+9-18, lucene.version=9.0.0, timestamp=1621920782361, os=Linux, java.runtime.version=15.0.1+9-18, mergeFactor=10\
, os.arch=amd64, source=merge, java.vendor=Oracle Corporation, os.version=5.11.2-arch1-1, mergeMaxNumSegments=-1}
[Segment 221]     no deletions
[Segment 221]    test: open reader.........[Segment 221] OK [took 0.172 sec]
[Segment 221]    test: check integrity.....[Segment 221] OK [took 0.562 sec]
[Segment 221][LiveDocs]    test: check live docs.....[Segment 221][LiveDocs] OK [took 0.000 sec]
[Segment 221][FieldInfos]    test: field infos.........[Segment 221][FieldInfos] OK [23 fields] [took 0.000 sec]
[Segment 221][FieldNorms]    test: field norms.........[Segment 221][FieldNorms] OK [2 fields] [took 0.009 sec]
[Segment 221][Postings]    test: terms, freq, prox...OK [15879702 terms; 497899770 terms/docs pairs; 726182953 tokens] [took 31.904 sec]
[Segment 221][StoredFields]    test: stored fields.......[Segment 221][StoredFields] OK [14934096 total field count; avg 3.0 fields per doc] [took 0.869 sec]
[Segment 221][TermVectors]    test: term vectors........[Segment 221][TermVectors] OK [0 total term vector count; avg 0.0 term/freq vector fields per doc] [took 0.000 sec]
[Segment 221][DocValues]    test: docvalues...........[Segment 221][DocValues] OK [15 docvalues fields; 4 BINARY; 2 NUMERIC; 7 SORTED; 0 SORTED_NUMERIC; 2 SORTED_SET] [took 3.234 sec]
[Segment 221][Points]    test: points..............[Segment 221][Points] OK [4 fields, 19912128 points] [took 0.618 sec]
[Segment 221][Vectors]    test: vectors..............[Segment 221][Vectors] OK [1 fields, 4978032 vectors] [took 0.153 sec]
[Segment 221]
[Segment 332]   3 of 15: name=_98 maxDoc=4978032
[Segment 332]     version=9.0.0
[Segment 332]     id=5kpcv7rf0icb64q1ern1xs3o3
[Segment 332]     codec=Lucene90
[Segment 332]     compound=false
[Segment 332]     numFiles=20
[Segment 332]     size (MB)=3,837.561
[Segment 332]     diagnostics = {java.version=15.0.1, java.vm.version=15.0.1+9-18, lucene.version=9.0.0, timestamp=1621922132326, os=Linux, java.runtime.version=15.0.1+9-18, mergeFactor=10\
, os.arch=amd64, source=merge, java.vendor=Oracle Corporation, os.version=5.11.2-arch1-1, mergeMaxNumSegments=-1}
[Segment 332]     no deletions
[Segment 332]    test: open reader.........[Segment 332] OK [took 0.137 sec]
[Segment 332]    test: check integrity.....[Segment 332] OK [took 0.516 sec]
[Segment 332][LiveDocs]    test: check live docs.....[Segment 332][LiveDocs] OK [took 0.000 sec]
[Segment 332][FieldInfos]    test: field infos.........[Segment 332][FieldInfos] OK [23 fields] [took 0.000 sec]
[Segment 332][FieldNorms]    test: field norms.........[Segment 332][FieldNorms] OK [2 fields] [took 0.012 sec]
[Segment 332][Postings]    test: terms, freq, prox...OK [16460318 terms; 475335907 terms/docs pairs; 694693997 tokens] [took 28.982 sec]
[Segment 332][StoredFields]    test: stored fields.......[Segment 332][StoredFields] OK [14934096 total field count; avg 3.0 fields per doc] [took 0.870 sec]
[Segment 332][TermVectors]    test: term vectors........[Segment 332][TermVectors] OK [0 total term vector count; avg 0.0 term/freq vector fields per doc] [took 0.000 sec]
[Segment 332][DocValues]    test: docvalues...........[Segment 332][DocValues] OK [15 docvalues fields; 4 BINARY; 2 NUMERIC; 7 SORTED; 0 SORTED_NUMERIC; 2 SORTED_SET] [took 3.323 sec]
[Segment 332][Points]    test: points..............[Segment 332][Points] OK [4 fields, 19912128 points] [took 0.623 sec]
[Segment 332][Vectors]    test: vectors..............[Segment 332][Vectors] OK [1 fields, 4978032 vectors] [took 0.151 sec]
[Segment 332]

...

[Segment 614]   15 of 15: name=_h2 maxDoc=46742
[Segment 614]     version=9.0.0
[Segment 614]     id=5kpcv7rf0icb64q1ern1xs4bl
[Segment 614]     codec=Lucene90
[Segment 614]     compound=false
[Segment 614]     numFiles=20
[Segment 614]     size (MB)=39.37
[Segment 614]     diagnostics = {java.vendor=Oracle Corporation, os=Linux, timestamp=1621925578914, java.version=15.0.1, java.vm.version=15.0.1+9-18, lucene.version=9.0.0, source=flush, os\
.arch=amd64, java.runtime.version=15.0.1+9-18, os.version=5.11.2-arch1-1}
[Segment 614]     no deletions
[Segment 614]    test: open reader.........[Segment 614] OK [took 0.004 sec]
[Segment 614]    test: check integrity.....[Segment 614] OK [took 0.009 sec]
[Segment 614][LiveDocs]    test: check live docs.....[Segment 614][LiveDocs] OK [took 0.000 sec]
[Segment 614][FieldInfos]    test: field infos.........[Segment 614][FieldInfos] OK [23 fields] [took 0.000 sec]
[Segment 614][FieldNorms]    test: field norms.........[Segment 614][FieldNorms] OK [2 fields] [took 0.001 sec]
[Segment 614][Postings]    test: terms, freq, prox...OK [344318 terms; 3965374 terms/docs pairs; 6341319 tokens] [took 0.342 sec]
[Segment 614][StoredFields]    test: stored fields.......[Segment 614][StoredFields] OK [140226 total field count; avg 3.0 fields per doc] [took 0.009 sec]
[Segment 614][TermVectors]    test: term vectors........[Segment 614][TermVectors] OK [0 total term vector count; avg 0.0 term/freq vector fields per doc] [took 0.000 sec]
[Segment 614][DocValues]    test: docvalues...........[Segment 614][DocValues] OK [15 docvalues fields; 4 BINARY; 2 NUMERIC; 7 SORTED; 0 SORTED_NUMERIC; 2 SORTED_SET] [took 0.075 sec]
[Segment 614][Points]    test: points..............[Segment 614][Points] OK [4 fields, 186968 points] [took 0.006 sec]
[Segment 614][Vectors]    test: vectors..............[Segment 614][Vectors] OK [1 fields, 46742 vectors] [took 0.002 sec]
[Segment 614]
No problems were detected with this index.

Took 202.570 sec total.

I think we should try to remove the duplicate segment/partId (e.g.[Segment 614][StoredFields]) in some lines?  And in fact since each segment is still done "at once" (we don't do concurrency across segments, yet), we shouldn't need the e.g. [Segment 614] prefix on each line?
Hmm, also why are we calling it Segment 614 when its name is _h2?  Hmm, is that the decimal translation of the base36 value?

Finally I ran CheckIndex -threadCount 128:
beast3:core[main]$ java -cp ../core/build/libs/lucene-core-9.0.0-SNAPSHOT.jar org.apache.lucene.index.CheckIndex /l/indices/trunk.nightly.index.prev/index -threadCount 128

NOTE: testing will be more thorough if you run java with '-ea:org.apache.lucene...', so assertions are enabled

Opening index @ /l/indices/trunk.nightly.index.prev/index

0.00% total deletions; 27625038 documents; 0 deleteions
Segments file=segments_2 numSegments=15 version=9.0.0 id=5kpcv7rf0icb64q1ern1xs4bp userData={userData=multi}
[Segment 110]   1 of 15: name=_32 maxDoc=4978031
[Segment 110]     version=9.0.0
[Segment 110]     id=5kpcv7rf0icb64q1ern1xs35l
[Segment 110]     codec=Lucene90
[Segment 110]     compound=false
[Segment 110]     numFiles=20
[Segment 110]     size (MB)=3,890.496
[Segment 110]     diagnostics = {os.arch=amd64, mergeFactor=10, java.runtime.version=15.0.1+9-18, os=Linux, timestamp=1621919422678, lucene.version=9.0.0, java.vm.version=15.0.1+9-18, java.version=15.0.1, mergeMaxNumSegments=-1, os.version=5.11.2-arch1-1, java.vendor=Oracle Corporation, source=merge}
[Segment 110]     no deletions
[Segment 110]    test: open reader.........[Segment 110] OK [took 0.161 sec]
[Segment 110]    test: check integrity.....[Segment 110] OK [took 0.617 sec]
[Segment 110][LiveDocs]    test: check live docs.....[Segment 110][LiveDocs] OK [took 0.001 sec]
[Segment 110][FieldInfos]    test: field infos.........[Segment 110][FieldNorms]    test: field norms.........[Segment 110][FieldInfos] OK [23 fields] [took 0.001 sec]
[Segment 110][Postings]    test: terms, freq, prox...[Segment 110][StoredFields]    test: stored fields.......[Segment 110][TermVectors]    test: term vectors........[Segment 110][DocValues]    test: docvalues...........[Segment 110][TermVectors] OK [0 total term vector count; avg 0.0 term/freq vector fields per doc] [took 0.001 sec]
[Segment 110][Points]    test: points..............[Segment 110][Vectors]    test: vectors..............[Segment 110][FieldNorms] OK [2 fields] [took 0.032 sec]
[Segment 110][Vectors] OK [1 fields, 4978031 vectors] [took 0.190 sec]
[Segment 110][Points] OK [4 fields, 19912124 points] [took 0.675 sec]
[Segment 110][StoredFields] OK [14934093 total field count; avg 3.0 fields per doc] [took 1.229 sec]
[Segment 110][DocValues] OK [15 docvalues fields; 4 BINARY; 2 NUMERIC; 7 SORTED; 0 SORTED_NUMERIC; 2 SORTED_SET] [took 3.357 sec]
OK [16412141 terms; 525829996 terms/docs pairs; 765284459 tokens] [took 33.251 sec]
[Segment 110]
[Segment 221]   2 of 15: name=_65 maxDoc=4978032
[Segment 221]     version=9.0.0
[Segment 221]     id=5kpcv7rf0icb64q1ern1xs3eu
[Segment 221]     codec=Lucene90
[Segment 221]     compound=false
[Segment 221]     numFiles=20
[Segment 221]     size (MB)=3,873.589
[Segment 221]     diagnostics = {os.arch=amd64, mergeFactor=10, java.runtime.version=15.0.1+9-18, os=Linux, timestamp=1621920782361, lucene.version=9.0.0, java.vm.version=15.0.1+9-18, java.version=15.0.1, mergeMaxNumSegments=-1, os.version=5.11.2-arch1-1, java.vendor=Oracle Corporation, source=merge}
[Segment 221]     no deletions
[Segment 221]    test: open reader.........[Segment 221] OK [took 0.166 sec]
[Segment 221]    test: check integrity.....[Segment 221] OK [took 0.533 sec]
[Segment 221][LiveDocs]    test: check live docs.....[Segment 221][FieldInfos]    test: field infos.........[Segment 221][LiveDocs] OK [took 0.000 sec]
[Segment 221][FieldNorms]    test: field norms.........[Segment 221][Postings]    test: terms, freq, prox...[Segment 221][FieldInfos] OK [23 fields] [took 0.000 sec]
[Segment 221][StoredFields]    test: stored fields.......[Segment 221][TermVectors]    test: term vectors........[Segment 221][DocValues]    test: docvalues...........[Segment 221][Points]    test: points..............[Segment 221][Vectors]    test: vectors..............[Segment 221][TermVectors] OK [0 total term vector count; avg 0.0 term/freq vector fields per doc] [t\
ook 0.000 sec]
[Segment 221][FieldNorms] OK [2 fields] [took 0.012 sec]
[Segment 221][Vectors] OK [1 fields, 4978032 vectors] [took 0.163 sec]
[Segment 221][Points] OK [4 fields, 19912128 points] [took 0.646 sec]
[Segment 221][StoredFields] OK [14934096 total field count; avg 3.0 fields per doc] [took 1.029 sec]
[Segment 221][DocValues] OK [15 docvalues fields; 4 BINARY; 2 NUMERIC; 7 SORTED; 0 SORTED_NUMERIC; 2 SORTED_SET] [took 3.073 sec]
OK [15879702 terms; 497899770 terms/docs pairs; 726182953 tokens] [took 31.948 sec]
[Segment 221]
[Segment 332]   3 of 15: name=_98 maxDoc=4978032
[Segment 332]     version=9.0.0
[Segment 332]     id=5kpcv7rf0icb64q1ern1xs3o3
[Segment 332]     codec=Lucene90
[Segment 332]     compound=false
[Segment 332]     numFiles=20
[Segment 332]     size (MB)=3,837.561
[Segment 332]     diagnostics = {os.arch=amd64, mergeFactor=10, java.runtime.version=15.0.1+9-18, os=Linux, timestamp=1621922132326, lucene.version=9.0.0, java.vm.version=15.0.1+9-18, java.version=15.0.1, mergeMaxNumSegments=-1, os.version=5.11.2-arch1-1, java.vendor=Oracle Corporation, source=merge}
[Segment 332]     no deletions
[Segment 332]    test: open reader.........[Segment 332] OK [took 0.136 sec]
[Segment 332]    test: check integrity.....[Segment 332] OK [took 0.554 sec]
[Segment 332][LiveDocs]    test: check live docs.....[Segment 332][FieldInfos]    test: field infos.........[Segment 332][LiveDocs] OK [took 0.000 sec]
[Segment 332][FieldNorms]    test: field norms.........[Segment 332][Postings]    test: terms, freq, prox...[Segment 332][FieldInfos] OK [23 fields] [took 0.000 sec]
[Segment 332][StoredFields]    test: stored fields.......[Segment 332][TermVectors]    test: term vectors........[Segment 332][DocValues]    test: docvalues...........[Segment 332][Points]    test: points..............[Segment 332][TermVectors] OK [0 total term vector count; avg 0.0 term/freq vector fields per doc] [took 0.000 sec]
[Segment 332][Vectors]    test: vectors..............[Segment 332][FieldNorms] OK [2 fields] [took 0.014 sec]
[Segment 332][Vectors] OK [1 fields, 4978032 vectors] [took 0.157 sec]
[Segment 332][Points] OK [4 fields, 19912128 points] [took 0.639 sec]
[Segment 332][StoredFields] OK [14934096 total field count; avg 3.0 fields per doc] [took 1.032 sec]
[Segment 332][DocValues] OK [15 docvalues fields; 4 BINARY; 2 NUMERIC; 7 SORTED; 0 SORTED_NUMERIC; 2 SORTED_SET] [took 3.124 sec]
OK [16460318 terms; 475335907 terms/docs pairs; 694693997 tokens] [took 30.290 sec]
[Segment 332]

...

[Segment 614]   15 of 15: name=_h2 maxDoc=46742
[Segment 614]     version=9.0.0
[Segment 614]     id=5kpcv7rf0icb64q1ern1xs4bl
[Segment 614]     codec=Lucene90
[Segment 614]     compound=false
[Segment 614]     numFiles=20
[Segment 614]     size (MB)=39.37
[Segment 614]     diagnostics = {java.vm.version=15.0.1+9-18, java.version=15.0.1, timestamp=1621925578914, os=Linux, java.vendor=Oracle Corporation, os.version=5.11.2-arch1-1, java.runtime.version=15.0.1+9-18, os.arch=amd64, source=flush, lucene.version=9.0.0}
[Segment 614]     no deletions
[Segment 614]    test: open reader.........[Segment 614] OK [took 0.004 sec]
[Segment 614]    test: check integrity.....[Segment 614] OK [took 0.011 sec]
[Segment 614][LiveDocs]    test: check live docs.....[Segment 614][FieldInfos]    test: field infos.........[Segment 614][FieldNorms]    test: field norms.........[Segment 614][Postings]    test: terms, freq, prox...[Segment 614][LiveDocs] OK [took 0.000 sec]
[Segment 614][StoredFields]    test: stored fields.......[Segment 614][TermVectors]    test: term vectors........[Segment 614][DocValues]    test: docvalues...........[Segment 614][Points]    test: points..............[Segment 614][Vectors]    test: vectors..............[Segment 614][FieldInfos] OK [23 fields] [took 0.000 sec]
[Segment 614][TermVectors] OK [0 total term vector count; avg 0.0 term/freq vector fields per doc] [took 0.000 sec]
[Segment 614][FieldNorms] OK [2 fields] [took 0.002 sec]
[Segment 614][Vectors] OK [1 fields, 46742 vectors] [took 0.002 sec]
[Segment 614][Points] OK [4 fields, 186968 points] [took 0.006 sec]
[Segment 614][StoredFields] OK [140226 total field count; avg 3.0 fields per doc] [took 0.012 sec]
[Segment 614][DocValues] OK [15 docvalues fields; 4 BINARY; 2 NUMERIC; 7 SORTED; 0 SORTED_NUMERIC; 2 SORTED_SET] [took 0.079 sec]
OK [344318 terms; 3965374 terms/docs pairs; 6341319 tokens] [took 0.331 sec]
[Segment 614]
No problems were detected with this index.

Took 176.318 sec total.

It went a bit faster!  (203 down to 176 seconds).
But the output is jumbled, I think because we are missing newlines somewhere, or maybe necessary locking?

Thanks @mikemccand for the feedback comment, as well as running the index check and posting the results here! I was planning to do that next after adding some more tests, but you beat me to it.

 But I don't think that should block this change -- we can open a follow-on issue to get better direct unit tests for CheckIndex? Those would be fun tests to write: make a healthy index, then make a random single bit change to one of its files, and then see if CheckIndex catches it. Hmm I think we have such a test somewhere :) But not apparently in BaseTestCheckIndex...

Sounds good. Yes I think I came across them before, but didn't recall now exactly where they are now...but will work on them in a follow-up PR.

I think the issue is that opts.threadCount is 0 if you don't explicitly set the thread count. Can we fix it to default to number of cores on the box, maybe capped at a maximum (4?
8?), when CheckIndex is invoked interactively from the command-line?

Ah sorry about this (embarrassing) bu ! There was a default 1 for threadCount set in the code, but when it was not provided via the command -line, the default was overwritten by 0, causing this exception to be thrown. I'll fix it and cap at 4.

I think we should try to remove the duplicate segment/partId (e.g.[Segment 614][StoredFields]) in some lines?

But the output is jumbled, I think because we are missing newlines somewhere, or maybe necessary locking?

Yes these repeated segment / part ids are due to concurrent threads printing messages without newlines:
https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/index/CheckIndex.java#L934. I also took a look at the implementation of PrintStream#println and it is synchronized on this, so it shouldn't require additional locking (assuming its sub classes are also similar).
I think this is indeed a problem with this approach to use segment / part id to organize messages, as it still requires certain way of printing the messages. I'll switch over to the other approach then to use per part buffer.

Hmm, also why are we calling it Segment 614 when its name is _h2? Hmm, is that the decimal translation of the base36 value?

This was due to segmentName was used for id there, but it should actually use info.info.name . I'll fix that.

Finally I ran CheckIndex -threadCount 128

It went a bit faster! (203 down to 176 seconds).

Glad to see it actually improved the speed there :D ! I think 128 might be too big of a threadCount though for the current implementation, as it only parallelize (up to 11) part checking in each segment at a time. I'll cap the threadCount to 11 and print out a message to alert the user if big value was passed in as well

I just tried to run CheckIndex#checkIndex via command line with index built for other task, but it failed from Version check (even without my latest changes):
if (major > 255 || major < 0) {
    throw new IllegalArgumentException("Illegal major version: " + major);
}

I'll dig into that more tomorrow.
From the unit test TestCheckIndex#testCheckIndexAllValid, I'm able to get this log output:
0.00% total deletions; 5 documents; 0 deleteions
Segments file=segments_1 numSegments=1 version=9.0.0 id=bqdz36uoui8ymz6pd2tp9wmet
1 of 1: name=_0 maxDoc=5
    version=9.0.0
    id=bqdz36uoui8ymz6pd2tp9wmeq
    codec=Asserting(Lucene90)
    compound=false
    numFiles=26
    sort=<int: "sort_field">!
    size (MB)=0.004
    diagnostics = {source=flush, os.arch=x86_64, java.runtime.version=11.0.9+11, os.version=10.15.5, java.vendor=AdoptOpenJDK, os=Mac OS X, timestamp=1622013701421, java.version=11.0.9, java.vm.version=11.0.9+11, lucene.version=9.0.0}
    no deletions
    test: open reader.........OK [took 0.004 sec]
    test: check integrity.....OK [took 0.001 sec]
    test: check live docs.....OK [took 0.000 sec]

    test: field infos.........OK [8 fields] [took 0.000 sec]

    test: field norms.........OK [1 fields] [took 0.000 sec]

    test: stored fields.......OK [8 total field count; avg 1.6 fields per doc] [took 0.001 sec]

    test: terms, freq, prox...OK [5 terms; 8 terms/docs pairs; 12 tokens] [took 0.004 sec]

    test: docvalues...........OK [2 docvalues fields; 0 BINARY; 2 NUMERIC; 0 SORTED; 0 SORTED_NUMERIC; 0 SORTED_SET] [took 0.002 sec]

    test: term vectors........OK [4 total term vector count; avg 1.0 term/freq vector fields per doc] [took 0.006 sec]

    test: vectors..............OK [2 fields, 8 vectors] [took 0.001 sec]

    test: index sort..........OK [took 0.002 sec]

    test: check soft deletes.....
    test: points..............OK [1 fields, 4 points] [took 0.005 sec]

No problems were detected with this index.

Took 0.046 sec total.

Here's the latest console output I got from running against a local small index
> Task :lucene:core:CheckIndex.main()
-threadCount currently only supports up to 11 threads. Value higher than that will be capped.

NOTE: testing will be more thorough if you run java with '-ea:org.apache.lucene...', so assertions are enabled

Opening index @ /Users/xichen/IdeaProjects/benchmarks/data/tmp/index-msmacropassages/

Checking index with async threadCount: 11
0.00% total deletions; 8841823 documents; 0 deleteions
Segments file=segments_2 numSegments=1 version=9.0.0 id=arx9mawlpijzmvgvi7ehv0ret
1 of 1: name=_a maxDoc=8841823
    version=9.0.0
    id=arx9mawlpijzmvgvi7ehv0req
    codec=Lucene90
    compound=false
    numFiles=12
    size (MB)=2,658.426
    diagnostics = {source=merge, java.vendor=AdoptOpenJDK, os.version=10.15.5, mergeMaxNumSegments=1, java.version=11.0.9, java.vm.version=11.0.9+11, lucene.version=9.0.0, timestamp=1622049007198, os=Mac OS X, java.runtime.version=11.0.9+11, mergeFactor=10, os.arch=x86_64}
    no deletions
    test: open reader.........OK [took 0.051 sec]
    test: check integrity.....OK [took 12.599 sec]
    test: field infos.........OK [2 fields] [took 0.000 sec]
    test: check live docs.....OK [took 0.000 sec]
    test: term vectors........OK [0 total term vector count; avg 0.0 term/freq vector fields per doc] [took 0.000 sec]
    test: docvalues...........OK [0 docvalues fields; 0 BINARY; 0 NUMERIC; 0 SORTED; 0 SORTED_NUMERIC; 0 SORTED_SET] [took 0.000 sec]
    test: points..............OK [0 fields, 0 points] [took 0.000 sec]
    test: vectors.............OK [0 fields, 0 vectors] [took 0.000 sec]
    test: field norms.........OK [1 fields] [took 0.382 sec]
    test: stored fields.......OK [17683646 total field count; avg 2.0 fields per doc] [took 25.261 sec]
    test: terms, freq, prox...OK [11795964 terms; 363490228 terms/docs pairs; 514795921 tokens] [took 82.775 sec]

No problems were detected with this index.

Took 95.669 sec total.

I'll search for the nightly benchmark index to test next.

There is a typo in "deleteions"?

There is a typo in "deleteions"?

Fixed.

-threadCount currently only supports up to 11 threads. Value higher than that will be capped.

LOL!!  Now what am I going to do with my 117 remaining cores!!

OK I re-ran with the latest PR and I LOVE the output -- the lines come as each check finishes, so you can see what is fast/slow.  It seems postings is slowest, preceded by doc values, and everything else is super fast.
It's curious to me that checking vectors seems to be super fast -- is our vectors check also checking integrity the full HNSW graph, not just the float vectors?
I need a tool that randomly flips a single bit somewhere in the index, to see how CheckIndex now behaves on a corrupt index!  Presumably there is some compact UNIX one-liner to do this?  Maybe a perfect job for a quick Python script!

OK, I have good news and bad news.
Good news first!
Bad news!  I ran the tool, confirmed it seems to flip just the one bit, then ran the CheckIndex here, and no corruption was detected!!  Then I also ran CheckIndex from a clean main checkout, and we still fail to detect the corruption.  WTF?  Surely the bit flip would alter the checksum and we should have detected that in CheckIndex?  Or is it possible CheckIndex does not actually fully checkIntegrity too?
For the record, this is how I ran the new bit-flipper tool:
 python3 -u /l/util/src/python/flip_random_bit.py /l/indices/trunk.nightly.index.prev.broken/index -seed 7 -real

and this is its output:
python3 -u /l/util.nightly/src/python/flip_random_bit.py /l/indices/trunk.nightly.index.prev.broken/index -seed 7 -real

RANDOM SEED: 0x7

Directory has 302 files:
  _32.fdm
  _32.fdt
  _32.fdx
  _32.fnm
  _32.kdd
  _32.kdi
  _32.kdm
  _32.nvd
  ...
    _h2.fdm
  _h2.fdt
  _h2.fdx
  _h2.fnm
  _h2.kdd
  _h2.kdi
  _h2.kdm
  _h2.nvd
  _h2.nvm
  _h2.si
  _h2_Lucene90HnswVectorFormat_0.vec
  _h2_Lucene90HnswVectorFormat_0.vem
  _h2_Lucene90HnswVectorFormat_0.vex
  _h2_Lucene90_0.doc
  _h2_Lucene90_0.dvd
  _h2_Lucene90_0.dvm
  _h2_Lucene90_0.pos
  _h2_Lucene90_0.tim
  _h2_Lucene90_0.tip
  _h2_Lucene90_0.tmd
  segments_2
  write.lock

**WARNING**: this tool will soon corrupt bit 39544 (of 152368 bits) in /l/indices/trunk.nightly.index.prev.broken/index/_gm.kdi!!!

Be really certain this is what you want... you have 5 seconds to change your mind!

5...
4...
3...
2...
1...

**BOOOOOOOM**

And then cmp and diff confirm the file is indeed changed, yet CheckIndex (with or without this PR) doesn't catch it.  I'll try a few more bit flips.

Phew, aha!  I caught my mistake!  When I ran CheckIndex, I failed to pass the .broken index -- I was just checking the original (presumably NOT corrupt) index.  I reran with the right argument (.broken version), and good news!:
beast3:core[main]$ java -cp ../core/build/libs/lucene-core-9.0.0-SNAPSHOT.jar org.apache.lucene.index.CheckIndex /l/indices/trunk.nightly.index.prev.broken/index -segment _gm

NOTE: testing will be more thorough if you run java with '-ea:org.apache.lucene...', so assertions are enabled

Opening index @ /l/indices/trunk.nightly.index.prev.broken/index

Checking index with async threadCount: 4
0.00% total deletions; 27625038 documents; 0 deletions
Segments file=segments_2 numSegments=15 version=9.0.0 id=4tts2a3tdztvmyrx4eavuuvca userData={userData=multi}

Checking only these segments: _gm:
9 of 15: name=_gm maxDoc=497803
    version=9.0.0
    id=4tts2a3tdztvmyrx4eavuuvau
    codec=Lucene90
    compound=false
    numFiles=20
    size (MB)=402.17
    diagnostics = {os.arch=amd64, mergeFactor=10, java.runtime.version=15.0.1+9-18, os=Linux, timestamp=1622011837396, lucene.version=9.0.0, java.vm.version=15.0.1+9-18, java.version=15.0.\
1, mergeMaxNumSegments=-1, os.version=5.11.2-arch1-1, java.vendor=Oracle Corporation, source=merge}
    no deletions
    test: open reader.........OK [took 0.054 sec]
    test: check integrity.....FAILED
    WARNING: exorciseIndex() would remove reference to this segment; full exception:
org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=681c742f actual=5f6ae61b (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/l/indices/\
trunk.nightly.index.prev.broken/index/_gm.kdi")))
        at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:440)
        at org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:614)
        at org.apache.lucene.codecs.lucene90.Lucene90PointsReader.checkIntegrity(Lucene90PointsReader.java:143)
        at org.apache.lucene.index.CodecReader.checkIntegrity(CodecReader.java:267)
        at org.apache.lucene.index.SegmentReader.checkIntegrity(SegmentReader.java:391)
        at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:745)
        at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:527)
        at org.apache.lucene.index.CheckIndex.doCheck(CheckIndex.java:4046)
        at org.apache.lucene.index.CheckIndex.doMain(CheckIndex.java:3919)
        at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:3851)

WARNING: 1 broken segments (containing 497803 documents) detected
Took 0.176 sec total.
WARNING: would write new segments file, and 497803 documents would be lost, if -exorcise were specified

OK all is good!  It looks like corruption is indeed still detected with this PR, wonderful!

Not bitcoin mining I hope
On Thu, May 27, 2021 at 8:56 AM Michael McCandless @.***>
wrote:

-threadCount currently only supports up to 11 threads. Value higher than
that will be capped.
LOL!! Now what am I going to do with my 117 remaining cores!!
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/apache/lucene/pull/128#issuecomment-849611238, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAHHUQNLVSCOXD754HTBBLDTPY6PHANCNFSM44GLV3QQ
.

bq. LOL!! Now what am I going to do with my 117 remaining cores!!
I think this is very appropriate:
https://en.wikipedia.org/wiki/Core_War

the lines come as each check finishes, so you can see what is fast/slow. It seems postings is slowest, preceded by doc values, and everything else is super fast.
Can't wait to see the next CheckIndex time in nightly benchmarks after we push this :)

I was finally able to rebuild my local index with wikibigall generating 15 segments, and performed 2 test runs with different threadCount.  With 11 threadCount, it took 359.293 sec in total to finish, and with 1 threadCount it took 378.583 sec in total, so about 5% time saving. I feel faster machine will have better time saving, but in general the speed up seems to be limited given the skewed distribution of checking speed of different segment parts  (e.g. posting check can account for around 85% ~ 90%  of the total time spent in the first segment check).

Good news first!
It looks like corruption is indeed still detected with this PR, wonderful!

The python tool looks very cool and thanks for testing it! One issue though is that this bit flipping is causing checksum integrity check failures before the concurrent segment part checks kick in, so it may not test the changes here? I think we may actually need to write a semantically buggy segment file with good checksum verification to see the error still gets detected and propagated correctly?
With the above two, I feel maybe I should also look into parallelizing across segments and keeping it single-threaded / simple within each segment (much of the learnings here can be applied there anyway)? The other approach to get better speed up could be to split up posting check, so that it can be handled by multiple threads as well. But I'm not sure now if it's easily parallelize-able and also need to look into it further.
I'm also good with merging this in and see how it performs.

I was finally able to rebuild my local index with wikibigall generating 15 segments, and performed 2 test runs with different threadCount. With 11 threadCount, it took 359.293 sec in total to finish, and with 1 threadCount it took 378.583 sec in total, so about 5% time saving. I feel faster machine will have better time saving, but in general the speed up seems to be limited given the skewed distribution of checking speed of different segment parts (e.g. posting check can account for around 85% ~ 90% of the total time spent in the first segment check).

I really don't think we should add this complexity to checkindex if it only improves it 5%. Let's take a step back and do this issue differently (e.g. just check whole segments concurrently with different threads). This should also require less complexity and be less invasive to the checkindex tool.

The python tool looks very cool and thanks for testing it! One issue though is that this bit flipping is causing checksum integrity check failures before the concurrent segment part checks kick in, so it may not test the changes here? I think we may actually need to write a semantically buggy segment file with good checksum verification to see the error still gets detected and propagated correctly?

This is also why checkindex has its -fast option, to only verify file checksums, which is probably what most end-users actually want when trying to verify that their index is intact and correct. The current (historical) slow defaults are really only useful for debugging a bug in lucene itself. And there is even a -slow that can do even more of that.

 The previous -slow could be moved to -slower and the previous current behavior could be activated by -slow.
I think the tool's defaults are unnecessarily slow just for historical reasons? (not having checksums originally)

I've gone ahead and reverted the changes to parallelize within segment, and then added the code that used many of the same ideas to parallelize across segments - with 11 threads the total runtime has been cut down to 130+ seconds, around 65% reduction! I also removed the restriction of using up to 11 threads, as large index may well contain more than 11 segments, so idle cores can be utilized as well!
Please let me know how this looks to you. @mikemccand @rmuir @dweiss

I'm gonna throw out the crazy idea to make -fast the new default. The previous -slow could be moved to -slower and the previous current behavior could be activated by -slow.
I think the tool's defaults are unnecessarily slow just for historical reasons? (not having checksums originally)

This also makes sense and seems to be an easy change to switch the default? Is there anything I need to add specifically so that users can be made aware of this change when they upgrade lucene version (e.g. extra log to indicate the switch) ?

Oh one more thing. As the log output was buffered during parallel execution and printed later in sequential fashion to maintain order, to help out those who might be eager to see the output, for the first segment (which consumes the most of time during check) I have used the "global" infoStream  to print log as they are available - this gives the "weird" printing behavior that the first segment check prints slowly while it progress, and once the first segment finishes then all the subsequent segment outputs got printed at once.  Not sure if this behavior is ok as it may be perceived as buggy by the user?

Latest ok output from 12 threads (my machine actually only has 6 physical cores though) :
4:39:26 PM: Executing task 'CheckIndex.main()'...

> Task :buildSrc:compileJava UP-TO-DATE
> Task :buildSrc:compileGroovy NO-SOURCE
> Task :buildSrc:processResources NO-SOURCE
> Task :buildSrc:classes UP-TO-DATE
> Task :buildSrc:jar UP-TO-DATE
> Task :buildSrc:assemble UP-TO-DATE
> Task :buildSrc:compileTestJava NO-SOURCE
> Task :buildSrc:compileTestGroovy NO-SOURCE
> Task :buildSrc:processTestResources NO-SOURCE
> Task :buildSrc:testClasses UP-TO-DATE
> Task :buildSrc:test NO-SOURCE
> Task :buildSrc:check UP-TO-DATE
> Task :buildSrc:build UP-TO-DATE

> Configure project :
IntelliJ Idea IDE detected.

> Task :errorProneSkipped
WARNING: errorprone disabled (skipped on non-nightly runs)

> Task :lucene:core:processResources UP-TO-DATE
> Task :lucene:core:compileJava
> Task :lucene:core:classes

> Task :lucene:core:CheckIndex.main()

NOTE: testing will be more thorough if you run java with '-ea:org.apache.lucene...', so assertions are enabled

Opening index @ /Users/xichen/IdeaProjects/benchmarks/indices/wikibigall.lucene_baseline.facets.taxonomy:Date.taxonomy:Month.taxonomy:DayOfYear.sortedset:Month.sortedset:DayOfYear.Lucene90.Lucene90.nd6.64758M/index

Checking index with async threadCount: 12
0.00% total deletions; 6647577 documents; 0 deletions
Segments file=segments_2 numSegments=15 version=9.0.0 id=59c6he3dhebad46x7proh30nq userData={userData=multi}
1 of 15: name=_32 maxDoc=1197893
    version=9.0.0
    id=59c6he3dhebad46x7proh2zhm
    codec=Lucene90
    compound=false
    numFiles=17
    size (MB)=2,531.843
    diagnostics = {os.version=10.15.5, mergeMaxNumSegments=-1, java.version=11.0.9, java.vm.version=11.0.9+11, lucene.version=9.0.0, timestamp=1622100146526, os=Mac OS X, java.runtime.version=11.0.9+11, mergeFactor=10, os.arch=x86_64, source=merge, java.vendor=AdoptOpenJDK}
    no deletions
    test: open reader.........OK [took 0.100 sec]
    test: check integrity.....OK [took 18.376 sec]
    test: check live docs.....OK [took 0.000 sec]
    test: field infos.........OK [17 fields] [took 0.000 sec]
    test: field norms.........OK [2 fields] [took 0.053 sec]
    test: terms, freq, prox...OK [20065511 terms; 450728331 terms/docs pairs;<PHONE_NUMBER> tokens] [took 107.241 sec]
    test: stored fields.......OK [3593679 total field count; avg 3.0 fields per doc] [took 0.955 sec]
    test: term vectors........OK [0 total term vector count; avg 0.0 term/freq vector fields per doc] [took 0.000 sec]
    test: docvalues...........OK [10 docvalues fields; 3 BINARY; 0 NUMERIC; 5 SORTED; 0 SORTED_NUMERIC; 2 SORTED_SET] [took 3.127 sec]
    test: points..............OK [2 fields, 2395786 points] [took 0.263 sec]
    test: vectors.............OK [0 fields, 0 vectors] [took 0.000 sec]

2 of 15: name=_65 maxDoc=1197893
    version=9.0.0
    id=59c6he3dhebad46x7proh2zqv
    codec=Lucene90
    compound=false
    numFiles=17
    size (MB)=1,539.981
    diagnostics = {os.version=10.15.5, mergeMaxNumSegments=-1, java.version=11.0.9, java.vm.version=11.0.9+11, lucene.version=9.0.0, timestamp=1622100810971, os=Mac OS X, java.runtime.version=11.0.9+11, mergeFactor=10, os.arch=x86_64, source=merge, java.vendor=AdoptOpenJDK}
    no deletions
    test: open reader.........OK [took 0.100 sec]
    test: check integrity.....OK [took 12.488 sec]
    test: check live docs.....OK [took 0.000 sec]
    test: field infos.........OK [17 fields] [took 0.000 sec]
    test: field norms.........OK [2 fields] [took 0.057 sec]
    test: terms, freq, prox...OK [15042354 terms; 274837439 terms/docs pairs; 686566591 tokens] [took 74.407 sec]
    test: stored fields.......OK [3593679 total field count; avg 3.0 fields per doc] [took 1.087 sec]
    test: term vectors........OK [0 total term vector count; avg 0.0 term/freq vector fields per doc] [took 0.000 sec]
    test: docvalues...........OK [10 docvalues fields; 3 BINARY; 0 NUMERIC; 5 SORTED; 0 SORTED_NUMERIC; 2 SORTED_SET] [took 2.439 sec]
    test: points..............OK [2 fields, 2395786 points] [took 0.185 sec]
    test: vectors.............OK [0 fields, 0 vectors] [took 0.000 sec]

...
...

14 of 15: name=_h1 maxDoc=11979
    version=9.0.0
    id=59c6he3dhebad46x7proh30nj
    codec=Lucene90
    compound=false
    numFiles=17
    size (MB)=12.824
    diagnostics = {source=flush, os.arch=x86_64, java.runtime.version=11.0.9+11, os.version=10.15.5, java.vendor=AdoptOpenJDK, os=Mac OS X, timestamp=1622102788648, java.version=11.0.9, java.vm.version=11.0.9+11, lucene.version=9.0.0}
    no deletions
    test: open reader.........OK [took 0.014 sec]
    test: check integrity.....OK [took 0.172 sec]
    test: check live docs.....OK [took 0.000 sec]
    test: field infos.........OK [17 fields] [took 0.000 sec]
    test: field norms.........OK [2 fields] [took 0.002 sec]
    test: terms, freq, prox...OK [290488 terms; 1843478 terms/docs pairs; 4383419 tokens] [took 0.788 sec]
    test: stored fields.......OK [35937 total field count; avg 3.0 fields per doc] [took 0.014 sec]
    test: term vectors........OK [0 total term vector count; avg 0.0 term/freq vector fields per doc] [took 0.000 sec]
    test: docvalues...........OK [10 docvalues fields; 3 BINARY; 0 NUMERIC; 5 SORTED; 0 SORTED_NUMERIC; 2 SORTED_SET] [took 0.072 sec]
    test: points..............OK [2 fields, 23958 points] [took 0.003 sec]
    test: vectors.............OK [0 fields, 0 vectors] [took 0.000 sec]

15 of 15: name=_h2 maxDoc=11248
    version=9.0.0
    id=59c6he3dhebad46x7proh30nm
    codec=Lucene90
    compound=false
    numFiles=17
    size (MB)=10.617
    diagnostics = {source=flush, os.arch=x86_64, java.runtime.version=11.0.9+11, os.version=10.15.5, java.vendor=AdoptOpenJDK, os=Mac OS X, timestamp=1622102791291, java.version=11.0.9, java.vm.version=11.0.9+11, lucene.version=9.0.0}
    no deletions
    test: open reader.........OK [took 0.011 sec]
    test: check integrity.....OK [took 0.124 sec]
    test: check live docs.....OK [took 0.000 sec]
    test: field infos.........OK [17 fields] [took 0.000 sec]
    test: field norms.........OK [2 fields] [took 0.001 sec]
    test: terms, freq, prox...OK [253387 terms; 1570705 terms/docs pairs; 3390075 tokens] [took 0.589 sec]
    test: stored fields.......OK [33744 total field count; avg 3.0 fields per doc] [took 0.013 sec]
    test: term vectors........OK [0 total term vector count; avg 0.0 term/freq vector fields per doc] [took 0.000 sec]
    test: docvalues...........OK [10 docvalues fields; 3 BINARY; 0 NUMERIC; 5 SORTED; 0 SORTED_NUMERIC; 2 SORTED_SET] [took 0.066 sec]
    test: points..............OK [2 fields, 22496 points] [took 0.004 sec]
    test: vectors.............OK [0 fields, 0 vectors] [took 0.000 sec]

No problems were detected with this index.

Took 130.404 sec total.

BUILD SUCCESSFUL in 2m 12s
4 actionable tasks: 3 executed, 1 up-to-date
4:41:39 PM: Task execution finished 'CheckIndex.main()'.

Some test results with corrupted index (_gx_Lucene90_0.dvd):
Full check
> Task :lucene:core:CheckIndex.main()

NOTE: testing will be more thorough if you run java with '-ea:org.apache.lucene...', so assertions are enabled

Opening index @ /Users/xichen/IdeaProjects/benchmarks/indices/corrupted/index/

Checking index with async threadCount: 12
0.00% total deletions; 6647577 documents; 0 deletions
Segments file=segments_2 numSegments=15 version=9.0.0 id=59c6he3dhebad46x7proh30nq userData={userData=multi}
1 of 15: name=_32 maxDoc=1197893
    version=9.0.0
    id=59c6he3dhebad46x7proh2zhm
    codec=Lucene90
    compound=false
    numFiles=17
    size (MB)=2,531.843
    diagnostics = {timestamp=1622100146526, lucene.version=9.0.0, java.vm.version=11.0.9+11, java.version=11.0.9, mergeMaxNumSegments=-1, os.version=10.15.5, java.vendor=AdoptOpenJDK, source=merge, os.arch=x86_64, mergeFactor=10, java.runtime.version=11.0.9+11, os=Mac OS X}
    no deletions
    test: open reader.........OK [took 0.125 sec]
    test: check integrity.....OK [took 20.451 sec]
    test: check live docs.....OK [took 0.000 sec]
    test: field infos.........OK [17 fields] [took 0.000 sec]
    test: field norms.........OK [2 fields] [took 0.044 sec]
    test: terms, freq, prox...OK [20065511 terms; 450728331 terms/docs pairs;<PHONE_NUMBER> tokens] [took 109.702 sec]
    test: stored fields.......OK [3593679 total field count; avg 3.0 fields per doc] [took 0.967 sec]
    test: term vectors........OK [0 total term vector count; avg 0.0 term/freq vector fields per doc] [took 0.000 sec]
    test: docvalues...........OK [10 docvalues fields; 3 BINARY; 0 NUMERIC; 5 SORTED; 0 SORTED_NUMERIC; 2 SORTED_SET] [took 2.575 sec]
    test: points..............OK [2 fields, 2395786 points] [took 0.204 sec]
    test: vectors.............OK [0 fields, 0 vectors] [took 0.000 sec]

2 of 15: name=_65 maxDoc=1197893
    version=9.0.0
    id=59c6he3dhebad46x7proh2zqv
    codec=Lucene90
    compound=false
    numFiles=17
    size (MB)=1,539.981
    diagnostics = {timestamp=1622100810971, lucene.version=9.0.0, java.vm.version=11.0.9+11, java.version=11.0.9, mergeMaxNumSegments=-1, os.version=10.15.5, java.vendor=AdoptOpenJDK, source=merge, os.arch=x86_64, mergeFactor=10, java.runtime.version=11.0.9+11, os=Mac OS X}
    no deletions
    test: open reader.........OK [took 0.124 sec]
    test: check integrity.....OK [took 13.612 sec]
    test: check live docs.....OK [took 0.000 sec]
    test: field infos.........OK [17 fields] [took 0.000 sec]
    test: field norms.........OK [2 fields] [took 0.042 sec]
    test: terms, freq, prox...OK [15042354 terms; 274837439 terms/docs pairs; 686566591 tokens] [took 76.072 sec]
    test: stored fields.......OK [3593679 total field count; avg 3.0 fields per doc] [took 0.982 sec]
    test: term vectors........OK [0 total term vector count; avg 0.0 term/freq vector fields per doc] [took 0.000 sec]
    test: docvalues...........OK [10 docvalues fields; 3 BINARY; 0 NUMERIC; 5 SORTED; 0 SORTED_NUMERIC; 2 SORTED_SET] [took 2.351 sec]
    test: points..............OK [2 fields, 2395786 points] [took 0.194 sec]
    test: vectors.............OK [0 fields, 0 vectors] [took 0.000 sec]

...
...

10 of 15: name=_gx maxDoc=119789
    version=9.0.0
    id=59c6he3dhebad46x7proh30n7
    codec=Lucene90
    compound=false
    numFiles=17
    size (MB)=129.046
    diagnostics = {timestamp=1622102767300, lucene.version=9.0.0, java.vm.version=11.0.9+11, java.version=11.0.9, mergeMaxNumSegments=-1, os.version=10.15.5, java.vendor=AdoptOpenJDK, source=merge, os.arch=x86_64, mergeFactor=10, java.runtime.version=11.0.9+11, os=Mac OS X}
    no deletions
    test: open reader.........OK [took 0.125 sec]
    test: check integrity.....FAILED
    WARNING: exorciseIndex() would remove reference to this segment; full exception:
org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=87e2aa4 actual=7b3afcbd (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/Users/xichen/IdeaProjects/benchmarks/indices/corrupted/index/_gx_Lucene90_0.dvd")))
	at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:440)
	at org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:614)
	at org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer.checkIntegrity(Lucene90DocValuesProducer.java:1656)
	at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.checkIntegrity(PerFieldDocValuesFormat.java:364)
	at org.apache.lucene.index.CodecReader.checkIntegrity(CodecReader.java:252)
	at org.apache.lucene.index.SegmentReader.checkIntegrity(SegmentReader.java:391)
	at org.apache.lucene.index.CheckIndex.testSegment(CheckIndex.java:913)
	at org.apache.lucene.index.CheckIndex.lambda$checkIndex$0(CheckIndex.java:742)
	at org.apache.lucene.index.CheckIndex.lambda$callableToSupplier$1(CheckIndex.java:842)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

11 of 15: name=_gy maxDoc=11979
    version=9.0.0
    id=59c6he3dhebad46x7proh30na
    codec=Lucene90
    compound=false
    numFiles=17
    size (MB)=13.473
    diagnostics = {os.version=10.15.5, java.runtime.version=11.0.9+11, os.arch=x86_64, source=flush, lucene.version=9.0.0, java.vm.version=11.0.9+11, java.version=11.0.9, timestamp=1622102778710, os=Mac OS X, java.vendor=AdoptOpenJDK}
    no deletions
    test: open reader.........OK [took 0.124 sec]
    test: check integrity.....OK [took 0.065 sec]
    test: check live docs.....OK [took 0.000 sec]
    test: field infos.........OK [17 fields] [took 0.000 sec]
    test: field norms.........OK [2 fields] [took 0.005 sec]
    test: terms, freq, prox...OK [295343 terms; 1930448 terms/docs pairs; 4662478 tokens] [took 1.269 sec]
    test: stored fields.......OK [35937 total field count; avg 3.0 fields per doc] [took 0.077 sec]
    test: term vectors........OK [0 total term vector count; avg 0.0 term/freq vector fields per doc] [took 0.000 sec]
    test: docvalues...........OK [10 docvalues fields; 3 BINARY; 0 NUMERIC; 5 SORTED; 0 SORTED_NUMERIC; 2 SORTED_SET] [took 0.130 sec]
    test: points..............OK [2 fields, 23958 points] [took 0.014 sec]
    test: vectors.............OK [0 fields, 0 vectors] [took 0.000 sec]

...
...

15 of 15: name=_h2 maxDoc=11248
    version=9.0.0
    id=59c6he3dhebad46x7proh30nm
    codec=Lucene90
    compound=false
    numFiles=17
    size (MB)=10.617
    diagnostics = {os.version=10.15.5, java.runtime.version=11.0.9+11, os.arch=x86_64, source=flush, lucene.version=9.0.0, java.vm.version=11.0.9+11, java.version=11.0.9, timestamp=1622102791291, os=Mac OS X, java.vendor=AdoptOpenJDK}
    no deletions
    test: open reader.........OK [took 0.014 sec]
    test: check integrity.....OK [took 0.155 sec]
    test: check live docs.....OK [took 0.000 sec]
    test: field infos.........OK [17 fields] [took 0.000 sec]
    test: field norms.........OK [2 fields] [took 0.001 sec]
    test: terms, freq, prox...OK [253387 terms; 1570705 terms/docs pairs; 3390075 tokens] [took 0.703 sec]
    test: stored fields.......OK [33744 total field count; avg 3.0 fields per doc] [took 0.015 sec]
    test: term vectors........OK [0 total term vector count; avg 0.0 term/freq vector fields per doc] [took 0.000 sec]
    test: docvalues...........OK [10 docvalues fields; 3 BINARY; 0 NUMERIC; 5 SORTED; 0 SORTED_NUMERIC; 2 SORTED_SET] [took 0.074 sec]
    test: points..............OK [2 fields, 22496 points] [took 0.004 sec]
    test: vectors.............OK [0 fields, 0 vectors] [took 0.000 sec]

WARNING: 1 broken segments (containing 119789 documents) detected
Took 134.361 sec total.
WARNING: would write new segments file, and 119789 documents would be lost, if -exorcise were specified

> Task :lucene:core:CheckIndex.main() FAILED

Execution failed for task ':lucene:core:CheckIndex.main()'.
> Process 'command '/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home/bin/java'' finished with non-zero exit value 1

Check with -segment _gx flag
> Task :lucene:core:CheckIndex.main() FAILED

NOTE: testing will be more thorough if you run java with '-ea:org.apache.lucene...', so assertions are enabled

Opening index @ /Users/xichen/IdeaProjects/benchmarks/indices/corrupted/index/

Checking index with async threadCount: 12
0.00% total deletions; 6647577 documents; 0 deletions
Segments file=segments_2 numSegments=15 version=9.0.0 id=59c6he3dhebad46x7proh30nq userData={userData=multi}

Checking only these segments: _gx:
10 of 15: name=_gx maxDoc=119789
    version=9.0.0
    id=59c6he3dhebad46x7proh30n7
    codec=Lucene90
    compound=false
    numFiles=17
    size (MB)=129.046
    diagnostics = {timestamp=1622102767300, lucene.version=9.0.0, java.vm.version=11.0.9+11, java.version=11.0.9, mergeMaxNumSegments=-1, os.version=10.15.5, java.vendor=AdoptOpenJDK, source=merge, os.arch=x86_64, mergeFactor=10, java.runtime.version=11.0.9+11, os=Mac OS X}
    no deletions
    test: open reader.........OK [took 0.086 sec]
    test: check integrity.....FAILED
    WARNING: exorciseIndex() would remove reference to this segment; full exception:
org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=87e2aa4 actual=7b3afcbd (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/Users/xichen/IdeaProjects/benchmarks/indices/corrupted/index/_gx_Lucene90_0.dvd")))
    at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:440)
    at org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:614)
    at org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer.checkIntegrity(Lucene90DocValuesProducer.java:1656)
    at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.checkIntegrity(PerFieldDocValuesFormat.java:364)
    at org.apache.lucene.index.CodecReader.checkIntegrity(CodecReader.java:252)
    at org.apache.lucene.index.SegmentReader.checkIntegrity(SegmentReader.java:391)
    at org.apache.lucene.index.CheckIndex.testSegment(CheckIndex.java:913)
    at org.apache.lucene.index.CheckIndex.lambda$checkIndex$0(CheckIndex.java:742)
    at org.apache.lucene.index.CheckIndex.lambda$callableToSupplier$1(CheckIndex.java:842)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)

WARNING: 1 broken segments (containing 119789 documents) detected
Took 0.734 sec total.
WARNING: would write new segments file, and 119789 documents would be lost, if -exorcise were specified

Execution failed for task ':lucene:core:CheckIndex.main()'.
> Process 'command '/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home/bin/java'' finished with non-zero exit value 1

This also makes sense and seems to be an easy change to switch the default? Is there anything I need to add specifically so that users can be made aware of this change when they upgrade lucene version (e.g. extra log to indicate the switch) ?

Maybe we should just open a separate issue for that. The change could be for 9.0-only, have its own issue, CHANGES.txt, entry MIGRATE.txt entry, etc.

Thank you for all the awesome iterations here @zacharymorn!
To get the best speedup, even at -slow, we should do concurrency both ways, and then sort those tasks by decreasing expected cost.  This way the work queue would first output all postings checks (across all segments), one per thread, followed by doc values, etc.
But we can add such complexity later -- the PR now ("thread per segment") is surely a great step forward too :)
And +1 to spinoff a separate issue to change CheckIndex to default to -fast -- this is really long overdue since we added end-to-end checksums to Lucene!

One issue though is that this bit flipping is causing checksum integrity check failures before the concurrent segment part checks kick in, so it may not test the changes here? I think we may actually need to write a semantically buggy segment file with good checksum verification to see the error still gets detected and propagated correctly?

You're right -- it is able to do that.  It's random, so if it flips a "really important" bit, such that the segments_N file cannot be loaded, then CheckIndex will fail, quickly, without checking the segments.  If you watch its output and re-run it until you get a single segment corruption event, then it should showcase the gains in this PR.

Oh one more thing. As the log output was buffered during parallel execution and printed later in sequential fashion to maintain order, to help out those who might be eager to see the output, for the first segment (which consumes the most of time during check) I have used the "global" infoStream to print log as they are available - this gives the "weird" printing behavior that the first segment check prints slowly while it progress, and once the first segment finishes then all the subsequent segment outputs got printed at once. Not sure if this behavior is ok as it may be perceived as buggy by the user?

Hmm, could we instead just output the full segment's output as each finishes?  So we would presumably see small segments finish first, then all checks for that small segment are printed?

To get the best speedup, even at -slow, we should do concurrency both ways, and then sort those tasks by decreasing expected cost.

This fine-grained concurrency (one task for "part X segment") would then mean we can get the postings check of the big segments all going, early, concurrently across all segments, and likely the biggest segments postings check would be the long pole, and {{CheckIndex}} would finish in (about) that much time.
Versus "thread per segment" concurrency, where the long pole will be one thread checking the biggest segment.
The really fun part of concurrency across and within segments will be figure out the rough heuristic of how to assign estimated cost to each task :)
But this all can come later!   "thread per segment" is already a big step forward!

Also, note that the "first segment" might not necessarily always be the largest!  The way our default merge policy works, that tends to be the case, but users can customize the merge policy, and we recently added a feature to let IndexWriter customize how it sorts the segments.  We should maybe sort the segments by descending docCount and work on them (concurrently) in that order?

To get the best speedup, even at -slow, we should do concurrency both ways, and then sort those tasks by decreasing expected cost.

This fine-grained concurrency (one task for "part X segment") would then mean we can get the postings check of the big segments all going, early, concurrently across all segments, and likely the biggest segments postings check would be the long pole, and {{CheckIndex}} would finish in (about) that much time.
Versus "thread per segment" concurrency, where the long pole will be one thread checking the biggest segment.
The really fun part of concurrency across and within segments will be figure out the rough heuristic of how to assign estimated cost to each task :)
But this all can come later! "thread per segment" is already a big step forward!

Yup this indeed will be both fun and challenging to implement! I think maybe it's also worth looking into breaking posting check into parallelize-able parts, as they alone can often take up 80% - 90% of the entire segment check time?

I've created a spinoff issue to track the default setting change https://issues.apache.org/jira/browse/LUCENE-9984

Hi @mikemccand, I have addressed the comment above with some additional changes and posted some updated results. Could you please let me know if they look ready?

Thanks @zacharymorn -- the changes look awesome! I'm looking forward to faster CheckIndex!

Thanks @mikemccand again for the review and approval, and I look forward to the speedup as well! I'll merge in a few days just in case if anyone else may have further comment.

Hi @mikemccand, I've merged this PR and will wait for an update on the nightly check index time page. Once the result there looks good, I believe we should backport this change to 8x as well?


validation dataset folder structure for dataset type 2

Hi, I'm training a vocals_mdx23c so am using dataset 2 from my understanding

The training folder and config file is all set up
I've confirmed this is working with a quick run
The validation folder isn't correct and is bugging

I'm a little confused, how do I set up the validation folder? The same as the training folder (training -> (vocals and other folders))

validation dataset is in musdb format so if you are using vocals have folders as such
valid_folder -
1 - vocals.wav other.wav mixture.wav
2 - vocals.wav other.wav mixture.wav
3 - vocals.wav other.wav mixture.wav
4 - vocals.wav other.wav mixture.wav

Dataset must be the same as Type 1 for training.
More details here:
https://github.com/ZFTurbo/Music-Source-Separation-Training/blob/main/docs/dataset_types.md#dataset-for-validation

Thank you @ZFTurbo working now :)


GET parameters from main URL lost when exporting

Summary of problem or feature request
The URL for exporting is the current domain and path, with a set of GET parameters. The page being exported may already have its own GET parameters, for filtering the data for example. These are discarded, so the exported data is not filtered like the page it is being exported from.
I need to leave the office now, but will come back and add further details. Just getting this in while it's fresh.
Code snippet of problem

System details

Operating System
PHP Version
Laravel Version
Laravel-DataTables Version

I'm going to have to leave this for another day, as I can't work out how this plugin is supposed to be used anyway (something to do with models needs to be setup, I believe, except that I'm exporting from queries with filters, not models). So I'll come back to this when exporting becomes more important on my project.


Strip Lombok from all modules

In anticipation of Spring Boot 3.2 and Java 21, specifically the ability to produce Native images utilizing GraalVM, we'll want to strip away Lombok from all child modules.
Remove dependencies on lombok in pom.xml files
Refactor source to make use of Record types (where necessary)
Refactor source to replace @Slf4j annotations with private static final Logger
Remove acrobatics in pom.xml files for delombok-ing source

Completed.


Please include complete code(include thread id) instead of snippet

[Enter feedback here]
Please include complete code(include thread id,main() method etc) instead of snippet below. This will make it easy to run and possibly understand . Make sure the code can be executed just by copying .
For this Snippet code :
private readonly HttpClient _httpClient = new HttpClient();
[HttpGet, Route("DotNetCount")]
public async Task GetDotNetCount()
{
// Suspends GetDotNetCount() to allow the caller (the web server)
// to accept another request, rather than blocking on this one.
var html = await _httpClient.GetStringAsync("https://dotnetfoundation.org");
return Regex.Matches(html, @"\.NET").Count;

}

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: d885f1eb-2b37-693b-c91f-74b748c9424a
Version Independent ID: 77bdee53-c2a0-e16f-c93a-3ca0afae0b93
Content: Asynchronous programming - C#
Content Source: docs/csharp/asynchronous-programming/async-scenarios.md
Product: dotnet-csharp
Technology: csharp-async
GitHub Login: @BillWagner
Microsoft Alias: wiwagn

Thanks for noting this @kirancobe
Our newer samples generally do include the full source, with a link to our samples repo. As we update older samples, we make sure to add them.
Adding this to our backlog for when we next update this article.

@BillWagner  this might be a good issue to add the code help wanted label like we've done. We've had good luck with getting help on those

@BillWagner I will handle that!


sriov-cni plugin silently throws away issues with LoadConfFromCache

What happened?
sriov-cni plugin silently throws away issues with LoadConfFromCache: https://github.com/k8snetworkplumbingwg/sriov-cni/blob/c44726a013e0b87470151bebe6a87034f181fa95/cmd/sriov/main.go#L201
What did you expect to happen?
That's on purpose and it makes sense, however, by doing so, we lose insight into valid errors. There should be a logging mechanism to the crio logs (via stderr) which will at least log the failure.
What are the minimal steps needed to reproduce the bug?
I saw this in a customer environment, I actually don't know how to best reproduce this with minimal steps.
Anything else we need to know?
I think it'd be nice to have optional debug logging (loglevel and logfile) for the sriov-cni. That would make troubleshooting in the field (e.g. of race conditions) a lot easier.
Component Versions
Please fill in the below table with the version numbers of applicable components used.

Component
Version

SR-IOV CNI Plugin

Multus

SR-IOV Network Device Plugin

Kubernetes

OS

Config Files
Config file locations may be config dependent.
CNI config (Try '/etc/cni/net.d/')
Device pool config file location (Try '/etc/pcidp/config.json')
Multus config (Try '/etc/cni/multus/net.d')
Kubernetes deployment type ( Bare Metal, Kubeadm etc.)
Kubeconfig file
SR-IOV Network Custom Resource Definition
Logs
SR-IOV Network Device Plugin Logs (use kubectl logs $PODNAME)
Multus logs (If enabled. Try '/var/log/multus.log' )
Kubelet logs (journalctl -u kubelet)

@andreaskaris can we close this one ?

of course!


Doc: fix typo on network typo in doc/rest-api.yaml

Signed-off-by: Hiroaki Nakamura<EMAIL_ADDRESS>

This pull request didn't trigger Jenkins as its author isn't in the whitelist.
An organization member must perform one of the following:

To have this branch tested by Jenkins, use the "ok to test" command.
To have a one time test done, use the "test this please" command.

Those commands are simple Github comments of the format: "jenkins: COMMAND"

Thanks!


Schema Deconflict

Description
As discussed in the comments of PR https://github.com/haskell-works/avro/pull/117 (see https://github.com/haskell-works/avro/pull/117#issuecomment-549036337 in particular), schema deconflicting would be a little more sensible if it functioned directly on the schemas and not on avro values.  That is, it would make more sense for schema deconflicting to have a type like
deconflict :: Schema -> Schema -> Either String Schema

than
deconflict :: Schema -> Schema -> T.Value Type -> Either String (T.Value Type)

Indeed, acting directly on the schemas allows us to go one step further: it becomes simple to make the deconflicting process lazy, only forcing strictness if the user wants to call fromAvro strictly.  Thus, we can actually make deconflicting have  the type:
deconflict :: Schema -> Schema -> Schema

This PR introduces such a deconflict function.
Changes
Internally, I needed to slightly extend the type of Schema and Field to make this possible.  This had knock-on effects on both the strict and lazy versions of getAvroOf.  Specifically, for Schema, I added  the following constructors:
data Type = ...
  | IntLongCoercion
  | IntFloatCoercion
  | IntDoubleCoercion
  | LongFloatCoercion
  | LongDoubleCoercion
  | FloatDoubleCoercion
  | FreeUnion { ty :: Type }
  | Panic { ty :: Type, err :: String }

The first 6 are for numeric type coercions and are pretty straightforward.  We need to know the original type so that we parse correctly, and then we need to know the reader type so that we do  the right conversion.
The FreeUnion constructor is to allow the situation where the writer does not expect a union type but the reader does.  In this case, we parse assuming no union and then inject the value into a "free" singleton union.  (Note that the reverse situation weirdly used to be allowed in deconflicting, but I didn't support it in this version.   It doesn't make a lot of sense to me, although it could be supported with something like FakeUnion { ty :: Type, options :: Vector Type }.)
The Panic constructor is what allows the Schema to be lazy.  It contains the type that the reader expects to parse (so lazy parsing can continue) but its presence indicates that deconflicting failed.  For lazy parsing, an Error value is produced with  the given error message, and for strict parsing, the error message is immediately thrown.
The Field type has one new field: fldStatus  :: FieldStatus where
data FieldStatus =
    AsIs
  | Ignored
  | Defaulted (Ty.Value Type)

By default, all fields  have  status AsIs, indicating that they are normal fields.  If a field is not present in the writer but is in the reader (with a default value), then the status is set to Defaulted with the default (and if no default is present, the field's type is set to Panic).  On the other hand, if the field is in the reader but not the writer, then its status is set to Ignored.  We keep it present in the schema so that it can be properly parsed, but the parsing functions then ignore the values they found.
Deprecation
I'm not sure what you guys want to do with the old deconflicting code, but to prevent breakages, I've currently left the modules there with deprecation pragmas.
Performance
The benchmark suite is not set up to handle benchmarking deconflicting in this way; it produces many avro objects that are then converted to different avro objects via the old deconflicting process.  The new method skips this step entirely by doing a single pass through the schema and then parsing directly into the final avro object.  I'm not sure the best way to compare these methods.
Looking Further
Having the avro object as an intermediate type used to be necessary for schema deconflicting, and the ability to deconflict (i.e., be backwards compatible) is a really great feature of avro.  With this new deconflicting style, the intermediate avro representation is no longer strictly necessary.  In other words, one could write a custom version of getAvroOf and hook it up to decodeContainerWith and parse straight to some target value they want.  Basically, that probably means we can write a version of fromAvro that goes straight from bytestring to Haskell types, which would be pretty cool!

Wow, thanks, great work! I will find time to look at it soon!

the intermediate avro representation is no longer strictly necessary
This is definitely a big step into the right direction!

@dwincort I started looking at this PR and realised that with these changes our Schema becomes non-trippable from/to JSON. That's it, we can't encode it to JSON and decode into the same value.
I don't know how actually important this property is, but it is a nice property to have, so I need to look closer and think about it.
@TikhonJelvis, you have done a fair amount of work on Schema, what do you think?
Alternatively we could maybe have this extra information, that is only needed for deconflicting into another data type? Such that deconflicting would result into some Either Error (Schema, Fixes)?
It would complicate FromAvro a bit, like instead of being
fromAvro :: Value Type -> Result a

it would be
fromAvro :: Fixes -> Value Type -> Result a

but I don't know if it could be a major showstopper...
@newhoggy what do you think?

@dwincort I started looking at this PR and realised that with these changes our Schema becomes non-trippable from/to JSON. That's it, we can't encode it to JSON and decode into the same value.

Hmm, is this simply because I didn't extend the parseSchemaJSON and schemaToJSON functions with the new data I added to schemas?  If so, I believe that's an easy (backwards-compatible) fix.
If this is about the fact that there are now schemas for which the before and after of a decode/encode round-trip on a value are no longer identical, then yes, perhaps that's worth thinking about.  If we extend schemas to allow for deconflicting, then there will no longer be a one-to-one mapping between schemas and avro values (that is, the Value type).  But, that's the point, right?  That's what deconflicting is all about: the fact that there are multiple ways to turn the bytes into a value.
Your suggestion to change fromAvro seems to stem from this issue of the one-to-one mapping: if Value no longer maps one-to-one to the bytes, then rather than extending the schema, we can add Fixes to fromAvro to reconstruct that mapping.  This does make the lack of bijectivity more obvious (an extra argument is needed), but now it means that fromAvro and toAvro no longer round-trip (unless Fixes is empty or something).  Perhaps an alternative is to embrace the idea that values of Value no longer uniquely identify their schema.  Maybe there should be a way to provide a schema when encoding that will override default behavior (for instance, telling an Int to be encoded as a Float).
When I made the new deconflict function, I thought about what it would mean to provide a binary operator to the Schema type.  Can deconflict be called multiple times?  If so, it's certainly not commutative, but is it associative?  Is Schema a semigroup now?  Or some sort of partial semigroup?  I basically punted on all of these questions: deconflict is an internal, somewhat advanced feature that really only ever needs to be called once (at the time of decoding), so I decided not to worry too much.  I think it's associative, but I would say doing more than one call to deconflict or using a deconflicted schema more than just for decoding should probably be "unsupported behavior".

is this simply because I didn't extend the parseSchemaJSON and schemaToJSON functions with the new data I added to schemas?

My concern is that if we do extend FromJSON/ToJSON, then our JSON representation for Schema will have extra properties that are not defined by Avro specification.
For example, I am not sure if we could send such a JSON to services like Schema Registry and likewise.
And if we don't extend it, then the round trip is broken.

but now it means that fromAvro and toAvro no longer round-trip (unless Fixes is empty or something).

Yes, I guess that there could be empty Fixes, or perhaps it can even have a Monoid instance (for example, if we define it similarly to newtype Fixes = Fixes (Map FieldPath ExtraInfo))

My concern is that if we do extend FromJSON/ToJSON, then our JSON representation for Schema will have extra properties that are not defined by Avro specification.

Ahhh, I understand now.  Perhaps it would make more sense to treat deconflicted schemas as a special type: essentially use a newtype wrapper (e.g., newtype DeconflictedSchema = DeconflictedSchema Schema) to keep them separate and distinct from regular schemas.  After all, deconflicted schemas "should" only be used for parsing data (they can't even be reliably used to go in the other direction due to sum types).
This is similar to the idea of having a paired Fixes type (consider that DeconflictedSchema could be defined as (Fixes, Shema)), but I actually think Fixes will be pretty painful to implement.  Keep in mind that it's not just fields that can be different but also numeric types and sum types, and although your suggestion for FieldPath over Field may contain extra differentiating information about fields, it may be annoying to keep track of.  Also, Fixes seems to still require an intermediate avro type, preventing a future where we can go directly from bytes to Haskell.
So, perhaps we should leave the JSON instances as they are, we mark that the new coercions and error types are not properly supported Schema components, and we make clear (through documentation or something like a newtype) that deconflicted schemas are special.  How does that sound?
Basically what I'm proposing comes down to something like:
-- Constructor only exposed in some internal module
newtype DeconflictedSchema = DeconflictedSchema { getDeconflictedSchema :: Schema }
  deriving (...) -- Not deriving ToJSON or FromJSON

-- Member function only exposed in some internal module
class IsSchema a where
  getSchema :: a -> Schema

instance IsSchema Schema where
  getSchema = id
instance IsSchema DeconflictedSchema where
  getSchema = getDeconflictedSchema

deconflict :: (IsSchema s1, IsSchema s2) => s1 -> s2 -> DeconflictedSchema
decodeContainerWithSchema :: (IsSchema s, FromAvro a) => s -> ByteString -> [[a]]

Just pinging this to see if there's any further thoughts.
I came across a case at work where it would be great if we could see whether two schemas were compatible (i.e., if they could be deconflicted) even though we have no actual values to deconflict.  The approach in this PR (or something similar) would let us get a deconflicted schema, and by scanning it for Panic, we could see if the schema were valid or not.

@dwincort Sorry, it is a busy time of the year for me now.
I experimenting on your branch (slowly) and investigating if we can fit the extended schema into an Avro Specification.
The specification says:
{"type": "typeName" ...attributes...}
where typeName is either a primitive or derived type name, as defined below. Attributes not defined in this document are permitted as metadata, but must not affect the format of serialized data.

and I thought that we could indeed serialise/deserialise extended schema by adding extra attributes, but now I think that it can indeed be treated as "affecting the format of serialised data" because what we say here is that, for example, Long should be read as Int from the serialised bytes.
And these options should only affect decoding and should not (and must not) take any effect when data is encoded.
So I had an idea of maybe annotating Schema constructors with some read rules, such as:
data ReadLong = LongAsIs | LongFromInt
data ReadDouble = DoubleAsIs | DoubleFromFloat | DoubleFromLong

data Schema
  = Int
  | Long ReadLong
  | Double ReadDouble

or maybe use GADTs?
data ReadRule a where
  AsIs :: ReadRule a
  LongFromInt :: ReadRule Int64
  DoubleFromFloat :: ReadRule Double
  DoubleFromLong :: ReadRule Double

data Schema
  = Int
  | Long (ReadRule Int64)
  | Double (ReadRule Double)

or maybe even GADTs with DataKinds?
data ReadRule a where
  AsIs :: ReadRule a
  LongFromInt :: ReadRule 'Schema.Long
  DoubleFromFloat :: ReadRule 'Schema.Double
  DoubleFromLong :: ReadRule 'Schema.Double

This way we probably won't need new constructors in Schema, we could simply adjust read rules in annotations.
If that works then can we do it as per Trees that grow paper, or even with a simplified approach and hide these annotations in "decoration"?
This is what I am trying to do now in your branch: to see if I can avoid having new constructors and serialise this schema to JSON as something like
{ "type": { "type": "Long", "readRule": "FromInt" } }

and it'd not violate the Avro Specification.
I think that it could also be achieved with extra constructors that you have added, but I'd like to see if it can be nicer with annotations (or not).
One thing that I'd like to change is to get rid of Panic constructor.
I would much prefer to make Schema strict and to get Either err Schema from a deconflicting function.
Another question that I had was: FieldStatus has Defaulted constructor. Why do we need it instead of setting the existing fldDefault field? If we can do that, then we perhaps don't need FieldStatus at all, we could just add fldReadIgnore flag to Field?
Again, a huge question for me is how to serialise changes in Field to JSON. Avro Specification doesn't mention anything about having extra attributes at that level.

Alternatively we could try making an extended copy of Schema, call it ReadSchema and only use it internally...
This way we would keep Schema Avro Spec compatible, although that would require some code duplication...

So I had an idea of maybe annotating Schema constructors with some read rules

Clever!  All of those options looks good, although I think I'm partial to the middle one (GADTs without DataKinds).

One thing that I'd like to change is to get rid of Panic constructor.  I would much prefer to make Schema strict and to get Either err Schema from a deconflicting function.

This is fine by me.  The only reason I added the Panic constructor was to support lazy deconflicting, but if you don't want this, I'm happy to cut it.

Another question that I had was: FieldStatus has Defaulted constructor. Why do we need it instead of setting the existing fldDefault field? If we can do that, then we perhaps don't need FieldStatus at all, we could just add fldReadIgnore flag to Field?

Yup, looking at the code now after being away from it for a bit, FieldStatus seems redundant and overkill.  A boolean fldReadIgnore flag in Field looks like it would be sufficient.

we could try making an extended copy of Schema, call it ReadSchema and only use it internally

This is very much what I was thinking about with the DeconflictedSchema newtype.  It's not exactly my favorite option, but I don't think it would be that much code duplication.

I have two more questions to consider.
FreeUnion
writing with the schema

@dwincort Well, I have tried the annotations approach (https://github.com/haskell-works/avro/pull/127) and I tried two schemas approach. Both went terribly complicated.
So perhaps your solution is the best middle-ground :)
I think I am willing to compromise on the JSON thing and to keep new constructors in Schema.
I will add them to JSON encoding/decoding thing and will ping you on that PR.
I think this excuse can be used: If you serialise a deconflicted Schema to JSON then you'll get a "standard" AVSC JSON with one extra field on Field type.
Most likely it won't break anything (since it is just an extra field) and it is highly unlikely that anyone would do it anyway...

Yes, that works.
I also think now that schemaToJson should just serialise these new constructors as their original counterparts:
  IntLongCoercion     -> A.String "long"
  IntFloatCoercion    -> A.String "float"
  IntDoubleCoercion   -> A.String "double"
  LongFloatCoercion   -> A.String "float"
  LongDoubleCoercion  -> A.String "double"
  FloatDoubleCoercion -> A.String "double"
  FreeUnion ty        -> schemaToJSON context ty

The reason is that I'd like to stay conservative just in case if anyone (future me) gets this "extended" schema and uses it somewhere in a container or in Schema Registry. Then the values that are written with this schema could not be read with this schema anymore.
This is still a problem: now we have a variation of a schema that doesn't obey fromAvro . toAvro = id property. This is kind of big, but I don't have a solution to it at the moment.
Now, if you remove Panic from Schema we can merge this PR and iterate on it.

I removed Panic as  a constructor from Schema.
Would you like to make the schemaToJSON changes or should I?

I know I just removed Panic, but it occurred to me that if we wanted, we could keep the laziness in this library and have Panic be precisely that: an error.  In other words, we could make schemaToJSON :: Maybe TypeName -> Schema -> Either String A.Value and define the ToJSON instance as:
toJSON = either error id . schemaToJSON Nothing

This is a little ugly because, presumably, toJSON shouldn't throw an error.  But, in some sense, this is no worse than any time one has a lazy value that might throw an error when it is fully evaluated.  Furthermore, we could introduce a function like strictifySchema :: Schema -> Either String Schema which is basically like seq combined with an error handler.

Yup, looking at the code now after being away from it for a bit, FieldStatus seems redundant and overkill. A boolean fldReadIgnore flag in Field looks like it would be sufficient.

I just tried implementing this, and I see now why I made FieldStatus in the first place.  We need to be able to distinguish the case of simply having a default vs knowing when to use it.  Basically, we have the following cases:

The field is in both schemas, and there is no default.  Deconflicting works fine, and we should read the value during decoding.
The field is in both schemas, and there is a default.  Deconflicting works fine, and we should read the value during decoding.  The default is ignored (but should still be in the schema).
The field is in the reader but not the writer, and there is no default.  Deconflicting fails.
The field is in the reader but not the writer, and there is a default.  Deconflicting succeeds, and we should not read the value during decoding.  Use the default instead.
The field is in the writer but not the reader (we never use defaults from the writer, so whether there is a default or not doesn't matter).  We should read the value, but we should ignore what it is.

There are 5 cases, but a pair of (Maybe Type, Bool) only covers 4 cases.  Cases 1 and 2 are the critical ones that are hard to distinguish with (Maybe Type, Bool).

We need to be able to distinguish the case of simply having a default vs knowing when to use it.

Yes, you are right! Maybe in this case instead of Bool it'd be cleaner to use a variation of your initial implementation like
data FieldStatus = AsIs | Ignored | Defaulted

This at least would describe the intent cleaner for future maintenance. We can do it later, too.
Could you please rebase do we can merge?

I had a thought...
When we eventually remove the Value type and do direct encoding/decoding, it could probably look like:
class FromEncoding a where
   fromEncoding :: Schema -> ByteString -> Either Text a

instance FromEncoding Int64 where
  fromEncoding Schema.Long bs = getLong bs
  fromEncoding Schema.Int bs = fromIntegral <$> getInt bs
  fromEncoding sch _ = Left ("Unable to decode as Long: " <> sch)

With this approach coercion becomes simpler and deconflicting would only need to make sure schema's compatibility (and ignored/defaulted fields).
So perhaps in time we will be able to improve this library quite a lot: better performance, removal of Value, removal lazy/strict duplication...
This PR is a first step to a greater future! :)

I have moved this PR to schema-deconflict branch so that we can finish it off there (JSON encoding, etc.).
I will make my own PR(s) to that branch and once it is done will merge to master.


Pick 8.20~2025.01: first preview with all packages that work as is

Merging despite the Windows failure to get forward.
The Windows failure likely needs a port of
https://github.com/coq/platform/blob/main/opam/opam-repository/packages/coq-core/coq-core.8.19.2/files/0001-Windows-fix-path-normalization-in-coqdep.patch
to 8.20

Since I anyway can't do the "please pick" issues today, I decided to try to fix the Windows issue before merging.

Merging - the remaining Windows issues will be fixed in a separate step.


Support mirroring existing sessions

Mirroring works only for new TCP sessions.
We can make it work with ongoing TCP sessions by finding HTTP requests and starting to stream from there.
Should resolve #397

It would be nice to have this supported. This makes mirrord more straight forward with how the doc describes it.

I just needed this for already established websocket connections

We won't be able to support mirroring established websocket connections because those are stateful connections

Is it doable for stealing?

It already works this way for stealing.

With #1887 this should now work if your session is using HTTP!
There is a small caveat in the implementation:
Currently only handle it if the packet data starts with valid HTTP bytes, something along the lines of: ["GET / HTTP"]. It fails to detect HTTP traffic if the bytes are "messed up", e.g. ["some other data \0 GET / HTTP]. There is an implicit assumption here that we're going to be hitting the common case of seeing a "well-behaved" packet most of the time.

A quick fix to this, if it ever poses a problem:

Perform HTTP detection on the packet, and move the bytes' cursor to the start of the HTTP part, example:
["some other data \0 GET / HTTP]
 /\                 /\
 |                  |
we're starting here
                    |
                    |
      create a slice that starts from here


Cannot load the paxml gpt3 tokenizer

Hello,
I have followed the instructions in here to download paxml weights of gpt3 and its tokenizer (vocab folder) and tried using it in tokenizer_path like this. But it results in the following error:
0: Traceback (most recent call last):
0:   File "/home/goramesh/maxtext/MaxText/train.py", line 682, in <module>
0:     app.run(main)
0:   File "/home/goramesh/.local/lib/python3.10/site-packages/absl/app.py", line 308, in run
0:     _run_main(main, args)
0:   File "/home/goramesh/.local/lib/python3.10/site-packages/absl/app.py", line 254, in _run_main
0:     sys.exit(main(argv))
0:   File "/home/goramesh/maxtext/MaxText/train.py", line 678, in main
0:     train_loop(config)
0:   File "/home/goramesh/maxtext/MaxText/train.py", line 590, in train_loop
0:     example_batch = load_next_batch(data_iterator, example_batch, config)
0:   File "/home/goramesh/maxtext/MaxText/train.py", line 99, in load_next_batch
0:     return next(train_iter)
0:   File "/home/goramesh/maxtext/MaxText/multihost_dataloading.py", line 119, in __next__
0:     return get_next_batch_sharded(self.local_iterator, self.global_mesh)
0:   File "/home/goramesh/maxtext/MaxText/multihost_dataloading.py", line 78, in get_next_batch_sharded
0:     local_data = next(local_iterator)
0:   File "/pyenv/versions/3.10.14/lib/python3.10/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 4700, in __next__
0:     return nest.map_structure(to_numpy, next(self._iterator))
0:   File "/pyenv/versions/3.10.14/lib/python3.10/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 814, in __next__
0:     return self._next_internal()
0:   File "/pyenv/versions/3.10.14/lib/python3.10/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 777, in _next_internal
0:     ret = gen_dataset_ops.iterator_get_next(
0:   File "/pyenv/versions/3.10.14/lib/python3.10/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 3028, in iterator_get_next
0:     _ops.raise_from_not_ok_status(e, name)
0:   File "/pyenv/versions/3.10.14/lib/python3.10/site-packages/tensorflow/python/framework/ops.py", line 6656, in raise_from_not_ok_status
0:     raise core._status_to_exception(e) from None  # pylint: disable=protected-access
0: tensorflow.python.framework.errors_impl.InternalError: {{function_node __wrapped__IteratorGetNext_output_types_6_device_/job:localhost/replica:0/task:0/device:CPU:0}} external/com_google_sentencepiece/src/sentencepiece_processor.cc(842) [!IsUnknown(PieceToId(absl::string_view(model_->bos_piece().data())))] id for `<s>` is not defined.
0:       [[{{node SentenceTokenizer/SentenceTokenizer/SentencepieceTokenizeOp}}]] [Op:IteratorGetNext] name:

So is the path in s3 the right path to the tokenizer?

Thank you @gramesh-amd.
It might be an permission issue.
Have you tried to download the tokenizer file to your own bucket and change the path tokenizer_path accordingly?
In parallel, we are looking at bucket permission gs://mlperf-llm-public2/.

Yes I have downloaded the tokenizer from S3 bucket that was provided here and the name seems to match what I see in google's mlperf submission scripts
But I get the above error (id for <s> is not defined), so im not sure if its the right tokenizer

Hi, @gramesh-amd
We haven't seen this error before.
Do you have a service account in your project, and we can grant you the access to the original bucket gs://mlperf-llm-public2/.

<EMAIL_ADDRESS>is my email (ive also created a google account with this same address)

We should have granted you the access. Could you take another try?

Thanks, will check

it works after i download from the gs://mlperf-llm-public2/ bucket
Thank you


Add a workflow to scan vulnerable dependencies

Add a new workflow of GitHub Actions to detect vulnerable dependencies using Dependency Review
dependency-review-action

Done -> https://github.com/lucoenergia/conluz/blob/main/.github/workflows/dependency-review.yml


Added cached auth tokens

Part of #7
Added implementation of AuthTokens with guava evicted cache under the hood.

@g4s8 please, take a look/merge

@g4s8 ping

@olenagerasimova I granted you write permissions in this repo, please try to merge

@g4s8 thanks, all good


[BUG] Adding a new attack before the last attack in a simfile behaves strangely

Please give your StepMania Info
StepMania Version: StepMania 5.3-git-744831257b (issue also confirmed on 5.1.0-b1)
Theme: Soundwaves (5.3), Lambda (5.1)
Describe the problem
Adding an attack in the editor that occurs before the last attack in the file is subject to strange behavior, specifically when dealing with attacks that override/cancel other attacks.
As an example: Let's say I add an attack at measure 1 that applies No Beat, and then add an attack later at measure 2 that applies Beat. If I play this chart, Beat is applied.
However, let's say I add an attack at measure 2 that applies Beat, then go back in at a later time and add an attack at measure 1 to apply No Beat. Beat is not applied in this example, where it was applied in the last one despite No Beat coming before Beat in the chart in both cases.
Describe what should happen
In both cases, I would assume the intention would be for the behavior to be consistent no matter the order I add the attacks in (it shouldn't matter if I added X or Y mod first, as long as X mod comes before Y mod in the final chart).
Logs and Additional context
info.txt
log.txt

likely won't be fixed as we're working on the new editor which doesn't suffer from this method. Will keep this open until then.