Concurrent Searching (Experimental)

Signed-off-by: Andriy Redko<EMAIL_ADDRESS>Description
Allows to use experimental Apache Lucene low-level API which allows to parallelize execution of the search across segment.

[x] Implement multi-collectors query context (possibly, needs API change to use CollectorManagers instead)

[X] Implement profilers support
The query profiling is supported but it has some limitations and differences with respect to naming. The profilers capture the time each collection type has taken and return it as the cumulative summary. The fact that segments may have been searched concurrently is not reflected (it requires significant overhauling of the profiling implementation), so the "real" perceived time may be smaller then profilers will report. The collector names are replaced with collector manager names (fe MinimumScoreCollector vs MinimumCollectorManager).
To compare, here is the usual search profile
"collector" : [
   {
     "name" : "MinimumScoreCollector",
     "reason" : "search_min_score",
     "time_in_nanos" : 4702942,
     "children" : [
       {
         "name" : "SimpleTopScoreDocCollector",
         "reason" : "search_top_hits",
         "time_in_nanos" : 3450252
       }
     ]
   }
 ]

vs the one where concurrent segment search is enabled
"collector" : [
  {
    "name" : "MinimumCollectorManager",
    "reason" : "search_min_score",
    "time_in_nanos" : 74827,
    "children" : [
      {
        "name" : "SimpleTopDocsCollectorManager",
        "reason" : "search_top_hits",
        "time_in_nanos" : 98044
      }
    ]
  }
]

Additionally, since TopDocs are merged, the query's shardIndex field is also populated, usual search profile has it set to -1, for example:
"query" : [
  {
    "type" : "ConstantScoreQuery",
    "description" : "ConstantScore(SearchAfterSortedDocQuery(sort=<int: \"rank\">, afterDoc=doc=287 score=NaN shardIndex=-1 fields=[-2062713761]))",
    "time_in_nanos" : 1009712,
    ...
  }
]

vs the one where concurrent segment search is enabled
"query" : [
  {
    "type" : "ConstantScoreQuery",
    "description" : "ConstantScore(SearchAfterSortedDocQuery(sort=<int: \"rank\">, afterDoc=doc=287 score=NaN shardIndex=1 fields=[-2062713761]))",
    "time_in_nanos" : 35667,
      ....
  }
]

[X] Implement early termination and timeout support
The forced early termination is simulated (without abrupt search termination) using Apache Lucene normal collection termination mechanism. The number of results is reduced to the desired one (if necessary) at the post search step. Essentially, to summarize, when early termination is requested and concurrent segment search is enabled, be the engine is going to do more work and (very likely) come up with more search hits than have been asked.
If query finished by timeout, no results are being returned, even if returning partial results is acceptable. The reason for that exception propagation mechanism does not leave a chance for reducers to execute.

[x] Add benchmarks suite

Splitting into subtasks:

[x] Extract changes related to profiler support (https://github.com/opensearch-project/OpenSearch/pull/1673)
[ ] Extract changes related to query context support
[ ] Move the concurrent search implementation to sandbox

Future Enhancements:

[ ] Implement aggregations support

Issues Resolved
Fixes https://github.com/opensearch-project/OpenSearch/issues/1286
Check List

[X] New functionality includes testing.

[X] All tests pass

[X] New functionality has been documented.

[X] New functionality has javadoc added

[X] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@anasalkouz still work in progress, but large chunk of the work has been done already, would be great to hear your thoughts, thank you.

Preliminary macro benchmarks:
Using rally with pmc track:
rally race --track pmc --target-hosts localhost:9200 --pipeline benchmark-only --track-params="force_merge_max_num_segments:20"   --kill-running-processes

Note: desired throughput for search tasks (default, term, phrase) has been increased from 20 ops/s to 50 ops/s. The benchmarks were running on the same node as Opensearch server.
Opensearch configuration: 1 node, -Xmx4g, all other JVM options kept unchanged
Hardware: i7 10th gen (12 cores), Ubuntu 21.10
Side by side Comparison

Metric
Task
#1 seq
#2 seq
#1 concur
#2 concur
Unit

Segment count

248
240
262
265

Min Throughput
default
49.54
49.54
49.41
49.42
ops/s

Mean Throughput
default
49.59
49.59
49.48
49.48
ops/s

Median Throughput
default
49.59
49.59
49.48
49.49
ops/s

Max Throughput
default
49.64
49.64
49.54
49.54
ops/s

50th percentile latency
default
8.01098
9.04569
6.98968
6.67651
ms

90th percentile latency
default
12.0628
17.3791
10.4292
11.063
ms

99th percentile latency
default
21.9861
32.1529
17.705
12.3328
ms

100th percentile latency
default
30.4458
36.6906
24.7913
12.392
ms

50th percentile service time
default
6.634
7.88215
5.67168
5.32377
ms

90th percentile service time
default
10.7652
14.7414
8.47473
9.57959
ms

99th percentile service time
default
16.0205
17.5388
12.6657
10.8485
ms

100th percentile service time
default
29.1359
35.5346
23.4873
11.27
ms

error rate
default
0
0
0
0
%

Min Throughput
term
49.83
49.69
49.86
49.82
ops/s

Mean Throughput
term
49.85
49.72
49.88
49.84
ops/s

Median Throughput
term
49.85
49.73
49.88
49.84
ops/s

Max Throughput
term
49.86
49.76
49.9
49.86
ops/s

50th percentile latency
term
10.3155
9.07136
7.7421
7.84862
ms

90th percentile latency
term
13.0895
10.3129
9.42092
10.5664
ms

99th percentile latency
term
23.3423
17.8859
27.9807
21.3598
ms

100th percentile latency
term
28.1488
25.9053
31.4844
31.0409
ms

50th percentile service time
term
9.10882
8.0295
6.21632
6.66419
ms

90th percentile service time
term
11.5806
8.76473
7.94414
8.36192
ms

99th percentile service time
term
15.5951
14.6678
15.1197
14.1959
ms

100th percentile service time
term
15.7594
23.4487
23.4279
30.0025
ms

error rate
term
0
0
0
0
%

Min Throughput
phrase
49.81
49.78
49.79
49.77
ops/s

Mean Throughput
phrase
49.83
49.8
49.81
49.8
ops/s

Median Throughput
phrase
49.84
49.8
49.82
49.81
ops/s

Max Throughput
phrase
49.85
49.83
49.83
49.83
ops/s

50th percentile latency
phrase
6.97764
6.50972
5.8718
5.91395
ms

90th percentile latency
phrase
8.76465
8.82422
8.03951
7.22674
ms

99th percentile latency
phrase
16.792
14.9502
11.5943
12.4171
ms

100th percentile latency
phrase
28.3097
29.4971
21.9315
12.977
ms

50th percentile service time
phrase
5.61003
5.35381
4.51923
4.7856
ms

90th percentile service time
phrase
7.38752
7.42578
6.4436
5.31315
ms

99th percentile service time
phrase
11.0224
9.91337
9.79194
10.3351
ms

100th percentile service time
phrase
26.4146
27.7785
20.5301
11.4582
ms

error rate
phrase
0
0
0
0
%

The index_searcher thread pool stats (averaged):
"index_searcher" : {
    "threads" : 12,
    "queue" : 0,
    "active" : 0,
    "rejected" : 0,
    "largest" : 12,
    "completed" : 519319
}

FYI - gradle check ran out of JVM and stalled
...
> Task :qa:full-cluster-restart:v1.3.0#bwcTest
> Task :qa:full-cluster-restart:bwcTestSnapshots
> Task :qa:full-cluster-restart:check
Expiring Daemon because JVM heap space is exhausted
Expiring Daemon because JVM heap space is exhausted
Build was aborted
Aborted by petern

x   Gradle Check failure 4e5eb47 Log 1047
Reports 1047

This time another one:
> Task :test:fixtures:minio-fixture:composeUp

ERROR: for 81df22525be5077429a7afdcff93efee_minio-fixture__minio-fixture-other_1  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for 81df22525be5077429a7afdcff93efee_minio-fixture__minio-fixture-for-snapshot-tool_1  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for 81df22525be5077429a7afdcff93efee_minio-fixture__minio-fixture_1  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for minio-fixture-other  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for minio-fixture-for-snapshot-tool  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for minio-fixture  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

x   Gradle Check failure 8c3283f Log 1113
Reports 1113

org.opensearch.cluster.routing.allocation.decider.ConcurrentRecoveriesAllocationDeciderTests > testClusterConcurrentRecoveries FAILED
    java.lang.AssertionError:
    Expected: <40>
         but: was <38>
        at __randomizedtesting.SeedInfo.seed([4129940D97376593:9BC7C76A97C50944]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.opensearch.cluster.routing.allocation.decider.ConcurrentRecoveriesAllocationDeciderTests.testClusterConcurrentRecoveries(ConcurrentRecoveriesAllocationDeciderTests.java:221)

@anasalkouz @nknize @dblock guys, I think the implementation wise we are done, I have documented all the concerns / differences encountered so far in the description. I have left aggregations support out of scope right now and re-targeted it as a future enhancement, otherwise the change would be quite large.
If you could find time to go over the changes, would be awesome. I am working on benchmarks at the moment, but I don't expect any significant changes in the implementation (:crossed_fingers: ), only recommendations regarding where concurrent segment search gives most benefits.
Thanks!

Benchmarking: 1 Opensearch node, -Xmx8g, all other JVM options kept unchanged
Using rally with pmc and ny taxis tracks (running on the same node)
Hardware: i7 10th gen (12 cores), Ubuntu 21.10
Baseline: Opensearch-2.0.0-SNAPSHOT, built locally
Compared with: Opensearch-2.0.0-SNAPSHOT with concurrent segment search on/off

rally with pmc (the test suite does not respect the force_merge_max_num_segments setting strictly)

search.allow_concurrent_segment_search: true

    esrally race
        --track pmc
        --target-hosts localhost:9200
        --pipeline benchmark-only
        --track-params="force_merge_max_num_segments:20"
        --kill-running-processes

Metric
Task
Baseline
CSS is on
Diff
Unit

Cumulative indexing time of primary shards

73.353
71.4803
-1.87265
min

Min cumulative indexing time across primary shard

14.5511
14.1031
-0.44792
min

Median cumulative indexing time across primary shard

14.62
14.3464
-0.27363
min

Max cumulative indexing time across primary shard

14.8236
14.482
-0.34168
min

Cumulative indexing throttle time of primary shards

0
0
0
min

Min cumulative indexing throttle time across primary shard

0
0
0
min

Median cumulative indexing throttle time across primary shard

0
0
0
min

Max cumulative indexing throttle time across primary shard

0
0
0
min

Cumulative merge time of primary shards

27.059
26.6039
-0.45517
min

Cumulative merge count of primary shards

38
41
3

Min cumulative merge time across primary shard

4.63982
4.6181
-0.02172
min

Median cumulative merge time across primary shard

5.09635
5.14325
0.0469
min

Max cumulative merge time across primary shard

6.29838
6.22662
-0.07177
min

Cumulative merge throttle time of primary shards

15.7683
14.9557
-0.81265
min

Min cumulative merge throttle time across primary shard

2.42433
2.45457
0.03023
min

Median cumulative merge throttle time across primary shard

2.90923
3.09637
0.18713
min

Max cumulative merge throttle time across primary shard

3.81042
3.65813
-0.15228
min

Cumulative refresh time of primary shards

3.82447
3.85612
0.03165
min

Cumulative refresh count of primary shards

122
119
-3

Min cumulative refresh time across primary shard

0.436283
0.44525
0.00897
min

Median cumulative refresh time across primary shard

0.794433
0.800833
0.0064
min

Max cumulative refresh time across primary shard

1.02345
0.929133
-0.09432
min

Cumulative flush time of primary shards

5.52572
4.9777
-0.54802
min

Cumulative flush count of primary shards

53
53
0

Min cumulative flush time across primary shard

0.86085
0.848833
-0.01202
min

Median cumulative flush time across primary shard

1.1037
0.996583
-0.10712
min

Max cumulative flush time across primary shard

1.34618
1.19045
-0.15573
min

Total Young Gen GC time

7.752
7.673
-0.079
s

Total Young Gen GC count

610
541
-69

Total Old Gen GC time

0
0
0
s

Total Old Gen GC count

0
0
0

Store size

19.6312
19.6209
-0.01029
GB

Translog size

2.56114e-07
2.56114e-07
0
GB

Heap used for segments

0.475765
0.434555
-0.04121
MB

Heap used for doc values

0.0863686
0.0776367
-0.00873
MB

Heap used for terms

0.260193
0.237183
-0.02301
MB

Heap used for norms

0.0448608
0.0408936
-0.00397
MB

Heap used for points

0
0
0
MB

Heap used for stored fields

0.084343
0.0788422
-0.0055
MB

Segment count

147
134
-13

Min Throughput
index-append
1438.19
1427.61
-10.5783
docs/s

Mean Throughput
index-append
1479.18
1498.1
18.9174
docs/s

Median Throughput
index-append
1478.8
1501.25
22.4509
docs/s

Max Throughput
index-append
1513.22
1529.06
15.8383
docs/s

50th percentile latency
index-append
1766.4
1660.51
-105.882
ms

90th percentile latency
index-append
4318.88
4082.85
-236.032
ms

99th percentile latency
index-append
10543.6
9281.66
-1261.9
ms

100th percentile latency
index-append
12085.1
13125.8
1040.66
ms

50th percentile service time
index-append
1766.4
1660.51
-105.882
ms

90th percentile service time
index-append
4318.88
4082.85
-236.032
ms

99th percentile service time
index-append
10543.6
9281.66
-1261.9
ms

100th percentile service time
index-append
12085.1
13125.8
1040.66
ms

error rate
index-append
0
0
0
%

Min Throughput
default
19.9224
19.9455
0.02302
ops/s

Mean Throughput
default
19.9333
19.953
0.01965
ops/s

Median Throughput
default
19.934
19.9533
0.01929
ops/s

Max Throughput
default
19.9422
19.9594
0.01722
ops/s

50th percentile latency
default
13.2887
14.4078
1.1191
ms

90th percentile latency
default
15.8232
15.4572
-0.366
ms

99th percentile latency
default
16.402
17.1835
0.78149
ms

100th percentile latency
default
17.9581
20.8119
2.85384
ms

50th percentile service time
default
11.9169
13.0856
1.16871
ms

90th percentile service time
default
14.339
13.7786
-0.56044
ms

99th percentile service time
default
14.7929
16.1739
1.38093
ms

100th percentile service time
default
16.2705
18.7671
2.4966
ms

error rate
default
0
0
0
%

Min Throughput
term
20.0068
20.0072
0.00034
ops/s

Mean Throughput
term
20.0082
20.0085
0.00024
ops/s

Median Throughput
term
20.0081
20.0084
0.00027
ops/s

Max Throughput
term
20.0098
20.0101
0.00033
ops/s

50th percentile latency
term
12.3187
13.1509
0.83219
ms

90th percentile latency
term
13.0416
14.0367
0.9951
ms

99th percentile latency
term
14.0721
14.9672
0.89504
ms

100th percentile latency
term
20.6105
15.3812
-5.22924
ms

50th percentile service time
term
10.9906
11.7696
0.77906
ms

90th percentile service time
term
11.4454
12.3908
0.9454
ms

99th percentile service time
term
12.4976
12.9763
0.47875
ms

100th percentile service time
term
18.8219
13.1709
-5.65103
ms

error rate
term
0
0
0
%

Min Throughput
phrase
19.9959
20.0058
0.00986
ops/s

Mean Throughput
phrase
19.9964
20.007
0.01062
ops/s

Median Throughput
phrase
19.9963
20.007
0.01077
ops/s

Max Throughput
phrase
19.997
20.0083
0.01135
ops/s

50th percentile latency
phrase
11.4029
10.6266
-0.77634
ms

90th percentile latency
phrase
12.4928
11.4927
-1.00008
ms

99th percentile latency
phrase
12.8898
12.426
-0.46377
ms

100th percentile latency
phrase
13.0831
23.833
10.7499
ms

50th percentile service time
phrase
10.0775
9.2115
-0.86595
ms

90th percentile service time
phrase
11.0157
10.0319
-0.98375
ms

99th percentile service time
phrase
11.4366
10.9051
-0.53148
ms

100th percentile service time
phrase
11.5139
23.0101
11.4962
ms

error rate
phrase
0
0
0
%

Min Throughput
articles_monthly_agg_uncached
19.9501
19.9041
-0.04605
ops/s

Mean Throughput
articles_monthly_agg_uncached
19.9569
19.9173
-0.03957
ops/s

Median Throughput
articles_monthly_agg_uncached
19.9572
19.9179
-0.03937
ops/s

Max Throughput
articles_monthly_agg_uncached
19.963
19.9284
-0.03459
ops/s

50th percentile latency
articles_monthly_agg_uncached
26.1548
25.9931
-0.16163
ms

90th percentile latency
articles_monthly_agg_uncached
36.2858
35.9679
-0.31789
ms

99th percentile latency
articles_monthly_agg_uncached
37.3572
36.5001
-0.8571
ms

100th percentile latency
articles_monthly_agg_uncached
37.592
36.9453
-0.64679
ms

50th percentile service time
articles_monthly_agg_uncached
24.4331
24.3308
-0.10232
ms

90th percentile service time
articles_monthly_agg_uncached
34.9095
34.5745
-0.33495
ms

99th percentile service time
articles_monthly_agg_uncached
35.5737
35.0186
-0.55514
ms

100th percentile service time
articles_monthly_agg_uncached
35.8452
35.2612
-0.58397
ms

error rate
articles_monthly_agg_uncached
0
0
0
%

Min Throughput
articles_monthly_agg_cached
20.019
20.0163
-0.00273
ops/s

Mean Throughput
articles_monthly_agg_cached
20.0219
20.0191
-0.00286
ops/s

Median Throughput
articles_monthly_agg_cached
20.0216
20.019
-0.00264
ops/s

Max Throughput
articles_monthly_agg_cached
20.026
20.022
-0.00394
ops/s

50th percentile latency
articles_monthly_agg_cached
6.09055
6.05907
-0.03148
ms

90th percentile latency
articles_monthly_agg_cached
6.60807
6.6646
0.05653
ms

99th percentile latency
articles_monthly_agg_cached
6.87594
7.2865
0.41057
ms

100th percentile latency
articles_monthly_agg_cached
7.29908
12.1076
4.80848
ms

50th percentile service time
articles_monthly_agg_cached
4.68761
4.71812
0.03051
ms

90th percentile service time
articles_monthly_agg_cached
4.83927
4.82664
-0.01263
ms

99th percentile service time
articles_monthly_agg_cached
5.07305
5.54625
0.4732
ms

100th percentile service time
articles_monthly_agg_cached
5.30228
11.0191
5.71682
ms

error rate
articles_monthly_agg_cached
0
0
0
%

Min Throughput
scroll
12.5441
12.5444
0.00027
pages/s

Mean Throughput
scroll
12.5725
12.573
0.00046
pages/s

Median Throughput
scroll
12.566
12.5664
0.00043
pages/s

Max Throughput
scroll
12.6307
12.6316
0.00094
pages/s

50th percentile latency
scroll
841.209
802.79
-38.419
ms

90th percentile latency
scroll
856.124
816.946
-39.1778
ms

99th percentile latency
scroll
1081.3
1039.82
-41.4734
ms

100th percentile latency
scroll
1288.54
1046.17
-242.379
ms

50th percentile service time
scroll
838.475
799.652
-38.8226
ms

90th percentile service time
scroll
852.988
814.058
-38.9299
ms

99th percentile service time
scroll
1078.36
1035.92
-42.4385
ms

100th percentile service time
scroll
1283.86
1043.37
-240.492
ms

error rate
scroll
0
0
0
%

Conclusions: slightly better phrase search latencies, much better scroll latencies
search.allow_concurrent_segment_search: false

Metric
Task
Baseline
Contender
Diff
Unit

Cumulative indexing time of primary shards

73.353
74.4449
1.09187
min

Min cumulative indexing time across primary shard

14.5511
14.5689
0.01783
min

Median cumulative indexing time across primary shard

14.62
14.8743
0.2543
min

Max cumulative indexing time across primary shard

14.8236
15.1092
0.28557
min

Cumulative indexing throttle time of primary shards

0
0
0
min

Min cumulative indexing throttle time across primary shard

0
0
0
min

Median cumulative indexing throttle time across primary shard

0
0
0
min

Max cumulative indexing throttle time across primary shard

0
0
0
min

Cumulative merge time of primary shards

27.059
27.9727
0.91367
min

Cumulative merge count of primary shards

38
38
0

Min cumulative merge time across primary shard

4.63982
5.03685
0.39703
min

Median cumulative merge time across primary shard

5.09635
5.4399
0.34355
min

Max cumulative merge time across primary shard

6.29838
6.53803
0.23965
min

Cumulative merge throttle time of primary shards

15.7683
14.229
-1.53933
min

Min cumulative merge throttle time across primary shard

2.42433
2.45758
0.03325
min

Median cumulative merge throttle time across primary shard

2.90923
2.8375
-0.07173
min

Max cumulative merge throttle time across primary shard

3.81042
3.25628
-0.55413
min

Cumulative refresh time of primary shards

3.82447
3.72902
-0.09545
min

Cumulative refresh count of primary shards

122
121
-1

Min cumulative refresh time across primary shard

0.436283
0.465983
0.0297
min

Median cumulative refresh time across primary shard

0.794433
0.7847
-0.00973
min

Max cumulative refresh time across primary shard

1.02345
0.899417
-0.12403
min

Cumulative flush time of primary shards

5.52572
5.90765
0.38193
min

Cumulative flush count of primary shards

53
53
0

Min cumulative flush time across primary shard

0.86085
0.982517
0.12167
min

Median cumulative flush time across primary shard

1.1037
1.11095
0.00725
min

Max cumulative flush time across primary shard

1.34618
1.37955
0.03337
min

Total Young Gen GC time

7.752
7.316
-0.436
s

Total Young Gen GC count

610
506
-104

Total Old Gen GC time

0
0
0
s

Total Old Gen GC count

0
0
0

Store size

19.6312
19.6199
-0.01128
GB

Translog size

2.56114e-07
2.56114e-07
0
GB

Heap used for segments

0.475765
0.457466
-0.0183
MB

Heap used for doc values

0.0863686
0.0805893
-0.00578
MB

Heap used for terms

0.260193
0.251343
-0.00885
MB

Heap used for norms

0.0448608
0.043335
-0.00153
MB

Heap used for points

0
0
0
MB

Heap used for stored fields

0.084343
0.0821991
-0.00214
MB

Segment count

147
142
-5

Min Throughput
index-append
1438.19
1344.01
-94.1829
docs/s

Mean Throughput
index-append
1479.18
1405.28
-73.9064
docs/s

Median Throughput
index-append
1478.8
1409.59
-69.2131
docs/s

Max Throughput
index-append
1513.22
1439.74
-73.4733
docs/s

50th percentile latency
index-append
1766.4
1605.22
-161.172
ms

90th percentile latency
index-append
4318.88
4247.33
-71.556
ms

99th percentile latency
index-append
10543.6
10040.4
-503.137
ms

100th percentile latency
index-append
12085.1
11392.1
-693.05
ms

50th percentile service time
index-append
1766.4
1605.22
-161.172
ms

90th percentile service time
index-append
4318.88
4247.33
-71.556
ms

99th percentile service time
index-append
10543.6
10040.4
-503.137
ms

100th percentile service time
index-append
12085.1
11392.1
-693.05
ms

error rate
index-append
0
0
0
%

Min Throughput
default
19.9224
19.9848
0.06235
ops/s

Mean Throughput
default
19.9333
19.987
0.05372
ops/s

Median Throughput
default
19.934
19.9871
0.05308
ops/s

Max Throughput
default
19.9422
19.9887
0.04658
ops/s

50th percentile latency
default
13.2887
15.429
2.14036
ms

90th percentile latency
default
15.8232
17.0159
1.19271
ms

99th percentile latency
default
16.402
17.735
1.333
ms

100th percentile latency
default
17.9581
21.2407
3.2826
ms

50th percentile service time
default
11.9169
14.2218
2.30488
ms

90th percentile service time
default
14.339
15.5023
1.16323
ms

99th percentile service time
default
14.7929
16.2826
1.48966
ms

100th percentile service time
default
16.2705
19.7867
3.51622
ms

error rate
default
0
0
0
%

Min Throughput
term
20.0068
20.0078
0.00094
ops/s

Mean Throughput
term
20.0082
20.009
0.00075
ops/s

Median Throughput
term
20.0081
20.0089
0.00078
ops/s

Max Throughput
term
20.0098
20.0104
0.00057
ops/s

50th percentile latency
term
12.3187
15.581
3.26234
ms

90th percentile latency
term
13.0416
16.9058
3.86423
ms

99th percentile latency
term
14.0721
17.7038
3.63163
ms

100th percentile latency
term
20.6105
17.9863
-2.62418
ms

50th percentile service time
term
10.9906
14.4275
3.43696
ms

90th percentile service time
term
11.4454
15.4056
3.96026
ms

99th percentile service time
term
12.4976
16.1373
3.63976
ms

100th percentile service time
term
18.8219
16.2621
-2.55981
ms

error rate
term
0
0
0
%

Min Throughput
phrase
19.9959
20.0051
0.00912
ops/s

Mean Throughput
phrase
19.9964
20.006
0.00954
ops/s

Median Throughput
phrase
19.9963
20.006
0.00974
ops/s

Max Throughput
phrase
19.997
20.0069
0.00998
ops/s

50th percentile latency
phrase
11.4029
11.9631
0.56024
ms

90th percentile latency
phrase
12.4928
12.788
0.29521
ms

99th percentile latency
phrase
12.8898
13.7797
0.88995
ms

100th percentile latency
phrase
13.0831
13.8166
0.73349
ms

50th percentile service time
phrase
10.0775
10.6109
0.53342
ms

90th percentile service time
phrase
11.0157
11.3261
0.31049
ms

99th percentile service time
phrase
11.4366
12.0162
0.57966
ms

100th percentile service time
phrase
11.5139
12.43
0.91613
ms

error rate
phrase
0
0
0
%

Min Throughput
articles_monthly_agg_uncached
19.9501
19.9446
-0.00554
ops/s

Mean Throughput
articles_monthly_agg_uncached
19.9569
19.952
-0.00487
ops/s

Median Throughput
articles_monthly_agg_uncached
19.9572
19.9525
-0.00472
ops/s

Max Throughput
articles_monthly_agg_uncached
19.963
19.9587
-0.0043
ops/s

50th percentile latency
articles_monthly_agg_uncached
26.1548
25.5446
-0.61016
ms

90th percentile latency
articles_monthly_agg_uncached
36.2858
35.9621
-0.32362
ms

99th percentile latency
articles_monthly_agg_uncached
37.3572
37.0854
-0.27182
ms

100th percentile latency
articles_monthly_agg_uncached
37.592
67.3733
29.7813
ms

50th percentile service time
articles_monthly_agg_uncached
24.4331
23.9472
-0.48589
ms

90th percentile service time
articles_monthly_agg_uncached
34.9095
34.1571
-0.75231
ms

99th percentile service time
articles_monthly_agg_uncached
35.5737
35.437
-0.13672
ms

100th percentile service time
articles_monthly_agg_uncached
35.8452
65.6705
29.8254
ms

error rate
articles_monthly_agg_uncached
0
0
0
%

Min Throughput
articles_monthly_agg_cached
20.019
20.0173
-0.00173
ops/s

Mean Throughput
articles_monthly_agg_cached
20.0219
20.0198
-0.00212
ops/s

Median Throughput
articles_monthly_agg_cached
20.0216
20.0196
-0.00203
ops/s

Max Throughput
articles_monthly_agg_cached
20.026
20.0232
-0.00275
ops/s

50th percentile latency
articles_monthly_agg_cached
6.09055
6.22025
0.12969
ms

90th percentile latency
articles_monthly_agg_cached
6.60807
6.70296
0.09489
ms

99th percentile latency
articles_monthly_agg_cached
6.87594
6.98675
0.11081
ms

100th percentile latency
articles_monthly_agg_cached
7.29908
7.59941
0.30033
ms

50th percentile service time
articles_monthly_agg_cached
4.68761
4.7369
0.04928
ms

90th percentile service time
articles_monthly_agg_cached
4.83927
4.84741
0.00813
ms

99th percentile service time
articles_monthly_agg_cached
5.07305
5.24661
0.17357
ms

100th percentile service time
articles_monthly_agg_cached
5.30228
5.32888
0.0266
ms

error rate
articles_monthly_agg_cached
0
0
0
%

Min Throughput
scroll
12.5441
12.546
0.00185
pages/s

Mean Throughput
scroll
12.5725
12.5754
0.00286
pages/s

Median Throughput
scroll
12.566
12.5688
0.00275
pages/s

Max Throughput
scroll
12.6307
12.6363
0.00565
pages/s

50th percentile latency
scroll
841.209
831.541
-9.66844
ms

90th percentile latency
scroll
856.124
853.704
-2.41952
ms

99th percentile latency
scroll
1081.3
1102.74
21.44
ms

100th percentile latency
scroll
1288.54
2215.96
927.412
ms

50th percentile service time
scroll
838.475
828.236
-10.2391
ms

90th percentile service time
scroll
852.988
846.114
-6.87344
ms

99th percentile service time
scroll
1078.36
1100.06
21.7012
ms

100th percentile service time
scroll
1283.86
2213.21
929.346
ms

error rate
scroll
0
0
0
%

Conclusions: very small variance across search-related latencies

rally with nyc_taxis

search.allow_concurrent_segment_search: true

    esrally race
        --track nyc_taxis
        --target-hosts localhost:9200
        --pipeline benchmark-only
        --track-params="force_merge_max_num_segments:20"
        --kill-running-processes

Metric
Task
Baseline
Contender
Diff
Unit

Cumulative indexing time of primary shards

158.785
159.293
0.50845
min

Min cumulative indexing time across primary shard

158.785
159.293
0.50845
min

Median cumulative indexing time across primary shard

158.785
159.293
0.50845
min

Max cumulative indexing time across primary shard

158.785
159.293
0.50845
min

Cumulative indexing throttle time of primary shards

0
0
0
min

Min cumulative indexing throttle time across primary shard

0
0
0
min

Median cumulative indexing throttle time across primary shard

0
0
0
min

Max cumulative indexing throttle time across primary shard

0
0
0
min

Cumulative merge time of primary shards

57.0389
55.9897
-1.04917
min

Cumulative merge count of primary shards

99
104
5

Min cumulative merge time across primary shard

57.0389
55.9897
-1.04917
min

Median cumulative merge time across primary shard

57.0389
55.9897
-1.04917
min

Max cumulative merge time across primary shard

57.0389
55.9897
-1.04917
min

Cumulative merge throttle time of primary shards

6.91375
5.56918
-1.34457
min

Min cumulative merge throttle time across primary shard

6.91375
5.56918
-1.34457
min

Median cumulative merge throttle time across primary shard

6.91375
5.56918
-1.34457
min

Max cumulative merge throttle time across primary shard

6.91375
5.56918
-1.34457
min

Cumulative refresh time of primary shards

3.48515
3.47125
-0.0139
min

Cumulative refresh count of primary shards

89
92
3

Min cumulative refresh time across primary shard

3.48515
3.47125
-0.0139
min

Median cumulative refresh time across primary shard

3.48515
3.47125
-0.0139
min

Max cumulative refresh time across primary shard

3.48515
3.47125
-0.0139
min

Cumulative flush time of primary shards

3.23018
3.47077
0.24058
min

Cumulative flush count of primary shards

36
38
2

Min cumulative flush time across primary shard

3.23018
3.47077
0.24058
min

Median cumulative flush time across primary shard

3.23018
3.47077
0.24058
min

Max cumulative flush time across primary shard

3.23018
3.47077
0.24058
min

Total Young Gen GC time

51.866
50.881
-0.985
s

Total Young Gen GC count

4699
4171
-528

Total Old Gen GC time

0
0
0
s

Total Old Gen GC count

0
0
0

Store size

24.1888
24.1747
-0.01407
GB

Translog size

5.12227e-08
5.12227e-08
0
GB

Heap used for segments

0.0626526
0.059433
-0.00322
MB

Heap used for doc values

0.0142365
0.0107574
-0.00348
MB

Heap used for terms

0.0266418
0.0268555
0.00021
MB

Heap used for norms

0
0
0
MB

Heap used for points

0
0
0
MB

Heap used for stored fields

0.0217743
0.0218201
5e-05
MB

Segment count

20
20
0

Min Throughput
index
101371
101251
-120.297
docs/s

Mean Throughput
index
105690
103852
-1837.72
docs/s

Median Throughput
index
105155
103810
-1345.55
docs/s

Max Throughput
index
111784
107728
-4055.92
docs/s

50th percentile latency
index
578.89
572.464
-6.42686
ms

90th percentile latency
index
712.863
695.115
-17.7477
ms

99th percentile latency
index
4674.72
4868.49
193.772
ms

99.9th percentile latency
index
6688.37
7989.65
1301.28
ms

99.99th percentile latency
index
10130.2
9129.41
-1000.84
ms

100th percentile latency
index
10215.7
10804.1
588.395
ms

50th percentile service time
index
578.89
572.464
-6.42686
ms

90th percentile service time
index
712.863
695.115
-17.7477
ms

99th percentile service time
index
4674.72
4868.49
193.772
ms

99.9th percentile service time
index
6688.37
7989.65
1301.28
ms

99.99th percentile service time
index
10130.2
9129.41
-1000.84
ms

100th percentile service time
index
10215.7
10804.1
588.395
ms

error rate
index
0
0
0
%

Min Throughput
default
3.01917
3.01554
-0.00363
ops/s

Mean Throughput
default
3.03125
3.02527
-0.00599
ops/s

Median Throughput
default
3.02849
3.02304
-0.00545
ops/s

Max Throughput
default
3.05522
3.04456
-0.01065
ops/s

50th percentile latency
default
9.74398
11.0182
1.27423
ms

90th percentile latency
default
10.399
13.0243
2.62526
ms

99th percentile latency
default
10.6896
13.8311
3.1415
ms

100th percentile latency
default
10.7933
18.1735
7.38024
ms

50th percentile service time
default
8.39205
9.21252
0.82047
ms

90th percentile service time
default
8.51849
11.2065
2.68806
ms

99th percentile service time
default
8.84073
11.9656
3.12484
ms

100th percentile service time
default
8.84986
16.4717
7.62186
ms

error rate
default
0
0
0
%

Min Throughput
range
0.702312
0.70303
0.00072
ops/s

Mean Throughput
range
0.703796
0.704976
0.00118
ops/s

Median Throughput
range
0.703456
0.704527
0.00107
ops/s

Max Throughput
range
0.706837
0.708979
0.00214
ops/s

50th percentile latency
range
609.298
218.801
-390.498
ms

90th percentile latency
range
646.756
233.172
-413.584
ms

99th percentile latency
range
750.559
242.807
-507.752
ms

100th percentile latency
range
805.287
247.387
-557.9
ms

50th percentile service time
range
606.64
216.431
-390.209
ms

90th percentile service time
range
644.171
230.879
-413.292
ms

99th percentile service time
range
748.935
240.366
-508.568
ms

100th percentile service time
range
803.681
244.485
-559.195
ms

error rate
range
0
0
0
%

Min Throughput
distance_amount_agg
2.012
2.01206
6e-05
ops/s

Mean Throughput
distance_amount_agg
2.01974
2.01985
0.0001
ops/s

Median Throughput
distance_amount_agg
2.01794
2.01805
0.0001
ops/s

Max Throughput
distance_amount_agg
2.03549
2.03564
0.00015
ops/s

50th percentile latency
distance_amount_agg
6.03545
7.59386
1.55841
ms

90th percentile latency
distance_amount_agg
6.45989
8.2049
1.74501
ms

99th percentile latency
distance_amount_agg
6.66398
8.95673
2.29274
ms

100th percentile latency
distance_amount_agg
6.73026
9.09743
2.36717
ms

50th percentile service time
distance_amount_agg
4.16372
5.79158
1.62785
ms

90th percentile service time
distance_amount_agg
4.28412
6.16922
1.88511
ms

99th percentile service time
distance_amount_agg
4.48571
6.64509
2.15938
ms

100th percentile service time
distance_amount_agg
4.80145
6.83914
2.03769
ms

error rate
distance_amount_agg
0
0
0
%

Min Throughput
autohisto_agg
1.49575
1.49477
-0.00097
ops/s

Mean Throughput
autohisto_agg
1.49763
1.49709
-0.00054
ops/s

Median Throughput
autohisto_agg
1.49785
1.49735
-0.0005
ops/s

Max Throughput
autohisto_agg
1.49856
1.49821
-0.00035
ops/s

50th percentile latency
autohisto_agg
506.685
485.511
-21.1741
ms

90th percentile latency
autohisto_agg
568.946
503.067
-65.8784
ms

99th percentile latency
autohisto_agg
595.732
519.817
-75.9155
ms

100th percentile latency
autohisto_agg
596.373
521.69
-74.6826
ms

50th percentile service time
autohisto_agg
504.978
483.595
-21.3835
ms

90th percentile service time
autohisto_agg
567.723
501.678
-66.0448
ms

99th percentile service time
autohisto_agg
594.446
518.441
-76.0042
ms

100th percentile service time
autohisto_agg
594.787
520.713
-74.0745
ms

error rate
autohisto_agg
0
0
0
%

Min Throughput
date_histogram_agg
1.50023
1.50038
0.00015
ops/s

Mean Throughput
date_histogram_agg
1.50032
1.50053
0.00021
ops/s

Median Throughput
date_histogram_agg
1.50031
1.50051
0.0002
ops/s

Max Throughput
date_histogram_agg
1.50046
1.50077
0.0003
ops/s

50th percentile latency
date_histogram_agg
545.918
498.707
-47.2106
ms

90th percentile latency
date_histogram_agg
574.626
516.146
-58.4804
ms

99th percentile latency
date_histogram_agg
590.896
530.055
-60.8414
ms

100th percentile latency
date_histogram_agg
597.37
553.235
-44.1355
ms

50th percentile service time
date_histogram_agg
544.724
496.971
-47.7531
ms

90th percentile service time
date_histogram_agg
572.973
514.892
-58.0809
ms

99th percentile service time
date_histogram_agg
589.08
528.245
-60.8349
ms

100th percentile service time
date_histogram_agg
595.714
551.565
-44.1496
ms

error rate
date_histogram_agg
0
0
0
%

Conclusions: very noticeable reduction of latencies for range queries
search.allow_concurrent_segment_search: false

Metric
Task
Baseline
Contender
Diff
Unit

Cumulative indexing time of primary shards

158.785
159.132
0.34725
min

Min cumulative indexing time across primary shard

158.785
159.132
0.34725
min

Median cumulative indexing time across primary shard

158.785
159.132
0.34725
min

Max cumulative indexing time across primary shard

158.785
159.132
0.34725
min

Cumulative indexing throttle time of primary shards

0
0
0
min

Min cumulative indexing throttle time across primary shard

0
0
0
min

Median cumulative indexing throttle time across primary shard

0
0
0
min

Max cumulative indexing throttle time across primary shard

0
0
0
min

Cumulative merge time of primary shards

57.0389
54.4416
-2.59727
min

Cumulative merge count of primary shards

99
101
2

Min cumulative merge time across primary shard

57.0389
54.4416
-2.59727
min

Median cumulative merge time across primary shard

57.0389
54.4416
-2.59727
min

Max cumulative merge time across primary shard

57.0389
54.4416
-2.59727
min

Cumulative merge throttle time of primary shards

6.91375
5.42017
-1.49358
min

Min cumulative merge throttle time across primary shard

6.91375
5.42017
-1.49358
min

Median cumulative merge throttle time across primary shard

6.91375
5.42017
-1.49358
min

Max cumulative merge throttle time across primary shard

6.91375
5.42017
-1.49358
min

Cumulative refresh time of primary shards

3.48515
3.43525
-0.0499
min

Cumulative refresh count of primary shards

89
90
1

Min cumulative refresh time across primary shard

3.48515
3.43525
-0.0499
min

Median cumulative refresh time across primary shard

3.48515
3.43525
-0.0499
min

Max cumulative refresh time across primary shard

3.48515
3.43525
-0.0499
min

Cumulative flush time of primary shards

3.23018
3.38402
0.15383
min

Cumulative flush count of primary shards

36
36
0

Min cumulative flush time across primary shard

3.23018
3.38402
0.15383
min

Median cumulative flush time across primary shard

3.23018
3.38402
0.15383
min

Max cumulative flush time across primary shard

3.23018
3.38402
0.15383
min

Total Young Gen GC time

51.866
51.452
-0.414
s

Total Young Gen GC count

4699
3722
-977

Total Old Gen GC time

0
0
0
s

Total Old Gen GC count

0
0
0

Store size

24.1888
24.1291
-0.05967
GB

Translog size

5.12227e-08
5.12227e-08
0
GB

Heap used for segments

0.0626526
0.0592346
-0.00342
MB

Heap used for doc values

0.0142365
0.0108032
-0.00343
MB

Heap used for terms

0.0266418
0.0266418
0
MB

Heap used for norms

0
0
0
MB

Heap used for points

0
0
0
MB

Heap used for stored fields

0.0217743
0.0217896
2e-05
MB

Segment count

20
20
0

Min Throughput
index
101371
99943.4
-1428.05
docs/s

Mean Throughput
index
105690
102843
-2847.31
docs/s

Median Throughput
index
105155
102218
-2937.06
docs/s

Max Throughput
index
111784
107899
-3884.51
docs/s

50th percentile latency
index
578.89
570.709
-8.18176
ms

90th percentile latency
index
712.863
703.71
-9.15365
ms

99th percentile latency
index
4674.72
4948.41
273.688
ms

99.9th percentile latency
index
6688.37
8050.86
1362.49
ms

99.99th percentile latency
index
10130.2
11069.9
939.673
ms

100th percentile latency
index
10215.7
13915.3
3699.6
ms

50th percentile service time
index
578.89
570.709
-8.18176
ms

90th percentile service time
index
712.863
703.71
-9.15365
ms

99th percentile service time
index
4674.72
4948.41
273.688
ms

99.9th percentile service time
index
6688.37
8050.86
1362.49
ms

99.99th percentile service time
index
10130.2
11069.9
939.673
ms

100th percentile service time
index
10215.7
13915.3
3699.6
ms

error rate
index
0
0
0
%

Min Throughput
default
3.01917
3.01532
-0.00386
ops/s

Mean Throughput
default
3.03125
3.02494
-0.00632
ops/s

Median Throughput
default
3.02849
3.02271
-0.00578
ops/s

Max Throughput
default
3.05522
3.04395
-0.01127
ops/s

50th percentile latency
default
9.74398
10.7946
1.05065
ms

90th percentile latency
default
10.399
12.1291
1.73011
ms

99th percentile latency
default
10.6896
14.2935
3.60394
ms

100th percentile latency
default
10.7933
18.9675
8.17426
ms

50th percentile service time
default
8.39205
9.14095
0.7489
ms

90th percentile service time
default
8.51849
10.5664
2.04796
ms

99th percentile service time
default
8.84073
12.6854
3.8447
ms

100th percentile service time
default
8.84986
17.8632
9.01331
ms

error rate
default
0
0
0
%

Min Throughput
range
0.702312
0.701887
-0.00042
ops/s

Mean Throughput
range
0.703796
0.703102
-0.00069
ops/s

Median Throughput
range
0.703456
0.702826
-0.00063
ops/s

Max Throughput
range
0.706837
0.705585
-0.00125
ops/s

50th percentile latency
range
609.298
606.512
-2.78676
ms

90th percentile latency
range
646.756
646.448
-0.3076
ms

99th percentile latency
range
750.559
663.352
-87.2073
ms

100th percentile latency
range
805.287
692.011
-113.276
ms

50th percentile service time
range
606.64
604.751
-1.88868
ms

90th percentile service time
range
644.171
642.89
-1.28118
ms

99th percentile service time
range
748.935
659.661
-89.2739
ms

100th percentile service time
range
803.681
690.599
-113.081
ms

error rate
range
0
0
0
%

Min Throughput
distance_amount_agg
2.012
2.0119
-0.0001
ops/s

Mean Throughput
distance_amount_agg
2.01974
2.01958
-0.00016
ops/s

Median Throughput
distance_amount_agg
2.01794
2.01781
-0.00014
ops/s

Max Throughput
distance_amount_agg
2.03549
2.03516
-0.00033
ops/s

50th percentile latency
distance_amount_agg
6.03545
7.65055
1.6151
ms

90th percentile latency
distance_amount_agg
6.45989
8.4301
1.97022
ms

99th percentile latency
distance_amount_agg
6.66398
8.76383
2.09985
ms

100th percentile latency
distance_amount_agg
6.73026
9.0628
2.33254
ms

50th percentile service time
distance_amount_agg
4.16372
5.83697
1.67324
ms

90th percentile service time
distance_amount_agg
4.28412
6.30701
2.0229
ms

99th percentile service time
distance_amount_agg
4.48571
6.70348
2.21777
ms

100th percentile service time
distance_amount_agg
4.80145
6.90733
2.10588
ms

error rate
distance_amount_agg
0
0
0
%

Min Throughput
autohisto_agg
1.49575
1.49177
-0.00398
ops/s

Mean Throughput
autohisto_agg
1.49763
1.49543
-0.00221
ops/s

Median Throughput
autohisto_agg
1.49785
1.49585
-0.002
ops/s

Max Throughput
autohisto_agg
1.49856
1.49721
-0.00135
ops/s

50th percentile latency
autohisto_agg
506.685
499.525
-7.16041
ms

90th percentile latency
autohisto_agg
568.946
519.422
-49.5237
ms

99th percentile latency
autohisto_agg
595.732
530.96
-64.7724
ms

100th percentile latency
autohisto_agg
596.373
538.004
-58.3689
ms

50th percentile service time
autohisto_agg
504.978
497.424
-7.55413
ms

90th percentile service time
autohisto_agg
567.723
518.127
-49.5965
ms

99th percentile service time
autohisto_agg
594.446
529.934
-64.5113
ms

100th percentile service time
autohisto_agg
594.787
536.229
-58.5576
ms

error rate
autohisto_agg
0
0
0
%

Min Throughput
date_histogram_agg
1.50023
1.50034
0.00011
ops/s

Mean Throughput
date_histogram_agg
1.50032
1.50047
0.00015
ops/s

Median Throughput
date_histogram_agg
1.50031
1.50046
0.00015
ops/s

Max Throughput
date_histogram_agg
1.50046
1.50069
0.00022
ops/s

50th percentile latency
date_histogram_agg
545.918
508.833
-37.0851
ms

90th percentile latency
date_histogram_agg
574.626
530.466
-44.1606
ms

99th percentile latency
date_histogram_agg
590.896
541.335
-49.5615
ms

100th percentile latency
date_histogram_agg
597.37
554.333
-43.0373
ms

50th percentile service time
date_histogram_agg
544.724
507.088
-37.6356
ms

90th percentile service time
date_histogram_agg
572.973
528.838
-44.1354
ms

99th percentile service time
date_histogram_agg
589.08
540.033
-49.0469
ms

100th percentile service time
date_histogram_agg
595.714
553.075
-42.6392
ms

error rate
date_histogram_agg
0
0
0
%

Conclusions: slight variance across search-related latencies

@reta Thanks for your effort to put up this PR. Glad to see the improvement into the benchmark results. We will look into this soon.
Question: Does concurrent search on segment level conflict with the fact to get an accurate query results, segments should be queried in sequence.

Thanks @anasalkouz, regarding your question

Question: Does concurrent search on segment level conflict with the fact to get an accurate query results, segments should be queried in sequence.

AFAIK, it should not conflict, the outcomes of the each concurrent operation are merged together, essentially producing the same results. I have checked the queries on same data sets using main build, and this PR build with concurrent segment search on/off: the same query returned identical hits and scores The only divergence in the behavior is early forced termination and query timeout.

Thats great. One topic worth discussion and can be considered as potential future work. We will loose the benefit of this feature when Lucene merge small segments to a single large segment. which might shows as shark tooth on search query performance. In the future, I would like to figure out the possibility to do concurrent queries for the single segment (Large Segments).

@reta , Could you please create an issue for the aggregation part? to track in the future.

@anasalkouz here we are, https://github.com/opensearch-project/OpenSearch/issues/1610, thank you

Thats great. One topic worth discussion and can be considered as potential future work. We will loose the benefit of this feature when Lucene merge small segments to a single large segment. which might shows as shark tooth on search query performance. In the future, I would like to figure out the possibility to do concurrent queries for the single segment (Large Segments).

That is correct, Lucene falls back to sequential search if there is only 1 segment, the future improvement you are suggesting sounds reasonable.

@nknize thank you for looking.

Nice! I'll look into this a little deeper but at first glance I think it should split into a couple smaller PRs. Perhaps 1. Multi-Collector query context support, 2. profiling support, 3. early termination and timeout support. Let's track through a meta issue

I think I could extract changes needed for profiler support into separate pull request, with that the separation to "query context support" and "query context support with profiler" would be not needed (very likely), what do you think?
The early termination is supported, and it will be very difficult to extract this part just on it own (this is a chain of collector managers essentially). But the "forced"  early termination (abruptly terminate search) and timeout returning partial results are not implemented at all right now.

Also, since this is experimental I'd like to look at adding it as an experimental feature (module) through the sandbox mechanism. Even if it means exposing some core server methods we should look at adding this through sandbox via SearchPlugin (and maybe ClusterPlugin) interfaces.

Yeah, sure, I will look at it, thanks for pointing it out

What shall we do with this?

@andrross Let's do a CR of this? Sorry @reta this feel through the cracks.

@andrross Let's do a CR of this? Sorry @reta this feel through the cracks.

@dblock that's fine, it would probably make sense to wait for https://github.com/opensearch-project/OpenSearch/pull/1931 to get merged, it will reshape this pull request significantly

Yeah, I've been waiting for #1931 because it will change things here. I think we should be able to get that merged pretty soon. I'll take a look as soon as this one is rebased with those changes.

We could merge this for 3.0, wdyt @nknize?

@andrross another CR please?

Test failure #2442
start gradle check

start gradle check

We could merge this for 3.0, wdyt @nknize?

Since we're targeting sandbox we can merge in 2.x and don't have to wait for 3.0.

We could merge this for 3.0, wdyt @nknize?

Since we're targeting sandbox we can merge in 2.x and don't have to wait for 3.0.
Also, the 2.0 branch is cut (as well as the 2.x branch). So when this is ready we can merge to main and backport to 2.x without having to wait for the version bump.

I merged and marked to backport 2.x.

:tada:

@dblock @nknize @andrross thanks guys! I will create followup improvement issue to cover aggregations and early termination

Should we create a meta issue to listing all the things needing to be done to promote it out of the sandbox?

Should we create a meta issue to listing all the things needing to be done to promote it out of the sandbox?

:+1: , I will do that shortly

https://github.com/opensearch-project/OpenSearch/issues/2587


[Accessibility] Terminal a11y buffer is not automatically updated

CC @meganrogge
Reproducible Steps

Load NVDA.
Open terminal.
Change directory to anywhere having lots of files and sub-directories.
Type ls -1
As soon as typing ls -1, press Shift+Tab to focus in the terminal a11y buffer.
Press Ctrl+End to go to the end of the buffer and see if it reflects the last file name of the ls command: It is not actualy.
Press Tab to terminal input and Shift+Tab to come back to a11y output buffer: This time the last line of the buffer reflects the the last item on the screen.

Expected Behavior
The terminal a11y buffer needs to get updated automatically without manual focus refresh.
System Info
Version: 1.75.0-insider (system setup)
Commit: 07d6f5b35fac559504dec65e3073d796b2d25a55
Date: 2023-01-26T05:23:05.014Z
Electron: 19.1.9
Chromium: 102.0.5005.194
Node.js: 16.14.2
V8: <IP_ADDRESS>-electron.0
OS: Windows_NT x64 10.0.22621
Sandboxed: Yes

Performance is a concern but since it's an edge case I think it makes sense to listen for buffer changes and refresh it. We will probably need to append and not replace though

Another problem encountered is that if I type some text in the accessibility buffer, it is no longer updated. It seems that the problem  only happens if the text is typed very close to the last line of the buffer. I tried to reproduce the problem by typing a text at the beginning of the buffer but I couldn't.
To reproduce try the following steps:

Run some commands in the VSCode terminal.
Press the shift+tab keys and explore the accessibility buffer using the arrow keys. Note that the executed commands result are present as expected.
Move the cursor to the last line of the buffer and type any garbage.
Press tab and run more commands in the terminal.
Press the shift+tab keys and explore the accessibility buffer.

In my environment the result of the commands executed in step 4 are not present in the accessibility buffer, only the commands executed in step 1 are shown.

Please treat this as a significant bug and not an "edge case".  It's currently impossible to review the log of any running task or command until the task completes or the shell prompt is returned.  That eliminates access to many common things like:

man pages
python or node in interactive mode
logs of significant builds or test suites
development watch tasks

@steverep this is a plan item for the March iteration IE we plan to tackle it soon

Repro:

Run while sleep 1; do echo "Hi"; done
Focus the accessible buffer, focus the terminal, focus the accessible buffer again
🐛 it doesn't update

This still repros for me on Windows (without Narrator enabled):

Enable "editor.accessibilitySupport"
Run ls
Shift+Tab while step 2 is running
:bug: accessible buffer contents are incomplete

If I click away from the VS Code window and then click back into it, the accessible buffer updates

this is false when I'm typing in the editor. this should instead check if the accessible buffer is visible, which it did originally
https://github.com/microsoft/vscode/blob/f584ecc7e63975401db1c0a0076f023f559bba3e/src/vs/workbench/contrib/terminalContrib/accessibility/browser/terminalAccessibleBuffer.ts#L119

This still repros for me with the latest insiders.

@joyceerhl a gif would be helpful
https://user-images.githubusercontent.com/29464607/227318087-eb807fbb-a566-4081-8d35-fa5b4a7f0542.mov

https://user-images.githubusercontent.com/30305945/227319300-bc01d522-66e2-4916-9583-7dc6749b7c50.mp4
(in the recording above, at about the 6 second mark I click out of the terminal and back in, and that apparently forces an update)

can you repro if you start with a new terminal, don't run clear? i'm trying exactly what you show and not seeing an issue - in the latest insider's (as of 2 hours ago)

I can repro but it's quite difficult, there's probably some race condition where output occurs before the listener is registered. I suggest we action this as part of the _getContent testing/refactoring

Considering this verified because it works in most cases and we will have a TPI next iteration to test the improvements we plan to make


NR Dependency Protobuf 3.18.0 Not Compatible With Python 2.7

May not be the right place for this, but hopefully it helps someone with the same issue.
Description
Traceback (most recent call last):
  File "xxx/utils.py", line 4, in <module>
    import newrelic.agent
  File "/xxx/lib/python2.7/site-packages/newrelic/agent.py", line 15, in <module>
    from newrelic.config import (
  File "/xxx/lib/python2.7/site-packages/newrelic/config.py", line 30, in <module>
    from newrelic.core.config import (Settings, apply_config_setting,
  File "xxx/lib/python2.7/site-packages/newrelic/core/config.py", line 43, in <module>
    from newrelic.core.infinite_tracing_pb2 import Span as _  # NOQA
  File "xxx/lib/python2.7/site-packages/newrelic/core/infinite_tracing_pb2.py", line 16, in <module>
    from google.protobuf import descriptor as _descriptor
  File "xxx/lib/python2.7/site-packages/google/protobuf/descriptor.py", line 113
    class DescriptorBase(metaclass=DescriptorMetaclass):
                                  ^
SyntaxError: invalid syntax

See: https://issueexplorer.com/issue/GoogleCloudPlatform/python-docs-samples/6703
Steps to Reproduce
In a python 2.7 env:
pip install newrelic==<IP_ADDRESS>
$ python
>>> import newrelic.agent
Your Environment

Python 2.7
newrelic==<IP_ADDRESS>

Additional context
Rolling back to protobuf~=3.17.3 fixes this.

Yes thank you, that's the correct issue. I was looking for the link earlier and didn't find it, was looking at gRPC not protobuf.


feature request: allow select+copy to clipboard

It'd be handy to be able to select text+copy to the clipboard from the log panels. It's much more cumbersome to find the log txt first just to share a few rows with others.

+1

This feature is planned.

I started testing this feature. If you get v1.3.15, it's added to the isolated ping window (hover over a ping window to get the mini title bar, which has the open isolated view button).  One problem I'm having is if you are highlighting text from right to left, your highlight gets reset every time the ping window updates. I don't think there's any way to fix that. Highlighting text from left to right works fine. The other problem I'm having is the output auto-scrolls to the bottom on each update, so it's difficult to highlight text. This should be fixable. And this is only when active. If it's stopped, highlighting works as expected.

Thank you ! I will try it

Added here 59537f481a96a9e26fdd0b3f15137d06ee9c0f6e and here 7fbeae6f36560b91eb6d4bbeeed5e79e9cc43e24. Available in release 1.3.16.


fix: no version check between browser extension and native app

Added an additional version check: Native app's major version must not be lower than the extension's major version
When the extension returns warnings in the status response, they will now be logged to console by the library
Renamed "Versions" interface name to "Status" as it can also contain the warnings array
Updated the documentation
Fixed an issue where a pending action was rejected or resolved before it's removed from the pending action queue
Updated TypeScript version to version 3.9.10
Fixed a setTimeout return type conflict between Node.js and browser version's introduced by the TypeScript update
Introduced jest and ts-jest as dev-dependency
Tests are executed during NPM pack's prepack cycle
Added tests for version checking

Inadvertantly checks that versions match in package.json and src/config.ts

Added tests for warning logging
Version bump to 1.1.0 - minor version increased as warning logging is a new feature.

Modified the original commit:

In addition to the status() function, authenticate() and sign() functions now also send the library version to the extension
Removed warnings from the Status class
Renamed "Status" class back to the origina name "Versions"
Fixed an additional type related linting errors
Removed some unnecessary logging related to "ack" messages
Handling "web-eid:warning" event, so warnings can be messaged for authenticate and sign commands without waiting for their promise completion
Warning messages are saved internally and logged to browser console only once per page load
Tests added/updated
Reverted documentation change as the warnings are no longer related to the JS API


13 clean up simulatecombat 2

For posterity: This is a copy of another PR that David approved. I recreated it because I branched off a branch other than main and didn't want the commits from that other branch to appear in the PR


Add from_reader / from_slice / from_value function.

Add from_slice and from_reader and ~from_value~ function  to be compatible with serde_json,  serde_yaml.

I'm curious what your interest in these is.
Is there a case where this is more convenient?
My main concern is that someone will see these "lower level" functions and assume they are more performant when they aren't.

This is simply to provide compatibility with json and yaml implementations and reduce developer thinking time.
I'm making conversion tools for data validation, and I noticed that toml crate were not operation providing consistently.
I also aim to provide an easy way for developers to change data formats if they want to change it in their projects.

To me, that motivation feels very weak.  Yes, this provides consistency but we aren't being consistent in their implementation.
For serde_json, they either are independent implementations or the higher-level ones are implemented in terms of the lower level ones.
For toml, we are implementing the lower level ones in terms of the higher level.  As I said, I'm worried people will "drop down" to this API expecting better performance when it makes no difference.

Let me get my perception straight in order to decide if I should continue with this PR.
"toml is implemented the lower level ones in terms of the higher level" means that ser/de is using toml_edit internally in its implementation.
I recognize that the problem is that finer tuning is forgotten in the serialization or deserialization operations between toml strings and Rust structures.
When derializing, I believe that the string to structure can be uniquely determined.
On the other hand, when serializing, the output pattern cannot be uniquely determined and may not reflect user preferences
(e.g., how to represent arrays of tables).
And if user want to output customizing format, he needs to implemented Serialize for the Rust struct.
Is my rough understanding correct?
In the case of the above perception, I think users will implement their own Serialize according to their own convenience.
If the problem is not only with the format, you recognize the performance of the conversion process, I don't think many users understand the implementation of crates that well.
Performance tuning documentation will be needed.

Not quite.
I was referring to the fact that for serde_json, a from_str (higher level) is convenience wrapper around from_slice (lower level).  Contrast that with toml where from_slice (lower level) is a convenience wrapper around from_str (higher level).  People optimizing their code might assume working directly with a File / from_reader would be faster than a File and from_slice which would be faster than File and from_str when in fact the performance would all be the same (or worse).

When deserializing toml from a file, I think you end up needing to do the same thing I implemented with from_reader.
I suspect that users who use from_reader do not use it to improve performance, but simply to serialize from a file.
As for from_slice, it would certainly be better not to implement it, since toml requires utf-8 encoding in specification, so from_str is sufficient.

When deserializing toml from a file, I think you end up needing to do the same thing I implemented with from_reader.

I think my own biases in how I write code are showing through here and I would like help in better understanding other view points, not just for this PR but for PRs generally.
When/why are you using File over alternatives?
For myself, I pretty much always use std::fs::read_to_string and std::fs::write.  I think last week is the first time in years that I used File and it was because I needed to append.

It seems natural for toml to handle File when reading from a configuration file.
Certainly relaying a String to a structure is the actual process (as is the implementation of from_reader), but I felt that going through a string was not the intent. As you say, this seems to be a bias.

would it be possible to add from_reader without reading into a string first? imo the benefit of from_reader is to avoid the upfront cost of having the entire input in memory.

@yassun4dev

It seems natural for toml to handle File when reading from a configuration file.

Could you expand on this?  All of my own config handling in all of my applications call std::fs::read or std::fs::read_to_string.  In general, I find the File API a hassle to use and only do so when needed.  I'm not seeing why the use case of config files has an influence on what API is used.
@nissaofthesea

would it be possible to add from_reader without reading into a string first? imo the benefit of from_reader is to avoid the upfront cost of having the entire input in memory.

Doing so is out of scope at this time and I think you are highlighting why I'm hesitant to move forward with this.

@epage
I think I have come to understand why there is reluctance to introduce it.
The toml crate is focused on its function as a parse and isolates the io issue. I did not understand this when I raised the issue, but it is an excellent approach.

It seems natural for toml to handle File when reading from a configuration file.

Could you expand on this?

To explain a little, The File API is a hassle, but that process is hidden in the crate. I think this code expresses the purpose of mapping from toml format files to objects.
let data: MyStruct = serde_toml::from_reader(std::fs::File::open("sample.toml")?)?;

However, I would like to close this issue. My question has been resolved.
Thakns. I appreciate your explanation.

Any further discussion on this should be on #326 until this has been approved.


Always need to specify overlap information

Hi,
This is more a question than an issue really. Do I need to specify overlap information for every possible combination? I'm working with a lot of data, but not necessary a lot of overlap. Is there any way that I don't need to specify the overlap for every possible combination, only for the ones that do overlap? When I don't give information about the overlap of a certain combination, I get the error message "Need overlap information for set {"x":10000000000,"y":10000000000,"size":1,"radius":0.5641895835477563}".
Thanks in advance

Right now you need to specify overlaps for every pairwise combination - but don't combinations of 3+ sets.
I'm going to fix the error message so its more descriptive, and remove this constraint (missing values will be assumed to of size 0) - hopefully this weekend. Also see https://github.com/benfred/venn.js/issues/42

@benfred Any update on this issue?

Well, I was creating sets greater than 3 combinations, so I fixed that and now I’m using just 3 entries. I now specify information for all possible combinations of these entries, so I just worked around it.

On 22 Sep 2015, at 01:18, John Babak<EMAIL_ADDRESS>wrote:
@benfred https://github.com/benfred Any update on this issue?
—
Reply to this email directly or view it on GitHub https://github.com/benfred/venn.js/issues/42#issuecomment-142135592.

@sebastianstoelen I worked this around, too, by adding all the pairs missing from my input data with the size of 0. I wanted to ask the project maintainer if he's going to fix the issue or have already done this.

@sompylasar @sebastianstoelen this should be fixed now - sorry about the delay

@benfred great, many thanks!


Update _index.en.md

Updating docker container that gets pulled upon first invocation.
To line up with https://cicd.serverlessworkshops.io/sam/init.html
Choose nodejs12.x for runtime
Description of changes:
Updating docs to reference fact that nodejs12 docker container is pulled
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Great catch @carltondickson! Thank you! Do you mind doing the contribution to the dev branch please? This is as per the Contribution guidelines we are trying to follow the process of no direct commits to master, all new development goes to dev and we merge to master when we do the next release.

@ferdingler This ok?

@ferdingler This ok?

Perfect. Thank you, I will merge now and it will be included in the next release: March 24th, 2020.


Hospitalization length of stay algorithm

The VDEC team, including myself, were examining some of records generated via synthea; specifically the hospitalization data, for which we wondered what algorithm you use.
We defined hospitalization as:

Hospital Admission
OR

Admission to XYZ Department, EXCEPT

Emergency room admissions
Non-urgent orthopedic admission
Obstetric emergency admission

Current Canadian health statistics note that average length of stay of a patient is typically about 7 days:
https://www.statista.com/statistics/497128/canada-average-hospital-stay-length/
I ran synthea and generated 10,000 entries, checked for hospitalizations (514 records), this yields:
  Length of |
Stay (days) |      Freq.     Percent        Cum.
------------+-----------------------------------
          0 |         73       14.20       14.20
          1 |        413       80.35       94.55
          2 |         27        5.25       99.81
          5 |          1        0.19      100.00
------------+-----------------------------------
      Total |        514      100.00

Where we define length of stay as stop_date - start_date
So roughly 80% of the data points from synthea show a length of stay of 1 day, and 95% <= 1 day.
The Canadian medical data (and maybe the US as well) seems to point to a different distribution for the length of stay. With this in mind, could you tell us more about the algorithms and assumptions have you have built into the distribution of length of stay in synthea?

How it works
Encounters have a minimum amount of time required:

Type
Minimum Time

Emergency
1 hour

Inpatient
1 day

Other
15 minutes

If a module tries to end an encounter early, the system will use the minimum times list above. Otherwise, the system will end the encounter when the module tries to close it.
After that, the system will add additional time to the encounter for each activity (e.g. procedures) that occurred during an encounter so long as the module defined a duration for an activity.
So, for example, if an Encounter for Dialysis (and nothing else) occurs:
Start with a minimum of 15 minutes for the Encounter and then add 3 hours for the Dialysis, resulting in 3 hours 15 minutes.
There are few inpatient procedures being modeled now. If a specific disease module doesn't define delays or introduce activities that require time, the system ends the encounter after a day.
Thoughts
Obviously, this can be improved. What are your suggestions?
Perhaps we can get some statistics for common inpatient encounter causes, and lengths of stay for each type, and update the modules to reflect that? We're happy to have help on this.

The algorithm seems reasonable for emergency and physician visits (adding some random wait times/lengths could be a future idea) as it ends (either send home or admitted to hospital) once all procedures have finished.
Inpatient hospitalization is a bit different (more complex) though, as discharge is based on the condition of the patient as judged by the physician and there can be significant time between procedures (e.g., needing a blood sample every 12 hours). Later procedures can also depend on whether the patient is discharged or not (e.g. daily procedures while in hospital). Not knowing exactly how you program is set up, I cannot say what the best way of action is. One option might be for each day (after the first (or a later) procedure) compute some status score or discharge probability based on morbidity etc. @DrGo any other thoughts?

How do you people with more complex health symptoms? Would the hospitalization for one (e.g., stroke) be influenced by other diseases (e.g., diabetes)?

Currently, more complex health comorbidities are handled independently.

My recently pulled modifications to the congestive_heart_failure module have a crude implementation of a more realistic inpatient encounter that you can look at. I model the encounter as a daily loop with a certain set of daily activities. At the end of the day I branch either into another day of inpatient stay, or into the discharge from the hospital. The likelihood of a discharge is increased sharply as the typical ALOS for heart failure is reached (5 days).

Related to #1168, #1169, #1170, #1171


Add compat, update packages

This PR adds the compat entries from the Compat PRs, and updates the packages.
Closes #25, #26, #27, #28, #29, #30, #32

bors r+

bors r+


dif_kmac smoke tests use dif_kmac_toggle_t instead of bool for entropy_fast_process

Noticed this while working on #7757. A grep for entropy_fast_process should show the lines that should be fixed.

This has been fixed (at some point).


Update to v1.22.14

Signed-off-by: matttrach<EMAIL_ADDRESS>

Drone CI: https://drone-pr.k3s.io/k3s-io/k3s/4724
failed - "no space left on device"

Codecov Report
Base: 9.58% // Head: 9.58% // No change to project coverage :thumbsup:

Coverage data is based on head (0456d5c) compared to base (42190ac).
Patch has no changes to coverable lines.

Additional details and impacted files
@@             Coverage Diff              @@
##           release-1.22   #6145   +/-   ##
============================================
  Coverage          9.58%   9.58%
============================================
  Files               136     136
  Lines              9936    9936
============================================
  Hits                952     952
  Misses             8786    8786
  Partials            198     198

Flag
Coverage Δ

unittests
9.58% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

are we good to merge this then?

I think this should be merged now.


Update translation.json

Small language style changes

Small update on https://github.com/pixelfed/project-website/pull/52


Pretrain on COCO and Cityscapes

Hi, thank you for your great work!
I downloaded your pre-train on COCO and CityScape.
It seems the checkpoint files of two versions on Cityscapes dataset 1024,2048 and 512,1024  are the same files.
Could you check the link and verify it?
Thank you

After double-checking,
I found that we need to change the size in the config file.
For example, change the  MIN_SIZE_TEST and MAX_SIZE_TEST of the input to 1024, 2048.
The checkpoint is the same in both cases.


fix: deep links on ios

Fixes #744
Will upload it to the bucket

For the record, the following is the real previous version. The file hasnt been modified since december 2022:
{
    "applinks": {
        "apps": [],
        "details": [
            {
                "appID": "KJ6LCYQ3A8.tv.brunstad.app.v4",
                "paths": [
                    "NOT /tvlogin/*",
                    "/*"
                ]
            },
            {
                "appID": "KJ6LCYQ3A8.tv.brunstad.app",
                "paths": [
                    "NOT /tvlogin/*",
                    "/*"
                ]
            }
        ]
    }
}


Deletion of history events on workflow completion

We mark the workflow execution row with a TTL in executions table on completion.  This takes care of workflow execution entry in executions table but we still need leak space in the events table as we don't cleanup the history associated with that execution.
We could use the timer queue processor for this purpose and queue up a timer task to delete the execution history after retention period.

PR #255 fixes this.


[RFE] reload button in all views

Current situation
list and detail views seem static, i'd have to reload the whole interface to see updates.
Impact
QOL update
Ideal future situation
little button to click. clicking reload currently viewed resources
Implementation options
?
Additional information

The list and details views do update automatically. If they are not updating for you, we should look into it.
As a test, you can edit a resource to add annotations and see that it reloads. It should show the new annotations whether you edited it from Headlamp or from kubectl.


drafts: Fix cant delete on android bug.

Fixes #22512
This fixes the bug that a user was not able to delete drafts in mobile or narrow view.
The problem was that the message-content box overlapped with the drafts-controls.

I posted one comment, but it'd also be super helpful if the commit message explained the reasoning behind this change -- what was going wrong and why these specific changes fix it.

@timabbott i pushed this again with a better commit message, i also explained why i chose this specific padding above, could you review this again?

OK, that explanation was quite helpful; I ended up rewriting this with a different approach that uses float: right to have the CSS more obviously do what it's clearly intended to do. It'd be great if you can do some added testing and comment in #frontend if there are further issues to investigate.
Thanks @juliaBichler01!


Plan integration is broken

Relevant integration (if any):

Plan integration
🐛 Describe the bug
time tracking button doesn't show up
Steps to reproduce

Steps to reproduce the behaviour:

Go to plan.toggl.com
click "new task"
observe that the button does not render

Other details or context

Looks like the task form has been updated so our selectors will have to be updated as well

case

@niyiAO Can you assign it to me please?


[LUIS] Delete multiple suggested utterances with LUIS api

Hi Team,
I want to delete multiple suggested utterances from LUIS api. I tried with delete request "https://westus.api.cognitive.microsoft.com/luis/api/v2.0/apps/{appId}/versions/{versionId}/suggest" and below are my findings,

Above request works well for deleting single utterances
Dn't know how to pass multiple utterances to above request.
Tried with passing data in body like as "Test","Test1" but getting badArgument response.
{  "error": {      "code": "BadArgument",      "message": "Cannot find an unlabeled query with the given text."  } }

Unfortunately we currently don't support deleting multiple utterances through Programmatic API. The only solution is through dashboard.


[TASK] Use merge sort for nested types when cudf supports it

When https://github.com/rapidsai/cudf/issues/8050 is done we should remove all of the work that was done in https://github.com/NVIDIA/spark-rapids/pull/2251 to work around the lack of cudf support.

Reopen the issue #8050 since cudf::merge is still missing the array and map type support.


MovR Documentation

We need to create a documentation section that introduces the MovR application and dataset, and provides instructions for setting up the database and tables, to run the application.
This page will include the following:

A general overview of the MovR application and dataset.
Instructions on using cockroach demo and cockroach workload for MovR.
Instructions on manual setup (perhaps first on a simple local cluster, and then on a larger, distributed GCP cluster).

Originally posted by @ericharmeling in https://github.com/cockroachdb/docs/issues/2849#issuecomment-509316569
See also https://github.com/cockroachdb/movr/issues/90 and https://github.com/cockroachdb/docs/issues/5062

See #5075


Quill permaplays behave strangely on IE 11:

You can get stuck on a knot or stuck in a tight loop. Choices aren't processing correctly.
It's pretty consistent, I got it from a couple different machines. Sometimes opening up the dev console fixes, but I was able to repro with the dev console open on one machine.
See http://jeejah.xyz/quill/play/9vzhcwmdq9 as an example.
Also check the regular Quill page, not just the permaplay page.

Could not repro on my laptop.

The problem is caused by IE caching AJAX responses (yay!).  This can be fixed with $.ajaxSetup({ cache: false }); once I get Quill updated to Core 1.1.

This is fixed in development and will presumably make its way to production at some point.


wip: Add LCMScheduler

What type of PR is this? (check all applicable)

[ ] Refactor
[ ] Feature
[ ] Bug Fix
[ ] Optimization
[ ] Documentation Update
[ ] Community Node Submission

Have you discussed this change with the InvokeAI team?

[ ] Yes
[ ] No, because:

Have you updated all relevant documentation?

[ ] Yes
[ ] No

Description
DOES NOT WORK -- Need to fix the incorrect shapes and then we probably need to isolate the type for LCM Models so we can control the schedulers for them. But the Scheduler is implemented.
Related Tickets & Documents

Related Issue #
Closes #

QA Instructions, Screenshots, Recordings

Added/updated tests?

[ ] Yes
[ ] No : please replace this line with details on why tests
have not been included

[optional] Are there any post deployment tasks we need to perform?

I approved before I saw the note regarding the need to isolate the scheduler. I think this is the first time that we've had a scheduler that only works with certain models. Do we need to add a required_scheduler field to the model configs?

I approved before I saw the note regarding the need to isolate the scheduler. I think this is the first time that we've had a scheduler that only works with certain models. Do we need to add a required_scheduler field to the model configs?

I think this is a broader question of how we anticipate people using LCM models, and where we want this type of validation to live.
My read on LCMs is that the flow will likely be some form of "Ideate/explore rapidly in LCM space, then push to the "Full" variant of the model for the final asset". In that world, I think the UI/UX might be more like each full model has an LCM variant that only generates using LCM schedulers when LCM mode is activated.
This is probably less "modular" than people might like, though, and opinionated

As mentioned in discord, think the answer here is moving towards "Default Settings" that can be configured on a per model basis (optionally). I don't think we want to too tightly couple "requirements"

This is not functional yet though. The LCM Scheduler fully fails. My understanding is that the LCM models do not take negative conditioning. So the shapes end up being mismatched. I think we need to set a bypass for that in our own conditioning. Something I haven't done before. If someone who is more familiar with that part of the code could take a look, I think that would be better.

With the latest changes to diffusers package, this PR works for the LCMScheduler / LCM-LoRA combo!
(I had just got a similar branch working, then saw this draft PR)
I've been testing LCMScheduler/LCM-LoRA with diffusers StableDiffusionXLPipeline as well. The outputs aren't identical, but are similar.
Example:
Left picture is with InvokeAI, right picture is with diffusers StableDiffusionXLPipeline. Both using base SDXL 1.0 model, LCMScheduler and SDXL LCM-LoRA, steps = 4, CFG scale = 1.1, seed=0, prompt="old man".

I've found that using a very low CFG Scale is crucial -- this was also the case for the earlier "SimianLuo/LCM_Dreamshaper_v7" distilled LCM model. Results in InvokeAI are also very sensitive to LoRA scale. In the example above, for StableDiffusionXLPipeline I didn't specify a LoRA scale, which I thought would default to 1.0. But for InvokeAI the results with LoRA scale = 1.0 were terrible. For the left image above the LoRA scale = 0.45.
Here's InvokeAI, holding everything fixed as above except changing CFG scale -- top row left to right 0.3, 0.4, 0.5, bottom row left to right 0.6, 0.75, 1.0:

I don't have an explanation for the differences between resulsts with InvokeAI and StableDIffusionXLPipeline, but I think the results are good enough at this point to merge the PR.


update American river rearing habitat to the Water Forum flow to WUA relationship (currently used in VAs)

[x] Use American Rearing WUA from VAs in place of IFIMWUA for rearing habitat in DSMhabitat in data-raw/watersheds/american/american_river.Rmd - Refer to VA processing to see how they got data into flow to acre format.
[x] Update both inchannel rearing and floodplain rearing
[x] existing data on American is in WUA sqft/1000 ft. New data is in WUA acres, use deer as an example for what WUA acres should look like. https://github.com/CVPIA-OSC/DSMhabitat/blob/main/data-raw/watershed/deer/deer_creek.Rmd
[x] Saved data object should be in sqm (use deer as example) for inchannel rearing, I think acres okay for floodplain, confirm on deer creek
[x] Duplicate juv data to work for both Juv and Fry...erin will check for what we have done in this case in the past
[x] Make plots that are x - flow to y - area (acres)
[x] Update documentation to describe change in data, list new data sources.
[x] Rerun markdown to save updated data in american_river.rmd
[ ] update DSMhabitat::watershed_methods to change from wua to sqm (check current nomenclature used in the file)
[ ] Rerun data-raw/cache_habitat.R file to resave new American river data.

[ ] ask Mark to contact Chris Hammersmark to get link for data source


use multistage build to reduce image size

The current image is over 600MB which is way to much.
Use multistage build to reduce image size to ~10MB
Before:
npalm/docker-discovery-agent   latest              258cc58cecae        11 months ago        684MB
After:
local/docker-discovery-agent   latest              3a99a10b0f65        About a minute ago   11.5MB

@kofemann sorry I missed you PR. I do not maintain the module anymore for more then a year since the same features are supported by the AWS ecs-agent.


ERR_BLOCKED_BY_CLIENT

I encountered a problem when I wanted to try this extension with the following details

I have topped up my credit balance on OpenAI by $5.00 and the problem still occurs, is there a solution to this problem?

Thank you in advance!

And the date for you grant acess are weird too "Grant 2: 1 october 2024"

Thank you, it's fixed


Add support for external HAL modules

Having added about 6 variations of private IREE HALs including the public ROCM HAL it would be good to support out of tree HALs.  We have a few more HALs that require carry forks that we would rather have built out of tree.
There is no need for binary compatibility etc in this ask -- all sources will be built together but we would like IREE HAL to look for "external" modules in some CMAKE specified path.

I’ll add a cmake list var to supersede what we have in iree/hal/drivers/ - you’ll then ensure you have a static library with the right name and a function that registers the driver and it’ll get linked and called by any iree tool using the driver registry. You can still manually register things in your own binaries if you want smaller binary sizes/deps but I suspect what you’re asking for here is for them to appear in iree-benchmark-module and such.

Rough idea is that we add an IREE_EXTERNAL_HAL_DRIVERS cmake var similar to LLVM_EXTERNAL_PROJECTS - you'd then declare your driver:
ROCM_HAL_DRIVER_SOURCE_DIR=experimental/rocm/
ROCM_HAL_DRIVER_TARGET=experimental_rocm_driver
list(APPEND IREE_EXTERNAL_HAL_DRIVERS "rocm")

then have a file with the registration function in it:
iree_status_t rocm_hal_driver_register(iree_hal_driver_registry_t* registry) {
  // do registration stuff here
}

and in iree/hal/drivers/ we'd include it:
foreach(driver ${IREE_EXTERNAL_HAL_DRIVERS})
  # include cmakelists that contains the library and fetches/finds/etc deps
  get_external_hal_driver_var(source_dir ${driver} SOURCE_DIR)
  add_subdirectory(${source_dir})
  # allow the external cmakelists to decide whether it's actually enabled (find_package success, etc)
  get_external_hal_driver_var(driver_usable ${driver} FOUND)
  if(${driver_usable})
    # add dep on the target so that it gets pulled into the build
    get_external_hal_driver_var(target ${driver} TARGET)
    list(APPEND IREE_HAL_DRIVER_MODULES ${target})
    # add the registration call to the list
    list(APPEND IREE_HAL_DRIVER_REGISTRATION_CALLS "${driver}_register")
  endif()
endforeach()
...
# produce an init_external.c with calls to each of the IREE_HAL_DRIVER_REGISTRATION_CALLS
configure_file(init_external.c.in init_external.c)
...
iree_cc_library(
  drivers ...
  SRCS "init.c" "init_external.c"
  DEPS ${IREE_HAL_DRIVER_MODULES}
  COPTS "-DIREE_HAL_EXTERNAL_DRIVER_REGISTRATION=1"

then in the init.c we do:
extern void iree_hal_register_external_drivers(iree_hal_driver_registry_t* registry);
iree_hal_register_all_available_drivers(iree_hal_driver_registry_t* registry) {
  iree_hal_register_external_drivers(registry);
}

Or something like this with the cmake goo resolved.
For bazel we could check in a file with the backends or genrule a templated one, but the cmake stuff should be this straightforward and make it possible to include external drivers without changing any IREE code.
/cc @ScottTodd

Nice this is exactly what we want on the CMake side. We haven't had to use bazel with externals so whatever is best on bazel side we are ok.

We may want something similar for executable loaders - that'd let us solve the #4298 issue by having vmvx/dylib/etc optional alongside static libraries and bigger deps (wasm/jits/etc).

A gotcha here will be that the benchmark infra is not setup to handle external drivers; it currently relies on knowing about the drivers in python instead of relying on what's available in cmake (what we can easily extend). There will need to be some work to remove all references to driver names in the code if we care about letting people use the benchmark infra (which would be really nice); basically the only place that should know about compilation targets/driver names/etc should be the definition of the benchmark suite and then those should be passed unmodified down to the lower level tooling (iree-compile/iree-benchmark-module/etc).


skip e2e binary during image builds

make was changed to build the e2e test binary after we broke it once and realized that e2e binary build was not covered by CI.
However, we do not need the e2e binary in the image builds for hypershift
This PR switches to a make target that doesn't build e2e.
@ironcladlou @enxebre

This shouldn't affect e2e but let's be sure:
/test-e2e

/test e2e-aws

/lgtm


Rounding issue when setting data

When trying to set data it may happen – probably due to rounding errors – that the crop area will be calculated as "too high" so it will fallback to something smaller. Having an image with 640x427 dimensions and setting the following data:
width:  427,
height: 427,
x:      106.5,
y:      0

with an aspectRatio of 1 will result in the following crop area data:
width:<PHONE_NUMBER>9999997
height:<PHONE_NUMBER>9999997
x: 106.5
y: 0

You can see it in action here. The code including a good old fiddle can be found here, but unfortunately I have some issues with my server, so the scripts will not execute. For looking at the code it's good enough, I think.

 I'll fix it soon. Thank you very much!!

Thanks a lot, works now.


Fixes for SCons shared cache for Travis-CI and AppVeyor-CI (master)

Fix pruning when the option progress is set to 'no'
Use separate caches for each job (Travis)
Use separate folders for each brach (AppVeyor)
Limit the cache size to 1024 MB for AppVeyor (512 MB per branch) which functions as a single cache, because AppVeyor still doesn't support a separate cache per branch.
Limit the cache size to 1024 MB per job for Travis, giving a total of 9GB (5GB for master branch + 4GB for branch 2.1) that function independently of each other.
Programs and libraries are not added to the cache, because those are too big and change too often, only intermediate build artifacts are added.

As explained here:
https://docs.travis-ci.com/user/caching/#Caches-and-build-matrices
It changes the URL of the cache so each  job has a different URL. It can be
used to share the same cache for some jobs while having separate caches for
others.
On Nov 28, 2017 3:42 AM, "Rémi Verschelde"<EMAIL_ADDRESS>wrote:

@akien-mga commented on this pull request.
In .travis.yml
https://github.com/godotengine/godot/pull/13359#discussion_r153413531:

@@ -18,22 +18,22 @@ matrix:
- env: STATIC_CHECKS=yes
os: linux
compiler: clang

env: GODOT_TARGET=x11 TOOLS=yes

env: GODOT_TARGET=x11 TOOLS=yes CACHE_NAME=${GODOT_TARGET}-gcc-tools

What does CACHE_NAME do? It doesn't seem used.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/pull/13359#pullrequestreview-79412320,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABwVjKr9pnoLt71R3_cpLa0S5IY7NkrKks5s67l7gaJpZM4QsxYa
.


GH-308: KStreamBuilderFB Polishing

null check on state listener and exception handler in case user has set manually
use normal method order (ctor, accessors, other methods)

cherry-pick to 1.2.x, 1.1.x (minus state listener)

Doh - that will teach me not to "think" while lying in bed.


Support Next.js App Router

Your integration tutorial (https://www.ory.sh/docs/getting-started/integrate-auth/nextjs) only shows how to do it with the pages router. However, my project uses the newer App router. How do I set it up with App router?

I think that a version that supports the app router is ready but hasn't been published to NPM yet. awaiting the release!

Hey @aryaniyaps thanks for your response! :) Do you have any ETA when the new version can be released? Are we talking about weeks / months / years? :D

Would love an update, we're all in on app router and can't integrate this without this.

App Router support would be great!

Another +1... all in on app router here too and desperate to migrate to Ory

Hey @aryaniyaps thanks for your response! :) Do you have any ETA when the new version can be released? Are we talking about weeks / months / years? :D

I'm not sure.. I'm not a maintainer here actually.. :)

@aryaniyaps tell us anyways!!111one

Any updates on this?

Hello, Ory Elements 1.0 is on the horizon where we support the latest Next features such as app router.

Hello, Ory Elements 1.0 is on the horizon where we support the latest Next features such as app router.

hi could you provide a timeline or roadmap to support this?
what kind of timescale are we talking about? weeks? quarters?

Hello there! For the people who are super into cutting edge "in development" stuff, here is a working app router example. We're still working on finishing up the last pieces (in particular add tests to ensure everything works):
https://github.com/ory/elements/tree/aeneasr/add-nextjs-app-router-example-2/examples/nextjs-app-router

I believe we still need to publish @ory/nextjs package which can happen once the PR is merged.

Am I too anxious? The dependency @ ory/extjs for the example was not found, in fact, it does not exist. Where should I look for it

When I used it, I found that it was in development, but it was an exciting feat

We are still in development so this is only a preview for now but we wanted to share our progress:)

Thank you for sharing the development progress! While it's still in development, it's really exciting to see a preview version with App Router support. I'll keep following the project's progress and looking forward to the official release.


Type 'x' does not conform to protocol 'TargetType'

FYI
I'm having this issue on the latest version, if regress to version  1.3.0 this doesn't happens.
After making some searches I see people having the same problem, but they are using Carthage and its not my case.
This even happens with the Demo project (after doing pod update). And Xcode doesn't provide more info

This was left hanging here for quite some time, sorry for the really late response.
This issue was due to the addition of var multipartBody: [Moya.MultipartFormData]? to the TargetType protocol.
I've updated the sample project in afc7f9fb which should fix this particular issue.

HI,
I am still getting this issue even after adding multiform method..please update on this ASAP


reduce amount of data loaded during atx validation

earlier this week i shared results of fs profiling with stacks.
reads pattern may vary, but in general it is somewhat x10-15 to writes, e.g for every 2GB of atxs written go-spacemesh loads 20GB-30GB.
it looks like that

in this change i applied several simple optimizations that follow directly from the pprof image shared above. all of them are in last commit, i made on top of v1.6.8

cache poet proofs, as they usually shared and quite large, so it is a waste to load them for every atx
identities stored without row_id, that leads to ineffecient IsMalicious check because of how clustered indexes are stored in sqlite, and also IsMalicious is run twice in close proximity
atx downloader was doing a lot of repetitive updates, that required reading before write

the read rate was reduced quite significantly to ~7.5GB per 2GB of writes.

the remaining problems for atx validation are:

still a lot of reads to validate maliciousness or atx, e.g loading prev/commitment/positioning atx
atxs themself require a lot of updates in indexes, you can see it specifically here, as how many btree updates are required for atx insertion

To me it looks like a lot of the changes of this PR have already been integrated into go-spacemesh. Feel free to re-open if that's not the case 🙂


Add task to push documentation to Heroku

This adds a documentation task that pushes current master to Heroku. You can now just run:
inv documentation

I tested this when upgrading the documentation to 0.2.6.

Would it be worth allowing updating the docs requirements with this?

Would your expectation be that this would edit docs/requirements.txt file? If so, happy to add that in. An alternative might be to:

bundle bumping the version in docs/requirements.txt as part of inv release. It probably makes sense to update the documentation version as part of a release.
bundle pushing to heroku as part of inv postrelease. Once we've verified that a release is correct, we probably always want to update the documentation.

Would your expectation be that this would edit docs/requirements.txt file?

Yes, that's what I had in mind.
I think bundling the version bump for docs/requirements.txt into inv release makes sense, I can't really think of a scenario in which we wouldn't want to run the docs on the lastest version.
While I don't have a problem with pushing to heroku as part of inv postrelease, did you have in mind that this would replace inv documentation? I think there are cases where we would want to update the docs without making a new release of the python package, so it might be useful to have inv documentation available in such cases.

I think bundling the version bump for docs/requirements.txt into inv release makes sense, I can't really think of a scenario in which we wouldn't want to run the docs on the lastest version.

Great! I can set that up.

While I don't have a problem with pushing to heroku as part of inv postrelease, did you have in mind that this would replace inv documentation? I think there are cases where we would want to update the docs without making a new release of the python package, so it might be useful to have inv documentation available in such cases.

Good question. I was probably thinking of keeping inv documentation as a stand-alone task and declare a dependency on it in inv postrelease. That would deploy documentation automatically after a release, but it would also let you deploy doc changes automatically.

I think this is now ready for review again. To summarise, this PR:

bumps the version of dbc in docs/requirements.txt as part of invoke release.
pushes documentation as part of invoke postrelease
adds a stand-alone invoke documentation task that pushes the current master branch to heroku.

This looks great to me. Happy for you to merge and test.

Great! Will do that tomorrow (Sunday).


Allow user to enter tags for the webpage

There is no way to add custom tags for the webpage from the user. All notes are currently tagged with wbjs.

Feature added in v3.0.5 and up


URL parsing failure

The following link:
https://medium.com/@philipla/the-two-types-of-product-virality-8ae744b1c4d7
gets rendered as:

Care to open a pull request? It should be easy to add @ as a valid auto linking character (with a test).