Extreme CPU Usage on small cluster

Environmental Info:
K3s Version:
k3s version v1.30.1+k3s1 (80978b5b)
go version go1.22.2

Node(s) CPU architecture, OS, and Version:
Linux k3s-server.internal.self-hosted.io 4.18.0-513.24.1.el8_9.x86_64 #1 SMP Thu Apr 4 18:13:02 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
1 server, 2 agents
Describe the bug:
k3s server has extremely high CPU usage, even scaling pods back to 0 there's no change. A 4 core/16GB RAM machine has a load average of and often spikes to 50/60!
load average: 13.42, 20.74, 22.08

Steps To Reproduce:
k3s has been running for a while with the upgrade controller/operator.
Expected behavior:
CPU is within sane numbers
Actual behavior:
Extremely high CPU usage, slow API response times and often timeouts. Appears to spam /var/log/messages with trace logging.
Additional context / logs:
Nodes have NVMe local drives. The k3s-server process is using 400%, there's no other large processes on the system. It has high %USR with about 20% SYS/Kernel time. The state.db is also 12GB~ and SQLLite fails to vacuum. Cluster age is 270d.

Okay, this is certainly SQLLite related. Migrating to etcd the cluster has returned to normal levels by adding --cluster-init to the systemd unit, however the Trace: logging is still present.
The server is now at 0.4 1m load average and responding to requests quickly. The SQLLite vacuum did not do anything, but migrating to etcd did. Trying to set -v=0 in the systemd unit doesn't seem to have an effect on the logs either.
Happy to close as the root issue seems clear.


text_match fusing

When executing queries like:
select col1, col2 from Table where text_match(col1, '/r1/') or text_match(col1, '/r2/')

Pinot has to scan the referred column twice. This PR creates an optimization that tries to fuse boolean algebra and text_match predicates. Specifically, as indicated in the Javadoc:

Queries where text_match(col1, '/r1/') and text_match(col1, '/r2/') will be translated to where text_match(col1, '/(?=r1)(?=r2)/')
Queries where text_match(col1, '/r1/') or text_match(col1, '/r2/') will be translated to where text_match(col1, '/(?:r1)|(?:r2)/')
Queries where not text_match(col1, '/r1/') will be translated to where text_match(col1, '/(?!r1)/')

There are some tests that apply the optimization to more advanced cases.
Regex can be quite complex and it isn't clear to me how expressive is the regex language we (and lucene) support. By doing some analysis I'm sure that this optimization will break some regex like the ones that use backreferences.
To know if the optimization can be applied or not, it would be necessary to analyze the regex, which would require to parse the regex into a AST. As far as I know Pinot doesn't have that, so this optimization is disabled by default and can be enabled by activating a new query option.
Future improvements may include to translate some text_match predicated that are not regex to regex, so we would be able to merge things like where text_match(col1, '/r1/') or text_match(col1, '"literal1" and "literal2"') into where text_match(col1, '/(?:r1)|(?=(?:literal1)|(?:literal2))/')

Codecov Report

Merging #8818 (1a54b02) into master (b597226) will decrease coverage by 42.74%.
The diff coverage is 22.18%.

:exclamation: Current head 1a54b02 differs from pull request most recent head f6d6db3. Consider uploading reports for the commit f6d6db3 to get more accurate results

@@              Coverage Diff              @@
##             master    #8818       +/-   ##
=============================================
- Coverage     69.72%   26.97%   -42.75%
+ Complexity     4621        1     -4620
=============================================
  Files          1735     1725       -10
  Lines         91320    91055      -265
  Branches      13644    13616       -28
=============================================
- Hits          63670    24560    -39110
- Misses        23228    64157    +40929
+ Partials       4422     2338     -2084

Flag
Coverage Δ

integration1
26.97% <22.18%> (-0.01%)
:arrow_down:

integration2
?

unittests1
?

unittests2
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...e/pinot/common/utils/FileUploadDownloadClient.java
46.28% <ø> (-13.15%)
:arrow_down:

...ler/api/resources/TableConfigsRestletResource.java
0.00% <0.00%> (-75.44%)
:arrow_down:

...ava/org/apache/pinot/core/plan/FilterPlanNode.java
48.09% <0.00%> (-34.08%)
:arrow_down:

...t/core/query/optimizer/filter/FilterOptimizer.java
0.00% <0.00%> (ø)

.../optimizer/filter/FlattenAndOrFilterOptimizer.java
88.88% <ø> (ø)

...ery/optimizer/filter/MergeEqInFilterOptimizer.java
72.83% <ø> (-19.76%)
:arrow_down:

...optimizer/filter/TimePredicateFilterOptimizer.java
40.66% <ø> (-41.34%)
:arrow_down:

...nt/creator/impl/inv/BitmapInvertedIndexWriter.java
0.00% <0.00%> (-90.00%)
:arrow_down:

...gment/index/readers/BitmapInvertedIndexReader.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ache/pinot/segment/local/utils/IngestionUtils.java
0.00% <0.00%> (-28.24%)
:arrow_down:

... and 1261 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b597226...f6d6db3. Read the comment docs.

Wondering how much performance gain are we expecting here? text_match will use the Lucene text index to get the matching docs for the request. Essentially the comparison is between 2 simple requests vs 1 combined request. Some perf number would be helpful to justify the change.

I thought this was supposed to be targeting regexp_like

I thought this was supposed to be targeting regexp_like

My bad. I looked for regexp in the docs and given that only found regexpExact, I thought it was text_match the operation we want to optimize.
Anyway, the idea is the same, so most of the code can be reused

I don’t think it’s possible to express backtracking behaviour in a LIKE statement, this optimisation would target long unions of LIKE statements generated by ORMs, and should be most effective in reducing the impact on a cluster when there is no text or FST index in place. I expect this would also help FST based regular expression evaluation, so getting some numbers to justify extending to FST regex evaluators would be great.

ORMs are not going to generate them, but a user may try to do something like regexp_like(col1, '(.+)=\1'), which will be positive in texts like a=a but false in b=a. I think this is very bizarre, but possible.
What I'm going to do is to try to detect the usage of backreferences in the regexp. To do so, I'm going to discard any regexp_like whose predicate matches with .*(?:(?:\\[0-9]+)|(?:\\k\<\w+\>)).*. The function may have false positives (for example the regexp\\1 will be a false positive) and false negatives (where is more difficult to find examples, maybe playing with utf8?), but it may be good enough to activate the query option by default in a future release.

ORMs are not going to generate them

On the contrary, this exact pattern has been observed generated by customer systems querying Pinot

I've changed the implementation and the description of the PR to be focused on regexp_like instead of text_match.
Anyway, I would like to have some benchmarks before this feature is merged.

I don’t believe the native FST supports backreferences, and I don’t think it would be easy to add support either. Not sure about the lucene FST index. As mentioned above, LIKE statements (e.g. LIKE ‘%foo%’) can’t express all regular expression features, which should reduce the scope of the edge cases to be considered.
This definitely needs benchmarking for at least the following scenarios;

Single like statement
N like statements combined with OR

for each of:

no FST index - both raw and dictionarized
Lucene FST index
Native FST index

I didn't know LIKE expressions were evaluated as REGEX_LIKE. Reading the code, it seems that the transformation is made in RequestContextUtils, which is evaluated after the filter optimization is done. This means that (in case I don't miss something else) the optimizations included here will not affect LIKE expressions.
I can modify the optimizer to add the ability to transform LIKE expressions into REGEX_LIKE the same way RequestContextUtils does right now. But I'm going to focus on the benchmark that tests the scenarios @richardstartin mentioned, but explicitly using REGEXP_LIKE for now.

@gortiz Thanks for the detailed analysis. This is awesome, and very interesting!

We cannot use this optimization with Lucene
The cost is too high when the expression gets complex. It even with  java.lang.RuntimeException: org.apache.lucene.util.automaton.TooComplexToDeterminizeException: Determinizing .*(?:^.*domain0.*$)|(?:^.*domain1.*$)|(?:^.*domain2.*$)|(?:^.*domain3.*$)|(?:^.*domain4.*$)|(?:^.*domain5.*$)|(?:^.*domain6.*$)|(?:^.*domain7.*$)|(?:^.*domain8.*$)|(?:^.*domain9.*$).* would result in more than 10000 states. when the expression have 10 branches.

Curious how native FST handles it, and why is it so much faster than the Lucene FST. cc @atris

In general COL like 'whatever' expression is 2x-3x worse than regexp_like(COL, 'similar expression')
Reading the code it looks to me that Pinot automatically translate one to the other, but it that transformation is not applied when BaseQueriesTest.getBrokerResponse is called (which is great, because we can test the difference).

I think the reason is that it is not apple-to-apple comparison. col LIKE 'abc' is actually equivalent to regexp_like(col, '^abc$')

Pinot regex execute faster wildcards than a single values
It is quite clear in the optimal cases:
-- this is just a `where regexp_like(DOMAIN_NAMES, 'domain\d')`
BenchmarkFuseRegexp.optimal10                 null  avgt    5     0.149 ±    0.082  ms/op
-- this is just a `where regexp_like(DOMAIN_NAMES, 'domain0')`
BenchmarkFuseRegexp.optimal1Regex             null  avgt    5   243.160 ±   59.722  ms/op

At the beginning I thought it was an error in my code, but it is easy to try that using the github_events dataset.
select * from githubEvents where regexp_like(actor_url, 'victor\d') limit 1

Responds quite faster than
select * from githubEvents where regexp_like(actor_url, 'victor7') limit 1

The difference is not the 2.000x that the benchmark shows, but is consistently between 2-3x.
I'm guessing that the issue actually comes from Java regex engine, but it is just a hunch, I didn't actually test it.

Let's re-do the benchmark. I don't believe there can be 2000x difference. Also, this is very counter-intuitive, so I'd suggest let's figure out if it is actually from the java regex engine, or because of our code.

When there are several regexp in the actual expression, indexes are very useful.
Examples:
BenchmarkFuseRegexp.decreasing9Regex        LUCENE  avgt    5     0.593 ±    0.362  ms/op
BenchmarkFuseRegexp.decreasing9Regex        NATIVE  avgt    5     0.422 ±    0.222  ms/op
BenchmarkFuseRegexp.decreasing9Regex          null  avgt    5  1053.673 ±  379.729  ms/op

BenchmarkFuseRegexp.increasing10Regex       LUCENE  avgt    5     0.590 ±    0.208  ms/op
BenchmarkFuseRegexp.increasing10Regex       NATIVE  avgt    5     0.435 ±    0.159  ms/op
BenchmarkFuseRegexp.increasing10Regex         null  avgt    5  1096.238 ±  289.928  ms/op

I feel the inefficiency is from regexp_like(DOMAIN_NAMES, 'domain0') (exact match) which takes much longer than expected. We should check what is causing the huge performance difference

Yeah, the REGEXP_LIKE translation from LIKE is not equivalent.
Also, remember that any FST traversal will benefit most from a selective expression rather than a full term match.
Lastly, why do you say that indices don't help for single regexes? Unless I am misreading the benchmark result, even for a single regex case, native index is pretty fast?

I will check out the branch and give some comments tomorrow

Thanks @atris for taking a look. I also have a question around how Lucene and native FST handle the query, specifically for these results:
SELECT INT_COL FROM MyTable WHERE regexp_like(DOMAIN_NAMES, '(?:^.*domain9.*$)|...|(?:^.*domain1.*$)')
BenchmarkFuseRegexp.decreasing9Fusing       LUCENE  avgt    5    24.198 ±   13.439  ms/op
BenchmarkFuseRegexp.decreasing9Fusing       NATIVE  avgt    5     0.263 ±    0.019  ms/op
BenchmarkFuseRegexp.decreasing9Fusing         null  avgt    5     0.476 ±    0.040  ms/op

SELECT INT_COL FROM MyTable WHERE regexp_like(DOMAIN_NAMES, '(?:^.*domain0.*$)|...|(?:^.*domain9.*$)')
BenchmarkFuseRegexp.increasing10Fusing      LUCENE  avgt    5     ERROR
BenchmarkFuseRegexp.increasing10Fusing      NATIVE  avgt    5     0.326 ±    0.138  ms/op
BenchmarkFuseRegexp.increasing10Fusing        null  avgt    5     0.226 ±    0.062  ms/op

We can see that Lucene FST is very slow in this case, but native is still performing okay. Trying to understand what is causing this difference.

The difference will be due to determinization of the automaton, which the native implementation doesn’t do.

@Jackie-Jiang one thing is that the native index aggressively tries to prune branches, both while building the query automaton and during the traversal.
That is also what causes general suffix and prefix queries to be much faster.

Thanks @atris for taking a look. I also have a question around how Lucene and native FST handle the query, specifically for these results:
SELECT INT_COL FROM MyTable WHERE regexp_like(DOMAIN_NAMES, '(?:^.*domain9.*$)|...|(?:^.*domain1.*$)')
BenchmarkFuseRegexp.decreasing9Fusing       LUCENE  avgt    5    24.198 ±   13.439  ms/op
BenchmarkFuseRegexp.decreasing9Fusing       NATIVE  avgt    5     0.263 ±    0.019  ms/op
BenchmarkFuseRegexp.decreasing9Fusing         null  avgt    5     0.476 ±    0.040  ms/op

SELECT INT_COL FROM MyTable WHERE regexp_like(DOMAIN_NAMES, '(?:^.*domain0.*$)|...|(?:^.*domain9.*$)')
BenchmarkFuseRegexp.increasing10Fusing      LUCENE  avgt    5     ERROR
BenchmarkFuseRegexp.increasing10Fusing      NATIVE  avgt    5     0.326 ±    0.138  ms/op
BenchmarkFuseRegexp.increasing10Fusing        null  avgt    5     0.226 ±    0.062  ms/op

We can see that Lucene FST is very slow in this case, but native is still performing okay. Trying to understand what is causing this difference.

The difference will be due to determinization of the automaton, which the native implementation doesn’t do.

I doubt determinization alone can cause such a massive difference.

Well, it’s an exponential time algorithm and Lucene refuses to even attempt to determinize some of the automata in the benchmark because they are too large. For an apples to apples comparison, native determinization should be re-enabled (by temporarily reverting/reversing #8237)

Thanks @atris for taking a look. I also have a question around how Lucene and native FST handle the query, specifically for these results:
SELECT INT_COL FROM MyTable WHERE regexp_like(DOMAIN_NAMES, '(?:^.*domain9.*$)|...|(?:^.*domain1.*$)')
BenchmarkFuseRegexp.decreasing9Fusing       LUCENE  avgt    5    24.198 ±   13.439  ms/op
BenchmarkFuseRegexp.decreasing9Fusing       NATIVE  avgt    5     0.263 ±    0.019  ms/op
BenchmarkFuseRegexp.decreasing9Fusing         null  avgt    5     0.476 ±    0.040  ms/op

SELECT INT_COL FROM MyTable WHERE regexp_like(DOMAIN_NAMES, '(?:^.*domain0.*$)|...|(?:^.*domain9.*$)')
BenchmarkFuseRegexp.increasing10Fusing      LUCENE  avgt    5     ERROR
BenchmarkFuseRegexp.increasing10Fusing      NATIVE  avgt    5     0.326 ±    0.138  ms/op
BenchmarkFuseRegexp.increasing10Fusing        null  avgt    5     0.226 ±    0.062  ms/op

We can see that Lucene FST is very slow in this case, but native is still performing okay. Trying to understand what is causing this difference.

The difference will be due to determinization of the automaton, which the native implementation doesn’t do.

I doubt determinization alone can cause such a massive difference.

Well, it’s an exponential time algorithm and Lucene refuses to even attempt to determinize some of the automata in the benchmark because they are too large. For an apples to apples comparison, native determinization should be re-enabled (by temporarily reverting/reversing #8237)

I am still failing to understand what the concern is.
Even if its plain determinization that causes native to be significantly faster, is that not a good thing?

Thanks @atris for taking a look. I also have a question around how Lucene and native FST handle the query, specifically for these results:
SELECT INT_COL FROM MyTable WHERE regexp_like(DOMAIN_NAMES, '(?:^.*domain9.*$)|...|(?:^.*domain1.*$)')
BenchmarkFuseRegexp.decreasing9Fusing       LUCENE  avgt    5    24.198 ±   13.439  ms/op
BenchmarkFuseRegexp.decreasing9Fusing       NATIVE  avgt    5     0.263 ±    0.019  ms/op
BenchmarkFuseRegexp.decreasing9Fusing         null  avgt    5     0.476 ±    0.040  ms/op

SELECT INT_COL FROM MyTable WHERE regexp_like(DOMAIN_NAMES, '(?:^.*domain0.*$)|...|(?:^.*domain9.*$)')
BenchmarkFuseRegexp.increasing10Fusing      LUCENE  avgt    5     ERROR
BenchmarkFuseRegexp.increasing10Fusing      NATIVE  avgt    5     0.326 ±    0.138  ms/op
BenchmarkFuseRegexp.increasing10Fusing        null  avgt    5     0.226 ±    0.062  ms/op

We can see that Lucene FST is very slow in this case, but native is still performing okay. Trying to understand what is causing this difference.

The difference will be due to determinization of the automaton, which the native implementation doesn’t do.

I doubt determinization alone can cause such a massive difference.

Well, it’s an exponential time algorithm and Lucene refuses to even attempt to determinize some of the automata in the benchmark because they are too large. For an apples to apples comparison, native determinization should be re-enabled (by temporarily reverting/reversing #8237)

I am still failing to understand what the concern is.
Even if its plain determinization that causes native to be significantly faster, is that not a good thing?

@Jackie-Jiang was asking about the root cause of the difference. I suggested that the root cause is likely to be determinization, but it needs to be investigated. Where has concern been expressed?

Just circling back here having taken a quick look at BenchmarkFuseRegexp.decreasing9Fusing - I got similar results to Gonzalo:
Benchmark                              (_fstType)  Mode  Cnt   Score   Error  Units
BenchmarkFuseRegexp.decreasing9Fusing      LUCENE  avgt    5  32.525 ± 2.733  ms/op
BenchmarkFuseRegexp.decreasing9Fusing      NATIVE  avgt    5   0.487 ± 0.123  ms/op
BenchmarkFuseRegexp.decreasing9Fusing        null  avgt    5   0.845 ± 0.204  ms/op

Lucene spends virtually all of its time in minimization (which I mistakenly referred to as determinization earlier):

Now if I enable minimization in the native implementation by reverting #8237, the scores change drastically:
Benchmark                              (_fstType)  Mode  Cnt    Score    Error  Units
BenchmarkFuseRegexp.decreasing9Fusing      LUCENE  avgt    5   32.399 ±  3.005  ms/op
BenchmarkFuseRegexp.decreasing9Fusing      NATIVE  avgt    5  366.520 ± 40.574  ms/op
BenchmarkFuseRegexp.decreasing9Fusing        null  avgt    5    0.819 ±  0.172  ms/op

As do the profiles. Without minimization, there is no clear single bottleneck:

With minimization, there is a single bottleneck, and it's in minimization :

As to whether this is cause for concern is beside the point; this is the root cause of the difference observed, rather than a quirk of the benchmark which needs to be analysed and removed.

Lastly, why do you say that indices don't help for single regexes? Unless I am misreading the benchmark result, even for a single regex case, native index is pretty fast?

My benchmark may not be the best and it is only measuring a few regexp cases, but I wouldn't say that native gives us a significant advantage:
-- this actually has a single, quite complex regex expression
BenchmarkFuseRegexp.decreasing9Fusing       LUCENE  avgt    5    24.198 ±   13.439  ms/op
BenchmarkFuseRegexp.decreasing9Fusing       NATIVE  avgt    5     0.263 ±    0.019  ms/op <-- here native is faster by 1.8x
BenchmarkFuseRegexp.decreasing9Fusing         null  avgt    5     0.476 ±    0.040  ms/op
-- this actually has a single, quite complex regex expression
BenchmarkFuseRegexp.increasing10Fusing      LUCENE  avgt    5     ERROR
BenchmarkFuseRegexp.increasing10Fusing      NATIVE  avgt    5     0.326 ±    0.138  ms/op <-- here native is slower by 1.4x, although the difference is lower than the error margin
BenchmarkFuseRegexp.increasing10Fusing        null  avgt    5     0.226 ±    0.062  ms/op
-- this is just a `where regexp_like(DOMAIN_NAMES, 'domain\d')`
BenchmarkFuseRegexp.optimal10               LUCENE  avgt    5     0.169 ±    0.066  ms/op
BenchmarkFuseRegexp.optimal10               NATIVE  avgt    5     0.136 ±    0.011  ms/op  <-- here native is faster by 1.1x, although the difference is lower than the error margin
BenchmarkFuseRegexp.optimal10                 null  avgt    5     0.149 ±    0.082  ms/op

So we have one case where native is 1.8x faster and some others were it is equivalent to the java regex engine, which is known by its lack of performance. I mean, it is not like the native index is atrocious, but I would expect more from an index than a situational 1.8x performance increase.
To be clear, as I already said, this is not a benchmark on the FST itself and it is very narrow, so I don't want to get conclusions about whether we should or we should not use these indexes. I think this benchmark proves that there are very specific cases where these indexes are not very useful and therefore it would be nice to invest time to design proper tests with different kind of regex to verify whether FST indexes are in general useful evaluating regex or not.

I was able to invest more time in this topic today and I have to say that my initial benchmark was completely useless. Apart from some typos in the queries (which may have some impact), there were two problems that completely fooled me:

The implicit limit introduced by Pinot. As only the first 10 elements were returned, queries that actually found elements could finish faster.
Something is wrong with Lucene and Native indexes. I guess I'm not configuring them correctly, but they simply return 0 results when the regexp is a little bit complex (like having groups).

Therefore I've changed all the tests to return a count and each iteration is actually verifying whether the expected value is returned or not, in which case the benchmark is stopped. As a result, all fusing tests fail whenever there is an index. I hope it is something related to my test (ie I didn't configure the index properly) and not some semantic discrepancy these indexes have in relation how each engine evaluates regexp, as that could mean that there are customers receiving false results.
These are the results I've got. Note that some combinations are not shown. For example, decreasing9Fusing and LUCENE or NATIVE. That means that these combinations did not return what it was expected (in all these particular cases it means that they return 0 rows). Note that apart from the fusing case, benchmarks like optimal10, which is regexp_like(DOMAIN_NAMES, 'domain\d') did also failed.
Benchmark                                  (_fstType)  Mode  Cnt     Score      Error  Units
BenchmarkFuseRegexp.decreasing9Fusing            null  avgt    5  1438.185 ±   73.963  ms/op
BenchmarkFuseRegexp.decreasing9Like            LUCENE  avgt    5    41.927 ±    0.984  ms/op
BenchmarkFuseRegexp.decreasing9Like            NATIVE  avgt    5    40.232 ±    5.597  ms/op
BenchmarkFuseRegexp.decreasing9Like              null  avgt    5  3957.902 ±  154.228  ms/op
BenchmarkFuseRegexp.decreasing9Regex           LUCENE  avgt    5    40.744 ±    2.059  ms/op
BenchmarkFuseRegexp.decreasing9Regex           NATIVE  avgt    5    61.443 ±    1.301  ms/op
BenchmarkFuseRegexp.decreasing9Regex             null  avgt    5  2116.747 ±  602.570  ms/op
BenchmarkFuseRegexp.increasing10Fusing           null  avgt    5  1320.869 ±   45.509  ms/op
BenchmarkFuseRegexp.increasing10Like           LUCENE  avgt    5    42.253 ±    0.695  ms/op
BenchmarkFuseRegexp.increasing10Like           NATIVE  avgt    5    39.049 ±    1.829  ms/op
BenchmarkFuseRegexp.increasing10Like             null  avgt    5  5071.207 ± 3134.887  ms/op
BenchmarkFuseRegexp.increasing10Regex          LUCENE  avgt    5    44.084 ±    2.737  ms/op
BenchmarkFuseRegexp.increasing10Regex          NATIVE  avgt    5    39.650 ±    1.567  ms/op
BenchmarkFuseRegexp.increasing10Regex            null  avgt    5  2318.479 ±   75.632  ms/op
BenchmarkFuseRegexp.optimal10                    null  avgt    5   185.757 ±    3.426  ms/op
BenchmarkFuseRegexp.optimal10NotFound          LUCENE  avgt    5     0.212 ±    0.136  ms/op
BenchmarkFuseRegexp.optimal10NotFound          NATIVE  avgt    5     0.201 ±    0.140  ms/op
BenchmarkFuseRegexp.optimal10NotFound            null  avgt    5   248.525 ±   60.819  ms/op
BenchmarkFuseRegexp.optimal1Like               LUCENE  avgt    5    16.431 ±    3.533  ms/op
BenchmarkFuseRegexp.optimal1Like               NATIVE  avgt    5    16.586 ±    0.828  ms/op
BenchmarkFuseRegexp.optimal1Like                 null  avgt    5   468.731 ±  185.623  ms/op
BenchmarkFuseRegexp.optimal1LikeNotFound       LUCENE  avgt    5     0.197 ±    0.062  ms/op
BenchmarkFuseRegexp.optimal1LikeNotFound       NATIVE  avgt    5     0.170 ±    0.072  ms/op
BenchmarkFuseRegexp.optimal1LikeNotFound         null  avgt    5   477.176 ±  129.445  ms/op
BenchmarkFuseRegexp.optimal1Regex              LUCENE  avgt    5    15.485 ±    0.517  ms/op
BenchmarkFuseRegexp.optimal1Regex              NATIVE  avgt    5    16.711 ±    0.857  ms/op
BenchmarkFuseRegexp.optimal1Regex                null  avgt    5   205.676 ±    6.921  ms/op
BenchmarkFuseRegexp.optimal1RegexNotFound      LUCENE  avgt    5     0.212 ±    0.123  ms/op
BenchmarkFuseRegexp.optimal1RegexNotFound      NATIVE  avgt    5     0.186 ±    0.029  ms/op
BenchmarkFuseRegexp.optimal1RegexNotFound        null  avgt    5   239.459 ±   13.846  ms/op
BenchmarkFuseRegexp.selective2Fusing             null  avgt    5   541.530 ±  168.173  ms/op
BenchmarkFuseRegexp.selective2Like             LUCENE  avgt    5    19.961 ±    0.937  ms/op
BenchmarkFuseRegexp.selective2Like             NATIVE  avgt    5    20.387 ±    1.953  ms/op
BenchmarkFuseRegexp.selective2Like               null  avgt    5   916.620 ±   96.702  ms/op
BenchmarkFuseRegexp.selective2Regex            LUCENE  avgt    5    23.366 ±    0.795  ms/op
BenchmarkFuseRegexp.selective2Regex            NATIVE  avgt    5    20.037 ±    0.601  ms/op
BenchmarkFuseRegexp.selective2Regex              null  avgt    5   389.965 ±    7.284  ms/op

I've also changed RegexpPatternConverterUtils (and its relative test) to do not introduce useless ^.* at the beginning of the expression nor .*$ at the end. I've repeated the benchmark with that change and these are the results I've got. As you can see, most benchmarks didn't change that much, but the ones that use like and have no index are twice as fast, so they achieve the same numbers than their equivalent optimalXRegex, as expected.
Benchmark                                  (_fstType)  Mode  Cnt     Score     Error  Units
BenchmarkFuseRegexp.decreasing9Fusing            null  avgt    5  1365.403 ±  91.822  ms/op
BenchmarkFuseRegexp.decreasing9Like            LUCENE  avgt    5    40.553 ±   1.394  ms/op
BenchmarkFuseRegexp.decreasing9Like            NATIVE  avgt    5    39.057 ±   0.905  ms/op
BenchmarkFuseRegexp.decreasing9Like              null  avgt    5  2046.090 ±  40.428  ms/op
BenchmarkFuseRegexp.decreasing9Regex           LUCENE  avgt    5    42.584 ±   2.387  ms/op
BenchmarkFuseRegexp.decreasing9Regex           NATIVE  avgt    5    61.583 ±   1.893  ms/op
BenchmarkFuseRegexp.decreasing9Regex             null  avgt    5  2054.523 ±  75.452  ms/op
BenchmarkFuseRegexp.increasing10Fusing           null  avgt    5  1305.249 ±  65.576  ms/op
BenchmarkFuseRegexp.increasing10Like           LUCENE  avgt    5    43.328 ±   2.040  ms/op
BenchmarkFuseRegexp.increasing10Like           NATIVE  avgt    5    61.580 ±   1.149  ms/op
BenchmarkFuseRegexp.increasing10Like             null  avgt    5  2272.112 ±  99.393  ms/op
BenchmarkFuseRegexp.increasing10Regex          LUCENE  avgt    5    54.319 ±   2.048  ms/op
BenchmarkFuseRegexp.increasing10Regex          NATIVE  avgt    5    39.080 ±   1.817  ms/op
BenchmarkFuseRegexp.increasing10Regex            null  avgt    5  2267.062 ±  49.525  ms/op
BenchmarkFuseRegexp.optimal10                    null  avgt    5   187.012 ±  13.054  ms/op
BenchmarkFuseRegexp.optimal10NotFound          LUCENE  avgt    5     0.210 ±   0.103  ms/op
BenchmarkFuseRegexp.optimal10NotFound          NATIVE  avgt    5     0.201 ±   0.066  ms/op
BenchmarkFuseRegexp.optimal10NotFound            null  avgt    5   242.423 ±  17.612  ms/op
BenchmarkFuseRegexp.optimal1Like               LUCENE  avgt    5    16.395 ±   0.884  ms/op
BenchmarkFuseRegexp.optimal1Like               NATIVE  avgt    5    15.506 ±   0.591  ms/op
BenchmarkFuseRegexp.optimal1Like                 null  avgt    5   209.389 ±  11.456  ms/op
BenchmarkFuseRegexp.optimal1LikeNotFound       LUCENE  avgt    5     0.209 ±   0.129  ms/op
BenchmarkFuseRegexp.optimal1LikeNotFound       NATIVE  avgt    5     0.172 ±   0.021  ms/op
BenchmarkFuseRegexp.optimal1LikeNotFound         null  avgt    5   245.542 ±  35.323  ms/op
BenchmarkFuseRegexp.optimal1Regex              LUCENE  avgt    5    15.421 ±   0.804  ms/op
BenchmarkFuseRegexp.optimal1Regex              NATIVE  avgt    5    15.908 ±   0.491  ms/op
BenchmarkFuseRegexp.optimal1Regex                null  avgt    5   204.505 ±  13.134  ms/op
BenchmarkFuseRegexp.optimal1RegexNotFound      LUCENE  avgt    5     0.210 ±   0.089  ms/op
BenchmarkFuseRegexp.optimal1RegexNotFound      NATIVE  avgt    5     0.203 ±   0.107  ms/op
BenchmarkFuseRegexp.optimal1RegexNotFound        null  avgt    5   235.262 ±   8.533  ms/op
BenchmarkFuseRegexp.selective2Fusing             null  avgt    5   517.545 ± 193.977  ms/op
BenchmarkFuseRegexp.selective2Like             LUCENE  avgt    5    23.622 ±   1.178  ms/op
BenchmarkFuseRegexp.selective2Like             NATIVE  avgt    5    20.547 ±   0.778  ms/op
BenchmarkFuseRegexp.selective2Like               null  avgt    5   399.638 ±  54.746  ms/op
BenchmarkFuseRegexp.selective2Regex            LUCENE  avgt    5    23.883 ±   0.802  ms/op
BenchmarkFuseRegexp.selective2Regex            NATIVE  avgt    5    20.113 ±   0.858  ms/op
BenchmarkFuseRegexp.selective2Regex              null  avgt    5   416.987 ±  28.789  ms/op

With this results, the optimization introduced in this PR doesn't seem to be amazing, but to be honest I'm quite worried about what is going on with regexp and Lucene/Native indexes. As said above, I hope that the benchmark is not correctly configuring the indexes, but even with that, it seems very dangerous that an index can be configured in such a way that it affects the semantics of the queries.
@Jackie-Jiang @atris I would really appreciate if any of you can take a look at the benchmark trying to find what is going on.

@gortiz are you creating a FST index or a text index?

@gortiz another thing to remember is that neither Lucene nor native FST index follow the java regex pattern language.
If you look up the Lucene regex documentation, and look at the FST index and LIKE tests in Pinot, you will notice the difference.
Unfortunately, since we do not actually parse the regex engine until deep inside the FST territory, we have no way of reporting unknown characters. That may be a factor leading to 0 results for your queries

@gortiz another thing to remember is that neither Lucene nor native FST index follow the java regex pattern language.
If you look up the Lucene regex documentation, and look at the FST index and LIKE tests in Pinot, you will notice the difference.
Unfortunately, since we do not actually parse the regex engine until deep inside the FST territory, we have no way of reporting unknown characters. That may be a factor leading to 0 results for your queries

Wow, that is shocking and I would see that as a huge problem to use Pinot in a big team. In cases like that I would expect to have a new operation that does the search using the Lucene expression language or whatever we want to use and fail if there is no index. What you are saying is that if I have a column that is not indexed and suddenly a colleague of mine decides to index that column, queries I had will start to return different (and incorrect) results.
Anyway, if that is what was decided, that is what we have. I'm going to close this PR, as it doesn't make sense to apply this optimization if it can be produce false results when someone adds an index.

@gortiz There has been a long comment trail on this PR so I am summarising my responses in this comment:

Performance -- Like any other index, FST indices also have their worst case. FST is an automaton, and in an extreme scenario where a high number of branches are traversed for a query, the performance of the index will approach the brute force approximation.

In this case, the query you tried had an abnormally large number of branches, to the extent that Lucene decided to not even try and process the query. The average user rarely fires such queries, and in the corner cases where they do, they should be ok with a brute force latency, as long as their 99th percentile queries keep performing significantly better, which is the case as is evident from the benchmarks that test the FST index.

Language Semantics -- Lucene has, for long, published its regex dialect clearly. Engines based on Lucene, such as ElasticSearch and Apache Solr, follow the same dialect (https://help.oncrawl.com/en/articles/1685982-lucene-regex-cheat-sheet)

It is arguable whether this is correct or not, but we have to keep historical perspective in mind, since many Pinot users migrate from one of these engines.
Please note that, from native FST index's perspective, it is pretty simple to support java regex language. The reason I did not do that was because the existing FST index was based on Lucene and thus supported Lucene's regex dialect. For back compatibility, native FST index supports the same dialect.

Parsing And Reporting Invalid Regexes: I agree that in an ideal scenario, we should be parsing regexes and reporting characters that we do not accept. However, given the complexity of an average regex, the time complexity of doing this will be non trivial.

Specific To This PR: I looked into the PR and while the effort is appreciative, I do not see significant performance improvement. Also, the API surface changes required for this change are high, which makes me a bit uneasy.

Yeah, I completely understand that we should maintain the Lucene language when Natife FST was created. My concern is with the fact that when Lucene index was added, we decided to accept a different language in regexp_match. IMHO that is a consistency problem that is not easy to explain to Pinot customers. Pinot doesn't include documentation about this operation and how it changes depending on whether there is an index or not and I feel sorry for whoever has to write that doc, because it seems quite difficult to describe.
IMHO a better solution would be to do not optimize regexp_match with indexes and to have another operation whose string argument was a Lucene expression.
It is too late to do that, so let's focus on useful things. This PR is closed, but I'm going to cherry pick the like to regex optimization to add it in another PR.


[ModuleInterface] AliasModuleNames workaround support for indirect dependencies and missing imports

In AliasModuleNames, avoid wrongfully printing aliased names for modules that
were not aliased, just use the real name in that case. This can happen in the case of modules indirectly
imported via a reexport.
If there's a reference in API to a module that's not imported, the
import is inserted automatically in the swiftinterface. This PR ensures the
inserted import is correctly aliased in AliasModuleNames mode.

rdar://102262019

@swift-ci Please smoke test

@swift-ci Please smoke test macOS

@swift-ci Please smoke test


Huge latency increase with AWQ models at medium context lengths

Using an awq quantized model from thebloke (TheBloke/manticore-13b-chat-pyg-AWQ), generation is fine and starts after a few seconds with only a few sentences in the context window, but anything more than three or four makes it take ~30 seconds to start generation. Tried different awq models, and the issue doesn't happen with unquantized models. Is this normal/something to do with the way it handles awq models?

This is not normal behavior. You can go to AutoAWQ and try to generate using Huggingface methods to see how it normally works.


⚠ Redlib has degraded performance

In 0690791, Redlib (https://redlib.perennialte.ch) experienced degraded performance:

HTTP code: 200
Response time: 2273 ms

Resolved: Redlib performance has improved in 83e19c1 after 7 minutes.


Typo in IDynamicLinkCustomTypeProvider name

This might well be an annoying breaking change now, but just for future reference there might be a typo in the name of IDynamicLinkCustomTypeProvider. All the other concrete types use the correct spelling Linq while this one uses Link.
No complaints here, but just wanted to flag the inconsistency between type names.

@glopesdev Good catch ;-)
Best would be to add the correct class IDynamicLinqCustomTypeProvider and let the older interface extend this one in order to maintain backward compatibility.

https://github.com/zzzprojects/System.Linq.Dynamic.Core/pull/476

https://github.com/zzzprojects/System.Linq.Dynamic.Core/pull/476

Changed interface name will be included in next version.


Improve "Add emoji reaction" three-dot menu option

As discussed on CZO, we should make the following improvements to the "Add emoji reaction" message menu item:

[ ] Add the (:) keyboard shortcut
[ ] Do not show the option on other users' messages, where the button to add an emoji is available on hover.

@zulipbot claim


Update prop-types 15.6.0 to 15.7.2

Coverage remained the same at 100.0% when pulling 7103da098e3dc6ccd0196c11ca9a38ae43905c73 on petri-nc:update-prop-types-library into ff834d0667f437c9d0affcd0208c2b5ce09e92ac on ethanselzer:master.


Dynamically calculate ETH-FOX APR

Overview
We currently have an env var to set the ETH-FOX APR in the /defi/earn page
This can be calculated programatically, as per fox.shapeshift.com
References and additional details

https://github.com/shapeshift/foxfarm - APR calculation logic is in here somewhere

Acceptance Criteria

Don't rely on env var for this APR
Dynamically calculate APR via web3 call

Need By Date
n/a
Screenshots/Mockups
currently hardcoded

dynamically calculated

Estimated effort
2 hours

@gomesalexandre - can we knock this out with your plugin page as it will be the same code to calc APR?

@gomesalexandre - can we knock this out with your plugin page as it will be the same code to calc APR?

Will be the exact same info, but IMO we still want to keep this issue to consume it since currently it will be plugin-scoped.


Build pipeline

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: lh190330 sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.


PHP Universal Version

Hello,
is it possible to do a php universal version without a composer?

We won't be providing a none composer version unfortunately.
It's fairly trivial to get this working in your own project without using composer though.


New icon to distinguish normal commits from indexed commits (e.g. commits with rich navigation support)

Again for the Remote Repositories extension we need an icon to use to convey the difference between a normal commit and an indexed commit.

Added the following icons:

layers
layers-dot
layers-active
compass
compass-dot
compass-active

I integrated the new icons.


Errors in files that are not sent through webpack are not reported

Reported by @s-panferov
Simple case would be an error in a declaration file.

This has been fixed in v0.3.3


Strip zero width non joiners

Originally reported in html2text_ruby: https://github.com/soundasleep/html2text_ruby/pull/5
This test case needs to be copied over to html2text.

Done in 1.1 !


Unable to use Neuron Cores while fine-tuning BERT on Trainium

Hey!
I am trying to follow this guide: https://huggingface.co/docs/optimum-neuron/tutorials/fine_tune_bert and fine tune BERT on a trn1.2xlarge instance. I setup the datasets as mentioned in the blog and then ran the training script but the usage of neuron cores is still at 0%. The reason why this is relevant for me is because the expected training time for me is close to 5 hours.

cc: @philschmid

Thank you for reporting can. When did you create your environment? It seems that there is an error with the new AMI. Can you use the previous one?

I created the environment yesterday, using this AMI:  huggingface-neuron-2023-06-26T09-27-02.137Z-692efe1a-8d5c-4033-bcbc-5d99f2d4ae6a. I can try the previous one.

Trying huggingface-neuron-2023-04-20T11-02-28.279Z-692efe1a-8d5c-4033-bcbc-5d99f2d4ae6a

Ok that AMI works, thanks for your quick response!
I am trying to train a T5 model. Do you know if this AMI can be used to train a T5 model?

Thank you! We are working on fixing that ASAP!


to continue with convert error

to continue, in this case if we convert 3 Grams to Petagrams we get 3.0000000000000002e-15 and rounding this number leads to errors

Closed for the same reason as #555


change the securityContext as recommended by the SDK

The change is already present in the main branch. closing this one now.


Dropping Python 3.3 support

Currently the CI builds are failing on Python 3.3. While we could spend some time to get them working again, I'm questioning the value of supporting Python 3.3 at this point. We do still support Python 3.4. Though honestly am considering dropping that as well as soon as Python 3.6 comes out. Do people still use Python 3.3? Does this affect anyone? Feedback welcome.

The CI builds on Python 3.3 have been fixed. That being said, it is worth asking at some point how much effort we want to spend on Python 3.3 issues.

My general thinking is that by default we should try to be compatible with the versions supported by Jupyter (and adjust from there if compatibility is too difficult). Currently they support 3.3.


python311Packages.google-cloud-resource-manager: 1.12.1 -> 1.12.2

Changelog: https://github.com/googleapis/google-cloud-python/blob/google-cloud-resource-manager-v1.12.2/packages/google-cloud-resource-manager/CHANGELOG.md
Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 290848 run on x86_64-linux 1

2 packages built:

python311Packages.google-cloud-resource-manager
python311Packages.google-cloud-resource-manager.dist


similar struct to hat.T but for benchmarking

Going to close as it has been a long time and I'm not sure if this is relevant anymore.


The hide folder is show again

when I close the obsidian and then open it again，the hide folder is show again. thans

Duplicate of #12 , please check for similar issues before opening a new one, and use the bug report issue template.


Swipe from left gesture

As well as the custom shrink effect, Pinterest lets you swipe from left to go back. Is it possible to include this in PinterestSwift? Alternatively, could the custom transition be animated as the user swipes from the left?

To pop view controller by interactive gesture, we can override the other functions when implement UIViewControllerTransitioningDelegate protocol, looks like interactionControllerForPresentation  or something else, it seems more complex interactive transition will be available, you can try it whatever you like. :smiley:

Is there any code you could share for this?

@cannyboy
here is a repo to demonstrate a lot of transitions, interacted transition is included in. Good luck.


Cilium IPCache inconsistent for node IPs in cluster

General Information

Cilium version: 1.2.6 88f6431
Kernel version: 4.14.81-coreos

Background

The cluster had previously been running v1.3.0

Issue
On different nodes, the local node's IP is being mapped to host, while the other identities are mapped to CIDR identities:
One node:
$ ks exec -ti cilium-7994f cilium map get cilium_ipcache  | grep 172.0
<IP_ADDRESS>/32                14787   sync
<IP_ADDRESS>/32                42668   sync
<IP_ADDRESS>/32                 26307   sync
<IP_ADDRESS>/32                23705   sync
<IP_ADDRESS>/32                  17896   sync
<IP_ADDRESS>/32                47277   sync
<IP_ADDRESS>/32                 30143   sync
<IP_ADDRESS>/32                 1       sync
<IP_ADDRESS>/32                  1781    sync
<IP_ADDRESS>/32                61548   sync
<IP_ADDRESS>/32                 22976   sync
<IP_ADDRESS>/32                 47616   sync
<IP_ADDRESS>/32                 35707   sync

Another node:
$ ks exec -ti cilium-wttbz cilium map get cilium_ipcache  | grep 172.0
<IP_ADDRESS>/32                 26307   sync
<IP_ADDRESS>/32                  1781    sync
<IP_ADDRESS>/32                42668   sync
<IP_ADDRESS>/32                61548   sync
<IP_ADDRESS>/32                14787   sync
<IP_ADDRESS>/32                47277   sync
<IP_ADDRESS>/32                 22976   sync
<IP_ADDRESS>/32                 30143   sync
<IP_ADDRESS>/32                 47616   sync
<IP_ADDRESS>/32                 34232   sync
<IP_ADDRESS>/32                23705   sync
<IP_ADDRESS>/32                 1       sync
<IP_ADDRESS>/32                  17896   sync

Interestingly, there is actually a CIDR identity allocated for the local node's IP, it's just not propagated into the IPCache for the local node:
$ ks exec -ti cilium-wttbz cilium identity list | grep 172.0
1781    cidr:<IP_ADDRESS>/32
14787   cidr:<IP_ADDRESS>/32
17896   cidr:<IP_ADDRESS>/32
22976   cidr:<IP_ADDRESS>/32
23705   cidr:<IP_ADDRESS>/32
26307   cidr:<IP_ADDRESS>/32
30143   cidr:<IP_ADDRESS>/32
34232   cidr:<IP_ADDRESS>/32
35707   cidr:<IP_ADDRESS>/32
42668   cidr:<IP_ADDRESS>/32
47277   cidr:<IP_ADDRESS>/32
47616   cidr:<IP_ADDRESS>/32
61548   cidr:<IP_ADDRESS>/32

This has implications on policy.

Cilium v1.3.0 is not affected by this, but has the same output of the ipcache commands above.

For what it's worth, looking at the ipcache, all of the cilium_host IP addresses (typically 10.2.x.1/32 in the node's CIDR range) are being mapped to the host identity (1). However only the local node's physical (ie outward-facing) IP is considered host from an ipcache perspective. This means that traffic from local pods to node addresses may be considered using a CIDR identity or the host identity depending on where the pod is (and this is inconsistent across the cluster, with every node considering its own physical address as host).

One question I have is how we are affected during downgrade from 1.3.0 --> 1.2.6. This is because we removed the cluster identity in 1.3 (https://github.com/cilium/cilium/pull/5581). So, I believe that all nodes in the cluster are allocated their own CIDR identity if traffic is to be allowed to them by policy. However, during downgrade to v1.2.6, the cluster identity is reintroduced. But, the identities from when 1.3.0 was running are still in the key-value store, which causes confusion in the Cilium agent because the same IP is mapping to multiple identities(?). This is all conjecture, so I could very well be wrong in my assumptions.

@joestringer I don't fully understand what is the problem here, you were expecting <IP_ADDRESS>/32 to have identity 1 on both nodes?

@aanm If someone writes an egress CIDR policy that allows sending packets to each node in the cluster, then each node can access any other node but not the current node, even if the address of the local node is included in the CIDR policy. This is because the local node's IP is mapped to host identity.
On the flip side, if someone writes an egress policy to allow "to host", then it only allows sending traffic to the local node.
When you compare this to ingress, I believe that traffic from any node in the cluster to a pod on any node is considered as coming "from host", "to pod" identity.

@joestringer now I'm even more lost, I thought by default we were always allowing traffic to/from host.

@aanm IIRC we only allow ingress traffic from the host by default, not egress.

@ianvernon bingo.
When the policy is working as we'd like it to, here's the policymap:
# cilium bpf policy get 33106 -n | grep " 1 "
Ingress     1          ANY          NONE         8233254    5592
Egress      1          ANY          NONE         3165530    6016

When it's not working as intended:
# cilium bpf policy get 33106 -n | grep " 1 "
Ingress     1          ANY          NONE         14313798   10506
Ingress     1          9090/TCP     NONE         370331     4294
Egress      1          9404/TCP     NONE         0          0
Egress      1          9091/TCP     NONE         0          0

@joestringer I still don't fully grasp what exactly the traffic is that should be allowed. Is the goal to allow traffic to all hosts within the cluster on a certain port, not just the localhost?

@joestringer ok, thanks.
The next question I have is, does this affect 1.3, or is this only an issue in downgrade from 1.3 --> 1.2.x?

@joestringer should we look into this still ? Or do you think we can close ?

Removing backport labels for the following release branches / reasons:

v1.3 is going to be EOL'd with the release of v1.6
v1.4 is moving to a security-fix-only backport model

I am going to nominate this for v1.6 backport so we can track this to see if it still is a problem. I will not nominate this for v1.5 backport because I'd like to reduce the churn in release branches before the most recent release (e.g., 1.6 is the most recent branch, do not backport such fixes to v1.5).


Samplesheet multilevel nesting ick4

This branch adds an option to choose which files to create the samplesheet (and thus, run the pipeline) on.  It defaults to only running the pipeline on fastq files nested in subdirectories, the default CDC Core format.  But you can instead choose to run the pipeline on all the files (top-nested and one-dir-under nested) or only the top-level files.
Test cmd:
nextflow run main.nf --indir /scicomp/groups-pure/Projects/scbs_mpob/JRowell_ick4/mpox_testing/paired/ --file_levels <nested, all, top> --outdir /scicomp/groups-pure/Projects/scbs_mpob/JRowell_ick4/mpox_testing/results/ --project_name test -config /scicomp/reference/nextflow/configs/cdc-dev.config -profile singularity,rosalind

PR checklist

[x] This comment contains a description of changes (with reason).
[x] If you've fixed a bug or added code that should be tested, add tests!

[ ] If you've added a new tool - have you followed the pipeline conventions in the contribution docs
[ ] If necessary, also make a PR on the nf-core/mpxvassembly branch on the nf-core/test-datasets repository.

[ ] Make sure your code lints (nf-core lint).
[x] Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
[x] Usage Documentation in docs/usage.md is updated.
[x] Output Documentation in docs/output.md is updated.
[ ] CHANGELOG.md is updated.
[x] README.md is updated (including new tool citations and authors/contributors).

Please remove option --file_levels all because it utilizes inconsistent sample name derivation and reinforces poor data organization practices. Users should have option to define:
top -- look for fastq files in listed directory only
nested -- look for fastq files organized in subdirs one level down
in either scenario, ignore any other fastq files present in the directory tree passed to --indir.

I have removed the --file_levels all option and updated the documentation to reflect this.

This needs some param validation because running with --file_levels all attempts to execute but returns warning:
NOTE: Process POLKAPOX_ASSEMBLY:POLKAPOX:CREATE_SAMPLESHEET terminated with an error exit status (2) -- Error is ignored

I've enforced the valid values for --file_levels in nextflow


Power Rename crashing Explorer

Microsoft PowerToys version
0.43.0
Running as admin

[X] Yes

Area(s) with issue?
PowerRename
Steps to reproduce
Select several files
Enter rename parameters (using RegEx)
Rename
✔ Expected Behavior
Files renamed
❌ Actual Behavior
Before opening rename form, a popup window showing files being enumerated (There are only 18 files in the directory); message is "Wait while the selected items are enumerated.")
Sometimes:
Task bar icon shows moving green bar filling the "PowerRename" icon continually until I click Cancel
Most times The window disappears by itself after a few seconds
Rename form opens.
Parameters entered
Click Rename
Popup form disappears
Sometimes:
Task bar icon shows moving green bar filling the "PowerRename" icon continually until I click in Explorer window
Mostly:
PowerRename window disappears
Always:
All open Explorer instances (including the desktop) crash, the PowerRename icon disappears from the task bar.
On restarting Exporer and locating the directory, I see the files have been renamed.
Application Event log contains 4 error messages:
Application: explorer.exe Framework Version: v4.0.30319 Description: The process was terminated due to an unhandled exception. Exception Info: exception code c0000005, exception address 00007FFE07CC67BF
Faulting application name: explorer.exe, version: 10.0.19041.1151, time stamp: 0x2885d2b8 Faulting module name: PowerRenameExt.dll, version: <IP_ADDRESS>, time stamp: 0x6108753e Exception code: 0xc0000005 Fault offset: 0x00000000000367bf Faulting process ID: 0x5a0c Faulting application start time: 0x01d794d7afca6552 Faulting application path: C:\WINDOWS\explorer.exe Faulting module path: C:\Program Files\PowerToys\modules\PowerRename\PowerRenameExt.dll Report ID: 4209e2ad-eb8b-448e-9406-7d304d8a140c Faulting package full name:  Faulting package-relative application ID:
Fault bucket , type 0 Event Name: APPCRASH Response: Not available Cab Id: 0
The shell stopped unexpectedly and explorer.exe was restarted.
Other Software
Win10 20H2 (19042.1165), fully patched.

Failed to repro this on my computer.
@ggrinton Could you attach a bug report file?
Right-click on the PowerToys icon in the tray menu and select Report Bug

Drag and drop report into a GitHub comment.

Thanks For the quick response.
I have repeated the exercise (same result) and attach the Report file.
I also did a screen capture. To make it small enough to load I edited it to keep only key moments - it's a bit clunky but may help you see what is going on.
https://user-images.githubusercontent.com/33444245/130062847-617b8c55-9d70-4c2a-8acd-1cf1f9019340.mp4
PowerToysReport_2021-08-19-21-26-04.zip

Not sure of the cause, but I see the progress window does not disappear when it's done. There have been one or two fixed for this very recent, maybe those are the solutions? Also, I couldn't see if the full path of the filenames are very long, so a small possibility #8748 is related?

@Jay-o-Way  I've the path is C:\Users\ggrin\Desktop\New folder\ according to the video.

It's possible that this will help once released: https://github.com/microsoft/PowerToys/pull/12771

After moving to Windows 11 a three weeks ago PowerRename has not crashed once, whereas on 10 it would crash daily. I didn't apply a PowerToys update around the same time to explain this.

This is one of those hard to replicate bugs. Thank you for the feedback @Superenigmatix

@ggrinton version 0.51 has just been released. Can you try and update and see if there are still issues?

I have not had recurrences of this issue for some time. Happy to have it closed.


win-12 low on disk space for local Temp directory.

Type of Request
bug
Infrastructure Environment
LUCI, Github, Cocoon scheduler, Autosubmit, etc...
What is happening?
win-12 low on disk space for local Temp directory.

Steps to reproduce
Step 1:
Step 2:
..
Step n:
Expected results
I expect to see X when Y is finished.

Removing old directories in C:\users\swarming\AppData\Local\Temp\ to free up disk space.


Login/password doesn't fill in if form.method is 'get'

Check on https://www.instagram.com
It skips login form because of this check:
end.js
if(form.method.toLowerCase() === "get") {
	continue;
}

but it should not.

@issoft-nsiniakevich Thank you for your report! I will take a look at this and probably remove this check.


Cop idea: Ranges or between

Description
Use ranges or Comparable#between? instead of complex comparison logic when possible.
https://rubystyle.guide/#ranges-or-between
Behavior
# bad
do_something if x >= 1000 && x <= 2000

# good
do_something if x.between?(1000, 2000)

https://github.com/rubocop/rubocop-rspec/issues/1684

The expression x >= min && x <= max is a very popular convention and is also performance:
#!/usr/local/bin/ruby

require 'benchmark/ips'

array = (0...100).to_a

Benchmark.ips do |x|
  value = 42

  x.report('>= && <=') { value >= 1000 && value <= 2000 }
  x.report('include?') { (1000..2000).include?(value) }
  x.report('between?') { value.between?(1000, 2000) }

  x.compare!
end

$ ruby -v
ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-darwin19]

$ ruby bench.rb
Warming up --------------------------------------
            >= && <=     2.351M i/100ms
            include?     1.139M i/100ms
            between?     1.244M i/100ms
Calculating -------------------------------------
            >= && <=     23.476M (± 1.3%) i/s -    117.558M in   5.008408s
            include?     11.117M (± 8.2%) i/s -     55.807M in   5.057945s
            between?     11.021M (± 6.8%) i/s -     55.959M in   5.102033s

Comparison:
            >= && <=: 23475928.0 i/s
            include?: 11117073.1 i/s - 2.11x  slower
            between?: 11020978.6 i/s - 2.13x  slower

It might be worth considering its validity in the described style guide.

Discussing the trade-off between idiom and speed in the style guide seems like the better first step.

https://github.com/fastruby/fast-ruby/issues/225


Bug in region_psite()

Hi Fabio,
I wanted to plot the p-site distribution across different mRNA regions and found that the current version of region_psite() bugs out with the following error:
Error in as.character(transcript) %in% c_transcript :
object 'transcript' not found``
Best

Hi René.
I just run some tests using the datasets provided in the package and other datasets and I got no errors.
The problem here seems to be related to the column names of the data.table passed either through data or through annotation. I guess data is the one to check out since annotation is used at the beginning of the function and also in that case the column "transcript" is involved. Do your data come from the psite_info function? If so, could you please check the name of the first column of each data.table of the list?

Never mind, I found the issue after some testing. The problem was that the naming of data did not match the provided sample, the function works as intended.
Thanks for the super quick response though!


Possibility to delete a user profile

It should be possible to delete your own profile on the 'user profiel' page. In order to do this we need to make an API endpoint for this.

We hebben besloten niets te doen met gebruikersaccounts binnen de tool.


Test scripts in bin by using --help

Take a look at this ( http://conda.pydata.org/docs/bdist_conda.html#setup-options ).


Does this library support continuous pomdp problems?

Seems the examples in the documentation are mostly discrete?
Many thanks!

The base classes are general enough to allow defining continuous POMDPs (see light dark example), but currently there is no solver for them -- I just added an item for this in call for contributions!
If your continuous domain has a discrete action space, and you should be able to still apply POUCT but it may not work well (flat tree, for example).


Create Blog “2023-05-19-new-blog-on-friday”

Automatically generated by Netlify CMS

👷 Deploy Preview for spectacular-sopapillas-3c0fe8 processing.

Name
Link

🔨 Latest commit
3f3cff79f8621e5326dc76e4dacb9a2b25e2d1a9

🔍 Latest deploy log
https://app.netlify.com/sites/spectacular-sopapillas-3c0fe8/deploys/6466d9b856867b00086ca406


Minimize and Maximize Visible during runtime

What steps will reproduce this issue?
Upgrading from version 1.3.0-ALPHA139 to ALPHA47
When in full screen mode, the Minimize and Maximize buttons are visible in the TitleBar
Expected outcome
The Minimize and Mazimize should not be visible
I am using the following which was giving the desired results:
WindowStartupLocation="CenterScreen"
ResizeMode="NoResize"
ShowTitleBar="False"
ShowCloseButton="False"
WindowStyle="None"
--
Environment

MahApps.Metro 1.3.0-ALPHA147
Windows 10
Visual Studio 2013
.NET Framework 4.6.1

@akhalid should be fixed with 143168e382764507f49a6a4ea41928e238c71fe1

Thank you very much!


Deno linter errors not prettyfied

I had an issue while using deno and deno linter (emits deno-ts errors)

No problem with regular ts errors in node. Maybe some options should be swithced on?
Spec:
deno 1.32.4 (release, x86_64-pc-windows-msvc)
v8 <IP_ADDRESS>
typescript 5.0.3
vscode 1.77.3
Win 11

fixed by reupdating vscode


Error while using the syntax table.columnName for the field argument of the countIf measure

To Reproduce

Given you have to table named recommendation with the following columns: (simulation, final_price)
Execute a query similar to the following one

  const query = from("recommendation")
    .select(
      [],
      [],
      [
        countIf(
          "myCountIf",
          "recommendation.simulation",
          criterion("recommendation.final_price", gt(0))
        ),
      ]
    )
    .build();

You will get an error in the squashql application similar to
2023-04-05T15:09:58.640+02:00 ERROR 13532 --- [nio-5000-exec-7] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed: java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.clickhouse.client.ClickHouseException: Code: 47. DB::Exception: Missing columns: 'recommendation.simulation' while processing query: 'SELECT count(if(final_price > 0, `recommendation.simulation`, NULL)) AS myCountIf, count() AS _contributors_count_ FROM recommendation LIMIT 10000', required columns: 'final_price' 'recommendation.simulation', maybe you meant: 'final_price'. (UNKNOWN_IDENTIFIER) (version <IP_ADDRESS> (official build))
, server ClickHouseNode [uri=http://localhost:8123/default]@1228268188] with root cause

java.io.IOException: Code: 47. DB::Exception: Missing columns: 'recommendation.simulation' while processing query: 'SELECT count(if(final_price > 0, `recommendation.simulation`, NULL)) AS myCountIf, count() AS _contributors_count_ FROM recommendation LIMIT 10000', required columns: 'final_price' 'recommendation.simulation', maybe you meant: 'final_price'. (UNKNOWN_IDENTIFIER) (version <IP_ADDRESS> (official build))

	at com.clickhouse.client.http.HttpUrlConnectionImpl.checkResponse(HttpUrlConnectionImpl.java:194) ~[clickhouse-jdbc-0.3.2-patch11-all.jar:clickhouse-jdbc 0.3.2-patch11 (revision: 27f8951)]
	at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:289) ~[clickhouse-jdbc-0.3.2-patch11-all.jar:clickhouse-jdbc 0.3.2-patch11 (revision: 27f8951)]
	at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:123) ~[clickhouse-jdbc-0.3.2-patch11-all.jar:clickhouse-jdbc 0.3.2-patch11 (revision: 27f8951)]
	at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:156) ~[clickhouse-jdbc-0.3.2-patch11-all.jar:clickhouse-jdbc 0.3.2-patch11 (revision: 27f8951)]
	at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:289) ~[clickhouse-jdbc-0.3.2-patch11-all.jar:clickhouse-jdbc 0.3.2-patch11 (revision: 27f8951)]

Affected version
1.0.0
Database
Clickhouse 22.12
When I execute the following query it works
  const query = from("recommendation")
    .select(
      [],
      [],
      [
        countIf(
          "myCountIf",
          "simulation",
          criterion("recommendation.final_price", gt(0))
        ),
      ]
    )
    .build();

I think it is fixed by #147. Reopen the issue if it is still there.


Adding test cases for OAuth 2.0 DCRM create, read and delete requests

Resent the same improvement with https://github.com/wso2/product-is/pull/1363. Hence closing the PR


Multiple Explorers

I know this might be a far-fetched idea, but for me personally it would be very useful to have multiple project explorers side-by-side. I.e. use case: I have a subdirectory in my project root dir that contains results (logs) of each run of  my program. A new log file is generated at each run, so this folder quickly gets populated with 100+ files and it becomes difficult to scroll up and down. Another use case is when node_modules folder is open with its 1000+ packages. It'd be practical to have one explorer open for the project as now, and another on its right side to browse other folders.

This is a fair feature request, however we do not have it on our immediate plan. Thus assigning to backlog for now. Thanks


Fixed typo. j doesn't exist. The variable is p.

The path library has been imported as p not j.

Indeed, thanks :+1:


Feature/juan/integration testing

ISSUE
#83
SUMMARY
Add instrumentation tests to the app. Despite FavoriteCharacterViewModelIntegrationTest and CharacterDetailViewModelIntegrationTest are both not necessary, they were added just for practice purposes

For integration tests I would consider providing an in-memory roomdatabase with: Room.inMemoryDatabaseBuilder.

Hello @joseantoniocampillo but this approach uses fakes instead of real data, so that's why I don't use it


Travis

Add BDD on Travis and pass the build.

IBM Cloud toolchain: Delivery Pipeline deployed wishlists to dev, including commit a0413e5ca7f53d6a8ed65ce7bd31e7db93dd53f3


Refactoring for Merge statement

This PR does not have a test case as Merge statement is underdeveloped on calcite end and even the Rel created is a custom Rel for DM.


pdfstudio / pdfstudioviewer: 2021.2.0 -> 2022.x.x

Checklist

[X] Checked the nixpkgs master branch

[X] Checked the nixpkgs pull requests

Project name
nix search name: pdfstudio
nix search name: pdfstudioviewer

current version: 2021.2.0
desired version: 2022.x.x
Notify maintainers

maintainers: @PhilippWoelfel
Note for maintainers
Please tag this issue in your PR.

@PhilippWoelfel  Just purchased license for this product but can't use it as it's a 2022 license. Thanks for your help!


Need clear instructions on installing from scratch

Hi,
I'm having trouble installing oncall in our linux environment using the instructions given.
Here is one of the errors while running "python setup.py develop" :
I have a venv set up with python 3.6.3
Processing dependencies for oncall==1.1.0                  Searching for slackclient                                  Reading https://pypi.python.org/simple/slackclient/        Downloading https://files.pythonhosted.org/packages/1f/9e/dd5bb327786be753e5d8de976082eb714cb83abeb00f67f0646ba2e166ad/slackclient-1.2.1.tar.gz#sha256=e9de0c893e8c5473107f5927ae1e543d35246f0c834f5e86470b22b762211577                error: Download error for https://files.pythonhosted.org/packages/1f/9e/dd5bb327786be753e5d8de976082eb714cb83abeb00f67f0646ba2e166ad/slackclient-1.2.1.tar.gz#sha256=e9de0c893e8c5473107f5927ae1e543d35246f0c834f5e86470b22b762211577: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)

Can we have clear step by step instructions to download and install oncall for both development and separately for production use ? Please note that not everyone uses Docker.

Is it possible to package oncall so we can start it easily ? i.e. minimum run. For example in a VM or as a single zip file ?

Thanks

This appears to be a problem with SSL verification on files.pythonhosted.org; probably transient. We have installation instructions for development in our Readme, and Docker for quick setup. Support for services other than Docker is something we'd like to have in the future, and if you'd like to contribute an integration for k8s, AWS, or something else, that would be awesome.

Support for services other than Docker is something we'd like to have in the future

I meant simple install and start for bare-metal hosts and VMs.
Also some instructions for running a production instance. All the info seems to be development oriented.

@foequeue Go for it!  I doubt anyone would shred it.  I'm definitely not a programmer, but have had a couple of minor changes committed with absolutely no shredding.

Yes, it would be good to have some explanation of the installation process. Also integration with AD/LDAP.

Yes, it would be good to have some explanation of the installation process. Also integration with AD/LDAP.

Did you find any explanation? please share with me if you have.

Yes, it would be good to have some explanation of the installation process. Also integration with AD/LDAP.

Did you find any explanation? please share with me if you have.

Unfortunately, we didn't get any method and so we  built our own internal app in the end.


Add relations-api data sync dashboard configmap.

PR Template:
Describe your changes

Add configmap for Rbac -> relations-api data sync grafana dashboard to be references in app-interface.

Ticket reference (if applicable)
Fixes #RHCLOUD-36275.
Checklist

[x] Are the agreed upon acceptance criteria fulfilled?

[ ] Was the 4-eye-principle applied? (async PR review, pairing, ensembling)

[x] Do your changes have passing automated tests and sufficient observability?

[x] Are the work steps you introduced repeatable by others, either through automation or documentation?

[ ] If automation is possible but not done due to other constraints, a ticket to the tech debt sprint is added
[ ] An SOP (Standard Operating Procedure) was created

[ ] The Changes were automatically built, tested, and  - if needed, behind a feature flag - deployed to our production environment. (Please check this when the new deployment is done and you could verify it.)

[ ] Are the agreed upon coding/architectural practices applied?

[ ] Are security needs fullfilled? (e.g. no internal URL)

[x] Is the corresponding Ticket in the right state? (should be on "review" now, put to done when this change made it to production)

[ ] For changes to the public API / code dependencies: Was the whole team (or a sufficient amount of ppl) able to review?

/lgtm


Add node/npm implementation

Codecov Report

Merging #8 (bb872e1) into main (4a94c50) will decrease coverage by 24.22%.
The diff coverage is 16.66%.

@@             Coverage Diff             @@
##             main       #8       +/-   ##
===========================================
- Coverage   95.65%   71.42%   -24.23%
===========================================
  Files           3        4        +1
  Lines          23       28        +5
===========================================
- Hits           22       20        -2
- Misses          1        8        +7

Flag
Coverage Δ

unittests
71.42% <16.66%> (-24.23%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

python/src/lib.rs
60.00% <0.00%> (-30.00%)
:arrow_down:

npm/src/lib.rs
20.00% <20.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4a94c50...bb872e1. Read the comment docs.

This should be provided with a separated repository instead monorepo. napi-rs generates GitHub Actions scripts to run on project root directory.
I tried very long time to fix paths to match. But it will take more time to work fine.


The lich changes are not appearing in the UMM UI

I believe it is simply missing a line in the UI file

This will be fixed in the next release, which should be with WotR 1.3.0


fioctl: 0.35 -> 0.38

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/foundriesio/fioctl/releases.
meta.description for fioctl is: A simple CLI to manage your Foundries Factory
meta.homepage for fioctl is: https://github.com/foundriesio/fioctl
Updates performed

Golang update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS
Warning: a test defined in passthru.tests did not pass
found 0.38 with grep in /nix/store/dbmd5vx95h7l8s9fxggzhj2kdx1a5ajj-fioctl-0.38
found 0.38 in filename of file in /nix/store/dbmd5vx95h7l8s9fxggzhj2kdx1a5ajj-fioctl-0.38

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

First fifty rebuilds by attrpath
fioctl

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/dbmd5vx95h7l8s9fxggzhj2kdx1a5ajj-fioctl-0.38 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A fioctl https://github.com/r-ryantm/nixpkgs/archive/af4194c742842c9f76b66f50fb55e3b70851481a.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/af4194c742842c9f76b66f50fb55e3b70851481a#fioctl

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/dbmd5vx95h7l8s9fxggzhj2kdx1a5ajj-fioctl-0.38
ls -la /nix/store/dbmd5vx95h7l8s9fxggzhj2kdx1a5ajj-fioctl-0.38/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

fioctl

Maintainer pings
cc @nixinator @MatthewCroughan for testing.

LGTM


added section title support above touchable element

Added section title support.

Could you describe a use case where this would be useful?

Sure. If I want to dynamically create collapsible sections with section titles this is a nice feature.

No way to make sections have there own titles without it being in the touchable element. I could wrap it in a higher order but this is a clean solution that others I think can benefit from.

Alright, makes sense. Can you rebase this PR from the next branch

No problem will do now. @iRoachie

@iRoachie There we go.

when can this be merged?


Nested USB redirection using RemoteFX

Hello,
Not sure anybody will answer my question here but let's try.
Running two VRA (VMWare) hosted VMs, both VMs run latest "Windows 10 Enterprise LTSC 2019".
I connect from my local pysical machine (runs a WIN10 v1909) w/ RemoteFX enabled into first VM.
There I can indeed redirect USB devices as expected.
I now want to further redirect my USB devices from first VM into a second VM.
So far I managed to enable RemoteFX on all machines (my local host and both VMs).
But I only can redirect into first level VM my USB devices using RemoteFX, further nested level VMs is not working - list of devices RemoteFX would present normally remains empty.
Question: are there any GPOs or registry hacks I could apply to enable such scenario (I indeed am needing)? Or is this impossible? Thanks Christophe

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 341357d9-570c-043d-489b-74701a61c55c
Version Independent ID: dc864602-6186-c5d7-f5b8-852aec46fb58
Content: Explaining Microsoft RemoteFX
Content Source: virtualization/community/team-blog/2010/20100317-explaining-microsoft-remotefx.md
GitHub Login: @scooley

Hello, @coco21
Thank you for your feedback.
We are actively reviewing your comments and will get back to you soon.
Kind regards,
Microsoft DOCS International Team

Hi MS Team,
Great for getting so quickly news, thank you!
To ensure we perfectly understand each other please check this URL:
https://www.thewindowsclub.com/use-devices-resources-remote-desktop-session
There it's said under "Major Consideration" section (was not able to verify source of this statement):
Plug and Play device redirection and RemoteFX USB device redirection are not supported over cascaded Remote Desktop connections. This means that when you’re connected remotely to one computer, and from within that session you connect to another computer, the second connection is cascaded. For example, you can redirect and use a Plug and Play device attached to your local computer when you connect to a remote computer. However, if you connect to a second remote computer from the first one, you can’t redirect and use the Plug and Play device with the second computer.
Hope this turns out not being true. A way out of this may be Hyper-V virtual bus. But then I may need some support to perform this correctly.
Good then, I wait on your input here.
Cheers Christophe

Hello teams,
Did anybody got a chance to have a look at this?
Thank you & regards,
Christophe

We hugely appreciate your contribution, feedback, and support to our technical documentation.
We’ve phased out GitHub Issues on this repository as the feedback mechanism for content and have replaced it with a new feedback system. For more information see, Content User Feedback. As part of this rollout, we are migrating your issue where our team will use your comments to improve the documentation experience. Unfortunately, this does mean you won't be able to access this issue after the migration is complete.
All issues will be migrated by Thursday 29th August 2024.
Once again, thank you for your valuable feedback, we hope you’ll continue to contribute using our new feedback system.


Discharge color data frame

Three columns:
site, timestamp (index....), discharge class

lat lon too?

that would already be in the shapefile....I'm on this one, nearly done, ish

is this ready to go?

Looks like there are more sites in the classification than there are in the set of sites

oh, more like multiple values for the same site and the same timestep


Does not take new target groups into use

On version 1.0-alpha.3, starting from a simple ingress with path / pointed to a backend. Edit the ingress to add a second path rule and point it to a different backend. The controller creates a new target group with the correct instances and port specified, and creates a new path rule. The new rule points to the target group of the original rule, and not to the new target group.
If I delete the new rule and let the controller recreate it I see the expected result of the new rule pointing at the new target group.
I haven't tested the latest version from git, so this may have been fixed since 1.0-alpha.3.

After some more debugging, it seems that the point is that the controller waits for the target group to enter state "healthy" before taking it into use. This is a chickend and egg problem, since it remains in state "unused" until it's enabled somewhere..

Same issue here! New target group gets created but ALB rules are never updated to use it...!

This should be resolved in the latest alphas 1.0-alpha.9


Added code for ways to create max heap

No of Different ways to construct a max heap from n distinct integers.

Added Code for Find no of Different ways to construct a max heap from n distinct integers.

Added Heap Solution
Discarded Postfix solution as already existing.
Please fetch upstream to be even with main repo.
Congo on First PR


[XR] main camera's projection matrix

Please first wait before merging, waiting for @ryantrem (#8944)

LOL I misclicked!

Looks like this should solve the problem in our case. I will test it out once we get a new @babylonjs/core package.

Just published beta 2


Feature request: unbuffered text matching

(Probably blocked on #656 restructuring, and potentially better exposed as part of regex-automata.)
TL;DR:
In the advanced/"lower level" impl section, or perhaps on applicable Regex types provided by regex-automata:
fn Regex::is_match_streaming<O: Display>(&self, text: O) -> bool;

pub fn is_match_streaming<O: Display>(&self, text: O) -> bool
Returns true if and only if there is a match for the regex in the display representation of the object given.
If you have or can get &str for the text to match, you should use is_match instead. This method is potentially useful to reduce allocations, as you don't have to first write the object's display representation into a buffer to match it.
However, this can lead to less performant regex execution. Normal regex execution does multiple optimizations that rely on having the full text available to scan, which can't be done in a streaming fashion. Instead, this method has to always run the most general regex VM byte-by-byte over the input.
To test against streaming text which is not solely a single object's Display impl, use format_args! to manufacture a Display impl with the text you want to match.
Examples
// Fictional people from a name generator.
// Any similarity to actual persons, living or dead, is purely coincidental.
let tiffany = "Ms Tiffany Northman";
let raymond = "Dr Raymond Hyde";
let regex = Regex::new("(?i)tiffany .* loves .* raymond").unwrap();
assert!(regex.is_match_streaming(format_args!("{tiffany} loves {raymond}")));

Motivation
TL;DR: the matchers crate, as used in tracing-subscriber.

The regex crate implements regular expression matching on strings and byte arrays. However, in order to match the output of implementations of fmt::Debug and fmt::Display, or by any code which writes to an instance of fmt::Write or io::Write, it is necessary to first allocate a buffer, write to that buffer, and then match the buffer against a regex.
In cases where it is not necessary to extract substrings, but only to test whether or not output matches a regex, it is not strictly necessary to allocate and write this output to a buffer. This crate provides a simple interface on top of the lower-level regex-automata library that implements fmt::Write and io::Write for regex patterns. This may be used to test whether streaming output matches a pattern without buffering that output.
Users who need to extract substrings based on a pattern or who already have buffered data should probably use the regex crate instead.

tracing spans/events structured logging facilities capture non-primitive types by their Debug formatting. It's expected for such formatting to be relatively cheap — cheaper to do multiple times than to allocate a buffer and do so a singular time, anyway. In most cases, the captured values will only be formatted once, into a large buffer used for recording the event, with all recorded fields printing into the same buffer.
Where the regex comes in is filtering. Filters can observe the recorded field, and filter spans/events in/out depending on the value of the field, and this is primarily done via regex matching. When doing this filtering, tracing-subscriber wants to avoid allocating a buffer for the fields to test, just to throw it away afterwards. This is done by writing the formatting machinery directly into driving a regex-automata DFA byte-by-byte.
tracing-subscriber filters are not supposed to be resilient to untrusted construction, but using the regex crate instead of regex-automata would reduce the failure case from regex compilation taking near-unbounded CPU/RAM to filter application taking O(n) time on the length of the filter string.
Implementation sketch
For simplicity's sake, I'm providing an implementation sketch using regex_automata@0.2.0::hybrid::dfa::DFA. This implementation is derived from matchers', which is MIT licensed[^1].
[^1]: I believe the code as presented here is simple/axiomatic enough that it is uncopyrightable. As much as is permissible by law, I release copyright on my modifications to the code, to the detriment of my heirs. (My modifications can be considered licensed under the Unlicense.)
Implementation sketch
impl DFA {
    fn is_match_streaming(
        &self,
        cache: &mut Cache,
        bytes: impl Display,
    ) -> Result<bool, CacheError> {
        let mut matcher = StreamingMatcher {
            dfa: self,
            // NB: we need a start_state method that doesn't take the
            //     text as input, like the one in<EMAIL_ADDRESS>            state: self.start_state_anchored(cache, None)?,
            cache: cache,
        };
        use io::Write;
        write!(&mut matcher, "{}", bytes).expect("infallible Write impl");
        matcher.finish()
    }
}

struct StreamingMatcher<'a> {
    dfa: &'a DFA,
    state: Result<LazyStateID, CacheError>,
    cache: &'a mut Cache,
}

impl<'a> io::Write for StreamingMatcher<'a> {
    fn write(&mut self, bytes: &[u8]) -> io::Result<usize> {
        let mut i = 0;
        for &byte in bytes {
            self.advance(byte);
            i += 1;
            if self.is_dead() {
                break;
            }
        }
        Ok(i)
    }
}

impl StreamingMatcher<'_> {
    fn advance(&mut self, input: u8) {
        let Ok(state) = self.state else { return; }
        // TODO: use next_state_untagged/next_state_untagged_unchecked
        // TODO: replace CacheError with MatchError::GaveUp
        self.state = self.automaton.next_state(self.cache, state, input);
    }

    fn is_dead(&self) -> bool {
        // TODO: handle quit state
        match self.state {
            Ok(state) => state.is_dead(),
            Err(_) => true,
        }
    }

    fn finish(self) -> Result<bool, MatchError> {
        let state = self.state?;
        let state = self.dfa.next_eoi_state(self.cache, state)?;
        Ok(state.is_match())
    }
}

(I do not know how such an implementation would handle unicode word boundaries, as AIUI those are handled by annoying special cases and not the bytewise (hybrid) DFA.)
By my quick scanning through the regex implementation, the pikeVM is always driven byte-by-byte, and could be driven in a streaming fashion this same way.

This looks like a special case of #425 to me. And yes, I would expect that this would show up in regex-automata. Or rather, I think what I would prefer for the time being is that folks would develop these sorts of matching routines by writing their own regex engines. It sounds a little crazy maybe, but that's actually one of my goals with regex-automata: it should be possible to use the finite state machines to write your own search implementation using those finite state machines. (Just like you did above.)

// NB: we need a start_state method that doesn't take the
//     text as input, like the one in<EMAIL_ADDRESS>state: self.start_state_anchored(cache, None)?,

Hmmm, yes, indeed. The start state computation does require doing look-behind (for start anchors and one half of the word boundary assertion). But this is at most one byte and it only happens when start > 0. When start == 0 (which I would imagine is the case for this problem), then bytes is never accessed. So I do believe that we can provide a more elementary method that would work here. The start_state_reverse case is harder, because the check is no longer start == 0, but rather, end == bytes.len(). But, if you're doing a reverse search, then it's likely you have the buffer in the first place anyway.
I do in part think that #425 subsumes this, but I would like to keep this open since there is a very clear and actionable use case that drives things here. It's also a bit simpler than the general case.

(I do not know how such an implementation would handle unicode word boundaries, as AIUI those are handled by annoying special cases and not the bytewise (hybrid) DFA.)
By my quick scanning through the regex implementation, the pikeVM is always driven byte-by-byte, and could be driven in a streaming fashion this same way.

Right, the DFAs cannot and probably will never be able to handle Unicode word boundaries. You can use its heuristic support for it, and if the search quits, you could in theory give up and switch over to the PikeVM. But... doing this without a buffer seems tricky/impossible. So the heuristic might not be possible. It seems likely then that you'll need to do:
if nfa.has_unicode_word_boundary() {
    // do streaming version of PikeVM
} else {
    // do streaming version of lazy DFA
}

The PikeVM has its own problems with streaming, regrettably. For example, it will try to look up to 4 bytes behind and up to 4 bytes ahead in order to deal with Unicode word boundaries. (Up to 1 byte behind or ahead for other types of assertions, currently.)
So there are problems to be worked out... One simple way forward for y'all would be to do this instead:
if nfa.has_unicode_word_boundary() {
    // copy string to buffer, run normal regex
} else {
    // use lazy DFA
}

And that might be good enough. Unless, of course, you find that most queries in practice really want Unicode word boundaries.
Although, it seems likely that you are currently blocked by the start_state_method. But there is a hack for you. If you are indeed always starting the search with start == 0, then you should be able to call start_state_forward(cache, None, &[], 0, 0) and that will ultimately do what you want. I can't guarantee that it will always work, but it should work today.
Finally, I feel compelled to remind you of the warning at the top of the regex-automata 0.2 docs. :-) https://docs.rs/regex-automata/0.2.0/regex_automata/index.html

Finally, I feel compelled to remind you of the warning at the top of the regex-automata 0.2 docs. :-)

matchers currently uses<EMAIL_ADDRESS>so yeah, this is more a "nice future" kind of feature request, nothing's really blocked. :-)
Once<EMAIL_ADDRESS>is more cooked, it can implement these patterns on top of the building blocks provided.
(I'd like to remove the (?:){HUGE} footgun, but filters are currently assumed to be trusted, so it's low/"nice future" priority.)

Aye. And to be clear, I hope there are no more 0.2.x releases. If things go my way, the next release will be 0.3.0 and that will roughly coincide with: 1) moving regex-automata into this repo and 2) building regex on top of regex-automata. Those are my victory conditions. :-)

(I'd like to remove the (?:){HUGE} footgun, but filters are currently assumed to be trusted, so it's low/"nice future" priority.)

Hmmm. I don't believe regex-automata has ever suffered from this footgun? But maybe I'm wrong.

On the topic of footguns, if you haven't read the security advisory, you might enjoy doing so: https://github.com/rust-lang/regex/security/advisories/GHSA-m5pq-gvj9-9vr8
In summary, there are always ways of making regex compilation take quite a bit of time. The (?:){HUGE} footgun was a little special in that it was particularly egregious and also kinda didn't respect our API guarantee for size limits. But there is no API guarantee that, e.g., "regex compilation will always take less than 1 second." :)

Yeah, regex-automata suffers from a similar issue; I just timed regex_automata::hybrid::dfa::DFA::new("(?:){4294967295}").unwrap() and it took 101.5 seconds in release mode. For comparison, regex::Regex::new with<EMAIL_ADDRESS>took 71.5 seconds.
But you also very explicitly don't offer mitigation for this in regex-automata,

Compilation can take an exponential amount of time and space in the size of the regex pattern. While most patterns do not exhibit worst case exponential time, such patterns do exist. For example, [01]*1[01]{N} will build a DFA with 2^(N+1) states. For this reason, untrusted patterns should not be compiled with this library. (In the future, the API may expose an option to return an error if the DFA gets too big<EMAIL_ADDRESS>regex-automata@0.2.0::dfa]

This might break the lazy DFA's guarantees, though, since it sort of implies being linear on pattern length / text length, and I don't think this is.

It avoids the worst case exponential time complexity of DFA compilation by guaranteeing that it will only build at most one state per byte searched. While the worst case here can lead to a very high constant, it will never be exponential. [regex-automata@0.2.0::hybrid]

OK... so much to untangle here. Haha. This is why I didn't want to publish regex-automata 0.2. It just isn't done yet. I'll just lay out a sequence of statements that I think collectively will help to clarify things:

Creating a lazy DFA does essentially no additional work beyond building the NFA. The actual additional work that a lazy DFA needs to do isn't done until search time.
That (?:){HUGE} takes a long time doesn't necessarily mean things aren't linear. The catch here is that the size of the regex is unfortunately not len(pattern), but rather, len(pattern-with-all-bounded-repetitions-expanded).
Currently, regex-automata sets no size limits on things.  For example, try regex-cli debug nfa thompson '(?:){200000}' --nfa-size-limit 2000000.
The exponential time complexity warning at build time definitely only applies to regex_automata::dfa in 0.2.0.

So for example, if I do end up sticking with no size limits by default for regex-automata, then that would be pretty clearly documented at the crate level for example. I don't know if I will do that or not though.

I'm going to close this out in favor of the monster #425 issue. Namely, I don't think we need more than one issue for the streaming use case.
Once #656 is done, my hope is that it will enable some experimentation. Note that this may include writing your own PikeVM if you need Unicode word boundary support. it is possibly easier than it sounds, given that regex-automata will expose a way to compile an NFA. But I don't currently have any plans to pursue streaming APIs in regex-automata proper. I want to see people build their own first so we can get a better idea of what is actually required, and whether it makes sense to add streaming APIs.


rdp session always requires password  (Vers. 1909)

RDP session always asks for a password, even though the password was saved correctly. Does anyone know the bug?

is computer member of an directory domain ?

no its not

i think the problem ist in the patched "termsrv.dll"


BUG: Required fields are not checked in Playground

The Playground allows the user to submit the form even if there are required empty fields that should not be empty.

This is not a bug. This is to let api users get error responses and eventually tune their clients to properly handle them.

I think it is a bug because in my case i have two APIs like:
/{type}/ - that gets all entries of type {type}
/{type}/{id} - that gets the entry with the provided {id}
If the playground allows providing empty value for the {id} then the first API will be called - > problem!

Ok I got it. For path parameters it makes sense to do a check before submitting.Thank you.


UI Adjustments

Put "Source Environment" input before "Target Environment"
Auto scroll down in the terminal on new inputs
Add "have a good day!" in the websocket messages after deploylemnt, to give it that full retro feel
Make the terminal selectable via tab (and if possible, controllable by up and down buttons)

Terminal auto scroll functionality is stopped when user manually scrolls up past the 87% scroll height threshold. (scroll height is counted from top to bottom)


I missed a small bug on the last PR

Hi @yuval-k,
My eyes must be failing me.
Could you please merge this and release?
Appreciate that.
Thanks, Mario

no problem - thank you for fixing.
released new version


diagrams: create state diagram for coordination

this diagram will be a good starting point for thinking about how the networking part of payment channels will work.
At first this diagram will look at the initial steps of setting up a channel:
user A creates channel, sending message to user B
User B responds successfully || doesn't respond || bad response
what state is user A in now, and what are there possible options
I think that pattern will be similar for any of the messages between user A and B

the diagrams added thus far have fulfilled this ticket


HTML does not validate

There are numerous issues with aria roles, nesting of elements etc. which would fail WCAG. In all there are 25 issues,

Note that this issue cannot be resolved without upstream changes to WET.
e.g. "ARIA role tabpanel must be removed when the element is made visible, as it is not allowed for the element"
This code is added by WET Tabbed Interface:
<details id="details-step1" open="open" class="wb-auto-2-grp fade out noheight" role="tabpanel" aria-hidden="true" aria-expanded="false" aria-labelledby="details-step1-lnk">


Created howto on patching android application to accept user CA

Description
I figured out how to patch an app that would refuse to run on a rooted device to accept the user CA certificate store. I hope others find this useful as well.
Checklist

[ ] I have updated tests where applicable.
[ ] I have added an entry to the CHANGELOG.

Thanks! We already have a short section on certificate pinning here: https://docs.mitmproxy.org/dev/concepts-certificates/#certificate-pinning
How does your approach compare to using apk-mitm or objection?

I looked into apk-mitm and it is exactly what I describe here but automated... Well at least I know of an even better way now. I hadn't looked into the certificate pinning section, because I understood it to mean certificates that are packaged with the app, which was not relevant in my case.


Change play store listing to use a company name and email

Company name
Email Address
Website

Done: https://play.google.com/store/apps/details?id=coatapp.coat&hl=en-GB&ah=rTTxN6oS1gerkFk1rzKmTHC_Qwo