Add a catch/retry mechanism to lakectl local operations

This enhancement request stems from a bug report about rare failures of lakectl local commits.

@itaiad200 is this issue https://github.com/treeverse/lakeFS/issues/7705 closing this item?

I just ran into an issue where I was running lakectl local clone on about 4TB of data. It was about 1/2 done when one download failed with:
download <path> failed: could not write file '/mnt/data/frames-bbox/<file>': stream error: stream ID 167; INTERNAL_ERROR; received from peer

(I've redacted the actual file path as it contains the name of a confidential partner). The error is likely because the destination volume is an NFS volume and there was some kind of network error.
As a result I had to then cd into that directory and lakectl local pull which first required a diff of what it had pulled and what was in the cloud. As that was about 2TB of data and due to this issue that diff takes a very long time. It's still running now but I suspect it's going to be about the same amount of time as just deleting everything and trying the clone again from scratch.
I'm not sure but I think https://github.com/treeverse/lakeFS/pull/7723 fixed issues with communication with the LakeFS server but I don't think it addressed issues like the above.
In addition to this being a bit slow, manual, and annoying it can be a really significant issue in some cases. We sometimes launch ML training jobs as Argo workflows. The first step in that workflow is to download the relevant training data to faster, local storage. If that fails the entire workflow fails. The initial clone can take about 8 hours so that's a significant issue if it fails after 6 or 7 hours and we have to restart the job.

Just had another one of these after hours of data download. This time it wasn't an NFS volume.
download<path>/camera_b_raw/frames.h5 failed: could not write file '/mnt/data/frames_bbox/<frames>/camera_b_raw/frames.h5': stream error: strea │
│ Error executing command.

As above, I've redacted the full filename a bit.

Like @oliverdain mentioned, #7723 handles retries with lakeFS requests, and won't address issues like the above, where the response failed to be written to the destination file (here).
Like the title suggests, in order to solve this we would have to retry the entire apply flow in situations where it's not lakeFS server requests failing (to avoid multiplying the retries).

I just had another data transfer fail with:
download <redacted>/camera_b_raw/frames.h5 failed: could not write file '/<redacted>/camera_b_raw/frames.h5': stream error: stream ID 325; INTERNAL_ERROR; received from peer

This time I'm using a regular GKE persistent disk not an NFS volume so I think that error is coming from the LakeFS server. This is now the 3rd time in a row my data transfer has failed after about 10 hours.


client: Implement multiple camera angles per fire event

Index incoming fire events by camera ID/timestamp in a lookup table
Add ._anglesByKey property to incoming fire events
Populate ._anglesByKey with camera angles that overlap the given
fire event within the last 15 minutes
Bring camera markers corresponding to secondary angles to the front
z-Index of the map to facilitate angle selection by clicking markers


advice needed: producer under gunicorn never sends messages

TLDR: is there a recommended way to use pykafka in a web app to produce messages?
I need to produce some messages as part of a web application and I can do so by creating the connection when the request comes in, but the overhead is too much with sync=true. messages never arrive when connection/topic/producer are module level.
The app in question is running in gunicorn with preload_app=True and worker_class="gthread"
Right now I'm getting around this by spawning a thread and producing the message for the request there but I have concerns that spawning an additional thread per request /response to write message would cause issues under a production load.
I would be running into the same kinds of problems with the database if I was spawning a connection per thread or even worse, per request, however the author(s) of sqlalchemy gives me a connection pool and it works like a charm even with 48 worker threads per 8 processes.
I'm thinking something similar is needed for kafka but i'm kind of surprised it hasn't been written already.
This leads me to believe that maybe i'm on the wrong track here and there is a way in pykafka for producing messages in the context of a web app without the web developer having to know all the intricacies of threading/processes with this library.
Using something like celery seems redundant here and I would rather not take on the complexity of putting rabbitmq or redis in our infrastructure so I can produce messages into kafka.

here's source that demonstrates the problem. https://github.com/twillis/kafka_web


per-job duration increases linearly with the number of commits

What happened?: per-job duration increases linearly with the number of commits
What you expected to happen?: pachyderm administration overhead increases sublinearly, probably logarithmically, with the number of commits/files.
How to reproduce it (as minimally and precisely as possible)?:

create pipeline that does minimal work, such as sort an input file into directories
create one commit per input file, and create some input files (a thousand or so)
note that as jobs complete, job.finished - job.started increases linearly

Environment?:

Kubernetes version (use kubectl version):

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T05:28:34Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.7", GitCommit:"b30876a5539f09684ff9fde266fda10b37738c9c", GitTreeState:"clean", BuildDate:"2018-01-16T21:52:38Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Pachyderm CLI version (use pachctl version):

$ pachctl version
COMPONENT           VERSION
pachctl             1.6.5
pachd               1.6.5

Cloud provider: AWS, k8s deployed via kops
OS (e.g. from /etc/os-release):

$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 8 (jessie)"
NAME="Debian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=debian
HOME_URL="http://www.debian.org/"
SUPPORT_URL="http://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Anything else we need to know?:
Is this behavior expected? Is there a chance of user error? Are there any places I can check/tweak/configure to ensure performance remains close to constant? I would like to be able to push 10s of thousands of files in this manner, but worry about overwhelming overhead.
Below are some diagrams that demonstrate decreasing performance.
There are three pipelines:

user-idb is a simple file sort pipeline
idb-push-names and idb-sync-fid do a few seconds of seconds of CPU work, and then HTTP POST their results elsewhere

I created around 1200 commits, each with a single input file. Pachyderm is still catching up. You can see that job durations are increasing, while CPU and network activity remain constant. My interpretation is that pachyderm administration overhead is increasing with each job, file, or commit (unsure which once).

Other theories I've considered:

overhead from job depth; however, after I stopped creating new input files, job duration continued to increase as before
too many processes/threads/tasks running on worker nodes; however, via manual inspection (and CPU utilization charts above), CPU load has remained between 3 and 5 on four core nodes.

in comparing job logs from early and recent jobs from the same pipeline, I noticed that the time before the first log entry (before process call started - request: ...) has increased dramatically. see cherry-picked example below:
early job
130ms: process call started - request: job_id:"ced77a2e-3f67-4402-81de-e0bb2ea2975e" data:<file_info:<file:<commit:<repo:<name:"idbs" > id:"cdc35bcc1ec243809dc8561248f30d20" > path:"/73665ca317de49300c8a373d180b2b619a8b865e913688120d97635c1c57f8e0" > file_type:FILE size_bytes:4607861 hash:"\025\335c\263y\256\300\342\267\271\374R\214!s\266G\037\330\005\026p_E*8\253\020)j\rw" > name:"idbs" branch:"master" > enable_stats:true
174ms: input has not been processed, downloading data
325ms: input data download took (151.445356ms)
325ms: beginning to run user code
327ms: inputs:
328ms: 73665ca317de49300c8a373d180b2b619a8b865e913688120d97635c1c57f8e0
330ms: input:  73665ca317de49300c8a373d180b2b619a8b865e913688120d97635c1c57f8e0
378ms: WARNING:idb.fileformat:section class not implemented: id2
386ms: INFO:idb.netnode:resolved string netnode Root Node to ff00000000000002
386ms: INFO:idb.netnode:resolved string netnode Root Node to ff00000000000002
420ms: output:
421ms: 00d6cfd5bdf6998f39ce1af16d7b4e6f.i64
421ms: done
422ms: finished running user code - took (97.095435ms) - with error (<nil>)
438ms: starting to upload output
746ms: finished uploading output - took 307.179174ms

recent job
10948ms: process call started - request: job_id:"8e8a712a-a917-40d3-8fc3-62d22c5abd73" data:<file_info:<file:<commit:<repo:<name:"idbs" > id:"1ba6e5d8590141d69a41cd5e5193c046" > path:"/e2cca7bed72a04647422b4d9adc214c9f69acd25bb5ceeacd4e633e7b2ec2c34" > file_type:FILE size_bytes:1273955 hash:"\304\341\306p\332_\335S\350\222\362z\374\303\265\032*mG&/21W\230\211\270\231n\266\257," > name:"idbs" branch:"master" > enable_stats:true
10996ms: input has not been processed, downloading data
11192ms: input data download took (195.252644ms)
11192ms: beginning to run user code
11195ms: inputs:
11196ms: e2cca7bed72a04647422b4d9adc214c9f69acd25bb5ceeacd4e633e7b2ec2c34
11198ms: input:  e2cca7bed72a04647422b4d9adc214c9f69acd25bb5ceeacd4e633e7b2ec2c34
11262ms: WARNING:idb.fileformat:section class not implemented: id2
11289ms: INFO:idb.netnode:resolved string netnode Root Node to ff000002
11289ms: INFO:idb.netnode:resolved string netnode Root Node to ff000002
11315ms: output:
11316ms: 3442f17ac4cb2559d69fcefe79fce345.idb
11318ms: done
11319ms: finished running user code - took (126.904087ms) - with error (<nil>)
11338ms: starting to upload output
11825ms: finished uploading output - took 486.345898ms

in reviewing a slack conversation from earlier, i see this explanation:

@jdoliner:  so basically what's going on here is that for each commit you create the system needs to figure out what all the datums are for that commit. Then for each datum either process, if it's new, or skip it if it's already been processed, and then merge the results together. Skipping datums is much faster than processing them, but there's still some overhead associated with it since you have to check if you have the right objects in the object store. Because you're committing one datum at a time Pachyderm needs to do n^2 checks for already processed datums to process n datums. One of the major things we're going to make faster is the check for already processed datums. But in the meantime you can make this a lot faster by committing things in batches.

I won't self-close the issue yet to allow for some discussion. I find the existing behavior surprising (though, understanding after explanation), and definitely look forward future enhancements.

To summarize: the number of past commits/jobs has no bearing on processing time of future jobs (other than to potentially reduce the amount of work required, if some of the current job's datums have already been processed), but job runtime does increase linearly with the number of datums in that job (which, in your case, are one-per file).
At the moment, the added overhead of an additional skip-able datum is fairly small (check if it exists in the object store), but non-zero.  There are also, however, potentially design changes that could be made to effectively eliminate it (by, say, storing and only processing a diff per commit, rather than scanning all datums), but that gets complicated when files in an upstream commit don't map one-to-one with downstream datums, and would require (much) more significant changes.

Hey @gabrielgrant thanks for adding to the discussion here!
Would you mind clarifying a bit (so that my future pipelines are most efficient):

job runtime does increase linearly with the number of datums in that job

So with one datum per job, then overhead is at its minimum?

At the moment, the added overhead of an additional skip-able datum is fairly small.

Cool, this makes sense to me. In my example above, none of the datums are skippable. Is there an overhead for non-skippable datums?
Does the increasing overhead reported here comes from having to check "for the new datum, does it match any of the old datums so that i can skip it?". If so, is the correct name for this github issue "per-job duration increases linearly with the number processed datums"? Ultimately, I'm trying to understand the crux of the issue and avoid linear performance degradation as I add commits.

Whoops, probably just getting tripped up on terminology. I had been using "datum" to refer to a group of files added in a commit that are grouped and processed together.
In the example "user-idb" pipeline from above (and whose contents are below), the inputs are unique files that can be processed independently.
{
    "pipeline": {
        "name": "user-idb"
    },
    "input": {
        "atom": {
            "repo": "idbs",
            "glob": "/*"
        }
    },
    "parallelism_spec": {
        "constant": 1
    },
    "transform": {
        "image": "...",
        "image_pull_secrets": ["dockerhubkey"],
        "cmd": ["/bin/bash"],
        "stdin": [
            "cd /pfs/idbs/;",
            "echo 'inputs: ';",
            "ls -1;",
            "FILE=$(ls -1 | head -n 1); ",
            "echo 'input: ' $FILE; ",
            "IDBMD5=$(python /opt/idb_md5.py $FILE);",
            "FMD5=$(md5sum $FILE | cut -d ' ' -f 1);",
            "mkdir -p /pfs/out/$IDBMD5/$FMD5/;",
            "if [ $(head -c 4 $FILE) = 'IDA1' ]; then ",
            "  cp $FILE /pfs/out/$IDBMD5/$FMD5/$FMD5.idb ;",
            "else ",
            "  cp $FILE /pfs/out/$IDBMD5/$FMD5/$FMD5.i64 ;",
            "fi; ",
            "echo 'output:';",
            "ls /pfs/out/$IDBMD5/$FMD5/;",
            "echo 'done'; ",
            "date > /pfs/out/$IDBMD5/$FMD5/success.txt;",
            "exit 0;"
        ]
    },
    "resource_requests": {
        "cpu": 0.1,
        "memory": "100M"
    },
    "resource_limits": {
        "cpu": 2.0,
        "memory": "500M"
    },
    "enable_stats": true
}

@williballenthin so you're not reprocessing the old datums it sounds like, only processing the new ones that are being added in the new commit. So if you do inspect-job, you should see a number of datums processed and number skipped, right?

Ah, I see that now:
$ pachctl inspect-job 378cca63-622c-4575-b95f-f32fb413b955
ID: 378cca63-622c-4575-b95f-f32fb413b955
Pipeline: p1
Parent: cf6ccaac-5f64-403a-941d-96282e504b2d
Started: 2 minutes ago
Duration: 5 seconds
State: success
Reason:
Processed: 1
Skipped: 151
Total: 152

So, as new files are added and processed independently, the other existing files are considered skipped. And there is a constant overhead for each other file that is skipped. So as more files are added to the repo, jobs will take longer and longer. Thanks for walking me through this behavior @gabrielgrant and @JoeyZwicker.
So, given the above, is there anything I can do to keep overhead down, when my use case is to process small subsets of a repo independently? I've heard a suggestion to batch commits to the input repo by @jdoliner, but I think this will only delay the problem, rather than solve it.

We've got an improvement incoming that will drastically reduce the overhead for skipped files which should pretty much completely resolve this issue.

For future reference, here are scripts and a pipeline spec to reproduce the behavior described in this issue.
p1.json
{
    "pipeline": {
        "name": "p1"
    },
    "input": {
        "atom": {
            "repo": "inputs",
            "glob": "/*"
        }
    },
    "transform": {
        "image": "alpine:3.7",
        "cmd": ["/bin/sh"],
        "stdin": [
            "for F in /pfs/inputs/*;",
            "do",
            "  HASH=$(sha256sum ${F} | cut -d ' ' -f 1);",
            "  mkdir -p /pfs/out/${HASH}/;",
            "  ln -s ${F} /pfs/out/${HASH}/${HASH};",
            "done;"
        ]
    },
    "enable_stats": true
}

setup.sh
#!/bin/bash

pachctl create-repo inputs
pachctl create-pipeline -f p1.json

trigger.sh
#!/bin/bash

for i in $(seq 1 1000); do
    echo "$i";
    echo "$i" > "/tmp/${i}.txt";
    pachctl put-file inputs master "$i" -c -f "/tmp/${i}.txt";
    rm "/tmp/${i}.txt";
done

reset.sh
#!/bin/bash

pachctl delete-pipeline p1 --delete-jobs --delete-repo
pachctl delete-repo inputs
pachctl garbage-collect

For my particular scenario, I've found a solution that reduces job overhead. I'll describe it here so that other's can decide if its relevant to their setups. @pachyderm-masters you can close this issue out if appropriate, or maintain it as a discussion thread.
In my scenario, I don't care about file names, only file content. Therefore, I can name inputs as cryptographic hashes of the contents (or anything, really). By placing input files into nested directories derived from their content hash, per-job overhead is reduced to a logarithmic amount.
For example: given file a.txt with md5 hash abcdef012345..., commit this file to: /inputs/ab/cd/abcdef012345.... Downstream pipelines read from /inputs/*/*/*. Configuration dials to tune the performance include: number of nested subdirectories, and length of each component.
Here's the per-job duration plots for three strategies:

1000 input files (though not yet complete processing).

p1: naive approach with all input files in same directory
p3: nested directory depth two, each component is two characters (branch factor 256)
p4: nested directory depth two, each component is one character (branch factor 16)

@brycemcanally Is this issue fixed by the changes you've been making recently?

This issue is largely fixed, most of the time here was being spent checking for previously processed datums and performing the merge, these have both been optimized in the append only case to be mostly skipped. With the new storage layer this will get even better, but I think the original ask has been fulfilled here so I'm closing. Feel free to reopen if there are further issues of this sort.


feat(import-glob): support read module from alias pattern

Description
refs #12180

Additional context

What is the purpose of this pull request?

[ ] Bug fix
[x] New Feature
[ ] Documentation update
[ ] Other

Before submitting the PR, please make sure you do the following

[x] Read the Contributing Guidelines.
[x] Read the Pull Request Guidelines and follow the PR Title Convention.
[x] Check that there isn't already a PR that solves the problem the same way to avoid creating a duplicate.
[x] Provide a description in this PR that addresses what the PR is solving, or reference the issue that it solves (e.g. fixes #123).
[x] Ideally, include relevant tests that fail without this PR but pass with it.

Can you help review this pr ? @bluwy @sapphi-red  Thanks.

Having both isn't possible because then users would need to de-duplicate them. Imagine if they do a glob and iterate over the result to do something with each.
So if we change something, I think the paths could keep the alias instead of using absolute paths. But I don't know how much ergononic is to have the unresolved path as the key instead of the absolute path here. cc @brillout @antfu, interested in your opinion here.

I think we should preserve the path format. In other words:

If the glob path is relative => return relative paths.
if the glob path is absolute => return absolute paths.
If the glob path contains an alias => return paths containing the same alias.

The pro here is that the user has the flexibility to choose the format. And, AFAICT, I don't see any drawback for this.
I believe that's what we decided back in the days of vite-plugin-glob (but correct me if I'm wrong @antfu).

Having both isn't possible because then users would need to de-duplicate them. Imagine if they do a glob and iterate over the result to do something with each.

Agreed.

I think the paths could keep the alias instead of using absolute paths

I think so too.

But I don't know how much ergononic is to have the unresolved path as the key instead of the absolute path here

If the user wishes to have absolute paths, then he can provide an absolute glob path instead of an aliased one.

I've been thinking of a solution for this, I feel like the current behaviour while strange is able to scale to more cases. For example,
<EMAIL_ADDRESS><EMAIL_ADDRESS>'./extras/*.json'])

It would be hard to determine what the keys should be. Some will have aliases, some will not. Today, the keys for this is ['/src/data/one.json', '/src/data/two.json', '/src/pages/extras/one.json'].
The keys are root-relative if one of the globs are not relative.

So to summarize and document how it works today:

If the globs are all relative, the keys will be relative to the file.
If one of the globs are not relative, the keys will be relative to the root (with root-relative syntax).
For virtual files, the keys will also be relative to the root (makes sense in general).

If we want to change something, I think we need to add compat keys to prevent a big breaking change, maybe through a Proxy. But in this case, I think documenting it should be enough.

I'll move this off the Vite 5 milestone for now. From the above comment, we could start with a documentation update. If we want to change the behaviour, we might need to plan out a migration path and some sort of compat so it doesn't cause a big breaking change.


Delete CustomWidget

Because the underlying QtWidgetElement can't be typed.

Haven't decided to merge this or not.


Don't use filters with Edge. Fixes #124.

Edge supports filters, but they act funny (flickering when hovering over the course tiles). To prevent this, just filter:none; when we're in Edge - means Edge users don't get the colourization effect, but that's better than the flickering by far (and Edge users don't get the colourization anyway right now).

Checked this in Edge, IE11, Chrome and Firefox. No effect in Chrome/FF, IE11 doesn't support CSS filters anyway (so no colourization, no flickering), and Edge now does the zoom without the flicker (no colourization, but that wasn't there before either).


Export headers for SSG apps to import in nginx.conf

Is your feature request related to a problem? Please describe.
When using this modules for a SSG app, the headers are correctly used in development, but there's no way to extract the generated headers to import them in the server configuration for the deployment server.
Describe the solution you'd like
I would like the headers to be exported into a headers.conf file for example. So that I can use the nginx include function to import them.
Describe alternatives you've considered
The alternative is that I set all headers manually and don't use this module at all.
Additional context

Hey buddy,
Thanks for reporting this issue idea. I have very little experience with Nginx to be honest so I cannot really say much about it.
I completely understand the reasoning why you may want it but I am not sure if we can deliver something from the module side that would be useful for the majority of module users.
Maybe you have some idea already? I understand the case of exporting headers as headers.conf but how do you see it in more details? Would it be a script that would transform headers from defaultConfig.ts file? Or maybe a log in the console after build with the headers that are set for the app?
I would appreciate your feedback :)

I am not sure whether this falls within the scope of this module.
To me it sounds more like

Either a new module idea for Nginx users
Or a feature request for an nginx_static preset in unjs/nitro

On the other hand, we do already compute headers for the exact same use case that OP is talking about. But currently they can only be used with the Vercel and Netlify Nitro presets. There is no nginx preset in Nitro.
One way we could introduce some flexibility would be to provide a custom hook with the value of the headers. It would then be the responsibility of the user to generate the correct headers.conf file or whatever other output ?

Hey @vejja
Great to have you back! 🚀
Thanks for providing this idea to @mtdvlpr . Maybe it will allow to get the desired output already.
In terms of implementing this as a part of the module. I agree with you that it can be outside of the scope of the security module to have this functionality.
If the solution you provided will work, I would recommend to add a note in FAQ about how this can be achieved for easier discovery.
Let me know guys what you think about it.

@mtdvlpr have you tried the solution provided by @vejja ?

Hi @Baroshem I haven't tested it yet, but I have tested the Netlify preset and that's exactly what I want! Only get the headers in a different format for nginx. I will create a custom preset and maybe create a PR so it gets added to the nitro presets as an nginx preset. So I consider this issues resolved.

Awesome, glad that you managed to make it work! 🚀
I will close the issue now.
If you will have any ideas in the future about what we could do to make the module better, do not hesitate to let us know through issues here.
Happy coding! ⚡

Hi @Baroshem, quick question. Since I'm now extracting the headers, including CSP, and importing them in my nginx config, I don't need the CSP meta tag that's generated. Is there a way to disable the meta tag but keep the CSP configuration?

Hey @mtdvlpr
AFAIK, we dont have this option right now so if you select this rendering mode in your build config nuxt security will automatically detect it and set up either the headers (SSR) or meta (SSG).


[NEW] Off-by-One Conference 2024

Name: Off-by-One Conference
Website: https://offbyone.sg/
Date(s): June 26-27 2024
Location: Singapore
Twitter/Mastodon Handle:  @offbyoneconf
Free: N

Thanks!


Added Spanish translation for keyboards.md

Added a Spanish translation for the keyboards.md file as a way to complement the efforts made by #67 in providing Spanish translations for the documentation.

Unfortunately, this is a file that really can't be translated.  It's automatically generated each time that QMK Firmware updates, and is regenerated and replaced.
The translations would need to go to the script generating this page, not the page itself.

Closing, for the reasons @drashna detailed above.
I've filed an issue regarding this matter. Hopefully we can swing back around and do this.


Some performance tuning

Wanted so see what's this new shiny thing a saw things that might improve perf, so here's some.

improve array access by having array as local using uint access to avoid bound checks  when needed / using foreach if possible
prefer read-only structs, get/set properties and methods are bad for structs as they cause copying for CLR to be safe

generally should make structs less class-like, it's kind of either or with the properties and methods, extension methods can help with structs

tried to make some property access inlined

To further improve performance, some suggestions:

Ditch the IParser/IParser<T> and always use abstract class access, interface dispatch is more costly, these are called a lot
Consider if possible for benchmarks to use TextSpan instead of string for getting JSON properties, it might make the parsers con-comparable though, the ToString() needed for materializing properties isn't awfully nice. I did try this with custom Marvin hasher for ROS (netstandard2.1 doesn't have GetHashCode for ROS)
traversing the string with pointers instead of index might be faster, haven't tried it out though

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19042
AMD Ryzen 7 2700X, 1 CPU, 16 logical and 8 physical cores
.NET Core SDK=5.0.101
  [Host]   : .NET Core 5.0.1 (CoreCLR <IP_ADDRESS>516, CoreFX <IP_ADDRESS>516), X64 RyuJIT
  ShortRun : .NET Core 5.0.1 (CoreCLR <IP_ADDRESS>516, CoreFX <IP_ADDRESS>516), X64 RyuJIT

Job=ShortRun  IterationCount=3  LaunchCount=1
WarmupCount=3

Parlot.Benchmarks.ExprBench

Diff
Method
Mean
Error
Gen 0
Gen 1
Gen 2
Allocated

Old
ParlotRawSmall
784.8 ns
308.0 ns
0.1411
0.0000
0.0000
592 B

New

759.9 ns (-3%)
305.91 ns
0.1411 (0%)
0.0000
0.0000
592 B (0%)

Old
ParlotFluentSmall
2,623.9 ns
320.7 ns
0.1755
0.0000
0.0000
744 B

New

1,975.2 ns (-25%)
54.83 ns
0.1755 (0%)
0.0000
0.0000
736 B (-1%)

Old
PidginSmall
14,292.6 ns
376.3 ns
0.1831
0.0000
0.0000
816 B

New

14,645.4 ns (+2%)
1,034.18 ns
0.1831 (0%)
0.0000
0.0000
816 B (0%)

Old
ParlotRawBig
3,909.5 ns
1,367.0 ns
0.6447
0.0000
0.0000
2712 B

New

4,059.4 ns (+4%)
106.52 ns
0.6485 (+1%)
0.0000
0.0000
2712 B (0%)

Old
ParlotFluentBig
13,842.5 ns
2,039.8 ns
0.7477
0.0000
0.0000
3136 B

New

10,241.6 ns (-26%)
599.34 ns
0.7477 (0%)
0.0000
0.0000
3128 B (0%)

Old
PidginBig
72,480.7 ns
1,090.6 ns
0.8545
0.0000
0.0000
4072 B

New

72,651.2 ns (0%)
2,222.91 ns
0.8545 (0%)
0.0000
0.0000
4072 B (0%)

Parlot.Benchmarks.JsonBench

Diff
Method
Mean
Error
Gen 0
Gen 1
Gen 2
Allocated

Old
BigJson_Parlot
367.6 μs
2.37 μs
24.9023
3.4180
0.0000
101.84 KB

New

292.0 μs (-21%)
3.01 μs
24.9023 (0%)
3.4180 (0%)
0.0000
101.84 KB (0%)

Old
BigJson_Pidgin
556.1 μs
6.77 μs
24.4141
2.9297
0.0000
101.7 KB

New

553.3 μs (-1%)
3.96 μs
24.4141 (0%)
2.9297 (0%)
0.0000
101.7 KB (0%)

Old
BigJson_Sprache
3,595.3 μs
254.49 μs
1308.5938
50.7813
0.0000
5349.63 KB

New

3,511.6 μs (-2%)
39.13 μs
1308.5938 (0%)
50.7813 (0%)
0.0000
5349.63 KB (0%)

Old
BigJson_Superpower
2,133.4 μs
148.34 μs
222.6563
3.9063
0.0000
913.43 KB

New

2,144.2 μs (+1%)
126.62 μs
222.6563 (0%)
3.9063 (0%)
0.0000
913.43 KB (0%)

Old
LongJson_Parlot
289.6 μs
2.72 μs
25.3906
0.4883
0.0000
104.39 KB

New

236.2 μs (-18%)
1.35 μs
25.3906 (0%)
0.2441 (-50%)
0.0000
104.38 KB (0%)

Old
LongJson_Pidgin
491.8 μs
12.61 μs
25.3906
3.9063
0.0000
104.25 KB

New

485.2 μs (-1%)
4.95 μs
25.3906 (0%)
2.4414 (-38%)
0.0000
104.25 KB (0%)

Old
LongJson_Sprache
2,847.4 μs
161.38 μs
1054.6875
11.7188
0.0000
4311.36 KB

New

2,895.6 μs (+2%)
161.87 μs
1054.6875 (0%)
11.7188 (0%)
0.0000
4311.36 KB (0%)

Old
LongJson_Superpower
1,734.5 μs
34.44 μs
171.8750
3.9063
0.0000
706.79 KB

New

1,730.9 μs (0%)
103.56 μs
171.8750 (0%)
3.9063 (0%)
0.0000
706.79 KB (0%)

Old
DeepJson_Parlot
248.6 μs
35.45 μs
20.0195
0.4883
0.0000
82.39 KB

New

198.4 μs (-20%)
3.77 μs
20.0195 (0%)
0.2441 (-50%)
0.0000
82.38 KB (0%)

Old
DeepJson_Pidgin
573.1 μs
49.33 μs
45.8984
0.9766
0.0000
187.79 KB

New

567.8 μs (-1%)
20.04 μs
45.8984 (0%)
0.9766 (0%)
0.0000
187.79 KB (0%)

Old
DeepJson_Sprache
3,190.5 μs
37.78 μs
562.5000
226.5625
0.0000
2946.56 KB

New

3,211.3 μs (+1%)
261.43 μs
554.6875 (-1%)
218.7500 (-3%)
0.0000
2946.56 KB (0%)

Old
WideJson_Parlot
204.9 μs
3.67 μs
11.7188
0.2441
0.0000
48.56 KB

New

178.0 μs (-13%)
4.43 μs
11.7188 (0%)
0.7324 (+200%)
0.0000
48.55 KB (0%)

Old
WideJson_Pidgin
290.6 μs
4.36 μs
11.7188
0.9766
0.0000
48.42 KB

New

293.6 μs (+1%)
2.93 μs
11.7188 (0%)
0.9766 (0%)
0.0000
48.42 KB (0%)

Old
WideJson_Sprache
1,676.3 μs
123.49 μs
683.5938
11.7188
0.0000
2797.28 KB

New

1,635.4 μs (-2%)
113.14 μs
683.5938 (0%)
11.7188 (0%)
0.0000
2797.28 KB (0%)

Old
WideJson_Superpower
1,108.5 μs
65.53 μs
111.3281
3.9063
0.0000
459.74 KB

New

1,104.4 μs (0%)
14.52 μs
111.3281 (0%)
3.9063 (0%)
0.0000
459.74 KB (0%)

This PR doesn't have complex changes and yet produces great benefits. I wonder which one is the most significant. I might try a few of the most obvious ones to see.
For the Json benchmarks changes, you are right as I don't want to change it. I want to comply with what Pidgin was doing. The goal is just to compare the parsers, not the raw outcome of the benchmark.
The IParser usage is an issue right now. I have struggled finding a good compromise with the class heerarchy, and how to provide a good Fluent API. I think I haven't found the best approach. Maybe I should try to ditch the interface altogether if it doesn't provide any meaningful abstraction. Maybe the base abstract class would be sufficient. I would also love to have a covariant type that would allow for Parser to be castable to Parser.
Good idea for the string pointers. That's what the ROS algorithms do internally. Maybe the scanner/cursor should only deal with a ROS.

This PR doesn't have complex changes and yet produces great benefits. I wonder which one is the most significant. I might try a few of the most obvious ones to see.
For the Json benchmarks changes, you are right as I don't want to change it. I want to comply with what Pidgin was doing. The goal is just to compare the parsers, not the raw outcome of the benchmark.
The IParser usage is an issue right now. I have struggled finding a good compromise with the class heerarchy, and how to provide a good Fluent API. I think I haven't found the best approach. Maybe I should try to ditch the interface altogether if it doesn't provide any meaningful abstraction. Maybe the base abstract class would be sufficient. I would also love to have a covariant type that would allow for Parser to be castable to Parser.
Good idea for the string pointers. That's what the ROS algorithms do internally. Maybe the scanner/cursor should only deal with a ROS.


TelemetryInitializerBase is internal

Commit https://github.com/Microsoft/ApplicationInsights-aspnetcore/commit/b7db1a4940e715bd4081fdb4e1dde92592eff0f7 made the TelemetryInitializerBase internal. Was it really the intent? Isn't the purpose of this class to be reused to easily access the HttpContext from the request in a custom TelemetryInitilizer?
The other initializers are also internal, which would prevent from using them...

@pakrym, since he made the commit. Did we intend to hide them? For contrast, the initializers in the ApplicationInsights-dotnet-server repo are public and so is the WebTelemetryInitializerBase.

@pharring our goal was to shrink public API surface so we can change things without breaking changes to public API's. Accessing current HttpContext can pretty easily be done with IHttpContextAccessor

Thanks. It's very reasonable to limit the public surface area, especially early on.
Agreed that getting the current HttpContext is straightforward.
I think folks are being surprised to find that the derived initializers are also internal (#346 and one other I can't find right now). Right now, it's "all or nothing": i.e. You can call AddApplicationInsightsTelemetry and get our curated set of initializers; or, if you want a customized subset, then you're left re-implementing our list.
I recommend we monitor the feedback on this one and find out which initializers folks would really like to make public.

Hi @pharring,
I'm not so sure getting the current HttpContext is straightforward. I had to reimplement something similar to TelemetryInitializerBase since it requires a lock. Nothing overly complex, but considering current documentation does not make any mention, I am still confused on the correct implementation.


changed sample to use services dropdown as filter for selecting packages

@john0005 can you review this once more. There was some confusion as to what kathleen wanted. Hardik says these changes will work for her.

@tyle8552 Image in README is slightly off.

Made the changes. Thanks.


Dissenter plugin is setting the new tab behaviour

Plugin disabled until this behaviour can be disabled.

If I wanted this, I'd download bonzai buddy.

UH OH, Someone goofed up ! ;-)
Not mine. I signed this myself using the old fork from ~1week ago. Can install it without any developer mode BS. https://www.mediafire.com/file/434u2g3vr0v33l7/dissenter_extension-0.1.5-fx.xpi/ ^^

but really now by doing this dissenter becomes nothing more than like that disgusting 2000s new tab malware

@kabo, @TehSViN24, @githubsux, @gary9872 in the latest release, r11, the Dissenter new tab experience has been made optional. It is off by default and can be toggled on from within the extension's options/preferences page.


Update Xcode version to 16

Description

[!IMPORTANT]
We should likely not merge this PR until Xcode 16.0 RC is available.

This PR updates Xcode version to 16.
Changes made

Update Xcode version.
Update fastlane to avoid build issue with Xcode 16.

Checklist

[x] APIs have been properly documented (if relevant).
[x] The documentation has been updated (if relevant).
[x] New unit tests have been written (if relevant).
[x] The demo has been updated (if relevant).

We sadly cannot use the new Xcode 16 synchronized folders because they are not yet supported by xcodeproj, leading to fastlane failing when archiving the projects.


Segmentation Fault Error after initial successful run

I am able to build this project, but running the inference after the engine has been generated throws a segmentation fault error
vanguard@vanguard-jetson:~/dev/tensorrt-cpp-api/build$ make -j$(nproc)
[ 25%] Building CXX object CMakeFiles/tensorrt_cpp_api.dir/src/engine.cpp.o
/home/vanguard/dev/tensorrt-cpp-api/src/engine.cpp: In member function ‘bool Engine::build(std::__cxx11::string)’:
/home/vanguard/dev/tensorrt-cpp-api/src/engine.cpp:81:16: warning: unused variable ‘output’ [-Wunused-variable]
     const auto output = network->getOutput(0);
                ^~~~~~
[ 50%] Linking CXX shared library libtensorrt_cpp_api.so
[ 50%] Built target tensorrt_cpp_api
[ 75%] Building CXX object CMakeFiles/driver.dir/src/main.cpp.o
[100%] Linking CXX executable driver
[100%] Built target driver
vanguard@vanguard-jetson:~/dev/tensorrt-cpp-api/build$ ls
CMakeCache.txt  CMakeFiles  cmake_install.cmake  driver  libtensorrt_cpp_api.so  Makefile
vanguard@vanguard-jetson:~/dev/tensorrt-cpp-api/build$ ./driver
Searching for engine file with name: trt.engine.a220528a4ef634d2ac5172ebc267ecf9.fp32.16.2_4_8.4000000000
Engine not found, generating...
onnx2trt_utils.cpp:364: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
DLA requests all profiles have same min, max, and opt value. All dla layers are falling back to GPU
Detected invalid timing cache, setup a local cache instead
Tactic Device request: 2193MB Available: 492MB. Device memory is insufficient to use tactic.
Skipping tactic 3 due to oom error on requested size of 2193 detected for tactic 5.
Tactic Device request: 2193MB Available: 500MB. Device memory is insufficient to use tactic.
Skipping tactic 3 due to oom error on requested size of 2193 detected for tactic 5.
Tactic Device request: 457MB Available: 453MB. Device memory is insufficient to use tactic.
Skipping tactic 4 due to oom error on requested size of 457 detected for tactic 5.
Tactic Device request: 2193MB Available: 482MB. Device memory is insufficient to use tactic.
Skipping tactic 3 due to oom error on requested size of 2193 detected for tactic 5.
Tactic Device request: 2193MB Available: 484MB. Device memory is insufficient to use tactic.
Skipping tactic 3 due to oom error on requested size of 2193 detected for tactic 5.
Success, saved engine to trt.engine.a220528a4ef634d2ac5172ebc267ecf9.fp32.16.2_4_8.4000000000
Success! Average time per inference: 15.2175 ms, for batch size of: 4

After the successful generation of the engine when I run the driver again I get the following error.
Searching for engine file with name: trt.engine.a220528a4ef634d2ac5172ebc267ecf9.fp32.16.2_4_8.4000000000
Engine found, not regenerating...
Segmentation fault (core dumped)

Environment
TensorRT Version : 8.0.1-1
CUDA Version : 10.2
Operating System + Version : Ubuntu 18.04.6 LTS
Inference Network : AlexNet (using the conversion tutorial here : AlexNet from PyTorch to ONNX ) . I used the dynamic_axes flag while exporting the model.
torch.onnx.export(model, dummy_input, "alexnet_dynamic.onnx", verbose=True, input_names = ['input'],   # the model's input names
                  output_names = ['output'], # the model's output names
                  dynamic_axes={'input' : {0 : 'batch_size'},    # variable length axes
                                'output' : {0 : 'batch_size'}})

Device for TRT engine builder: Jetson Nano 4GB

Hi @vanguard478, sorry for the late response.
Can you please upload the exported onnx model so I can debug on my end and determine what is causing the segfault.

I have the same error and this is my onnx model
mobilenetv3_small_100.zip

same problem.
I try this official onnx and my OS is Ubuntu18.04-cuDNN8.2.2-TensorRT-8.4.1 :
import torchvision.models as models
import torch
import torch.onnx
import numpy as np
import onnx

# load the pretrained model
resnet50 = models.resnet50(pretrained=True, progress=False).eval()

BATCH_SIZE = 32
USE_FP16 = True
target_dtype = np.float16 if USE_FP16 else np.float32

dummy_input = torch.randn(1, 3, 224, 224)
torch_out = resnet50(dummy_input)
torch.onnx.export(resnet50, dummy_input, "resnet50_pytorch_d.onnx",
    verbose=False,
    input_names=['input'],
    output_names=['output'],
    dynamic_axes={
        'input': {0: 'batch_size'},
        'output': {0: 'batch_size'},
    }
)

Hi @LicharYuan
The model you have provided has an input of (1, 3, 223, 224), whereas the input image in the repo is of size (3, 112, 112). Therefore the dimensions don't match. I can't be certain if you used the input image in the repo, or your own one. I am therefore closing this ticket.
That being said, I have made the code more robust such that it will give an error message if you try to provide an input which is not the correct size.
Please test again with the patches I have added and if you can still reproduce the bug, you can re-open this ticket and I will look into it.


Formatcode

Keep in mind that all other open PRs have to be reformatted too.

SonarQube analysis reported 295 issues

 50 critical
 196 major
 41 minor
 8 info

Watch the comments in this conversation to review them.
Top 10 extra issues
Note: The following issues were found on lines that were not modified in the pull request. Because these issues can't be reported as line comments, they are summarized here:

 ForwardRateAgreement.cs#L118: Rename parameter 't' to 'incomeDiscountCurve' to match the base class declaration.
 ImpliedVolatility.cs#L75: Rename parameter 'x' to 'v' to match the base class declaration.
 Makevanillaswap.cs#L224: Refactor this method to reduce its Cognitive Complexity from 38 to the 15 allowed.
 Payoffs.cs#L46: Rename parameter 'k' to 'price' to match the base class declaration.
 ChiSquareDistribution.cs#L52: Refactor this method to reduce its Cognitive Complexity from 29 to the 15 allowed.
 NormalDistribution.cs#L51: Rename parameter 'x' to 'v' to match the interface declaration.
 NormalDistribution.cs#L91: Rename parameter 'z' to 'v' to match the interface declaration.
 NormalDistribution.cs#L304: Refactor this method to reduce its Cognitive Complexity from 18 to the 15 allowed.
 NormalDistribution.cs#L419: Rename parameter 'x' to 'v' to match the interface declaration.
 NormalDistribution.cs#L534: Rename parameter 'x' to 'v' to match the interface declaration.


updated vep cache version

Updated required vep cache version

Matthias -- thanks again for keeping these up to date. When you update the configuration file you also need to manually increment the versions in the YAML files so bcbio knows to download the new versions on existing installs. I did that in a separate commit following on this so we should be good to go now. Thanks again.


feat: add ability to include dimensions per-metric

Been using the exclude feature that we put in yesterday (#28) and liking it, but found it would be useful to have a whitelist option as well. Tried to match inclusion/exclusion logic of INCLUDE_METRICS / EXCLUDE_METRICS.

Thanks for the quick review @aknysh. Sounds good, I'll make sure of that.

type StringSet map[string]bool

func set(slice []string) StringSet {
	boolMap := make(StringSet, len(slice))

	for i := range slice {
		boolMap[slice[i]] = true
	}

	return boolMap
}

func shouldIncludeDimension(dimName string, includeSet, excludeSet StringSet) bool {
	// blacklist first
	if excludeSet != nil && excludeSet[dimName] {
		return false
	}
	// if no whitelist, keep it as it passed the blacklist
	if includeSet == nil {
		return true
	}
	// otherwise, check the whitelist
	return includeSet[dimName]
}

func main() {
	println(shouldIncludeDimension("dim1", nil, set([]string{"dim1"})))                   // false, in exclude set
	println(shouldIncludeDimension("dim2", set([]string{"dim1"}), nil))                   // false, not in include set
	println(shouldIncludeDimension("dim3", set([]string{"dim1"}), set([]string{"dim2"}))) // false, not in include set yet also not in exclude set
	println(shouldIncludeDimension("dim1", nil, set([]string{"dim2"})))                   // true, no include set specified and not in exclude set
	println(shouldIncludeDimension("dim1", set([]string{"dim1"}), set([]string{"dim2"}))) // true, in the include set, not in exclude set
}

I think we should be good - here's a runnable example of the current logic.

Thanks @aknysh - I'm not sure if I'll have time soon but might in a few weeks.


Invalid paragraph validation error using the "Font  Size" toolbar menu item mixed with links

Example:
1.
(prefix) SomeText
(prefiex) Some more link text
``
2. make "link" a hyperlink

3. add new line(s) between the two lines and paragraph validation failes

PR: #2137

:+1:


Composer is not ready to perform different importation paths in the same instance

Many importation code rely on ComposerBase.get_module_path() to get the module path to import. The problem is that the method use the _MODULE_PYTHONPATH attribute to build the path.
But _MODULE_PYTHONPATH is designed to be overwritten by composer inheriter (django, textcontent, etc..) to directly point to a specific application module (settings, urls, etc..).
But now with collection order resolving, the composer always first go to scan application module (its __init__ file) to search for app parameter variable like DEPENDENCIES, but when using a ComposerBase inheriter like ComposeDjangoSettings the module path is not right since _MODULE_PYTHONPATH have been overwritten to search for APPLICATION_MODULE.settings and so it is unable to find the application parameter variables.
So currently, the application order resolving is broken with  ComposerBase inheriters.

This issue is solved in the v0.6.0, yet to release, that include a refactoring in composer.


Markdown table for part5 output results in the vignette does not display well

Probably a typo in the syntax
https://cran.r-project.org/web/packages/GGIR/vignettes/GGIR.html

now fixed in GitHub repo


Bundle issue on 'fast_blank'

Hi, In doing a bundle install most of the process works fine. But it trips up on 'fast_blank' for some reason. Below is the output. Any idea how to fix? Thanks!

Operating system: Linux Mint18
Last upstream commit (run git log --author="Robert Lord" | head -n 1): 2245e9e489f5a2e583890a118aadc1bab964eac
Browser version(s): Chome Version 66.0.3359.181
Ruby version (run ruby -v): 2.3.1p112

Solved the problem using
sudo apt-get install zlib1g-dev build-essential ruby-full


Since hashie/mash can be required alone, require its dependencies

Thanks for considering the fix here.
Just requiring mash is not optimal.
I did put a PR in to omniauth to request they require 'hashie' at the root level.
This is my suggestions for minimal changes.
If you prefer, I can move the logging into its own module (along with utils) - wasn't sure how much of a change you wanted.
Let me know how you would prefer to proceed.

To reproduce I used the following:
before:
be irb
require 'omniauth-google-oauth2'

NameError: uninitialized constant Hashie::Extensions::RubyVersionCheck::ClassMethods::RubyVersion

after:
be irb
require 'omniauth-google-oauth2'
puts "yay"
# => "yay"

Fixes #391
require ruby_version
ruby_version_check accesses ruby_version.
Since array is one of the first files required, this is sufficient to
resolve this class.
move logger
The only component that uses logger is mash. Move the logger to
where it is used.

I have a slightly cleaner version (re: logger) with a spec in #392. Closing in favor of that, can you please check this with me @kbrock ?


Do not display denial shares on Shared-with-me

Filter the denial shares (permission = 0) out of the Shared-with-me UI view.
Also, denials work regardless whether they are accepted or not, therefore there's no point to expose them.

@glpatcern I think it just needs rebasing on the latest cs3apis

@glpatcern I think it just needs rebasing on the latest cs3apis

Indeed, now I updated all dependencies (via VS Code) and all compiles, however many other dependencies were updated. Do you think it's fine to merge or better stick to the previous versions (except for the cs3apis bindings of course)?


ilmerge only targeting framework 4.0

Even though the referenced assemblies are for 4.5, it still is showing as 4.0 in dnSpy
Consider using dnlib to merge assemblies instead.

This may not actually be an issue, as the compressed file seems to run fine at 4.0 framework, even if the compressed file is using higher framework version.


PlayModeStateChange not found in UnityEditor namespace

I get this error when I put the VSCode.cs File into Assets/Plugins/Editor folder

Assets/Plugins/Editor/VSCode.cs(970,56): error CS0234: The type or namespace name PlayModeStateChange' does not exist in the namespace UnityEditor'. Are you missing an assembly reference?

Unity Editor 5.6.3xf1 Linux Personal on Ubuntu 16.04 and Visual Studio Code 1.20.1

I'm using the following modification locally for 2017.1.3
diff --git a/Assets/Editor/VSCode.cs b/Assets/Editor/VSCode.cs
index b516da6b64..68afdb6c2b 100644
--- a/Assets/Editor/VSCode.cs
+++ b/Assets/Editor/VSCode.cs
@@ -966,7 +966,11 @@ namespace dotBunny.Unity
         /// <summary>
         /// Executed when the Editor's playmode changes allowing for capture of required data
         /// </summary>
+#if UNITY_2017_2_OR_NEWER
         static void OnPlaymodeStateChanged(UnityEditor.PlayModeStateChange state)
+#else
+        static void OnPlaymodeStateChanged()
+#endif
         {
             if (UnityEngine.Application.isPlaying && EditorApplication.isPlayingOrWillChangePlaymode)
             {
@@ -980,8 +984,13 @@ namespace dotBunny.Unity
         [UnityEditor.Callbacks.DidReloadScripts()]
         static void OnScriptReload()
         {
+#if UNITY_2017_2_OR_NEWER
             EditorApplication.playModeStateChanged -= OnPlaymodeStateChanged;
             EditorApplication.playModeStateChanged += OnPlaymodeStateChanged;
+#else
+            EditorApplication.playmodeStateChanged -= OnPlaymodeStateChanged;
+            EditorApplication.playmodeStateChanged += OnPlaymodeStateChanged;
+#endif
         }

         /// <summary>

In Unity 2017.4.2 I'm getting
Assets/VSCode/Plugins/Editor/VSCode.cs(983,31): warning CS0618: UnityEditor.EditorApplication.playmodeStateChanged' is obsolete: Use EditorApplication.playModeStateChanged and/or EditorApplication.pauseStateChanged'

It's been fixed on the master branch.  Maybe we need a new release?

On the asset store? shiver

I was getting my unitypackage from a Github download thing that's not maintained anymore. Getting it from the asset store worked.


unset the END_HEADERS flag for stream concurrency test

Since HTTP/2 runs on a bi-directional stream, servers are allowed to send responses (and close the streams) as they receive the requests.
This PR drops the END_HEADERS flag used for the stream concurrency test so that it would not report false positives when used against such server implementations.

Oops. It's my mistake.
Thank you for your contribution!

Your welcome!
h2spec really helps me improving the conformance of H2O to the HTTP2 specification.  Thank you for creating it and for continuing to improve.


use fips flag to enable fips endpoints

See - https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html
This adds support for calling the FIPS compliant endpoints wherever possible.
This PR is an "all or nothing" approach which is largely convention based. If run with --fips all regions will attempt to use the fips compliant endpoints, which may or may not exist for the associated region(s).
The typical pattern is configuration based wherever the region is set, allow a corresponding endpoint to be set.
It seemed too difficult to maintain this in config.file since the service calls behind the region are abstracted.

Thank you for your new PR :) Appreciate your work and thoughts.

Thank you for your new PR :) Appreciate your work and thoughts.


remove gh package and update pnpm

This is for the licensing errors we're getting. Testing something with this PR

@dayhaysoos Nice one! So we can just remove the dependency and nothing breaks? We're not even using it?
If we're not using it, heck yeah, approve this PR and get the dep outta town.

@dayhaysoos With @angiejones and my ✅ you're clear to merge.


Fix 3D editor gizmos appearing in GI following default GI mode change

This fixes a gression from https://github.com/godotengine/godot/pull/60935. Your gizmos are no longer globally illuminated now :slightly_smiling_face:
Here's what it looked like:

I feel like there's a lot of redundant code in there, maybe there could be a helper method to create a gizmo MeshInstance with all the default parameters set?
Like:
RID Node3DEditorViewport::create_gizmo_instance(int p_idx, uint32_t p_layer) {
	RID rid = RS::get_singleton()->instance_create();
	RS::get_singleton()->instance_set_base(rid, spatial_editor->get_rotate_gizmo(p_idx)->get_rid());
	RS::get_singleton()->instance_set_scenario(rid, get_tree()->get_root()->get_world_3d()->get_scenario());
	RS::get_singleton()->instance_set_visible(rid, false);
	RS::get_singleton()->instance_geometry_set_cast_shadows_setting(rid, RS::SHADOW_CASTING_SETTING_OFF);
	RS::get_singleton()->instance_set_layer_mask(rid, p_layer);
	RS::get_singleton()->instance_geometry_set_flag(rid, RS::INSTANCE_FLAG_IGNORE_OCCLUSION_CULLING, true);
	RS::get_singleton()->instance_geometry_set_flag(rid, RS::INSTANCE_FLAG_USE_BAKED_LIGHT, false);
	return rid;
}

...

		move_plane_gizmo_instance[i] = create_gizmo_instance(i, layer);
		...

Needs checking that the code is indeed always the same.

Does this also fix gizmo plugins using create_material and create_handle_material, or were those not affected?

Does this also fix gizmo plugins using create_material and create_handle_material, or were those not affected?

I changed how gizmo instances are created in node_3d_editor_gizmos.cpp, so these should be fixed as well.
That said, from my testing, I only noticed the issue with editor origin lines, not individual node gizmos.

Thanks!


NTR: Liquid chromatography mass spectrometry evidence

Proposed name(s): Liquid chromatography coupled with tandem mass spectrometry evidence (one for use in manual assertion and one for use in automatic assertion)
Proposed definition: A type of spectrometry evidence using liquid chromatography to separate particles in solution, followed by identifying the amount and type of material entities present in a sample by fragmenting it and measuring the mass-to-charge ratio of the resulting particles.
Relationships: parent term - ECO:0001096 (mass spectrometry evidence)
References: PMID:30366363
NTR for CACAO class.
@jimhu-tamu, @sandyl27, @jrr-cpt

This term request is related to the discussion in issue 204.

Hi @jaydclark, @jimhu-tamu, @sandyl27, @jrr-cpt,
The following were added and pushed to GitHub.
NTR: liquid chromatography coupled with tandem mass spectrometry evidence
child of: mass spectrometry evidence
def: A type of mass spectrometry evidence where liquid chromatography is used to separate particles in solution, followed by fragmentation and measurement of the mass-to-charge ratio of the resulting particles to identify the amount and type of material entities present in a sample.
xref: PMID:30366363
NTR: liquid chromatography coupled with tandem mass spectrometry evidence used in automatic assertion
child of: liquid chromatography coupled with tandem mass spectrometry evidence
def: A type of liquid chromatography coupled with tandem mass spectrometry evidence that is used in an automatic assertion
NTR: liquid chromatography coupled with tandem mass spectrometry evidence used in manual assertion
child of: liquid chromatography coupled with tandem mass spectrometry evidence
def: A type of liquid chromatography coupled with tandem mass spectrometry evidence that is used in a manual assertion
Reaching out to @rctauber for help with OBI logical def.
Closing this ticket.
James


Add device-items count to output

Could be used to detect changes to devices.

@CoMPaTech the short solution was impossible to type so I went for a more difficult solution that needed no extra typing :)


Suggestion about your link for install.sh

This is actually not an issue, but a suggestion about your link for install.sh
Instead of doing this:
https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh
make one folder there on that server called 'latest' so the link would look like this:
https://raw.githubusercontent.com/nvm-sh/nvm/latest/install.sh
and then just always symlink the latest script into it when you make one.
Then we developers don't have to change our bash scripts to download the nvm.
Thank you in advance, and hope you will consider it.

This is intentionally not made easy. Just like package.json, you should be changing something every time you get a different version.
Duplicate of #400.


Stopping a receiving messages when an error occurs

When an error occurs in HandleUpdateAsync and HandleErrorAsync methods, bot stop receiving messages.
Is that how it's supposed to work?
https://github.com/TelegramBots/Telegram.Bot.Extensions.Polling/blob/52c1edf4ff13d97c0b6d5ed8a62631cceadd9210/src/Telegram.Bot.Extensions.Polling/DefaultUpdateReceiver.cs#L99-L115

There is a way to automatically restart the bot after an exception? I also have the same problem, but I'm receiving exception because a user has blocked the bot. I don't want to wrap each call to sendMessageAsync with a try-catch to avoid interruption, is there another way?

#1106

i have this problem,
my bot stop receiving messages for any problem and after restarting bot , it working fine again.
how i can restart bot automatically when an error occur ?

This issue was fixed in v21.3


ItemStack JSON don't work with all chars and empty lines (lore).

If you have empty line in lore, it will be changed to "" (visible in item lore).
If you use "" then you get empty line.
if you use """  then you get "Invalid Item" message.
If you use " then you get "Invalid Item" message.
So parsing " is broken.
(Not tested how that works with Achievements and Statistics)

Sounds like an issue with the conversion to JSON. Will look into it when I am back on my computer.


collapse/expand individual tables

the global expand/collapse tables feature is nice. But even better would be a feature to individually expand/collapse tables

I think the use case for this (being able to quickly scan the entire page) is pretty well satisfied by having the tables collapsed by default now. It also wouldn't be a trivial change for the UI.

Well, not exactly the usecase I was interested in: I like to see historical data (expanded table) of 1 vs various others last 2/3 updates. That makes it easier to analyse 1 particular table in comparison to others in a dashboard mode. That is impossible when all tables are expanded. Especially since they keep growing.


Initial circleCI settings

@miniway it's good to see you contributing again!
These files seem very specific to Treasure Data. Did you mean to submit them here?

sorry it's wrong repository


[Crash] Dedicated server crashes when mob walks on spike

crash-2021-08-02_03.22.31-server.txt
Running Forge 36.2.0 Dedicated server, latest version of Java with Miniutilities 1.1.3.
Created a Netherite spike and the moment a zombie walked over it, the server crashed with the above log.  If you need more from me just ask!

Same crash here_
https://paste.sunekaer.com/mikekadima.rb

fixed in dev


in PR checks Error: Failed to get ID token: Error message: Unable to get ACTIONS_ID_TOKEN_REQUEST_URL env variable

I was looking to add attestations to https://github.com/jqlang/jq/pull/3220 and it fails in the PR checks but works in the downstream repo.
The error seems to be the same as seen when not providing correct permissions, though they are present.
permissions:
  id-token: write
  contents: read
  attestations: write

https://github.com/jqlang/jq/actions/runs/12267835999/job/34228631988?pr=3220
The changes
https://github.com/jqlang/jq/pull/3220/files
Am i overlooking something here? As in this in an intended or unintended outcome.
I did not see anything gin the readme to suggest a solution
https://github.com/actions/attest?tab=readme-ov-file#actionsattest
What is the correct way forward to allow attestations in a pr action check or should this be avoided?
Thanks in advance.

I think this is a limitation of the id-token: write permission when operating in a forked repo. There is some more information here which may be useful.

So it's probably best to have some conditions to skip running on a pr move on?
I think there is some sense to have pr artifacts attested but probably not as important and release assets.


We should operate only on sub_fields

https://github.com/vrancurel/ntl/blob/78f391b2753649de1a488fcca7b36ac99c10c974/src/poly.h#L14
so this notion should disappear, especially gf2n should not create a poly with itselg (GF(2^n)) but with GF(2)
One day we should assert that the field is a prime field

👍


Reduce node count for e2e tests

The number of VMs created simultaneously for e2e tests can put a strain
on the vcenter env causing tests to fail.
This commit reduces the node count for testing features such as oidc, flux,
proxy and certain upgrade use cases to use single etcd, controlplane and worker nodes
to avoid test failures caused due to resource shortage.
Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: mrajashree
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [mrajashree]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


Update grafana-agent.service.erb

Template is pointing at mult-user rather than multi-user causing issues with starting the service.

@harrisondscott thanks!


APP-643 HC: Check minimum versions for Agent, Key Manager and POD

Support for check minimum versions for Agent, Key Manager and POD

LGTM. @dnsymphony could you review please?

@dnsymphony I'm closing this PR because the code is included on the PR #5


Bump Python version to 3.8

Some functionality we just introduced only exists in py38+. I forgot to change that in setup.py. py37's end-of-life is in 3 months either way: https://endoflife.date/python. I should have realized, because we run CI using py38.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 97.74%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
5449

Relevant Lines:
5575

💛  - Coveralls


In CouchDB Query regex with the caseless (?i) modifier is not case-insensitive for unicode strings

Description
In CouchDB Query (Mango) when you use the $regex operator with the (?i} modifier which is supposed to make the match case-insensitive, the query is not in fact case-insensitive if the regex contains a character that is not in the ASCII character repertoire.
This appears to be due to the way that unicode strings are represented internally and interaction with re:run/3.
Steps to Reproduce
$ curl -u admin:password -X PUT http://localhost:5984/test
{"ok":true}

$ curl -u admin:password -X POST http://localhost:5984/test -H "Content-Type: application/json" -d '
{
  "_id": "1",
  "data": "xxxöxxx"
}'
{"ok":true,"id":"1","rev":"1-96be014c47e090c7705b66c4b646d6f6"}

$ curl -u admin:password -X POST http://localhost:5984/test -H "Content-Type: application/json" -d '
{
  "_id": "2",
  "data": "xxxÖxxx"
}'
{"ok":true,"id":"2","rev":"1-903d0f61be4c2eda02c1bd72d3ba92bc"}

$ curl -u admin:password -X POST http://localhost:5984/test/_find -H "Content-Type: application/json" -d '
{
  "selector": {
    "data": {
      "$regex": "(?i)ö"
    }
  }
}'

{"docs":[
{"_id":"1","_rev":"1-96be014c47e090c7705b66c4b646d6f6","data":"xxxöxxx"}
],
"bookmark": "g1AAAAAyeJzLYWBgYMpgSmHgKy5JLCrJTq2MT8lPzkzJBYozGoIkOGASEKEsAEqyDRk",
"warning": "No matching index found, create an index to optimize query time."}

Expected Behaviour
The query selector should match both documents.
Your Environment

CouchDB version used: 3.1.0, Erlang 20
Browser name and version:
Operating system and version: Verified on RHEL and Debian

Additional Context
Tracing the call to re:run/3 in Mango reveals that the data string from the 2 documents is treated differently:
(<0.31742.114>) call mango_selector:match({[{<<"data">>,{[{<<"$regex">>,<<40,63,105,41,195,150>>}]}}]},{[{<<"_id">>,<<"2">>},
  {<<"_rev">>,<<"1-903d0f61be4c2eda02c1bd72d3ba92bc">>},
  {<<"data">>,<<120,120,120,195,150,120,120,120>>}]},#Fun<mango_json.cmp.2>)
(<0.31742.114>) call mango_selector:match({[{<<"$regex">>,<<40,63,105,41,195,150>>}]},<<120,120,120,195,150,120,120,120>>,#Fun<mango_json.cmp.2>)
(<0.31742.114>) call re:run(<<120,120,120,195,150,120,120,120>>,<<40,63,105,41,195,150>>,[{capture,none}])

(<0.31706.114>) call mango_selector:match({[{<<"data">>,{[{<<"$regex">>,<<40,63,105,41,195,150>>}]}}]},{[{<<"_id">>,<<"1">>},
  {<<"_rev">>,<<"1-96be014c47e090c7705b66c4b646d6f6">>},
  {<<"data">>,<<"xxxöxxx">>}]},#Fun<mango_json.cmp.2>)
(<0.31706.114>) call mango_selector:match({[{<<"$regex">>,<<40,63,105,41,195,150>>}]},<<"xxxöxxx">>,#Fun<mango_json.cmp.2>)
(<0.31706.114>) call
re:run(<<"xxxöxxx">>,<<40,63,105,41,195,150>>,[{capture,none}])

The string "xxxöxxx" is represented as <<"xxxöxxx">>, but the string "xxxÖxxx" is represented as <<"xxxÖxxx"/utf8>>. re:run/3 behaviour differs depending on the string representation (in Erlang 20, at least):

data
regex
options
result

<<"xxxöxxx">>
<<"(?i)ö">>

match

<<"xxxöxxx"/utf8>>
<<"(?i)ö">>

nomatch

<<"xxxöxxx"/utf8>>
<<"(?i)ö"/utf8>>

match

<<"xxxöxxx">>
<<"(?i)ö"/utf8>>
unicode
nomatch

<<"xxxÖxxx"/utf8>>
<<"(?i)ö"/utf8>>
unicode
match

<<"xxxÖxxx"/utf8>>
<<"(?i)ö"/utf8>>

nomatch

<<"xxxöxxx">>
<<"(?i)ö">>
unicode
** exception error: bad argument

Whether the string is represented as utf8 encoded or not seems to stem all the way from couch_doc:to_json_obj/2 so I wonder if anything else in Mango would be impacted by this.

Excellent write up. Unfortunately the Erlang shell and its weirdo UTF-8 behavior caused you more pain that it helped. The issue is that Erlang's displaying binaries and interpreter binaries differently that you'd normally expect.
A quick test to see that in action is to print out the actual binary values to see how things have been interpreted (Running on Erlang 22 locally):
Also for clarity, ö is 192 186 and Ö is 192 150.
Forms = [
    <<"xxxöxxx">>,
    <<"xxxöxxx"/utf8>>,
    <<"xxxÖxxx">>,
    <<"xxxÖxxx"/utf8>>,
    <<"(?i)ö">>,
    <<"(?i)ö"/utf8>>
],
lists:foreach(fun(F) ->
    io:format("~w~n", [F])
end, Forms).

<<120,120,120,246,120,120,120>>
<<120,120,120,195,182,120,120,120>>
<<120,120,120,214,120,120,120>>
<<120,120,120,195,150,120,120,120>>
<<40,63,105,41,246>>
<<40,63,105,41,195,182>>

You'll notice that the /utf8 flag is merely telling the shell to correctly interpret unicode characters that have been typed into the shell. So the table from above is just showing this albeit in a round about fashion.
Internally since we know everything has gone through Jiffy we know everything is valid UTF8, so it then becomes a question of providing that flag and whether or not its something we should do since it could be a behavior change.
Although, we can also control that flag from the pattern as well, so this works as expected:
curl -u admin:password -X POST http://localhost:5984/test/_find -H "Content-Type: application/json" -d '
{
  "selector": {
    "data": {
      "$regex": "(*UTF8)(?i)ö"
    }
  }
}'
{"docs":[
{"_id":"1","_rev":"1-96be014c47e090c7705b66c4b646d6f6","data":"xxxöxxx"},
{"_id":"2","_rev":"1-903d0f61be4c2eda02c1bd72d3ba92bc","data":"xxxÖxxx"}
],
"bookmark": "g1AAAAAyeJzLYWBgYMpgSmHgKy5JLCrJTq2MT8lPzkzJBYozGoEkOGASEKEsAErJDRs",
"warning": "No matching index found, create an index to optimize query time."}

thansks @davisp. Specifying (*UTF8) seems like a reasonable workaround for users that need to match unicode. Closing.


fix(curriculum): rework Project Euler 73

Reworks challenge to use argument in function call.
Adds more tests.
Adds solution.
Uses MathJax to improve look of fractions and other math notation.
Related to #40896.

Checklist:

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the main branch of freeCodeCamp.
[x] I have tested these changes either locally on my machine, or GitPod.

Hey @gikf - just out of curiosity, have you been copy/pasting any of the solutions you have been adding to any of these PR's from elsewhere?

@moT01 I have solutions for first hundred saved from when I was solving them on fCC. If it's still working (couple of times, for some reason, they were not efficient enough anymore) I'm using it, after trying to make it more clear and readable. If challenge, to not timeout first, requires using algorithms or math beyond my comprehension, I'm basing code on the solutions made by clever people understanding it, I think I've been adding link to the original solution in comment. For example https://www.mathblog.dk/project-euler-solutions/ has such very math powered (and very efficient) solutions. But again, after trying to make it more readable rather than straight copy/paste.

Thanks - yea, that should be fine then 👍


Add support for specifying raw category warnings along with those with modifiers.

Closes #27.

LGTM

yep new release sounds good, thanks for the fix!


Preflight Checklist

[x] I agree to follow the Code of Conduct that this project adheres to.
[x] I have searched the issue tracker for an issue that matches the one I want to file, without success.
[x] I am not looking for support or already pursued the available support channels without success.

Version
1.1.3
Installation Type
Official Docker Compose
Service Name
Deploy(DongTai-deploy)
Describe the details of the bug and the steps to reproduce it
docker-compose启动后，admin/admin登录失败
Additional Information
No response
Logs
No response

This problem should be that the data of dongtai-mysql service has not been initialized, and it needs to wait for 2-3 minutes.


Disambiguation of Documentation

Hey. This looks awesome, but confuses me. There is an OLAP cube engine named Mondrian. Is this tool designed to work with that http://mondrian.pentaho.com ? I would be super stoked if the Mondrian I used had a better IDE (and did graph design).

Mondrian is a visualization engine that has no relationship with the OLAP engine


Can't install write_stylus-209

Describe the bug
The installation of write_stylus fails because the file https://oyra.eu/write/icon.tar.gz does not exists.
To Reproduce
Steps to reproduce the behavior:

run nix-env -i write_stylus

Expected behavior
The package is installed.
Additional context
Note that the version 209 is outdated, and the version 300 is available upstream.

Hello this was actually already fixed by https://github.com/NixOS/nixpkgs/commit/e92866d74f3a795dc4b6f50d53815b50dab01b37#diff-d91944d0c56d0ad196f89f861a306072


Consider guaranteeing vendor list sort order

The vendor consent vecs will almost always be used for membership checks. If encoding is ever added to this crate, guaranteeing ascending order and non-repeats through modification makes that much easier. You'd hope that people would encode their TC strings as sorted (always true for a bitfield, but possibly not true for ranges), but they might not, so it would be good to guarantee it.
Using a BTreeSet instead of a Vec guarantees ascending order and uniqueness, even if people modify it for re-encoding later. Otherwise, a sorted vec is an improvement as it allows binary searches. An &mut self method to sort all the internal vendor lists would be minimally invasive.

Hello @clbarnes,
There are currently no plans to implement encoding for this crate.
We’re open for contributions if you (or anyone else reading this) want(s) to implement TCString encoding for this crate.
I agree with you that bitfield lists should be sorted, but I (personally) would prefer a solution which is optional either via an option or crate feature.
With the current implementation it is not possible to pass options, be it a callback function that can be used to sort the bitfield lists or an option to signal the implementation, it should sort these lists.
So it would require a rewrite of the user-facing API as well.

The sorting was a bit of an aside - switching out the vec of vendor IDs for a btreeset would coincidentally guarantee order while primarily making containment checks more efficient (order being a useful tiebreaker vs a hashset). Containment checks are the entire purpose of vendor list so it seems like a use case worth prioritising. And serde serialises them exactly the same way so no changes needed there!
I only mentioned sorting as well because in a read-only workflow as currently supported, a sorted vec is probably about as good as a btreeset because you can binary search it.

The overhead for using the BTreeSet are probably also beneficial for the allowed_vendors and disclosed_vendors not just the vendors_consent and vendors_li_consent.
I will have to do some testing whether this overhead is worth placing behind an option / feature or if it should be the new default behavior.

The overhead for using BTreeSet is roughly 39%.

baseline (without BTreeSet)
TCString V2 (core only) time:   [346.08 ns 346.62 ns 347.23 ns]
Found 3 outliers among 100 measurements (3.00%)
  3 (3.00%) high mild

TCString V2 (core + disclosed vendors)
                        time:   [1.5183 µs 1.5202 µs 1.5225 µs]
Found 5 outliers among 100 measurements (5.00%)
  3 (3.00%) high mild
  2 (2.00%) high severe

TCString V2 (core + disclosed vendors + allowed vendors)
                        time:   [2.9701 µs 3.0044 µs 3.0392 µs]
Found 24 outliers among 100 measurements (24.00%)
  21 (21.00%) low severe
  2 (2.00%) low mild
  1 (1.00%) high severe

TCString V2 (core + publisher tc)
                        time:   [574.17 ns 574.92 ns 575.73 ns]
Found 6 outliers among 100 measurements (6.00%)
  1 (1.00%) low severe
  1 (1.00%) low mild
  3 (3.00%) high mild
  1 (1.00%) high severe

TCString V2 (core + disclosed vendors + allowed vendors + publisher tc)
                        time:   [779.89 ns 780.60 ns 781.31 ns]
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high mild

with BTreeSet
TCString V2 (core only) time:   [361.11 ns 361.55 ns 362.04 ns]
                        change: [+4.4258% +4.7797% +5.1308%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 10 outliers among 100 measurements (10.00%)
  1 (1.00%) low mild
  7 (7.00%) high mild
  2 (2.00%) high severe

TCString V2 (core + disclosed vendors)
                        time:   [2.2539 µs 2.2557 µs 2.2576 µs]
                        change: [+48.396% +48.672% +48.959%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

TCString V2 (core + disclosed vendors + allowed vendors)
                        time:   [5.0106 µs 5.0166 µs 5.0229 µs]
                        change: [+65.906% +67.190% +68.548%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 19 outliers among 100 measurements (19.00%)
  1 (1.00%) high mild
  18 (18.00%) high severe

TCString V2 (core + publisher tc)
                        time:   [715.91 ns 717.07 ns 718.23 ns]
                        change: [+24.120% +24.467% +24.861%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 5 outliers among 100 measurements (5.00%)
  4 (4.00%) high mild
  1 (1.00%) high severe

TCString V2 (core + disclosed vendors + allowed vendors + publisher tc)
                        time:   [1.1880 µs 1.1980 µs 1.2080 µs]
                        change: [+51.470% +52.049% +52.751%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 11 outliers among 100 measurements (11.00%)
  1 (1.00%) high mild
  10 (10.00%) high severe

So this would need to be placed behind an opt-in feature.
The code used in the latter benchmark can be found here: https://github.com/advancedSTORE/lib_tcstring/commit/09a54c9dbeacfa4e228a2b97ce8aaab09b4b4e52


controlplane cannot change the password required pref

controlplane cannot change the password required pref for when screensaver comes on or when computer sleeps.

same as https://github.com/dustinrue/ControlPlane/issues/421 I beliefe. Still waiting for this to be fixed too.

This is an issue that comes up a lot and unfortunately I don't have a good answer for you. Remember that ControlPlane is tapping into undocumented features of OS X to get certain tasks done. As far as I know, there is no other utility out there that can modify the screensaver password setting in the background and for good reason. Apple doesn't want people doing that.
It's very likely I'll be removing this feature from ControlPlane in the near future.


Environments are not created automatically (Prod &Dev) on Azure DevOps Pipelines

Describe the bug or the issue that you are facing
3.10 Under Pipelines, select Environments and ensure both "Prod" and "Dev" environments are created. Create the "Dev" environment manually, if necessary. -> I followed deployment guide. It worked fine for CLIv2-bicep-Classical parameters but there is no env created automatically.
Steps/Code to Reproduce
Follow the deployment guide till 3.10 by selecting CLIv2-bicep-Classical parameters
Expected Output
3.10 step should be validated but there is no env on AzureDevOps pipelines.
Versions

Which platform are you using for deploying your infrastrucutre?
Azure DevOps (ADO)
If you mentioned Others, please mention which platformm are you using?
No response
What are you using for deploying your infrastrucutre?
Bicep
Are you using Azure ML CLI v2 or Azure ML Python SDK v2
Azure ML CLI v2
Describe the example that you are trying to run?
3.10 step is not happening when I follow the deployment guide in order.

This issue is under investigation. Autocreate behavior for environments when referenced changed last October. Investigating to determine fix or workaround. For now, deployment guide has been updated (https://github.com/Azure/mlops-v2/pull/108) to include guidance for manually creating environments prior to running the infra pipeline.

Closing this. @yelizkilinc  I hope the updated documentation has resolved your issue.

@yelizkilinc I hope this has resolved your issue.


RequestsDependencyWarning with urllib3 and chardet

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

[X] I'm asking a question and not reporting a bug or requesting a feature
[X] I've looked through the README
[X] I've verified that I have updated yt-dlp to nightly or master (update instructions)
[X] I've searched known issues and the bugtracker for similar questions including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue

Please make sure the question is worded well enough to be understood
I'm asking specifically about the RequestsDependencyWarning.  It shows up on every invocation.  Is this an issue with my environment?

Output of `pip list`
Package                Version
---------------------- --------------------
aiodns                 3.0.0
aiohttp                3.7.0
aiohttp-socks          0.4.1
aiosignal              1.2.0
async-timeout          3.0.1
attrs                  21.4.0
beautifulsoup4         4.11.1
Brotli                 1.0.9
cchardet               2.1.7
certifi                2022.12.7
cffi                   1.15.0
chardet                3.0.4
charset-normalizer     2.0.12
command-not-found      0.3
dataclasses            0.6
dbus-python            1.2.16
distro                 1.4.0
distro-info            0.23+ubuntu1.1
dnspython              1.16.0
elastic-transport      8.1.2
elasticsearch          8.2.0
fake-useragent         0.1.11
frozenlist             1.3.0
geographiclib          1.52
geopy                  2.2.0
googletransx           2.4.2
gpg                    1.13.1
idna                   2.8
iotop                  0.6
language-selector      0.1
Markdown               3.1.1
meson                  0.53.2
multidict              6.0.2
mutagen                1.46.0
netifaces              0.10.4
numpy                  1.22.3
packaging              20.3
pandas                 1.4.2
pip                    20.0.2
pycares                4.1.2
pycparser              2.21
pycrypto               2.6.1
pycryptodomex          3.17
Pygments               2.3.1
PyGObject              3.36.0
pyparsing              2.4.6
PySocks                1.7.1
python-apt             2.0.1+ubuntu<IP_ADDRESS>
python-dateutil        2.8.2
python-socks           2.0.3
pytz                   2022.1
PyYAML                 5.3.1
requests               2.27.1
requests-unixsocket    0.2.0
schedule               1.1.0
setuptools             45.2.0
six                    1.14.0
soupsieve              2.3.2.post1
ssh-import-id          5.10
twint                  2.1.21
typing-extensions      4.2.0
ubuntu-advantage-tools 8001
ufw                    0.36
unattended-upgrades    0.1
urllib3                1.26.9
websockets             10.4
wheel                  0.34.2
yarl                   1.7.2
yt-dlp                 2023.2.17

Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[ ] If using API, add 'verbose': True to YoutubeDL params instead
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
/usr/lib/python3/dist-packages/requests/__init__.py:89: RequestsDependencyWarning: urllib3 (1.26.9) or chardet (3.0.4) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
[debug] Command-line config: ['-vU']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version<EMAIL_ADDRESS>from yt-dlp/yt-dlp-master-builds [8056a3026] (zip)
[debug] Python 3.8.10 (CPython x86_64 64bit) - Linux-5.4.0-163-generic-x86_64-with-glibc2.29 (OpenSSL 1.1.1i  8 Dec 2020, glibc 2.31)
[debug] exe versions: ffmpeg 4.2.7, ffprobe 4.2.7
[debug] Optional libraries: Crypto-broken 2.6.1, certifi-2019.11.28, requests-2.22.0, sqlite3-3.31.1, urllib3-1.26.9
[debug] Proxy map: {}
[debug] Request Handlers: urllib
[debug] Loaded 1810 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-master-builds/releases/latest
Latest version<EMAIL_ADDRESS>from yt-dlp/yt-dlp-master-builds
yt-dlp is up to date<EMAIL_ADDRESS>from yt-dlp/yt-dlp-master-builds)

Some of your libraries are severly outdated. If you cannot uodate them through the system package manager (apt update && apt upgrade), use yt-dlp_linux or install it using pipx instead (pipx install 'yt-dlp[default]').

Thank you.  Aptitude showed no packages to update but I was able to update packages with this command:
pip list --outdated | awk '{print $1}' | tail -n +3 | xargs -n1 pip install -U


Implements task queue in the project

Checklist

[x] I have read the Contribution & Best practices Guide.
[x] My branch is up-to-date with the Upstream development branch.

Short description of what this resolves:
Implements task queue in the project, which enables multiple web app generation without the loss of request from the user.
Changes proposed in this pull request:

Implemented queue backed by redis server.

Fixes #1856
Link to test the changes: https://openeveweb.herokuapp.com/
API endpoints for testing:

v1:
https://eventyay.com/api/v1/events/200
https://eventyay.com/api/v1/events/275

v2:
https://raw.githubusercontent.com/fossasia/open-event/master/sample/FOSSASIASummit2017

@geekyd please review.

@agbilotia1998 Please check the codacy issues

@geekyd can you please mention the test data for which event generation failed, as I tried to handle all the errors and tested multiple cases for Error in read contents of zip and the events are getting generated after this.

Yes sure, I will add the documentation for the changes.

@agbilotia1998 I used the API you provided in the description.

@geekyd I tested for several kinds of error and it worked fine. Also, I checked Heroku logs but couldn't find anything there as well, it would be really helpful if you mention the error encountered. Thanks :smile:

@geekyd can you please help me here, I tested for null zip upload as well but the app was working fine. Maybe when you tested it resulted into memory exceed for Heroku, actually this endpoint https://eventyay.com/api/v1/events/275 is for FOSSASIA Summit 2018 which is a large event, so on Heroku account with no dynos enabled, multiple times generating this app may result into the memory related error.

@geekyd can you please help me here, I tested for null zip upload as well but the app was working fine. Maybe when you tested it resulted into memory exceed for Heroku, actually this endpoint https://eventyay.com/api/v1/events/275 is for FOSSASIA Summit 2018 which is a large event, so on Heroku account with no dynos enabled, multiple times generating this app may result into the memory related error.
@ParthS007 @pradeepgangwar @srv-twry can also checkout if I am missing something?

@agbilotia1998 It works for me. :+1:

@geekyd @ParthS007 @mayank8318 I have resolved the conflicts, please check.
Link to test the changes: https://openeveweb.herokuapp.com/


I cant remove cookie

use API remove method u give us is  completely ineffective；in Cookies.js remove function i find when u delete this.cookie[name] ,this.cookie is change but after execute document.cookie = _cookie2.default.serialize(name, '', finalOptions); document.cookie is still there,how can i fix it ?

see #110 (and PR #112)

This should be fixed in 2.1.1. Thanks for reporting!


powerrename 文件夹检索增强

Description of the new feature / enhancement
在读取文件目录前添加选项判断是否只更改父文件夹名称，以降低不必要的性能占用和可能的子文件名称的错误名称更改
Scenario when this would be used?
仅需更改父文件夹名称时，若文件夹内包含大量文件夹，powerrename占用大量性能和内存，此时添加选项判断是否读取子文件夹内容可以增强工作效率和降低不必要的性能占用

Supporting information
No response

35分钟过去了

40min

45min

最终

Is this asking the utility to rename a half a million files?

/needinfo

Is this asking the utility to rename a half a million files?

我的需求仅仅为更改最顶层文件夹的名称，然而在过程中必须完全检索所有文件，导致大量计算和性能浪费产生并且计算完成后也几乎不可用
Google trs：My requirement is only to change the name of the topmost folder. However, all files must be completely retrieved in the process, resulting in a lot of calculation and performance waste, and it is almost unusable after the calculation is completed.


Handle composite builds

This PR extracts source sets for composite builds - previously, included builds would be ignored.
I've had to get rid of the unique display name code and go with project [sourceset] for all target display names.  It just gets far too complicated for little reward when composite builds are added into the mix.
Composite builds have to be handled in the GradleAPIConnector as each included build requires a new ProjectConnection

I've had to get rid of the unique display name code and go with project [sourceset] for all target display names. It just gets far too complicated for little reward when composite builds are added into the mix.

👍. If it's possible, would you mind separate the display name handling change to another PR?

👍. If it's possible, would you mind separate the display name handling change to another PR?

I've submitted another PR.  When that's merged I'll rebase this PR onto it.

rebased

rebased

I tried this change with a sample in Gradle document site: https://docs.gradle.org/current/samples/sample_composite_builds_basics.html, but it's not working:
Caused by: org.gradle.internal.resolve.ModuleVersionNotFoundException: Could not find org.sample:number-utils:1.0.

It happens when calling sourceSet.getCompileClasspath().getFiles() for the project :app

I may have to rethink how this works.  I hadn't taken into account dependency substitutions.
Even though it looks like a maven dependency, implementation 'org.sample:number-utils:1.0' in project app actually refers to the project number-utils in the other included build.

Hi @Arthurm1
I'd be happy to take a look at this PR and potentially contribute. Let me know if there's anything I can do to help move this forward.

@Tanish-Ranjan You're welcome to take over.  The main thing is that dependencies into included builds aren't handled as stated here

Okay, I'll start from there.

@Tanish-Ranjan Hi.  Are you working on this?  No rush - I just noticed that it's a bit tricky to create the project dependencies

Hi @Arthurm1, what kind of issue are you facing?

what kind of issue are you facing?

No issue really - I just noticed that we'll have to move the project dependency collection out of SourceSetsModelBuilder and into GradleAPIConnector.  Because the included builds have to be retrieved separately on different ProjectConnection.
The archive output jars will also have to be returned by the SourceSetsModelBuilder.
I was going to have a go but wasn't sure if @Tanish-Ranjan was working on it.

I see. Just to heads up, @Tanish-Ranjan is a participant for the event GSoC (Google Summer of Code). He will work on this open source project during the summer. The composite build is one of the tasks he would like to take.
@Tanish-Ranjan, please let us know if you would like to keep doing this task. If yes, @Arthurm1 is a very experienced and active contributor to this project, and I believe he can definitely give you some guidance for this issue.
To me, I hope we can address this issue ASAP since this will increase the Gradle project import success rate in our VS Code extensions. My plan is to fix it in June.

Yes @jdneo, I'm happy to take on this task for fixing composite builds. I've been preparing myself by researching composite builds and familiarizing myself with the relevant code. I'd also like to reach out to @Arthurm1 for his guidance. Given his experience with the project, I am confident we would be able to resolve this issue soon.

No issue really - I just noticed that we'll have to move the project dependency collection out of SourceSetsModelBuilder and into GradleAPIConnector. Because the included builds have to be retrieved separately on different ProjectConnection.
The archive output jars will also have to be returned by the SourceSetsModelBuilder.

May I ask how to handle the problem mentioned in https://github.com/microsoft/build-server-for-gradle/pull/122#issuecomment-2031099285? Since we still need to get the normal dependencies.

May I ask how to handle the problem mentioned in #122 (comment)? Since we still need to get the normal dependencies.

If the archive dirs are returned in the GradleSourceSet (currently they are not) then in GradleAPIConnector we can check whether they feature in any project's compile classpath.  If they do then we can add a project dependency between those 2 projects.
The includedBuilds also need to be returned in GradleSourceSets for each projectdir so GradleAPIConnector can retrieve source sets from all the included builds before the project dependencies can be worked out.
I described this to Tanish on Slack and do have a PR showing it works but I was leaving it to Tanish.

It's resolved by https://github.com/microsoft/build-server-for-gradle/pull/154.
Thank you @Arthurm1 for all the effort and help you provided for this issue.


GitHub workflows that include pytest are still known as pylint. Change the name to 'python testing' or 'testing'

The current behavior or issue
The current testing workflows use the term pylint when running both pylint and pytest commands. This is misleading
The steps taken to reproduce the behavior or issue, or specify a location where the steps were recorded
To reproduce: take a look at https://github.com/AgPipeline/transformer-rgb-indices/runs/995443455?check_suite_focus=true
Down near the bottom is the pytest command
Expected behavior
The workflows should use the terminology of "Testing", or similar, when running disparate tests
Screenshots

Add other supporting information that may be useful
GitHub workflows
Completion criteria
This is done when:

[x] Workflows using pylint and pytest reference "testing" instead of pylint or pytest

Plot clip: https://github.com/AgPipeline/transformer-plotclip/pull/16
Soil mask: https://github.com/AgPipeline/transformer-soilmask/pull/19
Canopy Cover: https://github.com/AgPipeline/transformer-canopycover/pull/22
RGB Indicies: https://github.com/AgPipeline/transformer-rgb-indices/pull/9
Same PRs as #235


fix: retry put if cloud return 499.

Some cloud providers may send response with http code 499 if the connection not send data in 1 min.
So we retry the put request if this happens.

make format done.

I submit another PR to finish what you want. You can upgrade megfile to 3.0.5 and try new features. So I close this PR.
In 3.0.5, there are some new environments for setting retry times, you can read changlog to get them.


Include git header files

Hello,
I have a project that needs to be installed in the rtd virtualenv so that I can scrape it for information to add to the documentation
(http://harpoon.readthedocs.io/en/latest/docs/configuration.html#Image)
This project includes git support and I've recently moved from using dulwich to pygit2.
The problem with this is that pygit2 requires the libgit header files to compile (https://readthedocs.org/projects/harpoon/builds/3961730/).
Seeing as git is such a common thing, can we please add it's header files to rtd?
Thanks
Stephen.

@delfick not sure, but I think the quick answer is "No, they won't add the git headers file" as they explain in the FAQ: http://docs.readthedocs.io/en/latest/faq.html#i-get-import-errors-on-libraries-that-depend-on-c-modules
I will close this issue as invalid. Feel free to re-open it if you consider.


Add Markdown tables support

Overview:
This PR introduces a small yet impactful change to enhance the parsing of tables within our markdown files. By integrating the turndown-gfm-plugin, we can now successfully parse HTML tables, ensuring that they are converted into clean and readable markdown format.
Key Changes:
•	Integrated turndown-gfm-plugin to handle the parsing of HTML tables.
•	Improved markdown output by converting HTML tables into well-structured markdown tables, enhancing readability and consistency.

Impact:
This update will make our markdown files look better by ensuring that tables are properly formatted and displayed. It should be particularly useful for documents where tabular data is frequently used.

Awesome find, @dscherdi!  This'll help out a lot of folks who had been looking for this functionality, and I had no idea it could be solved so easily.
This will go out in the next release, and if things go well, I expect to be submitting that for review by Chrome's team sometime this weekend (maybe even tonight).


Update UploadFile docs to reference using curl for CLI file upload

Update the UploadFIle docs to reference using curl to upload files from the CLI, ie:
curl http://localhost:8080/upload -F name=john -F avatar=@"C:\path\to\some\picture.png"

This will upload the file, as a form, and allow Save-PodeRequestFile to work.
Extra:
Investigate doing the same with Invoke-RestMethod, but it seems it doesn't work properly.

I've been trying to upload files using Invoke-WebRequest method but no success so far... it seems that using -Form @{avatar =Get-Item avatar.png fails and gives server error as the avatar key wasn't provided...

I've updated the UploadFile docs to show how to upload a file using curl, but also Invoke-WebRequest!
I found a gotcha bug with how Invoke-WebRequest sets the form-data boundary, and fixing that now allows it to work 😄

Very nice! Thank you for this!! I would like to rely on PS as much as possible instead of using other binaries for tests like curl!
How can I check the dev docs instead of main docs? I'd like to keep watching the edge work that you're doing.
I'm waiting for the next release 🚀


Improve document event binding for colum resizer

feat(column-resize): improve document event binding for resizer

Use case: the table is mounted into a non-src iframe

This PR will not be merged as it has conflicts and is not clear about what functionality is it providing.

thanks @gary-menzel I think it is fair to do that.


[SPARK-36926][SQL] Decimal average mistakenly overflow

What changes were proposed in this pull request?

This bug was introduced by https://github.com/apache/spark/pull/33177
When checking overflow of the sum value in the average function, we should use the sumDataType instead of the input decimal type.
Why are the changes needed?

fix a regression
Does this PR introduce any user-facing change?

Yes, the result was wrong before this PR.
How was this patch tested?

a new test

@karenfeng @maropu @viirya @gengliangwang

Test build #143831 has started for PR 34180 at commit 413f668.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48344/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48344/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48344/

Merging to master/3.2

@cloud-fan This PR can't be merged to branch-3.2 directly. Could you open a backport PR?

Test build #143831 has finished for PR 34180 at commit 413f668.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143831/

Thanks for fixing this quickly @cloud-fan !