nwjs: 0.82.0 -> 0.83.0

Description of changes
Closes #276393
The original PR from r-ryantm didn't update all hashes and, as a result, the sdk version didn't build.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

10 packages built:
betaflight-configurator emuflight-configurator gridtracker inav-configurator nwjs nwjs-sdk onlykey pinegrow pinegrow6 popcorntime
Result of nixpkgs-review pr 276397 run on x86_64-linux 1

10 packages built:

betaflight-configurator
emuflight-configurator
gridtracker
inav-configurator
nwjs
nwjs-sdk
onlykey
pinegrow
pinegrow6
popcorntime


Inline chat UI glitches on Web

open an untitled on Web (I was using codespace)
ctrl/cmd+i

The inline chat widget input seems to have conflicts with the title bar. It seems like an zindex issue.

This seems to be a duplicate of https://github.com/microsoft/vscode-copilot/issues/4658 which is supposedly fixed

Thanks for creating this issue! We figured it's covering the same as another one we already have. Thus, we closed this one as a duplicate. You can search for similar existing issues. See also our issue reporting guidelines.
Happy Coding!


Why multiply 2 on the partial value?

Hi,
thanks for this excellent work.
I have one question about the source code.
https://github.com/thughost2/Padam/blob/master/Padam.py#L86
Why did you multiply 2 on the partial value? You did not do that in the paper.

Thank you for your interests in our work!
Note that the "denom" is already the square root of the second order momentum (see
https://github.com/thughost2/Padam/blob/master/Padam.py#L78).
Therefore the power of 2p gives you exactly what we described in the paper. We implement in this way simply want to follow the framework of the original amsgrad implementation in Pytorch and make minimum changes.

I may check the paper again.
thank you.


First Name + Last Name Tagging not working

You can try a react-native-controlled-mentions that is not rich in configuration yet but enough stable for use in projects.
https://www.npmjs.com/package/react-native-controlled-mentions


[HPRO-765] Update to Composer 2, resolve conflicts

Q
A

Target Release?
Next available

Bug fix?
❌

New feature?
❌

Database migration?
❌

New configuration?
❌

Composer updates?
✅

NPM updates?
❌

Jira Ticket(s)
HPRO-765

Summary
Updates the Dockerfile to explicitly pull in Composer 2, which has some breaking changes for a few of our dependencies.
Instructions for testing
While the prior version called for :latest, it always assumed that tag was current unless you ran docker pull composer:latest manually. You can avoid a full Docker rebuild with:
$ docker exec -it healthpro-web composer self-update
$ docker exec -it healthpro-web composer install

Note: Must be running docker-compose up at the time to run commands in container.

docker run us.gcr.io/gae-runtimes/buildpacks/php74/builder:php74_20210203_7_4_14_RC00 composer --version
Composer version 1.10.17 2020-10-30 22:31:58


checkValidity for CRL certificates.

Hi, first of all, wow... great job with this library :)
Now, there is a way to make the checkValidity method works with CRL certificates? as i can see in the object created, the values for the dates are the following:
    UTCTIME : 2019-05-08 22:00:35 +0000
    UTCTIME : 2019-06-07 22:00:35 +0000

and with a CRL, the values are:
    SEQUENCE {
        UTCTIME : 2014-01-20 15:45:35 +0000
        GENERALIZEDTIME : 2060-12-31 15:24:38 +0000
    }

Is there any chance to get the dates correctly for the CRL??
Thank you!

Hi!
What do you specifically mean with "CRL certificate" ? A CRL file is not a certificate, although it is encoded with ASN.1.
The checkValidity method is specific for X.509  certificate, so it's not going to work with a CRL, if you for example tried to instantiate a X509Certificate with a CRL file.
CRL should have a different class to help parse a CRL structure.

Sorry for the misunderstanding, effectively i mean CRL file.
Is in your plans to do a parser for this kind of file on future releases? I will much appreciated.
Thanks again for your work.
Regards.

I haven't plan a release for CRL.
Anyhow if you just need the CRL dates interval, you can just use this:
let asn1 = try ASN1DERDecoder.decode(data: crlData)
let dateFrom = asn1.first?.sub(0)?.sub(3)?.value as? Date
let dateTo = asn1.first?.sub(0)?.sub(4)?.value as? Date

Thanks @filom i will try it :)


Machine learning project to predict total power of the Farm

Project Request
Machine learning project to predict total power of the Farm using wave energy converter dataset.

Define You

[x] GSSOC Participant
[x] Contributor

I would like to work for this project under gssoc'23


Generating swagger documentation in support for  Retrofit

Hi Team ,
Im looking for generating swagger documentation in support of Retrofit . How can i generate for it ?

Can some one reply on this open issue.


Feature request: support for Bedrock (via litellm)

Litellm supports function calls; not sure how well they work but this would enable the use of ChatDBG with other AI providers, notably Amazon AWS Bedrock.

Thanks for using litellm - do you have any feedback on how we can improve ?
Regarding function calling here's how to enable it for non openai models https://docs.litellm.ai/docs/completion/function_call#function-calling-for-non-openai-llms

cc @emeryberger

I actually just added litellm for part of this tool yesterday:
https://github.com/plasma-umass/ChatDBG/blob/c7ea3525d5ded127be64580058a3b16df968c8ca/src/chatdbg/assistant/lite_assistant.py#L103-L112

How can we improve it?

We already contribute and report issues when we run into them 👍  Thanks!
Pretty sure we only use LiteLLM for ChatDBG API calls at this point so Bedrock, Ollama, etc. should be supported out-of-the-box. @stephenfreund can you confirm and close this?

That's right.  Should all work just fine.


[AMBARI-23614] Move NameNode wizard is stuck on Review step after enabling federation

What changes were proposed in this pull request?

Infinite spinner is displayed
'Next' button is disabled
JS error is thrown: {{app.js:31501 Uncaught TypeError: Cannot read property 'indexOf' of undefined}}

How was this patch tested?
UI unit tests:
21516 passing (25s)
48 pending

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/1911/
Test PASSed.


Bubble sort algorithm

Example Problem:
given an array of n integers: [4, 2, 5, 6, 8], we sort it using the bubble sort sorting algorithm
Example Solution:
The expected output for above array will be a sorted array: [2, 4, 5, 6, 8]
Further explanation:
We use the bubble sort algorithm which repeatedly traverses the array and swaps integers based on their magnitude to sort the whole array.
Would you like to be assigned to this:
yes

Hello @arpitg24, May I work on this?

Hi, I'd like to work on this issue

Hello, I send a pull request for this issue

This is not what this repository is for.


Pass rate value into entity

Hi,
I have successfully use your bundle and I want to use rate value from your bundle and save it in a field of my table in the db through my entity as an integer.
So how can I save the rating in that field?
Thanks in advanced.

Create an entity for example, 'Rating'
Create a form, with field 'RatingType' for example:
"$builder
->add('rating', RatingType::class, [
'label' =>'Rating',
])
;"
Once the form is submitted, the value will be saved to your database


Add support for overlays

This would allow for the support of non-visible utilities like a file picker directly from flet.

I've added overlay support and subsequently added a file picker widget in my forked repo. If the project is still active, I am willing to pull.


Opensearch-dashboard download reporting columns are not in order.

Hi Team,
We want to download the dashboard report based on field search.
It was showing correctly in dashboard, but after download the report some columns are not in sync, EX: 3rd column showing in 11th column.
Can anyone please tell us why columns are not sync, if we want to sync those columns what need to do from our end.
Thanks,
Anil

[Triage]
Transferring this to https://github.com/opensearch-project/OpenSearch-Dashboards repo.
@bbarani @peterzhuamazon


initial searchprotocol implementation in cpp

@vekterli please review
@geirst FYI

@geirst you might want to have a look at the actual search integration code, I focused primarily on the protobuf parts

still some issues with cmake/protobuf it seems:
CMake Error at /usr/share/cmake3/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
Could NOT find Protobuf (missing: Protobuf_INCLUDE_DIR)
Call Stack (most recent call first):
/usr/share/cmake3/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
/usr/share/cmake3/Modules/FindProtobuf.cmake:595 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
functions.cmake:62 (find_package)
searchlib/src/vespa/searchlib/CMakeLists.txt:40 (vespa_add_target_package_dependency)


Update for blast radius containment

Readme needs updating of course. 🙂

Merging as is - thanks @gunnargrosch


Allow interception of Completion list and its transformation

Ability to modify a Completion list right before it is rendered seems like missed opportunity, that would allow Plugin dev to change way completions are displayed to the user, this can be done in IntelliJ IDEs rather easily, and would be great if it was possible for VS-Code
I searched the whether it is possible similarly to IntelliJ IDEs to intercept the list of completions, inspect them and then modify the final list the way you like before it gets rendered.
There could be two approaches

same as IntelliJ - have CompletionItemProvider receive 'Completion list so far' and allow it to be inspected/manipulated right there
have a dedicated CompletionInterceptor that gets the list before it is rendered, so transformation is possible before rendering

same as IntelliJ - have CompletionItemProvider receive 'Completion list so far' and allow it to be inspected/manipulated right there

We run them (mostly) in parallel and therefore a "list so far" approach doesn't work

have a dedicated CompletionInterceptor that gets the list before it is rendered, so transformation is possible before rendering

Unsure about that because it enables an extension to "break" another extension (by bugs) and that's something we aren't aiming for.

First is understandable.
Second - I don't thin processing/filtering result is breaking something, this would affect the output the way user wishes, and add some flexibility, autocompletion is the part you see all the time, and it would certainly open options to make it more convenient.
Another option would be to have 'interceptable completion action' standing separately that would allow such modification, and you could use it if you like.


error when connecting TortoisePanel

Perhaps the same technique as SQLAlchemy can be used.
Please refer to my response.
https://github.com/mongkok/fastapi-debug-toolbar/issues/16


sign-root-targets for mnm678

Summary

Release Note

Documentation

looks good
STAGED METADATA

Outputting metadata verification at /Users/kommendorkapten/git/root-signing/repository...

Verifying targets.json...
	Contains 1/3 valid signatures from the current staged metadata
	targets version 9, expires 2024/09/06

Verifying root.json...
	Contains 1/3 valid signatures from the current staged metadata
	Contains 1/3 valid signatures from the previous root
	root version 9, expires 2024/09/06

Verifying registry.npmjs.org.json...
	Success! Signatures valid and threshold achieved
	registry.npmjs.org version 3, expires 2024/09/06


PackageReference requires all sources to be available

Fixes https://github.com/NuGet/docs.microsoft.com-nuget/issues/1716

Docs Build status updates of commit 6e91f75:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/consume-packages/Package-References-in-Project-Files.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


fix: warn message for scss option

Fixes: #24

Updated dependencies
Fixed warning message when scss option was enabled

After updating, I met this error:
Nuxi 3.0.0                                                                                14:20:41

 ERROR  Error while requiring module nuxt-swiper: Error [ERR_MODULE_NOT_FOUND]: Cannot find module<EMAIL_ADDRESS>imported from /app/node_modules/nuxt-swiper/dist/module.mjs

 ERROR  Cannot find module<EMAIL_ADDRESS>imported from /app/node_modules/nuxt-swiper/dist/module.mjs

  at new NodeError (node:internal/errors:400:5)
  at finalizeResolution (node:internal/modules/esm/resolve:326:11)
  at moduleResolve (node:internal/modules/esm/resolve:945:10)
  at defaultResolve (node:internal/modules/esm/resolve:1153:11)
  at nextResolve (node:internal/modules/esm/loader:163:28)
  at ESMLoader.resolve (node:internal/modules/esm/loader:842:30)
  at ESMLoader.getModuleJob (node:internal/modules/esm/loader:424:18)
  at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:77:40)
  at link (node:internal/modules/esm/module_job:76:36)

I strongly believe that this path /Users/cpreston/Documents/projects should not be there.

@nuvolapl I am deeply sorry about this. v0.1.8 should fix this.
Thanks for the heads up!


using weakly supervised dataset

Hello Yassine,
I have some questions about adding weakly labeled images.
First, I generated pseudo labels using irn, and the 10.582 .png files were saved correctly. Then, I set use_weak_labels: true in the configs file. However, when I run the main train.py script, it gives me the following error: (see image as well)
TypeError: forward() missing 2 required positional arguments: 'curr_iter' and 'epoch'
(with use_weak_labels: false I do not have this issue)

Second, how exactly are the pseudo labeled images utilized? When using the weakly labeled method, the model still trains on the labeled and unlabeled datasets as well? With each iteration the same amount of labeled, unlabeled, and pseudo labeled images? Some images will be in the labeled and the pseudo labeled data set, or some in both the unlabeled and pseudo labeled data sets?
I hope my questions are clear. I would appreciate your help.
ps: how many epochs did the model require to achieve the results published in the paper?

Hi @SuzannaLin, thank you for your interest,
I guess you are using abCE loss, and in this case, we must also pass curr_iter and epoch to compute the weighting. I just pushed a correction, you'll simply need to use the CE loss.
The correction is simply:
loss_weakly = sum([self.sup_loss(outp, target_ul, ignore_index=self.ignore_index) for outp in outputs_ul]) / len(outputs_ul)

to
 from utils.losses import CE_loss
........
loss_weakly = sum([CE_loss(outp, target_ul, ignore_index=self.ignore_index) for outp in outputs_ul]) / len(outputs_ul)

In terms of the usage of weak labels, after generating the pseudo-pixel-level labels, the aux. decoders & encoder are trained with a loss MSE(aux. pred, main pred) + CE(pseudo labels, aux.pred). So we simply use the generated labels to train the aux. decoder using the standard CE.
I hope I answered your question.

Yes! It's working :)
thank you!

Some more questions:
When using the weakly labeled setting, the weakly labeled images are the same as the unlabeled dataset?
Using my own images, I thought about creating 3 different datasets: fully labeled, weakly labeled, and unlabeled dataset. But in the train.py script, there is only the unlabeled dataset (which includes the weakly-labeled images).
Do I need to generate weak labels for all the unlabeled images if I use this setting?

Hi,
As you mentioned, the weakly labeled examples are also considered unlabeled, so they are used for both weakly loss and unsupervised loss.
If you want to use three distinct datasets, labeled, unlabeled, and weak, you'll need to modify the code a little bit where the unlabed and weakly labeled are seperate, but it won't take that much time I think.


Add Switch task implementation

Implements https://github.com/serverlessworkflow/specification/blob/main/dsl-reference.md#switch

Hi @fjtirado could you assign it o me?

@mcruzdev Im already working on it

@mcruzdev Knowing that you want to work on the reference implementation, once I have completed this one and error handlng (both affects the current TaskExecutor interface) I would open more issues and discuss which ones can you help me with. Im thinking on authorization support for http calls, but, in order to avoid collisions, I have to finish this one first.


dart2js mirrors: metadata missing from some variables in a variable declaration

I think this is a dart2js issue but not 100% sure. Consider the following program:

import 'dart:mirrors';
main() {
  for (var field in reflect(new FooBar()).type.variables.values) {
    print('field ${field.simpleName} has metadata '
        '${field.metadata.map((m) => m.reflectee).join()}');
  }
}
const hello = 'hello';
class FooBar {
  @hello var x, y, z;
}

On VM it prints:
field Symbol("y") has metadata hello
field Symbol("x") has metadata hello
field Symbol("z") has metadata hello
On dart2js+d8 it prints:
field Symbol("x") has metadata hello
field Symbol("y") has metadata
field Symbol("z") has metadata
Note that "y" and "z" are missing metadata.
Since you can only specify metadata on the declaredIdentifier in the variableDeclaration (http://www.dartlang.org/docs/spec/latest/dart-language-specification.html#h.55kzc4r0q21p) , it makes me think metadata should apply to all of the identifiers.

Mirrors is no longer supported in Dart2JS.


Cannot generate CHANGELOG

Describe the bug
Trying to generate changelog but get:
Received issues: 154
Pull Request count: 118
Filtered pull requests: 52
Filtered issues: 0
Fetching events for issues and PR: 52
Fetching closed dates for issues: 52/52
Warning: PR 220 merge commit was not found in the release branch or tagged git history and no rebased SHA comment was found
Warning: PR 217 merge commit was not found in the release branch or tagged git history and no rebased SHA comment was found
Warning: PR 216 merge commit was not found in the release branch or tagged git history and no rebased SHA comment was found
Warning: PR 210 merge commit was not found in the release branch or tagged git history and no rebased SHA comment was found
Warning: PR 191 merge commit was not found in the release branch or tagged git history and no rebased SHA comment was found

To Reproduce
Follow the release instructions at https://github.com/apache/arrow-ballista/blob/master/dev/release/README.md
Expected behavior
Additional context

@alamb Any idea why this is happening?
I am testing this in my branch at https://github.com/apache/arrow-ballista/pull/183

fyi @avantgardnerio @iajoiner this is blocking me from creating a release candidate - any idea what the issue may be?

I am running the following command in branch https://github.com/apache/arrow-ballista/pull/183:
CHANGELOG_GITHUB_TOKEN=<token> ./dev/release/update_change_log-ballista.sh apache/master 0.8.0 0.7.0

Never mind - I have it working now somehow


AVRO-4046: [PHP] Handling of default values

What is the purpose of the change
This pull request fixes the behavior that values are accidentially overwritten by default values from the reader's schema.
Detailed description: AVRO-4046
Verifying this change
This change added tests and can be verified as follows:

Added test that serializes some data and ensures that the data survives deserialization.

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? not applicable

Issues: AVRO-4046

@Fokko can you take a look at this?

Thank you, @MichelHartmann !


istio-config-getting-started.yaml path fixed

istio-config-getting_started.yaml was changed to istio-config-getting-started.yaml and not updated in the documentation. The result was giving a 404: Not Found error.
Signed-off-by: Marcus Vaal<EMAIL_ADDRESS>

Cheers @mvaal :slightly_smiling_face:
/lgtm
/approve
/ok-to-test


Document need for unique count names

I noticed in #43 that @boxofrox mentions the need to publish counts with a unique name. I didn't notice that in the docs (forgive me if I overlooked it). It didn't occur to me after reading the README that the counts would require a unique name per client to be unique per client. If that is the case, would be nice to mention it (or make it bolder if it's already there) in the docs.
Happy to submit a PR if you'd like one.

Sure thing.

Wait... I realized that unfortunately the names had to be unique when listing groups and trying to show the number of items in each group. But reading @chmac 's comment I wonder, do names also have to be unique per client?

@ubald, my mind's a bit hazy tonight, and I may be overlooking something here, so take what I say skeptically and verify it makes sense.
Currently, the documented way of publishing counts in the browser works through Meteor subscriptions.  Using this mechanism, publish-counts avoids having to keep a server-side cache of all counters for every client.  Instead, we use the Meteor.publish callback to create a closure for a new counter variable and wire it up to the calling client through the this instance of the Meteor.publish callback function.  It just works.
So to answer your question, names don't need to be unique per client... normally.  If one client has multiple counts, then each count for that client should have a unique name.
There is an edge case/bug where you subscribe to the exact same count twice and it doesn't matter that the names are the same because the counts are calculated with the exact same algorithm, so even though the record in the Counts collection is written more than once per update, the count still works out to the correct value.
Abnormally, you might want to pursue something like in #63 where a user wanted to use Counts.get on the server, too.  (see related commented)  In this case, the server side would need to expose the count variables hidden behind closures... probably through a global registry on the server... which would be used to store all counts for all clients.  Names would have to be unique across clients to avoid collisions.
@chmac, the docs don't mention the need for unique names.  You're welcome to submit a patch to add that.
It was likely overlooked for the fact that with many simple counts (e.g. Counts.publish(this, 'posts',  Posts.find()); Counts.publish(this, 'comments', Comments.find());), it's fairly obvious that two counts with the same name could only provide one of those counts, not both.
Where it isn't so obvious, and the reason for the comment in #43, is when you have subscription parameters that reduce the result set of a subscription, and then a single client subscribes multiple times to the same subscription with different parameters.
// One client will view a page with three groups of posts, each group with
// a different category. That one client will subscribe three times to
// 'postsByCategory' and expects 3 unique counts, one for each column.
// Each 'name' in publish-counts has one value, so you must make three
// names for three values.

if (Meteor.isServer) {
  Meteor.publish('postsByCategory', function (category, limit) {
    Counts.publish(this, 'posts-by-' + category, Posts.find({ category: category }));
    if (limit)
      return Posts.find({ category: category });
    else
      return Posts.find({ category: category }, { limit: limit });
  });
}

Sorry for the excessively long post.

Thanks @boxofrox


Adding more evaluation metrics

I've seen the project, and it would be better if more evaluation metrics are added such as R-squared, Mean squared logarithmic error (MLSE) and coefficient of variation to evaluate the performance of the algorithms.

@Varun-malviya5  I appreciate your curiosity to this repository, so I encourage you to come up with a new issue.
All the best


SAS Deployment Operator is getting Failed

While running ansible command, I am getting the following error:
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Logging into cr.sas.com for user 9CV4JS failed - 500 Server Error for http+docker://localhost/v1.41/auth: Internal Server Error ("Get "[https://cr.sas.com/v2/\](https://cr.sas.com/v2/)": unauthorized: incorrect username or password")"}
Though it's taking the credentials from the secrets.yaml
Anyone is having an idea on this or faced the same then please update me with the resolution. Thanks in advance!

What release version of DAC are you using? Could you check out this Troubleshooting tip and validate that you have the required version of the Docker daemon running on the workstation you are running DAC from? Here is the text from the solution of that troubleshooting entry:

As of release 6.0.0, it's required that if you are running this project using Ansible directly on your workstation, it needs Docker to be installed and the executing user should be able to access it. This is so that we can consume the sas-orchestration tool, which is available as a Docker image to generate the SASDeployment Custom Resource file


Bump package to 1.0.2

The Tasks tutorial blows up on the second example without the changes from 1f21350d28ff5562caf141bec2ec2ea7fa0f3712. Spending an hour trying to figure out what I'd done wrong (only to find out it wasn't my mistake for once!) makes the tutorial a bit of an underwhelming experience. Please bump the package version to include the changes so that we (both myself and others) don't keep downloading a broken version.

Thanks for opening an issue for this, should be released now!

Thanks!


Dependencies not installed:  nothing provides *** needed by ****

Snakemake errors: Many dependencies not being downloaded or installed?
(tobias_snakemake_env) user@pop-os:~/TOBIAS_snakemake$ snakemake --configfile example_config.yaml --use-conda --cores 6
Building DAG of jobs...
Creating conda environment environments/uropa.yaml...
Downloading and installing remote packages.
CreateCondaEnvironmentException:
Could not create conda environment from /home/user/TOBIAS_snakemake/environments/uropa.yaml:
Command:
mamba env create --quiet --file "/home/user/TOBIAS_snakemake/.snakemake/conda/bd9e0dc4179dc2ab76c5ce61377855a3_.yaml" --prefix "/home/user/TOBIAS_snakemake/.snakemake/conda/bd9e0dc4179dc2ab76c5ce61377855a3_"
Output:
Encountered problems while solving:
  - nothing provides r 3.2.2* needed by r-ggplot2-2.0.0-r3.2.2_0
  - nothing provides r 3.2.2* needed by r-magrittr-1.5-r3.2.2_0
(tobias_snakemake_env) user@pop-os:~/TOBIAS_snakemake$ snakemake --configfile example_config.yaml --use-conda --cores 6
Building DAG of jobs...
Creating conda environment environments/uropa.yaml...
Downloading and installing remote packages.
CreateCondaEnvironmentException:
Could not create conda environment from /home/user/TOBIAS_snakemake/environments/uropa.yaml:
Command:
mamba env create --quiet --file "/home/user/TOBIAS_snakemake/.snakemake/conda/bd9e0dc4179dc2ab76c5ce61377855a3_.yaml" --prefix "/home/user/TOBIAS_snakemake/.snakemake/conda/bd9e0dc4179dc2ab76c5ce61377855a3_"
Output:
Encountered problems while solving:
  - nothing provides r 3.2.2* needed by r-ggplot2-2.0.0-r3.2.2_0
  - nothing provides r 3.2.2* needed by r-magrittr-1.5-r3.2.2_0
  - nothing provides libcurl 7.79.0 h2574ce0_0 needed by curl-7.79.0-hea6ffbf_0

Hi @c2b2pss,
Can you share which OS you are using? linux/mac? I can't reproduce this on debian, so I am thinking this might be a system specific / conda issue.

Hi,
I am using Linux (Pop OS/Ubuntu-based). I am just running this in the TOBIAS yaml generated environment.  Does this and subsequent steps pull the necessary dependencies?
BTW, I am running TOBIAS without snakemake and that works fine.  Except I don't know how to run multiple files rather than entering commands for each file (I have 7 bams+peaks). I presume snakemake would run all these one after another???

Yes, when you run snakemake with "--use-conda" it automatically builds individual conda environments for the run. However, it seems that r=3.2.2 is not available for the system, and it can therefore not build the environment. To solve this you might try:
Option 1: If you are able to, you can try to install all requirements for the whole pipeline, and then run snakemake without "--use-conda". The main requirements are:

samtools
uropa
bedtools
macs2
igvtools
tobias

Option 2: R is only needed for specific options in the pipeline, so if you are just running default parameters, you can remove the requirements from the file at environments/uropa.yaml to look like:
name: uropa_env

channels:
  - bioconda
  - conda-forge
  - defaults

dependencies:
  - python
  - numpy <1.16
  - pysam >=0.15.3
  - pip
  - pip:
    - uropa

The pipeline is meant to deal with multiple .bam-files such that you do not have to run them manually (like you say). But other than that, the same TOBIAS tools are used.

Hi c2b2pss,
I am not from Looso lab but thought to contribute to the thread as I have also encountered a problem similar to the one mentioned above.
For me it had to do with the conda set up (of channel priority)
I ran this conda command:
conda config --set channel_priority disabled
then could install TOBIAS_snakemake version.
Thanks
Deep


Missing LICENSE file

Hi, I'd like to test out these asys bindings. I'm guessing the code in this repository is MIT, since the plan is for it to go into the Haxe std, but just wanted to double check as there's currently no LICENSE file.

Yes, MIT is fine, I've just committed a license.

Thanks! And thank you for your work on this project.


iOS (Mobile Safari) - Keyboard covers up messages after orientation c…

Summary
This change addresses an issue for iOS Mobile Safari where, after screen orientation is changed, the keyboard overlays most recent messages in the window and user cannot scroll down to see them. The window will now hide the keyboard on orientation change for this scenario.
Ticket Link
N/a
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] Ran make check-style to check for style errors (required for all pull requests)
[x] Ran make test to ensure unit and component tests passed
[ ] Added or updated unit tests (required for all new features)
[ ] Needs to be implemented in mobile (link to PR or User Story)
[ ] Has server changes (please link)
[ ] Has redux changes (please link)
[ ] Has UI changes
[ ] Includes text changes and localization file (.../i18n/en.json) updates
[ ] Touches critical sections of the codebase (auth, posting, etc.)

Setup Test Server label detected. Spinmint test server created if build succeeds (checks pass and no conflicts with base branch).

Spinmint test server created at: https://i-0b06bf62da850a399.test.spinmint.com
Test Admin Account: Username: sysadmin | Password: sysadmin
Test User Account: Username: user-1 | Password: user-1
Internal IP: <IP_ADDRESS>
Instance ID: i-0b06bf62da850a399

Spinmint test running for more than 7 days. This test server was terminated.

Setup Test Server label detected. Spinmint test server created if build succeeds (checks pass and no conflicts with base branch).

Spinmint test server created at: https://i-077bd06d1abeab0bf.test.spinmint.com
Test Admin Account: Username: sysadmin | Password: sysadmin
Test User Account: Username: user-1 | Password: user-1
Internal IP: <IP_ADDRESS>
Instance ID: i-077bd06d1abeab0bf

Spinmint test server destroyed

@csduarte Please help rebase so we can merge, thanks!

@saturninoabril This should be ready now.


clatd: init at 1.6

Description of changes
A CLAT component for 464XLAT: https://github.com/toreanderson/clatd
I will do NixOS module later on and proper testing.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

I built a test for clatd some time ago. Feel free to use. https://github.com/hax404/nixpkgs/commit/6be04700ef7f5441b44102add9dca95b681f7c45

Please ignore this PR given the existence of superior https://github.com/NixOS/nixpkgs/pull/240982 containing a clatd component AFAIK.

Reopened because Jool is actually inferior in practical usecases.


Make guardConfig and interceptorConfig nullable

The official docs provide an example where these values are null:

The type definitions, however, do not allow this currently:

@EmanH Thanks for your contribution. The official docs you linked was intended to be a step-by-step tutorial for setting up an Angular app with msal-angular. If you see later on in the docs, both the msal-guard and the msal-interceptor are configured (instead of left as null) and in fact, this is required for msal-angular to run properly. We are aware that the way this tutorial is set up creates unnecessary confusion, and are prioritizing a fix for this tutorial to be more user-friendly.
Linking a previous issue with this same feedback: #4795
@EmLauber @mmacy for docs visibility.

@jo-arroyo Ok I see. Yes, I think the docs are a bit misleading there. It looks like a working code snippet while not obvious that its not.

Closing as change should be to documentation rather than code.


Upgrade to Python 3.9

In order to use a version of Python with longer support, this commit upgrades the build to Python 3.9.
This is currently running in Sandbox.

Closing until I've found a solution to a problem with the updater.


Removing redundant metadata; making span robust to NaN

The metadata dict we were returning from ModelView was redundant with
the table returned by CampusPower.metadata. This changes the API:
campus = CampusPower("...endpoint...")
# old
model, md = campus.model([building], ['Meter'])
# new
model = campus.model([building], ['Meter'])

In the implementation of span, the interpolation function returned by 'interp1d' cannot use any NaN values. I've changed span so that it only builds the interpolation function from non-NaN values. Seems to be working OK

Thanks


Missing Software License

I'd like to use your software and its nuget packages, but it lacks a license. Would you be able to add one? Preferably MIT License or other OSS? Thanks for this cool project!

Hey @rkd-xom, thanks for opening up this issue. I have just pushed an update making explicit the use of the MIT license in this project :)

Wow, so fast @rkd-xom . Many thanks and looking forward to using the project in the real world.


Typo fix in PromiseConstructor interface

This doesn't seem like a typo fix. Can you provide more context for what is wrong with the current types and how your solution fixes it?
Just at first glance, I think this breaks compatibility for non-promise values passed into Promise.all().

Your right I was using it wrong.


fix: strict dependencies to fix build

hi,
at some point i came up with necessity to build js-libp2p, but found that it's not buildable.
this change was the cause (libp2p/interfaces)
if semver rules were followed this shouldn't have happened.

Thanks for the PR.  The relevant changes needed to support the updated interfaces have been done as part of #1206 so if you pull from master and reinstall the node deps it should build successfully again.

if semver rules were followed this shouldn't have happened.

This project is in a bit of a weird space while we transition to TypeScript - the interfaces have grown organically over time so are subject to a bit of churn while we apply the learnings of the past few years to them so there will be some breakage.  The good news is this breakage is contained to the @libp2p/* modules and libp2p@next - the previous 0.36.x release tree is unaffected because it doesn't use and @libp2p/* modules.
Long story short: as soon as #1178 is merged and 0.37.x is released, semver will be strictly followed again.  In the interim you should use the 0.36.x release (see releases for the latest tag) and not libp2p@next or master.

thank you for the news and the explanation, it makes things clearer


Remove version package dependency in favour of built-in sys

This removes the need for using another package just to get "2.7" from a system.

The intention is to get the version of datacube software. The correct behaviour has been restored in 9f6bc7430a4e9d174160abd2720d6c29380392f8.
A system will be put in place to make sure failed documentation builds don't go unnoticed for so long. Thank you for bringing it to our attention.

Thanks @v0lat1le.


revisit the “old” integrated assignment

keep it up to date,
part1: go from EM to String/genemania;
review the Chipseq assignment and update it;
clusterProfiler: look at vignette and be ready to answer questions

updated integrated assignment with result figures from latest versions of all softwares

completed.


Add new testing dataset

The dataset layout has been updated and we need to have something for testing

Closed by #36


Issues uploading files to API View

Hi, unfortunately we are encountering issues uploading files to create API views for JavaScript and Python.

In the Python SDK, we generally generate the API Views as follows:

Download .whl from pypi.org from the latest released version, this becomes the base revision
Checkout the feature branch, call python setup.py bdist_wheel -d [dest_folder] to generate the .whl file as the diff revision

However, due to a combination of factors (lots of changes going in since the officially released version on pypi.org, etc.), we decided to go with an alternate workflow to cut down on the noise in the APIView. The only change is to step 1:

Checkout the main branch, call python setup.py bdist_wheel -d [dest_folder] to generate the .whl file as the base revision

This alternate workflow was fine for file-share but I am running into issues with it for datalake (didn't even upload the revision to datalake yet because the base .whl not working properly)
Some other observations is that it seems to be hanging just on the .whl files generated through python setup.py bdist_wheel -d [dest_folder], as I have attempted to do the "original" workflow on datalake and the .whl from pypi.org works fine, but again I run into the issue of the feature branch .whl not working as seen here


kubelet unable to authenticate to API server

What happened:
My worker nodes are unable to register with my cluster due to an authentication issue,
I checked out v20191213, built the AMI (1.14) into the us-west-2 region, copied it into us-east-2, and tried to create a node group in my cluster using the CloudFormation template in the repository. The nodes are not getting registered with the cluster, i.e. running kubectl get nodes doesn't show them.
I signed into a node and looked around a little bit, and the problem seems to be an authentication issue. Running systemctl status -l kubelet shows messages like this:
E0203 23:07:15.496759    3824 kubelet_node_status.go:94] Unable to register node "ip-x-x-x-x.us-east-2.compute.internal" with API server: Unauthorized

There are also similar messages like this, occurring for several different types of resources (i.e. Service, Pod)
E0203 23:10:24.695072    3824 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.CSIDriver: Unauthorized

And messages like this are occurring in my cluster's API server log:
E0203 23:07:15.496759       1 webhook.go:107] Failed to make webhook authenticator request: unknown
E0203 23:07:15.496779       1 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, unknown]

What you expected to happen:
Worker nodes to authenticate and register properly with the API server.
How to reproduce it (as minimally and precisely as possible):
Checkout v20191213, build 1.14, copy to us-east-2, and try to create a node group in a cluster in that region using the CloudFormation template int the repository.
Anything else we need to know?:
I've also tried:

Tweaking Makefile so that I can build v20191213 directly into us-east-2, instead of copying from us-west-2.
Building with v20190906 (1.13)
Building with master (1.14)

I get the same results in either case. I'm not sure if the problem is specific to us-east-2 -- I have not tried setting up a cluster in any other region.
My cluster was recently set up using the Getting Started Guide (using AWS Console) here:
https://docs.aws.amazon.com/eks/latest/userguide/getting-started-console.html
I am able to create Managed Node Groups using the EKS console without any problems.
Environment:

AWS Region: us-east-2
Instance Type(s): t3.medium
EKS Platform version: eks.7
Kubernetes version: 1.14
AMI Version: v20191213
Kernel: Linux 4.14.158-129.185.amzn2.x86_64 #1 SMP Tue Dec 24 03:15:32 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Release information:

BUILD_TIME="Mon Feb  3 22:03:11 UTC 2020"
BUILD_KERNEL="4.14.154-128.181.amzn2.x86_64"
ARCH="x86_64"

Problem solved. I needed to add an entry for the node instance role created by CloudFormation to the aws-auth ConfigMap, as described here


Switch signUrl method in google-cloud-storage to use v4 as default signature version

V2 is discouraged in favor of V4.  We should figure out a way to migrate users to use V4, and at some point in the future (TBD), we should make the default signature version V4 instead of V2.

A migration plan is defined internally but pending v4 switch timeline.

I mostly made this issue so that code TODOs could reference it :)
Internal doc, for Google folks referencing this bug: go/v2-to-v4-migration-plan


Add a gh workflow for frontend tests

Purpose
To run admin-ui and privacy-center frontend tests in CI 🎉
Changes

Adds a github workflow file to run frontend unit tests
In the future Cypress tests can go in here too

Checklist

[ ] Applicable documentation updated (guides, quickstart, postman collections, tutorial, fidesdemo, database diagram.
If docs updated (select one):

[ ] documentation complete, or draft/outline provided (tag docs-team to complete/review on this branch)
[ ] documentation issue created (tag docs-team to complete issue separately)

[ ] Good unit test/integration test coverage
[ ] This PR contains a DB migration. If checked, the reviewer should confirm with the author that the down_revision correctly references the previous migration before merging
[ ] The Run Unsafe PR Checks label has been applied, and checks have passed, if this PR touches any external services

Ticket
Fixes https://github.com/ethyca/fidesops/issues/508

@elliotbonneville do you think you could go and fix the things CI is complaining about?

Looks good to me as soon as we can get those failing tests figured out.


cbmbasic: init at unstable-2022-12-18

Description of changes
Homepage: https://github.com/mist64/cbmbasic

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 261153 run on x86_64-linux 1

1 package built:

cbmbasic


-cells/--expected_cells still required when -wl provided?

This isnt a huge issue, but I gather you're near to a release so I thought I'd bring it up. So far as I can tell, the tool requires the user to enter "-cells", even when a whitelist (-wl) is provided. I know there's a workaround to enter "-cells 0", but that's not especially intuitive and ideally one would not need to supply -cells when -wl is used. Not a huge problem, but thought i'd mention it.

Hey Ben, thanks for the issue.
I've looked at this in the past as well and here is the problem I'm faced with.
Here are two ways I looked at it:
A. Ask the user for the number of cells or whitelist. Use the size of the whitelist when provided
B. Ask for the user for the number of cells and the whitelist (if exists).
If the user provides the whitelist as a "short reference" coming from the subsetted mRNA data, (A) makes the most sense. it requires one less input from the user and the data comes out as you need it. (B) still works but the user needs to add one more variable.
If the user provides the "full whitelist", as you might do for 10xv3, (A) will catch a lot of cells we don't really want with really low umi counts, etc... Since I'm not doing any filtering in CITE-seq-Count so that user is in control, we might end up ina place where we process, correct cell barcodes and UMIs of cells we don't care about. (B) Solves this issue by having the user filter the data themselves.
I had an option C in mind where the user would provide a "full whitelist" + either cell number or a short subset of cells as list. But this option is more explanation, more potential misunderstanding and I'd like to keep the number of questions down if it comes down to bad design.
For now, I would rather keep option B, but let me know if you have an idea I might have missed.

Hey Ben, thanks for the issue.
I've looked at this in the past as well and here is the problem I'm faced with.
Here are two ways I looked at it:
A. Ask the user for the number of cells or whitelist. Use the size of the whitelist when provided
B. Ask for the user for the number of cells and the whitelist (if exists).
If the user provides the whitelist as a "short reference" coming from the subsetted mRNA data, (A) makes the most sense. it requires one less input from the user and the data comes out as you need it. (B) still works but the user needs to add one more variable.
If the user provides the "full whitelist", as you might do for 10xv3, (A) will catch a lot of cells we don't really want with really low umi counts, etc... Since I'm not doing any filtering in CITE-seq-Count so that user is in control, we might end up ina place where we process, correct cell barcodes and UMIs of cells we don't care about. (B) Solves this issue by having the user filter the data themselves.
I had an option C in mind where the user would provide a "full whitelist" + either cell number or a short subset of cells as list. But this option is more explanation, more potential misunderstanding and I'd like to keep the number of questions down if it comes down to bad design.
For now, I would rather keep option B, but let me know if you have an idea I might have missed.

Interesting - I didnt realize that's how whitelist and --cells interacted. I was under the impression providing a whitelist was like using option A (hence why it seemed odd to need to provide '--cells=0', when the size could be inferred from the whitelist.
One usage that might be useful would be to treat the whitelist as barcodes to always include, but then also include additional high-UMI-count cells (governed by --cells). Our usage might explain that:
We are using Cite-Seq-count as basically a first-pass. We do filtering and QC downstream in R (currently our cellhashR package), which filters cells by min count. Having Cite-Seq-Count provide an input count matrix with all requested cells is useful, since the difference between whitelist and passing (i.e. above min counts) can tell you something about the library.
Our primary use-case is to provide a whitelist generated using either: 1) the whitelist of cell barcodes with passing GEX data, or 2) the whitelist of barcodes with passing TCR data (which is not necessarily identical the passing GEX). In each case, we often want maximal calling rate, and we might want to take steps to recover cells with low cell hashing-counts. I have no real problems that I know about with the current behavior of providing CITE-Seq-Count whitelist, and it producing a matrix with those cells.
The theoretical reason an output that is the union of whitelist and high-UMI-count cells could be advantageous is that in the case of TCR, the TCR whitelist might actually be a subset of true cells. For example, if you sequence PBMC, only ~15% of cells might actually be T-cells. I could probably solve this myself by giving Cite-Seq-Count a whitelist that is actually the union of GEX and TCR cell barcodes. This is not something Cite-Seq-Count needs to do itself.
Anyway, that's a long explanation. The core thing I was asking is that it seems unnecessary for Cite-Seq-Count to force the user to enter '--cells-0' when a whitelist is provided. It seems that if a whitelist is provided and --cells is not provided, that the tool should do something like default to the length of the whitelist.  I dont have any real problems with the actual performance of Cite-Seq-Count as-is.

Interesting - I didnt realize that's how whitelist and --cells interacted. I was under the impression providing a whitelist was like using option A (hence why it seemed odd to need to provide '--cells=0', when the size could be inferred from the whitelist.
One usage that might be useful would be to treat the whitelist as barcodes to always include, but then also include additional high-UMI-count cells (governed by --cells). Our usage might explain that:
We are using Cite-Seq-count as basically a first-pass. We do filtering and QC downstream in R (currently our cellhashR package), which filters cells by min count. Having Cite-Seq-Count provide an input count matrix with all requested cells is useful, since the difference between whitelist and passing (i.e. above min counts) can tell you something about the library.
Our primary use-case is to provide a whitelist generated using either: 1) the whitelist of cell barcodes with passing GEX data, or 2) the whitelist of barcodes with passing TCR data (which is not necessarily identical the passing GEX). In each case, we often want maximal calling rate, and we might want to take steps to recover cells with low cell hashing-counts. I have no real problems that I know about with the current behavior of providing CITE-Seq-Count whitelist, and it producing a matrix with those cells.
The theoretical reason an output that is the union of whitelist and high-UMI-count cells could be advantageous is that in the case of TCR, the TCR whitelist might actually be a subset of true cells. For example, if you sequence PBMC, only ~15% of cells might actually be T-cells. I could probably solve this myself by giving Cite-Seq-Count a whitelist that is actually the union of GEX and TCR cell barcodes. This is not something Cite-Seq-Count needs to do itself.
Anyway, that's a long explanation. The core thing I was asking is that it seems unnecessary for Cite-Seq-Count to force the user to enter '--cells-0' when a whitelist is provided. It seems that if a whitelist is provided and --cells is not provided, that the tool should do something like default to the length of the whitelist.  I dont have any real problems with the actual performance of Cite-Seq-Count as-is.

There is another person who tested the new version and prefers the old paradigm.
That's a high return rate over the few people who tried the new version.
I guess I should think about it.
The next step for CSC is to use a centralized chemistry definition, which will hold the full reference list.
The main idea is to propose to the use to only have to use --chemistry 10xV3 and it will download reference lists, knows barcodes position, etc...
I could change this a bit by keeping the old way of asking for the "short" reference list, but, proposing a new argument being the  full reference list.
I'm just afraid this could get confusing to users.

There is another person who tested the new version and prefers the old paradigm.
That's a high return rate over the few people who tried the new version.
I guess I should think about it.
The next step for CSC is to use a centralized chemistry definition, which will hold the full reference list.
The main idea is to propose to the use to only have to use --chemistry 10xV3 and it will download reference lists, knows barcodes position, etc...
I could change this a bit by keeping the old way of asking for the "short" reference list, but, proposing a new argument being the  full reference list.
I'm just afraid this could get confusing to users.

on all these points, we're pretty flexible. we'll adapt to a new argument paradigm in a new version or keep as-is.
the only thing i intended to bring up on this thread was what seemed like redundancy in CSC requiring cells=0 when a whitelist is provided. i now see it's more complicated.

on all these points, we're pretty flexible. we'll adapt to a new argument paradigm in a new version or keep as-is.
the only thing i intended to bring up on this thread was what seemed like redundancy in CSC requiring cells=0 when a whitelist is provided. i now see it's more complicated.


Split Jenkins server and LibreCores CI SaaS repositories

Right now we have everything in a single repository, and it is hard to maintain it. My plan is to split out  the SaaS repo to the separate repo. Likely I will start there from Helm charts so that we can have a more modern deployment.

https://github.com/librecores/ci.librecores.org was created, but we still need to move the docs


sys/config: Automatically restore on startup

Before PR: the application needed to manually restore the persisted configuration via a call to conf_load(), typically from main().
After PR: conf_load() gets called automatically at startup.  It doesn't get called directly from sysinit(), as the underlying storage hasn't been configured at that point.  Instead, values are loaded by an event which config init function enqueues to the default event queue.  When main() has completed its runtime configuration, including that of the sys/config storage, it processes the default eventq, at which point conf_load() gets called.
There is one danger here: some packages may try to use a configuration setting before conf_load() has been called.  However, this is not a new issue; high priority tasks, created by sysinit(), would already preempt main() before conf_load() could be called.  To address this problem, I believe we need a separate fix.

I think this PR warrants an email to dev-list, as this will have effect on all apps using sys/config.


RatedShipment_TransportationCharges - MonetaryValue minLength

The minLength is 16, however, returned values are not padded to meet this minLength:
      "TransportationCharges": {
        "CurrencyCode": "USD",
        "MonetaryValue": "160.02"
      },

https://github.com/UPS-API/api-documentation/blob/4962e982cfca654bee7fbc4c9a0b80105cf480aa/Rating.yaml#L4601-L4622

Hi, thank you for your comment. We have pushed updated OpenAPI specs.


pathrules: merge path rules such that duplicate paths are ignored

Checklist

[x] The title of the PR is clear and informative
[ ] If applicable, the changes made in the PR have proper test coverage
[ ] Issues addressed by the PR are mentioned in the description followed by Fixes.

Description
Merge path based rules such that

duplicate paths are ignored
ingress error message is logged with details of the service having duplicate path

Fixes

TODO: E2E


Crash when trying to replicate

Trying to replicate the exploit in kernel 6.2.0, the terminal prints this and the pc freezes.
Any feedback on what may be wrong?

I thought that could be the problem but I did try disabling the wifi adapter through BIOS as you suggested on my laptop, tried turning off the wifi in a virtual box machine and also tried disabling through virtualbox the adapter and always got the crash...

I've seen this behaviour on AWS EC2 instances as well. Servers would freeze, kernel panic, and then reboot.

So I think I found out what the problem is, when I tried to execute the exploit I downloaded a new ubuntu 22.04 LTS and then changed the kernel to one of the supported ones but the problem I mentioned kept ocurring, but I have found out that using an older ubuntu version such as 20.04 makes it possible to run the exploit successfully. Maybe I can make a pr noting in the readme that ubuntu 20.04 is required?

Great exploit by the way and congrats on finding it! I'd love to know if there are any plans to stabilize the exploit? I read in the blog pages ways it could be stabilized, unfortunately I'm not too familiar with such techniques, do you know of any colleagues or people who may create or develop a more stabilized versions?

I mean it does come with an unsupported kernel, but I suppose that as long as I downgraded the kernel to one of the supported ones, like I did, the exploit should theoretically work right?
20.04 comes with an unsupported kernel as well but downgrading makes the trick

But it doesn't, I guess something changed from ubuntu version to another. What ubuntu version did you test with?


Delay (beta) does not work

Hi there,
the delay function does not work for me.
Hombridge Service does not start after setting delay to true.
It would be nice to have a delay, because for roomcleaning. If set multiple rooms the cleaning mode should have a delay for 3 seconds to wait until every room switch is set to on.

Hi @Christian-CFP, I think the option you're after is roomTimeout (explained in the readme)
The option delay is for something else: It tries to wait for the connection to be established on startup before registering the plugin in Homebridge, although it doesn't work in all OSs.

Sounds great but I tried it and set it to 10 sec. and no. Did not get all rooms. It started. Did not wait 10 sec.

1 or 2 sec will work but I wanted to know if the robo will wait the time.

Hmm! I'll need to test it myself to debug the reason. AFAIK, it should wait the specified time.


Windows: "This device cannot start. (Code 10)"

I have a code for a simple macro pad using the raspberry platform with arduino framework, and it uses the PluggableUSBHID, USBKeyboard and USBMouse that are included with the framework, plus the Serial is also printing through USB.
It works fine on Linux, but Windows 10/11 fails to start the device with a STATUS_DEVICE_DATA_ERROR code.
I saw similar issues with pico-sdk and a workaround for a hardware bug that manually sleeps for 800us, but I haven't seen anything for PlatformIO.
Is there anything I'm missing?

Hi @Caian, it seems that your problem has nothing to do with PlatformIO itself, probably there is an issue with Arduino framework or Windows drivers for your device which is out of scope for this repository.


3D Perspective on logo.

Just an idea:
http://youtu.be/-D_3vw69Mkc
Just add the CSS perspective property to the .logo.

@sokra Is there some JavaScript that triggers the animation? It'd be easier to do it there without actually modifying the logo.

I also prefer the isometric look of the logo but @trusktr if you're still interested in updating the logo; the animation is timed just by the CSS. So you would need to modify the translations to each side or maybe get fancy with the perspective css declaration.
Feel free to send a PR to https://github.com/webpack/docs/blob/master/app/css/mixins.styl#L17-L94
Cheers!


Videos from the Montreal-Python usergroup are missing

See https://github.com/pytube/data/issues/7

Hi @mlhamel,
In https://github.com/pytube/data/issues/7 you commented

If I can help in anyway that would be a pleasure :) Just let me know what's need to be done add those videos on pyvideo.org

I have scarped all videos from the Montreal-Python usergroup [ https://www.youtube.com/user/MontrealPython ] into montreal-python branch. But data needs to be corrected.
Example a-hobbist-introduction-to-the-kivy-framework-mp47:
{
  "copyright_text": "Creative Commons Attribution license (reuse allowed)",
  "description": "Montreal, June 9, 2014 - Ghislain Hivon presents kivy, a Python library for rapid development of applications that make use of innovative user interfaces, such as multi-touch apps.\n\nUseful links:\n* Ghislain: https://twitter.com/GhislainHivon\n* Kivy: http://kivy.org\n* MP47: http://sflx.ca/mp47\n* Montreal Python: http://montrealpython.org\n* Savoir-faire Linux: http://www.savoirfairelinux.com",
  "duration": 274,
  "language": "eng",
  "recorded": "2014-07-16",
  "related_urls": [
    "http://montrealpython.org",
    "http://sflx.ca/mp47",
    "https://twitter.com/GhislainHivon",
    "http://www.savoirfairelinux.com",
    "http://kivy.org"
  ],
  "speakers": [
    "TODO",
    "enter_manually"
  ],
  "tags": [],
  "thumbnail_url": "https://i.ytimg.com/vi/fAK74eBCIBk/maxresdefault.jpg",
  "title": "A hobbist introduction to the kivy framework #MP47",
  "videos": [
    {
      "type": "youtube",
      "url": "https://www.youtube.com/watch?v=fAK74eBCIBk"
    }
  ]
}

Fields to review:

description: Have to be in proper RestructuredText to render correctly.
language: eng or fra in your case.
recorded: If you have the real data you can substitute the Youtube upload date scraped.
related_urls: slides, repos, and relevant urls go here.
speakers: Couldn't automate your case, it have to be fixed manually.
tags: Having the videos tagged is nice when searching.
title: Speaker name, user group, etc shouldn't be here.

All the files have to be correct json.
To test the data use make test in the root of pyvideo/data repo.
To preview the result use make html && make serve.
Anyone that want to fix it could make a pull request to montreal-python branch.

Hi @mlhamel,
Please let know to Montreal Python group that this issue is almost done.
Any review/fix/improvement is welcome (after or before the merge).

I think we can consider this as done! Now we are gonna take care of maintaining those! Thanks everyone.


[v1.4] Fix bugs found in Login and Signup commands

Pull Request Test Coverage Report for Build 1111

4 of 4 (100.0%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.01%) to 87.904%

Totals

Change from base Build 1105:
-0.01%

Covered Lines:
2231

Relevant Lines:
2538

💛  - Coveralls


Fix #1863 validation issue with schedule names

Thanks for reporting this @JamiePhonic!

@dsander There were validation errors when comparing the OpenStruct to a string, so I went this direction instead.

No problem!


Azure refresher bottlenecked by Orchestration Stacks

The longest portion of an EMS refresh for Azure is Orchestration Stack information collection. It appears that we're running a single select for every Orchestration Stack instance, which is quite slow.
You can see what's happening by creating an Azure provider in the eastus (using our dev environment) and watch what's happening:
EmsRefresh.refresh(ManageIQ::Providers::Azure::CloudManager.first)

  ManageIQ::Providers::Azure::CloudManager::OrchestrationStack Inst Including Associations (0.1ms - 1rows)
  ManageIQ::Providers::Azure::CloudManager::OrchestrationStack Load (0.6ms)  SELECT  "orchestration_stacks".* FROM "orchestration_stacks" WHERE "orchestration_stacks"."type" IN ('ManageIQ::Providers::Azure::CloudManager::OrchestrationStack') AND "orchestration_stacks"."name" = $1 AND "orchestration_stacks"."ems_ref" = $2 AND "orchestration_stacks"."ems_id" = 33 LIMIT $3  [["name", "Microsoft.StorageAccount-123"], ["ems_ref", "/subscriptions/abc/resourceGroups/walateststorage/providers/Microsoft.Resources/deployments/Microsoft.StorageAccount-123"], ["LIMIT", 1]]
  ManageIQ::Providers::Azure::CloudManager::OrchestrationStack Inst Including Associations (0.0ms - 0rows)
  ManageIQ::Providers::Azure::CloudManager::OrchestrationStack Load (0.5ms)  SELECT  "orchestration_stacks".* FROM "orchestration_stacks" WHERE "orchestration_stacks"."type" IN ('ManageIQ::Providers::Azure::CloudManager::OrchestrationStack') AND "orchestration_stacks"."name" = $1 AND "orchestration_stacks"."ems_ref" = $2 AND "orchestration_stacks"."ems_id" = 33 LIMIT $3  [["name", "RedHat.RedHatEnterpriseLinux72-123"], ["ems_ref", "/subscriptions/abc/resourceGroups/whitty/providers/Microsoft.Resources/deployments/RedHat.RedHatEnterpriseLinux72-123"], ["LIMIT", 1]]
  ManageIQ::Providers::Azure::CloudManager::OrchestrationStack Inst Including Associations (0.1ms - 1rows)
  ManageIQ::Providers::Azure::CloudManager::OrchestrationStack Load (0.5ms)  SELECT  "orchestration_stacks".* FROM "orchestration_stacks" WHERE "orchestration_stacks"."type" IN ('ManageIQ::Providers::Azure::CloudManager::OrchestrationStack') AND "orchestration_stacks"."name" = $1 AND "orchestration_stacks"."ems_ref" = $2 AND "orchestration_stacks"."ems_id" = 33 LIMIT $3  [["name", "VMAccessWindowsPasswordResetXXX"], ["ems_ref", "/subscriptions/abc/resourceGroups/xxx/providers/Microsoft.Resources/deployments/VMAccessWindowsPasswordResetXXX"], ["LIMIT", 1]]
 # ... and so on, for every Orchestration Stack.

Can this collection of queries be collated into a single query?

Looks like an N+1.  Do you know where these queries are coming from?  Most of the save_inventory code is designed to mass query, so this surprises me.

@Fryguy I think it's https://github.com/ManageIQ/manageiq/blob/master/app/models/manageiq/providers/azure/cloud_manager/refresh_parser.rb#L340-L346

Can this logic be moved into the deployment object? And can it become a relation?
@bzwei what percentage of the time will this bring back an object?

The slowness may come from either factor:
https://github.com/ManageIQ/manageiq/blob/master/app/models/manageiq/providers/azure/cloud_manager/refresh_parser.rb#L333
For each stack we want to find its template and link them. The template is EITHER downloaded OR fetched from VMDB. So the DB inquiry is not for EVERY stack. Downloading files take time. When you observe the slowness it might come from the downloading which we cannot boost.
It is true we attempt to query the stack record for each deployment object that does not have an URI for its template. I don't know how to boost the performance at this time. The deployment object is a raw object, not yet a VMDB record, so a relation cannot help.

Can we try to fetch a collection of templates for all qualified deployments? Yes. That's the direction we can work on.

@djberg96 Can you close this issue if #11662 solves the problem?

@bzwei It's still a bit slow with that big "IN" clause, but overall I think it's faster. I haven't done any formal benchmarking yet, though.

Closing this for now, as we have some similar discussions already on the Azure provider repo.


added extra metrics for scheduler threads

Scope & Purpose
Added scheduler thread creation/destruction metrics:

arangodb_scheduler_threads_started: Number of scheduler threads started

arangodb_scheduler_threads_stopped: Number of scheduler threads stopped

[ ] :hankey: Bugfix

[x] :pizza: New feature

[ ] :hammer: Refactoring

[x] :book: CHANGELOG entry made

[x] :muscle: The behavior in this PR was manually tested

[ ] :computer: The behavior change can be verified via automatic tests

Backports:

[ ] No backports required
[x] Backports required for: 3.7

Testing & Verification

[x] This change is a trivial rework / code cleanup without any test coverage.

http://<IP_ADDRESS>:8080/view/PR/job/arangodb-matrix-pr/11865/

http://<IP_ADDRESS>:8080/view/PR/job/arangodb-matrix-pr/11870/


[Android] Canvas disapears on screen on/off

Description
Hello,
I've noticed that my component (a simple ring) disapears on screen ON, in Android release mode only.
Component re-appears as soon as it re-renders.
Tested with a OnePlus 8T
Version
0.1.175
Steps to reproduce

Open your app
Turn OFF screen with hardware button while app is in foreground
Turn ON screen

Snack, code example, screenshot, or link to a repository
import React from 'react';
import { View } from 'react-native';
import {
    Canvas,
    Group,
    SweepGradient,
    Circle
} from "@shopify/react-native-skia";

type Props = {
    colors: string[];
    size?: number;
    strokeWidth?: number;
    children?: React.ReactNode;
}

export const ColorRing = ({ colors, size = 45, strokeWidth = 5, children }: Props) => {
    return (
        <View>
            <View style={{ position: 'absolute' }}>
                {children}
            </View>
            <Canvas style={{ width: size, height: size }} >
                <Group style="stroke" strokeWidth={strokeWidth / 2}  >
                    <Circle cx={size / 2} cy={size / 2} r={(size - strokeWidth) / 2}>
                        <SweepGradient
                            c={{ x: size / 2, y: size / 2 }}
                            start={0}
                            end={360}
                            colors={colors}
                        />

                    </Circle>
                </Group>
            </Canvas>
        </View>
    );
}

Hi,
I have the same problem on a Xperia XA1 with Android 8. Do you find a solution ?
I think we have to redraw the canvas when the phone wakes up but it's supposed to be already done by Skia ?
I tried with a button that redraws the whole application component and the canvas reappears, so we have to redraw the whole canvas when the phone wakes up.

Hi,
we have the same problem and we ended up re-rendering all Skia canvases when AppState changes.


Fix header and cookie parameter locations

@grktsh thank you for the contribution. It's similar to #46

@p1c2u Sorry, overlooked it.  Closing this.


Enhancement Request: Enable changing the file size upper threshold for read cache

Thank you for the excellent work with this script!
I wanted to ask if you know how to increase the size of files that the Synology will put into the read cache?
From what I have read (and I could be wrong), it will only cache files up to 1M? (I'll try to find where I read that)
With a large SSD it would be great if we could specify what the upper limit is for files that would be cached.

I believe that DSM originally cached all files, and Synology added the file size limit in a later DSM version. I'll see if I can find what they changed and if it can be edited.


Can Instagram restrict access to certain API functions for web only?

Can it be that Instagram restricts access to certain API methods based on the plathorm: web or mobile? That is, even if I log in into my account via browser normally and then call some API method via DevTools of the browser, and such a method is not present in the web version via some sort of a button, it'll return an error.
For instance, api/live/create. Theoretically it may very well be, but has it confirmed by someone?
I've confirmed it for myself, but indirectly.

@GildedHonour I don't really understand what you are suggesting, but you can log in with a browser and copy the sessionid cookie. After that, through instagrapi (cl.login_by_sessiond) call any endpoints from the Instagram mobile API.
If this is not so, then tell me which specific endpoints do not work?


Webpack sometimes doesn't recompile files on watching.

I have latest cloned version of NG6-starter and in PHPStorm when I make changes in files and tries to save it by CTRL + S, webpack nothing do. Sometimes I need to make change in one of file (doesn't matter which one) and again save file. If still not work, then need to repeat process.
It always work after few tries :)
NodeJS: v7.5.0
NPM: v4.1.2
Windows 10 PRO, 64 bit

This is exactly the same issue we're having and we, continually, need to pseudo-save files to have webpack pick up changes.
Would love to see what kind of fix we can get in for this...

Just turn off 'Safe write' setting in PHPStorm. Follow these steps
https://stackoverflow.com/a/34225091/6396555

^ Will give that a try...

Closing this since issue is related to webpack. If you have this problem please check webpack documentation or found/write issue in their repository.


[Epic] Unify AggregateFunction Interface (remove built in list of AggregateFunction s), improve the system

Is your feature request related to a problem or challenge?
For many of the same reasons as listed on https://github.com/apache/arrow-datafusion/issues/8045, having two types of aggregate functions ("built in" -- datafusion::physical_plan::aggregates::AggregateFunction) and AggregateUDF is problematic for two reasons:

There are some features not available to User Defined Aggregate Functions (such as the faster GroupsAccumulator interface)
Users can not easily choose which aggregate functions to include (for example, do they want to allow (and pay the code size / compile time) for the Statistical and Approximate  functions

The second also ends up causing pushback on adding new aggregates like ARRAY_SUM in https://github.com/apache/arrow-datafusion/pull/8325 and geospatial support https://github.com/apache/arrow-datafusion/issues/7859.
Describe the solution you'd like
I propose moving DataFusion to only use AggregateUDFs and remove the built in list of AggregateFunctions for the same reasons as https://github.com/apache/arrow-datafusion/issues/8045
We will keep the existing AggregateUDF interface as much as possible, while also potentially providing an easier way to define them.
Describe alternatives you've considered
Additional context
Proposed implementation steps:

[x] https://github.com/apache/arrow-datafusion/issues/8710
[x] https://github.com/apache/arrow-datafusion/issues/8793
[ ] https://github.com/apache/arrow-datafusion/issues/8984
[ ] https://github.com/apache/arrow-datafusion/issues/9526

I will work on FirstValue UDF together with #9249

Can we introduce state_fields and fields for AggregateUDFImpl.  We can see that types in AggregateUDFImpl are for  building fields, why not just return fields directly, we can not only define the types but also the field name and is_nullalbe.
https://github.com/apache/arrow-datafusion/blob/e337832946fc69f6ceccd14b96a071cc2cd4693d/datafusion/physical-plan/src/udaf.rs#L86-L106

Thank you @jayzhan211  -- I am traveling this week so I am very behind on reviews. I will try and respond later this week

Thank you @jayzhan211  -- I am traveling this week so I am very behind on reviews. I will try and respond later this week

I leave the comment so I won't forget.
Have a good trip 😊

I have another question. I think our goal for the aggregate function is similar to functions. we will move them into separate create. If we need to avoid importing physical-expr. It seems we need to move some struct from physical-expr to expr, like PhysicalExpr and pub type LexOrdering = Vec<PhysicalSortExpr>;
https://github.com/apache/arrow-datafusion/blob/3eeb108125b35424baac39dd20ba88433b347419/datafusion/physical-expr/src/aggregate/first_last.rs#L44-L45
But, does moving PhysicalExpr to datafusion_expr make sense? 😕 Where/When should be convert expr to physical-expr  like sort_exprs to LexOrdering.  Should we go through the whole process in aggregate-functions crate like what functions do or we should seperate logical-expr and physical-expr for aggregate-functions and find a way to link between them (convert from logical-expr to physical-expr)?

After https://github.com/apache/datafusion/pull/10648 and https://github.com/apache/datafusion/issues/10389 I think we have a pretty good set of examples of how to move aggregates out of the core (thanks to all the foundations layed by @jayzhan211 )
Would it be helpful to file a few "good first issue" type tickets for some of the more straightforward aggregates (I am thinking the statistical variance etc)?

After #10648 and #10389 I think we have a pretty good set of examples of how to move aggregates out of the core (thanks to all the foundations layed by @jayzhan211 )
Would it be helpful to file a few "good first issue" type tickets for some of the more straightforward aggregates (I am thinking the statistical variance etc)?

Before this, I think it would be nice to determine the API of aggregate expr first https://github.com/apache/datafusion/pull/10560#discussion_r1611644593


Implement predicate pruning for like expressions (prefix matching)

The idea is that we can push certain like expressions down into statistics pruning.
For example, a filter like url LIKE 'https://www.google.com%' can be (basically, some caveats and other tricks) used to filter url_min <= 'https://www.google.com' and 'https://www.google.com' <= url_max such that a row group that only has https://www.example.com would be excluded.
Closes https://github.com/apache/datafusion/issues/507

cc @alamb

This is very clever -- I will review it tomorrow

@alamb can these stats be truncated? I know stats in pages truncate large strings, e.g. if the min value is "B" could it be that the actual min value is "BA"? If so I think this approach may not work at all. Imagine we have a row group with data ["BA", "ZD"] which generates min/max stats ["B", "Z"]. Now we want to know if col LIKE '%A%' is possible. Clearly the answer should be yes but if we convert it to the predicate form we get 'B' <= '' AND '' <= 'Z' which gives false 😞

@adriangb  in theory I think parquet statistics can be truncated.

Now we want to know if col LIKE '%A%'

I don't think we can use statistics for substring match -- we can only use statistics for equality and prefix matching
so like col LIKE 'A%'
The predicate would be transformed into 'B' <= 'A' AND 'A' <= 'Z' which I do think is correct

Consider the values ["ABC", "XYZ"] with stats ["AB", "XY"] and the filter col like 'A%'. This becomes 'AB' <= 'A' AND 'A' <= 'XY' which is false, but we need true. To fix this we'd need to truncate the stats to the length of the filter to get 'A' <= 'A' AND 'A' <= 'XY' which then gives the right result. %A% is just an obvious case because you get '' as the prefix which gives obvious issues.

Okay @alamb I pushed a pretty big rework. Lots of new test cases, lots of comments explaining what's going on. I removed the not like part; I'm thinking this is complex enough as is and most of the benefit (maybe even in ClickBench?) will come from like. We can tackle not like, ilike and not ilike in future PRs. Especially since it's going to be important to evaluate each test case carefully.
I will note that I am a bit concerned about the interaction of truncated stats and how we apply these filters. Take the (absurd) case of stats that were truncated so that all you have is "","". You basically know nothing about the data, there could be anything in there. Yet col = 'A' transforms into '' <= 'A' and 'A' <= '' which is false. Substitute in non-truncated stats and 'A' <= 'A' and 'A' <= 'Z' is true. I added a test case for this behavior on the existing code. It doesn't differentiate between "ABC" truncated to "" and "" actually being the min string but it shows the behavior which would be the same in both cases.

Argh the only ClickBench query this maybe could improve is 23:
WHERE "Title" LIKE '%Google%' AND "URL" NOT LIKE '%.google.%'

Since the like starts with a wildcard this won't help. Maybe we can get not like to do something smart there, tbd...

I am wondering if simple like patterns are not already converted to startswith etc, such that pruning already is applied.

I am wondering if simple like patterns are not already converted to startswith etc, such that pruning already is applied.

Good question. Based on performance of some queries I saw I'd say no, but it it's worth double checking. Any suggestions as to a definitive easy way to check? I guess I can run datafusion-cli against a very large parquet file in object storage (high latency) and a query that should filter (col like 'A%') and one that can't?
I don't see where startswith or any other transformations (lower, upper, etc.) are handled in the pruning transformation.

I would take a look at the (physical plan) of queries involving like first to see if it still uses like or is transformed to another function.

I made a big parquet file as follows:
import random
import string
import polars as pl

df = pl.DataFrame({'col': ["A" + "".join(random.choices(string.ascii_letters, k=1_000)) for _ in range(1_000_000)]})
df.write_parquet('data.parquet', compression='uncompressed')

This came out to ~1GB. I then uploaded it to a GCS bucket.
I ran queries col = 'Z' and col like 'Z' against it and got 2s and 23s respectively. IMO that means it's not getting pushed down.
The explain plans show the following:
ParquetExec: file_groups={10 groups: [[data.parquet:0..100890471], [data.parquet:100890471..201780942], [data.parquet:201780942..302671413], [data.parquet:302671413..403561884], [data.parquet:403561884..504452355], ...]}, projection=[col], predicate=col@0 = Z, pruning_predicate=CASE WHEN col_null_count@2 = col_row_count@3 THEN false ELSE col_min@0 <= Z AND Z <= col_max@1 END, required_guarantees=[col in (Z)], metrics=[output_rows=0, elapsed_compute=10ns, predicate_evaluation_errors=0, bytes_scanned=19368790, row_groups_pruned_bloom_filter=0, row_groups_pruned_statistics=3, pushdown_rows_filtered=0, page_index_rows_filtered=0, row_groups_matched_statistics=0, row_groups_matched_bloom_filter=0, file_scan_errors=0, file_open_errors=0, num_predicate_creation_errors=0, time_elapsed_scanning_until_data=18.748µs, time_elapsed_opening=7.717746249s, time_elapsed_processing=64.457827ms, page_index_eval_time=10.134µs, pushdown_eval_time=20ns, time_elapsed_scanning_total=19.21µs]

ParquetExec: file_groups={10 groups: [[data.parquet:0..100890471], [data.parquet:100890471..201780942], [data.parquet:201780942..302671413], [data.parquet:302671413..403561884], [data.parquet:403561884..504452355], ...]}, projection=[col], predicate=col@0 LIKE Z, metrics=[output_rows=1000000, elapsed_compute=10ns, predicate_evaluation_errors=0, bytes_scanned=1006955145, row_groups_pruned_bloom_filter=0, row_groups_pruned_statistics=0, pushdown_rows_filtered=0, page_index_rows_filtered=0, row_groups_matched_statistics=0, row_groups_matched_bloom_filter=0, file_scan_errors=0, file_open_errors=0, num_predicate_creation_errors=0, time_elapsed_scanning_until_data=49.346124581s, time_elapsed_opening=2.18377s, time_elapsed_processing=1.545583231s, page_index_eval_time=20ns, pushdown_eval_time=20ns, time_elapsed_scanning_total=49.654700084s]

So it doesn't seem like it's being transformed into another expression. It probably would be smart to do so as a general optimization outside of pruning.

I also think pruning should handle whatever that produces (startswith in the case of like 'A%' or = in the case of like 'A') as well as additional simple cases like upper(), lower(), etc.

That certainly makes sense to me
Perhaps we could implement some sort of simplification in https://github.com/apache/datafusion/blob/main/datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs for LIKE into starts_with https://docs.rs/datafusion/latest/datafusion/functions/expr_fn/fn.starts_with.html (though since it is a function figuring out how to make the rewrite work might be tricky)
Then we can implement the rules in pruning predicate for starts_with 🤔

I'm happy to do it that way if you'd like. Or we can leave that for a future optimization.
Do you have any thoughts on my concerns for possibly truncated stats, in particular how = may even be wrong as of today if the stats are truncated enough?

I think it’s fine to support like for now and leave the further combination / optimization for future work. I see that only simplifying to starts_with won't get all the benefits.
Of course, the pruning needs to be correct :). We could add (negative) test cases / add issues if the already implemented logic for = pruning is incorrect.

Well there's no tests for hypothetical cases with truncated stats. All of the tests are against the stats themselves.
The current implementation is certainly not wrong in the real world, but I'm not sure if that's because it's not used in situations where stats are truncated, if truncation only happens at extremes like a 10MB value where practically it's not a problem, etc.

I suggest we (I can help tomorrow):

File a ticket to simplify like to = and starts_with when possible (to help follow on optimizations like this)
File / find a ticket about testing with truncated statistics
Determining what, if anything, is left to add directly to pruning prediate

Sounds good.
All of that said, I think this PR is currently as correct as = and had pretty good test coverage. Does it need to wait on those tasks or can it proceed in parallel?

@alamb I re-arranged some of the comments on assertions in https://github.com/apache/datafusion/pull/12978/commits/ae3426da4b19ff9e8adf038547b6a4e552190f1f which I feel like helped a lot with readability of the tests. There's a couple other tests with a similar pattern that I think could benefit.
I was also thinking about doing some more black-box testing: I think given any min, max you can always convert that into a  RecordBatch with an array of the form [min, max] and you should never have the pruning say the array should be excluded but the array has any matches. Does that make sense? Maybe this could even be fuzz tested?

I was also thinking about doing some more black-box testing: I think given any min, max you can always convert that into a RecordBatch with an array of the form [min, max] and you should never have the pruning say the array should be excluded but the array has any matches. Does that make sense? Maybe this could even be fuzz tested?

We could also randomly generate an array with an element or elements within the min max range.
This would cover not only logic at the edges, but also for the values inside the range.

Some sqllogictests failed and this is expected. @adriangb can you please update them with
cargo test --test sqllogictests -- --complete

?

Some sqllogictests failed and this is expected. @adriangb can you please update them with
cargo test --test sqllogictests -- --complete

You will need to merge with main to reproduce the failures (the failing tests are new).
 There is no min,max predicate we can practically derive from LIKE %a%  expression.
  character is problematic to read and to work with. Some tools don't handle it properly. For example when pasting into iterm with python3 running for inspection, it got displaed as \U+0FFFF even though this is code pointx10FFFF.

Some sqllogictests failed and this is expected. @adriangb can you please update them with
cargo test --test sqllogictests -- --complete

You will need to merge with main to reproduce the failures (the failing tests are new). When you do so, you should get a diff like this

Thanks I was wondering how to update that. When I run it I get various errors along the lines of:
Execution error: Error completing "string/string_view.slt": failed to rename file from test_files/string/init_data.slt.part.temp to test_files/string/./init_data.slt.part
Execution error: Error completing "string/large_string.slt": failed to rename file from test_files/string/string_query.slt.part.temp to test_files/string/./string_query.slt.part
Execution error: Error completing "string/dictionary_utf8.slt": failed to rename file from test_files/string/string_query.slt.part.temp to test_files/string/./string_query.slt.part

Any idea what's going on?

Yes -- #12752
but don't worry. see #12978 (comment) . this test case shouldn't change after all

Having other issues now. I tried in 46531f5bf7be7a59255fc8723f0be3efd3beb908 but got a bunch of errors from missing files. I tried to follow what CI is doing but got to compiling the databricks repo and got a missing malloc.h so I stopped there since I wasn't even sure if I was barking up the right tree. Is this the expected workflow for updating these snapshots? It might be helpful if failed CI runs offered them as a downloadable artifact.

for a predicate LIKE %a% we should not generate additional conditions at all. There is no min,max predicate we can practically derive from LIKE %a% expression

By the way I added a special case for the wildcard being the first character (no prefix) to bail out and add no filtering

Looks like the special case on '%...' actually eliminated the diff in those tests, so no need to help me figure out how to update them now. I'll re-request review from both of you. Thank you for your help so far.

This is on my list to review today

I didn't make it here today -- it is the top of my list for tomorrow

Thanks for the review Andrew. Let's open an issue to track a fuzz tester for predicate pushdown.

@alamb @findepi I think I've accepted or applied all of your excellent feedback, and I added some coverage / more test cases to the increment_utf8 function 😄

This is one of the PRs I hope to push through this weekend -- what I would really like to do given the potential for very subtle bugs in this code, is to write some more tests. I keep hoping I will find time, but I haven't. Maybe this weekend will be better

Given LIKE can get desugared to something else during planning (eg  https://github.com/apache/datafusion/pull/13061), maybe it would be great to support this functionality with end-to-end tests.

Given LIKE can get desugared to something else during planning (eg #13061), maybe it would be great to support this functionality with end-to-end tests.

I feel like we need both e2e and more focused tests: e2e tests risk depending on a combination of the optimizer and pruning rewrites that could be broken if e.g. someone customized the optimizer (which I believe is pluggable?).

This PR should possibly indicate that it Closes #507.

This PR should possibly indicate that it Closes #507.

Updated the description

No worries, I'll try to take a look at #13253. I agree it's better to be cautious and make sure we get this right rather than create a bug that results in incorrect data.

I wrote up some fuzz tests here: #13253
They don't yet pass -- I think there might be some bug in the test.

Good tests should find https://github.com/apache/datafusion/issues/12637 and therefore shouldn't pass.

To follow up here -- what I think this PR needs prior to merging is to add fuzz testing as I think it has the potential to have hard to track down / subtle bugs
Some important corner cases:

With truncated statistics (actual min/max different than stats)
With truncated prefix matching -- as in min / max is actually foo but the query has a predicate like column='fo'

Hi @alamb I took a stab at fuzz tests in 969e83c82.
They're heavy and slow so I had to restrict the search space a lot more than I would have liked. Maybe you or @tustvold can suggest ways to cut out the heavy parsing of Parquet metadata and such to speed these up? Ultimately I do think it's worth re-using whatever creates parquet stats from data so that we use the "real" thing but I don't think we need to test the serialization / deserialization repeatedly like this does.
Also happy to restrict the search space by being more deliberate about how we build the values, row groups, predicates, etc.

Thanks @adriangb  -- I have put this PR on my review queue and will attempt to re-review it carefully tomorrow

Clearly I failed to review this -- I will do so hopefully later today but may be tomorrow

This is still on my list, hopefully other people can check it out too

This is my top priority after DF 44 is released:

#13334

Thanks @adriangb -- I think this PR is ready to go
One thing I noticed is that the fuzz test takes over a minute on my machine:
        SLOW [> 60.000s] datafusion::fuzz fuzz_cases::pruning::test_fuzz_utf8
        PASS [  65.772s] datafusion::fuzz fuzz_cases::pruning::test_fuzz_utf8
------------
     Summary [  72.749s] 47 tests run: 47 passed (1 slow), 0 skipped
andrewlamb@Mac:~/Software/datafusion$

Is there some way to make it faster? Maybe with multiple threads or crank down the number of things to teset?

Yeah this is what I was hinting at in https://github.com/apache/datafusion/pull/12978#issuecomment-2542335627.
I'm happy to throw threads at it for a start, and restricting the search space might be necessary but I think requires a more careful eye to minimize how much valuable testing is discarded. The other thing that I think we can do is speed up the tests themselves, in particular minimizing unnecessary round trips to Parquet, but I'm not sure where the right places to hook in would be that still give us a realistic test but remove the need to re-parse the same data over and over again.

Yeah this is what I was hinting at in https://github.com/apache/datafusion/pull/12978#issuecomment-2542335627.
I'm happy to throw threads at it for a start, and restricting the search space might be necessary but I think requires a more careful eye to minimize how much valuable testing is discarded. The other thing that I think we can do is speed up the tests themselves, in particular minimizing unnecessary round trips to Parquet, but I'm not sure where the right places to hook in would be that still give us a realistic test but remove the need to re-parse the same data over and over again.

Awesome -- I'll try and find time later today or tomorrow to give it a critical eye. Otherwise I'll plan to merge this PR later today or tomorrow as well.

Yeah this is what I was hinting at in #12978 (comment).
I'm happy to throw threads at it for a start, and restricting the search space might be necessary but I think requires a more careful eye to minimize how much valuable testing is discarded. The other thing that I think we can do is speed up the tests themselves, in particular minimizing unnecessary round trips to Parquet, but I'm not sure where the right places to hook in would be that still give us a realistic test but remove the need to re-parse the same data over and over again.

Filed https://github.com/apache/datafusion/issues/13946 to track this

Thank you again @adriangb  for bearing with us -- I know this took a long time
However, I am pretty stoked that we now have this optimization and it is an example of the very careful engineering required for this kind of optimization. The fact we are at this point in DataFusion is pretty sweet in my mind
🚀


Test scenarios for Datasource authentication

Bearer Authentication Scenarios

[x] Datasource authentication & connection with valid bearer token

[x] Valid token for bearer authentication

[x] End-to-End test - pass valid bearer token & ensure recommendations are generated

[x] Exception handling when incorrect token is passed

[x] Bearer authentication

[x] Expired token
[x] Invalid token
[x] Empty token
[ ] Invalid characters in token

[ ] Authentication of Multiple datasources (Is it supported?)

[x] Test with supported datasources

[x] Test logging for any authentication messages, retries

[x] Validate documentation

Updated the above list after the 1307 merge.

The above issue has been updated to include only bearer auth scenarios and the basic auth scenarios has been moved to #1425 .
Closing this issue since the bearer auth tests was added as part of #1307


[9.4.x] ISPN-10080 Test cache managers created in other threads cannot join the cluster with UDP

https://issues.jboss.org/browse/ISPN-10080
Backport of #6794

Putting on hold becaues https://github.com/infinispan/infinispan/pull/6443 wasn't integrated in 9.4.x