Diff with Non-Intersecting Polygons

Not sure whether this is a bug or a lack in my understanding:
I got surprised that I get an null result for the diff of non-intersecting polygons.
Is this expected and should my code first check whether the polygons are intersecting and only then try to calculate the diff?
Examples
const gh = require('greiner-hormann')
const pencilPolygon = [{"x":0.11612903225806452,"y":0.33455451259745367},{"x":0.11720430107526882,"y":0.33455451259745367},{"x":0.11720430107526882,"y":0.33064826259745367},{"x":0.11612903225806452,"y":0.33064826259745367}];
const eraserPolygon = [{"x":0.14516129032258066,"y":0.4109600939072074},{"x":0.14623655913978495,"y":0.4109600939072074},{"x":0.14623655913978495,"y":0.3718975939072074},{"x":0.14516129032258066,"y":0.3718975939072074}];

gh.diff(pencilPolygon, eraserPolygon); // -> null
gh.intersection(pencilPolygon, eraserPolygon); // -> null

I would assume either one or the other to return something. In this case diff returning the pencilPolygon.

That's strange. True, diff should return first polygon in this case.

Interestingly "martinez" does:
const m = require('martinez-polygon-clipping');
const p2arr = (p) => ([p.x, p.y])
m.diff([ pencilPolygon.map(p2arr) ], [ eraserPolygon.map(p2arr) ]);

[ [ [ 0.11612903225806452, 0.33455451259745367 ],
    [ 0.11720430107526882, 0.33455451259745367 ],
    [ 0.11720430107526882, 0.33064826259745367 ],
    [ 0.11612903225806452, 0.33064826259745367 ] ] ]

Maybe it would make sense to compare the two libs in test to find such discrepancies? They should always return the same results, right?

No, you already found it) Greiner-Hormann is a very primitive library with it's own problems, so this just has to be fixed in the output here https://github.com/w8r/GreinerHormann/blob/master/src/polygon.js#L273


[bn] adding files in content/bn/example

adding files in content/bn/example
Fixes : #44993

wonderful @SAShameem .
Thanks for contributing

/lgtm

/lgtm

To fix netlify build  errors follow this steps :

1 ) go to your fork repo .
2) go to dev-1.24-bn.1 branch
3 ) sync fork

go to PR section and create new PR

take your own forked repo as a base repository

take your PR branch as base

take dev-1.24-bn.1 as compare and create PR

merge this PR

10 ) and done

@SAShameem  Kubernetes is one of the largest open-source repositories in the world, and as such, we encounter many obstacles along the way. It's our responsibility to address and overcome them  as a contributor .

wonderful work @SAShameem .
Thanks for contributing

Thanks Bhaiya for giving the opportunity. Happy to be a part of Kubernetes with my little contributions;
I will fix the netlify build errors soon.

Well done vaiya
/lgtm


Moment Difference returns wrong number if locale is set to 'fa'

Describe the bug
The package returns wrong number when you want to find the differences (day, month, year, ...) if the locale is set to 'fa'.
moment('2019-10-26').locale('fa').diff(moment('2019-10-28').locale('fa'), 'month') // 108!!
moment('2019-10-26').locale('en').diff(moment('2019-10-28').locale('en'), 'month') // 0
To Reproduce
see demo

I added a test and it worked, but why the demo is wrong? can you check the demo?

Thank for your input.
The fiddle is pretty simple. I don't know how the tests are passing but the demo result is wrong. I even tested it again in a node environment but the result is the same. I get wrong numbers if locale is fa.

can you please check the code and find the problem?

Can you please check the 3.3.5?

Sorry I didn't get a chance to get at it sooner.

good points. thank you.
can you please create a PR?

Sure, I'll work on that.

I created this pull request https://github.com/fingerpich/jalali-moment/pull/96
I borrowed code from the original moment diff method. The tests are passing. Hope it resolves the issue.

please check v3.3.7

I did several other tests in the browser and it looks fine.
Thanks!


Create Auth ENV config prop

Feature Suggestion
We want to be able to change the authentication environment for passing onto AuthProviders like the Okta Provider https://github.com/backstage/backstage/blob/master/packages/core-api/src/apis/implementations/auth/okta/OktaAuth.ts#L43
Possible Implementation
Have a top level auth.environment key in the config which gets sent to the frontend for use by AuthProviders

Interesting project to explore! I am fairly new to the code, but I am sure I can take up this task to get a hang of code base.

OK, assigned!

Hi @freben Thank you for assigning the issue. I have raised a PR.
I will try to explain my understanding to give more context on the changes I have made in the PR;
Intention
Should be able to override the hard-coded value provided for environment attr when calling static create func for any auth provider.
Solution

Added auth.environment attr to the top level  of app-config.yaml file as mentioned
Introduced relevant Config schema attributes to pass the value through configApi
Passed the value through createApiFactory authProvider1.create function so default environment value gets overridden with the value passed.

I hope we can close this issue

For sure!


Allow the main Function Apps view to be pinned

You can go to the function apps view and hit the pin icon and you'll get this message:

But unfortunately if you then try to use this tile, it will take you back to a specific function app, probably due to #1061.
This issue is specifically tracking the ability to pin a tile for the main function apps view. The new navigation makes it so easy to switch between function apps, I think I would use this more than individual pinned tiles for each function app.

You can pin it to the sidebar

I didn't realize there is now a "function apps" category. That's great. Closing.


Challenge support: tls-sni-01

Add support for the tls-sni-01 challenge.

This is removed in ACME v2


Norfair Vs. ByteTracker or DeepSort Inference Speed

Hello @joaqo,
I have a question regard to speed of Norfair tracking...
Im using YoloV5 as a object detector for both NorFair and ByteTracker (or DeepSort) tracking algorithms with the same setting. However, all the process for one frame (Detection and tracking) took almost ~14ms for ByteTracker and ~45ms for Norfair. I would like to ask you how can I increase the inference time of Norfair?
Thanks!

Hi @Rm1n90,
I am also using Bytetracker recently, what you think about the tracking performance comparison between these two trackers?

I think one difference between NorFair and other trackers is that NorFair is designed more as a framework for tracking, allowing the user to bring their own method for associating keypoints. Meaning other trackers can be implemented using NorFair in a sense.

@mgultekin  Hi ,
would you mind to share the python repo for bytetracker ?  if yolov7 integration would be best

This comparison dives that faster algotitm is ByteTracker
https://mpolinowski.github.io/docs/IoT-and-Machine-Learning/ML/2023-02-15-as-one-yolo-object-tracking/2023-02-15/


Add Integration section

I added this so that http://swiftpack.co will automatically index this package. The crawler searches for this pattern in readme.md of github repos.

Thanks! I'll get this tagged with 0.0.3 after I make a few more changes.


Runtime error: table index is out of bounds.

Is your feature request related to a problem? Please describe.
After connecting wallet and request to mint, I'm getting " table index is out of bounds " and screen is not responding.

Describe the solution you'd like

Check the size of the table before accessing an index: Before accessing an index in a table, always check that the index is less than the size of the table.
Use the ipairs function for iterating over tables: The ipairs function in Lua is specifically designed for iterating over tables. It skips over any nil values and only iterates over the existing indices.
Use the pairs function for iterating over tables: The pairs function in Lua iterates over all elements in the table, including nil values. If you're using pairs, make sure to check for nil values.
Check the size of the table after modifying it: If you're modifying the table (adding or removing elements) during iteration, make sure to check the size of the table again after each modification.

@gangstarr60 can you please share the full stack trace and perhaps a video/screenshot on how to reproduce the issue?
Would be helpful to know the wallet and browser used.
The described solution you posted seems GPT generated. We are not using Lua.

I got the explained error and tried with ai. I don't have more information about this language.
Wallet : phantom
Browser : chrome

No problems. Thanks for reporting it @gangstarr60 .
Can you please copy paste the full stack trace (the error logged in the console) and describe the steps to reproduce the issue?

My page is not responding after when i click somewhere. Then gives runtime error
[ An error occurred running the Unity content on this page. See your browser JavaScript console for more info. The error was:
RuntimeError: memory access out of bounds
at https://magicblock-labs.github.io/Solana.Unity-SDK/Build/ Solana.Unity-SDK.wasm:wasm-function[548]:0x7a3f1
at https://magicblock-labs.github.io/Solana.Unity-SDK/Build/ Solana.Unity-SDK.wasm:wasm-function[15496]:0x8ab693
at https://magicblock-labs.github.io/Solana.Unity-SDK/Build/ Solana.Unity-SDK.wasm:wasm-function[26517]:0xfeb587
at https://magicblock-labs.github.io/Solana.Unity-SDK/Build/ Solana.Unity-SDK.wasm:wasm-function[35210]:0x15867ab
at invoke_viiii (https://magicblock-labs.github.io/Solana.Unity- SDK/Build/Solana.Unity-SDK.framework.js:10:382263)
at https://magicblock-labs.github.io/Solana.Unity-SDK/Build/ Solana.Unity-SDK.wasm:wasm-function[31256]:0x132c075
at https://magicblock-labs.github.io/Solana.Unity-SDK/Build/ Solana.Unity-SDK.wasm:wasm-function[65776]:0x1d57469
at https://magicblock-labs.github.io/Solana.Unity-SDK/Build/ Solana.Unitv-SDK.wasm:wasm-functionl449461:0x1733bd1
at https://magicblock-labs.github.io/Solana.Unity-SDK/Build/ Solana.Unity-SDK.wasm:wasm-function[1256]:0xb349b
at https://magicblock-labs.github.io/Solana.Unity-SDK/Build/ Solana.Unity-SDK.wasm:wasm-function[35214]:0x15867e5
at invoke_viii (https://magicblock-labs.github.io/Solana.Unity- SDK/Build/...
]

I'm unable to reproduce the issue unfortunately, the hosted demo work correctly on all the setup I tested.
Could you please specify OS, browser version and anything you think is relevant to reproduce on my end.

Windows 10 Home Single Language
Chrome 121.0.6167.140
Phantom wallet


Select second star when clicking outside of stars

Hi, I'm just trying your library and it looks really good, thanks!
I discovered a bug, when clicking outside of a star, the second star get selected.
You can reproduce this bug when clicking on a tooltip in your demo.
I guess it's coming from elementFromPoint that resolve to the tag gl-star-rating--stars instead of targeting star tags but I'm not sure.
Best,
Jason

Thanks. Fixed by disabling pointer-events on pseudo-elements.


Confusing error "An item with the same key has already been added."

log.txt
issue64.zip
When you install the two attached files in Custom Connectors, you will receive this error on startup in Power BI (tested on August release).
Expectation: an error that guides the developer what to fix.

Background:
reason was this line present in pqs:
[DataSource.Kind="Invantive.ExactOnline.GB", Publish="UI.Root"]
shared Database = () as table => Database.Database();

and
[DataSource.Kind="Invantive.TeamLeader", Publish="UI.Root"]
shared Database = () as table => Database.Database();

The sections were different in the pqs, so there should be no name clash (when I understand the M language spec correctly). But it seems that
Microsoft.Mashup.Client.ClientShared.UiExtensibility.UiExtensionHost.ExtensionWebResources.AddImage(String suffix, IBinaryValue image)

does not take into account the section name. Again, when I understand the spec correctly. Just browser it.

Section's aren't exactly like namespaces in .NET/other languages. They do provide scoping for local members, but all shared members are put into a global 'namespace' once they are loaded, and must be unique.
You could have different Data Connector projects in visual studio with duplicate shared members, and run them without error, because the SDK will only load a single extension.
Power BI Desktop will load all extensions in the Custom Connectors directory, which will cause name conflicts when duplicate symbols are detected.

Thanks for the explanation; familiar concept. But IMHO the error text might be changed to better help the developer understand what action is needed. The developer has no access to the source code of power bi and now needs to look at the call stack and gamble what might help.


Remove dataproxy api

As we are using lambda as our server, there is some limitation of lambda.

The response and request payload can not be more that 6MB

That is why we are fetching resources directly from front-end using s3 url. That is why our previously created apis:

/api/dataproxy/{publisher}/{package}/r/{resource}.json
/api/dataproxy/{publisher}/{package}/r/{resource}.csv
are not getting used. So it is safe to delete these two apis.

Quick comments:

For most front-end rendering or API work 6mb is fine so I am not worried about that limitation

right now if we try and render > 5mb CSV into front-end view page stuff will not work well

In fact, 6Mb limit could be good.

So simple answer is: 6 MB limit is not an issue. Overall, I think we will want an API but we need to do more work on it and think how it will work.

WONTFIX.


Use stable Julia version

This PR updates the CI setup and the Manifest files of the tutorials to the latest stable Julia version (currently Julia 1.8.5). With the recent move away from CompatHelper to a weekly update of the Manifest files (maybe could be relaxed to a two weeks interval, to reduce maintenance burden, CI time, and size of artifacts?), it seems feasible now to (try to) keep up with the stable Julia versions.
Fixes #378.

Closed in favour of #394.


AUTOCOMMIT not available, but required for not-journaled files

Hi, I need to do an insert/update operation on a file that doesn't have journaling enabled.  So it gives the following error, you can't use transactions without journaling:
sqlalchemy.exc.ProgrammingError: (ibm_db_dbi.ProgrammingError) ibm_db_dbi::ProgrammingError: Statement Execute Failed: [IBM][CLI Driver][AS] SQL7008N  REXX variable "VARSS     " contains inconsistent data.  SQLSTATE=55019 SQLCODE=-7008
on the docs here: https://docs.sqlalchemy.org/en/20/tutorial/dbapi_transactions.html it mentions that the AUTOCOMMIT can be set as the isolation_level on the engine.  However, it gives this error:
sqlalchemy.exc.ArgumentError: Invalid value 'AUTOCOMMIT' for isolation_level. Valid isolation levels for ibm_db_sa are RR, REPEATABLE READ, READ STABILITY, CURSOR STABILITY, RS, UNCOMMITTED READ, UR, CS
Is there any way to enable AUTOCOMMIT so i can insert/upsert operations with SqlAlchemy ORM on non-journaled files?

sure,
smt = f"db2+ibm_db://{user}:{password}@{hostname}:{port}/{database}"
engine = create_engine(smt, echo=True)

sm = sessionmaker(engine), autocommit=True, autoflush=True)

with sm.begin() as session:
    record = Users(**user_dict)
    session.add(record)


Adjustable verbosity

Would be nice to be able to turn down the verbosity on the miner logs.
Right now the greenish text tends to be redundant with the full file name etc. and it's a very long line that wraps that even fullscreen on 1080p for me causes the text to wrap.  Perhaps an option to disable that Processed: messages in config
Precision on the MBps could also be set so 21.1112341413245235Mbps is shown as 21.1 or 21.11 instead as that level of prevision just complicates the log viewing.

This should be configurable already but I'll add additional configuration for rounding and move the log level of these messages.


obs-studio: fix build with alsa/pulseaudio/pipewire support disabled

Description of changes
Fixes #273205

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

darwin isn't listed in meta.platforms, think it's just borgo being slow

yeah I noticed that too, not sure if it goes with this PR or should be separate.

Please rebase on master

yeah I noticed that too, not sure if it goes with this PR or should be separate.

Either seems fine to me.

In that case I'd rather do a separate PR, as it will require effort to make sure things work properly.


[REQUEST] Library of Ruina (Nintendo Switch ver.)

Engine is Unity so AFAIK shouldn't be hard to make a script for the novel sections
https://www.nintendo.com/us/store/products/library-of-ruina-switch/

PC version works with Textractor sorry, no need for this


Left nav free training

Deploy preview for calico ready!
Built with commit 0c11c22383b973cf717ef5f8be064a735fb29a37
https://deploy-preview-3473--calico.netlify.app


Console Spaming with GLTF current state

After https://github.com/decentraland/godot-explorer/pull/82
I get a lot of logs with the following messages:
2023-11-10T21:28:53.768229Z  INFO decentraland_godot_lib::scene_runner::components::gltf_container:  scene_id 23 	 entity_idSceneEntityId { number: 515, version: 0 } 	 current_state Loading 	 current_state_godot: Loading
2023-11-10T21:28:53.768343Z  INFO decentraland_godot_lib::scene_runner::components::gltf_container:  scene_id 23 	 entity_idSceneEntityId { number: 519, version: 0 } 	 current_state Loading 	 current_state_godot: Loading
2023-11-10T21:28:53.784365Z  INFO decentraland_godot_lib::scene_runner::components::gltf_container:  scene_id 23 	 entity_idSceneEntityId { number: 515, version: 0 } 	 current_state Loading 	 current_state_godot: Loading
2023-11-10T21:28:53.784457Z  INFO decentraland_godot_lib::scene_runner::components::gltf_container:  scene_id 23 	 entity_idSceneEntityId { number: 519, version: 0 } 	 current_state Loading 	 current_state_godot: Loading

Repro steps: Go to Goerli Plaza center

Closed by https://github.com/decentraland/godot-explorer/pull/85


Microphone support

Hi, is there a way to access the microphone of the HoloLens2 beside of the speech mappings?

Yes, you can use standard UWP audio capture API
https://docs.microsoft.com/en-us/uwp/api/windows.media.devices.mediadevice?view=winrt-22000


Can't access FirewallWAS.Rules in v<IP_ADDRESS>

Hi,
I'm using FirewallHelper in a .NET Framework 4.8 project and have just updated from 1.6xx to <IP_ADDRESS>. I can't use the FirewallWAS.Rules method because it tells me that I need C# 8.0 or later and Framework 4.8 is limited to 7.3. CS8370 is raised saying that recursive patterns are not available in C#7.3. See embedded image.
I had to look through the change log to get some essential migration information; a useful addition to your documentation would be a basic migration guide saying things such as "profiles are now specified in the GetProfile constructor in FirewallManager.Instance.GetProfile()" and "change all FirewallAPIV2 references to FirewallWAS etc.
Many thanks, Peter

You are trying to compare the type of a variable to a property. This error is irrelevant to the C# version it is just a bad error. With .Net4.7.2 and C# 7.3 the following piece of code works:

As for the migration, you do have a point there however unfortunately I am quite busy at the moment. But I will keep that in mind. Thank you.

Many thanks I have now successfully migrated to version 2. I cloned the repository in the end so that I could examine it in Visual Studio to see how members had been reworked and renamed, which helped a lot.
Regards, Peter


merge package template from Kronos-Tools/npm-package-template

Updated standard to latest version

Codecov Report

Merging #42 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #42   +/-   ##
=======================================
  Coverage   83.33%   83.33%
=======================================
  Files           1        1
  Lines          12       12
=======================================
  Hits           10       10
  Misses          2        2

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ad8c5c9...6ed937d. Read the comment docs.


Streaming candles on futures after-hours

I am having trouble streaming candles for futures in after-hours.  I have tried the code listed below, based on some of the examples provided:
async def main():
    session = Session('my_username', 'my_password')
    async with DXLinkStreamer(session) as streamer:
        await streamer.subscribe_candle(['/ES'], '15s', datetime.now(), extended_trading_hours=True)
        async for candle in streamer.listen(Candle):
            print(candle)

asyncio.run(main())

My code executes but no candles are ever printed.  I am running this during the extended hours for ES trading.

Hello! The problem here is twofold:
First, you're just using the symbol /ES, which is ambiguous as there are a whole host of futures under that family. One you could try that's currently active is /ESZ4.
Second, the streamer uses a different symbology than Tasty. So the way to handle this is to first make a call to Tasty to get the streamer symbol, then use that to subscribe to quotes/candles like so:
from tastytrade.instruments import Future
future = Future.get_future(session, '/ESZ4')
# ...
await streamer.subscribe_candle([future.streamer_symbol], '15s', datetime(2024,12,1,0,0), extended_trading_hours=True)
async for candle in streamer.listen(Candle):
    print(candle)

If you want an algorithmic way of getting the future instead of using its symbol, you can use tastytrade.instruments.FutureProduct which will work with just the base symbol (/ES) and return which month codes are currently active.
Hope this helps!

Thanks for the quick reply @Graeme22 !  That works perfectly.

Great! If you're willing to add something like that to the documentation for posterity, that would be appreciated. Good luck trading!


[.run] add context about .run folder

What does this PR do?
Add context .run containing out of the box tasks for IntelliJ Goland
Motivation
I did not know what .run was
Describe how to test/QA your changes
Possible Drawbacks / Trade-offs
Additional Notes

/merge


opens/ : Suggested Open List : Wrong order ?

The list is ordered by score then ppm : .order_by('-score', 'ppm')
But, the order on ppm is from small to big, don't we want to peer with the one we pay the most : .order_by('-score', '-ppm') ?
https://github.com/cryptosharks131/lndg/blob/31a02f9279e034c43d4fd15d9bd41bb1bd0a54d7/gui/views.py#L411

Order is intentionally this way.
ppm represents the fees paid to go through that peer. We want to avoid connecting to expensive peers.


add index.d.ts

@nakagawa424 thank you very much, this is absolutely another great idea!
Sorry to be so brute, I did not merged the PR, just roughly copyed files (but your contribution is really welcome:)
Now it is released everything on the npm registry :metal:


Include cumulativetodeltaprocessor from contrib

https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/cumulativetodeltaprocessor
I would like to use this Processor with the Sumo collector distro.
I don't see it defined in https://github.com/SumoLogic/sumologic-otel-collector/blob/main/otelcolbuilder/.otelcol-builder.yaml
Is there a reason for this? Or am I misunderstanding something?
Thanks!
Edit: https://github.com/SumoLogic/sumologic-otel-collector/pull/597

Fixed by https://github.com/SumoLogic/sumologic-otel-collector/pull/597


Sync Main with UAT

Read the title.

@exquisitebin @Taz17 @emmapainter @Mhudlark why would you approve with a failed CI? Disappointing...

fix ci

Fixed. Please approve.


Minor reword of pull secret notification to apply to more scenarios

This represents a minor tweak to the wording of scenarios where the cluster owner has modified the cloud.redhat.com pull secret, so that it is more reusable for the different ways in which the secret can be modified.

/approve

/lgtm


bitsy hd option doesn't work when exported

I'm new here so I'm sorry if I don't understand a lot of stuff. Making the game works perfectly fine. I can even test the game perfectly fine but it looks like it goes back to 8x8 pixels once I export it. Is there a way around this? Thank you!

Same issue here, I think. The game loads when exported, but it tries to go back to 8x8 pixels, and only loads part of the tiles/sprites as a result. I have only part of a window showing up, for example.


Add pse-tools24 extension

Description

Screencast

Checklist

[ ] I read the extension guidelines
[ ] I read the documentation about publishing
[ ] I ran npm run build and tested this distribution build in Raycast
[ ] I checked that files in the assets folder are used by the extension itself
[ ] I checked that assets used by the README are placed outside of the metadata folder

Please disregard this request.  I am trying to publish this extension privately.


want to change the hover button color

want to change the color to light red of the button on hovering the button

Please assign me this as my first issue @saurabhsingh720

I know how to fix this , So would like to work on this project. Please assign it to me under GSSOC'23 @pranjay-poddar @saurabhsingh720

hey @pranjay-poddar  please assign it to me . I know how to handle it.

I want to work on this issue ,please assign it to me under GSSOC'23 @pranjay-poddar

Please assign me this as  issue

Hey @saurabhsingh720 , Thanks for opening the issue. It would be great if you provide more description for the issue. Then only I can assign it to you.

Hey @saurabhsingh720 , Thanks for opening the issue. It would be great if you provide more description for the issue. Then only I can assign it to you.

I want to change the color as well as some transition on hovering the button

I want to work on this issue, please assign it to me.


Improved examples and added notes about how to run them

@AlexCppns Merging so your can rebase on it.


Incorrect grad on complex number

Description
Hi, thanks for maintaining JAX. I believe the behavior of taking gradient on the imag part of complex-valued input is incorrect, differing in sign. Here's an example
import jax
import jax.numpy as np

jax.grad(lambda x: x.imag**2)(np.array(2j))

The above code gives Array(0.-4.j, dtype=complex64). However considering the definition of grad, it should be positive. Here's a torch version of it, which gives the correct grad (tensor(0.+4.j),):
import torch

x = torch.tensor(2j, requires_grad=True)
y = x.imag**2
torch.autograd.grad(y, x)

What jax/jaxlib version are you using?
jax/jaxlib v0.4.5 on GPU, v0.4.8 on CPU
Which accelerator(s) are you using?
Tried on both CPU/GPU
Additional system info
Python 3.8. Tried on both Ubuntu 22.04 LTS (x86) & MacOS 13.0
NVIDIA GPU info
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.30.02              Driver Version: 530.30.02    CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+

Thanks for raising this!
Actually this is just a different convention for what grad means; torch and jax follow different conventions, which differ by a complex conjugate. In JAX we define grad(f)(x) == jax.vjp(f, x)[1](1.0)[0] (following a choice we made in the original Autograd, basically from at the time preferring grad(lambda x: c * x) == c for complex c). That means for complex inputs it does not correspond to what you'd get from the more natural mathematical definitions, e.g. defining the gradient as the representer for the vjp-with-ones covector in terms of the standard (complex) inner product; instead it differs by a conjugate.
I'd like to switch conventions to the more mathematically motivated one (i.e. the same as what torch does). See #4891. But it just hasn't been a high priority. At the end of the day, it's just a convention, and you can just apply jnp.conjugate, or define your own grad function in terms of jax.vjp. (Notice, interestingly, that jax.vjp is unambiguous here: there's no need to choose some convention for what it means!)
The current JAX convention is documented in the autodiff cookbook.
Maybe we can keep discussion about changing the convention on the thread in #4891? But until we do, just apply a conjugate (or call jax.vjp directly).

Hi mattjj,
Thanks for prompt response. I encountered this when I was performing regression on complex inputs/outputs, and I was surprised that I can't directly minimize loss with jax/flax. To me it's a little bit counter-intuitive that I should .conj before gradient descent. I'll keep up with the thread you mentioned later, thanks!


[NTR/gene]

Preferred gene-related syndrome label
CPOX-related hereditary coproporphyria
Synonyms
Parent term (use OLS, or your favorite ontology browser)
inherited porphyria
Definition
Any inherited porphyria in which the cause of the disease is monoallelic or biallelic variants in the CPOX gene.
Definition source (Please give PubMed ID, if applicable, in format PMID:#######)
Per criteria outlined by the ClinGen Lumping and Splitting Working Group, the molecular mechanism (CPOX loss-of-function) was found to be consistent between the harderoporphyria cases and hereditary coproporphyria cases. In addition, the phenotypic differences between the biallelic and monoallelic cases appeared to represent a single spectrum of disease. Therefore, cases caused by inherited CPOX variants have been lumped into a single disease entity referred to as CPOX-related hereditary coproporphyria, with a semidominant mode of inheritance.
Children terms (if applicable) Should any existing terms be re-classified as children underneath this new proposed term?
hereditary coproporphyria (MONDO:0007369), harderoporphyria (MONDO:0030048)
Your nano-attribution (ORCID) or URL for a working group
https://clinicalgenome.org/affiliation/40097/
https://clinicalgenome.org/working-groups/clinical-domain/inborn-errors-of-metabolism/

Thank you in advance for your help in considering this proposed term.


2 graph validator errors blocking dataset submission

I have uploaded a dataset with valid metadata to ingest prod. However, there are many graph validator errors:

There are 3 protocols linked to no entity and tagged with 'invalid graph = yes'. These protocols shouldn't be in the spreadsheet, can they please be deleted? I can't see an option to delete protocols currently.

The fastq files are all uploaded and shown as valid, but something is going wrong with the linking in ingest. It is showing each cell suspension linked to only 1 sequence fastq file, whereas in the data tab it can be seen that all expected fastq files are present and valid.

Submission: https://contribute.data.humancellatlas.org/submissions/detail?uuid=13beb31c-1aa4-4355-9562-7e0fad2065b2

Could whichever dev who picks this up please time-track roughly how long it takes to complete (1.)? Would be really useful to know!

@ami-day the linking in the spreadsheet has issues:

You have to specify a process id if you want files to be included in the same assay process.
I'm afraid you'd have to delete this submission and reupload the files again. We dont have bulk updates of linking supported yet. :(

Ohhh I see, i'll modify and re-upload and see if it fixes all the linking errors

Issue is fixed.


[enhancement]: Import Model - Auto detect file name and auto fill that name into Model Name field

Is there an existing issue for this?

[X] I have searched the existing issues

Contact Details
No response
What should this feature add?
It could save more time when importing new model by auto detect file name and auto fill that file name into Model Name field.

Alternatives
No response
Aditional Content
No response

Added: #4445


Cloud backup improvements

The user shouldn't be prompted to enter a password every time they click backup, if a password is set the user should have somewhere an option to change the password (maybe in a settings page), but it should not ask the user to enter a password every time.
Should reconsider whether to use BSMS in the PDF, it may be better to store a normal descriptor, or store an actual .bsms
Healthcheck doesn't check much, only that files exists and accessible and how many there are. Also not much info is shown on healtcheck success/ failure, making it less useful. We could show a toast with more info for now.

@AreejKhalid21 we could use here a design update with the make the backup now button popup say this will save the backup to your cloud (ie. some general explanation), and have an option to change password, but it should by default just continue with the saved password.


auto generated C# bindings / Toolchains iOS/Android

Hi,
Hope you are all well !
I am actually working on some C++ libraries, like Dlib or OpenCV, and I wanted to auto generate unsafe bindings for some computer vision libraries to build native plugins for Unity3D.
After a couple of researches, I found your great work and some questions popped out of my brain :-)
Context:
I wanted to test this example, dnn_mmod_dog_hipsterizer.cpp, from  dlib, in order unity3d to leverage the WebCamTexture class. I saw a module in the asset store using dlib but I wanted to test ccv and some other CMake driven projects. So, I am looking for something scriptable and more flexible for my studies.
So, in a nutshell, I would like to auto-generate bindings from some C++ libraries and allow interoperability and build these projects with CMake, with polly, a collection of toolchains to cross-platform build and test these projects..
I found CppSharp and still trying to find out the best way to get something, like FFmpeg.AutoGen, also using CppSharp.
 To make it more flexible, I was thinking to wrap all these commands inside  a docker container with onbuild instructions. Even if you cannot compile xcode projects from a docker container,  it would be great for Android, Linux or Windows builds.
Questions:
bindings related:

How did you proceed for aruco-unity like Calib3d.cs ? Did you write all the bindings yourself ? (if yes, that's a lot of work :-) )

Did you try to use Swig or CppSharp for the aruco-unity project ?

How would you see a much more structured and semi-automated flow to build native modules,  based on Cmake driven projects, for Unity3d ?

cmake related:

Would aruco-unity work for iOS or Android if we provide a cmake toolchain ?

Thanks in advance for your input on my quest :-)
Wish you a great week !
Cheers,
Richard

Hi,
I had an equivalent context than you: there is some AR modules in the asset store but I wanted to use the OpenCV's ArUco module in Unity. And I wanted a free license module. That's why I started this project.
To answer your questions:
bindings related:

I wrote all the bindings by myself. First I just wanted to test the ArUco library on Unity. At this time, I didn't know anything about bindings, so I started with manual bindings in order to learn. And it worked pretty well as I expected so I continued on to finish the project. Retrospectively I may have used Swig or CppSharp, because yes it's a lot of work!
I haven't take the time to try Swig or CppSharp yet! I think I should, my manual bindings are far from perfect. For example I should use IDisposable interface to manage the memory, there is no tests, or there may be memory leaks with the actual bindings. On the other hand, the core of ArUco Unity is almost finished (port the features of ArUco for Unity for Windows).
It could be very interesting. I wanted first with this project to make ArUco available for Unity for Windows. Then port the project to other desktop and mobile platforms. I think it could be a great idea to use Hunter and Polly.

cmake related:

Yes this is possible (see this ArUco port for Unity for Android, and OpenCV for Unity works for Android and iOS). When the port for Windows will be finished, I will port to Android, iOS, Linux and Mac OS. But I'm wondering of the performances on mobile with the current project's architecture.

I don't know anything about docker, but it may be interesting to use.
Tell me if you test ccv and Dlib on Unity, I'm interested about it. If you want to use WebCamTexture with these libraries, take a look ArucoCameraWebcam.cs and ArucoCamera.cs to put the WebCamTexture to a more easier to use Texture2D with which you can use GetRawTextureData() and LoadRawTextureData() to get and set pixels as an array of bytes.
Does that answer well your questions?
Best
Erwan


error x509: certificate signed by unknown authority when using container image v1.3.0

Hi,

just for the paper work, I add next to PR also an issue report here.

With the v1.3.0 we are getting the following error message:
time="2022-12-..." level=fatal msg="Could not retrieve quotas and limits: failed to list quotas: WebIdentityErr: failed to retrieve credentials\ncaused by: RequestError: send request failed\ncaused by:

Post "https://sts.amazonaws.com/": x509: certificate signed by unknown authority"

A possible fix is provided in this PR: https://github.com/thought-machine/aws-service-quotas-exporter/pull/36

resolved in #36


Support custom log implementations

It would be great if this library could be instructed to use user-configurable logging mechanisms. Specifically, this would make integration into larger applications (and their log handling) easier. On top of this, some logging implementations can be turned on/off during the runtime of an application for improved debugging.
Thinking of this goal, the following would be great to have in kafka-node:

Set custom logging implementations, e.g.

var kafka = require('kafka-node');
kafka.setLoggerProvider(myLoggerProvider);

Where myLoggerProvider can be called with a name (string) to create a new logger. A logger is any object with a .warn, .info and .error function, e.g. console, bunyan, winston or pino loggers.
Example provider for debug logger:
var debug = require('debug');

function debugLoggerProvider(name) {
  var logger = debug(name);
  logger = logger.bind(logger);

  return {
    info: logger,
    warn: logger,
    error: logger
  };
}

Continue to use debug as the default logging implementation for full backward compatibility.
Adapt the kafka-node code to use the user configured logger with reasonable log levels.

The change would be backward compatible and I am willing to contribute it. So if you agree, I'll get right to it.

That sounds great. Go for it!


Add a way to change black key notes

There should be a setting so you can change it so that it reads B flat instead of A#

Will be added in the next release

Added in 6b6e30fffd15715504beb104ab384fb9bd6c41ee


OPC-N3 Power Requirements

Hi @Nighthater,
Thanks for this work. I do not have much experience in electronics, so I am facing some trouble in providing proper power to the OPC-N3. Please suggest me how should I power it when using it with along either ESP32 or Raspberry Pi?
Should I power it via RPi/ESP or should I power it separately via 5V mobile charging adapter? I tried to power it via ESP32 Vin and GND pins, but its not working. Your help would be greatly appreciated. Thanks :)

Hi @yash-dahima,
You can power the OPC with either the VIN/GND pins (RPi: 5V/GND) or a 5V Charging adapter. When using an external power supply, like an external charger, make sure that the GND of the Charger and the GND of the ESP/RPi are connected together.
The sensor also won't power on after supplying it with power. You have to send it commands to start up and start measuring data.
Do you currently use an ESP or a RPi to interface with the OPC?
You should also consider that the SPI logic levels on the OPC-N3 Sensor are 3.3V, despite its 5V power supply. I chose an ESP because of that. A RPi uses 5V logic and probably wont recognize a 3.3V high Signal.

Thanks you very much @Nighthater for your prompt response. I will attempt as per your suggestions and will let you know how it goes.
Again thanks and best wishes :)


[extractor/safari] Partially fix extractor

Please follow the guide below

You will be asked some questions, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your pull request (like that [x])
Use Preview tab to see how your pull request will actually look like

Before submitting a pull request make sure you have:

[x] Searched the bugtracker for similar pull requests
[x] Read adding new extractor tutorial
[x] Read youtube-dl coding conventions and adjusted the code to meet them
[x] Covered the code with tests (note that PRs without tests will be REJECTED)
[x] Checked the code with flake8

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

[x] I am the original author of this code and I am willing to release it under Unlicense
[x] I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

[x] Bug fix
[x] Improvement
[ ] New extractor
[ ] New feature

Description of your pull request and other information
Hey @dirkf, @epsilonSpider, @Kerruba I just tried to modify this code a bit with updated urls to fix the downloading at least but still getting 1 minute video download error. Seems like kaltura extractor is having problem downloading correct index.m3u8.

Is there an issue that should be linked? Otherwise you could post a failing verbose log (using the master or nightly code).

Sometimes I am getting kaltura extractor error after downloading first video which is also defaulting to the 1:00 minute.
BTW I am sharing the verbose logs
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--username', 'PRIVATE', '--password', 'PRIVATE', 'https://learning.oreilly.com/course/python-complete/9781835465141', '--verbose']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python 3.12.4 (CPython x86_64 64bit) - Linux-6.9.3-zen1-1-zen-x86_64-with-glibc2.39 - OpenSSL 3.3.1 4 Jun 2024 - glibc 2.39
[debug] exe versions: ffmpeg 6.1.1, ffprobe 6.1.1
[debug] Proxy map: {}
[generic]<PHONE_NUMBER>141: Requesting header
[redirect] Following redirect to https://www.oreilly.com/videos/python-complete/9781835465141/
[safari:course]<PHONE_NUMBER>141: Downloading course JSON
[download] Downloading playlist: Python – Complete Python, Django, Data Science and ML Guide
[safari:course] playlist Python – Complete Python, Django, Data Science and ML Guide: Collected 470 video ids (downloading 470 of them)
[download] Downloading video 1 of 470
[safari:api]<PHONE_NUMBER>141/video1_1: Downloading part JSON
[Kaltura]<PHONE_NUMBER>141-video1_1: Downloading webpage
[Kaltura] 1_4kzn6p7b: Downloading video info JSON
[Kaltura] 1_4kzn6p7b: Checking mp4-1829 URL
[Kaltura] 1_4kzn6p7b: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/p/1926081/sp/192608100/playManifest/entryId/1_4kzn6p7b/format/url/protocol/http/flavorId/1_lmo6gks3'
[download] Destination: Introduction to the Complete Python Guide-1_4kzn6p7b.mp4
[download] 100% of 17.36MiB in 00:30
[download] Downloading video 2 of 470
[safari:api]<PHONE_NUMBER>141/video1_2: Downloading part JSON
[Kaltura]<PHONE_NUMBER>141-video1_2: Downloading webpage
[Kaltura] 1_6w8js3cg: Downloading video info JSON
[Kaltura] 1_6w8js3cg: Checking mp4-656 URL
[Kaltura] 1_6w8js3cg: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/p/1926081/sp/192608100/playManifest/entryId/1_6w8js3cg/format/url/protocol/http/flavorId/1_h93c65eo'
[download] Destination: Course Overview-1_6w8js3cg.mp4
[download] 100% of 5.00MiB in 00:09
[download] Downloading video 3 of 470
[safari:api]<PHONE_NUMBER>141/video1_3: Downloading part JSON
[Kaltura]<PHONE_NUMBER>141-video1_3: Downloading webpage
[Kaltura] 1_shb1bbeq: Downloading video info JSON
[Kaltura] 1_shb1bbeq: Checking mp4-517 URL
[Kaltura] 1_shb1bbeq: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/p/1926081/sp/192608100/playManifest/entryId/1_shb1bbeq/format/url/protocol/http/flavorId/1_9fkfiea4'
[download] Destination: Where to Write and Run Python Code-1_shb1bbeq.mp4
[download] 100% of 6.04MiB in 00:11
[download] Downloading video 4 of 470
[safari:api]<PHONE_NUMBER>141/video1_4: Downloading part JSON
[Kaltura]<PHONE_NUMBER>141-video1_4: Downloading webpage
[Kaltura] 1_n5y3sn9d: Downloading video info JSON
[Kaltura] 1_n5y3sn9d: Checking mp4-595 URL

I have again tried to simplify the extractor and updated it. Please check that out

Also, see https://github.com/yt-dlp/yt-dlp/issues/3526#issuecomment-2129882177.

Probably the free version is what was analyzed. For an entire course, your API URL can be fetched with _download_json() and the playlist extracted. Tyhe playlist items like https://learning.oreilly.com/api/v1/book/9781835465141/chapter/video1_2.html also give JSON (despite appearances) and then you have a block like this for each video clip:
    {
      'thumbnail': null,
      'reference_id': '9781835465141-video1_2',
      'brightcove_id': null,
      'video_still': null,
      'created_time': null,
      'last_modified_time': null,
      'duration_seconds': 346,
      'streaming_source': 'Kaltura'
    }

This was served to me in a non-JS browser with no authentication.
IIRC Kaltura needs the video ID (or ref_id, as here, I guess) and partner/account IDs, which are probably effectively constant but should be found in the page or linked JS code. From the clip parameters one might infer that some clips are hosted at Brightcove, so it would be good to find a case where that applies.

Nope I am not using any free version. I have also tried to create an trial account today and tried to watch the full video using that but the extractor maybe only trying to fetch kaltura link wrongly so that the full duration file is not properly fetched. I already fixed the

Also, see yt-dlp/yt-dlp#3526 (comment).

I have actually fixed the KeyError('next') bug using this pull request already. The problem is its fetching the kaltura demo link off the video instead of the full video length. Oreilly changed something to their API so its no more working. Actually I was using it for a long time to download playlists. But its no more working to download the playlist with full video length.

Also, see yt-dlp/yt-dlp#3526 (comment).

Tried to patch this file same results as me. Limiting downloading to 1:00 minute

Got another interesting problem where it really downloads the wrong format even if I set the --format mp4-2363 for an example in the flag it always downloads the format mp4-1829 with less file size than the original one.
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--username', 'PRIVATE', '--password', 'PRIVATE', 'https://www.oreilly.com/library/view/python-complete/9781835465141', '--playlist-items', '1', '--list-formats', '--verbose']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python 3.12.4 (CPython x86_64 64bit) - Linux-6.9.3-zen1-1-zen-x86_64-with-glibc2.39 - OpenSSL 3.3.1 4 Jun 2024 - glibc 2.39
[debug] exe versions: ffmpeg 6.1.1, ffprobe 6.1.1
[debug] Proxy map: {}
[safari:course]<PHONE_NUMBER>141: Downloading course JSON
[download] Downloading playlist: Python – Complete Python, Django, Data Science and ML Guide
[safari:course] playlist Python – Complete Python, Django, Data Science and ML Guide: Collected 470 video ids (downloading 1 of them)
[download] Downloading video 1 of 1
[safari:api]<PHONE_NUMBER>141/video1_1: Downloading part JSON
[Kaltura]<PHONE_NUMBER>141-video1_1: Downloading webpage
[Kaltura] 1_4kzn6p7b: Downloading video info JSON
[Kaltura] 1_4kzn6p7b: Checking mp4-1829 URL
[Kaltura] 1_4kzn6p7b: Downloading m3u8 information
[info] Available formats for 1_4kzn6p7b:
format code  extension  resolution note
mp4-56       mp4        audio only   56k , isom container, 0fps, ~1.04MiB
hls-58       mp4        audio only   58k , mp4a.40.2
mp4-464      mp4        640x360     464k , isom container, avc1, 30fps, ~8.60MiB
hls-508      mp4        640x360     508k
mp4-660      mp4        854x480     660k , isom container, avc1, 30fps, ~12.22MiB
hls-713      mp4        854x480     713k
mp4-827      mp4        960x540     827k , isom container, avc1, 30fps, ~15.32MiB
hls-887      mp4        960x540     887k
mp4-1272     mp4        1280x720   1272k , isom container, avc1, 30fps, ~23.55MiB
hls-1353     mp4        1280x720   1353k
mp4-1356     mp4        1280x720   1356k , isom container, avc1, 30fps, ~25.11MiB
mp4-1829     mp4        1920x1080  1829k , m4v container, avc1, 60fps, ~33.85MiB
mp4-2363     mp4        1920x1080  2363k , isom container, avc1, 30fps, ~43.74MiB (best)
[download] Finished downloading playlist: Python – Complete Python, Django, Data Science and ML Guide

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--username', 'PRIVATE', '--password', 'PRIVATE', 'https://www.oreilly.com/library/view/python-complete/9781835465141', '--playlist-items', '1', '--format', 'mp4-2363', '--verbose']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python 3.12.4 (CPython x86_64 64bit) - Linux-6.9.3-zen1-1-zen-x86_64-with-glibc2.39 - OpenSSL 3.3.1 4 Jun 2024 - glibc 2.39
[debug] exe versions: ffmpeg 6.1.1, ffprobe 6.1.1
[debug] Proxy map: {}
[safari:course]<PHONE_NUMBER>141: Downloading course JSON
[download] Downloading playlist: Python – Complete Python, Django, Data Science and ML Guide
[safari:course] playlist Python – Complete Python, Django, Data Science and ML Guide: Collected 470 video ids (downloading 1 of them)
[download] Downloading video 1 of 1
[safari:api]<PHONE_NUMBER>141/video1_1: Downloading part JSON
[Kaltura]<PHONE_NUMBER>141-video1_1: Downloading webpage
[Kaltura] 1_4kzn6p7b: Downloading video info JSON
[Kaltura] 1_4kzn6p7b: Checking mp4-1829 URL
[Kaltura] 1_4kzn6p7b: Downloading m3u8 information
[debug] Invoking downloader on 'http://cdnapi.kaltura.com/p/1926081/sp/192608100/playManifest/entryId/1_4kzn6p7b/format/url/protocol/http/flavorId/1_lmo6gks3'
[download] Destination: Introduction to the Complete Python Guide-1_4kzn6p7b.mp4
[download] 100% of 17.36MiB in 00:10
[download] Finished downloading playlist: Python – Complete Python, Django, Data Science and ML Guide

I made a version of your PR from yt-dl with yt-dlp (other than conventional) changes back-ported.
With no login, I'm able to get the formats for https://www.oreilly.com/library/view/python-complete/9781835465141, with durations != 1 minute (470 items):
$ python3 -m youtube_dl --get-filename -o  '%(duration)s' 'https://www.oreilly.com/library/view/python-complete/9781835465141'
155
346
311
162
184
323
467
165
288
435
665
729
587
350
487
488
...

However, when I get the video it comes as 60s rather than the expected duration.
Presumably with a valid login it takes the if self.LOGGED_IN: branch and gets the actual course media.
Please show the actual command that gives the unexpected format. I very much hope that there is some CLI problem that is causing the specified format to be ignored.

Thank you very much @dirkf for suggesting me changes with your valuable time. BTW I have found something interesting this time.
I have currently tried to patch the changes with this commit the latest codebase in yt-dlp which can now download the full length video with this commit when --cookies are applied. But when credentials are applied its not working and getting errors. But yotube-dl downloads 1:00 minute video when applied this patch no matter the --cookies are applied or not.
Using yt-dlp-git
[debug] Command-line config: ['--continue', '--ignore-errors', '--retries', 'infinite', '--username', 'PRIVATE', '--password', 'PRIVATE', '--output', '%(playlist)s/%(playlist_index)s - %(title)s.%(ext)s', 'https://www.oreilly.com/library/view/python-complete/9781835465141', '--verbose']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version<EMAIL_ADDRESS>from yt-dlp/yt-dlp [12b248ce6]
[debug] Python 3.12.4 (CPython x86_64 64bit) - Linux-6.9.4-zen1-1-zen-x86_64-with-glibc2.39 (OpenSSL 3.3.1 4 Jun 2024, glibc 2.39)
[debug] exe versions: ffmpeg 6.1.1 (setts), ffprobe 6.1.1
[debug] Optional libraries: certifi-2024.06.02, requests-2.32.3, sqlite3-3.46.0, urllib3-1.26.18
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests
[debug] Loaded 1820 extractors
[safari:course] Checking if logged in
[debug] [safari:course] Unable to login: Authentication credentials were not provided.
[safari:course] Logging in
ERROR: [safari:course]<PHONE_NUMBER>141: Unable to download JSON metadata: HTTPSConnectionPool(host='www.oreilly.com', port=443): Read timed out. (read timeout=20.0) (caused by TransportError("HTTPSConnectionPool(host='www.oreilly.com', port=443): Read timed out. (read timeout=20.0)"))
  File "/usr/lib/python3.12/site-packages/yt_dlp/extractor/common.py", line 730, in extract
    self.initialize()
  File "/usr/lib/python3.12/site-packages/yt_dlp/extractor/common.py", line 636, in initialize
    self._perform_login(username, password)
  File "/usr/lib/python3.12/site-packages/yt_dlp/extractor/safari.py", line 43, in _perform_login
    auth, urlh = self._download_json_handle(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/yt_dlp/extractor/common.py", line 1092, in download_handle
    res = self._download_webpage_handle(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/yt_dlp/extractor/common.py", line 953, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/yt_dlp/extractor/common.py", line 902, in _request_webpage
    raise ExtractorError(errmsg, cause=err)

  File "/usr/lib/python3.12/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3.12/site-packages/urllib3/connectionpool.py", line 462, in _make_request
    httplib_response = conn.getresponse()
                       ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse
    response.begin()
  File "/usr/lib/python3.12/http/client.py", line 331, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/http/client.py", line 292, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/socket.py", line 708, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/ssl.py", line 1252, in recv_into
    return self.read(nbytes, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/ssl.py", line 1104, in read
    return self._sslobj.read(len, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TimeoutError: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/urllib3/connectionpool.py", line 799, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/urllib3/util/retry.py", line 525, in increment
    raise six.reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/usr/lib/python3.12/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/urllib3/connectionpool.py", line 469, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/lib/python3.12/site-packages/urllib3/connectionpool.py", line 358, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='www.oreilly.com', port=443): Read timed out. (read timeout=20.0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/yt_dlp/networking/_requests.py", line 326, in _send
    requests_res = session.request(
                   ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/requests/adapters.py", line 713, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='www.oreilly.com', port=443): Read timed out. (read timeout=20.0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/yt_dlp/extractor/common.py", line 889, in _request_webpage
    return self._downloader.urlopen(self._create_request(url_or_request, data, headers, query, extensions))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/yt_dlp/YoutubeDL.py", line 4140, in urlopen
    return self._request_director.send(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/yt_dlp/networking/common.py", line 117, in send
    response = handler.send(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/yt_dlp/networking/_helper.py", line 208, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/yt_dlp/networking/common.py", line 337, in send
    return self._send(request)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/yt_dlp/networking/_requests.py", line 350, in _send
    raise TransportError(cause=e) from e
yt_dlp.networking.exceptions.TransportError: HTTPSConnectionPool(host='www.oreilly.com', port=443): Read timed out. (read timeout=20.0)

I see that the https://github.com/sulyi/yt-dlp/blob/oreilly-login-fix/yt_dlp/extractor/safari.py code is using a different set of URLs for auth. Presumably there's some issue about setting cookies if you're getting different behaviour with yt-dl vs yt-dlp


feat(protocol): Accept, Accept-Charset and Content-Encoding protocol enforcement rules

Hi,
I created this patch that contains two new protocol enforcement rules. One of them is aimed at the Accept header, which only blocks any charset parameter that's not explicitly whitelisted. The other blocks any Accept-Charset headers, which can be used to accomplish the same thing. The latter is a forbidden header, clients should not use it and servers should not respect it. Since it's never actually used by browsers of any kind, it should be blocked completely.
The Accept header was a tricky one, considering that PCRE lookarounds are not allowed due to compatibility and performance issues. However, I've found a workaround and created a python script to generate a regex capable of almost perfectly mirroring a negative lookahead. This is much better than the alternatives, considering that a positive match or an inverted regex match could be bypassed by simply including the charset parameter twice:
Accept: text/html;charset=UTF-8;charset=foo

This is now no longer the case, as the new rule will only match invalid charset parameter values, such as foo in this case. Both rules also have their own regression tests. That should guarantee their stability through time. The regexp python script can be found at util/regexp-tricks. It still has limitations and issues that are properly commented inside the script.
Edit: This PR contains fixes for the bypasses reported in FBAZCKVI and 5LT92MNR

This PR should fix the bypass reported in FBAZCKVI.

CRS Bug Bounty PR assessment

Rules affected (list rules): FIXME
Paranoia Level addressed (1, 2, 3, 4, full or explain): FIXME
FTW passes (yes or no) : FIXME
Rule(s) picked for solution (correct or not-correct or explain) : FIXME
Risk for false positives (irrelevant, adequate, substantial or explain) : FIXME
Regular expression quality (inspirational, decent base, needs work, adequate or explain) : FIXME
Documentation (needs work, adequate or explain) : FIXME
Tests (none or some or adequate) : FIXME

Verdict (Unusable, inspirational, usable, almost perfect or perfect) : FIXME

This is not meant to be final. As a CRS dev, feel free to comment below and edit this form directly. As committer or observer, feel free to comment below with feedback and we will think about updating the assessment accordingly.

The @unconditionalMatch operator can be used to match anything and still report the MATCHED_VAR in logdata. This is a lot faster than @rx.

SecRule REQUEST_HEADERS:Content-Encoding "@rx .*?" \
We want to detect 'anything' so the lazy modifier might harm performance.
Also, if I understand correctly you allowlisted only a small number of charsets - there are likely more character sets in wild use and we might need to whitelist them too. For instance in Europe iso-8859-15 is popular and more countries will use their variant. Will this work already or does it require a change?
The tool is really cool and a useful contribution. Thank you.

@dune73, I didn't use the regular expression for capturing the header's value, the content shouldn't matter much to us as there is no legitimate use for it. Also, .*? will always return a zero-width match after 2 steps max, performance really isn't an issue here:

Not sure why I was inconsistent on that between the Accept-Charset and Content-Encoding rules though.
Are we sure @unconditionalMatch is right for this use-case? I spotted it in the ModSecurity docs but I figured that it would always return true, or does the REQUEST_HEADERS:<header> part of the rule correct that?

The performance impact of @rx over other operators is the setup of the regex, which is quite substantial.
The @unconditionalMatch is cool here, since it will always trigger, yet the rule will only execute if the request header in question is actually present.
(I am not implying CRS is doing this correctly elsewhere. But for a new rule, it is worth doing it this way. We assumed you use this over counting the number of occurrences since you want to display the value in logdata. If you feel that is not necessary, then the counter would be fine too.)

@lifeforms, thanks :)

For instance in Europe iso-8859-15 is popular and more countries will use their variant. Will this work already or does it require a change?

That's a not so simple question to answer. The current regex does work for iso-8859-15, but that's actually an unintended side-effect of the not-yet perfect generation script. The reason that it's complex is because of the common 'substrings' (e.g. iso-8859-1 is contained within iso-8859-15), which can cause accidental matching if not handled properly. I've managed to account for some of these cases, hence the experimental notice in the script. It should work fine for one or two strings though.
It's quite difficult to get this behavior right in a generation script, but it's doable manually. I've edited the generated regex so that it fully performs as desired:
https://regex101.com/r/BBpIu2/1 (scroll down to "Unit Tests")
As for the character sets that were included by default. I simply stuck to the charsets that were already allowed within CRS elsewhere (which seems limited to UTF-8 and iso-8859-1).

@dune73, pushed the necessary changes just now. Let me know if that works :)

Thank you @karelorigin. This is clean and to the point how. Yet only if we assume your fix carries the correct architecture. You could say perfect, but only when looking at it from a limited perspective.
Like @lifeforms I am not so sure we should hardcode the charsets into a rule. Yet the regex itself is generated via a script on top. That means it's tweaking is very difficult for users.
Are there other opinions?

Thanks @karelorigin for the work!
I've been thinking about how to avoid the whole look-ahead issue. We've only recently gotten rid of a script that would generate a look-ahead-free expression and I don't want to reintroduce it. At that time we had solved the issue by using a chained rule and the @within operator. So here's the idea (your tests pass this):
SecRule REQUEST_HEADERS:Accept "@rx ;\s*charset\s*=\s*([a-z\d-]+)" \
    "id:920600,\
    phase:1,\
    block,\
    t:none,t:lowercase,\
    msg:'Illegal Accept header: charset parameter %{TX.1}',\
    logdata:'%{TX.0}',\
    capture,\
    tag:'application-multi',\
    tag:'language-multi',\
    tag:'platform-multi',\
    tag:'attack-protocol',\
    tag:'paranoia-level/1',\
    tag:'OWASP_CRS',\
    ver:'OWASP_CRS/4.0.0-rc1',\
    severity:'CRITICAL',\
    setvar:'tx.inbound_anomaly_score_pl1=+%{tx.critical_anomaly_score}',\
    setvar:'tx.foo=/%{tx.1}/',\
    chain"
SecRule TX:foo "!@within /utf-8/ /iso-8859-1/" \
    "t:none,t:lowercase"

First, we extract the (first) charset key-value pair. Then we put / delimiters around the match and then use @within to look for the allowed character sets. It's still not perfect but at least it's readable and doesn't require a complex regular expression. The list of valid encodings could come from a variable btw, @within supports macro expansion. I think the same could be done with @pm or @pmf if we wanted to.
Now, this will not protect against multiple charset, but I would argue that such a header should lead to a block anyway and should get its own rule.
One other thing: you write that you don't think that Accept-Charset should ever be allowed. From looking tat the spec (https://httpwg.org/specs/rfc7231.html#header.accept-charset), it appears to me that we can't really rule out that clients will send the header (CRS could be deployed within an organization that does this, for example). But maybe it's ok to block this header at PL2 or PL3.

The big Q will be what will happen if clients in the wild do submit a "forbidden" header. The Mozilla docs says the server should ignore these headers, but that is different from blocking the request. Nevertheless, I am willing to try it out - running a WAF means you are willing to be more strict than the RFC's, and we could go by what we see in live traffic.
So, in my opinion, we could merge this, and we'll have enough time to test it out on live traffic. If it turns out that one or more of the headers are too common, we could move their check to a higher paranoia level like PL2.
Any other opinions?

Like @lifeforms I am not so sure we should hardcode the charsets into a rule

@dune73, a whitelist is the only approach that makes sense. All major browsers (and any other clients) rely on the standard UTF-8 encoding. I've simply taken a look at what other character encodings CRS generally accepts and these were the only two I could find. Example:
https://github.com/coreruleset/coreruleset/blob/1eb870bddbb784e6709d38e0cc5ae77987bde23e/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf#L1013-L1016
Where the default is types are:
https://github.com/coreruleset/coreruleset/blob/fc827de163a2b41608cb70078e1b5324c5bd35cd/crs-setup.conf.example#L504

Yet the regex itself is generated via a script on top. That means it's tweaking is very difficult for users.

True, but I don't see how that's any different from regexp-assemble.py.

I've been thinking about how to avoid the whole look-ahead issue. We've only recently gotten rid of a script that would generate a look-ahead-free expression and I don't want to reintroduce it.

@theseion, I wasn't aware that a similar script existed at one point. Could you show me where perhaps?
I went with this solution because it was the only one that actually solved the problem. As you already know, any alternatives can be bypassed by specifying the charset more than once. The solution you suggested definitely came to mind.
I don't really see the point in adding a rule that is known to be broken and has a publicly available bypass (this conversation discloses that part).

Now, this will not protect against multiple charset, but I would argue that such a header should lead to a block anyway and should get its own rule.

Could you explain to me how another rule would solve the multiple charset problem? Any positive/inverse matches will return on the first (non-)match.

The Mozilla docs says the server should ignore these headers, but that is different from blocking the request.

@lifeforms, very true. I think the docs were misleading because I interpreted the big red warning plus the "Forbidden header name" as not allowed in requests, even though the "forbidden" part means that only user agents should control them.
For what it's worth, any major browsers have stopped sending it for at least 9 years:

https://hsivonen.fi/accept-charset/
https://src.chromium.org/viewvc/chrome?revision=188237&view=revision

Nevertheless, I am willing to try it out - running a WAF means you are willing to be more strict than the RFC's, and we could go by what we see in live traffic.

Agreed! CRS already doesn't allow many character encodings out of the box (without these rules). Following RFCs to the letter, in a security landscape, is a bit too idealistic.

@karelorigin Link to the script in the commit before I removed it: https://github.com/coreruleset/coreruleset/blob/98590c959a477f58dc7c4cc090ca41118ba6a491/util/regexp-assemble/lib/negativelookbehind.py.
To find out whether there's more than one charset you could do the following:

match the first charset and create a match group for the rest of the string, e.g. .*charset=([\da-z-]+);)(.*)
in a second rule, search the second match group for the occurrence of charset with the same value as the first match group

We've done something similar for SQL tautologies: https://github.com/coreruleset/coreruleset/blob/1eb870bddbb784e6709d38e0cc5ae77987bde23e/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf#L597.

@theseion, I'm afraid the solution you suggested won't work. The problem is finding any charset parameters that have illegal values, not blocking requests that have more than one charset parameter present. This is a perfectly legitimate Accept header value and should not be blocked:
text/html;charset=iso-8859-1;q=0.8,text/html;charset=utf-8;q=0.9

My regex will not match, as intended. That doesn't appear to be the case for the suggested implementation.

@karelorigin You're right... I'm currently wrapping my head around the issue and trying to come up with a different solution. I really want to avoid having to use any look-around hacks.

I think I have a working solution. The idea is to match the entire header value and expect it to be valid, then block when no match is found, using inversion (!). The following regular expression matches the value of the Accept header per RFC (as far as necessary). It looks for charset=<valid charsets> or any other valid parameter value that is not charset:
^(?:(?:\*|[^\"(),/:;<=>?![\x5c\]{}]+)/(?:\*|[^\"(),/:;<=>?![\x5c\]{}]+))(?:\s*;\s*(?:(?:charset\s*=\s*(?:iso-8859-1|utf-8)\b)|(?:[^c\"(),/:;<=>?![\x5c\]{}][^\"(),/:;<=>?![\x5c\]{}]*|c[^h\"(),/:;<=>?![\x5c\]{}][^\"(),/:;<=>?![\x5c\]{}]*|ch[^a\"(),/:;<=>?![\x5c\]{}][^\"(),/:;<=>?![\x5c\]{}]*|cha[^r\"(),/:;<=>?![\x5c\]{}]*|char[^s\"(),/:;<=>?![\x5c\]{}][^\"(),/:;<=>?![\x5c\]{}]*|chars[^e\"(),/:;<=>?![\x5c\]{}][^\"(),/:;<=>?![\x5c\]{}]*|charse[^t\"(),/:;<=>?![\x5c\]{}][^\"(),/:;<=>?![\x5c\]{}]*)\s*=\s*[^(),/:;<=>?![\x5c\]{}]+))*(?:\s*,\s*(?:(?:\*|[^\"(),/:;<=>?![\x5c\]{}]+)/(?:\*|[^\"(),/:;<=>?![\x5c\]{}]+))(?:\s*;\s*(?:(?:charset\s*=\s*(?:iso-8859-1|utf-8)\b)|(?:[^c\"(),/:;<=>?![\x5c\]{}][^\"(),/:;<=>?![\x5c\]{}]*|c[^h\"(),/:;<=>?![\x5c\]{}][^\"(),/:;<=>?![\x5c\]{}]*|ch[^a\"(),/:;<=>?![\x5c\]{}][^\"(),/:;<=>?![\x5c\]{}]*|cha[^r\"(),/:;<=>?![\x5c\]{}]*|char[^s\"(),/:;<=>?![\x5c\]{}][^\"(),/:;<=>?![\x5c\]{}]*|chars[^e\"(),/:;<=>?![\x5c\]{}][^\"(),/:;<=>?![\x5c\]{}]*|charse[^t\"(),/:;<=>?![\x5c\]{}][^\"(),/:;<=>?![\x5c\]{}]*)\s*=\s*[^(),/:;<=>?![\x5c\]{}]+))*)*$

The rule would look as follows (with proper backslash and double quote escaping):
SecRule REQUEST_HEADERS:Accept "!@rx ^(?:(?:\*|[^\"(),/:;<=>?![\x5c\]{}]+)/(?:\*|[^\"(),/:;<=>?![\x5c\]{}]+))(?:\s*;\s*(?:(?:charset\s*=\s*(?:iso-8859-1|utf-8)\b)|(?:[^c\"(),/:;<=>?![\x5c\]{}][^\"(),/:;<=>?![\x5c\]{}]*|c[^h\"(),/:;<=>?![\x5c\]{}][^\"(),/:;<=>?![\x5c\]{}]*|ch[^a\"(),/:;<=>?![\x5c\]{}][^\"(),/:;<=>?![\x5c\]{}]*|cha[^r\"(),/:;<=>?![\x5c\]{}]*|char[^s\"(),/:;<=>?![\x5c\]{}][^\"(),/:;<=>?![\x5c\]{}]*|chars[^e\"(),/:;<=>?![\x5c\]{}][^\"(),/:;<=>?![\x5c\]{}]*|charse[^t\"(),/:;<=>?![\x5c\]{}][^\"(),/:;<=>?![\x5c\]{}]*)\s*=\s*[^(),/:;<=>?![\x5c\]{}]+))*(?:\s*,\s*(?:(?:\*|[^\"(),/:;<=>?![\x5c\]{}]+)/(?:\*|[^\"(),/:;<=>?![\x5c\]{}]+))(?:\s*;\s*(?:(?:charset\s*=\s*(?:iso-8859-1|utf-8)\b)|(?:[^c\"(),/:;<=>?![\x5c\]{}][^\"(),/:;<=>?![\x5c\]{}]*|c[^h\"(),/:;<=>?![\x5c\]{}][^\"(),/:;<=>?![\x5c\]{}]*|ch[^a\"(),/:;<=>?![\x5c\]{}][^\"(),/:;<=>?![\x5c\]{}]*|cha[^r\"(),/:;<=>?![\x5c\]{}]*|char[^s\"(),/:;<=>?![\x5c\]{}][^\"(),/:;<=>?![\x5c\]{}]*|chars[^e\"(),/:;<=>?![\x5c\]{}][^\"(),/:;<=>?![\x5c\]{}]*|charse[^t\"(),/:;<=>?![\x5c\]{}][^\"(),/:;<=>?![\x5c\]{}]*)\s*=\s*[^(),/:;<=>?![\x5c\]{}]+))*)*$" \
    "id:920600,\
    phase:1,\
    block,\
    t:none,t:lowercase,\
    msg:'Illegal Accept header: charset parameter %{tx.0}',\
    capture,\
    logdata:'%{MATCHED_VAR}',\
    tag:'application-multi',\
    tag:'language-multi',\
    tag:'platform-multi',\
    tag:'attack-protocol',\
    tag:'paranoia-level/1',\
    tag:'OWASP_CRS',\
    ver:'OWASP_CRS/4.0.0-rc1',\
    severity:'CRITICAL',\
    setvar:'tx.inbound_anomaly_score_pl1=+%{tx.critical_anomaly_score}'"

The expression isn't readily DoS-able. The only thing you don't get in this case is the actual value of the invalid character set, but I think that's acceptable.
With this approach we can also have a single block of valid encodings in a data file. The only thing we can't do is use macro expansion, because that doesn't work properly, so using a variable for tuning won't be an option.
What do you think @karelorigin?

@theseion, that's uhh quite the regex. This solution goes far beyond the scope of finding invalid charset values, which makes it more prone to bypasses. This regex101 is one such example.
The only upside I can think of is that this should be easier to maintain because of the data file part. However, besides that, it's a lot more confusing and a lot less performant. The new proposed regex takes 51454 steps to determine that the following Accept header value is invalid:
text/html; charset=utf-8;q=0,text/html; charset=utf-8;q=0,text/html; charset=utf-8;q=0,text/html; charset=utf-8;q=0,text/html;charset=utf-x;q=0

while this one does it in 223.

You're right, it is more complicated, although only as long as you don't account for the complexity of the script you wrote.
Thanks for the bypass hint. I was able to fix that by using a possessive quantifier on the optional white space matches (\s*+). Now the performance is much better (100 and ~700 steps). Your regex still easily outperforms it of course.
I don't want to dismiss your work but I also don't want to be the one to finish the script and make sure it properly works with more than 2 inputs (I've checked, it doesn't). I can also see, that we might run into similar situations again, and having your script would be useful, because it might not be as simple as it was this time, to write a fully matching regular expression. However, my solution would work out of the box with the tooling we have.
If you are up to finishing the script (substring overlaps, > 2 inputs) and provide a suite of tests to cover it I will integrated it into regexp-assemble and we can use your solution. I would like that. Otherwise my advice for this PR is to use my solution (with the possessive quantifiers, of course).
Rationale for using the script to convert negative-lookahead into non-lookahead syntax:

more flexibility in the future
expressions generated with the script will be less prone to bypasses than expressions trying to match fully
writing a fully matching regular expression isn't trivial, not every contributor will be able to do it
verifying a fully matching regular expression isn't trivial, it is likely that we will miss possible bypasses

Wow, this is way above my regexp knowledge, thanks to all for contributing...
@karelorigin What do you think about Max' proposal of improving the script and using it?

He @theseion / @karelorigin, following theseion's request to review this, I sat down and read up on this PR again. As for @lifeforms, this is intellectually challenging.
Here is what I understand - please tell me where I am wrong.

We can do look-around regular expression and everything is cool but we do not want that for known reasons.
@karelorigin contributed a proof-of-concept script that comes up with a speedy regular expression that does away with the need for look-around. The script needs work, though.
@theseion created a very complex regular expression that does the job as well, but is less performant.

What I fail to see is the difference in the regular expressions. Could you put them side by side? I am not sure I look at the right ones.
Having a script that allows us to simplify look-around constructs into non-look-around constructs would be desirable, but it has to be maintainable obviously. The work @theseion has done with the regex-assembly is awesome, but I also feel like he is the only one that really, really understands what is happening. It's still OK, but we need to keep an eye on it.
And finally: This is Paranoia-Level 1, if we run an anchored allow-list on known header-fields on the Accept-Header we make future changes to this header impossible. The HTTP protocol guys pointed out quite clearly that this may lead to the creation of additional headers since the existing ones can no longer be expanded.
Thank you for all your good work. I love the level of scrutiny that our projects gets, it's just hard for me - and others - to keep up amidst all the other stuff going on.

@theseion,
Sorry for the delayed response. This is a complicated discussion and I couldn't find the right moment for me to get my story straight.

I don't want to dismiss your work but I also don't want to be the one to finish the script and make sure it properly works with more than 2 inputs (I've checked, it doesn't)

This is correct, the script itself also contains a comment referencing the (then potential, now confirmed) problem. I think I know what the root cause might be though and should be able to fix it.

I can also see, that we might run into similar situations again, and having your script would be useful, because it might not be as simple as it was this time, to write a fully matching regular expression.

In a lot of cases it will be outright impossible. https://github.com/coreruleset/coreruleset/pull/2666 proves this.

However, my solution would work out of the box with the tooling we have.

If you're still very much against chaining regexp-assemble with external scripts, we might be able to find a middle ground by either extending the regexp-assemble wrapper or perl module. This will require some pretty intensive changes, but might be necessary to reliably solve cases such as https://github.com/coreruleset/coreruleset/pull/2666. It would surely be the ideal experience from a CRS contributor point of view. Perl is honestly outside of my current skill set though.
It gets even more complicated if we want to negative lookahead patterns instead of just literal strings. e.g. bob and bab vs b[oa]b

Rationale for using the script to convert negative-lookahead into non-lookahead syntax:

more flexibility in the future
expressions generated with the script will be less prone to bypasses than expressions trying to match fully
writing a fully matching regular expression isn't trivial, not every contributor will be able to do it
verifying a fully matching regular expression isn't trivial, it is likely that we will miss possible bypasses

Agree with all of these points. In the end, I think it's either dropping the PCRE restrictions or finding a way around it like we are doing right now.

@lifeforms,

What do you think about Max' proposal of improving the script and using it?

If we were to use @theseion's solution, I would recommend at least finding out whether it can be simplified or not. It will be difficult to maintain in its current state.
Also can't help but feel like it would be an attempt to avoid the inevitable. This solution only works because we're able to somewhat emulate what a header parser would do (assuming it follows the RFC). In a scenario where the surrounding context is unknown, such as the PR I linked in my previous comment, we'll be out of luck.

@dune73,

@karelorigin contributed a proof-of-concept script that comes up with a speedy regular expression that does away with the need for look-around. The script needs work, though.

Correct!

@theseion created a very complex regular expression that does the job as well, but is less performant.

Also correct. Though in @theseion's defence, my version might (not sure if or by how much) see a performance decrease when adding additional charsets to the negative lookahead. @theseion's version will probably be more stable in that regard. Could be trivial when doing regexp-assemble-like optimizations, however.

What I fail to see is the difference in the regular expressions. Could you put them side by side? I am not sure I look at the right ones.

Sure! I've actually linked to both solutions in a previous comment, but I'll list them here again.
You can find mine here:
regex101
And @theseion's here:
regex101
The difference is that @theseion's version attempts to validate the entire Accept header value by running it against an RFC-following regex that matches any inputs containing exotic charsets. This also explains its complexity. That definitely makes things difficult false positive/negative and maintenance wise.

And finally: This is Paranoia-Level 1, if we run an anchored allow-list on known header-fields on the Accept-Header we make future changes to this header impossible. The HTTP protocol guys pointed out quite clearly that this may lead to the creation of additional headers since the existing ones can no longer be expanded.

Anchored in what sense? That they're not configurable without commits? Unfortunately, this seems to be the case regardless of how we solve the issue. What we can do is try to find a good way to improve the CRS contributor experience.
In https://github.com/coreruleset/coreruleset/pull/2591#issuecomment-1188277759 I suggested going as far as making lookahead emulation regexp-assemble native functionality. This will require quite a bit of work on our side though but might make any additional cases (https://github.com/coreruleset/coreruleset/pull/2666) possible to solve, which right now we cannot.

Thank you for all your good work. I love the level of scrutiny that our projects gets, it's just hard for me - and others - to keep up amidst all the other stuff going on.

I understand, and no worries, I don't have any issues answering questions :)

Thank you @karelorigin. Your explanation are really helpful.
With anchored, I mean an anchored regular expression. One that defines the value of the Accept-Header from the start to the end. An allowlist or whitelist of that header if you will.
If we introduce this and it's being deployed with CRS across the globe, any experimental changes to the HTTP protocol on the Accept header will get a forbidden on a certain percentage of requests. This is what is called "protocol ossification". You can not longer change certain parts of the protocol (-> the Accept header). This will force people to add new headers for functionality that could in fact be added via the Accept-Header.
So while we try to improve security, we make HTTP more complicated for everybody else (and force people to burn more CO2 from here going forward).

So while we try to improve security, we make HTTP more complicated for everybody else (and force people to burn more CO2 from here going forward).

That's a good point and one more reason to use the script instead of my "parser emulation". As @karelorigin says, it will take some work to getting this into regexp-assemble and I'd rather have this PR closed than wait for that moment. We have so much work to do currently that we can't run this as an additional project at the moment. That's why I was hoping (😇) that @karel would be able to contribute the script in a polished state. Otherwise I suggest that we use my regex for closing this PR and then replace it once we have the script integrated.

Thank you @theseion. I got it and I understand your position. Glad to hear your estimation with regards to regex-assembly integration.
@karelorigin could you give your estimate in this regard: How much work do you think polishing your script is? Do you see yourself performing that? How would you integrate into our regex assembly?
I'm adding this to the agenda tonight, it's one of the bigger open questions right now.

The lowercase transformation solves the case part, but the optional quoting should be there I think.

Yes, absolutely.

Guys, can you get a fix for across the line ASAP? Ideally before the meeting tomorrow. It really is one of our biggest showstoppers right now.
Here is what we decided in August:

Need to make a tough decision about the direction we are taking here. We have 2 options to deal with:

    Max has created an admittedly hard and probably relatively slow regular expression. A manual regex.
    Karel has created a PoC script that writes an optimized regular expression. But the script is not ready for prime time. Integrating it into the regex-assembly would take some really hard work. And we're a bit worn down on that front.
    -> We agree on starting with Max' (fixed) regexp for now, and revisit the issue when the script by Karel is going places.

Working on it.

I've opened #2766. It's still in draft as it relies on two fixes to regexp-assemble: #2764 and #2765.

@karelorigin I'm closing this PR. It has been replaced with #2766. The negative-lookahead script is also in that PR, so you can continue working on it and provide PRs to this repository.
Unfortunately, I wasn't able to confirm all of this with you but we had to move quickly due to the release we are planning.


Add some icons and replace others

Referencing stuff in icons which would be deleted by the other push.
Close, please rework and repush


Bluetooth: ipsp: Fix not checking return of build_reply

net_pkt_sendto uses size_t as parameter for len so the value would be
treat as unsigned which may cause and invalid memory to be read.
Fixes #14950
Fixes #14955
Signed-off-by: Luiz Augusto von Dentz<EMAIL_ADDRESS>

@Vudentz is there not a matching GitHub issue for this? AFAIK one was created for all Coverity issues. Please reference it so that it gets automatically closed once this PR is merged.

@Vudentz is there not a matching GitHub issue for this? AFAIK one was created for all Coverity issues. Please reference it so that it gets automatically closed once this PR is merged.

I probably miss that then, will take a look.

@jhedberg updated with issue number.

@jhedberg updated with issue number.

@Vudentz thanks. There also seems to be a similar issue with tcp_received (issue #14955). Are you planning to submit a separate PR for that? I guess it could be fixed in the context of this PR as well.

@jhedberg updated with issue number.

@Vudentz thanks. There also seems to be a similar issue with tcp_received (issue #14955). Are you planning to submit a separate PR for that? I guess it could be fixed in the context of this PR as well.

Right, I will fix that as well.

Codecov Report

Merging #15004 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #15004   +/-   ##
=======================================
  Coverage   52.92%   52.92%
=======================================
  Files         309      309
  Lines       45268    45268
  Branches    10451    10451
=======================================
  Hits        23956    23956
  Misses      16544    16544
  Partials     4768     4768

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9e81cbe...9012799. Read the comment docs.


Bump alpine-lima example from Alpine 3.14 to 3.15

Signed-off-by: Jan Dubois<EMAIL_ADDRESS>

Thanks, but covered in https://github.com/lima-vm/lima/pull/811


Watch pane does not use evaluateName when executing "Copy Value"

I've had a bug raised where values copied (using Copy Value) from the Watch pane are truncated. I investigated, and it seems like there is a difference in behaviour between the Variables and Watch panes. I'm examining an array with a single long string in it:

In both the scopesRequest and the evaluateRequest I return a variablesReference for the array. In the variablesRequest they both return the same data, which is the following (an array of Variable):
[
	{
		"evaluateName": "longStrings[0]",
		"indexedVariables": null,
		"name": "[0]",
		"type": "String (_OneByteString)",
		"value": "\"This is a long string that is 300 characters! This is a long string that is 300 characters! This is a long string that is 300 ch…\"",
		"variablesReference": 0
	}
]

This has a truncated string for display, and an evaluateName that will give the full string. However it seems that when I choose Copy Value from the Variables window the expression gets evaluated, but choosing Copy Value from the Watch window just copies the truncated value into the clipboard.

(@isidorn does this seem like a bug, or are they intended to work differently?)

You are right, this is an issue and I can push a fix for it by passing variable instead of variable.value here
However this seems to uncover another bug from node-debug. Which is that the evaluateName for variables in watch window are bogus. @weinand @roblourens can you please investigate? Or if you prefer i can push a fix for this issue which will uncover the other issue

I pushed a fix for the evaluateName issue.


Update akka-actor to 2.5.26

Updates com.typesafe.akka:akka-actor from 2.5.25 to 2.5.26.
Release Notes/Changelog
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.typesafe.akka", artifactId = "akka-actor" } ]

labels: semver-patch

HI @scala-steward it looks like the links point to a fork of akka, not to the akka project itself.

That is because the official akka POMs point to that URL: https://github.com/fthomas/scala-steward/issues/1048#issuecomment-543486737 :man_shrugging:


Simplify ATPOutputView close method

The detach() and lastOpenedView = null should occur regardless of the
outcome of the conditional. It is cleaner to just have these lines
follow the after the if statement.
Coincidentally, this fixes isis97/atom-terminal-panel#57 which allowed
multiple panels to be open at the same time.

@isis97 ping


Only allow docker-buildx plugin >= 5.0.0 to be priviledged by default

bump buildx plugin to only be privileged if version is pinned and greater than v5.0.0 or latest.
https://codeberg.org/woodpecker-plugins/docker-buildx/releases/tag/v5.0.0
so by default only this images will get privildges:

woodpeckerci/plugin-docker-buildx (latest)
woodpeckerci/plugin-docker-buildx:latest
woodpeckerci/plugin-docker-buildx:5
woodpeckerci/plugin-docker-buildx:5.0
woodpeckerci/plugin-docker-buildx:5.0.0

well that's the as much non breaking as posible way
we don't have a better way to make users secure by default!!!
and if it breaks they where insecure and can decide to either stay so or fix it!

and to "FIX" it ... you can just read the old images you use ... but then it's up to you to secure it - I'll write that as heading into the release note

@qwerty287  see https://github.com/woodpecker-ci/woodpecker/pull/4214


feat: create text file in project root repository when plan returned changes

This allows to add logic in the digger workflow based on if the plan returned changes or not.
For instance, to run checkov only when the plan returned changes:
workflows:
  default:
    plan:
      steps:
        - init
        - plan
        - run: if [ -f "isNonEmptyPlan.txt" ]; then checkov --directory .; fi

The -no-color plan default argument was also added in this PR to fix a bug in cleanupTerraformOutput() function when nonEmptyOutput is false. The following logic would not work because of color characters around the No changes substring:
	} else if nonEmptyOutput {
		start = "Terraform will perform the following actions:"
	} else {
		start = "No changes. Your infrastructure matches the configuration."
	}

why can't we check the plan file itself?

Do you mean the content of the plan file? It's binary, and the content by itself represents some infrastructure configuration, but not the diff vs the current infrastructure state.
Also in the digger cmds, seems like we cannot use pipes with multiple commands.
How would you do it?

that's true my bad, for some reason I thought it's gonna be actualy empty file.
We have now enabled -detail-exitcode by default. Will it help with your original suggestion?

Realised after looking at the code that isNonEmptyPlan variable is what was really needed, which is why I based my proposal on it.
From the workflow itself, I cannot read that the plan command returned 2, and I don't think there is an alternative to either my proposal or setting an env variable. Using an env variable would be a bit more complicated since it would need to be reset between each project in a mono-repo.
Maybe digger could add a run_when_plan_changes step that would rely on this file to execute the cmd instead of using bash like my example
workflows:
  default:
    plan:
      steps:
        - init
        - plan
        - run_when_plan_changes:  checkov --directory .


Add support for flow, deployment and deployment access resources.

Added support for multiple resources:

Flow
Deployment
DeploymentAccess

This PR adds basic support for these resources but does not full implement all attributes for deployments currently.
If approved the additional support will be implemented.
All acceptance tests are passing.
Test output:
TBD

Hi. I'll modify the deployment access test to use a service account and that should allow the test to pass.
Can you confirm these tests run in a pro account with ability to create workspaces?

@parkedwards I have engineered these tests to create a workspace and then delete the workspace. This is to ensure the tests work for anyone and does not require the "evergreen" workspace used in the other tests. Can you please confirm that the the workspace can be created. It looks like there is an issue there somewhere.
I can run this test in my environment using a generic api key or a service account api key with admin access.

Hi @skyscrapr, appreciate the contribution here. Would you be willing to split this PR up so each one adds one resource? As you can probably tell from our merged PRs, we try to keep them as small as possible to make peer reviews more effective. Let us know if you have any questions.

closing this PR and splitting as requested in new PRs


Issue token callback does not find provider with overwritten clientId or clientSecret values

When a provider config is overwritten (config/patron.php):
<?php
use craft\helpers\App;

return [
    'providers' => [
        'salesforce' => [
            'clientId' => App::env("SALESFORCE_CLIENT_ID"),
            'clientSecret' => App::env("SALESFORCE_CLIENT_SECRET")
        ]
    ]
];

When issuing a token, the overwrite value are not properly evaluated when the callback is handled and therefore a token is not saved.

Fixed w/ version https://github.com/flipboxfactory/patron/releases/tag/2.4.0


Compilation problems

I am working on a react app created with create-react-app.
I am using nats.ws with the following import as mentioned in README:
import { connect } from 'nats.ws/nats.js'.
In a method I am calling:
const nc = await connect({servers: 'nats:4442'});
The compilation process gives me the following fail:
Failed to compile.

./node_modules/nats.ws/nats.js 132:9
Module parse failed: Unexpected token (132:9)
File was processed with these loaders:
 * ./node_modules/babel-loader/lib/index.js
You may need an additional loader to handle the result of these loaders.
|
| class DataBuffer1 {
>   buffers;
|   byteLength;
|

Tools versions I am using are:
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>How can I solve this issue?
Thank you in advance

@alessandroperetti try this https://github.com/nats-io/nats.ws/blob/master/developer_notes.md#working-with-react-create-app