How should we store installed cask information?

Verification

[X] This issue's title and/or description do not reference a single formula e.g. brew install wget. If they do, open an issue at https://github.com/Homebrew/homebrew-core/issues/new/choose instead.

Provide a detailed description of the proposed feature
When you install a package from Homebrew, we store the package definition along with the installed package. This allows us to always be able to load the package again in the future even if it gets removed from a tap. It also means that any uninstall information is more accurate since files can get moved around between versions.
A year and a half ago when we went all in on the API with Homebrew v4.0, we moved core casks to use the API. When you install a cask without the API, it will store the Ruby cask file along with the cask itself as metadata. When you install cask with the API, it will store the JSON cask file which represents the Ruby cask file 99% of the time.
If a flight or language block is present, we download the original Ruby cask file during the installation process and use that instead.

https://github.com/Homebrew/brew/pull/14304
https://github.com/Homebrew/brew/pull/14456

That means that there are two possible cask files that could be stored along with an installed cask. This means that we there is more cask loading code that we have to support along with exceptions like those for casks with certain blocks defined. We should try to simplify that if possible.

For reference, we always store the Ruby formula file whether we use the API or not. If we're not using the API, the Ruby formula file gets stored along with the installed formula. If we're using the API, we download a bottle which contains the Ruby formula file on install. This means that we don't need to include install and uninstall info in the formula JSON for the API since we always use the Ruby formula file for that instead.

Current Ideas

Only store the Ruby cask file

This would make things much simpler on our end but would require an extra network request to fetch the Ruby cask file every time a new cask gets installed. We already do this for casks with flight or language blocks.

Only store the JSON cask file

This would require figuring out how to serialize the fight and language blocks in the cask DSL. That's why we currently download the Ruby cask file before installing casks with those blocks. They cannot be expressed in the current iteration of the cask API JSON.

Create a new format to represent an installed cask

This might actually end up being more code to support but would be more consistent because we'd be able to specify the minimum amount of information necessary to work with an installed cask.
What is the motivation for the feature?
Having one file format will mean that the codebase is more consistent. It will also make it easier for us to maintain compatibility with older casks going forward.
Me and @MikeMcQuaid have been going back and forth on this for a week and I think it'd be helpful to get more voices in this discussion. There might be some simple solution that we've missed. We'd like to get this sorted out before releasing the internal JSON API v3.

https://github.com/Homebrew/brew/pull/16896#discussion_r1527590673
https://github.com/Homebrew/brew/pull/16896#discussion_r1527590946

How will the feature be relevant to at least 90% of Homebrew users?
It's relevant to anyone who installs casks as it could affect old cask file loading reliability.
What alternatives to the feature have been considered?
Keeping things the way they are where we support both the JSON and Ruby cask file formats.

@Homebrew/cask It'd be great to hear your opinion on this topic.

Thanks for opening this @apainintheneck! ❤

Only store the Ruby cask file

This would make things much simpler on our end but would require an extra network request to fetch the Ruby cask file every time a new cask gets installed. We already do this for casks with flight or language blocks.

This feels like we lose a lot of benefits of the API and make offline installation impossible.

Only store the JSON cask file

This would require figuring out how to serialize the fight and language blocks in the cask DSL. That's why we currently download the Ruby cask file before installing casks with those blocks. They cannot be expressed in the current iteration of the cask API JSON.

This feels like the best option to me and worth tackling in JSON v3.

Create a new format to represent an installed cask

This might actually end up being more code to support but would be more consistent because we'd be able to specify the minimum amount of information necessary to work with an installed cask.

If this format is a e.g. superset/subset of JSON v3 this seems desirable. Otherwise, it feels like it will have all of the disadvantages of one or both of the above.

Only store the Ruby cask file

This would make things much simpler on our end but would require an extra network request to fetch the Ruby cask file every time a new cask gets installed. We already do this for casks with flight or language blocks.

This feels like we lose a lot of benefits of the API and make offline installation impossible.

The only downsides I know of are the extra network request and the fact that the tap info is not included in the Ruby cask file.

Only store the JSON cask file

This would require figuring out how to serialize the fight and language blocks in the cask DSL. That's why we currently download the Ruby cask file before installing casks with those blocks. They cannot be expressed in the current iteration of the cask API JSON.

This feels like the best option to me and worth tackling in JSON v3.

@Rylan12 worked a bit on this over a year ago before we decided to to download the source files for casks with flight or language blocks. The approach at the time was to store the flight blocks as strings in the API JSON and then evaluate that code when loading the cask.

Loader code: https://github.com/Homebrew/brew/pull/14304

Changed in final commit since we decided not to use that approach

Removed in: https://github.com/Homebrew/brew/pull/14611

We never really attempted to load the language tabs from the API. I'll have to look into that a bit more. I think the firefox cask is the most complex example.

The approach at the time was to store the flight blocks as strings in the API JSON and then evaluate that code when loading the cask.

Yeh. I don't love it but: don't really see any alternatives, sadly.

We never really attempted to load the language tabs from the API. I'll have to look into that a bit more. I think the firefox cask is the most complex example.

Thanks ❤

We currently store the tap when installing from the API since the JSON cask already includes that information. We were also discussing whether or not it should be included when installing with internal JSON v3.

Note: I think the JSON should be structured such that we're able to use the same schema (or some sort of schema inheritance/inclusion if that's possible?) validation eventually for the public API schema, internal API schema and JSON that's written to disk.

So the method_source gem only stores the code for the block/proc itself but not any surrounding variables that it closes over. This should be fine for internal use with the main cask repo but could cause problems for third-party taps. We could try some workarounds where we capture all local variables when defining the proc with a custom proc class but that would up the difficulty dramatically.
There have been some gems that have tried to solve this sort of problem in the past but it's not obvious to me that any of them were ever production-ready and all of them have been abandoned for about a decade. We could potentially use them for inspiration but I don't think it's worth it.

https://github.com/ngty/serializable_proc
https://github.com/joker1007/proc_to_ast
https://github.com/ngty/sourcify

Good investigation, thanks @apainintheneck!

This should be fine for internal use with the main cask repo but could cause problems for third-party taps.

We should optimise for our own use cases here and we can fix up issues with 3rd party taps and e.g. provide a longer deprecation cycle.

This should be fine for internal use with the main cask repo but could cause problems for third-party taps.

We should optimise for our own use cases here and we can fix up issues with 3rd party taps and e.g. provide a longer deprecation cycle.

I don't think this problem will just go away if we tell people not to do it and have a longer deprecation cycle. Inevitably it will be misused because valid Ruby syntax just cannot be described in JSON assuming we don't want to get rid of the Ruby DSL entirely. It also means that subtle errors could be introduced that wouldn't be clear until trying to load an installed cask from JSON.
It seems like a mistake to choose a flawed option which is trying to serialize and deserialize arbitrary blocks of Ruby code as strings in JSON when the option of just evaluating Ruby code like we have been for the life of the Homebrew project is on the table. Doing this research has reminded me why I was against the Ruby strings in JSON approach from the beginning.

I don't think this problem will just go away if we tell people not to do it and have a longer deprecation cycle. Inevitably it will be misused because valid Ruby syntax just cannot be described in JSON assuming we don't want to get rid of the Ruby DSL entirely. It also means that subtle errors could be introduced that wouldn't be clear until trying to load an installed cask from JSON.

This is something we can audit for, adjust the way we evaluate the DSL (i.e. what scope we use and what variables are captured), warn/deprecate/error for at installation time, etc.

deserialize arbitrary blocks of Ruby code as strings in JSON when the option of just evaluating Ruby code like we have been for the life of the Homebrew project is on the table.

The logical conclusion of this argument is we shouldn't have used the JSON API for casks at all.
Here's the requirements we have here:

whatever we introduce: it must be the same data written to the Caskroom whether installed from API or tap and whether a core or third-party cask
it must be possible to do offline cask installation such that brew fetch <cask> and then brew install <cask> works with no internet connection
if we're writing JSON to disk: it should be a subset of a public (or what will eventually aim to become public) JSON API
we have had many problems with deprecating DSLs in casks causing issues on uninstall. whatever the solution is here: it should not make the problem worse and should ideally make it better.

and, as it relates to the API:

any internal v3 API should be designed to be similar to and evolve into a public v3 API
human readability trumps uncompressed or compressed file size
a JSON schema should be eventually possible

Hopefully that helps!

This is something we can audit for, adjust the way we evaluate the DSL (i.e. what scope we use and what variables are captured), warn/deprecate/error for at installation time, etc.

I didn't know that it was possible to reduce the scope of closures in Ruby. Can you point me to some references on that because I've been unable to find anything with a quick search?

The logical conclusion of this argument is we shouldn't have used the JSON API for casks at all.

There were problems initially for this very reason but they got ironed out with time. What I'm saying is that we can't generalize what we've made work reasonably well with the core cask repos to third-party repos.

Here's the requirements we have here:

whatever we introduce: it must be the same data written to the Caskroom whether installed from API or tap and whether a core or third-party cask

I disagree. It's not obvious to me that this requirement will result in a more maintainable codebase or a better experience for users.

it must be possible to do offline cask installation such that brew fetch <cask> and then brew install <cask> works with no internet connection

Good point, I hadn't thought of that but it seems ideal and achievable.

if we're writing JSON to disk: it should be a subset of a public (or what will eventually aim to become public) JSON API

This seems ideal but there might be a few fields in the internal JSON that are really no useful or relevant for the publish JSON. I agree in general though.

we have had many problems with deprecating DSLs in casks causing issues on uninstall. whatever the solution is here: it should not make the problem worse and should ideally make it better.

Could you point me to some threads or information about this problem? It's not obvious to me how pervasive this is or when it was most acute.

and, as it relates to the API:

any internal v3 API should be designed to be similar to and evolve into a public v3 API

👍

human readability trumps uncompressed or compressed file size

50/50 on this one.

a JSON schema should be eventually possible

I think the question is how painful would it be to have a JSON schema for all casks and will it be worth that effort.

Ruby within JSON sucks and there's a reason we have already tried that before and declared it not a good idea.
With formulae, there's already a two-level system:

JSON API is used where possible
If postinstall_defined is declared then it will load the Ruby file from the bottle.

Formulae store the Ruby file always in the install directory (though not usually necessary to use given there's no postuninstall).
With casks, it's actually quite similar except it's two downloads rather than one since we don't repackage:

JSON API is used where possible
If a postflight is declared then it will download and use the Ruby file

The reason casks don't always store the Ruby file because we don't always fetch the Ruby file.
However, in the end we will always support installing from third-party taps and they use Ruby files, so we will always support both JSON and API install data. Unless the plan was to generate JSON on-the-fly during install, but that will be extra code rather than simplifying.
Ruby files are downloaded when postflights are used, so we should already know when a Ruby file is needed and when it isn't for uninstall.

I didn't know that it was possible to reduce the scope of closures in Ruby.

Blocks can be evaluated with instance_eval. You can decide which object you evaluate it on.

What I'm saying is that we can't generalize what we've made work reasonably well with the core cask repos to third-party repos.

We don't need to. These repos always install from Cask Ruby files in taps rather than a JSON API. The way these repos do things can be audited, deprecated, disabled and removed.

Here's the requirements we have here:

whatever we introduce: it must be the same data written to the Caskroom whether installed from API or tap and whether a core or third-party cask

I disagree. It's not obvious to me that this requirement will result in a more maintainable codebase or a better experience for users.
...

human readability trumps uncompressed or compressed file size

50/50 on this one.

I am the technical lead on this project (https://docs.brew.sh/Homebrew-Governance#6-project-leader) and I am stating that these are a hard requirements. I have explained my reasoning here enough times. Let's continue discussion on other points, please.

Could you point me to some threads or information about this problem? It's not obvious to me how pervasive this is or when it was most acute.

Please use GitHub's search to find these or just trust me.

a JSON schema should be eventually possible

I think the question is how painful would it be to have a JSON schema for all casks and will it be worth that effort.

You do not have to implement this schema. Others have expressed a desire to do so. I am stating that others want to do this so it's important to not prevent it.

However, in the end we will always support installing from third-party taps and they use Ruby files, so we will always support both JSON and Ruby install scenarios, unless the plan was to add extra code to generate JSON on-the-fly during install?

@Bo98 Yes, but this does not mean that we should vary the file that is written to the Caskroom depending on the installation method here (which is my main argument).
Having an INSTALL_RECEIPT.json for formulae that's written sometimes and not others is not useful for users.

Critically: installation information should not be broken ever, and that includes whatever is already there right now.

Firstly, "not be broken ever" is not a realistic API contract to have as an open source project.
Secondly, we have broken this many times before on purpose and by accident and we will likely break this again in future.

So if that install information switches from JSON API v2 to v3, then that means supporting both v2 and v3 indefinitely unless we do some similarly indefinite migration behaviour.

This doesn't make a lot of sense to me because: we're not proposing a switch from JSON API v2 to v3, I'm proposing a switch from "sometimes Ruby, sometimes JSON (which is not the same as JSON v3) to "the same data is consistently written regardless of the way the cask was installed" and "we want this to somewhat follow the new v3 API design" and "we should have a longer-than-usual deprecation/migration window for handling this".
I am game for e.g. Ruby files to be written only when needed (and, note, they will break in future when we make deprecations) but a JSON file is written every single time (like the INSTALL_RECEIPT.json for formulae, perhaps even with a similar name/format to that for formulae).

 Is there anything super complex there that cannot ever be represented by a DSL? If we ban uninstall_pre/postflight then I think we don't need to store any Ruby?

This is a really great idea and the direction I would like things to go in 👍🏻

Firstly, "not be broken ever" is not a realistic API contract to have as an open source project.

We've never (to my knowledge) broken brew uninstall in formulae so breaking brew uninstall on casks sounds like really bad user experience to be treating as normal and I don't think we should be accepting of that.

I'll add another vote in favor of flight DSLs. I also think that any additional cask metadata that we want to include at install time could probably be added separately from the discussion of cask file formats and it would likely be pretty uncontroversial. I don't see any reason why for instance tap couldn't get added to the config.json metadata file or something similar.

Yes there doesn't need to be any overlap at all between the formats. We just need to store basic metadata + the uninstall instructions.
Flight Ruby code is OK at install-time but is not really OK for an uninstall stage executed potentially many months/years in the future.

We've never (to my knowledge) intentionally broken brew uninstall in formulae so breaking brew uninstall on casks sounds like really bad user experience to be treating as normal/expected behaviour and I don't think we should be accepting of that.

I think it's worth clarifying what "intentionally broken" means here. It means: after a long period of deprecation/warnings/perhaps migration: we will not use the uninstall hooks from the Cask JSON/Ruby file in the Caskroom but instead use those from the Cask file, if available. The absolute worst case scenario here is you have software you installed years ago, haven't upgraded it at all since, ignored all the warnings and then when you brew uninstall it it does not remove all the files you might expect it to.

(To be clear: I'm not saying the current formats are great for that - but the next format really should be)
Cool - we can likely eliminate large parts of this problem if we can do that.
Flight Ruby code is OK at install-time but is not really OK for an uninstall stage executed potentially many months/years in the future.

I agree very strongly here with all of this. Running arbitrary Ruby with context to a Caskfile/Homebrew internals is terrible for long-term backwards compatibility.
My desire to use JSON here was specifically to limit how much dynamism is possible here so that we can maintain support for (the new format) pretty much indefinitely and make any future migration to another format dramatically easier.

Yes there doesn't need to be any overlap at all between the formats and I suggest making them separate, like how the very-stable formula tab format is.

I think some overlap would be nice if possible but I also think the INSTALL_RECEIPT.json/Tab for formulae is a good example of a very stable format that's fairly easy for third-parties to parse (and we should really probably document somewhere).
That's exactly the sort of model I'm hoping for with the same for Casks.

Across Homebrew/brew and Homebrew/homebrew-cask, issues related to how we store cask information is coming up on a weekly basis. It is very clearly a big pain point in casks.
In formulae we have tabs that we've not broken compatibility for in years. For casks, we use Ruby files which we break regularly on both Homebrew minor versions and Ruby updates. I do feel there's room to be more like formulae here.
My suggestion is:

Introduce a install receipt for casks that contains all the information you need to know about the install state of a cask, including uninstall information.
Keep a ruby file around that is only loaded when pre/postflights are needed (the install receipt will have a uninstall_preflight: true just like how formula API has a postinstall_defined: true). Never read the Ruby file other than the scenarios it's required.
Introduce more uninstall DSLs so that uninstall_pre/postflight can be banned in homebrew-cask.
For the case of third-party casks that don't switch, support best-effort upgrades when a uninstall pre/postflight errors (warning the user that things may have been left behind). This may include switching to the latest cask Ruby and/or skipping flights entirely.

I think this is better if split into separate issues but reopening this one for now until then.

Introduce a install receipt for casks that contains all the information you need to know about the install state of a cask, including uninstall information.

Agreed.

Keep a ruby file around that is only loaded when pre/postflights are needed (the install receipt will have a uninstall_preflight: true just like how formula API has a postinstall_defined: true). Never read the Ruby file other than the scenarios it's required.

If we're going to keep around a Ruby file: we may as well just use the cask file for this. I really don't want to introduce yet another Ruby-on-disk file format.

Introduce more uninstall DSLs so that uninstall_pre/postflight can be banned in homebrew-cask.

Agreed. This provides even more support for "let's not have another Ruby file here".

For the case of third-party casks that don't switch, support best-effort upgrades when a uninstall pre/postflight errors (warning the user that things may have been left behind). This may include switching to the latest cask Ruby and/or skipping flights entirely.

We should just skip flights when they error and allow uninstallation regardless of the rest here.
We should deprecate/disable/remove this behaviour at the normal rate. That gives the best part of a year to migrate which, for a rolling release package manager, is pretty decent really.

If we're going to keep around a Ruby file: we may as well just use the cask file for this. I really don't want to introduce yet another Ruby-on-disk file format.

Yes, that's what I meant. Exactly like how formulae work.

That approach makes sense to me too. Ideally we'd have some way to migrate existing installed files to the new format or gracefully handle legacy options somehow as well.

Yes, that's what I meant. Exactly like how formulae work, though formulae are only read at install-time and not uninstall-time hence the ambition to also try eliminate the usage of uninstall_(pre|post)flight.

Cool, we're agreed then 👍🏻

Ideally we'd have some way to migrate existing installed files to the new format or gracefully handle legacy options somehow as well.

This is a "nice to have" when the rest is done. Let's not blow up the scope too much here.

@Rylan12 did great work in https://github.com/Homebrew/brew/pull/17554 to add cask install receipts. That means that the remaining part of this ticket that still needs to be done is figuring out how to simplify the flight block DSLs so they can be serialized and added to the JSON API without including raw Ruby code.


Cannot Open Folder under Remote Development

Lapce Version
nightly-16e0775
System information
Windows 10 Pro 22H2
Describe the bug
After correctly connecting to the remote server, remote server button is green and commands can be executed in the lapce terminal, but cannot open folder, in other words, lapce do nothing when i click "Open Folder" button times. The following gif picture depicts this process:

Please let me know if need additional configuration or log information.

Duplicate of #2985


Vm rollup defaults (ruby solution)

Fixes #14393  [Alternate to #14448 ]
Introduced by #14285, the default value for Vm#provisioned_storage became nil when there is no hardware record. This PR changes the default back to 0.
Details
When delegating a column to another model, and the other model does not exist in the database, the SQL for the attribute comes back as a null. So the attribute looks like it is coming back from SQL, but in truth, it is just a null. Since there is no value, the default needs to be returned.

@kbrock as I have stated in private previously, I am not a big fan of changing the "default default behavior" since we are already using it in other places and this might cause issues relying on that previous behavior.
What I would prefer to see is either:

Confirmation that the 10ish virtual_delegates that already use a :default are un-affected by this (not sure how you could actually do that thought).  With that, then further documentation around virtual_delegates that this is the new behavior.
Make this an opt-in feature with an additional flag to trigger this functionality:

   virtual_delegate :allocated_disk_storage, :used_disk_storage, :provisioned_storage,
                    :to => :hardware, :allow_nil => true, :default => 0, :uses => {:hardware => :disks},
                    :use_default_for_nil_sql => true

# lib/extensions/ar_virtual.rb

         method_def = <<-METHOD
            def #{method_name}(#{definition})
 -            return self[:#{method_name}] if has_attribute?(:#{method_name})
 +            return self[:#{method_name}]#{default if use_default_for_nil_sql} if has_attribute?(:#{method_name})
              _ = #{to}
              if !_.nil? || nil.respond_to?(:#{method})
                _.#{method}(#{definition})
             end#{default}
           end
         METHOD

The latter makes it so the default behavior is explicit to the user of both the person assigning or modifying the virtual_delegate in the model, but also anyone who is hacking on virtual_delegate in the future.

I compared the current virtual_delegate code with the original code.
All cases returned the default when there was no target. (e.g. Vm delegating to a null Hardware returns the default value for the attribute, typically a 0)
I believe that the current code on master is wrong for these values. Making this change fixes these methods.
What @NickLaMuro said is totally true. The recently written code may assume that the attributes will return a nil instead of a 0. This behavior may be "wrong", but it is the way the code has been written. So "fixing" the delegates may still introduce bugs. Again, this is unquestionably true.
I just feel it is less risky to change the way default works than to leave it as is.

Here is the reference to the diffs: https://gist.github.com/kbrock/c08919f7ec51fd7d399bc6f237462804
I'm going through the references to all these variables and looking for any code like || that would handle a 0 differently than a nil.


Created an endpoint to Calculate Student loan

closes #236

Congrats @filza2112  🎉, your PR got merged. Star this repo to get more awesome swags!

@ighoshsubho done, Thank you!


[WIP] Add support for QuantumCircuit.layout to qpy

Summary
This commit adds the missing support for QuantumCircuit.layout to the qpy format. This necessitates bumping the QPY format version to 8 to accommodate the extra data needed for representing the details of the layout. The tricky piece with representing the 3 TranspileLayout attributes is representing the virtual bits in the initial layout because there is no guarantee that the input circuit's registers are in the output circuit (typically they are not when transpile() is used).
Details and comments
TODO:

[ ] Finish documentation
[x] Add backwards compatibility tests
[x] ~Add test coverage for layout with standalone qubits (no associated register)~
[ ] Add release note

Fixes #10112

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

103 of 105 (98.1%)  changed or added relevant lines in 3 files are covered.
8 unchanged lines in 2 files lost coverage.
Overall coverage increased (+0.03%) to 85.953%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

qiskit/qpy/binary_io/circuits.py
96
98
97.96%

Files with Coverage Reduction
New Missed Lines
%

crates/accelerate/src/vf2_layout.rs
3
94.74%

crates/qasm2/src/lex.rs
5
90.89%

Totals

Change from base Build<PHONE_NUMBER>:
0.03%

Covered Lines:
71302

Relevant Lines:
82955

💛  - Coveralls


Format vital

Still working on VitalFormat, but almost done :)

hi @dldmstjs500, thanks for the contribution! i can take a quick look now to see if i can fix the test.

@dldmstjs500 apologies, in hindsight i should have left this to you, but i just rebased on main and updated the requirements file with vitaldb.

@dldmstjs500 You can find the current test results at:
https://github.com/chorus-ai/chorus_waveform/actions/runs/8946501783/job/24577232681
Run FORMAT_FILES=$(cat format_files.txt)
Running benchmarks for classes in waveform_benchmark.formats.vital
Benchmarking waveform_benchmark.formats.vital.VitalFormat
Traceback (most recent call last):
  File "/home/runner/work/chorus_waveform/chorus_waveform/./waveform_benchmark.py", line 6, in <module>
    waveform_benchmark.__main__.main()
  File "/home/runner/work/chorus_waveform/chorus_waveform/waveform_benchmark/__main__.py", line 52, in main
    run_benchmarks(input_record=opts.input_record,
  File "/home/runner/work/chorus_waveform/chorus_waveform/waveform_benchmark/benchmark.py", line 129, in run_benchmarks
    filedata = filedata[channel]
KeyError: 'II'

Benchmarks are running for me locally after adding .vital to the filenames.
The benchmarking workflow on GitHub is taking a lo-o-o-ong time to complete. I think this may be a resource issue, rather than a problem with the implementation of the format.

@dldmstjs500 the tests are now running, so  I'll merge this pull request.
There are some errors being reported in the fidelity checks (see above). Please could you open a new pull request with any necessary fixes?


Add the links of the DSA in a better way

Add the links of the DSA in a better and more styled way. Removing its layout of the excel file and adding the links like a list.

we are planning to remove that list. if you have some ideas to add new dsa problems then please elaborate on your idea  @K-jais

I am thinking of adding those questions which are there in the excel but with a better frontend in the form of lists with categories like array, dynamic programming,etc.
And when the person clicks on that particular problem , he will be straight away go to that problem.

@ravikr126

We want something different from the sheet .Please tag me on discord server .i will tell you more details

Sir @ravikr126 I have tagged you in the discord server.

@ravikr126 i would like to work on this issue

@Prajwal1110 I am not associated with algo phantom right now

@harikesh409 could you please assign me this work


resilient: feature - python3 compatibility and on-poll feature

Please ensure your pull request (PR) adheres to the following guidelines:

Please refer to our contributing documentation for any questions on submitting a pull request, link: Contribution Guide

Pull Request Checklist
Please check if your PR fulfills the following requirements:

[x] Testing of all the changes has been performed (for bug fixes / features)
[ ] The manual_readme_content.md has been reviewed and added / updated if needed (for bug fixes / features)
[x] Use the following format for the PR description: <App Name>: <PR Type> - <PR Description>
[ ] Provide release notes as part of the PR submission which describe high level points about the changes for the upcoming GA release.
[ ] Verify all checks are passing.
[ ] Do NOT use the next branch of the forked repo. Create separate feature branch for raising the PR.
[ ] Do NOT submit updates to dependencies unless it fixes an issue.

Pull Request Type
Please check the type of change your PR introduces:

[ ] New App
[ ] Bugfix
[x] Feature
[ ] Code style update (formatting, renaming)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation
[ ] Other (please describe):

Security Considerations (REQUIRED)

[ ] If you are exposing any endpoints using a REST handler,
please document them in the manual_readme_content.md.
[ ] If this is a new connector or you are adding new actions

[ ] Please document in the manual_readme_content.md all methods (eg, OAuth) used to authenticate
with the service that the connector is integrating with.
[ ] If any actions are unable to run on SOAR Cloud, please document this in the manual_readme_content.md.

[ ] Are you introducing any new cryptography modules? If yes, please elaborate their purpose:
[ ] Are you are accessing the file system? If yes, please verify that you are only accessing paths returned through
the Vault API.
[ ] Are you are marking code to be ignored by Semgrep with nosemgrep?
If yes, please provide justification in an additional comment next to the ignored code.

Release Notes (REQUIRED)

Python 3 compatibility.
Added on_poll action which ingests Resilient incidents and artifacts.

What is the current behavior? (OPTIONAL)

Current version of app uses Python 2 and does not work with latest versions of SOAR

What is the new behavior? (OPTIONAL)

See release notes please.

Other information (OPTIONAL)

Any other information that is important to this PR such as screenshots of how the component looks before and after the change.

Pay close attention to (OPTIONAL)

Any specific code change or test case points which must be addressed/reviewed at the time of GA release.

Screenshots (if relevant)

Thanks for contributing!

Hi team, I need help with getting the static tests to pass. I believe there's around 3 still failing test cases.

Hi @bliew-splunk ,
We have made some changes according to our dev standards along with changes due module vulnerabilities. We have added the directly used dependencies in the requirements.txt file. Also, the cryptography and jwcrypto modules have liscence and security failure due to which we have removed those from the dev-requirements.txt file as they are not used directly anywhere in the code. Could you please verify the changes and let us know incase you have any concerns.

Hi @bliew-splunk , We have made some changes according to our dev standards along with changes due to module vulnerabilities. We have added the directly used dependencies in the requirements.txt file. Also, the cryptography and jwcrypto modules have license and security failures due to which we have removed those from the dev-requirements.txt file as they are not used directly anywhere in the code. Could you please verify the changes and let us know in case you have any concerns?

Hi @dhwanis-crest, what is the difference between pip39_dependencies and pip_dependencies?
Does "input_file": "wheels/py3/resilient-<IP_ADDRESS>.430-py3-none-any.whl" mean it is installed from a wheel on disk? Or does this still pull from pypi?

@dhwanis-crest  - I got this error when trying to install the app. I think the issue is referencing the wheel file that does not exist on disk. Can we not just install from Pypi?
{
  "log_type": "django",
  "component": "webserver",
  "product_version": "<IP_ADDRESS>5",
  "time": "2024-03-04T07:27:39.710436",
  "logger": "app_install",
  "pid": 95709,
  "level": "ERROR",
  "file": "/opt/phantom/www/phantom_ui/phapps/installer.py",
  "line": 1065,
  "message": "Error importing app by user: Installation of wheel modules failed for app b108b9d0-4611-4abf-af34-4334e084031f. Exception: Command '['/opt/phantom/bin/phenv', 'python3', '-m', 'pip', 'install', '--no-cache-dir', '--target', '/opt/phantom/apps/app_install_br86_xvj/resilient/dependencies', '--no-deps', '/opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/resilient-<IP_ADDRESS>.430-py3-none-any.whl', '/opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/cachetools-5.3.2-py3-none-any.whl']' returned non-zero exit status 1.\nOutput: Processing /opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/resilient-<IP_ADDRESS>.430-py3-none-any.whl\nError output: WARNING: Requirement '/opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/resilient-<IP_ADDRESS>.430-py3-none-any.whl' looks like a filename, but the file does not exist\nWARNING: Requirement '/opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/cachetools-5.3.2-py3-none-any.whl' looks like a filename, but the file does not exist\nERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/resilient-<IP_ADDRESS>.430-py3-none-any.whl'\n\n\n[notice] A new release of pip is available: 23.0 -> 24.0\n[notice] To update, run: pip3 install --upgrade pip",
  "ctx_request_id": "30aa1127624e9b7e8511e97a73124a46",
  "ctx_request_method": "POST",
  "ctx_user_id": 1,
  "ctx_request_remote_addr": "<IP_ADDRESS>",
  "ctx_request_user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36",
  "ctx_request_view": "app_install",
  "ctx_request_route": "^app_install$",
  "ctx_app_install_source": "tgz_upload",
  "ctx_app_install_is_forced": false,
  "ctx_app_id": "b108b9d0-4611-4abf-af34-4334e084031f",
  "ctx_app_package_name": "phantom_resilient",
  "ctx_app_version": "2.0.0",
  "ctx_app_min_phantom_version": "6.1.1",
  "ctx_connector_id": 188,
  "ctx_connector_name": "Resilient",
  "ctx_connector_version": "2.0.0",
  "ctx_connector_is_custom": false,
  "exc_info": [
    "Traceback (most recent call last):",
    "  File \"/opt/phantom/www/phantom_ui/phapps/installer.py\", line 611, in handle_pip_dependencies",
    "    phproc.run(pip_cmd, error_fn=_clean_pip_error, stream=False)",
    "  File \"/opt/phantom/pycommon3/phantom_common/phproc.py\", line 346, in run",
    "    proc.check_returncode()",
    "  File \"/opt/phantom/pycommon3/phantom_common/phproc.py\", line 486, in check_returncode",
    "    raise PhCalledProcessError(",
    "phantom_common.phproc.PhCalledProcessError: Command '['/opt/phantom/bin/phenv', 'python3', '-m', 'pip', 'install', '--no-cache-dir', '--target', '/opt/phantom/apps/app_install_br86_xvj/resilient/dependencies', '--no-deps', '/opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/resilient-<IP_ADDRESS>.430-py3-none-any.whl', '/opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/cachetools-5.3.2-py3-none-any.whl']' returned non-zero exit status 1.",
    "Output: Processing /opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/resilient-<IP_ADDRESS>.430-py3-none-any.whl",
    "Error output: WARNING: Requirement '/opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/resilient-<IP_ADDRESS>.430-py3-none-any.whl' looks like a filename, but the file does not exist",
    "WARNING: Requirement '/opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/cachetools-5.3.2-py3-none-any.whl' looks like a filename, but the file does not exist",
    "ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/resilient-<IP_ADDRESS>.430-py3-none-any.whl'",
    "",
    "",
    "[notice] A new release of pip is available: 23.0 -> 24.0",
    "[notice] To update, run: pip3 install --upgrade pip",
    "",
    "During handling of the above exception, another exception occurred:",
    "",
    "Traceback (most recent call last):",
    "  File \"/opt/phantom/www/phantom_ui/phapps/installer.py\", line 999, in complete_install",
    "    handle_pip_dependencies(app_json, tmp_app_dir)",
    "  File \"/opt/phantom/www/phantom_ui/phapps/installer.py\", line 613, in handle_pip_dependencies",
    "    raise PipDependencyError(",
    "phantom_ui.phapps.installer.PipDependencyError: Installation of wheel modules failed for app b108b9d0-4611-4abf-af34-4334e084031f. Exception: Command '['/opt/phantom/bin/phenv', 'python3', '-m', 'pip', 'install', '--no-cache-dir', '--target', '/opt/phantom/apps/app_install_br86_xvj/resilient/dependencies', '--no-deps', '/opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/resilient-<IP_ADDRESS>.430-py3-none-any.whl', '/opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/cachetools-5.3.2-py3-none-any.whl']' returned non-zero exit status 1.",
    "Output: Processing /opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/resilient-<IP_ADDRESS>.430-py3-none-any.whl",
    "Error output: WARNING: Requirement '/opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/resilient-<IP_ADDRESS>.430-py3-none-any.whl' looks like a filename, but the file does not exist",
    "WARNING: Requirement '/opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/cachetools-5.3.2-py3-none-any.whl' looks like a filename, but the file does not exist",
    "ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/resilient-<IP_ADDRESS>.430-py3-none-any.whl'",
    "",
    "",
    "[notice] A new release of pip is available: 23.0 -> 24.0",
    "[notice] To update, run: pip3 install --upgrade pip"
  ]
}

This issue is occurring as folder containing all wheel files have been ignored by gitignore. I will add the necessary wheels files here.

Hi @bliew-splunk , We have made some changes according to our dev standards along with changes due to module vulnerabilities. We have added the directly used dependencies in the requirements.txt file. Also, the cryptography and jwcrypto modules have license and security failures due to which we have removed those from the dev-requirements.txt file as they are not used directly anywhere in the code. Could you please verify the changes and let us know in case you have any concerns?

Hi @dhwanis-crest, what is the difference between pip39_dependencies and pip_dependencies? Does "input_file": "wheels/py3/resilient-<IP_ADDRESS>.430-py3-none-any.whl" mean it is installed from a wheel on disk? Or does this still pull from pypi? Also, after making these changes were you able to package the app and test it?

pip_39 dependencies specifies any dependencies required for your Python 3 app. pip_dependencies is ignored, if this key is present.
"input_file": "wheels/py3/resilient-<IP_ADDRESS>.430-py3-none-any.whl" means that which wheel file is to be installed from the package's wheels folder.
As this is Not a Splunk supported app, we do not have an asset for this app and hence, we are unable to test it.

@dhwanis-crest - I got this error when trying to install the app. I think the issue is referencing the wheel file that does not exist on disk. Can we not just install from Pypi?
{
  "log_type": "django",
  "component": "webserver",
  "product_version": "<IP_ADDRESS>5",
  "time": "2024-03-04T07:27:39.710436",
  "logger": "app_install",
  "pid": 95709,
  "level": "ERROR",
  "file": "/opt/phantom/www/phantom_ui/phapps/installer.py",
  "line": 1065,
  "message": "Error importing app by user: Installation of wheel modules failed for app b108b9d0-4611-4abf-af34-4334e084031f. Exception: Command '['/opt/phantom/bin/phenv', 'python3', '-m', 'pip', 'install', '--no-cache-dir', '--target', '/opt/phantom/apps/app_install_br86_xvj/resilient/dependencies', '--no-deps', '/opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/resilient-<IP_ADDRESS>.430-py3-none-any.whl', '/opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/cachetools-5.3.2-py3-none-any.whl']' returned non-zero exit status 1.\nOutput: Processing /opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/resilient-<IP_ADDRESS>.430-py3-none-any.whl\nError output: WARNING: Requirement '/opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/resilient-<IP_ADDRESS>.430-py3-none-any.whl' looks like a filename, but the file does not exist\nWARNING: Requirement '/opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/cachetools-5.3.2-py3-none-any.whl' looks like a filename, but the file does not exist\nERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/resilient-<IP_ADDRESS>.430-py3-none-any.whl'\n\n\n[notice] A new release of pip is available: 23.0 -> 24.0\n[notice] To update, run: pip3 install --upgrade pip",
  "ctx_request_id": "30aa1127624e9b7e8511e97a73124a46",
  "ctx_request_method": "POST",
  "ctx_user_id": 1,
  "ctx_request_remote_addr": "<IP_ADDRESS>",
  "ctx_request_user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36",
  "ctx_request_view": "app_install",
  "ctx_request_route": "^app_install$",
  "ctx_app_install_source": "tgz_upload",
  "ctx_app_install_is_forced": false,
  "ctx_app_id": "b108b9d0-4611-4abf-af34-4334e084031f",
  "ctx_app_package_name": "phantom_resilient",
  "ctx_app_version": "2.0.0",
  "ctx_app_min_phantom_version": "6.1.1",
  "ctx_connector_id": 188,
  "ctx_connector_name": "Resilient",
  "ctx_connector_version": "2.0.0",
  "ctx_connector_is_custom": false,
  "exc_info": [
    "Traceback (most recent call last):",
    "  File \"/opt/phantom/www/phantom_ui/phapps/installer.py\", line 611, in handle_pip_dependencies",
    "    phproc.run(pip_cmd, error_fn=_clean_pip_error, stream=False)",
    "  File \"/opt/phantom/pycommon3/phantom_common/phproc.py\", line 346, in run",
    "    proc.check_returncode()",
    "  File \"/opt/phantom/pycommon3/phantom_common/phproc.py\", line 486, in check_returncode",
    "    raise PhCalledProcessError(",
    "phantom_common.phproc.PhCalledProcessError: Command '['/opt/phantom/bin/phenv', 'python3', '-m', 'pip', 'install', '--no-cache-dir', '--target', '/opt/phantom/apps/app_install_br86_xvj/resilient/dependencies', '--no-deps', '/opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/resilient-<IP_ADDRESS>.430-py3-none-any.whl', '/opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/cachetools-5.3.2-py3-none-any.whl']' returned non-zero exit status 1.",
    "Output: Processing /opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/resilient-<IP_ADDRESS>.430-py3-none-any.whl",
    "Error output: WARNING: Requirement '/opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/resilient-<IP_ADDRESS>.430-py3-none-any.whl' looks like a filename, but the file does not exist",
    "WARNING: Requirement '/opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/cachetools-5.3.2-py3-none-any.whl' looks like a filename, but the file does not exist",
    "ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/resilient-<IP_ADDRESS>.430-py3-none-any.whl'",
    "",
    "",
    "[notice] A new release of pip is available: 23.0 -> 24.0",
    "[notice] To update, run: pip3 install --upgrade pip",
    "",
    "During handling of the above exception, another exception occurred:",
    "",
    "Traceback (most recent call last):",
    "  File \"/opt/phantom/www/phantom_ui/phapps/installer.py\", line 999, in complete_install",
    "    handle_pip_dependencies(app_json, tmp_app_dir)",
    "  File \"/opt/phantom/www/phantom_ui/phapps/installer.py\", line 613, in handle_pip_dependencies",
    "    raise PipDependencyError(",
    "phantom_ui.phapps.installer.PipDependencyError: Installation of wheel modules failed for app b108b9d0-4611-4abf-af34-4334e084031f. Exception: Command '['/opt/phantom/bin/phenv', 'python3', '-m', 'pip', 'install', '--no-cache-dir', '--target', '/opt/phantom/apps/app_install_br86_xvj/resilient/dependencies', '--no-deps', '/opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/resilient-<IP_ADDRESS>.430-py3-none-any.whl', '/opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/cachetools-5.3.2-py3-none-any.whl']' returned non-zero exit status 1.",
    "Output: Processing /opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/resilient-<IP_ADDRESS>.430-py3-none-any.whl",
    "Error output: WARNING: Requirement '/opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/resilient-<IP_ADDRESS>.430-py3-none-any.whl' looks like a filename, but the file does not exist",
    "WARNING: Requirement '/opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/cachetools-5.3.2-py3-none-any.whl' looks like a filename, but the file does not exist",
    "ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/opt/phantom/apps/app_install_br86_xvj/resilient/wheels/py3/resilient-<IP_ADDRESS>.430-py3-none-any.whl'",
    "",
    "",
    "[notice] A new release of pip is available: 23.0 -> 24.0",
    "[notice] To update, run: pip3 install --upgrade pip"
  ]
}

Thats fine as well, I have added these changes.

@dhwanis-crest Sure, happy to have the wheel files embedded in the app

Hi @bliew-splunk ,
Is the dev-requirements file needed?  We have added cachetools and resilient to requirements.txt file. Most of the libraries are not used directly anywhere in the code, so if not required we can delete the dev-requirements.txt file

@

Hi @bliew-splunk , Is the dev-requirements file needed? We have added cachetools and resilient to requirements.txt file. Most of the libraries are not used directly anywhere in the code, so if not required we can delete the dev-requirements.txt file
@dhwanis-crest You can remove dev-requirements.txt, thanks

Hi @bliew-splunk ,
We have modified the contains for "vault_id" parameter in "add_attachment" action, based on the contains mentioned for this parameter in the output datapaths, as the contains for both were different. Please let us know if this change is fine and check if it doesn't affect any of your playbooks.

Hi @bliew-splunk , We have modified the contains for "vault_id" parameter in "add_attachment" action, based on the contains mentioned for this parameter in the output datapaths, as the contains for both were different. Please let us know if this change is fine and check if it doesn't affect any of your playbooks.

All good, thanks


oidc silent-renew unnecessary decoding of tokens in loop

Current implementation of the silent-renew decodes the id_token every 3 seconds to parse the expiration claim and check it against the clock to see, if we have to silently renew the token with our iframe.
Why don't we parse the id_token expiration once, and set a timeout to exactly a configurable number of seconds before expiry?
Then we only decode it once, we have one fewer 3 second loop, and we also have better control over when we actually want to renew the token

good idea,
Also I would renew only every 5 minutes


Update sample app image name in release workflow

Issue #, if available:
Description of changes:
Updated sample app ECR repository name in release workflow
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Codecov Report

Merging #33 (24b9368) into main (9cbf158) will increase coverage by 50.00%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             main      #33       +/-   ##
===========================================
+ Coverage        0   50.00%   +50.00%
- Complexity      0        3        +3
===========================================
  Files           0        2        +2
  Lines           0       10       +10
  Branches        0        1        +1
===========================================
+ Hits            0        5        +5
- Misses          0        3        +3
- Partials        0        2        +2

Impacted Files
Coverage Δ
Complexity Δ

...emetry/javaagent/providers/AwsAgentProperties.java
0.00% <0.00%> (ø)
0.00% <0.00%> (?%)

...y/javaagent/providers/AwsOpenTelemetryFactory.java
62.50% <0.00%> (ø)
3.00% <0.00%> (?%)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9cbf158...24b9368. Read the comment docs.


Show last added action on Input Map

Fix #6121

This broke some of my code but only on the 2.1 branch, on master I have no problem. Still not sure what is the actual problem.

@vnen That's #6376 for which I haven't cherry-picked the fix to 2.1 yet.

@akien-mga oh, I see. Sorry for the disturbance then.


mio::io::MapNonBlock should be public?

With the deprecation of TryWrite, there's no replacement. The deprecated one just calls on the MapNonBlock trait. I'm left vendoring MapNonBlock into my program so I don't have to import deprecated mio's and don't have to maintain the implementation.
Perhaps this should be made public.
For example, the code used to be:
use mio::deprecated::TryWrite;

match socket.try_write(buf)? {
    None => break,
    Some(len) => // do stuff,
}

Now it would be
use std::io::Write;
use mio::io::MapNonBlock;

match socket.write(buf).map_non_block()? {
    None => break,
    Some(len) => // do stuff,
}

Thanks for your time.

Thanks for the report. TryRead and `TryWrite can live in a separate crate. There will be something similar provided by Tokio soon. The goal for Mio is to slim it down as much as possible, so map_non_block will not be exposed.
I also moved as much as possible of the deprecated code to mio-more


end-to-end training

Proposed changes:

introduced RasaCoreInterpreter -- NLU pipeline inside the core
state featurizer which outputs BOW vectors (one based on CountVectorizer) -- BOWSingleStateFeaturizer
creating special symbols for e2e processing;
(will create flag which will make it go through the pipeline and allow for e2e as well as non-e2e)

details

rasa/core/domain.py: change to allow processing of actions stored in Message datatype
rasa/core/events/init.py: adding message as a property of ActionExecuted and UserUttered events to store the processed utterance in message
rasa/core/featurizers.py: introducing a new featurizer, E2ESingleStateFeaturizer, MaxHistoryFeaturizer: default maxhistory -- None, i.e., it takes the whole dialog
rasa/core/interpreter.py: RasaE2EInterpreter, which collects training data and trains NLU pipeline
rasa/core/ted_policy.py: TEDPolicy input redone: it takes sparse and dense text features; There will be a HierarchicalTEDPolicy, right now it lives on the hierarchical_e2e branch because I am still running some experiments with it;
rasa/core/training/dsl.py and dsl.py: added processing for e2e data format; and added a function to process e2e user utterance format.
rasa/nlu/training_data/formats/markdown.py: function to parse actions when they come in e2e format
Ignore: tests/core/test_featurizer.py

Status (please check what you already did):

[ ] added some tests for the functionality
[ ] updated the documentation
[ ] updated the changelog (please check changelog for instructions)
[x] reformat files using black (please check Readme for instructions)

@evgeniiaraz Can we rebase this and e2e to master / merge in master?

@evgeniiaraz As part of the merge I had to move your changes from the dsl module to markdown_story_reader and story_reader. I added TODO so we don't forget that these are temporary hacks and should be removed before merging this


[Config] Add guzzle ssl verify config

Hello,
I've added a configuration for Guzzle SSL verification. Sometimes, issues arise on the development server when it doesn't use a valid SSL, leading to a GuzzleHttp Exception with cURL error 60: SSL certificate problem.
With this addition, we can disable Guzzle verification through the .env file or config.php
Thanks
to enable Guzzle SSL verify (its default to true)
ODOO_SSL_VERIFY=true

to disable Guzzle SSL verify
ODOO_SSL_VERIFY=false

ah i see, ill change it

@vheins Thank you for contributing. Released in v1.7.0
https://packagist.org/packages/obuchmann/odoo-jsonrpc#v1.7.0


Lint jenkins

add linting jenkinsfile
requires: https://github.com/SUSE-Cloud/socok8s/pull/330

@Itxaka you need to add a job template to suse-cloud/automation so we can test this PR directly

automation PR at https://github.com/SUSE-Cloud/automation/pull/3468

will send a new PR with the linting as part of the pipeline instead!


[Bug]: VirusTotal Reports Installer as Malicious

All of the following criteria must be met

[X] All Requirements must be installed.
[ ] Full Latest.log file included. If no file exists then leave this unchecked and state so.

All of the following are optional to answer

[ ] Tried reinstalling the Game.
[X] Tried reinstalling MelonLoader.
[ ] Tried restarting PC.
[ ] Was able to see the Start Screen.

Describe the issue.
Downloaded MelonLoader.Installer.exe and uploaded it to VirusTotal.com
The scan reported by 4 separate security vendors identifies it as malicious.
Cynet: Malicious Score 100
TrendMicro-HouseCall TROJ_GEN.R002V01JC23
SecureAge Malicious
Webroot W32.Malware.gen
https://www.virustotal.com/gui/file/ad3015fcf72af7fa32386e2d584ddc00ba10fef82e84b4f57adafb6183177542

Hybrid analysis also shows it as suspicious and it's labelled as malware
https://www.hybrid-analysis.com/sample/ad3015fcf72af7fa32386e2d584ddc00ba10fef82e84b4f57adafb6183177542

There's no log file to attach, I have not attempted to install this on my computer due to these identifications.
Did you attach your log file?

[ ] Yes, I attached my log file to the text box above.
[X] No, I could not find a log file at {Game_Directory}\MelonLoader\Latest.log

MelonLoader proxies windows dlls to start with the game, and injects code into games. Obviously some antiviruses will false flag MelonLoader for this. It's entirely open source so you can see that we don't do anything Malicious. We have no interest in changing anything to make these Antiviruses not flag us, because ultimately, we can't. We use some techniques to inject that some Malware might use i guess.

@RinLovesYou
This issue is with the installer, not MelonLoader itself.
Is the installer open source? Or do you only have the code for MelonLoader itself on the github?

Yes the installer is also open source and can be found here https://github.com/LavaGang/MelonLoader.Installer
I think the installer gets flagged because it downloads files from the internet (GitHub) and auto unzips it which can maybe be seen as sus


Typo "a Azure Database"→"an Azure Database"

https://learn.microsoft.com/en-us/azure/mysql/flexible-server/how-to-connect-tls-ssl
https://github.com/MicrosoftDocs/azure-docs/blob/main/articles/mysql/flexible-server/how-to-connect-tls-ssl.md
#PingMSFTDocs

Thanks @hyoshioka0128!


Inconsistant dictonary iteration behavior with content vs. expression

Description
The automatic joining functionality of looping through a dictionary works in an inconsistent and unexpected way depending on if content or expressions are returned.
The behavior of returning expressions from the loop has the ability to "lose" iterations depending on the type of expression returned. In particular, returning a dictionary causes the iteration to only return the final dictionary, and all earlier ones are lost. I think this behavior is at best confusing and rarely what an author intends.
Screenshot:

Source:
// dummy data
#let myDict = (
  a: "foo",
  b: "bar",
  c: "baz",
  d: "qux",
)

// a function that makes a dictionary out of the two parameters
#let makeXY(x, y) = {
  (x: x, y: y)
}

\#1. Printing keys and values #linebreak()
Works as expected.

#for (k, v) in myDict {
  k + ": " + v + linebreak()
}

\#2. Printing "XY" content from keys and values #linebreak()
Works as expected.

#for (k, v) in myDict {
  [ #makeXY(k, v) #linebreak() ]
}

\#3. Outputting a single *array* expression from keys and values #linebreak()
Works as expected.

#for (k, v) in myDict {
  (k,)
}

\#4. Outputting "XY" *dictionary* expression from keys and values #linebreak()
Only last element shown, does not work as expected.

#for (k, v) in myDict {
  makeXY(k, v)
}

\#5. Outputting "XY" *array* expression from keys and values #linebreak()
Works as expected, but this is the behavior I had assumed \#4 would produce.

#for (k, v) in myDict {
  (makeXY(k, v),)
}

Cheers and thanks a heap!
Reproduction URL
https://typst.app/project/rxNys3KjfbyLESBX0KE-s-
Operating system
Web app
Typst version

[X] I am using the latest version of Typst

A dictionary can only have one value per key. If every iteration produces a dictionary with the same key, only the last one will remain.
Do you want literally "x" as the key or rather the value of the variable x? In the latter case, you need to manually create an empty dictionary with (:) and call .insert on it with the pair. In the next Typst version (already on main), this will become easier as you can just write ((key): value) to force the key to be evaluated.

The case where I ran into this I was trying to produce an array of dictionaries (like in example 5.)
From my point of view, each iteration of the loop was returning a single dictionary object and I had expected them to all be returned as a single array. Similar to how in example 1, each value is present (despite being all joined into a single object.)
While the explanation of A dictionary can only have one value per key. makes perfect sense, I think it was how the loop's "join" behavior works with the dictionary that was the gotcha for me:
In none of the other cases (be it content, or array) is there any chance for any of the previously iterated values to be overwritten. Only in the dictionary loop returning a dictionary is there the potential for the overwriting of previous results.
Does my explanation make sense or do I sound completely out of left field? 😂

I see what you mean. There are some ideas and plans which impact joining. Those might alleviate this issue by having joining always perform the content-like join instead of having special behaviour for arrays and dictionaries.
That said, for now it does work as intended for now, so I'm gonna close this.


Upgrading from 4.0.1 to 5.3.1 breaks generic method UnityContainer.Resolve

I'm currently using type registration with configuration file by means of MapTo xml element and by specifying LifecycleManager (PerThreadLifetimeManager, Singleton, etc...) and constructor parameters and I'd like to continue Resolving by type only without using "name" specification.
Is that still possible?
Or what kind of changes I need to implement?
regards,
Alessandro

Would help If you could provide code sample where all these failing

yes,
here the code failing that simply invokes UnityContainer.Resolve() and after upgrade it doesn't compile with the error:
"The non-generic method 'UnityContainer.Resolve(Type, string, params ResolverOverride[])' cannot be used with type arguments"
The configuration excerpt and I'd like to have still working is:

regards,
Alessandro

configuration was stripped by saving my reply.
Trying with proper html escaping.
<configSections>

</configSections>
<unity>
<container>
<register type="Interfaces.MyInterface, Interfaces" mapTo="Implementation.MyInterfaceImpl, Implementation">
<lifetime type="PerThreadLifetimeManager" />
<constructor>
<param name="param1" value="param2Value"/>
<param name="param2" value="param2Value"/>
</constructor>
</register>
</container>
</unity>

Try:
<configSections>

...

I'm sorry, but I noticed even the method mentioned has been stripped out by angular parenthesis so I need to rephrase the issue.
The issue is that after upgrade to 5.3.1 the compilation fails, with error already mentioned before, in the line UnityContainer.Resolve<MyInterface>()
I already fixed all namespaces but this method signature change prevent me to use "legacy resolution way".
I'd like to continue using the same "legacy" way to resolve types, without needing to use the overload with "name" parameter.
Is there a workaround to obtain the same behavior?
About the configuration, I just wanna be ensured it will work at the same way as before. It's was not the focus of the issue.
many thanks

If you change namespaces correctly it should work exactly as before. We have about 1300 tests still passing with all legacy syntax. Check this project for samples.
Unfortunately documentation is still not there so you would have to do it by example.

                container.RegisterType<IX, X>("x", new ContainerControlledLifetimeManager());
                var x = container.Resolve<X>("x");

                // true for 4.0.1  false for 5.3.1
                Console.WriteLine(x == container.Resolve<X>("x"));

Yes. Think about it. You are registering IX with name "x" as container controlled instance of X. Once resolved, every other IX will return the same.
On the other hand X is not registered with container with any lifetime manager. If you resolve X it will be resolved with transient lifetime manager.
If you want X to be singleton you need to change registration into:
container.RegisterType<X>("x", new ContainerControlledLifetimeManager());
container.RegisterType<IX, X>("x");

var x = container.Resolve<X>("x");
Console.WriteLine(x == container.Resolve<X>("x"));

This change is due to this bug fix.

@ENikS
will Unity have a NotRegisteredException in order to fail fast?

But why? It is not an error that Unity can distinguish from valid case. Both of these are valid cases as far as Unity is concerned.


No longer maintained?

I see a note on the Shopware 6 extension page that the plugin will no longer be maintained. Is this true?
https://store.shopware.com/en/adyen70424242359f/adyen-payments-for-shopware-6.html

Hi @dude2511,
this was a mistake and the message has been removed now. We are still actively maintaining our Shopware plugins


Missing upgrade docs from 6.9 to 7.0

Expected Behavior
Section called "Upgrading from 6.9"
Current Behavior
The last section is "Upgrading from 6.8", as if 6.9 didn't exist or has no migrations/deprecations.
Context
If that's the case, please add a section for 6.8 to 6.9 with the message.
Steps to Reproduce
Open https://docs.gradle.org/current/userguide/upgrading_version_6.html
Your Environment
Have a project with Gradle 6.9.1
Have an intent to upgrade to 7.3

I know the naming of the chapters is a bit misleading, but the documentation is available: https://docs.gradle.org/current/userguide/upgrading_version_7.html#changes_7.1

Really confusing. So "7 and earlier" section is 6.9 to 7.1?
Where are the changes from 6.9 to 7.0?

The outcome of this might be twofold in my reading:

Rename sections to be clear. The main use case of that document is that you have a project using X Gradle and have a target Y version you want to upgrade to. It's really hard to use it like this.

Split/add sections to document 6.8 to 6.9, 6.9 to 7.0, 7.0 to 7.1 upgrades as independent sections. (Whichever is actually missing out of these.)

Will be fixed in 8.0


Bug bounty testing - please do not approve  rcntio

Bug bounty testing - please do not approve  rcntio

Hey yeah, if you could just like... drown? That would be great. Thanks.


[RFC] drivers: watchdog: Make WDT_DISABLE_AT_BOOT default to y

This commit consolidates the meaning of WDT_DISABLE_AT_BOOT option as
"disabling watchdog at Zephyr system startup for the SoCs that enable
the watchdog by default after reset", and makes this default to y in
order to prevent unintentional processor reset by the watchdog when
not explicitly configured and fed by the application.
Related discussion at https://github.com/zephyrproject-rtos/zephyr/pull/20607#issuecomment-560992779, refer to the second approach.
Signed-off-by: Stephanos Ioannidis<EMAIL_ADDRESS>

Note that, with this PR, we are (somewhat implicitly) consolidating the following rule for the watchdog drivers:

For all SoCs, the "init" function of watchdog driver shall not enable the watchdog (it should only configure the watchdog to a known state if necessary).
Watchdog shall be enabled only after the driver "setup" function is called.

As far as I can tell, these rules align with the current watchdog API as well.

@gon1332 (as stm32 watchdog user), can you have a look ?

Motivation behind this PR was to enforce the consistent default behavior of the watchdog to be "disabled by default until explicitly enabled by the application." However, this approach is wrong. There is a very good reason why some vendors start the watchdog automatically upon reset: the code can hang during the boot process before the software has a chance to configure it.
There are several major issues with having WDT_DISABLE_AT_BOOT enabled by default.

it's a relatively common feature among different watchdog implementations that the module can be configured only once upon boot. Unless the user explicitly sets WDT_DISABLE_AT_BOOT to n the watchdog subsystem will be disabled for good. Any code that wants to be portable will always need to explicitly set WDT_DISABLE_AT_BOOT to n.
watchdog that can be re-programmed and was started automatically upon reset will be temporarily disabled. The option is easy to overlook and this defective behavior is unlikely to be caught by a testcase. Unless the user explicitly sets WDT_DISABLE_AT_BOOT to n the functionality of the watchdog subsystem will be degraded.
there are already users who configured their watchdog subsystem and didn't set WDT_DISABLE_AT_BOOT to n. If they happen to have a watchdog module that can be configured only once, e.g. Atmel SAM family, this PR effectively - and quietly - disables their watchdog subsystem. Some of them may not have a testcase that will catch the issue. And the broken firmware will get shipped.

Having WDT_DISABLE_AT_BOOT set to y makes it easy for us to maintain samples and testcases since we almost never want the watchdog to be enabled. Maybe apart from the watchdog testcase. However, in a production environment the opposite is true, users virtually always want the watchdog to be enabled. Now we force everyone out there to explicitly set WDT_DISABLE_AT_BOOT to n.
In fact, there is no good reason to ever have WDT_DISABLE_AT_BOOT option enabled in the production environment.
To fix the issue reported in #21025 we can set the option to y in a board Kconfig.defconfig file. It's more work for us since we need to modify every evaluation board that is starting the watchdog automatically but it saves extra work and frustration for Zephyr users.
I believe this PR should be reverted.

@mnkp The real problem is that we have different behaviours across different platforms and this can lead to even more confusion: https://github.com/zephyrproject-rtos/zephyr/pull/20607#issuecomment-560992779.

This means that we have two possible states for the watchdog after OS initialisation, across different SoCs when CONFIG_WDT_DISABLE_AT_BOOT=n (note that wdt_nrfx.c, for instance, does not enable watchdog when CONFIG_WDT_DISABLE_AT_BOOT=n).

There is indeed a good reason for enabling the watchdog by default after reset, and this is a useful feature to have. Nevertheless, I believe this is a "feature" and should not be activated without user's knowledge unless we introduce an API-level change/specification that states the watchdog state after init is 'enabled', in which case we will need to enforce this behaviour across all existing platforms (think of it as WDT_KEEP_ENABLED instead of WDT_DISABLE_AT_BOOT, because that is really what it is).
With the well-defined behaviour that the watchdog shall only enter the 'enabled' state after the setup API function is called, the only problem with WDT_KEEP_ENABLED=n (or WDT_DISABLE_AT_BOOT=y) approach is, as previously noted, that some platforms do not allow the watchdog to be re-enabled once disabled.
IMO, this problem should be addressed by adding assertion on the enable status of the watchdog (either from a HW register if available, or simply asserting CONFIG_WDT_DISABLE_AT_BOOT == n) in the setup API implementation for such platforms; hence, seamlessly guiding users to take advantage of the "WDT_KEEP_ENABLED" feature). Note that, with this approach, there will be no unexpected reset if an application does not use the watchdog; if one ever uses the watchdog (i.e. calls the setup API function), an error will be displayed guiding them to set WDT_KEEP_ENABLED=y (WDT_DISABLE_AT_BOOT=n).
To summarise the status quo:

The default WDT_KEEP_ENABLED=y (or WDT_DISABLE_AT_BOOT=n) configuration does not conform to the current API (AFAICT, the watchdog should only enter the 'enabled' state after the setup API function is called).
The default WDT_KEEP_ENABLED=y (or WDT_DISABLE_AT_BOOT=n) configuration behaves differently across different platforms and this can lead to portability issues as well as extreme user confusion (as we have already seen in a few issues).

I do not think this PR should be reverted; instead, I believe we should implement the "watchdog state assertion in the setup API function approach" stated above.

IMO, this problem should be addressed by adding assertion on the enable status of the watchdog (either from a HW register if available, or simply asserting CONFIG_WDT_DISABLE_AT_BOOT == n) in the setup API implementation for such platforms; hence, seamlessly guiding users to take advantage of the "WDT_KEEP_ENABLED" feature). Note that, with this approach, there will be no unexpected reset if an application does not use the watchdog; if one ever uses the watchdog (i.e. calls the setup API function), an error will be displayed guiding them to set WDT_KEEP_ENABLED=y (WDT_DISABLE_AT_BOOT=n).

This would be a valid approach, however it's not elegant and it's breaking current API. It will not be enough to enable the watchdog driver to use it, we also force users to always set  CONFIG_WDT_DISABLE_AT_BOOT == n. In effect we need to enable it twice. As I mentioned, in the production environment this will always be the case.
The design of watchdog API takes into account that by default some watchdog modules can be enabled upon reset. We have wdt_disable function but there is no wdt_enable or  wdt_init. There is only wdt_setup which does just that, it can be used to configure or reconfigure the watchdog but makes no promises about its previous state. The watchdog API was designed around the notion that unless user explicitly configures the module it's left in its default configuration. That's what we generally do and users expect. To write a portable application on a platform that has a watchdog it's enough to configure or disable it. We provide users all the necessary tools.
We may improve the documentation but we shouldn't set CONFIG_WDT_DISABLE_AT_BOOT == y by default. Especially that there are other less intrusive methods to deal with issues like #21025.


Write up a guide about null vs undefined

c.f. https://2ality.com/2021/01/undefined-null-revisited.html


feat(move): Add --insert flag

This adds an -i/--insert flag to git move to attempt to insert the moved commits between the destination and it's children.

This is still draft b/c I still need to confirm a few things, and I don't like my error messages.
This only works if the the moved subtree is a "stick", ie the subtree root commit (ie source commit) has exactly 1 parent and the subtree has exactly 1 head.

I have a suspicion that the "exactly 1 parent" restriction is technically unnecessary, but it made the code easier to write and reason about.

This also updates the rebase planner to support moving commit ranges instead of just entire subtrees, to handle the case where the moved commits are already descendants of the destination. This works, but I see that there has been some recent work in https://github.com/arxanas/git-branchless/tree/move-individual-commits that might be a more elegant/less-hacky solution. I'm happy to take guidance on how to improve this!

Haven't looked at this PR at all yet, but I recommend using -I instead of -i for the short form. For example, we might want an equivalent to git commit -i to interactively select changes and git commit -I to insert the commit into the subgraph.

I recommend using -I instead of -i for the short form

Sure thing. Done.

I see that there has been some recent work in https://github.com/arxanas/git-branchless/tree/move-individual-commits

So, I went ahead and updated/rebased this work onto that existing work just for the sake of exploration. I went ahead and implemented Constraint::ImmediateChild as well as plan::move_commit, but I have not wired up an --exact flag.

FYI I just pushed a "wip" commit that is my first attempt at a struct/impl. If you like the general initial approach of treating the constraint_graph as a HashMap<NonZeroOid, HashMap<NonZeroOid, Option<NonZeroOid>>>, then perhaps this refactor may provide a more ergonomic way to manipulate those constraints instead of directly, repeated querying the keys/values of the Map.
Emphasis on WIP. 😄 😴

I've chopped up the wip changes I mentioned and pushed them back up into the commit graph so the changes flow a bit better. I know this isn't done yet, but I think it's ready for an initial review to point out all of the things that are hiding right in front of me. 😄  I'll update the PR description when I get some more time in the next few days, but the summary of the changes are:

pick up the changes from @arxanas move-individual-commits branch
extend those forward and extract a ConstraintGraph struct in the rebase planner. (This is the bulk of the changes, but it wasn't rocket science; it was just a straightforward refactoring to hide some code behind a more expressive (maybe? I think?) API.)
updating the planner to support moving ranges of commits
connecting a few wires to implement --insert

Thank you for this detailed guidance; that makes a ton of sense! I've played with it a bit and I was able to remove ChildAndSomeDescendants w/o issue. I have some more error handling to work on before I want to push, though. 🙈
While removing ChildAndSomeDescendants, I started wondering if we even need ImmediateChild. I am assuming that I'm just not seeing something that you have in mind for it, but it strikes me that we could treat it like a range with only 1 commit. That is, in your example of moving C on top of A w/ ImmediateChild, couldn't that also be accomplished with a ChildAndDescendants from C to A and a ChildAndDescendants from D to B?

While removing ChildAndSomeDescendants, I started wondering if we even need ImmediateChild. I am assuming that I'm just not seeing something that you have in mind for it, but it strikes me that we could treat it like a range with only 1 commit. That is, in your example of moving C on top of A w/ ImmediateChild, couldn't that also be accomplished with a ChildAndDescendants from C to A and a ChildAndDescendants from D to B?

I think you're right, it could be all implemented with one operation. The main concerns in my eyes are:

When we query the DAG. Currently, we don't query it at all when building the initial set of constraints. If we wanted to issue two constraints to move a single commit, we'd have to query for the children commits at the time that we insert the constraint into the set.
Error-reporting. I forget what we do currently. Is it an error to issue another constraint for a commit which is already constrained? If we decide we want to move a child commit somewhere else in the graph, we have to be sure that the new constraint overrides the old one properly.

Feel free to proceed however you think is cleaner, whether it's having either one or multiple types of constraint.
By the way, you might be interested in contributing the same feature either instead or in addition to Jujutsu. In the long term, I think it will make most sense to concentrate efforts there.

I'll try to pick this up again soon (sorry, I've been distracted). And thanks for clarifying about querying the DAG while building the initial constraints vs building the actual plan. I hadn't been thinking about the performance penalty of DAG queries since most of my repos are relatively small. I'll keep tinkering and see what I come up with.

By the way, you might be interested in contributing the same feature either instead or in addition to Jujutsu. In the long term, I think it will make most sense to concentrate efforts there.

I'm looking forward to trying out jj sometime, but for the time being I'm still really liking the way that branchless just augments my existing workflow. Once I land this PR here, maybe that's a great excuse to finally dig into jj and learn more about it, though!

This now includes my latest WIP. The main API change is removing Constraint::ImmediateChild and adding Constraint::ChildrenOfToParentOf.[^1] (Terrible name ... any suggestions?) Doing so allows moving single commits and ranges w/o doing any dag queries while collecting the initial constraints. There are some dag queries needed in r#move to figure some things out, but I need to think about those some more before finishing this. I also haven't addressed your question about error reporting yet.
[^1]: I also expanded the scope of the new ConstraintGraph struct a bit, moving many fns that were only related to populating the constraint graph from RebasePlanBuilder into ConstraintGraph.

OK, I think I'm ready for some more guidance on this. I have some questions based on your comments:

When we query the DAG. Currently, we don't query it at all when  building the initial set of constraints. If we wanted to issue two  constraints to move a single commit, we'd have to query for the children commits at the time that we insert the constraint into the set.

I interpreted this as meaning that we should avoid DAG queries while collecting the initial constraints. Is that what you intended? If so, how strongly should be we be avoiding that? I ask b/c I'm finding it very difficult to handle some cases. I think that doing some DAG queries while collecting these constraints might be helpful, but I don't want to dig into that if it's a blocker related to performance or something.

Error-reporting. I forget what we do currently. Is it an error to  issue another constraint for a commit which is already constrained? If  we decide we want to move a child commit somewhere else in the graph, we have to be sure that the new constraint overrides the old one properly.

Currently, no, it is not an error to "reconstrain" a commit. As I understand it, in fact, doing so will just duplicate the constrained subtree. I have code locally that handles this better, and I'm just working on the details of some specific cases (like the aforementioned adjacent commits).
Here's an example that I'm struggling with. Consider this graph:
flowchart RL
    E --> D --> C --> B --> A

Now consider these rebase instructions:
builder.move_commit(C, A);
builder.move_commit(D, A);

I would expect this to produce the following rebased graph, which just moves the exact commits C and D out of the B-C-D-E lineage:
flowchart RL
    E --> B --> A
    D --> A
    C --> A

The problem I'm facing is that, w/o doing any DAG queries initially, it's very difficult (impossible?) to know whether E should end up on B or C, because of how the first move_commit affects the second:

the first call says "move C to A" and "move D to B"

As you know, moving D also implies it's descendent E

the second then says "move D to A" and "move E to C"

However, as C is already constrained, I've opted to keep looking at it's ancestors until we find an unconstrained commit and then using that.

The issue I'm facing is that I would expect those 2 move_commit calls to be order-independent (so we could do them in any order), but because of the need to handle child commits, doing these in the reverse order isn't working:
builder.move_commit(D, A);
builder.move_commit(C, A);

now the first says "move D to A" and "move E to C"
the second call says "move C to A" and "move D to B"

Note how the first order (where move C then D) doesn't constrain E until the second call. In this order, though, E is constrained first, and the second call doesn't/can't know that it should cancel the constraint.

I think that doing some more interrogation of the DAG and existing constraints while the initial constraints are being collected could help with this, but I want a sense of how careful to be w/ DAG queries before proceeding. Thank you!
Phew! This has proven to be much more complicated than I expected. The basic cases (ie handling the needs of move --insert) seem to be easy, but generalizing things to be able to handle possibly arbitrary range constraints is proving to be tricky. I guess that leads me to another question: is it even worth it to support arbitrary range constraints at this point? If we limited the algorithm to only accept 1 range constraint, it would delay much of the aforementioned complexity until we really need/want it.
What do you think?

I interpreted this as meaning that we should avoid DAG queries while collecting the initial constraints. Is that what you intended? If so, how strongly should be we be avoiding that? I ask b/c I'm finding it very difficult to handle some cases. I think that doing some DAG queries while collecting these constraints might be helpful, but I don't want to dig into that if it's a blocker related to performance or something.

This isn't a blocker for performance. After all, we'd have to do the same DAG queries immediately afterward when we want to execute the rebase plan anyways. It's just a stylistic matter, so we don't need to restrict ourselves here if it makes something more difficult to implement, but see below.

The problem I'm facing is that, w/o doing any DAG queries initially, it's very difficult (impossible?) to know whether E should end up on B or C, because of how the first move_commit affects the second

The problem is that you're trying to do all the work in one pass, but since later constraints can affect earlier constraints, this won't work. Really, you want to break this up into two steps:

Pass 1, apply direct constraints:

Set D's parent to A.
Set C's parent to A.

Pass 2, apply implied constraints:

Find D's immediate unconstrained children, then find D's nearest unconstrained ancestor. Its immediate parent C is already constrained (by pass 1), so we skip it and go to B, which is unconstrained. So we'll set E's parent to B.
Find C's immediate unconstrained children. D is its only child and it's constrained (by pass 1), so we have no more work to do.

Since we're doing a two-step constraint resolution process, I think it also works out that we don't need to do any DAG queries when issuing the initial constraints with move_*. We have to do the second pass later anyways to make sure we're constraining child nodes correctly.

I guess that leads me to another question: is it even worth it to support arbitrary range constraints at this point? If we limited the algorithm to only accept 1 range constraint, it would delay much of the aforementioned complexity until we really need/want it.

It looks like you the single-range case working, so I'd be happy merging this as-is. There would be immediate use-cases, such as git moveing ranges of commits (obviously) but also for a git commit --insert command and possibly also a git split command.
Thanks for all your work so far! 😊

Thank you for your guidance on this! These comments were very helpful. And you're correct, handling the constraints in 2 passes means we still don't need any DAG queries while collecting the initial constraints. Also, I was able to get everything working w/o limiting us to moving a single-range, so no worries there either.
The only piece that I needed to add was that we need to handle the "ancestor search" for constrained ranges (and commits) differently than constrained subtrees. (I'm open to other ways to handle this, of course, but this is what I've come up with.) Basically, in the above example, moving the 2 commits C and D will produce different results than if we were to move the subtree @ C and then commit D (in either order). In the former, we want E to end up @ B (as covered in previous comments), but in the latter we want it to end up @ C, even though C is also being moved. I think that I've covered the cases in all of the tests that I wrote, but I would not be surprised if you find some that I didn't cover.
Speaking of tests, I added a bunch of tests for the rebase planner. You'll note that they're very similar to some tests for the move command in that they directly manipulate a repo and test against smartlog output. If that's to "end-to-end-y", I can redo them, but I found the smartlog output very easy to work with in those tests. Since the core of this feature is really the ability to add numerous, arbitrary ranges to the rebase planner, I tried to be exhaustive in the tests. If they're too much (or not enough!) I can add or remove as desired.
I've updated the main PR description w/ some more overview details.

Basically, in the above example, moving the 2 commits C and D will produce different results than if we were to move the subtree @ C and then commit D (in either order). In the former, we want E to end up @ B (as covered in previous comments), but in the latter we want it to end up @ C, even though C is also being moved.

I actually never thought of this case, so I'm glad you discovered it and figured it out 😅.


libwebsockets: added support to build uwp versions

This PR adds support for building uwp versions of libwebsockets. The patch is only applied to the source code if the portfile.cmake file detects a WindowsStore build.
This PR depends on PR #524 to pass WACK testing. (openssl dlls were failing WACK)

By default (and in the case of this port) we share the source tree across all triplets. Therefore, to get reproducible results, could you modify the patch to always be applied? My brief skim says it shouldn't significantly change the non-uwp build, but I'd like a second opinion.

I authored the patch to work across all triplets so I will modify the pull request to patch the source for all builds.

@ras0219-msft The patch is now applied to all builds triplets.

Thanks!
I bumped the version since this modifies the produced package.


Feature: added as gen custom types to proto cli

Unable to make it work for now, on hold

Rebased and working, pls test on your side before merging to avoid the "it works on my machine" syndrome!


Website server start should check mixer readiness

the two codeql errors look like good candidates for a fix, but they are not related to the PR at all.

the two codeql errors look like good candidates for a fix, but they are not related to the PR at all.

Okay, looks like they are not trivial to fix. Going to look at the codeql errors in batch later.

Thanks for review!


support troubleshooting inconsistent results when run in a script

I'm trying to figure out why we are consistently seeing a handul of https? == false results when we run this script in a batch (500 sites total).  Restarting our ruby app and testing the same sites again causes https? to be true.

This normally happens to sites on a particular server or servers, so Its likely related to some kind of timeout or other server/network issue
I'm seeing similar (but different) results when using a proxy, We get about 1-2% false positives (also on a server, but perplexingly a different server than when not using the proxy)

I was going to put some kind of delay or reduce the currency and see if that helped.  Also due to memoization the false positives are saved... not sure if there is a strategy to get around that.

I think I know whats going on, but running with debug on, i see
App 10921 stdout: About to connect() to [domain] port 443 (#0)
App 10921 stdout:   Trying [ip]...
App 10921 stdout: Closing connection #12
App 10921 stdout: Connection (#12) was killed to make room (holds 16)
Also, googling "connection was killed to make room" is ... interesting....
I'm trying to run this with typhoeus_options = { forbid_reuse: true }..
Although, since I'm running into these issues, this seems like I'm doing something wrong.  Shoudl I not be using this for this use case?

It looks like connection ... was killed to make room is a cURL thing, not a Typhoeus thing:
http://blog.gmane.org/gmane.comp.web.curl.php/month=20081001
http://curl.haxx.se/mail/lib-2006-09/0130.html
http://tekfacts.com/ow-do-pull-from-a-it-repository-through-an-proxy/
I'm not sure if the curl CLI client shares state between independent requests, or if it means something else entirely.

This normally happens to sites on a particular server or servers, so Its likely related to some kind of timeout or other server/network issue

I observe a number of "troublemakers" in the .gov space that exhibit common fluctuations week-to-week, where they appear go on and off. I think it relates to them either having intermittent outages, or taking long enough to time the client out. There's also sometimes issues with us making cURL requests from an EC2 instance and how requests from Amazon get handled by enterprise network edges.
Entropy is high out there. :/

Thanks @konklone, I think the forbid_reuse thing worked for me, I can see the connections getting killed instead of re-used...  I've also had to bump up timeouts to 30 seconds (!!!) to cut down on errors.

Sadly, 30s is the minimum timeout I need to apply for "government websites" as well.


Checklist for on-boarding new people in contracting roles

Make a checklist in the wiki for things that have to be done when on-boarding new people in contracting roles. This should include

time tracking setup
access to repositories on GitHub
announcement at news blog

@ovivoda since you are onboarding now, can you compile the items that you see?

@wipfli here it is what I have so far, but I will keep updating this with new things that will pop-up

time tracking setup
announcement at news blog
access to repositories in GitHub
include the person into the right Slack channels
GitHub Projects guide on creating tasks. TO be refined
guide on work methodology. TO be refined
hand over onboarding material. Eg. introduction to MapLibre project, a bit of history, architecture documents, etc... TO be collected/defined
setup objectives/goals

Will put into wiki page

https://github.com/maplibre/maplibre/wiki/Maintainer-Onboarding-Checklist


Pull requests

Hello there!
Awesome project.
I've been working with OPS from python for 2.5 months now and might be able to help with
Legal service and Bulk operations if I can find the time

Coverage remained the same at 99.749% when pulling bb35b90a01845a699e2db58425ea88b6917f7982 on development into 457cf17560124c3a912e146ccc7730d3fb264874 on master.

Coverage remained the same at 99.749% when pulling bb35b90a01845a699e2db58425ea88b6917f7982 on development into 457cf17560124c3a912e146ccc7730d3fb264874 on master.


Removes generated_plugin_registrant.dart from lint

Codecov Report

:exclamation: No coverage uploaded for pull request base (collection-execution@8b0676f). Click here to learn what that means.
The diff coverage is n/a.

@@                   Coverage Diff                   @@
##             collection-execution      #61   +/-   ##
=======================================================
  Coverage                        ?   74.41%
=======================================================
  Files                           ?       26
  Lines                           ?      469
  Branches                        ?        0
=======================================================
  Hits                            ?      349
  Misses                          ?      120
  Partials                        ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8b0676f...edf8c63. Read the comment docs.


[Global Function] GetWord - Codeunit 5054 WordManagement

Copy of #1007 -> Request for 'Global' function (to allow at least OnPrem with customized AL base app) instead of 'External'
In the latest BCoP CU3 release, a fix is shipped to avoid pending Winword.exe sessions. The solution is using a new WordApplicationHandler codeunit, subscribing to events raised in codeunit 5054 WordManagment. 5 new functions are added, most of them external (Activate, IsActive, ...)
In our solution, we've heavily customized the WordManagement codeunits (in a separate object). However, we'd like to apply the same pattern / fix, using the new WordApplicationHandler codeunit. Therefore we re-use the new functions (Activate, IsActive, ...) from codeunit 5054. However, function GetWord is not global.
Sample code of our WordManagement codeunit (CreateWordAttachment function)
WordManagement.Activate(WordApplicationHandler,50000);
WordManagement.GetWord(WordApplication);  // GetWord is not Global
...
WordManagement.Deactivate(50000);

Can you please set 'GetWord' at least to Global, untill a final solution is available?
Still struggling with the original issue about how to extend Cu5054 and thus applying the standard pattern Activate (=External) / GetWord (=Local) / ... DoYourThing with Word ... / Deactivate (=External) without duplicating all code from Cu5054 / Cu5068.

@AlexanderYakunin, any status update on the above, setting 'GetWord' to global?
I would like to avoid having to merge our custom WordManagement module to the standard BC one and request 100+ events :(
PS: Seems like COD 5054 contains an IntegrationEvent OnGetWord(VAR NewWordApplication : DotNet "Microsoft.Office.Interop.Word.ApplicationClass" RUNONCLIENT;VAR IsFound : Boolean), that allows the use DotNet parameters?

This is OnPrem feature, it will not work for cloud.

No longer needed ...


fixed missing trt_backend option & remove un-need data layout check in Cast

update .gitignore
fixed missing trt_backend option bug when init from trt
remove un-need data layout and add pre-check for dtype
changed RGB2BRG to BGR2RGB in ppcls model

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.