Policy validation process for cosign policy | sget

[sget change] sget should be able to validate previous policies for consistent proof of previous signers.
[cosign change] signatures will be moved from the policy.json and place into a cosign annotation.
[cosign change] We add in previous root reference by digest blob of the policy to the new policy.

@imjasonh @jyotsna-penumaka @asraa

So the general idea is that cosign policy init would write only the policy (maintainers, threshold) to an OCI registry, then cosign policy sign would effectively become cosign sign reg.io/policy@digest, which pushes reg.io/policy:sha256-digest.sig as normal. Policy signatures effectively become the same as cosign signatures, which is nice.
A benefit is that, when a policy changes, it immediately invalidates all the signatures attached to it, because they all point to the previous policy's digest.
When a policy is updated (cosign policy edit or something), the new policy should include a pointer to the previous policy's digest -- "previousRoot": "sha256:digest" -- which verifiers can lookup if they want to ensure that the policy wasn't edited in some malicious way.
If the policy doesn't have a previousRoot defined, verifiers should still lookup in Rekor to see if there had been any previous policies defined, since a malicious policy pusher might not have included it to cover their tracks.

A benefit is that, when a policy changes, it immediately invalidates all the signatures attached to it, because they all point to the previous policy's digest.

Would this mean we lose the previous sigs?
For example; Bob run policy init and then signs the policy. Bill (a 2nd maintainer) then signs the policy, would they remove Bobs sig?
Or are we thinking only the policy creator signs the policy?

Would this mean we lose the previous sigs?

Nope, when a policy changes the previous sigs would still be present at reg.io/policy:<previous-policy-digest>.sig.
When a second person signs the policy they get appended to reg.io/policy:<current-policy-digest>.sig, and don't replace existing signatures, just like with cosign sign today.

When a policy is updated (cosign policy edit or something), the new policy should include a pointer to the previous policy's digest -- "previousRoot": "sha256:digest" -- which verifiers can lookup if they want to ensure that the policy wasn't edited in some malicious way.
If the policy doesn't have a previousRoot defined, verifiers should still lookup in Rekor to see if there had been any previous policies defined, since a malicious policy pusher might not have included it to cover their tracks.

I think this question is rooted in my need to learn more about how these policies work 🙊  but I'll ask anyway...  What is the advantage of having new policy versions point to the prior version, if consumers should check Rekor even when previousRoot is absent? And could a malicious actor go ahead and add the previousRoot field and value to avoid suspicion?

A totally reasonable question! (At least, to me, someone who may also need to learn more about how these policies work)
AIUI the previousRoot isn't intended to replace the need to lookup the previous policy in Rekor, it's basically just another bit of information that's available at policy edit-time that can provide added assurance that something isn't Afoot. Verifying clients like sget should still keep looking up in Rekor and report/fail if anything is missing.

Another question, I can't recall what (or if) we decided on how the signature of the artifact should be handled?
I presume we will run `cosign sign-blob <name_space> and as long as there are maintainer signatures greater than the threshold within the policy, we can then move to checking previousRoot instances?
So effectively there will only be one manifest?

I started to sketch out what would be in the cosign issue here [0], please all weigh in with views / amendments:
https://docs.google.com/document/d/1CavafvqJxWrlFf9FidF0LZ-YjKNUIs5LvTuqWQbbZPs/edit#

Yeah, I think we should walk through the overlap with cosign upload-blob and cosign sign-blob -- is the blob we're signing the script.sh? Or the policy for script.sh?
When the policy held the signatures, it was easier to tell, but now I think we can actually simplify the UX and get rid of cosign policy sign.
The flow could be:

Maintainer writes script.sh, shares it (cosign upload-blob script.sh reg.io/project) --> emits script blob digest
Maintainer inits a policy for script.sh (cosign policy init --blob script.sh reg.io/project --maintainers=X,Y,Z --threshold=2), which is attached to the script's manifest, like an SBOM*
Maintainer X signs the blob -- not the policy (cosign sign-blob script.sh reg.io/project)
sget reg.io/project script.sh finds the blob and policy, checks whether the script's attached signatures meet the policy; they don't, so sget fails.
Maintainer Y signs the blob (cosign sign-blob script.sh reg.io/project)
sget reg.io/project script.sh finds the blob and policy, now the signatures do satisfy the policy, download and run.

* Does this mean we allow multiple policies attached to a blob? 🤔
NB: In the absence of a policy (omitting #2 above) sget could check for any valid signature, and #4 would run the script.
This starts to make the script blob flow more similar to the image sign/verify flow -- you sign the blob, not the policy -- and makes policies more easily applicable to images, so other verifying clients (e.g., cosigned) can start to align with sget. I'm not sure if this fundamentally undermines policies though, since they're only consulted during verification, and aren't signed anymore.

AIUI the previousRoot isn't intended to replace the need to lookup the previous policy in Rekor, it's basically just another bit of information that's available at policy edit-time that can provide added assurance that something isn't Afoot. Verifying clients like sget should still keep looking up in Rekor and report/fail if anything is missing.

Just on the flip side, as I've thought of this -- if for whatever reason you need to totally re-write a policy (let's say, some tragic event where the maintainers all disappear), then you would need to overhaul the root policy. We probably want the UX to handle this like "Verified up to blah point"

I'm not sure if this fundamentally undermines policies though, since they're only consulted during verification, and aren't signed anymore.

This is the only thing I'm curious about digging in to. On one hand, you have some sort of trust in the people who have push access to the repository. They might have implicit "root". However, maintainers Y and Z never agreed to be part of this policy, i.e. no one else with push access explicitly allowed maintainer X to define the policy. Trust needs to start somewhere... but I don't know the answer beyond everyone signing the policy, and clients having an initial way of "trusting" the starting users.
Just send a request to access the doc!

One thing we need to keep in mind is a lot of users will already have the install script in their repository. It's going to feel disconnected for them to have the script available in change control git and yet need to re-upload to a registry and mess about with blobs etc.
I expect the dominant UX will be:
I am user with my install script already inside the repository. I inform my users to run curl https://raw.githubusercontent.com/blah/install.sh | bash. I like this as its very easy for me to maintain.
I would like to sign this and have my users download it safely and then execute
I might want to allow multiple users to sign my install script as my project has several maintainers (but it could be just one individual).

It's hard to argue with the existing UX's ease of use. I also wonder if requiring maintainers to copy it to some registry is just going to be too high a bar. rget (archived) lets users fetch scripts/artifacts from any URL, relying on the tlog.
Something like that would prevent a moving reference from working (e.g., https://raw.ghc.com/foo/bar/master/script.sh) but that's a pretty terrible practice anyway, users should fetch from a tag or commit SHA.
Crazy stream-of-consciousness ideas follow, I don't know if any of them are good:

A more rget-like sget could transparently fetch the URL and validate the contents against the tlog, then mirror it in an OCI registry and remember to go to their registry for future fetches. That solves the UX issue a bit, and transparently opts the user into the best practice of keeping a local copy in case GitHub is down (unless your registry is also down).
To define a maintainer-signoff policy in that mode, we could have some convention around a policy.json file alongside the artifact in the repository, which we'd also verify against the tlog.
We couldn't store signatures in the repository alongside the artifact+policy, or else the commit SHA would change from the tag, or you'd have to move the tag. This also prevents policy.json from being changed, but maybe we just say if you want to change the policy you have to change the tag and make people fetch that instead? 🤔
(All these cases assume the GitHub raw content server behavior; these limitations wouldn't apply if you're just serving script+policy+signatures from https://myserver.com)


Csutomize checkbox checked view

Hi, current checkbox checked has a blue X apparance. i tried to get normal checkbox checked on zebra as like html but could not find. is it possible and how?
thansk.

Do you need to change X element color ?

No, i want to change X with cheched icon as used in HTML. There is no need to change to it now. i use below code and get it.
var checkbox = document.createElement('input');
checkbox.type = "checkbox";

zebra.ui.HtmlElement ( checkbox )

when i use HTML element and create shape some html code remains code side although i delete the item (for example HtmlTextArea). If i enter new page the removed shapes shown because remained Html codes.
i additionaly delete the html codes and solved it but i wonder what the right way for it.

Hi there, have done this for the zebkit theme builder project which should be able to make public soon.
Basically you need to define your own Checkbox View, for example this one changes the 'X' to a tick -
pkg.CheckboxCustomView = Class(pkg.View, [
        function(color) {
            if (arguments.length > 0) {
                this.color = color;
            }
        },

        function $prototype() {
            /**
             * Ticker color.
             * @attribute color
             * @type {String}
             * @readOnly
             * @default "rgb(65, 131, 255)"
             */
            this.color = "rgb(65, 131, 255)";

            this.paint = function(g,x,y,w,h,d){
                g.beginPath();
                g.strokeStyle=this.color;
                g.lineWidth=2;
                g.moveTo(x+5,y+h-2);
                g.lineTo(x+1,y+h-7);
                g.stroke();
                g.beginPath();
                g.moveTo(x+w-2,y+2);
                g.lineTo(x+5,y+h-2);
                g.stroke();
                g.lineWidth=1;
            };
        }
    ]);

And then override the correct views for Checkbox class -
"view": {
                "pressed.out"       : {<EMAIL_ADDRESS>[ "%{palette.mainColor}", "%{palette.mainColor}" ] },
                "out"               : {<EMAIL_ADDRESS>[] },
                "over"              : {<EMAIL_ADDRESS>[ "lightGray" ] },
                "pressed.over"      : {<EMAIL_ADDRESS>[ "rgb(126, 239, 104)" ] },
                "pressed.disabled"  : {<EMAIL_ADDRESS>[ "%{palette.borderColor}", "%{palette.borderColor}" ] },
                "disabled"          : {<EMAIL_ADDRESS>[] }
            }

I injected my code directly into the zebkit.draw package and then modified the ui.json theme file (note you will need add/modify for the palette colours), you can also do at a user code level, but there are some things you can't theme in json without changing/adding to the core files, and this was one of them...

thanks @Gumphrie


How to continue training if the program is interrupted during the use of Lora training. The code script is as follows:torchrun --standalone --nnodes 1 --nproc-per-node 1 vla-scripts/finetune.py \   --vla_path "/mnt/d/ll/openvla/openvla-7b" \   --data_root_dir "/mnt/d/ll/openvla/BridgeData" \   --dataset_name bridge_orig \   --run_root_dir "/mnt/d/ll/openvla/checkpoint" \   --adapter_tmp_dir "/mnt/d/ll/openvla/adapter_tmp" \   --lora_rank 32 \   --batch_size 4 \   --grad_accumulation_steps 1.5 \   --learning_rate 5e-4 \   --image_aug  False \   --wandb_project openvla142 \   --wandb_entity my \   --save_steps 1000

We save LoRA checkpoints by merging the LoRA weights into the model backbone. So you could try simply loading the last checkpoint's weights right after creating the model, here: https://github.com/openvla/openvla/blob/46b752f477cc5773cc1234b2e82c0e2130e4e890/vla-scripts/finetune.py#L164
Our finetuning code currently does not support saving of the optimizer state though, so it's a bit experimental, but could work.


Added oauth2 token exchange functionality in command line

General Checkups

[x] Have you checked that there aren't other open pull requests for the same issue/update/change?
[ ] If your code includes new features and not just bug fixes/copy editing: Did you include new tests?

Description

Related Issue

Example

Hi @madprime , @gedankenstuecke , @mcescalante , I have added token exchange functionality to be directly used through command line. Please suggest some way to solve the codeclimate issue. Thanks :)

@gedankenstuecke here is an example of the output
{'expires_in': 36000, 'scope': 'wildlife write read open-humans american-gut pgp go-viral', 'token_type': 'Bearer', 'access_token': 'ACCESS_TOKEN', 'refresh_token': 'REFRESH_TOKEN'}
If any change in the format is required, I will be happy to change it!

Ah, so it's a regular python dictionary that's printed? It could maybe be run through json.dumps(dict) to make sure it's a valid json, that might be more useful downstream. What do you think @mcescalante?

Hi @mcescalante adding sphinx documentation to click commands requires additional modules. I have added that in my other PR 59. Once that is merged, it will be easier for me to add the docstrings here. :)

@DartingMelody #59 is merged and I've merged master into this PR, so feel free to git pull and add the docstrings now!

Hi @mcescalante , I have added the docstring please review :)

Works for me! I'm going to merge now, thank you @DartingMelody :smile:


Module lineup list

Contains
Reorganization and updating of docs/modules.md
Cross referenced list in the omega properties with module lineup in docs/modules.md.
Updated module list.
Added Other Module subsection.
Added Tutorials section.
I don't have much experience with git or documentation so criticism / thoughts welcome.
Refs #4040

I really like this organization to the the README I'm sure @Cervator would have some comments on this.

I really like this organization to the the README I'm sure @Cervator would have some comments on this.

I mostly just moved things around and copied descriptions from the individual modules.
Few questions I had:
Should archived modules be left in the secondary list?
Should all Tutorial repos be listed here?

IMO we don't need the archived modules in the secondary list.
And I'm in favor of listing all tutorial repos there, yes. Didn't check, which are missing?

And I'm in favor of listing all tutorial repos there, yes. Didn't check, which are missing?

I think all these:
TutorialTelemetry
TutorialPathfinding
TutorialProfiling
TutorialGroups
TutorialEventsInteractions
TutorialEntitySystem
TutorialMultiplayerExtras
TutorialI18n
TutorialBlockFamily
TutorialSnakeTournament
TutorialParticleSystem

Okay, then I would say the following should definitely be listed:
TutorialPathfinding
TutorialParticleSystem
TutorialEntitySystem
TutorialEventsInteractions
TutorialMultiplayerExtras
TutorialBlockFamily
TutorialI18n
TutorialTelemetry (needs updating now that we moved it to k8s @agent-q1 )
TutorialProfiling
I'm not so sure about the following... opinion @Cervator ?
TutorialGroups
TutorialSnakeTournament

@devindoherty From the conversation it looks like this PR would be ready to merge when the following tutorial modules are listed as well:

TutorialPathfinding
TutorialParticleSystem
TutorialEntitySystem
TutorialEventsInteractions
TutorialMultiplayerExtras
TutorialBlockFamily
TutorialI18n
TutorialTelemetry (needs updating now that we moved it to k8s @agent-q1 )
TutorialProfiling

We'd highly appreciate if you could add those, will gladly merge then 🙃

We'd highly appreciate if you could add those, will gladly merge then 🙃

Should all be added.


Publish website to branch

Contributor Checklist

[ ] Did you add Scaladoc to every public function/method?
[ ] Did you add at least one test demonstrating the PR?
[ ] Did you delete any extraneous printlns/debugging code?
[ ] Did you specify the type of improvement?
[ ] Did you add appropriate documentation in docs/src?
[ ] Did you state the API impact?
[ ] Did you specify the code generation impact?
[ ] Did you request a desired merge strategy?
[ ] Did you add text to be included in the Release Notes for this change?

Type of Improvement

API Impact

Backend Code Generation Impact

Desired Merge Strategy

Release Notes

Reviewer Checklist (only modified by reviewer)

[ ] Did you add the appropriate labels?
[ ] Did you mark the proper milestone (Bug fix: 3.4.x, [small] API extension: 3.5.x, API modification or big change: 3.6.0)?
[ ] Did you review?
[ ] Did you check whether all relevant Contributor checkboxes have been checked?
[ ] Did you do one of the following when ready to merge:

[ ] Squash: You/ the contributor Enable auto-merge (squash), clean up the commit message, and label with Please Merge.
[ ] Merge: Ensure that contributor has cleaned up their commit history, then merge with Create a merge commit.

@mwachs5 this is a PR against a branch so I'm guessing you don't want any review? Or what's the intent here?

the test is what happens when I merge to the website-branch, so I am gonna merge this, and then it should push to the gh-pages

Well, the automation worked to try, but it massively failed for some reason https://github.com/chipsalliance/chisel3/actions/runs/4009946484

Ok, the re-trigger passed but I guess because it was a manual trigger it did not work to push to the gh-pages branch https://github.com/chipsalliance/chisel3/actions/runs/4009946484/jobs/6886077173


Fixes again

Some stuff to fix most of the current blocking issues.
@Spuds can you have a look at ebc46342082554c785b8b7372341079915c41886 and see if it actually fixes the problem? ;D

Seems to have fixed the admin search management actions, however actually performing a search results in various fatal errors.
One was in search.controller
$this->_search = new Search(); changing that to $this->_search = new \ElkArte\Search\Search(); That results in ElkArte\Search\API\Errors' not found in /sources/subs/Search/API/SphinxqlSearch.class.php on line 198 which is the static call to Errors::instance()->fatal_lang_error but its now looking inside the search namespace

hmm... my search miss the Search instantiation... dunno why...
I'll add it! :D
For the error it's "just" the matter of adding a \ in front of the class:
\Errors::instance()->fatal_lang_error

I'll check that too (and all the other classes inside them.

One was in search.controller

In addition to the already changed at https://github.com/elkarte/Elkarte/pull/2277/files#diff-4c33a887d2de4691b03e11decc3f85e4L164 ?
Where?

@spuds the last commit should fix the errors reported.

Then lets give it a try !


fix: do not override openedx docker image

Description
This PR makes to not override the openedx docker image by default, and only updates it in CI.

Thanks for the pull request, @Ian2012!  Please note that it may take us up to several weeks or months to complete a review and merge your PR.
Feel free to add as much of the following information to the ticket as you can:

supporting documentation
Open edX discussion forum threads
timeline information ("this must be merged by XX date", and why that is)
partner information ("this is a course on edx.org")
any other information that can help Product understand the context for the PR

All technical communication about the code itself will be done via the GitHub pull request interface. As a reminder, our process documentation is here.
Please let us know once your PR is ready for our review and all tests are green.

@Ian2012 🎉 Your pull request was merged!  Please take a moment  to answer a two question survey so we can improve your experience in the future.


Adding AD groups as members in various places

In some places in SharePoint you can't add groups, only users.
This is an administrative pain in the ***.
Best case would be to have SharePoint to allow groups and not only users for different administrative rights and permissions.
In the mean time one could implement a pattern for SharePoint DSC resources.
Every resources that handles administrative rights and permissions could have a flag, like "ResolveGroups = $true"
Then when DSC runs it could resolve all groups recursively to user.
Good or bad idea? But I'd like it.
If you 'd like I can participate in development of this if you find it is a solid idea.
BR // Ilian

Hi @IlleNilsson I like your idea. Just wondering in what circumstance this would be useful. Do you have an example of a place where you can't use groups?

Hi @IlleNilsson, could you please share a little more information for this issue/idea? Right now I don't know what you mean exactly, which makes resolving this issue quite hard :wink:

Closing this issue due to inactivity. We need more information to investigate this issue.
Please reopen the issue if this is still relevant and provide the requested information.


Migrate documentation for the component framework

I moved this directory into the new documentation framework whilst fixing formatting and typos. I split the directory into guidelines, architecture pages, and one how-to document.

Created DOCS-7394 for docs review.


Implementatin mpsks clients

Content
Sipa implementation for mpsk clients so users can create delete and change them on there own
Please make sure though you did the following (tick off everything you
already did):

[x] Run the tests and see them pass
[x] Rebase your branch on top of develop
[ ] Include tests for features you introduced / bugs you fixed

I can't enter the usersuite unfortunately:
CRITICAL 2024-12-27 11:56:04,613 sipa.blueprints.generic Backend error: pycroft
Traceback (most recent call last):
  File "/tmp/sipa/sipa/model/pycroft/user.py", line 44, in __init__
    self.user_data: UserData = UserData.model_validate(user_data)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/sipa/venv/lib/python3.12/site-packages/pydantic/main.py", line 503, in model_validate
    return cls.__pydantic_validator__.validate_python(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for UserData
mpsk_clients
  Field required [type=missing, input_value={'birthdate': '1111-11-11..., 'wifi_password': None}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.4/v/missing

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/tmp/sipa/venv/lib/python3.12/site-packages/flask/app.py", line 880, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/sipa/venv/lib/python3.12/site-packages/flask/app.py", line 865, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/sipa/sipa/blueprints/generic.py", line 143, in login
    user = User.authenticate(username, password)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/sipa/sipa/model/pycroft/user.py", line 78, in authenticate
    user = cls.get(result['id'])
           ^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/sipa/sipa/model/pycroft/user.py", line 57, in get
    return cls(user_data)
           ^^^^^^^^^^^^^^
  File "/tmp/sipa/sipa/model/pycroft/user.py", line 47, in __init__
    raise PycroftBackendError("Error when parsing user lookup response") from e
sipa.model.pycroft.exc.PycroftBackendError: Error when parsing user lookup response

interesting yesterday it worked for me will check

just forgot to push a naming change sorry


"Base schema exists but missing which project it belongs to"

I'm trying to set up the connection, I've been able to troubleshoot myself so far but now I get above error message.
What does this error message entail? It's unclear to me what the 'project' is, it's referring to.

So the problem is within the DB_SCHEMA_MODIFIED and DB_SCHEMA_BASE I'm assuming. What should I do if I do not want to create new schema's but want to just update an existing table within my public schema based on a GeoPackage that's already active within Input/Mergin?

Hi. I have the same problem, I can't configure an existing schema to start with. I saw that the issue was labeled as bug. If it helps I could do some test.

Assuming that you have your database tables in a schema, let's call it "my_data", then this is the value you should use for DB_SCHEMA_MODIFIED. The value for DB_SCHEMA_BASE should be a new schema name that db-sync will create and maintain by itself (to keep the sync working correctly). You could call it e.g. "my_data_db_sync_internal":

DB_SCHEMA_MODIFIED=my_data
DB_SCHEMA_BASE=my_data_db_sync_internal

Does that make sense? (sorry the naming is somehow confusing)

Your base schema should not exist at the time you initialize db sync - when you delete it, things should work... (and after the init you should not modify anything in the "base" schema as it would break the sync mechanism).

I've created the schema by hand because I've this previous error
File "/mergin-db-sync/dbsync.py", line 257, in _set_db_project_comment
cur.execute(query.as_string(conn), (json.dumps(comment), ))
psycopg2.errors.InvalidSchemaName: schema "base" does not exist

Problem seem partially to due to starting data. If I change my real table with a very simple test table the base schema was created successfully, but this error remains [0] .
I also hav this comment for the base schema {"name": "admin/ha", "version": "v0", "error": "Initialization of db-sync failed due to a bug in geodiff"}
[0]
File "dbsync_daemon.py", line 65, in
main()
File "dbsync_daemon.py", line 37, in main
dbsync.dbsync_init(mc, from_gpkg=False)
File "/mergin-db-sync/dbsync.py", line 522, in dbsync_init
changes_gpkg_base = _compare_datasets("sqlite", "", gpkg_full_path, config.db_driver,
File "/mergin-db-sync/dbsync.py", line 211, in _compare_datasets
_geodiff_create_changeset_dr(src_driver, src_conn_info, src, dst_driver, dst_conn_info, dst, tmp_changeset)
File "/mergin-db-sync/dbsync.py", line 203, in _geodiff_create_changeset_dr
_run_geodiff([config.geodiff_exe, "diff", "--driver-1", src_driver, src_conn_info, "--driver-2", dst_driver, dst_conn_info, src, dst, changeset])
File "/mergin-db-sync/dbsync.py", line 153, in _run_geodiff
raise DbSyncError("geodiff failed!\n" + str(cmd))
dbsync.DbSyncError: geodiff failed!
['/geodiff/build/geodiff', 'diff', '--driver-1', 'sqlite', '', '--driver-2', 'postgres', '****', '/tmp/dbsync/ha.gpkg', 'base', '/tmp/clTQBxZY']

Ok. The last problem was resolved by add some data to the test table. So the main problem remain my real data (may be some void table ?).

@lucalanteri That's strange that the copy failed when you had zero rows :thinking:  ...would it be possible to provide a dump of your schema in Postgres in the state when it causes the error during db-sync init? I'd like to see if we can find the cause (and fix it). If your dump is sensitive, please feel free to send it by email: martin.dobias at lutraconsulting.co.uk
If I can replicate the bugs, I would raise tickets as needed (in db-sync repo or in geodiff repo).

Hi Martin, after some other tests it seems that changing data type from numeric(10,2) to real resolve also the problem of void table.
Thanks and sorry for the noise

I'm experiencing similar issue as above:  trying to initialize from db but get this error...
Status: Downloaded newer image for lutraconsulting/mergin-db-sync:latest
== starting mergin-db-sync daemon == version 1.1.2 ==
Logging in to Mergin...
Processing Mergin Maps project 'Cleveland-Metroparks/nr_test'
Connecting to the database...
Downloading latest Mergin Maps project Cleveland-Metroparks/nr_test to /tmp/dbsync/nr_test
Traceback (most recent call last):
File "dbsync_daemon.py", line 63, in
main()
File "dbsync_daemon.py", line 35, in main
dbsync.dbsync_init(mc, from_gpkg=False)
File "/mergin-db-sync/dbsync.py", line 666, in dbsync_init
init(conn, mc, from_gpkg)
File "/mergin-db-sync/dbsync.py", line 625, in init
raise DbSyncError(f"The output GPKG exists but the base schema is missing: {conn_cfg.base}")
dbsync.DbSyncError: The output GPKG exists but the base schema is missing: sync_base
But when I create the base schema (and add {"name": "admin/db-sync", "version": "v0"} comment to the schema)  and try to initialize, I get the following:
Status: Downloaded newer image for lutraconsulting/mergin-db-sync:latest
== starting mergin-db-sync daemon == version 1.1.2 ==
Logging in to Mergin...
Processing Mergin Maps project 'Cleveland-Metroparks/nr_test'
Connecting to the database...
Modified and base schemas already exist
Downloading version v0 of Mergin Maps project Cleveland-Metroparks/nr_test to /tmp/dbsync/nr_test
There are pending changes on server, please run pull command after init
Traceback (most recent call last):
File "dbsync_daemon.py", line 63, in
main()
File "dbsync_daemon.py", line 35, in main
dbsync.dbsync_init(mc, from_gpkg=False)
File "/mergin-db-sync/dbsync.py", line 666, in dbsync_init
init(conn, mc, from_gpkg)
File "/mergin-db-sync/dbsync.py", line 627, in init
raise DbSyncError(f"The base schema exists but the output GPKG exists is missing: {gpkg_full_path}")
dbsync.DbSyncError: The base schema exists but the output GPKG exists is missing: /tmp/dbsync/nr_test/cm_wetlands.gpkg
How do I troubleshoot this last problem?


Download credentials on env if not available

Follow up to #52, relies on NewCredentialsCommand for building the context and help, followed by calling out to carina.Download(pc).
One minor nit is that the help lists the path arg as "path to write credentials out to". Any idea what I should change this to or should we just have a separate setup for the env command (like before)?
This now creates a separate command that doesn't do Auth (should work without a remote call to the API - yay fast :bullettrain_side:). If the user doesn't have credentials downloaded, it will try to Auth then Download.
/cc @minrk

TODO: Also make sure we don't Auth when using carina env.

Apparently I can't overwrite a previous PreAction with kingpin. Had to create a wholly new command for this instead of using NoAuth.

Awesome, thanks.

I'm guessing this is one of the style nits you pointed out in #52, but why is EnvCommand not a new type like all the others? (#52 was my first PR in go, so I'm not worried about having done things all kinds of wrong). Is it so it inherits the Download action?

Actually, the only nits were in 9d9737628533f576aa12f7074f590f223d2a5581 which I just did directly after merging your PR. Primarily:

camelCase variables
lined up parameters in structs
comments preceding exported functions start with the function name

The only reason that EnvCommand is not a new type is that the variables stored as part of the overall context are the same as the credentials command. There may be a friendlier way to write that; it was just convenient to do embedded types of commands, building up from those that held similar contexts.


Upgrade to SpatialOS SDK 13.6.0

After #24, need to then upgrade to 13.6.0! Has some minor changes to KCP and some schema compiler semantic changes.

Fixed an issue where load_all_schema_on_schema_path would only validate schema. The argument will now generate code for any .schema files in the schema_path without them having to additionally be passed in as input files. You can still only validate schema by using load_all_schema_on_schema_path without any *_out arguments.

Oh terrific! This should mean we no longer need to manually search for schema files and pass them into the schema compiler, right? Should simplify the logic in cargo-spatial 😁


Can't open documentation website

Hey, today I wanted to take a look into the documentation of erela.js but the domain to the website doesn't work anymore. Looking around I found that you are working on a new version of erela.js and that you have a new website. Could you by any chance reupload the old erela documentation?

solaris announced docs are back up on his  new domains https://projects.solaris.codes/erelajs/ https://projects.solaris.codes/erelajs-rewrite/


[Snyk] Security upgrade python from 3.9-slim to 3.13.0rc2-slim

Snyk has created this PR to fix 5 vulnerabilities in the dockerfile dependencies of this project.
Keeping your Docker base image up-to-date means you’ll benefit from security fixes in the latest version of your chosen image.
Snyk changed the following file(s):

cmd/suggestion/hyperopt/v1beta1/Dockerfile

We recommend upgrading to python:3.13.0rc2-slim, as this image has only 40 known vulnerabilities. To do this, merge this pull request, then verify your application still works as expected.
Vulnerabilities that will be fixed with an upgrade:

Issue

Integer Overflow or Wraparound SNYK-DEBIAN12-EXPAT-7855502

Integer Overflow or Wraparound SNYK-DEBIAN12-EXPAT-7855503

XML External Entity (XXE) Injection SNYK-DEBIAN12-EXPAT-7855507

Integer Overflow or Wraparound SNYK-DEBIAN12-ZLIB-6008963

Integer Overflow or Wraparound SNYK-DEBIAN12-KRB5-1549480

[!IMPORTANT]

Check the changes in this PR to ensure they won't cause issues with your project.
Max score is 1000. Note that the real score may have changed since the PR was raised.
This PR was automatically created by Snyk using the credentials of a real user.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information:
🧐 View latest project report
📜 Customise PR templates
🛠 Adjust project settings
📚 Read about Snyk's upgrade logic

Learn how to fix vulnerabilities with free interactive lessons:
🦉 XML External Entity (XXE) Injection

Pull Request Test Coverage Report for Build<PHONE_NUMBER>7
Details

0 of 0   changed or added relevant lines in 0 files are covered.
2 unchanged lines in 1 file lost coverage.
Overall coverage increased (+0.1%) to 72.974%

Files with Coverage Reduction
New Missed Lines
%

pkg/controller.v1beta1/trial/trial_controller.go
2
70.83%

Totals

Change from base Build<PHONE_NUMBER>:
0.1%

Covered Lines:
3116

Relevant Lines:
4270

💛  - Coveralls


Add support for AAC codec in webviews to read MP4 audio correctly

Currently MP4 videos with an AAC audio codec don't play audio in VSCode.
AAC is currently the most popular audio codec, it is the default on many softwares like OBS and other capture softwares.
As such it is used in screen recordings  for documentations in many projects.
Supporting AAC in webviews would allow the user to preview documentation .md files containing videos. And also allow the user to use mp4s in python notebooks (eg: Machine learning / data science applications).
Currently it is impossible to make a video readable both by MacOS (QuickTime) and VSCode because there is no overlap in the supported audio formats. Supporting AAC is essential to make sure that the documentation is readable on all platforms.
VSCode includes FFMPEG as a dependency and it already includes the necessary AAC codecs. This proposal will not include new dependencies but only enable codecs already present in VSCode through FFMPEG.

Useful link: Release Notes: supported audio and video formats
After reading the release note above, I am curious as to why this hasn't been fixed at the same time.

Duplicate of https://github.com/microsoft/vscode/issues/162062
But we can keep this one open to track user requests.
In short - we can not ship AAC as part of VS Code because we did not get approval from our Lawyers due to some codec rights and fees.

AAC is the audio format used in most MP4 video files so would really help if it can be supported.
If it cannot be supported in core vscode due to legal reasons then why not have one of the devs either release an extension with code that does or release it as a mp4 preview extension as vscode support having multiple video previewers.
That way the extension does not have to be released under Microsoft. There are other Microsoft developers like lostintangent and mattbierner who release popular extensions in their own name.
There is currently another mp4 preview extension but doesn't support AAC due to webview audio restrictions
https://marketplace.visualstudio.com/items?itemName=analytic-signal.preview-mp4
Not sure if the legal limitation is the webview itself that needs support for AAC.

As I understand it from the replies and other duplicate issues. It's not that there definitely is a legal problem with using AAC. It's that the dev team didn't get a response from their legal department. That means that for the time being, this issue is on Microsoft's side.
That said I don't think anyone here is opposed to a workaround. But it would be wise to first check if there is a problem before trying to solve it.

Not just a problem with .mp4 video files, this seems also to prevent playback of AAC audio files in browser views, internal live preview, etc. Any solution should fix audio-only playback and decoding.

Impossible to get full demand as issues are just closed with "as-design" as reason. Some issues had as many as 51 upvotes before closed.
#158972
[#167685]](https://github.com/microsoft/vscode/issues/167685)
#179385
#72280
#162062
#187525

Is this the same reason why if I embed an mp4 video in the vscode editor using the Custom CSS and JS loader extension the video won't play past the first frame if videoElement.muted is not set to true?


Allow layers control section to resize to contents

Description

Set min-height and min-width to the empty widget used by the layers controls container widget instead of the widget itself to allow it to resize dependending on the controls being shown
Preview

Type of change

[x] Bug-fix (non-breaking change which fixes an issue)

References

closes #2337
closes #5472
How has this been tested?

[x] all tests pass with my change
[x] I check if my changes works with both PySide and PyQt backends
as there are small differences between the two Qt bindings.

Final checklist:

[x] My PR is the minimum possible work for the desired functionality

Not sure what's up with Windows CI, but I've re-run it twice and it's the same, so I think it's not random?
https://github.com/napari/napari/actions/runs/3905492648/jobs/6674079073#step:8:221
napari\_qt\_tests\test_qt_viewer.py .........................ss......... [ 80%]
  ss...Windows fatal exception: access violation

I tried bumping windows CI to python 3.9 (maybe we should do that anyways, since that's the version we suggest in installation?) but it didn't help.
https://github.com/psobolewskiPhD/napari/actions/runs/3919686014/jobs/6700847765
Can we assume from the 3 dots after skip:

That the last 3 tests completed, so it's the first of the next test file?
I'ma try that that in my branch.

could you try to run a verbose test (to find failing job).
It will be also nice to try to run only napari\_qt\tests\test_qt_viewer.py to check if fils are related only to these tests.

I got pyqt to pass with +1 skip (def test_surface_mixed_dim(make_napari_viewer):
https://github.com/psobolewskiPhD/napari/actions/runs/3920033829/jobs/6701406395
But now pyside is still cranky 😡

Ok, I ended up having to skip 3 tests plus the octree.
https://github.com/psobolewskiPhD/napari/actions/runs/3920450156/jobs/6702331229
The octree one fails and gives a similar error from the subprocess, so I think it's related, but 🤷♂

Should I try checking if reverting the min-height change in the qss file helps with the failing checks?

Should I try checking if reverting the min-height change in the qss file helps with the failing checks?

I guess? hard to imagine that being the culprit, but your commit before mu suggestion seems to have flown by. However, my fork branch is using the 295 value and still shows the same CI issues.

We also have some flaky tests that w're slowly gettin grid of, so some failures are unfortunately fake :/

Note: Re-runs of the bundle workflow jobs for failures to upload artifacts probably due to GitHub Actions degradation status. See https://github.com/actions/upload-artifact/issues/383 and https://www.githubstatus.com/incidents/n18j5g7wpg0s

Note: seems like the skipped tests are failing due to an access violation error on Windows when Vispy code is creating/managing the OpenGL related context/objects?

Yeah those failures are familiar... Very annoying, sorry about that :/

Before merging, how do others feel about these skipped tests? Should we be more responsible and try to solve the problem instead of shoving it under the rug, or are we happy like this? @Czaki @andy-sweet tagging you since the three of us particularly have fought with these flaky tests before...

I do not understand how these changes may reuires so many changes in tests.

I do not understand how these changes may reuires so many changes in tests.

My assumption is that the skipped tests have underlying issues that are independent of the implementation changes in this PR that mean they fail sometimes. The implementation changes here affect the layout, which in turn affects drawing the widgets, which is something that could be relatively slow on CI runners. My other more speculative guess is that the Windows runners might be a bit slower in this regard and that's the reason we often skip on Windows CI only rather than there being an actual Windows issue.
We already have quite a lot of skips, so I have no issue with the ones added here, especially given the fix is simple and desirable.

Ok, I guess I'll merge then. Let's hope that as these issues keep showing up, we can figure out exactly how to prevent them :)


Import all civil data from IFC

Description 📝
Including alignment, sections, etc.
Suggested solution 💡
No response
Alternative ⛕
No response
Additional context ☝
No response
Validations ✅

[X] Read the docs.
[X] Check that there isn't already an issue that requests the same feature to avoid creating a duplicate.

Done!


Updated the deprecated links in contribute/desktop code-signing.md

Summary
Updated the deprecated links in contribute/desktop code-signing.md
Ticket Link

NA

Hello @Rutam21,
Thanks for your pull request! A Core Committer will review your pull request soon. For code contributions, you can learn more about the review process here.

@cwarnermm Please review and add the necessary labels for Docathon.

Thank you for your PR. The existing links are working as expected and don't need to be changed. As a result this PR will not be merged. :)


K8s AuthZ Reactor

The idea is to be able to specify a git repo to m8 where m8 itself then puts RBAC yamls which can be applied to customer clusters by their method of choice (ArgoCD, Flux, …).
This reactor creates the necessary yaml from the user/rolebding/tenant/cluster all the stuff events and reconciles the yaml in the external git repo. Additionally it can reconcile OPA Gatekeeper rules for namespaces and other resources.

Coverage remained the same at 56.164% when pulling a368e0056bee0d290683e047100f230b0725d5c9 on feature/k8s-git-repo-reactor into 54fededb5584b113686f8059e7e935385e75f0e2 on main.


Hotfix/base href

This is the second time I've made this mistake.  I'll have to write up some docs on how to compile for the gh-pages

The base href is easy to forget.


Improvement weightClass check

Description
Improvement of com.typenetwork/check/usweightclass

Great!
Please, also add an entry to changelog


adding top-down error oversight

adding top-down error oversight along with the data file required for this and the test file. I shall add some other test file after some time. I need some suggestion regarding the oversight suggestion string.

Overall logic looks good.


Type inference fails in ark-gl

Specifically here:
pub func (g: ^Game) render() {
	mut i := uint(0);
	for i < g.entities.getLength() {
		entity: ^mut entity::Entity = g.entities.get(i);
		match entity {
			some(e) => {
				e.render();
			},
		}
		i += 1;
	}
}

I get this error:
error: [game:45:5] Unable to infer type of member `render` on type `$104`
                e.render();
                ^

Most likely related to/dupe of #706


Fix Chilean Currencies

There are two Chilean Currencies, one is CLF also know UF, mostly used for real states, it is a currency with inflation included. The other currency is CLP, Chilean Peso. the currency used daily by Chilean people.

Author is unwilling to sign CLA.  We will fix this more generally via #42

Author is unwilling to sign CLA.  We will fix this more generally via #42


Question about modifying a node

Is there an easy way to modify a node after it's parsed? I ask this because nodes contain childNodes, parentNode, parentElement, _ancestors, etc. Do i need to relink all those things or there is a method/way to replace one node with another?

Yes, Deno DOM implements replaceWith https://developer.mozilla.org/en-US/docs/Web/API/Element/replaceWith

Yes, Deno DOM implements replaceWith https://developer.mozilla.org/en-US/docs/Web/API/Element/replaceWith

i didn't know it! big thanks!!!!!


WIP: CheckedFunction utils

This is an example of utils which I would like to create. Purpose of interfaces like this is to simplify usage of java streams

Coverage decreased (-0.01%) to 95.221% when pulling 3bf5df5013c24523c5d3a2c33a3748c0d3e38201 on asciborek:LANG_1424 into 60b8f18e463be4673732e2a31867a7c3b02a4a1e on apache:master.


MudSplitter questions

Hello,
Maybe this has been asked and answered before, but I was not able to find those answers.
I'm trying to achieve a sort of file explorer type layout behavior (one content is locked while the rest fills the space) and I was wondering if it's possible with MudSplitter or not.
What I'm trying to achieve is the following:

The StartContent has a starting/initial size in pixels (or some other unit) and the EndContent fills the remaining area.
The StartContent size should remain the same if the browser window is resized.
The user can use the splitter to change the size/dimension.
The Start- and EndContent should have a minimum size set which prevents the splitter bar from moving all the way to the left or right.

The only property I could find to set an initial size was Dimension, however that is percentage based and thus means different things depending on the browser window.
So, is it possible to set initial pixel sizes or placement of the divider, and lock the StartContent so it doesn't resize when the browser window changes in size etc.?

Nice idea. But splitter controlled with a slider, I think it's not currently possible to set pixels. But you can use resize observer and set StartContent percentage manually when user resize the screen.
Setting min and max value is possible i think. We already added min and max for the MudRangeSlider.

I'm guessing that what I'm trying to do could be seen as an anti-pattern or anti-design as it is the opposite of being flexible/responsive with the sizes, sort of the whole point of flexbox.
Regardless, I was able to find a library that had implemented this and if anyone is looking for the same and stumbles on this point then I'm using this:
https://github.com/jsakamoto/Toolbelt.Blazor.SplitContainer
It allows me to set a fixed size for one of the containers (as well as minimum etc.) and let the rest adapt/fill the remaining spaces.
It would however be nice to have the same features on MudSplitter if anyone has the knowledge or experience to be able to. I am fairly new with Blazor so I feel it's safer to try and use what is out there to start with.

The main idea of MudSplitter is make a non-js and smooth solution. We can think what we can do about pixel sizing.


Fixes the slowdowns and bugs caused by the prior improved compute practices, but requires a nested tensor package.

This should fix #73 as well

@mmcdermott what polars and numpy version were used for this branch?
I tested this building the dataset works fine.
When running pretrain, I initially got a runtime error here:
https://github.com/mmcdermott/EventStreamGPT/blob/29c3b9f53f8b1d1e16188f81a18face2d1c4adce/EventStream/data/pytorch_dataset.py#L253-L255
The error being:
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (55877,) + inhomogeneous part.
When I change the numpy array dtype to object, it just hangs (it runs for an incredibly long time, and I haven't been patient enough to wait it out). I don't know whether safetensors or the nested_ragged_tensors are to blame (I assume the latter). This occurs when trying to run pretrain, sampling about 90,000 subjects from a full dataset of about 600,000.

@juancq , definitely don't feel obliged to wait it out when setting it to an object; that will be very slow and defeats the purpose of this change. One question, though; in your data do you have reason to believe that your subjects will have varying numbers of observations of static measurements per row? I believe that what is going on here is that something that is treated as a dense tensor is in reality a ragged one.

@juancq , definitely don't feel obliged to wait it out when setting it to an object; that will be very slow and defeats the purpose of this change. One question, though; in your data do you have reason to believe that your subjects will have varying numbers of observations of static measurements per row? I believe that what is going on here is that something that is treated as a dense tensor is in reality a ragged one.

I don't follow, what would be an example?
This branch also has parts that break with certain polars versions. This makes it hard to test because I don't know which exact polar version to use.

@mmcdermott I have sorted out polars issues and have gotten further testing this.
My pretraining now hangs here:
https://github.com/mmcdermott/EventStreamGPT/blob/fa33387247f43e760d11c117c3eec5a983778f2d/EventStream/data/pytorch_dataset.py#L161
When I kill the script, the stack trace is something along the following lines (in ragged tensors):
get__item https://github.com/mmcdermott/nested_ragged_tensors/blob/b3a01a661e73e9c55541430dbb985c51d054fa4d/src/nested_ragged_tensors/ragged_numpy.py#L398
vstack https://github.com/mmcdermott/nested_ragged_tensors/blob/b3a01a661e73e9c55541430dbb985c51d054fa4d/src/nested_ragged_tensors/ragged_numpy.py#L655
concatenate https://github.com/mmcdermott/nested_ragged_tensors/blob/b3a01a661e73e9c55541430dbb985c51d054fa4d/src/nested_ragged_tensors/ragged_numpy.py#L754
If this is too cryptic, let me know and I'll rephrase or can post an issue on nested_ragged_tensors repo.

Thank you @juancq -- I've been travelling quite a bit and been otherwise occupied for the last month and a half, but I'm trying to push a new major version of ESGPT that addresses these issues and the other memory issues. I assume your last comment still reflects the state of things with this change for you?

@coderabbitai review

Thank you @juancq -- I've been travelling quite a bit and been otherwise occupied for the last month and a half, but I'm trying to push a new major version of ESGPT that addresses these issues and the other memory issues. I assume your last comment still reflects the state of things with this change for you?

Yes, the state of things on my end are the same as of the time I wrote the last comment.

This is blocked by #104

@juancq I know this update is long overdue, but the recent pushes use the nested ragged tensor code in a new way that should dramatically reduce the CPU memory burden during model training. You'll need to re-build the base dataset object first to produce the right cached files (though I may write a conversion script to ease that cost, if that would be helpful) but once that is done this should, with minimal to no impact on throughput, set things up to only pull the patient data from disk as needed rather than loading it all in memory at all. Note this requires updating to the latest version of nested_ragged_tensors as well. If you try it and find it useful or find issues with it, I'd be very appreciative and curious of your findings!

@mmcdermott thanks for all the hard work. I tested this branch on my dataset. The previous bugs are gone. I am now seeing about a 7% runtime improvement per epoch and about 30% lower memory usage.

Fantastic! Thanks so much @juancq . I'll do some final testing just to make sure there are no issues and plan to merge this branch in soon. Glad this has resolved your issues and induced other improvements besides.


lay out goals of the project on the homepage

Sets out the goals of the project on the homepage - let me know if I've missed anything or what i've laid out don't align with the consensus.
Also added a quick acknowledgments section at the bottom.
Preview: https://c2edd824.peopleplus-components.pages.dev/
Closes #177
Closes #84

Can you update playwright_tests/test.spec.ts?  Ta.


Complete merged anndata docs

Closes #241 🎉
This PR should 🤞 wrap up the merged object docs page. I added sections for experiment metadata (uns), reduced dimensions, and CITE-seq considerations.
I also made a couple other small changes:

Gratuitous change to de-magrittr the spell check script
Some parallel changes about that discard column, as in #261
Fixed some relative links throughout the docs that were not properly formatted, mostly a bunch of places that were missing a closing >!
We missed a few spots in #258 to document adt_id (not adt_name!) in sce rowdata & anndata .var.

Tables have been purged!


[Core feature] Convert List[Any] to a single pickle file

TL;DR
Currently, flyte create N (size of the list) pickle files if the output type is List[Any]. This slows down serialization. it takes more than 15 mins to upload the pickles to s3 if the size of the list is 1000 (see Slack Thread)
This PR:

converts the entire list into a single pickle file by default.
takes batches of list items and uploads each batch as a separate pickle file if the user annotates the batch size(see below).

@task
def t1() -> Annotated[List[Any],  100]

Type

[ ] Bug Fix
[x] Feature
[ ] Plugin

Are all requirements met?

[ ] Code completed
[ ] Smoke tested
[ ] Unit tests added
[ ] Code documentation added
[ ] Any pending items have an associated Issue

Complete description
Originally, in ListTransformer.to_literal, Items are converted one by one. It is time-consuming, especially for large lists.
https://github.com/flyteorg/flytekit/blob/8ccb5dde044f9dce3e216a8cea58dd464fb3ab6b/flytekit/core/type_engine.py#L977
This PR checks if python_type is list[flytePickle]. If so, batch the list of items and call FlytePickleTransformer.to_literal.
Measure the performance
Test file:
from typing import List, Dict, Any
from flytekit import task, Resources, workflow

@task(
    limits=Resources(mem="4Gi",cpu="1"),
    disable_deck=True,
)
def test() -> List[Any]:
    return  [{"a": {0: "foo"}}] * 10000

@workflow
def wf():
    test()

if __name__ == "__main__":
    wf()

# yichenglu/flytekit:mytest5 contains this PR
# pyflyte run  --image yichenglu/flytekit:mytest5 --remote ./test.py  wf
# cr.flyte.org/flyteorg/flytekit:py3.10-1.4.0b1 is the default image
# pyflyte run  --image cr.flyte.org/flyteorg/flytekit:py3.10-1.4.0b1 --remote ./test.py  wf

In an 4VCPU, 16G RAM EC2 instance, measuring the performance in flyte cluster using the above test file and the commands：

Prior to these changes, , converting a list with type List[Any]  with 10000 items needs 157 seconds.
After this PR, convertinga list with type List[Dict[str, Any]]  with 10000 items only needs 7 seconds.

backward compatibility:

case1: old task's cached output -> new task

from typing import List, Dict, Any, Annotated
from flytekit import task, Resources, workflow
@task(
    limits=Resources(mem="4Gi",cpu="1"),
    disable_deck=True,
    container_image="{{.image.oldImage}}",
    cache=True, cache_version="1.0"
)
def task0() -> List[List[Any]]:
    return   [ ["foo","foo","foo"] ] * 2
@task(
    limits=Resources(mem="4Gi",cpu="1"),
    disable_deck=True,
    container_image="{{.image.prImage}}",
)
def task1(data: List[List[Any]]) -> List[List[Any]]:
    print(data)
    return  data
@workflow
def wf():
    data = task0()
    task1(data=data)
# pyflyte run --image prImage="yichenglu/flytekit:mytest4" --image oldImage="cr.flyte.org/flyteorg/flytekit:py3.10-1.4.1" --remote ./test.py  wf

Backward compatibility is preserved in this case. Can see pod log ton double check.

case2: new task(image include this PR) -> old task(image without this PR)

from typing import List, Dict, Any, Annotated
from flytekit import task, Resources, workflow
@task(
    limits=Resources(mem="4Gi",cpu="1"),
    disable_deck=True,
    container_image="{{.image.prImage}}",
)
def task0() -> List[List[Any]]:
    return   [ ["foo","foo","foo"] ] * 2
@task(
    limits=Resources(mem="4Gi",cpu="1"),
    disable_deck=True,
    container_image="{{.image.oldImage}}"
)
def task1(data: List[List[Any]]) -> List[List[Any]]:
    print(data)
    return  data
@workflow
def wf():
    data = task0()
    task1(data=data)
# pyflyte run --image prImage="yichenglu/flytekit:mytest4" --image oldImage="cr.flyte.org/flyteorg/flytekit:py3.10-1.4.1" --remote ./test.py  wf

[[['foo', 'foo', 'foo']], [['foo', 'foo', 'foo']]] will be printed if see the pod's log
This occurs because in Task0, the outputs are batched, while in Task 1, which does not include this PR, the results are not expanded.
While slightly impacting backward compatibility, it is justifiable to ask users to update their images for two primary reasons:

Before the implementation of this PR, users faced lengthy wait times for tasks involving pickling to be completed.
Only tasks with the type list[any] will be affected. In most scenarios, users can still utilize the old images or cached outputs.

Tracking Issue
https://github.com/flyteorg/flyte/issues/3207
Follow-up issue
NA

Prior to these changes, , converting a list with type List[Dict[str, Any]] with 1000 items needs<PHONE_NUMBER>37 seconds.
After this PR, convertinga list with type List[Dict[str, Any]] with 1000 items only needs 2.990736610998283 seconds.

Amazing, thanks for testing it

thanks! can we also do a quick end to end test please on sandbox if you haven't already

Certainly! May I know which test you are referring to?  I have tested the following two cases(shown in the PR description):

old task's cached output -> new task
new task(image include this PR) -> old task(image without this PR)

Do you mean to add cases that demonstrate successful execution?

I think with the Annotation functionality I added in #1615 you can accomplish this much more clearly and cleanly without bringing the functionality into TypeEngine, where it adds a lot of complexity, and doesn't even really provide this feature in many cases (for example what if I want to batch types which are already registered in the TypeEngine).
BatchedList = Annotated[List[Any], FlytePickleTransformer()]

@task
def my_task(...) -> BatchedList:
   ...

Or even using a custom transformer with a fixed batch size, and explicit type;

class BatchedList:

    def __class_getitem__(self, type: Type, batch_size: int):
         return  Annotated[List[type],  MyBatchingTransformer(t=type, batch_size=int)]

@task
def my_batching_task(...) -> BatchedList[int, 5E10]:
   ...

@Yicheng-Lu-llll Does this meet your needs?
If so perhaps we can revert this as it adds significant complexity. @wild-endeavor
I'm sorry I didn't catch this before I went in, the Annotation override feature is newer and not well documented.

Hi @elibixby, Thank you for commenting! We can further discuss about this.

I think with the Annotation functionality I added in https://github.com/flyteorg/flytekit/pull/1615 you can accomplish this much more clearly and cleanly without bringing the functionality into TypeEngine, where it adds a lot of complexity, and doesn't even really provide this feature in many cases (for example what if I want to batch types which are already registered in the TypeEngine).

The primary goal of this PR is to address the performance issues. Some users find it is extremely slow if they have large list with type Any.  This issue is not apparent with other types. Hence, this PR focuses on batching the List[Any] type to enhance performance, while leaving all other types unaffected.

In my initial implementation,I also change the behavior of get_transformer,  However, as you pointed out, , this approach will lose the ability to change the batch size. From a user's perspective, creating a custom transformer to support batch size can be overly complex. Also, adding batch size to FlytePickleTransformer would introduce equivalent complexity.

But if we do not care about the batch size, I do strongly agree your approach is much much better. But if we choose to disregard batch size and proceed with your suggestions , more modifications will be required beyond #1615. For example, We need to make sure translate_inputs_to_literals and TypeEngine are consistent.

In my initial implementation,I also change the behavior of get_transformer,  However, as you pointed out, , this approach will lose the ability to change the batch size. From a user's perspective, creating a custom transformer to support batch size can be overly complex. Also, adding batch size to FlytePickleTransformer would introduce equivalent complexity.

I agree if we were to use annotations, we should provide a convenience wrapper for batch size to users, so they don't need to implement their own type transformer. I disagree that it would introduce equivalent complexity to this PR. It would be a completely isolated subclass of FlytePickleTransformer.  (as a side note I am working on a PR right now which alters the way FlytePickleTransformer is used to be a default in get_transformer rather than a special case only for function signatures as implmented in #1565 )

If we do not care about the batch size, I do strongly agree your approach is much better. But even we choose to disregard batch size and proceed with your suggestions, more modifications will be required beyond https://github.com/flyteorg/flytekit/pull/1615. For example, We need to make sure translate_inputs_to_literals and TypeEngine are consistent.

This is something I've discussed pretty extensively with @wild-endeavor . I think the preferred solution here is to make it so that adding a TypeTransformer by itself (either through registration or annotation), is sufficient to fully the support the type in Flyte.
The main thrust of this is in supporting pyflyte run (as you point out), but it could fairly easily be supported by adding a from_json to the TypeTransformer base class which pyflyte uses to unmarshal yaml data.
Would you be open to a solution that addresses both of these issues through an Annotated type as I've just described?

I disagree that it would introduce equivalent complexity to this PR. It would be a completely isolated subclass of FlytePickleTransformer

If no batch size is specified, when a list comes in, the FlytePickleTransformer can simply transform the entire list into a single pickle file. However, with a batch size, things may be different. For example, we can split the list into three, and then convert them into three separate pickle files. But then, how to store and handle it as a literal seems to be a problem. Because it appears we have to change the literal type (from blob to LiteralCollection), otherwise we can not simply deal with the three separate pickle files. This is why I said "may introduce equivalent complexity to the current FlytePickleTransformer". It would make things easier to handle the batching process within the ListTransformer, which naturally supports the LiteralCollection.

Would you be open to a solution that addresses both of these issues through an Annotated type as I've just described?

Overall make sense to me. But I am not sure how to deal the above concern .


Unable to add .dev domain

I seem to be unable to add my website URL (https://mxb.dev) to bridgy for twitter polling. After accepting the app on twitter, I can't enter the correct URL, the screen always says that "mxb.dev doesn't look like your website".
It is correctly set on my twitter profile at https://twitter.com/mxbck though. Could this be because of the new .dev TLD?

Thank you!

hah, sorry for the trouble! good guess, .dev domains are indeed blacklisted, back from when Google originally bought the TLD and said they would keep it internal only. now that they're reversed that stance, I'll remove the blacklist entry.

done! feel free to try again.

yup, works now! that must have been one of the fastest responses I've ever got ;)
thank you for your help!


Generate sample viz data for lung tissue

https://czi-ie-single-cell-prod-databricks-workspace.cloud.databricks.com/?o=1454199489318564#notebook/3783100054742660/command/2836064880204318

[ ] Determine how frequently a cell contains equivalent gene expression values.
[ ] If common, the rankit computation should assign the same rank, and thus probability, for these values.


requirements.txt conflict robotframework-sherlock and mkdocs

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
mkdocs 1.5.2 requires pathspec>=0.11.1, but you have pathspec 0.9.0 which is incompatible.

I used the same venv for both mkdocs and osfv. Using new clean venv solves issue for me.

@macpijan I have no idea how to classify this. I wasted some time before I realized there was conflict, and I assume it can happen also to other users. Separate venv for each repo would eliminate the problem but may lead to redundancy and storage waste, so the question is, what is our KPI here? My rough guess is we would like to deliver a reliable environment. Also, I wonder if we shouldn't get back to the docker image instead of using venv and system-native python. It would eliminate this problem also.

Separate venv for each repo would eliminate the problem

Yes, and this is what README suggests and is sane to do.
TBH this is quite unusual (and a bad practice in general) to use the same venv for multiple projects at the same time (with different requirements.txt files).

what is our KPI here?

That following documented steps produces a working environment?

lead to redundancy and storage waste,

Also, I wonder if we shouldn't get back to the docker image instead of using venv and system-native python. It would eliminate this problem also.

So 100MB venv is redundancy and storage waste, but 300+MB container is not? In the end, container has the same as venv inside + more.
We might go back to docker. We would need to add some wrapper scripts so it would be easier to use.
I must admit, that using venv is a bit easier and more convenient to use, especially if we need to modify the venv, not just execute scripts.
I think it might be easier to keep in sync by placing Dockerfile in this repo (so the same requirements.txt can be resued), rather than having to sync another repo: https://github.com/3mdeb/rf-docker

TBH this is quite unusual (and a bad practice in general) to use the same venv for multiple projects at the same time (with different requirements.txt files).

Correct, this is bad practice. I will close this issue since it is caused by having too much in one venv.

So 100MB venv is redundancy and storage waste, but 300+MB container is not? In the end, container has the same as venv inside + more.

The question is what is more reliable venv or docker.

We might go back to docker. We would need to add some wrapper scripts so it would be easier to use.
I must admit, that using venv is a bit easier and more convenient to use, especially if we need to modify the venv, not just execute scripts.

It really depend if you create this project for users or for developers.

I think it might be easier to keep in sync by placing Dockerfile in this repo (so the same requirements.txt can be resued), rather than having to sync another repo: https://github.com/3mdeb/rf-docker

Agree, maybe even CI for that?

https://github.com/Dasharo/open-source-firmware-validation/issues/60


Two factor authentication failure

The following files are to be populated.

app/Http/Controllers/RegisterController.php
app/Models/Authentication.php
resources/views/emails/normal.blade.php
resources/views/users/authenticate.blade.php

Please assign me the issue


Update to latest fix

In this version there is StackOverflowError which was fixed 7 months ago. Could you update this repository (last update 9months ago) to newest sliding library code (https://github.com/sysu2012zzp/SlidingMenu/commit/1d84040914b645ab836c8b02cbd0747ad3dedc66)?

Fixed in latest commit


Execution of featurecount incorrect

Description of the bug
Dear nf-core team,
while I was switching from version 1.4.2 to version 3.5 I have seen that the outputs of featurecounts are different regarding the grouping of the counts according to the attribute type. I have found the problem in at least version 3.4 and 3.5. There I have detected the function call of featurecounts being set up erroneously as parameter -g is the attribute type in the GTF file (default: gene_id). The value provided instead corresponds with the biotype of the annotation entry.
You can see the mentioned code snippet in commit https://github.com/nf-core/rnaseq/pull/779/commits/550ce6a487e6ac0b96a960874238f83b4de354f5 in my PR https://github.com/nf-core/rnaseq/pull/779.
Command used and terminal output
No response
Relevant files
No response
System information
No response

Hi @tglomb ! featureCounts is no longer used to do any formal gene quantification in the pipeline which is where I think this confusion is arising. We have replaced it with Salmon and RSEM to get more accurate gene/transcript quantification in v3.0.
featureCounts is only used in this pipeline to quantify biotypes in the annotation as I elude to in https://github.com/nf-core/rnaseq/issues/765#issuecomment-1046296271
Apologies for any confusion!

Hi @tglomb ! Hopefully, I have clarified the confusion in the comment above. We should probably document this a little better but I will close this issue as the pipeline is working as expected.


Add zsh completion for '--cluster-store-opt'

Zsh completion for docker daemon --cluster-store-opt~
ref. #16644
@thaJeztah Thanks for the ping
@vincentbernat
Signed-off-by: Steve Durrheimer<EMAIL_ADDRESS>

Thanks @sdurrheimer !

LGTM!

Thanks for reviewing @vincentbernat !
ping @tianon @jfrazelle for the merge

Thanks! LGTM


After docker-compose up and i go to the site, method /api/analyze/query_graphs return 500

Hi! I decided to try your tool, but I can't use it
In localhost:3000 i see erros:
backend: http api method /api/analyze/query_graphs return 500 with body {"type":"server_error","code":"error","detail":"A server error occurred.","attr":null}
frontend:

logs:
housewatch-web-1     |  HTTP  7/14/2023 12:45:08 PM <IP_ADDRESS> GET /
housewatch-web-1     |  HTTP  7/14/2023 12:45:08 PM <IP_ADDRESS> Returned 304 in 7 ms
housewatch-web-1     |  HTTP  7/14/2023 12:45:08 PM <IP_ADDRESS> GET /static/css/1.f7ef86df.chunk.css
housewatch-web-1     |  HTTP  7/14/2023 12:45:08 PM <IP_ADDRESS> GET /static/css/main.6e4c166f.chunk.css
housewatch-web-1     |  HTTP  7/14/2023 12:45:08 PM <IP_ADDRESS> Returned 304 in 5 ms
housewatch-web-1     |  HTTP  7/14/2023 12:45:08 PM <IP_ADDRESS> Returned 304 in 5 ms
housewatch-web-1     |  HTTP  7/14/2023 12:45:08 PM <IP_ADDRESS> GET /static/js/1.fb96ed73.chunk.js
housewatch-web-1     |  HTTP  7/14/2023 12:45:08 PM <IP_ADDRESS> GET /static/js/main.a91ea030.chunk.js
housewatch-web-1     |  HTTP  7/14/2023 12:45:08 PM <IP_ADDRESS> Returned 304 in 2 ms
housewatch-web-1     |  HTTP  7/14/2023 12:45:08 PM <IP_ADDRESS> Returned 304 in 2 ms
housewatch-web-1     |  HTTP  7/14/2023 12:45:08 PM <IP_ADDRESS> GET /static/css/main.6e4c166f.chunk.css.map
housewatch-web-1     |  HTTP  7/14/2023 12:45:08 PM <IP_ADDRESS> GET /static/css/1.f7ef86df.chunk.css.map
housewatch-web-1     |  HTTP  7/14/2023 12:45:08 PM <IP_ADDRESS> Returned 304 in 1 ms
housewatch-web-1     |  HTTP  7/14/2023 12:45:08 PM <IP_ADDRESS> Returned 304 in 1 ms
housewatch-web-1     |  HTTP  7/14/2023 12:45:08 PM <IP_ADDRESS> GET /static/js/1.fb96ed73.chunk.js.map
housewatch-web-1     |  HTTP  7/14/2023 12:45:08 PM <IP_ADDRESS> GET /static/js/main.a91ea030.chunk.js.map
housewatch-web-1     |  HTTP  7/14/2023 12:45:08 PM <IP_ADDRESS> Returned 304 in 1 ms
housewatch-web-1     |  HTTP  7/14/2023 12:45:08 PM <IP_ADDRESS> Returned 304 in 2 ms
housewatch-web-1     |  HTTP  7/14/2023 12:45:08 PM <IP_ADDRESS> GET /static/media/HindSiliguri-Medium.41fd138d.ttf
housewatch-web-1     |  HTTP  7/14/2023 12:45:08 PM <IP_ADDRESS> Returned 304 in 1 ms
housewatch-app-1     | [14/Jul/2023 12:45:08] "GET /api/analyze/hostname HTTP/1.1" 200 29
housewatch-web-1     |  HTTP  7/14/2023 12:45:08 PM <IP_ADDRESS> GET /favicon.ico
housewatch-web-1     |  HTTP  7/14/2023 12:45:08 PM <IP_ADDRESS> Returned 304 in 2 ms
housewatch-web-1     |  HTTP  7/14/2023 12:45:08 PM <IP_ADDRESS> GET /manifest.json
housewatch-web-1     |  HTTP  7/14/2023 12:45:08 PM <IP_ADDRESS> Returned 304 in 1 ms
housewatch-app-1     | [14/Jul/2023 12:45:08] "GET /api/analyze/query_graphs HTTP/1.1" 500 86
I use environment variables: CLICKHOUSE_SECURE=False, CLICKHOUSE_HOST , CLICKHOUSE_CLUSTER, CLICKHOUSE_USER and CLICKHOUSE_PASSWORD
With respect, Andrei

@Dolso there can be many reasons for 500 Internal server error. Can you click on the link to the API http://localhost:8000/api/analyze/query_graphs and open it in a new tab. Check if there's any additional explanation about the error, like trace logs.
If there will be just the pretty error-page try to start the API in debug mode. Like so:
DEBUG=1 python manage.py runserver <IP_ADDRESS>:8000


Fix for WFCORE-4406 and Upgrade to Galleon 4.0.0.alpha

F- ix for https://issues.jboss.org/browse/WFCORE-4406

Upgrade to galleon 4.0 alpha. Final upgrade is tracked by (https://issues.jboss.org/browse/WFCORE-4422)
Produces FP with Galleon 4.0, generates and deploy wildfly-core offliner. That is a required step to address: https://issues.jboss.org/browse/WFLY-11989
Make dependencies provided, it is required to not download all dependencies when building User feature-packs from wildfly feature-packs.

@psakar @bstansberry FYI this is not labelled with deps-changed as it does not go into the feature packs

@jmesnil , I think that it should be ok, @aloubyansky we have 2 weeks before the final release, is it fine with you?

Yes, we'll have the final release by that date.


Add terser (JS compressor for ES5 and ES6)

Add terser as a JS compressor which can handle both ES5 and ES6 syntax.

Coverage remained the same at ?% when pulling 530b8c418b35a493f6d75784d8006bbaf206d86b on felix-last:terser into decffd39ae3361a89c5b832e74c8009973b76636 on jazzband:master.

@mathjazz Seems like the tests failed for CI related issues in lastest Django master. I've fixed it in master..

@mathjazz Seems like the tests failed for CI related issues in lastest Django master. I've fixed it in master..

Thanks! Master seems to be failing, though.

Thanks! Master seems to be failing, though.

Thanks! Master seems to be failing, though.

Yeah, seems like some things that got removed following the deprecation policy. I've just opened #742 to be clear what's failing.

Thanks! Master seems to be failing, though.

Yeah, seems like some things that got removed following the deprecation policy. I've just opened #742 to be clear what's failing.

Hey @cyberdelia, we'd like to switch https://github.com/mozilla/pontoon/pull/1819#issuecomment-762189263 to the terser compressor, support for which just landed on master.
Could you please publish the new release on PyPI?

Hey @cyberdelia, we'd like to switch https://github.com/mozilla/pontoon/pull/1819#issuecomment-762189263 to the terser compressor, support for which just landed on master.
Could you please publish the new release on PyPI?

just created the 2.0.6 release.
Thanks a lot @jezdez for making it so easy.

just created the 2.0.6 release.
Thanks a lot @jezdez for making it so easy.

just created the 2.0.6 release.
Thanks a lot @jezdez for making it so easy.

Happy you feel that way! Remember to release the files to PyPI on https://jazzband.co/projects/django-pipeline#2.0.6

just created the 2.0.6 release.
Thanks a lot @jezdez for making it so easy.

Happy you feel that way! Remember to release the files to PyPI on https://jazzband.co/projects/django-pipeline#2.0.6

@cyberdelia any chance to have the 2.0.6 package uploaded to PyPI? 🙏 Thanks!

@cyberdelia any chance to have the 2.0.6 package uploaded to PyPI? 🙏 Thanks!

Done.

Done.


update concat_elimination pass for openvino use

since openvino has special handling for removing ops / friendly_name handling, atleast for this concat null op case i created these changes in below PR
https://github.com/NervanaSystems/ngraph/pull/4536


Explain todos

It's just a reminder, since I visit this repo three-four times a week and still see the todos. It's very interesting to me. Thanks

This project is not currently being worked on and its code is not even in a usable state.  I don't have any plans to continue working on this project at this time.

@tonyshkurenko You could fork this project and create a library out of it.

I ain't sure I'm able to do that :)

@tonyshkurenko Sure you can do that. And even if the library never gets finished you'll learn a lot from it. I guess @JakeWharton csn confirm that ;-) When I started my first open source library it helped a lot looking at best practices and comparing code style and API style to other common libraries. Jake's libraries are just perfect so don't hesitate to look at his code and learn from it. That's the whole purpose of open source


SPFx - How to load jsom dependencies in a proper way

Target SharePoint environment
SharePoint Online
What SharePoint development model, framework, SDK or API is this about?
💥 SharePoint Framework
Developer environment
Windows
What browser(s) / client(s) have you tested

[ ] 💥 Internet Explorer
[ ] 💥 Microsoft Edge
[ ] 💥 Google Chrome
[ ] 💥 FireFox
[ ] 💥 Safari
[ ] mobile (iOS/iPadOS)
[ ] mobile (Android)
[x] not applicable
[ ] other (enter in the "Additional environment details" area below)

Additional environment details
 "dependencies": {
    "@microsoft/sp-core-library": "1.17.1",
    "@microsoft/sp-loader": "^1.17.2",
    "@microsoft/sp-lodash-subset": "1.17.1",
    "@microsoft/sp-office-ui-fabric-core": "1.17.1",
    "@microsoft/sp-property-pane": "1.17.1",
    "@microsoft/sp-webpart-base": "1.17.1",
    "classnames": "^2.3.2",
    "office-ui-fabric-react": "^7.199.1",
    "papaparse": "^5.4.1",
    "react": "17.0.1",
    "react-dom": "17.0.1",
    "tslib": "2.3.1"
  },
  "devDependencies": {
    "@microsoft/eslint-config-spfx": "1.17.1",
    "@microsoft/eslint-plugin-spfx": "1.17.1",
<EMAIL_ADDRESS>"0.4.0",
    "@microsoft/sp-build-web": "1.17.1",
    "@microsoft/sp-module-interfaces": "1.17.1",
    "@rushstack/eslint-config": "2.5.1",
    "@types/microsoft-ajax": "^0.0.37",
    "@types/papaparse": "^5.3.7",
    "@types/react": "17.0.45",
    "@types/react-dom": "^17.0.17",
    "@types/sharepoint": "^2016.1.11",
    "@types/webpack-env": "~1.15.2",
    "ajv": "^6.12.5",
    "eslint": "8.7.0",
    "eslint-plugin-react-hooks": "4.3.0",
    "gulp": "4.0.2",
    "typescript": "4.5.5"
  }

Describe the bug / error
I am getting crazy trying to load scripts like Runtime.js and SP.js. In the beginning I added the dependencies in the config.json as mentioned here.
I ended up to load the SharePoint JSOM scripts with the SPComponentLoader. In the debug it works, once deployed without -p, I get:
SP.Runtime.js:2 Uncaught Error: SP.Runtime.js is already loaded
Uncaught RangeError: Maximum call stack size exceeded
    at Boolean (<anonymous>)
    at SupplementDefaultMethod.window.<computed> [as OriginalOnError] (init.js:1:180299)
    at ULSSendExceptionImpl (init.js:1:106758)
    at ULSOnError (init.js:1:106829)
    at SupplementDefaultMethod.window.<computed> [as OriginalOnError] (init.js:1:180335)
    at ULSSendExceptionImpl (init.js:1:106758)
    at ULSOnError (init.js:1:106829)
    at SupplementDefaultMethod.window.<computed> [as OriginalOnError] (init.js:1:180335)
    at ULSSendExceptionImpl (init.js:1:106758)
    at ULSOnError (init.js:1:106829)

If I package the solution with -p I get:
GET https://relative-path.invalid/sp-init.js net::ERR_NAME_NOT_RESOLVED
GET https://relative-path.invalid/sp-runtime.js net::ERR_NAME_NOT_RESOLVED
GET https://relative-path.invalid/sharepoint.js net::ERR_NAME_NOT_RESOLVED
GET https://relative-path.invalid/microsoft-ajax.js net::ERR_NAME_NOT_RESOLVED

Steps to reproduce
Actually, I am looking for the right way to load these dependencies.
Expected behavior
Have the code running in all three scenarios.

I figured out, that using the flag --build, like gulp build --debug and gulp package-solution --debug, work.

As suggested by another user on another forum:

    if (window.location.pathname.indexOf("/_layouts/15/workbench.aspx") > -1) {
      SPComponentLoader.loadScript("/_layouts/15/init.js", {
        globalExportsName: "$_global_init",
      })
        .then((): Promise<{}> => {
          return SPComponentLoader.loadScript("/_layouts/15/MicrosoftAjax.js", {
            globalExportsName: "Sys",
          });
        })

        .then((): Promise<{}> => {
          return SPComponentLoader.loadScript("/_layouts/15/SP.Runtime.js", {
            globalExportsName: "SP",
          });
        })

        .then((): Promise<{}> => {
          return SPComponentLoader.loadScript("/_layouts/15/SP.js", {
            globalExportsName: "SP",
          });
        });
    } else {
      SPComponentLoader.loadScript("/_layouts/15/init.js", {
        globalExportsName: "$_global_init",
      });
    }

It works, really ugly... wonder if Microsoft has a clean solution.

@emaborsa - do you have a solution you could share?
As mentioned in the docs you referenced you can use either declarative or code approach to reference the jsom libraries.
but not both.

Ok, lets start with the references in the config.json externals:
https://github.com/HGV/SPO.JSOM.Test

check out the repo
checkout using-externals
edit the my.sharepoint.com with your host
npm i
gulp serve
add the web part and click on + Click here
I get the alert ReferenceError: SP is not defined

Now move using ComponentLoader.loadScript:

checkout using-component-loader
gulp serve
click on + Click here
should alert It works!
gulp bundle -p
gulp package-solution -p
properly install solution and add web part
Add loading, before clicking ``+ Click here` I get the following error:

SP.Runtime.js:2 Uncaught Error: SP.Runtime.js is already loaded
(anonymous) @ SP.Runtime.js:2
init.js:1 Uncaught RangeError: Maximum call stack size exceeded
    at Boolean (<anonymous>)
    at ULSSendExceptionImpl (init.js:1:106252)
    at ULSOnError (init.js:1:106829)
    at SupplementDefaultMethod.window.<computed> [as OriginalOnError] (init.js:1:180335)
    at ULSSendExceptionImpl (init.js:1:106758)
    at ULSOnError (init.js:1:106829)
    at SupplementDefaultMethod.window.<computed> [as OriginalOnError] (init.js:1:180335)
    at ULSSendExceptionImpl (init.js:1:106758)
    at ULSOnError (init.js:1:106829)
    at SupplementDefaultMethod.window.<computed> [as OriginalOnError] (init.js:1:180335)

@emaborsa - I'm not seeing this part in your project:
require('sp-init');
require('microsoft-ajax');
require('sp-runtime');
require('sharepoint');

@AJIXuMuK ,
did you try to clone and fix it?

checkout using-externals
add  import "microsoft-ajax"; import "sharepoint"; import "sp-init"; import "sp-runtime"; in Listview.tsx
gulp serve
click on + Click here
It works!
gulp bundle -p
gulp package-solution -p
properly install solution and add web part
At loading, before clicking + Click here I get the following error:

SP.Runtime.js:2 Uncaught Error: SP.Runtime.js is already loaded
(anonymous) @ SP.Runtime.js:2
init.js:1 Uncaught RangeError: Maximum call stack size exceeded
    at Boolean (<anonymous>)
    at ULSSendExceptionImpl (init.js:1:106252)
    at ULSOnError (init.js:1:106829)
    at SupplementDefaultMethod.window.<computed> [as OriginalOnError] (init.js:1:180335)
    at ULSSendExceptionImpl (init.js:1:106758)
    at ULSOnError (init.js:1:106829)
    at SupplementDefaultMethod.window.<computed> [as OriginalOnError] (init.js:1:180335)
    at ULSSendExceptionImpl (init.js:1:106758)
    at ULSOnError (init.js:1:106829)
    at SupplementDefaultMethod.window.<computed> [as OriginalOnError] (init.js:1:180335)

@AJIXuMuK any update?

@emaborsa - sorry for the really long delay.
I tested both branches in your sample with the steps you provided.
I'm not experiencing the issue for the using-externals

for the using-component-loader I needed to remove import statements as you don't have the modules loaded in that case.
After that I can both debug and deploy the solution without any issues.


FluentAvalonia does not compile with avalonia 11.2

When upgrading to avalonia 11.2, the FluentAvalonia project does not compile.
It shows various missing items in the DateTimePicker style file

I'm also having a problem with the TimePicker the code compiles but the designer stops working.
the output shows an error but not much information
19:13:02.441 [Debug] 29776 <= UpdateXamlResultMessage { Error: "System.NullReferenceException: …", Handle: null, Exception: ExceptionDetails { ExceptionType: "NullReferenceException", Message: "Object reference not set to an …", LineNumber: null, LinePosition: null } }
19:13:02.441 [Error] 29776 UpdateXamlResult error
System.Xaml.XamlException: Object reference not set to an instance of an object.
19:13:02.446 [Verbose] 29776 Finished PreviewerProcess.OnMessageAsync()

Confirm same issue with AbaloniaUI 11.2
https://github.com/AvaloniaUI/Avalonia/issues/17419

A fix for this has just been committed into Avalonia
Fix TimePicker breaking change, when using pre-11.2 styles #17516

Closing, as this is result of Avalonia breaking change in a minor version + should also be fixed now.


[4.x] Fix checksum computation

It seems that the checksum of changesets are only based on queries.
They should also include:

author
id
preconditions
postconditions

However, fixing that now is tricky as we need to migrate all the stored changesets from previous versions (basically, update their checksum to the new definition).
This means introducing another internal migration such as this one.
Note: Even if we migrate the stored changesets, this still is a breaking change (previously modified changesets may now break if run-on-change is not set to true).

just reading this, I'm not sure it's true, afaik liquibase does not include these things... I've changed all of these properties in liquibase 3 without triggering a checksum failure. (maybe liquibase 4 is changing this?)

I haven't used liquibase for a while, I need to have a look at the codebase again.
That being said, it feels like the attributes listed also form the identity of a changeset. Changing any of them should result in a different changeset (though changing the id of a changeset is indeed the same as introducing a new changeset).

Though the rest is not part of the identity of a changeset, I feel that changing preconditions or postconditions is cheating a bit.
Once executed against a target database, a changeset should never change at all (by default).

just my 2 cents, and that's not how liquibase does it, but as you will. I think there are reasons for preconditions and postconditions... something like a changeset ran in one environment, passed them, then in another environment couldn't, and so you need to refine the precondition so that it will correctly do or do not in the new environment, think... making a change in staging against one set of data, then finding out your precondition doesn't exactly do the right thing in production... You don't want to rerun that changeset in staging though. Obviously in liquibase though you can also set multiple valid checksums (I don't know if you can do that here)

@xenoterracide could you share documentation of the "multiple valid checksums" feature you mention? I'm interested in learning how that works :)

it's under available sub tags https://docs.liquibase.com/concepts/basic/changeset.html

I was trying to find out how the checksum is calculated, but it turns out its really complicated, and depends on the subclasses of this https://github.com/liquibase/liquibase/blob/master/liquibase-core/src/main/java/liquibase/change/AbstractChange.java

Thanks @xenoterracide. Having multiple checksums could come in handy in some scenarios (such as upgrading Neo4j that brings incompatible cypher changes that forces to rewrite some of the changeset queries).
However, the focus of this issue, i.e. revising how checksum is computed, is not a good idea, as you proved it well.
Thanks a lot for chiming in!


test: split tests by resource type. fix query time by limiting by timestamp

This should reduce the time for the integration tests from ~20 minutes to ~3 minutes.

ListSink tests fixed in #226


Does it work on github?

Thanks for great work.  I am a newbie in jekyll
I have a question, the paginate-v2 works fine on local server,
But when I push to my github account, It is not work , do I  missed something?
Thanks

Currently supported plugins for Github Pages can be found here - https://pages.github.com/versions/
I do not think they have allowed for use of this plugin yet.

Oh, thanks,
It's so sad they don't allowed it.


backend_qemu: Avoid file ID collisions, when multiple devices

Currently we get a useful warning like:
9p: Multiple devices detected in same VirtFS export, which might lead to file ID collisions and severe misbehaviours on guest!
You should either use a separate export for each device shared from host or use virtfs option 'multidevs=remap'!
So let's add the option to avoid the issue. Requires QEMU >= 4.2.0
[Emil Velikov]

Rebase, tweak commit message

All done - sorry for the delay.

bors r+


Add Tag to Entry

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@0e8934d). Click here to learn what that means.
The diff coverage is 40.00%.

@@            Coverage Diff            @@
##             master      #70   +/-   ##
=========================================
  Coverage          ?   67.40%
  Complexity        ?      471
=========================================
  Files             ?       85
  Lines             ?     1620
  Branches          ?      156
=========================================
  Hits              ?     1092
  Misses            ?      490
  Partials          ?       38

Impacted Files
Coverage Δ
Complexity Δ

.../java/seedu/address/model/util/SampleDataUtil.java
8.82% <0.00%> (ø)
1.00 <0.00> (?)

...n/java/seedu/address/storage/JsonAdaptedEntry.java
0.00% <0.00%> (ø)
0.00 <0.00> (?)

src/main/java/seedu/address/ui/EntryCard.java
0.00% <0.00%> (ø)
0.00 <0.00> (?)

...c/main/java/seedu/address/model/journal/Entry.java
87.50% <75.00%> (ø)
14.00 <1.00> (?)

...ess/logic/parser/AddJournalEntryCommandParser.java
100.00% <100.00%> (ø)
6.00 <0.00> (?)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0e8934d...6b042b9. Read the comment docs.


Improve CIN description and colors to be more accurate

Hey, I tried to email this to<EMAIL_ADDRESS>but it bounced. I studied meteorology and noticed the last sentence of the CIN description (and chosen colors) is wrong because when CIN is closer to 0, it is more dangerous to fly. (Zero inhibition means there is nothing stopping air from convecting).
I've also updated the CIN ranges used for weak, moderate, and strong to reflect what I've seen in scientific papers.
It's confusing because of the negative values used to describe negative energy, but it's important for pilots to know that CIN values closer to zero are more dangerous and CIN values farther from zero are more stable. This all depends on CAPE as well.
I am NOT a programmer at all and made a GitHub account just for this.

@aeharding First, thank you for all your great work on ppg.report and making it open source!
I'd really like to see this reviewed and merged if confirmed to be correct. If this is correct, it is a very important, safety-critical issue that should be addressed ASAP.

Hey thanks for the PR.
Yes, you're definitely correct that high levels of convective inhibition typically indicate more stability in the atmosphere.
However, from the perspective of a paramotor pilot, the best flying is when CIN and CAPE are both 0.
If CIN is high, even if CAPE is low, it might technically be a stable atmosphere, but in practice, high levels of CIN mean that something is happening in the atmosphere, and for paramotor pilots, when something is happening in the atmosphere, it's generally not good.
I think it would definitely be a problem when CIN is rendered as a green color. But in PPG.report, white doesn't mean "good" it just means neutral. Which a CIN of 0 is.
I'm happy to take a look at the language here, because I agree, it's not ideal. But in terms of color coding, I think the existing is good.
As feedback regarding your proposed language, I think I would note that high levels of CIN do indicate stable atmosphere as you said, high levels of CIN is also unusual and should probably be scrutinized.
Again, in practice, every time I've seen high CIN with low CAPE, there's been something weird happening in the atmosphere that at very least deserves more scrutiny before flying.
I think we all agree however, it's best to err on the side of caution in terms of what is shown to the user, so that the user can make their own informed decisions.

Closing due to previously mentioned concerns. Thanks for starting this discussion!