K8SPG-600: Fix failed backup status

https://perconadev.atlassian.net/browse/K8SPG-600
DESCRIPTION
Problem:
If the user has the wrong pgbackrest credentials, the failed backup doesn't get the failed state in the pg-backup .status..
Reason:
When incorrect pgbackrest credentials are used, the pg cluster reconcile fails, which stops the process of updating crunchy's postgrescluster resource. It is necessary to delete backup annotations from crunchy's resource, and this problem doesn't allow it to continue.
Solution:
Don't stop the reconcile process with the pgbackrest error. Log the error message instead. Also, the operator should avoid trying to use pgbackrest when the backup job has failed.
CHECKLIST
Jira

[x] Is the Jira ticket created and referenced properly?
[x] Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
[x] Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

[ ] Is an E2E test/test case added for the new feature/change?
[ ] Are unit tests added where appropriate?

Config/Logging/Testability

[x] Are all needed new/changed options added to default YAML files?
[x] Did we add proper logging messages for operator actions?
[x] Did we ensure compatibility with the previous version or cluster upgrade process?
[x] Does the change support oldest and newest supported PG version?
[x] Does the change support oldest and newest supported Kubernetes version?

Test name
Status

custom-extensions
passed

demand-backup
failure

init-deploy
passed

major-upgrade
failure

monitoring
passed

one-pod
failure

operator-self-healing
passed

pitr
passed

scaling
failure

scheduled-backup
failure

self-healing
failure

start-from-backup
passed

tablespaces
passed

telemetry-transfer
passed

upgrade-consistency
passed

upgrade-minor
passed

users
passed

We run 17 out of 17

commit: https://github.com/percona/percona-postgresql-operator/pull/820/commits/b5f61e04a7a1417d8af7068bfa928a89bc398a13
image: perconalab/percona-postgresql-operator:PR-820-b5f61e04a

Test name
Status

custom-extensions
passed

demand-backup
failure

init-deploy
passed

major-upgrade
passed

monitoring
passed

one-pod
failure

operator-self-healing
passed

pitr
passed

scaling
failure

scheduled-backup
failure

self-healing
failure

start-from-backup
passed

tablespaces
passed

telemetry-transfer
passed

upgrade-consistency
passed

upgrade-minor
passed

users
passed

We run 17 out of 17

commit: https://github.com/percona/percona-postgresql-operator/pull/820/commits/b5f61e04a7a1417d8af7068bfa928a89bc398a13
image: perconalab/percona-postgresql-operator:PR-820-b5f61e04a

Test name
Status

custom-extensions
passed

demand-backup
passed

init-deploy
passed

major-upgrade
passed

monitoring
passed

one-pod
passed

operator-self-healing
failure

pitr
passed

scaling
passed

scheduled-backup
passed

self-healing
passed

start-from-backup
passed

tablespaces
passed

telemetry-transfer
passed

upgrade-consistency
passed

upgrade-minor
passed

users
passed

We run 17 out of 17

commit: https://github.com/percona/percona-postgresql-operator/pull/820/commits/066cdc9dea65922f6120ac661e16ca1c1dd60022
image: perconalab/percona-postgresql-operator:PR-820-066cdc9de

Test name
Status

custom-extensions
passed

demand-backup
passed

init-deploy
passed

major-upgrade
passed

monitoring
passed

one-pod
passed

operator-self-healing
failure

pitr
passed

scaling
passed

scheduled-backup
passed

self-healing
passed

start-from-backup
passed

tablespaces
passed

telemetry-transfer
passed

upgrade-consistency
passed

upgrade-minor
passed

users
passed

We run 17 out of 17

commit: https://github.com/percona/percona-postgresql-operator/pull/820/commits/066cdc9dea65922f6120ac661e16ca1c1dd60022
image: perconalab/percona-postgresql-operator:PR-820-066cdc9de

Test name
Status

custom-extensions
passed

demand-backup
passed

init-deploy
passed

major-upgrade
passed

monitoring
passed

one-pod
passed

operator-self-healing
passed

pitr
passed

scaling
passed

scheduled-backup
passed

self-healing
passed

start-from-backup
passed

tablespaces
passed

telemetry-transfer
passed

upgrade-consistency
passed

upgrade-minor
passed

users
passed

We run 17 out of 17

commit: https://github.com/percona/percona-postgresql-operator/pull/820/commits/066cdc9dea65922f6120ac661e16ca1c1dd60022
image: perconalab/percona-postgresql-operator:PR-820-066cdc9de


create student.txt, example @student.htldornbirn.at

Please add HTL Dornbirn

Please remove /student and change the path to at/htldornbirn.txt - all subdomains will be covered this way. Thank you.


let's improve this recipe (and set a standard for python packages calling julia packages)

Solution to issue cannot be found in the documentation.

[X] I checked the documentation.

Issue
This line shouldn't end like that and it useless; the package on anaconda.org doesn't have any of these crazy julia modules being installed: https://anaconda.org/conda-forge/pysr/files
https://github.com/conda-forge/pysr-feedstock/blob/cbf0bb3c7efa0262fe442a4f8495799fc1eb471c/recipe/meta.yaml#L14
If you really want && {{ PYTHON }} -c 'import pysr; pysr.install() you should add it in an activation script.
@ocefpaf, sorry I just saw your question on gitter. I am never there, and usually not very good at instant messaging 😅 it is probably easier to get my attention to something via tag in an open pull request or issue.
Also tagging @isuruf if you have time to look into this with us.
The original message on gitter:
Filipe @ocefpaf Aug 06 10:55
Folks, what are the best practices for packaging a Python module that depends on a Julia pkg? @ngam I recall you packaging some Julia/Python stuff, do you have any advice for me?

Do you have a specific recipe/package? I am happy to work on it together in staged-recipes! Just tag me or respond here :)
Installed packages
n/a

Environment info
n/a

I agree it would be better if the pysr.install() ran as part of the conda install. Do you know how to implement that?

The post-link trick sounds like the best bet to me. Happy to try this, let me know when I can get started.

Hm, I'm reading the docs on it now and I'm not sure post-link.sh is the right strategy:

Post-link and pre-unlink scripts should:

Be avoided whenever possible.
Not touch anything other than the files being installed.
Not write anything to stdout or stderr, unless an error occurs.
Not depend on any installed or to-be-installed conda packages.
Depend only on simple system tools such as rm, cp, mv, and ln.

Specifically the last two points. I'm worried if trying to go this route will introduce a variety of other operating system-specific bugs to solve in the future... What do you think?

Yes, I am not sure about it either. One of the main questions to answer is how deterministic the script can be. Users expect conda packages to be reproducible, so if it has dynamic behaviour, it better be a controlled one. I am not aware of what pysr.install() does, but if it has to find Julia dependencies, can we make sure they are always the same versions?

One thing we could do is fix the Manifest.toml file. Otherwise the Julia dependencies will be different every single install. These are bounded by the Project.toml file but those constraints are pretty soft rather than set to specific versions.

Manifest.toml is good, but I'm not sure if that is a good thing for us to distribute. It would be similar to distributing a conda-lock file.
Perhaps it would be useful to tighten the version bounds further?
https://pkgdocs.julialang.org/v1/compatibility/

I guess even if you tighten the version bounds on all the direct dependencies, the dependencies of those dependencies could still change their version. (and this sometimes could result in distinct behavior as a function of date the package is installed...). You could add [compat] bounds on the entire dependency tree, but then it seems is the same as the Manifest.toml file, no?

I agree it would be better if the pysr.install() ran as part of the conda install. Do you know how to implement that?

I'm not so sure. I like the module.install(), at least for now, b/c that makes an easy transition to folks who wants to manually install Julia vs those who are packaging in systems like conda-forge, debian, etc. However, I have zero knowledge of Julia packaging and my thoughts here should be taken with a grain of salt.

 script: {{ PYTHON }} -m pip install . -vv'
 script: {{ JULIA }} -c 'using Pkg; Pkg.install(".")'

That would be the dream, right?


packaging: Fedora: update for RHEL8.2

Set __cmake_builddir conditionally to enable full builds on RHEL8.2.
Signed-off-by: Tim Whisonant<EMAIL_ADDRESS>

Coverage remained the same at 64.873% when pulling 2b5873200a6bc40c6372ecaef7ba60c21767067c on tswhison/fedora_spec into 245c6dd89d0c90497a0f4ffdc70a5e6739e19145 on master.


community: Add support for DOT_PRODUCT distance for BigQueryVectorStore

PR Description
Add support for DOT_PRODUCT distance for BigQueryVectorStore.
DOT_PRODUCT is supported in BQ: doc
Relevant issues

Type

🆕 New Feature
📖 Documentation (auto-generated)
Changes(optional)
Add DOT_PRODUCT in the list of supported distance types.

@lkuligin for final approval


Proposal: Create a website enabling users to connect to people all around the world to communicate on Kaia

Description

Project: Connect to stranger
Target:

Create an online space enable users to anonymous and fetterless communicate with other people.
Encourage cultural exchange and the exchange of ideas between individuals from many different countries.
Provides a safe and friendly platform to discuss diverse topics.
Main feature:
Friendly user interface: Simple design and easy to use, suitable for all ages.
Incognito mode: Users can choose a nickname and do not need to provide personal information.
Blockchain & Kaia integration
Bounty track

[add the name of the bounty track here]

I posted this before in another proposal

imagine a smart contract to host all your posts with a common interface or entry point. then external indexers can aggregate all of these smart contracts and posts to build up a feed/client
works like a personal blog (not a social media). So i can send posts (by calling the function), then the posts will be stored there.

I have a few notes for you:

Create an online space enable users to anonymous and fetterless communicate with other people.
Encourage cultural exchange and the exchange of ideas between individuals from many different countries.
Provides a safe and friendly platform to discuss diverse topics.

This is contradictory. Anonymous and fetterless often != safe and friendly. Try to read 4chan and you will see.
I would like you to focus more on anonymous and uncensored. We don't need another web2 social media.

Incognito mode: Users can choose a nickname and do not need to provide personal information.

Wallet address is already somewhat incognito and good enough for this proposal.

The application was completed. I submitted it

@zxstim

please send all the links here (deployment, github links)

deployment: https://vent-it-client-m2i2gan0l-menh1505s-projects.vercel.app/
github:

front end: https://github.com/Menh1505/vent_it-client.git
contract: https://github.com/Menh1505/vent_it-contract

why does it say sepolia here? should be kaia kairos testnet

added kaia kairos: https://vent-it-client-git-feat-post-menh1505s-projects.vercel.app/

Still need a decent amount of polishing but nice try
approved for 300 KAIA bounty


the bind mount content is private and unshared

Issue #676
Description of changes:
The Z option indicates that the bind mount content is private and unshared.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

These seems pretty risky to me. I don't fully understand this change either and how it impacts all OS's.
From here:
" Use extreme caution with these options. Bind-mounting a system directory such as /home or /usr with the Z option renders your host machine inoperable and you may need to relabel the host machine files by hand."

This is a risky change. The risk is not worth the benefits. A better solution will be to actually copy files over instead of bind mounting. #976 is implementing a copy functionality to support Golang builds.


Depend on electron-updater v4

Otherwise running yarn build -p always errors out with At least electron-updater 4.0.0 is recommended by current electron-builder version. Please set electron-updater version to "^4.0.0"

Thank you!


Test code

Excuse me, can you provide the code of the test process? The input of the test is only an image and does not require the target caption.

Hi @ExitPath
Do you mean the test code for test dataset?

Hi @ExitPath
Do you mean the test code for test dataset?

Yes, it is test.py or eval.py. The input is just an image，and the predicted caption can be output.

Sure. Maybe I can upload the inference.py for evaluation within next month. I'll let you know when I'm done with work.

Ok, thank you very much！

Hi @ExitPath
I upload the inference.py. Please check the repo and let me know if you have any problem.
Thanks :)


system-update.sh seems to keep failing (10.9 / Mavericks)

It looks like the updates are totally failing for Mavericks:
2015/11/13 14:41:22 ui:     virtualbox-iso: Downloading and installing system updates...
    virtualbox-iso: Downloading and installing system updates...
2015/11/13 14:41:34 ui:     virtualbox-iso: Software Update Tool
    virtualbox-iso: Software Update Tool
2015/11/13 14:41:34 ui:     virtualbox-iso: Copyright 2002-2012 Apple Inc.
    virtualbox-iso: Copyright 2002-2012 Apple Inc.
2015/11/13 14:41:34 ui:     virtualbox-iso:
    virtualbox-iso:
2015/11/13 14:41:34 ui:     virtualbox-iso: Finding available software
    virtualbox-iso: Finding available software
2015/11/13 14:41:34 ui:     virtualbox-iso:
    virtualbox-iso:
2015/11/13 14:41:34 ui:     virtualbox-iso: Downloading OS X Update Combined
    virtualbox-iso: Downloading OS X Update Combined
2015/11/13 14:41:34 ui:     virtualbox-iso: Downloading iBooks Update
    virtualbox-iso: Downloading iBooks Update
2015/11/13 14:45:45 ui:     virtualbox-iso: Downloading iTunes
    virtualbox-iso: Downloading iTunes
2015/11/13 14:45:52 ui:     virtualbox-iso: Error downloading iTunes: Didn't get a response from the Apple Software Update server when downloading iTunes.
    virtualbox-iso: Error downloading iTunes: Didn't get a response from the Apple Software Update server when downloading iTunes.
2015/11/13 14:47:00 ui:     virtualbox-iso: Error downloading iBooks Update: Didn't get a response from the Apple Software Update server when downloading iBooks Update.
    virtualbox-iso: Error downloading iBooks Update: Didn't get a response from the Apple Software Update server when downloading iBooks Update.
2015/11/13 14:47:00 ui:     virtualbox-iso: Downloaded OS X Update Combined
    virtualbox-iso: Downloaded OS X Update Combined
2015/11/13 14:53:45 ui:     virtualbox-iso: Installing OS X Update Combined, iBooks Update, iTunes
    virtualbox-iso: Installing OS X Update Combined, iBooks Update, iTunes
2015/11/13 14:53:45 ui:     virtualbox-iso: Done with OS X Update Combined
    virtualbox-iso: Done with OS X Update Combined
2015/11/13 14:53:45 ui:     virtualbox-iso: Done.
    virtualbox-iso: Done.
2015/11/13 14:53:45 ui:     virtualbox-iso:
    virtualbox-iso:
2015/11/13 14:53:45 ui:     virtualbox-iso: Error downloading updates.
    virtualbox-iso: Error downloading updates.

I saw some errors in the "Closed" list which sound a lot like these. Is this possibly a problem with Apple no longer providing these downloads?

I see the SUS servers have issue from time to time. (A couple weeks ago they were having issues for the better part of a day, leading the CLI tools downloads to fail). I'd suggest waiting a while and trying again.
Not sure what's the "Closed" list you're referring to, though..

Aha... I shall try again on Monday, let's see what happens.
I was referring to this closed ticket: https://github.com/timsutton/osx-vm-templates/issues/33
Which seems vaguely related...

#33 was specifically about the CLI tools not being available in Apple's software update server catalogs for 10.9. The error you're posting is about a general response from their update server for the system updates script, which in this template comes later. You might want to check whether the earlier step of installing CLI tools worked or not.

That one did work...

Tried again today, failed again, but with a slightly different error:
2015/11/18 16:25:00 ui:     virtualbox-iso: Error downloading updates.
    virtualbox-iso: Error downloading updates.
2015/11/18 16:25:00 ui:     virtualbox-iso: Nov 18 08:24:58 osx-10_9.vagrantup.com softwareupdate CLI[588] <Error>: /usr/sbin/softwareupdate: XPC error in __60-[SUCommandLineTool _runSessionWithUpdates:skippingInstall:]_block_invoke (Error Domain=NSCocoaErrorDomain Code=4099 "Couldn’t communicate with a helper application." (The connection was invalidated from this process.) UserInfo=0x7fa201c1b480 {NSDebugDescription=The connection was invalidated from this process.})
    virtualbox-iso: Nov 18 08:24:58 osx-10_9.vagrantup.com softwareupdate CLI[588] <Error>: /usr/sbin/softwareupdate: XPC error in __60-[SUCommandLineTool _runSessionWithUpdates:skippingInstall:]_block_invoke (Error Domain=NSCocoaErrorDomain Code=4099 "Couldn’t communicate with a helper application." (The connection was invalidated from this process.) UserInfo=0x7fa201c1b480 {NSDebugDescription=The connection was invalidated from this process.})

I'd be curious to see the exit code from softwareupdate if you can reproduce this. People who manage large fleets of Macs saw some intermittent errors like this specifically with Mavericks.

So, just to make sure: just log into the box and try softwareupdate manually, let you know what happens?

Strange: this time no errors were reported while creating the box:
2015/11/19 10:50:54 ui:     virtualbox-iso: Downloading and installing system updates...
    virtualbox-iso: Downloading and installing system updates...
2015/11/19 10:51:06 ui:     virtualbox-iso: Software Update Tool
    virtualbox-iso: Software Update Tool
2015/11/19 10:51:06 ui:     virtualbox-iso: Copyright 2002-2012 Apple Inc.
    virtualbox-iso: Copyright 2002-2012 Apple Inc.
2015/11/19 10:51:06 ui:     virtualbox-iso:
    virtualbox-iso:
2015/11/19 10:51:06 ui:     virtualbox-iso: Finding available software
    virtualbox-iso: Finding available software
2015/11/19 10:51:06 ui:     virtualbox-iso:
    virtualbox-iso:
2015/11/19 10:51:06 ui:     virtualbox-iso: Downloading OS X Update Combined
    virtualbox-iso: Downloading OS X Update Combined
2015/11/19 10:51:06 ui:     virtualbox-iso: Downloading iBooks Update
    virtualbox-iso: Downloading iBooks Update
    virtualbox-iso: Downloading iTunes
2015/11/19 10:55:10 ui:     virtualbox-iso: Downloading iTunes
2015/11/19 10:55:10 ui:     virtualbox-iso: Downloaded OS X Update Combined
    virtualbox-iso: Downloaded OS X Update Combined
2015/11/19 10:55:10 ui:     virtualbox-iso: Downloaded iBooks Update
    virtualbox-iso: Downloaded iBooks Update
2015/11/19 10:55:10 ui:     virtualbox-iso: Downloaded iTunes
    virtualbox-iso: Downloaded iTunes
    virtualbox-iso: Installing OS X Update Combined, iBooks Update, iTunes
2015/11/19 11:03:17 ui:     virtualbox-iso: Installing OS X Update Combined, iBooks Update, iTunes
2015/11/19 11:03:17 ui:     virtualbox-iso: Done with OS X Update Combined
    virtualbox-iso: Done with OS X Update Combined
2015/11/19 11:03:17 ui:     virtualbox-iso: Done with iBooks Update
    virtualbox-iso: Done with iBooks Update
2015/11/19 11:03:17 ui:     virtualbox-iso: Done with iTunes
    virtualbox-iso: Done with iTunes
2015/11/19 11:03:17 ui:     virtualbox-iso: Done.
    virtualbox-iso: Done.
2015/11/19 11:03:17 ui:     virtualbox-iso:
    virtualbox-iso:
2015/11/19 11:03:17 ui:     virtualbox-iso: You have installed one or more updates that requires that you restart your
    virtualbox-iso: You have installed one or more updates that requires that you restart your
2015/11/19 11:03:17 ui:     virtualbox-iso: computer.  Please restart immediately.
    virtualbox-iso: computer.  Please restart immediately.

However, logging into the box and running softwareupdate -i -a a bunch of packages are downloaded and installed. I suppose this is the more normal situation... Especially as one of the updates needed a restart...


The creepers' explosion was much wider

As usual, the explosion was small than installed mod
Like using Flint and Steel

fixed in 1.5-beta.2


Update-AdfsArtifactDatabasePermission not available with Server 2019

The cmdlet is not available with ADFS 2019: Update-AdfsArtifactDatabasePermission
and probably not needed?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: f0238d6f-4073-172e-de62-e2d973d993b7
Version Independent ID: 3789bcce-bebd-8a9d-c814-45ce6c1bfe47
Content: Configure AD FS Extranet Smart Lockout Protection
Content Source: WindowsServerDocs/identity/ad-fs/operations/Configure-AD-FS-Extranet-Smart-Lockout-Protection.md
Product: windows-server
Technology: ad-fs
GitHub Login: @billmath
Microsoft Alias: billmath

#assign:robinharwood
Thanks for your dedication to our documentation. Unfortunately, at this time we have been unable to review your issue in a timely manner and we sincerely apologize for the delayed response. The requested updates have not been made, although there have been further updates with subsequent releases which supersede the updates in your issue. We are closing this issue for now, but if you feel that it's still a concern, please respond and let us know. If you determine another possible update to our documentation, please don't hesitate to reach out again. #please-close


Update libcalico-go to latest

Also, pin runc to avoid incompatibility between it and kubernetes

LGTM.


fix test typo

fix test calling wrong function.
https://travis-ci.org/CargoSense/ex_aws/builds/231123985?utm_source=github_status&utm_medium=notification

yeah just saw that, thanks for the fix!


Docs: Add missing closing curly brace to @service

Add a missing curly brace in the @service decorator.

/cadleng format

Thanks kai687 for the fix!


Droplet deletion: File name too long

Hi all,
we are running CF 228 and have encountered an issue on one of our runners. In /var/vcap/data/dea_next/droplets, there are a lot of directories with multiple ".deleted." suffixes:
drwxr-xr-x  2 root root  4096 Feb 10 09:45 b636736c01441e9ecbb37349347049f7efff26a7/
drwxr-xr-x  2 root root  4096 Feb  9 23:04 bb12bd92715ea80dd669e8acff5466c84b6486c6.deleted.1455059188.deleted.1455059332.deleted.1455059481/
drwxr-xr-x  2 root root  4096 Feb  9 21:53 c6180fc237fd50d6df29a9af5e4fd80020a22b02.deleted.1455054955.deleted.1455055101.deleted.1455055550/
drwxr-xr-x  2 root root  4096 Feb 11 09:40 cead64c00362bfd4414fc1656192a47dbad50275/
drwxr-xr-x  2 root root  4096 Feb 11 06:55 d598dbd50b5132716b57318c570c7ed9b82070f5.deleted.1455173880.deleted.1455173952.deleted.1455174027.deleted.1455174106/

Some directory names have become too long and can't be deleted by the dea_next job. The dea_ctl.err.log is full of such entries:
[2016-02-11 10:19:57+0000] ------------ STARTING dea_ctl at Thu Feb 11 10:19:57 UTC 2016 --------------
[2016-02-11 10:20:58+0000] Encountered error: File name too long @ sys_fail2 - (/var/vcap/data/dea_next/droplets/9db700956fb72ff889f34708c9c029a89f793297.deleted.1455045650.deleted.1455045803.deleted.1455045951.deleted.1455046135.deleted.1455046281.deleted.1455046467.deleted.1455046622.deleted.1455046772.deleted.1455047111.deleted.1455047260.deleted.1455047559, /var/vcap/data/dea_next/droplets/9db700956fb72ff889f34708c9c029a89f793297.deleted.1455045650.deleted.1455045803.deleted.1455045951.deleted.1455046135.deleted.1455046281.deleted.1455046467.deleted.1455046622.deleted.1455046772.deleted.1455047111.deleted.1455047260.deleted.1455047559.deleted.1455186058)
[2016-02-11 10:20:58+0000] ------------ STARTING dea_ctl at Thu Feb 11 10:20:58 UTC 2016 --------------
[2016-02-11 10:21:59+0000] Encountered error: File name too long @ sys_fail2 - (/var/vcap/data/dea_next/droplets/9db700956fb72ff889f34708c9c029a89f793297.deleted.1455045650.deleted.1455045803.deleted.1455045951.deleted.1455046135.deleted.1455046281.deleted.1455046467.deleted.1455046622.deleted.1455046772.deleted.1455047111.deleted.1455047260.deleted.1455047559, /var/vcap/data/dea_next/droplets/9db700956fb72ff889f34708c9c029a89f793297.deleted.1455045650.deleted.1455045803.deleted.1455045951.deleted.1455046135.deleted.1455046281.deleted.1455046467.deleted.1455046622.deleted.1455046772.deleted.1455047111.deleted.1455047260.deleted.1455047559.deleted.1455186119)

CPU load is very high and we assume that apps deployed to this runner are failing. We could resolve the issue by re-creating the runner VM, but we would like to understand why this problem occurred and how it can be prevented in the future?
Thanks for your assistance and best regards,
Jochen.

I was wondering if you have any errors reported in the dea_next.log?
What seemed to happen is that the deletion failed the first time and it continues to append each time the DEA restarts.
I am curious what happened the very first time that prevented the deletion from completing.
We will be putting in guards to prevent the constant name growth but that doesn't fix the root problem.

We had to recreate the DEA VM to fix the problem. I can't remember if there were any errors reported in the dea_next.log. If the problem occurs again, we'll try to save all log files for analysis.
Thanks for your support!

Fix delivered.


docs: updated UG and personal PPP

Merge this first so everyone can pull the UG and edit it where necessary. I've already copied our UG draft over.

Codecov Report

Merging #15 (84d395d) into master (52b1800) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master      #15   +/-   ##
=========================================
  Coverage     72.15%   72.15%
  Complexity      399      399
=========================================
  Files            70       70
  Lines          1232     1232
  Branches        125      125
=========================================
  Hits            889      889
  Misses          311      311
  Partials         32       32

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 52b1800...84d395d. Read the comment docs.

LGTM!

Closes #17


Adding installation instructions in Chinese

Hiya!
I can see how this project can benefit from a Chinese version for the README and or installation instructions. The repo is used by international corporations and international users worldwide. Although English is primarily recognized, there may be users who will better benefit from a Chinese translation.
URL
https://github.com/openstreetmap/iD/blob/develop/README.md
How to reproduce the issue?
n/a
Screenshot(s) or anything else?
n/a
Which deployed environments do you see the issue in?
The most up to date version of iD
What version numbers does this issue effect?
The most up to date version of iD
Which browsers are you seeing this problem on?
Chrome

What languages would the readme need too be translated into, and who would both translate it and keep it up to date?

I'm interested in working on this issue and would appreciate being assigned to it.


Add some french translation.

All is in the title.

Poedit crop line are fixed, and I add spaces before the ":"
The fuzzy was intentional, I can't say if this is "la scène" or "une scène" without context.

Great, thanks :)


defaultSelectedUsers for Person field on Dynamic Form

Thank you for reporting an issue, suggesting an enhancement, or asking a question. We appreciate your feedback - to help the team understand your
needs please complete the below template to ensure we have the details to help. Thanks!
Please check out the documentation to see if your question is already addressed there. This will help us ensure our documentation is up to date.
Category
[ ] Enhancement
[ ] Bug
[ X ] Question
Version
Please specify what version of the library you are using: [ 3.7.2 ]
If you are not using the latest release, please update and see if the issue is resolved before submitting an issue.
Expected / Desired Behavior / Question
If you are reporting an issue please describe the expected behavior. If you are suggesting an enhancement please
describe thoroughly the enhancement, how it can be achieved, and expected benefit. If you are asking a question, ask away!
Hello,
The DynamicForm Component is great and saves me the trouble of manually building a new/edit form for my project. I have a requirement to populate a person field when the DynamicForm is being used as a new item form (no item id supplied). I see the PeoplePicker Component has a property of defaultSelectedUsers to achieve this on a field level. I would like to do the same for a person column while using the DynamicForm Component.
Is their a supported way of achieving this?
Best, Dan

It would be nice to generalize this and be able to not only edit values in onSubmitted after the user has finished filling the form but to set initial values too.


NPM Deployment

@stoneman1, should we deploy to NPM under a different name to stop this confusion between pointing to master branch? @yamill doesn't seem to be active at all in GitHub and I doubt we'd be able to have him move the react-native-orientation npm package to one of us. What are your thoughts? We could do react-native-device-orientation or something along those lines, but then the name will differ from the GitHub package name.

I think we should ask @yamill to allow us to do updates on npm. If nothing happens next week perhaps we have to make a new npm package.

@stoneman1 cool that sounds like a plan.
@yamill, what are your thoughts? Any chance you can hook us up with access to do NPM updates? Should be a quick 5 second thing really.

@stoneman1 Please publish a new npm package if you can.

Seems like we can't catch @yamill, so let's do our own npm package?  We probably should fork this into a new repo then to match it with the package name?

@stoneman1 I guess we could write that up in the README file. What name are you thinking? react-native-orient & react-native-orientation & react-native-orientation-controller are all taken.

Why is there so many packages? :(

I'm about to get react-native-device-orientation. That's the best one so far.

It is taken already :(

Maybe I try to email @yamill ?

Ah damn. Yamill doesn't seem to be repsonsive or interested in dealing with this Open Source stuff. I'm guessing he's also got some issues with doing OSS while working for SnapChat.

react-native-orientation-handler? lol
react-native-orientation-package? lol

Not good names :D

Yeah I know. Just thinking of anything at this point.

Sent @yamill a email and explaining the situation.

Alright cool. Sounds good.

Any updates? Would really love to import this properly :)

@andrerfneves Maybe we fork it and use name react-native-orientation2?

Hi, I can give access to the npm. Can you list the usernames who need access? Thanks!

Just gave access to @stoneman1 and @andrerfneves

@yamill thank you for the access. We should now be good to move along with our deployment. A release will be made. Be in the lookout for it on an updated README. @stoneman1 I can probably branch a release tomorrow. Closing this for now.


Minimal cargo-c support

Fixes #437

@lu-zero I think we need to preserve the Makefile because you want to be able to run the same tests with both Limbo and SQLite?

Sure, I can expand it to use pkg-config if you deem it useful.


restrict-steps-plugin hosting request

Repository URL
https://github.com/samrocketman/restrict-steps-plugin
New Repository Name
restrict-steps-plugin
Description
I wrote a plugin for https://issues.jenkins.io/browse/JENKINS-69606 and am ready to publish the POC plugin.
GitHub users to have commit permission
@samrocketman
Jenkins project users to have release permission
sag47
Issue tracker
Jira

Security audit, information and commands
The security team is auditing all the hosting requests, to ensure a better security by default.
This message informs you that a Jenkins Security Scan was triggered on your repository.
It takes ~10 minutes to complete.
Commands
The bot will parse all comments, and it will check if any line start with a command.
Security team only:

/audit-ok => the audit is complete, the hosting can continue :tada:.
/audit-skip => the audit is not necessary, the hosting can continue :tada:.
/audit-findings => the audit reveals some issues that require corrections :pencil2:.

Anyone:

/request-security-scan => the findings from the Jenkins Security Scan were corrected, this command will re-scan your repository :mag:.
/audit-review => the findings from the audit were corrected, this command will ping the security team to review the findings :eyes:.
	It's only applicable when the previous audit required changes.

Only one command can be requested per comment.

(automatically generated message, version: 1.19.2)

:x: Jenkins Security Scan failed.
The Security team was notified about this.

I don’t want the repository to be forked.  That affects contribution stats

The repository will be forked and then you delete the original which will make it the root of the network.
You need to post the github repository link in the description so that hosting can continue

I don’t want the repository to be forked. That affects contribution stats

Once forked and deleting the upstream repository, doesn't GitHub keep track of all past contributors? git history isn't affected by it 🤔
Could you add the repository URL nonetheless please, allowing automatic tooling to run? We can see about going a different route later.

I didn't consider that thanks.  I'll publish later

/hosting re-check

@timja @NotMyFault okay plugin published

/request-security-scan

The Jenkins Security Scan did not find anything dangerous with your plugin, congratulations! :tada:

:bulb: The Security team recommends that you are setting up the scan in your repository by following our guide.

My current plan is to release once manually and then onboard it to auto-release

Looks all good to me, but two recommendations:

https://github.com/samrocketman/restrict-steps-plugin/tree/main/src/main/groovy/net/gleske/restrictsteps diverges from the groupId specified in the pom.xml, better move it to io.jenkins.plugins.restrictsteps
I can see a few spotbugs issues left to address, you can generate a list via mvn clean verify

@NotMyFault I can move the package to that convention; I typically use my domain but I understand this helps make it obvious that an imported class (in other plugins) is coming from an actual Jenkins plugin.  I'll be able to make this change in the evening (as I don't normally code during work hours on OSS projects).

@NotMyFault https://github.com/gradle/gradle/issues/8317 it's groovy so it triggers SE_NO_SERIALVERSIONID all the time.  For example, you can't do someMap.each { k, v -> ... }.  It feels silly to restrict my programming in Groovy due to this spotbugs quirk.

callSiteArray appears to be an under the hood feature I can't control as part of groovy https://stackoverflow.com/questions/67825890/groovy-getcallsitearray-implementation

Here's the remaining spotbugs which are not possible to resolve
[ERROR] Medium: Incorrect lazy initialization of static field io.jenkins.plugins.restrictsteps.RestrictStepsListener.$callSiteArray in io.jenkins.plugins.restrictsteps.RestrictStepsListener.$getCallSiteArray() [io.jenkins.plugins.restrictsteps.RestrictStepsListener] In RestrictStepsListener.groovy LI_LAZY_INIT_STATIC
[ERROR] Medium: io.jenkins.plugins.restrictsteps.RestrictStepsListener$_resolveRestricted_closure1 is Serializable; consider declaring a serialVersionUID [io.jenkins.plugins.restrictsteps.RestrictStepsListener$_resolveRestricted_closure1] At RestrictStepsListener.groovy:[line 76] SE_NO_SERIALVERSIONID

It's possible newer versions of Groovy handle this somehow but Jenkins remains on Groovy 2.4.21.

I've:

Renamed the package to io.jenkins.plugin.restrictsteps
Fix all the spotbugs errors that I feasibly could.
Updated license text.

@NotMyFault gradle/gradle#8317 it's groovy so it triggers SE_NO_SERIALVERSIONID all the time. For example, you can't do someMap.each { k, v -> ... }. It feels silly to restrict my programming in Groovy due to this spotbugs quirk.

Oic, can you annotate the method with @SuppressFBWarnings to put a local suppression in place? Releases won't be successful if the build is failing.

Hey! Thanks for the tip! I'll see if I can also figure out how to exclude just those two errors via spotbugs config because in general I like spotbugs operating on the code.  It did find some legitimate issues.

Two things that are neat.  Rather than using suppress warnings one can use the GUI via mvn spotbugs:gui.

The GUI has a function to mark bugs for exclusion (right click on the bug and mark for exclusion; a popup will give you a series of checkboxes to add as match rules for the generated exclusion).
The GUI can export exclusions to a Filter file.

With the filter file it gets me 90% there.  I can tweak the file for generic Groovy classes and apply the configuration via spotbugs maven plugin config excludeFilterFile.  I'll let you know when I'm able to update this.  I'll be off from work at around 1pm ET today so I can push this earlier than usual.

@NotMyFault I believe the remaining issues are resolved.

/request-security-scan

/hosting re-check

The Jenkins Security Scan did not find anything dangerous with your plugin, congratulations! :tada:

:bulb: The Security team recommends that you are setting up the scan in your repository by following our guide.

/hosting host

:tada:


(MODULES-773) Use mco_array_to_string() in more locations

The fix committed in e750b674f7f0 is on the right track, but incomplete. This
commit uses mco_array_to_string() in three more locations affected by the same
bug, allowing the module to work with the future parser as of Puppet 3.7.0.

This looks good to me. @hunner are those uses of the function valid?
I'd love the ability to write tests that expect function calls upon class compilation, as pure rspec tests do for Ruby code. rspec-puppet cannot do this currently, nor are there plans. So this should not be a blocker.
Can there be sensible tests to see whether the results of those calls are used properly?

Looks like this was already pulled in commit 1899dce6a2e39, so I'll close this pull request.


Can't update the KYC document

Hello, guys!
I am trying to update the KYC document like this written in the documentation:
$KycDocument->Type = "IDENTITY_PROOF";
$Result = $mainmp->create_kyc_document($UserId, $KycDocument); // returns KYC document
$Page = $mainmp->create_kyc_page_from_file($UserId, $Result->Id, $file); // returns true

$KycDocument->Status = \MangoPay\KycDocumentStatus::ValidationAsked;
$ResultUpdate = $mainmp->update_kyc_document($UserId, $KycDocument); // returns error

But I am getting this error:
Fatal error: Uncaught MangoPay\Libraries\ResponseException: Method not allowed. The requested resource does not support http method 'PUT'. in /sdk/MangoPay/Libraries/RestTool.php:366
Stack trace:
#0 /sdk/MangoPay/Libraries/RestTool.php(152): MangoPay\Libraries\RestTool->CheckResponseCode(405, Object(stdClass))
#1 /sdk/MangoPay/Libraries/ApiBase.php(346): MangoPay\Libraries\RestTool->Request('/users/96113714...', 'PUT', Array)
#2 /sdk/MangoPay/ApiUsers.php(282): MangoPay\Libraries\ApiBase->SaveObject('kyc_documents_s...', Object(MangoPay\KycDocument), '\\MangoPay\\KycDo...', 96113714)

Please, help.

Hello @khrigo,
Thanks for reaching us. Are you using the last version of the SDK ?
As you can see below, you should use UpdateKycDocument in User class to submit the KYC document to validation.
//submit the doc for validation
$KycDocument = new MangoPay\KycDocument();
$KycDocument->Id = $KycDocumentId;
$KycDocument->Status = \MangoPay\KycDocumentStatus::ValidationAsked;
$result3 = $mangoPayApi->Users->UpdateKycDocument($_SESSION["MangoPayDemo"]["UserNatural"], $KycDocument);

More demos about KYC in demos/worflow/
Tell me if it's fix your issue.

Hello @khrigo,
Thanks for reaching us. Are you using the last version of the SDK ?
As you can see below, you should use UpdateKycDocument in User class to submit the KYC document to validation.
//submit the doc for validation
$KycDocument = new MangoPay\KycDocument();
$KycDocument->Id = $KycDocumentId;
$KycDocument->Status = \MangoPay\KycDocumentStatus::ValidationAsked;
$result3 = $mangoPayApi->Users->UpdateKycDocument($_SESSION["MangoPayDemo"]["UserNatural"], $KycDocument);

More demos about KYC in demos/worflow/
Tell me if it's fix your issue.

Hello @fredericdelordm,
Yes, it is the last version. I just use WordPress plugin and use update_kyc_document which has:
public function update_kyc_document($userId,$KycDocument){
     return $this->mangoPayApi->Users->UpdateKycDocument($userId,$KycDocument);
}

@khrigo
you are using the official plugin : https://fr.wordpress.org/plugins/mangopay-woocommerce/ ? With WC Vendors ?
If yes, you should open a topic in the official forum : https://wordpress.org/support/plugin/mangopay-woocommerce/.
If not, let me know.

@fredericdelordm I have a problem with PHP SDK, not in the plugin, I send you an error.

Our tech team is going to review this and will get back to you soon.
Have a nice day,

Hi @khrigo
`

@param string $userId User Id
@param \MangoPay\KycDocument $kycDocument Document to save
@return \MangoPay\KycDocument Document returned from API`

If you take a look at this, you will find that the first parameter needs to be a string, not an object.

Hi @khrigo
`

@param string $userId User Id
@param \MangoPay\KycDocument $kycDocument Document to save
@return \MangoPay\KycDocument Document returned from API`

If you take a look at this, you will find that the first parameter needs to be a string, not an object.

If you check my request, you can see what user_id is a string.

@fredericdelordm please, open the issue.

@fredericdelordm @SoloJr any news?

Hello @khrigo,
We cannot reproduce your error. The SDK is doing a PUT request as requested by the documentation.
Are you sure this error is coming from the SDK call to update the KYC Document status? Do you have more logs?
Just to be sure. Are you creating the KYC Document correctly (example here) before trying the update?
Regarding your code below, I think you are not creating the document as you should.
$KycDocument = new MangoPay\KycDocument();
$KycDocument->Id = $KycDocumentId;


Failing builds

due to https://github.com/ai/autoprefixer-rails/issues/203
project is using middleman-autoprefixer
According to their changelog, latest version is 3.0.0 which includes autoprefixer v10, but rubygems only lists v2.10.1

middleman-autoprefixer updated the changelog to mention version 3.0.0 in November 2020, but haven't released that version to rubygems or even tagged it in GitHub :/ It seems unmaintained to me, perhaps we should fork it?

Ah, we might be able to get around it by downgrading middleman-prefixer to 2.10.0, which uses a more optimistic version spec https://github.com/middleman/middleman-autoprefixer/compare/v2.10.0...v2.10.1.
I'm trying this in #233.


Support building sdks that can build self-contained / ReadyToRun / NativeAOT applications without using pre-builts from nuget.org

Thanks to the reference packs, a source-build sdk can now build .NET Core applications that can be considered themselves source-buildable.
This doesn't apply to self-contained applications, and ReadyToRun applications. These require downloading a Microsoft.NETCore.App.Runtime.<rid> package from nuget.org. (This includes framework dependent ReadyToRun, probably because it uses the package to aquire crossgen.)
This issue is for building and including the Runtime package with the sdk. This makes it possible for the maintainer to apply patches during source-build, and the applications built this way using the sdk can be considered source-buildable.
cc @dagood @nguerrera @omajid

This issue is for building and including the Runtime package with the sdk.

Runtime packs are similar to targeting packs, so IMO they should be included in e.g. dotnet-runtime-pack-x-y packages, which the SDK package would have a dependency on.

This makes it possible for the maintainer to apply patches during source-build, and the applications built this way using the sdk can be considered source-buildable.

A note about this from https://github.com/dotnet/source-build/issues/1202, to put it in context: the runtime pack has limitations on what it supports, due to glibc compatibility and portable vs. non-portable builds. An SCD publish for linux-x64 would need to bring down the extra-compatible prebuilt, but e.g. fedora.30-x64 would be source-built.

I believe the SDK and perhaps Core-SDK repos will need to add support for this. The SDK currently locates targeting packs with TargetingPackRoot ($(NetCoreTargetingPackRoot)) which points to e.g. /usr/bin/dotnet/packs/. The runtime pack resolver task doesn't seem to have anything similar.
The definition of NetCoreTargetingPackRoot is over in Core-SDK.
The apphost pack resolver uses TargetingPackRoot too, despite the name, so I guess it might be fine to use it for runtime packs too.

It makes sense to make it work similar to the apphost package.
Can we set a target milestone for this?
I can probably implement this based on these references.

Adding @dsplaisted

The current .NET 6 SDK already checks for runtime packs in the packs/ subdirectory, it just doesn't find any because none were installed.  I've been able to do using the following straightforward patch to the installer:
Index: installer/src/redist/targets/GenerateLayout.targets
===================================================================
--- installer.orig/src/redist/targets/GenerateLayout.targets
+++ installer/src/redist/targets/GenerateLayout.targets
@@ -134,6 +134,13 @@
         <RelativeLayoutPath>packs/%(PackageName)/%(PackageVersion)</RelativeLayoutPath>
       </BundledLayoutPackage>

+      <BundledLayoutPackage Include="MicrosoftNetCoreAppRuntimePackNupkg">
+        <PackageName>Microsoft.NETCore.App.Runtime.$(SharedFrameworkRid)</PackageName>
+        <PackageVersion>$(MicrosoftNETCoreAppRefPackageVersion)</PackageVersion>
+        <TargetFramework>$(TargetFramework)</TargetFramework>
+        <RelativeLayoutPath>packs/%(PackageName)/%(PackageVersion)</RelativeLayoutPath>
+      </BundledLayoutPackage>
+
       <BundledLayoutPackage Include="MicrosoftNetStandardTargetingPackNupkg">
         <PackageName>NETStandard.Library.Ref</PackageName>
         <PackageVersion>$(NETStandardLibraryRefPackageVersion)</PackageVersion>
@@ -146,6 +153,13 @@
         <PackageVersion>$(MicrosoftAspNetCoreAppRefPackageVersion)</PackageVersion>
         <TargetFramework>$(TargetFramework)</TargetFramework>
         <RelativeLayoutPath>packs/%(PackageName)/%(PackageVersion)</RelativeLayoutPath>
+      </BundledLayoutPackage>
+
+      <BundledLayoutPackage Include="MicrosoftAspNetCoreAppRuntimePackNupkg">
+        <PackageName>Microsoft.AspNetCore.App.Runtime.$(SharedFrameworkRid)</PackageName>
+        <PackageVersion>$(MicrosoftAspNetCoreAppRefPackageVersion)</PackageVersion>
+        <TargetFramework>$(TargetFramework)</TargetFramework>
+        <RelativeLayoutPath>packs/%(PackageName)/%(PackageVersion)</RelativeLayoutPath>
       </BundledLayoutPackage>

       <BundledLayoutPackage Include="MicrosoftNetCoreAppHostPackNupkg">

This likely still needs a bit of tweaking to use an appropriate condition when to do it.   Also, it currently installs only the $(SharedFrameworkRid) ... maybe it also should install alternate RIDs as is done for the Apphost?
As a drawback, this approach significantly increases the size of the SDK installation on disk.  This is annoying in particular as most of the files installed as part of the target runtime pack are in fact already present in the SDK on-disk installation, under the shared/ directory ... but it looks like that cannot be used as a drop-in replacement because the directory structure is different.

.NET 7 will add support for NativeAOT.
We'd like a source-build SDK to be able to build self-contained NativeAOT applications.
This issue was already tracking adding support for self-contained, and I've added NativeAOT in the title too. We can split it up when that makes sense.

I've summarized what I believe to be the current state of play and the agreed-upon solution for NativeAOT in 7.0 in https://github.com/dotnet/runtime/pull/76206.  Please check this out and let me know if you have any concerns.

@MichaelSimons you were asking about this issue earlier today?

@tmds would you agree this is more important than supporting building and using workloads from source?

Yes. These features apply to any app, while the workloads are for specific use-cases.

@richlander I'm wondering, do you think it would be beneficial in the future if Microsoft sdk images include some of these features (so: without having to reach out to nuget.org) as the features are becoming more prevalent?

For self-contained deployment -
Self-contained deployment acquires runtime packs and AppHost packs (cross-target) from NuGet (MS Built). These are not packaged with SDK due to the size constraint. And so, as the packages are not offered with Linux distribution-built SDK as well and end-up acquired from the NuGet. To acquire packages which are source-built by Linux distributions, here are few options.

Package with SDK (if small size). Starting .NET 6, SDK does check for runtime packs and AppHost packs in dotnet/packs folder first before acquiring from NuGet. Alternatively, when user opt to use Linux package manager to get those packages, by default Install to dotnet/packs folder.
Another option is to build and offer those as NuGet package, but I’m not sure if maintainers will prefer this option.

A few points to note –

Linux distributions only build packages specific to their distribution. As in RHEL only builds runtime pack rhel-x64 and not linux-x64 or ubuntu-x64. So, the size of packages to offer must be smaller compared to Microsoft Build.
Source-built does offer Linux distros to build AppHost and Runtime packages.
For cross-targeting scenarios, ex- user building on rhel-x64 and target platform being ubuntu-x64. This would still require pulling packages from NuGet.org (MS Built) as Linux distributions do not build packages for other distributions. So, if packages are not acquired with distributions preferred mechanism, then the packages will be acquired using NuGet.org.

These are not packaged with SDK due to the size constraint.

I was wondering about just that size issue.  The main problem appears to me that if we place a copy of the Microsoft.NETCore.App.Runtime.<rid> package under the packs/ directory, then all those files are present twice: under packs and under shared/Microsoft.NETCore.app.
Is this duplication really necessary?   Would it be possible to rework the dotnet loader so it takes all the runtime assemblies (and other files) from the packs/ directory if present, so we would then just not need the shared/ directory at all?  Or am I missing something here?

I also think size is an important factor, esp if we'd like to include these features in a container image.
The duplication isn't needed, but it comes from staying close to how things work now with the nuget.org packages.
And since that is working well, I imagine there isn't much interest upstream in exploring the design to minimize the size. And besides the effort, there's also the risk of breaking things by entangling them.

For self-contained deployment

What about R2R and NativeAOT?

The idea has come up before that we could use the shared framework from shared instead of the runtime pack from packs in order to save space or downloads.  I think that even though they are mostly the same, there are still some differences.  It's possible those differences wouldn't apply to source-built versions of the .NET SDK.  I think @jkotas knows the details.

We do not want to have hard coupling between the runtime used by the build output with the runtime that that the SDK happens to run on. The output of the build can use different (servicing or major) version, different architecture, different runtime flavor (mono, aot, ...), etc.
In theory, we can special case the case where the shared runtime exactly matches the runtime that the self-contained deployment wants to use, and skip the download of the runtime pack in that case. We did not think that the self-contained publish to be common enough for this special casing and all complications that come with it to be worth it.

I also think size is an important factor, esp if we'd like to include these features in a container image.

This size factor can be fixed by linking the files to the same physical file.

This size factor can be fixed by linking the files to the same physical file.

I was wondering about that.
Whether it could be symbolic links, or have to be hard links?
And if the runtime repo could provide such an output, or whether source-build repo could/should do it?

Whether it could be symbolic links, or have to be hard links?

I do not know. Some tools may have problems with symbolic links.

And if the runtime repo could provide such an output, or whether source-build repo could/should do it?

Build of any more complicated app ends up restoring something from nuget. I do not think we want to include more stuff in the default SDK or otherwise complicate things to fine-tune the amount of stuff that gets downloaded. For example, we have long-standing issue in that we download all runtime packs (even ASP.NET Core and Windows Deskop packs) when publishing self-contained console apps that we did not care enough to fix.
If the distro builds want to include more packages in place (and potentially optimize the amount of diskspace used by linking), I think it would be a specific to just the distro builds.

Some tools may have problems with symbolic links.

Yes, symbolic links may not work for different reasons.
And, hard links come with a packaging challenge because the hard links link to files that are in a different package.
For example, a pack package with a hard link to a file that is provided by a runtime package.
Though the archive format may support this, the default archive tooling doesn't*.
So packages with hard links need to be crafted.
(*: it can't distinguish between what package provides, and what package hard links)

And, hard links come with a packaging challenge because the hard links link to files that are in a different package. For example, a pack package with a hard link to a file that is provided by a runtime package.

Couldn't the runtime package just automatically always provide both the shared runtime and the runtime pack?  If this is hard links, there wouldn't be any disk space reason not to, right?

Couldn't the runtime package just automatically always provide both the shared runtime and the runtime pack? If this is hard links, there wouldn't be any disk space reason not to, right?

Yes, though the runtime pack is used by the sdk, we could ship it in the runtime package (with no additional size cost).
We also need to package the R2R and NativeAOT tooling.

@ashnaga @MichaelSimons can we for an upcoming preview of .NET 8 target to include the runtime pack as part of the vmr output?
Once we have that, we can then for the subsequent preview focus on reducing its size either by using symbolic or hard links.
And, can you propose how we could include R2R and NativeAOT?

can we for an upcoming preview of .NET 8 target to include the runtime pack as part of the vmr output?

I would expect that vmr is building the runtime pack and R2R .nupkg files. Is that not the case?

I would expect that vmr is building the runtime pack and R2R .nupkg files. Is that not the case?

Yes, some of these nuget packages get built.
The output of the vmr is "DOTNET_ROOT" file layout which is then packaged up by the distro maintainers.
The request include the files from the nuget package at the appropriate location.

@ashnaga @MichaelSimons can we for an upcoming preview of .NET 8 target to include the runtime pack as part of the vmr output?

Can we do this?

@ashnaga @MichaelSimons what's the next step?

@ashnaga is driving this one.  I will let her respond.

@tmds
Thank you for your patience. I made an evaluation to ensure that packages are picked from dotnet/packs folder if available and not being pulled from NuGet.
Currently, if there is a package with following naming convention - "dotnet/packs/Microsoft.NETCore.App.Runtime.linux-x64" then only it will be picked by SDK. The naming convention does not imply distribution/RID specific packages which we are using in case of AppHost packages - https://learn.microsoft.com/en-us/dotnet/core/distribution-packaging
In order to make it work for distribution/RID specific packages, the source-built version will need to change how the Bundled versions targets file is generated:
https://github.com/dotnet/installer/blob/7eea2f60fdcf419d65872c6173f2bdcd8381905a/src/redist/targets/GenerateBundledVersions.targets#L196-L212
@tmds
To make this work in preview timeframe, are you happen to make these changes?
cc @dsplaisted @MichaelSimons if you have specific questions on changes in the mentioned file.

To make this work in preview timeframe, are you happen to make these changes?

If no one else is available that is more familiar, I will look into it.
In the meanwhile we should try to determine what is needed to provide NativeAOT. Then we can start working on that too.

@ashnaga do you have a picture of what is needed for NativeAOT?
A heads up: I'll be on summer leave from mid July to mid August. It would be good if we can make some more progress before that.

do you have a picture of what is needed for NativeAOT?

Add dotnet/llvm-project to https://github.com/dotnet/dotnet, and make it build objwriter there. (It should be built before dotnet/runtime.)
Find and delete all places where we are disabling native AOT for source build. For example: https://github.com/dotnet/runtime/blob/aca0eab8fcc5e4c82f7895b017595788752dc3f7/src/coreclr/tools/aot/crossgen2/crossgen2.csproj#L7-L8
Bug-fix level work to make things actually work.
Figure out how to distribute the packages (discussion above)

Add dotnet/llvm-project to https://github.com/dotnet/dotnet, and make it build objwriter there. (It should be built before dotnet/runtime.)

This requires adding some things to installer src/SourceBuild/content/repo-projects.
With the move to the vmr, I don't know how to go about building that.
@MichaelSimons can you provide some help?

@tmds this should be a starting point for adding new repositories into Source Build: https://github.com/dotnet/source-build/blob/main/Documentation/sourcebuild-in-repos/new-repo.md

this should be a starting point for adding new repositories into Source Build: https://github.com/dotnet/source-build/blob/main/Documentation/sourcebuild-in-repos/new-repo.md

I'm looking for the command that you can run on the installer repo to make it build up the complete sources (which is what is put into dotnet/dotnet). I don't see it mentioned in these docs.

I think the right way is to clone the VMR, manually put the llvm in src/. Register the repository (repo-project...) and build it.
Once you get that working, we will open a PR against installer to put the repo-project file under src/VirtualMonoRepo and also change the source-mappings.json. Then, when you open a PR, the PR build will validate everything.
You can also start with a PR and work your way through that but it will possibly cost a lot of iterations and compute as the PR build builds much more than the VMR.

Couple of notes to the above: To make everything work in the PR build of installer, we will also have to make sure the LLVM flows into the VMR which means it would have to be declared as a dependency in one of the repositories (in eng/Version.Details.xml). It is still possible to have the following workflow:

Make a dev branch in a fork of installer
Declare the dependency on llvm-project there (in eng/Version.Details.xml)
Add the llvm-project repo to src/VirtualMonoRepo/source-mappings.json in the installer dev branch
Push the dev branch into your fork and start a Codespace (you have to select the VMR .devcontainer)

The installer VMR-specific Codespace is configured to create and sync the VMR into /workspaces/dotnet so it should bring in the llvm-project there.
In this workflow, you can:

Keep pushing new changes into the installer dev branch and awlays start a fresh Codespace
In the installer branch, you can already add the new repo-project files and everything - this way your changes will persist between attempts.
Build the VMR in the Codespace right away to test things as you go

Do not open an installer PR in this workflow - it's not required. Also I am not sure if you will need to have billing set up for the Codespace to work for you.

Something like the following, but what to use as the Name and Version?

See https://github.com/dotnet/installer/pull/14697 specifically https://github.com/dotnet/installer/pull/14697/files#diff-fb62e94a1d6f29f863e3d0a22aa38269f6cd1d7f03b109dc06e2cbf2548b86d3L20.  This shouldn't use the SourceBuildTarball element as we are trying to remove it.  Instead it should use SourceBuild but that requires LLVM be onboarded to source-build so that it has a source-build leg and publishes a source-build intermediate.

I've opened two PRs for getting llvm-project in the vmr: https://github.com/dotnet/installer/pull/16757, https://github.com/dotnet/llvm-project/pull/439.

@ashnaga @MichaelSimons @premun
Based on where we are with source-buildable NativeAOT, it makes sense to defer this to .NET 9 so we can focus our efforts on the upcoming .NET 8 release.

@jkotas @MichalStrehovsky
Does this merge of ObjWriter in C# impact the instruction on what is needed to enable NativeAOT with source-build instructions.
PRto build  llvm-project is closed now, considering this as an alternative.

We also need to some work so the ILCompiler pack becomes part of the source-built SDK that is built by the vmr.
The work should be similar to the runtime pack we've added in .NET 8 to enable self-contained publish.
I'll look into that after looking at the bug-fix.

I think the SDK needs these 3 NuGet packages to publish a NativeAOT application:

Microsoft.DotNet.ILCompiler
Microsoft.NET.ILLink.Tasks
runtime..Microsoft.DotNet.ILCompiler

Like with self-contained, we'd like to provide NativeAOT through distro packages.
For self-contained we placed the files from Microsoft.NETCore.App.Runtime.<rid> and Microsoft.AspNetCore.App.Runtime.<rid> under DOTNET_ROOT/packs.
I think the SDK only looks at DOTNET_ROOT/packs for specific types of packages, so we can't use the same mechanism for these 3 NuGet packages, can we?
Is there some other mechanism we can use?
cc @dsplaisted @jkotas

I think the SDK only looks at DOTNET_ROOT/packs for specific types of packages, so we can't use the same mechanism for these 3 NuGet packages, can we?
Is there some other mechanism we can use?
cc @dsplaisted @jkotas

I think we would need changes in the .NET SDK to enable this.  I think we should put these assets in the packs folder in the same way as the other assets, and modify the SDK to pick them up from there.
Basically use the same mechanism that we already use for other packs, but add support for using it for these ones.

I've opened 2 PRs: https://github.com/dotnet/runtime/pull/99148 and https://github.com/dotnet/installer/pull/18835.
With these PRs, the resulting SDK can be used with the 3 source-built NuGet packages (Microsoft.DotNet.ILCompiler, Microsoft.NET.ILLink.Tasks, runtime.<rid>.Microsoft.DotNet.ILCompiler) to publish NativeAOT apps!
I also did a second stage build and validated it works.

I think we would need changes in the .NET SDK to enable this. I think we should put these assets in the packs folder in the same way as the other assets, and modify the SDK to pick them up from there.

I'm not sure how this will look. Will the NuGet packages that are available through packs be defined in Microsoft.NETCoreSdk.BundledVersions.props? Should we create an issue for adding this feature to the SDK?

https://github.com/dotnet/runtime/pull/99148 and https://github.com/dotnet/installer/pull/18835.

These have been merged.
The next step is to work out how we deliver this in rpms (distro packages).
@dsplaisted is this something you can do? Or describe how it should work, so I can help implement it?

@dsplaisted @jkotas @baronfel
In case of Microsoft SDK, are these packages packaged with SDK or being acquired from NuGet during NativeAOT operations like build/publish?
Microsoft.DotNet.ILCompiler
Microsoft.NET.ILLink.Tasks
runtime..Microsoft.DotNet.ILCompiler

They are acquired from NuGet for Microsoft SDK. We do not want to include them in the default Microsoft SDK install due to size.

I was doing a bit of investigation into the ProcessFrameworkReferences MSBuild Task, which is where most of the 'should we download anything from NuGet' logic lives for the runtime packs and so on, and it doesn't appear like we even support loading Microsoft.DotNet.ILCompiler, Microsoft.NET.ILLink.Tasks, and runtime.**RID**.Microsoft.DotNet.ILCompiler from the packs directory at all - they are always assumed to be acquired from NuGet. So I expect that for a source-built SDK to light up in the presence of these packs we'd need to change ProcessFrameworkReferences to support locating Tool Packs from the TargetingPackRoot like we do for runtime packs and targeting packs

So I expect that for a source-built SDK to light up in the presence of these packs we'd need to change ProcessFrameworkReferences to support locating Tool Packs from the TargetingPackRoot like we do for runtime packs and targeting packs

Do we need a new ToolPackType? which? And would, like for the other packs, the packages be listed under Microsoft.NETCoreSdk.BundledVersions.props?

The design would be something like

Known Package References can either be located from the Workloads installed in a given SDK, the SDK's TargetingPackRoot, or from the configured user NuGet feeds. When locating a pack of any type, first the workloads installation pack directory will be probed for that pack, then the TargetingPackRoot location will be probed for that pack. If neither of those locations can satisfy the pack, the pack will be acquired as an implicit PackageReference

Practically speaking, this means that the method that currently unconditionally creates an implicit PackageReference for the ToolPacks would instead follow the pattern that runtime packs and targeting packs use in their matching methods:

Probe the pack paths for the pack
if found, set metadata that points to the statically-known location
If not found, and the current build environment supports package downloads, then create an implicit package reference for the pack

@baronfel can you also answer my questions here: https://github.com/dotnet/source-build/issues/1215#issuecomment-2020305034?

I don't think we'd need new toolpack types - the existing toolpacks just would all become viable targets for this kind of sdk-local location? @dsplaisted anything you see wrong with that? There's kind of a philosophical difference here where the MS-built SDKs wouldn't use this because we'd rather float to package feeds, but the source-built SDKs want to be more batteries-included, and this is our existing pattern for making batteries-included work.
cc @jkotas do you have any thoughts on this?

cc @jkotas do you have any thoughts on this?

I do not have anything to add.

Probe the pack paths for the pack

How does ProcessFrameworkReferences know what packages it should probe the packs directory for, and when?

How does ProcessFrameworkReferences know what packages it should probe the packs directory for, and when?

It's in the logic of ProcessFrameworkReferences-- when it gets to a pack it needs, it calls GetPackPath to see if it's already available in various pack folders, and if so resolves it there.  If not, then it adds a PackageDownload to download the pack.
Over time, the set of packs that the task would do this for has grown, as we had more scenarios for including different types of packs in the SDK install.
The problem with the ILCompiler and ILLink packs is that they are referenced as PackageReference instead of PackageDownload items and have MSBuild .props and .targets files in them that need to be imported.  We don't have a good way of importing targets files from something resolved to the packs folder by the ProcessFrameworkReferences task, as by the time the task runs evaluation has already happened and we can't then import an additional .targets file.
Some ideas for what we could do:

Move the .targets files for ILLink and ILCompiler from the NuGet packages to the .NET SDK.  The NuGet package would then be used just for delivering the tool binaries, but not the targets that call them.  I'm not sure how tightly coupled the targets are with the tools and how hard it would be to extract the targets separately.
Include the .nupkg file itself in the library-packs folder of the .NET SDK.  NuGet would restore it from there and extract it to the packages folder.  We might want to use a different package identity for the source-built versions of these packages so there's no chance that the Microsoft-built ones from NuGet would get restored instead.

I think the first option is overall better, but the second one may be easier to implement.

Move the .targets files for ILLink and ILCompiler from the NuGet packages to the .NET SDK. The NuGet package would then be used just for delivering the tool binaries, but not the targets that call them. I'm not sure how tightly coupled the targets are with the tools and how hard it would be to extract the targets separately

These targets are tightly coupled with the compiler binary. Doing this would effectively break the ability to override the NativeAOT package version to workaround bugs, test privates, etc.

We might want to use a different package identity for the source-built versions of these packages so there's no chance that the Microsoft-built ones from NuGet would get restored instead.

Also for the user, I think we should try to only use the source-built versions when the user explicitly opts in by specifying the source-built rid.

The problem with the ILCompiler and ILLink packs is that they are referenced as PackageReference instead of PackageDownload items and have MSBuild .props and .targets files in them that need to be imported. We don't have a good way of importing targets files from something resolved to the packs folder by the ProcessFrameworkReferences task, as by the time the task runs evaluation has already happened and we can't then import an additional .targets file.

Could we extend PackageReference so ProcessFrameworkReferences can output PackageReferences that are found as extracted packages in the packs folder?
        PackageReference=
            Microsoft.DotNet.ILCompiler
                    IsImplicitlyDefined=true
                    PrivateAssets=all
                    PackageDirectory=/usr/lib64/dotnet/packs/Microsoft.DotNet.ILCompiler/8.0.3
                    Version=8.0.3
...

Could we extend PackageReference so ProcessFrameworkReferences can output PackageReferences that are found as extracted packages in the packs folder

This suggestion is meant to meet the needs of making ProcessFrameworkReferences capable of using the source-built assets when needed, and continue use their .props and .targets as it is working now.
@dsplaisted @jkotas @baronfel what do you think?

Could we extend PackageReference so ProcessFrameworkReferences can output PackageReferences that are found as extracted packages in the packs folder

This suggestion is meant to meet the needs of making ProcessFrameworkReferences capable of using the source-built assets when needed, and continue use their .props and .targets as it is working now.
@dsplaisted @jkotas @baronfel what do you think?

No, NuGet doesn't support setting a PackageDirectory on a PackageReference where the package is already extracted.
It sounds like the best solution is probably to put the source-built ILCompiler and ILLink NuGet packages in the library-packs folder.

It sounds like the best solution is probably to put the source-built ILCompiler and ILLink NuGet packages in the library-packs folder (as .nupkg files).

There are some downsides to this:
The packages that are produced by source-build don't have stripped symbols which bloats them significantly. This is due to the stripping being performed at the end of the packaging process to produce separate debug packages.
In general (I don't know how it applies to these packages specifically) the source-build packages can be different from the Microsoft built packages due to limitations in what can be source-built. This can cause functional differences.
Afaik, NuGet doesn't have a preference between different feeds. We can't force it to prefer library-packs over nuget.org packages.
With the files held in a nupkg archive, we can't try to optimize the size by referencing common files like we have done when adding support for self-contained publish.

No, NuGet doesn't support setting a PackageDirectory on a PackageReference where the package is already extracted.

Is it a feature we can consider?

@dsplaisted and I chatted about this just now and we see two potential solutions, with short- and long- time frames:
Short term
We think the least-friction, least-dependency way to get this working is to use library-packs. This would imply source-build producing the packages above (cc @jkotas - is there any set of functionality that source-built versions of these packages would lack compared to the MS-produced nuget.org versions?), and changing the names of the two without runtime-specific package identifiers. This would allow us to not have to worry about content mismatches if/when the MS-produced packages would be used. You would need to handle the symbol concerns you mention above, but the up-side to this is that you would be entirely in control - no need to wait on any other teams inside Microsoft to unblock this functionality.
Longer term
Longer term, we'd want to work with the IL Compiler team to split out the contents of the above packages into completely separate components for build logic vs tools/binaries, ensuring that the tools are more standalone. With this done, they could be treated as Tool Packs in the ProcessFrameworkReferences Task and just laid down into an expected place in the SDK for the build logic to assume.

is there any set of functionality that source-built versions of these packages would lack compared to the MS-produced nuget.org versions?

They should be identical.

changing the names of the two without runtime-specific package identifiers

The names of these packages are hardcoded in number of other places. Would this mean that all those other places have to have some sort of condition for including the package under a new name?

@tmds How do you envision regular package references working in source-buildable apps? What is going to guarantee that the source-building of the apps uses the copy of the package built from source and not the copy of the package on nuget.org?

The names of these packages are hardcoded in number of other places. Would this mean that all those other places have to have some sort of condition for including the package under a new name?

More or less.  It could be a condition evaluated when the SDK is being built, or a condition in the SDK evaluated when building an app, or it could be a source-build patch that changes the name.

What is going to guarantee that the source-building of the apps uses the copy of the package built from source and not the copy of the package on nuget.org?

I think the package identity should be different, for example Redhat.DotNet.ILCompiler, and the only place that package would be built would be via source-build.

I think the package identity should be different, for example Redhat.DotNet.ILCompiler

I meant how does the source-build work for random 3rd party packages. Let's say I want to source-build an app that references https://www.nuget.org/packages/Serilog package. Where is the source-built version of Serilog package going to show up in the system and how is it going to be referenced by the app?
I would like to see whether it would make sense to naturally extend the solution for 3rd party packages to packages like ILCompiler.

I meant how does the source-build work for random 3rd party packages. Let's say I want to source-build an app that references https://www.nuget.org/packages/Serilog package. Where is the source-built version of Serilog package going to show up in the system and how is it going to be referenced by the app?

There is an open issue for that: https://github.com/dotnet/core/issues/7443. It might involve using nuget packages from a local folder that are all source-built and having no other feeds available, like when source-building .NET.

I would like to see whether it would make sense to naturally extend the solution for 3rd party packages to packages like ILCompiler.

I think ILCompiler is different because the integration/coupling with the SDK.

I think ILCompiler is different because the integration/coupling with the SDK

I would not call it coupling. We have been intentionally keeping the AOT compiler package very loosely coupled with the SDK. The integration is basically just about automatically auto-referencing the ILCompiler package when the PublishAot property is set to provide better UX. We lived without this integration in the early days of the AOT project and it worked just fine for years. (The projects were required to reference the ILCompiler package manually.)
I think it would be best to treat the ILCompiler nuget package just like any other nuget package that the app source building may need.

It might involve using nuget packages from a local folder that are all source-built and having no other feeds available, like when source-building .NET.

This makes sense to me.

Package source mapping https://learn.microsoft.com/en-us/nuget/consume-packages/package-source-mapping#enable-by-manually-editing-nugetconfig can help with mixing and matching local packages and nuget.org packages.

Where is the library-packs NuGet directory added as a NuGet source? Is it something the SDK does unconditionally?

NuGet doesn't support setting a PackageDirectory on a PackageReference where the package is already extracted.

Why aren't we considering implementing this option?
My understanding is that it would allow to make ProcessFrameworkReferences to pick the source-built ILCompiler (and friends) from packs when needed.

Renaming packages in order to use them in source build feels wrong to me.

My opinion is that this is OK.  These packs or packages are implicitly referenced by the SDK.  If they were package references in a project file and you had to update that identity in order to use source-built packages, that would not be so nice.  It sounds like maybe they can be referenced separately, but the main scenario today is when they're automatically referenced by the SDK.

I think it would be best to treat the ILCompiler nuget package just like any other nuget package that the app source building may need.

I don't think we know how we would handle source-built NuGet packages, and it could well be a large feature.  If we want to support source-built NativeAOT soon, then it's probably better not to wait for general source-built NuGet package support.

Where is the library-packs NuGet directory added as a NuGet source? Is it something the SDK does unconditionally?

Yes, it does it unconditionally in Microsoft.NET.NuGetOfflineCache.targets.  The library-packs folder was added to support workloads, but it is included as a source unconditionally and would work here.

NuGet doesn't support setting a PackageDirectory on a PackageReference where the package is already extracted.

Why aren't we considering implementing this option?

It seems to me that this would conflict with some design aspects of NuGet.  When NuGet extracts packages, it computes hashes of the packages and may verify that the package was correctly signed.  This information is saved to the extracted package folder and helps verify and maintain the integrity of the package.  If you try to point it to a folder you've created where the package is already extracted, then it won't have that extra information and I think NuGet won't be able to maintain the guarantees it tries to make.
I'm not an expert on a lot of the details of NuGet, so I could be wrong.  But even then, it would take time to design and implement a new NuGet feature for this.  If possible, I would recommend trying to go with a design that can be implemented mostly with existing functionality of the SDK and its components.

If we want to support source-built NativeAOT soon, then it's probably better not to wait for general source-built NuGet package support.

Creating steps for how one can use source-built Native AOT makes sense to me (e.g. using a custom Package Source Mapping rules that I have linked above). However, I do not think it make sense to spend a lot of one-off effort on polishing the experience for source-built NativeAOT without solving the larger problem of source-built nuget packages.

I do not think it make sense to spend a lot of one-off effort on polishing the experience for source-built NativeAOT without solving the larger problem of source-built nuget packages.

They are related, but I don't expect https://github.com/dotnet/core/issues/7443 to become a high priority for .NET upstream in the near future.
For this issue we focus on a .NET SDK distro maintainer providing an SDK that includes NativeAOT support as a built from source feature.
For https://github.com/dotnet/core/issues/7443 we focus on a .NET app distro maintainer building an app from source.
These are different people.
For the second, some of the app maintainers already go through the effort of dealing with building their dependencies from source, and https://github.com/dotnet/core/issues/7443 is about providing tooling to assist them.
The second issue is mostly about dealing with 3rd party NuGet packages. I don't consider the ILCompiler a 3rd party NuGet package. I think we should try to deliver the source-built NativeAOT (as .NET SDK distro maintainers) so it just works for the SDK user (.NET app distro maintainer or regular distro user). They shouldn't be aware/care how the ILCompiler gets packaged with the SDK.

I'm not an expert on a lot of the details of NuGet, so I could be wrong. But even then, it would take time to design and implement a new NuGet feature for this. If possible, I would recommend trying to go with a design that can be implemented mostly with existing functionality of the SDK and its components.

I'm not an expert either but it seems like an option that would give us much flexibility.

The packages that are produced by source-build don't have stripped symbols which bloats them significantly. This is due to the stripping being performed at the end of the distro packaging process to produce separate distro debug packages.

Some numbers: the Microsoft ILCompiler nupkg is 48MB and the source-built is 126MB. Uncompressed they are 142MB and 326MB. I think we should try to strip those symbols to reduce the package size. I'm not sure where to strip them as we'd still want the non-stripped files to flow through for distro packaging. If the files are stripped inside a nupkg, the distro won't know about them, and it can't provide debug symbols through debug packages either.

For this issue we focus on a .NET SDK distro maintainer providing an SDK that includes NativeAOT support as a built from source feature.

It would help to describe the user scenario that this about. Why do the users want to use this and what do they expect to get out of it? My assumption has been that the motivating scenario for this work is ".NET app distro maintainer building an app from source", but it sounds like that it is not the case.

The second issue is mostly about dealing with 3rd party NuGet packages. I don't consider the ILCompiler a 3rd party NuGet package.

Yes, native AOT toolchain is a .NET feature delivered via NuGet packages. We have many other .NET Runtime and SDK features that are delivered via NuGet packages. My point is that we should keep that in mind and design something that can work for any .NET feature delivered via NuGet package. The library-packs idea is going in this direction that's good.
(I would expect the solution to be naturally extendable to 3rd party packages as well, but I agree that building 3rd packages from source is a separate problem.)

It would help to describe the user scenario that this about. Why do the users want to use this and what do they expect to get out of it? My assumption has been that the motivating scenario for this work is ".NET app distro maintainer building an app from source", but it sounds like that it is not the case.

The reason is the same as why .NET is buildable by others: it enables 3rd parties to build/deliver/support .NET independent of Microsoft.
The app maintainer is an important use-case.
The reason the app maintainer isn't allowed to use the NativeAOT because the distro want to be capable of changing/fixing the feature.
This applies also to RHEL: Red Hat won't claim we support NativeAOT on RHEL if we can't build it from source. The best we can do (and say) is: we work with Microsoft to ensure NativeAOT works well on RHEL.
Another use-case is to be able to build the feature on a distro/architecture that is not supported by Microsoft.

Yes, native AOT toolchain is a .NET feature delivered via NuGet packages. We have many other .NET Runtime and SDK features that are delivered via NuGet packages. My point is that we should keep that in mind and design something that can work for any .NET feature delivered via NuGet package. The library-packs idea is going in this direction that's good.

I listed what I think are the main challenges with the nuget packages in source-build in an earlier comment: https://github.com/dotnet/source-build/issues/1215#issuecomment-2090941658.

I listed what I think are the main challenges with the nuget packages in source-build in an earlier comment: #1215 (comment).

Well, the alternative is to create a parallel path through the whole system where the native AOT toolchain does not look like a package. It sounds very complicated to me. I think addressing the issues with the package-centric approach is easier.

Hi folks,
Some updates, and (I think) some good news.
While thinking this over I eventually came up with an idea that I think would let .props and .targets files be imported from packs in the packs folder.  The idea is that ProcessFrameworkReferences would output the list of files to be imported, and the SDK would have a target that ran during restore that would write out generated .props and .targets files that would import the specified files and would be automatically imported during the build.  This is how NuGet supports importing .props and .targets files from packages, so the SDK would do something similar also during restore.
However, while I was writing up a design for this idea, I looked more at how the three packages (Microsoft.DotNet.ILCompiler, Microsoft.NET.ILLink.Tasks, and runtime.<rid>.Microsoft.DotNet.ILCompiler) needed for NativeAOT are used.  It turns out that the runtime one, which is the large one, doesn't have any .props and .targets in it and is already downloaded as a PackageDownload.  So I think that package could be placed in the packs folder, and slight modifications to ProcessFrameworkReferences (probably mostly in the AddToolPack method) would allow it to be resolved from there.
The other two packages are way smaller, so I think including them as .nupkg files in the library-packs folder should work.  You still might want to change the package ID to something source-build specific if there are concerns that the Microsoft-built ones from NuGet.org could be used instead of the ones that ship with the source-built SDK in the library-packs folder.  We could also use the idea I had to allow these two packages to also be shipped in the packs folder if that was necessary.
As for fallback folders, I had forgotten about them, but was remembering the lessons we learned from them.  I believe fallback folders aren't supported (at least not well) anymore.  I believe we stopped using fallback folders around .NET Core 3, partly because we switched to framework references and package downloads, but also because fallback folders didn't work with features NuGet was adding such as package signing and hash validation.

I've started to implement the design proposed by @dsplaisted.
After patching the ProcessFrameworkReferences.cs so it looks for the source-built runtime.<rid>.Microsoft.DotNet.ILCompiler in packs, there is still another reference which triggers a nuget.org download for this package.
That reference comes from the runtime.json file that is in the source-built Microsoft.DotNet.ILCompiler package. It looks like:
{
  "runtimes": {
    "fedora.39-x64": {
      "#import": [],
      "Microsoft.DotNet.ILCompiler": {
        "runtime.fedora.39-x64.Microsoft.DotNet.ILCompiler": "[9.0.0-preview.5.24272.2, )"
      }
    },

We need the Microsoft.DotNet.ILCompiler property to be empty, like:
{
  "runtimes": {
    "fedora.39-x64": {
      "#import": [],
      "Microsoft.DotNet.ILCompiler": { }
    },

@dsplaisted @baronfel @ViktorHofer do you have a suggestion how/where to do this?

Interesting.  I imagine that reference in the runtime.json file is there to support explicitly referencing the Microsoft.DotNet.ILCompiler package.  However, it seems that the PackageDownload added by the ProcessFrameworkReference task would be redundant.  @jkotas, do you know how this works?
I don't know where the Microsoft.DotNet.ILCompiler package is created, but for source build I think you could omit the runtime.json file entirely.

Yes, explicit reference of Microsoft.DotNet.ILCompiler package is expected to add the RID-specific Microsoft.DotNet.ILCompiler package. runtime.json in Microsoft.DotNet.ILCompiler package on nuget.org has entries for all portable RIDs supported by native AOT to make that work.

The runtime.json feature is undocumented and got added in the .NET Core 1.x days to support including RID specific packages. The ILCompiler package is one of the remaining ones that use runtime.json and our plan is to eventually get rid off them all and replace them with alternative solutions (i.e. the Known* items in the SDK for anything in-built). https://github.com/dotnet/runtime/issues/49137 is also related.
Mentioning that as I think now would be a good time to change the ILCompiler package and make it use the common SDK infrastructure (framework references / known* items).

explicit reference of Microsoft.DotNet.ILCompiler package is expected to add the RID-specific Microsoft.DotNet.ILCompiler package

I assume this means we can not just remove the runtime.json file.

I think the effect of removing the file entirely would be the same as removing the element you are suggesting.  The only thing that runtime.json file does is add the RID-specific package reference.  If you remove that element, then that should be the same as removing the entire file (except possibly for projects that are targeting other RuntimeIdentifiers).
I don't know if things will work if you remove the reference.  I'm not sure how the ILCompiler tasks locate the assets in the RID-specific ILCompiler package.  As a first step though I'd suggest trying out removing the runtime.json entirely and seeing what happens.

@jkotas are there cases where that won't work?

Is explicit reference of Microsoft.DotNet.ILCompiler package going to pick RID-specific Microsoft.DotNet.ILCompiler of the same version?
The explicit references of Microsoft.DotNet.ILCompiler are used e.g. here: https://github.com/dotnet/runtime/blob/main/src/coreclr/nativeaot/docs/compiling.md#using-daily-builds

Is explicit reference of Microsoft.DotNet.ILCompiler package going to pick RID-specific Microsoft.DotNet.ILCompiler of the same version?

I don't think so. We'd break this scenario by removing runtime.json.

How can we update/patch the runtime.json file so we eliminate the package reference only for the source-built rid?

@jkotas would it be ok if we'd remove runtime.json from Microsoft.DotNet.ILCompiler only when it gets built with source-build configuration?

Is my understanding correct?

This is a good overview, and it matches with how I expect it to behave.

Also, have you considered installing the native AOT compilers as separate distro package (e.g. dnf install dotnet-sdk-aot-8.0) to reduce the baseline size of .NET SDK? This package can transitively install the native toolchain and other dependencies so that users do not have to install them manually.

Yes, we'll put this in a separate package to reduce the default SDK size.
I like the idea of making this package install the native toolchain through dependencies. It will be up to the package maintainer to actually implement this.
We'll extend https://learn.microsoft.com/en-us/dotnet/core/distribution-packaging with these guidelines.

This all sounds reasonable to me, but I want to make sure we have a long-term design in place that's specified and maintained. Do we have a doc that explains exactly what library-packs and packs directories are, how they interact with PackageDownload, how they are used as hooks for source build, and what semantics we have to maintain in the future?
This area has been difficult to understand in the past.

These are the PRs that implement https://github.com/dotnet/source-build/issues/1215#issuecomment-2116395577:

https://github.com/dotnet/sdk/pull/41198
https://github.com/dotnet/arcade/pull/14805
https://github.com/dotnet/runtime/pull/102908

I verified that with these changes the resulting source-built SDK can be used to publish native-aot apps, and that it works as an SDK to build the vmr.
There is still some work to be done, esp on the sdk PR. I hope we can get these into the preview that the main branches are currently targeting.
Once these changes are in the vmr, we may look to do some optimizations that reduce the size of the ILCompiler pack.

@baronfel @dsplaisted @jkotas @MichaelSimons @ViktorHofer friendly reminder these PRs are still open, and this is something we'd really like to be in preview6.
cc @omajid

@tmds, @omajid, @ashnaga - Can we close this issue now?  I am thinking any remaining bugs found can be tracked with instance specific issues.

We still need to add tests, and update https://learn.microsoft.com/en-us/dotnet/core/distribution-packaging.

@tmds - Is testing the last piece here?  Is that still something you are planning to contribute?

Yes. I forgot about this. I'll look into it.


Add ESLint rules to mandate braces around if-else and switch-case

Description
This PR incorporates changes to the ESLint configuration to enforce the use of braces in one-line if-else statements and case blocks inside switch statements. This update aims to enhance code readability and prevent potential issues in the future. The specific ESLint rules added are:

curly: Enforce braces for all control statements.
default-case: Enforce a default case in switch statements.
eqeqeq: Enforce the use of strict equality (===) and inequality (!==).

For more details on ESLint rules, please refer to the official ESLint documentation.
Testing Instructions

Ensure ESLint is installed by running npm install eslint --save-dev.
Run ESLint on the project to check for adherence to the new rules: npx eslint .
If necessary, automatically fix fixable issues using: npx eslint --fix .

Checklist

[x] My pull request has a descriptive title.
[x] My pull request targets the default branch of the repository (main) or a parent feature branch.
[x] My commit messages follow best practices.
[x] My code follows the established code style of the repository.
[x] I added or updated documentation regarding the ESLint rule changes.
[x] I ran the project locally and verified that there are no visible errors.

Hey @dhruvkb
I don't know what happen with the repository I have forked to my github. I am not able to find it. So, I am forking it again and creating new pull request.

@bhavik001 it seems that instead of closing the PR and deleting your branch, you have accidentally deleted the entire repo.

I don't know how it get deleted because I am still working on it. At the time of pushing the code I was getting the error and I saw the repository is not there.


maintenance status == discountinued?

The kernel lives on as JJava:

https://github.com/SpencerPark/IJava/issues/161
https://github.com/dflib/jjava


Compute f1_score per MeSH tree level

Btw you can add this to the dvc as part of the evaluation or creating graphs. You can also use dvc metrics or dvc plots


docs: Add && & || to syntax table

The diff is bigger than it looks, rendered comparison forthcoming

Existing:

Proposed:

It's not great that it spans two lines. But the existing approach already spanned two lines in other columns, and I'm not sure of a better way — we def don't want to have a line per operator where we announce the meaning of +...

@richb-hanover for your review...

Good idea to add && and || Merge it.
I'm still thinking about how to handle parentheses and will build off what's committed at the time.


Compile error

Program type already present: com.github.florent37.expansionpanel.R$dimen
Message{kind=ERROR, text=Program type already present: com.github.florent37.expansionpanel.R$dimen, sources=[Unknown source file], tool name=Optional.of(D8)}
Caused by: com.android.builder.dexing.DexArchiveBuilderException: Failed to process C:...\app\build\intermediates\transforms\instantRunSlicer\debug\1
Caused by: com.android.builder.dexing.DexArchiveBuilderException: Error while dexing.
Caused by: com.android.tools.r8.CompilationFailedException: Compilation failed to complete
Caused by: com.android.tools.r8.utils.AbortException

Fixed by running a clean build, sorry.


sql binlog writes to orders_extract and then transfers to s3 bucket

created a dictionary to write to orders_extract using keys (update, delete, insert) which gets stored to local machine. It then gets tranferred to s3

gucci