Entry Points for RO-Crate Profiles

In the current specification, profiles are always defined on a complete RO-Crate. In case that such profile specifies requirements on the root data entity, they cannot be used in a modular or composable way, since subdirectories in RO-Crates don't specify a root data entity. An example for such a profile is the Workflow-Run-RO-Crate.
This issue came up at the BioHackathon Europe 2024, project 19 (@elichad @dnlbauer @floWetzels). Since composed research objects seem to be common and it should be possible to model them in RO-Crates without redundance in profiles, we propose the introduction of a Entrypoint mechanic into the RO-Crate specification, see https://github.com/dnlbauer/bh24-ro-crate-extension for details (will be fleshed out in the future).
Subscribe to this issue to stay updated on the development.

To consider:

can/should this work recursively? Having an entry point inside another entry point?
what should happen if multiple entry points conform to the same profile? e.g. in the case of uploading a crate with multiple workflow entry points to WorkflowHub

what should happen if multiple entry points conform to the same profile? e.g. in the case of uploading a crate with multiple workflow entry points to WorkflowHub

In this case, the service could decide to decompose the RO-Crate into "atomic" (for the lack of a better term) RO-Crates which can be handled like normal RO-Crates. I.e. WorkflowHub could decide to create multiple entries - one for each entry point in the crate; a workflow execution engine executing a workflow from RO-Crate could present a drop down selection to the user or require to specify an entrypoint during workflow submisson.

Intending to discuss this issue at the RO-Crate community call at 8:00 UTC tomorrow

Missed the meeting and thus the discussion in the call.
Still sharing my two cents.
From a semantic point of view there is nothing preventing you to declare additional triples with the dcterms:conformsTo-predicate attached to any available part (subject) in the graph.  And I don't think the ro-crate spec is formulating any restriction on that either.  If anything, the jsonld-context just makes it handy to use conformsTo: keys in the json-ld to add these.
Its value is expected to contain an identifier (URI) for a standard, that

just conceptually represents a number of assumptions clients can make about the subject
allows those clients to verify if they have the knowledge on board to deal with that

As such one could be using the conformsTo in ro-crates in combination with

data entities of type File to express e.g. the file is not just a netcdf file but conforming to th cf-conventions, or even a CSV file that sticks to some layout or schema, ...
conceptual entities describing dataservices that e.g. conform to some webserrvice api standard (like ogc-wms, erddap, ...)

This way of applying dcterms:conformsTo exists outside the RO-Crate concept and can be applied to any part of it as far as I see.  The fact that the RO-Crate specification additionally introduced some specific suggestions to express conformity of ro-crates was considered as a useful and clear mechanism to guide people into some kind of "duck-type" declaring of valid assumptions on the crate contents.  The fact RO-Crate 1.2 introduces some guidance on this level

does not in any way limit other usage of this mechanism (including the suggested nesting)
nor should it raise the expectation that because of that the RO-Crate specification suddenly needs to control, document, or worse: forbid any more nested/detailed application of that same mechanism.

If anything, IMHO the RO-Crate spec should state it does deliberately not want to interfere with that detail level. And

Summarising discussion from community call 2024-11-14:

Using the EntryPoint type is not strictly required since it could be inferred from checking if any conformsTo on an entity is an RO-Crate profile. However it is convenient (for tooling) to make entry points explicit within the crate metadata.
Using @type to indicate entry points may not be the best choice, as @type usually describes what the actual thing represented by the entity is (e.g. a File, a Person, a Place), and an entry point is just a construct in the metadata

the existing EntryPoint type in schema.org is intended for describing API endpoints and such, this isn't quite the same as our idea, we wouldn't want someone to describe an API using RO-Crate and end up with weird conflicts because of the overloaded type
we could find a different property to use (but we haven't found a good one so far)
in our example we also included the entry points under "about" in the metadata entity, which is another way to make them easily discoverable by tools
ISA profile uses additionalType to indicate Investigation, Study, Assay https://github.com/nfdi4plants/isa-ro-crate-profile/blob/release/profile/isa_ro_crate.md - potentially do something similar to indicate entry points?
Do we need a “Crate” type?

Highlighting of entrypoints GUI wise - they are possible views of the crate
Profiles often talk about the root crate - the entrypoint would be a mechanic for profiles to talk about their root without necessarily that being the RO-Crate Root

Want to make RO-Crate profiles  more like mix-ins that don’t require being in an RO-Crate
https://www.researchobject.org/ro-crate/specification/1.2-DRAFT/data-entities.html#referencing-other-ro-crates uses conformsto and type Dataset to indicate an external crate.  But what if it only conforms to a profile without being a RO-Crate?
RO-Crate V2 is moving towards "fragments". Also does not have to be a Dataset that is the “root”. Increasingly profiles of fragments can then be used. This idea fits well with that vision

If we add EntryPoint there may be only be certain properties that we should follow recursively to scope the “sub-crate” e.g. hasPart, mentions, mainEntity. But what links NOT to follow?
Alternative of using named @graph { fragments } to isolate the scope? Can get quite complicated.


LLAPI modularisation.

AB#61
This PR is the result of a discussion on discord surrounding the current LLAPI structure and how we want to handle object deletion. As a result this PR resolves #51 .

Overall LGTM.
I called out a few nits/nice to haves, but didn't repeat them if they occured more than once.


Glock Ammo Type

Glock currently uses 10mm ammo, which seems to have been replaced everywhere outside the vault, could be good to make it use 9x19 parabellum instead.

Will be added in with upcoming update


ModuleNotFoundError: No module named 'openTSNE'

Dear Michael,
i'm struggling with the use of the cellplato software.
Running the notebook "cellPLATO_StepByStep_main" notebook, and executing the first cell i run into the mentionned issue.
Do you have any clue about this,
thank you in advance for your support,
Fabrice

Hello,
i made a fresh install and now I run into a new problem:
INTEL oneMKL ERROR: The specified module could not be found. mkl_intel_thread.2.dll.
Intel oneMKL FATAL ERROR: Cannot load mkl_intel_thread.2.dll.
Do you have any clue about such an issue ?
Thank you,
Fabrice

hello again,
i pip installed MKL and made some progress ;)

Hi Fabrice. I know exactly what is going on with the tSNE part, I will give you a fix in the next few days. Sorry for the delay, I moved to a new institute and was setting up here.

Dear Michael,
I fully understand. In the meantime I made lots of progress.
By know I'm stucked into another problem related to BTrack and Cellpose. It seems that the file generated via napari-Btrack doesn't retain the label layer, which is obviously needed.
Don't worry, you're kind of belated answer made me make huge progress in some rather basic aspects.
Thank you and enjoy your new position.
Let's keep in touch,
Fabrice
-----Message original-----
De: Michael @.>
à: Michael-shannon @.>
Cc: fabricesenger @.>; Author @.>
Envoyé: jeudi 11 juillet 2024 17:40 CEST
Sujet : Re: [Michael-shannon/cellPLATO] ModuleNotFoundError: No module named 'openTSNE' (Issue #2)
Hi Fabrice. I know exactly what is going on with the tSNE part, I will give you a fix in the next few days. Sorry for the delay, I moved to a new institute and was setting up here. —
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

Hello Michael,
here the latest problem I encounter (let me know if you prefer me opening a new issue
AssertionError                            Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_4652\4009738624.py in
1 # Load, process and combine the dataframes (including segmentation and migration calculations)
----> 2 comb_df = cp.combine_dataframes(exp_list)
3
4 comb_df, new_factors = cp.measurement_pipeline(comb_df, mixed=cp.MIXED_SCALING, factors_to_timeaverage = cp.ALL_FACTORS)
5 display(new_factors)
c:\Users\fabrice_senger\cellplato\cellPLATO\cellPLATO\cellPLATO\data_processing\data_io.py in combine_dataframes(exp_list_df, fmt, dedup_columns)
324                     this_file = os.path.join(DATA_PATH,cond,rep) + TRACK_FILENAME
325                     f = h5py.File(this_file)
--> 326                     assert 'segmentation' in f.keys(), 'segmentation not found'
327                     if (DEBUG):
328                         print('h5 file contents: ',f.keys())
AssertionError: segmentation not found
the h5 file has been generated using a cellpose segmentation, this has been converted to a label layer and the labels have been tracked with BTrack in napari. The BTrack output has been saved as h5 file. Until this part of the code all the cell outputs from the notebook are now ok (at least i didn't get any further error message.
Thank you for your help and support,
Fabrice

Hello again,
from what i understood my data structure is not ok.
From your publication i found "btrack was run using Jupyter notebook (included with the cellPLATO package) in batch on the segmented masks in the master folder. Segmentations, xy coordinates and track IDs for each cell over time are saved into a separate h5 file per replicate." If i understand right, this notebook -not included in the package to my knowledge- generates the files used for further processing ?
I would happily give this a try
Fabrice

Hello,
to generate an h5 file with the segmentation and the tracking data i got this code snippet from Alan Lowe
with btrack.io.HDF5FileHandler("/path/to/file.h5", "w") as hdf:
hdf.write_segmentation(cellpose_segmentation_as_numpy_arr)
hdf.write_tracks(btrack_tracks)
While I have all the needed envs (Cellpose3 ; Btrack ) and can run them via notebooks , I'm still wondering about the data structure, i.e. , the names to give to my files
Could you give me some feedback about that ?
Hope you're doing fine in your new position,
All the best,
Fabrice.

Hi Fabrice
Thanks for raising these issues.

I have added the notebook used for btrack , and accompanying config file as a json. The notebook already points to the json, so you should just need to change the master folder name to whatever yours is. I am sure you have your own settings for btrack based on using it in napari, so please enter them in the required places in the jupyter notebook to run it. And yes, cellplato generally expects the segmentation to be written into the h5 file too.

Do you mean folder structure or data structure? I would be happy to set up a zoom with you next week if you need to discuss this and anything else.

I see you solved the dependency issue (MKL). Did you still have a problem with the tSNE issue?
Best,
Michael

Dear Michael,
To me it's a blast to interact with you.
I will be of for a BIG summer break. So for the zoom, given I'll be in Nature... It's going to be complicated. I will look into the files and let you know how it goes. To make a long story short, I look into cellular monolayers, not sure at all, what will be the outcome. For the software you developed it might be an interesting generalization...
Thank you once more for your time.
Have a nice and peaceful summer,
Fabrice
-----Message original-----
De: Michael @.>
à: Michael-shannon @.>
Cc: fabricesenger @.>; Author @.>
Envoyé: vendredi 19 juillet 2024 18:45 CEST
Sujet : Re: [Michael-shannon/cellPLATO] ModuleNotFoundError: No module named 'openTSNE' (Issue #2)
Hi Fabrice Thanks for raising these issues.

I have added the notebook used for btrack , and accompanying config file as a json. The notebook already points to the json, so you should just need to change the master folder name to whatever yours is. I am sure you have your own settings for btrack based on using it in napari, so please enter them in the required places in the jupyter notebook to run it. And yes, cellplato generally expects the segmentation to be written into the h5 file too.
Do you mean folder structure or data structure? I would be happy to set up a zoom with you next week if you need to discuss this and anything else.   I see you solved the dependency issue (MKL). Did you still have a problem with the tSNE issue? Best,
Michael —
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

Sure, enjoy your summer!
Also for the tSNE issue, you don't need that module really, unless you want to do tSNE specifically. So you can just comment it out of the first notebook cell.
Michael

Dear Michael,
Once I'm back we can organize a zoom. It would be cool, as I could show you what I'm working on. Besides that, I resolved the tsne problem, and the mkl problem, and the hdbscan/joblib problem... I think, I'm the basic user you're thinking of in your publication, thus, the problems I encounter with the solution will contribute to a wider adoption of the software you developed... I'm back around mid August.
Enjoy your summer and your new position.
Best,
Fabrice
-----Message original-----
De: Michael @.>
à: Michael-shannon @.>
Cc: fabricesenger @.>; Author @.>
Envoyé: samedi 20 juillet 2024 00:03 CEST
Sujet : Re: [Michael-shannon/cellPLATO] ModuleNotFoundError: No module named 'openTSNE' (Issue #2)
Sure, enjoy your summer!
Also for the tSNE issue, you don't need that module really, unless you want to do tSNE specifically. So you can just comment it out of the first notebook cell. Michael —
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>


Laravel 11.x Compatibility

This is an automated pull request from Shift to update your package code and dependencies to be compatible with Laravel 11.x.
Before merging, you need to:

Checkout the l11-compatibility branch
Review all comments for additional changes
Thoroughly test your package

If you do find an issue, please report it by commenting on this PR to help improve future automation.

:warning: The latest version of Laravel has a dependency for phpunit/phpunit of ^10.5. If you would like to also upgrade your tests, you may run the PHPUnit 10 Shift for free.

:warning: Shift detected GitHub Actions which run jobs using a version matrix. Shift attempted to update your configuration for Laravel 11. However, you should review these changes to ensure the desired combination of versions are built for your package.

:alembic: Using this package? If you would like to help test these changes or believe them to be compatible, you may update your project to reference this branch.
To do so, temporarily add Shift's fork to the repositories property of your composer.json:
{
    "repositories": [
        {
            "type": "vcs",
            "url": "https://github.com/laravel-shift/laravel-ciphersweet.git"
        }
    ]
}

Then update your dependency constraint to reference this branch:
{
    "require": {
        "spatie/laravel-ciphersweet": "dev-l11-compatibility",
    }
}

Finally, run: composer update


Add retry mechanism for DefaultCredentialsProvider

Development Task
we had the downgrade order to get the access token. for example when web_identity_provider didn't work(maybe network problem). we'll use default_provider.
 async fn credentials(&self) -> Result<AwsCredentials, CredentialsError> {
        // Prefer the web identity provider first for the kubernetes environment.
        // Search for both in parallel.
        let web_creds = self.web_identity_provider.credentials();
        let def_creds = self.default_provider.credentials();
        let k8s_error = match web_creds.await {
            res @ Ok(_) => return res,
            Err(e) => e,
        };
        let def_error = match def_creds.await {
            res @ Ok(_) => return res,
            Err(e) => e,
        };
        Err(CredentialsError::new(format_args!(
            "Couldn't find AWS credentials in default sources ({}) or k8s environment ({}).",
            def_error.message, k8s_error.message,
        )))
    }

but when user only set k8s pod credential. we must ensure web_identity_provider works. we should retry until it succeed. or it may be still failed due to access denied in next request to s3.

I was thinking can we check the if role_arn not nil in web_identity_provider then we should retry until succeed?
But the question is how to distinguish retryable errors(e.g. network error) from k8s_error. WDYT @kennytm @YuJuncen

I guess a limited time (even maybe huge number of retires, say, 30 times) of retries would be fine (or we may get stuck if the service is inaccessible by definition)?

I guess maybe we also need some rule for fast fail, or we may be stuck for a long time if the user forgot to provide the access key.

@YuJuncen  Pls test this solution:
https://github.com/awslabs/aws-sdk-rust/issues/540
amazonec2-metadata-token=required
amazonec2-metadata-token-response-hop-limit=2

@cncoder Thanks for your hint! It seems like the problem in docker container consistently occurs as long as the workaround configuration hasn't be set. However we met this problem occasionally, hence I guess probably there are some other things went wrong?


occursed structure : how to click on the three left columns ?

Bonjour ,
after research , like innerHTML , no success
can help me
i want to click for exemple column 4 , 5 or 6 of line contains 'spufi'
(with attached ZIP (not clean , security reasons ))
thanks
ttt.zip

ok
the solution is in example file : the module usage_table


AWS::OpenSearchService::Domain

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-opensearchservice-domain.html

Thanks for making this change but I think I'm going to split this out into a separate file. I doubt AWS will make further improvements to Elasticsearch and OpenSearchService is their replacement. Might was well get this in its own namespace sooner than later.

Got it. Do you want me to separate it? Or do you already have the codes?


⚠ Secret Website 3 has degraded performance

In ae626ae, Secret Website 3 ($SECRET_WEBSITE_3) experienced degraded performance:

HTTP code: 200
Response time: 2506 ms

Resolved: Secret Website 3 performance has improved in c646a28.


Help wUST stuck

Hey,
I tried sending wUST through wormhole bridge v1. But I rejected the ETH gas fees and now my wUST is stuck. Could some please help? Thanks!
FZuUgE41NAFPAFCZXNaKCtFcVpsEDVeJPomPsVjLxNykYLdevKCBbQjaZXe2SzshiNdMLh1WnQctUAb1ShabLWk

The transaction was successfully processed by the Wormhole network, but the client-side transaction submission may have failed. Try clearing your cache and reloading https://v1.wormholebridge.com to see if the transaction can be completed there. Make sure both wallets are connected, and select both tokens from the picker.
If v1.wormholebridge.com doesn't work, you can manually submit the signed VAA bytes below to the submitVAA method on the Wormhole contract: https://etherscan.io/address/0xf92cd566ea4864356c5491c177a430c222d7e678#writeContract
Note that Wormhole v2 was recently launched on wormholebridge.com, which fixes the bug you encountered! You will still have to use v1 to complete your in-progress transfer.
You can also try using the third-party https://www.interstellar.tk site to retry your VAA submission.

Digest (for reference only): 51a7a465a6b0de94d855ea01d6660573fcf16e9440eaa3d88326170d4cfac402
Signed VAA bytes to submit to the contract:

0x01000000010d00ad1d039156086e0c628568aeb4fc1341a219c26a20b9155fcdca7b752641211c4a1c827318c603771cb3f30b3aeebbf4c0d0c4eddbe606b976262f9dbee5bb250001d56265b48e5cb1abec6e8c8cce07183f3c9212bf538976b3536f5837545354343de4d557b966624c2fff6eda63091fcc00b2310630c0d53d7eaaab7d277bece001030069311dc5182bca2ddb390d51a16c57a7f2d973aef79d83e928fb8c268ba3184db025b55cfba70459a89a8be6a3888fba2650e8885f4f1214ca8d39c5a0f26c01046c1d15aef24c7cfeff84a7d4e750228380d188b2a571cfeb2127bcf11cfe872c0d1df96c1f6f3d83c0927f025c7f0e72f033920f30d12bcdc8b4fd9b06a1a2670105c6d83f29d88dbc14f97eee27be6179f8fe8e4a365430631c9b6d73e3fdd233d367fef33284604d21e7394ef4e9a41e9cc5dd7db296ec84f1ec25393bda9203e7000620061c5de6a93c1288b4cf0f111c62af13a6d66ba98bc033aee7c82a17380c037101fd4ae7be45e03c6a4b94225541db8498a042bec19c404d8808d20d9ec83e00076355eb5bcdea866eaa461d81c03e84c5cb42fbd88b360fda69e9aa782c2c7bd7436fa38a0cdab6d68311f180da344e7ea983f2decfef53249658c96d2af9e8c400093004c90d44f575fef2d04d395ffe2cc48fc6c52dbeb461b8c25d02a9da4d73e21e4c74bdf2f80661b5f41b409ee77934ab82a57c1f8a67e0e220c302f68e5dfd010a9220a45dcc0a4cbb5905d766b7ea302abd60bd91b20406928811aea330872897416c5c6ad44566f1e953aafdb012160d3bb709b57b067fd8ef55bf0a22aad27c010c3ba77de5d84669eaa192d645780eafa6519bef5d048d494ce12ae70b187a7c302b931ddb47f99a01e673093ce736c870503aef8faccab92fc42177d4dbc489d1000d7ee31c3426bdb41d4fb202445c257b981eea924ff46063638131bd898ea229b45f160b2e2cfecf8b83f246f7c42ba76d3dfd65266d9d3410e4e7604c6933cdec0110ae04bcb40fa9f3108169978091ba6dadb77ed08c9f5d58c7e3d43359f6d1856a35a0f0c94c9e0e0394a15c51a4e315d7b5c10986240637e56c9fa775d890684b011130ab8d0550b11d408de7c9cd63404f2924321edee582fd73ee5fcb6faad03bf618f2dfeed0bb8120d228c9e0bc06ed4441820b49ff8a1723e11f6f842204294c006162f4771000015ce30102b7cb89d0a68640364fc3b381b5038e43befade15a9a0373a01bd5c46db6a9266000000000000000000000000dcb0dad8ee75ba75cb8d4641bd357dd1f441a48f02000000000000000000000000a47c8bf37f92abed4a126bda807a7b7498661acd090000000000000000000000000000000000000000000000000000005d5d766a00


Game Keeps Crashing with Dynamic Surroundings Installed

I recently installed dynamic surroundings into my mods folder and it crashes afew seconds into the world. Here is the log file.
latest.log

What is your particle render level set to in the game options?

Minimal

It's what I suspected.  Two things you can do right now: increase from minimal to decreased, or disable the waterfall effect. I know what the issue is. I am in the midst of making other changes and will be uploading another build toward the weekend.

Pushed update to CurseForge and Modrinth.


incorrect shortTextAlgorithm documentation

The documentation about shortTextAlgorithm(int) says: Defaults to 0, but it actually seems to default to 50.

That is indeed correct as you can see here: https://github.com/optimaize/language-detector/blob/master/src/main/java/com/optimaize/langdetect/LanguageDetectorBuilder.java#L47


README: Remove ember-cli-eslint requirement

Let's remove ember-cli-eslint requirement mention from README - it's not necessary to use the plugin by any means, and we certainly can assume some people only linting code within text editor (for example).

Or we can leave this information but slightly changed, that you need either pure eslint or ember-cli-eslint if you wan't to check your code during development and testing by ember.

I thought about this, but do we really need to include information that eslint is required by ESLint plugin? And eslint command is what ember-cli-eslint uses underneath, as well. WDYT?

It is not a standalone package and has only one requirement, so in my opinions it seems quite natural to put this information in README and point that you need either just eslint or ember-cli-eslint that uses eslint under the hood.

Let's agree to disagree here, but I included short note about ESLint dependency in latest commit - however obvious, it's still an actual requirement, so it's understandable to include it. 👍


WEBAPP-325: Fix url encoding

This pull request belongs to an issue on our bugtracker.
You can find it there by looking for an issue with the key which is mentioned in the title of this pull request.
It starts with the keyword WEBAPP.

This wants to be merged still :D


Add checkstyle with the default rules

Add checkstyle, file issues for every set of bugs.

done pending PR


How can i remove subclass_prefix?

Hello, i want to remove subclass_prefix thing in my codeigniter project. How can i do that?

I tried this. This doesnt work. I dont want to use subclass prefix.

Questions should be asked on our forum, thanks.
We use guthub for bug tracking.


Update  Tracker.py

Is your feature request related to a problem? Please describe.
Tracker.py isn't working due to import  module error i.e.,
Traceback (most recent call last):
File "Tracker.py", line 1, in
from phonenumbers import geocoder
ImportError: No module named phonenumbers
Describe the solution you'd like
I would like to suggest an update to this python program and want to replace it with the script which I had recently created.
Do I want to work on this:

[x]  Yes
[ ] No

lemme check after that I will assign. thanks

@kuzuri-03 for me there was no import error, but later it shown some code which maybe bcoz of code.

Make ir working with proper requirement.

@kuzuri-03 for me there was no import error, but later it shown some code which maybe bcoz of code.

Make ir working with proper requirement.

I made some changes and it work pretty well
But I'm not satisfied with the output it gave ...
It gave only carrier name and country name nothing else ....
I stilll wanna work on it for  GSSoC..
Can I ??

I made some changes and it work pretty well
But I'm not satisfied with the output it gave ...
It gave only carrier name and country name nothing else ....
I stilll wanna work on it for GSSoC..
Can I ??

I already assigned you

can you assign this issue to me?

Sure Go Ahead @enrich4real

Hi Im gssoc 2023 contributor , I'm new to open source can you elaborate little more what is needed to be done and can you assign it to me !!

hi @avinashkranjan
I am a GSSOC'23 contributor and would really like to work on this issue. Please assign this to me. I have worked on projects in machine learning and have my expertise in python.

Go Ahead @NIveX2169


Release 0.3.1

@JuliaRegistrator register

@JuliaRegistrator register

Registration pull request created: JuliaRegistries/General/11251
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if Julia TagBot is installed, or can be done manually through the github interface, or via:
git tag -a v0.3.2 -m "<description of version>" cddf2ebe662e3613198b9686310cc6188570197e
git push origin v0.3.2


utils: added super init call for SmartDict init

I need this super call for proper inheritance in invenio-records.
Signed-off-by: Łukasz Chrząszcz<EMAIL_ADDRESS>

What about changing the commit message to utils: super call added to SmartDict
I guess we can make this PR for maint-2.0, can you rebase on top of it instead of master? (don't close this PR just rebase)
And you can add Reviewed-by: Esteban J. G. Gabancho<EMAIL_ADDRESS>too :wink:

s/added/addition/ - no verbs on first line - http://invenio.readthedocs.org/en/latest/developers/git-workflow.html?highlight=git#r2-remarks-on-commit-log-messages - we should mention this change with BETTER label

ping @jirikuncar

@Theer108 please use global component and rebase. Thanks

Done

I guess we can make this PR for maint-2.0, can you rebase on top of it instead of master? (don't close this PR just rebase)

@Theer108 see https://github.com/inveniosoftware/invenio/pull/3386#issuecomment-123626931 (cc @egabancho)

Done again


BEAN EMOJI

omg coronavirus will eat me if i dont get bean emoji pls help discord add it right now or i will eat bacon egg sandwich

here is crude drawing of bean hope u like it and see the use of it in discord

What is this? Bean emojis aren't part of the official unicode standard.

Can someone close this troll issue?

Bean has been proposed thrice and has finally been added to the list of shelved emojis that might be added solutions at the discretion of the Unicode Consortium, which might be never.
https://unicode.org/emoji/emoji-requests.html

As lovely a bean drawing as that is, there's no bean emoji in the Unicode standard, so closing this.


Add holding notes to bib items document

Closes #856

Coverage increased (+0.2%) to 82.676% when pulling d38814f539d8577b35a56bf1ce5a42413ae59fa4 on holdings_notes into 6111d339441b6b4d80e63502121aea085fabb416 on alma.

@hackartisan Update the PR with your suggestions and added a new spec for a record with only a permanent holding location note.


Story/complex status code

This branch is to fix the issues with the status codes always returning "OK" regardless of if the request is good or bad

@erchiggins
Ready for review


Update Sandbox sync to use DEA-notebooks "stable" branch (not "master")

We have just renamed the master branch of DEA notebooks to stable to better describe how we use it in contrast to develop:
https://github.com/GeoscienceAustralia/dea-notebooks/issues/589
This means the DEA Notebooks cloning to the Sandbox needs to be updated. Would it be possible to get this change made:

Make the DEA Sandbox git clone from the DEA Notebooks stable branch (not master)

Branch in question here: https://github.com/GeoscienceAustralia/dea-notebooks/tree/stable

pretty sure Tisham fixed that


Add some defensive checks and make code more uniform.

This fixes #448

Can Name() return a null pointer in any sane case?

I think, for these checks, the element->Name() should be an assert?

@leethomason This was already resolved with assertions inside StringEqual()

No longer relevant.


Can we create a cluster layer with a feature collection instead of service url?

This is not an issue but an enhancement suggestion.
Currently, a service url seems to be required to generate a cluster layer. Is there any way to create a cluster layer by passing a feature collection? Thanks,

Is there any way to create a cluster layer by passing a feature collection?

nope, but a pull request would be welcome.


feat: allow proxies to be injected using a native sidecar

Reason for Change:

Fixes Issue 773 by using native sidecars so that pods exit cleanly when running as cronjobs, currently azwi sidecar prevents the pod from terminating.
It also addresses an annoyance where the first container in the pod is the proxy, so any kubectl commands target that container, previously I was working around this by adding the default-container annotation.
I've kept the current behaviour as the default because this change requires at least k8s 1.28, adding the annotation azure.workload.identity/use-native-sidecar enables native sidecars. But it seems to work well so at some point in the future you could just make this the default behaviour.

No update to deployment.yaml

No change to Helm chart
Requirements

[X] squashed commits
[X] included documentation
[X] added unit tests and e2e tests (if applicable).

Issue Fixed:

Fixes #733
Please answer the following questions with yes/no:
Does this change contain code from or inspired by another project? If so, did you notify the maintainers and provide attribution?

[ ] yes
[X] no

Notes for Reviewers:
Tested on v1.30.0-eks-fff26e3

@microsoft-github-policy-service agree

@aramase @enj any update on this PR? I've just rebased it


Refactor/consensus related renaming

This PR implements the renaming as suggested in RFC 0018.

Perhaps next_staking_epoch_ledger? However, if we are going to have this discussion, we should move it back to the RFC. People who have stake in the names (including evan and izzy) approved those names, and I would want to make sure they are on board with whatever we choose since a lot of the confusion around curr vs next has been between evan and myself. I believe the term next maps more directly to how evan thinks about the consensus stuff.


Add hacking version constraint

To reduce time of dependency resolution.

Codecov Report

Merging #17 (d6ae6b8) into master (8213c29) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #17   +/-   ##
=======================================
  Coverage   98.63%   98.63%
=======================================
  Files           3        3
  Lines         220      220
  Branches       31       54   +23
=======================================
  Hits          217      217
  Misses          1        1
  Partials        2        2

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8213c29...d6ae6b8. Read the comment docs.


Store index of candidate in search result

Hey! Thanks for this crate!
I think it would be convenient to store the index along or instead of the string. That allows to identify results more easily, e.g. when there’s other data associated with the strings.

Thanks for your interest. I'm not actively working on this right now, but will bear this in mind if I come back to it.


rfc: make cargo install extensible

Rendered
The goal of this RFC is to introduce the ability for an end user to be able to extend the cargo install command arbitrarily to include instructions that should be executed occur after cargo install <project>.

This RFC was a collaborative effort between @joshtriplett @spacekookie and @ashleygwilliams.

A note about cargo as a distribution tool
We are in no way suggesting that cargo install should be the definitive or exclusive distribution mechanism, or that it should supplant other distribution mechanisms such as platform package management tools; we are only proposing to improve the experience for the class of applications that already depend on cargo as a distribution platform today, notably small CLI applications and development tools for the Rust ecosystem.
Additionally, this proposal will make cargo install more useful for the larger set of rust applications that leverage the larger, often preferred, platform-specific distribution workflows, which typically want to install into a temporary directory and package the result. This point is discussed further in the “Rationale and Alternatives” section.
Fundamentally, the aim of this RFC is aligned directly with Rust’s value of “productivity”. It aims to make both building and using small developer CLI tools simpler and easier which has a direct positive effect on the ability of all to improve their workflow.

for context i would really like this (if considered to be viable) to be blocked on https://github.com/rust-lang/rfcs/pull/2196/ because i believe the pattern that this RFC introduces builds the affordances in a way that mitigates much of what i believe could be seen as a downside of this RFC

Currently, cargo install tracks what files it installs, and supports cargo uninstall; this extension mechanism does not hook into that. Potentially, install.rs could emit a list of installed files and let cargo install install them, producing a log of those files for later uninstallation. However, we do not intend for cargo install to become a full-featured package management mechanism; rather, we expect cargo install to work analogously to make install.

I guess I'm one of those who think that clean uninstallation is an important feature. I hope the statement that we are few is wrong.
I avoid make install like the plague and if I actually have to use it, if just invoking some binary in some build dir is not enough, I create a separate root fs and let it install inside there. This is a major hassle that I do just to have the simple pleasure of not getting my file system messed up with random stuff where I don't even know where it came from.
Currently, Cargo solves this in a beautiful way: it just puts a binary into a directory. It is simple, yes, not very extensible, yes. But it allows for clean and simple uninstallation. Updating is equivalent to "forcing" the installation.
With cargo install I actually started installing things into a per-user context because I trusted cargo: it just puts a binary into a directory. But if cargo install becomes extendible, it might put stuff anywhere in my home directory. And I'm not talking about malicious uses, but people who want to "do good" by instaling manpages or desktop shortcuts or file associations or similar.
If the RFC is merged in the current form, I'd probably stop using cargo install, and will be going back to patterns I've learned from C/C++: cargo build'ing and invoking binaries by their path inside the build directory.
Instead I think we should focus on artifact generation: allowing post-build scripts to create .msi files or .snap files or .deb files and allowing users to install them on their own, or maybe even on the user's behalf.

Maybe a more general way is post-build.rs, which can be used to invoke bindgen for example (not only for cargo install).

@est31

Instead I think we should focus on artifact generation: allowing post-build scripts to create .msi files or .snap files or .deb files and allowing users to install them on their own, or maybe even on the user's behalf.

I'm all for supporting that as well. I'd like to make packaging easier. (That also includes policy-compliant packaging that could go into distributions, not just creating a .deb from thin air without a source package.)
What I'm looking for here is a way to let crates provide additional files, not just binaries. Generating manpages, generating completions, and putting them in the appropriate places.
We'd be open to talking about how exactly that should work. For instance, we certainly could have install.rs output a list of files to install and where they should go (including paths to files in the output directory that it just generated), and then cargo install could do the actual installation, and track the files for later uninstallation. That has downsides, in that it's a lot less flexible and puts more policy into Cargo, but it would support cargo uninstall.
Another possibility would be to just install those files into a temporary directory under the output directory, and cargo install would install everything under that temporary directory to the directory specified as its --root.
We'd like to do the simplest thing that could work, here.

@joshtriplett

For instance, we certainly could have install.rs output a list of files to install and where they should go (including paths to files in the output directory that it just generated), and then cargo install could do the actual installation, and track the files for later uninstallation.

That would work as well. in fact we could have post-build.rs generate those files. In the traditional make install set up, the installation step is only copying stuff to the appropriate places, while the previous steps did the remainder of the work.
Your proposal (calling it "microcargo") sounds fine @joshtriplett but I'd like to give an alternative "monolithic Cargo" proposal to just be able to compare the two:
You could have post-build.rs output whether installation of the program is supported and if yes, outputs a list of the files together with a semantic information what they actually are: executables, manpages, completion files, etc... all need to be put into different places and some would require an additional set up step.
Cargo would then store that list and cargo install would just use that list as a recipe for installation. It would then decide based on input parameters whether to install on a global, or on a per-user basis, and depending on that it could then also tell users to set up a ~/.bash_completion or decide which of the completion files to install based on the used shell. It would then track all the files it installed and support uninstallation. Of course this would require to put some logic into Cargo, but after all, Cargo is not a generic make replacement but a tool specifically built for the Rust ecosystem. It does many opinionated choices already.
So with the "microcargo" approach you have install.rs parse all the input parameters and environment info (per-user or global, name of the shell, distro directory layout, which files the user wants to be installed) and output a list of desired destinations. The advantage is flexibility. The disadvantage is that writing a install.rs becomes far more complicated... I've seen so many issues with make install. Often this is not the main attention of the developer and so often it sucks in one way or another. With the monolithic approach, you would have a central place where improvement can happen. However, this downside of the minicargo approach would be fixed by the metabuild RFC.
So summarizing, compared to the "shellout" proposal of the current RFC, I think "minicargo" and the monolithic approaches are both better, with me preffering the minicargo approach.

Something that seems worth mentioning, explicitly: some responses have expressed the sentiment that the right answer is to work towards distribution packaging and similar, rather than making cargo install better. It seems worth pointing out, as someone who does care about distribution packaging as well, that making cargo install better will make it much easier to do distribution packaging. As mentioned in the RFC, we'd like to be able to cargo install to a directory and then package the result, and we'd like to make that include things like completions, manpages, and other associated files.

So, as was mostly expected, the primary question brought up by the comments so far is the issue of uninstall. The current RFC presents the opinion that we should make no effort to offer an uninstall.rs, however it is an open question as to when an install.rs what we can do to mitigate the absence of an uninstall.rs. Several suggestions have been made, but I think that ensuring that cargo emits any files systems changes when an install.rs is run is a reasonable strategy to take.
Would that appease any of those who are concerned about uninstall? If not, let's try to aggregate other suggestions about mitigating the uninstall issue and discuss which if any would be reasonable to implement alongside this, with an eye that there is still the goal of not packing cargo with so many features that it begins to resemble a platform-specific package manager. (which i think we all agree is something we DON'T want to happen.)

ensuring that cargo emits any files systems changes when an install.rs is run is a reasonable strategy to take.

I'm not sure I understand your proposal fully. Could you explain it? Suppose I did cargo install foo. With which command would I uninstall it? And how would it look implementation wise?

I think she's talking of a general class of solutions (correct me if I'm wrong), but one such proposal would be that install.rs emits cargo:installed-file:/path/to/manpage, and cargo keeps these strings around. On uninstall it removes these files and then removes the binary.
(This doesn't cover all cases of uninstall, but as the RFC mentions this isn't intended to be a generalized distribution solution)

@Manishearth Right, the idea would be that install.rs either prints a list of all filesystem changes it wants to make but doesn't make them directly, or alternatively that install.rs only installs things to a subdirectory of the output directory and then cargo install installs those things. Either of those would allow cargo install to optionally remember what it installed for later uninstallation.

@est31 yup, what both @Manishearth and @joshtriplett expansion of what i said is correct. i'm curious if that's enough to address the issue- and if not, i'm curious what strategies would best address the issue without growing cargo into a thing similar to a full fledged platform-specific package manager, which, i believe, we all agree that we don't want.

This strategy seems reasonable, and mostly assuages my concerns. There's of course the issue of regedit and such things on Windows where you are dealing with non-files. But I think that falls under not having a "full fledged platform-specific package manager"?
Personally I prefer the variant where cargo moves the actual files and the install.rs creates them and points to where they should be.
I think an argument in favor of this RFC is that we can always decide to permit an (optional) uninstall.rs file later on (if we wish / there's sufficient demand for it). And so we can start with a conservative initial feature.

One of the problems that crops up with other languages' package managers (hi, Python!) is that the tool chooses one particular convention for where to install files and uses it on every platform in the name of consistency. Usually this convention is highly-POSIX-flavoured, making things weird and awkward to use on Windows.
I realise that ship has kind of already sailed for Cargo, since it installs binaries into $CARGO_HOME/bin, but is there a risk that adding more "install" support to Cargo will make it even more awkward on Windows, or might even make some tools unnecessarily Windows-incompatible if they try to install a file named "aux" or something?

@Screwtapello so, i share this concern but i think this RFC alleviates it instead of exacerbating it. we specifically are picking to do a install.rs instead of a post-build type situation so that authors can detect platform and perform platform specific functions. there's no way to necessarily guarantee that authors will do it, but the motivation for this RFC, at least partially, is the fact that authors WANT a better cross-platform story.

I didn't see any mention of passing arguments to cargo install - is this a deliberate choice or just unspecified? If unspecified I'd like to propose allowing it (to avoid a world of ad-hoc environment variable usage).

@aidanhs it was not opinionated! i think that would be fine, and likely uncontroversial? but i'm curious what others think (assuming no one objects i can specifically add that to the RFC)

the idea would be that install.rs either prints a list of all filesystem changes it wants to make but doesn't make them directly, or alternatively that install.rs only installs things to a subdirectory of the output directory and then cargo install installs those things.

@ashleygwilliams

i'm curious if that's enough to address the issue

Yes, both variants as laid out by @joshtriplett would be enough (see quote above). I'd personally prefer the variant where install.rs is not doing the changes directly and would like install.rs to be sandboxed. It could have unrestricted read access but write access restricted to one directory inside the target/ hierarchy to ensure that install.rs is actually using cargo to install stuff.
Especially, it seems to me that the intention is not for install.rs to be as flexible as possible, as this would mean support for any kind of installation of applications (cargo replacing an OS package manager), but rather to allow the minimum of features for CLI applications to install their manpages and completion files.

I realise that ship has kind of already sailed for Cargo, since it installs binaries into $CARGO_HOME/bin

@Screwtapello  This may not be true in the future if something like rust-lang/cargo#5183 is merged.

Installing to a staging area, and from there copying to either ~/.cargo or /usr/local would seem sensible (depending on whether the install is global or user-local). Then the uninstall actions can be easily generated just by walking the staging area directory tree. Possibly custom install/uninstall commands could also be added for things like registry edits on Windows. Alternatively, the staging area could be a permanent one, with symlinks into one of the two aforementioned dirs. this is how Mac’s Homebrew does it.

I think that I'm against this RFC. Instead of allowing crate authors to put arbitrary steps inside of install.rs, I would prefer that we extend cargo and Cargo.toml to support additional crate artifacts types.
To make it concrete, I'll use man pages as an example. With this RFC, crate authors would write an install.rs script which would generate and move man pages into the correct location. I'll refer to this as the imperative approach.
An alternate approach would be to allow crate authors to put something like man-dir = "man" into their Cargo.toml, and let Cargo put any man pages it finds there in the appropriate place. I'll refer to this as the declarative approach.
I think that the declarative approach is superior for a number of reasons.

Under the imperative approach, all crate authors must write their own install.rs scripts. Although they can certainly draw from each other's scripts, this is a great deal of duplicated effort. Additionally, installation is complicated by different distro and platform quirks and conventions, so many install.rs scripts will likely have bugs. With the declarative approach, this code lives in one place. This allows for less duplicated effort. And, when bugs are found, they can be fixed for everyone in one place.

Different distros have different ideas about where things should go. For example, man pages will go somewhere entirely different on Ubuntu, homebrew, macports, nix, or windows. With the declarative approach, they can write a single adaptor script that parses Cargo.toml, which will work with all crates without further effort. With an imperative approach each distro will have to examine and adapt the install.rs script to identify man pages and put them into the correct location.

With the declarative approach, we can add additional support for special handling of different kinds of crate artifacts. For example, we could support a command line flag to only install binaries and not man pages, or vice versa. It would be very hard to support this using the imperative approach, since all install.rs files would need to support this flag. Additionally, with the declarative approach, different crate artifact post-processing steps can be added as needed, for example converting to CRLF line endings on windows.

So, to sum up, I think we should extend cargo with the ability to understand and install crate artifacts on a type-by-type basis, so that they can be handled consistently and appropriately across platforms, and not left to individual crate authors.

@casey I think that's fair for a large number of cases, but what about certain Rust projects like tectonic, for which this RFC would be hugely useful in its present form?

@alexreg, what kinds of artifacts does tectonic need to install? I'm wondering if there might be a better way to support them.

@alexreg, what kinds of artifacts does tectonic need to install? I'm wondering if there might be a better way to support them.

Probably best we check with the Tectonic people, like @pkgw, since I'm not 100% sure at this point.

@alexreg I think that Tectonic is pretty happy with the current version of cargo install — you could imagine downloading some resource files upon install, but I'm not even sure if I'd want to do that if it was an option. Is there a pain point that you have in mind?

@alexreg I think that Tectonic is pretty happy with the current version of cargo install — you could imagine downloading some resource files upon install, but I'm not even sure if I'd want to do that if it was an option. Is there a pain point that you have in mind?

Man pages and font files are two things I was thinking of.

i honestly don't want cargo install downloading files. That's precisely the kind of misuse I was talking about: not malicious but abusing the feature in order "to do good" or assuming things that the user might want. Downloading stuff in install.rs would be absolutely horrible and bad and not okay. It means that you are forced to use cargo install in order to fully use the program. It also means that it just assumes network connectivity... you don't always have internet!
I quite like @casey 's declarative approach. It is basically a built-in metabuild feature. That's okay.
Otherwise I think we should use sandboxing. The suggestion of abusing install.rs for downloading has made my opinion on sandboxing clear.

So, to sum up, I think we should extend cargo with the ability to understand and install crate artifacts on a type-by-type basis, so that they can be handled consistently and appropriately across platforms, and not left to individual crate authors.

@casey your proposal sounds like an excellent outcome, but I wonder about the practicalities. Who maintains this matrix of artifact behaviours per platform? I'm a little concerned that the end result effectively ends up being a bunch of slowly-iterated on 'officially supported' code in Cargo that has to be right for all artifact types, for all different targets (Windows, NixOS, Ubuntu, etc). This is a huge undertaking, and also bottlenecks any new type of artifact on long discussions.
One thing that I appreciate about build.rs files is the recognition of the world as it is today, not as we wish it to be. There are a few well known C and C++ projects out there that download things during configure and build - it's not great but it happens.
This RFC seems to be along similar pragmatic lines. I see the attraction of something more controlled. But I then also have questions like "can I ask the user if they want to add an environment variable to their shell environment script" and "can I run ldconfig if the user has done a sudo install of my cdylib". Something more minimal gives the community a chance to a) get going asap and b) build things like the cc crate in the longer term for modularity (and this is a bit richer than just "draw from each other's scripts"). I don't know that a small set of people spending a lot of time ironing out your proposal puts us in a better position than getting things into the hands of users and letting things grow without needing constant bandwidth from people involved in Cargo.

Who maintains this matrix of artifact behaviours per platform? [...] This is a huge undertaking, and also bottlenecks any new type of artifact on long discussions.

If cargo developers can't do it properly, why should crate authors be able to? crate authors are less experienced with these matters, and generally care less. If an user installed a crate if we had install.rs, at worst we would have the behaviour of the crate on crates.io with the worst behaviour. If an user installed a crate with the "cargo managing everything" scenario, the worst case would be the average and the best case. No bad surprises here. Any crate author which wanted to improve the experience for its own crate would improve it for everyone.
And even if they used some library in their install.rs or metabuild whose authors do care, they will most likely use an outdated buggy version of it. rustc-serialize is still in the the top 20 of directly depended on crates on crates.io, just to give an example. And in the case of install.rs it's even worse as the crates are all binaries and their Cargo.lock files are uploaded to crates.io so there is nothing the authors of the library/metabuild crates can do about it (releasing a minor version will still require a cargo update).

Something more minimal gives the community a chance to a) get going asap

I'd argue that install.rs is not minimal. You allow any crate to run any code for whichever purpose it wants.  If you limit the scope of the feature, that's minimal.

can I ask the user if they want to add an environment variable to their shell environment script

rustup is very transparent about the fact that it adds an environment variable to ~/.profile. This kind of high standard might be in place by many codebases, but some people might also think that this slows down installation and informing the user makes user experience worse... If everything is unified in one place (cargo), users can tell cargo about their prefferences.

can I run ldconfig if the user has done a sudo install of my cdylib

This was not listed in the use cases of this RFC. The use case was for CLI apps.
Something like installing libraries should definitely be left to your OS's package manager.
If the library is less used so it is not packaged by debian and friends, you can just let the program's build system use cargo... Cargo being able to integrate into larger build systems was I think on the 2017 roadmap.

One thing that I appreciate about build.rs files is the recognition of the world as it is today, not as we wish it to be.

I am a supporter of build.rs. You need to be able to shell out in order to compile C code. Sometimes a C dep has a complicated build system. So you need to do more than just invoke gcc.
I am a supporter of post-build.rs. You should be allowed to modify the built artifacts and generate forms that are more interesting to you, and use any tool for them you want.
But I think that users get the gist of build.rs and post-build.rs much better than they get the gist of install.rs.

There are a few well known C and C++ projects out there that download things during configure and build - it's not great but it happens.

C/C++ stuff won't be invoked by install.rs... will it? Why should it be invoked?

@casey your proposal sounds like an excellent outcome, but I wonder about the practicalities. Who maintains this matrix of artifact behaviours per platform? I'm a little concerned that the end result effectively ends up being a bunch of slowly-iterated on 'officially supported' code in Cargo that has to be right for all artifact types, for all different targets (Windows, NixOS, Ubuntu, etc). This is a huge undertaking, and also bottlenecks any new type of artifact on long discussions.

Exactly this. The precedent has been set for allowing arbitrary build.rs scripts, so why not arbitrary install.rs scripts? Sure, we could publish guidelines, but being overly restrictive is not a good approach, despite the benefits gained from declarative installation scripts -- for the reasons you mention above. Some "safeguards" could be created, like crates.io having CI that checks that the crate  installs without an Internet connection... but I wouldn't go further.

@casey Declarative is exactly what I had in mind, with the idea of metainstall. The thought was simply that those individual declarative steps could get implemented outside of cargo itself, in separate crates in the ecosystem.

A strong benefit of the declarative approach is that distros can use the declarations to do the distro-specific right thing with artifacts. If it's in rust code in a build.rs script, then this benefit is lost, and distro package maintainers will have to do a lot more work.

@casey your proposal sounds like an excellent outcome, but I wonder about the practicalities. Who maintains this matrix of artifact behaviours per platform?

The metadata is in Cargo.toml, and individual distros maintain an scripts, in whatever form, to process that metadata and package it in a distro-appropriate way.

This is a huge undertaking, and also bottlenecks any new type of artifact on long discussions.

I think that this is actually desireable. I think that cargo install should stay minimal, and it should only support installing things that it can do well, i.e. deterministically uninstall.

Exactly this. The precedent has been set for allowing arbitrary build.rs scripts, so why not arbitrary install.rs scripts?

The purpose and scope of build.rs scripts is far clearer than that of install.rs scripts. As a result, even though a crate author can make a build.rs script do weird stuff, they probably won't. I don't think that this is the case with install.rs scripts. Will crate authors try to install files to /? Will they try to mess with dotfiles files? Will they try to add and remove user services? I think that these are all possibilities and are all highly undesirable, not least because there is no guarantee that they will be deterministically undone.

The purpose and scope of build.rs scripts is far clearer than that of install.rs scripts. As a result, even though a crate author can make a build.rs script do weird stuff, they probably won't. I don't think that this is the case with install.rs scripts. Will crate authors try to install files to /? Will they try to mess with dotfiles files? Will they try to add and remove user services? I think that these are all possibilities and are all highly undesirable, not least because there is no guarantee that they will be deterministically undone.

The purpose of install.rs can be made just as clear with documentation and guidelines. In any case, your reasons aren't good ones, because build.rs files can in theory do all of these things too, and the only thing stopping the author of either a build.rs or install.rs script is guidelines and convention.

i agree with @alexreg here. i am not convinced that the distinction between build.rs and install.rs is enough to legislate in theinstall.rs and NOT build.rs. part of the developer experience i like about the current proposal for install.rs is the symmetry between it and build.rs.
am i right in saying there is some consensus on @est31's comment here which suggests that @joshtriplett's articulation of the install.rs mitigation to  allow uninstall is the way we'd like to go? if so, i'll update the PR today. just want to check that my sense of some consensus is accurate before doing so :)

also, just to reiterate, i do think we should block on the metainstall|build RFC because as has been articulated repeatedly, we definitely want conventions to emerge.
we seem to still be figuring out if we want those conventions to be directed by cargo itself or by the ecosystem, but at least we agree we want conventional behavior to be the happy path here. i'm currently of the mind that leaving it to the ecosystem is both FINE and potentially preferable but i'm open to being convinced about that.

@ashleygwilliams

am i right in saying there is some consensus on @est31's comment here which suggests that @joshtriplett's articulation of the install.rs mitigation to allow uninstall is the way we'd like to go?

Let me +1 this. :)  I full share the concerns that have been raised above about make install being a horrible mess. make install is a great way to get your computer thrashed beyond repair. I have had to learn the hard way that I should never run sudo make install. Please let's not use that as a model. "People don't expect a working make uninstall" is a statement akin to "people don't expect a reasonable build system for a systems language"; if that's our standard, we might as well go back to makefiles. I think we can do better :)
Providing a guarantee that cargo uninstall will properly undo everything is hard, but I think we can at least make some effort to make this a common case -- just like build.rs could do all sorts of horrible things to my local system but most of them just invoke gcc. I think I prefer the variant laid out by @joshtriplett that lets cargo do the actual copying, combined with some libraries offering wrappers around libstd file accessing functions. We could then have the clear rule that if you fs::copy in your install.rs, you're doing it wrong, and I think it is not hard to explain that this helps with uninstallation. I like to think that the reason make uninstall is broken is not that software authors want to make the lives of people that uninstall their software miserable, I'd like to think that they actually would prefer uninstallation to work so if we tell them "that's easy, just use the wrappers in this lib for installation and you are all set", they'll just use it. I hope I am not being too naive here :D

@RalfJung Right. This is why I was suggesting having a staging area is highly beneficial for a) containing cargo install operations (at least if one follows the conventions), and b) producing reliable uninstalls.

@ashleygwilliams Yes, that sounds quite reasonable, and was similar to what I was suggesting above too. I would also lean towards an install.rs script not actually messing with the system or the filesystem except within some staging directory (subdirectory of the project target dir perhaps). Then cargo install would copy everything from the staging dir (it would have subdirs for bin, lib, man, etc, and so on), and save this information at the same time, so it knows exactly how to uninstall. I'd also like to see cargo install able to read the output of the install.rs script just like cargo build reads the output of the builds.rs script, though – this would be useful for additional directives, like perhaps installing cron/launchd tasks, or registry entries on Windows. Then they too could be reliably noted and uninstalled.

@aidanhs A println like that, combined with putting generated files in a temporary staging directory under the build output directory. (I'll avoid bikeshedding numerous bits about the previous line for the target directory.)

I basically agree with @aidanhs & @joshtriplett's last commits, with the proviso that I think we should make everything as convention-based as possible. Specifically, we should install to bin, lib, man, etc. dirs under the staging area, then Cargo can handle the rest. See my above comment on this for more info. The lines printed to stdout can be used for more specific directives to cargo.

@aidanhs

the Rust community has come up with some amazing libraries/tools/solutions that have then gained widespread de-facto adoption - in general I'm not entirely comfortable with the line of thinking in the quote above.

I'm not worried about the 80% of people who will use such libraries. I'm worried about the 20% who don't. And even if you use such a library, you might be using an older version. Just look at how many crates still depend on rustc_serialize.
Also see my other point: If stuff is managed in cargo, improvements will immediately reach everyone who updates cargo. If stuff was managed in libraries, improvements would not just require that the library author is on the maintained semver branch of the library, they would also require that the author is running cargo update in the first place and uploads a new release to crates.io, as cargo install IIRC takes the Cargo.lock of the crate file, it doesn't create its own. For some reason, I think that most crate authors won't do this. And I don't think we should force them to. It is not ergonomic to update your crate every month just to bump the Cargo.lock.

If you put all the logic into the crate (via install.rs or whatever), even if there's reliable and robust helper-libraries for crate authors to rely on, there'll always be problems with crates that don't use the libraries, or use the wrong libraries, or use an older version of a library that doesn't understand the platform you're trying to build for, and the version that does understand is on the other side of a compatibility-breaking major version bump.
If you put all the logic into cargo, then presumably cargo that runs on platform X will always know how to install apps on platform X, so compatibility won't be an issue. However, there's only one cargo, so it needs to be extremely conservative about adding features, and that makes it hard to explore and prototype.
The third option is to put all the logic into third-party tools. Have a cargo-install-posix tool that knows about manpages and cron-jobs. Have a cargo-install-win32 tool that knows about registry keys and system services. Have cargo-build-rpm for RedHat and cargo-build-pkg for macOS and whatever other tools that handle other platforms, and can have their own, separate release cycles and compatibility guarantees. Ideally there'd be some well documented conventions that these tools could share (in the same way that cargo hands examples/*.rs files specially without configuration, these third-party packaging tools might automatically treat docs/*.md files as documentation to be converted to a platform-specific format and installed in a platform-specific location without explicitly being told to do so) but each platform will always have some special quirks of its own.

Specifically, we should install to bin, lib, man, etc. dirs under the staging area, then Cargo can handle the rest.

I worry that's already too platform-specific.

I'm maintaining cargo-deb, and I'm having hard time seeing how this helps packaging.
How could that work with packaging for different targets than the host OS?
For packaging I use a declarative list of assets to bundle plus additional metadata in Cargo.toml. I don't think I could reverse engineer a good Debian package from just installed files on disk, so I'd still have to combine scraping Cargo.toml with the install process?
Also cargo-deb users ask for more power features around packaging, e.g. multiple variants of a package built from the same crate (for internal/test and external use), as well as separate packages with and without debug symbols. I guess it could work if cargo install was able to install multiple copies to multiple chroot-like environments?
But the current solution where cargo-deb reads cargo metadata and invokes build itself (with custom flags if required) seems to work well for packaging, since it gives full control over the process.

You're saying you're not making a package manager, but everything about this RFC implies that you are making one (just an incomplete one).
As a user my main grief with cargo install is that it's just dumps files on disk and abandons them. There is no update mechanism. And it's going to be even worse when uninstallation will leave files behind. If install configures installed files to be run somewhere, then uninstall deleting executables, but not config that enables them, will likely leave broken stuff behind.
So please, make it a proper package manager. Or 100% defer to other package managers - e.g. invoke homebrew on macOS, apt on Debian.

I agree with @kornelski that we should be taking one of those two approaches. That said, the approach we've proposed above with the install.rs script is still a good one. Homebrew effectively does something similar.

@kornelski It would support cross-building exactly as well as cargo itself does. But in any case, it helps packaging in that you can install into a temporary directory and package the result, which works far more easily than handling individual files twice, once in the build and once in packaging. The result would work exactly like packaging a program that uses make install into a temporary directory.
And yes, we will make it uninstallable.

You're saying you're not making a package manager, but everything about this RFC implies that you are making one (just an incomplete one).

Whether or not it's a package manager, is not an important thing. Just make it do what we want it to do.
I think the goal of cargo install is: a light tool to help rustaceans (who have installed cargo) downloading CLI development tools easily. 3 key points: light&easily, rustaceans, unpopular devtools.

e.g. invoke homebrew on macOS, apt on Debian.

It's a more heavy way. It's not easy to upload unpopular tools to homebrew/apt.

@ashleygwilliams

also, just to reiterate, i do think we should block on the metainstall|build RFC because as has been articulated repeatedly, we definitely want conventions to emerge.

Update: That RFC is now merged. =)

Currently cargo install is safe and easy. It happens to have most properties of a proper package manager because of its 1-file simplicity:

it only puts files in locations it owns (.cargo/bin)
it can correctly delete everything it installed (i.e. one file)
it's aware of conflicts between packages and other files (i.e. none, except the trivial case of reinstallation of the same executable)
is fully portable across operating systems (they all support a single executable in PATH)

This RFC allows breaking all of these key properties:

allows installation of files in other, arbitrary locations.
does not have a way to track what has been installed nor uninstall extra files.
has no way of tracking conflicts between packages and with other package managers.
enables features that are system-dependent, and requires install scripts to have OS-specific support.

As a user, I won't know what a package is going to install. Is it going to install a bash completion file that will start erroring after I remove the executable? How am I going to track all the files down to remove the package?
Is it going to run daemons, add itself to autostart? Will it know how to create launchd XML definition files to properly run on macOS, or is it just going to append a line to my bashrc?
If I already have the same package installed via other package manager, is it going to just blindly overwrite parts of it and make my package manager freak out about it?
How am I going to handle upgrades? If the package installs configuration files as part of its own install process, is it going to overwrite my customized config? Is it going to merge configs? apt can deal with that. Is any random install.rs going to?
What's stopping packages from trying to install stuff in /usr and spectacularly fail on Windows?
So I'm afraid these new properties open Pandora's box. I don't like that Cargo wants to add more capabilities without taking responsibility for them.
The best case scenario I can image is that someone else will write fully-featured package manager as a "metainstall" script. But then there will be nothing to enforce use of the fully-featured package manager, and as a user I'll still be afraid that someone just wrote install.rs with a bunch of fs::copy that's going to screw up my system.

it can correctly delete everything it installed (i.e. one file)

Actually, no. Cargo does not keep track of the files it installed. If you do cargo install foo, and then foo changes the name of its binary, and you do cargo install foo again, you now have both binaries sitting around.

@RalfJung OK, so that's a bug. But is that a justification to completely give up on ability to uninstall packages?

@Kornelski Lots of package managers allow modification of system-wide directories, but deny it by convention. One can even deny permissions for the program to modify stuff outside of a certain dir, using things like chroot. You overstate the issue here.

@kornelski I think I agree with you that this RFC isn't yet complete, but I do think it just needs a bit of expansion to be in a workable form. As has been stated before, build.rs scripts can already modify anything on your system, but don't, by convention/guidelines. I think guidelines are probably the best way to stop build.rs and install.rs doing stupid system-wide things, as in other package managers (e.g. Homebrew), but if we could use chroot to help with the filesystem aspect, that would be good at least. Support for things like cron/launchd could be done via "notes" at first (that's what some package managers do still) – i.e. simple instructions on how to set it up, but the install not actually carrying it out itself.

I think guidelines are probably the best way to stop build.rs and install.rs doing stupid system-wide things

install.rs's very purpose is to be unhygienic. To change something. On a per-user level or a system wide level. Saying "build.rs should not change stuff outside the build directory or temporary directories" is a very clear rule. But saying "this way of changing the system is bad but that way of changing it is good" is much harder to formulate and might mean different things to different people.
Can someone tell me about a use case that is important enough that we give up sandboxing or guaranteed uninstallability? The original use cases (completion files, manpages) would all be covered by sandboxed solutions or even declarative ones.

@est31 I think we've already established in this thread that we should find an approach to install.rs that builds things in a build directory and emits information that cargo install can use to install (and optionally remember to uninstall).

install.rs's very purpose is to be unhygienic. To change something. On a per-user level or a system wide level. Saying "build.rs should not change stuff outside the build directory or temporary directories" is a very clear rule. But saying "this way of changing the system is bad but that way of changing it is good" is much harder to formulate and might mean different things to different people.
Can someone tell me about a use case that is important enough that we give up sandboxing or guaranteed uninstallability? The original use cases (completion files, manpages) would all be covered by sandboxed solutions or even declarative ones.

Absolute nonsense! We've already been over this, and collectively countered all your arguments. It can be made perfectly clear by guidelines what an install.rs script should do; this is no serious issue.

We don't have sandboxing for build.rs, and any sandboxing mechanism would necessarily be platform-specific. I would suggest that we leave sandboxing as an open question, and reconsider it after some experimentation with install.rs and what kinds of things people want to use it for.
Right now, if someone wanted to, they could have build.rs go install arbitrary files on the system, and yet they don't. I think if we make it clear what install.rs is supposed to do (install files to a temporary location and print some output), we'll get similar results.

@joshtriplett Yeah, sounds fair. Sandboxing is a somewhat complex problem, and can come later if necessary.

reconsider it after some experimentation with install.rs and what kinds of things people want to use it for.

I could live with this being converted to an experimental RFC, or alternatively a nightly only feature that turns off sandboxing. But introducing sandboxing after the fact is quite a breaking change...

I think if we make it clear what install.rs is supposed to do (install files to a temporary location and print some output), we'll get similar results.

I admit, the model is a bit limited. You don't need more however if you just want to add shell completion files. However, those people who will run into limitations of that model, will inevitably just ignore it.
We have the choice what to do with these use cases:

allow them by not sandboxing, turning install.rs into a mess similar to make install or
disallow them and tell them that install.rs is not ready yet for these use cases.

We can either err on safety, or we can err on flexibility. There is no between. And you probably know what I think that should be done =).

I also agree that some experimentation first would be a good idea so we can see what people use it for and so on.

Things like shell completion, launchd/cron jobs, etc. can be handled either by writing directives to stdout, like previously mentioned (and thus retaining sandboxing), or writing manual instructions of shell commands to run to a NOTES file. The latter could be a solution until we flesh out the former, at least.

I think most build systems are converging on a model of installing the files to a given directory. Distribution-specific tools can then take it from there, and maybe re-organize things a little. Others dump everything in the build directory, and let the packager pick the destination files.
I wouldn't say that make install is a mess, however:

it doesn't track what it installed, so it can leave a mess behind
people do sudo make install and get root-owned build files
setting the installation path/prefix isn't always obvious

cargo install can avoid these issues, assuming the crate author is not malicious. Even in that case, the risk is not greater than for build scripts. E.g. I saw a build script that compiled C code to the cargo repository instead of OUT_DIR, and cargo clean didn't know about it.

The user has to trust the installed executable anyway (and all build.rs scripts involved), so any protection against actively malicious install.rs scripts is future. The remaining problem is helping well-meaning authors to make well-behaved scripts, especially when said authors are not familiar with nuances of system-wide installations and installations on all operating systems they don't use.
The idea of telling install.rs scripts to install files to a chroot-like directory sounds good from perspective of keeping the OS clean, but I'm not sure how script authors would know what subdirectory names to use. In this solution the "API" would be the directory layout, and that still is very system-dependent. Does writing to $OUT_DIR/bin put stuff in C:\Program Files? What paths are mapped to roaming profile directory?

Would it be possible to limit this RFC only to the "metainstall" feature, and not provide "bare" install.rs scripts, at least initially?
The worries I have are mainly about simplistic install.rs scripts that would just blindly copy-and-forget files to system directories. However, it would be fine if a de-facto standard metainstall crate emerged that worked like a well-behaved package manager.

For improved uninstallation I see two options:

Allow scripts to print cargo:delete-when-uninstalling={path}, remember these paths, and delete them when cargo uninstall is run.

Simple to implement and understand, and gives consistent behavior that Cargo can control.
Doesn't support advanced cases, e.g. deleting files created after the installation, or tracking renamed files.

Extend the "metainstall" API to also have an uninstall call.

It would support all advanced cases, and metainstall crates that performed installation are most qualified to perform the uninstallation.
It would mean that uninstallation runs code, and possibly even needs to compile it.
"Add or Remove programs" on Windows takes that approach. Quality of uninstallers varies greatly.

@kornelski We’ve covered most of these points thoroughly already. I don’t see why this discussion is continuing...

@kornelski,

The idea of telling install.rs scripts to install files to a chroot-like directory sounds good from perspective of keeping the OS clean, but I'm not sure how script authors would know what subdirectory names to use. In this solution the "API" would be the directory layout, and that still is very system-dependent. Does writing to $OUT_DIR/bin put stuff in C:\Program Files? What paths are mapped to roaming profile directory?

There are libraries that provide those mappings based on XDG-basedir-spec, XDG-user-dirs, FHS, known folders and standard directories. Quick search on crates.io suggests directories as good candidate.
And then on top of this we need to find out where to install the shell completion files, user systemd units, desktop files etc., which will call for another layer on top of that. There needs to be some guidance provided on that, but it does mean that cargo install will have to provide an option to install to arbitrary path, because cargo can't know all the shell and tools that may need to be integrated with.

Just look at how many crates still depend on rustc_serialize.

@est31 isn't rustc_serialize an official(ish) solution replaced by the community with a better one? In this RFC comment thread the design is moving to prevent alternative approaches. Regarding "everyone gets the improvements" - this is the large vs small stdlib argument and we won't resolve that here :smile:

I can that consensus is forming behind the general thrust of the sandboxing approach, so I'll just jot down some final 'contrary' thoughts:

Minor: the discussion seems unix-centric - isn't regedit is a fairly important part of CLI app install on Windows (e.g. to put it on your path / to identify where your app is installed to find assets)? But files on disk is step 1 (and devs can perform 'post-install' steps in their binary) so maybe it's ok.
One point of discussion is "who sets conventions - ecosystem or cargo". The sandboxing thing (to me) ends up implying "cargo decides conventions" (since you're only empowered to make decisions on things cargo says you can). I'm ok with collectively selecting that as the approach, just wanted to point out that these are linked.

I personally just want curl ... | sh rewritten in Rust, so I'll wait for a cargo 'run-script' RFC that will allow something like cargo run-script cratename install.

I could live with this being converted to an experimental RFC, or alternatively a nightly only feature that turns off sandboxing. But introducing sandboxing after the fact is quite a breaking change...

To echo others: robust sandboxing that supports Linux kernel 2.6.18, Windows 7 and OSX is hard (basically requiring a SUID helper binary on Linux). 'Good enough' sandboxing (assuming we agree on what that means) is still tough. Let's bear in mind that the goal of this RFC is modest - "make CLI apps easier to install". I'm sure we can come up with a way to make it so easy to do things the 'right way' that most authors never even think of anything else, or just explicitly document that anything you do that isn't creating a file in a specific directory may break in the future.
I like @RalfJung's take:

I like to think that the reason make uninstall is broken is not that software authors want to make the lives of people that uninstall their software miserable, I'd like to think that they actually would prefer uninstallation to work so if we tell them "that's easy, just use the wrappers in this lib for installation and you are all set", they'll just use it. I hope I am not being too naive here :D

Minor: the discussion seems unix-centric - isn't regedit is a fairly important part of CLI app install on Windows (e.g. to put it on your path / to identify where your app is installed to find assets)? But files on disk is step 1 (and devs can perform 'post-install' steps in their binary) so maybe it's ok.

Well, my above suggestion was to use some convention based (Unix-like) FHS-like approach (@jan-hudec expanded on this with good ideas). For registry edits, I suggested we just have install.rs scripts write directive to stdout, which get interpreted by cargo. They'd be ignored on all platforms but Windows (possibly with warnings output, since ideally you'd want install.rs script authors to check the OS before emitting them).

One point of discussion is "who sets conventions - ecosystem or cargo". The sandboxing thing (to me) ends up implying "cargo decides conventions" (since you're only empowered to make decisions on things cargo says you can). I'm ok with collectively selecting that as the approach, just wanted to point out that these are linked.

Makes sense to me.

One point of discussion is "who sets conventions - ecosystem or cargo". The sandboxing thing (to me) ends up implying "cargo decides conventions" (since you're only empowered to make decisions on things cargo says you can). I'm ok with collectively selecting that as the approach, just wanted to point out that these are linked.
[...]
I personally just want curl ... | sh rewritten in Rust

I'd say that the approach is already "cargo decides conventions".
Make is basically a glorified shell... for the execution of targets, make actually invokes a shell, the syntax is similar, etc etc.
CMake is no different. It has some special syntax built in but every library did its own thing and reimplemented the same stuff over and over and it is still an ugly mess.
Cargo's traditional approach is radically different. Instead of being a generic tool that allows everything, it has a ton of stuff hardcoded that the code has no say over.

it only supports Rust
non-git non-path deps are hardcoded to the crates.io domain (at least unless the user changes .cargo/config)
it has the concepts of build, run, and test

When I came to Rust, it was surprising to me to see all of this stuff hardcoded. But then I saw how all of this hardcoding is benefitting the Rust ecosystem.
You don't have non-rust stuff to worry about, you don't have everything spread out on several hosts with possibly several policies, you know instantly how to compile something (just do cargo build).
So I think we should follow that approach and be opinionated for the user's sake :).

To echo others: robust sandboxing that supports Linux kernel 2.6.18, Windows 7 and OSX is hard (basically requiring a SUID helper binary on Linux). 'Good enough' sandboxing (assuming we agree on what that means) is still tough.

It doesn't have to be watertight. I think something that causes big headaches for people who ignore the convention (or who don't know... most times it is negligence :p ) is enough. I don't want to go into detail about implementation but I think the two options on the table are gaol and interpreting stuff with miri.

Any progress on this lately?

I suppose this hasn't been high on people's priority lists lately, given the lack of news. Any chance of reviving things?
Some questions I had:

Is the plan now to make cargo closer to a lightweight package manager, per above discussions?
Should we retain separate "repositories" of Cargo binaries & their support files that switch depending in the toolchain, in a pyenv-like manner?
What's the interaction with Cargo components going to look like in the future? It seems like components could just be "official" versions of Cargo packages, if we extend the cargo install system to support a more general concept of "packages".

It makes the emergence of conventions significantly more difficult as the option to reuse, share, or automate this task has significantly affordance than integrating it into the familiar cargo step.

I'm struggling to parse this. Can you tell me what you mean?

Looks like this is being revived as part of https://github.com/rust-lang/crates-io-cargo-teams/issues/14. @ashleygwilliams / anyone else, how are we going to proceed here?

This RFC sounds excellent! In particular, this approach to offering cargo uninstall strikes me as very reasonable:

....Right, the idea would be that install.rs either prints a list of all filesystem changes it wants to make but doesn't make them directly, or alternatively that install.rs only installs things to a subdirectory of the output directory and then cargo install installs those things. Either of those would allow cargo install to optionally remember what it installed for later uninstallation.

I was surprised to see that there hasn't been any movement on this in a year and a half. Has it been superseded by more recent initiatives? If so, could someone provide a link?

This came up in today's @rust-lang/cargo meeting. This hasn't seen activity in a long time, and as a result, we'd like to postpone it. This should not be taken as a lack of interest; we just don't want to leave this open and unresolved indefinitely. If someone would like to pick this up, they should feel free to do so, and post a new RFC.
In general, if someone is interested in reviving this, they should specifically write this in terms of being able to stage things in an output directory and letting Cargo install them, rather than installing things directly. We're generally not in favor of a completely general "install things directly onto the filesystem" script. (We're also uncertain about the amount of complexity this might create in Cargo, and we'd want to know this wouldn't impact long-term maintainability.)
Hopefully this guidance helps someone looking to work on this in the future.
@rfcbot postpone


Not rebuild application if same version already deployed.

Sometimes one only wants to do a deployment for changing some configuration, it would be nice if this were possible without re-building and downloading everything. This way the guy managing configs does not need to be a developer with a beefy machine, also the re-building is obviously not needed.

I'm surprised that things get rebuilt in this situation. Nix should know how to not duplicate work. Or perhaps you mean: When someone else deploys, they shouldn't need to rebuild on their machine? In this case the target server would have to act as a cache?

Right, that's what I mean. If  you have built it on your machine nothing gets rebuilt of course.

Well I guess this could be implemented by having the server act as a cache I suppose (should obviously happen via ssh), yeah I guess this would make sense!

I think you can already do this manually by using nix-instantiate on your deployment (you'd have to reconstruct the right command by looking at the verbose output of ob deploy). With that you could use nix-copy-closure --include-outputs from the server to your local machine.

There are two major issues with the idea of pulling build artifacts from the deployed servers.

The build machine should not trust the contents of the store on any deployed machine.  What if the deployed machines are ever hacked?  If the build machine is compromised, recovery will be very difficult.
The build machine should not be stealing resources - in this case download bandwith - from the production machines.  What if you're trying to deploy because you're underprovisioned and the system is therefore struggling?  The last thing you want to do is have the build machine start leaning on the production cluster.

Instead, I think we should make the build machine not lose the artifacts in the first place, e.g. by creating GC roots with nix.  Failing that, the build machine could pull from the CI machine.

I definitely agree that using the target server as a cache and/or copying from it is not a good idea for default behavior. But I must confess I've wanted this several times before when I knew that the server wouldn't be adversely impacted. The use case has usually been a "quick fix" to some simple configuration, often to a staging server where I'm not as worried.

But why did your build server not have that stuff?

I didn't have a build server :)

Oh, but I suppose your question still applies for just my local machine. Cases I've run into before:

Using a different machine than the one I built on
Trying to deploy to an old server where I no longer had the cache
Using a VM for the builder and the VM was previously deleted (especially common on Mac)

Examples of quick fixes I've wanted to do:

Change domain name
Change nginx rule
Change various configs (like SMTP settings, etc.)

I don't really have a problem with someone using nix-copy-closure in a situation like that, but I really would prefer if this were treated as more of an "emergency recovery" approach, not a recommended way of doing anything.

Definitely agree.