Feature request: new meta attr for source code repository

Problem
most packages use meta.homepage to link to the source code repository, but for projects with an actual homepage (such as luajit), finding the source code often requires clicking around various links to try to find the official repo.
for packages that use fetchFromGitHub or similar, you can usually piece the url back together with some effort, but for projects that use a source tarball (again, like luajit), you're out of luck.
Proposal
a meta.repository field that points to an http-browsable source tree.
for packages without a separate homepage, you could just set the meta.repository field instead.
Checklist

[ ] checked latest Nixpkgs manual (source) and latest NixOS manual (source)
[ ] checked open documentation issues for possible duplicates
[ ] checked open documentation pull requests for possible solutions

Add a :+1: reaction to issues you find important.

There is meta.downloadPage which is documented https://github.com/NixOS/nixpkgs/blob/5bfab70cdf63ad75f1d7d0facc59fb9f49668811/doc/stdenv/meta.chapter.md#downloadpage-var-meta-downloadpage

a download page and browsable source tree are entirely different things. a package that has a dedicated homepage is also very likely to have a dedicated page for downloading binaries.

Problem
most packages use meta.homepage to link to the source code repository, but for projects with an actual homepage (such as luajit), finding the source code often requires clicking around various links to try to find the official repo.

So, what?
This is a decision from the upstream maintainers to not provide an easy link to their CVS repos.
Ask them for an easier-to-find link.

This is a decision from the upstream maintainers to not provide an easy link to their CVS repos.
Ask them for an easier-to-find link.

same logic can be applied to say that meta.downloadPage and meta.changelog shouldn't exist.
project homepages are usually designed for users, not developers, in my experience they rarely put the git repo front and center.

Indeed there are many times we find the opposite: the software has only the repository as its homepage.
E.g. I keep bqn-mode in github and never set a dedicated homepage for it.

if you mean packages that don't have a dedicated homepage, and use the git forge instead: yes, i am well aware of that.  i mention it in the original post.
if you mean packages that have a dedicated homepage, and use this git forge despite this: as far as i can tell this is incorrect metadata.  the homepage should point to the homepage, doing otherwise is just confusing.

same logic can be applied to say that meta.downloadPage and meta.changelog shouldn't exist.

meta.downloadPage is useful when the download page is wildly different from the homepage.
That's useful for automated scripts.
meta.changelog has useful info for package maintainers.
A pointer for an eye-candy webpage showing the files unpacked in the CVS? This is just eyecandy.
And the CVS itself is useful for those who want to contribute or hack with the project themselves.
For a package maintainer, this is just downloadPage, albeit it is not an adequate name.

project homepages are usually designed for users, not developers

Yep.
Convince them we programmers are more important than those mere users...

meta.repositories used to exist but was removed in https://github.com/nixos/nixpkgs/commit/33cce15e42e4086ea26b5fc026a2de3ca2e07f29

I've mentioned the same problem here: https://discourse.nixos.org/t/problems-regarding-meta-homepage-and-link-to-repository/39821. So thanks for raising an issue.

meta.downloadPage is useful when the download page is wildly different from the homepage. That's useful for automated scripts.

The real download link is always obvious: it's in the src  field. This excludes a small number of packages with a download link hidden behind the paywall. But in this case, downloadPage is still useless for automated scripts or package maintainers. In fact, not a large proportion of software packages really use downloadPage.
We often do this instead:
nix-repl> :l <nixpkgs>

nix-repl> pkgs.mailspring.src.url
"https://github.com/Foundry376/Mailspring/releases/download/1.13.3/mailspring-1.13.3-amd64.deb"

nix-repl> pkgs.go-musicfox.src.url
"https://github.com/go-musicfox/go-musicfox/archive/v4.3.1.tar.gz"

nix-repl> pkgs.transmission.src.url
"https://github.com/transmission/transmission.git"

I believe the automated scripts do the same, because in this way they even don't have to parse the source code themselves. For package maintainers, they can read the code and figuring out how to bump source versions shouldn't be a problem.

A pointer for an eye-candy webpage showing the files unpacked in the CVS? This is just eyecandy.

And the CVS itself is useful for those who want to contribute or hack with the project themselves.
For a package maintainer, this is just downloadPage, albeit it is not an adequate name.

In fact, link to a repository is often much more useful than the downloadPage. We can roughly observe whether this project is in a good development state (I will not define this term here, it is related to the user's personal needs). On third-party code hosting platforms, we can often find the issue area in a fixed location (rather than a random place), so that we can have a general understanding of the quality of the program, submit issues, or find workarounds to known problems.
This also applies to package maintainers (including the maintainer who created this package and some random people who wants to bump the package), since some dependencies are often hidden in the cmake or meson build files, some files may need to be installed manually, and some code needs to be patched to adapt to NixOS. However, unlike the downloadPage, sometimes we are not using the src related to the repository (such as mirror, upstream deb package, or simply download link not including the repo)

meta.changelog has useful info for package maintainers.

For maintainers, code modifications between version tags are far more important than the changelog, because they really directly reflect changes in dependencies and other metadata.

Convince them we programmers are more important than those mere users...

Users and developers are often not separate in the Linux community, especially in a distribution that requires writing code in an unfamiliar language to configure the system. It makes no sense to discuss the needs of the two in isolation.

Indeed there are many times we find the opposite: the software has only the repository as its homepage.

Yes, this begs another question: What exactly is a homepage? In my opinion, a repository is a repository, while a special introduction page is a homepage. Although I did see many repositories with very detailed READMEs, and they may indeed be used as homepages, the repository itself still brings too much distraction. I would prefer that in tools like search.nixos.org, when users click on a link named "homepage" or "repository", they have clear expectations of where they will be directed. This is also a UX issue, but there may be my personal preference, and other people may have different views.

On third-party code hosting platforms, we can often find the issue area in a fixed location (rather than a random place), so that we can have a general understanding of the quality of the program, submit issues, or find workarounds to known problems.

Again, eye-candy for people interested in hacking/contributing to the code.

meta.changelog has useful info for package maintainers.

For maintainers, code modifications between version tags are far more important than the changelog, because they really directly reflect changes in dependencies and other metadata.

Are you seriously saying that a whole diff -Naur between two releases of Linux kernel, with nothing else, is way more important than the changelog the kernel developers wrote?
Worse, are you suggesting programmers (including but not limited to Linux kernel programmers) are so unorganized that the changelogs they wrote have zero relevance or negative reliability when compared to a machine-generated diff -Naur?

Users and developers are often not separate in the Linux community

Often not is not the same as never.
Usually programmers are smarter than users, and smart enough to figure themselves by being very keen to look for some keywords such as “star”, “contribute”, “submit an issue”, or “development”, or find the GitHub icon in the sidebar.

This is especially true about

It does not change the argument: the upstream webmasters should provide an easier-to-find link for their VCSes to begin with, not the package managers.

On third-party code hosting platforms, we can often find the issue area in a fixed location (rather than a random place), so that we can have a general understanding of the quality of the program, submit issues, or find workarounds to known problems.

Again, eye-candy for people interested in hacking/contributing to the code.

I don't see what I'm saying having anything to do with hacking/contributing to the code; on the contrary, it's what you would call "normal users" doing on a daily basis.

For maintainers, code modifications between version tags are far more important than the changelog, because they really directly reflect changes in dependencies and other metadata.

Are you seriously saying that a whole diff -Naur between two releases of Linux kernel, with nothing else, is way more important than the changelog the kernel developers wrote?

No; for some software, the author may have a very well-written changelog, including changes in the construction method; but for other projects, the diff of the build script and the addition and subtraction of some special data files have become The maintainer provides sufficient information. Changes to specific functional code often do not result in package changes such as adding or removing dependencies or modifying the build process, unless a specific error is thrown in the build process.

Worse, are you suggesting programmers (including but not limited to Linux kernel programmers) are so unorganized that the changelogs they wrote have zero relevance or negative reliability when compared to a machine-generated diff -Naur?

To some extent, yes. I've seen nearly all maintainers who write in great detail about functional changes for their users, but there aren't that many maintainers who inform the build process or dependency changes for the package maintainers.

Users and developers are often not separate in the Linux community

Often not is not the same as never.

I'm not trying to please everyone. There are certainly an amount of users who will not benefit from it. But this doesn't render it useless.

Usually programmers are smarter than users, and smart enough to figure themselves by being very keen to look for some keywords such as “star”, “contribute”, “submit an issue”, or “development”, or find the GitHub icon in the sidebar.

But that doesn't change the unpleasant process. If we can save them time and energy, then why not?

It does not change the argument: the upstream webmasters should provide an easier-to-find link for their VCSes to begin with, not the package managers.

Yes, logically they should. But what actually happens? We don't always create tools to solve a modeled problem, we also need to solve these problems in reality, especially if we happen to have the ability to do so (I mean, as shown in the PR above, some common fetchers already contain this information).

Let me be the devil's advocate here.
What is your goal here, by providing meta.repository? It is just an eye-candy view of the VCS? Or is the repository itself?
Will you prefer a webpage ready to be opened in a web browser, like this GitWeb interface for GCC or this mirror of GNU Emacs?
Or the URL for the repo ready to be consumed by Git, like this from GCC - git://gcc.gnu.org/git/gcc.git - or this from Emacs - git.savannah.gnu.org/git/emacs.git?
meta.repository only makes sense if it can be used as input for a git command (or the corresponding for other VCSes).

@Artturin thanks, i did not know that.  i would hope that automatically setting the field from src metadata would make it more useful.  additionally, having a single value should make it easier for sites like search.nixos.org to use and present it.

A pointer for an eye-candy webpage showing the files unpacked in the CVS? This is just eyecandy.

issue trackers exist outside of the source tree.  they are not viewable via a git clone, but nearly every git forge links them from the repo page.
cloning the entirety of a large project when all you want is a quick glance at a single file is a waste of time, bandwidth, and disk space.

Convince them we programmers are more important than those mere users...

making drastic changes to the culture of every open source project is a lot harder than adding a few lines of code.

Yes, this begs another question: What exactly is a homepage? In my opinion, a repository is a repository, while a special introduction page is a homepage. Although I did see many repositories with very detailed READMEs, and they may indeed be used as homepages, the repository itself still brings too much distraction. I would prefer that in tools like search.nixos.org, when users click on a link named "homepage" or "repository", they have clear expectations of where they will be directed. This is also a UX issue, but there may be my personal preference, and other people may have different views.

i already brought up a possible solution to this in issue i submitted to nixos search: if meta.homepage and meta.repository are set to the same value, only show the repository link.

A pointer for an eye-candy webpage showing the files unpacked in the CVS? This is just eyecandy.

issue trackers exist outside of the source tree.  they are not viewable via a git clone, but nearly every git forge links them from the repo page.

Therefore you do not want the source code repository, you want a very specific eyecandy. You are naming it in a misleading way - and arguing for it in a misleading way too.
The "issue tracker" of Linux kernel is a freaking mailing list. Indeed many projects still use them - hell, SourceHut was created a decade ago and they use mailing lists!
Is a mailing list an acceptable value for the meta.eyecandySite you call "repository"? (Don't worry, many of them have HTML-rendered, HTTP-reachable backups...)

cloning the entirety of a large project when all you want is a quick glance at a single file is a waste of time, bandwidth, and disk space.

fetchgit does this and no one complained.

Convince them we programmers are more important than those mere users...

making drastic changes to the culture of every open source project is a lot harder than adding a few lines of code.

Therefore you do not want the source code repository, you want a very specific eyecandy. You are naming it in a misleading way - and arguing for it in a misleading way too.

issue trackers exist outside of the source tree, therefore i don't want the source code repository?
i know that's being overly literal, but i don't know what you're actually trying to say.
the issue tracker being available via the repository webpage is not a an official feature, it is simply a nice situational benefit for users.

The "issue tracker" of Linux kernel is a freaking mailing list. Indeed many projects still use them - hell, SourceHut was created a decade ago and they use mailing lists!

ok, so the repository page of those projects simply won't link to the issue tracker.  i don't think that's a huge deal.

Is a mailing list an acceptable value for the meta.eyecandySite you call "repository"? (Don't worry, many of them have HTML-rendered, HTTP-reachable backups...)

no, only an http-browsable source tree is an acceptable value, as the documentation states

fetchgit does this and no one complained.

fetchgit puts its files in a garbage-collected, deduplicated, cached file store.
people do complain about nix build speeds being slow.  it's the main downside of nix.

Also, git allows capturing a single file since at least 1.8.

additional mental load of:

remembering/looking up the command to do that
reconstructing the git url from the fetcher, or entering a nix repl to evaluate src.gitRepoUrl
deciding which files to download
deciding where to put those files
remembering to remove those files when you're done

as opposed to just typing !nixpkgs PACKAGE_NAME and clicking a few links.

Like making all of them using GitHub exclusively?

what?
this proposal works equally well with sites like gitea, gitlab, and SourceHut.  it can even work with custom solutions like git.kernel.org.
95% of software projects have some form of http-browsable source tree, and those that don't can simply not set the field and be no worse off than if the field didn't exist.

project homepages are usually designed for users, not developers, in my experience they rarely put the git repo front and center.

Then maybe instead of ‘http-browsable source tree’ being the defining characteristic of this new field, we could say that it's for homepages targeting contributors, and meta.homepage is for homepages targeting users? Then there's less confusion over when to use this (for humans who want to interact with contributor-centric resources for the package) versus when to use src (for automated processes to get the current source) or src.gitRepoUrl (for getting an entire Git repository) or meta.downloadPage (for manually downloading the current source or binaries).

Then maybe instead of ‘http-browsable source tree’ being the defining characteristic of this new field, we could say that it's for homepages targeting contributors, and meta.homepage is for homepages targeting users?

personally i think "http-browsable source tree" is much more descriptive than "homepage targeting contributors", as the latter requires making a subjective judgement on what the intended target audience of the page is (and most pages usually target both)
maybe "a webpage where the package's source code can be viewed" would be easier to understand.

(and most pages usually target both)

I don't know, if your archetypal example is get-project.com for meta.homepage and github.com/project-team/project for meta.repository, I think it's pretty clear that one page is user-facing and one page is contributor-facing.
And if a project only exposes a Bugzilla instance but not an HTTP-browsable VCS, I'd still consider that to be a contributor-facing home.

I don't know, if your archetypal example is get-project.com for meta.homepage and github.com/project-team/project for meta.repository, I think it's pretty clear that one page is user-facing and one page is contributor-facing.

And if a project only exposes a Bugzilla instance but not an HTTP-browsable VCS, I'd still consider that to be a contributor-facing home

i think in that case (if it was frequent enough), we would want a separate issueTracker or bugReport field, instead of overloading the meaning of meta.repository.
my goal with this is to make metadata easier to understand, as currently homepage is somewhat overloaded, and often has an unexpected value (eg. the homepage for gforth is set to its git repo, instead of gforth.org)
when you click the url labeled "homepage", you should go to the project's homepage, and when you click the url labeled "repository" you should go to the project's repository.

Then I guess I don't know why you want this field either, given that src.gitRepoUrl will take you to the project's repository.

src.gitRepoUrl is specific to git, undocumented, only set by certain fetchers, and not designed to be overridden by package maintainers.

Therefore you do not want the source code repository, you want a very specific eyecandy. You are naming it in a misleading way - and arguing for it in a misleading way too.

issue trackers exist outside of the source tree, therefore i don't want the source code repository? i know that's being overly literal, but i don't know what you're actually trying to say.

Your intention is to allow only eye-candy forges or forge-like sites, and call them a generic and misleading name "repository"

ok, so the repository page of those projects simply won't link to the issue tracker. i don't think that's a huge deal.

Repositories that are not meta.repositories.
I have not seen such an ironic stance since the C keyword const not meaning constant.

no, only an http-browsable source tree is an acceptable value, as the documentation states

Why such a discrimination (against ugly non-eyecanded repos), I ask again?

1. fetchgit puts its files in a garbage-collected, deduplicated, cached file store.

Was this an argument for or against using fetchgit?

2. people do complain about nix build speeds being slow.  it's the main downside of nix.

In the worst of cases, you can grab the raw git command.

additional mental load of:
1. remembering/looking up the command to do that

2. reconstructing the git url from the fetcher, or entering a nix repl to evaluate `src.gitRepoUrl`

3. deciding which files to download

4. deciding where to put those files

5. remembering to remove those files when you're done

There is a thing called shell script. It can automate a tuckload of boring and forgettable tasks.

Like making all of them using GitHub exclusively?

what?
this proposal works equally well with sites like gitea, gitlab, and SourceHut. it can even work with custom solutions like git.kernel.org.

You are not targetting general-purpose repositories, but a very specific style of site.
In other words, you are targetting forges.

95% of software projects have some form of http-browsable source tree, and those that don't can simply not set the field and be no worse off than if the field didn't exist.

Then your arguments about being hard to find those source trees are weaker than mine about multi-valued backup mirrors.

"But only if its project's repository is an HTTP-browsable eyecandied site" - FTFY.

Your intention is to allow only eye-candy forges or forge-like sites, and call them a generic and misleading name "repository"

don't tell me what my intention is.
git.kernel.org would certainly qualify for this value, and that doesn't have any of the features you would associate with a typical forge

You are not targetting general-purpose repositories, but a very specific style of site.
In other words, you are targetting forges.

once again: stop telling me what i am trying to do.

"But only if its project's repository is an HTTP-browsable eyecandied site" - FTFY.

i don't care about whether a webpage looks good or not.  you simply decided that i did and keep asserting that i do every two sentences.
it's 2024, and the world runs on http.  i'm not really happy about it, but i'm not going to decrease the utility of my metadata fields out of spite.

Your intention is to allow only eye-candy forges or forge-like sites, and call them a generic and misleading name "repository"

don't tell me what my intention is.

I don't need - you did it already:
issue trackers exist outside of the source tree. they are not viewable via a git clone, but nearly every git forge links them from the repo page.
Further, the whole "search engine runs on HTTP, then it can't include non-HTTP links" argument plus the examples about multiple purposes of forges meta.repositories...

git.kernel.org would certainly qualify for this value, and that doesn't have any of the features you would associate with a typical forge

Then nothing hinders a Gemini link.

once again: stop telling me what i am trying to do.

Are you targeting general-purpose repositories?

it's 2024, and the world runs on http.  i'm not really happy about it, but i'm not going to decrease the utility of my metadata fields out of spite.

Allowing Gemini or IPFS links decrease the utility of your metadata fields? How?

Your intention is to allow only eye-candy forges or forge-like sites, and call them a generic and misleading name "repository"

don't tell me what my intention is.

I don't need - you did it already:
issue trackers exist outside of the source tree. they are not viewable via a git clone, but nearly every git forge links them from the repo page.
Further, the whole "search engine runs on HTTP, then it can't include non-HTTP links" argument plus the examples about multiple purposes of forges meta.repositories...

git.kernel.org would certainly qualify for this value, and that doesn't have any of the features you would associate with a typical forge

Then nothing hinders a Gemini link.

once again: stop telling me what i am trying to do.

Are you targeting general-purpose repositories?

it's 2024, and the world runs on http.  i'm not really happy about it, but i'm not going to decrease the utility of my metadata fields out of spite.

Allowing Gemini or IPFS links decrease the utility of your metadata fields? How?

@AndersonTorres if you are not capable of phrasing your criticism of the proposal in a constructive manner, please consider leaving the discussion. Especially, calling a feature you don't like "eye-candy" in a derogatory way is not okay. Same for the PR thread.
Within the use cases for the feature have been detailed, I have never encountered any repository whose primary forge web site was not reachable by HTTP. Focusing on this is whataboutism and derailing the conversation. It increasingly feels to me like you are filibustering a change you don't like, instead of constructively working towards a solution that fits the needs of those who ask for it while also taking your concerns into account.

Especially, calling a feature you don't like "eye-candy" in a derogatory way is not okay.

I am not arguing against eye-candy all-in-one forges. I am arguing against refusing other, less typical forges - that are being refused with no reason besides "less than any%" or "they doesn't look cool on that search engine".
And such reasons in a codebase with things like fetchcvs and fetchpijul are less than convincing.

I have never encountered any repository whose primary forge web site was not reachable by HTTP

If that "less than any%" argument is not a reasonable motivation for removing fetchers like fetchpijul and fetchsvn from Nixpkgs codebase, then this is not a reasonable motivation to not allow non-HTTP-reachable forges as possible values for meta.repository.

instead of constructively working towards a solution that fits the needs of those who ask for it

A solution that allows non-HTTP-reachable links along with HTTP-reachable ones certainly fits the needs of those who ask for HTTP-reachable links.
Why not allowing anything besides HTTP-reachable links? This arbitrary restriction benefits no one.
What reasons were given for such discrimination?

while also taking your concerns into account.

Since you bring this about, summarizing my concerns:

Not banishing otherwise valid values with questionable reasons like "it does not look good enough on this particular search engine" or "I have not ever seen this before"
Making that meta attribute future-proof
Automatic filling fields based on questionable heuristics

Questionable heuristics that introduced bugs in otherwise functioning code already in the Nixpkgs codebase, yay!

Then nothing hinders a Gemini link.

honestly, fine.  allow gemini, or ipfs, or whatever.  it shouldn't be too difficult for users of this field to proxy that over http, i guess.
can we at least say "https links are preferred when available"?  but it has to be a browsable tree, no linking to tarballs, that can go in downloadPage if necessary.

How about "web links"?

Not banishing otherwise valid values with questionable reasons like "it does not look good enough on this particular search engine"

i never said this.  i just wanted it to be easily viewable in a browser.  popular web browsers do not support "gemini" links at the time of writing

Making that meta attribute future-proof

going back later and saying "actually this can be ipfs" is a trivial change only affecting documentaion.
the bug i'm worried about is code assuming it will always be http while the documentation does not specify that.

Based on questionable heuristics like inspecting opaque fields

src.meta.homepage is not opaque.
any derivation can have a meta.homepage attribute.  src is a derivation.
it is, however, somewhat unintuitive.  do you want me to document it better?

Questionable heuristics that introduced bugs in otherwise functioning code already in the Nixpkgs codebase, yay!

that is a misrepresentation of what happened.  the only thing that happened is i evaluated code that does not evaluate.
you should not pass stdenv.mkDerivation a src attribute that does not evaluate.
additionally, the actual package was not broken, the only thing that was broken was a transient value only reachable by ofborg-eval recursing for derivations.

can we at least say "https links are preferred when available"? but it has to be a browsable tree, no linking to tarballs, that can go in downloadPage if necessary.

Yes.

the bug i'm worried about is code assuming it will always be http while the documentation does not specify that.

For what it matters to Nix evaluator, this is just (a list of?) strings. The search engine user application decides what to do with them later.

any derivation can have a meta.homepage attribute. src is a derivation.

Hum... I believe this is not necessarily true.
It's not unusual to point to local files as src in Nix expressions.
It happens all the time with parameterized files (just rg substituteAll).
And meta makes no sense for a local file.
Further, being pedantic, derivations as defined by Nix language manual have only three required attrs: name, system and builder.
Nonetheless, it is better to not "try to be smart".
It is more reliable to let the package writers to populate the meta.repository field with such typical meta.repository = src.meta.homepage snippets instead of introducing complexity at the core of Nixpkgs and being hit by a clash of planets years later (like a transient value only reachable by ofborg-eval recursing for derivations).

any derivation can have a meta.homepage attribute. src is a derivation.

Hum... I believe this is not necessarily true. It's not unusual to point to local files as src in Nix expressions. It happens all the time with parameterized files (just rg substituteAll). And meta makes no sense for a local file. Further, being pedantic, derivations as defined by Nix language manual have only three required attrs: name, system and builder.

yes, but if you look at the code i wrote, you'll notice it checks whether src.meta.repository exists before adding it as a fallback.  the derivation could be missing src entirely and it would still work fine.
not everything will have a src.meta.homepage field, but my point is, if src.meta.homepage exists, it has a well-defined meaning.

Nonetheless, it is better to not "try to be smart".

citation needed
also, making a value default to another value is extremely common throughout all of programming.

It is more reliable to let the package writers to populate the meta.repository field with such typical meta.repository = src.meta.homepage snippets instead of introducing complexity at the core of Nixpkgs and being hit by a clash of planets years later (like a transient value only reachable by ofborg-eval recursing for derivations).

any change has the potential to cause issues down the line.  but you know what we do? we fix those issues.
i put a lot of work into those two lines of code and considered a lot of factors, down to stuff like making sure the error message you got when typing meta.repository on a package with the field unset actually points to the right piece of code.
also, the stability of metadata fields has never been an important priority in nixpkgs, which i can tell because i've found a large number of obviously invalid urls just sitting around.

citation needed

Example: that person hit by a transient value only reachable by ofborg-eval recursing for derivations.

also, making a value default to another value is extremely common throughout all of programming.

Explicit is better than implicit.
- Zen of Python

any change has the potential to cause issues down the line. but you know what we do? we fix those issues.

Avoiding issues by taking preventative measures is usually something we do too.
Fixing the issues before they appear - it's a kind of magic!

i put a lot of work into those two lines of code and considered a lot of factors, down to stuff like making sure the error message you got when typing meta.repository on a package with the field unset actually points to the right piece of code.

Hit by a transient value only reachable by ofborg-eval recursing for derivations.

also, the stability of metadata fields has never been an important priority in nixpkgs, which i can tell because i've found a large number of obviously invalid urls just sitting around.

Is this an argument for progressive degradation and uselessness of meta (including to pinging maintainers that didn't touch their forks for years) or for tool-assisted sprints fixing this issue?

So what is a "transient value only reachable by ofborg-eval recursing for derivations"?

src.gitRepoUrl is specific to git, undocumented, only set by certain fetchers, and not designed to be overridden by package maintainers.

Ehr, the same can be said about src.meta.homepage.
Indeed all fetchers are underdocumented, and the chapter about fetchers say nothing about meta attributes from src.

Indeed all fetchers are underdocumented, and the chapter about fetchers says nothing about meta attributes from src

well maybe they should

Example: that person hit by a transient value only reachable by ofborg-eval recursing for derivations.

you seem to think that error is much more severe than it is.
the only thing it caused was a CI failure.  it did not cause any package to actually stop working.

Is this an argument for progressive degradation and uselessness of meta (including to pinging maintainers that didn't touch their forks for years) or for tool-assisted sprints fixing this issue?

i'm sorry, how does adding more data to meta make it useless?
i would argue that having a meta field that isn't set by any package would be much more useless.
not having the field at all is the most useless of all.
or would you rather i changed 10 thousand lines instead of 10?  because that's what would be required if the field has no default.

including to pinging maintainers that didn't touch their forks for years

how is that relevant here?

or would you rather i changed 10 thousand lines instead of 10? because that's what would be required if the field has no default.

A third alternative is to place the defaulting logic in consumers instead of the producer.

@rhendric it's certainly an option, but i'm not sure it's a good one.
instead of having nixpkgs code rely on undocumented nixpkgs behavior (which is quite common, not every obscure utility function has all of it's features publicly documented), you have external code outside the nixpkgs repo depending on undocumented behavior.
by handling it within nixpkgs, if we ever want to change the behavior of src.meta.homepage, all we have to do is update check-meta.nix, instead of telling everyone who uses meta.repository to change how their code works.

Indeed all fetchers are underdocumented, and the chapter about fetchers says nothing about meta attributes from src

well maybe they should

Then underdocumentation is not exactly a problem for using this or that attribute.
On the other hand, this is not the crux of matter: src should have a meta.repositories field.

you seem to think that error is much more severe than it is.

You asked for an example of error, not for an example of catastrophe.
Nonetheless this error pointed out to something more promising.

Is this an argument for progressive degradation and uselessness of meta (including to pinging maintainers that didn't touch their forks for years) or for tool-assisted sprints fixing this issue?

i'm sorry, how does adding more data to meta make it useless?

Let's get back to the paragraph:

"the stability of metadata fields has never been an important priority in nixpkgs" - your words

This is an argument for what, exactly?
To me, it points to the need of improvement of a bad situation.
But the impression you transmitted was something like "don't worry, there is so much mess here that my code will not look so relevant".

or would you rather i changed 10 thousand lines instead of 10? because that's what would be required if the field has no default.

The default of empty list was suggested, wasn't it? Indeed it was precisely what would happen in cases when src has no meta.homepage - with the small problem that, as you yourself said, a homepage is not the same thing as a repository.
This is a bit ironic relying on a homepage that is a repository in order to set the value of a repository that is not always a homepage.
Further, if I am correct on my suppositions, it would require less than a hundred lines of diff. And this being conservative: there are 40 directories called pkgs/build-support/fetch*. If each one has only 3 files each, and each one requiring one line of code, it would be 120 changes.

how is that relevant here?

I was talking about (an example of) metadata maintenance.

On the other hand, this is not the crux of matter: src should have a meta.repositories field.

for what purpose?
you seem to have a problem with the existing behavior of fetchers, which i think is out of the scope of this issue.

The default of empty list was suggested, wasn't it?

none of the other meta attributes default to null or an empty list, so i would like to keep things consistent.

Indeed it was precisely what would happen in cases when src has no meta.homepage

no, it is not.  if no explicit value is given, and src.meta.homepage is not set, then meta.repository will not be set either.  trying to access it in this case will raise an error (this behavior is in line with other metadata fields).

Further, if I am correct on my suppositions, it would require less than a hundred lines of diff. And this being conservative: there are 40 directories called pkgs/build-support/fetch*. If each one has only 3 files each, and each one requiring one line of code, it would be 120 changes

hold on, i thought you wanted each package to explicitly set its own meta.repository?
in any case, 120 is a lot more than 15, an i like to keep my PRs small to minimize the possibility of conflicts, and to make them easy to review.

On the other hand, this is not the crux of matter: src should have a meta.repositories field.

for what purpose?

Two purposes you proposed:

sometimes src can have meta attributes, especially when generated by the fetchers
meta.repositories should exist as an entity, not as an overload of meta.downloadPage or meta.homepage

Therefore, src.meta.homepage should not be conflated with src.meta.repository.
Consistency, that's it.

you seem to have a problem with the existing behavior of fetchers, which i think is out of the scope of this issue.

Since you are seeking for consistency, this is not so out of scope after all.
Certainly it will require more coordination and many PRs, but this will not be so hard.
After all, meta does not trigger a rebuild.

none of the other meta attributes default to null or an empty list, so i would like to keep things consistent.

They are more or less well-defined and easy to obtain. Rarely a person has doubts about descriptions, homepages,  licenses and the like.
The same can't be said about meta.repositories.
E.g. live555 explicitly has no public VCS. They don't even release older tarballs.
For such a case, live555.meta.repositories = []; is not only perfectly acceptable, it's the only acceptable value.
Technically meta.maintainers can be empty, since we do not impose lifetime maintenance on packages.

no, it is not. if no explicit value is given, and src.meta.homepage is not set, then meta.repository will not be set either. trying to access it in this case will raise an error (this behavior is in line with other metadata fields).

What is the value of an unset field?
Remember, you also opened a feature request on the Nixpkgs Search Engine. And parsing a field that can raise an error is a pain in the butt, even when using a memory-paranoid language like Rust.

hold on, i thought you wanted each package to explicitly set its own meta.repository?

Initially this was the only acceptable stance, since the src.meta.homepage is a bad heuristic.
Nonetheless, for some (many) fetchers meta.repositories make perfect sense and it is easy to generate.
Picking the most ubiquitous fetchFromGitHub,  it is just ${domain}/${owner}/${repo}.
Further, it is reasonable to inherit repository from source - since meta.repositories == src.meta.repositories by "definition".
As I have said above, src should have a meta.repositories field.

in any case, 120 is a lot more than 15

If fetchFromGitHub can return a proper meta.repositories (and it can), this return can be used to auto-set 14k meta.repositories. And it will be just one line of code - ${domain}/${owner}/${repo}.
This is a huge save. A huge and reliable save.
I believe showing the code will be more convincing.

Remember, you also opened a feature request on the https://github.com/NixOS/nixos-search/issues/741. And parsing a field that can raise an error is a pain in the butt, even when using a memory-paranoid language like Rust.

but they already have infrastructure for handling that, as other meta attributes are frequently unset.
and remember, indexing an unset field is only an error if it there is no default value
if anything, being inconsistent would probably require more code paths.

Initially this was the only acceptable stance, since the src.meta.homepage is a bad heuristic.

why is it a bad herustic?

This is a huge save. A huge and reliable save.

so, using src.meta.homepage is bad, but if we add a new field that does the exact same thing, that's reliable?

and remember, indexing an unset field is only an error if it there is no default value

"Unset" is not the same as "set as empty by default".

why is it a bad herustic?

How do I start...

Such a polemic PR with potential to touch substantial and sensitive parts of Nix should be proposed via RFC.

According to you yourself and @Aleksanaa , fields like downloadPage and  homepage are less than suitable for the purposes of storing links for repositories. Why should we expect the same will not happen with an undocumented feature? Your own words argue against you here.

Being undocumented, such src.meta feature can be easily discarded, and way more when they don't trigger a rebuild.
Indeed, why not deleting such instances of src.meta right now?

We know from Artturin that meta fields can be deleted without much hassle and no one will care.
After all, no one cared to remove meta.repositories field in that long past (miss you, WOP), whereas a single guy ignited a holy war because Nixpkgs nuked a SLAPPed software.

Further, arguments for removing changelog and downloadPage were given here, in a conversation for adding a new, arguably more useful field.
On the other hand, adding a single meta.categories field required a whole year.
Using such a sliding quicksandy ground is asking for trouble.

so, using src.meta.homepage is bad, but if we add a new field that does the exact same thing, that's reliable?

Correct!
Because that new field does precisely what it is purposed to do: point to a repository.
Also, because it can be documented in an unambiguous way. No one will need to update meta.homepage in order to say something like "sometimes it point to the upstream home page, but in some corner, undocumented cases it points to a source code repository".

Since this got reverted in #300247 and there still seems to be interested in this feature coming to fruition, I'm reopening this issue for now.

I was mildly skeptical before that computing attributes in meta from other parts of the derivation is a good idea, and now I'm more so. Why are we doubling down on this design choice and telling contributors (in a comment they have to find by following the stack trace!) to add redundant data to their packages if this causes more problems? We could instead let this defaulting logic live in a non-central, non-critical location, namely the one place that OP wants to consume this, and let meta be a dumb source of raw data that can't possibly cause evaluation problems again.

@rhendric a lot of meta attributes are already computed from other attributes, for the record.
there's several problems with putting this logic outside nixpkgs:

no longer testable via CI
will need to be re-implemented by everyone that wants this data
makes it much more work if anyone wants to change the logic in the future
one of the main ways to access this data is via repl, so "just implement it at the other end" means telling every user to handle it themselves
putting the logic outside nixpkgs doesn't fix the eval errors, it just ignores them (and most consumers, like nixos-search, will have no way of handling them, since there is no way to catch an abort)

unfortunatly the only viable options i see are:

no defaulting logic (would require 1000s of PRs to make the field useful)
put the defaulting logic in nixpkgs (requires some effort to not break eval)

maybe it would be better to instruct package maintainers to simply make their src eval on all platforms (eg. by adding or { })

Aborts can be catched outside Nix.

with great difficulty, yes.  instead of just doing nix eval --json nixpkgs#somepackage.meta to extract every metadata field at once, you instead need a seperate nix eval command for ever metadata field, increasing the number of required procs tenfold.

treewide + shell scripts, like the by-name migration

link? i can't find any evidence of any sort of automatic by-name migration, and there are still a lot of packages using the category hierarchy.
that seems like it would encounter a ton of problems, from constant merge conflicts to unusual syntactic constructions.  this would almost certainly end up being more work than the runtime approach, which has already had most of the required work completed.
even if we did somehow manage that, requiring new packages to specify the field (meaning reviewers need to inform new contributors about it) would still be a lot of work in the future.

link?

 Did you born yesterday?
because RFC 140 https://github.com/NixOS/rfcs/pull/140 is happening right now.

i can't find any evidence of any sort of automatic by-name migration

well, I can: https://github.com/NixOS/nixpkgs/issues/258650
You're welcome.

, and there are still a lot of packages using the category hierarchy.

because it was issued a recommendation for not to convert packages using the older convention (except in special circumstances), since - surprise surprise - they will be mass-migrated.
inb4 "link?":
https://github.com/NixOS/nixpkgs/blob/master/pkgs%2Fby-name%2FREADME.md

@rhendric a lot of meta attributes are already computed from other attributes, for the record.

Yes, but:

That isn't an argument that it's a good idea and should be continued.

From my reading of check-meta.nix, I see:

name—only references scalar attributes
outputsToInstall—references a list attribute that only contains strings
position—kind of magic
unfree, broken, unsupported, insecure—does a lot of stuff but mostly appears to only reference other things from meta?
available—optionally traverses dependencies, can be disabled

repository would be the first to pull directly from a complex attribute not already part of meta, which is a threshold worth pausing at, because of precisely the sort of problem we've just witnessed.

1. no longer testable via CI

More like no longer needs to be tested via CI, because for most packages there'll be nothing to test.

2. will need to be re-implemented by everyone that wants this data

As soon as ‘everyone’ is more than one nice-to-have link in the search app, I'll entertain this objection.

3. makes it much more work if anyone wants to change the logic in the future

Fully disagree; I think changing logic in check-meta.nix is much more fraught than changing logic in the search app, because the former is at the root of everything for everyone.

4. one of the main ways to access this data is via repl, so "just implement it at the other end" means telling every user to handle it themselves

Telling every user who looks this information up in the REPL (and how many of those are there going to be) to check two attributes instead of one is not much of a cost.

5. putting the logic outside nixpkgs doesn't fix the eval errors, it just ignores them (and most consumers, like nixos-search, will have no way of handling them, since there is no way to catch an abort)

As already noted, addressing eval errors outside of Nix is easier than inside, and consumers can always filter the packages down to a set they're interested in. You're proposing changing what src attributes are required to evaluate under what conditions just so that you can put logic intended to populate a search result field in Nixpkgs instead of in the search app—this is clearly a case of the tail wagging the dog.

Did you born yesterday?

@AndersonTorres, this is rude. OP may be misguided but that isn't license to take this tone.

  `repository` would be the first to pull directly from a complex attribute not already part of `meta`, which is a threshold worth pausing at, because of precisely the sort of problem we've just witnessed.

@rhendric what is the problem we just witnessed? we made a mistake fixed it and have a better implementation because of it that is immune to the previous failure. requiring a perfect first implementation is not a good faith argument.

More like no longer needs to be tested via CI, because for most packages there'll be nothing to test.

most packages.  the entire point of testing is to catch edge cases.

As soon as ‘everyone’ is more than one nice-to-have link in the search app, I'll entertain this objection.

well guess what, i have immediate plans for other stuff i would like to do with it so... yeah it's more that nixos-search

As already noted, addressing eval errors outside of Nix is easier than inside, and consumers can always filter the packages down to a set they're interested in.

as already noted, doing so results in massive performance penalties.

You're proposing changing what src attributes are required to evaluate under what conditions just so that you can put logic intended to populate a search result field in Nixpkgs instead of in the search app—this is clearly a case of the tail wagging the dog.

this is simply misrepresenting my intent.  i already said there are other usecases.  i would really appreciate if people would stop telling me what i want.

But you don't have to test what you don't implement! I'm really confused by our disagreement here. You want to change src and then test that the change is good; I want to refrain from changing src and then there's nothing to do or test.

i thought you wanted to not implement it in nixpkgs.
guess what, if you move the code somewhere else, it still exists!

github sucks on mobile

i thought you wanted to not implement it in nixpkgs.
guess what, if you move the code somewhere else, it still exists!

This isn't the gotcha that you think it is, so the attempt at a dunk isn't exactly going to be convincing.
What you would be testing if this logic were in Nixpkgs is not the correctness of the logic itself. The logic is pretty dang simple—use meta.repository if it exists, and use src.meta.repository if it doesn't, and adjust for lists appropriately. That's a function that could be unit tested very easily, but it's not what we're discussing, I think. It seems to me that when you say you want the correctness tested in Nixpkgs, what you want to ensure is that every derivation in Nixpkgs has a src that can be accessed in this way, across all platforms and under any other weird conditions. That's hard to test and hard to guarantee going forward.
I'm proposing not making that change to the expectations around src. Implement the trivial function elsewhere. If you plan on using it in multiple places, stick it in a library. Maybe it even deserves to live in lib. So yes, that code still exists! And you can test it if you want! What won't be tested, because it won't change, is that src can be accessed in all the contexts that you might want meta. If consumers try to access src in those contexts, they can fail and recover from the failure in whatever way is appropriate for them, because that's not Nixpkgs's concern.

Did you born yesterday?

@AndersonTorres, this is rude. OP may be misguided but that isn't license to take this tone.

Not my intention, just an expression of surprise.

requiring a perfect first implementation is not a good faith argument

the first one broke - twice.
(oh my, being far from my production machine hurts)

That's hard to test

it's really not, it's basically just nesting two for-each loops.
in my experience it isn't difficult to make src eval either.

hard to guarantee going forward.

nixpkgs doesn't guarantee stability outside of stable releases
i've only ever seen one pattern that breaks this expectation: indexing src based on the value of the current system

That's hard to test

it's really not, it's basically just nesting two for-each loops.

That'll test that it works in the specific configuration that it's tested in. But a given src could depend on any configuration value, not just system, and you aren't going to test every possible permutation in CI.

hard to guarantee going forward.

nixpkgs doesn't guarantee stability outside of stable releases
i've only ever seen one pattern that breaks this expectation: indexing src based on the value of the current system

Yeah, so this really comes down to a risk appetite thing. If you're a move-fast-break-things person, you might see the risk as worth it. You're looking at the things you see and saying they don't look so bad. I'm looking at the spaces where there can be things I don't see and saying why risk it? I don't think there have been enough eyeballs on this for me to feel comfortable with work that has a good chance of slipping time bombs into Hydra, which take precious time and effort to resolve. Particularly when consumers who want this logic can shoulder the burden of evaluating it themselves. Why not start there and move things into Nixpkgs after you've proved the value of the work?

@a-n-n-a-l-e-e, I'm not requiring a perfect first implementation.
The issue is that the scope increased from ‘add a field to meta’ to ‘add a field to meta and also create a new guarantee for src to evaluate successfully in more contexts’, and that second bit is harder and more error-prone.

so we are not trying to guarantee that src evaluates correctly. the new implementation doesn't require src to evaluate meta attributes.

That'll test that it works in the specific configuration that it's tested in. But a given src could depend on any configuration value, not just system, and you aren't going to test every possible permutation in CI.

theoretically, but i don't think they're likely to in practice.
anyways, by that argument, some combination of config options could cause nixpkgs as a whole to stop evaluating, since we're not testing every permutation.

@a-n-n-a-l-e-e

so we are not trying to guarantee that src evaluates correctly. the new implementation doesn't require src to evaluate meta attributes.

The new implementation being #300313? Which contains this code?
      # NOTE: this will fail if src fails to eval, in that case either set meta.repository manually to prevent this default from being evaluated, or just make sure src doesn't fail to eval.
      repository =
        if attrs ? src.meta.homepage
        then attrs.src.meta.homepage
        else if attrs ? srcs && isList attrs.srcs
        then unlist (map (src: src.meta.homepage) (filter (src: src ? meta.homepage) attrs.srcs))
        else [];

That's the code that doesn't require src? The code that fails ‘if src fails to eval’? Can you clarify?
@lolbinarycat

anyways, by that argument, some combination of config options could cause nixpkgs as a whole to stop evaluating, since we're not testing every permutation.

Yes, some combination of config options could do that. I don't know what point you're trying to make with that statement.

@a-n-n-a-l-e-e

so we are not trying to guarantee that src evaluates correctly. the new implementation doesn't require src to evaluate meta attributes.

The new implementation being #300313? Which contains this code?
      # NOTE: this will fail if src fails to eval, in that case either set meta.repository manually to prevent this default from being evaluated, or just make sure src doesn't fail to eval.
      repository =
        if attrs ? src.meta.homepage
        then attrs.src.meta.homepage
        else if attrs ? srcs && isList attrs.srcs
        then unlist (map (src: src.meta.homepage) (filter (src: src ? meta.homepage) attrs.srcs))
        else [];

That's the code that doesn't require src? The code that fails ‘if src fails to eval’? Can you clarify?

right. src not evaluating will not prevent access to other fields in meta, which was the issue before. trying to evaluate repository will still be problematic but i'm not sure it's worth adding additional checks to repository to prevent errors related to src. it's possible, checking that sourceProvence isSource, but perhaps not necessary. my point is that adding the repository field will not break current usage.

it's possible, checking that sourceProvence isSource

this is another assumption that can be broken easily.

sourceProvenance == isSource does not imply it "evaluates" in all platforms. What about  assembly?

sourceProvenance == binaryNativeCode does not imply it does not "evaluate" in all platforms. What about object files for Windows that are used in forensics and pentesting?
Indeed we can argue font glyphs are exactly like that.

(currently I am more inclined to an "outside nix" approach, in case someone asks me.)

What about assembly?

unless you are using import-from-derivation (which is banned in nixpkgs), assembly will never be interpreted at eval time, only at build time.

Indeed we can argue font glyphs are exactly like that.

font glyphs are binary data, not binary code.  it could be argued that truetype fonts contain vm bytecode, but they certainly do not contain native code.

so i wanted to see how many aborts are generated from trying pkg.meta.repository over all packages with the hydra systems x64/aarch64 linux & darwin. so hacked the script below which runs the check meta eval. Running the script i found 19 files that needed to be modified to throw. (probably better way to do this but here is a hacked 1st attempt)
script.sh
#!/usr/bin/env bash
# https://github.com/NixOS/ofborg?tab=readme-ov-file#running-meta-checks-locally
curl -OL https://raw.githubusercontent.com/NixOS/ofborg/released/ofborg/src/outpaths.nix
GC_INITIAL_HEAP_SIZE=4g nix-env -f ./outpaths.nix -qaP --no-name --out-path --arg checkMeta true > out-paths

# transform out-paths to:
# pkgs: with pkgs; [
#   pkg-name.meta.repository or null
#   ...
# ]
# need to remove packages with names that include '+' and .if. and pkgs-lib-tests
cat out-paths  | \
    gawk '{gsub("\\.[^.]*$", ".meta.repository or null", $1); print $1}' | \
    sort -u | \
    sed -e '/haskellPackages.if/d' \
        -e '/pkgs-lib-tests/d' \
        -e '/+/d' | \
    awk 'BEGIN{ print "pkgs: with pkgs; ["}
         { print $0 }
         END{ print "]"}' > out-paths.nix

for sys in x86_64-darwin x86_64-linux aarch64-darwin aarch64-linux; do
    nix eval --impure --expr 'with import ./. { system = '"\"$sys\""'; }; map (t: builtins.tryEval t) ((import ./out-paths.nix) pkgs)'
done

nixpkgs.diff add repository to meta and add throw to 19 files
diff --git a/pkgs/applications/audio/renoise/default.nix b/pkgs/applications/audio/renoise/default.nix
index d3462ecc6ad5..a56d43876fb0 100644
--- a/pkgs/applications/audio/renoise/default.nix
+++ b/pkgs/applications/audio/renoise/default.nix
@@ -39,7 +39,7 @@ in stdenv.mkDerivation rec {
     releasePath
   else
     let
-      platform = platforms.${stdenv.system};
+      platform = platforms.${stdenv.system} or (throw "unsupported system ${stdenv.hostPlatform.system}");
       urlVersion = lib.replaceStrings [ "." ] [ "_" ] version;
     in fetchurl {
       url =
diff --git a/pkgs/applications/gis/udig/default.nix b/pkgs/applications/gis/udig/default.nix
index 8510918d4cab..264445336cd3 100644
--- a/pkgs/applications/gis/udig/default.nix
+++ b/pkgs/applications/gis/udig/default.nix
@@ -13,7 +13,7 @@ let
       hash = "sha256-Ihk3InHB3/tEYRqH2ozhokz2GN8Gfig5DJkO/8P1LJs=";
     };
   };
-  src = srcs.${stdenv.hostPlatform.system};
+  src = srcs.${stdenv.hostPlatform.system} or (throw "unsupported system ${stdenv.hostPlatform.system}");

   meta = with lib; {
     description = "User-friendly Desktop Internet GIS";
diff --git a/pkgs/applications/graphics/sane/backends/brscan4/default.nix b/pkgs/applications/graphics/sane/backends/brscan4/default.nix
index ec8efcc27c2a..c15b3033a265 100644
--- a/pkgs/applications/graphics/sane/backends/brscan4/default.nix
+++ b/pkgs/applications/graphics/sane/backends/brscan4/default.nix
@@ -21,7 +21,7 @@ stdenv.mkDerivation rec {
       url = "https://download.brother.com/welcome/dlf006645/${pname}-${version}.amd64.deb";
       sha256 = "sha256-Gpr5456MCNpyam3g2qPo7S3aEZFMaUGR8bu7YmRY8xk=";
     };
-  }."${stdenv.hostPlatform.system}";
+  }."${stdenv.hostPlatform.system}" or (throw "unsupported system ${stdenv.hostPlatform.system}");

   unpackPhase = ''
     ar x $src
diff --git a/pkgs/applications/misc/1password-gui/default.nix b/pkgs/applications/misc/1password-gui/default.nix
index 32825d3ba523..5a7e4914f8cd 100644
--- a/pkgs/applications/misc/1password-gui/default.nix
+++ b/pkgs/applications/misc/1password-gui/default.nix
@@ -51,7 +51,7 @@ let
   };

   src = fetchurl {
-    inherit (sources.${channel}.${stdenv.system}) url hash;
+    inherit (sources.${channel}.${stdenv.system} or (throw "unsupported system ${stdenv.hostPlatform.system}")) url hash;
   };

   meta = with lib; {
diff --git a/pkgs/applications/networking/cluster/hadoop/default.nix b/pkgs/applications/networking/cluster/hadoop/default.nix
index d5bae9ad885b..5f44e39d008c 100644
--- a/pkgs/applications/networking/cluster/hadoop/default.nix
+++ b/pkgs/applications/networking/cluster/hadoop/default.nix
@@ -35,7 +35,7 @@ let
       src = fetchurl {
         url = "mirror://apache/hadoop/common/hadoop-${finalAttrs.version}/hadoop-${finalAttrs.version}"
               + optionalString stdenv.isAarch64 "-aarch64" + ".tar.gz";
-        inherit (platformAttrs.${stdenv.system}) hash;
+        inherit (platformAttrs.${stdenv.system} or (throw "Unsupported system: ${stdenv.system}")) hash;
       };
       doCheck = true;

diff --git a/pkgs/applications/networking/instant-messengers/discord/default.nix b/pkgs/applications/networking/instant-messengers/discord/default.nix
index 40e9c77e57a7..423f6cb2f1db 100644
--- a/pkgs/applications/networking/instant-messengers/discord/default.nix
+++ b/pkgs/applications/networking/instant-messengers/discord/default.nix
@@ -52,7 +52,7 @@ let
     };
     aarch64-darwin = x86_64-darwin;
   };
-  src = srcs.${stdenv.hostPlatform.system}.${branch};
+  src = srcs.${stdenv.hostPlatform.system}.${branch} or (throw "${stdenv.hostPlatform.system} not supported on ${branch}");

   meta = with lib; {
     description = "All-in-one cross-platform voice and text chat for gamers";
diff --git a/pkgs/applications/science/electronics/picoscope/default.nix b/pkgs/applications/science/electronics/picoscope/default.nix
index c7242117b68c..3aa5cb22aa52 100644
--- a/pkgs/applications/science/electronics/picoscope/default.nix
+++ b/pkgs/applications/science/electronics/picoscope/default.nix
@@ -49,7 +49,7 @@ let
   # If we don't have a platform available, put a dummy version here, so at
   # least evaluation succeeds.
   sources =
-    (lib.importJSON ./sources.json).${stdenv.system} or { picoscope.version = "unknown"; };
+    (lib.importJSON ./sources.json).${stdenv.system} or (throw "unsupported system ${stdenv.system}");

   scopePkg = name:
     { url, version, sha256 }:
diff --git a/pkgs/applications/version-management/radicle-upstream/default.nix b/pkgs/applications/version-management/radicle-upstream/default.nix
index 97fec6257dd1..69c960ed78e8 100644
--- a/pkgs/applications/version-management/radicle-upstream/default.nix
+++ b/pkgs/applications/version-management/radicle-upstream/default.nix
@@ -15,7 +15,7 @@ let
       sha256 = "sha256-EuWGbn6qggi8/9Rci8iaXfuVKE+QXb1BHEYDvotR/q4=";
     };
   };
-  src = srcs.${stdenv.hostPlatform.system};
+  src = srcs.${stdenv.hostPlatform.system} or (throw "unsupported system ${stdenv.hostPlatform.system}");

   contents = appimageTools.extract { inherit name src; };

diff --git a/pkgs/applications/virtualization/firecracker/default.nix b/pkgs/applications/virtualization/firecracker/default.nix
index d9bab2169623..2c3ec6c9a75b 100644
--- a/pkgs/applications/virtualization/firecracker/default.nix
+++ b/pkgs/applications/virtualization/firecracker/default.nix
@@ -13,7 +13,7 @@ let

   dlbin = sha256: fetchurl {
     url = "${baseurl}/v${version}/firecracker-v${version}-${suffix}.tgz";
-    sha256 = sha256."${stdenv.hostPlatform.system}";
+    sha256 = sha256."${stdenv.hostPlatform.system}"or (throw "unsupported system ${stdenv.hostPlatform.system}");
   };

 in
diff --git a/pkgs/development/compilers/oraclejdk/jdk-linux-base.nix b/pkgs/development/compilers/oraclejdk/jdk-linux-base.nix
index 40681faae8ae..d03419e96570 100644
--- a/pkgs/development/compilers/oraclejdk/jdk-linux-base.nix
+++ b/pkgs/development/compilers/oraclejdk/jdk-linux-base.nix
@@ -80,7 +80,7 @@ let result = stdenv.mkDerivation rec {
     in requireFile {
       name = "jdk-${productVersion}u${patchVersion}-${platformName}.tar.gz";
       url = "http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html";
-      sha256 = sha256.${stdenv.hostPlatform.system};
+      sha256 = sha256.${stdenv.hostPlatform.system} or (throw "unsupported system ${stdenv.hostPlatform.system}");
     };

   nativeBuildInputs = [ file makeWrapper ]
diff --git a/pkgs/development/compilers/temurin-bin/jdk-darwin-base.nix b/pkgs/development/compilers/temurin-bin/jdk-darwin-base.nix
index<PHONE_NUMBER>3d..8f6831a7907c 100644
--- a/pkgs/development/compilers/temurin-bin/jdk-darwin-base.nix
+++ b/pkgs/development/compilers/temurin-bin/jdk-darwin-base.nix
@@ -24,7 +24,7 @@ let
       sourcePerArch.${cpuName}.version or (throw "unsupported CPU ${cpuName}");

     src = fetchurl {
-      inherit (sourcePerArch.${cpuName}) url sha256;
+      inherit (sourcePerArch.${cpuName} or (throw "unsupported system ${stdenv.hostPlatform.system}")) url sha256;
     };

     # See: https://github.com/NixOS/patchelf/issues/10
diff --git a/pkgs/development/libraries/libcef/default.nix b/pkgs/development/libraries/libcef/default.nix
index d6d098110af8..ce83c3c22cd4 100644
--- a/pkgs/development/libraries/libcef/default.nix
+++ b/pkgs/development/libraries/libcef/default.nix
@@ -73,7 +73,7 @@ let
   platforms."aarch64-linux".sha256 = "16sbfk599h96wcsmpbxlwsvq0n1pssmm8dpwmjsqfrn1464dvs68";
   platforms."x86_64-linux".sha256 = "1wa4nv28saz96kar9svdarfz6c4rnbcqz0rqxzl9zclnhfzhqdiw";

-  platformInfo = builtins.getAttr stdenv.hostPlatform.system platforms;
+  platformInfo = platforms.${stdenv.hostPlatform.system} or (throw "unsupported system ${stdenv.hostPlatform.system}");
 in
 stdenv.mkDerivation rec {
   pname = "cef-binary";
diff --git a/pkgs/development/tools/azure-static-sites-client/default.nix b/pkgs/development/tools/azure-static-sites-client/default.nix
index 08e56de7335b..452976b7282a 100644
--- a/pkgs/development/tools/azure-static-sites-client/default.nix
+++ b/pkgs/development/tools/azure-static-sites-client/default.nix
@@ -21,7 +21,7 @@ let
   };
   sources = {
     "x86_64-linux" = fetchBinary "linux-x64";
-    "x86_64-darwin" = fetchBinary "macOS";
+    "x86_64-darwin" = fetchBinary "osx-x64";
   };
 in
 stdenv.mkDerivation {
diff --git a/pkgs/servers/meteor/default.nix b/pkgs/servers/meteor/default.nix
index 0ce2189568cb..7830d309b082 100644
--- a/pkgs/servers/meteor/default.nix
+++ b/pkgs/servers/meteor/default.nix
@@ -20,7 +20,7 @@ in
 stdenv.mkDerivation {
   inherit version;
   pname = "meteor";
-  src = srcs.${system};
+  src = srcs.${system} or (throw "unsupported system ${system}");

   #dontStrip = true;

diff --git a/pkgs/stdenv/generic/check-meta.nix b/pkgs/stdenv/generic/check-meta.nix
index 1cd1ae6dd72e..f1f486b115a8 100644
--- a/pkgs/stdenv/generic/check-meta.nix
+++ b/pkgs/stdenv/generic/check-meta.nix
@@ -10,7 +10,9 @@ let
     concatMapStrings
     concatMapStringsSep
     concatStrings
+    filter
     findFirst
+    head
     isDerivation
     length
     concatMap
@@ -39,6 +41,11 @@ let
     toPretty
   ;

+  unlist = list:
+  if length list == 1
+  then head list
+  else list;
+
   # If we're in hydra, we can dispense with the more verbose error
   # messages and make problems easier to spot.
   inHydra = config.inHydra or false;
@@ -303,6 +310,10 @@ let
       (listOf str)
       str
     ];
+    repository = union [
+      (listOf str)
+      str
+    ];
     downloadPage = str;
     changelog = union [
       (listOf str)
@@ -452,8 +463,14 @@ let
     let
       outputs = attrs.outputs or [ "out" ];
       hasOutput = out: builtins.elem out outputs;
-    in
-    {
+    in {
+      repository =
+        if attrs ? src.meta.homepage
+        then attrs.src.meta.homepage
+        else if attrs ? srcs && isList attrs.srcs
+        then unlist (map (src: src.meta.homepage) (filter (src: src ? meta.homepage) attrs.srcs))
+        else [];
+
       # `name` derivation attribute includes cross-compilation cruft,
       # is under assert, and is sanitized.
       # Let's have a clean always accessible version here.
diff --git a/pkgs/tools/archivers/rar/default.nix b/pkgs/tools/archivers/rar/default.nix
index 3799afd736ef..fabcfde887e0 100644
--- a/pkgs/tools/archivers/rar/default.nix
+++ b/pkgs/tools/archivers/rar/default.nix
@@ -37,7 +37,7 @@ stdenv.mkDerivation {
   pname = "rar";
   inherit version;

-  src = fetchurl (srcs.${stdenv.hostPlatform.system});
+  src = fetchurl (srcs.${stdenv.hostPlatform.system} or (throw "unsupported system ${stdenv.hostPlatform.system}"));

   dontBuild = true;

diff --git a/pkgs/tools/misc/geekbench/5.nix b/pkgs/tools/misc/geekbench/5.nix
index 7784b3632a73..49efe2ea0bf9 100644
--- a/pkgs/tools/misc/geekbench/5.nix
+++ b/pkgs/tools/misc/geekbench/5.nix
@@ -26,7 +26,7 @@ stdenv.mkDerivation {
   inherit version;
   pname = "geekbench";

-  src = fetchurl (sources.${stdenv.system});
+  src = fetchurl (sources.${stdenv.system} or (throw "unsupported system ${stdenv.hostPlatform.system}"));

   dontConfigure = true;
   dontBuild = true;
diff --git a/pkgs/tools/misc/geekbench/6.nix b/pkgs/tools/misc/geekbench/6.nix
index 4ac5f1d0e49a..373d93dc193c 100644
--- a/pkgs/tools/misc/geekbench/6.nix
+++ b/pkgs/tools/misc/geekbench/6.nix
@@ -27,7 +27,7 @@ stdenv.mkDerivation {
   inherit version;
   pname = "geekbench";

-  src = fetchurl (sources.${stdenv.system});
+  src = fetchurl (sources.${stdenv.system} or (throw "unsupported system ${stdenv.hostPlatform.system}"));

   dontConfigure = true;
   dontBuild = true;
diff --git a/pkgs/tools/misc/hakuneko/default.nix b/pkgs/tools/misc/hakuneko/default.nix
index 0ad8c5907a70..0359da603ccb 100644
--- a/pkgs/tools/misc/hakuneko/default.nix
+++ b/pkgs/tools/misc/hakuneko/default.nix
@@ -36,7 +36,7 @@ stdenv.mkDerivation rec {
       url = "https://github.com/manga-download/hakuneko/releases/download/v${version}/hakuneko-desktop_${version}_linux_i386.deb";
       sha256 = "32017d26bafffaaf0a83dd6954d3926557014af4022a972371169c56c0e3d98b";
     };
-  }."${stdenv.hostPlatform.system}";
+  }."${stdenv.hostPlatform.system}" or (throw "unsupported system ${stdenv.hostPlatform.system}");

   dontBuild = true;
   dontConfigure = true;
diff --git a/pkgs/tools/security/sonar-scanner-cli/default.nix b/pkgs/tools/security/sonar-scanner-cli/default.nix
index 553019299ba7..b97a2bc3a105 100644
--- a/pkgs/tools/security/sonar-scanner-cli/default.nix
+++ b/pkgs/tools/security/sonar-scanner-cli/default.nix
@@ -19,7 +19,7 @@ in stdenv.mkDerivation rec {
   inherit version;
   pname = "sonar-scanner-cli";

-  src = fetchurl sonarScannerArchPackage.${stdenv.hostPlatform.system};
+  src = fetchurl sonarScannerArchPackage.${stdenv.hostPlatform.system} or (throw "unsupported system ${stdenv.hostPlatform.system}");

   nativeBuildInputs = [ unzip ];

What about assembly?

unless you are using import-from-derivation (which is banned in nixpkgs), assembly will never be interpreted at eval time, only at build time.

Irrelevant. There is nothing hindering someone to write code like "throw when host is not x86" when facing a fromSource.

font glyphs are binary data, not binary code.

Data-code distinction lose most of its explanatory power since before the first worm, passing through undergrounds like Forth, Lisp and Phrack, and nowadays with the dreadful xzdoor.
Nonetheless my catastrophic example does not affect the argument: neither fromSource implies src will eval, nor non-fromSource implies src will break.

There is nothing hindering someone to write code like "throw when host is not x86"

a throw error would be fine, actually.  it would be caught by a tryEval in release-attrpaths-superset
calling abort would not be fine, and would probably cause CI to fail.
the only exceptions i know are calling abort within src on an unsupported platform, or in a transient derivation which is not exposed to hydra.
so... don't do that?

right. src not evaluating will not prevent access to other fields in meta, which was the issue before. trying to evaluate repository will still be problematic but i'm not sure it's worth adding additional checks to repository to prevent errors related to src. it's possible, checking that sourceProvence isSource, but perhaps not necessary. my point is that adding the repository field will not break current usage.

You don't know that. Current usage can include toJSONing the meta attributes of every package in the list. This currently works—I just confirmed it—and would break with your proposed change.
I'm fine with #300286 but I think any PR that makes meta depend on src, even in the way you're currently endorsing, should be reviewed by more of the community before it is merged.

This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/when-should-meta-attributes-be-computed/42833/1

Current usage can include toJSONing the meta attributes of every package in the list

if that is a usage we want to support then it should have a CI test.

In PR #301947, the new attribute still depends on src, but it is wrapped in a tryEval, meaning that toJSON etc. will always work. I'll merge it soon unless another objection is raised.


Add polyfills section to expo documentation

When using ElectricSQL with react native, the build fails because electric is using some browser APIs that are not available in  react native / expo. To get it to compile, you need to polyfill the missing function calls. This is definitely not the only way to do this, but the only one I got working. This should also work for react native without expo, but I don't have this set up to test it properly right now.

Thank you for updating the docs!
We have pushed a fix for this to avoid requiring people to install polyfills for electric to work, see merged PR https://github.com/electric-sql/electric/pull/1009
The only polyfill that will not be strictly required but included in the drivers and encouraged is the Crypto API getRandomValues polyfill.
I'm reluctant to add to the documentation as we should be releasing a new version that will render this obsolete.

Nice to see this beeing fixed! How long until the release that fixes this? We could merge this in the meantime to avoid confusing users that want to try out electric right now.

Hey @malte-j - version 0.9.6 which released last Fri no longer requires all those polyfills, only the getRandomValues one and there's a section in the docs for it on both the Expo and RN integrations.
Thanks for the contribution and sorry for the bad timing! I will be closing this PR as it no longer applies.


Add Edge browser and driver versions for 85.0.564.44

Is it possible to update the edge browser and drivers on the windows 2019 azure hosted environment? I see both the existing and the one marked pre-release are still on v84. v85 appears to be the current version being installed with new Edge browser installs.

Hi @bgoggan!
Edge version will be updated automatically during next week's windows deployments.
macOS image with edge 85.0.564.44 is being deployed at the moment, the update is supposed to be finished in a few hours.
As a temporary workaround you can use the following steps:
Windows:
choco upgrade microsoft-edge
macOS:
brew upgrade microsoft-edge

@miketimofeev that sounds great. Thanks for the update.

@miketimofeev thanks for the additional info but I actually download and keep the edge drivers contained as part of my test project so once I added the command to update the existing browser they ran as expected.

@bgoggan The new image with Edge 85.0.564.51 has finally deployed.
I'm going to close the issue, please feel free to contact us if you have any concerns.
Thank you!

@miketimofeev Thanks, I'll remove the update from my release.


phpCAS relies on curl being available

If curl is not available, or curl_exec() has been disabled, as it is on some systems, there is no alternative used, eg using stream_context_create() and file_get_contents().

Happy to receive a PR for this new feature


Timeout option

Can you please add a timeout option for the Web request?

Implemented in dev-master release(commit https://github.com/merorafael/telegram-handler/commit/3a3a360d7ff24446b9ec803982be400724854277). Do you can test this?

Thank you! Can you please add both options: CURLOPT_CONNECTTIMEOUT and CURLOPT_TIMEOUT? Link


Generate and Publish sha256sum.txt in RKE Releases

https://github.com/rancher/rke/issues/3652
There used to be a sha256sum.txt file in RKE's releases, for example v1.5.10.
The sha256sum.txt is missing in the recent releases since we switched from Drone CI to GitHub Action.
We should generate and add the missing sha256sum.txt back.

Confirmed, its getting generated and added to the release.
ref: https://github.com/rancher/rke/releases/download/v1.6.2-rc1/sha256sum.txt
Closing the issue.


Issue warning whenever RR or DR is zero

The _TP_estimator should raise a warning whenever there is division by zero, which occurs whenever RR or DR has an entry with 0.
This was pointed out by @mclaughlin6464, who noticed that that the tpcf_jackknife function sometimes returns a covariance matrix with NaN values in it, without warning the user.

Probably an exception should be raised if any RR are zero, describing the situation to the user, and how they can proceed to either:

increase bin size and/or Nran
directly call the pair-counter themselves and do whatever homebrewed masking they want

One additional nicety for the error message (and the docstring) would be to provide a simple estimate for how many randoms the user should probably use. The simplest estimation comes from requiring the expectation value of, say, ten, for the expected number count of randoms inside a sphere of radius rbins[0]. This is straightforward, since this expectation value is just Nran_tot*(4pi/3)(rbins[0]/Lbox)**3. Slightly fancier estimations could come from exploiting the knowledge that fluctuations in the randoms will be purely Poissonian, and so it is also straightforward to estimate the probability that at least one of Njack_samples contains an RR with zero, but that is almost certainly overkill.


[#2983] fix operator.

#2983

Codecov Report

Merging #3112 into master will decrease coverage by 0.08%.
The diff coverage is 67.74%.

@@            Coverage Diff             @@
##           master    #3112      +/-   ##
==========================================
- Coverage   38.82%   38.74%   -0.09%
==========================================
  Files        2003     2003
  Lines       66141    66120      -21
  Branches     9246     9236      -10
==========================================
- Hits        25681    25618      -63
- Misses      37874    37911      +37
- Partials     2586     2591       +5

Impacted Files
Coverage Δ

...ument/matcher/operator/BracketMatcherOperator.java
60% <ø> (-6.67%)
:arrow_down:

...rument/matcher/operand/AbstractMatcherOperand.java
100% <100%> (ø)
:arrow_up:

.../operand/SuperClassInternalNameMatcherOperand.java
64% <100%> (ø)
:arrow_up:

...r/operand/InterfaceInternalNameMatcherOperand.java
68% <100%> (ø)
:arrow_up:

...instrument/matcher/operator/OrMatcherOperator.java
73.91% <60%> (+4.68%)
:arrow_up:

...nstrument/matcher/operator/AndMatcherOperator.java
73.91% <60%> (+4.68%)
:arrow_up:

...nstrument/matcher/operator/NotMatcherOperator.java
68.42% <66.66%> (-4.31%)
:arrow_down:

.../java/com/navercorp/pinpoint/rpc/cluster/Role.java
0% <0%> (-42.86%)
:arrow_down:

...p/pinpoint/rpc/client/WriteFailFutureListener.java
56.25% <0%> (-18.75%)
:arrow_down:

...vercorp/pinpoint/rpc/packet/ServerClosePacket.java
52% <0%> (-16%)
:arrow_down:

... and 19 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f123b46...3cabecc. Read the comment docs.


InfoSubmission pulling questions from backend

Once the form has been edited, questions in InfoSubmission should pull from the backend.
Acceptance Criteria

sample text

User Story

sample text

Story Points

sample text

Closed by #99


Update predictor.py

Adding weight loading from saved pth file.

Hey @KelvinCPChiu
Thanks for the PR :) I've been using it for quite long, but now we can merge it as well

why need it? since the MODEL.WEIGHT can do the thing.


ORecordDuplicateException for non-duplicate record

I am using latest version of orientdb i.e. 2.1.16
and I have a index at phoneNumber
While inserting (massive inserts) in embedded mode at document level the below exception comes.
Phone number to be insert is *141*10 and it is found duplicate with phone number *141*10# present at
RID#24:117 and following exception comes
com.orientechnologies.orient.core.storage.ORecordDuplicatedException: Cannot index record contact_info#17:9387{phoneNumber:*141*10,lastname:loan,firstname:Air
tel,spamScore:0.0,circle:null,fullname:Airtel loan,city:null,in_contact_link:[#38:9373]} v1: found duplicated key '*141*10' in index 'contact_info.phoneNumber
' previously assigned to the record #24:117 RID=#24:117
at com.orientechnologies.orient.core.index.OIndexTxAwareOneValue.checkEntry(OIndexTxAwareOneValue.java:224)
at com.orientechnologies.orient.core.index.OClassIndexManager.checkIndexedPropertiesOnUpdate(OClassIndexManager.java:376)
at com.orientechnologies.orient.core.index.OClassIndexManager.checkIndexes(OClassIndexManager.java:583)
at com.orientechnologies.orient.core.index.OClassIndexManager.onRecordBeforeUpdate(OClassIndexManager.java:430)
at com.orientechnologies.orient.core.hook.ODocumentHookAbstract.onTrigger(ODocumentHookAbstract.java:256)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.callbackHooks(ODatabaseDocumentTx.java:1070)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeSaveRecord(ODatabaseDocumentTx.java:1936)
at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveRecord(OTransactionNoTx.java:159)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:2492)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:122)
at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:1768)
at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:1759)

Hi @nsitanurag ,
Why do you think that you do not have duplicated records ?

Because one number is *141*10 which is found to be duplicate with *141*10# which should not be the case since second number contains # at the end.

@nsitanurag I got the case, is it possible to send us database for this issue ?

Sorry, I had flushed all data and it's not possible. You can make an index on a String property and try inserting 14110# first and then 14110 as second record in it.

Same problem was observed with String containing ; (semi-colon) at the end. e.g. *141*10;

@nsitanurag, I have tried following commands on 2.1.16 and develop branches, everything works fine. Could you please make it fail on your side ;)
create database plocal:/tmp/phones
create class Person extends V
create property Person.phone string
create index Person.phone unique

create vertex Person set phone='*141*10#'
create vertex Person set phone='*141*10'

Ok. I will try and let you know.

I am using unique_hash_index and inserting records using document api in embedded mode and not using sql commands in console. Can you try that and check?

@nsitanurag yes, I will check it in the embedded mode.

Tested on 2.1.16 and develop branches, no exceptions:
final OClass class_ = db.getMetadata().getSchema().createClass("Person");
class_.createProperty("phone", OType.STRING);
class_.createIndex("Person.phone", OClass.INDEX_TYPE.UNIQUE_HASH_INDEX, "phone");

db.begin();
db.newInstance("Person").field("phone", "*141*10#").save();
db.newInstance("Person").field("phone", "*141*10").save();
db.commit();

Please make it fail.

@nsitanurag could you send us the code which you use for massive insertion, to help us to reproduce the issue ?

Close issue because of no reply. If you reproduce given issue and have information which is requested in issue feel free to reopen it.


Hide screen markers

Yo, there's not much activity at my feature request on the runelite github  and I figured this might fit into your plugin.

this falls under the feature requested here: https://github.com/PresNL/hide-widgets/issues/2 for now the workaround is simply to hide the runelite overlays yourself using their respective plugins.
I also suspect this issue will remain open unless I can figure out a proper way of hiding all overlays from runelite which I don't think exists at the moment


UI thread rendering

Description
Hey,
Apologies if this is already out there. Is there any plan to offer rendering off the main thread similar to Reanimated?
It would be handy for animations to continue when the JS thread is blocked (doing compute), especially for gestures.
Here is an example (code below). Panning on the x-axis uses Reanimated, and panning on the y-axis uses Skia. A blocking function runs every 3 seconds.
skia-blocking.webm
import * as skia from "@shopify/react-native-skia";
import { useEffect } from "react";
import { useWindowDimensions, View } from "react-native";
import { Gesture, GestureDetector, GestureHandlerRootView } from "react-native-gesture-handler";
import Animated, { useAnimatedStyle, useSharedValue } from "react-native-reanimated";

const block = () => {
  for (let i = 0; i < 50000000; i++) {}
};

export const Root = (): JSX.Element => {
  const { width, height } = useWindowDimensions();

  useEffect(() => {
    const i = setInterval(block, 3000);
    return () => clearInterval(i);
  }, []);

  const xPositionReanimated = useSharedValue(0);
  const yPositionReanimated = useSharedValue(0);

  const panGesture = Gesture.Pan().onChange((e) => {
    xPositionReanimated.value += e.changeX;
    yPositionReanimated.value += e.changeY;
  });
  const reanimatedStyle = useAnimatedStyle(() => ({
    translateX: xPositionReanimated.value,
  }));

  const skiaY = skia.useValue(0);
  skia.useSharedValueEffect(() => {
    skiaY.current = yPositionReanimated.value;
  }, yPositionReanimated);

  return (
    <View style={{ width, height }}>
      <GestureHandlerRootView>
        <GestureDetector gesture={panGesture}>
          <Animated.View style={reanimatedStyle}>
            <skia.Canvas style={{ height: height, width: width }}>
              <skia.Rect x={0} y={skiaY} height={100} width={100} color="red" />
            </skia.Canvas>
          </Animated.View>
        </GestureDetector>
      </GestureHandlerRootView>
    </View>
  );
};

Hi @louix! Thanks for suggesting this :) It is not available yet - but this is definitely a feature we're looking forward to release :) We showed some examples of it in the latest video on @wcandillon 's channel and we're working hard to bring this to you all 🥳

We are now rendering the UI thread but most animations need to be done via the JS thread at the moment, we are looking at the next steps there.

As @wcandillon says, from version Release 0.1.159 beta we're rendering on the main thread - which should offer rendering on first frame - so I'm closing this issue :)
PS. Eventually we'll also start updating our computed animation values using workouts as well.


[Improvement] Impose Overlap

Can the overlays not overlap the impose screen? (The screen when you long press PS button)

i'm not sure but an idea would be to make impose screen transparent then the background would be visible

i'm not sure but an idea would be to make impose screen transparent then the background would be visible

@jordywastaken made it by hooking the process in system_plugin.rco but when on XMB i don't know how it would be achived in-game

i'm not sure but an idea would be to make impose screen transparent then the background would be visible

About a transparent impose screen idk if it will be a good idea, i think it's kinda ugly
i made a transparent premo.rco and this is how it looks like

i'm not sure but an idea would be to make impose screen transparent then the background would be visible

@jordywastaken made it by hooking the process in system_plugin.rco but when on XMB (to mimic the dex explorer_plugin behaviour, it hides ifself when needed, but the previous version worked on every XMB screen and hides only when on the impose screen) i don't know how it would be achived in-game

is this for a particular use case?

i'm not sure but an idea would be to make impose screen transparent then the background would be visible

@jordywastaken made it by hooking the process in system_plugin.rco but when on XMB (to mimic the dex explorer_plugin behaviour, it hides ifself when needed, but the previous version worked on every XMB screen and hides only when on the impose screen) i don't know how it would be achived in-game

is this for a particular use case?

You mean @jordywastaken plugin? if so, yes for HEN users that want to customize the explorer_plugin.sprx (dex) ip information without the need of a remap and a xmb reload.
If you mean about the improviment request, no, I just think it would look better and get official behavior as the other overlays don't show up in the impose screen

Can the overlays not overlap the impose screen? (The screen when you long press PS button)

Easiest way to do that is to change the fps counter parent widget, currently it uses 'page_autooff_guide' which is displayed on top of anything else. Each page in system_plugin has a different behavior, one of which has the behavior you described, it is 'page_notification'.
Long story short:
I recently noticed that the ccapi vsh menu isn't displayed on top of the impose screen, so with a bit of reverse engineering I found out it used 'page_notification' as a parent, did a quick test and the fps counter behaved exactly like the ccapi menu.

@TheRouletteBoi can you take a look at it?
when on xmb to show only on xmb itself not while playing movies or while on web browser

While watching a movie it is even more annoying. i think you should keep it only on the xmb using @jordywastaken suggestion

@TheRouletteBoi can you take a look at it?
Make it show only on xmb itself not while playing movies or while on web browser

While watching a movie it is even more annoying. i think you should keep it only on the xmb using @jordywastaken suggestion

is this for the fps counter?

@TheRouletteBoi can you take a look at it?
Make it show only on xmb itself not while playing movies or while on web browser

While watching a movie it is even more annoying. i think you should keep it only on the xmb using @jordywastaken suggestion

is this for the fps counter or vsh menu?

FPS counter

here is the sprx you can test. let me know if it works. https://file.io/BIjodnZ0JNlT

Nothing changed, it still shows under videos/music/web browser.


Cache setup doesn't work on first run

Have you experienced this bug with the latest version of the template?
Yes, with a newly generated template
Steps to Reproduce
Generate the indie stack and follow the fly setup guide. No code has been changed at all.
Expected Behavior
Expected it to complete the build and deploy to Fly
Actual Behavior
After pushing to GitHub and running the deploy action, the Build step will fail with the following message:
2022-07-03T18:19:23.2351707Z #26 [stage-4 9/9] COPY --from=build /myapp/prisma /myapp/prisma
2022-07-03T18:19:23.2351952Z #26 DONE 0.0s
2022-07-03T18:19:23.3848909Z
2022-07-03T18:19:23.3849344Z #27 exporting to image
2022-07-03T18:19:23.3849638Z #27 exporting layers
2022-07-03T18:19:31.5031990Z #27 exporting layers 8.1s done
2022-07-03T18:19:31.5032863Z #27 exporting manifest sha256:399ea710e69ea4a50399d69e21d9c97c102e3fabbe24f75032ddc54e69fa218c done
2022-07-03T18:19:31.5033348Z #27 exporting config sha256:3600d6bfa7a170576c647912aa4ac13c683983ca24889ee63db5f5d6993592a2 done
2022-07-03T18:19:31.5033684Z #27 pushing layers
2022-07-03T18:19:31.6539904Z #27 ...
2022-07-03T18:19:31.6540128Z
2022-07-03T18:19:31.6542961Z #28 [auth] sharing credentials for registry.fly.io
2022-07-03T18:19:31.6543435Z #28 DONE 0.0s
2022-07-03T18:19:31.8043599Z
2022-07-03T18:19:31.8044012Z #27 exporting to image
2022-07-03T18:19:32.0626042Z #27 pushing layers 0.7s done
2022-07-03T18:19:32.0664787Z #27 ERROR: unexpected status: 404 Not Found
2022-07-03T18:19:32.0666088Z ------
2022-07-03T18:19:32.0666304Z  > exporting to image:
2022-07-03T18:19:32.0666547Z ------
2022-07-03T18:19:32.0667177Z WARNING: local cache import at /tmp/.buildx-cache not found due to err: could not read /tmp/.buildx-cache/index.json: open /tmp/.buildx-cache/index.json: no such file or directory
2022-07-03T18:19:32.0741515Z error: failed to solve: unexpected status: 404 Not Found
2022-07-03T18:19:32.0894846Z ##[error]buildx failed with: error: failed to solve: unexpected status: 404 Not Found

Same issue on my side
#26 [stage-4 9/9] COPY --from=build /myapp/prisma /myapp/prisma
#26 DONE 0.0s
#27 exporting to image
#27 exporting layers
#27 exporting layers 10.0s done
#27 exporting manifest sha256:2627e8644049e176e24285ad769c73793083f79317c7b6dc101c9b6a7215285d done
#27 exporting config sha256:994aa7b16ee309e132effa7771b3d3b1836acedb5cdf7df208c57b6bed755719 done
#27 pushing layers
#27 ...
#28 [auth] sharing credentials for registry.fly.io
#28 DONE 0.0s
#27 exporting to image
#27 pushing layers 0.7s done
#27 ERROR: unexpected status: 404 Not Found
------
 > exporting to image:
------
WARNING: local cache import at /tmp/.buildx-cache not found due to err: could not read /tmp/.buildx-cache/index.json: open /tmp/.buildx-cache/index.json: no such file or directory
error: failed to solve: unexpected status: 404 Not Found
Error: buildx failed with: error: failed to solve: unexpected status: 404 Not Found

I'm having the exact same problem.
#25 pushing layers 0.1s done
#25 ERROR: unexpected status: 404 Not Found
------
 > exporting to image:
------
WARNING: local cache import at /tmp/.buildx-cache not found due to err: could not read /tmp/.buildx-cache/index.json: open /tmp/.buildx-cache/index.json: no such file or directory
error: failed to solve: unexpected status: 404 Not Found
Error: buildx failed with: error: failed to solve: unexpected status: 404 Not Found

Seemed like I had the wrong app name configured in fly.toml https://fly.io/docs/reference/configuration/#the-app-name
It didn't match the name of the fly app I've created.

Ahh, yea - that was why. I was also using a different app name on fly than the one in fly.toml. I was following the README on github - not the one generated locally after creating the stack.


listMonth does not render event with the last date of month

Looks like the fullCalendar is not rendering the event with the last date of the month. For example, I created an event on the 31st of march with the start time as 6:30 p.m., then that event is not shown in the listMonth view.
This issue is only happening for the events with the last date of the month whose start time is 6:00 p.m. or after that.
I have attached two images. When the fullCalendar view is "month" the event is visible, but when the event is rendered with the "listMonth" view the event is not rendered.
Steps to reproduce:

Create an event with date:
start: 2022-03-31 6:30 PM, end: 2022-03-31 7:30 PM

Render the event in listMonth view.

Yes, @jyotisingh-tudip I am also facing the same issue.
Would wish this to get solved soon.
Thanks!

@jyotisingh-tudip @rohandani-tudip you seem to be experiencing an issue with fullcalender, but this GitHub project is only the packaging of the fullcalendar project for Ruby on Rails. You probably want to let the maintainers of fullcalendar know about your issue here: https://github.com/fullcalendar/fullcalendar

@gr8bit, thanks for the reply. We will reach out to the fullcalendar team.


Adding Kingsdale Foundation School

This school is a High School, but it offers a Sixth Form program.
The real domain for the email is @students.kingsdale.southwark.sch.uk but I had trouble trying to add that.
School Website:
https://kingsdalefoundationschool.org.uk/
Sixth Form Program Info:
https://s3-eu-west-1.amazonaws.com/sh22-kingsdalefoundationschool-org-uk/media/downloads/kfssixthformstudyprogrammescoursesforseptember2024year12entry.pdf

@The8BitGamer887 First stage of verification passed, official domain is kingsdalefoundationschool.org.uk as far as I verified. This is just an informational note. Pull request is still in the review. I appreciate your patience while waiting for the final decision. Proof of domain ownership: https://kingsdalefoundationschool.org.uk/

@The8BitGamer887 Thank you for your patience while waiting for a review and processing your request. Pull request merged.

@philipto No problem, thank you for verifying and merging this :)