Resolve dependencies using UnityNuGet scoped registry

To facilitate the installation and above all, to avoid collisions with assemblies, I add in this PR the changes with instructions included to adapt it to use this scoped registry.
By the way, I am mantainer of the mentioned repository: https://github.com/xoofx/unitynuget

Thank you for your pull request!
Let me explain our distribution policy including dependent libraries.
First of all, I knew there is a great project called UnityNuGet where you can get dependent packages. However, we decided not to adopt that approach in our project.
The first reason is the conflict of assemblies.
This is not a problem when originally configured with UnityNuGet, but it becomes a problem in cases where we have our own internal UPM server and managed assemblies are distributed within those packages. In such cases, it will be necessary to remove the dependency from one or the other, or to localize the package and delete the DLL.
The second reason is the responsibility for distribution and installation.
We prefer not to assume dependency on further 3rd party registries that are different from us. For instance, what if UnityNuGet has an outage?
In enterprise game development companies, for compliance reasons, they may scrutinize the libraries they are using.
In such cases, you might need to convince legal to use UnityNuGet in addition to GitHub and this repository/Organization.
Of course, this does not mean that we do not trust UnityNuGet. However, whether or not the user trusts it is different and cannot be forced.
Initially, we considered having users obtain the dependent libraries from NuGet.
However, we found that this was difficult, so we decided to redistribute using the most basic yet simple method of .unitypackage.
While UnityNuGet is convenient and provides a clean solution, it is realistically challenging to ask our customers to use it, from what we have observed with several developers.
Therefore, although we are sorry that you took the time to create a PR, we do not currently have any plans to accept methods that rely on unofficial UPMs, not just UnityNuGet.

@mayuki I understand the concerns but let me argue them and try to partly redirect the purpose of this PR with some adjustments.
Conflict resolution
I really missed the point of your explanation because I see that you agree with me that UnityNuGet is the best solution for this case.
Security concerns / Enterprise world
We develop a product which serves as a platform for XR content developers.
We solve our dependencies with UnityNuGet / OpenUPM (which does upstream https://openupm.com/nuget/) because this target of developers in our case has very little experience in development and that is why we opted for the simplest way which is the scoped registries and that everything is solved "magically" without any action to be performed by users other than the initial setup of the project by adding the scoped registries through the UI or in some cases not even that because we distribute a "template" project with everything preconfigured.
As you can understand, in our case it is crucial that all dependencies can be resolved through packages automatically because our users simply install our package and this in turn brings all the necessary dependencies either ours or third parties. And in successive updates they only have to update our package and nothing else, they don't have to deal with copying dlls by hand, etc... And if we have more advanced users who want to use other assets of OpenUPM, NuGet or Asset Store, the fact of solving everything as packages, facilitates this work enormously.
There are more special cases as you mentioned.
For these cases you can "defend" how this project (UnityNuGet) is open source, is on GitHub and is nothing "obscure" and is supported by a Unity employee (xoofx). Besides being a Docker container it is very easy to host it on your own without relying on Azure hosting by default (and then it would only be to change the url of the scoped registry in the manifest.json file). On the other hand, for cases where it is "not defensible" or external connections are not allowed, you can embed it, i.e. copy from Library/PackageCache to Packages and thus not depend on an external scoped registry.
Therefore, and if you agree, I propose the following modifications in this PR:

Copy the packages from Library/PackageCache to Packages and thus not depend on the scoped registry but at the same time keep the dependencies section of the package.json file so that it can get to work directly with OpenUPM / UnityNuGet at the same time.
Modify the script that exports the *.unitypackage file to do it with these new folders that are in Packages.

This way it would work as it works until now only that the assemblies instead of in Assets/Plugins will hang from Packages as embedded packages.
It also makes it easier for you to maintain this project because if you need to update the dependencies in the future you don't have to go to each NuGet page in the dependency tree and download them one by one. Just update the top-level packages (System.IO.Pipelines and System.Runtime.CompilerServices.Unsafe) and copy back from Library/PackageCache to Packages.

@mayuki what do you think of my previous message? It would allow to solve the problem this PR is trying to solve while not hurting the use cases you are concerned about.

Is it possible to use UnityNuGet to get the package dependencies and copy them when creating the .unitypackage (which will be extracted under Plugins)?
Also, if we use UnityNuGet to resolve dependencies, can we choose .NET Standard 2.1 as the target framework for the assembly?

Sorry for the delay in response.
The idea of using UnityNuGet to resolve dependencies and packaging the result sounds good!
I would appreciate it if you could modify the PR to that approach if possible.

@mayuki the PR is ready.
I have updated the README.md indicating the 2 ways of installation: with UnityNuGet or without it. In the second case it works exactly the same as before, you have to import a *.unitypackage for YetAnotherHttpHandler dependencies and another one for Grpc.Net.Client, the difference is that now instead of importing in Assets/Plugins you import UPM packages in /Packages as I said.
2. I have updated the export script of *.unitypackage files using Hybrid Packages.
The nice thing is that the way I set it up makes it infinitely easier to lift the dependencies because you only have to edit the version in the manifest.json file for the gRPC ones and the packages.json file for the YetAnotherHttpHandler ones. You no longer have to manually download anything from the NuGet web site.
The only thing missing on your side after the merge is to release another release to be able to publish the 2 new *.unitypackage and change the links in the README.md file.

Thanks for the update! I will review it now.

It looks very nice and simple.👍
I found that the display name of the package from the unitypackage (built-in Package) looks like org.nuget.system.buffers.
Is it possible to set the displayName when UnityNuGet generates package.json?
I feel a little uncomfortable having the package IDs listed under Packages in the Package Manager and the Project View.

@mayuki If you clear the UPM cache on your computer and reimport the project from scratch, you will see that packages now have a display name.

@mayuki check my changes because I think everything we talked about is already resolved.


Cannot use Golden Shovel on Arcane Pedestal

Needed to create a Golem core of Block Breaking.
Infusion recipe calls for a golden shovel amongst other things, to be placed on an arcane pedestal.
Normally you right click to place it, but the shovel is always trying to create a new claim.
Not sure if there is a way to temporarily disable the shovel, cant find one.
Maybe need to remove the golden shovel from recipe list?

what?

This sounds like a sponge issue but can be resolved by using a hopper to put the item on the pedestal.

There's a command to make the tool no longer be an actual claiming tool. I can't remember the command at the moment, however if I figure it out I will either edit this comment or post another comment.


6.11.2 tag missing from Docker Hub

I noticed 6.11.2 has been released a couple of days ago but Docker Hub only has 6.11.2-rc.2.
Is there no automated pipeline for this?

Hi @rafaelmagu,
Thanks for the report. I have checked and 6.11.2 is actually not released yet.
I am wondering what made you think it was? Was it the github release page?
This page is a bit confusing since it lists all git tags as "releases". Tagging is part of our build process but does not necessarily mean that an artifact has been made public yet.
If you go to the second page of the release list you will see that 6.11.1 is marked as "Latest release".
6.11.2 should be available later today
Arthur

6.11.2 artifacts are out, closing
Thanks again for being vigilant.


Some characters is not printing

It's not issue of this project, but a problem was inherited by them. It's problem of vncserver (x11vnc), but author didn't answer.
x11vnx works fine on english layout, but if me need to type cyrillic characters than some characters is not printing.
https://github.com/LibVNC/x11vnc/issues/23
I found out how to solve a problem:
apk add setxkbmap
setxkbmap ru
but it's  annoying.
Another aouthor found out earlier, that problem in keyboard.c file
Problem was solved by increasing size of array.
Could you make jlesage firefox with custom x11vnc ?

Latest image no longer use x11vnc.  Can you check if this is still an issue ?


Dynamic Change AudioContext between Speaker phone and  Earpiece

Checklist

[X] I made sure that the feature I am proposing doesn't already exist

Use case / Problem
I'm trying to have a player (for an audioguide) capable of switching between speakers of earpiece.
In version 0.x I had a command earpieceOrSpeakersToggle, but now the implementation is different.
I've used as suggested the audio context.
So for Android I've tested two configurations:
Earpiece:
AudioContext audioContextEar = const AudioContext(
    android: AudioContextAndroid(
      isSpeakerphoneOn: false,
      usageType: AndroidUsageType.voiceCommunication,
      audioMode: AndroidAudioMode.inCommunication,
    ),
  );

Speaker:
AudioContext audioContextSpeaker = const AudioContext(
    android: AudioContextAndroid(
      isSpeakerphoneOn: true,
      usageType: AndroidUsageType.media,
      audioMode: AndroidAudioMode.normal,
    ),
  );

if I set the context in the initialization phase it works.
If I try to change the context when audio is playing, I have no change at all.
I've look in the example, but also the example does not work as expected, if I try to change speakerPhone on the user interface it does not work.
Desired solution
A working example on how to change AudioContext dynamically, even stopping and restarting the audio file.

Thanks for a well written issue!


setProjectKey deprecated

I have a library used to set the signature in a WebApp
and yesterday started to say "Access not grant or expired"
I realized I was using the v.17, but the ProjectKey is deprecated
I switch to v21 removing from the function getService(ServiceName,Scope) the
.setProjectKey("xxxx")
how is managed the ScriptID?
the Aouth2 library is load in a library load by my script/webapp could be this the problem?
thanykou
Giorgio
function getService(ServiceName,Scope) {

  // Create a new service with the given name. The name will be used when
  // persisting the authorized token, so ensure it is unique within the
  // scope of the property store.

  // ServiceName: sharedcontacts
  //              AdminMail
  return OAuth2.createService(ServiceName)

      // Set the endpoint URLs, which are the same for all Google services.
      .setAuthorizationBaseUrl('https://accounts.google.com/o/oauth2/auth')
      .setTokenUrl('https://accounts.google.com/o/oauth2/token')

      // Set the client ID and secret, from the Google Developers Console.
      .setClientId(PropertiesService.getScriptProperties().getProperty("Client_Id"))
      .setClientSecret(PropertiesService.getScriptProperties().getProperty("Client_Secret"))

      // Set the project key of the script using this library.
      //.setProjectKey('xxxx')

      // Set the name of the callback function in the script referenced
      // above that should be invoked to complete the OAuth flow.
      .setCallbackFunction('authCallback')

      // Set the property store where authorized tokens should be persisted.
      .setPropertyStore(PropertiesService.getUserProperties())

      // Set the scopes to request (space-separated for Google services).
      .setScope(Scope)

      // Below are Google-specific OAuth2 parameters.

      // Sets the login hint, which will prevent the account chooser screen
      // from being shown to users logged in with multiple accounts.
      .setParam('login_hint', Session.getActiveUser().getEmail())

      // Requests offline access.
      .setParam('access_type', 'offline')

      // Forces the approval prompt every time. This is useful for testing,
      // but not desirable in a production application.
  //   .setParam('approval_prompt', 'auto');
  .setParam('approval_prompt', 'force');
}

Even before the deprecation we made a change such that the project key and script ID are determined automatically. There is no need to set the Script ID under normal circumstances.

thankyou,
but is not clear for me why, just removing the
.setProjectKey('xxxx')
and using the v21
now I have
Access not granted or expired. (line 352, file "Service", project "OAuth2")
:(

I'm not sure why either. The simplest solution is probably to reset the service and authorize it again.

well, is what I DID

Switching to a script ID also means that the callback URL has changed, so you may need to update that URL in your developers console project.

fyi
I just switch to a Service Account and is working


Enabled: false not working

I don't know Ruby well so I might be wrong, but I think it might be due to this line:
globally_enabled = @settings['enabled'] || true

If I set enabled to false, that is of course falsy and || kicks in setting it to true again.

Give it another shot with 0.1.9.

Works! Thanks for fixing this :+1:


[Request] Add ability to disable plugin for specific tabs

For example,
You should be able to disable tab 1 and it will go back to its default state, while still using the plugin for Tab 2-8

There is already checkbox for that!

There is already checkbox for that!

I'm referring to ALL the tabs, not just tab 0.

@Beagon the legend completed this one!


PSR-12 code standards

please learn & adopt the PSR-12 code standard in your code (all your PHP projects)
this will make it easier for other developers to review your code & contribute.

looks like fixed in 9d5527856633d055ed025d43f9e56e01121f5168

yes it is, i will clean other project code as well, thank 👍


When to support typescript?

Help us to manage our issues by answering the following:

How would you tag this issue?

Question
Bug
Discussion
Feature request
Tip
Enhancement
Performance

Describe your issue:

What is the current behavior?
What is the expected behavior?
Which versions of Rax, and which browser / OS are affected by this issue?
Did this work in previous versions of Rax?

If your issue is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem via Bug Report Template on CodePen
.

There is currently no plan to support typescript, but welcome PR. 😆

@janryWang OK, thank to reply me.

@janryWang Can you give me a copy of your email? If I support the typescript program, I hope to contact you for a more precise type definition.


add python 3.8, 3.9 on setup.py

related to https://github.com/spotify/luigi/pull/3048
I added python 3.8, 3.9 on setup.py.

Description

I missed to add python versions on setup.py in https://github.com/spotify/luigi/pull/3048
Motivation and Context

To show python versions at PyPI.
Have you tested this? If so, how?

tested on CI

@dlstadther @Tarrasch @ravwojdyla Sorry I missed to add python versions on setup.py in https://github.com/spotify/luigi/pull/3048
I just added it.


[Nimble] [v4.4.1][ESP32-S3] ble_gattc_notify_custom returns ENOMEM (BLE_HS_ENOMEM) (IDFGH-7534)

ESP-IDF: v4.1.1 (ae0e3e2e2a, May 31 10:28:03)
ESP32-S3
ble_gattc_notify_custom() is returning ENOMEM, and there's no obvious reason.
🎉EDIT: It appears that I'm calling ble_gattc_notify_custom() too fast. Adding some sleeps fixes the issue! This seems like an Espressif bug. See next comment.🎉
I've tracked down that  ENOMEM is coming from ble_att_clt_tx_write_cmd in ble_att_clt.c
int
ble_att_clt_tx_write_cmd(uint16_t conn_handle, uint16_t handle,
                         struct os_mbuf *txom)

    ... truncated....
    cmd = ble_att_cmd_get(BLE_ATT_OP_WRITE_CMD, sizeof(*cmd), &txom2);
    if (cmd == NULL) {
        os_mbuf_free_chain(txom);
        return BLE_HS_ENOMEM; // this is where it comes from
    }

But unfortunately ble_att_cmd_get() does not exist? It looks like it must be a proprietary blob from the linker?
My situation:

I have plenty of free ram. 100+ KB of all RAM types.
ble_gattc_notify_custom() sometimes works.
In my code I use a SPP over BLE architecture.
I have 2 Characteristics, 1 with NOTIFY, and 1 with WRITE_NO_RESP.
I BLE Write a serial command, and the ESP32 BLE Notifies JSON data back.
For Notifying large JSON data, I call split_notify_msg() (proprietary).
ble_gattc_notify_custom() returns ENOMEM when whensplit_notify_msg() msgLen > ~4000 bytes. This is bewildering to me.  I call '/api/time/get' and it works (3963 json bytes) but /api/device-state' fails, (4623 json bytes).
I've tried hardcoding the JSON response, so that all serial commands Notify the same JSON data. It fails when I make the hardcoded JSON string > ~4000.
Again, I have plenty of ram. The JSON that I'm Notifying is just sitting in ram in both cases.
When It fails, it fails on the 2nd / 3rd call to ble_gattc_notify_custom().
rebooting makes no difference. The order of the serial commands I invoke makes no difference.
MTU size is 512 bytes

My Code:
bool split_notify_msg(uint16_t gapConnHandle, uint16_t attrHandle, const uint8_t* msg,  uint32_t msgLen)
{

    uint16_t mtu = get_mtu(gapConnHandle);

    uint8_t* pkt = (uint8_t*) calloc_safe2(MALLOC_CAP_SPIRAM, mtu, 1, HERE);

    // how many packets needed?
    uint32_t roundUp = msgLen % mtu == 0 ? 0 : 1;
    uint32_t numPackets = (msgLen / mtu) + roundUp;

    for (int i = 0; i < numPackets; i++){

    	// copy to pkt
        uint32_t offset = mtu * i;
        uint32_t pktLen = PDMIN(msgLen - offset, mtu);
        memcpy(pkt, msg + offset, pktLen);

    	// this is returning ENOMEM
    	struct os_mbuf* txom = ble_hs_mbuf_from_flat(msg, msgLen);

    	int rc = ble_gattc_notify_custom(gapConnHandle, attrHandle, txom);
    	if (rc != 0){
    		return false;
    	}
    }

    free_safe(pkt);
    return true
}

Adding a Sleep (or even a ESP_LOGI call) in between each call to ble_gattc_notify_custom() seems to fix the issue. Is there a recommended way to do rate limiting without Sleep?

The bleprph_throughput example uses this approach:
                /* Check if the MBUFs are available */
                if (os_msys_num_free() >= MIN_REQUIRED_MBUF) {
                    om = ble_hs_mbuf_from_flat(payload, sizeof(payload));
                    if (om == NULL) {
                        /* Memory not available for mbuf */
                        ESP_LOGE(tag, "No MBUFs available from pool, retry..");
                        vTaskDelay(100 / portTICK_PERIOD_MS);
                        om = ble_hs_mbuf_from_flat(payload, sizeof(payload));
                        assert(om != NULL);
                    }

                    rc = ble_gattc_notify_custom(conn_handle, notify_handle, om);
                    if (rc != 0) {
                        ESP_LOGE(tag, "Error while sending notification; rc = %d", rc);
                        notify_count -= 1;
                        xSemaphoreGive(notify_sem);
                        /* Most probably error is because we ran out of mbufs (rc = 6),
                         * increase the mbuf count/size from menuconfig. Though
                         * inserting delay is not good solution let us keep it
                         * simple for time being so that the mbufs get freed up
                         * (?), of course assumption is we ran out of mbufs */
                        vTaskDelay(10 / portTICK_PERIOD_MS);
                    }
                } else {
                    ESP_LOGE(tag, "Not enough OS_MBUFs available; reduce notify count ");
                    xSemaphoreGive(notify_sem);
                    notify_count -= 1;
                }

This was very tricky to find.
Googling ble_gattc_notify_custom() and BLE_HS_ENOMEM yeilded nothing useful I could find.
These search terms should be added to the FAQ: https://espressif-docs.readthedocs-hosted.com/projects/espressif-esp-faq/en/latest/software-framework/ble-bt.html

Hello @chipweinberger,
You can find ble_att_cmd_get() function here.
This issue occurs when we try to send too many packets before controller can process them. In this case, the packets get queued. As we queue more number of packets, we reach a point where we run out of memory. In this case, BLE_HS_ENOMEM is returned.
There are two methods in which this issue can be solved -

Using suggested default data length read/write commands at the time of connection. These commands, will suggest controller to increase the number of data packets that can be transmitted. Please find the PR for the same here.
Introducing a delay (sleep) gives time for packet transmission, and the issue is not seen. Thus, adding delay works. If you want to avoid using delay, you can add check to retry the transmission in case of failure.

Thanks for the detailed reply.
I'll close this issue as your suggestions do work.
It would be useful for others to get this error code documented, and all the examples to use these suggestions.


Methods should not be empty

https://sonarcloud.io/organizations/apache/issues?languages=java&open=AXXSUeHQHNgm4sakNRxb&projects=apache_incubator-hop&resolved=false&rules=java%3AS1186
Following this checklist to help us incorporate your contribution quickly and easily:

[ ] Make sure there is a JIRA issue filed
for the change (usually before you start working on it).  Trivial changes like typos do not
require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
[x] Each commit in the pull request should have a meaningful subject line and body.
[ ] Format the pull request title like [HOP-XXX] - Fixes bug in SessionManager,
where you replace HOP-XXX with the appropriate JIRA issue. Best practice
is to use the JIRA issue title in the pull request title and in the first line of the commit message.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Run mvn clean install apache-rat:check to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
[ ] If you have a group of commits related to the same change, please squash your commits into one and force push your branch using git rebase -i.

Trivial changes like typos do not require a JIRA issue (javadoc, comments...).
In this case, just format the pull request title like (DOC) - Add javadoc in SessionManager.
If this is your first contribution, you have to read the Contribution Guidelines
If your pull request is about ~20 lines of code you don't need to sign an Individual Contributor License Agreement
if you are unsure please ask on the developers list.
To make clear that you license your contribution under the Apache License Version 2.0, January 2004
you have to acknowledge this by using the following check-box.

[ ] I hereby declare this contribution to be licenced under the Apache License Version 2.0, January 2004
[ ] In any other case, please file an Apache Individual Contributor License Agreement.

This is a NullMonitorListener, the methods should be empty... see comment in https://github.com/apache/incubator-hop/pull/638


sty file: Remove old dependencies, cleanup code

luatexbase-attr is now just a dummy loader for luatexbase.
luatextra is pretty outdated (hasn't seen an update for 7 years, way before LuaTeX became stable) and pulls in very old dependencies, which
are not needed any more. In fact, luatextra does nothing more than pulling in these dependencies:
\RequirePackage{fontspec}
\RequirePackage{luatexbase}
\RequireLuaModule{lualibs}
\RequirePackage{metalogo}
\RequirePackage{luacode}
\RequirePackage{fixltx2e}

(Source: http://mirrors.sorengard.com/ctan/macros/luatex/latex/luatextra/luatextra.dtx)
Out of these

fontspec is not used by lua-check-hyphen
luatexbase is already being required by lua-check-hyphen
ifluatex is already being required by lua-check-hyphen
lualibs is not being used
metalogo is not being used
luacode is not being used
fixltx2e is deprecated. Loading it results in a warning that the fixes it provides are no longer necessary since 2015.

\luatexluaescapestring is just another name for \luaescapestring nowadays anyway:
\let\luatexluaescapestring\luaescapestring

See http://ctan.mirrors.hoobly.com/macros/luatex/generic/luatexbase/luatexbase.dtx line 803.

Thank you!

Well, thank you!


formula_installer: Use bottle_dependencies

Modify bottle_dependencies to use Keg.relocation_formulae.
Revert Fix dependency cycle when install patchelf.

Codecov Report

Merging #774 into master will increase coverage by 0.03%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #774      +/-   ##
==========================================
+ Coverage   70.43%   70.46%   +0.03%
==========================================
  Files         386      386
  Lines       20711    20702       -9
==========================================
+ Hits        14587    14588       +1
+ Misses       6124     6114      -10

Impacted Files
Coverage Δ

formula_installer.rb
63.76% <0%> (+1.03%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6f165d5...05aef1b. Read the comment docs.


Provide major version lower and upper bounds on all dependencies

Is your feature request related to a problem? Please describe.
To avoid pvlib breakages when its dependencies release a major version with breaking changes, pvlib should include major version lower and upper bounds on all its dependencies.
Describe the solution you'd like
In the pyproject.toml, instead of
dependencies = [
    'numpy >= 1.17.3',
    'pandas >= 1.3.0',
    ...
]

do
dependencies = [
    'numpy >=1.17.3, <2',
    'pandas >=1.3.0, <2',
    ...
]

Describe alternatives you've considered
Keep living dangerously.
Additional context
This should be done for ALL dependencies (even the optional and testing-related ones). Some dependencies are currently given no range info, so some choices would need to be made, but usually one starts with the full range within the latest major version.

I believe the maintainers are usually updating pvlib to allow using the latest packages when an upcoming release is made. At least that was done with numpy, see #2027

When a major version of a dependency is released and an upper bound isn't already in place, fresh installs can go south before anyone has a chance to update for the breaking changes, because pip aggressively pulls the latest in fresh installs. Big packages like numpy do usually give fair warning, but after, say, numpy 2.0 releases with breaking changes, one probably won't be able to easily install an older version of pvlib-python "out of the box" without pip pulling an incompatible (newer) numpy version.
TBH: I've probably been stung by this more by dev dependencies (like pytest, or even older pip version updates) that all of a sudden stop working one day out of the blue, usually in CI after pushing because my local install hasn't updated likewise.

Placing upper bounds on dependency versions is a controversial topic in the python packaging world.  The distinction between applications and libraries is relevant when reasoning about how strict to be with dependency specifications.  This page goes into quite some detail about the reasons against doing it:  https://iscinumpy.dev/post/bound-version-constraints/
For the reasons detailed in that page, I'm of the view that imposing caps for pvlib would create more problems than it would solve.

For the reasons detailed in that page, I'm of the view that imposing caps for pvlib would create more problems than it would solve.

I'm in agreement.
I guess the ideal solution would be to retroactively add upper dependencies when new major versions start breaking code..

I guess all I can say is that I nearly entirely disagree with the arguments put forth in https://iscinumpy.dev/post/bound-version-constraints/
I have seen Python development environments and production builds fail miserably “out of the blue” (and with deadlines looming or blocking critical bug fixes) due to an either laziness or lack of awareness of the importance disciplined SemVer. I would claim that if a project cannot determine if it breaking SemVer, then it has probably gotten too complex or has some other management issue.
The TensorFlow example of a library pinning its dependency libraries to the patch level is not an example of a failure of SemVer, rather it’s an example of the misuse/abuse of SemVer. (I have literally hit this “impossible to build” issue with TensorFlow in my current position, so I do get how maddening that is.) Capping at the next major version is considerably more flexible than the TensorFlow “worst practice”.
More realistic difficulties to take issue with SemVer about are things like when a bug fix causes a breaking change. Because the Python ecosystem is so beautifully rich, a downside is that Python libraries tend to have monster dependency trees, which is where I might be more inclined to direct criticism and suggest dependency parsimony in a library like pvlib.
 However, I will say that I’ve seen Python get a reputation as to not being “production worthy” due to a perceived difference in community attitudes about proper “guardrails” and code hygiene.
Closing the issue.

When a major version of a dependency is released and an upper bound isn't already in place, fresh installs can go south before anyone has a chance to update for the breaking changes, because pip aggressively pulls the latest in fresh installs. Big packages like numpy do usually give fair warning, but after, say, numpy 2.0 releases with breaking changes, one probably won't be able to easily install an older version of pvlib-python "out of the box" without pip pulling an incompatible (newer) numpy version.

Just as pip aggressively pulls in the latest release of Numpy, then it would also pull in the latest release of pvlib (which should be compatible). I guess the incompatible issue arises if you fix your pvlib version and not the versions of the major dependencies such as numpy?

I'd like to add that it may be a good idea to add some "latest" requirements to the test matrix - maybe no requirements at all and let the manager resolve to whatever it thinks suits best. So problems with newer versions can be caught early on.

When a major version of a dependency is released and an upper bound isn't already in place, fresh installs can go south before anyone has a chance to update for the breaking changes, because pip aggressively pulls the latest in fresh installs. Big packages like numpy do usually give fair warning, but after, say, numpy 2.0 releases with breaking changes, one probably won't be able to easily install an older version of pvlib-python "out of the box" without pip pulling an incompatible (newer) numpy version.

Just as pip aggressively pulls in the latest release of Numpy, then it would also pull in the latest release of pvlib (which should be compatible). I guess the incompatible issue arises if you fix your pvlib version and not the versions of the major dependencies such as numpy?

In some (perhaps even the majority of) cases, this would sort itself out as you say. For purposes of, say, computation repeatability, a consumer may very well have pinned pvlib to a specific version. (Although, you might argue here that they would likely have pinned numpy too, but it's hard to know exactly what people will do and why. Indeed, maybe in some use case they didn't even have to import numpy directly.)
I don't mean to oversimplify this hard problem by suggesting that capping at the next breaking major version magically solves/avoids all problems. Rather, it's what I perceive as a best practice for combating dependency hell, which most likely none of us likes to spend our time in (and I certainly have some PTSD over). Problems would still pop up, but hopefully they would be more "contained" when they do. And just to reiterate, I am not proposing the "pin-to-patch-level zealotry" of TensorFlow, which is not appropriate for a library in the Python ecosystem. (My understanding is that in the Golang ecosystem all dependencies are called out as a specific commit/tag, which is quite the incentive to create libraries with very few dependencies. My sense is that makes the Go dependency ecosystem more "flat" and modular, but with correspondingly less code reuse, quite unlike the scientific Python ecosystem.)
I view the dependency ranges in a pyproject.toml as contract for the package so that for example, even in 5 years, the user should be able to dust off their old jupyter notebook pinned at pvlib v0.9.3, do pip install ., and have everything just work.

I view the dependency ranges in a pyproject.toml as contract for the package so that for example, even in 5 years, the user should be able to dust off their old jupyter notebook pinned at pvlib v0.9.3, do pip install . in a fresh virtual environment, and have everything just work.

This is certainly a worthy ideal, but it conflicts with another worthy ideal: not unnecessarily tying the hands of users.  Providing strict upper bounds on our dependencies (plus our indirect dependencies, since we can't trust our dependencies to always use upper bounds either) would make pvlib a "bad neighbor" in deployments where (1) it is one of many packages being installed and (2) our upper bounds turned out to be unnecessary, which I suspect is the case for the majority of our dependencies' major releases.  Which of these two ideals takes priority is the question of library vs application.  Unless we rename the project to pvapp, I think flexibility is the better approach ;)
Besides that, I don't want us taking on the maintenance burden of upper bounds.  Maintenance time is precious!
So there are some meaningful negatives regarding pvlib trying to provide long-term environment reproducibility.  I also see a much more general and complete alternative for users to achieve that reproducibility: $ pip freeze > requirements.txt.  With such a good alternative readily available, I don't think our pyproject.toml should be in the business of trying to keep pvlib installable after half a decade of the ecosystem moving on.

I'd like to add that it may be a good idea to add some "latest" requirements to the test matrix - maybe no requirements at all and let the manager resolve to whatever it thinks suits best. So problems with newer versions can be caught early on.

If I understand correctly, almost all of our test matrix does indeed leave the dependencies unpinned (see here), resulting in the package manager installing the latest versions :)

I'd like to add that it may be a good idea to add some "latest" requirements to the test matrix - maybe no requirements at all and let the manager resolve to whatever it thinks suits best. So problems with newer versions can be caught early on.
If I understand correctly, almost all of our test matrix does indeed leave the dependencies unpinned (see here), resulting in the package manager installing the latest versions :)

Aha! That's right @kandersolar , I misinterpreted the role of the requirements files under ci/

With breaking-changes numpy 2.0.0 now released (cf. https://github.com/pvlib/pvlib-python/issues/2026):
python -m pip install pvlib==0.10.4
python -c "import pvlib"

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/campy/Documents/Employment/IMS/solar/PVfit/venv/pvlib_0p10p4/lib/python3.11/site-packages/pvlib/__init__.py", line 3, in <module>
    from pvlib import (  # noqa: F401
  File "/home/campy/Documents/Employment/IMS/solar/PVfit/venv/pvlib_0p10p4/lib/python3.11/site-packages/pvlib/ivtools/__init__.py", line 7, in <module>
    from pvlib.ivtools import sde, sdm, utils  # noqa: F401
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/campy/Documents/Employment/IMS/solar/PVfit/venv/pvlib_0p10p4/lib/python3.11/site-packages/pvlib/ivtools/sdm.py", line 15, in <module>
    from pvlib.pvsystem import calcparams_pvsyst, singlediode, v_from_i
  File "/home/campy/Documents/Employment/IMS/solar/PVfit/venv/pvlib_0p10p4/lib/python3.11/site-packages/pvlib/pvsystem.py", line 22, in <module>
    from pvlib import (atmosphere, iam, inverter, irradiance,
  File "/home/campy/Documents/Employment/IMS/solar/PVfit/venv/pvlib_0p10p4/lib/python3.11/site-packages/pvlib/singlediode.py", line 61, in <module>
    NsVbi=np.Inf, breakdown_factor=0., breakdown_voltage=-5.5,
          ^^^^^^
  File "/home/campy/Documents/Employment/IMS/solar/PVfit/venv/pvlib_0p10p4/lib/python3.11/site-packages/numpy/__init__.py", line 397, in __getattr__
    raise AttributeError(
AttributeError: `np.Inf` was removed in the NumPy 2.0 release. Use `np.inf` instead.. Did you mean: 'inf'?

@markcampanelli support for Numpy 2.0 was given in v0.10.5. https://pvlib-python.readthedocs.io/en/stable/whatsnew.html#v0-10-5-may-6-2024 , see Bug Fixes section.

Understood. Just putting a finer point on a previously working version
install breaking anew.
Mark Campanelli
LinkedIn https://www.linkedin.com/in/markcampanelli/
Intelligent Measurement Systems LLC
https://intelligentmeasurementsystems.com
Try PVfit https://pvfit.app today!
On Mon, Jun 24, 2024 at 08:26 Echedey Luis @.***> wrote:

@markcampanelli https://github.com/markcampanelli support for Numpy 2.0
was given in v0.10.5.
https://pvlib-python.readthedocs.io/en/stable/whatsnew.html#v0-10-5-may-6-2024
, see Bug Fixes section.
—
Reply to this email directly, view it on GitHub
https://github.com/pvlib/pvlib-python/issues/2051#issuecomment-2186713594,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAISX43GMFMK4GJSSOKENILZJAUCVAVCNFSM6AAAAABHSQ2XQSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBWG4YTGNJZGQ
.
You are receiving this because you were mentioned.Message ID:
@.***>


Develop a protocol for communications between E2E runner and the browser

Currently, our very simplistic protocol allows to just report success.
We need to report the following events:

the wallet requested access confirmation
the wallet requested a signature
the contract ended successfully
the contract failed
arbitrary user messages

data BrowserEvent a
  = ConfirmAccess
  | Sign
  | Success
  | Failure String
  | Message a

Add "hooks" to sign, access confirmation, success and failure events to allow triggering messages between CTL and the browser.

type Hooks =
  { beforeSign :: Maybe (Effect Unit)
  , beforeInit :: Maybe (Effect Unit)
  , onSuccess :: Maybe (Foreign -> Effect Unit)
  , onError :: Maybe (Error -> Effect Unit)
  }

I decided to skip adding Message and the type parameter to BrowserEvent, because there're no specific product requirements for user-specified messages with payloads.


[VC] fix data race when there are multi sa secret

data race:
suppose there are multi sa secret both in super and tenant. we first get sa secret according to sa,
and using the first one to mutate pod spec, we may missmatch if we got the wrong one this pod used.
vSecret is not recognize by super master.
whats more, we enable user to mount sa secret manually, so we also mutate these secret mount.
this solution find pSecret according to the pod spec mount.

/lgtm
/approve


Getting error while installing pod

I follow the installation steps but getting below error while running pod install command

[!] Invalid Podfile file: undefined method `enable_user_defined_build_types!' for #Pod::Podfile:0x00007fad70382fc8.

Could be this issue https://github.com/joncardasis/cocoapods-user-defined-build-types/issues/3

@KishanVyas You'll get this error if you haven't installed and imported the plugin.
gem install cocoapods-user-defined-build-types

Podfile
plugin 'cocoapods-user-defined-build-types'

enable_user_defined_build_types!

target 'NiftyIOSApp' do
  ...
  pod 'Alamofire'
  pod "SwiftyJSON", :build_type => :dynamic_framework
end

Add to your Gemfile so it works for CI/CD tools
gem 'cocoapods-user-defined-build-types'
Then just do with bundle install

I'm solved this issue here

Add to your Gemfile so it works for CI/CD tools gem 'cocoapods-user-defined-build-types'
Then just do with bundle install

This solution works for me


coordinates saved in zarr are saved in top-left format but xarray interprets them as center-pixel

see title.
either we shift the coord values to represent the center of the pixel or idk

This means for users:

the data and coordinates are correct, its just that coordinates for a pixel represent the top-left corner, as is often standard
if you load with xarray, then xarray shows incorrect coords that are shifted by PIXEL_SIZE/2 from the true coordinates


Provide utils from purescript-globals instead of FFI

As of 4.1.0, purescript-globals provides the decodeURIComponent and unsafeEncodeURIComponent functions that correspond to those exported from HTTPure.Utils. With this change, HTTPure.Utils will re-export those functions and no longer require the FFI.

Don't worry about CI, it's temporarily broken...

Since this requires a custom package set, how will this affect folks who try to use httpure from bower?
Would it be better to hold off on this until I get a chance to upgrade purescript and dependencies(see #143 )?

@nsaunders do you have any need to cut a release to include this?

@cprussin thanks for asking, but I have no immediate need for a release. I just submitted this because of the new additions to purescript-globals and thought I could help you rip out some code.    😁


Added October CMS to CMS plugins section

Our ResponsiveImages plugin for October CMS is using your great library to convert images to WebP and serve them on-the-fly.
I have added a link to the plugin to the README.

thanks.
And I am very excited to have a new CMS on the list :)


Publishing Bot Composer v1.4.0 Project throwing different kind of Issues

Hi Team,
When we are trying to publish the Bot Composer to Azure, it is throwing 2 kind of errors.
1.A popup says to raise issue in github.
2.As page wrong as shown below. Same while opened in desktop or using localhost:5000/

@LakshummaC,  do you get any error in the provisioning step or just this above error in the publish step after importing the Azure resources? Also, does this happen to a specific bot? Do you get any additional error on the dev console?

@LakshummaC, do you get any error in the provisioning step or just this above error in the publish step after importing the Azure resources? Also, does this happen to a specific bot? Do you get any additional error on the dev console?

Hi Anusha,
Initially with localhost:5000 and then opened our designed Bot and tried publishing. Till Import from existing resource it worked fine for the first time. After a min we started getting above screenshot error.
There is no additional error on dev console and browser goes to not responding stage and throws above snapshot error.

After multiple retries in local desktop Bot composer it is throwing below error:

{
"name": "",
"environment": "dev",
"runtimeIdentifier": "win-x64",
"resourceGroup": "",
"botName": "",
"subscriptionId": "",
"region": "",
"settings": {
"applicationInsights": {
"InstrumentationKey": ""
},
"cosmosDb": {
"cosmosDBEndpoint": "",
"authKey": "",
"databaseId": "botstate-db",
"containerId": "botstate-container"
},
"blobStorage": {
"connectionString": "",
"container": ""
},
"luis": {
"authoringKey": "",
"authoringEndpoint": "",
"endpointKey": "",
"endpoint": "",
"region": "westus"
},
"qna": {
"subscriptionKey": "",
"endpoint": ""
},
"MicrosoftAppId": "",
"MicrosoftAppPassword": ""
}
}
on a safe note, can you explain the starting 5 fields what I have to put?

any update on this? I need it ASAP

Issue Resolved, you can close this ticket


Fs0025

Summary
Describe your changes here.
Fixes #Issue_Number (if available)

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: brettrowberry sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

I don't know what to do about Snippets 5000 / build (pull_request):
Project missing. A project (and optionally a solution file) must be in this directory or one of the parent directories to validate and build this code.
I don't think I should have a project or solution file and as far as I can tell, no other .fs files do.

@brettrowberry Thanks for the contribution. It's a recent change to CI; any new additions of files need to be buildable. What happens if you rename it to .fsx?
If that fails, you can just new up a quick project file and include it. It's annoying but it does at least ensure that things compile before they go live.

@cartermp That looks like it worked

That means I need to update the links in my .md file. I should probably update the script generator and convert Chet’s .fs to .fsx.

@brettrowberry Yep, that seems like a reasonable thing. Thanks! If you go ahead and do that then this should be good to merge.

@cartermp I think I'm done

Did I break something or is this a super busy repository?

I think that Bob's Machine Under The Desk (Azure?) is a little busy at the moment

Closing and reopening was a stroke of brilliance!

When does it go live?
https://docs.microsoft.com/en-us/dotnet/fsharp/language-reference/compiler-messages/

Usually within a day or two.


Why does computed change trigger filters?

Version
2.6.8
Reproduction link
https://codepen.io/GreenMashimaro/pen/qvYNGE?editors=1111
Steps to reproduce
Create a new timer, add a computed, and the computed change triggers the filter change
What is expected?
filters:flt. It will only be triggered once
My understanding is that filters will only be triggered once, because there is no correlation between myself. step_count and cpt_val.
What is actually happening?
Computed change trigger filters.

My understanding is that filters will only be triggered once, because there is no correlation between myself. step_count and cpt_val.

You are mistaken. The whole template is re-evaluated whenever a dependency changes, which means the filter runs each time as well.


feat: select measurement on drop

closes : https://github.com/zakodium-oss/analysis-ui-components/issues/250

Codecov Report
Base: 91.59% // Head: 91.59% // No change to project coverage :thumbsup:

Coverage data is based on head (e4f9b5d) compared to base (0a17957).
Patch has no changes to coverable lines.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #268   +/-   ##
=======================================
  Coverage   91.59%   91.59%
=======================================
  Files           9        9
  Lines         583      583
  Branches      100      100
=======================================
  Hits          534      534
  Misses         49       49

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Auto select measurement should only be done if no measurement is currently selected.
In the current version there is a weird behaviour. If you drag / drop ir2, mass and uv there is auto select of ir2. This is ok.
However after if you add one more 'ir' suddently there is the UV that is selected.

Using as rule that you auto-select only if nothing is selected you will not have this issue.

Auto select measurement should only be done if no measurement is currently selected.

Fixed, could you please try now.

Yes the functionality is ok now.


[Suggestion] Having the blacklist be a separate file

Currently if a pack dev were to include the blacklist in a pack it'd overwrite the user's data every update. I'd like to suggest having the blacklist in a separate file

Can it maybe grab the blacklist from JEI?

index customization can be done via resource pack. please see the wiki for more information.

Kat is correct, the local edit mode hidden stacks are per user, but can be copied to a resource pack file shipped with modpacks using common data scripters/loaders like KubeJS. That is to say this suggestion already exists.


chore(TDI-41415): get reference connection api version

What is the current behavior? (You can also link to an open issue here)
Wrong API version if referenced connection is used, every time it is 2018. It breaks work with custom fields.
What is the new behavior?
Fix get API version if referenced connection is used.
Please check if the PR fulfills these requirements

[x] The commit message follows Talend standard
[ ] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features) ?
[ ] The code coverage on new code >75%
[x] The new code does not introduce new technical issues (sonar / eslint)

What kind of change does this PR introduce?

[x] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build / CI related changes
[ ] Other... Please describe:

Does this PR introduce a breaking change?

[ ] Yes
[x] No

If this PR contains a breaking change, please describe the impact and migration path for existing applications: ...
Other information:

Pull request test result: failure :x:


Work on appending info to a temp JSON before being shifted to a main master json file (for caching purposes)

Can successfully append to file, now we need to save parameters and events as one thing


disable SSL verification in Spring Vault

It will help in quick unit testing

As of now, getting following exception:-
Cannot login using org.springframework.web.client.ResourceAccessException: I/O error on POST request for : PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target; nested exception is javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
org.springframework.vault.authentication.VaultLoginException: Cannot login using org.springframework.web.client.ResourceAccessException: I/O error on POST request for : PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target; nested exception is javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

RestTemplate can give any of the below error if SSL certificate of the target host is not valid:

PKIX path building failed: sun.security.provider.certpath
PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
javax.net.ssl.SSLHandshakeException: PKIX path building failed

Please use plaintext. We do not plan to disable SSL verification.


nixos/sourcehut: updates, fixes, hardening

This PR is still baking a bit, but is ready for testing and reviewing.
Motivation for this change
Be able to install and use latest https://sr.ht/~sircmpwn/sourcehut/ .
Beware, this is a huge change, and it may or may not cause troubles for people using the previous NixOS module. Which was no longer functional when I started this PR (due to a wrong --loglevel=info passed to celery instead of --loglevel INFO).

[X] Add a working metasrht-manager into systemPackages.
[X] Repurpose the services option to only serve as selecting what (local or remote) services are displayed, and use an enable option for each local service of sourcehut.
[X] Factorize configuration of Sourcehut services.
[X] Type-check and describe many settings.
[X] Configuring of credentials outside the Nix store.
[X] systemd hardening of Sourcehut services.
[X] Use systemd timers instead of cron.
[X] Use a dedicated (and thus hardenable) gitsrht-fcgiwrap.service instead of services.fcgiwrap.
[ ] Manual check services of timers work as expected.
[X] Only metasrht.service has access to the stripe-secret-key now.
[X] Only enable docker when needed.
[X] Use mkEnableOption.
[X] Manual check webhooks are working as expected.
[X] Per service config.ini.
[X] Better working integration into sshd.
[X] Now configuring of local nginx, postfix, postgresql, and redis is made optional.
[X] Better nginx integration.
[X] Fix celery calls by specifying --loglevel INFO instead of  --loglevel=info.
[X] Configure celery to use the eventlet concurrent networking library.
[X] Configure celery services to share a single Redis database.
[ ] Manual testing of buildssrht. I don't use it, but the service and the worker both run.
[ ] Manual testing of dispatchsrht. I don't use it.
[X] Manual testing of gitsrht.
[ ] Manual testing of hgsrht. I don't use it.
[X] Manual testing of hubsrht.
[X] Manual testing of listssrht.
[X] Manual testing of metasrht.
[X] Manual testing of mansrht.
[ ] Manual testing of pagessrht. The service is running, but I don't know how I should put content in it.
[X] Manual testing of pastesrht.
[X] Manual testing of todosrht.
[X] Update the meta.doc. But there may be more to do.
[X] Add myself in maintainers.
[ ] Add a release entry. I'm exhausted for now.

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[X] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
21.11 Release Notes (or backporting 21.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[X] Fits CONTRIBUTING.md.

Example of config:
{ pkgs, lib, config, ... }:
let
  inherit (config.services) sourcehut;
  inherit (config.users) users groups;
  # This `gnupg` module is from https://github.com/NixOS/nixpkgs/pull/93659
  # but use whatever secret scheme you prefer.
  inherit (config.security) gnupg;
  domain = "example.com";
in
{
services.sourcehut = {
  enable = true;
  listenAddress = "localhost";

  # Enable locally some Sourcehut services
  builds = {
    enable = true;
    enableWorker = true;
    images.nixos.unstable.x86_64 =
      let
        systemConfig = { pkgs, ... }: {
          # passwordless ssh server
          services.openssh = {
            enable = true;
            permitRootLogin = "yes";
            extraConfig = "PermitEmptyPasswords yes";
          };

          users = {
            mutableUsers = false;
            # build user
            extraUsers."build" = {
              isNormalUser = true;
              uid = 1000;
              extraGroups = [ "wheel" ];
              password = "";
            };
            users.root.password = "";
          };

          security.sudo.wheelNeedsPassword = false;
          nix.trustedUsers = [ "root" "build" ];
          documentation.nixos.enable = false;

          # builds.sr.ht-image-specific network settings
          networking = {
            hostName = "build";
            dhcpcd.enable = false;
            defaultGateway.address = "<IP_ADDRESS>";
            usePredictableInterfaceNames = false; # so that we just get eth0 and not some weird id
            interfaces."eth0".ipv4.addresses = [{
              address = "<IP_ADDRESS>";
              prefixLength = 25;
            }];
            enableIPv6 = false;
            nameservers = [
              # OpenNIC anycast
              "<IP_ADDRESS>"
              "<IP_ADDRESS>"
              # Google as a fallback :(
              "<IP_ADDRESS>"
            ];
            firewall.allowedTCPPorts = [ 22 ]; # allow ssh
          };

          environment.systemPackages = [
            pkgs.gitMinimal
            #pkgs.mercurial
            pkgs.curl
            pkgs.gnupg
          ];
        };
        qemuConfig = { pkgs, ... }: {
          imports = [ systemConfig ];
          fileSystems."/".device = "/dev/disk/by-label/nixos";
          boot.initrd.availableKernelModules = [
            "ahci"
            "ehci_pci"
            "sd_mod"
            "usb_storage"
            "usbhid"
            "virtio_balloon"
            "virtio_blk"
            "virtio_pci"
            "virtio_ring"
            "xhci_pci"
          ];
          boot.loader = {
            grub = {
              version = 2;
              device = "/dev/vda";
            };
            timeout = 0;
          };
        };
        config = (import (pkgs.path + "/nixos/lib/eval-config.nix") {
          inherit pkgs; modules = [ qemuConfig ];
          system = "x86_64-linux";
        }).config;
      in
      import (pkgs.path + "/nixos/lib/make-disk-image.nix") {
        inherit pkgs lib config;
        diskSize = 16000;
        format = "qcow2-compressed";
        contents = [
          { source = pkgs.writeText "gitconfig" ''
              [user]
                name = builds.sr.ht
                email =<EMAIL_ADDRESS>            '';
            target = "/home/build/.gitconfig";
            user = "build";
            group = "users";
            mode = "644";
          }
        ];
      };
  };
  #dispatch.enable = true;
  git.enable = true;
  hub.enable = true;
  lists.enable = true;
  meta.enable = true;
  man.enable = true;
  pages.enable = true;
  paste.enable = true;
  todo.enable = true;

  # Integration into local services
  postgresql.enable = true;
  postfix.enable = true;
  redis.enable = true;
  nginx.enable = true;

  settings = {
    "sr.ht" = {
      environment = "production";
      global-domain = domain;
      origin = "https://${domain}";
      owner-email = "root+srht@${domain}";
      owner-name = "Owner Name";
      site-blurb = "blurb";
      site-info = "https://${domain}";
      site-name = "Site Name";
      # nix shell nixpkgs#sourcehut.coresrht -c srht-keygen network
      network-key = gnupg.secrets."sourcehut/network-key".path;
      # nix shell nixpkgs#sourcehut.coresrht -c srht-keygen service
      service-key = gnupg.secrets."sourcehut/service-key".path;
    };

    # This is only needed for the pages.sr.ht service.
    # But may also be used to offer release storage in git.sr.ht,
    # and build artifact storage in builds.sr.ht
    objects = {
      s3-upstream = "localhost";
      s3-access-key = "12345";
      s3-secret-key = pkgs.writeText "s3-secret-key" "12345678";
    };

    # oauth-client-id of services are automatically authorized when the `metasrht-api.service` starts
    # TODO: I think it's better if oauth-client-id are different from one another.
    "builds.sr.ht" = {
      oauth-client-secret = gnupg.secrets."sourcehut/oauth-client-secret".path;
      oauth-client-id = "299db9f9c2013170";
    };
    "dispatch.sr.ht" = {
      oauth-client-secret = gnupg.secrets."sourcehut/oauth-client-secret".path;
      oauth-client-id = "299db9f9c2013170";
    };
    "hub.sr.ht" = {
      oauth-client-secret = gnupg.secrets."sourcehut/oauth-client-secret".path;
      oauth-client-id = "299db9f9c2013170";
    };
    "git.sr.ht" = {
      outgoing-domain = "http://git.${domain}";
      oauth-client-secret = gnupg.secrets."sourcehut/oauth-client-secret".path;
      oauth-client-id = "299db9f9c2013170";
    };
    "lists.sr.ht" = {
      oauth-client-secret = gnupg.secrets."sourcehut/oauth-client-secret".path;
      oauth-client-id = "299db9f9c2013170";
    };
    "lists.sr.ht::worker" = {
    };
    "man.sr.ht" = {
      oauth-client-secret = gnupg.secrets."sourcehut/oauth-client-secret".path;
      oauth-client-id = "299db9f9c2013170";
    };
    "meta.sr.ht" = {
    };
    "meta.sr.ht::settings" = {
      onboarding-redirect = "https://meta.${domain}";
      registration = true;
      # Or register users with something like:
      # metasrht-manageuser -t admin -e<EMAIL_ADDRESS>john
    };
    "meta.sr.ht::api" = {
    };
    "pages.sr.ht" = {
      oauth-client-secret = gnupg.secrets."sourcehut/oauth-client-secret".path;
      oauth-client-id = "299db9f9c2013170";
      s3-bucket = "pagesbuck";
    };
    "paste.sr.ht" = {
      oauth-client-secret = gnupg.secrets."sourcehut/oauth-client-secret".path;
      oauth-client-id = "299db9f9c2013170";
    };
    "todo.sr.ht" = {
      oauth-client-secret = gnupg.secrets."sourcehut/oauth-client-secret".path;
      oauth-client-id = "299db9f9c2013170";
    };

    # nix shell nixpkgs#sourcehut.coresrht -c srht-keygen webhook
    webhooks.private-key = gnupg.secrets."sourcehut/webhook-key".path;

    mail = {
      smtp-host = "localhost";
      smtp-port = 25;
      smtp-user = null;
      smtp-password = null;
      smtp-from =<EMAIL_ADDRESS>      error-to =<EMAIL_ADDRESS>      error-from =<EMAIL_ADDRESS>      pgp-privkey = null;
      pgp-pubkey = null;
      pgp-key-id = null;
    };
  };
};

# Minio provides an S3 bucket, it is only needed for the `pages` service.
services.minio = {
  enable = true;
  accessKey = "12345";
  secretKey = "12345678";
  #region = "";
  browser = true;
};
environment.systemPackages = [ pkgs.minio-client ];
}

Note that other settings are needed for enabling and properly configuring local nginx, postfix, postgresql, and redis services, but I don't have the energy to put a minimal example here right now.
Beware that origin= settings are recorded into the SQL databases and not updated if you later change them (eg. by switching from HTTP to HTTPS). See for instance:
meta.sr.ht=# select * from user_webhook_subscription;
 id |          created           |                          url                           |           events           | user_id | token_id
----+----------------------------+--------------------------------------------------------+----------------------------+---------+----------
  1 | 2021-08-14 01:37:05.252564 | http://git.sourcephile.wg/oauth/webhook/profile-update | profile:update             |       1 |        1
  2 | 2021-08-14 01:37:05.288726 | http://git.sourcephile.wg/webhook/notify/keys          | ssh-key:remove,ssh-key:add |       1 |        1
(2 rows)

Note that AFAIK those ssh-key:remove,ssh-key:add user webhooks will ultimately be phased out of sourcehut and authorized SSH public keys no longer cached into a Redis database.

This is amazing... it will take me a bit of time to review and test.

Should the restrictions allow the pre-start services to write to /var/lib/sourcehut/*srht/*.oauth?

This might have been a problem with my own setup, but I had to wipe the entire previous installation, DB+state. Note: clear cookies after changing secrets.
When pushing to git. Should check if the warnings/permissions for logs need to be fixed.
2021/08/15 02:58:24 repo ID 1; name 'thing'; owner ID 1; owner name 'tomberek'; visibility 'public'; pusher type 'admin'; pusher suspension notice ''; access grant ''
2021/08/15 02:58:24 Executing command: [git-receive-pack /var/lib/sourcehut/gitsrht/repos/~tomberek/thing]
Enumerating objects: 3, done.
Counting objects: 100% (3/3), done.
Writing objects: 100% (3/3), 203 bytes | 203.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
remote: 2021/08/15 02:58:24 Warning: unable to open log file: open /var/log/gitsrht-update-hook: permission denied (using stderr instead)
remote: hooks/pre-receive 2021/08/15 02:58:24 [hooks/pre-receive]
remote: hooks/pre-receive 2021/08/15 02:58:24 Running pre-receive for push f04eb06d-7eb0-4289-a2fe-cb1a9ae9d915
remote: 2021/08/15 02:58:24 Warning: unable to open log file: open /var/log/gitsrht-update-hook: permission denied (using stderr instead)
remote: hooks/update 2021/08/15 02:58:24 [hooks/update refs/heads/master<PHONE_NUMBER>000000000000000000000000000000 a98c53622b5ad4f79e4158cd7e6aabc13f510c73]
remote: hooks/update 2021/08/15 02:58:24 Running update for push f04eb06d-7eb0-4289-a2fe-cb1a9ae9d915
remote: 2021/08/15 02:58:24 Warning: unable to open log file: open /var/log/gitsrht-update-hook: permission denied (using stderr instead)
remote: hooks/post-update 2021/08/15 02:58:24 [hooks/post-update refs/heads/master]
remote: hooks/post-update 2021/08/15 02:58:24 Running post-update for push f04eb06d-7eb0-4289-a2fe-cb1a9ae9d915
remote: hooks/post-update 2021/08/15 02:58:25 Skipping stage 3, no work
To git.srht.tomberek.info:~tomberek/thing
 * [new branch]      master -> master
Branch 'master' set up to track remote branch 'master' from 'origin'.

Result of nixpkgs-review pr 133984 at 749d922f run on x86_64-linux 1

1 package failed to build:

nixos-install-tools

12 packages built successfully:

sourcehut.buildsrht
sourcehut.coresrht
sourcehut.dispatchsrht
sourcehut.gitsrht
sourcehut.hgsrht
sourcehut.hubsrht
sourcehut.listssrht
sourcehut.mansrht
sourcehut.metasrht
sourcehut.pagessrht
sourcehut.pastesrht
sourcehut.todosrht

11 suggestions:

warning: no-python-tests
Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
Near pkgs/applications/version-management/sourcehut/builds.nix:61:0:
   |
61 |     description = "Continuous integration service for the sr.ht network";
   | ^

warning: unused-argument
Unused argument: writeText.
Near pkgs/applications/version-management/sourcehut/core.nix:28:3:
   |
28 | , writeText
   |   ^

warning: no-python-tests
Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
Near pkgs/applications/version-management/sourcehut/git.nix:77:0:
   |
77 |     description = "Git repository hosting service for the sr.ht network";
   | ^

warning: no-python-tests
Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
Near pkgs/applications/version-management/sourcehut/dispatch.nix:36:0:
   |
36 |     description = "Task dispatcher and service integration tool for the sr.ht network";
   | ^

warning: no-python-tests
Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
Near pkgs/applications/version-management/sourcehut/man.nix:34:0:
   |
34 |     description = "Wiki service for the sr.ht network";
   | ^

warning: unclear-gpl
agpl3 is a deprecated license, please check if project uses agpl3Plus or agpl3Only and change meta.license accordingly.
Near pkgs/applications/version-management/sourcehut/pages.nix:34:5:
   |
34 |     license = licenses.agpl3;
   |     ^

warning: no-python-tests
Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
Near pkgs/applications/version-management/sourcehut/paste.nix:34:0:
   |
34 |     description = "Ad-hoc text file hosting service for the sr.ht network";
   | ^

warning: no-python-tests
Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
Near pkgs/applications/version-management/sourcehut/lists.nix:42:0:
   |
42 |     description = "Mailing list service for the sr.ht network";
   | ^

warning: no-python-tests
Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
Near pkgs/applications/version-management/sourcehut/meta.nix:71:0:
   |
71 |     description = "Account management service for the sr.ht network";
   | ^

warning: missing-patch-comment
Consider adding a comment explaining the purpose of this patch on the line preceeding.
Near pkgs/applications/version-management/sourcehut/core.nix:49:5:
   |
49 |     ./disable-npm-install.patch
   |     ^

warning: no-python-tests
Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
Near pkgs/applications/version-management/sourcehut/hg.nix:37:0:
   |
37 |     description = "Mercurial repository hosting service for the sr.ht network";
   | ^

Note that build failures may predate this PR, and could be nondeterministic or hardware dependent.
Please exercise your independent judgement. Does something look off? Please file an issue or reach out on IRC.

Should the restrictions allow the pre-start services to write to /var/lib/sourcehut/*srht/*.oauth?

@tomberek, AFAICS metasrht-api is the only service using those *.oauth files, owned by metasrht, so I don't see a problem here.
Thinking about it, I don't see the need for such caching, avoiding a few psql runs in preStart= is not worth the added complexity IMHO. I've removed the code for these *.oauth files.
I'm also making the psql command to use connexion-string so it will also work with a remote postgresql.

I had to rebase with master because of fbafeb7ad5dd6196fcc5d84264e1706653a62f81, this messes up the diff shown by GitHub, but nothing has changed except what I've mentioned here.

Pushed some more work on the buildsrht and buildsrht-worker front.
Those should now be configured. I've put in the demo in the intro of this PR @tomberek's example.

Pushed some more work to fix and harden the integration of git.sr.ht into nginx.

Pushed some work to enable a merge the Redis database of webhooks and other Celery workers into a shared database. And some more work to fix/improve the nginx integration.

# This is only needed for the pages service

FYI this is also used by git.sr.ht and builds.sr.ht to offer release storage and build artifact storage, respectively.

Pushed a few fix:

Fix default internal-ipnet=, according to https://lists.sr.ht/~sircmpwn/sr.ht-dev/patches/11682
Remove workaround removing X-Real-IP:, this is being fixed upstream
Create /var/lib/sourcehut/gitsrht/repos at gitsrht.service startup instead of failing services (including sshd, oops) because BindPaths= is currently not used with a failsafe -.

Pushed a revamping of services.redis to support per-application redis-server as it's supposed to be used since Redis databases are only mere key prefixes without any configuration or ACL of their own.
I'm trying push upstream some support for using Unix sockets with redis-server.
The rest of the diff is mostly options to configure Celery (the Python library used by Sourcehut to manage webhooks and other stuffs like listssrht-process), and a better integration of listssrht-lmtp into postfix (AFAIU some work will have to me done upstream to avoid backscattering).

[X] redis: make sure to disable TCP socket when there is no bind address.
[X] sourcehut: make nixosTest.sourcehut work again (other services will have to be tested to).

[X] Fix redis url for celery/kombu.
[X] Fix todosrht's integration into postfix.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-ready-for-review/3032/585

Having a hard time getting buildsrht working. Currently have a "panic: invalid redis URL scheme: unix" which presume is related to the recent Redis changes.

@tomberek, thanks for your review, I've pushed a v3 of the patches to Sourcehut including buildsrht's worker. Not yet able to test it above the building step though.

[X] Rebase after merge conflict due to bc3bca822a32fbbc73a9d55394991cef92dba3b9

[X] Fix mistake on naming a patch.

[X] Fix build error in redis module due to https://github.com/NixOS/nixpkgs/pull/133166

[X] Update to latest sourcehut releases, removing a merged patch and wrong cp in buildsrht.
[X] Remove buildsrht from scmsrht's propagatedBuildInputs. I guess others like redis should be removed too.

I'm having some issues building the gitsrht-dispatcher package:
gitsrht-dispatcher> unpacking sources
gitsrht-dispatcher> unpacking source archive /nix/store/dqi0ps96z9kisqmyzp2k3y0ip6ik8wdw-source
gitsrht-dispatcher> source root is source/gitsrht-dispatch
gitsrht-dispatcher> patching sources
gitsrht-dispatcher> applying patch /nix/store/srfgfamw4sb4jj40r0f2c85882j6qsp2-v3-0003-gitsrht-dispatch-add-support-for-supplementary-gr.patch
gitsrht-dispatcher> can't find file to patch at input line 15
gitsrht-dispatcher> Perhaps you used the wrong -p or --strip option?
gitsrht-dispatcher> The text leading up to this was:
gitsrht-dispatcher> --------------------------
gitsrht-dispatcher> |From fcbec39a406562c29dfcf7eeef6f284da28bc619 Mon Sep 17 00:00:00 2001
gitsrht-dispatcher> |From: Julien Moutinho<EMAIL_ADDRESS>gitsrht-dispatcher> |Date: Fri, 27 Aug 2021 17:42:33 +0200
gitsrht-dispatcher> |Subject: [PATCH git.sr.ht v3 3/3] gitsrht-dispatch: add support for
gitsrht-dispatcher> | supplementary groups
gitsrht-dispatcher> |
gitsrht-dispatcher> |---
gitsrht-dispatcher> | gitsrht-dispatch/main.go | 17 ++++++++++++++---
gitsrht-dispatcher> | 1 file changed, 14 insertions(+), 3 deletions(-)
gitsrht-dispatcher> |
gitsrht-dispatcher> |diff --git a/gitsrht-dispatch/main.go b/gitsrht-dispatch/main.go
gitsrht-dispatcher> |index d7aee14..5f17b75 100644
gitsrht-dispatcher> |--- a/gitsrht-dispatch/main.go
gitsrht-dispatcher> |+++ b/gitsrht-dispatch/main.go
gitsrht-dispatcher> --------------------------
gitsrht-dispatcher> File to patch:
gitsrht-dispatcher> Skip this patch? [y]
gitsrht-dispatcher> Skipping patch.
gitsrht-dispatcher> 3 out of 3 hunks ignored

I don't understand where the problem lies. The patch file looks fine to me...

I don't understand where the problem lies. The patch file looks fine to me...

@malte-v the patch is fine but buildGoModule's go-modules inherits patches without inheriting patchFlags, loosing the -p2.
I was unable to reproduce the problem at first because of the fixed-output-derivation being already in my Nix store from a time where the patch was not yet there (applying that patch does not change the vendorSha256 hash).

[X] Add a commit to fix buildGoModule, removing the need for passing patches and patchFlags through overrideModAttrs.

Rebase after 2ddc335e6f32b875e14ad9610101325b306a0add's treewide changes.

[X] Fix OnCalendar of gitsrht-periodic and hgsrht-periodic: "20min" was not a correct syntax there.
[X] Fix post-update-script integration: it's sshd.service which needs it, not gitsrht.service. Moreover, because we have to wrap gitsrht-update-hook to access sourcehut's config.ini (sic) the hooks/stage-3 needs a hack (exporting a custom environment variable STAGE3) because bash does not preserve $0 when used as a script, example:

$ echo -e '#!/usr/bin/env bash\necho $0' | install -m 750 /dev/stdin test.sh
$ bash -c 'exec -a toto ./test.sh'
/home/julm/test.sh

In that example, one would expect $0 to be toto instead of /home/julm/test.sh.
I've also increased the tracing (set -x) of those git hooks to help future debugging.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-ready-for-review/3032/628

Hi. After trying this update to nixpkgs i'm still getting a ValueError: 'name' may not contain a dot '.' character. from sourcehut.
@ddevault provide some help, he stated it was because flask was updated but sourcehut was not

Hi. After trying this update to nixpkgs i'm still getting a ValueError: 'name' may not contain a dot '.' character. from sourcehut.

@zkjanus, could you show the relevant parts of your Nix config (including the commit revision of Nixpkgs you're using) and what you do to reach this error?

@ju1m I started with the previous installation state before this pull request with already existing users and repos, with a system.stateVersion = 20.09
Is there no upgrade path from the old install to get this working?  I basically followed the above example with the same services i used before; there were a few small additions like todo/paste having oauth credentials that I didn't have before.
I rebased your commits on top of nixos-unstable.

https://lists.sr.ht/~sircmpwn/sr.ht-dev/<20210527000914.170829-1-admin%40xenrox.net>#<CBNLDA78STJ5.2JCL50DHQIRE0@taiga> the error i am seeing pertains to this. After the switch to flask 2.0.

Is there no upgrade path from the old install to get this working?

@zkjanus, I haven't tried this myself so you're pioneering here, user profiles and databases should upgrade (that's actually what is about to happen in your log), but some things may not for pre-existing repositories, like the /var/lib/sourcehut/gitsrht/repos/~*/*/hooks/{pre-receive,update,post-update} I mentioned earlier. Thinking about it, I guess I should put this fix in gitsrht.service's preStart.

I understand that you were using an older version of this PR and that upgrading to the latest fixed the Flask error.
Concerning the managing of secrets, you now have to pass them using absolute paths, those should be outside the Nix store, but for testing they could be in, as shown in the intro example, by using something like pkgs.writeText "s3-secret-key" "12345678". Note that when the *srht.services are running/starting you'll find the resulting config.ini in paths like /run/sourcehut/metasrht/config.ini and thus may have a short time window to grep '.*'-key= /run/sourcehut/metasrht/config.ini, otherwise you may temporarily:

systemd.services.metasrht.preStart = lib.mkForce ''
  sleep 42
  exit 1
'';

Do not hesitate to post here again logs of the errors you hit, so that I may help you and that it may help others later.

[X] Fix /var/lib/sourcehut/gitsrht/repos/~*/*/hooks/{pre-receive,update,post-update} for system.stateVersion < 22.05.

[X] Add linting from @SuperSandro2000
[X] Run pkgs/applications/version-management/sourcehut/update.sh. Note that this script requires a NIX_PATH= envvar to find nixpkgs which may not be the case for those configuring their system only with flake.nix. A more portable setup could be to use a passthru.updateScript. In the meantime one could:

$ nix shell -f . git mercurial common-updater-scripts
$ bash pkgs/applications/version-management/sourcehut/update.sh

Hello again,
I am now trying to install the services from scratch.
Nov 26 04:48:57 services gunicorn[1184249]: [2021-11-26 04:48:57 +0100] [1184249] [INFO] Starting gunicorn 20.1.0
Nov 26 04:48:57 services gunicorn[1184249]: [2021-11-26 04:48:57 +0100] [1184249] [INFO] Listening at: http://<IP_ADDRESS>:5000 (1184249)
Nov 26 04:48:57 services gunicorn[1184249]: [2021-11-26 04:48:57 +0100] [1184249] [INFO] Using worker: sync
Nov 26 04:48:57 services gunicorn[1184296]: [2021-11-26 04:48:57 +0100] [1184296] [INFO] Booting worker with pid: 1184296
Nov 26 04:48:59 services gunicorn[1184296]: [2021-11-26 04:48:59 +0100] [1184296] [ERROR] Exception in worker process
Nov 26 04:48:59 services gunicorn[1184296]: Traceback (most recent call last):
Nov 26 04:48:59 services gunicorn[1184296]:   File "/nix/store/1n56ps7kjlg8fcy0kfi16f6y44qxwnfi-python3-3.9.6-env/lib/python3.9/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
Nov 26 04:48:59 services gunicorn[1184296]:     worker.init_process()
Nov 26 04:48:59 services gunicorn[1184296]:   File "/nix/store/1n56ps7kjlg8fcy0kfi16f6y44qxwnfi-python3-3.9.6-env/lib/python3.9/site-packages/gunicorn/workers/base.py", line 134, in init_process
Nov 26 04:48:59 services gunicorn[1184296]:     self.load_wsgi()
Nov 26 04:48:59 services gunicorn[1184296]:   File "/nix/store/1n56ps7kjlg8fcy0kfi16f6y44qxwnfi-python3-3.9.6-env/lib/python3.9/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
Nov 26 04:48:59 services gunicorn[1184296]:     self.wsgi = self.app.wsgi()
Nov 26 04:48:59 services gunicorn[1184296]:   File "/nix/store/1n56ps7kjlg8fcy0kfi16f6y44qxwnfi-python3-3.9.6-env/lib/python3.9/site-packages/gunicorn/app/base.py", line 67, in wsgi
Nov 26 04:48:59 services gunicorn[1184296]:     self.callable = self.load()
Nov 26 04:48:59 services gunicorn[1184296]:   File "/nix/store/1n56ps7kjlg8fcy0kfi16f6y44qxwnfi-python3-3.9.6-env/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
Nov 26 04:48:59 services gunicorn[1184296]:     return self.load_wsgiapp()
Nov 26 04:48:59 services gunicorn[1184296]:   File "/nix/store/1n56ps7kjlg8fcy0kfi16f6y44qxwnfi-python3-3.9.6-env/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
Nov 26 04:48:59 services gunicorn[1184296]:     return util.import_app(self.app_uri)
Nov 26 04:48:59 services gunicorn[1184296]:   File "/nix/store/1n56ps7kjlg8fcy0kfi16f6y44qxwnfi-python3-3.9.6-env/lib/python3.9/site-packages/gunicorn/util.py", line 359, in import_app
Nov 26 04:48:59 services gunicorn[1184296]:     mod = importlib.import_module(module)
Nov 26 04:48:59 services gunicorn[1184296]:   File "/nix/store/97w52ckcjnfiz89h3lh7zf1kysgfm2s8-python3-3.9.6/lib/python3.9/importlib/__init__.py", line 127, in import_module
Nov 26 04:48:59 services gunicorn[1184296]:     return _bootstrap._gcd_import(name[level:], package, level)
Nov 26 04:48:59 services gunicorn[1184296]:   File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
Nov 26 04:48:59 services gunicorn[1184296]:   File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
Nov 26 04:48:59 services gunicorn[1184296]:   File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
Nov 26 04:48:59 services gunicorn[1184296]:   File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
Nov 26 04:48:59 services gunicorn[1184296]:   File "<frozen importlib._bootstrap_external>", line 850, in exec_module
Nov 26 04:48:59 services gunicorn[1184296]:   File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
Nov 26 04:48:59 services gunicorn[1184296]:   File "/nix/store/1n56ps7kjlg8fcy0kfi16f6y44qxwnfi-python3-3.9.6-env/lib/python3.9/site-packages/metasrht/app.py", line 59, in <module>
Nov 26 04:48:59 services gunicorn[1184296]:     app = MetaApp()
Nov 26 04:48:59 services gunicorn[1184296]:   File "/nix/store/1n56ps7kjlg8fcy0kfi16f6y44qxwnfi-python3-3.9.6-env/lib/python3.9/site-packages/metasrht/app.py", line 24, in __init__
Nov 26 04:48:59 services gunicorn[1184296]:     from metasrht.blueprints.oauth2 import oauth2
Nov 26 04:48:59 services gunicorn[1184296]:   File "/nix/store/1n56ps7kjlg8fcy0kfi16f6y44qxwnfi-python3-3.9.6-env/lib/python3.9/site-packages/metasrht/blueprints/oauth2.py", line 32, in <module>
Nov 26 04:48:59 services gunicorn[1184296]:     "scopes": r.json()["scopes"],
Nov 26 04:48:59 services gunicorn[1184296]:   File "/nix/store/1n56ps7kjlg8fcy0kfi16f6y44qxwnfi-python3-3.9.6-env/lib/python3.9/site-packages/requests/models.py", line 910, in json
Nov 26 04:48:59 services gunicorn[1184296]:     return complexjson.loads(self.text, **kwargs)
Nov 26 04:48:59 services gunicorn[1184296]:   File "/nix/store/97w52ckcjnfiz89h3lh7zf1kysgfm2s8-python3-3.9.6/lib/python3.9/json/__init__.py", line 346, in loads
Nov 26 04:48:59 services gunicorn[1184296]:     return _default_decoder.decode(s)
Nov 26 04:48:59 services gunicorn[1184296]:   File "/nix/store/97w52ckcjnfiz89h3lh7zf1kysgfm2s8-python3-3.9.6/lib/python3.9/json/decoder.py", line 337, in decode
Nov 26 04:48:59 services gunicorn[1184296]:     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
Nov 26 04:48:59 services gunicorn[1184296]:   File "/nix/store/97w52ckcjnfiz89h3lh7zf1kysgfm2s8-python3-3.9.6/lib/python3.9/json/decoder.py", line 355, in raw_decode
Nov 26 04:48:59 services gunicorn[1184296]:     raise JSONDecodeError("Expecting value", s, err.value) from None
Nov 26 04:48:59 services gunicorn[1184296]: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Nov 26 04:48:59 services gunicorn[1184296]: [2021-11-26 04:48:59 +0100] [1184296] [INFO] Worker exiting (pid: 1184296)
Nov 26 04:48:59 services gunicorn[1184296]: Discovering APIs...
Nov 26 04:48:59 services gunicorn[1184296]:   Found man.sr.ht
Nov 26 04:48:59 services gunicorn[1184249]: [2021-11-26 04:48:59 +0100] [1184249] [INFO] Shutting down: Master
Nov 26 04:48:59 services gunicorn[1184249]: [2021-11-26 04:48:59 +0100] [1184249] [INFO] Reason: Worker failed to boot.
Nov 26 04:48:59 services systemd[1]: metasrht.service: Main process exited, code=exited, status=3/NOTIMPLEMENTED

here's my config:

      services.sourcehut = {
        enable = true;
        listenAddress = "<IP_ADDRESS>";

        # Enable locally some Sourcehut services
        builds = {
          enable = true;
          enableWorker = true;
          images.nixos.unstable.x86_64 =
            let
              systemConfig = { pkgs, ... }: {
                # passwordless ssh server
                services.openssh = {
                  enable = true;
                  permitRootLogin = "yes";
                  extraConfig = "PermitEmptyPasswords yes";
                };

                users = {
                  mutableUsers = false;
                  # build user
                  extraUsers."build" = {
                    isNormalUser = true;
                    uid = 1000;
                    extraGroups = [ "wheel" ];
                    password = "";
                  };
                  users.root.password = "";
                };

                security.sudo.wheelNeedsPassword = false;
                nix.trustedUsers = [ "root" "build" ];
                documentation.nixos.enable = false;

                # builds.sr.ht-image-specific network settings
                networking = {
                  hostName = "build";
                  dhcpcd.enable = false;
                  defaultGateway.address = "<IP_ADDRESS>";
                  usePredictableInterfaceNames = false; # so that we just get eth0 and not some weird id
                  interfaces."eth0".ipv4.addresses = [{
                    address = "<IP_ADDRESS>";
                    prefixLength = 25;
                  }];
                  enableIPv6 = false;
                  nameservers = [
                    # OpenNIC anycast
                    "<IP_ADDRESS>"
                    "<IP_ADDRESS>"
                    # Google as a fallback :(
                    "<IP_ADDRESS>"
                  ];
                  firewall.allowedTCPPorts = [ 22 ]; # allow ssh
                };

                environment.systemPackages = [
                  pkgs.gitMinimal
                  #pkgs.mercurial
                  pkgs.curl
                  pkgs.gnupg
                ];
              };
              qemuConfig = { pkgs, ... }: {
                imports = [ systemConfig ];
                fileSystems."/".device = "/dev/disk/by-label/nixos";
                boot.initrd.availableKernelModules = [
                  "ahci"
                  "ehci_pci"
                  "sd_mod"
                  "usb_storage"
                  "usbhid"
                  "virtio_balloon"
                  "virtio_blk"
                  "virtio_pci"
                  "virtio_ring"
                  "xhci_pci"
                ];
                boot.loader = {
                  grub = {
                    version = 2;
                    device = "/dev/vda";
                  };
                  timeout = 0;
                };
              };
              config = (import (pkgs.path + "/nixos/lib/eval-config.nix") {
                inherit pkgs; modules = [ qemuConfig ];
                system = "x86_64-linux";
              }).config;
            in
              import (pkgs.path + "/nixos/lib/make-disk-image.nix") {
                inherit pkgs lib config;
                diskSize = 16000;
                format = "qcow2-compressed";
                contents = [
                  { source = pkgs.writeText "gitconfig" ''
                [user]
                  name = builds.sr.ht
                  email =<EMAIL_ADDRESS>              '';
                    target = "/home/build/.gitconfig";
                    user = "build";
                    group = "users";
                    mode = "644";
                  }
                ];
              };
        };
        dispatch.enable = true;
        git.enable = true;
        hub.enable = true;
        lists.enable = true;
        meta.enable = true;
        man.enable = true;
        #pages.enable = true;
        paste.enable = true;
        todo.enable = true;

        # Integration into local services
        postgresql.enable = true;
        #postfix.enable = true;
        redis.enable = true;
        nginx.enable = true;

        settings = {
          "sr.ht" = {
            environment = "production";
            global-domain = config.networking.domain;
            origin = "https://${config.networking.domain}";
            owner-email =<EMAIL_ADDRESS>            owner-name = "Janus";
            site-blurb = "forge";
            site-info = "https://${config.networking.domain}";
            site-name = "DF";
            # nix shell nixpkgs#sourcehut.coresrht -c srht-keygen network
            network-key = "${builtins.readFile ./secrets/sourcehut/network_key}";
            #builtins.path { path = ./secrets/sourcehut/network_key; };
            secret-key= "${builtins.readFile ./secrets/sourcehut/secret_key}";
            # nix shell nixpkgs#sourcehut.coresrht -c srht-keygen service
            service-key = "${builtins.readFile ./secrets/sourcehut/service_key}";
            private-key= "${builtins.readFile ./secrets/sourcehut/private_key}";
          };

          # This is only needed for the pages.sr.ht service.
          # But may also be used to offer release storage in git.sr.ht,
          # and build artifact storage in builds.sr.ht
          # objects = {
          #   s3-upstream = "localhost";
          #   s3-access-key = "12345";
          #   s3-secret-key = pkgs.writeText "s3-secret-key" "12345678";
          # };

          # oauth-client-id of services are automatically authorized when the `metasrht-api.service` starts
          # TODO: I think it's better if oauth-client-id are different from one another.
          "builds.sr.ht" = {
            oauth-client-secret = "${builtins.readFile ./secrets/sourcehut/builds_client_secret }";
            oauth-client-id = "${builtins.readFile ./secrets/sourcehut/builds_client_id}";
          };
          "dispatch.sr.ht" = {
            oauth-client-secret = "${builtins.readFile ./secrets/sourcehut/builds_client_secret }";
            oauth-client-id = "${builtins.readFile ./secrets/sourcehut/builds_client_id}";
          };
          "hub.sr.ht" = {
            oauth-client-secret = "${builtins.readFile ./secrets/sourcehut/code_client_secret }";
            oauth-client-id = "${builtins.readFile ./secrets/sourcehut/code_client_id}";
            outgoing-domain = "https://code.${config.networking.domain}";
            origin = "https://code.${config.networking.domain}";
          };
          "git.sr.ht" = {
            outgoing-domain = "https://git.${config.networking.domain}";
            oauth-client-secret = "${builtins.readFile ./secrets/sourcehut/git_client_secret}";
            oauth-client-id = "${builtins.readFile ./secrets/sourcehut/git_client_id}";
          };
          "lists.sr.ht" = {
            oauth-client-secret = "${builtins.readFile ./secrets/sourcehut/lists_client_secret}";
            oauth-client-id = "${builtins.readFile ./secrets/sourcehut/lists_client_id}";
          };
          "lists.sr.ht::worker" = {
          };
          "man.sr.ht" = {
            oauth-client-secret = "${builtins.readFile ./secrets/sourcehut/builds_client_secret}";
            oauth-client-id = "${builtins.readFile ./secrets/sourcehut/builds_client_id}";
          };
          "meta.sr.ht" = {
            #outgoing-domain = "https://code.${config.networking.domain}";
          };
          "meta.sr.ht::settings" = {
            onboarding-redirect = "https://meta.${config.networking.domain}";
            registration = false;
            # Or register users with something like:
            # metasrht-manageuser -t admin -e<EMAIL_ADDRESS>john
          };
          "meta.sr.ht::api" = {
          };
          # "pages.sr.ht" = {
          #   oauth-client-secret = gnupg.secrets."sourcehut/oauth-client-secret".path;
          #   oauth-client-id = "299db9f9c2013170";
          #   s3-bucket = "pagesbuck";
          # };
          "paste.sr.ht" = {
            oauth-client-secret = "${builtins.readFile ./secrets/sourcehut/paste_client_secret}";
            oauth-client-id = "${builtins.readFile ./secrets/sourcehut/paste_client_id}";
          };
          "todo.sr.ht" = {
            oauth-client-secret = "${builtins.readFile ./secrets/sourcehut/builds_client_secret}";
            oauth-client-id = "${builtins.readFile ./secrets/sourcehut/builds_client_id}";
          };
          webhooks.private-key = "${builtins.readFile ./secrets/sourcehut/webhooks_private_key}";

          mail = {
            smtp-host = "localhost";
            smtp-port = 1025;
            smtp-user =<EMAIL_ADDRESS>            smtp-password = "${builtins.readFile ./secrets/smtp_pass}";
            smtp-from =<EMAIL_ADDRESS>            error-to =<EMAIL_ADDRESS>            error-from =<EMAIL_ADDRESS>            # pgp-privkey = null;
            # pgp-pubkey = null;
            # pgp-key-id = null;
          };
        };
      };

services.nginx:
  nginx = {
        enable = true;
        recommendedGzipSettings = true;
        recommendedOptimisation = true;
        recommendedTlsSettings = true;
        recommendedProxySettings = true;

        virtualHosts = {
          "${config.networking.domain}" = {
            enableACME = true;
            forceSSL = true;
            root = "/var/www/html";
          };
          "builds.${config.networking.domain}" = {
            forceSSL = true;
            enableACME = true;
          };
          "dispatch.${config.networking.domain}" = {
            forceSSL = true;
            enableACME = true;
          };
          "git.${config.networking.domain}" = {
            forceSSL = true;
            enableACME = true;
          };
          "lists.${config.networking.domain}" = {
            forceSSL = true;
            enableACME = true;
          };
          "man.${config.networking.domain}" = {
            forceSSL = true;
            enableACME = true;
          };
          "meta.${config.networking.domain}" = {
            forceSSL = true;
            enableACME = true;
          };
          "code.${config.networking.domain}" = {
            forceSSL = true;
            enableACME = true;
          };
          "hub.${config.networking.domain}" = {
            forceSSL = lib.mkForce false;
            enableACME = false;
          };
          "paste.${config.networking.domain}" = {
            forceSSL = true;
            enableACME = true;
          };
          "todo.${config.networking.domain}" = {
            forceSSL = true;
            enableACME = true;
          };
        }; # virtualhosts
      };

I am using the nixpkgs PR of this pull request (ie on top of the same nixpkgs base commit). I've only adjusted the secrets to take a string by switching out the types.path with types.str in the modules definitions, as seen above. That's only temporary until I work out using my secrets scheme with this module. The other adjustment I expected to be able to do is custom endpoints, eg code.${config.networking.domain} instead of the default hub, but it's not a big concern.
Any idea what the s value is supposed to be, thats missing, from this?
    19   │ for s in config:
    20   │     if not s.endswith(".sr.ht"):
    21   │         continue
    22   │     origin = cfg(s, "api-origin", default=get_origin(s))
    23   │     try:
    24   │         r = requests.get(f"{origin}/query/api-meta.json", timeout=5)
    25   │         if r.status_code != 200:
    26   │             continue
    27   │     except (requests.exceptions.ConnectionError, requests.exceptions.ReadTimeout):
    28   │         continue
    29   │     print(f"  Found {s}")
    30   │     access_grants.append({
    31   │         "name": s,
    32   │         "scopes": r.json()["scopes"],
    33   │     })
    34   │     service_scopes[s] = r.json()["scopes"]
    35   │ print(f"Discovered {len(access_grants)} APIs")

Also, this error with buildsrht enabled:
Nov 26 21:34:39 services systemd[2627007]: nginx.service: Failed to set up mount namespacing: /run/systemd/unit-root/var/log/nginx/buildsrht/logs: No such file or directory
Nov 26 21:34:39 services systemd[2627007]: nginx.service: Failed at step NAMESPACE spawning /nix/store/4q7z1zr4y7880ab1hl5w3vhwijspcm70-unit-script-nginx-pre-start/bin/nginx-pre-start: No such file or directory

@zkjanus, I've pushed more tests, and should have fixed the buildsrht/logs: No such file or directory (not tested whether it's enough to write/read the logs though). Beware that I've not fixed buildsrht-worker above this because I do not use this service, and it is taking tens of minutes to start on my computer which induces a painful development cycle.
Should you want to fix buildsrht-worker, you may hack those tests with something like:
$ nix -L build -f nixos/tests/sourcehut.nix driverInteractive
$ result/bin/nixos-test-driver --interactive
>>> start_all()
>>> machine.shell_interactive()
systemctl status buildsrht

I may or may not find the time to add other tests for the other services.

@ju1m well the more serious one comes from metasrht, core to sourcehut services. I'm not sure what's wrong there with the missing s value.
Thanks for the update!

[X] Rebase after the merge of https://github.com/NixOS/nixpkgs/pull/142635
[X] Remove patches merged upstream implementing Unix socket support for Redis.
[X] Revert recent upstream commits breaking Unix socket support for Redis.

[X] Rebase after treewide change fb0e5be84331188a69b3edd31679ca6576edb75a

This is a huge PR, should we go ahead with merging for now?

@Mic92, the sooner the better for me. Some services like builds-worker are not working, but I do not plan in using those so I will likely not fix them in this PR. @tomberek, is this PR good enough for you?

Yes, my intention was to give people a bit of time with the Redis PR and then merge this. It would be easier to get buildsrht working by merging. I’d be okay with a merge now. @ju1m thanks for all the work and sticking with it for the long process!

[X] Update to latest upstream versions (pkgs/applications/version-management/sourcehut/update.sh)