Switching between similar routes results in the components in separate routes sharing state.

Full disclosure - I am fairly new to react-router and this may be expected behavior, but I couldn't find anything that confirmed this either way and it certainly is not intuitive to me that things would behave this way in this specific situation.  Is definitely a very edge case - and there are multiple workarounds that I can think of so is not blocking anything for me.  Just letting you know.
Version
5.1.2
Test Case
https://codesandbox.io/s/routes-stateful-component-a-dt7y9
Steps to reproduce
A switch with more than one route.  Multiple routes must have the exact same component (or combination of components) that maintains their own state.  These routes need to be defined as child components of the route - not with "component" attribute.  Switch routes between these similar routes and check the component state.
Expected Behavior
Switching routes would rerender the component and reset the state. Similar to this example that has each route changed slightly with a different wrapper.
https://codesandbox.io/s/routes-stateful-component-b-xjbjy
In this example the buttons counter resets when switching routes.
Actual Behavior
The state of the component in the previously selected route is present in the component in the current route.  In the example - you can see that the button counter doesn't reset to 0 when switching routes.

This is just how React works. If you render the same component tree, it will retain the state of those components. There is no fix we can apply at the library level, because you may not want the component state to reset when the location changes, such as with a persistent header.
In most applications this won't be an issue. You generally have a component-per-route structure, which will be unique for each route. But if you need a workaround, use a key prop with a unique value for each component in each route. That will tell React each one is different and should be reinstantiated.


[$1000] Dev - excessive increase in Heap size indicating a major memory leak

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

Run the app in Dev
Open chrome devtools and wait until the app is fully loaded
In devtools navigate to memory tab and take a Heap snapshot
Continuously switch between two small or empty reports back and forth for a while
Take a second Heap snapshot and notice the heap size
Repeat the process switching back and forth again between the same two reports for a while
Take a third Heap snapshot and notice the size still increasing even though it's the same two reports

Expected Result:
Heap memory should remain relatively stable when switching back and forth between the same two reports. This is indicative of effective memory management and garbage collection.
Actual Result:
There's a notable increase in the heap size with repeated actions, leading to degraded performance and even occasional crashes. This suggests a significant memory leak.
Workaround:
Can the user still use Expensify without this being fixed? Have you informed them of the workaround?
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android / native
[ ] Android / Chrome
[ ] iOS / native
[ ] iOS / Safari
[x] MacOS / Chrome / Safari
[ ] MacOS / Desktop

Version Number: Dev
Reproducible in staging?: dev
Reproducible in production?: dev
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation
https://github.com/Expensify/App/assets/43996225/d8124dae-76be-4610-b445-a23a0d370ea4
Expensify/Expensify Issue URL:
Issue reported by: @AmjedNazzal
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1692349529815459
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~019948e426cdb9307a
Upwork Job ID:<PHONE_NUMBER>651981824
Last Price Increase: 2023-09-05

Reproducible

Its probably JS debugging stuff like storing logs and things, since this doesn't happen on Prod, should this be a bug? I mean yeah someone can surely take a look at this but it's not a prod-affecting bug, right?

@AJV009 I would say someone should take care of this yes, this is making working on dev very difficult to the point that yesterday after using the app for a while it was taking almost 8GB of RAM which is insane. this is not normal at all.

@AJV009 This is repro on prod also

Not overdue

@shubham1206agra I did try it on prod too.
My observations were: It was automatically getting cleared on prod (not on dev)
Appreciate that you found the root cause.
Your test data isn't enough for prod. Look carefully and wait for a few seconds, the sizes go up and also come down. I tested for quite some time trying to push it beyond 300 MB but then the garbage collection or whatever was responsible was working fine.
I assume a fix for this involves rewriting or maybe modifying the way variables or listeners are being created/executed?

No I waited for some time before taking snapshot

Let me just open the tab for 10 minutes and see the heap size

See snapshot 3 and current heap size

@AmjedNazzal Can you point out the PR? Or what kind of PR was that?

Sounds like we know what the issue is, is that right? Please submit proposals so that we can get this fixed!

@narefyev91 is it correct that we're still looking for proposals?

@narefyev91 is it correct that we're still looking for proposals?

Yup - waiting for some proposal here

Proposal
Please re-state the problem that we are trying to solve in this issue.
Major memory leak, causing performance issues. Possibly only on dev (though comments above disagree).
What is the root cause of that problem?
I think I have found the root cause of at least one problem. In @expensify/react-native-web there is a ModalPortal component that creates an empty div to use as the portal for the modal when it opens. These are not getting cleaned up, so the DOM is growing with every switch between reports. I added some debugging code to pass the modal ID through to ModalPortal so that I could see what's going on, and you can see that these are not going away. I'm getting 3 of these with every switch so they grow quickly.

The reason it's so memory hungry is there are a lot of React event handlers and FiberNodes hanging off these (at least in dev). I see those when comparing sequential memory snapshots.
At the React component level we seem to be recycling ModalPortal instances correctly, which is surprising. I don't know why the ModalPortal cleanup useEffect is not running when the ModalPortal component instance is deallocated. That needs digging into.
What changes do you think we should make in order to solve the problem?
I don't have code changes ready to go yet. I will need to dig into why the cleanup effects are not running, and that will take some time. I'm looking for approval to go ahead with this first.
Of course, this might only be one leak out of many, and I can't guarantee that this is the only leak. It does look like big one though.

Agree with @ewanmellor - investigation maybe a much more deeper here - but in any case fixing portals will make app faster
🎀 👀 🎀 C+ reviewed

Interesting find! Looking forward to the results of the investigation.

@robertjchen OK, so the thing I was chasing turns out to be pretty fundamental to the whole app.
Inside CentralPaneNavigator we use FreezeWrapper > Freeze from react-freeze > Suspense from React 17. This means that when we navigate forward, the screens further back in the nav stack are frozen in the sense they don't receive event updates, but they're otherwise preserved, so all the memory for them remains allocated. For a rich view like ReportScreen this can mean 10s or even 100s of MB of memory for every one.
When you switch between items in the LHN, you're moving forward in nav history, so even toggling back and forth between two items allocates new screens every time. In general, simply navigating around the app without using the browser back button is costing huge amounts of memory for every nav. This memory does get released if you use the browser history to go back, but it's not surprising that devs are seeing crashes during ordinary use.
Specifically for ModalPortal as mentioned above, the problem is that the modals are instantiated by their host page (e.g. a lot of things have a ConfirmModal attached to some functionality) and so as long as all the pages are on the nav stack, so are all their modal portals. There are DOM elements for each of these because ModalPortal allocates a div behind the scenes, and so all of these remain in the DOM until the user navigates back using the browser history.
I don't really have a good way to proceed from here. It seems that the app fundamentally is deliberately trying to keep all this memory allocated. The question is really whether you think you can afford that. 10s-100s of MB per LHN click seems insanely expensive and very likely to keep biting people. We could certainly reduce the cost of ModalPortal specifically (why do they need to exist when the modal is not open?) but I think that's probably scratching around the edges of a larger issue.

@ewanmellor great investigation! Maybe let's take this back to Slack to discuss with the broader team on how we'll move forward from here  🤔

@robertjchen I'm new to this project (only joined Slack on Monday). Can you lead that discussion?

yesss, I'll start a thread and drop the link here

Starting the discussion sometime this week

@robertjchen did you get a change to start the discussion? If yes, could you link to it so that I can follow along and help move the discussion/resolution of this issue?

@robertjchen I know you've been busy with other issues but could we get an update on this? If you're swamped, maybe you can find someone to take over this issue for you? Not an engineer but sounds like this memory leak is causing issues for contributors so it would be good to get fixed

Yep sorry! Let me kick things off in #expensify-open-source and in internal channels. Will hand this off to those who are more familiar with the low-level aspects 👍

Let me clean up the above comments, as more people will be participating in the discussion here soon

Discussion here! https://expensify.slack.com/archives/C01GTK53T8Q/p1696998420646989

Thanks for the investigation and analysis here, it sounds like we'll be addressing this specifically in the issue here: https://github.com/Expensify/App/issues/27651
To avoid duplication and confusion, I'll close this one out and maybe we can continue the conversation there? 🙏

@robertjchen do I get the bug report pay? I've reported this a long time ago before the bounty amounts were changed.


C++ support

It would be nice to use a stream instead of raw pointers like we do in C.

Added in daf28de13a84cebc4eecf6b490c4f6110b80f.


Effects with weather not working properly

Issue reported with one effect having "weatherCode" and another effect having "weatherCodeMin" and "weatherCodeMax". The current weather is 802. Seems to be showing both effects.
{
	weatherCodeMin: "801",
	weatherCodeMax: "803",
	images: ["heart1.png"],
	direction: "left-right",
	size: "6"
},
{
	weatherCode: "804",
	images: ["heart1.png"],
	direction: "left-right",
	size: "6"
}

Can't seem to reproduce


Pushing to remote failed

[I'm supposed to describe steps to replicate my issue here and if I don't do that, I understand that my issue will be probably ignored.]

Brackets 1.3.0-16022 (win), Git 1.9.5.msysgit.1 & Brackets-Git 0.14.24
Pushing to remote failed
Error: error: failed to push some refs to<EMAIL_ADDRESS>hint: Updates were rejected because the remote contains work that you do
hint: not have locally. This is usually caused by another repository pushing
hint: to the same ref. You may want to first integrate the remote changes
hint: (e.g., 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

call: git push --porcelain --progress origin master
Error: error: failed to push some refs to<EMAIL_ADDRESS>hint: Updates were rejected because the remote contains work that you do
hint: not have locally. This is usually caused by another repository pushing
hint: to the same ref. You may want to first integrate the remote changes
hint: (e.g., 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
    at Object.exports.toError (file:///C:/Users/gaurav/AppData/Roaming/Brackets/extensions/user/zaggino.brackets-git/src/ErrorHandler.js:162:19)
    at NodeConnection.<anonymous> (file:/

git pull ...') before pushing again.


Add stability and version support details

Some of these linters aren't very stable and some aren't updated for so long time then too old to use in newish go versions.
IMHO, would be good improvement in providing these details ?

Hello! Thank you for idea, we think it would be useful. Do you have any suggestions how to make it in "Awesome List" format?


Track average response time by agent

Problem
As a customer support manager I want to see response time break down by agent
Solution
On the reporting page we should have average response time broken down by agent over time.

Other option instead of having it in the dashboard is to export this data in a CSV file so that people can run analytics on it. I've seen companies basically set up some pretty complex excel spreadsheets so that they can do the analysis themselves.

Other option instead of having it in the dashboard is to export this data in a CSV file so that people can run analytics on it. I've seen companies basically set up some pretty complex excel spreadsheets so that they can do the analysis themselves.


Update dependencies and pin klog version

Here: https://github.com/Peripli/service-broker-proxy-k8s/pull/96

Coverage remained the same at 68.214% when pulling 72d0afa62189acd8627cff939a8b80cde3e7776f on upd_dep into a409029c892b350fc1efd5a35be4c6e207852653 on master.


Update all current book content to be "Crosslink aware", and include link to in-progress Crosslink hackmd.

The first step in completing this ticket is #67.


ceylon version --dependencies broken by typechecker changes

Since 3599380cf3ef55ffa55ca9eed91bfb132c353d77 the name of imports is set by a visitor that can only be set up by configuring lots of stuff, so ceylon version --dependencies is too simplistic to work now.

I've a fix for that.

Fixed.


[iOS] Added missing features

Changelog:

Added missing isBankingAppAvailable method;
Added missing fetchTransactionDetails method and sample app option;
Added documentation to the JudoPay class;

what is 'Get Transaction Details' feature? I don't recall anything like that on the native side


Web support

Depends on #62 and #60.
This PR allows Coffee to compile to Wasm! :tada:
Examples
I would greatly appreciate it if you report any issues you find!

Particles
User Interface
Mesh
Fourier (thanks to @PvdBerg1998 :tada:) | Source

Known issues

Text does not render on some environments.

Implementation details
The changes basically use the new gl backend coming to wgpu to run graphics on the web using WebGL2.
The PR uses my own forks of:

winit – implements a new web platform with support for web-sys and stdweb. Coffee uses the web-sys backend on web.
gfx-backend-gl – fixes some issues I have detected when dealing with WebGL. More specifically, it implements the CopyImageToSurface command and it fixes a big memory leak.
wgpu-native – implements new wiring to support the new winit features.
wgpu-rs – adapts to the changes in wgpu-native.
wgpu_glyph – fixes an issue with drawArraysInstanced in WebGL2 on Firefox.
instant – fixes an Illegal invocation error when calling performance.now(). A fix should be merged soon (https://github.com/sebcrozet/instant/pull/5).

This PR does not contain any additional breaking changes to the ones found in #62. The API remains unchanged, meaning web support will be mostly free for projects that are already using Coffee.
Pending work

[ ] Contribute winit Wasm support

[x] https://github.com/rust-windowing/winit/pull/797
[x] https://github.com/rust-windowing/winit/pull/1038
[x] https://github.com/rust-windowing/winit/pull/1040
[x] https://github.com/rust-windowing/winit/pull/1089
[ ] https://github.com/rust-windowing/winit/issues/1072

[ ] Contribute gfx-backend-gl fixes

[x] https://github.com/gfx-rs/gfx/pull/2887
[x] https://github.com/gfx-rs/gfx/pull/2891
[x] https://github.com/gfx-rs/gfx/pull/2892
[x] https://github.com/gfx-rs/gfx/pull/2893
[x] https://github.com/gfx-rs/gfx/pull/2895

[ ] Contribute wgpu-native  and wgpu-rs wiring code
[ ] Improve and merge wgpu_glyph fixes
[ ] Wait for instant fix (https://github.com/sebcrozet/instant/pull/5)
[ ] Wait for gfx backends to integrate with the new winit (and maybe contribute!)
[x] Wait for https://github.com/rustwasm/wasm-bindgen/issues/1513 to reach the stable toolchain
[ ] Investigate why BGRA* texture formats do not seem to work
[ ] Implement gamma correction
[ ] Remove single-layered texture arrays dirtyfix (if possible)
[ ] Investigate why twox-hash has so many collisions on Wasm
[ ] Investigate and fix text issues on some environments
[ ] Swap File::open with an alternative on Wasm
[ ] Test all backends with the new event loop (#62)
[x] Merge #60
[ ] Merge #62

rust-windowing/winit#1072

It seems that it was closed by rust-windowing/winit#1221, so it should be checked.
I suspect that many other issues are solved by now too!


Crash on World Load: RandomBlockEffectScanner.tick() Null

Hello!
I am creating a rather large modpack and when loading into a world I crashed. Here are the logs. Any help is appreciated. Thank you!
crash-2022-11-13_09.53.54-client.txt
latest.log

Crash happened through no fault of Dynamic Surroundings
Judging by the logs, to blame Armor Station from "PSWG Extra Mod" who initiated crash and starting closing Minecraft. After that, an error has occurred in Dynamic Surroundings


Prediction support for custom text

@udibr, I tried the prediction model and it works well with the dataset on which I trained it on. However, I am not able to figure out a way to train it on a custom text/article.
Kindly provide support for that also.

@vibrantabhi19 hey could you please share your code ... i am facing some errors with tensorflow 0.12 , keras 1.2.2

@vibrantabhi19 did you predict for custom tex?


chore(license): add header to remaining files

Signed-off-by: Lukas Mertens<EMAIL_ADDRESS>
Stack:

#83
#78
#70
#68
#67
#64
#63
#62
#61
#60
#58
#57
#56
#52
#49 ⬅
#47
#44
#43
#48

⚠ Part of a stack created by spr. Do not merge manually using the UI - doing so may have unexpected results.

✓ Commit merged in pull request #57


Replace distutils Version class with packaging.version

Using distutils.version classes emits a DeprecationWarning when building docs with setuptools>=60.0.0 and Sphinx fails:
DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

The new way to compare versions is using packaging.version. This PR replaces the use of disutils.version with packaging.version. Does this need any tests?

Thanks @choldgraf @damianavila!


Add support for treeshaking

According to Rollup guide adding module field in package.json helps tree shaking aware tools like webpack 2 or rollup import the ES6 modules directly to use the tree shaking feature and reduce the bundle size.

@javivelasco can you take a look as its a small change.

Thank you and sorry for the late response!!!

This is definitely a cool feature!
I can certainly see it exists in the current dev branch with package.json saying its version of 2.0.0-beta.11. But neither does this tag exists in the git tag of 2.0.0-beta.11 or 2.0.0-beta.12. Something wrong?

Can anyone making treeshaking to work with latest 2.0.0-beta.12 version?


Add instructions for running the examples

For anyone just looking to validate that they can indeed build and run the examples on their own computer, the README is missing the important note that git clone must be given the --recursive flag. Without that flag, the result is a baffling error (see below). This PR adds instructions for cloning and running the examples.
% cargo run --example basic_use
   Compiling memchr v2.5.0
   Compiling libc v0.2.146
   Compiling proc-macro2 v1.0.60
   Compiling glob v0.3.1
   Compiling quote v1.0.28
   Compiling unicode-ident v1.0.9
   Compiling minimal-lexical v0.2.1
   Compiling syn v1.0.109
   Compiling cfg-if v1.0.0
   Compiling bindgen v0.64.0
   Compiling libloading v0.7.4
   Compiling either v1.8.1
   Compiling regex-syntax v0.7.2
   Compiling rustc-hash v1.1.0
   Compiling bitflags v1.3.2
   Compiling log v0.4.19
   Compiling lazycell v1.3.0
   Compiling clang-sys v1.6.1
   Compiling lazy_static v1.4.0
   Compiling shlex v1.1.0
   Compiling peeking_take_while v0.1.2
   Compiling hound v3.5.0
   Compiling regex v1.8.4
   Compiling which v4.4.0
   Compiling nom v7.1.3
   Compiling cexpr v0.6.0
   Compiling whisper-rs-sys v0.6.0 (whisper-rs/sys)
The following warnings were emitted during compilation:

warning: Unable to generate bindings: clang diagnosed error: wrapper.h:1:10: fatal error: 'whisper.h' file not found
warning: Using bundled bindings.rs, which may be out of date

error: failed to run custom build command for `whisper-rs-sys v0.6.0 (whisper-rs/sys)`

Caused by:
  process didn't exit successfully: `whisper-rs/target/debug/build/whisper-rs-sys-2d620388fe53b4e8/build-script-build` (exit status: 101)
  --- stdout
  cargo:rustc-link-lib=dylib=c++
  cargo:rustc-link-lib=framework=Accelerate
  cargo:rustc-link-search=whisper-rs/target/debug/build/whisper-rs-sys-2976390336e0b243/out
  cargo:rustc-link-lib=static=whisper
  cargo:rerun-if-changed=wrapper.h
  cargo:warning=Unable to generate bindings: clang diagnosed error: wrapper.h:1:10: fatal error: 'whisper.h' file not found

  cargo:warning=Using bundled bindings.rs, which may be out of date

  --- stderr
  wrapper.h:1:10: fatal error: 'whisper.h' file not found
  CMake Warning:
    Ignoring extra path from command line:

     ".."

  CMake Error: The source directory "whisper-rs/sys/whisper.cpp" does not appear to contain CMakeLists.txt.
  Specify --help for usage, or press the help button on the CMake GUI.
  thread 'main' panicked at 'Failed to run `cmake` (is CMake installed?)', sys/build.rs:120:9
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

PS. @tazz4843 Great to see this project dedicated to the public domain using the Unlicense! (I'm the original author of the Unlicense.)

Ah yes I've hit this error myself several times and gotten incredibly confused. Thanks for the PR, and sponsor!


⚠ Bofrost.de has degraded performance

In 2c73e3c, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1842 ms

Resolved: Bofrost.de performance has improved in fcc07ca after 13 minutes.


Needs an option to enable 'capitalise first char of every word'

This extention of yours has been a life saver for me for a long time but there was a feature I wanted more than life itself. That is to get capitalise first character of every word.
If you could add the feature.
Have a nice day Dev.

@khagendra-poudel I'm happy for you to raise a PR for this.


[docker-compose] Review the contents in  docs folder and transfer to readme

https://github.com/dojot/docker-compose/tree/master/docs

Should we remove this folder 'docs'?

Should we remove this folder 'docs'?


merged with casa cube and added power law fitting of emission surfaces

Thanks, I have included your changes in measure_height2.py and will try to merge back in measure_height.py


About the test set results

Hello, I have made improvements to your baseline, and all three categories have improved on the validation set. However, the test results submitted to the kitti official website are much lower than your results. Could you please give me some suggestions?

@sgrrlll have you sloved this problem

The score threshold used to submit to test set is different from validation set, as the KITTI metric is much more sensitive to object recall. Please use a lower score threshold as follows and train your detector on all training data to attain a good performance.
    POST_PROCESSING:
        RECALL_THRESH_LIST: [0.3, 0.5, 0.7]
        SCORE_THRESH: 0.3 #  using 0.1 if your results are still not good
        OUTPUT_RAW_SCORE: False

The score threshold used to submit to test set is different from validation set, as the KITTI metric is much more sensitive to object recall. Please use a lower score threshold as follows and train your detector on all training data to attain a good performance.
    POST_PROCESSING:
        RECALL_THRESH_LIST: [0.3, 0.5, 0.7]
        SCORE_THRESH: 0.3 #  using 0.1 if your results are still not good
        OUTPUT_RAW_SCORE: False

Thanks for your suggestion. I have some doubts, when I use train+val to train the model, how do I pick the best model?
Or if I use 80% of the data and the checkpoint with the highest mAP doesn't work well in the test set, should I choose based on the maximum of the recall or judge from the PR curve. Looking forward to your reply.


Add new fork to the README

Fixes https://github.com/kube-libsonnet/kube-libsonnet/issues/3

Thanks @carrodher , @agarcia-oss 🎉
will try to keep it alive & kicking =)


Plan to update to Miniaudio v0.9.1?

Is there any plan to update to Miniaudio v0.9.1? I think there are some breaking changes though.

Sure, will be updated, not sure when will that happen, as you noticed there are some breaking changes and some renaming have to be done.

@nkev I did a rough port here https://github.com/godsic/malgo

@nkev what are the rough parts? Can you send a PR if everything works as expected? I have some local copy where I tried to update to latest API and I failed but didn't have time to check all details.

@gen2brain

0.9 introduced new features that are not ported to malgo at the moment;
I am tracking dev branch for wasapi resampling/sample format control;
alsa and wasapi backends work, others were not tested;
capture and full duplex modes were not tested.

If that’s ok with you I can go ahead with making a PR.

Yes please, new features can wait a bit, but if it works on Linux and Windows it can be merged. If there are some issues after that they can be fixed later, much better then to be too much behind upstream lib.

I guess this can be closed now that #11 was merged?

Right, closing.


InitializeMCJITCompilerOptions does not work

While fixing the "out string" problem, I discovered that InitializeMCJITCompilerOptions overwrites the stack. MCJITCompilerOptions is a struct containing four ints and a pointer. As such, the Value field of the MCJITCompilerOptions must be a struct or it must be a pointer to a struct.

A special case for MCJITCompilerOptions must be written. The "typedef struct TYPE { } TYPE;" in the Swig specification files has the effect of nuking all fields from the data type, as it overrides any further definitions of the data type. (Adding a comment in the code so people don't have to rediscover this like I had to.)

Fixed in upcoming 4.0.6-alpha release.


How to switch to YouTube Music on Windows?

Hello,
Can't find the option to switch to the YT Music. Can a kind soul help me out on this? Thanks! BTW, thank you for this awesome work!

It's in the tray icon, right click and change the service to YouTube music.


Items created with "Add selection button" are not selectable when background is locked

See video please, i ve created a new button from editor and it can select it to edit when only when background image is unlocked
https://github.com/DreamWall-Animation/dwpicker/assets/638393/7b4e0bbb-fe08-42de-85e0-9a504e1b045a
When i manually add something to "action.targets": ["something"] in picker file, it goes well. PR will take some time, but may you have a clue where to look at.

It seems that there is no clear issue with dwPicker, it works as stated, "Add selection button", it only was not expected that in case of empty selection and locked background button become locked. It is better to resolve this confuse but it is up to you, if it will become clear on how it have to be resolved, i would prepare PR. My ideas are: 1. prevent button creation - show query "do you really want to create button with empty selection", or 2. disabled action in menu, 3. make background item instead of button-looking-item, 4. make script button doing nothing or printing "button was created with empty selection", 5. add None targets, so  item will not behave as background.

Hi Kalemas,
Indeed, this is a "normal" behavior reading the code. I guess the best solution is that I should add an attribute on buttons to define if they are background or not instead of guessing from their content. I will always wrongly guess in some cases. In state, it means that if you create a command button, it will be not selectable before you add a commande :/

Okay, I added this attributes, so now background shapes can be explicitely specified.

This field will be automatically filled importing old picker files, using old rules to determine whether or not a shape is background.
Cheers !


Added more constant-time code / removed biases in the prime number generation routines.

This pull request brings two changes:

I have improved the time- and flow- constantness of some parts of the code handling big integers;

I have removed statistical biases in the prime generation routines, and implemented a rejection method instead, which is the proper way to do. Such biases might have devastating consequences in terms of security for certain public-key cryptosystems. Note that I don't claim any attack here: I am not an expert in this field of cryptanalysis. However, look at the paper of Aranha et al. GLV/GLS Decomposition, Power Analysis, and Attacks on ECDSA Signatures With Single-Bit Nonce Bias presented at ASIACRYPT'14 as well as the references therein to be convinced that biases are bad in cryptography.

Thanks for your contribution! A few questions/remarks:

Regarding the time- and flow- constantness, I know people tend to use bit operations rather than equality comparisons, but I've never seen a convincing argument for avoiding them so far. I've looked at the generated assembly code

(Edit: previous message send before it was complete, sorry.)
Thanks for your contribution!

Regarding constantness: I know people tend to avoid equality comparisons because they might be implemented using a branch, but I checked the assembly produced for a = (b == 0) by common compilers with various optimization options for x86 and arm, and it looks constant time to me. (Which is not surprising: branches are bed for performance too.) I would like to avoid impairing readability for no good reason. Do you have an example of compiler&platform where == produces a branch? Or a solid reference for why == should be avoided?

Regarding biases, I'm quite convinced they should be avoided, but thanks for the reference anyway. Our key generation function for ECC (including ephemeral ECDSA exponents) already uses a rejection method for this reason. I forgot to  check mpi_gen_prime(), thanks.

Regarding ...by common compilers with various optimization...: for sure, it is impossible to check every version of every compiler on every platform with every flag combination. Therefore, I tend to prefer writing flow-constant code, for security hygiene reasons, even if it is less readable. For a solid reference, see e.g. https://cryptocoding.net/index.php/Coding_rules#Compare_secret_strings_in_constant_time

I agree it's impossible. But for the same reason, it's impossible to guarantee that flow-constant C code will always be translated to flow-constant assembly code by every compiler on every platform with every flag combination, and the cited reference indeed gives an example when that fails to happen: uint32_t ct_select_u32(uint32_t x, uint32_t y, _Bool bit).
This reference (which I already knew) is indeed solid on the how, but IMO no so solid on the why. For constant-time string comparison, it jumps straight from the obviously bad "stop early" version to the version with only bit operations without a word about why they think a version with a constant number of iterations but using == would not be enough. What I would love to see is a comparative table that, for a reasonable set of platforms/compilers/constructs, tells which way of coding (eg with == vs only bit operations) results in branches and which doesn't. I'd like hard data rather than just making assumptions (even if those are indeed quite reasonable).
OTOH, I agree that the way of coding you recommend is indeed the accepted "best practice". I just feel chronically uncomfortable about this "best practice" being, IMHO, not that well motivated. (This probably shows in my code, which currently contains a mix of == and bit-operation equivalents.)

PS: just to be clear, I've been arguing about the constant-time part because I was hoping you could convince me, rather that the other way round :)

Ok, I finally convinced myself that replacing == by bit operations is indeed useful, by collecting some data: https://github.com/mpg/ct/blob/master/results :)
I guess I'm in for some code review and fixing (I'm sure there are other places than these, at least one in our Bleichenbacher counter-measure in ssl_srv.c, but maybe others as well).

Thank you to provide this information: it confirms my feelings (and those of many applied cryptographers), and the strategy of being conservative regarding those aspects! Are you going to keep the link  https://github.com/mpg/ct/blob/master/results alive? If yes, one might add a link to it on https://www.cryptocoding.net.

I fully intend to keep the data available (since having to rely on feelings rather than data was the main reason for my earlier hesitations, so I hope it will help others), but it's not entirely clear yet if this specific link will continue to work. However, I can update the link on cryptocoding.net if/when it changes.

Hi Pascal,
I'm currently busy getting a standard CLA for mbed TLS within ARM. When that is done, I'll send it to you, so we can integrate your improvements into the core codebase!

Pascal,
It is done. Can you send me your e-mail address? paul dot bakker at arm dot com


[WIP] implement wp-tearing-control-v1 & immediate presentations

DO NOT USE YET
Implementation is not working due to wlr's lack of async flip support.
FPS are uncapped, meaning you can literally fry your GPU.
safeguard in window.cpp for now disables the tearing feature altogether.
Depends on wlroots@MR3871
Adds automatic tearing hints from the wp-tearing-control-v1 protocol, can be forcefully enabled for a window with the immediate windowrule.

FPS are uncapped, meaning you can literally fry your GPU.

 This rumor came from a faulty batch of EVGA gpus.
https://www.pcgamer.com/evga-explains-why-some-of-its-rtx-3090s-were-blowing-up-in-new-world/

better safe than sorry.

superseded by https://github.com/hyprwm/Hyprland/pull/1880


Early stopping in fit with KeyboardInterrupt

Allow users to stop the Sequential.fit(...) method with a KeyboardInterrupt and still return the training history.

Good for users prototyping in iPython, etc.

I see how this can be valuable, but I think try / except should be reserved for exceptions (which should be exceptional), not regular control flow. Try / excepting the entire main loop seems a bit heavy-handed.
Have you looked into using an interrupt handler to achieve the same result? https://docs.python.org/2/library/signal.html

Hmmm, I've seen this used a bunch, but I can understand that it's not the most semantically consistent trick.
How would you feel about incorporating a class like this into utils/generic_utils.py?
Then, we can do something like:
with TrainingHandler() as training_loop:
    for epoch in range(nb_epoch):
        ...
        if training_loop.interrupted:
            print('blah')
            break

@lukedeo @fchollet any particular reason why this was closed out? I think this would be highly useful and I don't see that this could be implemented using keras callbacks so it would need to go into the main training loop.

is there maybe a way to implement this in a custom generator fed into fit_generator?


Error: EMFILE on OSX when requiring lots of files

When requiring 256 files from a single file on OSX browserify throws:
Error: EMFILE, open '/Users/karl/Sites/KVP/GuruFramework/package.json'

From Googling around it looks like this is caused by attempting to open too many files in parallel.
It looks like this could be solved by using the graceful-fs module to limit the number of files opened in parallel.
https://github.com/isaacs/node-graceful-fs

ulimit -n 2560 solved the issue

@karl Thanks for the fix!

@karl How did you use graceful-fs as a work around? I'm new to this. I tried copying and pasting your work around to my ~/.bash_profile and it did not work.

I never actually implemented a graceful-fs fix I'm afraid, I just suggested it as a possibility.
I'm still using the ulimit workaround.

just ran into this as well

I just tried to upgrade to<EMAIL_ADDRESS>and started getting EMFILE errors on our CI server (Ubuntu 12.04 - which reports unlimited for ulimit).  Can this issue be retitled to remove the OSX specific part?  Using graceful-fs would solve the issue everywhere.

Make sure you are running ulimit -n not just ulimit to determine the maximum number of open file descriptors.

On Aug 14, 2015, at 12:41 PM, Tim Schaub<EMAIL_ADDRESS>wrote:
I just tried to upgrade to<EMAIL_ADDRESS>and started getting EMFILE errors on our CI server (Ubuntu 12.04 - which reports unlimited for ulimit). Can this issue be retitled to remove the OSX specific part?  Using graceful-fs would solve the issue everywhere.
—
Reply to this email directly or view it on GitHub https://github.com/substack/node-browserify/issues/431#issuecomment-131206094.

Make sure you are running ulimit -n not just ulimit to determine the maximum number of open file descriptors.

Thanks @programmarchy, my bad.  So 1024.  I think it is still valid to say that this should not be considered an OS specific issue.

Getting this in Windows now and sadly it's a garbage OS and doesn't seem to have a simple way to change the limit.

@RangerMauve
http://blogs.technet.com/b/markrussinovich/archive/2009/09/29/3283844.aspx
My understanding is that Windows has a limit of somewhere around 16.5
million handles, so upping it might not solve your problem.
Maybe you could npm dedupe and try again?
--
Jack Viers
333 E. Grand Ave.
#115
Des Moines, IA 50309
515.313.7838

@jackcviers I'm not so sure that's true anymore. I only started having this issue after a Windows 10 update earlier in the week. Before then it was all working properly.
Also, it's not so much that I have a lot of dupes, I'm just running several builds at once in a gulp script.

Here's a possible answer:
http://serverfault.com/questions/108827/windows-file-handle-limit
I know that even one build running through briwserify of a fairly large app
on mac would trigger the handle limit right away.
I know the point of gulp is lost if you just use a single synchronous
build, but for workaround purposes you could start with one and add a build
at a time in parallel until you hit the limit. Then try to create a
sequence of tasks for all your builds, limiting your parallelism to each
one to the amount that runs; e.g build one thru 3 don't depend on anything,
3 thru six depend on one, two, and three, etc. That will be a minor pain in
gulp, but might still give you acceptable throughput without crashing.
Other possible workarounds, asuming w10 handler limit cannot be changed,
are watchify, running your build in a docker / vagrant / environment where
you can set the limit appropriately, or vreaking your project into smaller
pieces with individual builds and common dependencies.
On Fri, Sep 11, 2015, 09:39 RangerMauve<EMAIL_ADDRESS>wrote:

@jackcviers https://github.com/jackcviers I'm not so sure that's true
anymore. I only started having this issue after a Windows 10 update earlier
in the week. Before then it was all working properly.
Also, it's not so much that I have a lot of dupes, I'm just running
several builds at once in a gulp script.
—
Reply to this email directly or view it on GitHub
https://github.com/substack/node-browserify/issues/431#issuecomment-139563348
.

--
Jack Viers
333 E. Grand Ave.
#115
Des Moines, IA 50309
515.313.7838

What I ended up doing was ensuring that the steps run serially rather than in parallel. It makes the build take more time, but at least it works now.

Updating to the latests node.js solved this issue for me!

The similar issue happen to me when i run gulp with watch, it says events.js throw err without handling and watch null emfile. So i searched emfile error and saw someone saying reboot. I just rebooted my pc and everything is fine now.

Hello,
I'm facing the same issues as @RangerMauve but on Windows 7. This started to happen when my project got bigger with more modules. Changing the ulimit in Windows (if that's possible) is no option because these machines are administrated by the customer.
Is it still planned to use graceful-fs instead of fs? Currently I'm not able to build my project entirely. I have to split it into two parts.
Thanks!

Try using npm 3. It's more agressive at keeping the node_modules folder flat.

I'm already using npm 3... And with modules I ment application entry points => bundles for browserify. The problem is that I'M creating multiple bundles at the same time. I need to implement some kind of scheduler which watches over it and limits it e.g. to 5 parallel browserify bundlings...


Use frame pointer opcodes for functions - part 1

AVM 8+ has support for frame pointers which allow for more efficient stack & slot usage for subroutines (functions in Tealish).
Tealish can generate Teal that uses proto at the start of function definitions instead of stores. frame_dig can be used to access the function args from the stack without having to store them in slots.
Teal example for reference:
pushbytes "ab"
pushbytes "cd"
// log(myfunc("ab", "cd"))
callsub myfunc
log
pushint 1
return

// myfunc(x: bytes, y: bytes) bytes:
myfunc:
// specify that our function takes 2 arguments and has one return value
proto 2 1 // internally puts the first 2 values from the stack into the frame
frame_dig -2 // first arg
frame_dig -1 // second arg
concat
retsub

I now understand that we can use frame pointers with frame_dig & frame_bury for function locals too instead of scratch slots. This is will allow for safe recursive functions and negate the need to analyse the call graph to find a safe slot allocation.
It is a more invasive change however so I suggest to tackle it as a separate issue after this one.


Switched to Functional React Components and added API documentation for Login/Register

Looking good

 Switch to Functional Components for Login/Registration


HHH-12892 - Fix spelling issues in the User Guide

https://hibernate.atlassian.net/browse/HHH-12892

Merged. Also merged the other commit + this one to 5.3.


document user facing functions

Codecov Report

Merging #11 (70a538a) into main (a3c68cd) will decrease coverage by 0.20%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main      #11      +/-   ##
==========================================
- Coverage   93.54%   93.34%   -0.21%
==========================================
  Files           9        9
  Lines         496      496
==========================================
- Hits          464      463       -1
- Misses         32       33       +1

Impacted Files
Coverage Δ

src/question.jl
97.40% <ø> (ø)

src/question_group.jl
60.00% <ø> (ø)

src/response_scale.jl
94.11% <ø> (ø)

src/subquestion.jl
100.00% <ø> (ø)

src/xml.jl
87.28% <ø> (-0.58%)
:arrow_down:

src/language_settings.jl
100.00% <100.00%> (ø)

src/survey.jl
100.00% <100.00%> (ø)

src/survey_component.jl
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a3c68cd...70a538a. Read the comment docs.


SPARK-1094 Support MiMa for reporting binary compatibility accross versions.

PR 585 from incubator repo.

Test build #15 has started for PR 20 at commit 01e6b1d.

Test build #15 has finished for PR 20 at commit 01e6b1d.

This patch fails RAT tests.
This patch does not merge cleanly.
This patch adds no public classes.

Test build #33 has started for PR 20 at commit 01e6b1d.

Test build #33 has finished for PR 20 at commit 01e6b1d.

This patch fails RAT tests.
This patch does not merge cleanly.
This patch adds no public classes.


[Help] Push Notification Certificate has expired

Hi I'am getting this error when sending push notification from dashboard and via code.
This my code:
Parse.Push.send({
        where: query,
        data: {
            title: title,
            alert: message,
            page: page
        }
    }, {
        success: function() {
            console.log(message);
        },
        error: function(error) {
            // Handle error
        }
    });
}

This the error I'm getting.
Error: certificate has expired: 2017-02-04T07:11:57.000Z
    at validateCredentials (/root/parse-server/node_modules/apn/lib/credentials/validate.js:12:9)
    at /root/parse-server/node_modules/apn/lib/connection.js:163:5
    at _fulfilled (/root/parse-server/node_modules/q/q.js:834:54)
    at self.promiseDispatch.done (/root/parse-server/node_modules/q/q.js:863:30)
    at Promise.promise.promiseDispatch (/root/parse-server/node_modules/q/q.js:796:13)
    at /root/parse-server/node_modules/q/q.js:604:44
    at runSingle (/root/parse-server/node_modules/q/q.js:137:13)
    at flush (/root/parse-server/node_modules/q/q.js:125:13)
    at _combinedTickCallback (internal/process/next_tick.js:67:7)
    at process._tickDomainCallback (internal/process/next_tick.js:122:9)
/root/parse-server/node_modules/parse-server/lib/ParseServer.js:437
            throw err;

@makoy21 push notifications certificates from Apple are valid only for one year, expiring after that. That's was happened to yours.
Create a new one in developer.apple.com and update it in your Parse Server.

Hi how can i handle that error without crash the parse server?


ES6 Modules

ES6 Modules with Rollup/terser (in place of browserify/uglify-js);
add module to package.json and change main
npm: Add recommended fields to package.json

The source is the same as before except with use strict and the IIFE removed (given module context), indentation adjusted accordingly, and and of course the export altered.

Since you are contributing in various place, I'd like to tell you how all my repositories are, in terms of consistency, and how I want to keep it that way.
ESM
The entry point of any ESM module is esm/index.js and everything related to that module should stay inside the esm folder.
Every ESM import should be Web compatible unless it's not possible other ways. Example, import anything from './some/file' is not acceptable, import anything from './some/file.js' is.
If there is no syntax to do special things like CJS would do, then it's OK to let bundlers resolve that (i.e. import 'module' since there is no standard way yet to resolve external dependencies.
The entry point should be the module field of package.json.
CJS
The entry point of any CJS module is cjs/index.js and and everything related to that module should stay inside the cjs folder.
You can create out of the box CJS equivalent via ascjs and the command "ascjs esm cjs" saved as "cjs" script in the package.json.
The entry point should be the main field of package.json.
No interoperability
Nothing between CJS and ESM is shared, I terms of files, those are two pure, independent, different, module systems.
Web
The main entry point for the web, if any, is the index.js. It should have a min option that creates a min.js file, and that file should be the unkpg entry of the `package.json.
UMD
I don't believe in UMD. I publish modules that work either on the web as global exports, or as CJS or as ESM. UMD makes no sense these days, because UMD cannot deliver ESM and also it cannot deliver for other platforms different from NodeJS.
Yes, I publish modules compatible also with GJS, or JSC, or even Nashorn, I don't want over bloated fairy tales in my repositories, and I consider UMD one of those.
A single source of truth
Sometimes it's the index.js, copied over as cjs/index.js and esm/index.js plus the related export, and sometimes it's the esm/index.js made CJS via ascjs and bundled for Web as index.js.
In the first case, I usually write ES5 compatible syntax myself to avoid the need of transpilers or over bloated final output due transpilers. As example, query-result was 60 LOC and transpiled about 400 lines. I had to manually craft, chose, and pick modern features to keep it smaller and non breaking once transpiled, and I hate doing that ... so I often just write older code.
However, like in hyperHTML case, if I pay attention to obvious things that bloat the transpilation without any real-world improvement (usually quite the opposite, where everything goes slower on already slower browsers) I write relatively modern code but I avoid everything that involves Symbol.iterators such for/of and [...spread] or ...rest.
This also helps with performance, whenever it's cryptical.
Anyway, to close this chapter, whenever somebody would like to help with a PR, there should be only changes in such source of truth, and an environment easy enough to install via npm I and work with through npm run build.
Apologies for the long answer.

About this PR, please follow above convention. The esm/index.js feel should be the entry point for ESM, using exact same code the index.js has, and so should do the CJS.
You can remove the exporting bit at the end of index.js file, since once there are 3 ways to load it it's clear which part should do what (i.e. the CJS/ESM should just export a module while the browser should overwrite).
It might be interesting to figure out how to create these 3 distinct files so that maybe, an extra core.js file with the shared logic could be used to generate index.js, esm/index.js and cjs/index.js.
If you think it's too much, I'll try to have a go whenever I have time, but I'm surrounded by requests these days and I am a full time employee and I also have a life after that ( or trying to 😂 ) so thanks for your help or patience.

I've updated this PR per my understanding of your instructions, but let me know what further needs fixing...
I didn't remove the lines at the end of index.js because it is currently derived from core.js (via Rollup). If you want me to add a Rollup plugin routine to strip out the code which is not relevant for the browser or CJS, let me know.
Btw, that block at the end is problematic in a different way now that there is no module.exports there causing the block to throw... I'd expect though that we not only want to avoid overwriting the native console for CJS, but also by default for ESM--unless expose a method that could do that overriding...

nothing between CJS and ESM is shared means that a file outside cjs or esm folder cannot be used in both modules systems. There is no require('../extra') and import from '../extra' 'cause the module system is different.

by insisting on the distinctness of each module system ...

Those are different modules system, I don't care about bundlers 'cause bundlers can work out what to import, standard ESM doesn't.


[Tutorial proposal] Rapid Classification of croplands: Case Study of smallholder maize cultivated lands

What is the objective of the proposed tutorial?
The objective of this tutorial is to guide institutional and individual users to implement an automated routine/tool (in GEE) for classification of croplands within Nigeria's farming system, based on contextual peculiarities related to season and cropping practices. This tutorial is focused on developing the basic competency of the prospective user(s) to ingest ground-truth data and generate quick outputs about the regional estimate of land area sown with Maize.
What is the scope of the proposed tutorial?
Landcover classification for croplands
Please provide an outline of the structure of the proposed tutorial?

Introduction
a. Need to classify croplands
b. Value for users
c. Data Sources

Data preparation and checking
a. Groundtruth reference
b. Importing boundary files
c.  Sentinel 2A Imageries

Analytics
a. Defining model
b. Selecting/optimizing Classifiers
c. Generating results

Platting the charts to generate insights

Exporting Outputs

In what format will you be submitting the tutorial?
Select one: Markdown
This request will be reviewed by the Earth Engine community maintainers, who will reply on this issue tracker with any questions or suggestions. Once approved, this issue will be assigned to you and you can begin work on the tutorial following instructions in Writing a tutorial. When ready, enter “Closes #issueno” in the description of your Pull Request to link the tutorial to this issue.
Issue labels:
proposal, tutorial

Hi Tyler (@tylere): Is there any chance to get this tutorial proposal reviewed and approved this week pls?

Hi @PJNation , your community tutorial proposal has been accepted! To get
started writing a new tutorial, please follow
these instructions.
Looking forward to receiving your contributions!


MockWebServer should have public APIs for HeldCertificate and SslClient

Possibly with examples. Would be especially nice to have an example of mutual auth.

[ ] HeldCertificate: promote to okhttp3.mockwebserver
[ ] SslClient: new APIs for my certs vs. peer’s certs. Should accept a cert to trust.
[ ] MockWebServer: setWantClientAuth(true) on incoming SSL sockets
[ ] RecordedRequest: expose the server’s handshake as a new field

Now that these are ready I’m tempted to take one more step – to move HeldCertificate and TlsNode into a new “okhttp3.tls” module.
It would allow us to remove bouncycastle as a MockWebServer dependency. (Anyone using TlsNode would still need bouncycastle). It would give us the option to later support non-test use cases, such as parsing PEM files and generating CSRs. And we’d also have the option to replace our current bouncycastle dependency with something focused on just our use-cases.

Okay, that’s all done.
New next steps:

[ ] Be clear in the docs about whether this is for test code, or fix the defaults to be appropriate for production code
[ ] API to add the platform’s default trust roots (possibly opt-in or opt-out)
[ ] Try to find a replacement name for TlsNode


NumbaWarning: The TBB threading layer requires TBB version 2019.5 or later i.e., TBB_INTERFACE_VERSION >= 11005. Found TBB_INTERFACE_VERSION = 9002. The TBB threading layer is disabled.

hi,dear
how to modify the warning ?
could you pls help me ?
thx

@ucasiggcas thank you for asking about this. My guess is that you need to update your version of tbb.

hi,thanks in advance
but tbb is what in python ? or how to update the module ?
thx

If you installed with conda, you can use conda install tbb to update it.
If you installed with pip, use pip install --upgrade tbb

It's updated by 'pip install --upgrade tbb' to 2020.3.254. But it doesn't work. There is  same 'NumbaWarning' message.

@eagle40 pip can sometimes not upgrade properly so old versions have a tendency to "hang around". I would suggest starting with a fresh virtualenvironment (or whatever other means you have for installing packages) to avoid such cases.

This most likely occurs when a user is using @jit(parallel=True) and one of:

the user is using conda and has installed TBB and it's of a too old version for Numba to use.
the user is using conda and hasn't installed TBB, but is running in the base conda environment which has an older version of TBB.
the user has installed TBB via pip and it's unlikely that the pip library directories are on a path that Numba can find and so a potentially too old system level TBB is found.
the user hasn't installed TBB but there is a potentially too old version of this library present at system level.

Quickest path to getting TBB working is to use conda and install tbb like:
$ conda install tbb

quickest way to stop the message if the tbb threading layer isn't wanted is to specify the threading layer that is wanted instead. e.g.
$ NUMBA_THREADING_LAYER='omp' python <my program>

see https://numba.readthedocs.io/en/stable/user/threading-layer.html#selecting-a-threading-layer-for-safe-parallel-execution for more information.

On a related note: when running python with -We (treat warnings as errors mode), the logic within numba.np.ufunc.parallel._launch_threads() does not work anymore as it expects an ImportError to be raised but this code:
            warnings.warn(problem)
            raise ImportError("Problem with TBB. Reason: %s" % msg)

raises a numba.core.errors.NumbaWarning instead.
This leads further to things like:
...
During handling of the above exception, another exception occurred:
...
/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/numba/core/decorators.py:218: in wrapper
    disp.compile(sig)
/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/numba/core/compiler_lock.py:32: in _acquire_compile_lock
    return func(*args, **kwargs)
/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/numba/core/dispatcher.py:819: in compile
    cres = self._compiler.compile(args, return_type)
/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/numba/core/dispatcher.py:78: in compile
    status, retval = self._compile_cached(args, return_type)
/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/numba/core/dispatcher.py:92: in _compile_cached
    retval = self._compile_core(args, return_type)
/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/numba/core/dispatcher.py:105: in _compile_core
    cres = compiler.compile_extra(self.targetdescr.typing_context,
/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/numba/core/compiler.py:627: in compile_extra
    return pipeline.compile_extra(func)
/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/numba/core/compiler.py:363: in compile_extra
    return self._compile_bytecode()
/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/numba/core/compiler.py:425: in _compile_bytecode
    return self._compile_core()
/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/numba/core/compiler.py:405: in _compile_core
    raise e
/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/numba/core/compiler.py:396: in _compile_core
    pm.run(self.state)
/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/numba/core/compiler_machinery.py:341: in run
    raise patched_exception
/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/numba/core/compiler_machinery.py:332: in run
    self._runPass(idx, pass_inst, state)
/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/numba/core/compiler_lock.py:32: in _acquire_compile_lock
    return func(*args, **kwargs)
/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/numba/core/compiler_machinery.py:291: in _runPass
    mutated |= check(pss.run_pass, internal_state)
/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/numba/core/compiler_machinery.py:264: in check
    mangled = func(compiler_state)
/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/numba/core/typed_passes.py:442: in run_pass
    NativeLowering().run_pass(state)
/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/numba/core/typed_passes.py:370: in run_pass
    lower.lower()
/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/numba/core/lowering.py:136: in lower
    self.lower_normal_function(self.fndesc)
/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/numba/core/lowering.py:190: in lower_normal_function
    entry_block_tail = self.lower_function_body()
/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/numba/core/lowering.py:216: in lower_function_body
    self.lower_block(block)
/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/numba/core/lowering.py:230: in lower_block
    self.lower_inst(inst)
/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/contextlib.py:131: in __exit__
    self.gen.throw(type, value, traceback)
/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/numba/core/errors.py:752: in new_error_context
    reraise(type(newerr), newerr, tb)
/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/numba/core/utils.py:81: in reraise
    raise value
E   numba.core.errors.LoweringError: Failed in nopython mode pipeline (step: nopython mode backend)
E   The TBB threading layer requires TBB version 2019.5 or later i.e., TBB_INTERFACE_VERSION >= 11005. Found TBB_INTERFACE_VERSION = 9107. The TBB threading layer is disabled.

Unintuitive and very tricky to understand.
Perhaps it would be worth to refactor the logic in select_known_backend() so it does not use exceptions for control flow?
Or catch both ImportError and NumbaWarning?

@slayoo thanks, I've opened #6592 to track.

There's more information on this topic on the Numba discourse forum here: https://numba.discourse.group/t/suppress-tbb-warning
Closing this issue as it seems to be resolved. If this is not the case please re-open with a comment about any item that appears to be unresolved. Many thanks.


spankies

Allow all mods to spank each other regardless of wether it works or not.

resolved - #302f26


Improve section tag constants in archives

See commit messages for details. This should close out issue #58.
Tested by listing contents of a volume, and packing a new clm file.

Okay, I thought it involved using the default keyword in enum, not defining a new member that was Default, but not the keyword. I deleted it. Also moved the static_assert to the header file.


Problems importing bert vectorizer on windows

When trying to import bert_vectorizer it shows the error
ImportError: cannot import name 'BertModel' from 'transformers'
BertModel doesn't seem to exist in transformers.
Looking at transformers.init.py it seems that BertModel is only imported if torch is available.
Torch doesn't automatically install when installing wellcomeml in windows. I think this is because you have to pip install torch slightly differently in windows (pip install torch==1.5.1+cpu torchvision==0.6.1+cpu -f https://download.pytorch.org/whl/torch_stable.html, from here - https://pytorch.org/).
Even though I have installed torch separately transformers still wont import BertModel.

Have you tried in stalling transformers after torch, i.e.:
pip install --no-cache-dir transformers==2.0.0

You might have to uninstall it and try it again.

Thanks, this worked

is this something we should add in as logic to the Makefile so future Windows users don't have the same problem?

I think it will be hard to add/test a make recipe to windows (if @nonieA wants to open a PR on that happy to do so), but as a minimum we could update the documentation to include this workaround.


akkoma: init at 3.2.0

Description of changes
Packaging Akkoma, a fork of Pleroma: https://coffee-and-dreams.uk/development/2022/06/24/akkoma.html.
Special thanks to @yu-re-ka for debugging.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[ ] Fits CONTRIBUTING.md.

 yaya | error: hash mismatch in fixed-output derivation '/nix/store/b5s2gx9yj2rchx9wfj4iihswp9yl5938-akkoma.drv':
 yaya |          specified: sha256-mJtgBkoENgDE8KAhtwm4ual8pQnGQRJMvPgkubVTPQM=
 yaya |             got:    sha256-TGC7pxUoK6krgEB7Mz9pd5QMl2VFRJ3robaz4c+kqFE=

 yaya | error: hash mismatch in fixed-output derivation '/nix/store/b5s2gx9yj2rchx9wfj4iihswp9yl5938-akkoma.drv':
 yaya |          specified: sha256-mJtgBkoENgDE8KAhtwm4ual8pQnGQRJMvPgkubVTPQM=
 yaya |             got:    sha256-TGC7pxUoK6krgEB7Mz9pd5QMl2VFRJ3robaz4c+kqFE=

I’ll correct the mistake and push the corrections after running the tests successfully.

Unlike pleroma, they don't ship a pre-built frontend. I think that's good, but also means that we should probably provide users with a way of installing a frontend, and possibly even package Akkoma's soft-fork of pleroma-fe.

It is a bit unfortunate that their mix.exs still specifies "pleroma" and they don't properly update the version attribute either. mixRelease requires the pname and version to match what's in the mix.exs. Would be nice if we could find a way around that. The latest upstream release is tagged as stable-202209.

I could adjust mix.exs automatically to correspond to the tagged version number. I tried that successfully for 3.2.0.

Unlike pleroma, they don't ship a pre-built frontend. I think that's good, but also means that we should probably provide users with a way of installing a frontend, and possibly even add Akkoma's soft-fork of pleroma-fe as a source-built package.

Installing a frontend can be done with pleroma_ctl, but the RELEASE_COOKIE environment variable has to be set manually.
Frontends could also be included as bind mounts into the Nix store.

I could adjust mix.exs automatically to correspond to the tagged version number. I tried that successfully for 3.2.

That could work for now, but maybe someone should talk to upstream about matching the version in mix.exs to the tag name for future releases.
While there are also tagged versions with version numbers, the actual stable releases with changelogs etc. are named after by the month they were released in:
https://meta.akkoma.dev/c/releases/7

Installing a frontend can be done with pleroma_ctl, but the RELEASE_COOKIE environment variable has to be set manually.

Well, yeah, that works. But it is not really great that the user has to set up the state to get a usable setup.

I could adjust mix.exs automatically to correspond to the tagged version number. I tried that successfully for 3.2.

That could work for now, but maybe someone should talk to upstream about matching the version in mix.exs to the tag name for future releases.
While there are also tagged versions with version numbers, the actual stable releases with changelogs etc. are named after by the month they were released in: https://meta.akkoma.dev/c/releases/7

It looks to me that the stable tags just point to the commit that merges the release from the develop branch into the stable branch. So stable-202209 essentially points to the same code as v3.2.0. In that sense the versions already match the tag names.

Installing a frontend can be done with pleroma_ctl, but the RELEASE_COOKIE environment variable has to be set manually.

Well, yeah, that works. But it is not really great that the user has to set up the state to get a usable setup.

This problem might be difficult to solve without adding a new module for Akkoma.
Such a module could allow the user to specify the frontends, defaulting to pleroma-fe and admin-fe as built from the store.

Installing a frontend can be done with pleroma_ctl, but the RELEASE_COOKIE environment variable has to be set manually.

Well, yeah, that works. But it is not really great that the user has to set up the state to get a usable setup.

This problem might be difficult to solve without adding a new module for Akkoma.
Such a module could allow the user to specify the frontends, defaulting to pleroma-fe and admin-fe as built from the store.

My first attempt at packaging the frontend: https://github.com/illdefined/nixpkgs/commit/b9ed43e222246df9b52674fc43b11d07eb9b9108
I was not sure about the canonical way of integrating it with the rest of the Nix packages.
admin-fe appears to be built‐in, but there is masto-fe and firebird-fe.

Instead of relying on the pleroma module, this PR now introduces a dedicated akkoma module that uses elixirConf to generate the configuration file from an attribute set, supporting secrets through replace-secret. I copy‐edited the pleroma test to make use of it.
Since Akkoma does not bundle a frontend, the module takes care of that and includes a few options to override its default settings, styles and themes.
I wasn’t sure about the proper way to integrate the (primary) frontend with the rest of nixpkgs. It is currently available at akkoma-frontends.pleroma-fe.

mixRelease requires the pname and version to match what's in the mix.exs.
This hasn’t been an issue for me

as for admin-fe, i applied the following patch to get it to build:
https://git.chir.rs/darkkirb/nix-packages/src/branch/main/akkoma/akkoma-admin.patch

lgtm
Personally my setup generates static_dir in a derivation, although this is probably better suited towards a local nixos module
Also this doesn’t contain admin_fe which might make moderation harder on multi-user instances

I reworked the frontend integration somewhat so that it should both be easy to have a setup that simply works out of the box or extensive modifications with a custom derivation.
With respect to admin_fe I was confused by the out‐of‐date information in the Git repository. I’ll try to package it and add it in the module as default admin frontend.

I corrected an error in the module documentation concerning the encoding of some of the secrets. They aren’t actually base64‐encoded, even though the mix tasks use base64 to generate them. Since they are used as inputs for hash functions, I assume that any byte string with sufficient entropy is suitable, although strings which are not valid UTF-8 or contain null bytes may pose problems when treated as character strings.
Limiting secrets to unreserved characters according to RFC 3986, like in the examples in the module documentation, provides a more than sufficient amount of entropy (~386.84 bits for the secrets and ~48.36 bits for the salts) and will probably never break any tooling involved in handling them.
Assuming that making the frontends available under akkoma-frontends is acceptable, I feel like this PR is ready now.

This is a service which goes under services instead of networking.

It’s currently placed under services/networking like pleroma. I could move it to services/web-apps like mastodon and peertube which also implement ActivityPub.

Just as a note: I added a dependency on network-online.target because this service actively connects to other ActivityPub servers on the Internet. Just ordering it after network.target may not be sufficient to ensure proper functionality.

Also please amend the review commits to the init ones.

Reordered and squashed the commits.

@illdefined A small design question: do you think an easy-mode option for reverse-proxying should be added (e.g. using nginx) with some sensible defaults if enabled and the ability to customize the resulting vhost? A lot of modules seem to contain it (e.g. bookstack, dolibarr, fluidd, matomo, nextcloud -- though the latter does only seem to support nginx from what I remember, and I had to hack it to make it work with Caddy)

I personally don’t like it that much because of the high variability of deployment environments. While the test and the config guide rely on nginx, my own playground uses HAProxy, which may be more difficult to configure but better suited to my needs.
I’ll give it some more thought.

You can use some sql to check if the database exists before executing the setup script and secrets can be replaced from files with replace-secret or use systemd's EnvironmentFile.
NixOS mostly uses nginx and if there is more than adding the server to an upstream there should be some help to get it into nginx. The nextcloud module is a good example.

When encountering issues with configuring the database in an idempotent fashion, the creators of the matrix-synapse module opted to not configure the database at all, leaving that to the user. Perhaps we could go the same way concerning database configuration.
As for nginx, I re-read the previous guide again and I think I should retract my note about potential easy-mode reverse proxy option, as the config seems trivial.
As a result, I have no more concerns about this module.

After a quick review by @Kloenk I made the following changes:

Package

Static frontend files are digested and compressed (cf. https://hexdocs.pm/phoenix/Mix.Tasks.Phx.Digest.html)

Module

The RELEASE_COOKIE variable is set to the cookie itself instead of its path.
The HTTP listener can be bound to a Unix socket.

@Kloenk, do you think that adjusting the socket permissions in ExecStartPost (see https://github.com/NixOS/nixpkgs/pull/192285/commits/b2e7d091cd10761f1154873756c4de80de5b72dd#diff-d907f02975661d36f2b743feb0c62acb21de8b98b769d1dd9915c6c91765fd35R283) is reasonable or would you rather patch Akkoma to set them?

@Kloenk, do you think that adjusting the socket permissions in ExecStartPost (see socketScript) is reasonable or would you rather patch Akkoma to set them?

Doing it in ExecStartPost sounds very reasonable to me, as I did not find a way to patch rancher (the underlying lib) to create the proper permission on the socket/use systemd socket. So I think the script is the better way to do it.

Added documentation on media proxy configuration, federation and upload filters.

Any objections to merging this?

Updated versions of akkoma and pleroma-fe:
diff --git a/pkgs/servers/akkoma/default.nix b/pkgs/servers/akkoma/default.nix
index 5fff455608a..9b3635cc3d2 100644
--- a/pkgs/servers/akkoma/default.nix
+++ b/pkgs/servers/akkoma/default.nix
@@ -1,7 +1,7 @@
 { lib
 , beamPackages
 , fetchFromGitea, fetchFromGitHub, fetchFromGitLab
-, cmake, file
+, cmake, file, libxcrypt
 , writeText
 , nixosTests
 , ...
@@ -9,14 +9,14 @@

 beamPackages.mixRelease rec {
   pname = "pleroma";
-  version = "3.2.0";
+  version = "3.3.1";

   src = fetchFromGitea {
     domain = "akkoma.dev";
     owner = "AkkomaGang";
     repo = "akkoma";
     rev = "v${version}";
-    hash = "sha256-dVWmqPan0trvoU6FsG/U7Aj2Z6KnxpZ3AlpaI00b1Qc=";
+    hash = "sha256-sdQki7/tiaDxdJRv3G95i/6TqP/UjyTLJdbptVGZSIA=";
   };

   postPatch = ''
@@ -65,6 +65,8 @@ beamPackages.mixRelease rec {
           sha256 = "sha256-ZYhZTe7cTITkl8DZ4z2IOlxTX5gnbJImu/lVJ2ZjR1o=";
         };

+        buildInputs = [ libxcrypt ];
+
         postInstall = ''
           mv $out/lib/erlang/lib/crypt-${version}/priv/{source,crypt}.so
         '';
diff --git a/pkgs/servers/akkoma/pleroma-fe/default.nix b/pkgs/servers/akkoma/pleroma-fe/default.nix
index abdf6fa629f..50795aa4055 100644
--- a/pkgs/servers/akkoma/pleroma-fe/default.nix
+++ b/pkgs/servers/akkoma/pleroma-fe/default.nix
@@ -7,19 +7,19 @@

 stdenv.mkDerivation rec {
   pname = "pleroma-fe";
-  version = "unstable-2022-09-10";
+  version = "unstable-2022-10-08";

   src = fetchFromGitea {
     domain = "akkoma.dev";
     owner = "AkkomaGang";
     repo = "pleroma-fe";
-    rev = "d7499a1f91fb5e4ec947445c566f8ae41cec6c72";
-    hash = "sha256-I+d+oTOTNaMPkk9vdFHhm/g8EtPTZnsWv8yPUa0uNHQ=";
+    rev = "c8c8d4082706802677beb0fb26073eeaaafce879";
+    hash = "sha256-TUppR+yaafWuYNm0x812IDdRxj4iXpcZtdHgqvIDgZc=";
   };

   offlineCache = fetchYarnDeps {
     yarnLock = src + "/yarn.lock";
-    hash = "sha256-UkSdVRqFp6bj7zz4ZRMD47tVpuS7x7F1MiEei+87kyg=";
+    hash = "sha256-EKlIJ2PfCvgE6b+VVgZo+xn1h+drZiNYqKenXOukLYI=";
   };

   nativeBuildInputs = [

Corrected a reference in an option description:
diff --git a/nixos/modules/services/web-apps/akkoma.nix b/nixos/modules/services/web-apps/akkoma.nix
index 7232e635351..eafff1ce2b0 100644
--- a/nixos/modules/services/web-apps/akkoma.nix
+++ b/nixos/modules/services/web-apps/akkoma.nix
@@ -342,7 +342,7 @@ in {
             and the ownership of the database.

             The database settings are configured through
-            [{option}`config.services.akkoma.config.":pleroma"."Pleroma.Repo"`](#opt-services.akkoma.config._pleroma.Pleroma.Repo).
+            [{option}`config.services.akkoma.config.":pleroma"."Pleroma.Repo"`](#opt-services.akkoma.config.__pleroma_._Pleroma.Repo_).

             If disabled, the database has to be set up manually:

Use the headless variant of ffmpeg:
diff --git a/nixos/modules/services/web-apps/akkoma.nix b/nixos/modules/services/web-apps/akkoma.nix
index eafff1ce2b0..613a6adce89 100644
--- a/nixos/modules/services/web-apps/akkoma.nix
+++ b/nixos/modules/services/web-apps/akkoma.nix
@@ -836,7 +836,7 @@ in {
     };

     systemd.services.akkoma = let
-      runtimeInputs = with pkgs; [ coreutils gawk gnused imagemagick ffmpeg exiftool ];
+      runtimeInputs = with pkgs; [ coreutils gawk gnused imagemagick ffmpeg-headless exiftool ];
     in {
       description = "Akkoma social network";
       documentation = [ "https://docs.akkoma.dev/stable/" ];

Summary

Set PropagatesReloadTo in the correct section of the akkoma-config service unit,
move environment setup from akkoma service unit into a wrapper script (previously just used for release cookie),
add a wrapper around pleroma_ctl using that script, with installation controlled by config.services.akkoma.installWrapper (default true),
issue a warning if wrapper is installed but sudo disabled,
adjust documentation to

use wrapper,
enable nginx slice module for media proxy config, and
properly document media proxy config.

Diff
diff --git a/nixos/modules/services/web-apps/akkoma.md b/nixos/modules/services/web-apps/akkoma.md
index ee9f0512301..5689e588887 100644
--- a/nixos/modules/services/web-apps/akkoma.md
+++ b/nixos/modules/services/web-apps/akkoma.md
@@ -46,7 +46,7 @@ After the Akkoma service is running, the administration utility can be used to
 administrative user can be created with

```ShellSession
-$ sudo -u akkoma pleroma_ctl user new <nickname> <email> --admin --moderator --password <password>
+$ pleroma_ctl user new <nickname> <email> --admin --moderator --password <password>
 ```

 ## Proxy configuration {#modules-services-akkoma-proxy-configuration}
@@ -78,6 +78,9 @@ Without the media proxy function, Akkoma does not store any remote media like pi
 locally, and clients have to fetch them directly from the source server.

 ```nix
+# Enable nginx slice module distributed with Tengine
+services.nginx.package = pkgs.tengine;
+
 # Enable media proxy
 services.akkoma.config.":pleroma".":media_proxy" = {
   enabled = true;
@@ -107,7 +110,7 @@ nginx.virtualHosts."cache.fediverse.example.com" = {
       proxy_cache_key $host$uri$is_args$args$slice_range;
       proxy_set_header Range $slice_range;

-      # Decouple client and upstream requests
+      # Decouple proxy and upstream responses
       proxy_buffering on;
       proxy_cache_lock on;
       proxy_ignore_client_abort on;
diff --git a/nixos/modules/services/web-apps/akkoma.nix b/nixos/modules/services/web-apps/akkoma.nix
index 613a6adce89..9bf3ebaecf2 100644
--- a/nixos/modules/services/web-apps/akkoma.nix
+++ b/nixos/modules/services/web-apps/akkoma.nix
@@ -269,16 +269,38 @@ let
     '';
   };

-  cookieWrapper = name: pkgs.writeShellApplication {
-    inherit name;
+  envWrapper = pkgs.writeShellApplication {
+    name = "akkoma-env";
     text = ''
-      RELEASE_COOKIE="''${RELEASE_COOKIE:-$(<"''${RUNTIME_DIRECTORY:-/run/akkoma}/cookie")}" \
-        exec "${cfg.package}/bin/${name}" "$@"
+      cmd="$1"
+      shift
+
+      RUNTIME_DIRECTORY="''${RUNTIME_DIRECTORY:-/run/akkoma}"
+      AKKOMA_CONFIG_PATH="$RUNTIME_DIRECTORY/config.exs" \
+      ERL_EPMD_ADDRESS="${cfg.dist.address}" \
+      ERL_EPMD_PORT="${toString cfg.dist.epmdPort}" \
+      ERL_FLAGS="${concatStringsSep " " [
+        "-kernel inet_dist_use_interface '${erlAddr cfg.dist.address}'"
+        "-kernel inet_dist_listen_min ${toString cfg.dist.portMin}"
+        "-kernel inet_dist_listen_max ${toString cfg.dist.portMax}"
+      ]}" \
+      RELEASE_COOKIE="$(<"$RUNTIME_DIRECTORY/cookie")" \
+        exec "${cfg.package}/bin/$cmd" "$@"
     '';
   };

-  pleroma = cookieWrapper "pleroma";
-  pleromaCtl = cookieWrapper "pleroma_ctl";
+  userWrapper = pkgs.writeShellApplication {
+    name = "pleroma_ctl";
+    text = ''
+      if [ "''${1-}" == "update" ]; then
+        echo "OTP releases are not supported on NixOS." >&2
+        exit 64
+      fi
+
+      exec sudo -u ${escapeShellArg cfg.user} \
+        "${envWrapper}/bin/akkoma-env" pleroma_ctl "$@"
+    '';
+  };

   socketScript = if isAbsolutePath web.http.ip
     then writeShell {
@@ -405,6 +427,15 @@ in {
         '';
       };

+      installWrapper = mkOption {
+        type = types.bool;
+        default = true;
+        description = mdDoc ''
+          Whether to install a wrapper around `pleroma_ctl` to simplify administration of the
+          Akkoma instance.
+        '';
+      };
+
       termsOfService = mkOption {
         type = with types; nullOr path;
         default = null;
@@ -781,6 +812,11 @@ in {
   };

   config = mkIf cfg.enable {
+    warnings = optionals (!config.security.sudo.enable) [''
+      The pleroma_ctl wrapper enabled by the installWrapper option relies on
+      sudo, which appears to have been disabled through security.sudo.enable.
+    ''];
+
     users = {
       users."${cfg.user}" = {
         description = "Akkoma user";
@@ -797,8 +833,8 @@ in {
       description = "Akkoma social network configuration";
       reloadTriggers = [ configFile ] ++ secretPaths;

+      unitConfig.PropagatesReloadTo = [ "akkoma.service" ];
       serviceConfig = {
-        PropagateReloadTo = [ "akkoma.service" ];
         Type = "oneshot";
         RemainAfterExit = true;
         UMask = "0077";
@@ -856,17 +892,6 @@ in {
       confinement.packages = runtimeInputs;
       path = runtimeInputs;

-      environment = {
-        AKKOMA_CONFIG_PATH = "%t/akkoma/config.exs";
-        ERL_EPMD_ADDRESS = cfg.dist.address;
-        ERL_EPMD_PORT = toString cfg.dist.epmdPort;
-        ERL_FLAGS = concatStringsSep " " [
-          "-kernel inet_dist_use_interface '${erlAddr cfg.dist.address}'"
-          "-kernel inet_dist_listen_min ${toString cfg.dist.portMin}"
-          "-kernel inet_dist_listen_max ${toString cfg.dist.portMax}"
-        ];
-      };
-
       serviceConfig = {
         Type = "exec";
         User = cfg.user;
@@ -890,10 +915,10 @@ in {
               "${val.package}:${staticDir}/frontends/${val.name}/${val.ref}:norbind")
             cfg.frontends;

-        ExecStartPre = "${pleromaCtl}/bin/pleroma_ctl migrate";
-        ExecStart = "${pleroma}/bin/pleroma start";
+        ExecStartPre = "${envWrapper}/bin/akkoma-env pleroma_ctl migrate";
+        ExecStart = "${envWrapper}/bin/akkoma-env pleroma start";
         ExecStartPost = socketScript;
-        ExecStop = "${pleroma}/bin/pleroma stop";
+        ExecStop = "${envWrapper}/bin/akkoma-env pleroma stop";
         ExecReload = "${pkgs.coreutils}/bin/kill -HUP $MAINPID";

         ProtectProc = "noaccess";
@@ -939,7 +964,7 @@ in {
       };
     };

-    environment.systemPackages = [ pleromaCtl ];
+    environment.systemPackages = mkIf (cfg.installWrapper) [ userWrapper ];

     services.nginx.virtualHosts = mkIf (cfg.nginx != null) {
       ${web.url.host} = mkMerge [ cfg.nginx {
diff --git a/nixos/modules/services/web-apps/akkoma.xml b/nixos/modules/services/web-apps/akkoma.xml
index bc79f910d8c..d06729ca281 100644
--- a/nixos/modules/services/web-apps/akkoma.xml
+++ b/nixos/modules/services/web-apps/akkoma.xml
@@ -59,7 +59,7 @@ services.akkoma.config = {
       with
     </para>
     <programlisting>
-$ sudo -u akkoma pleroma_ctl user new <nickname> <email> --admin --moderator --password <password>
+$ pleroma_ctl user new <nickname> <email> --admin --moderator --password <password>
 </programlisting>
   </section>
   <section xml:id="modules-services-akkoma-proxy-configuration">
@@ -95,6 +95,9 @@ services.nginx = {
         fetch them directly from the source server.
       </para>
       <programlisting language="nix">
+# Enable nginx slice module distributed with Tengine
+services.nginx.package = pkgs.tengine;
+
 # Enable media proxy
 services.akkoma.config.":pleroma".":media_proxy" = {
   enabled = true;
@@ -124,7 +127,7 @@ nginx.virtualHosts."cache.fediverse.example.com" = {
       proxy_cache_key $host$uri$is_args$args$slice_range;
       proxy_set_header Range $slice_range;

-      # Decouple client and upstream requests
+      # Decouple proxy and upstream responses
       proxy_buffering on;
       proxy_cache_lock on;
       proxy_ignore_client_abort on;

Akkoma may not unlink its listening socket by itself. Removing it during service shutdown helps:
diff --git a/nixos/modules/services/web-apps/akkoma.nix b/nixos/modules/services/web-apps/akkoma.nix
index 9bf3ebaecf2..0d90cc37387 100644
--- a/nixos/modules/services/web-apps/akkoma.nix
+++ b/nixos/modules/services/web-apps/akkoma.nix
@@ -919,6 +919,9 @@ in {
         ExecStart = "${envWrapper}/bin/akkoma-env pleroma start";
         ExecStartPost = socketScript;
         ExecStop = "${envWrapper}/bin/akkoma-env pleroma stop";
+        ExecStopPost = if isAbsolutePath web.http.ip
+          then "${pkgs.coreutils}/bin/rm -f '${web.http.ip}'"
+          else null;
         ExecReload = "${pkgs.coreutils}/bin/kill -HUP $MAINPID";

         ProtectProc = "noaccess";

Allow users to override packages made available to the service unit:
diff --git a/nixos/modules/services/web-apps/akkoma.nix b/nixos/modules/services/web-apps/akkoma.nix
index 0d90cc37387..90c84ab1e29 100644
--- a/nixos/modules/services/web-apps/akkoma.nix
+++ b/nixos/modules/services/web-apps/akkoma.nix
@@ -436,6 +436,22 @@ in {
         '';
       };

+      extraPackages = mkOption {
+        type = with types; listOf package;
+        default = with pkgs; [ exiftool ffmpeg_5-headless graphicsmagick-imagemagick-compat ];
+        defaultText = literalExpression "with pkgs; [ exiftool graphicsmagick-imagemagick-compat ffmpeg_5-headless ]";
+        example = literalExpression "with pkgs; [ exiftool imagemagick ffmpeg_5-full ]";
+        description = mdDoc ''
+          List of extra packages to include in the executable search path of the service unit.
+          These are needed by various configurable components such as:
+
+          - ExifTool for the `Pleroma.Upload.Filter.Exiftool` upload filter,
+          - ImageMagick for still image previews in the media proxy as well as for the
+            `Pleroma.Upload.Filters.Mogrify` upload filter, and
+          - ffmpeg for video previews in the media proxy.
+        '';
+      };
+
       termsOfService = mkOption {
         type = with types; nullOr path;
         default = null;
@@ -872,7 +888,7 @@ in {
     };

     systemd.services.akkoma = let
-      runtimeInputs = with pkgs; [ coreutils gawk gnused imagemagick ffmpeg-headless exiftool ];
+      runtimeInputs = with pkgs; [ coreutils gawk gnused ] ++ cfg.extraPackages;
     in {
       description = "Akkoma social network";
       documentation = [ "https://docs.akkoma.dev/stable/" ];

Akkoma may not unlink its listening socket by itself. Removing it during service shutdown helps:

@illdefined Sorry for the drive-by comment, but can't you rely on systemd removing the whole RuntimeDirectory after the service si stopped?

Akkoma may not unlink its listening socket by itself. Removing it during service shutdown helps:

@illdefined Sorry for the drive-by comment, but can't you rely on systemd removing the whole RuntimeDirectory after the service si stopped?

The RuntimeDirectory is set up by the akkoma-config service. It would be possible to have it removed by the main akkoma service, but then the configuration file would have to be regenerated every time the service is restarted.
Considering that I only split up the services to enable better separation of privileges – the configuration script requires access to secret files likely outside of the main service’s mount namespace – and that it does not take a lot of time, that should be acceptable. Or I could just place the generated config into CacheDirectory.
But as the socket does not necessarily need to reside in RuntimeDirectory and will impair service restart if not unlinked, I still think that using ExecStopPost is reasonable.

Minor change to the documentation: Add the media proxy to the same virtual host as the main Akkoma service.
The standard frontend Pleroma-FE uses JavaScript fetch() to preload still images, but the Content-Security-Policy header generated by Akkoma does not include media sources in connect-src. As a result, preload requests will be blocked if images are loaded across origins (cf. https://akkoma.dev/AkkomaGang/akkoma/issues/263).
Users with complex setups may adjust the Content-Security-Policy header until the issue is resolved or just do without image preloading, but I think that the example given in the documentation should simply provide images from the same origin.
diff --git a/nixos/modules/services/web-apps/akkoma.md b/nixos/modules/services/web-apps/akkoma.md
index 5689e588887..e33ad4aab90 100644
--- a/nixos/modules/services/web-apps/akkoma.md
+++ b/nixos/modules/services/web-apps/akkoma.md
@@ -84,7 +84,6 @@ services.nginx.package = pkgs.tengine;
 # Enable media proxy
 services.akkoma.config.":pleroma".":media_proxy" = {
   enabled = true;
-  base_url = "https://cache.fediverse.example.com";
   proxy_opts.redirect_on_failure = true;
 };

@@ -98,8 +97,8 @@ services.nginx.commonHttpConfig = ''
     inactive=1y use_temp_path=off;
 '';

-nginx.virtualHosts."cache.fediverse.example.com" = {
-  locations."/" = {
+services.akkoma.nginx = {
+  locations."/proxy" = {
     proxyPass = "http://unix:/run/akkoma/socket";

     extraConfig = ''
@@ -121,7 +120,6 @@ nginx.virtualHosts."cache.fediverse.example.com" = {

       # Allow serving of stale items
       proxy_cache_use_stale error timeout invalid_header updating;
-
     '';
   };
 };
diff --git a/nixos/modules/services/web-apps/akkoma.xml b/nixos/modules/services/web-apps/akkoma.xml
index d06729ca281..70d6273fff7 100644
--- a/nixos/modules/services/web-apps/akkoma.xml
+++ b/nixos/modules/services/web-apps/akkoma.xml
@@ -101,7 +101,6 @@ services.nginx.package = pkgs.tengine;
 # Enable media proxy
 services.akkoma.config.":pleroma".":media_proxy" = {
   enabled = true;
-  base_url = "https://cache.fediverse.example.com";
   proxy_opts.redirect_on_failure = true;
 };

@@ -115,8 +114,8 @@ services.nginx.commonHttpConfig = ''
     inactive=1y use_temp_path=off;
 '';

-nginx.virtualHosts."cache.fediverse.example.com" = {
-  locations."/" = {
+services.akkoma.nginx = {
+  locations."/proxy" = {
     proxyPass = "http://unix:/run/akkoma/socket";

     extraConfig = ''
@@ -138,7 +137,6 @@ nginx.virtualHosts."cache.fediverse.example.com" = {

       # Allow serving of stale items
       proxy_cache_use_stale error timeout invalid_header updating;
-
     '';
   };
 };

akkoma: 3.3.1 → 3.4.0
pleroma-fe: unstable-2022-10-08 → unstable-2022-11-12

I got it working by setting
systemd.services.pleroma.environment.RELEASE_DISTRIBUTION = "none";
systemd.services.akkoma.environment.RELEASE_DISTRIBUTION = "none";

I got it working by setting
systemd.services.pleroma.environment.RELEASE_DISTRIBUTION = "none";
systemd.services.akkoma.environment.RELEASE_DISTRIBUTION = "none";

I assume that this makes administration of the instance via pleroma_ctl impossible. But explicitly setting RELEASE_NAME to something other than pleroma should allow running Pleroma and Akkoma on the same host without having to disable distribution. I’ll adjust the module accordingly.

This seems to break cross platform evaluation, which works without having the module enabled.

What's the migration path from a running pleroma instance? Is it mentioned in the docs somewhere I couldn't find?

What's the migration path from a running pleroma instance? Is it mentioned in the docs somewhere I couldn't find?

https://docs.akkoma.dev/stable/installation/migrating_to_akkoma/ I'd assume that someone would need to create a NixOS specific migration path though

What's the migration path from a running pleroma instance? Is it mentioned in the docs somewhere I couldn't find?

https://docs.akkoma.dev/stable/installation/migrating_to_akkoma/ I'd assume that someone would need to create a NixOS specific migration path though

yeah some of these things are not applicable to a nix-based setup. you certainly don't want to use pleroma_ctl to directly manage your pleroma install...

What's the migration path from a running pleroma instance? Is it mentioned in the docs somewhere I couldn't find?

https://docs.akkoma.dev/stable/installation/migrating_to_akkoma/ I'd assume that someone would need to create a NixOS specific migration path though

It should be as easy as copying the database and the upload directory or pointing Akkoma to the existing ones, and enabling the service. The database migrations are run automatically during startup of the service.
Should I add a chapter to the manual?

I've migrated an existing non-NixOS Pleroma install to this module by doing pretty much that, plus rewriting the config in Nix.
The only problem was that the initial migration takes a while, and because it's run in ExecStartPre, it's subject to getting killed by Systemd before completion. Upping TimeoutStartSec might be a good idea in this module.

86802981ba642c284e2bd7022071b43c9b72a770 seems to fix it, but also maybe we should just symlink that stuff instead of copying it?

Looks like frontend building is currently broken - the built files end up in ...akkoma-static/frontends/$name/stable/$storepath/ instead of ...akkoma-static/frontends/$name/stable.

I’ll fix it.

K900@8680298 seems to fix it, but also maybe we should just symlink that stuff instead of copying it?

Symlinking would be an option, but that might break confinement.

K900@8680298 seems to fix it, but also maybe we should just symlink that stuff instead of copying it?

Symlinking would be an option, but that might break confinement.

Couldn't the store paths for the frontends be allowed into the sandbox? Presumably we know them already.

K900@8680298 seems to fix it, but also maybe we should just symlink that stuff instead of copying it?

Symlinking would be an option, but that might break confinement.

Couldn't the store paths for the frontends be allowed into the sandbox? Presumably we know them already.

The most reliable option would probably be to walk the static directory tree, resolving all symlinks to collect the relevant paths. That would also work with a custom static dir derivation.

The static directory is now built by symlinking all the configured frontends etc.
With confinement enabled, all store paths in the closure are bind‐mounted to make them accessible. I used pkgs.closureInfo to collect the list of paths, but I am not sure if that is the best way to do it.

This is on track to become the most thoroughly reviewed PR in existence 😀

Let's get this in then. We can always add more later.

I just noticed that running both akkoma and pleroma still fails as described in https://github.com/NixOS/nixpkgs/pull/192285#pullrequestreview-1187247030. (I forgot to comment out my workaround)
Opened #208027 to track this issue.


Moving Issue Labeler to official repo

Moving the GitHub Issue Labeler into an official repository.

@microsoft-github-policy-service agree company="Microsoft"

You should add a ci.yml file similar to https://github.com/Azure/azure-sdk-tools/blob/816e86402f50a79b4b2f08688c087e1bf9217c9b/tools/webhook-router/ci.yml#L25 assuming this is a azure functions project.

/azp run prepare-pipelines

/check-enforcer override


Dead Hyperlink

The link at https://pypi.org/project/starlette-api/ still points to https://starlette-api.perdy.io/. Maybe you can add a permanent redirect to the new homepage.

Good point. There is a permanent redirect now.
Thanks !


[BUG] Please fix light theme

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior
Light theme is too dim. Not enough contrast for comfortable work.

Expected Behavior
No response
Steps To Reproduce

Run Altair GraphQL Client
Switch theme to light

Environment
- OS: MacOS
- Version: 7.3.4

Additional context
No response

Hey! Do you have any specifics you want to call out?

@imolorhe Hey man, i would like to contribute into this issue, can you assign me! Thanks


Removed "medium" dictionary from test_data and modified the tests dependen on it

We have two entried(keys) named "small" and "medium" inside test_data.py to specify the vm properties and access details. However, both contain the same values. So deleted one of them and also modified the tests accordingly.
Ran all the tests and all of them passed.
Test advanced zone virtual router ... === TestName: test_advZoneVirtualRouter | Status : SUCCESS ===
ok
Test Deploy Virtual Machine ... === TestName: test_deploy_vm | Status : SUCCESS ===
ok
Test Multiple Deploy Virtual Machine ... === TestName: test_deploy_vm_multiple | Status : SUCCESS ===
ok
Test Stop Virtual Machine ... === TestName: test_01_stop_vm | Status : SUCCESS ===
ok
Test Start Virtual Machine ... === TestName: test_02_start_vm | Status : SUCCESS ===
ok
Test Reboot Virtual Machine ... === TestName: test_03_reboot_vm | Status : SUCCESS ===
ok
Test destroy Virtual Machine ... === TestName: test_06_destroy_vm | Status : SUCCESS ===
ok
Test recover Virtual Machine ... === TestName: test_07_restore_vm | Status : SUCCESS ===
ok
Test migrate VM ... === TestName: test_08_migrate_vm | Status : SUCCESS ===
Test to create service offering ... === TestName: test_01_create_service_offering | Status : SUCCESS ===
ok
Test to update existing service offering ... === TestName: test_02_edit_service_offering | Status : SUCCESS ===
ok
Test to delete service offering ... === TestName: test_03_delete_service_offering | Status : SUCCESS ===
ok
Test to change service to a small capacity ... === TestName: test_04_change_offering_small | Status : SUCCESS ===
ok

LGTM

All the tests success results are given in the PR.
LGTM