performance regression with new archiver code

Output of restic 0.8.3 - compiled with go1.10 on darwin/amd64
password is correct
using parent snapshot 41fe8ca6
scan [/Users/thomasko]
scanned 203642 directories, 923001 files in 1:40
[2:50] 100.00%  350.991 GiB / 350.988 GiB  1126648 / 1126643 items  1 errors  ETA 0:00
duration: 2:50
snapshot 84d57f41 saved
281.77 real       317.25 user       169.60 sys
Output of restic 0.8.3 (v0.8.3-217-g8026e6fd) compiled with go1.10.1 on darwin/amd64
[59:00] 36.97%  473781 files 129.896 GiB, total 922319 files 351.322 GiB, 3 errors ETA 1:40:34
[1:41:27] 94.40%  693174 files 331.637 GiB, total 922319 files 351.322 GiB, 18 errors ETA 6:01
[2:11:05] 99.63%  917467 files 350.037 GiB, total 922319 files 351.322 GiB, 18 errors ETA 0:28
8045.00 real      2588.35 user       588.89 sys
How did you run restic exactly?
i run these commands on an existing repository.
/usr/bin/time restic backup --hostname patmos
--exclude '.DS_Store'
--exclude '/Users/thomasko/Library/Caches/'
--exclude '/Users/thomasko/Library/Developer/'
--exclude '/Users/thomasko/Library/Mail/'
--exclude '/Users/thomasko/Library/Logs/'
--exclude '/Users/thomasko/.npm/'
--exclude '/Users/thomasko/.sonar/'
--exclude '/Users/thomasko/.m2/repository/'
--exclude '/Users/thomasko/.gradle/caches/'
--exclude '/Users/thomasko/.gradle/daemon/'
--exclude '/Users/thomasko/.gradle/wrapper/'
--exclude '/Users/thomasko/.gradle/native/'
--exclude '/Users/thomasko/.cache/'
/Users/thomasko
What backend/server/service did you use to store the repository?
the repository is on a local server. S3 (minio) - both executions 0.8.3 and 0.8.3-217 (compiled from master) ran with the same repository.
Expected behavior
run as fast as before
Actual behavior
runs slower
Steps to reproduce the behavior
run restic against an existing repository.
Do you have any idea what may have caused this?
i ran restic-0.8.3-release, restic-v0.8.3-191-g6d4c40f8 and restic-v0.8.3-217-g8026e6fd but i only have the above output available. - i ran the backups multiple times to exclude a possible server slowdown causing this behaviour. but the only difference (repository server, network, backup source, running programs at the time of backup, ...) was the restic version.
Do you have an idea how to solve the issue?
maybe printing the filename during backup causes the performance problem?

Output of restic 0.8.3 (v0.8.3-221-g26757ae2) compiled with go1.10.2 on darwin/amd64
repository 7e5130b9 opened successfully, password is correct
Files:         245 new,   664 changed, 921454 unmodified
Dirs:            0 new,     1 changed,     0 unmodified
Added:      556.837 MiB
processed 922363 files, 351.563 GiB in 1:21:47
snapshot 3a0895a0 saved
4907.81 real       309.80 user       281.43 sys
Observation
Maybe only the first run of the new archiver code causes a slowdown because of the file metadata rewrite - so i did a second run with the new-archiver code. The slowdown here is not as bad as the first run but still "extremely noticable"...

Hey, thanks for the report! What hardware is that? It sounds like a potent OS X machine with a fast SSD, is that the case?

Maybe only the first run of the new archiver code causes a slowdown because of the file metadata rewrite - so i did a second run with the new-archiver code. The slowdown here is not as bad as the first run but still "extremely noticable"...

You're right in your analysis: The first backup with the new archiver code re-reads everything, which explains the runtime. This is due to #549 being resolved, and the directory structure within the snapshot will therefore be different.
You can test your theory with the status output by running restic backup [...] | cat. When stdout is piped into cat, restic detects that it does not have a terminal, and will not print any status display.
Also, what's different to the old archiver code: We're using much less concurrent file access. By default, only two files are read in parallel, according to our experiments this seems to be the sweet spot for most machines. If you like to try different values, you can change it here: https://github.com/restic/restic/blob/26757ae2e53e17b75f98c9bc9c74d343ea226b6d/internal/archiver/archiver.go#L98-L113
Please report back!

Sorry for the late reply - testing took a while
summary
no significant differences in performance ... results below.
machine: MacBook Pro (latest model with Core i7 and 1TB SSD, APFS encrypted)
result: pipe output to 'cat'
restic backup | cat ...
5141.78 real       331.37 user       279.78 sys
result: concurrent set to 4
restic 0.8.3 (v0.8.3-223-g2a672588-dirty) compiled with go1.10.2 on darwin/amd64
repository 7e5130b9 opened successfully, password is correct
Files:          61 new,   368 changed, 921962 unmodified
Dirs:            0 new,     1 changed,     0 unmodified
Added:      287.575 MiB
processed 922391 files, 351.995 GiB in 1:25:05
snapshot cd9ad2d7 saved
5106.11 real       308.92 user       278.84 sys
result: concurrent set to 16
restic 0.8.3 (v0.8.3-223-g2a672588-dirty) compiled with go1.10.2 on darwin/amd64
repository 7e5130b9 opened successfully, password is correct
Files:        2667 new,   784 changed, 921584 unmodified
Dirs:            0 new,     1 changed,     0 unmodified
Added:      184.121 MiB
processed 925035 files, 352.036 GiB in 1:27:01
snapshot 222abb7b saved
5222.08 real       307.56 user       275.67 sys

Hm, interesting. The stats at the end say that it does not re-read all the data. So where does it spend so much time?
Did you maybe disable the local cache? Can you maybe try a backup with 0.8.3 again, to see if it still fast? (maybe you need to run two, because the first one may be very slow again, it may be re-read all local files)...
One thing you could try is building restic with the profile tag, go run build.go --tags profile, and then running it like this to write a CPU profile:
$ ./restic --cpu-profile . backup [...]

It'll then write a file cpu.pprof into the current directory. Maybe also do that with a so-called "block profile", via --block-profile .. You can then upload the files (there's nothing secret in them, just timing information and symbols from the program) somewhere for me to look at.
Another idea would be to find a small subset (maybe one directory or so) of your data that's fast archiving with 0.8.3, and much slower with the current master code. Then we could run with backup -v -v and maybe write a debug log (go run build.go -tags debug, then DEBUG_LOG=log ./restic backup [...]).
I don't have any more ideas right now :)
Thanks!

a quick feedback:

i haven't explicitly disabled the cache (please see the restic call from my first posting)
my first take is running the new-archiver profilings
after that i will run plain 0.8.3 multiple times
it will take a little time

Great, thank you very much for your time and patience :)
I'm eager to find out what's going on here.

Hm, the profiles unfortunately don't tell me anything. The CPU profile shows that it spent just about 320 seconds CPU time overall, which matches with the output of time you got. So it must be something else than CPU usage.
I have a different idea: Maybe your directory is structured in a way which triggers a very odd corner case I didn't foresee when rewriting the archiver. Can you paste a rough structure (e.g. the output of tree) somwhere, maybe obfuscated file/dir names?
Do you have tons of files in a single directory? Or a very deep directory structure?

debug log
i have created a debug log, but it contains all (sensitive) file names ... i don't want to upload it publicly. But i have collected some more information fragments...
distribution chart of directory depth in backup set
occurence   depth
60 /                               1
414 //                              2
2353 ///                             3
4559 ////                            4
15392 /////                           5
30170 //////                          6
30289 ///////                         7
59849 ////////                        8
55796 /////////                       9
20075 //////////                     10
22863 ///////////                    11
33963 ////////////                   12
32604 /////////////                  13
11369 //////////////                 14
4930 ///////////////                15
3747 ////////////////               16
5005 /////////////////              17
1158 //////////////////             18
646 ///////////////////            19
127 ////////////////////           20
89 /////////////////////          21
21 //////////////////////         22
10 ///////////////////////        23
so i think its deeply nested ... - when i back up my mailserver there are many more files in a single directory - so that's also a scenario i have.
first run of restic 0.8.3-release
using parent snapshot 110aa9c6
scan [/Users/thomasko]
scanned 207499 directories, 932817 files in 1:49
error for thomasko/Library/Preferences/com.apple.AddressBook.plist.y5sJ9tb: Lstat: lstat /Users/thomasko/Library/Preferences/com.apple.AddressBook.plist.y5sJ9tb: no such file or directory
error for thomasko/cpu.pprof: Lstat: lstat /Users/thomasko/cpu.pprof: no such file or directory
warning for /Users/thomasko/restic-stable-firstrun.cast: file has changed
warning for /Users/thomasko/restic-stable-firstrun.cast: expected 23399135 bytes, saved 23399860 bytes
[1:15:01] 100.00%  351.792 GiB / 351.774 GiB  1140324 / 1140316 items  2 errors  ETA 0:00
duration: 1:15:01
snapshot 2be02880 saved
4625.49 real      4199.42 user       641.53 sys
the first run is slow ... that was clear.
second run of restic 0.8.3-release
using parent snapshot 2be02880
scan [/Users/thomasko]
scanned 207498 directories, 932825 files in 2:04
warning for /Users/thomasko/restic-stable-secondrun.cast: file has changed
warning for /Users/thomasko/restic-stable-secondrun.cast: expected 1857442 bytes, saved 1857672 bytes
[4:14] 100.00%  351.801 GiB / 351.799 GiB  1140323 / 1140323 items  0 errors  ETA 0:00
duration: 4:14
snapshot a7279208 saved
396.06 real       375.05 user       196.19 sys
the second run took only a little over 4 minutes for processing 930.000 files!
wild guess
the interesting part about this is that all new-archiver-backups took nearly about as much time as the first run of 0.8.3-release. i haven't looked at the code but could it be that 0.8.3-release uses the cache on its second run(s) and new-archiver not?
screencasts
i have attached the 3 asciinema screen casts:

restic-new-archiver.cast (run of new archiver code) - slow, so i only recorded the first 7 minutes; progressed to about 0,5 % finished.
restic-stable-firstrun.cast - slow
restic-stable-secondrun.cast - fast (again)
(you can run it with 'asciinema play ....cast' to get a live impression of the performance degradation)

screencasts.zip

Hm. Random observations:

The first run with stable spends 4200s (almost the complete run time of the process) in user mode, which means that it uses a lot of CPU. That's expected: it's hashing/encrypting all the data.

The second run with stable is short, still almost all time is spent in user mode, probably decrypting/comparing files

For a subsequent run with the new archiver code, it spends time somewhere else, not in user and not in kernel mode. It could be blocked by syscalls, or it could be waiting on something. We need to find out what that is.

Oh, I have a suspicion, would you mind running the new archiver code again with this patch applied? https://gist.github.com/0832a521b20837a6fc34a5b0549cc996

And please run it with strace -c -e getdents64 :)

Ok, scratch the strace, that'll just falsify the measurements (and won't work on darwin out of the box anyway).

Can you please confirm that the restic binary you compiled (the one which has the performance regression) was built without the debug tag?

first run of new-archiver without tags (again)
repository 7e5130b9 opened successfully, password is correct
Files:       924071 new,     0 changed,     0 unmodified
Dirs:            1 new,     0 changed,     0 unmodified
Added:      53.763 MiB
processed 924071 files, 351.800 GiB in 2:16:13
snapshot 628a3713 saved
8174.72 real      2445.02 user       592.21 sys
next run of new-archiver without tags
(with quiet)
5012.34 real       299.37 user       308.51 sys
run new-archiver without tags including patch
repository 7e5130b9 opened successfully, password is correct
Files:           9 new,   431 changed, 920748 unmodified
Dirs:            0 new,     1 changed,     0 unmodified
Added:      76.497 MiB
processed 921188 files, 351.492 GiB in 1:22:19
snapshot ca3153f2 saved
time spent in readdirnames(): 13.313692311s
4940.35 real       301.78 user       300.43 sys

Hm, very odd. I don't have any idea what's going on here...

wild guess - i stumbled over a performance regression in go 1.10 (https://github.com/golang/go/issues/24078) - maybe there are such effects in go 1.10.2 too...
To get some evidence i re-compile 0.8.3 myself with go 1.10.2 and run the tests. That evidence may not conclusive because the new-archiver code works differently and triggers other effects - i will try to run benchmarks with my data on freebsd (0.8.3 & new-archiver branches).

Another idea would be to compile and test the new archiver code with Go 1.9

i ran 0.8.3-release - recompiled with go1.10.2 - that thing is even faster that the 0.8.3-release using go 1.10!
firstrun (slow - expected)
password is correct
using parent snapshot ca3153f2
scan [/Users/thomasko]
scanned 206072 directories, 929769 files in 3:16
[1:22:53] 100.00%  351.372 GiB / 351.372 GiB  1135840 / 1135841 items  0 errors  ETA 0:00
duration: 1:22:53
snapshot d5d5c339 saved
5199.23 real      4322.91 user       682.33 sys
secondrun over vpn (official build runs in 4:14 on local network!!!)
password is correct
using parent snapshot d5d5c339
scan [/Users/thomasko]
scanned 206072 directories, 929760 files in 1:51
[2:44] 100.00%  351.379 GiB / 351.379 GiB  1135833 / 1135832 items  0 errors  ETA 0:00
duration: 2:44
snapshot 35663b66 saved
289.06 real       305.34 user       163.90 sys

i ran 0.8.3-release - recompiled with go1.10.2 - that thing is even faster that the 0.8.3-release using go 1.10!

New Go releases usually improve the speed :)

i ran tests with 0.8.3-stable and new-archiver on freebsd. Data is exactly the same as with all previous backups, but now run on freebsd.
test sequence:

repo A - restic 0.8.3 - first run
repo A - restic 0.8.3 - second run
repo B - restic-new-archiver - first run
repo B - restic-new-archiver - second run
repo A - restic-new-archiver - first run
repo A - restic-new-archiver - second run

freebsd performs well in contrast to macosx (all the previous tests) with all the same data. new-archiver on freebsd shows both filename output lines changing during the backup process. on macosx the second line barely changes. the runtime implementation seems to be different on macosx somehow...
./restic_backup_newarchiver

restic init
Fatal: create key in repository at s3:http://archive1.home.intra:9000/restic.tests failed: repository master key and config already initialized

restic backup
repository 62fd6200 opened successfully, password is correct

Files:           0 new,     0 changed, 914483 unmodified
Dirs:            0 new,     0 changed,     4 unmodified
Added:      0 B
processed 914483 files, 351.127 GiB in 8:11
snapshot 1c5b59d7 saved
491,85 real       701,27 user       231,04 sys

./restic_backup_stable

restic init
created restic repository 62fd6200c7 at s3:http://archive1.home.intra:9000/restic.tests

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.

restic backup
password is correct
scan [/tank/sysbackup/patmos/Users/thomasko]
scanned 205757 directories, 923238 files in 3:50
[1:11:11] 100.00%  351.127 GiB / 351.127 GiB  1128995 / 1128995 items  0 errors  ETA 0:00
duration: 1:11:11
snapshot 236c9533 saved
4502,78 real     10839,38 user      1388,48 sys

./restic_backup_stable

restic init
Fatal: create key in repository at s3:http://archive1.home.intra:9000/restic.tests failed: repository master key and config already initialized

restic backup
password is correct
using parent snapshot 236c9533
scan [/tank/sysbackup/patmos/Users/thomasko]
scanned 205757 directories, 923238 files in 4:33
[8:50] 100.00%  351.127 GiB / 351.127 GiB  1128995 / 1128995 items  0 errors  ETA 0:00
duration: 8:50
snapshot f9a6c84f saved
812,99 real       925,89 user       302,11 sys

./restic_backup_newarchiver

restic init
created restic repository 6b08850c98 at s3:http://archive1.home.intra:9000/restic.newtests

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.

restic backup
repository 6b08850c opened successfully, password is correct
[1:12:47] 65.42%  497874 files 229.691 GiB, total 914483 files 351.127 GiB, 0 errors ETA 38:29

Files:       914483 new,     0 changed,     0 unmodified
Dirs:            4 new,     0 changed,     0 unmodified
Added:      294.696 GiB
processed 914483 files, 351.127 GiB in 2:02:48
snapshot ff43f151 saved
7368,54 real      8263,37 user      1244,54 sys
./restic_backup_newarchiver

restic init
Fatal: create key in repository at s3:http://archive1.home.intra:9000/restic.newtests failed: repository master key and config already initialized

restic backup
repository 6b08850c opened successfully, password is correct

Files:           0 new,     0 changed, 914483 unmodified
Dirs:            0 new,     0 changed,     4 unmodified
Added:      0 B
processed 914483 files, 351.127 GiB in 8:38
snapshot 9d1ed61e saved
518,73 real       736,89 user       258,30 sys
# ./restic_backup_newarchiver

restic init
Fatal: create key in repository at s3:http://archive1.home.intra:9000/restic.tests failed: repository master key and config already initialized

restic backup
repository 62fd6200 opened successfully, password is correct
[31:51] 36.96%  471273 files 129.761 GiB, total 914483 files 351.127 GiB, 0 errors ETA 54:20
/tank/sysbackup/patmos/Users/thomasko/Library/Application Support/iMazing/Library/Apps/Icons/org.videolan.vlc-ios-305

Files:       914483 new,     0 changed,     0 unmodified
Dirs:            4 new,     0 changed,     0 unmodified
Added:      1.795 KiB
processed 914483 files, 351.127 GiB in 1:19:39
snapshot 03fdb8ed saved
4779,20 real      3749,19 user       547,83 sys
./restic_backup_newarchiver

restic init
Fatal: create key in repository at s3:http://archive1.home.intra:9000/restic.tests failed: repository master key and config already initialized

restic backup
repository 62fd6200 opened successfully, password is correct

Files:           0 new,     0 changed, 914483 unmodified
Dirs:            0 new,     0 changed,     4 unmodified
Added:      0 B
processed 914483 files, 351.127 GiB in 8:11
snapshot 1c5b59d7 saved
491,85 real       701,27 user       231,04 sys

very interesting information, thank you already! Just to confirm, did you try to increase the files read concurrently for OS X to 4 or even 16 yet? The place in the source code is here:
https://github.com/restic/restic/blob/cf1fb50f9c3958537aea2603c929cf2c57f42125/internal/archiver/archiver.go#L102

i have alread changed that to 4 and 16 - please see the posting 5 days ago - changing that made no difference.
on freebsd both lines change very frequently - on macosx some files simply hang there for a very long time (one was 800 bytes long, the other one 2 MB). they hung half a minute there ...
the question is: what is different on macosx your new code triggers?

I've got no idea, sorry. Is anybody able to help here?

@tk7r Does this issue still exist for you? For me the backup performance on macOS is ok.

it seemed to be an issue with my virus scanner. „old“ restic was running fine, but some new restic behaviour or access pattern triggered more checks in the virus scanner than before. - i think this issue can be closed.
i am running restic on macos too and it works fine so far (at least with disabled virus scanner).

Thanks @tk7r, then we'll close this now!


fix(inline_macro): remove trivia from macro name

Fixes #798

awesome thank you @LucasLvy!
could you address the ci failure and we can get this merged


[XI-6068] Controls show a current time of 'NaN' while seeking

What does this change?
This PR fixes the controls of Kaltura players showing a current time of NaN while seeking.
The payload of the 'seeked' event is undefined in the Kaltura API.
This caused the current time of the player to be NaN for a short period of time since this.status.progress.seconds was being updated with a value of undefined. Right after that, the 'timeupdate' event was fired and the current time was restored.

before
after

Decisions / Choices I made
Moving the update of the seconds from the 'seeked' event listener to the seek method solves the issue.
For the same reason, seeking through the transcript was also broken for Kaltura videos (see here). The seconds sent to cueUpdate were undefined.
Release Notes
[Bugfix] Video player controls from Kaltura provider display a current time of 'NaN' while seeking (XI-6068)

Very nice, thanks!

Looks good already, thank you for taking care of two bugs at once! I'm thinking of a way to test this, but our setup makes this notoriously difficult. I'll get back to you.

e2e: add support for puppeteer v19

https://github.com/openHPI/xikolo-video-player/pull/60 this major update finally supports a newer version of puppeteer. I don't expect a breakthrough now, but maybe it will make things a little better?


Azure Resource Management - Front Door Sample Issue

Section link1, link2

Reason:
1.

2.

Suggestion:
1.add property location.
2.update to 429.
@joheredi, @dw511214992 for notification.

@kazrael2119 Could you help take a look at this issue ? Thanks

this issue is fixed in this pr:https://github.com/Azure/azure-sdk-for-js/pull/26380


Add unit tests

How to categorize this PR?

/area testing
/kind test
What this PR does / why we need it:
This PR adds unit tests across several packages.
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Release note:

NONE

/hold for PR triage

/lgtm

/unhold


[Enhancement] Open shulker where a target is focused

Hello,
I just would suggest this enhancement : open a shulker when a target is in the target range.

Are you using NPC shulkers with Sentinel?

Yes, and it works.

Spigot seems to lack a method for this as far as I can see.
Perhaps @fullwall can help here by providing a Citizens trait or NMS wrapper call to trick it open/close?

Added

:D!
Sentinel DEV-145+ contains autoshulker.
http://ci.citizensnpcs.co/job/Sentinel/
Important note: do not use the shulker command or trait now available in Citizens, as it will
A: Override the Sentinel input
B: Spam sound effects for some reason


prevent cursor from entering specific monitor

Description
One of my monitors is a TV which I exclusively use for running RetroArch. Having a way to prevent the mouse cursor from entering that monitor would prevent me from loosing my cursor.
If you currently don't have time to implement this, it'd be great if you could explain what you'd like the configuration to look like so I can take a look at it.

you can move the display to not touch any other and the mouse won't travel from/to it.


Buttons do not support dark theme

Buttons on the Coordinate Conversion pane do not follow Dark theme schema used by ArcGIS Pro.
Input section of the add-in:

[ ] Select Point
[ ] Edit Properties
[ ] Map Point Tool

MT 3.3.1 February 2018 release
Tested in Pro 2.2 and 2.3

This was tested on ArcGIS Pro. Good to merge

Testing from \DefenseSolutions\CoordinateConversion\March2019\Dev\43 looks correct.


Allow filter in quick navigation (Ctrl-Q)

Now: Ctrl-Q shows quick navigation and no fast jump to some item
Request:  show quick navigation and allow typing letters to jump to an item

Have you tried using using F1 > View: Open View?

Have you tried using using F1 > View: Open View?

Thanks! I assigned Ctrl-Q to this command and it dose exactly what I wanted.
I think this can be closed.


Submit MindsDB Docker Extension to Docker Extension Marketplace

MindsDB Arm platform support has recently been introduced. It's a perfect time to build Docker Extension and submit to Docker.

Completed. MindsDB is now in Extension Marketplace.


Add .eslintignore to project root for vscode support

The eslint extension in vscode were still running eslint on all dist/ folders in the workspace projects.  Centrally locating .eslintignore in the project root makes the ignores apply nicely for npm run lint and for vscode.
If you ever look at the transpiled or packaged code under dist/, this tweak will make life nicer.

@sjd78
Good catch!


fix: make ollama model selection work

Thank you for developing such an excellent plugin!
I found ollama model name selection does not work.

The current implementation appears to have two issues:

[x] Selected models are not saved into data.json.

fixed in l712-715

[x]  Selected model names are reset to the default ones when opening or refreshing the settings tab.

This happens because dropdown.getValue() returns nothing if there is no corresponding option.

I should mention that I’m not very familiar with this project or, more generally, with Obsidian plugin development. If there are any conventions or best practices that I’ve overlooked, please know it’s unintentional, and I’d appreciate it if you could point them out.

Sorry for the bug. Thanks for your contribution!


Allow changing the size of the viewing window for human rendering

Description
In debugging I often want to view what's happening with human rendering, but having the window take up a quarter of my screen every time it opened and me needing to resize it with the mouse was frustrating. This change allows (width=, height=) to be passed to gym.make() which will set the human window size properly.
With the way I have currently implemented it the "quarter of screen area" that is currently the default will never be hit because of the defaults on width/height that presently exist. Perhaps we could add a human_width/human_height optionals which will still allow the current default behaviour and then allow customizing with this pull requests changes.
Type of change
Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

Checklist:

[x] I have run the pre-commit checks with pre-commit run --all-files (see CONTRIBUTING.md instructions to set it up)
[x] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes

Thanks for the PR, looks good to me.


Fix javadoc for isAnyNotEmpty

Coverage increased (+0.03%) to 94.301% when pulling 014c616aeb119a1bf0184b688f64338f6a24c924 on martin-tarjanyi:patch-1 into 2cd3f0f10a797b744509903b8eca4fd2fd4ed6f6 on apache:master.

Thanks!


Get the HDMI2Ethernet target working on the Opsis board

Currently it is only available for the Atlys board.
See https://github.com/timvideos/HDMI2USB-misoc-firmware/blob/master/targets/atlys_hdmi2ethernet.py

@aomtoku - Please take a look at this github issue.

This is duplicate of https://github.com/timvideos/HDMI2USB-litex-firmware/issues/223


Add commutator control based on swing-twist decomposition

Summary
A common pattern of commutation is to use rotation measurements from a BNO055 or similar device to compute a feedback control signal for stabilizing twist in the tether.
This PR introduces a high-level operator AutoCommutator which provides a reusable solution for this pattern. Additionally, we introduce two operators to allow low-level interface with the commutator, and to compute the feedback signal in isolation. This allows composing custom commutation solutions where the control signal may be combined with manual operation, or to enable fine-grained rate control and logging, as demonstrated in the final example.
Operators

TurnMotor: the low-level interface to the motor, operated in units of full turns, in the range [-1, 1]. The input is a sequence of turn values. The operator internally formats and sends to a serial port a sequence of JSON strings used to drive the motor controller.
QuaternionTwistController: calculates the feedback control signal to stabilize twisting in the tether about the specified axis from a series of Quaternion rotation measurements. The twist about the specified axis is used to compute the differential feedback signal which is then normalized into units of turns.
AutoCommutator: an embedded workflow which samples the rotation measurements at 10 Hz, computes the differential feedback signal and drives the motor at the specified serial port.

Examples
Driving the commutator with a BNO055 from the Onix1 library

Logging BNO055 measurement timestamps with each command

Fixes #1

Its funny -- I think that if we are going to make these elements composable, then we are in an unhappy medium here. The original meganode on the issue-1 branch was an attempt at dead simplicity for the user. The AutoCommutator workflow in this PR provides  a similar level of useability, so that's covered. However, the TurnMotor sink feels kind of unnecessary to me. Right now it does three things

Provide access to write to a serial port. WriteSerialLine does this pretty effectively already
Formats a string. Expressions.Format seems like its fine for this already
Toss ridiculous turn commands (NaN or inifnity). I guess a custom expression script would be needed to do this filtering in Bonsai, so thats really the only major thing I see.

Aside from the limited gains from this node, what if we want more control over the Commuator state beyond turning the motoro? e.g. what if we wanted to have some Property that provides control over the commutator's LED state using {led: false}. We could

Make a property in TurnMotor that modifies the string sent to the motor to include the the additional LED state (non-reactive since it changes when the turn command issued)
Make a subject in TurnMotor that merges its output sequence with turn commands
We could add a parallel stream in the AutoCommutator workflow that sent {led: false} to a SerialWriteLine with the same port as the TurnMotor. This feels hacky because PortName needs to be defined for both the TurnMotor node and the SerialWriteLine node.

These all feel kinda hacky in comparison to definging AutoCommuator as the following:

So my suggestion is to remove TurnMotor and replace specific functionality with embedded bonsai workflows instead.

@jonnew Thanks for clarifying the additional functionality of the commutator device. Given this I agree we should refactor the TurnMotor operator to be either a CommutatorDevice or simply Commutator and expose a more general interface to the entire device functionality.
Detailed reasons below:

Formats a string. Expressions.Format seems like its fine for this already

While it is possible to format the JSON string with the Format operator, this locks us to a specific communication protocol. If we ever need to upgrade or extend the details of the JSON API in any way (or swap the protocol to binary, etc), we would not be able to easily upgrade old workflows. With the operator abstracting away the communication internals, we can get away with just upgrading the package, even for hand-crafted workflows.
I don't currently see any particular advantage of exposing the internal JSON strings, especially given the protocol does not comply to an existing standard and there is little cost or complexity added in doing this abstraction.

Toss ridiculous turn commands (NaN or inifnity). I guess a custom expression script would be needed to do this filtering in Bonsai, so that's really the only major thing I see.

Input validation is definitely an important consideration. Similar to the above, having the packaged node would make it much easier to upgrade legacy workflows to benefit from improved validation. For example, in the future we could extend this operator to drop or throw if inputs are sent too fast, etc.

Aside from the limited gains from this node, what if we want more control over the Commuator state beyond turning the motor?

This is where having a general CommutatorDevice node would really make this interface shine and again allow us to extend the device or API with extra functionality by simply updating the package.
The easiest way to do this would have been for the commutator to use one of the standard control protocols already integrated into bonsai such as Harp or even ONI itself. This would have allowed us to leverage a bunch of existing infrastructure for configuration, control, logging and synchronization with other devices. In fact, if it was Harp we could even have generated the entire Bonsai interface automatically.
All the questions you asked about how to dynamically parameterize device configuration and having multiple control streams have already been solved for all Harp devices for example. I don't know what microcontroller the commutator uses, but there is now an existing Harp core for the RP2040 (https://github.com/AllenNeuralDynamics/harp.core.rp2040) which would have been perfect for this. A simple version could also be designed on top of other microcontroller architectures, including software forms of clock synchronization which have been implemented for other devices.
Assuming this is not possible for the current hardware / firmware combination:

Make a property in TurnMotor that modifies the string sent to the motor to include the the additional LED state (non-reactive since it changes when the turn command issued)

This is definitely a possibility and I think we could make this pattern work by using a BehaviorSubject in the backend. Whether this is appropriate depends on how big is the full API, and on whether a simple LED toggle is really the only extra functionality. If there are other configuration or control commands which would be useful to expose as streams (now or in the future) this might quickly become too complicated to evolve.

Make a subject in TurnMotor that merges its output sequence with turn commands

I'm not sure I understood this suggestion correctly, but I'm interpreting this to be the same pattern we use for the Harp Device node, which is basically a node which takes a sequence of commands and formats / pipes them to the serial port.
I think this would definitely be flexible enough for all present and future needs. Basically in this pattern we need just two concepts:

A set of commutator nodes (TurnMotor, EnableLed, etc), which format inputs into the corresponding serializable JSON strings (or other protocol) to send to the serial port.
A CommutatorDevice node which establishes a connection to the commutator and receives formatted strings.
(optional) Make the "command" nodes output an abstract base class instead of strings, so that the CommutatorDevice node can declare a "strongly-typed" input. This would immediately disallow accidentally sending random or malformed strings to the device.

I think the last two bullet points would on their own also justify having a CommutatorDevice node.
Even though SerialWriteLine may suffice to send arbitrary strings to the serial port, this operator does not make any assumptions on port configuration properties such as baud rate, etc. Even if the commutator uses the default settings of the bonsai node for now, there is no guarantee this won't change in the future, either on the side of the commutator, or the side of bonsai, so relying on this alignment to be stable in the future is dangerous IMO. Having a common device node would ensure the alignment is enforced and the port configured correctly.
The last point is optional but easy to implement and important for validation to ensure the commutator does not go into an undefined state by accidentally receiving gibberish data from upstream.

We could add a parallel stream in the AutoCommutator workflow that sent {led: false} to a SerialWriteLine with the same port as the TurnMotor. This feels hacky because PortName needs to be defined for both the TurnMotor node and the SerialWriteLine node.

This could actually be done very easily by linking both properties via the same ExternalizedMapping. However, I do agree that it is still hacky because of all the issues discussed above.

These all feel kinda hacky in comparison to defining AutoCommuator as the following:

Seeing the manual control of the motor with keys and LED control functionality added to the AutoCommutator workflow actually makes me feel that what we should really remove is the AutoCommutator node itself.
It feels there are currently two things being conflated in this package:

The commutator device is actually a complex device, so we need all of its functionality to be exposed in a structured way. This is so we can freely compose what we want to do with it in arbitrary ways, e.g. reset commutator on trials, blink LED to signal something, apply a slower drift correction algorithm, untwist commutator with computer vision tracking, etc the list will just keep growing.

There is a need for a "plug-and-play" commutator solution that users can drop into their acquisition workflows to get started quickly doing some recordings.

I feel that 2. could just as easily be provided simply as an example in the commutator docs, with no need for an embedded workflow. This recommended example usage workflow could be updated regularly as we improve the pattern without requiring updates to the commutator package.
That said, I don't disagree we could also make it into an embedded workflow in the package, although in that case I feel it should be fairly minimalistic, e.g. without assuming specific keys or options, since the more we assume the higher is the risk that it will not work for someone's specific workflow.


Path parameters with "-" character (i.e. mount-name) to be changed

For all the callbacks with parameters in the path containing the hyphen (for example mount-name or local-id), when the api is called both via swagger or postman a 404 not found response is received.
Debugging the code the controllers are not called at all as they seems to be not recognized at all by the framework.
Fixed manually removing the hyphen and changing the parameter name in lower camel case.

Solution:
we will change the path-parameters in all here and in all future specifications to be in lower camel case.

Moved to milestone 1.1.0_spec as there is a lot of new code prepared that suffers from the same problem.

Discussed today in AppOwner meeting that the hyphen normally should not cause any problems. Therefore asked TechM for some feedback on it. Prathiba answered:

In the server side implementation, we are using a package called OAS3tool that validates the API’s syntax.
By default , this OAS3tool package don’t have the capability to consider the path parameter with a hyphen.
Either,
•	the {mount-name}, {local-id} in the OAS specification shall be modified to {mountName}, {localId}
(or)
•	we need to try extending the OAStool to see the possibility to support hypen in the path parameter
Note :  This problem will happen only in the serving application(where the paths are defined) and not in the consuming applications(where the paths are used in the callbacks).

Next step: discuss whether we first try to extend the OAStool

As a parameter, mount-name and local-id have been replaced by mountName and localID.


Post back event issue

i ready don't understand why nothing any postback event in log
have app need permission? or any config missing with me?
Pls help me
page config

but log don't have event postback

Nothing is wrong with the gem. Maybe ur webhook url is the issue. Create another fb page and try a fresh clean app

Thanks you so much
We have any issue to discuss about submit request permission for app?

There's no issue for that, but feel free to make one!

thanks so much


docs: clearStaleState method doesn't actually clear local storage

I'm trying to clear localstorage manually:
const mng = new UserManager(config);
mng.clearStaleState().then(() => {
    console.log('clearState success');
}).catch((e) => {
    console.log('clearStateState error', e.message);
});

After that i see 'clearState success' at console, but localstorage is still fill with oidc.${hash} items.
Could you provide any suggestions for fixing it? Thanks for your library.

There is a created property which tells when the local storage got created, https://github.com/IdentityModel/oidc-client-js/blob/dev/src/State.js#L63
you can see why it is not removing that, it only removes the storage keys that are fallen in the criteria of when it actually got created,

clearStaleState only cleans entries older than the configured age. perhaps this is why you are still seeing entires?

Any update?

I think there is confusion here because the documentation for the userStore optional setting says that sessionStorage is the default, but the original question was about clearing localStorage.  So, if you used the default storage for userStore but want to clear entries in localStorage using clearStaleState you need to pass an instance of WebStorageStateStore as the first argument to clearStaleState.
Something like await clearStaleState(new WebStorageStateStore({ storage: window.localStorage })).
Do the signinSilent methods make entries into localStorage by default?

Do the signinSilent methods make entries into localStorage by default?

For those, yes -- that's where the nonce/state is stored. localStorage is used (instead of sessionStorage) since in IE/Edge they flush sessionStorage when the browser changes hosts/domains. So in short, we need to use localStorage for the nonce/state piece to survive the round trip.
The user's session is in sessionStorage (by default).

Thanks for the timely response.
Perhaps the documentation or samples could be updated to indicate that siginSilent methods make use of localStorage by default so making use of clearStaleState requires an argument of WebStorageStateStore pointed at localStorage to remove those entries (using default settings)?
It wasn't particularly clear and I only stumbled upon the right solution after seeing this thread and reading source code. Those entries can add up pretty quickly depending on how the expiration of your token is configured by the provider.

Call clearStaleState in your app, as it's not called automatically. The stale lifetime is a setting set via the options staleStateAge and defaults to 15m. This clears local storage entries from incomplete OIDC requests. This state is unrelated to a user session, which is kept in session storage by default.

Perfect, Thanks :)

Thanks for the timely response.
Perhaps the documentation or samples could be updated to indicate that siginSilent methods make use of localStorage by default so making use of clearStaleState requires an argument of WebStorageStateStore pointed at localStorage to remove those entries (using default settings)?
It wasn't particularly clear and I only stumbled upon the right solution after seeing this thread and reading source code. Those entries can add up pretty quickly depending on how the expiration of your token is configured by the provider.

Hey, I want to know how the configuration of expiration of token is related in this case? The entries in my local storage got filled up preety quickly once the expiration time was set to 7 days for some testing purpose.


use backpack_auth() function from 0.9.x updates

this is a break changing commit ( if user uses old Base version this will cause problems  )
i would like to use \Auth::guard('admin')->check(); but they may uses another guard name for an admin panel.

@tabacitu I feel like this is a good idea with the move to having multi-auth support and such we need to convert to using backpack_auth() instead of just Auth::check() thoughts?

Totally agree. I think it's about time we support only Backpack\Base 0.9.x and its features, so I've also added the composer.json requirement and launched this as 1.2.0 - thank you @iMokhles !


How to load 3D images into the AutoFormer architecture

I am trying to modify the AutoFormer code to allow 3D images then combine a custom decoder. However, I am having trouble on modifying for 3D, any advise on how I can do so? Thanks

One simple way is to split the 3D images into multiple 2D images and feed them to AutoFormer. Another way is to split the 3D images into many small patches and embed them. Remember to modify the position embeddings to make the architecture reasonable.

Hi @Harsh-Gill. Hope you have solved the problem. I will close the issue now. If you have further questions, feel free to reopen it.


[Request] Allow verifying / update loading bar to persist in advanced options

Summary
Being able to see the progress of the verification / updating process from anywhere, including the advanced options menu, would be convenient as it'd appear everywhere in the application. It may also be helpful if #24 gets added and volume sliders get put into launcher settings, thus being able to quickly change audio levels before music / sounds play or stop.
Current behaviour:

Proposed behaviour:

This needs a persistence file, will be added soon

wait misread this issue

This is surprisingly hard due to how the bar's implemented -
it's a child of a VBoxContainer which has the blogpanel, etc but the advanced panel isn't in this - it literally flies in over the top of it.
There's a few ways to go about sorting this, one being have a copy of the advanced panel in the VBoxContainer and then modifying the flying-in box to the current size of that, so when it flies in it looks correct.
This will be ..attempted at some point


bevy_reflect: Recursive registration

Objective
Resolves #4154
Currently, registration must all be done manually:
#[derive(Reflect)]
struct Foo(Bar);

#[derive(Reflect)]
struct Bar(Baz);

#[derive(Reflect)]
struct Baz(usize);

fn main() {
  // ...
  app
    .register_type::<Foo>()
    .register_type::<Bar>()
    .register_type::<Baz>()
    // .register_type::<usize>() <- This one is handled by Bevy, thankfully
  // ...
}

This can grow really quickly and become very annoying to add, remove, and update as types change. It would be great if we could help reduce the number of types that a user must manually implement themselves.
Solution
As suggested in #4154, this PR adds automatic recursive registration. Essentially, when a type is registered, it may now also choose to register additional types along with it using the new GetTypeRegistration::register_type_dependencies trait method.
The Reflect derive macro now automatically does this for all fields in structs, tuple structs, struct variants, and tuple variants. This is also done for tuples, arrays, Vec<T>, HashMap<K, V>, and Option<T>.
This allows us to simplify the code above like:
#[derive(Reflect)]
struct Foo(Bar);

#[derive(Reflect)]
struct Bar(Baz);

#[derive(Reflect)]
struct Baz(usize);

fn main() {
  // ...
  app.register_type::<Foo>()
  // ...
}

This automatic registration only occurs if the type has not yet been registered. If it has been registered, we simply skip it and move to the next one. This reduces the cost of registration and prevents overwriting customized registrations.
Considerations
While this does improve ergonomics on one front, it's important to look at some of the arguments against adopting a PR like this.
Generic Bounds
Since we need to be able to register the fields individually, we need those fields to implement GetTypeRegistration. This forces users to then add this trait as a bound on their generic arguments. This annoyance could be relieved with something like #5772.
Registration Assumptions
By automatically registering fields, users might inadvertently be relying on certain types to be automatically registered. If Foo auto-registers Bar, but Foo is later removed from the code, then anywhere that previously used or relied on Bar's registration would now fail.

Changelog

Added recursive type registration to structs, tuple structs, struct variants, tuple variants, tuples, arrays, Vec<T>, HashMap<K, V>, and Option<T>

Migration Guide
All generics on types that derive Reflect will need to add GetTypeRegistration as a bound on each type argument:
// OLD
#[derive(Reflect)]
struct Foo<T: FromReflect> {
  data: Vec<T>
}

// NEW
#[derive(Reflect)]
struct Foo<T: FromReflect + GetTypeRegistration> {
  data: Vec<T>
}

We might actually be able to improve the usability here by adding GetTypeRegistration to the where clause of the impl like we do with Reflect 🤔

I can't speak to the actual implementation but as a user I have found it quite painful without this recursive registration, so this is a very welcome change, thanks a lot! Is there any chance that it makes it into 0.11?
I understand the concern that when a outer type is removed, the inner types won't be automatically registered any more. But in this case there's at least a warning thrown by Bevy, so that does give some guidance and should mitigate the issue, imo.

I'm actually realizing this might not be in an ideal state yet thanks to https://github.com/bevyengine/bevy/issues/8965. This will prevent users from including dynamic types in the fields of their reflected items (since they do not implement GetTypeRegistration). For example, this will fail to compile:
#[derive(Reflect)]
struct Foo {
  data: DynamicStruct
}

On the other hand, this is an issue anyways since this problem slipped through the cracks in #6056.
If this isn't that big a deal, we can move forward with it. Otherwise, I think we really need to rethink how we add type bounds.

@MrGVSV, could you resolve conflicts? This is the only remaining reflection PR in the milestone and you and @soqb have said you'd like to land this in 0.12.

I updated the Migration Guide:

Deriving Reflect now requires that all active fields also implement GetTypeRegistration (which is automatically implemented when deriving Reflect). This means that types which do not implement GetTypeRegistration now need to be ignored:
// OLD
#[derive(Reflect)]
#[reflect(from_reflect = false)]
struct Foo {
  data: DynamicStruct
}

// NEW
#[derive(Reflect)]
struct Foo {
  #[reflect(ignore)]
  data: DynamicStruct
}

This is already an issue with FromReflect (hence why we need to opt-out of it). I think the fix would be something like #9046.

Rebased and added a new commit (0ada41186f2fac6b7773948d98056c39a82b2e85) based on a Discord suggestion from @soqb that re-allows dynamic types in active fields.
Before this commit, this PR would have made the following impossible:
#[derive(Reflect)]
#[reflect(from_reflect = false)]
struct Foo {
  data: DynamicStruct
}

And would have required users to #reflect(ignore)] the dynamic field (see https://github.com/bevyengine/bevy/pull/5781#issuecomment-1781964531).
However, this new commit adds a hidden wrapper trait around GetTypeRegistration. This allows us to avoid implementing GetTypeRegistration for dynamic types while still enabling all type dependency registrations to go through a single function.

Having seen so many PRs come in asking to register one type or another, it's pretty clear that we cannot afford to not include some form of recursive type registration of some kind.

Agreed!

I really am not happy with the manual implementation on RenderAssetUsages, but the rest looks straightforward to me. LGTM.

Yeah I'm not sure why RenderAssetUsages is implemented manually, but my guess is that it could be replaced by a macro (#12025). But I chose not to mess with it in this PR.

@MrGVSV say the word and I'll merge this in :)

@MrGVSV say the word and I'll merge this in :) @SkiFire13 or @soqb your review on the last batch of changes would be nice too.

I mean I’m happy with it haha. We could maybe schedule it to be merged as part of the merge train tomorrow in case anyone has any last minute concerns?

Great: removing the controversial label for now so I don't miss it 😅

One final thing that came to mind was maybe to annotate register_type_dependencies as #[inline(never)] to avoid excessive code generation at compile time.

One final thing that came to mind was maybe to annotate register_type_dependencies as #[inline(never)] to avoid excessive code generation at compile time.

Annotate just the default trait method or add it to the generated GetTypeRegistration impls?

@james7132 Annotate just the default trait method or add it to the generated GetTypeRegistration impls?

The empty default is fine, but the ones that have an actual body need to be annotated to avoid inlining the recursion.

LGTM, although the #[inline(never)] feels like premature optimizations. Is it certain that LLVM will inline that method normally (and that this will actually be bad, since it seems it's only used in one function)?


login account

Could you provide an account for the official demo?
https://next-learn-dashboard.vercel.sh/login

try with
id<EMAIL_ADDRESS>pw: 123456

Thank you 🙏🏼


BitmapBatchIterator.hasNext() - incorrect result

Describe the bug
BitmapBatchIterator.hasNext() is implemented as:
  public boolean hasNext() {
    return wordIndex < 1024;
  }

This will produce incorrect results (return true for hasNext) when their is actually nothing at that index (bitmap.bitmap[wordIndex] == 0 for all remaining indexes).
From the looks of it, it might have been done this way for some well-intentioned performance reasons.
Can be argued though that performance impact of those extra checks amortized over the entire iteration will be same as before.
And the work that the user does when hasNext() returns true might be more expensive and even cause unintentional bugs (expected non-empty).
Proposed fix:
  @Override
  public boolean hasNext() {
    if (wordIndex == 1024) {
      return false;
    }
    while (word == 0) {
      if (wordIndex == 1024) { // reached end without a non-empty word
        return false;
      }
      word = bitmap.bitmap[++wordIndex];
    }
    return word != 0; // found some non-empty word, so hasNext
  }

Happy to contribute the fix !
To Reproduce
Obvious to recognise the bug from code, let me know though if reproduce code is necessary
RoaringBitmap version: 1.0.0
Java version: 21

Thanks. Let us document the behavior.
https://github.com/RoaringBitmap/RoaringBitmap/pull/731

Why not fix behaviour though? Since the behaviour is accurate in all other container iterators and fixing here is trivial.
And the perf overhead seems tolerable (consider this is a batched iterator)

Added PR https://github.com/RoaringBitmap/RoaringBitmap/pull/732
@lemire


Add Initial OpMode for moving the arm

Create the op mode
Init the correct motors
Init the Limit Switch
Init gamepad (if needed)
Configure game pad 2
** Joystick 1 up - arm up; Joysitck down - arm down
** Joystick 2 up - arm out; down - arm in
Make arm stop on arm in when hits the limit switch
Make the arm stop on out when the motor encoder reaches some spot

Do we need stops for lifting the arm?

No limits needed for the arm actuation, however the end limit may need a hold power.


SSL Node server help

Could you please tell us how to enable https

Well, you should set at backend side in socket.io the key and certificates, and set te client with secure: true Maybe I will work on this in a new branch, but may we need first to update socket.io to latest stable.
Leave this issue open in order to take care of this.

The backend is fine because i could use chrome socket io tester to send event and it works fine
but when i tried to use elephant i could not emit any event


Typescript to minor match

Contributes to https://github.com/aspnet/JavaScriptServices/issues/705
Doesn't solve the compat problem; but does solve the issue with the templates at least for now.

if (Model.IsMSEmployee && !Model.IsMSVendor) {
@benaadams
thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
} else if (Model.HasCLA) {
@benaadams
Thanks for having already signing the Contribution License Agreement.
Your agreement was validated by .NET Foundation.    We will now review your pull request.
} else if (!Model.IsSignificant) {
This seems like a small (but important) contribution, so no Contribution License Agreement is required at this point. We will now review your pull request.
}
else {
if (Model.IsMSEmployee && Model.IsMSVendor) {
@benaadams,
    Thanks for your contribution.
    To ensure that the project team has proper rights to use your work, please complete the Contribution License Agreement at https://cla2.dotnetfoundation.org
    The agreement will cover your contributions to all Microsoft-managed open source projects.

} else {
    @__benaadams__,

    Thanks for your contribution.

    To ensure that the project team has proper rights to use your work, please complete the Contribution License Agreement at https://cla2.dotnetfoundation.org
    It will cover your contributions to all Microsoft-managed open source projects.
}

}
Thanks, .NET Foundation Pull Request Bot

May have done too many files

Thanks for this. Rather than putting in this as a temporary fix, I've applied a more long term fix as described in #705. Hope that's OK!


pre-create hook not being executed

Question
I am using tusd with the tus-js-client. I have two hooks files, one for pre-create and one post-finish. Both files have the same permissions. While post-finish is working the pre-create hook is not fired.
Logfile output:
Sep 22 11:59:57 tusd[45098]: [tusd] 2020/09/22 11:59:57 event="RequestIncoming" method="OPTIONS" path="" requestId=""

Sep 22 11:59:57 tusd[45098]: [tusd] 2020/09/22 11:59:57 event="ResponseOutgoing" status="200" method="OPTIONS" path="" requestId=""

Sep 22 11:59:57 tusd[45098]: [tusd] 2020/09/22 11:59:57 event="RequestIncoming" method="POST" path="" requestId=""

Sep 22 11:59:57 tusd[45098]: [tusd] 2020/09/22 11:59:57 event="HookInvocationStart" type="pre-create" id=""

Sep 22 11:59:58 tusd[45098]: [tusd] 2020/09/22 11:59:58 event="HookInvocationFinish" type="pre-create" id=""

Sep 22 11:59:58 tusd[45098]: [tusd] 2020/09/22 11:59:58 event="UploadCreated" id="71745f257afa2ed21fee222f76b381c9" size="24548475" url="https://server/files/71745f257afa2ed21fee222f76b381c9"

Sep 22 11:59:58 tusd[45098]: [tusd] 2020/09/22 11:59:58 event="ResponseOutgoing" status="201" method="POST" path="" requestId=""

Sep 22 11:59:58 tusd[45098]: [tusd] 2020/09/22 11:59:58 event="HookInvocationStart" type="post-create" id="71745f257afa2ed21fee222f76b381c9"

Sep 22 11:59:58 tusd[45098]: [tusd] 2020/09/22 11:59:58 event="HookInvocationFinish" type="post-create" id="71745f257afa2ed21fee222f76b381c9"

Sep 22 11:59:58 tusd[45098]: [tusd] 2020/09/22 11:59:58 event="RequestIncoming" method="OPTIONS" path="71745f257afa2ed21fee222f76b381c9" requestId=""

Sep 22 11:59:58 tusd[45098]: [tusd] 2020/09/22 11:59:58 event="ResponseOutgoing" status="200" method="OPTIONS" path="71745f257afa2ed21fee222f76b381c9" requestId=""

Sep 22 11:59:58 tusd[45098]: [tusd] 2020/09/22 11:59:58 event="RequestIncoming" method="PATCH" path="71745f257afa2ed21fee222f76b381c9" requestId=""

Sep 22 11:59:58 tusd[45098]: [tusd] 2020/09/22 11:59:58 event="ChunkWriteStart" id="71745f257afa2ed21fee222f76b381c9" maxSize="24548475" offset="0"

Sep 22 11:59:59 tusd[45098]: [tusd] 2020/09/22 11:59:59 event="HookInvocationStart" type="post-receive" id="71745f257afa2ed21fee222f76b381c9"

Sep 22 11:59:59 tusd[45098]: [tusd] 2020/09/22 11:59:59 event="HookInvocationFinish" type="post-receive" id="71745f257afa2ed21fee222f76b381c9"

Sep 22 12:00:00 tusd[45098]: [tusd] 2020/09/22 12:00:00 event="HookInvocationStart" type="post-receive" id="71745f257afa2ed21fee222f76b381c9"

Sep 22 12:00:00 tusd[45098]: [tusd] 2020/09/22 12:00:00 event="HookInvocationFinish" type="post-receive" id="71745f257afa2ed21fee222f76b381c9"

Sep 22 12:00:01 tusd[45098]: [tusd] 2020/09/22 12:00:01 event="HookInvocationStart" type="post-receive" id="71745f257afa2ed21fee222f76b381c9"

Sep 22 12:00:01 tusd[45098]: [tusd] 2020/09/22 12:00:01 event="HookInvocationFinish" type="post-receive" id="71745f257afa2ed21fee222f76b381c9"

Sep 22 12:00:02 tusd[45098]: [tusd] 2020/09/22 12:00:02 event="HookInvocationStart" type="post-receive" id="71745f257afa2ed21fee222f76b381c9"

Sep 22 12:00:02 tusd[45098]: [tusd] 2020/09/22 12:00:02 event="HookInvocationFinish" type="post-receive" id="71745f257afa2ed21fee222f76b381c9"

Sep 22 12:00:03 tusd[45098]: [tusd] 2020/09/22 12:00:03 event="HookInvocationStart" type="post-receive" id="71745f257afa2ed21fee222f76b381c9"

Sep 22 12:00:03 tusd[45098]: [tusd] 2020/09/22 12:00:03 event="HookInvocationFinish" type="post-receive" id="71745f257afa2ed21fee222f76b381c9"

Sep 22 12:00:04 tusd[45098]: [tusd] 2020/09/22 12:00:04 event="HookInvocationStart" type="post-receive" id="71745f257afa2ed21fee222f76b381c9"

Sep 22 12:00:04 tusd[45098]: [tusd] 2020/09/22 12:00:04 event="HookInvocationFinish" type="post-receive" id="71745f257afa2ed21fee222f76b381c9"

Sep 22 12:00:04 tusd[45098]: Post-Finish Hook executing...

pre-create Python script:
#!/usr/bin/python3
import json
import sys
import os

print ("Pre-Create Hook executing...")
stdin = sys.stdin.readlines()
data = json.loads('\n'.join(stdin))
print (data)

I found a similar report here: https://stackoverflow.com/questions/60795064/tusd-pre-create-hook-not-being-called
Setup details
Please provide following details, if applicable to your situation:

Operating System: Linux (Ubuntu 20.04)
Used tusd data storage: disk store
Used tusd configuration: -upload-dir=/data/incoming -host=<IP_ADDRESS> -behind-proxy -expose-metrics=false -hooks-dir=/data/tushooks
Used tus client library: tus-js-client

The pre-create hook is invoked but its output is not forwarded to tusd's stdout. You can see its execution from the logs:

Sep 22 11:59:57 tusd[45098]: [tusd] 2020/09/22 11:59:57 event="HookInvocationStart" type="pre-create" id=""
Sep 22 11:59:58 tusd[45098]: [tusd] 2020/09/22 11:59:58 event="HookInvocationFinish" type="pre-create" id=""

Since pre-create is a blocking hook, only its stderr will be piped to tusd's output. The output from stdout is not forwarded. The documentation does right now not address this fact properly (https://github.com/tus/tusd/blob/master/docs/hooks.md#blocking-hooks), which should be improved.

Thanks for your quick and very helpful reply. Sending the output to stderr did the trick.


API for removing all components from an entity

Hi,
lets say I have an entity I want to reload without modifying other entities which refer to this one by id. For this I would first like to strip all components from the entity but that already poses a problem:
Knowing what registry.clear() vs registry.clear<Component>() does I thought registry.remove(entity) might do it – turns out it doesn't, it is a no-op instead. registry.visit(entity, [](auto const component){…}) might have worked if I could have figured out a way of doing the equivalent of registry.remove<component>(entity) with a component only known at runtime. All this runtime machinery seems rather powerful, but sadly I couldn't find too much documentation on how to make that actually work in practice. What seems to work is registry.destroy(entity); assert(entity == registry.create(entity));, but is that the way it should be done? I feel kinda uneasy about it for some reason. I had no qualms if that API were registry.destroy(entity); assert(entity == registry.recycle(entity)) instead so I know its a baseless fear, but still. I can also not remove the feeling of a previous and by now probably ancient version having a registry.clear(entity) for this propose, but I haven't dug into history to checked.
So this is as much a question of how this should be done nowadays as it is a request to fix and/or at least document the pitfalls a user might encounter along the way. Thanks!

All this runtime machinery seems rather powerful, but sadly I couldn't find too much documentation on how to make that actually work in practice.

Because it entered EnTT with the last release and only in the form of the visit so far.
More will come but it's a completely new part that must be designed carefully to avoid problems.

Knowing what registry.clear() vs registry.clear() does I thought registry.remove(entity) might do it – turns out it doesn't, it is a no-op instead.

Actually it does not but it may be an interesting feature. Why not?

What seems to work is registry.destroy(entity); assert(entity == registry.create(entity));, but is that the way it should be done?

When you destroy an entity, all its components are removed and its version is increased. Then it enters the list of available entities. It's a LIFO implicit list actually, that's why the same entity is returned if you do a create immediately after.

I can also not remove the feeling of a previous and by now probably ancient version having a registry.clear(entity) for this propose, but I haven't dug into history to checked.

Oh... it did exist? Then is has been removed accidentally. Another good reason to put it back in place.
I didn't even remember it. Usually either I destroy entities or remove specific components.
That's probably why I didn't notice the error. I apologize. Is this the case then?

for the record, I wasn't criticizing the design or anything, I was "just" seeing three wildly different potential directions I could explore while trying to (re)learn EnTT. I am fine with either as the "canonical path" and just though if there is one it should be implemented and/or documented.
If remove() vs remove<>() is the way to go I am happy to supply a patch, if it's visit() I am fine with waiting a little longer and helping documenting the eventual solution and if its the destroy, create dance I guess I will try coming up with documentation and a testcase. Or there is a fourth (fifth, sixth, …) way I do not know – or at the other side of the spectrum: A reason that should not be done at all.

I wasn't criticizing the design or anything

I'm fine with criticisms though, especially if they help to improve the library. Don't worry. :wink:

If remove() vs remove<>() is the way to go I am happy to supply a patch, if it's visit() I am fine with waiting a little longer and helping documenting the eventual solution and if its the destroy, create dance I guess I will try coming up with documentation and a testcase. Or there is a fourth (fifth, sixth, …) way I do not know – or at the other side of the spectrum: A reason that should not be done at all.

I never had this requirement and this is probably the reason for which I missed this point.
Since remove receives a parameter pack, I think the sizeof...(Component) == 0 case is a good candidate for doing what you pointed out. Isn't it?
visit is another beast and perhaps I'll extend it to create a parallel API that works with runtime component identifiers but there is no reason not to have the same functionality on both sides.

I'm adding a function named remove_all to address this. I'm pushing it on experimental.

Is this still experimental? It's quite a useful function to 'reset' an Entity without losing its ID.
In fact, I would love to have remove_all<entt::exclude<Comp>>(...) as well so that I can remove all but the component(s) I want to keep.

@samaursa with the last version, remove_all translates to something similar:
for(auto [id, storage]: registry.storage()) {
    storage.remove(entt);
}

As for excluding one or more types from the list, you can check them as:
for(auto [id, storage]: registry.storage()) {
    if(storage.type() != entt::type_id<Comp>()) {
        storage.remove(entt);
    }
}

Does it make sense?

@skypjack thanks! it would be great to see that last function in the registry's API

Mmm 🤔 maybe an utility file? I don't like to add functions to a class API when they don't really need to access the internals of the class itself.

@skypjack yes, that would work (is there a utility file already with other utilities or would this be the first?)

@skypjack it's not a big deal for personal projects, however optically it matters when trying to convince a larger team. I find that having functionality like this increases confidence in the library as it optically looks like a more 'complete' package.
Just to be clear, entt is already quite complete. Since you asked why I cared about it I thought I would elaborate :)

No, I see the reasons and I agree with you 100%. I meant why you care about if the file exists already or not. 😅
As for the implementation of this feature, can I ask you to open a dedicate issue? It helps to keep track of it and to not forget about it too. 🙂
Thanks in advance.

Ah yes, the file I don't care about.
Sure, I will open a dedicated issue.


County is 'colored in', but graphs populate no data

On click, there's no data in the graphs, just the hospitalization line.
In these counties:
Coffee | Alabama
Waupaca | Wisconsin
Gallia | Ohio
Garfield | Washington
Hampton | South Carolina
Conway | Arkansas
Henry | Missouri
Mercer | North Dakota
Pasquotank, Chowan, Swain | North Carolina
Gage County | Nebraska

Example broken page: https://covidactnow.org/us/al/county/coffee_county
FIPS 01031
It looks like the modeled data is just all 0's: https://covidactnow.org/data/county/AL.01031.0.json
@ghop02 Maybe you could take a look?  Perhaps we need to have the backend omit counties for which we have no cases or something?

So it looks like all of those have cases - on my local branch with (almost) updated data most of them did.
I think (though haven't fully confirmed) this was happening because of the 4 day interval. I think there were cases for less than 4 days and then it was starting the model from a day before there was any data.  It would make sense, but haven't 100% confirmed the logic, just my theory for now

Beaver County, Utah has the same behaviour, FYI

Fix for this is now deployed.  Confirmed Beaver, UT and Elko County, NV both work now.

[image: 🚀]
James Tamplin
tinyletter.com/jamest (2-3 emails / yr on what I'm up to)
On Mon, Apr 06, 2020 at 4:55 PM, Michael Lehenbauer <
<EMAIL_ADDRESS>wrote:

Fix for this is now deployed. Confirmed Beaver, UT and Elko County, NV
both work now.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/covid-projections/covid-projections/issues/413#issuecomment-610096316,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAHIUC7URKOQX4C6E5ZONLLRLJTWNANCNFSM4L7IGDCQ
.


Support resource deployment configuration options

I've only experimented with a few of the Aspire components so far and I've wanted to make changes to how several of them deployed in Azure. Would you be open to some PRs that add options to several of these components that would be configured in the AppHost and passed to the Bicep templates to handle some of their configuration properties?
The idea would be not unlike what I proposed for the Front Door/DNS PR.
For example, off the top of my head:

Allow serverless CosmosDB instead of strictly provisioned
Apply max RUs to a provisioned CosmosDB database
Specify backup policy for CosmosDB instance
Toggle whether Key Vaults enable soft-delete

Thanks for the consideration!

cc @mitchdenny

@WhitWaldo we recently just merged in some changes which introduce a callback on resources like AddCosmosDB(...) which allows you to mutate the resource. It builds on a new library coming from the Azure SDK team (Azure.Provisioning).
We actually use it internally to configure the Azure resources, but you can setup a callback like this:
builder.AddAzureCosmosDB("mycosmos", (resource, construct, account, databases) =>
{
    var database = databases.Single();
    database.AssignProperty(x => x.Options.AutoscaleMaxThroughput, "1000");
});

The API is still experimental but it will allow for quite a bit of customization. This ultimately affects the Bicep that is produced that AZD uses when deploying the resources.

@mitchdenny Is this expected to make it into the next preview?
This definitely addresses the concern I had here more meaningfully than what I took from the latest preview 4 release notes. They seemed to indicate a greater reliance on the creation and maintenance of Bicep files as part of the AppHost project and that Aspire would be limited to simply passing parameters into these self-maintained adjacent files. That's nice as a fallback, but where Aspire otherwise generates all the rest of the Bicep from scratch, with a rare exception, this feels like it'd be a notable step backwards to be generating all manner of deployment markup just to have to opt out of it for whole swaths of resources that need minor configuration tweaks.
How well does this handle Azure resources with several levels of nested objects (e.g. Front Door or Event Hubs)? And is it able to handle specifying quirky behavior like assigning existing user-assigned managed identities to resources?
I looked around but was unable to find anything about Azure.Provisioning (Bing just pulls up Entra and IoT provisioning scripts). Is there anything public about it yet?
Thanks for the follow-up!

This builds on that layer and is in preview 5.


Issue when specifying ax object for plotting

Thanks for this code, it's been useful so far !
I discovered a small issue in the _plot() method. When specifying an ax object only the first reference would be plot.
The issue can be replicated as:
fig, ax = plt.subplots(ncols=3,nrows=3,figsize=(16,12))
ind1 = detectPeaks.detectPeaks(a1, show=True, ax=ax[2,0], threshold=0, kpsh=True, mpd=2, mph=5e-2)
ind2 = detectPeaks.detectPeaks(a2, show=True, ax=ax[2,1], threshold=0, kpsh=True, mpd=2, mph=5e-2)
The workaround is easy:
if ax is None:
_, ax = plt.subplots(1, 1, figsize=(8, 4))
noAx = True
else:
noAx = False
(...)
if noAx:
plt.show()

corrected. thank you.


Connect Users with Groups

Right now, when someone would sign up to create an account, the groups listed on the homepage are not unique/specific to the user. We need to figure out how to connect each user to his/her own list of groups.

Need to improve schema representation (Users: need an id; Create an association table (stores both userIds and groupIds, to associate users with their groups); Groups: needs an id, can probably scrap the "group members" attribute; Need a calendars; Events: needs id, should be endtime instead of duration.)
and figure out how ruby exactly handles the database (SQLight files written how and where?)


Fix example

The next reference is missing in the sample:
var encoder = mp4.encode()
var decoder = mp4.decode()

decoder.on('box', function (headers, next) {
  decoder.decode(function (box) {
    encoder.box(box, next)
  })
})

fs.createReadStream('my-movie.mp4').pipe(decoder)
encoder.pipe(fs.createWriteStream('my-movie-copy.mp4'))

I think this is the old syntax that is no longer supported in v2 - is that correct @mafintosh?


Overwrite mode batch change

The overwrite mode is useful, but it does not cover batches, which is the main use for which it was requested. The idea is that you can overwrite files if they already exist instead of adding to the existing sequence. That way you can repeat batches in the same folder. Currently, it overwrites the same file, which limits its use.

It's unfortunately not possible. The only reason it can make increment files is because it checks the files in the dir. So it counts all the files, and then increments + 1 for the new file, so it doesn't overwrite files.
It's not possible for the node to incrementally overwrite files, because it has no idea what it overwrote on each run (every time you run the script, it's accessing the node files fresh, as if launched for the first time). So every run, it would just overwrite ComfyUI_00001_.png every time, because that's the first file it would encounter when it starts a counter to check files. And since it runs fresh every time, will always start at 1.

The numeric condition only reads if there is a change, so any other condition should fulfill the same function. For example, store the file's last modification date and add a +1 if it has changed, or so on. That would allow batches to be read and overwritten freely.

Separate question. How do I reset the batch counter? It doesn't matter what I put in the index textbox, it continues from the image where the last batch ended. Also, now there is no txt so I can't reset the counter manually.
Not having txt is good, anyway. It crashed batches in Auto1111 and some scripts. I would just need to be able to decide where my batch starts.

Separate question. How do I reset the batch counter? It doesn't matter what I put in the index textbox, it continues from the image where the last batch ended. Also, now there is no txt so I can't reset the counter manually. Not having txt is good, anyway. It crashed batches in Auto1111 and some scripts. I would just need to be able to decide where my batch starts.

Delete the settings database, or change the label name. It also resets if you change the path under same label.

The numeric condition only reads if there is a change, so any other condition should fulfill the same function. For example, store the file's last modification date order and add a +1 if it has changed, or so on. That would allow batches to be read and overwritten freely.

That would then lead to confusion and possible unrecoverable images because it would have the same flaws as batch loading and relying on a database.

I guessed about changing the label, but it's very impractical in complex workflows with multiple nodes, let alone the other options. A selectable "continue counter" or "reset batch" mode would solve the root problem and thousands of unnecessary clicks and typing in a work day.
I didn't understand why you should care what people overwrite. Whoever chooses an overwrite mode will know that they are replacing files with the same name format. No one is going to choose a folder with files they don't want to overwrite, and if they do, they aren't very smart. I don't think it's a reasonable approach to not implement an override mode, in any case, any mode will have an "what if" that could cause problems by mistake. That is something that rests on the user.

I guessed about changing the label, but it's very impractical in complex workflows with multiple nodes, let alone the other options. A selectable "continue counter" or "reset batch" mode would solve the root problem and thousands of unnecessary clicks and typing in a work day.

That's a request for ComfyUI to support such features.

 From zero knowledge of coding I have written a program and combined thousands of lines of functional code the last few weeks just looking at examples and asking Bing. I understand what can and cannot be done, and I understand that your nodes can perform the functions efficiently if modified correctly. You talk about flaws, but the limitations of your nodes are flaws. If you don't want to face them, that's fine. You don't want to add features, that's fine. But don't make cheap excuses or talk like you've already achieved the most flexible and efficient functionality and it would be remiss to change it, because it's not true. Good luck with your suite. 🖐

You don't seem to understand. I know they are flaws. And all you've suggest is I do the same thing I've done with batch loading, which I've immediately explained why that's a terrible idea. Messing with some code with Bing is a far cry from the fundamentals and practices. We don't need to introduce weird functionality like batch loading when it comes to overwriting files. Just make a new directory.


Parse channels similar to how names are parsed

@JohnDoe#1337 is turned into a user mention, channels (#example) should be turned into channel links.

I've done a pull request that should fix this – #476. If you know how to, you could check out that branch and test it out in the meantime.


Backports opened against wrong parent PR

I've run into a strange scenario every once in a while that I haven't been able to track down.
Rarely, we will have the backport script run and one backport PRs that have the correct commits, but are associated with the wrong parent PR.
For example, PR 1234 will merge, which kicks off the script. It runs with a command similar to:
backport --repo "repo" \
                --accessToken "${GITHUB_TOKEN}" \
                --pullNumber 1234 \
                --noFork \
                --nonInteractive \
                --commitConflicts \
                --label "automated backport" \
                --source-branch "sourceBranch" \
                --target-branch "targetBranch"

the output of the backport job correctly lists the commits from PR 1234, but incorrectly lists the source PR as something different, say 2345 (which is a still-open PR).
Any debugging suggestions would be helpful. I haven't been able to track down why this mismatch is occurring, and it's been causing some confusion when it happens.

We are still hitting this on a somewhat regular occasion. In some cases, the calculated "source pull request" is one that hasn't even merged yet. In some cases, the actual commit that is cherry-picked is from a third PR unrelated to the other two.
I haven't been able to replicate locally, as it works as expected each time. I'm not sure if there is some git state that I am getting stuck in causing this issue.
Any input on how we can debug or resolve this would be greatly appreciated.

Hey, sorry for the slow reply. I would like to look into this problem and will need a bit to work with. Next time this happens can you try getting the relevant logs?
The logs are places in your homefolder: ~/backport/backport.debug.log).
If you have jq installed you can get the logs from a specific data:
 cat .backport/backport.debug.log |  jq 'select(.timestamp >= "2024-08-31")'

We don't currently have access to those logs. I will make some changes to store those so that we can provide them if the issue occurs again I the future.

We hit one variation of this recently and were able to get the logs.
In this case, the backport was made against "PR A" from branch release/v4.8 into branch release/v4.9. The backport was able to pull the correct commit, but the "sourcePullRequest" was incorrect, leading to the tool commenting on the wrong parent PR (and the link in the backport PR pointing to the wrong PR despite having text indicating it was pointing to the correct parent PR).
In addition, the incorrect PR that was listed as the source PR had not merged at the time that this tool ran.
backport.debug.log


Add example notebooks for sklearn, xgboost

Need to create multiple notebooks with multiple scenarios for both sklearn, xgboost for local deployment.

single function call
nested function call
multi data object
no data object.

@chaudharyachint08

let's also fix issue along the way.

Hello @asingh9530
can I solve this issue?


return value of save_result

https://github.com/Open-EO/openeo-processes/blob/60fb5e22579f34010de42af71c3d2d70f2dea906/save_result.json#L43-L48
return value of save_result is true or false, but if save_result fails, a backend will probably throw a descriptive error before returning false, so nobody is going to check the actual return value.
Wouldn't it be more useful to just return (pass through) the input data so that it can be re-used for further processing?
This would allow for example:

store  intermediate results in a batch job
store result in multiple formats without having to re-execute same job

This is totally valid, but you are saying it is not necessary?

that's a valid process graph, but is a back-end forced to "execute" nodes that do not contribute to the result?

Yes, I think so. The API says "multiple end nodes are possible".

(sorry for delay)

Yes, I think so. The API says "multiple end nodes are possible".

It currently says indeed

One of the nodes  .. (the final one) MUST have the result flag set to true, ... Having such a node is important as multiple end nodes are possible,

There s is some room for interpretation here.  It doesn't explicitly say that a backend must fully evaluate non-result end nodes. Analogy: a C or Java program can have multiple functions, but only main is triggered when the program is executed and there is no guarantee that all code will be visited.
The obligation to execute all end nodes instead of just the "result": True node has quite a big impact:

the process graph parsing and (spark based) execution in VITO driver heavily depends  on  the assumption that there is only one final node to evaluate
The data cube building approach in the python client using chained methods on a "DataCube" object also depends on that: you call .execute() (or its batch variant) on a "DataCube", which is by design a reference to a single node (the result node). The current machinery in the Python client does not support building or executing a process graph with multiple result nodes.

I think it should be made more explicit in the API description how to handle a process graph with multiple end nodes.
From the above it should be clear that I favor the "only execute the result node" model. I think it's a simpler model, more straightforward to implement in both clients and back-ends and allows cleaner client API's for the end user.
Note that it still possible to "emulate" multiple result nodes in this model: you just collect all your result nodes in a final array_create process that acts as single result node. If necessary it could be handy to define a dedicated process for this (e.g. collect) at the API level  or just provide a helper at the client level.

(sorry for delay)

No worries. I migrated over to https://github.com/Open-EO/openeo-api/issues/427


Lazy components can return arrays now

Fix #695 only (with corresponding tests but without other code in #684)

Does it add support for returning an array from a root view?

No, @frenzzy, not now.

Thanks, @infinnie, but I'm not going to add this.


Store the Ado bugs and title in a table

To reduce the load on Ado. and give us the opportunity to implement better deduplication algorithm,
We should store every bug we open from in ADO in the database.
This could be a duplicate of #3367 But keeping this scenario separate in case the storage format is different for both usages.
AB#164198

and Bug ID


Enable Sourcegraph

I want to use Sourcegraph for autoform-demo code search, browsing, and usage examples. Can an admin enable Sourcegraph for this repository? Just go to https://sourcegraph.com/github.com/aldeed/autoform-demo. (It should only take 30 seconds.)
Thank you!

Done


Need help with dialouge

Describe your problem here.
I need help with the dialogue, ive seen tutorials but it doesnt play
Are you modding a build from source or with Lua?
Lua
What is your build target?
Mac
Did you edit anything in this build? If so, mention or summarize your changes.
No response

just code with source code

just code with source code

Well i cant, im on mac.

just code with source code

Well i cant, im on mac.

why do u have a apple device

Anyways, Im going to get a windows 11 for christmas, so ill just start doing it later.

Anyways, Im going to get a windows 11 for christmas, so ill just start doing it later.

just install a cracked version of win10....

Can I see your Code?

Can I see your Code?

for?

local allowCountdown = false
function onStartCountdown()
-- Block the first countdown and start a timer of 0.8 seconds to play the dialogue
if not allowCountdown and isStoryMode and not seenCutscene then
setProperty('inCutscene', true);
runTimer('startDialogue', 0.8);
allowCountdown = true;
return Function_Stop;
end
return Function_Continue;
end
function onTimerCompleted(tag, loops, loopsLeft)
if tag == 'startDialogue' then -- Timer completed, play dialogue
startDialogue('dialogue', 'roses-tylenol');
end
end
-- Dialogue (When a dialogue is finished, it calls startCountdown again)
function onNextDialogue(count)
-- triggered when the next dialogue line starts, 'line' starts with 1
end
function onSkipDialogue(count)
-- triggered when you press Enter and skip a dialogue line that was still being typed, dialogue line starts with 1
end


1.12.2材质问题

https://github.com/HoshinoTented/Mod-1.12.2/issues/1

thank you, it is useful for me


⚠ Bofrost.de has degraded performance

In a479a3e, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1246 ms

Resolved: Bofrost.de performance has improved in 49d4b41 after 9 minutes.


Dia.app requires XQuartz but it is not listed as dependency

Dia.app (package name: dia) requires XQuartz (package name: xquartz) to work. If XQuartz is not installed a dialog is opened when starting the app suggesting the installation of it.
Isn't it better if XQuartz was listed as a dependency of Dia, instead of the app suggesting a manual install?

Done.


use the right $ as UMD Module

Hey :)
thank you first for this plugin, it makes things a lot easier while dealing with DFP :).
I changed the way the module loads, because the current UMD implementation is not correct.
(function($, window, undefined)(window.jQuery....)

The $ shadows the underlying implementation and so a global $ is always used by the plugin (which is not available when loaded with a module loader). I changed it in a way that if a module loader is in scope all dependencies will be resolved first and then delegated to your implementation.
Best
David

Thanks for this... I will merge once I get a chance to look at the change properly... github is giving me a really messy diff!

Sorry for the diff :/, I had to move everything and restore the intention. Thanks for looking.

There we go... thanks for the fix!

Just an fyi with regards to the messy diff: if you add "?w=1" to the url, whitespace changes are ignored and it clears it up quite nicely.


Prepare for v4 LTS and v5

From https://github.com/nodejs/node/issues/3000 where this question is something to be resolved by the @nodejs/website WG:

[ ] How do publicise and promote LTS lines? Does @nodejs/website have ideas on how to make this happen when we have something called "LTS" and then start to get new "Stable" releases with a 5 at the front? Ideally, users will be self-selecting and would end up with the kind of release suited to them, perhaps we need to outline on the front page of the website and/or on the download page why you would choose LTS over Stable?

Until the first item in the list @ https://github.com/nodejs/node/issues/3000 is resolved I think you need to be prepared for the following scenarios:

a v5 and a v4 LTS: this will be the eventual situation no matter what but this is a possible scenario from next week onward
a v4 LTS only: do you do anything special for this or just treat it as yet another v4? It's possible we have this scenario next week

Also, we probably should have wording for what LTS actually means, to put somewhere, feel free to kick that over to @nodejs/lts and make them do it if you want, either open a new issue and describe exactly what you need or add it in https://github.com/nodejs/LTS/issues/43.

Will there be changes to https://nodejs.org/dist/index.json?

cc @nodejs/evangelism
@mikeal can we bring anyone from foundation-level marketing in to discuss this?

@phillipj perhaps a lts column/field in index.json and index.tab would be good, it could include the codename for lts releases

@rvagg sounds like a good plan. I'll could do some tests with that in mind for scenario 1, probably in the weekend / early next week.
For scenario 2, as long as file structures on /dist/ stays the same and we find it in index.json, no changes would be needed for us. Although adding some hard coded LTS strings around could be helpful to emphasize it, e.g. Current version: v4.x.x LTS on the frontpage

From an Evangelism perspective, there shouldn't be much of a problem. Weekly or twice-weekly tweets, as well as listing of updates in blog posts, are easy. However, will we be attempting to pull people into either using or contributing to LTS? Will it have independent contributors, outside of core?

Alright, to get the ball rolling I'll post some screenshots with some minimal changes necessary to host two release lines at the same time. All kind of feedback is much appreciated!
I'm thinking we should keep it as simple as possible to download the version of node most devs will use, which I've been told is stable, soon to be v5.
On the frontpage an LTS-link could been added beneath the big green download button, sending users directly to e.g. https://nodejs.org/dist/v4.2.1/.

As for the download page, a couple of links could be added to do some JS toggling of the URLs in the download matrix. An explanation of what LTS is and its pros/cons should also be added, either as a new section on the download page, or a completely new page which explains the release lines in great detail with content provided by @nodejs/lts as mentioned in https://github.com/nodejs/LTS/issues/44.

Any thoughts?

I would much rather see the two releases side by side with clearly marked
download buttons for each rather than having LTS behind some easily
overlooked small link text.
On Oct 15, 2015 12:58 PM, "Phillip Johnsen"<EMAIL_ADDRESS>wrote:

Alright, to get the ball rolling I'll post some screenshots with some
minimal changes necessary to host two release lines at the same time.
All kind of feedback is much appreciated!
I'm thinking we should keep it as simple as possible to download the
version of node most devs will use, which I've been told is stable, soon
to be v5.
On the frontpage an LTS-link could been added beneath the big green
download button, sending users directly to e.g.
https://nodejs.org/dist/v4.2.1/.
[image: screen shot 2015-10-15 at 21 19 42]
https://cloud.githubusercontent.com/assets/1231635/10525570/c2808778-7385-11e5-9981-045c40c5e992.png
As for the download page, a couple of links could be added to do some JS
toggling of the URLs in the download matrix. An explanation of what LTS is
and its pros/cons should also be added, either as a new section on the
download page, or a completely new page which explains the release lines in
great detail with content provided by @nodejs/lts
https://github.com/orgs/nodejs/teams/lts as mentioned in nodejs/LTS#44
https://github.com/nodejs/LTS/issues/44.
[image: screen shot 2015-10-15 at 21 39 14]
https://cloud.githubusercontent.com/assets/1231635/10525576/c78239d8-7385-11e5-80ec-05511653c5a1.png
Any thoughts?
—
Reply to this email directly or view it on GitHub
https://github.com/nodejs/new.nodejs.org/issues/200#issuecomment-148504034
.

Yes, my first preference is that if you don't choose carefully you end up with the LTS release.  ie unless you know what you are doing and know that you want the latest/greatest you end up with an LTS release.

I would much rather see the two releases side by side with clearly marked
download buttons for each rather than having LTS behind some easily
overlooked small link text.

Understandable. I appreciate when website are focused on the most important part(s) which should be relevant for the majority of users. On the other side, we cant really predict what our users really want before seeing the download figures, although we'll ofc influence those numbers by just having one button from the get go.

Could there be something user focused like: "looking for updated solid and
stable? → LTS • want to try out the latest features? → latest"
So try to switch based on the users self identification.

Thanks for your input so far! A couple of new screenshots:
1) Two green buttons side by side

2) Use oriented texts w/links

If I misunderstood your thoughts @jasnell @srl295, just say the word and I'll happily do some changes.
Anyone got a favorite amongst those two alternatives?

I prefer the first one. People tend to skip reading the "small print", so IMHO it's better to have some big shiny buttons in this case.

+1 to the first one. I like it a lot.

+1 to big buttons

+1 Buttons, can we maybe get some hover text?

Also, maybe we should have some slightly different coloring associated with them.

How about moving some of the text from 2 into the buttons ?  Maybe the "Solid and Stable"
for LTS and " Latest features" for Stable

Would that cover your needs for some hover text @Fishrock123?

Maybe. But we need to change it from having a stable in each.
Perhaps we should have the lts description be how many months it's supported?

(the stable branch is stable)

Possibly "Solid and Supported" for  LTS ?

Again, the stable branch is most definitely supported. :)
I think we need wording about the longevity of the LTS's support, not about it's supposed stability compared to the stable.

I think we need wording about the longevity of the LTS's support, not about it's supposed stability compared to the stable.

Sounds like something @nodejs/lts could help us with.

How about:
LTS: "Mature and Dependable"
Stable: "Active Development", or "Latest Features" is fine too.

Sometimes these things have a small but colorful "button'/link below that sais something like
"trying to decide which version you need?"
that takes them to a place where they can understand the difference between the two and choose wisely. I don't like the idea of them navigating to another page for this but maybe simply a little drop down model(or whatever those are called) that explains in a few lines.

Went with "Mature and Dependable" and "Latest Features" for LTS and Stable respectively in #260.
These taglines and other methods of explaining LTS/Stable differences should be discussed and iterated on going forward. As the v5 release is getting dangerously close (I assume), I think its time to merge the proposed changes ASAP, to ensure users of node.js can download these different versions as soon as they're available.

@phillipj  / all - should there a link to some text such as https://github.com/nodejs/LTS/  if users need more help? I'm seeing lots of confusion on twitter :(   Maybe "Unsure of which version?"
Examples:

Tomcat: Which version?
Python: Wondering which version to use? Here’s more about the difference between Python 2 and 3.

should there a link to some text such as https://github.com/nodejs/LTS/ if users need more help?

Yes, it would be great if we could do a better job helping people choose version! Might be worth trying to resurrect the stale discussion in https://github.com/nodejs/LTS/issues/44.