Postgres

Feel free to Edit, Suggest, Comment, Push, etc.

2FA is a field on User table.
2FA Secret is Base32 as best practice according to standard.
Added Terms Of Service (TOS) page, pointing to a static HTML TOS Template.
Added Bulma styles to Logs page.
Added Bulma styles to Server Info page.

... firejail
.. People seems to love it:
...

And I do to! THIS IS AWESOME! I wasn't aware of firejail, but I'm gonna dig into it now.

-d:noUnsplash to Disable.

Postgres
(extract(epoch from now()))

Gives:
Error: unhandled exception: ERROR:  column "creation" is of type timestamp but default expression is of type double precision

I changed it to now(), but can be improved, it seems it accepts ISO formatted but not literal numbers?.

Ready to test it out @ThomasTJdev :tada: :boom: :fireworks:

Ready to test it out @ThomasTJdev    To try all new stuff...
I tested this on another PC and worked ok.

Great! I'll try add a checklist, please pitch in, so we are testing everything.

Do you prefer -d:demo to automatically force -d:sqlite -d:noWebp -d:noUnplash

I don't think so. I'll evaluate during the test.

[ ] General UI check
[ ] Files - UI
[ ] Files - Folder (from image)
[ ] Files - Count (#45)
[ ] Plugin - UI
[ ] Plugin - Enable/Disable route
[ ] Docket
[ ] Firejail
[ ] WebP
[ ] Unsplash
[ ] Postgres
[ ] DevOps folder
[ ] Datetime2human integration
[ ] Online config
[ ] Admin panel
[ ] Honeypot
[ ] 2FA
[ ] Password generater (random)
[ ] Logs - UI
[ ] Nimble install on a clean system
[ ] Nim latest devel
[ ] Check changelog

Other

[ ] Error using Summernote with Bulma (#37)
[ ] Nim's symlink proc cant overwrite existing symlink (#36)
[ ] Version bump to 5

I did not meant to Fix #36 , #37 , #45 nor Bump to 5, but I think is good to have small iterations,
and come back to master before Diff gets uncontrollably huge and hard to review.
Dockerfile is meant as a template, something you copy over and modify, not something that will run as-is.

This is a really nice PR with so many great features and improvements!! Great work @juancarlospaco !

2FA
Would it be okay with you, if I make these changes?

I would like to make it optional to enable 2fa.
The 2fa random-key should be generated and shown to the user. The user can then use a 2fa-software/app to save it.
The user needs to actively enable 2fa from the profile page.

To do: We should first check pwd and email, if they match we should ask for 2FA. Or we can keep the current layout to force the user to think about 2FA on every login.
nimwc2fa.nim
Can't get the file to work:
os.nim(1385)             paramStr
Error: unhandled exception: invalid index [IndexError]
Error: execution of an external program failed: './nimwc2fa '

Unsplash
I couldn't get it to work.. where should the pictures change? How to use it? My main folder is getting the pics.
Would it be an idea to move it to a plugin?
TOS
Could be expandede? Like https://cxmanager.dk/eula ?
The font didn't work for me. I changed it to font-family sans-serif.
Shouldn't we make it editable? Like the config file. Then the user can specify the details to comply with GDPR.
-d:noFields
Purpose?
History table
Purpose?
Postgres "WITHOUT OIDS"
Is this needed? https://www.postgresql.org/docs/9.6/datatype-oid.html
"Object identifiers (OIDs) are used internally by PostgreSQL as primary keys for various system tables. OIDs are not added to user-created tables, unless WITH OIDS is specified when the table is created, or the default_with_oids configuration variable is enabled. "
Webp
The quality is really low - we should make it optional, where the user can choose to use webp.
nim.cfg
I have added -d:noFirejail and -d:noUnsplash into the cfg file. Firejail requires foreign dependency and I'm not sure of unsplash.
I'm okay by including both, but I think they should be disabled as standard, and the user could then enable them from the UI and NimWC could then recompile to enable them?
To do:
files.tmpl should use the files-table instead of walkdir on files/privat-public. We can then also show how many times, a file has been downloaded.
Split readme features into categories for better overview
Update NimWC.org with information about all the new features :)

I would like to delete nimwc2fa.nim after df183369e81c40ad9853ab2cb193de2836356b30.
Ok?

I think we should go for a version 5 :smiley: . What do you think? c03f7eea5884923b3f5d9e957685c313f5ecabe2

Gatabase is only used when connecting to the DB. If you don't use in other places, I suggest we remove it for now to strip down dependencies.

I have tried to split the README into sections. I think it needs a double check 65b9202458be9227cd0a64126925d7895411f60f

I would like to Merge this for the next month or before, like 1 march or near, 325 Commits already.
Looks good to me, what do you think @ThomasTJdev :grey_question:

I would like to Merge this for the next month or before, like 1 march or near, 325 Commits already.
Looks good to me, what do you think @ThomasTJdev

Agree. I would like to do some more test, but we will merge before monday the 4th of March - hopefully before?

Just for historical reference:
DropBox added Unsplash: https://medium.com/unsplash/unsplash-for-dropbox-paper-d7c0c425d2d0
Sketch, Adobe, Trello, Google, Medium, Square, Semplice, Typeform, Weebly, PicsArt, Ghost, CodePen, Unfold, Product Hunt, Anchor, InVision, Unbounce, FiftyThree already had Unsplash.
Docker only is Insecure: https://snyk.io/blog/top-ten-most-popular-docker-images-each-contain-at-least-30-vulnerabilities

DropBox added Unsplash: https://medium.com/unsplash/unsplash-for-dropbox-paper-d7c0c425d2d0
Sketch, Adobe, Trello, Google, Medium, Square, Semplice, Typeform, Weebly, PicsArt, Ghost, CodePen, Unfold, Product Hunt, Anchor, InVision, Unbounce, FiftyThree already had Unsplash.

Unsplash is currently not active as standard (not present in nimwc.nim.cfg). When you create a website - blog, portfolio etc. don't you think, that the user would like to choose static background image? I'm all in for having it as a part of NimWC, but I would like the user to manually turn it on.

I would like to implement these 2 things before release - ok?

a view-counter on blogpost (like the one on file-downloads)
a "fake" publishing date for blogpost. You can write a date, which is shown in the blogoverview. Sometimes you might want to create a blogpost in private mode and publish it later, but the end-user is not supposed to know when you created it.

I removed Unsplash and Gatabase.
For this Pull requests I go with vanilla Postgres but we may implement Norm later on,
and for Unsplash a better UX is needed so maybe it come back someday. I think is important to come back to master.
The features you mention sounds kind of cool!,
but I dont know how to implement them properly from that short description,
but no problem just push it and ship it!. I am just super happy with having Firejails in. :smile:

You can choose how much CPU & RAM NimWC can use.

https://gist.github.com/juancarlospaco/9ef1b803ef0cf6f71cde2dbec6839116#file-example-txt

Nice! Will you write a small How to config firejail for the README? Or can set the default settings, so it'll work on most systems (RPi's, EC2 tier, etc.)?

Good update on firejail in the readme :+1:

Question: What should our users standard DB be? SQLite or Postgres?
We currently have -d:sqlite in the nimwc.nim.cfg file, but inside the code we use when defined(sqlite). This is double negative, and the user needs to remove the line from nimwc.nim.cfg to enable postgres - there's no other way to force postgres. If sqlite is our primary DB, we should use when defined(postgres) and remove -d:sqlite from nimwc.nim.cfg.
I would prefer sqlite as standard. It makes it easier for a user to try NimWC without setting up a Postgres DB.
What do you think?

@ThomasTJdev I agree :+1:

@juancarlospaco I'm ready for merging :smiley: . You just say go, when you are ready!

GO :grey_exclamation:

@ThomasTJdev Need to Re-Deploy the Demo.
:fireworks: :fireworks: :fireworks: :fireworks: :fireworks: :fireworks: :fireworks: :fireworks:

I'm having 2 problems :disappointed:
Nimble file
Should we remove this line? It is a bin-file, so Nimble should compile it itself.
exec "nim c -r -d:release -d:ssl nimwc.nim"

Firejail
When running I get this:
ubuntu@development:~/tmp/nim_websitecreator$ nimwc
2019-03-01T21:54:21+00:00: Nim Website Creator: Launcher starting.
Error: invalid --keep-dev-shm command line option
2019-03-01T21:54:23+00:00: Restarting in 1 second.
Error: invalid --keep-dev-shm command line option
2019-03-01T21:54:26+00:00: Restarting in 1 second.
^CError: unhandled exception: No such process [OSError]

Remove the line if you want, you are right.
For the Demo you can compile without Firejail.
Nimble should copy that file anyways, you are right.
Are you sure you have the latest Firejail module?, you ran nimble refresh ?.
I updated both Firejail and WebP during this Pull Request, so reinstall them to update them.

This is a clean test install the server. I can compile without firejail.
I have the newest firejail-nim module from nimble.
Firejail version:
ubuntu@development:~/.nimble/pkgs/nimwc-5.0.0$ firejail --version
firejail version 0.9.52

Compile time support:
	- AppArmor support is enabled
	- AppImage support is enabled
	- bind support is enabled
	- chroot support is enabled
	- file and directory whitelisting support is enabled
	- file transfer support is enabled
	- git install support is disabled
	- networking support is enabled
	- overlayfs support is enabled
	- private-home support is enabled
	- seccomp-bpf support is enabled
	- user namespace support is enabled
	- X11 sandboxing support is enabled

I'm using the raw config.cfg - no changes has been made.

<IP_ADDRESS> is latest release of Firejail https://firejail.wordpress.com/download-2/
If you want leave the Demo without Firejail, whatever you like no problem.

<IP_ADDRESS> is latest release of Firejail https://firejail.wordpress.com/download-2/
If you want leave the Demo without Firejail, whatever you like no problem.

Demo without firejail is ok.
Is v<IP_ADDRESS> required? On ubuntu I can't get higher than *.52 on standard library:
ubuntu@development:~/.nimble/pkgs/nimwc-5.0.0$ apt search firejail
Sorting... Done
Full Text Search... Done
firejail/bionic,now 0.9.52-2 amd64 [installed]
  sandbox to restrict the application environment

firejail-profiles/bionic,now 0.9.52-2 all [installed,automatic]
  profiles for the firejail application sandbox

firetools/bionic 0.9.50-1 amd64
  Qt frontend for the Firejail application sandbox

They have the *.DEB packages here: https://sourceforge.net/projects/firejail/files/firejail/ (is Pure C).

They have the *.DEB packages here: https://sourceforge.net/projects/firejail/files/firejail/ (is Pure C).
--keep-dev-shm is Not being used, if you like just remove the line from nim-firejail.
But being kinda security package I dont feel like supporting legacy or deprecated too much lol.
Whatsoever is Distros fault not our fault.

I like the security approach, but it could make it hard to support AWS Ubuntu images or Raspberry Pi's. RPi has a backport version 9.44.
Can you compile on a RPi? I'm having trouble (with and without firejail).
nimwc_main.nim(582, 15) template/generic instantiation of `generateRoutes` from here
/home/pi/nim/Nim/lib/core/macros.nim(496, 1) template/generic instantiation of `routes` from here
../../jester-0.4.1/jester.nim(1269, 35) template/generic instantiation of `async` from here
resources/email/email_registration.nim(48, 6) Warning: 'sendEmailActivationManual' is not GC-safe as it calls 'sendEmailActivationManual_continue' [GcUnsafe2]
nimwc_main.nim(582, 15) template/generic instantiation of `generateRoutes` from here
/home/pi/nim/Nim/lib/core/macros.nim(496, 1) template/generic instantiation of `routes` from here
../../jester-0.4.1/jester.nim(1269, 35) template/generic instantiation of `async` from here
/home/pi/nim/Nim/lib/pure/asyncmacro.nim(267, 31) Warning: 'matchIter' is not GC-safe as it calls 'sendEmailActivationManual' [GcUnsafe2]
/home/pi/nim/Nim/lib/core/macros.nim(1214, 49) Error: conversion from int64 to int is invalid
 Compile Error

  ⚠ Compile-time or Configuration or Plugin error occurred.
  ➡ You can check your source code with: nim check YourFile.nim
  ➡ Check the Configuration of NimWC and its Plugins.
  ➡ Remove new Plugins, restore previous Configuration.
   Check that you have the latest Version. Check the Documentation.

They have the *.DEB packages here: https://sourceforge.net/projects/firejail/files/firejail/ (is Pure C).
--keep-dev-shm is Not being used, if you like just remove the line from nim-firejail.
But being kinda security package I dont feel like supporting legacy or deprecated too much lol.
Whatsoever is Distros fault not our fault.

www.nimwc.org is up and running with firejail :smiley: (I need to perform some upgrades, so there might be a little downtime next hour)
If --keep-dev-shm  isn't used, could it then be comment out. And if it's going to be used in the future, it can be reimplemented?

I just bought this to test Nimwc on ARM, but did not have time to install the OS yet.

Are you sure the OS is 64Bit?.
The versions available for ARM should be the same versions than desktop.
Ok about the --keep-dev-shm.

Can you compile on a RPi? I'm having trouble (with and without firejail).

I have tried compiling from source, but with no luck. Have you had any luck upgrading firejail on a RPi? If we can't support RPi out of the box, we should:

Inform about it in the readme and do a os-check to avoid compiling with firejail
Check local firejail version, and only compile with firejail if version is supported
Update nim-firejail to only use "old" functions when it's an old firejail version

Have you tried ArchLinux on it?, maybe is Distro-specific Bug.

Have you tried ArchLinux on it?, maybe is Distro-specific Bug.

It's running standard Raspbian. But I'm running Jessie, and the current stable is Stretch. I'm upgrading now to retest since Strecth has 9.58.

Have you tried ArchLinux on it?, maybe is Distro-specific Bug.

It's running standard Raspbian. But I'm running Jessie, and the current stable is Stretch. I'm upgrading now to retest since Strecth has 9.58.

Stretch has firejail 0.9.58 - so firejail is fine on a RPi :+1: . But I'm still getting the conversion error from macros.nim with both 0.19.9 and 0.19.4.


#16: Move logic about getting access token to a separate class

@rultor merge

@rultor merge

@proshin-roman OK, I'll try to merge now. You can check the progress of the merge here

@rultor merge

@proshin-roman Done! FYI, the full log is here (took me 3min)


New PR to test SignCLA at 2019-04-18 15:12:17.488007267 -0700 PDT m=+6.208421264

Body of PR

unsigned-fail-fork-staging: test succeeded


:bug: Fixed a bug in asdf list-all bun (because repo name was changed)

The bun repo name had been changed, so fixed it.
And also I changed curl option to prefer redirected url if repo name was changed.

Thanks !


Add combined option "Europe" to quick-hop region

If you are based in Europe, the UK and Germany worlds usually have very similar ping, being able to utilize a larger pool of worlds for quick hopping while maintaining a decent ping would be handy.

added in f6a3d222b4986ab7beab54b650797f2821f5852c


Chrome plugin: Error saving bookmark: TypeError: Failed to fetch

Chrome plugin: Error saving bookmark: TypeError: Failed to fetch

Please open an issue in the extension repo and provide some more details, such as which URL you tried to bookmark and maybe what your configuration looks like. Otherwise I have no way to figure out what your problem is.

It seems like this error randomly pops up on different website, but on the same website works just fine after a while. Looks like a network issue, because I am behind a company proxy.


Add launch param overrides

It became more expedient to test certain run configurations by directly overriding sim params in mbari_wec.launch.py
@hamilton8415 I also added the swap for Custom spectra to use freq in Hz in this PR. Incidentally, this is the reason I wanted to be able to override params to test specific configs without batching. I found an instability in the sim for the following config:
ros2 launch buoy_gazebo mbari_wec.launch.py inc_wave_spectrum:='inc_wave_spectrum_type:Custom' seed:=42 physics_rtf:=10 physics_step:=0.09 door_state:=closed scale_factor:=0.5 battery_soc:=0.25 x_mean_pos:=0.5

It fails almost immediately. If I change to physics_step:=0.01, it still crashes but takes longer. And, physics_step:=0.001 didn't crash in the few minutes I checked, but I can't guarantee it won't still run away eventually.
The buoy & heave cone fly off the screen pretty quickly.
The crash log:
[ruby $(which gz) sim-1] ODE INTERNAL ERROR 1: assertion "aabbBound >= dMinIntExact && aabbBound < dMaxIntExact" failed in collide() [collision_space.cpp:460]
[ruby $(which gz) sim-1] Stack trace (most recent call last):
[ruby $(which gz) sim-1] #31   Object "/lib/x86_64-linux-gnu/libruby-3.0.so.3.0", at 0x7fd3df434c96, in
[ruby $(which gz) sim-1] #30   Object "/lib/x86_64-linux-gnu/libruby-3.0.so.3.0", at 0x7fd3df431fc5, in
[ruby $(which gz) sim-1] #29   Object "/lib/x86_64-linux-gnu/libruby-3.0.so.3.0", at 0x7fd3df42fc34, in
[ruby $(which gz) sim-1] #28   Object "/lib/x86_64-linux-gnu/libruby-3.0.so.3.0", at 0x7fd3df37ba1e, in
[ruby $(which gz) sim-1] #27   Object "/lib/x86_64-linux-gnu/libruby-3.0.so.3.0", at 0x7fd3df2a69ac, in rb_protect
[ruby $(which gz) sim-1] #26   Object "/lib/x86_64-linux-gnu/libruby-3.0.so.3.0", at 0x7fd3df43ec61, in rb_yield
[ruby $(which gz) sim-1] #25   Object "/lib/x86_64-linux-gnu/libruby-3.0.so.3.0", at 0x7fd3df43a30c, in rb_vm_exec
[ruby $(which gz) sim-1] #24   Object "/lib/x86_64-linux-gnu/libruby-3.0.so.3.0", at 0x7fd3df434c96, in
[ruby $(which gz) sim-1] #23   Object "/lib/x86_64-linux-gnu/libruby-3.0.so.3.0", at 0x7fd3df431fc5, in
[ruby $(which gz) sim-1] #22   Object "/lib/x86_64-linux-gnu/libruby-3.0.so.3.0", at 0x7fd3df42fc34, in
[ruby $(which gz) sim-1] #21   Object "/usr/lib/x86_64-linux-gnu/ruby/3.0.0/fiddle.so", at 0x7fd3df5c844b, in
[ruby $(which gz) sim-1] #20   Object "/lib/x86_64-linux-gnu/libruby-3.0.so.3.0", at 0x7fd3df3fd088, in rb_nogvl
[ruby $(which gz) sim-1] #19   Object "/usr/lib/x86_64-linux-gnu/ruby/3.0.0/fiddle.so", at 0x7fd3df5c7d6b, in
[ruby $(which gz) sim-1] #18   Object "/lib/x86_64-linux-gnu/libffi.so.8", at 0x7fd3df5b9492, in
[ruby $(which gz) sim-1] #17   Object "/lib/x86_64-linux-gnu/libffi.so.8", at 0x7fd3df5bce2d, in
[ruby $(which gz) sim-1] #16   Object "/usr/lib/x86_64-linux-gnu/libgz-sim7-gz.so.7.5.0", at 0x7fd3df09980d, in runServer
[ruby $(which gz) sim-1] #15   Object "/lib/x86_64-linux-gnu/libgz-sim7.so.7", at 0x7fd3da3397e5, in
[ruby $(which gz) sim-1] #14   Object "/lib/x86_64-linux-gnu/libgz-sim7.so.7", at 0x7fd3da34647a, in gz::sim::v7::SimulationRunner::Run(unsigned long)
[ruby $(which gz) sim-1] #13   Object "/lib/x86_64-linux-gnu/libgz-sim7.so.7", at 0x7fd3da345b40, in gz::sim::v7::SimulationRunner::Step(gz::sim::v7::UpdateInfo const&)
[ruby $(which gz) sim-1] #12   Object "/lib/x86_64-linux-gnu/libgz-sim7.so.7", at 0x7fd3da3423e2, in gz::sim::v7::SimulationRunner::UpdateSystems()
[ruby $(which gz) sim-1] #11   Object "/usr/lib/x86_64-linux-gnu/gz-sim-7/plugins/libgz-sim-physics-system.so", at 0x7fd3a0127a86, in gz::sim::v7::systems::Physics::Update(gz::sim::v7::UpdateInfo const&, gz::sim::v7::EntityComponentManager&)
[ruby $(which gz) sim-1] #10   Object "/usr/lib/x86_64-linux-gnu/gz-sim-7/plugins/libgz-sim-physics-system.so", at 0x7fd3a0128101, in gz::sim::v7::systems::PhysicsPrivate::Step(std::chrono::duration<long, std::ratio<1l,<PHONE_NUMBER>l> > const&)
[ruby $(which gz) sim-1] #9    Object "/usr/lib/x86_64-linux-gnu/gz-physics-6/engine-plugins/libgz-physics-dartsim-plugin.so", at 0x7fd35c391a3b, in gz::physics::dartsim::SimulationFeatures::WorldForwardStep(gz::physics::Identity const&, gz::physics::SpecifyData<gz::physics::RequireData<gz::physics::WorldPoses>, gz::physics::ExpectData<gz::physics::ChangedWorldPoses, gz::physics::Contacts, gz::physics::JointPositions> >&, gz::physics::CompositeData&, gz::physics::ExpectData<gz::physics::ApplyExternalForceTorques, gz::physics::ApplyGeneralizedForces, gz::physics::VelocityControlCommands, gz::physics::ServoControlCommands, std::chrono::duration<long, std::ratio<1l,<PHONE_NUMBER>l> > > const&)
[ruby $(which gz) sim-1] #8    Object "/lib/x86_64-linux-gnu/libdart.so.6.12", at 0x7fd2d7566e73, in dart::simulation::World::step(bool)
[ruby $(which gz) sim-1] #7    Object "/lib/x86_64-linux-gnu/libdart.so.6.12", at 0x7fd2d754c705, in dart::constraint::ConstraintSolver::solve()
[ruby $(which gz) sim-1] #6    Object "/lib/x86_64-linux-gnu/libdart.so.6.12", at 0x7fd2d754afe1, in dart::constraint::ConstraintSolver::updateConstraints()
[ruby $(which gz) sim-1] #5    Object "/lib/x86_64-linux-gnu/libdart-collision-ode.so.6.12", at 0x7fd3629761cb, in dart::collision::OdeCollisionDetector::collide(dart::collision::CollisionGroup*, dart::collision::CollisionOption const&, dart::collision::CollisionResult*)
[ruby $(which gz) sim-1] #4    Object "/lib/x86_64-linux-gnu/libode.so.8", at 0x7fd35c51a266, in dxHashSpace::collide(void*, void (*)(void*, dxGeom*, dxGeom*))
[ruby $(which gz) sim-1] #3    Object "/lib/x86_64-linux-gnu/libode.so.8", at 0x7fd35c5227b7, in dDebug
[ruby $(which gz) sim-1] #2    Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7fd3dee287f2, in abort
[ruby $(which gz) sim-1] #1    Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7fd3dee42475, in raise
[ruby $(which gz) sim-1] #0    Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7fd3dee96a7c, in pthread_kill
[ruby $(which gz) sim-1] Aborted (Signal sent by tkill()<PHONE_NUMBER>3)

Couple of things.   Changing this to Hz requires the latest fs_hydrodynamics library, which I haven't quite get put on the ppa server.  I will do that.  Perhaps you're compiling that library though, and in that case I have to think about what the default "custom" spectra might be, that could be nonsense causing a really fast input wave...

At the moment, without the new library, it is only a variable name change (no actual math changed to Hz instead of rad/s)

We should probably figure out how to protect the sim from crashing in this manner with a loud warning that some values are getting clipped

Oh yes, of course, just a name change...  OK, but what spectra are you supplying with the custom designation. Perhaps the default being used in gazebo (or in the library) is screwy.  I will try to re-create...

default is
w = [0.0, 0.2, 0.4, 0.6, 2.0]
Szz = [0.0, 0.4, 1.0, 1.0, 0.0]

I also noticed the return code from the crash was not being captured by the LaunchService so I added some stuff to get the return code directly and properly log it to batch_runs.log. So, now we can actually see that the sim crashed on those runs.

I decreased testing time on buoy_tests down to ~35% of what it used to be, and it's still timing out in CI on github... We may need to look into using a faster self-hosted / cloud runner or pay for upgraded github runners. This is getting painful.

@mjcarroll @mabelzhang no matter what I do I cannot get the future to be ready here:
https://github.com/osrf/mbari_wec_gz/blob/3b97b6b8855918880d462075eae880253ecf7a0d/buoy_tests/testing_utils/utils.py#L327-L333
called from the pc_bias_damping_ros_feedback_py.launch.py test in buoy_tests:
https://github.com/osrf/mbari_wec_gz/blob/3b97b6b8855918880d462075eae880253ecf7a0d/buoy_tests/launch/pc_bias_damping_ros_feedback_py.launch.py#L70
for the service here:
https://github.com/osrf/mbari_wec_gz/blob/3b97b6b8855918880d462075eae880253ecf7a0d/buoy_tests/tests/fixture_server_sinusoidal_piston.cpp#L96-L129
The issue is only on the slow github CI. I've tried my trick of rate limiting the executor spin among various other things to no avail. The service call shows complete but the future isn't updated and the await blocks forever. It is repeatable in CI but I can't get it to happen on my machine.

@mjcarroll @mabelzhang no matter what I do I cannot get the future to be ready here:

Looking

osrf/mbari_wec_utils#51 should fix test issue

woot! tests pass in 28 minutes now!

@hamilton8415 a few good changes and bugfixes in this PR. I kept the original buoy height and piston position for tests and set mbari_wec.launch.py to use a better initial piston position and buoy height by default

Here's a comparison between a Bretschneider spectrum (Hs = 2m, Tp = 10s) with what we have been using as the default "custom" spectrum.  Presumably the crashes are occuring due to the excess high-frequency content, which would make some sense that a smaller time-step stabilizes things some.

Here's a candidate spectrum for a default custom spectrum, just a sub-sampling of the Bretschider spectrum from above:
w = [0.007, 0.164, 0.321, 0.479, 0.634, 0.793, 0.948, 1.106, 1.262, 1.419, 1.576, 1.736];  // rad/sec
Szz = [ 0,  0.000, 0.000, 0.760, 2.278, 1.518,  0.800, 0.414,  0.225, 0.129,  0.078,  0.0484];  m^2/(rad/sec)

And of course, once we switch to using the latest fs_hydro library which expects these spectrum relative to Hertz, these values will need to become:
f = [0.0011,  0.0262, 0.0511, 0.0762, 0.1009, 0.1262, 0.1509, 0.1760, 0.2009, 0.2259, 0.2508, 0.2762];  // Hz
Szz = [ 0, 0.0000, 0.0000, 4.7783, 14.3138, 9.5359, 5.0249, 2.6016, 1.4153, 0.8100, 0.4880, 0.3043];  // m^2/(Hz)

those values do better, but the sim still crashes after 45 mins (sim time) using RTF 10 and step 0.01

	Hmm, but it runs longer with bretscneider 2m/10s?  If that’s true it has to do with the smoothness of the supplied spectrum, which is surprising, and likely a bug…

	Get Outlook for iOS

From: andermi @.***>Sent: Friday, June 2, 2023 12:58 PMTo: osrf/mbari_wec_gzCc: Andrew Hamilton; MentionSubject: Re: [osrf/mbari_wec_gz] Add launch param overrides (PR #150)
those values do better, but the sim still crashes after 45 mins (sim time) using RTF 10 and step 0.01
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

bretschneider went for over an hour without crashing

@hamilton8415 try this with osrf/mbari_wec_utils#52

With regard to the issue of a custom spectrum being selected without a spectrum being specified, I have worked on this to try and identify why this is, it seems that it's related to the sparseness of the custom spectra specifications (i.e. spectrum specified at only a few frequencies).  I have investigated this and haven't found anything obvious, and I don't have an idea of a physical or numerical reason this would matter.
I think we could do two things, one is to specify the default custom spectrum used at a higher resolution and see if the crash still occurs.  Or, since it's a bit non-nonsensical anyway to specify a custom spectrum and not list what that spectrum is, we could have it default to using the internally generated 1m/10s Bretschnieder spectrum in the case that "custom" is specified without a spectrum description.
In either case, we should not this as an issue/bug somewhere as it's sure to turn back up...

Well, I think the issue is that a user could specify a spectrum that crashes the sim. Is there a way we could detect unreasonable forces in the wave plugin and cap them with a warning that the spectrum doesn't have enough resolution or even error out and close the sim?

Yes, agreed.  I will do some more work on this to try and figure out what's going on.  Without that understanding, it's hard to check for...  Even if I could find a characterization of the spectrum that fails reliably, we could at least warn/prohibit that...  Or I suppose a pre-interpolation, but that doesn't make sense that would change anything...  Interpolate now, interpolate later...


Fix import in DiagnosisKeyRepository

Resolve #273

Consider  https://github.com/corona-warn-app/cwa-server/issues/273#issuecomment-633269311

Confirmed with @michael-burwig


How to grant a service account this role: roles/cloudkms.cryptoKeyDecrypter

I created a service account in my yaml:
- type: iam.v1.serviceAccount
  name: downloaders-sa
  properties:
    accountId: downloaders-sa
    displayName: "downloaders-sa"

And I'm trying to add the roles/cloudkms.cryptoKeyDecrypterto it. Until now I'm using a workaround to do it, by gcloud cmdline:
gcloud projects add-iam-policy-binding ${GOOGLE_CLOUD_PROJECT} \
    --member="serviceAccount:email" \
    --role=roles/cloudkms.cryptoKeyDecrypter

But, how can I do by DM? I tried using gcp-types, like https://github.com/GoogleCloudPlatform/deploymentmanager-samples/blob/58b4d5db56081f9a9c0dd595a9ef264e4f6389ae/google/resource-snippets/iam-v1/service_accounts.jinja#L21-L34
But this role specifically is not supported. How can I proceed?

Can you please send here the jinja template you are trying to use including the IAM binding?
I am not sure what do you mean with this role specifically is not supported.
Do you get an error for this?
   accessControl:
     gcpIamPolicy:
       bindings:
       - role: roles/cloudkms.cryptoKeyDecrypter
         members:
         - "serviceAccount:tarrito-{{ properties['accountIdSuffix'] }}@{{ projectName }}.iam.gserviceaccount.com"

I still believe this should work, however feel free to take a look at the Cloud Foundation Toolkit IAM Member binding template which supports IAM bindings on Project, Folder, Org level.

Was able to do following this snippet:
https://github.com/GoogleCloudPlatform/deploymentmanager-samples/blob/master/google/resource-snippets/cloudresourcemanager-v1/patch_iam_project.yaml

@ocsig when I say "this role", it's specifically roles/cloudkms.cryptoKeyDecrypter


Add NDTI to band_indicies and add it to water quality notebook

Proposed changes
Include a brief description of the changes being proposed, and why they are necessary.
Checklist (replace [ ] with [x] to check off)

[ ] Remove any unused Python packages from Load packages
[ ] Remove any unused/empty code cells
[ ] Remove any guidance cells (e.g. General advice)
[ ] Ensure that all code cells follow the PEP8 standard for code. The jupyterlab_code_formatter tool can be used to format code cells to a consistent style: select each code cell, then click Edit and then one of the Apply X Formatter options (YAPF or Black are recommended)
[ ] Include relevant tags in the first notebook cell and re-use tags if possible
[ ] Ensure appropriate colour schemes  have been used to maximise accessibility for vision impairment. Test your images or learn more with Coblis or TPGI
[ ] Clear all outputs, run notebook from start to finish, and save the notebook in the state where all cells have been sequentially evaluated

Closes issues (optional)

Closes Issue #000

@fangfy more text needed but wondering if the flow is correct:

Identify areas of likely water
Assess for turbidity
Plot/animate

The animation shows distinct fluctuations in the relative NDTI, including times where it looks very washed out, but I can't tell if that's due to seasonal/climate events or the data itself. This wetlands area has a big tributary, but the little side areas flip between turbid and not between the years

Nice one @eefaye! Agree with your flow and logic noted above for NDTI. Wondering if we can mask images to water using WOfS and plot NDTI in 1D through time, so interpretation doesn't just rely on the animation? That would help infer seasonality etc. I will give it a go today/tomorrow.

@fangfy more text needed but wondering if the flow is correct:

Identify areas of likely water
Assess for turbidity
Plot/animate

The animation shows distinct fluctuations in the relative NDTI, including times where it looks very washed out, but I can't tell if that's due to seasonal/climate events or the data itself. This wetlands area has a big tributary, but the little side areas flip between turbid and not between the years

Flow sounds good to me.

Thanks @mickwelli  your additions are great! Interesting thought about rainfall vs turbidity.
I've added the required markdown so @nanaboamah89 it's all yours now - welcome to approve, merge, and share.


change icon to package

It seems package is more accurate than archive
https://code.visualstudio.com/api/references/icons-in-labels#icon-listing

@swift-server-bot test

@swift-server-bot test this please

Just because the icon is called package doesn't make it correct. It probably looks better, but at the same time it has an annoying space before it as well. If we could get rid of that I'd be happy

Just because the icon is called package doesn't make it correct. It probably looks better, but at the same time it has an annoying space before it as well. If we could get rid of that I'd be happy

what do you mean annoying space

what do you mean annoying space

There is a space before the icon

it's not a space, I just screenshot them from https://code.visualstudio.com/api/references/icons-in-labels#icon-listing
I didn't consider the alignment

It looked like it added a space when I used. I may have been wrong


barPercentage & categoryPercentage do not put the bars right next to each other

In Chart JS documentation, barPercentage is said to put the bars right next to each other if set to 1.0 since it will take the whole category width.

However, I get this

I want no spacing/padding between the two bars. Note: this occurs only if categoryPercentage is equal to anything other than 1.0. I don't want to set categoryPercentage to 1.0 because  I don't want the category to take full width and I want the bars to be thin.
Live demonstration: http://codepen.io/anon/pen/KgoKyg?editors=0010

@KHKKhalid i tried to look at your codepen but it is infinitely refreshing the page :/

I have no clue on why this would happen. Here's the code anyway
var ctx = document.getElementById("myChart");
var chart = new Chart(ctx, {
        type: 'bar',
        data: {
            labels: ['X', 'Y'],
            datasets: [{
                label: 'Magnitude',
                data: [1, 2],
                backgroundColor: ['yellow', 'orange']
            }]
        },
        options: {
            scales: {
                yAxes: [{
                    ticks: {
                        beginAtZero: true,
                        fixedStepSize: 5,
                        max: 15,
                        min: 0
                    },
                    gridLines: {
                        color: 'black',
                        zeroLineColor: 'black',
                        zeroLineWidth: 4
                    }
                }],
                xAxes: [{
                    gridLines: {
                        display:false,
                        zeroLineWidth: 4,
                    },
                  barPercentage: 1.0,
                        categoryPercentage: 0.1
                }]
            }
        }
    });

Does this work for you http://codepen.io/khalidahmedshalabi/pen/mAxJmp

For anyone facing this issue, I figured out a workaround, set categoryPercentage and barPercentage to 1.0 (this will put bars next to each other), then keep pushing empty data to your graph (transparent color, 0 values) at the front and back (to make the bars centralized) of your graph until you reach the thickness you want.
Here's a snippet of my project that sorta illustrates how it works
var AddEmptyDataToChart = function ()
    {
        if(bars_shown < 4 * MAX_TABS)
        {
            var i;
            var data_count_on_each = Math.ceil(((4 * MAX_TABS) - bars_shown) / 2);

            // Add at the beginning and at the end
            for(i = 0; i < data_count_on_each; i ++)
            {
                // Beginning
                labels.push(" ");
                values.push(0);
                bgcol.push("rgba(0, 0, 0, 0.0)");
                bordercol.push("rgba(0, 0, 0, 0.0)");

                // End
                labels.unshift(" ");
                values.unshift(0);
                bgcol.unshift("rgba(0, 0, 0, 0.0)");
                bordercol.unshift("rgba(0, 0, 0, 0.0)");
            }
        }
    };

Same issue here.  Can't get the bars to go full width.
http://jsfiddle.net/beehe4eg/445/

I am facing the same issue in horizontal bar chart. Is there any fix for it?

@ennyta check the workaround I posted above.

It works as documented, you are just misunderstanding what a category is. 1 label = 1 category.
You can also split the data to individual datasets (so the bars can be in same category): https://codepen.io/kurkle/pen/oNgoNjR?editors=0010


Save exported GPG key to lksec after provision succeeds

r? @maxtaco
Fixes https://keybase.atlassian.net/browse/CORE-1976 and https://keybase.atlassian.net/browse/CORE-2146

:+1:


fix(tag-filter): prevent click propagation

Closes #1485

Can we still get clicks on the immediate element? If not we may need a @Output click/onClick to handle that usecase

It seems to 🤔  Unless I'm misunderstanding this seems to work


Percentiles

Is there a command to calculate percentiles?
I am currently using xray to create max, min and mean plots for each season for 40 years of data. I have tried replacing where I use 'max' with 'percentile' or 'quantile' and then change ('time') to ('time, 90) but there is no attribute 'percentile' or 'quantile'
ds = xray.open_mfdataset('myfiles///*temp.nc')
mytemp=ds['temp']
eachseasonmax=mytemp.groupby('time.season').max('time')
Also is this the correct place for these types of questions?
Thanks

Yes, this is an appropriate place to ask these questions.
No, we don't have a percentile/quantile method currently. This would be a welcome addition if you are interested in putting together a pull request.
For now, you can work around it by using the reduce method: ds.reduce(functools.partial(np.percentile, 95), dim='time')

Closing as a dupe of https://github.com/xray/xray/issues/303


Add Contact form Backend

Remove Google form for Contact us and add backend to send data to a real-time database on firebase

done on Google forms

done on Google forms


Unmarshalling array of polymorphic type in body parameter

Fixes #1637 (unmarshalling array of polymorphic type in body parameter)

NOTE: polymorphism still gets a number of uncovered edge cases.
This provides minimal patching for this particular case.

Again codecov report is out of this world... Removed 1 line => relative coverage decreases...


Client Allocation Request for: Santanu Das

Client Allocation Questions
Core Information

Name: Santanu Das
Website / Social Media: genrait.com
Region: North America
DataCap Requested: 5TiB
Addresses to be Notarized:
Notary Requested: andrewxhill

@d2as2 Please subscribe to notifications for this Issue to be aware of updates. Notaries may request additional information on the Issue.

Hi @d2as2, so I can evaluate your application, could you answer the following questions for me.

Can you introduce yourself? Where can I find you online? Examples include Github, Twitter, LinkedIn, WeChat, etc., and more than one is encouraged.
If you represent an organization, what is the name, nature, registration country, time since inception, and named officers of your organization?
In which region(s) do you intend to use the DataCap?
What is your use case for the DataCap being requested?
How many miners do you plan on allocating the DataCap? Please specify if you have a specific miner(s) you would like to allocate the DataCap.
Do you agree on providing transparency of DataCap allocation decisions & execution details in public to prevent excessive/unfair self-dealing? If so, how will you plan to report it?
Will your organization be responsible for allocating the full DataCap? Examples where this might not be the case, include app builders where the application end-users may have access to storage backed by the DataCap. Please read our full proposal for details on this approach. Unfortunately, our proposed multisig enabled DataCap provider is not ready today, but when it is, we will provider further documentation.

thanks!

This user’s identity has been verified through filplus.storage


Just think 💭


Update summary.md

Added a link

@EdPrice-MSFT : Thanks for your contribution! The author(s) have been notified to review your proposed change.

#sign-off


Fixes for global display sample

Contents of BingLocation.ts were not exported from imodeljs-frontend.
GlobeAnimator constructor and other methods were private.

@Mergifyio backport master


android bugs

[ ] ETA sometimes looks weird (hard to reproduce)
[x] terminal_size does not compile

Let's close this. I'm using an other lib for terminal size.
Weird ETA is too hard to reproduce.


Run unit tests in isolated processes

Motivation
Use different JVM forks to execute the unit tests. This prevent resource leaks from some tests (which are very difficult to track down) to lead to failures in subsequent tests.
Next step would be to run multiple forks in parallel, to speed up the execution.

When I saw the travis output - I didn't see INFO logs being printed for pulsar-broker (besides the testng normal INFO messages) - so am not sure if the log4j.properties change is working.

By specifying <redirectTestOutputToFile>true</redirectTestOutputToFile>, maven sends the test output to individual files in target/surefire-reports. In travis we cannot see that anyway, but in Jenkins (when we switch) or in local tests, you can see the logs.

I'm still testing with this PR. I have the changes to run in parallel. There are just a couple of tests from ModularLoadManager that are failing and looking into it.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/pulsar-pull-request/14/

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/pulsar-pull-request/16/

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/pulsar-pull-request/17/

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/pulsar-pull-request/18/

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/pulsar-pull-request/19/

is there any way we can prevent asfgit comment on every PR?

I'm really trying to find it out...
You can check the config of the job here: https://builds.apache.org/job/pulsar-pull-request/configure
Also, there's docs here: https://wiki.jenkins.io/display/JENKINS/GitHub+pull+request+builder+plugin
So far I haven't found. I've disabled the build for now.

I think I don't have permission to check config for job
Error-Message:  rdhabalia is missing the Job/ExtendedRead permission

Error-Message: rdhabalia is missing the Job/ExtendedRead permission

Uhm I don't know how to add more people to Jenkins..
Though I think I should have fixed the commenting part :)

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/pulsar-pull-request/21/

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/pulsar-pull-request/22/

Already changed in #769


Support a variety format for Eddystone-URL.

When I tried with Eddystone beacons made by kontakt.io, it crashed, because [[BRMEddystoneURLBeacon alloc] initWithAdvertiseData:advertiseData] returns nil and it causes inserting nil into array.
The kontakt.io beacons seems not to advertise the last 2 "RFU" bytes, so I changed the data size condition.

Thanks for the fix.


Package.getActivationCommands is deprecated.

Use activationCommands instead of activationEvents in your package.json
Commands should be grouped by selector as follows:
  "activationCommands": {
    "atom-workspace": ["foo:bar", "foo:baz"],
    "atom-text-editor": ["foo:quux"]
  }

Package.getActivationCommands (/usr/local/share/atom/resources/app.asar/src/package.js:808:9)
Package.hasActivationCommands (/usr/local/share/atom/resources/app.asar/src/package.js:733:20)
<unknown> (/usr/local/share/atom/resources/app.asar/src/package.js:185:24)
Package.measure (/usr/local/share/atom/resources/app.asar/src/package.js:163:15)
Package.load (/usr/local/share/atom/resources/app.asar/src/package.js:177:12)
PackageManager.loadPackage (/usr/local/share/atom/resources/app.asar/src/package-manager.js:355:14)

Done ;]


[Feature request] Skew text

will be helpful for 3D diagrams

Also requesting this feature.

Since this was closed by the bot, should I open another issue? In any case, I too support this feature request of skewing, at least for text.

We can open it, but it'll never be done, so kinda misleading.

We're not going to implement this.


Errors when attempting to use ferc1_to_sqlite

When attempting to run the ferc1_to_sqlite script processing all tables for years 2020 and 2021 (so, 1 year each from the two FERC data sources) I get a bunch of errors about files not existing (but looking in the cache directory, they do seem to exist):
2022-07-07 17:53:38 [    INFO] catalystcoop.ferc_xbrl_extractor.xbrl:155 Parsing taxonomy from https://eCollection.ferc.gov/taxonomy/form1/2022-01-01/form/form1/form-1_2022-01-01.xsd
2022-07-07 17:53:39 [    INFO] catalystcoop.ferc_xbrl_extractor.xbrl:155 Parsing taxonomy from https://eCollection.ferc.gov/taxonomy/form1/2022-01-01/form/form1/form-1_2022-01-01.xsd
2022-07-07 17:53:39 [    INFO] catalystcoop.ferc_xbrl_extractor.xbrl:155 Parsing taxonomy from https://eCollection.ferc.gov/taxonomy/form1/2022-01-01/form/form1/form-1_2022-01-01.xsd
2022-07-07 17:53:39 [    INFO] catalystcoop.ferc_xbrl_extractor.xbrl:155 Parsing taxonomy from https://eCollection.ferc.gov/taxonomy/form1/2022-01-01/form/form1/form-1_2022-01-01.xsd
2022-07-07 17:53:42,621 [webCache:cacheDownloadRenamingError] [Errno 2] No such file or directory: '/home/zane/.config/arelle/cache/https/eCollection.ferc.gov/taxonomy/form1/2022-01-01/schedules/ScheduleAccumulatedDeferredIncomeTaxes/sched-234_2022-01-01.xsd.tmp' -> '/home/zane/.config/arelle/cache/https/eCollection.ferc.gov/taxonomy/form1/2022-01-01/schedules/ScheduleAccumulatedDeferredIncomeTaxes/sched-234_2022-01-01.xsd'
Unsuccessful renaming of downloaded file to active file /home/zane/.config/arelle/cache/https/eCollection.ferc.gov/taxonomy/form1/2022-01-01/schedules/ScheduleAccumulatedDeferredIncomeTaxes/sched-234_2022-01-01.xsd
Please remove with file manager. -

2022-07-07 17:53:42,624 [IOerror] sched-234_2022-01-01.xsd: file error: [Errno 2] No such file or directory: '/home/zane/.config/arelle/cache/https/eCollection.ferc.gov/taxonomy/form1/2022-01-01/schedules/ScheduleAccumulatedDeferredIncomeTaxes/sched-234_2022-01-01.xsd' - https://eCollection.ferc.gov/taxonomy/form1/2022-01-01/form/form1/form-1_2022-01-01.xsd 2

2022-07-07 17:53:54,479 [webCache:cacheDownloadRenamingError] [Errno 2] No such file or directory: '/home/zane/.config/arelle/cache/https/eCollection.ferc.gov/taxonomy/form1/2022-01-01/ferc-core_2022-01-01.xsd.tmp' -> '/home/zane/.config/arelle/cache/https/eCollection.ferc.gov/taxonomy/form1/2022-01-01/ferc-core_2022-01-01.xsd'
Unsuccessful renaming of downloaded file to active file /home/zane/.config/arelle/cache/https/eCollection.ferc.gov/taxonomy/form1/2022-01-01/ferc-core_2022-01-01.xsd
Please remove with file manager. -

I notice there's some weird case-sensitive stuff going on in my arelle cache directory, with the version from the last time I played around in April not having any uppercase in the domain name / directory:
drwxr-xr-x 3 zane zane 4.0K 2022-04-12 21:51 ecollection.ferc.gov/
drwxr-xr-x 3 zane zane 4.0K 2022-07-07 17:53 eCollection.ferc.gov/

There are also some syntax errors coming up...
2022-07-07 17:58:45,875 [xmlSchema:syntax] Char 0x0 out of allowed range, sched-304_2022-01-01_def.xml, line 223, column 128 - ../../../../../../../sched-304_2022-01-01_def.xml , https://eCollection.ferc.gov/taxonomy/form1/2022-01-01/schedules/ScheduleSalesOfElectricityByRateSchedules/sched-304_2022-01-01.xsd 5
2022-07-07 17:58:45,875 [xmlSchema:syntax] AttValue: ' expected, sched-304_2022-01-01_def.xml, line 223, column 128 - ../../../../../../../sched-304_2022-01-01_def.xml , https://eCollection.ferc.gov/taxonomy/form1/2022-01-01/schedules/ScheduleSalesOfElectricityByRateSchedules/sched-304_2022-01-01.xsd 5
2022-07-07 17:58:45,875 [xmlSchema:syntax] attributes construct error, sched-304_2022-01-01_def.xml, line 223, column 128 - ../../../../../../../sched-304_2022-01-01_def.xml , https://eCollection.ferc.gov/taxonomy/form1/2022-01-01/schedules/ScheduleSalesOfElectricityByRateSchedules/sched-304_2022-01-01.xsd 5
2022-07-07 17:58:45,875 [xmlSchema:syntax] Couldn't find end of Start Tag loc line 223, sched-304_2022-01-01_def.xml, line 223, column 128 - ../../../../../../../sched-304_2022-01-01_def.xml , https://eCollection.ferc.gov/taxonomy/form1/2022-01-01/schedules/ScheduleSalesOfElectricityByRateSchedules/sched-304_2022-01-01.xsd 5
2022-07-07 17:58:45,875 [xmlSchema:syntax] Premature end of data in tag definitionLink line 17, sched-304_2022-01-01_def.xml, line 223, column 128 - ../../../../../../../sched-304_2022-01-01_def.xml , https://eCollection.ferc.gov/taxonomy/form1/2022-01-01/schedules/ScheduleSalesOfElectricityByRateSchedules/sched-304_2022-01-01.xsd 5

Despite the errors above, it seems to be attempting to extract some filings. However in the end I get an error from concurrent.futures:
2022-07-07 18:02:37 [    INFO] catalystcoop.ferc_xbrl_extractor.xbrl:126 Extracting a6879fc6-64be-47e6-a830-0b9b4aaa24eb
2022-07-07 18:02:39 [    INFO] catalystcoop.ferc_xbrl_extractor.xbrl:126 Extracting 90b8961f-01d5-426c-b9e2-2c7b39884ced
2022-07-07 18:02:41 [    INFO] catalystcoop.ferc_xbrl_extractor.xbrl:126 Extracting f863e3cd-6ca9-42a9-b013-ca12fd301d2a
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/home/zane/mambaforge/envs/pudl-dev/lib/python3.10/concurrent/futures/process.py", line 246, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/home/zane/mambaforge/envs/pudl-dev/lib/python3.10/concurrent/futures/process.py", line 205, in _process_chunk
    return [fn(*args) for args in chunk]
  File "/home/zane/mambaforge/envs/pudl-dev/lib/python3.10/concurrent/futures/process.py", line 205, in <listcomp>
    return [fn(*args) for args in chunk]
  File "/home/zane/mambaforge/envs/pudl-dev/lib/python3.10/site-packages/ferc_xbrl_extractor/xbrl.py", line 93, in process_batch
    instance_dfs = process_instance(instance, db_path, taxonomy, save_metadata)
  File "/home/zane/mambaforge/envs/pudl-dev/lib/python3.10/site-packages/ferc_xbrl_extractor/xbrl.py", line 124, in process_instance
    tables = get_fact_tables(taxonomy, db_path, save_metadata)
  File "/home/zane/mambaforge/envs/pudl-dev/lib/python3.10/site-packages/ferc_xbrl_extractor/xbrl.py", line 156, in get_fact_tables
    taxonomy = Taxonomy.from_path(taxonomy_path, save_metadata)
  File "/home/zane/mambaforge/envs/pudl-dev/lib/python3.10/site-packages/ferc_xbrl_extractor/taxonomy.py", line 175, in from_path
    roles = [
  File "/home/zane/mambaforge/envs/pudl-dev/lib/python3.10/site-packages/ferc_xbrl_extractor/taxonomy.py", line 176, in <listcomp>
    LinkRole.from_list(role, concept_dict) for role in view.jsonObject["roles"]
  File "/home/zane/mambaforge/envs/pudl-dev/lib/python3.10/site-packages/ferc_xbrl_extractor/taxonomy.py", line 147, in from_list
    concepts=Concept.from_list(linkrole_list[3], concept_dict),
IndexError: list index out of range
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/zane/mambaforge/envs/pudl-dev/bin/ferc1_to_sqlite", line 33, in <module>
    sys.exit(load_entry_point('catalystcoop.pudl', 'console_scripts', 'ferc1_to_sqlite')())
  File "/home/zane/code/catalyst/pudl/src/pudl/convert/ferc1_to_sqlite.py", line 113, in main
    pudl.extract.ferc1.xbrl2sqlite(
  File "/home/zane/code/catalyst/pudl/src/pudl/extract/ferc1.py", line 618, in xbrl2sqlite
    xbrl.extract(
  File "/home/zane/mambaforge/envs/pudl-dev/lib/python3.10/site-packages/ferc_xbrl_extractor/xbrl.py", line 61, in extract
    for i, batch in enumerate(results):
  File "/home/zane/mambaforge/envs/pudl-dev/lib/python3.10/concurrent/futures/process.py", line 570, in _chain_from_iterable_of_lists
    for element in iterable:
  File "/home/zane/mambaforge/envs/pudl-dev/lib/python3.10/concurrent/futures/_base.py", line 609, in result_iterator
    yield fs.pop().result()
  File "/home/zane/mambaforge/envs/pudl-dev/lib/python3.10/concurrent/futures/_base.py", line 446, in result
    return self.__get_result()
  File "/home/zane/mambaforge/envs/pudl-dev/lib/python3.10/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
IndexError: list index out of range
[1]    50211 exit 1     ferc1_to_sqlite --clobber src/pudl/package_data/settings/etl_full.yml

I tried re-running this with all of the years of FERC data and it still errored out, but after only about half as many successfully processed filings, so it seems kind of stochastic.

Welp, I merged dev into xbrl_integration and xbrl_integration into xbrl_steam in the PUDL repo, and now the error messages seem to have gone (because of the new logging setup I think), and also the XBRL to SQLite conversion seems to have worked. Not sure what changed in there but I guess it's all okay now and I can continue trying to work with @cmgosnell's new transform functions / classes.

I did notice that in the old version, it wasn't really attempting to parallelize the XBRL processing, whereas after the merge it saturated all my CPUs, so I guess there was some change in how the parallel processing was working and it fixed the concurrent.futures issue whatever it was.

Hmmm I ran into a lot of similar errors:
2022-07-08 10:22:02,027 [IOerror] sched-234_2022-01-01_def.xml: file error: [Errno 2] No such file or directory: '/Users/apple/Library/Caches/Arelle/https/eCollection.ferc.gov/taxonomy/form1/2022-01-01/schedules/ScheduleAccumulatedDeferre
dIncomeTaxes/sched-234_2022-01-01_def.xml' - https://eCollection.ferc.gov/taxonomy/form1/2022-01-01/schedules/ScheduleAccumulatedDeferredIncomeTaxes/sched-234_2022-01-01.xsd 4

I haven't encountered these Arelle cache issues before, but I'll investigate and see if I can figure out what the cause is.
Regarding the parallelization, initially ferc1_to_sqlite was not properly initializing some parameters so it was only using one cpu. I'm guessing Christina branched off before that change, so that makes sense it was behaving that way.

After quite a bit of investigation, I haven't been able to consistently find a way to recreate this issue. I do have a few theories as to what might be happening, and why I hope this will not continue.
First, I was previously parsing the taxonomy from each worker process individually. I think it is possible that this was causing a conflict in the cache that Arelle creates, which led to the error. Now I am only parsing the taxonomy once before spawning the worker sub-processes and passing the data structures to each worker.
My second theory involves some strange dependency issue that I haven't been able to track down. Evidence for this theory comes from the fact that after reproducing this issue, I was able to fix it simply by removing and recreating my conda environment. Since then, I have created a mirror of Arelle and packaged it on pypi to improve consistency with dependencies (Arelle does not provide an official package on pypi, so before this it was hard to guarantee consistency).
I'm going to close this issue for now, and will reopen if this problem comes up again.

Would you be up for getting Arelle packaged on conda-forge as well?  That's crazy they don't have any official packaged versioned releases.


Include exported modules with completions

Not sure if this is a duplicate of #41, if it is then please close this.
I noticed that some completion candidates are not being included in the list when they are exposed as modules.  For example here:
https://github.com/slamdata/purescript-halogen/blob/50c06d684d34592fc19007bb77a618ca9a847f8b/src/Halogen/HTML/Events/Indexed.purs
The module ExposedEvents at the bottom includes the items input and input_ and don't come over in the completion list.

The current resolution algorithm for reexports(Reexports.hs) does only support exporting a module by it's original name.
It's a little unfortunate that we have to reproduce the entire reexport spec inside psc-ide. But it's absolutely doable.

Yea I agree and I'm not even sure how prevalent that feature is used.  It just caught me off guard since I was testing completions on a module that happened to use it.

There is more pathological cases... You can for example do this:

module Lol where
(
module C
)

import qualified A as C
import qualified B as C

And have it export both A and B...

can we call this fixed? We are not yet tracking selective Reexports but I think that's for another day... :D

Yea it alleviates the main pain I was having.  I'm not too concerned about showing some extra items in the completion list at this point either.  Thanks for the suggestions with the PR also!

Thank you for fixing it in the first place! :+1:


Production - make highly available, and set up autoscaling

Production should be highly available (i.e. at least two Fargate tasks, one in each availability zone). We should should also scale up/down depending on certain metrics (i.e. high CPU or memory usage, or high amount of traffic).

Set desired count to two, and set up auto-scaling based on CPU utilization.


Unable to use keyboard shortcuts

hi, first of all I'm on NixOS and using cinnamon DE(X11)
I've successfully been able to set up keyd and I can confirm this because remapping capslock = q works as expected whoever I've tried the following:
[control+shift]
p = command(firefox)
m = command(firefox)
w = command(wezterm)
and it does not work no matter the variations i try
my intention is to make Control+Shift+p open Firefox the other keybinds are just to see if it's a problem with p, keyd successfully registers key presses when using "sudo keyd monitor" (sudo is necessary for some reason) I am unaware if it's a compatibility problem with cinnamon or if I just miss used the syntax

sudo is necessary for some reason

Intercepting and generating keypresses is inherently a privileged action. You wouldn't want all your programs to be able to do that.
That is also the reason why using command() is discouraged for arbitrary applications, especially something like a web browser. You may be able to get it to work by changing the user and setting the appropriate DISPLAY variable, but for hotkeys you're much better off using your WM's config or something like <https//github.com/baskerville/sxhkd>.
cf #692, #593, https://github.com/rvaiya/keyd/issues/211, https://github.com/rvaiya/keyd/issues/229, https://github.com/rvaiya/keyd/issues/243


REST Timestamp with additional fraction not accepted

In the In-coordiator (DropWizard based) REST API, the timestamp of below format is accepted and the same fails in the Quarkus based REST API
2023-07-13T14:02:06.018850Z

Steps to reproduce:

Create table with a timestamp column

/v2/schemas/keyspaces/testks1/tables
{
    "name": "employee1",
    "ifNotExists": false,
    "columnDefinitions": [
        {
            "name": "id",
            "typeDefinition": "int"
        },
        {
            "name": "name",
            "typeDefinition": "text"
        },
        {
            "name": "createdAt",
            "typeDefinition": "timestamp"
        }
    ],
    "primaryKey": {
        "partitionKey": [
            "id"
        ]
    },
    "tableOptions": {
        "defaultTimeToLive": 0
    }
}

Insert a record

/v2/keyspaces/testks1/employee1
{
    "id": 123,
    "name": "emp2",
    "createdAt": "2023-07-13T14:02:06.018850Z"
}

In DropWizard REST API, it is working without any error and while getting back the row, we could see the last three 3 digits got truncated and returned as below
/v2/keyspaces/testks1/employee1/123
{
    "count": 1,
    "data": [
        {
            "name": "emp2",
            "createdAt": "2023-07-13T14:02:06.018Z",
            "id": 123
        }
    ]
}

In Quarkus based REST API, the request to insert fails with below error
{
    "description": "Invalid path for row to create, problem: Failed to deserialize field 'employee1.createdAt', problem: Text '2023-07-13T14:02:06.018850Z' could not be parsed, unparsed text found at index 23",
    "code": 400,
    "grpcStatus": null,
    "internalTxId": null
}

I suspect this is related to fix for #2223 (via PR #2241): probably need to figure out DateTimeFormat that allows more fraction digits.


chore(master): release 0.0.1

:robot: I have created a release beep boop
0.0.1 (2022-11-08)
Features

Initial release (#1) (3037efb)

Miscellaneous Chores

release 0.0.1 (#3) (193cac2)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/jtwatson/logger/releases/tag/v0.0.1 :sunflower:


Discourage/prohibit inheriting from package sets in callPackage

in all-packages.nix when something is inherited (ex, {inherit (pkgs.darwin.apple_sdk.frameworks) Security}) in a callPackage invocation the inherited attr will not have __spliced and so won't be cross-compilation compatible.
one way to fix this would be to change the with pkgs; in all-packages to with pkgs.__splicedPackages like tried in #58327
however that will result in other issues like infinite recursion.
$ nix build ".#pkgsCross.aarch64-multiplatform.bash"
error: infinite recursion encountered

other way to work around it would be to to inherit from the __splicedPackages set (ex, {inherit (pkgs.__splicedPackages.darwin.apple_sdk.frameworks) Security})
the way i prefer would be to just not inherit from package sets
Demo: show-splices.nix
let
  pkgs = import ./. { };
  pkgsCross = pkgs.pkgsCross.aarch64-multiplatform;
  workingSplices = pkgsCross.callPackage
    ({ darwin, xorg }: {
      d = darwin.apple_sdk.frameworks.Security.__spliced;
      x = xorg.libX11.__spliced;
    })
    { };

  notWorkingSplices = pkgsCross.callPackage
    ({ Security, libX11 }: {
      d = Security.__spliced;
      x = libX11.__spliced;
    })
    {
      inherit (pkgsCross.darwin.apple_sdk.frameworks) Security;
      inherit (pkgsCross.xorg) libX11;
    };
in
{
  # for --json to work
  workingSplices =
    removeAttrs workingSplices [ "override" "overrideDerivation" ];
  notWorkingSplices =
    removeAttrs notWorkingSplices [ "override" "overrideDerivation" ];
}

working:
$ NIXPKGS_ALLOW_UNSUPPORTED_SYSTEM=1 nix eval -f show-splices.nix workingSplices --json | jq
{
  "d": {
    "buildBuild": "/nix/store/ls83bixygn2bzqq1w2z66dyccfn61wic-apple-framework-Security",
    "buildHost": "/nix/store/ls83bixygn2bzqq1w2z66dyccfn61wic-apple-framework-Security",
    "buildTarget": "/nix/store/ls83bixygn2bzqq1w2z66dyccfn61wic-apple-framework-Security",
    "hostHost": "/nix/store/vhqjns1zfab6cnv2lbkr6gff1fa8n3kw-apple-framework-Security-11.0.0",
    "hostTarget": "/nix/store/vhqjns1zfab6cnv2lbkr6gff1fa8n3kw-apple-framework-Security-11.0.0"
  },
  "x": {
    "buildBuild": "/nix/store/w3zzhfl4a7xp0xfflz2gawv02y8ba9z8-libX11-1.8.1",
    "buildHost": "/nix/store/w3zzhfl4a7xp0xfflz2gawv02y8ba9z8-libX11-1.8.1",
    "buildTarget": "/nix/store/w3zzhfl4a7xp0xfflz2gawv02y8ba9z8-libX11-1.8.1",
    "hostHost": "/nix/store/gjh7n6myi7miwprjkk4lll8cp1j7kmz8-libX11-aarch64-unknown-linux-gnu-1.8.1",
    "hostTarget": "/nix/store/gjh7n6myi7miwprjkk4lll8cp1j7kmz8-libX11-aarch64-unknown-linux-gnu-1.8.1"
  }
}

broken:
}
$ NIXPKGS_ALLOW_UNSUPPORTED_SYSTEM=1 nix eval -f show-splices.nix notWorkingSplices --json | jq
error: attribute '__spliced' missing

       at /home/artturin/nixgits/my-nixpkgs/show-splices.nix:5:74:

            4|   workingSplices = pkgsCross.callPackage ({ darwin, xorg }: { d = darwin.apple_sdk.frameworks.Security.__spliced; x = xorg.libX11.__spliced;}) { };
            5|   notWorkingSplices = pkgsCross.callPackage ({ Security, libX11 }: { d = Security.__spliced; x = libX11.__spliced;}) { inherit (pkgsCross.darwin.apple_sdk.frameworks) Security; inherit (pkgsCross.xorg) libX11; };
             |                                                                          ^
            6| in
{
  "d": null
}

related issues:
https://github.com/NixOS/nixpkgs/issues/49526

@Artturin , please, can you (or someone) explain what is the matter around the splicing?
I have seen today some code on XFCE mega-attrset, and I am confused.
Also, I have some packages that use Darwin libs, so I will start to cleanup them.

@Artturin , please, can you (or someone) explain what is the matter around the splicing?

splicing adds the __spliced attribute set to the package
mkDerivation then picks __spliced.buildHost for nativeBuildInputs https://github.com/NixOS/nixpkgs/blob/86c83ab79d71084b4b107adf4c3838b7cbcc6446/pkgs/stdenv/generic/make-derivation.nix#L212
and __spliced.hostTarget for buildInputs https://github.com/NixOS/nixpkgs/blob/86c83ab79d71084b4b107adf4c3838b7cbcc6446/pkgs/stdenv/generic/make-derivation.nix#L212

I think it is not possible to get rid of it entirely, especially if you consider the API benefit of being able to override individual inputs which is certainly an intention in some of the cases. Also, in sub package sets the solution you propose in #195874 won't work, of course.

There's some magic going on with __spliced that's not obvious to me. Given that, it sounds hard to teach to contributors, so how do we prohibit this pattern?

I concur that, ideally, we do not want to pass sets of packages to packages. Same reason we don't just pass pkgs everywhere and call it a day.
Why are the Darwin frameworks hidden away in a subset anyways? If there's no good reason for it, we might want to consider just adding them to the top-level.
Another thing we could consider is a special attribute where we can simply pass attribute sets from which attrs should be considered as if they were at the top-level but at that point, we might as well use the top-level in the first place?

Why are the Darwin frameworks hidden away in a subset anyways?

I don't believe they are hidden, properly speaking. Anyone can access them by the cumbersome inherit mechanism.
The problem to me is the magic behind them and that huge amount of if isDarwin then [ Security OpenGL Metal ... ]

If there's no good reason for it, we might want to consider just adding them to the top-level.

This is a bit personal and preciousness-y, but I do not like the currently huge top-level file and its propensity to scramble the asciibetical ordering.

stuff like this should be discouraged/prohibited too
  keybinder = callPackage ../development/libraries/keybinder {
    automake = automake111x;
    lua = lua5_1;
  };

nix-repl> pkgsCross.aarch64-multiplatform.keybinder.nativeBuildInputs
[ ... «derivation /nix/store/yj8vj3gbd78scrj4xjmdx4rm5p87dyb9-automake-aarch64-unknown-linux-gnu-1.11.6.drv» ]

if it has to be done then it should be
  keybinder = callPackage ../development/libraries/keybinder {
    automake = __splicedPackages.automake111x;
    lua = __splicedPackages.lua5_1;
  };

nix-repl> pkgsCross.aarch64-multiplatform.keybinder.nativeBuildInputs
[ ... «derivation /nix/store/c9g2kiyrwv9hq33wa3ckrkz95y4dq1qh-automake-1.11.6.drv» ]

can we fix that by changing (all-packages.nix L13)
with pkgs;

to
with pkgs.__splicedPackages;

?

i linked https://github.com/NixOS/nixpkgs/pull/58327 in the OP
a thing we could try is reorganize all-packages.nix so we don't get infinite recursion
quick POC
diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix
index 5e0e798be20..765ce0b92c0 100644
--- a/pkgs/top-level/all-packages.nix
+++ b/pkgs/top-level/all-packages.nix
@@ -8,7 +8,7 @@
 { lib, noSysDirs, config, overlays }:
 res: pkgs: super:

-with pkgs;
+(with pkgs;

 {
   # A module system style type tag
@@ -15618,6 +15618,9 @@ with pkgs;

   yaml-language-server = nodePackages.yaml-language-server;

+}) // (with pkgs.__splicedPackages; {
+
+
   # prolog
   yap = callPackage ../development/compilers/yap { };

@@ -38170,4 +38173,4 @@ with pkgs;
   aitrack = libsForQt5.callPackage ../applications/misc/aitrack { };

   widevine-cdm = callPackage ../applications/networking/browsers/misc/widevine-cdm.nix { };
-}
+})

https://github.com/NixOS/nixpkgs/pull/204370

I don't believe they are hidden, properly speaking. Anyone can access them by the cumbersome inherit mechanism.

That's what I mean.

I do not like the currently huge top-level file and its propensity to scramble the asciibetical ordering.

Note that I said "top-level", not "top-level file". I don't care what file they're in but I do want them to be accessible from pkgs so that the auto-discovery mechanism, splicing and all the other quirks of callPackage's "just work" with them.
Alternatively, an extra attr that automagically extends the main package set for the purpose of discovering dependencies for that specific package would also be fine in my eyes. i.e.
  bar = callPackage ../foo/bar {
    extraPackageSets = [ darwin xorg ];
  }

{
  stdenv,
  libX11,
  Security,
  ...
}

...

stuff like this should be discouraged/prohibited too
  keybinder = callPackage ../development/libraries/keybinder {
    automake = automake111x;
    lua = lua5_1;
  };

In cases like this, I prefer to simply reference automake111x and lua5_1 explicitly in the inputs of the package's function. I.e.
{
  lib,
  stdenv,
  automake111x,
  lua5_1,
  ...
}:

let
  lua = lua5_1;
  automake = automake111x;
in

stdenv.mkDerivation {
  nativeBuildInputs = [ automake lua ];
}

(Or just in-line without the let.)
IMO, we should have as little logic as possible in all-packages.

It's becoming confusing...

The most ergonomic will always be to not have package sets and have everything under the top-level as I initially suggested.
My alternative suggestion is just something that came to my mind because we might need something like that anyways because we do have huge package sets whose packages other packages depend on and, unlike the smaller ones we've been discussing so far, we probably don't want to dump those into the top-level.
If you can think of a more ergonomic solution to that problem, I'm all ears.

Taking the idea to a ridiculous extreme, all the expressions organized in subdirectories can be "concatenated" in a huge 4GB single file, sqlite-amalgam style.
Indeed, I have refactored some Elisp packages that were written directly onto manual-packages.nix some time ago.
Thinking a bit about it, this makes few to no sense. We have custom code to shelter platforms like FreeBSD and PowerPC, it would be insane to make all of it accessible to the top-level.

Why are the Darwin frameworks hidden away in a subset anyways?

The frameworks come from different sdks, so sometimes we want the ones from apple_sdk, sometimes apple_sdk_11_0 (and there will be more).

(On a second read... did you ask why the sdks are under Darwin, not top level?)

In a certain sense, yes.

a issue with that is the use of // so we won't get errors about duplicate attrs and the RHS will override the LHS

@Artturin - I'm a bit late in responding to this, but don't we have lib.attrsets.unionOfDisjoint for this reason (which throws when the sets intersect) if we wanted to go the route of that POC you posted above?

To me this looks more like a deficiency in the way cross-compilation has been implemented (splicing). Is there really no alternative that would work with inheriting attributes? Because what you say should be discouraged/prohibited seems pretty useful.

To me this looks more like a deficiency in the way cross-compilation has been implemented (splicing). Is there really no alternative that would work with inheriting attributes? Because what you say should be discouraged/prohibited seems pretty useful.

In all-packages where with pkgs is used something like https://github.com/NixOS/nixpkgs/issues/204303#issuecomment-1336269127 would fix it there
Everywhere else we just need to use callPackage like we already are

a issue with that is the use of // so we won't get errors about duplicate attrs and the RHS will override the LHS

https://nix.dev/anti-patterns/language#attr1-attr2-merge-operator

Another idea I've had here would be to consequently use callPackage wrappers. This is already being done to some degree by i.e. the QT packages but I see no reason why we couldn't expand this pattern to Darwin packages, xorg packages and everything else:
  bar = callPackage ../foo/bar {
    inherit (darwin.apple_sdk.frameworks) AppKit;
  }

->
  bar = darwin.apple_sdk.callPackage ../foo/bar { };

The example package here would reference AppKit in its args and darwin.callPackage would do callPackage but add the (properly spliced) darwin frameworks.
We could generalise this callPackage wrapper creation to allow for ad-hoc building of custom wrappers for more complicated packages which depend on multiple package sets:
  bar = callPackageWith [ darwin.apple_sdk.frameworks xorg ] ../foo/bar { };

and then the derivation could reference both Darwin frameworks and xorg packages in its args.
What do you think?

  bar = darwin.apple_sdk.callPackage ../foo/bar { };

I believe this is, strong words follow, madness. Why should I call a specific function hidden inside the Darwin subset in order to build non-Darwin stuff?
Yes, "this function will delegate to normal callPackage when building nonDarwin", but still it is not something well enginneered. This is hacky.

  bar = callPackageWith [ darwin.apple_sdk.frameworks xorg ] ../foo/bar { };

and then the derivation could reference both Darwin frameworks and xorg packages in its args.
What do you think?

Way better! +1 for this.

Why should I call a specific function hidden inside the Darwin subset in order to build non-Darwin stuff?

We already do that for a lot of things that require apple sdk 11.0 but are not specific to darwin, search darwin.apple_sdk_11_0.callPackage

bar = callPackageWith [ darwin.apple_sdk.frameworks xorg ] ../foo/bar { };

How do we know wait the attribute names of the packages are supposed to be, and how does it differ from the current callPackage exactly?

Yes, "this function will delegate to normal callPackage when building nonDarwin", but still it is not something well enginneered. This is hacky.

Well, darwin.apple_sdk.callPackage would be defined as callPackageWith [ darwin.apple_sdk.frameworks ]. It'd just be a handy shorthand to not have to type that out every time because calling a package with darwin frameworks is a very common occurrence.

How do we know wait the attribute names of the packages are supposed to be

I'm not sure what you mean? The attrs given to the function are the sets of packages on whose attrs you can depend in the derivation.
You depend on the individual packages just like before.

and how does it differ from the current callPackage exactly?

It actually doesn't differ much; callPackage is defined in such a manner already. These functions I mentioned mostly already exists actually (under different names), we just need to wire them up.

How do we know wait the attribute names of the packages are supposed to be

I'm not sure what you mean? The attrs given to the function are the sets of packages on whose attrs you can depend in the derivation.

The sets will have to be passed as strings to we can get them from __splicedPackages

We already do that for a lot of things that require apple sdk 11.0 but are not specific to darwin, search darwin.apple_sdk_11_0.callPackage

And this is the problem I am pointing out.

How do we know wait the attribute names of the packages are supposed to be, and how does it differ from the current callPackage exactly?

How do I know what callPackage to use, after all?
The one exposed in the top-level?
The one buried on Darwin toolchain - after two indirections?
Or, in the foreseeable future when Nix/Nixpkgs tools cover other OSes, the one buried on BSD, after God knows how much indirection?

Yes, "this function will delegate to normal callPackage when building nonDarwin", but still it is not something well enginneered. This is hacky.

  bar = callPackageWith [ darwin.apple_sdk.frameworks xorg ] ../foo/bar { };

and then the derivation could reference both Darwin frameworks and xorg packages in its args.
What do you think?

Way better! +1 for this.

POC https://github.com/NixOS/nixpkgs/pull/210970

I made splicing (nativeDrv and crossDrv are older, but were done differently). Splicing is bad. I never wanted it. I causes major headaches when getting anywhere near bootstrapping. It is gross. It is probably slow. It should be removed.

While I don't have a better solution to propose now, just wanted to mention that using both a different apple_sdk version and using Qt libs does not have an obvious example (you can list them explicitly in the set after callPackage, but I'm likely not the only person who wondered "how do I do the equivalent of qt5.callPackage and darwin.apple_sdk_11_0.callPackage at the same time". (without reading the implementation and understanding the internals) callPackageWith would make it more explicit. (there are more combinations with other namespaces too)
So, a more composable / approachable way would be great.

It may be just because I don't know what I'm doing, but do we really need more than this to change the with pkgs; at the top of all-packages.nix to with pkgs.__splicedPackages;?
diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix
index bb5f941b0a2..28786dc713f 100644
--- a/pkgs/top-level/all-packages.nix
+++ b/pkgs/top-level/all-packages.nix
@@ -8,7 +8,7 @@
 { lib, noSysDirs, config, overlays }:
 res: pkgs: super:

-with pkgs;
+with pkgs.__splicedPackages;

 {
   # A module system style type tag
@@ -14205,6 +14205,7 @@ with pkgs;
   # The GCC used to build libc for the target platform. Normal gccs will be
   # built with, and use, that cross-compiled libc.
   gccCrossStageStatic = assert stdenv.targetPlatform != stdenv.hostPlatform; let
+    binutilsNoLibc = pkgsHostTarget.binutilsNoLibc;
     libcCross1 = binutilsNoLibc.libc;
     in wrapCCWith {
       cc = gccFun {

A handful of tests for standard and cross attrs indicate it evaluates to the same store paths as before, including for those with the problematic dependency paths (like nix which uses pkgsStatic.busybox). Running nixpkgs-review also successfully evals the whole set and reports no changed packages
That said, I would still like to have callPackageWithExtraSet like @Artturin offered regardless of this

With callPackagesWithExtraSet how would pkgs.python310Packages.zstd specify that it does not need itself but pkgs.zstd?

Something like
  callPackagesWithExtraSets [ pkgs python3Packages ] ...;

or
  callPackagesWithExtraSet (python3Packages // { inherit (pkgs) zstd; }) ...;

could work. Here, pkgs would take precedence over python.

With callPackagesWithExtraSet how would pkgs.python310Packages.zstd specify that it does not need itself but pkgs.zstd?

the bracket part of callPackage ./blah { } would still function normally

I also think splicing should be removed, in favor of something more explicit or less leaky.
I have a rough idea of what I think should replace it (I may have mentioned this to @Ericson2314 already). I meant to write a longer proposal, but since we're having this discussion here's a summary:

In the most common case, packages would take two sets of arguments: buildHost stuff and hostTarget stuff. More complex packages could take even more sets of arguments.{ lib
, stdenv
, cmake
, sometool
}:
{ libfoo
, libbar
}:
stdenv.mkDerivation ...

Some sort of deep pattern matching in the nix language would help with this. Instead of multiple (easy to confuse) arguments, we could have{ lib # ← could be in a "platformIndependent" set instead of top level
, buildHost =
    { stdenv
    , cmake
    , sometool
    }
, hostTarget =
    { libfoo
    , libbar
    }
}:
stdenv.mkDerivation ...

callPackage would pass buildHost packages as first argument and hostTarget packages as second.

There could be a callPackage variant with more platform combinations (the nix extension mentioned above would avoid this)

buildInputs, nativeBuildInputs etc would be merged into a single inputs argument
There would be an override function for each input set: overrideBuildHost, overrideHostTarget...
Existing packages could be migrated gradually either by giving the new callPackage a different name, like callPackageV2 or by making it detect when a package takes only a single attribute set.

It looks like meat for a future RFC!

@Artturin IMHO if this is a discussion and not a mandate, title should reflect it. As it seems to be still being debated, it should not be fixed at nixpkgs [yet] to avoid confusion. And honestly, I'm still unsure of what I should be doing about this.

I also think splicing should be removed, in favor of something more explicit or less leaky.

@fgaz I have been working on something similar.  It's not finished, but I've posted the work-in-progress before I factor in your overrideHost/overrideBuild idea, which solves one of the problems I ran into.  https://github.com/NixOS/nixpkgs/issues/227327
Thanks to @Artturin for letting me know about this.

when something is inherited (ex, {inherit (pkgs.darwin.apple_sdk.frameworks) Security}) in a callPackage invocation

Note that RFC 140 will deprecate callPackage _ { /* non-empty */ } so this problem might solve itself.

@amjoseph-nixpkgs No it won't deprecate that, such invocations will still be valid! It's in scope to look at for the Nixpkgs Architecture Team, but until custom arguments have an adequate replacement of its use cases (which RFC 140 isn't), it shouldn't be deprecated

Some sort of deep pattern matching in the nix language would help with this

I've experimented with a similar approach that doesn't require changes to the Nix language. In https://github.com/NixOS/nixpkgs/compare/master...AtnNn:nixpkgs:deepcall callPackage looks for parameters that contain ' and "replaces" it with .. So packages can take as arguments llvmPackages'bintools, pkgsBuildHost'nodejs, perlPackages'CGIFast, targetPackages'stdenv'cc, xorg'libX11, python3Packages'docutils, gst_all_1'gstreamer, unixODBCDrivers'mariadb, etc.

@AtnNn very nice, thanks for trying it out!
While yours is a clever solution given the current Nix language limitation, I think in the long run a feature like the one I proposed is necessary to keep everything clean. Nix is a domain specific language after all!
At the moment I'm busy, but I might draft an RFC later this year

until custom arguments have an adequate replacement of its use cases

What can callPackage _ { /* non-empty */ } do that .override cannot?

Thanks for remembering me about this!

This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/is-this-the-current-state-of-things/45944/14

Random suggestion, why not implement cross via an argument in mkDerivation? Have an offset parameter denoting the current derivation's position in the (build, ..., build, build, host, host, host, ..., host) chain which determines their internal platform triplet. Then dependent mkDerivations can call overrideAttrs on their nativeBuildInputs with an offset one less then their own, and on their depsTargetTarget with an offset one higher. Then, we no longer need to worry about callPackage injecting slices in all the places they're needed. As a bonus, external consumers of Nixpkgs expressions calling mkDerivation can participate in cross as dependencies without having to inject their own __spliced.buildHost and __spliced.hostTarget. I don't have full knowledge of the history of cross and why it is the way it is right now, so maybe there's some massive hidden downside I'm not aware of? More thunks, I guess?

Then dependent mkDerivations can call overrideAttrs on their nativeBuildInputs with an offset one less then their own

I'm not sure how you're imagining this to work with overrideAttrs?

We'd add offset to derivationArg in make-derivation.nix, which I presume is how it becomes overrideable via overrideAttrs. To be a bit clearer, I'm proposing we add this lever to all mkDerivation calls everywhere; the universality is what makes it useful. Thinking about it a little bit more, we might also need a second offset to accomodate the more esoteric combinations like depsHostHost or depsBuildTarget.

This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/frustrations-about-splicing/49607/1

This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/frustrations-about-splicing/49607/7

The way forward for Darwin is to get rid of frameworks and provide an SDK in the stdenv. It doesn’t help other package sets, but it should at least get rid of one of the major contributors to this problem.

https://github.com/NixOS/nixpkgs/pull/346043 replaces the individual framework packages with top-level SDK packages for Darwin, which should help reduce some of the Darwin-caused pain here.


Get failed to fetch error only when import bcrypt

I don't know why but when i call bcrypt methods i get this error. When omit it works. I even know bug it is or not.  Bcrypt last version. I tested it on seperate file bcrypt work without any problem.

Works without a problem with 'bcryptjs' package. But with bcrypt not working.


Update to StyleCop.Analyzers 1.0.0-rc2 release

:+1:

Current coverage is 79.39%

Merging #1927 into stabilization will decrease coverage by -0.01% as of e026e47

@@            stabilization   #1927   diff @@
=============================================
  Files                 559     559
  Stmts               35094   35094
  Branches             9711    9711
  Methods
=============================================
- Hit                 27866   27864     -2
+ Partial              5673    5672     -1
- Missed               1555    1558     +3

Review entire Coverage Diff as of e026e47

Powered by Codecov. Updated on successful CI builds.


Support for Apple M1 silicon

Description
The library doesn't support the Apple M1 silicon (ARM64).
The .NET version is 6.0.100-rc.2.21505.57 which already supports this architecture.
The exception call stack is:
System.DllNotFoundException: Failed to load the librdkafka native library.
   at Confluent.Kafka.Impl.Librdkafka.TrySetDelegates(List`1 nativeMethodCandidateTypes)
   at Confluent.Kafka.Impl.Librdkafka.LoadOSXDelegates(String userSpecifiedPath)
   at Confluent.Kafka.Impl.Librdkafka.Initialize(String userSpecifiedPath)
   at Confluent.Kafka.Producer`2..ctor(ProducerBuilder`2 builder)
   at Confluent.Kafka.ProducerBuilder`2.Build()

Maybe related to https://github.com/edenhill/librdkafka/issues/3546
How to reproduce

Create a .NET 6 RC project and install the Confluent.Kafka package (1.8.2) in a Mac with M1 silicon
Create a producer instance via ProducerBuilder
Run the solution

Checklist
Please provide the following information:

[x] Confluent.Kafka nuget version: 1.8.2
[x] Operating system.

Hmmm, building the librdkafka from the source and referencing it via Library.Load seems to do the trick:
git clone https://github.com/edenhill/librdkafka.git
cd librdkafka
./configure --install-deps
brew install  openssl zstd pkg-config
./configure
make
sudo make install

Library.Load("/usr/local/lib/librdkafka.dylib");

So there is something work in the brew distribution of librdkafka. But anyway, it would be nice if the Nuget package included the correct runtime version.

@andrebires did you find a workaround for the above error? I've made all the steps you described using the latest HEAD revision and I received the same error as you when trying to publish a message. Consume works well so far.

@andrebires did you find a workaround for the above error? I've made all the steps you described using the latest HEAD revision and I received the same error as you when trying to publish a message. Consume works well so far.

No, unfortunately I didn't. I just gave up using Kafka with .NET and started using with Go, which is working.

same error with Macbook Air M1, dotnet 6.0

@mhowlett Are you planning to look into this issue in the near future?

@edenhill - any plans on this for librdkafka.redist?
I think what we ideally want is a way for you to add a candidate librdkafka binary to the set included in librdkafka.redist simply by referencing an additional nuget package. Then we, or anyone, could make and distribute librdkafka nuget packages for any platform. I think this might not have been feasible in the past, but it might be worth looking into this again.

As a workaround, this library works when installing Rosetta 2 and x64 version of .NET 6

This is my workaround to have both ARM and x64 Sdks side by side

Install .NET x64
Add this to .csproj file

<RuntimeIdentifier Condition="$([MSBuild]::IsOSPlatform('OSX'))">osx-x64</RuntimeIdentifier>

In other words for projects that use Kafka I force x64 SDK on OSX.

Any ETA for arm/M1 build? It's already half of Q1, is it going to happen or is it delayed?

This is my workaround to have both ARM and x64 Sdks side by side

Install .NET x64
Add this to .csproj file

<RuntimeIdentifier Condition="$([MSBuild]::IsOSPlatform('OSX'))">osx-x64</RuntimeIdentifier>

In other words for projects that use Kafka I force x64 SDK on OSX.

This worked for me too.
To be clear - i think this just "tricks" Confluent.Kafka to use the same librdkafka x64 binaries. It doesn't actually run the .NET6 in emulation mode

also relevant: https://github.com/confluentinc/confluent-kafka-dotnet/issues/1769

This is my workaround to have both ARM and x64 Sdks side by side

Install .NET x64
Add this to .csproj file

<RuntimeIdentifier Condition="$([MSBuild]::IsOSPlatform('OSX'))">osx-x64</RuntimeIdentifier>

In other words for projects that use Kafka I force x64 SDK on OSX.

This worked for me too.
To be clear - i think this just "tricks" Confluent.Kafka to use the same librdkafka x64 binaries. It doesn't actually run the .NET6 in emulation mode

Yeah, this seems to be used only by Nuget for restoring the dependencies of the project:

RID values are used to identify target platforms where the application runs. They're used by .NET packages to represent platform-specific assets in NuGet packages.

https://docs.microsoft.com/en-us/dotnet/core/rid-catalog

@mhowlett any updates for this? I'm currently blocked on this.

this is in 1.9.0-RC10


DJL-TensorRT-LLM : inference no longer working, tokenizer error.

Description
Inference no longer working for TensorRT-LLM, tried docker tags "tensorrt-llm-nightly" and "0.28.0-tensorrt-llm".
Expected Behavior
The serving engine should return a generated response from TensorRT-LLM.
Error Message
{"error":"Cannot provide chat completion for tokenizer: \u003cclass \u0027NoneType\u0027\u003e, please ensure that your tokenizer supports chat templates.","code":424}
How to Reproduce?

Create model directory, for example my-model.
Download Mistral-v0.2-7b-Instruct model files with tokenizer into my-model directory.
Create serving.properties file : For example
engine=MPI option.dtype=fp16 option.rolling_batch=trtllm option.trust_remote_code=true option.max_input_len=8192 option.max_output_len=8192 option.tensor_parallel_degree=1

start DJL docker container.
the container starts, performs convertion of the HF model to TRT model.
it reaches state of readiness :
"BOTH API bind to: http://<IP_ADDRESS>:8080"
When trying to execute a request :
curl  -H "Content-Type:application/json"   -d<EMAIL_ADDRESS>http://localhost:8080/v1/chat/completions
with body
{   "messages" : [             {"role": "user", "content": "Explain the process of osmosis."}   ],     "n": 1,     "do_sample": true,     "temperature": 0.3,     "top_p" : 0.9,     "top_k" : 50,     "repetition_penalty" : 1.1,     "max_tokens": 600 }
error is accepted :
{"error":"Cannot provide chat completion for tokenizer: \u003cclass \u0027NoneType\u0027\u003e, please ensure that your tokenizer supports chat templates.","code":424}
Tokenizer can be loaded from the directory using Transformers code. Works without issues.

@maaquib Can you take a look?

Was able to reproduce this. We're setting the tokenizer on InputFormatConfigs in huggingface and tnx handlers but not on the TRT-LLM handler which is causing this error

Fix #2100 merged. Will backport this to 0.28.0


Enable scheduling in run command

Version - v0.1.1-rc1
Comments:  Need to be able to schedule services to specific nodes.
Currently the rio run -h has info:

We need to either support those options or update help when it's enabled as well.

To test again v0.6.x (post Kubecon)

Help has been updated in the CLI to remove this. It looks like the CLI doesn't support this, but API does have NodeAffinity, PodAffinity, and PodAntiAffinity so we can leave this open if we want the CLI to support those in the feature.


Updated build.gradle

Configuration 'compile' is obsolete and has been replaced with 'implementation' and 'api'.

While compile is deprecated, compileOnly is not. It's actually the other way around: compileOnly is the replacement for the deprecated provided configuration.
Details can be found in the official docs. Thanks.


[HOLD] allow language suppression to be set as a property attribute

Why was this change made?
Fixes #3274 - allow users to indicate a given property template should be able to suppress language selection; if selected, don't show language selection box for that template and then set the default value to null instead of 'eng'
Note: a lot of the diff is updating the fixture data to add the new attribute.
How was this change tested?

Updated existing tests verifying new property being set in state
Added a new test verifying language dialog is suppressed when attribute is set
Localhost browser

Which documentation and/or configurations were updated?
Readme

I'm doing a major refactoring of some tests and this will cause a massive merge conflict. Let's hold this and I may create a new PR for merging this.