Readonly Flag Does Not Work

Readonly flag that's implemented in the connector does not get properly handed down through the create_engine flag.  This may be a use problem on my end and just need an example of this being used correctly, but based on this

https://github.com/cloudflare/sqlalchemy-clickhouse/pull/5 - PR Added readonly support.
https://github.com/cloudflare/sqlalchemy-clickhouse/issues/1 - Issue AttributeError: 'Connection' object has no attribute 'readonly'

this has been a problem area before that could use some more clarity.
Bug:
import sqlalchemy as sa
engine = sa.create_engine("clickhouse://USERNAME:PASSWORD@HOSTNAME/DATABASE", readonly=True)

Stack Trace:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "python2.7/site-packages/sqlalchemy/engine/__init__.py", line 419, in create_engine
    return strategy.create(*args, **kwargs)
  File "python2.7/site-packages/sqlalchemy/engine/strategies.py", line 161, in create
    engineclass.__name__))
TypeError: Invalid argument(s) 'readonly' sent to create_engine(), using configuration ClickHouseDialect/QueuePool/Engine.  Please check that the keyword arguments are appropriate for this combination of components.

Version Information:
$ pip freeze
SQLAlchemy==1.2.2
sqlalchemy-clickhouse==0.1.3.post0

You have two problems:

engine = sa.create_engine("clickhouse://USERNAME:PASSWORD@HOSTNAME/DATABASE", connect_args={'readonly': True})
The current pip version (0.1.3.post0) doesn't yet have the readonly flag update.


[ci] Hardcode number of jobs and revise action trigger

The compiler CI job failures are due to circleci misreporting the number of processors, this seems to fix it.

Interesting. It shouldn't crash/hang though.

It is a memory issue. If I simulate using a docker container with 8 CPUs and 16 GB RAM, running skargo test --jobs 36 and monitoring with htop shows that memory is exhausted almost instantly, and all hell breaks loose then. That is only ~450MB per process so it makes sense.

That makes sense. I have a branch somewhere that mitigates (some of) the memory hungriness of sktest, I'll dig it back.


⚠ Fortnite has degraded performance

In 1ee9a9d, Fortnite (https://fortnite.com) experienced degraded performance:

HTTP code: 200
Response time: 1345 ms

Resolved: Fortnite performance has improved in 0643be3.


FPGA SRIO-V is not supportted by the FPGA device plugin

I am trying FPGA device plugin in my local env with SRIOV-enabled A10 FPGA. It has 1 VF and 1 PF.
The FPGA device plugin is always in CrashLoopBackOff status.
The output of  k logs intel-fpga-plugin-kgnz7 --namespace kube-system is the following:
FPGA device plugin (OPAE) started in af mode
Device scan failed: intel-fpga-dev.1: AFU without corresponding FME found
main.(*devicePlugin).scanFPGAs
/intel-device-plugins-for-kubernetes/cmd/fpga_plugin/fpga_plugin.go:296
main.(*devicePlugin).Scan
/intel-device-plugins-for-kubernetes/cmd/fpga_plugin/fpga_plugin.go:204
github.com/intel/intel-device-plugins-for-kubernetes/pkg/deviceplugin.(*Manager).Run.func1
/intel-device-plugins-for-kubernetes/pkg/deviceplugin/manager.go:96
runtime.goexit
/usr/lib/golang/src/runtime/asm_amd64.s:1357
Thanks in advance.

Which model of the Arria10 card do you use? Which kernel driver(DFL or OPAE) do you use? Please, show the output of ls -la /sys/class/fpga command here.

Which model of the Arria10 card do you use? Which kernel driver(DFL or OPAE) do you use? Please, show the output of ls -la /sys/class/fpga command here.

Here are my outputs:
$ lspci | grep 9c
04:00.0 Processing accelerators: Intel Corporation Device 09c4
04:00.1 Processing accelerators: Intel Corporation Device 09c5
$ ls -al /sys/class/fpga
total 0
drwxr-xr-x  2 root root 0 Apr 27 14:20 .
drwxr-xr-x 79 root root 0 Apr 23 22:13 ..
lrwxrwxrwx  1 root root 0 Apr 24 03:44 intel-fpga-dev.0 -> ../../devices/pci0000:00/0000:00:03.0/0000:04:00.0/fpga/intel-fpga-dev.0
lrwxrwxrwx  1 root root 0 Apr 27 14:15 intel-fpga-dev.1 -> ../../devices/pci0000:00/0000:00:03.0/0000:04:00.1/fpga/intel-fpga-dev.1
$ ls | grep opae
opae-intel-fpga-driver-1.3.0-2
opae-intel-fpga-driver-1.3.0-2.tar.gz
opae-sdk-1.3.0-2

@xxinran looks good so far. Next two things I'd like to see are the output of these 2 commands:
ls -la /sys/class/fpga/intel-fpga-dev.0/ and lsmod |grep fpga. I suspect that intel-fpga-fme module is not loaded according to plugin error, but let's see what it is in reality.

@bart0sh Thanks for your quick reply, here is my outputs:
$ ls -la /sys/class/fpga/intel-fpga-dev.0/
total 0
drwxr-xr-x  4 root root    0 Apr 27 15:15 .
drwxr-xr-x  3 root root    0 Apr 24 03:43 ..
lrwxrwxrwx  1 root root    0 Apr 24 03:49 device -> ../../../0000:04:00.0
drwxr-xr-x 11 root root    0 Apr 24 03:43 intel-fpga-fme.0
drwxr-xr-x  2 root root    0 Apr 24 03:49 power
lrwxrwxrwx  1 root root    0 Apr 24 03:43 subsystem -> ../../../../../../class/fpga
-rw-r--r--  1 root root 4096 Apr 24 03:43 uevent
$ ls -la /sys/class/fpga/intel-fpga-dev.1/
total 0
drwxr-xr-x 4 root root    0 Apr 27 15:16 .
drwxr-xr-x 3 root root    0 Apr 27 14:15 ..
lrwxrwxrwx 1 root root    0 Apr 27 14:20 device -> ../../../0000:04:00.1
drwxr-xr-x 4 root root    0 Apr 27 14:15 intel-fpga-port.1
drwxr-xr-x 2 root root    0 Apr 27 15:16 power
lrwxrwxrwx 1 root root    0 Apr 27 14:15 subsystem -> ../../../../../../class/fpga
-rw-r--r-- 1 root root 4096 Apr 27 14:15 uevent
$ lsmod |grep fpga
intel_fpga_afu         36864  0
intel_fpga_fme         61440  0
intel_fpga_pci         32768  2 intel_fpga_fme,intel_fpga_afu
fpga_mgr_mod           16384  1 intel_fpga_fme

@xxinran thanks for the info. It looks incorrect from plugin point of view. It should be intel-fpga-fme.1 directory under /sys/class/fpga/intel-fpga-dev.1/. Plugin expects it to be present. Here is an example from my system:
> ls -la /sys/class/fpga/intel-fpga-dev.0/
total 0
drwxr-xr-x  5 root root    0 Apr  7 12:29 .
drwxr-xr-x  3 root root    0 Apr  7 12:29 ..
lrwxrwxrwx  1 root root    0 Apr  8 13:49 device -> ../../../0000:06:00.0
drwxr-xr-x 12 root root    0 Apr  7 12:29 intel-fpga-fme.0
drwxr-xr-x  4 root root    0 Apr  7 12:29 intel-fpga-port.0
drwxr-xr-x  2 root root    0 Apr  8 04:28 power
lrwxrwxrwx  1 root root    0 Apr  7 12:29 subsystem -> ../../../../../../class/fpga
-rw-r--r--  1 root root 4096 Apr  7 12:29 uevent
ed@r720-1:~> ls -la /sys/class/fpga/intel-fpga-dev.1/
total 0
drwxr-xr-x  5 root root    0 Apr  7 12:29 .
drwxr-xr-x  3 root root    0 Apr  7 12:29 ..
lrwxrwxrwx  1 root root    0 Apr  8 13:49 device -> ../../../0000:42:00.0
drwxr-xr-x 12 root root    0 Apr  7 12:29 intel-fpga-fme.1
drwxr-xr-x  4 root root    0 Apr  7 12:29 intel-fpga-port.1
drwxr-xr-x  2 root root    0 Apr  8 04:28 power
lrwxrwxrwx  1 root root    0 Apr  7 12:29 subsystem -> ../../../../../../class/fpga
-rw-r--r--  1 root root 4096 Apr  7 12:29 uevent

I'd suggest to investigate why does it happen by looking at the dmesg output or asking OPAE devs. You can also try to re-insert cards one by one to see if they both show intel-fpga-fme entry in the /sys/class/fpga/intel-fpga-dev.0/ directory.

@bart0sh  It seems that you can 2 physical FPGA card, not one vf and one pf.  Did you enable SRIOV and virtualize the FPGA into VFs. If I disabled SRIOV, it will be only one fme and one port in intel-fpga-dev-0, and there is no intel-fpga-dev-1 repo, and the FPGA device plugin will report correctly.
But if I enable SRIOV,  the /sys/class/fpga will have 2 intel-fpga-dev-*, as I copyed above, in this case, the plugin will show this error: AFU without corresponding FME found.

@xxinran can you please show output of /opt/intel/fpga-sw/fpgatool list and /opt/intel/fpga-sw/fpgatool -d intel-fpga-port.1 fpgainfo on your system?

@xxinran The plugin doesn't support SRIO-V setup at the moment. However, this is just because we didn't test this configuration. We'll work on this issue and report here when it's fixed.

@bart0sh  Ah, got it. Thanks for your explaination :)

@xxinran can you still reply with output of fpgatool in your system, so we can validate that at least part of functionality working in such setups?

Sure.
$ sudo /opt/intel/fpga-sw/fpgatool -d intel-fpga-port.1 fpgainfo

//****** PORT ******//
Name                             : intel-fpga-port.1
Device Node                      : /dev/intel-fpga-port.1
SysFS Path                       : /sys/devices/pci0000:00/0000:00:03.0/0000:04:00.1/fpga/intel-fpga-dev.1/intel-fpga-port.1
PCIe s:b:d:f                     : 0000:04:00.1
Physical Function PCIe s:b:d:f   : 0000:04:00.0
Device Id                        : 0x8086:0x09c5
Device Class                     : 0x120000
Local CPUs                       : 0-7,16-23
NUMA                             : 0
FME Name                         : intel-fpga-fme.0
Port Id                          : 0
Interface UUID                   : 9926ab6d6c925a68aabca7d84c545738
Accelerator UUID                 : f7df405cbd7acf7222f144b0b93acd18
Kernet API Version               : 0
Port Regions                     : 2
Port Region (Index/Size/Offset)  : 0 / 262144 / 0
Port Region (Index/Size/Offset)  : 1 / 4096 / 262144

$ /opt/intel/fpga-sw/fpgatool list
Detected FPGA FMEs: 1
intel-fpga-fme.0
Detected FPGA Ports: 1
intel-fpga-port.1

$ ls  /sys/class/fpga
intel-fpga-dev.0  intel-fpga-dev.1

$ ls /sys/class/fpga/intel-fpga-dev.0
device  intel-fpga-fme.0  power  subsystem  uevent

$ ls  /sys/class/fpga/intel-fpga-dev.1
device  intel-fpga-port.1  power  subsystem  uevent

Thanks @xxinran, so it is only discovery phase of the plugin will require fix.


buildMix: fix: initial try

Motivation for this change
buildMix is broken
More details on https://github.com/NixOS/nixpkgs/issues/105002
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[ ] NixOS
[x] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

Here are instructions on how to test.

clone the following repo https://github.com/happysalada/nix_demo
run nix-build ./build.nix in it.
(I have added the instructions in the beam-section to the build.nix file).

(upon your first try, nix will complain that the sha256 is empty, I'm just leaving this empty so that everybody can verify by themselves.)
(please remember that in order to run the release you will need to set the env var RELEASE_TMP to a writable directory)
Here are the questions I have that are open

by including the building of the js dependencies, the build does not seem to be a Fixed Output Derivation. By trying to add the following three lines at the end of the buildMix file

  outputHashAlgo = "sha256";
  outputHashMode = "recursive";
  outputHash = sha256;

it can be verified, that whatever sha256 output is given, the resulting hash will always be different.
I have to say I'm a little puzzled as to why that is the case.
.

@hauleth you might be interested in this. Absolutely no worries if you don't have the time.
Most of the stuff here is based off your stuff. Thanks again for a lot of gems!

@cw789 you might be interested into this. I want to distill some of the greatness you put in nix_elixir_example in the doc ideally. No worries if you don't have time.

Initial adding of documentation

Hi Raphael.
I can have a look, but won't be able to test at the moment, as I'm currently sick.
First thing I notice, buildMix seems to be a different thing then.
Former it was to build (compile) Elixir dependencies (packages / libraries).
Now it will create a release (services).
mkdir: cannot create directory ‘/nix/store/n8wm8iy4w51na8b4bzizx6yxhwd693fl-union-0.0.1/tmp’: Permission denied

To this trouble - releases always need a writable tmp directory,
whish default is release_directory/tmp.
As your release is within the nix store which is never writable,
you need to change the tmp directory before starting the release.
This could be archived by setting env variable RELEASE_TMP - to e.g. /home/xxx/my_release_tmp
You can do this also within env.sh (documented within mix release).

@cw789 Thanks Chris, your help is much appreciated. ❤
No rush at all on the review, I don't think this will be merged that quick.
Thanks a lot for the tips. I will have a look at those tomorrow.
I agree with you that the buildMix seems to have been different. I removed large part of what was there before in the hope that somebody would say why they were needed. I think this will take a while to be merged, so hopefully someone can say if we forgot/missed something.

As an aside, I would never have thought that you could spin up postgresql like that!
For testing I personally would create a nixos VM, but your method is good to have in mind :smiley:

Also, if we are changing the meaning of buildMix, like @cw789 said, shouldn't we be using mix releases? I think they weren't actually part of Mix when buildMix was created, so it might make more sense now.
Also, I have a question that is maybe out of scope for this PR, but might be nice to keep in mind. It seems to me that configuration of Mix projects are given at build time. Does that mean that for future NixOS services of Mix applications, rebuilding your system will mean you will have to rebuild every configured Mix applications? Is there a way around it?

@minijackson On the mix release comment, unless I misunderstand something, this is using mix releases to pack your application.
After @cw789 's comment, here is what I am thinking about.

ask user to mix release.init and add the RELEASE_TMP="/private/tmp"
I feel a bit weird having to define the tmp directory before creating a service, but I don't see any way around it at the moment.

I have added a build.nix file to make the process simpler and it includes a demonstration of the RELEASE_TMP variable. It's working great.
There are two more variables needed to run a basic release though. DATABASE_URL and SECRET_KEY
I still feel weird about encoding the DATABASE_URL on building.
Perhaps we should include something in the doc to ask users to put that variable in the dynamically configurable variables? like in config.prod.exs.
Maybe it's just me.

All things considered, would it be a possibility to ask users to put their DATABASE_URL and SECRET KEY in the build.nix file?
We could ask them to supply RELEASE_TMP at the same time.
Does it make sense to everybody?

unless I misunderstand something, this is using mix releases to pack your application.

oops, that's my bad, I forgot to read after the comma. I got the impression that it wasn't when I read that there was no install phase, I figured that it was up to the end-user to choose what to install. In retrospect, that is obviously not the case.

ask user to mix release.init and add the RELEASE_TMP="/private/tmp"

Isn't mix release.init for upstream to do? I am not sure what the RELEASE_TMP variable would mean, can you give more details?
I've had a bit of thinking and experimenting with your PR. First of all, I managed to compile the Elixir part of Mobilizon without any particular issues :tada:
I was thinking that it would be best if we could specify in the configuration that we want these variables to be resolved at runtime, and fetch them from the environment (instead of fetching the environment at build time). I found this nice feature while looking through the docs: you can specify a runtime.exs file, which is exactly what I wanted.
With this, I added my custom Mobilizon runtime.exs in preBuild:
{
  # ...
  preBuild = ''
    cp "${./runtime.exs}" config/runtime.exs
  '';
}

With this, here's what I'm proposing for the moment: we add a runtimeConfiguration argument to buildMix which "environmentalizes" the configuration, and we automatically put in $src/config/runtime.exs and mix release takes care of the rest. This allows not recompiling the whole application on each configuration change, and would prevent secrets from being world readable in the Erlang bytecode stored in the Nix store.
An even better solution in my opinion would be to generate the runtime.exs in the style of RFC42, instead of going through environment variables. This would be nicer since there can be quite a lot of configuration options, so providing a runtime.exs which allows the modification of every variable can become quite tedious and error prone. The drawback for now is that we probably have to find a way to provide the runtime.exs outside of the release, which may be hacky. I'll try to find if this is possible.
But the good news is that I managed to get Mobilizon local instance up and running :tada:, although I had to compile the yarn part manually...

@minijackson I like your idea of using runtime variables.
What about letting the user define their runtime.exs file, and then just patching the env.sh.exs file in the end.
Since env.sh.exs is actually just a shell file, it would just be a matter of concatenating export VAR="my_value" at the end of it.
I personally only use runtime env vars for secret stuff, however I'm concerned about the experience for people starting a new phoenix project. I don't know why, but the phoenix default is to have DATABASE_URL as a compile time dependency. I was thinking to add compileEnvVars and just inject that in the preConfigure hook if people need.
Then we could define runtimeEnvVars inside a service definition (in a later PR perhaps?) and those would just patch the env.sh file inside the release.
That way we stay completely out of defining runtime env vars when we are just building the service.
PS: Here is the explanation around RELEASE_TMP and mix release.init.

mix release.init
When you mean mix release.init done by upstream, we mean the end user right? The reason I mentioned asking the user to do it, is that it's not mandatory, some people never do it. Maybe we can add a check for those files on build to make sure they are there. The reason it can be useful is that it provides a file env.sh.exs that is just a shell file to pass on extra runtime environment variables. In the official elixir doc, they use the runtime.exs by just calling System.fetch_env!/1 . It looks like the job of supplying environment variable is instead carried out by env.sh

RELEASE_TMP
Upon starting a release, some space on the disk will be used to write some configuration variables in a file. It does so in the RELEASE_TMP specified directory. Usually it's just the same directory as the release, but if we build the release in the nix store, that location won't be writable. So we need to supply a writable location that will be used to store the BEAM config variables. If you don't set this variable and try to run your release, it should throw an error regarding a not writable path. I'm not sure why you didn't get an error.

After some more tests, here is what I found out.

mix release.init is unecessary, no need to force people to use it.
In order to run the release, simply setting RELEASE_TMP to a writable dir will work. (of course you will also need to set your other envirenment variable for your release).
I kept the depsPreConfigure hook in there, just in case people actually have build time env variable dependency. Even though I can not personally think of a good reason why. I just added a recommendation in the docs.

Here are the tasks that could be improved.

Add docs about yarn support? Maybe with yarn2nix?
Add an exemple service file? I don't think anybody is going to run their release raw.
Here is an early draft, just putting it here for discussion
with an example service of union

    users.groups.union = { };
    users.users.union = {
      description = "Union service user";
      group = "union";
      isSystemUser = true;
    };
    systemd.services.union = {
      description = "My super service";
      wantedBy = [ "multi-user.target" ];
      after = [ "network-online.target" ];
      requires = [ "network-online.target" ];
      ExecStart = "${pkgs.vector}/bin/vector --config ${validateConfig conf}";
      User = "union";
      Group = "union";
      Restart = "no";
      StateDirectory = "union";
      ExecReload = "${pkgs.coreutils}/bin/kill -HUP $MAINPID";
      Environment = [
          "RELEASE_TMP=/private/tmp"
          "MY_VAR="MY_VAL"
        ];
    };

What about letting the user define their runtime.exs file, and then just patching the env.sh.exs file in the end.
Since env.sh.exs is actually just a shell file, it would just be a matter of concatenating export VAR="my_value" at the end of it.

What do you mean by user? Do you mean the package maintainer, or nixos module maintainer?
If we want to go through with the RFC42-style module, I think it would be best to patch runtime.exs directly, instead of going through environment variables, which was a possibility I hadn't considered, thanks!
If we want to go through with runtime variables, though, I would personally use systemd.services.<service>.environment or systemd.services.<service>.serviceConfig.EnvironmentFile for secrets.

I personally only use runtime env vars for secret stuff, however I'm concerned about the experience for people starting a new phoenix project. I don't know why, but the phoenix default is to have DATABASE_URL as a compile time dependency. I was thinking to add compileEnvVars and just inject that in the preConfigure hook if people need.

Do you mean that the DATABASE_URL is used at compile time by Phoenix, and cannot be changed at runtime? From my understanding, this is against the official library guidelines, but it is still possible. From my experience with Mobilizon, I managed to change the Ecto database hostname without issues. But if you want to change a variable at build time, a package manager can easily do:
{ ... }:

buildMix {
  # ...
  DATABASE_URL = "...";
}

When you mean mix release.init done by upstream, we mean the end user right?

I meant the software developers (as in the Mobilizon devs). But as you said, it is not necessary, one less thing to worry about ^^

Maybe we can add a check for those files on build to make sure they are there

I think there is no need for that, if they are not there, mix release will create them from a template, and we can specify environment variables through systemd, of other means (for example, direnv, or just plain export when we are not using a nixos module).

If you don't set this variable and try to run your release, it should throw an error regarding a not writable path. I'm not sure why you didn't get an error.

Thanks for the clarification, it helped.
It seems RELEASE_TMP is only used when we are starting the application using "daemon" or "daemon_iex", or using "eval" with RUNTIME_CONFIG=true is set in the sys.config file (whatever that means). For comparison, I have RUNTIME_CONFIG=false in my Mobilizon build. I don't think it is a blocker if we just want NixOS modules, but this can definitely be annoying for development. But I think this can easily be fixed by making a wrapper.

PPS: out of curiosity, did you use the yarn2nix tool? I'm wondering how well it works, if it works well, I was thinking we could add some documentation about it too.

I did try to use it, but the "resolutions" feature of package.json doesn't seem to be implemented, so it fails to build for Mobilizon.

I don't think anybody is going to run their release raw.

I just did, to test out things ^^
I can also see it being useful for development.

Here is an early draft, just putting it here for discussion
with an example service of $name

I think your systemd service looks good. Here is some small nitpicks:
{
  systemd.services.${name} = {
    description = "My super service";
    wantedBy = [ "multi-user.target" ];
    after = [ "network-online.target" ];
    requires = [ "network-online.target" ];
    # Raw systemd service options inside serviceConfig
    serviceConfig = {
      ExecStart = "${pkgs.${name}}/bin/${name} start";
      # Encouraging people of having a nice set of privacy defaults,
      # but might not be applicable everywhere.
      # Best if it used the software developers upstream.
      DynamicUser = true;
      StateDirectory = "${name}";
      # Implied by DynamicUser, but just to emphasize due to RELEASE_TMP
      PrivateTmp = true;
      EnvironmentFile = "/path/to/my/secrets.env";
      # I am not sure about the Restart and ExecReload options,
      # does sending a SIGHUP signal does something to the Erlang VM?
    };
    # Using the nixos declarative way
    environment = {
      # Set to "/tmp", made private by systemd, although not necessary here
      # Maybe this should be in buildMix, set in the wrapper
      RELEASE_TMP = "/tmp";
      # Maybe an example for the StateDirectory?
      MY_DATA_DIR = "/var/lib/${name}";
    };
  };
}

Sorry for the long message, and thanks again!

What about letting the user define their runtime.exs file, and then just patching the env.sh.exs file in the end.
Since env.sh.exs is actually just a shell file, it would just be a matter of concatenating export VAR="my_value" at the end of it.

What do you mean by user? Do you mean the package maintainer, or nixos module maintainer?
If we want to go through with the RFC42-style module, I think it would be best to patch runtime.exs directly, instead of going through environment variables, which was a possibility I hadn't considered, thanks!
If we want to go through with runtime variables, though, I would personally use systemd.services.<service>.environment or systemd.services.<service>.serviceConfig.EnvironmentFile for secrets.

I personally only use runtime env vars for secret stuff, however I'm concerned about the experience for people starting a new phoenix project. I don't know why, but the phoenix default is to have DATABASE_URL as a compile time dependency. I was thinking to add compileEnvVars and just inject that in the preConfigure hook if people need.

Do you mean that the DATABASE_URL is used at compile time by Phoenix, and cannot be changed at runtime? From my understanding, this is against the official library guidelines, but it is still possible. From my experience with Mobilizon, I managed to change the Ecto database hostname without issues. But if you want to change a variable at build time, a package manager can easily do:
{ ... }:

buildMix {
  # ...
  DATABASE_URL = "...";
}

When you mean mix release.init done by upstream, we mean the end user right?

I meant the software developers (as in the Mobilizon devs). But as you said, it is not necessary, one less thing to worry about ^^

Maybe we can add a check for those files on build to make sure they are there

I think there is no need for that, if they are not there, mix release will create them from a template, and we can specify environment variables through systemd, of other means (for example, direnv, or just plain export when we are not using a nixos module).

If you don't set this variable and try to run your release, it should throw an error regarding a not writable path. I'm not sure why you didn't get an error.

Thanks for the clarification, it helped.
It seems RELEASE_TMP is only used when we are starting the application using "daemon" or "daemon_iex", or using "eval" with RUNTIME_CONFIG=true is set in the sys.config file (whatever that means). For comparison, I have RUNTIME_CONFIG=false in my Mobilizon build. I don't think it is a blocker if we just want NixOS modules, but this can definitely be annoying for development. But I think this can easily be fixed by making a wrapper.

PPS: out of curiosity, did you use the yarn2nix tool? I'm wondering how well it works, if it works well, I was thinking we could add some documentation about it too.

I did try to use it, but the "resolutions" feature of package.json doesn't seem to be implemented, so it fails to build for Mobilizon.

I don't think anybody is going to run their release raw.

I just did, to test out things ^^
I can also see it being useful for development.

Here is an early draft, just putting it here for discussion
with an example service of $name

I think your systemd service looks good. Here is some small nitpicks:
{
  systemd.services.${name} = {
    description = "My super service";
    wantedBy = [ "multi-user.target" ];
    after = [ "network-online.target" ];
    requires = [ "network-online.target" ];
    # Raw systemd service options inside serviceConfig
    serviceConfig = {
      ExecStart = "${pkgs.${name}}/bin/${name} start";
      # Encouraging people of having a nice set of privacy defaults,
      # but might not be applicable everywhere.
      # Best if it used the software developers upstream.
      DynamicUser = true;
      StateDirectory = "${name}";
      # Implied by DynamicUser, but just to emphasize due to RELEASE_TMP
      PrivateTmp = true;
      EnvironmentFile = "/path/to/my/secrets.env";
      # I am not sure about the Restart and ExecReload options,
      # does sending a SIGHUP signal does something to the Erlang VM?
    };
    # Using the nixos declarative way
    environment = {
      # Set to "/tmp", made private by systemd, although not necessary here
      # Maybe this should be in buildMix, set in the wrapper
      RELEASE_TMP = "/tmp";
      # Maybe an example for the StateDirectory?
      MY_DATA_DIR = "/var/lib/${name}";
    };
  };
}

Sorry for the long message, and thanks again!

Thanks for all the suggestions. I need a couple of days before I can come back to this. I'll reply soon.

Just found out about this PR https://github.com/NixOS/nixpkgs/pull/103138 (packaging pleroma otp from a binary)
Very informative!

I have added two TODOs

improve overriding various steps of the derivation. (at the moment I personally don't use it, but I'm sure somebody will need this soon)
Figure out a way to do Fixed output derivation with a hash with a way that plays well with Phoenix projects that have frontend dependencies.

I've removed the passthru stuff since we don't use it anymore
I've added a wrapper inside the build to set a default for RELEASE_TMP
I've added an example service in the docs for people who are interested. I've taken it from here thanks @cw789 , I've incorporated @minijackson 's suggestions, thanks too!
I believe this PR is ready for prime time.
Please let me know if you have any more comments in the next few days before I ask someone with merge rights to have a look at it.

@happysalada dumb idea maybe, but would it make sense to let the frontend be handled in its own step for now, like people do their own derivation and we link it in the correct place in priv ?
After all, this is a release thing right ?

@DianaOlympos I think you're right on this one. Let me have a look at this tomorrow.

@DianaOlympos I've checked again, and I need precisions on what you are suggesting.
buildMix is using mix release and passing nodeDependencies externally.
I'm not sure how to do it differently. If you have a suggestion, I'm open.

@DianaOlympos I've checked again, and I need precisions on what you are suggesting.
buildMix is using mix release and passing nodeDependencies externally.
I'm not sure how to do it differently. If you have a suggestion, I'm open.

What I had in mind, was to have a separate package, where you compile the frontend, and then in the buildMix part, you can copy the compiled version over the source, for example in the preBuild phase. I would prefer letting package maintainer doing this, since Phoenix are only a part of Elixir projects, and each Phoenix project may have different frontend building system (NPM vs Yarn vs Webpack, etc.). The disadvantage, is that it might seem "hacky" to copy over the source, unfortunately. But this is what I've seen in nixpkgs to compile cross-language projects, so far.

@minijackson @DianaOlympos this is slowly starting to make sense to me.
Here is what I am thinking so far.

The build step is just doing mix compile. (I'm thinking that there should be the ability to pass compile time flags probably. I know I almost always use the --warnings-as-errors
The install step would be doing the mix release. Not sure about this one. Just thinking that those two steps need to be separated in order to enable passing frontend dependencies.
I'm still not sure about having the buildMix also fetch the dependencies. Perhaps this should be separated and left to the user? The user would then fetch-mix-deps and then pass that to buildMix ?

The build step is just doing mix compile. (I'm thinking that there should be the ability to pass compile time flags probably. I know I almost always use the --warnings-as-errors
The install step would be doing the mix release. Not sure about this one. Just thinking that those two steps need to be separated in order to enable passing frontend dependencies.

I don't think there is a need to separate these two steps for the frontend, we can install the compiled frontend over the source before mix compile is run.

I'm still not sure about having the buildMix also fetch the dependencies. Perhaps this should be separated and left to the user? The user would then fetch-mix-deps and then pass that to buildMix ?

Yes, I also think it is best that buildMix does only mix-specific things

The problem that I see with passing the frontend before mix compile is run is that if you change something in the frontend assets, then you need to recompile your whole app, which is not ideal if you are asking me.
I understand your point with regards to breaking compatibility, and it's a good one. I feel thought that if they update their nixpkgs, they will see the breakage and they should be able to fix it? Not sure what is the best way to proceed here.

The problem that I see with passing the frontend before mix compile is run is that if you change something in the frontend assets, then you need to recompile your whole app, which is not ideal if you are asking me.

Do you mean that when developing the package changing an asset leads to recompiling the whole package? To me, that seems like a normal Nix drawback. From my understanding, this cannot be circumvented by separating the phases, we would need to have three separate derivations, which would complicate things, in my opinion.
But when developing, I think most people would use the nix-shell, and in that case, mix compile wouldn't do anything, since it can reuse the build directory.

I understand your point with regards to breaking compatibility, and it's a good one. I feel thought that if they update their nixpkgs, they will see the breakage and they should be able to fix it? Not sure what is the best way to proceed here.

If we want to break compatibility, I think nice thing we could do is to provide an easy fix, which is why I proposed the dontFetchMixDeps attribute.

The problem that I see with passing the frontend before mix compile is run is that if you change something in the frontend assets, then you need to recompile your whole app, which is not ideal if you are asking me.

Do you mean that when developing the package changing an asset leads to recompiling the whole package? To me, that seems like a normal Nix drawback. From my understanding, this cannot be circumvented by separating the phases, we would need to have three separate derivations, which would complicate things, in my opinion.

I was under the impression that if you pass the same arguments to a function it would evaluate to the same results, even within a derivation. I might be wrong there. For me the advantage of passing front end assets after is that the compile step would have exactly the same elements and so would be cached? (not sure about this).
If that's not the case, then i agree with you, the only way would be to have 3 separate derivations which doesn't make sense

But when developing, I think most people would use the nix-shell, and in that case, mix compile wouldn't do anything, since it can reuse the build directory.

I understand your point with regards to breaking compatibility, and it's a good one. I feel thought that if they update their nixpkgs, they will see the breakage and they should be able to fix it? Not sure what is the best way to proceed here.

If we want to break compatibility, I think nice thing we could do is to provide an easy fix, which is why I proposed the dontFetchMixDeps attribute.

I see your point. The other way would be to pass dependencies in the configure phase for example. Which is the behavior that seem to have taken place before, since buildMix was not fetching dependencies. I don't understand how it could have worked without copying dependencies to the /deps folder. I couldn't find this step anywhere. It makes me wonder really how this think could ever have worked in the past.
I agree with you on the fact that every other builder out there goes on to fetch dependencies. I personally don't think that many people use the current buildMix, so it would be better to not add weirdness to fix backward compatibility for non existing users.

For me the advantage of passing front end assets after is that the compile step would have exactly the same elements and so would be cached? (not sure about this).

Unfortunately, Nix doesn't cache per-phase, only per-derivation. This allows you to do non-deterministic things in certain phases (like installing Hex, here), as long as the result is deterministic.

I don't understand how it could have worked without copying dependencies to the /deps folder. I couldn't find this step anywhere. It makes me wonder really how this think could ever have worked in the past.

I think this is why we've seen several projects for compiling Mix projects in Nix, as you said people were most likely not using nixpkgs' buildMix at all.

I agree with you on the fact that every other builder out there goes on to fetch dependencies. I personally don't think that many people use the current buildMix, so it would be better to not add weirdness to fix backward compatibility for non existing users.

Sounds good to me :+1:

things I had to patch:

the .git directories in the mix-deps drv were non-deterministic, thus breaking build

copied some code from nix-prefetch-git to make them deterministic

afterwards mix was complaning there was no git command

added git via beamDeps

mix was complaning the git deps were out of sync with the lock

i made it stop complaning about things being out of sync using a patch to elixir

it was trying to wrap some of the executables that weren't executable

i made it stop doing that

you can view the diff at https://github.com/happysalada/nixpkgs/compare/fix_build_mix...mkg20001:fix_build_mix

hello @mkg20001!

the .git directories in the mix-deps drv were non-deterministic, thus breaking build

copied some code from nix-prefetch-git to make them deterministic

Very nice find, I hadn't thought about that. What do you mean by breaking the build, though? To my understanding, having an non-deterministic doesn't lead to a compilation failure.
Is the removal of git directories, etc. the reason the lock check fails?
Probably completely out-of-scope for this PR, but I wonder if we could extract make_deterministic_repo and make it available globally for nixpkgs, this seems very useful for a lot of cases.

afterwards mix was complaning there was no git command

added git via beamDeps

I'm not seeing this change in your diff, could you point us to it?

mix was complaning the git deps were out of sync with the lock

i made it stop complaning about things being out of sync using a patch to elixir

I've looked a bit at your patch, and to my understanding, this effectively ignores the lock check failure, is this correct? If so, I unfortunately don't think this would be a good idea to integrate the patch into nixpkgs, as the lock check is useful for both NixOS users in development mode, and for us to check that the mix deps is not out of sync (e.g. when someone updates a package, but forgets to update depsSha256, or if we have made a mistake in fetchMixDeps / buildMix).

it was trying to wrap some of the executables that weren't executable

i made it stop doing that

Do you mean that the wrapping that sets the RELEASE_TMP variable? Is it the executable bit that weren't set?
Thank you!

Very nice find, I hadn't thought about that. What do you mean by breaking the build, though?

After generating the hash the first time, for some reason it tries to generate it again when running the build again. And then it fails because that outputs a different hash. Not sure what's causing it.

Is the removal of git directories, etc. the reason the lock check fails?

Maybe. I'm not entirely sure what it is checking, some things like remote branches, etc get removed.

could you point us to it?

I call buildMix like this
buildMix {
 ...
 beamDeps = [ git ];
 ...
}

but putting it in nativeBuildInputs achieves the same thing

this effectively ignores the lock check failure, is this correct

yes. it's more of a quick-and-dirty approach, as I didn't had the time to read elixir's code for a real fix.

Is it the executable bit that weren't set?

most likely. doing chmod a+x should prob work out.
having the abbility to override postFixup (or set dontWrap=true;) in case the warp is not wanted might possibly be a good idea, in case a more complex setup has binaries that the wrap does not apply to

I added git as nativeBuildInputs
I added the ability to prevent wrapping to set the default env var
I also separated the build and install step.
I'm having a look at making the frontend dependencies a separate derivation that would be passed to buildMix

@mkg20001 I included your initial approach of just removing the git dirs in the fetch mix deps. Did that approach not work for some reason?
I like that approach because of its simplicity.
If we need to go all the way and re-use make_deterministic_repo, then we should probably abstract it into a function somewhere?

mix was throwing a different error, but not exactly sure
It seems to use some git commands in dep.ex when validating the repository so I assume we either have to disable that or make the .git directory available
For make_deterministic_repo global availability: It's possible to simply copy the bash function and make a script that calls it make_deterministic_repo() { ... } make_deterministic_repo "$@", writeShellScriptBin and then use that script from path in the fetchgit script aswell

@mkg20001 do you know in which step this is happening?
Is this on mix deps.get? Maybe it's just a matter of passing a flag to it? https://hexdocs.pm/mix/Mix.Tasks.Deps.Get.html#content
For example --no-archives-check ?
Is this on mix deps.compile?
Here is the list of flags
https://hexdocs.pm/mix/Mix.Tasks.Compile.html
perhaps we also have to do the archives check here?

@minijackson @DianaOlympos I have updated the docs to build the frontend dependencies in a separate derivation.
The example I put is using node2nix, I would be happy to include any other way you use to build frontend dependencies.
The way I understand it, the last thing left to solve is the git dependency problem.
@mkg20001 Do you have an example of a dependency i can test this with?
I will be happy to add it to the nix_demo repo so everybody can see.

@happysalada mix.loadPaths. You can test with https://github.com/mkg20001/hubs-flake
Just change defaultPackage to .reticulum, change the nixpkgs input so it points to your branch and run "nix build" using the nixFlakes version of nix

@mkg20001 I think I have got it this time.
I used the following file to test
with import ../nixpkgs { };
let
  packages = beam.packagesWith beam.interpreters.erlang;
  src = builtins.fetchGit {
    url =<EMAIL_ADDRESS>    rev = "5f216106aef34a1953a5e9b90f09ee096f4d5e01";
  };

in
packages.buildMix {
  name = "reticulum";
  version = "0.0.1";

  mixEnv = "prod";
  depsSha256 = "sha256-EmVS+vry0cg1N/ODLtJ27t/OnJr2Ock3B9PapGempLA=";
  inherit src;
  buildEnvVars = {
    DATABASE_URL = "";
    SECRET_KEY_BASE = "";
  };
  defaultEnvVars = false;
  nativeBuildInputs = lib.optionals stdenv.isDarwin [ darwin.apple_sdk.frameworks.CoreServices ];
}

The commit I used is the latest master commit. Please not that it doesn't compile because of a compile error in their code.
If you find a commit that works, it should work.
I don't think you need the buildEnvVars, I just copied and pasted.
If anybody has some feedback, I would be curious. I think we should be able to merge after one last round of review.
@hauleth Thanks a ton again. This PR has your code all over.

@yurrriq I think this is ready, I would like your feedback when you have a moment

@yurrriq I think this is ready, I would like your feedback when you have a moment

I've added some inline suggestions for now. I'll plan to take a deep dive this weekend.

@yurrriq Thanks!

@Mic92 sorry to bother you with this, I personally think this is ready to merge. When you have a moment, do you think you could have a look?

Sorry for beeing a bit more picky here but designing new builders is something that affects potentially may packages and refactorings are more expensive. I would like to discuss https://github.com/NixOS/nixpkgs/pull/112477#discussion_r589191070 more

@Mic92 thanks a lot for taking a look so quickly!
I'm honestly just looking for feedback, so your comments are really welcome!

@Mic92 I am now passing the deps to buildMix, let me know if that is you had in mind. I updated the docs as well to reflect the change.
I updated the build.nix in the nix_demo repo to take into account the new architecture.
@mkg20001 I added a build_reticulum.nix to the nix_demo repo in case you want to test with the new setup.
Always interested if you think there are ways to make this better.

I'll plan to take a deep dive this weekend.

Didn't end up finding the time... can try some evening this week, but from just reading, this is looking pretty good to me.

@yurrriq no worries! I'll wait another week before sending a reminder to everyone.
Thanks for having had a look!

Also worthy of note is that the Erlang bytecode was not 10% reproducible, I'm still inspecting it to see why, but to my knowledge, impurity in VM bytecode is not new in NixOS and shouldn't matter that much for us.

@minijackson that mobilizon module is looking great!
The elixir formating thing looks amazing. You should definitely open a PR for it.
Here are my comments.

I like where you are going with runtime.exs. However I personally use it with environment variables. I'm not sure why mobilizon is not using it like that, but I think it would be more common to have env vars in there (not sure though). I don't know if many packages will actually require to build a runtime.exs. I would like some more opinions on that. (it doesn't change the fact that your elixir format thing is great and nixpkgs should have it!)
You are right about the COOKIE.
You are right about the bat file.

I'm not sure I understand your point about copying the wrapper and pointing to the wrong location. The wrapper has an absolute path /tmp so it shouldn't point to a different location?

I've removed the RELEASE_TMP wrapper anyways. If we need to provide another env var (RELEASE_COOKIE) that cannot be wrapped, we might as well get rid of it completely.
Do we really need to remove the .bat file though? On building I always get
* skipping elixir.bat for windows (bin/elixir.bat not found in the Elixir installation)
* skipping iex.bat for windows (bin/iex.bat not found in the Elixir installation)

As far as I am concerned, this is ready, let me know if I have forgotten something

I like where you are going with runtime.exs. However I personally use it with environment variables. I'm not sure why mobilizon is not using it like that, but I think it would be more common to have env vars in there (not sure though). I don't know if many packages will actually require to build a runtime.exs. I would like some more opinions on that. (it doesn't change the fact that your elixir format thing is great and nixpkgs should have it!)

Thank you ^^
The reason that I'm not going with environment variables for Mobilizon, is that it uses Application.get_env to get its configuration, and to get it to work with environment variables, either:
The library needs to specify an environment variable as fallback with something like this:
Application.get_env(:app, :root_key)
|> Keyword.get(:key, System.get_env("ENV_FALLBACK", "final fallback"))

but not every library does this for every configuration option.
Or, we can specify in the configuration (e.g. config/runtime.exs) that we want to configure each key with an environment variable like this:
config :app, :root_key,
  key1: System.get_env("APP_KEY1", "final fallback 1"),
  key2: System.get_env("APP_KEY2", "final fallback 2")

But Mobilizon itself as a lot of configuration options, and this is not even referencing configuration options from its 134 dependencies, so this does not seems feasible either.
I might have missed something, though.

I'm not sure I understand your point about copying the wrapper and pointing to the wrong location. The wrapper has an absolute path /tmp so it shouldn't point to a different location?

The issue wasn't about the /tmp variable, but with the path of the executable itself. If the derivation of buildMix is built under /nix/store/aaaaaaaa-package-0.0.1/ and if we kept the wrapProgram the wrapper would look something like this:
#! /nix/store/.../bin/bash -e
export RELEASE_TMP=${RELEASE_TMP-'/tmp'}
exec -a "$0" "/nix/store/aaaaaaaa-package-0.0.1/bin/.package-wrapped"  "$@"

Then, I wanted to override the runtime.exs so I copied every file onto /nix/store/bbbbbbbb-package-with-config-0.0.1/ and modified the runtime.exs. But the wrapper stayed the same and still called /nix/store/aaaaaaaa-package-0.0.1/bin/.package-wrapped, and it leads the package to load the empty runtime.exs that was in the previous aaaaaaaa store path.

Thanks for the explanation about the wrapProgram !
My understanding is that env vars are handled  in the second way you described. The only problem I see is that if those environment variables need to be secret, then we need to at least document how those should be handled (i.e. not in the nix store).

The only problem I see is that if those environment variables need to be secret, then we need to at least document how those should be handled (i.e. not in the nix store).

Yes! This should be possible with the elixirConf format that I'm proposing (see here for an example). Haven't gotten around to writing the documentation yet ^^

Just removed the reference to the release cookie and added a TODO to document how to properly define it.

Are we not touching the config/runtime.exs file by default then? I do not mind either way, just checking if it just went under the radar or not.

Hi.
My thoughts are still, shouldn't this release thing finally be called releaseMix or simliar?
For buildMix I would only admire to get a build only (example for building a project / dependency used as a library).
But I'm very very happy to see so much progress in here.
Thanks everybody. Thanks Raphael.

@minijackson sorry, I don't understand what you mean by touching runtime.exs by default.
@cw789 I think you're right on this one. What do we do about the old buildMix broken thing, do we just leave it and mark it as broken?

I does make sense to me to call this new builder releaseMix or buildMixRelease. It may be out of scope, but @cw789 can you give more detail about what you would want for buildMix? If I recall correctly, there was some discussions about packaging individual Elixir libraries, but it seems the consensus was that it was not feasible.
@happysalada during my testing, I discovered that elixir releases doesn't read from the runtime.exs unless one was present during the build phase. My solution was to create an empty one in preBuild. If we want to favor using the elixir conf format for nixos modules, it might be nice to do it here automatically, but if we do not want to favor any configuration method by default, then feel free to skip this ^^

@minijackson Thanks for the clarification. I definitely thing we should encourage people to configure their app with nix. However what about the others? Do we want to add a runtime.exs by default? I don't see any downsides but it wouldn't be needed for most right? That would tend me to leave it to overriding. Perhaps we should add a note in the documentation about it once your PR is merged?

@happysalada are you talking about end users using Nix to build an Elixir release, but not using NixOS to deploy it? I admit this possibility completely slipped my mind 😅
To my knowledge, an empty runtime.exs doesn't do anything if it stays empty. We could also add a runtimeConfigSupport option to releaseMix.

Naming
I would name it mixRelease as it is the same output you will get with mix release.
But something like buildMixRelease is also fine. Or elixirMixRelease.
I just won't keep the name buildMix as it is not the same output you will get with mix build.
Old deprecated broken buildMix
I'm a cruel person. If it is broken and nobody cares, why should we keep it?
New mixBuild
Just theoretical now. I followed the discussions.
If it would exist again I would assume it to do what mix build does.
But in my usecases not to package Elixir libraries but to use it whithin CI and archive excellent chaching (no rebuild if nothing changed).
In a perfect world no rebuild of dependency X if depenency did't changed.
But then again we are at the topic how to convert mix.lock dependencies to Nixpkg builds.
Built releases not deployed on NixOS
Of course this is a use case! But this will really shine, when we will archive to make Erlang cross compilation a polished thing (where DianaOlympus is working on).

I removed the buildMix (I feel strongly about this, but open to change).
I renamed the tool to mixRelease, I quite like the name. Anybody is free to add his opinion on this.

I tested the adding touch config/runtime.exs.
Here is the problem I ran into
warning: both "config/runtime.exs" and "config/releases.exs" have been found, but only "config/runtime.exs" will be used

I am actually still using the deprecated releases.exs
(see ref )
I had no idea it was deprecated before I read this.
For developpers who haven't made the migration, the behavior is unwanted, since their releases.exs won't be used.
I will make the migration to runtime.exs, however for other people that haven't done it, unless there is a hard deprecation, I don't think we should add a runtime.exs ourselves.
My opinion is that we should leave that to a preBuild overriding.
Let me know of course if you see flaws in my reasoning.
PS: That should be the last issue, let me know if I forgot something.

@cw789 I agree with you that the new name is much better, and quite like the 1-to-1 relation with mix commands.
On the topic of the new mixBuild, I think we should open a new issue, or continue discussing on #105002, but my personal opinion is that since Elixir libraries are allowed to do basically anything at compile time, having built libraries package in nixpkgs is not feasible without maintenance work for each library, and deep understanding of what each library does at compile-time. The mix.lock issue of not being stable can be circumvented since it is a central point of the ecosystem, but maintaining each library seems harder unless we have a maintainer for each one, like we have for C libraries.

For those interested. I recently discovered pnpm
https://pnpm.js.org/
it stores js packages in a content addressable store. For those who are not yet settled on the best way to build frontend dependencies it might be a good try. I will give it a go as soon as this is merged.

@Mic92 Can you take another look when you have a moment? I think this is ready.

@Mic92 @yurrriq sorry to bother you with this. Can you have a look when you have a moment? I think this is ready.

Can you have a look when you have a moment?

I'll plan to review Sunday evening CDT, if no one beats me to it.

@SuperSandro2000 @yurrriq thanks a lot for taking the time to review!

@SuperSandro2000 @yurrriq Is it possible to merge this for now and try to figure out the private git deps later?
@yurrriq Could you give more information of what that dependency looks like in the mix.exs? I've never seen a mix project with private git deps. Also, can you try running it with the debug flag set to true, to see if you get more informations? Or Maybe that was the output with the debug flag/

@Mic92 sorry to bother you with this. If you have some time, would you mind taking a look. I think this is ready to merge. There are some things that will need to be improved, but I think it is important to merge so that lots of software packaged with elixir can be added to nixos.

@yurrriq sorry, I just realized I missed some of your suggestions!

@NobbZ @yurrriq @SuperSandro2000 I think it's ready for another round of review. Let me know if I forgot anything of course.
Thanks again for having spent time on this.

@SuperSandro2000 @Mic92 sorry to bother you with this. I don't know who the best person to ask is. I think this is ready to merge. If you have a moment, please take a look.

Do we want to squash merge this?

nixpkgs-review or ofborg is not building any of those packages by default. I think we need recurseIntoAttrs somewhere but I am not sure where. Maybe https://github.com/NixOS/nixpkgs/pull/115246/files#diff-47bba45b2355a731c60917b1df71d696c8faf13cd40ca1415b4669470685d408R1472 can help you a bit?

This is awesome @happysalada! I've managed to use it to (sloppily) get elixir-ls packaged! It is kind of a special case, as it relies on some custom release tasks...
For your own testing purposes, my derivation is below:
        erlangPackages.mixRelease {
          pname = "elixir-ls";
          version = "v0.6.5";
          src = pkgs.fetchFromGitHub {
            owner = "elixir-lsp";
            repo = "elixir-ls";
            rev = "v0.6.5";
            sha256 = "sha256-3M0M/L2m9P9YYngCHcQBhnRPgr23Esj1Foyl2kQDJ2Y=";
          };
          installPhase = ''
            runHook preInstall
            mkdir -p $out/bin
            mix do compile --no-deps-check, elixir_ls.release -o "$out/bin"

            touch $out/bin/elixir-ls.bat

            mkdir -p $out/releases
            touch $out/releases/COOKIE

            mv "$out/bin/language_server.sh" "$out/bin/elixir-ls"
            runHook postInstall
          '';
          mixDeps = erlangPackages.fetchMixDeps {
            pname = "elixir-ls-deps";
            version = "v0.6.5";
            src = pkgs.fetchFromGitHub {
              owner = "elixir-lsp";
              repo = "elixir-ls";
              rev = "v0.6.5";
              sha256 = "sha256-3M0M/L2m9P9YYngCHcQBhnRPgr23Esj1Foyl2kQDJ2Y=";
            };
            sha256 = "fXdo/hKEMJVqZGbKgP8bcCsmlQT/4lO9YM5AdQo7G08=";
          };
        }

nixpkgs-review or ofborg is not building any of those packages by default. I think we need recurseIntoAttrs somewhere but I am not sure where. Maybe https://github.com/NixOS/nixpkgs/pull/115246/files#diff-47bba45b2355a731c60917b1df71d696c8faf13cd40ca1415b4669470685d408R1472 can help you a bit?

My understanding of nix might be lacking here, so forgive me if I say something that doesn't make sense at all.
In this PR, no packages are being touched right? Those are modules, or functions that would help you build packages. Unless I'm wrong, nothing needs to be built on hydra. Just in the future, new packages will use this function to build, and those will need to be built on hydra.
Let me know if I'm making sense.

Let me know if I'm making sense.

Oh, maybe you are right. I just assumed there are rebuilds.

Do we want to squash merge this?

I've squashed into 2 commits
one for the change in format, and one for the rest.
Let me know if that works.
@yurrriq in the process, I've lost your commit attributions. I hope you don't mind.

@SuperSandro2000 let me know if you need anything else.

/marvin opt-in
/status awaiting_merger

/status awaiting_reviewer

@SuperSandro2000 Thanks a lot for spending time on this!
I've taken care about the comments that I could take care of and I've responded to the other comments.
Please take a look when you have a moment.
Thanks again!

@SuperSandro2000 I've taken care of the last comment. (I should have thought about adding it before).
It should be ready now.
Let me know if anything.
Thanks again for your patience with this whole PR.

I wonder how we can get this on https://github.com/orgs/NixOS/projects/6#card-56314431

@NobbZ great idea! I think it's a matter of asking the release manager right?
Let me know if you want to have a go at it, otherwise I don't mind taking care of it.
if it doesn't make it by the time the release is made, I can probably backport it. I've never done it before, but I've been wanting to try for some time.

@balsoft I believe this is ready to merge. No worries at all if you don't want to take the responsability or don't have the time to invest in this.

@balsoft ❤ thanks a lot ❤
@SuperSandro2000 @minijackson @cw789 @NobbZ @schneebyte @afontaine @kalbasit @hauleth
thanks a lot for the help on this!

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/announcing-mixnix-build-elixir-projects-with-nix/2444/20


Import/Export Rules

We need to re-implement the ability to save and load blawx rules to a file.

Implemented in add_users branch


Licensing unclear

Hi, the licensing of this bot is not clear. The LICENSE file in the repo tells that it is under Apache 2.0 license, but the package.json says it's under ISC license. Which one is the right one?
In case you don't know, you can also license it under both so the user can follow their favorite.
In case you need help, just ask me, I will do a PR with the requested modifications.

Hey. Good catch. The license within the package json is probably outdated. The correct one is Apache. Feel free to submit a PR request.

As you can see above, I just made the PR. Don't hesitate if there's anything!


STORM-762: uptime for worker heartbeats is lost when converted to thrift

I made uptime_secs required because this is a bug where falling back to broken behavior does not feel right, and the current code has not been officially released yet.  If you want me to make it able to do a rolling upgrade I can without too much difficulty.

There are some unchanged files in the generated packages that should be removed from the commit.  I  think given we have not released this version yet it is ok to make this required field.

Some of the python files had #/bin/env python added at the beginning.  This was part of the script that generated them.  I left them in, because it looked like they were dropped out somehow.  Not totally sure how.  If you want me to remove them I can, but then every time someone generates them again using the current scripts they will show up again.

Ok, I think we can keep them as is. I am +1 for the change and thanks for catching this.


Management of non-Data Dictionary master metadata: Data Dictionary for Care (SNOMED CT refsets)

04/08/2020 - Moved to blocked sure resource availability.

Epic: https://nhsd-jira.digital.nhs.uk/browse/DPNDA-165
Discovery :
Initial Plan:
DD4C Current situation Ticket #1
Describe what DD4C is
Describe the current technologies and platforms it uses
Describe the current costs
Describe future developments (none due to cost?)
Describe the Technical Debt issues
Remediation Ticket #2
Options

Do Nothing - unacceptable to EAB - may result in service closure. Assumption this isn't what the customers wants but assumption needs to be tested.
Uplift/refresh - Ticket #3 Describe what needs to be done, how long it will take and any risk/issues. Technical debt will continue to resurface. Run and maintenance costs will continue along the current trajectory
Incorporate DD4C into Mauro
Ticket #4 Describe Mauro in terms the DD4C customer will understand and how current DD4C functionality will be accommodated to achieve their buy in
Will need to understand platform and maintenance costs and how or if they will be attributed to customers
Will need to understand dependencies with DD and other systems and Workflow (Cath and Michael?)
Will need to bring in as wider group of Data Products people as possible to develop and share Mauro skillset
Will need to understand Migration issues and develop a migration plan Ticket #5
Ticket #6 Build a light demonstration?
Ticket #7 explore and describe future opportunities/benefits for DD4C

Present the above to Nicholas and DD4C owners/users for final decision/funding
Ticket 8 - Implement the decision

12/5 - still blocked, awaiting stable instance of Mauro.  Team also do not have capacity.

23/6 - Update from BC:  Meeting required between Ali and Nicholas to review the design for data migration of DD4C in NHSD Mauro design

Meeting b/w Ali, Nicholas and Paul happened. Now Ali (@BobCox-afk) needs to review and arrange review sessions with Cath and Angela #279

Closed, see https://github.com/NHSDigital/DataDictionaryPublication/issues/279


Bug fixes for getResemblePath and resource errors in debug mode

Contains a fix for Huddle/grunt-phantomflow#11 and half of the fix for Huddle/PhantomFlow#46 (the other half of the fix is in pull request Huddle/PhantomFlow#47).

Thanks


Import auto-scroll update from Icon-O-Matic that use system auto-scroll

In hrev57439 auto-scroll was re-introduced on BListView's. Unfortunately this conflicts with the auto-scroll feature in Icon-O-Matic and WonderBrush which are using GetMouse() to do their own auto-scroll. So instead, I have taken the auto-scroll feature out of Icon-O-Matic here https://review.haiku-os.org/c/haiku/+/7215 and would like to do the same in WonderBrush. This alters WonderBrush\ to use the BListView version of auto-scroll instead.

Hi! Thanks for the patch. To be honest, I don't think it is good to make changes in Haiku which /require/ patches to applications to make them work again. Instead, the changes to be compatible and applications should be allowed to "opt in". From what I understand, Icon-O-Matic was now crashing and instead of fixing libbe, Icon-O-Matic was changed instead. Please correct me if I misunderstood things.

Fixed in upstream in hrev57966. The similar fix I made in Icon-O-Matic can stay as it fixes some other issues like redrawing the drop indicator correctly. However that's not a problem on WonderBrush and the crash is now fixed for R1B5.


Increase number of person dectected

Hello good day, how can I increase the number of person detected?. I tried to change the max_humans=10 in utils/tracker.py but it will only detect up to 5 only. Thanks

Did you change here? https://github.com/felixchenfy/Realtime-Action-Recognition/blob/master/utils/lib_tracker.py#L26
That doesn't make sense... It should work


Instance type label in Nodes tab

It would be great to have the instance-type label in the Nodes tab. This will help when you navigate between multiple cluster environments where you have different instance types.
node.kubernetes.io/instance-type: "m3.medium" - https://kubernetes.io/docs/reference/labels-annotations-taints/#nodekubernetesioinstance-type

How does kubectl show this today?

Not by default. Only possible via custom columns
-o custom-columns=type:"metadata.labels.beta\.kubernetes\.io/instance-type"

I'm bit hesitant to add features that are very unique in demand and not
supported by default in Kubectl. I believe this feature would be useful
only when you use a cloud provider. Let me think about it.
Thanks & Regards,
Deepu
On Wed, Jul 27, 2022 at 11:49 AM Shinu Suresh @.***>
wrote:

Not by default. Only possible via custom columns
-o custom-columns=type:"metadata.labels.beta.kubernetes.io/instance-type"
—
Reply to this email directly, view it on GitHub
https://github.com/kdash-rs/kdash/issues/223#issuecomment-1196511830,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAIOKF6RCXUR5A2MABTMK5LVWEA33ANCNFSM53O4RG4Q
.
You are receiving this because you commented.Message ID:
@.***>

Understood.
Would it make sense to have a toggle columns feature around?

Could you explain? what do you mean by toggle column?
Thanks & Regards,
Deepu
On Sat, Jul 30, 2022 at 6:27 PM Shinu Suresh @.***>
wrote:

Understood.
Would it make sense to have a toggle columns feature around?
—
Reply to this email directly, view it on GitHub
https://github.com/kdash-rs/kdash/issues/223#issuecomment-1200247107,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAIOKF4TPD6J47B4EZKMDHDVWVJXNANCNFSM53O4RG4Q
.
You are receiving this because you commented.Message ID:
@.***>

Something like this in a typical aws console

Thats gonna make the implementation and UI quite complex TBH so I don't
think that fits the goal of KDash which is to provide a fast and simple
alternative for kubectl with good UX
On Wed, 3 Aug 2022, 9:48 pm Shinu Suresh, @.***> wrote:

Something like this in a typical aws console
[image: image]
https://user-images.githubusercontent.com/2842927/182696151-649b69e3-d1bf-4d7d-b893-d4516cdf6c6d.png
—
Reply to this email directly, view it on GitHub
https://github.com/kdash-rs/kdash/issues/223#issuecomment-1204403882,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAIOKF6MT64NBI7MFSD4GLTVXLEHFANCNFSM53O4RG4Q
.
You are receiving this because you commented.Message ID:
@.***>

Ok. No worries.
On Thu, Aug 4, 2022, 9:24 AM Deepu K Sasidharan @.***>
wrote:

Thats gonna make the implementation and UI quite complex TBH so I don't
think that fits the goal of KDash which is to provide a fast and simple
alternative for kubectl with good UX
On Wed, 3 Aug 2022, 9:48 pm Shinu Suresh, @.***> wrote:

Something like this in a typical aws console
[image: image]
<
https://user-images.githubusercontent.com/2842927/182696151-649b69e3-d1bf-4d7d-b893-d4516cdf6c6d.png
—
Reply to this email directly, view it on GitHub
https://github.com/kdash-rs/kdash/issues/223#issuecomment-1204403882,
or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/AAIOKF6MT64NBI7MFSD4GLTVXLEHFANCNFSM53O4RG4Q
.
You are receiving this because you commented.Message ID:
@.***>

—
Reply to this email directly, view it on GitHub
https://github.com/kdash-rs/kdash/issues/223#issuecomment-1204929434,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAVWCLZ2HUL45PDGH7K6G3DVXN4Z7ANCNFSM53O4RG4Q
.
You are receiving this because you authored the thread.Message ID:
@.***>


FIX: get example data from cproof

@callumrollo Does this seem reasonable?  We can just have a static place to keep some example data.  It need not be in sync with the test data, though we could make it so periodically.


CONSOLE 3096: Promote ConsolePlugin CRD to v1

Story: https://issues.redhat.com/browse/CONSOLE-3069
/assign @spadgett

Just noticed as well that this PR is missing the CRD with generated schema, I'm not sure how we manage the CRDs across versions but I would expect to see a merged CRD with the schema of both versions somewhere in this repository

@JoelSpeed I've addressed your comments. PTAL

will generate run necessary generation script after we align on the v1 structure

@JoelSpeed @spadgett I've addressed your comments. PTAL :)

Ok I think this is good to go now, please make sure to resolve the verify issue and then squash the commits into something logical and clean.
/hold cancel

/test verify

Hmm, it would seem the verify job can't handle having both versions of the schemas generated, we should fix that (CC @deads2k)

Hmm, it would seem the verify job can't handle having both versions of the schemas generated, we should fix that (CC @deads2k)

Ok this is a path issue, I think we can adjust the makefile to set the paths for the generation correctly to resolve this, swapping console/v1 to console/...
$(call add-crd-gen,console,./console/...,./console/v1,./console/v1)
$(call add-crd-gen,console-alpha,./console/...,./console/v1alpha1,./console/v1alpha1)

Verify job broken here, please run the update-scripts to make sure this is up to date

@JoelSpeed there was a conflict due to which I had to rebase and push.

QE Approver:

/assign @yapei
Docs Approver:

/assign @opayne1
PX Approver:

/assign @RickJWagner

/label px-approved

/label docs-approved

/lgtm

/label qe-approved

/label qe-approved

/approve


Fade in...

Hi, is there way to stop the fade in when loading the website?

Is there anyone who can follow this up?


Send e-mail with questions about any tags

The user could register the tags that he would answer questions about and frequency to receive e-mail.

Did anyone spend some thoughts on the functional requirements yet?

where would we place a button to subscribe to a tag?
how would a user unsubscribe from a tag?
which actions would generate an email (apart from creating a question obviously - what about during "edit question" when a tag is added or what about answers/comments to a question)?


feat: Parse dataview compatible dates

Description
Adds the ability of using the following prefixes instead of emoji:

start::
scheduled::
due::
completion::
recur::

These are the same that are supported by the dataview plugin: https://blacksmithgu.github.io/obsidian-dataview/data-annotation/#tasks
Note: The last one is not found in dataview, I made that one up

Motivation and Context
Coming from zim's tasklist plugin, I missed the ability to fully type out a task without having to type special characters or wait for intellisense to pop up - both of which I feel like break my 'flow' of typing. I quite like zims format of using > for start dates and < for due dates (e.g. - [ ] do the thing > 2022-10-10 < 2022-10-31) but suspected that this would produce a lot of parsing errors / difficulties.
Through searching Issues & Discussions, I found a couple of threads mentioning dataviews tasks and one where it was mentioned, that parsing them would be implemented at some point (https://github.com/obsidian-tasks-group/obsidian-tasks/discussions/373 / https://forum.obsidian.md/t/task-management-devs-define-a-shared-date-and-action-format/26464/78)

How has this been tested?

I implemented a unit test for this
Screenshots (if appropriate)
not applicable
Types of changes

Changes visible to users:

[ ] Bug fix (prefix: fix - non-breaking change which fixes an issue)
[x] New feature (prefix: feat - non-breaking change which adds functionality)
[ ] Breaking change (prefix: feat!! or fix!! - fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation (prefix: docs - improvements to any documentation content)
[ ] Sample vault (prefix: vault - improvements to the Tasks-Demo sample vault)

Internal changes:

[ ] Refactor (prefix: refactor - non-breaking change which only improves the design or structure of existing code, and making no changes to its external behaviour)
[ ] Tests (prefix: test - additions and improvements to unit tests and the smoke tests)
[ ] Infrastructure (prefix: chore - examples include GitHub Actions, issue templates)

Checklist

[x] My code follows the code style of this project and passes yarn run lint.
[x] My change requires a change to the documentation.
[x] I have updated the documentation accordingly.
[x] My change has adequate Unit Test coverage.

Terms

[x] My contribution follow this project's contributing guide
[x] I agree to follow this project's Code of Conduct

Notes:

I am not 100% happy with the docs I added, maybe someone else has a better idea for this. Maybe just make an 'alternative formatting' page?
Thank you for having extensive tests for this code base. It made it very easy to judge the extent of a change even when I was completely new to the project.

Hi @j6s, Many thanks for this. I am sure it will be a very popular addition.
I am not currently able to try it out, so will have some more comments when I can.
From reading the code, I am wondering:
Format parsed
It will be important that the Tasks behaviour matches, as much as possible, the dataview behaviour on the save markdown content.
Have you compared the results of your new version of Tasks with the results from the equivalent dataview searches?
I ask because the format you use is different from the documented dataview format: https://blacksmithgu.github.io/obsidian-dataview/data-annotation/#tasks. It uses square bracket around values. I think it also allows some other syntax, perhaps parentheses around inline fields.
So in your test cases, I actually think Tasks should not recognise any of the example dates, because none of them have [] so are not valid dataview inline fields, as far as I understand.
Behaviour
What happens when you complete a Task with dataview fields? I presume you get an emoji-based completion date.
Do you think this is OK? Do you think that it should at least be documented?
What happens when you use the Add or edit Task modal to change one of these values? I suspect the task would be re-written with emojis.
What happens when you view these tasks in Live Preview or Reading mode. I wonder whether the Tasks rendering code will re-draw them with emojis rather than the dataview-like format that is currently parsed.

There is an old, partial PR - #384 - that might be worth looking at, to see if it has any code that would help with this PR, or if it could be picked up and finished off.

So in your test cases, I actually think Tasks should not recognise any of the example dates, because none of them have [] so are not valid dataview inline fields, as far as I understand.

I think it’s a bit more complicated than I first thought. I’m not totally sure of this, but from my experimenting it seems that the square brackets are not required for recognising fields in free text, but they are required for recognition in task lines.

Hi @j6s,
Thank you again for the contribution.
Just to update on the current status, I didn't pick this up at the time, because of the unanswered questions above...
However, now that we have an abstraction for reading and writing tasks in different formats - see #1728 - I expect that the best approach for this functionality would be to use that abstraction to fully define a new Dataview format for reading and writing tasks in Dataview format.
The questions above about when/if [...] are needed will remain, I suspect.


Record the author of each guide edition

Assign current user id as user_id on an edition. This is in order to keep track
of who's made changes to a guide over time. The latest editor is shown in the
list table of all guides.

Is :user validated in the model?

Thanks. Renamed the helper method and added a validation for user presence on Edition.

lgtm


Use of External Libraries

I am very excited that Cloudflare is hosting this task for summer interns. However, I think there is an important aspect of this task that should be clarified: are external libraries (outside of the standard libraries that come with the languages) explicitly allowed? If so, how will they be bundled with our source code?
Thanks!

just wanted to add:
for C/C++, could a header library like catch be used for testing?
iirc, go and rust both have testing as part of stdlib, while C/C++ folk would have to roll their own if they wanted to test

@sudopluto , feel free to use catch.

@Daniel-Liu-c0deb0t , whatever you submit should be easy to build so please do not include binaries.  You are allowed to use other libraries as long as they build as part of your submission.


naive bayes missing dataset

Hi joel, i figure it out on naive_bayes.py the data that you refer is not in this github. can you give me a link to download it?
if __name__ == "__main__":     #train_and_test_model(r"c:\spam\*\*") train_and_test_model(r"/home/joel/src/spam/*/*")
Thanks

ah you can download here http://spamassassin.apache.org/old/publiccorpus/


Particle2d randomness

Currently any randomness within the implementation doesn't work properly.
Any randomness would not take effect since the internal working of how seeds are produced for each particle made a too small of a seed for the random function to work properly.

A few things:

I don't think that xor is necessary or useful, and typically such constants for shift/multiplication need to be carefully chosen through statistical tests (so things like "0xdeadc0de" tend to be bad idea).
The original code uses multiplication because it scatters bits.
seed and random have different sizes (64-bit and 32-bit), meaning you'd need two random values to construct a seed.
pointers can be used as a seed, but again, a point on 32-bit systems is not large enough.

Also I think this PR can be simpler, because just changing the initialization of p.seed should solve this problem.
If you add a get_seed() in math_funcs.h which returns the value of the current seed, you can 1) use it directly for p.seed 2) add a randomize_from_seed(), and derive a new seed from the current seed as the initial value of p.seed.

Update just removes the seed per particle and uses a single seed for the entire system. There is no need for a separate seed per particle. It just uses the clock for initial seed.

random_seed = OS::get_singleton()->get_unix_time(); doesn't work well PCG64 (it causes issues the initial random value being 0 all the time), so can you please instead add the following function under Math and make use of it rather than using local custom random alhorithms:
uint64_t Math::randomize_from_seed(seed) {
	seed(OS::get_singleton()->get_ticks_usec() * seed + PCG_DEFAULT_INC_64);
}

There is no need for a separate seed per particle.

Why?

We don't need a seed per particle since we don't care about reproducing the random sequence.
if the PRNG works it should be indistinguishable to a casual observer from using multiple seeds.
It's a particle system not a sha1 key generator. It needs to be fast and light.

if the PRNG works it should be indistinguishable to a casual observer from using multiple seeds.

No, that's not how it works. PRNGs are simply long sequences of numbers. The only entropy may come from the seed. If that's shared, there will be correlations.

It's a particle system not a sha1 key generator.

This is why I'm not talking about hardware RNG or a cryptographic hash functions (of which SHA1 is a bad example anyway).

It needs to be fast and light.

I don't see how having different seeds will realistically cause any performance issues. You're degrading statistical quality for no real gains.

The current code uses uint64_t rand_seed = p.seed * (i + 1); to derive the new seed. It doesn't make use of clock but that is still preferable to this PR because this still has more entropy: it feeds some external information that is not a part of the PRNG's internal state (namely, i).

pcg32 is tested to produce high quality random numbers.
This means that as long as I keep advancing the seed for each call (which I do) I should get statistically random numbers.
Saying that you have to add entropy to the seed every few iterations makes no sense. The job of the PRNG is to produce random numbers. If that trick would make the PRNG produce better random numbers it would just be part of the algorithm.
An 8 byte per particle for a 512 particle is 4k extra data that adds nothing.
Just to make my point stronger I generated two histograms the left one uses a single seed the right one uses 100 different seeds. As you can see using a single seed is a better random source. This makes sense as the algorithm wasn't designed to have the different seeds produce statistically random different sequences in relation to each other but only for each sequence to be random within it's own stream.
What the test shows is that not only is the old method slower it produces less random numbers.
Finally, I would just like to repeat that even if the histograms were the other way around, it's were still random enough for the purpose of a particle system. But, as it stands using a single seed produces better random numbers takes less memory and less CPU.

OK, I don't have time to teach you statistics here, what a correlation function, entropy, randomness mean or spend time pin-pointing what's wrong your home-made statistical quality tests.
I don't agree with your PR because:

You're killing off all the sources of entropy, with will introduce correlations if analyzed with any proper standard randomness test suite.
The current code, which also uses different seeds for each, is not a performance bottleneck, so your trade-off isn't justified
Fixing the issue you mention in your original post is just a few lines of change, as I explained above.

You can't just dismiss what I'm saying. I have a CS degree and I know what I'm talking about.
pcg32() already produces strong enough random data by itself for non cryptographic purposes it passes all the tests. The only reason to add entropy to it to reduce it's predictability not distribution. Since we don't care about predictability only distribution there is no need for entropy. As long as the output is uniformly distributed it's good for these purposes.
Particle systems don't need unpredictable numbers, they need a source of uniformly distributed numbers. Correlation doesn't matter and predictability doesn't matter.
This is not optimization or tradeoff, it just removes code that does absolutely nothing that makes a difference. We loose literally nothing and make the code simpler, take less memory and take less CPU.
If I saw a loop calculating digits of PI I would remove it too even tough it's not a performance bottleneck.

In any case, this goes nowhere. Either you address my comments instead of being condescending or, if you are unable to, I'll do whatever @akien-mga, @Hinsbart or @reduz say since they seem to have the authority in the codebase.
I might be wrong, and I'm willing to accept that, but just saying that you are smarter doesn't make it so.

What? I'm not the one who's talking about credentials. But you really don't know what you're talking about, and no, they don't teach that stuff to undergrad or majority of the grad students in CS.
I already gave you an explanation above. Please read it properly.
I'm not in the mood to prove yet another perpetual engine wrong, so that's all I'll say.

@ficoos reduz is currently working on creating the new Particles2D system, so your changes would be overwritten anyway :( Thanks for the initiative though! 💙


VM sizes managed disk OS

I can't find a documented list of VM sizes available for managed disks. I'm trying to update my powershell script and need these value to ensure we are using the managed VM OS comparable to what we are already using.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ff5b5898-88d9-7aec-1516-e157e92564a7
Version Independent ID: f9807092-172c-d8d0-fc7d-17204faea51b
Content: Windows Virtual Machines Overview - Azure
Content Source: articles/virtual-machines/windows/overview.md
Service: virtual-machines-windows
GitHub Login: @cynthn
Microsoft Alias: cynthn

Thanks for the question! We are currently investigating and will update you shortly.

@Placid7 I couldn't find a single doc containing this inofrmation but as per the following docs:
https://docs.microsoft.com/en-us/azure/virtual-machines/windows/disks-standard-ssd
https://docs.microsoft.com/en-us/azure/virtual-machines/windows/standard-storage
Standard SSDs can be used with all Azure VMs, including the VM types that do not support Premium Disks.
Premium SSD Managed Disks are supported by DS-series, DSv2-series, FS-series, and GS-series VM sizes which are specifically targeted for Premium SSD Managed Disks.
Standard HDD disks can be attached to all Azure VMs including size-series VMs used with Premium Storage such as the DSv2 and GS series.


High variance in latency in speech-to-text with recognizeOnceAsync + Pronunciation Assessment

Describe the bug
We use the speech sdk in the browser to transcribe user-generated audio against reference text using the Pronunciation Assessment feature. For ~95% of the time, we are able to get a response in ~2 seconds, which is great. But for ~5% of the time, response time can be as long as ~20 seconds.
Extremely high response time seems to go away when Pronunciation Assessment is disabled.
I'm not sure if this issue is related to #1074 and #1124. Let me know if I can do anything else on my side to capture more debug information.
To Reproduce
const azureStt = require('microsoft-cognitiveservices-speech-sdk');
const speechConfig = azureStt.SpeechConfig.fromAuthorizationToken(...);
speechConfig.speechRecognitionLanguage = 'en-US';
speechConfig.requestWordLevelTimestamps();
speechConfig.outputFormat = azureStt.OutputFormat.Detailed;

const audioConfig = azureStt.AudioConfig.fromWavFileInput(wavFile);
let recognizer = new azureStt.SpeechRecognizer(speechConfig, audioConfig);

const pronunciationAssessmentConfig =
    new azureStt.PronunciationAssessmentConfig(
      referenceText,
      azureStt.PronunciationAssessmentGradingSystem.HundredMark,
      azureStt.PronunciationAssessmentGranularity.Phoneme,
      true /* EnableMiscue */,
);
pronunciationAssessmentConfig.applyTo(recognizer);

recognizer.recognizeOnceAsync(...);

Expected behavior
Consistent response time when using recognizeOnceAsync + Pronunciation Assessment
Version of the Cognitive Services Speech SDK
Tested with these SDK versions:
"microsoft-cognitiveservices-speech-sdk": "^1.18.0"
"microsoft-cognitiveservices-speech-sdk": "^1.19.0"
Platform, Operating System, and Programming Language

OS: macOS
Hardware: x64
Programming language: JavaScript
Browser: Chrome

Additional context
WebSocket messages from Chrome DevTools (note the "Time" field of the two highlighted messages):

Here's some details for more interesting messages in the stream:
[speech-config] (upstream)
Path: speech.config
X-RequestId: 5DF51D09A4B44239B7B5B4919663FCB8
X-Timestamp: 2021-11-23T02:00:59.743Z
Content-Type: application/json

{"context":{"system":{"name":"SpeechSDK","version":"1.19.0","build":"JavaScript","lang":"JavaScript"},"os":{"platform":"Browser/MacIntel","name":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.20 Safari/537.36","version":"5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.20 Safari/537.36"},"audio":{"source":{"bitspersample":16,"channelcount":1,"connectivity":"Unknown","manufacturer":"Speech SDK","model":"File","samplerate":16000,"type":"File"}}},"recognition":"interactive"}

[speech.context] (upstream)
Path: speech.context
X-RequestId: 2C20CAE7DCD04F98A60C2A313139BE37
X-Timestamp: 2021-11-23T02:00:59.743Z
Content-Type: application/json

{"phraseDetection":{"enrichment":{"pronunciationAssessment":{"referenceText":"My third story is about death.","gradingSystem":"HundredMark","granularity":"Phoneme","dimension":"Comprehensive","enableMiscue":true}}},"phraseOutput":{"detailed":{"options":["PronunciationAssessment","WordTimings","SNR"]},"format":"Detailed"}}

[turn.start] (downstream)
X-RequestId:2C20CAE7DCD04F98A60C2A313139BE37
Path:turn.start
Content-Type:application/json; charset=utf-8

{
  "context": {
    "serviceTag": "a726a8ce39db461085edff14194e64e9"
  }
}

[speech.startDetected] (downstream)
X-RequestId:2C20CAE7DCD04F98A60C2A313139BE37
Path:speech.startDetected
Content-Type:application/json; charset=utf-8

{"Offset":900000}

[telemetry] (upstream)
Path: telemetry
X-RequestId: 2C20CAE7DCD04F98A60C2A313139BE37
X-Timestamp: 2021-11-23T02:01:22.567Z
Content-Type: application/json

{"Metrics":[{"End":"2021-11-23T02:00:59.496Z","Name":"ListeningTrigger","Start":"2021-11-23T02:00:59.496Z"},{"End":"2021-11-23T02:01:22.566Z","Name":"Microphone","Start":"2021-11-23T02:00:59.565Z"},{"End":"2021-11-23T02:00:59.742Z","Id":"795FC1A610254DDB8A58CFC2E362ECA0","Name":"Connection","Start":"2021-11-23T02:00:59.497Z"},{"PhraseLatencyMs":[22999]},{"FirstHypothesisLatencyMs":[22452]}],"ReceivedMessages":{"turn.start":["2021-11-23T02:00:59.813Z"],"speech.startDetected":["2021-11-23T02:01:21.909Z"],"speech.hypothesis":["2021-11-23T02:01:22.017Z","2021-11-23T02:01:22.421Z","2021-11-23T02:01:22.422Z"],"speech.endDetected":["2021-11-23T02:01:22.437Z"],"speech.phrase":["2021-11-23T02:01:22.564Z"],"turn.end":["2021-11-23T02:01:22.566Z"]}}

Thanks @owenchu for reporting this issue, @yinhew could you help to take a look?

@owenchu Thanks for reporting this issue!
We've looked into this issue and partially understood the cause, but we still need more time to fully resolve it.
The issue only happened to Central US region per our investigation, you can switch to a different region like East US as a quick work around if it's doable for you.
We will update this thread if there is any progress.

Thanks for the update!
I just switched to West US. I'll report back if it happens again.

This issue seems to also happen in westus.
What are the fields in SpeechRecognitionResult that I can log on my side to help you debug the issue?

This somehow has worsened in our production environment. I've seen requests that took as long as 67 seconds.

@owenchu this looks a different issue. How many times did you hit this for?
Can you follow below sample code to print the session id, which can help us debug?
https://github.com/Azure-Samples/cognitive-services-speech-sdk/blob/master/samples/js/browser/index.html#L731

Here's some session ID / latency pairs:
973EF2551CFA494F9B7D3B27C379BF4C / 11,986 ms
56D8E35FB53A4A99B49E46326BA9A776 / 10,999 ms
929B7E3A15124368A549AF8571E24289 / 10,579 ms

Thanks for these session IDs! They helped a lot to my debugging.
Per debugging, the above cases were due to network transition issue, the audio chunks were uploaded using nearly 10 seconds.
So it was a different issue.
Where was the client invoking our speech service? Was it far from West US (California)?
Can you share me some cases with more significant latency when you hit?

Thanks for getting back to me, @yinhew!
The audio chunks were indeed sent from countries quite far from California. My use case involves directly streaming audio from the user's browser from all over the world. Is there a reliable way on the client side to find out the best endpoint to send audio over?

Here is our full region list:
https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/regions#speech-to-text-text-to-speech-and-translation
If you have CDN and distributed deployments of your website on different countries, you can point to different regions for different deployments.
If you don't have your backend server, you can try to roughly detect the user location by IP or language info in client code, and map to the nearest region in above list.

Thanks for the info! I'll give it a try.

Hi @owenchu, were you able to resolve your issue using @yinhew's suggestion?

The issue has been largely resolved, although I still see latency spikes at times. I'm not sure if it's because of slow network connections or code start issues of the API instances. That said, feel free to close the issue for now. :-)

Glad it was solved. Please open a new issue if you run into other problems. Thanks!


Shutdown Core3 before reboots?

Can we shutdown the server before linux reboots or shutsdown?

This is really about play servers, moved to https://github.com/Zonama/ZonamaPlay/projects/1


check_key_fields validation optimization for projects with heavily reused fragments

Summary
Codegen takes substantially longer in v3 compared to v2 in projects with heavily reused fragments due to check_key_fields validation.
Version
3.0.0-beta05
Description
Hello - I am in the process of migrating our project (which makes extensive use of nested and reused fragments) to Apollo 3, and noticed that the codegen task takes significantly longer for us than it does in v2. For comparison, codegen (excluding compilation) on our root module takes ~30s in v2 and ~3 min in v3.
To investigate, I cloned the Apollo compiler and found that most of the time was spent on the checkKeyFields call on fragments. Within check_key_fields.kt, I added a println and noticed that identical paths are being checked more than once. Am I correct in understanding that checkField should only be called once per unique path value (and that selections remain stable)? If so, could a set of checked paths be maintained so that duplicate calls are avoided? This change substantially reduces the time this class takes to run (our root module codegen comes down to 40s with this optimization) but I'm unsure if it breaks the completeness.

Thanks for sending this! We are monitoring the compile time but don't have any tests to detect such regressions at the moment. That's definitely something to add.

Am I correct in understanding that checkField should only be called once per unique path value

In CheckKeyFieldsScope.checkField, parentType might be important too, not sure.

This change substantially reduces the time this class takes to run (our root module codegen comes down to 40s with this optimization)

That's impressive 👏 ! Want to open a pull request and we can go from there?

Hi 👋
Trying to look at this, for now I couldn't reproduce a case where checkKeyFields takes significant time relatively to the rest of the compilation - albeit trying on various test cases, not a real project. Would you mind sharing how you singled out this method in your investigation? If possible it would be awesome if we could also have a look at your schema / operations (we understand if that's not possible). Thanks!

@apramana can you check with 3.0.0 and https://github.com/apollographql/apollo-kotlin/pull/3720 ? It has an optimization to disable this check early if not needed

@martinbonnin Awesome thanks for the patch, the build speed is much better now. For reference, I tested compilation of Apollo 3.0 GA in a multi-module project using compat codegen (though it probably doesn’t matter which one is used since this bug is in AST generation.)
Also, I’ve posted a PR for optimizing fragment spreads. Please take a look and provide any feedback.

All the thanks goes to @BoD who did all the work there :-)
Thanks for the PR, we'll take a look quickly !

@martinbonnin @BoD I noticed that Apollo React provides the ability to disable normalization for specific types. Does apollo-kotlin support this? (And if so, can this be used to exclude types from validation?)

@apramana we don't have anything like this at the moment. Feel free to open a separate issue to track this.

I'm closing this one in favor of https://github.com/apollographql/apollo-kotlin/issues/3966. As of 3.1.0, anyone not using @fieldPolicy or @typePolicy should never bump into this issue.
In the future, we'll want to optimize that part of the codegen but before we optimise, we'll need a measure of what we're optimizing, which is what the above issue is about.