WIP: Demonstrating Native Addons

Description
In helping solve the snapshot isolation problem in https://github.com/MatrixAI/js-db/pull/18, we needed to lift the hood and go into the C++ level of nodejs.
To do this, I need to have a demonstration of how native addons can be done in our demo lib here.
There are 2 ecosystems for building native addons:

prebuild
node-pre-gyp

Of the 2, the prebuild ecosystem is used by UTP and leveldb. So we will continue using that. Advantages from 2016 was commented here: https://github.com/prebuild/prebuild/issues/159
The basic idea is that Node supports a "NAPI" system that enables node applications to call into C++. So it's a the FFI system of NodeJS. It's also a bidirectional FFI as C++ code can call back into the NodeJS JS functions.
The core library is node-gyp. In the prebuild ecosystem is wrapped with node-gyp-build, which you'll notice is the one that we already using in this repo. The main feature here is the ability to supply prebuilt binaries instead of expecting the end-user to always compile from source.
Further details here: https://nodejs.github.io/node-addon-examples/build-tools/prebuild (it also compares it to node-pre-gyp).
The node-gyp-build has to be a dependency, not devDependencies, because it is used during runtime to automatically find the built shared-object/dynamic library and to load it.
It looks like this:
import nodeGypBuild from 'node-gyp-build';
const bindings = nodeGypBuild('./path/to/dir/containing/gyp/file');
bindings.someNativeFunction()

Internally nodeGypBuild ends up calling the require() function inside NodeJS. Which supports the ability to load *.node binaries (which is the shared-object that is compiled using the NAPI C++ headers). See: https://github.com/prebuild/node-gyp-build/blob/2e982977240368f8baed3975a0f3b048999af40e/index.js#L6
The require is supplied by the NodeJS runtime. If you execute the JS with a different runtime, they may support the commonjs standard, and thus understand the require calls, but they may be compatible with native modules that are compiled with NAPI headers.  It's all dynamic linking under the hood. This is also why you use node-gyp-build which automates some of this lookup procedure.
As a side-note about bundlers. Bundlers are often used part of the build process that targets web-platforms. Since the web platform does not understand require calls, bundlers will perform some sort of transclusion. This is also the case when ES6 import targets files on disk. Details on this process is here: https://github.com/evanw/esbuild/blob/master/docs/architecture.md#notes-about-linking. Bundlers will often call this "linking", and when targetting web-platforms, this is basically a form of static linking since JS running in browsers cannot load JS files from disk. This is also why in some cases, one should replace native addons with WASM instead, as bundlers can support static linking of WASM (which are cross-platform) into a web-bundle. But some native addons depend on OS features (like databases with persistence), and fundamentally cannot be converted into WASM binaries. In the future, our crypto code would make sense to turn into WASM binaries. But DB code is likely to always be native, as they have to be persistent. As the web develops can gains extra features, then eventually it may be possible that all native code can be done via WASM (but this may be a few years off).
Now the native module itself is just done with a C++ file like index.cpp. We should prefer using .cpp and .h as the most portable extensions.
Additionally, there must be binding.gyp file that looks like this:
{
  "targets": [{
    "target_name": "somename",
    "include_dirs": [
      "<!(node -e \"require('napi-macros')\")"
    ],
    "sources": [ "./index.cpp" ]
  }]
}

Basically another configuration file that configures node-gyp and how it should be compiling the C++ code. The target_name specifies the name of the addon file, so the output result will be somename.node. The sources are self-explanatory. The include_dirs entries have the ability to execute shell commands, in this case, it is using node -e to execute a script that will return some string that is a path to C++ headers that will be included during compilation.
The C++ code needs to use the NAPI headers, however there's a macro library that makes writing NAPI addons easier: https://github.com/hyperdivision/napi-macros. I've seen this used in the utp-native and classic-level.
The C++ code may look like this:
#include <node_api.h>
#include <napi-macros.h>

NAPI_METHOD(times_two) {
  NAPI_ARGV(1)
  NAPI_ARGV_INT32(number, 0)

  number *= 2;

  NAPI_RETURN_INT32(number)
}

NAPI_INIT() {
  NAPI_EXPORT_FUNCTION(times_two)
}

This ends up exporting a native module containing the times_two function that multiples a number by 2, and returns an int32 number.
It's also important that node-gyp-build is setup as a install script in the package.json:
  "scripts": {
    "install": "node-gyp-build"
  }

This means when you run npm install (which is used to install all the dependencies for a NPM package, or to install a specific NPM package), it will run the node-gyp-build durin the installation process.
This means that currently in our utils.nix node2nixDev expression still requires the npm install command. This used to exist, however I removed it during https://github.com/MatrixAI/TypeScript-Demo-Lib/pull/37 thinking it had no effect. But it was confirmed by https://github.com/svanderburg/node2nix/issues/293#issuecomment-1110567221 that the npm install command is still run in order to execute build scripts. And node-gyp-build is now part of the installation process. We should include: https://github.com/svanderburg/node2nix/blob/8264147f506dd2964f7ae615dea65bd13c73c0d0/nix/node-env.nix#L380-L387 with all the necessary flags and parameters too. We may be able to make it work if we hook our build command prior to npm install. I imagine that this should be possible since the npm rebuild command is executed prior. So we need to investigate this.
In order to make this all work, our Nix environment is going to need all the tools for source compilation. Now according to https://github.com/nodejs/node-gyp#on-unix we will need python3, make and gcc. Our shell.nix naturally has make and gcc because we are using pkgs.mkShell which must extend from stdenv.mkDerivation. However python3 will be needed as well.
The node2nix has some understanding of native dependencies (this is why it also brings in python in its generated derivation https://github.com/svanderburg/node2nix/issues/281), and I believe it doesn't actually build from source (except in some overridden dependencies).
Some npm dependencies are brought in via nixpkgs nodePackages because node2nix derivation isn't enough to build them (because they have complex native dependencies).  Such as node-gyp-build itself or vercel's pkg. This is also why I had to provide nodePackages.node-gyp-build in our buildInputs overrides in utils.nix. It is important that any dependencies acquired via nixpkgs must be the same version we use in our package.json. And this is the case for:
    "node-gyp-build": "4.4.0"
    "pkg": "5.6.0",

Ideally we won't need to do this our own native packages if js-db ends up forking classic-level or leveldown. I think this trick is only relevant in our "build tools" and not our runtime dependencies.
The remaining problem is cross-compilation, as this only enables building from source if you are on NixOS and/or using Nix. Windows and MacOS will require their own setup. Since our development environment is all Nix focused, we don't have to worry about those, but for end-users who may want to rebuild from scratch, they will need to setup their development environent based on information in https://github.com/nodejs/node-gyp. A more pressing question is how we in our Nix development environment will be capable of cross-platform native addons for distribution.
This is where the prebuild ecosystem comes in and in particular https://github.com/prebuild/prebuildify-cross. This is used in leveldb to enable them to build for different platforms, and then save these cross-compiled objects. These objects are then hosted on GitHub releases, and automatically downloaded upon installation for downstream users. In the case they are not downloadable, they are then built from source. https://github.com/Level/classic-level/blob/f4cabe9e6532a876f6b6c2412a94e8c10dc5641a/package.json#L21-L26
However in our Nix based environment, I wonder if we can avoid using docker to do cross compilation, and instead use Nix to provide all the tooling to do cross-compilation. We'll see how this plays out eventually.
Some additional convenience commands now:
# install the current package and install all its dependencies, and build them ALL from source
npm install --build-from-source
# install a specific dependency and build it from source
npm install classic-level --build-from-source
# runs npm build on current package and all dependencies, and also recompiles all C++ addons
npm rebuild
# runs npm build on current package and all dependencies, and specifically recompiles sqlite3 package which has C++ addon
npm rebuild --build-from-source=sqlite3

Issues Fixed

Related https://github.com/MatrixAI/js-polykey/issues/270
Related https://github.com/MatrixAI/js-polykey/issues/155
Related https://github.com/MatrixAI/js-polykey/issues/234
Related https://github.com/svanderburg/node2nix/issues/295
Related https://github.com/prebuild/prebuildify/pull/63 - this PR introduces the ability for optionalDependencies to contain platform-specific native addon packages, which the parent package relies on, this can avoid having to put all platform-specific native addons into a single prebuild directory that is installed from npm.
Related https://github.com/prebuild/node-gyp-build/pull/45 - this adds support for node-gyp-build to find the prebuilt binaries that come from optional dependencies
https://ownyourbits.com/2018/06/13/transparently-running-binaries-from-any-architecture-in-linux-with-qemu-and-binfmt_misc/ - When we want to deal with ARM
https://amir.rachum.com/blog/2016/09/17/shared-libraries/ - How runpath works.
https://github.com/NixOS/nixpkgs/issues/173025
https://developer.apple.com/documentation/apple-silicon/building-a-universal-macos-binary - How to build universal binaries

Tasks

[x] 1. Integrate node-gyp-build
[x] 2. Create a native module exporting a demo functions like addOne for primitives and setProperty for reference-passing procedure and makeArray for heap allocation
[x] 3. Fix the nix expressions to support node-gyp-build and other build scripts, and see if we can eliminate our postInstall hook, by relying on package.json hooks instead
[x] 4. Integrate prebuildify to precompile binaries and host them on our git release... but this depends on whether typescript-demo-lib is used as a library or as an application, if used as an application, then the pkg builds is used, if used as a library, then one must install the native binary from the same github release, this means the native binary must be part of the same release page.

The pkg integration may just be a matter of setting the assets path in package.json to the local prebuilds directory.
See the scripts that other projects used https://github.com/MatrixAI/TypeScript-Demo-Lib/pull/38#issuecomment-1120829967
Ensure that compiled native addons in nix have their rpath removed, because nodejs addons shouldn't have an rpath set, and this enables them to be portable

~[ ] 5. Cross compilation, prebuildify-cross or something else that uses Nix~ - we must use CI/CD to do cross compilation (not sure about other architectures like ARM)
[x] 6. Update the @typescript-eslint packages to match js-db to avoid the warning message.
[x] 7. Add typescript typings to a native module
~[ ] 8. Update README.md to indicate the 2 branches of typescript-demo-lib, the main and the native branch, where the native branch indicates how to build native addons~ - this will be done in a separate repo: https://github.com/MatrixAI/TypeScript-Demo-Lib-Native based off https://gitlab.com/MatrixAI/Employees/matrix-team/-/issues/8#note_885403611
[ ] 9. Migrate changes to https://github.com/MatrixAI/TypeScript-Demo-Lib-Native and request mac access to it the repository on gitlab. This branch will just be for development first. The changes here are too significant to keep within the same template repository.
[ ] 10. The pkg bundle can receive optimisation on which prebuild architectures it bundles, right now it bundles all architectures, when the target architecture implies only a single architecture is required. This can slim the final output pkg so it's not storing random unnecessary things. This may mean that pkg requires dynamic --config to be generated.
[ ] 11. See if nix-build ./release.nix -A application can be use prebuilds/ directory as well, as this can unify with pkg. That way all things can use prebuilds/ directory. But we would want to optimise it with task 10.
[ ] 12. Ensure that npm test can automatically run general tests, and platform-specific tests if detected on the relevant platform
[ ] 13. Automatic npm publish for prerelease and release based on staging and master branches

Future Tasks

Cross compilation for win-arm64, linux-arm64 (linux will require the necessary nix-shell environment)
Android and iOS builds for library
Consider using Nix directly on MacOS instead of homebrew, need to figure out how xcode works with nix
Use gitlab shared arm64 runners to do integration tests when it becomes available
If nodejs and vercel/pkg has universal binary become available, then use that instead...

Final checklist

[ ] Domain specific tests
[ ] Full tests
[ ] Updated inline-comment documentation
[ ] Lint fixed
[ ] Squash and rebased
[ ] Sanity check the final build

In the Node-API docs:

Node-API is a C API that ensures ABI stability across Node.js versions and different compiler levels. A C++ API can be easier to use. To support using C++, the project maintains a C++ wrapper module called node-addon-api. This wrapper provides an inlineable C++ API. Binaries built with node-addon-api will depend on the symbols for the Node-API C-based functions exported by Node.js. node-addon-api is a more efficient way to write code that calls Node-API. Take, for example, the following node-addon-api code. The first section shows the node-addon-api code and the second section shows what actually gets used in the addon.

So NAPI is actually a C-based API. Whereas node-addon-api is what enables you to write C++ instead.
That means leveldown and classic-level is just using C, and not C++. I'm guessing if we wanted to use C++, we would actually have to also bring in node-addon-api.
I wonder then why it's called binding.cc used in classic-level, and not just binding.c. Perhaps the reason is that although it is using NAPI's C API, the underlying code is still C++, and you only need node-addon-api if you wanted to have utilities to help with writing C++ native addons. So maybe you don't really need it.

Using vscode, there's some nice features that can be done with .vscode/c_cpp_properties.json. However it's not very automatic. I had to manually add the nodejs headers path there.
{
    "configurations": [
        {
            "name": "Linux",
            "includePath": [
                "${workspaceFolder}/**",
                "/nix/store/zl4bvsqfxyx5vn9bbhnrmbmpfvzqj4gd-nodejs-16.14.2/include/**"
            ],
            "defines": [],
            "compilerPath": "/nix/store/58pwclg9yr437h0pfgrnbd0jis8fqasd-gcc-wrapper-11.2.0/bin/gcc",
            "cStandard": "gnu17",
            "cppStandard": "gnu++17",
            "intelliSenseMode": "linux-gcc-x64"
        }
    ],
    "version": 4
}

Until this is figured out https://github.com/arrterian/nix-env-selector/issues/68, this is a project-by-project things.

Some additional findings:

The npm upstream package depends on node-gyp package.
The node-gyp package ends up being installed as part of the Node nix store path. Because nodejs brings in npm and npm brings in node-gyp.
The node-gyp binary is found by npm run scripts because it exists in /nix/store/...-nodejs-16.14.2/lib/node_modules/npm/bin/node-gyp-bin/node-gyp.
However the node-gyp executable itself is not found when in the nix-shell, this is because the the ${nodejs}/bin does not actually contain the node-gyp path.
This is why node-gyp-build works inside a run script, but not inside our nix-shell.
The node-gyp is overridden in nixpkgs and its expression is:node-gyp = super.node-gyp.override {
  nativeBuildInputs = [ pkgs.makeWrapper ];
  # Teach node-gyp to use nodejs headers locally rather that download them for>
  # This is important when build nodejs packages in sandbox.
  postInstall = ''
    wrapProgram "$out/bin/node-gyp" \
      --set npm_config_nodedir ${nodejs}
  '';
};

This means node-gyp if brought in via nixpkgs, already has the nodedir set correctly to the where the include headers are.
We will bring in node-gyp as dev dependency. And this will make our node-gyp-build actually run the node-gyp installed in our package.
However we will not prioritise the nixpkgs version in our $PATH for shell.nix as it is not relevant.
There's nothing special about node-gyp-build as a package. The only reason to bring it in from nixpkgs is to ensure that we are using the same version. However this can be enforced just by setting the version in the in the package.json to match what Nix provides, rather than overriding the dependency. But the same could be said for pkg. The only reason we are using utils.pkg and pkgs.node-gyp-build is that these are used inside the nix expressions to build things, and we just wanted ensure we had the same version in our nix-shell. Note that pkg has overrides applied, that we originally thought could be fixed, but didn't work last time.
Therefore, instead prioritising it in the $PATH, we can just use an npm install --save-dev --save-exact and npm install --save --save-exact commands to propagate our nix dependency versions in the package.json. As a bonus, the npm install also ends up installing all the other node modules.

The relevant lifecycle script is install. When npm install runs, it also runs the install script. By default if a binding.gyp file exists in the project root, it will run node-gyp rebuild. However since we have overridden it with node-gyp-build, that means it decides whether to run node-gyp rebuild or to download precompiled binaries. You can skip the check for precompiled binaries by doing node-gyp-build --build-from-source, or npm install --build-from-source.
A side note about npm rebuild. The npm rebuild has nothing to do with npm run rebuild. The rebuild script is just classic-level addition. The npm rebuild command actually rebuilds from source all dependencies. I believe this command is intended to be used when the node version itself has been updated, and you want to rebuild ALL native addons. However I believe most of the time you don't need to run this, as npm install --build-from-source should suffice.
A problem right now is that upon running our 2 npm install --save-dev --save ... npm install --save --save-exact ... commands it no longer runs the lifecycle install script. Only npm install itself runs the script.

Therefore if we only want to recompile our own C++ addon, then we must run node-gyp rebuild directly. It will force recompile our own native addon without doing anything else, even if the build has already been built.
Now this leaves a build/ directory in our project root. I'm not sure if this is configurable, but this does seem the build/ directory is then reserved for this usage. This directory may need to exist so that we can actually run tests, and run our distribution.
I may then link up this rebuilding call to the npm run build as well.

GTherefore if we only want to recompile our own C++ addon, then we must run node-gyp rebuild directly. It will force recompile our own native addon without doing anything else, even if the build has already been built.
Now this leaves a build/ directory in our project root. I'm not sure if this is configurable, but this does seem the build/ directory is then reserved for this usage. This directory may need to exist so that we can actually run tests, and run our distribution.
I may then link up this rebuilding call to the npm run build as well.

Because the build/ directory is auto generated by node-gyp configure. Then it can be ignored. And I notice this being done in classic-level too.
Now node-gyp-build is supposed to be used with prebuildify, so that would be the next thing I investigate. But now I need to test if might built native addon works for testing and bin execution.

The tests work and also running the executable works. Now to see how prebuildify works, and to see if npm install works for this as a library.

The changes here are likely to be quite significant, so it's going to be worthwhile to keep this as a separate top level branch as native. And just cherry pick common changes between the 2.

In order to ensure that our CI/CD during testing is using the same node-gyp structure, we have to change our nix-shell usage in the CI/CD to just run the normal shell.
I originally thought I could just bring in -p nodejs nodejs.python nodePackages.node-gyp, but the node-gyp getting executed by node-gyp-build still isn't aware of the nodedir setting. This means I've changed it to just running nix-shell so it can use the shellHook inside our shell.nix. Now it can stop downloading the nodejs headers since Nix already has it.

Something I've noticed that that deep strict equality doesn't work on arrays and objects returned from the native module.
I'm not sure why, but even assert.deepStrictEqual cannot properly find the difference.
But otherwise, using JSON.stringify returns what you'd expect.
So there must be some difference between objects created in JS land, or objects created in n-api, and returned, that prevents deep strict equality from working.
This affects jest testing and any usage of assert.deepStrictEqual.

For some quick debugging we can use:
for(size_t i = 0; i < bufferSize; i++) {
  printf("%#x ", buffer[i]);
}

But I'm still getting refamiliarised with C/C++ code here... and I suspect a better debugging experience can be had by using a debugger in the future.
Now during jest testing, the printf statements don't get outputted by jest for some reason. So it only shows up when using ts-node.
Furthermore, programming at NAPI level is really quite low level. It's a C focused API. So if we really doing hardcore C++, it's much better to use https://github.com/nodejs/node-addon-api. The alternative is rust.

Ok so prebuildify is not packaged under nodePackages atm, so you cannot get the same version as in our nix expressions in package.json. However if we want to use it as a CLI in our nix-build, then we will need to push it into nodePackages to maintain the same version. Otherwise, putting it into our run scripts will also work for now.

It appears that it is expected that the developer just goes and fetches all the prebuild binaries for all the platforms from a CI system, and then puts it all together manually into the prebuilds directory before calling npm publish.

Publishing is mildly more complicated, because npm publish must be done after compiling and fetching prebuilt binaries (typically in CI).
https://github.com/prebuild/prebuild-install

Crucial information is located here: https://github.com/prebuild/prebuild-install as prebuildify is an evolution of prebuild-install.
This basically means after the CI/CD has built everything and pushed it to the release page, you then download them and put them together into the prebuilds directory.
This seems rather clunky, I'd prefer that we have a single command to assemble the prebuilds directory. This solution https://github.com/prebuild/prebuildify/pull/63 might be helpful for doing this too.

The CI/CD workflow used by prebuildify ecosystem is ultimately explained here: https://github.com/mafintosh/prebuildify-ci
Basically the proposed flow is:

Create a new NPM tag with npm version ...
Do a git push && git push --tags to push the tags to the repo on GitHub
The CI/CD is now triggered on new version/tag to prebuildify on each of their virtual environments: Linux, Mac, Windows... maybe even ARM.. .etc. (Not sure about android/ios).
The CI/CD publishes the prebuilds to the GH release page (and the release page corresponds to the new tag)
Once all the CI/CD builds are done, on your local dev env, use prebuildify-ci download (which needs to be configured for download capability), and that downloads all of them to the ./prebuilds directory
Now you can do npm publish which will actually take the ./dist AND the ALL of the ./prebuilds binaries together and put them into the NPM package

Right now we cannot use prebuildify-ci because its workflow doesn't match our Gitlab CI/CD architecture.
This workflow does have some problems:

It is manual, requiring an asynchronous step of waiting CI/CD to finish before finally downloading and doing a npm publish.
It gathers ALL the binary dependencies even the ones that are unnecessary into the same npm package, and that means users end up downloading useless binaries for their host platforms.

The solution to 1. is to make use of the CI/CD features more, such that npm publish itself is part of the CI/CD process. The only problem is that publishing occurs with a specific publish token, and now the bot must have the right authentication token to publish on behalf of MatrixAI.
The solution to 2. involves these new developments:

https://github.com/prebuild/prebuildify/pull/63
https://github.com/prebuild/node-gyp-build/pull/45

Basically the idea is that each platform build goes into their own NPM package, and optionalDependencies in package.json is used to download the correct one. I'm not sure how it works entirely so we shall see.
There's an overall problem with distributing binaries. The lack of determinisim/reproducibility and the supply chain verification. These binaries are just put into the npm package without any oversight, and no source code verification can be done. It does seem that the existence of binaries is only for convenience, and a good build should be done from source.

The CI/CD workflow used by prebuildify ecosystem is ultimately explained here: https://github.com/mafintosh/prebuildify-ci
Basically the proposed flow is:

Create a new NPM tag with npm version ...
Do a git push && git push --tags to push the tags to the repo on GitHub
The CI/CD is now triggered on new version/tag to prebuildify on each of their virtual environments: Linux, Mac, Windows... maybe even ARM.. .etc. (Not sure about android/ios).
The CI/CD publishes the prebuilds to the GH release page (and the release page corresponds to the new tag)
Once all the CI/CD builds are done, on your local dev env, use prebuildify-ci download (which needs to be configured for download capability), and that downloads all of them to the ./prebuilds directory
Now you can do npm publish which will actually take the ./dist AND the ALL of the ./prebuilds binaries together and put them into the NPM package

Right now we cannot use prebuildify-ci because its workflow doesn't match our Gitlab CI/CD architecture.
This workflow does have some problems:

It is manual, requiring an asynchronous step of waiting CI/CD to finish before finally downloading and doing a npm publish.
It gathers ALL the binary dependencies even the ones that are unnecessary into the same npm package, and that means users end up downloading useless binaries for their host platforms.

The solution to 1. is to make use of the CI/CD features more, such that npm publish itself is part of the CI/CD process. The only problem is that publishing occurs with a specific publish token, and now the bot must have the right authentication token to publish on behalf of MatrixAI.
The solution to 2. involves these new developments:

https://github.com/prebuild/prebuildify/pull/63
https://github.com/prebuild/node-gyp-build/pull/45

Basically the idea is that each platform build goes into their own NPM package, and optionalDependencies in package.json is used to download the correct one. I'm not sure how it works entirely so we shall see.
There's an overall problem with distributing binaries. The lack of determinisim/reproducibility and the supply chain verification. These binaries are just put into the npm package without any oversight, and no source code verification can be done. It does seem that the existence of binaries is only for convenience, and a good build should be done from source.

Getting CI/CD to do everything will require the more sophisticated release pipeline that was first specced out in our strategy letter 3 https://gitlab.com/MatrixAI/Employees/matrix-team/-/issues/8#note_885403611.
This might be complicated if we are going to maintain 2 branches for TypeScript-Demo-Lib. I'm starting to think this should be a separate repository so TypeScript-Demo-Lib-Native. We would have to request another repository to be added to the gitlab mac beta.

I'm going to push this to https://github.com/MatrixAI/TypeScript-Demo-Lib-Native later at the end, and then submit request to https://gitlab.com/gitlab-com/macos-buildcloud-runners-beta/-/issues

Ok so the basic idea behind https://gitlab.com/MatrixAI/Employees/matrix-team/-/issues/8#note_885403611 is to have an additional staging branch before the production branch.
Right now most repos have master or main as the production branch. And we have feature branches like this one.
Basically with an additional staging branch it would be possible to run builds on the staging branch, do "staging" deployment on the staging branch, do tests against staging deployment, and if successful proceed to automatically merge into production branch for further testing.
The staging branch could have its history rewritten if necessary, however in general, history will mostly go forward, even in the case of a feature branch merging something incorrect, it would be resolved in another feature branch to be merged into staging. Staging won't merge into master until its state has passed all staging tests.
In PK, the branches are mainnet and testnet and feature branches.
However in library and other repos, it may just be main/master and staging instead since it doesn't directly correspond to any kind of network.
A question would why not just use the CI/CD to do the builds on the feature branches as well, and not just the master branches. The primary reason is that feature branches have alot of noise and it would likely tie up the CI/CD resources quite a bit if we did it on every feature branch. So CI/CD focuses only on local unit tests and not cross-platform testing on feature branches.

Since we want to do this on the new repo, but we can practice first on this repo by introducing a staging branch.

We will create a staging branch off master branch right now.
In this staging branch, we will update our CI/CD instructions to start performing builds, since we are already doing pkg builds, we will now extend this to also doing the native addon builds. In fact native addon builds are necessary prior to pkg builds or any kind of testing since it is part of the source code that is being tested in the unit tests.
So our staging branch can also proceed to cross platform builds. Our nix job currently produces multi-platform binaries, but this is done by merging with existing nodejs binaries that were already cross-compiled. This process may need to change now. We will need to make use of the multiple platforms immediately for cross compilation first since they would be done via source. So pkg builds are still going to be done this way and won't change yet, but linux, windows, macos runs will occur earlier now.

The prepare script a special script that is intended for "source-compilation". https://docs.npmjs.com/cli/v8/using-npm/scripts
By adding in:
    "prepare": "rm -r ./dist || true; tsc -p ./tsconfig.build.json",

This means the tsc build will run on these commands:

npm ci - similar to install but intended in CI environments
npm install - this is RUN after the "install": "node-gyp-build", script, so node-gyp rebuild is run before the TSC source compilation is done
npm pack - we don't really use this, but this would be useful to inspect the package sent to npm
npm publish - makes sense, compile the TSC before publishing to npm
npm rebuild - but only if the current directory is linked to somewhere else
npm diff - not relevant here

This means when entering the nix-shell, we now have something like this:
npm install

><EMAIL_ADDRESS>install
> node-gyp-build

><EMAIL_ADDRESS>prepare
> rm -r ./dist || true; tsc -p ./tsconfig.build.json

Meaning we always end up compiling our C/C++ code first into the shared object, before compiling the TS to JS. Note that node-gyp-build is lazy so if the ./build or ./prebuilds directory already exists, it won't bother building. However our prepare script isn't lazy, so it will always recompile even if needlessly, such as when we have to enter 2 nix-shells.
Is there a way to make our prepare script a bit lazier so it doesn't bother rebuilding if the dist already exists? Would that make sense though? Maybe by checking if the dist already exists.
Additionally by doing this, we should be able to eliminate our node2nixDev npm install override. We can allow nix to just do the npm install, because after doing the installation, it will run the prepare script to compile TSC.

I just realised the usage of
    "prebuild": "prebuildify --napi --strip",

means that when running npm run build, it ends up running the prebuildify script as well.
This is bit weird, since npm run build right now does a full rebuild of both TSC and C/C++ by directly calling node-gyp rebuild.
I think prebuildify should only be called by the CI/CD, it's not something that local development should be doing. So I'm going to be removing these 2 scripts:
    "prebuild": "prebuildify --napi --strip",
    "prebuild-ia32": "prebuildify --napi --strip --arch=ia32",

And maybe bring them back when inspecting the CI/CD.

With the new incremental compilation, compilation can be very fast, so it's no longer a problem to always remove the dist. I could have done something where we check if the dist exists, but then the prepare isn't really doing much when being called before npm publish.
This also mean npm run build is only necessary to do some minor build checking of the dist and the C/C++. And node-gyp rebuild should always be used when you want to rebuild the C/C++ code.

Ok so I have managed to get node-gyp integrated into the nix-build ./release.nix -A application. Some things I learned:

The node-gyp-build is not necessary, as we learned before, this is a runtime dependency and would be installed when necessary. I'm actually not sure why I had this, maybe just old node2nix dependencies often had this dependency injected. But removing this still worked.
This might mean the need to enforce same-version in shell.nix is less needed, except for of course ensuring easier debuggability. So we will leave those there for now.
This means utils.nix has been simplified.node2nixProd = (import (node2nixDrv false) { inherit pkgs nodejs; }).nodeDependencies.override (attrs: {
  # Use filtered source
  src = src;
  # Do not run build scripts during npm rebuild and npm install
  npmFlags = "--ignore-scripts";
  # Do not run npm install, dependencies are installed by nix
  dontNpmInstall = true;
});
node2nixDev = (import (node2nixDrv true) { inherit pkgs nodejs; }).package.override (attrs: {
  # Use filtered source
  src = src;
  # Ensure that build scripts are foregrounded during npm rebuild and npm install
  npmFlags = "--foreground-scripts";
});

In order to use such a simplified version we made use of the prepare script since it is intended for source compilation. Now npm install will call install script first, then prepare script afterwards. So no more postInstall hook needed in nix.
Now prepare script is not called by npm rebuild because we are not a symlinked directory during building. But it will end up building the C/C++, and because npm install calls node-gyp-build it will find that the build/Release exists and won't bother rebuilding.
I found that the --foreground-scripts is very useful to ensure that the any scripts that  run are now outputted to the foreground rather than running in the background, it can be disabled if we want faster build in the future.
Now for node2nixProd, you can see that there's a new nodeDependencies attribute, we don't have to go through the shell. And in this situation, it still tries to run npm rebuild and npm install. So we have to disable the npm install with dontNpmInstall, and we have to tell npm rebuild to ignore all build scripts with --ignore-scripts. This can still be a problem in case any of our production dependencies require source-recompilation atm.
In the future, we would probably want to optimise the node2nixProd and node2nixDev so we only use 2 node2nix derivation, and just figure out what dependencies are production, and only install does by directly copying them out of the node_modules rather than right now which is installing node_modules twice. This will be important when we want to ensure supply-chain security by always building our deps from source.
Finally in default.nix, we now need to copy over the build/Release/*.node binaries so that the final nix store output can have the binaries and our scripts can find them. Here we just check if that build directory exists at all.
I realised that $packageName was available due to the runCommandNoCC environment. So that simplified the default.nix.
We made a decision to use the build/Release rather than prebuilds here at least for nix-build ./release.nix -A application and derivatives of this. This is because prebuilds would only come out of prebuildify, but build/Release take precedence over prebuilds, and this is the default path of building C/C++ addons.

So nix-build ./release.nix -A application works, and the ./result/bin/typescript-demo-lib also works and calls the right native addon shared object. There are some warnings about patch elf, but that's because this shared object isn't a standalone executable. Furthermore shared objects are supposed to have executable permissions.
»» ~/Projects/TypeScript-Demo-Lib
 ♖ tree -a -L 6 ./result                                                                                (native) pts/2 20:55:26
./result
├── bin
│   └── typescript-demo-lib -><EMAIL_ADDRESS>└── lib
    └── node_modules
        └── @matrixai
            └── typescript-demo-lib
                ├── build
                │   └── Release
                ├── dist
                │   ├── bin
                │   ├── index.d.ts
                │   ├── index.js
                │   ├── index.js.map
                │   ├── lib
                │   ├── native
                │   ├── test.json
                │   ├── utils.d.ts
                │   ├── utils.js
                │   └── utils.js.map
                ├── node_modules
                │   ├── abstract-leveldown
                │   ├── base64-js
                │   ├── .bin
                │   ├── buffer
                │   ├── callsites
                │   ├── catering
                │   ├── debug
                │   ├── deferred-leveldown
                │   ├── encoding-down
                │   ├── esm
                │   ├── fd-lock
                │   ├── ieee754
                │   ├── inherits
                │   ├── is-buffer
                │   ├── is-observable
                │   ├── level
                │   ├── level-codec
                │   ├── level-concat-iterator
                │   ├── leveldown
                │   ├── level-errors
                │   ├── level-iterator-stream
                │   ├── level-js
                │   ├── level-packager
                │   ├── level-supports
                │   ├── levelup
                │   ├── ltgt
                │   ├── ms
                │   ├── napi-macros
                │   ├── node-gyp-build
                │   ├── observable-fns
                │   ├── queue-microtask
                │   ├── readable-stream
                │   ├── run-parallel-limit
                │   ├── string_decoder
                │   ├── threads
                │   ├── timeout-refresh
                │   ├── tiny-worker
                │   ├── unordered-set
                │   ├── util-deprecate
                │   ├── utp-native
                │   └── uuid
                └── package.json

53 directories, 9 files

In light of realising that node-gyp-build is not that important to maintain sync between nix and dev environment.
      npm install \
        --package-lock-only \
        --save-dev \
        --save-exact \
<EMAIL_ADDRESS>\
<EMAIL_ADDRESS>
      npm install \
        --package-lock-only \
        --save \
        --save-exact \
<EMAIL_ADDRESS>
I believe we can reduce this down to only pkg and node-gyp.

Note that node2nix uses nodejs.src rather than just nodejs as we discovered when using node-gyp rebuild. Created an issue for this https://github.com/svanderburg/node2nix/issues/295

I actually forgot about https://github.com/NixOS/nixpkgs/issues/170279 that we encountered before, so for now the docker builds still fail in NixOS. So I'm going to assume that the docker builds do work.

Now to address task 4...
Can our pkg working with our locally compiled build/Release? It can, but then it would only work on NixOS/Linux.
The pkg has 3 outputs: linux, macos, windows. These 3 outputs require 3 differently compiled native modules. And normally it would expect to get the prebuilds.
See that the pkg configuration is:
  "pkg": {
    "assets": [
      "node_modules/leveldown/**/*",
      "node_modules/utp-native/**/*",
      "node_modules/fd-lock/**/*",
      "dist/**/*.json"
    ],
    "scripts": [
      "dist/lib/workers/worker.js"
    ]
  },

We now know that those assets only really need to keep track of the binaries in prebuilds/ directory, so having node_modules/leveldown/**/* is actually overkill as it brings in all of the node_modules directory. @tegefaulkes
Additional reading here: https://github.com/vercel/pkg#native-addons
Therefore pkg can only work after we have done our prebuildify.
We can imagine a revised pkg configuration like this:
  "pkg": {
    "assets": [
      "node_modules/leveldown/prebuilds/**/*.node",
      "node_modules/utp-native/prebuilds/**/*.node",
      "node_modules/fd-lock/prebuilds/**/*.node",
      "prebuilds/**/*.node",
      "dist/**/*.json"
    ],
    "scripts": [
      "dist/lib/workers/worker.js"
    ]
  },

The only issue would be that this still makes pkg package up all the platforms. But I don't think pkg should be solving this here, instead, we would wait for https://github.com/prebuild/prebuildify/pull/63 and https://github.com/prebuild/node-gyp-build/pull/45 to be first settled before proceeding to optimise.

I believe we will need change to using Gitlab CICD DAG model to express what we want to do with the staging branch. Basically after local testing, we will start the building process. The building process cannot actually use Nix on the other platforms like mac and windows. This is because nix won't work on those platforms (and also because our docker container that we use for nix doesn't apply when running in windows and mac). We will need to run it like a normal nodejs application and ensure we have the necessary development tooling.
The problem with changing the pkg is that it now will be looking for prebuilds/ directory.
However that won't exist until the CI/CD has done its job. This means the nix-build ./release.nix -A package.linux.x64.elf won't work anymore without additional tweaks that brings in the prebuilds directory with all the prebuilt executables after the CI/CD has finished building the outputs.
Alternatively, the expectation that if you are building it locally, then you would assemble your own prebuild directory manually or instead get it from the build/Release, which should work for the package.linux.x64.elf attribute, but not sure about the other ones.

Yes by changing to:
  "pkg": {
    "assets": [
      "node_modules/leveldown/prebuilds/**/*",
      "node_modules/utp-native/prebuilds/**/*",
      "node_modules/fd-lock/prebuilds/**/*",
      "prebuilds/**/*.node",
      "build/Release/*.node",
      "dist/**/*.json"
    ],
    "scripts": [
      "dist/lib/workers/worker.js"
    ]
  },

The nix-build ./release.nix -A package.linux.x64.elf does in fact work, and I ran it on Ubuntu.
Notice that having prebuilds/**/*.node and build/Release/*.node means that if both directories exist, it will end up bundling both. But it makes more sense to only bundle one of them. Not sure about this atm. We might need to change this later since, our changes in https://github.com/MatrixAI/TypeScript-Demo-Lib/pull/38#issuecomment-1120829967 has meant that we use build/Release/*.node normally in Nix.
If we can unify it all under prebuilds directory that could work better.
We may also need to be able to dynamically configure the pkg with the --config option.
Note I need to find out if having npm_config_build_from_source = "true"; still makes sense, since pkg just bundles up the binaries, and thus there's actually no native addons being rebuilt anyway.

Some further learning about using pkg with prebuilds.
This PR https://github.com/vercel/pkg/pull/1066 introduced the usage of prebuild-install into pkg. What it did was to run prebuild-install to bring in the cross-compiled dependency when you run pkg.
There are several problems with this:

All of our packages are not using prebuildify not prebuild-install, meaning that prebuild-install is not necessary since we already have the fully compiled binaries provided in the NPM package.
This is just trying to download over the network, and we can't do that inside nix build environment.
Since it's not actually building from source, it's not really useful here, as we expect that the node-gyp tooling require virtualised infrastructure to complete the builds.

I had disabled it prior with:
      # ensure that native modules are built from source
      npm_config_build_from_source = "true";

But this was actually incorrect, better was the flag --no-native-build.
We don't pkg to do any source compilation. We just want it to do bundling for now. So this prevents pkg from doing but bundling.

Now to integrate CI/CD compilation with node-gyp.
Windows Gitlab CI/CD
The preinstalled-software that windows VMs have are: https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/gcp/windows-containers/blob/main/cookbooks/preinstalled-software/README.md.
Of primary importance is choco. Which we can use to automate the installation of windows packages that we need.
In particular we need 3 things:

visualstudio2019community

this is Microsoft's SDK, it bundles the ability to compile C/C++ on windows
SDK releases are structured in years then versions, such as Visual Studio 2022, Visual Studio 2019... etc
the SDK seems to be forward-compatible, but may not be backwards compatible
installing visual studio requires a large amount of space, which the VM may not have enough
because the windows VM already has visualstudio2019community, we will just update the 2019 release

nodejs
python

The choco command allows us to specify the version.
    - choco install nodejs --version=16.14.2 -y
    - choco install python --version=3.9.12 -y

Notice that we must use the -y to ensure that the installation can proceed unattended.
After doing the installation, the system PATH gets updated. To ensure we get the new PATH, we must use: refreshenv.
Once all the windows packages are installed, we can proceed to build our project.
Firstly using dir . shows us that the project is cloned to the current working directory:
C:\GitLab-Runner\builds\MatrixAI\open-source\typescript-demo-lib\

Because our project will execute node-gyp rebuild, then we must ensure that node-gyp find the right visual studio with the msvs_version setting.
Running npm config set msvs_version 2019 will edit the user profile ~/.npmrc, and make node-gyp find the 2019 version of visual studio.
    - npm config set msvs_version 2019
    - npm install
    - $env:Path = "$(npm bin);" + $env:Path

After installation, we update the PATH directly by passing npm bin. This will provide us access executables in the node_modules/.bin.

Now we have an error in compiling our index.cpp file.
C:\GitLab-Runner\builds\MatrixAI\open-source\typescript-demo-lib\src\native\index.cpp(127,3): error C2131: expression did not evaluate to a constant [C:\GitLab-Runner\builds\MatrixAI\open-source\typescript-demo-lib\build\native.vcxproj]
C:\GitLab-Runner\builds\MatrixAI\open-source\typescript-demo-lib\src\native\index.cpp(127,3): message : failure was caused by a read of a variable outside its lifetime [C:\GitLab-Runner\builds\MatrixAI\open-source\typescript-demo-lib\build\native.vcxproj]
C:\GitLab-Runner\builds\MatrixAI\open-source\typescript-demo-lib\src\native\index.cpp(127,3): message : see usage of 'keyValueLength' [C:\GitLab-Runner\builds\MatrixAI\open-source\typescript-demo-lib\build\native.vcxproj]
C:\GitLab-Runner\builds\MatrixAI\open-source\typescript-demo-lib\src\native\index.cpp(128,44): error C3863: array type 'char [keyValueLength+1]' is not assignable [C:\GitLab-Runner\builds\MatrixAI\open-source\typescript-demo-lib\build\native.vcxproj]
C:\GitLab-Runner\builds\MatrixAI\open-source\typescript-demo-lib\src\native\index.cpp(130,38): error C2131: expression did not evaluate to a constant [C:\GitLab-Runner\builds\MatrixAI\open-source\typescript-demo-lib\build\native.vcxproj]
C:\GitLab-Runner\builds\MatrixAI\open-source\typescript-demo-lib\src\native\index.cpp(130,23): message : failure was caused by a read of a variable outside its lifetime [C:\GitLab-Runner\builds\MatrixAI\open-source\typescript-demo-lib\build\native.vcxproj]
C:\GitLab-Runner\builds\MatrixAI\open-source\typescript-demo-lib\src\native\index.cpp(130,23): message : see usage of 'keyValueLength' [C:\GitLab-Runner\builds\MatrixAI\open-source\typescript-demo-lib\build\native.vcxproj]

It appears that our C code written in index.cpp is failing some compiler checks that the windows compiler does. Perhaps we're not writing proper C or C++ code? Perhaps we should be writing it as .cc? Not sure.

It appears the reason for the compilation failure is because of the variable length array https://stackoverflow.com/a/26815571/582917. While GCC supports it via C99 feature of variable length array, MSVC does not support it.
In fact it is actually necessary to it like:
  // Concatenate the string
  char fullString[8 + keyValueLength + 1];
  // Variable length array initialisation requires memset
  memset(fullString, 0, sizeof(fullString));

I was only able to see this warning by switching to index.c instead of index.cpp. Indeed we are not actually using any C++ code at all. The reported error in windows was not at all accurate...
But anyway, MSVS will never support variable length arrays, so it's necessary to do it a different way. https://stackoverflow.com/questions/58462881/how-to-use-visual-studio-as-an-ide-with-variable-length-arrayvla-working
Replacing it with calloc should work:
  // Concatenate the string
  char * fullString = calloc(8 + keyValueLength + 1, sizeof(char));

Going to setup RDP to our local windows and mac machines as it will be faster to iterate than to wait for the build machines.

Some important things about compilation.
When given a .c file, node-gyp will use gcc. When given a .cpp file, node-gyp will use g++.
Therefore cflags only applies to gcc and C files. While cflags_cc applies to g++ and C++ files.
Note that when using g++, it actually applies both cflags and cflags_cc. However the cflags_cc applies last and so it overrides whatever flags where originally in cflags.
It appears that for Windows, the minimum version is C90 standard. It does however support some features of C99, but not all, and VLAs are not supported, and this is where we have to be careful not to use those C99 features. So I kept the C standard to c99.
As for C++, windows best support and gcc best support is C++17 standard.
Therefore our flag options can be:
    "cflags": [ "-std=c99", "-Wpedantic" ],
    "cflags_cc": [ "-std=c++17", "-Wpedantic" ]

Note that Windows and Mac has different set of settings, we will need to find them in https://github.com/nodejs/node/blob/eaaee92d9b0be82d7f40b2abb67f30ce525d4bc4/common.gypi to find them.

When using printf or fprintf, if you use npm test -- ./sometestfile.test.ts, you must pass --verbose flag to see these logs. Quite a strange behaviour.

Ok finally we have it working..., after changing to malloc for all of the input strings. All of them are dynamic, so it is necessary to use NPM_UTF8_MALLOC.
It does have strange behaviour regarding name_len and name_size, I would have expected that name_len be the JS string length, while name_size would be the JS string length plus 1 for the null byte.
Anyway I'm now testing conditional flags on binding.gyp, it appears we can conditionally set settings for linux, windows, and mac.

Ok after running this on an actual windows machine, some interesting things have been revealed.
We will need to run local tests on the different machines, this means doing npm test on Windows as well.
When doing this, we hit the bug of leveldb not being closed. So that's a great way of seeing whether our native libraries, but even our normal tests are cross platform.
In doing so, when doing npm install, we were using POSIX shell concepts like rm -r. This won't work if we expect to run on Windows.
In such a case, that's where rimraf library becomes useful.
However now our scripts section needs to be alot more careful. We cannot just use command ; command. When running on windows, npm uses the CMD to execute commands. Not powershell. So the ; means parameter separator, not command separator.
There's only one common one, and that's && where it means do the first command before the second command. So I've swapped all rm -r to rimraf and using && when multiple commands must be run.
Note that npm can be set to use powershell directly https://stackoverflow.com/questions/23243353/how-to-set-shell-for-npm-run-scripts-in-windows. But I'm not sure if we want to add more ambient configuration.
Anyway, I can confidently say that prebuildify and tests all work on the windows side too.

It's important to note that prebuildify has its own cache, and requires explicit target setting: https://github.com/prebuild/prebuildify/issues/64

I've found that by overriding the call package of mkShell, we can set the stdenv to multiStdenv, which gives a gcc/glibc multi architecture, enabling us to build 32 bit and 64 bit binaries.
Now these 2 commands finally work https://github.com/MatrixAI/TypeScript-Demo-Lib/pull/38#issuecomment-1119485823
prebuildify --napi --target=16.14.2
prebuildify --napi --target=16.14.2 --arch=ia32

[nix-shell:~/Projects/TypeScript-Demo-Lib]$ tree prebuilds/
prebuilds/
├── linux-ia32
│   └── node.napi.node
└── linux-x64
    └── node.napi.node

This works also on windows, because windows has the linkable node object already.
It produces win32-ia32 and win32-x64.

I've also realised that the produced binaries on Nix are dynamically linked. In doing so, the binaries would only work on NixOS, which would bundle up both glibc and gcc.
Basically the opposite issue of pkg, which actually doesn't compile from source.
Anyway this does mean we would need to use a Ubuntu VM to compile relevant binaries for linux.
However I think the binaries are portable simply because things like fd-lock were too. We can test on ubuntu to check.

Indeed it is the case that we have hardcoded nix paths in the built shared object.
[nix-shell:~/Projects/TypeScript-Demo-Lib]$ readelf -d ./prebuilds/linux-x64/node.napi.node

Dynamic section at offset 0x2ca8 contains 33 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000e (SONAME)             Library soname: [native.node]
 0x000000000000001d (RUNPATH)            Library runpath: [/nix/store/fyhgzci1q7ml9c3npmxzva4qgj37mhk4-nix-shell/lib64:/nix/store/fyhgzci1q7ml9c3npmxzva4qgj37mhk4-nix-shell/lib:/nix/store/ayrsyv7npr0lcbann4k9lxr19x813f0z-glibc-2.34-115/lib:/nix/store/psijdi9190zgbp053y6dj3ax4y2l70gk-gcc-11.2.0-lib/lib]
 0x000000000000000c (INIT)               0x1000
 0x000000000000000d (FINI)               0x1a9c
 0x0000000000000019 (INIT_ARRAY)         0x3c90
 0x000000000000001b (INIT_ARRAYSZ)       16 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x3ca0
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x0000000000000004 (HASH)               0x270
 0x000000006ffffef5 (GNU_HASH)           0x340
 0x0000000000000005 (STRTAB)             0x680
 0x0000000000000006 (SYMTAB)             0x380
 0x000000000000000a (STRSZ)              873 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000003 (PLTGOT)             0x3ef8
 0x0000000000000002 (PLTRELSZ)           480 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0xc08
 0x0000000000000007 (RELA)               0xa70
 0x0000000000000008 (RELASZ)             408 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000000000001e (FLAGS)              BIND_NOW
 0x000000006ffffffb (FLAGS_1)            Flags: NOW
 0x000000006ffffffe (VERNEED)            0xa30
 0x000000006fffffff (VERNEEDNUM)         1
 0x000000006ffffff0 (VERSYM)             0x9ea
 0x000000006ffffff9 (RELACOUNT)          6
 0x0000000000000000 (NULL)               0x0

It actually has alot more libraries than I expected. Compare this to fd-lock's binary:
[nix-shell:~/Projects/TypeScript-Demo-Lib]$ readelf -d ./node_modules/fd-lock/prebuilds/linux-x64/node.napi.node

Dynamic section at offset 0x1df8 contains 25 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000e (SONAME)             Library soname: [lock_fd.node]
 0x000000000000000c (INIT)               0x938
 0x000000000000000d (FINI)               0xe5c
 0x0000000000000019 (INIT_ARRAY)         0x201dd8
 0x000000000000001b (INIT_ARRAYSZ)       16 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x201de8
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x1f0
 0x0000000000000005 (STRTAB)             0x490
 0x0000000000000006 (SYMTAB)             0x238
 0x000000000000000a (STRSZ)              517 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000003 (PLTGOT)             0x202000
 0x0000000000000002 (PLTRELSZ)           240 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x848
 0x0000000000000007 (RELA)               0x6f8
 0x0000000000000008 (RELASZ)             336 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffe (VERNEED)            0x6c8
 0x000000006fffffff (VERNEEDNUM)         1
 0x000000006ffffff0 (VERSYM)             0x696
 0x000000006ffffff9 (RELACOUNT)          6
 0x0000000000000000 (NULL)               0x0

We do use more of the C standard library though.
I need to find out more about how our Nix is building these, because it does seem by default the binaries produced by fd-lock and such hardcode any paths.
Even leveldb:
[nix-shell:~/Projects/TypeScript-Demo-Lib]$ readelf -d ./node_modules/leveldown/prebuilds/linux-x64/node.napi.glibc.node

Dynamic section at offset 0x77b30 contains 29 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000e (SONAME)             Library soname: [leveldown.node]
 0x000000000000000c (INIT)               0x133b8
 0x000000000000000d (FINI)               0x65ba0
 0x0000000000000019 (INIT_ARRAY)         0x276a00
 0x000000000000001b (INIT_ARRAYSZ)       16 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x276a10
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x1f0
 0x0000000000000005 (STRTAB)             0x5f40
 0x0000000000000006 (SYMTAB)             0x1638
 0x000000000000000a (STRSZ)              30561 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000003 (PLTGOT)             0x278000
 0x0000000000000002 (PLTRELSZ)           10536 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x10a90
 0x0000000000000007 (RELA)               0xdd48
 0x0000000000000008 (RELASZ)             11592 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffe (VERNEED)            0xdcb8
 0x000000006fffffff (VERNEEDNUM)         3
 0x000000006ffffff0 (VERSYM)             0xd6a2
 0x000000006ffffff9 (RELACOUNT)          124
 0x0000000000000000 (NULL)               0x0

It is possible to remove the rpath with patchelf --remove-rpath ./prebuilds/linux-x64/node.napi.node. This then produces a binary without the rpath embedded. I wonder if this is already done via nix-build and if the preservation of the rpath is only there during nix-shell.
See: https://stackoverflow.com/questions/13769141/can-i-change-rpath-in-an-already-compiled-binary
Resulting binary is now:
[nix-shell:~/Projects/TypeScript-Demo-Lib]$ readelf -d ./prebuilds/linux-x64/node.napi.node

Dynamic section at offset 0x2ca8 contains 32 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000e (SONAME)             Library soname: [native.node]
 0x000000000000000c (INIT)               0x1000
 0x000000000000000d (FINI)               0x1a9c
 0x0000000000000019 (INIT_ARRAY)         0x3c90
 0x000000000000001b (INIT_ARRAYSZ)       16 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x3ca0
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x0000000000000004 (HASH)               0x270
 0x000000006ffffef5 (GNU_HASH)           0x340
 0x0000000000000005 (STRTAB)             0x680
 0x0000000000000006 (SYMTAB)             0x380
 0x000000000000000a (STRSZ)              873 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000003 (PLTGOT)             0x3ef8
 0x0000000000000002 (PLTRELSZ)           480 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0xc08
 0x0000000000000007 (RELA)               0xa70
 0x0000000000000008 (RELASZ)             408 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000000000001e (FLAGS)              BIND_NOW
 0x000000006ffffffb (FLAGS_1)            Flags: NOW
 0x000000006ffffffe (VERNEED)            0xa30
 0x000000006fffffff (VERNEEDNUM)         1
 0x000000006ffffff0 (VERSYM)             0x9ea
 0x000000006ffffff9 (RELACOUNT)          6
 0x0000000000000000 (NULL)               0x0

After removing the rpath, the binary still works. Possibly because the resulting binary is loaded by the node executable which doesn't bother using the rpath encoded in it.

So I've noticed that the pkg compiled executable is very similar. The only library that it cannot find using ldd is the libstdc++.so.6 this is the same for the prebuilt binary here and in leveldb.
When the rpath is set, the libstdc++.so.6 is found. And it's found in /nix/store/psijdi9190zgbp053y6dj3ax4y2l70gk-gcc-11.2.0-lib/lib/libstdc++.so.6. Notice that this is different the other shared object paths being in glibc:
[nix-shell:~/Projects/TypeScript-Demo-Lib]$ ldd ./prebuilds/linux-x64/node.napi.node
	linux-vdso.so.1 (0x00007ffcde14e000)
	libstdc++.so.6 => /nix/store/psijdi9190zgbp053y6dj3ax4y2l70gk-gcc-11.2.0-lib/lib/libstdc++.so.6 (0x00007ff7e5cab000)
	libm.so.6 => /nix/store/ayrsyv7npr0lcbann4k9lxr19x813f0z-glibc-2.34-115/lib/libm.so.6 (0x00007ff7e5bd2000)
	libgcc_s.so.1 => /nix/store/ayrsyv7npr0lcbann4k9lxr19x813f0z-glibc-2.34-115/lib/libgcc_s.so.1 (0x00007ff7e5bb8000)
	libpthread.so.0 => /nix/store/ayrsyv7npr0lcbann4k9lxr19x813f0z-glibc-2.34-115/lib/libpthread.so.0 (0x00007ff7e5bb3000)
	libc.so.6 => /nix/store/ayrsyv7npr0lcbann4k9lxr19x813f0z-glibc-2.34-115/lib/libc.so.6 (0x00007ff7e59b4000)
	/nix/store/ayrsyv7npr0lcbann4k9lxr19x813f0z-glibc-2.34-115/lib64/ld-linux-x86-64.so.2 (0x00007ff7e5ec6000)

So something is different about libstdc++.so.6, and I wonder why it was not part of glibc? Possibly because glic is the C standard library, whereas libstdc++.so.6 is the C++ standard library. But it's weird that it's distributed as part of GCC.

It appears that gcc-11.2.0-lib libraries are GCC libraries. And these libraries can be considered the standard library required for language implementations that GCC implements. These are different from the glibc which are part of the operating system.

Beyond the library facilities required of such an implementation, the rest of the C library is supplied by the vendor of the operating system.
https://gcc.gnu.org/onlinedocs/gcc/Standard-Libraries.html

In https://nixos.wiki/wiki/C it explains that the GCC are wrappers provided by our nix standard environment, this means the tools are capable of adding additional options and flags before and after we are executing our build commands.
The addition of the NIX_DEBUG=1 environment variable in our shell.nix reveals how our compilation is actually being done.
This now gives us a full log of the build options and flags and we can see how the runpath setting is being set.
The runpath setting will come from 2 sources:

The $NIX_LDFLAGS
The compiler wrapper

The result is that any compiled object from using our Nix tools will end up with a runpath embedded in the binary.
For example using our multiStdenv, we get now:
[nix-shell:~/Projects/TypeScript-Demo-Lib]$ readelf -d ./prebuilds/linux-x64/node.napi.node  | grep RUNPATH
 0x000000000000001d (RUNPATH)            Library runpath: [/nix/store/iiq1niirvd1s2qvzaa80153lry5m0cr2-nix-shell/lib64:/nix/store/iiq1niirvd1s2qvzaa80153lry5m0cr2-nix-shell/lib:/nix/store/f6sqc6f2kcijdaq1gsyzd838c8k730yf-glibc-multi-2.34-115/lib:/nix/store/iyx5q794wrx0d5ir5bmq3a5n4szkjd84-gcc-11.2.0-lib/lib64:/nix/store/iyx5q794wrx0d5ir5bmq3a5n4szkjd84-gcc-11.2.0-lib/lib]

To explain:
    # these 2 are coming from $NIX_LDFLAGS, and would only exist because our nix-shell is also a "derivation", and therefore would exist in our regular nix-builds enabling the ability to look up other binary libraries in our $out/lib and $out/lib64
    /nix/store/rvw49dqfiign4xg37khfwixg5xl8ksbz-nix-shell/lib64:
    /nix/store/rvw49dqfiign4xg37khfwixg5xl8ksbz-nix-shell/lib:
    # 64 + 32 bit GLIBC (C standard library)
    /nix/store/f6sqc6f2kcijdaq1gsyzd838c8k730yf-glibc-multi-2.34-115/lib:
    # this is GCC standard library 64bit
    /nix/store/iyx5q794wrx0d5ir5bmq3a5n4szkjd84-gcc-11.2.0-lib/lib64:
    # this is GCC standard library 32bit
    /nix/store/iyx5q794wrx0d5ir5bmq3a5n4szkjd84-gcc-11.2.0-lib/lib

And therefore, having this RUNPATH being set in the output is what is going to prevent these binaries from being runnable in non-Nix linux.
Technically, non-Nix linux could still run them by ignoring the runpath or if the runpath directories don't exist.
This would mean to create portable binaries (for non-Nix linux), we should be stripping them the run path using the patchelf command.
Of course within nix, the runpath setting should be kept, since that is what expected of binaries running within a nix system.

Additionally the additional options/default options being added in by node-gyp come from: https://github.com/nodejs/node-gyp/blob/master/addon.gypi.

There is a program called ld-linux-x86-64.so.2. This is the "dynamic linker/loader". It is the interpreter of compiled binary programs. https://linux.die.net/man/8/ld-linux.so
On NixOS you can find this interpreter program by doing:
# finds the 32 bit dynamic linker and loader
ls /nix/store/*glibc*/*/ld-linux.so.2
# finds the 64 bit dynamic linker and loader
ls /nix/store/*glibc*/*/ld-linux-x86-64.so.2

When you are in a nix-shell or a nix build environment, it's easier to do this:
cat $NIX_CC/nix-support/dynamic-linker

This program is not usually put on the PATH, but you running it directly will give you options that allow you to specialise the execution of ELF executables on Linux operating systems.
According to the FHS, the designated locations for the dynamic linker and loader are:
# for 32 bit
/lib/ld-linux.so.2
# for 64 bit
/lib64/ld-linux-x86-64.so.2

This is why for portable Linux binaries, you generally will want the interpreter path set to one of those paths depending on if you are producing 32 bit binary or 64 bit binary. In fact this is the case for the nodejs executables when downloaded directly off https://nodejs.org.
However not all distros follow this process. NixOS is one of them that does not have anything in /lib nor /lib64.
On NixOS, binaries are expected to have their interpreter path set directly to one of the paths within the /nix/store. Furthermore, binaries are expected to have their DT_RUNPATH also set to relevant library paths to the /nix/store.
This is why so-called portable binaries do not run on NixOS normally, give a weird error message when you execute them:
bash: ./result: No such file or directory

Portable binaries will generally not have the DT_RUNPATH set, because they would expect the operating system to have the correctly configured search paths either via LD_LIBRARY_PATH or the ld.so.cache or finally compiled "system search path" that is embedded into the dynamic linker & loader.
When you use ldd on a binary executable, ldd knows how to find the ld-linux.so so when it shows up, that won't affect the actual execution of the program.
Ok so the easiest way to run a non-NixOS executable in this case is to do these things:
$(cat $NIX_CC/nix-support/dynamic-linker) --library-path '<setsomepath>:<setsomepath>' ./binary

Note that the library path will depend on the libraries that it needs, which you can try to resolve with ldd or with readelf -d.
After trying this on the vercel/pkg built executable, I find that I can run the nodejs executable, but the vercel pkg code internally fails to work, it says Pkg: Error reading from file.. So I'm guessing vercel/pkg bundles some platform-specific code that fails to work on NixOS.

At any case, it appears that what we need to do is to remove the rpath from any binaries created in nix when intending for portable environments. This should be doable with a a final fixup phase being something like:
patchelf --remove-rpath ./prebuilds/linux-x64/node.napi.node

We would have to test to see if after doing this, it would actually end up working when on Ubuntu.

Dynamic libraries like .so can have their own rpath. We can see this in our Nix built shared objects.
Because they have their own rpath, it should mean that these shared objects are not portable to non-nix linux operating systems.
To make them portable, we would want to remove the rpath when building them in a Nix build environment. Which we can do with patchelf --remove-rpath.
This is the case with other prebuilt native addons I see like in leveldown and fd-lock. They do not have the rpath set.
However dynamic libraries are allowed to have their own rpath, and this can aid the dynamic loader in finding the dependencies that the dynamic library needs. When the dynamic library doesn't have its own rpath, it ends up relying on the main executable's rpath.
If the dynamic library depends on something that isn't found via the main executable's rpath, then the dynamic library will fail to load. For example see: https://unix.stackexchange.com/questions/571861/is-there-an-rpath-for-dynamic-linked-libraries
It is not clear what the relationship is between the dynamic library's own rpath and the main executable's rpath. It might be the case that the dynamic library's rpath overrides, or it might just be searched recursively.

The conclusion is that if your native module depends on libraries that is unlikely to be found by nodejs itself, then you do actually want to do something so that the native addon when loaded will find its needed libraries.
Right now on Nix, the node executable already has its own rpath:
[nix-shell:~/Projects/TypeScript-Demo-Lib]$ patchelf --print-rpath /nix/store/zl4bvsqfxyx5vn9bbhnrmbmpfvzqj4gd-nodejs-16.14.2/bin/node
/nix/store/b36ilvc5hhfpcp7kv1kvrkgcxxpmxfsd-zlib-1.2.12/lib:/nix/store/nr73vd4apmn4sd6mmzw9fwpg0dq82g03-libuv-1.44.1/lib:/nix/store/v8vpzh3slc5hm4d9id5bim4dsb4d2ndh-openssl-1.1.1n/lib:/nix/store/7gkya4n7b0jj12r3k5v2x9gizwqhlbgm-icu4c-70.1/lib:/nix/store/ayrsyv7npr0lcbann4k9lxr19x813f0z-glibc-2.34-115/lib:/nix/store/psijdi9190zgbp053y6dj3ax4y2l70gk-gcc-11.2.0-lib/lib

As you can see, this pretty much includes all of the main nodejs headers:

zlib
libuv
openssl
icu4c
glibc
gcc standard library

But if our native module will depend on things outside of this, we would need to ensure that those will be available as well.
Now that could be fixed with setting the rpath in the shared object itself, but that would only work for Nix environments. For non-Nix, you are now needing to create a "portable" environment.
This most likely means statically linking the other libraries instead of dynamic linking. Or it could mean distributing the dynamic libraries with your native addon in the same directory, and using the $ORIGIN in the run path so you can refer to those other shared objects. https://stackoverflow.com/a/24574295/582917
I believe static linking would the right way to go. Bundling seems complicated because other platforms like windows may not even have equivalents.

I have managed to get prebuildify integrated into the nix build. This was done by setting --ignore-scripts for the node2nixDev as well. Finally in postInstall hook we use npm --nodedir=${nodejs} run build to fully execute the build scripts.
Because we have prebuild and build, this ends up building the native addons using prebuildify and the TSC compilation in one go. This avoids prebuildify running after node-gyp has already built.
Finally we use patchelf --remove-rpath prebuilds/*/*.node which will remove the rpath setting, thus enabling these native addons to be portable to non-Nix linux.

The build works on MacOS. I've tested on our own matrix-mac-1.
The main addition is the usage of hash -r that operates like refreshenv in powershell. This is because shells may cache previously used commands, and when we are using homebrew to install things on the VMs, we may be overriding existing binaries.
For Mac, there's an opportunity to eventually incorporate its build into the Nix ecosystem as well, since Nix works on macos. That way we can avoid having to use brew. However I'm not sure how nix mac's ecosystem works, so I'll leave it as it is right now.
The resulting build looks like this:
cmcdragonkai@matrix-mac-1 TypeScript-Demo-Lib % tree ./prebuilds
./prebuilds
└── darwin-arm64
  └── node.napi.node
1 directory, 1 file

Notice that because this is a M1 mac, we actually have darwin-arm64. Our current pkg build doesn't get the arm64 variant. We could add it in from: https://github.com/vercel/pkg-fetch/releases/download/v3.3/node-v16.14.2-macos-arm64
However our gitlab job currently specifies the shared-macos-amd64 tag. Gitlab has only added recent support for arm64 macs: https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/2274 so there's not much documentation about it.
Also there's apple universal binaries, perhaps that should be used instead?

I've managed to get a working build for universal binary.
It was a matter of PREBUILD_ARCH=x64+arm64 in our macos build job.
As well as updating the binding.gyp to pass the relevant -arch options to OTHER_CFLAGS and OTHER_CPLUSPLUSFLAGS and OTHER_LDFLAGS.
We also brought in the MACOSX_DEPLOYMENT_TARGET based on what nodejs v16.14.2 is programmed with, I assume that this version can be bumped as we go along with new nodejs versions.
There are now some places where version numbers are being manually referenced. If we ever update v16.14.2 this will require also updating the:

MACOSX_DEPLOYMENT_TARGET in binding.gyp to what is provided in common.gypi. https://github.com/nodejs/node/blob/v16.14.2/common.gypi
The --target setting for prebuild script in package.json. I would have assumed that this was part of the nodejs being used, but on windows prebuildify is downloading additional cache for no reason.

The nodejs binary does not universal binaries as part of their release, so vercel/pkg is not fetching those. Although they are building it for the .pkg. So we will need to bring in the arm64 variant of the nodejs into our utils.nix but both can bundle the same native binary.
I believe we have successfully integrated prebuildify to all platforms, and once I run the staging jobs, it should show that all prebuilds are going to be uploaded as artifacts and passed to the integration:builds job.

I've added in the nix instructions to also bundle arm64. The lack of universal binary for nodejs and therefore pkg makes it difficult to do the integration test.
The integration:macos job runs on shared-macos-amd64.
When arm64 of macos becomes available in the shared runners, we can also run a separate integration:macos:arm64 job to also do integration test on the arm64 macs. For now any integration test has to be done manually on matrix-mac-1.

I'm not going to bother with 32 bit cross compilation atm. This is because 32 bit machines are most likely going out of business. So our current shell.nix that supports cross compilation of 32 bit is sort of unnecessary.
For now we have these platforms:

linux-x64
win-x64
macos-x64
macos-arm64

The only relevant other architectures would be linux-arm64 and win-arm64 and android and ios. See classic-level for more inspiration.
For now we can stop here, and focus on the remaining tasks:

Get pkg to properly bundle only the necessary assets, rather than all the assets.
Test our integration builds and our new CI/CD pipeline.
Test out our pre-release with npm and on gh
Test out the master branch release job with npm and gh

The expectation is that npm publish will no longer by the developer, it is done automatically as part of the CI/CD. If it is done by the developer, they have to manually assemble the prebuilds.

By monkey patching the os, we can trick node-gyp-build in telling us what binary it needs to load:
node - <<EOF
var os = require('os');
os.platform = () => 'win32';
os.arch = () => 'x64';
console.log(require("node-gyp-build").path());
EOF

If the build is not found, we get exit code of 1, and an error message.
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/node-gyp-build/index.js:60
  throw new Error('No native build was found for ' + target + '\n    loaded from: ' + dir + '\n')
  ^

Error: No native build was found for platform=win32 arch=x64 runtime=node abi=93 uv=1 libc=glibc node=16.14.2
    loaded from: /home/cmcdragonkai/Projects/TypeScript-Demo-Lib

    at Function.load.path (/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/node-gyp-build/index.js:60:9)
    at [stdin]:4:39
    at Script.runInThisContext (node:vm:129:12)
    at Object.runInThisContext (node:vm:305:38)
    at node:internal/process/execution:76:19
    at [stdin]-wrapper:6:22
    at evalScript (node:internal/process/execution:75:60)
    at node:internal/main/eval_stdin:29:5
    at Socket.<anonymous> (node:internal/process/execution:213:5)
    at Socket.emit (node:events:538:35)

If it is found, we get:
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/prebuilds/linux-x64/node.napi.node

We can then use this path when constructing the asset configuration that vercel pkg needs in order to ensure that pkg only bundles the right binary.
However, we would also need to do this on all the native modules that are used in our dependencies such as leveldown, utp-native, and fd-lock.
And to merge in any default assets to be saved like dist/**/*.json and the scripts like dist/lib/workers/worker.js.
We need npm to tell us which packages have native addons.

Using npm ls --all --prod --parseable we can see all the production dependencies (transitively) a list of paths:
[nix-shell:~/Projects/TypeScript-Demo-Lib]$ npm ls --all --prod --parseable
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/fd-lock
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/level
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/node-gyp-build
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/threads
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/utp-native
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/uuid
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/napi-macros
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/level-js
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/level-packager
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/leveldown
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/callsites
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/debug
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/is-observable
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/observable-fns
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/tiny-worker
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/readable-stream
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/timeout-refresh
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/unordered-set
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/abstract-leveldown
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/buffer
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/inherits
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/ltgt
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/run-parallel-limit
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/encoding-down
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/levelup
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/ms
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/esm
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/string_decoder
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/util-deprecate
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/catering
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/is-buffer
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/level-concat-iterator
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/level-supports
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/queue-microtask
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/base64-js
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/ieee754
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/level-codec
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/level-errors
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/deferred-leveldown
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/level-iterator-stream
/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/string_decoder/node_modules/safe-buffer

Using this list of paths, we can use find to help us find possible locations where native node addons exist.
Afterwards, we can use this to find out what node-gyp-build would actually load:
node_platform=win32
node_arch=x64
node_package=/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/fd-lock
node - <<EOF
var os = require('os');
os.platform = () => '$node_platform';
os.arch = () => '$node_arch';
console.log(require("node-gyp-build").path('$node_package'));
EOF

Due to nix's way of interpolating strings with '', it is possible to use this easily in our buildPhase.
Finally we just need to generate the pkg config, and pass it in via pkg . --config
The only issue here is that how does --config interact with the existing package.json. It's not clear. But most likely it overrides it completely. We would want to ensure common pkg configuration remains or to inherit the configuration so things like dist/**/*.json is included.
Note that by doing this, pkg may not be well usable inside nix-shell directly anymore. But that's ok, majority of the special configuration was done in Nix anyway. If we really wanted to do it properly, it would have to be specified as package.json script. Which would enable pkg bundling to be done on cross-platform during CI/CD similar to how we are using prebuildify.

Stripping out all the un-used native addons for pkg build mean that typescript-demo-lib went from 49 MiB to 42 MiB for the linux executable. This included the leveldown, fd-lock, and utp-native.

Inspecting the --debug flag of pkg has revealed that pkg is currently bundling all the unnecessary files in each dependencies which includes all the markdown files like CHANGELOG.md for every package. There's a discussion about this upstream: https://github.com/vercel/pkg/discussions/1377. It's likely not going to be fixed until pkg adds in some way of ignoring specific file patterns. I would expect that markdown files should be excluded.

It turns out that the pkg --config option can take a javascript file that is required by the pkg. But this is not documented, so I'm going to assume that I didn't read it so we can proceed with the current usage of pkg.
I've created scripts/pkg-build.js that is a script that can be executed that actually performs the pkg build.
We can add this into our npm run pkg scripts. Which would perform the pkg bundling and because it is written in nodejs code, it can be ran on all platforms too.
What this script does is that it uses node-gyp-build to find the relevant native addons paths, and then constructs a pkg-config-XXXXXX.json file will contain the actual pkg config.
Since pkg only takes package.json or the pkg config. I'm merging configuration from the package.json pkg configuration with the new configuration, that way additional static configuration can still be setup in package.json:
  "pkg": {
    "assets": [
      "dist/**/*.json"
    ],
    "scripts": [
      "dist/lib/workers/worker.js"
    ]
  },

This ends up becoming something like:
{
  assets: [
    'dist/**/*.json',
    '/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/prebuilds/linux-x64/node.napi.node',
    '/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/fd-lock/prebuilds/linux-x64/node.napi.node',
    '/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/utp-native/prebuilds/linux-x64/node.napi.node',
    '/home/cmcdragonkai/Projects/TypeScript-Demo-Lib/node_modules/leveldown/prebuilds/linux-x64/node.napi.glibc.node'
  ],
  scripts: [ 'dist/lib/workers/worker.js' ]
}

I still need to test if the absolute paths actually work.
There is how parameterisation that is required.
In our release.nix, it specifies utils.nodeVersion and arch and also the target platform. But the target platform string is not the same as the os.platform(). For example, os.platform() can return darwin, while pkg uses macos, same with win vs win32.
The parameters we are passing are:

The bin executable that we want to bundle
The os.platform() that we want to bundle for - default to the host platform
The os.arch() that we want to bundle for - default to the host arch
The node major version that we want to bundle for - default to the node execution

We will need to "map" the darwin platform to macos and win32 to win.

For posterity I wrote a shell script before the JS script.

#!/usr/bin/env bash

node_platform='linux'
node_arch='x64'

pkg_config() {

  # Inherit base pkg config
  cat <<- EOF
	{
	  "scripts": [
	    "dist/lib/workers/worker.js"
	  ],
	  "assets": [
	    "dist/**/*.json"
	EOF

  # Acquire platform-specific node-gyp-build assets
  while IFS= read -r node_package; do
    if [ \
      -n "$(find \
        "$node_package/prebuilds" \
        -name "*.node" \
        -print \
        -quit \
        2>/dev/null \
      )" \
    ]; then
      # print indentation and leading comma
      printf '    ,'
      node - <<- EOF
			const os = require('os');
			os.platform = () => '$node_platform';
			os.arch = () => '$node_arch';
			console.log('"' + require("node-gyp-build").path('$node_package') + '"');
			EOF
    fi
  done < <(npm ls --all --prod --parseable)

  # Finish the pkg config JSON
  cat <<- EOF
	  ]
	}
EOF

}

I've added npm run pkg script. Which runs the pkg-bundle.
We've sorted out the leveldown.js patch issue too so the nix override is now removed.
It's time to integrate this into release.nix.

This nix command has been very useful for debugging:
nix-build -E '((import ./pkgs.nix {}).callPackage ./utils.nix {}).node2nixDev'

The pkg is now integrated with the prebuilds.
We know that pkg is still bundling alot of other useless stuff, and also the leveldown.js is currently hardcoded into the scripts/pkg.js, that stuff should be overridden above or via an additional configuration. But this is sufficient for now, we'll create future tasks to clean that up.

Onto the final setup of the integration stage of CI/CD and npm publishing bot, as well as setting up the npm test to be capable of platform-specific tests.

This is now entirely unnecessary:
      # Some dependencies must match the Nix provided versions
      # because they are used during the build expressions
      # our nix-shell development environment should match those
      packages=(
<EMAIL_ADDRESS>      )
      for p in ''${packages[@]}; do
        npm ls --package-lock-only $p >/dev/null || \
        echo "WARNING: Build dependency version mismatch: $p"
      done

So we can now install whatever pkg or node-gyp we would like.

I've updated the scripts/pkg.js with a basic options parser. This is similar to the bash options parser documented here: https://stackoverflow.com/questions/192249/how-do-i-parse-command-line-arguments-in-bash.
It supports long options with = and space separated.
This makes it easier for package.json to specify --no-dict=leveldown.js to be ignored for packaging atm.
The release.nix expressions have been updated to now copy without overwrite the ./prebuilds directory in case there are builds there that sit outside the Nix build. This is the case with windows and macosx native addons.
Interestingly I took a look at mingw-w64 which is a toolchain that allows one to compile on linux to target windows. This toolchain currently isn't supported by node-gyp, and this is because native addons compiled with mingw-w64 is not binary/link compatible with msvc compiled code. Since node itself doesn't have a mingw-w64 compiled version, then producing native addons in mingw-w64 won't be useful. But if it did work, then it would have been nice to be able to compile for Linux and Windows from a single host system on Linux. Especially since nix doesn't work on Windows atm.
It's now time to test the CICD staging branch jobs.

Merged this into staging, seeing this pipeline diagram:

Had to add: export HOMEBREW_NO_AUTO_UPDATE=1 into the build:macos because brew install triggers auto updates which is just a waste of time. We should want to use nix on mac eventually (maybe even a container...?).

I may need to ensure that the cache between Windows, Mac and Linux is separated. Perhaps something like:
cache:
  key: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"

Which apparently has separate cache per-job and per-branch. The only issue is that some jobs should share. I'm not sure atm.

After stopping brew from auto updating, we came across a problem, the brew doesn't know about the node@16. I think it's looking at its local formula index. We need to update brew, but probably not update everything. Just the index.

Homebrew's update is so slow, it looks like we need to potentially swap to using Nix on darwin instead of homebrew.

For arm64 binary on macos:

Due to the mandatory code signing requirement, before the
executable is distributed to end users, it must be signed.
Otherwise, it will be immediately killed by kernel on launch.
An ad-hoc signature is sufficient.
To do that, run pkg on a Mac, or transfer the executable to a Mac
and run "codesign --sign - ", or (if you use Linux)
install "ldid" utility to PATH and then run pkg again

The ldid command seems strange, and not available normally, so we could do this with codesign --sign later.

I noticed that the windows integration run command doesn't fail even if when the command fails. I guess the ForEach is rather forgiving?
$ Get-ChildItem -File ./builds/*-win-* | ForEach {& $_.FullName}
pkg/prelude/bootstrap.js:1833
      throw error;
      ^
Error: No native build was found for platform=win32 arch=x64 runtime=node abi=93 uv=1 libc=glibc node=16.14.2
    loaded from: C:\snapshot\typescript-demo-lib\node_modules\leveldown
    at Function.load.path (C:\snapshot\typescript-demo-lib\node_modules\node-gyp-build\index.js:60:9)
    at load (C:\snapshot\typescript-demo-lib\node_modules\node-gyp-build\index.js:22:30)
    at Object.<anonymous> (C:\snapshot\typescript-demo-lib\node_modules\leveldown\binding.js:1:43)
    at Module._compile (node:internal/modules/cjs/loader:1103:14)
    at Module._compile (pkg/prelude/bootstrap.js:1894:32)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:11[57](https://gitlab.com/MatrixAI/open-source/typescript-demo-lib/-/jobs/2468682176#L57):10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at Module.require (pkg/prelude/bootstrap.js:1812:31)

Either that or pkg nodejs executable is not returning the correct exit code. We should check that on matrix-win-1.

The integration runs are still failing. I need to debug this locally, by seeing why our bundled executable isn't running.
Right now both windows and macos takes quite a bit a time to startup, we could aid this by changing the cache:

For chocolatey we can change the choco cache with choco config set cacheLocation fullPath.
For macosx HOMEBREW_CACHE which can be set to "$(pwd)/tmp/Homebrew".

This might be enough to proceed. Nix caching is even more difficult since fundamentally it's a network cache, and we aren't able to get gitlab to cache /nix/store since it is outside the project root. However by using our own s3 caching and the usage of the nixos upstream cache, the difference is probably marginal, and most importantly we can cache the base foundation in our docker image itself. So I'm wary of installing nix directly on the macos runner, but it's worth checking out to compare too.

It turns out xcode-select --install is a GUI program, so there's no point running:
xcode-select --install || true

Cause if xcode wasn't installed we would end up with this problem anyway.
The automatic ways of install of xcode is actually quite complicated (and very flaky, none of it is intended to be automatic), so for now we will simply expect that the gitlab runner already has xcode installed and we don't need to install them ourselves.
I found this out by trying to install the new brew on the newly update matrix-mac-1.

The integration runs are still failing. I need to debug this locally, by seeing why our bundled executable isn't running.
Right now both windows and macos takes quite a bit a time to startup, we could aid this by changing the cache:

For chocolatey we can change the choco cache with choco config set cacheLocation fullPath.
For macosx HOMEBREW_CACHE which can be set to "$(pwd)/tmp/Homebrew".

This might be enough to proceed. Nix caching is even more difficult since fundamentally it's a network cache, and we aren't able to get gitlab to cache /nix/store since it is outside the project root. However by using our own s3 caching and the usage of the nixos upstream cache, the difference is probably marginal, and most importantly we can cache the base foundation in our docker image itself. So I'm wary of installing nix directly on the macos runner, but it's worth checking out to compare too.

The caching didn't really save that much time for both windows and macos.

I noticed that the windows integration run command doesn't fail even if when the command fails. I guess the ForEach is rather forgiving?
$ Get-ChildItem -File ./builds/*-win-* | ForEach {& $_.FullName}
pkg/prelude/bootstrap.js:1833
      throw error;
      ^
Error: No native build was found for platform=win32 arch=x64 runtime=node abi=93 uv=1 libc=glibc node=16.14.2
    loaded from: C:\snapshot\typescript-demo-lib\node_modules\leveldown
    at Function.load.path (C:\snapshot\typescript-demo-lib\node_modules\node-gyp-build\index.js:60:9)
    at load (C:\snapshot\typescript-demo-lib\node_modules\node-gyp-build\index.js:22:30)
    at Object.<anonymous> (C:\snapshot\typescript-demo-lib\node_modules\leveldown\binding.js:1:43)
    at Module._compile (node:internal/modules/cjs/loader:1103:14)
    at Module._compile (pkg/prelude/bootstrap.js:1894:32)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:11[57](https://gitlab.com/MatrixAI/open-source/typescript-demo-lib/-/jobs/2468682176#L57):10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at Module.require (pkg/prelude/bootstrap.js:1812:31)

Either that or pkg nodejs executable is not returning the correct exit code. We should check that on matrix-win-1.
See: https://gitlab.com/MatrixAI/open-source/typescript-demo-lib/-/jobs/2468682176

The reason for this is because pkg configuration JSON has to:

Always relative paths, absolute paths will end up being put in the wrong place in the virtual filesystem
The configuration JSON file itself MUST be in the same directory as the project root, and together with package.json, otherwise relative paths are resolve relative to the location of the configuration file
The file must have the file extension .json. Otherwise it may load it like a javascript file.

For arm64 binary on macos:

Due to the mandatory code signing requirement, before the
executable is distributed to end users, it must be signed.
Otherwise, it will be immediately killed by kernel on launch.
An ad-hoc signature is sufficient.
To do that, run pkg on a Mac, or transfer the executable to a Mac
and run "codesign --sign - ", or (if you use Linux)
install "ldid" utility to PATH and then run pkg again

The ldid command seems strange, and not available normally, so we could do this with codesign --sign later.

I've requested it to be packaged at top level here: https://github.com/NixOS/nixpkgs/issues/173467.
But there's a fork of ldid available in pkgs.ios-cross-compile.ldid.

Apple now requires mandatory code signing for all executables, application bundles, packages. This applies all iOS and MacOS devices. Any program attempting execution without a legitimate signature is automatically closed.
In order to do code-signing on our CLI executable, we must acquire an X.509 certificate. It is called "Developer ID Application" certificate.
This certificate is necessary to distribute applications outside of the Mac app store.  This is because we are distributing this CLI executable directly out of GitHub release page.
Apple has other certificates intended for other distribution purposes. Each certificate requires its own keypair. You cannot use the same keypair for different certificates.
To acquire this certificate, we must have an account on developer.apple.com. We had already set this up back in 2020.
Although the X.509 certificate system can be done entirely in openssl. Apple's workflow is designed to be done in the GUI. So we will follow this process.
Launch the Keychain Access program.
By default there is already a <key> keypair, we will be creating a new keypair for this.
Click Keychain Access -> Certificate Assistant -> Request a Certificate From a Certificate Authority.
This will open up a panel where you can fill in:

User Email Address - put the company email
Common Name - MATRIX AI PTY. LTD.

The CA Email Address is unnecessary.
Select Saved to disk.
The keypair must be RSA 2048 dictated by Apple, this is by default, so we don't need to change anything for the keypair generation.
Once this is done, the keypair will appear in your Keychain Access program. And the CSR "Certificate Signing Request" file will appear where you saved it.
The keypair in your Keychain Access will be named according to the Common Name. This is quite ambiguous. So you need to rename both the public and private keys to MATRIX AI PTY. LTD. - Developer ID Application directly in the GUI. This is just a tag in the Keychain Access program, it will not affect anything else.
Then go https://developer.apple.com/account/resources/certificates/add and select Developer ID Application.
On the next page, select the latest Profile Type, and then provide the CSR file.
The certificate will then be generated, and you can download it to your computer.
Now you need to import this into your Keychain Access, this will import it into the default keychain, which should be login.
security import ~/Downloads/developerID_application.cer

Once this is done, you will be able to check whether the certificate was properly imported:
security find-identity

It should show the Developer ID Application: MATRIX AI PTY. LTD. (${TeamIdentifier}) where TeamIdentifier is a 10 alphanumeric characters. This is the actual Common Name string stored on the X.509 certificate. The Keychain Access GUI was lying to you!! Copy the full common name, we will need this later for code signing.
If you selected the latest profile type, it is possible that the Mac has an outdated Developer ID Certificate Authority intermediate certificate. To acquire the latest intermediate certificates, go to: https://apple.com/certificateauthority/. For the Developer ID Application certificate based on the G2 Sub-CA (Xcode 11.4.1 or later) profile, we need the Developer ID - G2 intermediate certificate. Download that, and import it with:
security import ./DeveloperIDG2CA.cer

Now in order to do code signing, we can follow the information in https://wiki.lazarus.freepascal.org/Code_Signing_for_macOS. The way we code sign changes depending on what we are code signing.
 We must do mandatory notarization. https://wiki.lazarus.freepascal.org/Notarization_for_macOS_10.14.5%2B. In fact code signing without notarization is actually useless.
...TBD
The codesign command only exists on MacOS, suppose we want to run this on different MacOS (such as in our CI/CD situation), we need to first transfer both the certificate and keypair to that computer before we can use in the new MacOS.
First we must export our keypair and certificate as p12 or "Personal Information Exchange" file. This file must contain both the certificate and the private key, the public key is not necessary.
...TBD
Alternatively on Linux we have the ldid command. I haven't tried to figure out how it works yet, we will leave that for the future.
Only a maximum of 5 developer ID certificates can be created. To revoke any of them it requires an support email to Apple. Go to https://developer.apple.com/contact/topic/select. It requires you be signed in as an Apple developer.
Additionally you can use these command line commands to manipulate the keypairs:
# Extract private key as encrypted PEM from p12 (it will now have a password)
openssl pkcs12 \
  -nocerts \
  -in './MATRIX AI PTY. LTD. - Developer ID Application.p12' \
  -out './MATRIX AI PTY. LTD. - Developer ID Application'

# Extract private key as unencrypted PEM from p12
openssl pkcs12 \
  -nocerts \
  -nodes \
  -in './MATRIX AI PTY. LTD. - Developer ID Application.p12' \
  -out './MATRIX AI PTY. LTD. - Developer ID Application'

# Generate public key from private key
openssl rsa \
  -pubout \
  -in './MATRIX AI PTY. LTD. - Developer ID Application' \
  -out './MATRIX AI PTY. LTD. - Developer ID Application.pub'

Everytime xcode is installed or updated, do this:
sudo xcode-select --reset

Otherwise the xcode command line tools might be out of date.
The codesign command only exists on MacOS, suppose we want to run this on different MacOS (such as in our CI/CD situation), we need to first transfer both the certificate and keypair to that computer before we can use in the new MacOS.
First we must export our keypair and certificate as p12 or "Personal Information Exchange" file. This file must contain both the certificate and the private key, the public key is not necessary.

Since I've confirmed that the staging setup works right now, I'm transitioning this PR to https://github.com/MatrixAI/TypeScript-Demo-Lib-Native

All relevant issues have been created in this repo and typescript-demo-lib-native. Remaining development will continue on https://github.com/MatrixAI/TypeScript-Demo-Lib-Native/pull/2. We will be requesting access to macos for that repository too.

We will have some changes brought in here after first merging in https://github.com/MatrixAI/TypeScript-Demo-Lib-Native/pull/2