Add HSM integration

This PR depends on https://github.com/PegaSysEng/signers/pull/79 and dependencies need to be adjusted accordingly.
Prerequisites
HSM (PKCS11) tests are run against an HSM emulator called SoftHSM and Cavium (AWS CloudHSM) tests require a real AWS CloudHSM client to be set up locally.
Below are some instructions that might help in setting this up.
SoftHSM
You can install and initialize SoftHSM locally as below.
Linux
As a prerequisite you might need to install the following dependencies if not already present
apt install gcc make automake libtool autoconf pkg-config libssl-dev -y

Clone the repo
git clone https://github.com/opendnssec/SoftHSMv2
cd SoftHSMv2/
sh autogen.sh
./configure
make
make install

The last command may require super user permissions in which case one should also do
sudo chmod 1777 /var/lib/softhsm

Mac
As a prerequisite you might need to install the libtool dependency if not already present
brew install libtool
brew install softhsm

Initialize SoftHSM
softhsm2-util --show-slots

Look at the listed slots and determine the next slot index by taking the index of the last slot listed. Make sure you
do not have duplicated slots with the same label as this might cause problems. If there are only one default slot listed
(as it would be for a fresh SoftHSM installation) you will see Slot 0 displayed, and you should start from --slot 0
in the commands below. If you already have slots initialized and want to remove them first, you can use
softhsm2-util --delete-token --serial <Serial Number> to remove them. You will notice that the index of the last listed
(empty) slot will be decremented.
softhsm2-util --init-token --slot 0 --label WALLET-000 --pin us3rs3cur3 --so-pin sup3rs3cur3
softhsm2-util --init-token --slot 1 --label WALLET-001 --pin us3rs3cur3 --so-pin sup3rs3cur3
softhsm2-util --init-token --slot 2 --label WALLET-002 --pin us3rs3cur3 --so-pin sup3rs3cur3
softhsm2-util --init-token --slot 3 --label WALLET-003 --pin us3rs3cur3 --so-pin sup3rs3cur3
softhsm2-util --init-token --slot 4 --label cavium --pin us3r:s3cur3 --so-pin sup3r:s3cur3

AWS CloudHSM
First thing to set up is your CloudHSM client. You can follow the instructions in their guides to install the latest version:
https://docs.aws.amazon.com/cloudhsm/latest/userguide/install-and-configure-client-linux.html
On Linux you also need the libjson library as a prerequisite.
wget http://mirrors.kernel.org/ubuntu/pool/main/j/json-c/libjson-c2_0.11-4ubuntu2_amd64.deb
sudo dpkg -i libjson-c2_0.11-4ubuntu2_amd64.deb
wget https://s3.amazonaws.com/cloudhsmv2-software/CloudHsmClient/Xenial/cloudhsm-client_latest_amd64.deb
sudo dpkg -i cloudhsm-client_latest_amd64.deb

You will need to get a AWS certificate to be able to connect to your AWS CloudHSM cluster. This should be copied to your installation directory. Eg. /opt/cloudhsm/etc/customerCA.crt. You will also need to know the CloudHSM's IP with which you can configure our client:
sudo service cloudhsm-client stop
sudo /opt/cloudhsm/bin/configure -a <IP address>
sudo service cloudhsm-client start

Most HSM as a service providers support a proprietary interface and a generic interface called PKCS11. PKCS11 is just a fancy term for a standard way to talk to a HSM.
AWS supports both prop and PKCS#11 interaction. We favour the use of PKCS11 as it uses a generic implementation where the prop interface requires a custom implementation.
It worked in our Golang backend but not in Java where we needed to add a custom implementation. You only need the JCE provider for EthSigner.
wget https://s3.amazonaws.com/cloudhsmv2-software/CloudHsmClient/Xenial/cloudhsm-client-pkcs11_latest_amd64.deb
sudo dpkg -i cloudhsm-client-pkcs11_latest_amd64.deb
wget https://s3.amazonaws.com/cloudhsmv2-software/CloudHsmClient/Xenial/cloudhsm-client-jce_latest_amd64.deb
sudo dpkg -i cloudhsm-client-jce_latest_amd64.deb

AWS provides two main console applications by which you can perform administrative tasks on HSMs in the cluster. To add new users you can use the CloudHSM management utility:
/opt/cloudhsm/bin/cloudhsm_mgmt_util /opt/cloudhsm/etc/cloudhsm_mgmt_util.cfg

We have set up a dummy user (called alice) that can be used in our tests.
aws-cloudhsm>loginHSM CO admin <password>
aws-cloudhsm>createUser CU alice <password>

To work with keys you can use the key management console application:
/opt/cloudhsm/bin/key_mgmt_util

The HSMs in the cluster all pick up as one distributed slot so there is no need to set up dedicated slots like with SoftHSM.
A docker file was also included that has the CloudHSM client installed.
Account generation
We created an account generation utility from extracts of (an outdated) EthSigner, available at https://github.com/Adhara-Tech/accountgenerator.
Account generation did not fit into EthSigner, but we still wanted the user to be able to integrate it easily alongside EthSigner.
Key Management Services like Azure Vault and Hashicorp Vault allows for an external party to generate their own keys, provide a client application with the correct signing access rights, along with the alias (or name) of the key, and you can sign transactions. The alias is added to a toml file configuration that EthSigner requires to be able to sign with that key. In order to get away from managing config files for every key in our own signing solution, we use the checksumed Ethereum address as alias for keys stored in an HSM or Cloud-based service.
In theory, an external party can still generate their own keys, provided that they set the labels of the key correctly.
Build AccountGenerator
git clone git@github.com:Adhara-Tech/accountgenerator.git
cd accountgenerator
./gradlew clean build
cd build/distributions/
tar -xzf accountgenerator-<version>.tar.gz
cd accountgenerator-<version>/
bin/accountgenerator --help

Start AccountGenerator
The AccountGenerator is a JSON RPC server exposing a single eth_generateAccount endpoint. A listen address and a
path to a writable directory are required when starting up the AccountGenerator. The subcommand input specifies where
the key should be generated as illustrated in the examples below. A configuration to be used by EthSigner will be generated
in the specified directory. Mutual TLS can be enabled on the AccountGenerator in the same way as documented by EthSigner.
Keystore
accountgenerator
    --http-listen-host=<IP_ADDRESS>
    --http-listen-port=4542
    --logging="INFO"
    --directory="../../ethsigner/keys"
      file-based-account-generator
        --password-file="../../ethsigner/keys/passwordFile"

SoftHSM
accountgenerator
    --http-listen-host=<IP_ADDRESS>
    --http-listen-port=4542
    --logging="INFO"
    --directory="../../ethsigner/keys"
      hsm-account-generator
        --library="/usr/local/lib/softhsm/libsofthsm2.so"
        --slot-label="WALLET-001"
        --slot-pin="us3rs3cur3"

AWS CloudHSM
accountgenerator
    --http-listen-host=<IP_ADDRESS>
    --http-listen-port=4542
    --logging="INFO"
    --directory="../../ethsigner/keys"
      cavium-account-generator
        --library="/opt/cloudhsm/lib/libcloudhsm_pkcs11.so"
        --slot-pin="alice:391019314"
        --sas="../scripts/sas.sh"

Once the AccountGenerator is running, one can request a new account to be generated via the command line
curl -w "\n" -X POST --data '{"jsonrpc":"2.0","method":"eth_generateAccount","params":[],"id":1}' http://<IP_ADDRESS>:4542

Generated toml files for HSM storage will generically have the form:
[metadata]
createdAt = 2020-08-19T12:33:06.114490Z
description = "HSM configuration"

[signing]
type = "hsm-signer"
address = "0xeD042EDEB7e721eA48F304ff551b9f90574F6De7"
slot = "WALLET-001"

Generated toml files for AWS CloudHSM storage will have the form:
[metadata]
createdAt = 2020-08-19T14:55:53.004576Z
description = "Cavium configuration"

[signing]
type = "cavium-signer"
address = "0x97Ad4eCdebc5E477518eaC0FC96844D686669f47"

The private part of the generated EC key pair has its label set to the derived Ethereum address, and the public part of the
key has its label set to an alteration of the address by using 1x instead of 0x. This workaround was put in place solely cause AWS CloudHSM does not allow you to have the same label assigned to both its public and private parts when using a JCE provider.
Ethereum Signing
We always use EthSigner with the multikey-signer subcommand. Single key signing subcommands were extended just to be consistent.
For HSM-based solutions we had to change the design a little due to security and performance concerns. Instead of adding credentials in every key's toml file (like the other signing solutions), we let the multikey-signer take in a config file containing the slot (or client) credentials. If the tables in the config file are empty, we try to read it from environment variables. This is used to initialise the signer factories, e.g. initialise the PKCS11 module or set up a cloud-based client. It does limit one instance of Ethsigner to only be able to sign out of one HSM (actual hardware) slot or one AWS CloudHSM cluster or one Azure vault. For our use of Ethsigner, where a client will only be using one type of key storage at a time, this was acceptable. Storing client connection credentials (or passwords) in the key's toml files also meant that EthSigner had to initialise the module and log into a slot for every transaction it wants to sign (comparable with creating a new Azure vault client for each signature) which yielded such low performance it was not usable.
Starting the service for multiple key signing require the normal parameters along with a config file
that will differ depending on the key storage we want to use for signing. Ensure that the configuration (toml) files generated
by the AccountGenerator utility is present in the directory that EthSigner is started up with.
ethsigner
    --chain-id=44844
    --http-listen-host=<IP_ADDRESS>
    --http-listen-port=9545
    --downstream-http-host=<IP_ADDRESS>
    --downstream-http-port=8545
    --downstream-http-request-timeout=30000
    --logging="DEBUG"
      multikey-signer
        --directory="keys"
        --config="/path/to/config/file"

Below we list various configuration files depending on different vendors.
Keystore
A config file is not needed. Works as stock-standard EthSigner

SoftHSM
[hsm-signer]
library = "/usr/local/lib/softhsm/libsofthsm2.so"
slot = "WALLET-005"
pin = "us3rs3cur3"

AWS CloudHSM
[cavium-signer]
library = "/opt/cloudhsm/lib/libcloudhsm_pkcs11.so"
pin = "alice:391019314"

We will try to read environment variables if an empty table is present in the config file. Eg with
[hsm-signer]

it will try to read
library = System.getenv("PKCS11_HSM_LIB")
slot = System.getenv("PKCS11_HSM_SLOT")
pin = System.getenv("PKCS11_HSM_PIN")

and with
[cavium-signer]

it will try to read
library = System.getenv("AWS_HSM_LIB")
pin = System.getenv("AWS_HSM_PIN")

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Collision detection for PBC in DEM and CFD-DEM

Description of the problem

Mapping of periodic boundaries and particle displacement was added in the PR #557
This previous PR did not consider collision detection of particles in periodic boundaries

Description of the solution

Enabling the collision detection for particles in the cells of the periodic boundaries
The particle containers and the way it is implemented is really similar to the way current collision detection are (broad search, updating of candidate, fine search, force calculation, ...). But it needed proper containers since we need sort of tracking of the particles that has periodic candidates in periodic boundary for the neighbor threshold search and the displacement with offset.
3 new types of particle collisions : periodic local-local collisions,  periodic local-ghost collisions & periodic ghost-local collisions.
One direction periodic mapping : periodic cells or periodic collisions are only mapped related to the cells/particles on the periodic boundary 0 (this is why there's local-ghost & ghost local containers)
It is not possible to use more than one pair of periodic boundaries now

How Has This Been Tested?

Add the periodic_boundary_collisions application test for DEM where 2 particles collide in the periodic boundaries (mpirun=1 & mpirun=2)
periodic_multiple_boundaries is removed since we don't support more than 1 pair of periodic boundaries
All other tests worked

Future changes

Need some clean up in contact forces code with ContactType to remove recurring code in future PR
Find a cleaner way to get offset
Add some content to the DEM hopper example to show the periodic boundary options

@blaisb

[x] Fix code for DEM tests
[x] Add basic documentation in the Lethe wiki for PBC
[x] Improve description of functions in code
[ ] Modify 3d hopper example (will be in another PR)


Unified ARA & Franklin-Paxinos atlas

Continuing #22
Added non-terminal mesh generation, but need to check that this still works properly.

As this atlas can be the same resolution as the ARA, it can be much bigger (100x) than other atlases that need the meshes extracting automatically (like the rat atlas). The mesh extraction for high resolutions can therefore be slow. Theoretically we shouldn't have to do this very often, so do we live with it taking a long time (potentially hours), or do we:

Try to parallelise across CPU cores
Find a quicker way to extract the surface
Use lower resolution meshes for all atlas resolutions
Some combination of the above

The step that is slow, in the code I'm setting up for extracting the mesh, is the smoothing.
Everything else happens in <1 minute.
There might be clever ways to either speedup the smoothing or to find a way to smooth the decimated mesh. Though this last step introduces more artifacts.
Performing in parallel should also be fine, the smoothing is a bit memory hungry but it should be okay. I don't know if it's worth setting it up to run on the HPC even.
But otherwise I don't mind it being slow. As you said it should be a one off per atlas.

The step that is slow, in the code I'm setting up for extracting the mesh, is the smoothing.
Everything else happens in <1 minute.

Is 1 minute for all regions or for one? Extracting 1400 regions for the 10um mouse atlas looks like it will take ages.

have a look at the code here: https://github.com/brainglobe/brainatlas-api/pull/27
maybe you can find a way to speed things up

This is ready to review, it all seems to work ok, but I haven't fully tested it at 10um. It should be easy to speed up via parallelisation across cores, but I need to see how much RAM it uses.

Looks good. How do the meshes look?
I'm testing my mesh creation code, without the smoothing the 'root' mesh takes 40s. I think that smaller meshes will be faster though.

Looks good. How do the meshes look?

They're ok, but not perfect. The original atlas was only defined at 50um in Z. When Steve and I made the 10um atlas, we didn't interpolate in Z, so it's a bit blocky in that axis.

I think as it is currently, the mesh extraction uses too much RAM to be usefully parallelised.

The blockiness might be improved using mesh_utils.extract_mesh_from_mask with smooth=True from https://github.com/brainglobe/brainatlas-api/pull/27

I've added a function to convert the json to a cellfinder-compatible csv. I haven't added this to the other atlas generation scripts yet, assuming we're going to extract common functionality from these.

nice, looks good. We can add to atlas_gen.metadata_utils a function to save the .csv file with the other text files.
Can the structures in the .csv file be in alphabetical order?

Can the structures in the .csv file be in alphabetical order?

Done

Should we merge?
Also, are you working on code to make meshes for non-terminal nodes or do you want me to?

Yep, this is just finished. This code generates meshes for terminal nodes, it can probably be copied into the other atlases, but I haven't fully checked that everything works.

it will need some changes because I've improved the mesh generation function, but it should be just a matter of changing the name function.  I'm also adding methods to look at meshes and volume side by side to make sure it looks good.

The acronyms in this atlas are a bit weird, as is the hierarchy. Also considering the meshes are probably not going to be as nice as the ARA ones, somewhere we should suggest that people use the ARA unless they specifically need the extra segmentation.

Are you happy if we merge this, then we merge your PR?

yeah let's merge this first, mine needs some more work


Gun Crash / Door Change

Seed: R520-LCN1-ZTNE26JLL631

If you use a weapon in this room, the game will crash

same thing in this room
So this is a weird one but without debug menu i'd be softlocked, leon A

If i take this route, the game went to normal for about 4 rooms and then randomizes back to how it should be, but my way back is also changed softlocking me between 2 dead ends

even more crashes if i try use weapon

so it first started happening with the magnum, so i switched to the GL an just exactly same thing so just figured it was all weapons

anyway hope this information helps with future updates, thanks so much guys!!


[#130] Fix for EarthChem contributors limit

@sblack-usu - We'd like to push this change as a hotfix.


chore: enable AREnableSignupLoginFusion

This PR enables the AREnableSignupLoginFusion feature flag, which will display the new auth flow to users. It also has an accompanying Echo flag: artsy/echo#687
PR Checklist

[ ] I have tested my changes on iOS and Android.
[ ] I hid my changes behind a feature flag, or they don't need one.
[ ] I have included screenshots or videos, or I have not changed the UI.
[ ] I have added tests, or my changes don't require any.
[ ] I added an app state migration, or my changes do not require one.
[ ] I have documented any follow-up work that this PR will require, or it does not require any.
[ ] I have added a changelog entry below, or my changes do not require one.

To the reviewers 👀

[ ] I would like at least one of the reviewers to run this PR on the simulator or device.

Changelog updates
Changelog updates

Cross-platform user-facing changes

Display a redesigned sign up and login flow to users - iskounen

iOS user-facing changes

Android user-facing changes

Dev changes

Need help with something? Have a look at our docs, or get in touch with us.

This PR contains the following changes:

Cross-platform user-facing changes (Display a redesigned sign up and login flow to users - iskounen - iskounen)

  Generated by :no_entry_sign: dangerJS against bb44f29966c7c5610577b1032bcd602148ea7c12


ENHS: additional tool types

implement additional tool types for ENHS, eg. enhancement spot definitions(HOOK_DEF)

can we close this? Or is something else still missing?

lets just close


Feature/checkbox slot - [merged]

In GitLab by @VitorLamego on Aug 31, 2023, 23:12
Merges feature/checkbox-slot -> develop
Gitlab MR Web Template
Descrição
Esta MR é responsável por adicionar slots no componente Checkbox.

In GitLab by @VitorLamego on Sep 1, 2023, 13:05
added 1 commit
44788c42 - Checkbox (style): remove unused style
Compare with previous version

In GitLab by @DaniloRaposo on Sep 1, 2023, 13:13
requested review from @DaniloRaposo

In GitLab by @DaniloRaposo on Sep 1, 2023, 19:47
added 7 commits
44788c42...66434a12 - 5 commits from branch developd9498e5b - Merge branch 'develop' into feature/checkbox-slotb06b8c62 - changelog (chore): add change log
Compare with previous version

In GitLab by @DaniloRaposo on Sep 1, 2023, 13:13
approved this merge request

In GitLab by @DaniloRaposo on Sep 1, 2023, 20:06
mentioned in commit 4f1bef699883040424e2b14c382d4c4f2843c573


Add Corporacion Universitaria Iberoamericana

Hi, I hope you're doing well.
I'd like to kindly request that my university's domain is added to the list. As requested, I've added the .txt file. Additionally, here are the:

Institution's official website: https://www.ibero.edu.co/
IT-related course link: https://www.ibero.edu.co/programas/pregrado-ingenieria-de-software-virtual
University domain recognition proof: https://www.ibero.edu.co/blog/noticias/cambiamos-nuestro-dominio-de-correo-electronico

Please let me know if this works. Have a great day!

@sacuervo I am sorry, the domain ibero.edu.co is removed from our trusted domains list because it was abused in the past. If you are the student of this school, please enroll to GitHub Student Developer Program. Then apply for JetBrains educational subscription with your GitHub account (GitHub tab at https://www.jetbrains.com/shop/eform/students).

@philipto, ok I understand. Thanks for your help👍


chore: Included githubactions in the dependabot config

This should help with keeping the GitHub actions updated on new releases. This will also help with keeping it secure.
Dependabot helps in keeping the supply chain secure https://docs.github.com/en/code-security/dependabot
GitHub actions up to date https://docs.github.com/en/code-security/dependabot/working-with-dependabot/keeping-your-actions-up-to-date-with-dependabot
https://github.com/ossf/scorecard/blob/main/docs/checks.md#dependency-update-tool
Signed-off-by: neilnaveen<EMAIL_ADDRESS>

Thanks, but GitHub actions are currently disabled because they just won't work. Closing.


Installation from conda-forge

Hey @slundberg ,
I want to enable SHAP to be installed from conda, so I was going to add it to conda-forge.  Do you want to be added as a recipe maintainer ?  The consequence is that you will get emails from the conda-forge bots if there are issues.
Best,
Andrew

Sounds great. Go ahead and add me. I just pushed some updates that merge the iml package into shap and so remove the issues you encountered (because iml is not loner a dependency). I think everything will be cleaner this way since we didn't end up sharing code with LIME (which motivated the initial separation into iml and shap).

Thanks for the latest releases @slundberg , the checks passed for conda-forge for staged-recipes, it can take up to a week for them to merge https://github.com/conda-forge/staged-recipes/pull/6406

great. thanks!

Hey @slundberg, I think we are go to close this out!
So the staged-recipes PR was merged and the feedstock setup. I was able to run without issues:
conda config --add channels conda-forge
conda install shap

Which installed v0.22.2.
It also detected the version bump from your last release, and auto created a PR which I merged: https://github.com/conda-forge/shap-feedstock/pull/1 .  It updated automatically and ran via terminal.  It's also visible here: https://anaconda.org/conda-forge/shap
The following packages will be UPDATED:
shap: v0.22.2-py27h7eb728f_0 conda-forge --> v0.22.3-py27h7eb728f_0 conda-forge

Proceed ([y]/n)? y
Downloading and Extracting Packages
shap v0.22.3: ################################################################################ | 100%

https://github.com/slundberg/shap/pull/212

@mrandrewandrade Something has gone wrong with the conda-forge process, would you mind taking a look? I think it had to do with the update for python3.7 which I didn't notice to merge right away.