Different conda environments

We need to think how the framework can install isolated conda environments which are application specific. This issue case up with the Hydronet benchmark which is very sensitive to library versions. If we don't install specific versions of libraries it will not work. The problem is that these dependencies might be in conflict with the previously installed libraries and it would be better to create a specific environment just for running Hydronet.
The Hydronet dependencies can be installed by the following commands:

Create conda environment
conda create --name hydronet2 python=3.8

Activate conda environment
activate conda hydronet2

Installing pytorch:
conda install pytorch==1.12.0 cudatoolkit=11.3 -c pytorch -c conda-forge

conda install pyg -c pyg

conda install -c conda-forge tensorboard ase fair-research-login h5py tqdm

conda install -c conda-forge gdown

pip install torch-scatter torch-sparse torch-cluster torch-spline-conv torch-geometric -f https://data.pyg.org/whl/torch-1.12.0+cu113.html

Yes I agree this is an issue. As discussed I think there are two options:

Add conda as a requirement and then use conda to create a different environment for each benchmark.
Do it with containers (e.g. singularity) and each benchmark gets a container. However this can be headache when it comes to multi-gpu, multi-node runs.

Keeping track of benchmark requirements: here is a script for correctly installing tensorflow & requirements for the mnist benchmark. Other tensorflow benchmarks (optics, cloud, etc.) will be similar and will mostly differ in the last line.
#!/bin/bash
set -x

# Create new environment
ENV_NAME=sciml-bench-mnist_tf_keras
conda remove -n $ENV_NAME --all -y --quiet
conda create -n $ENV_NAME python=3.9 -y --quiet
ENV_PATH=$(dirname $(dirname /home/lhs18285/miniconda3/bin/conda))/envs/$ENV_NAME

# Install conda requirements
conda install -n $ENV_NAME -c conda-forge cudatoolkit=11.2.2 cudnn=8.1.0 -y --quiet
conda install -n $ENV_NAME -c nvidia cuda-nvcc=11.3.58 -y --quiet

# Configure environment variables
mkdir -p $ENV_PATH/etc/conda/activate.d
echo "export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${ENV_PATH}/lib/" >> $ENV_PATH/etc/conda/activate.d/env_vars.sh
echo "export XLA_FLAGS='--xla_gpu_cuda_data_dir=${ENV_PATH}/lib'" >> $ENV_PATH/etc/conda/activate.d/env_vars.sh

# Work around for Ubuntu 22.04. See: https://www.tensorflow.org/install/pip
mkdir -p $ENV_PATH/lib/nvvm/libdevice
cp $ENV_PATH/lib/libdevice.10.bc $ENV_PATH/lib/nvvm/libdevice/

# Install pip requirements
conda run -n $ENV_NAME LD_LIBRARY_PATH=$ENV_PATH/lib/ python -m pip install --upgrade pip -q
conda run -n $ENV_NAME LD_LIBRARY_PATH=$ENV_PATH/lib/ python -m pip install . -q
conda run -n $ENV_NAME LD_LIBRARY_PATH=$ENV_PATH/lib/ python -m pip install "tensorflow==2.11.*" scikit-image -q

And here's the script for stemdl (and pytorch). It also includes fixing to the correct pytoch_lightning version.
#!/bin/bash
set -e
set -x

# Create new environment
ENV_NAME=sciml-bench-stemdl_classification
conda remove -n $ENV_NAME --all -y --quiet
conda create -n $ENV_NAME python=3.9 -y --quiet
ENV_PATH=$(dirname $(dirname /home/lhs18285/miniconda3/bin/conda))/envs/$ENV_NAME

# Install conda requirements
conda install -n $ENV_NAME -y --quiet pytorch==1.13.1 torchvision==0.14.1 pytorch-cuda=11.6 -c pytorch -c nvidia

# Install pip requirements
conda run -n $ENV_NAME python -m pip install -q --upgrade pip
conda run -n $ENV_NAME python -m pip install -q "pytorch_lightning==1.9.*" scikit-learn tensorboard
conda run -n $ENV_NAME python -m pip install -q .

I started capturing install scripts for each environment in: dev/install_scripts/*.sh. They are quite useful for testing & will be a useful documentation of the dependencies for whatever refactoring solution we design in future,


[Discussion] Using create-playwright on macOS 10.13

Hi all.
I'm arguably on an old system but can not upgrade to a later macOS.
Because the latest supported OS on this hardware is macOS 10.13.6 High Sierra, I'm currently fixed on the following versions (+ indicates I'm able to use the latest version and can probably continue to upgrade for a while):

nodejs: 16.x (npm 8.19.3)
Corepack: 0.15.1
Firefox: 108+
Chromium: 109+
AppleWebKit: 605.1.15

There is already a closed PR, https://github.com/microsoft/playwright/issues/1941#issuecomment-1008338376, with a work-around for npm init playwright@latest failing on macOS 10.13.
I would like to contribute to a better Developer eXperience (DX) for us on old Apple hardware, if possible.
Below is my notes for getting Playwright to work on 10.13. Any of the sections could be turned into a PR, if desired by you. I don't see any guidelines for what is supported other than comments here and there.
ERROR: Playwright does not support webkit on mac10.13
Steps to reproduce
npm init playwright@latest
Result
Error: ERROR: Playwright does not support webkit on mac10.13
Error: Command failed: npx playwright install
    at checkExecSyncError (node:child_process:861:11)
    at execSync (node:child_process:932:15)
    at executeCommands (/Volumes/mystuff/dotnet/.npm/_npx/d352e76cc6b4974c/node_modules/create-playwright/lib/index.js:4519:39)
    at Generator.run (/Volumes/mystuff/dotnet/.npm/_npx/d352e76cc6b4974c/node_modules/create-playwright/lib/index.js:4660:5)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async /Volumes/mystuff/dotnet/.npm/_npx/d352e76cc6b4974c/node_modules/create-playwright/lib/index.js:4928:3 {
  status: 1,
  signal: null,
  output: [ null, null, null ],
  pid: 2004,
  stdout: null,
  stderr: null
}

create-playwright npm test
Steps to reproduce
npm i && npm test
Result
All of the tests fails.
1) [NPM] › integration.spec.ts:20:1 › should generate a project in the current directory =========

    Error: expect(received).toBe(expected) // Object.is equality

    Expected: 0
    Received: 1

      21 |   test.slow();
      22 |   const { exitCode, dir, stdout } = await run([], { installGitHubActions: true, testDir: 'tests', language: 'TypeScript', installPlaywrightDependencies: false, installPlaywrightBrowsers: true });
    > 23 |   expect(exitCode).toBe(0);

The same error happens in all of the tests.
Possible solution
Since the test output does not say what why all of the tests failed, I added a stderr property to type RunResult and in integration.spec.ts I print to console.error if exitCode is not above 0.
I could then see that the tests expects yarn and pnpm to be globally installed. I don't know if it makes sense to add package managers to the devDependencies for create-playwright but I'm assuming it does not.
A fix could be to just write that npm, yarn and pnpm is required for the tests to run, in the README.
After installing yarn 2 and still failing all tests, I took a peek at what the macOS github action is using and it turns out to be yarn@1. This should also be clear in the README.
The steps required, according to .github/workflows/.ci.yml, are the following:

npm i -g npm@8
npm i -g yarn@1
npm i -g pnpm@7
npm ci
npx playwright install-deps
npm run build
npx tsc --noEmit
npm test

Despite successfully doing the above steps, all of the tests still fails. Fortunately, with the stderr property, I am now able to see which commands fails.
The first one being, npx playwright install, which brings be back to my original issue, "Playwright does not support webkit on mac10.13".
Running the test with PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=true npm test gives the same errors.
It seems that I can not get any further without looking into the source of microsoft/playwright...
I will update this issue with my findings :)
Related issues:

1535
1941

Right, after looking through the source in playwright, I see that a lot of things has changed since the before mentioned issues - I simply can not find out where add logic for which browsers to download by default.
I think it is somewhere in playwright/packages/playwright-core/src/server/registry/index.ts. Preferably it should not be logic somewhere in the belly of playwright but flags/arguments to playwright CLI with logic in create-playwright.

Supporting macOS 10.15 or lower is unfortunately out of scope, since e.g. WebKit does not support them anymore due to their minimal xcode version constrains.
macOS 10.15 does no longer receive security updates by Apple as well, so we recommend updating.
Closing since this is not actionable for us.

I can not update. I would have to throw out my hardware and buy new... I understand that Microsoft don't want to support outdated OS's but the community could support macOS 10.13+. The latest webkit for iOS 15.7.2 (webkit 605.1.15), is also supported on 10.13 - I am running Safari 13.1.2 AppleWebKit/605.1.15.

WebKit does not support them anymore due to their minimal xcode version constrains.

@mxschmitt how does the xcode version come into play with playwright? I see relative download paths to a builds/ directory but fail to get a sample of the corresponding .zip file.

Also, it strikes me that playwright is also very useful on macoS 10.13 for Chromium and FIrefox browsers, which runs just fine.

Alternative if playwright does not run on old hardware, then there is only karma and puppeteer - though I can not find supported OS's in Puppeteer's documentation.
@mxschmitt can you confirm that cross-platform testing (for developer machines) would be possible with Docker?

The latest WebKit version is 16.2, which is much more recent than yours, see here. 605.1.15 is not the version number.

@mxschmitt that's not strictly true. The version numbers can be found here: https://github.com/WebKit/WebKit/blob/main/Configurations/Version.xcconfig
But you're correct that my webkit version is not the latest. It is however, the latest on iOS 15.7 which is still supported by Apple.
Again, I fail to see why you impose this requirement to playwright? Is it because you do not have old macOS's to test on? Is it because playwright depends on a specific xcode version - if yes, where and how?
Sorry about all this questions - but I'm very confused to why playwright would not work on macOS 10.13-10.15.
Previously (in August 2022) Version.xcconfig was here: https://trac.webkit.org/browser/trunk/Source/WebCore/Configurations/Version.xcconfig
Versions were unified in these two PR's: https://bugs.webkit.org/show_bug.cgi?id=244450 (https://github.com/WebKit/WebKit/pull/3747) and https://bugs.webkit.org/show_bug.cgi?id=249718 (https://github.com/WebKit/WebKit/pull/7958).


Remove the global comment flag and made it toggle the new comment flag

Remove the global comment flag
Made the flag for new comments toggle the newly posted comment's flag

Good job @hammerfunk :+1: