try new building strategy

Checklist

[ ] Used a personal fork of the feedstock to propose changes
[ ] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[ ] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[ ] Ensured the license file is being packaged.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

The depot we package should only contain two directories.

packages
artifacts

That's why I created a temporary directory for the depot and then only copied those two directories.

Hi! This is the friendly automated conda-forge-linting service.
I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.
Here's what I've got...
For recipe:

Non noarch packages should have python requirement without any version constraints.
Non noarch packages should have python requirement without any version constraints.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

How do you want to populate the new depot? I don't see code to do that at the moment.

Like environments, the depots also stack. With the new depot, we will deliver a set of packages and a preconfigured environment. This will be a read-only environment behind the main one.

We have to create a new depot otherwise Julia packages within the depot might collide with other Julia packages installed by other conda-forge packages.

So the idea is for each package to have its own depot in the hierarchy? What if someone has both pysr and xbitinfo in the same conda env? Which one do we choose to satisfy?

That's the beauty of this. Both depots are stacked below the main one. All the depots do is serve as a versioned package store. Julia can pull packages from both depots if needed. Julia will only write to the main depot named after the conda environment.

I've got commits pending after we see the results of this run.

Do you know what this is about?

julia.core.JuliaError: Exception 'failed to clone from https://github.com/MilesCranmer/SymbolicRegression.jl, error: GitError(Code:ERROR, Class:SSL, Your Julia is built with a SSL/TLS engine that libgit2 doesn't know how to configure to use a file or directory of certificate authority roots, but your environment specifies one via the JULIA_SSL_CA_ROOTS_PATH variable. If you believe your system's root certificates are safe to use, you can export JULIA_SSL_CA_ROOTS_PATH="" in your environment to use those instead.)' occurred while calling julia code:

We may need to set the environment variable JULIA_PKG_USE_CLI_GIT=true then?

Are the activate scripts functional before the tests?

This run doesn't work, same error. The rm -rf "${FAKEDEPOT}" doesn't seem to be working correctly, as it is still packaging something like share/SymbolicRegression.jl.

This the intention. It should package share/SymbolicRegression.jl/depot, which contains all the dependencies for the package.
This way we can package the high level packages first, and then eventually do the low levels ones as needed.

There are two binary artifacts installed.

libopenspecfun from
https://github.com/JuliaBinaryWrappers/OpenSpecFun_jll.jl
conda-forge should be able to replace this with https://github.com/conda-forge/openspecfun-feedstock

LLVMExtra from
https://github.com/JuliaBinaryWrappers/LLVMExtra_jll.jl
https://github.com/maleadt/LLVM.jl
https://github.com/JuliaPackaging/Yggdrasil/blob/e99675ef8f0c852e163dbf39ecd2f4dd246d884c/L/LLVMExtra/build_tarballs.jl

Also a note to @MilesCranmer , this allows us to install a Manifest.toml as part of the depot.

Here's some reading about how the depot stack works.
https://docs.julialang.org/en/v1/manual/environment-variables/#JULIA_DEPOT_PATH
https://docs.julialang.org/en/v1/base/constants/#Base.DEPOT_PATH
https://docs.julialang.org/en/v1/manual/code-loading/#code-loading

We are kind of close to a julia package wrapping. Not sure how far down that road we want to go but I'd love to see a feedstock for say, SymbolicRegression.jl, that pysr and other packages could use.

This is not going to work. What happens when there's another python package that relies on the same Julia package? There'll be conflicts right?

This is not going to work. What happens when there's another python package that relies on the same Julia package? There'll be conflicts right?

No, there will be no conflicts. The Julia depots stack. Each conda package can supply their own read-only depot containing their dependencies and a shared environment. It is not a problem that multiple depots supply the same package since the stack has a priority order. The packages themselves are also versioned and multiple versions of the same package are allowed to exist within the same depot.
What the user may need to do is switch Julia projects within the conda environment or manually combine them.
See https://docs.julialang.org/en/v1/base/constants/#Base.DEPOT_PATH for further reading. In particular,

The first entry is the "user depot" and should be writable by and owned by the current user. The user depot is where: registries are cloned, new package versions are installed, named environments are created and updated, package repos are cloned, newly compiled package image files are saved, log files are written, development packages are checked out by default, and global configuration data is saved. Later entries in the depot path are treated as read-only and are appropriate for registries, packages, etc. installed and managed by system administrators.

The Julia depots stack. Each conda package can supply their own read-only depot containing their dependencies and a shared environment. It is not a problem that multiple depots supply the same package since the stack has a priority order. The packages themselves are also versioned and multiple versions of the same package are allowed to exist within the same depot.

The priority order changes depending on which conda package was activated first. Then you'll use a depot from another conda package, right?

The priority order changes depending on which conda package was activated first. Then you'll use a depot from another conda package, right?

For packages where they overlap, yes, but they should be the same package at the same version if they have the same hash. The user's depot in $CONDA_PREFIX/share/julia has the highest priority.
Note that the depot structure is $DEPOT/packages/$PACKAGE_NAME/$HASH/. For example, I have $DEPOT/packages/PyCall/{BcTLp,BD546,L0flP} at the moment representing PyCall 1.92.1, 1.92.3, and 1.93.0 in my main Julia depot at the moment.
Let's keep this in perspective that we're currently talking about two Python packages at the moment that have Julia dependencies. Their direct dependencies overlap at PyCall.jl and its dependency Conda.jl. Those are the first dependencies to target.
Eventually, we would like to put each Julia package into its own conda-forge package and ultimately installing these into a single depot would be the goal. However, having multiple (currently 2) conda installed depots is a step in the the right direction as we move towards that. What we gain from this initial move is conda management and installation of Julia packages. This greatly improves upon the current user experience which involves the user running an additional install step that uses the Julia package manager independently of Julia.
If we want to bring more languages into conda-forge, we need transitional strategies that allow for incremental steps.

@mkitti can you tell us more about the sharing between depots? What prevents us from mixing the depots or having them automatically share? Don't these packages know where to look? Or will pysr get confused by, say, PyCall X if PyCall Y is also available?

Julia Pkg.jl will go down the depot stack looking for compatible packages. If it does not find them, then it will try to download the latest code into the primary depot at the top of the stack. As configured here, we include a pysr-0.1.10 project environment that also includes a Manifest.toml. That Manifest.toml specifies the use of particular versions of packages, the exact versions we put into the depot.
Now there might be sharing between depots if another depot provides the exact same package at the exact same version. If it's the exact same package though, is there really a problem? There is a potential other issue which is that some packages such as Conda.jl currently stores some configuration state in a deps.jl relative to the code:
https://github.com/JuliaPy/Conda.jl/blob/master/src/Conda.jl#L20
We should change that to use https://github.com/JuliaPackaging/Preferences.jl

Can we customize Pkg such that it works on "environments" as opposed to "projects"? In conda-forge, we have an $PREFIX/lib, $PREFIX/bin, and $PREFIX/lib/python3.X/site-packages, etc. etc. and all packages in a given environment share and access all these packages/libraries. Is there a we can target the equivalent of this here instead of these depots? If not, why not?

These depots are essentially the same as $PREFIX/lib/python3.X/, perhaps slightly larger in scope because it occasionally brings in some binaries via artifacts. I think we should eventually have a single conda-forge managed depot that is read-only to the user from the tooling perspective. That is Pkg.jl should not be trying to modify the conda-forge managed repository. To get there we need to have individual conda-forge packages for individual Julia packages. The problem with having a single depot is that the user can modify the top one, easily.
Note that the standard Julia configuration uses three depots:

~/.julia where ~ is the user home as appropriate on the system;
an architecture-specific shared system directory, e.g. /usr/local/share/julia;
an architecture-independent shared system directory, e.g. /usr/share/julia.

Currently we have moved ~/.julia to $PREFIX/share/julia. Now what I'm proposing is that we replace /usr/local/share/julia.
As far as I can tell, interference between package managers exists with other languages. If I need to use pip, I usually use it after I use conda and then I avoid manipulating the environment. Instead imagine if pip and conda had distinct places to store packages but they were still stacked so that I can safely use conda after using pip. Perhaps my understanding of this is slightly outdated.

I was referring to package versions above, and yes you can install multiple package versions simultaneously within the same depot.
I think you are talking about Python versions. The depot is designed such that multiple Julia versions can use the same depot. That said we are not setup here to have multiple Julia versions in the same conda environment, and our depots are entirely contained within the conda environment.

If it does not find them, then it will try to download the latest code into the primary depot at the top of the stack.

Where's the primary depot?

Now there might be sharing between depots if another depot provides the exact same package at the exact same version. If it's the exact same package though, is there really a problem?

If two conda packages 'a' and 'b' provide the same depot SymbolicRegression.jl then there'll be clobbering and if b is uninstalled, a will stop working because SymbolicRegression.jl depot will be deleted when b is removed.
I have another question, where is the project being created? (AFAIK Depots in julia are like pkg_dirs in conda where conda packages are stored. A project in julia is like a conda environment where the conda pkgs are extracted to)

Where's the primary depot?

The primary depot is in $PREFIX/share/julia per the julia-feedstock activate.sh

If two conda packages 'a' and 'b' provide the same depot SymbolicRegression.jl then there'll be clobbering and if b is uninstalled, a will stop working because SymbolicRegression.jl depot will be deleted when b is removed.

We're getting slightly confused here about the name of the depot, the name of the Julia package, and the name the conda-forge package.
I probably should have named the depot $PREFIX/share/pysr. I named it $PREFIX/share/SymbolicRegression.jl because the plan is to spin that out into its own conda-forge package named after SymbolicRegression.jl. The leading candidate for that name is julia-symbolicregression.
To be clear the depot $PREFIX/share/SymbolicRegression.jl/depot/packages contains the Julia source for 100 packages:
AbstractFFTs                    LLVMExtra_jll
AbstractTrees                   LogExpFunctions
Adapt                           LossFunctions
ArgCheck                        MacroTools
ArrayInterface                  Metatheory
ArrayInterfaceCore              MicroCollections
ArrayInterfaceStaticArrays      Missings
ArrayInterfaceStaticArraysCore  MultivariatePolynomials
AutoHashEquals                  MutableArithmetics
BangBang                        NaNMath
Baselet                         NLSolversBase
Bijections                      OpenSpecFun_jll
CEnum                           Optim
ChainRules                      OrderedCollections
ChainRulesCore                  Parameters
ChangesOfVariables              Parsers
ClusterManagers                 PositiveFactorizations
Combinatorics                   PreallocationTools
CommonSubexpressions            Preferences
Compat                          PyCall
CompositionsBase                RealDot
Conda                           RecipesBase
ConstructionBase                RecursiveArrayTools
DataAPI                         Reexport
DataStructures                  Referenceables
DataValueInterfaces             Requires
DefineSingletons                ReverseDiff
DiffResults                     Setfield
DiffRules                       SortingAlgorithms
DocStringExtensions             SpecialFunctions
DynamicPolynomials              SplittablesBase
ExprTools                       Static
FillArrays                      StaticArrays
FiniteDiff                      StaticArraysCore
ForwardDiff                     StatsAPI
FunctionWrappers                StatsBase
GPUArrays                       StructArrays
GPUArraysCore                   StructTypes
IfElse                          SymbolicRegression
InitialValues                   SymbolicUtils
InverseFunctions                Tables
IrrationalConstants             TableTraits
IRTools                         TermInterface
IteratorInterfaceExtensions     ThreadsX
JLLWrappers                     TimerOutputs
JSON                            Transducers
JSON3                           UnPack
LabelledArrays                  VersionParsing
LineSearches                    Zygote
LLVM                            ZygoteRules

Now let's say we have a theoretical conda-package conforming to BitInformation.jl or maybe julia-bitinformation. The Python interface is called xbitinfo and was recently packaged in conda-forge: https://github.com/conda-forge/xbitinfo-feedstock .
We could have a BitInformation.jl depot at $PREFIX/share/BitInformation.jl/depot or maybe $PREFIX/share/julia-bitinformation/depot. The packages subfolder contains all the dependencies that BitInformation.jl needs:
BitInformation       FillArrays               Preferences
Calculus             HypergeometricFunctions  QuadGK
ChainRulesCore       InverseFunctions         Reexport
ChangesOfVariables   IrrationalConstants      Rmath
Compat               JLLWrappers              Rmath_jll
DataAPI              LogExpFunctions          SortingAlgorithms
DataStructures       Missings                 SpecialFunctions
DensityInterface     NaNMath                  StatsAPI
Distributions        OpenSpecFun_jll          StatsBase
DocStringExtensions  OrderedCollections       StatsFuns
DualNumbers          PDMats

Both depots happen to contain code for FillArrays.jl, and we have potentially redundant installs of FillArrays.jl.
If either the SymbolicRegression.jl or the BitInformation.jl depot is uninstalled, the remaining one will continue to work. Each depot contains all the packages that the main package of the depot needs.
Two packages should not be providing depots of the same name because the depots should be closely related to the package that provides them. In this case pysr is a proxy for SymbolicRegression.jl and xbitinfo is a proxy for BitInformation.jl.
The depots are separate exactly to avoid clobbering each other. They can function independently or as part of the same stack.

I have another question, where is the project being created? (AFAIK Depots in julia are like pkg_dirs in conda where conda packages are stored. A project in julia is like a conda environment where the conda pkgs are extracted to.

In Julia there are "shared" environments which are stored in DEPOT_PATH[n]/environments/ and there are just regular directories wherever the user might want one. These usually only contain Project.toml and Manifest.toml.
This branch currently installs a Julia project in $PREFIX/share/SymbolicRegression.jl/depot/environments/pysr-0.10.1. Since it's within the depot the shorthand for this within Julia is @pysr-0.10.1. For example one may do the following.
julia> using Pkg

julia> pkg"activate @pysr-0.10.1"
  Activating project at `~/mambaforge/envs/pysr/share/SymbolicRegression.jl/depot/environments/pysr-0.10.1`

julia> pkg"status"
      Status `$PREFIX/share/SymbolicRegression.jl/depot/environments/pysr-0.10.1/Project.toml`
  [34f1f09b] ClusterManagers v0.4.2 `https://github.com/JuliaParallel/ClusterManagers.jl#14e7302`
  [438e738f] PyCall v1.94.1
  [8254be44] SymbolicRegression v0.10.1 `https://github.com/MilesCranmer/SymbolicRegression.jl#v0.10.1`

julia> Base.load_path()
3-element Vector{String}:
 "$PREFIX/share/SymbolicRegression.jl/depot/environments/pysr-0.10.1/Project.toml"
 "$PREFIX/share/julia/environments/pysr/Project.toml"
 "$PREFIX/share/julia/stdlib/v1.8"

Also note that the project folder only contains toml files:
$ ls $CONDA_PREFIX/share/SymbolicRegression.jl/depot/environments/pysr-0.10.1/
Manifest.toml  Project.toml

All packages are stored within the depot stack not within the project environment folders.

Currently the activate.sh script on this branch will activate the Julia environment @pysr-0.10.1. However, the Python package pysr should probably set the environment variable JULIA_PROJECT to be @pysr-0.10.1.

On licensing, note that all the packages have a license file with them.
Regarding the binary artifacts, we need to create an Overrides.toml to tell Julia where to find the replacement binaries:
https://pkgdocs.julialang.org/v1/artifacts/#Overriding-artifact-locations

I've seen enough, I think the issue with conflicting julia packages is resolved. Updated todo:

How? Do you want my input or not? If you do, don't prematurely say things like that.

I don't understand how JULIA_PROJECT is supposed to work? What happens when pysr and xbitinfo are both installed? Both of them would try to set JULIA_PROJECT to their own name right?

Projects do stack. JULIA_PROJECT manipulates the current active environment. To manipulate the environment stack we need to modify JULIA_LOAD_PATH.
I think we should not change the current active environment here. Rather PySR should set the environment variable JULIA_PROJECT before loading Julia.
Currently,  PySR requires that PyCall be installed in the current active environment, but that can easily not be the case. Thus the solution is for PySR to install PyCall and it's own Julia environment and then activate that.
Alternatively, we could modify JULIA_LOAD_PATH and add @pysr-0.10.1 to the stack.

export JULIA_LOAD_PATH="@:@pysr-0.10.1:@{CONDA_PREFIX##*/}:@stdlib"

The load order then be:

The current active Julia project. Initially $JULIA_PROJECT
@pyrsr-0.10.1
@$CONDA_PREFIX the default one of the julia-feedstock
The standard library

If we do this approach, then we should have a more modular way of building the load path or I need to get better at string mugging with bash.

What does the first @: do?

The documentation for JULIA_LOAD_PATH environment variable and the LOAD_PATH variable within Julia is located here:
https://docs.julialang.org/en/v1/base/constants/#Base.LOAD_PATH

@ refers to the "current active environment", the initial value of which is initially determined by the JULIA_PROJECT environment variable or the --project command-line option.
@stdlib expands to the absolute path of the current Julia installation's standard library directory.
@name refers to a named environment, which are stored in depots (see JULIA_DEPOT_PATH) under the environments subdirectory. The user's named environments are stored in ~/.julia/environments so @name would refer to the environment in ~/.julia/environments/name if it exists and contains a Project.toml file. If name contains # characters, then they are replaced with the major, minor and patch components of the Julia version number. For example, if you are running Julia 1.2 then @v#.# expands to @v1.2 and will look for an environment by that name, typically at ~/.julia/environments/v1.2.

The fully expanded value of LOAD_PATH that is searched for projects and packages can be seen by calling the Base.load_path() function.

The user can change the active project to whatever they want by either manipulating JULIA_PROJECT or using julia --project. However, they might still want to be able to load PyCall.jl and SymbolicRegression.jl regardless of what the active project is.

Or does it become redundant because we are activating the same project twice?

By default, there is usually a redundancy since @ is usually is the same as @#.#

There are two ways to call Python from Julia.

PyCall.jl which depends on Conda.jl: This is the first implementation, mainly by Professor Stephen G. Johnson and is under community maintenance. Isuru and I have contributed to Conda.jl.
PythonCall.jl which depends on CondaPkg.jl, which in turn depends on MicroMamba.jl: This is a separate implementation by Christopher Rowley.

There are now also two ways to call Julia from Python, corresponding with the above, respectively:

pyjulia - the julia package in PyPI. This uses PyCall.jl behind the scenes. This is mainly authored by Takafumi Arakaki and is under community maintenance.
juliacall. This uses PythonCall.jl behind the scenes. This is also authored by Christopher Rowley

You can kind of mix the two, but they otherwise form two independent methods of interop.
There is a pyjulia package in conda-forge. That should depend on PyCall.jl.
There probably should be a py-juliacall package in conda-forge as well that would in turn depend on PythonCall.jl.
I really do not think installing PyCall.jl by default in the julia-feedstock is a good idea. This would create a dependency on Python. If you want to depend on PyCall.jl, you should depend on PyCall.jl. Thus there should be a PyCall.jl package in conda-forge. Maybe it'll be called jl-pycall or julia-pycall.

macos tests pass on cc4a619. I will try to simplify.
We should probably merge https://github.com/conda-forge/julia-feedstock/pull/221 until we can figure out what has gone wrong with libgit2 on macos.

Is this ready for review? Let me know and I can try it out.
Cheers,
Miles

I need to clean up some settings that we pushed into the julia-feedstock.
One thing I think we should change in the Python code is to activate the pysr environment sooner by setting the JULIA_PROJECT environment variable. This way the PyCall dependency is entirely contained  within the environment.

@MilesCranmer , this is about where I want it without making modifications elsewhere.
The main thing is that we're putting PyCall into the pysr-0.10.1 environment and loading that environment directly rather than depending on whatever the current default environment is. I think this will be more robust than trying to install PySR in whatever the current environment is.
In the future, the plan for Julia packages in conda-forge would be:

Package PyCall.jl
Package SymbolicRegressions.jl
Package ClusterManagers.jl

When this is done, we should just be able to depend on this packages via conda-forge rather than having to package them explicitly here.
@ngam I think this is ready for @MilesCranmer to review.

This is not going to work, unless you package the julia packages as individual conda packages.
Say pysr depended on foo.jl>=v1 and packaged foo.jl=v1 and another conda package pybar depended on baz=v2 which depended on foo.jl>=v2. Installing both of them and JULIA_LOAD_PATH is set to pysr:pybar, then pysr will work correctly, but pybar will not because JULIA_LOAD_PATH has pysr in the front and foo.jl=v1 will have priority.

This is not going to work, unless you package the julia packages as individual conda packages.
Say pysr depended on foo.jl>=v1 and packaged foo.jl=v1 and another conda package pybar depended on baz=v2 which depended on foo.jl>=v2. Installing both of them and JULIA_LOAD_PATH is set to pysr:pybar, then pysr will work correctly, but pybar will not because JULIA_LOAD_PATH has pysr in the front and foo.jl=v1 will have priority.

This is why I have suggested to @MilesCranmer that we specifically activate the pysr-0.10.1 project in the Python code earlier. This will make the pysr-0.10.1 project the highest priority. pysr actually currently does this, but it depends on PyCall.jl being in the active project before trying to activate the pysr-0.10.1 project.
The only dependency is that currently sensitive to the scenario you outlined is PyCall.jl. Having PySR set JULIA_PROJECT will resolve that issue.

Here are some specific line references
This is where PySR activates it's Julia project:
https://github.com/MilesCranmer/PySR/blob/d09ade8628f541eb94028009bacdb9a55cb22ef5/pysr/julia_helpers.py#L30
This is where the julia project is defined.
https://github.com/MilesCranmer/PySR/blob/d09ade8628f541eb94028009bacdb9a55cb22ef5/pysr/julia_helpers.py#L63
The proposed change is that we insert this
import os
os.environ["JULIA_PROJECT"] = "pysr-0.10.1"

before PySR does
    import julia
    julia.install(quiet=quiet)

@MilesCranmer I think you can stream line the install by supplying both PackageSpec via a Vector at the same time.
https://github.com/MilesCranmer/PySR/blob/d09ade8628f541eb94028009bacdb9a55cb22ef5/pysr/julia_helpers.py#L107-L119
def _add_sr_to_julia_project(Main, io_arg):
    Main.sr_spec = Main.PackageSpec(
        name="SymbolicRegression",
        url="https://github.com/MilesCranmer/SymbolicRegression.jl",
        rev="v" + __symbolic_regression_jl_version__,
    )
    Main.clustermanagers_spec = Main.PackageSpec(
        name="ClusterManagers",
        url="https://github.com/JuliaParallel/ClusterManagers.jl",
        rev="14e7302f068794099344d5d93f71979aaf4fbeb3",
    )
    Main.eval(f"Pkg.add([sr_spec, clustermanagers_spec], {io_arg})")

Let's review the status quo ante:
mamba create -n pysr_test -c conda-forge pysr

$ python
Python 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:36:39) [GCC 10.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np, pysr
>>> X = 2 * np.random.randn(100, 5)
>>> y = 2.5382 * np.cos(X[:, 3]) + X[:, 0] ** 2 - 0.5
>>> from pysr import PySRRegressor
>>>
>>> model = PySRRegressor(
...     model_selection="best",  # Result is mix of simplicity+accuracy
...     niterations=40,
...     binary_operators=["+", "*"],
...     unary_operators=[
...         "cos",
...         "exp",
...         "sin",
...         "inv(x) = 1/x",
...     # ^ Custom operator (julia syntax)
...     ],
...     extra_sympy_mappings={"inv": lambda x: 1 / x},
...     # ^ Define operator for SymPy as well
...     loss="loss(x, y) = (x - y)^2",
...     # ^ Custom loss function (julia syntax)
... )
>>> model.fit(X, y)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/mkitti/mambaforge/envs/pysr_test/lib/python3.10/site-packages/pysr/sr.py", line 1771, in fit
    self._run(X, y, mutated_params, weights=weights, seed=seed)
  File "/home/mkitti/mambaforge/envs/pysr_test/lib/python3.10/site-packages/pysr/sr.py", line 1464, in _run
    Main = init_julia()
  File "/home/mkitti/mambaforge/envs/pysr_test/lib/python3.10/site-packages/pysr/julia_helpers.py", line 88, in init_julia
    raise ImportError(import_error_string())
ImportError:
    Required dependencies are not installed or built.  Run the following code in the Python REPL:

        >>> import pysr
        >>> pysr.install()

After running pysr.install() we three packages installed across two environments. PyCall is installed in the default environment because that was what was active before doing pysr.install().
$ julia
(@pysr_test) pkg> st
      Status `~/mambaforge/envs/pysr_test/share/julia/environments/pysr_test/Project.toml`
  [438e738f] PyCall v1.94.1

(pysr-0.10.1) pkg> activate @pysr-0.10.1
  Activating project at `~/mambaforge/envs/pysr_test/share/julia/environments/pysr-0.10.1`

(@pysr-0.10.1) pkg> st
      Status `~/mambaforge/envs/pysr_test/share/julia/environments/pysr-0.10.1/Project.toml`
  [34f1f09b] ClusterManagers v0.4.2 `https://github.com/JuliaParallel/ClusterManagers.jl#14e7302`
  [8254be44] SymbolicRegression v0.10.1 `https://github.com/MilesCranmer/SymbolicRegression.jl#v0.10.1`

With this pull request, and more so if Miles implements my recommendation and sets JULIA_PROJECT before import julia, we should have this:
(@pysr_test) pkg> st
Status `~/mambaforge/envs/pysr_test/share/julia/environments/pysr/Project.toml` (empty project)

(@pysr-0.10.1) pkg> st
Status `~/mambaforge/envs/pysr_test/share/pysr/depot/environments/pysr-0.10.1/Project.toml`
  [34f1f09b] ClusterManagers v0.4.2 `https://github.com/JuliaParallel/ClusterManagers.jl#14e7302`
  [438e738f] PyCall v1.94.1
  [8254be44] SymbolicRegression v0.10.1 `https://github.com/MilesCranmer/SymbolicRegression.jl#v0.10.1`

That is PySR will no longer depend on having PyCall in the default Julia project environment.

To summarize @isuruf :

The only package that pysr currently depends on being in the current Julia environment is PyCall.jl
If it does not see PyCall.jl, then it will prompt the user to initiate an install step which will install PyCall.jl into the current environment.
pysr.install() will then create it's own versioned environment (e.g. @pysr-0.10.1) and install ClusterMangers.jl and SymbolicRegressions.jl into that environment.

After the changes in this pull request,

@pysr-0.10.1 will more likely be the active Julia environment and will have PyCall.jl installed.
If @pysr-0.10.1 is not the current active environment, then pysr will prompt the user to do pysr.install(). This will install PyCall.jl into the current environment and then switch to the existing @pysr-0.10.1. It should then see that the other packages are installed.

My recommendation to Miles is that we ensure that @pysr-0.10.1 is the current active environment before even trying to load to pyjulia / PyCall.jl. In that case, then we should not need to set the JULIA_PROJECT in the activate.sh here.

In the worse case scenario where JULIA_PROJECT and JULIA_LOAD_PATH are clobbered by some other package, then we're back at the status quo ante. PySR will install PyCall.jl into the current environment, create a new environment for itself, and then activate it.

This is why I have suggested to @MilesCranmer that we specifically activate the pysr-0.10.1 project in the Python code earlier.

I created a pull request to help address Isuru's points:
https://github.com/MilesCranmer/PySR/pull/186
The pull request will set JULIA_PROJECT to the pysr environment before trying to load pyjulia and PyCall.jl.

Okay, but it is not ideal. I misunderstood earlier, I thought you were saying that one could have foo.jl==v1 and foo.jl==v2 in the same stack and two different packages can still find their correct dep somehow. That's why I said "I've seen enough" prematurely.

We can have multiple versions in the stack. What gets loaded will depend on the environment stack order though. Setting JULIA_PROJECT will move that environment to the top of stack.
PySR already explicitly activates an environment. The main reason for activating it earlier is decrease its dependence on PyCall.jl being in the current environment. If we merge https://github.com/MilesCranmer/PySR/pull/186, then we can remove the JULIA_PROJECT and JULIA_LOAD_PATH modifications here, and then this will still work.
The added JULIA_DEPOT_PATH depot will just act as a local package cache.

I think, for now, it is better to let the package take care of installing its required artifacts after the fact. So this is not ready for prime time. My personal interest in this is to come up with a productive policy/strategy, and not a one-off solution. I am not even a proper user of Julia...

What we need is an evolvable strategy that allows us to start somewhere and grow. This is an evolvable strategy. It allows to start somewhere practical. As we add individual packages to conda-forge and add those as dependencies to this project, the size of this depot will shrink.

I have PyCall.jl and Conda.jl ready to go. Someone just needs to tell me what to call the Julia packages. Even then, the overall strategy is exactly the same. You still will need to ship individual depots for both of them for the moment. To support PySR though, we need to get up to 100 packages. I'm definitely not doing that by myself. Also, to be clear doing this does not solve the problem that Isuru laid out.

Let's be practical and discuss the two large packages we actually have before us. To resolve the problem for SymbolicRegression.jl (used by pysr) and BitInformation.jl (used by xbitinfo) we need to add them to the same Julia project environment. I'll first point out that is not practical for either pysr or xbitinfo because both Python packages activate individual Julia environments when you run the Python packages. Let's say we do it anyways.
For either package, having both depots present does not hinder anything. For this discussion I will create two depots

pysr_depot
xbitinfo_depot

These each respectively contain an environment.

pysr_env
xbitinfo_env

To the environments, I added a single package:

SymbolicRegressions.jl
BitInformation.jl

Now we have two individual depots.
$ ls pysr_depot/packages | wc -l
93
$ ls xbitinfo_depot/packages | wc -l
32

Now I'll construct a common depot and put at the top of the depot stack while placing the two other depots underneath it.
julia> empty!(DEPOT_PATH)
String[]

julia> mkpath("common_depot/environments/common_env")
"common_depot/environments/common_env"

julia> push!(DEPOT_PATH, "common_depot")
1-element Vector{String}:
 "common_depot"

julia> push!(DEPOT_PATH, "pysr_depot")
2-element Vector{String}:
 "common_depot"
 "pysr_depot"

julia> push!(DEPOT_PATH, "xbitinfo_depot")
3-element Vector{String}:
 "common_depot"
 "pysr_depot"
 "xbitinfo_depot"

julia> Pkg.activate("common_depot/environments/common_env")
  Activating new project at `~/src/depottest/common_depot/environments/common_env`

julia> pkg"add SymbolicRegression BitInformation"
   Resolving package versions...
...

julia> pkg"status"
Status `~/src/depottest/common_depot/environments/common_env/Project.toml`
  [de688a37] BitInformation v0.6.0
  [8254be44] SymbolicRegression v0.10.2

When I added both packages to the same environment, no downloads had to occur. In fact, if we look for common_depot/packages we see that the directory does not even exist. We didn't need to "install" any additional packages!
julia> readdir(DEPOT_PATH[1])
4-element Vector{String}:
 "compiled"
 "environments"
 "logs"
 "scratchspaces"

What is in common_depot/environments/common_env? Two files

Project.toml
Manifest.toml

$ ls common_depot/environments/common_env/
Manifest.toml  Project.toml

$ cat common_depot/environments/common_env/Project.toml
[deps]
BitInformation = "de688a37-743e-4ac2-a6f0-bd62414d1aa7"
SymbolicRegression = "8254be44-1295-4e6a-a16d-46603ac705cb"

$ cat common_depot/environments/common_env/Manifest.toml
# This file is machine-generated - editing it directly is not advised

julia_version = "1.8.0"
manifest_format = "2.0"
project_hash = "c901445fd3719cd02549f8ebb4f6634de8639c29"

[[deps.AbstractFFTs]]
deps = ["ChainRulesCore", "LinearAlgebra"]
git-tree-sha1 = "69f7020bd72f069c219b5e8c236c1fa90d2cb409"
uuid = "621f4979-c628-5d54-868e-fcf4e3e8185c"
version = "1.2.1"
...

Full Manifest.toml
# This file is machine-generated - editing it directly is not advised

julia_version = "1.8.0"
manifest_format = "2.0"
project_hash = "c901445fd3719cd02549f8ebb4f6634de8639c29"

[[deps.AbstractFFTs]]
deps = ["ChainRulesCore", "LinearAlgebra"]
git-tree-sha1 = "69f7020bd72f069c219b5e8c236c1fa90d2cb409"
uuid = "621f4979-c628-5d54-868e-fcf4e3e8185c"
version = "1.2.1"

[[deps.AbstractTrees]]
git-tree-sha1 = "5c0b629df8a5566a06f5fef5100b53ea56e465a0"
uuid = "1520ce14-60c1-5f80-bbc7-55ef81b5835c"
version = "0.4.2"

[[deps.Adapt]]
deps = ["LinearAlgebra"]
git-tree-sha1 = "195c5505521008abea5aee4f96930717958eac6f"
uuid = "79e6a3ab-5dfb-504d-930d-738a2a938a0e"
version = "3.4.0"

[[deps.ArgCheck]]
git-tree-sha1 = "a3a402a35a2f7e0b87828ccabbd5ebfbebe356b4"
uuid = "dce04be8-c92d-5529-be00-80e4d2c0e197"
version = "2.3.0"

[[deps.ArgTools]]
uuid = "0dad84c5-d112-42e6-8d28-ef12dabb789f"
version = "1.1.1"

[[deps.ArrayInterface]]
deps = ["ArrayInterfaceCore", "Compat", "IfElse", "LinearAlgebra", "Static"]
git-tree-sha1 = "d6173480145eb632d6571c148d94b9d3d773820e"
uuid = "4fba245c-0d91-5ea0-9b3e-6abc04ee57a9"
version = "6.0.23"

[[deps.ArrayInterfaceCore]]
deps = ["LinearAlgebra", "SparseArrays", "SuiteSparse"]
git-tree-sha1 = "5bb0f8292405a516880a3809954cb832ae7a31c5"
uuid = "30b0a656-2188-435a-8636-2ec0e6a096e2"
version = "0.1.20"

[[deps.ArrayInterfaceStaticArrays]]
deps = ["Adapt", "ArrayInterface", "ArrayInterfaceStaticArraysCore", "LinearAlgebra", "Static", "StaticArrays"]
git-tree-sha1 = "efb000a9f643f018d5154e56814e338b5746c560"
uuid = "b0d46f97-bff5-4637-a19a-dd75974142cd"
version = "0.1.4"

[[deps.ArrayInterfaceStaticArraysCore]]
deps = ["Adapt", "ArrayInterfaceCore", "LinearAlgebra", "StaticArraysCore"]
git-tree-sha1 = "a1e2cf6ced6505cbad2490532388683f1e88c3ed"
uuid = "dd5226c6-a4d4-4bc7-8575-46859f9c95b9"
version = "0.1.0"

[[deps.Artifacts]]
uuid = "56f22d72-fd6d-98f1-02f0-08ddc0907c33"

[[deps.AutoHashEquals]]
git-tree-sha1 = "45bb6705d93be619b81451bb2006b7ee5d4e4453"
uuid = "15f4f7f2-30c1-5605-9d31-71845cf9641f"
version = "0.2.0"

[[deps.BangBang]]
deps = ["Compat", "ConstructionBase", "Future", "InitialValues", "LinearAlgebra", "Requires", "Setfield", "Tables", "ZygoteRules"]
git-tree-sha1 = "b15a6bc52594f5e4a3b825858d1089618871bf9d"
uuid = "198e06fe-97b7-11e9-32a5-e1d131e6ad66"
version = "0.3.36"

[[deps.Base64]]
uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"

[[deps.Baselet]]
git-tree-sha1 = "aebf55e6d7795e02ca500a689d326ac979aaf89e"
uuid = "9718e550-a3fa-408a-8086-8db961cd8217"
version = "0.1.1"

[[deps.Bijections]]
git-tree-sha1 = "fe4f8c5ee7f76f2198d5c2a06d3961c249cce7bd"
uuid = "e2ed5e7c-b2de-5872-ae92-c73ca462fb04"
version = "0.1.4"

[[deps.BitInformation]]
deps = ["Distributions", "Random", "StatsBase"]
git-tree-sha1 = "9003617f8ba48b69c3316b9e6f45fa376ef24fd0"
uuid = "de688a37-743e-4ac2-a6f0-bd62414d1aa7"
version = "0.6.0"

[[deps.CEnum]]
git-tree-sha1 = "eb4cb44a499229b3b8426dcfb5dd85333951ff90"
uuid = "fa961155-64e5-5f13-b03f-caf6b980ea82"
version = "0.4.2"

[[deps.Calculus]]
deps = ["LinearAlgebra"]
git-tree-sha1 = "f641eb0a4f00c343bbc32346e1217b86f3ce9dad"
uuid = "49dc2e85-a5d0-5ad3-a950-438e2897f1b9"
version = "0.5.1"

[[deps.ChainRules]]
deps = ["Adapt", "ChainRulesCore", "Compat", "Distributed", "GPUArraysCore", "IrrationalConstants", "LinearAlgebra", "Random", "RealDot", "SparseArrays", "Statistics", "StructArrays"]
git-tree-sha1 = "a5fd229d3569a6600ae47abe8cd48cbeb972e173"
uuid = "082447d4-558c-5d27-93f4-14fc19e9eca2"
version = "1.44.6"

[[deps.ChainRulesCore]]
deps = ["Compat", "LinearAlgebra", "SparseArrays"]
git-tree-sha1 = "8a494fe0c4ae21047f28eb48ac968f0b8a6fcaa7"
uuid = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4"
version = "1.15.4"

[[deps.ChangesOfVariables]]
deps = ["ChainRulesCore", "LinearAlgebra", "Test"]
git-tree-sha1 = "38f7a08f19d8810338d4f5085211c7dfa5d5bdd8"
uuid = "9e997f8a-9a97-42d5-a9f1-ce6bfc15e2c0"
version = "0.1.4"

[[deps.Combinatorics]]
git-tree-sha1 = "08c8b6831dc00bfea825826be0bc8336fc369860"
uuid = "861a8166-3701-5b0c-9a16-15d98fcdc6aa"
version = "1.0.2"

[[deps.CommonSubexpressions]]
deps = ["MacroTools", "Test"]
git-tree-sha1 = "7b8a93dba8af7e3b42fecabf646260105ac373f7"
uuid = "bbf7d656-a473-5ed7-a52c-81e309532950"
version = "0.3.0"

[[deps.Compat]]
deps = ["Base64", "Dates", "DelimitedFiles", "Distributed", "InteractiveUtils", "LibGit2", "Libdl", "LinearAlgebra", "Markdown", "Mmap", "Pkg", "Printf", "REPL", "Random", "SHA", "Serialization", "SharedArrays", "Sockets", "SparseArrays", "Statistics", "Test", "UUIDs", "Unicode"]
git-tree-sha1 = "78bee250c6826e1cf805a88b7f1e86025275d208"
uuid = "34da2185-b29b-5c13-b0c7-acf172513d20"
version = "3.46.0"

[[deps.CompilerSupportLibraries_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "e66e0078-7015-5450-92f7-15fbd957f2ae"
version = "0.5.2+0"

[[deps.CompositionsBase]]
git-tree-sha1 = "455419f7e328a1a2493cabc6428d79e951349769"
uuid = "a33af91c-f02d-484b-be07-31d278c5ca2b"
version = "0.1.1"

[[deps.ConstructionBase]]
deps = ["LinearAlgebra"]
git-tree-sha1 = "fb21ddd70a051d882a1686a5a550990bbe371a95"
uuid = "187b0558-2788-49d3-abe0-74a17ed4e7c9"
version = "1.4.1"

[[deps.DataAPI]]
git-tree-sha1 = "fb5f5316dd3fd4c5e7c30a24d50643b73e37cd40"
uuid = "9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a"
version = "1.10.0"

[[deps.DataStructures]]
deps = ["Compat", "InteractiveUtils", "OrderedCollections"]
git-tree-sha1 = "d1fff3a548102f48987a52a2e0d114fa97d730f0"
uuid = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8"
version = "0.18.13"

[[deps.DataValueInterfaces]]
git-tree-sha1 = "bfc1187b79289637fa0ef6d4436ebdfe6905cbd6"
uuid = "e2d170a0-9d28-54be-80f0-106bbe20a464"
version = "1.0.0"

[[deps.Dates]]
deps = ["Printf"]
uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"

[[deps.DefineSingletons]]
git-tree-sha1 = "0fba8b706d0178b4dc7fd44a96a92382c9065c2c"
uuid = "244e2a9f-e319-4986-a169-4d1fe445cd52"
version = "0.1.2"

[[deps.DelimitedFiles]]
deps = ["Mmap"]
uuid = "8bb1440f-4735-579b-a4ab-409b98df4dab"

[[deps.DensityInterface]]
deps = ["InverseFunctions", "Test"]
git-tree-sha1 = "80c3e8639e3353e5d2912fb3a1916b8455e2494b"
uuid = "b429d917-457f-4dbc-8f4c-0cc954292b1d"
version = "0.4.0"

[[deps.DiffResults]]
deps = ["StaticArrays"]
git-tree-sha1 = "c18e98cba888c6c25d1c3b048e4b3380ca956805"
uuid = "163ba53b-c6d8-5494-b064-1a9d43ac40c5"
version = "1.0.3"

[[deps.DiffRules]]
deps = ["IrrationalConstants", "LogExpFunctions", "NaNMath", "Random", "SpecialFunctions"]
git-tree-sha1 = "992a23afdb109d0d2f8802a30cf5ae4b1fe7ea68"
uuid = "b552c78f-8df3-52c6-915a-8e097449b14b"
version = "1.11.1"

[[deps.Distributed]]
deps = ["Random", "Serialization", "Sockets"]
uuid = "8ba89e20-285c-5b6f-9357-94700520ee1b"

[[deps.Distributions]]
deps = ["ChainRulesCore", "DensityInterface", "FillArrays", "LinearAlgebra", "PDMats", "Printf", "QuadGK", "Random", "SparseArrays", "SpecialFunctions", "Statistics", "StatsBase", "StatsFuns", "Test"]
git-tree-sha1 = "8579b5cdae93e55c0cff50fbb0c2d1220efd5beb"
uuid = "31c24e10-a181-5473-b8eb-7969acd0382f"
version = "0.25.70"

[[deps.DocStringExtensions]]
deps = ["LibGit2"]
git-tree-sha1 = "b19534d1895d702889b219c382a6e18010797f0b"
uuid = "ffbed154-4ef7-542d-bbb7-c09d3a79fcae"
version = "0.8.6"

[[deps.Downloads]]
deps = ["ArgTools", "FileWatching", "LibCURL", "NetworkOptions"]
uuid = "f43a241f-c20a-4ad4-852c-f6b1247861c6"
version = "1.6.0"

[[deps.DualNumbers]]
deps = ["Calculus", "NaNMath", "SpecialFunctions"]
git-tree-sha1 = "5837a837389fccf076445fce071c8ddaea35a566"
uuid = "fa6b7ba4-c1ee-5f82-b5fc-ecf0adba8f74"
version = "0.6.8"

[[deps.DynamicPolynomials]]
deps = ["DataStructures", "Future", "LinearAlgebra", "MultivariatePolynomials", "MutableArithmetics", "Pkg", "Reexport", "Test"]
git-tree-sha1 = "d0fa82f39c2a5cdb3ee385ad52bc05c42cb4b9f0"
uuid = "7c1d4256-1411-5781-91ec-d7bc3513ac07"
version = "0.4.5"

[[deps.ExprTools]]
git-tree-sha1 = "56559bbef6ca5ea0c0818fa5c90320398a6fbf8d"
uuid = "e2ba6199-217a-4e67-a87a-7c52f15ade04"
version = "0.1.8"

[[deps.FileWatching]]
uuid = "7b1f6079-737a-58dc-b8bc-7a2ca5c1b5ee"

[[deps.FillArrays]]
deps = ["LinearAlgebra", "Random", "SparseArrays", "Statistics"]
git-tree-sha1 = "87519eb762f85534445f5cda35be12e32759ee14"
uuid = "1a297f60-69ca-5386-bcde-b61e274b549b"
version = "0.13.4"

[[deps.FiniteDiff]]
deps = ["ArrayInterfaceCore", "LinearAlgebra", "Requires", "Setfield", "SparseArrays", "StaticArrays"]
git-tree-sha1 = "5a2cff9b6b77b33b89f3d97a4d367747adce647e"
uuid = "6a86dc24-6348-571c-b903-95158fe2bd41"
version = "2.15.0"

[[deps.ForwardDiff]]
deps = ["CommonSubexpressions", "DiffResults", "DiffRules", "LinearAlgebra", "LogExpFunctions", "NaNMath", "Preferences", "Printf", "Random", "SpecialFunctions", "StaticArrays"]
git-tree-sha1 = "187198a4ed8ccd7b5d99c41b69c679269ea2b2d4"
uuid = "f6369f11-7733-5829-9624-2563aa707210"
version = "0.10.32"

[[deps.Future]]
deps = ["Random"]
uuid = "9fa8497b-333b-5362-9e8d-4d0656e87820"

[[deps.GPUArrays]]
deps = ["Adapt", "GPUArraysCore", "LLVM", "LinearAlgebra", "Printf", "Random", "Reexport", "Serialization", "Statistics"]
git-tree-sha1 = "45d7deaf05cbb44116ba785d147c518ab46352d7"
uuid = "0c68f7d7-f131-5f86-a1c3-88cf8149b2d7"
version = "8.5.0"

[[deps.GPUArraysCore]]
deps = ["Adapt"]
git-tree-sha1 = "6872f5ec8fd1a38880f027a26739d42dcda6691f"
uuid = "46192b85-c4d5-4398-a991-12ede77f4527"
version = "0.1.2"

[[deps.HypergeometricFunctions]]
deps = ["DualNumbers", "LinearAlgebra", "OpenLibm_jll", "SpecialFunctions", "Test"]
git-tree-sha1 = "709d864e3ed6e3545230601f94e11ebc65994641"
uuid = "34004b35-14d8-5ef3-9330-4cdb6864b03a"
version = "0.3.11"

[[deps.IRTools]]
deps = ["InteractiveUtils", "MacroTools", "Test"]
git-tree-sha1 = "af14a478780ca78d5eb9908b263023096c2b9d64"
uuid = "7869d1d1-7146-5819-86e3-90919afe41df"
version = "0.4.6"

[[deps.IfElse]]
git-tree-sha1 = "debdd00ffef04665ccbb3e150747a77560e8fad1"
uuid = "615f187c-cbe4-4ef1-ba3b-2fcf58d6d173"
version = "0.1.1"

[[deps.InitialValues]]
git-tree-sha1 = "4da0f88e9a39111c2fa3add390ab15f3a44f3ca3"
uuid = "22cec73e-a1b8-11e9-2c92-598750a2cf9c"
version = "0.3.1"

[[deps.InteractiveUtils]]
deps = ["Markdown"]
uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240"

[[deps.InverseFunctions]]
deps = ["Test"]
git-tree-sha1 = "b3364212fb5d870f724876ffcd34dd8ec6d98918"
uuid = "3587e190-3f89-42d0-90ee-14403ec27112"
version = "0.1.7"

[[deps.IrrationalConstants]]
git-tree-sha1 = "7fd44fd4ff43fc60815f8e764c0f352b83c49151"
uuid = "92d709cd-6900-40b7-9082-c6be49f344b6"
version = "0.1.1"

[[deps.IteratorInterfaceExtensions]]
git-tree-sha1 = "a3f24677c21f5bbe9d2a714f95dcd58337fb2856"
uuid = "82899510-4779-5014-852e-03e436cf321d"
version = "1.0.0"

[[deps.JLLWrappers]]
deps = ["Preferences"]
git-tree-sha1 = "abc9885a7ca2052a736a600f7fa66209f96506e1"
uuid = "692b3bcd-3c85-4b1f-b108-f13ce0eb3210"
version = "1.4.1"

[[deps.JSON3]]
deps = ["Dates", "Mmap", "Parsers", "StructTypes", "UUIDs"]
git-tree-sha1 = "fd6f0cae36f42525567108a42c1c674af2ac620d"
uuid = "0f8b85d8-7281-11e9-16c2-39a750bddbf1"
version = "1.9.5"

[[deps.LLVM]]
deps = ["CEnum", "LLVMExtra_jll", "Libdl", "Printf", "Unicode"]
git-tree-sha1 = "e7e9184b0bf0158ac4e4aa9daf00041b5909bf1a"
uuid = "929cbde3-209d-540e-8aea-75f648917ca0"
version = "4.14.0"

[[deps.LLVMExtra_jll]]
deps = ["Artifacts", "JLLWrappers", "LazyArtifacts", "Libdl", "Pkg", "TOML"]
git-tree-sha1 = "771bfe376249626d3ca12bcd58ba243d3f961576"
uuid = "dad2f222-ce93-54a1-a47d-0025e8a3acab"
version = "0.0.16+0"

[[deps.LabelledArrays]]
deps = ["ArrayInterfaceCore", "ArrayInterfaceStaticArrays", "ChainRulesCore", "ForwardDiff", "LinearAlgebra", "MacroTools", "PreallocationTools", "RecursiveArrayTools", "StaticArrays"]
git-tree-sha1 = "3926535a04c12fb986028a4a86bf5a0a3cf88b91"
uuid = "2ee39098-c373-598a-b85f-a56591580800"
version = "1.12.0"

[[deps.LazyArtifacts]]
deps = ["Artifacts", "Pkg"]
uuid = "4af54fe1-eca0-43a8-85a7-787d91b784e3"

[[deps.LibCURL]]
deps = ["LibCURL_jll", "MozillaCACerts_jll"]
uuid = "b27032c2-a3e7-50c8-80cd-2d36dbcbfd21"
version = "0.6.3"

[[deps.LibCURL_jll]]
deps = ["Artifacts", "LibSSH2_jll", "Libdl", "MbedTLS_jll", "Zlib_jll", "nghttp2_jll"]
uuid = "deac9b47-8bc7-5906-a0fe-35ac56dc84c0"
version = "7.84.0+0"

[[deps.LibGit2]]
deps = ["Base64", "NetworkOptions", "Printf", "SHA"]
uuid = "76f85450-5226-5b5a-8eaa-529ad045b433"

[[deps.LibSSH2_jll]]
deps = ["Artifacts", "Libdl", "MbedTLS_jll"]
uuid = "29816b5a-b9ab-546f-933c-edad1886dfa8"
version = "1.10.2+0"

[[deps.Libdl]]
uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb"

[[deps.LineSearches]]
deps = ["LinearAlgebra", "NLSolversBase", "NaNMath", "Parameters", "Printf"]
git-tree-sha1 = "7bbea35cec17305fc70a0e5b4641477dc0789d9d"
uuid = "d3d80556-e9d4-5f37-9878-2ab0fcc64255"
version = "7.2.0"

[[deps.LinearAlgebra]]
deps = ["Libdl", "libblastrampoline_jll"]
uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"

[[deps.LogExpFunctions]]
deps = ["ChainRulesCore", "ChangesOfVariables", "DocStringExtensions", "InverseFunctions", "IrrationalConstants", "LinearAlgebra"]
git-tree-sha1 = "94d9c52ca447e23eac0c0f074effbcd38830deb5"
uuid = "2ab3a3ac-af41-5b50-aa03-7779005ae688"
version = "0.3.18"

[[deps.Logging]]
uuid = "56ddb016-857b-54e1-b83d-db4d58db5568"

[[deps.LossFunctions]]
deps = ["InteractiveUtils", "Markdown", "RecipesBase"]
git-tree-sha1 = "53cd63a12f06a43eef6f4aafb910ac755c122be7"
uuid = "30fc2ffe-d236-52d8-8643-a9d8f7c094a7"
version = "0.8.0"

[[deps.MacroTools]]
deps = ["Markdown", "Random"]
git-tree-sha1 = "3d3e902b31198a27340d0bf00d6ac452866021cf"
uuid = "1914dd2f-81c6-5fcd-8719-6d5c9610ff09"
version = "0.5.9"

[[deps.Markdown]]
deps = ["Base64"]
uuid = "d6f4376e-aef5-505a-96c1-9c027394607a"

[[deps.MbedTLS_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "c8ffd9c3-330d-5841-b78e-0817d7145fa1"
version = "2.28.0+0"

[[deps.Metatheory]]
deps = ["AutoHashEquals", "DataStructures", "Dates", "DocStringExtensions", "Parameters", "Reexport", "TermInterface", "ThreadsX", "TimerOutputs"]
git-tree-sha1 = "a160e323d3684889e6026914576f1f4288de131d"
uuid = "e9d8d322-4543-424a-9be4-0cc815abe26c"
version = "1.3.4"

[[deps.MicroCollections]]
deps = ["BangBang", "InitialValues", "Setfield"]
git-tree-sha1 = "6bb7786e4f24d44b4e29df03c69add1b63d88f01"
uuid = "128add7d-3638-4c79-886c-908ea0c25c34"
version = "0.1.2"

[[deps.Missings]]
deps = ["DataAPI"]
git-tree-sha1 = "bf210ce90b6c9eed32d25dbcae1ebc565df2687f"
uuid = "e1d29d7a-bbdc-5cf2-9ac0-f12de2c33e28"
version = "1.0.2"

[[deps.Mmap]]
uuid = "a63ad114-7e13-5084-954f-fe012c677804"

[[deps.MozillaCACerts_jll]]
uuid = "14a3606d-f60d-562e-9121-12d972cd8159"
version = "2022.2.1"

[[deps.MultivariatePolynomials]]
deps = ["ChainRulesCore", "DataStructures", "LinearAlgebra", "MutableArithmetics"]
git-tree-sha1 = "393fc4d82a73c6fe0e2963dd7c882b09257be537"
uuid = "102ac46a-7ee4-5c85-9060-abc95bfdeaa3"
version = "0.4.6"

[[deps.MutableArithmetics]]
deps = ["LinearAlgebra", "SparseArrays", "Test"]
git-tree-sha1 = "4e675d6e9ec02061800d6cfb695812becbd03cdf"
uuid = "d8a4904e-b15c-11e9-3269-09a3773c0cb0"
version = "1.0.4"

[[deps.NLSolversBase]]
deps = ["DiffResults", "Distributed", "FiniteDiff", "ForwardDiff"]
git-tree-sha1 = "50310f934e55e5ca3912fb941dec199b49ca9b68"
uuid = "d41bc354-129a-5804-8e4c-c37616107c6c"
version = "7.8.2"

[[deps.NaNMath]]
deps = ["OpenLibm_jll"]
git-tree-sha1 = "a7c3d1da1189a1c2fe843a3bfa04d18d20eb3211"
uuid = "77ba4419-2d1f-58cd-9bb1-8ffee604a2e3"
version = "1.0.1"

[[deps.NetworkOptions]]
uuid = "ca575930-c2e3-43a9-ace4-1e988b2c1908"
version = "1.2.0"

[[deps.OpenBLAS_jll]]
deps = ["Artifacts", "CompilerSupportLibraries_jll", "Libdl"]
uuid = "4536629a-c528-5b80-bd46-f80d51c5b363"
version = "0.3.20+0"

[[deps.OpenLibm_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "05823500-19ac-5b8b-9628-191a04bc5112"
version = "0.8.1+0"

[[deps.OpenSpecFun_jll]]
deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "13652491f6856acfd2db29360e1bbcd4565d04f1"
uuid = "efe28fd5-8261-553b-a9e1-b2916fc3738e"
version = "0.5.5+0"

[[deps.Optim]]
deps = ["Compat", "FillArrays", "ForwardDiff", "LineSearches", "LinearAlgebra", "NLSolversBase", "NaNMath", "Parameters", "PositiveFactorizations", "Printf", "SparseArrays", "StatsBase"]
git-tree-sha1 = "ad8de074ed5dad13e87d76c467a82e5eff9c693a"
uuid = "429524aa-4258-5aef-a3af-852621145aeb"
version = "1.7.2"

[[deps.OrderedCollections]]
git-tree-sha1 = "85f8e6578bf1f9ee0d11e7bb1b1456435479d47c"
uuid = "bac558e1-5e72-5ebc-8fee-abe8a469f55d"
version = "1.4.1"

[[deps.PDMats]]
deps = ["LinearAlgebra", "SparseArrays", "SuiteSparse"]
git-tree-sha1 = "cf494dca75a69712a72b80bc48f59dcf3dea63ec"
uuid = "90014a1f-27ba-587c-ab20-58faa44d9150"
version = "0.11.16"

[[deps.Parameters]]
deps = ["OrderedCollections", "UnPack"]
git-tree-sha1 = "34c0e9ad262e5f7fc75b10a9952ca7692cfc5fbe"
uuid = "d96e819e-fc66-5662-9728-84c9c7592b0a"
version = "0.12.3"

[[deps.Parsers]]
deps = ["Dates"]
git-tree-sha1 = "3d5bf43e3e8b412656404ed9466f1dcbf7c50269"
uuid = "69de0a69-1ddd-5017-9359-2bf0b02dc9f0"
version = "2.4.0"

[[deps.Pkg]]
deps = ["Artifacts", "Dates", "Downloads", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "REPL", "Random", "SHA", "Serialization", "TOML", "Tar", "UUIDs", "p7zip_jll"]
uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"
version = "1.8.0"

[[deps.PositiveFactorizations]]
deps = ["LinearAlgebra"]
git-tree-sha1 = "17275485f373e6673f7e7f97051f703ed5b15b20"
uuid = "85a6dd25-e78a-55b7-8502-1745935b8125"
version = "0.2.4"

[[deps.PreallocationTools]]
deps = ["Adapt", "ArrayInterfaceCore", "ForwardDiff"]
git-tree-sha1 = "50bf1eae0ace9a993ddfa78e83f9b76a898faf36"
uuid = "d236fae5-4411-538c-8e31-a6e3d9e00b46"
version = "0.4.1"

[[deps.Preferences]]
deps = ["TOML"]
git-tree-sha1 = "47e5f437cc0e7ef2ce8406ce1e7e24d44915f88d"
uuid = "21216c6a-2e73-6563-6e65-726566657250"
version = "1.3.0"

[[deps.Printf]]
deps = ["Unicode"]
uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"

[[deps.QuadGK]]
deps = ["DataStructures", "LinearAlgebra"]
git-tree-sha1 = "3c009334f45dfd546a16a57960a821a1a023d241"
uuid = "1fd47b50-473d-5c70-9696-f719f8f3bcdc"
version = "2.5.0"

[[deps.REPL]]
deps = ["InteractiveUtils", "Markdown", "Sockets", "Unicode"]
uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb"

[[deps.Random]]
deps = ["SHA", "Serialization"]
uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"

[[deps.RealDot]]
deps = ["LinearAlgebra"]
git-tree-sha1 = "9f0a1b71baaf7650f4fa8a1d168c7fb6ee41f0c9"
uuid = "c1ae055f-0cd5-4b69-90a6-9a35b1a98df9"
version = "0.1.0"

[[deps.RecipesBase]]
git-tree-sha1 = "6bf3f380ff52ce0832ddd3a2a7b9538ed1bcca7d"
uuid = "3cdcf5f2-1ef4-517c-9805-6587b60abb01"
version = "1.2.1"

[[deps.RecursiveArrayTools]]
deps = ["Adapt", "ArrayInterfaceCore", "ArrayInterfaceStaticArraysCore", "ChainRulesCore", "DocStringExtensions", "FillArrays", "GPUArraysCore", "IteratorInterfaceExtensions", "LinearAlgebra", "RecipesBase", "StaticArraysCore", "Statistics", "Tables", "ZygoteRules"]
git-tree-sha1 = "3004608dc42101a944e44c1c68b599fa7c669080"
uuid = "731186ca-8d62-57ce-b412-fbd966d074cd"
version = "2.32.0"

[[deps.Reexport]]
git-tree-sha1 = "45e428421666073eab6f2da5c9d310d99bb12f9b"
uuid = "189a3867-3050-52da-a836-e630ba90ab69"
version = "1.2.2"

[[deps.Referenceables]]
deps = ["Adapt"]
git-tree-sha1 = "e681d3bfa49cd46c3c161505caddf20f0e62aaa9"
uuid = "42d2dcc6-99eb-4e98-b66c-637b7d73030e"
version = "0.1.2"

[[deps.Requires]]
deps = ["UUIDs"]
git-tree-sha1 = "838a3a4188e2ded87a4f9f184b4b0d78a1e91cb7"
uuid = "ae029012-a4dd-5104-9daa-d747884805df"
version = "1.3.0"

[[deps.Rmath]]
deps = ["Random", "Rmath_jll"]
git-tree-sha1 = "bf3188feca147ce108c76ad82c2792c57abe7b1f"
uuid = "79098fc4-a85e-5d69-aa6a-4863f24498fa"
version = "0.7.0"

[[deps.Rmath_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "68db32dff12bb6127bac73c209881191bf0efbb7"
uuid = "f50d1b31-88e8-58de-be2c-1cc44531875f"
version = "0.3.0+0"

[[deps.SHA]]
uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce"
version = "0.7.0"

[[deps.Serialization]]
uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b"

[[deps.Setfield]]
deps = ["ConstructionBase", "Future", "MacroTools", "Requires"]
git-tree-sha1 = "38d88503f695eb0301479bc9b0d4320b378bafe5"
uuid = "efcf1570-3423-57d1-acb7-fd33fddbac46"
version = "0.8.2"

[[deps.SharedArrays]]
deps = ["Distributed", "Mmap", "Random", "Serialization"]
uuid = "1a1011a3-84de-559e-8e89-a11a2f7dc383"

[[deps.Sockets]]
uuid = "6462fe0b-24de-5631-8697-dd941f90decc"

[[deps.SortingAlgorithms]]
deps = ["DataStructures"]
git-tree-sha1 = "b3363d7460f7d098ca0912c69b082f75625d7508"
uuid = "a2af1166-a08f-5f64-846c-94a0d3cef48c"
version = "1.0.1"

[[deps.SparseArrays]]
deps = ["LinearAlgebra", "Random"]
uuid = "2f01184e-e22b-5df5-ae63-d93ebab69eaf"

[[deps.SpecialFunctions]]
deps = ["ChainRulesCore", "IrrationalConstants", "LogExpFunctions", "OpenLibm_jll", "OpenSpecFun_jll"]
git-tree-sha1 = "d75bda01f8c31ebb72df80a46c88b25d1c79c56d"
uuid = "276daf66-3868-5448-9aa4-cd146d93841b"
version = "2.1.7"

[[deps.SplittablesBase]]
deps = ["Setfield", "Test"]
git-tree-sha1 = "39c9f91521de844bad65049efd4f9223e7ed43f9"
uuid = "171d559e-b47b-412a-8079-5efa626c420e"
version = "0.1.14"

[[deps.Static]]
deps = ["IfElse"]
git-tree-sha1 = "f94f9d627ba3f91e41a815b9f9f977d729e2e06f"
uuid = "aedffcd0-7271-4cad-89d0-dc628f76c6d3"
version = "0.7.6"

[[deps.StaticArrays]]
deps = ["LinearAlgebra", "Random", "StaticArraysCore", "Statistics"]
git-tree-sha1 = "dfec37b90740e3b9aa5dc2613892a3fc155c3b42"
uuid = "90137ffa-7385-5640-81b9-e52037218182"
version = "1.5.6"

[[deps.StaticArraysCore]]
git-tree-sha1 = "ec2bd695e905a3c755b33026954b119ea17f2d22"
uuid = "1e83bf80-4336-4d27-bf5d-d5a4f845583c"
version = "1.3.0"

[[deps.Statistics]]
deps = ["LinearAlgebra", "SparseArrays"]
uuid = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"

[[deps.StatsAPI]]
deps = ["LinearAlgebra"]
git-tree-sha1 = "f9af7f195fb13589dd2e2d57fdb401717d2eb1f6"
uuid = "82ae8749-77ed-4fe6-ae5f-f523153014b0"
version = "1.5.0"

[[deps.StatsBase]]
deps = ["DataAPI", "DataStructures", "LinearAlgebra", "LogExpFunctions", "Missings", "Printf", "Random", "SortingAlgorithms", "SparseArrays", "Statistics", "StatsAPI"]
git-tree-sha1 = "d1bf48bfcc554a3761a133fe3a9bb01488e06916"
uuid = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91"
version = "0.33.21"

[[deps.StatsFuns]]
deps = ["ChainRulesCore", "HypergeometricFunctions", "InverseFunctions", "IrrationalConstants", "LogExpFunctions", "Reexport", "Rmath", "SpecialFunctions"]
git-tree-sha1 = "5783b877201a82fc0014cbf381e7e6eb130473a4"
uuid = "4c63d2b9-4356-54db-8cca-17b64c39e42c"
version = "1.0.1"

[[deps.StructArrays]]
deps = ["Adapt", "DataAPI", "StaticArraysCore", "Tables"]
git-tree-sha1 = "8c6ac65ec9ab781af05b08ff305ddc727c25f680"
uuid = "09ab397b-f2b6-538f-b94a-2f83cf4a842a"
version = "0.6.12"

[[deps.StructTypes]]
deps = ["Dates", "UUIDs"]
git-tree-sha1 = "ca4bccb03acf9faaf4137a9abc1881ed1841aa70"
uuid = "856f2bd8-1eba-4b0a-8007-ebc267875bd4"
version = "1.10.0"

[[deps.SuiteSparse]]
deps = ["Libdl", "LinearAlgebra", "Serialization", "SparseArrays"]
uuid = "4607b0f0-06f3-5cda-b6b1-a6196a1729e9"

[[deps.SymbolicRegression]]
deps = ["Distributed", "JSON3", "LineSearches", "LinearAlgebra", "LossFunctions", "Optim", "Pkg", "PreallocationTools", "Printf", "Random", "Reexport", "SpecialFunctions", "SymbolicUtils", "Zygote"]
git-tree-sha1 = "46a3fcb433f06f00f60f706cf281a6d8bcb509dc"
uuid = "8254be44-1295-4e6a-a16d-46603ac705cb"
version = "0.10.2"

[[deps.SymbolicUtils]]
deps = ["AbstractTrees", "Bijections", "ChainRulesCore", "Combinatorics", "ConstructionBase", "DataStructures", "DocStringExtensions", "DynamicPolynomials", "IfElse", "LabelledArrays", "LinearAlgebra", "Metatheory", "MultivariatePolynomials", "NaNMath", "Setfield", "SparseArrays", "SpecialFunctions", "StaticArrays", "TermInterface", "TimerOutputs"]
git-tree-sha1 = "027b43d312f6d52187bb16c2d4f0588ddb8c4bb2"
uuid = "d1185830-fcd6-423d-90d6-eec64667417b"
version = "0.19.11"

[[deps.TOML]]
deps = ["Dates"]
uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76"
version = "1.0.0"

[[deps.TableTraits]]
deps = ["IteratorInterfaceExtensions"]
git-tree-sha1 = "c06b2f539df1c6efa794486abfb6ed2022561a39"
uuid = "3783bdb8-4a98-5b6b-af9a-565f29a5fe9c"
version = "1.0.1"

[[deps.Tables]]
deps = ["DataAPI", "DataValueInterfaces", "IteratorInterfaceExtensions", "LinearAlgebra", "OrderedCollections", "TableTraits", "Test"]
git-tree-sha1 = "5ce79ce186cc678bbb5c5681ca3379d1ddae11a1"
uuid = "bd369af6-aec1-5ad0-b16a-f7cc5008161c"
version = "1.7.0"

[[deps.Tar]]
deps = ["ArgTools", "SHA"]
uuid = "a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e"
version = "1.10.0"

[[deps.TermInterface]]
git-tree-sha1 = "7aa601f12708243987b88d1b453541a75e3d8c7a"
uuid = "8ea1fca8-c5ef-4a55-8b96-4e9afe9c9a3c"
version = "0.2.3"

[[deps.Test]]
deps = ["InteractiveUtils", "Logging", "Random", "Serialization"]
uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[[deps.ThreadsX]]
deps = ["ArgCheck", "BangBang", "ConstructionBase", "InitialValues", "MicroCollections", "Referenceables", "Setfield", "SplittablesBase", "Transducers"]
git-tree-sha1 = "d223de97c948636a4f34d1f84d92fd7602dc555b"
uuid = "ac1d9e8a-700a-412c-b207-f0111f4b6c0d"
version = "0.1.10"

[[deps.TimerOutputs]]
deps = ["ExprTools", "Printf"]
git-tree-sha1 = "9dfcb767e17b0849d6aaf85997c98a5aea292513"
uuid = "a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f"
version = "0.5.21"

[[deps.Transducers]]
deps = ["Adapt", "ArgCheck", "BangBang", "Baselet", "CompositionsBase", "DefineSingletons", "Distributed", "InitialValues", "Logging", "Markdown", "MicroCollections", "Requires", "Setfield", "SplittablesBase", "Tables"]
git-tree-sha1 = "c76399a3bbe6f5a88faa33c8f8a65aa631d95013"
uuid = "28d57a85-8fef-5791-bfe6-a80928e7c999"
version = "0.4.73"

[[deps.UUIDs]]
deps = ["Random", "SHA"]
uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"

[[deps.UnPack]]
git-tree-sha1 = "387c1f73762231e86e0c9c5443ce3b4a0a9a0c2b"
uuid = "3a884ed6-31ef-47d7-9d2a-63182c4928ed"
version = "1.0.2"

[[deps.Unicode]]
uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"

[[deps.Zlib_jll]]
deps = ["Libdl"]
uuid = "83775a58-1f1d-513f-b197-d71354ab007a"
version = "1.2.12+3"

[[deps.Zygote]]
deps = ["AbstractFFTs", "ChainRules", "ChainRulesCore", "DiffRules", "Distributed", "FillArrays", "ForwardDiff", "GPUArrays", "GPUArraysCore", "IRTools", "InteractiveUtils", "LinearAlgebra", "LogExpFunctions", "MacroTools", "NaNMath", "Random", "Requires", "SparseArrays", "SpecialFunctions", "Statistics", "ZygoteRules"]
git-tree-sha1 = "c3f4af6b167f0c181148650221a4cbabf6bbb8a6"
uuid = "e88e6eb3-aa80-5325-afca-941959d7151f"
version = "0.6.47"

[[deps.ZygoteRules]]
deps = ["MacroTools"]
git-tree-sha1 = "8c1a8e4dfacb1fd631745552c8db35d0deb09ea0"
uuid = "700de1a5-db45-46bc-99cf-38207098b444"
version = "0.2.2"

[[deps.libblastrampoline_jll]]
deps = ["Artifacts", "Libdl", "OpenBLAS_jll"]
uuid = "8e850b90-86db-534c-a0d3-1478176c7d93"
version = "5.1.1+0"

[[deps.nghttp2_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "8e850ede-7688-5339-a07c-302acd2aaf8d"
version = "1.48.0+0"

[[deps.p7zip_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "3f19e933-33d8-53b3-aaab-bd5110c3b7a0"
version = "17.4.0+0"

Practically, it may be possible to build the Project.toml file. We can then generate the Manifest.toml by invoking Pkg.resolve().
The summary of this experiment is as follows.

We created individual depots and environments for two packages: SymbolicRegression.jl and BitInformation.jl
We created a common depot and environment for both of them. Because the individual depots were lower on the depot stack, we didn't need to install any additional packages.
Having the two depots around did not hinder the creation of the common environment. In fact, it made it significantly faster since the packages were already locally available.
Note that I did not manipulate the JULIA_LOAD_PATH. The only modification to code loading I did was creating the depot stack and activating different projects.
The bottom line here is the depot preloading does not interfere with creating the common environment between the two packages.

You guys are getting distracted by the JULIA_PROJECT and JULIA_LOAD_PATH manipulation. This is only temporarily filling in for something the Python packages themselves can do. The main point is that we can package depots with preconfigured environments that do not need further installation.
Solving the conflicting version dependency issue is one step further. It either needs the user to create the common environment or we need to do something in a post-link script. Nonetheless, packaging the depots help rather than hinder the creation of the common environment. They also allow the distinct environments to be completely installed.

Were there any version clashes? Yes.
pysr_env/Manifest.toml
[[deps.DocStringExtensions]]
deps = ["LibGit2"]
git-tree-sha1 = "b19534d1895d702889b219c382a6e18010797f0b"
uuid = "ffbed154-4ef7-542d-bbb7-c09d3a79fcae"
version = "0.8.6"

xbitinfo_env/Manifest.toml
[[deps.DocStringExtensions]]
deps = ["LibGit2"]
git-tree-sha1 = "5158c2b41018c5f7eb1470d558127ac274eca0c9"
uuid = "ffbed154-4ef7-542d-bbb7-c09d3a79fcae"
version = "0.9.1"

common_env/Manifest.toml
[[deps.DocStringExtensions]]
deps = ["LibGit2"]
git-tree-sha1 = "b19534d1895d702889b219c382a6e18010797f0b"
uuid = "ffbed154-4ef7-542d-bbb7-c09d3a79fcae"
version = "0.8.6"

Here's the individual output of pkg"status -m" for each of the Manifests.toml

common_env manifest status
(@common_env) pkg> st -m
Status `~/src/depottest/common_depot/environments/common_env/Manifest.toml`
  [621f4979] AbstractFFTs v1.2.1
  [1520ce14] AbstractTrees v0.4.2
  [79e6a3ab] Adapt v3.4.0
  [dce04be8] ArgCheck v2.3.0
  [4fba245c] ArrayInterface v6.0.23
  [30b0a656] ArrayInterfaceCore v0.1.20
  [b0d46f97] ArrayInterfaceStaticArrays v0.1.4
  [dd5226c6] ArrayInterfaceStaticArraysCore v0.1.0
  [15f4f7f2] AutoHashEquals v0.2.0
  [198e06fe] BangBang v0.3.36
  [9718e550] Baselet v0.1.1
  [e2ed5e7c] Bijections v0.1.4
  [de688a37] BitInformation v0.6.0
  [fa961155] CEnum v0.4.2
  [49dc2e85] Calculus v0.5.1
  [082447d4] ChainRules v1.44.6
  [d360d2e6] ChainRulesCore v1.15.4
  [9e997f8a] ChangesOfVariables v0.1.4
  [861a8166] Combinatorics v1.0.2
  [bbf7d656] CommonSubexpressions v0.3.0
⌅ [34da2185] Compat v3.46.0
  [a33af91c] CompositionsBase v0.1.1
  [187b0558] ConstructionBase v1.4.1
  [9a962f9c] DataAPI v1.10.0
  [864edb3b] DataStructures v0.18.13
  [e2d170a0] DataValueInterfaces v1.0.0
  [244e2a9f] DefineSingletons v0.1.2
  [b429d917] DensityInterface v0.4.0
  [163ba53b] DiffResults v1.0.3
  [b552c78f] DiffRules v1.11.1
  [31c24e10] Distributions v0.25.70
⌅ [ffbed154] DocStringExtensions v0.8.6
  [fa6b7ba4] DualNumbers v0.6.8
  [7c1d4256] DynamicPolynomials v0.4.5
  [e2ba6199] ExprTools v0.1.8
  [1a297f60] FillArrays v0.13.4
  [6a86dc24] FiniteDiff v2.15.0
  [f6369f11] ForwardDiff v0.10.32
  [0c68f7d7] GPUArrays v8.5.0
  [46192b85] GPUArraysCore v0.1.2
  [34004b35] HypergeometricFunctions v0.3.11
  [7869d1d1] IRTools v0.4.6
  [615f187c] IfElse v0.1.1
  [22cec73e] InitialValues v0.3.1
  [3587e190] InverseFunctions v0.1.7
  [92d709cd] IrrationalConstants v0.1.1
  [82899510] IteratorInterfaceExtensions v1.0.0
  [692b3bcd] JLLWrappers v1.4.1
  [0f8b85d8] JSON3 v1.9.5
  [929cbde3] LLVM v4.14.0
  [2ee39098] LabelledArrays v1.12.0
  [d3d80556] LineSearches v7.2.0
  [2ab3a3ac] LogExpFunctions v0.3.18
  [30fc2ffe] LossFunctions v0.8.0
  [1914dd2f] MacroTools v0.5.9
  [e9d8d322] Metatheory v1.3.4
  [128add7d] MicroCollections v0.1.2
  [e1d29d7a] Missings v1.0.2
  [102ac46a] MultivariatePolynomials v0.4.6
  [d8a4904e] MutableArithmetics v1.0.4
  [d41bc354] NLSolversBase v7.8.2
  [77ba4419] NaNMath v1.0.1
  [429524aa] Optim v1.7.2
  [bac558e1] OrderedCollections v1.4.1
  [90014a1f] PDMats v0.11.16
  [d96e819e] Parameters v0.12.3
  [69de0a69] Parsers v2.4.0
  [85a6dd25] PositiveFactorizations v0.2.4
⌅ [d236fae5] PreallocationTools v0.4.1
  [21216c6a] Preferences v1.3.0
  [1fd47b50] QuadGK v2.5.0
  [c1ae055f] RealDot v0.1.0
  [3cdcf5f2] RecipesBase v1.2.1
  [731186ca] RecursiveArrayTools v2.32.0
  [189a3867] Reexport v1.2.2
  [42d2dcc6] Referenceables v0.1.2
  [ae029012] Requires v1.3.0
  [79098fc4] Rmath v0.7.0
⌅ [efcf1570] Setfield v0.8.2
  [a2af1166] SortingAlgorithms v1.0.1
  [276daf66] SpecialFunctions v2.1.7
  [171d559e] SplittablesBase v0.1.14
  [aedffcd0] Static v0.7.6
  [90137ffa] StaticArrays v1.5.6
  [1e83bf80] StaticArraysCore v1.3.0
  [82ae8749] StatsAPI v1.5.0
  [2913bbd2] StatsBase v0.33.21
  [4c63d2b9] StatsFuns v1.0.1
  [09ab397b] StructArrays v0.6.12
  [856f2bd8] StructTypes v1.10.0
  [8254be44] SymbolicRegression v0.10.2
  [d1185830] SymbolicUtils v0.19.11
  [3783bdb8] TableTraits v1.0.1
  [bd369af6] Tables v1.7.0
⌅ [8ea1fca8] TermInterface v0.2.3
  [ac1d9e8a] ThreadsX v0.1.10
  [a759f4b9] TimerOutputs v0.5.21
  [28d57a85] Transducers v0.4.73
  [3a884ed6] UnPack v1.0.2
  [e88e6eb3] Zygote v0.6.47
  [700de1a5] ZygoteRules v0.2.2
  [dad2f222] LLVMExtra_jll v0.0.16+0
  [efe28fd5] OpenSpecFun_jll v0.5.5+0
  [f50d1b31] Rmath_jll v0.3.0+0
  [0dad84c5] ArgTools v1.1.1
  [56f22d72] Artifacts
  [2a0f44e3] Base64
  [ade2ca70] Dates
  [8bb1440f] DelimitedFiles
  [8ba89e20] Distributed
  [f43a241f] Downloads v1.6.0
  [7b1f6079] FileWatching
  [9fa8497b] Future
  [b77e0a4c] InteractiveUtils
  [4af54fe1] LazyArtifacts
  [b27032c2] LibCURL v0.6.3
  [76f85450] LibGit2
  [8f399da3] Libdl
  [37e2e46d] LinearAlgebra
  [56ddb016] Logging
  [d6f4376e] Markdown
  [a63ad114] Mmap
  [ca575930] NetworkOptions v1.2.0
  [44cfe95a] Pkg v1.8.0
  [de0858da] Printf
  [3fa0cd96] REPL
  [9a3f8284] Random
  [ea8e919c] SHA v0.7.0
  [9e88b42a] Serialization
  [1a1011a3] SharedArrays
  [6462fe0b] Sockets
  [2f01184e] SparseArrays
  [10745b16] Statistics
  [4607b0f0] SuiteSparse
  [fa267f1f] TOML v1.0.0
  [a4e569a6] Tar v1.10.0
  [8dfed614] Test
  [cf7118a7] UUIDs
  [4ec0a83e] Unicode
  [e66e0078] CompilerSupportLibraries_jll v0.5.2+0
  [deac9b47] LibCURL_jll v7.84.0+0
  [29816b5a] LibSSH2_jll v1.10.2+0
  [c8ffd9c3] MbedTLS_jll v2.28.0+0
  [14a3606d] MozillaCACerts_jll v2022.2.1
  [4536629a] OpenBLAS_jll v0.3.20+0
  [05823500] OpenLibm_jll v0.8.1+0
  [83775a58] Zlib_jll v1.2.12+3
  [8e850b90] libblastrampoline_jll v5.1.1+0
  [8e850ede] nghttp2_jll v1.48.0+0
  [3f19e933] p7zip_jll v17.4.0+0
Info Packages marked with ⌅ have new versions available but cannot be upgraded.   If there are blockers a priori, then pRIP. This would be my next step in this exciting story. If we can establish at least that a process is possible to achieve your vision, then we can ask for this to be reviewed and merged.
TLDR, how about these in order for now?

Can we get a process rolling to packaging julia-stuff?
Submit a CFEP and ask for a vote/feedback from more members of the community.
Ask maintainer here to review and merge.

Or do you think this is unnecessarily elaborate? Instead, should we just experiment here for now?

That should happen but the path for this pull request is much simpler.

I need to work on the upstream PySR request with Miles: https://github.com/MilesCranmer/PySR/pull/186
Once we merge a version of that pull request, we can see how much activation script support we need here. I think we can pull out the JULIA_PROJECT and JULIA_LOAD_PATH manipulation entirely which should squash Isuru's concerns.
What should be left here is the JULIA_DEPOT_PATH manipulation and the additional depot. All that does is locally cache Julia packages making it easier to use the Python packages.

The goal here is to make the pysr.install() is no longer necessary. All that needs to be happen is we need to move the activation of the pysr Julia environment before we try to load pyjulia. Once that happens, then the conda environment does not need to directly manipulate the Julia environment stack directly. This means there is no longer potential interference between conda packages. At that point, I think the final decision is almost entirely within Miles' domain.

In the meantime, could you update the package on you have on your channel, @ngam before I start breaking things again?

Once we merge a version of that pull request, we can see how much activation script support we need here. I think we can pull out the JULIA_PROJECT and JULIA_LOAD_PATH manipulation entirely which should squash Isuru's concerns.

No, it doesn't. It just moves the problem to pysr and whatever the python package is. import julia will only use the project env variable the first time it's used and for the second project it's a no-op. This is also a bad fix, because depending on
import pysr
import xbitinfo

vs
import xbtinfo
import pysr

the location will change.
In my opinion, the current solution (before this PR) avoids these issues by having one common environment and is superior to the solution proposed in this PR.

In my opinion, the current solution (before this PR) avoids these issues by having one common environment and is superior to the solution proposed in this PR.

The current solution (before this PR) does not exclusively use one common environment. It starts pyjulia and calls julia.install() which installs PyCall.jl into whatever the current environment is active. Then it activates @pysr-0.10.1 via Pkg.activate to install SymbolicRegression.jl and ClusterManagers.jl.
While there is a good argument for a conda package to install PyCall.jl into the common environment that package should probably be either pyjulia itself or perhaps julia-pycall.

the location will change.

No because neither package does anything upon import. Currently pyjulia, pysr, and soon xbitinfo are configured to install packages via [pkg].install(). The environment they activate is also configurable.

For cross reference here is the xbitinfo pull request which copies the current pysr approach:
https://github.com/observingClouds/xbitinfo/pull/132
https://github.com/observingClouds/xbitinfo/blob/5b59b00fb191a12ea8ab5b7804ce4ded87aca593/xbitinfo/julia_helpers.py#L14-L47
The install function in julia_helpers.py will do the following:

Invoke julia.install() which will install PyCall.jl into whatever the current Julia environment is.
Activate the @xbitinfo-{version} environment.
Install BitInformation.jl and StatsBase.jl into the @xbitinfo-{version} environment
Instantiate the @xbitinfo-{version} environment
Precompile the @xbitinfo-{version} environment

To be clear, I'm not the one introducing new environments per Python package with this pull request. That existed in pysr before and is now being duplicated into xbitinfo.
What I'm trying to do is move the install() step into build phase so they can be managed by conda. The order of the install() invocations is what will result in variability, and I'm trying to eliminate that variability. If pysr.install() is invoked first and then xbitinfo.install() is invoked second, then xbitinfo.install() will end up installing PyCall.jl into the pysr environment.
What we want at the end of the day is for conda or mamba to actually do the installation so that eventually conda/mamba can eventually do dependency management.

If two conda packages (pysr, xbitinfo) try to create two different environments and activate both julia environments, those two conda packages are fundamentally not compatible with each other and we should disallow installing both conda packages to the same conda environment.

If two python packages (pysr, xbitinfo) try to create two different environments and activate both julia environments, those two python packages are fundamentally not compatible with each other and we should disallow installing both python packages to the same conda environment.

I think you are beginning to the understand the developing situation now. Note that the julia_project can be configured.
https://github.com/MilesCranmer/PySR/blob/d09ade8628f541eb94028009bacdb9a55cb22ef5/pysr/sr.py#L508-L512
During install() it can also be defined as an argument.
https://github.com/MilesCranmer/PySR/blob/d09ade8628f541eb94028009bacdb9a55cb22ef5/pysr/julia_helpers.py#L9
Thus it's possible to have them in the same conda environment if we configure them correctly. The question is when can we configure them or do we let the end-user figure it out?

Just merged @mkitti's PR, so the latest PySR version (0.10.2) will now use JULIA_PROJECT.

Folks, most of the conversation here is kind of above my head, sorry for not contributing more. However, we do have tons of nice discussions here and it would be nice to summary it, preferably as docs (even if in draft form), somewhere.

To clarify, what the PySR pull request allows us to do is tell PySR which Julia environment to find PyCall to start pyjulia as well as the rest of the components.
We can specify the julia_project argument to the PySRRegressor class or the julia_project argument to install. Previously, this would only affect the environment where PySR would locate SymbolicRegression.jl and ClusterManagers.jl. Now, it will also look for PyCall.jl in the same environment. This allows the PySR dependencies to be completely contained within a single environment rather than potentially being split among two environments.
Because of the stacking order defined in the julia-feedstock this automatically will fallback to looking for PyCall.jl in the default "base" Julia environment named after the conda environment.
For now this means the environment can be entirely contained with pysr-0.10.x. For later this allows us to potentially redirect PySR to use a common environment with another package.

Folks, most of the conversation here is kind of above my head, sorry for not contributing more. However, we do have tons of nice discussions here and it would be nice to summary it, preferably as docs (even if in draft form), somewhere.

@ocefpaf, sorry. There is a lot going on here and a lot more to debate.
The main innovation here is that we can embed a Julia depot within the conda package which allows us to "install" Julia packages for a specific project in a predefined Julia environment. Essentially this allows us to move the [package].install() step into the conda-forge build process.
The main question is how this will work when integrating multiple packages. For example, if someone wanted to install both xbitinfo and pysr in the same conda environment, how will that work?
If we figure out there is some utility to this approach, I'll send a pull request your way. It looks promising to me, but not everyone is convinced. For now, I need to unravel and simplify a few things in the feedstock here. Perhaps the main practical thing for you is examining the changes to the julia_helpers.jl in PySR: https://github.com/MilesCranmer/PySR/commit/b14e38ac6e7c719c26d1b936f2f960ab5363348f

In 8a4dcb3 I disabled manipulation of JULIA_LOAD_PATH and JULIA_PROJECT. However, I have not updated the PySR source.
My expectation is that this run will fail because Julia will not be able to locate PyCall.jl.

@ocefpaf, sorry. There is a lot going on here and a lot more to debate.

Don't be sorry. The discussion here is important! I'm sorry I don't have much to contribute to it.

@mkitti - sorry, it looks like 0.10.2 didn't actually include your change. Released 0.10.3 just now, which does.

@mkitti - sorry, it looks like 0.10.2 didn't actually include your change. Released 0.10.3 just now, which does.

Oh, hah, I thought you were waiting for me to test it before tagging the release.

I'm perplexed why I'm getting this error on osx only:
Traceback (most recent call last):
  File "/Users/runner/miniforge3/conda-bld/pysr_1662508039913/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.10/site-packages/pysr/julia_helpers.py", line 126, in init_julia
    from julia import Main as _Main
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 672, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 632, in _load_backward_compatible
  File "/Users/runner/miniforge3/conda-bld/pysr_1662508039913/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.10/site-packages/julia/core.py", line 247, in load_module
    JuliaMainModule(self, fullname))
  File "/Users/runner/miniforge3/conda-bld/pysr_1662508039913/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.10/site-packages/julia/core.py", line 149, in __init__
    self._julia = loader.julia
  File "/Users/runner/miniforge3/conda-bld/pysr_1662508039913/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.10/site-packages/julia/core.py", line 239, in julia
    self.__class__.julia = julia = Julia()
  File "/Users/runner/miniforge3/conda-bld/pysr_1662508039913/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.10/site-packages/julia/core.py", line 483, in __init__
    raise UnsupportedPythonError(jlinfo)
julia.core.UnsupportedPythonError: It seems your Julia and PyJulia setup are not supported.

If line 126 throws an UnsupportedPythonError, shouldn't line 129 catch it?
https://github.com/MilesCranmer/PySR/blob/b521bff08ed6553f9464172d2687a0f42abff39c/pysr/julia_helpers.py#L125-L138

Yes, that error should be caught. Very very weird. Is there any way conda could be using a custom traceback mechanism here, that would ignore the try-except?

I figured it out. There was an additional error. I'm thinking the original error might have been that "some_env" did not resolve to a valid path.
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/runner/miniforge3/conda-bld/pysr_1662506003093/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.10/site-packages/pysr/julia_helpers.py", line 32, in install
    Main = init_julia(julia_project)
  File "/Users/runner/miniforge3/conda-bld/pysr_1662506003093/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.10/site-packages/pysr/julia_helpers.py", line 133, in init_julia
    jl = Julia(compiled_modules=False)
  File "/Users/runner/miniforge3/conda-bld/pysr_1662506003093/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.10/site-packages/julia/core.py", line 513, in __init__
    self._call("const PyCall = Base.require({0})".format(PYCALL_PKGID))
  File "/Users/runner/miniforge3/conda-bld/pysr_1662506003093/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.10/site-packages/julia/core.py", line 549, in _call
    self.check_exception(src)
  File "/Users/runner/miniforge3/conda-bld/pysr_1662506003093/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.10/site-packages/julia/core.py", line 603, in check_exception
    raise JuliaError(u'Exception \'{}\' occurred while calling julia code:\n{}'
julia.core.JuliaError: Exception 'ArgumentError' occurred while calling julia code:
const PyCall = Base.require(Base.PkgId(Base.UUID("438e738f-606a-5dbb-bf0a-cddfbfd45ab0"), "PyCall"))

Could it be two different pyjulia installations - one raises the error, the other is used to check it, hence the incompatibility with the except?

Ah, sorry, the thread didn’t update. Nice going!

@conda-forge-admin, please rerender

I'm at a stopping point on this. This package functions much as it did before this pull request. It still uses a versioned Julia environment @pysr-0.10.3 by default.
What's new

The Julia environment used can be modified by passing the julia_project argument to pysr.install and pysr.sr.init_julia. It defaults to @psr-$version.
PyCall is installed into the Julia environment.
We package a Julia depot containing the needed Julia packages and their dependencies. We put it on the depot stack by modifying the JULIA_DEPOT_PATH with an activate script.
Within the depot we include a configured, resolved, and ready to use @psr-$version environment.
The user does not need to invoke pysr.install() before using the package.

Basically this conda package now installs the Julia components for you in a default Julia environment. For other Julia environments, pysr.install("/path/to/env") can still be used to install the needed Julia packages. For example, the user might want to do pysr.install(os.environ["JULIA_PROJECT"]). If one then specified the julia_project argument to PySRRegressor, then pysr can used along side other packages such as xbitinfo.
Example use
import numpy as np
import os

X = 2 * np.random.randn(100, 5)
y = 2.5382 * np.cos(X[:, 3]) + X[:, 0] ** 2 - 0.5

from pysr import PySRRegressor

model = PySRRegressor(
    model_selection="best",  # Result is mix of simplicity+accuracy
    niterations=40,
    binary_operators=["+", "*"],
    unary_operators=[
        "cos",
        "exp",
        "sin",
        "inv(x) = 1/x",
	# ^ Custom operator (julia syntax)
    ],
    extra_sympy_mappings={"inv": lambda x: 1 / x},
    # ^ Define operator for SymPy as well
    loss="loss(x, y) = (x - y)^2",
    # ^ Custom loss function (julia syntax)
    # Uncomment the line below if installed into the default Julia environment
    # julia_project=os.environ["JULIA_PROJECT"]),
)

# Previously this would complain about the user needing to invoke pysr.install()
# Now it just works. No `pysr.install()` needed
model.fit(X, y)

No post-link script
As far as I understand, we are discouraged from doing pysr.install(os.environ["JULIA_PROJECT"]) for the user via a post-link script. Thus, the user is responsible for completing that step if they want to integrate pysr with another Python package that also uses Julia such as xbitinfo.
Potential post-link script for another pull request
Using simple bash commands in a post-link script might allow us to parse the active Project.toml and insert a few lines.
The Project.toml for @pysr-$version looks like this:
[deps]
ClusterManagers = "34f1f09b-3a8b-5176-ab39-66d58a4d544e"
PyCall = "438e738f-606a-5dbb-bf0a-cddfbfd45ab0"
SymbolicRegression = "8254be44-1295-4e6a-a16d-46603ac705cb"

We might be able to echo the last few lines into the user's Project.toml for them via a simple bash script?
Concerns

And introducing silently inconsistent conda envs is a blocker, so we have to ensure that's not the case or at least make the solver aware of conflicts somehow.

At the end of the day the user needs to do some final integration. In particular, they need to configure a common Julia environment if they want to combine different sets of Julia packages. By doing so, they will resolve any conflicts.
This situation was not introduced by this pull request. This pull request does make it easier to resolve by providing a Julia package cache that the combined Julia environment can draw upon. The packaged depot itself does not create any conflicts with other packages.

I'm up for trying this out. I also don't see potential conflicts as an issue right now.
With the current approach before this PR, I don't think you could have both xbitinfo and pysr running at the same time anyways - whereas with this PR, you might be able to. At the very least, this lets us test this.

No matter, I think to satisfy the main concern of isuruf, we need to think about a way to establish a conflict between two julia-packages depending on two explicitly different versions of the same third julia-package. This could be done manually, e.g. run_constrained, but we should automate it if we really want this to be adopted.

To be clear, there is a solution on the Julia side between BitInformation.jl and SymbolicRegression.jl. The two can exist in the same Julia environment.
(jl_DhIDzb) pkg> add BitInformation
   Resolving package versions...
    Updating `/tmp/jl_DhIDzb/Project.toml`
  [de688a37] + BitInformation v0.6.0
    Updating `/tmp/jl_DhIDzb/Manifest.toml`
...
  [34da2185] + Compat v4.2.0
...
  [ffbed154] + DocStringExtensions v0.9.1

(jl_DhIDzb) pkg> add SymbolicRegression
   Resolving package versions...
    Updating `/tmp/jl_DhIDzb/Project.toml`
  [8254be44] + SymbolicRegression v0.10.2
    Updating `/tmp/jl_DhIDzb/Manifest.toml`
...
⌅ [34da2185] ↓ Compat v4.2.0 ⇒ v3.46.0
...
⌅ [ffbed154] ↓ DocStringExtensions v0.9.1 ⇒ v0.8.6

The problem is on the Python side between pysr and xbitinfo at the moment. The solution is that pysr and xbitinfo need to use the same Julia environment in order to be compatible, but currently default to using different environments.

pysr defaults to using the Julia environment pysr-$version
xbitinfo defaults to using the Julia environment xbitinfo-$version

Solutions:

The end user configures both packages to use the same Julia environment
We, the conda-forge packagers, configure both packages to use $JULIA_PROJECT environment by default.
The problem with this is that we do not know what $JULIA_PROJECT is at build time, so the user has to manually install the Julia packages into $JULIA_PROJECT.
We, the conda-forge packagers, preconfigure pysr-$version and xbitinfo-$version Julia environments to be compatible. Based on the above we just need to pin Compat.jl to v3.46.0 and DocStringExtensions.jl to v0.8.6. Since we are shipping the projects in a depot, we can do this at build time. The problem is the user could then modify the environments to be incompatible.
We, the conda-forge packagers, patch both packages to use the @conda-forge Julia environment by default. We then curate a common Julia environment for Python packages and ship that. The @conda-forge environment can exist in the environment stack below the user's environments. This may be a fully concrete environment that we pre-build, or it could be an environment that we manage for the user.

sounds good to me as well.

@conda-forge admin, please restart ci

I don't have any objections. The needed actions are all on the xbitinfo-feedstock side at the moment where we should pin Compat.jl and DocStringExtensions.jl. For some reason SymbolicRegressions.jl is not compatible with the latest versions of those. Perhaps we should investigate that later as well.
Basically all that we have done here is included a cache of Julia packages within the pysr conda-forge package, which happens to satisfy Julia's initial installation requirements. If someone really wanted to use the packages together, the easiest route would be to set the julia_project of PySRRegressor to @xbitinfo-{version} and install SymbolicRegressions.jl there.

Incidentally, pythoncall is being added to conda-forge:

Technically PythonCall.jl is a Julia package that calls Python code. It plays a similar role as PyCall.jl.
The two feedstocks linked above are actually the Python components.

The conda version of PySR a bit out-of-date so I'll plan to merge on Monday (afternoon eastern) unless @isuruf has any objections.
Best,
Miles

Can someone summarize what this PR does?

Can someone summarize what this PR does?

See https://github.com/conda-forge/pysr-feedstock/pull/43#issuecomment-1239243732
We install a secondary Julia depot in $CONDA_PREFIX/share/pysr/depot. This contains a package cache containing all of PySR's dependencies. It also contains a shared environment @pysr-0.10.4 that is ready to use by PySR by default without the user having to perform an explicit install step. The cache is calculated during build time allowing conda-forge the ability precisely control Julia package versions such as by pinning.
The cache can also be used by the user if they specify a julia_project environment to pysr.install(). Julia will not need to download additional packages if the versions contained in the package cache are correct. Othereise, they can be ignored. Importantly, the package cache does interfere with the user's package cache in $CONDA_PREFIX/share/julia/packages nor does it interfere with potential other depots. It is simply a cache. This is because Julia packages are stored in hashed directories depending on their version and and git commit.
If the user wants to use other Julia packages either directly or via a Python package, they should explicitly specify an environment containing those other packages.
We can only supply a resolved Project.toml and Manifest.toml. We do have a mechanism to modify the  Project.toml and Manifest.toml in the current Julia environment via post-link scripts. Thr required simplicity of what we can do in post-link prevents use from invoking the Julia package manager to do this for us.
In summary, we supply a Julia package cache and environment in a depot that makes PySR ready to use.

Why can this not be done in an activate script?

Why can this not be done in an activate script?

You want to run the Julia package manager every time someone activates an environment?
Perhaps you mean a post-link script?
In that case, I'm taking the advice here from here:
https://docs.conda.io/projects/conda-build/en/latest/resources/link-scripts.html

Post-link and pre-unlink scripts should:

Be avoided whenever possible.
Not touch anything other than the files being installed.
Not write anything to stdout or stderr, unless an error occurs.
Not depend on any installed or to-be-installed conda packages.
Depend only on simple system tools such as rm, cp, mv, and ln.

If this doesn't apply, we could certainly take that approach as we started to in #41.

Also how do you deal with packages that may have incompatible mutual dependencies? e.g. if you release packages A and B, which depend on package C, how do you ensure that C is compatible with both versions of A and B that are installed? The logical answer seems to be that you need C to be a Conda package too. The conclusion of which is that you need to release all Julia packages you depend on as individual Conda packages, if you want to ensure compatible versions. This seems like a huge undertaking, similar in scope to conda-forge itself.

Part of the point here is to deliver the PySR software as ready-to-use. By invoking the Julia package manager during conda package build time we can control the versions included, pinning the dependencies as needed. The status quo that exists before this pull request is that the user has to take additional deliberate steps to make this package function. As such, Julia package management is lies completely out side of conda-forge's control.
I concur that packaging individual Julia packages would be the ideal. To support PySR, this would require adding ~100 individual packages. That's a tremendous barrier to starting though. What I'm trying to sort out here is a way to get started while delivering a practical solution.

What guarantees that Pkg will use bundled versions of the packages? If one of the dependencies gets updated in the future, then Julia will end up installing it from the internet instead of from the stacked depot, right? And if you're OK with that, then what was the point of bundling the dependencies in the first place?

We package a complete environment with a Project.toml and Manifest.toml, so that specifies the exact package versions to use. That complete environment is the default used by PySR and it is "installed" after conda finishes. As time goes on this package will be rebuilt with new Julia dependencies. The point of bundling the dependencies in the first place is that conda and conda-forge are now managing those files. The bundled solution has been tested as part of the CI here. There is some appeal in that. In my understanding, this is what conda-forge is about and what separates from just using conda and any other channel. conda-forge aims to deliver a completely tested solution built from source.

Also how do you deal with packages that may have incompatible mutual dependencies? e.g. if you release packages A and B, which depend on package C, how do you ensure that C is compatible with both versions of A and B that are installed? The logical answer seems to be that you need C to be a Conda package too. The conclusion of which is that you need to release all Julia packages you depend on as individual Conda packages, if you want to ensure compatible versions. This seems like a huge undertaking, similar in scope to conda-forge itself.

I discussed this above. We are currently addressing a N = 2 situation. The other conda package is https://github.com/conda-forge/xbitinfo-feedstock . By bundling the packages at build time, we have a chance to reconcile the package versions. I've done the analysis in this case and we only need to pin two package versions, Compat.jl and DocStringExtensions.jl. All other packages are compatible at the latest versions.
The next targets for packaging are those at the intersection of these two packages.
Let's consider another question for a second. Why should conda-forge package pure Python or R packages? Why not leave that to pip or poetry? I think this is exactly analogous to the Julia package situation. If I can just use the Julia package manager in an activate script, why not use pip in an activate script? That might work in another conda channel, but my understanding is that is not how conda-forge works. Maybe I'm wrong.

@ngam I bumped this to version v0.11.0 and build 1, anticipating that #47 will be merged before this one. Please update and resolve conflicts as needed.

To clear up any confusion, I am already pro this proposal. Having one package manager be able to resolve both Python and Julia (and other) dependencies for us sounds great. I'm just trying to figure out how it works in practice and if it is scalable to many packages.

We package a complete environment with a Project.toml and Manifest.toml, so that specifies the exact package versions to use. That complete environment is the default used by PySR and it is "installed" after conda finishes. As time goes on this package will be rebuilt with new Julia dependencies.

Is this literally the Manifest created when you build PySR? What happens when we use Conda to install another Julia package with its own Manifest? It's not possible to instantiate two Manifests, do you merge them somehow?
A few smaller questions:

As discussed, all Julia dependencies will ultimately need to be added, namely 100s or 1000s of packages. Are there plans/ambitions to automate this, otherwise it is a huge undertaking.
Currently you package all packages/artifacts in the depot. But if (in the future) all Julia dependencies are their own Conda packages, you only actually need whichever packages/artifacts are new in the depot.
Is there a reason you don't install the packages/artifacts into the existing depot? The Julia distro in conda-forge already creates its own depot in the Conda environment, and these packages/artifacts will have distinct paths, so this should work fine I believe.  Thinking about it, separate depots is probably necessary in the current setup just because there are a few common dependencies, so there would indeed be a few file clashes, but this problem would go away in a world where every Julia package is a Conda package.

Thinking out loud, just as a point of comparison, there is also my JuliaPkg Python package for managing Julia dependencies. This is the default way that JuliaCall manages its dependencies. It basically provides a way for a Python package to declare any Julia dependencies it needs, and then JuliaPkg will ensure those dependencies are met.
Some pros of JuliaPkg compared to the conda-forge proposal:

It's very simple, just a pure-python package, which doesn't require Conda.
It is based on Pkg, Julia's package manager, so should be very reliable and non-surprising.
There is no separate registry of packages required (whereas the conda-forge way requires registering all Julia packages again).

Some cons:

It IS possible to get into an inconsistent state using JuliaPkg, because two Python packages may declare incompatible Julia dependencies. But this will raise an error when dependencies are resolved, leaving it to the user to fix things (unlike say Pip which can just install an inconsistent set of packages) so at least it is not a silent error.

However I don't anticipate this will be a problem very often in practice because JuliaPkg/JuliaCall is mostly intended for writing wrapper packages, which will only normally have one dependency (the Julia package being wrapped).
It can also be solved by creating Python packages such as juliapkg-SymbolicRegression which just declare a JuliaPkg dependency on SymbolicRegression at a particular version. Then the Python package manager can ensure all the Julia packages are compatible too. This is a similar complexity of effort to creating a Conda package for every Julia package.

There is a post-install step (juliapkg.resolve()) to resolve/install Julia dependencies, whereas the conda-forge way has everything already installed.
If a Python package using JuliaPkg depends on some Julia package which itself depends on some Python package, then the Python->Python dependency is not visible to the Python package manager, so will be missed, which will cause some runtime error. Similarly for Julia->Python->Julia dependency chains. The Conda way does not have this issue.

Again, it can be solved by having Python packages for each Julia package, because then these can also declare their own Python dependencies.

So if we instead just used JuliaPkg everywhere, with the addition of juliapkg-X Python packages for every Julia package X to declare a dependency on X at a specific version, then we get something very similar to the current proposal. This has one key down-side, namely there is a post-install step (juliapkg.resolve()) required before severing the internet connection. But a large up-side is that this works in any Python package manager (Pip/Poetry/Python/etc).

@conda-forge-admin, please rerender

Is this literally the Manifest created when you build PySR? What happens when we use Conda to install another Julia package with its own Manifest? It's not possible to instantiate two Manifests, do you merge them somehow?

Yes, this is the manifest we used to build. We put it in a dedicated environment called @pysr-{version} which is the default environment that PySR will try to use. The other Python package also creates it's own environment. Thus we don't actually merge them on the user's machine.
The current plan is to reconcile the two environments by pinning packages while building the conda-forge packages such that the Julia packages are at the same version. At the moment, this is only requires pinning two packages.

As discussed, all Julia dependencies will ultimately need to be added, namely 100s or 1000s of packages. Are there plans/ambitions to automate this, otherwise it is a huge undertaking.

Ultimately, I think automation is the key. conda-forge seems to have a tradition of automating as much as possible with Github bots, so there is precedence. I'm just not sure if conda-forge is ready for me to stage 100 packages at once. That's why I'm taking a more evolutionary approach where we start with a small number of packages with larger bundles and then focus on the intersection of the current packages. PyCall.jl and PythonCall.jl would definitely be one of the first Julia packages I think.

Is there a reason you don't install the packages/artifacts into the existing depot? The Julia distro in conda-forge already creates its own depot in the Conda environment, and these packages/artifacts will have distinct paths, so this should work fine I believe.  Thinking about it, separate depots is probably necessary in the current setup just because there are a few common dependencies, so there would indeed be a few file clashes, but this problem would go away in a world where every Julia package is a Conda package.

The existing depot, which we moved from ~/.julia to $CONDA_PREFIX/share/julia is still under the user's control. Given the lack of Julia packages in conda-forge, I would expect the user to have installed some Julia packages directly rather than through conda. Rather than deal with a potential collision of files, Julia's depot stacking allows us to bypass this issue entirely. Separation of "user" and "system" installed packages is the intended use case of depot stacking. In this case, conda-forge is playing the role of the "system".
Earlier I had proposed moving towards a common conda-forge managed depot, but this requires us to eliminate the overlap in packages. In principle this is possible, but it will require additional conda packages to implement.

Thinking out loud, just as a point of comparison, there is also my JuliaPkg Python package for managing Julia dependencies. This is the default way that JuliaCall manages its dependencies. It basically provides a way for a Python package to declare any Julia dependencies it needs, and then JuliaPkg will ensure those dependencies are met.

This is a nice approach. Conda-forge could use JuliaPkg at build time to deliver the initial package state. Thus, JuliaPkg is complementary to the approach taken here.

Thanks this is all really helpful.

The current plan is to reconcile the two environments by pinning packages while building the conda-forge packages such that the Julia packages are at the same version. At the moment, this is only requires pinning two packages.

OK so you install one environment per Conda package, giving you a consistent set of Julia packages. Is there a mechanism to merge these into a single environment, so that it's possible to access a bunch of unrelated Julia packages in a single session? I think maybe that's what you meant in the quote above but I'm not sure. How does it work?

OK so you install one environment per Conda package, giving you a consistent set of Julia packages. Is there a mechanism to merge these into a single environment, so that it's possible to access a bunch of unrelated Julia packages in a single session? I think maybe that's what you meant in the quote above but I'm not sure. How does it work?

One could use environment stacking:
insert!(LOAD_PATH, 2<EMAIL_ADDRESS>insert!(LOAD_PATH, 2<EMAIL_ADDRESS>
See https://docs.julialang.org/en/v1/manual/code-loading/#Environments
That still requires us to ensure consistency of the dependencies. In the short term, this means pinning packages in the depots. In the long term, we split out the mutual dependencies into their own packages

Is this PR good to go?
I tried to unpin Julia in #50 but having some issues (which I assume are solved in this PR) - so would be great if I could merge this.
Cheers,
Miles

My current plan is to attend the conda-forge core meeting next week to discuss this.
That said, I don't think the world will end if you merged this.  The worse case scenario is that we'll have to back out the changes later. Let me take a look at #50

Thanks, sounds good to me! I am eager to hear the results of the discussion.
Since it seems like the current package (even before recent versions, possibly) is failing on macOS, and this one seems to fix those issues, I will go ahead and merge now. I think it will also allow for us to study the stability of the build strategy which you could discuss at the meeting, too. Happy to back out the changes too, if a different strategy is decided on!

Just tested it locally - works perfectly! Super cool to not have to do any extra install steps.
Since this fixes the Manifest during build, it would be interesting in the future to think about packaging sysimages created by PackageCompiler.jl for reduced latency.

Nice! I'd be grateful if you could keep me in the loop as this stuff develops.

Since this fixes the Manifest during build, it would be interesting in the future to think about packaging sysimages created by PackageCompiler.jl for reduced latency.

Shipping a system image is interesting although a fully optimized approach would still involve building the system image on the user's machine. I've noticed that custom system images often can take advantage of advanced features like AVX-512 if built on the user's machine.

Nice! I'd be grateful (@mkitti, @ngam) if you could keep me in the loop as this stuff develops.

We should take a look at replacing pyjulia with juliacall upstream. It would be nice if we were less dependent on environment variables for communication. For example, it would be nice to set the initial Julia environment via the JLOptions struct rather than JULIA_DEPOT_PATH environment variable.
The next feedstock that we may visit will be xbitinfo / BitInformation.jl. I would like to have the conversation with conda-forge core first though before proceeding further.
https://github.com/conda-forge/xbitinfo-feedstock