Error: Error building ADCME:

┌ Error: Error building ADCME:
│ ┌ Warning: Pkg.installed() is deprecated
│ └ @ Pkg C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Pkg.jl:554
│ ┌ Warning: Pkg.installed() is deprecated
│ └ @ Pkg C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Pkg.jl:554
│ [ Info: Your Julia version is 1.5.3, current ADCME version is 0.5.13, ADCME dependencies installation path: C:\Users\Farzad.julia\adcme
│ [ Info: --------------- (1/6) Install Tensorflow Dependencies ---------------
│ [ Info: ADCME dependencies have already been installed
│ [ Info: --------------- (2/6) Check Python Version ---------------
│ ┌ Warning: Pkg.installed() is deprecated
│ └ @ Pkg C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Pkg.jl:554
│ Building Conda ─→ C:\Users\Farzad.julia\packages\Conda\x5ml4\deps\build.log
│ Building PyCall → C:\Users\Farzad.julia\packages\PyCall\BcTLp\deps\build.log
│ ┌ Info: PyCall Python version: C:\Users\Farzad.julia\adcme\python.exe
│ └ Conda Python version: C:\Users\Farzad.julia\adcme\python.exe
│ [ Info: --------------- (3/6) Looking for TensorFlow Dynamic Libraries ---------------
│ ERROR: LoadError: PyError (PyImport_ImportModule
│
│ The Python package tensorflow could not be imported by pyimport. Usually this means
│ that you did not install tensorflow in the Python version being used by PyCall.
│
│ PyCall is currently configured to use the Python version at:
│
│ C:\Users\Farzad.julia\adcme\python.exe
│
│ and you should use whatever mechanism you usually use (apt-get, pip, conda,
│ etcetera) to install the Python package containing the tensorflow module.
│
│ One alternative is to re-configure PyCall to use a different Python
│ version on your system: set ENV["PYTHON"] to the path/name of the python
│ executable you want to use, run Pkg.build("PyCall"), and re-launch Julia.
│
│ Another alternative is to configure PyCall to use a Julia-specific Python
│ distribution via the Conda.jl package (which installs a private Anaconda
│ Python distribution), which has the advantage that packages can be installed
│ and kept up-to-date via Julia. As explained in the PyCall documentation,
│ set ENV["PYTHON"]="", run Pkg.build("PyCall"), and re-launch Julia. Then,
│ To install the tensorflow module, you can use pyimport_conda("tensorflow", PKG),
│ where PKG is the Anaconda package the contains the module tensorflow,
│ or alternatively you can use the Conda package directly (via
│ using Conda followed by Conda.add etcetera).
│
│ ) <class 'ImportError'>
│ ImportError('\n\nIMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!\n\nImporting the numpy C-extensions failed. This error can happen for\nmany reasons, often due to issues with your setup or how NumPy was\ninstalled.\n\nWe have compiled some common reasons and troubleshooting tips at:\n\n https://numpy.org/devdocs/user/troubleshooting-importerror.html\n\nPlease note and check the following:\n\n * The Python version is: Python3.7 from "C:\Users\Farzad\AppData\Local\JuliaPro-1.5.3-1\Julia-1.5.3\bin\julia.exe"\n * The NumPy version is: "1.19.1"\n\nand make sure that they are the versions you expect.\nPlease carefully study the documentation linked above for further help.\n\nOriginal error was: DLL load failed: The specified module could not be found.\n')
│ File "C:\Users\Farzad.julia\adcme\lib\site-packages\tensorflow_init_.py", line 99, in
│ from tensorflow_core import *
│ File "C:\Users\Farzad.julia\adcme\lib\site-packages\tensorflow_core_init_.py", line 34, in
│ from tensorflow.python import pywrap_tensorflow # pylint: disable=unused-import
│ File "C:\Users\Farzad.julia\adcme\lib\site-packages\tensorflow_init_.py", line 50, in getattr
│ module = self.load()
│ File "C:\Users\Farzad.julia\adcme\lib\site-packages\tensorflow_init.py", line 44, in load
│ module = importlib.import_module(self.name)
│ File "C:\Users\Farzad.julia\adcme\lib\importlib_init.py", line 127, in import_module
│ return bootstrap.gcd_import(name[level:], package, level)
│ File "C:\Users\Farzad.julia\adcme\lib\site-packages\tensorflow_core\python_init.py", line 47, in
│ import numpy as np
│ File "C:\Users\Farzad.julia\adcme\lib\site-packages\numpy_init.py", line 140, in
│ from . import core
│ File "C:\Users\Farzad.julia\adcme\lib\site-packages\numpy\core_init.py", line 48, in
│ raise ImportError(msg)
│
│ Stacktrace:
│ [1] pyimport(::String) at C:\Users\Farzad.julia\packages\PyCall\BcTLp\src\PyCall.jl:547
│ [2] top-level scope at C:\Users\Farzad.julia\packages\ADCME\JzSON\deps\build.jl:77
│ [3] include(::String) at .\client.jl:457
│ [4] top-level scope at none:5
│ in expression starting at C:\Users\Farzad.julia\packages\ADCME\JzSON\deps\build.jl:77
└ @ Pkg.Operations C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Operations.jl:949

@kailaix commented yesterday
Hello, you had an old installation "current ADCME version is 0.5.13". Could you try the following
Remove the directory C:\Users\Farzad.julia\adcme
Reinstall and rebuild ADCME.
using ADCME
Pkg.build("ADCME")
Make sure you have the latest version (0.6.7) (you can use Pkg.status("ADCME") to check.

@kailaix commented yesterday
Hello, you had an old installation "current ADCME version is 0.5.13". Could you try the following
Remove the directory C:\Users\Farzad.julia\adcme
Reinstall and rebuild ADCME.
using ADCME
Pkg.build("ADCME")
Make sure you have the latest version (0.6.7) (you can use Pkg.status("ADCME") to check.

farzadforouzanfard87 commented 22 hours ago
Hi kailaix, I did all of your commands, but I get the below error ☹
⛔
(@JuliaPro_v1.5.3-1) pkg> add ADCME
Updating registry at C:\Users\Farzad.julia\registries\General
Updating registry at C:\Users\Farzad.julia\registries\JuliaComputingRegistry
Resolving package versions...
Installed ADCME ─ v0.6.7
Updating C:\Users\Farzad.julia\environments\JuliaPro_v1.5.3-1\Project.toml
[07b341a0] + ADCME v0.6.7
Updating C:\Users\Farzad.julia\environments\JuliaPro_v1.5.3-1\Manifest.toml
[07b341a0] + ADCME v0.6.7
[9e28174c] + BinDeps v1.0.2
[a74b3585] + Blosc v0.7.0
[0b7ba130] + Blosc_jll v1.14.3+1
[e1450e63] + BufferedStreams v1.0.0
[631607c0] + CMake v1.2.0
[f67ccb44] + HDF5 v0.13.7
[0234f1f7] + HDF5_jll v1.10.5+7
[5ced341a] + Lz4_jll v1.9.2+2
[23992714] + MAT v0.8.1
Building ADCME → C:\Users\Farzad.julia\packages\ADCME\hCmEo\deps\build.log
┌ Error: Error building ADCME, showing the last 100 of log:
wrapt-1.12.1 | 45 KB | ########## | 100%
libblas-3.8.0 | 3.5 MB | ########## | 100%
opt_einsum-3.3.0 | 51 KB | ########## | 100%
ca-certificates-2020 | 184 KB | ########## | 100%
google-auth-1.20.1 | 55 KB | ########## | 100%
keras-applications-1 | 29 KB | ########## | 100%
click-7.1.2 | 64 KB | ########## | 100%
cachetools-4.1.1 | 12 KB | ########## | 100%
tensorflow-probabili | 1.2 MB | ########## | 100%
mkl-service-2.3.0 | 52 KB | ########## | 100%
tensorboard-2.3.0 | 6.2 MB | ########## | 100%
│ Preparing transaction: ...working... done
│ Verifying transaction: ...working... done
│ Executing transaction: ...working... done
│
│
│ ==> WARNING: A newer version of conda exists. <==
│ current version: 4.8.3
│ latest version: 4.9.2
│
│ Please update conda by running
│
│ $ conda update -n base -c defaults conda
│
│
│ #
│ # To activate this environment, use
│ #
│ # $ conda activate base
│ #
│ # To deactivate an active environment, use
│ #
│ # $ conda deactivate
│
│ ┌ Warning: Pkg.installed() is deprecated
│ └ @ Pkg C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Pkg.jl:554
│ ┌ Warning: Pkg.installed() is deprecated
│ └ @ Pkg C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Pkg.jl:554
│ [ Info: Your Julia version is 1.5.3, current ADCME version is 0.6.7, ADCME dependencies installation path: C:\Users\Farzad.julia\adcme
│ [ Info: --------------- (1/6) Install Tensorflow Dependencies ---------------
│ [ Info: Installing miniconda...
│ [ Info: Installing conda dependencies...
│ [ Info: --------------- (2/6) Check Python Version ---------------
│ ┌ Warning: Pkg.installed() is deprecated
│ └ @ Pkg C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Pkg.jl:554
│ Building Conda ─→ C:\Users\Farzad.julia\packages\Conda\x5ml4\deps\build.log
│ Building PyCall → C:\Users\Farzad.julia\packages\PyCall\BcTLp\deps\build.log
│ ┌ Info: PyCall Python version: C:\Users\Farzad.julia\adcme\python.exe
│ └ Conda Python version: C:\Users\Farzad.julia\adcme\python.exe
│ [ Info: --------------- (3/6) Looking for TensorFlow Dynamic Libraries ---------------
│ ERROR: LoadError: PyError (PyImport_ImportModule
│
│ The Python package tensorflow could not be imported by pyimport. Usually this means
│ that you did not install tensorflow in the Python version being used by PyCall.
│
│ PyCall is currently configured to use the Python version at:
│
│ C:\Users\Farzad.julia\adcme\python.exe
│
│ and you should use whatever mechanism you usually use (apt-get, pip, conda,
│ etcetera) to install the Python package containing the tensorflow module.
│
│ One alternative is to re-configure PyCall to use a different Python
│ version on your system: set ENV["PYTHON"] to the path/name of the python
│ executable you want to use, run Pkg.build("PyCall"), and re-launch Julia.
│
│ Another alternative is to configure PyCall to use a Julia-specific Python
│ distribution via the Conda.jl package (which installs a private Anaconda
│ Python distribution), which has the advantage that packages can be installed
│ and kept up-to-date via Julia. As explained in the PyCall documentation,
│ set ENV["PYTHON"]="", run Pkg.build("PyCall"), and re-launch Julia. Then,
│ To install the tensorflow module, you can use pyimport_conda("tensorflow", PKG),
│ where PKG is the Anaconda package the contains the module tensorflow,
│ or alternatively you can use the Conda package directly (via
│ using Conda followed by Conda.add etcetera).
│
│ ) <class 'ImportError'>
│ ImportError('\n\nIMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!\n\nImporting the numpy C-extensions failed. This error can happen for\nmany reasons, often due to issues with your setup or how NumPy was\ninstalled.\n\nWe have compiled some common reasons and troubleshooting tips at:\n\n https://numpy.org/devdocs/user/troubleshooting-importerror.html\n\nPlease note and check the following:\n\n * The Python version is: Python3.7 from "C:\Users\Farzad\AppData\Local\JuliaPro-1.5.3-1\Julia-1.5.3\bin\julia.exe"\n * The NumPy version is: "1.19.1"\n\nand make sure that they are the versions you expect.\nPlease carefully study the documentation linked above for further help.\n\nOriginal error was: DLL load failed: The specified module could not be found.\n')
│ File "C:\Users\Farzad.julia\adcme\lib\site-packages\tensorflow_init_.py", line 99, in
│ from tensorflow_core import *
│ File "C:\Users\Farzad.julia\adcme\lib\site-packages\tensorflow_core_init_.py", line 34, in
│ from tensorflow.python import pywrap_tensorflow # pylint: disable=unused-import
│ File "C:\Users\Farzad.julia\adcme\lib\site-packages\tensorflow_init_.py", line 50, in getattr
│ module = self.load()
│ File "C:\Users\Farzad.julia\adcme\lib\site-packages\tensorflow_init.py", line 44, in load
│ module = importlib.import_module(self.name)
│ File "C:\Users\Farzad.julia\adcme\lib\importlib_init.py", line 127, in import_module
│ return bootstrap.gcd_import(name[level:], package, level)
│ File "C:\Users\Farzad.julia\adcme\lib\site-packages\tensorflow_core\python_init.py", line 47, in
│ import numpy as np
│ File "C:\Users\Farzad.julia\adcme\lib\site-packages\numpy_init.py", line 140, in
│ from . import core
│ File "C:\Users\Farzad.julia\adcme\lib\site-packages\numpy\core_init.py", line 48, in
│ raise ImportError(msg)
│
│ Stacktrace:
│ [1] pyimport(::String) at C:\Users\Farzad.julia\packages\PyCall\BcTLp\src\PyCall.jl:547
│ [2] top-level scope at C:\Users\Farzad.julia\packages\ADCME\hCmEo\deps\build.jl:77
│ [3] include(::String) at .\client.jl:457
│ [4] top-level scope at none:5
│ in expression starting at C:\Users\Farzad.julia\packages\ADCME\hCmEo\deps\build.jl:77
│
│ Full log at C:\Users\Farzad.julia\packages\ADCME\hCmEo\deps\build.log
└ @ Pkg.Operations C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Operations.jl:949

farzadforouzanfard87 commented 22 hours ago
Hi kailaix, I did all of your commands, but I get the below error ☹
⛔
(@JuliaPro_v1.5.3-1) pkg> add ADCME
Updating registry at C:\Users\Farzad.julia\registries\General
Updating registry at C:\Users\Farzad.julia\registries\JuliaComputingRegistry
Resolving package versions...
Installed ADCME ─ v0.6.7
Updating C:\Users\Farzad.julia\environments\JuliaPro_v1.5.3-1\Project.toml
[07b341a0] + ADCME v0.6.7
Updating C:\Users\Farzad.julia\environments\JuliaPro_v1.5.3-1\Manifest.toml
[07b341a0] + ADCME v0.6.7
[9e28174c] + BinDeps v1.0.2
[a74b3585] + Blosc v0.7.0
[0b7ba130] + Blosc_jll v1.14.3+1
[e1450e63] + BufferedStreams v1.0.0
[631607c0] + CMake v1.2.0
[f67ccb44] + HDF5 v0.13.7
[0234f1f7] + HDF5_jll v1.10.5+7
[5ced341a] + Lz4_jll v1.9.2+2
[23992714] + MAT v0.8.1
Building ADCME → C:\Users\Farzad.julia\packages\ADCME\hCmEo\deps\build.log
┌ Error: Error building ADCME, showing the last 100 of log:
wrapt-1.12.1 | 45 KB | ########## | 100%
libblas-3.8.0 | 3.5 MB | ########## | 100%
opt_einsum-3.3.0 | 51 KB | ########## | 100%
ca-certificates-2020 | 184 KB | ########## | 100%
google-auth-1.20.1 | 55 KB | ########## | 100%
keras-applications-1 | 29 KB | ########## | 100%
click-7.1.2 | 64 KB | ########## | 100%
cachetools-4.1.1 | 12 KB | ########## | 100%
tensorflow-probabili | 1.2 MB | ########## | 100%
mkl-service-2.3.0 | 52 KB | ########## | 100%
tensorboard-2.3.0 | 6.2 MB | ########## | 100%
│ Preparing transaction: ...working... done
│ Verifying transaction: ...working... done
│ Executing transaction: ...working... done
│
│
│ ==> WARNING: A newer version of conda exists. <==
│ current version: 4.8.3
│ latest version: 4.9.2
│
│ Please update conda by running
│
│ $ conda update -n base -c defaults conda
│
│
│ #
│ # To activate this environment, use
│ #
│ # $ conda activate base
│ #
│ # To deactivate an active environment, use
│ #
│ # $ conda deactivate
│
│ ┌ Warning: Pkg.installed() is deprecated
│ └ @ Pkg C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Pkg.jl:554
│ ┌ Warning: Pkg.installed() is deprecated
│ └ @ Pkg C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Pkg.jl:554
│ [ Info: Your Julia version is 1.5.3, current ADCME version is 0.6.7, ADCME dependencies installation path: C:\Users\Farzad.julia\adcme
│ [ Info: --------------- (1/6) Install Tensorflow Dependencies ---------------
│ [ Info: Installing miniconda...
│ [ Info: Installing conda dependencies...
│ [ Info: --------------- (2/6) Check Python Version ---------------
│ ┌ Warning: Pkg.installed() is deprecated
│ └ @ Pkg C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Pkg.jl:554
│ Building Conda ─→ C:\Users\Farzad.julia\packages\Conda\x5ml4\deps\build.log
│ Building PyCall → C:\Users\Farzad.julia\packages\PyCall\BcTLp\deps\build.log
│ ┌ Info: PyCall Python version: C:\Users\Farzad.julia\adcme\python.exe
│ └ Conda Python version: C:\Users\Farzad.julia\adcme\python.exe
│ [ Info: --------------- (3/6) Looking for TensorFlow Dynamic Libraries ---------------
│ ERROR: LoadError: PyError (PyImport_ImportModule
│
│ The Python package tensorflow could not be imported by pyimport. Usually this means
│ that you did not install tensorflow in the Python version being used by PyCall.
│
│ PyCall is currently configured to use the Python version at:
│
│ C:\Users\Farzad.julia\adcme\python.exe
│
│ and you should use whatever mechanism you usually use (apt-get, pip, conda,
│ etcetera) to install the Python package containing the tensorflow module.
│
│ One alternative is to re-configure PyCall to use a different Python
│ version on your system: set ENV["PYTHON"] to the path/name of the python
│ executable you want to use, run Pkg.build("PyCall"), and re-launch Julia.
│
│ Another alternative is to configure PyCall to use a Julia-specific Python
│ distribution via the Conda.jl package (which installs a private Anaconda
│ Python distribution), which has the advantage that packages can be installed
│ and kept up-to-date via Julia. As explained in the PyCall documentation,
│ set ENV["PYTHON"]="", run Pkg.build("PyCall"), and re-launch Julia. Then,
│ To install the tensorflow module, you can use pyimport_conda("tensorflow", PKG),
│ where PKG is the Anaconda package the contains the module tensorflow,
│ or alternatively you can use the Conda package directly (via
│ using Conda followed by Conda.add etcetera).
│
│ ) <class 'ImportError'>
│ ImportError('\n\nIMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!\n\nImporting the numpy C-extensions failed. This error can happen for\nmany reasons, often due to issues with your setup or how NumPy was\ninstalled.\n\nWe have compiled some common reasons and troubleshooting tips at:\n\n https://numpy.org/devdocs/user/troubleshooting-importerror.html\n\nPlease note and check the following:\n\n * The Python version is: Python3.7 from "C:\Users\Farzad\AppData\Local\JuliaPro-1.5.3-1\Julia-1.5.3\bin\julia.exe"\n * The NumPy version is: "1.19.1"\n\nand make sure that they are the versions you expect.\nPlease carefully study the documentation linked above for further help.\n\nOriginal error was: DLL load failed: The specified module could not be found.\n')
│ File "C:\Users\Farzad.julia\adcme\lib\site-packages\tensorflow_init_.py", line 99, in
│ from tensorflow_core import *
│ File "C:\Users\Farzad.julia\adcme\lib\site-packages\tensorflow_core_init_.py", line 34, in
│ from tensorflow.python import pywrap_tensorflow # pylint: disable=unused-import
│ File "C:\Users\Farzad.julia\adcme\lib\site-packages\tensorflow_init_.py", line 50, in getattr
│ module = self.load()
│ File "C:\Users\Farzad.julia\adcme\lib\site-packages\tensorflow_init.py", line 44, in load
│ module = importlib.import_module(self.name)
│ File "C:\Users\Farzad.julia\adcme\lib\importlib_init.py", line 127, in import_module
│ return bootstrap.gcd_import(name[level:], package, level)
│ File "C:\Users\Farzad.julia\adcme\lib\site-packages\tensorflow_core\python_init.py", line 47, in
│ import numpy as np
│ File "C:\Users\Farzad.julia\adcme\lib\site-packages\numpy_init.py", line 140, in
│ from . import core
│ File "C:\Users\Farzad.julia\adcme\lib\site-packages\numpy\core_init.py", line 48, in
│ raise ImportError(msg)
│
│ Stacktrace:
│ [1] pyimport(::String) at C:\Users\Farzad.julia\packages\PyCall\BcTLp\src\PyCall.jl:547
│ [2] top-level scope at C:\Users\Farzad.julia\packages\ADCME\hCmEo\deps\build.jl:77
│ [3] include(::String) at .\client.jl:457
│ [4] top-level scope at none:5
│ in expression starting at C:\Users\Farzad.julia\packages\ADCME\hCmEo\deps\build.jl:77
│
│ Full log at C:\Users\Farzad.julia\packages\ADCME\hCmEo\deps\build.log
└ @ Pkg.Operations C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Operations.jl:949

@kailaix commented 15 hours ago •
What is the output of
$ C:\Users\Farzad.julia\adcme\python.exe
import tensorflow

@kailaix commented 15 hours ago •
What is the output of
$ C:\Users\Farzad.julia\adcme\python.exe
import tensorflow

farzadforouzanfard87 commented 14 hours ago
Dear kailai, here is the output of your above command:
import tensorflow
Traceback (most recent call last):
File "C:\Users\Farzad.julia\adcme\lib\site-packages\numpy\core_init_.py", line 22, in
from . import multiarray
File "C:\Users\Farzad.julia\adcme\lib\site-packages\numpy\core\multiarray.py", line 12, in
from . import overrides
File "C:\Users\Farzad.julia\adcme\lib\site-packages\numpy\core\overrides.py", line 7, in
from numpy.core._multiarray_umath import (
ImportError: DLL load failed: The specified module could not be found.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\Farzad.julia\adcme\lib\site-packages\tensorflow_init_.py", line 99, in
from tensorflow_core import *
File "C:\Users\Farzad.julia\adcme\lib\site-packages\tensorflow_core_init_.py", line 34, in
from tensorflow.python import pywrap_tensorflow # pylint: disable=unused-import
File "C:\Users\Farzad.julia\adcme\lib\site-packages\tensorflow_init_.py", line 50, in getattr
module = self.load()
File "C:\Users\Farzad.julia\adcme\lib\site-packages\tensorflow_init.py", line 44, in load
module = importlib.import_module(self.name)
File "C:\Users\Farzad.julia\adcme\lib\importlib_init.py", line 127, in import_module
return bootstrap.gcd_import(name[level:], package, level)
File "C:\Users\Farzad.julia\adcme\lib\site-packages\tensorflow_core\python_init.py", line 47, in
import numpy as np
File "C:\Users\Farzad.julia\adcme\lib\site-packages\numpy_init.py", line 140, in
from . import core
File "C:\Users\Farzad.julia\adcme\lib\site-packages\numpy\core_init.py", line 48, in
raise ImportError(msg)
ImportError:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.
We have compiled some common reasons and troubleshooting tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html
Please note and check the following:
The Python version is: Python3.7 from "C:\Users\Farzad.julia\adcme\python.exe"
The NumPy version is: "1.19.1"
and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.
Original error was: DLL load failed: The specified module could not be found.

farzadforouzanfard87 commented 14 hours ago
Dear kailai, here is the output of your above command:
import tensorflow
Traceback (most recent call last):
File "C:\Users\Farzad.julia\adcme\lib\site-packages\numpy\core_init_.py", line 22, in
from . import multiarray
File "C:\Users\Farzad.julia\adcme\lib\site-packages\numpy\core\multiarray.py", line 12, in
from . import overrides
File "C:\Users\Farzad.julia\adcme\lib\site-packages\numpy\core\overrides.py", line 7, in
from numpy.core._multiarray_umath import (
ImportError: DLL load failed: The specified module could not be found.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\Farzad.julia\adcme\lib\site-packages\tensorflow_init_.py", line 99, in
from tensorflow_core import *
File "C:\Users\Farzad.julia\adcme\lib\site-packages\tensorflow_core_init_.py", line 34, in
from tensorflow.python import pywrap_tensorflow # pylint: disable=unused-import
File "C:\Users\Farzad.julia\adcme\lib\site-packages\tensorflow_init_.py", line 50, in getattr
module = self.load()
File "C:\Users\Farzad.julia\adcme\lib\site-packages\tensorflow_init.py", line 44, in load
module = importlib.import_module(self.name)
File "C:\Users\Farzad.julia\adcme\lib\importlib_init.py", line 127, in import_module
return bootstrap.gcd_import(name[level:], package, level)
File "C:\Users\Farzad.julia\adcme\lib\site-packages\tensorflow_core\python_init.py", line 47, in
import numpy as np
File "C:\Users\Farzad.julia\adcme\lib\site-packages\numpy_init.py", line 140, in
from . import core
File "C:\Users\Farzad.julia\adcme\lib\site-packages\numpy\core_init.py", line 48, in
raise ImportError(msg)
ImportError:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.
We have compiled some common reasons and troubleshooting tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html
Please note and check the following:
The Python version is: Python3.7 from "C:\Users\Farzad.julia\adcme\python.exe"
The NumPy version is: "1.19.1"
and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.
Original error was: DLL load failed: The specified module could not be found.

@kailaix commented 11 hours ago
It looks like your NumPy DLL was not loaded.
Could you try
$ C:\Users\Farzad.julia\adcme\python.exe
import numpy

@kailaix commented 11 hours ago
It looks like your NumPy DLL was not loaded.
Could you try
$ C:\Users\Farzad.julia\adcme\python.exe
import numpy

farzadforouzanfard87 commented 10 hours ago
import numpy
Traceback (most recent call last):
File "C:\Users\Farzad.julia\adcme\lib\site-packages\numpy\core_init_.py", line 22, in
from . import multiarray
File "C:\Users\Farzad.julia\adcme\lib\site-packages\numpy\core\multiarray.py", line 12, in
from . import overrides
File "C:\Users\Farzad.julia\adcme\lib\site-packages\numpy\core\overrides.py", line 7, in
from numpy.core._multiarray_umath import (
ImportError: DLL load failed: The specified module could not be found.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\Farzad.julia\adcme\lib\site-packages\numpy_init_.py", line 140, in
from . import core
File "C:\Users\Farzad.julia\adcme\lib\site-packages\numpy\core_init_.py", line 48, in
raise ImportError(msg)
ImportError:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.
We have compiled some common reasons and troubleshooting tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html
Please note and check the following:
The Python version is: Python3.7 from "C:\Users\Farzad.julia\adcme\python.exe"
The NumPy version is: "1.19.1"
and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.
Original error was: DLL load failed: The specified module could not be found.

farzadforouzanfard87 commented 10 hours ago
import numpy
Traceback (most recent call last):
File "C:\Users\Farzad.julia\adcme\lib\site-packages\numpy\core_init_.py", line 22, in
from . import multiarray
File "C:\Users\Farzad.julia\adcme\lib\site-packages\numpy\core\multiarray.py", line 12, in
from . import overrides
File "C:\Users\Farzad.julia\adcme\lib\site-packages\numpy\core\overrides.py", line 7, in
from numpy.core._multiarray_umath import (
ImportError: DLL load failed: The specified module could not be found.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\Farzad.julia\adcme\lib\site-packages\numpy_init_.py", line 140, in
from . import core
File "C:\Users\Farzad.julia\adcme\lib\site-packages\numpy\core_init_.py", line 48, in
raise ImportError(msg)
ImportError:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.
We have compiled some common reasons and troubleshooting tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html
Please note and check the following:
The Python version is: Python3.7 from "C:\Users\Farzad.julia\adcme\python.exe"
The NumPy version is: "1.19.1"
and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.
Original error was: DLL load failed: The specified module could not be found.

@kailaix commented 7 hours ago
I think the problem is with your numpy installation. Could you try reinstall numpy using
C:\Users\Farzad.julia\adcme\Scripts\pip.exe install -U numpy
and retry import numpy?

@kailaix commented 7 hours ago
I think the problem is with your numpy installation. Could you try reinstall numpy using
C:\Users\Farzad.julia\adcme\Scripts\pip.exe install -U numpy
and retry import numpy?

farzadforouzanfard87 commented 31 minutes ago
I think the problem is with your numpy installation. Could you try reinstall numpy using
C:\Users\Farzad.julia\adcme\Scripts\pip.exe install -U numpy
and retry import numpy?
julia@DESKTOP-0RUTMSV:~$ /mnt/c/Users/Farzad/.julia/adcme/Scripts/pip.exe install --trusted-host pypi.org --trusted-host files.pythonhosted.org numpy
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Requirement already satisfied: numpy in c:\users\farzad.julia\adcme\lib\site-packages (1.19.1)

farzadforouzanfard87 commented 31 minutes ago
I think the problem is with your numpy installation. Could you try reinstall numpy using
C:\Users\Farzad.julia\adcme\Scripts\pip.exe install -U numpy
and retry import numpy?
julia@DESKTOP-0RUTMSV:~$ /mnt/c/Users/Farzad/.julia/adcme/Scripts/pip.exe install --trusted-host pypi.org --trusted-host files.pythonhosted.org numpy
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Requirement already satisfied: numpy in c:\users\farzad.julia\adcme\lib\site-packages (1.19.1)

@jiaqiwang969
Hi, @kailaix , the Same problem I have met, and as you suggested, with "install -U numpy":
@farzadforouzanfard87
C:\Users\wjq>C:\Users\wjq.julia\adcme\Scripts\pip.exe install -U numpy
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/numpy/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/numpy/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/numpy/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/numpy/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/numpy/
Could not fetch URL https://pypi.org/simple/numpy/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/numpy/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
Requirement already up-to-date: numpy in c:\users\wjq.julia\adcme\lib\site-packages (1.19.1)
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴
hi @jiaqiwang969 , i solve this issue by using the below command :
pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org <package_name>
for example in my Windows Subsystem for Linux :
julia@DESKTOP-0RUTMSV:~$ /mnt/c/Users/Farzad/.julia/adcme/Scripts/pip.exe install --trusted-host pypi.org --trusted-host files.pythonhosted.org numpy
🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴
but dear @kailaix , after reinstall numy , in JuliaPro , Pkg.build("ADCME") not work and the same error occurs as I mentioned in my first post ☹☹☹☹

@jiaqiwang969
Hi, @kailaix , the Same problem I have met, and as you suggested, with "install -U numpy":
@farzadforouzanfard87
C:\Users\wjq>C:\Users\wjq.julia\adcme\Scripts\pip.exe install -U numpy
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/numpy/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/numpy/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/numpy/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/numpy/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/numpy/
Could not fetch URL https://pypi.org/simple/numpy/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/numpy/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
Requirement already up-to-date: numpy in c:\users\wjq.julia\adcme\lib\site-packages (1.19.1)
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴
hi @jiaqiwang969 , i solve this issue by using the below command :
pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org <package_name>
for example in my Windows Subsystem for Linux :
julia@DESKTOP-0RUTMSV:~$ /mnt/c/Users/Farzad/.julia/adcme/Scripts/pip.exe install --trusted-host pypi.org --trusted-host files.pythonhosted.org numpy
🔴🔴🔴🔴🔴🔴🔴🔴🔴🔴
but dear @kailaix , after reinstall numy , in JuliaPro , Pkg.build("ADCME") not work and the same error occurs as I mentioned in my first post ☹☹☹☹

Hi, I guess the reason is that import numpy is trying to use DLLs from another numpy installation. Following the instruction here, I suggest you activate your ADCME conda environment and try importing numpy (the same for Linux/OSX)
$ C:\Users\Farzad\.julia\adcme\Scripts\conda.bat activate
(base) $ python
>>> import numpy

Hi, I guess the reason is that import numpy is trying to use DLLs from another numpy installation. Following the instruction here, I suggest you activate your ADCME conda environment and try importing numpy (the same for Linux/OSX)
$ C:\Users\Farzad\.julia\adcme\Scripts\conda.bat activate
(base) $ python
>>> import numpy

Hi, I guess the reason is that import numpy is trying to use DLLs from another numpy installation. Following the instruction here, I suggest you activate your ADCME conda environment and try importing numpy (the same for Linux/OSX)
$ C:\Users\Farzad\.julia\adcme\Scripts\conda.bat activate
(base) $ python
>>> import numpy

Dear @kailaix , i solve this problem by doing two tasks:
1-in visual studio 2017: install "Desktop development with C++"
2-set an extra set of PATH environment variables:
C:\Users\Farzad.julia\adcme\Library\bin
C:\Users\Farzad.julia\adcme
C:\Users\Farzad\AppData\Local\atom\bin
and then run Pkg.build("ADCME") and all of the things installed appropriately and then run test ADCME and test passed.
Thanks for your commands, ADCME installed on my JuliaPro successfully.
and now I have a new issue with installing AdFem.jl package that I post in AdFem.jl issue page.

Hi, I guess the reason is that import numpy is trying to use DLLs from another numpy installation. Following the instruction here, I suggest you activate your ADCME conda environment and try importing numpy (the same for Linux/OSX)
$ C:\Users\Farzad\.julia\adcme\Scripts\conda.bat activate
(base) $ python
>>> import numpy

Dear @kailaix , i solve this problem by doing two tasks:
1-in visual studio 2017: install "Desktop development with C++"
2-set an extra set of PATH environment variables:
C:\Users\Farzad.julia\adcme\Library\bin
C:\Users\Farzad.julia\adcme
C:\Users\Farzad\AppData\Local\atom\bin
and then run Pkg.build("ADCME") and all of the things installed appropriately and then run test ADCME and test passed.
Thanks for your commands, ADCME installed on my JuliaPro successfully.
and now I have a new issue with installing AdFem.jl package that I post in AdFem.jl issue page.

Hi, I guess the reason is that import numpy is trying to use DLLs from another numpy installation. Following the instruction here, I suggest you activate your ADCME conda environment and try importing numpy (the same for Linux/OSX)
$ C:\Users\Farzad\.julia\adcme\Scripts\conda.bat activate
(base) $ python
>>> import numpy

Dear @kailaix , i solve this problem by doing two tasks:
1-in visual studio 2017: install "Desktop development with C++"
2-set an extra set of PATH environment variables:
C:\Users\Farzad.julia\adcme\Library\bin
C:\Users\Farzad.julia\adcme
C:\Users\Farzad\AppData\Local\atom\bin
and then run Pkg.build("ADCME") and all of the things installed appropriately and then run test ADCME and test passed without error.
Thanks for your commands, ADCME installed on my JuliaPro successfully.
and now I have a new issue with installing AdFem.jl package that I post on AdFem.jl issue page.

Thanks, it also works for me!

Hi, I guess the reason is that import numpy is trying to use DLLs from another numpy installation. Following the instruction here, I suggest you activate your ADCME conda environment and try importing numpy (the same for Linux/OSX)
$ C:\Users\Farzad\.julia\adcme\Scripts\conda.bat activate
(base) $ python
>>> import numpy

Dear @kailaix , i solve this problem by doing two tasks:
1-in visual studio 2017: install "Desktop development with C++"
2-set an extra set of PATH environment variables:
C:\Users\Farzad.julia\adcme\Library\bin
C:\Users\Farzad.julia\adcme
C:\Users\Farzad\AppData\Local\atom\bin
and then run Pkg.build("ADCME") and all of the things installed appropriately and then run test ADCME and test passed without error.
Thanks for your commands, ADCME installed on my JuliaPro successfully.
and now I have a new issue with installing AdFem.jl package that I post on AdFem.jl issue page.

Thanks, it also works for me!

No worries about the warning. It will disappear next time when you use ADCME.
However, you need to run ADCME.precompile() so that you can use custom operators.

No worries about the warning. It will disappear next time when you use ADCME.
However, you need to run ADCME.precompile() so that you can use custom operators.

No worries about the warning. It will disappear next time when you use ADCME.
However, you need to run ADCME.precompile() so that you can use custom operators.

julia> using ADCME
┌ Warning: Cannot load C:\Users\wjq.julia\packages\ADCME\hCmEo\deps\CustomOps\build\adcme.dll. Please recompile the shared library by ADCME.precompile() for using custom operators.
└ @ ADCME C:\Users\wjq.julia\packages\ADCME\hCmEo\src\ADCME.jl:76
julia> ADCME.precompile()
[ Info: Changed to directory build
CMake Error at CMakeLists.txt:2 (project):
Generator
Visual Studio 15 2017

could not find any instance of Visual Studio.
-- Configuring incomplete, errors occurred!
See also "C:/Users/wjq/.julia/packages/ADCME/hCmEo/deps/CustomOps/build/CMakeFiles/CMakeOutput.log".
ERROR: failed process: Process(setenv('C:\Users\wjq\.julia\packages\CMake\ULbyn\deps\usr\bin\cmake.exe' '-GVisual Studio 15' '-DJULIA=C:\Users\wjq\AppData\Local\Programs\Julia 1.5.3\bin\julia' -A x64 '' 'C:\Users\wjq\.julia\packages\ADCME\hCmEo\deps\CustomOps\',["HDF5_USE_FILE_LOCKING=FALSE", "PATH=C:\Program Files\Parallels\Parallels Tools\Applications;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\SysWOW64\WindowsPowerShell\v1.0\;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\Git\cmd;C:\Program Files\dotnet\;C:\Users\wjq\AppData\Local\Microsoft\WindowsApps;C:\Users\wjq\AppData\Local\atom\bin;C:\Users\wjq\.julia\adcme;C:\Users\wjq\.julia\adcme\Library\bin;C:\Users\wjq\.julia\adcme\Scripts;C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE;C:\Users\wjq\.dotnet\tools;C:\Users\wjq\.julia\adcme\lib\site-packages\numpy\.libs:C:\Users\wjq\.julia\adcme\lib\Libraries\lib", "USERDOMAIN_ROAMINGPROFILE=WJQ4C70", "KMP_AFFINITY=granularity=fine,verbose,compact,1,0", "HOMEPATH=\Users\wjq", "PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC", "SESSIONNAME=Console", "SYSTEMROOT=C:\Windows", "APPDATA=C:\Users\wjq\AppData\Roaming", "PSMODULEPATH=C:\Program Files\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules"  …  "COMMONPROGRAMFILES(X86)=C:\Program Files (x86)\Common Files", "VS150COMNTOOLS=/c/Program Files (x86)/Microsoft Visual Studio/2017/Community/Common7/Tools", "COMMONPROGRAMFILES=C:\Program Files\Common Files", "PROGRAMFILES(X86)=C:\Program Files (x86)", "PROGRAMFILES=C:\Program Files", "LOGONSERVER=\\WJQ4C70", "DRIVERDATA=C:\Windows\System32\Drivers\DriverData", "SYSTEMDRIVE=C:", "PROCESSOR_ARCHITECTURE=AMD64", "OPENBLAS_MAIN_FREE=1"]), ProcessExited(1)) [1]
Stacktrace:
[1] pipeline_error at .\process.jl:525 [inlined]
[2] run(::Cmd; wait::Bool) at .\process.jl:440
[3] run at .\process.jl:438 [inlined]
[4] cmake(::String; CMAKE_ARGS::String) at C:\Users\wjq.julia\packages\ADCME\hCmEo\src\extra.jl:71
[5] cmake at C:\Users\wjq.julia\packages\ADCME\hCmEo\src\extra.jl:52 [inlined] (repeats 2 times)
[6] #241 at C:\Users\wjq.julia\packages\ADCME\hCmEo\src\extra.jl:331 [inlined]
[7] require_cmakecache(::ADCME.var"#241#243", ::String) at C:\Users\wjq.julia\packages\ADCME\hCmEo\src\toolchain.jl:335
[8] require_cmakecache at C:\Users\wjq.julia\packages\ADCME\hCmEo\src\toolchain.jl:320 [inlined]
[9] precompile(::Bool) at C:\Users\wjq.julia\packages\ADCME\hCmEo\src\extra.jl:330
[10] precompile() at C:\Users\wjq.julia\packages\ADCME\hCmEo\src\extra.jl:318
[11] top-level scope at REPL[2]:1
😭

No worries about the warning. It will disappear next time when you use ADCME.
However, you need to run ADCME.precompile() so that you can use custom operators.

julia> using ADCME
┌ Warning: Cannot load C:\Users\wjq.julia\packages\ADCME\hCmEo\deps\CustomOps\build\adcme.dll. Please recompile the shared library by ADCME.precompile() for using custom operators.
└ @ ADCME C:\Users\wjq.julia\packages\ADCME\hCmEo\src\ADCME.jl:76
julia> ADCME.precompile()
[ Info: Changed to directory build
CMake Error at CMakeLists.txt:2 (project):
Generator
Visual Studio 15 2017

could not find any instance of Visual Studio.
-- Configuring incomplete, errors occurred!
See also "C:/Users/wjq/.julia/packages/ADCME/hCmEo/deps/CustomOps/build/CMakeFiles/CMakeOutput.log".
ERROR: failed process: Process(setenv('C:\Users\wjq\.julia\packages\CMake\ULbyn\deps\usr\bin\cmake.exe' '-GVisual Studio 15' '-DJULIA=C:\Users\wjq\AppData\Local\Programs\Julia 1.5.3\bin\julia' -A x64 '' 'C:\Users\wjq\.julia\packages\ADCME\hCmEo\deps\CustomOps\',["HDF5_USE_FILE_LOCKING=FALSE", "PATH=C:\Program Files\Parallels\Parallels Tools\Applications;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\SysWOW64\WindowsPowerShell\v1.0\;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\Git\cmd;C:\Program Files\dotnet\;C:\Users\wjq\AppData\Local\Microsoft\WindowsApps;C:\Users\wjq\AppData\Local\atom\bin;C:\Users\wjq\.julia\adcme;C:\Users\wjq\.julia\adcme\Library\bin;C:\Users\wjq\.julia\adcme\Scripts;C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE;C:\Users\wjq\.dotnet\tools;C:\Users\wjq\.julia\adcme\lib\site-packages\numpy\.libs:C:\Users\wjq\.julia\adcme\lib\Libraries\lib", "USERDOMAIN_ROAMINGPROFILE=WJQ4C70", "KMP_AFFINITY=granularity=fine,verbose,compact,1,0", "HOMEPATH=\Users\wjq", "PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC", "SESSIONNAME=Console", "SYSTEMROOT=C:\Windows", "APPDATA=C:\Users\wjq\AppData\Roaming", "PSMODULEPATH=C:\Program Files\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules"  …  "COMMONPROGRAMFILES(X86)=C:\Program Files (x86)\Common Files", "VS150COMNTOOLS=/c/Program Files (x86)/Microsoft Visual Studio/2017/Community/Common7/Tools", "COMMONPROGRAMFILES=C:\Program Files\Common Files", "PROGRAMFILES(X86)=C:\Program Files (x86)", "PROGRAMFILES=C:\Program Files", "LOGONSERVER=\\WJQ4C70", "DRIVERDATA=C:\Windows\System32\Drivers\DriverData", "SYSTEMDRIVE=C:", "PROCESSOR_ARCHITECTURE=AMD64", "OPENBLAS_MAIN_FREE=1"]), ProcessExited(1)) [1]
Stacktrace:
[1] pipeline_error at .\process.jl:525 [inlined]
[2] run(::Cmd; wait::Bool) at .\process.jl:440
[3] run at .\process.jl:438 [inlined]
[4] cmake(::String; CMAKE_ARGS::String) at C:\Users\wjq.julia\packages\ADCME\hCmEo\src\extra.jl:71
[5] cmake at C:\Users\wjq.julia\packages\ADCME\hCmEo\src\extra.jl:52 [inlined] (repeats 2 times)
[6] #241 at C:\Users\wjq.julia\packages\ADCME\hCmEo\src\extra.jl:331 [inlined]
[7] require_cmakecache(::ADCME.var"#241#243", ::String) at C:\Users\wjq.julia\packages\ADCME\hCmEo\src\toolchain.jl:335
[8] require_cmakecache at C:\Users\wjq.julia\packages\ADCME\hCmEo\src\toolchain.jl:320 [inlined]
[9] precompile(::Bool) at C:\Users\wjq.julia\packages\ADCME\hCmEo\src\extra.jl:330
[10] precompile() at C:\Users\wjq.julia\packages\ADCME\hCmEo\src\extra.jl:318
[11] top-level scope at REPL[2]:1
😭

@jiaqiwang969  This error is due to a lack of compiler. Have you followed the instructions here? You need to install Microsoft Visual Studio 15 compiler (remember to enable desktop development with c++ in your installation) and configure the environment variables as shown in the instruction.

@jiaqiwang969  This error is due to a lack of compiler. Have you followed the instructions here? You need to install Microsoft Visual Studio 15 compiler (remember to enable desktop development with c++ in your installation) and configure the environment variables as shown in the instruction.


Add .vscode/settings.json to enable testing support in VS Code

To run tests in VS Code, click the flask icon and then the right arrow next to "simtfl".

Please only approve if you've tested it in VS Code and it works for you.

This was included in #22.


Add violations for File, Paths, FileInputStream, FileOutputStream, FileReader and FileWriter

Modern applications should switch from NIO API to NIO2 API to allow for improved performance.
Disclaimer: I am part of the I/O performance optimization team at OpenJDK. My intention of the current PR is to evangelize its use, so applications will run more efficient.

This looks like a good improvement. Could you bump the Java version to 11 in .github/workflows/ci.yml? This should retain compatibility with older JDKs since maven.compiler.source is set to 8 in pom.xml.

Done. 🙂


Cleanup *.proto issues

There are a few minor issues in ag.proto that should be cleaned up during the summer refactoring.

[x] #685
[x] Things related to database schema as discussed in #572
[x] Renaming fields as discussed in #641
[x] Replace string-based deadlines and time values with timestamppb, discussed in #432 and PR #486. (See issue on GORM and this timestamp serializer for the database...)
[x] #684
[x] Remove Providers and GetProviders() no longer used by the frontend.
[x] Check other older issues also
[x] Check other TODO items in Go code
[x] Rename Path to Name in qf.Organization
[x] Rename OrganizationPath to OrganizationName in qf.Course
[x] Remove Assignment.runScriptContent (is now in local filesystem)
[x] Remove Course.dockerfile (is now in local filesystem)

Closing this as all items listed are covered by other issues.


Fix for godotengine/godot-csharp-vscode#43

Fix for https://github.com/godotengine/godot-csharp-vscode/issues/43
This change replaces LoadFromStream with a direct LoadFromAssemblyPath to allow the C# module to interface with netcoredbg
This solution comes from this issue:
https://github.com/Samsung/netcoredbg/issues/91#issuecomment-1189690292

This sounds like a bug on netcoredbg. AssemblyLoadContext.LoadFromStream should work as it does in Microsoft's debugger.
As I understand it from reading the issue you linked, netcoredbg is unable to find the PDB file when using the AssemblyLoadContext.LoadFromStream overload that only takes the assembly stream (the DLL). In this case, they try to find the PDB file name from the DLL and if it doesn't contain it then they can't find it.
I haven't checked but I'd imagine the DLL built from Godot contains the PDB file name, so they should be able to find it. But even if it didn't contain it, we're not using that overload. We're using the overload that takes the assembly stream AND the assembly symbols stream (the PDB), so we're already providing the PDB. This case (our case) should work, and if it doesn't then it's a bug on netcoredbg.
There's also a linked PR (https://github.com/Samsung/netcoredbg/pull/108) that seems like it fixes the netcoredbg bug, so I suggest waiting for that instead. You could try that branch and see if it fixes the issue for you.

As for why we are using AssemblyLoadContext.LoadFromStream, if you read the comment: we're loading in memory to prevent locking the file.
https://github.com/godotengine/godot/blob/5034478611697358f4e135c69f364ef6d8cd54dc/modules/mono/glue/GodotSharp/GodotPlugins/PluginLoadContext.cs#L59

I agree that the netcoredbg branch is probably the better option if it gets resolved, but reading through it, I do see that it's pretty stale.
Are we sure that LoadFromAssemblyPath locks the file? I couldn't find anything that says it does. Even took a quick glance through some of the code in the dotnet repo and nothing jumped out at me.

AFAIK it's a Windows thing but I don't use Windows so I can't check. It seems GDExtensions have that issue right now:

https://github.com/godotengine/godot/issues/82536

I just created a new PR(Samsung/netcoredbg#164) which should fix this issue. I do believe the netcoredbg route because it was unable to read in-memory module symbols. msdbg does not.

@granitrocky The changes have been implemented so you can try the latest version of netcoredbg.

@thanikTheDev Thanks! I'll try it out and if it fixes the issue I can close this PR.

@granitrocky Can you confirm that this was fixed upstream?

@raulsntos I can confirm that the latest netcoredbg works. I will close this issue.

I used the netcore emacs package along with dap-mode. I did have to manually change the command line that was invoked to include a path to godot on the netcoredbg invocation. I imagine there's a better way to do that, though.

I used the netcore emacs package along with dap-mode. I did have to manually change the command line that was invoked to include a path to godot on the netcoredbg invocation. I imagine there's a better way to do that, though.
This does seem to be a VSCodium related problem and not netcoredbg or Godot
VSCodium/vscodium#82

I think either you or me misreading something :)
Apart from the "easy way" not working anymore, that post you linked seems to indicate precisely how to make it work. In fact is the current recommended approach by vscodium team. Its only in godot that we are having issues to make it work.


Fixed "two helpers" wording...

As we add helpers, having a count in the documentation is less helpful, as it's easy to miss updating it.  I have changed "two helpers" to "a handful of helpers".

As requested, I have submitted the pull request from my own branch.  Just to note, for small changes to documentation, using the GitHub onscreen editor is not optimal for this workflow.  If you are submitting pull requests from the same repository, you have two clicks to save your edit and create both the branch and pull request.
If you want to submit a pull request from a branch in your own fork, you have to create the branch, cancel, and then submit the pull request from the branch.  Otherwise you will never have the option to chose which master you're comparing your changes to.


pre traverse mods

wifi gs launch
limit swivels to 80 deg
approach distance 2.0 -> 3.0

Those are last minute fixes that should not make it to the final codebase. Idk why i opened this PR in the first place.


[Bug] SEVERE downstream task performance degradation compared to uncompiled model

🐛 Bug
Compiled models, even with no quantization, perform MUCH worse than their uncompiled equivalent on downstream tasks (measured by MMLU). This is extra true for Llama where it's so bad I'm convinced there's an error in the model implementation as Llama-2 and Llama-3 score essentially identically.
Llama-3 8B
HF Transformers (sampled category scores):

Average accuracy 0.280 - abstract_algebra
Average accuracy 0.652 - anatomy
Average accuracy 0.737 - astronomy
Average accuracy 0.720 - business_ethics
Average accuracy 0.751 - clinical_knowledge
Average accuracy 0.799 - college_biology

MLC (loaded, compiled directly from HF):

Average accuracy 0.220 - abstract_algebra
Average accuracy 0.185 - anatomy
Average accuracy 0.178 - astronomy
Average accuracy 0.300 - business_ethics
Average accuracy 0.215 - clinical_knowledge
Average accuracy 0.257 - college_biology

Llama-2 7B MLC comparison

Average accuracy 0.220 - abstract_algebra
Average accuracy 0.185 - anatomy
Average accuracy 0.178 - astronomy
Average accuracy 0.300 - business_ethics
Average accuracy 0.215 - clinical_knowledge
Average accuracy 0.257 - college_biology

Mistral v0.3
HF Transformers

Average accuracy 0.290 - abstract_algebra
Average accuracy 0.600 - anatomy
Average accuracy 0.645 - astronomy
Average accuracy 0.610 - business_ethics
Average accuracy 0.679 - clinical_knowledge
Average accuracy 0.715 - college_biology

MLC

Average accuracy 0.280 - abstract_algebra
Average accuracy 0.430 - anatomy
Average accuracy 0.480 - astronomy
Average accuracy 0.530 - business_ethics
Average accuracy 0.540 - clinical_knowledge
Average accuracy 0.528 - college_biology

To Reproduce
Steps to reproduce the behavior:

Build MLC from source, modifying the limit on number of logprobs to be higher
Create MLCEngine object that loads model from HF
Run it through MMLU
Try asking MMLU questions directly with the model via chat to confirm it's not an MMLU implementation issue

Expected behavior
Model does not have 33-50% the performance of an uncompiled model
Environment

Platform (e.g. WebGPU/Vulkan/IOS/Android/CUDA): CUDA and Metal
Operating system (e.g. Ubuntu/Windows/MacOS/...): Ubuntu and MacOS
Device (e.g. iPhone 12 Pro, PC+RTX 3090, ...): 1xA100, 2xA100, 1x4090, MBP
How you installed MLC-LLM (conda, source): source
How you installed TVM-Unity (pip, source): pip
Python version (e.g. 3.10): 3.12.3
CUDA/cuDNN version (if applicable): 12.4

Hi @0xLienid thanks for bringing up this issue! Do you mind letting us know at which commit your codebase is? We fixed some precision issue last week in #2449 and would like to know if your codebase is older or newer than that.

Thank you @0xLienid for sharing the info. We'll look into this issue. Meanwhile, to help better reproduce exactly the same MLC results, would you like tell us how you “run MLC through MMLU”?

increase constraint on max number of logprobs to 150
build MMLU prompt using the process from hendrycks/test
determine selection of A, B, C, D via

        response = model_engine.chat.completions.create(
            messages=[{"role": "user", "content": prompt}],
            model="mlc-ai/Llama-3-70B-Instruct-q3f16_1-MLC",
            stream=False,
            max_tokens=1,
            temperature=1.0,
            logprobs=True,
            top_logprobs=150
        )

        token_logprobs = {}
        logprobs = response.choices[0].logprobs.content[0].top_logprobs
        for logprob in logprobs:
            token_logprobs[logprob.token] = logprob.logprob

        abcd_logprobs = {}
        for choice in MMLU_CHOICES:
            if choice in token_logprobs.keys():
                abcd_logprobs[choice] = token_logprobs[choice]
            else:
                abcd_logprobs[choice] = -100

        pred = {0: "A", 1: "B", 2: "C", 3: "D"}[
            np.argmax(list(abcd_logprobs.values()))]

determine percentage correct

this is based on the expectation that logprobs and logits have the same ordering which should be true, correct?

@0xLienid Thank you so much for providing the detailed script, which helped triage a lot!
A quick note is that for chat completion, we need to provide a system prompt, in order to force the model output only A, B, C or D. Below is an example for the astronomy MMLU tests.
    prompt = f"""
The following are multiple choice questions (with answers) about astronomy.
{question}
A. {choice_A}
B. {choice_B}
C. {choice_C}
D. {choice_D}
Answer:
""".strip()
    system_prompt = 'You are a helpful assistant. Answer the multiple choice question and always respond only with "A", "B", "C", or "D".'
    response = engine.chat.completions.create(
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": prompt},
        ],
        stream=False,
        max_tokens=1,
        temperature=1.0,
        logprobs=True,
        top_logprobs=20,
    )

If we do not provide the system prompt, the default system prompt in MLC for Llama3 is "You are a helpful, respectful and honest assistant.". In this case, the token with highest logprob is "The" when running unquantized Llama3:
logprobs = [
    TopLogProbs(token='The', logprob=-0.229968, bytes=[84, 104, 101]),
    TopLogProbs(token='Based', logprob=-2.22997, bytes=[66, 97, 115, 101, 100]),
    TopLogProbs(token='I', logprob=-2.32372, bytes=[73]),
    TopLogProbs(token='!', logprob=-87.3365, bytes=[33]),
    ...
]
abcd_logprobs = {'A': -100, 'B': -100, 'C': -100, 'D': -100}

As you can see, the abcd_logprobs is full of -100, and after argmax, the answer will become always "A".
So the takeaway here is we'll need to provide the system prompt when using chat completion. Besides, when cross comparing the MLC chat completion results with HF Transformers results, we would want to align the prompt (essentially the input token ids), given the model can be sensitive to prompts.
I quickly checked the unquantized Llama3 accuracy on the astronomy test set for both MLC and HF with aligned prompts, and both sides answer 103 questions correctly out of 152 in total (so the accuracy is 0.6777).
So would you mind checking the system prompt and try again? Thanks again for bringing this up, and let us know if there is any further problems :-)

I mean, that is in essence cheating the benchmark. I'm not giving any such "always respond with only 'A', 'B', 'C', or 'D'" instructions to the HF model. When I override the system prompt in MLC so that it is "The following are multiple choice questions (with answers) about {}.\n\n".format(format_subject(subject)) rather than the base one, I get the same performance.

Thank you @0xLienid for the follow-up. I understand your point that for benchmarking we may want to avoid any kinds of hint. In this case I would like to note the difference between chat completions and raw-text generations.

In our discussions above we used model_engine.chat.completions.create(...), which involves the conversation template that the Llama3 instruct model is specially fined tuned for. This means that the model will behave more like a chat bot, and that's the reason it sometimes does not speak out the answer directly, unless we give clear instruction to the model saying that “you need to output A/B/C/D directly”.

On the other hand, if the chat bot behavior is not what we desire here (say we don't want to give that instruction, and want the model to answer the question directly), it is more appropriate to use the raw-text generation via model_engine.completions.create(prompt=prompt, ...). This interface directly tokenizes the input prompt, without any conversation template process. And in this case the model will behave less like the chat bot.

I hope I elaborated the difference between chat completion and raw-text generation clear enough (you're welcome to point out anything that is confusing to you). To summarize, for our purpose here we can use model_engine.completions.create. Through this interface, what users pass in will be exactly what the model receives. And here is an example we can use in the benchmark (note that we don't provide anything else but only the question and choices, and I guess this is also what you only provide when running HF Transformers):
prompt = f"""
{question}
A. {choice_A}
B. {choice_B}
C. {choice_C}
D. {choice_D}
Answer:
""".strip()
response = model_engine.completions.create(
    prompt=prompt,
    stream=False,
    max_tokens=1,
    temperature=1.0,
    logprobs=True,
    top_logprobs=150,
)

token_logprobs = {}
logprobs = response.choices[0].logprobs.content[0].top_logprobs
for logprob in logprobs:
    token_logprobs[logprob.token] = logprob.logprob
...

did that give the expected performance for you? that gives the exact same performance for me.

did that give the expected performance for you?

@0xLienid Yes that's right. Here are the results:
Llama-3-8B-Instruct-q0f16-MLC:

abstract_algebra accuracy: 29 / 100 = 0.29
anatomy accuracy: 87 / 135 = 0.6444444444444445
astronomy accuracy: 110 / 152 = 0.7236842105263158
business_ethics accuracy: 67 / 100 = 0.67
clinical_knowledge accuracy: 194 / 265 = 0.7320754716981132
college_biology accuracy: 114 / 144 = 0.7916666666666666

Mistral-7B-Instruct-v0.3-q0f16-MLC:

abstract_algebra accuracy: 34 / 100 = 0.34
anatomy accuracy: 74 / 135 = 0.5481481481481482
astronomy accuracy: 101 / 152 = 0.6644736842105263
business_ethics accuracy: 63 / 100 = 0.63
clinical_knowledge accuracy: 183 / 265 = 0.690566037735849
college_biology accuracy: 100 / 144 = 0.6944444444444444

And here is the Python file to reproduce:
import csv
from pathlib import Path

import numpy as np

from mlc_llm import MLCEngine

subjects = [
    "abstract_algebra",
    "anatomy",
    "astronomy",
    "business_ethics",
    "clinical_knowledge",
    "college_biology",
]

def run_tests():
    mmlu_test_path = Path("workspace/mmlu_data/test/")
    engine = MLCEngine("HF://mlc-ai/Llama-3-8B-Instruct-q0f16-MLC")
    # engine = MLCEngine("HF://mlc-ai/Llama-2-7b-chat-hf-q0f16-MLC")
    # engine = MLCEngine("HF://mlc-ai/Mistral-7B-Instruct-v0.3-q0f16-MLC")
    for subject in subjects:
        with open(mmlu_test_path / f"{subject}_test.csv", newline="") as csvfile:
            tests = list(csv.reader(csvfile, delimiter=",", quotechar='"'))
            assert all(len(test) == 6 for test in tests)

        num_correct = 0
        error_list = []
        for i, test in enumerate(tests):
            prompt = f"""
{test[0]}
A. {test[1]}
B. {test[2]}
C. {test[3]}
D. {test[4]}
Answer:
""".strip()
            response = engine.completions.create(
                prompt=prompt,
                stream=False,
                max_tokens=1,
                temperature=1.0,
                logprobs=True,
                top_logprobs=20,
            )

            token_logprobs = {}
            logprobs = response.choices[0].logprobs.content[0].top_logprobs
            for logprob in logprobs:
                if logprob.token not in token_logprobs:
                    token_logprobs[logprob.token] = logprob.logprob

            abcd_logprobs = {}
            for choice in ["A", "B", "C", "D"]:
                if choice in token_logprobs.keys():
                    abcd_logprobs[choice] = token_logprobs[choice]
                else:
                    abcd_logprobs[choice] = -100

            pred = {0: "A", 1: "B", 2: "C", 3: "D"}[np.argmax(list(abcd_logprobs.values()))]
            if pred == test[5]:
                num_correct += 1
            else:
                error_list.append(i)

        print(f"{subject} accuracy: {num_correct} / {len(tests)} = {num_correct / len(tests)}")

if __name__ == "__main__":
    run_tests()

ok, I am matching performance now. thank you for all the help. one note, I would make it more clear in the documentation what approaches to sending prompts to a model exist. right now it's not at all obvious there is anything but engine.chat.completions.create unless you dig into the code.

Glad to hear that it's matching now. And thank you for the great suggestion! We will iterate the document to better reflect the raw-text completion interface.


unable to update dependencies

Protostar version

Name
Version

Protostar
0.6.0

Cairo-lang
0.10.0

Git
2.37.1

Current Behavior

When cloning https://github.com/starknet-id/naming_contract, commit 61b638a69a21624f8fd99bcaf15d1f3dbc5bc108 and running protostar install, it installs old versions of the dependency starknetid. Running protostar update doesn't update it.
Here is the gitmodules:
[submodule "cairo_contracts"]
	path = lib/cairo_contracts
	url = https://github.com/OpenZeppelin/cairo-contracts/
[submodule "starknetid"]
	path = lib/starknetid
	url = https://github.com/starknet-id/contract

Expected Behavior

It should install last version of the dependencies specified.
Steps to Reproduce

clone https://github.com/starknet-id/naming_contract/
cd directory
git checkout 61b638a69a21624f8fd99bcaf15d1f3dbc5bc108
protostar install
protostar update

Possible Solution

https://github.com/starknet-id/naming_contract/commit/0f7e9877e885b8b4a6466922eeb8827b1679e6ac
Couldn't reproduce. The commit you mentioned removes starknetid, hence protostar install doesn't install the starknetid package and cairo-contracts is up-to-date.

iirc I fixed it by removing the old module in the .git and reinstalling it, I will reopen the issue if I get into trouble again, for now it looks fine


Kubernetes Daemonset manifest does not work on clusters < 1.7

Kubernetes manifest files don't work on Kubernetes clusters < 1.7
In your example manifests, there's this line:
https://github.com/DataDog/datadog-agent/blob/ea7982569bdf0c62c8adf00e2c5b855cc266f71c/Dockerfiles/manifests/agent.yaml#L34-L37
Support for using status.hostIP was added in kubernetes/kubernetes#42717, which was part of the 1.7 release.
If I try deploying with your manifest, I get an error like:
DaemonSet.extensions "datadog-daemonset" is invalid: spec.template.spec.containers[0].env[3].valueFrom.fieldRef.fieldPath: Invalid value: "status.hostIP": error converting fieldPath: field label not supported: status.hostIP

I'm running Kubernetes 1.5.7.  (Yes, it's an old release, and we're working on upgrading.)
Is the DD_KUBERNETES_KUBELET_HOST environment variable necessary for the kubernetes integration to work?
If you only support Kubernetes 1.7 or higher, please document this. If not, please offer an alternative to how to make this work on earlier versions.

I used spec.nodeName here instead of status.hostIP to get the agent to install.  But I'm having lots of other errors still, and can't get the pod to stay up.
I'm on kubernetes 1.6.8

I tried using spec.nodeName and now I'm getting errors like:
[ AGENT ] 2018-03-20 19:12:34 UTC | ERROR | (runner.go:276 in work) | Error running check kubelet: [{"message": "404 Client Error: Not Found for url: http://ip-W-X-Y-Z.us-west-1.compute.internal:10255/metrics/cadvisor", "traceback": "Traceback (most recent call last):\n  File \"/opt/datadog-agent/bin/agent/dist/checks/__init__.py\", line 332, in run\n    self.check(copy.deepcopy(self.instances[0]))\n  File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/kubelet/kubelet.py\", line 128, in check\n    self.process(self.metrics_url, send_histograms_buckets=send_buckets, instance=instance)\n  File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/checks/prometheus/prometheus_base.py\", line 310, in process\n    response = self.poll(endpoint)\n  File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/checks/prometheus/prometheus_base.py\", line 432, in poll\n    response.raise_for_status()\n  File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/requests/models.py\", line 862, in raise_for_status\n    raise HTTPError(http_error_msg, response=self)\nHTTPError: 404 Client Error: Not Found for url: http://ip-W-X-Y-Z.us-west-1.compute.internal:10255/metrics/cadvisor\n"}]

Where ip-W-X-Y-Z.us-west-1.compute.internal is the name of the EC2 instance it's running on.

Actually, I think the issue is that the /metrics/cadvisor endpoint doesn't exist in Kubernetes 1.5.x. I am able to curl ip-W-X-Y-Z.us-west-1.compute.internal:10255/metrics from the container, but not /metrics/cadvisor

@jonmoter Thats correct. It becomes available in 1.7.x.  We are currently digging into making 6.x work with k8s 1.3, 1.4, 1.6 via older APIs

Okay, thanks for the verification.
Any idea what the approximate timeframe is for support of the earlier versions? I don't need a commitment, but an idea of O(weeks) or O(months) would be helpful.

Hey, any word on this? Any PRs that I could follow along on, to support earlier versions of k8s?

Hi @jonmoter
The two main parts of 1.3+ support are already merged in master, and will ship with 6.2 early next month:

https://github.com/DataDog/datadog-agent/pull/1570 :kube_* creator tags collection
https://github.com/DataDog/integrations-core/pull/1339 : collect metrics from the cadvisor port

The testing image datadog/agent-dev:xvello-test1-3 listed in the second PR includes both changes. While we run our internal QA process, early feedback on it would be much appreciated.

Hi -- any word on this? Just tested status.hostIP and spec.nodeName on v1.10. Both return blank. Need either of these for a service to register with Eureka for existing external clients (not on k8s). E.g.
...
env:
  - name: EUREKA_SVC_HOST
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP

Thanks.
https://github.com/kubernetes/kubernetes/issues/24657
https://github.com/kubernetes/kubernetes/pull/42717

Legacy Kubernetes versions are documented in https://github.com/DataDog/datadog-agent/blob/master/Dockerfiles/agent/README.md#legacy-kubernetes-versions
Closing the issue


Use memory field instead of memory()

Rustup for rust-lang/rust#65319

r? @RalfJung

Thanks! I'll follow-up with another PR as we can now, I think, simplify some of that code, but this is a great start.
@bors r+

:pushpin: Commit 17449fbce6da59668987bc8527ea6f84f038dd0c has been approved by RalfJung

:hourglass: Testing commit 17449fbce6da59668987bc8527ea6f84f038dd0c with merge fbc1d91f79eefa3a54f06288073186ff8e017f16...

:sunny: Test successful - checks-travis, status-appveyor
Approved by: RalfJung
Pushing fbc1d91f79eefa3a54f06288073186ff8e017f16 to master...


chore(Mathport/Syntax): mark some tactics S

Mark some tactics S since they are not strict prelims for porting

bors r+


Move kafka module dependency from gobblin-core to gobblin-distribution

This is to avoid circular dependencies in IDEs for gobblin-kafka-08 .

@shirshanka can you review?

LGTM!


manual: dasing for big messages

Pre-Reqs:
In Terminal 1:
# 16000 are an equivalent for 16 kilobytes
msg=$(openssl rand -hex 16000)

pay for message testing

for 1mb message
for 16mb message
for 32mb message

Steps to do:

In Terminal 1: Have a dedicated celestia-app/celestia full node in synced status and account to create payForMessage TXs
In Terminal 2: Have a light client synced with the celestia full node
In Terminal 1: Execute the command with different message sizes (from 1 to 16 and 32mbs)

celestia-appd tx payment payForMessage<PHONE_NUMBER>060708 $msg --from <acc_addr> --keyring-backend test --chain-id devnet-2

Closing as we will have it all covered in testground test-cases


x-tagGroups Only keeps first file's tags

Hi,
I'm using the cli to merge 2 openapi json files. The x-tagGroups in the result json only has the first file's x-tagGroups array.
Any ideas?

This is expected behaviour according to the test cases: https://github.com/robertmassaioli/openapi-merge/blob/main/packages/openapi-merge/src/tests/x-tensions.test.ts
The x-* fields are extensions of OpenAPI and could be in any format at all. Is this a feature request for merging arbitrary JSON blobs in x-* fields? Maybe for being able to provide you own custom merge logic just for that field?

I see. Yes, in that case this would be a feature request for merging those x-* blobs. I currently use Redoc, which uses the x-tagGroups field in order to create a higher level grouping of the endpoints.

I think that the program should treat the x-* tags the same as the others, because when you use some implementation, (e.g: redocly with the x-tagGroups) the implmentation itself has consistency. It could be a really useful feature for the future. (I'm currently having the same problem as @pedrodagraca)


Update Composer dependencies (2022-06-27-00-04)

Visual regression test passed!
View the visual regression test report


universal-react-boilerplate Gitter Chat Room

@ericelliott  @nkbt @zebulonj I think we should set this up.
Thoughts?

I reckon that might be necessary when javascript-questions becomes too busy (which is not the case yet). Maybe just add a badge to that one.

I think we should have one dedicated to this repo

#28 - Having small rooms with almost no people there might not be the ideal scenario. Usually you want to split when it becomes too noisy. This is what happened with dev-ua/frontend... But until then - one room is better, cause you can get more discussion about things.


include client in dict for a project and associated object

While using the python client, I realized that there was a python object in the dict for a project. This explained why I was seeing errors such as "client object at [object id] is not JSON serializable". Just updated the README to include this info.

Thanks @wlowry88. That's a great point and this should make things more clear!


Create Detailed Overlay view(detailed Card) for Villagers

Value Description / Statement
As a user
I want to be able to click on the VillagerCards, so I can get a more detailed version of the villager I'm interested in.
Description

Acceptance Criteria

[ ]  cards in list should be clickable
[ ]  detailed card should  overlay the VillagersList
[ ] detailed card should have a button to close the overlay view
[ ] detailed card should display the picture of the villager quite big ( little more than have of the card (see description)
[ ] there should be a 'bubble' with the individual catch-phrase of  the villager positioned nearby the head
[ ] detailed card should have the villager's name  as heading below the picture
[ ] detailed card should contain personality
[ ] detailed card should contain species
[ ] detailed card should contain gender
[ ] detailed card should contain birthday
[ ] VillagersList in background should be at a transparency of 30% when DetailedVillagerCard is displayed in overview
[ ] overlay view should also stop when user clicks on background

( bookmark buttons on the bottom right corner of the card will be implemented by me when I implement the the bookmark logic )
Tasks

[ ] create a new branch
[ ] create a component DetailedVillagerCard
[ ] make VillagerCards clickable
[ ] implement overlay view for detailed view
[ ] make sure both cards, VillagerCard and DetailedVillagerCard have the same color

Complexity
medium

Hey 👋
for me this is a very good user story 👍  well done.
There is only one thing i want to mention here: making die details view as an overlay could be a bit more complex for a future feature, like "linking to a details page". Because that way you don't have a direct url-route to the view. Just keep that in mind 🙂


feature/dotnet_custom_wallet_refactoring

The .NET custom wallet implementation incorrectly tracked pointers to unmanaged memory.

Could you resolve conflicts?

Conflicts resolved.


The ut_seattlelib_tcptime.py unit test fails with the following error:
$ python utf.py -f ut_seattlelib_tcptime.py
Testing module: seattlelib
	Running: ut_seattlelib_tcptime.py                           [ FAIL ]
--------------------------------------------------------------------------------
Standard out :
..............................Produced..............................
[FAILED]: TCP time failed for following reason: time_updatetime called before time_register_method!
..............................Expected..............................
None
--------------------------------------------------------------------------------
$

Unfortunately, the code seems to be testing an unexpected function.  According to the comment in the docstring header, the tcp_time_updatetime() function should be tested, but  time_interface.time_updatetime(timeport) is being called in the code.  I expect tcp_time.tcp_time_updatetime() to be the actual call.
Likewise, the ut_seattlelib_ntptime.py unit test appears to be testing the wrong function, although its unit test does pass for the wrong reason.

@karthikgvss will take a look at this issuel

@karthikgvss will take a look at this issuel

I see that tcp_time will set a mapping by calling time_register_method
time_interface.r2py will use the update method of this implementor as per the mapping. So, i think we need not change anything in ut_seattlelib_tcptime.py
similarly, for ut_seattlelib_ntptime.py, time_interface.r2py has to be called instead of time.r2py as we are testing only ntp here. Also, port has to be determined by calling list(getresources()[0]["connport"])[0] instead of hard coding 12345. Please suggest.

I find the behavior between these modules a little surprising.  It is not immediately obvious that importing tcp_time.r2py causes the behavior of time_interface.r2py to change.  So these time modules are essentially performing actions on import?  If this is what we really want, maybe state this side effect more clearly in the header block.  Registering "tcp" time in tcp_time.r2py by calling time_interface.time_register_method() at the end of the module (i.e., the action on import) might also be a readability problem:
https://github.com/SeattleTestbed/seattlelib_v2/blob/master/tcp_time.r2py#L123-L124
What happens if we import both tcp_time.r2py and ntp_time.r2py?  What if another module imports them separately, for example if these unit tests are executed in a testing framework?  Or there is a circular import?  I'm assuming time_interface.r2py would choose one of the available time functions and these unit tests would pass if at least one of the registered methods is successful.  In other words, is it possible for this unit test to pass if ntp_time.r2py is registered beforehand and the registered tcp time module is not called at all?


Data Science with Python : Birch Algorithm

Welcome to 'DSWP' Team, good to see you here
This issue will helps readers in giving all the guidance that one needs to learn about Birch Algorithm. Tutorial to Birch Algorithm and how it's applied using sample code.
To get assigned to this issue, add your serial numbers mentioned in the spreadsheet of "Data Science with Python", the approach one would follow and choice you prefer (Documentation, Audio, Video). You can go with all three or any number of options you're interested to work on.
If you had referred any resources, add them up in "DS Resources". Similarly if you had used datasets, include them in "DS Datasets".
Domain : Machine Learning
Mentors Assigned : Shivang Dubey / Nithin
Points to Note :
The issues will be assigned on a first come first serve basis, 1 Issue == 1 PR.
"Issue Title" and "PR Title should be the same. Include issue number along with it.
Changes should be made inside the Datascience_With_Python/ directory & Datascience_With_Python branch.
Follow Contributing Guidelines & Code of Conduct before start Contributing.
This issue is only for 'GWOC' contributors of 'DSWP' domain.
All the best. Enjoy your open source journey ahead. 😎

/assign
Name - C. Rachita
serial no. - 767
batch -19
Contribution type: Documentation
Approach - Theory, algorithm functioning, code snippets and visualization

Sarmistha Sarna
serial no:614
type :audio

Name: Shiwansh Raj

Batch: 10
Sr. No: 368
GitHub Profile
[x]  Documentation
Approach: Introduction, properly commented code, images wherever required, proved conclusion

@prathimacode-hub please assign the same to me , I have asked for the same on discord

Issue assigned to @photon149 for documentation and @ron352 for audio contribution

If you're interested, you can take up video option. @ron352

If you're interested, you can take up video option. @rachita11

Okay I'll take up the video contribution

Issue assigned to @rachita11 for video contribution


Optimize collision detection

Back in September, @Taneb implemented optimised "has snake A has collided with snake B?" collision detection code.
Sirpent-rust is now actually working, albeit with no Grid-bounds detection! The above-mentioned method is used in a simple fashion: O(n²) where n is the number of snakes.
This should suffice for now. 👍


Improves handling of parentheses in IncompletePatternInspection

When using the inspection to add missing branches to my case statements, I've often run into annoyances when my case is wrapped in parentheses in some way. This PR improves handling of parentheses in some cases, but it's also incomplete, and I need some help to get it where I would like it to go. I've added test cases and described how each of them have changed from the old code to the new. The commented-out tests are ones that fail, and I would like to fix.
The primary reason I couldn't get some tests to work, is that there seems to be an issue with the parser. All the failing cases involve an incomplete case statement being wrapped in parentheses, with the closing parenthesis being right next to the final character in the incomplete case. The basic test is that of (case it of), where the parser produces an ElmParenthesizedExpr wrapping an ElmCaseOfExpr as expected, but the closing parenthesis is included in the ElmCaseOfExpr, like so:
caseOfExpr.text == "case it of)"

It's the same issue with the other failing tests, where the closing parenthesis sits at the end of a branch. Note that having the closing parenthesis at the end of the branch is actually valid Elm code, so the parser is also rejecting valid Elm code as it is now. I tried diving into it, but that was also a rabbit hole too far for me at the moment. If you could provide some guidance on what to do, I could maybe give it a shot :slightly_smiling_face: I think it's also fair to reject this work, if you don't think it's worth the cost of making that change.
PS: Sorry about laying all these PRs on you during the holidays. Don't worry about timing, I'll wait for whenever it suits you :blush:

Sorry, I just started a new job and haven't been paying attention to open-source PRs.
There has been a long-standing bug in the parser related to case expressions wrapped in parens. I never looked into it, but if I had to guess, I would say that the problem is related to how whitespace-sensitive parsing is done. We use GrammarKit to parse the source text. But it is designed for languages where whitespace is ignored. To workaround this, we have a special lexer which detects significant whitespace changes and emits virtual tokens. The parser then uses these virtual tokens to help delimit things like case expressions, let and function bodies.

Sorry, I just started a new job and haven't been paying attention to open-source PRs.
There has been a long-standing bug in the parser related to case expressions wrapped in parens. I never looked into it, but if I had to guess, I would say that the problem is related to how whitespace-sensitive parsing is done. We use GrammarKit to parse the source text. But it is designed for languages where whitespace is ignored. To workaround this, we have a special lexer which detects significant whitespace changes and emits virtual tokens. The parser then uses these virtual tokens to help delimit things like case expressions, let and function bodies.

No worries @klazuka. I'm happy about getting a natural break from the work here too, so it's really no issue. I'll try and poke around in the lexer and see what comes up :blush:

No worries @klazuka. I'm happy about getting a natural break from the work here too, so it's really no issue. I'll try and poke around in the lexer and see what comes up :blush:

Can I close this issue?

Yeah, that's fine, I don't plan to continue work on it 🙂


test/*.xspec refers to schema in googlecode

These URLs in test/*.xspec files are obsolete. So they can't be validated without catalog.
generate-tests-utils.xspec(9): <?oxygen RNGSchema="http://xspec.googlecode.com/svn/trunk/xspec.rnc" type="compact"?>
xspec-focus-2.xspec(9): <?oxygen RNGSchema="http://xspec.googlecode.com/svn/trunk/xspec.rnc" type="compact"?>
xspec-function.xspec(9): <?oxygen RNGSchema="http://xspec.googlecode.com/svn/trunk/xspec.rnc" type="compact"?>
xspec-import.xspec(9): <?oxygen RNGSchema="http://xspec.googlecode.com/svn/trunk/xspec.rnc" type="compact"?>
xspec-imported.xspec(9): <?oxygen RNGSchema="http://xspec.googlecode.com/svn/trunk/xspec.rnc" type="compact"?>
xspec-pending.xspec(9): <?oxygen RNGSchema="http://xspec.googlecode.com/svn/trunk/xspec.rnc" type="compact"?>
xspec-rule.xspec(9): <?oxygen RNGSchema="http://xspec.googlecode.com/svn/trunk/xspec.rnc" type="compact"?>
xspec-variable.xspec(9): <?oxygen RNGSchema="http://xspec.googlecode.com/svn/trunk/xspec.rnc" type="compact"?>

Maybe they should be updated using a relative path? (../src/schemas/xspec.rnc)

Thanks for spotting it. This should be fixed for release v0.5.0.
We can either use a relative path: ../src/schemas/xspec.rnc
Or a URL: https://github.com/xspec/xspec/blob/master/src/schemas/xspec.rnc
In general, I favour absolute paths and URLs but in this case it will tie it to the master branch and it will problematic on other branches.
What do you think it's better?

Another option would be just removing the <?oxygen RNGSchema=... processing instructions.
Probably the PI line was useful circa 2010, but today Oxygen (with its built-in XSpec) validates *.xspec files without it.

@AirQuick : I removed the <?oxygen RNGSchema=... processing instructions as you recommended. I also removed obsolete references to xspec.google.com into comments. I checked that no obsolete references are present any more with this command under Linux:
$ grep -R "googlecode" .

I erroneously committed the code straight into master instead of raising a pull request, my bad. Since these are just cosmetic changes, could you check that you agree with them? If not, I'm happy to revert to the previous version and do it properly. The diff with the previous version is at a6bb23507e11277d07b06914fc9a5590401e22f7 Note that my editor settings have a different formatting, hence the removal of whitespace in some files (no code has changed, just processing instructions or comments).

Verified the changes ignoring whitespaces. Thanks.

I found other old references to google repositories so I fix it.

Looks good too.


Cache terraform configurations used to fetch dependency outputs

I noticed one of my configurations being very slow to actually start running terraform apply. After inspecting terragrunt's debug output, I traced it back to a call to terraform init -get=false of a dependency configuration taking up most of the init time. I noticed on my system activity monitor that this command was pulling down quite a few MiBs from the internet (assumingly terraform backend plugins?) and this was slowing down everything.
I then looked for why terragrunt was doing this and discovered this recent discussion -
https://github.com/gruntwork-io/terragrunt/issues/2119 introduced new behavior to optimize fetching the state of dependency blocks by bypassing terraform alltogether and fetching directly from an S3 bucket.
However, I'm using GCS backend and that optimization isn't implemented for GCS yet. The issue also discusses the challenges of that optimization and the robust fallback behavior.
Now, based on this comment:

But this cache only works for the current process, so it is only used when you do a run-all.

Originally posted by @yorinasub17 in https://github.com/gruntwork-io/terragrunt/issues/2119#issuecomment-1135291831
I wonder if not a "naive" but more robust implementation could (persistently) cache the terraform configuration that's used to get the terraform output of the dependency configuration? This way terragrunt would still use terraform output to fetch the dependency but can avoid expensive terraform init calls on that dependency configuration.
This optimization could easily shave of 10s and up to 60s of "init" time from some of my more complex terragrunt configurations and yield a huge boost for interactive development.

We are experiencing a similar "lag" when running plans on modules with 4-5 dependency blocks, and we're using AWS s3.
More specifically, planning with mock_ouptuts and skip_outputs flag on each dependency block results in a 26s plan, while not using these leads to a 1min40secs plan.
That may not sound a lot but often you're trying to iterate on the main module (not on the dependencies) and this slow aspect builds up.
To mitigate this I had in mind a similar suggestion to @JohannesRudolph
More specifically, it would be great if there was an optional TG CLI flag for caching dependency outputs so they're available for subsequent plans.
In this way plan speeds could be improved significantly instead of constantly re-calculating ouputs that may never change while testing changes in the main module.

I have the exact same issue, terragrunt is taking a minute to run and iterating on it is frustrating 😢

@fenos maybe add a 👍 on this issue descr

@Ido-DY Could you please clarify whether you're suggesting you suspect this new terragrunt-fetch-dependency-output-from-state is related to OP's original issue, or if it is a new non-causal feature that you think could resolve the slow Terragrunt commands from a different angle?

@stevenpitts If I got that right, the issue is all about the slowness of Terragrunt mostly when handling dependencies. I actually investigated it myself before opening #2119, and I found out that it actually terraform init and terraform output -json that takes 99% of the time, then I compared the output of the Terraform command with the output stored within the state file and confirmed it's identical. At this point I wonder why it takes Terraform more than 10 seconds to generate the exact output that can be fetched from the state directly with a few milliseconds, and I'm still not sure. Caching the Terragrunt configuration is constructed from a few components:

The wrapping configuration (such as the backend and credentials preparations)
The Terraform plugins
The output of packages

The 1st component is generated super fast and there is no point to my mind to cache it. The seconed one can be cached using TF_CACHE_DIR but it'll lead to other issues when having concurrent executions.
And the last one is cached by default only within the memory of the same run. Caching the last layer locally without knowing if there was another execution on the same resource is not safe.
It takes a few seconds for Terragrunt to run on a single component and most of the time is spent on the Terraform commands themselves, however when resolving dependencies throughout Terraform commands this time usually multiplied.
Is this answer your question?

I think this makes sense, thank you!
It's felt like the time it takes to "start up" Terragrunt (number 3 in your post) had increased recently, which was the reason I originally searched for this issue, but that may just be my imagination or something unrelated.
But doesn't it make sense that generating output of all dependent modules would take longer than fetching the output from state, since it involves many provider API calls, rather than a single call to the authoritative state file (local or remote)?
I get the sense that this is a different conversation, and I don't want to derail this issue, so I'll leave it as something I don't quite understand yet about the Terragrunt flow :grin:
Thanks for the clarification!

Thanks everyone who joined in on the discussion so far. I'll summarize my understanding of the issue and will try to clarify my original enhancement suggestion.
The key problem is that when terragrunt encounters a dependency it needs to resolve the output of the dependency terraform configuration. The correct approach is to perform a terraform init -get=false (we don't need modules, only providers) followed by terraform output -json on the dependency terraform configuration.
Since this is slow, various forms of caching have been proposed here. Any approach needs to maintain correctness though. The solutions and their respective limitations are

Bypassing terraform alltogether and fetching state directly from the state backend via terragrunt-fetch-dependency-output-from-state cli flag. Only supported for S3.
Speed up terraform init using TF_CACHE_DIR. This is a terraform feature, but is not safe with concurrent execution and can cause issues with lockfiles
Cache the initialized dependency terraform configuration inside the .terragrunt-cache dir, speeding up terraform init because the configuration is already initialized (except on the first run). This could use the same terragrunt behavior already present for "auto init".
Cache the output of terraform output -json. This is already done inside a single execution of terragrunt run-all and helps speeding up executions with common shared dependency configurations.

@Ido-DY suggested extending 1) to GCS. While that's possible, my next feature request would be to extend that to Azure as well as we have some state backends there. I expect sooner or later there will be further requests to add first-party support in terragrunt for more state backends.
Another variant is extending 4) to persist the cache between different terragrunt run-all configurations and somehow maintain correctness.
My personal favorite would be 3) as this builds on terragrunt behavior that is already there i.e. maintaining the initialized terraform configuration inside a .terragrunt-cache dir and auto-init. While this will still incur the cost of running terraform output -json, it will save the terraform init step which is super expensive without TF_CACHE_DIR.