[world rebuild|toolchain] Fix try compile handling

while playing with qtbase on x64-uwp i found that it could not pass a try_compile command since important variables are not correctly forwarded. As such I concluded that the handling of try_compile in vcpkg is completely wrong.
(just learned qt6 does not support uwp. Nevertheless fixing try_compile is still and issue. )

/azp run

@ras0219-msft Could you please help review this PR?

Tagged reviewed so other members will review this changes (doesn't means approved).

Tagged reviewed so other members will review this changes (doesn't means approved).

As soon as CMake 3.24 gets released parts of this PR can be dropped. I'll update CMake in vcpkg then and after that adjust this PR to the newer CMake. (Meaning the try_compile override isn't necessary any more with 3.24)

Considering upstream CMake's policy on overriding functions (undocumented, undefined behavior, only use for debugging, don't rely on it in production), we would very strongly like to avoid adding any more overrides and to do as much as we can to reduce the overrides we have -- especially for core functions like try_compile(). It's great to hear that 3.24 avoids that so I'd like to wait for the new version before merging.

will draft the PR until cmake 3.24 is out

CMake 3.24 is out 🙂

@dg0yt Which log do i actually need to look at for the cmake-user failure?

stdout says:
-- Running tests with CMake 3.7.2 for 'release'
CMake Error at scripts/cmake/vcpkg_execute_required_process.cmake:96 (message):
    Command failed: D:/buildtrees/cmake-user/cmake-3.7.2-win32-x86/3.7.2-61b52342c7.clean/bin/cmake C:/a/1/s/scripts/test_ports/cmake-user/project -G Ninja "-DCMAKE_MAKE_PROGRAM=C:/Program Files/Microsoft Visual Studio/2022/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/Ninja/ninja.exe" -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_TOOLCHAIN_FILE=C:/a/1/s/scripts/buildsystems/vcpkg.cmake -DVCPKG_CHAINLOAD_TOOLCHAIN_FILE=C:/a/1/s/scripts/toolchains/windows.cmake -DVCPKG_TARGET_ARCHITECTURE=x64 -DVCPKG_TARGET_TRIPLET=x64-windows -DVCPKG_CRT_LINKAGE=dynamic -DVCPKG_HOST_TRIPLET=x64-windows -DVCPKG_INSTALLED_DIR=D:/installed -DCMAKE_INSTALL_PREFIX=D:/buildtrees/cmake-user/x64-windows-3.7.2-release/install -DVCPKG_MANIFEST_MODE=OFF -DCHECK_CMAKE_VERSION=3.7.2 -DBUILD_SHARED_LIBS=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_PREFIX_PATH=SYSTEM_LIBS -DVCPKG_PREFER_SYSTEM_LIBS=OFF
    Working Directory: D:/buildtrees/cmake-user/x64-windows-3.7.2-release
    Error code: 1
    See logs for more information:
      D:\buildtrees\cmake-user\x64-windows-3.7.2-release-config-out.log
      D:\buildtrees\cmake-user\x64-windows-3.7.2-release-config-err.log

and the error log says:
CMake Error at C:/a/1/s/scripts/toolchains/windows.cmake:5 (cmake_policy):
  Policy "CMP0067" is not known to this version of CMake.
Call Stack (most recent call first):
  C:/a/1/s/scripts/buildsystems/vcpkg.cmake:206 (include)
  D:/buildtrees/cmake-user/cmake-3.7.2-win32-x86/3.7.2-61b52342c7.clean/share/cmake-3.7/Modules/CMakeDetermineSystem.cmake:88 (include)
  CMakeLists.txt:2 (project)

CMake Error at C:/a/1/s/scripts/toolchains/windows.cmake:6 (cmake_policy):
  Policy "CMP0137" is not known to this version of CMake.
Call Stack (most recent call first):
  C:/a/1/s/scripts/buildsystems/vcpkg.cmake:206 (include)
  D:/buildtrees/cmake-user/cmake-3.7.2-win32-x86/3.7.2-61b52342c7.clean/share/cmake-3.7/Modules/CMakeDetermineSystem.cmake:88 (include)
  CMakeLists.txt:2 (project)

CMake Error at C:/a/1/s/scripts/toolchains/windows.cmake:5 (cmake_policy):
  Policy "CMP0067" is not known to this version of CMake.
Call Stack (most recent call first):
  C:/a/1/s/scripts/buildsystems/vcpkg.cmake:206 (include)
  D:/buildtrees/cmake-user/x64-windows-3.7.2-release/CMakeFiles/3.7.2/CMakeSystem.cmake:6 (include)
  D:/buildtrees/cmake-user/x64-windows-3.7.2-release/CMakeFiles/CMakeTmp/CMakeLists.txt:2 (project)

CMake Error at C:/a/1/s/scripts/toolchains/windows.cmake:6 (cmake_policy):
  Policy "CMP0137" is not known to this version of CMake.
Call Stack (most recent call first):
  C:/a/1/s/scripts/buildsystems/vcpkg.cmake:206 (include)
  D:/buildtrees/cmake-user/x64-windows-3.7.2-release/CMakeFiles/3.7.2/CMakeSystem.cmake:6 (include)
  D:/buildtrees/cmake-user/x64-windows-3.7.2-release/CMakeFiles/CMakeTmp/CMakeLists.txt:2 (project)

CMake Error: Internal CMake error, TryCompile configure of cmake failed
CMake Error at D:/buildtrees/cmake-user/cmake-3.7.2-win32-x86/3.7.2-61b52342c7.clean/share/cmake-3.7/Modules/CMakeTestCCompiler.cmake:51 (message):
  The C compiler "C:/Program Files/Microsoft Visual
  Studio/2022/Enterprise/VC/Tools/MSVC/14.33.31629/bin/Hostx64/x64/cl.exe" is
  not able to compile a simple test program.

  It fails with the following output:

  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  CMakeLists.txt:2 (project)

@dg0yt: yeah the error is that the check is trying to also check windows.cmake for 3.7.2 compat which is unnecessary since that is completely internal.

@dg0yt: yeah the error is that the check is trying to also check windows.cmake for 3.7.2 compat which is unnecessary since that is completely internal.

How can you say it is internal? It is the toolchain used by users, externally. That's the point of this test.

How can you say it is internal? It is the toolchain used by users, externally. That's the point of this test.

vcpkg.cmake is external; windows.cmake is not. The error is because i set policies in windows.cmake. The test is automatically using windows.cmake as VCPKG_CHAINLOAD_TOOLCHAIN_FILE which is not needed to be supported by vcpkg with a lower cmake version.

The test is automatically using windows.cmake as VCPKG_CHAINLOAD_TOOLCHAIN_FILE which is not needed to be supported by vcpkg with a lower cmake version.

It is supported and tested until now, even if YMMV depending on the ports of interest. Your PR is about to change it.
(I'm not defending CMake 3.7 support for windows. I'm just reflecting my understanding of documented/expected behaviour.)

@dg0yt About the CMake test. It has to ideally set VCPKG_CHAINLOAD_TOOLCHAIN_FILE to a dummy toolchain which is empty and not use the internal vcpkg toolchains.

@Cheney-W Why is this PR marked as: requires:author-response?

Just a reminder that there is a new comment here.

Tagging vcpkg-team-review for discussion on required CMake versions.

@ras0219-msft @dan-shaw @JavierMatosD and I discussed this today and we could make the argument to drop Debian 9.x (because it went out of support of the Debian project in January 2020 https://endoflife.software/operating-systems/linux/debian ). That would let us change the required cmake version to 3.10.2 (Ubuntu 18.04), but we aren't going to consider anything close to 3.24 at this time. Older copies of CMake are distributed with in support systems we care about like Ubuntu 18.x. Even if something is out of support, dropping it needs specific rationale in order to justify breaking existing customers.

Fetching modern CMake is like two lines lines for any Linux distro...
Does vcpkg's telemetry track what CMake versions people use?

@BillyONeal Why exactly were you discussing CMake versions in this PR?

@BillyONeal Why exactly were you discussing CMake versions in this PR?

Because that's what's going on in the discussion above about whether we can require that these policies be turned on.

Because that's what's going on in the discussion above about whether we can require that these policies be turned on.

if(POLICY CMP0137)
    cmake_policy(SET CMP0137 NEW)
endif()

will work no matter the CMake version. If the policy doesn't yet exist it will just use old behavior.

/azp run

Sorry, needed to update the host of that mac VM

When this completes, I will update xbox.cmake to be consistent with these changes.

@JavierMatosD @BillyONeal @vicroms: I feel like this PR got forgotten which is why i ping.

This seems reasonable to me. I'm not sure how to verify that it is correct though. Can you show what a before/after of this change would imply?
I realize it's been a long time :)

Can you show what a before/after of this change would imply?

Test case added in https://github.com/microsoft/vcpkg/pull/24493/commits/7bcc75aa78ee17de02c1eb091a97322c485602b8. Port fails in current master.
Here is the diff of the CMakeCache generated for the try_compile command:
diff.txt

Shouldn't we be concerned about the extra '?

That is CMake automagically doing it. I assume it is due to the additional space at the end of the flags. Since flags without that space don't have the extra '

@BillyONeal arm windows CI hanging building GTK3 here?

@BillyONeal arm windows CI hanging building GTK3 here?

Yeah, I kicked it by pushing a merge

@BillyONeal arm windows CI hanging building GTK3 here?

Remember https://github.com/microsoft/vcpkg/pull/32327. Would have supplied logs. Might have enabled a fix.

Thanks for the fix and sorry you were in a holding pattern for 2 years

OK, so once I remove the property test and add the other elements to xbox.cmake that are in windows.cmake, how do I verify this is working as intended?

@walbourn

OK, so once I remove the property test and add the other elements to xbox.cmake that are in windows.cmake, how do I verify this is working as intended?

You can run the test port from this PR:
PS D:\vcpkg> .\vcpkg.exe install try-compile:x64-windows --overlay-ports scripts\test_ports --enforce-port-checks
Computing installation plan...
The following packages will be built and installed:
    try-compile:x64-windows@1 -- D:\vcpkg\scripts\test_ports\try-compile
Detecting compiler hash for triplet x64-windows...
Compiler found: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.40.33807/bin/Hostx64/x64/cl.exe
Restored 0 package(s) from C:\Users\bion\AppData\Local\vcpkg\archives in 245 us. Use --debug to see more details.
Installing 1/1<EMAIL_ADDRESS>Building<EMAIL_ADDRESS>D:\vcpkg\scripts\test_ports\try-compile: info: installing overlay port from here
-- Configuring x64-windows
-- Skipping post-build validation due to VCPKG_POLICY_EMPTY_PACKAGE
Stored binaries in 1 destinations in 22.6 ms.
Elapsed time to handle try-compile:x64-windows: 1.6 s
try-compile:x64-windows package ABI: f0f78eaa272af4799922a190f78bc5bcdb8d2396051fb83abbefe0c3b8989f8f
Total install time: 1.6 s
PS D:\vcpkg> .\vcpkg.exe install try-compile:x64-xbox-scarlett --overlay-ports scripts\test_ports --enforce-port-checks
Computing installation plan...
The following packages will be built and installed:
    try-compile:x64-xbox-scarlett@1 -- D:\vcpkg\scripts\test_ports\try-compile
Detecting compiler hash for triplet x64-xbox-scarlett...
Compiler found: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.40.33807/bin/Hostx64/x64/cl.exe
Restored 0 package(s) from C:\Users\bion\AppData\Local\vcpkg\archives in 217 us. Use --debug to see more details.
Installing 1/1<EMAIL_ADDRESS>Building<EMAIL_ADDRESS>D:\vcpkg\triplets\community\x64-xbox-scarlett.cmake: info: loaded community triplet from here. Community triplets are not built in the curated registry and are thus less likely to succeed.
D:\vcpkg\scripts\test_ports\try-compile: info: installing overlay port from here
-- Configuring x64-xbox-scarlett
CMake Error at scripts/cmake/vcpkg_execute_required_process.cmake:112 (message):
    Command failed: D:/vcpkg-downloads/tools/ninja/1.10.2-windows/ninja.exe -v
    Working Directory: D:/vcpkg/buildtrees/try-compile/x64-xbox-scarlett-rel/vcpkg-parallel-configure
    Error code: 1
    See logs for more information:
      D:\vcpkg\buildtrees\try-compile\config-x64-xbox-scarlett-dbg-CMakeCache.txt.log
      D:\vcpkg\buildtrees\try-compile\config-x64-xbox-scarlett-rel-CMakeCache.txt.log
      D:\vcpkg\buildtrees\try-compile\config-x64-xbox-scarlett-dbg-CMakeConfigureLog.yaml.log
      D:\vcpkg\buildtrees\try-compile\config-x64-xbox-scarlett-rel-CMakeConfigureLog.yaml.log
      D:\vcpkg\buildtrees\try-compile\config-x64-xbox-scarlett-out.log

Call Stack (most recent call first):
  installed/x64-windows/share/vcpkg-cmake/vcpkg_cmake_configure.cmake:269 (vcpkg_execute_required_process)
  scripts/test_ports/try-compile/portfile.cmake:5 (vcpkg_cmake_configure)
  scripts/ports.cmake:191 (include)

error: building try-compile:x64-xbox-scarlett failed with: BUILD_FAILED
Elapsed time to handle try-compile:x64-xbox-scarlett: 1.1 s
Please ensure you're using the latest port files with `git pull` and `vcpkg update`.
Then check for known issues at:
  https://github.com/microsoft/vcpkg/issues?q=is%3Aissue+is%3Aopen+in%3Atitle+try-compile
You can submit a new issue at:
  https://github.com/microsoft/vcpkg/issues/new?title=[try-compile]+Build+error+on+x64-xbox-scarlett&body=Copy+issue+body+from+D%3A%2Fvcpkg%2Finstalled%2Fvcpkg%2Fissue_body.md

PS D:\vcpkg>

D:\vcpkg\buildtrees\try-compile\config-x64-xbox-scarlett-out.log says:
[1/2] "C:/Program Files/CMake/bin/cmake.exe" -E chdir ".." "C:/Program Files/CMake/bin/cmake.exe" "D:/vcpkg/scripts/test_ports/try-compile" "-G" "Ninja" "-DCMAKE_BUILD_TYPE=Release" "-DCMAKE_INSTALL_PREFIX=D:/vcpkg/packages/try-compile_x64-xbox-scarlett" "-DFETCHCONTENT_FULLY_DISCONNECTED=ON" "-DCMAKE_MAKE_PROGRAM=D:/vcpkg-downloads/tools/ninja/1.10.2-windows/ninja.exe" "-DXBOX_CONSOLE_TARGET=scarlett" "-DBUILD_SHARED_LIBS=ON" "-DVCPKG_CHAINLOAD_TOOLCHAIN_FILE=D:/vcpkg/scripts/toolchains/xbox.cmake" "-DVCPKG_TARGET_TRIPLET=x64-xbox-scarlett" "-DVCPKG_SET_CHARSET_FLAG=ON" "-DVCPKG_PLATFORM_TOOLSET=v143" "-DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON" "-DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=ON" "-DCMAKE_FIND_PACKAGE_NO_SYSTEM_PACKAGE_REGISTRY=ON" "-DCMAKE_INSTALL_SYSTEM_RUNTIME_LIBS_SKIP=TRUE" "-DCMAKE_VERBOSE_MAKEFILE=ON" "-DVCPKG_APPLOCAL_DEPS=OFF" "-DCMAKE_TOOLCHAIN_FILE=D:/vcpkg/scripts/buildsystems/vcpkg.cmake" "-DCMAKE_ERROR_ON_ABSOLUTE_INSTALL_DESTINATION=ON" "-DVCPKG_CXX_FLAGS= -DVCPKG_LET_ME_PASS" "-DVCPKG_CXX_FLAGS_RELEASE=" "-DVCPKG_CXX_FLAGS_DEBUG=" "-DVCPKG_C_FLAGS= -DVCPKG_LET_ME_PASS" "-DVCPKG_C_FLAGS_RELEASE=" "-DVCPKG_C_FLAGS_DEBUG=" "-DVCPKG_CRT_LINKAGE=dynamic" "-DVCPKG_LINKER_FLAGS=" "-DVCPKG_LINKER_FLAGS_RELEASE=" "-DVCPKG_LINKER_FLAGS_DEBUG=" "-DVCPKG_TARGET_ARCHITECTURE=x64" "-DCMAKE_INSTALL_LIBDIR:STRING=lib" "-DCMAKE_INSTALL_BINDIR:STRING=bin" "-D_VCPKG_ROOT_DIR=D:/vcpkg" "-D_VCPKG_INSTALLED_DIR=D:/vcpkg/installed" "-DVCPKG_MANIFEST_INSTALL=OFF"
FAILED: ../CMakeCache.txt
"C:/Program Files/CMake/bin/cmake.exe" -E chdir ".." "C:/Program Files/CMake/bin/cmake.exe" "D:/vcpkg/scripts/test_ports/try-compile" "-G" "Ninja" "-DCMAKE_BUILD_TYPE=Release" "-DCMAKE_INSTALL_PREFIX=D:/vcpkg/packages/try-compile_x64-xbox-scarlett" "-DFETCHCONTENT_FULLY_DISCONNECTED=ON" "-DCMAKE_MAKE_PROGRAM=D:/vcpkg-downloads/tools/ninja/1.10.2-windows/ninja.exe" "-DXBOX_CONSOLE_TARGET=scarlett" "-DBUILD_SHARED_LIBS=ON" "-DVCPKG_CHAINLOAD_TOOLCHAIN_FILE=D:/vcpkg/scripts/toolchains/xbox.cmake" "-DVCPKG_TARGET_TRIPLET=x64-xbox-scarlett" "-DVCPKG_SET_CHARSET_FLAG=ON" "-DVCPKG_PLATFORM_TOOLSET=v143" "-DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON" "-DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=ON" "-DCMAKE_FIND_PACKAGE_NO_SYSTEM_PACKAGE_REGISTRY=ON" "-DCMAKE_INSTALL_SYSTEM_RUNTIME_LIBS_SKIP=TRUE" "-DCMAKE_VERBOSE_MAKEFILE=ON" "-DVCPKG_APPLOCAL_DEPS=OFF" "-DCMAKE_TOOLCHAIN_FILE=D:/vcpkg/scripts/buildsystems/vcpkg.cmake" "-DCMAKE_ERROR_ON_ABSOLUTE_INSTALL_DESTINATION=ON" "-DVCPKG_CXX_FLAGS= -DVCPKG_LET_ME_PASS" "-DVCPKG_CXX_FLAGS_RELEASE=" "-DVCPKG_CXX_FLAGS_DEBUG=" "-DVCPKG_C_FLAGS= -DVCPKG_LET_ME_PASS" "-DVCPKG_C_FLAGS_RELEASE=" "-DVCPKG_C_FLAGS_DEBUG=" "-DVCPKG_CRT_LINKAGE=dynamic" "-DVCPKG_LINKER_FLAGS=" "-DVCPKG_LINKER_FLAGS_RELEASE=" "-DVCPKG_LINKER_FLAGS_DEBUG=" "-DVCPKG_TARGET_ARCHITECTURE=x64" "-DCMAKE_INSTALL_LIBDIR:STRING=lib" "-DCMAKE_INSTALL_BINDIR:STRING=bin" "-D_VCPKG_ROOT_DIR=D:/vcpkg" "-D_VCPKG_INSTALLED_DIR=D:/vcpkg/installed" "-DVCPKG_MANIFEST_INSTALL=OFF"
-- The C compiler identification is MSVC 19.40.33811.0
-- The CXX compiler identification is MSVC 19.40.33811.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.40.33807/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.40.33807/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Error at CMakeLists.txt:8 (message):
  Compiler flags are not passed consistently to try_compile!

-- Configuring incomplete, errors occurred!
[2/2] "C:/Program Files/CMake/bin/cmake.exe" -E chdir "../../x64-xbox-scarlett-dbg" "C:/Program Files/CMake/bin/cmake.exe" "D:/vcpkg/scripts/test_ports/try-compile" "-G" "Ninja" "-DCMAKE_BUILD_TYPE=Debug" "-DCMAKE_INSTALL_PREFIX=D:/vcpkg/packages/try-compile_x64-xbox-scarlett/debug" "-DFETCHCONTENT_FULLY_DISCONNECTED=ON" "-DCMAKE_MAKE_PROGRAM=D:/vcpkg-downloads/tools/ninja/1.10.2-windows/ninja.exe" "-DXBOX_CONSOLE_TARGET=scarlett" "-DBUILD_SHARED_LIBS=ON" "-DVCPKG_CHAINLOAD_TOOLCHAIN_FILE=D:/vcpkg/scripts/toolchains/xbox.cmake" "-DVCPKG_TARGET_TRIPLET=x64-xbox-scarlett" "-DVCPKG_SET_CHARSET_FLAG=ON" "-DVCPKG_PLATFORM_TOOLSET=v143" "-DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON" "-DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=ON" "-DCMAKE_FIND_PACKAGE_NO_SYSTEM_PACKAGE_REGISTRY=ON" "-DCMAKE_INSTALL_SYSTEM_RUNTIME_LIBS_SKIP=TRUE" "-DCMAKE_VERBOSE_MAKEFILE=ON" "-DVCPKG_APPLOCAL_DEPS=OFF" "-DCMAKE_TOOLCHAIN_FILE=D:/vcpkg/scripts/buildsystems/vcpkg.cmake" "-DCMAKE_ERROR_ON_ABSOLUTE_INSTALL_DESTINATION=ON" "-DVCPKG_CXX_FLAGS= -DVCPKG_LET_ME_PASS" "-DVCPKG_CXX_FLAGS_RELEASE=" "-DVCPKG_CXX_FLAGS_DEBUG=" "-DVCPKG_C_FLAGS= -DVCPKG_LET_ME_PASS" "-DVCPKG_C_FLAGS_RELEASE=" "-DVCPKG_C_FLAGS_DEBUG=" "-DVCPKG_CRT_LINKAGE=dynamic" "-DVCPKG_LINKER_FLAGS=" "-DVCPKG_LINKER_FLAGS_RELEASE=" "-DVCPKG_LINKER_FLAGS_DEBUG=" "-DVCPKG_TARGET_ARCHITECTURE=x64" "-DCMAKE_INSTALL_LIBDIR:STRING=lib" "-DCMAKE_INSTALL_BINDIR:STRING=bin" "-D_VCPKG_ROOT_DIR=D:/vcpkg" "-D_VCPKG_INSTALLED_DIR=D:/vcpkg/installed" "-DVCPKG_MANIFEST_INSTALL=OFF"
FAILED: ../../x64-xbox-scarlett-dbg/CMakeCache.txt
"C:/Program Files/CMake/bin/cmake.exe" -E chdir "../../x64-xbox-scarlett-dbg" "C:/Program Files/CMake/bin/cmake.exe" "D:/vcpkg/scripts/test_ports/try-compile" "-G" "Ninja" "-DCMAKE_BUILD_TYPE=Debug" "-DCMAKE_INSTALL_PREFIX=D:/vcpkg/packages/try-compile_x64-xbox-scarlett/debug" "-DFETCHCONTENT_FULLY_DISCONNECTED=ON" "-DCMAKE_MAKE_PROGRAM=D:/vcpkg-downloads/tools/ninja/1.10.2-windows/ninja.exe" "-DXBOX_CONSOLE_TARGET=scarlett" "-DBUILD_SHARED_LIBS=ON" "-DVCPKG_CHAINLOAD_TOOLCHAIN_FILE=D:/vcpkg/scripts/toolchains/xbox.cmake" "-DVCPKG_TARGET_TRIPLET=x64-xbox-scarlett" "-DVCPKG_SET_CHARSET_FLAG=ON" "-DVCPKG_PLATFORM_TOOLSET=v143" "-DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON" "-DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=ON" "-DCMAKE_FIND_PACKAGE_NO_SYSTEM_PACKAGE_REGISTRY=ON" "-DCMAKE_INSTALL_SYSTEM_RUNTIME_LIBS_SKIP=TRUE" "-DCMAKE_VERBOSE_MAKEFILE=ON" "-DVCPKG_APPLOCAL_DEPS=OFF" "-DCMAKE_TOOLCHAIN_FILE=D:/vcpkg/scripts/buildsystems/vcpkg.cmake" "-DCMAKE_ERROR_ON_ABSOLUTE_INSTALL_DESTINATION=ON" "-DVCPKG_CXX_FLAGS= -DVCPKG_LET_ME_PASS" "-DVCPKG_CXX_FLAGS_RELEASE=" "-DVCPKG_CXX_FLAGS_DEBUG=" "-DVCPKG_C_FLAGS= -DVCPKG_LET_ME_PASS" "-DVCPKG_C_FLAGS_RELEASE=" "-DVCPKG_C_FLAGS_DEBUG=" "-DVCPKG_CRT_LINKAGE=dynamic" "-DVCPKG_LINKER_FLAGS=" "-DVCPKG_LINKER_FLAGS_RELEASE=" "-DVCPKG_LINKER_FLAGS_DEBUG=" "-DVCPKG_TARGET_ARCHITECTURE=x64" "-DCMAKE_INSTALL_LIBDIR:STRING=lib" "-DCMAKE_INSTALL_BINDIR:STRING=bin" "-D_VCPKG_ROOT_DIR=D:/vcpkg" "-D_VCPKG_INSTALLED_DIR=D:/vcpkg/installed" "-DVCPKG_MANIFEST_INSTALL=OFF"
-- The C compiler identification is MSVC 19.40.33811.0
-- The CXX compiler identification is MSVC 19.40.33811.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.40.33807/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.40.33807/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Error at CMakeLists.txt:8 (message):
  Compiler flags are not passed consistently to try_compile!

-- Configuring incomplete, errors occurred!
ninja: build stopped: subcommand failed.

Thanks!
The updated xbox triplet toolchain file is in this PR: https://github.com/microsoft/vcpkg/pull/39654


FtpTrace.LogToConsole ignored?

How do I stop the client from filling my Console with verbose information?
I've tried all the below in every combination :(
        FtpTrace.LogToFile = null;
        FtpTrace.LogFunctions = false;
        FtpTrace.AddListener(new TextWriterTraceListener("log_file.txt") {
            Filter = new EventTypeFilter(SourceLevels.Error)
        });
        FtpTrace.LogToConsole = false;

Show the logs and specify what part you find verbose?

LogToConsole is honored at this line - https://github.com/robinrodricks/FluentFTP/blob/master/FluentFTP/Helpers/FtpTrace.cs#L189

Thank you very much. I needed to set the constant EnableTracing to false. Thank you for your great work on this library.

You are most welcome.


Got ValueError: Both labels and logits must be provided while both labels and logits have been provided.

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): v1.2.0-5-g435cdfc 1.2.1
Python version:  3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 13:51:32)  GCC 4.4.7
Bazel version (if compiling from source):
CUDA/cuDNN version: 5.1
GPU model and memory: GTX-1050 4GB

Describe the problem
Got ValueError: Both labels and logits must be provided while both labels and logits have been provided.
Source code / logs
# Softmax example in TF using the classical Iris dataset
# Download iris.data from https://archive.ics.uci.edu/ml/datasets/Iris

import tensorflow as tf
print(tf.__version__)
import os

def combine_inputs(X):
    res =  tf.matmul(X, W) + b
    res = tf.identity(res, name="linear_out")

def inference(X):
    return tf.nn.softmax(combine_inputs(X), "softmax_out")

def loss(X, Y):
    return tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=Y, logits=combine_inputs(X), name="softmax_entropy"), name="loss")

def read_csv(batch_size, file_name, record_defaults):
    filename_queue = tf.train.string_input_producer([os.path.dirname(__file__) + "/" + file_name])
    reader = tf.TextLineReader(skip_header_lines=1)
    key, value = reader.read(filename_queue)
    # decode_csv will convert a Tensor from type string (the text line) in
    # a tuple of tensor columns with the specified defaults, which also
    # sets the data type for each column
    decoded = tf.decode_csv(value, record_defaults=record_defaults)
    # batch actually reads the file and loads "batch_size" rows in a single tensor
    return tf.train.shuffle_batch(decoded,
                                  batch_size=batch_size,
                                  capacity=batch_size * 50,
                                  min_after_dequeue=batch_size)

def inputs():
    sepal_length, sepal_width, petal_length, petal_width, label =\
        read_csv(100, "iris.csv", [[0.0], [0.0], [0.0], [0.0], [""]])
    # convert class names to a 0 based class index.
    label_number = tf.to_int32(tf.argmax(tf.to_int32(tf.stack([
        tf.equal(label, ["Iris-setosa"]),
        tf.equal(label, ["Iris-versicolor"]),
        tf.equal(label, ["Iris-virginica"])
    ])), 0))
    # Pack all the features that we care about in a single matrix;
    # We then transpose to have a matrix with one example per row and one feature per column.
    features = tf.transpose(tf.stack([sepal_length, sepal_width, petal_length, petal_width]))
    return features, label_number

def train(total_loss):
    learning_rate = 0.01
    return tf.train.GradientDescentOptimizer(learning_rate).minimize(total_loss)

def evaluate(sess, X, Y):
    predicted = tf.cast(tf.arg_max(inference(X), 1), tf.int32)
    print(sess.run(tf.reduce_mean(tf.cast(tf.equal(predicted, Y), tf.float32))))

# Launch the graph in a session, setup boilerplate
with tf.Graph().as_default():
    with tf.Session() as sess:
        # this time weights form a matrix, not a column vector, one "weight vector" per class.
        W = tf.Variable(tf.zeros([4, 3]), name="weights")
        # so do the biases, one per class.
        b = tf.Variable(tf.zeros([3], name="bias"))
        tf.global_variables_initializer().run()
        X, Y = inputs()
        total_loss = loss(X, Y)
        train_op = train(total_loss)
        summary_out = os.path.join(os.path.dirname(__file__), "tf_summary")
        import shutil
        shutil.rmtree(summary_out)
        writer = tf.summary.FileWriter(summary_out, graph=sess.graph)
        coord = tf.train.Coordinator()
        threads = tf.train.start_queue_runners(sess=sess, coord=coord)
        # actual training loop
        training_steps = 1000
        for step in range(training_steps):
            sess.run([train_op])
            # for debugging and learning purposes, see how the loss gets decremented thru training steps
            if step % 10 == 0:
                print("loss: ", sess.run([total_loss]))
        evaluate(sess, X, Y)
        coord.request_stop()
        coord.join(threads)
        writer.close()

Error:
/home/kaiyin/miniconda3/envs/tf/bin/python3.6 /home/kaiyin/PycharmProjects/tensorflow-mnist/learntf_008_softmax.py
2017-08-01 15:12:57.469597: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-08-01 15:12:57.469610: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-08-01 15:12:57.469613: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-08-01 15:12:57.469616: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-08-01 15:12:57.469618: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2017-08-01 15:12:57.554651: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:893] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2017-08-01 15:12:57.554839: I tensorflow/core/common_runtime/gpu/gpu_device.cc:940] Found device 0 with properties:
name: GeForce GTX 1050
major: 6 minor: 1 memoryClockRate (GHz) 1.493
pciBusID 0000:01:00.0
Total memory: 3.95GiB
Free memory: 1.97GiB
2017-08-01 15:12:57.554848: I tensorflow/core/common_runtime/gpu/gpu_device.cc:961] DMA: 0
2017-08-01 15:12:57.554852: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0:   Y
2017-08-01 15:12:57.554859: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1050, pci bus id: 0000:01:00.0)
Traceback (most recent call last):
  File "/home/kaiyin/PycharmProjects/tensorflow-mnist/learntf_008_softmax.py", line 69, in <module>
    total_loss = loss(X, Y)
  File "/home/kaiyin/PycharmProjects/tensorflow-mnist/learntf_008_softmax.py", line 17, in loss
    return tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(None, Y, combine_inputs(X)), name="loss")
  File "/home/kaiyin/miniconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 1555, in softmax_cross_entropy_with_logits
    labels, logits)
  File "/home/kaiyin/miniconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 1512, in _ensure_xent_args
    raise ValueError("Both labels and logits must be provided.")
ValueError: Both labels and logits must be provided.

The combine_inputs function does not return any value.
This sort of question is better asked on stackoverflow.

TMKB You could answer the question


[Spark-Doris-Connector][Bug-Fix] Resolve deserialize exception when Spark Doris Connector in aync deserialize mode

Proposed changes
fix #5335
Types of changes
What types of changes does your code introduce to Doris?
Put an x in the boxes that apply

[x] Bugfix (non-breaking change which fixes an issue)
[] New feature (non-breaking change which adds functionality)
[] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[] Documentation Update (if none of the other choices apply)
[] Code refactor (Modify the code structure, format the code, etc...)

Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

[] I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail
[x] Compiling and unit tests pass locally with my changes
[] I have added tests that prove my fix is effective or that my feature works
[] If these changes need document changes, I have updated the document
[] Any dependent changes have been merged

Further comments
None

Gentle ping @wuyunfeng

@924060929 This is a nice capture. I am curious about whether  the lock is necessary when running in sync mode?

@924060929 This is a nice capture. I am curious about whether the lock is necessary when running in sync mode （default mode）?

Unnecessary, just for simplify. Synchronized has little effect on performance in sync mode.
Shall we remove synchronized in sync mode?

@924060929 I suggest that there can be such an optimization

@wuyunfeng OK, I will add this optimization soon

@924060929 I suggest that there can be such an optimization

@wuyunfeng done

I just using Java describe the implementation which I would prefer:
First define a NoOpLock:
protected static final class NoOpLock implements Lock {

        @Override
        public void lock() {
        }

        @Override
        public void lockInterruptibly() throws InterruptedException {
        }

        @Override
        public boolean tryLock() {
            return true;
        }

        @Override
        public boolean tryLock(long time, TimeUnit unit) throws InterruptedException {
            return true;
        }

        @Override
        public void unlock() {
        }

        @Override
        public Condition newCondition() {
            throw new UnsupportedOperationException("NoOpLock can't provide a condition");
        }
    }

We just define a private final Lock lock = new NoOpLock();
If using async mode,  let the lock using the real Lock, If not, we can replace the lock with NoOpLock.
If we do this, the first code is more elegant, and second, we can avoid excessive branch switching.
I think your approach is fine, would you like to think about my suggestion

@wuyunfeng OK, first I think the same solution as you note, but don't know where the place the NoOpLock, we can add a inner class, or add some public class in any package, or any third-party package has same implement?
So I add the if condition without Lock implement.
Can you give more advice?

I think you can just use a inner class

I think you can just use a inner class

done

@Youngwb already fix bug which use deserializeArrowToRowBatchAsync before initialize

@924060929 I have merge this into master. Thanks


feature: change constant name style

Which PikiwiDB functionalities are relevant/related to the feature request?
No response
Description
Which PikiwiDB functionalities are relevant/related to the feature request?
No response
Description
统一 define 宏、const 常量、enum 常量 的命名方式：kCmdOK
Proposed solution
none
Alternatives considered
none

ok


Error when compiling rabbit reorder

(CLAP) root@03d38d594625:~/CLAP-triangle-counting/reorder_preprocess/rabbit_order/src# make
g++     -MD -fopenmp -std=c++14 -O3 -I/root/miniconda3/envs/CLAP/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/data.h -o reorder reorder.cpp -ltcmalloc_minimal -lnuma
cc1plus: warning: /root/miniconda3/envs/CLAP/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/data.h: not a directory
reorder.cpp:10:10: fatal error: torch/data.h: No such file or directory
10 | #include <torch/data.h>
|          ^~~~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:14: reorder] Error 1
I have included the path to the data.h in the make file but the error persists

It seems to be a environment issue related to torch. Could you provide the system version, torch version, cuda version (if exists) that you are using?

You do not have to run make. Did you try follow the instruction in readme to run the script in correct path?

I followed the instructions and I ran the setup.py
This is the error that  I get (base) root@03d38d594625:~/CLAP-triangle-counting/reorder_preprocess/rabbit_order/src# python setup.py build
running build
running build_ext
building 'rabbit' extension
creating /root/CLAP-triangle-counting/reorder_preprocess/rabbit_order/src/build/temp.linux-aarch64-cpython-311
Emitting ninja build file /root/CLAP-triangle-counting/reorder_preprocess/rabbit_order/src/build/temp.linux-aarch64-cpython-311/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/1] c++ -MMD -MF /root/CLAP-triangle-counting/reorder_preprocess/rabbit_order/src/build/temp.linux-aarch64-cpython-311/reorder.o.d -pthread -B /root/miniconda3/compiler_compat -Wsign-compare -DNDEBUG -fwrapv -O3 -Wall -fPIC -O3 -isystem /root/miniconda3/include -fPIC -O3 -isystem /root/miniconda3/include -fPIC -I/root/miniconda3/lib/python3.11/site-packages/torch/include -I/root/miniconda3/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/root/miniconda3/lib/python3.11/site-packages/torch/include/TH -I/root/miniconda3/lib/python3.11/site-packages/torch/include/THC -I/root/miniconda3/include/python3.11 -c -c /root/CLAP-triangle-counting/reorder_preprocess/rabbit_order/src/reorder.cpp -o /root/CLAP-triangle-counting/reorder_preprocess/rabbit_order/src/build/temp.linux-aarch64-cpython-311/reorder.o -O3 -fopenmp -mcx16 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1016"' -DTORCH_EXTENSION_NAME=rabbit -D_GLIBCXX_USE_CXX11_ABI=1 -std=gnu++14
FAILED: /root/CLAP-triangle-counting/reorder_preprocess/rabbit_order/src/build/temp.linux-aarch64-cpython-311/reorder.o
c++ -MMD -MF /root/CLAP-triangle-counting/reorder_preprocess/rabbit_order/src/build/temp.linux-aarch64-cpython-311/reorder.o.d -pthread -B /root/miniconda3/compiler_compat -Wsign-compare -DNDEBUG -fwrapv -O3 -Wall -fPIC -O3 -isystem /root/miniconda3/include -fPIC -O3 -isystem /root/miniconda3/include -fPIC -I/root/miniconda3/lib/python3.11/site-packages/torch/include -I/root/miniconda3/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/root/miniconda3/lib/python3.11/site-packages/torch/include/TH -I/root/miniconda3/lib/python3.11/site-packages/torch/include/THC -I/root/miniconda3/include/python3.11 -c -c /root/CLAP-triangle-counting/reorder_preprocess/rabbit_order/src/reorder.cpp -o /root/CLAP-triangle-counting/reorder_preprocess/rabbit_order/src/build/temp.linux-aarch64-cpython-311/reorder.o -O3 -fopenmp -mcx16 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1016"' -DTORCH_EXTENSION_NAME=rabbit -D_GLIBCXX_USE_CXX11_ABI=1 -std=gnu++14
c++: error: unrecognized command-line option ‘-mcx16’
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "/root/miniconda3/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1905, in _run_ninja_build
subprocess.run(
File "/root/miniconda3/lib/python3.11/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/root/CLAP-triangle-counting/reorder_preprocess/rabbit_order/src/setup.py", line 7, in
setup(
File "/root/miniconda3/lib/python3.11/site-packages/setuptools/init.py", line 107, in setup
return distutils.core.setup(**attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
return run_commands(dist)
^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
dist.run_commands()
File "/root/miniconda3/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
self.run_command(cmd)
File "/root/miniconda3/lib/python3.11/site-packages/setuptools/dist.py", line 1234, in run_command
super().run_command(command)
File "/root/miniconda3/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/root/miniconda3/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 131, in run
self.run_command(cmd_name)
File "/root/miniconda3/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "/root/miniconda3/lib/python3.11/site-packages/setuptools/dist.py", line 1234, in run_command
super().run_command(command)
File "/root/miniconda3/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/root/miniconda3/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 84, in run
_build_ext.run(self)
File "/root/miniconda3/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
self.build_extensions()
File "/root/miniconda3/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 848, in build_extensions
build_ext.build_extensions(self)
File "/root/miniconda3/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
self._build_extensions_serial()
File "/root/miniconda3/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
self.build_extension(ext)
File "/root/miniconda3/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 246, in build_extension
_build_ext.build_extension(self, ext)
File "/root/miniconda3/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 548, in build_extension
objects = self.compiler.compile(
^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 663, in unix_wrap_ninja_compile
_write_ninja_file_and_compile_objects(
File "/root/miniconda3/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1578, in _write_ninja_file_and_compile_objects
_run_ninja_build(
File "/root/miniconda3/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1921, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error compiling objects for extension

The problem seems to be caused by c++: error: unrecognized command-line option ‘-mcx16'.
Are you running the code on a x86 architecture 64-bit machine? Could you try running on another machine that support CMPXCHG16b? I find this that might be helpful.

I am running on a mac with clang compiler. I will try running on a windows machine that supports CMPXCHG16b and get back

While it's not our recommendation to run the code on a Windows machine, if feasible, we suggest considering a Linux environment. Our code has been thoroughly tested on Ubuntu 20.04 with the gcc 9.4.0 compiler. Using a Linux setup will likely to get program run. Thank you for your understanding!

Oh okay, I will try on an ubuntu machine and get back. Thanks

Hi, we wanted to check if there have been any updates regarding the issue of running it on an Ubuntu machine. For the time being, we plan to close this issue, as it seems we have done all we can for now. However, please feel free to re-open it if any new related issues arise in the future.

Thanks so much for the help
On Fri, Jan 5, 2024 at 2:50 AM Tianyu Fu @.***> wrote:

Hi, we wanted to check if there have been any updates regarding the issue
of running it on an Ubuntu machine. For the time being, we plan to close
this issue, as it seems we have done all we can for now. However, please
feel free to re-open it if any new related issues arise in the future.
—
Reply to this email directly, view it on GitHub
https://github.com/thu-nics/CLAP-triangle-counting/issues/5#issuecomment-1878262362,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BC44EU2RI3KCCKOM22RCXWTYM6WEBAVCNFSM6AAAAAA66PXSVSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZYGI3DEMZWGI
.
You are receiving this because you authored the thread.Message ID:
@.***>


Deleting submissions

Dear @ckcollab,
In the frame of the COMETH project, sensitive clinical datasets will be submitted use the bot-submission mode from the METEOR web interface.
As COMETH competition admins, we are committed to deleting (completely) these datasets after a given amount of time of time.
I would like to now if there is an easy way :
[1] to do it manually? If I go in the admin>submissions menu of the competition and click on the delete submission button, will that definitely delete the submission in the codabench storage cloud?

[2] to do it semi-automatically? For instance, erase all bot-submission, or all submissions from a particular user?
Thank you very mmuch

[1] In response to your question, yes. When you delete a submission the corresponding data file is deleted from storage.

[1] In response to your question, yes. When you delete a submission the corresponding data file is deleted from storage.

With our contract end, we didn't have time to add a multi delete feature, but I did add the ability to sort the columns on the table of submissions. Hopefully, that will make things easier for you.

With our contract end, we didn't have time to add a multi delete feature, but I did add the ability to sort the columns on the table of submissions. Hopefully, that will make things easier for you.

Perfect. Thank you!

Perfect. Thank you!

You can now select multiple submissions in the submission manager and delete or re-run them all through one button. You can also filter and use the select all feature in combination to make selecting many submissions easy.

You can now select multiple submissions in the submission manager and delete or re-run them all through one button. You can also filter and use the select all feature in combination to make selecting many submissions easy.

One relevant question. When we delete the competition from codalab website, will the corresponding data file (eval script, data, submission, etc.) deleted from the storage too?


Building into single .js

I'm having trouble building the build/package *.js files into a single .js file.  I have cloned the repo, ran npm install, tsd install, gulp compile and gulp packageFiles in it's root and it compiled perfectly.
What's the best entry point for the package, is it decorators/Validator.js?  I'm still trying to get my head around compiling, concatenating and minifying typescript projects and their dependancies so apologies for the novice question.  How do you use this in production?  I have the following in a model file:
import {Validator} from "validator.ts/Validator";
import {NotEmpty, IsInt} from "validator.ts/decorator/Validation";

And great work on the project by the way, thanks very much for sharing it.  I'm really looking forward to putting it to use.

Thanks,
Why are you trying to to create a single .js file? Its not supposed to work this way, e.g. there is not entry point of the package and not need of it. Simply import classes/decorators in your model as you wrote:
import {Validator} from "validator.ts/Validator";
import {NotEmpty, IsInt} from "validator.ts/decorator/Validation";

You should not pack anything into a single file. What you need is to run npm install validator.ts, tsd link and simply use this package. Did you look into samples ?

Side note: I think the .ts prefix is a bad idea. It might try to look for this as a actual script instead of a module.

@amcdnl its kinda similar to ".js" prefixes in javascript world. there are thousands of libraries with .js prefix, thats why I choose ts prefix. But I also don't like a lot such name. But its hard to come up with a good name, especially when most of names already busy on npm. But I think we can open a new issue with new naming suggestions

I have no problem with the name, its just having it called validator.ts feels like a file in my project vs a npm module.

package name changed.


Pycromanager and Micromagellan: ''MagellanAcquisition' object has no attribute '_acq"

I am trying to run an acquisition in Micromagellan using Pycromanager:
from pycromanager import Core, MagellanAcquisition, Magellan
core = Core()
core.set_config('Brightfield', 'OFF')
core.set_config('Objectives', '5x_SlideScanner')
core.set_config('SidePort', 'Camera')
core.set_config('Colors', 'Violet>Blue')
x, y = core.get_x_position(), core.get_y_position()
mag_api = Magellan()
new_grid = mag_api.create_grid('Grid', 13, 19, x, y)
acq_settings = mag_api.get_acquisition_settings(0)
acq_settings.set_saving_dir('C:/Users/Lab/Test')
acq_settings.set_use_channel('Violet>Blue', True)
acq = MagellanAcquisition(magellan_acq_index=0)
acq.acquire()
acq.await_completion()

I get this error
>>> acq = MagellanAcquisition(magellan_acq_index=0)
Traceback (most recent call last):
  File "C:\Users\Lab\anaconda3\envs\pycro\Lib\site-packages\pycromanager\acquisition\java_backend_acquisitions.py", line 293, in __init__
    args=[self._acq], port=self._port, new_socket=False,
          ^^^^^^^^^
AttributeError: 'MagellanAcquisition' object has no attribute '_acq'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Lab\anaconda3\envs\pycro\Lib\site-packages\pycromanager\acquisition\java_backend_acquisitions.py", line 769, in __init__
    super().__init__(**named_args)
  File "C:\Users\Lab\anaconda3\envs\pycro\Lib\site-packages\pycromanager\acquisition\java_backend_acquisitions.py", line 308, in __init__
    self._acq.start()
    ^^^^^^^^^
AttributeError: 'MagellanAcquisition' object has no attribute '_acq'

I've confirmed that this attribute is missing:
>>> hasattr(MagellanAcquisition,'_acq')
False
>>> MagellanAcquisition.__dict__
mappingproxy({'__module__': 'pycromanager.acquisition.java_backend_acquisitions', '__doc__': 'Class used for launching Micro-Magellan acquisitions. Must pass either magellan_acq_index\nor magellan_explore as an argument', '__init__': <function MagellanAcquisition.__init__ at 0x00000255B32F8900>, '_start_events': <function MagellanAcquisition._start_events at 0x00000255B32F94E0>, '_create_event_queue': <function MagellanAcquisition._create_event_queue at 0x00000255B32F9580>, '_create_remote_acquisition': <function MagellanAcquisition._create_remote_acquisition at 0x00000255B32F9620>, '__abstractmethods__': frozenset(), '_abc_impl': <_abc._abc_data object at 0x00000255B32F66C0>})
>>> dir(MagellanAcquisition)
['__abstractmethods__', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__enter__', '__eq__', '__exit__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_abc_impl', '_add_storage_monitor_fn', '_call_image_process_fn', '_check_for_exceptions', '_create_event_queue', '_create_remote_acquisition', '_initialize_hooks', '_initialize_image_processor', '_start_events', '_start_hook', '_start_notification_dispatcher', '_start_processor', '_start_receiving_notifications', 'abort', 'acquire', 'await_completion', 'get_dataset', 'get_viewer', 'mark_finished']

At one time I also got this error but this doesn't seem to be present since restarting the computer and microscope:
C:\Users\Lab\anaconda3\envs\pycro\Lib\site-packages\pycromanager\acquisition\java_backend_acquisitions.py:292: UserWarning: Could not create acquisition notification handler. Update Micro-Manager and Pyrcro-Manager to the latest versions to fix this. '
Pycromanager is fully upgraded using pip install pycromanager --upgrade.
Micromanager is the nightly build.
Do you have any ideas on how to fix this error?  Thanks!

Could you reformat those errors with with correct line breaks? Hard to read what is going on there

Sorry about that -- copy/pasted from my phone. Edited!

Should be fixed in https://github.com/micro-manager/pycro-manager/pull/779
You'll also need a new nightly build

Unfortunately I'm still getting the same issue.
>>> acq = MagellanAcquisition(magellan_acq_index=0)
Traceback (most recent call last):
  File "C:\Users\Lab\anaconda3\envs\pycro\lib\site-packages\pycromanager\acquisition\java_backend_acquisitions.py", line 293, in __init__
    args=[self._acq], port=self._port, new_socket=False,
AttributeError: 'MagellanAcquisition' object has no attribute '_acq'
C:\Users\Lab\anaconda3\envs\pycro\lib\site-packages\pycromanager\acquisition\java_backend_acquisitions.py:300: UserWarning: Could not create acquisition notification handler. Update Micro-Manager and Pyrcro-Manager to the latest versions to fix this
  warnings.warn('Could not create acquisition notification handler. '
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Lab\anaconda3\envs\pycro\lib\site-packages\pycromanager\acquisition\java_backend_acquisitions.py", line 769, in __init__
    super().__init__(**named_args)
  File "C:\Users\Lab\anaconda3\envs\pycro\lib\site-packages\pycromanager\acquisition\java_backend_acquisitions.py", line 308, in __init__
    self._acq.start()
AttributeError: 'MagellanAcquisition' object has no attribute '_acq'
>>>

Using the 20240623 nightly build of MicroManager (https://download.micro-manager.org/nightly/2.0/Windows/MMSetup_64bit_2.0.3_20240623.exe)
Upgraded pycromagellan within the 'pycro' Python environment using pip install pycromanager --upgrade
Steps I tried to attempt to fix the issue:

Completely uninstalled Anaconda and MicroManager, searched for any remaining installations or install files on the computer and removed them.
Reinstalled Anaconda3 (Anaconda3-2024.02-1-Windows-x86_64).
Reinstalled MicroManager using the 20240623 nightly build.
In Anaconda Prompt created a new environment 'pycro'
In pycro environment installed pycromanager using pip install pycromanager
In pycro environment upgraded pycro using pip install pycromanager --upgrade
Opened MicroManager
Opened python within pycro environment, entered the following:

>>> from pycromanager import Core, MagellanAcquisition, Magellan
>>> core = Core()
>>> core.set_config('Brightfield', 'OFF')
>>> core.set_config('Objectives', '5x_SlideScanner')
>>> core.set_config('SidePort', 'Camera')
>>> core.set_config('Colors', 'Violet>Blue')
>>> x, y = core.get_x_position(), core.get_y_position()
>>> mag_api = Magellan()
>>> new_grid = mag_api.create_grid('Grid', 13, 19, x, y)
>>> acq_settings = mag_api.get_acquisition_settings(0)
>>> acq_settings.set_acquisition_name('test')
>>> acq_settings.set_saving_dir('C:/Users/Lab/Desktop')
>>> acq_settings.set_use_channel('Violet>Blue', True)
>>> acq = MagellanAcquisition(magellan_acq_index=0)

The attribute is still missing
>>> hasattr(MagellanAcquisition,'_acq')
False
>>> MagellanAcquisition.__dict__
mappingproxy({'__module__': 'pycromanager.acquisition.java_backend_acquisitions', '__doc__': 'Class used for launching Micro-Magellan acquisitions. Must pass either magellan_acq_index\nor magellan_explore as an argument', '__init__': <function MagellanAcquisition.__init__ at 0x0000022FF681A1F0>, '_start_events': <function MagellanAcquisition._start_events at 0x0000022FF681A280>, '_create_event_queue': <function MagellanAcquisition._create_event_queue at 0x0000022FF681A310>, '_create_remote_acquisition': <function MagellanAcquisition._create_remote_acquisition at 0x0000022FF681A430>, '__abstractmethods__': frozenset(), '_abc_impl': <_abc_data object at 0x0000022FF67F9C90>})
>>> dir(MagellanAcquisition)
['__abstractmethods__', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__enter__', '__eq__', '__exit__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_abc_impl', '_add_storage_monitor_fn', '_call_image_process_fn', '_check_for_exceptions', '_create_event_queue', '_create_remote_acquisition', '_initialize_hooks', '_initialize_image_processor', '_start_events', '_start_hook', '_start_notification_dispatcher', '_start_processor', '_start_receiving_notifications', 'abort', 'acquire', 'await_completion', 'get_dataset', 'get_viewer', 'mark_finished']

Here is the text of my java_backend_acquisitions.py file:

java_backend_acquisitions.py
"""
The Pycro-manager Acquisiton system
"""
import json
import logging
import warnings
import weakref

import numpy as np
import multiprocessing
import threading
from inspect import signature
import time
from pyjavaz import deserialize_array
from pyjavaz import PullSocket, PushSocket, JavaObject, JavaClass
from pyjavaz import DEFAULT_BRIDGE_PORT as DEFAULT_PORT
from pycromanager.mm_java_classes import ZMQRemoteMMCoreJ, Magellan
from pycromanager.acquisition.RAMStorage_java import NDRAMDatasetJava

from ndstorage import Dataset
import os.path
import queue
from docstring_inheritance import NumpyDocstringInheritanceMeta
from pycromanager.acquisition.acquisition_superclass import Acquisition
import traceback
from pycromanager.acq_future import AcqNotification, AcquisitionFuture
import json

logger = logging.getLogger(__name__)

### These functions are defined outside the Acquisition class to
# prevent problems with pickling when running them in differnet process
# although they are currently only used in different threads

def _run_acq_event_source(acquisition, event_port, event_queue, debug=False):
    event_socket = PushSocket(event_port, debug=debug)
    try:
        while True:
            events = event_queue.get(block=True)
            if debug:
                logger.debug(f"got event(s): {events}")
            if events is None:
                # Initiate the normal shutdown process
                if not acquisition._acq.is_finished():
                    # if it has been finished through something happening on the other side
                    event_socket.send({"events": [{"special": "acquisition-end"}]})
                    # wait for signal that acquisition has received the end signal
                    while not acquisition._acq.is_finished():
                        acquisition._acq.block_until_events_finished(0.01)
                break
            # it may have been shut down remotely (e.g. by user Xing out viewer)
            # if we try to send an event at this time, it will hang indefinitely
            if acquisition._acq.is_finished():
                break
            # TODO in theory it could be aborted in between the check above and sending below,
            #  maybe consider putting a timeout on the send?
            event_socket.send({"events": events if type(events) == list else [events]})
            if debug:
                logger.debug("sent events")
    except Exception as e:
        acquisition.abort(e)
    finally:
        event_socket.close()

def _run_acq_hook(acquisition, pull_port,
                  push_port, hook_connected_evt, event_queue, hook_fn, debug=False):

    push_socket = PushSocket(pull_port, debug=debug)
    pull_socket = PullSocket(push_port, debug=debug)
    hook_connected_evt.set()

    exception = None
    while True:
        event_msg = pull_socket.receive()

        if "special" in event_msg and event_msg["special"] == "acquisition-end":
            push_socket.send(event_msg)
            push_socket.close()
            pull_socket.close()
            return
        else:
            if "events" in event_msg.keys():
                event_msg = event_msg["events"]  # convert from sequence
            params = signature(hook_fn).parameters
            if len(params) == 1 or len(params) == 2:
                try:
                    if len(params) == 1:
                        new_event_msg = hook_fn(event_msg)
                    elif len(params) == 2:
                        new_event_msg = hook_fn(event_msg, event_queue)
                except Exception as e:
                    acquisition.abort(e)
                    # Cancel the execution of event because there was an exception
                    new_event_msg = None
                    # don't return here--wait for the signal from the acq engine
            else:
                acquisition.abort(Exception("Incorrect number of arguments for hook function. Must be 1 or 2"))

        if isinstance(new_event_msg, list):
            new_event_msg = {
                "events": new_event_msg
            }  # convert back to the expected format for a sequence
        push_socket.send(new_event_msg)

def _run_image_processor(
        acquisition, pull_port, push_port, sockets_connected_evt, process_fn, event_queue, debug
):
    acquisition._process_fn = process_fn
    push_socket = PushSocket(pull_port, debug=debug)
    pull_socket = PullSocket(push_port, debug=debug)
    if debug:
        logger.debug("image processing sockets connected")
    sockets_connected_evt.set()

    def process_and_sendoff(image_tags_tuple, original_dtype):
        """

        Parameters
        ----------
        image_tags_tuple :

        Returns
        -------

        """
        if len(image_tags_tuple) != 2:
            acquisition.abort(Exception("If image is returned, it must be of the form (pixel, metadata)"))

        pixels = image_tags_tuple[0]
        metadata = image_tags_tuple[1]

        # only accepts same pixel type as original
        if not np.issubdtype(image_tags_tuple[0].dtype, original_dtype) and not np.issubdtype(
            original_dtype, image_tags_tuple[0].dtype
        ):
            acquisition.abort(Exception(
                "Processed image pixels must have same dtype as input image pixels, "
                "but instead they were {} and {}".format(image_tags_tuple[0].dtype, pixels.dtype)
            ))

        if metadata['PixelType'] == 'RGB32':
            if pixels.shape[-1] == 3:
                #append 0 for alpha channel because thats whats expected
                pixels = np.concatenate([pixels, np.zeros_like(pixels[..., 0])[..., None]], axis=2)
        else:
            #maybe pixel type was changed by processing?
            metadata["PixelType"] = "GRAY8" if pixels.dtype.itemsize == 1 else "GRAY16"

        processed_img = {
            "pixels": pixels.tobytes(),
            "metadata": metadata,
        }
        push_socket.send(processed_img, suppress_debug_message=True)

    while True:
        message = None
        while message is None:
            message = pull_socket.receive(timeout=30, suppress_debug_message=True)  # check for new message

        if "special" in message and message["special"] == "finished":
            pull_socket.close()
            push_socket.send(message)  # Continue propagating the finished signal
            push_socket.close()
            return

        metadata = message["metadata"]
        # TODO: this should probably be handled by the push socket...
        pixels = deserialize_array(message["pixels"])
        if metadata['PixelType'] == 'RGB32':
            image = np.reshape(pixels, [metadata["Height"], metadata["Width"], 4])[..., :3]
        else:
            image = np.reshape(pixels, [metadata["Height"], metadata["Width"]])

        processed = acquisition._call_image_process_fn(image, metadata)

        if processed is None:
            continue

        if type(processed) == list:
            for image in processed:
                process_and_sendoff(image, pixels.dtype)
        else:
            process_and_sendoff(processed, pixels.dtype)

def _notification_handler_fn(acquisition, notification_push_port, connected_event, debug=False):
    monitor_socket = PullSocket(notification_push_port)
    connected_event.set()

    events_finished = False
    data_sink_finished = False
    while True:
        try:
            message = monitor_socket.receive()
            notification = AcqNotification.from_json(message)

            # these are processed seperately to handle image saved callback
            # Decode the Data storage class-specific notification
            if AcqNotification.is_image_saved_notification(notification): # it was saved to RAM, not disk
                if not notification.is_data_sink_finished_notification():
                    # check if NDTiff data storage used
                    if acquisition._directory is not None:
                        index_entry = notification.payload.encode('ISO-8859-1')
                        axes = acquisition._dataset.add_index_entry(index_entry)
                        # swap the notification.payload from the byte array of index information to axes
                        notification.payload = axes
                    else: # RAM storage
                        axes = json.loads(notification.payload)
                        acquisition._dataset.add_available_axes(axes)
                        notification.payload = axes

                acquisition._image_notification_queue.put(notification)
                # check size
                if acquisition._image_notification_queue.qsize() > acquisition._image_notification_queue.maxsize * 0.9:
                    warnings.warn(f"Acquisition image notification queue size: {acquisition._image_notification_queue.qsize()}")

            acquisition._notification_queue.put(notification)
            if AcqNotification.is_acquisition_finished_notification(notification):
                events_finished = True
            elif AcqNotification.is_data_sink_finished_notification(notification):
                data_sink_finished = True
                acquisition._image_notification_queue.put(notification)
            if events_finished and data_sink_finished:
                break

        except Exception as e:
            traceback.print_exc()
            acquisition.abort(e)
            continue # perform an orderly shutdown

    monitor_socket.close()

class JavaBackendAcquisition(Acquisition, metaclass=NumpyDocstringInheritanceMeta):
    """
    Pycro-Manager acquisition that uses a Java runtime backend via a ZeroMQ communication layer.
    """

    def __init__(
        self,
        directory: str=None,
        name: str='default_acq_name',
        image_process_fn : callable=None,
        event_generation_hook_fn: callable=None,
        pre_hardware_hook_fn: callable=None,
        post_hardware_hook_fn: callable=None,
        post_camera_hook_fn: callable=None,
        notification_callback_fn: callable=None,
        image_saved_fn: callable=None,
        show_display: bool=True,
        napari_viewer=None,
        saving_queue_size: int=20,
        timeout: int=2500,
        port: int=DEFAULT_PORT,
        debug: int=False
    ):
        # Get a dict of all named argument values (or default values when nothing provided)
        arg_names = [k for k in signature(JavaBackendAcquisition.__init__).parameters.keys() if k != 'self']
        l = locals()
        named_args = {arg_name: (l[arg_name] if arg_name in l else
                                     dict(signature(JavaBackendAcquisition.__init__).parameters.items())[arg_name].default)
                                     for arg_name in arg_names }

        superclass_arg_names = [k for k in signature(Acquisition.__init__).parameters.keys() if k != 'self']
        superclass_args = {key: named_args[key] for key in superclass_arg_names}
        super().__init__(**superclass_args)

        if directory is not None:
            # Expend ~ in path
            directory = os.path.expanduser(directory)
            # If path is relative, retain knowledge of the current working directory
            self._directory = os.path.abspath(directory)
        else:
            self._directory = None
        named_args['directory'] = self._directory

        # Java specific parameters
        self._port = port
        self._timeout = timeout
        self._nd_viewer = None

        self._create_event_queue()
        self._create_remote_acquisition(**named_args)
        self._initialize_image_processor(**named_args)
        self._initialize_hooks(**named_args)

        try:
            self._remote_notification_handler = JavaObject('org.micromanager.remote.RemoteNotificationHandler',
                                                           args=[self._acq], port=self._port, new_socket=False,
                                                           timeout=self._timeout)
            self._acq_notification_recieving_thread = self._start_receiving_notifications()
            self._acq_notification_dispatcher_thread = self._start_notification_dispatcher(notification_callback_fn)
        # TODO: can remove this after this feature has been present for a while
        except:
            traceback.print_exc()
            warnings.warn('Could not create acquisition notification handler. '
                          'Update Micro-Manager and Pyrcro-Manager to the latest versions to fix this')

        # Start remote acquisition
        # Acquistition.start is now deprecated, so this can be removed later
        # Acquisitions now get started automatically when the first events submitted
        # but Magellan acquisitons (and probably others that generate their own events)
        # will need some new method to submit events only after image processors etc have been added
        self._acq.start()
        self._dataset_disk_location = (
            self._acq.get_data_sink().get_storage().get_disk_location()
            if self._acq.get_data_sink() is not None
            else None
        )

        self._start_events()

        # Load remote storage
        data_sink = self._acq.get_data_sink()
        # load a view of the dataset in progress. This is used so that acq.get_dataset() can be called
        # while the acquisition is still running, and (optionally )so that a image_saved_fn can be called
        # when images are written to disk/RAM storage
        storage_java_class = data_sink.get_storage()
        summary_metadata = storage_java_class.get_summary_metadata()
        if directory is not None:
            # NDTiff dataset saved to disk on Java side
            self._dataset = Dataset(dataset_path=self._dataset_disk_location, summary_metadata=summary_metadata)
        else:
            # Saved to RAM on Java side
            self._dataset = NDRAMDatasetJava(storage_java_class)
        # Monitor image arrival so they can be loaded on python side, but with no callback function
        # Need to do this regardless of whether you use it, so that it signals to shut down on Java side
        self._storage_monitor_thread = self._add_storage_monitor_fn(image_saved_fn=image_saved_fn)

        if show_display:
            if napari_viewer is None:
                # using NDViewer
                self._nd_viewer = self._acq.get_data_sink().get_viewer()
            else:
                # using napari viewer
                try:
                    import napari
                except:
                    raise Exception('Napari must be installed in order to use this feature')
                from pycromanager.napari_util import start_napari_signalling
                assert isinstance(napari_viewer, napari.Viewer), 'napari_viewer must be an instance of napari.Viewer'
                self._napari_viewer = napari_viewer
                start_napari_signalling(self._napari_viewer, self.get_dataset())

    ########  Public API methods with unique implementations for Java backend ###########

    def await_completion(self):
        try:
            while not self._acq.are_events_finished() or (
                    self._acq.get_data_sink() is not None and not self._acq.get_data_sink().is_finished()):
                self._check_for_exceptions()
                self._acq.block_until_events_finished(0.01)
            # This will block until saving is finished, if there is a data sink
            self._acq.wait_for_completion()
            self._check_for_exceptions()
        finally:
            for hook_thread in self._hook_threads:
                hook_thread.join()

            if hasattr(self, '_event_thread'):
                self._event_thread.join()

            # need to do this so its _Bridge can be garbage collected and a reference to the JavaBackendAcquisition
            # does not prevent Bridge cleanup and process exiting
            self._remote_acq = None

            # Wait on all the other threads to shut down properly
            if hasattr(self, '_storage_monitor_thread'):
                self._storage_monitor_thread.join()

            if hasattr(self, '_acq_notification_recieving_thread'):
                # for backwards compatiblitiy with older versions of Pycromanager java before this added
                self._acq_notification_recieving_thread.join()
                self._remote_notification_handler.notification_handling_complete()
                # need to do this so its _Bridge can be garbage collected and a reference to the JavaBackendAcquisition
                # does not prevent Bridge cleanup and process exiting
                self._remote_notification_handler = None
                self._acq_notification_dispatcher_thread.join()

            try:
                # one final check for exceptions for stuff that may have happened during shutdown
                self._check_for_exceptions()
            finally:
                self._acq = None
                self._finished = True

    def get_viewer(self):
        if self._napari_viewer is None:
            return self._nd_viewer
        else:
            return self._napari_viewer

    ########  Private methods ###########
    def _start_receiving_notifications(self):
        """
        Thread that runs a function that pulls notifications from the acquisition engine and puts them on a queue
        """
        connected_event = threading.Event()

        pull_port = self._remote_notification_handler.get_port()
        notification_thread = threading.Thread(
            target=_notification_handler_fn,
            args=(
                self,
                pull_port,
                connected_event,
                self._debug,
            ),
            name="NotificationHandlerThread",
        )
        #
        # Wait for pulling to start before you signal for pushing to start
        notification_thread.start()
        connected_event.wait()

        # start pushing out all the notifications
        self._remote_notification_handler.start()
        return notification_thread

    def _check_for_exceptions(self):
        """
        Check for exceptions on the python side (i.e. hooks and processors)
        or on the Java side (i.e. hardware control)
        """
        # these will throw exceptions
        self._acq.check_for_exceptions()
        if self._exception is not None:
            raise self._exception

    def _start_events(self, **kwargs):

        self.event_port = self._acq.get_event_port()

        self._event_thread = threading.Thread(
            target=_run_acq_event_source,
            args=(self, self.event_port, self._event_queue, self._debug),
            name="Event sending",
        )
        self._event_thread.start()

    def _initialize_image_processor(self, **kwargs):

        if kwargs['image_process_fn'] is not None:
            java_processor = JavaObject(
                "org.micromanager.remote.RemoteImageProcessor", port=self._port, timeout=self._timeout
            )
            self._acq.add_image_processor(java_processor)
            self._processor_thread = self._start_processor(
                java_processor, kwargs['image_process_fn'],
                # Some acquisitions (e.g. Explore acquisitions) create events on Java side
                self._event_queue if hasattr(self, '_event_queue') else None,
                process=False)

    def _initialize_hooks(self, **kwargs):
        self._hook_threads = []
        if kwargs['event_generation_hook_fn'] is not None:
            hook = JavaObject(
                "org.micromanager.remote.RemoteAcqHook", port=self._port, args=[self._acq], timeout=self._timeout
            )
            self._hook_threads.append(self._start_hook(hook, kwargs['event_generation_hook_fn'],
                                                       self._event_queue, process=False))
            self._acq.add_hook(hook, self._acq.EVENT_GENERATION_HOOK)
        if kwargs['pre_hardware_hook_fn'] is not None:
            hook = JavaObject(
                "org.micromanager.remote.RemoteAcqHook", port=self._port, args=[self._acq], timeout=self._timeout
            )
            self._hook_threads.append(self._start_hook(hook,
                                            kwargs['pre_hardware_hook_fn'], self._event_queue,
                                                       process=False))
            self._acq.add_hook(hook, self._acq.BEFORE_HARDWARE_HOOK)
        if kwargs['post_hardware_hook_fn'] is not None:
            hook = JavaObject(
                "org.micromanager.remote.RemoteAcqHook", port=self._port, args=[self._acq], timeout=self._timeout
            )
            self._hook_threads.append(self._start_hook(hook, kwargs['post_hardware_hook_fn'],
                                                       self._event_queue, process=False))
            self._acq.add_hook(hook, self._acq.AFTER_HARDWARE_HOOK)
        if kwargs['post_camera_hook_fn'] is not None:
            hook = JavaObject(
                "org.micromanager.remote.RemoteAcqHook", port=self._port, args=[self._acq], timeout=self._timeout
            )
            self._hook_threads.append(self._start_hook(hook, kwargs['post_camera_hook_fn'],
                                                       self._event_queue, process=False))
            self._acq.add_hook(hook, self._acq.AFTER_CAMERA_HOOK)

    def _create_remote_acquisition(self, **kwargs):
        core = ZMQRemoteMMCoreJ(port=self._port, timeout=self._timeout, debug=self._debug)
        acq_factory = JavaObject("org.micromanager.remote.RemoteAcquisitionFactory",
            # create a new socket for it to run on so that it can have blocking calls without interfering with
            # the main socket or other internal sockets
            new_socket=True,
            port=self._port, args=[core], debug=self._debug, timeout=self._timeout)
        show_viewer = kwargs['show_display'] is True and kwargs['napari_viewer'] is None
        self._acq = acq_factory.create_acquisition(kwargs['directory'], kwargs['name'], show_viewer,
                                                   kwargs['saving_queue_size'], self._debug,)

    def _start_hook(self, remote_hook, remote_hook_fn : callable, event_queue, process):
        """

        Parameters
        ----------
        remote_hook :

        remote_hook_fn :

        event_queue :

        process :

        Returns
        -------

        """
        hook_connected_evt = multiprocessing.Event() if process else threading.Event()

        pull_port = remote_hook.get_pull_port()
        push_port = remote_hook.get_push_port()

        hook_thread = (multiprocessing.Process if process else threading.Thread)(
            target=_run_acq_hook,
            name="AcquisitionHook",
            args=(
                self,
                pull_port,
                push_port,
                hook_connected_evt,
                event_queue,
                remote_hook_fn,
                self._debug,
            ),
        )
        # if process else threading.Thread(target=_acq_hook_fn, args=(), name='AcquisitionHook')
        hook_thread.start()

        hook_connected_evt.wait()  # wait for push/pull sockets to connect
        return hook_thread

    def _start_processor(self, processor, process_fn, event_queue, process):
        """

        Parameters
        ----------
        processor :

        process_fn :

        event_queue :

        process :

        Returns
        -------

        """
        # this must start first
        processor.start_pull()

        sockets_connected_evt = multiprocessing.Event() if process else threading.Event()

        pull_port = processor.get_pull_port()
        push_port = processor.get_push_port()

        processor_thread = (multiprocessing.Process if process else threading.Thread)(
            target=_run_image_processor,
            args=(
                self,
                pull_port,
                push_port,
                sockets_connected_evt,
                process_fn,
                event_queue,
                self._debug,
            ),
            name="ImageProcessor",
        )
        processor_thread.start()

        sockets_connected_evt.wait()  # wait for push/pull sockets to connect
        processor.start_push()
        return processor_thread

class XYTiledAcquisition(JavaBackendAcquisition):
    """
    For making tiled images with an XY stage and multiresolution saving
    (e.g. for making one large contiguous image of a sample larger than the field of view)
    """

    def __init__(
            self,
            tile_overlap : int or tuple,
            directory: str=None,
            name: str=None,
            max_multi_res_index: int=None,
            image_process_fn: callable=None,
            pre_hardware_hook_fn: callable=None,
            post_hardware_hook_fn: callable=None,
            post_camera_hook_fn: callable=None,
            show_display: bool=True,
            image_saved_fn: callable=None,
            saving_queue_size: int=20,
            timeout: int=1000,
            port: int=DEFAULT_PORT,
            debug: bool=False,
    ):
        """
        Parameters
        ----------
        tile_overlap : int or tuple of int
            If given, XY tiles will be laid out in a grid and multi-resolution saving will be
            actived. Argument can be a two element tuple describing the pixel overlaps between adjacent
            tiles. i.e. (pixel_overlap_x, pixel_overlap_y), or an integer to use the same overlap for both.
            For these features to work, the current hardware configuration must have a valid affine transform
            between camera coordinates and XY stage coordinates
        max_multi_res_index : int
            Maximum index to downsample to in multi-res pyramid mode. 0 is no downsampling,
            1 is downsampled up to 2x, 2 is downsampled up to 4x, etc. If not provided, it will be dynamically
            calculated and updated from data
        """
        self.tile_overlap = tile_overlap
        self.max_multi_res_index = max_multi_res_index
        # Collct all argument values except the ones specific to XY Tiled acquisitions
        arg_names = list(signature(self.__init__).parameters.keys())
        arg_names.remove('tile_overlap')
        arg_names.remove('max_multi_res_index')
        l = locals()
        named_args = {arg_name: l[arg_name] for arg_name in arg_names}
        super().__init__(**named_args)

    def _create_remote_acquisition(self, port, **kwargs):
        core = ZMQRemoteMMCoreJ(port=self._port, timeout=self._timeout)
        acq_factory = JavaObject("org.micromanager.remote.RemoteAcquisitionFactory",
                                 port=self._port, args=[core], timeout=self._timeout)

        show_viewer = kwargs['show_display'] is True and\
                      kwargs['napari_viewer'] is None and\
                      (kwargs['directory'] is not None and kwargs['name'] is not None)
        if type(self.tile_overlap) is tuple:
            x_overlap, y_overlap = self.tile_overlap
        else:
            x_overlap = self.tile_overlap
            y_overlap = self.tile_overlap

        self._remote_acq = acq_factory.create_tiled_acquisition(
            kwargs['directory'],
            kwargs['name'],
            show_viewer,
            True,
            x_overlap,
            y_overlap,
            self.max_multi_res_index if self.max_multi_res_index is not None else -1,
            kwargs['saving_queue_size'],
            self._debug,
        )

class ExploreAcquisition(JavaBackendAcquisition):
    """
    Launches a user interface for an "Explore Acquisition"--a type of XYTiledAcquisition
    in which acquisition events come from the user dynamically driving the stage and selecting
    areas to image
    """

    def __init__(
            self,
            directory: str,
            name: str,
            z_step_um: float,
            tile_overlap: int or tuple,
            channel_group: str = None,
            image_process_fn: callable=None,
            pre_hardware_hook_fn: callable=None,
            post_hardware_hook_fn: callable=None,
            post_camera_hook_fn: callable=None,
            show_display: bool=True,
            image_saved_fn: callable=None,
            saving_queue_size: int=20,
            timeout: int=2500,
            port: int=DEFAULT_PORT,
            debug: bool=False,
    ):
        """
        Parameters
        ----------
        z_step_um : str
            Spacing between successive z planes, in microns
        tile_overlap : int or tuple of int
            If given, XY tiles will be laid out in a grid and multi-resolution saving will be
            activated. Argument can be a two element tuple describing the pixel overlaps between adjacent
            tiles. i.e. (pixel_overlap_x, pixel_overlap_y), or an integer to use the same overlap for both.
            For these features to work, the current hardware configuration must have a valid affine transform
            between camera coordinates and XY stage coordinates
        channel_group : str
            Name of a config group that provides selectable channels
        """
        self.tile_overlap = tile_overlap
        self.channel_group = channel_group
        self.z_step_um = z_step_um
        # Collct all argument values except the ones specific to ExploreAcquisitions
        arg_names = list(signature(self.__init__).parameters.keys())
        arg_names.remove('tile_overlap')
        arg_names.remove('z_step_um')
        arg_names.remove('channel_group')
        l = locals()
        named_args = {arg_name: l[arg_name] for arg_name in arg_names}
        super().__init__(**named_args)

    def _create_remote_acquisition(self, port, **kwargs):
        if type(self.tile_overlap) is tuple:
            x_overlap, y_overlap = self.tile_overlap
        else:
            x_overlap = self.tile_overlap
            y_overlap = self.tile_overlap

        ui_class = JavaClass('org.micromanager.explore.ExploreAcqUIAndStorage')
        ui = ui_class.create(kwargs['directory'], kwargs['name'], x_overlap, y_overlap, self.z_step_um, self.channel_group)
        self._remote_acq = ui.get_acquisition()

    def _start_events(self, **kwargs):
        pass # These come from the user

    def _create_event_queue(self, **kwargs):
        pass # Comes from the user

class MagellanAcquisition(JavaBackendAcquisition):
    """
    Class used for launching Micro-Magellan acquisitions. Must pass either magellan_acq_index
    or magellan_explore as an argument
    """

    def __init__(
            self,
            magellan_acq_index: int=None,
            magellan_explore: bool=False,
            image_process_fn: callable=None,
            event_generation_hook_fn: callable=None,
            pre_hardware_hook_fn: callable=None,
            post_hardware_hook_fn: callable=None,
            post_camera_hook_fn: callable=None,
            image_saved_fn: callable=None,
            timeout: int=500,
            port: int=DEFAULT_PORT,
            debug: bool=False,
    ):
        """
        Parameters
        ----------
        magellan_acq_index : int
            run this acquisition using the settings specified at this position in the main
            GUI of micro-magellan (micro-manager plugin). This index starts at 0
        magellan_explore : bool
            Run a Micro-magellan explore acquisition
        """
        self.magellan_acq_index = magellan_acq_index
        self.magellan_explore = magellan_explore
        # Collct all argument values except the ones specific to Magellan
        arg_names = list(signature(self.__init__).parameters.keys())
        arg_names.remove('magellan_acq_index')
        arg_names.remove('magellan_explore')
        l = locals()
        named_args = {arg_name: l[arg_name] for arg_name in arg_names}
        super().__init__(**named_args)

    def _start_events(self, **kwargs):
        pass # Magellan handles this on Java side

    def _create_event_queue(self, **kwargs):
        pass # Magellan handles this on Java side

    def _create_remote_acquisition(self, **kwargs):
        magellan_api = Magellan()
        if self.magellan_acq_index is not None:
            self._remote_acq = magellan_api.create_acquisition(self.magellan_acq_index, False)
        elif self.magellan_explore:
            self._remote_acq = magellan_api.create_explore_acquisition(False)
        self._event_queue = None

Please let me know if you need any more information.
Thanks!

Looks like the new versions didn't deploy (https://github.com/micro-manager/pycro-manager/actions/runs/9633056103)
I just updated versions so hopefully that will happen now https://github.com/micro-manager/pycro-manager/pull/782

It works great, thank you very much!
On Tue, Jun 25, 2024 at 2:16 AM Henry Pinkard @.***>
wrote:

Looks like the new versions didn't deploy (
https://github.com/micro-manager/pycro-manager/actions/runs/9633056103)
I just updated versions so hopefully that will happen now #782
https://github.com/micro-manager/pycro-manager/pull/782
—
Reply to this email directly, view it on GitHub
https://github.com/micro-manager/pycro-manager/issues/780#issuecomment-2188158619,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADAMUTX2TPOGCEME7TDRLJDZJEKLVAVCNFSM6AAAAABJSNGWUOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBYGE2TQNRRHE
.
You are receiving this because you authored the thread.Message ID:
@.***>


BUG: stats.fit: intermittent failure in doctest

Describe your issue.
Refguide check is failing intermittently (e.g. here) with the error below due to a scipy.stats.fit doctest failure.
The fix is probably to move the last example (the one that shows the custom optimizer with fixed seed) above the failing test and use it in the failing test.

Reproducing Code Example
NA

Error message
===========
scipy.stats
===========

scipy.stats.fit
---------------

File "build/testenv/lib/python3.9/site-packages/scipy/stats/_fit.py", line 362, in fit
Failed example:
    res3.nllf() > res.nllf()
Expected:
    True
Got:
    False

SciPy/NumPy/Python version information
main

I am wondering if we should add seed, in the end it's an optimizer behind. Users might complain about that even though it can be fixed by supplying your optimizer. What do you think?
Otherwise yes we can simply fix the doc. And for a case without custom optimizer we should have a doctest that cannot fail.

I am hesitant to add seed because there is nothing in fit that is stochastic. It just so happens that the default optimizer right now is stochastic, but that is not by preference. My hope is to get a deterministic optimizer in its place. So for now, yeah, let's just change the order to avoid the failure.

I decided to mark it # may vary and address it explicitly in the commentary.

I like your solution, thanks!


Can't install. Windows 8.1

Error is below:

Versions:

Here's npm-debug.log
https://gist.githubusercontent.com/ttocskcaj/b42952835ba31ade22ca/raw/node-sass log

Hello, thanks for reporting this issue.
Can you please try this:
:: this is cmd
npm install node-sass
:: it fails with the aforementioned error
node
:: you have entered node interactive console (REPL)
var spawnSync = require('child_process').spawnSync;
spawnSync('C:\\Program Files\\Internet Explorer\\iexplore.exe');
:: does it open IE?

var spawnSync = require('child_process').spawnSync
returns undefined
spawnSync('C:\\Program Files\\Internet Explorer\\iexplore.exe')
Does open IE

I asked that because it seemed like a path issue with spawn (where path has a space).
To further investigate, please download the zip file from https://github.com/sass/node-sass/releases/tag/v2.0.1, extract and then in scripts\install.js, throw in some console.log statements, save and then:
:: this is cmd
cd into\extracted\node-sass\v2.0.1\src\directory
node scripts/install

It will not install like that anyway, but we just need to figure out which line is causing this issue:

/c/program files no such file or directory

Also, please show the output of:
cd \temp1
node -p process.execPath
``

Running node scripts/install is coming up with this error:
https://gist.github.com/ttocskcaj/576f17c97b9a220e3941
And node -p process.execPath returns this:
C:\Program Files (x86)\nodejs\node.exe
(I've tried the 64bit and 32bit versions)

Sorry, please try with:
:: this is cmd
cd into\extracted\node-sass-v2.0.1
npm install

Also, add some console.log('test 1') (test 2, test 3 ...) in scripts\install.js, so it appear in output and try to detect at which line it throws.

OK none of the console.logs are showing on the screen.
Including one on the first line of scripts/install.js
Maybe it's not reaching this file?

Once npm install succeeds, please try explicitly running node scripts/install (again).

I've got it to install, though I think the problem was with my "ComSpec" environmental variable rather than node-sass. Cheers for the help.


Request to add an external readonly method "getPoolTotalAt" on DEX contract

Our team is working on foligator- an analytics tool for balanced.
Balanced Dex score has a method "getPoolTotal(self, _id: int, _token: Address)" which returns the current pool total for base or quote.
We want something like getPoolTotalAt(self, _id: int, _token: Address, day: int) on DEX contract. It will improve accuracy of our results.

Hey @habbas33, it would be better for you to use new feature in ICON blockchain where you can query read only methods using block height.
Or use this method to use the time
https://tracker.v2.mainnet.sng.vultr.icon.community/api/v1/docs/index.html#/Blocks/get_api_v1_blocks_timestamp__timestamp_
This method will not be included in dex because there are no snapshots for pool total.


BigInteger support

#6
Fixed using json-bigint

shouldn't this be also changed https://github.com/qdrant/openapi-typescript-fetch/blob/f33b84870048e75efe68e1ebaf6876bf87548ab1/src/fetcher.ts#L97C2-L97C79 ?

shouldn't this be also changed https://github.com/qdrant/openapi-typescript-fetch/blob/f33b84870048e75efe68e1ebaf6876bf87548ab1/src/fetcher.ts#L97C2-L97C79 ?

Fixed


Error using custom_step duplicate function

It looks liker there is an issue with self.dup() in custom_step. It expect attribute 'setup_root' but is has not this attribute.
You can create this off course, but it should give a better warning that it needs this attribute to be created before you can use this dup function.
How should this be fixed @miquelcampos ?
Should a setup_root be created in the mgear_run? Or it could if not found use self.mgear_run.setupWS?
  File "/Users/liudger/Documents/projects/benjamin bat/scenes/bejamin/custom_steps/scripts/post/wing_proximity.py", line 48, in proximity_wrap
    self.wing_prox_geo = self.dup(self.custom_step("rig_base").wing_base_geo, name="wing_proximity_geo")
  File "/Users/liudger/Git/mgear4/release/scripts/mgear/shifter/custom_step.py", line 89, in dup
    pm.parent(dup, self.setup_root)
AttributeError: 'CustomShifterStep' object has no attribute 'setup_root'

@liudger I will check also this ASAP. Thanks!


fix(#979): GetHubFromContext may crash instead of returning nil

fix #979

Thanks for approving! I habe no write access, so a merge would be nice.

@kohnalex pls add description to the readme

@kohnalex Can you update the README with the new function. Also add a unit-test for it. Thanks!

@kohnalex friendly ping, pls check the last hints

[ ] Add description in readme
[ ] Add unittests for the new/adjusted code

Sorry for the wait... I had to restructure the test cases a little so app.Use(New()) will be called after the test cases I have added.

What else is needed to merge this?


chore: release version 0.0.5

Update related files to release v0.0.5

Thanks !!

 I can get python realtime 0.0.5 to work with websockets 10.3 on Python 3.10. But it would be really cool if you could update the supabase python main package so I could just use that package for everything instead of the individual packages. It still references realtime 0.0.4 and poetry gives an error with websockets 10.3.


Vertex label separator

Coverage remained the same at 44.89% when pulling 64126dbc971e7de997da46ccaf63d8b55239c7e7 on unit-tests into ecf1271f3806ced85d37b87223d159c8b42dd2da on master.


Fix bug with state named 'is'.

Sympy doesn't tolerate variable names called is.

Fixes #1258.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 86.35%. Comparing base (8126294) to head (9e9a3a3).

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1263   +/-   ##
=======================================
  Coverage   86.35%   86.35%
=======================================
  Files         175      175
  Lines       13214    13214
=======================================
  Hits        11411    11411
  Misses       1803     1803

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Will still break if the following keywords are declared as NMODL variables:
'nonlocal', 'except', 'else', 'from', 'not', 'with', 'for', 'or', 'try', 'and', 'class', 'None', 'True', 'async', 'continue', 'break', 'False', 'if', 'return', 'assert', 'await', 'global'

I understand that it's rather partial solution. Some are tricky: should if be a legal variable name? I don't know and I want to punt. Same for global. I'd be quite happy if it continues to crash for all of those.
The problem is we only have strings. We don't know if a particular substring is a variable or if it's a keyword. We know if a particular string is used as a variable name; but we don't know if a particular occurrence of the string is a variable or a keyword.

Re: legality of variable naming, shouldn't NOCMODL be the reference for what is valid and what isn't (however strange the rules may be)? In any case, the variables which NOCMODL doesn't seem to like (out of the ones I listed above) are apparently else not for or and if return assert, while the other ones compile fine, so maybe we can just add those in the list of variables to rename and call it a day for now.

Makes sense. The proper place to fix all of this is when NMODL create the string it sends to SymPy. That string should not contain any troublesome identifiers, i.e. all variable should probably be mangled first; and demangled after SymPy. There we'd have a better understanding of each substring.

I'm surprised class and try would work.

I'm surprised class and try would work.

NOCMODL seems to define those, which is apparently legal (though it raises a warning on clang): https://godbolt.org/z/sa9qddYTY

It feels very strange to allow C++ keywords as a variable names. It feels even more off to "fix" it by patching the string sent to SymPy, meaning the C++ keywords are returned from SymPy again.
I get your point of nocmodl being the reference. Since it's all not very clean, I've reduced the number of keywords we sanitize in this unsatisfactory way to a small number of what I believe a plausible variable names. I'd be open to reducing what works even futher to just is and maybe in, to make the bug Koen is seeing go away.

Since keeping a list of both C++ and Python keywords is a bit cumbersome, we could add prefix/suffix to all variables internally or something, but we can postpone the details of that discussion for later. For now, as long as this PR fixes the issue, I'm fine with it as a temporary workaround.


Support Rails 8.0

Note that Rails 8.0 requires Ruby >= 3.2.0.
https://rubygems.org/gems/rails/versions/8.0.0

@smtlaissezfaire
Hi, could you review, merge and release a new version that supports Rails 8.0? 😃

0.17.0 released - https://mongomapper.com/blog/2024/12/10/release-0-17-0

Thank you!!


brew install envoy

Hi, thanks for this great project! Just wanted to let you know that Envoy is now available on Homebrew, so Mac users can install it with:
brew install envoy

Thanks to @rebello95 and others involved in the -Wrange-loop-analysis work for making this possible.
Ref: https://github.com/Homebrew/homebrew-core/pull/61858

thanks @ankane (ref #13461)

Awesome work! Thank you!