pybind11_add_module adds OS-installed ROS include path before ament include paths

This isn't exactly a problem with this repository, but I think it might be the best place to fix it, so I'm creating an issue here...
First, the problem: rosbag2_py fails to build if there are binary-installed versions of other rosbag2 packages it depends on.  See ros2/rosbag2#583 for more details.
This is because it uses pybind11_add_module, provided by the pybind11 package in pybind11Tools.cmake, to declare its targets.  That macro declares the target and then calls target_include_directories; it adds system-installed include paths such as /opt/ros/rolling/include to the include directories before it is possible to call ament_target_dependencies, which adds include paths from the source workspace.  Because the system-installed directories are added first, gcc prefers to use them to locate header files, which causes the compilation to fail if there are any API incompatibilities.  If you declare targets with add_executable or add_library, the right way to fix this is to call ament_target_dependencies before you call target_include_directories or target_link_libraries, but that is not possible with pybind11 because pybind11_add_module does this (at line 154):
  add_library(${target_name} ${lib_type} ${exclude_from_all} ${ARG_UNPARSED_ARGUMENTS})

  if(ARG_SYSTEM)
    set(inc_isystem SYSTEM)
  endif()

  target_include_directories(${target_name} ${inc_isystem}
    PRIVATE ${PYBIND11_INCLUDE_DIR}  # from project CMakeLists.txt
    PRIVATE ${pybind11_INCLUDE_DIR}  # from pybind11Config
    PRIVATE ${PYTHON_INCLUDE_DIRS})

The best solution I've thought of to fix this is to add something like a ros_pybind11_add_module macro that has a AMENT_DEPENDENCIES option; it would duplicate the behavior of pybind11_add_module but the above block of code would call ament_target_dependencies with the contents of that option between add_library and target_include_directories.  The down side to this approach is it would also need to be updated any time pybind11 is updated to ensure that ros_pybind11_add_module's functionality is parallel with pybind11_add_module.
Since pybind11 isn't tied to ROS, it shouldn't need to know anything about ament, but I don't think rosbag2_py is the right place to put any custom cmake code that would be useful for any ROS package that uses pybind11; this package seems like the right place to me.  Does that sound reasonable to you, or do you have any other suggestions?

Thanks for the report! It looks like something else is going on here. I see a different compile error that doesn't involve pybind11_add_module(). I didn't put rcutils from source in the workspace. Should I have?
[ 84%] Building CXX object CMakeFiles/test_sqlite_storage.dir/test/rosbag2_storage_default_plugins/sqlite/test_sqlite_storage.cpp.o
[ 84%] Building CXX object CMakeFiles/test_sqlite_wrapper.dir/test/rosbag2_storage_default_plugins/sqlite/test_sqlite_wrapper.cpp.o
/usr/bin/c++  -DDEFAULT_RMW_IMPLEMENTATION=rmw_cyclonedds_cpp -DPLUGINLIB__DISABLE_BOOST_FUNCTIONS -DRCUTILS_ENABLE_FAULT_INJECTION -I/opt/ros/rolling/src/gmock_vendor/include -I/opt/ros/rolling/src/gtest_vendor/include -I/home/sloretz/bigssd/rosbag/src/rosbag2/rosbag2_storage_default_plugins/include -isystem /home/sloretz/bigssd/rosbag/install/rosbag2_test_common/include -isystem /opt/ros/rolling/include -isystem /home/sloretz/bigssd/rosbag/install/rosbag2_storage/include -isystem /opt/ros/rolling/opt/yaml_cpp_vendor/include  -Wall -Wextra -Wpedantic -Werror -std=gnu++14 -o CMakeFiles/test_sqlite_storage.dir/test/rosbag2_storage_default_plugins/sqlite/test_sqlite_storage.cpp.o -c /home/sloretz/bigssd/rosbag/src/rosbag2/rosbag2_storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/test_sqlite_storage.cpp
/usr/bin/c++  -DDEFAULT_RMW_IMPLEMENTATION=rmw_cyclonedds_cpp -DPLUGINLIB__DISABLE_BOOST_FUNCTIONS -DRCUTILS_ENABLE_FAULT_INJECTION -I/opt/ros/rolling/src/gmock_vendor/include -I/opt/ros/rolling/src/gtest_vendor/include -I/home/sloretz/bigssd/rosbag/src/rosbag2/rosbag2_storage_default_plugins/include -isystem /home/sloretz/bigssd/rosbag/install/rosbag2_test_common/include -isystem /opt/ros/rolling/include -isystem /home/sloretz/bigssd/rosbag/install/rosbag2_storage/include -isystem /opt/ros/rolling/opt/yaml_cpp_vendor/include  -Wall -Wextra -Wpedantic -Werror -std=gnu++14 -o CMakeFiles/test_sqlite_wrapper.dir/test/rosbag2_storage_default_plugins/sqlite/test_sqlite_wrapper.cpp.o -c /home/sloretz/bigssd/rosbag/src/rosbag2/rosbag2_storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/test_sqlite_wrapper.cpp
In file included from /home/sloretz/bigssd/rosbag/src/rosbag2/rosbag2_storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/test_sqlite_storage.cpp:29:
/home/sloretz/bigssd/rosbag/src/rosbag2/rosbag2_storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/storage_test_fixture.hpp: In member function ‘rosbag2_storage::StorageOptions StorageTestFixture::make_storage_options_with_config(const string&, const string&)’:
/home/sloretz/bigssd/rosbag/src/rosbag2/rosbag2_storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/storage_test_fixture.hpp:154:22: error: too many initializers for ‘rosbag2_storage::StorageOptions’
  154 |       "", yaml_config};
      |                      ^
In file included from /home/sloretz/bigssd/rosbag/src/rosbag2/rosbag2_storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/test_sqlite_wrapper.cpp:27:
/home/sloretz/bigssd/rosbag/src/rosbag2/rosbag2_storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/storage_test_fixture.hpp: In member function ‘rosbag2_storage::StorageOptions StorageTestFixture::make_storage_options_with_config(const string&, const string&)’:
/home/sloretz/bigssd/rosbag/src/rosbag2/rosbag2_storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/storage_test_fixture.hpp:154:22: error: too many initializers for ‘rosbag2_storage::StorageOptions’
  154 |       "", yaml_config};
      |                      ^
/home/sloretz/bigssd/rosbag/src/rosbag2/rosbag2_storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/test_sqlite_storage.cpp: In member function ‘virtual void StorageTestFixture_storage_configuration_file_applies_over_storage_preset_profile_Test::TestBody()’:
/home/sloretz/bigssd/rosbag/src/rosbag2/rosbag2_storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/test_sqlite_storage.cpp:334:11: error: ‘struct rosbag2_storage::StorageOptions’ has no member named ‘storage_preset_profile’
  334 |   options.storage_preset_profile = "resilient";
      |           ^~~~~~~~~~~~~~~~~~~~~~
/home/sloretz/bigssd/rosbag/src/rosbag2/rosbag2_storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/test_sqlite_storage.cpp: In member function ‘virtual void StorageTestFixture_storage_preset_profile_applies_over_defaults_Test::TestBody()’:
/home/sloretz/bigssd/rosbag/src/rosbag2/rosbag2_storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/test_sqlite_storage.cpp:353:82: error: too many initializers for ‘rosbag2_storage::StorageOptions’
  353 |   rosbag2_storage::StorageOptions options{storage_uri, kPluginID, 0, 0, 0, "", ""};
      |                                                                                ^
/home/sloretz/bigssd/rosbag/src/rosbag2/rosbag2_storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/test_sqlite_storage.cpp:355:11: error: ‘struct rosbag2_storage::StorageOptions’ has no member named ‘storage_preset_profile’
  355 |   options.storage_preset_profile = "resilient";

This similarly looks like it's finding the installed header for rosbag2_storage::StorageOptions before the one in the workspace. What looks suspicous to me is -isystem instead of -I includes:  -isystem /opt/ros/rolling/include -isystem /home/sloretz/bigssd/rosbag/install/rosbag2_storage/include -isystem /opt/ros/rolling/opt/yaml_cpp_vendor/include. I'm pretty sure that changes the include order, but I'm struggling to find clear documentation about it. Do you know if anything in rosbag2 is using the  SYSTEM keyword argument to target_include_directories()?

Thanks for the report! It looks like something else is going on here. I see a different compile error that doesn't involve pybind11_add_module(). I didn't put rcutils from source in the workspace. Should I have?
[ 84%] Building CXX object CMakeFiles/test_sqlite_storage.dir/test/rosbag2_storage_default_plugins/sqlite/test_sqlite_storage.cpp.o
[ 84%] Building CXX object CMakeFiles/test_sqlite_wrapper.dir/test/rosbag2_storage_default_plugins/sqlite/test_sqlite_wrapper.cpp.o
/usr/bin/c++  -DDEFAULT_RMW_IMPLEMENTATION=rmw_cyclonedds_cpp -DPLUGINLIB__DISABLE_BOOST_FUNCTIONS -DRCUTILS_ENABLE_FAULT_INJECTION -I/opt/ros/rolling/src/gmock_vendor/include -I/opt/ros/rolling/src/gtest_vendor/include -I/home/sloretz/bigssd/rosbag/src/rosbag2/rosbag2_storage_default_plugins/include -isystem /home/sloretz/bigssd/rosbag/install/rosbag2_test_common/include -isystem /opt/ros/rolling/include -isystem /home/sloretz/bigssd/rosbag/install/rosbag2_storage/include -isystem /opt/ros/rolling/opt/yaml_cpp_vendor/include  -Wall -Wextra -Wpedantic -Werror -std=gnu++14 -o CMakeFiles/test_sqlite_storage.dir/test/rosbag2_storage_default_plugins/sqlite/test_sqlite_storage.cpp.o -c /home/sloretz/bigssd/rosbag/src/rosbag2/rosbag2_storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/test_sqlite_storage.cpp
/usr/bin/c++  -DDEFAULT_RMW_IMPLEMENTATION=rmw_cyclonedds_cpp -DPLUGINLIB__DISABLE_BOOST_FUNCTIONS -DRCUTILS_ENABLE_FAULT_INJECTION -I/opt/ros/rolling/src/gmock_vendor/include -I/opt/ros/rolling/src/gtest_vendor/include -I/home/sloretz/bigssd/rosbag/src/rosbag2/rosbag2_storage_default_plugins/include -isystem /home/sloretz/bigssd/rosbag/install/rosbag2_test_common/include -isystem /opt/ros/rolling/include -isystem /home/sloretz/bigssd/rosbag/install/rosbag2_storage/include -isystem /opt/ros/rolling/opt/yaml_cpp_vendor/include  -Wall -Wextra -Wpedantic -Werror -std=gnu++14 -o CMakeFiles/test_sqlite_wrapper.dir/test/rosbag2_storage_default_plugins/sqlite/test_sqlite_wrapper.cpp.o -c /home/sloretz/bigssd/rosbag/src/rosbag2/rosbag2_storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/test_sqlite_wrapper.cpp
In file included from /home/sloretz/bigssd/rosbag/src/rosbag2/rosbag2_storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/test_sqlite_storage.cpp:29:
/home/sloretz/bigssd/rosbag/src/rosbag2/rosbag2_storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/storage_test_fixture.hpp: In member function ‘rosbag2_storage::StorageOptions StorageTestFixture::make_storage_options_with_config(const string&, const string&)’:
/home/sloretz/bigssd/rosbag/src/rosbag2/rosbag2_storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/storage_test_fixture.hpp:154:22: error: too many initializers for ‘rosbag2_storage::StorageOptions’
  154 |       "", yaml_config};
      |                      ^
In file included from /home/sloretz/bigssd/rosbag/src/rosbag2/rosbag2_storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/test_sqlite_wrapper.cpp:27:
/home/sloretz/bigssd/rosbag/src/rosbag2/rosbag2_storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/storage_test_fixture.hpp: In member function ‘rosbag2_storage::StorageOptions StorageTestFixture::make_storage_options_with_config(const string&, const string&)’:
/home/sloretz/bigssd/rosbag/src/rosbag2/rosbag2_storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/storage_test_fixture.hpp:154:22: error: too many initializers for ‘rosbag2_storage::StorageOptions’
  154 |       "", yaml_config};
      |                      ^
/home/sloretz/bigssd/rosbag/src/rosbag2/rosbag2_storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/test_sqlite_storage.cpp: In member function ‘virtual void StorageTestFixture_storage_configuration_file_applies_over_storage_preset_profile_Test::TestBody()’:
/home/sloretz/bigssd/rosbag/src/rosbag2/rosbag2_storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/test_sqlite_storage.cpp:334:11: error: ‘struct rosbag2_storage::StorageOptions’ has no member named ‘storage_preset_profile’
  334 |   options.storage_preset_profile = "resilient";
      |           ^~~~~~~~~~~~~~~~~~~~~~
/home/sloretz/bigssd/rosbag/src/rosbag2/rosbag2_storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/test_sqlite_storage.cpp: In member function ‘virtual void StorageTestFixture_storage_preset_profile_applies_over_defaults_Test::TestBody()’:
/home/sloretz/bigssd/rosbag/src/rosbag2/rosbag2_storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/test_sqlite_storage.cpp:353:82: error: too many initializers for ‘rosbag2_storage::StorageOptions’
  353 |   rosbag2_storage::StorageOptions options{storage_uri, kPluginID, 0, 0, 0, "", ""};
      |                                                                                ^
/home/sloretz/bigssd/rosbag/src/rosbag2/rosbag2_storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/test_sqlite_storage.cpp:355:11: error: ‘struct rosbag2_storage::StorageOptions’ has no member named ‘storage_preset_profile’
  355 |   options.storage_preset_profile = "resilient";

This similarly looks like it's finding the installed header for rosbag2_storage::StorageOptions before the one in the workspace. What looks suspicous to me is -isystem instead of -I includes:  -isystem /opt/ros/rolling/include -isystem /home/sloretz/bigssd/rosbag/install/rosbag2_storage/include -isystem /opt/ros/rolling/opt/yaml_cpp_vendor/include. I'm pretty sure that changes the include order, but I'm struggling to find clear documentation about it. Do you know if anything in rosbag2 is using the  SYSTEM keyword argument to target_include_directories()?

Looks like -isystem is because includes from imported targets are treated as system includes by default: https://cmake.org/cmake/help/v3.0/prop_tgt/NO_SYSTEM_FROM_IMPORTED.html

Looks like -isystem is because includes from imported targets are treated as system includes by default: https://cmake.org/cmake/help/v3.0/prop_tgt/NO_SYSTEM_FROM_IMPORTED.html

One way to build rosbag2_storage_default_plugins is to set -DCMAKE_NO_SYSTEM_FROM_IMPORTED and reorder the includes. Put the build include directories first, then add rosbag2_storage before pluginlib.
diff --git a/rosbag2_storage_default_plugins/CMakeLists.txt b/rosbag2_storage_default_plugins/CMakeLists.txt
index f519aba..264b251 100644
--- a/rosbag2_storage_default_plugins/CMakeLists.txt
+++ b/rosbag2_storage_default_plugins/CMakeLists.txt
@@ -34,20 +34,19 @@ add_library(${PROJECT_NAME} SHARED
   src/rosbag2_storage_default_plugins/sqlite/sqlite_storage.cpp
   src/rosbag2_storage_default_plugins/sqlite/sqlite_statement_wrapper.cpp)

+target_include_directories(${PROJECT_NAME}
+  PUBLIC
+  $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>
+  $<INSTALL_INTERFACE:include>
+)
 ament_target_dependencies(${PROJECT_NAME}
-  pluginlib
   rosbag2_storage
+  pluginlib
   rcpputils
   rcutils
   SQLite3
   yaml_cpp_vendor)

-target_include_directories(${PROJECT_NAME}
-  PUBLIC
-  $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>
-  $<INSTALL_INTERFACE:include>
-)
-
 # Causes the visibility macros to use dllexport rather than dllimport,
 # which is appropriate when building the dll but not consuming it.
 target_compile_definitions(${PROJECT_NAME} PRIVATE

One way to build rosbag2_storage_default_plugins is to set -DCMAKE_NO_SYSTEM_FROM_IMPORTED and reorder the includes. Put the build include directories first, then add rosbag2_storage before pluginlib.
diff --git a/rosbag2_storage_default_plugins/CMakeLists.txt b/rosbag2_storage_default_plugins/CMakeLists.txt
index f519aba..264b251 100644
--- a/rosbag2_storage_default_plugins/CMakeLists.txt
+++ b/rosbag2_storage_default_plugins/CMakeLists.txt
@@ -34,20 +34,19 @@ add_library(${PROJECT_NAME} SHARED
   src/rosbag2_storage_default_plugins/sqlite/sqlite_storage.cpp
   src/rosbag2_storage_default_plugins/sqlite/sqlite_statement_wrapper.cpp)

+target_include_directories(${PROJECT_NAME}
+  PUBLIC
+  $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>
+  $<INSTALL_INTERFACE:include>
+)
 ament_target_dependencies(${PROJECT_NAME}
-  pluginlib
   rosbag2_storage
+  pluginlib
   rcpputils
   rcutils
   SQLite3
   yaml_cpp_vendor)

-target_include_directories(${PROJECT_NAME}
-  PUBLIC
-  $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>
-  $<INSTALL_INTERFACE:include>
-)
-
 # Causes the visibility macros to use dllexport rather than dllimport,
 # which is appropriate when building the dll but not consuming it.
 target_compile_definitions(${PROJECT_NAME} PRIVATE

Thanks for the report! It looks like something else is going on here. I see a different compile error that doesn't involve pybind11_add_module(). I didn't put rcutils from source in the workspace. Should I have?

These errors are expected if you're building the master branch of rosbag2 and you also have binary rosbag2 packages installed; right now there are several packages that have dependency ordering issues similar to the one I reported here with rosbag2_py.  I've got a PR open at https://github.com/ros2/rosbag2/pull/585 that should fix the issue for everything except rosbag2_py, which is what lead me to opening up this issue.
I'll experiment and see if -DCMAKE_NO_SYSTEM_FROM_IMPORTED=1 helps with rosbag2_py...

Thanks for the report! It looks like something else is going on here. I see a different compile error that doesn't involve pybind11_add_module(). I didn't put rcutils from source in the workspace. Should I have?

These errors are expected if you're building the master branch of rosbag2 and you also have binary rosbag2 packages installed; right now there are several packages that have dependency ordering issues similar to the one I reported here with rosbag2_py.  I've got a PR open at https://github.com/ros2/rosbag2/pull/585 that should fix the issue for everything except rosbag2_py, which is what lead me to opening up this issue.
I'll experiment and see if -DCMAKE_NO_SYSTEM_FROM_IMPORTED=1 helps with rosbag2_py...

I'll experiment and see if -DCMAKE_NO_SYSTEM_FROM_IMPORTED=1 helps with rosbag2_py...

The -isystem bit might not be the real issue. Looks like with the diff I posted everything builds fine after a clean build, including rosbag2_py. I'm not sure what -isystem is doing to the search order, but it didn't need to be changed for the issue to go away.

I'll experiment and see if -DCMAKE_NO_SYSTEM_FROM_IMPORTED=1 helps with rosbag2_py...

The -isystem bit might not be the real issue. Looks like with the diff I posted everything builds fine after a clean build, including rosbag2_py. I'm not sure what -isystem is doing to the search order, but it didn't need to be changed for the issue to go away.

It looks like since I opened the original issue, there's been a new Rolling release that has, for better or worse, made the issue obsolete; I just tried again in a brand new ros:rolling container and  rosbag2_py builds fine in my branch.  So the issue probably still exists but is going to be hard to reproduce until there's another API-breaking change...

It looks like since I opened the original issue, there's been a new Rolling release that has, for better or worse, made the issue obsolete; I just tried again in a brand new ros:rolling container and  rosbag2_py builds fine in my branch.  So the issue probably still exists but is going to be hard to reproduce until there's another API-breaking change...


MINOR: Handle some null cases in BrokerMetadataPublisher

Found a few NPEs when running system tests. Added coverage for a few null cases in BrokerMetadataPublisher as well as a unit test

Cherry-picked to 3.0 as 89c9c94f877


Add Disqus support to my blog

Add disqus commenting support to my blog

Added utteranc.es instead under
#33


Refactor Host and related traits into a new traits module

This is a draft implementation of #294. I'll leave this open for
feedback and potentially better trait naming suggestions or better
solutions in general!
Closes #294.
cc @ishitatsuyuki

Seeing as there has been no further feedback on this PR or at #294 I'm going to move on and merge this. Feel free to open a follow-up PR if you feel a better solution exists!


Script halts on "Matching hosts lines" with OpenBSD

Hi!
I was trying the script on OpenBSD 6.8 and the script halts almost at the end, on "Matching hosts lines", is there any debug flag I can use to help?
Thanks!

I wouldn't call it halt, it just takes longer on OpenBSD because GNU awk, sed and grep are much faster.
I just checked hBlock 3.1.1 on OpenBSD 6.8 and FreeBSD 12.2 and all the tests passed correctly.

I'm sorry, you're correct, just ran hBlock with just one small list and it ran with no issues. I guess I'm just using a lot of lists. Sorry for the false alarm!


RegularParser = Uncaught Error: Maximum function nesting level of '256' reached, aborting!

We're using thunderer/shortcode in our Grav CMS shortcode-core plugin: https://github.com/getgrav/grav-plugin-shortcode-core
And it was reported to me that while using also using the Gantry5 Plugin while Shortcode Core was active, resulted in an infinite loop bug.  I dug into this a bit and it appears that some HTML code output by the plugin causes this PHP infinite loop bug when using the RegularParser.  It works fine with the RegexParser or WordpressParser BTW.
I have created a little test project where I'm using JUST the thunderer/shortcode library and the sample content in question, and it's definitely repeatable even out of the context of Grav and the Shortcode Core plugin.
shortcode-infinite-loop.zip
In this file, you can see the HTML that is causing the issue in question.  I've tried 'cleaning' the HTML, but that does not change the results.
My hunch is that it's the HTML data attributes that are throwing the parser for a loop:
...

        Positions

        Menu

...

I'm inclined to disable the Grav Shotcode Core while Gantry5 plugin is processing, as it's really not being used here, it's just that Grav is processing ALL page content for potential shortcodes.

Hi @rhukster, thanks for your report, please look at #72. Can you verify my results using dev-parser-improvements branch?
Your test data runs in ~20ms and uses around 10MiBs of memory in my local environment, Blackfire reports 80ms and ~9MiBs, baseline master ran in 12.5s and used 514MiBs of memory. All tests pass so I'm pretty sure it works correctly, but let me know how does it look on your side.
If everything is fine we have over 100x performance gain on time and memory and RegularParser can be used in all situations without any penalty.

I can confirm that I've tested the parser-improvements branch and the test case I added as part of #72 now works, an also the actual issue as reported to in https://github.com/getgrav/grav-plugin-shortcode-core/issues/53 and https://github.com/gantry/gantry5/issues/2426 have also been resolved!
I notice now that when there are no actual shortcodes, the regular parser is indeed very fast to process the content, in fact, the fastest of all the 3 parsers.
I've not done any comprehensive performance testing on the parsers for actual shortcode work, but it definitely is faster, and I can find no visible issues with my custom shortcodes page I use for testing.
Nice work! 👍

@rhukster During the verification phase, I managed to reproduce #70 and added the fix to #72. Could you please verify the results again? I promise I'm not doing anything else with the PR, so it will be merged once I get your approval.
Unfortunately, the fix reduces the performance gains I mentioned above, but we're still well above 10x improvement in both time and memory (850ms and 47MiB on Blackfire, 660ms and 46MiB on local environment, for your test data vs the same baseline). Details are described in the PR if you're interested.

Please see my comments on the PR itself.  Alas, the issue has returned with the latest commit, but does seem to be related to xdebug being on. (was this way with stable release too, I just didn't realize it before).

Update... latest commits in #72 branch again fixes the issue for me.

If this PR fixed the issue, may I ask you to close it? I will tag the next version soon, I just need to review the commits since the last version and prepare some release notes.

Ya all good for me...

Any word on timing for a new release?

@rhukster I'm very happy to announce that v0.7.0 was just tagged, you can find the release notes here. While I was writing them I found a lot of small things which could be improved, so #73 will probably become v0.7.1 in the near future.
Thanks a lot for using Shortcode!


NDRS-791: Introducing effect for putting a trie into the trie store

@henrytill we're going to need to keep track of the key tries are inserted under, since fast sync can only know its done if there's no pending tries we are waiting to be inserted.
I had the EE also return any trie keys missing beneath the trie we insert, since we'll need to queue them up in fast sync to be fetched from over the wire.

@henrytill we're going to need to keep track of the key tries are inserted under, since fast sync can only know its done if there's no pending tries we are waiting to be inserted.
I had the EE also return any trie keys missing beneath the trie we insert, since we'll need to queue them up in fast sync to be fetched from over the wire.

bors merge

bors merge


Basic formulars as cell content (simple arithmetics)

Problem
I have some use cases for which it is not enough to be able to calculate a "result" for a column.
Solution
I would like to use a column of type "calculation" or "formula" and in which I can put simple calculations.
Additional details
No response

@dclausen01 Thanks for your feature request. I have seen multiple issues related to this. Unfortunately, there are some other big features that seem more urgent. i.e. importing/export and embedding into tables. I am also fixing a few current bugs with the software. This will be on my radar after those things are completed though.


Text Input Sms code is showing on every input, and its showing already read codes.

Two different text inputs on different screens, second screen receive code via sms and we can populate code we get from message with textContentType="oneTimeCode", next step login everything ok. But
If I kill the app and start it again on first screen where I've declared textContentType="none", or what ever else, keyboard will always offer me last sms code received as sms and also on the second screen until new one is received.
Is there a way to prevent this behaviour? I mean the real stuff would be to read just last message and present code to populate. I would call this a bug on textContentType prop.

Don't pass textContentType prop to first screen text input

Don't pass textContentType prop to first screen text input

Tried, does not help.

Sorry, RN issue tracker is not intended for developer support, but for RN issues


about pdf files

is there some plans to display pdf files?
thank's!

For viewing PDF files this library looks nice: https://pub.dev/packages/pdfx

thank you!


stack overflow when trying to use text generation.

When trying to use text generation and template generation, I get the same call stack size exceeded error.
The following is the error it outputs:

[!failure]- Failure
RangeError: Maximum call stack size exceeded

JSON.stringify

plugin:obsidian-dev-tools:128 Object.stringify
plugin:obsidian-dev-tools:128:539

plugin:obsidian-dev-tools:945 ConsolePanel.logArrayEntry
plugin:obsidian-dev-tools:945:62

plugin:obsidian-dev-tools:1047 ConsolePanel.markLogEntry
plugin:obsidian-dev-tools:1047:47

plugin:obsidian-dev-tools:1111 console.eval
plugin:obsidian-dev-tools:1111:174

plugin:obsidian-dev-tools:498 object. [as log]
plugin:obsidian-dev-tools:498:24

plugin:obsidian-textgenerator-plugin:2017 o3.generateStreamInEditor
plugin:obsidian-textgenerator-plugin:2017:3034

plugin:obsidian-textgenerator-plugin:2093 async I.callback
plugin:obsidian-textgenerator-plugin:2093:4733

seems to be an issue with interacting with the dev-tools plugin.  I'm not sure what is interacting weirdly, but for now I will close.


Limited foresight

Hi Jon and Mustafa!
I'm making a pull request for the implementation of carryover storage value function (COSVF) with particle swarm optimization (PSO). It's a pretty large set of code to review.  I've boiled it down to three commit chunks, although not equally sized!:

documentation restructure: I'm organizing the documentation into markdown files under sections in the ./documentation directory. These are rendered nicely in the GitHub integrated GitBook which I've set up; I'll send you invitations to join that.
limited-foresight with pso: this is all the new code for the COSVF-PSO. I think it's a good milestone to push now and get some eyes on it. All of this was used for a pilot study, the two-sr-pilot-lf, which is a small network of two surface and groundwater reservoirs representing Tuolumne and Merced subsets of the CALVIN network. I wrote up a report (for ECI273!) which describes the results so far.
Create get-model-data.py: this is just a script to download the Calvin network links referenced in the Quick Start doc, as well as the two-sr-pilot-lf example files which can be used to run the COSVF-PSO.

So I will keep developing on my limited-foresight branch and pushing. Let me know if a conference call would be good to step through the main additions.
The next updates I intend to make are:

finish out the documentation on the additions
adding a linear penalty cost (function) for groundwater aquifer (this is more in line with Draper's original control on groundwater overdraft and a shift away from the pilot's idea of limiting groundwater storage capacity)
implement a ring topology for the PSO global best position
add plotting and post-processing code and a folder of Jupyter notebooks to demonstrate their use

Sending my best!

Hi Jon and Mustafa,
After more testing with the full Calvin network, looks like the particle swarm search approach is not working. I’m going to close this pull request until another approach can be worked out. So far I’ve been testing with multi-objective evolutionary search.
Thanks,
Wyatt


(Draft) Merge dev into main

todo:

[ ] address outstanding issues
[ ] adequate testing in vitro and in vivo
[ ] bump version
[ ] reset alive branch to main

Trio
This app splits from Artificial-Pancreas/iAPS at v2.3.3 on the main branch and this commit. Very minimal UI changes were made in this initial update, but while it may look and for the most part seem the same as iAPS v2.3.3, you can see from the number of commits and files/lines of code changed in this PR that there are extensive changes under the hood. These changes were done to ensure it is truly open source, peer-reviewed and tested, to keep it in line with updates from Loop's submodules (the stuff that connects your pump, cgm, and services), to fix some bugs, and to add a few new features.
The Team
Trio is a collaborative project set up by members of the DIY community who welcome a peer reviewed, Open Source iOS implementation of the Oref/APS algorithm.
The team consists of:

Software architects
Developers
Code reviewers
Designers
Testers
Documentation specialists
Project managers / agile coaches

Anyone who wants to contribute is welcome to join the team!
The Name
The name is inspired by several joint efforts:

Algorithm/CGM/Pump: the three components that make up your DIY loop
Basal/CR/ISF: the three main settings that, when tuned, treat your TxD
Carbs/Fat/Protein: the three main food components that impact your blood glucose
FAX/iAPS/Trio: the three stages of Oref/APS leading to the birth of Trio
In true DIY fashion, make the Trio definition individualized to you

Hosted by The Nightscout Foundation
The most obvious change is that the repository location has changed to be housed by The Nightscout Foundation, alongside Nightscout and AndroidAPS. The Nightscout Foundation has a long track record of facilitating innovation in DIY diabetes technology and stoking collaboration between many developers and projects. This will keep Trio in the hands of the community rather than a single individual, and ensure all development is peer-reviewed and tested before being released.
New Bundle ID
If you're currently using iAPS, another big change is the app's Bundle ID, which is now org.nightscout.TEAMID.trio. This means Trio will not overwrite the iAPS app on your iPhone, but all settings must be re-entered and all devices re-connected. Trio will need about 24 hours of data before the dynamic features can be enabled. If you're using Omnipod DASH or Eros, you'll have to pair a new pod to start using Trio, but all other devices can transfer mid-session.
New Icons
New icons have been created as a symbol of the name Trio.
(by @tmhastings), plus some old ones (by @tqb43)

Default Setting Changes
Default CGM: None (instead of Nightscout)
Show high/low lines on main chart: On (instead of Off)
High value for main chart lines and statistics: 180 (instead of 145)
Sensitivity Raises Target: Off (instead of On)
(by @MikePlante1 with input from @bjornoleh and many others)
Sigmoid Adjustment Factor
With Dynamic ISF toggled on, the effects of Adjustment Factor vary drastically between having Sigmoid turned on and off. Because of this, we've split the Adjustment Factor in two and given Sigmoid its own setting. This allows Adjustment Factor to be defaulted to 0.8 with Sigmoid turned off and to 0.5 with Sigmoid turned on. (by @MikePlante1)

External Insulin & Glucose
External insulin entry (to log non-pump insulin, eg. from an insulin pen) has been moved to the History screen rather than the Bolus screen. External insulin entries will display a warning if amount is larger than Max Bolus setting and will not be allowed if over three times the Max Bolus setting. The History screen also now displays if a bolus was an SMB or External, and uses swipe gestures to delete entries. (by @BrianWieder with many cherry picks from @dnzxy)

History
Add External Insulin

LoopKit Submodules
The code that's used to connect to pumps, CGMs, and other things was taken from their implementations in Loop. This update changes where that code is housed from within this app to point to the code in @LoopKit's repositories. This allows for a better development as it makes it easy to see the differences between the code used in Trio vs the code being used in LoopKit and allows updates in the LoopKit repo to easily be pulled into Trio. (by @bjornoleh with help from @marionbarker and @bjornoleh)
Oref
This part is rather technical. Oref (the algorithm from OpenAPS that drives Trio) is written in the JavaScript programming language. In order to perform well in an iOS app, the algorithm is implemented in a minimized format. Until the algorithm is rewritten into the Swift programming language, it needs to be like this to ensure performance. The source code used to build the minimized files is now included in this repo as well as a separate repo trio-oref. This work is quite extensive and has been done and peer-reviewed by several contributors: (beautified/reconstructed by @JeremyStorring and updated by @bjornoleh, @mountrcg, @avouspierre, @MikePlante1, @tmhastings)
Schedule SMBs Off in Profile Overrides
The Schedule when SMBs are Off option in profile overrides has been improved to make it easier to understand and use. Now, if you want to disable SMBs for the entire time the override is active, toggle on Always Disable SMBs, but if you want to specify a timeframe for SMBs to be disabled, you can toggle off Always Disable SMBs toggle on Schedule when SMBs are Off and set the timeframe. Note that setting a timeframe of 22-6 would disable SMBs from 22:00 (10PM) through 5:59 as long as the override is active during that time.

Apple Health Fix
The 2.3.3 version of iAPS had a bug with Apple Health. This part of the code was re-written and fixed (by @AndreasStokholm), then thoroughly tested and peer-reviewed by the team. Feel free to toggle Apple Health back on if you'd like.
Tidepool Integration
Tidepool can now be uploaded to directly from Trio instead of using Apple Health and the Tidepool app on your iPhone. Note that if both are enabled, you will see duplicate entries in Tidepool, and even after turning one off, you could still see up to a week of duplicate data. The Tidepool integration was written by (@avouspierre), then thoroughly tested and peer-reviewed by the team.
Meal Settings
There is a new section in settings called Meal Settings. In it, you'll find Max Carbs (which used to be in Pump Settings) and the Fat and Protein Conversion Settings that used to be found in it's own section. (by @bjornoleh)

⚙
Meal Settings

Max Bolus
Manually entering a bolus larger than the Max Bolus setting used to just deliver the max bolus amount, but now this setting prevents the bolus from being delivered until the amount is lower than the max setting. (cherry-picked from @Jon-b-m and modified by @MikePlante1 with input from @bjornoleh)

Live Activities (Dynamic Island & Lockscreen Widget)
Live Activities have been added to show relevant data without opening Trio. This displays glucose, delta, and trend on the lock screen (iOS 16.2+) as well as in the Dynamic Island (iPhone 14+). Optionally, a glucose graph and current IOB & COB can also be displayed. Consistency updated every 5 minutes with every loop cycle. (cherry-picked from @nas10, @polscm32 and resolved by @MikePlante1)

Live Activity
Screenshot

Lockscreen
.

Dynamic Island

Dynamic Island(long press)
.

Dexcom ONE+
Added support for Dexcom ONE+ (taken from this Zulip discussion)
Tests Enabled
Added ability to run tests on Trio and submodules. (by @avouspierre)
Browser Build Updates
Browser Build updated to new Apple/GitHub standards. (by @bjornoleh)
TestFlight Expiration is now displayed in Settings. (by @bjorkert)
List of individual PRs included in this large PR:

#1
#3
#5
#6
#14
#15
#17
#19
#22
#26
#28
#31
#34
#36
#39
#41
#42
#43
#46
#48
#50
#52
#55
#59
#60
#61
#62
#63
#66
#67
#68
#72
#74
#75
#76
#78
#79
#80
#81
#82
#83
#86
#89
#90
#91
#93
#94
#95
#97
#99
#100
#101
#103
#110
#112
#114
#116
#117
#118
#120
#121
#122
#124
#125
#127
#128
#134
#135
#140
#141
#143
#143
#148
#149
#150
#151
#155
#156
#157
#160
#163
#164
#165
#166
#168
#169
#170
#172
#173
#176

Go Team!🎉
Thank you to the following developers who have added commits to this PR and the initial version of Trio:
@AndreasStokholm, @avouspierre, @bjornoleh, @BrianWieder, @JeremyStorring, @kylmcw, @lifeguardz, @marionbarker, @MikePlante1, @Sjoerd-Bo3, @tmhastings
A heartfelt thank you to everyone who has played a crucial role behind the scenes with their valuable contributions in discussions, project management, coaching, documentation, testing, and more:

Dana Lewis (@danamlewis) and Scott Leibrand (@scottleibrand) for their unwavering support and guidance regarding our Oref inquiries.
Tim Street for his invaluable insights into the implementation of Oref code.
Tim Gunn (@Tornado-Tim) for his dedicated efforts in setting up and maintaining the Nightscout hosting of the Trio repository.
Auggie Fisher (@aug0211) for generously sharing the Discord server for Trio and providing ongoing support for both current and future developments of Trio's caregiver implementation.
Pete Wilkins (@petewilkins) for optimizing the Trio Discord home base throughout development and in anticipation of future discussions.
Marion Barker (@marionbarker) for her invaluable contributions, including creating the Build Scripts for Trio, enhancing documentation, and offering expertise in various areas not fully reflected on Github.
Mike Plante (@MikePlante1) for taking the lead and ensuring a successful initial launch of Trio.
Sjoerd Bozon (@Sjoerd-Bo3) and Bjørn Ole (@bjornoleh) for their tireless efforts in driving collaboration and continuously advancing the project.
Kelsey Huss (@kelseyhuss) for stepping up without hesitation to communicate progress to the community.
Theresa Hastings (@tmhastings) for pulling all the strings, getting the right people involved and spending much of her days on this project one way or another. Without you this would never have happened!
Tom Barrows (@tqb43) for sharing your branding and logo expertise and providing support to users in the Facebook group.
Magnus Reintz (@t1dude) for his consistent availability (🦇) and invaluable assistance with Facebook community management, clear language and translation management.
Jonas Hummelstrand for his commitment to maintaining open communication on Facebook, even during challenging times.
Dan (@dnzxy) for consistently fostering discourse aimed at enhancing and enriching the DIY community and Trio project.
To Dan, Marvin, Andreas, Mike, Tom, Magnus, Kelsey, and Theresa, Trio owes you its inception. Your proactive approach and DIY spirit didn't just recognize the critical safety aspect but also reshaped the implementation of Oref on iOS. Your bold actions turned necessity into reality. Thank you for your invaluable contributions.

@t1dude @tmhastings Could I get a shoutout as an agile coach? I kid. Actually, based on Theresa's brilliant idea to take up NS on their offer to host Trio, I reached out to James Wedding and introduced him to @AndreasStokholm -- and James said: Whatever you need. I believe it took several others to get the repo set up in NS, and in fact, at one point it was at the wrong NS domain. And super early on I suggested we use some of Princeton's “Ground Rules for Respectful Discussion” which after much work by @aug0211 became a code of conduct, which I thought was really important to set the stage early on to avoid what had happened in the past. Any mention of any of this would be appreciated, tiny as it is, b/c this stuff is really important to me as I know it is to all of us. thank you!


[Feature Request] Remove background side borders that encompass open windows

Implemented but not yet on main store release. It is called Borderless Window in Window Settings (read tips to see how to open it). You are welcome to clone and build the project to get it.

Now available for Experimental InDev Version, link in Readme


Minilib.Task.Future should not be freezed when the task pool is shut down

Currently it is freezed when the task pool is shut down.

Resolved by #36.


⚠ Secret Website 3 has degraded performance

In 4fe8d83, Secret Website 3 ($SECRET_WEBSITE_3) experienced degraded performance:

HTTP code: 200
Response time: 2874 ms

Resolved: Secret Website 3 performance has improved in db6e292.


[Registry Mobile][WP theme] Remove sections from mobile version

"About the Digital Public Good Standard" Button

Acceptance Criteria:
URL: https://digitalpublicgoods.net/registry/
"About the Digital Public Good Standard" is visible on desktop

Set screen size to mobile such as iPhone SE, and "About the Digital Public Good Standard" text and link should disappear.


English translation for frontend page "about"

Please find the english translation for the frontend page "about" in the attachment.
Uber-JILA---statische-Seite-522917-EN_1.docx
There are also two additions to the german frontend page (in yellow):
Über JILA - Datenstand DE_1.docx

Added the paragraphs to the German version.
The About page was previously defined to be monolingual, so we did not implement provisions for internationalisation

Ok I didn't know that, as there was an english "about" page implemented already when I started working on JILA.
Does this mean there is no technical possibilty at the moment to fill in an english version, like on the "persons" page?


Reduce CI costs

cargo clippy and cargo doc need to do the same work as cargo build, so put them in the linux job, and reduce the duplicate effort.
In addition, use the free windows runner as the cost for solang-windows-latest is much too high. @ryjones
Before this PR, we would running clippy with the latest stable. After this PR, we run clippy with 1.70.0 (or the current version we're using for building, rather than latest).

This PR was merged before I could submit my review.

For future reference, @ryjones in our project, the PR author merges once all the maintainers have reviewed the change and given their  :heavy_check_mark: approval


🐛 BUG: wrangler build documentation is missing from wrangler --help

Which Cloudflare product(s) does this pertain to?
Wrangler core
What version(s) of the tool(s) are you using?
3.57.0
What version of Node are you using?
20.12.2
What operating system and version are you using?
macOS 13.6.3
Describe the Bug
Observed behavior
wrangler build command documentation / description is missing from wrangler --help output.
Expected behavior
wrangler --help must show description of the wrangler build command.
Steps to reproduce
> wrangler --version
 ⛅ wrangler 3.57.0 (update available 3.57.1)
-------------------------------------------------------
> wrangler --help
wrangler

Commands:
  wrangler docs [command..]            📚 Open wrangler's docs in your browser
  wrangler init [name]                 📥 Initialize a basic Worker project, including a wrangler.toml file
  wrangler generate [name] [template]  ✨ Generate a new Worker project from an existing Worker template. See https://github.com/cloudflare/workers-sdk/tree/main/templates
  wrangler dev [script]                👂 Start a local server for developing your worker
  wrangler deploy [script]             🆙 Deploy your Worker to Cloudflare.  [aliases: publish]
  wrangler delete [script]             🗑  Delete your Worker from Cloudflare.
  wrangler tail [worker]               🦚 Starts a log tailing session for a published Worker.
  wrangler secret                      🤫 Generate a secret that can be referenced in a Worker
  wrangler secret:bulk [json]          🗄  Bulk upload secrets for a Worker
  wrangler kv:namespace                🗂  Interact with your Workers KV Namespaces
  wrangler kv:key                      🔑 Individually manage Workers KV key-value pairs
  wrangler kv:bulk                     💪 Interact with multiple Workers KV key-value pairs at once
  wrangler pages                       ⚡ Configure Cloudflare Pages
  wrangler queues                      🇶 Configure Workers Queues
  wrangler r2                          📦 Interact with an R2 store
  wrangler dispatch-namespace          📦 Interact with a dispatch namespace
  wrangler d1                          🗄  Interact with a D1 database
  wrangler hyperdrive                  🚀 Configure Hyperdrive databases
  wrangler ai                          🤖 Interact with AI models
  wrangler vectorize                   🧮 Interact with Vectorize indexes
  wrangler pubsub                      📮 Interact and manage Pub/Sub Brokers
  wrangler mtls-certificate            🪪 Manage certificates used for mTLS connections
  wrangler login                       🔓 Login to Cloudflare
  wrangler logout                      🚪 Logout from Cloudflare
  wrangler whoami                      🕵  Retrieve your user info and test your auth config
  wrangler types [path]                📝 Generate types from bindings & module rules in config
  wrangler deployments                 🚢 List and view details for deployments
  wrangler rollback [deployment-id]    🔙 Rollback a deployment

Flags:
  -j, --experimental-json-config  Experimental: Support wrangler.json  [boolean]
  -c, --config                    Path to .toml configuration file  [string]
  -e, --env                       Environment to use for operations and .env files  [string]
  -h, --help                      Show help  [boolean]
  -v, --version                   Show version number  [boolean]

A git repo we can clone and run a test suit on, or which has a README with step-by-step instructions, is even better. In this case, please use the field below to provide a link to the minimal repro.
Please provide a link to a minimal reproduction
No response
Please provide any relevant error logs
No response

wrangler build is deprecated: https://developers.cloudflare.com/workers/wrangler/deprecations/#build
Closing.


Upgrade to node 12

Description
A few sentences describing the overall effects and goals of the pull request's commits.
What is the current behavior, and what is the updated/expected behavior with this PR?
Other changes
Describe any minor or "drive-by" changes here.
Tested
An explanation of how the changes were tested or an explanation as to why they don't need to be.
How others should test
Does this need to be tested by QA in the next release cycle? If so please give a brief explanation of how to test these changes.
Related issues

Fixes #[issue number here]

Backwards compatibility
Brief explanation of why these changes are/are not backwards compatible.

Can we close this one since we have #477?

Closed in favor of #477


Toast When Clearing Pings (Dev)

Test 49 requires us to have a toast pop up when the user clears his pings. This is just so he knows that the operation was successful.

1 minute
fixed


Add YugabyteDB to list of third party DB Backends

YugabyteDB is a Postgres compatible database and has a backend for Django overriding features supported differently in YugbayteDB as compared to Postgres.
We would love to get listed on the list of third party backends on the databases page.
Please let us know in case any changes are required to help us get listed here.
Thanks!

We have run the django test suite to verify that it works. The usage is also documented here to confirm that it works.
** Note- The module name was recently changed from yb-bqckend to django_yugabytedb which is not yet updated in the doc

We have run the django test suite to verify that it works. The usage is also documented here to confirm that it works.

But we cannot verify this without testing it manually. Moreover, in the future, we will not be able to easily check if it supports the latest versions of Django. Please, first set up CI in django-yugabytedb with all supported versions of Django.

Okay, we will setup a CI and give an update here. Thanks!

@felixxm We have added the CI setup, with the migrations and schema test modules added. Please take a look. Thanks!

IMO, there is more work to do:

merge CI,
add classifiers, e.g. Framework :: Django, Framework :: Django :: 3.2, Framework :: Django :: 4.0.
prepare a stable release (django-yugabytedb==4.0.0 is pre-alpha),
update outdated docs, e.g.

"Installing in Python Virtual Environment From Source": you've released on PyPI so users don't have to install from source anymore

document more caveats:

dropping primary keys is not supported,
changing the column types doesn't work in many cases.

@felixxm  All the changes are made according to the previous comment. Please let us know the next steps. Thanks!

@felixxm - Please let us know if this PR can be merged now? Please treat this on priority, many of our community users looking forward to this integration.

@felixxm - Please let us know if this PR can be merged now? Please treat this on priority, many of our community users looking forward to this integration.

This PR is only about mentioning django-yugabytedb in the list of 3rd-party database backend and does not in any way block folks from using it.

@felixxm, since the CI setup PR is approved and merged now, could you tell the next steps to get this one approved as well.

@Sfurti-yb Thanks :+1:  Welcome aboard :boat:

Thank you @felixxm for being such a great help. When do the changes made here reflect on this page


Index out of range exception

so DateFavorited is a nullable DateTime.
In some combinations, such as when both TestVm's have a date set, and you remove one from the ItemCache, you get an index out of range error.
if you comment out the first item's date and remove the second one from the ItemCache, there is no error.
It also only happens if you are sorting on the DateFavorited field.
once this error occurs, changes to the itemcache stop refreshing the UI properly, so it's pretty nasty.
using System;
using System.Reactive.Linq;
using System.Windows;
using DynamicData.Binding;
using DynamicData;
using DynamicData.Controllers;
using DynamicData.Operators;

namespace WpfApplication3
{
    public partial class MainWindow : Window
    {
        public ObservableCollectionExtended<TestVm> Items { get; set; }
        public SourceCache<TestVm, int> ItemCache;
        public TestVm SelectedItem { get; set; }
        public PageController PageController;

        public MainWindow()
        {
            InitializeComponent();

            PageController = new PageController();
            var sortController = new SortController<TestVm>(SortExpressionComparer<TestVm>.Ascending(t => t.DateFavorited ?? DateTime.MinValue));
            var filterController = new FilterController<TestVm>(myVm => myVm.Id != 0);
            Items = new ObservableCollectionExtended<TestVm>();
            ItemCache = new SourceCache<TestVm, int>(myVm => myVm.Id);

            var item1 = new TestVm(1) { DateFavorited = DateTime.Now };
            var item2 = new TestVm(2) { DateFavorited = DateTime.Now };

            ItemCache.AddOrUpdate(item1);
            ItemCache.AddOrUpdate(item2);

            ItemCache.Connect()
                            .Filter(filterController)
                            .Sort(sortController).Do(OnNext)
                            .Page(PageController).Do(OnNext) //error doesnt occur with paging disabled
                            .ObserveOnDispatcher()
                            .Bind(Items)
                            .Subscribe();

            PageController.Change(new PageRequest(1, 100));

            DataContext = this;
        }

        private void OnNext(IChangeSet<TestVm, int> changeSet)
        {

        }

        private void BtnRemove_OnClick(object sender, RoutedEventArgs e)
        {
            if (SelectedItem != null)
            {
                SelectedItem.DateFavorited = null;
                ItemCache.Remove(SelectedItem); //ERROR!
            }
        }
    }

    public class TestVm
    {
        public int Id { get; set; }
        public DateTime? DateFavorited { get; set; }

        public TestVm(int id)
        {
            Id = id;
        }
    }
}

<Window x:Class="WpfApplication3.MainWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        Title="MainWindow" Height="350" Width="525">
    <Grid>
        <Grid.ColumnDefinitions>
            <ColumnDefinition />
            <ColumnDefinition />
        </Grid.ColumnDefinitions>

        <ListBox SelectedItem="{Binding SelectedItem}" ItemsSource="{Binding Items}" />

        <StackPanel Orientation="Vertical" Grid.Column="1">
            <Button Name="BtnRemove" Content="Remove" Click="BtnRemove_OnClick" />
        </StackPanel>
    </Grid>
</Window>

I will look into this over the weekend

I found the problem, will fix and deploy over the weekend

this caused me so much pain i am glad to hear it

The problem was exactly what you found - sorting on a mutable value. When the page mechanism applies page logic it is relying on a binary search.  Changing a value of an object effectively changes the sorting and therefore breaks the binary search and it could not therefore find the object in the list. I have removed BinarySearch() and replaced it with the less efficient IndexOf() operator and the problem goes away. Although less efficient the performance impact should be negligible.

Fixed and unit test added, Try installing Dynamic Data <IP_ADDRESS>9 and let me know how you get on


NobodyNada: Watch jobsnotice.in

NobodyNada requests the watch of the watch_keyword jobsnotice.in. See the Metasmoke search here

!!/approve


[AMBARI-25286] : Avoid conditional logic for creating ViewURLResource

What changes were proposed in this pull request?
Removing conditional logic inside createViewUrlResource, removing redundant AuthorizationException and Optional as argument
How was this patch tested?
View APIs were tested with changes successfully

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/5242/
Test PASSed.

@g-boros please review

Please review @g-boros

@virajjasani  please open a PR against branch-2.7 to have this included in the next maint release

Created PR: https://github.com/apache/ambari/pull/2999
Thanks @g-boros

Please review @zeroflag


replace sigint signal handler, with atexit, to let run in a thread

if there is merge conflict: final result example:
https://github.com/shimondoodkin/zero/blob/main/zero/server.py#L106-L111

Hi @shimondoodkin thanks for the PR!
Can you please add tests for using threads? TIA

Hi @shimondoodkin thanks for the PR!
Can you please add tests for using threads? TIA

maybe yes. i will look into it (if all ok this week).

Hi @shimondoodkin just a gentle knock, if you have some time on this. I appreciate the work! :pray:

Hi @shimondoodkin thanks for the tests. I found some issues for server termination in the tests. I see you are trying to fix them, thanks for that! 🙌 But if you want you can ignore that. Just test for the threads should be enough. I am trying to fix them in this PR => https://github.com/Ananto30/zero/pull/28

Hi @shimondoodkin I have fixed the server close issue in this PR => https://github.com/Ananto30/zero/pull/28 (exactly these lines server.py#111, server.py#137 and making fixtures in specific folders tests/single_server/conftest.py#14, tests/multiple_servers/conftest.py#10)
So I already merged to main, if you rebase, you dont have to spend time on solving that (hopefully, same thing helps in thread)

here it returns value by reference if it works then it works
https://github.com/shimondoodkin/zero/blob/replace-signal-handler-atexit/tests/server_test.py#L72

i was trying to solve the exit problem.
using zmq pooling worked for me in another project and it exited ok nicely and avoided blocking in recv indefinitely.
seems it exiting well when using sigint but not when using sigterm.
seems it is stuck in native module in socket.recv().
maybe zmq doesn't handle sigterm, maybe only sigint.
or
maybe sigterm is switched. maybe it is the reason why it is not working.  could be that it is not calling the previous  original signal sig term signal handler..

@shimondoodkin very good point! I have introduced polling for the sync client. For the async client, there are still some issues. And for the server, I haven't tried polling. Maybe I should.


Change onclick not on mouseup

I notice that in all demos, change happen only after mouse click, I want as soon as mouse button is pressed

Mousedown and mouseup trigger on the same element will trigger the click event.
Bind the mouseup event on the component will not change the value...

Sorry, I have to clarify.
Slider do not behave as native .
When you click in different area than knob(to change value), it do not change value immediately(have to release mouse button), also after that if you keep press mouse down you can't just move it to change value.

Yes, this is currently the case. The click event does not determine when it was pressed until it was released.

+1 for this.
@NightCatSama could we have an option for this in the future?
The default behaviour in both native browser html sliders and other polyfills (https://rangeslider.js.org) is to update the value/dot position on mousedown not on mouseup.
I think it's a much better user experience to be able to click down on a part of the slider, have the slider update its value and dot position, then be able to continue dragging to refine the value.
Otherwise you have to click twice.
A first click to get the dot in position and then a second click on the dot to refine the value.


[FEA] Add support for HiveTableScanExec

Describe the bug
Steps/Code to reproduce bug
this is my configuration of pyspark shell
pyspark --jars /root/hadoop/mysql-connector-java-5.1.37-bin.jar,/root/downloads/rapids-4-spark_2.12-21.10.0.jar,/root/downloads/cudf-21.10.0.jar,/root/downloads/rapids-4-spark-tools_2.12-21.10.0.jar,/root/downloads/xgboost4j_3.0-1.4.2-0.1.0.jar,/root/downloads/xgboost4j-spark_3.0-1.4.2-0.1.0.jar --master yarn --deploy-mode client  --num-executors 2 --executor-cores 1 --executor-memory 1g --driver-memory 1g \
--conf spark.sql.session.timeZone=UTC \
--conf spark.driver.extraJavaOptions=-Duser.timezone=UTC \
--conf spark.executor.extraJavaOptions=-Duser.timezone=UTC \
--conf spark.plugins=com.nvidia.spark.SQLPlugin         \
--conf spark.kryo.registrator=com.nvidia.spark.rapids.GpuKryoRegistrator \
--conf spark.resources.discoveryPlugin=com.nvidia.spark.ExclusiveModeGpuDiscoveryPlugin \
--conf spark.rapids.shims-provider-override=com.nvidia.spark.rapids.shims.spark312.SparkShimServiceProvider \
--conf spark.rapids.memory.gpu.pooling.enabled=true    \
--conf spark.rapids.python.memory.gpu.allocFraction=0.8 \
--conf spark.rapids.python.memory.gpu.maxAllocFraction=0.9 \
--conf spark.executor.resource.gpu.amount=1   \
--conf spark.task.resource.gpu.amount=0.5 \
--conf spark.executor.resource.gpu.discoveryScript=/root/hadoop/getGpusResources_executor.sh \
--conf spark.rapids.sql.enabled=true  \
--conf spark.rapids.sql.incompatibleOps.enabled=true \
--conf spark.rapids.sql.concurrentGpuTasks=2 \
--conf spark.rapids.sql.udfCompiler.enabled=true \
--conf spark.rapids.sql.format.csv.read.enabled=true \
--conf spark.rapids.sql.format.csv.enabled=true \
--conf spark.rapids.sql.variableFloatAgg.enabled=true \
--conf spark.rapids.sql.explain=ALL \
--conf spark.rapids.sql.batchSizeBytes=512M \
--conf spark.sql.hive.convertMetastoreParquet=true \
--py-files /root/downloads/sparkxgb.zip

then I run sparksql(connect to hive, saprk thrift server)
df=spark.sql("""
select "1015a" `station`, t1.d1 `datetime`, cast(t1.`1015a_O3` as int) `O3`, cast(t2.`1015a_PM10` as int) `PM10`,
cast(t3.`1015a_PM25` as int) `PM25`,cast(t4.`1015a_SO2` as int) `SO2`,cast(t5.`1015a_NO2` as int) `NO2`,
cast(t6.`1015a_CO` as float) `CO`, cast(t7.`1015a_CO` as int) `target`
from
(select `datetime` as d1,`1053a` as `1015a_O3` from aq_station where  `type`="O3") t1
join
(select `datetime` as d2,`1015a` as `1015a_PM10` from aq_station  where  `type`="PM10") t2
on (t1.d1=t2.d2)
join
(select `datetime` as d3,`1015a` as `1015a_PM25` from aq_station  where  `type`="PM2.5") t3
on (t1.d1=t3.d3)
join
(select `datetime` as d4,`1015a` as `1015a_SO2` from aq_station  where `type`="SO2") t4
on (t1.d1=t4.d4)
join
(select `datetime` as d5,`1015a` as `1015a_NO2` from aq_station  where  `type`="NO2") t5
on (t1.d1=t5.d5)
join
(select `datetime` as d6,`1015a` as `1015a_CO` from aq_station  where `type`="CO") t6
on (t1.d1=t6.d6)
order by `datetime`
""")

I got such errors:
 !NOT_FOUND <HiveTableScanExec> cannot run on GPU because no GPU enabled version of operator class org.apache.spark.sql.hive.execution.HiveTableScanExec could be found
                                                        @Expression <AttributeReference> datetime#3451 could run on GPU
                                                        @Expression <IsNotNull> isnotnull(type#3453) could run on GPU
                                                          @Expression <AttributeReference> type#3453 could run on GPU
                                                        @Expression <EqualTo> (type#3453 = PM10) could run on GPU
                                                          @Expression <AttributeReference> type#3453 could run on GPU
                                                          @Expression <Literal> PM10 could run on GPU
                                            *Exec <BroadcastExchangeExec> will run on GPU
                                              *Exec <ProjectExec> will run on GPU
                                                *Expression <Alias> datetime#5165 AS d3#4 will run on GPU
                                                *Exec <FilterExec> will run on GPU
                                                  *Expression <IsNotNull> isnotnull(datetime#5165) will run on GPU

Expected behavior
The code should work.
Environment details (please complete the following information)

Environment location: YARN, hadoop3.2.2, hive3.1.2, spark3.1.2, scala2.12.11
Spark configuration settings related to the issue

Additional context
I want to run the code on gpu, but now it is still run on cpu.

the spark settings above allocated two gpus, when I run sparksql, the volatile gpu-util of each gpu is 5%

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.91.03    Driver Version: 460.91.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 208...  Off  | 00000000:04:00.0 Off |                  N/A |
| 27%   30C    P8     5W / 260W |  10233MiB / 11019MiB |      5%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  GeForce RTX 208...  Off  | 00000000:05:00.0 Off |                  N/A |
| 27%   31C    P8    21W / 260W |      1MiB / 11019MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  GeForce RTX 208...  Off  | 00000000:08:00.0 Off |                  N/A |
| 27%   31C    P8     5W / 260W |      1MiB / 11019MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  GeForce RTX 208...  Off  | 00000000:09:00.0 Off |                  N/A |
| 27%   31C    P8    21W / 260W |      1MiB / 11019MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   4  GeForce RTX 208...  Off  | 00000000:85:00.0 Off |                  N/A |
| 27%   33C    P8    21W / 260W |      1MiB / 11019MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   5  GeForce RTX 208...  Off  | 00000000:86:00.0 Off |                  N/A |
| 27%   32C    P8     7W / 260W |      1MiB / 11019MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   6  GeForce RTX 208...  Off  | 00000000:89:00.0 Off |                  N/A |
| 27%   31C    P8     5W / 260W |      1MiB / 11019MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   7  GeForce RTX 208...  Off  | 00000000:8A:00.0 Off |                  N/A |
| 27%   32C    P8     1W / 260W |      1MiB / 11019MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      4091      G   /usr/lib/xorg/Xorg                 33MiB |
|    0   N/A  N/A      8262      C   ...0-openjdk-amd64//bin/java    10195MiB |
+-----------------------------------------------------------------------------+

Thanks for the report, @sybest1259!

The code should work.

To clarify, did the query not execute correctly even with the table scan being performed by the CPU?

Thanks for your anwser, @jlowe
I run pysparkshell with 2 cpu cores and 2 gpu cards, the sql statement is simple, it took about 1 minute to finish the job with the warning above. GPU-Util is very low but cpu is very high.
I got jars from https://mvnrepository.com/: rapids-4-spark_2.12-21.10.0.jar, cudf-21.10.0-cuda11.jar
when I check pom.xml in the source code of rapids-4-spark and cudf from https://github.com/NVIDIA/spark-rapids and https://github.com/rapidsai/cudf/tree/branch-21.10, I find that the default version of scala is 2.12.8, spark is 3.0.1. I wonder if I need to change their version to 2.12.11 and 3.1.2(I am using now), if that warning is  due to the incompatible compilation.

I have tried to recompile rapids-4-spark_2.12-21.10.0.jar with spark3.1.2(I also have changed <hive.storage.api.version>2.8.1</hive.storage.api.version> in pom.xml), but it still do not work. I got the same warning : !NOT_FOUND  cannot run on GPU because no GPU enabled version of operator class org.apache.spark.sql.hive.execution.HiveTableScanExec could be found

when I use the command below, the job takes 10 seconds. So I think GPU is not really used
pyspark --jars /root/hadoop/mysql-connector-java-5.1.37-bin.jar --master yarn --deploy-mode client  --num-executors 20 --executor-cores 2 --executor-memory 2g --driver-memory 10g

it took about 1 minute to finish the job with the warning above. GPU-Util is very low but cpu is very high.

So the query did complete successfully despite the warning?  That is the expected behavior.  The warning is not an error, it's just informing the user that the HiveTableScanExec in the query plan will not be replaced because the GPU does not support that query operation yet.  This warning would not be emitted at all with the default settings for spark.rapids.sql.explain.  It arguably should be an info-level message, but spark-shell by default suppresses all logging below warnings, so we decided to log it at a warning level so users would see it even in the shell.

I wonder if I need to change their version to 2.12.11 and 3.1.2(I am using now), if that warning is due to the incompatible compilation.

The warning is not due to an incompatible compilation.  The message is correctly stating that there is no GPU override for HiveTableScanExec.  The RAPIDS Accelerator does not support that query operation, so it is falling back to the CPU implementation to ensure the query still runs.
It sounds like this query is operating properly (i.e.: producing the correct results) but processing it mostly on the CPU because the RAPIDS Accelerator does not yet support HiveTableScanExec.  If so, this is technically a feature request and not a bug, as the RAPIDS Accelerator does not claim that every query operation is supported by the RAPIDS Accelerator.  See the supported operations document for details on what specific query operations and types are supported by the RAPIDS Accelerator.
I'm updating this to a feature request, and we can look into what it would take to support accelerating HiveTableScanExec with RAPIDS.

Thanks very much. @jlowe
Spark-rapids is quite a useful tool, it could bring us efficience and convenience.
As my team is working on sparksql-on-gpu, we look forward to the development of this function as soon as possible.
I‘d like to convey my heartfelt gratitude to all the developers.

@sybest1259
I can't repro when just simply reading a hive and doing join. Could you provide a minimum repro step about how to create the hive table and the data?  Thx

I created table in hive shell:
CREATE external TABLE aq_station (date string,hour int,1013a string,1014a,...,2882a string, datetime TIMESTAMP) partitioned by(year int,type string);
Then I imported data into table via pysaprk:
from pyspark.sql import HiveContext
from pyspark import SparkContext
from pyspark import SparkConf
from pyspark.sql import SparkSession
spark = SparkSession.builder.config("hive.metastore.uris","thrift://<IP_ADDRESS>:9083").config("hive.exec.dynamic.partition.mode","nonstrict").config("spark.sql.storeAssignmentPolicy","LEGACY").appName("Python Spark").enableHiveSupport().getOrCreate()
spark.sql("use weatherpollution")
df = spark.read.format('com.databricks.spark.csv').options(header='true', inferschema='true').load("hdfs://mycluster/user/yjy2021/aq_station.csv",header=True)
df.write.insertInto('weatherpollution.aq_station', overwrite=False)
Here is the sample data
aq_demo.csv

Thx @sybest1259,
Previously,  Spark default to FileSourceScanExec when reading from hive table backended by Parquet (we can also tune the conf spark.sql.hive.convertMetastoreParquet to fallback to HiveTableScanExec). So I can't repro it.
Now I tried with CSV, and can repro.

The hive table backed by parquet/orc will be converted to HadoopFsRelation by default which will finally be converted to FileSourceScanExec, and it can be replaced by GPU.
But if the hive table is backed by other types will be converted to HiveTableRelation and finally be planed to HiveTableScanExec which is not supported by GPU for now.
@sybest1259, For now, maybe you can convert your data into parquet/orc, and alter your table stored as parquet or orc

Thx @wbo4958
Sorry for replying late, I just came back from a business trip.
I will follow your advice to rebuild the data.
Thanks very much.

GPU-Util is still very low(1%-7%), the running time of the job is still a bit long, took about 1 minute.
Are there some other configuration problems?

GPU-Util is still very low(1%-7%), the running time of the job is still a bit long, took about 1 minute.

Without more information, my guess is that this query is I/O bound, i.e.: almost all of the time is spent accessing the input from the distributed filesystem.
Can you take a look a the Spark web UI for the job, specifically the SQL tab showing the graph of the query plan?  There should be metrics on each node showing how much time was spent in various steps of processing for the node (e.g.: buffering time, GPU decode time for the Parquet load).  It would be interesting to check how much time is being reported for the metrics and where all the time appears to be spent for the query.  With such a low GPU utilization I suspect you'll see it's almost all in buffer time (i.e.: time spent reading in the distributed filesystem data).
If you want to see low-level details of what's going on with the GPU, you can try capturing an Nsight Systems profile of an executor.  See the developer docs on capturing Nsight Systems profiles for more information.  If you are not building your own custom cudf jar you can skip the first step since we now ship the official cudf artifact with NVTX code included.

The configuration "--executor-cores 1 --executor-memory 1g" may have restricted the parallel, which means at any time there is only 1 task running on each Executor.
You can change to "--executor-cores 12 --executor-memory 30g" according to your real physical env.
and then add the below configurations when submitting
  --conf spark.task.cpus=1
  --conf spark.executor.resource.gpu.amount=1
  --conf spark.task.resource.gpu.amount=0.08
  --conf spark.rapids.sql.concurrentGpuTasks=2

spark.task.resource.gpu.amount=0.08 is calculated by 1/12

@sybest1259 Have you tried the configurations I provided in https://github.com/NVIDIA/spark-rapids/issues/4025#issuecomment-987518881 ?

Thx, @wbo4958
I have tried new configurations with your advice. As the CPU of nodes in the cluster is limited to 8 cores，I set --conf park.task.resource.gpu.amount=0.125 --executor-cores 12
The entire configurations below
pyspark --jars /root/hadoop/mysql-connector-java-5.1.37-bin.jar,/root/downloads/rapids-4-spark_2.12-21.10.0-SNAPSHOT.jar,/root/downloads/cudf-21.10.0-cuda11.jar,/root/downloads/xgboost4j-spark-gpu_2.12-1.6.0-SNAPSHOT.jar,/root/downloads/xgboost4j-gpu_2.12-1.6.0-SNAPSHOT.jar,/root/downloads/jpmml-xgboost-1.5.6.jar,/root/downloads/jpmml-sparkml-1.8.0.jar --master yarn --deploy-mode client  --num-executors 2 --executor-cores 8 --executor-memory 12g --driver-memory 10g
--conf spark.sql.session.timeZone=UTC
--conf spark.driver.extraJavaOptions=-Duser.timezone=UTC
--conf spark.executor.extraJavaOptions=-Duser.timezone=UTC
--conf spark.plugins=com.nvidia.spark.SQLPlugin
--conf spark.kryo.registrator=com.nvidia.spark.rapids.GpuKryoRegistrator
--conf spark.resources.discoveryPlugin=com.nvidia.spark.ExclusiveModeGpuDiscoveryPlugin
--conf spark.rapids.shims-provider-override=com.nvidia.spark.rapids.shims.spark312.SparkShimServiceProvider
--conf spark.rapids.memory.gpu.pooling.enabled=true
--conf spark.task.cpus=1
--conf spark.rapids.python.memory.gpu.allocFraction=0.8
--conf spark.rapids.python.memory.gpu.maxAllocFraction=0.9
--conf spark.executor.resource.gpu.amount=1
--conf spark.task.resource.gpu.amount=0.125
--conf spark.executor.resource.gpu.discoveryScript=/root/hadoop/getGpusResources_executor.sh
--conf spark.rapids.sql.enabled=true
--conf spark.rapids.sql.incompatibleOps.enabled=true
--conf spark.rapids.sql.concurrentGpuTasks=2
--conf spark.rapids.sql.udfCompiler.enabled=true
--conf spark.rapids.sql.format.csv.read.enabled=true
--conf spark.rapids.sql.format.csv.enabled=true
--conf spark.rapids.sql.variableFloatAgg.enabled=true
--conf spark.rapids.sql.explain=ALL
--conf spark.rapids.sql.batchSizeBytes=512M
--conf spark.sql.hive.convertMetastoreParquet=true
--conf spark.rapids.sql.castFloatToString.enabled=true
--conf spark.rapids.sql.csv.read.float.enabled=true
--conf spark.rapids.sql.castStringToFloat.enabled=true
--py-files /root/downloads/sparkxgb2.zip
I ran the task 3 times，the task duration reduced to 11 seconds.

But I still wonder if it's because of the increase of CPU.

The configurations enable more spark tasks executed parallelly.  @sybest1259

Thx @wbo4958
I'm testing now and  I'll ask you if I have any problems.
Thanks to all developers.


Bug: can't assign product variants weight

What are you trying to achieve?
can't assign product variants weight in shippable products
there are only can assign variant name and SKU for each variant

Steps to reproduce the problem

create a product with 2 variants and test it

What did you expect to happen?
assign product variants weight
Logs
No response
Environment
Saleor version:
dashboard v3.18.2
core v3.18.13
OS and version: …

Hi @Qasem-h,
You can change the weight on the variant details page, just click on the pencil icon in the data grid.

Setting a weight on the data grid view is currently not available, you can add a feature request for that in dashboard repo.


[writeset-builder]  Implement a command to output WriteSetPayload

…he stdlib.

Motivation
Fixed a number of issues around the writeset-builder tool:

Added a flag to output either the WriteSetPayload(required by AOS writeset portal) or the Transaction (required by the db-bootstrapper tool)
Fixed the json argument for building custom script.

There will be more follow up items. The most important TODO item is to hook the tool up with real state and generate the release patch for the updateed stdlib.
Have you read the Contributing Guidelines on pull requests?
Yes
Test Plan
Make sure buid_custom_script works with remove_validators.move via following command:
cargo run -- -o out.bin build-custom-script remove_validators.move '{"addresses": ["00000000000000000000000000000000"]}'
Fixed existing tests.

After a discussion with @sblackshear, we suggest that it is safer to generate the writeset patch by computing the diff between remote stdlib in production and stdlib file on disk. Thus I got rid of the cherry-picking part in this PR.

After a discussion with @sblackshear, we suggest that it is safer to generate the writeset patch by computing the diff between remote stdlib in production and stdlib file on disk. Thus I got rid of the cherry-picking part in this PR.

/land

/land


status2d patch comparability with integrated status text.

Why isn't status2d compatible  with integrated status text? Is it an issue with integrated status text itself or is it an incompatibility with your build?
I'm trying to figure out if I'd be able to merge status2d into my build with your integrated status text patch already applied. As most bigish patches tend to have conflicts which can be resolved, but I've been thrown off by your comment leading me to believe it wasn't possible to make the two patches work together.
So is it possible to merge the two patches together, and have it work?

Well than I guess I don't need status 2d. Appreciate the response.

I think it should be medium to easy level of difficulty, generally speaking. I can take a look, since all I need to do is follow the logic of an already existent patch. If you are curious enough, im sure u can work it as well.
Not that much time for it rn tho

Do you mean it'd be possible to merge the two patches? If so I don't mind doing it myself manually.

Yes but what is that you want from the status2d?

what is that you want from the status2d?

I don't really need it I just find it neat. It offers color handling which I don't think my build has capability of. I also like how it draws boxes.

the integrated status text I've updated handles the foreground color by default.

Yeah I missed that despite having it updated, and having to set the color property that's a goof on my end. Don't need status2d for that.

About the drawing yes it is possible to re impmement it.

What do you mean by re-implement? Was it something you had implemented with in integrated status text before, or that since switching from external dwmblocks with status2d to integrated status text that function was lost? Or something else I'm missing.

The seconds one, Im still not sure how it does stuff. it should be straightforward but tweaking a lot to adapt to the blocks context.
And yes, the functionality of dwm that consisted of reading from xsetroot -name to set the status bar is lost, now its integrated and one could argue more efficient.


Code coverage fix

Could you please provide meaningful names/descriptions for commits? It’s quite hard to track what is the purpose of the change and how it is supposed to change code coverage

https://github.com/MIPT-ILab/mipt-mips/blob/ed9aefd4fcd8eb8547f8c98f83d40c3797f2254a/visualizer/test/test.js#L65-L69
This loop dispatched mouse events for each connection and then tests handler
https://github.com/MIPT-ILab/mipt-mips/blob/03308e91741e7fe18ebf324f7913873ee1708c9f/visualizer/src/topology.js#L114-L116
In the previous version, handlers were not called.

This loop dispatches mouse events for each connection and then tests handler

Please add a comment to the code and/or wrap to a function "test_mouse_event_handler"


master/scheduler: rebalance support

What problem does this PR solve?

Issue Number: close https://github.com/pingcap/tiflow/issues/5053
What is changed and how it works?
support rebalancer interface and tableNumberRebalancer in dm. This interface will periodically pick victim sources, sort them in orders and try to pick up workers for these sources.
Check List
Tests

Unit test

Code changes

Has exported function/method change

Side effects

Possible performance regression
Increased code complexity

Related changes

Need to update the documentation

Release note
support table number rebalancer for dm-master

replaced by https://github.com/pingcap/tiflow/pull/5078


feat(preventDuplicates): Changed to list

The prevent duplicates feature works by checking the previously rendered message with the current message and avoids duplicates. There is a test case which will not pass in the following situation - success('hello') --- followed by --> success('bye') --- followed by--> success('hello'). The last "hello" should not render until the previous "hello" has unmounted.
Now, the code maintains a list of messages that are currently being rendered. This makes sure that aforementioned issue doesn't happen.

@tomchentw The previous pull request helps us avoid the duplicate messages. Kindly approve this pull request since it completes the functionality.

@RangarajKaushikSundar I'd like to invite you to be a collaborator of this project. Are you interested?

@tomchentw I would be happy to.

Released v2.8.0

@RangarajKaushikSundar invited!


"Poienarii Burchii" is listed in the drop-down menu

The commune is  listed in the drop-down menu alongside its corresponding county, according to the law
Preconditions : the website (https://peviitor-ro.github.io/ui.orase/) is up and runing. After every step delete the text you typed from the search bar.

Step 1
Write in the search bar "Cosminele"
Expected results
The location appears in the drop-down menu as "Comuna Cosminele, PRAHOVA"


Add Brazilian Portuguese translation

Please, add Brazilian Portuguese translation!

@cgarwood please accept the translation, it is very important for us Brazilians to have its component in our language! Only 5% of Brazilians know English and only 3% are fluent.
Please!

@cgarwood  please accept the translation!


Suggestion: Add a wealth ranking for admins to better managing the economy

I suggested a open wealth ranking before, I know you might not approve it, but is there a chance that we can have a monitoring tool for admins to see the wealth ranking and prevent some player from gaining abnormal amount of money by bug or cheating?

I do not know if this is feasible. The nature of Numismatic Overhaul is item-based. Any wealth could literally be placed beneath your floorboards in chests should you desire, hiding it from any reasonable wealth query

Recently I founded an abnormal number of serverworth in my server (96 Trillion total asset), I want to found out which account has this amount of money, could you please tell me where to look for player’s numismatic account data? Or could you please add a ranking function?

The wealth stored in players' purses is in the players' NBT data under (this might be outdated!) cardinal_components > numismatic-overhaul:currency > Value. For currency stored in item form, check the players' inventories or chests.

Is it possible to search it in all of them but not one by one? Maybe a wealth ranking?

There isn't anything built into Numismatic Overhaul for this (afaik). I described the way to do it manually with an NBT editor.


Registration on societies results in blank page on redirect

When registering on mla.hcommons-dev.org both hctester10 and hctester50 were returned to a blank page on successful registration. I had to clear cookies to log in as both users. I then expunged both users to try on HC vs. MLA.
When trying to register hctester50 through humanities commons it returned a message stating that the user was already member and set that attempt to the status "duplicate" when trying to set up a Google authentication. I can't find another user with that email address. I have seen this happen with users stuck in the LDAP, but the users used only Google auth, not an HCID. Have also seen stuck organizational identity records.
Was not able to replicate the duplicate error with HCTester10, and did not receive a blank screen when registering hctester10 on Humanities Commons. Once registered hctester10 was able to visit all of the Commons they are a member of and  an issue with society roles.

This was an order of operations issue. When a user logs in, a number of things have to happen:

The user needs to be authenticated
BuddyPress needs to initialize
The user needs to have their memberships sync'd from COmanage / their SAML session

Previously, the order of these operations was 1-3-2, which caused problems because adding members to groups in 3 depends on BuddyPress to be initialized (2). To get around this requirement, (3) had been written using direct database calls, but that made it slow and it was still partially dependent on BuddyPress, causing it to fail in some circumstances.
The initial fix (https://github.com/MESH-Research/humanities-commons/commit/a9c487195bf2cc6a8bff2829cb032e35576634aa) made the order 2-1-3, which meant that BuddyPress failed to initialize correctly, eventually resulting in PHP crashing with an out-of-memory error further down the line.
The revised fix (makes the order 1-2-3 by changing the action hook for  hc-simplesaml.php::hcommons_set_user_member_types( https://github.com/MESH-Research/humanities-commons/commit/284b98c661f8d631428ffcf9233b7ae97f2fa5ca) from 'wp_saml_auth_existing_user_authenticated' to 'bp_init' with a late priority to ensure that it fires after BuddyPress has initialized.
The commits also include refactoring and code cleaning.


Multiple instructions

How can i execute multiple instructions in a single transaction? This method seems to not work anymore im getting this error:
Argument of type 'MethodsBuilder<Idl, IdlInstruction & { name: string; }>' is not assignable to parameter of type 'Transaction | TransactionInstruction | TransactionInstructionCtorFields'.ts(2345)
my code:
const tx = new Transaction();
tx.add( await program.methods.sendPost(topic, content, 'username').accounts({             author: wallet.publicKey.toBase58(),             post: post.publicKey.toBase58(),         }).signers([post]));

its because the last function call in the inner chain is .signers which returns MethodsBuilder in order to maintain chain-ability in the function calls. if you want to construct an instruction to add into a transaction, you need to end the chain with .instruction().
const tx = new Transaction();
tx.add(
  await program
    .methods
    .sendPost(topic, content, 'username')
    .accounts({ author: wallet.publicKey.toBase58(), post: post.publicKey.toBase58() })
    .signers([post])
    .instruction()
);

Thanks @callensm for the answer. @AlandKM if you have further questions, please join discord.


Inline buttons: more than 1 per line

I would like to have several inline buttons in one line. But every inline button I use, gets its own line.
If i try to move them into a table, the table is broken.
What is a solution?

Sorry, my mistake. I had a  display: block;  in my cls. Changed it to inline


WebRTC header and lib: broken link

Hello,
In the README file you mentioned that you have the webrtc header and lib in the following repo https://github.com/SoonyangZhang/webrtc-header-lib but the link seems to be broken. I cannot download the files from the Baidoo service either. Can you please reupload these files to enable building your project?
Cheers
Nikita

try this：
url: https://pan.baidu.com/s/18F26BAmZhj_CAQzKNUeDSA
auth code: j4ts

Thanks for the answer. I already tried this link, but unfortunately I cannot register a new account on this service because it does not support phone numbers from Germany.

it is  restored.  https://github.com/SoonyangZhang/webrtc-header-lib

If any change is planed for congestion control algorithm in webrtc, you should build from source code.


Explain how to use package in VScode

what to do with the environment created

how to activate it for running code

how to activate it for notebooks

open the OptoSim folder as project folder to be able to source the created environment

change the default location for VS code server in nikhef login node to a place with some disk space like /data

what exactly is teh issue here?


[14.0] ISPN-14755 Don't serialize empty role arrays

https://issues.redhat.com/browse/ISPN-14755

Thanks @tristantarrant


Downgrade azure sdk

Azure sdk 4.39 pulls in netty 4.1.82 which does not have a method bioSetFd used by SSL in ambry. So as a short term solution, we downgrade azure sdk.

Codecov Report
Base: 68.68% // Head: 14.81% // Decreases project coverage by -53.86% :warning:

Coverage data is based on head (fa81496) compared to base (88a2053).
Patch has no changes to coverable lines.

Additional details and impacted files
@@              Coverage Diff              @@
##             master    #2314       +/-   ##
=============================================
- Coverage     68.68%   14.81%   -53.87%
+ Complexity     9916     2365     -7551
=============================================
  Files           757      757
  Lines         58241    58241
  Branches       7152     7152
=============================================
- Hits          40002     8628    -31374
- Misses        15901    49097    +33196
+ Partials       2338      516     -1822

Impacted Files
Coverage Δ

.../src/main/java/com/github/ambry/router/Router.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../src/main/java/com/github/ambry/store/Message.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...rc/main/java/com/github/ambry/named/PutResult.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...rc/main/java/com/github/ambry/quota/QuotaMode.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...rc/main/java/com/github/ambry/quota/QuotaName.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...rc/main/java/com/github/ambry/rest/RestMethod.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...c/main/java/com/github/ambry/config/KMSConfig.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...c/main/java/com/github/ambry/config/SSLConfig.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...c/main/java/com/github/ambry/router/ByteRange.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../main/java/com/github/ambry/quota/QuotaAction.java
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 570 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Prevent unquoting whole URI (fix #4089)

requirementslib is an affiliate library that powers Pipenv, and we don't update it in the vendors dir.
@techalchemy is working on updating these libraries.
If it is a bug, you can report it to requirementslib repo.
Close the PR now.


Docs: not clear if javascript is executed before page is crawled

Prerequisites

[x] I searched to see if the issue already exists.

Description
Would be nice to mention if crawler is executing js when visiting pages. A section in readme titled Javascript execution or similar would be good enough.


About the release of synthesized dataset

Dear Author,
Thanks for your awesome work!
Are you willing to release the synthesized dataset for training/testing in the paper?
Best wishes,
Xiaodong Cun.

Dear Author,
Thanks for your awesome work!
Are you willing to release the synthesized dataset for training/testing in the paper?
Best wishes, Xiaodong Cun.

hello vinthony,
could you please share the pretrained model with me? There is currently no other way to obtain the model since the url is wrong.
Best wishes!