[CT-3195] [spike+] unit testing versioned models

Is this your first time submitting a feature request?

[X] I have read the expectations for open source contributors
[X] I have searched the existing issues, and I could not find an existing issue for this feature
[X] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

If a unit test is added to a model, with no supplied version, the unit test will run on all versions of said model

unit-tests:
  - name: test_is_valid_email_address # this is the unique name of the test
    model: my_model # name of the model I'm unit testing
    given: # optional: list of inputs to provide as fixtures
      - input: ref('users')
        rows:
         - {user_id: 1, email<EMAIL_ADDRESS>    email_top_level_domain: example.com}
         - {user_id: 2, email<EMAIL_ADDRESS>    email_top_level_domain: unknown.com}
         - {user_id: 3, email: badgmail.com,         email_top_level_domain: gmail.com}
         - {user_id: 4, email: missingdot@gmailcom,  email_top_level_domain: gmail.com}
      - input: ref('top_level_domains')
        rows:
         - {tld: example.com}
         - {tld: gmail.com}
    expect: # required: the expected output given the inputs above
      - {user_id: 1, is_valid_email_address: true}
      - {user_id: 2, is_valid_email_address: false}
      - {user_id: 3, is_valid_email_address: false}
      - {user_id: 4, is_valid_email_address: false}

So if I have version 1, 2, and 3 of my_model, my test_is_valid_email_address unit test will run on all 3 versions.

To to only unit test a specific version (or versions) of a model, you can include the desired version(s) in the model config

unit-tests:
  - name: test_is_valid_email_address # this is the unique name of the test
    model: my_model # name of the model I'm unit testing
      versions:
        include:
          - 2
    given: # optional: list of inputs to provide as fixtures
      - input: ref('users')
        rows:
         - {user_id: 1, email<EMAIL_ADDRESS>    email_top_level_domain: example.com}
         - {user_id: 2, email<EMAIL_ADDRESS>    email_top_level_domain: unknown.com}
         - {user_id: 3, email: badgmail.com,         email_top_level_domain: gmail.com}
         - {user_id: 4, email: missingdot@gmailcom,  email_top_level_domain: gmail.com}
      - input: ref('top_level_domains')
        rows:
         - {tld: example.com}
         - {tld: gmail.com}
    expect: # required: the expected output given the inputs above
      - {user_id: 1, is_valid_email_address: true}
      - {user_id: 2, is_valid_email_address: false}
      - {user_id: 3, is_valid_email_address: false}
      - {user_id: 4, is_valid_email_address: false}

So if I have version 1, 2, and 3 of my_model, my test_is_valid_email_address unit test will run on ONLY version 2.

To to unit test all versions except a specific version (or versions) of a model, you can exclude the relevant version(s) in the model config

unit-tests:
  - name: test_is_valid_email_address # this is the unique name of the test
    model: my_model # name of the model I'm unit testing
      versions:
        exclude:
          - 1
    given: # optional: list of inputs to provide as fixtures
      - input: ref('users')
        rows:
         - {user_id: 1, email<EMAIL_ADDRESS>    email_top_level_domain: example.com}
         - {user_id: 2, email<EMAIL_ADDRESS>    email_top_level_domain: unknown.com}
         - {user_id: 3, email: badgmail.com,         email_top_level_domain: gmail.com}
         - {user_id: 4, email: missingdot@gmailcom,  email_top_level_domain: gmail.com}
      - input: ref('top_level_domains')
        rows:
         - {tld: example.com}
         - {tld: gmail.com}
    expect: # required: the expected output given the inputs above
      - {user_id: 1, is_valid_email_address: true}
      - {user_id: 2, is_valid_email_address: false}
      - {user_id: 3, is_valid_email_address: false}
      - {user_id: 4, is_valid_email_address: false}

So if I have version 1, 2, and 3 of my_model, my test_is_valid_email_address unit test will run on ONLY version 2 and 3.
(similar to include & exclude for warn_error_options)

If you want to unit test a model that references a pinned version of model, you should specify that in the ref of your input:

unit-tests:
  - name: test_is_valid_email_address # this is the unique name of the test
    model: my_model # name of the model I'm unit testing
    given: # optional: list of inputs to provide as fixtures
      - input: ref('users', v=1)
        rows:
         - {user_id: 1, email<EMAIL_ADDRESS>    email_top_level_domain: example.com}
         - {user_id: 2, email<EMAIL_ADDRESS>    email_top_level_domain: unknown.com}
         - {user_id: 3, email: badgmail.com,         email_top_level_domain: gmail.com}
         - {user_id: 4, email: missingdot@gmailcom,  email_top_level_domain: gmail.com}
      - input: ref('top_level_domains')
        rows:
         - {tld: example.com}
         - {tld: gmail.com}
    expect: # required: the expected output given the inputs above
      - {user_id: 1, is_valid_email_address: true}
      - {user_id: 2, is_valid_email_address: false}
      - {user_id: 3, is_valid_email_address: false}
      - {user_id: 4, is_valid_email_address: false}

Describe alternatives you've considered
No response
Who will this benefit?
No response
Are you interested in contributing this feature?
No response
Anything else?

@dbeatty10 What are your thoughts here if you want to only run the unit test on the "latest" version on the model?

@MichelleArk looking at some previous notes, it sounds like we may actually already support unit testing a specific version of a model, by setting model: model_name.v<version>
@dbeatty10 What are y'all's thoughts on that naming convention vs. setting the version in the unit test yml block

I think 3 (f you want to unit test a model that references a pinned version of model, you should specify that in the ref of your input:) is already supported (?) but we should ensure we have a test for that

For the future, we may want to consider enabling folks to only unit test a specific version (or versions) of a model, by supplying the version(s) in my model config
unit-tests:
  - name: test_is_valid_email_address # this is the unique name of the test
    model: my_model # name of the model I'm unit testing
      versions:
        - v2
    given: # optional: list of inputs to provide as fixtures
      - input: ref('users')
        rows:
         - {user_id: 1, email<EMAIL_ADDRESS>    email_top_level_domain: example.com}
         - {user_id: 2, email<EMAIL_ADDRESS>    email_top_level_domain: unknown.com}
         - {user_id: 3, email: badgmail.com,         email_top_level_domain: gmail.com}
         - {user_id: 4, email: missingdot@gmailcom,  email_top_level_domain: gmail.com}
      - input: ref('top_level_domains')
        rows:
         - {tld: example.com}
         - {tld: gmail.com}
    expect: # required: the expected output given the inputs above
      - {user_id: 1, is_valid_email_address: true}
      - {user_id: 2, is_valid_email_address: false}
      - {user_id: 3, is_valid_email_address: false}
      - {user_id: 4, is_valid_email_address: false}

So if I have version 1, 2, and 3 of my_model, my test_is_valid_email_address unit test will run on ONLY version 2.
But this it out of scope for the time being

Notes from refinement:

creating multiple unit test definitions for each model that's versioned
could have partial parsing implications
how do we know all of the versions for a given model

@grace-arends How do we want to handle fixture files?  If we have versioned models and want to test them all but need to use different versions of the fixture file, will we allow it by naming them my_fixture_v1.csv or similar?  Or does each unit test need to be defined separately, as in the last option outlined above?

If we have versioned models and want to test them all but need to use different versions of the fixture file

@emmyoop I don't think we want to allow folks to use different fixtures depending on the version in a single unit test - a unit test should be defined on a static set of inputs.
If you want to create a unit test on version 1 of your model, and a different unit test (with different fixtures) on versions 2 and 3 of your model, I would expect you to need to use 2 different unit test definitions (regardless of the format you're supplying the mock input/expected output data in):
unit-tests:
  - name: test_is_valid_email_address # this is the unique name of the test
    model: my_model # name of the model I'm unit testing
      versions:
        include:
          - 1
    given: # optional: list of inputs to provide as fixtures
      - input: ref('users')
        format: csv
        fixture: users_emails_fixture
      - input: ref('top_level_domains')
        rows:
         - {tld: example.com}
         - {tld: gmail.com}
    expect: # required: the expected output given the inputs above
      - {user_id: 1, is_valid_email_address: true}
      - {user_id: 2, is_valid_email_address: false}
      - {user_id: 3, is_valid_email_address: false}
      - {user_id: 4, is_valid_email_address: false}
  - name: test_is_valid_email_address_2 # this is the unique name of the test
    model: my_model # name of the model I'm unit testing
      versions:
        exclude:
          - 1
    given: # optional: list of inputs to provide as fixtures
      - input: ref('users')
        rows:
        format: csv
        fixture: users_emails_fixture_2
      - input: ref('top_level_domains')
        rows:
         - {tld: example.com}
         - {tld: gmail.com}
         - {tld: hotmail.com}
    expect: # required: the expected output given the inputs above
      - {user_id: 1, is_valid_email_address: true}
      - {user_id: 2, is_valid_email_address: false}
      - {user_id: 3, is_valid_email_address: false}
      - {user_id: 4, is_valid_email_address: false}

If we get feedback to the contrary, we can always come back to this!

Question: What's the use case for unit testing all versions of a model with a single unit test definition?  How are the versions changing but the fixtures are staying the same?
Answer: You could imagine needing to bump the version of my_model because you’re deprecating a column (breaking the model contract), but you could have a unit test on my_model that’s not related to the column you’re deprecating - so you would expect that unit test to still apply for the new version of the model as well as the older version. The input mock data only needs to be specified for the columns relevant to the specific unit test (not all columns). By having the default be “run this unit test on all version of my_model”, if a new version breaks that unit test you will immediately know about it!

I'm closing this out in favor of https://github.com/dbt-labs/dbt-core/issues/9344.  The spike is complete, the implementation is now.


ValueError: Directory data/preprocessed-power/test does not exist

I installed speechT with:
sudo apt install python3-pip portaudio19-dev ffmpeg
pip3 install git+https://github.com/timediv/speechT

Then I wanted to test it with the pre-trained model by following this section.
On speecht-cli evaluate --run-name best_run I'm getting this error:
Determine input size from first sample
Traceback (most recent call last):
  File "/home/mertyildiran/.local/bin/speecht-cli", line 221, in <module>
    cli.run()
  File "/home/mertyildiran/.local/bin/speecht-cli", line 207, in run
    self.command_executor.run()
  File "/home/mertyildiran/.local/lib/python3.5/site-packages/lazy/lazy.py", line 28, in __get__
    value = self.__func(inst)
  File "/home/mertyildiran/.local/bin/speecht-cli", line 200, in command_executor
    }[self.parsed.command](self.parsed)
  File "/home/mertyildiran/.local/bin/speecht-cli", line 169, in _get_evaluation_executor
    return speecht.evaluation.Evaluation(flags)
  File "/home/mertyildiran/.local/lib/python3.5/site-packages/speecht/execution.py", line 33, in __init__
    self.input_size = self.determine_input_size()
  File "/home/mertyildiran/.local/lib/python3.5/site-packages/speecht/execution.py", line 41, in determine_input_size
    return next(self.create_sample_generator(limit_count=1))[0].shape[1]
  File "/home/mertyildiran/.local/lib/python3.5/site-packages/speecht/preprocessing.py", line 261, in load_samples
    raise ValueError('Directory {} does not exist'.format(load_directory))
ValueError: Directory data/preprocessed-power/test does not exist

I found that it's related to this line but I don't know how to fix this error. @timediv could you help me please?
Thank you so much with this great project :+1: :blush:

I also tried to run speecht-cli record --run-name best_run with the hope of testing the model with my microphone but this time I got this error:
Traceback (most recent call last):
  File "/home/mertyildiran/.local/bin/speecht-cli", line 221, in <module>
    cli.run()
  File "/home/mertyildiran/.local/bin/speecht-cli", line 207, in run
    self.command_executor.run()
  File "/home/mertyildiran/.local/lib/python3.5/site-packages/speecht/recording.py", line 44, in run
    model.restore(sess, self.flags.run_train_dir)
  File "/home/mertyildiran/.local/lib/python3.5/site-packages/speecht/speech_model.py", line 260, in restore
    raise FileNotFoundError('No checkpoint for evaluation found')
FileNotFoundError: No checkpoint for evaluation found

I think it's unable to load the pre-trained model (checkpoint) but I don't know how to fix that too.

Now it's working fine with speecht-cli record --run-name best_run command. I don't know what was wrong, probably it was an issue with my tensorflow installation. So I'm closing the issue.


How to start  nni tuning based on historical tuning records?

I have a question about using nni. Consider a certain scenario, when the automl tuning process continues for a long time, but suddenly terminates abnormally. In this case, how to save the current tuning results and continue the automl process based on the historical tuning records?

I think this case is similar to saving checkpoint during neural network training.

Yes, this is an important feature and nni have supported. All tuning results will be saved under ~/nni-experiments/<experiment_id>/db/nni.sqlite and you can use nnictl resume [experiment_id] to resume a stopped experiment.
FYI: https://nni.readthedocs.io/en/stable/Tutorial/Nnictl.html#resume

@J-shang Thank you for your answer. One more question, can I dynamically change the parameter search space while the nni process is running?

@J-shang Thank you for your answer. One more question, can I dynamically change the parameter search space while the nni process is running?

I found nnictl  update in the doc can answer my question, I will close this issue.


Colcon build error on fresh install complaining about Behaviour Tree package

Hi,
I have Ubuntu 22 with ROS2 Humble. I followed the tutorial on how to install PlanSys2:
cd <your_workspace>
git clone -b humble-devel https://github.com/IntelligentRoboticsLabs/ros2_planning_system_examples.git src
colcon build --symlink-install

It failed on the colcon build with the following output:
Starting >>> plansys2_executor
--- stderr: plansys2_executor
CMake Error at CMakeLists.txt:20 (find_package):
  By not providing "Findbehaviortree_cpp_v3.cmake" in CMAKE_MODULE_PATH this
  project has asked CMake to find a package configuration file provided by
  "behaviortree_cpp_v3", but CMake did not find one.

  Could not find a package configuration file provided by
  "behaviortree_cpp_v3" with any of the following names:

    behaviortree_cpp_v3Config.cmake
    behaviortree_cpp_v3-config.cmake

  Add the installation prefix of "behaviortree_cpp_v3" to CMAKE_PREFIX_PATH
  or set "behaviortree_cpp_v3_DIR" to a directory containing one of the above
  files.  If "behaviortree_cpp_v3" provides a separate development package or
  SDK, be sure it has been installed.

---
Failed   <<< plansys2_executor [4.00s, exited with code 1]
Aborted  <<< rosbag2_transport [7min 58s]

I cloned the humble-devel branch from the following link: https://github.com/BehaviorTree/BehaviorTree.ROS2
I then removed build, install and log folders and started a fresh build of everything and I am still getting the above error.
Has anyone else stumbled upon this?
EDIT:
I found the file "behaviortree_cpp_v3Config.cmake" hiding in /opt/ros/humble/share and added that path to the CMAKE_PREFIX_PATH. I reran colcon and now I am getting a different error for the behaviour tree:
Starting >>> plansys2_lifecycle_manager
--- stderr: behaviortree_ros2
CMake Error at CMakeLists.txt:17 (find_package):
  By not providing "Findbehaviortree_cpp.cmake" in CMAKE_MODULE_PATH this
  project has asked CMake to find a package configuration file provided by
  "behaviortree_cpp", but CMake did not find one.

  Could not find a package configuration file provided by "behaviortree_cpp"
  with any of the following names:

    behaviortree_cppConfig.cmake
    behaviortree_cpp-config.cmake

  Add the installation prefix of "behaviortree_cpp" to CMAKE_PREFIX_PATH or
  set "behaviortree_cpp_DIR" to a directory containing one of the above
  files.  If "behaviortree_cpp" provides a separate development package or
  SDK, be sure it has been installed.

---
Failed   <<< behaviortree_ros2 [0.82s, exited with code 1]
Aborted  <<< plansys2_lifecycle_manager [1.32s]

I cannot find any of of those cmake files on my computer.

Hi @DigitalGabriele
PlanSys2 still uses BehaviorTreeCPP 3.x . Remove the version 4.x that you installed. Use apt to install the correct version of behavior tree. As far as I remember, it is sudo apt install ros-humble-behaviortree-cpp-v3. It also could be fixed using rosdep with your workspace.
Francisco.

Hi,
It says its already installed when I ran sudo apt install ros-humble-behaviortree-cpp-v3 but it doesnt work.
ros-humble-behaviortree-cpp-v3 is already the newest version (3.8.6-1jammy.20240126.102626).
ros-humble-behaviortree-cpp-v3 set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 5 not upgraded.

Did you sourced the underlay before building?
Is the package in the list when you execute `ros2 pkg list|grep behavior"?

Try this:
git clone -b rolling https://github.com/PlanSys2/ros2_planning_system.git
git clone -b rolling https://github.com/PlanSys2/ros2_planning_system_examples

@DigitalGabriele Did these answers help you? Can we close the issue?

Hi! I have the same problem, when I do the colcon build I get an error in behaviortree_cpp.
--- stderr: plansys2_executor
CMake Error at CMakeLists.txt:23 (find_package):
By not providing "Findbehaviortree_cpp.cmake" in CMAKE_MODULE_PATH this
project has asked CMake to find a package configuration file provided by
"behaviortree_cpp", but CMake did not find one.
Could not find a package configuration file provided by "behaviortree_cpp"
with any of the following names:
behaviortree_cppConfig.cmake
behaviortree_cpp-config.cmake

Add the installation prefix of "behaviortree_cpp" to CMAKE_PREFIX_PATH or
set "behaviortree_cpp_DIR" to a directory containing one of the above
files.  If "behaviortree_cpp" provides a separate development package or
SDK, be sure it has been installed.

Failed   <<< plansys2_executor [8.91s, exited with code 1]
I am using Ros2 version foxy and I have done the steps you indicated and as you can see the behaviortree_cpp_v3 package is installed.
:~/ros2_ws/prueba$ ros2 pkg list|grep behavior
behaviortree_cpp_v3


Can combine skip intermediate calculations for sync values?

When streams are syncronized and im trying to combine them, there is intermediate calculations, which im trying to avoid
Here is example: http://requirebin.com/?gist=ed0ccb7df29f651b13a63fa83a59ecdc
var a$ = most.periodic(1000).delay(1000).scan(x => x + 1, 0).take(5);
var b$ = a$.map(x => x + 1);
most.combine(sum, a$, b$)
	//.debounce(0)
	.forEach(x => console.log('a+b=', x, new Date().toISOString()));

function sum(a, b) { return a + b; }

And a diagram:
a$            -0--1--2--3--4--
b$            -1--2--3--4--5--
sum:expected  -1--3--5--7--9--
sum:real      -1--23-45-67-89-

On 1st second i have a=1 and b=2, so i expect sum=3, but i had sum=2 and then sum=3
Using debounce will help, but its a bad decision to my mind. Becouse it slow, unrealiable, and may lead to race conditions (for example, if next i want to combine sum and a$), etc
What if combine, or a new method, can detect when variables are calculating and push only when last of them done?

If you know there's a 1:1 correlation between the data in each stream, using zip, which is the same as combine, but with synchronised output. That is the function you pass it will be called once for each complete set of emissions on each channel.

Yes, but only for 1:1, so zip will fail when you try to filter one of streams
a$            -0--1--2--3--4--
b$            -1-----3-----5--
sum:expected  -1-----5-----9--
zip           -1-----4-----7--

I guess it's just a case of the fact that you can't have it both ways. How would most know what you consider to be an intermediate calculation? Would sample solve the problem?

Hi @standy.  Thanks for jumping in with good info, @axefrog!
As you noticed, in push-based implementations, simultaneity poses an interesting problem.  It's much less of a problem in pull-based impls (e.g. catalyst handles it in its mergeWith operator)

can detect when variables are calculating and push only when last of them done?

That is the essence of the problem: How would one detect when variables are calculating?
I do believe that there is a reasonable solution to be had ... we just haven't managed to figure it out yet :)
One way might be to wait "long enough" to deem that all events in the "current time unit" (whatever that unit is, millisecond, etc.) must have occurred.  But how long is long enough?  If the time unit is millisecond, then we'd need at least need to wait until the end of the current millisecond.  While most's scheduler does have millisecond resolution, this still poses a few interesting problems:

It means that event propagation through combine() cannot be synchronous anymore.  combine must use the scheduler, so will incur a small, but non-zero perf penalty.

Things get really interesting when you stack multiple combines: combine(f1, s1, combine(f2, s2, combine(f3, s3, combine(f4, s4, s5))))  If the innermost combine waits until the end of the current millisecond, how long should each of the outer combines wait?  Is the perf penalty cumulative?

I'd love to find some key insights that allow a solution to be implemented efficiently, so if you have thoughts/suggestions/etc. I'd certainly love to hear them.

@standy In your specific example, zip would work, as @axefrog mentioned.  But like you said, if you change the proportion of one of the streams, i.e. via filter, throttle, etc. zip wouldn't work.  One possible solution would be to zip first, and then filter afterward.

Did some research, and now im pretty sure it is possible with this implementation.
Actually, when there is a list of values that about to start calculating, you could mark them all as "not done yet", and unmark one by one right after. So inside combine method all you need is don't push if there is more then one marked value.
My commit is just dirty hack, but i hope it helps

@standy Thanks for investigating.  Unfortunately, that approach won't work in all cases.  The scheduler is a task scheduler, and not an event scheduler.  Not all events pass through the scheduler, and sometimes many events are produced by a single task.  As one example, DOM events (in fact, any EventTarget) don't use the scheduler at all, since they are naturally async and most.js doesn't need to introduce extra asynchrony.  So, something like this wouldn't detect simultaneity:
const s = fromEvent('input', textBox)
combine(add2, s, s.map(x => x+1))

I read a paper recently (can't find the link atm, but I'll post it here if I find it) that proposed a solution.  The author's observation was that in a single-threaded push-based system, simultaneous events can only occur when they originated from the same source.  The proposed solution was to tag every originating event source (i.e. fromEvent) with an id.  They proposed to use that to determine if events from multiple inputs had ultimately come from the same source, and thus might be simultaneous.
I'm not convinced it can work in all cases in the face of filtering (and other combinators that drop or insert new events), but still an interesting idea.


Integrate host config integrity monitoring

In order to be alerted when the configuration on a production VM changes from the expected state, we want to have Tripwire monitoring expected-static directories on all hosts and alerting when there are changes.
Acceptance Criteria

[ ] Tripwire is installed on everything
[ ] Tripwire's known-good configuration snapshot is generated as part of CD
[ ] Tripwire excludes paths that we expect to be changing based on actual production usage
[ ] Notifications go to Riemann and alerts will be raised via PagerDuty
[ ] Tripwire deployment is triggered based on an upstream release

https://github.com/s-matyukevich/bosh-broker  This is a POC for the broker

Long since handled in the FedRAMP push.


TF 2.2.0-rc2 tf.Keras API:  model.loss_functions no longer exists

System information

Have I written custom code (as opposed to using a stock
example script provided in TensorFlow):  Yes
OS Platform and Distribution:  Ubuntu 18.04
TensorFlow installed from (binary): v2.2.0-rc1-34-ge6e5d6df2a 2.2.0-rc2
CUDA/cuDNN version: cuda-10.1

Describe the current behavior
In tf.Keras API in  TensorFlow 2.2.0, model.loss_functions no longer exists.  Instead the loss functions are buried and seem to be only accessible with code like this:
model.compiled_loss._get_loss_object(model.compiled_loss._losses).fn

Being able to easily access loss functions associated with a model is important for many applications that are built on TensorFlow.  Moreover, this is an undocumented breaking change.
Describe the expected behavior
The loss functions associated with a model should be more directly and easily accessible like model.loss_functions, which is the way loss functions are accessed in TensorFlow 2.1.0.
Standalone code to reproduce the issue
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Activation
model = Sequential()
model.add(Dense(10, input_dim=10, activation='softmax'))
model.compile(optimizer='sgd', loss='categorical_crossentropy', metrics=['accuracy'])

# works in TF 2.2.0-rc2, but does NOT work in previous versions like 2.1.0
model.compiled_loss._get_loss_object(model.compiled_loss._losses).fn

# does NOT work in TF 2.2.0-rc2, but works in TF 2.1.0
model.loss_functions[0].fn

This looks related to #38150

The tf.keras.losses module does not allow  you to identify the loss function tied to a compiled model.  So, this solution isn't really satisfactory.  The loss function associated with a compiled model should be publicly accessible, which was part of the point of this issue.

Oh i meant you can use tf.keras.losses.get to get the loss function using a string. Since loss value is passed to the compile API it should be easy to compile list of loss functions from what's being passed. You can also use model.compiled_loss.metrics.

I see - thanks.
So, the idea is to obtain the loss function string names and supply that to tf.keras.losses.get.   In situations where you may not have access to the loss function string names originally supplied to model.compile (e.g., libraries that accept a compiled model as input to facilitate training), what is the best way to obtain the loss function string names directly from the model?  If a model is compiled with a loss function using a string name or loss object, it would make sense for it be possible to query the model and retrieve the loss function (or loss function string name) from the model using public API.
You suggested model.compiled_loss.metrics, but this is an empty list using the example above.  Only  model.compiled_loss._losses contains the string names:
In [1]: from tensorflow.keras.models import Sequential
   ...: from tensorflow.keras.layers import Dense, Activation
   ...: model = Sequential()
   ...: model.add(Dense(10, input_dim=10, activation='softmax'))
   ...: model.compile(optimizer='sgd', loss='categorical_crossentropy', metrics=['accuracy'])

In [2]: model.compiled_loss.metrics
Out[2]: []

In [3]: model.compiled_loss._losses
Out[3]: 'categorical_crossentropy'

model.compiled_loss.metrics gets populated after a model is actually trained /evaluated.

I see that model.compiled_loss.metrics is populated after training, but it doesn't return what you claim it returns:
In [18]: model.compiled_loss.metrics # no loss function string name
Out[18]: [<tensorflow.python.keras.metrics.Mean at 0x7fc85af8a6d8>]

There should be an easy and public way to query a model to identify either the loss function or loss function string name (in this case 'categorical_crossentropy') without resorting to underscore atrributes like this::
In [19]: model.compiled_loss._losses
Out[19]: [<tensorflow.python.keras.losses.LossFunctionWrapper at 0x7fc85aed4a90>]

In [20]: model.compiled_loss._losses[0].fn
Out[20]: <function tensorflow.python.keras.losses.categorical_crossentropy(y_true, y_pred, from_logits=False, label_smoothing=0)>


Generic support for enums, custom (read_write)_json_value not used

I'm trying to support enum values in JSON (de)serialization. For this I'm using https://github.com/Neargye/magic_enum
It works when I define the custom IO functions with the literal enum type, but it does not work when I make the function generic (so it works for all enums).
This works:
enum bla : uint16_t
{
    value_one,
    value_two
};

namespace json_dto {

    template <>
    inline void write_json_value(const heatpump::mode& value, rapidjson::Value& object, rapidjson::MemoryPoolAllocator<>& allocator)
    {
        // first retrieve the enum name
        std::string name{ magic_enum::enum_name(value) };

        // now write it to the value
        write_json_value(name, object, allocator);
    }

}

This, however, does not:
enum bla : uint16_t
{
    value_one,
    value_two
};

namespace json_dto {

    template <typename T>
    std::enable_if_t<std::is_enum_v<T>>
    write_json_value(const T& value, rapidjson::Value& object, rapidjson::MemoryPoolAllocator<>& allocator)
    {
        // first retrieve the enum name
        std::string name{ magic_enum::enum_name(value) };

        // now write it to the value
        write_json_value(name, object, allocator);
    }

}

The latter case will not find the function, and then fail because it tries to call T::json_io.

It seems that that overload:
template <typename T>
std::enable_if_t<std::is_enum_v<T>>
write_json_value(const T& value, rapidjson::Value& object, rapidjson::MemoryPoolAllocator<>& allocator);

won't work without such patch for pub.hpp:
@@ -886,7 +886,8 @@ json_io( Io & io, Dto & dto )

 template< typename Dto >
 std::enable_if_t<
-               !details::meta::is_stl_like_container<Dto>::value,
+               !details::meta::is_stl_like_container<Dto>::value
+                       && !std::is_enum<Dto>::value,
                void >
 read_json_value(
        Dto & v,
@@ -898,7 +899,8 @@ read_json_value(

 template< typename Dto >
 std::enable_if_t<
-               !details::meta::is_stl_like_container<Dto>::value,
+               !details::meta::is_stl_like_container<Dto>::value
+                       && !std::is_enum<Dto>::value,
                void >
 write_json_value(
        const Dto & v,

It also seems that the current version of json_dto doesn't support such a generic way of reading/writing enums. I'll think about some way of adding such support to json_dto.

The best way that I could come up with at the moment is:

Definition of some empty tag type:
https://github.com/Stiffstream/json_dto/blob/db0fb723fc5163083d763e0b2a8262e74e05e450/dev/test/tagged_proxy/main.cpp#L12

Usage of the new json_dto::tagged_proxy function during enumeration of fields to be (de)serialized:
https://github.com/Stiffstream/json_dto/blob/db0fb723fc5163083d763e0b2a8262e74e05e450/dev/test/tagged_proxy/main.cpp#L82-L87

Writing own partial specialization of the new json_dto::tagged_proxy_io_t type for your tag type:
https://github.com/Stiffstream/json_dto/blob/db0fb723fc5163083d763e0b2a8262e74e05e450/dev/test/tagged_proxy/main.cpp#L16-L53

If such an approach looks appropriate for you then I'll finish that sketch and release an updated version of json_dto.

I think there is another possibility based on the approach described above:
namespace test {

struct my_enum_image {}; // Tag type.

// Enums to be serialized a special way.
enum class level_t {...};
enum class category_t {...};

} /* namespace test */

namespace json_dto {

// Partial specialization for tagged_io_t.
template<typename T>
struct tagged_io_t<test::my_enum_image, T> {
   static void read_json_value(...) {...}
   static void write_json_value(...) {...}
};

// THE MAIN TRICK: specialization of tagged_t for every enums from test namespace.
template<> struct tagged_t<test::level_t> : public tagged_as<test::my_enum_image> {};
template<> struct tagged_t<test::category_t> : public tagged_as<test::my_enum_image> {};

} /* namespace json_dto */

namespace test {

struct data_t
{
	level_t m_level;
	category_t m_category;

	template<typename IO>
	void json_io( IO & io )
	{
		// NOTE: fields are described as usual.
		io & json_dto::mandatory( "level", m_level )
			& json_dto::mandatory( "cat", m_category );
	}
};

} /* namespace test */

I didn't check that approach, but it seems that it could be implemented in C++14.

I'm not sure those ideas are the best we can come up with. The fact that you have to do extra work to specifically enable support for each enum in particular feels suboptimal.
As it is, currently it breaks because the read_json_value and write_json_value are over-eager templates which then themselves call an over-eager json_io free-standing function which then breaks because it tries to invoke a member function on the given variable.
The free-standing json_io function should be easily fixable. We can just ensure it is only enabled if the given type is a class type. So if we change this function prototype to something like this:
template< typename Io, typename Dto >
std::enable_if_t<
        std::is_class<Dto>::value,
        void >
json_io( IO & io, Dto & dto )

we ensure that this function is not called unduly. This allows us to create a type trait so we can enable the read_json_value and write_json_value functions only if we can call the free-standing json_io function.

The free-standing json_io function should be easily fixable. We can just ensure it is only enabled if the given type is a class type.

The json_io function is one of the ways to define custom (de)serialization for a user type (see here). In most cases json_io will be used for struct/classes but there is some probability that json_io can be used for non-struct types like non-standard extension (__int128, for example).
I think that probability is very low and we can neglect it but there is another drawback:
Suppose someone writes a header-only library A using json_dto and defines its own specialization for read_json_value/write_json_value for enums. And someone else writes another header-only library B using json_dto with different specialization of read_json_value/write_json_value for working with enums.
There won't be any problems until A and B are used separately. But if someone includes both A and B in one translation unit then there will be ambiguous specializations for read_json_value/write_json_value. And that ambiguity can't be easily solved.
An idea with tagged types has no that drawbacks.

Another possible solution is the introduction of some form of extractors/formatters like shown here.

I understand that the free-standing json_io function may be used to define user-types for serialization. However, that should not be an issue. Simply removing the default json_io function (which does not support non-class types) from the overload set does not prevent someone from writing another version that does support it. I think doing this is useful in its own right.
In fact, I think it could be even better if we write a type trait for a class-type that has a proper implementation of a json_io member function and only enable the free-standing version for these types. Perhaps something like this (untested):
template< typename, typename = void_t<> >
struct has_json_io : public std::false_type{};

template< typename T >
struct has_json_io<
        T,
        void_t<
            decltype(T::json_io(std::declval<json_input_t>())),
            decltype(T::json_io(std::declval<json_output_t>()))
        > : public std::true_type{};

If we do this, then it becomes possibly to write a generic free-standing json_io function, because the one from json_dto does not over-eagerly try to take it

I don't get the point.
The current scheme has several levels but is rather simple (lets use deserialization for an example):

read_json_value is called for a reference of type T;
if there is a specialization/overload of read_json_value for T it is called;
if no, then the default version of read_json_value is called, this version calls json_io function;
if there is a specialization/overload of json_io for T it is called;
if no, then the default version of json_io if called, this version calls T::json_io method.

This scheme allows a user to make own deserialization for a type T at two points:

own overload/specialization of read_json_value, or
own overload/specialization of json_io.

The main drawback of that scheme is that it is oriented to a specific type T, not to a group or family of types. So, if you have a group of types that should be (de)serialized the similar way you have to make an overload of read/write_json_value or json_io functions for every type from your group.
And I don't see how your proposal of the introduction of has_json_io solves that drawback.
Suppose that we implement has_member_json_io and has_freestanding_json_io traits and make two specializations of read_json_value based on them:
template< typename Dto >
std::enable_if_t<
   !details::meta::is_stl_like_container<Dto>::value
      && details::meta::has_member_json_io<Dto>::value,
   void >
 read_json_value(Dto & v, const rapidjson::Value & object) {...}

template< typename Dto >
std::enable_if_t<
   !details::meta::is_stl_like_container<Dto>::value
      && details::meta::has_freestanding_json_io<Dto>::value,
   void >
 read_json_value(Dto & v, const rapidjson::Value & object) {...}

It allows you to make a specialization for enums like:
template< typename Dto >
std::enable_if_t<
   !details::meta::is_stl_like_container<Dto>::value
      && std::is_enum<Dto>::value,
   void >
 read_json_value(Dto & v, const rapidjson::Value & object) {...}

But such specialization can lead to conflicts in the case if someone else defines a specialization like:
template< typename Dto >
std::enable_if_t<
   !details::meta::is_stl_like_container<Dto>::value
      && (std::is_same<Dto, SomeMyEnum>::value || std::is_same<Dto, AnotherEnum>::value),
   void >
 read_json_value(Dto & v, const rapidjson::Value & object) {...}

or if someone else defines json_io for his/her enum types.
That is why I came to an idea with tagged types. And later to an idea with custom extractors/formatters. The idea with custom extractors/formatters looks even better because it also allows solving another problem (see #8).

@omartijn
Version 0.2.10 is just released. Now you can try to use magic_enum library with your types via custom Reader_Writer types as described here.

That looks pretty nice! Thanks for implementing.


Music will not start playing

Hello, I followed the guide step by step and everything seems to work fine until I tried to let my bot play a song from Youtube. When I type: "!!q faded" the bot will respond: "Playing song - Alan Walker - Faded", but I heard nothing.
Then if I type: "!!nowplaying", the bot will say: "Now playing: Alan Walker - Faded Vol100% 00:00/03:33". I think that means the music did not start playing for some reason...?
Before I had some problem with FFMPEG but I think I already fixed those problems. I restarted my computer many times, too.
I looked through the closed issues, searched on Google, and checked the FAQs but I did not found solutions to this problem.
Thank you!

Screenshot your console/ terminal with the error.

@samdivaio Like this?

 It's not supported.

@Kwoth Oops. So I can not use Nadekobot on a 32bit system?

Not music unless you get 32bit .dlls for libopus and libsodium and replace the current ones in NadekoBot/src/NadekoBot 🤔 I think this could be done automatically in the installers. Will ask samvaio to look into it.

@Kwoth O..okay..?? Thank you:))!

You can try extracting this zip, and copying these files to NadekoBot/src/NadekoBot (replacing current ones)
32bitdlls.zip

Hi, can you please test this new installer on a new folder, let me know if you see
Your System Architecture is 32bit...
and Dependencies Downloaded... I added 10secs timer to it so it should be easy to notice.
Thanks 😄
nadeko-with-systemarch.zip

@Kwoth I tried and still it will not start playing... 😭
@samdivaio I opened it and it just said: "git not found, make sure it's been installed as per the guides instructions!
Press any key to exit." (I have git installed already though..)
(Thank you guys for helping me! Usually if I ask something like this on a forum or something no one will reply or help. I'm really happy😆!)

Weird if you already have git but can you check running this command git --version in cmd?
It should show you not found if its environment variable path is somehow removed or charged.
If so, try reinstalling git with selecting option to use it with command prompt.
NadekoInstaller.zip
I have updated the installer for you, you can use two files in the zip, NadekoInstaller.bat to run or install ffmpeg for 32bit and same as before Latest.bat to install latest build of Nadeko.


Current master branch cannot profile Firefox

I build latest master branch (without any GPU support) and try to run it with Firefox. I did the experiment on gpu.cs.rice.edu.  I ran it with hpcrun -e cycles firefox. firefox is a shell script which will eventually launch a  browser window. The Firefox window cannot load any webpage and says "Restart Required"

This is a situation where hpcrun breaks the behavior of the application. Firefox is extremely complicated software, so I am surprised that things do not work out-of-box. Based the error messages on the terminal, there seems to be two issues:

There seems to be file descriptor conflicts between hpcrun and Firefox:

monitor warning [33170,121] monitor_thread_shootdown: timeout exceeded (10): unable to deliver monitor_fini_thread() to 114 threads
hpctoolkit warning: unable to access log file (maybe application closed the file descriptor)
monitor warning [33340,121] monitor_thread_shootdown: timeout exceeded (10): unable to deliver monitor_fini_thread() to 71 threads
hpctoolkit warning: unable to access log file (maybe application closed the file descriptor)

hpctoolkit warning: unable to access log file (maybe application closed the file descriptor)
hpctoolkit: unable to open hpcrun file: '/home/xm13/projects/test/hpctoolkit-firefox-measurements/firefox-000000-001-2a805680-32935-0.hpcrun': File exists

hpcrun seems to affect epoll as Firefox complains:

[warn] Epoll ADD(1) on fd 3 failed. Old events were 0; read change was 1 (add); write change was 0 (none); close change was 0 (none): Operation not permitted

Are we currently do anything to prevent applications to close log, profile, and trace files?
This issue is definitely not a release blocker, but it would be nice to be able to profile complicated code.

The problem seems to be caused by calls to close EVERY file, not just ones Firefox opened in the destructor in the following code. https://chromium.googlesource.com/chromium/src/+/39.0.2171.99/ipc/file_descriptor_set_posix.cc.
I think the only way we can get around this is to wrap close and not let it close our files.
Given the error report at the end of Xiaozhu's entry, it seems that we either need to make changes for epoll as well.

This repository has moved to GitLab, please continue this issue there.


[keymgr] Minor refactor for long files

move data fsm, op fsm and error collection into their own
separate modules.
also some minor clean-up.

first commit will be rebased away.

LGTM (assume the sim runs fine :))


[Bug] Creating A Plot does not recognize required fields

What happened?
ISSUE:
While creating a Plot we noticed that fields that should be required are not. A user is allowed to Create a plot specifically without selecting locations for Step 4 [Select Temporary Directory] and Step 5 [Select Final Directory] or step 6 [Join a Pool or Create a Plot NFT].
STEPS TO REPRODUCE:
User Scenario 1

Given I have a wallet created with TXCH
When I navigate to Plots tab on left navigation bar
Click > ADD A PLOT
Click > CREATE
EXPECTED RESULT: User should be instructed to complete all required fields.
ACTUAL RESULT: User is directed back to Plots main page

User Scenario 2

If user completes step 6 without completing steps 4 and step 5
Click > CREATE
EXPECTED RESULT: User should be instructed to complete all required fields.
ACTUAL RESULT: User is directed back to Plots main page and status % never goes pass 0

Version
1.6.2rc1
What platform are you using?
macOS
What ui mode are you using?
GUI
Relevant log output or stacktrace
No response

@ChiaMineJP can you take a look at this as part of the plotting changes for the next release?


k6 0.45.1 issue on build

From 0.45.0 to 0.45.1 I have this issue:
2023/08/05 14:01:15 [INFO] Build environment ready
2023/08/05 14:01:15 [INFO] Building k6
2023/08/05 14:01:15 [INFO] exec (timeout=0s): /usr/local/go/bin/go mod tidy -compat=1.17
2023/08/05 14:01:15 [INFO] exec (timeout=0s): /usr/local/go/bin/go build -o /go/k6 -ldflags=-w -s -trimpath
# go.k6.io/k6/js/modules
/go/pkg/mod/go.k6.io/k6@v0.45.1/js/modules/resolution.go:161:20: source.URL.JoinPath undefined (type *url.URL has no field or method JoinPath)
note: module requires Go 1.19
2023/08/05 14:01:54 [INFO] Cleaning up temporary folder: /tmp/buildenv_2023-08-05-1400.2697366931
2023/08/05 14:01:54 [FATAL] exit status 2
The command '/bin/sh -c xk6 build --with github.com/grafana/xk6-amqp@latest' returned a non-zero code: 1

@Laurianti this error seems to indicate you are compiling with a version of go that is below 1.19.
The code referenced there wasn't update inv v0.45.1 - it was added for v0.45.0.
There is good chance that you never compiled with v0.45.0 as the update to it was tagged the day before the update to v0.45.1 was made.
Can you confirm your go version by running go version ?

You are right, I had Go version 1.18 installed, and that was indeed the issue. After updating to Go 1.19, the compilation worked correctly. Thank you!


Update mathjax to v3

Is your feature request related to a problem? Please describe.
Mathjax v3 is 60%–80% faster, and has other advantages over mathjax v2, which is used by the sphinx-book-theme.
Describe the solution you'd like
Update mathjax to v3

This is not set by sphinx-book-theme. It is a sphinx option which is already updated in sphinx v4: https://www.sphinx-doc.org/en/master/usage/extensions/math.html#module-sphinx.ext.mathjax


Replace source-map with @jridgewell/trace-mapping

The trace-mapping module is faster than source-map in every benchmark
https://github.com/jridgewell/trace-mapping#benchmarks
It's also being adopted by many packages across the ecosystem (Jest, Babel, Terser at least have included it already)
And it doesn't need to be async like source-map 0.7.*
This would replace #286 and #206
Regarding the changes in this PR

This project seems to have a wide compatibility with older Node.js versions, is that still needed ?
I tested the changes with the browsers tests but they failed, but I also tested /without/ my changes and it still failed (on Firefox, does it only work on some browsers ? )

@LinusU thoughts on this? 🙂


Added CIRT default usernames/passwords from https://cirt.net/passwords

Great stuff!

uhmmm that file looks escaped because all of the _ are prefixed with
is this desirable\by design?

Based on the source, https://cirt.net/passwords?vendor=OpenMarket
I say no @KLuuKer. Nice catch.
Would someone like to make a PR or shall I do it?


Add program notifications

What I forgot in my review: I don't like using dictionaries instead of (data)classes. That's very easy to mess up and no tool can help you with keeping the structure aligned in all places.

Things to do tomorrow:

Address comments on session_to_embed
Fix tests

I think all comments are taken care of and this is ready to be merged. Can you do a final check @NMertsch?

@egeakman Here is an example of how I would test the embed title in test_session_to_embed.py:

A fixture is creating a generic Session object for each test
Each test sets up the tested aspect of the Session object (e.g. session.title = "Short Title")
Each test calls create_session_embed(session)
Each test makes assertions about the tested aspect of the Embed object (e.g. assert embed.title == "Short Title")

This way, the test code itself contains the information about the test: Arrange, Act, Assert. The fixture is doing generic setup logic: Our test requires that "a session is created", but does not care about "how the session is created". Also, it cares about the title of the session, but it does not care about all the other fields of it.
from datetime import datetime, timezone

import pytest

from program_notifications.session_to_embed import create_session_embed
from program_notifications.models import Session, Speaker

@pytest.fixture
def session() -> Session:
    """Create a dummy session object."""
    return Session(
        code="AAAAAA",
        duration=60,
        event_type="session",
        level="beginner",
        rooms=["Exhibit Hall", "Forum Hall"],
        session_type="Announcements",
        slug="example-session",
        speakers=[Speaker(code="BBBBBB", name="Jane Doe", avatar="", website_url="")],
        start="2024-07-10T08:00:00+00:00",
        title="Example Session",
        track=None,
        tweet="",
        website_url="",
    )

def test_short_title(session: Session) -> None:
    session.title = "Short Session Title"

    embed = create_session_embed(session)
    assert embed.title == "Short Session Title"

def test_long_title(session: Session) -> None:
    session.title = (
        "This is a long title which exceeds our maximum embed title length, "
        "so we expect it to be shortened by our session-to-embed converter. "
    )

    embed = create_session_embed(session)
    assert embed.title == (
        "This is a long title which exceeds our maximum embed title length, "
        "so we expect it to be shortened by our session-to-embed [...]"
    )

Please fix test_get_now_with_simulation, then this can get merged.

Should be ready to merge, then!


Consider bringing back value class for TickTime

[ ] Establish performance benchmark for kalasim
[ ] Reintroduce value class for TickTime
[ ] Ensure java-compat with regression test

Blocked by https://youtrack.jetbrains.com/issue/KT-50618


Shader-level texture wrap modes in conjunction with mipmapping and texture atlasses

This PR addresses several things (from pair programming with @smoogipoo ):

Implements shader-level texture wrap modes to allow texture wrapping in conjunction with our use of texture atlasses.
Implements edge-padding of atlassed textures which fixes texture filtering (as well as mipmapping) near the edges of textures. Not only does this address a number of existing problems (e.g. darkening of mania and notes in osu!lazer), but it is also an enabler for the wrap modes.
Implements an API in the Sprite class for specifying a sub-rectangle of the sprite to which the sprite's texture should be stretched and around which the texture is wrapped according to its wrap modes in the S (horizontal) and T (vertical) directions. See the provided test case for an implementation example.

Here is the effect of different wrap modes on a texture. The red-bordered region indicates the sub-rectangle of the sprite around which the wrap modes are applied.

Does this fix https://github.com/ppy/osu-framework/issues/1291 ?

Good chance it does. Let me know if you try and you find that it does work. :)

Needed to make the PR quite a bit bigger just now to fix bleeding problems with fonts. Turns out, certain textures (such as characters) rely on being surrounded by black padding as opposed to repeated padding, so now the way padding is generated depends on the wrap mode of the texture.
This change necessitated making wrap modes readonly per texture and adding their specification to TextureStore.Get.

So I just ran this on my box and textures look kind of... crispy?

Note the fonts looking off, and also the weird moiré-like patterns on the sprite. Not quite sure what that's about, I'll have a read of the source now, but I figured I'd just raise that as a concern once I saw it.

Not a 100% on some of the CI inspections that are reported here. This patch would fix up all of them as far as I can see, but some are kind of weird (the worst one by far is the Call to base member with implicit default parameters inspection, since the cases where I put null, null either report the warning if the params are missing, or show a The parameter ... has the same value message, which is not a warning but still mildly annoying), so I refrained from pushing that in directly for now.

i think previously the solution was using default instead

@smoogipoo can you quickly check over my refactors (should be nothing too offensive)? good to merge otherwise

Gonna leave it there for today. The only remaining issue is that None should not be the default wrap mode, since there is very few cases where you want the edge-bleed by default especially when the bleed is the transparent black background.

@smoogipoo is this okay to get in as-is? would be nice to get it merged.

Yeah I think this is fine as is for now. We can consider changing the default after-the-fact.

Added one more obsoleted member that's currently used in a ruleset Sprite.WrapTexture.
Considered adding TextureGL.WrapMode back as well, but it's not used throughout either osu! or any rulesets, so I think it's pretty safe...

@peppy please add a comment on the max smoothness variable if you can understand the reasoning there.

I've just added a link to the above discussion for now.


Request: Removal of the "NoiseRouter" information line

Kind of annoying since, in my display resolution, that specific line obstructs a lot of my view... That one single line.

I added an option to hide this in the latest release :)


refactor(image): :recycle: file select event to model update

#405

:tada: This PR is included in version 2.5.5 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


fix(convertPathData): fix q control point behavior when item is removed

fix #1926 by only resetting the control point if an item is included
realistically this should be fine as it'll still get reset on move commands, and likely if a command is dropped the control point stayed the same
there's another issue with how in the case M0 0Q10 0 10 10Q10 10 10 10T20 20, convertPathData should convert T to longhand and update the control point before dropping the useless command (changing the control point), but that's out of scope for this PR. (if my thinking is correct, i can't think of any cases where converting from q to t is affected by this behavior)

Is there a release being planned soon?
I've broken my teeth on this and was a bit baffled to find it was fixed three months ago.
Thanks to the both of you for fixing it.

Hey! I was planning to make a release by the end of the month.
Maintenance was stalled last month as I got ill, then had some client work that took priority. I expect to be back on SVGO next week, though. 👍🏽

Looks like this commit reverted in 3.3.2. Is it intentioned?
Got the same error

Yeah, if you read the release notes, due to breaking changes we deprecated and reverted the last release.
If you aren't impacted by the breaking changes in v3.3.1, you're welcome to continue using that until v4 comes out. Meanwhile, I'd recommend waiting for v4 (or at least the v4 release candidates) which will include these bug fixes.

That's unfortunate, as latest stable will bring back old bugs, would it perhaps be a good idea to keep the bug fixes and reverting just the module changes?

as latest stable will bring back old bugs

If you were using v3.3.0 or v3.3.1 without issue, you're welcome to continue using it!

keep the bug fixes and reverting just the module changes?

I would've done that, but the switch to ESM was done before any of the bug fixes, so I couldn't just cherry-pick that change away. With v4 being planned anyway, it seemed more worthwhile to focus on that rather than backport.
I was hoping to have a v4 release candidate out this week, but I was held up. I'd expect SVGO v4.0.0-rc-1 to be out next week, though. That will include the bug fixes, though you'll also need to peek at the release notes to see breaking changes.
(Mostly impacting imports, and a few changes to default plugins like removeViewBox, removeTitle, and also renaming removeScriptElement to removeScripts etc.)
Meanwhile, if it's urgent, I'd recommend either disabling the optimization, or using the patch feature of Yarn/pnpm (or patch-package if using npm) to apply an adapter version of this PR to your repository until the next release is ready.

Ah got it, thanks.

Guess this label confused me

Just a heads-up, the release candidate for SVGO v4 is now available. Feel free to use/try it, and if you encounter any other problems, report an issue!
https://github.com/svg/svgo/releases/tag/v4.0.0-rc.0


Add support for native rating

There are probably better ways to check for native rating, like via protondb. However, I couldn't find any protondb api docs, so I simply used the steam requirements tab.

Apparently my vs code formatted some of the css as well. My bad. I didn't modify any of the existing values, I simply added a class for the native rating. I just chose some random green values, so feel free to change those!

There is no way to get whether or not there is native support of a game from ProtonDB without web scraping and there are no docs for proton's API(I stumbled upon the link for the JSON on Reddit randomly). You did it the exact way I was thinking I was going to!
If you feel like adding any other more experimental features feel free to reach out on discord @ Panda#4464. I have plans to expand it to the game store list too if it is not too laggy to do so. (i.e. this page https://store.steampowered.com/search/)


Tests allow user to modify challenge seed object instead of modifying function to pass Record Collection Challenge

I was working on a way to remove the lines from the Record Collection challenge,
// Keep a copy of the collection for tests
var collectionCopy = JSON.parse(JSON.stringify(collection));

While doing some testing, I discovered that a camper could simply modify the original challenge seed collection object without modifying the original shell function and still pass all the tests (see code at bottom).  This bug is present in the beta and the current production.
I am hoping the fix can also still allow the removal of the above two lines.
var collection = {
    "2548": {
      "album": "Slippery When Wet",
    },
    "2468": {
      "album": "1999",
      "artist": "Prince",
      "tracks": [
        "1999",
        "Little Red Corvette"
      ]
    },
    "1245": {
      "artist": "Robert Palmer",
      "album": "Riptide",
      "tracks": [ "Addicted to Love" ]
    },
    "5439": {
      "album": "ABBA Gold",
      "artist": "ABBA",
      "tracks": ["Take a Chance on Me"]
    }
};
// Keep a copy of the collection for tests
var collectionCopy = JSON.parse(JSON.stringify(collection));

// Only change code below this line
function updateRecords(id, prop, value) {
  return collection;
}

@ValeraS Any thoughts about how best to approach both of the objects of this issue?

How will you validate the original code has not changed in the first test?  Remember, the other goal here is to remove the following from the challenge seed.
// Keep a copy of the collection for tests
var collectionCopy = JSON.parse(JSON.stringify(collection));

I did not explain earlier why I wanted to removed these lines.  The reason is because we get a ton of questions on the forum about that line.  We have not explained what JSON.parse or JSON.stringify does.  Also, I think we should be removing code used for tests because if the user accidentally changes a line which the tests depend on, it could either mess up the testing (even with a valid solution) or maybe allow tests to pass without a valid solution.
One thought I had was to not even use the object shown in the challenge seed, but put a comment in the challenge see that the object used in the challenge looks exactly like the one shown in the challenge seed,  Then, we would use a replicate of the object behind-the-scenes for the actual tests.  We did something similar in another PR, where we actually moved the object from the challenge see and put it in the instructions, but that was a much smaller object which did not take up much space, so I don't think we could do that.
Anyway, if you can fix both issues raised in my original post above, go for it.


README link to Linux GPU Python 3.5 nightly build is broken

I figured you wouldn't need the form information. The link:
https://ci.tensorflow.org/view/Nightly/job/nightly-matrix-linux-gpu/TF_BUILD_IS_OPT=OPT,TF_BUILD_IS_PIP=PIP,TF_BUILD_PYTHON_VERSION=PYTHON3.5,label=gpu-linux/lastSuccessfulBuild/artifact/pip_test/whl/tensorflow_gpu-1.3.0-cp35-cp35m-linux_x86_64.whl
My response:
HTTP ERROR 404

Problem accessing /view/Nightly/job/nightly-matrix-linux-gpu/TF_BUILD_IS_OPT=OPT,TF_BUILD_IS_PIP=PIP,TF_BUILD_PYTHON_VERSION=PYTHON3.5,label=gpu-linux/lastSuccessfulBuild/artifact/pip_test/whl/tensorflow_gpu-1.3.0-cp35-cp35m-linux_x86_64.whl. Reason:

    Not Found
Powered by Jetty://

They link worked for me. Please re-open if issue persists...


if a Job contains a failure, downstream chokes on that failure and stops

If a Job has a failure, downstream Jobs will choke on that bad record and stop.  Some records are written to DB, but ES indexing is not performed, nor are any other actions.
Errors in job processing have been minimal, and so this went unnoticed.  Would benefit from a test.
Likely should "shunt" only good records when using a Job as input.

Fixed.


Inventory with IP address

Is your feature request related to a problem? Please describe.
While it is possible to generate an inventory file, I would like to be able to manipulate the file to add an IP address. The reason, is that smaller firewall in the environment do not have a management ip-address, but one of the interfaces is used as management. For this reason, Panorama reports an ip-address that it is not correct and will not allow access to the firewall. If it was possible to add the ip-address on the inventory file, this would be solved.
Describe the solution you'd like
Add the ability to add ip-address on the inventory or when choosing the option -firewall, add the possibility to add multiple firewalls, instead of just one.

Interesting, so in batch mode (the only mode that uses the inventory.yaml file) the script doesn't connect directly to the device but rather proxies all communication to remote devices through Panorama.
Panorama itself refers to devices by serial rather than hostname or IP address, since those two things are configurable to change at any point of time whereas the serial is both unique and permanent.
The inventory.yaml file is a human-friendly way of selecting serial numbers for Panorama to target, there is an operation that maps the listed devices to their serial number, and then Firewall objects can be made with the info that Panorama needs to target the devices.
Using a data plane interface (like ethernet1/1, for example) should have zero impact on the scripts ability to connect successfully to that device.
The only situation where this could be an issue is where a firewall doesn't have a configured hostname, thereby making it impossible to target the device with the inventory.yaml file.
If you're having issues in your environment, please provide the script's logs and we can investigate what's going on here


goreleaser: build and release arm64 images

what's the motivation for this?

Speed. So I can run ctlptl and tilt images natively on my M1 instead of with emulation. I thought it might be nice for the community to have arm images available too.


Which CDS Entity has Resource master ?

I need to know resource master which Project for the web refers to. As "msdyn_resourceassignments" has several id which refer to resources. I am unable to identify how to map resource id/owner id without master data.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 20c839da-0f06-c409-9744-4d285e405dbb
Version Independent ID: 161f4096-13a0-1368-bdbd-0e530bcffbf0
Content: Export user data from Project for the Web - Project for the web
Content Source: ProjectWeb/export-user-data-from-project-for-the-web.md
Service: project-web
GitHub Login: @serdars
Microsoft Alias: serdars

@parthrawal Thank you for submitting feedback.
I think the best way forward is if you open a service ticket in your tenant so this can get resolved ASAP. Based on the outcome let me know if it is something that can be called out in the docs.
Please follow this link to contact support for business products:https://docs.microsoft.com/office365/admin/contact-support-for-business-products
Please keep us posted here on the resolution so we can feed whatever information you discover into the content.


Adding Redis om spring dependency in spring cloud stream project gives startup error

Hello,
I have a spring cloud stream project with kafka streams binder dependency - no issues at startup.
Next I add redis om spring dependency 0.6.0, add a model object annotate it with @Document and annotate main class with @EnableRedisDocumentRepositories. Now I get below error at startup
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
at com.redis.om.spring.RedisModulesConfiguration.getBeanDefinitionsFor(RedisModulesConfiguration.java:392)
Apologies for any mistake in posting this issue with details - new to this, so please guide me.

same issue

Map<String, Object> annotatedBeans = ac.getBeansWithAnnotation(SpringBootApplication.class); // return an empty LinkedHashMap
Class<?> app = annotatedBeans.values().toArray()[0].getClass();

RedisModulesConfiguration.ensureIndexesAreCreatedis invoked twice. The first oneac.getBeansWithAnnotation(SpringBootApplication.class)` is empty. The second one is not

See my Comment in the commit, this might be the cause. My previous fix has somehow regressed, I don't know the consequences of reverting this, since a lot has changed, so you might look into this @bsbodden.
The error occurs when I try to send an event via Spring Cloud Stream.
Best,
David

Oops, wrong account. This is the one... :D

I am stuck because of this issue - any workaround suggestions to get going till the fix?

Next release will have solve this issue. Likely out by Monday.

@praks84 @roux-jerome This should be fix in https://s01.oss.sonatype.org/content/repositories/snapshots/com/redis/om/redis-om-spring/0.6.2-SNAPSHOT/ a proper 0.6.2 release is pending


added the name

added name
Related Issues or bug #11

Info about Issue or bug

Fixes: #[issue number that will be closed through this PR]
Proposed Changes

Info about changes Goes here

Additional Info

Anything related Issues

Screenshots

Original
Updated

original screenshot
**updated screenshot **

Hi i am merging this pr but this is not having any score


ERROR when doing GET request for vm with cluster name as filter

Environment

Python version: 3.6.7
NetBox version:  2.5.8

Steps to Reproduce

Doing a GET request with the following

2 .Execute

Expected Behavior
Response code 200 with either a:

Empty response if None found
or
Corresponding VM

Observed Behavior
Error code 400 Bad request

This is expected behavior. The request is specifying a cluster which does not exist. There's some discussion in #2819 about whether this error message should be improved, but as it stands currently this is not a bug.


Set to publish on every merge

@benmcmorran - I think this is ready to be enabled in ADO.
I've altered it so it will pull the version from the package.json after updating it with rush set-versions which updates it with the checkin sequence number  (ie, third digit always increases with each commit to main)
I've removed the checks for a tag to be present, since we want to publish to this repo on every commit (and push a tag when we do.)


[Feature request] Remove the jQuery dependency

Ok. I know what you are thinking : What? Why would you want to do that? That's surely impossible!
But, having looked a bit at the source, I still have some genuine questions ;

Is removing the jQuery dependency feasible?
Is autoNumeric core really tightly linked to the jQuery library?
What amount of work would that require?
Would you need help for specific parts of the code?

I'm looking forward to minimize the need to import third party libraries, and so far I only need jQuery for autoNumeric (the latter being awesome ;))

To answer my own questions :

Is removing the jQuery dependency feasible?

yes

Is autoNumeric core really tightly linked to the jQuery library?

oh yes :( I had to jump through hoops to remove it

What amount of work would that require?

a few months as you can see ;)

Would you need help for specific parts of the code?

No thank you, I was able to help myself :P


Magicdns

:shipit:

@justinsb I have this fundamental question? should we cache or should we serve the DNS directly from each Nodes ... imaging we embed kube-proxy and coredns, both reuse the same information for EndpointSlices and Services and you save 2 components ... and kindnet becomes the only networking component needed in the cluster


Update topology-aware-hints.md

Added back to correct sentence

Thanks @seancrasto!
/lgtm

@robscott No worries. It is the least that I can do. The quality of the documentation is top notch.

/approve


Inline styles

We want them because that's the only way how to share styles across all platforms. But I am not sure whether there is a "winner takes it all" library yet. Aphrodite? Anything else?

https://github.com/Khan/aphrodite
https://github.com/rofrischmann/react-look

react-look? (untested but works with native)

Maybe the most simple and straightforward approach is just put FooBla.scss to FooBla.react.js as JavaScript object via StyleSheet the browser version. Then, we can share styles across platforms easily. Browser dependent stuff like responsive CSS, :hover, etc. can be still defined in SCSS. Maybe inline styles are just fine until you render ugly huge views and we don't need any React inline styles library at all.


Account Registration and Login

I have made the registration and login functions. Currently the successful login takes you to the FAQ page as we don't have patient or doctor dashboards yet. Can change where it points to in future if we get those pages done.

closing due to wrong branch.


"forge add project --help" Shows parameter information for adding references

Describe the bug
When executing forge add project --help, the output is as follows (having just installed forge from scoop):
project : Adds project reference to project
Available parameters:
USAGE: forge.exe [--help] [--name <string>] [--project <string>]

OPTIONS:

--name, -n <string>   Project reference path
--project, -p <string>
                      Project to which reference will be added
--help                display this list of options.

You can see the all information details how one adds a reference to a project and not a project to a solution (or what to call the project, what type of project it should be, etc.).
To Reproduce
Execute forge add project --help
Expected behaviour
Parameters and descriptions should reflect how one can customize a project's creation in a solution. Expected output folder, project name (clearly denoting whether a subdirectory is made for the project inside the output folder, or if it is just placed directly in the output folder), customization of what sort of project we are creating (ClassLib, console app, web app, some other template installed via dotnet new under-the-hood, etc.). I'm not sure what all features are available in Forge yet since I just learned about it recently.
Screenshots
N/A
Environment:

OS: Windows 10
Visual Studio version: 2017
dotnet SDK version: 2.2.106
mono / .Net Framework version: wanted to create a net452 project adjacent to two netcoreapp2.2 projects.

Additional context
N/A

Looks like the AddProjectArgs type outlines the help information and values and the addProject function contains the implementation. Not sure what the <@ @> operators are doing, nor where the Furnace module / load method combo is defined. Wouldn't mind taking a crack at fixing this myself if I had an idea of how those things worked (I'm quite new to the whole .NET ecosystem, but this doesn't seem terribly difficult, on the surface of it).

Given all the changes in the ecosystem, Forge will be just utility for moving/adding files. Adding projects can be done with dotnet add reference


Token and vouching mechanics

This PR adds a new package implementing the ZEP token and the Vouching mechanics, closing #101 and #108.
Pending Work:

The contracts are not yet upgradeable, we are tackling this on a new branch on top of this one (#116).
TPL is not yet integrated, so the token contract is some of a placeholder. Same as above regarding further work (#103).

We are opening this as is so as to get a first round of review on the vouching mechanics, which are independent of the above.
Notes:

Right now, an attacker can register a new dependency with a fresh name, but with an address already listed in the system. This does not open any obvious vulnerability, but might cause confusion with respect to who the owner of a dependency is. Ultimately, this stems from the fact that the indexing is done with dependency names instead of their addresses.

(Opening on master since that's where we started from.. Cannot automatically merge on next :-/ )

Already pushed some updates. Now waiting for the opened discussions to be settled to make the final changes.

I'm coming up with a possible implementation for indexing dependency names. Still testing locally. When it's ready, I'll branch out from vouching and then create a PR to merge again.

Already fixed latest issues, should be ready now.

@facuspagnuolo do you want to update your review, as it is blocking the merge?


hotfix: readme formating

What does this PR do?
The introduced spacing at the end of the line violates formatting and prevents any PR from landing, so suggesting to have each sentence on a separate line which I believe also increases readability...

Before submitting

Was this discussed/agreed via a GitHub issue? (not for typos and docs)
[x] Did you read the contributor guideline, Pull Request section?
[x] Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
[x] Did you verify new and existing tests pass locally with your changes?
Did you list all the breaking changes introduced by this pull request?
Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review
Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

Reviewer checklist

[x] Is this pull request ready for review? (if not, please submit in draft mode)
[x] Check that all items from Before submitting are resolved
[x] Make sure the title is self-explanatory and the description concisely explains the PR
[x] Add labels and milestones (and optionally projects) to the PR so it can be classified

cc @borda

CI is deadlocked: this PR needs https://github.com/Lightning-AI/lightning/pull/17134 and https://github.com/Lightning-AI/lightning/pull/17134 needs this PR.
One will have to be force merged.

CI is deadlocked: this PR needs #17134 and #17134 needs this PR.

maybe typo, you refer to the same pr twice


freemode.momentum false causes autoplay to stop.

Check that this is really a bug

[X] I confirm

Reproduction link
https://codepen.io/andershagbard/pen/wvmxjdz
Bug description
Initially when you open the page, autoplay is working. When you start dragging, it won't start playing again.
Expected Behavior
I would expect the autoplay to start again.
Actual Behavior
Autoplay stops after drag.
Swiper version
8.3.2
Platform/Target and Browser Versions
MacOS Chrome 104
Validations

[X] Follow our Code of Conduct
[X] Read the docs.
[X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.
[X] Make sure this is a Swiper issue and not a framework-specific issue

Would you like to open a PR for this bug?

[ ] I'm willing to open a PR

I have additional context to add to this. This isn't using freemode.momentum false but it is using free mode and I am having the same issue: https://codesandbox.io/s/swiper-autoplay-bug-u5xqci?file=/src/App.jsx

Swiper v9 comes with fully reworked autoplay module. If you have similar issues in Swiper 9, open a new issue with a CodeSandbox showing the issue.


Update birthing_centre.json, add doctors-PH.json

Added additional terms for healthcare=birthing_centre preset. Fix typo.
Add regional variant for doctors-PH.json, including aliases.

Thanks for reviewing this @tyrasd .
As for the doctors-PH.json, I'm not really sure if translation if the way to go.
The "RHU"/Rural Health Unit is a local type of field health facility operated by a government, under the supervision of a government physician, which (locally) may be then tagged as amenity=doctor/healthcare=doctor,  and similar to a doctor's office.
I wonder if it's appropriate to create a different preset to display it with a different name, rather than using translate.


Is it really MFA?

The documentation for PIN registration and recovery refers to multifactor authentication. This document Windows Hello for Business Provisioning shows a connection to "Azure MFA". But what methods are allowed to be registered? Are they the methods configured in the old MFA Settings page in Azure AD? Or the methods enabled for SSPR? Or something else? We find that people can register a phone for text messages, even though this is not enabled as a valid method for MFA.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: d9dae9da-eaf0-c209-7c65-be8d2c1ada12
Version Independent ID: 28b6788d-7982-23c2-5f19-f80690ebc0ce
Content: Pin Reset - Microsoft 365 Security
Content Source: windows/security/identity-protection/hello-for-business/hello-feature-pin-reset.md
Product: w10
GitHub Login: @mapalko
Microsoft Alias: mapalko

So the question is: is it accurate to call it Azure MFA? Or does it really just mean a Microsoft PIN recovery registration service?

Hello @Air-Git ,
what methods are allowed to be registered? Are they the methods configured in the old MFA Settings page in Azure AD? Or the methods enabled for SSPR? - methods set in your MFA provider
is it accurate to call it Azure MFA? - no, it is not. Third-party MFA providers could be used as well
Or does it really just mean a Microsoft PIN recovery registration service? - those are two different services. MFA provides authentication, Microsoft PIN reset service only allows to change PIN
Thank you


[Bug]: Dialog's top right close icon cannot close the dialog if open the dialog programmatically

Environment
Radix Vue version: 1.2.4

Link to minimal reproduction
https://github.com/sunny-repo88/shadcn-vue-dialog-bug
Steps to reproduce
`
<script setup lang="ts">
import {
  Dialog,
  DialogContent,
  DialogDescription,
  DialogFooter,
  DialogHeader,
  DialogTitle,
  DialogTrigger,
} from '@/components/ui/dialog'
import { Button } from '@/components/ui/button'
import { ref } from 'vue'

const open = ref(false)

</script>

<template>
  <Dialog :open="open">
    <DialogTrigger>
      <Button @click="open = true">Edit Profile</Button>
    </DialogTrigger>
    <DialogContent>
      <DialogHeader>
        <DialogTitle>Edit profile</DialogTitle>
        <DialogDescription>
          Make changes to your profile here. Click save when you're done.
        </DialogDescription>
      </DialogHeader>

      <DialogFooter>
        Save changes
      </DialogFooter>
    </DialogContent>
  </Dialog>
</template>
`

Describe the bug

right top icon cannot close the dialog
Expected behavior
right top icon should be able to close the dialog
Conext & Screenshots (if applicable)
No response

You are using Controlled Component which you control the (open) state (show/hide)
in this case, you should handle the hiding or showing of the component
You could use

v-model:open="open" easy way ✅ or
:open="open" and @update:open="(openState) => (open = openState)" 🙅

Dialog demo here: https://stackblitz.com/edit/github-eurciq?file=src%2FApp.vue

If you want to use UnControlled Component and still want to show dialog by default you could use
defaultOpen prop which you can read about in MAIN radix-vue documentation

@sadeghbarati , thanks! It works like a charm. v-model:open="open" is working fine. Maybe we can enhance the documentation since open the dialog programatically is quite common usage.


Add more github action tests

Run checks under multiple python versions
and also check the package build and lint.
Reviewers: greg-b

#
head
base
diff
date
summary

0
fe552788
b64cf3b0
diff
Sep 14 8:34 PM
1 file changed, 18 insertions(+), 7 deletions(-)

Reviews in this chain:
└https://github.com/Skydio/revup/pull/39 Add more github action tests

i set it to python3.8 to be guaranteed to work (at least on my computer). python3 for me is 3.6 so i imagine it wouldn't be guaranteed to work for others as well. afaik there's no 100% incantation for py>=3.8 so the PYTHON= is probably the way to go


Enable 2-D input length tensor with second dimension 1

Since the input length tensor provided by users is typically a 2-D tensor of dimension (nsamples, 1) when training with mini-batches, this modification simplifies client code by flattening the second dimension to avoid rank mismatch errors down the chain.

Agreed, please add a test for the previous error case, in backend tests.


Compatible with Angular 13

Summary
Is it compatible with Angular version 13?
Screenshots

Version
^3.15.3

@saurabhShukla621
Angular is not officially supported.


Remove trailing new lines from subject

Fixes #2188

The subject shouldn't contain any newlines at all.

If you paste in rich text (e.g. a heading from a webpage), then the subject EditText shows the rich text.
When you call subjectView.getText() it returns an Editable. Either the Editable retains original new lines that were pasted or it converts the rich text formatting to new lines.
Either way the result is:
Text\n\n which kills the parser.
Maybe a better plan is a TextWatcher that removes spannables... I'll test that out.

While it makes for a nicer user experience not to display styles we don't support anyway (Editable.toString() will strip out all formatting), that's not a critical issue.
Retaining newlines on the other hand is a bug. We definitely need to avoid that. A good first step is probably removing the inputType flag textImeMultiLine from the EditText.

So I looked at this and it does fix the problem of displaying rich-text but it doesn't fix the \n problem. We agreed to just implement a regex replacement for that. There's a pattern in the Utility class already.


Unicode issue occurs from webhook invoke

Dear support,
Seems if we put some Chinese characters, and the action execution will end up with following errors:
{
  "traceback": "  File \"/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2actions/container/base.py\", line 99, in _do_run\n    (status, result, context) = runner.run(action_params)\n  File \"/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py\", line 49, in wrapped_f\n    return Retrying(*dargs, **dkw).call(f, *args, **kw)\n  File \"/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py\", line 206, in call\n    return attempt.get(self._wrap_exception)\n  File \"/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py\", line 247, in get\n    six.reraise(self.value[0], self.value[1], self.value[2])\n  File \"/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py\", line 200, in call\n    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)\n  File \"/opt/stackstorm/runners/mistral_v2/mistral_v2.py\", line 218, in run\n    result = self.start(action_parameters=action_parameters)\n  File \"/opt/stackstorm/runners/mistral_v2/mistral_v2.py\", line 255, in start\n    **options)\n  File \"/opt/stackstorm/st2/local/lib/python2.7/site-packages/mistralclient/api/v2/executions.py\", line 56, in create\n    return self._create('/executions', data)\n  File \"/opt/stackstorm/st2/local/lib/python2.7/site-packages/mistralclient/api/base.py\", line 95, in _create\n    self._raise_api_exception(resp)\n  File \"/opt/stackstorm/st2/local/lib/python2.7/site-packages/mistralclient/api/base.py\", line 143, in _raise_api_exception\n    error_message=error_data)\n",
  "error": "RemoteError: Remote error: UnicodeEncodeError 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128)\n[u'Traceback (most recent call last):\\n', u'  File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py\", line 155, in _process_incoming\\n    res = self.dispatcher.dispatch(message)\\n', u'  File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py\", line 222, in dispatch\\n    return self._do_dispatch(endpoint, method, ctxt, args)\\n', u'  File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py\", line 192, in _do_dispatch\\n    result = func(ctxt, **new_args)\\n', u'  File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistral/engine/engine_server.py\", line 98, in start_workflow\\n    (rpc_ctx, workflow_identifier, utils.cut(workflow_input),\\n', u'  File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistral/utils/__init__.py\", line 283, in cut\\n    return cut_dict(data, length=length)\\n', u'  File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistral/utils/__init__.py\", line 197, in cut_dict\\n    v = str(value)\\n', u\"UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128)\\n\"]."
}

Body in webhook:
{"pool_id": "9300", "assigned_group": "上海-OPS-APP", "assignee": "cjie2"}

Which version of ST2 are you running? There were several Unicode-related fixes in version 2.3, which has just been released.
Also, what does your action look like? From the traceback, the error is occurring when running the action, not at the webhook stage.

ST2 version is 2.2.1, I remembered last time, I input some Chinese characters in the payload body via Webhook, the action was print the trigger.body to local file, and the whole body result is not as expected, means Chinese characters were transfered as strange characters like "-----OPS-APP" from its original which should be "上海-OPS-APP", anyway maybe you can test it out at your side. Just copy these characters and put it to body, print it or save to local file seem what happens.

If I use a simple rule like the one in the examples pack, that writes the body to file using core.local, I get this written to file:
{u'foo': u'\u4e0a\u6d77-OPS-APP', u'name': u'st2'}
That is with 2.3.0.
You would need to test it with your full action though.

Yes, this seems correct. I will test our tomorrow on  both 2.2.1 and 2.3, will come back once tested, thanks Lindsay.

Thanks @leoxhj - I don't know if we've fixed all of the Unicode issues, but it is something we want to do.

@LindsayHill I have tested on 2.2.1, same result as yours:
root@1eabe73d2837:/opt/stackstorm/packs/examples/rules# cat /home/st*/st2*out
{'foo': 'bar', 'name': 'st2'}
{'you': 'too', 'name': 'st2'}
{'you': 'too', 'name': 'st2'}
{'you': 'too', 'name': 'st2'}
{'you': u'\u4e0a\u6d77-OPS-APP', 'name': 'st2'}

@leoxhj can you please provide a rule + workflow/action that reproduces the problem, since using a simple core.local action works?

UnicodeEncodeError: 'ascii' codec can't encode character u'\\\\xb7' in position 791: ordinal not in range(128)
I have a 3rd party that is sending a webhook with the middle dot character
https://en.wikipedia.org/wiki/Interpunct
and I can receive the webhook just fine, but stackstorm/mistral seems to crash when processing the middle dot. Is there a reasonable workaround for this? Or something that I can slip into the existing code where I replace all the middle dots with normal periods?  Here is python stack trace:
  "traceback": "  File \"/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2actions/container/base.py\", line 99, in _do_run\n    (status, result, context) = runner.run(action_params)\n  File \"/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py\", line 49, in wrapped_f\n    return Retrying(*dargs, **dkw).call(f, *args, **kw)\n  File \"/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py\", line 206, in call\n    return attempt.get(self._wrap_exception)\n  File \"/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py\", line 247, in get\n    six.reraise(self.value[0], self.value[1], self.value[2])\n  File \"/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py\", line 200, in call\n    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)\n  File \"/opt/stackstorm/runners/mistral_v2/mistral_v2.py\", line 218, in run\n    result = self.start(action_parameters=action_parameters)\n  File \"/opt/stackstorm/runners/mistral_v2/mistral_v2.py\", line 260, in start\n    **options)\n  File \"/opt/stackstorm/st2/local/lib/python2.7/site-packages/mistralclient/api/v2/executions.py\", line 56, in create\n    return self._create('/executions', data)\n  File \"/opt/stackstorm/st2/local/lib/python2.7/site-packages/mistralclient/api/base.py\", line 95, in _create\n    self._raise_api_exception(resp)\n  File \"/opt/stackstorm/st2/local/lib/python2.7/site-packages/mistralclient/api/base.py\", line 143, in _raise_api_exception\n    error_message=error_data)\n",
  "error": "RemoteError: Remote error: UnicodeEncodeError 'ascii' codec can't encode character u'\\xb7' in position 791: ordinal not in range(128)\n[u'Traceback (most recent call last):\\n', u'  File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py\", line 155, in _process_incoming\\n    res = self.dispatcher.dispatch(message)\\n', u'  File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py\", line 222, in dispatch\\n    return self._do_dispatch(endpoint, method, ctxt, args)\\n', u'  File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py\", line 192, in _do_dispatch\\n    result = func(ctxt, **new_args)\\n', u'  File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistral/engine/engine_server.py\", line 98, in start_workflow\\n    (rpc_ctx, workflow_identifier, utils.cut(workflow_input),\\n', u'  File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistral/utils/__init__.py\", line 283, in cut\\n    return cut_dict(data, length=length)\\n', u'  File \"/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistral/utils/__init__.py\", line 197, in cut_dict\\n    v = str(value)\\n', u\"UnicodeEncodeError: 'ascii' codec can't encode character u'\\\\xb7' in position 791: ordinal not in range(128)\\n\"]."
}```
Here is my webhook payload information:

{"trigger":"linux.DataDog",
"payload": { "body": {
"id": "$ID",
"hostname" : "$HOSTNAME",
"user": "$USER",
"priority" : "$PRIORITY",
"last_updated" : "$LAST_UPDATED",
"date" : "$DATE",
"event_type" : "$EVENT_TYPE",
"body" : "$EVENT_TITLE",
"org_id" : "$ORG_ID",
"name" : "$ORG_NAME",
"plain_text" : "$EVENT_TITLE",
"aggregate_key" : "$AGGREG_KEY",
"alert_scope" : "$ALERT_SCOPE",
"picture" : "$SNAPSHOT",
"link" : "$LINK",
"tags" : "$TAGS",
"alert_id" : "$ALERT_ID",
"alert_metric" : "$ALERT_METRIC",
"alert_transition" : "$ALERT_TRANSITION",
"alert_status" : "$ALERT_STATUS",
"alert_title" : "$ALERT_TITLE"
}}}

Would need to investigate further, but based on what I'm seeing it looks like the mistral runner's handling Unicode characters is the culprit.

I think you are 100% correct. I'm working on a change to just find and replace those characters within the JSON before the resp object gets created here:
/opt/stackstorm/st2/local/lib/python2.7/site-packages/mistralclient/api/base.py
def _create(self, url, data, response_key=None, dump_json=True):
        if dump_json:
            data = json.dumps(data)
---->>insert terrible hack
        resp = self.http_client.post(url, data)

        if resp.status_code != 201:
            self._raise_api_exception(resp)

        return self.resource_class(self, extract_json(resp, response_key))

@LindsayHill
the output should be fine, and this can be handled decode in python script:
{'you': u'\u4e0a\u6d77-OPS-APP', 'name': 'st2'}, you can close this issue, thanks a lot for your support.

@leoxhj it still seems to be a problem for other users when running Mistral workflows, as seen above. I'll leave it open a little longer - hopefully someone will be able to provide a complete workflow that reproduces this. No-one has given us a full workflow that triggers it yet. If we can get that, we can work on fixing it.

@LindsayHill
Today, I tested our code, it's indeed a problem of unicode, please find attached rules, workflows.
you can test with below payload data:
{"pool_id": "8883", "assigned_group": "上海-OPS-APP", "assignee": "cjie2"}
code structure:
rules\pool-decommission.yaml
actions\pool-decommission.yaml
actions\workflows\pool-decommission.yaml
traceback as follows:
{
"traceback": "  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2actions/container/base.py", line 99, in _do_run\n    (status, result, context) = runner.run(action_params)\n  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py", line 49, in wrapped_f\n    return Retrying(*dargs, **dkw).call(f, *args, **kw)\n  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py", line 206, in call\n    return attempt.get(self._wrap_exception)\n  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py", line 247, in get\n    six.reraise(self.value[0], self.value[1], self.value[2])\n  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py", line 200, in call\n    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)\n  File "/opt/stackstorm/runners/mistral_v2/mistral_v2.py", line 218, in run\n    result = self.start(action_parameters=action_parameters)\n  File "/opt/stackstorm/runners/mistral_v2/mistral_v2.py", line 255, in start\n    **options)\n  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/mistralclient/api/v2/executions.py", line 56, in create\n    return self._create('/executions', data)\n  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/mistralclient/api/base.py", line 95, in _create\n    self._raise_api_exception(resp)\n  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/mistralclient/api/base.py", line 143, in _raise_api_exception\n    error_message=error_data)\n",
"error": "RemoteError: Remote error: UnicodeEncodeError 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128)\n[u'Traceback (most recent call last):\n', u'  File "/opt/stackstorm/mistral/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming\n    res = self.dispatcher.dispatch(message)\n', u'  File "/opt/stackstorm/mistral/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 222, in dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', u'  File "/opt/stackstorm/mistral/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 192, in _do_dispatch\n    result = func(ctxt, **new_args)\n', u'  File "/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistral/engine/engine_server.py", line 98, in start_workflow\n    (rpc_ctx, workflow_identifier, utils.cut(workflow_input),\n', u'  File "/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistral/utils/init.py", line 283, in cut\n    return cut_dict(data, length=length)\n', u'  File "/opt/stackstorm/mistral/local/lib/python2.7/site-packages/mistral/utils/init.py", line 197, in cut_dict\n    v = str(value)\n', u"UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128)\n"]."

@LindsayHill just verified in st2vagrant (st2 version 2.3), no error.
so I assume it fixed in 2.3, you can double confirm this.

Thanks for the additional info @leoxhj. I will check it is fixed in both 2.3 (current stable) and the latest master.

In my tests, this worked on 2.3. So I think it has been fixed. Please do continue to let us know if you find any more Unicode issues. We do want to fix them.
Thanks!