Enhance Toast Notifications: Add Update Functionality and Prevent Duplication Using Custom Keys or IDs

Provide a general summary of the feature here
I’m using toast creation tools, but I’m facing an issue where I can’t pass a custom key or ID to prevent duplication or update existing toasts.
This functionality is crucial in scenarios where the notification represents task progress and needs to update based on the task status. Additionally, I need to be able to trigger the same notification from different places in the application. To avoid duplication, the notification should update if it’s already displayed.
🤔 Expected Behavior?
ToastQueue provides additional tools for updating and avoiding duplication
😯 Current Behavior
At the moment, no other tool has such functionality.
💁 Possible Solution
No response
🔦 Context
I resolved the duplication issue by including the ID in the options. I then check if this ID already exists in the toast array. If it’s not present, I add it to the queue.
I also implemented update functionality. While it works to some extent—updating the content—the changes only take effect when the queue is refreshed or the container is focused.
💻 Examples
No response
🧢 Your Company/Team
No response
🕷 Tracking Issue
No response

Thanks for the issue, can you provide a little more information? Are you using the hooks? or are you using ReactSpectrum? It sounds like you're using the hooks.
A codesandbox or stackblitz would also be helpful to see what you're trying to accomplish.

I can’t pass a custom key or ID to prevent duplication

Preventing duplication seems like something that should be handled at the application layer.

or update existing toasts.

I think Toasts are mostly meant to be immutable. If the content on one becomes stale, you can programmatically remove it and create a new one with the latest information.

Additionally, I need to be able to trigger the same notification from different places in the application.

I think this is something that can be handled at the application level. If two different areas of your app can create toasts for a similar incident, they should share state on whether one has already been created.

Okay, I think I could implement all of the above at the application level. Changing the toast content by closing and opening sounds pretty weird, but maybe it makes sense.

The issue is more that I think screen reader or visual user might not notice a change to an existing toast, so I'm not convinced it's a good idea to support updating the content.

Good point! Thanks for your help ❤

@reidbarber I have to use toasts to display progress on a process that runs for a couple of seconds. Closing and re-opening the toast for every percentage change is not really an option so updating it would be the only option here which I'm blocked from doing.


PublishTestResults@2 also publishes the VS coverage hyperlink, making it impossible to publish Cobertura report with PublishCodeCoverageResults@1 task.

Required Information
Question, Bug, or Feature?
Type: Bug
Enter Task Name: PublishTestResults@2 and/or PublishCodeCoverageResults@1

https://github.com/microsoft/azure-pipelines-tasks/tree/master/Tasks/PublishCodeCoverageResultsV1
https://github.com/microsoft/azure-pipelines-tasks/tree/master/Tasks/PublishTestResultsV2

Environment

Server - TFS on-premises

If using TFS on-premises, provide the version: Dev17.M153.3

Agent - Private

name
value

Agent.Version
2.153.1

Agent.OS
Windows_NT

Agent.OSArchitecture
X64

Agent.OSVersion
10.0.14393

Issue Description
When running DotNetCoreCLI@2 with the test command and --collect "Code coverage" the VS coverage binary report is generated. I then convert it to Cobertura report and publish it using the guidelines in https://docs.microsoft.com/en-us/azure/devops/pipelines/ecosystems/dotnet-core?view=azure-devops#collect-code-coverage, i.e. I do the following steps:

Convert the binary VS coverage report to XML
Install reportgenerator tool
Run reportgenerator tool to convert the XML VS coverage result to Cobertura report
Publish the test results (PublishTestResults@2)
Publish the Cobertura coverage result (PublishCodeCoverageResults@1)

However, the Code Coverage page only shows the hyperlink to the VS binary coverage file, which is completely useless to us.
Task logs
logs_754278.zip
Snapshots
This is what we get:

We find it useless. Disabling the Publish Test Results task yields the correct coverage page:

But, of course, the Tests page is now empty:

I found the workaround, please see my answer to https://stackoverflow.com/questions/61143454/how-to-publish-cobertura-coverage-report-from-the-vs-coverage-produced-by-dotnet

@MarkKharitonov This seems by design for now. We are working on the new experience.
Thanks for sharing the workaround.


tests failing / not running

It looks like we need to go in and update our github workflows for treating as seen in #44

Working on this in #43. Do not know what is up with the windows tests though

Thanks Joe! You are on top of things. Yeah, those error messages from windows aren't too insightful...


Added custom rule group interval

Pull from https://github.com/slok/sloth/pull/515

This allows for more expensive SLO recording rules by letting users
create a custom Prometheus rule_group.interval, as rule group default or per type of rule.
https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/#rule_group
Now instead of all rule groups using the default global evaluation
interval, a custom interval can be set on all three types of recording rules
for an SLO.
If no interval is set then the global default will be assumed,
matching current Sloth behavior.
Resolves: https://github.com/slok/sloth/issues/367
Edit: I didn't touch the kubernetes operator side and probably need adjustments there

@r3code nice to see you using this! let me know if you see any issues


Start Pulsar in TLS Only mode and deprecate tlsEnabled flag.

Motivation
Start Pulsar services (broker, proxy, websocket, discovery) in TLS only mode, so that they only listen on TLS ports.
Once TlsPort is set tlsEnabled flag becomes redundant information - hence getting rid of the flag in relevant components.
Modifications
a. Changed the Ports to Option in the configuration file.
b. In Websocket Service there was a bug where we used 'tlsEnabled' flag to start listening on a TLS port and to talk to broker in on serviceUrlTls - separated the flag into two (tlsEnabled and brokerClientTlsEnabled) and deprecated tlsEnabled.
c. Fixed a lot of tests which relied on tlsEnabled flag.
Result
Brokers can now listen to TLS only port.

rerun java8 tests

rerun C++ tests

rerun cpp tests

rerun integration tests

rerun java8 tests

@jai1 I think you might need to look into the cpp tests and integration tests. I guess your changes change some tls settings.

rerun java8 tests


chore: update dependencies

License: MIT
Signed-off-by: Henrique Dias<EMAIL_ADDRESS>

@hacdias i don't see the travis builds in the details list for this PR. What happened?

Hm?

oh github. that is weird.


Error parsing '@' character when using json5

Reporting a bug?
If a translation file is in json5 format, and contains the "@", even if escaped or written as a \u0040, results in an error code 10: Invalid link format. For example, if someone uses different email addresses per locale, or an account name, etc.
Not sure if there are other characters that also break.
According to json5 documentation (https://spec.json5.org/#escapes), the '@' doesn't need to be escaped. Even if escaped, it creates the same error.
Error log:
Invalid linked format (error code: 10) in /project/sandbox/translations/en.json5
target message<EMAIL_ADDRESS>target message path: anEmailString
Expected behavior
According to json5 docs, should support the @ symbol.
Reproduction
CodeSandBox: https://codesandbox.io/p/sandbox/determined-vaughan-5qbybf
Issue Package
unplugin-vue-i18n
System Info
All regular browsers.

  System:
    OS: Linux 5.15 Ubuntu 20.04.5 LTS (Focal Fossa)
    CPU: (16) x64 Intel(R) Core(TM) i7-10700K CPU @ 3.80GHz
    Memory: 25.56 GB / 31.05 GB
    Container: Yes
    Shell: 5.0.17 - /bin/bash
  Binaries:
    Node: 18.13.0 - ~/.nvm/versions/node/v18.13.0/bin/node
    Yarn: 1.22.17 - ~/.yarn/bin/yarn
    npm: 8.19.3 - ~/.nvm/versions/node/v18.13.0/bin/npm

Screenshot
No response
Additional context
No response
Validations

[X] Read the Contributing Guidelines.
[X] Read the README
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A open a GitHub Discussion.

@ is special character, because @ is used in linked messages
docs says:

linked message
https://vue-i18n.intlify.dev/guide/essentials/syntax.html#linked-messages

Special Characters
https://vue-i18n.intlify.dev/guide/essentials/syntax.html#special-characters

You can avoid with literal interpolation:
https://vue-i18n.intlify.dev/guide/essentials/syntax.html#literal-interpolation
Thanks!

Is there a way to disable this? In some cases translations are exposed to the customers and it's a pain to explain to e.g. business people they have to code emails as follows<EMAIL_ADDRESS>

The @ is a really bad choice to use as a special character because it's used literally everywhere in language now, not least emails and tagging like @kazupon !
And literal interpolation is just broken, no-one in their right mind is going to do a spaghetti mess like<EMAIL_ADDRESS>like the example above, it's also not that flexible for different languages (which is like the whole point we're here).
Can there not be a way to either change the special character or to exclude it as in this comment https://github.com/intlify/bundle-tools/issues/53#issuecomment-1080330930
I can't be the only one with this issue. It is a great library otherwise though.

@cozumel424 yes agree with you. I must note, however, following started working for me<EMAIL_ADDRESS>In any case, I can't provide it to translators.


WhitespaceAround fails on typedefs without a semicolon

Apparently this is valid Haxe code:
typedef Test = Int

(http://try.haxe.org/#3055d)
./source/Test.hx:1: character 0 : Error: Check WhitespaceAround failed: no more tokens

{
	"type": "WhitespaceAround",
	"props": {
		"severity": "WARNING",
		"tokens": ["="]
	}
}

Man, issues are not open for long on this repo! :+1:

I need to switch to my regular job now but @AlexHaxe is on a roll :) :+1:


Refactor usage of vtk to avoid libgl errors

See: https://github.com/plotly/dash-docs/pull/1091

About

Description of changes

Pre-Merge checklist

[ ] The project was correctly built with npm run build.
[ ] If there was any conflict, it was solved correctly.
[ ] All changes were documented in CHANGELOG.md.
[ ] All tests on CircleCI have passed.
[ ] All Percy visual changes have been approved.
[ ] Two people have :dancer:'d the pull request. You can be one of these people if you are a Dash core contributor.

You are still using<EMAIL_ADDRESS>I would think so...

Let me upgrade the vtk dependencies

All the tests failed:
 tests/test_render.py::test_usage_vtk_cfd ⨯                       75% ███████▌
 tests/test_render.py::test_usage_algorithm ✓                     88% ████████▊
 tests/test_render.py::test_synthetic_volume_rendering ✓         100% ██████████
=============================== warnings summary ===============================
tests/test_render.py::test_pyvista_terrain_following_mesh
  /home/circleci/project/venv/lib/python3.7/site-packages/transforms3d/quaternions.py:26: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
  Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
    _MAX_FLOAT = np.maximum_sctype(np.float)

tests/test_render.py::test_pyvista_terrain_following_mesh
  /home/circleci/project/venv/lib/python3.7/site-packages/transforms3d/quaternions.py:27: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
  Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
    _FLOAT_EPS = np.finfo(np.float).eps

tests/test_render.py::test_pyvista_point_cloud
tests/test_render.py::test_pyvista_point_cloud
  /home/circleci/project/venv/lib/python3.7/site-packages/vtk/util/numpy_support.py:74: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
  Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
    _vtk_np = {vtk.VTK_BIT:numpy.bool,

-- Docs: https://docs.pytest.org/en/stable/warnings.html

Results (67.61s):
       4 passed
       4 failed
         - tests/test_render.py:32 test_pyvista_terrain_following_mesh
         - tests/test_render.py:32 test_slice_rendering
         - tests/test_render.py:32 test_volume_rendering
         - tests/test_render.py:32 test_usage_vtk_cfd

Exited with code exit status 1
CircleCI received exit code 1

So it seems like vtkmodules was added in 8.2.0 (see release 8.2.0 vs 8.1.2 where it didn't exist). I'll change the requirements

Wow, seems like circleci for python-3.6 can't even find vtk==9.0.1 on pypi 🤦
  Could not find a version that satisfies the requirement vtk>=8.2.* (from -r requirements.txt (line 4)) (from versions: 8.0.0.dev20170717, 8.1.0, 8.1.1, 8.1.2)
No matching distribution found for vtk>=8.2.* (from -r requirements.txt (line 4))
You are using pip version 18.1, however version 21.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

I feel we should go with a try/except when it comes to importing the library, this way if vtkmodules are not available it falls back to vtk. For dash_vtk/utils/vtk.py, it'd be something like that:
try:
    from vtkmodules.util.numpy_support import vtk_to_numpy
    from vtkmodules.vtkFiltersGeometry import vtkGeometryFilter
except:
    from vtk.util.numpy_support import vtk_to_numpy
    from vtk.vtkFiltersGeometry import vtkGeometryFilter

Thoughts? @jourdain

Sounds good
On Sat, Feb 6, 2021 at 14:01 Xing Han Lu<EMAIL_ADDRESS>wrote:

I feel we should go with a try/except when it comes to importing the
library, this way if vtkmodules are not available it falls back to vtk. For
dash_vtk/utils/vtk.py, it'd be something like that:
try:
from vtkmodules.util.numpy_support import vtk_to_numpy
from vtkmodules.vtkFiltersGeometry import vtkGeometryFilterexcept:
from vtk.util.numpy_support import vtk_to_numpy
from vtk.vtkFiltersGeometry import vtkGeometryFilter
Thoughts? @jourdain https://github.com/jourdain
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/plotly/dash-vtk/pull/21#issuecomment-774542835, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AACH45UXLS3ZSYM6FY4QLFTS5WU4HANCNFSM4XDYDOHA
.

@jourdain I don't think vtkmodules resolve the problem.
See the logs from a test heroku app that imports vtkmodules instead of `vtk:
2021-02-08T18:36:15.429362+00:00 app[web.1]:     return _bootstrap._gcd_import(name[level:], package, level)
2021-02-08T18:36:15.429362+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/vtkmodules/all.py", line 29, in <module>
2021-02-08T18:36:15.429363+00:00 app[web.1]:     from .vtkRenderingOpenGL2 import *
2021-02-08T18:36:15.429430+00:00 app[web.1]: ImportError: libGL.so.1: cannot open shared object file: No such file or directory
2021-02-08T18:36:15.430248+00:00 app[web.1]: [2021-02-08 18:36:15 +0000] [10] [INFO] Worker exiting (pid: 10)
2021-02-08T18:36:15.593373+00:00 heroku[router]: at=error code=H13 desc="Connection closed without response" method=GET path="/" host=test-vtk-deploy.herokuapp.com request_id=68847d42-ab72-4b70-a9ff-94be92a42c1e fwd="<IP_ADDRESS>" dyno=web.1 connect=1ms service=582ms status=503 bytes=0 protocol=https
2021-02-08T18:36:15.696799+00:00 app[web.1]: [2021-02-08 18:36:15 +0000] [4] [INFO] Shutting down: Master
2021-02-08T18:36:15.697103+00:00 app[web.1]: [2021-02-08 18:36:15 +0000] [4] [INFO] Reason: Worker failed to boot.

Arg... I guess you may have to do the patching if you can like Jcr was talking about...

Do you have the full stack? I'm not sure why we get vtkmodules.all

Unfortunately heroku truncates the full stack :/

I'm not sure how to patch a library inside dash-vtk's setup.py, which is what we would need if we are to use it inside utils; right?

I don't know if there is a post-setup or something that can be executed once vtk get installed to edit/patch it.

This should work for 9.0.1 but I feel this approach might be very brittle and would require a lot of error catching since there's a lot of variables (people uninstalling/reinstalling vtk, someone installing dash-vtk alongside vtk but uses vtk for other purposes and finds out they can't import libgl anymore, etc.). In the end it could create more friction.
I'd like to backtrack on the reasons why this PR was started and my various attempts at bypassing libgl. I feel it might be better to write instructions in the Readme.md and the docs to point people towards the right approach to install libgl (aptfile on heroku, apt-packages file on dash enteprise, apt-get instructions for local, etc.). Although those capabilities are not needed for dash-vtk, it sounds safer than modifying the source code of a dependency just to use dash-vtk. Sorry for all the trouble.

@jourdain I think this is ready to be merged

Yes LGTM


Platform dependency + version constraint not deduped in POM

Expected Behavior
dependencies {
  compile platform("org.apache.spark:spark-dist_2.11-hadoop-palantir-bom")
  constraints {
    compile "org.apache.spark:spark-dist_2.11-hadoop-palantir-bom:2.5.0-palantir.7"
  }
}

When I depend on a published platform without a version, and inject the version separately through a vanilla constraint, I expect the resulting POM to contain a single entry in the <dependencyManagement> section.
  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-dist_2.11-hadoop-palantir-bom</artifactId>
        <version>2.5.0-palantir.7</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>

Current Behavior
I get two entries for the same dependency (same groupId and artifactId), one of which doesn't have a version. This is not a correct POM.
  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-dist_2.11-hadoop-palantir-bom</artifactId>
        <version>2.5.0-palantir.7</version>
        <scope>compile</scope>
      </dependency>
      <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-dist_2.11-hadoop-palantir-bom</artifactId>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>

Note that gradle imports the correct BOM, it's just the published POM that ends up being incorrect.
Context

I'm trying to declare a BOM dependency as described here, but have the version injected from somewhere else, which is why it ends up as a constraint.
Steps to Reproduce (for bugs)
Please see linked repo.
Your Environment

Build scan URL: https://gradle.com/s/ilk7ycapy6w5u
Repo: https://github.com/dansanduleac/gradle-test/tree/ds/constrained-platform-double-entry

Don't we resolve prior to writing the pom file though?

No we don't. We only do this when you have configured a version mapping strategy, which is fairly recent addition to Gradle.

However, imagining that platform imports are just read out of the default configuration, then could we not use the resolved default to figure out the correct version?

First, I don't really understand what you mean. Beside that, there's no "platform imports are read of out a configuration". We parse the POM files, build a model, and use the model consistency. There shouldn't be anything using the default configuration, which even doesn't exist at all for Maven.


Build Errors - openssl

I am getting build errors while compiling flocker for repository flocker-emc and branch feat_xio_emc.
Steps to recreate:
a) clone repository
b) vagrant up
c) vagrant ssh
d) cd /vagrant
e) tox .
Any help is appreciated.
Errors look like:
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c src/cryptography/hazmat/bindings/pycache/_Cryptography_cffi_f3e4673fx399b1113.c -o build/temp.linux-x86_64-2.7/src/cryptography/hazmat/bindings/pycache/_Cryptography_cffi_f3e4673fx399b1113.o
src/cryptography/hazmat/bindings/pycache/_Cryptography_cffi_f3e4673fx399b1113.c:216:25: fatal error: openssl/aes.h: No such file or directory
#include <openssl/aes.h>
^
compilation terminated.
error: command 'gcc' failed with exit status 1
----------------------------------------
Command "/vagrant/.tox/sphinx/bin/python2.7 -c "import setuptools, tokenize;__file__='/tmp/pip-build-IX20By/cryptography/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-gkwNcF-record/install-record.txt --single-version-externally-managed --compile --install-headers /vagrant/.tox/sphinx/include/site/python2.7/cryptography" failed with error code 1 in /tmp/pip-build-IX20By/cryptography

Hi,
It looks like the problem is that you are missing the OpenSSL development headers - apt-get install libssl-dev on Ubuntu and yum install openssl-devel on RH, CentOS etc
There may be some other non-python dependencies that are required (libffi development headers for example).  You can see what the dependencies are needed by each platform by looking in the Dockerfiles:
Ubuntu 14.04
CentOS 7
Fedora-20
Hope this helps!

The main culprit over here the cryptography package … it has dependency on open-ssl and libffi
https://cryptography.io/en/latest/installation/
I also ran into issues with a bug in the same package
https://bugs.launchpad.net/openstack-gate/+bug/1446882
The bug  has a workaround …

Hey,
thanks for your investigations :-)
So, have you installed open-ssl / libffi? If yes, have they had an effect on the original error?

Hi
I followed the steps as given in above link and the bug. I am able to compile anf run test case for loop back driver. I still have to test tux but I believe that should work too.
Thanks


Add BFCL Evaluation GitHub Action on Pull Requests

Description
Adds a GitHub Actions workflow that automatically runs on pull requests and comments with evaluation results generated by bfcl generate and evaluated by bfcl evaluate.
Setup Required
Repository administrators need to add:

Settings → Secrets and Variables → Actions
New repository secret:

Name: OPENAI_API_KEY
Value: Your OpenAI API key

The workflow will automatically run on PRs once the secret is configured.
Results
It should look something like this:

Permissions for the workflow to comment on PRs will also need to be provided.


stubs mappings directory should be the same for convert and run mojo

Current coverage is 56.54% (diff: 50.00%)

Merging #95 into master will not change coverage

@@             master        #95   diff @@
==========================================
  Files           166        166
  Lines          4550       4550
  Methods           0          0
  Messages          0          0
  Branches        763        763
==========================================
  Hits           2573       2573
  Misses         1667       1667
  Partials        310        310

Powered by Codecov. Last update 61ca5a4...9ba31ee


ip match fixes

It looks like  --nodata-radius is not having an effect.
It would be nice to filter IPs from the top/bottom, as well as the sides. It could be useful if there are distortions at the edges of the images which are unaccounted for, and which complicate the bundle_adjustment.

--nodata-radius will filter out along the edges which will work for removing edge_ip as long as nodata in the image interior does not make that problematic.  Closing this unless it becomes a problem.


pishrink with corrupt image fails after 10 minutes copying.

my command:    pishrink  'imagefile'   'newimagefile'
in this case, pishrink copies the imagefile and than runs parted.
I have a corrupt imagefile.  Linux command cp doesn't recognize a problem.
Copying takes 10 minutes on a RPi 3B.  After copying, Linux parted runs in an error.
I would prefer running parted checking the imagefile before copying the image.

The parted commands change the image, so it would be counter productive to the whole goal of having to copy it. I also feel the frustration from having to wait on the copy and I'm trying to come up with ways to make it faster if I can.


[MINOR] Fix spark2.4 do not support bucket index row writer

Change Logs
CI fix after #9156. Add bucket index bulk insert row writer support for spark2
Impact
fix ci
Risk level (write none, low medium or high below)
none
Documentation Update
NONE
Contributor's checklist

[x] Read through contributor's guide
[x] Change Logs and Impact were stated clearly
[x] Adequate tests were added if applicable
[x] CI passed

@hudi-bot run azure

The second failure case passed in first CI. Let's merge it to unblock other PRs.


Incorrect use of "Release Name" in release pipeline

The release checklist indicates that the release name should be in the form of X.0 for a VMR-based servicing release. This is incorrect because it creates a GH release in the dotnet/dotnet repo with that title. For example, it originally created this 8.0.2 release with a title of ".NET 8.0" when it should have been ".NET 8.0.2".
Changing the checklist to use a three-part version number is simple enough. However, changing that impacts the release announcement that gets created in the dotnet/source-build repo. For example, the corresponding announcement of the 8.0.2 release is here: https://github.com/dotnet/source-build/discussions/4126. This has a title of ".NET 8.0 February 2024 Update - .NET 8.0.2 and SDK 8.0.102". If we changed the checklist to say that the release name should be ".NET 8.0.2", that would impact the beginning portion of this title by changing it from ".NET 8.0" to ".NET 8.0.2".
We either need to change the pipeline to vary the value according between the release title and announcement title OR we need to change the naming patterns between these two titles to more closely align so that no variation is needed.

I guess we can switch the release title to use ReleaseVersion from the release config, which in this example would be equal to 8.0.2, and leave the announcement to use releaseName parameter mentioned in the release checklist

PR - https://dev.azure.com/dnceng/internal/_git/dotnet-release/pullrequest/37633

PR merged, closing


Json file not properly formatted in case a string contains a single double quote

In Quick.Config.JSON when a string contains a single double quote, the resultant file is not formatted properly

In file JSON Utils, you need to swap the order of two lines at the end. It worked for me, but please review.
From:
isEscape := (c = '') and not isEscape;
if not isEscape and (c = '"') then isIntoString := not isIntoString;
to:
if not isEscape and (c = '"') then isIntoString := not isIntoString;
isEscape := (c = '') and not isEscape;

When a string contains a single double quote? Then it's not a good json object...Could you explain please?

If you have a string property (e.g. SQL) and store in this string for example:
select * from actor where name like '% ABC" %'
then after you saving, you will notice that the config file is not properly formatted. Making the change I mentioned it worked for me.

You're right. I've applied your changes. Thanks.


How do I convert my EOS ERC20 to EOS?

How do I convert my EOS ERC20 to EOS?

Off topic for eosjs


pass in Pipeline to ClientBuilder

I did not make Pipeline settable in #527 because I wasn't sure how it should be done in #510. Should it be added to ClientBuilder and Client like this. Would we expect someone to call new_pipeline and then set the retry policies?

That's a good question; given we're on track to convert all endpoints to builders, it might make sense for the client to follow a similar pattern. I was hoping that once we have the endpoints in place, we could take a look at the client builders after. But it seems I wasn't the only one thinking of that (:
What is a pipeline? [^me]
[^me]: I'm writing this down in part for myself as well, since I haven't really gone through the pipeline code until now. I'm trying to keep this section brief tho (:
A "pipeline" in our SDK is an HTTP backend + corresponding middleware [^surf]. It's defined as follows:
[^surf]: In the surf http client this would correspond to a Client instance.
#[derive(Debug, Clone)]
pub struct Pipeline {
    http_client: Arc<dyn HttpClient>,
    pipeline: Vec<Arc<dyn Policy>>,
}

Pipeline::new creates a new pipeline and sets up default policies. For now this is "telemetry", and "custom headers injector". And depending on the build mode, we can switch out to a "recorder" HTTP backend.
What would usage of our clients look like with this PR?
Policies on the pipeline currently cannot be modified or extended, which is what the PR under discussion attempts to address. Taking CosmosClient as an example again, assuming we would implement this PR as-is, using it to extend the pipeline with a custom policy would be as follows:
let account = std::env::var("COSMOS_ACCOUNT").unwrap();
let master_key = std::env::var("COSMOS_MASTER_KEY").unwrap();
let auth = AuthorizationToken::primary_from_base64(&master_key)?;

// Create a new Cosmos client with a custom policy
let options = CosmosOptions::default();
let mut pipeline = Pipeline::new(
    option_env!("CARGO_PKG_NAME"),
    option_env!("CARGO_PKG_VERSION"),
    ClientOptions::default(),
    vec![],                               // per-call policies
    vec![Arc::new(ExamplePolicy::new())], // per-retry policies
);
let client = CosmosClient::new(account, auth, options, pipeline);

This PR moves the construction of Pipeline into the customer's code, requiring them to handle pipelines in every SDK. The Pipeline::new_pipeline constructor simplifies this somewhat (not shown here; it doesn't take any parameters) -- but when a customer desires to add a custom policy, it definitely is a lot to take in.
An alternative interface for constructing clients
I see the "pipeline" and "options" parameters as mirrors to the interface issues we faced with "context" and "options" in our endpoint constructors. And I believe we should be able to address these in a similar manner. I'd like to propose we consider constructing clients using an API along these lines instead:
let account = std::env::var("COSMOS_ACCOUNT").unwrap();
let master_key = std::env::var("COSMOS_MASTER_KEY").unwrap();
let auth = AuthorizationToken::primary_from_base64(&master_key)?;

// Create a new Cosmos client with a custom policy
let client = CosmosClient::new(account, auth)
    .retry_policy(ExamplePolicy::new());

This would move all CosmosClientOptions optional parameters into a builder, and add two new methods: retry_policy and call_policy which take an impl Policy each. I feel like this would be a more ergonomic experience, with the benefit of having the flexibility to be able to extend the policies.

This requires further discussion, so I opened:

#591.


Blazor Preview4 updates

Fixes #11837
Here's the sample app :point_right: https://github.com/guardrex/BlazorSamplePre4

Overview
Get started
Hosting models
Build your first app
Components
Forms and validation
Component libraries
Layouts
Dependency injection
Routing
JavaScript interop
Debug
Host and deploy

Overview
Client-side
Server-side
Configure the Linker

Common sample app runs, but ...

I had to comment out the code-behind experience (BlazorRocks.razor and BlazorRocksBase.cs) to get the app to compile. The error (thrown dozens of times) is:
AZORGENERATE : error RZ3008: Tag helpers cannot target tag name '<global namespace>.BlazorRocksBase' because it contains a '<' character.

The TabSet example regressed: No styles now (but the code that toggles the hidden tab works).

The Templated Components regressed: I had to comment out the TableTemplate and ListViewTemplate in Pages/TemplatedComponents.razor. The error thrown is ...
obj\Debug\netstandard2.0\Razor\Pages\TemplatedComponents.razor.g.cs(56,36): error CS0103: The name 'context' does not exist in the current context

From the blog post, I still need to add ...

Improved SignalR connection lifetime handling
Improved disconnect & reconnect handling opportunities

Would you like me to continue added them to the Server-side section of the Hosting models topic where I added the other blog post sections?

Would you like me to add them to the Server-side section of the Hosting models topic where I added the other blog post sections?

That sounds reasonable. But I'd also like to get the Get Started and Overview updates in ASAP.

Sure ... already done. I was collecting changes here as you go. I'll shoot those up now.

Looks like if I move TableTemplate and ListViewTemplate into the Pages folder, it fixes that problem.

Now, I'm wondering if the TabSet example is broken for the same reason.
Do I need to do something special now for those components if they're over in the Components folder but being used by a component in the Pages folder?

Confirmed! All problems fixed by moving the components in the Components folder over to the Pages folder. Should we just go with that?

@guardrex Components have a namespace like normal .NET types, and in Preview4 we resolve component names using the in scope namespaces. The BlazorSample.Components namespace just needs to be in scope for you to use the components in that namespace. Just add @using BlazorSample.Components to your root _Imports.razor file.

oh ... ok ... I'll fix it back up ... properly :smile:

Very good ... I'll merge it when the build completes.
I'll reread everything in the morning and see if I spot any 🙈 in the topics.

No response ... he must be 🏃. Merging now!


The console in the remote desktop fails

Do not receive any input
Steps to reproduce
123456

Expected behavior
123456

Actual behavior
Null

Environment data

win 7.0.1
system server1909

Please provide a more thorough issue description and reproduction steps.

Not compatible with rdp10

Not compatible with rdp10

I am seeing the same thing. I have a Windows 10 box, with pwsh v7.0.1 installed. If I use Remote Desktop 10.3.11 on MacOS 10.15.4 to remote into the Windows 10 machine and do:

WIN+R
pwsh and enter

If I type nothing shows up in the window. I can hit the <enter> key and it will do a newline.
But if I open cmd.exe or the older powershell.exe then the keyboard works fine.

Please provide a more thorough issue description and reproduction steps.

@vexx32 I'm pretty sure I see the same thing @he852100 mentions.

Seems to be reproducible in a Win10 session when connecting via HTML5 client to a remote Windows Virtual Desktop session.
Using the Windows Virtual Desktop Windows client pwsh v7.0.1 can receive inputs normally but when using the HTML5 client it fails to pass any input outside of CTRL-V or Enter

I'm using Remmina (Linux remote-session client) to a Windows 10 Pro 21H2 system via its RDP plugin (HTTP transport), and exactly the same. Input works to any other program without issue.
Running pwsh.exe (version 7.2.1), either in a newly-launched Windows Terminal window, or by executing it from the command prompt of an existing cmd.exe session, results in no response to any of the normally-echoed basic ASCII command line characters.
The Enter key works, Ctrl combinations seem to work, keyboard paste via Shift+Insertworks, PgUp/PgDn work... it's just the shell prompt itself that seems to be ignoring most attempts to communicate with it.
Pasting in a command line and typing Enter DOES execute the pasted command, if that's any help.

Running pwsh.exe (version 7.2.1), either in a newly-launched Windows Terminal window

Sorry, not Windows Terminal. That runs Windows PowerShell, and encourages me to upgrade to PowerShell 6. PowerShell 7 executes under the Console Window Host.

Unfortunately, no changes following the 7.2.2 update. 😞

I'm experiencing a similar problem on a brand new install of Windows Server 2022 21H2, I can type anywhere on notepad, cmd.exe, however as soon as I type: "powershell" and powershell is running I can't type on the window, same if I start powershell on its own. In my case I'm accessing the Win2022 instance via another Windows 2019 server acting as a bastion. So The full chain is Linux/Remmina->Windows Server 2019->Windows Server 2022.
Hope this helps.

And even more bizarre, if I run an interactive powershell script the powershell console (running powershell via cmd.exe) accepts input normally as soon as the script ends the console stops accepting input.

I had this issue and fixed it via a Remmina preference change.
Remmina Preferences -> RDP -> untick "use client keyboard mapping"
Also have confirmed that re-ticking the option causes the problem to come back

I was experiencing the same issue. The fix suggested by @barrybingo above fixed it for me. Thanks!

I had this issue and fixed it via a Remmina preference change.
Remmina Preferences -> RDP -> untick "use client keyboard mapping"
Also have confirmed that re-ticking the option causes the problem to come back

@barrybingo @ngcdev
Huh — holy crap, that does work. Nice find, thanks @barrybingo. Fascinating that should be the issue, especially since it's a systemwide preference and not a per-connection thing.
I looked at the Remmina code to see why it would make a difference.
When that option is enabled in the RDP preferences, the RDP plugin's code translates certain keypresses (ones that aren't modifiers, spaces, function keys, numpad keys, or any key combined with a modifier) into unicode code points, instead of into raw keycodes:
Remmina/plugins/rdp/rdp_event.c@763:809:
		if (!rfi->use_client_keymap) {
			hardware_keycode = event->hardware_keycode;
			if (rfi->keymap) {
				for (ik = 0; ik < rfi->keymap->len; ik++) {
					kep = &g_array_index(rfi->keymap, RemminaPluginRdpKeymapEntry, ik);
					if (hardware_keycode == kep->orig_keycode) {
						hardware_keycode = kep->translated_keycode;
						break;
					}
				}
			}
			scancode = freerdp_keyboard_get_rdp_scancode_from_x11_keycode(hardware_keycode);
			if (scancode) {
				rdp_event.key_event.key_code = scancode & 0xFF;
				rdp_event.key_event.extended = scancode & 0x100;
				remmina_rdp_event_event_push(gp, &rdp_event);
				keypress_list_add(gp, rdp_event);
			}
		} else {
			unicode_keyval = gdk_keyval_to_unicode(event->keyval);
			/* Decide when whe should send a keycode or a Unicode character.
			 * - All non char keys (Shift, Alt, Super) should be sent as keycode
			 * - Space should be sent as keycode (see issue #1364)
			 * - All special keys (F1-F10, numeric pad, Home/End/Arrows/PgUp/PgDn/Insert/Delete) keycode
			 * - All key pressed while Ctrl or Alt or Super is down are not decoded by gdk_keyval_to_unicode(), so send it as keycode
			 * - All keycodes not translatable to unicode chars, as keycode
			 * - The rest as Unicode char
			 */
			if (event->keyval >= 0xfe00 ||                                                  // Arrows, Shift, Alt, Fn, num keypad…
			    event->hardware_keycode == 0x41 ||                                          // Spacebar
			    unicode_keyval == 0 ||                                                      // Impossible to translate
			    (event->state & (GDK_MOD1_MASK | GDK_CONTROL_MASK | GDK_SUPER_MASK)) != 0   // A modifier not recognized by gdk_keyval_to_unicode()
			    ) {
				scancode = freerdp_keyboard_get_rdp_scancode_from_x11_keycode(event->hardware_keycode);
				rdp_event.key_event.key_code = scancode & 0xFF;
				rdp_event.key_event.extended = scancode & 0x100;
				if (rdp_event.key_event.key_code) {
					remmina_rdp_event_event_push(gp, &rdp_event);
					keypress_list_add(gp, rdp_event);
				}
			} else {
				rdp_event.type = REMMINA_RDP_EVENT_TYPE_SCANCODE_UNICODE;
				rdp_event.key_event.unicode_code = unicode_keyval;
				rdp_event.key_event.extended = False;
				remmina_rdp_event_event_push(gp, &rdp_event);
				keypress_list_add(gp, rdp_event);
			}

When a keyboard event of type REMMINA_RDP_EVENT_TYPE_SCANCODE_UNICODE is processed, it's delivered to the remote server using the RDP message TS_UNICODE_KEYBOARD_EVENT.
Remmina/plugins/rdp/rdp_plugin.c@228-234:
		case REMMINA_RDP_EVENT_TYPE_SCANCODE_UNICODE:
			/*
			 * TS_UNICODE_KEYBOARD_EVENT RDP message, see https://msdn.microsoft.com/en-us/library/cc240585.aspx
			 */
			flags = event->key_event.up ? KBD_FLAGS_RELEASE : KBD_FLAGS_DOWN;
			input->UnicodeKeyboardEvent(input, flags, event->key_event.unicode_code);
			break;

Apparently, presumably due to its own fairly low-level keyboard processing, Unicode input events aren't properly handled by PowerShell 7.

For people having this problem with Microsoft Remote Desktop for Mac, switching Keyboard Mode solves it!

Did you try the fixes folks mentioned above (disabling client keyboard mapping in the RDP settings)?
Beyond that, I have a vague recollection that it might be worth seeing what version of PSReadLine is on the box you're connecting to and whether it might need updating.

As far as i am aware the RD web client is delivered from from RDS services and there is no way to confgiure the settings for the session other than suppressing telemetry. https://learn.microsoft.com/en-us/windows-server/remote/remote-desktop-services/clients/remote-desktop-web-client-admin i have raised this at the suggested place in that article too.

@vexx32 well there never used to be any settings but it appears there is an option now to turn on "Input Method Editor" in the RDP Web client. Turning that on and setting keyboard to "other" has fixed the issue for me!

Ah heck yeah, nice find! 💖

Indeed. In fact, perhaps someone from the PowerShell team (if anyone's out there listening) — or @he852100, of course, if you're still around — could retitle this issue to something slightly more precise for the benefit of future users searching the open issues, now that we've nailed down the exact pathology? I'm thinking something like,
"Can't type into PowerShell window when connected via Remote Desktop"
or
"Keyboard input ignored by PowerShell running in Remote Desktop session"

Issue still exists. But Microsoft prefers to ignore the issue. since there is a workarround, Please specify
= WILL NOT INVESTIGATE OR SOLVE

@fd694309 I mean, that's pretty heavily implied by the issue being auto-closed. I don't see why they have to be any more explicit than that.
And in fairness, I doubt anyone from Microsoft has seen this issue since 2020; certainly not since the workaround was discovered.
You could try opening a new issue, specifically reporting that PowerShell only responds to low-level scancode keyboard input, but not Unicode keyboard input, over Remote Desktop. Now that we know precisely what the issue actually is.


XML parsing error when trying to delete lots of objects on s3

When I open a zarr-like array on s3 with w mode (i.e., I want to delete the original objects), I occasionally get an xml parsing error:

Long traceback
---------------------------------------------------------------------------
ParseError                                Traceback (most recent call last)
~/miniconda3/envs/cosem-flows/lib/python3.8/site-packages/botocore/parsers.py in _parse_xml_string_to_dom(self, xml_string)
    451             parser.feed(xml_string)
--> 452             root = parser.close()
    453         except XMLParseError as e:
ParseError: no element found: line 2, column 0
During handling of the above exception, another exception occurred:
ResponseParserError                       Traceback (most recent call last)
 in
3     source_arrays = dict(source_group.arrays())
4     create_success = False
----> 5     target_group, target_arrays = initialize_group(dest_store_path,
6                                                source_group.path,
7                                                arrays=list(source_arrays.values()),
~/miniconda3/envs/cosem-flows/lib/python3.8/site-packages/fibsem_tools/io/io.py in initialize_group(container_path, group_path, arrays, array_paths, chunks, group_attrs, compressor, array_attrs, mode, **kwargs)
295 ) -> Tuple[zarr.hierarchy.group, Tuple[zarr.core.Array]]:
296
--> 297     zgroup = access(
298         os.path.join(container_path, group_path), mode=mode, attrs=group_attrs, **kwargs
299     )
~/miniconda3/envs/cosem-flows/lib/python3.8/site-packages/fibsem_tools/io/io.py in access(path, mode, **kwargs)
157
158         if is_container:
--> 159             return accessor(path_outer, path_inner, mode=mode, **kwargs)
160         else:
161             return accessor(path_outer, mode=mode, **kwargs)
~/miniconda3/envs/cosem-flows/lib/python3.8/site-packages/fibsem_tools/io/zarr.py in access_n5(dir_path, container_path, **kwargs)
165 ) -> Any:
166     dir_path = N5FSStore(dir_path, **kwargs.get("storage_options", {}))
--> 167     return access_zarr(dir_path, container_path, **kwargs)
168
169
~/miniconda3/envs/cosem-flows/lib/python3.8/site-packages/fibsem_tools/io/zarr.py in access_zarr(dir_path, container_path, **kwargs)
155         ):
156             delete_zbranch(tmp)
--> 157     array_or_group = zarr.open(dir_path, path=str(container_path), **kwargs)
158     if kwargs.get("mode") != "r" and len(attrs) > 1:
159         array_or_group.attrs.update(attrs)
~/miniconda3/envs/cosem-flows/lib/python3.8/site-packages/zarr/convenience.py in open(store, mode, **kwargs)
84             return open_array(store, mode=mode, **kwargs)
85         else:
---> 86             return open_group(store, mode=mode, **kwargs)
87
88     elif mode == "a":
~/miniconda3/envs/cosem-flows/lib/python3.8/site-packages/zarr/hierarchy.py in open_group(store, mode, cache_attrs, synchronizer, path, chunk_store, storage_options)
1169
1170     elif mode == 'w':
-> 1171         init_group(store, overwrite=True, path=path, chunk_store=chunk_store)
1172
1173     elif mode == 'a':
~/miniconda3/envs/cosem-flows/lib/python3.8/site-packages/zarr/storage.py in init_group(store, overwrite, path, chunk_store)
486
487     # initialise metadata
--> 488     _init_group_metadata(store=store, overwrite=overwrite, path=path,
489                          chunk_store=chunk_store)
490
~/miniconda3/envs/cosem-flows/lib/python3.8/site-packages/zarr/storage.py in _init_group_metadata(store, overwrite, path, chunk_store)
500     if overwrite:
501         # attempt to delete any pre-existing items in store
--> 502         rmdir(store, path)
503         if chunk_store is not None:
504             rmdir(chunk_store, path)
~/miniconda3/envs/cosem-flows/lib/python3.8/site-packages/zarr/storage.py in rmdir(store, path)
120     if hasattr(store, 'rmdir'):
121         # pass through
--> 122         store.rmdir(path)
123     else:
124         # slow version, delete one key at a time
~/miniconda3/envs/cosem-flows/lib/python3.8/site-packages/fibsem_tools/io/storage.py in rmdir(self, path)
209         store_path = self.dir_path(path)
210         if self.fs.isdir(store_path):
--> 211             self.fs.rm(store_path, recursive=True)
212
213     def getsize(self, path=None):
~/miniconda3/envs/cosem-flows/lib/python3.8/site-packages/fsspec/asyn.py in wrapper(*args, **kwargs)
70     def wrapper(*args, **kwargs):
71         self = obj or args[0]
---> 72         return sync(self.loop, func, *args, **kwargs)
73
74     return wrapper
~/miniconda3/envs/cosem-flows/lib/python3.8/site-packages/fsspec/asyn.py in sync(loop, func, timeout, *args, **kwargs)
51     event.wait(timeout)
52     if isinstance(result[0], BaseException):
---> 53         raise result[0]
54     return result[0]
55
~/miniconda3/envs/cosem-flows/lib/python3.8/site-packages/fsspec/asyn.py in _runner(event, coro, result, timeout)
18         coro = asyncio.wait_for(coro, timeout=timeout)
19     try:
---> 20         result[0] = await coro
21     except Exception as ex:
22         result[0] = ex
~/miniconda3/envs/cosem-flows/lib/python3.8/site-packages/s3fs/core.py in _rm(self, path, recursive, **kwargs)
1508         dirs = [p for p in paths if not self.split_path(p)[1]]
1509         # TODO: fails if more than one bucket in list
-> 1510         await asyncio.gather(
1511             *[
1512                 self._bulk_delete(files[i : i + 1000])
~/miniconda3/envs/cosem-flows/lib/python3.8/site-packages/s3fs/core.py in _bulk_delete(self, pathlist, **kwargs)
1496         for path in pathlist:
1497             self.invalidate_cache(self._parent(path))
-> 1498         await self._call_s3("delete_objects", kwargs, Bucket=bucket, Delete=delete_keys)
1499
1500     async def _rm(self, path, recursive=False, **kwargs):
~/miniconda3/envs/cosem-flows/lib/python3.8/site-packages/s3fs/core.py in _call_s3(self, method, *akwarglist, **kwargs)
250             except Exception as e:
251                 err = e
--> 252         raise translate_boto_error(err)
253
254     call_s3 = sync_wrapper(_call_s3)
~/miniconda3/envs/cosem-flows/lib/python3.8/site-packages/s3fs/core.py in _call_s3(self, method, *akwarglist, **kwargs)
231         for i in range(self.retries):
232             try:
--> 233                 out = await method(**additional_kwargs)
234                 return out
235             except S3_RETRYABLE_ERRORS as e:
~/miniconda3/envs/cosem-flows/lib/python3.8/site-packages/aiobotocore/client.py in _make_api_call(self, operation_name, api_params)
138             http, parsed_response = event_response
139         else:
--> 140             http, parsed_response = await self._make_request(
141                 operation_model, request_dict, request_context)
142
~/miniconda3/envs/cosem-flows/lib/python3.8/site-packages/aiobotocore/client.py in _make_request(self, operation_model, request_dict, request_context)
158     async def _make_request(self, operation_model, request_dict, request_context):
159         try:
--> 160             return await self._endpoint.make_request(operation_model, request_dict)
161         except Exception as e:
162             await self.meta.events.emit(
~/miniconda3/envs/cosem-flows/lib/python3.8/site-packages/aiobotocore/endpoint.py in _send_request(self, request_dict, operation_model)
85         request = await self.create_request(request_dict, operation_model)
86         context = request_dict['context']
---> 87         success_response, exception = await self._get_response(
88             request, operation_model, context)
89         while await self._needs_retry(attempts, operation_model,
~/miniconda3/envs/cosem-flows/lib/python3.8/site-packages/aiobotocore/endpoint.py in _get_response(self, request, operation_model, context)
118         # If an exception occurs then the success_response is None.
119         # If no exception occurs then exception is None.
--> 120         success_response, exception = await self._do_get_response(
121             request, operation_model)
122         kwargs_to_emit = {
~/miniconda3/envs/cosem-flows/lib/python3.8/site-packages/aiobotocore/endpoint.py in _do_get_response(self, request, operation_model)
178         protocol = operation_model.metadata['protocol']
179         parser = self._response_parser_factory.create_parser(protocol)
--> 180         parsed_response = parser.parse(
181             response_dict, operation_model.output_shape)
182         if http_response.status_code >= 300:
~/miniconda3/envs/cosem-flows/lib/python3.8/site-packages/botocore/parsers.py in parse(self, response, shape)
243                 parsed = self._do_error_parse(response, shape)
244         else:
--> 245             parsed = self._do_parse(response, shape)
246
247         # We don't want to decorate event stream responses with metadata
~/miniconda3/envs/cosem-flows/lib/python3.8/site-packages/botocore/parsers.py in _do_parse(self, response, shape)
807         final_parsed['ResponseMetadata'] = self._populate_response_metadata(
808             response)
--> 809         self._add_modeled_parse(response, shape, final_parsed)
810         return final_parsed
811
~/miniconda3/envs/cosem-flows/lib/python3.8/site-packages/botocore/parsers.py in _add_modeled_parse(self, response, shape, final_parsed)
816         self._parse_non_payload_attrs(response, shape,
817                                       member_shapes, final_parsed)
--> 818         self._parse_payload(response, shape, member_shapes, final_parsed)
819
820     def _do_modeled_error_parse(self, response, shape):
~/miniconda3/envs/cosem-flows/lib/python3.8/site-packages/botocore/parsers.py in _parse_payload(self, response, shape, member_shapes, final_parsed)
856                     body_shape, original_parsed)
857         else:
--> 858             original_parsed = self._initial_body_parse(response['body'])
859             body_parsed = self._parse_shape(shape, original_parsed)
860             final_parsed.update(body_parsed)
~/miniconda3/envs/cosem-flows/lib/python3.8/site-packages/botocore/parsers.py in _initial_body_parse(self, xml_string)
942         if not xml_string:
943             return ETree.Element('')
--> 944         return self._parse_xml_string_to_dom(xml_string)
945
946     def _do_error_parse(self, response, shape):
~/miniconda3/envs/cosem-flows/lib/python3.8/site-packages/botocore/parsers.py in _parse_xml_string_to_dom(self, xml_string)
452             root = parser.close()
453         except XMLParseError as e:
--> 454             raise ResponseParserError(
455                 "Unable to parse response (%s), "
456                 "invalid XML received. Further retries may succeed:\n%s" %
ResponseParserError: Unable to parse response (no element found: line 2, column 0), invalid XML received. Further retries may succeed:
b'\n'

The TL;DR is that this operation: await self._call_s3("delete_objects", kwargs, Bucket=bucket, Delete=delete_keys) for a lot of objects (100k+) leads to this exception:  ResponseParserError: Unable to parse response (no element found: line 2, column 0), invalid XML received. Further retries may succeed: b'<?xml version="1.0" encoding="UTF-8"?>\n'
My guess is that this is a side-effect of hitting some API request rate limit, but it seems like that phenomenon should manifest as something a little more explicit than an XML parsing error.
s3fs==2021.4.0 in case that matters

ping: do you intend to chase this or try my suggestion, @d-v-b ?

I wasn't planning on it, since I don't think I understand your proposed fix at all :)

Maybe the simpler fix, is making use of asyn._run_coros_in_chunks in rm as put and get already do.

I've been investigating rate-limiting behavior with s3, and have come up with a pretty reliable reproducer for this. It does seem to be related to rate-limiting: there tend to be a lot of concurrent errors in the other delete_object() that ask me to SlowDown. But sometimes the errors response sent by AWS is invalid XML, in that it only has a header element.
My reproducer clearly isn't perfect, since it relies on a non-public image and s3 bucket, but I think it's still decent.  I'm using coiled and dask to create a large number of small files (~180K), but that's not actually the important part: the error all happens on the client side:
from dask.distributed import Client
import distributed
import dask.dataframe as dd
import coiled
import s3fs

cluster=coiled.Cluster(
    name="rate-limit",
    software="ian-r-rose/parquet",   # pretty standard dask + s3fs image
    n_workers=100,
    worker_vm_types=["t3.medium"],
    scheduler_vm_types=["t3.medium"],
)
client = distributed.Client(cluster)

df.to_parquet(
    "s3://dask-io/ian-many-partitions.parquet",
    engine="pyarrow-dataset",
    overwrite=True,
    write_metadata_file=False,
    compute=False,
    compression="snappy",
).compute()

# And here we run into troubles
fs = s3fs.S3FileSystem()
fs.rm("s3://dask-io/ian-many-partitions.parquet", recursive=True)

s3fs == 0.8.1
dask == 2022.4.1
distributed == 2022.4.1

Thanks for the reporducer, @ian-r-rose
So do you think it would be enough to add this XML decode error to the list of retriable ones, since those retries have an exponential backoff policy? Did you try my suggestion of batching coroutines?

So do you think it would be enough to add this XML decode error to the list of retriable ones, since those retries have an exponential backoff policy?

It might be, I can give this a shot. I suppose there is a chance for false positives there, but it's probably not too harmful.

Did you try my suggestion of batching coroutines?

Yes, I was actually about to submit a PR with my own hand-rolled proposal for batching them when I saw this issue. I'm not certain if _run_coros_in_chunks exactly fits my needs. I'll open it for discussion now.


Go rewrite

As we planned, I rewrote the CLI in Go.
It currently only has the init command and help and version commands finished. Other stuff will be added later, as I believe it would be better to merge first.
The stuff that I will add after the merge are these:

Extras
Code refactoring
More CLI components

I closed the PR because this branch will be renamed to main instead.


[Blazor] Proposal: Async wrapper components

Summary
Today it's not possible to await calls inside markup in blazor, however a very similar effect can be accomplished by wrapping the task in a component that awaits it in ParametersSetAsync and then outputs a renderfragment with the result. This approach also enabled rendering other markup while waiting for the task, such as a "loading" text.
I believe it will be very common for people to want to await things in markup, we certainly use it quite a bit, so it would useful to have in the framework.
Besides tasks there is also the async stream support in c#8 and being able to update values as they come in has been very useful for us as well.
I propose three components to enable this.
AsyncValue
Takes a task used to show a single value in an async fashion
AsyncList
Takes an IAsyncEnumerable<T> and appends values as they come in, useful for a log or a chat
AsyncUpdatedValue
Takes an IAsyncEnumerable<T> and updates the value in place, useful for a status label or progress bar
Open questions:
Is this even a good idea?
Does this already exist?
Use a renderfragment for loading or expose a property that can be bound to by the caller?
Better names?
Multiple components or a single one with configuration properties?
Better handling of prerendering?
Example
@page "/"

<AsyncValue Source="@SomeValue">
    <LoadingContent>

Loading

    </LoadingContent>
    <ChildContent>

@context

    </ChildContent>
</AsyncValue>

<AsyncValue Source="@SomeValue">

@context

</AsyncValue>

<AsyncUpdatedValue Source="@ASequenceOfValues">

@context%

</AsyncUpdatedValue>

    <AsyncList MaxCount="10" Source="@ASequenceOfValues">

@context

    </AsyncList>

<button @onclick="GetValue">GetValue</button>

@code {
    public Task<int> SomeValue { get; set; }
    public IAsyncEnumerable<int> ASequenceOfValues { get; set; }

    protected override void OnAfterRender(bool firstRender)
    {
        if (firstRender)
        {
            SomeValue = GenerateValue();
            ASequenceOfValues = GenerateValues();
            StateHasChanged();
        }
    }

    public async Task<int> GenerateValue()
    {
        await Task.Delay(1000);
        return 5;
    }

    public async Task GetValue()
    {
        SomeValue = GenerateValue();
        ASequenceOfValues = GenerateValues();
    }

    public async IAsyncEnumerable<int> GenerateValues()
    {
        for (int i = 0; i < 100; i++)
        {
            await Task.Delay(100);
            yield return i;
        }
    }
}

AsyncValue
@typeparam T

@if (_loading)
{
    @LoadingContent
}
else
{
    @ChildContent(_value)
}

@code {
    T _value;
    bool _loading  = true;

    [Parameter]
    public Task<T> Source { get; set; }
    [Parameter]
    public RenderFragment<T> ChildContent { get; set; }
    [Parameter]
    public RenderFragment LoadingContent { get; set; }

    protected override async Task OnParametersSetAsync()
    {
        _loading = true;
        if (Source != null)
        {
            _value = await Source;
            _loading = false;
        }
    }
}

AsyncList
@typeparam T

@if (_loading)
{
    @LoadingContent
}

else
{
    @foreach (var item in _values)
    {
        @ChildContent(item)
    }
}

@code {
    Queue<T> _values = new Queue<T>();
    bool _loading = true;

    [Parameter]
    public IAsyncEnumerable<T> Source { get; set; }
    [Parameter]
    public int MaxCount { get; set; } = int.MaxValue;

    [Parameter]
    public RenderFragment<T> ChildContent { get; set; }
    [Parameter]
    public RenderFragment LoadingContent { get; set; }

    protected override async Task OnParametersSetAsync()
    {
        _loading = true;
        _values.Clear();

        if (Source != null)
        {
            await foreach (var item in Source)
            {
                _values.Enqueue(item);
                _loading = false;
                if (_values.Count > MaxCount)
                    _values.Dequeue();
                StateHasChanged();
            }
        }
    }
}

AsyncUpdatedValue
@typeparam T

@if (_loading)
{
    @LoadingContent
}
else
{
    @ChildContent(_value)
}

@code {
    T _value;
    bool _loading = true;

    [Parameter]
    public IAsyncEnumerable<T> Source { get; set; }
    [Parameter]
    public RenderFragment<T> ChildContent { get; set; }
    [Parameter]
    public RenderFragment LoadingContent { get; set; }

    protected override async Task OnParametersSetAsync()
    {
        _loading = true;
        if (Source != null)
        {
            await foreach (var item in Source)
            {
                _value = item;
                _loading = false;
                StateHasChanged();
            }
        }
    }
}

@aL3891 Thanks for contacting us.
Blazor already supports this type of behavior without the need to wrap things inside a component. When you do async work inside a component we call StateHasChanged after the synchronous part has finished running and again later when the asynchronous part finishes running.
For example, if you override OnInitAsync and do something like this:
if(_data == null){

...Loading

}else{
   @* Show data *@
}

@code{
  List<string> _data;
  OnInitAsync(){

  // A render will be triggered here
  _ data = await GetData()
  // Another render will be triggered here
  }
}

Isn't this what you are looking for?

Not really, i mean that is also good, but what i wanted to enable was the equivalent of awaiting something inside the markup itself, witch is not supported as far as i understand.
Also, doing this with a lot of tasks in a view can get a little verbose in my opinion, especially if i don't want to specify the order in witch they should be awaited.
Having IAsyncEnumerable support is also something that would be nice for the same reason, and that also results in a lot of very similar code. StateHasChanged is also necessary here because the async work is not finished, the iteration of the async stream is still going on, but the view should still re-render. At least that seems to be the case from my testing, but perhaps i missed something.

I don't think we plan to introduce something like this on the framework itself. This is a good space for component libraries to fill-in if they feel there is a gap in the experience.
The framework goes to great lengths to avoid requiring the user to call state has changed, and for the most parts it succeeds in doing so.
With regards to the IAsyncEnumerable I feel that rather than wrapping the component inside another component it's easier to wrap the IAsyncEnumerable in some other IAsyncEnumerable that calls StateHasChanged through an extension method.
protected Task OnInitializedAsync()
{
    await foreach(var result in GetAsyncEnumerable().WithStateUpdate(this))
    {
        elements.Add(result);
    }
}

In your async IEnumerable you would call StateHasChanged inside each call to MoveNext() and that would be a potential solution.

Fair enough, I thought I'd share it since I've found this approach useful, perhaps it will be useful to someone else as well :)
Wrapping the iasyncenumerale in another one that calls StateUpdateChanged is a good approach, however the main value of this approach in my opinion is the ability to essentially use async calls directly in the markup without adding intermediate properties to the component.
One example for this is a list of users for example, where upon load, each item kicks of an async request for the user name or some other data. Of course this can be solved in alot of different ways but I've found this to be a very clean way to do it.

@aL3891 Rendering in Blazor is sync by design. It needs to be really fast and if it was async, it would be much harder to make it performant.

Thanks for the suggestion.
We are closing this issue as we don't plant to include this functionality in the framework.

I know that, but i think a solution like this would enable users to get the best of both worlds by buth having the actual rendering to not be async while still semantically "awaiting" async code inside the markup.


#map doesn't recognize a block delimeted with do .. end

#map{..} and #map do .. end are equivalent :
arr = [1, 2]
a = arr.map {|i| i.to_s}.join        # "12"
b = arr.map do |i| i.to_s end.join   # "12"

but replace a =  by puts  and only {..} compiles in the code below
puts arr.map {|i| i.to_s}.join       # "12"

puts arr.map do |i| i.to_s end.join
'Array(Int32)#map' is expected to be invoked with a block, but no block was given

Workaround: Assigning to a variable works. https://carc.in/#/r/xpt

method {...} and method do ... end are not really equivalent. When you have a method chain without parenteses, this happens:
puts arr.map {|i| i.to_s }.join        # read as   puts(arr.map() {|i| i.to_s }).join
puts arr.map do |i| i.to_s end.join    # read as   puts(arr.map()) {|i| i.to_s }.join

A do block refers to the most far method, in this case puts. But {} refers to the closest one, and works as intended.
You will need to do this:
puts(arr.map do |i| i.to_s end.join)

Or this:
arr.map do |i| i.to_s end.join.tap {|x| puts x }

@lbguilherme
It's consistent with Ruby (see example below).
The question is: does it make the most sense in this case?
# in Ruby
arr = [1, 2]
puts  (arr.map do |i| i.to_s end.join )  # 12
puts   arr.map do |i| i.to_s end.join    # NoMethodError: undefined method ‘join’ for nil:NilClass

I've updated the docs to explain the difference between do ... end and { ... }. We won't change this behaviour.


Incorrect sort of leading zeros within an alphanumeric value

The result of this array is incorrect:
["a55", "a010", "a102", "a02", "a1", "a255", "a33", "a43", "a87"].sort( naturalSort );
// result : ["a010", "a02", "a1", "a33", "a43", "a55", "a87", "a102", "a255"]

My feeble attempts at fixing this problem caused the failure of a few other unit tests.

LOL ok I think this is a quick & easy solution:
normChunk = function(s, l) {
    // normalize spaces; find floats not starting with '0', string or 0 if not defined (Clint Priest)
    return (naturalSort.ignoreLeadingZeros || !s.match(ore) || l == 1) && parseFloat(s) || s.replace(snre, ' ').replace(sre, '') || 0;
},

This just adds a naturalSort.ignoreLeadingZeros option; then for a unit test just add this:
naturalSort.ignoreLeadingZeros = true;
wrapTest(
  ["a55", "a010", "a102", "a02", "a1", "a255", "a33", "a43", "a87"],
  ["a1", "a02", "a010", "a33", "a43", "a55", "a87", "a102", "a255"],
  "issue - leading zero in alphanumeric");
naturalSort.ignoreLeadingZeros = false;

And btw, this is the same issue as #6... I just realized that it was a duplicate.


Issue 2395 ode tests

Summary
To address #2395 , the PR uses typed fixtures to test multiple ode solvers. It includes:

apparatus macro to generate ODE solver functor from ode solver functions.
A Base test fixture that automates comparing AD vs finite difference gradients.
All ODE test fixtures are derived from the base fixture. Each ODE test fixture defines an ODE system, tests, and required methods for base fixture to work, based on CRTP pattern.
An ODE tests fixture can be applied to all existing ODE solvers supplied testing types for typed-parameter test suite.

Example:
The Lorenz test fixture has parent class test_fixture_ode which defines basic test and finite diff tests. The Lorenz test fixture itself provides a handful of functions required by the parent class to do finite diff tests. The typed tests for this particular ODE can then performed on all existing ODE solvers:
bash-3.2$ ./runTests.py -j4 test/unit/math/rev/functor/lorenz_ode_typed_rev_test.cpp
------------------------------------------------------------
make -j4 test/unit/math/rev/functor/lorenz_ode_typed_rev_test
clang++ -std=c++1y -Wno-unknown-warning-option -Wno-tautological-compare -Wno-sign-compare -D_REENTRANT -Wno-ignored-attributes      -I lib/tbb_2019_U8/include   -O3  -I . -I lib/eigen_3.3.9 -I lib/boost_1.75.0 -I lib/sundials_5.6.1/include -I lib/benchmark_1.5.1/googletest/googletest/include -I lib/benchmark_1.5.1/googletest/googletest -I lib/benchmark_1.5.1/googletest/googletest/include -I lib/benchmark_1.5.1/googletest/googletest      -DBOOST_DISABLE_ASSERTS           -c -o test/unit/math/rev/functor/lorenz_ode_typed_rev_test.o test/unit/math/rev/functor/lorenz_ode_typed_rev_test.cpp
clang++ -std=c++1y -Wno-unknown-warning-option -Wno-tautological-compare -Wno-sign-compare -D_REENTRANT -Wno-ignored-attributes      -I lib/tbb_2019_U8/include   -O3  -I . -I lib/eigen_3.3.9 -I lib/boost_1.75.0 -I lib/sundials_5.6.1/include -I lib/benchmark_1.5.1/googletest/googletest/include -I lib/benchmark_1.5.1/googletest/googletest     -DBOOST_DISABLE_ASSERTS                -Wl,-L,"/Users/yiz/Work/temp/cmdstan/stan/lib/stan_math/lib/tbb" -Wl,-rpath,"/Users/yiz/Work/temp/cmdstan/stan/lib/stan_math/lib/tbb"      test/unit/math/rev/functor/lorenz_ode_typed_rev_test.o lib/benchmark_1.5.1/googletest/googletest/src/gtest_main.cc lib/benchmark_1.5.1/googletest/googletest/src/gtest-all.o lib/tbb/libtbb.dylib lib/tbb/libtbbmalloc.dylib lib/tbb/libtbbmalloc_proxy.dylib lib/sundials_5.6.1/lib/libsundials_nvecserial.a lib/sundials_5.6.1/lib/libsundials_cvodes.a lib/sundials_5.6.1/lib/libsundials_idas.a lib/sundials_5.6.1/lib/libsundials_kinsol.a        -Wl,-L,"/Users/yiz/Work/temp/cmdstan/stan/lib/stan_math/lib/tbb" -Wl,-rpath,"/Users/yiz/Work/temp/cmdstan/stan/lib/stan_math/lib/tbb"   -o test/unit/math/rev/functor/lorenz_ode_typed_rev_test
------------------------------------------------------------
test/unit/math/rev/functor/lorenz_ode_typed_rev_test --gtest_output="xml:test/unit/math/rev/functor/lorenz_ode_typed_rev_test.xml"
Running main() from lib/benchmark_1.5.1/googletest/googletest/src/gtest_main.cc
[==========] Running 4 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 1 test from StanOde/lorenz_test/0, where TypeParam = std::tuple<ode_rk45_functor, ode_rk45_functor, double, double, double>
[ RUN      ] StanOde/lorenz_test/0.param_and_data_finite_diff
[       OK ] StanOde/lorenz_test/0.param_and_data_finite_diff (21 ms)
[----------] 1 test from StanOde/lorenz_test/0 (21 ms total)

[----------] 1 test from StanOde/lorenz_test/1, where TypeParam = std::tuple<ode_ckrk_functor, ode_ckrk_functor, double, double, double>
[ RUN      ] StanOde/lorenz_test/1.param_and_data_finite_diff
[       OK ] StanOde/lorenz_test/1.param_and_data_finite_diff (26 ms)
[----------] 1 test from StanOde/lorenz_test/1 (26 ms total)

[----------] 1 test from StanOde/lorenz_test/2, where TypeParam = std::tuple<ode_bdf_functor, ode_bdf_functor, double, double, double>
[ RUN      ] StanOde/lorenz_test/2.param_and_data_finite_diff
[       OK ] StanOde/lorenz_test/2.param_and_data_finite_diff (176 ms)
[----------] 1 test from StanOde/lorenz_test/2 (217 ms total)

[----------] 1 test from StanOde/lorenz_test/3, where TypeParam = std::tuple<ode_adams_functor, ode_adams_functor, double, double, double>
[ RUN      ] StanOde/lorenz_test/3.param_and_data_finite_diff
[       OK ] StanOde/lorenz_test/3.param_and_data_finite_diff (116 ms)
[----------] 1 test from StanOde/lorenz_test/3 (116 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 4 test suites ran. (380 ms total)
[  PASSED  ] 4 tests.

Pro

simplify existing ODE tests by removing redundancy & avoiding boilerplate.
reduce burden to add new ODE tests and apply it to desired solvers (we need MORE test problems!)

Con

By supply a single list of solver types testing::Types<rk45, ckrk, bdf, adams> to a single test we're blurring the boundary that separate prim and rev tests. Currently rk45 & ckrk tests are in prim but bdf & adams are in rev(I'm missing the point of doing this BTW). We can follow the suit by testing a single fixture on two lists: testing::Types<rk45, ckrk> in prim and testing::Types<bdf, adams> in rev, but IMO it's less ideal.

Tests
Naming

test_fixture_ode_xxx.hpp for a fixture of an ODE model
xxx_ode_typed_prim/rev_testname_test for a typed test for xxx model. testname, if supplied, indicates specific test item.

I've ported most existing ODE tests to the above setup:

Cos 1 & 2 Arg tests
harmonic oscillator prim tests
harmonic oscillator finite diff tests
harmonic oscillator error tests
Lorenz finite diff tests

Side Effects
n/a
Release notes
Replace this text with a short note on what will change if this pull request is merged in which case this will be included in the release notes.
Checklist

[x] Math issue #2395

[x] Copyright holder:  Metrum Research Group
The copyright holder is typically you or your assignee, such as a university or company. By submitting this pull request, the copyright holder is agreeing to the license the submitted work under the following licenses:
- Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
- Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

[ ] the basic tests are passing

unit tests pass (to run, use: ./runTests.py test/unit)
header checks pass, (make test-headers)
dependencies checks pass, (make test-math-dependencies)
docs build, (make doxygen)
code passes the built in C++ standards checks (make cpplint)

[ ] the code is written in idiomatic C++ and changes are documented in the doxygen

[ ] the new changes are tested

@bbbales2 @rok-cesnovar what do you think of the setup? My aim is mainly to simplify adding new ODE model & tests but #2395 can be resolved along with it.

Nice!

we need MORE test problems

Agree.

we're blurring the boundary that separate prim and rev tests

I think that is fine and the way to go. To me it is  better if things are in one place. We do this with mix testing where we dont have separate rev testing for stuff in mix anymore, in OpenCL we rev tests also run tests for prim, so I would keep everything in rev.

@wds15 can you do review here?
Ideally this pull makes testing new ODE solvers easy and makes the tests in general more manageable.
The best test I can think of for this test framework is the adjoint ODE solver, and I saw tests being discussed yesterday there.

That doesn't mean this needs blocked for adjoint or they need to go in at the same time -- but if @wds15 can understand these tests and see how to extend them to adjoint solver and it is cleaner than what was there, then they it is probably a good test refactor.

Maybe the right move here is to get this in and then use it for the adjoint tests

@SteveBronder 's suggestion makes more sense to me, as in general I prefer a single-action PR addressing a single-task issue. But I'm ok either way. Perhaps what a reviewer can do is to examine the design to ensure:

it covers existing ode tests
it can be used to adjoint solver (but doesn't have to be applied in this PR).

then use it for the adjoint tests

Oh yeah sorry, I didn't mean to imply getting it to work for adjoint right now. What @yizhang-yiz posted is what I was hoping for.

I can definitely apply these tests on an extra branch to the adjoint ode things as a test drive. Some early comments on this:

very cool - we need this!
can we avoid the need for macros? That does not feel right to me... if it's really needed, then so be it. The least we should attempt is to reduce the macro sizes if possible
I see that we currently need to somewhat manually still vary the types being piped through the functions, but can't we automate that (in particular the combinatorics)?
Do you see a way to test against analytic gradients?
If no, we have analytic gradient tests - should we keep them? I think we should, but I would be curious about your suggestions.
What I haven't yet looked into is how error checks are tested which can be specific to the integrators (stuff like error messages users get when asking for a negative tolerance, etc.). Is there a unified interface for that as well?

@wds15

can we avoid the need for macros? That does not feel right to me... if it's really needed, then so be it. The least we should attempt is to reduce the macro sizes if possible

We can write out those functors manually but it's tedious. Being not a subject matter expert I'm not sure how to reduce macro sizes, It'd be great if someone can help out.

I see that we currently need to somewhat manually still vary the types being piped through the functions, but can't we automate that (in particular the combinatorics)?

It is automated through testing types. See, e.g., https://github.com/stan-dev/math/blob/6693a48a6acdcaebbae9b8c9142c51e925d54a28/test/unit/math/rev/functor/fho_ode_typed_rev_test.cpp#L8

Do you see a way to test against analytic gradients?

I think I've ported most of the tests including sho analytical ones. See
https://github.com/stan-dev/math/blob/6693a48a6acdcaebbae9b8c9142c51e925d54a28/test/unit/math/rev/functor/sho_ode_typed_rev_test.cpp#L118
It's also tested against various var/double combinations.

What I haven't yet looked into is how error checks are tested which can be specific to the integrators (stuff like error messages users get when asking for a negative tolerance, etc.). Is there a unified interface for that as well?

I'd like to think error tests are only needed once in certain ODE context. In our case it's the sho model that contains most such tests. I've ported them here:
https://github.com/stan-dev/math/blob/6693a48a6acdcaebbae9b8c9142c51e925d54a28/test/unit/math/rev/functor/sho_ode_typed_rev_error_test.cpp#L56
and here
https://github.com/stan-dev/math/blob/6693a48a6acdcaebbae9b8c9142c51e925d54a28/test/unit/math/rev/functor/sho_ode_typed_prim_test.cpp#L17
In general, I'm thinking several types of ODE tests:

tests that can be based on any ODE and should be passed by every integrator. Such tests include error message check, bad ODE check, parameter pack check, etc. Currently we mainly use sho model and cos 1 & 2 arg model for this type of tests.
tests that are based on a "nice-behaving" ODE model and should be passed by every integrator. Such tests include solution value & sensitivity check. This type of tests is the bottom line of verification for our ODE solvers.
integrator-specific tests used for verification of certain solver's implementation. An obvious use case is to use stiff ODE to test BDF. In this type of tests we are mostly interested in checking value/gradients against precomputed numerical value, analytical solution, or finite difference solution.

Type 1 tests are one-time effort, so I port them in sho & cos 1 & 2 Arg model fixtures and have every solver test on them. Type 2 & 3 are the ones that we'll be incrementally enhancing/tuning in the future, and since they require comparison finite diff & analytical solution comparison, I put these tests in parent fixture test_fixture_ode.hpp so we can have all future test models derive from it.

This is ready for review.

I am trying to adapt this to the adjoint branch.
Is t0 as var being tested?

All previously tested items have been ported. t0 as var test is here:
https://github.com/stan-dev/math/blob/4f9b46aaa6f525fe89eb7b21d63bcbd92bf02fe2/test/unit/math/rev/functor/test_fixture_ode_cos_scalar.hpp#L541
and is tested on all integrators:
https://github.com/stan-dev/math/blob/4f9b46aaa6f525fe89eb7b21d63bcbd92bf02fe2/test/unit/math/rev/functor/cos_ode_typed_test.cpp#L51

Some more notes:

cos_ode_type_test used an integer vector in two instances for time...those should be double vectors. We have never allowed ints for time (at least not intentionally).
I think I managed to integrate the adjoint solver. What was a pain is to add all those combinations of type lists. If we can reduce that burden with clever macro or tuple magic, then that would be cool.
The test suite seems to identify some issues in the adjoint solver or at least some expectations are not met. That's fine, but the sho_ode_typed_test test fails for me now for many more tests than just the adjoint tests, see the message below. That is odd to me - I would think that only the adjoint related tests would fail.
I saw a lot of set_zero_adjoints() and recover memory calls. I think those would look nicer when replaced by the newer nested mechanism.

Other than that it's a bit of a thing to get used to this logic, but it is what the google test library seems to suggest. So that's nice.
Error message:
... many more...
[  FAILED  ] StanOde/harmonic_oscillator_analytical_test/17.dv, where TypeParam = std::tuple<ode_bdf_functor, ode_bdf_functor, double, stan::math::var_value<double, void>, stan::math::var_value<double, void> >
[  FAILED  ] StanOde/harmonic_oscillator_analytical_test/17.vd, where TypeParam = std::tuple<ode_bdf_functor, ode_bdf_functor, double, stan::math::var_value<double, void>, stan::math::var_value<double, void> >
[  FAILED  ] StanOde/harmonic_oscillator_analytical_test/17.vv, where TypeParam = std::tuple<ode_bdf_functor, ode_bdf_functor, double, stan::math::var_value<double, void>, stan::math::var_value<double, void> >
[  FAILED  ] StanOde/harmonic_oscillator_analytical_test/18.dv, where TypeParam = std::tuple<ode_adams_functor, ode_adams_functor, double, stan::math::var_value<double, void>, stan::math::var_value<double, void> >
[  FAILED  ] StanOde/harmonic_oscillator_analytical_test/18.vd, where TypeParam = std::tuple<ode_adams_functor, ode_adams_functor, double, stan::math::var_value<double, void>, stan::math::var_value<double, void> >
[  FAILED  ] StanOde/harmonic_oscillator_analytical_test/18.vv, where TypeParam = std::tuple<ode_adams_functor, ode_adams_functor, double, stan::math::var_value<double, void>, stan::math::var_value<double, void> >
[  FAILED  ] StanOde/harmonic_oscillator_analytical_test/19.dv, where TypeParam = std::tuple<ode_adjoint_functor, ode_adjoint_functor, double, stan::math::var_value<double, void>, stan::math::var_value<double, void> >
[  FAILED  ] StanOde/harmonic_oscillator_analytical_test/19.vd, where TypeParam = std::tuple<ode_adjoint_functor, ode_adjoint_functor, double, stan::math::var_value<double, void>, stan::math::var_value<double, void> >
[  FAILED  ] StanOde/harmonic_oscillator_analytical_test/19.vv, where TypeParam = std::tuple<ode_adjoint_functor, ode_adjoint_functor, double, stan::math::var_value<double, void>, stan::math::var_value<double, void> >

82 FAILED TESTS

cos_ode_type_test used an integer vector in two instances for time...those should be double vectors. We have never allowed ints for time (at least not intentionally).

Good catch!

I think I managed to integrate the adjoint solver. What was a pain is to add all those combinations of type lists. If we can reduce that burden with clever macro or tuple magic, then that would be cool.

Can you point to me how it's utilized in adj solver?

but the sho_ode_typed_test test fails for me now for many more tests than just the adjoint tests, see the message below. That is odd to me - I would think that only the adjoint related tests would fail.

That's odd. Isn't the "full unit tests" passed on Jenkins?

I saw a lot of set_zero_adjoints() and recover memory calls. I think those would look nicer when replaced by the newer nested mechanism.

Yeah I just copied from existing tests. I'll update them.

@wds15
I've pushed "debug_ode_typed_test.cpp" to feature/adjoint-odes-test. It contains a half-ass solution to item 2(add macro to simplify tuple types) because I can't get var_value<double, void> to work with the macro (because of the comma) but var_value<double> works fine. It also shows that the test doesn't fail with var_value<double>. I'll stop investing on improving the macro because it's not a deal breaker for this issue, and the macro there does simplify things significantly.

@yizhang-yiz cool. Will look at it... and yeah don't over-engineer this type list thing. Maybe an easy copy pattern will already help actually.

@rok-cesnovar Was the Windows tests timed out? The test there seems irrelevant to this PR.

Indeed, its capped at 6 hours. That tests can take between 5h20min to 6h+, depending on what instance we get from Github. Annoying, but apart from splitting them again, nothing we can do. I am hoping the next time we need to touch these will be to switch to Rtools 4.0.
Restarted them.

Test passed but I'm just not sure about the plan at this point. Is it to wait till adjoint goes in? @bbbales2

No. This one should go in first. I am happy to review further. For now I am missing some overview doc on this or is there some?

For now I am missing some overview doc on this or is there some?

I'm not sure if it warrants  adesign doc as it's just an application of googletest. Is it ok to add more comments in the parent test fixture `test_fixture_ode.hpp"?

Not a design doc. Just some doc which is more than the code itself. Doxygen comments do. I had to trace all function calls myself and thought this would be helpful. Nothing big really.

@wds15 I've added doc & templates that generates test types. This is ready for review.

Is there a test for passing NaN to relative_tolerance and absolute_tolerance... which should be thowing that this is not ok? I only found the ones for passing -1 to that argument.

Same for the max_num_steps_errors (as we had these tests in the prim ode tests)... is that tested for? I was grepping for that string, but was not able to find anything.

Are the old interfaces being tested for at all? I am not sure if integrate_ode_adams for example is being tested for in any typed test. At least my grep failed for that. We probably do not need to test the full thing again, but at least something. Or I overlooked that?

@wds15

Is there a test for passing NaN to relative_tolerance and absolute_tolerance... which should be thowing that this is not ok? I only found the ones for passing -1 to that argument.
Same for the max_num_steps_errors (as we had these tests in the prim ode tests)... is that tested for? I was grepping for that string, but was not able to find anything.

See https://github.com/stan-dev/math/blob/f1971b9cbf674715b4c49b12168d5b4d7c7631f6/test/unit/math/rev/functor/test_fixture_ode_cos_scalar.hpp#L451-L513. I only put them in this model as we need not to test them in every ODE.

Are the old interfaces being tested for at all?

I only use them in the original "bad ode test" (test ode input/output dimension).
https://github.com/stan-dev/math/blob/f1971b9cbf674715b4c49b12168d5b4d7c7631f6/test/unit/math/rev/functor/sho_ode_typed_error_test.cpp#L54-L69
But we can add more if necessary.

LGTM.

Old to new tests map:

old test
new test
test fixture
tested interface

sho_value_test
test_value
harmonic_oscillator_test
variadic

sho_data_test
test_value
harmonic_oscillator_data_test
variadic

error_conditions & error_conditions_nan (prim, harm_osc)
test_bad & bad_param_and_data
harmonic_oscillator_data_test
variadic

error_conditions(rev, harm_osc)
test_bad
harmonic_oscillator_test
variadic

error_conditions_bad_ode
test_bad_ode
harmonic_oscillator_bad_ode_test
fixed

too_much_work
test_too_much_work
cos_arg_test
variadic

(w & w/o tol control)y0_errors
test_y0_error
cos_arg_test
variadic

(w & w/o tol control)t0_errors
test_t0_error
cos_arg_test
variadic

(w & w/o tol control)ts_errors
test_ts_error
cos_arg_test
variadic

(w & w/o tol control)one_arg_error
test_one_arg_error
cos_arg_test
variadic

(w & w/o tol control)two_arg_error
test_two_arg_error
cos_arg_test
variadic

(w & w/o tol control)rhs_wrong_size_errors
test_rhs_wrong_size_error
cos_arg_test
variadic

(w & w/o tol control)error_name
test_error_name
cos_arg_test
variadic

rtol_errors
test_rtol_error
cos_arg_test
variadic

atol_errors
test_atol_error
cos_arg_test
variadic

max_num_size_errors
test_max_num_error
cos_arg_test
variadic

init_t0
test_tol_int_t0
cos_arg_test
variadic

init_ts
test_tol_int_ts
cos_arg_test
variadic

(w & w/o tol contrtol)t0
test_t0
cos_arg_test
variadic

(w & w/o tol contrtol)ts
test_ts
cos_arg_test
variadic

(w & w/o tol contrtol)ts_repeat
test_ts_repeat
cos_arg_test
variadic

(w & w/o tol contrtol)scalar_arg
test_scalar_arg
cos_arg_test
variadic

(w & w/o tol contrtol)scalar_arg_multi_time
test_scalar_arg_multi_time
cos_arg_test
variadic

(w & w/o tol contrtol)std_vector_arg
test_std_vector_arg
cos_arg_test
variadic

(w & w/o tol contrtol)matrix_arg
test_matrix_arg
cos_arg_test
variadic

(w & w/o tol contrtol)
test_matrix_arg
cos_arg_test
variadic

(w & w/o tol contrtol)scalar_std_vector_args
test_scalar_std_vector_args
cos_arg_test
variadic

(w & w/o tol contrtol)std_vector_std_vector_args
test_std_vector_std_vector_args
cos_arg_test
variadic

(w & w/o tol contrtol)std_vector_vector_args
test_std_vector_vector_args
cos_arg_test
variadic

(w & w/o tol contrtol)std_vector_row_vector_args
test_std_vector_row_vector_args
cos_arg_test
variadic

(w & w/o tol contrtol)std_vector_matrix_args
test_std_vector_matrix_args
cos_arg_test
variadic

(w & w/o tol contrtol)arg_combos_test
test_arg_combos_test
cos_arg_test
variadic

finite diff
test_fd_vd
sho & lorenz test
variadic

test_fd_dv
sho & lorenz test
variadic

test_fd_dv
sho & lorenz test
variadic

time_steps_as_param_AD
test_ts_ad
forced_harm_osc_ts_test
variadic


GCE + RancherOS + Rancher > not working

Rancher versions:
rancher/server: 1.6.5
Operating system and kernel: (cat /etc/os-release, uname -r preferred)
rancheros-v1.0.3
Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)
GCE
Steps to Reproduce:
Upload RancherOS as custom Image
Create and Boot VM (1 CPU, 4 GB RAM) with RancherOS
Start Rancher on VM: $ docker run -d --restart=unless-stopped -p 80:8080 rancher/server
Adjust Firewall to let Port 80 pass through
Try to connect to: http:///
Results:
Rancher shows first contact, but is stuck at "Loading..." in title bar and doesn't continue.
If using Ubuntu 16.04LTS instead of RancherOS and start the Rancher Docker Container there, it works

Closing as duplicate of https://github.com/rancher/rancher/issues/8910, can you please try the workaround mentioned here: https://github.com/rancher/rancher/issues/8910#issuecomment-317780649


This commit adds rest api testing to the openshift-ci deployment

It also provides better function descriptions for the functions and adds output formatting.

@tonyxrmdavidson kindly rebase on top of main to include GHA fixes, thank you.

/retest

@tonyxrmdavidson kindly rebase on top of main to include GHA fixes, thank you.

/retest

/lgtm

/approve


Weak formulation of the Example 9 : DG Advection

Hi !
I searched in your various publications the weak formulation of the time-dependent advection equation (Ex 9), I found the conservative form, but I woulk like to know the non-conservative form. Where can I find that ?
I don't understand very well the difference between NonconservativeDGTraceIntegrator and ConservativeDGTraceIntegrator. What does TransposeIntegrator  do in NonconservativeDGTraceIntegrator ?
Best wishes.

With exact integration, the two formulations are equivalent. With approximate integration (because of variable coefficients, curvature, quadrature, etc.), they are not exactly equivalent.
The TransposeIntegrator corresponds to an integration-by-parts.
In the variational form, the derivative can either be on the test function or trial function. If the derivative is on the test function, the method is automatically conservative. If the derivative is on the trial function, the method is automatically constant-preserving. If the quadrature is exact, both of the above properties hold.

With exact integration and divergence-free velocity field, the two formulations are equivalent. With approximate integration (because of variable coefficients, curvature, quadrature, etc.), they are not exactly equivalent.
The TransposeIntegrator corresponds to an integration-by-parts.
In the variational form, the derivative can either be on the test function or trial function. If the derivative is on the test function, the method is automatically conservative. If the derivative is on the trial function, the method is automatically constant-preserving. If the quadrature is exact, both of the above properties hold.

Thank you for your answer ! But what is the difference between NonconservativeDGTraceIntegrator and ConservativeDGTraceIntegrator in the weak formulation ?

Integration by parts over each element will give rise to additional face terms, that are taken into account in each of the integrators. The specific forms represented by each of ConservativeDGTraceIntegrator and NonconservativeDGTraceIntegrator` are described in the comments/documentation:
https://github.com/mfem/mfem/blob/aef972353a0cd18d58d73fda666c8226fa727d17/fem/bilininteg.hpp#L2907-L2932
https://github.com/mfem/mfem/blob/aef972353a0cd18d58d73fda666c8226fa727d17/fem/bilininteg.hpp#L2992-L2993
https://github.com/mfem/mfem/blob/aef972353a0cd18d58d73fda666c8226fa727d17/fem/bilininteg.hpp#L2995-L3002

Yes I saw this documentation, and I don't understand, the only difference I see between the two forms is the sign of alpha

It looks like there may be a typo in the comment, since NonconservativeDGTraceIntegrator uses the transpose of DGTraceIntegrator, it should probably be
-alpha < rho_u (u.n) [v],{w} > + beta < rho_u |u.n| [v],[w] >

Hum, I don't see how to find a upwind flux with this formulation...
Normally it must remain at the end : v1(u.n) or v2(u.n)

DGTraceIntegrator will reduce exactly to the upwind flux in the case of specific values of alpha and beta. NonconservativeDGTraceIntegrator will have an additional face term that comes from the integration by parts.

Which integration by part?

Conservative form : dv/dt + div(uv) = 0
Non-Conservative form : dv/dt + u.gradv = 0

The weak form for conservative form is (I think) :
<dv/dt , w> -  <u .grad(w),v> + (0.5<(u.n)v, w> - 0.5<|u.n|v, w>) + (<(u.n) {v},[w]> + 0.5<|u.n| [v],[w]>) = 0
The weak form for non-conservative (Ex 9) would be :
<dv/dt , w> + <u.grad(v) ,w> + (0.5<(u.n)v, w> - 0.5<|u.n|v, w>) + (-<(u.n) [v],{w}> + 0.5<|u.n| [v],[w]>) = 0   ??

The conservative form for the steady problem is the following (from this paper)

where in the above, beta is the velocity field, u is the trial function, and v is the test function (g is the inflow condition, and f is the forcing function). There is also a reaction term gamma, you can take gamma = 0.
Under the condition  this is equivalent to using the upwind flux.
This is exactly the formulation that is obtained using ConservativeConvectionIntegrator and ConservativeDGTraceIntegrator with alpha = 1 (and beta = 1/2) — it should be possible to see this by matching terms with what is listed in the documentation.
The non-conservative form is obtained by integrating the first volume integral by parts over each element, which will change -(u*velocity, grad(v)) into (div(u*velocity), v) and also give rise to new face terms. Under the condition that the velocity is divergence free, the volume term will be (velocity*grad(u), v), which is given by the NonconservativeConvectionIntegrator. The appropriate face integrals will be given by NonconservativeDGTraceIntegrator with alpha = 1 and beta = 1/2. This can be seen using the following identity

Sorry but I don't understand very well the second part of the answer. Unfortunately, I don't have access to the cited paper...
With NonconservativeDGTraceIntegrator  we take alpha = -1 and not alpha = 1 no ??
I don't understant the terms in the last equation

Sorry but I don't understand very well the second part of the answer. Unfortunately, I don't have access to the cited paper...
With NonconservativeDGTraceIntegrator  we take alpha = -1 and not alpha = 1 no ??
I don't understand the terms in the last equation

Sorry but I don't understand very well the second part of the answer. Unfortunately, I don't have access to the cited paper...
With NonconservativeDGTraceIntegrator  we take alpha = -1 and not alpha = 1 no ??
I don't understant the terms in the last equation

With NonconservativeDGTraceIntegrator we take alpha = -1 and not alpha = 1 no ??

You can take alpha = 1. The reason the sign is flipped in ex9 is because that term is moved to the RHS for time integration, so you pick up a minus sign on the differential operator.