ruff: validate pep8-naming

What does this PR do?
enable pep8-naming and fix issues
Before submitting

[ ] Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
[x] Did you read the contributor guideline, Pull Request section?
[x] Did you make sure to update the docs?
[x] Did you write any new necessary tests?

PR review
Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
Did you have fun?
Make sure you had fun coding 🙃

the remaining issue may be fixed in https://github.com/charliermarsh/ruff/issues/2250

seeing strange errors on both GHA and pre-commit bot, but can't reproduce them locally, it also seems unrelated to the changes of this PR:
          error: subprocess-exited-with-error

          × Preparing metadata (pyproject.toml) did not run successfully.
          │ exit code: 1
          ╰─> [17 lines of output]
              Traceback (most recent call last):
                File "/pc/clone/FAyrB0nVQOi505WZ2l1j_w/py_env-python3/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
                  main()
                File "/pc/clone/FAyrB0nVQOi505WZ2l1j_w/py_env-python3/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
                  json_out['return_val'] = hook(**hook_input['kwargs'])
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/pc/clone/FAyrB0nVQOi505WZ2l1j_w/py_env-python3/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 152, in prepare_metadata_for_build_wheel
                  return hook(metadata_directory, config_settings)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/tmp/pip-build-env-10kfuqyr/overlay/lib/python3.11/site-packages/poetry/core/masonry/api.py", line 40, in prepare_metadata_for_build_wheel
                  poetry = Factory().create_poetry(Path(".").resolve(), with_groups=False)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/tmp/pip-build-env-10kfuqyr/overlay/lib/python3.11/site-packages/poetry/core/factory.py", line 57, in create_poetry
                  raise RuntimeError("The Poetry configuration is invalid:\n" + message)
              RuntimeError: The Poetry configuration is invalid:
                - [extras.pipfile_deprecated_finder.2] 'pip-shims<=0.3.4' does not match '^[a-zA-Z-_.0-9]+$'

              [end of output]

          note: This error originates from a subprocess, and is likely not a problem with pip.
        error: metadata-generation-failed

and check on master seems to be green, too... any idea @asottile?

try googling the error first, if you can't find it then I can help

seems like suddenly pre-commit does not like isort :(
ref: https://github.com/home-assistant/core/issues/86892

just for the record: https://github.com/pre-commit/pre-commit/issues/2730


In Identify, choosing two annotation filters doesn't work

eg https://www.inaturalist.org/observations/identify?taxon_id=56582&term_id=1&term_value_id=2&without_term_id=9 will show observations marked as adult, but many of those observations will already have sex annotations.
Expected behavior: observations with Adult life stage annotations should be shown, but none of them should have any Sex annotation.

Mostly for my own reference, https://github.com/inaturalist/iNaturalistAPI/commit/6dd6e95cbb01aa073d5a4e50cc2c7634a31c23fe gets this working for term_id=x&without_term_id=y, but for some reason term_id=x&term_value_id=ywithout_term_id=z still doesn't work.

I think this is working for combinations of term_id, term_value_id, and without_term_id. If I set with Life Stage: Adult, then I can select without Alive or Dead and the count drops and I see obs w/o Alive or Dead annotations.
One bit that's confusing is the without_term_value_id. The API documentation says Must be combined with the term_id parameter, and the code implements it this way - without_term_value_id is only in the context of term_id. Meaning you can search for obs with a Life Stage but not Life Stage: Adult. But, you cannot search for obs with a Life Stage, without a Alive or Dead: Alive annotation as the Identify UI implies, as Alive isn't a value of Life Stage.
So I think we should either update the Identify UI or change the behavior of the API to either add some new param(s), or make it so without_term_value_id is in the context of term_id unless there's a without_term_value_id (i.e. without_term_id defaults to term_id but can be overridden)


Tabs Component Dropdown does not close when clicking outside of the dropdown on the XS breakpoint

Detailed description
On the XS breakpoint, the Tabs component does not close when clicking outside of the dropdown when it is opened. This happens on both Carbon Components and Carbon Components React.

What did you expect to happen? What happened instead? What would you like to see changed?

Upon clicking outside of the dropdown the Tabs component should close, similar to how it closes on the Dropdown component below:

What browser are you working in?

Chrome Version 72.0.3626.119 (Official Build) (64-bit). Though this happens in Firefox and I'm assuming all major browsers.

What version of the Carbon Design System are you using?

Carbon Components 9.77.1 & Carbon Components React 6.97.0

What offering/product do you work on? Any pressing ship or release dates we should be aware of?

IBM Cloud
Steps to reproduce the issue

Visit either the Carbon Components or Carbon Components React repository.
Select the Tabs Component
Reduce the size of the viewport to less than 768px.
Open the Tabs component by selecting the visible Dropdown
Click outside of the Tabs component elsewhere on the screen.
The open Tabs Dropdown should not close.

hi @electrostaticfleece thank you for reporting this! we will probably create a separate issue to track this in the React repo as well


no volumes in fig.yml

Would it be better to put arbitrary volumes under fig.yml?

@so0k whhat do you mean by arbitrary volumes?

sorry for late reply, maybe arbitrary is the wrong word (non-native speaker here).
I meant - add a 'default' volume mapping for the data store in the fig file (and rename fig to docker-compose.yml)

@so0k docker compose feature is not released yet right? I don't see docker up option in 1.4.1. Until the compose feature is available it does not make sense to rename the fig.yml file. Once the feature is out in the open the required changes can be done with updates to the readme as well.
Regarding the volume statement in fig.yml, since it is a sample configuration it does not make sense to add a volume instruction in there. User are expected to build upon this sample as per their requirements.

you are right


Building the binaries failing.

Disclaimer: I'm relatively new to Go so this may be caused by a lack of understanding.
When I run make binaries I get the following error:
🐳 bin/swarmd
# _/home/liam/Desktop/swarmkit/cmd/swarmd
cmd/swarmd/main.go:138: cannot use client (type *"github.com/docker/engine-api/client".Client) as type "github.com/docker/swarmkit/vendor/github.com/docker/engine-api/client".APIClient in argument to "github.com/docker/swarmkit/agent/exec/container".NewExecutor:
	*"github.com/docker/engine-api/client".Client does not implement "github.com/docker/swarmkit/vendor/github.com/docker/engine-api/client".APIClient (wrong type for ContainerAttach method)
		have ContainerAttach("golang.org/x/net/context".Context, string, "github.com/docker/engine-api/types".ContainerAttachOptions) ("github.com/docker/engine-api/types".HijackedResponse, error)
		want ContainerAttach("github.com/docker/swarmkit/vendor/golang.org/x/net/context".Context, string, "github.com/docker/swarmkit/vendor/github.com/docker/engine-api/types".ContainerAttachOptions) ("github.com/docker/swarmkit/vendor/github.com/docker/engine-api/types".HijackedResponse, error)
make: *** [bin/swarmd] Error 2

Go version is go version go1.6.2 linux/amd64

/cc @stevvooe

I can't reproduce this on a clean Debian VM, so it's likely a problem with the environment.

@liamawhite can you post echo $GOPATH and echo $GOROOT? NB the latter may not output anything.

liam@liam-ThinkPad-W550s:~/Desktop/swarmkit$ echo $GOPATH
/home/liam/Desktop/swarmkit
liam@liam-ThinkPad-W550s:~/Desktop/swarmkit$ echo $GOROOT

Can you also give me the output of go env as well? That'll give more information.

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/liam/Desktop/swarmkit"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT="1"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

Ok, so the problem is likely related to you not having a standard project structure for go.
If you go to https://golang.org/doc/code.html#Workspaces you'll see that there is a standard go workspace layout, and you're going to have to create some of those directories manually.
Here is an example of the tree of my home directory.  You also need to set GOPATH=$HOME/gocode in your .profile. Reload your shell after all of this and try again.

@liamawhite Hey, what @dperny said earlier.
We use vendoring so, once you have a proper setup, you won't need to run any go get - just make binaries.

Here is a good place to start on building a Go project:
https://github.com/docker/distribution/blob/1b9ab303a477ded9bdd3fc97e9119fa8f9e58fca/docs/recipes/building.md
We should probably add the same thing to the root of the project. Somehow, this got sucked into docs in distribution.

@dperny changing to that directory structure sorted everything, thanks.


Foster epic: Create FosterAccount

Blocked by https://github.com/rubyforgood/pet-rescue/issues/448
We differentiate user types in this app by associating User with either AdopterAccount or StaffAccount. We need to add a third user type FosterAccount.
Acceptance criteria

Create migration to add table
Create model, associations: belongs_to user, has_one adopter_foster_profile
Update User model has_one FosterAccount and update AdopterFosterProfile belongs_to FosterAccount
Add model tests
(other assoications will be added later once those models are in place)

I have started this but I will have follow-up questions.

@ErinClaudio for sure
Note you'll want to wait for this issue https://github.com/rubyforgood/pet-rescue/issues/448 to be merged so we can add the association to the adopter_foster_profile on FosterAccount model.

Draft PR in progress: https://github.com/rubyforgood/pet-rescue/pull/452

https://gist.github.com/ErinClaudio/372b5d6e6bb03b3d604ab934a8c563ac

https://gist.github.com/ErinClaudio/372b5d6e6bb03b3d604ab934a8c563ac


[Conformance checker] Capture potential matches for missing witnesses.

Rework the data structures we use in the conformance checker to talk
about missing witnesses, so they can capture the set of potential
matches. This will allow us to delay more diagnostics to later,
more uniformly.

@swift-ci smoke test


TypeError Bad Argument

I get an error saying:
TypeError in plugin 'gulp-uncss'
Message:
    Bad argument

With a fairly typical setup:
var gulp = require('gulp');
var plugins = require('gulp-load-plugins')({
	pattern: ['gulp-*'],
	lazy: true
});
    gulp.src(['app/css/app.scss'])
    .pipe(plugins.plumber())
    .pipe(plugins.sass({
		includePaths: [
		'./app/vendor',
		'./bower_components/bootstrap/scss',
		'./bower_components/open-sans-fontface',
		'./app/vendor/compass-sass-mixins/lib',
		'./app/vendor/compass-sass-mixins/lib/css3',
		'./node_modules/include-media/dist',
		'./node_modules/modularscale-sass/stylesheets'
		],
	}))
    .pipe(plugins.postcss([autoprefixer]))
	.pipe(plugins.concat('app.css'))
    .pipe(gulp.dest('public/css'))
	.pipe(plugins.uncss({
		html: ['public/**/*.html']
	}))
    .pipe(gulp.dest('public/css'))
    .pipe(plugins.rename({suffix: '.min'}))
    .pipe(plugins.cssnano())
    .pipe(gulp.dest('public/css'))

However, running through the UnCSS CLI it progresses as expected. This is with<EMAIL_ADDRESS>and<EMAIL_ADDRESS>The html-option seems correct enough, what could be causing the error?

Where is the origin of this error? Could you please provide a full stack trace?

I'm not quite sure how to obtain a full stack trace, would the equivalent from gulp-plumber work? Here is the output from its errorHandler (slightly formatted for readability):
{ [TypeError: Bad argument]
  name: 'TypeError',
  message: 'Bad argument',
  stack: 'TypeError: Bad argument
at TypeError (native)
at ChildProcess.spawn (internal/child_process.js:274:26)
at Object.exports.spawn (child_process.js:362:9)
at node_modules\\phridge\\lib\\spawn.js:103:38
at init (node_modules\\when\\lib\\makePromise.js:39:5)
at new Promise (node_modules\\when\\lib\\makePromise.js:27:53)
at Function.promise (node_modules\\when\\when.js:97:10)
at startPhantom (node_modules\\phridge\\lib\\spawn.js:73:25)
at tryCatchReject (node_modules\\when\\lib\\makePromise.js:845:30)
at runContinuation1 (node_modules\\when\\lib\\makePromise.js:804:4)
at Fulfilled.when (node_modules\\when\\lib\\makePromise.js:592:4)\n
   at Pending.run (node_modules\\when\\lib\\makePromise.js:483:13)
at Scheduler._drain (node_modules\\when\\lib\\Scheduler.js:62:19)
at Scheduler.drain (node_modules\\when\\lib\\Scheduler.js:27:9)
at nextTickCallbackWith0Args (node.js:452:9)
at process._tickCallback (node.js:381:13)',
  showStack: false,
  showProperties: true,
  plugin: 'gulp-uncss' }

Okay. Looks like this is a duplicate of this bug; https://github.com/giakki/uncss/issues/75


Multiple I/O on disk for creating LSTM training dataset

https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/tesseract-ocr/OJI3bsi_JMY/NHlt8uEfEAAJ

I'm new to tesseract and wondered why the lstm dataset creation for the training process has to write the file again and again in TrainLineRecognizer. I've seen 200MB/s IO on the disk while creating the training data set. As far I can see for the training case it would be sufficient to just load it once and write it at the end. The same applies to the box and tif file - but these are only read and not written...

LSTM training takes a lot of time and has no well defined duration. Therefore saving intermediate results (with a configurable frequency) is a good thing. It allows tests based on those intermediate results, and those results can also be used to resume the learning process if it was interrupted for whatever reason.

Thanks @stweil .


Feature request: Add directive estimation support for "initial complexity"

Imagine we have an external service that we're calling out to in order to populate the foo array field.
I can request any number of elements for that, but the issue is that asking for any number drastically increases complexity, whereas the relative complexity between requesting 1 and 2 entries is much lower.
It would be good to have an "initial" field on the complexity directive, e.g.
@complexity(value: 1, multipliers: :["limit"], initial: 10)
in this example, requesting with a limit of 1 would be a complexity of 11, and requesting 2 would be a complexity if 12, etc.
If we like this suggestion I'm happy to take a crack at implementation

This seems like a valuable addition to add. I'm happy to review / merge a PR if you are still interested in adding this.


slots filling with using NLU

Rasa Core version: 0.9.7
Python version:  3.6
Operating system (windows, osx, ...): windows 10
Issue:
I have questions about slots filling without using FormAction, which I want let nlu to parse it. In this case, if I want to fill the slots as someone's name. Obviously, there are thousands of names.

Do I need to list all of the possible names in my intent?
Do I need to list all of the stories with all names?
If I use FormAction, does that mean I don't need to specify all of the possible names?
below in my story 2 and intent My name is [Daniel](user_name), after I typed "My name is Daniel", the bot didn't reply me and just stopped.Here is the debug messages:

my name is Daniel
2018-07-05 19:51:43 DEBUG    rasa_core.tracker_store  - Recreating tracker for id 'default'
2018-07-05 19:51:43 DEBUG    rasa_core.processor  - Received user message 'my name is Daniel' with intent '{'name': 'usrname', 'confidence': 0.6202647573096165}' and entities '[]'
2018-07-05 19:51:43 DEBUG    rasa_core.processor  - Logged UserUtterance - tracker now has 6 events
2018-07-05 19:51:43 DEBUG    rasa_core.processor  - Current slot values:
        user_name: None
2018-07-05 19:51:43 DEBUG    rasa_core.policies.memoization  - Current tracker state [{'intent_greet': 1.0, 'prev_action_listen': 1.0}, {'prev_utter_what_name': 1.0, 'intent_greet': 1.0}, {'intent_usrname': 1.0, 'prev_action_listen': 1.0}]
2018-07-05 19:51:43 DEBUG    rasa_core.policies.memoization  - There is no memorised next action
2018-07-05 19:51:43 DEBUG    rasa_core.policies.ensemble  - Predicted next action using policy_1_KerasPolicy
2018-07-05 19:51:43 DEBUG    rasa_core.policies.ensemble  - Predicted next action 'action_listen' with prob 0.85.
2018-07-05 19:51:43 DEBUG    rasa_core.processor  - Action 'action_listen' ended with events '[]'
2018-07-05 19:51:43 DEBUG    rasa_core.processor  - Current topic: None

Content of domain file (if used & relevant):
domain:
intents:
  - greet
  - inform
  - request
  - usrname

actions:
- utter_what_name
- utter_ask_user_name

entities:
  - user_name

slots:
  user_name:
    type: text

templates:
  utter_what_name:
  - text: "What is your name?"
  utter_ask_user_name:
  - text: "Hello, {user_name}"

part of intent of usrname:
## intent:usrname
- [Mike](user_name)
- [John](user_name)
- [Mary](user_name)
- My name is [Daniel](user_name)

stories:
## writing path 1<!-- name of the story - just for debugging -->
* greet
  - utter_what_name
* usrname{"user_name": "Mike"}
  - utter_ask_user_name
## writing path 2
* greet
  - utter_what_name
* usrname{"user_name": "Daniel"}
  - utter_ask_user_name

from the debug log, NLU is parsing "my name is Daniel" as:
Received user message 'my name is Daniel' with intent '{'name': 'usrname', 'confidence': 0.6202647573096165}' and entities '[]'````

the problem is that Daniel is not picked up as a name entity, so also won't be set as a slot. Do you have `ner_crf` in your NLU pipeline? What does your NLU config file look like?

@amn41 Hi,
I dont have ner_crf though. How can I get a complete NLU config file?
My NLU config file:
language: "en"

pipeline: "spacy_sklearn"

I found it on https://nlu.rasa.com/pipeline.html
Now my NLU config:
language: "en"

pipeline:
- name: "nlp_spacy"
- name: "tokenizer_spacy"
- name: "intent_entity_featurizer_regex"
- name: "intent_featurizer_spacy"
- name: "ner_crf"
- name: "ner_synonyms"
- name: "intent_classifier_sklearn"

But it still doesn't work....
my name is Daniel
2018-07-06 02:47:41 DEBUG    rasa_core.tracker_store  - Recreating tracker for id 'default'
2018-07-06 02:47:41 WARNING  py.warnings  - C:\Users\e8180\Anaconda3\lib\site-packages\sklearn\preprocessing\label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.
  if diff:

2018-07-06 02:47:41 DEBUG    rasa_core.processor  - Received user message 'my name is Daniel' with intent '{'name': 'usrname', 'confidence': 0.6037633475397749}' and entities '[]'
2018-07-06 02:47:41 DEBUG    rasa_core.processor  - Logged UserUtterance - tracker now has 6 events
2018-07-06 02:47:41 DEBUG    rasa_core.processor  - Current slot values:
        user_name: None
2018-07-06 02:47:41 DEBUG    rasa_core.policies.memoization  - Current tracker state [{'prev_action_listen': 1.0, 'intent_greet': 1.0}, {'intent_greet': 1.0, 'prev_utter_what_name': 1.0}, {'prev_action_listen': 1.0, 'intent_usrname': 1.0}]
2018-07-06 02:47:41 DEBUG    rasa_core.policies.memoization  - There is no memorised next action
2018-07-06 02:47:41 DEBUG    rasa_core.policies.ensemble  - Predicted next action using policy_1_KerasPolicy
2018-07-06 02:47:41 DEBUG    rasa_core.policies.ensemble  - Predicted next action 'action_listen' with prob 0.85.
2018-07-06 02:47:41 DEBUG    rasa_core.processor  - Action 'action_listen' ended with events '[]'
2018-07-06 02:47:41 DEBUG    rasa_core.processor  - Current topic: None

yes, your original pipeline is also fine but it looks like you don't have enough training data for the name entity, because 'Daniel' is not recognized as a name

Thank you, after I added more training data, it works now.

@akelad Is the custom action you mentioned referred to action.py(FormActions)?
more specifically, how can I do this? any tutorial?

Have you seen this part of the documentation on custom actions? https://core.rasa.com/domains.html#defining-custom-actions

in your intent, the training data you have submitted. I would suggest that the names you provided in the file be all unique. Do not repeat the names!
Next change your pipeline slightly to take in all types of names
I had the same issue with names so this is what I did and it worked. I tried weird characters like 'afgrgwersag' and it would still take in as a name and i believe that's a good thing
this is the config i used
language: "en"

pipeline:
  - name: "nlp_spacy"
    model: "en_core_web_md"
    case_sensitive: false
  - name: "tokenizer_whitespace"
  - name: "tokenizer_spacy"
  - name: "intent_featurizer_spacy"
  - name: "intent_classifier_sklearn"
  - name: "ner_crf"
    BILOU_flag: true
    features: [
                ["low", "title", "upper"],
                ["bias", "low", "prefix5", "prefix2", "suffix5", "suffix3", "suffix2", "upper", "title", "digit", "pattern", "pos", "pos2"],
                ["low", "title", "upper"]
              ]
  - name: "ner_spacy"
  - name: "ner_crf"
    features: [
                ["low", "upper"],
                ["bias", "low", "prefix5", "prefix2", "suffix5", "suffix3", "suffix2", "upper", "digit", "pattern", "pos", "pos2"],
                ["low", "upper"]
              ]
  - name: "ner_synonyms"
  - name: "intent_entity_featurizer_regex"
  - name: "intent_featurizer_count_vectors"
    token_pattern: r'(?u)\b\w\w+\b'
    max_ngram: 2
  - name: "intent_classifier_tensorflow_embedding"
    intent_tokenization_flag: true
    intent_split_symbol: "+"

with this config i made the nlu to ignore case sensitivity and as u can see i have used 2 ner_crf. i did this so that if one them fails to catch the entity the other one can still be there.
i tried this out and it works for me. i have tried many cases with different/mixed cases and it recognizes it as PERSON entity but with low confidence. It may not be the best solution as I am just a beginner and a student, so rasa and everything is new to me. but try this out and let me know if it works.
I am getting a warning when I run and enter a text "rasa_nlu.featurizers.count_vectors_featurizer  - There is no trained CountVectorizer: component is either not trained or didn't receive enough training data" but it doesnt affect my bot. If you guys could help me on this I would appreciate it!
my files.zip


re-generate with svd2rust v0.14.0

bors r+

I will release this as v0.14.0-alpha.1. This is a pre-release. Depend on it at your own risk. This version will be yanked when the proper release is made and there may be breaking changes between this pre-release and the real release.
cc @rnestler @simonvandel


[SPARK-38521][SQL][WIP] Insert overwrite partitioned datasource table with dynamic mode

Why are the changes needed?
We have supported partition overwrite with static mode by default while partition manged by catalog for a long time. It looks weird to me as static mode behaves catastrophically and deletes all data in table. Maybe setting partition overwrite with dynamic mode by default while partition manged by catalog, and user can set partitionOverwriteMode in table properties to support old behaviors. And also, this change can also bring a better experience to users when migrating from the original hive table to the datasource table, rather than the old data being deleted when the user does not know the meaning of partitionOverwriteMode.
Does this PR introduce any user-facing change?
Yes. We will no longer delete old partition data by default when user insert overwrite partitioned datasource tables.
How was this patch tested?
Origin UT

cc @cloud-fan @HyukjinKwon @dongjoon-hyun

@cloud-fan @HyukjinKwon I wan't to bring a discuss about this, as setting partitionOvermode with static by default is realy not a good way. It may result in the loss of unrecoverable data when user use datasource table without knowing this config, such as migrating from the original hive table to the datasource table.

Can one of the admins verify this patch?


Womanium Final Assignment Víctor_Hernán_Torres_Brauer

Homework for womanium and classiq certification
PR Description

Some notes

[ ] Please make sure that you placed the files in an appropriate folder

[ ] And that the files have indicative names.

[ ] Please note that Classiq runs automatic code linting, which may minorly alter some files.

[ ] If you're familiar with pre-commit, you may run pre-commit install, and then at each commit, your files will be altered in a similar way

Good job!
Score: 8/10.


#635: Simplify TkProduces and TkConsumes by using TkFork instead of RsFork

Fix for #635
The goal of this PR is to simplify TkProduces and TkConsumes by using TkFork instead of RsFork. For the need of this task, I added one new constructor in the class FkTypes in order to allow to provide a Take instead of a static Response. This way we can still provide a static response if we want but now we can provide a take for dynamic responses.

@essobedo Thanks for the pull request, let me find a reviewer..

@mkordas it's yours, please review

@essobedo I'll check it today

@essobedo just one comment

@mkordas PR description updated

@essobedo looks good!

@rultor merge

@rultor merge

@mkordas Thanks for your request. @yegor256 Please confirm this.

@yegor256 review needed here as well

@rultor try to merge

@rultor try to merge

@yegor256 OK, I'll try to merge now. You can check the progress of the merge here

@rultor try to merge

@yegor256 Done! FYI, the full log is here (took me 17min)

@elenavolokhova review this ticket please, for compliance with our quality rules

@rultor please deploy

@rultor please deploy

@davvd OK, I'll try to deploy now. You can check the progress here

@rultor please deploy

@davvd Done! FYI, the full log is here (took me 20min)

@mkordas
According to our quality rules:

The code reviewer found at least three problems in the code.

There were no issues found during review.
Please confirm that you will try to find at least three problems next time.

@elenavolokhova I confirm

@davvd Quality is acceptable here.

@davvd Quality is acceptable here.

@elenavolokhova thanks for the QA review, we'll work better next time

@mkordas added 10 mins to @elenavolokhova, for QA review, payment code is 82175912. I added 23 mins to your account, many thanks for working with the project (56ffb8fd33253e121b0001c9)! The completion time here was 74 hours and 14 mins.. extra minutes for review comments (c=8). +23 to your rating, your total score is +9169


Update php.ts

Add magic constants
Add self keywords
Extend list of keywords
Group variable types

Also need help with adding class constants

Many of these don't seem to actually be emitted by the parser. Defining a highlighting tag for a name that won't appear in the parse tree doesn't actually work.

@marijnh you mean many in this PR or at all?

In this PR.

@marijnh

Defining a highlighting tag for a name

I'm add only self keywords and magic constants in other cases I'm only add missed keywords. please, specify exactly what's wrong?

These changes don't work, for things like __CLASS__, as I'm sure you can confirm. The names in this context refer to syntax tree node names, so you can't just define new constants here.

@marijnh ok, constants are removed. anything else?

Attached patch adds the ones that were missed. The others don't exist in the parser output.

@marijnh why declare enddeclare not in control keywords? why eval isset empty die exit __halt_compiler don't exist in the parser output?

@marijnh why declare enddeclare not in control keywords?

Oh, I see, they were already there in definition keywords, which seems a reasonable place for them as well. My patch actually adds them to control keywords.

why eval isset empty die exit __halt_compiler don't exist in the parser output?

Mostly because they didn't appear in the tree-sitter grammar our grammar is based on, which I guess is because you still get a reasonable parse even if you don't handle these specially.


Possiblly a bug in the README sample

Information

Qiskit Nature version 0.1.3:
Python version 3.7:
Operating system Linux:

What is the current behavior?
ansatz.compose(init_state, front=True)

This line probably does nothing because by default compose is not in place as of qiskit version 0.17.4
Suggested solutions
Change the line to
ansatz.compose(init_state, front=True, inplace=True)

That line you indicate indeed has no consequence on the subsequent code. Either the returned circuit needs to be used in VQE or as you suggest use inplace=True to alter the ansatz cct itself.
There is a unit test that checks the outcome of the readme code test/test_readme_sample.py that needs to stay in sync with whatever change is done too.

Hi, I am new to contributions but I would like to work on this issue. Can I be assigned?

Done - thank you for offering to look at this.


chore(main): release volcano-data-deconvolution 1.0.5

:robot: I have created a release beep boop
1.0.5 (2024-06-03)
Continuous Integration

github: update citation.cff and validate after (8def5c5)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/engeir/volcano-data-deconvolution/releases/tag/volcano-data-deconvolution-v1.0.5 :sunflower:


Ability to revert a dependency to pending

This changes enables a no longer pending dependency to move back to
pending if all hard edges are removed.

Will try an alternate implementation that uses counter instead of EdgeState reference

@melix Thanks for the suggestion, this version no longer shows any performance impact. All failures are caused by network issues with JCenter.


Saving a DataArray of datetime objects as zarr is not a lazy operation despite compute=False

What happened?
Trying to save a lazy xr.DataArray of datetime objects as a zarr forces a dask.compute operation and retrieves the data to the local notebook. This is generally not a problem for indices of datetime objects as that is already locally store and generally small in size.
However, if the whole underlying array is a datetime object, that can be a serious problem. In my case it simply crashed the scheduler upon attempting to retrieve the data persisted on workers.
I managed to isolate the problem on this call stack. The issue is in the encode_cf_datetime function
What did you expect to happen?
Storing the data in zarr format to be performed directly by dask workers bypassing the scheduler/Client if compute=True, and complete lazy operation if compute=False
Minimal Complete Verifiable Example
import numpy as np
import xarray as xr
import dask.array as da
test = xr.DataArray(
    data = da.full((20000, 20000), np.datetime64('2005-02-25T03:30', 'ns')),
    coords = {'x': range(20000), 'y': range(20000)}
).to_dataset(name='test')

print(test.test.dtype)
# dtype('<M8[ns]')

test.to_zarr('test.zarr', compute=False)
# this will take a while and trigger the computation of the array. No data will be actually saved though

MVCE confirmation

[X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
[x] Complete example — the example is self-contained, including all data and the text of any traceback.
[x] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
[X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output
File /env/lib/python3.8/site-packages/xarray/core/dataset.py:2036, in Dataset.to_zarr(self, store, chunk_store, mode, synchronizer, group, encoding, compute, consolidated, append_dim, region, safe_chunks, storage_options)
   2033 if encoding is None:
   2034     encoding = {}
-> 2036 return to_zarr(
   2037     self,
   2038     store=store,
   2039     chunk_store=chunk_store,
   2040     storage_options=storage_options,
   2041     mode=mode,
   2042     synchronizer=synchronizer,
   2043     group=group,
   2044     encoding=encoding,
   2045     compute=compute,
   2046     consolidated=consolidated,
   2047     append_dim=append_dim,
   2048     region=region,
   2049     safe_chunks=safe_chunks,
   2050 )

File /env/lib/python3.8/site-packages/xarray/backends/api.py:1431, in to_zarr(dataset, store, chunk_store, mode, synchronizer, group, encoding, compute, consolidated, append_dim, region, safe_chunks, storage_options)
   1429 writer = ArrayWriter()
   1430 # TODO: figure out how to properly handle unlimited_dims
-> 1431 dump_to_store(dataset, zstore, writer, encoding=encoding)
   1432 writes = writer.sync(compute=compute)
   1434 if compute:

File /env/lib/python3.8/site-packages/xarray/backends/api.py:1119, in dump_to_store(dataset, store, writer, encoder, encoding, unlimited_dims)
   1116 if encoder:
   1117     variables, attrs = encoder(variables, attrs)
-> 1119 store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)

File /env/lib/python3.8/site-packages/xarray/backends/zarr.py:500, in ZarrStore.store(self, variables, attributes, check_encoding_set, writer, unlimited_dims)
    498 new_variables = set(variables) - existing_variable_names
    499 variables_without_encoding = {vn: variables[vn] for vn in new_variables}
--> 500 variables_encoded, attributes = self.encode(
    501     variables_without_encoding, attributes
    502 )
    504 if existing_variable_names:
    505     # Decode variables directly, without going via xarray.Dataset to
    506     # avoid needing to load index variables into memory.
    507     # TODO: consider making loading indexes lazy again?
    508     existing_vars, _, _ = conventions.decode_cf_variables(
    509         self.get_variables(), self.get_attrs()
    510     )

File /env/lib/python3.8/site-packages/xarray/backends/common.py:200, in AbstractWritableDataStore.encode(self, variables, attributes)
    183 def encode(self, variables, attributes):
    184     """
    185     Encode the variables and attributes in this store
    186
   (...)
    198
    199     """
--> 200     variables = {k: self.encode_variable(v) for k, v in variables.items()}
    201     attributes = {k: self.encode_attribute(v) for k, v in attributes.items()}
    202     return variables, attributes

File /env/lib/python3.8/site-packages/xarray/backends/common.py:200, in <dictcomp>(.0)
    183 def encode(self, variables, attributes):
    184     """
    185     Encode the variables and attributes in this store
    186
   (...)
    198
    199     """
--> 200     variables = {k: self.encode_variable(v) for k, v in variables.items()}
    201     attributes = {k: self.encode_attribute(v) for k, v in attributes.items()}
    202     return variables, attributes

File /env/lib/python3.8/site-packages/xarray/backends/zarr.py:459, in ZarrStore.encode_variable(self, variable)
    458 def encode_variable(self, variable):
--> 459     variable = encode_zarr_variable(variable)
    460     return variable

File /env/lib/python3.8/site-packages/xarray/backends/zarr.py:258, in encode_zarr_variable(var, needs_copy, name)
    237 def encode_zarr_variable(var, needs_copy=True, name=None):
    238     """
    239     Converts an Variable into an Variable which follows some
    240     of the CF conventions:
   (...)
    255         A variable which has been encoded as described above.
    256     """
--> 258     var = conventions.encode_cf_variable(var, name=name)
    260     # zarr allows unicode, but not variable-length strings, so it's both
    261     # simpler and more compact to always encode as UTF-8 explicitly.
    262     # TODO: allow toggling this explicitly via dtype in encoding.
    263     coder = coding.strings.EncodedStringCoder(allows_unicode=True)

File /env/lib/python3.8/site-packages/xarray/conventions.py:273, in encode_cf_variable(var, needs_copy, name)
    264 ensure_not_multiindex(var, name=name)
    266 for coder in [
    267     times.CFDatetimeCoder(),
    268     times.CFTimedeltaCoder(),
   (...)
    271     variables.UnsignedIntegerCoder(),
    272 ]:
--> 273     var = coder.encode(var, name=name)
    275 # TODO(shoyer): convert all of these to use coders, too:
    276 var = maybe_encode_nonstring_dtype(var, name=name)

File /env/lib/python3.8/site-packages/xarray/coding/times.py:659, in CFDatetimeCoder.encode(self, variable, name)
    655 dims, data, attrs, encoding = unpack_for_encoding(variable)
    656 if np.issubdtype(data.dtype, np.datetime64) or contains_cftime_datetimes(
    657     variable
    658 ):
--> 659     (data, units, calendar) = encode_cf_datetime(
    660         data, encoding.pop("units", None), encoding.pop("calendar", None)
    661     )
    662     safe_setitem(attrs, "units", units, name=name)
    663     safe_setitem(attrs, "calendar", calendar, name=name)

File /env/lib/python3.8/site-packages/xarray/coding/times.py:592, in encode_cf_datetime(dates, units, calendar)
    582 def encode_cf_datetime(dates, units=None, calendar=None):
    583     """Given an array of datetime objects, returns the tuple `(num, units,
    584     calendar)` suitable for a CF compliant time variable.
    585
   (...)
    590     cftime.date2num
    591     """
--> 592     dates = np.asarray(dates)
    594     if units is None:
    595         units = infer_datetime_units(dates)

Anything else we need to know?
Our system uses dask_gateway in a AWS infrastructure (S3 for storage)
Environment

INSTALLED VERSIONS
------------------
commit: None
python: 3.8.10 (default, Jun 22 2022, 20:18:18)
[GCC 9.4.0]
python-bits: 64
OS: Linux
OS-release: 5.4.209-116.367.amzn2.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: C.UTF-8
LANG: C.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.10.4
libnetcdf: 4.7.3
xarray: 2022.3.0
pandas: 1.5.0
numpy: 1.22.4
scipy: 1.9.1
netCDF4: 1.6.1
pydap: installed
h5netcdf: 1.0.2
h5py: 3.7.0
Nio: None
zarr: 2.13.2
cftime: 1.6.2
nc_time_axis: None
PseudoNetCDF: None
rasterio: 1.3.2
cfgrib: None
iris: None
bottleneck: 1.3.5
dask: 2022.9.2
distributed: 2022.9.2
matplotlib: 3.6.0
cartopy: 0.20.2
seaborn: 0.12.0
numbagg: None
fsspec: 2022.8.2
cupy: None
pint: None
sparse: 0.13.0
setuptools: 65.4.1
pip: 22.2.2
conda: None
pytest: 7.1.3
IPython: 8.5.0
sphinx: None

This is correct -- CFDatetimeCoder.encode is not lazy, even if the inputs are Dask arrays.
We would welcome contributions to fix this. This would entail making the encode look similar to the decode method (using lazy_elemwise_func).
We would also need a fall-back method for determining appropriate time units without looking at the array values. Something like seconds since 1900-01-01T00:00:00 would probably be a reasonable choice.

@dcherian Thanks; I agree that this seems to be the same as #7028. Just as a note, I have had 3 people reach out to me (1 from UNH, 2 from across the globe) thanking me for the work around in my message. So this does seem to be a commonly encountered issue.


Creating beaver pid directory when missing

#564 Fixing by creating pid directory when missing during beaver start

Updated beaver_pre_install.yml to remove the pid dir

Oh yes you're right, cut'n paste victim... let me change it

can you make that a proper conditional?

Yeah "proper" is subjective I personally like on liners... but I'm fine with that

:+1:

@BjoernT update the commit message, rebase and :+1:

Yeah I had it re based this morning and updated the message already. Re based again.


Performance issue with NVSHMEM example

Hello,
I am observing significant performance issues running the nvshmem benchmark over InfiniBand. I'm running RHEL 7.9 on two compute nodes, each with 8x NVIDIA A100-SXM4 (40 GB RAM) and 2x AMD EPYC CPU 7352 (24 cores). The A100s inside the compute nodes are connected via NVLink, the compute nodes are connected via InfiniBand.
I'm configuring the nvshmem example with nx = ny = 32768. Running it with two GPUs on a single compute node yields the expected result of about half the single GPU runtime (from ~10 seconds to ~5 seconds). Running the same example with two GPUs on 2 compute nodes (so one GPU per compute node) results in a runtime of ~43 seconds, so about 4.3 times slower than the single GPU version.
I'm NVSHMEM 2.5.0 and OpenMPI 4.1.1. Do you have any idea what could cause this issue? Did I make a mistake while configuring/installing NVSHMEM?
I'd greatly appreciate any tips or suggestions.

@pazkaI for multinode runs please use nvshmem_opt directory as that is optimized for multinode runs.

@akhillanger thanks for the suggestion - I tried the nvshmem_opt version with multiple compute nodes and it seems to scale as expected.

Hi! I'm also trying to run nvshmem on multi-node environment. However, I encountered an error that two GPUs from two different nodes could not access each other.
When I run my application on SLURM using:
mpirun -np $SLURM_NTASKS $APP_PATH $DATA $SLURM_NTASKS

I got the following error report:
mpi_info: rank: 0, nranks: 2
mpi_info: rank: 1, nranks: 2
PE-0, local_gpu_num: 1, local_gpu_id: 0
PE-1, local_gpu_num: 1, local_gpu_id: 0

src/topo/topo.cpp:61: [GPU 1] Peer GPU 0 is not accessible, exiting ...
src/init/init.cpp:276: non-zero status: 3 building transport map failed
src/mem/mem.cpp:nvshmem_malloc:342: nvshmem initialization failed, exiting
src/topo/topo.cpp:61: [GPU 0] Peer GPU 1 is not accessible, exiting ...
src/init/init.cpp:276: non-zero status: 3 building transport map failed
src/mem/mem.cpp:nvshmem_malloc:342: nvshmem initialization failed, exiting

To clarify, these two nodes have an IB connection. Does the problem come from my mpirun command? Or do I have to configure nvshmem to notify it to use the IB connection?
It's very glad to see that you have successfully run NVSHMEM on multi-node. And, could you please help me solve this problem?

@zzzlxhhh can you share a bit more about your system configuration? What GPUs are you using? Are you using dmabuf or nvidia-peermem to setup GPUDirect RDMA? Working GPUDirect RDMA is a requirement for multi node NVSHMEM.

Thanks for your reply. I'm running my application on two nodes each with 8 A800 GPUs on the school cluster. It's very tricky to solve all the dependencies required for NVSHMEM on multi-node since I’m not the cluster administrator. I'm going to use Nvidia HPC SDK to build a container to run my application.
To clarity, I can run my application successfully on single-node multi-GPU.
By the way, to enable GPUDirect RDMA means I have to install GDRCopy and nvidia-peermem? Also, before nvidia-peermem, there exists nv_peer_mem. Do all NVSHMEM versions support the latest nvidia-peermem? Or some versions of NVSHMEM rely on nv_peer_mem?

My main concern is about the installation guide provided by Nvidia. It is unclear what installation settings should be met to run NVSHMEM on multi-node multi-GPU platform, since cross-node communication is more complicated than single-node multi-GPU. They list so many optional settings, especially the driver level softwares.

To update, I use nvidia hpc sdk container 22.11-devel-cuda11.8-ubuntu22.04 to run my application to ensure that all the dependencies are installed properly. However, I still run into trouble.
The SLURM script is as follows:
#!/bin/bash
#SBATCH -p g078t2
#SBATCH -N 2
#SBATCH --ntasks=2
#SBATCH --ntasks-per-node=1
#SBATCH --gres=gpu:1
#SBATCH --time=10:00
#SBATCH --comment=idmg_bupt

mpirun -n 2 singularity exec --nv -B /usr/local/nvidia \
        ~/nvsdk_118.sif ./coco.sh

The content in ./coco.sh is to call singularity to run my application:
#!/bin/bash
export LD_LIBRARY_PATH=/usr/local/nvidia/lib:$LD_LIBRARY_PATH
export PATH=/usr/local/nvidia:$PATH
source /usr/share/lmod/lmod/init/bash
module load nvhpc/22.11

~/distSpMM/AdaCo_AE/build/distSpMM com-Amazon 2 32 0 0

However, the error report is:
INFO:    Mounting image with FUSE.
INFO:    Mounting image with FUSE.
WARNING: underlay of /usr/bin/nvidia-smi required more than 50 (428) bind mounts
WARNING: underlay of /usr/bin/nvidia-smi required more than 50 (428) bind mounts
--------------------------------------------------------------------------
WARNING: No preset parameters were found for the device that Open MPI
detected:

  Local host:            gpu8
  Device name:           mlx5_0
  Device vendor ID:      0x02c9
  Device vendor part ID: 4123

Default device parameters will be used, which may result in lower
performance.  You can edit any of the files specified by the
btl_openib_device_param_files MCA parameter to set values for your
device.

NOTE: You can turn off this warning by setting the MCA parameter
      btl_openib_warn_no_device_params_found to 0.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There are more than one active ports on host 'gpu8', but the
default subnet GID prefix was detected on more than one of these
ports.  If these ports are connected to different physical IB
networks, this configuration will fail in Open MPI.  This version of
Open MPI requires that every physically separate IB subnet that is
used between connected MPI processes must have different subnet ID
values.

Please see this FAQ entry for more details:

  http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid

NOTE: You can turn off this warning by setting the MCA parameter
      btl_openib_warn_default_gid_prefix to 0.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
No OpenFabrics connection schemes reported that they were able to be
used on a specific port.  As such, the openib BTL (OpenFabrics
support) will be disabled for this port.

  Local host:           gpu8
  Local device:         mlx5_2
  Local port:           1
  CPCs attempted:       rdmacm, udcm
--------------------------------------------------------------------------
mpi_info: rank: 0, nranks: 2
mpi_info: rank: 1, nranks: 2
PE-1, local_gpu_num: 1, local_gpu_id: 0
PE-0, local_gpu_num: 1, local_gpu_id: 0
group_size is set to 0 for rowSpMM
graph_name: com-Amazon, num_GPUs: 2, dim: 32
IO read from bin time: 42.304001 ms
[gpu9:16180] *** Process received signal ***
[gpu9:16180] Signal: Segmentation fault (11)
[gpu9:16180] Signal code: Invalid permissions (2)
[gpu9:16180] Failing at address: 0x14724700e790
[gpu9:16180] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x147247bf0520]
[gpu9:16180] [ 1] [0x14724700e790]
[gpu9:16180] *** End of error message ***
[gpu8:16579] *** Process received signal ***
[gpu8:16579] Signal: Segmentation fault (11)
[gpu8:16579] Signal code: Invalid permissions (2)
[gpu8:16579] Failing at address: 0x147ff9535790
[gpu8:16579] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x147ff99f0520]
[gpu8:16579] [ 1] [0x147ff9535790]
[gpu8:16579] *** End of error message ***
/home/u2022110987/distSpMM/AdaCo_AE/multi_node/coco.sh: line 7: 16579 Segmentation fault      (core dumped) ~/distSpMM/AdaCo_AE/build/distSpMM com-Amazon 2 32 0 0
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
/home/u2022110987/distSpMM/AdaCo_AE/multi_node/coco.sh: line 7: 16180 Segmentation fault      (core dumped) ~/distSpMM/AdaCo_AE/build/distSpMM com-Amazon 2 32 0 0
[gpu8:16510] 7 more processes have sent help message help-mpi-btl-openib.txt / no device params found
[gpu8:16510] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[gpu8:16510] 5 more processes have sent help message help-mpi-btl-openib.txt / default subnet prefix
[gpu8:16510] 3 more processes have sent help message help-mpi-btl-openib-cpc-base.txt / no cpcs for port
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[37160,1],0]
  Exit code:    139
--------------------------------------------------------------------------

Does it because the openmpi-4.1.1 in the host is not the same with openmpi-4.1.5a1 in the sigularity container? Should I recompile an openmpi on the host side?

@zzzlxhhh sorry I can't help you much with solving what the issue with your container is. Regarding the NVSHMEM requirements: For multi node runs you need one of:

Mellanox OFED and nvidia-peermem
UCX which at least for performant exectuion also requires Mellanox OFED and nvidia-peermem if you are using a NVIDIA IB network
libfabrics e.g. if you are running on a system with slingshot 11.

These dependencies are described as optional because you can use NVSHMEM single node without them. As described if GDRCopy is needed or not depends on your use of atomics.
Hope this helps.

Thank you so much! Your explanation helps a lot. It's much more clear.

@zzzlxhhh sorry I can't help you much with solving what the issue with your container is. Regarding the NVSHMEM requirements: For multi node runs you need one of:

Mellanox OFED and nvidia-peermem
UCX which at least for performant exectuion also requires Mellanox OFED and nvidia-peermem if you are using a NVIDIA IB network
libfabrics e.g. if you are running on a system with slingshot 11.

These dependencies are described as optional because you can use NVSHMEM single node without them. As described if GDRCopy is needed or not depends on your use of atomics.
Hope this helps.

Sorry to bother you again! I deeply appreciate your kind response!
I have prepared a system with Mellanox OFED and nvidia-peermem. However, there are still errors when running on multi-node multi-GPU. (running on single-node multi-GPU is OK).
The error report is:
Running on nodes: gpu[8-9]
Executable path: ./distSpMM
mpi_info: rank: 0, nranks: 2
mpi_info: rank: 1, nranks: 2
PE-0, local_gpu_num: 1, local_gpu_id: 0
PE-1, local_gpu_num: 1, local_gpu_id: 0
group_size: 1 for XpMM
graph_name: com-Amazon, num_GPUs: 2, dim: 32
PE-0, l_split_ratio:1.001
PE-1, l_split_ratio:1.001
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
PE-1, local_if_nonempty_nvshmem: 0x10038c80600, remote_if_nonempty_d: (nil)
PE-0, local_if_nonempty_nvshmem: 0x10038c80600, remote_if_nonempty_d: (nil)
PE-1, local_row:150255 remote_col:149059 reduce_cnt:0
PE-0, local_row:150160 remote_col:148809 reduce_cnt:0
IO read from bin time: 463.783997 ms
/home/u2022110987/lib_src/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1025: non-zero status: 4 ibv_poll_cq failed, status: 4

/home/u2022110987/lib_src/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1085: non-zero status: 7 progress_send failed, outstanding_count: 1023

/home/u2022110987/lib_src/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1115: non-zero status: 7 check_poll failed

/home/u2022110987/lib_src/nvshmem_src/src/include/internal/host/nvshmem_internal.h:nvshmemi_process_multisend_rma:182: aborting due to error in process_channel_dma

/home/u2022110987/lib_src/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1025: non-zero status: 4 ibv_poll_cq failed, status: 4

/home/u2022110987/lib_src/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1085: non-zero status: 7 progress_send failed, outstanding_count: 1023

/home/u2022110987/lib_src/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1115: non-zero status: 7 check_poll failed

/home/u2022110987/lib_src/nvshmem_src/src/include/internal/host/nvshmem_internal.h:nvshmemi_process_multisend_rma:182: aborting due to error in process_channel_dma

--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[59241,1],1]
  Exit code:    255
--------------------------------------------------------------------------

To emphasize, the main issues are:
/home/u2022110987/lib_src/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1025: non-zero status: 4 ibv_poll_cq failed, status: 4

/home/u2022110987/lib_src/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1085: non-zero status: 7 progress_send failed, outstanding_count: 1023

/home/u2022110987/lib_src/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1115: non-zero status: 7 check_poll failed

/home/u2022110987/lib_src/nvshmem_src/src/include/internal/host/nvshmem_internal.h:nvshmemi_process_multisend_rma:182: aborting due to error in process_channel_dma

So, I checked the corresponding Q&A in NVSHMEM FAQ.

The Q&A is still very unclear. What behavior could trigger this issue?
Does Error or status: 4 - Local Protection Error mean I have some wrong operation about the GPU shared memory?
More specifically, in my case, I use in-kernel registers as the local buffer to store the remote data from another PE(GPU) as follows:
input is registered as nvshmem
float *tmp2 = (float *)&tmp[warpPerBlock * dim];
nvshmemx_float_get_warp((float *)&tmp2[blk_wid * dim], &input[r_offset * dim], dim, r_GPUid);

To my understanding, this behavior is not allowed in multi-node multi-GPU NVSHMEM, but allowed in single-node multi-GPU NVSHMEM?
Am I right? Could you please give some suggestions about my next step? Whether should I modify all the code? My project is huge, so I have to evaluate the possible time I have to invest to modify the code.

Problem solved. This is indeed caused by using local scratchpad memory as the buffer to receive remote data.

Great and thanks for closing the loop @zzzlxhhh . In general please use our developer forums for these kind of questions as they are not about the code in this repository. For NVSHMEM starting new Topic in the GPU-Accelerated Libraries Category: https://forums.developer.nvidia.com/c/accelerated-computing/gpu-accelerated-libraries/12 with the Tag nvshmem would be best.

Thanks for your suggestions! So sorry to bother you again. However, I also encountered a severe performance drop when running application on multi-node multi-GPU platform.  This issue has not yet been reported in the nvida developer forum. I will later post my issue in the developer forum too. This is also encountered by @covoge. And @akhillanger suggests to use nvshmem_opt directory, but I cannot find this directory.

Hello,
I am observing significant performance issues running the nvshmem benchmark over InfiniBand. I'm running RHEL 7.9 on two compute nodes, each with 8x NVIDIA A100-SXM4 (40 GB RAM) and 2x AMD EPYC CPU 7352 (24 cores). The A100s inside the compute nodes are connected via NVLink, the compute nodes are connected via InfiniBand.
I'm configuring the nvshmem example with nx = ny = 32768. Running it with two GPUs on a single compute node yields the expected result of about half the single GPU runtime (from ~10 seconds to ~5 seconds). Running the same example with two GPUs on 2 compute nodes (so one GPU per compute node) results in a runtime of ~43 seconds, so about 4.3 times slower than the single GPU version.
I'm NVSHMEM 2.5.0 and OpenMPI 4.1.1. Do you have any idea what could cause this issue? Did I make a mistake while configuring/installing NVSHMEM?
I'd greatly appreciate any tips or suggestions.

To clarify, the cross node connection is based on IB with around 20GB/s bandwidth, intra-node GPU connection is based on NVLINK with around 180 GB/s bandwidth. However, the performance drop drastically, from 0.1 ms (with 8 GPUs in 1 node) to 70 ms (2 nodes each with 4 GPUs). What could cause these performance degradation?


Calibfilter

This feature contains additional scripts for processing the RHI data, analyzing the PPIs and estimating the calibrations offsets based on daily data.

@prabshr Thanks for adding this. I'll have a look tomorrow early, checking on the errors.

I have made some important bug fixes to the calibfilter branch

@prabsh Hang on, I'll try to fix the CI errors in your branch. I'll upload the BoXPol RHI Data to the test-repo, too.

@prabsh The big issue is, that the calib notebook needs rain over the radar (vertical scan) but in our one hour test-data (2015-07-05 1400 to 1500) there is no rain over the radar.
To make this work we would need to select a fitting one-hour time range. Please always have in mind that the notebooks should represent the workflow to be run within the CI with the miubrt-test-data.
So how do we proceed? Agree on another hour of that day? Which? We also need to talk if Jülich Data is necessary to be represented in the notebooks. If so, we would need to also have test-data for this.

@kmuehlbauer we can have the vertical scan data in the repository, then ! The calib filter can create the vertical scan, with file generate flag set to True.
Similarly, we could also have RHI data in the repository top, then. The RHI script should create the data.
Maybe this resolves the issue ?

@prabshr Yes, but the problem is, that the proposed hour of that day (2015-07-05 1400 to 1500) there is no rain over the radar. SO we need to choose another time, right?

@kmuehlbauer - Ok understood, then lets change it to 2015-07-05 1530 to 1630, the cell should directly pass over both radars, and still constrain the data for 1 hour period only

@prabshr Good, I'll update the data repo.

Any updates ?

Any updates ?


feat(popover-edit): Helper to reduce form state boilerplate

Honestly not sure if this is worthwhile, but throwing it out there for comment.

The one should be ready now too.


Bottom placement on SE3 is different from iPhone14Pro

SE3 is on the left, iPhone14Pro on the right.

I think we don't have bottom margins when the physical home button is present. Let's leave it that way at the moment.


run_program for op_pubkey_for_exp is slow. Even slower than Python's clvm

It is really strange, but I got the same results even I tried benchmark multiple times.
I am releasing production-ready version of clvm_tools-js and I am currently benchmarking to compare performance with:

clvm_tools(js) with clvm(js)
clvm_tools(python) with clvm(python)
clvm_tools(python) with clvm(rust)

In summary, clvm in Rust implementation works great in run_program.
It is around 30-100 times faster than other implementation in most cases.
(But be aware, to_sexp_f benchmark is 5-10 times slower than other implementation. This is another story.)
When I got the benchmark result, I found that op_pubkey_for_exp is slower than Python implementation of clvm.
This is really strange. I haven't started investigation yet, but I'd like to share the current data with you.
The benchmark was done using benchmark files generated by costs/generate-benchmark.py in clvm_tools.
The benchmark parameters are:

value size: 4bytes(32bit)
operations count: 200

I modified generate-benchmrk.py a bit to produce benchmark files with such parameters.
I thought value_size = 4bytes and operations count = 200 seems to be typical among most of chialisp programs.
run_program

to_sexp_f
If you are interested, here is the result of to_sexp_f benchmark.

The reason for the difference in pubkey_for_exp is because clvm uses bls-signatures (aka blspy), and clvm_rs uses https://github.com/zkcrypto/bls12_381.

Note that clvm_rs is now an npm package https://www.npmjs.com/package/clvm_rs containing wasm32. The api is not good yet, but PRs are most welcome. See also https://github.com/Chia-Network/clvm_rs/tree/main/src/wasm

In clvm_rs, op_point_add does from_compressed instead of from_compressed_unchecked, which is probably slower. However, making this change would probably be a hard fork.

@richardkiss I'll try to import clvm_rs into clvm_tools-js soon.
I already have an idea on how to integrate clvm_rs into clvm_tools-js.


Removes StorableAccountsWithHashes from AppendVec

Problem
We no longer store account hashes in storages, yet we still haul 'em around and sometimes (re)calculate them. This is unnecessary.
AppendVec always stores default account hashes, so the hashes in StorableAccountsWithHashes are unnecessary. Since that was the only reason to use StorableAccountsWithHashes, it can be replaced back with the simpler StorableAccounts.
Summary of Changes
Updates AppendVec  to use StorableAccounts directly

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 81.8%. Comparing base (03b5965) to head (16739a0).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #853   +/-   ##
=======================================
  Coverage    81.8%    81.8%
=======================================
  Files         851      851
  Lines      231744   231742    -2
=======================================
+ Hits       189692   189718   +26
+ Misses      42052    42024   -28


Distance and Direction widget will not load into an app

WAB 2.7 and 2.8 snapshot

Create a new app
Try to load the Distance and Direction widget into the application
The widget hangs on the configuration page with the three "working" dots just cycling

On 2.7 I was adding our widget in, on 2.8 I just used what was already in WAB.

@adgiles @kgonzago  anything I could be missing? @Dbarnes1 saw the same.

@dfoll I no longer have access to the latest D&D code since it has been moved up to devtopia, so I cannot check

@dfoll
Looks like there are issues with some of the localisation in the settings.js, instead of this.nls it is this.nnls (double n), on the following lines:
76:  title: this.nnls.lineColorLabel,
84:  title: this.nnls.lineWidthLabel,
112:  title: this.nnls.feedbackLabel,
117:  title: this.nnls.textColorLabel,
125:  title: this.nnls.textSizeLabel,
225:  feedbackLabel: this.nnls.feedbackLabel,
I am not sure what @kgonzago fixed in this PR mentioned here: https://github.com/Esri/distance-direction-addin-dotnet/issues/445#issuecomment-367118734

@dfoll all issues encountered were fixed in this PR. Unfortunately that PR has not been merged, sitting as a PR for the past 10 days. Please look at this branch if you want to test

@kgonzago
I think your fixes to the localization string errors fix this issue:
https://github.com/Esri/distance-direction-addin-dotnet/issues/507
If so, could you comment in it, remove the Dev tag?
Issue https://github.com/Esri/distance-direction-addin-dotnet/issues/508 is not addressed, right?
Issue https://github.com/Esri/distance-direction-addin-dotnet/issues/506 is about typographic/formatting issues with ellipse labels.  Probably an easy fix.
@dfoll - FYI

#506 and #507 can be tested using this branch.
#508 is a core WAB issue. An issue will be submitted in Devtopia.

develop branch from devetopia no longer has this issue


More graceful abort when rate limit is hit on list import

yeah, like just skip that list, and then either move onto the next one, or wait a bit and try again, or just return all the results that have already been found before the limit was hit


[bug]: lncli listpayments --count_total_payments should honor --creation_date_start --creation_date_end

Background
lncli listpayments --count_total_payments should honor --creation_date_start and --creation_date_end
Steps to reproduce
$ lncli listpayments --count_total_payments|grep total_num_payments
    "total_num_payments": "30560"
$

and
$ lncli listpayments --creation_date_start<PHONE_NUMBER> --count_total_payments --creation_date_end<PHONE_NUMBER>|grep total_num_payments
    "total_num_payments": "30560"
$

both yield the same thing.
I would expect
$ lncli listpayments --creation_date_start<PHONE_NUMBER> --count_total_payments --creation_date_end<PHONE_NUMBER>|grep total_num_payments
    "total_num_payments": "0"
$

I would like to work on this issue! Can I take it?

Hi @AndySchroder , since the issue doesn't have an assignee, I took a stab at it. Could you please have a look? I already answered the bot reviewer's questions

Okay, looks like https://github.com/lightningnetwork/lnd/pull/8565 has been cancelled and this issue will need a new attempt.

Okay, looks like #8565 has been cancelled and this issue will need a new attempt.

I've been investigating this issue and the code associated with it ... https://github.com/lightningnetwork/lnd/blob/8adec8692ad2b8e1d1a4023c56ccc6db8ddcce76/channeldb/payments.go#L514
I would like to do a new attempt to solve it.


Castblock claims to be skipping, but nothing happens

Ran into this just now - It looks like it's trying to skip the same segment but nothing actually happened:
Apr 28 11:46:20 pihole castblock[76792]: 2021-04-28T15:46:20.632400Z   INFO - castblock.blocker: Found a sponsor segment, skipping it. -- id: "kMxcw_lQBjI", start: 40.812607, end: 102.831894
Apr 28 11:46:33 pihole castblock[76792]: 2021-04-28T15:46:33.636287Z   INFO - castblock.blocker: Found a sponsor segment, skipping it. -- id: "kMxcw_lQBjI", start: 40.812607, end: 102.831894
Apr 28 11:46:43 pihole castblock[76792]: 2021-04-28T15:46:43.642162Z   INFO - castblock.blocker: Found a sponsor segment, skipping it. -- id: "kMxcw_lQBjI", start: 40.812607, end: 102.831894
Apr 28 11:46:53 pihole castblock[76792]: 2021-04-28T15:46:53.648376Z   INFO - castblock.blocker: Found a sponsor segment, skipping it. -- id: "kMxcw_lQBjI", start: 40.812607, end: 102.831894
Apr 28 11:47:06 pihole castblock[76792]: 2021-04-28T15:47:06.650518Z   INFO - castblock.blocker: Found a sponsor segment, skipping it. -- id: "kMxcw_lQBjI", start: 40.812607, end: 102.831894
Apr 28 11:47:16 pihole castblock[76792]: 2021-04-28T15:47:16.652738Z   INFO - castblock.blocker: Found a sponsor segment, skipping it. -- id: "kMxcw_lQBjI", start: 40.812607, end: 102.831894
Apr 28 11:47:26 pihole castblock[76792]: 2021-04-28T15:47:26.657088Z   INFO - castblock.blocker: Found a sponsor segment, skipping it. -- id: "kMxcw_lQBjI", start: 40.812607, end: 102.831894
Apr 28 11:47:39 pihole castblock[76792]: 2021-04-28T15:47:39.662750Z   INFO - castblock.blocker: Found a sponsor segment, skipping it. -- id: "kMxcw_lQBjI", start: 40.812607, end: 102.831894
Apr 28 11:47:49 pihole castblock[76792]: 2021-04-28T15:47:49.664129Z   INFO - castblock.blocker: Found a sponsor segment, skipping it. -- id: "kMxcw_lQBjI", start: 40.812607, end: 102.831894
Apr 28 11:47:59 pihole castblock[76792]: 2021-04-28T15:47:59.666159Z   INFO - castblock.blocker: Found a sponsor segment, skipping it. -- id: "kMxcw_lQBjI", start: 40.812607, end: 102.831894

Unfortunately I wasn't running in debug mode. Also worth noting that it seems to be permanently stuck in this state.
I'm not familiar at all with the go-chromecast http api, any ideas on what might be the cause?
Edit: Could it be that the watch thread is still connected but the http api used for skipping lost the device?

The log for error when calling /seek-to uses debug level :roll_eyes: . I changed it to error level and I added the error from go-chromecast in the message. You can use master (or latest docker images), if it happens again we will see the error.
Anyway the timing is weird. watch is set to send a message every second, and the code just react to those messages, but thoses log indicate that it is trying to seek every 10 or 13s. It seems like something (the httpserver probably) is either slow to timeout or does automatic retries.

I just add the same bug as you!
This is what I saw:

/devices still see the device
/connect?uuid=... says the device is already connected
but /seek-to complains with "unable to seek-to device: unable to write binary format: write tcp <IP_ADDRESS>:58146-><IP_ADDRESS>:8009: write: broken pipe"

Unfortunately I didn't try to call /disconnect, then /connect again, then /seek-to. If you do have the error again could you try to do that please?

The error comes from here: https://github.com/vishen/go-chromecast/blob/0795e28217f8b18586ebb69c1569c46451dbab0b/cast/connection.go#L121. It seems like the connection is just broken. I will implement a disconnect / reconnect mechanism.

Sounds good, I was too slow to reproduce the issue :)

Looks like that fix makes disconnecting and reconnecting work to fix the issue. I guess all that's needed is to auto reconnect when it gets disconnected

Tell me if it happens again. I spotted a weird bug in go-chromecast where the /seek-to endpoint can return a 200 even if the connection is broken (https://github.com/vishen/go-chromecast/issues/106). If that happens we will not do a disconnect / reconnect.
But I suspect that in real life conditions the endpoint would return an error, and in that case the code will try to reconnect.


Android Signing Task V3.0 fails when password string contains an ampersand

Environment
VSTS
Hosted 2017 agent
Issue Description
Android Signing Task V3.0
If the keystore file password or the keystore key password have an ampersand (&) in the string then the commands fail because it seems like they attempt some kind of variable substitution resulting in the parts after the & raising errors (* is not recognised as an internal or external command
Error logs
2018-08-05T06:29:10.6428349Z ##[section]Starting: Signing and aligning APK file(s) $(build.binariesdirectory)/Release/*.apk
2018-08-05T06:29:10.6433658Z ==============================================================================
2018-08-05T06:29:10.6433848Z Task         : Android Signing
2018-08-05T06:29:10.6433974Z Description  : Sign and align Android APK files
2018-08-05T06:29:10.6434114Z Version      : 3.137.0
2018-08-05T06:29:10.6434229Z Author       : Microsoft Corporation
2018-08-05T06:29:10.6434385Z Help         : [More Information](https://go.microsoft.com/fwlink/?LinkID=613717)
2018-08-05T06:29:10.6434531Z ==============================================================================
2018-08-05T06:29:15.5160490Z [command]C:\Windows\system32\cmd.exe /D /S /C ""C:\Program Files (x86)\Android\android-sdk\build-tools\24.0.3\apksigner.bat" sign --ks D:\a\_temp\Sample.keystore --ks-pass "pass:dummypassword&RND1" --ks-key-alias sample --key-pass "pass:dummypassword&RND2" --verbose D:\a\1\b\Release\com.sample.app.apk"
2018-08-05T06:29:20.8298982Z 'RND1' is not recognized as an internal or external command,
2018-08-05T06:29:20.8299639Z operable program or batch file.
2018-08-05T06:29:20.8418881Z 'RDN2' is not recognized as an internal or external command,
2018-08-05T06:29:20.8419284Z operable program or batch file.
2018-08-05T06:29:23.4293263Z Failed to load signer "signer #1"

@ericsciple hi Eric, do you know how we can escape the & in this case?  The task is passing in the secret with the arg function:
 apksignerRunner.arg(['--ks-pass', 'pass:' + keystorePass]);

@yacaovsnc you could try to put quotes around the value: instead of 'pass:bad&password' have it send 'pass:"bad&password"' but then the apksignerRunner may consider the quotes are part of the password... so, maybe '"pass:bad&password"' will work...

@GZidar we spawn the child process with child_process.  I don't think we should change the quotes around the argument input.
Can you give this a try?  Can you escape the & char?

@yacaovsnc I tried that before posting the issue. When using the ^ before the & the error goes away but the password is considered invalid.

@yacaovsnc @ericsciple I have tried some testing just using the command line and the call to the apksigner.bat file seems to work properly if the two passwords are passed like this pass:"" note that the double quotes are placed around only the password string. It might be a matter of just adjusting this and everything will work as expected.
I hope that helps.

It turns out my issue was made more complex by the fact that not only did the password contain an ampersand it also contained a per cent symbol, both of which cause issues to the apksigning batch file.
By escaping the % (using %%) and not the & (^&) the apk was signed even though the console reported errors.
Ideally the task should not require the values to be escaped in the task definition but instead does the appropriate escaping within its own processing, but it is good to know that it was possible to get the task to work.

@GZidar Glad it worked at last!  As to escaping the password within the task, I need to think about it a bit more as I usually shy away from changing user inputs since if we introduce any bugs in those logic, user will have no way to workaround those issues.
I am closing this issue as the & and % can be escaped manually for now.

I'm facing the same issues in de AzureCli task. If my input parameters contain a % it gets dropped from the input.

https://github.com/Microsoft/vsts-tasks/issues/8104


Buttons do not work

Need to lean how to animate listview and use scrollcontroller.animateTo

Sad, but the only way to resolve this was to get rid of the buttons-- was taking too much time. If anyone wants to ahead and fix that just create a pull request.


Investigate how to use S3 buckets

DuckDB supports querying directly from the cloud storage.  I think we need to refactor the queries to do this.  Some things I'm pretty sure we need to do.

[ ] Create a connection and then conn.execute() instead of duckdb.query.  This will allow us to install and enable the S3 extension.
[ ] Need to allow the user to specify the region, access key and access secret.
[ ] May make sense to make the queries a class where we can specify the required info once.  For example on init, the user could specify AWS credentials, timeseries paths, etc.  The there could be query methods...need to think this through.

We are moving away from DuckDB for queries and also know how to do this now.


Fix tests on windows, enable windows to github auto-testing

Fixes a test test_evolutionary_agent that was failing on Windows because it was using bash specific path syntax.
Updates Github Actions workflow so that

tests automatically run on windows (in addition to existing envs that include MacOS and Linux)
tests automatically run after a push to any branch (but only on PRs to master, which is current behavior). This will make sure the tests are run when we push changes to our forks so that we can see test github test results before we submit our PRs. NOTE that creating a draft PR is another way to run the tests while preparing a PR.

Fixes #39.

@nickjalbert, I believe this PR that fixes/adds windows testing is ready for review!

@nickjalbert, I believe this PR that fixes/adds windows testing is ready for review!

Yeah, I agree that having the dependencies for example agents duplicated is asking for trouble.
I originally had intended I have each example agent be as standalone as possible,so that folks can get the best sense of how an Agent Directory is intended to encapsulate everything about an agent, including its dependencies.
I also wanted to demonstrate how different agents can make their own choice for dependency management tooling, so I wanted to show some using conda, and some using virtualenv (and eventually some using pipenv).
But that slightly complicates the centralized agentos testing strategy, since then you need to create a separate env for each example Agent, then install agentos and that agents dependencies in that env, and then run that agents tests in it. That’s what I was doing originally, but it got cumbersome enough that I broke down and started consolidating dependencies for example agents into the [what was then called] test-requirements.txt.
Setting up a virtual env and installing dependencies from scratch (esp. tensorflow, which is heavyweight) for each test also made the tests take longer to run.
We should probably come up with a well thought out set of conventions for example agents.
It should also address how we decide what type of agent goes into the example_agents dir vs into <proj_root>/agentos/agents.py which currently only contains RandomAgent.
It seems like we will want be able to build and test the agentos core functionality—including APIs and tooling (e.g., CLI)... basically anything in the agentos dir—independent of example agents, which are likely to be heavier weight, and have much more extensive dependencies.
Using an entirely different repo (or many repos) for example agents is a good option, but that also makes them more tricky to maintain and more, less easy for new users to try out or explore, and more likely that we neglect to update them as the core expands.
We could just keep them in the same repo like they are now, but separate out there dependencies and build file from the core stuff.
However, having 3 places agents can live (agents.py, example_agents dir, and separate repos) is probably too confusing and complex at this stage of the project.

Yeah, I agree that having the dependencies for example agents duplicated is asking for trouble.
I originally had intended I have each example agent be as standalone as possible,so that folks can get the best sense of how an Agent Directory is intended to encapsulate everything about an agent, including its dependencies.
I also wanted to demonstrate how different agents can make their own choice for dependency management tooling, so I wanted to show some using conda, and some using virtualenv (and eventually some using pipenv).
But that slightly complicates the centralized agentos testing strategy, since then you need to create a separate env for each example Agent, then install agentos and that agents dependencies in that env, and then run that agents tests in it. That’s what I was doing originally, but it got cumbersome enough that I broke down and started consolidating dependencies for example agents into the [what was then called] test-requirements.txt.
Setting up a virtual env and installing dependencies from scratch (esp. tensorflow, which is heavyweight) for each test also made the tests take longer to run.
We should probably come up with a well thought out set of conventions for example agents.
It should also address how we decide what type of agent goes into the example_agents dir vs into <proj_root>/agentos/agents.py which currently only contains RandomAgent.
It seems like we will want be able to build and test the agentos core functionality—including APIs and tooling (e.g., CLI)... basically anything in the agentos dir—independent of example agents, which are likely to be heavier weight, and have much more extensive dependencies.
Using an entirely different repo (or many repos) for example agents is a good option, but that also makes them more tricky to maintain and more, less easy for new users to try out or explore, and more likely that we neglect to update them as the core expands.
We could just keep them in the same repo like they are now, but separate out there dependencies and build file from the core stuff.
However, having 3 places agents can live (agents.py, example_agents dir, and separate repos) is probably too confusing and complex at this stage of the project.

I created #44 to track creation of a design doc for improved example agent processes, conventions, tooling.

I created #44 to track creation of a design doc for improved example agent processes, conventions, tooling.


Service Manual Publisher post-DB switch configuration changes on prod

This is opened as a draft PR it is intended to be merged in as part of the production DB upgrade.
This updates the db hostname, sets up the env sync config for the new Service Manual Publisher RDS instance and removes the previous config from the main DB Admin instance.
Trello: https://trello.com/c/st7HeiYK/60-plan-production-upgrade-for-small-postgresql-databases

@davidgisbey It looks like this one needs rebasing against the main branch before we can merge it


Simple rewind concept

I'd like to be able to re-use objects with animation sequencer with some kind of pool. To able able to do so I noticed that a Rewind method call would be handy.

You should be able to just play the Sequence again right? With Play();
Or you want a PlayBackwards();


Development of DDOFTrolleySuspension dynamic systems

New system is to implement in module motorcoach.py. It should be inherited from similar class being implemented within scope of the module if possible.

The system works properly, when called using DynSysOverviewReport it returns all the necessary information.
Same issue as #152


Rename starter name from azure-starter-* into starter-azure-*

Codecov Report

Merging #312 into master will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #312   +/-   ##
=========================================
  Coverage     51.28%   51.28%
  Complexity      272      272
=========================================
  Files            99       99
  Lines          1248     1248
  Branches         50       50
=========================================
  Hits            640      640
  Misses          577      577
  Partials         31       31

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e6ac2bb...d7d96f3. Read the comment docs.

We also need to update the titles and terms in all README files, right?

Accidentally clicked on the close button. Sorry about that. 😂


tests(dbless) add regression test for DB-less plugin iterator build f…

…ailure
"plugins iterator was changed while rebuilding it" from a6a3080
This test was not included during the Workspaces refactor, however it is
still efficient at catching such bugs in the future. Therefore it is
picked back to the main branch.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

@gszr Since the 2.1 release is already out I think we don't need this on master immediately. Merging it on next makes sense since that is where the development happens and this test is trying to catch bugs caused by big changes anyway.


Verification of Test-ID

The entered Test-ID (Testergebnis erfassen) should be checked if it is a valid ID from the BfArM-Liste.
A tool tip that shows the Name (from the list) could also be added for a manual verification.

@flo-so This is how it looks in my interface, is it different for you?

For me in the mobile Version, it looks live this, and IT accepts the wrong Test-ID

Maybe this is just a test or mobile version issue

The TestID field has two functions:

provide a quicksearch function to the underlying BfArM List
provide freetext functionality for testkids that are currently not in the BfArM List, but are indeed eligible and chargeable.

Be aware that, whatever you type in as testkid id, will be printed to the pdf from that is generated with the test result.


Frontend implementation of the platform

Hi @soledadli! Awesome work on the platform's design!!! It's so cool to see it implemented.
I'll go through this PR in the next few days and add any review comments that would be helpful.

This is truly fantastic work @soledadli ! I've been looking at the deployed version and the design is awesome. There are a few things I've noticed around the text content and flow which need changing but I think this makes sense to add as separate issues and fix forwards so we can get this merged to the main repo!

Are you able to take a look @BrainonSilicon? Do you reckon we can go ahead and merge this now?

@helendduncan and I reviewed most of this today. I think the design is really nice. There are obviously bits that we would want to change (e.g. hardcoded experiences, some links here and there not working...), but I think we should merge this to main now, and branch off from there, since the whole site is still in dev. This PR makes a big positive change to the look and feel of the site, well done @soledadli!

@GeorgiaHCA @BrainonSilicon do we have a practice for merging? I'd be happy to just go ahead and merge but want to check how you work first.


Redesign a non-destructive vector node data format

Duplicate of #1162

Sorry for not writing a more descriptive explanation about each of these two issues. #1162 refers to a short-term goal of encoding a diff for our existing SVG-based vector format. This issue refers to a series of upcoming design meetings and refactors that will move us from the MVP format we're using today to something supporting arbitrary bezier meshes (like the OBJ 3D format but with beziers instead of straight edge connections) and a way of laying out the data in a way that can be operated on procedurally like in Houdini and Blender geometry nodes. This issue refers to this item on the roadmap:


The following pictures show the display effect when select Sarasa term sc regular, Sarasa term sc semibold, Sarasa term sc nerd regular and Sarasa term sc nerd semibold in Visual Studio setting for code editor's font:

It can be seen that the font display effects in CODE EDITOR is same when select Sarasa term sc nerd regular or semibold. This is very strange, because it is correctly displayed in the SAMPLES in SETTING WINDOWS, and Sarasa term sc also correctly displayed in both regular and semibold in CODE EDITOR and SAMPLES in SETTING WINDOWS.
I have confirmed that both Sarasa term sc font and Sarasa term sc nerd font is installed with Regular and semibold versions:

Thanks for reporting this issue. I can reproduce this bug on my end.
I think this bug is caused by the font name Sarasa Term SC Nerd Font SemiBold being too long for Visual Studio.
Some Windows applications may encounter problems when the font name is excessively long.
As a result, the SemiBold font may not be rendered properly by Visual Studio.
The solution I have come up with is to release a separate set of fonts that have shorter font names.
I will probably try to fix this bug next weekend while I have some free time.

Hi @EVANGELION-Air,
I recreated the Sarasa Term SC Nerd Font SemiBold font, renaming it as Sarasa Term SC NF SmBd, and it appears to have resolved the issue:

Before

After

Here is the link to download the fixed fonts for now:
shorten-sarasa-term-sc-semibold-nerd-font.zip
You could try it out and see if the issue is resolved.


fix user agent header for sparql queries

-> elastic-wikidata/0.1
Also look at retry-after header.
See https://github.com/TheScienceMuseum/heritage-connector/commit/4208d593fec57f50161524b90ab0958b30cbf7dd

closed by #5


Address the norm bug in issue 1140

Description
fix the issue found here
Fixes #(issue)
Make a function for uniform diagonal tensors that correctly computes the norm of the tensor
Minimal demonstration of previous behavior
using ITensors
i, j = Index(2), Index(2)
norm(delta(i,j)) == √(2) # returns false

Minimal demonstration of new behavior
using ITensors
i, j = Index(2), Index(2)
norm(delta(i,j)) == √(2) # returns true

How Has This Been Tested?
Added a test to diag.jl

Codecov Report

Merging #1141 (3fafae9) into main (72b2cc3) will decrease coverage by 31.85%.
The diff coverage is n/a.

:exclamation: Current head 3fafae9 differs from pull request most recent head 9aaff92. Consider uploading reports for the commit 9aaff92 to get more accurate results

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
@@             Coverage Diff             @@
##             main    #1141       +/-   ##
===========================================
- Coverage   85.27%   53.42%   -31.85%
===========================================
  Files          87       85        -2
  Lines        8334     7790      -544
===========================================
- Hits         7107     4162     -2945
- Misses       1227     3628     +2401

see 68 files with indirect coverage changes


NIFI-10751 Add Username to User and Group CRUD

Summary
NIFI-10751
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking

[x] Apache NiFi Jira issue created

Pull Request Tracking

[x] Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
[x] Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

[x] Pull Request based on current revision of the main branch
[x] Pull Request refers to a feature branch with one commit containing changes

Verification
Please indicate the verification steps performed prior to pull request creation.
Build

[ ] Build completed using mvn clean install -P contrib-check

[ ] JDK 8
[x] JDK 11
[x] JDK 17

Licensing

[x] New dependencies are compatible with the Apache License 2.0 according to the License Policy
[x] New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

[x] Documentation formatting appears as expected in rendered files

@shawnweeks this looks good, I know its a fairly simple change, but could we update the unit test for the event factory? TestEventFactory it would help to avoid situations like this

I'll take care of that today @bbende

@bbende added test cases


Unable to start the cassandra container

I am trying to run wasabi on CentOS, after building everything i am unable to start the wasabi service with the following error.
[root@###### wasabi]# ./bin/wasabi.sh start
[ERROR] Failed to start cassandra container, please check the logs
docker: Error response from daemon: open /var/run/docker/libcontainerd/containerd/b2c4004939ad484d3affc9c30d853a40480345103af95d1e1cb18295d239dd3a/init/shim-log.json: no such file or directory.
error: unable to run command: % docker run --net=wasabi_nw -it --rm mysql:5.6 mysql -h -P3306 -uroot -pmypass -e " create database if not exists wasabi;
grant all privileges on wasabi. to 'readwrite'@'localhost' identified by 'readwrite';
grant all on .* to 'readwrite'@'%' identified by 'readwrite';
flush privileges;"

follow,the same error

any updates?


New sparkle-stroke icon contains flawed references

Describe the bug
The sparkle-stroke.svg file contains an illegal reference mask="url(#path-1-inside-1_920_5084)" that when run caused the Alaska Credit Card icon to fail.
Expected behavior
All icons need to render correctly.

Here is a re-exported sparkle-stroke. I renamed one of the layers in hopes that it will help resolve the issue at hand.

@KaylaBlyton1 do you know what is creating a mask over the icon? I believe that is where the issue is coming from.
When I remove the code in the SVG that creates the mask over the image, I get the following. It's this mask that is causing the issue.

This icon is proving to be a trick. This SVG is making use of a stroke and fill to create the shape. This is causing issues on how we process the icons for use. But looking at the star-stroke icon, this is just the shape, not stroke and fill. Can the sparkle-stroke be updated to be like the star?

@blackfalcon Thank you for explaining! The star example was very helpful. I have flattened this icon into oblivion - it SHOULD work now.

OK... 4th time is the charm?! @blackfalcon

@blackfalcon I reached out to June, but in the meantime I tried one more thing. Let me know if this works? Strokes are outlined, flattened, and I double checked and there are no masks in the layers.

@KaylaBlyton1 WE HAVE A WINNER!

Whatever was learned from this, can you work with @leeejune to be sure to document this?