GWT: AssetFilter changed to match Android build and file system restrictions

Fixed two issues with DefaultAssetFilter:
DefaultAssetFilter accepts files beginning with full stops (for example .gitignore) and adds this files to the assets.txt. However, these files are (under Windows) not copied to war/assets leading to a JavaScript warning.
DefaultAssetFilter accepts files and folders beginning with an underscore in contrast to the Android build process were these files are omitted when packaging the Android apk.

Are you sure that windows doesn't copy all dot-files instead of only those listed here
Likewise there seems to be a workaround for android:
https://stackoverflow.com/questions/9206117/how-to-workaround-autoomitting-fiiles-folders-starting-with-underscore-in
And yet another question, but couldn't you do
		String normFile = file;
		if (normFile.contains("\\.")) return false;
		if (normFile.contains("\\_")) return false;

instead of
		String normFile = file.replace('\\', '/');
		if (normFile.contains("/.")) return false;
		if (normFile.contains("/_")) return false;

?

Interesting some see this as an issue on Android, for me it is a feature. I have seen several projects using folders in the assets directory starting with an underscore with the intention that it is not copied. I adapted that and use underscore folders for unpacked skins, drawables or marketing material in my projects. So I hope if they "fix" the Android build they make this configurable.
For Windows, your changed code would work, but I assume it would not work on Unix and Mac. That would lead to differing builds depending of the operating systems.
To make it clear, you can always override this AssetFilter if you don't like the behaviour. As a default filter, I am convinced one that resembles the Android build filter makes the most sense especially for new users.


Add a CONTRIBUTING.md

Summary
Add a document that describes something like the following:

How to get your dev environment set up
How to find an issue
Things to do before submitting your patch
Things we like to see in a patch

on ubuntu 18.04 personally just had to
apt install docker.io xorriso

We should probably also add make dev-init too.


Fixing code vuln

CI was broken.

Codecov Report

Merging #628 (3e635b0) into main (5440bee) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #628   +/-   ##
=======================================
  Coverage   26.64%   26.64%
=======================================
  Files          55       55
  Lines        5855     5855
=======================================
  Hits         1560     1560
  Misses       4023     4023
  Partials      272      272


Feature request: return as a string

Would be good to be able to return the table as a string, instead of printing it

Maybe it can be done from the client side, configuring a StringIO to collect the output then converting it to a plain String.


Allow GlyphBrushBuilder to take the 'static lifetime

Replace slice parameter of GlyphBrushBuilder::using_font with Cow
Update examples and documentation accordingly

This crate is fantastic - thank you so much!

Thanks for the PR!
I think I prefer using Into<SharedBytes<'a>> rather than Cow. It is more in-line with rusttype, ie closer to the truth in this case. It also requires less code change. (But you do need to cast the bytes to &[u8] annoyingly)
Are you happy with the changes?

I didn't realise the include_bytes! issue was nightly related. Thanks for pointing that out.
I'm happy with the changes. :smile:

@alexheretic when are you planning to publish next version with this fix?

@vitvakatu I'll publish 0.3 later tonight.


Sync.pull_exn fails with Invalid_argument exception when called twice on two different brances

I discovered this when writing tests for irmin-graphql, but the issue can be easily reproduced:
module Store = Irmin_unix.Git.FS.KV(Irmin.Contents.String)
module Sync = Irmin.Sync(Store)
let cfg = Irmin_git.config "./tmp-store"
let origin = "git://github.com/mirage/irmin"

open Lwt.Infix
let main =
  Store.Repo.v cfg >>= fun repo -> Store.of_branch repo "gh-pages" >>= fun gh_pages ->
  Store.master repo >>= fun t ->
  Sync.pull_exn t (Store.remote origin) `Set >>= fun () ->
  Lwt_io.printl "Pulled master" >>= fun () ->
  Sync.pull_exn gh_pages (Store.remote origin) `Set >>= fun () ->
  Lwt_io.printl "Pulled gh-pages"

let () = Lwt_main.run main

Which fails with the following error:
Fatal error: exception (Invalid_argument
  "Sync.pull_exn: {`Smart (`No_assert_predicate #predicate)}")

/cc @dinosaure, since this seems like it is probably an ocaml-git issue

Indeed, I will investigate why we have this.

Ok the bug definitely comes from ocaml-git on the negotiation engine which does not expect much more noticed commits when it sends the done status. I forked irmin to keep the current state of the git repository. However, this bug will take a long time to be fixed when we don't have any documentation about negotiation/smart protocol in Git. Thanks for your report.

I'm closing this as mirage/ocaml-git#338 has been merged.


Missing TorchConfig.cmake

The CMakeLists.txt for cpp-torch rely on a  find_package(Torch REQUIRED)
even if the torch Lib never actually gets built (we only builf TH TH_NN ...)
If you try to compile from a machine that include a full torch install, it's going to pick it up from there, but if you try to compile from a new setup, it's going to complain about

Could not find a package configuration file provided by "Torch" with any of
the following names:

TorchConfig.cmake
torch-config.cmake

Add the installation prefix of "Torch" to CMAKE_PREFIX_PATH or set
"Torch_DIR" to a directory containing one of the above files.  If "Torch"
provides a separate development package or SDK, be sure it has been
installed.

@gheaeckkseqrz did you get around the problem?

No, I ended up writting my own NN runtime in C++/OpenCL.


Authentication failed

[INF]: LN: Failed to retrieve the shadow entry .
I installed sysrepo and netopeer2 and changed the port from default to 8000 in<EMAIL_ADDRESS>After that I tried to connect using netopeer2-cli and this was the result
In netopeer2-cli it shows
"nc ERROR: Unable to authenticate to the remote server (all attempts via supported authentication methods failed).
cmd_connect: Connecting to the localhost:8000 as user "harmandeep" failed."

I do not think the problem is the changed port, does the user harmandeep exist on the system?


reverse MFMA order in inner loop for odd outer iteration

The idea is to use same source (A or B) at the end of even outer loop and at the beginning of odd outer loop in mfmaIter.
ex.)
before this change (always same order in inner loop):
outer loop 0:
inner loop: 0, 1, 2, .... 15
outer loop 1:
inner loop: 0, 1, 2, .... 15
after this change (inner loop index is reversed at each odd outer loop):
outer loop 0:
inner loop: 0, 1, 2, .... 15
outer loop 1: (starting from same index as inner loop 0)
inner loop: 15, 14, 13,....
Using same source for MFMA is a little power efficient and it can improve perf slightly.

This logic is applied in general but some exceptions.

PrefetchGlobalRead=2 and last inner loop and DirectToVgpr
-> DirectToVgpr load scheduling requires non-reversed order
gfx11 (wmma)
-> untested
complex type
-> src vgpr is not same even if we reverse index


unable to launch on iOS emulator

Error (Xcode): Building for 'iOS-simulator', but linking in object file (.pub-cache/hosted/pub.dev/esp_blufi-0.0.7/ios/Classes/BlufiLibrary/Security/openssl/libcrypto.a[arm64]2) built for 'iOS'
Error (Xcode): Linker command failed with exit code 1 (use -v to see invocation)
Uncategorized (Xcode): Command SwiftCompile failed with a nonzero exit code
Could not build the application for the simulator.
Error launching application on iPhone 15 Pro Max.

ESP Blufi do not support simulator.

looks is openssl issue, there's some way to avoid this error and compile on simulator , of course without using library

I see openssl works on simulator https://github.com/x2on/OpenSSL-for-iPhone


Support for the Strict-Transport-Security header

EQUELLA installs which should only be accessible by HTTPS can make the browser enforce this by using the Strict-Transport-Security header. This can be a server level config setting in optional-config.properties
https://www.owasp.org/index.php/HTTP_Strict_Transport_Security_Cheat_Sheet
Enable with:
strictTransportSecurity.maxage = 86400 # length in seconds

Closing, manual testing was successful


ConnectionError: Error 32 while writing to socket. Broken pipe

Hi,
We are running using python 2.7 on AKS (azure kubernetes service) and have a redis cluster deployed in our AKS deployed with kubedb.
Due to some issue between azure load balancer and confluent kafka we added an init container that does the following:
sysctl -w net.ipv4.tcp_keepalive_time=60
sysctl -w net.ipv4.tcp_keepalive_intvl=30

Since introducing the init container every few minutes we get this error:
Traceback (most recent call last):
  File \"/usr/local/lib/python2.7/dist-packages/rediscluster/client.py\", line 625, in _execute_command
  connection.send_command(*args)
  File \"/usr/local/lib/python2.7/dist-packages/redis/connection.py\", line 619, in send_command
    self.send_packed_command(self.pack_command(*args))
  File \"/usr/local/lib/python2.7/dist-packages/redis/connection.py\", line 612, in send_packed_command
    (errno, errmsg))
ConnectionError: Error 32 while writing to socket. Broken pipe.

As far as I see it this error is self contained within the library and doesn't actually cause any issues with our system, please correct me if i'm wrong
We have a few services running in that kubernetes cluster who work with the redis cluster that are not experiencing this issue. The only thing different about the services that is experiencing this issue is that it is running with multi processing.
Any help in resolving this issue would be much appreciated.
Please let me know if there is more information I can provide you with

@NadavkOptimalQ Based on previous issues inside redis-py itself, the problem might be that you have to update the config option health_check_interval into the RedisCluster instance. If that dont work you have to investigate any other keepalive or timings/timeouts that is hardcoded to some value inside redis-py and update them to stop this error from happening. My guess is, that your system has terminated the socket on the system level and not really informed the python level, so when you try to write to the socket your system has terminated it already for any reason.
Also note that this error is not really inside this lib since this lib only wraps around redis-py that actually performs the socket handling etc, and you can see that part in your stack trace where it happens inside redis/connection.py. So unless you find a solution by your own tinkering i will advise you to submit this error up to redis-py repo instead to solve it there, or attempt to submit it at the google groups or stack-overflow for any assistance with this.

Thank you.
Can you tell me how can I set health_check_interval using redis-py-cluster?

@NadavkOptimalQ RedisCluster(health_check_interval=10)

I ran this:
from rediscluster import RedisCluster
nodes =  [
      {
        "host": "<IP_ADDRESS>",
        "port": "7000"
      },
      {
        "host": "<IP_ADDRESS>",
        "port": "7001"
      },
      {
        "host": "<IP_ADDRESS>",
        "port": "7002"
      }
    ]

r = RedisCluster(startup_nodes=nodes, health_check_interval=10)

And I got this error:
  File "/snap/pycharm-community/214/plugins/python-ce/helpers/pydev/pydevd.py", line 1448, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/snap/pycharm-community/214/plugins/python-ce/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/nadav/workspace/sandbox/main.py", line 60, in <module>
    r = RedisCluster(startup_nodes=nodes, health_check_interval=10)
  File "/home/nadav/workspace/sandbox/venv/lib/python3.6/site-packages/rediscluster/client.py", line 217, in __init__
    super(RedisCluster, self).__init__(connection_pool=pool, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'health_check_interval'

Runnin
ubuntu 18.04
python 3.6 with
hiredis            1.0.1
redis              3.0.1
redis-py-cluster   2.1.0

@NadavkOptimalQ That variable was only added in redis 3.3.0 so you need to run that version in order to have access to that variable. https://github.com/andymccurdy/redis-py/blob/b80d423cc531c5db972d35ba72424cf9cf8772ff/CHANGES#L169

This fixed our issue. Thank you very much!
If anyone runs into the same issue we upgraded to new redis version and set health_check_interval=20


SNOW-749361: [Local Testing] Emualte Snowflake datatypes

Please answer these questions before submitting your pull requests. Thanks!

What GitHub issue is this PR addressing? Make sure that there is an accompanying issue to your PR.
Fixes #SNOW-749361

Fill out the following pre-review checklist:

[ ] I am adding a new automated test(s) to verify correctness of my new code
[ ] I am adding new logging messages
[ ] I am adding a new telemetry message
[ ] I am adding new credentials
[ ] I am adding a new dependency

Please describe how your code solves the related issue.
Please write a short description of how your code change solves the related issue.

the test failures are not related to local testing, those should be fixed after rebasing the main branch (specially this PR: https://github.com/snowflakedb/snowpark-python/pull/912).
all local test related tests are passing


Count and say problem in CPP

In a given sequence count the number of the same digit number and print the count of that number and what is that number
Please assign this issue to me I would like to add this question in cpp

/assign


fix(gemfile): unbreaks platform-ios automation

We need to include rexml in the Gemfile in order to unbreak automation on the server. Otherwise, builds fail with things like this.
This also removes a duplicate Gemfile. The original, under example/ios, was introduced by @dcvz . The more recent one, under example/, was introduced by @jspizziri . I've deleted the old one and updated the new one.
fix #1868

Awesome. @dcvz : This will unblock the other PRs that I've got active -- would you mind reviewing and merging so that those others can be unblocked? Thanks!

Updated as requested, @jspizziri!
I spent some time digging around, but I couldn't find the exact toolchain commit that changed this. But I suspect it's within our workflow yaml, where we run with macos-latest. Since we don't use actions/setup-ruby@v1 to lock the CI to Ruby 2.x, we probably inherited the macos-latest Ruby bump to 3.x some time between last Oct and today.
This, BTW, suggests an alternative fix, which is that we lock Ruby to 2.x in the workflow via setup-ruby. LMK if you'd prefer that instead.

@fivecar thanks for the explanation. LGTM.


update admin/base.html for Django 1.8

admin/base.html has changed with Django 1.8
https://github.com/django/django/pull/3762/files

Thanks for reporting. Unfortunately using a new context variable (has_permission) makes it difficult to handle compatibility with django 1.7. I'll try to sort something out.

I think this has been fixed in:
3fca84c0c2fcc7bf91d2ba85d14726ca0182e4bd
Care to double check? Thanks!


Fix memory leak, issue-59

Resets a class variable after each extraction to prevent memory leak. More details in #59 .

@Kdoggett887 thanks for PR! Sorry if the answer to my question is obvious, as I'm new to extruct, but I wonder why is _lookups a class attribute instead of an instance attribute? As far as I understand, if it would be an instance attribute, then it would be cleaned up automatically since we create a new parser in extruct.extract function?

@lopuhin Thanks for the suggestion! I was looking into doing this, but was unsuccessful at first and gave up on it fairly quickly since it caused some upstream issues in the lxml package which I'm unfamiliar with.
After your suggestion, I decided to look into it more and found a solution that puts the _lookups variable into the constructor. When I had tried it before, I wasn't instantiating the HtmlElementClassLookup, so the constructor just instantiates the super class and assigns self._lookups = {}. I'm testing the memory usage on a large sample of files now, but if it works this should be a better solution.

@lopuhin thanks for the feedback! I fixed up the spacing and removed the unused import. I ran a test checking the speed and the updated version took ~85% of the time master did, but more testing would be needed to confirm the results.

Awesome, thanks @Kdoggett887!


osbuild2/rpm: deduplicate rpmStageOptions() and do not import GPG key if repo gpgcheck is disabled

Move rpmStageOptions() from stage_options.go to osbuild2

osbuild2/rpm: do not import GPG keys if repo gpgcheck is disabled

[x] adequate testing for the new functionality or fixed issue

[ ] adequate documentation informing people about the change such as

[ ] submit a PR for the guides repository if this PR changed any behavior described there: https://www.osbuild.org/guides/

I should have marked this as a draft. I still need to regenerate image test cases, because we are not consistent in setting the gpg_check in the repos that we use to generate them. As a result, some GPG keys will be deleted from the image test manifests.
In addition, the second commit changes the behavior only for V2 based distro definitions. Which means that Fedora, RHEL-84 and RHEL-83 keep the behavior of always importing the GPG key. I'm in doubt whether these should be changed as well. And I'm also I'm not sure if the change is actually worth it, since one could as well not include the GPG key in the repo definition with the same effect. I initially thought that it makes sense to be consistent with DNF behavior, but I'm now not sure if it is work the behavior change. What's your opinion @gicmo, @teg ?
Note that I still think that deduplicating rpmStageOptions() makes sense...

I decided to not change the default behavior related to importing GPG keys provided with the repository.
In situation when it is needed to not import the GPG key, one can also not include it in the repo definition, which has the same effect as if the GPG signature check was disabled (the GPG key is not imported).
I'm happy to revisit this in the future, but for now it is IMO not worth it. I submitted the de-duplication change as #2503


Add ethkey functionality as alicenet built int CLI commands

Scope
Add ethkey functionality as alicenet built int CLI commands.
Why?
As an alicenet user, I would like to rely less on external tools, especially very hidden tools like ethkey. Therefore, we should port the generate and possibly other functionalities from ethkey in alicenet built-in CLI commands.

👍


[OSSM-6734] Test coverage for Tempo/Otel integration

Added test for Tempo
Added Minio app and Tempo/Otel installation via OLM
Installation via OLM was generalized and simplified (affects also certmanageroperator)

Ready for code review. I left it as a draft since I need to check if minio/busybox images also work on arm/IBM clusters

Instead of trying random busybox images from the quay, I have mirrored the official one from dockerhub under the maistra private repo https://quay.io/repository/maistra/busybox

/retest

/retest

Added dynamic wait for first check ( since it can take some time to spans are ready )


Some more consistent and correct text

Accouht
The title for error.html is changed to be consistent with the style used in login.html.

nooooo the funny typo


show firewall rules created in Interface Groups on Interfaces tabs

It would be nice to see rules created in Interface Groups (maybe in a different color) on the Interface tabs.

timeout (https://github.com/opnsense/core/commit/7338549ba5f295f5420e45ba74112af10d20c713)


chore(brillig): refactor register allocation [DO NOT MERGE]

Description
This is mainly to be used as inspiration for the function call PR and not merged as-is (although it can be).
Problem*

Resolves
Summary*

Documentation

[ ] This PR requires documentation updates when merged.

[ ] I will submit a noir-lang/docs PR.

[ ] I will request for and support Dev Rel's help in documenting this PR.

Additional Context

PR Checklist*

[ ] I have tested the changes locally.
[ ] I have formatted the changes with Prettier and/or cargo fmt on default settings.

Breaking out some of the ideas here, see my other register abstraction PR


files.catbox.moe ERR_HTTP2_PROTOCOL_ERROR

Any image hosted on files.catbox.moe won't load. ERR_HTTP2_PROTOCOL_ERROR
Image loads on Chrome and other browsers.
Ex: https://files.catbox.moe/cvd34y.jpeg
EDIT: Switching the user agent to a different browser makes the page load correctly. Perhaps catbox is blocking certain versions of chrome on their end?

Have the same issue

Also have the same problem


fix: LogbackConfigTask crash on Gradle 7.0 to 7.4, close gatling/gatling#4394

Motivation:
On Gradle 7.0 to 7.4, the property annotation analysis fails for our task LogbackConfigTask because the logger property isn't annotated as input, output, or internal.
Modifications:

Annotate LogbackConfigTask#logger as internal
Add test with several Gradle versions in WhenRunSimulationSpec

Great job!


chore: deprecate clickedArtistSeriesGroup helper

Description
This PR resolves FX-3345
Deprecation of clickedArtistSeriesGroup helper
This is the cohesion related PR, should be reviewed and merged after this.

#squashOnGreen

awesome! thanks George!


ci(chore): build arm binary in release and docker

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
https://github.com/risingwavelabs/risingwave/issues/14898
Checklist

[ ] I have written necessary rustdoc comments
[ ] I have added necessary unit tests and integration tests
[ ] I have added test labels as necessary. See details.
[ ] I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features #7934).
[ ] My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
[ ] All checks passed in ./risedev check (or alias, ./risedev c)
[ ] My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

[ ] My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

[ ] My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

test pipeline: https://buildkite.com/risingwavelabs/docker/builds/15159

LGTM, thanks!


Plugin requires reconfiguration

Hi,
I have been using the integration successfully for some time, its great.
It now claims it needs reconfiguration on the integrations page, but if I attempt reconfiguration it says it's already configured. How can I remove the warning?
Thanks in advance.

I have the same issue, any ideas how to fix it ?

same issue here

@pabutterworth Hi, did you find solution ?

Same issue here.
Any resolution for this?

Same issue for me.  Any suggestions?

I have the same issue after upgrading Home Assistant Core to 2022.7.0

I had the same issue, resolved with the following procedure:

Removed Sonoff integration
Reboot HA
Added Sonoff integration, when requested in the popup:

in the email input field insert a different email
leave the password empty

The integration will be added with 2 account, the older and the new (fake) one. Remove the new (fake) and the integration will work ok as before.

Please check latest release v3.1.0

Done, the integration was already upgraded to release v3.1.0, I had to follow the procedure above.

Not exactly sure what solved the issue for me:

I removed the integration as per @rdndnl's recommendation,
I ALSO loged-out from the eWeLink app on my phone (had forgotten that previously),
after restarting HASS, I have my Sonoff integration working fine and no more message to "reconfigure" and I did not have to do the rest of @rdndnl's actions.

Same issue here. Any resolution for this?

I have upgraded to 2022.7.0 and SOnoff 3.1.0
While most of my devices have now reappeared, I have a few devices which were working previously and which are now unavailable although they are available in the ewelink app.
Also, I couldnt find the documentation on how to enable multiple homes in the configuration

Can't reproduce problem on latest integration version.
Please reopen the issue if there are still any problems.


Why the status always set to 200 when respond with a gRPC trailers-only scenario

Title: Why the response status always set to 200 when respond with a gRPC trailers-only scenario
Description:
Why is the response status always set to 200 when respond with a gRPC trailers-only scenario ?
https://github.com/envoyproxy/envoy/blob/95038feabf260c3937465951d5da603d31ea3bd4/source/common/http/utility.cc#L538-L548

The gRPC spec states that responses should be http-200, and use grpc-specific headers for the actual grpc status.


use TypeError/ValueError for argument checking errors

https://github.com/mjpost/sacrebleu/issues/189

Sorry this took so long. I'll release it shortly.


Docfx v3 migration

Migration PR to turn docfx build engine to v3

Docs Build status updates of commit 1125b24:
:warning: Validation status: warnings

File
Status
Preview URL
Details

articles/CNTK-Library-Native-Eval-Interface.md
:warning:Warning
View
Details

articles/Setup-Linux-Binary-Manual.md
:warning:Warning
View
Details

articles/Loading-Data-Overview.md
:warning:Warning
View
Details

articles/Unary-Operations.md
:warning:Warning
View
Details

articles/Archive/CNTK-Shared-Libraries-Naming-Format.md
:warning:Warning
View
Details

articles/Tutorial2/Tutorial2.md
:warning:Warning
View
Details

articles/Tutorial/Tutorial.md
:warning:Warning
View
Details

articles/Setup-Windows-Binary-Manual.md
:warning:Warning
View
Details

articles/CNTK-CSharp-Examples.md
:warning:Warning
View
Details

articles/AAAI-2018-Tutorial.md
:warning:Warning
View
Details

.openpublishing.publish.config.json
:white_check_mark:Succeeded

articles/breadcrumb/toc.yml
:white_check_mark:Succeeded

articles/CNTK-Library-Native-Eval-Interface.md

Line 8: [Warning-ms-topic-invalid] Invalid value for 'ms.topic': 'get-started-article'

articles/Setup-Linux-Binary-Manual.md

Line 2: [Warning-ms-prod-or-service-missing] One of the following attributes is required: 'ms.prod', 'ms.service'. Use ms.prod for on-premise products, or ms.service for cloud services.

articles/Loading-Data-Overview.md

Line 2: [Warning-author-missing] Missing required attribute: 'author'. Add the current author's GitHub ID.

articles/Unary-Operations.md

Line 2: [Warning-author-missing] Missing required attribute: 'author'. Add the current author's GitHub ID.
Line 2: [Warning-title-missing] Missing required attribute: 'title'

articles/Archive/CNTK-Shared-Libraries-Naming-Format.md

Line 8: [Warning-ms-topic-invalid] Invalid value for 'ms.topic': 'get-started-article'

articles/Tutorial2/Tutorial2.md

Line 8: [Warning-ms-service-subservice-invalid] Invalid value for 'ms.service': 'Cognitive-services'

articles/Tutorial/Tutorial.md

Line 8: [Warning-ms-service-subservice-invalid] Invalid value for 'ms.service': 'Cognitive-services'

articles/Setup-Windows-Binary-Manual.md

Line 2: [Warning-ms-prod-or-service-missing] One of the following attributes is required: 'ms.prod', 'ms.service'. Use ms.prod for on-premise products, or ms.service for cloud services.

articles/CNTK-CSharp-Examples.md

Line 4: [Warning-ms-author-invalid] Invalid value for 'ms.author', 'liqun' is not a valid Microsoft alias

articles/AAAI-2018-Tutorial.md

Line 4: [Warning-ms-author-invalid] Invalid value for 'ms.author', 'yuqtang' is not a valid Microsoft alias
Line 63: [Warning-insecure-link] Link 'http://notebooks.azure.com/cntk/libraries/tutorials' is insecure. Links to Microsoft sites must use 'https' instead of 'http'.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel


[SPARK-50001][PYTHON][PS][CONNECT] Adjust "precision" to be part of kwargs for box plots

What changes were proposed in this pull request?
Adjust "precision" to be kwargs for box plots in both Pandas on Spark and PySpark.
Why are the changes needed?
Per discussion here (https://github.com/apache/spark/pull/48445#discussion_r1804042377), precision is Spark-specific implementation detail, so we wanted to keep “precision” as part of kwargs for box plots.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Existing tests.
Was this patch authored or co-authored using generative AI tooling?
No.

@xinrong-meng please rebase to enable a recent test image refactor

Rebased, thank you!

Merged to master, thank you!


[Issue #471] Update Angular GUI to make it prettier and easier to use

This PR modifies some visual part of the new angular GUI:

the operator bar, sidebar, and main panel have different colors
the "Run" button is bigger and more obvious
the titles and orders of the operators are adjusted to make them more user friendly
some uncommon attributes are now hidden
website title is changed from "Angular quickstart" to "TextDB"


Release 22.04.2022

IREC fixes mostly

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.1 out of 2 committers have signed the CLA.:white_check_mark: soanvig:x: github-actions[bot]You have signed the CLA already but the status is still pending? Let us recheck it.


Add CrossValidator transformer

Not sure if we actually want to implement this. Let's think on it a minute before we do

We aren't going to add this model as serializable, closing for now.


Made a few changes here and there

If you want to contribute make a flask server which directly sends the audio instead of using the NodeJs Express server


make-dbus-conf: prefer local build

Motivation for this change
I use remote builders to upgrade my laptop (nix.buildMachines option), and after a mass rebuild the final step that takes a lot of time looks like this (for the brevity of the example, I've interrupted the last step before running the switch again):
# nixos-rebuild switch
building Nix...
building the system configuration...
these derivations will be built:
  /nix/store/sacraxp35l3k0mcz92bmisf7rrqvxbdf-dbus-1.drv
  /nix/store/5ny75i0xcwic0plq30jrpjax9jfx5d5i-unit-dbus.service.drv
  /nix/store/2rwjbklbxzsgl6s6mnyrvj34svgdkyxk-system-units.drv
  /nix/store/s3p5db30hjp1aw847ml5q7xmsrrhr671-user-units.drv
  /nix/store/yxmxcclkchff3x18srpdz6h0aybb5qsz-etc.drv
  /nix/store/r8yw56k71dpn8wq30jp539kk5v7qaf4f-nixos-system-orivej-18.03.git.f575062.drv
copying 500 missing paths (5000 MiB) to<EMAIL_ADDRESS>
All of this derivations prefer local build except dbus-1. If I enable it for dbus-1, the switch finishes instantaneously. Previously I had to rerun the switch with --no-build-hook when I wanted it to finish fast.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[x] Fits CONTRIBUTING.md.

lgtm


Microsoft.EntityFrameworkCore.SqlServer in Class Library - RC2

I'm having trouble getting Microsoft.EntityFrameworkCore.SqlServer working within a class library.  It is telling me that IX-Async and Remotion.Linq are not compatible with .NETStandard,Version=v1.5.
This is very easy to reproduce.  I just installed RC2.  Created a new project and chose Class Library (.NET Core).  Then installed Microsoft.EntityFrameworkCore.SqlServer rc2 via nuget, and get a compiler error.
Any ideas on how to get around this?
project.json
{
  "version": "1.0.0-*",

  "dependencies": {
    "Microsoft.EntityFrameworkCore.SqlServer": "1.0.0-rc2-final",
    "NETStandard.Library": "1.5.0-rc2-24027"
  },

  "frameworks": {
    "netstandard1.5": {
      "imports": "dnxcore50"
    }
  }
}

Try changing your imports to look like this:
      "imports": [
        "dnxcore50",
        "portable-net452+win81"
      ]

That worked thank you very much!
What exactly is adding that doing?  I'm assuming this is kind of binding me to Windows (which is fine) given that sql server only works on windows.

Imports can be used to allow packages that use .NET Standard to interoperate with packages that were originally built against other targets. We are working with the owners of IX-Async and Remotion.Linq so that they can add .NET Standard support in their packages, after which this workaround should become unnecessary.

sql server only works on windows

Not really :smile: The .NET Core version of System.Data.SqlClient works cross-platform and even SQL Server on Linux is coming https://www.microsoft.com/en-us/server-cloud/sql-server-on-linux.aspx

FYI the issue tracking this is https://github.com/aspnet/EntityFramework/issues/5176

@divega You should probably mention this import thing in the upgrading manual.

@gdoron - @natemcmaster added some info to that article.

Hi ,
I am trying to upgrade my project to RC2.
In my previous project.json, i was able to refer gac assemblies using the frameworkassemblies section. But after upgrading to RC2, assemblies in gac are not getting recognized.
Even i tried using the import , but no luck.
Can anyone help me ?
sample project.json :
"frameworks": {
"net451": {
"imports": [ "portable-net452+win81" ],
"frameworkAssemblies": {
"gac-assmebly_name ": {
"version": "<IP_ADDRESS>",
"type": "build",
},
I am getting the below error
The dependency gac-assmebly_name >= <IP_ADDRESS> could not be resolved .
I have verified that the assembly is present in the GAC.

GAC assembly references are not supported in the .NET CLI. File an issue here https://github.com/dotnet/cli

we have an assembly which because of various dependencies, can not be created as package.
So what is the work around or alternative to referring the assembly present in GAC ?

You can't reference it from the GAC for compilation so you'd need to put it in a nuget package

@davidfowl As of now nuget package is not possible, do  have any other way to do it ?

Thanx  @divega  sir for your post of import to use entity framework in class library
Now i m facing the problem
Microsoft.EntityFrameworkCore.Tools in class library
This preview of Entity Framework tools does not support targeting class library projects in ASP.NET Core and .NET Core applications.
Need Help

I had the same issue, I found that the name had been changed,
"Microsoft.EntityFrameworkCore.Tools.Core": "1.0.0-rc2-final",
instead of the original EntityFrameworkCore.Tools

@ermithun "Microsoft.EntityFrameworkCore.Tools does not support .NET Core class libraries. See https://github.com/aspnet/EntityFramework/issues/5320.
@ofer the class library was in RC1 EntityFramework.Commands and we renamed it to Microsoft.EntityFrameworkCore.Tools version "1.0.0-preview1-final" for RC2. See https://docs.efproject.net/en/latest/miscellaneous/rc1-rc2-upgrade.html#package-names-and-versions

What is the relationship between Microsoft.EntityFrameworkCore.Tools.Core and Microsoft.EntityFrameworkCore.Tools
Are they different libraries or does one supersede the other?
Is the documentation found here current and correct: https://docs.efproject.net/en/latest/miscellaneous/rc1-rc2-upgrade.html#package-names-and-versions

@sam-wheat Microsoft.EntityFrameworkCore.Tools is the package you should install. Microsoft.EntityFrameworkCore.Tools.Core is some infrastructure used by the tooling and is a dependency of the other package.

@rowanmiller Hi, I am using EF Core Code First, and I am trying to move my DbContext in separate class library. But the problem is I cannot Add-Migrations there. When I try to install Microsoft.EntityFrameworkCore.Tools it says "The dependency Microsoft.EntityFrameworkCore.Tools 1.0.0-preview1-final does not support framework .NETStandard, Version=v1.5".
This is my project.json to the class library:
`{
"version": "1.0.0-*",
"dependencies": {
"NETStandard.Library": "1.5.0-rc2-24027",
"Npgsql.EntityFrameworkCore.PostgreSQL": "1.0.0-rc2-release1",
"Microsoft.EntityFrameworkCore": "1.0.0-rc2-final",
"Microsoft.EntityFrameworkCore.Tools": "1.0.0-preview1-final", ---> shows error
},
"tools": {
"Microsoft.EntityFrameworkCore.Tools": {
"version": "1.0.0-preview1-final",
"imports": [
"portable-net452win81+dnxcore50",
"portable-net452+win81"
]
}
},
"frameworks": {
  "netstandard1.5": {
    "imports": [
      "dnxcore50",
      "portable-net452+win81"
    ]
  }
}

}`

But the problem is I cannot Add-Migrations there.
I hope this one is fixed or high on the to-do list.

See also: http://benjii.me/2016/05/dotnet-ef-migrations-for-asp-net-core/

@dbrezoev @sam-wheat you are hitting https://github.com/aspnet/EntityFramework/issues/5320, there are details of workarounds etc. at https://docs.efproject.net/en/latest/cli/dotnet.html#targeting-class-library-projects-is-not-supported


Set config var for running on different execution contexts

e.g., spark-ec2 scripts, director cluster, or local machine

Closing for cleanup/refactor.


Error during build

Hi,
I got this error after building ImageViewer on my Ubuntu 20.04 system and using "make".
(base) nicholas:~/dev/ImageViewer-BUILD$ make
Consolidate compiler generated dependencies of target QtImageViewer
[  7%] Building CXX object QtImageViewer/CMakeFiles/QtImageViewer.dir/RulerWidget.cxx.o
/home/nicholas/dev/ImageViewer/QtImageViewer/RulerWidget.cxx: In member function ‘void RulerToolCollection::handleMouseEvent(QMouseEvent*, double*)’:
/home/nicholas/dev/ImageViewer/QtImageViewer/RulerWidget.cxx:203:83: error: cannot bind non-const lvalue reference of type ‘RulerTool::PointType3D&’ {aka ‘itk::Point<double, 3>&’} to an rvalue of type ‘RulerTool::PointType3D’ {aka ‘itk::Point<double, 3>’}
  203 | ue_ptr< RulerTool > r(new RulerTool(this->parent, RulerTool::PointType3D(index), metaDataFactory->getNext()));
      |                                                              ^~~~~~~~~~~~~~~~~~

/home/nicholas/dev/ImageViewer/QtImageViewer/RulerWidget.cxx:64:59: note:   initializing argument 2 of ‘RulerTool::RulerTool(QtGlSliceView*, RulerTool::PointType3D&, std::unique_ptr<RulerToolMetaData>)’
   64 | RulerTool::RulerTool(QtGlSliceView* parent0, PointType3D &index0, std::unique_ptr< RulerToolMetaData > metaData) :
      |                                              ~~~~~~~~~~~~~^~~~~~
make[2]: *** [QtImageViewer/CMakeFiles/QtImageViewer.dir/build.make:116: QtImageViewer/CMakeFiles/QtImageViewer.dir/RulerWidget.cxx.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:118: QtImageViewer/CMakeFiles/QtImageViewer.dir/all] Error 2
make: *** [Makefile:136: all] Error 2
(base) nicholas:~/dev/ImageViewer-BUILD$

Does anyone know how I can resolve this? Thanks!

I think it wants the RulerTool constructor to define that as a const reference, but I'll dig into it.  The reason why you are seeing the error is Visual Studio (which I use) is a bit more lax than gcc by default.


Bump ruby to 2.5.6

It is already like that in the docker image.

The Docker image actually has 2.5.5p157.
docker pull quay.io/fundingcircle/shipment_tracker
docker run --rm  quay.io/fundingcircle/shipment_tracker ruby -v

replaced by https://github.com/FundingCircle/shipment_tracker/pull/430


Implement is (rfc 5)

Implements the is operation as described in rfc #5.
Checklist for requesting a review
The change in this PR is (choose one, and delete the other options):

[x] A breaking change requiring a major version bump to cedar-policy (e.g., changes to the signature of an existing API).

I confirm that this PR (choose one, and delete the other options):

[x] Updates the "Unreleased" section of the CHANGELOG with a description of my change (required for major/minor version bumps).

I confirm that cedar-spec (choose one, and delete the other options):

[x] Requires updates, and I have made / will make these updates myself. (Please include in your description a timeline or link to the relevant PR in cedar-spec, and how you have tested that your updates are correct.)

Disclaimer
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Reviewers should feel free to limit their their review to the portions of this PR that they are familiar with. In adding a new language feature it touches everything.
e.g.:

@cdisselkoen est changes
@shaobo-he-aws formatting changes
@khieta validation changes

Similar to the has operator, what error messages should Cedar produce for expressions like principal is if. Currently, they are different from that of principal is like.
echo 'permit(principal is if, action, resource);' | cargo run check-parse
    Finished dev [unoptimized + debuginfo] target(s) in 0.17s
     Running `target/debug/cedar check-parse`
Error: cedar_policy_core::parser::to_cst_error

  × failed to parse policy set
  ╰─▶ unexpected token `,`
   ╭─[<stdin>:1:1]
 1 │ permit(principal is if, action, resource);
   ·                       ┬
   ·                       ╰── expected `::`
   ╰────

Error: cedar_policy_core::parser::to_cst_error

  × unexpected token `,`
   ╭─[<stdin>:1:1]
 1 │ permit(principal is if, action, resource);
   ·                               ┬
   ·                               ╰── expected `(`
   ╰────
Error: cedar_policy_core::parser::to_cst_error

  × unexpected token `)`
   ╭─[<stdin>:1:1]
 1 │ permit(principal is if, action, resource);
   ·                                         ┬
   ·                                         ╰── expected `(`
   ╰────

echo 'permit(principal is like, action, resource);' | cargo run check-parse
    Finished dev [unoptimized + debuginfo] target(s) in 0.17s
     Running `target/debug/cedar check-parse`
Error: cedar_policy_core::parser::to_ast_err

  × failed to parse policy set
  ╰─▶ this identifier is reserved and cannot be used: `like`

These seem to be consistent for has.
cedar % echo 'permit(principal, action, resource) when { principal has if };' | cargo run check-parse
    Finished dev [unoptimized + debuginfo] target(s) in 0.17s
     Running `target/debug/cedar check-parse`
Error: cedar_policy_core::parser::to_ast_err

  × failed to parse policy set
  ╰─▶ this identifier is reserved and cannot be used: `if`

echo 'permit(principal, action, resource) when { principal has like };' | cargo run check-parse
    Finished dev [unoptimized + debuginfo] target(s) in 0.17s
     Running `target/debug/cedar check-parse`
Error: cedar_policy_core::parser::to_ast_err

  × failed to parse policy set
  ╰─▶ this identifier is reserved and cannot be used: `like`

Similar to the has operator, what error messages should Cedar produce for expressions like principal is if?

@shaobo-he-aws has does some special handling for if here https://github.com/cedar-policy/cedar/blob/main/cedar-policy-core/src/parser/grammar.lalrpop#L183 . I'll have to look at that closer.
It also seems like like has a similar problem
jkastner@88665a03aa97 cedar-policy-core % echo 'permit(principal, action, resource) when { principal like if};' | cargo run -p cedar-policy-cli check-parse
    Finished dev [unoptimized + debuginfo] target(s) in 0.64s
     Running `/Users/jkastner/cedar/cedar/target/debug/cedar check-parse`
Error: cedar_policy_core::parser::to_cst_error

  × failed to parse policy set
  ╰─▶ unexpected token `}`
   ╭─[<stdin>:1:1]
 1 │ permit(principal, action, resource) when { principal like if};
   ·                                                             ┬
   ·                                                             ╰── expected `::`
   ╰────

jkastner@88665a03aa97 cedar-policy-core % echo 'permit(principal, action, resource) when { principal like like};' | cargo run -p cedar-policy-cli check-parse
    Finished dev [unoptimized + debuginfo] target(s) in 0.44s
     Running `/Users/jkastner/cedar/cedar/target/debug/cedar check-parse`
Error: cedar_policy_core::parser::to_ast_err

  × failed to parse policy set
  ╰─▶ this identifier is reserved and cannot be used: `like`

@shaobo-he-aws has does some special handling for if here https://github.com/cedar-policy/cedar/blob/main/cedar-policy-core/src/parser/grammar.lalrpop#L183 . I'll have to look at that closer.

Yes, this is exactly what I wanted to refer to.

It also seems like like has a similar problem

Good catch. We then need to decide what we should do generally.

Opened #407 for the parse error inconsistency Shaobo observed. The same affects most expressions one way or another, so I won't try to make a special case fix for is here.


Often returned feeds are not the last published on network

Context
I've different nodes, I can publish a feed update on the first, and require from the second.
Summary
Often the feed index changes based on the node that receive the request.
Expected behavior
If I ask for the last, it should be always the last
Actual behavior
Having load balanced nodes, requiring the feed can produce unexpected behaviors.
Steps to reproduce
Try to push several feeds updates on a node, and try to receive from the receiving one, and another
Possible solution
Don't know.

A possible solution could be to pass to bee an optional parameter with a minimum feed index to find.
If I know (me, the client) that an already seen feed index was at least a specific value, I also know that every response with the index less than it is invalid. And I can pass this information to bee.
In this way bee can look for any index >= to it, and return 404 if doesn't found anything. That is a better response than an invalid one.
If I don't pass this parameter, any feed-index response is valid, as now.

It is possible to make this change in the client side if you are using bee-js. Looking up the feed update is just using the index that is returned and the topic of the single owner chunk is calculated based on that.
Also as a workaround I may suggest to use a persistent cookie for the loadbalancer to return, so that subsequent requests in the same session would be served by the same server in the loadbalancer. This could mitigate the data inconsistencies which may happen during reads, but this gets especially important when writing, otherwise it's possible to end up with different single -owner chunks for the same address.

Yes, writing is already addressed with the postage batch, that will find the exact node owning it.
We need to find a solution with downloads, but I need to get more into how bee-js works.

It is possible to make this change in the client side if you are using bee-js. Looking up the feed update is just using the index that is returned and the topic of the single owner chunk is calculated based on that.

I think the problem here is that the feed index returned from one node is not the most recent, so I don't think it is possibile to resolve this using bee-js

It is possible to make this change in the client side if you are using bee-js. Looking up the feed update is just using the index that is returned and the topic of the single owner chunk is calculated based on that.

I think the problem here is that the feed index returned from one node is not the most recent, so I don't think it is possibile to resolve this using bee-js

The proposal from @tmm360 was to pass an optional parameter with a minimum feed index to find. and if the returned index is less than that ignore that. I wrote answering to this that this is possible to do with bee-js / on the client side if it uses the feed API and not the bzz API.
Actually the complete feed lookup algorithm is possible to do in bee-js, only by using the /soc endpoints.

Unfortunately it's difficult to use only bee-js in our situation.
We provide a gateway that can see traffic through it. It can store for each returned feed it's maximum returned index, and can inject that value for each new request as I suggested.
Instead, in a case of a pure bee-js use, the final user requests a feed, and we don't do any request modification.
Problem is that user, opening the page, can't know what the last feed-index was, so can't request it specifically. We don't have any way to inform users on last feed indexes, except to provide it with a specific query on our gateway, but it's not a very elegant solution.

We was using feeds wrong. Closing this now, will open again if we will found more specific issues.


it appears that the character '>' is filtered by the back-end server. ghauri will based data retrieval on BETWEEN operator

how to use tamper in ghauri for bypass WAF ??
and i have problem whene i retreive data "[WARNING] invalid character detected, retrying.." and "it appears that the character '>' is filtered by the back-end server. ghauri will based data retrieval on BETWEEN operator"
what sollution?

Ghauri don't have tamper support so no way to use tamper script. The warning you are facing is a handle to incorrect character so this is not an issue but a solution to avoid false characters. When > blocked by firewall it switches automatically to between and then assignment operator (=) to bypass firewall.
So all in all what you are facing is not an issue but a handle to an issue and no tamper support as of yet. Will add support in near future but don't know the exact time line as I work on Ghauri in my free times.


Sigma transaction size not being calculated correctly by coin control

Bug reported by @goodc0re on Telegram
Actual behaviour
Coin control is calculating the size of the Sigma spend transaction at 226 bytes and estimating fees based on that.
Actual is 1.4 kB which is calculated correctly on the confirmation screen.

Desired behaviour
Coin control for Sigma should calculate it based on Sigma spend size not regular transparent transactions.
Steps to reproduce

Go to coin control, inputs, select a mint.
Go to spend, enter in an address and select the whole amount of the mint. Make sure Subtract fee from amount is selected.
See the difference between Coin Control estimation and actual fee.

@levonpetrosyan93 @reubenyap
Verified. Can be closed
1)

2)

Waiting requested changes. And than re-check it again

pushed to 13.8.4


In multinode training with ddp each node duplicates logs and has node_rank=0

🐛 Bug
When training using ddp in a multi-node environment with seed_everything(workers=True) there are identical loss values logged on each node. For example this occurs in a 3 node environment with limit_val_batches=2 (logged via mlflow):

The NODE_RANK of each node is 0 (which shouldn't be the case) and identical loss values also appear in each node's process when printed to the screen so it appears each node is using the same data in the same order. One fix I attempted was to set NODE_RANK to OMPI_COMM_WORLD_RANK, but this resulted in the job hanging or crashing.
To Reproduce
Download the two python scripts below (edit as needed to run on a multinode multi-gpu Azure cluster) and then run python submit_job.py. AzureML is used to submit the job with an MPIConfiguration.
submit_job.py:
import os
from azureml.core.runconfig import PyTorchConfiguration, MpiConfiguration
from azureml.core import Environment, Workspace, ScriptRunConfig, Experiment
from azureml.core.authentication import ServicePrincipalAuthentication

experiment_name = "XXXXXXXXX"
environment_name = "XXXXXXXX"
cluster_name="XXXXXXX"
node_count=3
environment_version="XXXXXXX"

command="; ".join([
    "python -m pip freeze",
    "wget https://raw.githubusercontent.com/Lightning-AI/lightning/master/requirements/collect_env_details.py",
    "python collect_env_details.py",
    "python logging_reprex.py"
])

print(command)

ws = Workspace.get(
    name="XXXXXXXXX",
    resource_group="XXXXXXXXXX",
    subscription_id=os.environ["AZURE_SUBSCRIPTION_ID"],
    auth= ServicePrincipalAuthentication(
            tenant_id=os.environ.get("AZURE_TENANT_ID"),
            service_principal_id=os.environ.get("AZURE_CLIENT_ID"),
            service_principal_password=os.environ.get("AZURE_CLIENT_SECRET")
        )
    )

compute_target=ws.compute_targets[cluster_name]

env = Environment.get(
    workspace = ws,
    name = environment_name,
    version = environment_version
    )

print(env)
print(env.get_image_details(ws))

job_config = MpiConfiguration(node_count=node_count, process_count_per_node=1)

src = ScriptRunConfig(
    source_directory = ".",
    command=command,
    compute_target = compute_target,
    environment = env,
    distributed_job_config = job_config
)

# Submit job
run = Experiment(ws, experiment_name).submit(src)

logging_reprex.py:
import numpy as np # fix for https://github.com/pytorch/pytorch/issues/37377
import torch, os, logging, sys
from torch.utils.data import DataLoader, Dataset
#from deepspeed.ops.adam import FusedAdam
from azureml.core import Run, Workspace
from pytorch_lightning import LightningModule, Trainer, LightningDataModule, seed_everything
from pytorch_lightning.loggers import MLFlowLogger

divider_str="-"*40

def get_env_display_text(var_name):
    var_value = os.environ.get(var_name, "")
    return f"{var_name} = {var_value}"

def display_environment(header='Environmental variables'):
    """
    Print a few environment variables of note
    """
    variable_names = [
        "PL_GLOBAL_SEED",
        "PL_SEED_WORKERS",
        "AZ_BATCH_MASTER_NODE",
        "AZ_BATCHAI_MPI_MASTER_NODE",
        "MASTER_ADDR",
        "MASTER_ADDRESS",
        "MASTER_PORT",
        "RANK",
        "NODE_RANK",
        "LOCAL_RANK",
        "GLOBAL_RANK",
        "WORLD_SIZE",
        "NCCL_SOCKET_IFNAME",
        "OMPI_COMM_WORLD_RANK",
        "OMPI_COMM_WORLD_LOCAL_RANK",
        "OMPI_COMM_WORLD_SIZE",
        "OMPI_COMM_WORLD_LOCAL_SIZE"
    ]
    var_text = "\n".join([get_env_display_text(var) for var in variable_names])
    print(f"\n{header}:\n{divider_str}\n{var_text}\n{divider_str}\n")

def get_run():
    """
    Returns active run azureml object
    """
    run = Run.get_context()
    return run

class RandomDataset(Dataset):
    def __init__(self, size, length):
        self.len = length
        self.data = torch.randn(length, size)

    def __getitem__(self, index):
        return self.data[index]

    def __len__(self):
        return self.len

class BoringModel(LightningModule):
    def __init__(self):
        super().__init__()
        self.save_hyperparameters()
        self.model = torch.nn.Linear(32, 2)

    def forward(self, x):
        return self.model(x)

    def training_step(self, batch, batch_idx):
        loss = self(batch).sum()
        self.log("train_loss", loss, on_step=True, on_epoch=True)
        return {"loss": loss}

    def validation_step(self, batch, batch_idx):
        loss = self(batch).sum()
        self.log("valid_loss", loss)
        print(f"""self.global_step: {self.global_step}
gpu #: {torch.cuda.current_device()}
batch_idx: {batch_idx}
loss: {loss}
""")

    def configure_optimizers(self):
        return torch.optim.AdamW(self.model.parameters())
        #return FusedAdam(self.model.parameters())

    def setup(self, stage=None) -> None:

        # prevents hanging
        if stage != "fit":
            return
        display_environment(f"LightningModule setup(stage={stage})")

class DataModule(LightningDataModule):
    def __init__(self):
        super().__init__()
        self.num_workers = os.cpu_count()
        print(f"num_workers set to {self.num_workers}")

    def setup(self, stage=None) -> None:
        display_environment(f"DataModule setup(stage={stage})")
        self._dataloader = DataLoader(
            RandomDataset(32, 64),
             num_workers=self.num_workers,
             batch_size=1,
             pin_memory=True
             )

    def train_dataloader(self):
        return self._dataloader

    def test_dataloader(self):
        return self._dataloader

    def val_dataloader(self):
        return self._dataloader

if __name__ == "__main__":

    # Toggle comment here to make logged loss values identical (uncommented) or unique (commented)
    seed_everything(102938, workers = True)

    display_environment("__main__")
    model = BoringModel()
    dm = DataModule()

    # If running on azure, get the active tracking uri and run id
    # else, use the workspace to get a uri
    active_run = get_run()
    offline = False
    try:
        print(active_run.experiment)
    except:
        offline = True

    if offline:
        print("Running offline...")
        run_id = None
        ws = Workspace.get(
            subscription_id=os.environ.get("AZURE_SUBSCRIPTION_ID"),
            resource_group="XXXXXXX",
            name="XXXXXXXX",
        )
        tracking_uri = ws.get_mlflow_tracking_uri()
    else:
        tracking_uri =active_run.experiment.workspace.get_mlflow_tracking_uri()
        run_id = active_run.id

    logger = MLFlowLogger(
                experiment_name='reprex',
                tracking_uri=tracking_uri,
                run_id=run_id
    )

    trainer = Trainer(
        accelerator='gpu',
        auto_select_gpus=True,
        limit_train_batches=2,
        limit_val_batches=2,
        log_every_n_steps=1,
        logger=logger,
        enable_checkpointing=False,
        num_sanity_val_steps=0,
        max_epochs=2,
        enable_model_summary=False,
        strategy="ddp",
        precision=16
    )

    trainer.fit(model, datamodule=dm)

Expected behavior
Nodes should load different data from their dataloaders and should not duplicate logged loss values.
Environment

3 node Azure compute cluster
Each node has 2x V100 GPUs and the base docker image for the environment is mcr.microsoft.com/azureml/openmpi4.1.0-cuda11.3-cudnn8-ubuntu20.04

* CUDA:
	- GPU:
		- Tesla V100-PCIE-16GB
		- Tesla V100-PCIE-16GB
	- available:         True
	- version:           11.3
* Packages:
	- numpy:             1.22.3
	- pyTorch_debug:     False
	- pyTorch_version:   1.11.0
	- pytorch-lightning: 1.6.4
	- tqdm:              4.64.0
* System:
	- OS:                Linux
	- architecture:
		- 64bit
		- ELF
	- processor:         x86_64
	- python:            3.8.13
	- version:           #38-Ubuntu SMP Sun Mar 22 21:27:21 UTC 2020

Environmental variables are printed to the log in the python script at various points in the process. Below are the values of these environmental variables on each node before the LightningModule and LightningDataModule objects are initiated (ie. the display_environment("__main__") call in the script above).
Node 1
__main__:
----------------------------------------
PL_GLOBAL_SEED = 102938
PL_SEED_WORKERS = 1
MASTER_ADDR = <IP_ADDRESS>
MASTER_ADDRESS =
MASTER_PORT = 38581
RANK =
NODE_RANK = 0
LOCAL_RANK = 1
GLOBAL_RANK =
WORLD_SIZE = 2
NCCL_SOCKET_IFNAME = eth0
OMPI_COMM_WORLD_RANK = 0
OMPI_COMM_WORLD_LOCAL_RANK = 0
OMPI_COMM_WORLD_SIZE = 3
OMPI_COMM_WORLD_LOCAL_SIZE = 1
----------------------------------------

Node 2
__main__:
----------------------------------------
PL_GLOBAL_SEED = 102938
PL_SEED_WORKERS = 1
MASTER_ADDR = <IP_ADDRESS>
MASTER_ADDRESS =
MASTER_PORT = 57987
RANK =
NODE_RANK = 0
LOCAL_RANK = 1
GLOBAL_RANK =
WORLD_SIZE = 2
NCCL_SOCKET_IFNAME = eth0
OMPI_COMM_WORLD_RANK = 1
OMPI_COMM_WORLD_LOCAL_RANK = 0
OMPI_COMM_WORLD_SIZE = 3
OMPI_COMM_WORLD_LOCAL_SIZE = 1
----------------------------------------

Node 3
__main__:
----------------------------------------
PL_GLOBAL_SEED = 102938
PL_SEED_WORKERS = 1
MASTER_ADDR = <IP_ADDRESS>
MASTER_ADDRESS =
MASTER_PORT = 46819
RANK =
NODE_RANK = 0
LOCAL_RANK = 1
GLOBAL_RANK =
WORLD_SIZE = 2
NCCL_SOCKET_IFNAME = eth0
OMPI_COMM_WORLD_RANK = 2
OMPI_COMM_WORLD_LOCAL_RANK = 0
OMPI_COMM_WORLD_SIZE = 3
OMPI_COMM_WORLD_LOCAL_SIZE = 1
----------------------------------------

cc @borda @awaelchli @rohitgr7 @akihironitta @justusschock @kaushikb11 @ananthsub @carmocca @edward-io @kamil-kaczmarek @Raalsky @Blaizzy

RANK =

RANK is None on all the cases?

RANK is None on all the cases?

Yes that's correct.

you didn't specify num_nodes in Trainer?

That's correct. I also realized I need to set the process count per node to the number of gpus per node in the MPIConfiguration. So I made these two parameter adjustments to the scripts:

Trainer(num_nodes=3, auto_select_gpus=False, devices=2)
MpiConfiguration(process_count_per_node=2) # num gpus per node

I now have 6 driver logs (3 nodes x 2 gpus per node) whereas before I only had 3 driver logs (1 per node).  However, each driver log now hangs on this message:
Global seed set to 102938
Initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/6
Global seed set to 102938
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/6

Below are the environmental variable printouts for each driver log. NODE_RANK=0 and LOCAL_RANK=1 for all of the drivers. Master port is also a different value for each driver.
Driver 0
__main__:
----------------------------------------
PL_GLOBAL_SEED = 102938
PL_SEED_WORKERS = 1
AZ_BATCH_MASTER_NODE = <IP_ADDRESS>:6000
AZ_BATCHAI_MPI_MASTER_NODE = <IP_ADDRESS>
MASTER_ADDR = <IP_ADDRESS>
MASTER_ADDRESS =
MASTER_PORT = 47057
RANK =
NODE_RANK = 0
LOCAL_RANK = 1
GLOBAL_RANK =
WORLD_SIZE = 6
NCCL_SOCKET_IFNAME = eth0
OMPI_COMM_WORLD_RANK = 0
OMPI_COMM_WORLD_LOCAL_RANK = 0
OMPI_COMM_WORLD_SIZE = 6
OMPI_COMM_WORLD_LOCAL_SIZE = 2
----------------------------------------

Driver 1
__main__:
----------------------------------------
PL_GLOBAL_SEED = 102938
PL_SEED_WORKERS = 1
AZ_BATCH_MASTER_NODE = <IP_ADDRESS>:6000
AZ_BATCHAI_MPI_MASTER_NODE = <IP_ADDRESS>
MASTER_ADDR = <IP_ADDRESS>
MASTER_ADDRESS =
MASTER_PORT = 59425
RANK =
NODE_RANK = 0
LOCAL_RANK = 1
GLOBAL_RANK =
WORLD_SIZE = 6
NCCL_SOCKET_IFNAME = eth0
OMPI_COMM_WORLD_RANK = 1
OMPI_COMM_WORLD_LOCAL_RANK = 1
OMPI_COMM_WORLD_SIZE = 6
OMPI_COMM_WORLD_LOCAL_SIZE = 2
----------------------------------------

Driver 2
__main__:
----------------------------------------
PL_GLOBAL_SEED = 102938
PL_SEED_WORKERS = 1
AZ_BATCH_MASTER_NODE = <IP_ADDRESS>:6000
AZ_BATCHAI_MPI_MASTER_NODE = <IP_ADDRESS>
MASTER_ADDR = <IP_ADDRESS>
MASTER_ADDRESS =
MASTER_PORT = 48317
RANK =
NODE_RANK = 0
LOCAL_RANK = 1
GLOBAL_RANK =
WORLD_SIZE = 6
NCCL_SOCKET_IFNAME = eth0
OMPI_COMM_WORLD_RANK = 2
OMPI_COMM_WORLD_LOCAL_RANK = 0
OMPI_COMM_WORLD_SIZE = 6
OMPI_COMM_WORLD_LOCAL_SIZE = 2
----------------------------------------

Driver 3
__main__:
----------------------------------------
PL_GLOBAL_SEED = 102938
PL_SEED_WORKERS = 1
AZ_BATCH_MASTER_NODE = <IP_ADDRESS>:6000
AZ_BATCHAI_MPI_MASTER_NODE = <IP_ADDRESS>
MASTER_ADDR = <IP_ADDRESS>
MASTER_ADDRESS =
MASTER_PORT = 35447
RANK =
NODE_RANK = 0
LOCAL_RANK = 1
GLOBAL_RANK =
WORLD_SIZE = 6
NCCL_SOCKET_IFNAME = eth0
OMPI_COMM_WORLD_RANK = 3
OMPI_COMM_WORLD_LOCAL_RANK = 1
OMPI_COMM_WORLD_SIZE = 6
OMPI_COMM_WORLD_LOCAL_SIZE = 2
----------------------------------------

Driver 4
__main__:
----------------------------------------
PL_GLOBAL_SEED = 102938
PL_SEED_WORKERS = 1
AZ_BATCH_MASTER_NODE = <IP_ADDRESS>:6000
AZ_BATCHAI_MPI_MASTER_NODE = <IP_ADDRESS>
MASTER_ADDR = <IP_ADDRESS>
MASTER_ADDRESS =
MASTER_PORT = 34161
RANK =
NODE_RANK = 0
LOCAL_RANK = 1
GLOBAL_RANK =
WORLD_SIZE = 6
NCCL_SOCKET_IFNAME = eth0
OMPI_COMM_WORLD_RANK = 4
OMPI_COMM_WORLD_LOCAL_RANK = 0
OMPI_COMM_WORLD_SIZE = 6
OMPI_COMM_WORLD_LOCAL_SIZE = 2
----------------------------------------

Driver 5
__main__:
----------------------------------------
PL_GLOBAL_SEED = 102938
PL_SEED_WORKERS = 1
AZ_BATCH_MASTER_NODE = <IP_ADDRESS>:6000
AZ_BATCHAI_MPI_MASTER_NODE = <IP_ADDRESS>
MASTER_ADDR = <IP_ADDRESS>
MASTER_ADDRESS =
MASTER_PORT = 59631
RANK =
NODE_RANK = 0
LOCAL_RANK = 1
GLOBAL_RANK =
WORLD_SIZE = 6
NCCL_SOCKET_IFNAME = eth0
OMPI_COMM_WORLD_RANK = 5
OMPI_COMM_WORLD_LOCAL_RANK = 1
OMPI_COMM_WORLD_SIZE = 6
OMPI_COMM_WORLD_LOCAL_SIZE = 2
----------------------------------------

So far my attempts to manually reset LOCAL_RANK, MASTER_PORT, and NODE_RANK have resulted in an error or the process hanging.

Here is a quick template, no guarantee it will work. You may have to tweak it:

import os
from pytorch_lightning.plugins.environments import ClusterEnvironment

class AzureMLClusterEnvironment(ClusterEnvironment):
    @property
    def creates_processes_externally(self) -> bool:
        """Return True if the cluster is managed (you don't launch processes yourself)"""
        return True

    def world_size(self) -> int:
        return int(os.environ["OMPI_COMM_WORLD_SIZE"])

    def set_world_size(self, size: int) -> None:
        pass

    def global_rank(self) -> int:
        return int(os.environ["OMPI_COMM_WORLD_RANK"])

    def set_global_rank(self, rank: int) -> None:
        pass

    def local_rank(self) -> int:
        return int(os.environ["OMPI_COMM_WORLD_LOCAL_RANK"])

    def node_rank(self) -> int:
        # this may not exist, defaulting to 0
        return int(os.environ.get("OMPI_COMM_WORLD_NODE_RANK", 0))

    def main_address(self) -> str:
        return os.environ["MASTER_ADDRESS"]

    def main_port(self) -> int:
        return int(os.environ["MASTER_PORT"])

    @staticmethod
    def detect() -> bool:
        return "OMPI_COMM_WORLD_SIZE" in os.environ

trainer = Trainer(plugins=[MyClusterEnvironment()])

@jessecambon Wouldn't it be simpler to just set
os.environ["NODE_RANK"] = str(int(os.environ.get("OMPI_COMM_WORLD_RANK")) // devices)

Just wonder if you have any reason not to do that.

@thongonary OMPI_COMM_WORLD_RANK refers to the GPU rank relative to all the other GPUs in the known world (i.e. known world = devices x num_nodes).
NODE_RANK refers to the node a given collection of GPUs live on. Say you specify devices=2 and num_nodes=2, your proposal would result in NODE_RANK=0 in every case which would put us right back to the original bug/issue (every node being set to NODE_RANK=0 which causes stuff to hang).

@jmwoloso In your example, my proposal would result in

NODE_RANK = 0 for OMPI_COMM_WORLD_RANK = 0, 1
NODE_RANK = 1 for OMPI_COMM_WORLD_RANK = 2, 3

ah yeah yeah, you're correct @thongonary , devices not num_nodes (for whatever reason I was switching the two)

Here is an updated function based on @jessecambon's answer to scale it on both single node and multi node:
def set_environment_for_aml(num_nodes, gpus_per_node):

    if num_nodes > 1:
        os.environ['MASTER_ADDRESS'], os.environ['MASTER_PORT'] = os.environ.get("AZ_BATCH_MASTER_NODE").split(':')

    else:
        os.environ["MASTER_ADDRESS"] = os.environ["AZ_BATCHAI_MPI_MASTER_NODE"]
        os.environ["MASTER_PORT"] = "47586"

    os.environ["MASTER_ADDR"] = os.environ.get('MASTER_ADDRESS')

    os.environ["NODE_RANK"] = str(int(os.environ.get("OMPI_COMM_WORLD_RANK")) // gpus_per_node)
    os.environ["LOCAL_RANK"] = os.environ.get("OMPI_COMM_WORLD_LOCAL_RANK", "")
    os.environ["WORLD_SIZE"] = os.environ.get("OMPI_COMM_WORLD_SIZE")

This function can be inserted directly in the main training code:
if __name__ == "__main__":
    parser = ArgumentParser()
    parser.add_argument("--num_nodes", required=True)
    parser.add_argument("--devices", required=True)
    args = parser.parse_args()
    set_environment_for_aml(args.num_nodes, args.devices)

@thongonary yes, // gpus_per_node is more direct for setting NODE_RANK. This looks good. :+1: Is there a reason you used 47586 for the port when num_nodes=1 instead of setting it with AZ_BATCH_MASTER_NODE?

@jessecambon AZ_BATCH_MASTER_NODE is empty when launching with single node, so I just set a random high number (in the spirit of the inaccurate Azure ML documentation)

@thongonary Just confirming this works for me :rocket:
I tested in multinode and single node environments with deepspeed. The only change I made was to wrap num_nodes and gpus_per_node within the function with int() since the command line arguments come in as strings.

I did some more experimentation and discovered that while environmental variable configuration is needed for running with an MPIConfiguration that no environmental variable adjustments are needed when using a PytorchConfiguration. Additionally, an error regarding MKL requires an environmental variable adjustment when I used MPI, but I didn't have this issue with PytorchConfiguration.
I ended up configuring my scripts so that I could run jobs with either MPIConfiguration (with necessary adjustments) or  PytorchConfiguration. I haven't tested this extensively but runtimes seem to be comparable between configurations. These updated scripts and info on my environment are below (run python submit_job.py to submit the job on the Azure cluster). Also note that although the script is now configured for strategy="deepspeed_stage_3" that I also tested strategy="ddp" (and switched out the FusedAdam optimizer for torch.optim.AdamW).
Also, for reference here are the environmental variables for the first and last process on a 2 node, 2 gpu per node run with PytorchConfiguration. Note that RANK is now set where it was not with the MPIConfiguration.
Driver 0
__main__:
----------------------------------------
PL_GLOBAL_SEED = 102938
PL_SEED_WORKERS = 1
AZ_BATCH_MASTER_NODE = <IP_ADDRESS>:6000
AZ_BATCHAI_MPI_MASTER_NODE =
MASTER_ADDR = <IP_ADDRESS>
MASTER_ADDRESS =
MASTER_PORT = 6105
RANK = 0
NODE_RANK = 0
LOCAL_RANK = 0
GLOBAL_RANK =
WORLD_SIZE = 4
NCCL_SOCKET_IFNAME = eth0
OMPI_COMM_WORLD_RANK =
OMPI_COMM_WORLD_LOCAL_RANK =
OMPI_COMM_WORLD_SIZE =
OMPI_COMM_WORLD_LOCAL_SIZE =
----------------------------------------

Driver 3
__main__:
----------------------------------------
PL_GLOBAL_SEED = 102938
PL_SEED_WORKERS = 1
AZ_BATCH_MASTER_NODE = <IP_ADDRESS>:6000
AZ_BATCHAI_MPI_MASTER_NODE =
MASTER_ADDR = <IP_ADDRESS>
MASTER_ADDRESS =
MASTER_PORT = 6105
RANK = 3
NODE_RANK = 1
LOCAL_RANK = 1
GLOBAL_RANK =
WORLD_SIZE = 4
NCCL_SOCKET_IFNAME = eth0
OMPI_COMM_WORLD_RANK =
OMPI_COMM_WORLD_LOCAL_RANK =
OMPI_COMM_WORLD_SIZE =
OMPI_COMM_WORLD_LOCAL_SIZE =
----------------------------------------

Scripts
submit_job.py
import os
from azureml.core.runconfig import PyTorchConfiguration, MpiConfiguration
from azureml.core import Environment, Workspace, ScriptRunConfig, Experiment
from azureml.core.authentication import ServicePrincipalAuthentication

# ----------------------------------------------------------------------------------------

experiment_name = "myExperiments"
environment_name = "YourAzureEnvironmentName"
cluster_name="YourClusterName"
node_count=2
devices=2 # number of gpus per node
environment_version=None
run_config="pytorch"
script_name="train.py"

# -----------------------------------------------------------------------------------------

# fix for https://github.com/pytorch/pytorch/issues/37377 (only needed for MPIConfiguration)
if run_config=='mpi':
    env_settings="MKL_THREADING_LAYER='GNU'"
else:
    env_settings=""

command="; ".join([
    "conda env list",
    "python -m pip freeze",
    "wget -N https://raw.githubusercontent.com/Lightning-AI/lightning/master/requirements/collect_env_details.py",
    "python collect_env_details.py",
    f"{env_settings} python {script_name} --num_nodes={node_count} --devices={devices} --run_config={run_config}"
])

print(command)
###################

ws = Workspace.get(
    name="XXXXXXXX",
    resource_group="XXXXXXX",
    subscription_id=os.environ["XXXXXXXX"],
    auth= ServicePrincipalAuthentication(
            tenant_id=os.environ.get("XXXXXXXXXXXX"),
            service_principal_id=os.environ.get("XXXXXXXXX"),
            service_principal_password=os.environ.get("XXXXXXXX")
        )
    )

compute_target=ws.compute_targets[cluster_name]

env = Environment.get(
    workspace = ws,
    name = environment_name,
    version = environment_version
    )

print(env)
print(env.get_image_details(ws))

if run_config=='pytorch':
    job_config = PyTorchConfiguration(
        communication_backend="Nccl",
        process_count=node_count*devices,
        node_count=node_count
    )
elif run_config=='mpi':
    job_config = MpiConfiguration(node_count=node_count, process_count_per_node=devices)

src = ScriptRunConfig(
    source_directory = ".",
    command=command,
    compute_target = compute_target,
    environment = env,
    distributed_job_config = job_config
)

# Submit job
run = Experiment(ws, experiment_name).submit(src)

train.py
import torch, os, logging, sys
from torch.utils.data import DataLoader, Dataset
from deepspeed.ops.adam import FusedAdam
from azureml.core import Run, Workspace
from pytorch_lightning import LightningModule, Trainer, LightningDataModule, seed_everything
from pytorch_lightning.loggers import MLFlowLogger
from argparse import ArgumentParser

os.environ["TORCH_CPP_LOG_LEVEL"]="INFO"
os.environ["TORCH_DISTRIBUTED_DEBUG"] = "DETAIL"

def configure_environment_mpi(num_nodes, gpus_per_node):
    """
    Configure environmental variables for ddp and deepspeed. For use with MPIConfiguration
    """
    if int(num_nodes) > 1:
        os.environ['MASTER_ADDRESS'], os.environ['MASTER_PORT'] = os.environ.get("AZ_BATCH_MASTER_NODE").split(':')

    else:
        os.environ["MASTER_ADDRESS"] = os.environ["AZ_BATCHAI_MPI_MASTER_NODE"]
        os.environ["MASTER_PORT"] = "47586"

    os.environ["MASTER_ADDR"] = os.environ.get('MASTER_ADDRESS')
    os.environ["NODE_RANK"] = str(int(os.environ.get("OMPI_COMM_WORLD_RANK")) // int(gpus_per_node))
    os.environ["LOCAL_RANK"] = os.environ.get("OMPI_COMM_WORLD_LOCAL_RANK", "")
    os.environ["WORLD_SIZE"] = os.environ.get("OMPI_COMM_WORLD_SIZE")

divider_str="-"*40

def get_env_display_text(var_name):
    var_value = os.environ.get(var_name, "")
    return f"{var_name} = {var_value}"

def display_environment(header='Environmental variables'):
    """
    Print a few environment variables of note
    """
    variable_names = [
        "PL_GLOBAL_SEED",
        "PL_SEED_WORKERS",
        "AZ_BATCH_MASTER_NODE",
        "AZ_BATCHAI_MPI_MASTER_NODE",
        "MASTER_ADDR",
        "MASTER_ADDRESS",
        "MASTER_PORT",
        "RANK",
        "NODE_RANK",
        "LOCAL_RANK",
        "GLOBAL_RANK",
        "WORLD_SIZE",
        "NCCL_SOCKET_IFNAME",
        "OMPI_COMM_WORLD_RANK",
        "OMPI_COMM_WORLD_LOCAL_RANK",
        "OMPI_COMM_WORLD_SIZE",
        "OMPI_COMM_WORLD_LOCAL_SIZE"
    ]
    var_text = "\n".join([get_env_display_text(var) for var in variable_names])
    print(f"\n{header}:\n{divider_str}\n{var_text}\n{divider_str}\n")

class RandomDataset(Dataset):
    def __init__(self, size, length):
        self.len = length
        self.data = torch.randn(length, size)

    def __getitem__(self, index):
        return self.data[index]

    def __len__(self):
        return self.len

class BoringModel(LightningModule):
    def __init__(self):
        super().__init__()
        self.save_hyperparameters()
        self.model = torch.nn.Linear(32, 2)

    def forward(self, x):
        return self.model(x)

    def training_step(self, batch, batch_idx):
        loss = self(batch).sum()
        self.log("train_loss", loss, on_step=True, on_epoch=True)
        return {"loss": loss}

    def validation_step(self, batch, batch_idx):
        loss = self(batch).sum()
        self.log("valid_loss", loss)

    def configure_optimizers(self):
        #return torch.optim.AdamW(self.model.parameters())
        return FusedAdam(self.model.parameters())

    def setup(self, stage=None) -> None:

        # prevents hanging
        if stage != "fit":
            return

class DataModule(LightningDataModule):
    def __init__(self):
        super().__init__()
        self.num_workers = os.cpu_count()
        print(f"num_workers set to {self.num_workers}")

    def setup(self, stage=None) -> None:
        self._dataloader = DataLoader(
            RandomDataset(32, 64),
             num_workers=self.num_workers,
             batch_size=1,
             pin_memory=True
             )

    def train_dataloader(self):
        return self._dataloader

    def test_dataloader(self):
        return self._dataloader

    def val_dataloader(self):
        return self._dataloader

if __name__ == "__main__":
    parser = ArgumentParser()
    parser.add_argument("--num_nodes", required=True)
    parser.add_argument("--devices", required=True)
    parser.add_argument("--run_config")
    args = parser.parse_args()

    # If running on azure, get the active tracking uri and run id
    # otherwise, use the workspace to get a tracking uri
    active_run = Run.get_context() #  active run azureml object
    offline = False
    try:
        print(active_run.experiment)
    except:
        offline = True

    if offline:
        print("Running offline...")
        run_id = None
        ws = Workspace.get(
            subscription_id=os.environ.get("AZURE_SUBSCRIPTION_ID"),
            resource_group="abcinc-stg-rg",
            name="abcinc-stg-amls",
        )
        tracking_uri = ws.get_mlflow_tracking_uri()
    else:
        # MPIConfiguration requires environmental variable tweaking
        if args.run_config == "mpi":
            configure_environment_mpi(args.num_nodes, args.devices)

        tracking_uri=active_run.experiment.workspace.get_mlflow_tracking_uri()
        run_id = active_run.id

    seed_everything(102938, workers = True)

    display_environment("__main__")
    model = BoringModel()
    dm = DataModule()

    logger = MLFlowLogger(
                experiment_name='reprex',
                tracking_uri=tracking_uri,
                run_id=run_id
    )

    trainer = Trainer(
        num_nodes=args.num_nodes,
        accelerator='gpu',
        devices=args.devices,
        limit_train_batches=2,
        limit_val_batches=2,
        log_every_n_steps=1,
        logger=logger,
        default_root_dir = 'checkpoints',
        enable_checkpointing=False,
        num_sanity_val_steps=0,
        max_epochs=2,
        enable_model_summary=False,
        strategy="deepspeed_stage_3",
        precision=16
    )
    trainer.fit(model, datamodule=dm)

Environment

The cluster has 2 nodes with 2 V100 Gpus on each node
The base docker image for the environment was mcr.microsoft.com/azureml/openmpi4.1.0-cuda11.3-cudnn8-ubuntu20.04

* CUDA:
	- GPU:
		- Tesla V100-PCIE-16GB
		- Tesla V100-PCIE-16GB
	- available:         True
	- version:           11.3
* Packages:
	- numpy:             1.22.3
	- pyTorch_debug:     False
	- pyTorch_version:   1.11.0
	- pytorch-lightning: 1.6.5
	- tqdm:              4.64.0
* System:
	- OS:                Linux
	- architecture:
		- 64bit
		- ELF
	- processor:         x86_64
	- python:            3.8.13
	- version:           #38-Ubuntu SMP Sun Mar 22 21:27:21 UTC 2020

@awaelchli do you think this is a sufficiently general use-case that we provide an environment plugin for mpi?

@justusschock I would say we leave it up to the community here to experiment with it and contribute something that works for everybody. The skeleton is here https://github.com/Lightning-AI/lightning/issues/13639#issuecomment-1184350663 and the blanks could be filled with what was done by @jessecambon @thongonary here. If we can get an implementation that is general but not colliding with the existing clusters, then it could also be auto-detected like the other environments.
I'm not knowledgable enough about MPI to give any good insights otherwise.

@awaelchli I made an attempt at defining the MPI cluster environment class, but I'm having trouble getting the trainer to use the provided cluster environment. I keep on getting the error below regarding os.environ["MASTER_ADDR"] = cluster_environment.main_address.
I put in a print statement to see if OpenMPIClusterEnvironment.main_address() is being run and it doesn't appear to be. I've tried using the format trainer(plugins=[OpenMPIClusterEnvironment()] and trainer(strategy=DDPStrategy(cluster_environment=OpenMPIClusterEnvironment())).
I can post the full code if that would be helpful, but let me know if you can see what the issue is.
Error:
Traceback (most recent call last):
  File "train.py", line 228, in <module>
    trainer.fit(model, datamodule=dm)
  File "/azureml-envs/azureml_a97d4dbeffb8de54f1509e712d7ebe36/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 770, in fit
    self._call_and_handle_interrupt(
  File "/azureml-envs/azureml_a97d4dbeffb8de54f1509e712d7ebe36/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 723, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/azureml-envs/azureml_a97d4dbeffb8de54f1509e712d7ebe36/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 811, in _fit_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
  File "/azureml-envs/azureml_a97d4dbeffb8de54f1509e712d7ebe36/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1171, in _run
    self.strategy.setup_environment()
  File "/azureml-envs/azureml_a97d4dbeffb8de54f1509e712d7ebe36/lib/python3.8/site-packages/pytorch_lightning/strategies/ddp.py", line 152, in setup_environment
    self.setup_distributed()
  File "/azureml-envs/azureml_a97d4dbeffb8de54f1509e712d7ebe36/lib/python3.8/site-packages/pytorch_lightning/strategies/ddp.py", line 205, in setup_distributed
    init_dist_connection(self.cluster_environment, self._process_group_backend)
  File "/azureml-envs/azureml_a97d4dbeffb8de54f1509e712d7ebe36/lib/python3.8/site-packages/pytorch_lightning/utilities/distributed.py", line 352, in init_dist_connection
    os.environ["MASTER_ADDR"] = cluster_environment.main_address
  File "/azureml-envs/azureml_a97d4dbeffb8de54f1509e712d7ebe36/lib/python3.8/os.py", line 680, in __setitem__
    value = self.encodevalue(value)
  File "/azureml-envs/azureml_a97d4dbeffb8de54f1509e712d7ebe36/lib/python3.8/os.py", line 750, in encode
    raise TypeError("str expected, not %s" % type(value).__name__)
TypeError: str expected, not method
[2022-08-04T21:43:20.484362] Command finished with exit code 256 interpreted as return code 1

Partial code:
...

class OpenMPIClusterEnvironment(ClusterEnvironment):
    def __init__(self, devices: int = 1) -> None:
        """ devices : devices per node"""
        super().__init__()
        self.devices = devices

    @property
    def creates_processes_externally(self) -> bool:
        """Return True if the cluster is managed (you don't launch processes yourself)"""
        return True

    def world_size(self) -> int:
        return int(os.environ["OMPI_COMM_WORLD_SIZE"])

    def set_world_size(self, size: int) -> None:
        pass

    def global_rank(self) -> int:
        return int(os.environ["OMPI_COMM_WORLD_RANK"])

    def set_global_rank(self, rank: int) -> None:
        pass

    def local_rank(self) -> int:
        return int(os.environ["OMPI_COMM_WORLD_LOCAL_RANK"])

    def node_rank(self) -> int:
        # this may not exist, defaulting to 0
        return int(os.environ.get("OMPI_COMM_WORLD_RANK",0)) // int(self.devices)

    def main_address(self) -> str:
        # AZ_BATCH_MASTER_NODE should be defined when num_nodes > 1
        if "AZ_BATCH_MASTER_NODE" in os.environ:
            print(f"main_address : {os.environ['AZ_BATCH_MASTER_NODE'].split(':')[0]}") # debugging
            return os.environ["AZ_BATCH_MASTER_NODE"].split(':')[0]
        elif "AZ_BATCHAI_MPI_MASTER_NODE" in os.environ:
            print(f"main_address : {os.environ['AZ_BATCHAI_MPI_MASTER_NODE']}") # debugging
            return os.environ["AZ_BATCHAI_MPI_MASTER_NODE"]
        else:
            raise("main_address not found")

    def main_port(self) -> int:
        # AZ_BATCH_MASTER_NODE should be defined when num_nodes > 1
        if "AZ_BATCH_MASTER_NODE" in os.environ:
            return int(os.environ["AZ_BATCH_MASTER_NODE"].split(':')[1])
        else:
            return int(47586) # set port to arbitrary high number

    @staticmethod
    def detect() -> bool:
        return "OMPI_COMM_WORLD_SIZE" in os.environ

...

    trainer = Trainer(
        num_nodes=args.num_nodes,
        accelerator='gpu',
        devices=args.devices,
        limit_train_batches=2,
        limit_val_batches=2,
        log_every_n_steps=1,
        logger=logger,
        enable_checkpointing=False,
        num_sanity_val_steps=0,
        max_epochs=2,
        enable_model_summary=False,
        #strategy='ddp',
        plugins=[OpenMPIClusterEnvironment(devices=args.devices)],
        # strategy=DDPStrategy(
        #     cluster_environment = OpenMPIClusterEnvironment(devices=args.devices)
        # ),
        # strategy=DeepSpeedStrategy(
        #     stage = 3,
        #     cluster_environment = OpenMPIClusterEnvironment(devices=args.devices)
        # ),
        precision=16
    )
    trainer.fit(model, datamodule=dm)

adding the property decorator to main_address (and any others you need to access that way) should fix that:
@property
def main_address(self) -> str:
    # AZ_BATCH_MASTER_NODE should be defined when num_nodes > 1
    if "AZ_BATCH_MASTER_NODE" in os.environ:
        print(f"main_address : {os.environ['AZ_BATCH_MASTER_NODE'].split(':')[0]}") # debugging
        return os.environ["AZ_BATCH_MASTER_NODE"].split(':')[0]
    elif "AZ_BATCHAI_MPI_MASTER_NODE" in os.environ:
        print(f"main_address : {os.environ['AZ_BATCHAI_MPI_MASTER_NODE']}") # debugging
        return os.environ["AZ_BATCHAI_MPI_MASTER_NODE"]
    else:
        raise("main_address not found")

i suppose an alternative is also to call the method as well.
os.environ["MASTER_ADDR"] = cluster_environment.main_address()

Thanks @jmwoloso, I was missing @property on the main_address and main_port function definitions. I'll move the working solution over to https://github.com/Lightning-AI/lightning/issues/14014

I'm closing the stale issue now after we implemented an MPI environment (#16570). I hope this covers the use case here or at least provides a base to modify if needed for your cluster.

unset OMPI_COMM_WORLD_LOCAL_RANK solves the issue.


Remove binary blobs from history

Following the steps described in issue #11
Step 1. Remove src/test/resources/gcd-v1beta2-rev1-2.1.1.zip from index.
git filter-branch --prune-empty --index-filter 'git rm -rf --cached --ignore-unmatch src/test/resources/gcd-v1beta2-rev1-2.1.1.zip' --tag-name-filter cat -- --all

Step 2. Remove src/test/resources/gcd-head.zip from index.
git filter-branch --force --prune-empty --index-filter 'git rm -rf --cached --ignore-unmatch src/test/resources/gcd-head.zip' --tag-name-filter cat -- --all

Step 3. Ignore those files from now on, make a new commit.
echo 'src/test/resources/gcd-head.zip' >> .gitignore
echo 'src/test/resources/gcd-v1beta2-rev1-2.1.1.zip' >> .gitignore
git commit -am 'Remove files'

Step 4. Garbage collect outdated blobs and force push back to your fork.
(WARNING THIS WILL REWRITE HISTORY IN YOUR FORK)
git for-each-ref --format='delete %(refname)' refs/original | git update-ref --stdin
git reflog expire --expire=now --all
git gc --prune=now
git push origin --force --all

@ryanseys : Any chance you can double check this too? It's a big scary change that we want to make sure we don't have to do again........

Because it rewrites history we can't merge this in as a PR, it has to be force pushed to the repo and there's really no way for me to "double check" this. I'm not sure what I would do to double check it considering all the SHAs have changed, hence the monstrosity of a PR this is. Any thoughts?

Something strange.
When I clone gcloud-java from GoogleCloudPlatform and from aozarov (where I applied the pruning)
I don't see a difference in size (actually the clone is slightly bigger...)
ozarov /tmp $ git clone https://github.com/GoogleCloudPlatform/gcloud-java/
Cloning into 'gcloud-java'...
remote: Counting objects: 71093, done.
remote: Total 71093 (delta 0), reused 0 (delta 0), pack-reused 71093
Receiving objects: 100% (71093/71093), 225.40 MiB | 3.56 MiB/s, done.
Resolving deltas: 100% (66008/66008), done.

ozarov /tmp $ git clone https://github.com/aozarov/gcloud-java/
Cloning into 'gcloud-java'...
remote: Counting objects: 71176, done.
remote: Total 71176 (delta 0), reused 0 (delta 0), pack-reused 71176
Receiving objects: 100% (71176/71176), 225.40 MiB | 3.33 MiB/s, done.
Resolving deltas: 100% (66043/66043), done.

Does it make sense? Could it be that the clone has more branches to be the reason?
Also, I am not sure why the Pull request suggests 200 files change (Last time I tried it,
Before, using a different procedure I mentioned in issue #11 there were many commits to merge but no file differences (see see pull request #73).
A local diff between the 2 clones and also between the 2 zip files downloaded from both sites suggest only .gitingore as a difference (so it is probably OK, but just want to make sure).
Maybe a way to verify it is to duplicate the repository and then apply the procedure on a clone of it, push the change and see if it works?

@ryanseys mentioned in a previous post that he had to delete the old repository to actually get rid of things, but potentially we could simple prune old objects after a git push -f. Did you force push this to your fork?
As Ryan said, the PR won't work here, so this is just a debate on the change until we ultimately apply it forcefully.

Yes, I applied all the steps described by @ryanseys in issue #11. I also mentioned the steps I applied above, which include the "git push origin --force --all". Is that what you are referring to?
Oh, are you (and Ryan) saying that once agreed I will need to apply the same steps directly on the GCP repository instead of merging it via a push request from a fork? If so, I still don't understand why this fork does not seem to be smaller...

Yeah, apparently they only run garbage collection on the repo a maximum of once per day on a repo and it's triggered by a push so you may have to wait a couple days and push a couple dummy commits in order to see a difference in clone size once this GC runs.
Source: https://twitter.com/GitHubHelp/status/387926738161774592


Update gcp-cloudshell-open-in-command-injection.yaml

The quotes around "Open In" makes the YAML invalid.

thanks for submitting this fix! you've deleted the branch so i added it to the main branch.


BUG：The embedding model has been used for a while and is no longer available.

Describe the bug
The embedding model has been used for a while and is no longer available.
The actual situation is: I use the embedding model for some time, I can't find the embedding model in the webui interface, and the API can't be used, but I can still find the uid in the terminal, and if I use the ui web to create the model again, the uid is no longer bge-m3-1-0, so I need to delete the model in the terminal, and then create the model again in the ui. I need to delete the previous model in terminal and create the model in ui again.
Translated with DeepL.com (free version)
To Reproduce
docker：xprobe/xinference:latest
2024-04-22 03:16:25,220 xinference.core.worker 95 DEBUG    Enter terminate_model, args: (<xinference.core.worker.WorkerActor object at 0x7d960864b380>, 'bge-m3-1-0'), kwargs: {}
2024-04-22 03:16:25,221 xinference.core.worker 95 DEBUG    Destroy model actor failed, model uid: bge-m3-1-0, error: [Errno 111] Connection refused
2024-04-22 03:16:25,221 xinference.core.worker 95 DEBUG    Remove sub pool failed, model uid: bge-m3-1-0, error: '<IP_ADDRESS>:33245'
2024-04-22 03:16:25,221 xinference.core.worker 95 DEBUG    Leave terminate_model, elapsed time: 0 s
2024-04-22 03:16:25,222 xinference.core.worker 95 WARNING  Recreating model actor bge-m3-1-0 ...
2024-04-22 03:16:25,222 xinference.core.worker 95 DEBUG    Enter launch_builtin_model, args: (<xinference.core.worker.WorkerActor object at 0x7d960864b380>,), kwargs: {'model_uid': 'bge-m3-1-0', 'model_name': 'bge-m3', 'model_size_in_billions': None, 'model_format': None, 'quantization': None, 'model_type': 'embedding', 'n_gpu': 'auto', 'peft_model_path': None, 'image_lora_load_kwargs': None, 'image_lora_fuse_kwargs': None, 'request_limits': None, 'gpu_idx': None, 'event_model_uid': 'bge-m3', '_': 1, '__': 0}
2024-04-22 03:16:25,223 xinference.core.worker 95 DEBUG    GPU selected: [0] for model bge-m3-1-0
2024-04-22 03:16:28,479 xinference.model.embedding.core 95 DEBUG    Embedding model bge-m3 found in ModelScope.
2024-04-22 03:16:28,480 xinference.model.utils 95 INFO     Use model cache from a different hub.
2024-04-22 03:16:28,481 xinference.core.worker 95 ERROR    Failed to load model bge-m3-1-0
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/xinference/core/worker.py", line 662, in launch_builtin_model
    model, model_description = await asyncio.to_thread(
  File "/opt/conda/lib/python3.10/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
  File "/opt/conda/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/conda/lib/python3.10/site-packages/xinference/model/core.py", line 83, in create_model_instance
    return create_embedding_model_instance(
  File "/opt/conda/lib/python3.10/site-packages/xinference/model/embedding/core.py", line 329, in create_embedding_model_instance
    model = EmbeddingModel(model_uid, model_path, **kwargs)
TypeError: EmbeddingModel.__init__() got an unexpected keyword argument 'event_model_uid'
2024-04-22 03:20:29,113 xinference.core.worker 95 ERROR    Report status got error.
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/xinference/core/worker.py", line 781, in report_status
    status = await asyncio.to_thread(gather_node_info)
  File "/opt/conda/lib/python3.10/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
asyncio.exceptions.CancelledError

@JasonFlyBeauty Could you please show me the complete debug logs from the start of the xinference?

@JasonFlyBeauty Thanks for reporting. This is a bug. We will fix it ASAP.
By the way, the reason you are facing this problem is that, you have triggered a CUDA OOM while using embedding. This error is an error in the auto-recovery process.
Please check the status of your GPU memory while using the embedding model.

Thank you very much for your help.

@JasonFlyBeauty Thanks for reporting. This is a bug. We will fix it ASAP. By the way, the reason you are facing this problem is that, you have triggered a CUDA OOM while using embedding. This error is an error in the auto-recovery process. Please check the status of your GPU memory while using the embedding model.

Also, I would like to know how to specify the GPU card for the embedding model and if it supports multi-card running

@JasonFlyBeauty Thanks for reporting. This is a bug. We will fix it ASAP. By the way, the reason you are facing this problem is that, you have triggered a CUDA OOM while using embedding. This error is an error in the auto-recovery process. Please check the status of your GPU memory while using the embedding model.

Also, I would like to know how to specify the GPU card for the embedding model and if it supports multi-card running

@JasonFlyBeauty Thanks for reporting. This is a bug. We will fix it ASAP. By the way, the reason you are facing this problem is that, you have triggered a CUDA OOM while using embedding. This error is an error in the auto-recovery process. Please check the status of your GPU memory while using the embedding model.

Use --gpu-idx option when launching models. See https://inference.readthedocs.io/en/latest/reference/generated/xinference.client.Client.launch_model.html
xinference launch <other_options> --gpu-idx <your_specific_gpu_index>

Thanks again for your help!

另外，我想知道如何指定嵌入模型的GPU卡以及是否支持多卡运行

@JasonFlyBeauty感谢您的报告。这是一个错误。我们会加速修复它。顺便说一句，您遇到此问题的原因是，您在使用嵌入时触发了 CUDA OOM。该错误是自动恢复过程中的错误。使用嵌入模型时请检查GPU内存的状态。

启动模型时使用 --gpu-idx 选项。请参考https://inference.readthedocs.io/en/latest/reference/ generated /xinference.client.Client.launch_model.html
xinference launch <other_options> --gpu-idx <your_specific_gpu_index>