Lazily load regression test data

Instead of eagerly loading all regression test data at suite start, this PR changes the behavior to load test data on-demand, if it is missing. This makes it nicer when running a subset of the regression suite as only the required data is loaded.

Ah, looks like this is failing due to the letInSelectProjection.test not correctly declaring its data dependencies, will fix.

@wemrysi LGTM - I'll merge once jenkins is green

@wemrysi so the build for 3.0_RO failed with two instances of
[error]      java.lang.RuntimeException: File not found: /regression/cars

I actually don't know the difference between the 3.0 build and the 3.0_RO build - what is the difference?
Is there something in the RO build that would cause a file not to be found that was successfully found in the non-RO build?

Maybe something about the file permissions in read-only (RO) mode caused the file not to be located?

@alissapajer Yeah, see my earlier comment. The test in question doesn't declare that it depends on the cars data set, that's the root of the problem. As for why it happens in RO and not in the other cases, my (unconfirmed, but fairly confident) hypothesis is that another test does depend on cars and loads the dataset so that when this test is reached in the non-RO cases the data is there. However, in RO-mode we skip lots of tests that we know can't run with read-only permissions, I suspect the test that was loading cars is one of the ones we skip so when we reach the test in question, it attempts to query cars and it doesn't exist.

Come to think of it, we should probably be extracting data set names from the compiled queries (LogicalPlan), then we don't need to declare them separately. That's for another PR, however.

Current coverage is 66.59% (diff: 100%)

Merging #1461 into master will increase coverage by 0.95%

@@             master      #1461   diff @@
==========================================
  Files           295        297     +2
  Lines         14019      13844   -175
  Methods       13439      13475    +36
  Messages          0          0
  Branches        218        211     -7
==========================================
+ Hits           9202       9220    +18
+ Misses         4817       4624   -193
  Partials          0          0

Powered by Codecov. Last update 5a37edd...c2017e4


Migrate ERC721

Fixes #571. Supersedes #584's work by @Amxx.

also working on a SafeERC721 module to serve as a dual-interface dispatcher, although it can probably be on its own PR

(still waiting for the soon to be merged #620)

I don't think we need to test _check_on_erc721_received, it's an internal function and not part of the module's ABI..


Investigate: What dependencies should be made optional

Like contracts, staking and rustls stuff.
This will remove quite some dependencies in case they are not needed.

Could look similar to staking xt: https://github.com/scs/substrate-api-client/blob/10084fa48c77a16d78e36c48c1af2404c141b85b/Cargo.toml#L87

Unfortunately sp-wasm-interface has a dependency on wasmi as well (at least in std mode). See this issue/comment https://github.com/scs/substrate-api-client/issues/272#issuecomment-1484750527

Hmm, I see. However, we did not have these dependencies before in the worker: https://github.com/integritee-network/worker/pull/1263/files
Which intrigues me now.

@clangenb Hmm, I checked and most of the dependencies are not really new if I look at:
https://github.com/integritee-network/worker/blob/945571ae07f6efdebb7cf67b21e5e31b1b4b6ee4/Cargo.lock
Only wasmparser-nostd is completely new wasmi is now duplicated with different version numbers.
One thing I noticed: In the worker there are some dependencies to the substrate-api-client without default-features = false. This now activates the jsonrpsee-client feature which is not what you might want/need. I think disabling the default features could already help a bit with your dependencies. Could you check what happens?

Nice, this is what happens: https://github.com/integritee-network/worker/pull/1271/files.

Great. I was not really expecting to see changes regarding wasm, but at least one of the rustls dependencies is gone 👍


fix: Make default HttpHandler a WaterfallHandler

Accidentally stumbled upon this when making #428 . Currently all requests to /pods will also be sent to the LDP handler.
As it currently is, the config for inserting the pod http handler in the config has always been wrong. It used to work because the AllVoidCompositeHandler used to ignore errors, but since the change to SequenceHandler it probably never worked anymore. Perhaps this also caused the confusion in #421 .
This does mean that the middleware will only run for requests that reach the LDP handler. If we want the middleware to always run more changes will be required.
Should also investigate how the pod creation integration test didn't catch this. Perhaps a timing issue because the pod handler managed to respond first.

After some more testing it actually only starts failing after #428 is merged, so there might be an issue in that branch (even then, it still makes sense that this should be a WaterfallHandler).

Final update: the error was actually thrown by an unrelated problem. But I still suggest to make this change since this is the behaviour we actually want. If the middleware should run for all http handlers the order can be reversed (a Sequence containing a Waterfall).

If the middleware should run for all http handlers

Yeah!

the order can be reversed (a Sequence containing a Waterfall).

Done in 7599e978acef3d98ad7d8db24f58a417716a7aeb


Prevent scrolling bounce effect

Hi, I have a problem with my F7 app.
When I have a page with content who don't take all of the height, it's OK, for example like this :

But when I have content who take more than the heigh, the div is scrollable (it's normal, it's ok) but the top and the bottom "overflow and bounce". It's difficult for me to explain the problem because I'm french. So have a look to this picture who describe my problem :

I have tested  <preference name="DisallowOverscroll" value="true" />
But it's juste solve the navbar bounce, but note the content page bounce.
I have tested .page-content {    margin-bottom:1px; } .page {    overflow: auto;    -webkit-overflow-scrolling: touch; }
But it not works.
Thanks in advance

It is default iOS effect which is excpected by iOS users, not sure it can be disabled easily. Actually can't see any problems with it. Just make page background color same white as list bg color

This works well with current Framework7 version to eliminate the bounce effect:

Download https://github.com/lazd/iNoBounce/blob/master/inobounce.min.js

Include script <script type="text/javascript" src="js/inobounce.min.js"></script>

Add css to .views container

.views {
  height: 100%;
  overflow: auto;
  -webkit-overflow-scrolling: touch;
}


chore: release 1.2.171

:robot: I have created a release *beep* *boop*
1.2.171 (2022-09-07)
Bug Fixes

buildah: different processing of CMD/ENTRYPOINT by Stapel and Buildah backend (97e89b0)
cleanup: fallback to batch/v1beta1 when querying CronJobs for cleanup (2f53aa4)
telemetry: repair turn-off telemetry switch (f8559e9)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/werf/werf/releases/tag/v1.2.171 :sunflower:


An error occurred while parsing the item

🐞 bug report
Describe the bug

POE Overlay does not parse many HEIST specific items.
To Reproduce

CTRL+D on items
Expected behavior

price check with values of mods
Screenshots

cf attachment
poe overlay bug parse heist.docx
🌍  Your Environment

Version:  0.6.28
POE PC

Still no clue about this issue ?

Is poe overlay dead ? Should I use another app ?


Fix add_root_node() being no-op

The method was non-functional since e9bbb97acccc08ae03fde41e4cc6d2dc6722021a, for no reason.

I'm not sure it's a great idea to start revitalizing the EditorPlugin class since it's borderline entirely deprecated by other classes. This functionality seems like it would fit better as part of EditorInterface.

After adding the root node, the tab for the scene will still show as [empty].

Fixed.

I'm not sure it's a great idea to start revitalizing the EditorScript class since it's borderline entirely deprecated by other classes.

The base functionality of this class is running scripts directly in the editor, which is still relevant. Methods other than _run() are mostly extras that we can't remove for compatibility reasons.

The base functionality of this class is running utility scripts in the editor, which is still relevant. Methods other than _run() are mostly extras that we can't remove for compatibility reasons.

Would this method still be under the purview of compatibility after not having worked for nearly a decade haha. It has been marked unimplemented since 3.0. I can see a rational for the answer being yes, so I won't press the matter.
I still think it makes sense at least for a similar function to be implemented in EditorInterface, and this one to be marked as deprecated (along with get_scene(), since get_tree().current_scene does the same thing).

When using this method with scenes, would it make sense for this method to add an inherited instance of a scene instead of just opening a scene?

vs

This PR is supposed to make the method functional again the way it was designed, any additional changes can be done later.

When using this method with scenes, would it make sense for this method to add an inherited instance of a scene instead of just opening a scene?

It does not open a scene, you need to be already in empty scene to use it.

I made it create inherited scene when adding an instance.

Thanks!


Fix some lint errors [phab:D2294]

Created by danielmoy-google at 2018-05-09 23:36:47:

craigbarber wrote at 2018-05-10 16:32:01:
Thanks for catching these.

craigbarber accepted this revision at 2018-05-10 16:32:01

danielmoy-google closed via commit 006fd1149173162df2b6670f4bf3d4a38204ca4c at 2018-05-11 17:03:04


chore: upgraded deps and expanded react support range

What

upgraded dependencies and added react 18 as supported range

:tada: This PR is included in version 1.6.4 :tada:
The release is available on:

GitHub release
npm package (@latest dist-tag)

Your semantic-release bot :package::rocket:


[Android][iOS][Dev-tool] loadmore method only invoke once

loadmore at list tag  just can be invoked once. I have to invoke this.$refs['list'].resetLoadmore() method in loadmore method ,then it can have right behavior.But the document have nothing about resetLoadmore. What is the right to user loadmore?

v-for only can be used for cell. Otherwise loadmore will not behavior expectly.


Create SCC Eligibility download covering 3 annexes for legal exercises

Background
This ticket is connected with #2387, in which the Eligibility report display has been reconfigured to focus on specific areas of eligibility for legal exercises only, due to the fact that non-legal exercises are problematic thanks to the lack of distinction between statutory and ASC criteria, and reviews that are currently taking place. Hence, we aim to automate production of the following annexes for legal exercises only: Statutory eligibility criteria (Professional qualification and PQE), PJE and RLoS.
User Story
As an Admin, I would like to be able to download SCC Annex-ready eligibility annexes so that I can expedite production of the SCC paper.
Benefit(s) to user (if not already clear from User story)
Admins will be able to produce a Word download from the platform that covers some of the eligibility annexes required for the SCC report.
Feature(s) Description

[ ] The annexes described below should be produced as a single Word document download which is downloaded to user's desktop when you click the Download SCC Annex report button

[ ] Each section of the report should be preceded by a page break

[ ] The first section of the annex 'Candidates who do not meet the Statutory Eligibility Criteria' includes candidates who are 'Not met' for Professional qualification and/or Post-qualification experience sorted alphabetically by candidate surname

[ ] The content for the 'Reasons Statutory Eligibility Criteria is not satisfied' column is drawn from the 'Reasons not satisfied' field for Statutory eligibility criteria

[ ] The content for the next two sections is Candidate Previous Judicial Experience and plays back this section of the Eligibility report

[ ] It is split into 3 sections 'Recommended to proceed', 'Recommended to discuss', and 'Recommended to reject' according to the Recommendation selected from the dropdown in that section

[ ] The final section relates to Reasonable Length of Service and includes all candidates with a Not Met flag under RLoS ordered alphabetically by their surname

[ ] It is formatted as below

[ ] Column C returns candidates' ages at date that SCC is scheduled for (this has already been calculated by the platform to determine whether they meet the RLoS criteria or not)

Acceptance Criteria
It's done when for legal exercises only:

[ ] Users can trigger a download to their desktop based on the criteria described above
[ ] Each section starts on a fresh page
[ ] Each section is ordered alphabetically by candidates' surnames
[ ] Each section is formatted according to the images presented
[ ] Each section included the content as described
[ ] The PJE section is split into Proceed, Discuss and Reject sub-sections

Definition of Done

[ ] User stories/acceptance criteria met
[ ] Internal reviews passed (feedback actioned)
[ ] User testing passed
[ ] Relevant technical documentation updated
[ ] User guidance updated
[ ] Deployed and merged without errors

User Testing Steps
This feature is designed to automate as much of the production and formatting of the eligibility report as possible for legal exercises. Does the functionality deliver what is required to produce the annexes covering: 1) statutory eligibility criteria, 2) PJE and 3) Reasonable length of service?
Feature Champion
Nick

@nickaddy
I'm not very sure about these 2 places, could you help me to confirm them ? Thanks

Is ok that we use number (1, 2, 3 ...) for represent years in this place? Currently, it seems that we don't have functions for convert numbers to words (1 -> one, 2 -> two). And if we use same format for the same field, it may be more consistent.

Is this field the age of candidate at Character and selection SCC date? I've found a logic in our existing code, but not sure if it is the right one.

Thx @HalcyonJAC - this appears to work well. Can you pls set up say 5 exercises that can be used independently at UT tomorrow (this has not been UT'd yet?) Unless we can use those exercises that you set up previously for the character annex download?

@HalcyonJAC This failed UT yesterday with the following responses from users: 'Couldn't download - network issue.' See screenshot. It worked fine for me, but multiple users experienced the issue. Can you pls test with @Franceswog who also experienced the issue above?


Parser changes are breaking new created projects

see: https://github.com/vuejs/component-compiler-utils/issues/14

@omarsotillo Looking into this now.

@omarsotillo fixed in 2.1.4

You should delete node_modules and package-lock just in case and then do a fresh npm install

Awesome! 🤗 Just wanted to let you know in case someone asks you again! I fixed myself ☺

👍


Feedback: CockroachDB in Comparison

Exalate commented:
Page: https://cockroachlabs.com/docs/v22.1/cockroachdb-in-comparison.html
What is the reason for your feedback?
[x] Missing the information I need
[ ] Too complicated
[ ] Out of date
[ ] Something is broken
[ ] Other
Additional details
The comparison is great! I thought Couchbase is better than MongoDB, no?
Jira Issue: DOC-5676

Hi @guyklages! We're glad you like it, and appreciate the thoughtful suggestion. We do plan to add Couchbase in the future, though it's not slated to happen anytime soon. Our current priority is to ensure we stay up-to-date on the improvements made to each of the already listed database products so that this resource remains accurate.


Add all available blends of each color

Available blend can be found on page 24 of the iO style guide.

[IMPORTANT] A will implement a new brand soon, creating all available blends will be a wast of time.


chore: promote jx2-python-http-01-6 to version 0.0.1

chore: promote jx2-python-http-01-6 to version 0.0.1
this commit will trigger a pipeline to generate the actual kubernetes resources to perform the promotion which will create a second commit on this Pull Request before it can merge

@rocannon77: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

verify
0944744516e770328773926f904484436930b751
link
/test verify

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository. I understand the commands that are listed here.

ss

ss


Newsletters: add 176 (2021-11-24)

[x] (Monday) Add Bitcoin Stack Exchange section @bitschmidty
[x] (Monday) C-Lightning #4890 wallet/db_sqlite3.c: Support direct replication of SQLITE3 backends @jnewbery
[x] (Monday) Rust-Lightning #1166 Penalize large HTLCs relative to channels in default Scorer @dongcarl
[ ] (Tuesday) Add topic links @harding
[ ] (Tuesday) Update releases/RCs @harding

Added c-lightning 4890. Everything else looks good so far. Thanks all!

Squashed and merged! Thanks @harding @dongcarl @jnewbery for authoring this week and to @Xekyo for reviewing and pushing the stackexchange wording PR.


fix longitude in ESRL cmorizer

ongitude in ESRL cmorizer can be wrong. The table read for it uses the -180-180 range, the negative values don't get converted but just taken an absolute value of. Added 2 lines to fix this!

Right, had to change an attribute name cause I had mistakenly added a space instead of an underscore. Will run fine until the provenance tries to do the tree diagrams, then it returns an error of "ValueError: Invalid tag name 'altitude units'".
-> Changed to underscore, ready for review now!

Right, had to change an attribute name cause I had mistakenly added a space instead of an underscore. Will run fine until the provenance tries to do the tree diagrams, then it returns an error of "ValueError: Invalid tag name 'altitude units'".
-> Changed to underscore, ready for review now!

@axel-lauer Could you have a look at this? One of my IPCC figures uses data from this cmorizer so would be nice to get this fixed. Thanks in advance!


More specific error message for SAS expiration date in the future

Feature Request
If we attempt to create a SAS with an expiration date in the future, we get a very generic error message:
<?xml version=\"1.0\" encoding=\"utf-8\"?>
<Error>
	InvalidXmlNodeValue
	<Message>The value for one of the XML nodes is not in the correct format.
	RequestId:42e93fb3-401e-002a-3388-423fee000000
	Time:2024-01-08T23:12:02.3259066Z
	</Message>
	<XmlNodeName>Expiry</XmlNodeName>
	<XmlNodeValue>2024-01-08T23:11:55Z</XmlNodeValue>
</Error>

From reading this, one would think that the problem is caused by malformed XML in the request, rather than a timestamp in the future.  It would be great if this error message could be made more specific to reflect what's actually wrong.

Here SDK is reporting the error that was sent back by Azure Backend. SDK in itself has no way to understand why certain operation failed and forwards the same reply back to user. Nothing much we can do from SDK side here.


Multiline text conversion

If a certain keyword in comment based help has a multiline description (commonly .NOTES section), generated .md file will have exactly the same context. But .md is getting presented differently, so lines will appear merged.
Steps to reproduce
Generate .md help file for the following script and take a look at .NOTES section
https://gist.github.com/9to5IT/9620683
Expected behavior
Genrated .md file should be displayed like this:
NOTES
Version:        1.0
Author:
Creation Date:
Purpose/Change: Initial script development
To achieve that, .md file should contain text like this (notice additional blank lines)
## NOTES
Version:        1.0

Author:         <Name>

Creation Date:  <Date>

Purpose/Change: Initial script development

Actual behavior
Generated .md file is displayed like this:
NOTES
Version:        1.0 Author:          Creation Date:   Purpose/Change: Initial script development
Environment data
get-module platyPS | select -ExpandProperty Version

Major  Minor  Build  Revision
-----  -----  -----  --------
0      8      3      -1

Just confirming this is in relation to New-MarkdownHelp scenario?

@BernieWhite yes, basically the ask is to treat lines in notes as separate paragraphs when generating markdown.
I'm uncertain about it, to be honest. MAML has a separate <para> tags around paragraphs and the fact that powershell Get-Help for comment based (not MAML) help doesn't add them makes it more a bug about Get-Help.
I think in general it falls into a category of things that you have to clean-up with hands after transformation (there is always be some).
A good solution may be to add an extra parameter switch New-MarkdownHelp -FromCommentHelp that can be optimized for the comment-based input and particularly preserve new lines from Notes and others.

@vors Just a thought. #75 may be a good way to resolve some of these ad-hoc cases that need special handling without increasing complexity for once off or edge cases.

This is standard CommonMark behavior (well documented in the spec). If you need preformatted or table-style presentation then you need to use markdown tables or a fenced code block.


Add E2E test for manufacturer filter

Co-authored-by: Tobias Struckmeier<EMAIL_ADDRESS>Co-authored-by: Priyanka Kore<EMAIL_ADDRESS>Co-authored-by: Ramona  Schwering<EMAIL_ADDRESS>
1. Why is this change necessary?
Because you need a test for manufacturer
2. What does this change do, exactly?
Add a new test

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

waiting for replacement of demo-data.

Hi @J-Rahe ,
thx for review and passion. Should I get the manufacturer name with a callback and do the assertion values dynamic? So far I do not know how I can help here.
See you
Roland

@rogoit, sorry to confuse you. @leichteckig wanted to continue on this, i thought you were informed.

@rogoit Would you like to squash and rebase our MR? 🙂

Added a changelog now, so we might only need to squash the commits. 🙂

Hello,
thank you for creating this pull request.
I have opened an issue on our Issue Tracker for you. See the issue link: https://issues.shopware.com/issues/NEXT-15870
Please use this issue to track the state of your pull request.

Hi,
and thx all for your help to get our first PR done. So I do not know where I can answer the questions. I do it here and hope it helps.
Why is this change necessary?
The test will help to get a wider application test coverage for the filter on the product overview page. With this initial test we can go forward to do more validations for the correct working.
What does this change do, exactly?
The tests opens the startpage and click the dropdown of the manufacturer. Here it select "Shopware" and check if there are less products. This will validate the working of the filter. Also the test assert that the manufacturer of the filtered item has the selected brand.
Can I do anything else for you?
See you
Roland

Hey @rogoit, your PR was just merged : ) Thanks!


*: release 0.24.0

Adjust the CHANGELOG accordingly for 0.23.2.
Signed-off-by: Giedrius Statkevičius<EMAIL_ADDRESS>

@GiedriusS I say let's close this in favor of https://github.com/thanos-io/thanos/pull/4984, which updates the tutorials and other files. Does that sound ok?


Incorrect statement in documentation

The statements below which are located the "Which option to select" section are incorrect.
They are incorrect because by default, the "Allow members to share the site and individual files and folders" setting is enabled on ALL site collections, which means that users in the "Members" group (that does not have full control permission) can share sites, files and folders.
"Site owners or others with full control permission can share sites with external users."
"Site owners or others with full control permissions on a site can share documents with external users."
Should be
"Site owners and members can share sites with external users."
"Site owners and members of a site can share documents with external users."

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 6ed46514-8036-a543-d690-73d48acef690
Version Independent ID: 8542c3ce-3012-3d23-c3e4-4331dac033d2
Content: Turn external sharing on or off for SharePoint Online
Content Source: SharePoint/SharePointOnline/turn-external-sharing-on-or-off.md
Service: sharepoint-online
GitHub Login: @MikePlumleyMSFT
Microsoft Alias: mikeplum

Sorry for my delayed response on this and Closed #570https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2FOfficeDocs-SharePoint%2Fissues%2F570&data=02|01|jvasil%40microsoft.com|d2c40abae9f9408847cb08d68d5fe451|72f988bf86f141af91ab2d7cd011db47|1|0|636851840322529155&sdata=HCuKhxwVSrWZ1Ff1W4VJtfDN7Qv124yOa2qTWJ4jrtE%3D&reserved=0.
This point is that by default “Members” can share with external users.. and by default Members do NOT have full control permission.
Therefore the statements below which are located the "Which option to select" section are incorrect of that document are incorrect.
They are incorrect because by default, the "Allow members to share the site and individual files and folders" setting is enabled on ALL site collections, which means that users in the "Members" group (that does not have full control permission) can share sites, files and folders.
"Site owners or others with full control permission can share sites with external users."
"Site owners or others with full control permissions on a site can share documents with external users."
Should be
"Site owners and members can share sites with external users."
"Site owners and members of a site can share documents with external users."
From: Trevor Seward<EMAIL_ADDRESS>Sent: Wednesday, January 16, 2019 10:39 AM
To: MicrosoftDocs/OfficeDocs-SharePoint<EMAIL_ADDRESS>Cc: Joseph Vasil<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/OfficeDocs-SharePoint] Incorrect statement in documentation (#570)
In this case, the documentation is indeed correct. Sharing can be done with users who not external users and in AAD by site members, as you noted. The controls mentioned in the documentation are very specific to external users/anonymous links, however.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2FOfficeDocs-SharePoint%2Fissues%2F570%23issuecomment-454824481&data=02|01|jvasil%40microsoft.com|35014b456e99413d6d0c08d67bc8c59c|72f988bf86f141af91ab2d7cd011db47|1|0|636832499578081765&sdata=H8w%2ByKf7ms4n2F9mmHo2dz5c%2B2yp253yUQhjOoaMfVo%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FALim384LkJkH8x3K4XWvT6AhVQv6rzkIks5vD0ejgaJpZM4aB0V4&data=02|01|jvasil%40microsoft.com|35014b456e99413d6d0c08d67bc8c59c|72f988bf86f141af91ab2d7cd011db47|1|0|636832499578081765&sdata=hMIFE8QWfptfZ1cOori0MsHu5aIou94bEgF60PSxbUM%3D&reserved=0.

@Jvasil but keep in mind the context of the article. The context of the article is about SPO Admin Center settings, not the per-site setting you're referring to.

Yes I know and realize that. I believe your  missing my point.. with the highlighted setting selected at tenant level, the highlighted statement is incorrect because “site Members” can also share with external users.
Site Members do not have full control permission, nor are they Site Owners. However they can still share a site, they are a member of, with external users.
New and existing external users (sign-in required)
Require external users who have received invitations to view sites or content to sign-in with a Microsoft account before they can access the content.
Site owners or others with full control permission can share sites with external users.
Site owners or others with full control permissions on a site can share documents with external users.
All external users will be required to sign in before they can view content.
Invitations to view content can be redeemed only once. After an invitation has been accepted, it cannot be shared or used by others to gain access.
Joe Vasil
Senior Premier Field Engineer
Office 365 - SharePoint Online
US PFE - East Region
Office: +1<PHONE_NUMBER>
jvasil<EMAIL_ADDRESS>[MSFT_logo]
From: Trevor Seward<EMAIL_ADDRESS>Sent: Friday, February 8, 2019 4:10 PM
To: MicrosoftDocs/OfficeDocs-SharePoint<EMAIL_ADDRESS>Cc: Joseph Vasil<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/OfficeDocs-SharePoint] Incorrect statement in documentation (#570)
@Jvasilhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FJvasil&data=02|01|jvasil%40microsoft.com|0cc40c8a0cea453697ad08d68e09cd3f|72f988bf86f141af91ab2d7cd011db47|1|0|636852570086671170&sdata=sJtTg%2Fd%2B14B0UEZhhFbGizJ92wuirGsQQwDoY%2FI13p4%3D&reserved=0 but keep in mind the context of the article. The context of the article is about SPO Admin Center settings, not the per-site setting you're referring to.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2FOfficeDocs-SharePoint%2Fissues%2F570%23issuecomment-461947843&data=02|01|jvasil%40microsoft.com|0cc40c8a0cea453697ad08d68e09cd3f|72f988bf86f141af91ab2d7cd011db47|1|0|636852570086681166&sdata=ykY7EjdjmQLh9CNFpIQ046vxm4v7MyHDDGSO8aHANqg%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FALim30B0m7PAiuHyZ-56GwtynvZTblCmks5vLeeugaJpZM4aB0V4&data=02|01|jvasil%40microsoft.com|0cc40c8a0cea453697ad08d68e09cd3f|72f988bf86f141af91ab2d7cd011db47|1|0|636852570086681166&sdata=xRG2yUBtC0qDfwR6eofFFkTt1mi0d0ATkody9V837bk%3D&reserved=0.

@Jvasil, agreed, the current text is incorrect for the default settings. I've updated the article.

Excellent! Thanks Mike
Get Outlook for Androidhttps://aka.ms/ghei36

From: Mike Plumley<EMAIL_ADDRESS>Sent: Monday, February 11, 2019 7:05:30 AM
To: MicrosoftDocs/OfficeDocs-SharePoint
Cc: Joseph Vasil; Mention
Subject: Re: [MicrosoftDocs/OfficeDocs-SharePoint] Incorrect statement in documentation (#570)
@Jvasilhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FJvasil&data=02|01|jvasil%40microsoft.com|10fef17a250e4d301e1008d690325d9d|72f988bf86f141af91ab2d7cd011db47|1|0|636854943328206538&sdata=IbR%2BS%2FCtaBZBwQYCIW3hlqGaaI7ALWZtwSy5lhRrzsQ%3D&reserved=0, agreed, the current text is incorrect for the default settings. I've updated the article.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2FOfficeDocs-SharePoint%2Fissues%2F570%23issuecomment-462360459&data=02|01|jvasil%40microsoft.com|10fef17a250e4d301e1008d690325d9d|72f988bf86f141af91ab2d7cd011db47|1|0|636854943328206538&sdata=87hssB3WtCrceS0ZhcFl9fSo%2BAOELE%2FtDkc%2B1YJCSt8%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FALim3xV-1Vor_P7TInpAzo7tTC3j_6cyks5vMYa6gaJpZM4aB0V4&data=02|01|jvasil%40microsoft.com|10fef17a250e4d301e1008d690325d9d|72f988bf86f141af91ab2d7cd011db47|1|0|636854943328216533&sdata=EY4lcZ8rrgcnykUn17QbFvlqZF2ZFFVFV%2BzewucE9kk%3D&reserved=0.


DDP address in use

When I try to run more than one trainings (with a single job) on the same machine, I get this:
Traceback (most recent call last):
  File "mmi_bigram_train.py", line 475, in <module>
    main()
  File "mmi_bigram_train.py", line 238, in main
    setup_dist(rank=args.local_rank, world_size=args.world_size)
  File "/ceph-dan/.local/lib/python3.8/site-packages/snowfall-0.1-py3.8.egg/snowfall/dist.py", line 9, in setup_dist
  File "/ceph-dan/.local/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 423, in init_process_group
    store, rank, world_size = next(rendezvous_iterator)
  File "/ceph-dan/.local/lib/python3.8/site-packages/torch/distributed/rendezvous.py", line 179, in _env_rendezvous_handler
    store = TCPStore(master_addr, master_port, world_size, start_daemon, timeout)
RuntimeError: Address already in use

this is with mmi_bigram_train.py, with --world-size=1, --local-rank=0

... i.e. the defaults, no extra args.

We might need to make the port configurable. For a quick work-around you can change it here:
https://github.com/k2-fsa/snowfall/blob/master/snowfall/dist.py#L8

yeah I did that.
can't we randomly choose it in a range?  or does that create problems for co-ordination?
is the user just supposed to launch multiple copies of the job?  Or does it spawn?

We can choose it randomly - although I think with torch.distributed.launch we'd have to choose it outside of the python script, and with torch.distributed.spawn we can choose it inside the python script, before spawning jobs. All the spawned tasks would have to see the same address + port. It might be easier to implement with spawn that @csukuangfj has been using in another PR.

Getting this error:
de-74279-k2-dev-2-0331181900-7b69767657-72fhf:simple_v1: ngpus=2
de-74279-k2-dev-2-0331181900-7b69767657-72fhf:simple_v1: python3 -m torch.distributed.launch --nproc_per_node=$ngpus ./mmi_bigram_train.py --world_size $ngpus &
[1] 1489648
de-74279-k2-dev-2-0331181900-7b69767657-72fhf:simple_v1: *****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your app\
lication as needed.
*****************************************

de-74279-k2-dev-2-0331181900-7b69767657-72fhf:simple_v1: World size: 2 Rank: 0
World size: 2 Rank: 1

de-74279-k2-dev-2-0331181900-7b69767657-72fhf:simple_v1: Traceback (most recent call last):
Traceback (most recent call last):
  File "./mmi_bigram_train.py", line 474, in <module>
  File "./mmi_bigram_train.py", line 474, in <module>
    main()
  File "./mmi_bigram_train.py", line 238, in main
    main()
  File "./mmi_bigram_train.py", line 238, in main
    setup_dist(rank=args.local_rank, world_size=args.world_size)
  File "/ceph-dan/.local/lib/python3.8/site-packages/snowfall-0.1-py3.8.egg/snowfall/dist.py", line 9, in setup_dist
    setup_dist(rank=args.local_rank, world_size=args.world_size)
  File "/ceph-dan/.local/lib/python3.8/site-packages/snowfall-0.1-py3.8.egg/snowfall/dist.py", line 9, in setup_dist
  File "/ceph-dan/.local/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 442, in init_process_group
  File "/ceph-dan/.local/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 442, in init_process_group
    barrier()
  File "/ceph-dan/.local/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1947, in barrier
    barrier()
  File "/ceph-dan/.local/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1947, in barrier
    work = _default_pg.barrier()
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:784, invalid usage, NCCL version 2.7.8
    work = _default_pg.barrier()
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:784, invalid usage, NCCL version 2.7.8
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/ceph-dan/.local/lib/python3.8/site-packages/torch/distributed/launch.py", line 260, in <module>
    main()
  File "/ceph-dan/.local/lib/python3.8/site-packages/torch/distributed/launch.py", line 255, in main
    raise subprocess.CalledProcessError(returncode=process.returncode,
subprocess.CalledProcessError: Command '['/usr/bin/python3', '-u', './mmi_bigram_train.py', '--local_rank=1', '--world_size', '2']' returned non-zero exit status 1.

Hmm, I've never seen this one before...

I found something about why, when we use torch.distributed.launch, it was hanging at the end.  (Caution: my lhotse was not fully up to date, although sampling.py doesn't seem to have changed in the interim).
Firstly, at startup, I got a bunch of messages like this (I was confused with how many there were):
2021-04-10 23:35:58,870 INFO [mmi_bigram_train.py:274] About to create train dataset
2021-04-10 23:35:59,509 INFO [mmi_bigram_train.py:284] Using BucketingSampler.
2021-04-10 23:35:59,813 INFO [sampling.py:523] Distributed training with world size of 2 detected (node's local rank is 0. Splitting cuts into 2 partitions (this partition has cut IDs range\
 [(0, 1427)].
2021-04-10 23:35:59,814 INFO [sampling.py:523] Distributed training with world size of 2 detected (node's local rank is 0. Splitting cuts into 2 partitions (this partition has cut IDs range\
 [(0, 1427)].
2021-04-10 23:35:59,814 INFO [sampling.py:523] Distributed training with world size of 2 detected (node's local rank is 0. Splitting cuts into 2 partitions (this partition has cut IDs range\
 [(0, 1427)].
2021-04-10 23:35:59,814 INFO [sampling.py:523] Distributed training with world size of 2 detected (node's local rank is 0. Splitting cuts into 2 partitions (this partition has cut IDs range\
 [(0, 1427)].
2021-04-10 23:35:59,815 INFO [sampling.py:523] Distributed training with world size of 2 detected (node's local rank is 0. Splitting cuts into 2 partitions (this partition has cut IDs range\
 [(0, 1427)].
2021-04-10 23:35:59,815 INFO [sampling.py:523] Distributed training with world size of 2 detected (node's local rank is 0. Splitting cuts into 2 partitions (this partition has cut IDs range\
 [(0, 1427)].
2021-04-10 23:35:59,815 INFO [sampling.py:523] Distributed training with world size of 2 detected (node's local rank is 0. Splitting cuts into 2 partitions (this partition has cut IDs range\
 [(0, 1427)].
2021-04-10 23:35:59,816 INFO [sampling.py:523] Distributed training with world size of 2 detected (node's local rank is 0. Splitting cuts into 2 partitions (this partition has cut IDs range\
 [(0, 1427)].
2021-04-10 23:35:59,816 INFO [sampling.py:523] Distributed training with world size of 2 detected (node's local rank is 0. Splitting cuts into 2 partitions (this partition has cut IDs range\
 [(0, 1427)].
2021-04-10 23:35:59,816 INFO [sampling.py:523] Distributed training with world size of 2 detected (node's local rank is 0. Splitting cuts into 2 partitions (this partition has cut IDs range\
 [(0, 1427)].
2021-04-10 23:35:59,816 INFO [sampling.py:523] Distributed training with world size of 2 detected (node's local rank is 0. Splitting cuts into 2 partitions (this partition has cut IDs range\
 [(0, 1427)].
2021-04-10 23:35:59,817 INFO [sampling.py:523] Distributed training with world size of 2 detected (node's local rank is 0. Splitting cuts into 2 partitions (this partition has cut IDs range\
 [(0, 1427)].
2021-04-10 23:35:59,817 INFO [sampling.py:523] Distributed training with world size of 2 detected (node's local ran
...
2021-04-10 23:35:59,820 INFO [sampling.py:523] Distributed training with world size of 2 detected (node's local rank is 0. Splitting cuts into 2 partitions (this partition has cut IDs range\
 [(0, 1427)].
2021-04-10 23:35:59,820 INFO [sampling.py:523] Distributed training with world size of 2 detected (node's local rank is 0. Splitting cuts into 2 partitions (this partition has cut IDs range\
 [(0, 1427)].
2021-04-10 23:35:59,820 INFO [sampling.py:523] Distributed training with world size of 2 detected (node's local rank is 0. Splitting cuts into 2 partitions (this partition has cut IDs range\
 [(0, 1427)].
2021-04-10 23:35:59,821 INFO [sampling.py:523] Distributed training with world size of 2 detected (node's local rank is 0. Splitting cuts into 2 partitions (this partition has cut IDs range\
 [(0, 1427)].
2021-04-10 23:35:59,821 INFO [sampling.py:523] Distributed training with world size of 2 detected (node's local rank is 0. Splitting cuts into 2 partitions (this partition has cut IDs range\
 [(0, 1427)].
2021-04-10 23:35:59,821 INFO [sampling.py:523] Distributed training with world size of 2 detected (node's local rank is 0. Splitting cuts into 2 partitions (this partition has cut IDs range\
 [(0, 1427)].
2021-04-10 23:35:59,822 INFO [sampling.py:523] Distributed training with world size of 2 detected (node's local rank is 0. Splitting cuts into 2 partitions (this partition has cut IDs range\
 [(0, 1426)].
2021-04-10 23:35:59,824 INFO [mmi_bigram_train.py:298] About to create train dataloader

.. .that's just FYI.  I am confused why it never says the local rank is 1, and why the last message says 1426.
I am using world-size=2.  The rank=1 process doesn't print much logs, but I does print some, which are printed near the start of the log file for some reason, and not at all in sync with where you'd expect them to be:
2021-04-10 23:36:00,287 INFO [mmi_bigram_train.py:401] epoch 0, learning rate 0.001
2021-04-10 23:36:02,585 INFO [mmi_bigram_train.py:176] batch 0, epoch 0/10 global average 2021-04-10 23:36:02,597 INFO [distributed.py:607] Reducer buckets have been rebuilt in this iterati\
on.
2021-04-11 00:06:37,689 INFO [common.py:201] write training info to exp-lstm-adam-mmi-bigram-musan-dist/best-epoch-info
2021-04-11 00:06:37,689 INFO [common.py:201] write training info to exp-lstm-adam-mmi-bigram-musan-dist/epoch-0-info
2021-04-11 00:06:37,690 INFO [mmi_bigram_train.py:401] epoch 1, learning rate 0.001
2021-04-11 00:37:08,277 INFO [common.py:201] write training info to exp-lstm-adam-mmi-bigram-musan-dist/best-epoch-info
2021-04-11 00:37:08,277 INFO [common.py:201] write training info to exp-lstm-adam-mmi-bigram-musan-dist/epoch-1-info
2021-04-11 00:37:08,278 INFO [mmi_bigram_train.py:401] epoch 2, learning rate 0.001
2021-04-11 01:07:37,757 INFO [common.py:201] write training info to exp-lstm-adam-mmi-bigram-musan-dist/best-epoch-info
2021-04-11 01:07:37,757 INFO [common.py:201] write training info to exp-lstm-adam-mmi-bigram-musan-dist/epoch-2-info
2021-04-11 01:07:37,758 INFO [mmi_bigram_train.py:401] epoch 3, learning rate 0.001
2021-04-11 01:38:04,478 INFO [common.py:201] write training info to exp-lstm-adam-mmi-bigram-musan-dist/best-epoch-info
2021-04-11 01:38:04,478 INFO [common.py:201] write training info to exp-lstm-adam-mmi-bigram-musan-dist/epoch-3-info
2021-04-11 01:38:04,478 INFO [mmi_bigram_train.py:401] epoch 4, learning rate 0.001
2021-04-11 02:08:40,364 INFO [common.py:201] write training info to exp-lstm-adam-mmi-bigram-musan-dist/best-epoch-info
2021-04-11 02:08:40,365 INFO [common.py:201] write training info to exp-lstm-adam-mmi-bigram-musan-dist/epoch-4-info
2021-04-11 02:08:40,365 INFO [mmi_bigram_train.py:401] epoch 5, learning rate 0.001
2021-04-11 02:39:10,325 INFO [common.py:201] write training info to exp-lstm-adam-mmi-bigram-musan-dist/best-epoch-info
2021-04-11 02:39:10,326 INFO [common.py:201] write training info to exp-lstm-adam-mmi-bigram-musan-dist/epoch-5-info
2021-04-11 02:39:10,326 INFO [mmi_bigram_train.py:401] epoch 6, learning rate 0.001
2021-04-11 03:09:50,414 INFO [common.py:201] write training info to exp-lstm-adam-mmi-bigram-musan-dist/best-epoch-info
2021-04-11 03:09:50,415 INFO [common.py:201] write training info to exp-lstm-adam-mmi-bigram-musan-dist/epoch-6-info
2021-04-11 03:09:50,415 INFO [mmi_bigram_train.py:401] epoch 7, learning rate 0.0008
2021-04-11 03:36:48,757 WARNING [cut.py:1023] To perform mix, energy must be non-zero and non-negative (got 0.0).  Cut with id "c1c0349b-2b4f-3277-d8a3-c1a701cc4c32" will not be mixed in.
2021-04-11 03:40:24,571 INFO [common.py:201] write training info to exp-lstm-adam-mmi-bigram-musan-dist/epoch-7-info
2021-04-11 03:40:24,571 INFO [mmi_bigram_train.py:401] epoch 8, learning rate 0.0006400000000000002
2021-04-11 04:11:20,545 INFO [common.py:201] write training info to exp-lstm-adam-mmi-bigram-musan-dist/epoch-8-info
2021-04-11 04:11:20,546 INFO [mmi_bigram_train.py:401] epoch 9, learning rate 0.0005120000000000001
2021-04-11 04:42:13,339 INFO [common.py:201] write training info to exp-lstm-adam-mmi-bigram-musan-dist/best-epoch-info
2021-04-11 04:42:13,340 INFO [common.py:201] write training info to exp-lstm-adam-mmi-bigram-musan-dist/epoch-9-info
2021-04-11 04:42:13,340 WARNING [mmi_bigram_train.py:466] Done
erage objf: 1.353939 over 1285235.0 frames (100.0% kept), current batch average objf: 1.197140 over 9905 frames (100.0% kept) avg time waiting for batch 0.012s
2021-04-10 23:39:06,673 INFO [mmi_bigram_train.py:176] batch 140, epoch 0/10 global average objf: 1.346037 over 1383290.0 frames (100.0% kept), current batch average objf: 1.175396 over 974\
1 frames (100.0% kept) avg time waiting for batch 0.012s

Anyway, when I check the times of where it starts each epoch, this rank=1 job does not seem to be correctly synchronized with the rank=0 job.  It finishes epoch 0 when the rank=0 job has only finished about 90% of its minibatches.

Wouldn't it be easier, in order to support distributed training, to just have the sampler process things as normal and then return 1 out of every world_size minibatches?  The time that it processes that metadata will probably overlap with GPU stuff anyway, I don't really think that's going to be the limiting factor.
That way we can more easily ensure the number of minibatches is exactly the same between jobs.

That way we can more easily ensure the number of minibatches is exactly the same between jobs.

I confirm that if different nodes have a different number of utterances in its dataloader, the node with most utterances will hang in the end.
I suspect the reason is due to allreduce called inside backward. Since nodes with fewer utterances
exit earlier, the node with most utterances waits inside allreduce; but it does not receive responses from other already exited nodes. Therefore, it waits indefinitely.
A minimal example to reproduce it is given below.
The current approach to partition the dataset over different nodes cannot guranteen that every node receives the same amount of utterances. See
https://github.com/lhotse-speech/lhotse/blob/3c6dea9e90536e01b6ce7b937682ab85ae50d680/lhotse/dataset/sampling.py#L519-L522
        total = len(data_source)
        per_partition = int(ceil(total / float(world_size)))
        partition_start = rank * per_partition
        partition_end = min(partition_start + per_partition, total)

The node with the largest rank value receives fewer if  total % world_size != 0.
That may explain why the node with rank==0 hangs in the end.

#!/usr/bin/env python3

import os
import torch.distributed as dist
import torch.multiprocessing as mp
from torch.nn.parallel import DistributedDataParallel as DDP

import torch

import datetime

def run(rank: int, world_size: int):
    print(f'world_size: {world_size}')
    device = torch.device('cuda', rank)
    if rank != 0:
        data = [torch.tensor([1], device=device, dtype=torch.float32) for _ in range(world_size)]
    else:
        data = [torch.tensor([1], device=device, dtype=torch.float32) for _ in range(world_size*100)]
    # NOTE: `data` on rank 0 has more entries
    dist.barrier()

    model = torch.nn.Linear(1, 1).to(device)
    model = DDP(model, device_ids=[rank])
    for i, d in enumerate(data):
        model.zero_grad()
        y = model(d)
        y.backward()

    print(f'rank {rank} done')
    # node with rank==0 will exit after timeout (5 seconds)
    # The default time out is 30 minutes. But it comes into effect
    # only if one of the following environment variable is
    # set:
    #  - NCCL_ASYNC_ERROR_HANDLING
    #  - NCCL_ASYNC_ERROR_HANDLING
    # See https://pytorch.org/docs/stable/distributed.html#torch.distributed.init_process_group

def init_process(rank: int, world_size: int, fn):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12357'
    dist.init_process_group('nccl',
                            rank=rank,
                            world_size=world_size,
                            timeout=datetime.timedelta(0, 5))
    fn(rank, world_size)

if __name__ == '__main__':
    print(f'dist.is_available: {dist.is_available()}')
    world_size = 3
    processes = []
    mp.set_start_method('spawn')
    for rank in range(world_size):
        p = mp.Process(target=init_process, args=(rank, world_size, run))
        p.start()
        processes.append(p)

    for p in processes:
        p.join()

Its output is:
(NOTE: It has to set the environment variable NCCL_ASYNC_ERROR_HANDLING=1)
$ NCCL_ASYNC_ERROR_HANDLING=1 ./foo.py
dist.is_available: True
world_size: 3
world_size: 3
world_size: 3
rank 1 done
rank 2 done
Process Process-1:
Traceback (most recent call last):
  File "/root/fangjun/open-source/pyenv/versions/3.8.6/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/root/fangjun/open-source/pyenv/versions/3.8.6/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/root/fangjun/open-source/.n/pytorch/code/ddp/ex3.py", line 47, in init_process
    fn(rank, world_size)
  File "/root/fangjun/open-source/.n/pytorch/code/ddp/ex3.py", line 28, in run
    y.backward()
  File "/root/fangjun/py38/lib/python3.8/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/root/fangjun/py38/lib/python3.8/site-packages/torch/autograd/__init__.py", line 130, in backward
    Variable._execution_engine.run_backward(
RuntimeError: NCCL communicator was aborted.

@pzelasko
How about dropping some utterances in partition_cut_ids
so that all nodes get the same number of utterances?

Oooh, now it all finally makes sense. Thanks for debugging this guys. I'll add a fix to the cut ids partitioning in the sampler.

I'm going to use @danpovey's solution rather than @csukuangfj's solution -- unfortunately, it is not straightforward to estimate how many utterances should be dropped in partition_cut_ids, since we have a dynamic batch size. That means that equal number of cuts for each partition doesn't guarantee equal number of batches due to variations in duration and max_duration constraint. I think that dropping all batches except sth. like batch_idx + rank % world_size is much more viable. I'll test it and make a PR in Lhotse.

@danpovey @csukuangfj can you please try out the version in PR https://github.com/lhotse-speech/lhotse/pull/267 and let me know if it helped? I won't be able to test the snowfall distributed training setup today, but based on the unit tests I wrote it seems to have fixed the issues with an unequal number of batches in each worker.

@pzelasko
Thanks. Trying it.

@pzelasko
I confirm that the current change can solve the hanging problem.
Here is the tensorboard log of DDP training with 3 GPUs:

https://tensorboard.dev/experiment/UGf0fbC0QianY9WyeghJpQ/

And the WERs are
2021-04-13 09:12:14,242 INFO [common.py:356] [test-clean] %WER 7.39% [3885 / 52576, 505 ins, 330 del, 3050 sub ]
2021-04-13 09:14:29,223 INFO [common.py:356] [test-other] %WER 18.82% [9849 / 52343, 1149 ins, 863 del, 7837 sub ]

The WERs are worse than that of single GPU training. I believe the reason is due to the learning rate.
You can compare the learning rate from the above tensorboard log with the one from single GPU training.
I believe if we train it for more epochs, it can achieve similar results.
NOTE: The training time per epoch with 3 GPUs is about 16 minutes, which is about 1/3 of single GPU training.

Great!!
I think in order to get comparable results to the baseline we'd have to
divide the minibatch size by the number of workers.
Let's merge this?
On Tue, Apr 13, 2021 at 9:21 AM Fangjun Kuang @.***>
wrote:

@pzelasko https://github.com/pzelasko
I confirm that the current change can solve the hanging problem.
Here is the tensorboard log of DDP training with 3 GPUs:

https://tensorboard.dev/experiment/UGf0fbC0QianY9WyeghJpQ/

And the WERs are
2021-04-13 09:12:14,242 INFO [common.py:356] [test-clean] %WER 7.39% [3885 / 52576, 505 ins, 330 del, 3050 sub ]
2021-04-13 09:14:29,223 INFO [common.py:356] [test-other] %WER 18.82% [9849 / 52343, 1149 ins, 863 del, 7837 sub ]
The WERs are worse than that of single GPU training. I believe the reason
is due to the learning rate.
You can compare the learning rate from the above tensorboard log with the
one from single GPU training
https://tensorboard.dev/experiment/h3xiWY0oQ4WGd2dRgG8NWw.
I believe if we train it for more epochs, it can achieve similar results.
NOTE: The training time per epoch with 3 GPUs is about 16 minutes, which
is about 1/3 of single GPU training.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/k2-fsa/snowfall/issues/152#issuecomment-818361538,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAZFLO3I4TUSDORV2AUBS6TTIOMBBANCNFSM42WSEJPA
.

Oh it's a PR to lhotse, we'll wait for Piotr to merge.

Merged!

FYI this could be of interest to us https://huggingface.co/blog/accelerate-library


[Bug Report] Bot joins but then leaves

Bug Description
Okay so the bot works in my own server perfectly fine but when i invite it to my friends server i got admin and stuff and try to play a song it joins the call then leave instantly and i dont know why is there anyway to fix that
https://user-images.githubusercontent.com/95266355/143989355-8565fc91-40c1-4d0c-b2bb-fa83beb154ce.mp4
.
Steps to Reproduce
bot leaves and joins back
Expected Result
idk.
Debug Output
recent version
and heres my config

/////////////////////////////////////////////////////////
// Config for the JMusicBot                            //
/////////////////////////////////////////////////////////
// Any line starting with // is ignored                //
// You MUST set the token and owner                    //
// All other items have defaults if you don't set them //
// Open in Notepad++ for best results                  //
/////////////////////////////////////////////////////////

// This sets the token for the bot to log in with
// This MUST be a bot token (user tokens will not work)
// If you don't know how to get a bot token, please see the guide here:
// https://github.com/jagrosh/MusicBot/wiki/Getting-a-Bot-Token

token = my token is here just dont want to leak it

// This sets the owner of the bot
// This needs to be the owner's ID (a 17-18 digit number)
// https://github.com/jagrosh/MusicBot/wiki/Finding-Your-User-ID

owner = my id here just dont wanna leak it

// This sets the prefix for the bot
// The prefix is used to control the commands
// If you use !!, the play command will be !!play
// If you do not set this, the prefix will be a mention of the bot (@Botname play)

prefix = ";"

// If you set this, it modifies the default game of the bot
// Set this to NONE to have no game
// Set this to DEFAULT to use the default game
// You can make the game "Playing X", "Listening to X", or "Watching X"
// where X is the title. If you don't include an action, it will use the
// default of "Playing"

game = "DEFAULT"

// If you set this, it will modify the default status of bot
// Valid values: ONLINE IDLE DND INVISIBLE

status = ONLINE

// If you set this to true, the bot will list the title of the song it is currently playing in its
// "Playing" status. Note that this will ONLY work if the bot is playing music on ONE guild;
// if the bot is playing on multiple guilds, this will not work.

songinstatus=false

// If you set this, the bot will also use this prefix in addition to
// the one provided above

altprefix = "NONE"

// If you set these, it will change the various emojis

success = "🎶"
warning = "💡"
error = "🚫"
loading = "⌚"
searching = "🔎"

// If you set this, you change the word used to view the help.
// For example, if you set the prefix to !! and the help to cmds, you would type
// !!cmds to see the help text

help = help

// If you set this, the "nowplaying" command will show youtube thumbnails
// Note: If you set this to true, the nowplaying boxes will NOT refresh
// This is because refreshing the boxes causes the image to be reloaded
// every time it refreshes.

npimages = false

// If you set this, the bot will not leave a voice channel after it finishes a queue.
// Keep in mind that being connected to a voice channel uses additional bandwith,
// so this option is not recommended if bandwidth is a concern.

stayinchannel = false

// This sets the maximum amount of seconds any track loaded can be. If not set or set
// to any number less than or equal to zero, there is no maximum time length. This time
// restriction applies to songs loaded from any source.

maxtime = 0

// This sets the amount of seconds the bot will stay alone on a voice channel until it
// automatically leaves the voice channel and clears the queue. If not set or set
// to any number less than or equal to zero, the bot won't leave when alone.

alonetimeuntilstop = 0

// This sets an alternative folder to be used as the Playlists folder
// This can be a relative or absolute path

playlistsfolder = "Playlists"

// By default, the bot will DM the owner if the bot is running and a new version of the bot
// becomes available. Set this to false to disable this feature.

updatealerts=true

// Changing this changes the lyrics provider
// Currently available providers: "A-Z Lyrics", "Genius", "MusicMatch", "LyricsFreak"
// At the time of writing, I would recommend sticking with A-Z Lyrics or MusicMatch,
// as Genius tends to have a lot of non-song results and you might get something
// completely unrelated to what you want.
// If you are interested in contributing a provider, please see
// https://github.com/jagrosh/JLyrics

lyrics.default = "A-Z Lyrics"

// These settings allow you to configure custom aliases for all commands.
// Multiple aliases may be given, separated by commas.
//
// Example 1: Giving command "play" the alias "p":
// play = [ p ]
//
// Example 2: Giving command "search" the aliases "yts" and "find":
// search = [ yts, find ]

aliases {
  // General commands
  settings = [ status ]

  // Music commands
  lyrics = []
  nowplaying = [ np, current ]
  play = []
  playlists = [ pls ]
  queue = [ list ]
  remove = [ delete ]
  scsearch = []
  search = [ ytsearch ]
  shuffle = []
  skip = [ voteskip ]

  // Admin commands
  prefix = [ setprefix ]
  setdj = []
  setskip = [ setskippercent, skippercent, setskipratio ]
  settc = []
  setvc = []

  // DJ Commands
  forceremove = [ forcedelete, modremove, moddelete, modelete ]
  forceskip = [ modskip ]
  movetrack = [ move ]
  pause = []
  playnext = []
  repeat = []
  skipto = [ jumpto ]
  stop = [ leave ]
  volume = [ vol ]
}

// Transforms are used to modify specific play inputs and convert them to different kinds of inputs
// These are quite complicated to use, and have limited use-cases, but in theory allow for rough
// whitelists or blacklists, roundabout loading from some sources, and customization of how things are
// requested.
//
// These are NOT EASY to set up, so if you want to use these, you'll need to look through the code
// for how they work and what fields are needed. Also, it's possible this feature might get entirely
// removed in the future if I find a better way to do this.

transforms = {}

// If you set this to true, it will enable the eval command for the bot owner. This command
// allows the bot owner to run arbitrary code from the bot's account.
//
// WARNING:
// This command can be extremely dangerous. If you don't know what you're doing, you could
// cause horrific problems on your Discord server or on whatever computer this bot is running
// on. Never run this command unless you are completely positive what you are running.
//
// DO NOT ENABLE THIS IF YOU DON'T KNOW WHAT THIS DOES OR HOW TO USE IT
// IF SOMEONE ASKS YOU TO ENABLE THIS, THERE IS AN 11/10 CHANCE THEY ARE TRYING TO SCAM YOU

eval=false

Additional Info
No response
Checklist

[X] I have looked for information about this within the documentation
[X] I have searched for similar issues on the issues page
[X] I am running the latest version of the bot:

i know its been months but im having this too

This happens with tracks that have explicit lyrics, or tracks that aren't available in your region. In both cases YouTube displays a warning rather than providing the video. I'm not sure what a good solution would be for the latter case, but for the former case it should be possible to have jmusicbot automate the age verification.

Assuming this issue is stale/resolved.
If this problem occurs again, please open a new issue.


Connection problems

I'm experimenting a lot of connection issue since today 13:30. Basically tdlib client cannot connext to telegram server and the state change between connecting and connected and disconnected continuoilusly
Are there some telegram server issue? Or it's my connection problem?

There was no any issues in the last few hours..

So it's very strange because I'm receiving thousand of mwssages like
updateConnectionState {
state = connectionStateUpdating {
}
}
updateConnectionState {
state = connectionStateReady {
}
}
updateConnectionState {
state = connectionStateConnecting {
}
}
In last 2 hours, sometimes it's ok, and sometimes give me
UpdateMessageSendFail
With retry_after
I don't know what is going on but was all warking until 13:30

@levlam about retry_after it's a time you should wait before send any new message or a message to that id?
the strange thing is that I see even retry_after 2400 seconds....probably somthing is somehow heppening with telegeam because never seen that error

3 hours....same problems

@micheleselea This particular message. A failed to send message can be retried to send with resendMessages.

@micheleselea Send the full log with verbosity level 4 to @tdlib_bot, so I can tell you the cause.

I cannot now, I have to wayt monday I just hope the problem will be fixed before monday...

The problem is still there... I think that is the worst service issue since I use telegram for bot so since 2015. I think I have to find another sistem to make my notification more reliable...I'm realy sorry about because I like telegram and I like bot and tdlib but I cannot have all that problems...

@levlam what I can'rlt figure out is that: does tdlib can be use to develop a BOT that has thousand of users? Because ij 2/3 months of use I hit a lot of constraint and limitation that make me think the lib are only for "user like" use....it's a pity if so, because BOT are great but they do not suport secret chat, tdlib was to be the perfect solution but I thino cannot do what I have to do..I'm I wrong?

@micheleselea TDLib can handle millions of chats simultaneously, so thousands of users shouldn't be a problem for it. But usage by ordinary users is still the primary use case for TDLib.

Ok so, suppose your tdlib client have to send  1000messages every minutes to your thousand client, I mean 1000 secretchat messages...I think today I reach the limit because the server could bot satisfy request and drop my messages or disconnect the client. So do you think this is not a environment I can use? There is a way to tell telegram server thet the tdlib client is not a human but a bot? Or Have I to change notification system?
As I told you with bot api there is not such limitation, the problem is only the secret chat missing. @levlam what do you suggest in that situation?

I would suggest not to invent theories about limits reaching before TDLib logs show the real reason of reconnections.

I'm not "invent" theory I was just reporting you the facts. Yestetday I disabled using of tdlib for sending notification falling back on the standard BOT and all is working wonderfull, even the tdlib receiving messages are very fast. I think the problem is that the server consider my tdlib client hostile because I sent too much messages. That is the more logic conclusion. So I'll give another chance to tdlib reproducing yesterday behavior and give you the log. If, I hope not, the problem is that, I can't waste other time I have to move on some other system...and I'll be very sad because I like Telegram platform.
Anyway in the mean while che question I want to ask you as in my prev message: do you think you can build using tdlib a BOT clinet that can send thousands messages per minute to your thousands client? Is there the possibility that server consider you as spammer and block you as yesterday?

"I think the problem is" is exactly inventing a theory without facts. You have zero evidence for now that server did any kind of blocking for you. And the behavior you described is definitely not the way in which spammers are blocked. The real reason of reconnections can be seen only in TDLib logs.

@levlam ok

@levlam can you answer my question please?

Do you think you can build using tdlib a BOT clinet that can send thousands messages per minute to your thousands client?

Telegram limits for ordinary users are much lower. Only bots are allowed to send so much messages.

Is there the possibility that server consider you as spammer and block you as yesterday?

The server fails particular requests with 429th error if flood limits are exceeded, but not interfere with user connections.

Ok I'll try to reproduce the problem and let you view logs. I hope we find the reason, but from your preview message I understand that probably I took the wrong way anyway

this morning still no problem with the same version of Saturday and quite the same Traffic, maybe more traffic due to the work activity

@levlam about the retry_after fields of updateMessageSendFailed.
do you think that could happen if my tdlib BOT send too much messages? the retry after is for any kind of new message to be sent or for a specific type of message or receipient?

in log I found a lot of lines like:
error_code = 429
error_message = "Too Many Requests: retry after 150"

or
error_code = 429
error_message = "Too Many Requests: retry after 3019"

429 error code you told me is the code of "flood limits"

@micheleselea 429 is returned exactly when flood limit is exceeded. The number in the error is a number of seconds to wait, before the message could be retried to be sent. The same number is returned in updateMessageSendFailed.retry_after. There are different kinds of flood limits, so it isn't possible to tell which one is exceeded.

my message was about updateMessageSendFailed.retry_after  I usually do not resend message that fail to send because but I was asking if that retry_after is only about that message or is a time to wait before send next message in general

And I answered that there are different kinds of flood limits. The value of retry_after is related to a particular message. Usually other messages will also fail to send with the same retry_after time, but sometimes they will be succesfully sent.

ok understand, anyway the problems I had saturday generates a lot of that error logs. Today I did not yet see any of that logs. I'll keep you informed

12 hours without problems. I think saturday were something wrong with connection or telegram servere..

Ok after 2 days the system was working this morning I'm in the same saturday situation.
I have  messages like:
updateMessageSendFailed Too Many Requests: retry after 599[code:429]
updateMessageSendFailed Too Many Requests: retry after 599[code:429]

updateConnectionState {
  state = connectionStateReady {
  }
}

updateConnectionState {
  state = connectionStateConnecting {
  }
}

updateHavePendingNotifications {
  have_delayed_notifications = false
  have_unreceived_notifications = true
}
updateConnectionState {
  state = connectionStateReady {
  }
}
updateConnectionState {
  state = connectionStateConnecting {
  }
}

updateHavePendingNotifications {
  have_delayed_notifications = false
  have_unreceived_notifications = true
}
updateConnectionState {
  state = connectionStateReady {
  }
}
updateConnectionState {
  state = connectionStateConnecting {
  }
}
updateHavePendingNotifications {
  have_delayed_notifications = false
  have_unreceived_notifications = true
}
updateConnectionState {
  state = connectionStateReady {
  }
}

updateConnectionState  about 2000 messgages received in 30 minutes.
I'm trying collecting logs

@levlam  Ok I have the log collected, tell me where you want me to send. I temporary have to disable using the client for sending because it does not work, and even new message notification are very slow I think it takes about 1 minute from the moment I send message and the momento tdlib receive it

@levlam I think it's all related to Error : 429 : Too Many Requests: retry after 139
you will see the log.
Do you know exactly what is the limit after I can start receiving 429 error? so I can do a check on my application to verify if I'm near to that limit

I see a lot of
[ 3][t 4][1590559181.599406481][NetQuery.h:239][#1][!SessionMultiProxy:4:main][&net_query]	[Query:[id:8581939202][tl:0xa9776773][state:Query]] [debug:SessionMultiProxy:4:main: send to proxy #0]
[ 3][t 4][1590559181.599411011][NetQuery.h:239][#1][!SessionProxy:4:main][&net_query]	[Query:[id:8581939202][tl:0xa9776773][state:Query]] [debug:SessionProxy:4:main: sent to session]
[ 3][t 4][1590559181.599415302][NetQuery.h:239][#1][!Session:4:main][&net_query]	[Query:[id:8581939202][tl:0xa9776773][state:Query]] [debug:Session: received from SessionProxy]
[ 3][t 4][1590559181.599419594][Session.cpp:234][#1][!Session:4:main][&net_query]	Got query [Query:[id:8581939202][tl:0xa9776773][state:Query]]
[ 3][t 4][1590559181.599423647][NetQuery.h:239][#1][!Session:4:main][&net_query]	[Query:[id:8581939202][tl:0xa9776773][state:Query]] [debug:Session: pending]
[ 3][t 4][1590559181.599427938][NetQuery.h:239][#1][!Session:4:main][&net_query]	[Query:[id:8581939202][tl:0xa9776773][state:Query]] [debug:Session: trying to send to mtproto::connection]
[ 3][t 4][1590559181.599432230][NetQuery.h:302][#1][!Session:4:main][&net_query]	Got error [Query:[id:8581939202][tl:0xa9776773][state:Query]] [Error : 204 : ]
[ 3][t 4][1590559181.599436760][NetQuery.h:239][#1][!Session:4:main][&net_query]	[Query:[id:8581939202][tl:0xa9776773][state:Error][Error : 204 : ]] [debug:dispatch]
[ 3][t 4][1590559181.599441290][NetQuery.h:239][#1][!Session:4:main][&net_query]	[Query:[id:8581939202][tl:0xa9776773][state:Error][Error : 204 : ]] [debug:sent to callback]
[ 3][t 4][1590559181.599445581][SequenceDispatcher.cpp:142][#1][!SecretChat SequenceDispatcher][&net_query]	Resend [Query:[id:8581939202][tl:0xa9776773][state:Error][Error : 204 : ]]
[ 3][t 4][1590559181.599449873][NetQuery.h:80][#1][!SecretChat SequenceDispatcher][&net_query]	Resend[Query:[id:8581939202][tl:0xa9776773][state:Error][Error : 204 : ]]
[ 3][t 4][1590559181.599454165][NetQuery.h:239][#1][!SecretChat SequenceDispatcher][&net_query]	[Query:[id:8581939202][tl:0xa9776773][state:Query]] [debug:Waiting at SequenceDispatcher]
[ 4][t 4][1590559181.599459410][Session.cpp:1336][#1][!Session:4:main]	Wakeup after 0.000822

Don't know if it's useful but seems that my tdlib Linux application is connected to telegram trough ipv6
tcp6       0      0 IPSRC:57952 2001:67c:4e8:f004:::443 ESTABLISHED 16133/TelegramTdlibBot

after 3 hours I re enabled service and it seems to work for now: I don't know let me know when you have some suggestion

Send logs to @tdlib_bot in Telegram.

done

Thanks for the log. I see the cause of the issue in the log, but the log is too small to find the exact initial cause. Could you increase log size to 1GB, wait for the next time the issue happens, and send the log again to @tdlib_bot, so I can confirm the initial reason of the issue?

@levlam I start record the log after getting into problem that the reason you do not see the log before the problem. I'll try to check if I can enable log always and let you know.
I have a question about 429 retry_after SECONDS.
Suppose I have to send 100 messages
After 30 messages I received 429 retry_after SECONDS.
Suppose it's not important to "resend" the 30th message but I can keep going to the 31st message.
In that situation I can go ahead and send 31st message or have I to wait SECONDS before send that new message?

Sometimes yes and sometimes no. This is supposed to automatically handled by TDLib, so you can just always call sendMessage.

Ok so maybe something wrong in that control, because I usually see the SECONDS increase over time....sometimes I see 3000 seconds that it's quite absurde

@levlam Ok fine! I'm in the problem: and I have the log since the beginning

@levlam
sent in @tdlib_bot
I disable the use of tdlib and fall back in BOT and all is fine, I still got the notification connecting/ready yet usually the go after few hours

@micheleselea Unfortunately, the load is so high during problems that 756MB of the log contains only last 184 seconds. So it can't be seen why the issue appeared, but at least it is clear how to make it self-repair after the issue happened.

@levlam but is it depend from 429 flood limit?

Anyway take in mind the log switch happened at 13:14  and the problem start at about 13:30 so the 774Mb of log file should be start before the event

The log autorotates every 1GB (as set by the app), so it starts only at 13:41:41 and ends at 13:44:45.

@levlam I see a log.txt.old when it rotate, can be usefull? I don't think I have that now, but for the next time

@levlam In this case log.txt.old contained 1GB of data, which also shouldn't be enough to see the beginning of the issue.

So what you think it's the max ammount of time I can wait before stop application and grab log?

Ok so next time it happens, if there's no fix yet, I'll try to stop it as soon as I can

@micheleselea Unfortunately, the load is so high during problems that 756MB of the log contains only last 184 seconds. So it can't be seen why the issue appeared, but at least it is clear how to make it self-repair after the issue happened.

@levlam do you think the "self-repair" will be a tdlib patch or a server side change?

@levlam I sent another 2 logs captured this morning with the problem.
Tell me if you find it useful or if I can sto capturing log

This morning I had problem since 11:08 till now....a lot of 429 errors and some connection/reconnection problem

@levlam I really need to have an idea, even an approximate idea, of how can I reach that 429 limit. Because today happens dozen times, and the 3 days before never happened.
I need to understand if I completely miss the target so I'm using something I can never figure out to run well or if I just to make it "better".
For example latest 24 hour I sent about 6270 to 440 SecretChat. Do you think that is too much for a client developed with TDLib or it's still in the "good range"? If not, consider my tdlib application has only one instance from one server running (so just one client with that api id and that phone number logged in) there's a way to instruct Telegram Server that I'm not a human? but I have only one tdlib instance running?

This is a secret chats, so nothing should depend on the server. The fix, if possible, should be in TDLib.
The last log catched the moment when the problem started, so there is no need in more logs.

I've checked the log and found that the issue was triggered by 136 messages sent in batches from<PHONE_NUMBER> to<PHONE_NUMBER>. To avoid the issue you can avoid to sending more than 100 messages in any 30 seconds.

I've checked the log and found that the issue was triggered by 136 messages sent in batches from<PHONE_NUMBER> to<PHONE_NUMBER>. To avoid the issue you can avoid sending more than 100 messages in any 30 seconds.

Ok I'll try, at least now I have a way to make it work better, thanks

I put a 100 msg max in 30seconds, and I immediatly hit the condition :)
Let see what happen next few days, but I think probably I choose the wrong way,
Probably the best thing should be if BOTs support SecretChat, @levlam do you think one day there could be the possibility for a BOT to handle SecretChat? that probably should be my solution

I put the 100 messages max in 30 seconds but the 429 error still happen sometime with a retry_after time very high for example 128 or 400 seconds...still no solution for SecretChat I think. should I abandon my tdlib BOT SecretChat project? probably....

ok I'll try

nope, after restarting with 80 messages in 30seconds limit, without hittin that limit I start receiving
updateMessageSendFailed Too Many Requests: retry after 561[code:429]
and
updateConnectionState {
  state = connectionStateConnecting {
  }
}
updateConnectionState {
  state = connectionStateReady {
  }
}

I don't understand but or I'm making a bad code, and I hope so, or something else is wrong

Have you reached the limit previously? You need to wait the specified time first. Also if there are pending unsent messages, they should be also counted.

consider everytime I restar the application che queue is deleted, so I start over, so should not be any message queued on startup

@micheleselea This is not true. TDLib continues to send messages after restart, nothing is lost. Moreover, secret chat can't work without that, otherwise the server would be able to tamper with messages and remove arbitrary one from the middle.

Understand. I'll try again next week with secretchat and 80messages limit because for now was not succesfull....I give one other shot next week than Inhavr to give up

All my try to limit message even 80 on 30seconds make me receive 429 and than connecting disconnectig. Ok fine, I have to try another way because this one is not the right one, it's a shame but I cannot make it work as I need. Thank you anyway, bye

I tried not to send more than 4 messages every 2 seconds....and I got 429 anyway....that way I can't use it, I need to move away from SecretChat structures. Bye @levlam

In secret chats message deletions are also counted for message sending and in fact they was the primary cause of the issue. The behavior was improved in the latest version from master, so it should work better now. The server-side flood control still remains the same and you are limited to ~90 message actions (including deletions) in any 30 seconds.

Understand


Discussion: naming of conditional pass-through methods

Currently the methods for conditional pass-through are named trip for the happy and rise for the error path.
While these are handy and behave as desired, I've received feedback that the names are quite confusing. Furthermore, they don't follow the naming pattern of the other methods, where success and error variants indicate their intention by using and / or respectively.
As I've spent an annoyingly long time to come up with these names in the first place, I can totally relate to the concerns raised.
Ideas and suggestions warmly welcome!

Ok, I've thought quite a bit about this and propose the following:

.trip() will be renamed to .andEnsure()
.rise() will be renamed to .orEnsure()
in the upcoming minor release the new method names will be added and original ones will be marked as deprecated
the original ones will be removed in the upcoming major release (probably end of april)

The new naming is consistent with the and/or naming pattern and conveys the intent clearly IMO.


[fix bug 1138467] Set the maximum unavailability period to 3 months.

@johngian r?

r+wc :rabbit:

Coverage increased (+0.02%) to 91.75% when pulling e0d64f14831631c93727341e993de2c667c8d58f on akatsoulas:1138467-unavailability-timespan into 539e0da04db7d88c532d2cb81b6e083fee79e067 on mozilla:master.


re-configure envoy without restart?

Is your feature request related to a problem? Please describe.
no.
Describe the solution you'd like
when we run an envoy with func-e run -c <configfile>, if i later make a change to my config, the procedure is to ctrl+c the current envoy, and re-run the command.  since envoy by default supports "hot reloading" of configuration, it would be cool if all one had to do was edit and save the config file for the changes to get picked up by func-e to push the updated config to envoy, without a restart.
Describe alternatives you've considered
ctrl+c and re-run the command.
Additional context
none

hi there. I recall at some point we indirected the config file, but now we have tests that we don't mess with it. IOTW -c  passes that arg directly to envoy and doesn't change the path to the config at all (it can even be in the current directory).
I also became confused as to what envoy actually supports vs what is done with scripts, and seem not alone https://github.com/envoyproxy/envoy/issues/19515
It appears at first glance that it doesn't internally use inotify rather scripts to do that... That's an uneducated guess as I have no idea why you would need a script if it supported it internal. We probably need an envoy expert to answer that or docs to be clarified on the above envoy issue about if 'hot restart' support really means a python script. If that's all it means, then probably we could consider externalizing that here, but if internally it already does it and just no one knows how, maybe we can make an example instead

Not sure if it is related, but we could possibly use dynamic configuration from filesystem: https://www.envoyproxy.io/docs/envoy/latest/start/quick-start/configuration-dynamic-filesystem, also see: https://www.tetrate.io/blog/envoy-101-file-based-dynamic-configurations/.

thank you for sharing your thoughts.  this came up in a chat earlier today and it initially sounded very interesting and i wanted to make sure and capture it.  i suppose the value of such a feature depends on the use case.  if func-e is used in the context of experimenting with different configurations / learning, i'm starting to realize this "feature" is largely superfluous.  in other contexts such as when envoy is an active participant in a larger system (i.e. a mesh) then leaning on its configuration api becomes important.  and istio already does that.

thanks dio for the links.

⚠ This is a sketch ☢
Please don't hate me because I do a silly thing like this 🙏🏽.

I'm starting to realize this "feature" is largely superfluous

So editing a "static" config-only scenario?
How about this: so when you have a valid config, for example from https://raw.githubusercontent.com/envoyproxy/envoy/main/configs/envoyproxy_io_proxy.yaml, you can do:
go run github.com/dio/pc@main func-e run -c ./envoyproxy_io_proxy.yaml

Then edit the YAML, Envoy will get a new config.


Make metadata first slide 2-column or tabular

With more metadata, particularly with the style stuff, we should make the first slide better structured to hold it all.

In 1.2.


us-co-routt

ftp://ftp.co.routt.co.us/pub/

Their FTP is pitifully slow, we were able to scrape a REST Service here: https://github.com/openaddresses/openaddresses/blob/master/sources/us/co/routt_county.json


Console spam

05-15 18:33:22.332 11171-11171/recyclerviewpagerdemo D/test: dx:0.14092954
05-15 18:33:22.348 11171-11171/recyclerviewpagerdemo D/test: dx:0.1161919
05-15 18:33:22.358 11171-11171/recyclerviewpagerdemo D/test: dx:0.10194903
05-15 18:33:22.365 11171-11171/recyclerviewpagerdemo D/test: dx:0.10044978
05-15 18:33:22.383 11171-11171/recyclerviewpagerdemo D/test: dx:0.0

It goes from TabLayoutSupport library. I would be grateful if you will fix it :+1:

Thx!!! I will do it.


runtime: auxv parsing should be unified

Currently there are two runtime locations where auxv data is parsed, vsdo_linux_amd64.go, and os_linux_arm.go. Adding arm64 will add a third location. This logic should be unified into a single file to be reused by all linux implementations.

I just found a third location, os_linux_386.go

This was fixed by https://golang.org/cl/22061 (for #9993).


fmt.Printf("%v") panics when time.Month is embedded in a struct

Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (go version)?
1.7.4
What operating system and processor architecture are you using (go env)?
playground
What did you do?
https://play.golang.org/p/Zsst7dbgmd
What did you expect to see?
valid output
What did you see instead?
panic output

This is just how time.Month(0).String() behaves; no need for struct embedding.
https://play.golang.org/p/0ycnF_0YBg
Perhaps time.Month.String should specially handle the zero value, though.

This has nothing to do with Printf. A better recreation is here: https://play.golang.org/p/tvvroOOWZk
The problem is (was) that time.Month.String was panicking.
In any case it's fixed in 1.8.

$ cat x.go
package main
import "time"
func main() {
var m time.Month
println(m.String())
}
$ go run x.go
%!Month(0)
$


Kafka ACL table should print the mapped values instead of directly printing the enums from the backend

For the ACL CRUD commands we have mapped all enums from permission, operation, resource type to lower case forms. There are a couple of reasons making the user enter uppercase values is not preferred and we should try to avoid this:

It is more difficult to type, as the user must turn on CAPS LOCK.
It is more difficult to read, as uppercase letters are more uniform and harder to tell apart.

Current output:
  PRINCIPAL (6)    PERMISSION   OPERATION          DESCRIPTION
 ---------------- ------------ ------------------ ----------------------------
  All accounts     ALLOW        DESCRIBE_CONFIGS   TOPIC is "*"
  All accounts     ALLOW        DESCRIBE           TOPIC is "*"

Preferred output:
  PRINCIPAL (6)    PERMISSION   OPERATION          DESCRIPTION
 ---------------- ------------ ------------------ ----------------------------
  All accounts     allow        describe-configs   topic is "*"
  All accounts     allow        describe           topic is "*"

Taking this one now.

Should this wait for release?

No, it's fine, there are several places that need to be updated for this.


Correct minor documentation typos

Current coverage is 60.81%

Merging #235 into master will decrease coverage by -0.08% as of cb342f4

@@            master    #235   diff @@
======================================
  Files           48      48
  Stmts         1565    1567     +2
  Branches         0       0
  Methods
======================================
  Hit            953     953
  Partial          0       0
- Missed         612     614     +2

Review entire Coverage Diff as of cb342f4

Powered by Codecov. Updated on successful CI builds.

Thanks!


Software release checklist for ppa v3.9

release prep

[x] Pull updated copies of the develop and main branches
[x] Use git flow to create a new release branch with the appropriate version (e.g., git flow release start 0.5)
[x] Review database migrations to make sure updates look good (django only; should be caught in testing/qa)
[x] Update release version to appropriate number (for Python apps, set to final version without any -pre or -dev tags).
[x] Update changelog to make sure that all features, changes, bugfixes, etc included in the release are documented.  You may want to review the git revision history to be sure you’ve captured everything.
[x] Update deploy notes to document any manual steps that need to be done with the deploy (e.g., running any import scripts, configuring site domain in Django admin, etc)
[x] Update any README badges that are pointing to develop branches to use either main or default branch (travis-ci build, code coverage, etc).
[x] Make sure local settings are documented (for django app: sample local settings file includes any new configurations).
[x] Check python requirements for any internal dependencies that should be released (or at least pinned to a specific git commit)
[x] Release & publish any internal JS dependencies with updates; specfy the published version number in package.json
[x] Use npm audit to check and fix vulnerabilities in npm dependencies
[x] Make sure that unit tests are passing on travis-ci for release branch (fix them if they are not)
[x] Make sure code coverage is 95% or higher. Review for any important gaps.
[x] Review code documentation to make sure it is up to date. If the database has changed, generate and include current schema documentation images.
[x] Run pip freeze, save output as requirements.lock and check in to document known working versions of python dependencies
[x] Use git flow to finish the release (merge release branch into both main and develop, create a tag, remove the release branch, etc.). (git flow release finish 0.5)

after release

[x] Increase the develop branch version so it is set to the next expected release (i.e., if you just released 0.5 then develop will probably be 0.6-dev unless you are working on a major update, in which case it will be 1.0-dev)
[x] Push all updates to GitHub (main branch, develop branch, tags)
[x] Clean up any release branches on GitHub

Looks like we're ready to close this?


OCPBUGS-30063: Add version info to kargs for deploy in IBU

In the case where we're doing an IBU to a release with the same rhcos base image on a cluster that has previously been upgraded via IBU, we end up with the exact same set of boot loader config options in the deployments for both the old and new stateroots. This causes a problem for ostree, which is using the set of config options when comparing deployments to determine whether the boot loader entries need to be updated. When we're swapping deployments in order to pivot to the new release, the deployment boot serials get reassigned, however, as the deployments are in separate stateroots. Without also updating the boot loader entries, ostree ends up unable to correlate these entries with the boot links it has now swapped.
In order to avoid this issue, this update adds a karg to the new deployment with the image version, ensuring it will be unique when compared to the original deployment.
When doing a no-delta IBU from a cluster that has not previously undergone an IBU, the boot loader config options already have some differences when compared to the new deployment created for the seed, so this scenario does not occur.

/cc @browsell

/lgtm
/approve

/override ci/prow/integration

/cherrypick release-4.15


Fix bugs of copying scene root node or pasting node as scene root

Fix crash when pasting a node as a new scene root.
Fix owner was not set correctly when copying scene root node from unsaved scene.
The owner of the root node is nullptr, which was ignored before.

Bugsquad edit: Fixes #81507

Thanks!


Still need webpacker for Rails 5.1.3 above ?

So in the Readme under Steps to a New App with rails/webpacker v3 plus latest React on Rails, it says

First be sure to run rails -v and check that you are using Rails 5.1.3 or above. If you are using an older version of Rails, you'll need to install webpacker with React per the instructions here.

I am on rails version Rails 5.1.4  after setuping everything in Basic installation for a new Rails App and run it. I got an error for
undefined method `javascript_pack_tag'  for <%= javascript_pack_tag 'hello-world-bundle' %>
Which as far as I know javascript_pack_tag helper is from webpacker, Do does that mean I will have to add gem "webpacker", "~> 3.0" even for  Rails 5.1.3 or above

Yes to the above.


The example iggp-minimal-decay don't work anymore. Shows "No program returned."

The example examples/iggp-minimal-decay don't work anymore. After 1 minute it shows "No program returned"; same result with  and without the --test-all flag.
(I know that It worked as of 2021-04-29.)

Pushed a fix. Thanks!


Create Terraform definitions for FAM OIDC client (IDIR only)

FAM needs a public OIDC client to secure the web app and the API. It will be IDIR only to start.
AC:
We have an OIDC client in Cognito that we can use to secure our frontend application and API for FAM; MVP supports IDIR for login

Done. Testing process documented here: https://github.com/bcgov/nr-forests-access-management/wiki/OIDC-Client-Testing

@ArogeG  to review documentation


Inventory Grid

Created a plugin that shows a grid in your inventory to differentiate item slots

Images:
Resizeable

Fixed

why?

Related #1774


Snippets with empty lines

I have a problem with empty lines apparently terminating a snippet definition. Consider the following simple snippet:
    s('foo', {t('bar'','','baz')}),

which should expand to
bar

baz|

(with | the cursor). But what happens is
bar|

seems like is a problem with passing several strings to t  it works fine with
       s('foo', {t({'bar','','baz'})}),

Oh, I see, that isn't supported! Knowing that, the examples do only talk about the table syntax. 🤦🏻
(It's really hard going back and forth between the two -- equally valid -- styles of omitting parentheses around table arguments or not... You wouldn't happen to want to use neovim's stylua presets? :])

Oh, could you link those? I can't find them in neovim/neovim

Still under discussion in core (https://github.com/neovim/neovim/pull/14987) but many related plugins have already made similar settings. (The autoparenthesis setting is still a bit controversial as long as stylua insists on "all or nothing".)


Be clear about which release we're upgrading from and to

Although the plan is very clear about what's going to happen to the individual cloud components, we don't clarify what version the overall cloud is at, and which OpenStack release we're going to be upgrading to.

I implemented something that displays current and next cloud OS release (PR#120). Currently this is only a part of analysis result and will not show unless verbose mode is turned on. I can add it to the plan and always display it (except in quiet mode) before the actual steps


Read pubsub for receipt

What and why?
How to test?


Don't store Google account login in plain text

It's a little bit worrying to have my Google login info saved in plain text.
Can you somehow utilize oauth-style login similar to the way https://github.com/nylonee/hangupsbot does it?
It just gives me a URL to go to which returns a long string to paste back into hangupsbot. That hangupsbot never sees my raw login credentials.

I read all Google OAuth documentation but haven't found a way to do something like this :disappointed:
It's all because Ingress Intel map is an AppEngine application which doesn't allow logging in via OAuth, it uses OAuth itself. The only thing you can do is to use command-line arguments with phantomJS and no config files, as written here.
./phantomjs ice.js $VERSION $EMAIL $PASSWD $LINK $MIN_LEVEL $MAX_LEVEL $DELAY $WIDTH $HEIGHT $FOLDER $NUMBER $IITC $TIMESTAMP

which will look like
./phantomjs ice.js 1 some-email "This$is&my*passwd" "https://www.ingress.com/intel?ll=52.98348,5.071092&z=3" 8 8 300 4096 2160 /home/link/ice/ 0 0 0

Or use sacsid  (session id) and csrf token (ingress Intel timelaps from
github)
пт, 19 Июн 2015, 22:46, Nikita-kun<EMAIL_ADDRESS>
I read all Google OAuth documentation but haven't found a way to do
something like this [image: :disappointed:]
It's all because Ingress Intel map is an AppEngine application which
doesn't allow logging in via OAuth, it uses OAuth itself. The only thing
you can do is to use command-line arguments with phantomJS and no config
files, as written here
https://github.com/nibogd/ingress-ice/wiki/Config-file-format.
./phantomjs ice.js $VERSION $EMAIL $PASSWD $LINK $MIN_LEVEL $MAX_LEVEL $DELAY $WIDTH $HEIGHT $FOLDER $NUMBER $IITC $TIMESTAMP
which will look like
./phantomjs ice.js 1 some-email "This$is&my*passwd" "https://www.ingress.com/intel?ll=52.98348,5.071092&z=3" 8 8 300 4096 2160 /home/link/ice/ 0 0 0
—
Reply to this email directly or view it on GitHub
https://github.com/nibogd/ingress-ice/issues/19#issuecomment-113621973.

Finally done this (as @SniperRus suggested — using cookies).

Thank you! :smile:


Complete missing dataservice configuration rake tasks

Fixes #11777
Fixes #11823

@rafatower if you care to take a final look, I've cleaned up the code to comply with hound (and tidied up things a bit while I was at it) and added some documentation.

:+1:

Acceptance plan:
This could be performed in staging, with some staging user with a proper dataservices configuration (ping me if you need assistance with this).
Check the user configuration using the cdb_dataservices_client.cdb_service_quota_info() function; then use all of the six following rake tasks to make configuration changes and check with said SQL function that the changes are effective.

cartodb:services:set_user_provider[username,service,provider]
cartodb:services:set_organization_provider[orgname,service,provider]
cartodb:services:set_user_quota[username,service,quota]
cartodb:services:set_org_quota[orgname,service,quota]
cartodb:services:set_user_soft_limit[username,service,soft_limit_status]
cartodb:services:set_org_soft_limit[orgname,service,soft_limit_status]

It should be also checked that when not passing one of the parameters or when passing invalid values a proper error message is shown, and the configuration isn't changed.
The mapzen_routing service must be explicitly tested for changing its quota and soft limit.

Acceptance approved!! Everything working as expected in staging tests.

@jgoizueta sorry to bring bad news again... There is an issue around (https://github.com/CartoDB/cartodb/commit/575e5ce446ca5972097f0f965fc85d4b46324c60#diff-b9a50e367735318a20a042098a0be3fcL148)
# sudo su - carto -s /bin/bash -c "cd /opt/carto/builder/embedded/cartodb ; RAILS_ENV=production RAILS_CONFIG_BASE_PATH=/data-builder/production/config/builder RAILS_LOG_BASE_PATH=/data-builder/log/builder SUBDOMAIN=carto PASSWORD=location<EMAIL_ADDRESS>/opt/carto/builder/embedded/bin/bundle exec rake cartodb:services:set_org_soft_limit['organization',geocoding,true]"
rake aborted!
undefined method `soft_geocoding_limit=' for #<Organization:0x00000006d35e18>

Tasks: TOP => cartodb:services:set_org_soft_limit
(See full trace by running task with --trace)

This is rather lame: I implemented a rake task to set a configuration parameter that doesn't exist 😞
Service soft limits are only per-user parameters. There are columns for soft limits in the users tables of central and metadata and the values are stored in redis too. But organization soft limits are not a thing: such parameters do not exist!
You can only enable soft limits for a given service to individual users, not for a whole organization.
I'm very sorry for this time-wasting confusion; please do not use the set_org_soft_limit. I'll proceed to remove any trace of it from the code and documentation and thus bury this shameful episode as soon as possible.

Ok, cristal clear now!!! Thanks @jgoizueta