WebApp:Unhelpful error on az webapp up

This is autogenerated. Please review and update as needed.
Describe the bug
When trying to use az webapp up I was met with an 'Unauthorized' error message and that let me on a wild chase of permission issues with my account, RBAC rules, etc. as it led me to believe that my user account didn't have access to create the resource.
Turns out that was an unhelpful message and the actual underlying exception message would have saved me endless time:
msrest.http_logger: {"Code":"Unauthorized","Message":"This subscription has reached the limit of 1 Free Linux app service plan(s) it can create in this region. Please choose a different sku or region.","Target":null,"Details":[{"Message":"This subscription has reached the limit of 1 Free Linux app service plan(s) it can create in this region. Please choose a different sku or region."},{"Code":"Unauthorized"},{"ErrorEntity":{"ExtendedCode":"52041","MessageTemplate":"This subscription has reached the limit of {0} Free Linux app service plan(s) it can create in this region. Please choose a different sku or region.","Parameters":["1"],"Code":"Unauthorized","Message":"This subscription has reached the limit of 1 Free Linux app service plan(s) it can create in this region. Please choose a different sku or region."}}],"Innererror":null}
msrest.exceptions: Operation returned an invalid status code 'Unauthorized'

The specific portion: This subscription has reached the limit of 1 Free Linux app service plan(s) it can create in this region. Please choose a different sku or region would have alerted me immediately to the issue and I could have resolved it in 5 minutes rather than hours.  Alternatively the default error message of Operation returned an invalid status code 'Unauthorized' could have told me to turn on --debug option for more details and that would have equally saved me time.  As it stands today, the message is not helpful to the user in directing them to resolution.
Command Name
az webapp up
Errors:
Operation returned an invalid status code 'Unauthorized'

To Reproduce:
Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

Put any pre-requisite steps here...
az webapp up --os-type {} --debug

Expected Behavior
Environment Summary
Windows-10-10.0.19041-SP0
Python 3.6.8
Installer: MSI

azure-cli 2.21.0

Additional Context

@panchagnula - seems this error has been returned by the API. could you please check this one.

@timheuer apologize but the detailed error message currently is not displayed by default in CLI commands & --debug is usually the recommended approach for CLI commands - you do have a good point about adding the info about using --debug to all error/warning messages. Will check with core-cli folks on

displaying detailed error messages as default
exposing the use of --debug.
@yonzhan -- will start an email thread on this discussion internally -FYI.

Thank you @panchagnula -- just looking for a little more hinting as the current state was unhelpful and didn't direct me to how it could be more helpful.


Support for reverse engineering from SQL file?

Greetings very cool redesign of the tool.
It would be very interesting to be able to do reverse engineering of sql script for example I work with postgres. When you update the schema you can have the model updated.

@martinmurciego
I think a more straight forward way is to import the SQL into a temporary database and then convert the database to XML.


SCSS SourceMap pointing to the Parent selector only.

Mac OSX (El Capitan?)
angular-cli: 1.0.0-beta.18
node: 6.7.0
When running 'ng serve' or 'ng build' the SCSS SourceMaps only point to the Parent selector and not the correct selector.

But as you can see it is not pointing to the correct selector.  It should be pointing to line 13 not line 5.

This seems to be related more towards angular-cli and webpack as running the node-sass cli works fine and points to line 13
node-sass --source-map true src/styles.scss > styles.css
I have tried changing devtool to 'source-map' in
/node_modules/angular-cli/models/webpack-build-development.js
And still no luck.
PS in order to get SourceMaps to work I needed to add ?sourceMap to the css and sass loaders in
/node_modules/angular-cli/models/webpack-build-common.js
Also I needed to update angular-cli: 1.0.0-beta.17 to angular-cli: 1.0.0-beta.18
Here is my loader configurations in
/node_modules/angular-cli/models/webpack-build-common.js

//  in main, load css as raw text
{
exclude: styles,
test: /.css$/,
loaders: ['raw-loader', 'postcss-loader']
}, {
exclude: styles,
test: /.styl$/,
loaders: ['raw-loader', 'postcss-loader', 'stylus-loader'] },
{
exclude: styles,
test: /.less$/,
loaders: ['raw-loader', 'postcss-loader', 'less-loader']
}, {
exclude: styles,
test: /.scss$|.sass$/,
loaders: ['raw-loader', 'postcss-loader', 'sass-loader?sourceMap']
},
// outside of main, load it via style-loader
{
include: styles,
test: /.css$/,
loaders: ['style-loader', 'css-loader', 'postcss-loader']
}, {
include: styles,
test: /.styl$/,
loaders: ['style-loader', 'css-loader', 'postcss-loader', 'stylus-loader']
}, {
include: styles,
test: /.less$/,
loaders: ['style-loader', 'css-loader', 'postcss-loader', 'less-loader']
}, {
include: styles,
test: /.scss$|.sass$/,
loaders: ['style-loader', 'css-loader?sourceMap', 'postcss-loader', 'sass-loader?sourceMap']
}

I tried removing 'postcss-loader' and also rearranging the loaders, but still no luck.
I am thinking that maybe one of the loaders is running before the sass source-maps is running and getting the incorrect or compressing the css before it runs the source map, but I am not sure.
Any Help?
Thanks

+1 on this issue.
Hey @geoff-gedde, some of us are having this same issue in Stack Overflow, see:

How to Source Map Angular CLI CSS files in Chrome Dev Tools?

Except I don't even get a styles.scss just inline <style>...</style> but I'll read your issue here again tomorrow and try to get at least to where you're at and then edit this part out. Cheers!

The code is still inline, but because the source-map is running it will point to the source of the inline css which is styles.scss.

Looks like this issue is more related to Webpack then angular-cli as it also happens on vuejs 2.0 with webpack.  This also may be related to using the style loader instead of loading the file.  I will be testing some more.

Not sure why this was closed.
Are you suggesting that if you want Source Maps to work correctly that it will not work correctly with angular-cli and you that need to be using static files?
Using static files is not a fix, but a work-around.
I tried using "extract-text-webpack-plugin" instead of 'style-loader' to make the css an external file which works great to make the css files separate so they are cached, but the source map is still not pointing to the correct selector.  So it is not related to 'style-loader'.  This makes me think that the issue is related to sass-loader.
Yes,
If I use node-sass --watch everything is fine, but I don't like the idea of running 'ng serve' and 'node-sass --watch' as separate tasks.
I will file an issue with sass-loader though.

Posted on sass-loader here:
https://github.com/jtangelder/sass-loader/issues/272

@ggedde the PR that closed this issue (https://github.com/angular/angular-cli/pull/2646), added, among other things, sourcemap support to the CSS pipeline (https://github.com/angular/angular-cli/pull/2646/files#diff-d85c71bf928fadd8cda87cb456e03d30R8).
I believe that has fixed this issue. The change isn't released yet, it should be out in the next release. If this isn't fixed then I'm happy to reopen and investigate more.

Updates: I have managed to have sourcemaps in beta 20-4
The codebase is still failing in devel but removing style-loader, from the cssLoader array and adding ExtractTextPlugin.extract to loaders, solve the problem:
https://gist.github.com/Canos/ec7fba011c989881562c97a14d0a192d
Also seems beta 20.4 solved the misspointing line.

@ggedde were you able to get this to work for you ?

Sorry @nadhir-falta, I took a pause from this project.
Nope, the issue still exists for me.  I also just recently installed brand new ng-cli on a new laptop.
Angular4.0 with CLI 1.0.2
I am still getting this issue.
This also happens with ng serve --extract-css
jhnns from sass-loader has confirmed that it is happening on sass-loader.
https://github.com/webpack-contrib/sass-loader/issues/272
So I am not sure where the issue is exactly coming from and not sure what I can do to help out.

When you type ng serve by default angular cli doesn't include sourcemaps files for css files but you can include it by typing this command

ng serve -sm -ec

or

ng serve --sourcemap --extractCss

I use Angular Cli -v 1.0.3, and More information - Angular CLI special cases

Yes @Koziuk-S  I have been using both those tags.

I did find a partial solution.
I think css-loader expects sass-loader to be compressed, but it is not by default.
if you add:
outputStyle: 'compressed',
To the sass-loader [options] in<EMAIL_ADDRESS>Then it works fine in Chrome, but not Firefox.  (that might be a separate FF bug)
@filipesilva I think a quick solution for Chrome would be to add outputStyle: 'compressed',
To the sass-loader [options] in<EMAIL_ADDRESS>Thanks

@ggedde is the repro you posted originally still valid for the issue you're still experiencing? Reading through https://github.com/webpack-contrib/sass-loader/issues/272#issuecomment-326732443 it seems like your fix will break sourcemaps in firefox, so I don't think that's a good solution. https://github.com/postcss/postcss/issues/926#issuecomment-294062807 indicates that this problem will be fixed at some point though.

@filipesilva
Yes, The Repo is https://github.com/ggedde/ngtest.
I wouldn't call it a fix, but a work around until postcss or css-loader makes an update.  Yes, the issue still exists on Firefox even after the workaround, but at least it would work for everyone on Chrome and Safari.  Plus I did add bug for it on FireFox bugzilla.
https://bugzilla.mozilla.org/show_bug.cgi?id=1364232
Firefox's bug is not related to sass-loader.  It is just related to how node-sass compiles the sourcemap.
So I think making the change yo the sass-loader [options] in<EMAIL_ADDRESS>is still valid.
Thanks

Closing this as sourcemap support has been greatly improved in 6.x and further in the upcoming 7.0 (currently in release candidate).  If problems are still occurring with the latest versions, please open a new issue detailing the version and environment information.  Thank you.


Removing ::before ::after padding hack on markdown

Because of the ::before, ::after hack, the code blocks are not fully selectable. This pr removes the hack. It's been so long since this was introduced (~3 years ago) It's not fully clear what this was preventing. I think it has to do with scrollbars in Firefox and Android browsers.
I'd like to try and roll that back, then test on prod to see what bugs might crop up.
cc https://github.com/github/github/issues/58168

This was deployed to review-lab https://github.com/github/github/pull/79702 and I didn't find any regressions.


Random failures of the reference tests on the chrome

The reference tests are failing on the chrome. The results are not consistent. It appears that the fonts are not fully loaded before showText -- the failed pages are empty. See #1040 testing attempts.
CC @kkujala

Tests that were the suspects:
TEST-UNEXPECTED-FAIL | eq bpl13210 | in chrome | rendering of page 4 != reference rendering
TEST-UNEXPECTED-FAIL | eq sizes | in chrome | rendering of page 1 != reference rendering


Add form validation hints

Changes

Rewrite <Tooltip> component to make it more flexible
Incorporate prefix and suffix into <Input> component (i.e. merge <IconInput> and <Input>)
Add validation icons to <Input>
Add validation hint to <Input>, displayed as a <Tooltip>

Questions

How are validation hints displayed on other form elements, such as selects, checkboxes, or toggles? — Not part of this PR.
What's the most straightforward way of ensuring that the tooltips are always positioned within the viewport? — Should address this later.
Do we need an abstraction à la <ValidatedInput> that takes care of the positioning and visibility of the tooltip? Where would that component go, Circuit UI or Dashboard? — Yes, in the dashboard.

Thanks. I'll give the code a look soonish. Is this deployed already, if not can you take screenshots first? 😃
To your questions:

The elements you name, I don't think require validation hints. At least I can't recall any instances of that. Since all these options provide you with possible answers, I really don't think there is a need.
This is tough and I don't have any idea. All I can come up with is filthy JS hacks.
If we make such a component -- and I think we should -- I think it should be as specific to how we do forms as possible (i.e. we should be able to provided as a render prop to Field). If we do that, I don't think it should go into Circuit UI. For now, the Dashboard seems like the right place. But as we develop this and grow our web projects, I think it might be worth making this a library so that things like p2p can use it, too (distant project though IMO).

Haha, silly me. Like you told me, you do need something like required. Didn't think of it as validation.

@felixjung Thanks for the thorough review! I've addressed your comments, please have another look 😊


Docs addition

https://mobile.twitter.com/bruxisma/status/1575888655582756876

In case rest of Twitter dies:


Upgrade to Angular 19

Use the following new features:

NG_HMR_TEMPLATES=1 ng serve
{ angularCompilerOptions: { strictStandalone: true } } in tsconfig.json
ng generate @angular/core:signal-input-migration
ng generate @angular/core:signal-queries-migration
ng generate @angular/core:output-migration
ng generate @angular/core:inject-migration

Source: https://blog.angular.dev/meet-angular-v19-7b29dfd05b84

Done


Socket hung up / Worker respawn when restarting app

Hi,
as soon as I close my connection inside of my app (cordova) I'll receive a socket hung up inside of my server-logs with an warning, after that, a "socket respawned" message apears.
Why does the server crash if a user disconnect from a connection?
Is this a bug? Should it behave like that?

@Hirbod This is normal. It's a warning, not an error. Socket errors show up as warnings on the server.
In this case the problem is probably that the client did not close the socket connection properly and instead killed the connection abruptly. The WebSocket RFC requires that client sockets send a close control frame the server (I.e. socket.disconnect()) before they kill the connection.
When using SC, warnings are a normal part of day-to-day operations; they don't usually affect the user experience. it's good to be aware of them but you generally don't need to do anything about them. You can try to reduce how often they happen though.

Thank you very much for your answer. I found out, that cordova does not fire the unload event, thats the reason why "disconnectOnUnload: true," does not work, while it works inside the browser.
I though it is an error, since the log file say "worker exited 1; worker respawned". Sounds like a crash like more then a warning.

If it says that the worker exited then it is an error. Do you have some other custom code in there that could be causing a fatal error (maybe as a flow-on effect to the first warning)?

1519397254959 - Origin: Worker (PID 20442)
  [Warning] SocketProtocolError: Socket hung up
   at SCSocket._onSCClose (/srv/trivia/quiz/node_modules/socketcluster-server/scsocket.js:229:17)
   at WebSocket.internalOnClose (/srv/trivia/quiz/node_modules/socketcluster-server/scsocket.js:74:10)
   at process.nextTick (/srv/trivia/quiz/node_modules/uws/uws.js:445:27)
   at _combinedTickCallback (internal/process/next_tick.js:131:7)
   at process._tickCallback (internal/process/next_tick.js:180:9)
1519397254959 - Origin: Worker (PID 20442)
  [Error] UnknownError: Socket error - SocketProtocolError: Socket hung up
   at SocketCluster._convertValueToUnknownError (/srv/trivia/quiz/node_modules/socketcluster/index.js:449:13)
   at SocketCluster.emitFail (/srv/trivia/quiz/node_modules/socketcluster/index.js:460:14)
   at SocketCluster._workerErrorHandler (/srv/trivia/quiz/node_modules/socketcluster/index.js:492:8)
   at ChildProcess.workerHandler (/srv/trivia/quiz/node_modules/socketcluster/index.js:687:14)
   at emitTwo (events.js:126:13)
   at ChildProcess.emit (events.js:214:7)
   at emit (internal/child_process.js:772:12)
   at _combinedTickCallback (internal/process/next_tick.js:141:11)
   at process._tickCallback (internal/process/next_tick.js:180:9)
1519397254962 - Worker 2 exited - Exit code: 1
  >> Worker PID: 20739
1519397255229 - Worker 2 was respawned

This only happens when I call "window.location.reload()" inside my app (or if I change a .js file and socketcluster reloads).
As my app listens to some special events, I need to call that reload function sometimes which will flood or kill the server when 100k users are connected)

@Hirbod I cannot reproduce. Which socketcluster version and socketcluster-client version are you using?

"socketcluster": "~9.3.1",
"socketcluster-client": "~9.0.2"

Updated to 9.3.3, problem remains the same.

Can you reproduce the issue by connecting a client which is running inside Node.js somehow?
E.g. You can run a client inside Node.js with:
var socketCluster = require('socketcluster-client');

var options = {
  port: 8000,
  path: '/socketcluster/',
  hostname: '<IP_ADDRESS>',
  multiplex: false
};
var socket = socketCluster.connect(options);

For me I only get the warning but no error afterwards.

we have a node.js running, there are no issues. We also have the client inside a brower, also no issue. It only happens on a hard disconncet from my app.. :/

To give more informations. I've downloaded a copy of "socketcluster.js" into my cordova app, which is included via js.

UnknownError means that an unrecognised error type was thrown in one of the processes.
SC should never throw this error type internally. I can reproduce the same error if I add this line inside worker.js:
scServer.on('warning', (err) => {
  throw err.toString(); // Throwing a string; bad idea.
});

Then I open a client in the browser and I hold down the refresh key so that it keeps reloading until I see errors.
SC never throw strings as errors (it always uses Error objects) so it's more likely to be a problem with your custom application logic.

I will dig into this, but to be honest, the server/worker.js stuff was auto generated with "socketcluster create app". I did not do anything on that part, I will have a look if I can find something.

Dang... @jondubois here we go
...
scServer.on('connection', function (socket) {
		socket.on('error', function (err) {
			throw 'Socket error - ' + err;
		});

This is AUTOGENERATED. I did not write this, but thanks for figuring out that very annoying problem. You saved my day!

Changed throw 'Socket error - ' + err to console.log('Socket error - ' + err); and this fixed the problem. THANK YOU!

 Has this been pushed?
|handler  |<PHONE_NUMBER>970 - Origin: Worker (PID 13438)
1|handler  |    [Error] SocketProtocolError: Server ping timed out
1|handler  |     at SCSocket._onSCClose (/var/www/handler/node_modules/socketcluster-client/lib/scsocket.js:629:15)
1|handler  |     at SCTransport. (/var/www/handler/node_modules/socketcluster-client/lib/scsocket.js:280:12)
1|handler  |     at SCTransport.Emitter.emit (/var/www/handler/node_modules/component-emitter/index.js:133:20)
1|handler  |     at SCTransport._onClose (/var/www/handler/node_modules/socketcluster-client/lib/sctransport.js:204:28)
1|handler  |     at Timeout._onTimeout (/var/www/handler/node_modules/socketcluster-client/lib/sctransport.js:269:10)
1|handler  |     at ontimeout (timers.js:475:11)
1|handler  |     at tryOnTimeout (timers.js:310:5)
1|handler  |     at Timer.listOnTimeout (timers.js:270:5)
1|handler  |<PHONE_NUMBER>071 - Worker 0 exited - Exit code: 1
1|handler  |    >> Worker PID: 13523


Add file_path param to download_file() to directly download files to disk

Instead of keeping them in memory. Addresses #37.

@hoh I think codecov is incorrectly configured. It currently also measures test file coverage... Which doesn't make sense.

@hoh I think codecov is incorrectly configured. It currently also measures test file coverage... Which doesn't make sense.

Why not ? Tests that don't run do not make sense.

Why not ? Tests that don't run do not make sense.

Because I just added some cleanup checks after downloading to the file, in order to check the automatic directory and file creation every time correctly


opentrons-ot3-image.bb: add chinese language support

Overview
We are adding language localization support to the client, on the ODD we need to support the Chinese character set. This pull request adds support for that.
Closes: PLAT-573
Testing

[ ] Ensure there are no invalid characters when localization is enabled and Chinese is selected.

here we go


Triage new projects ingest - nxn

Projects to triage

[x] https://contribute.data.humancellatlas.org/projects/detail?uuid=b3491826-b335-4b4e-91d6-2f1d69206091
[x] https://contribute.data.humancellatlas.org/projects/detail?uuid=06d01e67-2c57-4767-96f9-33fdc6d4226a
[x] https://contribute.data.humancellatlas.org/projects/detail?uuid=d7820c50-6d86-400d-a6d6-9fec0c712a3b
[x] https://contribute.data.humancellatlas.org/projects/detail?uuid=08fe0e78-fe02-445f-97bb-98ac4842b985
[x] https://contribute.data.humancellatlas.org/projects/detail?uuid=560e3e84-fc22-448f-a3c6-9665f5e4c62f
[x] https://contribute.data.humancellatlas.org/projects/detail?uuid=615d1d2e-7dd9-47c9-8ef9-3ab81267d50f - duplicate project Wrangled by lattice - deleted our entry
[ ] https://contribute.data.humancellatlas.org/projects/detail?uuid=5086a7a2-d5d8-44b0-af94-93d5e4c46f8c

Acceptance criteria

[ ] All projects have been triaged according to our SOPs

Potentially for @idazucchi to pick up next week.

the last project looks very weird

Group discussion: the last project seems to be created in error.
@ESapenaVentura to add importer script output for the broken project.

I've added a new project here with the paper from Enrique's comment
This was probably not added to ingest because the doi cannot be found in Europe PMC
I've opened a ticket about this issue in the Ops board


42

"Answer to the Ultimate Question of Life, the Universe, and Everything".

your comment ruined the pr.  If one must ask then one does not know.


[WIP] Makes MediaType (MT) a wrapper around HttpAbstractions MediaTypeHeaderValue (MTHV)

Making this PR to identify the current issues between MT and MTHV. Here are the differences I identified.
MTHV is case sensitive and MT is case insensitive
[Fact]
public void GetParameter_IsCaseInsensitive()
{
    // Arrange
    var mediaType = "application/json;charset=utf-8";
    var expectedParameter = new StringSegment("utf-8");

    var parsedMediaType = new MediaType(mediaType);

    // Act
    var result = parsedMediaType.GetParameter("CHARSET");

    // Assert
    Assert.Equal(expectedParameter, result);
}

MT supports "*" for parameters, while MTHV says it is invalid.
[InlineData("text/entity+json;*", true)]

This will fail to parse in MTHV.
MT removes quotes around values that don't require it, while MTHV keeps them
This passes for MT because it removes quotes around utf-8.
public static TheoryData<string> MediaTypesWithParameters
{
    get
    {
        return new TheoryData<string>
        {
            "application/json+bson;format=pretty;charset=utf-8;q=0.8",
            "application/json+bson;format=pretty;charset=\"utf-8\";q=0.8",
        };
    }
}

[Theory]
[MemberData(nameof(MediaTypesWithParameters))]
public void Constructor_CanParseMediaTypesWithParameters(string mediaType)
{
    // Arrange & Act
    var result = new MediaType(mediaType, 0, mediaType.Length);

    // Assert
    Assert.Equal(new StringSegment("application"), result.Type);
    Assert.Equal(new StringSegment("json+bson"), result.SubType);
    Assert.Equal(new StringSegment("json"), result.SubTypeWithoutSuffix);
    Assert.Equal(new StringSegment("bson"), result.SubTypeSuffix);
    Assert.Equal(new StringSegment("pretty"), result.GetParameter("format"));
    Assert.Equal(new StringSegment("0.8"), result.GetParameter("q"));
    Assert.Equal(new StringSegment("utf-8"), result.GetParameter("charset"));
}

MT can take the full Accept header in the constructor, parse a single type, and not throw an exception.
With MT, we can call
var parsedMediaType = new MediaType("text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8", start, length: null);

and it would succeed, returning the first type in the list of MediaType, "text/html", in the MediaType struct. MTHV will fail to parse this. We can try to parse it as a list and take the first one, but that seems bad.
cc/ @muratg @Eilon

@jkotalik,
Thanks for having already signed the Contribution License Agreement. Your agreement was validated by .NET Foundation. We will now review your pull request.
Thanks,
.NET Foundation Pull Request Bot

There are around ~25 tests failing right. We will need to make modifications to MediaTypeHeaderValue to support some of the scenarios.

Let's go over the failing tests first to make sure it's behavior we need to keep.

@jkotalik thanks for sending this! For some of the issues above, what do the relevant RFCs say about things? E.g. case-sensitivity.

I just realized that the case sensitivity is my own implementation error. I will fix that.

The "*" syntax for parameters is not part of the RFC spec. But I think @javiercn had a good reason to support this?
Quotes around values that don't require quotes is a usability concern. In general MTHV will maintain the wire sent format, so it will keep the quotes.
Parsing the MediaType chunk by chunk will need to be fixed in the parsing logic.

Chunk by chunk? Like an IEnumerable parser? That's not a parser fix for MTHV, you need a very different kind of parser.
That said, that sounds like an internal consumption pattern we may not need to keep.

To clarify, I mean passing in a StringSegment with "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8" as the input to the MediaType constructor.
I believe I will just need to change the AcceptHeaderParser class to eventually call MediaTypeHeaderValue.TryParseList();.

Actually, should we port/remove AcceptHeaderParser to HttpAbstractions too?

It looks redundant with MTHV, you may be able to shim / replace it.

Re: the * syntax
I have a feeling that this is needed by MVC for where a developer is specifying which media types their formatter accepts/produces/whatever. So, I don't think that the * syntax would ever appear on the wire, but it would appear in code as a way to specify a constraint. Just the same as c:\*.txt is not a valid filename, but it's something a developer would specify to indicate an allowed pattern.

More investigation:

MTHV will use the first quality parameter if more than one quality parameter is provided. MT will use the last one.
Quality parameters are not allowed to be quoted. MTHV checks this correctly, while MT does not. (See rfc 2616.

Re: the * syntax
I have a feeling that this is needed by MVC for where a developer is specifying which media types their formatter accepts/produces/whatever. So, I don't think that the * syntax would ever appear on the wire, but it would appear in code as a way to specify a constraint. Just the same as c:*.txt is not a valid filename, but it's something a developer would specify to indicate an allowed pattern.

The only way I can see us support this is to change MediaTypeHeaderValue to parse * as a parameter. I have managed to fix the quotes around parameters, however all of the other issues still exist.
@rynowak for profiling this scenario, you wanted me to do a standard content negotiation scenario. Is there an existing scenario today?

Hold of on the * change, that's not content we want to accept off the wire. It's primarily used for configuring the formatters, correct?

👍 to what @Tratcher said.
We should figure out the right way to process the * scenario. Maybe it's a different constructor, or factory helper method, or something like that.

Closing this for now as it has gotten stale.


Python environment with matplotlib and jupyter causes run-time Qt version conflicts in LXQt desktop environment

Describe the bug
When constructing a python environment wrapper, picking up matplotlib and jupyter, executing python3 inside a terminal emulator on LXQt desktop environment, and plotting with plot() provided by matplotlib.pyplot, MatPlotLib complains
Cannot mix incompatible Qt library (5.15.11) with this library (5.15.12)
Aborted (core dumped)

Steps To Reproduce
Steps to reproduce the behavior:

Boot into a NixOS on commit cf28ee2 nixos-23.11 branch, where the version of Qt 5 is 5.15.11.
Start a graphical session with LXQt desktop environment.
Open a terminal emulator (I use Alacritty).
Type lxqt-about --version and see the Qt version. Mine is
cd to the Nixpkgs project directory.
Check out nixos-unstable or nixos-23.11 (56528ee or later) where the Qt 5 version is 5.15.12 or later.
See the Qt 5 version with nix eval .#qt5.full.name. The output on my machine is "qt-full-5.15.12".
Enter the Python interactive environment by running nix run --impure --expr "(import ./. { }).python3.withPackages (ps: with ps; [ matplotlib jupyter ])".
Type import matplotlib.pyplot as plt and press enter
Type plt.plot([1, 2, 3], [1, 1, 1]) and press enter

One-liner of the Python execution part:
nix shell --impure --expr "(import ./. { }).python3.withPackages (ps: with ps; [ matplotlib jupyter ])" -c python3 -c "__import__('matplotlib.pyplot').pyplot.plot([1, 2, 3], [1, 1, 1])"
(It cannot be replaced with nix run due to a Nix bug NixOS/nix#8900.)
Expected behavior
There should be no error and no abortion. The interactive environment should print something like [<matplotlib.lines.Line2D object at 0x7fb179f10d50>] after plot(), and show the plot after plt.show(). The one-liner should print nothing.
Screenshots

Additional context
$ lxqt-about -v
lxqt-about 1.4.0
liblxqt   1.4.0
Qt        5.15.11

Notify maintainers
@lovek323 @veprbl (MatPlotLib maintainers)
@costrouc (author of a Jupyter refactoring commit 6102dd1991e0cd5d3fa0e04ac6934bb90c0ea438)
@FRidh (Python package committer)
@romildo (LXQt maintainer)

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.
[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
output here

Add a :+1: reaction to issues you find important.

I believe I have a very related problem.
My set-up is:
flake.nix
{
  description = "Reproducer";

  inputs = {
    nixpkgs.url = "nixpkgs/nixos-24.05";
    flake-utils.url = "github:numtide/flake-utils";
  };
  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs { inherit system; };
        myPyPkgs = pkgs.python3Packages;

        base-dependencies = with myPyPkgs; [ matplotlib numpy ];
        bad-jupyter = with myPyPkgs; [ jupyter ];
        good-jupyter = with myPyPkgs; [ pkgs.jupyter ];

      in {
        devShells.default = pkgs.mkShell {
          nativeBuildInputs = base-dependencies;
        };

        devShells."badJupyter" = pkgs.mkShell {
          nativeBuildInputs = base-dependencies ++ bad-jupyter;
        };

        devShells."goodJupyter" = pkgs.mkShell {
          nativeBuildInputs = base-dependencies ++ good-jupyter;
        };

      });
}

flake.lock
{
  "nodes": {
    "flake-utils": {
      "inputs": {
        "systems": "systems"
      },
      "locked": {
        "lastModified":<PHONE_NUMBER>,
        "narHash": "sha256-SZ5L6eA7HJ/nmkzGG7/ISclqe6oZdOZTNoesiInkXPQ=",
        "owner": "numtide",
        "repo": "flake-utils",
        "rev": "b1d9ab70662946ef0850d488da1c9019f3a9752a",
        "type": "github"
      },
      "original": {
        "owner": "numtide",
        "repo": "flake-utils",
        "type": "github"
      }
    },
    "nixpkgs": {
      "locked": {
        "lastModified":<PHONE_NUMBER>,
        "narHash": "sha256-sga9SrrPb+pQJxG1ttJfMPheZvDOxApFfwXCFO0H9xw=",
        "owner": "NixOS",
        "repo": "nixpkgs",
        "rev": "12bf09802d77264e441f48e25459c10c93eada2e",
        "type": "github"
      },
      "original": {
        "id": "nixpkgs",
        "ref": "nixos-24.05",
        "type": "indirect"
      }
    },
    "root": {
      "inputs": {
        "flake-utils": "flake-utils",
        "nixpkgs": "nixpkgs"
      }
    },
    "systems": {
      "locked": {
        "lastModified":<PHONE_NUMBER>,
        "narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
        "owner": "nix-systems",
        "repo": "default",
        "rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
        "type": "github"
      },
      "original": {
        "owner": "nix-systems",
        "repo": "default",
        "type": "github"
      }
    }
  },
  "root": "root",
  "version": 7
}

main.py
import matplotlib.pyplot as plt
import numpy as np

arr = np.ones((12, 12))
plt.imshow(arr)
plt.savefig("foobar.png")

When I run
nix develop .
python3 main.py

all is good.
When I run
nix develop .#badJupyter
python3 main.py

I get:
qt.qpa.plugin: Could not find the Qt platform plugin "xcb" in ""
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

When I run
nix develop .#goodJupyter
python3 main.py

all is good again.
@lovek323 @veprbl, do you have any insights on this? Apparently python3Packages.jupyter is without maintainers at the moment.
Currently python3Packages.jupyter just packages https://pypi.org/project/jupyter/ from PyPI which installs all kind of things including a qtconsole which is probably incompatible with other Nix components.
What shall we do? Does it make sense to remove python3Packages.jupyter and just point to the jupyter package?

@phiadaarr  Do you really need python3Packages.jupyter? python3Packages.notebook should cover most of use cases.

The issue in the original post I experience myself. And moreover, there seems to be a requirement to have the same version of wayland stuff in flake as in the host system. And none of this is matplotlib-specific, I suspect.

@phiadaarr Do you really need python3Packages.jupyter? python3Packages.notebook should cover most of use cases.

Thanks for this (potential) workaround (I have not tried it yet). I believe in any way, the behaviour that I describe above is a bug.

The issue in the original post I experience myself. And moreover, there seems to be a requirement to have the same version of wayland stuff in flake as in the host system. And none of this is matplotlib-specific, I suspect.

I am on X11. So it appears to be independent of X11 vs wayland.

I am on X11. So it appears to be independent of X11 vs wayland.

My information may be outdated, but your issue is a known one, and is Qt-specific. The Qt in nixpkgs is setup to use QT_PLUGIN_PATH environment variable set by wrapQtAppsHook. In that approach, no qt backend can work from a normal "python3" invocation.


[Question] Refresh ENV during runtime

I make changes to my .env files during runtime but when I read the ENV variables during the same runtime it shows the old values.  Is there a way to refresh or reload the ENV from the latest files during runtime?

Sorry, this question was added to the wrong repo.


PostingsIterator: Address nil pointer dereferencing

If the postingsList of the iterator is nil or empty,
there's nothing to find, so we need not invoke the
nextDocNumAtOrAfterClean(..) method of PostingsIterator
when the iterator's postings is nil.

Related to: blevesearch/bleve#1606

This fix attempts to address ..

panic: runtime error: integer divide by zero

goroutine 95122 [running]:
github.com/blevesearch/zapx/v15.(*PostingsIterator).nextDocNumAtOrAfterClean(0xc0e8a4f180, 0x8a64, 0x0, 0x0, 0xc020b45ce0, 0xc02e4854a0)
	/go/src/github.com/fubotv/services-search/vendor/github.com/blevesearch/zapx/v15/posting.go:683 +0x3bb
github.com/blevesearch/zapx/v15.(*PostingsIterator).nextDocNumAtOrAfter(0xc0e8a4f180, 0x8a64, 0x36e9ac2ad10be90a, 0xc2676a1420, 0x360000c02e485540, 0xd2935e)
	/go/src/github.com/fubotv/services-search/vendor/github.com/blevesearch/zapx/v15/posting.go:566 +0x2d6
github.com/blevesearch/zapx/v15.(*PostingsIterator).nextAtOrAfter(0xc0e8a4f180, 0x8a64, 0x1, 0xd31d30, 0xc0095e3100, 0x8a64)
	/go/src/github.com/fubotv/services-search/vendor/github.com/blevesearch/zapx/v15/posting.go:476 +0x4d
github.com/blevesearch/zapx/v15.(*PostingsIterator).Advance(0xc0e8a4f180, 0x8a64, 0x0, 0x8a64, 0x0, 0x0
...

Let's discuss this in the next sync-up.
Though i.postings == nil check makes sense, and if that was the issue there, not clear about the stack trace crash dumps.

Updated the commit message (with the latest patch set) to indicate the exact stack trace that this commit attempts to address.
Note that https://github.com/blevesearch/bleve/issues/1606 captures several stack traces.

If postings was nil or chunkSize was zero, then it should have crashed at line no: 677 itself?
https://github.com/blevesearch/zapx/blob/master/posting.go#L677
So, not sure how this fix is going to address that? Or am I missing the obvious here.


As a player in-game, I want information about the current round's youtube video to be revealed incrementally to try and deduce if the comments in my hand match the video

Story: As a player in-game, I would like to see all the currently revealed information about the YouTube video to try and deduce whether my hand (comments) might match the video
Acceptance criteria:

[ ] sopra-fs23-group-08/client#17
[x] sopra-fs23-group-08/client#16
[ ] sopra-fs23-group-08/client#15

Priority: Critical
Estimate: 4-8h

Client receives information incrementally: 937317224af2009d37736890c155ed60dcb2fcf8


[Support] Need guideline and building unitTest for osx

Dear team,
We love your work, please help support us on how to build current unitTest for bip32 (available in sourcecode). We tried building but still got below error.
Many thanks
John

You need to include modm-donna-32bit.c into your project.


I get Thread 1 Signal error in App delegate

After opening podfile and downloading SearchTextField 1.1.4  I created an outlet as follows:
@IBOutlet weak var countrySearchTextField: SearchTextField!
And then in viewDidLoad, i put in this function
{            countrySearchTextField.filterStrings(["United States", "England", "China"])
}
I am doing autocomplete for countries, but when I run this code, the Thread 1 Signal error comes up. Much help appreciated!

I got it too. @Adrian-Yao , did you find any solution?
Thanks

i hade a problem like this too. Not sure if it is the same but on my side it was caused because i hadn't changed the textfield typ in the storyboard. There it still was UITextField and not SearchTextField.


Tests fail with model definitions

I added three model to the mirage dir in the dummy app. Some interesting tests fail although I'm not sure if they are related to my problem @samselikoff discussed on slack yesterday.
See https://github.com/alecho/ember-cli-mirage/tree/tests-fail-with-model-defintions

Thanks for opening this (and sorry for the delay!). I'll check it out before cutting 0.2.0!

Just cloned that repo and only got one jshint error..you still seeing anything here?

Did you check out the tests-fail-with-model-defintions branch? It's been awhile and I don't know if I have the same tests failing as before. Here's the two the fail for me:
not ok 37 PhantomJS 2.0 - Acceptance: Custom handlers: You can customize the response by passing a handler function that returns the desired body
    ---
        actual: >

        expected: >
            Max depth.,Max depth.
        Log: |
    ...
not ok 40 PhantomJS 2.0 - Acceptance: Custom handlers: returning a non-blank response from a custom handler whose default status is 204 changes the status to 200
    ---
        actual: >
            [object Object]
        expected: >
            [object Object]
        message: >
            The response is correct
        Log: |
    ...

This was a result of trying to use mirage for a JSON API mock a couple month ago and I know a lot of work has been done since then so I'll have to give the beta a go and see if I have any issues.

There are some models in the current test suite and it's passing so I'm going to close this for now. If you are still seeing this in beta.3 or later, we can reopen.
Thank you for opening this issue!

Thanks for taking the time to review this. I'll check out Beta 2 or 3 today, if I can get the chance.

On Jan 24, 2016, at 10:39 AM, Sam Selikoff<EMAIL_ADDRESS>wrote:
There are some models in the current test suite and it's passing so I'm going to close this for now. If you are still seeing this in beta.3 or later, we can reopen.
Thank you for opening this issue!
—
Reply to this email directly or view it on GitHub.

Beta 3 is working great! Thanks to Sam and everyone for their hard work.


chore: add @pjungermann to CODEOWNERS

Add @pjungermann as code owners for created modules
or those with many contributions.
Signed-off-by: Patrick Jungermann<EMAIL_ADDRESS>Hey, I just made a Pull Request!

:heavy_check_mark: Checklist

[ ] A changeset describing the change and affected packages. (more info)
[ ] Added or updated documentation
[ ] Tests for new functionality and regression tests for bug fixes
[ ] Screenshots attached (for UI changes)
[x] All your commits have a Signed-off-by line in the message. (more info)

Thanks! :pray:


DynamicBatchSampler

Hi,
I'm trying to understand the implementation of DynamicBatchSampler.
I would like to know, in _get_boundaries_through_warping, why would you use the lognorm of s=1 to get the quantiles and linearly scale up to max_batch_length. What about using lognorm.fit ?
cc: @popcornell
Thanks in advance !

cc @anautsch maybe as well

Hi @bofenghuang yeah, that warping idea is on me. It's resolving time resolution in the latent statistical space rather than needing to define it explicitly for every audio collection that one fancies at a time. I write the following bulk just to make sure we are on the same page - from your message I assume you know it intuitively already, just so others can follow when reading this when archived.
Please take a look at our tutorial for context. The prequel to your question:

long audios are a burden, random batches run easily oom for limited VRAM
reducing batch size to 1 takes for ever in training, it wastes time and power consumption (lots of padding also)
so, how do we create batches that need the least padding possible while maximising training efficiency?
the go-to approach was: let's create buckets that have properties and we treat these properties then - words - so - one bucket takes only one long audio, others some medium audios, and others all the small ones - to reduce padding, tiny audios are put in here & there to reduce padding -> note: VRAM determines what's the max length of a bucket with the longest audio

Now, how to get these buckets with their fancy properties?
Without the latent space/warping approach, one would need to define: 0 to 0.2; 0.2 to 0.6; 0.6 to 1.2; ... or to script some exponential growth there. Rationale for exponential growth: most datasets are log-normal distributed when it comes to audio duration. So, the buckets are projected to treatment in a linear space. Therefore, the log-normal distribution is used.
About your question.
Which log-normal distribution to use is perhaps somewhat arbitrary. ;-)
Here, the goal was to get an initial handle on:

max_batch_length which represents the VRAM limit
num_quantiles which represents the targeted resolution in the latent space

For this latent space, your question is about:

why use standard parameters for the assumed distribution?
why not using a good fit of the actual duration distribution?

The answer might be depressingly simple: to get the PR & tutorial out for later discussions like this one.
My gut feeling is that a distribution fit should play out better than some arbitrarily assumed distribution. Would you be up to dive into tests on this topic? It w/could make sense also to move away then from the log-normal assumption and use a general fit - in the end, what matters here is that quantiles become linear in their handling through warping distributions. Another issue might also rest in: for the rest of the dynamic batching, why not have simply three or five bucket types and sort the rest in from long to short audios?

Hi @anautsch, thank you very much for your explanations ! And yeah, my question was exactly why using the arbitrary lognorm but not one with the parameters fitted on dataset.
When checking the bucketing log in _generate_batches func, I found that the reason that current DynamicBatchSampler works very well with num_buckets=60, as mentioned in tuto (With Dynamic Batching: % True samples 0.9206810914442322, % of padding 0.07931890855576784, Total time 11.002478837966919), is that a large #buckets is used, but only 6 buckets are filled in fact. I think this has justified the need for a more fit distribution.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 0 with boundary 0.0-7.6 and batch_size 71: Num Examples 199.0, Num Full Batches 1.000, Pad Factor 120.711.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 1 with boundary 7.6-10.2 and batch_size 53: Num Examples 114.0, Num Full Batches 1.000, Pad Factor 28.153.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 2 with boundary 10.2-12.3 and batch_size 44: Num Examples 149.0, Num Full Batches 3.000, Pad Factor 18.298.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 3 with boundary 12.3-14.2 and batch_size 38: Num Examples 377.0, Num Full Batches 9.000, Pad Factor 14.158.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 4 with boundary 14.2-16.0 and batch_size 34: Num Examples 594.0, Num Full Batches 16.000, Pad Factor 11.702.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 5 with boundary 16.0-17.7 and batch_size 30: Num Examples 86.0, Num Full Batches 2.000, Pad Factor 7.803.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 6 with boundary 17.7-19.3 and batch_size 28: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 7 with boundary 19.3-21.0 and batch_size 25: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 8 with boundary 21.0-22.6 and batch_size 24: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 9 with boundary 22.6-24.2 and batch_size 22: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 10 with boundary 24.2-25.8 and batch_size 21: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 11 with boundary 25.8-27.4 and batch_size 19: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 12 with boundary 27.4-29.0 and batch_size 18: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 13 with boundary 29.0-30.7 and batch_size 17: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 14 with boundary 30.7-32.4 and batch_size 16: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 15 with boundary 32.4-34.1 and batch_size 15: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 16 with boundary 34.1-35.8 and batch_size 15: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 17 with boundary 35.8-37.5 and batch_size 14: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 18 with boundary 37.5-39.4 and batch_size 13: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 19 with boundary 39.4-41.2 and batch_size 13: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 20 with boundary 41.2-43.1 and batch_size 12: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 21 with boundary 43.1-45.0 and batch_size 12: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 22 with boundary 45.0-47.0 and batch_size 11: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 23 with boundary 47.0-49.1 and batch_size 11: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 24 with boundary 49.1-51.2 and batch_size 10: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 25 with boundary 51.2-53.4 and batch_size 10: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 26 with boundary 53.4-55.7 and batch_size 9: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 27 with boundary 55.7-58.1 and batch_size 9: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 28 with boundary 58.1-60.5 and batch_size 8: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 29 with boundary 60.5-63.0 and batch_size 8: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 30 with boundary 63.0-65.7 and batch_size 8: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 31 with boundary 65.7-68.4 and batch_size 7: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 32 with boundary 68.4-71.3 and batch_size 7: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 33 with boundary 71.3-74.3 and batch_size 7: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 34 with boundary 74.3-77.5 and batch_size 7: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 35 with boundary 77.5-80.8 and batch_size 6: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 36 with boundary 80.8-84.3 and batch_size 6: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 37 with boundary 84.3-88.0 and batch_size 6: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 38 with boundary 88.0-91.9 and batch_size 5: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 39 with boundary 91.9-96.1 and batch_size 5: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 40 with boundary 96.1-100.5 and batch_size 5: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 41 with boundary 100.5-105.2 and batch_size 5: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 42 with boundary 105.2-110.3 and batch_size 4: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 43 with boundary 110.3-115.7 and batch_size 4: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 44 with boundary 115.7-121.6 and batch_size 4: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 45 with boundary 121.6-128.0 and batch_size 4: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 46 with boundary 128.0-134.9 and batch_size 4: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 47 with boundary 134.9-142.6 and batch_size 3: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 48 with boundary 142.6-151.1 and batch_size 3: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 49 with boundary 151.1-160.5 and batch_size 3: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 50 with boundary 160.5-171.2 and batch_size 3: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 51 with boundary 171.2-183.3 and batch_size 2: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 52 with boundary 183.3-197.4 and batch_size 2: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 53 with boundary 197.4-214.0 and batch_size 2: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 54 with boundary 214.0-234.0 and batch_size 2: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 55 with boundary 234.0-258.8 and batch_size 2: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 56 with boundary 258.8-291.2 and batch_size 1: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 57 with boundary 291.2-336.0 and batch_size 1: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 58 with boundary 336.0-405.7 and batch_size 1: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.
INFO:speechbrain.dataio.sampler:DynamicBatchSampler: Bucket 59 with boundary 405.7-544.0 and batch_size 1: Num Examples 0.0, Num Full Batches 0.000, Pad Factor 0.000.

Based on this super tutorial, I have added some other bucketing methods and ended up with following results on MiniLibriSpeech:

Random Sampling
After sorting (minimum padding but no randomness)
Org Dynamic Batching (the one in speechbrain current version)
Mdf Dynamic Batching w/ fitted lognorm (bucket boundaries set up with lognormal distribution fitted on dataset)
Mdf Dynamic Batching w/ fitted beta (bucket boundaries set up with beta distribution fitted on dataset, mentioned in tuto)
Mdf Dynamic Batching w/ Kmeans (examples aggregated into buckets by Kmeans)

I have also measured the time for creating the buckets to make sure we don't add too much time overhead (see Sampler initialization time).
Random Sampling: % True samples 0.7691900805460599, % of padding 0.2308099194539402, Total time 8.152731657028198
After sorting: % True samples 0.9877999198583547, % of padding 0.012200080141645281, Total time 7.920250654220581

num_buckets: 5
Org Dynamic Batching: % True samples 0.8037098958878093, % of padding 0.19629010411219072, Total time 8.982778787612915, Sampler initialization time 0.00830388069152832
Mdf Dynamic Batching w/ fitted lognorm: % True samples 0.9105685331121831, % of padding 0.08943146688781692, Total time 8.017155408859253, Sampler initialization time 0.2598123550415039
Mdf Dynamic Batching w/ fitted beta: % True samples 0.9061257346980318, % of padding 0.09387426530196818, Total time 8.40556263923645, Sampler initialization time 0.12327051162719727
Mdf Dynamic Batching w/ Kmeans: % True samples 0.9172593243667561, % of padding 0.08274067563324397, Total time 8.077270984649658, Sampler initialization time 0.37424445152282715

num_buckets: 10
Org Dynamic Batching: % True samples 0.7820164173717347, % of padding 0.2179835826282653, Total time 8.377742528915405, Sampler initialization time 0.00844120979309082
Mdf Dynamic Batching w/ fitted lognorm: % True samples 0.9358722055686697, % of padding 0.06412779443133029, Total time 8.322029829025269, Sampler initialization time 0.2533447742462158
Mdf Dynamic Batching w/ fitted beta: % True samples 0.9443160562979621, % of padding 0.055683943702037865, Total time 8.235085248947144, Sampler initialization time 0.11742091178894043
Mdf Dynamic Batching w/ Kmeans: % True samples 0.9554278736419016, % of padding 0.044572126358098395, Total time 8.240896701812744, Sampler initialization time 0.12387490272521973

num_buckets: 15
Org Dynamic Batching: % True samples 0.7756018507479978, % of padding 0.22439814925200216, Total time 7.931443691253662, Sampler initialization time 0.007909774780273438
Mdf Dynamic Batching w/ fitted lognorm: % True samples 0.9518678718022273, % of padding 0.048132128197772656, Total time 8.096233129501343, Sampler initialization time 0.27107810974121094
Mdf Dynamic Batching w/ fitted beta: % True samples 0.9599471840190559, % of padding 0.04005281598094407, Total time 8.055798530578613, Sampler initialization time 0.1018073558807373
Mdf Dynamic Batching w/ Kmeans: % True samples 0.9694473523120741, % of padding 0.03055264768792584, Total time 8.201721429824829, Sampler initialization time 0.19640326499938965

num_buckets: 20
Org Dynamic Batching: % True samples 0.7710137381778893, % of padding 0.22898626182211065, Total time 7.569898366928101, Sampler initialization time 0.00859689712524414
Mdf Dynamic Batching w/ fitted lognorm: % True samples 0.9607128662922578, % of padding 0.03928713370774222, Total time 8.13912057876587, Sampler initialization time 0.31614184379577637
Mdf Dynamic Batching w/ fitted beta: % True samples 0.9690223254725432, % of padding 0.030977674527456776, Total time 8.036609172821045, Sampler initialization time 0.10276174545288086
Mdf Dynamic Batching w/ Kmeans: % True samples 0.9769193297979476, % of padding 0.023080670202052424, Total time 7.95473051071167, Sampler initialization time 0.20025897026062012

num_buckets: 60
Org Dynamic Batching: % True samples 0.9203420430290024, % of padding 0.07965795697099763, Total time 8.252700090408325, Sampler initialization time 0.008800029754638672
Mdf Dynamic Batching w/ fitted lognorm: % True samples 0.9846392039967643, % of padding 0.015360796003235686, Total time 7.790568828582764, Sampler initialization time 0.2609591484069824
Mdf Dynamic Batching w/ fitted beta: % True samples 0.9887281684322106, % of padding 0.011271831567789437, Total time 8.079726696014404, Sampler initialization time 0.10100769996643066
Mdf Dynamic Batching w/ Kmeans: % True samples 0.9925841676005036, % of padding 0.0074158323994964305, Total time 8.379063844680786, Sampler initialization time 0.3739185333251953

System Info
- SpeechBrain version: 0.5.11
- PyTorch version: 1.11.0+cu113
- Scipy version: 1.8.0
- Python version: 3.8.13
- Platform: Linux version 5.4.0-105-generic
- CPU: Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
- #logical cpus: 8

The code is here.
I can make a PR if this approach is approved and look forward to further discussions :)
Cheers

@bofenghuang neat! It's encouraging to see your enthusiasm :D
Any choice is arbitrary, having some facts selection to it, doesn't make choices systematic ;)
@popcornell worked a lot on the final tutorial!
As you demonstrated, the num_buckets helps DynamicBatching to do something but it is far from what one would think should happen (10% of the created buckets are actually used, the rest remains unfilled). Therefore, I'd be curious if it makes sense to have any distributional assumption at all - or to treat this entirely on the categorical level (have 4, 5, 6, ... bucket types to be filled whatever they are - the limit is VRAM). What I try to say:

we need to question what is the underlying problem actually
which parameters (among those we know/not know) are critical to solve it?
the actual problem might be much more simple but statistical understanding of its inner workings might help to find a more adequate solution

That's also what your findings on kmeans support: fitting distributions can help but in the end, we get a dataset and that's our entire population to take care about during operation. (New dataset, new task - everything back to start.) Yet, what does the batch creation of DynamicBatching imply to the overall training? How to test that for one and many datasets - how do we have the guarantees that we need to have?
If the number of total batches created is small and padding is small - how much random permutation of the files in these batches is then possible? Or would there only be one solution in how to draw a batch after kmeans?
What's your take: is there benefit in giving many choices here on which DynamicBatching approach to use? Are there "relevant" ones?
// The goal is to provide useful tools which make a complex issue to be handled intuitively.
% True samples & % of padding say the same thing - % of padding could be more useful to develop intuition here but it's not what one thinks of first (so both were in the tutorial). About the Sampler initialization time - it's relevant to decompose the Total time to understand what's going on internally.
DynamicBatching has a few use cases which need better testing if they are fulfilled:

low VRAM: create batches that are not running oom
low VRAM: don't be wasteful - reduce padding
low VRAM: see as much data as possible; don't remove too much bc of low VRAM
low VRAM: different epochs should feature different batch samplings = random permutations of files
low VRAM: while long audios might fill up an entire bucket, two half-long audios of that might also run oom - how to instantiate buckets properly for all audios to be processed
high VRAM: less padding means fewer batches means faster training

@TParcollet @popcornell please add to that list what I forgot - there might be more requirements to DynamicBatching ^^"
The dimensions to investigate a better DynamicBatching need to also work on:

mini LibriSpeech
full LibriSpeech
CommonVoice (some language sets there are at a few MB; some GB; lots GB)
...

What we observe on small datasets might not hold on large datasets. How about discussing next week a strategy for testing and developing DynamicBatching further?

@anautsch nice! Look forward to our discussion :)

Hi @anautsch ,
As discussed last week, I came back with the tests on more datasets :

MiniLibriSpeech
LibriSpeech
CommonVoice IT
CommonVoice FR

I used batch_size=32 for the random sampling and the sequential sampling after sorting. max_batch_len for DynamicBatchSampler is set to batch_size * max_dur to make sure they have the same memory peak.
Here are the results.

As per slack: Random Sampling & After sorting are w/o the dyn batching warping
I'll rewrite your resutls - let's see if this formatting helps. I'll add: %init time = sampler initialization time / total time
A = num_buckets | B = ratio of padding | C = #batches | D = total time | E = %init time
I think about them as:
resolution depth | avg energy waste | total energy spent | demanded impatience | one-time overhead
The init time is extra to the total time. After looking at the script, they are two separate timings. The total time refers to all batches within an epoch. To put this number in relation, one might think of distributed data parallel (DDP) & shards, where batches are composed once (init time effort), and then each batch is randomly sampled among the available batches (sets of batches can be delegated as shards to specific multi-gpu nodes).
The results below are for 1 single epoch.
Rnd = Random Sampling
Org = Org Dynamic Batching
log = Mdf Dynamic Batching w/ fit lognorm
bta = Mdf Dynamic Batching w/ fit beta
kms = Mdf Dynamic Batching w/ kmeans
srt = After sorting
MiniLibriSpeech
(max. 17.275s -> max. length: 552.8)
    | A  |   B   | C   |   D   |   E
Rnd | 32 | 23.0% |  48 | 1.35s | N/A
Org | 5  | 19.5% | 254 | 1.59s |  0.6%
    | 10 | 21.8% | 109 | 1.79s |  0.6%
    | 20 | 22.9% |  55 | 1.62s |  0.6%
    | 60 |  8.3% |  40 | 1.61s |  0.8%
log | 5  |  9.7% |  41 | 1.36s |  9.8%
    | 10 |  6.5% |  45 | 1.57s |  8.4%
    | 20 |  3.9% |  45 | 1.56s |  8.2%
    | 60 |  1.5% |  70 | 1.52s |  8.8%
bta | 5  |  9.3% |  42 | 1.23s |  4.5%
    | 10 |  5.6% |  43 | 1.81s |  3.2%
    | 20 |  3.1% |  46 | 1.47s |  4.2%
    | 60 |  1.1% |  72 | 1.71s |  3.2%
kms | 5  |  8.5% |  41 | 1.58s | 35.7%
    | 10 |  4.5% |  41 | 1.56s | 36.2%
    | 20 |  2.4% |  45 | 1.50s | 34.8%
    | 60 |  0.7% |  76 | 1.62s | 36.9%
srt | 32 |  1.2% |  48 | 1.35s | N/A

Log-normal and beta appear alike; log-normal will scale better with more epochs. K-means has a great init penalty but promises reduced padding (yet, 70+ batches are a lot compared with some 40 batches). Distribution fitting helps a lot compared to Org dynamic batching; but: could this one be better on its own?
LibriSpeech
(max. 29.735s -> max. length: 951.52)
Rnd | 32 | 25.2% |  8789 |  63.3s | N/A
Org | 5  | 21.0% | 46874 | 105.7s |  1.4%
    | 10 | 23.6% | 20089 |  72.6s |  2.0%
    | 20 | 25.0% | 10045 |  66.0s |  2.1%
    | 60 | 19.2% |  4727 |  61.8s |  2.2%
log | 5  | 10.7% |  4736 |  61.9s | 12.7%
    | 10 |  7.2% |  3999 |  56.8s | 10.0%
    | 20 |  4.4% |  3849 |  56.3s | 10.3%
    | 60 |  1.8% |  3766 |  52.2s | 11.0%
bta | 5  | 15.3% |  4913 |  61.9s | 10.7%
    | 10 | 10.7% |  4134 |  58.4s | 11.3%
    | 20 |  6.2% |  3923 |  56.4s | 11.5%
    | 60 |  2.3% |  3786 |  54.8s | 11.9%
kms | 5  |  9.6% |  5502 |  56.4s | 10.7%
    | 10 |  5.3% |  4427 |  56.6s |  9.6%
    | 20 |  2.8% |  3802 |  54.2s | 10.7%
    | 60 |  0.9% |  3730 |  52.7s | 16.8%
srt | 32 | 0.01% |  8789 |  50.7s | N/A

Beta suggest slight gains to log-normal, and k-means sits in the same boat. There is no clear advantage among the distribution fits of one over the other; it'd be some nitty gritty argument.
CommonVoice IT
(max. 11.556s -> max. length: 369.792)
Rnd | 32 | 35.6% |  4416 | 40.7s | N/A
Org | 5  | 24.4% | 23548 | 58.7s |  1.3%
    | 10 | 30.6% | 10092 | 43.6s |  1.6%
    | 20 | 34.8% |  5046 | 41.3s |  1.7%
    | 60 | 19.7% |  2454 | 35.5s |  1.9%
log | 5  | 18.2% |  2529 | 34.3s |  6.9%
    | 10 | 15.1% |  2290 | 32.5s |  5.4%
    | 20 | 14.2% |  2175 | 33.9s |  5.3%
    | 60 | 15.5% |  2120 | 34.6s |  5.1%
bta | 5  | 18.2% |  2511 | 34.4s |  3.5%
    | 10 | 15.0% |  2275 | 34.0s |  3.5%
    | 20 | 14.1% |  2173 | 32.2s |  3.7%
    | 60 | 15.7% |  2129 | 34.1s |  3.5%
kms | 5  | 17.1% |  2386 | 34.1s | 14.1%
    | 10 | 13.9% |  2219 | 33.3s | 16.9%
    | 20 | 13.1% |  2137 | 33.0s | 14.4%
    | 60 | 14.9% |  2111 | 33.5s | 15.7%
srt | 32 |  8.2% |  4416 | 31.9s | N/A

K-means > Log-norm > beta. Especially, for DDP & shareds.
None of the algorithms tackled the padding issue to a point of contempt satisfaction!
CommonVoice FR
(max. 17.856s -> max. length: 571.392)
Rnd | 32 | 36.8% | 13872 | 130.7s | N/A
Org | 5  | 24.7% | 73984 | 186.3s |  1.2%
    | 10 | 31.5% | 31708 | 136.0s |  1.8%
    | 20 | 36.0% | 15854 | 134.6s |  1.8%
    | 60 | 34.4% |  6345 | 131.4s |  1.7%
log | 5  | 17.8% |  5996 | 112.6s | 10.5%
    | 10 | 13.5% |  5018 | 111.4s |  7.0%
    | 20 | 11.4% |  4532 | 107.2s |  7.6%
    | 60 | 10.4% |  4233 | 108.9s |  7.4%
bta | 5  | 17.8% |  5958 | 114.5s |  2.4%
    | 10 | 13.5% |  4992 | 115.2s |  2.3%
    | 20 | 11.4% |  4531 | 109.4s |  2.5%
    | 60 | 10.4% |  4243 | 111.5s |  2.5%
kms | 5  | 16.7% |  5089 | 114.8s |  7.4%
    | 10 | 12.5% |  4437 | 107.6s |  6.6%
    | 20 | 10.6% |  4235 | 110.1s |  7.2%
    | 60 | 10.1% |  4133 | 109.0s | 12.5%
srt | 32 |  5.2% | 13872 |  99.5s | N/A

The init penalty behaviour is odd. None of the distribution fits is really, really good at getting rid off padding. While the run-time is somewhat similar, k-means has the fewest batches.
It looks like all of them have their justifications and it should be to the user to decide which to take. For this, a tool to assist batch composition could be helpful!
Yet, as for a standard configuration, I do not know. As we cannot control the users & their datasets, the best option might be the beta one. @bofenghuang  what's your take?
@TParcollet @mravanelli  any ideas on the impact of number of batches to the training outcome? (does it matter/not - somewhat, how?)

@anautsch thanks for the rewriting.
I didn't realize the dynamic sampler with beta distribution has the less initialization time (except on LibriSpeech). Just when fitting the beta distribution on datasets, I keep get this warning. I don't know if this stops you to use it as the default option.
/home/bhuang/anaconda3/envs/sb/lib/python3.8/site-packages/scipy/stats/_continuous_distns.py:647: RuntimeWarning: invalid value encountered in sqrt
  sk = 2*(b-a)*np.sqrt(a + b + 1) / (a + b + 2) / np.sqrt(a*b)

For me they all have the similar behaviors in terms of number of batches and ratio of padding, better than the original fixed one. Maybe bucketing by kmeans shouldn't be the default option because it will add one more dependency and don't have obvious gain.
I put all the code here.

Dependency is a valid remark.
It's good btw you handle that import inline!
Gosh, I needed to re-read this line a bit

the "total time" reported is just one iteration of dataloader without real training, so I/O basically

"total_time" it's the total time necessary to add up the number padding in ratio to total audio per batch for 1 epoch!
Ok, then my above analysis doesn't really hold xD
Then, the actual criterion is the number of batches - everything else just puts some nuances to the discussion. Got it, sorry I was slow here...
Just to summarise per dataset, the approach with fewest num_batches, so fastest epochs:
MiniLibriSpeech | Org | 60 |  8.3% |   40 |   1.6s |  0.8%
                  log | 5  |  9.7% |   41 |   1.4s |  9.8%
                  kms | 5  |  8.5% |   41 |   1.6s | 35.7%
                      | 10 |  4.5% |   41 |   1.6s | 36.2%
LibriSpeech     | kms | 60 |  0.9% | 3730 |  52.7s | 16.8%
                  log | 60 |  1.8% | 3766 |  52.2s | 11.0%
                  bta | 60 |  2.3% | 3786 |  54.8s | 11.9%
                  kms | 20 |  2.8% | 3802 |  54.2s | 10.7%
CommonVoice IT  | kms | 60 | 14.9% | 2111 |  33.5s | 15.7%
                  log | 60 | 15.5% | 2120 |  34.6s |  5.1%
                  bta | 60 | 15.7% | 2129 |  34.1s |  3.5%
                  kms | 20 | 13.1% | 2137 |  33.0s | 14.4%
CommonVoice FR  | kms | 60 | 10.1% | 4133 | 109.0s | 12.5%
                  log | 60 | 10.4% | 4233 | 108.9s |  7.4%
                  kms | 20 | 10.6% | 4235 | 110.1s |  7.2%
                  bta | 60 | 10.4% | 4243 | 111.5s |  2.5%

Interesting that fewest padding is not always fewest batches!
Yeah, k-means is all over the place.
Would you be up for creating a PR with the following?

log-norm with num_buckets=60 as default, since it does not need additional dependencies
leaving out beta, since it might throw errors and is not the striking candidate
making k-means available as per your implementation, num_buckets=60
creating a tool to run your pre-tests with a table output -> based on that, one could make a choice on which dynamic batching method to choose with which parameters etc.

I sense I might be missing sth. - what should the next iteration of dynamic batching include from your view?
Would it make sense to invest more time into another conceptual approach or is this as good as it might get?

Yeah it was the same "total time" used in the tutorial.

Interesting that fewest padding is not always fewest batches!

I think this is related to datasets, #buckets and how the batches are generated from buckets.
In the current implementation, a batch is generated when its size reaches the predefined _max_batch_ex, or _bucket_lens[i] which is max_batch_length divided by upper boundary of the bucket.
https://github.com/speechbrain/speechbrain/blob/a8d55321962f742d193dda5d4bc5cb500335355b/speechbrain/dataio/sampler.py#L594-L599
https://github.com/speechbrain/speechbrain/blob/a8d55321962f742d193dda5d4bc5cb500335355b/speechbrain/dataio/sampler.py#L487-L489
In my version, I defined _bucket_lens[i] as max_batch_length divided by length of the longest example in the bucket.
And yeah! I would love to make a PR. But I'm not sure if we should make  num_buckets=60 as default. Isn't it too much? As said in the docstring, num_buckets represents a trade-off between speed and randomization.
Now we come back to the question we have discussed. How could we quantify the randomness ? How will this approach impact the training performance ?


New metrics to calculate for GUI output

Is your feature request related to a problem? Please describe.
I'm often interested in metrics like the coefficient of variation of a fluorescence signal over space within a single cell, which should be easily calculated given mask and fluo channel info, but isn't currently part of the output of the main CellACDC GUI.
Describe the solution you'd like
A way to add the aforementioned coefficient of variation metric, as well as other calculations that might be relevant depending on the experiment. As a stretch goal, it would be neat to have a standard set of calculations that users can select from when saving the data from a position (so one could unselect unimportant ones to save time), as well as the ability to introduce custom calculations.

Hi Jordan, I just released v1.2.4rc7 that contains the fix for this.
You can now add custom metrics that takes the signal and the background value as input and returns whatever calculation you want to do with it. You just need to write the function and put it into the <user-path>/acdc-metrics. I added some instructions in the new menu on the menubar on the top of the main GUI called Measurements
In that menu there is also a new functionality called Set measurements... that will allow you to select which metrics you want to save, and for each metric there is an info button with a description of the specific metric.
Update with pip install --upgrade cellacdc and let me know if it works, thanks!

Hi Francesco, this menu appears to be working, and seems straightforward to use. I'll let you know if I find any issues later. Thanks for adding this!

Hi Jordan, I just thought now that I might have changed something with the new Set measurements function that broke the downstream analysis, I will fix it today, thanks!

Hi Francesco,
Quick question about the CV custom measurement--does it calculate the CV after performing background subtractions? If so, does it use autoBkgr or dataPrepBkgr?
More broadly, I'm trying to quantify nuclear localization of a protein using the CV as a proxy (which has worked in the past), and wanted to make sure the CV calculated in this instance is still usable the same way.
Thanks!

Hi Jordan, so for the CV you can actually choose what to do with the background signal. If you go to your user path (something like C:\Users on Windows) there is a folder called acdc-metrics and inside that folder there is a python script called CV.py. You can see this folder path if you go to the Measurements --> Add custom metric menu on the top of the GUI, where you will get some instructions. Inside the CV.py file you will find the CV function which has the arguments called correct_with_bkgr=False and which_bkgr='auto'. With these arguments, you can choose whether you want to correct the signal using the background value and which background you want to use (anything different from 'auto' will use the dataPrepBkgr value).
I hope it's clear enough, thanks.

Got it, thank you!

Hi Francesco,
I tried creating a custom metric that is basically CV_dataPrep_Bkgr; basically copy-pasted CV.py and changed a few arguments, and also updated the function name within the script, as described. It shows up correctly when I set measurements in the GUI, but it's not showing up in overall_df once I load the data into jupyter lab.  Is there something else I need to do to get it to make it to actually calculate the metric?

Hi Jordan, the 1D array is a vector with the signal from all the pixels of each cell, so no it is not a single number but the entire distribution. It's a 1D array because when you index a NumPy array with a boolean mask (i.e., the segmented cell) the output is a flattened 1D array.
Anyway, I will look into how to get the metric into overall_df now, I keep you updated.

Hi Jordan, so I just did some tests and, after creating the CV_dataPrep_Bkgr.py as you described, I saved the metrics in the GUI and I got a new column in the acdc_output.csv file called Dia_Ph3_CV_dataPrepBkgr (where my channel is called Dia_Ph3) as expected. When I then loaded the data into jupyter-lab I get the same column in the overall_df table, so everything seems to work fine on my side.
Did you recalculate the metrics in the GUI after creating your CV_dataPrep_Bkgr.py file? Can you check if after saving the metrics you get the new column in the acdc_output.csv file? Thanks

Hi Francesco, turns out I had to restart CellACDC to get it to work! Apologies for wasting your time; I must have missed it if it's in the instructions. Thanks!
And the CV calculation makes sense now, thanks! I don't suppose you also saw my email regarding my specific data?

Hi Jordan, absolutely no problem. In general, you always have to restart Cell-ACDC to get the updates. That's because Python loads the libraries only when you run it and you need to restart to load new stuff.
For the background signal, you are right, the doc string is wrong, it is a single numeric value. In any case, it works like you said, when you subtract a single numeric value to a numpy array that value is subtracted to each element of the array.
For the values that you can use in the custom metrics, as of now, you can only manipulate the 1D array signal, you cannot combine it with other columns. However, I will keep this issue open and I can implement that feature. I will keep you updated.
For your email, I actually saw it, downloaded your sample data and started having a look at it, but I forgot to reply :D, sorry about that.

Thanks again for helping me make sense of my data! And yes, I'd love to be able to work with other columns for custom metrics. In particular, since my signal to noise is so low, I need to do autofluorescence corrections where possible. I specifically want to use the autofluorescence-corrected signal to calculate the CV, so I want to apply the AF correction before the CV calculation; not sure what order ACDC's custom metrics are calculated in, but maybe this gives you an idea of how it can be implemented.

Wonderful, thank you! I'll give it a try and let you know if I run into any issues.
Also just wanted to say, I appreciate the aesthetic upgrades to the GUI too!

Hi Francesco, the custom and combined metrics stuff is working wonderfully!
What is the best way to edit combined measurements after the fact? For instance, I tried defining something called 'mCitrine_sum_AFcorrected', but when it saved it appended the channel name again so it became 'mCitrine_mCitrine_sum_AFcorrected'. Changing the name in the .ini file didn't work, so I deleted the .ini files and redid it.

Hi Jordan, you are right, if you put the mCitrine in the name and then you only use columns from the mCitrine signal that Cell-ACDC appends the mCitrine again. I did not think of that possibility, it's an easy fix anyway.
However, you don't need to prepend mCitrine if you use only measurements from mCitrine channel because Cell-ACDC will recognise that you are asking for another mCitrine-specific measurement and it will prepend the channel. If instead, you use a mix of channels that the name stays what you requested.
I will add a checkpoint anyway that will recognise that you already prepended the channel name. Thanks for finding that!

Got it, that makes sense!
Any insights on the second issue? I.e. using a custom measurement defined for one position/experiment for another?

Oh sorry, I misread your issue.
When you say that you are defining a custom measurement in one position you mean a combined measurement right? This is just to be sure what you are referring to. I call "custom" measurement a measurement defined by a python script in the acdc-metrics, while a combined measurement is defined through the menu Measurements --> Add combined measurement....

Hi Jordan, so I did some tests and everything seems to work as expected, however, I understand that it is not clear at all how it works. So here is what is supposed to happen:
Let's assume you have a channel called mCitrine. So you open the Position with this channel and define a combined measurement as mytest = mCitrine_sum - mCitrine_mean. Now, this equation is saved in the specific Position folder in the file that ends with custom_combine_metrics.ini, but it's also saved in your user folder in the folder called acdc-metrics (for example C:\Users\jordan\acdc-metrics).
When you re-open the same Position, cell-acdc will find the custom_combine_metrics.ini and add mytest to the list of saved metrics.
When you open another Position, cell-acdc will check if there is a file in the user_folder/acdc-metrics folder called combine_metrics.ini and, if the channel mCitrine is also present in this newly opened position, it will automatically add this combined metric to the list of saved metrics.
However, if you open a Position that does not have the mCitrine channel then acdc will ignore the previously saved metric mytest, since this metric was specific to mCitrine.
When you create a combined metric that is the result of combining multiple channels, for example my_mixed_test = mCitrine_sum - GFP_sum, then cell-acdc will save as usual but to the field in the .ini file called [mixed_channels_equations]. These measurements will then be added to the list of saved metrics for ANY other Position that you open because the mixed channel equations are independent of the channel name (the example above would be stored as my_mixed_test = channel_1_sum - channel_2_sum, where 1 and 2 are determined by alphabetical order).
So in general, if you always want to have the same channel-specific equation, then you need to have all the Positions with the exact same channel name.
The reason why it works like this is that I personally work with many different channel names depending on the fluorophore or experiment that I do, so, usually, I don't want the same combined measurement that I had for mNeon when I open a completely different experiment that used tdTomato for example.
Note that you can always choose to not save a specific combined measurement by unchecking it in the Set measurements... dialogue.
I hope it's clear and let me know if you think it should behave differently. Cheers!

Hi Francesco, thanks for the explanation! What you described does make sense, but somehow it doesn't seem to work on my data for some reason!
Long story short, I defined two parameters: one that falls under the "mixed channel" category, and one that should apply to one channel. I defined and saved it for one position, which worked; I can open acdcoutputs.csv and both fields are in there.
When I open another position within the same movie, so same channel names and everything, I can see the new metrics in the right places under "set measurements"; one under mixed, one under the specific channel. But when I actually save it doesn't appear in my acdcoutput.csv for the second position.
I first saw the issue in rc29, but just checked in rc31 and it's still happening.

On a somewhat related topic, is it possible to use area, volume, etc. and other metrics when writing a python file under custom metrics? I think that might actually be easier to use for the most part than the "combined measurements", since it's possible to leave comments and such.

Hi Jordan, now I got your issue.
I opened two new issues #

Hi Jordan, I released v1.2.4rc32 which has a fix for the bug described at issue #101
Combined measurements defined at one Position should also be saved in any other Position (if they have the same channel name).
Please update with pip install --upgrade cellacdc and let me know if it works! Cheers

Hi Francesco, I tested it out and it looks like the channel-specific "combined measurement" feature is saving correctly now. However, at the same time it looks like the "mixed channels combined measurement" still isn't saving.

Hi Francesco, I tested it out and it looks like the channel-specific "combined measurement" feature is saving correctly now. However, at the same time it looks like the "mixed channels combined measurement" still isn't saving.

Hi Jordan, I released v1.2.4rc33 which has a fix for this issue.
Please update with pip install --upgrade cellacdc and let me know if it works, thanks!

Hi Francesco, no idea what's going on but the "mixed channels combined measurement" I defined still isn't saving for any position besides the one it was originally defined for. Would it help if I provided any files?
I also noticed that the channel-specific "channel measurement" doesn't always save either--haven't figured out what the pattern is, but I was testing it on a channel that hadn't been complete tracked or m-b paired. Maybe that makes a difference? (The "mixed channel combined measurement" wasn't saving for any channel, regardless of complete tracking/m-b assignment, so I think something else might be going on here.)

Hi Jordan, I think sharing the data would help a lot sure. Can you send me a Position where you defined the "mixed channels combined measurement" and one where the same measurement doesn't get saved?
The complete tracked or paired shouldn't make any difference. Could you also share a Position where the channel measurement is saved and one where it is not saved? Thanks!

Hi Francesco, here's the experiment I was testing stuff in. I defined two new metrics: one called {ch_name}_test (single-channel combined measurement, a nonsensical quantity), and one called mCitrineFFC_AF_deduction (mixed channel combined measurement, but actually only uses constants and cell volume vox). Both of these were initially defined in Position 1. I have full seg/track/m-b pairing for Positions 1 and 2, but not for 6.
Last time I was testing, this is what I saw (and should be reflected in the acdc_output.csv for each position):
Position 1: both metrics save (test and deduction)
Position 2: only "test" saves, somehow "deduction" doesn't
Position 6: neither saves
https://drive.google.com/file/d/1ofm1XjEh60TartfVohpqhSmrozkc7Ig-/view?usp=sharing

Hi Jordan, I am doing some tests and still haven't fixed the issue, but I need some information from you first.
When I open your Position_2 it doesn't save the deduction metric, but that is expected because in my user folder inside the folder acdc-metrics I don't have the deduction measurement defined because it was defined by you. In this case, when I open Position_2 it will not find this measurement from my user folder.
Could you please check if you have the deduction measurement defined in your user folder inside the acdc-metrics/combine-metrics.ini file? Could you please send me also that file so that I can do tests in your same situation?

I took the opportunity to improve the feature a little bit. Now I added that it also scans all the other Positions in the same experiment and it automatically read combined metrics defined in other Positions (and not only the ones that were saved in the user folder).
This way you can take the Positions folder with you on another computer (where you did not define measurements) and all of the Positions will have the same measurements despite the fact there are no measurements defined in the user path.
I hope it makes sense!
Anyway, I released v1.2.4rc34 which has the fix plus the improvement. Please update with pip install --upgrade cellacdc and let me know if it works.  Thank you for your patience!

Hi Francesco, sorry about the delay, I was traveling this weekend. Thanks for the fix!


hello_world session_1

@PRASANNA-416
You have successfully completed the task, but our tests expected different results. But the objective of this assignment was to get you familiar with routing, displaying data in HTML tags & controller actions, which you have completed.

@PRASANNA-416
You have successfully completed the task, but our tests expected different results. But the objective of this assignment was to get you familiar with routing, displaying data in HTML tags & controller actions, which you have completed.


Installing anything always fails

Hi
So I have downloaded haxe and I want to install lime, openfl, and flixel using haxelib install lime and haxelib install openfl and also haxelib install flixel So the installing works but it always fails, So whats wrong?

How does it works and how does it fail?

To fail, It doesn't reach the max MB So like openfl is 2.7 MB and it never goes 2.7 MB its been 3/3 so it always fails

Oh wait its fixed


make the example in readme actually work.

should fix issue #11

The example should now work.
> core/console
[info] Starting scala interpreter...
[info]
Welcome to Scala 2.11.11 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_112).
Type in expressions for evaluation. Or try :help.

scala> :paste
// Entering paste mode (ctrl-D to finish)

import _root_.spray.json.DefaultJsonProtocol._
import com.qvantel.jsonapi._

implicit val apiRoot: ApiRoot = ApiRoot(None)

@jsonApiResource final case class Employee(id: String, name: String)
@jsonApiResource final case class Company(id: String, name: String, employees: ToMany[Employee])

val acme = Company("1", "acme", ToMany.loaded(Seq(Employee("1", "number one 1"))))

val json = rawOne(acme)
val parsed = readOne[Company](json, Set("employees"))

acme == parsed // true

// Exiting paste mode, now interpreting.

import _root_.spray.json.DefaultJsonProtocol._
import com.qvantel.jsonapi._
apiRoot: com.qvantel.jsonapi.ApiRoot = ApiRoot(None)
defined class Employee
defined object Employee
defined class Company
defined object Company
acme: Company = Company(1,acme,Loaded(List(Employee(1,number one 1))))
json: spray.json.JsObject = {"data":{"attributes":{"name":"acme"},"relationships":{"employees":{"data":[{"type":"employees","id":"1"}],"links":{"related":"/companies/1/employees"}}},"links":{"self":"/companies/1"},"id":"1","type":"companies"},"included":[{"type":"employees","attributes":{"name":"number one 1"},"id":"1","links":{"self":"/employees/1"}}]}
parsed: Company = Company(1,acme,Loaded(List(...

Coverage remained the same at 51.571% when pulling 578bb08fc5bdd46f2fa121becb98800ac14a153c on Doikor:bugfix/readme into 5f44db06c658a8adfd40bd8b34403855552c8224 on qvantel:master.

@Doikor Just found that with Scala version 2.12.3, I'm stilling having this issue.
Is there something wrong? :)
build.sbt
scalaVersion := "2.12.3"
libraryDependencies ++= Seq(
  "com.qvantel" %% "jsonapi-scala-core" % "5.1.2",
  "io.spray" %%  "spray-json" % "1.3.3"
)

Welcome to Scala 2.12.3 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_131).
Type in expressions for evaluation. Or try :help.

scala> :paste
// Entering paste mode (ctrl-D to finish)

import _root_.spray.json.DefaultJsonProtocol._
import com.qvantel.jsonapi._

implicit val apiRoot: ApiRoot = ApiRoot(None)

@jsonApiResource final case class Employee(id: String, name: String)
@jsonApiResource final case class Company(id: String, name: String, employees: ToMany[Employee])

val acme = Company("1", "acme", ToMany.loaded(Seq(Employee("1", "number one 1"))))

val json = rawOne(acme)
val parsed = readOne[Company](json, Set("employees"))

acme == parsed // true

// Exiting paste mode, now interpreting.

<console>:19: error: could not find implicit value for evidence parameter of type com.qvantel.jsonapi.Identifiable[Employee]
       val acme = Company("1", "acme", ToMany.loaded(Seq(Employee("1", "number one 1"))))
                                                    ^
<console>:22: error: Cannot find JsonApiReader or JsonApiFormat type class for Company
       val parsed = readOne[Company](json, Set("employees"))


Sample splash screen for the example app

Hiya! So you think it's broken or missing a feature eh?
Welllllll, it just might be.  Do me a favor and check the following.  Then feel free to delete this comment and above.

[x] Issue does not already exist
Identify exactly what aspect your issue is about

[x] ignite base application
[ ] ignite the cli
[ ] ignite generated code

Hi @ahmed1490 - We are considering writing a lib that allows splash screens to be easy.  Whenever we do add that, we'll integrate it here :)

Great work @GantMan !! :)

relevant: https://github.com/mehcode/rn-splash-screen

is the Splash Screen up already?
more power to Ignite :+1:

Perhaps someone can wrap the library @ahmed1490 mentioned into a plugin.


[BUG]Compreface integration not work, No files found, UI rotating indefinitely

Describe the bug
Describe the bug
I'm running a Kubernetes cluster with Double take, frigate, mqtt and compreface.
Frigate capturing objects.
Double take logs show detections but they're not showing up in UI.
In config tab, Compreface is red with error:
24-03-14 21:03:13 error: 1 validation error
24-03-14 21:03:13 error: notify is not of a type(s) object

It worked a couple of months ago, didn't check until now.
Additional Logs
Example of a detection
24-03-14 21:02:26 info: Double Take v<IP_ADDRESS>
24-03-14 21:02:26 verbose: {
  telemetry: false,
  auth: true,
  token: { image: '24h' },
  detect: {
    match: {
      save: true,
      base64: false,
      confidence: 60,
      purge: 168,
      min_area: 10000
    },
    unknown: {
      save: true,
      base64: false,
      confidence: 40,
      purge: 8,
      min_area: 0
    }
  },
  time: { timezone: 'Europe/Paris' },
  frigate: {
    attempts: { latest: 10, snapshot: 10, mqtt: true, delay: 0 },
    image: { height: 500 },
    labels: [ 'person' ],
    update_sub_labels: false,
    stop_on_match: true,
    min_area: 0,
    device_tracker_timeout: 30,
    cameras: [],
    url: 'http://frigate.surveillance:5000'
  },
  mqtt: {
    protocol: 'mqtt',
    port: 1883,
    tls: {},
    topics: {
      frigate: 'frigate/events',
      matches: 'double-take/matches',
      cameras: 'double-take/cameras',
      homeassistant: 'homeassistant'
    },
    host: 'rabbitmq.surveillance',
    password: '********',
    username: 'rabbit'
  },
  logs: { level: 'debug', sql: false },
  ui: {
    path: '',
    theme: 'bootstrap4-dark-blue',
    editor: { theme: 'nord_dark' },
    logs: { lines: 5000000 },
    pagination: { limit: 50 },
    thumbnails: { quality: 95, width: 500 }
  },
  server: { port: 3000 },
  cameras: { 'front-door': null, reolink1: { detect: [Object] } },
  detectors: {
    compreface: {
      det_prob_threshold: 0.8,
      timeout: 15,
      opencv_face_required: false,
      key: '********',
      url: 'http://compreface-api.surveillance:8080'
    }
  },
  notify: null,
  schedule: {},
  storage: {
    path: './.storage',
    config: { path: './.storage/config' },
    secrets: { path: './.storage/config', extension: 'yml' },
    media: { path: './.storage' },
    tmp: { path: '/dev/shm/double-take' }
  },
  version: '<IP_ADDRESS>'
}

Versions
double take: v<IP_ADDRESS>
compreface: 1.2.0
frigate: 0.13.2
Hardware
Kubernetes Cluster on top of Rocky OS on amd64
container image used:  skrashevich/double-take:v<IP_ADDRESS>

Expected behavior
Compreface to show up green.
UI to show detections that are showing up in logs.
Screenshots

i have the same issue and I cannot see the cameras etc

Same here. Worked for over a year (docker) and suddenly no images, no errors, log shows recognising and mqtt is fine, but spinning circle...


5 :arrow_right: 4 - EnumIface: suppress deprecation warning (#540)

🦟 Bug fix
Fixes #505
Summary
Backport of https://github.com/gazebosim/gz-common/pull/540
Checklist

[X] Signed all commits for DCO
[ ] Added tests
[ ] Updated documentation (as needed)
[ ] Updated migration guide (as needed)
[ ] Consider updating Python bindings (if the library has them)
[ ] codecheck passed (See contributing)
[ ] All tests passed (See test coverage)
[ ] While waiting for a review on your PR, please help review another open pull request to support the maintainers

Note to maintainers: Remember to use Squash-Merge and edit the commit message to match the pull request summary while retaining Signed-off-by messages.

This change also required calling target_link_library on the utils library, which I've done in https://github.com/gazebosim/gz-common/commit/6652132af4f72217dfb30ac105d001a0a2be8b30 in a way that will minimize the diff when merging forward to gz-common5. Feel free to cherry-pick into this branch.
I think it is safe for our Ubuntu packages because the libignition-common4-core-dev already depends on libignition-utils1-dev (see common4 control file)

A version that doesn't use the ign-utils macros is in https://github.com/gazebosim/gz-common/pull/604. We can still merge this since the macros are better.

I think everything is ready to go. Waiting for approvals and then merge

The examples are failing on CI. This also shows that we are changing the dependency on ign-utils from being a requirement of the graphics component to being a requirement for the core library. I'm not exactly sure why the example is failing, but I'm inclined to close this PR and live with the bare pragmas that were added in #603.