Allow specifying exposed port range for portallocator

For security reasons not all ports can be opened on firewall and there is no way to define port allocation range https://github.com/docker/docker/blob/73b7cee89c8cde951012c0b0004c5faa579a0028/daemon/networkdriver/portallocator/portallocator.go#L14-L16
If will be very suitable to allow specify this settings for daemon.
P.S. Also as i see portallocator has no strategies and by default uses +1 port for any newly run container. As separate enhancement  this issue can be masked with strategy that will search always for lowest available port (though it will be slow i suppose).

Hi!
If you are reporting a new issue, PLEASE make sure that does not have any duplicates already open.
We would like to take this time to remind you of the information we need to debug the problem you are seeing. This is an automated response so if this ticket is not about a bug, do not fret.
If you fail to provide this information within 7 days, we will close this because we cannot debug your issue. We can reopen whenever the information is provided.
Thank you.
Please see:
https://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues
Description of problem:

`docker version`:

`docker info`:

`uname -a`:

Environment details (AWS, VirtualBox, physical, etc.):

How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual Results:

Expected Results:

Additional info:

#ENEEDMOREINFO

Docker uses the kernel's ephemeral port range (specified in /proc/sys/net/ipv4/ip_local_port_range), and falls back to the range you mentioned if the kernel's range is not defined.
You can use this to configure the port range used.

Configuring global kernel configuration for single process sounds weird.

There is also some further discussion on this in #12622 and #12927

Strange, why this is still open?
I totally agree with @KostyaSha, modifying kernel global limits just for docker daemon is not acceptable solution at all.
Request sounds trivial and useful (from security POV)

#12927 is definitely better than nothing. Thank you for such feature. But it is still workaround because to configure exposed ports to use specific range I have to inspect image to get list of exposed ports in image, then I have to add to that list additional ports needed in runtime. Then use modified container configuration to create container. Instead of that I would want to use publishing to random ports with feature requested in that issue.
Maintainers, will you accept such feature if someone provides a PR or you are against such changes in docker?

Strange, why this is still open?

It's a feature request, and as you can see, there's been no activity on it for over a year.

modifying kernel global limits just for docker daemon

Given the original description at the top, no software (so not just docker) should pick ports outside that range, so for the original use case, changing that value makes sense to me.

because to configure exposed ports to use specific range I have to inspect image to get list of exposed ports

Off topic; but I consider doing that "best practice" overall. Running a random image, and publish all ports without knowing what it's exposing sounds dangerous to me.

Maintainers, will you accept such feature if someone provides a PR or you are against such changes in docker?

I think it would need to be supported in libnetwork first, because that code moved there; https://github.com/docker/docker/blob/3d13fddd2bc4d679f0eaa68b0be877e5a816ad53/vendor/src/github.com/docker/libnetwork/portallocator/portallocator.go#L12-17
I'm not against having a configuration option for other use cases, but can't talk for other maintainers.

Given the original description at the top, no software (so not just docker) should pick ports outside that range, so for the original use case, changing that value makes sense to me.

Ok, right, docker random range should be inside system range, but doesn't need to be exactly that range.

Running a random image, and publish all ports without knowing what it's exposing sounds dangerous to me.

Why?
IMHO publishing all declared as exposed ports is just as dangerous as root access. At least, the ports can be seen inspecting the random image:
docker inspect -f "{{.Config.ExposedPorts}}" <random-image>


support markdown tables as data sources

A regular markdown table
| name | foo | bar |
| - | - | - |
| something | 1 | |
| something else | 2 | 123 |

that renders as

name
foo
bar

something
1

something else
2
123

can be made accessible through the api. may be something like
```dataquery
table i.name as Name, i.foo as Foo, i.bar as Bar
FROM ...
FLATTEN file.items as i
```

Was just playing with obsidian-tracker, which can work on tables, and came here to see if I could use dataview to query a table to return just a subset of the table (think a constantly growing activity log, and wanting to just show the day's activities in the Daily Note).
Looks like pulling from tables with DataView is not available yet, but this ticket gives me hope it will be someday..

Block references would also work, that is a good idea. I'll make the data available in the index (as file.tables) and see if I can make it possible to query by the block ID.

What's the status on this? I'd love to experiment with dataview in order to query my inventory database :-)

cooperate and do it with cooperation with the advanced. table plugin people?

I think Notion-like Tables must have something like this under the hood, right?

I also give this feature a +1. In my daily notes, I use markdown tables to keep track of my daily habits. I'm thinking I could have a page to parse my habit logs with the hopes of displaying some sort of "don't break the chain" / summary idea.

Hmm I wonder how difficult this is. Obsidian does parse tables so wonder if we could leverage that somehow.

Just coming by to add my +1 to this as a useful feature.
I'm writing up a table of my home network and would love to query it to pull the IP into sub-pages.

That would be a relly nice feature, +1 to this for sure!

+1. My current workaround is to use lists with inline fields. It’s more verbose syntax, but also more flexible.

Since the markdown tables got a boost with the 1.5 update, supporting the tables directly as a data source would be a very good addition by Dataview.

The update does not bring table indexing to the metadata cache which means we'd still have to parse tables by hand. Given that this plugin is in maintenance mode, it is unlikely it will be added.

+1 This would be very useful to query manual data with.

+1, IMO it sounds like a lot more useful solution than files as data sources

+1, I would use this

+1 would use it too

+1

Just started using Obsidian and dataview, I Google'd and AI'd around for a while only to find this feature doesn't exist (yet).
+1 for interest in this feature.

+1
I want this feature to have a glossary functionality.
The system is simple: a column for term, definition, example and language (2 letter abbreviation, with LX meaning latex symbols for example).
I could do that with lists and sublists, but I would prefer to do it with tables, to make it more readable even without querying.  Having hundreds if not thousands of files just isn't as clear or structured as this.
This, together with a fuzzy search would be amazing. Also, since obsidian implements fuzzy search, can't it just provide this functionality via some API?


Failure in the build task: Type '(string | RegExp)[]' is not assignable to type 'Rules'

Error reported in the latest CI job, looks the build task fails because of type mismatches with the latest webpack:
https://github.com/licg9999/transpile-webpack-plugin/actions/runs/4927111373/jobs/8803615556

Closed by: https://github.com/licg9999/transpile-webpack-plugin/pull/42


Windows server 2022 21H2 10.0.20348.1547 support missing

Hello, version  10.0.20348.1547 is missing, I'm sending termsrv.dll in attachment.
Thank you for support and help.
termsrv.zip

https://github.com/sebaxakerhtc/rdpwrap.ini/commit/9d53561df16a05362f2e3146ca83794dc6162253

Reopened. Not included in official INI (by Stass'M)

It's working here...
https://raw.githubusercontent.com/sebaxakerhtc/rdpwrap.ini/master/rdpwrap.ini

不错，很优秀😂

Hi,
it didn't work for me, its still limited to 2 concurrent users.. I replaced the termsrv.dll file in system32 and restarted.. is there anything else I should do?
by the way it's a hyper-v VM with windows server 2022 standard..
Thanks..

There is no need to replace termsrv.dll, this is not how rdp wrapper works. You have to add a new memory offset into rdpwrap.ini file for specific version of windows. The build number of termsrv.dll was changed after windows update, so the new offset is need for rdp wrapper to work.

There is no need to replace termsrv.dll, this is not how rdp wrapper works. You have to add a new memory offset into rdpwrap.ini file for specific version of windows. The build number of termsrv.dll was changed after windows update, so the new offset is need for rdp wrapper to work.

sorry can u be more specific, what do we need to change? or just wait for new ini?

You can edit your's just add missing sections into ini.
rdpwrap.zip

You can edit your's just add missing sections into ini. rdpwrap.zip

It is working now on Win Server 2022, Tnx for reply. I did edit ini file, but it is seams with wrong one xD


fix(cors): avoid setting Access-Control-Allow-Origin if there is no matching origin

The author should do the following, if applicable

[x] Add tests
[x] Run tests
[x] bun run format:fix && bun run lint:fix to format the code
[ ] Add TSDoc/JSDoc to document the code

Even if the Origin header of a request does not match a origin specified by origin option, CORS middleware always sets Access-Control-Allow-Origin header. I don't see this as a serious problem, but in some cases it may unintentionally expose a server setting.
References

https://github.com/expressjs/cors/blob/v2.8.5/test/test.js#L251-L262 - cors package seems not to set the Access-Control-Allow-Origin if the Origin header of a request does not match allowed origins
https://fetch.spec.whatwg.org/#cors-check - A CORS check returns failure if the Access-Control-Allow-Origin header is missing

Currently, CORS middleware returns a origin specified by origin option as the Access-Control-Allow-Origin if the option does not match the Origin header of a request, so changes in this PR would have no particular negative impact

Hi @uki00a
Thanks! I've left one comment. Please check it!

@uki00a
Looks good! I'll merge it and release the new version soon (maybe tomorrow). Thank you for your contribution!


Evaluate GitHub actions for pipeline builds

Industrial CI also runs on GitHub actions. Main reason to consider switching: Travis-CI does not support skipping builds if only certain files or directories have changed (travis-ci/travis-ci#6301). This would be quite relevant for the documentation folder - it makes no sense to trigger a build when updating it.

I fully support this. I've read the same about skipping builds.

PR into master, great! :) Can we consider this as closed? All builds are migrated? Or are there any left out?

It's only done for this repo, and it's still sitting on prepare-for-open-collab branch. When we have an initial somewhat-ready state, we can merge to master imo.


Skips the linking Spotify step on registration

when going through the registration process following the steps. I received the email for verification of the account, then clicked the "here" link which then brought me to the home page without linking Spotify.

fixed kind of lets not talk about it


Enhancement: Add Filter Condition to "Open / Close" in Relief Camps

Subject of the issue
Add Relief Camps Status as Filter Condition to Identify - Open or Closed camps
https://www.keralarescue.in/relief_camps_list
Your environment

version of your python
which browser and its version
OS Type (iOS/Android/Linux/Windows)

Steps to reproduce
Go to https://www.keralarescue.in/relief_camps_list
Expected behavior
Users should be able to filter Open and Closed Relief  Camps
Actual behaviour
Currently functionality is not available to filter Open/Closed Relief camps.

It is already there in back emd


Differ the assertion checks

Prepare one test suite for Semver 1.0
And one test suite for Semver 2.0

Won't be solved / too late.


Adding visual cue for "Limited" & "Private" organizations.

Implements #13017 .
Label for "Limited" & "Private" organizations added to org header, org homepage, and navbar.
Before:

After:

Codecov Report

Merging #13040 into master will increase coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #13040      +/-   ##
==========================================
+ Coverage   42.57%   42.59%   +0.02%
==========================================
  Files         672      672
  Lines       73698    73785      +87
==========================================
+ Hits        31374    31432      +58
- Misses      37241    37259      +18
- Partials     5083     5094      +11

Impacted Files
Coverage Δ

modules/secret/secret.go
74.07% <0.00%> (-9.26%)
:arrow_down:

modules/setting/task.go
72.72% <0.00%> (-7.28%)
:arrow_down:

modules/setting/queue.go
71.42% <0.00%> (-4.77%)
:arrow_down:

routers/repo/attachment.go
42.85% <0.00%> (-3.38%)
:arrow_down:

services/pull/check.go
44.61% <0.00%> (-3.08%)
:arrow_down:

models/unit.go
46.57% <0.00%> (-2.74%)
:arrow_down:

modules/context/auth.go
18.47% <0.00%> (-2.58%)
:arrow_down:

modules/indexer/issues/bleve.go
69.28% <0.00%> (-1.43%)
:arrow_down:

models/issue_label.go
67.15% <0.00%> (-0.76%)
:arrow_down:

models/issue.go
56.58% <0.00%> (-0.07%)
:arrow_down:

... and 14 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 72636fd...d7e30c9. Read the comment docs.

@6543 Moved styles to .less files in commit 17fbba7 . Re: repository "Limited" labels -- do we want to find some way to visually show that the "Limited" attribute/label is inherited from the org (opposed to the "Private" attribute, which is set at the repository level & also creates a label next to the repo name)?

🚀

@6543 I think the 2nd one can be saved for later PR


clearHandlers not supported in models

hi,
i am trying to clear handlers in models but unable to do so.
when i checked the module files then found it's not bind in models
**this line should be added after line 190 in model.js**
this.$clearHandlers = this.$queryInterface.$clearHandlers.bind(this.$queryInterface);

created PR for this issue
https://github.com/BlinkUX/sequelize-mock/pull/59


Warn user about malformed email addresses

When we get malformed email addresses from teamsTeamAddEmailsBulkRpcPromise, throw a black bar with the culprits:

@keybase/react-hackers

wait

we dont' want to throw black bars to handle errors. we should show a real error in red on the screen

Talked over in meeting: this is alright in the short short term, but black bars should not be used to present things like user input errors - they should be incorporated into the designs.


MPI version of 3D model

MPI-parallel version of the 3D model.

Yes, just merging the latest parallel code + version update into main. Thanks!


Issue with Import

Exported all data.
Ran Import with generated token that has work access as well as tried with full access and am receiving the following error.
To Reproduce
Steps to reproduce the behavior:

create export
run import with token that has proper access or full access on work items
Error received
See error

Tool version

3.0.109

Please attach the following files:
ork Item Importer v3.0.109
Copyright (c) Solidify
[I][15:55:09] Connecting to Azure DevOps/TFS...
[E][15:55:10] Unexpected migration error.
[Microsoft.VisualStudio.Services.WebApi.VssServiceResponseException] Microsoft.VisualStudio.Services.WebApi.VssServiceResponseException: Page not found.
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.d__53.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.d__51.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.d__471.MoveNext() --- End of stack trace from previous location where exception was thrown ---    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at Microsoft.VisualStudio.Services.Location.Client.LocationHttpClient.<GetConnectionDataAsync>d__6.MoveNext() --- End of stack trace from previous location where exception was thrown ---    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.<GetConnectionDataAsync>d__56.MoveNext() --- End of stack trace from previous location where exception was thrown ---    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.<ConnectAsync>d__41.MoveNext() --- End of stack trace from previous location where exception was thrown ---    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.<EnsureConnectedAsync>d__39.MoveNext() --- End of stack trace from previous location where exception was thrown ---    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.<GetInstanceIdAsync>d__11.MoveNext() --- End of stack trace from previous location where exception was thrown ---    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at Microsoft.VisualStudio.Services.WebApi.Location.LocationService.<GetLocationDataAsync>d__2.MoveNext() --- End of stack trace from previous location where exception was thrown ---    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at Microsoft.VisualStudio.Services.WebApi.VssConnection.<GetClientInstanceAsync>d__20.MoveNext() --- End of stack trace from previous location where exception was thrown ---    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at Microsoft.VisualStudio.Services.WebApi.VssConnection.<GetClientServiceImplAsync>d__17.MoveNext() --- End of stack trace from previous location where exception was thrown ---    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at Microsoft.VisualStudio.Services.WebApi.VssConnection.<GetClientAsync>d__141.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Services.WebApi.TaskExtensions.SyncResult[T](Task1 task)    at Microsoft.VisualStudio.Services.WebApi.VssConnection.GetClient[T]()    at WorkItemImport.WitClientWrapper..ctor(String collectionUri, String project, String personalAccessToken)    at WorkItemImport.Agent.Initialize(MigrationContext context, Settings settings)    at WorkItemImport.ImportCommandLine.ExecuteMigration(CommandOption token, CommandOption url, CommandOption configFile, Boolean forceFresh, CommandOption continueOnCritical):    at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<HandleResponseAsync>d__53.MoveNext() --- End of stack trace from previous location where exception was thrown ---    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__51.MoveNext() --- End of stack trace from previous location where exception was thrown ---    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__471.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Services.Location.Client.LocationHttpClient.d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.d__56.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.d__41.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.d__39.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Services.WebApi.Location.LocationService.d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Services.WebApi.VssConnection.d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Services.WebApi.VssConnection.d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Services.WebApi.VssConnection.d__141.MoveNext() --- End of stack trace from previous location where exception was thrown ---    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at Microsoft.VisualStudio.Services.WebApi.TaskExtensions.SyncResult[T](Task1 task)
at Microsoft.VisualStudio.Services.WebApi.VssConnection.GetClientT
at WorkItemImport.WitClientWrapper..ctor(String collectionUri, String project, String personalAccessToken)
at WorkItemImport.Agent.Initialize(MigrationContext context, Settings settings)
at WorkItemImport.ImportCommandLine.ExecuteMigration(CommandOption token, CommandOption url, CommandOption configFile, Boolean forceFresh, CommandOption continueOnCritical)
[I][15:55:10] Import complete. Imported 0 items, 0 revisions (1 errors, 0 warnings) in 00:00:01.

Hi @GBRacerX1 - did you end up finding what caused the "Page not found" Exception? I am getting it too...


AzureStack Gallery Admin

Initial commit for the AzureStack Gallery admin specs.
PR information

[X] The title of the PR is clear and informative.
[X] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For information on cleaning up the commits in your pull request, see this page.
[X] Except for special cases involving multiple contributors, the PR is started from a fork of the main repository, not a branch.
[X] If applicable, the PR references the bug/issue that it fixes.
[X] Swagger files are correctly named (e.g. the api-version in the path should match the api-version in the spec).

Quality of Swagger

[X] I have read the contribution guidelines.
[X] My spec meets the review criteria:

[X] The spec conforms to the Swagger 2.0 specification.
[X] The spec follows the guidelines described in the Swagger checklist document.
[X] Validation tools were run on swagger spec(s) and have all been fixed in this PR.

Automation for azure-sdk-for-go

Automation for azure-libraries-for-java

Automation for azure-sdk-for-python

Automation for azure-sdk-for-node

@lmazuel Hmm, looks like it does not like objects with no properties.  Not really sure what to do with this.

@lmazuel Ping.


RP2040 SPI asserts equal length for transfer

It seems that the current implementation of Spi requires the read and write to be of equal lengths:
https://github.com/embassy-rs/embassy/blob/7e251a25509a02f9388a8352522e1a279ad857b1/embassy-rp/src/spi.rs#L385-L388
However, in embedded_hal_async:0.2.0-alpha.0 it is stated that:

It is allowed for read and write to have different lengths, even zero length

How should we fix this misalignment?

Yes, this should be allowed in embassy-rp.

So we should just remove the assertion?

No, it's a bit more complicated. I think:
If read is longer, we'd need to write a bunch of dummy zeros to get the SPI to clock more, to do the entire read.
If write is longer , it'd need to either do the extra dummy reads, or (preferably) don't do them and let the rx fifo overflow, and ignore whatever "overflow" error happens, and cleanup the garbage left in the fifo for the next transfer.

@Dirbaio would you mind giving me a few more points to guide me how to implement this?
I see that all transfer methods are using this transfer_inner method which I should adjust accordingly to what you explained in your comment:
https://github.com/embassy-rs/embassy/blob/b6663a013f8632cefbc6d6ab9a9aa7afbcd1314b/embassy-rp/src/spi.rs#L395-L406

let the rx fifo overflow, and ignore whatever "overflow" error happens, and cleanup the garbage left in the fifo for the next transfer
How would I allow the FIFO the overflow?

it seems that dma::read will only read to the read buffer length:
https://github.com/embassy-rs/embassy/blob/b6663a013f8632cefbc6d6ab9a9aa7afbcd1314b/embassy-rp/src/dma.rs#L46

it seems that dma::read will only read to the read buffer length:

yes, but if you write bytes to the SPI without reading, the SPI peripheral keeps clocking bits I think. At least it does in STM32. This is what I meant by "let the RX fifo overflow"

So if TX > RX length we can leave the RX overflow by not reading more bytes from the DMA and it differs from the first solution where "it'd need to either do the extra dummy reads". In a nutshell, I don't need to do anything in this case?
And what about RX > TX? I cannot extend the tx buffer as it's a *const correct?
But I also can't create a new tx buffer like e.g. let mut tx = [0_u8; to_len]; because to_len is not a const value.
How would I write dummy bytes in this case?

So if TX > RX length we can leave the RX overflow by not reading more bytes from the DMA and it differs from the first solution where "it'd need to either do the extra dummy reads". In a nutshell, I don't need to do anything in this case?

There's 2 ways:

Read the wanted bytes, then read dummy bytes.
Read the wanted bytes, let the rest overflow.

The 1st will work for sure. I think the 2nd one should work as well, and if it does it's the better way, because it'll be faster (less work). It might be a bit tricky because maybe it might leave unread data inthe SPI FIFO which can corrupt the next read afterwards, perhaps you have to clear it.

But I also can't create a new tx buffer like e.g. let mut tx = [0_u8; to_len]; because to_len is not a const value.
How would I write dummy bytes in this case?

you can read/write dummy bytes by setting incr_read=false, incr_write=false. https://dirbaio.github.io/rp2040-pac/rp2040_pac2/dma/regs/struct.CtrlTrig.html#method.incr_write and then pointing it to a 1-byte buffer. This way DMA will repeatedly read/write from the same memory location, instead of requiring an N-bytes buffer.

There's 2 ways:
* Read the wanted bytes, then read dummy bytes.

* Read the wanted bytes, let the rest overflow.

The 1st will work for sure. I think the 2nd one should work as well, and if it does it's the better way, because it'll be faster (less work). It might be a bit tricky because maybe it might leave unread data inthe SPI FIFO which can corrupt the next read afterwards, perhaps you have to clear it.

What do you mean by clearing?
Do you mean that I should just move the read pointer or re-write the data that was just written to it?

you can read/write dummy bytes by setting incr_read=false, incr_write=false. https://dirbaio.github.io/rp2040-pac/rp2040_pac2/dma/regs/struct.CtrlTrig.html#method.incr_write and then pointing it to a 1-byte buffer. This way DMA will repeatedly read/write from the same memory location, instead of requiring an N-bytes buffer.

If I'm doing read or write I need to set both to incr_read=false, incr_write=false or incr_read=false for read and incr_write=false for write?
I see that there is write_repeated function in the DMA module but it sets both to false.
https://github.com/embassy-rs/embassy/blob/b6663a013f/embassy-rp/src/dma.rs#L78-L95

What do you mean by clearing?
Do you mean that I should just move the read pointer or re-write the data that was just written to it?

the SPI peripheral has a data FIFO. DMA moves bytes from the FIFO to RAM. If you let the FIFO overflow, most of the bytes will be lost but a few will be lingering in the FIFO when the transfer is done. When you do another read, the first bytes popped from the FIFO will be these stale bytes, instead of the current ones.
write currently does the same (let the FIFO overflow), so it clears it after it's done, here: https://github.com/embassy-rs/embassy/blob/master/embassy-rp/src/spi.rs#L343-L348

If I'm doing read or write I need to set both to incr_read=false, incr_write=false or incr_read=false for read and incr_write=false for write?
I see that there is write_repeated function in the DMA module but it sets both to false.

normal write transfers (read mem -> write reg) do incr_read=true because you want to send several bytes from RAM from teh buffer, and incr_write=false because you want to write them all to the same reg.
normal read transfers (read reg -> write mem) do incr_read=false because you want to read all from the same reg, and incr_write=true because you want to write them all to the buffer one after each other
these "dummy" transfers have both to false, because you want to repeatedly read/write the same byte in mem instead.

Edit: Only problem with write_repeated I can see is that it always sets the read pointer to 0?!

it sets a pointer pointing to the dummy variable, that contains a zero. It'll write N zeros.


ServiceWorker is not being throttled

Tell us about your environment:

Puppeteer version: v1.14.0 (also tested with v.1.12.0)
Platform / OS version: OSX Sierra (also tested with Win10)
URLs (if applicable): https://kindaninja.dk/sw-throttle-test/
Node.js version: v11.12.0

What steps will reproduce the problem?
test.js
client = await puppeteer.launch({
    headless: false
});

let page = await client.newPage();

let cdp = await page.target().createCDPSession();

await cdp.send('Network.emulateNetworkConditions', {
    'offline': false,
    'downloadThroughput': 1 * 1024,
    'uploadThroughput': 1 * 1024,
    'latency': 0
});

await page.goto('https://kindaninja.dk/sw-throttle-test/');

The following test site is available here: https://kindaninja.dk/sw-throttle-test/
index.html
<script>
    navigator.serviceWorker.register('sw.js').then(function (reg) {
        console.log("worker_registration_succeeded");
    }).catch(function (err) {
        console.error("worker_registration_failed", err);
    });

    (async function() {
        var start = Date.now()
        let img1 = await fetch('https://kindaninja.dk/sw-throttle-test/1001.png')
        console.log("Fetch from page:", Date.now()-start, "ms")

        start = Date.now()
        let img2 = await fetch('https://kindaninja.dk/sw-throttle-test/1002.png')
        console.log("Fetch from sw:", Date.now()-start, "ms")
    })();
</script>

sw.js
self.addEventListener('install', event =>  {
    console.log("worker_install");
    event.waitUntil(self.skipWaiting());
});

self.addEventListener('activate', event => {
    console.log("worker_activate");
    event.waitUntil(self.clients.claim());
});

self.addEventListener('fetch', function(event) {
    // Only fetch second image from sw
    if(event.request.url.includes('1002')) {
        console.log("intercepted_request", event.request.url);
        event.respondWith(async function() {
            const response = await fetch(event.request)
            const blob = await response.blob()
            return new Response(blob, {status: 200, statusText: "OK"});
        }());
    }
    // Every thing else passes through
    else {
        console.log("passed_through", event.request.url);
        event.respondWith(fetch(event.request));
    }
});

What is the expected result?
That all requests are being throttled. When visiting the website in Chrome (without puppeteer), and throttling to Slow 3G in DevTools, the log prints the expected result:
Fetch from page: 2014 ms
Fetch from sw: 2021 ms

What happens instead?
Only requests from the web page is being throttled. The request from the service worker is not. From the console in Chromium:
Fetch from page: 1421 ms
Fetch from sw: 38 ms

What else?
I have tried using Puppeteer with Chrome instead of Chromium:
executablePath: '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome'

I have tried targeting the service worker directly with:
client.on('targetcreated', async (target) =>{
            if(target.type() === 'service_worker'){
                const cdp = await target.createCDPSession();
                console.log(cdp);
                await cdp.send('Network.emulateNetworkConditions', {
                    'offline': false,
                    'downloadThroughput': internetSpeed * 1024,
                    'uploadThroughput': internetSpeed * 1024,
                    'latency': 0
                });
            }
        })

It finds the worker, but throttling is not applied.
I'm trying to do a performance evaluation for my bachelor thesis, so any help will be greatly appreciated :)

@kindaninja service worker is a separate target in DevTools protocol, so you'd need to connect to it separately. Some time ago I did a demo on how to throttle SW with Puppeteer - here's the gist: https://gist.github.com/aslushnikov/d03eada6d12eedbd1b0a49d67f0b1d5b
Note: DevTools protocol network shaping is far from ideal, so I wouldn't expect too much of it.
Check out this Network Throttling & Chrome - status for details.
Copying the SW Throttling Demo here to save you some clicks.
const puppeteer = require('puppeteer');

(async() => {
  // Step 1: launch browser and open a new page.
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  // Step 2: PPTR.DEV has a service worker \m/
  // Go there and wait for SW to register.
  await page.goto('https://pptr.dev/');
  const swTarget = await browser.waitForTarget(target => target.type() === 'service_worker');

  // Step 3: Create a Chrome DevTools Protocol session to talk to service worker.
  // Make sure to enable Runtime agent so that we can do evaluations inside SW.
  const session = await swTarget.createCDPSession();
  await session.send('Runtime.enable');

  // Step 4: Fetch a dummy file from-inside SW and see how long it takes.
  console.log(`Fetching WITHOUT network throttling: ${await measureAPIFetch(session)}ms`);

  // Step 5: Enable network throttling.
  // Exact numbers for this throttling are taken from Chrome DevTools front-end:
  // https://github.com/chromium/chromium/blob/24c2704773b6d26f3fa65985dc54fb49eb449a6b/third_party/blink/renderer/devtools/front_end/sdk/NetworkManager.js#L247-L252
  await session.send('Network.enable');
  await session.send('Network.emulateNetworkConditions', {
    offline: false,
    latency: 400 * 5,
    downloadThroughput: 500 * 1024 / 8 * .8,
    uploadThroughput: 500 * 1024 / 8 * .8,
  });

  // Step 6: Fetch a dummy file from-inside SW and see how it long it takes now.
  console.log(`Fetching WITH network throttling: ${await measureAPIFetch(session)}ms`);

  // Step 7: Done. Close.
  await browser.close();
})();

async function measureAPIFetch(session) {
  return (await session.send('Runtime.evaluate', {
    awaitPromise: true,
    expression: `
      (async function() {
        const start = Date.now();
        const response = await fetch('https://raw.githubusercontent.com/GoogleChrome/puppeteer/master/docs/api.md', {cache: "no-cache"});
        const body = await response.text();
        const end = Date.now();
        return end - start;
      })();
    `,
  })).result.value;
}

Hope this helps!


Proper way to run scrapyd

There is a problem, in which I cant run scrapyd using the docs provided here.
(I added necessary config files to scrapy.cfg)
Using latest version.
It just cant deploy.

Post in the questions forum as this likely is a user error.


returns error body if it is impossible to parse JSON

Hi. I'm writing wrapper for YandexKassa API and realized that sometimes I get JSON::ParserError for failed requests. It happens because API responds with HTML page. So, in this case it is impossible to parse JSON from HTML and I guess it would be nice if we just use raw response body.

Thank you, we are also faced with this problem. I'll release fix in 1.0.1


[jax2tf] Remove legacy bfloat16 conversion code.

Now that tf.bfloat16 and jnp.bfloat16 are the same, there should not be a need for specific conversion code anymore between the two.

@googlebot I signed it!

@googlebot I signed it!

I am proposing that we hold off on merging this until jaxlib minimum version needs to be upgraded for other reasons as well.

I am proposing that we hold off on merging this until jaxlib minimum version needs to be upgraded for other reasons as well.

Oops, I meant to answer sooner! Sounds good to me, thanks :+1:

Oops, I meant to answer sooner! Sounds good to me, thanks :+1:


Redraw when an add- or subtract-button click is handled

Prior to https://github.com/droplet-editor/droplet/pull/150 the redraw was triggered indirectly via mousedown -> clearLassoSelection() -> redrawHighlights() -> redrawMain().
This fixes the bug where pressing the '+' button on an if block appears to have no effect until something else happens to trigger a redrawMain().

LGTM


How to generate job-archive file

I want to know, how to generate the job-archive files like the demo. I tried to use cc-metric-collector as information gathering, but it seems can ‘t generate a job-archive file, so it can't display by cc-backend.

Hi,
to setup the job archive is part of the configuration. You need to create a directory for each cluster and setup the cluster.jsob file within it to describe the processor and metrics you want to see. Did you already do the hands-on in the source tree ./docs directory?
I am sorry for the currently poor documentation of the overall setup. The basic setting is: the cc-metric-collector will run as node-agent on every node and collect metrics. Those metrics are then send to the cc-metric-store. There has to be an integration with your job scheduler to get the job meta data. This is implemented using a REST API provided by cc-backend. You can play with the REST API using a Swagger UI if you start cc-backend in dev mode. cc-slurm-sync is just an example how this could be done. The main problem is that the resources a job uses (extracted from your job scheduler) must match those used in cc-metric-collector/cc-metric-store. This is simple as long as you only have node lists, but becomes cumbersome as soon as you have shared jobs or jobs using GPUs. In the future we plan to integrate the support for slurm (using its REST API) into cc-backend. Till then you have to write a script that provides the job meta data. I hope this helps.

Thank you for answering my question. Can I setup the whole ClusterCokcpit like this? In every cluster node, install cc-metric-collector to collect metrics; Install cc-metric-store and cc-backend on a separate node to receive the metrics.
When the metrics stored on cc-metric-store/var/checkpoints/, I have to use my own script to collect and generate the matched job meta data.  Is it right?

Yes that is correct. This piece of documentation for the big picture is still missing :-(.
You have to extract the job meta data and communicate it to cc-backend using the Job REST API.
In the long term we plan to integrate a Slurm Job Scheduler adapter in cc-backend. Getting all the required information out of Slurm is not trivial, especially with node sharing. There is an example script in https://github.com/ClusterCockpit/cc-slurm-sync. Maybe that helps.


Update expected Go version to 1.20

$ make all
go vet github.com/chronosphereio/go-grpc-prometheus github.com/chronosphereio/go-grpc-prometheus/examples/grpc-server-with-prometheus/client github.com/chronosphereio/go-grpc-prometheus/examples/grpc-server-with-prometheus/protobuf github.com/chronosphereio/go-grpc-prometheus/examples/grpc-server-with-prometheus/server github.com/chronosphereio/go-grpc-prometheus/examples/testproto
go fmt github.com/chronosphereio/go-grpc-prometheus github.com/chronosphereio/go-grpc-prometheus/examples/grpc-server-with-prometheus/client github.com/chronosphereio/go-grpc-prometheus/examples/grpc-server-with-prometheus/protobuf github.com/chronosphereio/go-grpc-prometheus/examples/grpc-server-with-prometheus/server github.com/chronosphereio/go-grpc-prometheus/examples/testproto
./scripts/test_all.sh
TESTS FOR: for github.com/chronosphereio/go-grpc-prometheus
=== RUN   TestClientInterceptorSuite
=== RUN   TestClientInterceptorSuite/TestStartedStreamingIncrementsStarted
=== RUN   TestClientInterceptorSuite/TestStreamingIncrementsMetrics
=== RUN   TestClientInterceptorSuite/TestUnaryIncrementsMetrics
=== NAME  TestClientInterceptorSuite
    client_test.go:92: stopped grpc.Server at: <IP_ADDRESS>:60233
--- PASS: TestClientInterceptorSuite (0.01s)
    --- PASS: TestClientInterceptorSuite/TestStartedStreamingIncrementsStarted (0.00s)
    --- PASS: TestClientInterceptorSuite/TestStreamingIncrementsMetrics (0.00s)
    --- PASS: TestClientInterceptorSuite/TestUnaryIncrementsMetrics (0.00s)
=== RUN   TestServerInterceptorSuite
=== RUN   TestServerInterceptorSuite/TestRegisterPresetsStuff
=== RUN   TestServerInterceptorSuite/TestStartedStreamingIncrementsStarted
=== RUN   TestServerInterceptorSuite/TestStreamingIncrementsMetrics
=== RUN   TestServerInterceptorSuite/TestUnaryIncrementsMetrics
=== NAME  TestServerInterceptorSuite
    server_test.go:104: stopped grpc.Server at: <IP_ADDRESS>:60235
--- PASS: TestServerInterceptorSuite (0.37s)
    --- PASS: TestServerInterceptorSuite/TestRegisterPresetsStuff (0.06s)
    --- PASS: TestServerInterceptorSuite/TestStartedStreamingIncrementsStarted (0.20s)
    --- PASS: TestServerInterceptorSuite/TestStreamingIncrementsMetrics (0.11s)
    --- PASS: TestServerInterceptorSuite/TestUnaryIncrementsMetrics (0.00s)
PASS
        github.com/chronosphereio/go-grpc-prometheus    coverage: 65.9% of statements
ok      github.com/chronosphereio/go-grpc-prometheus    0.561s  coverage: 65.9% of statements

TESTS FOR: for github.com/chronosphereio/go-grpc-prometheus/examples/grpc-server-with-prometheus/client
?       github.com/chronosphereio/go-grpc-prometheus/examples/grpc-server-with-prometheus/client        [no test files]

TESTS FOR: for github.com/chronosphereio/go-grpc-prometheus/examples/grpc-server-with-prometheus/protobuf
?       github.com/chronosphereio/go-grpc-prometheus/examples/grpc-server-with-prometheus/protobuf      [no test files]

TESTS FOR: for github.com/chronosphereio/go-grpc-prometheus/examples/grpc-server-with-prometheus/server
?       github.com/chronosphereio/go-grpc-prometheus/examples/grpc-server-with-prometheus/server        [no test files]

TESTS FOR: for github.com/chronosphereio/go-grpc-prometheus/examples/testproto
?       github.com/chronosphereio/go-grpc-prometheus/examples/testproto [no test files]

Closing in favor of #4.


Issue with AzCopy sync wrt file sizes

Which version of the AzCopy was used? - AzCopy 10.18.1
Note: The version is visible when running AzCopy without any argument
Which platform are you using? (ex: Windows, Mac, Linux) - Windows
What command did you run? - azcopy sync
What problem was encountered? - Even though AZCopy claims that it does not copy files which are currently being written by other tools to target storages, we have seen multiple instances where the tool creates 4 kb ( default size ) files on target side (blob) with a different timestamp than the actual file commit time on source side. This results in the tool skipping those files to sync and results in failed export consistency as there is a difference in file size on source and target side.
We are unable to find a flag which compared the last modified time or the size of the files from source to target so that the sync happens accurately.
Have you found a mitigation/solution? - We manually sync files after a certain period of time using mirror-mode argument to overwrite the wrongly synced files with the files of their true size.

Hi Shubham,
AzCopy cannot modify or set the timestamps on blob storage.  Blob storate does not allow this. The timestamps is the time at which blob was committed.
Also, AzCopy does not claim that we do not copy blobs which are currently being writen by other tools. Again, blob storage does not allow to identify this.
If you are performing sync, and if a file is marked for transfer, we will fire PutBlob or PutBlock calls appropriately. If the source changed, after the copy was done, it would be copied again. AzCopy does not create placeholder blobs.
Can you let us know your usecase a bit more? Is the source being actively changed?

Hi nakulkar
I agree that azcopy should not set timestamps. I am saying that azcopy in SAP export scenarios, does not compare the last modified time stamp or the size of file on source wrt target to upload the files. This results in us getting partial files on target size which corrupts our import process

Hi, @shubhamsharma156!
In our last release, we added a flag to AzCopy Sync, namely --compare-hash. This flag instead of comparing LMTs opts to compare the hashes, and if they are mismatched, replaces. This probably could resolve your scenario.
Regards,
Adele

Hi @shubhamsharma156.  Did the --compare-hash flag resolve your issues?  We are experiencing exactly the same issues as you.  However, when I tested the --compare-hash flag, I found it didn't sync ANY source files to the destination, if say just one source file was locked.  In such a case it doesn't copy any source files at all, even those which are unlocked.  I've logged a new issue for this.

Oh! Admittedly, I think both me and my colleague misunderstood the scenario. Disregard the --compare-hash response.
We advise users to not be touching the source or destination while AzCopy is running, but I imagine some filtering could be done on files which have an open lock. @markmiddleton908 I'm going to close your other issue and include this in the discussion for next release.

Hi @adreed-msft
Could you please let us know as well if there is any ETA on this as we are running critical loads as of right now and a very crucial go live is upcoming in a few days.
Thanks a lot in advance
Regards
Shubham

Oh! Admittedly, I think both me and my colleague misunderstood the scenario. Disregard the --compare-hash response.
We advise users to not be touching the source or destination while AzCopy is running, but I imagine some filtering could be done on files which have an open lock. @markmiddleton908 I'm going to close your other issue (since they're the same issue under the hood) and include this in the discussion for next release.

Hi @adreed-msft
Are they the same issue?  This one concerns uploaded file sizes when running a sync and the recommendation was to run with --compare-hash to resolve that.  So in testing Adele's recommendation and using --compare-hash, I found the other issue below which is:
When running with --compare-hash, it doesn't sync ANY files if one of the source files is locked.  It aborts the whole sync.  This differs to current sync functionality without --compare-hash which will sync all files without a lock and merely ignore the locked ones.  So --compare-hash is breaking current functionality and that was my other issue.

Hi, all. I've spoken with the team, and we've decided to not fix this. Our advisory has always been that customers should not mutate any files within the source for the duration of the transfer.


Fargate support

Quite a few teams are using ECS Fargate in SST using CDK constructs, and are interested in native SST support.
Use cases
Is it fair to classify Fargate use cases into: "services" and "tasks".

A. "services" are long lived, and usually used with ALBs. ie, running web servers;
B. "tasks" are short lived, they perform a task and then ends.

What can SST simplify

Configuring networking: in use case A, configuring VPC and ALB to expose a port to the internet.
Configuring event triggers: in use case B, configuring the Fargate task invoked by an SNS Topic, SQS Queue

Interesting ideas

In use case A, optionally fallback to Lambda when Fargate is under load and cannot scale up fast enough

Please share your use case + how SST help you.
^ Join the discussion on Discord — https://discordapp.com/channels/983865673656705025/1024752575549022288

I think the following things would be valuable for integration.
My use case is that I use Fargate to run a Next.js 13 app. I think this will become the primary way that users run Next.js on AWS, since the current NextjsSite construct lacks even full Next.js 12 support, due to it's dependency on @sls-next/lambda-at-edge. With Next.js 13 leaning further towards SSR, I think a truly "serverless" approach will fail to support all the new features, leading to Fargate becoming a more popular approach.
Resource binding
The most useful thing for me would be having Resource Binding available.
I should be able to do container.bind([bucket]) to allow an ECS container access to my bucket. Then, I can access the name of the bucket using the @serverless-stack/node/bucket module.
ECS containers do support environment variables, so I imagine it would work similarly to Lambda does now.
esbuild integration
The Function construct (and all the other constructs which extend it), wraps esbuild to bundle the source files. The user simply points at their handler files, and they're good to go.
At the moment with ECS, users need to write a Dockerfile. They need to run their build script prior to running CDK, or run build scripts inside the Dockerfile as part of building the image.
It would be great to have this a little more magic. Like SST knows that you're trying to run a node app, so you just point at the entrypoint source file, and good to go, without writing up a Dockerfile.

@dangreaves this is great. It's completely along the lines of what we've been thinking about recently.

Hey @dangreaves, I'm beginning to explore Fargate. Yeah we should definitely support resource binding.
I'd love to take a glimpse at ur current setup, and have a few questions around the ideal sst dev experience u imagine. Would love to jump on quick zoom if u r up for it.
^ lemme throw my calendar here https://calendly.com/frank-w/30min; I'm also good if other times work better (think u r 14hr ahead of me :)

Shout out to everyone who liked this thread, would love to learn about ur current setup and get ur help prototyping on our first container construct.
Feel free to grab a time here — https://calendly.com/frank-w/30min, or shoot me an email at<EMAIL_ADDRESS>with some times that work better!
/cc @lukeshay, @mikestopcontinues, @notgiorgi, @joyfulelement, @pjvds, @MakChan, @penairdes, @NicholasFields

Added Service construct in v2.23.0. See examples here
Huge thanks to everyone here 🫡


Google map not getting displayed on device back button

I'm submitting a ...
[ ] question
[x] any problem or bug report
[ ] feature request
The plugin version:
[ ] 1.4.x
[x] 2.0.0-beta3
If you choose 'problem or bug report', please select OS:
[x] Android
[ ] iOS
cordova information:  (run $> cordova plugin list)
com.googlemaps.ios 2.3.0 "Google Maps SDK for iOS"
cordova-plugin-android-permissions 1.0.0 "Permissions"
cordova-plugin-compat 1.0.0 "Compat"
cordova-plugin-console 1.0.5 "Console"
cordova-plugin-device 1.1.4 "Device"
cordova-plugin-file 4.3.3 "File"
cordova-plugin-file-transfer 1.6.3 "File Transfer"
cordova-plugin-googlemaps 2.0.0-beta3-20170820-2025 "cordova-plugin-googlemaps"
cordova-plugin-media 3.0.1 "Media"
cordova-plugin-network-information 1.3.3 "Network Information"
cordova-plugin-splashscreen 4.0.3 "Splashscreen"
cordova-plugin-statusbar 2.2.2 "StatusBar"
cordova-plugin-whitelist 1.3.1 "Whitelist"
cordova-sqlite-storage 2.0.4 "Cordova sqlite storage plugin"
cordova.plugins.diagnostic 3.6.6 "Diagnostic"
ionic-plugin-keyboard 2.2.1 "Keyboard"
cli packages:
@ionic/cli-utils  : 1.9.2
ionic (Ionic CLI) : 3.9.2
global packages:
Cordova CLI : 7.0.1
local packages:
@ionic/app-scripts : 2.1.3
Cordova Platforms  : android 6.2.3 ios 4.4.0
Ionic Framework    : ionic-angular 3.6.0
I have created a tab on sidebar view to show google map. All is working fine if I navigate from ionic back button from header bar but if i redirect back using android device back button map will get disappear and it shows me a blank screen.
Please help me to solve this issue.
Thanks.

The question was asked and here, but could not help:
https://forum.ionicframework.com/t/google-map-not-getting-displayed-on-device-back-button/102407/2

Before asking, please try the latest version.
The cordova-plugin-googlemaps 2.0.0-beta3-20170820-2025 is too old.

And I appreciate if you write your comment with correct syntax of Github markdown.

In last version bug fixed. Thanks.


Brackets do not group strings which contain spaces and are chunked instead

Great project!
I've had one issue though. For a program i wrote i had a string which contained bracket, which raised a unrecognized parameter exception. something like :
application command "some (text)"
using a signature like
void Command(string name)
Is this a bug, or is it just incorrect/unexpected syntax?

I will try to reprpduce this

@bilal-fazlani thanks. You can use the following code to reproduce it:
public class StringValueParserTests : TestBase
{
    [Theory]
    [InlineData("test config sometext", 0)]
    [InlineData("test config \"sometext (and more)\"", 0)]
    public void TestValues(string input, int expectedCode)
    {
        var app = new AppRunner<StringValueApplication>(new AppSettings(){Case = Case.LowerCase});
        var result = app.Run(input.Split(' '));
        result.Should().Be(expectedCode);
    }

    private class StringValueApplication
    {
        public int Test(
            [Argument(Name = "action")]
            string actionType,
            string text)
        {
            return 0;
        }
    }

    /// <inheritdoc />
    public StringValueParserTests(ITestOutputHelper testOutputHelper) : base(testOutputHelper)
    {
    }
}

Alright I looked into it.
I simplified the app. and it looks like this:
class Issue55
{
    public void process(string key)
    {
        Console.WriteLine(key);
    }
}

and this test fails:
[Fact]
public void ShouldAcceptBracketAsInputParameter()
{
    var app = new AppRunner<Issue55>(new AppSettings(){Case = Case.LowerCase});
    var result = app.Run("process \"sometext (and more)\"".Split(' '));
    result.Should().Be(0);
}

let's look at what's happening- Since this is a tests and we don't have handle to the actual main method, we have use string.split to split our full command by spaces. To combine the argument sometext (and more) into one argument we used " double quotes.
The split by spaces doesn't create proper result though. It creates this array :
["process", ""sometext", "(and", "more)"]
Clearly this is not what we were expecting. We want ["process", "sometext (and more)"] for it to function properly
When I run this test it works:
[Fact]
public void ShouldAcceptBracketAsInputParameter2()
{
    var app = new AppRunner<Issue55>(new AppSettings(){Case = Case.LowerCase});
    var result = app.Run("process" ,"sometext (and more)");
    result.Should().Be(0);
}

I tested it from command line also (on a mac) :

It worked fine for me with double quotes. Has this failed for you from CLI ?

Thanks for verifying - yeah - it failed from cli. so it appears that the arguments are parsed like in the first case and have to be rebuilt before being processed further. I'll see if i can provide a PR for you to spare you some work which suceeds all tests. Gotta pay back some gratitude for all the work that has been put into your project :)

internal class ArgumentParser
{
    private static readonly Regex ArgumentsSplitRegex = new Regex(@"(?<!\\)"".+(?<!\\)""|(?<!\\)'.+(?<!\\)'|[^\s]+", RegexOptions.Compiled);

    public static IEnumerable<string> SplitFlags(params string[] args)
    {
        var joined = string.Join(" ", args);
        foreach (Match match in ArgumentsSplitRegex.Matches(joined))
        {
            yield return match.Value;
        }
    }
}

Changing the argument parser to this fixes the issue, but causes a couple other errors though. gonna check if those are expected errors or edge cases where the syntax would not be compatible.

according to the code samples at https://chris.beams.io/posts/git-commit/ using " or ' to group a string as argument, seems to be an established pattern, so it's actually the unit tests, which have the invalid syntax (if this PR were to be merged), instead of the regex behaving unexpected. Do you agree?

There is an issue with parameter combinations like -ab being equivalent to -a -b
Is this standard console behavior? I can't remember having seen such a syntax before, since that would also require additional validation whether a command which has -a and -b does not also have -ab.

https://blogs.msdn.microsoft.com/twistylittlepassagesallalike/2011/04/23/everyone-quotes-command-line-arguments-the-wrong-way/

Recall2 that the CommandLineToArgV convention3 stipulates that arguments containing spaces be surrounded by double quotation marks. Following the above approach and surrounding arguments with quotation marks produces good results for simple cases, and many people stop here.

Looks like quotation marks are a common convention then

@bilal-fazlani something not ok with the PR?

I need some time to go throug it. (Couple of days)

@bilal-fazlani @taori  I think there's a problem with this approach.  While creating the feature tests, I found the following 2 scenarios where the argument is split when I don't think it should be.
I don't think the rules are consistent across OS's for escaping strings and including spaces in arguments.
Windows - dos prompt

Windows - git bash prompt

I don't think this framework should be responsible for trying to account for those.
I think the only time we should change what is passed into string[] args is when splitting options that contain : or = for value assignment.  Any other changes can violate the principle of least surprise.  We'd need to document them clearly.

I agree.
I would be happy if we free this lib from taking on this responsibility.
As pointed by Drew it's behaviour can vary from one shell to to another.

Perhaps it would be best to compile a list of how it should be parsing on each platform to produce the expected results then.

In both examples above, I'm on the same machine.  There's not flag indicate which shell is running this.  This is a problem any application will have if it can be run across shells.  I think it will be the responsibility of the users to be aware of this.  Tool authors can help by including a dump-args command to review this and/or adding logging that can be used to troubleshoot.
I'm still working on filling out the feature tests.  Would either of you be willing to fix that branch :) ?

I will sign up for this.

@taori I missed your post before my last one and was about to type up the same thing.  It would be helpful to find or compile that list.   In addition to each platform, we'd need to include the shell's on that platform.  i.e. Windows would be dos, git bash, powershell.  Linux and Mac have bash, zsh and several others.  Is this something you'd be able to pull together?
On a related note, how we determine what to display in the AppUsage is problematic.  Help defaults to assuming the app is running as a dotnet global command if a command name is provided via ApplicationMetadata.  The example includes dotnet {command}.  This isn't appropriate in many cases.  I updated it so that apps ending with .exe don't include this and I added a couple of options so devs could be more explicit.  I bring it up in case someone finds a way to be more deterministic with tihs while compiling the list for the different shells.

@drewburlingame The whole reason i started using this lib was to have some sort of integrated argument parsing. If there was a way to have it behave the way ppl would expect it on each version that would be the ideal approach. unfortunately i have no idea how it should behave on mac/linus

@taori I agree.  I'd like that too.  The challenge as I see them

Arguments are already parsed to some extent before being passed to our framework and I don't know how we can determine what rules were used in that parsing and what characters will have been removed.
We can't know which shell the app is being called from (not that I think we should need to).
Devs using a shell may know how that particular shell parses the values and if we change the behavior, we could be violating the principle of least surprise.
Given the number of shells and new shells being created, it will require additional maintenance for us to try to account for all the variations.

My personal preference is to provide tools to help devs troubleshoot when args don't parse the way they expect.  I also like the idea of a echo-args tool that will let devs get rapid feedback on how their app is parsed.    I've added a --echo-args option to some of my apps in the past that have been helpful.  It's like --help, but prints out the string[] arg and how the values were mapped into arguments.  I think it would be worthwhile to make this something that could be enabled.

perhaps the best approach to solve this would be to have this controlled through parseroptions with something like a ICommandLineParser interface which splits the arguments. That way whenever someone experience issues he can just modify the behavior. Also that would allow to build in some shell detection possibly and replace the parsing behavior on demand.

I'm still stuck on how could we determine what the shell didn't give us.   For example, what was the original string passed to

This doesn't contain the quotes I used. Were they single or double quotes?
Since the shell is a different process, I don't see how we can replace the parsing behavior on demand.  The parsing is done before our process is spawned.  We could try to reconstruct the original string, but... we don't know what the shell removed so I don't think we can reliably reconstruct the string.
Is there a .net feature that can tell us which shell was used?   The only way I can think of to determine the shell is to walk the parent processes tree.  This will require complex, OS-specific code.  That's a lot of complexity to introduce and maintain.

it makes sense that the quotes are not parsed however. think of file names which might contain spaces - there has to be a way to pass that as argument. i can't imagine that it is impossible to get environmental info which might indicate how to parse arguments though. the sample you pointed out with different behavior depending on the shell is no doubt something which must be fixed.

I'm not sure what there is for us to fix. As shown in this PR, the data is parsed before we can do anything with it and we don't have enough data to try to recover.  What you're asking for doesn't appear to be technically possible.
This problem has existed since... well probably since the second shell was created.   And each shell has a way to escape strings so args with spaces can be passed in.  You just have to learn the behavior of the shell you choose.
I think this is a benefit instead of a problem.   If every program escaped strings differently, you'd have to learn how each program did it and each program would have to implement that logic.  Now, you only need to learn the behaviors of your chosen shell(s).  If you see the example on the PR, I know that double quotes will preserve spaces in both shells.  Single quotes only one.  So I use double quotes for both.   I just tested with Powershell and it has the same rules for double quotes but appears to have different rules for escaping double quotes.   I rarely use Powershell so I know I have to get up to speed on that.  I know that once I've learned the rules, they'll work the same for every program I use via Powershell.

as a side note, I just learned that Powershell appears to be implemented similar to the echo-args command in that PR.  Each parsed arg is emitted in it's own line.

System.CommandLine has a [parse] directive for this.  Perhaps that's something we should consider. https://github.com/dotnet/command-line-api/wiki/Features-overview#parse-preview

@taori after a coding frenzy the last couple of days, I've updated the parsing logic and introduced a way to keep standard parsing behavior but also enable custom transformations so you can achieve the behavior you're looking for.  See https://github.com/bilal-fazlani/commanddotnet/pull/142 for examples.
If you AppSettings.EnableDirectives you can use the [parse] directive to debug your custom transformation in the console.
I've created release 2.8 which should be available on nuget shortly.

@taori do you mind if we close this issue?

@drewburlingame i don't mind at all if tests still run fine really. More bilal's decision though.

👍  OK from my side . closing this.


Update Makefile

Make a new phony target that creates a graphColoring share object file

Hi
Could please  explain what specific parameter should I change to use other algorithms than DSatur?
My aim is to run Tabu col algorithm for my dataset and get coloured nodes as some form of output. I transformed my edge list to the given file type successfully but could not figure out the algorithm selection part.
Out of subject question:
I apologise for this super ignorant question but is there a specific algorithm that I can set my color (k). The reason that I am interested in to Tabu because I found this description in one the reviews.

Tabucol starts by generating a random coloring on the given graph G using k color classes. This will (almost certainly) not be a legal coloring. The objective function fc which Tabucol is trying to minimize is equal to the number of conflicts in a configuration: fc = 􏰂kn=1 |∀(i, j) ∈ E : c(i) = c(j)|. If a valid k-coloring C has been found fc(C) = 0.

Thank you very much for your help,
Best regards,
Tunc.


Don't connect to a host when detecting the type

Instead of connecting to the host to determine if the type is Host or
HostESX, just assume it is ESX.
@Fryguy what is the purpose of the ManageIQ::Providers::Vmware::InfraManager::Host instead of the ManageIQ::Providers::Vmware::InfraManager::HostEsx type?  Is this supposed to be GSX or workstation?  Can we just assume all hosts connected to vSphere are ESX/ESXi?
Also this is the last place VimConnectMixin is included in the main repo so we can move it to -providers-vmware after this.
cc @kbrock

I like this much better than #16451  (more ✂)

Also this is the last place VimConnectMixin is included in the main repo so we can move it to -providers-vmware after this.

Yay!
To reiterate, If this is a GSX feature, then it may be time to let that one go.
The VmwareHost seemed to only have 1 type when introducing STI:
commit 39c945ce73e7dfc7c7e87ca46f4bd61490ea4e2f
Author: Oleg Barenboim<EMAIL_ADDRESS>Date:   Fri Aug 5 19:49:32 2011 +0000

    Subclass Hosts using Single-Table Inheritance (STI)
    BugzID:12905

Later, it looks like a second type was added as a fallback:
commit 22507bf1635654f87b66e2ca7b28cbd00090bdfa
Author: Brandon Dunne<EMAIL_ADDRESS>Date:   Wed Jul 10 23:27:07 2013 -0400

    Refactor detect_type to return "Host" when the VENDOR_TYPE is unknown

I had assumed the distinction in these classes is to allow the EsxHost to be used without an Ems.
On the history of GSX, it was discontinued 1/2010 and support ended 6/2011 support pdf.

@bdunne Maybe you could chime in if you remember why this code was added. Think that coul put this discussion to rest quickly.
commit 22507bf1635654f87b66e2ca7b28cbd00090bdfa
Author: Brandon Dunne<EMAIL_ADDRESS>Date:   Wed Jul 10 23:27:07 2013 -0400

    Refactor detect_type to return "Host" when the VENDOR_TYPE is unknown

@kbrock I was working on https://bugzilla.redhat.com/show_bug.cgi?id=983101  I don't remember all of the details, but I'll look into it a bit more.

So this code needs authentication to be present for the host so to get this to run I had to discover the host, then add the credentials, then re-discover it.  I don't think we can possibly hit the connect case on the initial discover because there can't be any authentications yet.
Now that we have detect_hypervisor on the Vmware::InfraManager::Host (in https://github.com/ManageIQ/manageiq-providers-vmware/pull/106) after adding the credentials and re-discovering would actually go through that code not here so this should be safe to remove.

Host is used to be a "shell" for bare metal ESX provisioning.  Once the provisoining completes it is upgraded" to whatever type was installed.

@Fryguy The credentials statement really stuck out in my mind:
When this host is discovered, has_credentials?(:ws) == false. So this block is not called.
When credentials are later added, type == "Vmware::HostEsx", so this method will not be called but rather Vmware::HostEsx#detect_discovered_hypervisor.
So this code is never run.

@Fryguy can you take a look?  Unless it is possible to add authentication for a host record before it is created I don't think we can ever hit this code now that we added detect_discovered_hypervisor to the vmware host here https://github.com/ManageIQ/manageiq-providers-vmware/pull/106

ping @Fryguy


Add missing Permissions for CircleCI Deployer Role

These are required for Managing AWS Terraform Pipelines for DPR Project
A reference to the issue / Description of it
We Require additional AWS Permissions for CircleCI Deployer IAM Role to manage DPR Specific Resources
How does this PR fix the problem?
This PR will give required perms to successfully run the CIrcleCI Terraform Apply Pipeline
How has this been tested?
Please describe the tests that you ran and provide instructions to reproduce.
Yes, Tested on DEV-Sandbox Env
Deployment Plan / Instructions
Will this deployment impact the platform and / or services on it?
N/A
Checklist (check x in [ ] of list items)

[ ] I have performed a self-review of my own code
[x ] All checks have passed
[ ] I have made corresponding changes to the documentation
[ ] Plan and discussed how it should be deployed to PROD (If needed)

Additional comments (if any)
{Please write here}

@harichintala1 - is this one still relevant? If not, would you mind closing this PR?


Download http_api_endpoint

Added documentation to download the http_api_endpoint binaries.
@gbarr01
@meticulous-dft
Ming, I could not find the http_api_endpoint binaries for power and zlinux.
The http_api_endpoint binary is in the s3 bucket for linux and windows.
https://s3.amazonaws.com/store-logos-us-east-1/certification/windows/http_api_endpoint.exe
https://s3.amazonaws.com/store-logos-us-east-1/certification/linux/http_api_endpoint
Could not find it for zlinux and power.
https://s3.amazonaws.com/store-logos-us-east-1/certification/zlinux/http_api_endpoint
https://s3.amazonaws.com/store-logos-us-east-1/certification/power/http_api_endpoint

Proposed changes

Unreleased project version (optional)

Related issues (optional)

@gforghetti , looks like the endpoints for zlinux and power were added (by @meticulous-dft ?) since you filed this? Can you verifty?

Nevermind ... I was wrong.

@gbarr01 @gforghetti All the binaries for plugins and the http_api_endpoint are uploaded / updated.
@gbarr01 Please merge the PR once you have time. Thanks.


Bad signature

I've this problem when try to install Janus.
gpg: key A39CC3ADAEAEF36E: public key "Theodore Ratte Wilmes (CODE SIGNING KEY<EMAIL_ADDRESS>imported
gpg: key 66B85FF141802BA8: public key "Jason Plurad (CODE SIGNING KEY<EMAIL_ADDRESS>imported
gpg: Total number processed: 2
gpg:               imported: 2
gpg: Signature made чт, 12-жов-2017 00:16:31 +0300 EEST
gpg:                using RSA key 66B85FF141802BA8
gpg: BAD signature from "Jason Plurad (CODE SIGNING KEY<EMAIL_ADDRESS>[unknown]

My OS information:
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04 LTS
Release:	18.04
Codename:	bionic

Will be fixed in #276


Introduce onSuggestionsClearRequested

Current coverage is 97.38% (diff: 96.25%)

Merging #229 into master will decrease coverage by 0.26%

@@             master       #229   diff @@
==========================================
  Files             4          4
  Lines           510        535    +25
  Methods          49         50     +1
  Messages          0          0
  Branches         79         86     +7
==========================================
+ Hits            498        521    +23
- Misses           12         14     +2
  Partials          0          0

Powered by Codecov. Last update 4ba8900...a073ac7


Using vanilla break when attempting to cut down tree

I read on the mod page that multi-mine may be the cause of this issue but should have been resolved. I attempted to use the fix given on that issue and it kinda worked? If I hold down left click while trying to break the block, it breaks like vanilla. If I almost fully break it but stop, then finish the break, then it will treechop break twice. If I continue to hold it from there, it will treechop break the rest of the way. Using minecraft version 1.20.1 and mod version 0.19.0
https://github.com/hammertater/treechop/assets/147950902/f258871a-bec5-42dd-a662-745e65414cd9
latest.log

Weird, I tested this the other day and it worked well enough. The double-chopping thing has always been an issue that's tricky to solve, as it should really be fixed on MultiMine's end (they trigger block breaking twice sometimes, when doing what you showed). You can always add tree blocks to multimine's blacklist in their config file, which should fix both issues
I'll look into it more soon, thanks for reporting


SQL db cleanup fix

Description
This update will remove the sql dbs that were created during the compatibility testing phase.
Related issues
Addresses [issue #].
Testing
Ran the SqlServerSqlCompatibilityTests tests before the change and after the change locally. Before the change there were sql dbs created but not cleaned up. After the change, I ran the same SqlServerSqlCompatibilityTests tests and all testing dbs were now cleaned up.
FHIR Team Checklist

[X] Update the title of the PR to be succinct and less than 50 characters
[ ] Add a milestone to the PR for the sprint that it is merged (i.e. add S47)
[X] Tag the PR with the type of update: Bug, Dependencies, Enhancement, or New-Feature
[ ] Tag the PR with Azure API for FHIR if this will release to the Azure API for FHIR managed service (CosmosDB or common code related to service)
[ ] Tag the PR with Azure Healthcare APIs if this will release to the Azure Healthcare APIs managed service (Sql server or common code related to service)
[ ] CI is green before merge
Review squash-merge requirements

Semver Change (docs)
Patch|Skip|Feature|Breaking (reason)

/azp run


Should we create a null-prototype object?

Should the object returned by groupBy() have a null prototype? It could help add some safety when using the object returned by .groupBy() with unknown keys. Some of the value of this would be lessened if we provide a .groupByMap() function, as being discussed in #3, but it would still be useful either way.

Not just should, must. This was discussed in plenary; see #1.


As a user of the Household model converter, I'd like the primary contact to be set by contact created date if no giving information is available

Probably just need to order this query:
https://github.com/SalesforceFoundation/Cumulus/blob/dev/src/classes/CONV_Account_Conversion_BATCH.cls#L78

Included in beta release version 3.32 (Beta 7) and higher

Included in production release version 3.32


Simplify BabyBear memory machine

This removes a complicated comparison mechanism introduced in #1874.
Previously, time steps were compared differently from addresses. Now, both are compared the same way which is:

The prover provides a bit indicating whether the high limbs of the values are equal
If this bit is 1, a constraints asserts this is actually the case
The prover provides the 16-Bit difference minus one on the limb that differs. This asserts that the limb value is strictly increasing.

I guess we should also rename the memory machine. But I think this could happen in a different PR.

I guess we should also rename the memory machine. But I think this could happen in a different PR.

Done in https://github.com/powdr-labs/powdr/pull/1881


missed P2082R1 (Fixing CTAD for aggregates)

@fsb4000 : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@corob-msft : Thanks for your contribution! The author(s) have been notified to review your proposed change.


[main] Upgrade to latest dependencies

Cron -knative-prow-robot
/cc knative-sandbox/networking-wg-leads
/assign knative-sandbox/networking-wg-leads
Produced by: knative-sandbox/knobots/actions/update-deps
Details:
bumping knative.dev/pkg 787aec5...6352c0c:
  > 6352c0c Drop unnecessary list creation from bucketer (# 2360)
bumping knative.dev/networking aa82452...b615bc5:
  > b615bc5 upgrade to latest dependencies (# 580)
  > ea794f1 upgrade to latest dependencies (# 579)

/retest

Ah, we still did not get https://github.com/knative/pkg/pull/2361


WIP: valid data masking

Makes the "valid nav mask" available in UDFs, and propagates it to results.
Closes #1473, closes #1449
TODO

[x] improve test cases
[x] implement BufferWrapper.masked_data and friends
[ ] update plotting to use the per-buffer valid data mask instead of "global" damage
[x] figure out how this fits best with rois
[ ] documentation

Contributor Checklist:

[ ] I have added or updated my entry in the creators.json file
[ ] I have added a changelog entry for my contribution
[ ] I have added/updated documentation for all user-facing changes
[ ] I have added/updated test cases
[ ] I have included the rebuilt production build of the client (only if changes were made to the GUI)

Reviewer Checklist:

[ ] /azp run libertem.libertem-data passed
[ ] No import of GPL code from MIT code

/azp run libertem.libertem-data

I like the API! Really minimal and simple. Regarding masked array support in other backends, I came up with the following:

Dask: https://docs.dask.org/en/latest/generated/dask.array.ma.masked_array.html
CuPy: https://github.com/cupy/cupy/issues/2225
Sparse: https://github.com/pydata/sparse/issues/359

Since we have, as far as I am aware, only NumPy and Dask results, it should work. If we want to support other return backends, one could implement the thin wrapper alternative for them, essentially a "poor man's masked array" type. When the time comes we can think of an API that allows creating the right kind of array which is backwards-compatible with NumPy masked arrays.
Regarding ROI behavior, what about the following?
Internally, everything works on the ROI like before. That means the mask is relative to the nav space with ROI applied -- always of the same shape as the actual result buffer one is interacting with. Result buffer .data only sets the values within the ROI that are not masked, meaning masked parts of result buffers are treated the same way as areas outside the ROI.

Hm, the fill_value parameter is interesting relative to our ROI considerations. One could also argue that masked areas should have the fill value and not the "outside of ROI" value.
Now I'm not so sure if we should use masked arrays directly.
Pro

We use an existing API of NumPy
Potentially benefit from additional capabilities, see open question
It generally matches what we want to express with the valid mask

Con

We should support all features of masked arrays in a predictable way if we allow users to pass them directly. With ROIs it leads to an ambiguous situation.
The behavior of masked arrays is opposite of ROIs and damage -- "1" means invalid for masked arrays, but valid for ROI and damage.
Not available in CuPy, which would be a likely target for get_results() implementations that benefit from GPUs, like iCoM

Open questions

Will the additional features of masked arrays be useful for the user, such as numerical operations on them?

Conclusion
Looking at the implications, we might be better off to implement a thin wrapper type?

Open questions

Will the additional features of masked arrays be useful for the user, such as numerical operations on them?

Conclusion
Looking at the implications, we might be better off to implement a thin wrapper type?

My view on this part is that if we go with a thin wrapper, we can always offer a method on the wrapper which gives the user a true np.ma array with the most useful properties for them (valid only, roi only, valid + roi etc), and it gives us more flexibility in terms of fill values because we can handle that ourselves. We also don't have to construct anything new unless the user actually wants to take advantage of np.ma, which might avoid some copies ?
We also retain the option to support non-masked-compatible backends, as you've said already.
Overall I like the implementation and idea behind it 👍

The behavior of masked arrays is opposite of ROIs and damage -- "1" means invalid for masked arrays, but valid for ROI and damage.

Oof, yeah, I completely missed that one. I think that alone is enough to convince me that a thin wrapper can provide a better API.

Looking at the implications, we might be better off to implement a thin wrapper type?

My view on this part is that if we go with a thin wrapper, we can always offer a method on the wrapper which gives the user a true np.ma array with the most useful properties for them (valid only, roi only, valid + roi etc), and it gives us more flexibility in terms of fill values because we can handle that ourselves. We also don't have to construct anything new unless the user actually wants to take advantage of np.ma, which might avoid some copies ?
We also retain the option to support non-masked-compatible backends, as you've said already.

Thanks for the feedback, I'll work on an updated version of these changes. I'm thinking on what aspects have to be taken into account.

Since we have, as far as I am aware, only NumPy and Dask results

Yeah, according to our documentation, get_results is supposed to always return np.ndarray. This is actually wrong, as you correctly state, it can also be a dask.array.core.Array, with the delayed executor. I ran the test suite with the following patch, and at least for what we cover, Union[np.ndarray, dask.array.core.Array] is the true type there.
diff --git a/src/libertem/udf/base.py b/src/libertem/udf/base.py
index bc0b80b9..3efc03e3 100644
--- a/src/libertem/udf/base.py
+++ b/src/libertem/udf/base.py
@@ -1214,6 +1214,8 @@ class UDFBase(UDFProtocol):
         # wrap numpy results into `ResultBuffer`s:
         results = {}
         for name, arr in results_tmp.items():
+            import dask.array.core
+            assert isinstance(arr, (np.ndarray, dask.array.core.Array))
             self._check_results(decl, arr, name)
             buf_decl = decl[name]
             buf = results_buffer_cls[name](

We do not yet support returning cupy arrays from get_results, for the iCoM case, I'm currently doing a round-trip CPU->GPU->CPU, which is still very fast . In the future, if we want to optimize the "very large scan" use-case, we may want to keep the result buffers on the GPU, for example to directly apply a shader for visualization.
I'm also thinking about including some helper functions on the wrapper type, as there will be some common operations:

Is everything valid?
What is the (outer) bounding box (as 2d or nd slices)?
What is the inner bounding box, meaning the slices for the "rectangular" region where all values are valid?

Tangent: missing data and dropped frames
So, we also had the topic of missing data some time ago (#256), where we kind of punted out, IIRC with the data set returning zeros for missing portions or similar. Now, with live processing, it will sadly become more common that we drop frames (for example when optimizing a new UDF for live processing and can't keep up, or when the computer used is simply not fast enough to keep up with the chosen computation). With the valid_nav_mask or something similar, we now have the chance  to properly communicate this to the UDF in a consistent way. But maybe we actually want to have a separate mask for this, to distinguish dropped frames / missing data from (in the case of STEM) not having visited a given position yet.
A reason for keeping these separately is that UDFs may want to only show results in the "inner bounding box", a rectangular region where all positions have been seen. Otherwise, we may get artifacts/histogram/flickering issues for the live display, at borders etc. If we have a single mask only, and we are missing a single frame, this region will never grow beyond that position.
So maybe, in this PR, valid_nav_mask should actually be documented to include positions with missing data, and in the future we can add a second mask especially for the missing data. Agreed?

The web_api tests are a bit flaky now, which I managed to reproduce locally, too. I suspect this is related to #1577 and the changes to how the server creates executors etc. - before bugging out, the test was hanging for quite some time in make_executor:
Thread 1706497 (idle): "LiberTEM-background"
    stop (distributed/utils.py:606)
    close (distributed/client.py:1790)
    __del__ (distributed/client.py:1522)
    _make_invoke_excepthook (threading.py:1324)
    __init__ (threading.py:910)
    __init__ (distributed/utils.py:479)
    _start_unlocked (distributed/utils.py:590)
    start (distributed/utils.py:551)
    start (distributed/client.py:1209)
    __init__ (distributed/client.py:1014)
    connect (libertem/executor/dask.py:701)
    make_executor (libertem/web/state.py:128)
    _do_connect (libertem/web/connect.py:53)
    put (libertem/web/connect.py:45)
    _execute (tornado/web.py:1790)
    _run (asyncio/events.py:80)
    _run_once (asyncio/base_events.py:1914)
    run_forever (asyncio/base_events.py:607)
    run (conftest.py:939)
    _bootstrap_inner (threading.py:1045)
    _bootstrap (threading.py:1002)

(where "LiberTEM-background" is the thread used to run the web API in the background for testing - used in fixtures like base_url)

/azp run libertem.libertem-data

/azp run libertem.libertem-data

/azp run libertem.libertem-data

/azp run libertem.libertem-data

/azp run libertem.libertem-data

/azp run libertem.libertem-data

/azp run libertem.libertem-data

@uellue could you have a look specifically at the CoM changes?

Looks good to me. If there is anything missing it is a description in the doc of how this differs from the existing damage on UDFResults, which is linked but a less granular information about validity.

Thanks for having a look, I'll add a short note about the relation to UDFResults.damage.

/azp run libertem.libertem-data

/azp run libertem.libertem-data

/azp run libertem.libertem-data

/azp run libertem.libertem-data

/azp run libertem.libertem-data

/azp run libertem.libertem-data

/azp run libertem.libertem-data

/azp run libertem.libertem-data


Adding references to the MS Learn modules

Acrolinx Scorecards
A minimum Acrolinx score of 80 is required.
Click the scorecard links for each article to review the Acrolinx feedback on grammar, spelling, punctuation, writing style, and terminology.

Article
Acrolinxscore
Lowestcategoryscore
Spellingissues
Scorecard
Processed

articles/index.yml
87
54
0
link
:white_check_mark:

articles/welcome.md
85
73
0
link
:white_check_mark:

More information about Acrolinx

Install Acrolinx locally for VSCode
False positives or issues
Troubleshooting issues with Acrolinx

Docs Build status updates of commit a867a98:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

articles/index.yml
:white_check_mark:Succeeded
View

articles/welcome.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Acrolinx Scorecards
A minimum Acrolinx score of 80 is required.
Click the scorecard links for each article to review the Acrolinx feedback on grammar, spelling, punctuation, writing style, and terminology.

Article
Acrolinxscore
Lowestcategoryscore
Spellingissues
Scorecard
Processed

articles/index.yml
87
54
0
link
:white_check_mark:

articles/welcome.md
85
73
0
link
:white_check_mark:

More information about Acrolinx

Install Acrolinx locally for VSCode
False positives or issues
Troubleshooting issues with Acrolinx

Docs Build status updates of commit d6b72d5:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

articles/index.yml
:white_check_mark:Succeeded
View

articles/welcome.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Acrolinx Scorecards
A minimum Acrolinx score of 80 is required.
Click the scorecard links for each article to review the Acrolinx feedback on grammar, spelling, punctuation, writing style, and terminology.

Article
Acrolinxscore
Lowestcategoryscore
Spellingissues
Scorecard
Processed

articles/index.yml
87
54
0
link
:white_check_mark:

articles/welcome.md
85
73
0
link
:white_check_mark:

More information about Acrolinx

Install Acrolinx locally for VSCode
False positives or issues
Troubleshooting issues with Acrolinx

Docs Build status updates of commit 87e5d30:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

articles/index.yml
:white_check_mark:Succeeded
View

articles/welcome.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


What about babel-plugin-transform-flow-comments?

I think this is the simplest and most obvious solution. Better than potentially broken generation *.flow flies. So what about?

@rosskevin what do you think?

What problem is this a solution to?
What is broken in *.js.flow shadow files?

I think what @bigslycat trying to say is avoid shipping 2 seperate version (with flow and without flow by converting flow annotation to comment). This will make npm size smaller.

The npm size may be smaller, but it will have zero effect on any bundle size as these are never included.  And is a few kb smaller npm package size a win?
I think it is much easier to comprehend the flow typing in the context of the source file.  Trying to wrap my head around comments jammed into a transpiled file is far more difficult than seeing the source as *.js.flow file.  We use many plugins to optimize transpiled files, and this will only make it harder in userland to comprehend flow typings.
Unless I'm missing something compelling, I'm not in favor of switching.

@rosskevin i'm not in favor of switching either. The above comment is just what I got from reading this thread. And FYI: flow accepts comment as type annotation. See https://flow.org/blog/2015/02/20/Flow-Comments/
I use Flow comment sometimes but I have to admit, it's harder to read and pure editor support

@bigslycat feel free to provide more argument regarding "why" we should do such change. For now, let's close it. Thanks for the discusion :).


Ubuntu 20.04 LTS?

On the page of supported Ubuntu guest OS versions, it would be good to mention Ubuntu 20.04 LTS.  If it's supported, I'd expect a column; if it's unsupported, a note making that clear would be good.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 84d52296-198d-7c8a-db99-e8cb336e355f
Version Independent ID: 77670399-109f-3233-0466-c13d84542039
Content: Supported Ubuntu virtual machines on Hyper-V
Content Source: WindowsServerDocs/virtualization/hyper-v/Supported-Ubuntu-virtual-machines-on-Hyper-V.md
Product: windows-server
Technology: compute-hyper-v
GitHub Login: @shirgall
Microsoft Alias: shirgall

Fixed by https://github.com/MicrosoftDocs/windowsserverdocs/commit/fb78b64c3a2ffc434c1e04d16cf7804e9c00df81?

Fixed by https://github.com/MicrosoftDocs/windowsserverdocs/commit/fb78b64c3a2ffc434c1e04d16cf7804e9c00df81?

Fixed by https://github.com/MicrosoftDocs/windowsserverdocs/commit/fb78b64c3a2ffc434c1e04d16cf7804e9c00df81#diff-2c87b9243c51b6fdef6774ae8e77296f172eca2841618beee48d6e0430185e4b


feat: add Tabbar

Coverage increased (+5.4%) to 75.606% when pulling 07efd934bf44a488220382ca30d51eb8080d04b5 on Yang03:tabbar into 4049122c7be265f341222c528c004af2def9e5ff on ZhonganTechENG:dev.


spicedao.eth update SKIN & Avatar

Updated it @gokaiorg

Updated it @gokaiorg

There is a conflict in your PR, can you resolve it?

There is a conflict in your PR, can you resolve it?

@gokaiorg I am not sure how to do that .. I have pushed two image files and the scss file. Any ideas?

@gokaiorg I am not sure how to do that .. I have pushed two image files and the scss file. Any ideas?