Camera sample doesn't close NnApiDelegate

The code here
https://github.com/android/camera-samples/blob/42f76c4aab7c7c2d41110c3c127c20c08882e079/CameraXTfLite/app/src/main/java/com/example/android/camerax/tflite/CameraActivity.kt#L89
allocates an NnApiDelegate object, and never calls close() on that object.
   private val tflite by lazy {
        Interpreter(
            FileUtil.loadMappedFile(this, MODEL_PATH),
            Interpreter.Options().addDelegate(NnApiDelegate()))
    }

But the API documentation for NnApiDelegate says that the user is expected to explicitly call close():
https://github.com/tensorflow/tensorflow/blob/4e72b2da554ceb1b5aa5559c8881caa3d4a6b97b/tensorflow/lite/delegates/nnapi/java/src/main/java/org/tensorflow/lite/nnapi/NnApiDelegate.java#L177
The failure to call close() results in a memory leak, which can be difficult to debug;
for more details, see https://github.com/tensorflow/tflite-support/issues/269.

I think the fix could be fairly simple, e.g.

add an NnApiDelegate field to the CameraActivity class, initially null;
change the code referenced in the original bug report to check if the NnApiDelegate field is non-null, and if so, to use that object (rather than allocating a new NnApiDelegate object); else, i.e. if the field is null, go ahead and allocate a new NnApiDelegate object, just as in the existing code, but store that object in the new field before passing it to addDelegate.
call close() on the new NnApiDelegate field from the CameraActivity's onDestroy method.


api_key.delete( ) doesn't work

In tests, we are doing api_key.delete() and it doesn't work. So, we are creating a lot of API keys in the testing account

+1
Are there any plan to fix this bug?

This issue should be solved with the following PR


[BUG] - When perma banning I need to fill in length...

Describe the bug
When I try to perma ban someone I need to fill in the length...
Expected behavior
Just skip length choose perma and fill in reason and ban him.

This issue has been fixed with: https://github.com/11TStudio/SquadStatsJSPRO/commit/c9ed6e7cc7acce4bb9a73ff9b6de044ae186f9e5


"mkchain my-chain" in the readme fails - needs updating

The readme quickstart section of the readme contains the following:
mkchain my-chain

which when executed as instructed fails with the following message:
mkchain my-chain
usage: mkchain [-h] [--version] {generate-constants,create,invite} ...
mkchain: error: argument action: invalid choice: 'mkchain' (choose from 'generate-constants', 'create', 'invite')

The readme needs updating

Attempting to use mkchain generate-constants andrey-chain results in the error:
Traceback (most recent call last):
  File "/Users/andrey/Documents/TQ/tezos-k8s/.venv/bin/mkchain", line 8, in <module>
    sys.exit(main())
  File "/Users/andrey/Documents/TQ/tezos-k8s/.venv/lib/python3.9/site-packages/tqchain/mkchain.py", line 329, in main
    keys = gen_key(args.docker_image)
  File "/Users/andrey/Documents/TQ/tezos-k8s/.venv/lib/python3.9/site-packages/tqchain/mkchain.py", line 48, in gen_key
    "secret_key": keys[2].split(b":")[2].strip().decode("utf-8"),

Attempting to use mkchain generate-constants andrey-chain results in the error:
Traceback (most recent call last):
  File "/Users/andrey/Documents/TQ/tezos-k8s/.venv/bin/mkchain", line 8, in <module>
    sys.exit(main())
  File "/Users/andrey/Documents/TQ/tezos-k8s/.venv/lib/python3.9/site-packages/tqchain/mkchain.py", line 329, in main
    keys = gen_key(args.docker_image)
  File "/Users/andrey/Documents/TQ/tezos-k8s/.venv/lib/python3.9/site-packages/tqchain/mkchain.py", line 48, in gen_key
    "secret_key": keys[2].split(b":")[2].strip().decode("utf-8"),

This has been done

This has been done


Improve full text search for default views

Motivation
Performing full text search in Fusion is the project page does not bring satisfying results due to the way data is indexed:
Search in only one field where everything is pushed so everything that should have more weight (id/name/descroption) gets flooded in the rest
Acceptance criteria

Description is indexed as an independant field
The tokenizer/analyzer is changed so as to better index the id for example
The full text search part of the listing operation is changed so as to target several fields (already done in Global search)

@imsdu might pick up this next, although it might be more suited to someone with domain knowledge... Would it be ok to allow feeding the weightings through an endpoint with a small UI? Otherwise the feedback loop is long and it'll be hard to iterate towards a "better" search
Also who uses the search feature? I.e. who can verify that the results are better

The idea is not to break retro-compatibility and not to introduce any weighting for now but to change the full-text search part of the listing operation (the q param) so that it does not target only the allFields fields but also the other fields like the id, the name, the type that are defined in the default mapping.
That will be a first good step as the default TF/IDF algorithm should already help a lot.
For the id/types/_location and the other ones that are Iris/Uris, it would be nice to define a multifield with a subfield for full-text search and have a separate tokenizer that correctly handles Iris (this one would be a good candidate: https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-uaxurlemail-tokenizer.html)
For the other ones like name, description and other than all_fields, we could use this one: https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-edgengram-tokenizer.html
all_fieldscan stay as it is.

The idea is not to break retro-compatibility and not to introduce any weighting for now but to change the full-text search part of the listing operation (the q param) so that it does not target only the allFields fields but also the other fields like the id, the name, the type that are defined in the default mapping.
That will be a first good step as the default TF/IDF algorithm should already help a lot.
For the id/types/_location and the other ones that are Iris/Uris, it would be nice to define a multifield with a subfield for full-text search and have a separate tokenizer that correctly handles Iris (this one would be a good candidate: https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-uaxurlemail-tokenizer.html)
For the other ones like name, description and other than all_fields, we could use this one: https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-edgengram-tokenizer.html
all_fieldscan stay as it is.

For the query part, this can be a good candidate: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-multi-match-query.html
With * in the fields part

Also we can remove the nested for the different fieds


Weird hstring.empty() behavior immediately after using co_await

Say I have this block of code:
IAsyncAction DoThingAsync(hstring const& par)
{
	if (!par.empty())
		throw hresult_invalid_argument(L"first check went wrong: " + par);

	// Some intensive tasks
	using namespace std::chrono;
	co_await 1s;

	if (!par.empty())
		throw hresult_invalid_argument(L"second check went wrong: " + par);
}

Passing hstring{ } to DoThingAsync will throw on the second check, with some weird output:
second check went wrong: ˕翻

Am I missing something here? Commenting out co_await 1s; doesn't throw (as expected), and passing a non empty hstring makes it throw on the first check (as expected).
Using CppWinRT 2.0.220531.1 on VS2022 17.2.3, Platform Toolset set to v143. This is also within a Runtime Component.

You're taking the hstring by reference, so by the point you hit the second empty() check it has been freed by the caller (because the coroutine returns to the caller by the first suspension point). If you want a use the parameters in a coroutine, you should take them by value.

Thanks, Visual Studio gave me some weird warning about it but it was showing up inconsistently, thought I was going crazy.


Multi-language support. Link generated with english google drive cannot be opened on system with german google drive.

The main entry point paths into your google drive files "My drives" & "Shared drives" are translated depending on your google driver language setting (or OS language setting).
Our link contains a language specific pathCloudStorage/GoogleDrive-user_account/Shared Drives/...
This creates a problem when sharing links between users with different language settings.

Fixed in version 1.0.1


Centos disk image version updated

What are your changes?
What type of changes does your doc introduce?

[x] Bug fix
[ ] New feature

What is the urgency level of this change?

[ ] Blocker
[ ] High
[x] Medium

I have completed these items:

[x] I have tested all commands and code snippets.
[x] I have built locally and tested for broken links and formatting.
[x] I have made this change for all affected versions (e.g. 1.7, 1.8, and 1.9).
[x] My doc follows the style guide: https://github.com/dcos/dcos-docs#contributing.

Ping @emanic, @sascala, or @joel-hamill for reviewing pull request changes.
If you included a comment with your commit, it appears here:

Can one of the admins verify this patch?

I've also changed a subnet for europe-west1-c

I'm not sure if I have followed the correct process here. I have committed three separate changes into this PR. If I need to do something else I'll be happy to oblige.


[BUG] .env variables read a string instead of list

Describe the bug
A clear and concise description of what the bug is.
Environment variable city_name  and facilities read as string instead of list.
To Reproduce
Steps to reproduce the behavior:

Update city_name  and facilities with appropriate values.
Run the bot
See error

Expected behavior
Successfully interprets city_name  and facilities as list instead of strings.
Screenshots

Fixed in merge #7 therefore closing


Loop all resources in known namespaces

This removes all resources in known namespaces so namespace removal does not get stuck on remaining resources in that namespace
Also adds 2.5.x fleet namespace fleet-system

@aemneina will you merge for us please?


BitCheckbox Title issue

I am using BitCheckBox in this way
<BitCheckbox Value=false Title="برای تست"></BitCheckbox>
But nothing is shown as title in UI

Thanks for contacting us. We're investigating this issue.
We'll let you know if it's possible to work on this issue or not.

Title is an attribute for the CheckBox input. it is shown as a popup. If you need something like a label, you need to use BitLabel separately. @msangtarash
What do you think? @msynk

The issue is accepted and planned. Resolving it will start ASAP.

Resolving this issue has started. It will be announced when this issue is resolved.

This issue has been resolved and it will be available in the next pre-release version. It will be announced when the pre-release becomes ready.

The pre-release has been published. Please give it a try and give feedback.

Thank you this is applied :)


refactor: use SfLink

Related issue

Closes #1125
Scope of work

[x] replace <component> to <SfLink in SfProductCardHorizontal
[x] replace review counts from <a> to <SfButton> as its clickable element

Screenshots of visual changes

Checklist

[x] No commented blocks of code left
[x] Run tests and docs
[x] Self code-reviewed

If applicable:

[ ] I followed composition rules for my component
[ ] I tested the component in most common device sizes (can be tested in Storybook from viewport addon in top menu)
[ ] I accept Individual Contributor License Agreement

This was kinda breaking change, as now it seems impossible to get rid of <a> tag around SfImage in SfProductCardHorizontal, and it leads to side-effects in situation when SfImage slot is replace with smth else or handles a custom click event.

@martaradziszewska , can you address this please, as there's no workaround apart from downgrade.

Possible solution would be disabling not rendering SfLink if null is passed (and make it default prop value).


Fix README: npm run build

this fixes the command written in the readme in the installation section.

Awesome, thank you!


Missing and/or incorrect country codes

It seems the data extraction from http://commondatahub.com/live/geography/country/country_dialing_codes may have had a couple of issues.
There are multiple instances of countries sharing a country code. US & Canada are a good example. Today I was looking for Australia in phone_countries.yml and found
"61":
  :country_code: "61"
  :national_dialing_prefix: "0"
  :char_2_code: "0"
  :char_3_code: CC
  :name: Cocos (Keeling) Islands
  :international_dialing_prefix: "11"

Turns out Cocos is a territory of Australia (Thanks Wikipedia!).
I also see that Norway is not in the file, but Svalbard is. My guess is that the data extraction just overwrote previous country code entries as it progressed. Which means we got luck with the US and Italy as Italy and the Holy See share a code.
Sorting the commondatahub table on dialing country code shows some others.
My first instinct is to just change the offending entires in phone_countries.yml to the more commonly recognized countries. However, this doesn't make much sense if the countries file is going to be automatically regenerated in the future.

I have user-originated phone numbers with a mix of "phone / tel / ..." in front or at the end of my strings, and I need to support Norway (curently clobbered as described above). Is it only the data import feature and elf.find_by_country_code(code) that is affected? Sounds easy to fix. Patches welcome?


Remove global endpoints

Type of change

New feature

Description
Remove and prevent the use of global orderer endpoints on version V3 and above
Related issues

Looks good modulo the comment above.
@denyeart maybe you can also review this one?

@denyeart Thanks! I saw you addressed the comments while I was away on vacation.


Security groups vs. Power BI admin rights

Hello,
Please add to this document https://learn.microsoft.com/en-us/power-bi/admin/service-admin-metadata-scanning-setup#enable-tenant-settings-for-metadata-scanning a note somewhere.
Note that when you enable the API setting only for specific security groups, Power BI Admin will need to be added to the security group as well.

The information is from Teams Power BI escalation channels:
[13:29] Meir Shimon
Indeed, your observation is correct.
To utilize the Scanner API, you have two authentication options:
Authenticate as a delegated user of the Power BI admin account.
Authenticate as a service principal that has been granted the necessary permissions by the Power BI admin, with the first switch enabled.
In either case, in order to obtain the schema of the dataset (sub-artifact), you must first enable the switch and be a member of the corresponding security group.
So, to answer on your question, these switches setting not overriding Power BI Admin rights to run the API but they overriding his privilege to get the schema in the response until it will enable it and will be member in the SG that enabled for.

Thank you.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: b5057b20-527f-b796-47c6-e63add79b15b
Version Independent ID: e7305792-f359-d0a9-3c59-9e6cd2024b13
Content: Set up metadata scanning in an organization - Power BI
Content Source: powerbi-docs/admin/service-admin-metadata-scanning-setup.md
Service: powerbi
Sub-service: powerbi-eim
GitHub Login: @paulinbar
Microsoft Alias: painbar

Thanks for your comment, @MartinaPoddana. This issue has been assigned to the doc author for review.


Update static asset URLs to use CDN endpoint

Description
The CDN serving fluidframework.com has been set up with routes that allow it to serve static content from our storage account. This updates direct references to the storage account to use those routes instead, which unblocks removing anonymous access to the storage account (a security requirement as per recent MSFT initiatives). As a side benefit, this should be a performance improvement for commonly served content (e.g. images on https://fluidframework.com)

I've also updated internal documentation on FF website infrastructure, reviewer can feel free to check that out if they want a few more details.


Make ignore_interrupts thread-safe.

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[ ] Have you written new tests for your changes? Here's an example.

[x] Have you successfully run brew typecheck with your changes locally?
[x] Have you successfully run brew tests with your changes locally?

Make ignore_interrupts thread-safe to prevent race conditions when cancelling concurrent downloads.

No, there is no way to avoid it, but as far as I understand the code in safe_fork, the child will receive the interrupt and send it back to the parent via exit code 130, so the ignore_interrupts is only there to avoid the parent receiving the interrupt before the child.

Doesn't this break the whole nesting level stuff? How can it ever have $ignore_interrupts_nesting_level > 1 if the whole block is mutex locked?

Right, I assumed the standard Mutex is reentrant. The question is, do we ever even call it nestedly?

Not sure, would need to audit existing usages, but it seems very easy for something to do ignore_interrupts { system_command!("echo") } (system_command has an ignore_interrupts internally).

Got rid of ignore_interrupts in in SystemCommand and safe_fork, all remaining usages only clean up files, so should be good.


presto-cli should acquire password using stdin&stderr when stdout is a pipe

This should be possible:
presto-cli --user me --password --execute 'select 1' > /dev/null

Currently, this fails with
Exception in thread "main" java.lang.RuntimeException: No console from which to read password
	at io.prestosql.cli.Console.getPassword(Console.java:173)
	at io.prestosql.cli.Console.run(Console.java:136)
	at io.prestosql.cli.Presto.main(Presto.java:31)

It should work just the same way as sudo echo x > /tmp/x does.

Workaround
One can specify the password with the PRESTO_PASSWORD environment variable.

Dear All,
Could someone help me with the issue I'm facing currently, Im connecting to presto using Presto CLI by constructing presto string including user and password and trigger it in bash.
When I grep the process in unix, the whole presto string is being shown including the password.
Help me to overcome this issue.


Apply clang format

Here is the process(a derivative of microsoft/MixedReality-GraphicsTools-Unreal)
and a copy of https://github.com/microsoft/AdaptiveCards/pull/6799

After you checked out the repo, run scripts\SetupClangFormat.bat to setup the git hooks. It's an one time setup.
You edit and check in the code just as before. When you run git commit, the git hooks will verify the format for staged files
If there is format issue, commit will continue, but show the error and provide command to ask user to format the file
User format the files via powershell  -ExecutionPolicy Bypass scripts\FormatSource.ps1 -ModifiedOnly $False -Path yourchangedfolder to format them before submitting the PR
Pipeline will detect the failures if there is format issue

Error example:
c:\repo\adaptivecards>PowerShell.exe -ExecutionPolicy Bypass scripts\FormatSource.ps1 -Verify $True -Path source
[clang-format] Checking formatting: C:\repo\adaptivecards\source\uwp\Renderer\dll\dll.cpp
C:\repo\adaptivecards\source\uwp\Renderer\dll\dll.cpp:11:7: error: code should be clang-formatted [-Wclang-format-violations]
STDAPI
      ^
Errors found (see output). Please make sure to resolve all issues before opening a Pull Request.
Formatting can be applied by running:
   PowerShell.exe -ExecutionPolicy Bypass C:\repo\adaptivecards\scripts\FormatSource.ps1 -ModifiedOnly $False [-Path <path to file or directory>]

c:\repo\adaptivecards>

violations on pipeline:

About .clang-format, refer to https://clang.llvm.org/docs/ClangFormatStyleOptions.html
Note: In this PR, pipeline check is disabled and files will be formatted in another PR.
If you want to review the difference between the formatted files and unformatted files, you can check

/azp run

/azp run

/azp run

/azp run

/azp run

/azp run


Replies are not cleaned

Describe the bug
I have been trying to completely clean out a slack channel with:
docker run -it --rm sgratzl/slack-cleaner -c "slack-cleaner --token $SLACK_TOKEN --message --channel $CHANNEL --user \"*\" --bot --as_user --perform"

But it seems to keep the replies. If I am running the same command again, I get
Failed to delete (message_not_found)->
{   u'hidden': True,
    u'last_read': u'1587533958.000300',
    u'latest_reply': u'1587533958.000300',
    u'reply_count': 32,
    u'reply_users': [u'UUFELML9X', u'U75TBYH9N'],
    u'reply_users_count': 2,
    u'subscribed': True,
    u'subtype': u'tombstone',
    u'text': u'This message was deleted.',
    u'thread_ts': u'1587490251.011600',
    u'ts': u'1587490251.011600',
    u'type': u'message',
    u'user': u'USLACKBOT'}
Failed to delete (message_not_found)->
{   u'hidden': True,
    u'last_read': u'1587472946.007400',
    u'latest_reply': u'1587472946.007400',
    u'reply_count': 1,
    u'reply_users': [u'UUFELML9X'],
    u'reply_users_count': 1,
    u'subscribed': True,
    u'subtype': u'tombstone',
    u'text': u'This message was deleted.',
    u'thread_ts': u'1587472719.006800',
    u'ts': u'1587472719.006800',
    u'type': u'message',
    u'user': u'USLACKBOT'}
Failed to delete (message_not_found)->
{   u'hidden': True,
    u'latest_reply': u'1587105514.000400',
    u'reply_count': 2,
    u'reply_users': [u'UUFELML9X'],
    u'reply_users_count': 1,
    u'subscribed': False,
    u'subtype': u'tombstone',
    u'text': u'This message was deleted.',
    u'thread_ts': u'1587100977.000100',
    u'ts': u'1587100977.000100',
    u'type': u'message',
    u'user': u'USLACKBOT'}

To Reproduce
In my case it was just a channel full of messages by a bot or user followed by replies of users.
Expected behavior
It should clean out all messages
Screenshots

hm...
https://github.com/sgratzl/slack-cleaner/blob/85f62d9789e88c6b77cbb1ebc75ec20f81d4a2e7/slack_cleaner/cli.py#L170-L176
is the relevant code for deleting replies, so interesting what is going on. I try to reproduce it.
In the meanwhile you could try using https://github.com/sgratzl/slack_cleaner2 which is a python module allowing more control about messages and replies.
code wise it would be something like (untested)
from slack_cleaner2 import SlackCleaner

s = SlackCleaner('SECRET TOKEN')
channel = s.c.CHANNEL_NAME
for msg in channel.msgs():
  for reply in msg.replies():
    reply.delete()
  msg.delete()

Yes, I also felt it was weird because I saw https://github.com/sgratzl/slack-cleaner/pull/55
I will have a look at the slack cleaner 2.

seems to be a general bug, see https://github.com/sgratzl/slack_cleaner2/issues/17

Ah ok, are you planning on including it in this package as well ?

@sgratzl  any chance you will apply the same fix here? since a number of us may not be using or coding in python

see v0.7.1


\Illuminate\Mail\Events\MessageSent is triggered regardless whether the message has been sent or  not

Laravel Version: ~5.4
PHP Version: ~7.0

I am listening \Illuminate\Mail\Events\MessageSending to prevent sending some messages. It does, but \Illuminate\Mail\Events\MessageSent is triggered.

I would do it as following
if ($this->sendSwiftMessage($message->getSwiftMessage())) {
    $this->dispatchSentEvent($message);
}

Submitted a fix here https://github.com/laravel/framework/pull/19690
thanks


Drop support for --eth-account

The --eth-account is no longer necessary because nim-ethers now supports wallets. The fact that it is still present among the CLI options and takes precedence over --eth-private-key causes confusion even amongst ourselves.
We therefore should:

hide this option from the CLI;
get rid of it, adapting tests and such so that they use fake wallets instead.

@benbierens, will that break our tests?

Removing this should not break our tests.
We're using "ETH_PRIVATE_KEY" + script to inject private key into containers.


Rework Hashicorp DSL

Allows any key to be stored in the hashicorp vault.
Allows users of the HashicorpNode to get access to the underlying SelfSignedCertificate, such that it can be formatted appropriately, then used to construct a HashicorpConnection.

splitting into smaller PRs


Private deployment scripts - Azure Container Instance did not provision during the expected time

Bicep version
v0.22.6
Describe the bug
I am trying out the new private deployment scripts functionality, but the container instance "hangs" in "creating" state which later results in a timeout (The supporting resource: 'Azure Container Instance' did not provision during the expected time.). I am using the latest api-version: Microsoft.Resources/deploymentScripts@2023-08-01.
What I was expecting was a running Azure Container Instance which executed the "Hello World!" deployment script in a private environment.
What am I missing?
To Reproduce
What I have done:

Create a virtual network with 2 subnets:

Address space: <IP_ADDRESS>/23
Subnet: my-subnet with range <IP_ADDRESS>/24
Subnet: my-container-subnet with range <IP_ADDRESS>/24 and added delegation to Microsoft.ContainerInstance/containerGroups

Created a storage account
Added a private endpoint to the 'file endpoint' and added this to my-subnet
Ran the following Bicep which includes the deploymentScripts:

param parLocation string = 'westeurope'

resource resManagedIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2022-01-31-preview' = {
  name: 'john-managed-identity'
  location: parLocation
}

resource resVirtualNetworkRef 'Microsoft.Network/virtualNetworks@2023-05-01' existing = {
  name: 'vnet-private-network-ds'

  resource resVirtualNetworkSubnetRef 'subnets' existing = {
    name: 'my-subnet'
  }

  resource resVirtualNetworkSubnetCIRef 'subnets' existing = {
    name: 'my-container-subnet'
  }
}

resource resDeploymentScript 'Microsoft.Resources/deploymentScripts@2023-08-01' = {
  name: 'test-private-networking'
  location: parLocation
  kind: 'AzurePowerShell'
  identity: {
    type: 'UserAssigned'
    userAssignedIdentities: {
      '${resManagedIdentity.id}' : {}
    }
  }
  properties: {
    storageAccountSettings: {
      storageAccountName: 'dssatestplayground'
    }
    containerSettings: {
      subnetIds: [
        {
          id: resVirtualNetworkRef::resVirtualNetworkSubnetCIRef.id
        }
      ]
    }
    azPowerShellVersion: '9.0'
    retentionInterval: 'P1D'
    scriptContent: '''
      Write-Host "Hello world!"
    '''
  }
}

Additional context
The documentation doens't say anything about private endpoints, but I was wondering if I could use a private endpoint on a storage account instead of selected networks. I have a feeling I am not even reaching the private endpoint yet, but I might be wrong.

@majastrz - is this related to the incident?
@johnlokerse -- can you share the correlation ID from the failure?

@alex-frankel I have two correlation IDs from two different deployments (with the same error):

a90b79b8-8200-4815-8d52-548a75d3bd2b
ffdb03fe-1965-4be5-b86d-04dbae64490d

@majastrz @alex-frankel I have fixed it! I was missing a crucial part of my experimental setup... namely DNS. Yet again, it's always DNS 😉.
Is there a way to get a view of what is running inside the container? The timeout error message makes sense in a way, but the error notification could be better.
Also, the documentation on private deployment scripts only has a write-up on service endpoints. Is it okay that I add the private endpoint implementation documentation?

@johnlokerse I am also facing the same problem. Can you please explain in brief how you solved this?

@RoLfBOT Yes, I forgot to implement a private DNS zone. The container instance tried to query the storage account via FQDN (name.file.core.windows.net), but could not resolve it to the private IP address which resulted in a timeout. I implemented the private DNS zone and after a redeployment, it worked. :-)
Here is the overview:

If you want to know more I wrote a blog on private deployment scripts: https://johnlokerse.dev/2022/12/04/run-powershell-scripts-with-azure-bicep/ (Private Networking)

Thanks that worked.

Closing as it looks like the problem is solved, please reopen if otherwise!

Does this still work for you @johnlokerse ? I'm trying to replicate this setup, but the container instance stays in a "waiting" status.
param location string = 'westeurope'
param storageAccountName string = 'stgscriptsecretstesthjk'
param virtualNetworkName string = 'vnet-deploymentscript-secret'
param addressPrefixPE string = '<IP_ADDRESS>/26'
param addressPrefixAci string = '<IP_ADDRESS>/26'
param addressPrefixVnet string = '<IP_ADDRESS>/24'
param time string = replace(utcNow(), ':', '-')

resource virtualNetwork 'Microsoft.Network/virtualNetworks@2023-05-01' = {
  name: virtualNetworkName
  location: location
  properties: {
    addressSpace: {
      addressPrefixes: [
        addressPrefixVnet
      ]
    }
    subnets: [
      {
        name: 'privateendpoint'
        properties: {
          addressPrefix: addressPrefixPE
        }
      }
      {
        name: 'aci'
        properties: {
          addressPrefix: addressPrefixAci
          serviceEndpoints: [
            {
              service: 'Microsoft.Storage'
            }
            {
              service: 'Microsoft.KeyVault'
            }
          ]
          delegations: [
            {
              name: 'Microsoft.ContainerInstance.containerGroups'
              properties: {
                serviceName: 'Microsoft.ContainerInstance/containerGroups'
              }
            }
          ]
        }
      }
    ]
  }
}

resource storageAccount 'Microsoft.Storage/storageAccounts@2023-01-01' = {
  name: storageAccountName
  location: location
  sku: {
    name: 'Standard_LRS'
  }
  kind: 'StorageV2'
  properties: {
    allowBlobPublicAccess: false
    minimumTlsVersion: 'TLS1_2'
    allowSharedKeyAccess: false
    publicNetworkAccess: 'Disabled'
  }
}

resource userAssignedIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = {
  name: 'uai-deploymentscript'
  location: location
}

resource roleAssignmentStorageAccount 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
  name: guid(resourceGroup().id, storageAccount.name, '69566ab7-960f-475b-8e7c-b3118f30c6bd')
  scope: storageAccount
  properties: {
    principalId: userAssignedIdentity.properties.principalId
    roleDefinitionId: resourceId('Microsoft.Authorization/roleDefinitions', '69566ab7-960f-475b-8e7c-b3118f30c6bd')
    principalType: 'ServicePrincipal'
  }
}

resource deploymentScript 'Microsoft.Resources/deploymentScripts@2023-08-01' = {
  name: 'deploymentscript'
  location: location
  identity: {
    type: 'UserAssigned'
    userAssignedIdentities: {
      '${userAssignedIdentity.id}': {}
    }
  }
  kind: 'AzurePowerShell'
  properties: {
    forceUpdateTag: time
    azPowerShellVersion: '6.4'
    retentionInterval: 'P1D'
    cleanupPreference: 'Always'
    storageAccountSettings: {
      storageAccountName: storageAccountName
    }
    containerSettings: {
      subnetIds: [
        {
          id: '${virtualNetwork.id}/subnets/aci'
        }
      ]
    }
    scriptContent: "Hello World!"
  }
}

And this is my networking setup:

I'd suggest a small modification over @johnlokerse 's solution. Instead of creating a private endpoint, add service endpoint to the subnet that is delegated to the Azure Container Instances resource provider. This will ensure that the container instance is able to resolve the storage file share public IP. The storage account, as per above architecture, need not be integrated with the vent. This resolves any race conditions when setting up the private endpoint.

@Lucas-c-B did you get it working?

@Lucas-c-B did you get it working?

Unfortunately I did not

I did notice now that you have allowSharedKeyAccess: false, I think it must be enabled.
Does it still work for you @johnlokerse?

I did notice now that you have allowSharedKeyAccess: false, I think it must be enabled.
Does it still work for you @johnlokerse?

The identity should access the storage account through the RBAC role assignment, so I think it should also work with allowSharedKeyAccess: false

Sorry for the extremely late reply @Lucas-c-B and thanks for the ping @OskarKlintrot. The private endpoint solution works for me and running it in productions. I am not using service endpoints as seen in the example Lucas posted here. Also, the posted solutions seems like a mix of using service endpoints and wanting it to be fully private (property: publicNetworkAccess: disabled). This not seems correct and can be the reason why the container is in waiting state.
Can you try and add the networkAcls object and set publicNetworkAccess to enabled?
resource storageAccount 'Microsoft.Storage/storageAccounts@2023-01-01' = {
  name: storageAccountName
  location: location
  sku: {
    name: 'Standard_LRS'
  }
  kind: 'StorageV2'
  properties: {
    allowBlobPublicAccess: false
    minimumTlsVersion: 'TLS1_2'
    allowSharedKeyAccess: false
    publicNetworkAccess: 'Enabled'
    networkAcls: {
      bypass: 'AzureServices'
      defaultAction: 'Deny'
      virtualNetworkRules: [
        {
          id: virtualNetwork.properties.subnets[1].id
          action: 'Allow'
          state: 'Succeeded'
        }
      ]
    }
  }
}

If you don't want to use service endpoint, but rather private endpoint then take a look this snippet: https://github.com/johnlokerse/bicep-snippets/blob/main/privateDeploymentScript/privateDeploymentScripts.bicep
(Also, make sure mcr.microsoft.com is whitelisted in your firewall if you have any)

@johnlokerse Thanks! I turned on the public network access to be able to move forward, that worked (I did turn on allowSharedKeyAccess in the same commit though, but from my understand that shouldn't effect anything). The only difference I can see between you snippet and my bicep is that you add an A record to the private DNS zone, shouldn't it be auto registered? I don't have to register anything else that uses private DNS zones.

@OskarKlintrot Glad it works 😃. What happens when you set the property publicNetworkAccess to disabled it removes the possibility to connect to the data plane endpoint of the storage account from the internet. You are then required to configure private endpoints for the storage account.
If you are using service endpoints you are connecting through the public interface of the storage account, but over the Microsoft backbone. This means you are coming from a public network. This is why the example given by Microsoft shows that you need to have publicNetworkAccess to enabled including an 'allow' from virtual network configuration as seen below:

Source [Microsoft Learn - Access a private virtual network from a Bicep deployment script]https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/deployment-script-vnet

@johnlokerse I want to use private endpoint, so my service endpoints were wrong indeed. However, I still couldn't make it work after changing this. The container instance again doesn't provision in time.

@Lucas-c-B Have you tried to enable SAK? I teared down my env to set it up again but this time it didn't work. After I enabled SAK it did work again. I guess I must have had that turned on by mistake last time I tried.

tried using private endpoint + SAS key enabled : works
tried using private endpoint + SAS key disabled: container failed to provision in time

@va-chopra Same for me, I must use PE and SAS enabled for my deployment script to work.


Image not shown in some pages on MANGA contents

Link to PDF file (or attach file here):
AN_1000126_p100.pdf
Configuration:

Web browser and its version: doesn't seem browser specific
Operating system and its version: any
PDF.js version: current
Is an extension:

Steps to reproduce the problem:

Open the attached PDF

What is the expected behavior? (add screenshot)

What went wrong? (add screenshot)

Even it's MANGA contents, images aren't shown.
(Most pages are OK, but it happens with this page only.)
Link to a viewer (if hosted on a site other than mozilla.github.io/pdf.js or as Firefox/Chrome extension):

After PR #14175, the images are now very faintly visible.


Inserts do not set up index correctly

This issue was in the pre-badger code (<= dc5b9e3503ecba809eff2436d1ee69a4b171f36b). I'm putting it here for anyone else who wants to continue using the old storage.  Note that if you're doing this, you should also cherry-pick the other indexing bug fix, e736362eb15e75437a1404458148678902a022d4.
The behavior is that Insert() of documents after the first would result in an incorrect index since it always started with an offset of 0 regardless of how many documents were already in the collection. Attempts to access document N>0 would always get the wrong document. This usually resulted in bad JSON formatting errors, either unexpected end of JSON or invalid syntax. The patch is:
diff --git a/storage.go b/storage.go
--- a/storage.go
+++ b/storage.go
@@ -260,7 +260,7 @@ func (s *storageImpl) Insert(collection
                return err
        }

-       pointers, err := appendDocs(&collectionFile{File: tempFile, size: 0}, docs)
+       pointers, err := appendDocs(&collectionFile{File: tempFile, size: coll.file.size}, docs)
        if err != nil {
                return err
        }

@ostafen, I'm submitting and then closing; hopefully others searching for a fix will find this.

Thank you for pointing this out :+1:


Docker Desktop "feedback" button should go to issue type choice page

Bug Report
We have a few issue templates to guide the user and help categorize new issues. By default when opening an issue in GitHub, the user is brought to https://github.com/vmware-tanzu/community-edition/issues/new/choose where they can choose which issue type to open.
The "Feedback" button inside the Docker Desktop extension UI goes directly to https://github.com/vmware-tanzu/community-edition/issues/new so the user is given a blank issue page to fill out without any template to help guide them. We should have this link go to the same "choose" link that they would get if they were opening an issue through GitHub.

Is there a way to create a specific issue or to automagically create a template so that issues get triaged as coming from Docker Desktop?


Issue flashing Tidbyt Gen 2

@matslina I see that you recently added support for gen 2 devices, however it appears there are some errors popping up when running the ... pio run --environment tidbyt-gen2 --target upload command. I am attaching a screenshot of what I am seeing, and it does seem to link back to some diffs that were recently made.

Ah, needed to init the git sub-modules.


Paddle-2.0 fix zh doc

修复 pow， reciprocal，logical_* 的中文文档。
增加 multiply 的中文文档。
删除 elementwise_mul 的中文文档。

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.Paddle CI seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


Please add the badge for Weblate translations to your adapter repository

Please add the badge for Weblate translations to your adapter repository:
https://github.com/ioBrokerTranslator/doc#add-badge-to-readmemd

done for github version. Will be published with next version

As visible at github (head) this issue can be closed.


parse triples for object-like access

Just a thought - parse triples from graph for object-like access?
e.g. foo has foaf:knows triples pointing to goober and foo/bar.  What if this was available at:
# pseudo code
[In] foo.rdf.triples.foaf.knows
[Out] [rdflib.term.URIRef('http://localhost:8080/rest/goober'),rdflib.term.URIRef('http://localhost:8080/rest/foo/bar')]

With the handy SimpleNamespaces, not sure how difficult this would be, and might be quite handy.

Closes: https://github.com/ghukill/pyfc4/commit/5b0b0122d15fe4232b66d1922aaa755c78661d60


Remove six dependency

The six dependency should not be needed anymore, as the project only supports python3 \o/

Yup! I'm working on removing six, a little at a time. I think the only remaining uses are ensure_bytes() and ensure_str().


Mocktail Throws Error: type 'Null' is not a subtype of type 'Future'

When using mocktail to mock the configure method from the FlutterBackgroundService package, the mocked method unexpectedly returns Null, leading to the following error during testing:
type 'Null' is not a subtype of type Future<bool>
test/widget_test.dart 6:7                 MockFlutterBackgroundService.configure
widget_test.dart:6
package:experiment/experiment.dart 10:24  Experiment.initService
test/widget_test.dart 29:37               main.<fn>

Code Example:
Here is the relevant code:
Main Code:
import 'package:flutter_background_service/flutter_background_service.dart';

class Experiment {
  FlutterBackgroundService _service;

  Experiment({service}) : _service = service;

  Future<bool> initService() async {
    var running =
        await _service.configure(iosConfiguration: IosConfiguration(), androidConfiguration: AndroidConfiguration(onStart: (_) {}, isForegroundMode: false));

    return running;
  }
}

Test Case:
import 'package:experiment/experiment.dart';
import 'package:flutter_background_service/flutter_background_service.dart';
import 'package:flutter_test/flutter_test.dart';
import 'package:mocktail/mocktail.dart';

class MockFlutterBackgroundService extends Mock implements FlutterBackgroundService {}

void main() {
  late Experiment experiment;
  late MockFlutterBackgroundService mockFlutterBackgroundService;

  setUp(() {
    mockFlutterBackgroundService = MockFlutterBackgroundService();
    experiment = Experiment(service: mockFlutterBackgroundService);

    when(
      () => mockFlutterBackgroundService.configure(
        androidConfiguration: AndroidConfiguration(onStart: (_) {}, isForegroundMode: false),
        iosConfiguration: IosConfiguration(),
      ),
    ).thenAnswer(
      (_) async => false,
    );
  });

  test(
    "verify initService method working as Expected or not",
    () async {
      var result = await experiment.initService();

      expect(result, isFalse);
    },
  );
}

I resolved the issue by introducing FakeAndroidConfiguration and FakeIosConfiguration classes for mocking the configurations. These classes were registered with registerFallbackValue, and now the test works as expected.


Add Vote-Kick

Sorry for opening this in issue, i have no idea how i can contact you.
If you want, could you add a vote-kick/vote-ban system? i really think this mod is useful, i use it sometimes and a lot of players enjoy it.
I would like to host the mod on a machine, but if i go afk, cheaters or people who harass others can't be banned.
The vote-kick would work like this:
/votekick PlayerName Reason
a message will appear in chat with "Player1 wants to kick Player2 for: he's cheating, i saw him flying!" use /yes if you agree.
if half the players agree to kick the player with /yes, he will be kicked.
Again, sorry for proposing this in the "issues" section, Thanks!

 I'll see what I can do tomorrow, when I have some time

https://github.com/Danilo1301/crab-game-server-mod/releases/tag/v2.3


Add functions and operators to serialize and deserialize dictionaries

Changes allow you to use <~~ and ~~> to serialize dictionaries of serializable objects:
{ ... "colors":{   "red":{"red":255, "green":0, "blue":0},   "purple":{"red":255, "green":0, "blue:255} } }
can be serialized using:
`
let colors : [String:Color]
init?(json:JSON) {
...
colors = "colors" <~~ json
...
}
`
obviously assuming an appropriate declaration of Color as Decodable.  Also works in reverse to serialize [String:Encodable]

nice - i'll need to play with this a bit myself this weekend and add some tests. if i surface any issues, i'll respond here

this is cool - thanks @dwb357 :smile:


Update NH swagger to move Sku parameter from Resource to NamespaceResource

Regarding bug: https://dev.azure.com/msmobilecenter/NotificationHub/_workitems/edit/78678
Have moved Sku parameter from "Resource" to "NamespaceResource".

/azp run automation - sdk

Can one of the admins verify this patch?

/azp run automation - sdk

/azp run automation - sdk

@marstr  @markcowl can you please review the PR and advise what should be done about Python breaking change? Is is an FYI or do we need to do anything in this PR?

@stankovski, I just took a look and this is a real breaking change for most (all?) of the SDK, not just Python. While this is a bug in our Swagger, the fix would be more painful than the problem.
We should remember this when we're considering adding a new API Version in the future.


Block: Add cache-related options for block devices

Add block_device_cache_set, block_device_cache_direct and
block_device_cache_noflush.
They are cache-related options for block devices that are described in
https://github.com/qemu/qemu/blob/master/qapi/block-core.json.
block_device_cache_direct denotes whether use of O_DIRECT (bypass the host
page cache) is enabled.  block_device_cache_noflush denotes whether flush
requests for the device are ignored.
The json said they are supported since 2.9.
So add block_device_cache_set to control the cache options set to block
devices or not.  It will help to support the old version qemu.

Also directly related to #810 I think. /cc @amshinde for review/input as well.

/test

/test

/test

/test

/test

/retest

/test

http://jenkins.katacontainers.io/job/kata-metrics-runtime-ubuntu-16-04-PR/196/console
18:44:27 go build -ldflags "-X main.sysBaseFile=" -o checkmetrics
18:44:29 install -D checkmetrics /usr/bin/checkmetrics
18:44:29 ~/workspace/workspace/kata-metrics-runtime-ubuntu-16-04-PR/go/src/github.com/kata-containers/tests
18:44:29 time="2018-12-06T10:44:29Z" level=warning msg="Failed Maxval (3.045000 < 3.103300) for [boot-times]"
18:44:29 time="2018-12-06T10:44:29Z" level=warning msg="Check for [boot-times] failed [Failed]"
18:44:29 time="2018-12-06T10:44:29Z" level=warning msg=" with [[*F* boot-times 95.0% 107.0% 105.0% 10.0% 104.0% 110.7% 6.5% 1.5% 20]]"
18:44:29 time="2018-12-06T10:44:29Z" level=warning msg="Overall we failed"
18:44:29
18:44:29 Report Summary:
18:44:29 +-----+----------------------+-------+--------+--------+-------+--------+--------+------+------+-----+
18:44:29 | P/F |         NAME         |  FLR  |  MEAN  |  CEIL  |  GAP  |  MIN   |  MAX   | RNG  | COV  | ITS |
18:44:29 +-----+----------------------+-------+--------+--------+-------+--------+--------+------+------+-----+
18:44:29 | *F* | boot-times           | 95.0% | 107.0% | 105.0% | 10.0% | 104.0% | 110.7% | 6.5% | 1.5% |  20 |
18:44:29 | P   | memory-footprint     | 85.0% | 85.1%  | 115.0% | 30.0% | 85.1%  | 85.1%  | 0.0% | 0.0% |   1 |
18:44:29 | P   | memory-footprint-ksm | 95.0% | 100.9% | 105.0% | 10.0% | 100.9% | 100.9% | 0.0% | 0.0% |   1 |
18:44:29 +-----+----------------------+-------+--------+--------+-------+--------+--------+------+------+-----+
18:44:29 Fails: 1, Passes 2
18:44:29 Failed
18:44:29 checkmetrics FAILED (1)

This pr doesn't change the default behavior of runtime.

Upon the metrics test, this PR

increase 7% of the boot time
and save 15% of the memory footprint (w/o ksm)

However, it is a PR just for three new configuration items and does not affect the default behavior. I think it is the environment issue as stated by Graham.
Let's merge it.

Hi @teawater @gnawux - afaict, this does not look to change the default runtime behaviour - but, when I track what looks to be a performance regression (noted by the metrics CI), and run the metrics report along with a git-bisect on the runtime, it all points to this PR making the difference.
The boot time has gone up, and maybe the memory footprint had dropped a little.
Can you double check for me this code and the related govmm code to just check you cannot see any default behaviour change?
I'll carry on digging on Monday. My best guess is we somehow disabled some cacheing for one or more of our mounts and filesystems... it's hard to debug as this is all done via qmp, so does not show up on the qemu commandline ;-)
/cc @markdryan

@grahamwhaley This PR doesn't change the default qemu boot args, nor the default qmp command to hotplug a new disk. I don't think it can affect boot time either.


GitHub: Invite member

Who should be invited to Catena-X ng
GitHub User:
https://github.com/SteDue
Vouching person:
Steffen Düring BMW
@mkanal

Hi @mkanal! Invitation sent. After accepted, the maintainer of the team can onboard the user.


Requesting to show console logs only when added --debug

Requesting to show logs only when executed command with argument --debug
gidevice -u $UDID  xctest com.facebook.WebDriverAgentRunner.xctrunner --debug

XCTest output is the same as Xcode
I just think it is much more useful

it will be great to have two options - with debug logs and without logs(just get a message connected url)

if didn't use to launch WDA?
nothing to display?

I think that's good

v0.4.2

$ go run main.go xctest com.leixipaopao.WebDriverAgentRunner.xctrunner --contains="ServerURLHere->" --contains="Running tests..." --contains="Built at"

2021-04-28 23:15:56.229402+0800 WebDriverAgentRunner-Runner[2436:498709] Running tests...
2021-04-28 23:15:57.709613+0800 WebDriverAgentRunner-Runner[2436:498709] Built at Apr 24 2021 00:16:35
2021-04-28 23:15:57.719480+0800 WebDriverAgentRunner-Runner[2436:498709] ServerURLHere->http://<IP_ADDRESS>:8100<-ServerURLHere

works perfect. thanks


Bug fix for Issue 2334

Contains fix for Issue 2334, changed probability.py, updated the string formatting for python 3.

Thanks @EeshitaBiswas


Race condition with get_or_create on TaggedItem

When adding tags to items, the TaggedItem model is used for the intermediate table. In particular, get_or_create is used. There are no unique restrictions on this table, so get_or_create is vulnerable to race conditions if two threads/processes/servers try and add a tag to one item at the same time.
To fix this, I think it would work to add a unique_together contrainst on (tag, content_type, object_id) on the TaggedItem model. That way, the second creation will not happen (and get_or_create will recover correctly).

Thanks for pointing this out! If you create a pull request for this, I'd be happy to merge it

I filed a partial PR for this, but I'm having some trouble with migrations. I didn't see any examples or documentation about migrations. Can you provide some help in that PR?


Tutorial #1

Create a tutorial for APF data that shows the progression of 1D data products, starting from the reduced 1D spectra.
Zoe
Wavelength solution in constant RV space.
Shifted to rest frame.

complete


WHATSAPPER | Fix wweb.js issue

see issue
https://github.com/alexbelgium/hassio-addons/issues/1358

Ready to merge? Thanks!


pkg/report: tune get_maintainer.pl arguments

We currently invoke get_maintainer.pl with default args.
@wdebruij made a point that since syzbot is an automated system, it could use a slightly different mode:
https://groups.google.com/d/msg/syzkaller/QUzeqnczVw0/Xj0ws8I2CwAJ
I.e. the default args are for humans, the new special mode is for systems that report bugs automatically (has less assurance that the file is correct).
Proposals include using:
--nom --nor --git-min-percent 20

or:
-m --git-max-maintainers=2

To assess the options we could dump results for all kernel files for several combinations and diff them with the current results (would be handy to upload to github/gerrit for side-by-side diffs).

Using --nom looks somewhat questionable:
https://groups.google.com/d/msg/syzkaller/QUzeqnczVw0/6SnKI6h2CwAJ

FTR another bug with 50 emails:
https://groups.google.com/g/syzkaller/c/oiJQ-KyRRAA/m/JEjxJJfDAwAJ
But the bisected commit CC has 41 emails, so this mostly looks like WAI

I'm not sure it's a good idea to mail the people in Cc: on a bisected commit.  Often the people listed in Cc weren't really involved in the patch at all; someone just though they might be interested.  (And as we can see, some people use very long Cc lists.)  It's much more important to send to the Author and people who left tags indicating active involvement like Reviewed-by, Signed-off-by, Acked-by.
This is made worse by the fact that syzbot puts everyone on the To: line and doesn't actually use Cc: itself, so it's unclear how responsible each recipient is.  Ideally the most responsible people (e.g. Author, maintainer and list for affected code) would be on To:, and additional people or lists would be on Cc:.

Good question. I guess there is no single right answer. I am a bit afraid of the following cases: (1) if we include just 1-2 people they may be on vacation; left kernel; busy; don't care; etc; (2) emails may have changed (switch companies); (3) some lists are not read (lkml, netdev), (4) maintainer may not have been added Acked-by/Reviewed-by tags, or they simply came a bit later. On the other hand if one adds people to Cc I would expect them to be relevant and knowledgeable about the subsystem, so they are already on CC from get_maintainers for the file, or they need to be on CC.
But let's try removing Cc (esp if you take responsibility for decisions :)).

I think mailing people in Cc: on a bisected commit does more harm than good overall, especially given that sometimes developers use very long Cc: lists on patches and syzbot bisections are still often unreliable.  So yes, I support not adding those addresses to syzbot recipients after a bisection.
Separately, I'd also suggest that the syzbot recipent lists:

Put people more likely to be responsible on To: and people less likely to be responsible on Cc:.  I.e., maintainer(s), supporter(s), subsystem list(s), and bisected commit author and sign-off-er(s) on To:, and other people and lists such as designated reviewers, linux-kernel, and syzkaller-bugs on Cc:.  This would help people understand who is most likely to be responsible for the bug.  (Edge case: if the resulting To: list is empty, just put everyone in To: like the current situation.)
For human recipients, include people's names like Joe Example<EMAIL_ADDRESS>instead of just an email address like<EMAIL_ADDRESS> This is by far the more common convention, and it makes it easier for a human to understand who the email was sent to.

Regarding the get_maintainer.pl arguments, I suggest just adding --git-min-percent=15 for now to cut down on the noise a bit.  I don't think --nom or --nor makes sense, though I think reviewers should normally go on Cc: line rather than To: as mentioned above.  I'm not sure what --git-max-maintainers does exactly.

Some fun with get_maintainers.pl:
https://twitter.com/dvyukov/status/1204356404875874305

Checked 2 files that come up earlier:
$ ./scripts/get_maintainer.pl --git-min-percent 5  drivers/net/veth.c | grep -v "open list" | wc -l
16
$ ./scripts/get_maintainer.pl --git-min-percent 15 drivers/net/veth.c | grep -v "open list" | wc -l
13
$ ./scripts/get_maintainer.pl --git-min-percent 20 drivers/net/veth.c | grep -v "open list" | wc -l
13

$ ./scripts/get_maintainer.pl --git-min-percent 5  fs/proc/task_mmu.c | grep -v "open list" | wc -l
12
$ ./scripts/get_maintainer.pl --git-min-percent 15 fs/proc/task_mmu.c | grep -v "open list" | wc -l
8
$ ./scripts/get_maintainer.pl --git-min-percent 20 fs/proc/task_mmu.c | grep -v "open list" | wc -l
5

and looked at actual people being added/removed. Using 15 looks sane.

I've addressed 2 simpler and more critical issues. Filed 2 other issues for larger proposals.


Error during make flash with esp-idf-v3.1.1

Environment

Development Kit: ESP32
IDF version: v3.1.1
Build System: Make
Compiler version : (crosstool-NG crosstool-ng-1.22.0-80-g6c4433a) 5.2.0
Operating System:  macOS
Power Supply: USB

Problem Description
When trying to do make flash getting following errors during compilation:
CC build/app_trace/app_trace.o
In file included from /Users/arun/esp/esp-idf-v3.1.1/components/soc/esp32/include/soc/soc.h:20:0,
                 from /Users/arun/esp/esp-idf-v3.1.1/components/app_trace/app_trace.c:160:
/Users/arun/esp/esp-idf-v3.1.1/components/soc/esp32/include/soc/io_mux_reg.h: In function 'PIN_PULLUP_DIS':
/Users/arun/esp/esp-idf-v3.1.1/components/esp32/include/esp_assert.h:24:13: error: implicit declaration of function 'assert' [-Werror=implicit-function-declaration]
             assert(#MSG && (CONDITION));                                                                    \
             ^
/Users/arun/esp/esp-idf-v3.1.1/components/soc/esp32/include/soc/soc.h:146:38: note: in expansion of macro 'TRY_STATIC_ASSERT'
 #define ASSERT_IF_DPORT_REG(_r, OP)  TRY_STATIC_ASSERT(!IS_DPORT_REG(_r), (Cannot use OP for DPORT registers use DPORT_##OP));
                                      ^
/Users/arun/esp/esp-idf-v3.1.1/components/soc/esp32/include/soc/soc.h:177:13: note: in expansion of macro 'ASSERT_IF_DPORT_REG'
             ASSERT_IF_DPORT_REG((_r), REG_CLR_BIT);                                                                    \
             ^
/Users/arun/esp/esp-idf-v3.1.1/components/soc/esp32/include/soc/io_mux_reg.h:95:5: note: in expansion of macro 'REG_CLR_BIT'
     REG_CLR_BIT(PIN_NAME, FUN_PU);
     ^
cc1: some warnings being treated as errors
make[1]: *** [app_trace.o] Error 1
make: *** [component-app_trace-build] Error 2

Expected Behavior
No error during compilation (this error comes from code belonging to esp-idf)
Actual Behavior
See above
Steps to reproduce

Download esp-idf-v3.1.1.zip that includes submodules and unzip
cd esp-idf-v3.1.1
git submodule add https://github.com/siara-cc/esp32-idf-sqlite3 components/esp32-idf-sqlite3
git submodule add https://github.com/siara-cc/esp32-idf-sqlite3-examples examples/esp32-idf-sqlite3-examples
cd examples/esp32-idf-sqlite3-examples/spiffs
make flash

sdkconfig.txt

This is probably because the component you are including into the build, esp32-idf-sqlite3, provides a file called assert.h, which shadows assert.h provided by newlib.
In that file (https://github.com/siara-cc/esp32-idf-sqlite3/blob/master/include/assert.h), if NDEBUG is not defined, there is an include <assert.h> which will include the same file. In this case, assert macro ends up being undefined.
https://github.com/siara-cc/esp32-idf-sqlite3/blob/2527c3c630daa423dcb64985289a53510f0b1102/include/assert.h#L6
You can replace that include with include_next, then it should work as you have intended, i.e. include the "real" assert.h file.

The problem stems from newlib's definition of assert which is # define assert(__e) ((void) sizeof(__e)) when NDEBUG is defined.  This expects __e to be valid when NDEBUG is defined, which creates the problem with the way sqlite3.c uses assert macro (refer to line number 106235).
I am inclined to think that newlib's assert should not expect __e to be valid when NDEBUG is defined.  In this case I am removing the component's assert.h and handling by modifying sqlite3.c and closing this issue, but I am mentioning this because it may arise with other 3rd party components in future.  Also I am not sure if the components include files should affect other components.
Thank you for looking into it.

Linking same issue faced with ESP32 Arduino SDK: https://github.com/espressif/esp-idf/issues/2758

Also I am not sure if the components include files should affect other components

Component may have private include directories (which will be added to the compiler command line when compiling this component only), and public include directories (exported to other components). You can create a directory, say include_private, in your component, place assert.h there, and add this directory to COMPONENT_PRIVINCLUDEDIRS. Then this modified assert.h will only be used by your components

I added
COMPONENT_PRIV_INCLUDEDIRS=private_include
to component.mk and moved assert.h and osapi.h to private_include.
It works fine - Thanks!!


creative/inferred mode not using canonical_predicate from BioLink Model

BioLink Model specifies a canonical_predicate property, which is meant to indicate the direction of the edge that should be returned in TRAPI results.  For example, treats is listed with canonical_predicate: true, and treated_by lists inverse: treats.   See snippets from BioLink Model below.
  treats:
    aliases: [ 'is substance that treats', 'indicated for' ]
    is_a: ameliorates
    description: >-
      holds between a therapeutic procedure or chemical entity
      and a disease or phenotypic feature that it is used to treat
    domain: chemical or drug or treatment
    range: disease or phenotypic feature
    annotations:
      canonical_predicate: true
      opposite_of: contraindicated for

  treated by:
    is_a: is ameliorated by
    description: >-
      holds between a disease or phenotypic feature and a therapeutic
      process or chemical entity that is used to treat the condition
    domain: disease or phenotypic feature
    range: chemical or drug or treatment
    # range: chemical or drug or treatement
    in_subset:
      - translator_minimal
    inverse: treats

In the context of https://github.com/NCATSTranslator/testing/issues/222, BTE's standard mode is returning edges using biolink:treats, but creative mode is reversing the edge and using biolink:treated_by.
"normal" mode: https://arax.ncats.io/?r=14e4e94c-3a77-4ad6-ac90-8df7b1e34df5

creative/inferred mode: https://arax.ncats.io/?r=ff949174-c6c8-4de9-81c7-0da9d030ee75

Huh, I thought this was going to be a surprisingly complicated bug within the handling code, but it turn out it's just that our query templates aren't in the canonical direction.
I'm not 100% sure if there's an issue with BTE in general not respecting canonical_predicate, or if the query graph directionality (in this case in our templates) is taking precedence in the response formatting. I don't see any references to the property in our codebase...

@colleenXu could you make a PR fixing the templates? I'd do it but I'm afraid of breaking them >.<

@andrewsu do we want the canonical direction to take precedence over whatever direction is presented in a query, generally speaking?

Yes, AFAIK we return in query-direction, so if we're only queried in canonical direction, we should be fine (possibly worth making a low-priority issue).

The templates were specified in "non-canonical-predicates" because the QGraph was then easier to read (start with a Disease and find Chemicals). I am editing the creative-mode templates now to make them fit "canonical directions".

Closing for now since the linked PR addresses this for now. And the related issue is low priority.


Raise warning and return None on invalid WKB instead of raising exception

Resolves #320
For reasons stated in #320, it is problematic to raise exceptions when deserializing invalid geometries from formats such as WKB.  This provides a way to gracefully read those in, raise a warning, and allow the user to filter them out instead of failing hard.
This adds a new enum value PGWARN_GEOS_EXCEPTION which is used to raise the GEOS exception as a warning instead of an exception.

A couple things to consider:

I'm not sure that ignore_invalid is the best name for this parameter.  Is there something better?
WKT and WKB warning types are separate enum values now but this doesn't add much value.  These could be combined into a single enum value if we don't think these need to be kept separate.

+1 on making it a string option

This can be done now easily by only allowing raise and ignore

We could maybe also further use the terminology of np.seterr and use "raise" and "warn", which keeps the possibility open to add a "ignore" (which would not show the warning)?
About naming, I think I find on_invalid a bit more descriptive than errors.

Latest changes make this a string parameter and add an ignore option.


Swift 5.1 compatibility

The networking-related classes from Foundation have recently been moved into a separate FoundationNetworking module (see also here for discussion) and thus broken compatibility with Swift 5.1. Unfortunately, Vapor 2 also uses the old imports, thus we have to use Vapor 3 to build under Swift 5.1.
This PR merges the development branch and adds the required FoundationNetworking imports.

The branch builds successfully, but the WebSocket connecting to Discord's gateway repeatedly closes:
2019-08-27 23:23:42.787 D2[15183:b9f712c0] LOG: DiscordClient: Connecting
2019-08-27 23:23:43.019 D2[15183:b9f10700] LOG: DiscordWebSocketable: Connecting to wss://gateway.discord.gg/?v=6&encoding=json, shard: 0
2019-08-27 23:23:43.019 D2[15183:b9f10700] LOG: DiscordWebSocketable: Attaching WebSocket with hostname gateway.discord.gg, port nil, path: /, shard: shard: 0
2019-08-27 23:23:43.424 D2[15183:b9f10700] LOG: DiscordWebSocketable: Websocket connected, shard: shard: 0
2019-08-27 23:23:43.424 D2[15183:b9f10700] LOG: DiscordEngine: Sending handshake, shard: 0
2019-08-27 23:23:43.589 D2[15183:b9f712c0] LOG: DiscordClient: Handling ready
2019-08-27 23:23:43.692 D2[15183:b9f712c0] LOG: DiscordClient: Handling guild create
2019-08-27 23:23:43.701 D2[15183:b9f10700] LOG: DiscordWebSocketable: WebSocket did close, shard: 0;
2019-08-27 23:23:43.702 D2[15183:b9f10700] LOG: DiscordEngine: Disconnected, shard: 0, reason: unknown
2019-08-27 23:23:43.702 D2[15183:b9f10700] LOG: DiscordEngine: Trying to resume gateway session on shard: 0
2019-08-27 23:23:43.702 D2[15183:b9f10700] LOG: DiscordWebSocketable: Connecting to wss://gateway.discord.gg/?v=6&encoding=json, shard: 0
2019-08-27 23:23:43.702 D2[15183:b9f10700] LOG: DiscordWebSocketable: Attaching WebSocket with hostname gateway.discord.gg, port nil, path: /, shard: shard: 0
2019-08-27 23:23:44.138 D2[15183:b9f10700] LOG: DiscordWebSocketable: Websocket connected, shard: shard: 0
2019-08-27 23:23:44.138 D2[15183:b9f10700] LOG: DiscordEngine: Sending resume, shard: 0
2019-08-27 23:23:44.379 D2[15183:b9f10700] LOG: DiscordWebSocketable: WebSocket did close, shard: 0;
2019-08-27 23:23:44.379 D2[15183:b9f10700] LOG: DiscordEngine: Disconnected, shard: 0, reason: unknown
2019-08-27 23:23:44.379 D2[15183:b9f10700] LOG: DiscordEngine: Already trying to resume or closed, ignoring
2019-08-27 23:23:53.703 D2[15183:b9f10700] LOG: DiscordWebSocketable: Connecting to wss://gateway.discord.gg/?v=6&encoding=json, shard: 0
2019-08-27 23:23:53.703 D2[15183:b9f10700] LOG: DiscordWebSocketable: Attaching WebSocket with hostname gateway.discord.gg, port nil, path: /, shard: shard: 0
2019-08-27 23:23:54.061 D2[15183:b9f10700] LOG: DiscordWebSocketable: Websocket connected, shard: shard: 0
2019-08-27 23:23:54.061 D2[15183:b9f10700] LOG: DiscordEngine: Sending resume, shard: 0
2019-08-27 23:23:57.335 D2[15183:a97a0700] LOG: DiscordEngine: Invalid session received. Invalidating session
2019-08-27 23:23:57.335 D2[15183:a97a0700] LOG: DiscordEngine: Sending handshake, shard: 0
2019-08-27 23:23:57.501 D2[15183:b9f712c0] LOG: DiscordClient: Handling ready
2019-08-27 23:23:57.605 D2[15183:b9f712c0] LOG: DiscordClient: Handling guild create
2019-08-27 23:23:57.608 D2[15183:b9f10700] LOG: DiscordWebSocketable: WebSocket did close, shard: 0;
2019-08-27 23:23:57.608 D2[15183:b9f10700] LOG: DiscordEngine: Disconnected, shard: 0, reason: unknown
2019-08-27 23:23:57.608 D2[15183:b9f10700] LOG: DiscordEngine: Trying to resume gateway session on shard: 0
2019-08-27 23:23:57.608 D2[15183:b9f10700] LOG: DiscordWebSocketable: Connecting to wss://gateway.discord.gg/?v=6&encoding=json, shard: 0
2019-08-27 23:23:57.609 D2[15183:b9f10700] LOG: DiscordWebSocketable: Attaching WebSocket with hostname gateway.discord.gg, port nil, path: /, shard: shard: 0
2019-08-27 23:23:57.984 D2[15183:b9f10700] LOG: DiscordWebSocketable: Websocket connected, shard: shard: 0
2019-08-27 23:23:57.984 D2[15183:b9f10700] LOG: DiscordEngine: Sending resume, shard: 0
2019-08-27 23:23:58.211 D2[15183:b9f10700] LOG: DiscordWebSocketable: WebSocket did close, shard: 0;
2019-08-27 23:23:58.212 D2[15183:b9f10700] LOG: DiscordEngine: Disconnected, shard: 0, reason: unknown
2019-08-27 23:23:58.212 D2[15183:b9f10700] LOG: DiscordEngine: Already trying to resume or closed, ignoring
2019-08-27 23:24:03.704 D2[15183:b9f10700] LOG: DiscordWebSocketable: Connecting to wss://gateway.discord.gg/?v=6&encoding=json, shard: 0
2019-08-27 23:24:03.704 D2[15183:b9f10700] LOG: DiscordWebSocketable: Attaching WebSocket with hostname gateway.discord.gg, port nil, path: /, shard: shard: 0
2019-08-27 23:24:04.078 D2[15183:b9f10700] LOG: DiscordWebSocketable: Websocket connected, shard: shard: 0
2019-08-27 23:24:04.079 D2[15183:b9f10700] LOG: DiscordEngine: Sending resume, shard: 0
2019-08-27 23:24:04.216 D2[15183:a97a0700] LOG: DiscordEngine: Resumed gateway session on shard: 0
2019-08-27 23:24:04.216 D2[15183:a97a0700] ERROR: DiscordEngine: Tried heartbeating on disconnected shard, shard: 0

Since the Vapor 2 version connects fine (using an older Swift version), I suspect there is something wrong with the new WebSocket implementation. Perhaps the HTTPClient tries to do non-persistent requests causing the WebSocket to immediately close every time Discord sends a response?

I've reimplemented the WebSocket using Vapor 3's nio-websocket-client and this seems to work reasonably well, so I'm opening the PR for review/mergeability.

@fwcd Do you feel confident enough to open against the development branch? You'd be able to get CI going if you PR there.
I don't have time to check this second exactly what's changed. I think one of the last outstanding things that I was checking out was how to get voice working again in vapor 3. Is that something you wanted to checkout out as well?

This is great work btw, thanks for putting in the effort!

I haven't really experimented with voice yet, though from what I can tell, the new voice implementation uses BlueSocket and should be unaffected be these changes. I'd be willing to take a look at that too, but can't promise when, currently.
For reference, D2 is the Discord bot I'm working on and testing this with.

@fwcd no worries, I'll try and look at this sometime this week/weekend.

BTW if you could add this change so libsodium gets found that would be helpful https://github.com/tellowkrinkle/SwiftDiscord/commit/14867c14e8c453e86417412f762cf06297f7c33b

The websocket seems to reconnect unusually often (despite working fine):
2019-09-30 00:48:30.776 D2[14315:2dffb700] ERROR: DiscordEngine: Tried heartbeating on disconnected shard, shard: 0
2019-09-30 00:49:25.821 D2[14315:5a0d5700] LOG: DiscordWebSocketable: WebSocket closed, code: normalClosure, shard: 0;
2019-09-30 00:49:25.821 D2[14315:5a0d5700] LOG: DiscordEngine: Disconnected, shard: 0
2019-09-30 00:49:25.821 D2[14315:5a0d5700] LOG: DiscordEngine: Trying to resume gateway session on shard: 0
2019-09-30 00:49:25.823 D2[14315:5a0d5700] LOG: DiscordWebSocketable: WebSocket errored, uncleanShutdown, shard: 0
2019-09-30 00:49:25.823 D2[14315:5a0d5700] LOG: DiscordWebSocketable: Connecting to wss://gateway.discord.gg/?v=6&encoding=json, shard: 0
2019-09-30 00:49:25.823 D2[14315:5a0d5700] LOG: DiscordWebSocketable: Attaching WebSocket, shard: shard: 0
2019-09-30 00:49:26.342 D2[14315:5a0d5700] LOG: DiscordWebSocketable: Websocket connected, shard: shard: 0
2019-09-30 00:49:26.342 D2[14315:5a0d5700] LOG: DiscordEngine: Sending resume, shard: 0
2019-09-30 00:49:26.501 D2[14315:2dffb700] LOG: DiscordEngine: Resumed gateway session on shard: 0
2019-09-30 00:49:26.502 D2[14315:2dffb700] ERROR: DiscordEngine: Tried heartbeating on disconnected shard, shard: 0
2019-09-30 00:50:21.527 D2[14315:5a0d5700] LOG: DiscordWebSocketable: WebSocket closed, code: normalClosure, shard: 0;
2019-09-30 00:50:21.527 D2[14315:5a0d5700] LOG: DiscordEngine: Disconnected, shard: 0
2019-09-30 00:50:21.527 D2[14315:5a0d5700] LOG: DiscordEngine: Trying to resume gateway session on shard: 0
2019-09-30 00:50:21.529 D2[14315:5a0d5700] LOG: DiscordWebSocketable: WebSocket errored, uncleanShutdown, shard: 0
2019-09-30 00:50:21.529 D2[14315:5a0d5700] LOG: DiscordWebSocketable: Connecting to wss://gateway.discord.gg/?v=6&encoding=json, shard: 0
2019-09-30 00:50:21.529 D2[14315:5a0d5700] LOG: DiscordWebSocketable: Attaching WebSocket, shard: shard: 0
2019-09-30 00:50:22.041 D2[14315:5a0d5700] LOG: DiscordWebSocketable: Websocket connected, shard: shard: 0
2019-09-30 00:50:22.041 D2[14315:5a0d5700] LOG: DiscordEngine: Sending resume, shard: 0
2019-09-30 00:50:22.172 D2[14315:2dffb700] LOG: DiscordEngine: Resumed gateway session on shard: 0
2019-09-30 00:50:22.173 D2[14315:2dffb700] ERROR: DiscordEngine: Tried heartbeating on disconnected shard, shard: 0

Since these reconnects happen almost exactly once a minute, I suspect there to be some timeout that causes the connection to close periodically.

Yeah that sounds like some issue I've seen before... although I think it'd been happening for a while.


Need a gripper control in the realtime servoing

Description
I can use the Xbox controller to freely control the end effector, but I can't control the closing of the gripper. How can I do this conveniently? In addition to using rviz to control
Your environment

ROS Distro: Humble
OS Version: e.g. Ubuntu 22.04

@songhat can you please share more details like name of manipulator you are using and snippets of script for controlling gripper


Support Synology NAS Ecosystem

I had used Docker Desktop on my Mac (apple silicon) home lab for a while now, but so many times Docker Desktop had performance issues, I just flat out gave up and started looking for other options.  I came across some posts on Reddit and found OrbStack.  It looked promising.  I installed it, and wow, the difference was substantial.
I moved to OrbStack, and I couldn't be happier.  I am so happy that I would love to have another server with containers running and also use OrbStack.  That is my Synology NAS.  I would love to see a package or a way to install OrbStack instead of the native way Synology uses containers.  The current way Synology does container management is just too slow, just like it was on my Mac.  This is why I would love to see a way to run OrbStack there with the massive performance gains.
The performance difference is massive, and it shows when starting many containers.

Sorry, out of scope. No plans to support anything other than macOS for now — everything's built to work with Mac.


Build Agda with debug printing?

Agda <IP_ADDRESS> introduced a debug configure flag that controls whether debug printing (with -v) is enabled (previously it was always enabled).
Should we enable this? I am leaning strongly towards "yes" as I use debug printing quite often and it would suck to have to rebuild Agda.
cc @abbradar @iblech @turion @laMudri @alexarice

is there any downside to enabling it?

Performance is slightly worse, though I'd say not in a way that matters.

heavy opt, debug: 6m8,176s
heavy opt, no debug: 5m45,000s

If it was true by default before, we should keep that default. We can add an option to the derivation to make it configurable.

Opened https://github.com/NixOS/nixpkgs/pull/279188


Set Password Policy for AWS Account?

Are you able to create a password policy for an AWS account in Troposphere?
I can see creating policies for groups, users, roles etc.
But I want to know if this is possible for the entire Account?
Cheers!

As far as I know you can't, as it's not part of the resources available in cloudformation (currently:

AWS::IAM::AccessKey
AWS::IAM::Group
AWS::IAM::InstanceProfile
AWS::IAM::ManagedPolicy
AWS::IAM::Policy
AWS::IAM::Role
AWS::IAM::ServiceLinkedRole
AWS::IAM::User
AWS::IAM::UserToGroupAddition

)

https://github.com/aws-cloudformation/aws-cloudformation-coverage-roadmap/issues/107


Module 1 exercises [Samuel Fianyo-Adovor]

Please "drag & drop" exercise 1 file below.

Please UPDATE THE TITLE of this "Issue" in the title bar above with your name.

If you haven't done so, please go back to Module 1 and make sure to take
the post-training survey.
Exercise 1_Submission.docx

Thank you for your successful submission to the practice exercise! Please be sure to fill out the post-Module survey.


Support Shared Access Signatures with Azure Blob Storage

It would be nice if rclone supported SAS authorization instead of requiring the storage account's shared key. That way I can distribute a read-only SAS URL and use it with rclone.

Sounds like a reasonable request and it is probably supported by the SDK
Fancy doing a bit of research, and linking to some docs?  Or even better sending a pull request?

I'm not particularly versed enough in Go (yet) to contribute some code, but I'll pitch in with some info. For Blob Storage, there are two different kinds of SAS, one is an account level SAS which gives you access to all containers (or other storage products in this account if it's allowed, like tables or queues), the other one is a service level SAS which will give you access to a specific container (or file).
For an account level SAS, you'll only need a different constructor - instead of https://godoc.org/github.com/Azure/azure-sdk-for-go/storage#NewClient you'll either need https://godoc.org/github.com/Azure/azure-sdk-for-go/storage#NewAccountSASClient, https://godoc.org/github.com/Azure/azure-sdk-for-go/storage#NewAccountSASClientFromEndpointToken or https://godoc.org/github.com/Azure/azure-sdk-for-go/storage#NewClientFromConnectionString, depending on which info you have. NewClientFromConnectionString is probably the best one, since a connection string covers every connection method for account-level access:

Access with an account key:
DefaultEndpointsProtocol=https;AccountName=<ACCOUNTNAME>;AccountKey=<ACCOUNTKEY>;BlobEndpoint=https://<ACCOUNTNAME>.blob.core.windows.net/;
With an SAS:
SharedAccessSignature=sv=2017-04-17&ss=bt&srt=sco&sp=rl&st=2018-03-20T20%3A13%3A00Z&se=2018-03-21T20%3A13%3A00Z&sig=<SIGNATURE>;BlobEndpoint=https://<ACCOUNTNAME>.blob.core.windows.net/;

IMHO it would be great to be able to either configure an Azure Blob Storage remote storage using the traditional way (account name, key, endpoint) for backwards compatibility or with a connection string, which covers everything supported by the SDK, including the former.
A service level SAS is a bit different. An SAS for a single file doesn't make much sense for rclone since it's just used to give access to a single file, but a container SAS might be useful. Using an SAS URL you can get a container instance using https://godoc.org/github.com/Azure/azure-sdk-for-go/storage#GetContainerReferenceFromSASURI and from there a Client instance again https://godoc.org/github.com/Azure/azure-sdk-for-go/storage#Container.Client

@maetthu thank you for the writeup - very helpful :-)
So it looks like adding access via NewClientFromConnectionString would be useful.  That looks pretty easy too.
To cover the use case of @memotype (the original poster) I'd need to add config for a SAS URL too if I'm understanding you correctly.
So the 3 config options would be

account/key
connection string
SAS URL (for container level access only)

This looks reasonably straight forward.  If you @maetthu and @memotype are willing to test it, I'll add the code?

@ncw of course :) If needed, I can also provide you with access to a storage account.

I've made a first attempt at this:
https://beta.rclone.org/v1.40-019-gc76f1662-azureblob-auth/ (uploaded in 15-30 mins)
Run through rclone config and hopefully you'll work out what to do!
I haven't updated the docs yet.
I'd appreciate it if you could try the 3 auth methods - thanks!

Authentication from connection string works when using account/key method, although the connection string needs to have the following format (copied from Azure Portal):
DefaultEndpointsProtocol=https;AccountName=<account-name>;AccountKey=<account-key>;EndpointSuffix=core.windows.net
rather than the one I wrote in my earlier comment, which was copied from Azure Storage Explorer (apparently, not all connection strings are created equally :roll_eyes:)
The other two connection methods (SAS in connection string and SAS URL) unfortunately don't work, I'll check if I can dig a bit deeper into the Go SDK to find out what's wrong, it might be just an encoding thing (although I followed the example here: https://docs.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string#create-a-connection-string-using-a-shared-access-signature with no luck)

SAS in connection string reports following error:

2018/03/26 20:05:16 ERROR : : error reading source directory: storage: service returned error: StatusCode=400, ErrorCode=AccountRequiresHttps, ErrorMessage=The account being accessed does not support http.
RequestId:d89d2b4a-001e-0038-122c-c5161f000000
Time:2018-03-26T18:05:16.0525267Z, RequestInitiated=Mon, 26 Mar 2018 18:05:15 GMT, RequestId=d89d2b4a-001e-0038-122c-c5161f000000, API Version=, QueryParameterName=, QueryParameterValue=
2018/03/26 20:05:16 ERROR : Attempt 1/3 failed with 1 errors
2018/03/26 20:05:16 ERROR : : error reading source directory: storage: service returned error: StatusCode=400, ErrorCode=AccountRequiresHttps, ErrorMessage=The account being accessed does not support http.
RequestId:d89d2b5e-001e-0038-252c-c5161f000000

Although the error isn't really related to http vs https, if I enable http access to the storage account, the error changes to:
2018/03/26 20:24:33 ERROR : : error reading source directory: directory not found
2018/03/26 20:24:33 ERROR : Attempt 1/3 failed with 1 errors

SAS url reports

2018/03/26 20:10:00 ERROR : : error reading source directory: Get ?comp=list&delimiter=%2F&include=metadata&maxresults=5000&restype=container: unsupported protocol scheme ""
2018/03/26 20:10:00 ERROR : Attempt 1/3 failed with 1 errors

Judging from this issue here https://github.com/Azure/azure-sdk-for-go/issues/464 it looks like there is  a separate go SDK for blob storage (https://github.com/Azure/azure-storage-blob-go) now apparently, where these issues might be addressed. I also found this pull request https://github.com/Azure/azure-sdk-for-go/pull/1055 which indicates that the storage part in the main go SDK will be moved away to dedicated packages, which is also noted in here: https://github.com/Azure/azure-sdk-for-go/blob/master/storage/README.md
I'm a bit confused atm what these two Go SDKs actually support or not, I'll try to find some more info about that in the next couple of days.

I appreciate the enthusiasm for this feature. @ncw Sorry I wasn't able to dig up the docs, and I'm definitely not a Go developer, just a simple sysadmin, so I won't be able to send a PR. We've currently implemented a work-around that's working for us, but using the rclone method with the SAS would definitely be preferred. Anyway, just wanted to let you know all know I wasn't just pulling a drive-by feature request. :P

@maetthu thanks for testing that :-)

Authentication from connection string works

Hooray!

SAS in connection string reports following error:

I note this in the docs - did you set https?

DefaultEndpointsProtocol=[http|https];

This error

error reading source directory: directory not found

Looks like it is almost working...
If you wanted to do a bit more debugging with the code I posted, then I recommend using -vv --dump bodies which will show all the HTTP transactions - that usually helps a lot.

SAS url reports

error reading source directory: Get ?comp=list&delimiter=%2F&include=metadata&maxresults=5000&restype=container: unsupported protocol scheme ""

This looks like the URL is missing a protocol scheme, likely an https:\\  In fact it looks like it is missing the rest of the URL.  Did you put in something like "https://host.com/stuff` here?
--dump bodies may throw some light.
I've had interactions with the Azure Go devs and they indicated to me that the preferred package is now https://github.com/Azure/azure-storage-blob-go - however that package doesn't look particularly mature to me yet!  Also it is a lot of work swapping a backend implementation :-(
@memotype Thanks for the comment!  As far as this feature goes, happy to pull existing features out of the SDK, but I really don't have enough time to dive into the rabbit hole of the Microsoft auth system so really appreciate your help @maetthu.  Assuming we can get enough of this working to be useful, I'll ask for some help writing the docs too!

@memotype @maetthu just looking through branches I haven't merged and I found this one.
Does anyone want to help debug the bits which don't work, or shall I merge the connection string part (which does work) for v1.41?

@ncw I'm sorry, I totally forgot about this :frowning_face: I'll do some debugging in the next days and see if I can shed some light on it.
I'm not sure if merging the connection string part is a good idea though, since only the connection string with the account key works atm and this is already covered through the current method. Supporting connection strings but not all variations might just lead to more confusions w/o a real benefit. I guess it's better to wait until at least SAS through connection string also works.

Here we go - I did some testing and digging and managed to get some more things to work:

Fails: Config using account/key connection string from Azure Storage Explorer:

[azure]
type = azureblob
connection_string = DefaultEndpointsProtocol=https;AccountName=MYSTORAGEACCOUNT;AccountKey=KEY;BlobEndpoint=https://MYSTORAGEACCOUNT.blob.core.windows.net/;QueueEndpoint=https://MYSTORAGEACCOUNT.queue.core.windows.net/;TableEndpoint=https://MYSTORAGEACCOUNT.table.core.windows.net/;FileEndpoint=https://MYSTORAGEACCOUNT.file.core.windows.net/;

2018/04/24 17:31:39 DEBUG : rclone: Version "v1.40-019-gc76f1662-azureblob-authβ" starting with parameters ["./rclone" "-vvv" "--config" "config-connection-string-key.txt" "copy" "tmp/" "azure:container"]
2018/04/24 17:31:39 DEBUG : Using config file from "/home/maetthu/tmp/rclone/config-connection-string-key.txt"
2018/04/24 17:31:39 Failed to create file system for "azure:container": failed to make azure storage client from connection string: azure: base storage service url required

Works: Config using account/key connection string from Azure Portal:

[azure]
type = azureblob
connection_string = DefaultEndpointsProtocol=https;AccountName=MYSTORAGEACCOUNT;AccountKey=KEY;EndpointSuffix=core.windows.net;

Works: Config using account-level SAS, generated in Azure Portal. Again, Storage Explorer connection string does not work since it doesn't include the protocol and the SDK falls back to HTTP, which fails unless the account is configured to allow unencrypted connections:

[azure]
type = azureblob
connection_string = SharedAccessSignature=sv=2017-04-17&ss=b&srt=sco&sp=rwdl&st=2018-04-24T15%3A32%3A14Z&se=2018-04-25T15%3A32%3A14Z&sig=SIGNATURE;BlobEndpoint=https://MYSTORAGEACCOUNT.blob.core.windows.net/;

Fails: Config using an SAS-URL

[azure]
type = azureblob
sas_url = https://MYSTORAGEACCOUNT.blob.core.windows.net/?sv=2017-07-29&ss=b&srt=sco&sp=rwdlac&se=2018-04-25T00:25:09Z&st=2018-04-24T16:25:09Z&spr=https&sig=SIGNATURE

$ ./rclone --dump headers --dump bodies --dump auth --dump requests --dump responses -vvv --config config-connection-string-sas-url.txt copy tmp/ azure:container
2018/04/24 18:29:44 DEBUG : rclone: Version "v1.40-019-gc76f1662-azureblob-authβ" starting with parameters ["./rclone" "--dump" "headers" "--dump" "bodies" "--dump" "auth" "--dump" "requests" "--dump" "responses" "-vvv" "--config" "config-connection-string-sas-url.txt" "copy" "tmp/" "azure:container"]
2018/04/24 18:29:44 DEBUG : Using config file from "/home/maetthu/tmp/rclone/config-connection-string-sas-url.txt"
2018/04/24 18:29:44 INFO  : Azure container container: Modify window is 1ns
2018/04/24 18:29:44 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/04/24 18:29:44 DEBUG : HTTP REQUEST (req 0xc420322b00)
2018/04/24 18:29:44 DEBUG :
2018/04/24 18:29:44 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/04/24 18:29:44 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/04/24 18:29:44 DEBUG : HTTP RESPONSE (req 0xc420322b00)
2018/04/24 18:29:44 DEBUG : Error: unsupported protocol scheme ""
2018/04/24 18:29:44 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/04/24 18:29:44 ERROR : : error reading destination directory: Get ?comp=list&delimiter=%2F&include=metadata&maxresults=5000&restype=container: unsupported protocol scheme ""
2018/04/24 18:29:44 INFO  : Azure container container: Waiting for checks to finish
2018/04/24 18:29:44 INFO  : Azure container container: Waiting for transfers to finish
2018/04/24 18:29:44 ERROR : Attempt 1/3 failed with 1 errors

TL/DR:

Connection string using account key works (if you use the connection string format from Azure Portal)
Connection string using SAS works as well (if you manually craft a connection string from an SAS generated in Azure Portal)
Both account-level and container-level SAS-URL don't work - seems it can't parse the URL properly? A trace shows that no connection to Azure is made.

Is there any blocker on this? This would definitely be great to have!

@zifeo help testing it would be nice I guess. I couldn't get SAS-URLs to work, but since I'm neither very experienced with Go nor with Azure, I might have missed something. For a link to download the version where the feature is added, see https://github.com/ncw/rclone/issues/2118#issuecomment-375891719

Yes more testing please!  I need help with understanding how the Azure stuff works - I understand the API quite well but the myriad ways of authentication I find quite baffling ;-)

Here is a new version of the code.  The code is unchanged but this has updated Azure libraries which may make a difference!
https://beta.rclone.org/v1.42-006-gacf9078d-azureblob-auth/ (uploaded in 15-30 mins)
If someone could test the SAS URL feature I'd be grateful!  (Configure a new azureblob remote and you'll see the option).

I did some testing for the SAS-URL but only got this for copy: 2018/06/20 22:18:13 ERROR : Attempt 1/3 failed with 0 errors and: Head http://.blob./CONTAINER/FILES: dial tcp: lookup .blob.: no such host . Not sure I did everything correct though... I have only the sas_url set to the full url given by Azure?

@zifeo Thanks for testing this.  I found an issue on the upstream webpage: https://github.com/Azure/azure-sdk-for-go/issues/959 which isn't wildly helpful - it just says

Are you sure your SAS query params are correct? I tried this and it works for me using a SAS token generated via the Azure portal.

The token was generated through the Azure Portal as well. Maybe any encoding issue?

I managed to figure out that this method only works for container based SAS URLs so you have to open the storage explorer and right click on the container - there you will see an option to make a SAS URL.
This is almost working - well listings are working.  I can't get it to download files.
https://beta.rclone.org/v1.42-012-gef7b0541-fix-2118-azureblob-auth/ (uploaded in 15-30 mins)
I have made an upstream issue as I think there is a problem there: https://github.com/Azure/azure-sdk-for-go/issues/2119

It seems a fix was merged here: https://github.com/Azure/azure-sdk-for-go/pull/2123
I expanded your example with a quick'n dirty blob upload and delete, built against the latest branch of Azure SDK it works for me with an SAS URL created within Storage Explorer now: https://github.com/maetthu/azure-storage-playground

@maetthu great - thanks for testing :-)
I've pulled the latest  branch from the azure repo into the rclone branch and rebuilt - can you give this a go?  It seems to be working for me!
I'll wait until the Azure code hits master before merging it to rclone master most likely.
https://beta.rclone.org/branch/v1.42-016-g865b3303-fix-2118-azureblob-auth/ (uploaded in 15-30 mins)

I can confirm it works well for container specific SAS urls, nice job! Do you have an upstream issue for service wide SAS urls?

@zifeo I didn't test account level SAS URLs, but an account level SAS is already possible using a connection string, see the results of my tests for a previous version here https://github.com/ncw/rclone/issues/2118#issuecomment-384008230

Okay, in this case, we should make explicit that account level SAS urls does not work and rather connection string should be used.

@zifeo it does say this in the help
		Help: "SAS URL for container level access only\n(leave blank if using account/key or connection string)",

Maybe it could be more obvious.
Here is a first stab at some documentation - updates and corrections appreciated :-)
Authenticating with Azure Blob Storage
Rclone has 3 ways of authenticating with Azure Blob Storage:
Account and Key
This is the most straight forward and least flexible way.  Just fill in the account and key lines and leave the rest blank.
Connection string
This supports all the possible connection string variants.  Leave account, key and sas_url blank and put the connection string into the connection_string configuration parameter.
Use this method if using an account level SAS; the Azure Portal shows connection strings you can cut and paste.
SAS URL
This only for a container level SAS URL - it does not work with an account level SAS URL. For account level SAS use the connection string method.
To get a container level SAS URL right click on a container in the Azure Blob explorer in the Azure portal.
You will only be able to use the container specified in the SAS URL with rclone, eg
rclone ls azureblob:container

However these will not work
rclone lsd azureblob:
rclone ls azureblob:othercontainer

This would be useful for temporarily allowing third parties access to a single container or putting credentials into an untrusted environment.

I might have gone too fast on it, sorry. The document seems all good to me!

@ncw works here as well, using sas_url in config:
[azure]
type = azureblob
sas_url = https://ACCOUNT.blob.core.windows.net/CONTAINTER?sv=2017-07-29...

very nice :tada:

Excellent - thanks for testing :-) I'm going to leave this open for a while until the upstream fix hits the master branch, then I'll merge to master.

Apparently the upstream fix is due very soon so I'll hold off for a moment.

We're planning to release v18 first week of July (hopefully the 2nd), and that release will have the fix.

I have merged this to master now - thank you all for your help testing.
https://beta.rclone.org/v1.42-022-g03c5b823/ (uploaded in 15-30 mins)

Thanks, when is the release planned?

@ncw very nice, thank you :slightly_smiling_face:

The release is planned for 28th July.  No guarantees though!

Note that thanks to the hard work by @sandeepkru we've updated to use the newer SDK.
This is the recommended SDK for use with azure blob.
It has the advantage that SAS URLs now work properly both for account and container.
It has the disadvantage that we've had to drop connection string support.  Apparently connection strings are no longer going to be supported by Microsoft which is why the new SDK doesn't support them.  I don't view this as a big loss since we only just put that feature in, but it may affect people on this issue, so I apologise if this causes you any inconvenience.
You'll find this in
https://beta.rclone.org/v1.42-035-g99201f8b/ (uploaded in 15-30 mins)
and the latest beta from now on.


Fix the "running chaincode as an external service" doc section

The Fabric doc says that "Using this chaincode as an external service model, installing the chaincode on each peer is no longer required"
But this statement can confuse a user who has an organization with at least 2 endorsing peers:
Q: Do I need to install chaincode on a second endorsing peer too?
A: Using an chaincode as an external service model, you will need to install a chaincode package on every peer which will execute it.
I think that the paragraph should be modified to make it clear that if a user has at least 2 endorsing peers, he must install the chaincode in each peer
link:
https://hyperledger-fabric.readthedocs.io/en/release-2.4/cc_service.html#running-the-chaincode-as-an-external-service

I agree that it is confusing, because the chaincode package must be installed on every peer where the chaincode will be invoked, even though the installed chaincode package doesn't include the actual chaincode logic. This should definitely be clarified.
This is also probably the right place to discuss another frequent question - the cardinality of peers to external chaincode services. That is, can two peers point to the same external chaincode service endpoint? Alternatively, can one peer point to an external chaincode service endpoint that actually represents multiple chaincode service instances made available through a load balancer?
For the latter questions, let's ask @jkneubuh @mbwhite to comment as they have investigated this area in several deployment environments.

Hi @denyeart and @kmilodenisglez
The TL/DR summary and current recommendation is to deploy CCaaS endpoints 1:1 with the peers, utilizing the Gateway Client as a mechanism to distribute transaction load across a set of peers.
Please see the notes and discussion at #3350, or the original Discord thread #LOAD BALANCING CHAINCODE for some additional context.