Generating a SAS for a DataLake Blob Directory

In portal, I can click on a directory and generate a SAS for it and all it's children recursively.

The Resource types for BlobSasBuilder do not work. I also tried with DatalakeSasBuilder or something like that, but I saw it is mainly used for file system? (correct me if I'm wrong).
How can I do the same using the .Net SDK?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 077b3fa5-15a2-e3a8-10f6-2bd25e001b09
Version Independent ID: 85c7f2d2-d246-45a4-3e95-7abbbc207821
Content: BlobSasBuilder.Resource Property (Azure.Storage.Sas) - Azure for .NET Developers
Content Source: xml/Azure.Storage.Sas/BlobSasBuilder.xml
Service: azure
GitHub Login: @rloutlaw
Microsoft Alias: routlaw

Hi,
In order to create a SAS for the directory and all of its files and subdirectories you can either two ways.
Generate using a DataLakeDirectoryClient (recommended way to not have to specify a lot of parameters)
// client authenticated with a Shared Key (connection string contains the shared key)
DataLakeDirectoryClient directoryClient = new DataLakeDirectoryClient(connectionString, fileSystemName, path);
// generate the SAS with setting the permissions and the expirey time
Uri sasUri = directoryClient.GenerateSasUri(DataLakeSasPermissions.All, Recording.UtcNow.AddDays(1));
// OR to specify more parameters pass a DataLakeSasBuilder
DataLakeSasBuilder sasBuilder = new DataLakeSasBuilder(permissions, expiresOn)
{
    StartsOn = startsOn
};
Uri sasUri = directoryClient.GenerateSasUri(sasBuilder);

Using the DataLakeSasBuilder to generate a directory SAS
DataLakeSasBuilder sasBuilder = new DataLakeSasBuilder(permissions, expiresOn)
{
    FileSystemName = fileSystemName,
    Path = path,
    IsDirectory = true,
};
DataLakeUriBuilder expectedUri = new DataLakeUriBuilder(serviceUri)
{
    FileSystemName = fileSystemName,
    DirectoryOrFilePath = path
    **Sas = sasBuilder2.ToSasQueryParameters(constants.Sas.SharedKeyCredential);**
};

Please let me know if you have any questions.

Hi,
In order to create a SAS for the directory and all of its files and subdirectories you can either two ways.
Generate using a DataLakeDirectoryClient (recommended way to not have to specify a lot of parameters)
// client authenticated with a Shared Key (connection string contains the shared key)
DataLakeDirectoryClient directoryClient = new DataLakeDirectoryClient(connectionString, fileSystemName, path);
// generate the SAS with setting the permissions and the expirey time
Uri sasUri = directoryClient.GenerateSasUri(DataLakeSasPermissions.All, Recording.UtcNow.AddDays(1));
// OR to specify more parameters pass a DataLakeSasBuilder
DataLakeSasBuilder sasBuilder = new DataLakeSasBuilder(permissions, expiresOn)
{
    StartsOn = startsOn
};
Uri sasUri = directoryClient.GenerateSasUri(sasBuilder);

Using the DataLakeSasBuilder to generate a directory SAS
DataLakeSasBuilder sasBuilder = new DataLakeSasBuilder(permissions, expiresOn)
{
    FileSystemName = fileSystemName,
    Path = path,
    IsDirectory = true,
};
DataLakeUriBuilder expectedUri = new DataLakeUriBuilder(serviceUri)
{
    FileSystemName = fileSystemName,
    DirectoryOrFilePath = path
    **Sas = sasBuilder2.ToSasQueryParameters(constants.Sas.SharedKeyCredential);**
};

Please let me know if you have any questions or if this resolves your issue.

I have one more question. I think I'm not familiar with the terminology but does this create a SAS for the blobs or for files/directories in the file shares? I'm working with the blob storage. If you have a hierarchical namespace (datalake) then the terminology changes from blobs to files? I'm not getting it.

Hi,
Our documentation writers wrote up this great document about DataLake that would help you understand the difference.
https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction
Essentially just to answer your question, this would create a SAS for a DataLake Directory (not a SMB file share directory). When you have Hierarchical namespace (datalake) enabled, then the terminology changes to datalake file (not SMB file). This is because datalake works on top of blobs.

I followed the steps given earlier and always get the below exception. Azure side everything is open. I'm not sure what changes the name of my DFS endpoint https://gopdfsfoo.dfs.core.windows.net/ to gopblobfoo.dfs.core.windows.net.
One or more errors occurred. (Retry failed after 6 tries. Retry settings can be adjusted in ClientOptions.Retry. (No such host is known. (gopblobfoo.dfs.core.windows.net:443)) (No such host is known. (gopblobfoo.dfs.core.windows.net:443)) (No such host is known. (gopblobfoo.dfs.core.windows.net:443)) (No such host is known. (gopblobfoo.dfs.core.windows.net:443)) (No such host is known. (gopblobfoo.dfs.core.windows.net:443)) (No such host is known. (gopblobfoo.dfs.core.windows.net:443)))
Code below:
        public static async Task WTH(DataLakeSasPermissions permissions)
        {
            // client authenticated with a Shared Key (connection string contains the shared key)
            DataLakeDirectoryClient directoryClient = new DataLakeDirectoryClient("redacted connection string" , "gop-filesystem2", "gop-dir");
            // generate the SAS with setting the permissions and the expirey time
            Uri sasUri = directoryClient.GenerateSasUri(DataLakeSasPermissions.All, DateTimeOffset.UtcNow.AddDays(1));

            DataLakeDirectoryClient sasDirectoryClient = new DataLakeDirectoryClient(sasUri);

            // Act
            await foreach (PathItem pathItem in sasDirectoryClient.GetPathsAsync())
            {
                // Just make sure the call succeeds.
                Console.WriteLine(pathItem.Name);
            }
        }

@vivere-dally


Use existing tiff

I'm extracting all the pages of a PDF as TIFFs thru the mini_magick gem & I'd like to feed each of these to rtesseract w/o having it unnecessarily re-generate new, temporary tiffs. Short of monkey patching your image method, is there any way to do this?

@dshorthouse Did you try with .new('file.png, processor: 'mini_magick')?  Completely a stab in the dark, (processor seems to be undocumented in README).

@mjy Hard to tell what's going on since the signature has changed quite a bit since end-2017. Indeed, I was using MiniMagick to extract pages of a PDF to separate tiff files + altering density, stripping out alpha layer, etc. prior to passing to rtesseract, which itself then again converted to tiff tempfile thru a method that now no longer exists in this newer version of rtesseract. Assuming a tempfile is no longer created thru https://github.com/dannnylo/rtesseract/blob/b27f7e12110100a0fbd203b3b6ef37912e1f6178/lib/rtesseract/text.rb#L6 then perhaps this is no longer an issue.


Composer install steps

Simplify the install process by updating the Composer package definition.

Update platform dependencies to require ext-bcmath and suggest ext-gmp
Install pear.php.net/PHPUnit version 1.x as dev dependency
Install pear.php.net/MDB2 as dependency

I suggest an update to the README INSTALLATION instructions with steps for a composer install. Using this updated composer.json install is a lot simpler than it was.

I'm not sure why we'd favor bcmath over gmp. Either way I think these changes are certainly nice to have other than what we suggest over depend on.
However, I'm closing this due to age. Please rebase and update with a new PR if you wish to see this feature added!


Archiving documents following the instructions doesn't archive document.

Hello.
I am experiencing instances where some purchase orders do not auto-archive once they are fully received/invoiced.  The P/O's are showing as received & invoiced, noting that several (4) P/O's were received and then invoiced under one invoice.  The P/O's have remained visible int he open Purchase Orders, so I have attempted to Archive Document according to the instructions.
Following the instructions, I have selected the P/O, then selected Actions > Functions > Archive Document.
A display prompt is shown to Confirm "Archive Order no.: [order number]" and I select the YES button.
Another display box indicates that "Document [order number] has been archived.
The purchase orders are still showing in the open purchase orders screen.  Is there a missing process or step in the instructions that allows the archive process to occur, but the order itself isn't archived?  the instructions do not indicate there are any other steps required.  I can see that the No. of archived versions is increasing each time I archive the document.

Document details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 3f4ae087-a1fd-7343-f2dd-57d7b4536272
Version Independent ID: 46bdbddd-5b62-3970-aa86-0fb08f44147c
Content: Archive Sales and Purchase Documents - Business Central
Content Source: business-central/across-how-to-archive-documents.md
Service: dynamics365-business-central
GitHub Login: @brentholtorf
Microsoft Alias: bholtorf

Thanks for the feedback.
This looks like a bug; we have opened a product bug for this issue.


Add implementation for simple_query_string relevance search function in SQL and PPL

Description
Issues Resolved
#192
Check List

[x] New functionality includes testing.

[x] All tests pass, including unit test, integration test and doctest

[x] New functionality has been documented.

[ ] New functionality has javadoc added
[x] New functionality has user manual doc added

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #635 (30bc724) into main (26058b8) will decrease coverage by 31.81%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##               main     #635       +/-   ##
=============================================
- Coverage     94.57%   62.76%   -31.82%
=============================================
  Files           278       10      -268
  Lines          7430      658     -6772
  Branches        554      119      -435
=============================================
- Hits           7027      413     -6614
+ Misses          349      192      -157
+ Partials         54       53        -1

Flag
Coverage Δ

query-workbench
62.76% <ø> (ø)

sql-engine
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...rg/opensearch/sql/analysis/ExpressionAnalyzer.java

...opensearch/sql/data/model/ExprCollectionValue.java

...c/main/java/org/opensearch/sql/expression/DSL.java

...ensearch/sql/expression/ExpressionNodeVisitor.java

...h/sql/expression/function/BuiltinFunctionName.java

...h/sql/expression/function/OpenSearchFunctions.java

...arch/storage/script/filter/FilterQueryBuilder.java

...pensearch/sql/ppl/parser/AstExpressionBuilder.java

...pensearch/sql/sql/parser/AstExpressionBuilder.java

...org/opensearch/sql/data/model/ExprDoubleValue.java

... and 258 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 26058b8...30bc724. Read the comment docs.


Allow to use Godot as a GUI frontend for python external code execution (Feature proposal).

Hi, I wonder if somehow godot could be used as a GUI frontend and have external calls to python code so we can use python libraries for diferent purposes, this could mean libraries like tensorflow or twitter api or whatever python library the user could set up in his system be available to the engine.
The users could even write external scripts for whatever they want.
That way we could make non game applications on godot very easily, maybe use external services like twitter, facebook, google maps api.
I would love to use this to call a tensorflow library so I can make a chatbot game and use godot GUI and art capabilities as front end.

I've looked into this to experiment with AlphaGo-inspired game AI.

There's Python for Godot but the last release was a year ago - working for 3.0.6 at least but not 3.1. We dunno or cannot expect when the next one's out. The developers here can't help much, I guess.

We can call a Python script from Godot with

var output = []
var pid = OS.execute('play.py', [], true, output)

which freezes the game (may be worked around with threading) with no way to send subsequent
commands even with the PID, afaik.
Same with opening the Python interpreter:
var pid = OS.execute('python', [], true, output)

I believe this is where the devs here can step in.

Yes, all of this is already possible with the python bindings.
However this is not really the place to ask question about how to use the engine, but to report bugs and ask for new features. Please ask on the other community channels if you need more advises (discord,  forum, Q&A...)

Regarding the python binding issue, you can submit an issue on the bindings repository. Maybe the maintainers will update for 3.1


Problem with new line

Well... This problem is really really odd...
I am trying to use the transform to add new field to the payload and then using the new field in the actions.
But - i can't add new line to my new field.
That is my transform code(The rest of the alarm is not important):
  "script": {
    "script": "var results=payload.aggregations.group_by_hosts.buckets;var text='';for(var i=0;i<results.length;i++){text+=results[i].key + ':' + results[i].average.value + '\n';}payload.hosts=text;"
  }
}```

please paste a sample of input data and the expected output

For example: I have 2 servers(named ser1 and ser2) which have low CPU idle.
I want to send an E-mail to myself which will look like that:
ser1 low CPU idle: <idle_value>
ser2 low CPU idle: <idle_value>
I want to create a long string with '/n' in the middle. But as I said,the watcher is failing...

You can use email_html and loop through the reporting items with mustache:
{{#hosts}} <p> {{ some_value}} </p> {{/hosts}}
For more ideas see the mustache documentation and examples - hope this helps!

@lmangani  I figured it out. Just wanted to report on the bug :smile:

@masterofcloud1 thanks! I don't really consider this a bug, since all the "formatting" is to be done in actions, but its appreciated nevertheless to improve documentation and examples


[CI] Add SemaphoreCI

This PR adds SemaphoreCI as new CI platform.

Merged too soon. :-(
Please consider my comments post-mergem. :-)


please create laravel 5 branch

i will fix all the bugs in laravel 5 branch
i have project that is in laravel 5 and i want to use this package
it will be grate if you add me in contributors or i will create pull requests

i have created develop-l5 where you can create PRs to

we need 2 branch for laravel 4 and for laravel 5
laravel 5 version will not work for laravel 4 and
laravel 4 version will not work for laravel 5 and
currently master brunch is not compatible with laravel 4 and 5
v0.5.1 is compatible with laravel 4
but how we will update that ? i guess we need branch with illuminate/support and illuminate/filesystem to 4.x
#34 will make master compatible with laravel 5

i would keep the master branch compatible with L5 and deprecate L4,
As you said 0.5.1 is compatible with L4 and will be the last version for it.

have you tested current version on L5
it does not work at all
i have pull request #34
i guess there will be other issue but it makes this simple example work
$files = glob('public/files/*');
Zipper::make('public/test.zip')->add($files);

i was thinking we need to support L4
if not develop-l5 branch is useless
also i want to say thank you for this library
it makes world better place :laughing:

ok, i have created l4 and l5 as branches, we just need to put the right versions there

in L4 branch if you remove last 3 commits it will be fine
you said master will be for L5 so we don't need 'l5' branch
just update readme and say: master branch is for laravel 5 and there is L4 branch if you use laravel 4 click here
somthing like this

please tag current master branch as 0.6.1
also in readme
instead of '0.5.1' and '0.6.0' it will be better to put '0.5.' and '0.6.'
because usually last digit is for bug fixes


change: install SageMaker Python SDK into Python 3 images - TF 1.15.2

Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Similar issues reported when adding smdebug to Tensorflow 1.15.2
https://github.com/aws/sagemaker-tensorflow-container/pull/298
The error logs look nearly the same

that's because the buildspec isn't right. see #297. that's why we haven't required the PR build for merges yet

that's because the buildspec isn't right. see #297. that's why we haven't required the PR build for merges yet

Thanks!


Hide empty profile status on hover

Fixes #4160
Test URLs

Profile with empty status
Profile with status

Screenshot

Oh no, it's 2 rules. Let's move it to its own file 😃


How to do the inference on new data set?

I have trained the Context-Dependent Sentiment Analysis in User Generated Videos , and it went well! But how can I use the model to infer new data! Thanks! @soujanyaporia

For that, you need to replicate the feature extraction process for every modality. See some sample codes here: https://github.com/soujanyaporia/MUStARD


Change systemd unit names to pod IDs

This PR changes the systemd unit names to use pod IDs, because for some
use cases the pod names may not work.
For example when container name and pod name has same for different
podman version the podman generate different pod name.
Signed-off-by: Ondra Machacek<EMAIL_ADDRESS>

cc @jakub-dzon


Welders cannot be refueled.

1: Use a welder.
2: Try and refuel it on a fuel tank.
3: Be disappointed.
I've been vaguely hearing that this has been a bug for around a week, but no one ever confirmed it.

Same thing appears to be happening with syringes.

1: Use a welder. 2: Try and refuel it on a fuel tank. 3: Be disappointed.
I've been vaguely hearing that this has been a bug for around a week, but no one ever confirmed it.

I believe this is because it's conflicting with the welder/fuel tank interaction between a lit welding torch and the fuel tank exploding.

Actually it looks like I broke it with some anchorable changes. The welder tries to anchor the tank....


Need documentation for adding new recipe and ci

I started this in the developer section - what else would it need?


Support for .NET Core SDK?

Although this is to compile C# to Javascript which does not run on .NET Framework, there are still many reasons to migrate the whole toolchain to .NET Core:

.NET Framework is terminated.
Visual Studio independency.
.NET Core only or .NET Standard 2.1 features.
......

It does not need to depend on an obsoleted framework, does it?

It does not need to depend on an obsoleted framework, does it?

For the time being, it does.
Unfortunately, we just don't have the resources (time, people, money) available to upgrade Bridge.NET to full .NET Core.

We will do our best to support .NET Core as soon as we can. Once enabled, we will update this issue with the news.


IMU data time on Cartographer

about sensor delivered data out of order.
https://hackmd.io/s/r1fAEBGlX#issue2 from @YuehChuan

Thanks for openning this!

Hello @YuehChuan :)
We updated new firmware (v1.2.1) and software today
This update includes modified time function in roslib and amended config file for cartographer
After fixed it, the cartographer was not crashed with IMU and without IMU for a long time.
You can get more information about it in ROS Discourse
Thank you for your inquiry and suggestions!

Hi @YuehChuan,
Thank you for your consideration and supports.
With your help, we were able to find this issue early and to improve it and release it.
I close this issue. You can reopen this issue to show this issue to the users whenever.
Once again, thank you for your help and supports. :)


Fix build issues by adding pyproject.toml

Fixes gatagat/lap#38.
Previously, it was not always possible to install lap by simply listing it as a dependency e.g. in a requirements.txt file. This is because the setup.py build system depends on numpy.distutils, and so numpy has to be installed before setup.py could run successfully. This is not always guaranteed even if doing something like pip install numpy lap.
By using pyproject.toml, we can specify that numpy>=1.10.1 and cython>=0.21 are requirements for the build step. These will be installed in a build-only environment during the setup. Numpy is also required at runtime, so it is now listed as a dependency as well.

Hello, new v0.5.12 is now available, and you can install lap smoothly using pip install lap on most OS and platforms. Thanks!


Optimize icons

We use only several icons but the huge part of the projects(~ 600kb) is just font-awesome items.
They also slow down the build and rendering.

Done