Can I have multiple Intents in LUIS for the same knowledge base.

Issue Description:

I'm referring below github code:
https://github.com/microsoft/BotBuilder-Samples/blob/master/samples/javascript_nodejs/14.nlp-with-dispatch/
I am having trouble to access multiple knowledge bases using LUIS and Dispatcher tool.
Below are my issues:

Can I have multiple Intents in LUIS for the same knowledge base.
How to create intent per question in LUIS when you have more than 100 questions.

Can I have same intent in two knowledge bases.

Please help me to implement this situation.

Hi @tejasvinishelar, can you post your code so I can better advise you? Please don't include any keys.

Hello Sir,
I can't post my code publicly.
Is there any other way to send you my code?

@tejasvinishelar, apologies for the delay. You can send the code via email to me. My email is in my profile. Please reference this issue (#, link, etc.) so I can easily find.

@tejasvinishelar, in the future if you can use GitHub issues for product defect or product enhancement requests that would be aligned to the community's exceptions.  How-to questions should be vectored to Stackoverflow where we have an active community.  https://stackoverflow.com/search?q=bot+framework
Support, continue helping.  thank you

@stevkan, do you have any updates?

@tejasvinishelar, thank you for sending your code. I will take a look at respond as soon as possible with any other solutions/questions.

@stevkan please provide an update

Hi @tejasvinishelar, I believe the following example should help.
Essentially, I have two LUIS intents that have similar yet distinct utterances: home and vehicle automation (and movie times and list). Dispatcher first processes which LUIS app it belongs to and returns a value which is then mapped to a process (e.g "turn my Xbox on" => "l_HomeAutomation"). "l_HomeAutomation" and "l_VehicleAutomation" both map to this.processAutomation. Within the process the utterance is processed again against the specific home automation LUIS app returning a specific intent. The returned intent can then be mapped to the QnA KB to retrieve a response. In this scenario, home and vehicle automation questions pull from the same KB as do movie times and list.
Does this answer your question / solve your issue?
class DispatchBot extends ActivityHandler {
    constructor() {
        super();
        const dispatchRecognizer = new LuisRecognizer({
            applicationId: process.env.LuisAppId,
            endpointKey: process.env.LuisAPIKey,
            endpoint: `https://${process.env.LuisAPIHostName}.api.cognitive.microsoft.com`
        }, {
                includeAllIntents: true,
                includeInstanceData: true
            }, true);

        const qnaMaker = new QnAMaker({
            knowledgeBaseId: process.env.QnAKnowledgebaseId,
            endpointKey: process.env.QnAEndpointKey,
            host: process.env.QnAEndpointHostName
        });

        this.dispatchRecognizer = dispatchRecognizer;
        this.qnaMaker = qnaMaker;

        this.onMessage(async (context, next) => {
            console.log('Processing Message Activity.');

            // First, we use the dispatch model to determine which cognitive service (LUIS or QnA) to use.
            const recognizerResult = await dispatchRecognizer.recognize(context);

            // Top intent tell us which cognitive service to use.
            const intent = await LuisRecognizer.topIntent(recognizerResult);

            // Next, we call the dispatcher with the top intent.
            await this.dispatchToTopIntentAsync(context, intent, recognizerResult);
            await next();
        });

    async dispatchToTopIntentAsync(context, intent, recognizerResult) {
        switch (intent) {
            case 'l_HomeAutomation':
                await this.processAutomation(context, recognizerResult.luisResult);
                break;
            case 'l_VehicleAutomation':
                await this.processAutomation(context, recognizerResult.luisResult);
                break;
            case 'l_MovieTimes':
                await this.processMovies(context, recognizerResult.luisResult);
                break;
            case 'l_MovieList':
                await this.processMovies(context, recognizerResult.luisResult);
                break;
            default:
                console.log(`Dispatch unrecognized intent: ${intent}.`);
                await context.sendActivity(`Dispatch unrecognized intent: ${intent}.`);
                break;
        }
    }

    async processAutomation(context, luisResult) {
        console.log('processHomeAutomation');

        // Retrieve LUIS result for Process Automation.
        const result = luisResult.connectedServiceResult;
        const intent = result.topScoringIntent.intent;

        [...Make QnA call...]
    }
}

module.exports.DispatchBot = DispatchBot;

Hi @tejasvinishelar, I believe the following example should help.
Essentially, I have two LUIS intents that have similar yet distinct utterances: home and vehicle automation (and movie times and list). Dispatcher first processes which LUIS app it belongs to and returns a value which is then mapped to a process (e.g "turn my Xbox on" => "l_HomeAutomation"). "l_HomeAutomation" and "l_VehicleAutomation" both map to this.processAutomation. Within the process the utterance is processed again against the specific home automation LUIS app returning a specific intent. The returned intent can then be mapped to the QnA KB to retrieve a response. In this scenario, home and vehicle automation questions pull from the same KB as do movie times and list.
Does this answer your question / solve your issue?
class DispatchBot extends ActivityHandler {
    constructor() {
        super();
        const dispatchRecognizer = new LuisRecognizer({
            applicationId: process.env.LuisAppId,
            endpointKey: process.env.LuisAPIKey,
            endpoint: `https://${process.env.LuisAPIHostName}.api.cognitive.microsoft.com`
        }, {
                includeAllIntents: true,
                includeInstanceData: true
            }, true);

        const qnaMaker = new QnAMaker({
            knowledgeBaseId: process.env.QnAKnowledgebaseId,
            endpointKey: process.env.QnAEndpointKey,
            host: process.env.QnAEndpointHostName
        });

        this.dispatchRecognizer = dispatchRecognizer;
        this.qnaMaker = qnaMaker;

        this.onMessage(async (context, next) => {
            console.log('Processing Message Activity.');

            // First, we use the dispatch model to determine which cognitive service (LUIS or QnA) to use.
            const recognizerResult = await dispatchRecognizer.recognize(context);

            // Top intent tell us which cognitive service to use.
            const intent = await LuisRecognizer.topIntent(recognizerResult);

            // Next, we call the dispatcher with the top intent.
            await this.dispatchToTopIntentAsync(context, intent, recognizerResult);
            await next();
        });

    async dispatchToTopIntentAsync(context, intent, recognizerResult) {
        switch (intent) {
            case 'l_HomeAutomation':
                await this.processAutomation(context, recognizerResult.luisResult);
                break;
            case 'l_VehicleAutomation':
                await this.processAutomation(context, recognizerResult.luisResult);
                break;
            case 'l_MovieTimes':
                await this.processMovies(context, recognizerResult.luisResult);
                break;
            case 'l_MovieList':
                await this.processMovies(context, recognizerResult.luisResult);
                break;
            default:
                console.log(`Dispatch unrecognized intent: ${intent}.`);
                await context.sendActivity(`Dispatch unrecognized intent: ${intent}.`);
                break;
        }
    }

    async processAutomation(context, luisResult) {
        console.log('processHomeAutomation');

        // Retrieve LUIS result for Process Automation.
        const result = luisResult.connectedServiceResult;
        const intent = result.topScoringIntent.intent;

        [...Make QnA call...]
    }
}

module.exports.DispatchBot = DispatchBot;

Thank you Stevan for your reply.
But this is not my problem.
1] I want to handle multiple intents for the same QnA maker in dispatch LUIS app.
While creating dispatch app we use below command:
dispatch add -t qna -i "QNA_ID" -n "KBNAME" -k "KBID" --intentName name_of_intent
Here name_of_intent will be the only one intent for full knowledge base in dispatch LUIS app.
How to distribute questions of knowledge base to different intents in "dispatch LUIS app"?
2] Is there any other way to handle intents without switch case?
What if I have 100 intents. It will be difficult to manage 100 intents in switch case or in if else

Closing as answered. @tejasvinishelar, please let us know otherwise.


kandy can't be loaded

Notebook somehow pickup repository from dependencyResolutionManagement.repository or project.repository company internal repo:
The problem is found in one of the loaded libraries: check library imports, dependencies and repositories
Error compiling code:
@file:Repository("https://packages.jetbrains.team/maven/p/kds/kotlin-ds-maven")
@file:DependsOn("org.jetbrains.kotlinx:kandy-lets-plot:0.6.0")
@file:DependsOn("org.jetbrains.kotlinx:kotlin-statistics-jvm:0.2.1")

Errors:
Failed to resolve [Dependency(value=org.jetbrains.kotlinx:kandy-lets-plot:0.6.0, hasSources=MAYBE, isMultiplatform=NO), Dependency(value=org.jetbrains.kotlinx:kotlin-statistics-jvm:0.2.1, hasSources=MAYBE, isMultiplatform=NO)]:
File 'org.jetbrains.kotlinx:kandy-lets-plot:0.6.0' not found
File 'org.jetbrains.kotlinx:kotlin-statistics-jvm:0.2.1' not found
Could not find artifact org.jetbrains.kotlinx:kotlin-statistics-jvm:jar:0.2.1 in nexus (http://mvnrepo.company.com/nexus/content/groups/public)

org.jetbrains.kotlinx.jupyter.exceptions.ReplLibraryException: The problem is found in one of the loaded libraries: check library imports, dependencies and repositories
	at org.jetbrains.kotlinx.jupyter.exceptions.ReplLibraryExceptionKt.rethrowAsLibraryException(ReplLibraryException.kt:38)

Ok, it's not Gradle's fault, it's maven's settings.xml in .m2 that resolver apparently using


[Question] NSP patch build/packing process.

Hello,
I noticed that on the website, the patch is available as an NSP update.
I have been able to find building NSPs for homebrew, but seeing an unofficial NSP built as an update for an official game is extremely rare, I haven't seen that done elsewhere yet (although I've been away from the Switch Homebrew scene for a while now).
Do you mind if I ask how the NSP is built?
Thanks.

Yes, I also want to learn

But I want to know why we don't modify exefs to change the file structure. Is it because of legal risks.
In my opinion, the legal risk of packaging NSP seems to be greater.

But I want to know why we don't modify exefs to change the file structure.
Is it because of legal risks？
In my opinion, the legal risk of packaging NSP seems to be greater.

But I want to know why we don't modify exefs to change the file structure. Is it because of legal risks.

Now that our project is effectively complete, we are releasing mkbktr, the tool we built to handle creating BKTR update NCAs for switch titles. As far as we could tell, no tools already existed for creating these archives, only reading them. Since they provide a compact and inherently supported mechanism for distributing binary patches, we elected to use them as our patch distribution mechanism.
The tool uses an Adler rolling checksum + strong MD5 checksum to detect reusable sections across the base NCA and patch NCA, generating relocation/patch entries as needed. The implementation is based entirely off of information gleaned from the SwitchBrew Wiki and the implementations for reading BKTR files found in hactool and Yuzu.
The program certainly has room for improvement - I didn't bother to handle patches large enough to require more than a single relocation bucket, there is no backtracking on md5 match to minimize copying within chunks - but it seems to work correctly, at least for this title. We hope that it proves useful for others in the community.


Created basic Epic Key analyzer with unit test

Created Epic Key analyzer per Issue #97. Please note, the regex may be improved on if there are invalid characters that should not match.
$ python -m unittest discover -s . -v -p "epickeyanalyzer_test.py"
test_match_negative (pastepwn.analyzers.tests.epickeyanalyzer_test.TestEpicKeyAnalyzer)
Test if negatives are not recognized ... ok
test_match_positive (pastepwn.analyzers.tests.epickeyanalyzer_test.TestEpicKeyAnalyzer)
Test if positives are recognized ... ok

Ran 2 tests in 0.001s

Pull Request Test Coverage Report for Build 168

35 of 36 (97.22%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.7%) to 63.588%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

pastepwn/analyzers/tests/epickeyanalyzer_test.py
30
31
96.77%

Totals

Change from base Build 164:
0.7%

Covered Lines:
1081

Relevant Lines:
1700

💛  - Coveralls


Instructions for  setting the MIP value are incorrect

As per https://docs.microsoft.com/en-us/azure/active-directory/users-groups-roles/groups-settings-cmdlets , you need to create a settings object to assign the value to the EnableMIPLabels property. Was this tested?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ef84a3e0-879e-4312-a89d-7a3bb0cd9a51
Version Independent ID: 43851f8a-b71f-0a0a-8b1f-629e3528972f
Content: Assign sensitivity labels to groups - Azure AD
Content Source: articles/active-directory/users-groups-roles/groups-assign-sensitivity-labels.md
Service: active-directory
Sub-service: users-groups-roles
GitHub Login: @curtand
Microsoft Alias: curtand

@GavinHaak Thanks for the feedback! We are currently investigating and will update you shortly.

Hi @GavinHaak
Thanks for your dedication to our documentation! Unfortunately, at this time the engineering team has been unable to review your issue in a timely manner and we sincerely apologize for the delayed response. We have been unable to resolve your query since the creation of this issue, and the timeline for resolution may vary based on resourcing, so we've created an internal work item to incorporate your suggestions. We are closing this issue for now, but feel free to comment here as necessary. #please-close


Loading kitematic without admin on windows docker toolbox deletes config

Expected behavior
Loading kitematic without admin privileges when virtualbox requires admin permissions to:

request permission or
state admin permissions are needed

Actual behavior
Loading kitematic shows "We're Sorry!"
Loading kitematic without admin permissions looks to try and resetup the default machine and in the process will delete all the config files and ssh keys in C:\Users\xxx\.docker\machine\machines\default
Steps to reproduce the behavior

Load kitematic with admin permissions - let it create the default vm in virtualbox
Close kitematic
Load kitematic without admin permissions
Check C:\Users\xxx\.docker\machine\machines\default for config.json and ssh keys

up


Rewrite the Discovery Client in TypeScript

overview
As a followup to #7274, we should port one of our smaller leaf-projects, the Node.js Discovery Client, from Flow JavaScript to TypeScript.
acceptance criteria

[ ] Discovery Client written in TypeScript instead of Flow
[ ] Flow type shims in place so existing Flow dependents do not break
[ ] PR used as input into general Flow > TS guide

Closed by #7331


Cb Comments Fix

fixed English for comments of cb function

"identity" refers to the _.identity function. This could be marked a little clearer though.


Doesn't denote if AppCenter.Start is additive or not

The documentation doesn't say whether you can call AppCenter.Start multiple times with different modules.
My use case is I'd like to init crash reporting as early as possible (in the native parts of my Xamarin.Forms project), while my shared Xamarin.Forms project would initialise analytics later on. It is not clear from the documentation whether you can call Start multiple times, and if you can, it doesn't tell whether the modules sent in are additive or not.
Thanks.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 5d72f6fe-74d6-696e-e709-5f6d577c0a58
Version Independent ID: b3d0ea8b-5a1f-bf71-7dc2-9685cf23c00b
Content: Get Started with Xamarin - Visual Studio App Center
Content Source: docs/sdk/getting-started/xamarin.md
Service: vs-appcenter-sdk
Sub-service: app-center-general
GitHub Login: @King-of-Spades
Microsoft Alias: kegr

Hi there, sorry for the delay! For the described purpose, our SDK has a special API: https://docs.microsoft.com/en-us/appcenter/sdk/other-apis/xamarin#change-state-of-service-in-runtime
Thus, you will need to call Start from the earliest point with a required service ("Crashes"):
AppCenter.Start("ios={Your App Secret};android={Your App Secret};uwp={Your App Secret}", typeof(Crashes));
and then use Analytics.SetEnabledAsync(true); in another part of your application.

Hi there, sorry for the delay! For the described purpose, our SDK has a special API: https://docs.microsoft.com/en-us/appcenter/sdk/other-apis/xamarin#change-state-of-service-in-runtime
Thus, you will need to call Start from the earliest point with a required service ("Crashes"):
AppCenter.Start("ios={Your App Secret};android={Your App Secret};uwp={Your App Secret}", typeof(Crashes));
and then use Analytics.SetEnabledAsync(true); in another part of your application.

Hi there, sorry for the delay! For the described purpose, our SDK has a special API: https://docs.microsoft.com/en-us/appcenter/sdk/other-apis/xamarin#change-state-of-service-in-runtime
Thus, you will need to call Start from the earliest point with a required service ("Crashes"):
AppCenter.Start("ios={Your App Secret};android={Your App Secret};uwp={Your App Secret}", typeof(Crashes));
and then use Analytics.SetEnabledAsync(true); in another part of your application.

Thank you! I really appreciate the answer.

Hi there, sorry for the delay! For the described purpose, our SDK has a special API: https://docs.microsoft.com/en-us/appcenter/sdk/other-apis/xamarin#change-state-of-service-in-runtime
Thus, you will need to call Start from the earliest point with a required service ("Crashes"):
AppCenter.Start("ios={Your App Secret};android={Your App Secret};uwp={Your App Secret}", typeof(Crashes));
and then use Analytics.SetEnabledAsync(true); in another part of your application.

Thank you! I really appreciate the answer.

Actually let me reopen this -- as it may make sense to add what you said to the documentation for the Start method.

Actually let me reopen this -- as it may make sense to add what you said to the documentation for the Start method.

We've updated the SDK docs on the separate start of App Center services, soon they'll be available on the portal.

We've updated the SDK docs on the separate start of App Center services, soon they'll be available on the portal.

Thank you!

Thank you!


Message-relayer-fast stopped due to MessageRelayerService._getStateBatchHeader error

{"level":50,"time":1625456689806,"extra":{"message":"TypeError: Cannot read property 'data' of null","stack":"TypeError: Cannot read property 'data' of null\n    at MessageRelayerService._getStateBatchHeader (/opt/optimism-message-relayer-fast/dist/service.js:201:128)\n    at runMicrotasks ()\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)\n    at async MessageRelayerService._isTransactionFinalized (/opt/optimism-message-relayer-fast/dist/service.js:215:24)\n    at async MessageRelayerService._start (/opt/optimism-message-relayer-fast/dist/service.js:109:23)\n    at async MessageRelayerService.start (/opt/optimism-message-relayer-fast/node_modules/@eth-optimism/common-ts/dist/base-service.js:33:9)\n    at async main (/opt/optimism-message-relayer-fast/dist/exec/run.js:89:5)\n    at async /opt/optimism-message-relayer-fast/exec/r

I added this.logger.info to the message-relayer-fast to help us debug.

{"level":30,"time":1626589816741,"msg":"Checking for newly finalized transactions..."}
{"level":30,"time":1626589816741,"height":6228,"msg":"Checking if tx is finalized"}
{"level":30,"time":1626589816752,"startingBlock":8956000,"endBlock":8958000,"msg":"Querying events"}
{"level":30,"time":1626589816887,"event":{"blockNumber":8956595,"blockHash":"0xa0ddd447560eda51d7792987e24efb68a64f932fd96cc4bdb7b3bff99057044e","transactionIndex":13,"removed":false,"address":"0x1ba99640444B81f3928e4F174CFB4FF426B4FFAE","data":"0xad8dc011d344abc26e13d8baa0a6b88724d2d093f2a3761dafc43ef28fec3b3d00000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000001854000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000060f3c0470000000000000000000000007f3cdbe9906fd57373e8d18aaa159fc713f379b0","topics":["0x16be4c5129a4e03cf3350262e181dc02ddfb4a6008d925368c0899fcd97ca9c5","0x00000000000000000000000000000000000000000000000000000000000008bd"],"transactionHash":"0x96509977129070bc041a2b5c367c5a2f7f524d388afd64fe575873e326ec67c1","logIndex":31,"event":"StateBatchAppended","eventSignature":"StateBatchAppended(uint256,bytes32,uint256,uint256,bytes)","args":[{"type":"BigNumber","hex":"0x08bd"},"0xad8dc011d344abc26e13d8baa0a6b88724d2d093f2a3761dafc43ef28fec3b3d",{"type":"BigNumber","hex":"0x02"},{"type":"BigNumber","hex":"0x1854"},"0x0000000000000000000000000000000000000000000000000000000060f3c0470000000000000000000000007f3cdbe9906fd57373e8d18aaa159fc713f379b0"]},"msg":"Got state batch appended event"}
{"level":30,"time":1626589816908,"transaction":null,"msg":"Got event transaction"}
{"level":50,"time":1626589816908,"extra":{"message":"TypeError: Cannot read property 'data' of null","stack":"TypeError: Cannot read property 'data' of null\n    at MessageRelayerService._getStateBatchHeader (/opt/optimism/packages/omgx/message-relayer-fast/dist/service.js:207:128)\n    at runMicrotasks ()\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)\n    at async MessageRelayerService._isTransactionFinalized (/opt/optimism/packages/omgx/message-relayer-fast/dist/service.js:221:24)\n    at async MessageRelayerService._start (/opt/optimism/packages/omgx/message-relayer-fast/dist/service.js:109:23)\n    at async MessageRelayerService.start (/opt/optimism/packages/common-ts/dist/base-service.js:33:9)\n    at async main (/opt/optimism/packages/omgx/message-relayer-fast/dist/exec/run.js:89:5)\n    at async /opt/optimism/packages/omgx/message-relayer-fast/exec/run-message-relayer-fast.js:6:3"},"msg":"Caught an unhandled error"}

Fixed.


Accessing group key using group::key

Hi,
I'm unable to access database translation using this syntax:
group::key
Many Laravel packages using this syntax on their views and controllers (Here for Laravel Voyager).
My Waavi/Translation is misconfigured or something else ?

Further informations are below :
On my database

On my controller:
Not working:
trans('voyager::generic.dashboard');
Works:
trans('voyager.generic.dashboard');

How did you import voyager's language files? You see, for packages I believe it's the namespace that should be set to voyager, generic would be the group and dashboard the item. If you loaded them manually then it failed.
I'd have to look more into it.


Opening a number of tabs at once can result in no Master

Repro:

Open a number of tabs (~5) with the supplied sample page crosstab/samples/index.html
Using IE(10) right-click one of the tabs and select "Refresh All" from the context menu

You may need to do this a few times to see the behaviour, but I usually get it within a few tries.  Once in this mode of no Master performing subsequent "Refresh All"s does not fix the problem.  One of the tabs appears to think it is Master (but is not reflecting that state) as refreshing the individual pages one by one eventually fixes the issue (on a specific tab).
While this may not be a typical user scenario I have seen it in production when a number of pages load at once (automatic logon after re-authentication).
I've also noticed that under heavy load (JS intensive pages) localStorage can become a bit flaky with the timing of events, causing similar issues with the election process (this is a bit hard to show repro steps for).

Does this only happen in IE / IE10? Or can you reproduce it in other browsers?

IE10 is the only browser I've managed to consistently reproduce the issue with.
In FF I see different behaviour, if I have 4 tabs open and perform a "Reload All" it thinks there are 8 tabs (and there is no master), until the 4 old tabs timeout, then a new master is elected and all is good.
Chrome doesn't have a "Refresh All" method so I can't test that, but I am seeing strange behaviour too.  Such as multiple tabs becoming master when the previous master tab navigates or is refreshed, and occasionally there is no master.
A year or more ago I wrote something similar to what you have and found these same issues to prevail.  Working with a lightweight sample all seems to go quite well but once integrated into a real application problems with timing of events start to arise.  My first theory was that localStorage was flaky so I changed to using Cookies as the message bus (just to test the theory), this also worked well with lightweight pages but not under load.
I'm not sure what the solution is, some sort of outer timer doing a sweep for these erroneous conditions, issuing demotions when multiple masters exist, and promotion if none exists?

It's very odd that there would be no master in an unrecoverable state, as all the tabs should be polling to check that there is a master and that no old tabs are present. I'll try to replicate this soon, maybe tomorrow. Thanks for bringing it up.
The behavior you're describing with FireFox happens because it isn't sending a pageunload event to the open tabs, so they don't have a chance to cleanup properly, but it's expected that none of the new ones will be master until the polling cycle completes and there's a new master election. I'll also look into that.

Currently unable to replicate this.


[Improvement idea] Implement read model optimized for frame decoders

In my opinion the read model isn't really optimized for frame decoders.
There should be a way to read directly a frame into a new buffer without both slice() and copy() etc. more like in C
So a frame decoder should read a header directly from the socket, then decide the frame's length, and then read exactly X bytes into new buffer (or keep trying).
The current model has following problems:

forced cumulation if data comes over X reads, copying bytes over and over
slice() creates per-read gc pressure because the slices aren't pooled so I am not going to use it, no way
even if I were to use slice(), I couldn't grow the buffer and would have to copy the data to a new one anyway
if more than one slice is used on the stream the other stream data could be kept forever etc. and not discarded until disconnect
copy() would copy the bytes without need as the improved model could just read to a new separately buffer directly

With my understanding, this should be advantage for the app's version implemented for example in golang - is that correct?
I was pretty sure that splice is used by default, but it looks like it's not - should I consider using LEVEL_TRIGGERED which uses splice?
https://github.com/netty/netty/blob/master/transport-native-epoll/src/main/java/io/netty/channel/epoll/EpollChannelConfig.java#L134
I am using autoread=false for both sides of a proxied connection, but I am a bit sceptical to it since I am very sensitive to latency, I am not entirely sure if it can be affected by this.
What CPU improvement could I expect in that case?

@ninja- - Are you familiar with RecvByteBufAllocator (also see the interfaces / types that inherit from this class)? You can use this to control how to allocate ByteBufs while preforming reads from the underlying transport mechanism. In 4.1+ you can use this to control the read loop too in terms of how many max total bytes should be attempted to read, or how many max read operations should attempted per channel when there is data to read (the later can also be done in older versions of Netty). If this interface is not adequate can you explain why?

@Scottmitch thanks for a tip. do you think it can be useful to create a more low-level decoder? Is it worth the time at all?
for example:
I consider the header to be max 10 bytes including all the pipeline things decryption etc.
would it make sense if I read these 10 bytes and denied reading more if there isn't at least X bytes in the buffer to avoid read cumulations?
But the read cumulations are one thing and the other thing is that the frame decoder needs to copy the data to the new buffer which will be decoded frame. On reflection, I think that this couldn't be optimized given that the pipeline contains other elements before a frame can be decoded (encryption etc.)

I switched to level triggered, it works fine but with one regression: it will starting missing last packets when the remote end disconnects, even if autoread=true. Problem goes away when epoll mode is back to edge triggered. is this a bug?

That woukd be a bug... Let me investigate

Am 04.10.2015 um 07:18 schrieb ninja<EMAIL_ADDRESS>I switched to level triggered, it works fine but with one regression: it will starting missing last packets when the remote end disconnects, even if autoread=true. Problem goes away when epoll mode is back to edge triggered. is this a bug?
—
Reply to this email directly or view it on GitHub.

@normanmaurer any news on this?

@ninja- nope not yet... had no time today. Would you mind open a bug-report for this and mention me in it ?

#4313

denied reading more if there isn't at least X bytes in the buffer to avoid read cumulations?
How will you deny reading more data? You are using encryption so how do you know how many total clear text bytes there are after decryption (without reading and doing the decryption)?

@Scottmitch I could get the first 10 bytes, get the 10 bytes through the pipeline to get the header decoded and then decide. But maybe it's too crazy to optimize anything with a visible CPU difference.

@ninja- - I'm still not sure I understand. Are your header and body encrypted? How do you correlate number of encrypted bytes (what you read off the wire) to decrypted bytes (what the application sees)?
I guess general advise is to only optimize if there is a need, and analyze your bottlenecks with tools to know where to focus your efforts.

@Scottmitch  yeah I dont think that was a good idea but I am hoping for some improvements after fixing epoll


Eager/deferred/lazy?

I'm interested in using this to model very large Xmls, but I'm concerned that the library is effectively eagerly reading all the data in, and will take enormous amounts of data.
Is it just a thin layer on top of XDocument? (which I think is largely deferred execution, i.e. IEnumberable<>) or is it a bit more eager?

Good question; looking at the technical documentation in the wiki, which is a copy of the technical manual Microsoft published back in 2011 and the source code for the XTypedElement  class, it does appear to use XDocument internally for much of its typed-based operations.
I can't give you a 100% certain answer, but I think it's a safe conclusion that it is basically a wrapper for XDocument with code-gen sprinkled on top. I do recall using the library to read some pretty large XML files, some exceeding 500MB in size and I recall the program I was using the library in actually used much less than that at runtime in terms of memory usage.


Stop remote harvesting: textbox without description

0.5.19-BETA
When you stop remote harvesting, a popup window is displayed and there is a textbox without a description:

that was leftover from some old version, I'm not sure why and how it got there


Switch default DB from LMDB to flat

LMDB is great, however -

the size on disk is much larger
on cloud instances the memmap requirements are a killer

... to “—db-type file”

Closed in https://github.com/polkadot-js/client/pull/212


How to verify Microsoft Graph API permissions for the managed identity service principal?

This page state: After executing the script, you can verify in the [Azure portal](https://portal.azure.com/) that the requested API permissions are assigned to the managed identity.
Where in the portal?

Document details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 15507fb4-6ebe-f0f0-96b6-f6c578747d3f
Version Independent ID: b2ae30fa-60ae-a30a-b17b-2de525cf0370
Content: Tutorial - Web app accesses Microsoft Graph as the app - Microsoft Entra
Content Source: articles/active-directory/develop/multi-service-web-app-access-microsoft-graph-as-app.md
Service: app-service-web
GitHub Login: @rwike77
Microsoft Alias: ryanwi

Ah, I realise that the subsequent documentation steps answer my question.


table.setRowsSynced doesn't seem to update with new content

My understanding of table.setRowsSynced (or using enableSync: true when creating the table) is that I should only have to set the rows' model once, and then I can just modify the model and it should then be reflected in the table. But my current implementation doesn't seem to do this
model: <some model of records> ,
sortedModel: sort('model', 'sortBy').readOnly(),

init() {
  let table = new Table(this.get('columns'), A([]), { enableSync: true });
  this.set('table',table);
  table.setRowsSynced(this.get('sortedModel'));
}

actions: {
  onColumnClick(column) {
    this.set('sortBy','someval:asc');
  }
}

The sortedModel gets sorted on the column click, but the table rows don't get updated with the new sorted model
I can add the following code, but then the rows get completely replaced and I lose things like which rows are selected & hidden - so ideally I want to be able to sort while keeping the existing rows in the table
setRows: observer('sortedModel', 'sortedModel.[]', function() {
  console.log('sorted the model');
  let sortedModel = this.get('sortedModel');
  let table = this.get('table');

  table.setRowsSynced(sortedModel);
}),

Is there something I'm missing here? Thanks!

The problem is that sort('model', 'sortBy') returns a new array instance every time the sort definition (sortBy) changes.
This really sucks, I know. It's been nagging me for quite some time also. What we'd need is an ArrayProxy that is clever enough to diff the underlying content array with the new instance and move existing rows around.
But I think that is outside of the scope of this addon and deserves its own project.

As an immediate remedy, you could compare the old and newly sorted arrays yourself and set the required state, like selected, hidden, expanded yourself.

Thanks for the explanation! This was bugging me for hours, and explains another ELT issue I was having a few weeks ago haha - at least I have a path forward now!

For anyone coming to this issue wanting to know more, I used the suggested immediate remedy solution of remembering the old rows' properties of selected, hidden, expanded and applied them to the newly sorted rows.
To do this you'll need to make sure your data records have some kind of unique identifier in order to be able to match them between sorts so you can apply the remembered row properties to the correct new row (this is fine / easy if using ember models directly in ELT, you can just use the id attr, - but I was using a raw JSON array for one of my tables)
Here's a sample of what worked for me - there are probably more efficient ways to do this;
setRows: observer('sortedModel', function() {
  let sortedModel = this.get('sortedModel');
  let table = this.get('table');

  // Remember old row properties
  let oldSelectedRows = table.get('selectedRows');
  let oldExpandedRows = table.get('expandedRows');
  let oldHiddenRows = table.get('rows').filterBy('hidden',true);

  // Set the new rows
  table.setRows(sortedModel);
  let newRows = table.get('rows');

  // Now apply the remembered properties (selected, hidden, expanded) to the new rows, based on their primary key
  let primaryKeyAttr = 'id';
  oldSelectedRows.forEach((oldRow) => {
    let oldRowId = oldRow.get(primaryKeyAttr);
    let matchingNewRow = newRows.findBy(primaryKeyAttr,oldRowId);
    if(matchingNewRow) matchingNewRow.set('selected', true);
  });
  oldExpandedRows.forEach((oldRow) => {
    let oldRowId = oldRow.get(primaryKeyAttr);
    let matchingNewRow = newRows.findBy(primaryKeyAttr,oldRowId);
    if(matchingNewRow) matchingNewRow.set('expanded', true);
  });
  let visibilityChanged = false;
  oldHiddenRows.forEach((oldRow) => {
    let oldRowId = oldRow.get(primaryKeyAttr);
    let matchingNewRow = newRows.findBy(primaryKeyAttr,oldRowId);
    if(matchingNewRow) {
      visibilityChanged = true;
      matchingNewRow.set('hidden', true);
    }
  });
  // If a row visibility changed, need to notify table that visibleRows changed, otherwise the table doesn't update for some reason
  if(visibilityChanged)	table.notifyPropertyChange('visibleRows');
}).on('init'),


Parametrise PR values

This PR will:

[x] Remove the 'auto' action

This is a breaking change, so will need to be included in the v2 release

[ ] Recommend using two separate workflows to deploy and remove previews respectively
[ ] Add a recommendation for forks using workflow_run, which fixes #3
[x] To be able to call the action from a non-pull_request event, values that the action needs that previously came from the pull request directly will be parametrised
[ ] Create a bunch of examples for different use cases in dedicated dirs, including the aforementioned

Instead of just adding md readmes, make a simple Jekyll site and move examples there (possibly in another PR)

[x] Make the source-dir input no longer required, and explicitly fail if it is given when action=remove

@rossjrw hey, just found this action a perfect-match to our needs if pr previews can be enabled from forks. so, wondering if this pr can be merged soon?

@ryoqun Not soon, no - before I'm willing to release this kind of change I need to be sure that I can appropriately document the horrible security risks of allowing unmoderated forks from deploying code to your repository, and how to work around it.
I took a glance at your profile and it looks like you're using a different repo entirely to contain preview deployments, which seems very sensible. If you need this functionality right now, you could look at extracting the key steps from https://github.com/rossjrw/pr-preview-action/blob/022361539c71c58a7141d4fe8c3e0e4a1c34f9c5/action.yml and putting them in a GitHub Actions workflow directly in your main repo, with the necessary parameters. Alternatively, it looks like https://github.com/ACCESS-NRI/pr-preview-action/issues/1 had some luck using the commits from this as-yet-unmerged PR to parametrise the PR values.
As always my top recommendation is to use a tool that's actually designed to do this reliably and is maintained by people who are paid to maintain it well, e.g. Vercel, Netlify.

@rossjrw I'm wondering if you considered using the pull_request_target event. The issue I've encountered is with a PR from a fork trying to push to the gh-pages branch on the origin repo. The pull_request_target enables this, but in order to deal with the security issues you point out, it's necessary to isolate the workflows and even set it so that previews from forks require manual approval.

Is there a reason why forked repos PR's couldn't be hosted on the forks gh-pages? That would bypass the security risk, no?

Is there a reason why forked repos PR's couldn't be hosted on the forks gh-pages? That would bypass the security risk, no?

@zerothi That's an interesting proposal. I'd like to see a proof-of-concept to explore in more detail, but just off the top of my head / with cursory research:

Forking a repo doesn't automatically deploy a Pages site - the forker would still need to go into the forked repo's settings and turn it on themselves. That's probably feasible with clear enough instructions - the inital workflow run could leave a comment instructing the forker to do so, but it wouldn't then be able to automatically update when/if the forker fixes it
The coupling between 'branch in forked repo' and 'pull request in original repo' is one-way - the PR knows about the fork but the fork doesn't know about the PR. It'd be difficult (impossible?) to run the preview workflow in the forked repo with the right parameters without doing some sort of lookup on the original repo's PR, assuming that information is available (it's possible that the forked repo's github token doesn't even have read access to the original repo). (To be fair, off the top of my head the only information that's needed is the PR number to create the URL/directory - and that could be replaced with the branch name)
I don't know if it would be possible to have the workflow in the original repo (that leaves a comment) know anything about the workflow in the forked repo (that creates the preview). It would be good to have the comment only be posted if the deployment is successful, but it might not be possible to determine that. (re point number 1, it might not be possible to determine if the fork has Pages enabled)

True, I guess such a single shot thing would be ok to leverage for the committer, but of course, automating this later would be ideal.
I see, that is of course an issue, hmm... That may be a deal breaker here. Access to non-PR branches greatly limits its functionality.
Hmm, I see. I can also understand that this may be problematic for other security reasons, say if the PR requires some tokens for running.

I definitely appreciate and applaud the caution with regard to security here; however aren't these risks precisely what https://docs.github.com/en/actions/managing-workflow-runs/reviewing-deployments is designed to solve?  Personally I'd be fine with allowing workflows from forkers if I have to approve them before they can run.
Maybe I'm missing something but it doesn't feel to me like you need to carry the whole responsibility of making this safe.  Ultimately if you issue clear caveats and document the safety considerations, then it's up to users of the action to act responsibly based on that.  And if their repo breaks, they get to keep the pieces ;-)

I also see that #38 was merged, so the pattern of deploying to a separate repo is an option, with the security isolation that provides.  Nice!

@aspiers You're right - but it is my responsibility to avoid implying that things are guaranteed to be safe and secure. I don't know of any instances of malicious actors abusing this preview action and I'm very keen for it to stay that way.
This action is really very basic and most of the upcoming work is about documenting how to use it. You can do almost all of the things you describe right now using different configuration options around the action - e.g. the pull_request_target event gives a forked workflow permission to access the base repo, you can add deployment reviews if you like, as you pointed out you can even deploy to a different repo. There are a few things you can't do because of hardcoded PR information but that's what this PR #6 is for, to make the action even more generic.
I'm sure these options are valid approaches for an experienced user, but I feel that packaging an Action carries the implication that anyone can drop it into a project and use it regardless of skill level. I don't want to present these nuanced use cases to a user who might not understand the caveats that come with them, so in the meantime, stating that forks aren't supported is easiest on my end. But I certainly can't stop you from finding clever workarounds that work for your use case!

Thanks a lot for the reply an extra info - I think we're probably in violent agreement :-)  I wrote "Ultimately if you issue clear caveats and document the safety considerations" so I certainly wasn't advocating for you implying any misplaced guarantees around safety, or facilitating a more risky approach without the appropriate warnings.  Perhaps where we may slightly differ in opinion is that I generally prefer for advanced users not to be limited through aiming to protect users who completely disregard their responsibility for their own security.  But I can see arguments for both sides :-)

I've manually set this up for egui based on pull_request_target and it works great!
Here is the PR in case someone is interested: https://github.com/emilk/egui/pull/5131
I've created a separate github org to host the preview deployments, so they are deployed on a separate domain not directly associated with the project.

Hey @rossjrw,
Can we do something like this to run the workflow only if it's approved by the maintainer?
on:
pull_request_review:
types: [submitted]
jobs:
approved_job:
if: >
github.event.review.state == 'approved' &&
(
contains(github.event.review.author_association, 'COLLABORATOR') ||
contains(github.event.review.author_association, 'OWNER')
)
runs-on: ubuntu-latest
steps:
  # Your steps here

This will ensure the workflow runs only if it's approved by a collaborator or the owner of the repository.
Will it still have security issues ?

Hey @rossjrw,
Can we do something like this to run the workflow only if it's approved by the maintainer?
on:
  pull_request_review:
    types: [submitted]

jobs:
  approved_job:
    if: >
      github.event.review.state == 'approved' &&
      (
        contains(github.event.review.author_association, 'COLLABORATOR') ||
        contains(github.event.review.author_association, 'OWNER')
      )
    runs-on: ubuntu-latest

    steps:
     # Your steps here

This will ensure the workflow runs only if it's approved by a collaborator or the owner of the repository.
Will it still have security issues?

@Spiral-Memory It will have the same security issues. Best case scenario is that your contributors thoroughly review every revision of every pull request, ensuring no malicious attacks get through. Worst case scenario is that your contributors don't review thoroughly enough each and every single time, or, if you allow previews for new commits in PRs that have at some point been approved, the malicious actor simply adds the malicious content post-review.
The security hole is still there, even if there's a manual step in the way. It's up to you where to decide the right balance between security and convenience for your project. Your solution is possible, as far as I can see, but as the developer of this Action I can't recommend it.

Thank you so much for your response, @rossjrw .
Yes, I understand the security risks involved. However, I plan to ensure that workflow will run only if the PR is approved by the maintainer or owner of the repo, and it won’t run after approval by any other contributor, only by maintainers.
Also, I’m planning to separate the build workflow (which will be in the pull_request trigger) and deployment workflow, which will only run on the pull_request_target trigger, so that malicious actors can't execute any scripts while the project is building.
Though I’m not sure, and would appreciate confirmation, I think if someone adds a new commit post-approval, GitHub removes the approved tag. I haven’t tested it yet.
So, in my opinion, the security risk is the same as merging the PR. As you pointed out, it’s about balancing convenience and risk, and this approach feels suitable to me.
I would love to hear your thoughts on any other security risks I should consider while implementing this.

Just wanted to echo what others have said about the utility of allowing runs on PRs from forks, although the security implications are of course important to consider. Requiring maintainer approval seems like a good idea, but yes, that trusts that maintainers always thoroughly review everything. Something I would personally add for my repo would be "don't allow this workflow to run at all on fork PRs if anything other than the text source files [RestructuredText, .rst] are touched." We have other ways to test our builds that we would fall back to in such cases.


Chapter 2 document has too many summaries

I'm looking holistically at the problem of slow catch up.
I.e. loading from previously cached summary that might be a day old and spending substantial time to catch up.
There are various independent bugs opened based on size of the file / ops, this is one of them.
File has 633K ops, Last summary is at op 334K, i.e. half of ops are not summarized (due to bug with PUSH and stuck client in quorum).
There are almost 9K summary ops (and I'd assume as many summaryAk), which creates quite a few ops relative to total payload.
I.e. roughly 5% of payload is summary-related ops.
That means there are plenty of cases where we summarize less than 20 ops, and likely enough cases where we summarize less than 10 ops (including previous 2 summary related ops).
Can we be more efficient here?
Like have different time limits for when you have 10 vs. 100 un-summarized ops?
Can we find something simple that improves it 2x?
More analyzes is welcome! This issue is opened based on high-level observations.

I made a change a while back to support intervals for the configuration. I think that would have covered this. Based on the outstanding ops, it would choose different time configurations (so the current "maxOps" would map to a zero time).

Would a solution like this be suitable? #60

I think so. I'm not sure I buy into (general) direction of server controlling these things, I feel it's very payload specific.
I.e. if I'm building a game, my numbers would be quite different, vs. if I'm building a document container.

Related to this change: idle timer starts too frequently. It shouldn't start if there are no "data" ops since last summary, but sometimes it appears to be.


src/index.md is read-only

I just run newsite("blegat") and the index.md and config.md are read-only, is that expected ?
$ ll
total 40K
drwxr-xr-x 5 blegat users 4.0K Oct 11 11:10 assets
drwxr-xr-x 2 blegat users 4.0K Oct 11 11:10 css
-rw-r--r-- 1 blegat users 2.1K Oct 11 11:10 feed.xml
-rw-r--r-- 1 blegat users  14K Oct 11 11:10 index.html
drwxr-xr-x 4 blegat users 4.0K Oct 11 11:09 libs
drwxr-xr-x 2 blegat users 4.0K Oct 11 11:10 pub
drwxr-xr-x 5 blegat users 4.0K Oct 11 11:09 src
$ cd src
$ ll
total 28K
-r--r--r-- 1 blegat users  780 Oct 11 11:09 config.md
drwxr-xr-x 2 blegat users 4.0K Oct 11 11:09 _css
drwxr-xr-x 2 blegat users 4.0K Oct 11 11:09 _html_parts
-r--r--r-- 1 blegat users 8.3K Oct 11 11:09 index.md
drwxr-xr-x 2 blegat users 4.0K Oct 11 11:09 pages

I would expect to have -rw... instead of -r-... for index.md and config.md.
julia> versioninfo()
Julia Version 1.2.0
Commit c6da87ff4b (2019-08-20 00:03 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)

Ah crap, no, I had an issue with this a short while back and tried to nuke it chmoding everything but it looks like the issue is still there.
Apologies for that and thanks for reporting, I'll try to fix it asap

JoDoc v0.3.4 and JuDocTemplates master also work.

Yeah newsite is provided by JuDocTemplates so the JuDoc version doesn't matter.
I still need to investigate though, I'm baffled this happened in the first place; since newsite copies a bunch of files maybe I did something silly there.
Anyway thanks for reporting and glad it's working!

Btw, a bunch of bug fixes accumulated and are pending release 0.3.5 so for now you're probably best on JuDoc#master anyway 😄

It seems that Julia creates files with different access rights when doing ] add or ] dev
$ ll -R /home/blegat/.julia/dev/JuDocTemplates
...
/home/blegat/.julia/dev/JuDocTemplates/src/templates/vela/src/_html_parts:
total 8.0K
-rw-r--r-- 1 blegat users  316 Oct 11 11:36 foot.html
-rw-r--r-- 1 blegat users 1.5K Oct 11 11:36 head.html
$ ll -R /home/blegat/.julia/packages/JuDocTemplates/C6jgR
...
/home/blegat/.julia/packages/JuDocTemplates/C6jgR/src/templates/vela/src/_html_parts:
total 8.0K
-r--r--r-- 1 blegat users  316 Oct 11 11:01 foot.html
-r--r--r-- 1 blegat users 1.5K Oct 11 11:01 head.html

Thanks for that and that also explains why I didn't see it; I guess I can just squeeze a chmod in the newsite function to make sure... trying this now

Ok I think this is fixed now. I'll tag a release in a minute.
I tried activating a new environment and
] add JuDoc JuDocTemplates#master
newsite("blah")

the perms are now 644.
Thanks again for the heads up!


Proposal: Automatically select the namespace if there is only one option after fuzzy search

This PR will make kubectx and kubens select namespace directly if there is a single result based on fuzzy match.
Note: This will work only when fzf is enabled
Example:
Lets say we have three namespaces.
$ KUBECTX_IGNORE_FZF=false kubens
default
kube-public
kube-system

With this PR,
kubens def # will change to default namespace
kubens sys # will change to kube-sytem namespace
kubens pub # will change to kube-public namespace
kubens ksy # will change to kube-system namespace
kubens kube # will take you to interactive session where we have to select between kube-public and kube-system

Similarly, it will work for kubectx

I signed it!

⚠  NOTE: Please make a proposal next time you write the code for it so your work is not discarded when we don't agree on a change.

As for your change, I'm against these sort of fuzzy behavior. We incorporated fzf for two reasons:

it gives you ability to scroll between options using arrow keys
you can filter down options by writing some queries

I am not willing to add more features beyond simple integration with fzf. Here's why:

This feature would only work with fzf. And it would complicate documentation.
It creates different behavior for the same command when fzf is installed vs not.  This is not desirable.
FZF employs a fuzzy search, it's not a substring search.

If we don't make this change, you'll basically be typing only more keystroke (you can type those queries while in fzf interactive mode, and hit Return).
For these reasons I'm currently -2 for this change unless you have counter-arguments to change my mind.

Sure @ahmetb. I am relatively new to this OSS community. From next time, I will create an issue describing the request/proposal and if everyone agreed, we can start to implement the idea.
Few points that I feel this feature is useful:

I agree that it will give a different behavior for the same command when fzf is installed vs not. I am seeing this very similar to command kubens. Without fzf, kubens will list the namespaces and with fzf, kubens will take us to the interactive mode where we can filter, navigate and select  a namespace (interactive mode makes our life easier to select a namespace)
With this we can easily change the namespace with one command + less keystrokes
kubens kube-systems vs kubens sys.
If there are  multiple namespaces, we have to enter into interactive and query it and select it. Here if we remember the name vaguely(fuzzy), we can select the namespace in one command (no interactive, no extra typing, no extra enter)

So I feel this will increase the productivity of the user. And I totally agree we have to add more docs to explain this feature(I would love to do that)

With this we can easily change the namespace with one command + less keystrokes
kubens kube-systems vs kubens sys.

It's still 1 command, and as I said 1 keystroke fewer is not a good argument.

If there are  multiple namespaces, we have to enter into interactive and query it and select it. Here if we remember the name vaguely(fuzzy), we can select the namespace in one command (no interactive, no extra typing, no extra enter)

Here's a better counter-argument: A lot of people run kubectx or kubens to see the namespaces available. And most users start with 1 context. Therefore, making a choice instead of switching breaks that behavior.
If I'm running kubectx it means:

(in interactive mode) I want to see a list of {contexts,namespaces} and (maybe) choose one.
(non-interactive mode) I want to see a list of {contexts,namespaces}

Many people run kubectx+fzf and hit Ctrl+C to exit, because they came there to see the list. I don't think we should make this command have side-effects when there's only 1 item to show.


Ubuntu Kernel  5.11.0-40 issues.

It wont run at all on Ubuntu 20.04 with 5.11.0-40.  Below is error in the kernel ring buffer - dmesg when 5.11.0-40 is up:
[ 30.391112] module: x86/modules: Skipping invalid relocation target, existing value is nonzero for type 1, loc 000000004e8c48c2, val ffffffffc14c3d80

This problem extends to Kernel 5.13.0-22. Same message multiple times in dmesg.
I have tried the new 5.5 Beta release, however no joy...

+1
any workaround?

Just tried this on Ubuntu 22.04 running 5.15.0-33 and the problem remains
(I had intentionally delayed upgrading since DisplayLink would not support non-LTS releases, but I guess I was over optimistic...)

BTW: Also 5.15.0-35

For those who know how to dig deeper: here is the output from the dlsupporttool.
DLSupportTool_Output_2022-06-05T18:20:32.408950.zip

facing the same isse with 22.04 and kernel 5.15.0-40-generic. Any workaround?

Strange issue.
If this still reproduce, I would suggest first to install latest displaylink driver and update kernel in your distro.
Disable secure boot  in bios can also help.
Please reproduce it on latest evdi driver and updated linux kernel.
I am meanwhile closing this issue. Please raise new issue if this still happening.


Support cascading dropdown with restful API in matrix row

You may refer to the column value of the same row with "row." prefix. Here is the example:
{ type: "matrixdynamic", name: "salesByCountry", title: "Please set sales by country", addRowText: "Add new Country",
        columns: [{ name: "region", title: "Region", isRequired: true,  choices: ["Africa", "Americas", "Asia", "Europe", "Oceania"]},
        { name: "country", title: "Select the country...", isRequired: true,
         choicesByUrl: {url: "https://restcountries.eu/rest/v2/region/{row.region}", valueName: "name"} },
            { name: "sales", cellType:"text", isRequired: true, inputType:"numeric", title: "Sales" }],
        rowCount: 2}

I am trying to do the same but with panel dynamic. The restful request to pupulate the second dropdown does not seem to take the correct parameter (titleName insted of valueName) from the first dropdown
https://plnkr.co/edit/sv6pYceDt00QFhupVfTW?p=preview

@masciugo Hm... It looks like a bug to me. I will investigate it.
Thank you,
Andrew

@masciugo Thank you for reporting this issue to us. It has been fixed. A Unit test has been added.
The fix will be available in the next minor update on the next week.
Thank you,
Andrew


dua 2.29.0 fails to compile with rustc 1.80.0

rust version

$ rustc --version
rustc 1.80.0

compiler error:

error[E0282]: type annotations needed for `Box<_>`
  --> /home/x/cargo/registry/src/index.crates.io-6f17d22bba15001f/time-0.3.34/src/format_description/parse/mod.rs:83:9
   |
83 |     let items = format_items
   |         ^^^^^
...
86 |     Ok(items.into())
   |              ---- type must be known at this point
   |
help: consider giving `items` an explicit type, where the placeholders `_` are specified
   |
83 |     let items: Box<_> = format_items
   |              ++++++++

For more information about this error, try `rustc --explain E0282`.
error: could not compile `time` (lib) due to 1 previous error
warning: build failed, waiting for other jobs to finish...

The underlying issue has been fixed in time crate 0.3.36 version which was updated in 69a7cdd11068b2e4053460b42d8a4959501a7799 commit.

Where is the source code from?
Did you try cargo update?

I downloaded the source tarball from release https://github.com/Byron/dua-cli/archive/refs/tags/v2.29.0.tar.gz and ran cargo build in that.
cargo update does pull latest time crate 0.3.36 version.

Thanks for the information! I understand now what's happening and believe to have fixed it - future releases should not contain the Cargo.lock file anymore and automatically build with the latest dependencies.
Retroactively this can't be fixed, so it's as good as it gets.
Closing for that reason, thanks for your understanding.

@Byron thank you for the fix! Would it be possible to cut a new release containing the fix?
Nixpkgs is tracking the packages broken by Rust 1.80 here and the preferred way is to do a version update once there is a release with the fix.

No problem at all, here it is: https://github.com/Byron/dua-cli/releases/tag/v2.29.1

Thank you for the rapid fix! I hope you can consider re‐adding the Cargo.lock file but just bumping the time crate. Having a set of upstream known‐good versions to build against is valuable, it helps the build process with hermetic distributions like Nixpkgs/NixOS, and it can also be useful for suppy‐chain security. The Cargo manual has some discussion about this; it used to unambiguously recommend versioning the lock file for an application, though it has some more nuance these days. Generally Cargo has moved in the direction of encouraging versioning more lock files; for instance, in 2023 it stopped ignoring them by default for even libraries.

By the way, it looks like it was only actually removed from the GitHub archives, so we could get around this with forceFetchGit.
How long until the first supply chain attack that exploits the surprising amount of malleability GitHub archives have…?

I think it's all trade-offs and here I'd prefer to get the latest greatest, hoping it will 'make things better rather than worse'. Either way, keeping old versions or using the latest ones, can be the wrong decision, so maybe it helps that I do dependency upgrades and updates only rarely, and somewhat randomly.

An option you might be interested in is using a bot like dependabot to automatically do SemVer‐compatible upgrades to your lock file, and running with --locked in CI. You can, I think, even set up dependabot PRs to automatically merge if CI passes. That ensures that the test suite always runs successfully with your locked dependencies, and that they’re always as up to date as possible, while still fixing versions for release tags that can be used for distributions, reproducible builds, and supply chain scanning.

Thanks - dependabot indeed seems like an option. Typically I don't set it up as I don't fancy figuring out how its YAML works, while wanting settings that avoid too much noise.


Documentation

The lit-element-starter-ts repo came with some facilities for generating documentation. However, these are somewhat limited in scope and intended for a single element rather than a repository full of elements. The purpose of this epic is to tailor the documentation to our repository with an emphasis on end users of the components.

Completed with PR #39.


Idea dump for truely async resource loading

currently an entire resource file is loaded into a memory buffer, then a GL resource is created, and the memory buffer content is copied over; the copying happens in the main thread
it should be possible to create and fill resources (vertex buffers, index buffers, textures) in a separate thread using multiple GL context and shared resource lists (https://www.opengl.org/wiki/OpenGL_and_multithreading)
it should be possible to incrementally load a file via small fixed-size buffers to avoid memory spikes
new set of thread-safe resource creation functions called from IO thread, these take a SETUP object, and return an array of pointer/size pairs (e.g. pointer to vertex buffer, index buffer, texture mips), these methods must be THREAD-SAFE, the resource slot has already been reserved on the main thread in the Gfx::Resource().Load() method:

Array BeginSetupThreaded(Id id, const SETUP& setup);
void EndSetupThreaded(Id id);

between BeginSetupAsync() and EndSetupAsync() a loader thread will fill the returned memory areas and when done will call EndSetupAsync(), during this, the resource slot will be locked with a separate flag. Once the locking flag is cleared, the resource pool slot will switch from Pending to Valid (on the main thread), from then on it is safe to use the resource
question: how to implement destroying the resource while an async load is in progress? => mark as deleted, let resource loading finish, then destroy on main thread(?)
question: one separate loader object per resource? loader probably needs to keep and update tracking state during the loading
loader needs a generic OnDataAvailable(void* ptr, int32 size) method which can be called with any size by an IO thread (or on some platforms on the main thread), usually loader needs to read enough data until a file header is complete, then call BeginSetupAsync() to create the GL resource(s), and write the rest of the data into the returned pointers.
question: how to communicate loading errors and 'finished' to loader?

No threaded loading with multiple GL contexts, instead (later) for desktop GL (or all implementations which have mapping: (pre-)allocate GL resource(s), map-buffer in main thread, fill in worker thread, unmap in main thread, render. Probably need to separate between loader thread(s) and resource creation thread.

How a loader should actually work:

complete resource loading and creation in several steps:
on main thread: allocate resource id, set resource state to pending, create and fire ioRequest, watch for ioRequest to be handled
on IO thread: have an optional method to decode loaded data (e.g. to parse a file format into a canonical format), but don't create any GL resources in loader thread.
back on main thread (after ioReq has been marked as handled), either create resource directly, or delegate to a resource creation thread
Loader sets resource state to Valid or Failed, and is done
GfxResourceContainer should put all Loaders into a queue and trigger them as long as loader is not done.

Perfect resource streaming only possible on GL4.x+ (http://www.slideshare.net/CassEveritt/approaching-zero-driver-overhead). So need to implement at least 2 paths for GL2.1+exts, and "full AZDO" with GL4.x.
Also: pre-allocate and map one big vertex buffer, and let dlmalloc (or other existing allocator) do the allocation in this buffer? Or allow true streaming only with restrictions? (like knowing the required resource buffer sizes beforehand?)

or screw it for the the old GL2.1 path and implement it only on modern hardware ?


tsc cannot resolve the node temp module

I am trying to use node temp, but tsc is not finding the module. When I use this module in a raw javascript project, I have no problems.
I am attaching a very simple project that illustrates the problem. To reproduce:

npm install temp
create file.ts in same location with the following contents: import temp = require('temp');
tsc --module commonjs file.ts:
BUG: compiler output is:

file.ts(1,23): error TS2307: Cannot find module 'temp'.

You can see that this module is actually installed by running node from the terminal in the same directory:
$ node
> console.log(require('temp'))
{ dir: '/var/folders/36/yjgv05hn78b4xn331jdt8klrcvt5zc/T',
  track: [Function],
  mkdir: [Function: mkdir],
  mkdirSync: [Function: mkdirSync],
  open: [Function: open],
  openSync: [Function: openSync],
  path: [Function],
  cleanup: [Function: cleanup],
  cleanupSync: [Function: cleanupSync],
  createWriteStream: [Function: createWriteStream] }

Is the module itself somehow malformed?

You'll need to declare the structure of the "temp" module with a .d.ts file.
As a workaround, you can also just write something like the following:
declare function require(moduleName: string): any;

var temp = require("temp");

Hmmm...it works if I use var instead of import. Why is that?

My understanding is that when TypeScript originally did imports, we had AMD and CommonJS. It's not entirely clear when you're using one and not the other, and users often want to target both, so import temp = require("temp") was created as a specialized syntax.
As a side note, I'm working on a definition for you right now since this seems to be a simple-enough module to type. :smiley:

@aeisenberg can you check out https://github.com/DefinitelyTyped/DefinitelyTyped/pull/7234 and tell me if that type definition looks okay?

Commented on the commit. Looks great to me.
So, when using commonjs, are we better off using var or import? We had been using import, but it seems like there's a problem with that.

Alright, so my definitions for the temp library have gone in. Here's what you should do from here:

Install tsd by running npm install tsd
Use tsd to grab the type definitions by running tsd install temp --save
Add typings/tsd.d.ts to the list of files you're compiling.
Use import * as temp from "temp";

Let me explain each of these so I'm not just pushing you into a dark room and telling you to try and find the lights:

(1) tsd is sort of like npm for definition management. It's managed by DefinitelyTyped, a community dedicated to making distributing typings easier.

(2) tsd install temp will grab the typings for temp. Those typings will be placed in a folder called typings, where a file called tsd.d.ts will also be created.
--save will create a tsd.json file so that you can run tsd install on any computer and all your dependencies will get grabbed. It's very similar to a package.json file.
As a note, you shouldn't check in your typings folder to source control, just like you wouldn't put node_modules under source control either.

(3) Recall that tsd install created a tsd.d.ts as well. That file refers to all the typings you installed earlier.

(4) JavaScript introduced a new and standardized import syntax, so what we all should be using is
import * as temp from "temp";

However, there are certain times when that won't entirely work. If you've ever seen something like
module.exports = function Foo {

}

the way you'll need to import that is using the older style import syntax that TypeScript supported
import temp = require("temp");

Makes sense. Thanks for your help. Using it now. It's still a bit confusing on how many different ways there are to import modules. But this is probably not the proper forum to ask the question. I'll get a stackoverflow question together about this.

Yup, StackOverflow and our Gitter channel are usually better venues for this sort of thing.
Modules are a surprisingly subtle in JS, and it's pretty common to discover how when starting with TypeScript. We're in the process of trying to add more documentation on it (https://github.com/Microsoft/TypeScript-Handbook).


Concurrency: repair the build after #34902

libdispatch is not part of the system on Linux and Windows, and dispatch
has not been used for the standard library up until this point.  The
current usage is limited to the Apple platforms, so rather than adding
another build of dispatch, conditionally include the header instead.

Replace this paragraph with a description of your changes and rationale. Provide links to external references/discussions if appropriate.

Resolves SR-NNNN.

CC: @jckarter @DougGregor

@swift-ci please smoke test

@swift-ci please smoke test Linux platform


Node API + handle mail responses

This PR moves the submission code / sending emails to work directly with the express server via a post  route.

E-mail responses are now checked redirected per https://github.com/cds-snc/ircc-rescheduler/issues/237#issuecomment-404603417
Confirmation page has been updated to handle a failed user email
Logic for emails / templates was cleaned up a bit to get the code out of the route itself

👇 also, our CircleCI stuff is failing because it can't run the API tests anymore.
The config for it is all in here: .circleci
I think we want to:

remove api: and everything under it
remove deploy_api: and everything under it
remove - api and - deploy_api from workflows: (lines 106 -> 114)


Can you provide or package the scss file used to generate this theme?

I like it but need to modify it a bit for my purposes.

Losed it, sorry, but here's what I've used in development.
Element theme packager
Also sorry for the late answer, bit busy these days.


Exception in Azure SDK for CPP when access deferred result of async blob delete batch

Which service(blob, file, queue, table) does this issue concern?
blob
Which version of the Azurite was used?
3.22.0
Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)
npm
What's the Node.js version?
v18.14.2
What problem was encountered?
Get exception when call DeferredResponse::GetResponse() following BlobServiceClient::SubmitBatch() or BlobContainerClient::SubmitBatch()
Steps to reproduce the issue?
Could use the CMake project in repo https://github.com/adamhan-hitachi/test-azure-sdk-cpp/commits/main.
2023-02-24T12:35:24.854Z 9b3d53cd-9053-4cd5-aa7a-52e33a7d83c8 verbose: SerializerMiddleware: Start serializing...
2023-02-24T12:35:24.854Z 9b3d53cd-9053-4cd5-aa7a-52e33a7d83c8 info: EndMiddleware: End response. TotalTimeInMS=7 StatusCode=201 StatusMessage=undefined Headers={"server":"Azurite-Blob/3.22.0","etag":""0x201006C7D5F0640"","last-modified":"Fri, 24 Feb 2023 12:35:24 GMT","content-md5":"ymnDrMhuVrmVqie7pvKs3A==","x-ms-client-request-id":"d7ebcfab-8298-4e99-5110-40d13b437920","x-ms-request-id":"9b3d53cd-9053-4cd5-aa7a-52e33a7d83c8","x-ms-version":"2021-12-02","date":"Fri, 24 Feb 2023 12:35:24 GMT","x-ms-request-server-encrypted":"true"}
2023-02-24T12:35:24.854Z 9b3d53cd-9053-4cd5-aa7a-52e33a7d83c8 verbose: FSExtentStore:readExtent() Read stream closed. LocationId:Default extentId:516f2df9-30e6-4951-92dd-27ec670894cd path:/gfs/azurite/blobstorage/516f2df9-30e6-4951-92dd-27ec670894cd offset:64149718 count:100 end:64149817
2023-02-24T12:35:24.856Z b10e92cf-afdb-4b8b-a634-754eda3f19db info: BlobStorageContextMiddleware: RequestMethod=POST RequestURL=http://localhost/devstoreaccount1?comp=batch RequestHeaders:{"authorization":"SharedKey devstoreaccount1:QW/9uC7XhYDH0D4QOSICvbL7GExKLXg8LLiyaQjwQQM=","content-length":"533","content-type":"multipart/mixed; boundary=batch_35f851ae-7e1d-45db-52ed-28a9542a959a","host":"localhost","user-agent":"azsdk-cpp-storage-blobs/12.7.0-beta.1 (Linux 5.10.0-21-amd64 x86_64 #1 SMP Debian 5.10.162-1 (2023-01-21))","x-ms-client-request-id":"9a7a3a25-dfa2-40ca-6672-47c1058fb470","x-ms-date":"Fri, 24 Feb 2023 12:35:24 GMT","x-ms-version":"2021-12-02"} ClientIP=<IP_ADDRESS> Protocol=http HTTPVersion=1.1
2023-02-24T12:35:24.856Z b10e92cf-afdb-4b8b-a634-754eda3f19db info: BlobStorageContextMiddleware: Account=devstoreaccount1 Container=undefined Blob=
2023-02-24T12:35:24.856Z b10e92cf-afdb-4b8b-a634-754eda3f19db verbose: DispatchMiddleware: Dispatching request...
2023-02-24T12:35:24.856Z b10e92cf-afdb-4b8b-a634-754eda3f19db info: DispatchMiddleware: Operation=Service_SubmitBatch
2023-02-24T12:35:24.856Z b10e92cf-afdb-4b8b-a634-754eda3f19db verbose: AuthenticationMiddlewareFactory:createAuthenticationMiddleware() Validating authentications.
2023-02-24T12:35:24.856Z b10e92cf-afdb-4b8b-a634-754eda3f19db info: PublicAccessAuthenticator:validate() Start validation against public access.
2023-02-24T12:35:24.856Z b10e92cf-afdb-4b8b-a634-754eda3f19db debug: PublicAccessAuthenticator:validate() Getting account properties...
2023-02-24T12:35:24.856Z b10e92cf-afdb-4b8b-a634-754eda3f19db debug: PublicAccessAuthenticator:validate() Retrieved account name from context: devstoreaccount1, container: undefined, blob:
2023-02-24T12:35:24.856Z b10e92cf-afdb-4b8b-a634-754eda3f19db info: PublicAccessAuthenticator:validate() Skip public access authentication. Container name is undefined.
2023-02-24T12:35:24.856Z b10e92cf-afdb-4b8b-a634-754eda3f19db info: BlobSharedKeyAuthenticator:validate() Start validation against account shared key authentication.
2023-02-24T12:35:24.856Z b10e92cf-afdb-4b8b-a634-754eda3f19db info: BlobSharedKeyAuthenticator:validate() [STRING TO SIGN]:"POST\n\n\n533\n\nmultipart/mixed; boundary=batch_35f851ae-7e1d-45db-52ed-28a9542a959a\n\n\n\n\n\n\nx-ms-client-request-id:9a7a3a25-dfa2-40ca-6672-47c1058fb470\nx-ms-date:Fri, 24 Feb 2023 12:35:24 GMT\nx-ms-version:2021-12-02\n/devstoreaccount1/devstoreaccount1\ncomp:batch"
2023-02-24T12:35:24.856Z b10e92cf-afdb-4b8b-a634-754eda3f19db info: BlobSharedKeyAuthenticator:validate() Calculated authentication header based on key1: SharedKey devstoreaccount1:QW/9uC7XhYDH0D4QOSICvbL7GExKLXg8LLiyaQjwQQM=
2023-02-24T12:35:24.856Z b10e92cf-afdb-4b8b-a634-754eda3f19db info: BlobSharedKeyAuthenticator:validate() Signature 1 matched.
2023-02-24T12:35:24.856Z b10e92cf-afdb-4b8b-a634-754eda3f19db verbose: DeserializerMiddleware: Start deserializing...
2023-02-24T12:35:24.856Z b10e92cf-afdb-4b8b-a634-754eda3f19db info: HandlerMiddleware: DeserializedParameters={"options":{"requestId":"9a7a3a25-dfa2-40ca-6672-47c1058fb470"},"comp":"batch","contentLength":533,"multipartContentType":"multipart/mixed; boundary=batch_35f851ae-7e1d-45db-52ed-28a9542a959a","version":"2021-12-02","body":"ReadableStream"}
2023-02-24T12:35:24.857Z 62e2a1bc-2056-4b9a-9407-36feddd7b2f9 info: BlobStorageContextMiddleware: RequestMethod=DELETE RequestURL=http://localhost/devstoreaccount1/gfs/2 RequestHeaders:{"authorization":"SharedKey devstoreaccount1:wOwehYKLa0civWq+AiGxaGx4iih2sBNiGpLkpHBBY0g=","user-agent":"azsdk-cpp-storage-blobs/12.7.0-beta.1 (Linux 5.10.0-21-amd64 x86_64 #1 SMP Debian 5.10.162-1 (2023-01-21))","x-ms-client-request-id":"a45cf015-2699-435e-447e-124d7bad0cba","x-ms-date":"Fri, 24 Feb 2023 12:35:24 GMT"} ClientIP=http://localhost Protocol=http HTTPVersion=version
2023-02-24T12:35:24.857Z 62e2a1bc-2056-4b9a-9407-36feddd7b2f9 info: BlobStorageContextMiddleware: Account=devstoreaccount1 Container=gfs Blob=2
2023-02-24T12:35:24.857Z 62e2a1bc-2056-4b9a-9407-36feddd7b2f9 verbose: DispatchMiddleware: Dispatching request...
2023-02-24T12:35:24.857Z 62e2a1bc-2056-4b9a-9407-36feddd7b2f9 info: DispatchMiddleware: Operation=Blob_Delete
2023-02-24T12:35:24.857Z b10e92cf-afdb-4b8b-a634-754eda3f19db info: BlobBatchHandler: starting on subrequest 0
2023-02-24T12:35:24.857Z 9a97c786-920c-4c44-9ef7-05a398cd7440 info: BlobStorageContextMiddleware: RequestMethod=DELETE RequestURL=http://localhost/devstoreaccount1/gfs/2 RequestHeaders:{"authorization":"SharedKey devstoreaccount1:wOwehYKLa0civWq+AiGxaGx4iih2sBNiGpLkpHBBY0g=","user-agent":"azsdk-cpp-storage-blobs/12.7.0-beta.1 (Linux 5.10.0-21-amd64 x86_64 #1 SMP Debian 5.10.162-1 (2023-01-21))","x-ms-client-request-id":"a45cf015-2699-435e-447e-124d7bad0cba","x-ms-date":"Fri, 24 Feb 2023 12:35:24 GMT"} ClientIP=http://localhost Protocol=http HTTPVersion=version
2023-02-24T12:35:24.857Z 9a97c786-920c-4c44-9ef7-05a398cd7440 info: BlobStorageContextMiddleware: Account=devstoreaccount1 Container=gfs Blob=2
2023-02-24T12:35:24.857Z 9a97c786-920c-4c44-9ef7-05a398cd7440 verbose: DispatchMiddleware: Dispatching request...
2023-02-24T12:35:24.857Z 9a97c786-920c-4c44-9ef7-05a398cd7440 info: DispatchMiddleware: Operation=Blob_Delete
2023-02-24T12:35:24.857Z 9a97c786-920c-4c44-9ef7-05a398cd7440 verbose: AuthenticationMiddlewareFactory:createAuthenticationMiddleware() Validating authentications.
2023-02-24T12:35:24.857Z 9a97c786-920c-4c44-9ef7-05a398cd7440 info: PublicAccessAuthenticator:validate() Start validation against public access.
2023-02-24T12:35:24.857Z 9a97c786-920c-4c44-9ef7-05a398cd7440 debug: PublicAccessAuthenticator:validate() Getting account properties...
2023-02-24T12:35:24.857Z 9a97c786-920c-4c44-9ef7-05a398cd7440 debug: PublicAccessAuthenticator:validate() Retrieved account name from context: devstoreaccount1, container: gfs, blob: 2
2023-02-24T12:35:24.858Z 9a97c786-920c-4c44-9ef7-05a398cd7440 debug: PublicAccessAuthenticator:validate() Skip public access authentication. Cannot get public access type for container gfs
2023-02-24T12:35:24.858Z 9a97c786-920c-4c44-9ef7-05a398cd7440 info: BlobSharedKeyAuthenticator:validate() Start validation against account shared key authentication.
2023-02-24T12:35:24.858Z 9a97c786-920c-4c44-9ef7-05a398cd7440 info: BlobSharedKeyAuthenticator:validate() [STRING TO SIGN]:"DELETE\n\n\n\n\n\n\n\n\n\n\n\nx-ms-client-request-id:a45cf015-2699-435e-447e-124d7bad0cba\nx-ms-date:Fri, 24 Feb 2023 12:35:24 GMT\n/devstoreaccount1/devstoreaccount1/gfs/2"
2023-02-24T12:35:24.858Z 9a97c786-920c-4c44-9ef7-05a398cd7440 info: BlobSharedKeyAuthenticator:validate() Calculated authentication header based on key1: SharedKey devstoreaccount1:wOwehYKLa0civWq+AiGxaGx4iih2sBNiGpLkpHBBY0g=
2023-02-24T12:35:24.858Z 9a97c786-920c-4c44-9ef7-05a398cd7440 info: BlobSharedKeyAuthenticator:validate() Signature 1 matched.
2023-02-24T12:35:24.858Z 9a97c786-920c-4c44-9ef7-05a398cd7440 verbose: DeserializerMiddleware: Start deserializing...
2023-02-24T12:35:24.858Z 9a97c786-920c-4c44-9ef7-05a398cd7440 info: HandlerMiddleware: DeserializedParameters={"options":{"requestId":"a45cf015-2699-435e-447e-124d7bad0cba","leaseAccessConditions":{},"modifiedAccessConditions":{}}}
2023-02-24T12:35:24.858Z 9a97c786-920c-4c44-9ef7-05a398cd7440 verbose: SerializerMiddleware: Start serializing...
2023-02-24T12:35:24.858Z 9a97c786-920c-4c44-9ef7-05a398cd7440 info: EndMiddleware: End response. TotalTimeInMS=1 StatusCode=202 StatusMessage= Headers={"Server":"Azurite-Blob/3.22.0","x-ms-client-request-id":"a45cf015-2699-435e-447e-124d7bad0cba","x-ms-request-id":"9a97c786-920c-4c44-9ef7-05a398cd7440","x-ms-version":"2021-12-02","date":"Fri, 24 Feb 2023 12:35:24 GMT"}
2023-02-24T12:35:24.858Z b10e92cf-afdb-4b8b-a634-754eda3f19db info: BlobBatchHandler: completed on subrequest 0 9a97c786-920c-4c44-9ef7-05a398cd7440
2023-02-24T12:35:24.858Z b10e92cf-afdb-4b8b-a634-754eda3f19db verbose: SerializerMiddleware: Start serializing...
2023-02-24T12:35:24.858Z b10e92cf-afdb-4b8b-a634-754eda3f19db info: Serializer: Start returning stream body.
2023-02-24T12:35:24.859Z b10e92cf-afdb-4b8b-a634-754eda3f19db info: EndMiddleware: End response. TotalTimeInMS=3 StatusCode=202 StatusMessage=Accepted Headers={"server":"Azurite-Blob/3.22.0","content-type":"multipart/mixed; boundary=batch_35f851ae-7e1d-45db-52ed-28a9542a959a","x-ms-request-id":"b10e92cf-afdb-4b8b-a634-754eda3f19db","x-ms-version":"2021-12-02"}
Have you found a mitigation/solution?
No.
Refer to the stack trace picture, the exception is raised by BlobServiceClient::SubmitBatch(), internally in turn call this method ParseSubresponses().
ParseSubresponses()
{
...
try
{
auto response = subrequest.Client.Delete(
subrequest.Options,
Core::Context().WithValue(s_subresponseKey, &subresponses[subresponseCounter++]));
subrequest.Promise.set_value(std::move(response));
}
catch (...)
{
subrequest.Promise.set_exception(std::current_exception());          // exception reported in this issue
}
...
}

Refer to this issue https://github.com/Azure/azure-sdk-for-cpp/issues/4048, the way to use the Azure SDK seems correct. Also, it works fine with Azure cloud storage.
The stack trace  -

Extra Azurite log -
azurite_log.txt.txt
Network package by Wireshark -
exception_deferred_result_submitbatch.zip

Thanks for your help. Let me know if anything more required.

Hi @adamhan-hitachi , I'm still looking into the issue, seems Azurite has done the batch delete and returned response successfully , but the SDK cannot get response body. Will update you with any progress.

PR to fix it: https://github.com/Azure/Azurite/pull/1885

Excellent! I will verify it today and check if that fixed my another issue.

@EmmaZhu Thanks for the updates. But I am not sure any conclusion I could get. Hence I just noted my test procedure hope that would be more or less useful.
Did you pass any test similar to the one in this repo https://github.com/adamhan-hitachi/test-azure-sdk-cpp/commits/main? It is still failed in my side.
Note:
Initially, I installed node.js 18.x and NPM 9.x through https://deb.nodesource.com/setup_lts.x. But I had critical errors reported by NPM  during building from source code by 'npm run build'.
Then, I downgraded to node.js 16.19.1 and NPM 8.19.3. NPM only reports issues in src/blob/persistence/SqlBlobMetadataStore.ts (as at bottom). I ignored that and run the test in above repo. It failed in the same exception. ($ azurite -v 3.23.0)
Further more, I try to update the Azure SCD for C++ to release "azure-storage-common_12.3.1", which failed the same too.
------------ npm run build error ----------------------
$ npm run build

<EMAIL_ADDRESS>build
tsc

src/blob/persistence/SqlBlobMetadataStore.ts:559:9 - error TS2345: Argument of type 'object' is not assignable to parameter of type 'Optional<any, string>'.
Type 'object' is not assignable to type 'Omit<any, string>'.
Index signature for type 'number' is missing in type '{}'.
559         this.convertContainerModelToDbModel(container)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/blob/persistence/SqlBlobMetadataStore.ts:1171:31 - error TS2345: Argument of type 'object' is not assignable to parameter of type 'Optional<any, string>'.
1171       await BlobsModel.upsert(this.convertBlobModelToDbModel(blob), {
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/blob/persistence/SqlBlobMetadataStore.ts:1419:33 - error TS2345: Argument of type 'object' is not assignable to parameter of type 'Optional<any, string>'.
1419         await BlobsModel.upsert(this.convertBlobModelToDbModel(newBlob), {
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/blob/persistence/SqlBlobMetadataStore.ts:1662:31 - error TS2345: Argument of type 'object' is not assignable to parameter of type 'Optional<any, string>'.
1662       await BlobsModel.upsert(this.convertBlobModelToDbModel(commitBlockBlob), {
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/blob/persistence/SqlBlobMetadataStore.ts:1803:31 - error TS2345: Argument of type 'object' is not assignable to parameter of type 'Optional<any, string>'.
1803       await BlobsModel.upsert(this.convertBlobModelToDbModel(snapshotBlob), {
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/blob/persistence/SqlBlobMetadataStore.ts:2645:31 - error TS2345: Argument of type 'object' is not assignable to parameter of type 'Optional<any, string>'.
2645       await BlobsModel.upsert(this.convertBlobModelToDbModel(copiedBlob), {
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Found 6 errors in the same file, starting at: src/blob/persistence/SqlBlobMetadataStore.ts:559

@adamhan-hitachi , I wasn't be able to get https://github.com/adamhan-hitachi/test-azure-sdk-cpp/commits/main passed build, I leveraged cpp SDK's sample project. I copied main.cpp from https://github.com/adamhan-hitachi/test-azure-sdk-cpp to CPP SDK's sample project and get the Azurite work with it.
About this:

Did you pass any test similar to the one in this repo https://github.com/adamhan-hitachi/test-azure-sdk-cpp/commits/main? It is still failed in my side.

Do you mean the CPP project still throws exception with Azurite, or you cannot get Azurite pass build?

My project was created by Cmake extension in VS Code. Good to know you pass it. Did it output a message from Line 104 - "std::cerr << "unknown exception " << e.what() << std::endl;"?
I will try your approach to test again.
About your question. I was meaning I saw the same message indicating exception. (See the first point above, the code captured exception and output an message).
Regarding to Azurite build, I believe Blob storage working fine in my computer as I am using it now even NPM did reported some errors as in my last comments.

@EmmaZhu Thanks again for following up this issue. I checked my example in https://github.com/adamhan-hitachi/test-azure-sdk-cpp.git. I simplify the project and add missing logic to create Azure blob container. More importantly, I removed code capturing the exception under test. By the latest version, it should no exception if the issue fixed.
I would be very appreciated if you could run it again. Otherwise, I'd like to wait for next Azurite release to retest.
Following step should help you setup my project -

open the fold where you clone my project in by VS code
install VS code extension CMake, CMake Tools and ms-vscode.cpptools
Ctrl + P and choose " CMake; Build "
F5 to launch GDB if you need

OR run the executable directly
$./build/azure-sdk-build
libcurl 7.88.1 481281
PUT OK
SubmitBatch OK
terminate called after throwing an instance of 'std::runtime_error'
what():  failed to parse response body at 232
Aborted

I tested azurite v3.24.0 and could confirm this has been resolved. Thanks a lot!

@adamhan-hitachi
Thanks for the confirmation!
I will close this issue.
Feel free to contact us again if need any further assistance on Azurite.


Use a new variable for conda vendoringg

In order to override the url from where conda is installed, the environment variable BUILDPACK_CONTINUUM_URL must now be set. This change is to allow the buildpack to work in conjunction with the main python buildpack without clashing on environment variables.

@gzork yeah I plan on doing a round of tests as this affects like 4 repositories - the tests are documented in https://github.com/plotly/dash-deployment-server/pull/513.


(WIP!) Xbrl steam

See #1707 for my ongoing task list
I haven’t been doing deep design thinking about how to compile the transform steps bc i assume there will be some dagster stuff and other considerations that I'd like suggestions on from the team.

a note about the extract step:

I think it would be cleaner for the extract step to save dfs with pudl table names with _instant and _duration suffixes when applicable. Instead of the current setup which saves the ferc1 raw table names. This way we could automatically grab the extracted tables with just the pudl table name (which is now the argument for many many of these transform functions).

I asked some questions about the FERC1 transform refactor design in this comment on #1739

Also @bendnorman did you see this list of outstanding questions on the linked issue? https://github.com/catalyst-cooperative/pudl/issues/1739#issuecomment-1213323493

Issue moved to catalyst-cooperative/ferc-xbrl-extractor #16 via ZenHub

Yes, it's definitely under-documented right now. @cmgosnell once you and I have gotten through the steam & fuel tables, let's write up some documentation in the generic table transformer module that explains how these different parts connect to each other, and then we can give that to @aesharpe and @zschira as they head into the next round of tables, and they can let us know what else it needs.


feat(labels.json): Add cleanup value to labels

PR Checklist

[ x] Addresses an existing open issue: fixes #000
[x ] That issue was marked as status: accepting prs
[ ] Steps in CONTRIBUTING.md were taken

Overview

Closing in favor of #635


micro_ros without FreeRTOS

Hi,
I am trying to use stm32CubeIDE configuring my board without FreeRTOS. Is this possible?
I am getting these errors:
../Core/Src/custom_memory_manager.c:13:10: fatal error: FreeRTOS.h: No such file or directory
   13 | #include "FreeRTOS.h"
      |          ^~~~~~~~~~~~
compilation terminated.
make[1]: *** [Core/Src/subdir.mk:46: Core/Src/custom_memory_manager.o] Error 1
make[1]: *** Waiting for unfinished jobs....
../Core/Src/dma_transport.c:6:10: fatal error: cmsis_os.h: No such file or directory
    6 | #include "cmsis_os.h"
      |          ^~~~~~~~~~~~
compilation terminated.
make[1]: *** [Core/Src/subdir.mk:46: Core/Src/dma_transport.o] Error 1
make: *** [makefile:77: all] Error 2
"make -j8 all" terminated with exit code 2. Build might be incomplete

Try to not include custom_memory_manager.c and avoid setting custom allocators here
Let us know if this works.

Thanks @pablogs9
Still having errors with dma_trasnport.c ,  microros_allocators.c  and  microros_time.c
../Core/Src/microros_time.c:3:10: fatal error: cmsis_os.h: No such file or directory
    3 | #include "cmsis_os.h"
      |          ^~~~~~~~~~~~
compilation terminated.
make[1]: *** [Core/Src/subdir.mk:43: Core/Src/microros_time.o] Error 1
make[1]: *** Waiting for unfinished jobs....
../Core/Src/microros_allocators.c:3:10: fatal error: cmsis_os.h: No such file or directory
    3 | #include "cmsis_os.h"
      |          ^~~~~~~~~~~~
compilation terminated.
make[1]: *** [Core/Src/subdir.mk:43: Core/Src/microros_allocators.o] Error 1
../Core/Src/dma_transport.c: In function 'cubemx_transport_write':
../Core/Src/dma_transport.c:40:13: warning: implicit declaration of function 'osDelay' [-Wimplicit-function-declaration]
   40 |             osDelay(1);
      |             ^~~~~~~
../Core/Src/dma_transport.c: In function 'cubemx_transport_read':
../Core/Src/dma_transport.c:59:17: error: 'portTICK_RATE_MS' undeclared (first use in this function)
   59 |         osDelay(portTICK_RATE_MS);
      |                 ^~~~~~~~~~~~~~~~
../Core/Src/dma_transport.c:59:17: note: each undeclared identifier is reported only once for each function it appears in
make[1]: *** [Core/Src/subdir.mk:43: Core/Src/dma_transport.o] Error 1
make: *** [makefile:77: all] Error 2

Ok, so not possible with the current status of this repo. For sure it is possible to run micro-ROS bare metal using CubeMX but it will need a major refactor of this repo.
We should remove dependencies with FreeRTOS and CMSIS_OS and move then to HAL/libc functionality.

In general:

microros_allocators.c can be removed and rely on libc malloc
dma_transports.c can be used without the delays functions as you will be using a single-thread app. Ideas here @Acuadros95 ?
microros_time.c will need a complete refactor to retrieve time from a bare timer configured in the HAL.

We could also use HAL_Delay(ms) as an alternative for transport delays.

We should have a conditional compilation option because AFAIK HAL_Delay does not work with RTOS. Maybe we can check if CMSIS is enabled and use osDelay or HAL_Delay appropriately.

Any update on this?

Hi,
As I am sure that this modification can be done locally, allowing this project to support MicroROS without FreeRTOS will be a huge benefit.
Thanks in advance!

Why was this closed?


Custom date time Fields auto date time filter in templates and Date picker

Is your feature request related to a problem? Please describe.

Custom date time fields will not display properly from templates as programmed in Django until the filter for date or time are added. Create a method to automatically apply these filters in the helpdesk templates.
Custom date time fields on input need date picker or datetime pickers to aid in input.

Describe the solution you'd like
Automatic filter application in Templates for display of data if the custom field is a date/time type field.
Automatically apply datepicker JQuery or datetimepicker for time/date types.
Describe alternatives you've considered
Add help text for input formats allowed.

I'm finding this to be a difficult task due to the many date time formats that could be utilized. The TicketCustomFieldValue value field is a TextField. This created a need to translate a String to Boolean to correctly initialize the CheckBox input for Boolean values.
My first inclination is to add additional value fields to the TicketCustomFieldValue class for each data type supported. This would create another translation requirement to read and write the correct value field based on the CustomField type, but would provide native data type support in forms.

I decided I will work with the TextField and add error handling for conversion issues.


Update developing-windows-drivers-for-usb-function-controllers.md

introduce the inbox controller drivers and clarify that some platform bring-up steps are needed.

@tedhudek reviewed and approved; please merge.


NIFI-3702 AWS SDK Update:

Updated the AWS SDK dependency version to 1.11.8
Removed GetDynamoDBTest references to JsonObject class dropped from AWS SDK
Fixed integration test ITPutSNS.testPublish() - exception from not evaluating expressions
Ignored integration test ITPutS3Object.testS3MultipartAgeoff() - did not work before or after SDK update

Reviewers: In NIFI-2062 we upgraded the master branch from AWS SDK 1.10.32 to 1.11.8.  The change included some adjustments to unit tests to account for Amazon's JSON changes.  We probably should have merged this to 0.x last year, but didn't.  So I recycled commit a7c5b88, changed the names to protect the innocent, and present it fresh for 0.x.

Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[X] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[X] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[X] Has your PR been rebased against the latest commit within the target branch (typically master)?

[X] Is your initial contribution a single, squashed commit?

For code changes:

[X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[X] Have you written or updated unit tests to verify your changes?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

Thanks for this @jvwing.  Would you mind updating the nifi-nar-bundles/nifi-aws-bundle/nifi-aws-nar/src/main/resources/META-INF/NOTICE and nifi-assembly/NOTICE files to remove the 1 line from each that says "JSON parsing and utility functions from JSON.org - Copyright 2002 JSON.org" in the Amazon Web Services SDK section?  If you could take care of that in this PR, then I won't need another!

@mosermw, sorry I left that out.  I updated both NOTICE files and checked the AWS SDK NOTICE to verify that was the only change.  I force pushed the branch with the modified commit.

+1 will merge.  Thanks @jvwing.
It looks like I should also migrate the changes from NIFI-2763 and NIFI-2902 (V4 Signer support) over to the 0.x branch as well, to avoid breaking that functionality.  I will create a new JIRA for that.

Would you close this PR @jvwing?  Thank you.

Thanks, @mosermw.