add to schema Functionality to set Document set view

It would be nice to have the possibility to add a view to a document set per listinstance.
pnp:Documentset
<pnp:contenttype id="" name="">
<pnp:view name="all documents"></pnp:view>
pnp:contenttype
</pnp:Documentset>

Hi,
Thanks for your feedback. The idea is great, and someone else already submitted it, too. However, we need to figure out how to implement this capability at the engine level.

Hi Paolo,
thank you i wrote a extension method for it maybe you can use it in the core in combination with the schema change.

Hi Paul,
Sorry for the delayed answer, where is the extension method? Is it in Core or in your own code somewhere else? Can you share it? Thanks.

Hi Paolo,
will look it up and send a message thanks again.
Paul

Hi Paulo,
have not tested if it still works but i can test it if you want.
`  ///
/// sets the view for a Documentset welcome page.
///
/// The context.
/// The web.
/// The list.
/// The contenttype.
/// The contenttype.
private void SetDocumentSetView(ClientContext context, Web web, List spList, string pContentType, string welcomePageview)
{
        //var contenttype = spList.ContentTypes.GetById(pContentType);
        context.Load(spList, l => l.ContentTypes);
        context.ExecuteQuery();

        var contenttype = spList.ContentTypes.FirstOrDefault(c => c.Name == pContentType);
        var viewid = spList.GetViewByName(welcomePageview).Id;

        var docSetTemplate = DocumentSetTemplate.GetDocumentSetTemplate(context, contenttype);

        //context.Load(contenttype, c => c.SchemaXmlWithResourceTokens);
        context.Load(contenttype, c => c.SchemaXml, c => c.SchemaXmlWithResourceTokens);
        context.Load(docSetTemplate);
        context.ExecuteQuery();

        var xmlschema = string.IsNullOrEmpty(contenttype.SchemaXmlWithResourceTokens) ? contenttype.SchemaXml : contenttype.SchemaXmlWithResourceTokens;
        if (xmlschema.Contains("XmlDocuments"))
        {
            XmlDocument xmldocument = new XmlDocument();
            xmldocument.LoadXml(xmlschema);

            XmlNodeList xmldocs = xmldocument.SelectNodes("ContentType/XmlDocuments/XmlDocument");
            foreach (XmlNode xmldoc in xmldocs)
            {
                if (xmldoc.ChildNodes.Count > 0)
                {
                    var firstChildNodeName = xmldoc.FirstChild.Name;    //xmldoc.ChildNodes[0].Name;
                    if (firstChildNodeName == "WelcomePageView")
                    {
                        // Create new default view XML
                        string newDefaultViewXml = string.Format(@"<wpv:WelcomePageView xmlns:wpv='http://schemas.microsoft.com/office/documentsets/welcomepageview' ViewId='{0}' />", viewid);

                        // Replace default view XML
                        xmldoc.FirstChild.InnerXml = newDefaultViewXml;

                        // Exit foreach loop
                        break;
                    }
                }
            }

            StringWriter sw = new StringWriter();
            XmlTextWriter tx = new XmlTextWriter(sw);
            xmldocument.WriteTo(tx);
            xmlschema = sw.ToString();

             contenttype.Update(false);

        }
    }`

Hi @KbWorks,
We just added to the upcoming new schema version a XmlDocuments element for the DocumentSet. In the future, we will add corresponding functionality at the engine level. Thanks.

Hi KbWorks
Most likly your code is for SharePoint onpremises. I tried your approach on sharepoint online, but it did not work since SchemaXml is readonly property in online version.
Thaks anyway

Hi @PaoloPia ,
I am trying to define the default view for a docset using this code
 <XmlDocument NamespaceURI="http://schemas.microsoft.com/office/documentsets/welcomepageview">                 <WelcomePageView xmlns="http://schemas.microsoft.com/office/documentsets/welcomepageview" LastModified="1/1/1 0:00:01 AM" />               </XmlDocument>
but It doesnt work. Any suggestion idea how to fulfill this? thanks!!

So here we are 5 years later and there doesn't seem to be a programatic or declaritive way to set the view to be used within a document set.
One would assume this would have to be a setting at the library level - this is because:

There are no views defined at a site level (alone with site columns and site content types)
The library and the view need to exist for there to be a ViewID to reference

Please Microsoft, how do we use your tools in a scalable way?

Can we update the "ViewId" in the "SchemaXML"? I tried and the SchemaXML is read-only.

@MrRCG5 Yes, that's the way to do it currently, and it is pretty easy:
$welcomePageView = Get-PnPView -Identity $view.Displayname -List $list
$xml = [xml](Get-PnPContentType -Identity $ctType.Name -List $library | Select -Expand SchemaXml)
$node = $xml.ContentType.XmlDocuments.XmlDocument | where {$_.NamespaceURI -eq 'http://schemas.microsoft.com/office/documentsets/welcomepageview'}
$node.InnerXml = "<wpv:WelcomePageView xmlns:wpv=`"http://schemas.microsoft.com/office/documentsets/welcomepageview`" ViewId=`"$($welcomePageView.Id)`" />"

newDefaultViewXml

It is not working for me using SharePoint online. How is "$node.InnerXml" being updated? Do I need to execute or invoke a change?

newDefaultViewXml

It is not working for me using SharePoint online. How is "$node.InnerXml" being updated? Do I need to execute or invoke a change?

You are correct. I must have been testing the wrong variables and assumed wrongly it was working, but it is not. The property 'SchemaXml" is read-only. I don't know if this is a setting that can be changed per instance, or not.
Cross link: https://sharepoint.stackexchange.com/a/314039/68024


visit 3.4.1 rhel gl build missing

Could we please have a gl build for Linux based systems. The one I used is the rhel one. Mesa is simply much slower.

We no longer have access to a RHEL-7 system with an NVIDIA graphics card to build releases. If folks have ideas of how we can provide a build that supports accelerated GL (container recipe) we could create binaries in the future.

What does it take to build it, say on Fedora 39 with NVIDIA drivers installed? I have a number of systems.

@sambo57u  we almost have completed recipes for building on Fedora 39 w/ mesa, Would you be willing to see if that recipe can be followed for a Fedora 39 system that has NVIDIA drivers?
Here are the related PRS:
#19559
#19562

I just upgraded my workstations to Fedora 40 but I had retired a few that had 39 on them. I can reconnect those and try this. Let me know how to proceed.

PS; they do have the latest NVIDIA drivers installed.

Great!  @biagas Can you connect with @sambo57u  when you feel the recipes are ready?

@sambo57u, Which version of g++ is running on your Fedora 40 machine? If it is 13, then the scripts I modified on a Fedora39 docker container should run. I will let you know as soon as I have the updated build script.

Fedora 40 is running gcc 14.1, unfortunately.


SCons: Fix feature build profile being parsed too late

Also renames the option to just build_profile, the previous one was too verbose and had words in the wrong order. (Slight compat breakage but should be fine at this stage for such an advanced build option.)
Fixes #70643.

Since I'm breaking compat on the option name, I'm wondering whether I should also fix this weirdness in the build profile itself. Here's an example from @boruok:
{
	"disabled_build_options": {
		"disable_3d": true,
		"disable_3d_physics": true,
		"openxr": false,
		"rendering_device": false,
		"vulkan": false
	},
	"disabled_classes": [

	],
	"type": "build_profile"
}

disabled_classes is fine, but a section named disabled_build_options with stuff like disable_3d=true or vulkan=false seems confusing to me.
Does it mean that it's disabling the disable_3d build option (since the value is true), thus enabling 3D? Or does it mean that disable_3d is going to be set to true, thus disabling 3D?
Since this maps to SCons build options, which may be either negatively worded (disable_3d) or positively worded (rendering_device), I think it would be better to name that section build_options only.


Option to don't automatically remove an item from "Files -> Open Recent"

I use VSCode for editing network files or files on external drives. But sometimes I forget to mount a network drive before opening a file located on that drive. So here is my usual scenario:

Navigate to "Files -> Open Recent"
Click to some network file (e.g. /Volumes/homeassistant/configuration.yaml)
Alert "Path does not exist" pop-ups, because the parent drive is not currently mounted (see the screenshot below)
Click OK
Mount the drive
Navigate again to "Files -> Open Recent"
Observe, that the /Volumes/homeassistant/configuration.yaml item is missing (it was removed due to the failed attempt to open it)

So it would be useful to have an option in the VSCode Settings to disable that automatic removal of items which failed to open

I'd like to give this a try!

+1 for this, I have this problem when I forget to enable the VPN (or it is temporarily disconnected) and my network share is unavailable. This happens to me a couple of times a week... Another solution would be a favorites menu (bookmarks?) that is fully controlled and does not automatically change.


Update airflow-flower.conf

Following changes done:

added the console output to get the logs in /var/log/upstart/airflow-flower.log
updated the exec parameter to default location of airflow installation

Dear Airflow maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA

[ ] Not sure if this has ticket in JIRA but adds the functionality to airflow services

Description

[ ] With this change we can see the logs of airflow maintained by upstart in /var/log/upstart/.log

Tests

[ ] My PR does not need testing for this extremely good reason:
any user defined upstart conf files if have

Commits

[ ] Updated the upstart configuration file to include the upstart managed logs for the service

Codecov Report

Merging #2768 into master will decrease coverage by 0.09%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##           master    #2768     +/-   ##
=========================================
- Coverage   73.26%   73.17%   -0.1%
=========================================
  Files         157      157
  Lines       11934    11934
=========================================
- Hits         8744     8733     -11
- Misses       3190     3201     +11

Impacted Files
Coverage Δ

airflow/task_runner/bash_task_runner.py
93.33% <0%> (-6.67%)
:arrow_down:

airflow/utils/helpers.py
53.44% <0%> (-2.88%)
:arrow_down:

airflow/jobs.py
79.23% <0%> (-0.45%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3eb2dd8...281d7e9. Read the comment docs.

Reopen and please use one commit when it makes sense for your changes instead of spreading out the same change over 4-5 files. You created 4 similar commits.


[Bug]: Crash/Endless wait after claiming a large amount of native tokens

Issue description
When trying to claim a large amount of native tokens, we will never get a response back from the account.claimOutputs call.
The manager can also not be destroyed after that. Only after killing the process and restarting, everything works again, with the output claimed. The limit between when it works and when not is ~1,451,000,000,000,000.
Maybe related to this:
When calling account.prepareOutput for sending ~1,451,000,000,000,000 tokens, the error invalid output amount:<PHONE_NUMBER>000000 is thrown, but sending the output afterwards works fine
One time with and one time without expiration ULC. When we have no expiration ULC, the error gets thrown, and if we have it, it doesnt
Version
2.0.3-rc.27
Expected behaviour
Claim output without error or endless loading
Actual behaviour
Endless loading
Can the issue reliably be reproduced?
Yes
Steps to reproduce the issue

Send more than 1,451,000,000,000,000 native tokens with SDRUC from account A to account B (error should appear when preparing output)
On Account B, try claim the output -> endless load

Errors
wallet-2023-04-06-08-55.log
Duplicate declaration

[X] I have searched the issues tracker this issue and there is none

Claiming worked fine [OUTPUT_CLAIMING] Claiming transaction created: block_id: Some(BlockId(0x57ec6efca945d11d1006ed902c913ed1219316dc3a222e0d6db8ebf0f4e779f2)) tx_id: TransactionId(0xb1534c417c68275af395acfc26c9e0b68a23f79fb85ce71bd46539d74ad158e1)
https://explorer.shimmer.network/testnet/block/0x57ec6efca945d11d1006ed902c913ed1219316dc3a222e0d6db8ebf0f4e779f2
The first time you provided the native token amount as amount for the output, so this way it failed
prepare_output DEBUG [OUTPUT] prepare_output OutputOptions { recipient_address: "rms1qqctzt4p445zy5htsq0tr253y5pu96eckf3376rg03r33t5ylmgtx6kndp2", amount:<PHONE_NUMBER>000000,


[FZ Editor] snapping and moving splitters

Environment
Windows build number: 10.0.17763.973
PowerToys version: 0.14.1
PowerToy module for which you are reporting the bug (if applicable): FancyZones

Actual behavior
Splitters are placed incorrectly and they can't move throughout whole area

Screenshots

Fixed in https://github.com/microsoft/PowerToys/pull/4095

We just released 0.20 and this should be resolved there.  Please try it out.


Attributes & instances variables names on KEIndividual are not coherent

Why do we need attributes on KEIndividual and not instances variables on KEIndividual ?
Why currentStatus and nextStatus are not defined as attributes ?

#id attribute of KEIndividual has been removed.

Remove #alive attribute in KEIndividual and all the mess generated with this in Kendrick-SergeStinckwich.638
Individual-based simulations are still working :-)

Update with Kendrick-SergeStinckwich.640 (we take care of deaths/birth of individuals)


Update to new Govee API

https://developer.govee.com/reference/get-you-devices

hey @wiltodelta I pushed PR #4 with the necessary changes. Could you please review?
I've successfully tested on Home Assistant version 2024.7.4

Thank you for the contribution @ilyasubkhankulov!


provide example for opengl

hi,
is it possible to add an example of how to use this with opengl in golang? I just wanna render a video (with sound) in an openGL window.
thanks

@sk8245 Do you want to render video to a texture? If so, libVLC v3 does not support this. I know there's some development regarding this in v4, but v4 is in development. I will add bindings when there's a stable version of v4 as the API will likely change.
If you simply want to render a video in a window, I've added a GTK player example which renders  video to a gtk.DrawingArea: https://github.com/adrg/libvlc-go/blob/master/v3/examples/gtk_player.go

Closing this issue as it has been addressed. Please reopen if the provided solution is not suitable for your use case.


Add details about supported platforms

File: engine/installation/linux/index.md
Hi.  Not everything runs on x86.   There is Power Linux and Linux on z.  Please update this page to reflect what platforms are supported

This has already been done as part of previous work. :)


Update permute.py

Updated the permute.py file with the required changes with reference to the issue #41123.

Hey @qlzh727 , thanks for reviewing the PR, I shall make the necessary commits. Thanks

Hey @gbaned , please review my comments and suggest me the necessary changes. Thanks

Hey @gbaned , please assign me another reviewee, as @qlzh727, hasn't reviewed my PR past 2 days. Thanks

Hi @Bhavay192 ,
The example given in the documents is an example of permute in 3D tensor. What you saying is about a 4D tensor. Also when I say a rank, the batch is also included.
If you still can't follow please refer to this notebook
Thank you

@Bhavay192 Can you please check @old-school-kid's comments and keep us posted ? Thanks!

I am closing this PR since it is lack of activity. Feel free to reopen later.


Allow specify custom ntp servers on OpenStack

This PR adds support for specifying a custom ntp server array in your settings.yml file.
It also moved similar logic from the vsphere provider into the base class.

:+1:


Allow spaces in directory paths

Hi,
I have a small modification that you might find helpful.
The current implementation uses an os.system call to the native shell mkdir function when creating new folders in converter.py. The way this is implemented causes problems for directory paths that include spaces in the name.
This pull request replaces calls to os.system with os.makedirs which gets around this limitation.
The remainder of the code changes are actually just to the test suite.
I'm not entirely sure what test_warning_banana_bad_n() tests but I believe it is for trying to create an output folder when one already exists? If so, then the method of testing the assertion changes with the use of os.makedirs. An appropriate modification has been made.
Another test suite called test_convert_banana_spaces.py has been added. It is essentially a copy of test_convert_banana.py but uses 'bru banana' and 'nifti banana' (with spaces) as the paths to be used. The 'bru banana' folder is just a direct copy of the original 'bru_banana' folder. This test suite passes with this commit but fails on the original implementation.
Let me know your thoughts.
Thanks,
Jath

Hi Jath!
Thanks for considering helping the development of bruker2nifti, much much appreciated!
Only, I think there are still some improvements to do before the merge.

About giving the user the possibility of having paths with spaces, can you please add a warning to console (with a print, or with the more sophisticated logme or similar) when there are spaces, and stop the conversion? Most of the software in the post-processing are not permissive to this extent, so it is a practice that I would not like to encourage: it can simplify the life for the conversion step, but the user may need to remove the spaces from the paths manually afterwards anyway.
 Or maybe even removing the spaces automatically: There are spaces in the input paths. These will be replaced with underscores. Do you want to continue [Y/n]? (or similar). What do you think?

For the extra test module and the extra data test, I would like to keep the code as minimal as possible and avoid redundancies. It is already not best practice to have data test in the repo itself (in a future version I should better distinguish between unittests -no data and only mock- and a more comprehensive integration test - with data cloned from bruker2nifti_qa repo). This difference was not clear to me when I wrote this code, and it would greatly improve maintainability and efficiency. For your case, can you for example mock some data input path and do unittest of only the function that reads and creates the path with spaces, ensuring the warning is thrown, without considering a full copy of the dataset only for a single test case?

It is a good idea to move to pipenv, as it is going to be the future standard for production. At the same time, there may be some issues in installing this particular tool in development mode (pip install -e . is the usual way for research code, and especially this one, that needs still a lot of tweaks - see open issues!). About pipenv, I still do not know how to keep the pipfile and the requirements.txt coherent. Is there any "translator" to keep both files in sync automatically, to allow the user to install with the old pip? Any suggestion is more than welcome!

For what I can understand it is better to add the ignore the pipfile.lcok for code that is targeting more than one python version.

For what I remember, nosetest is almost not maintained anymore. Is it still the case? If yes, I would suggest using pytest instead (nosetest had been "recently" purged from the repo - see commit tagged "Nosetests is no more"), or only assert, to have the minimal amount of dependencies as possible.

...I know I am in your way for the merge and I'm asking you a lot...! On the top, this repo is far from being perfect under many perspective, and in particular in term of testing. At the same time I really do not want your efforts to be wasted in merging your work now and then having to delete part of it in the subsequent attempt of improving the testing structure and removing the data from the repo.
Thanks for your help again!
Sebastiano
P.S. An AUTHORS file is on its way. As soon as done, please pull the repo and add your name in the next PR.

Hi Sebastiano,
Thanks for the feedback.

...I know I am in your way for the merge and I'm asking you a lot...!

Firstly, not at all! I'd much rather you merge good code that fits the code base than something half-baked!

About giving the user the possibility of having paths with spaces, can you please add a warning to console (with a print, or with the more sophisticated logme or similar) when there are spaces, and stop the conversion?

Yes I think that's a good idea. I think printing a message to the console is probably the best way of doing it so I'll add the relevant code.

For the extra test module and the extra data test, I would like to keep the code as minimal as possible and avoid redundancies.
I would suggest using pytest

Yes actually I'd be very happy to use pytest instead. Sorry I got the impression you were using nosetest which is why I used it but that's easily fixed. In terms of keeping the test suite small, I could, yes I agree it makes sense to just mock a file path with spaces rather than have redundant test data. I'll make the necessary changes.

About pipenv, I still do not know how to keep the pipfile and the requirements.txt coherent. Is there any "translator" to keep both files in sync automatically, to allow the user to install with the old pip?
For what I can understand it is better to add the ignore the pipfile.lcok for code that is targeting more than one python version.

Sorry I added pipenv for managing the virtual environment as much as anything. If you wanted I could remove the pipenv sutff and just use virtualenv/venv  for the virtual environment and leave requirements.txt. Alternatively if we did keep the pipfile running pipenv run pip freeze > requirements.txt would update requirements.txt to match the current pipfile.lock. I think you're right about keeping it out of git if supporting both python 2 and python 3. I'll add it to gitignore.
Thanks again for the feedback. I'll get working on the changes and update the pull request in a little while.
Cheers,
Jath

Great!
Agree with all.
Please do keep pipenv alongside the old fashion requirements.txt!
Thanks, looking forward for the next PR!

Oh oh... I think I messed up my git foo. I could spend the next 10 hours trying to work out what I messed up and how to fix it but I think it'll be easier for me to just close this and reopen a new pull request. I will combine the multiple commits to a simple commit as well to make the merge cleaner.


Flow considers React types to be any-typed values if they are imported in a flow-typed module

Flow version: 0.132.0 and above
Expected behavior
Now flow error
Actual behavior
Flow emits an error when I use a React type that has been imported in a flow-typed module.
Github repo: https://github.com/debel27/flow-issue_value-as-type
To reproduce : clone the project, run yarn, then run yarn run flow
The emitted error is the following
Error ┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈ src/index.js:4:13

Cannot use Node as a type because it is an any-typed value. Type Node properly, so it is no longer any-typed, to use it
as an annotation. [value-as-type]

     1│ // @flow
     2│ import * as React from 'react';
     3│
     4│ const node: React.Node =
Hello world
;
     5│

One way to get around this - use the internal names. So rather than React.Element, use React$Element. A find/replace from React. to React$ in any libdef file should take care of it.

One way to get around this - use the internal names. So rather than React.Element, use React$Element. A find/replace from React. to React$ in any libdef file should take care of it.

Something's really funky here, it's because of the flow-typed/types.js file you have with a import type { Node } from 'react'; inside.

That flow-typed folder is handled as a "normal" file since it's not in the [libs] section of the config.
import type { Node } from 'react';  should be fine, either in a lib or "normal" file.
It shouldn't affect other imports...
:woman_shrugging:

Note: react types are still part of Flow core, it's only react-dom that got moved to flow-typed for now.

Something's really funky here, it's because of the flow-typed/types.js file you have with a import type { Node } from 'react'; inside.

That flow-typed folder is handled as a "normal" file since it's not in the [libs] section of the config.
import type { Node } from 'react';  should be fine, either in a lib or "normal" file.
It shouldn't affect other imports...
:woman_shrugging:

Note: react types are still part of Flow core, it's only react-dom that got moved to flow-typed for now.

In library definitions, the imports must be inside a declaration block.
declare module 'something' {
  import type { Node } from 'react'
}

In library definitions, the imports must be inside a declaration block.
declare module 'something' {
  import type { Node } from 'react'
}

Regarding 1., I'm expecting flow-typed to be considered as a lib folder by default (as stated in the docs).
I was unaware there were plans of moving out React types from Flow core. Thanks for the info.

Regarding 1., I'm expecting flow-typed to be considered as a lib folder by default (as stated in the docs).
I was unaware there were plans of moving out React types from Flow core. Thanks for the info.

TIL: "By default, the flow-typed folder in your project root directory is included as a library directory. "

TIL: "By default, the flow-typed folder in your project root directory is included as a library directory. "

Import statements are no longer allowed at the toplevel of library files. To use import statements in library files they must appear within a "declare module".

https://github.com/facebook/flow/blob/master/Changelog.md#01430

Import statements are no longer allowed at the toplevel of library files. To use import statements in library files they must appear within a "declare module".

https://github.com/facebook/flow/blob/master/Changelog.md#01430

This issue can be closed, because (as the last comment indicates) it was fixed in v0.143.0.
This is the same issue as #6414 and some other issue threads linked to that one. See https://github.com/facebook/flow/issues/6414#issuecomment-958648596 and the next comment for more detail on the fix.


Fix link to title capitalization

The referred link seems to not exist anymore.
I wasn't sure how to check for the original choice.
https://titlecaseconverter.com/ looked like a good option, but https://titlecapitalize.com/ seemed closer to the original webpage link, so went with that option.
Or I can change to something else, if the domain has just been replaced for something else entirely :P

I think I just copied this from some other CONTRIBUTING file, so I'm glad you checked it. Totally fine with the choice 👍🏻


Steps to generate Bearer Token - incomplete.

Steps to generate bearer token are incorrect. The URL is jumbled up
GET /MSI/token?resource=https://vault.azure.net&api-version=2019-08-01
The above URL doesn't make sense.
In the powershell section the steps is as below (again an invalid url how can you have ?resourceHttps"//....)
$tokenAuthURI = $env:IDENTITY_ENDPOINT + "?resource=$resourceURI&api-version=2019-08-01"
Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: f57d3177-ce9e-ebfa-03a7-45beb8ca3212
Version Independent ID: a59df23d-f3ac-9809-61d3-5741e31f020a
Content: Managed identities - Azure App Service
Content Source: articles/app-service/overview-managed-identity.md
Service: app-service
GitHub Login: @cephalin
Microsoft Alias: cephalin

@asubmani
Thanks for your feedback! We will investigate and update as appropriate.

@asubmani Thanks again for the feedback! I have assigned the issue to the content author to review further and update the document as appropriate.

Internal item created: https://dev.azure.com/msft-skilling/Content/_workitems/edit/288596
#please-close


lost original type of element in Set when invoke toArray()

new Set([1,2,3]).toArray() // suppose: [1,2,3] actual:['1','2','3']

fill this.set with the original value instead of true may solve this problem

No, the design of this set is to convert all the entered items into strings ie via builtin toString, and add to the set as keys (with value=true).
So what you see is by design, so anything can be used as a key (not just numbers).


Don't crash in Statement.next.

We're getting a lot of crashes in Statement.next() with unhelpful crash logs. I wanted to make it throws but that doesn't match the signature of the IteratorProtocol method it's implementing. This isn't my favorite solution, but I don't like crashes and I'm not sure what else to do. I would love some feedback and/or ideas about this.

How hard would it be to have next() throw the error instead of swallowing it here with a print?

This is not a good solution. Slightly better would be to use precondition here so the check can be disabled with -Ounchecked if desired. It would still not solve the actual problem though, which requires bigger changes.
What kind of errors do you see in your app? How does it recover from them?

You think force unwrapping and crashing is better?

@stephanheilner Obv not ideal, but still better than ignoring errors.

I agree that swallowing the error isn't ideal. I liked the assert idea. (I don't feel like it's really a precondition at that point since the work has already been done, but I can change it if you want. Assertions can also be turned off but I think it's a different flag.)
A couple other options I thought of are to not conform to IteratorProtocol and just make next throw, or change the return type to be some Result type that can include an error. The downside of both of those is that they won't be backwards compatible though. The assert/precondition approach can at least give users the option to not crash by turning them on or off.

Did you consider that Sequence may not be the correct type?
Hey, @nickmshelley just wrote it :-) Introducing Cursor:

pitch on swift-evolution (may well deserve a little support)
Cursor.swift

GRDB's cursors share traits with lazy sequences and iterators:
let cursor = /* some cursor of Row, for example */
while let row = try cursor.next() { ... }
try cursor.forEach { row in ... }
try cursor
    .filter { row in ... }
    .map { row in ... }...

@jberkel I just realized I never answered your question. The main problem is that we don't see crashes when running locally very often, but when we do we get a fairly helpful message from the error (like "Disk IO error") so we can try to prevent it in the future.
However, the crash reports from Fabric give stack traces and messages that aren't as helpful (see attached image). All crashes that happen in Statement.next() have similar beginning stack traces, but when looking through them the rest of the stack varies among all of the places we use SQLite.swift in our app (or libraries we include that use it), some of which take user input for FTS search, so it's nearly impossible to tell what went wrong with the information we have. And since that is the top crasher in our app and the next top one happens an order of magnitude less, I'd really like to get this figured out. I'm willing to do the bigger changes required to correctly solve this problem if you aren't ok with this assert/precondition stopgap.

Yes, looks like Sequence / IteratorProtocol does not help here and we need to introduce another type. I just checked my own project which uses SQLite.swift and the pattern employed is almost always:
return try connection.prepare(query).map { mapRow($0) }

I imagine that providing a few simple (throwing) operations such as map and reduce would handle the majority of use cases.
Interesting question is API evolution, for now we could keep the current state and return a new type explicitly:
return try connection.prepare(query).rowCursor.map { mapRow($0) }

This would make it opt-in, and a future major release of the library could make this the default.
@groue surprised nobody replied to your post on the swift mailinglist, I'd imagine this to be a more common problem. Sometimes I feel that mundane things like error handling get ignored in this pure functional developer view of the world.
But it occurred to me that instead of throwing errors we could just return a new type which encapsulates the error:
enum RowResult {
  case error(SQLite.Result)
  case row(RowData)
}

public func next() -> RowResult? {
}

This would lend itself to composition so that you could chain multiple operations via flatMap. A related universal Either type has already been suggested on swift-evolution but the proposal was not accepted.
The client code could then use the result type returned by the library and handle it internally or simply rethrow the error (one could image adding a set of methods which fail fast, to emulate current behaviour).
One could also look at it from a reactive programming perspective, where iterators are reversed into Observables (or Signals) which emit events:

Event is an enumerated type representing either a value or one of three terminal events:

The value event provides a new value from the source. [...]
The failed event indicates that an error occurred before the signal could finish. [...]
The completed event indicates that the signal finished successfully [...]

(RAC events)
Not explicitly throwing errors in our own code would facilitate this programming model as well.

@jberkel RowResult (or a Row wrapped in a generic Result or Either type) is an interesting idea.
May I brainstorm with you a little bit?
I guess that Result has serious consequences on client code, that you will eventually evaluate. I can't say if forcing users into mapping/flatMapping/unwrapping results is a good idea or not - it depends on how much "functional" you want SQLite.swift to look and feel. Besides, sequences of results can not be turned into lazy sequences of unwrapped userland types: the memory-efficient consumption of SQLite statements that return many rows would require users to use Result:
let lazySequenceOfRowResults = try connection.prepare(query)

let arrayOfUserResults = lazySequenceOfRowResults
    .map { $0.map { User(row: $0) } }

let lazySequenceOfUserResults = lazySequenceOfRowResults
    .lazy
    .map { $0.map { User(row: $0) } }

let userArray = try lazySequenceOfRowResults
    .map { try User(row: $0.unwrap()) }

let lazySequenceOfUsers = /* impossible */

// Consumption of result sequence:
for resultResult in lazySequenceOfUserResults {
    switch result {
    case .success(let user): print(user.name)
    case .failure(let error): ...
    }
}

Cursors, on the other hand, have the advantage of being always lazy, and to be able to wrap any type, including userland types. This means that they don't make users pay with functional idioms when they want a memory-efficient consumption of SQLite statements that return many rows:
// With GRDB
let users = try User.filter(...).fetchCursor(db)  // DatabaseCursor<User>
while let user = try users.next() { // or try users.forEach { user in ... }
    print(user.name)
}

Applied to SQLite.swift, this would read:
let rows = try connection.prepare(query)              // Cursor of Row
let users = rows.map { User(row: $0) }                // Cursor of User
let userArray = try Array(rows.map { User(row: $0) }) // Array of User

// Consumption of cursor
while let user = try users.next() { // or try users.forEach { user in ... }
    print(user.name)
}

As for support for reactive programming, some fast thoughts: cursors need glue code to be turned into sequences of Result that can feed a Reactive library. But sequences of SQLite.swift Results would need some glue code as well, because we're not talking about the same Result types. We don't escape glue code in both cases. I don't known which one is easier to write, though.

@jberkel The cursor API has another advantage:
// A sequence of Result is designed for individual error handling:
for result in results {
    switch result {
    case .success(let value): ...
    case .failure(let error):
        // Is it the last step? Will I get more successes? More failures?
        fud()
    }
}

// Cursor API is designed to stop on first error - just like SQLite statements:
try {
    while let value = try values.next() {
        ...
    }
} catch {
    // Fetch has eventually failed. No more success. No more failures.
}

You could actually consider that last point a disadvantage to using cursers, depending on how you look at it @groue. The result approach gives users the most control: if they want to abort on first failure like the cursor approach does by default, they can, but if they want to continue they can as well (but not with cursors as they are currently implemented).
Of course, maybe the extra control isn't worth the extra complexity, and I think aborting on the first failure is a very sensible default, even if it could be considered limiting.
@jberkel I think either the backwards-incompatible result type approach or the opt-in (at least for now) row cursor approach are both fine, both having their own advantages and disadvantages. I don't know enough about RAC to proper evaluate your last suggestion, but I'd be hesitant to force RAC on users of the library.
Let me know which direction you want me to proceed. Or if you're inclined to just implement whatever you have in mind yourself, I'm perfectly fine with that as well. 😉

You could actually consider that last point a disadvantage to using cursers, depending on how you look at it @groue. The result approach gives users the most control: if they want to abort on first failure like the cursor approach does by default, they can, but if they want to continue they can as well (but not with cursors as they are currently implemented).
Of course, maybe the extra control isn't worth the extra complexity, and I think aborting on the first failure is a very sensible default, even if it could be considered limiting.

Once sqlite3_step() has returned an error, a statement can not be reused (sqlite3_step, sqlite3_reset, etc. all return the same error ad nauseam).
So it's not a gratuitous limitation or an opinionated choice: it's API design.

Now if we speak precisely: once sqlite3_step() has returned an error, a statement can not be reused (sqlite3_step, sqlite3_reset, etc. all return the same error ad nauseam).

Thanks for pointing that out, I didn't know that. Pretty much makes my point moot then. 😄
I'm not familiar at all with the internal workings of SQLite, so I'm glad there are people around who are and are willing to make libraries that abstract that away for the rest of us.

Implemented in #647.


Could not load OpenAIEmbedding. Using HuggingFaceBgeEmbeddings with model_name=BAAI/bge-small-en. If you intended to use OpenAI, please check your OPENAI_API_KEY. Original error: No API key found for OpenAI.

Could not load OpenAIEmbedding. Using HuggingFaceBgeEmbeddings with model_name=BAAI/bge-small-en. If you intended to use OpenAI, please check your OPENAI_API_KEY.
Original error:
No API key found for OpenAI.
Here is the notebook for reference: https://colab.research.google.com/drive/1m7dlKYSFdB6CDMoV3iC5AdIBZBHe59b-?usp=sharing
How to use: llamaindex_huggingface_lm_format_enforcer with ServiceContext.from_defaults
import torch
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
from llama_index.llms import HuggingFaceLLM

model_id = 'TinyPixel/Llama-2-7B-bf16-sharded'
device = 'cuda'

if torch.cuda.is_available():
    config = AutoConfig.from_pretrained(model_id)
    config.pretraining_tp = 1
    model = AutoModelForCausalLM.from_pretrained(
        model_id,
        config=config,
        torch_dtype=torch.float16,
        load_in_8bit=True,
        device_map='auto'
    )
else:
    raise Exception('GPU not available')
tokenizer = AutoTokenizer.from_pretrained(model_id)
if tokenizer.pad_token_id is None:
    # Required for batching example
    tokenizer.pad_token_id = tokenizer.eos_token_id

llm_huggingface = HuggingFaceLLM(model=model, tokenizer=tokenizer)

from typing import Optional
from lmformatenforcer import CharacterLevelParser, JsonSchemaParser
from lmformatenforcer.integrations.transformers import build_transformers_prefix_allowed_tokens_fn

def llamaindex_huggingface_lm_format_enforcer(llm: HuggingFaceLLM, prompt: str, character_level_parser: Optional[CharacterLevelParser]) -> str:
    prefix_allowed_tokens_fn = None
    if character_level_parser:
        prefix_allowed_tokens_fn = build_transformers_prefix_allowed_tokens_fn(llm._tokenizer, character_level_parser)

    # If changing the character level parser each call, inject it before calling complete. If its the same format
    # each time, you can set it once after creating the HuggingFaceLLM model
    llm.generate_kwargs['prefix_allowed_tokens_fn'] = prefix_allowed_tokens_fn
    output = llm.complete(prompt)
    text: str = output.text
    return text

display_header("Prompt:")
display_content(prompt)

# display_header("Answer, Without json schema enforcing:")
# result = llamaindex_huggingface_lm_format_enforcer(llm_huggingface, prompt, None)
# display_content(result)

# display_header("Answer, With json schema enforcing:")
# result = llamaindex_huggingface_lm_format_enforcer(llm_huggingface, prompt, JsonSchemaParser(AnswerFormat.schema()))
# display_content(result)
service_context = ServiceContext.from_defaults(llm=llm_huggingface)

gives error:
******
Could not load OpenAIEmbedding. Using HuggingFaceBgeEmbeddings with model_name=BAAI/bge-small-en. If you intended to use OpenAI, please check your OPENAI_API_KEY.
Original error:
No API key found for OpenAI.
Please set either the OPENAI_API_KEY environment variable or openai.api_key prior to initialization.
API keys can be found or created at https://platform.openai.com/account/api-keys

This does not look related to lm-format-enforcer but to llamaindex. Can you get the example to work without the lm format enforcer, but with the same LLM that would be used?

This does not look related to lm-format-enforcer but to llamaindex. Can you get the example to work without the lm format enforcer, but with the same LLM that would be used?

i used the code shared by you

Hi,
The errors in the notebook do not look related to lm-format-enforcer, but rather to the general usage of llama-index.
Note that I also submitted a PR for better lm-format-enforcer integration in llama-index:
https://github.com/run-llama/llama_index/pull/8601
Once that PR is accepted, it will be much easier to integrate lm-format-enforcer with other facilities in llama-index.

Crated an article: https://medium.com/ai-in-plain-english/demystifying-llama-index-with-zephyr-7b-alpha-beta-a-pydantic-powered-query-engine-guide-fb355170b209

Very cool!
I am not a medium member so I can't read the whole thing. I hope you used the official integration that has been released in the meantime :)


Glibc Version

Thank you for making this docker image. Unfortunately for me the generated binary isn't statically linked and the GLIBC version used by the build process is newer than my machine, meaning I can't run the generated executable. Is there a way to specify a different version of Debian with an older Glibc or a way to set it to static linking?

Hey. I appreciate your interest.
I believe what you can do is modify this line and specify an older base Debian image: https://github.com/jonathanlinat/dockerized-trenchbroom/blob/main/Dockerfile.linux#L2C1-L2C26
Ref. https://hub.docker.com/_/debian/tags?page=1&page_size=&name=bookworm&ordering=-last_updated


esp_ota + encrypted firmware example (IDFGH-4334)

Hello. Have anybody example how make ota update with encrypted firmware using esp_ota_begin(), esp_ota_write(); esp_ota_end in release/v4.2 esp-idf ?

@kshyshkin
Default OTA examples shipped with ESP-IDF (ref: https://github.com/espressif/esp-idf/tree/master/examples/system/ota) should work with flash encryption enabled case as well.  esp_ota_write uses underlying partition manager API esp_partition_write, which transparently handles encrypted flash writes.
If you are facing any specific problem please feel to post detailed report per github issue template.

@mahavirj Thank you for your reply. Problem with esp_ota_write for pre-encrypted firmware. esp_ota_write check magic byte and etc as in non encrypted firmware. I try esp_ota_write_offset it works, but after in esp_ota_end i got same situation with checking magic byte. I cant find how solve problem with ota update with pre-encrypted firmware.

@kshyshkin Can you please explain more details about your use-case? Do you have flash encryption enabled on device? If yes, any specific reason to flash pre-encrypted firmware?

@mahavirj Yes all secure features on controller on flash encryption (RELEASE), secure boot reflashable, all needed fuses on. But i have OTA (not through webserver), with intermediate device. When i need update by OTA, i make project, sighn fw with pem key, encrypt fw by pem key, and send to  intermediate device. Intermediate device send pre-encrypted fw to esp32 by custom interface and protocol  dividing the firmware by chunks. I need that from my host computer to esp32 fw goes pre-encrypted.

@kshyshkin
At this moment, pre-encrypted firmware write is not supported in app_update component.

I see in documentation that esp_partition_write_raw function, i think it can help me solve this problem, bu i cant find in which release this function burn.

This API is part of latest master branch and it will reflect in ESP-IDF v4.3 release.
https://github.com/espressif/esp-idf/blob/178b122c145c19e94ac896197a3a4a9d379cd618/components/spi_flash/partition.c#L423
Probably we can add configuration option to write pre-encrypted firmware image in OTA component, will keep you posted on this.

@mahavirj Thank you for your work. I will be wait this option. It will be awesome, because i see that many issues by this feature.

This API is part of latest master branch and it will reflect in ESP-IDF v4.3 release.
https://github.com/espressif/esp-idf/blob/178b122c145c19e94ac896197a3a4a9d379cd618/components/spi_flash/partition.c#L423
Probably we can add configuration option to write pre-encrypted firmware image in OTA component, will keep you posted on this.

Hi @mahavirj
Any progress on this?
esp_partition_write_raw API is available now, but there is no config option to allow esp_https_ota() use it.

Hi @AxelLin,
We had tried to OTA with pre encrypted firmware using raw_write APIs, however there are more dependencies that should be considered.
For example, if the firmware is encrypted the image magic byte present in the header will be changed due to encryption. In order to verify magic byte, we need to write received data to flash and re-read decrypted data. Writing data to flash without verifying magic byte doesn't seem to be a feasible approach.
We are checking possible issues with this feature in esp_https_ota. I'll keep you posted with the updates.
Thanks,
Shubham

any specific reason to flash pre-encrypted firmware during OTA update?

A common use case is the customer's device cannot connect to internet so it cannot reach OTA server.
If it's necessary to provide binary to customers for bug fix, an unencrypted binary is not an option.

Hi @AxelLin,
We had tried to OTA with pre encrypted firmware using raw_write APIs, however there are more dependencies that should be considered.
For example, if the firmware is encrypted the image magic byte present in the header will be changed due to encryption. In order to verify magic byte, we need to write received data to flash and re-read decrypted data. Writing data to flash without verifying magic byte doesn't seem to be a feasible approach.

Hi @shubhamkulkarni97
This seems not a big problem.
If you must write the received data to flash for verifying image magic,
it just change the checking image magic from before writting to after writting.
BTW, do you have patch to test updating encrypted firmware even without checking image magic?
I tried replace esp_partition_write() with esp_partition_write_raw() but it does not work.
I got error message when boot: esp_image: image at ...... has invalid magic byte (nothing flashed here?)

@shubhamkulkarni97
Could you take a look what could be wrong in my changes?
https://gist.github.com/AxelLin/fc509e11f1da3b68bad97d6cd68b8777
The changes including replace esp_partition_write with esp_partition_write_raw,
and ignore image header magic checking (for testing purpose and simplify the changes).
The OTA processing seems done, but after reboot it fails to load new firmware due to invalid magic byte.

Hi @shubhamkulkarni97 @mahavirj
I figure out the problem now.
Encrypt the same image to different flash offset result in different encrypted-binary.
When your current image is running at ota_0, the ota image will be write to ota_1 then you need to use
the encrypted-binary generated for the ota1 offset. If use encrypted-binary generated for the ota0 offset,
in esp_ota_end the esp_image_verify failed.
This is an annoying issue because user needs to know what's current partition to decide which image to download.
(And multiple encrypted-binary for each ota partition offset is required)

For example, if the firmware is encrypted the image magic byte present in the header will be changed due to encryption. In order to verify magic byte, we need to write received data to flash and re-read decrypted data. Writing data to flash without verifying magic byte doesn't seem to be a feasible approach.

@shubhamkulkarni97
I re-read the code and don't understand what's your concern.
The flash is erased either in esp_ota_begin() or esp_ota_write().
i.e. flash is erased before checking image header anyway.
For encrypted firmware, you can check the image header after the first call of writing.
e.g.
            ret = esp_partition_write_raw(it->part, it->wrote_size, data_bytes, size);
            if(ret == ESP_OK){
#if 1
               // read back magic byte from flash
               if (it->wrote_size == 0 && it->partial_bytes == 0 && size > 0) {
                       uint8_t magic_byte;
                       ret = esp_partition_read(it->part, it->wrote_size, &magic_byte, 1);
                       if (ret != ESP_OK) {
                               return ret;
                       }

                       if (magic_byte != ESP_IMAGE_HEADER_MAGIC) {
                               ESP_LOGE(TAG, "OTA image has invalid magic byte (expected 0xE9, saw 0x%02x)", magic_byte);
                               return ESP_ERR_OTA_VALIDATE_FAILED;
                       }
               }
#endif

                it->wrote_size += size;
            }

@AxelLin
You are right about dependency of flash offset address in flash encryption scheme, it comes due to "key tweaking" as highlighted here.
We have been discussing internally on enabling "pre-encrypted firmware distribution" scheme and so far we are considering this to be entirely different than "platform flash encryption" feature.

Primarily because, even for devices where flash encryption is not enabled, it could be requirement that firmware image over OTA is still encrypted in nature.
Additionally for devices where flash encryption is enabled such that key is unique to device and generated by "bootloader" on first boot (recommended for production use-cases), pre-encryption with device key is not possible/feasible approach.

We are still at early discussion stage but probably approach will be to create "pre-encrypted" OTA image format, define encryption scheme, credential storage (use of "DS" peripheral or other approach) etc. This will be format of image for "on-the fly" decryption before writing to device flash, if flash encryption is enabled then device may "re-encrypt" image with platform key before writing to flash.
If you have any feedback on this, please feel free to share.

We are still at early discussion stage but probably approach will be to create "pre-encrypted" OTA image format, define encryption scheme, credential storage (use of "DS" peripheral or other approach) etc.

Note ESP32 does not has "DS" peripheral.

Hi @mahavirj
For firmware image, it can check the ESP_IMAGE_HEADER_MAGIC to recognize it's a vaild image.
For SPIFFS image, is there a standard way to recognize if a file is spiffs image?

@AxelLin
Sorry, I missed your comment.

For SPIFFS image, is there a standard way to recognize if a file is spiffs image?

I was unable to find anything obvious in quick glance through SPIFFS format, looks like successful mount would be required to check authenticity of image. @igrr Do you any other way round this?

I don't think there is an easy way to determine if something is a SPIFFS image just from the header. SPIFFS has a SPIFFS_probe_fs function that ends up reading magic values from 3 filesystem blocks.
If CONFIG_SPIFFS_USE_MAGIC_LENGTH=n (default is =y), then the magic numbers will be the same for all blocks, so one can read the magic value at the end of the 1st page header — see SPIFFS_MAGIC_PADDR macro definition in SPIFFS code for address calculation.

@kshyshkin  I have created a example which uses encrypted image.
https://github.com/hmalpani/esp-idf/tree/feature/encrypted_image/examples/system/ota/pre_encrypted_binary_ota
Please feel free to take a look. It also contains a tool for creating encrypting image which can be used for OTA updates.

So, is there any good news for developer that developed whole project with OTA update (with public supply firmware scenario) and than added secure boot v.2 and encryption and hangs on one month before releasing to production?
I see that there is (officially undocumented) issue with offset parameter used while encryption, dependent of OTA partition number. Happily, It is not big problem to make double image file and choose correct part depended of  running partition on device.
But the second issue is a huge challenge for me. I'm talking about magic byte and writing raw to partition.
Is there any chance that it will be available shortly?
@AxelLin
Did You succeed with Your changes with using esp_partition_write_raw() function?

Hello @jacek12345,
We are going to use the approach as here: https://github.com/hmalpani/esp-idf/tree/feature/encrypted_image/examples/system/ota/pre_encrypted_binary_ota.
The feature will be available soon in IDF.

@hmalpani  I Just try this code I see following error while OTA.
I (20010) esp_encrypted_img: Magic Verified
I (20010) esp_encrypted_img: Reading RSA private key
E (20130) esp_encrypted_img: failed
! mbedtls_pk_parse_keyfile returned -0x3d00
E (20130) esp_encrypted_img: Unable to decipher GCM key
E (20140) OTAUpdate: Error in decoding binary
E (20140) OTAUpdate: OTA failed
Let me know if you need more info. Any help will be appreciated
Thanks

@hmalpani I Just try this code I see following error while OTA. I (20010) esp_encrypted_img: Magic Verified I (20010) esp_encrypted_img: Reading RSA private key E (20130) esp_encrypted_img: failed ! mbedtls_pk_parse_keyfile returned -0x3d00
E (20130) esp_encrypted_img: Unable to decipher GCM key E (20140) OTAUpdate: Error in decoding binary E (20140) OTAUpdate: OTA failed
Let me know if you need more info. Any help will be appreciated
Thanks

It seems like you are using the wrong RSA private key. Please make sure you are using the correct RSA private key. You can find info about how to generate RSA key in README.md

Thanks for the response. I tried the same steps.
I am generating certificate on windows machine. Let me try it again. Let me know if there is any version dependency on Python version.

@mdmodi
This feature is now supported in official ESP-IDF and corresponding example is available at https://github.com/espressif/esp-idf/tree/master/examples/system/ota/pre_encrypted_ota
As a first step, I would recommend that you skip the step of "RSA key generation" and use the test key supplied in the example itself. Once we confirm that this setup works on your side, we can chase on key generation problem separately.
Please let us know update on this.

@mahavirj
i tested this example but i get this error
../main/pre_encrypted_ota.c:44:30: error: unknown type name 'decrypt_cb_arg_t'; did you mean 'esp_decrypt_cfg_t'?
static esp_err_t _decrypt_cb(decrypt_cb_arg_t *args)
^~~~~~~~~~~~~~~~
esp_decrypt_cfg_t
../main/pre_encrypted_ota.c: In function 'pre_encrypted_ota_task':
../main/pre_encrypted_ota.c:108:10: error: 'esp_https_ota_config_t' {aka 'struct '} has no member named 'decrypt_cb'
.decrypt_cb = _decrypt_cb,
^~~~~~~~~~
../main/pre_encrypted_ota.c:108:23: error: '_decrypt_cb' undeclared (first use in this function)
.decrypt_cb = _decrypt_cb,
^~~~~~~~~~~
../main/pre_encrypted_ota.c:108:23: note: each undeclared identifier is reported only once for each function it appears in
ninja: build stopped: subcommand failed.
ninja failed with exit code 1
Note: I didn't change anything in this examples

@hmalpani  thank you very much, but when I work on Arduino as an ESP-IDF component I got this error
main.cpp:80:42: error: invalid conversion from 'esp_decrypt_handle_t' {aka 'void*'} to 'void**' [-fpermissive]
ctx = esp_encrypted_img_decrypt_start(&cfg);

@saidsnanou1 This must be resolved by https://github.com/espressif/idf-extra-components/pull/17

Can we set the FLASH_CRYPT_CONFIG to 0x0, then any address offset of pre-encrypt binaries should be ok to read and write.

Can we set the FLASH_CRYPT_CONFIG to 0x0, then any address offset of pre-encrypt binaries should be ok to read and write.

Can you please elaborate? Just to note that, pre-encrypted OTA is entirely different scheme and has no co-relation to platform flash encryption feature as such.

@mdmodi
This feature is now supported in official ESP-IDF and corresponding example is available at https://github.com/espressif/esp-idf/tree/master/examples/system/ota/pre_encrypted_ota
As a first step, I would recommend that you skip the step of "RSA key generation" and use the test key supplied in the example itself. Once we confirm that this setup works on your side, we can chase on key generation problem separately.
Please let us know update on this.

Hello,
I am also interested in that feature but I would like to stay on the latest stable release (V4.4.X). Is there any luck to see that feature in stable branch v4.4.X in the near future?
Best regards

@cl-cr We are not planning to backport this feature to IDF v4.x. It will be part of v5.0 release. However, I had create a branch on top of v4.4.2 release with this feature, please see discussion here https://esp32.com/viewtopic.php?f=13&t=29917#p103742

@mdmodi , Any luck with resolving error -0x3d00 ? I'm getting the exact same error and i've tried the key gen process multiple times.
I (40450) esp_encrypted_img: Magic Verified
I (40452) esp_encrypted_img: Reading RSA private key
E (40454) esp_encrypted_img: failed
  ! mbedtls_pk_parse_keyfile returned -0x3d00

E (40455) esp_encrypted_img: Unable to decipher GCM key
E (40456) OTA: OTA_CMD_WRITE Encrypt failed: ABORT ESP_FAIL
E (40457) file_server: Failed writing block update -1

@mahavirj Is there any update on how to update intermediate device with encrypted firmware? The intermediate device will receive OTA chunks with UART communication protocol.

@mahavirj Yes all secure features on controller on flash encryption (RELEASE), secure boot reflashable, all needed fuses on. But i have OTA (not through webserver), with intermediate device. When i need update by OTA, i make project, sighn fw with pem key, encrypt fw by digest from pem key, and send to intermediate device. Intermediate device send pre-encrypted fw to esp32 by custom interface and protocol dividing the firmware by chunks. I need that from my host computer to esp32 fw goes pre-encrypted. Thats why i need store pre-encrypted fw by chunks to OTA partition.


Add oauth2client to reqs and bot url config to readme

@erickoledadevrel of course :) thanks for the catch. Changed accordingly.

Thanks for the submission!


Add CI job for running Kubernetes external-storage e2e tests

This adds a CI job for running the Kubernetes external-storage e2e tests against a PR. The job can be triggered with /test ci/centos/k8s-e2e-external-storage. There will be an attempt to test RBD and CephFS, of which RBD is expected to succeed, whereas CephFS seems to have some failure (probably in the driver configuration, TO BE TESTED).
This PR alone is not sufficient to run the tests, see #1526 for the additions in the master branch.
Merging this PR should not affect any other testing, the job is not started for all PRs, only when requested with a comment.

After this is merged, #1526 can be tested by leaving a comment
/test ci/centos/k8s-e2e-external-storage

mostly questions

The files with the description of the drivers and their capabilities have been moved to #1526. These are files with minimal functionality for testing the CI job. Enabling additional functions can be done at a later time.

@Yuggupta27 @Madhu-1 could you review and approve this? Once merged, the job is not active, but can be triggered with
/test ci/centos/k8s-e2e-external-storage

Thanks!

/retest ci/centos/jjb-validate

/retest ci/centos/jjb-validate

This mostly fails if there is an other jjb-validate job running, see #1624 for a fix

/retest ci/centos/jjb-validate

/retest ci/centos/jjb-validate


split elyra from jkg

Changes necessary to reference the elyra artifacts that have now been split from JKG.
Fixed issue where the proxy launch log was using /var/log instead of /tmp, but /var/log is only writable (on HDP at least) by root - so I reverted back to using /tmp until further notice.
Also removed reference to the data directory - that should have been removed in previous change.
Note that this PR should not be merged until the corresponding PR is merged in the Elyra project.

LGTM, PR Merged


Wheel support for linux aarch64[arm64]

Summary
Installing jpype on aarch64 via pip using command "pip3 install jpype" tries to build wheel from source code
Problem description
jpype doesn't have wheel for aarch64 on PyPI repository. So, while installing jpype via pip on aarch64, pip builds wheel for same resulting in it takes more time to install jpype. Making wheel available for aarch64 will benefit aarch64 users by minimizing jpype installation time.
Expected Output
Pip should be able to download jpype wheel from PyPI repository rather than building it from source code.
@jpype-team, please let me know if I can help you building wheel/uploading to PyPI repository. I am curious to make jpype wheel available for aarch64. It will be a great opportunity for me to work with you.

Do you feel like sending us a PR for this? You can take a look into .azure/release.yaml for the build steps of the release pipeline.
There is a manylinux docker image available for ARM64, so this should be usable and easy to integrate (https://quay.io/repository/pypa/manylinux2014_aarch64)

@marscher Thanks for your quick response, I will start working on it and will raise a PR.

I believe this was resolved with #910

I believe this was resolved with #910


Json Support

Currently the RosalilaParser class supports XML only. But we could switch to Json o better yet, support them both!

On mod.io I'm using JSON for Modern C++ and it has been working great
https://github.com/nlohmann/json


NoAvailableBackendException on Android devices

Everything compiles fine, but comes down to this. Am I missing a step?

Included nd4j-java.jar

@VicV
Can you give me a tutorial how to use nd4j for android?
many thanks for help


Release/2.7.1

Release 2.7.1
This is a patch release of 2.7.0 including fixes to build on Freia (UKAEA) and fixes to Cap'n Serialisation from issues found during development of IMAS remote data access.
We are also now including the C++ guideline support library (GSL) to avoid needing to rely on Boost::span which is only available in newer versions.

question why do we ship the configuration files like imas and mast (under /etc/machine.d)...these are specific files for a particular site?
is it the plan to collect there all the sites configuration?

question why do we ship the configuration files like imas and mast (under /etc/machine.d)...these are specific files for a particular site? is it the plan to collect there all the sites configuration?

It's mostly for convenience. These are the configuration files for running the server on those machines.
I agree this is probably the wrong place for them and we should look to move them elsewhere, especially now UDA is a public repo.

Merging this for 2.7.1 release. We will look at the ITER CODAC changes for 2.7.2 release.


Missing dependencies in package.json

Im using JSPM to package your module but it has problems doing so because package.json does not declare 'prop-types' or 'object-assign' as either dependencies or peerDependencies. This results in a runtime error loading your module because JSPM cant include undeclared dependencies without some extra JSPM override config.
Would you consider accepting a P.R to correct this?

Absolutely, please go ahead and I’ll have it merged ASAP
tir. 19. feb. 2019 kl. 05:05 skrev Colm<EMAIL_ADDRESS>
Im using JSPM to package your module but it has problems doing so because
package.json does not declare 'prop-types' or 'object-assign' as either
dependencies or peerDependencies. This results in a runtime error loading
your module because JSPM cant include undeclared dependencies without some
extra JSPM override config.
Would you consider accepting a P.R to correct this?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/svenanders/react-iframe/issues/44, or mute the thread
https://github.com/notifications/unsubscribe-auth/ABfQp_Kdv_9-GwwCaQp78-l5fnoUmSaZks5vO3fsgaJpZM4bCIOn
.

--
Mvh, Sven Anders Robbestad
https://www.robbestad.com http://www.robbestad.com/

Thanks for the quick turn around. Closing as this is resolved via https://github.com/svenanders/react-iframe/commit/e04af2d49aaca5bc4ffb4b4a0cad292394adee1f


colorlover v3

hey @santosjorge - I threw in the latest colorlover version in here too

Thank you very much Chris!


[Format] Consider adding an official variant type to Arrow

Describe the enhancement requested
This could be aligned with the new Spark variant type or it could not be.
Component(s)
Format

What would be the benefit of this over the current Union types? Is it just to alleviate the need to specify all the types up front?

What would be the benefit of this over the current Union types? Is it just to alleviate the need to specify all the types up front?

That's part of it, yes, but many sources also support variants containing semistructured types where you could conceivably need a combinatorial explosion of unions to reflect all the data. (See https://docs.snowflake.com/en/sql-reference/data-types-semistructured, for instance.)

I’m investigating using the Spark spec as an extension type in DataFusion. I’ll report back here whether it turns out to work well with Arrow layouts.

I talked to developers at Databricks who worked on adding this feature to Spark and Delta Lake. Here a few notes from that.

This is being added as a data type in Spark and Delta Lake. They intend to add this data type to Iceberg as well.
They called it the “Open Variant Data Type” with the intention that this data type would proliferate to other systems.

They have a standalone Java library that implements the data type. That’s the Java code at https://github.com/apache/spark/tree/master/common/variant

The data is stored as two binary fields: one to hold a string dictionary, the other to hold the binary representation of the values. It is generally kept as binary data in memory, but engines are free to manipulate it as they wish.
The eventual plan is to support record shredding, where fields that have dense values will be split out into their own columns. This allows row group / page pruning to happen with normal Parquet statistics / indices.

Record shredding will have to be the same per Parquet file, but could be different between files.
Once in memory, variants will be either recombined into the two binary columns or else have been selected back into their fully shredded forms. This is because most engines will require a common schema across files. The good news here is that means by the time it might be exported into Arrow data, we wouldn’t have to worry about the shredding.

Performance justification: JSON and BSON are not designed for OLAP queries.

The canonical pathological case is where you are extracting the last field in a large object. JSON has to do O(n) string comparisons, the variant form replaces them with integer comparisons.
The main performance optimization is that object keys (and other common strings) are pulled out into a common string dictionary. This reduces the size, but also replaces all the string comparisons needed in field lookups with integer comparisons.

An Arrow extension type?
In the near term, I think this would make a good Arrow extension type. This would be:
struct<
  metadata: dictionary<binary>,
  data: binary
>

The metadata will usually be a single binary shared across all rows, but could be multiple. (Multiple might happen if two different batches are concatenated together, for example.) Either dictionary or REE encoded array would be appropriate.
The data could be either binary, large binary, or binary view.
Binary view isn’t widely supported right now, but could be very useful for this data type. This is because sub-objects can be sliced out of variants. From the spec ^1:

Another motivation for the representation is that (aside from metadata) each inner Variant value is contiguous and self-contained. For example, in a Variant containing an Array of Variant values, the representation of an inner Variant value, when paired with the metadata of the full variant, is itself a valid Variant.

Where could this be useful?
A few immediate places I think this extension type could be useful:

Roundtrip variant Arrow ↔ Spark

Spark Connect (and any ADBC connector to that) would benefit from this

Extension type in PyArrow, roundtrip PySpark ↔ PyArrow
DataFusion function library (I’m experimenting with that now)

There's been substantial interest in DataFusion community for a way to handle semi-structured data efficiently.

Extension type pitfalls
The main pitfall of using an extension type for this is the storage type is meaningless to users. They need to have special libraries to interpret the bytes if pulled into a system that doesn't understand the variant extension type.
In addition, most existing Arrow systems I've worked with don't have a way to customize how extension arrays are printed. I think this is something we should fix. A reasonable workaround in the meantime is providing functions that convert these back to JSON strings for the purpose of printing.

I think this makes sense as a extension type.  I think given subcolumnarization work happening one might also want to store a union in the type as well for columns that have been split out

One other thought, I think the variant type in spark has a more limited type surface then Arrow, that is potentially something that might need reconciling

Yeah, I think there are really two different requests possible here: an Arrow-native variant type and a Spark-compatible variant type.   The surface area thing works both ways: like Parquet, the Spark variant supports 32-bit and 64-bit decimal values while Arrow does not.

Yeah, I think there are really two different requests possible here: an Arrow-native variant type and a Spark-compatible variant type.

I think it’s too early to say whether an Arrow-native one makes sense. The Spark / delta lake teams have intentions that their standard will proliferate to other engines. At which point it will not be a Spark specific thing and might make more sense to align with these types. If that succeeds, it would make sense for us to align with the format. The Open Variant Data Type has a version field, so it could be amenable to expansions of the types in there.
If this standard doesn’t proliferate to other engines and ends up being Spark specific while other engines maintain different standards, we will have to have a conversation about what kind of variant type would make a good interchange format. That would be a point where Arrow designing its own format would make sense.
Either way, it’s much too early to know which direction to go. Spark 4.0 isn’t even release yet. I think this is the stage where we should experiment with this type as a non-canonical extension type and keep an eye on the data types adoption in the wider ecosystem.

Yeah, fwiw there is an iceberg proposal to also support variant type and if IIUC the current incarnation is to support spark with iceberg types but it hasn't made it very far yet

Even if there were no Spark (or Iceberg) variant type there would still be variants stored in databases and it would be nice for ADBC to be able to return those in a somewhat-consistent fashion. I suppose ADBC could define its own extension type for this purpose.

For curious observers, there's a thread about the Iceberg proposal at https://lists.apache.org/thread/xnyo1k66dxh0ffpg7j9f04xgos0kwc34

Binary view isn’t widely supported right now, but could be very useful for this data type. This is because sub-objects can be sliced out of variants. From the spec 1:

BTW we are actively working on implementing StringView / BinaryView support in arrow-rs https://github.com/apache/arrow-rs/issues/5374 and DataFusion https://github.com/apache/datafusion/issues/10918 and thanks to @XiangpengXao, @Weijun-H  and other we are making good progress

Parquet recently added the variant spec and variant data type.
https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
Is there any plans in arrow to adopt it soon?

@julienledem and I were just talking about this. I agree it would be nice to add the variant type to arrow -- I think the challenge will be finding people willing to help implement it in two languages. I don't think I will have time to help with Rust anytime soon, though I can help coordinate and I'll see if I can muster anyone to help.
I filed this to track the ideae

https://github.com/apache/arrow-rs/issues/6736

What would be required on the Arrow side of things? Just an extension, or would we also need methods to access/manipulate the content as well?


Fixed  .\eslzArm\managementGroupTemplates\policyAssignments\DINE-VMBackupPolicyAssignment.json

Overview/Summary
Replace this with a brief description of what this Pull Request fixes, changes, etc.
This PR fixes/adds/changes/removes

This PR fixes Issue https://github.com/Azure/Enterprise-Scale/issues/791

Breaking Changes

Otherwise, we are not able to deploy DINE-VMBackupPolicyAssignment.json in the Azure Landing Zone accelerator DIY.

Testing Evidence
PS C:\Users\hoongfai\Enterprise-Scale> New-AzManagementGroupDeployment -Name "$($DeploymentName)-vm-lz-backup" `

                            -Location $Location `
                            -ManagementGroupId "$($ESLZPrefix)-landingzones" `
                            -TemplateFile .\eslzArm\managementGroupTemplates\policyAssignments\DINE-VMBackupPolicyAssignment.json `
                            -topLevelManagementGroupPrefix $eslzPrefix `
                            -Verbose

VERBOSE:
VERBOSE: 4:36:07 PM - Template is valid.
VERBOSE: 4:36:08 PM - Create template deployment 'Eslz-vm-lz-backup'
VERBOSE: 4:36:08 PM - Checking deployment status in 5 seconds
VERBOSE: 4:36:13 PM - Resource Microsoft.Authorization/roleAssignments '9427f73d-50fd-5d2f-92bb-2b2163c15203' provisioning status is succeeded
VERBOSE: 4:36:13 PM - Resource Microsoft.Authorization/policyAssignments 'Deploy-VM-Backup' provisioning status is succeeded
VERBOSE: 4:36:13 PM - Resource Microsoft.Authorization/policyAssignments 'Deploy-VM-Backup' provisioning status is succeeded
Id                      : /providers/Microsoft.Management/managementGroups/ALZ-landingzones/providers/Microsoft.Resources/deployments/Eslz-vm-lz-backup
DeploymentName          : Eslz-vm-lz-backup
ManagementGroupId       : ALZ-landingzones
Location                : chinaeast2
ProvisioningState       : Succeeded
Timestamp               : 8/09/2021 6:36:13 AM
Mode                    : Incremental
TemplateLink            :
Parameters              :
Name                             Type                       Value
===============================  =========================  ==========
topLevelManagementGroupPrefix    String                     ALZ
enforcementMode                  String                     Default
Outputs                 :
As part of this Pull Request I have

[x] Checked for duplicate Pull Requests
[x] Associated it with relevant issues, for tracking and closure.
[x] Ensured my code/branch is up-to-date with the latest changes in the main branch
[x] Performed testing and provided evidence.
[ ] Updated relevant and associated documentation.
[ ] Updated the "What's New?" wiki page (located: /docs/wiki/whats-new.md)

@faister - It looks like your branch contains a lot of files that are not related to this specific issue? Can you please remove them from the PR?

yup sorry, will clean this up, and recreate my branch and PR


Hard to separate working blocks from failing ones

I know the docs are autogenerated. But maybe there's a way to do that:
Currently, every block is formatted the same way in the blocks. It would be much clearer to use some sort of color coding to separate the "good" examples from the ones that fail tests. A good example for color coding is found on bootstrap.
Would be looking like that:

I agree that it would be good to put some thought into UI/UX for this.  I'm not sure if colors are the best way (considering r/g colorblind folks), but perhaps it could be used in conjunction with a larger redesign at some point?  I'd love to see some consistency between how options are explained and examples are given.  I just don't know how that design should look.

Colors were just the most obvious way I came up with to solve the problem. Currently, we are all treated as colorblinds.
If you want to tackle it at a more general level, perhaps a table is a better way to display qualities:
Rule: yoda
default: "never"

Configuration
validates
fails

"yoda": [2, "never"]
if (color === "red") {}
if ("red" === color) {}

The problem becomes how to generate such changes automatically. It's unlikely we will get a volunteer to reformat every for by hand, so what can be done such automation?

If we can find a solution for this that will also solve the remaining blocker in my mind on eslint/eslint#2271 (how do we lint valid examples but only check invalid examples for syntax?), then we could solve this at the same time we (likely meaning "I") go through and use configuration comments (related discussion eslint/eslint#3208) for all the examples.

Is it just a matter of wrapping code blocks with a div of an appropriate class?  I don't know how the file loader / documentation linting works, but that could potentially be used at least for styling.

And @btmills I would be happy to change docs to use configuration comments. Your mind/time is better utilized on less tedious activities.

I don't think it's a matter of wrapping the code in a div. We write everything in MD format. And code blocks are converted into
 tags by jekyll. I don't actually know how we can achieve valid vs. invalid styles with .md

// md code block here

Is valid markdown, and the background can be styled with:
.passing pre {
    background:green;
}

I've tested it out in my blog (which also uses markdown) and it worked, and can try it on the this repo tonight.

@IanVS If your example works out, would you agree with my idea to use tables? It'd be basically the same approach:

// md table here

// css:
.yep-nope-table {
    td:first-child { /* config block */ }

    td:first-child + td { /* yep block */ }
    td:first-child + td + td { /* nope block */ }
}

I can try tables as an experiment, but in my experience markdown tables are kind of a pain to work with.   I do think in general the result might look nicer and be clearer, but we could run into the issue of wrapping lines, which is always ugly in code examples. Also, changing to tables would require much more drastic changes to all of the rule docs. I'll try it out though.

Just to reiterate: anything manual is not a good solution. These are manual solutions which mean we are going to have to manually convert everything and also manually enforce their usage on pull requests. In both cases, this is a lot of work, so I'm skeptical this is a good direction.

What if the examples in the docs were all live? If the demo could be made into something reusable, then invalid examples would have the actual error messages right there.

And how would you automate that?

For illustration, I'll borrow an example from eqeqeq:
--
The following patterns are considered okay and do not cause warnings:
/* eslint eqeqeq: [2, "smart"] */
typeof foo == 'undefined'
'hello' != 'world'
0 == 0
true == true
foo == null

The following patterns are considered warnings with "smart":
/* eslint eqeqeq: [1, "smart"] */
// comparing two variables requires ===
a == b

// only one side is a literal
foo == true
bananas != 1

// comparing to undefined requires ===
value == undefined

--
npm test runs the linter on the docs using the Markdown preprocessor. Makefile.js will fail the build on errors (parse errors in valid and invalid examples and lint errors in the valid examples), and it will suppress warnings in the docs (from lint warnings in the invalid examples). This reduces our manual workload on PRs from manually checking and verifying examples for both syntax and linting down to making sure examples have a configuration comment that enables the rule.
The live demo part is hazier, as I don't know how the page build process works. @xjamundx is there (or could there be, with the work you're doing) a way to load a demo on code blocks within the page?

Interesting. And how do we update the already existing docs?

I've volunteered to update them.

Cool :)

I don't think it's a matter of wrapping the code in a div.

You're right, this doesn't work in redcarpet, Jekyll's markdown processor.

@Nirazul @IanVS @nzakas @btmills @ilyavolodin
Can you think of any additional objectives which are relevant to this issue?

Given incorrect or inconsistent code, find a rule (and its options) to prevent it.

Given a rule (and its options) in the configuration for a project, write correct code.

What do you think about the following pattern for the sentence preceding an example of code?

The sentence begins with Examples of.

Adjectives incorrect and correct decribe the code to avoid confusion with rule configuration error levels: warning or error.

Omit any verb to avoid the passive voice: are considered.

Make the rule configuration explicit:

Examples of incorrect code if you turn on the rule:
Examples of incorrect code for the default "always" option:
Examples of incorrect code for the "never" option:
Examples of incorrect code for the {"builtinGlobals": true} option:
Examples of incorrect code for the browser environment and the {"builtinGlobals": true} option:
Examples of incorrect code for the {"anonymous": "always", "named": "never"} option:
Examples of incorrect code for the {"anonymous": "never", "named": "always"} option:

Because the sentence includes configuration, what do you think if we omit punctuation in h3 headings for options to make a potential table of contents easier to scan?
Rule Details
Options
  always
  never
  anonymous
  named
When Not To Use It
Related Rules
Version
Resources

Considering the examples (in several following comments) which contrast baseline grammar and formatting with suggested changes, how well do you think the suggestions support the objectives?
ESLint members: and how realistic do you think it is for future editors of rule pages (including first-timers) to follow the suggested pattern (to a reasonable extent) after the initial rewriting, which I am willing to do?

Rule has no options: no-console
Baseline:

Suggested:

The default value of an option: consistent-this
Baseline:

Suggested:

A second set of examples for an option: consistent-this
Baseline:

Suggested:

Another value of an option: space-before-function-paren
Baseline:

Suggested:

An object value of an option: space-before-function-paren
Baseline:

Suggested:

@pedrottimark I like it. I never cared much for "patterns are considered problems" I also like green/red backgrounds too. Maybe we can combine those two?

I think the green/red problem is a technical one.  I haven't found a way to accomplish it, but would love it if we could do at least some subtle shading differences.

Let’s take a background distinction as part of the solution and trust that we will find a way. Just saw some resources for accessible color contrast ratios, so will make some example pictures to discuss.
In addition to distinguishing incorrect from correct, readers can quickly distinguish these examples from any general examples of code or configurations.
A question which I forgot to ask: If only some lines in a code example require es6 environment, what do you think about separating them? Compare the following to the preceding picture:

I think in the scope of a larger page, having a bunch of smaller wells might be slightly disconcerting. We should also think about possible future were we will support ES7 as well and possibly TypeScript/Flow. That might be just too many wells to deal with.

Good point that we need to limit the total visual complexity.

The lighter background for inline code has made some pages more readable for my eyes. #202
That reminds me to ask how likely the text of code examples might have coloring in the future, because that affects how much background color or pattern will be reasonable. #194

I'll try to do kramdown upgrade this week, that might bring syntax highlighting, but I'm not 100% sure.

@ilyavolodin any luck with Kramdown?

I tried it locally, and found two major issues. Code fences were not working and H1 headers were missing on rule docs. Code fences just required config switch, but H1 headers require modification of our build script. We do not leave empty line before H1s, so they don't render correctly. I'm not sure if there are any more issues, I've been pretty busy the last few days and haven't had the chance to look into it more.

The following comments have pictures for your feedback about how background color could help people quickly see whether code is an incorrect, correct, or general example.
After the example sentences become consistent, a possible solution to the technical problem is replace rules in the doc.html layout file:
replace: '
Examples of incorrect code', '
Examples of incorrect code'
replace: '
Examples of correct code', '
Examples of incorrect code'
corresponding to CSS rules:
p.incorrect + div > pre { background-color: … }
p.correct + div > pre { background-color: … }

Distinguish general code in the introduction of no-shadow from incorrect code:

Distinguish incorrect from correct code for the default option of comma-dangle:

Distinguish incorrect code from correct code in adjacent options of no-shadow:

That looks nice, and might work in terms of markup. Although it means we need to go and put paragraphs with classes around each code example. I'm not sure you can do that with markdown.

I was thinking the replace rules take effect after the conversion from markdown to HTML?
So that the only requirement in the content is consistency in the beginning of the sentence.
Imitating the following:
replace: '(recommended)', ''
replace: '(fixable)', ''

You are right, replace kicks in after conversion. Sorry, didn't notice that you mentioned replace. That might work

I'll try to do this over the weekend and see if it works.

Here is the CSS I used to make those picture:

What do you think about this blue/yellow color contrast, which is more accessible than red/green?

Here are the recommended changes:

inline code: black on light gray like GitHub, instead of ESLint theme lavender (not in this picture)
generic pre: light gray like GitHub
incorrect pre: ESLint theme lavender because this is the “lint”
correct pre: ivory

While I understand that red/green is less accessible to some, it's much more standard to most. I think I would just stick with light red/green for incorrect/correct examples.

I think red/green is okay, especially since we are not relying on the color coding.  We've all made it this long without it, after all.


Remove wasabi

+1
Wasabi is being listed but no disclaimer is in place to the fact they work with chain analysis and surveillance  companies to censor UTXOs that want to participate in the coinjoins. They effectively use their profits to help fund anti-freedom and surveillance companies and are a net negative for privacy in the space

ACK

Thanks

based


S3 doesn't work on Swarm bucket name containers invalid chars

Summary
When using  Docker swarm and deploy a stack with the volumes defined like the following:
version: '3'
...
volumes:
	test:
		driver: rexray/s3fs

When deployed with have the prefix of the stack name for example docker stack deploy -c ./deployment.yml web will make the volume name web_test which then means the rexray/s3fs driver tries to make a bucket web_test which has invalid characters (the underscore). I have tried Setting volumes.test.name and volumes.test.driver_opts.name but neither overwrite the volume name.
Version
rexray/s3fs:latest
Expected Behavior
A) Either accept the name/bucket as a driver_opts so you can set volumes.test.drivers_opts.bucket .
B) Get word after the last underscore web_test => test and use as bucket name.
Actual Behavior
Invalid bucket name from volume name in a Swarm stack deployment.

Is this a problem that was introduced in the latest versions?
Reading a non related issue from 6 months ago, I got the feeling that rexray+s3+swarm used to work before:
https://github.com/thecodeteam/rexray/issues/908

I might as well misunderstood #908. Seems swarm is configured to use a previously created volume. Would that be a good alternative to circumvent the present issue?

That Swarm as you say uses a volume already created, but for some reason when I did it that way it didn't find the volume, it was as if it wasn't accessible by Swarm because Swarm had not created it, I could be wrong.

@bweston92 I tried it earlier and didn't see any issue. Did you specify "external: true" in your volume specification in the compose file?
Nonetheless, I think the issue you reported is a basic feature that is required for rexray to work with Swarm.

@ptemmer not to go off topic but yes, Rexray created the default volumes for buckets and I was unable to use them when targeting them with the following I just got Swarm waiting to schedule onto a node with the volume. (Docker 17.12)
volumes:
  myvolumename:
    external: true

Ah. I just did a quick test with with a single node swarm. So the volume was there anyway.

Any update on this?

I encountered this as well today, and found this workaround:
If you use compose file version 3.4, you can specify a name option for the volume like this:
volumes:
  myvolumename:
    name: myvolumename
    driver: rexray/s3fs

This will create a volume with the name myvolumename.

Related to #1120

Christiaan, works like a charm!!
Bedankt man

@ptemmer it works with the name key?
I did try that, in my original message I put "I have tried Setting volumes.test.name  and volumes.test.driver_opts.name but neither overwrite the volume name."
Will try again later.

Could you check that you actually used compose version 3.4 or higher for the file? I've had it fail silently on 3.3

@christiaangoossens that might be the case, will report back asap


feat(profile): add organizations within profile modal

As a user, I want my organization to appear under the username in the Profile Modal so that I can confirm correct information.
Acceptance Criteria:
user organization will appear under the username and above email address on Profile Modal

It works, however, I'd make it stylistically look more like the Listings, rather than a floating bullet with no context.

Added styling

:+1:

👍


[FEATURE] Create a custom hook for async queries

Is your feature request related to a problem?
Currently 3 plugin components make direct queries to S3. Not having a high level function creates a lot of issues on code duplication and long term maintenance. Creating a custom hook for async queries will help solve these issues

[ ] Create a custom hook for async queries
[ ] Re-use hook in datasources plugin component
[ ] Re-use hook in integrations plugin component
[ ] Re-use hook in explorer plugin component

Can we document the API that the hook will expect from the server endpoint(s)? At some point I'm going to be moving integrations setup to async jobs and would like if we could reuse the same hook.

From @Swiddis : we need to also sanitize the queries in his new hook: https://github.com/opensearch-project/dashboards-observability/pull/1605#discussion_r1534580812
First implementation for the hook is here. Still can be improved: https://github.com/opensearch-project/dashboards-observability/blob/76d206c18bf412d9c4c541f06c991843a113ac7d/public/framework/datasources/direct_query_hook.tsx


Add support for pg 7.x

Pull Request Test Coverage Report for Build 779

0 of 0   changed or added relevant lines in 0 files are covered.
11 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.07%) to 71.05%

Files with Coverage Reduction
New Missed Lines
%

dist/plugins/integrations/PostgreIntegration.js
11
70.0%

Totals

Change from base Build 774:
-0.07%

Covered Lines:
2601

Relevant Lines:
3408

💛  - Coveralls


Support for URI (or similar) approach when specifying projects

From here: https://github.com/dagger/dagger/issues/3275#issuecomment-1271983052
Or if we want to be really fancy, you could pass a uri to the engine cli as an arg:

local://path/to/dagger.json
json://{"literal":"json"}
git://https://github.com/....

Also came up in a slightly different form here: https://github.com/dagger/dagger/pull/3281#issuecomment-1275522659
Point being that short ways of specifying the location of the project is desirable for CLI interfaces (or anything where it needs to be written by hand)

Addressed in modules implementation


Question:  what does the Sta in StaSh stand for?

Pythonista shell

Yep what @dgelessus said.
sta may not be the best brief for Pythonista but it forms a nice word with the following sh part. So I just thought the word reads nicely and kinda have the meaning that it is a confined environment and some awesome treasure could be found within.

I'll add this background info in README. Thanks!

Background section added.


Localized Gender for Name Generator

This allows the name generator to use localized genders. "male" and "female" can still be used.
Tested in the character creator and the name button of the NPC and Creature sheets.

Added a little change. Is this a better practice? If not I can revert it.

I do like it better