In SCIM2 multi attribute filtering fails when groups are retrieved which are in a secondary Postgre user store

Steps to reporduce

Create a Postgre secondary user store. Create a group( Eg: eng1) and assign a user (Eg: user1) to it.

Now list all the groups in secondary user store. Notice that group eng1 is retrieved.

Now filter user user1 in users endpoint. Notice that group eng1 is in the responce.

Now use a multi attribute filter in the users endpoint and filter from userName starts with u and groups sw e [1].

Expected behaviour is to retrieve 'user1' in the reponse. Instead following error is displayed [2]. This happens when a group is combined with another filter attribute. This needs to be fixed.
[1] - curl -v -k --user admin2@t.com:admin2 'https://localhost:9443/t/t.com/scim2/Users?filter=userName+sw+s+and+groups+sw+e&domain=test1'
[2] - {"schemas":["urn:ietf:params:scim:api:messages:2.0:Error"],"detail":"Error in filtering users by multi attributes ","status":"500"}

This issue is being closed due to extended inactivity. Please feel free to reopen it if further attention is needed. Thank you for helping us keep the issue list relevant and focused!


Modules list hash

Should hopefully close #55

LGTM :+1: Thanks :exclamation:

Released as v0.0.4 :rocket:


Bar Chart Rendering Issue in Angular Ionic

Expected Behavior
I've been following every available video tutorial or documentation on Chart.js to produce a line and bar chart on the same page of my Ionic app which is built on top of an Angular frontend. I would expect both the bar chart and the line to render.
Current Behavior
However, my expectation is not what happens. For some reason, the line will render on the start of the page, but the bar chart will not. I figure this may be a result of the line data being static and the bar data being dynamically loaded from Firebase. I also figure this may have something to do with my lifecycle events, but I've tried everything. Even stranger, the bar finally renders on resizing the window.
Environment

Chart.js version: Latest
Browser name: Chrome

https://paste.ofcode.org/TD7mjwraYbuiH6cqqpnueM

Did you call chart.update() after updating the bar chart data? If it still doesn't work, please provide a minimal jsfiddle or similar live test case that reproduces your issue because maintainers are not necessary familiar with Angular and Ionic.

I did try updating the charts, but that didn't do anything. And I'm not sure how to provide a demo given that this project relies on Ionic, Angular, and Firebase. For obvious reasons, I can't provide my Firebase credentials.

@mbrahimi617 in your paste.ofcode, I can't find where you are calling this.bChart.update() after updating your data in list.ref.get().then(...). Are you sure your are explicitly calling the update() method on your chart after updating your data asynchronously?

I just added it in my local code as per your suggestion and it didn't do anything.

Well, I'm not sure how we can help more without any way to debug your code :\


fleetctl and etcdctl are not getting compiled at vagrant up

Current work-around run the commands after vagrant ssh into the VM:
sudo env "PATH=$PATH" /usr/local/fleet/build
sudo env "PATH=$PATH" /usr/local/etcd/build


handling account identification variants

in this variant, the bank code is shorter
and the currency is separated with an additional
/. this was happening for an austian bank

is this somewhere specified?
or does the bank have a spec that we can link to?

happened in data received from the bank, i guess no spec ... like always :-/

I mean we have mt940 file descriptions from various banks.
maybe we can mention the bank somewhere with that format?
I am fine merging this, seems to not change any other behaviour.

@kangguru what do we do with this? should this go in the next release?


zoom-us-ios-sdk-pod throws issue when running in iOS

Description
After installing the "zoom-us-ios-sdk-pod" running the project, the following error gets thrown

Property 'userToken' not found on object of type 'MobileRTCMeetingStartParam4WithoutLoginUser *'; did you mean 'userType'?

Replace 'userToken' with 'userType'

Replacing  'userToken' with 'userType' would give more errors

Note: It was working before the last commit in "zoom-us-ios-sdk-pod"
React Native version:
System:
OS: macOS 10.15.3
CPU: (12) x64 Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
Memory: 287.49 MB / 16.00 GB
Shell: 5.7.1 - /bin/zsh
Binaries:
Node: 13.11.0 - /usr/local/bin/node
Yarn: 1.22.4 - /usr/local/bin/yarn
npm: 6.13.7 - /usr/local/bin/npm
Watchman: 4.9.0 - /usr/local/bin/watchman
SDKs:
iOS SDK:
Platforms: iOS 13.5, DriverKit 19.0, macOS 10.15, tvOS 13.4, watchOS 6.2
Android SDK:
API Levels: 27, 28, 29
Build Tools: 28.0.3, 29.0.0, 29.0.1, 29.0.3, 30.0.0
System Images: android-27 | Google Play Intel x86 Atom, android-29 | Google APIs Intel x86 Atom, android-29 | Google Play Intel x86 Atom
IDEs:
Android Studio: 4.0 AI-193.69<IP_ADDRESS>14223
Xcode: 11.5/11E608c - /usr/bin/xcodebuild
npmPackages:
@react-native-community/cli: ^4.10.0 => 4.10.1
react: 16.9.0 => 16.9.0
react-native: 0.61.4 => 0.61.4
npmGlobalPackages:
react-native-cli: 2.0.1
Steps To Reproduce
Provide a detailed list of steps that reproduce the issue.

Install pod 'ZoomSDK', :git => 'https://github.com/mokriya-org/zoom-us-ios-sdk-pod.git'
run in iOS

@NikhilCatalyst there was some breaking changes on the Zoom SDK 5.0. Just merged an update to remove userToken from the code. Waiting for npm update to occur.


Serialization and implementation support: System.Text.Json and Json.NET (Newtonsoft) in particular

The core model from https://github.com/decentralized-identity/did-common-dotnet/pull/4 can likely be made agnostic of the current JSON (de)serializer, System.Text.Json (STJ) with low amount of work (removing attributes).
I think first-hand for support for STJ is needed as it is the framework way and its design criteria of security and performance seem to align with this .NET DID (thinking security specifically).

Do we have a known case for using Json.NET where STJ is not supported?
If yes on 1., do we want to provide support for it upfront? Other option currently seem to be that we can maintain that with reasonable effort support can be provded either with slight refactoring on STJ implementation one (Extension Data comes into mind mostly) and/or remove all references to STJ in the core model in any case.

**Edit: I think we need to be (de)serialization agnostic in the sense that CBOR is mentioned in the specification. Even if not, I can easily imagine that sending JSON to IoT devices is not a good idea (part of my day job, even very large installations) and so one would want to (de)serialize the model in some other way. Then if one runs .NET on IoT device, and it may become a case, maybe not referencing STJ in the code makes it easier (I'm not sure about AOT etc.).
It's one case then if a parity, say, with CBOR ought to be provided out-of-the-box.

Ok moving my frustration from #4 to here https://github.com/decentralized-identity/did-common-dotnet/issues/5#issuecomment-783080103
"I think first-hand for support for STJ is needed as it is the framework way and its design criteria of security and performance seem to align with this."
STJ is new and Newtonsoft is the most widely adopted,

Both frameworks brings their own pros and cons, Newtonsoft most compatible with a richer feature set and STJ newer faster but with issues in compatibility.
For end users that will require either option as it stands currently the idea is to have the common layer. Now in an ideal scenario the common layer will use the System.Runtime DataMembers to allow to be fully agnostic of the Json deserialisation implemenation but  as it stands this will not be supported after .net 6 (currently we are in .net 5), so that might be in 2 years time.
If the STJ convertors can remove the dependency of the attributes (which obviously is a much harder work) then is a start, then in 2 years time these extensions can be removed at the time.
Other thought was to have interfaces of the model, and each serialiser implement their own model, this is a truly smelly patten, so it is not ideal.

"2. If yes on 1., do we want to provide support for it upfront? Other option currently seem to be that we can maintain that with reasonable effort support can be provded either with slight refactoring on STJ implementation one (Extension Data comes into mind mostly) and/or remove all references to STJ in the core model in any case."
All the references of STJ in the Common model do not exist anymore, these have been replaced with DataMembers. The lack of support of DataMembers is what is causing a complete failure of deserialization.
Going back to the question, do we want STJ now or can it wait for later?
I truly don't want to bring a full dependency to STJ, as it will force a nuget that is not required and wasm / webgl / aot users don't want to download that extra package if needed,

@juanfranblanco I concur with with a runtime dependency on STJ, it appears I was editing my original post when you replied. To me the number downloads of Json.NET and unquantified compatibility are not a case, though.
So one question now is that do we just remove the attributes and continue working without bringing JSON.NET up to parity with the idea it can be if needed? I am not personally keen to support JSON.NET unless I'm aware of a concrete case where it is needed.
I record here about STJ more explicitly:

The attributes are used to turn .NET PascalCase property names to JSON camelCase by providing a handler PropertyNamingPolicy and removing the attributes.

I am not sure about ExtensionData, but I think it can be done otherwise. That is more like a "catch all bag" that makes it possible to accept anything by way of an extension point unless on wants to inherit from the core model.
I have not checked why everything fails as the PR got reorgnized and merged and I have not yet pulled it. Maybe that mentioned helps us/someone to sort it out.

Ok to start with what I have done is to add a new class
 public class JsonCaseNamingPolicy : JsonNamingPolicy
    {
#pragma warning disable CS8764 // Nullability of return type doesn't match overridden member (possibly because of nullability attributes).
        public override string? ConvertName(string name)
#pragma warning restore CS8764 // Nullability of return type doesn't match overridden member (possibly because of nullability attributes).
        {
            if (name.StartsWith("@") && name.Length > 1)
            {
                return "@" + name.Substring(1, 1).ToLower() + name.Substring(2);
            }

            return name.Substring(0, 1).ToLower() + name.Substring(1);
        }
    }

this is included in specific converters like

and it should part of the generic service when deserialising as per

As all this project is WIP ill check in now so you can have a look.
Failing areas are:

[ ]  Extended Data
[ ] Object deserialisation

as an example AdditionalData in Service
and Context in a DidDocument

@juanfranblanco I need to be elsewhere the whole day next (day job etc.), but the policy already exist. Example of its use: https://stackoverflow.com/a/58476761.

@veikkoeeva  but that is support "@"? also that is not a biggy can be removed and replace, also make the code better anyway.. what it will require is to be included on custom convertors and on the future generic service. All wip.. make everything work and then refactor.

@veikkoeeva I am starting to think that the simplest way to support different json frameworks is to provide a common interface in Common, the each model implementation can provide its own attributes etc.
This way end users which are the ones that will be extending the dids with documents etc (think ceramic) can extend with their framework of choice, and if they want to support both they can do it at their own will, leveraging the best features of both.
For example ExtendedData, Subtypes etc can be implemented leveraging each framework and a end user can use them or extend them in their projects without a worry.

Futher thought on that.. the STJ implementation will just fit in as there won't be much work todo, now.. the issue will move to the newtonsonf implementation with the Context and the LD, i guess other stuff can fit in pretty easily
Mainly we can have an ServiceInterface that provides the writing and decoding using again IDidDocument as an input etc. Truly will remove the need of any fights with the frameworks, and eventually let nature decide which is best :) And we can concentrate on the crypto which is truly more fun than this. (<-- Defeated, too many hours.. )

Is there a case where STJ is not available for JSON operations and Newtonsoft is? Are there reasons to still prefer Newtonsoft over STJ? As note earlier, historical download numbers do seem to be provide a benefit in this codebase, maybe there are other reasons.
In passing: There probably other ways to have DTOs that can be (de)serialized with these frameworks than interfaces if that's what you propose.
Before any of this planning I would like to get back to a solution that compiles and test pass using STJ. I think if there is a need to support Newtonsoft we have a number of options for refactoring. Maybe just removing the attributes which I had been planning to take off in any case.

Yes I have been trying to make the test pass after removing the attributes, (STJ) pain point there is the extending data at the moment and the custom objects.  To cheat i tried to inject the attribute JsonDataExtented (stupid idea) and utter waste of time as it will be not in anyway good for a framework.
Overall i am glad the Context works, not in Service the extended data needs to be ignored

@veikkoeeva  One of the most compelling use cases to keep it simple for end users is something like Ceramic https://ceramic.network/, DIDs are an element in both the data structure, schema and of course identity.
Hence the emphasis on compatibility on something which is more widely used and supported that STJ.
If you imagine the journey of a user who is storing data in IPFS or anywhere else including DIDs as one of the elements, they might want to introduce these DID or multiple in a big structure that will carry much more information.
The user might have data elements already decorated with attributes from Json.Net, STJ or other, hence my total frustration on incompatibility.
If DIDs were isolated the only issue will be the framework (adding an extra assembly) but they are not. Maybe this is a bigger case to have those interfaces, or maybe not. What do you think?

@juanfranblanco The examples you linked seem to be Jest snap files, not plain DID documents (i.e. they are scaffolded with Jest snap specific structures). As far as I understand by just looking at them, if they were DID JSON documents, I would assume they can be (de)serialized fine.
I try to think of a scenario where user is using .NET Core or .NET5/6 and do not have STJ available. Currently I can think of is that there are DTOs like on those tests in services and the user of the library point-blank refuses to take them as strings and serialize using Newtonsoft and refuses to write a converter and well, just refuses to use STJ.
If there were other cases, I would still think that it would be easier to note it's possible later if someone comes along and has a case. Maybe they do a PR (since I think it's doable) and this can be solved then and not now.

Of course these are just examples, if you look how ceramic works or 3box a Did will be part of the composition. What I mean by composition is that a Did document will part of a bigger schema,
I think this is a better example https://hack.ethglobal.co/showcase/eridanos-recJwuShuxEibN3y7
So yes a user might use STJ or Newtonsoft, but when they have a document including multiple DIDs and verifiable credentials, and extra data (ie tiles using the ceramic example with their own data) they might require all this to be compatible with the framework of choice.

Regarding scenarios that people may not have STJ, yes i can think of some like blazor, unity, ios etc, also can work in the other way. Hence I am leaning now more towards interfaces to define the schema, as this will allow any user to extend or use the DIDs with their framework of choice without any limitations.

@juanfranblanco Out of curiosity, do you have examples of such JSON files (without scaffolding)?
So the discussion then would be that someone takes DID DTOs from this library and embeds them into other DTOs of their own and want that Newtonsoft works out of box (*) since they have a want or need to use Newtonsoft instead of STJ? And we are now building a case if we should or should not support such a case right from the start? Would it be sufficient to instruct to usea Newtonsoft converter in that case (I think it should work).
(*) Not just writing a Newtonsoft converter that use this library to (de)serialize the objects but that it needs to work without writing a Newtonsoft converter to do that.

@juanfranblanco I need to be off from the computer for a while now. So as not to get into a marathon thread where you come up cases and I every time tell that would a converter do, I think one way forwards would be to get a solution that runs the tests.
Then we remove the attributes from the DTOs so that the tests still pass. So then building a solution using Newtonsoft looks could be that someone creates converters and the model does not need interfaces, inheritance or anything to support multiple JSON (de)serialization frameworks. Would that work?
So basically I'm still postponing this uprfront work for better design and not pushing it into to this merged spike with the feeling it creates more wor and delays. When someone comes along and says that they need this library to work on Unity (I think Blazor has STJ, I don't know about if iOS has something limited there), we know it's possible to do.

Most of the stuff is framework based, ie mainly all written to ipfs, but as you have seen in the demo you end up composing many objects (i will need to spin up my ceramic stuff. etc) but ceramic works in tiles so the schema is extendable.
So if you are an end user, what will you expect to do? You will not want to create a convertor as that is not the normal scenario, you will want to, start creating your schema, add verification and your other custom stuff, a few dids which may have specific signatures for validation (full doc, not just a resolver link), and finally you might want to serialise all this, create a hash and sign it.
In this scenario you will pick your preferred framework, create your types etc and decorate the attributes and serialize. Obviously you won't do a combination of frameworks as that won't make sense for sure.
Also the user might want to extend the document (DID) to include extra data (typed data) including that information.
In a super ideal scenario we should have been able to use DataMembers and use this as a common attribute for both frameworks, but regardless there is  also a limitation as we don't have those JsonDataExtended attributes in system.runtime, hence i was trying to inject the attribute as a hacky way too.

@veikkoeeva I  personally think that it will be the best to have interfaces, is the only solution to share a common model across the two implementation, i personally would use newtonsoft over stj for the time being as it is supported in unity in older versions, blazor and ios (aot).
From a perspective of the STJ implementation will make it much easier as it will allow to move the model classes directly to the STJ. Of course we still need to have converters for the Context.

It looks like this to me now:
I think STJ is available in Blazor and Xamarin (iOS) if I recall correctly, maybe not be right now in Unity. STJ is in the framework the recommended way forwards. How many use Unity and need this library right now and do we need to add support for it right now? There was/is a working solution with STJ.
The driving scenario is that that someone has written DTOs and they cannot/do not want to use STJ and do not want to write a converter in Newtonsoft but want/need to use attributes. This is presumably for cases that have started using DIDs and VCs in .NET already now and these scenarios need to be supported. There maybe also something else that allow using STJ without converters if I understand you correctly. I do not know how what's discussed here makes STJ "much easier". I may have misunderstood this scenario.
Then here in this thread of the design space available it is decided DTOs will use interfaces that have attributes, if I understand correctly. We do this design and work upfront right now, or maybe you do and submit the code.
In any case I would like to see an an example of such documents that emded DIDs and VCs. I would like to see how DID DTOs with STJ look like and become easier. I'm not very keen to assume maintenance burden without a sample document and a clearer issue with this thread only, it would greatly ease my thinking to have concrete DTOs that embed these DIDs. I think they would help tests. Even then I might want to consider someting else than interfaces to DTOs if that's the case, at least consider a bit. Even before than I would like to resume to a working solution and refactor from there, not be in limbo and come up with something that makes some tests pass in the future.
I do not see a proposal that is "easier" so it's really difficult have an opinion on that.

My problem is that we are not agnostic to a framework, so I am not saying that STJ is going to be great but also I am not saying that Newtonsoft is going to die. It may have its counted days, but it won't be in the near future.
Regarding document examples, i can ask for some.. but truly the most simple thing is that things are going to be composed.
STJ vs Newtonsoft both do different things, it is not up to us to discuss this, but to provide a simple way for anyone to do whatever and don't force them in a particular way.
I am creating an example of interfaces, and we can see how bad / good it is.
I personally I am liking it as interfaces allow me to extend objects without needing to worry about the overall whole schema.
This is similar to your FormatKey implementation using an abstract class.

All right, we see what comes.
This how I understand the scenario: It looks to me you want to explicitly support a scenario where Newtonsoft is used with attributes, not just removing STJ attributes from the DTOs, make the solution compile, tests pass and move forwards and solve the Newtonsoft issue later if a need merges (maybe you have a need somewhere, see following). Not only that, you want to support both STJ and Newtonsot with attributes, also in environment where the users have both available and has a strong preference to use in all cases either STJ or Newtonsoft via attributes, or Newtonsoft in this case since STJ case exist already.

Yes that's about it :) !!

@veikkoeeva  check the current implementation using interfaces https://github.com/decentralized-identity/did-common-dotnet/tree/spike-interfaces there are some issues on the round trips of full documents and some IService but.. you can see the overall idea. Once the interfaces are sorted it is surely much easier than fixing with no attributes, I have restored the old attributes too.

The only argument I can make for Newtonsoft is the dependencies on other libraries, specifically, those supporting JSON-LD processing.
Currently, dotnetrdf and json-ld.net both use Newtonsoft.
If the intention of this library is to allow upstream implementation to utilize the data model, while being able to use the JSON-LD processing features, it might be better to support both.

@veikkoeeva https://github.com/decentralized-identity/did-common-dotnet/tree/spike-interfaces  migration to interfaces is done now, tests are passing STJ only no Newtonsoft.. check the structure for thoughts etc. I had to changes to support interfaces but mostly was already handled by the convertors..

Thanks @tmarkovski yes that is the thought, although it is proving a small headache, check that interface spike if you have any thougths or feedback, the only issue here is the duplication of the models, but there is no much way around it.  Ok bed time.. :)

@tmarkovski I think both Dotnetrdf and json-ld.net support POCOs. This is  because they are built on Newtonsoft that supports POCOs at its core case. If this is the case, I believe we have this covered in the future. It looks like this was a general observation from you and with this written, I believe it is an unblocked future scenario.
Referring to other isuses and this thread with @juanfranblanco, if you have a specific scenario in mind that needs to supported right now, I would like to see a more concrete use case we can also test to see it is covered and remains covered. It can also be just an example of something so that we can now think if it's unblocked even if not acting on it right now.
@juanfranblanco I thought the thought was

[Veikko, referring to other discussion] This how I understand the scenario: It looks to me you want to explicitly support a scenario where Newtonsoft is used with attributes, not just removing STJ attributes from the DTOs, make the solution compile, tests pass ...

[Juan, answering] Yes that's about it :) !!

I think it would be simpler to make them DTOs POCOs. Remove attributes. Then we would have one set of DTOs as POCOs and two sets of converters. DTOs as POCOs without interfaces.
Even using Newtonsoft I could very well imagine that if you have a case where a parent element or an element preceding in element order is telling the next type to read, as DID specifies, a converter would be written. Doing this with attributes is not a straightforward case.
Assuming someone just refuses to do this and insists on attributes, one another way without interfaces could be
public class BasePoco
    {
        public string SomeField { get; set; }
    }

    public class NewtonsoftPoco: BasePoco
    {
        [SomeNewtonsoftAttribute]
        new public string SomeField { get; set; }
    }

(I just wrote that on top of my head)
which would still allow attributes. In this case, like in this currently being built, we could reasonably ask if it would be a case for this library to support and distribute or could we reasonably instruct those potential users who insist using Newtonsoft attributes only to do this themselves. I can also think that if we want to maintain this as a core case and we insist on not making use of converts but attributes only, it can be that beyond just adding interfaces and two sets of POCOs, we need to start alterting the types and processing in other ways.
I think using converters here is reasonable. I think we can reasonably expect users write converters as part of their other developer flows where they embed POCOs from this library. When they do not embed DTOs, they maybe do not need to see converters being used.
I of course would like to build a scenario why would we now insist not using DTO POCO and remove the STJ attributes. Does it block us from some use cases we can think of? I do not think it blocks cases where one uses Newtonsoft or libraries based on those. Using POCOs likely warrants using converters by way of how DID specification is written, I think.
Moreover in general, why would we not write a more thoughtful issue about this and cartwheel this through in a thread where goal posts are moving and I try to dig up something (see the quotes)? Why not write an issue that entertains the scenario and some alternative designs? Why insist it goes goes into a spikes folder and not be a PR towards a working base like I could imagine this could be done? Do we in the future say we support every way of implementing things by way of it's not up to us pick supported scenarios?
This way of working feels a lot of work to me. I do not really any more time to put into this this week and maybe very little next week. So one way of moving towards is that I just note it's all fine to me and I come back in a week or two to see what has happened with all the code moving around and use case scenarios implemented.

@juanfranblanco If all you was add interfaces and the code compiles, I think the tests should pass and things work. See that of some notes on what could be an alternative design. It depends on how pathological the scenario is, of course.

@veikkoeeva Mainly this is what I have been doing, finding the best solution that allows any framework to work, have the lesser impact in your previous work but also maintaining for an end user all the capabilities of their chosen framework. Also I know that your busy so I want to put the less stress on you by providing you constant feedback on what is happening. But this is a great experience to get a good understanding of how each other see this project going in the future and obviously the pros and cons.
I am trying to get your best feedback as my ideas might be completely wrong as I am trying to experiment the different alternatives, without imposing any. Also I want to get the architecture out of the way so if we are happy with the overall structure we can convincedly say why everything is setup this way, currently and what i may look like in some years.
So after structuring the code. In one side I have been trying to eliminate the need of using attributes or use common attributes which provides a common model for both (ie not interfaces), similar to your current thoughts, and as we have seen the other alternative is to use interfaces. In both areas I tried make working solutions as fast as I can to validate pros and cons.
As you have seen we have found plenty issues, mainly STJ does not support yet a common set of attributes (DataMember, DataContract) sadly as we have seen per the issues that decided against that due to the lack of features they have implemented, mainly not able to provide ordering etc etc.   We know that possible this is coming after .Net 6 so in a couple of years time, this can be achieved.
This was pretty sad, and truly if it was a project that is not going to be extended by an end user I would not mind (mainly use POCOs) and come up with a complex way to handle AdditionalData. Talking of AdditionalData NewtonSoft handling of additional and attribute is all done internally in private classes https://github.com/JamesNK/Newtonsoft.Json/blob/666d9760719e5ec5b2a50046f7dbd6a1267c01c6/Src/Newtonsoft.Json/Serialization/DefaultContractResolver.cs#L443-L579, yet again this is pretty sad as an end user may want to change how AdditionalData is used as they extend the classes, etc using attributes. So mainly I inspected all the code and both frameworks and there was not an error prone way to enable end users to use an alternative. Both recommend the use is to use their attributes.
In a a truly ideal scenario, and something to push to the OSS community, both and any framework would share a common set of contracts / attributes etc as we had seen in the past what happened with Dependency Injection. Of course this not limited to this, but I was hoping to provide the users with the best experience they need, in a similar way that Javascript users when they deal with Json for them it just works.
So I pondered onto the other solution which was to have a common set of interfaces, I personally was not keen as I wanted to have a common model for both, this is your POCO, but so far seems to be the more flexible way to define the contract of how the model should work and leaving each framework, or future framework as we might see other ways to serialize and deserialise the data.
Obviously having interfaces brings their own set of issues, mainly the model needs to be duplicated in the implementation project, but this way any end user can use their preferred framework and extended the best way they like it.
Another solution could be to simplify the models and convertors, if this is even possible, so any end can easily modify or extend the convertors, simplifying inheritance selection of verification relationships, formats, the context etc. But I truly have not wanted to disrupt your initial code as you have put plenty thought on it (it can be seen by working on it :) )
Regarding the way of working I know that you have been busy, hence I created the two solutions highlighting the issues, one in the current main area, and the other in https://github.com/decentralized-identity/did-common-dotnet/tree/spike-interfaces. So all the tests pass there.
Once we pass this hurdle, hence I wanted to get it out of the way, things will be ideally super simple as we will be able to concentrate in specific features, and do the usual stuff. This is were we are with lots of ideas and merging them into a common architecture :) So please have a look at both areas, it takes truly 5 minutes to see what is happening in each side and we can take it from there. I personally going to start to create the deserialisation of Newtonsoft now that STJ is working so we can take it from there in this common model.
Overall my principles are, making it easy for a user to work, extend and don't break any solution / framework or way of working and seeing how DIDs are being used across different languages (which are years light ahead of .Net) how this can help to achieving at least something that is the same and compatible with them.
Truly this is pretty hard I know, and gives you an understanding of why MS (as an example with some Azure projects) decides in some areas to keep things whilst open source in their product projects, as it is much easier to focus in a product than to please everyone with a framework.

Now the POCO solution using inheritance, is actually taking us to the interfaces
public class BasePoco
    {
        public string SomeField { get; set; }
    }

    public class NewtonsoftPoco: BasePoco
    {
        [SomeNewtonsoftAttribute]
        new public string SomeField { get; set; }
    }

so if you look at here:
https://github.com/decentralized-identity/did-common-dotnet/blob/spike-interfaces/src/DidNet.Common/IService.cs
using System;

namespace DidNet.Common
{
    public interface IService: IAdditionalData
    {
        Uri? Id { get; set; }
        string? Type { get; set; }
        string? ServiceEndpoint { get; set; }
    }
}

if ends up in STJ like this.. https://github.com/decentralized-identity/did-common-dotnet/blob/spike-interfaces/src/DidNet.Json.SystemText/Service.cs#L10-L29
/// <summary>
    /// https://www.w3.org/TR/did-core/#service-endpoints
    /// </summary>
    [DebuggerDisplay("Service(Id = {Id})")]
    public class Service : IService
    {
        [JsonPropertyName("id")]
        public Uri? Id { get; set; }

        [JsonPropertyName("type")]
        public string? Type { get; set; }

        [JsonPropertyName("serviceEndpoint")]
        public string? ServiceEndpoint { get; set; }

        [JsonExtensionData]
        public IDictionary<string, object>? AdditionalData { get; set; }
    }

leaving the original model intact and not having to think of using in STJ the yet to be supported DataMember
Now we need to discuss what the namespaces will look like if we go with this approach.

So after structuring the code. In one side I have been trying to eliminate the need of using attributes or use common attributes which provides a common model for both (ie not interfaces), similar to your current thoughts, and as we have seen the other alternative is to use interfaces. In both areas I tried make working solutions as fast as I can to validate pros and cons.

STJ, Newtonsoft, some CBOR (e.g. System.Formats.Cbor) or whatever random (de)serializer one chooses support POCOs without attributes if that is the goal. This is how I think currently. In addition to this, you seem to have a goal to support attributes only Newtonsoft type hierarchy in addition to POCOs, if understand correctly. I think this case can be realized attributes only Newtonsoft types inheriting from POCOs. STJ cannot be made to work without converters in any complex scenario, it is how it has been designed. I keep repeating this, I see.
POCOs can be made by removing the attributes that currently do:

Rename attributes ('@' and PascalCase) -> converter.
Extension data -> converter.

Do we not know how to remove these attributes? I was thinking to do that but the code was merged (and reorganized) before that.
Do we want to avoid converters at any cost with Newtonsoft support? Why? Do we intend to uphold this decision in the future?
If we support attributes only Newtonsoft processing without converters in the future, is it necessary this decision affects all other types and processing also? See the following where I go back to the earlier point I keep repeating.
STJ certainly cannot be made to function without converters in any sceneario that is more complicated, it is how it has been designed. DID specification, and later VCs, presentation exchange and so on seem to be such complex scenarios. STJ can support POCOs without attributes, so can Newtonsoft as far as I know.

Now the POCO solution using inheritance, is actually taking us to the interfaces

While I did not think it much, it looks to me that particular solution would not force interfaces to all type hierarchies. In this case  attributes only Newtonsoft that will not use converters will inherit from POCOs. It looks simpler to me if the intention is support this particular scenario, I'm inclided to think metrics would support this. Does it work?
By inheriting the attribute only Newtonsoft that will not use converters types from POCOs seem to be another alternative that is less intrusive. Does it work? If it does, can that be used? If not, do we need to proceed with this idea and with what looks to me a self-imposed constraint of attributes only Newtonsoft without converters.
Do I misunderstand something?
Is is not only that, other future processing that need to happen via interfaces and not via data types as far as these parts are concerned. It looks to me this decision of attributes only Newtonsoft without converters constructed using interfaces and two type hierarchies with attributes has implication to whatever other processing and design there will be.
If we take it upon ourselves to impose a constraint that all processing in the future must be possible in Newtonsoft using attributes only since we do not want to use converters, it is intrusive on other parts of design. Is this scenario so important? I think this not only for this case. Since maybe this one can be solved if attributes only Newtosoft without using converters by inheriting from POCOs that others can use without interfaces and other potential constraints.
This future processing I see is VCs and presentation exchanges where I think STJ also needs converters and I had thought about going with about the same solution. But it might be more complex because of attributes only Newtonsoft that will not use converters implemented by interfaces and two different type interfaces that is on the design here.
I believe the mentioned upstream libraries that use Newtonsoft in their core allow using converters. Is this not the case? Even if that is not the case, see the previous of making DIDs, VCs and presentation exchange to use attributes only class hierarchies without converters.

Once we pass this hurdle, hence I wanted to get it out of the way, things will be ideally super simple as we will be able to concentrate in specific features, and do the usual stuff.

It certainly looks like you want to make this change and others. I keep repeating that why not POCOs if it support the case you wrote and I quoted. I keep repeating it because you explain to me these type hierarchies are needed to support STJ and Newtonsoft. It looks like wanting to support attributes only Newtonsoft specifically – so that converters would not ever be needed. I am not sure if that can remain the case in the future. The tradeoffs you propose over POCOs to support attributes only Newtonsoft without converters do not seem appealing to me if we can supports attributes only Newtonsoft without converters so that attributes only Newtonsoft without converters types inherit from POCOs.

Thanks @veikkoeeva  we are truly in the same page. We just need to validate the best approach for end users.

STJ, Newtonsoft, some CBOR (e.g. System.Formats.Cbor) or whatever random (de)serializer one chooses support POCOs without attributes if that is the goal.

Exactly hence an interface might fit better to match the structure
public interface IService: IAdditionalData
    {
        Uri? Id { get; set; }
        string? Type { get; set; }
        string? ServiceEndpoint { get; set; }
    }

Although having the capability to add DataMembers to describe the real name in the contracts is pretty important, as we know interop with other systems is necessary so it is easy to understand the json etc structure.
I removed the attritutes on the main branch, and replaced them with DataMember attributes, also here is where the limitations surfaced (thinking as an end user and end consumer)
BTW I am not saying that convertors are not going to be needed, I am actually now creating a convertor now in Newtonsoft for the VeficationMethod trying to be similar to the STJ implementation, so it is easier to maintain (and of of course as the structure truly dictates so) but what I want is for users to be able to pick and chose without any constraints if they want to use attributes and extend as they wish.
I think for an end user having examples using attributes so they can understand their inner and outs and can extend might be key, but it might be that is the opposite in some scenarios, or impossible due to inheritance / strategies, etc.
In the previous post I am highlighting the different pros and cons of having either POCOs, interfaces and a combination of both. We can continue to add other examples to see what will be best, i am personally working on the interface branch as all the tests are passing there.
Just to reinforce I am in no way committed to anything, but my principle is that users should not need to suffer if it is too complex they will end up using Javascript instead.

@juanfranblanco Why not having a plain POCO Service? What benefit does interface bring? Now it needs to be implemeted by a POCO.

I feel sorry also I need to leave this discussion like this for a while. It may be I can get back next week. I'm mostly on the move and need to accomplish other work also.

Sure don't worry I understand :) regarding the POCO vs interfaces this is what I was highlighting in the previous post, I think we can take this offline when you have more time. But overall interfaces can dictate the contract structure and mapping without forcing you to a given inheritance, attribute usage etc that the POCO dictates. And yes I truly hate having to copy the model into another implementation, hence my frustration as it most scenarios this might have been solved by having these common attributes.
You actually have done something similar to an interface, by having the Format abstract class with no internal implementation.
Like I said in the other post another alternative might be to have iboth interfaces and POCOs model in Common, so this way we can achieve the best common ground, allowing both for extensibility and or fully replacement if wanted by implementing the interface.

@veikkoeeva when you have a second check the https://github.com/decentralized-identity/did-common-dotnet/tree/spike-interfaces/src/DidNet.Json.Newtonsoft/Converters and overall port to Newtonsoft, I have tried to keep it the same as possible as the STJ so it is easier to maintain (and obviously create ;) ).
I think this will be great point of reference as we can see the pain points but also the duplication as we see with the POCOs and hopefully achieve something that can work for all. If there is any consolation I have just seen that JSON LD are also going to port and support both frameworks https://github.com/linked-data-dotnet/json-ld.net/issues/59#issuecomment-653392194 in the future, so mainly everyone is in the same boat.
BTW lots of great thoughts in the STJ code base, great to work with that.

@veikkoeeva check the latest https://github.com/decentralized-identity/did-common-dotnet/tree/spike-interfaces/src/DidNet.Json.Newtonsoft/Converters  all have been ported. I am going to try to move the model if both scenarios (STJ and Newtonsoft back to common but keeping the interfaces to enable more extensibility, then simplify abstract the convertors creation in newtonsoft as there is no factories.. hopefully something that can be extendable) Lets see how that looks and can be applied to STJ or try something different.

@veikkoeeva  Further update, if you check the latest commit https://github.com/decentralized-identity/did-common-dotnet/commit/1cebb6f087c95439d2e347816b12e584b0d85515. This combines back the models, keeps the interfaces and introduces extending models. All the models are decorated with DataMember and DataContract attributes where necessary.
Overall these are the findings, which provide good justification of structure.

Obviously duplicating the whole model makes no sense in different implementations

All extension objects are suffixed with "Ext" and in their own namespace as per @veikkoeeva  suggestion.

All the model objects properties are virtual so they can be overriden to extend with attributes. For example includes Non Nullable attributes as these were not ignored by the serializer

  [DebuggerDisplay("PublicKeyJwk(Crv = {Crv}, Kid = {Kid}, Kty = {Kty}, X = {X}, Y = {Y})")]
    [DataContract]
    public class PublicKeyJwkExt : PublicKeyJwk
    {
        [DataMember(Name ="kid")]
        [JsonProperty("kid", NullValueHandling = NullValueHandling.Ignore)]
        public override string? Kid { get; set; }

        [DataMember(Name ="y")]
        [JsonProperty("y", NullValueHandling = NullValueHandling.Ignore)]
        public override string? Y { get; set; }
    }

but also having the capability to add ExtendedData attribute
    [DebuggerDisplay("Service(Id = {Id})")]
    [DataContract]
    public class ServiceExt : Service
    {
        [JsonExtensionData]
        public override IDictionary<string, object>? AdditionalData { get; set; }
    }

All the model objects extend interfaces to allow a full new implementation, as an example the STJ DidDocument when extended was introducing a new "context" plus a "@context" duplicating the element. The only workaround at the moment is to create a full class that replicates the original.

This way a user remains full in control of the objects without adding dependencies to the model.
Example of issue:
 "@context": "https://www.w3.org/ns/did/v1",
    "authentication": [
        "did:example:123456789abcdefghi#keys-1",
        "did:example:123456789abcdefghi#keys-3",
        {
            "controller": "did:example:123456789abcdefghi",
            "id": "did:example:123456789abcdefghi#keys-2",
            "publicKeyBase58": "H3C2AVvLMv6gmMNam3uVAjZpfkcJCwDwnZn6z3wXmqPV",
            "type": "Ed25519VerificationKey2018"
        }
    ],
    "context": "https://www.w3.org/ns/did/v1",

Next it will be to experiment with a simpler way to instantiate and configure the opptions, convertors etc. From there anything is up to discussion, but we have a common ground to support both frameworks and to demonstrate benefits / issues from each framework. Nevertheless we can see how STJ is heavily based on Newtonsoft anyway. But as with .Net and .Netcore etc and don't see this going away in many many years.

Sure don't worry I understand :) regarding the POCO vs interfaces this is what I was highlighting in the previous post, I think we can take this offline when you have more time.

Online is fine.

Further update, if you check the latest commit 1cebb6f.

I'm not sure what to look at. Instead I'll put some effort on the (massive) PRs that are open.

Overall these are the findings, which provide good justification of structure.
Obviously duplicating the whole model makes no sense in different implementations
All extension objects are suffixed with "Ext" and in their own namespace as per @veikkoeeva suggestion.
All the model objects properties are virtual so they can be overriden to extend with attributes. For example includes NullValueHandling attributes as these were not ignored by the serializer

I think null values were handled with converters in STJ already and could be with Newtonsoft too. I don't remember proposing suffixes. I agree duplication is not needed, POCOs suffice on the core.
More broadly:
If the concern was to get Newtonsoft onboard, I think it could have been done by adding Newtonsoft converters and tests that excercise them (that too could have been a separate PR). Maybe add cases where specifically Newtonsoft could be used instead of STJ, e.g. with some particular library and see the code functionality enhanced with Newtonsoft (converters, and attribute-added types?) realize the case and as a way of example.
To reflect specifically to the topics here, soon after the work-in-progress merge of the branch and consequent reorganization and while this discussion was running, I updated the version with the changes I had mostly done to remove STJ attributes, added more tests and fixed some tests.
Now this codebase is quite different and I think here the idea is to do something different even if broadly the same approach is used. Which leads me to following point:
I do not have capacity to work like on spikes directory to pursue 'correct architecture' and build support on everything upfront.
I have a functional codebase (currently on my personal and company account). At the moment it is more important for my other goals to have a working version I can improve. If this codebase here is broadly based on same basic ideas, there may be possibility for cross-breeding code and I can try to keep up and help here if that is desirable.
Certainly I had already a plan to be able to make this useable with other ideas, such as DIDs (and VCs) inherting from Newtonsoft or STJ types by way of making them "hierarchical key-value bags" or even supporting a case of JSON to-the-whitespace that is signed (which likely means one should look into some proper signing scheme). I also try to look this so it is useable in more regulated environments, like payments systems.
Currently I have small refactorings in mind, in the order of adding Newtonsoft converters-only (or maybe CBOR), to refactor types and cryptograhic parts and tests. While I note this, I add that likely I work first on VCs and presentation exchange and so maybe there's a chance to cross-breed ideas here in the future.

@veikkoeeva I kept it as much as possible to match your code grab the pr and have a look, so you don't have to worry and can be built upon there. There are Newtonsoft convertors etc. I can give you an overview offline, it is not as different as it may sound in the first place. Check the latest or, it can be merged and work from there. I understand having proprietary code etc, so good to be aligned.

@veikkoeeva  i created this pull request, you will see that not much have changed but now it enables the architecture to include newtonsoft, but most important now that this is enabled we can validate each one of the models. https://github.com/decentralized-identity/did-common-dotnet/pull/10

@juanfranblanco I added a review. I apologize for the delay.


Add stop experiments queue processing immediately command

2/? main <- #3154 <- this
This PR implements the command alluded to here. This command not only stops all experiments queue workers but kills currently executing queued experiments. I've added this command so that it can be wired into the stop button.
Docs for the stop command are here.
Demo
https://user-images.githubusercontent.com/37993418/214470456-5f966ea2-dfc6-400e-8729-237c5ef442a9.mov

Dumping this in favour of adding the exact functionality to dvc.stopRunningExperiments.


Allow form-reducer to handle model at deep state path

With a complicated app, it'd be nice to be able to organize the state.
eg state = {forms: {user: {}, userForm: {}}}
Does this seem reasonable?
Thanks for this awesome library!

Yep, LGTM! Makes sense, since the model reducers can handle deep state paths as well.
Thanks for tackling this!


Setting up Push Notfication

Hi
I'm using Checkmk version 2.2.0. I copied the mail notification rule and i choosed "Push Notification (using Telegram)". I see only one field for entering the parameters of the bot. Your screenshot shows three input boxes.
Any idea what is going on?
Thanks!
Martin

Please have a look at #6

Hi,
That's really odd, nice trap :-) But thanks for the advice. I will try it later on and will give feedback.

It worked out.


query param to hide list, zoom map to state

If we can to use the Geocoding API the zooming would be cleaner. To do so we'd need to look up the state name using an object like this: https://gist.github.com/mshafrir/2646763
We need the mappings because for that API searching for "State of New York" works but "State of NY" doesn't

@nburt - This is great. I'll pull down and do some local testing. Thank you.


Temporal training takes much longer time than single-frame warmup epochs.

I train the 256x704 config with 8 L20 and finds that temporal training (after 6 epochs) takes much longer time than single-frame warmup epochs (before 6 epochs). I attach my training log here. Is that normal for this training speed?
20241002_223820.log

It is normal for the temporal model to be slower because of the need to load and calculate historical information.


Use alias declarations instead of 'typedef'

Description
Pure refactoring change. Mostly automated with clang-tidy, but also some manual work to fix-up some comments and some places where clang-tidy failed.
Tasks

[ ] Tested on Linux
[x] Tested on Windows
[ ] Tested on macOS
[ ] Tested on iOS
[ ] Tested on Android

How to test this PR?
CI. I've also ran some of the examples and everything was fine.

Looks good to me, I assume a type alias can do anything a typedef can (for the given changes), correct?

Yes, it has the same exact semantics. Just alternative syntax.


A11y asst text fixes

Fixes #1234 #1231
Additional description
Small fixes for assistive text in Icon Button and Stateful Button example
Pull Request Review checklist (do not remove)

[ ] Review the appropriate Storybook stories. Open http://localhost:9001/.
[ ] Review tests are passing in the browser. Open http://localhost:8001/.
[ ] Review markup conforms to SLDS by looking at snapshot strings.
[ ] Add year-first date and commit SHA to last-slds-markup-review in package.json and push.
[ ] Request a review of the deployed Heroku app by the Salesforce UX Accessibility Team.
[ ] Add year-first review date, and commit SHA, last-accessibility-review, to package.json and push.
[ ] While the contributor's branch is checked out, run npm run local-update within locally cloned site repo to confirm the site will function correctly at the next release.

This looks like a trailing comma issue. DSR requires trailing commas now.
npm run lint:fix in the future. I'll push a commit.


Some theme resources retrieved using Application.Current.Resources do not update with theme

Describe the bug
Retrieving resources from code using Application.Current.Resources lookup always returns the light theme version of the resource. In addition the value does not change when the application theme changes.
In contrast using a theme resource with XAML binding does return the correct resource value and is responsive to application theme changes.
Affected resources include at least:

TextFillColorPrimaryBrush
SystemFillColorSuccessBrush
SystemFillColorCautionBrush
SystemFillColorCriticalBrush
Probably many others...

Steps to reproduce the bug

Create an app with XAML and code behind:

    <Grid x:Name="RootGrid" Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">
        <StackPanel>
            <Button x:Name="ThemeButton" Content="Toggle Theme" Click="Button_Click"/>
            <TextBlock x:Name="CodeStyledText" Text="This text doesn't change color."/>
            <TextBlock Text="This text does change color." Foreground="{ThemeResource TextFillColorPrimaryBrush}"/>
        </StackPanel>
    </Grid>

        public MainWindow()
        {
            this.InitializeComponent();
            CodeStyledText.Foreground = (Brush)Application.Current.Resources["TextFillColorPrimaryBrush"];
        }

        private void Button_Click(object sender, RoutedEventArgs e)
        {
            if (RootGrid.RequestedTheme == ElementTheme.Dark)
            {
                RootGrid.RequestedTheme = ElementTheme.Light;
            }
            else
            {
                RootGrid.RequestedTheme = ElementTheme.Dark;
            }
        }

Click the theme button and observe the textblock with foreground set by the code behind does not change color.

Expected behavior
Theme resources loaded from C# code should update with the app theme the same way that they do when loaded via XAML.
Screenshots
No response
NuGet package version
WinUI 3 - Windows App SDK 1.1.4
Windows app type

[ ] UWP
[X] Win32

Device form factor
Desktop
Windows version
Windows 10 (21H2): Build 19044
Additional context
No response

Keep alive https://github.com/microsoft/microsoft-ui-xaml/discussions/8638

There does not seem to be an easy way to do ThemeResources in code.. Here is a related thread - https://stackoverflow.com/questions/33356217/setting-or-modifying-themeresource-in-code


Create new message with initial values taken from database

Hi,
I am using Python 3.7.3 and Cantools 36.2.0
I want to create a message taking the initial values from the database. But I don't now how to access the values.
Now I am creating a message with dataIni instead:
dataIni = [0, 0, 0, 0, 0, 0, 0, 0]
msgLWI_01 = dbCH0.get_message_by_name('LWI_01')
msgLWI_01_Simu = dbCH0.decode_message(msgLWI_01.frame_id, dataIni, decode_choices=False)
Can you help me with that? Thank you.

I am not sure if there is a way to apply that automatically. You should inspect the contents of the dbc_specifics object that your db has. You will find the initial value in there and can apply it to the signal value manually.

Well, I got the idea (get signals from DB message, get initial value from every signal, and then loop every signal in newmessage and set value). I think it may work.
However, will be very useful a 'data' propery in DBMessage returning a list of bytes as used to decode.
Thanks for your help juleq!!

To check that I understood your last point:
On a separate note, you propose this enhancement: Keep the raw byte array for the message after decoding for some other purpose later on.
Did I understand that the way you meant it?

Better an example :)
When I use this line:
msgLWI_01 = dbCH0.get_message_by_name('LWI_01')
I got a messageFrom the database. It would be fine to get the a list with the initial values (raw byte array) doing something like:
dataIni = msgLWI_01.data
so I could:
msgLWI_01_Simu = dbCH0.decode_message(msgLWI_01.frame_id, dataIni, decode_choices=False)
or
msgLWI_01_Simu = dbCH0.decode_message(msgLWI_01.frame_id, msgLWI_01.data, decode_choices=False)
Is more clear now?

Ah, ok, you want to encode a message using the initial values where available. Yes, that is a reasonable request. As I said, I do not know if automatically applying of initial values is implemented. If not, you could propose an implementation.
Should be around ten lines of code effectively. Can be done either on db level iterating over all signals in all messages or, better, cascade down like: db.ApplyDefaultValuesWhereAvailable() calls that same method on every message and that on every signal.

Ok, I will open a request.
Thanks a lot for your help!

Well, I don't know how to open a request. I though it was similar to open an issue but it is not.
Anyway, I managed to solve the problem with a small function. I leave it here an example so if someone wants to use it or create a proper function in the library.
Please, I accept suggestions of better code. And thanks again juleq.
#Example:
#Set initial values from given DB Message to given message
def fSetIniValues(msgDB, msg)
for signal in msgDB.signals:
if (signal.initial in not None):
msg[signal.name] = signal.initial
#Usage:
dataIni = [0, 0, 0, 0, 0, 0, 0, 0]
msgLWI_01 = dbCH0.get_message_by_name('LWI_01')
msgDecoded = dbCH0.decode_message(msgLWI_01.frame_id, dataIni, decode_choices=False
fSetIniValues(msgLWI_01, msgDecoded)
#Set other values (not initial values)
#msgDecoded [signalName] = value
#Encode values to data
data = msgLWI_01.encode(msgDecoded)
#Create CAN message with correct/simulated data
msgLWI_01_Simu = can.Message(arbitrarion_id=msgLWI_01.frame_id, data = data, extended_id = msgLWI_01.is_extended_frame)

Well, I don't know how to open a request. I though it was similar to open an issue but it is not.

it is, but with patches to the code attached. If you're not yet familiar with coding on git/github, https://guides.github.com/introduction/flow/ might help...

Thank you andlaus. As I understand, to open a request I need to make a commit first, so I need create a branch or fork the library in order to compare it with the master(for example). Am I right? I find this quite different from openning an issue.
Obviously I am not familiar with coding on git and sincerelly I think I am still far from creating a proper request. Thank you anyway.

As I understand, to open a request I need to make a commit first, so I need create a branch or fork the library in order to compare it with the master(for example). Am I right?

basically yes: you first need to fork the repository to your personal github account via the "fork" button on the original repository (i.e., https://github.com/eerimoq/cantools/ ), then you clone that new repository to your local computer, followed by implementing the changes. After that, commit them locally on your machine (if possible into a branch not called master), push them to your personal github repository and finally open a pull request against the original repository. That said, note that a "fork" on github is basically equivalent to copying the respective repository to your personal playground, i.e., not what is most people understand when talking about forks in the open source context...

Would this solve your request ?
def get_data_from_signals(sigs: dict, can_id: int,
                          dbc: database.can.Database):
    """Create frame data using signal dict and dbc.

    Set unspecified signals to their initial values."""
    msg_dbc = dbc.get_message_by_frame_id(can_id)
    for signal in msg_dbc.signals:
        if signal.name not in sigs.keys():
            val = signal.initial if signal.initial else 0
            sigs[signal.name] = val
    # Produce data
    data = dbc.encode_message(can_id, sigs)
    return data

This works for me, using cantools 36.4.0 : for each signal I get the initial value from dbc if it exists, else 0.

Thanks AlexDLSy! Now we have two different solutions. I will close the issue. Thank you all!

Note that my solution does not work for multiplexed signals, as it will force the multiplexer signal to its initial value or 0.
I will edit my answer once I have a countermeasure for that usecase.


Create image with empty database

I'm using docker-compose for testing with one of containers is postgres. Each time I start compose it takes postgres image and spends some to create container out of it because it runs docker-entrypoint.sh.
So I want to speed it up and create Docker image with empty seed database:
FROM postgres:10.1

ENV POSTGRES_PASSWORD mypassword
ENV POSTGRES_DB shell

RUN docker-entrypoint.sh postgres

ENTRYPOINT ["echo DB shell started"]
However it doesn't work because command RUN docker-entrypoint.sh postgres never exits. How to do it the right way?

When you RUN docker-entrypoint.sh postgres, you're literally starting up the PostgreSQL server, which expects to run indefinitely.
As to what you're trying to accomplish, see https://github.com/docker-library/postgres/issues/375 and https://github.com/docker-library/postgres/issues/319#issuecomment-327254512.  You'll need to set PGDATA, and likely run initdb yourself (because docker-entrypoint.sh's initialization code is tailored for starting up a persistent instance of PostgreSQL, as you've discovered).
In the future, these sorts of questions/requests would be more appropriately posted to the Docker Community Forums, the Docker Community Slack, or Stack Overflow.


(#182) Elevate current linter warnings to errors

Closes #182

Updated eslint config rules from warn to error
Removed linter rules from config since eslint:recommended is extended making these rules redundant

@seyilonge-ps Hey there! It seems that there are ESLinting errors here.  Can you fix and push?

@seyilonge-ps  Can you review what is wrong with the ESLint scan?  It's failing.

Can you look at the eslint checks? They are failing.

I just rebased with the main branch. Lint checks should pass now.


ActiveModel::ForbiddenAttributesError in permit.tap chain

Currently (Rails 4) such a piece of code is not working:
def product_params
  params.require(:product).permit(:name).tap do |whitelisted|
    whitelisted[:data] = params[:product][:data]
  end
end

take from http://edgeguides.rubyonrails.org/action_controller_overview.html#outside-the-scope-of-strong-parameters

It is working fine as long as you do not have
config.action_controller.action_on_unpermitted_parameters = :raise
in your environment enabled.
If enabled it will raise an error before going into the whitelisted block. Is it a Bug?

@fseefried I really don't remember what happened on Sep 30, 2013. This seems to be obsolete.

Hmm - I've the same problem on most recent Rails 4.2, as soon as I enable config.action_controller.action_on_unpermitted_parameters = :raise


instanceSigs

Is it necessary to use memo in the pure implementation?

We need memo so that the tests ensuring Pure and Spider produce the same results can be run relatively efficiently.  Perhaps we should add some comment telling readers they can ignore it?  memo is semantically equivalent to id.

Hmmm, I'm surprised that the tests are complex enough to create performance issues.


added novatel pp7 installation guide

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.1 out of 2 committers have signed the CLA.:white_check_mark: cyPurDue:x: cuiiyanncuiiyann seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


Configure sphinx.ext.extlinks for issues

Spotted this in the aspw documentation: https://github.com/rogerbinns/apsw/blob/3.34.0-r1/doc/conf.py#L29-L36
extlinks={
    'cvstrac': ('https://sqlite.org/cvstrac/tktview?tn=%s',
                'SQLite ticket #'),
    'sqliteapi': ('https://sqlite.org/c3ref/%s.html', 'XXYouShouldNotSeeThisXX'),
    'issue': ('https://github.com/rogerbinns/apsw/issues/%s',
              'APSW issue '),
    'source': ('https://github.com/rogerbinns/apsw/blob/master/%s',
               ''),
    }

Which lets you link to issues like this:
:issue:`268`

Relevant TIL: https://til.simonwillison.net/vscode/vs-code-regular-expressions


Throwing the exception after cleanup from SetUpLoad

Throwing the exception after cleanup from SetUpLoad to the caller method, so that it can be used later on

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.Jaikish Thamarasseril seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

Change committed through a different PR


Option to make the popper top of all DOM elements.

Is your feature request related to a problem? Please describe.
When using inside a custum popup or material ui table the popper of the datepicker is going under the view. It should be displayed above the table or popup. Changing the z-index didn't work.
Describe the solution you'd like
There should be an option to make the popup overlapping or above all DOM elements. it is going inside the table not displayed above.

Describe alternatives you've considered
making the position to fixed and dynamically changing the top and left properties may work perfect.
Additional context
Add any other context or screenshots about the feature request here.

You can create a React portal for it outside the normal DOM flow, take a look at Portal by id example. It should then work as you suggest, and you can change the z-index if you need it to be infront of a modal.

thank you, it got worked in two step:
step 1 - import createPortal
import {createPortal} from 'react-dom';step 2 - add this props to the datePicker:-popperContainer={({children}) => createPortal(children,document.body)}`
simple, it got worked in my case. Hope someone may get helpful


Have User arguments make more sense

It's unclear from looking at api.py whether you should pass in an email or a user ID or what.

For posterity: user is the "current_user" - automatically taken care of by APIResource in line 222 of api.py.
...
http_method = request.method.upper()
        user = session['user']

        if path is None:  # Index
            if http_method not in ('GET', 'POST'):
...


Split source and compiled files

While there could be arguments for having both the source TS and the compiled JS files in the same repository, I do not believe that there are arguments for having them in PRs / manual commits.
I suspect that code reviews will mostly focus on the TS part which means manual changes to the JS could sneak past reviewers which is a security concern. Furthermore I cannot imagine any sane person enjoys comparing TS and JS code just to see if they do the same thing..
If we need to have both in the same repo, the least we could do is have CI compile the files and commit them back every time a change is made to the TS files. This also guarantees that a uniform compiler configuration is used.
Just to be clear: I don't believe there is a good argument to have the JS files in this repo at all, they could either be compiled when the action runs or be compiled just for the tagged commit and added as a release artifact. I'm not sure how Github Actions uses the release though, so someone with a bit more expertise should look at that first.

We have to check in our node_modules as well as the .js files to make any action work. You could checkout this branch:https://github.com/Azure/k8s-deploy/tree/releases/v1.4
I do agree with your points. Also, we are currently in the process of standardising the way we maintain our actions right now. This seems like a good addition to the bunch of new standards we're planning for all of our azure actions. 😄
Looping in @N-Usha to take a look.

https://github.com/actions/typescript-action
I think the recommendation is to use a different branch for releases.

@SamMousa I don't think there are any apprehensions from the action team to not have .js files in your main branch either. Some actions could have their tests run on the .js alone.
I agree that they're redundant in our case as jest test runs work against .ts files. We will consider removing them.

I think they are a security risk unless you check everything every time... If I add a dependency, are you checking to see if I didn't change the dependency code during review?
The review load is bigger. An alternative I've used is to at least have a CI job for compilation and dep installation to see if it gets the same result. Than you can remove the risk and burden of manual review. Of course that requires the compilation to be deterministic..


Add disable_comments option

Motivation
In prost-build's Configurations, there is a disable_comments. This disables the generation of comments in proto as documentation comments. However, it only works for message and its fields, not service and its rpc.
This Pull Request adds similar functionality to tonic-build's Configurations.
Solution

Add disable_comments to tonic_build::Builder.
The documentation generation points for tonic_build::server::generate and tonic_build::client::generate have been changed.

I'll fix it later.

I rebased and fixed the conflicts and tests.
I didn't try it on ubuntu-latest, so it might fail.


Add folium 0.14.0

Changelog: https://github.com/python-visualization/folium/blob/v0.14.0/CHANGES.txt
License: https://github.com/python-visualization/folium/blob/v0.14.0/LICENSE.txt
Requirements:

https://github.com/python-visualization/folium/blob/v0.14.0/pyproject.toml
https://github.com/python-visualization/folium/blob/v0.14.0/requirements.txt
https://github.com/python-visualization/folium/blob/a457a868ff17c109df170afa9c08b6c179b29d10/setup.py#L61

Actions:

Remove noarch

Skip py<35

Add --no-deps to script

Add missing setuptools, setuptools_scm, wheel to host

Fix python in host and run

Update branca pinning

Add license_family

Add description

Add dev and doc urls

Notes

folium depends on branca https://github.com/AnacondaRecipes/branca-feedstock/pull/3

Do you know what's going on with the ibmz vs linux-s390x build. It seems odd that one machine can find branca 0.6.0 and the other one can't. This should be resolved before merging so that we can guarantee that the artifacts will be available on zeus.

Prefect has duplicate agents for linux-s390x (linux-s390x, ibmz, ibmz2) but only one of them will create artifacts. I hope this will be fixed soon

osx-64 and osx-arm64 agents on Prefect failed:
ssh: connect to host github.com port 22: Operation time out


[cherry-pick] [branch-2.3] [BugFix] Fix the problem of estimate memtable size (#7316)

The size calculation cost is relatively large for some types of Column such as BitmapColumn, so the estimation method of this MemTable Size is changed to the incremental before.
When importing, a Chunk will involve multiple Tablets. If there are large differences in length in different Tablets, for example, the average length of the string in Tablet 1 is 100 bytes, and the average length of string in Tablet 2 are small or is null. The estimated size of memtable is chunk_bytes_usage += chunk.bytes_usage() * size / chunk.num_rows().
However, in extreme cases, due to incorrect estimation, the MemTable will be too large or too small, or even exceed 4G, resulting in crash or data confusion.
The new strategy is to Incremental calculate directly with chunk of MemTable
Before modification, MemTable Size
...
FLUSH:16949250
FLUSH:16983890
FLUSH:17078640
FLUSH:17270930
FLUSH:17073270
FLUSH:16934340
FLUSH:16890280
FLUSH:17088330
FLUSH:17014780
FLUSH:17002530
FLUSH:17299380
FLUSH:7911050
FLUSH:2497327728
FLUSH:2488632580
FLUSH:3305326048
FLUSH:3309594000
FLUSH:3303867992
FLUSH:1495277328
FLUSH:4116982500
...

After modification, MemTable size
...
FLUSH:104910526
FLUSH:104868988
FLUSH:104885820
FLUSH:104871402
FLUSH:104864160
FLUSH:104893658
FLUSH:104868340
FLUSH:104864715
FLUSH:104868988
FLUSH:104890714
FLUSH:104861746
FLUSH:104868356
FLUSH:104860294
FLUSH:104874798
FLUSH:104893900
FLUSH:104876775
FLUSH:104872573
FLUSH:104861746
FLUSH:104871402
FLUSH:104888300
FLUSH:104876720
...

run starrocks_clang-format


Detect non-existing secret when mounted as a volume

When a template mounts a non-existing secret as an env var, we eagerly detect that and fail the deploy. However,  this detection is not working when it is mounted as a volume.

Right now the information needed to detect this is only exposed in events not in the pods status. (Capturing events is far too expensive to do during a deploy). The comment at https://github.com/Shopify/kubernetes-deploy/blob/master/test/integration/kubernetes_deploy_test.rb#L294 is still accurate.

😢 too bad. Thanks for looking into it.


 changes in this plugin conflicts with  changes in config.xml

Problem
Upon adding the platform Android, an error occurs while installing this plugin.
The error message is this:
Failed to install 'cordova-plugin-telerik-imagepicker': Error: cordova-plugin-telerik-imagepicker cannot be added. <edit-config> changes in this plugin conflicts with <edit-config> changes in config.xml. Conflicts must be resolved before plugin can be added. at PlatformMunger.add_plugin_changes (<path_to_project>/node_modules/cordova-common/src/ConfigChanges/ConfigChanges.js:135:19)
Analysis
From my analysis, the error stems from the commit that solves issue #209.
The change adds an edit-config to the plugin.xml which conflicts with an edit-config in my own app's config.xml.
Here's what I have in my app:
<edit-config file="app/src/main/AndroidManifest.xml" mode="merge" target="/manifest/application"
    xmlns:android="http://schemas.android.com/apk/res/android">

    <application android:usesCleartextTraffic="true" />
    <application android:networkSecurityConfig="@xml/network_security_config" />
    <application android:allowBackup="false" />
</edit-config>

i'm trying to find a way to solve this, but I've only solved it by explicitly removing the edit-config from the plugin's plugin.xml.

Solution
I've managed to solve this by using only the AndroidMainfest.xml in the edit-config of my app's config.xml.
It looked like this:
<edit-config file="AndroidManifest.xml" mode="merge" target="/manifest/application"
    xmlns:android="http://schemas.android.com/apk/res/android">

    <application android:usesCleartextTraffic="true" />
    <application android:networkSecurityConfig="@xml/network_security_config" />
    <application android:allowBackup="false" />
</edit-config>

I don't know if this is an adequate solution, but it fixed the error.
The configuration seems correct in the AndroidManifest.xml, too.

I got the same problem.
But when i chanved de  tag.
I got this https://github.com/Telerik-Verified-Plugins/ImagePicker/issues/212

@CaioMelo8 I tried your solution but didn't work for me. Still getting the error

Even I am getting this error.  Should this issue be opened ?

I had to downgrade the package to version 2.3.3 and i added the preference in my config for the timebeing

I had to downgrade the package to version 2.3.3 and i added the preference in my config for the timebeing

Do you resolve the error? I  also block by this error

Solution
I've managed to solve this by using only the AndroidMainfest.xml in the edit-config of my app's config.xml.
It looked like this:
<edit-config file="AndroidManifest.xml" mode="merge" target="/manifest/application"
    xmlns:android="http://schemas.android.com/apk/res/android">

    <application android:usesCleartextTraffic="true" />
    <application android:networkSecurityConfig="@xml/network_security_config" />
    <application android:allowBackup="false" />
</edit-config>

I don't know if this is an adequate solution, but it fixed the error.
The configuration seems correct in the AndroidManifest.xml, too.

@CaioMelo8 hi, which version do you install?

@saviour2008
Hi there.
I installed version 2.3.4, but I installed using the GitHub url and not the NPM registry.
Basically:
```cordova plugin add git+https://github.com/Telerik-Verified-Plugins/ImagePicker.git#2.3.4````
I haven't tested with the 2.3.5 version yet, but I'm planning to update the version soon.


Dummy Server is not called properly

When init.py is called for when app.py tries to import andor when the camera is not connected, it tries to import andor.py instead of dummy.py.

We also need to update dummy.py to share the same return type as the andor functions.


Update fastparse to 3.0.1

About this PR
📦 Updates com.lihaoyi:fastparse from 2.3.3 to 3.0.1 ⚠
📜 GitHub Release Notes - Version Diff
Usage
✅ Please merge!
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

⚙ Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.lihaoyi", artifactId = "fastparse" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "30 days" },
  dependency = { groupId = "com.lihaoyi", artifactId = "fastparse" }
}]

labels: library-update, early-semver-major, semver-spec-major, commit-count:1

Superseded by #109.


kernel: add s390 config for version 4.19.8

Remove configs s390_kata_kvm_4.14.x
The patch 0003-serial-forbid-8250-on-s390 is no longer necessary as it
has been upstreamed since version 4.16
The kernel configs have been generated as described in https://github.com/kata-containers/packaging/issues/246
plus the vsock options have been manually enabled:
CONFIG_VSOCKETS=y
CONFIG_VIRTIO_VSOCKETS=y
CONFIG_VIRTIO_VSOCKETS_COMMON=y
Fixes: #280
Signed-off-by: Alice Frosi<EMAIL_ADDRESS>

Thanks @alicefr.
lgtm

lgtm
/test

do we need to bump the version?

@devimc I guess yes, but I don't know which PR should be taken before between https://github.com/kata-containers/packaging/pull/278 and https://github.com/kata-containers/packaging/pull/269


State file topic configs

Hi there,
I've added some topic configs to my state file, and when running a plan, it's trying to add the configs that already exist?
Context:
Topic is pre-existing and was created using the ccloud cli tool. Some configs were passed at the time of topic creation using --config. I'm looking to 'import' the topic into the state file.
Examples:
State.yaml
topics:
  test_partitions:
    partitions: 128
    replication: 3
    configs:
      cleanup.policy: delete
      retention.ms: 604800000

ccloud kafka topic describe (removed a bunch of output)
Configuration

                   Name                   |        Value
+-----------------------------------------+---------------------+
  cleanup.policy                          | delete
  compression.type                        | producer
  retention.ms                            |           604800000
...

Plan:
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update

The following actions will be performed:

Topics: 0 to create, 1 to update, 0 to delete.

~ [TOPIC] test_partitions
	~ configs:
		+ cleanup.policy: delete
		+ retention.ms: 604800000

ACLs: 0 to create, 0 to update, 0 to delete.

Plan: 0 to create, 1 to update, 0 to delete.

My assumption (expected outcome) was that the plan would say There are no necessary changes; the actual state matches the desired state. and not need to modify anything.
If I remove the additional configs from the state file, it comes back as no necessary changes. Since we created these topics with specific configs, I'd like to have the state file show those configs.
Thanks

Hi @alex036, these are not showing as changes because they are considered broker defaults -- Confluent Cloud defaults to a cleanup.policy of delete and a retention time of 7 days, e.g. 604800000 ms.
So, when the tool looks for topic configs, it only looks for user-defined ones. Thus, the tool believes these configs have not been changed and ignores them. If you want to have the defaults explicitly defined as you have shown in your state file, you can just run an apply with the plan and it will put them in as a custom config. Confluent Cloud must not set custom configs if default values are provided. (I would do this with a test topic first to ensure nothing actually changes).
I tested out this scenario against a Confluent Cloud cluster and saw the same.

Hey @devshawn,
Appreciate the prompt reply.
I've been playing around with adding in other existing topics and I think I based most of my testing off of some test/unused topics that may have indeed been 100% default.
The other topics now have plans with the following as an example:
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update

The following actions will be performed:

Topics: 0 to create, 1 to update, 0 to delete.

~ [TOPIC] accounts_receivable
	~ configs:
		- cleanup.policy
		- min.insync.replicas
		- retention.ms
		- segment.ms

ACLs: 0 to create, 0 to update, 0 to delete.

Plan: 0 to create, 1 to update, 0 to delete.

Then I've just been adding in those custom configs to the state and re-planning to confirm the state and the cluster are the same state.
Thanks!


Support Microsoft Entra ID with Redisson client library

Context
Refer to https://github.com/MicrosoftDocs/azure-docs-pr/pull/260292#discussion_r1445151576, make MSI auth the default/recommended option
Purpose
Support Microsoft Entra ID with Redisson client library.
Pull Request Type

[ ] Bugfix
[x] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[ ] Other... Please describe:

Other Information

Seeking review and merge from @philon-msft .

Hello @philon-msft . As far as I know, @backwind1233 and I have addressed all outstanding concerns.

Hello @philon-msft ,
@backwind1233  and I went through the remaining issues and marked them Resolved based on his confirmation that the issues were addressed.
If you agree that the issues are all resolved, please do merge this PR.  If not, please help me understand what we missed.
Thanks,
Ed


Openscapes Update Merge Request

Thank you thank you @IleanaF for your amazing work on this! Thank you @stefaniebutland & @jules32 also!
🚀

Woohoo! Merging, and then will check and update the GitHub Action :)

Yay @gavinfay @IleanaF @stefaniebutland it's all updated and working, published with the updated github action!
Gavin there's obviously a lot more stylizing goodies to explore and we can share tips as we learn them too. Great job @IleanaF ! 🎉

Brilliant! 🙌


Change support for Pocket Casts

The extension works fine for Pocket Casts but only on the beta. Since they are identical, can you make it work on both?

Apparently, the problem was Brave messing with the extension, now it works just fine, sorry :D


feat(quickstart): Remove kafka-setup as a hard deployment requirement

This PR does a few improvements to our docker cli behaviour, namely:

Makes kafka-setup job optional behind the flag: --kafka-setup which works similar to standalone consumer containers.
Makes docker-compose Quickstart generation logic able to handle duplicate env vars with last-wins heuristic. This was done because we have conflicting configs coming from docker env files & explicit docker-compose environment variables.
Generates Neo4j M1-compatible docker-compose files which removes a long-standing limitation in our cli.
M1 Quickstart was not being generated by our internal scripts. This makes it very easy for changes in our partial compose files to not get mirrored in the final Quickstart file. This has now been fixed.
Makes standalone consumer configs for neo4j deployments consistent.
Improves Quickstart comparison logic more flexible with the use of yq

Checklist

[x] The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
[ ] Links to related issues (if applicable)
[x] Tests for the changes have been added/updated (if applicable)
[ ] Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
[ ] For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

Like the approach, probably you've found an issue with the mce consumer quickstart generation.

I am wondering if we even need a separate m1 flavor anymore? Seems like folks have moved to generate arm images consistently, however perhaps I am overlooking something here.

I am wondering if we even need a separate m1 flavor anymore? Seems like folks have moved to generate arm images consistently, however perhaps I am overlooking something here.

There are a few images that we need M1 alternatives for still such as Postgres & Neo4j.


Menu dropdown propType should support numbers for length

Fixes #
Additional description
As a consumer of DSR, currently when passing a number type length to MenuDropdown, I get a propType validation error. I am currently striving for 0 validation errors, but think that number type should be supported by the length prop. I should not have to know about internal implementation details that it gets cast to a string (which is a clean cast) for the class name.

CONTRIBUTOR checklist (do not remove)
Please complete for every pull request

[ ] First-time contributors should sign the Contributor License Agreement.  If you haven't before, wait a few minutes and a bot will comment on this pull request with instructions.
[ ] npm run lint:fix has been run and linting passes.
[ ] Mocha, Jest (Storyshots), and components/component-docs.json CI tests pass (npm test).
[ ] Tests have been added for new props to prevent regressions in the future. See readme.
[ ] Review the appropriate Storybook stories. Open http://localhost:9001/.
[ ] The Accessibility panel of each Storybook story has 0 violations (aXe). Open http://localhost:9001/.
[ ] Review tests are passing in the browser. Open http://localhost:8001/.
[ ] Review markup conforms to SLDS by looking at DOM snapshot strings.

REVIEWER checklist (do not remove)

[ ] TravisCI tests pass. This includes linting, Mocha, Jest, Storyshots, and components/component-docs.json tests.
[ ] Tests have been added for new props to prevent regressions in the future. See readme.
[ ] Review the appropriate Storybook stories. Open http://localhost:9001/.
[ ] The Accessibility panel of each Storybook story has 0 violations (aXe). Open http://localhost:9001/.
[ ] Review tests are passing in the browser. Open http://localhost:8001/.
[ ] Review markup conforms to SLDS by looking at DOM snapshot strings.

Required only if there are markup / UX changes

[ ] Add year-first date and commit SHA to last-slds-markup-review in package.json and push.
[ ] Request a review of the deployed Heroku app by the Salesforce UX Accessibility Team.
[ ] Add year-first review date, and commit SHA, last-accessibility-review, to package.json and push.
[ ] While the contributor's branch is checked out, run npm run local-update within locally cloned site repo to confirm the site will function correctly at the next release.

Thank you for the PR!  I'm going to review this by the end of the day tomorrow and leave comments and/or merge it.


Allow multiple examples for options

This makes it able to have multiple examples for custom messages if parser is updated.

I don't really see what that would improve so I'm just gonna close it for now


Update manifest.xml

Q
A

Bug fix?
yes

New feature?
no

New sample?
no

Related issues?
fixes #88

What's in this Pull Request?
Changing taskpane.html to index.html (see #88 )
It should technically make no difference as the projects seems to be rerouted to index.html anyway but formally the OP of #88 is right and the project does not have a taskpane.html
Guidance
No guidance, just stick to the instructions as defined in Read Me

@aafvstam This is great! Thanks for the help on resolving this issue!


Collection search uses paging as opposed to search-after header

The following code uses page_size and page number to iterate through results:
https://github.com/nsidc/earthaccess/blob/8fe60974ce0f6e5d6f8fbec679afb96f12f1506f/earthaccess/search.py#L282C13-L282C64
Limit could be set to a value sufficiently high to cause CMR problems. Search-After is used to combat this.
Documentation on search-after: https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html#search-after
See similar fix to another CMR python library here: https://github.com/nasa/python_cmr/commit/1702100ab3f8dcb4843ab9da184612b372ec1854

I plan on fixing this in the same way I did with nasa/python_cmr.

Great catch! Luis resolved this long ago for granules (#145), and collection search didn't even cross my mind at the time :laughing: It's just not something I normally do I guess.

Yeah, I got thrown because you have two classes in the same file! I would suggest factoring out the granule code to be used in the dataset code too since it should be the same except for marshaling the results.

Looking more closely at the granule get method,
Search-After is only used if the format of the requested data is umm_json?
https://github.com/nsidc/earthaccess/blob/8fe60974ce0f6e5d6f8fbec679afb96f12f1506f/earthaccess/search.py#L642

:eyes: Not sure why that's like that! The whole conditional is a bit confusing to me. I think we badly need a refactor on that piece of core code :grin:

@doug-newman-nasa would you be interested in attending one of our bi-weekly hackathons?

Yeah, this is mainly because earthaccess has only parsers for umm_json. Initially I wanted to write parsers for json, iso19115 and echo10 but I've always run into irregularities with the metadata and umm_json has been the most predictable flavor of the schemas.

So, the reason I'm 'looking' here is whenever I find myself in a repo for a python wrapper for CMR I immediately check they aren't deep-paging. You aren't where it counts (granule) but are in other places (collection). But it pointed out some redundancy in your two classes. I'd like to tackle some refactoring while addressing the main issue. Discussing that at a bi-weekly hackathon would be extremely useful (at least to me).

@betolink Do you think the next step is to remove the temporary (?) code for the other parsers? Or start considering writing parsers for the other available CMR formats?

... it pointed out some redundancy in your two classes. I'd like to tackle some refactoring while addressing the main issue. Discussing that at a bi-weekly hackathon would be extremely useful (at least to me).

:bow: This will be so immensely appreciated! It's hard to get refactoring on core code like that done because we have many high-demand features and bugs splitting our attention! That doesn't make the refactors any less important, though. Looking forward to seeing you in two weeks!

@mfisher87 I'm leaning towards just supporting umm_json for now and when the time comes we can map CMR responses to the proper results parsers. This brings me to 2 related topics,

We use umm_json because of its completeness and consistency but, could the same be achieved by GrapQL queries that in theory are faster?
If we refactor the way we handle response it would be interesting to follow what pystac-client does, the results are wrapped into a data structure that contains stac items, we could do something similar. This could bring UX improvements to the way we present the data.

GraphQL queries that in theory are faster?
GraphQL might be faster if you are doing multiple queries. The primary utility of GraphQL is reducing the number of queries by the client but there is still potentially 'multiple queries' going on behind GraphQL. If you are getting everything you need from umm_json then GraphQL isn't going to speed things up. It's still taking your query and querying the CMR API. But it might have utility elsewhere.
the results are wrapped into a data structure that contains stac items
If you want stac items why not use the CMR STAC API?

We do not necessarily need the stac items as such, just the way the response is handled. eg.
results = earthaccess.search_data(**params)

Right now, results is a list of parsed umm_json items. Each instance of this list is an enhanced Python dictionary with some convenient custom methods like data_links() etc.
I'm thinking of refactoring this in a way that results will be a wrapped on a class that contains the same umm_json items but the handy methods can operate on the entire list. The semantics are a bit different but that will save users all those for loops to collect links from the results or filter granules by a particular criteria etc Maybe this is not as urgent, I just thought it could be interesting to explore! this is the class in pystac-client: https://pystac-client.readthedocs.io/en/latest/api.html#pystac_client.ItemSearch

Just to check if we're on the same page, you're thinking a search would return a Results object containing Granule objects and having some special methods that can make operations on the whole list easier? I've been thinking this might be useful as well. E.g. results.get_links()?

@mfisher87 correct! we could also implement pagination like in the stac search results results.next_page() if we don't want to load all the results in one go.

I love that. Or we could promote a generator usage pattern? next(results)? Or to get them all list(results)!

Started work on this. What I want to do is add VCR to your testing. I think I only see one test that actually queries CMR: test_data_links. So there is nothing that really tests collection search url construction or results parsing at the collection level. Using VCR we can test this and remove the need to hit CMR each time the test is run. You are also relying on the result contents returned by test_data_links not changing in the future. Thoughts?

TIL VCR! I'm having one of those "how have I not heard of this?" moments ;) That's a really cool idea. I'm all for this! Would you be using the betamax implementation?

cc @danielfromearth this may be helpful for unit tests for #426

I'm using https://vcrpy.readthedocs.io/en/latest
I should have a PR ready today.

PR submiited: https://github.com/nsidc/earthaccess/pull/494

I can't stress enough how much I feel this has improved our tests and will improve our testing practices going forward. Thank you! :100:

Fixed by #494