[discussion] Java Async

I think it's worth having this conversation even just to keep track of the reasoning behind the decisions.
When I looked at the codebase, at first, I was pretty impressed by the fact that we are using async calls everywhere for Java.
This is a subject pretty much full of nuances in Java land as far as I know it.
Let's try to summarize a few key points:

CompletableFuture API is not great and doesn't play well with most of the rest of the standard library, also, using it makes the code much much harder to understand and follow (e.g. we miss an async/await construct)
Reactive programming has become quite popular in Java, and exposing a CompletableFuture interface is going to play nicely with all the frameworks adopting this paradigm (Akka, Vert-X etc. etc.)
Most of the libraries around are exposing still a blocking API(e.g. OkHttp), and, wrapping every call in CompletableFutures might not be the ideal solution given the number of options that comes (scheduling on which Thread Pool? Keep separated IO blocking operations from in-process computing etc. etc.)
"Idiomatic" Java 8 code (as per Android compatibility) is not using the async API and it doesn't play nicely with imperative and blocking code
Project Loom is coming with the promise to enable blocking idioms to easily work in contexts with concurrency and parallelism (since here we are keeping compatibility with Android I'm not sure about timelines here)

thanks for bringing this up. A lot of the thinking comes from the Microsoft Graph SDK work (pre-kiota) and was kind of recaped here and there.
Long story short; completable future seemed to be the best compromise at the time, only raising API level to 24 (we rose to 26 because of the DateTime and other things we needed). It's not ideal, but its the one with the broader compatibility and broader support.
API level 26 is now supported by 93% of devices (compared with 91% in late 2020), and 24 96.4%. Over time the gains of investing into a broader compatibility solution will naturally reduce.
As per investing into a better developer experience solution, we should demonstrate a significant improvement before doing so. I wouldn't be surprised if Fibers come up with helper functions to wrap completable future to ease up the migration path.
Lastly, we're currently wrapping the call using enqueue which OkHttp will defer to another thread or not depending on whether other requests are being executed, and depending on the dispatcher. That gives full control to the caller (they can implement/swap their dispatcher) rather than implementing our own custom logic.

Thanks a lot for the context @baywet , I think that your answer perfectly recaps the reasoning behind the current decisions/implementation 👍

do you think further discussion is needed or should we close this issue at this time?

Perfectly fine closing this, thanks for taking the time to share the background.

We briefly touched on this point during the community meeting today.
Reopening to gather additional feedback from @papegaaij and @mikrethor 🙏

For additional context we have one precedent of a language not being async: go. There is no primitive for that in go because usually defer the routines at the highest level.
I'm guessing that if we took away the async API surface, people could always do it similarly to go (in this case wrapping the call in a future or other construct themselves).
Do we have any visibility over a potential async/await set of keywords equivalent in Java being worked on? or the reason why this might have been declined? (hopefully we're not talking about compatibility again)

Do we have any visibility over a potential async/await set of keywords equivalent in Java being worked on? or the reason why this might have been declined? (hopefully we're not talking about compatibility again)

In Java land, there is an exciting turnaround with Project Loom and part of it is going to land in Java 21 (going to be released GA in September).
To play well with Loom execution model we should only have blocking calls 🙂
It would be interesting to do a quick experiment to see how much we can generalize some utility methods in an external "extension" to wrap the most relevant calls with async.

I don't think you can actually wrap a synchronous call in an async utility. For true async, you need to be async all the way down. I don't know how good the async implementation for OkHttp actually is, but in theory a good http client using Http 2 could multiplex multiple http requests in parallel over a limited set of connections using only a few threads. There is no way to get this if the SDK performs the calls sync.
That being said, very few people actually use async in Java and using it "the right way" is very hard. Many SDKs in Java provide both options. Many Java developers really prefer a sync API. A suggestion would be something like get and getAsync. This is very common in many Java SDKs and should be possible in the current request builders.

The current Microsoft Graph SDK offers both API surfaces for the last couple of years. However it's built in a different way (using our older generator) and we don't have telemetry over which API surface people are using. Our current plan for the Microsoft Graph Java SDK is to release a preview based of kiota and see what kind of feedback we get.
As per okHttp, we're calling enqueue which in turns calls enqueue on the dispatcher.
As far as I understand kotlin the dispatcher then uses a combination of thread pools and queues to defer the call to a different thread.
This is not "truly" async but between that and the fact we wrap the callback to a Future, it's close enough.

These pools are one of the major issues with using async APIs. We've had several major production outages to some of our other applications due to pools getting exhausted and the whole thing deadlocking.
We use the current Graph SDK in Topicus KeyHub and we use the sync API as do other applications in our company AFAIK.

Maybe someone like @brunoborges (on Microsoft side) and @Sanne (on RH) are interested in weighting in on this decision before we go GA 🙏

Hi all - I'm over on the Azure SDK side of the Microsoft house. We've been building client libraries since 2018, and have a fairly large amount of experience in this area. We did a lot of user testing and investigations back when we started, and we ended up using Project Reactor for our async story, but we ship sync and async clients side-by-side (in the same library). We have full sync and full async stacks. This allows developers to choose to use sync or async, as their needs dictate. Whilst I have zero evidence of this, my gut feeling is that most developers choose to use our sync stack implementation, for its simplicity and debuggability.  I imagine only our largest customers bother with the complexities of async, to eek out the final few percentage points worth of performance.
Obviously when we started the concept of virtual threads was still a long way off, and even then, it feels too low-level of an abstraction to give to developers, as it would be asking them to manage threads (and, realistically, block on them). Similarly, CompletableFuture is good in concept for many use cases, but in some places (in particular, streaming cases), it falls down. This is why we ended up going with Reactor for our async story - it gave a complete story for developers to understand, for all async cases.
Looking ahead to the future, I've done some early thinking about how I would do things today, and it would probably be a custom async API that wasn't based on a third party dependency. This is because Reactor does break more often than we would like, both in terms of implementation and public-facing API. If I had more time back in the 2018 time frame, I would probably have dived more into this, but as it stands I've only got high level thoughts (and a lot of investigations I would like to do).

@JonathanGiles thanks a lot for chipping into this!
I'm not familiar with the graph API, but, if you are shipping artifacts through Sonatype/Maven Central you should have easy access to the download statistics.
We should not trust the absolute numbers, but the delta in between the two distributions might give us an "objective enough" data point if available.
Re: Virtual Threads, they are not such a low level abstraction in the Loom incarnation ...

Do you really think this will make that much of a difference in jar size? Looking at the code, it should be enough to duplicate the method pairs like 'get' and 'post'. A quick test gave an increase in size of about 600 bytes per operation in the class file, which would be less after compression in the jar. I doubt it would be more than 1 or 2% in the total size for an API like the graph API.

I've asked @maisarissi, our Java SDK PM, to engage with existing customers so we understand better what usage of the API surface they are doing today. But the feedback here helps a lot! Thanks everyone!

I checked some numbers, and the graph API indeed has about 20k operations. All class files together are about 100MB, the resulting jar is about 42MB. I've then taken one of the generated request builders and modified the source in 2 steps:

The get() method now calls get(null) rather than duplicate the body. This saves about 180 bytes per operation in class files.
Both get(...) methods are duplicated to getAsync, with getAsync() calling getAsync(null). The get(...) were made sync by calling async(...).get() and the try-catch was modified to catch and ignore all exceptions, That's not very nice, but this is just a test. The real version should probably use a RuntimeException. This added about 740 bytes.

For serviceprincipals/delta/DeltaRequestBuilder.class this resulted in a class that went from 6312 to 6871 bytes, a difference of 559 bytes for one operation. This will probably result in about 11MB more class files and a jar that's about 4.3MB larger. It is however likely that it is possible to save some bytes by moving the exception handling to the abstraction layer.
Some observations I made: moving common code to a new method often increases the size of the class file. Exception handling (and throwing) takes a lot of bytes. toGetRequestInformation(...) declares to throw URISyntaxException, but I can't find any place where it actually does. Removing that exception saved over 400 bytes.
In my opinion, if 46MB is too big (and this can probably be brought back to about 43-44MB), 42MB also is. I think the solution would be to split the jar, not to try to save space on not providing both sync and async.

Thanks for leading that investigation! We should probably fork off to another thread to focus on size optimization.
The baseline for the previous generation of SDK is 11MB (5.66), but it supports far fewer endpoints (about 3 times fewer).
Mapping overloads with fewer parameters to their "complete" equivalents with default values instead: very good idea. I'm not sure why we did it for request generator methods and not for request executor methods. Let me create an issue after I post that reply.
UriSyntaxException. It all comes from here, I didn't realize exceptions were so costly in Java.
https://github.com/microsoft/kiota-java/blob/1a1fc247068bd646116422fbf2cbe575484decad/components/abstractions/src/main/java/com/microsoft/kiota/RequestInformation.java#L52
We should probably review that method to wrap these specific exceptions in runtime exception (bad practice I know) and simplify a great deal the generated code. (let me create another issue for that)
Splitting the SDK: in the case of Microsoft Graph there's no good split we could assume for the customers, because it's a Graph and we can't assume in advance what endpoints/models/properties people are going to need. The functional boundaries are inexistent as well. Best case scenario we'd end up with a lot of duplication and mapping code on the application side. Worst case we'd end up with gaps in the API surface area. This is actually one of the requirements that got kiota started! :-) The long term play would be to have people choose between an SDK (easy and ready to go) or generating their own client for Microsoft Graph (tailored to their needs, lower footprint). But on the Microsoft Graph we still have infrastructure and tooling to allow that new experience to be seamless.

Looking again at this discussion with fresh eyes and I completely understand that this decision is completely dominated by the fact that we are excluding "a-priori" supporting both sync and async.
I would like to challenge this requirement:

in the microsoft/kiota project generating code with and without CompletableFuture wrapper should be trivial, and, it should be possible to keep the logic for the switch pretty much isolated.
in this repo, most of the logic can be shared again, and we only need a thin layer to expose the 2 separate APIs

All in all, this seems to be:

low effort
easy to swap in/out in case a different decision has been taken

A possible approach here might be:

add the possibility to generate both sync/async and mark it as "experimental"
(not mandatory) delay a little going to GA with Java
start to check and track the usage of the variants based on the "Unique IPs" downloads of the core libraries
re-iterate

I can see that, having to decide upfront to limit the flags in Kiota, seems to generate lengthy conversations and opinion-sharing other than actual "data-based" directions.
Thoughts?

Thanks for the additional input here.
I'm all for data driven decisions. The challenge we've faced with questions that are so specific is that answering to them based only on data from our own telemetry/experiments is lengthy at best, often times inaccurate:

we could add telemetry for that specific point in the current SDK which offers both API surfaces today. But we'd need to allow some time (typically between 30 to 90 days) for people to pick up the update and run with that in production so we can collect enough sample data. Even with that the data would be biased towards sync since it was the first API surface offered, it's the API surface present in the docs, and most people just grab the update without changing their code.
we could experiment by putting a preview version out there with both API surfaces and adding telemetry. The data would probably be more accurate (only new projects) but getting a meaningful sample would take much longer (less than 1% of aps try preview builds if a stable release is available, and because most of them are net new apps, it takes a while for them to reach production)
experimenting by putting two libs out there, a sync and an async is pretty much going to yield the same results as the previous one. We'll probably incur a drop from the setup overhead though.

This is why for questions that are this specific we usually rely on readily industry data (yes that includes experts like yourselves opinions), interview with customers, experience from other teams at Microsoft...
The challenge with Java is there doesn't seem to be a clear line in the sand. Not only there's no clear consensus between async (like Task in dotnet) or sync (like Go), there are also multiple flavors of async.
The fact that server frameworks like spring, ORMs like hibernate and reactive libs like project reactor all integrate with Futures strongly leads me to think the best way forward will be future based.  This will cover scenarios like "inbound request, query to an API, query to a database" non-blocking all the way.
People who really need to block will be able to do so over the completablefuture. and one of the added benefit is to be able to cancel requests, which a sync API makes much more difficult.
Right now we'll probably focus on the size improvements before we come back to this topic, which should give time to @maisarissi to interview with customers and give us some additional feedback.

Hello all again,
apologies I'm definitely not an expert about kiota, but since I was asked to shed some light on our experience I feel the need to clarify some points here. I hope this migh e useful for the greater good but feel free to ignore me of course :)

I'm all for data driven decisions. The challenge we've faced with questions that are so specific is that answering to them based only on data from our own telemetry/experiments is lengthy at best, often times inaccurate:

It's really great that you all are attempting to make data driven decision, but please allow me to warn about the metrics that I'm seeing being called out here (several posts earlier too): they are fluctuating over time, and most old data shouldn't be taken into account. For example - the fact that the vast majority of artifacts in Maven Central is using "javax." prefix rather than "jakarta." prefix is totally expected as "javax." has been a pillar of integration among libraries for 20+ years. But most such libraries are out of date and abandoned; I'm fully confident that it won't be in the future, so please try to sample data by taking trends into account, and also that while many libraries in the Java ecosystem are fully committed into migrating to jakarta, several haven't done so yet as it's a slow ongoing process which will take years: the existing API is very well ingrained, although also totally not future proof for legal reasons (I hope you're aware: the package name is trademarked so it can't be evolved further).
Also to bear in mind: some popular libraries, for example the Java security APIs, are not part of the "enterprise libraries" but part of the base platform, so in those cases they will retain the javax. prefix as they part of the Java API. That's also expected, so you won't see any transition to a new naming strategy in this important area, and needless to say many such packages need to integrate with those APIs (on top, or exclusively).

The fact that server frameworks like spring, ORMs like hibernate and reactive libs like project reactor all integrate with Futures strongly leads me to think the best way forward will be future based. This will cover scenarios like "inbound request, query to an API, query to a database" non-blocking all the way.

I can't speak for the other libraries, but I'm the lead architect of Hibernate and designed the API you're liking to here; might be useful to bear in mind that we offer two "reactive" APIs: one based on the CompletionStage and one based on the more modern Mutiny library as described in this section.
The rationale at the time (4+ years ago) was that we wished to make it easy for users to integrate with various other reactive libraries, and the CompletionStage system, since it's being offered by the Java standard libraries, felt like an essential integration to have, while Mutiny was looking promising but was a brand new bet.
Nowadays, I regret integrating with CompletionStage as it has been extemely tricky to handle correctly and we regularly received feedback from users shooting themselves in the foot as they don't always understand all nuances of this very complex API.
Mutiny on the other hand has proven itself as a reliable choice and offers robust integrations with several other reactive types, so it alone would satisfy our requirement of easy integration with all such popular systems; of course I would have preferred there was a single type system so that people wouldn't even need to invoke a tiny type conversion layer, however this seems like the best solution available today.
However, basing one's integration on CompletableFuture is very tricky as the futures might (or might not) be executed right away, implying it's very hard to control several important semantics of asynchronous operations, I would suggest to stay clear of it unless your needs are extremely basic cc/ @jponge
HTH

Thank you for joining the discussion. This kind of library maintainer feedback is gold!
Yes, the caller does not have control over the scheduling of the work. This is because at the end of the day we end up calling enqueue from OkHttp, and OkHttp itself does the scheduling on a thread pool.
https://github.com/microsoft/kiota-java/blob/088694863ea312c45e8f8baa254e2baf3c769cac/components/http/okHttp/src/main/java/com/microsoft/kiota/http/OkHttpRequestAdapter.java#L660
We might be able to control the execution by implementing our own dispatcher and/or executor service here and make it so the request execution only begins when the chain of Futures is being started and/or asked for a result. But I don't think scheduling here was the primary goal, offering non-blocking APIs (for the main thread) was.
As for Munity it seems to be mixing two concerns: async programing and dataset streaming, like a lot of the afore-mentioned options. This later capability is not something kiota clients will need as in most cases API results are either objects or small collections. Additionally for large collections, most Java JSON serialization libraries lack the support of data streaming deserialization as far as I understand, so streaming the items coming from a big memory block would not help performance wise.
All this feedback, and those options, make a great base for discussion with customers for @maisarissi while we make good progress on the size optimization aspects. Keep it coming.

One of the arguments I haven't seen in this discussion so far is the reliability and scalability of an async API when used in a synchronous way (i.e. calling get() right away). As I've mentioned before, in our applications we almost exclusively use APIs in a synchronous way (we block the thread that's performing the external call). Our applications are designed to deal with this in a reliable way, with a high number of task threads, circuit breakers and rather aggressive timeouts.
We do however, occasionally need to use an SDK that only provides an async API (like what's kiota offering at the moment). These have turned out to be really problematic because they often require careful management of a separate thread pool. Let's say the application has 500 task threads. Every task might need to use the async API at some point. Often, these APIs by default use really small thread pools (like 10), which will then be a major bottleneck (this has even resulted in outage). However, you cannot make the thread pool too large either, as threads are expensive to create and keep around. Also, in many cases it's not a good idea to share thread pool between different API. In the end, you are wasting a large amount of resources on threads, spread over many different pools, that are idle most of the time and performance is still suffering when a large amount of concurrent requests needs to be processed because that particular pool is not large enough.

What about a totally different solution: providing both sync and async behavior via the same API? It should not be that difficult to change the implementation to return completed CompletableFutures. This will certainly be a compromise, mostly on the sync part, because the caller still needs to 'get()' the result, but at least it prevents the problems associated with having to deal with a separate thread pool.

When having both blocking (sync) and asynchronous capabilities in the same stack, we try really hard to avoid the two needing to interact: every time an asynchronous operation needs to invoke a blocking one it's a headache of integration and design issues, and every time a blocking method needs to invoke an asynchronous one it needs to, at very least, switch the physical carrier thread which comes at a significant cost of performance.
I wouldn't recommend that.
The one reason in other frameworks we exposed "natively asynch" APIs is precisly for these reasons; if you think you'd better be off with a very thin adaptation layer, I'd recommend rather expose only one of them and let the users deal with it explicitly - it would be a better experience as they would be aware and responsible for how exactly the translation layer works.

@baywet I'm suggesting that you can flip the http abstraction layer in sync-mode, so it will do the http call in the caller thread and simply return CompletableFuture.completedFuture(value). This will make the CompletableFuture nothing more than a wrapper for the actual value, without all the async behavior attached.
The main problem I'm having with an async API in an application that's designed to use sync behavior is the need for a separate thread pool to handle the async calls. If too small, you will get significant congestion on these calls (even up to the point of deadlocks), if too big, they will waste resources (even up to the point of OutOfMemoryError). We've seen both in our applications.

@Sanne yep I was aware of the kind of issues sync over async creates. And async over sync doesn't provide much value when compared to only sync.
@papegaaij I don't think returning a completed future over a sync call would provide much value. And we've already ruled out providing both APIs due to a size constraint/duplicating the pipelines.
The decision is sync XOR async API. And if we maintain async (with completable futures or something else) we should make sure it plays nicely with the HTTP client and its thread pool scheduler (if it uses any).
Another aspect we discussed when looking at this issue if we switch to a sync API surface, we'd probably have to declare the throws of the custom exceptions generated from the description.

If the actual async behavior is being handled in the OK layer (using a thread pool maintained by OK), then does that mean users/community can solve the problem @papegaaij has by contributing their own HTTP client layer?  I haven't dug into the library the way that @andreaTP has, but I know that was discussed.  We were interested because we wanted to e.g. provide a Quarkus specific HTTP client implementation.
But if we can do that, then someone could presumably implement the approach @papegaaij suggested, by providing an HTTP client that actually does the call synchronously and then just wraps the result in an already-completed future.
Personally, as a consumer of the generated SDKs, I would prefer a synchronous API because that's just easier to use and debug.  And for the use-cases I care about, client scalability isn't really much of an issue.  But I understand why async is being used now - it's probably only possible to practically adapt async->sync, not the other way around:  as mentioned, adapting a sync API to async doesn't really accomplish the point of async for scalability - it needs to be actually non-blocking/async all the way down.

We briefly touched on the subject another couple of times, I think that my mind is settled now, on 2 opposite directions 🙂 :

from the "theoretical" point of view(threads control etc.) the async implementation is by far preferable
from the pragmatic point of view, using async code in a sync codebase(which is the majority according to any data I've ever collected), is extremely unpractical and doesn't play well with most of the constructs (exceptions are always wrapped, controlling execution in lambdas is challenging, the "Functional interface" APIs are mostly unusable) and it's likely to force the introduction of breaking changes in public APIs

And I think it is safe to assume that with the introduction of virtual threads in Java 21, async will become more of niche. Virtual threads bring most of the benefits of async without the downsides of unreadable and impossible to debug code.
Go has adopted virtual threads from the start and uses blocking sync APIs everywhere (at least that's what I see in the code I work with). Threads (or goroutines) are cheap and you can spawn as many as you like. There's no problem in having lots of them blocking in a sync call. I wouldn't be surprised if Java is heading the same way when more application servers and frameworks start supporting virtual threads.

Hello everyone! Thanks for all the inputs and discussion here! Things like this is what help us to make the best product and better decisions.
After some considerations like the introduction of virtual thread in Java 21 and conversations with @JonathanGiles @brunoborges and internally on Graph DevX team, we believe that sync only is the best way to go. With that, we can improve debugging capabilities as well as decrease complexity on development.
We are building on top of @JonathanGiles work that have been working with Java community and got feedback like this https://twitter.com/JonathanGiles/status/1711583034007564484 where most developers believe sync only is also the way to go. This also gives people the possibility to wrap calls in their own preferred async way and gives us the possibility to provide our own wrappers in the future, if we want to.

Thank you @maisarissi for going the extra mile here and getting both quantitative data, and experts input in addition to the amazing feedback we got here.
@ramsessanchez please go ahead and clean up completable futures from the API surface here, also remove any async suffix we might have + make the relevant generation change (kiota/snippets/raptor).


Support RTL

It is just enough for these   two isFirst and  isLast attributes to reverse their behavior.
I think it's also better to renamed to isStart and isEnd.

I made a PR request.


only 1 sit coin for block reward[BUG]

Describe the bug
only 1 sit coin for block reward height 75885. 0.125 and 0.875

Incoming
sit1vdz568u893fafcus3ttr552u4z3mz6qjgwwfm4cz5hjrgl3s65zsgt99pw
11/21/2021 12:20:34 PM
Confirmed at height 75887
0.125
0

Incoming
sit1vdz568u893fafcus3ttr552u4z3mz6qjgwwfm4cz5hjrgl3s65zsgt99pw
11/21/2021 12:20:34 PM
Confirmed at height 75887
0.875
0

Steps to reproduce the behavior:

win a block
2 receive only 1 sit coin

Expected behavior
reward is 2 sit or am I incorrect
Screenshots

Desktop

OS: windows
OS 10
CPU: intel i7
Additional context

Not a bug. The reward is 1 SIT per block.


fixing ascii shopware 4 logo

Thank you @teiling88.
Are you actually using the shell functionality of the shopware console?
It was deprecated in Symfony 2.8 recently:
https://github.com/symfony/symfony/commit/1c17928e58fa1a4503c616c681aa50221d2aaf1d
https://github.com/symfony/symfony/commit/c23444985bedfdd4ba547973221d2484c60daa2c
I am thinking about removing it starting Shopware 5.2.
Also I don't wan't to be to picky here but the figlet tool we used to generate the logo gives me a slightly different logo:
$ figlet shopware5
     _                                       ____
 ___| |__   ___  _ ____      ____ _ _ __ ___| ___|
/ __| '_ \ / _ \| '_ \ \ /\ / / _` | '__/ _ \___ \
\__ \ | | | (_) | |_) \ V  V / (_| | | |  __/___) |
|___/_| |_|\___/| .__/ \_/\_/ \__,_|_|  \___|____/
                |_|

Would you mind updating you Pull Request to the above. Thanks in advance.

I would remove this kind of version information anyway. If it doesn't come from the VERSION static, it will just be another thing to maintain in the future.

Good thought @dnoegel. The banner image of the updater/install already got rid of the version information. See: https://github.com/shopware/shopware/blob/5.1/recovery/common/src/IOHelper.php#L230
     _
 ___| |__   ___  _ ____      ____ _ _ __ ___
/ __| '_ \ / _ \| '_ \ \ /\ / / _` | '__/ _ \
\__ \ | | | (_) | |_) \ V  V / (_| | | |  __/
|___/_| |_|\___/| .__/ \_/\_/ \__,_|_|  \___|
                |_|

Are you actually using the shell functionality of the shopware console?
yes - autocomplete is nice :)

i removed the version number like @dnoegel mentioned

:+1:

Thank you @teiling88


Lepton image compression

I saw enhancement about FLIF image compression. But its for lossless images. There are Lepton compression from Dropbox. It offers bit identical compression-decompression of JPEG with 22% gain.
https://github.com/dropbox/lepton

Precomp already uses PackJPG for lossless compression of JPG files. Lepton is similar (it is based on uncmpJPG by packJPG's author) and gives similar compression ratios. Testing the JPG file from issue #24:
Original file: 4,307,553 bytes
Lepton v1.2: 3,395,557 bytes, 2.4 s
PackJPG v2.5j: 3,415,122 bytes, 4.8 s
Precomp v0.4.6: 3,292,325 bytes, 3.5 s
paq8px_v112 -3: 3,076,748 bytes, 109 s
Lepton is faster because it's using multi-threading. Precomp compresses this particular file a bit better than packJPG because it also compresses the thumbnail it contains. paq8px has a more advanced JPG compression and gives even better results, but it's very slow.
Since I already have plans to do multi-threading too, Lepton doesn't give big advantages over packJPG.


[GOBBLIN-1187] cancel flow execution when dag manager is disabled

Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA

[x] My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"

https://issues.apache.org/jira/browse/GOBBLIN-1187

Description

[x] Here are some details about my PR, including screenshots (if applicable):

In this PR, flowExecution endpoint will call spec producer's delete method for http delete verb.
unit tests in kafka-09 module will be run sequentially. this has proven effective in dealing hanging issues in test cases

Tests

[x] My PR adds the following unit tests OR does not need testing for this extremely good reason:
existing test cases

Commits

[x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Codecov Report

Merging #3027 into master will increase coverage by 0.03%.
The diff coverage is 16.54%.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 31f2ef1...459a424. Read the comment docs.

The PR itself LGTM, but I don’t know indication of this change for the service itself, especially the backward-incompatible part. Will leave it to more knowledgable reviewers for this point


I changed gmail to allow only secure logins, & now K-9 cannot login via either ssl/tls or STARTTLS, Thunderbird can login via Oauth..

Expected behaviour
This is in 5.010
K-9 should have maximally-secure login methods, all of them, and should try them in sequence,
from the hardest to the softest security,
and should never use an insecure login tech without
a) first exhausting all other alternatives, &
b) getting explicit permission from the user to vulnerablize their credentials that way, while logging-in, &
c) ANY change in the security-situation ( ssl becoming unrecommended, as i remember happening a few years ago, eg ), and that module being re-jigged immediately to protect us against violation as much as possible.
( i remember reading that ssl is now untrustworthy and TLS is recommended, but have no idea if the initial setup is secure in TLS, or if the problem is simply Google acting like Microsoft and disallowing standard protocols/methods as a monopoly-pushing means making its apps the only "secure" ones users can know/see/have...
I hate the gmail app, and want to use K-9,
but having my email compromised??
-shudder- )
Actual behaviour
After telling gmail to not allow any insecure logins,
neither K-9 nor Thunderbird could login
In Thunderbird i discovered setting it to Oauth fixed things, but there are only 2 choices in K-9, neither of which works.
Account Options / Account Settings / Fetching Mail / Incoming Server
STARTTLS produces..
( NOW i discover there is an EncryptedPassword option? It was hidden under the keyboard, and the Next button i used suggested nothing under the keyboard...
..oh, well, that didnt work anyways! )
"Cannot connect to Server.
Unable to Connect"
When i try SSL/TLS, the result is
( again using the user-set option, which should have been automated: Encrypted Password )
Cannot connect to Server.
(Server doesnt support passwords encrypted using CRAM-MD5)
Steps to reproduce

Change gmail acct to disallow insecure apps logging in
A few hours later discover K-9 no longer is getting gmail
3.try diff settings and discover none work.

Environment
K-9 Mail version: 5.010
Android version: 5.1.1
Account type (IMAP): gmail imap

See issue #655.
Google calls apps that don't support OAuth "less secure". That doesn't make them insecure.

By committing K-9 to that path, instead of adding OAuth,
You make the market feel-need to abandon K-9 for security-reasons, due to Google's monopolistic move.
Gmail app isnt blocked for OAuth requirement, but K-9 is.
If Google is falsely indicating that the login methods without OAuth are insecure,
Either adapt, or allow Google's position to indicate K-9 WONT be made secure, whether objectively-true or not.
Evolution-requirements are never fair to minorities.
If AdaptOrDie-situation mixes with an Adopted Political Position that extinguishes a species/project's viability,
maybe having made A Statement was better than changing the world?
Pointless choice, to me...
Cheers ( deleting K-9 from my devices & telling others to continue defaulting to Maximal Security, even when some providers insist there is some technical-eror in 1 or another such policy )

that's quite the attitude you have there. you did notice the "see #665"? and maybe even the" duplicate" label? we are aware of this issue and there is working code for it, it's just not done.

Having now read the other issue, it looks like Google will prevent any non-Google app from having real security...
( authority theatre instead )
I dont know what the real ranking of security of login methods is, and am obviously incompetent to comment an anything anyways.
It does look like they are now Microsofting their users.
...

Cheers.

Having now read the other issue, it looks like Google will prevent any non-Google app from having real security...

No, you just have to use the OAuth 2 methods to get a token. The app is then associated with your account based on the application ID. It's not overly complex, it's just a bit fiddly testing all the edge cases.


Replace thirdweb to wagmi

Let's wagmi!!

wagmi latest version and next 12 and react 18 has issue
https://github.com/tmm/wagmi/issues/542#issuecomment-1144178142


Announcement: Renaming "master" to "main" in this repository

On the 18th of June, 2021 (PST) we will be renaming the master branch in this repository to main. When this occurs there will be some disruption to the engineering system as we address any issues that earlier preparations did not address. Please read the following centralized issue for more background on what to expect:

https://github.com/Azure/azure-sdk/issues/2745

Localized issues in this repository can be discussed here.
Engineering System Checklist
The following checklist will be executed by a member of the Azure SDK Engineering System team when the branch rename occurs. If errors occur they'll need to be addressed:

[x] Ensure repo mirroring is stopped (coordination across repos required).
[x] Rename the master branch to main on this repository.
[x] Rename the master branch on the -pr repository.
[x] Rename the master branch on the azure-sdk user's clone of this repository.
[x] Rename the master branch on the azure-sdk user's clone of the -pr repository.
[x] Ensure repo mirroring is started (coordination across repos required).
[x] Ensure pipeline generation has been run and pipelines have been updated to correct branch.
[x] Submit a PR to ensure that PR validation is executing correctly.
[x] Merge PR to ensure that CI triggers are executing correctly.
[x] Trigger template release pipeline to ensure end-to-end release process is working correctly.

Renaming "master" to "main"

python pip module on pypl is broken, when executing azdev setup, the following error will happen because the pip module is still pointing to master branch
~/env/bin/python -m pip install git+https://github.com/Azure/azure-cli@master#subdirectory=src/azure-cli-testsdk` failed with exit code 1:
Collecting git+https://github.com/Azure/azure-cli@master#subdirectory=src/azure-cli-testsdk
  Cloning https://github.com/Azure/azure-cli (to revision master) to /tmp/pip-req-build-k8x_5z_1
  Running command git clone -q https://github.com/Azure/azure-cli /tmp/pip-req-build-k8x_5z_1
  WARNING: Did not find branch or tag 'master', assuming revision or ref.
  Running command git checkout -q master
  error: pathspec 'master' did not match any file(s) known to git.
WARNING: Discarding git+https://github.com/Azure/azure-cli@master#subdirectory=src/azure-cli-testsdk. Command errored out with exit status 1: git checkout -q master Check the logs for full command output.
ERROR: Command errored out with exit status 1: git checkout -q master Check the logs for full command output.

python azdev module on pypl is broken, when executing azdev setup, the following error will happen because the pip module is still pointing to master branch
~/env/bin/python -m pip install git+https://github.com/Azure/azure-cli@master#subdirectory=src/azure-cli-testsdk` failed with exit code 1:
Collecting git+https://github.com/Azure/azure-cli@master#subdirectory=src/azure-cli-testsdk
  Cloning https://github.com/Azure/azure-cli (to revision master) to /tmp/pip-req-build-k8x_5z_1
  Running command git clone -q https://github.com/Azure/azure-cli /tmp/pip-req-build-k8x_5z_1
  WARNING: Did not find branch or tag 'master', assuming revision or ref.
  Running command git checkout -q master
  error: pathspec 'master' did not match any file(s) known to git.
WARNING: Discarding git+https://github.com/Azure/azure-cli@master#subdirectory=src/azure-cli-testsdk. Command errored out with exit status 1: git checkout -q master Check the logs for full command output.
ERROR: Command errored out with exit status 1: git checkout -q master Check the logs for full command output.

Thanks for your feedback, It's already fixed in this PR, which will be contained in next release.


TFLite YOLOX-Nano  int8 quantization error

Float16 quantization works, No test.
onnx-tf convert -i "yolox_nano.onnx" -o  nano

(base) se@stu:~/yolov5$ python  tf_testv2.py
2022-01-15 08:23:02.309548: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-01-15 08:23:02.309583: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2022-01-15 08:23:03.956724: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-01-15 08:23:03.956793: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublas.so.11'; dlerror: libcublas.so.11: cannot open shared object file: No such file or directory
2022-01-15 08:23:03.956826: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublasLt.so.11'; dlerror: libcublasLt.so.11: cannot open shared object file: No such file or directory
2022-01-15 08:23:03.956864: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcufft.so.10'; dlerror: libcufft.so.10: cannot open shared object file: No such file or directory
2022-01-15 08:23:03.956898: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcurand.so.10'; dlerror: libcurand.so.10: cannot open shared object file: No such file or directory
2022-01-15 08:23:03.956941: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusolver.so.11'; dlerror: libcusolver.so.11: cannot open shared object file: No such file or directory
2022-01-15 08:23:03.956975: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusparse.so.11'; dlerror: libcusparse.so.11: cannot open shared object file: No such file or directory
2022-01-15 08:23:03.957122: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2022-01-15 08:23:03.957408: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-01-15 08:23:05.187060: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:363] Ignored output_format.
2022-01-15 08:23:05.187116: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:366] Ignored drop_control_dependency.
2022-01-15 08:23:05.187121: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:372] Ignored change_concat_input_ranges.
2022-01-15 08:23:05.188210: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: nano
2022-01-15 08:23:05.201529: I tensorflow/cc/saved_model/reader.cc:107] Reading meta graph with tags { serve }
2022-01-15 08:23:05.201573: I tensorflow/cc/saved_model/reader.cc:148] Reading SavedModel debug info (if present) from: nano
2022-01-15 08:23:05.273244: I tensorflow/cc/saved_model/loader.cc:210] Restoring SavedModel bundle.
2022-01-15 08:23:05.527303: I tensorflow/cc/saved_model/loader.cc:194] Running initialization op on SavedModel bundle at path: nano
2022-01-15 08:23:05.743009: I tensorflow/cc/saved_model/loader.cc:283] SavedModel load for tags { serve }; Status: success: OK. Took 554801 microseconds.
2022-01-15 08:23:05.982566: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:237] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
2022-01-15 08:23:06.530075: I tensorflow/compiler/mlir/lite/flatbuffer_export.cc:1962] Estimated count of arithmetic ops: 1.624 G  ops, equivalently 0.812 G  MACs

Estimated count of arithmetic ops: 1.624 G  ops, equivalently 0.812 G  MACs

(base) se@stu:~/yolov5$
(base) se@stu:~/yolov5$ cat tf_testv2.py
import tensorflow as tf
import numpy as np

def representative_dataset():
    for _ in range(100):
      data = np.random.rand(1, 244, 244, 3)
      yield [data.astype(np.float32)]

converter = tf.lite.TFLiteConverter.from_saved_model("nano")
converter.target_spec.supported_types = [tf.float16]

converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_dataset
#converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS]
#converter.inference_input_type = tf.int8  # or tf.unint8
#converter.inference_output_type = tf.int8  # or tf.uint8
converter.experimental_new_quantizer = False
tflite_quant_model = converter.convert()
open('yolox_nano.tflite', 'wb').write(tflite_quant_model)

But when I do int8 quantization, get an error.
(base) se@stu:~/yolov5$ python  tf_testv2.py
2022-01-15 08:25:30.770253: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-01-15 08:25:30.770291: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2022-01-15 08:25:32.418275: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-01-15 08:25:32.418330: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublas.so.11'; dlerror: libcublas.so.11: cannot open shared object file: No such file or directory
2022-01-15 08:25:32.418363: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublasLt.so.11'; dlerror: libcublasLt.so.11: cannot open shared object file: No such file or directory
2022-01-15 08:25:32.418403: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcufft.so.10'; dlerror: libcufft.so.10: cannot open shared object file: No such file or directory
2022-01-15 08:25:32.418436: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcurand.so.10'; dlerror: libcurand.so.10: cannot open shared object file: No such file or directory
2022-01-15 08:25:32.418471: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusolver.so.11'; dlerror: libcusolver.so.11: cannot open shared object file: No such file or directory
2022-01-15 08:25:32.418503: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusparse.so.11'; dlerror: libcusparse.so.11: cannot open shared object file: No such file or directory
2022-01-15 08:25:32.418630: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2022-01-15 08:25:32.418864: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-01-15 08:25:33.335963: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:363] Ignored output_format.
2022-01-15 08:25:33.335995: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:366] Ignored drop_control_dependency.
2022-01-15 08:25:33.335999: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:372] Ignored change_concat_input_ranges.
2022-01-15 08:25:33.336767: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: nano
2022-01-15 08:25:33.345063: I tensorflow/cc/saved_model/reader.cc:107] Reading meta graph with tags { serve }
2022-01-15 08:25:33.345090: I tensorflow/cc/saved_model/reader.cc:148] Reading SavedModel debug info (if present) from: nano
2022-01-15 08:25:33.382117: I tensorflow/cc/saved_model/loader.cc:210] Restoring SavedModel bundle.
2022-01-15 08:25:33.558271: I tensorflow/cc/saved_model/loader.cc:194] Running initialization op on SavedModel bundle at path: nano
2022-01-15 08:25:33.706672: I tensorflow/cc/saved_model/loader.cc:283] SavedModel load for tags { serve }; Status: success: OK. Took 369907 microseconds.
2022-01-15 08:25:33.863306: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:237] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
2022-01-15 08:25:34.391789: I tensorflow/compiler/mlir/lite/flatbuffer_export.cc:1962] Estimated count of arithmetic ops: 1.624 G  ops, equivalently 0.812 G  MACs

Estimated count of arithmetic ops: 1.624 G  ops, equivalently 0.812 G  MACs
Traceback (most recent call last):
  File "/home/se/yolov5/tf_testv2.py", line 19, in <module>
    tflite_quant_model = converter.convert()
  File "/home/se/anaconda3/lib/python3.9/site-packages/tensorflow/lite/python/lite.py", line 775, in wrapper
    return self._convert_and_export_metrics(convert_func, *args, **kwargs)
  File "/home/se/anaconda3/lib/python3.9/site-packages/tensorflow/lite/python/lite.py", line 761, in _convert_and_export_metrics
    result = convert_func(self, *args, **kwargs)
  File "/home/se/anaconda3/lib/python3.9/site-packages/tensorflow/lite/python/lite.py", line 1043, in convert
    return self._optimize_tflite_model(
  File "/home/se/anaconda3/lib/python3.9/site-packages/tensorflow/lite/python/convert_phase.py", line 226, in wrapper
    raise error from None  # Re-throws the exception.
  File "/home/se/anaconda3/lib/python3.9/site-packages/tensorflow/lite/python/convert_phase.py", line 216, in wrapper
    return func(*args, **kwargs)
  File "/home/se/anaconda3/lib/python3.9/site-packages/tensorflow/lite/python/lite.py", line 721, in _optimize_tflite_model
    model = self._quantize(
  File "/home/se/anaconda3/lib/python3.9/site-packages/tensorflow/lite/python/lite.py", line 544, in _quantize
    return calibrate_quantize.calibrate_and_quantize(
  File "/home/se/anaconda3/lib/python3.9/site-packages/tensorflow/lite/python/convert_phase.py", line 226, in wrapper
    raise error from None  # Re-throws the exception.
  File "/home/se/anaconda3/lib/python3.9/site-packages/tensorflow/lite/python/convert_phase.py", line 216, in wrapper
    return func(*args, **kwargs)
  File "/home/se/anaconda3/lib/python3.9/site-packages/tensorflow/lite/python/optimize/calibrator.py", line 177, in calibrate_and_quantize
    self._feed_tensors(dataset_gen, resize_input)
  File "/home/se/anaconda3/lib/python3.9/site-packages/tensorflow/lite/python/optimize/calibrator.py", line 133, in _feed_tensors
    self._calibrator.Prepare([list(s.shape) for s in input_array])
RuntimeError: tensorflow/lite/kernels/concatenation.cc:158 t->dims->data[d] != t0->dims->data[d] (1 != 2)Node number 6 (CONCATENATION) failed to prepare.

(base) se@stu:~/yolov5$ cat  tf_testv2.py
import tensorflow as tf
import numpy as np

def representative_dataset():
    for _ in range(100):
      data = np.random.rand(1, 244, 244, 3)
      yield [data.astype(np.float32)]

converter = tf.lite.TFLiteConverter.from_saved_model("nano")
converter.target_spec.supported_types = [tf.int8]

converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_dataset
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
#converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS]
converter.inference_input_type = tf.int8  # or tf.unint8
converter.inference_output_type = tf.int8  # or tf.uint8
converter.experimental_new_quantizer = False
tflite_quant_model = converter.convert()
open('yolox_nano.tflite', 'wb').write(tflite_quant_model)

Oh, my fault.

I get the same error @Pillar1989. Did you manage to solve it?

The input should be (1, 3, 244, 244), not (1, 244, 244, 3)


Vite and WASM files

I'd love an example on how to use the package with Vite - can't get it quite right as of yet. Here's what I've tried so far:
vite.config.ts
import { defineConfig } from "vite"
import { viteStaticCopy } from "vite-plugin-static-copy"

export default defineConfig(async () => ({
  plugins: [
    // ...other plugins
    viteStaticCopy({
      targets: [
        {
          src: 'node_modules/web-tree-sitter/tree-sitter.wasm',
          dest: 'public/tree-sitter.wasm',
        },
        {
          src: 'node_modules/curlconverter/dist/tree-sitter-bash.wasm',
          dest: 'public/tree-sitter-bash.wasm',
        },
      ],
    }),
  ],
  // ...other config
}));

But I'm not sure what else I need to do. In my React code, I do the following
import * as cURL from "curlconverter"

const Try = () => {
  return (<button onClick={() => cURL.toAnsible("curl example.com")}>curlconverter</button>)
}

The errors I get when trying to it

[Error] Unhandled Promise Rejection: TypeError: Load failed
[Warning] wasm streaming compile failed: TypeError: Unexpected response MIME type. Expected 'application/wasm' (curlconverter.js, line 271)
[Warning] falling back to ArrayBuffer instantiation (curlconverter.js, line 271)
[Warning] failed to asynchronously prepare wasm: CompileError: WebAssembly.Module doesn't parse at byte 0: module doesn't start with '\0asm' (curlconverter.js, line 260)
[Warning] Aborted(CompileError: WebAssembly.Module doesn't parse at byte 0: module doesn't start with '\0asm') (curlconverter.js, line 202)
[Error] Unhandled Promise Rejection: RuntimeError: Aborted(CompileError: WebAssembly.Module doesn't parse at byte 0: module doesn't start with '\0asm'). Build with -sASSERTIONS for more info.

any help/advice would be much appreciated - thanks!

Check the Network tab of the devtools to see what file path is actually getting requested and the actual data being sent. I bet it's sending an HTML 404 page, which is obviously not a valid WASM file. Check the layout of the build directory (does Vite have something like that?) to see where the actual file is requested from. Both Wasm files need to be served from the root of your website, like localhost:8888/tree-sitter.wasm and localhost:8888/tree-sitter-bash.wasm

Check the Network tab of the devtools to see what file path is actually getting requested and the actual data being sent. I bet it's requesting localhost:8888/tree-sitter.wasm and getting an HTML 404 page, which is obviously not a valid WASM file. Check the layout of the build directory (does Vite have something like that?) to see where the actual file is placed, it looks like it'll be localhost:8888/public/tree-sitter.wasm. curlconverter needs both Wasm files to be served from the root of your website, like localhost:8888/tree-sitter.wasm and localhost:8888/tree-sitter-bash.wasm

Here's my Network tab (it's Safari, as I use it in my Tauri app):

the response is HTTP 200, but only one file is tree-sitter.wasm (and not tree-sitter-bash.wasm - is that a problem?) is requested. It seems that whatever's placed at the public directory in Vite, is successfully served at the root.

I confirmed that files placed in the public directory are successfully served at the project's root. However, in my network tab, I can see a single tree-sitter.wasm request, that's falsely HTTP 200, as it is sent to http://localhost:1420/**index.html/**tree-sitter.wasm for the reason uknown:

and (correctly) responds with my index.html file:

Do you have any idea why does it send an /index.html-prefixed request?

It's 200 because Vite and other single-page app frameworks handle the URL in JavaScript, so it just always returns index.html which contains some js that then determines that the user should be shown a 404 page after all.
To make it load /tree-sitter.wasm instead of /index.html/tree-sitter.wasm , you could fork curlconverter and then change this
https://github.com/curlconverter/curlconverter/blob/cf2aab4ce918792a2ae8ad9d689f1c4fac74becb/src/shell/webParser.ts#L7
to
await Parser.init({
  locateFile(scriptName: string, scriptDirectory: string) {
    return scriptName;
  },
});

https://github.com/tree-sitter/tree-sitter/tree/master/lib/binding_web#running-wasm-in-browser

Seems like a sane default for this package tho, doesn't it? You already require it to be at the server root, so it only makes sense to actually ignore the scriptDirectory altogether, right?

guess so.

So it turns out that tree-sitter (Emscripten-compiled Wasm in general) loads tree-sitter.wasm relative to the directory of the JavaScript file that loads it, not relative to the HTML file that includes the JavaScript that loads it. I didn't even know that was possible.
https://github.com/emscripten-core/emscripten/blob/228af1a7de1672b582e1448d4573c20c5d2a5b5a/src/shell.js#L242
So right now curlconverter requests these two URLs
require('url').fileURLToPath(new URL('./', import.meta.url)) + 'tree-sitter.wasm';
'/tree-sitter-bash.wasm';

We could change it to (as you suggest and as the documentation claims it works)
'/tree-sitter.wasm';
'/tree-sitter-bash.wasm';

or we could do what I tried to do in #649 properly
require('url').fileURLToPath(new URL('./', import.meta.url)) + 'tree-sitter.wasm';
require('url').fileURLToPath(new URL('./', import.meta.url)) + 'tree-sitter-bash.wasm';

Where are your JavaScript files being loaded from, http://localhost:1420/index.html/main.js ? or  http://localhost:1420/main.js? Or do you have multiple JavaScript files?

I've got the following in my index.html:
<!doctype html>

    <meta charset="UTF-8" />

    <title>SafeUtils</title>

Found in devtools:

and the src/main.tsx is a pretty standard React setup:
import ReactDOM from "react-dom/client";
import App from "./App";

saveWindowState(StateFlags.ALL);
restoreStateCurrent(StateFlags.ALL);

const container = document.getElementById("root") as HTMLElement

ReactDOM.createRoot(container).render(<App />);

Here's the index.html in Devtools:

As you can see, there are 2 additional Vite's


Remove Dockerfile. Have Travis install h5py

Maintaining our own Docker image was adding an unnecessary layer of complexity, and causing peculiar behavior in our continuous integration process. DockerHub does not make it very easy to see how images have changed.

Also here I'm no longer having Travis use requirements.txt with pip, since I've so far found this behavior perplexing.

Covered by PR #218


Handle LegacyFactoryFunctions in generator

Some interfaces in the Web IDL have a LegacyFactoryFunction defined in their metadata e.g. HTMLAudioElement that tells us that the factory for that interface is defined differently e.g. HTMLAudioElement here: https://html.spec.whatwg.org/multipage/media.html#the-audio-element.
The surface area of this metadata is small, affecting 3 interfaces today, all of which are elements and can be instantiated using createElement.

Copying your reply from https://github.com/flutter/flutter/issues/139125 :

This is a bug in package:web. I filed an issue here: https://github.com/dart-lang/web/issues/124. The workaround is simple for now: document.createElement('audio').
There are no custom elements involved in package:web. It lowers the constructor to the intuitive lowering new HTMLAudioElement() in JS. That JS code returns that error, since the working way is to do new Audio() instead.
Marking it closed and tracking it in the package instead.

Now is this a future compatible workaround, in the sense of, wiil the document.createElement('audio') approach continue to work once web addresses this internally in the future?

Yes, it should. createElement is a key DOM API, so it should not change even if we fix this issue.

Ah, got it. In my head, I was completely off track in what I had thought you meant.
Now that I understand what you meant, I guess my question just to confirm is, can I assume that the typecast document.createElement('audio') as HTMLAudioElement or document.createElement('audio') as AudioElement is future compatible?

I had the same issue for
  final HTMLMetaElement metaElement = HTMLMetaElement()
    ..name = 'google-signin-client_id'
    ..content = flavor.googleWebClientId;

Is this issue for all HTML elements or audio element only?

Is this issue for all HTML elements or audio element only?

The issue states that it's not all, not one, but exactly "three". The surface area is small. (HTMLMetaElement is not one of them.)

@ryanheise is correct here: this is a different bug even though it has similar results. Responded in #183.

@srujzs Based on the following description for [LegacyFactoryFunction] values,

If the [LegacyFactoryFunction] extended attribute appears on an interface, it indicates that the JavaScript global object will have a property with the specified name whose value is a function that can create objects that implement the interface. From IDN docs

Would you say the following would be a correct representation of such factory functions in dart?
HTMLAudioElement Audio(String src){
  return HTMLAudioElement()..src =src;
}

Are there any other specifics that we want to take into account if we have to add support for [LegacyFactoryFunction] in the generator?

I actually don't think this is needed anymore. These are legacy ways to construct 3 element types. We added default constructors to call document.createElement instead for these and other element types, and therefore, there's no need to call these legacy constructors. I think we can go ahead and close this as the original motivation is handled.


Merge pull request #1 from angular/master

Update from origin/master

merge from origin

merge from origin


Email history

It would be really nice to have emails history at HPI. First possible source could be google takeout mbox files. And it's also possible to add others mbox files via https://docs.python.org/3/library/mailbox.html library.
Will check and try to implement this feature.

Hey, I have this implemented in my modules, you can install them alongside this repository by following instructions here:
https://github.com/seanbreckenridge/HPI#install
Have docs for setting up mail/mbox in particular:
https://github.com/seanbreckenridge/HPI/blob/master/doc/MAIL_SETUP.md

Oh, cool, I'll check it, thanks.
Maybe there is some way to add this new features to karlicoss's HPI repo?

It's possible could merge that here in the future yeah, just for the convenience
But also it doesn't need to, HPI can install both into the single namespace package, allows for people creating their own modules/repos and combining them into HPI without having to merge all the modules here
Mbox is a recent feature and not incredibly stable yet -- also want to add some spam filtering to the module but haven't got around to that yet

@seanbreckenridge thanks for explaining. Closing this issue.

If you have any issues setting it up/suggestions for the module, feel free to create an issue there, is pretty barebones as a full 'email suite' right now


Test failures with Python 3.12 due to warnings (from pytest)

While technically the "problem" is in pytest itself, I think it'd be better if the test suite would be more resilient to warnings that don't affect the result.
========================================================= test session starts =========================================================
platform linux -- Python 3.12.0b1, pytest-7.3.1, pluggy-1.0.0
cachedir: .tox/py312/.pytest_cache
rootdir: /tmp/pytest-forked
configfile: tox.ini
plugins: forked-1.6.1.dev4+gd9d05e2
collected 10 items

testing/test_boxed.py ...xx.                                                                                [ 60%]
testing/test_xfail_behavior.py .F.F                                                                                [100%]

============================================================== FAILURES ===============================================================
______________________________________________________ test_xfail[strict xpass] _______________________________________________________

is_crashing = False, is_strict = True, testdir = <Testdir local('/tmp/pytest-of-mgorny/pytest-2/test_xfail1')>

    @pytest.mark.parametrize(
        ("is_crashing", "is_strict"),
        (
            pytest.param(True, True, id="strict xfail"),
            pytest.param(False, True, id="strict xpass"),
            pytest.param(True, False, id="non-strict xfail"),
            pytest.param(False, False, id="non-strict xpass"),
        ),
    )
    def test_xfail(is_crashing, is_strict, testdir):
        """Test xfail/xpass/strict permutations."""
        # pylint: disable=possibly-unused-variable
        sig_num = signal.SIGTERM.numerator

        test_func_body = (
            "os.kill(os.getpid(), signal.SIGTERM)" if is_crashing else "assert True"
        )

        if is_crashing:
            # marked xfailed and crashing, no matter strict or not
            expected_letter = "x"  # XFAILED
            expected_lowercase = "xfailed"
            expected_word = "XFAIL"
        elif is_strict:
            # strict and not failing as expected should cause failure
            expected_letter = "F"  # FAILED
            expected_lowercase = "failed"
            expected_word = FAILED_WORD
        elif not is_strict:
            # non-strict and not failing as expected should cause xpass
            expected_letter = "X"  # XPASS
            expected_lowercase = "xpassed"
            expected_word = "XPASS"

        session_start_title = "*==== test session starts ====*"
        loaded_pytest_plugins = "plugins: forked*"
        collected_tests_num = "collected 1 item"
        expected_progress = f"test_xfail.py {expected_letter!s}*"
        failures_title = "*==== FAILURES ====*"
        failures_test_name = "*____ test_function ____*"
        failures_test_reason = "[XPASS(strict)] The process gets terminated"
        short_test_summary_title = "*==== short test summary info ====*"
        short_test_summary = f"{expected_word!s} test_xfail.py::test_function"
        if expected_lowercase == "xpassed":
            # XPASS wouldn't have the crash message from
            # pytest-forked because the crash doesn't happen
            short_test_summary = " ".join(
                (
                    short_test_summary,
                    "The process gets terminated",
                )
            )
        reason_string = (
            f"reason: The process gets terminated; "
            f"pytest-forked reason: "
            f"*:*: running the test CRASHED with signal {sig_num:d}"
        )
        if expected_lowercase == "xfailed" and PYTEST_GTE_7_2:
            short_test_summary += " - " + reason_string
        total_summary_line = f"*==== 1 {expected_lowercase!s} in 0.*s* ====*"

        expected_lines = (
            session_start_title,
            loaded_pytest_plugins,
            collected_tests_num,
            expected_progress,
        )
        if expected_word == FAILED_WORD:
            # XPASS(strict)
            expected_lines += (
                failures_title,
                failures_test_name,
                failures_test_reason,
            )
        expected_lines += (
            short_test_summary_title,
            short_test_summary,
        )
        if expected_lowercase == "xpassed" and expected_word == FAILED_WORD:
            # XPASS(strict)
            expected_lines += ("  " + reason_string,)
        expected_lines += (total_summary_line,)

        test_module = testdir.makepyfile(
            f"""
            import os
            import signal

            import pytest

            # The current implementation emits RuntimeWarning.
            pytestmark = pytest.mark.filterwarnings('ignore:pytest-forked xfail')

            @pytest.mark.xfail(
                reason='The process gets terminated',
                strict={is_strict!s},
            )
            @pytest.mark.forked
            def test_function():
                {test_func_body!s}
            """
        )

        pytest_run_result = testdir.runpytest(test_module, "-ra")
>       pytest_run_result.stdout.fnmatch_lines(expected_lines)
E       Failed: fnmatch: '*==== test session starts ====*'
E          with: '========================================================= test session starts ========================================================='
E       nomatch: 'plugins: forked*'
E           and: 'platform linux -- Python 3.12.0b1, pytest-7.3.1, pluggy-1.0.0'
E           and: 'rootdir: /tmp/pytest-of-mgorny/pytest-2/test_xfail1'
E       fnmatch: 'plugins: forked*'
E          with: 'plugins: forked-1.6.1.dev4+gd9d05e2'
E       exact match: 'collected 1 item'
E       nomatch: 'test_xfail.py F*'
E           and: ''
E       fnmatch: 'test_xfail.py F*'
E          with: 'test_xfail.py F                                                                                [100%]'
E       nomatch: '*==== FAILURES ====*'
E           and: ''
E       fnmatch: '*==== FAILURES ====*'
E          with: '============================================================== FAILURES ==============================================================='
E       fnmatch: '*____ test_function ____*'
E          with: '____________________________________________________________ test_function ____________________________________________________________'
E       exact match: '[XPASS(strict)] The process gets terminated'
E       nomatch: '*==== short test summary info ====*'
E           and: '========================================================== warnings summary ==========================================================='
E           and: '../../../pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:927'
E           and: '  /tmp/pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:927: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead'
E           and: '    assertmsg = ast.Str("")'
E           and: ''
E           and: '../../../pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:929'
E           and: '  /tmp/pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:929: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead'
E           and: '    template = ast.BinOp(assertmsg, ast.Add(), ast.Str(explanation))'
E           and: ''
E           and: '../../../pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:817'
E           and: '  /tmp/pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:817: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead'
E           and: '    keys = [ast.Str(key) for key in current.keys()]'
E           and: ''
E           and: '../../../pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:941'
E           and: '  /tmp/pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:941: DeprecationWarning: ast.NameConstant is deprecated and will be removed in Python 3.14; use ast.Constant instead'
E           and: '    clear = ast.Assign(variables, ast.NameConstant(None))'
E           and: ''
E           and: '-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html'
E       fnmatch: '*==== short test summary info ====*'
E          with: '======================================================= short test summary info ======================================================='
E       exact match: 'FAILED test_xfail.py::test_function'
E       nomatch: '*==== 1 failed in 0.*s* ====*'
E           and: '==================================================== 1 failed, 4 warnings in 0.01s ===================================================='
E       remains unmatched: '*==== 1 failed in 0.*s* ====*'

/tmp/pytest-forked/testing/test_xfail_behavior.py:121: Failed
-------------------------------------------------------- Captured stdout call ---------------------------------------------------------
========================================================= test session starts =========================================================
platform linux -- Python 3.12.0b1, pytest-7.3.1, pluggy-1.0.0
rootdir: /tmp/pytest-of-mgorny/pytest-2/test_xfail1
plugins: forked-1.6.1.dev4+gd9d05e2
collected 1 item

test_xfail.py F                                                                                [100%]

============================================================== FAILURES ===============================================================
____________________________________________________________ test_function ____________________________________________________________
[XPASS(strict)] The process gets terminated
========================================================== warnings summary ===========================================================
../../../pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:927
  /tmp/pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:927: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
    assertmsg = ast.Str("")

../../../pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:929
  /tmp/pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:929: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
    template = ast.BinOp(assertmsg, ast.Add(), ast.Str(explanation))

../../../pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:817
  /tmp/pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:817: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
    keys = [ast.Str(key) for key in current.keys()]

../../../pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:941
  /tmp/pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:941: DeprecationWarning: ast.NameConstant is deprecated and will be removed in Python 3.14; use ast.Constant instead
    clear = ast.Assign(variables, ast.NameConstant(None))

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================================================= short test summary info =======================================================
FAILED test_xfail.py::test_function
==================================================== 1 failed, 4 warnings in 0.01s ====================================================
____________________________________________________ test_xfail[non-strict xpass] _____________________________________________________

is_crashing = False, is_strict = False, testdir = <Testdir local('/tmp/pytest-of-mgorny/pytest-2/test_xfail3')>

    @pytest.mark.parametrize(
        ("is_crashing", "is_strict"),
        (
            pytest.param(True, True, id="strict xfail"),
            pytest.param(False, True, id="strict xpass"),
            pytest.param(True, False, id="non-strict xfail"),
            pytest.param(False, False, id="non-strict xpass"),
        ),
    )
    def test_xfail(is_crashing, is_strict, testdir):
        """Test xfail/xpass/strict permutations."""
        # pylint: disable=possibly-unused-variable
        sig_num = signal.SIGTERM.numerator

        test_func_body = (
            "os.kill(os.getpid(), signal.SIGTERM)" if is_crashing else "assert True"
        )

        if is_crashing:
            # marked xfailed and crashing, no matter strict or not
            expected_letter = "x"  # XFAILED
            expected_lowercase = "xfailed"
            expected_word = "XFAIL"
        elif is_strict:
            # strict and not failing as expected should cause failure
            expected_letter = "F"  # FAILED
            expected_lowercase = "failed"
            expected_word = FAILED_WORD
        elif not is_strict:
            # non-strict and not failing as expected should cause xpass
            expected_letter = "X"  # XPASS
            expected_lowercase = "xpassed"
            expected_word = "XPASS"

        session_start_title = "*==== test session starts ====*"
        loaded_pytest_plugins = "plugins: forked*"
        collected_tests_num = "collected 1 item"
        expected_progress = f"test_xfail.py {expected_letter!s}*"
        failures_title = "*==== FAILURES ====*"
        failures_test_name = "*____ test_function ____*"
        failures_test_reason = "[XPASS(strict)] The process gets terminated"
        short_test_summary_title = "*==== short test summary info ====*"
        short_test_summary = f"{expected_word!s} test_xfail.py::test_function"
        if expected_lowercase == "xpassed":
            # XPASS wouldn't have the crash message from
            # pytest-forked because the crash doesn't happen
            short_test_summary = " ".join(
                (
                    short_test_summary,
                    "The process gets terminated",
                )
            )
        reason_string = (
            f"reason: The process gets terminated; "
            f"pytest-forked reason: "
            f"*:*: running the test CRASHED with signal {sig_num:d}"
        )
        if expected_lowercase == "xfailed" and PYTEST_GTE_7_2:
            short_test_summary += " - " + reason_string
        total_summary_line = f"*==== 1 {expected_lowercase!s} in 0.*s* ====*"

        expected_lines = (
            session_start_title,
            loaded_pytest_plugins,
            collected_tests_num,
            expected_progress,
        )
        if expected_word == FAILED_WORD:
            # XPASS(strict)
            expected_lines += (
                failures_title,
                failures_test_name,
                failures_test_reason,
            )
        expected_lines += (
            short_test_summary_title,
            short_test_summary,
        )
        if expected_lowercase == "xpassed" and expected_word == FAILED_WORD:
            # XPASS(strict)
            expected_lines += ("  " + reason_string,)
        expected_lines += (total_summary_line,)

        test_module = testdir.makepyfile(
            f"""
            import os
            import signal

            import pytest

            # The current implementation emits RuntimeWarning.
            pytestmark = pytest.mark.filterwarnings('ignore:pytest-forked xfail')

            @pytest.mark.xfail(
                reason='The process gets terminated',
                strict={is_strict!s},
            )
            @pytest.mark.forked
            def test_function():
                {test_func_body!s}
            """
        )

        pytest_run_result = testdir.runpytest(test_module, "-ra")
>       pytest_run_result.stdout.fnmatch_lines(expected_lines)
E       Failed: fnmatch: '*==== test session starts ====*'
E          with: '========================================================= test session starts ========================================================='
E       nomatch: 'plugins: forked*'
E           and: 'platform linux -- Python 3.12.0b1, pytest-7.3.1, pluggy-1.0.0'
E           and: 'rootdir: /tmp/pytest-of-mgorny/pytest-2/test_xfail3'
E       fnmatch: 'plugins: forked*'
E          with: 'plugins: forked-1.6.1.dev4+gd9d05e2'
E       exact match: 'collected 1 item'
E       nomatch: 'test_xfail.py X*'
E           and: ''
E       fnmatch: 'test_xfail.py X*'
E          with: 'test_xfail.py X                                                                                [100%]'
E       nomatch: '*==== short test summary info ====*'
E           and: ''
E           and: '========================================================== warnings summary ==========================================================='
E           and: '../../../pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:927'
E           and: '  /tmp/pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:927: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead'
E           and: '    assertmsg = ast.Str("")'
E           and: ''
E           and: '../../../pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:929'
E           and: '  /tmp/pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:929: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead'
E           and: '    template = ast.BinOp(assertmsg, ast.Add(), ast.Str(explanation))'
E           and: ''
E           and: '../../../pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:817'
E           and: '  /tmp/pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:817: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead'
E           and: '    keys = [ast.Str(key) for key in current.keys()]'
E           and: ''
E           and: '../../../pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:941'
E           and: '  /tmp/pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:941: DeprecationWarning: ast.NameConstant is deprecated and will be removed in Python 3.14; use ast.Constant instead'
E           and: '    clear = ast.Assign(variables, ast.NameConstant(None))'
E           and: ''
E           and: '-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html'
E       fnmatch: '*==== short test summary info ====*'
E          with: '======================================================= short test summary info ======================================================='
E       exact match: 'XPASS test_xfail.py::test_function The process gets terminated'
E       nomatch: '*==== 1 xpassed in 0.*s* ====*'
E           and: '=================================================== 1 xpassed, 4 warnings in 0.01s ===================================================='
E       remains unmatched: '*==== 1 xpassed in 0.*s* ====*'

/tmp/pytest-forked/testing/test_xfail_behavior.py:121: Failed
-------------------------------------------------------- Captured stdout call ---------------------------------------------------------
========================================================= test session starts =========================================================
platform linux -- Python 3.12.0b1, pytest-7.3.1, pluggy-1.0.0
rootdir: /tmp/pytest-of-mgorny/pytest-2/test_xfail3
plugins: forked-1.6.1.dev4+gd9d05e2
collected 1 item

test_xfail.py X                                                                                [100%]

========================================================== warnings summary ===========================================================
../../../pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:927
  /tmp/pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:927: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
    assertmsg = ast.Str("")

../../../pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:929
  /tmp/pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:929: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
    template = ast.BinOp(assertmsg, ast.Add(), ast.Str(explanation))

../../../pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:817
  /tmp/pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:817: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
    keys = [ast.Str(key) for key in current.keys()]

../../../pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:941
  /tmp/pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:941: DeprecationWarning: ast.NameConstant is deprecated and will be removed in Python 3.14; use ast.Constant instead
    clear = ast.Assign(variables, ast.NameConstant(None))

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================================================= short test summary info =======================================================
XPASS test_xfail.py::test_function The process gets terminated
=================================================== 1 xpassed, 4 warnings in 0.01s ====================================================
========================================================== warnings summary ===========================================================
.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:683
.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:683
.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:683
  /tmp/pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:683: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
    and isinstance(item.value, ast.Str)

.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:685
.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:685
  /tmp/pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:685: DeprecationWarning: Attribute s is deprecated and will be removed in Python 3.14; use value instead
    doc = item.value.s

.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:965
.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:965
.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:965
.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:965
.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:965
  /tmp/pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:965: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
    inlocs = ast.Compare(ast.Str(name.id), [ast.In()], [locs])

.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:968
.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:968
.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:968
.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:968
.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:968
  /tmp/pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:968: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
    expr = ast.IfExp(test, self.display(name), ast.Str(name.id))

.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:1102
.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:1102
  /tmp/pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:1102: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
    syms.append(ast.Str(sym))

.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:1104
.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:1104
  /tmp/pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:1104: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
    expls.append(ast.Str(expl))

.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:817
.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:817
.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:817
.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:817
.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:817
.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:817
.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:817
.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:817
.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:817
  /tmp/pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:817: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
    keys = [ast.Str(key) for key in current.keys()]

.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:927
.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:927
  /tmp/pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:927: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
    assertmsg = ast.Str("")

.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:929
.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:929
.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:929
  /tmp/pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:929: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
    template = ast.BinOp(assertmsg, ast.Add(), ast.Str(explanation))

.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:941
.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:941
.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:941
.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:941
  /tmp/pytest-forked/.tox/py312/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:941: DeprecationWarning: ast.NameConstant is deprecated and will be removed in Python 3.14; use ast.Constant instead
    clear = ast.Assign(variables, ast.NameConstant(None))

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================================================= short test summary info =======================================================
FAILED testing/test_xfail_behavior.py::test_xfail[strict xpass] - Failed: fnmatch: '*==== test session starts ====*'
FAILED testing/test_xfail_behavior.py::test_xfail[non-strict xpass] - Failed: fnmatch: '*==== test session starts ====*'
XFAIL testing/test_boxed.py::test_functional_boxed_capturing[sys] - capture cleanup needed
XFAIL testing/test_boxed.py::test_functional_boxed_capturing[fd] - capture cleanup needed
========================================= 2 failed, 6 passed, 2 xfailed, 37 warnings in 0.66s =========================================

Hmm, it seems that after patching pytest, it's now failing due to a problem in py:
E           and: 'test_xfail.py::test_function'
E           and: '  /usr/lib/python3.12/site-packages/py/_process/forkedfunc.py:45: DeprecationWarning: This process (pid=66) is multi-threaded, use of fork() may lead to deadlocks in the child.'
E           and: '    pid = os.fork()'

The warnings will be fixed by https://github.com/pytest-dev/pytest/pull/10894.

The warnings will be fixed by pytest-dev/pytest#10894.

This doesn't fix the warning from py.process.ForkedFunc().


Json alphabetization

This PR has two things:

Alphabetizes all the json properties.  I've validated that the json is exactly the same as before, with one tiny change:  Profundity (AI) had the same id as regular Profundity, so a new id was created for the AI version.  The gempId is still the same.
Added json schema for the set and card json files and updated the validation steps.  From what I can see, there isn't a way to enforce alphabetization of fields with json schema, unfortunately.  This should at least enforce the correct types and formatting.

FYI:  I used https://github.com/josephburnett/jd to validate the json files are all the same before/after the alphabetization (minus the one change I mentioned).

I would add mention of this new rule to the README (in addition to updating the example JSON there to also comply).

I updated the PR with improvements to the README.

rebased


wget directions returns 404 Not Found

In the readme the installation for build versions shows
wget https://github.com/vortex314/serial2mqtt/raw/master/Debug/serial2mqtt.arch.zip
I got a 404 Not Found.
Then changed it to https://github.com/vortex314/serial2mqtt/raw/master/build/serial2mqtt.arch.zip
and it works.  Think your Readme needs updated.

no more builds, please rebuild on target platform with cmake


ToOne and ToMany should allow for having just one of data, links or meta fields

Per the spec:

A “relationship object” MUST contain at least one of the following:

links: a links object containing at least one of the following:

self: a link for the relationship itself (a “relationship link”). This link allows the client to directly manipulate the relationship. > For example, removing an author through an article’s relationship URL would disconnect the person from the article without deleting the people resource itself. When fetched successfully, this link returns the linkage for the related resources as its primary data. (See Fetching Relationships.)
related: a related resource link

data: resource linkage
meta: a meta object that contains non-standard meta-information about the relationship.

"at least one of the following" does mean that it can have just one of data, links and meta.

I must apologise - I completely misread this issue when you opened it! I hadn't taken in it was in the laravel-json-api/spec repo.
So the part of the spec that is relevant here says this:

[When creating a resource] If a relationship is provided in the relationships member of the resource object, its value MUST be a relationship object with a data member. The value of this key represents the linkage the new resource is to have.
https://jsonapi.org/format/#crud-creating

And for updating:

If a relationship is provided in the relationships member of a resource object in a PATCH request, its value MUST be a relationship object with a data member. The relationship’s value will be replaced with the value specified in this member.
https://jsonapi.org/format/#crud-updating

The section you've quoted is about the serialization of a relationship within a resource object, i.e. when a server is serializing a resource.

 Guess now I'll have to see how I get my client lib to not serialize relationships for PATCH/POST when there is no data available (but links is)...