Add Dotnetcore support

.Net Core Support
We have long wanted to add support for .Net core and earlier this year IKVM was finally "revived" to have support for .net core. At first, I gave up because ikvmc.exe didn't seem to work at all (and still does not for our use case). But @dylanlangston created a proof of concept using IKVMReference and msbuild to extract dotnet assemblies from the tika .jar file.
Nugets

TikaOnDotNet nuget is now multi-targeting .Net framework 4.6.2 and .Net Core 3.1.
TikaOnDotNet.TextExtraction is now multi-targeting .Net framework 4.6.2 and .Net Core 3.1.

Do we need to target .Net 6?
Tests
All tests but one are passing. For some reason parsing our test .rtf file throws a java UnsatisfiedLinkError exception:
    TikaOnDotNet.TextExtraction.TextExtractionException : Extraction of text from the file 'files/Tika.rtf' failed.
  ----> TikaOnDotNet.TextExtraction.TextExtractionException : Extraction failed.
  ----> java.lang.UnsatisfiedLinkError : sun/java2d/Disposer.initIDs()V

If anyone has an idea what this might be related to please help!😖
Build / Deployment Automation
We are going to move away from Packet and the F# build automation to use GitHub actions to build/test and deploy nugets. I'd like updating the version of Tika to be a simple update of a version file. We are close with what @dylanlangston started for us.
Tests are "mostly" passing with plain msbuild and me hammering out this at the command line to produce a tika nuget.
dotnet pack ./src/TikaOnDotnet/TikaOnDotnet.csproj -p:NuspecFile=package.nuspec -p:NuSpecBasePath=. --configuration=Release

Nuget Packaging
The nuget has been updated to better represent the license, readme location, project url, and finally I've added an icon.
Icon
I spent 30 seconds creating an icon to prettify the Nuget presentation. I anyone would like to improve on what I started. Please do. I am not a design person.

TBD:
TikaOnDotnet.TextExtraction should use nuspec or jave csproj properties to make the listing as nice as TikaOnDotnet

Hey I'm not a designer, but if you like it I can add in a commit to this branch.

Thank you!

Hey KevM,

Do we need to target .Net 6?

Yes we do need Tika on .Net to target .Net 6 at my organization for a couple of projects,
When do you think we could expect a new release?

There is one failing test for rtf files. No idea why it is not working. I was going to work on getting a pre-release out and then let people try it out for a bit before committing to a release.
Note: I’d be willing to take a short contract to get this release out quicker. I am self employed.
On Wed, Sep 14, 2022, at 7:01 AM, Smiechowski Nathanael wrote:

Hey KevM,

Do we need to target .Net 6?

Yes we do need Tika on .Net to target .Net 6 at my organization for a couple of projects,
When do you think we could expect a new release?
—
Reply to this email directly, view it on GitHub https://github.com/KevM/tikaondotnet/pull/152#issuecomment-1246661398, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAMHPSFOCSWRMVEYDO473V6G5DLANCNFSM6AAAAAAQCWG3ME.
You are receiving this because you authored the thread.Message ID: @.***>

There is one failing test for rtf files. No idea why it is not working. I was going to work on getting a pre-release out and then let people try it out for a bit before committing to a release. Note: I’d be willing to take a short contract to get this release out quicker. I am self employed.
…
On Wed, Sep 14, 2022, at 7:01 AM, Smiechowski Nathanael wrote: Hey KevM, > Do we need to target .Net 6? > Yes we do need Tika on .Net to target .Net 6 at my organization for a couple of projects, When do you think we could expect a new release? — Reply to this email directly, view it on GitHub <#152 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAMHPSFOCSWRMVEYDO473V6G5DLANCNFSM6AAAAAAQCWG3ME. You are receiving this because you authored the thread.Message ID: @.***>

I was able to pretty easily target net6.0 throughout and pass tests (except the RTF one) without changing any other dependency versions by incrementing version numbers and adding net6.0 to the .csproj targets.
Regarding the RTF test - it seems to pass when the RTF file doesn't contain an image. Without digging into the Java side of things I can't provide much feedback beyond that.
If you'd like, I can submit a PR for the net6 support but that'll take a bit of approval on my end as I'm using this for an internal project.

Hey ya'll. I fell into this thread while following links blindly. I revived the IKVM project.
To get Core out, and because nobody really wanted to fix it, we didn't pay any attention to AWT. So, no AWT in IKVM. My guess is this is killing your attempted usage of Java2D. I don't really know though, since I didn't do any more investigation yet besides read this thread.
The previous AWT default toolkit was IKVM.AWT.WinForms. An attempt to map the AWT stuff to WinForms. As ya'll know, WinForms is quite different in Core. And it's not cross platform anyways. So we just didn't get it building, and probably aren't going to spend any time on it.
Instead though, you can probably configure IKVM to run in headless mode, just as you would configure OpenJDK to do so. Some System property you can set.
8.2.2 will end up with headless mode enabled by default.
Somebody try that.

Also, while I'm here, I want to make ya'll aware of IKVM.Maven.Sdk.
This is a new strategy for Java-Libraries-on-DotNet.
Instead of actually distributing cross compiled JAR -> DLL files on NuGet, which is error prone, as you don't own the assembly name or code, we allow users to directly add references to Maven packages in their C# projects. That way only one authoritative source for Java libraries exist: Maven.
We also embed some fancy information inside produced NuGet packages describing these references to Maven. So, say you write some .NET code that uses a library from Maven. And then you Pack that. Your produced NuGet file actually has a .pom file embedded into it. When somebody installs your NuGet package, and builds their own library, IKVM.Maven.Sdk downloads the Maven artifacts and cross compiles them on the fly.
It kind of obsoletes projects like TikaOnDotNet, unless you guys provide some functionality beyond the Java library. Like extension methods, etc.

Also, while I'm here, I want to make ya'll aware of IKVM.Maven.Sdk.
This is a new strategy for Java-Libraries-on-DotNet.
Instead of actually distributing cross compiled JAR -> DLL files on NuGet, which is error prone, as you don't own the assembly name or code, we allow users to directly add references to Maven packages in their C# projects. That way only one authoritative source for Java libraries exist: Maven. And the author of the actual product owns those artifacts. And the end developer is the one downloading and doing the conversion, and no middle man is redistributing licensed code.
We also embed some fancy information inside produced NuGet packages describing these references to Maven. So, say you write some .NET code that uses a library from Maven. And then you Pack that. Your produced NuGet file actually has a .pom file embedded into it. When somebody installs your NuGet package, and builds their own library, IKVM.Maven.Sdk downloads the Maven artifacts and cross compiles them on the fly.
It kind of obsoletes projects like TikaOnDotNet, unless you guys provide some functionality beyond the Java library. Like extension methods, etc.

This is great! However, it's building extremely slow for me. Is that expected for IKVM.Maven.Sdk? Any recommendations for how the Maven build process can be sped up?

Depends. The first build is definitely going to be a thing. Likely it has to download two dozen jars and convert them all. But that information is cached for subsequent builds.
Can you described what it looks like it's doing?

Also, while I'm here, I want to make ya'll aware of IKVM.Maven.Sdk.
It kind of obsoletes projects like TikaOnDotNet, unless you guys provide some functionality beyond the Java library. Like extension methods, etc.

Thanks for the information and suggestion of IKVM.Maven.SDK.
Unfortunately upon digging deeper I realized the IKVM Nuget package is licensed under GPL and as such won't work for me.
In addition this project likely needs to have a change of license to accommodate the requirements of GPL.

and as near as I can tell, the Java artifacts will be downloaded and built every single time

Nope. The dependency graph is cached until it's changed.

After searching on Google for "how to speed up maven builds", one of the suggestions is to build the artifacts

We don't build artifacts.
I need to know where in the process you are experiencing a slow down. What is the output at when it's slow?

On tiki-app, my understanding is that's not a library you're actually supposed to use as a dependency, but a JAR file with all of the dependencies embedded into it, and a main entry point, for running it as an app. Like, all the logging stuff is copied into it. Which would break trying to use tika-app along with other Java libraries that use the same logging libraries.
Like imagine if a user tried to use both tika-app and also, say, I don't know, Apache Foobar. And both depend on commons-logging. The tiki-app JAR has commons-logging copied into it. While Apache Foobar uses the version from commons-logging. They'd be be using different classes and assemblies, and configuration for one wouldn't work right for the other. Which is really the reason to favor using a single source in the first place.
There is probably documentation about how Tika users in Java make use of tika-core and the parsers properly.

Yeah, from the documentation:
tika-app/target/tika-app-0.7.jar
Tika application. Combines the above libraries and all the external parser libraries into a single runnable jar with a GUI and a command line interface.
They then go into details about which packages you're supposed to add for development purposes.

The first thing I tried was tika-core with tika-parsers-standard-package (as suggested here [https://tika.apache.org/2.5.0/gettingstarted.html]). I tried tika-app for a comparison out of curiosity mostly. In both cases, building the solution or project is painfully slow, with a couple minutes being spent at "Build started..." and a couple more minutes at "1>------ Build started: Project: IKVM_Testing, Configuration: Debug Any CPU ------" in the Output window.
I appreciate you taking a look at it for me, but this may be a case where nothing can be done about it.

Heh. Yeah. Okay, I got it reproduced on my end. Tika, with all the parsers, has 87 different dependencies. That's 87 JARs that need to be downloaded and individually converted to assemblies.
It looks like the holdup on subsequent builds is checking the cache itself. The cache is organized by hash of the transpiler information per-JAR. For instance, a JAR built with 20 different dependencies, is cached as long as the 20 dependencies themselves are cached, etc. Because any change in the graph could produce different results.
It's just taking a long time trying to even figure out if they've even changed. Let alone dealing with them if it has.
There are problem some optimizations I can put in here. Will look.

So it sounds like there is still a benefit of our project doing once what people would otherwise need to do on every build.
On Mon, Oct 24, 2022, at 1:54 PM, Jerome Haltom wrote:

Heh. Yeah. Okay, I got it reproduced on my end. Tika, with all the parsers, has 87 different dependencies. That's 87 JARs that need to be downloaded and individually converted to assemblies.
It looks like the holdup on subsequent builds is checking the cache itself. The cache is organized by hash of the transpiler information per-JAR. For instance, a JAR built with 20 different dependencies, is cached as long as the 20 dependencies themselves are cached, etc. Because any change in the graph could produce different results.
It's just taking a long time trying to even figure out if they've even changed. Let alone dealing with them if it has.
There are problem some optimizations I can put in here. Will look.
—
Reply to this email directly, view it on GitHub https://github.com/KevM/tikaondotnet/pull/152#issuecomment-1289458331, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAMHIU3IEHOSXIXZXUSHDWE3LO3ANCNFSM6AAAAAAQCWG3ME.
You are receiving this because you authored the thread.Message ID: @.***>

Sure. For the next couple hours or so, maybe. Heh.

IKVM 8.2.3 much improves the cache lookup speed.

hi,any news?

@gomep342 , I was having problems getting the Tika core to find the parsers (probably due to missing CLASSPATH), so I just built everything into one dll targeting .NET 6, and that works for me. I did not make a pull request as my solution felt like a one-off for my own situation.

Hey, this is a great project! I was trying to add some word document processing to my latest .NET program and came across this project (which is so great but unfortunately doesn't support .NET Core and I found it difficult to get this pull request branch to compile and work in a project) and GroupDocs.Parser (unfortunately a commercial piece of software with a very limited trial, not very useful for a  project I was only doing for fun, what a party pooper)
I noticed IKVM has been revived recently too and I threw together a small proof of concept that fits my needs, where I am able to use IKVM and tika to parse doc, docx, pdf files - https://github.com/souramoo/TikaOnDotNet - along with examples on how to use it in c#, for anyone who needs this functionality until this pull request gets merged!
I initially tried using IKVM.Maven.Sdk as suggested above, but actually this optimised a bit too much, leaving out the office and OOXml parsers, so I basically just made a quick java app that drags in these dependencies into a main function and then adding a reference to this jar in my csproj file (along with the IKVM dependency) got everything working :)

@souramoo
On the references. You need to include them exactly how you would in Maven. If, for example, Maven lists them as optional dependencies, MavenReference won't pull them in. However, if Maven does list them as optional dependencies, you can add them as MavenReferences, and they'll be ordered correctly.
If they're optional references but upstream forgot to actually add them as optional references, though, a bug should be filed upstream.

@wasabii thanks for the advice and great job on everything with ikvm-revived - it's very exciting stuff!
I was including both the tika-core tika-parsers-standard-package using MavenReference, but for some reason intellisense was pointing out that org.apache.tika.parsers.microsoft.OfficeParser was not available (despite being present in the jar file, I opened it up and checked!) - and despite other parsers such as AutodetectParser being available.
On top of that, AutodetectParser does some weird stuff to detect which Parser classes are included in the classpath which doesn't seem to quite work in IKVM so I had to specify manually (by building a quick and dirty function based on file extension).
By making another jar that drags in the parsers I want in the main function I think I convinced IKVM not to optimise the parser classes away (presumably because the main classes in the original package jar did not use the OfficeParser or OOXMLParser class directly)

Well, it'd be good to resolve some of those issues. The goal is it to work exactly like Maven would, as possible.
For instance, IkvmReference defaults to the "app domain assembly class loader" feature, where the static assemblies "believe" that they live inside a ClassLoader where their direct references are available first, followed by any other assemblies loaded into the current app domain (or available in the DependencyContext on Core).
So, if they do stuff like read resources (classLoader.getResource('')), it should scan properly: first directly referenced assemblies; second the entire loaded app domain.

@Arextion I don't see any remaining issues for IKVM on this thread.

@Arextion I don't see any remaining issues for IKVM on this thread that prevent Tika from running. Is there any I'm not aware of? It's been months since I tried.

I haven't tried running this, i was just curious if this PR would soon be merged, or there would be a release to try out.

Just tried. I don't know really how Tika works, but adding a MavenReference:
        <MavenReference Include="org.apache.tika:tika-core" Version="2.8.0" />

Resulted in Tika and it's dependencies being compiled, added, and seeming to work. Though I didn't get too deep into it.

@Arextion, using MavenReference is something you can just do in your own project.

@KevM Any news on this?

Sounds like people are getting it working using native features of ikvm <MavenReference .../>. So. I'd love to get a proof of concept from someone or better yet a PR I can take action on to demonstrate how to do this.

@KevM Any news on this?

Sounds like people are getting it working using native features of ikvm <MavenReference .../>. So. I'd love to get a proof of concept from someone or better yet a PR I can take action on to demonstrate how to do this.

YES please.

@KevM Any news on this?

Sounds like people are getting it working using native features of ikvm <MavenReference .../>. So. I'd love to get a proof of concept from someone or better yet a PR I can take action on to demonstrate how to do this.

I think this is what I did in my proof of concept repo at https://github.com/souramoo/TikaOnDotNet :) (which despite its name does not use anything from this project and directly uses maven references)

@souramoo  Looks like you used IkvmReference. So you have a Tika .JAR file sitting in your source repo, and you use IkvmReference to refer to that. So, similar. MavenReference just fronts all that for you by automatically managing artifact dependencies from Maven.

@KevM @wasabii I just tried with a .Net 6 project with MavenReference using:
<MavenReference Include="org.apache.tika:tika-core" Version="2.8.0" />
<MavenReference Include="org.apache.tika:tika-parsers-standard-package" Version="2.8.0" />

But it won't load any parsers it seems. I'm getting an empty result.
BodyContentHandler handler = new BodyContentHandler();
Parser parser = new AutoDetectParser();
Metadata metadata = new Metadata();
ParseContext context = new ParseContext();

parser.parse(stream, handler, metadata, context);

string text = handler.toString();

Any clue why?

@KevM @wasabii I just tried with a .Net 6 project with MavenReference using:
<MavenReference Include="org.apache.tika:tika-core" Version="2.8.0" />
<MavenReference Include="org.apache.tika:tika-parsers-standard-package" Version="2.8.0" />

But it won't load any parsers it seems. I'm getting an empty result.
using var fs = new FileStream(filePath, FileMode.Open);
using var stream = new InputStreamWrapper(fs);

BodyContentHandler handler = new BodyContentHandler();
Parser parser = new AutoDetectParser();
Metadata metadata = new Metadata();
ParseContext context = new ParseContext();

parser.parse(stream, handler, metadata, context);

string text = handler.toString();

Any clue why?

I got the same result - it's why I had to make a wrapper piece of code (https://github.com/souramoo/TikaOnDotNet/blob/main/java_src/src/main/java/Main.java) to generate a jar file containing all the dependencies and prevent the parsers being optimised away

Nothing is optimized away. At a minimum you've got to know what Tika expects, though. I don't, since I don't know Tika. How does it locate parsers?

Nothing is optimized away. At a minimum you've got to know what Tika expects, though. I don't, since I don't know Tika. How does it locate parsers?

I got the same error when trying to load a parser directly. It just never appeared in the namespace and I got a class not found error, as if the standard parsers package was never loaded.
So creating a new OfficeParser instance in c# was generating a class not found exception for example. This went away when I combined everything into a single jar to load into ikvm

So, I don't know how Tika is laided out. But, if the upstream authors have published a set of artifacts, and described those dependencies correctly, MavenReference will use them. Except for POM bundles. Those don't work.
If the upstream authors have missing dependencies in their packages, it won't, and you'd have to manually specify those. And open a bug with them, preferably.
Here's what I added:

        <MavenReference Include="org.apache.tika:tika-core" Version="2.8.0" />
        <MavenReference Include="org.apache.tika:tika-serialization" Version="2.8.0" />
        <MavenReference Include="org.apache.tika:tika-parsers-standard-package" Version="2.8.0" />
        <MavenReference Include="org.apache.tika:tika-parser-zip-commons" Version="2.8.0" />
        <MavenReference Include="org.apache.tika:tika-parser-text-module" Version="2.8.0" />
        <MavenReference Include="org.apache.tika:tika-parser-pdf-module" Version="2.8.0" />
        <MavenReference Include="org.apache.tika:tika-parser-image-module" Version="2.8.0" />
        <MavenReference Include="org.apache.tika:tika-parser-xml-module" Version="2.8.0" />
        <MavenReference Include="org.apache.tika:tika-parser-microsoft-module" Version="2.8.0" />

Then it worked.

So, advice here:
Yes, you can build your own JAR files. This works fine with IKVM as it stands today. Probably won't work when we eventually get JDK9 support (since JAR files can be named modules in JDK9+).
You should prefer to look over the dependencies for the library you want to use, and understand the parts, and add a MavenReference to only those parts you actually need. Same as any Java user would do. Same as you'd do for NuGet packages.
You should NOT publish resulting IKVM assemblies to NuGet. Please. Ever. There's a big notice at the end of the IKVM README.md file for why not.

I'd love it if someone wrote up a Readme with all of this knowledge so that others can leverage it. It sounds like this project needs to go on hiatus if y'all get things figured out. This is fine with me but I'd like to update the docs to educate people on the correct way to IKVM in the modern era.

In response to an email sent specifically by Erik Gavriluk, I have uncovered two issues in IKVM.Maven.SDK which could impact ya'll. Both of which have patches incoming.

Transitive dependencies between projects was broken in 1.5.0. This means if you have a MavenReference in ProjectB, referenced by ProjectA, that dependency wasn't making it over. This was trivial to work around by just specifying the MavenReference in both projects. But a hotfix is being published shortly that solves it.

Incorrectly culling all but one unified artifact version. This showed up in a few places with slf4j-api. If LibraryA referenced slf4j-api:2.6.0, and LibraryB referenced slf4j-api:2.7.0, only the first reference to slf4j-api was preserved and unified (to 2.7.0). So LibraryB would compile without the reference added. This will be fixed in the same hotfix.

There are two things ya'll need to be fully aware of when using IKVM.Maven.Sdk: we are just a wrapper for Maven. If the author of a library published something broken in Maven, it's going to break IKVM.Maven.Sdk. And it might break it slightly differently than it would in Java. One common example is underspecified dependencies. IKVM.Maven.Sdk relies on dependencies specified in Maven being correct, so that we can generate assemblies which properly reference each other. But, this might not break Java users, as Java JAR files have no actual dependencies: only the classes might depend on other classes, and this is only discoverable at runtime if the code is accessed. For instance, if an author forgets to depend on like, commons-logging, but nobody ever runs the code path that needs commons-logging, it'll work just fine in Java. It will also work if the users end up having commons-logging for some other reason, like they added it explicitely, or are using some other library which depends on it. In those cases, on Java, the .jar files will be added to the CLASSPATH, and it'll work fine. But, IKVM needs that information to generate assemblies.
In this case, the only true fix is to report the problem to the upstream authors of the library you are using, and have them properly fix their dependencies.
Second, assembly name generation. IKVM.Maven.SDK replies on the "automatic module name" specification of JDK9+ in order to choose assembly names. But, the situation with modules in Java is a bit weird. First, they don't exist in JDK8. Second, even in JDK9+ they are "sort of optional". That is, you can load a JAR file by specifying it on the MODULEPATH, or by adding the JAR to the CLASSPATH. This opens the situation where an upstream author may provide invalid module information: but nobody notices because all the users are using the CLASSPATH.
I have discovered at least one issue of this in Tika: tika-parser-crypto-module-2.8.0.jar.
Notice the file name is tika-parser-crypto-module-2.8.0.jar. However, if you open the JAR, and look in META-INF/MANIFEST.MF at the Automatic-Module-Name line, you'll notice the value is org.apache.tika.parser.code. This value is incorrect. The crypto-module JAR has a module-name for the parser code JAR. That's wrong. Each JAR file should have it's own unique module name.
As a consequence, IKVM attempts to name the assembly for tika-parser-crypto-module as org.apache.tika.parser.code.dll. But, it also attempts to name the assembly for org.apache.tika.parser.code to org.apache.tika.parser.code.dll. Resulting in two assemblies with the same name. IKVM then adds a reference to both to your .NET project. Except they have the same name. So they get copied into bin/ and clobber each other.
This is a bug in Tika upstream. It needs to be reported to Tika upstream and fixed there. There's not much I can do about it.

And this is how you do that:
https://issues.apache.org/jira/browse/TIKA-4061

Thank you @wasabii that info dump is sure to be helpful to many people moving forward. 🌬 ⛵
That said. How do people want to proceed? This project doesn't have a future it sounds like. How do we best encapsulate the learnings we've made here to help the .Net community of Tika users?

@KevM Are the custom .net classes something you want to continue to provide?

Custom Tika Wrapper
We currently do this to provide a good out of box experience for new users. My guess is that most people use the wrapper and if it works that is all they'll do. Others roll their own.
Question for the community what do you think we should do
It sounds like IKVM + Maven support gets you 90% of the way there. We could still provide a Tika text extraction wrapper but I'm honestly not sure what people would like or how best to support the way they use Tika.

Disclaimer:  I am not a Tika user. I barely knew what it was before an hour ago. But I just spent the better part of a day working through getting a sample project submitted to me by Erik Gavriluk. FYI, it worked.
If the code in the text extractor package is still useful for users, it can continue to be published as a NuGet package. That NuGet package can reference the Maven packages it requires. I quite enjoy doing this: having a NuGet package that depends on a Maven artifact, and having the NuGet package provide extension methods and stuff for working with Java classes in Maven in a more .NET-ish way.
Tika has a large library of parsers. That large library of parsers depends on an even larger set of other Maven artifacts. For instance, org.apache.tika:tika-parser-pdf-module implements it's PDF parsing using PDFBox. So it depends on PDF box.  Those graphics libraries depend on other libraries. For parsing SVGs, and BMPs, and JPGs. The text parser depends on CSV libraries. The HTML parser depends on many things. Adding half of the parsers I found available in Tika to a project produced about 100 different assemblies.
Users shouldn't be forced to include all of those unless they opt in. So, I would not have the TikaDotNet.TextExtractor package depend on the parsers. It would only depend on the core. And the user can add in whatever parser packages he happens to need.
Now, this isn't EXACTLY Tika related, but it is a place where value can be added: all of those 100s of packages ultimately have support for logging. These libraries log to java.util.logging. Some log to log4j. Some log to SLF4J. Some log to commons-logging. There are others. As described at https://cwiki.apache.org/confluence/display/TIKA/Logging:

Apache Tika include a lot of Apache and thirdparty libraries that have different approach to logging. Tika use slf4j-api as logging API and Apache Log4j 2.x as an implementation for modules that require it.

Basically, to get logging out of each aspect of Tika, you need to configure a dozen differnet logging libraries, independently.
Each of these logging libraries could use .NET helpers. For instance, a package that configures log4j to forward messages to Microsoft.Extensions.Logger.  A package that configures slf4j to log to Microsoft.Extensions.Logging. One for commons. Lots of little packages, useful beyond Tika (for anybody using slf4j for instance), to hook it up to Microsoft logging.
In anticipation of this, I started https://github.com/ikvmnet/ikvm-logging, who's goal it will be to write little bridges forwarding Java logger implementations into .NET.
There are probably other similar utilities like this. Simple bits of code that can make integrating Java code with .NET easier.

Tika upstream fixed the name bug.
https://issues.apache.org/jira/plugins/servlet/mobile#issue/TIKA-4061

MSBuild Kristen Renner

Very new to Tika, and trying to use it in .Net Core, and, following many links, ended up here!
Trying to use Tika to extract text on large quantities of files, so ideally would like to get AutoDetectParser working. Following @wasabii 's advice (thanks for your efforts!) and copying and pasting the MavenReferences you posted here, I'm able to Parse files, but only if I try something like @souramoo 's if docx then OOXMLParser type stuff. I don't want to have to do that for hundreds of file extensions / mime-types.
Has anyone got the AutoDetectParser working with .Net and IKVM.Maven.Sdk?
I suspect it's a problem with the DefaultDetector? (from Tika troubleshooting guide)
Hoping someone's already solved this problem and can share their solution! 🤞

I'm getting this error when using TikaOnDotNet from NET 6:
TypeLoadException: Could not load type 'System.Reflection.Emit.MethodToken' from assembly 'mscorlib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b77a5c561934e089'.
I suspect the error is caused by the System.Reflection.Emit.MethodToken type does not exist in NET 6 og .Net Core, only in .Net Framework.

 I know they do not allow emitting code.
On Sat, Sep 9, 2023, at 9:56 AM, Arextion wrote:

I'm getting this error when using TikaOnDotNet from NET 6:
TypeLoadException: Could not load type 'System.Reflection.Emit.MethodToken' from assembly 'mscorlib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b77a5c561934e089'.
I suspect the error is caused by the System.Reflection.Emit.MethodToken type does not exist in NET 6 og .Net Core, only in .Net Framework.
—
Reply to this email directly, view it on GitHub https://github.com/KevM/tikaondotnet/pull/152#issuecomment-1712530717, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAMHJ6KIETWBUCVXKEW4LXZR7SNANCNFSM6AAAAAAQCWG3ME.
 I know they do not allow emitting code.
…
On Sat, Sep 9, 2023, at 9:56 AM, Arextion wrote: I'm getting this error when using TikaOnDotNet from NET 6: TypeLoadException: Could not load type 'System.Reflection.Emit.MethodToken' from assembly 'mscorlib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b77a5c561934e089'. I suspect the error is caused by the System.Reflection.Emit.MethodToken type does not exist in NET 6 og .Net Core, only in .Net Framework. — Reply to this email directly, view it on GitHub <#152 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAMHJ6KIETWBUCVXKEW4LXZR7SNANCNFSM6AAAAAAQCWG3ME. You are receiving this because you were mentioned.Message ID: @.***>

No from Windows 11.

This is as simple as him trying to run TikaOnDotNet on .NET Core.
This very issue is about TokaOnDotNet not supporting .NET Core.

Very new to Tika, and trying to use it in .Net Core, and, following many links, ended up here!
Trying to use Tika to extract text on large quantities of files, so ideally would like to get AutoDetectParser working. Following @wasabii 's advice (thanks for your efforts!) and copying and pasting the MavenReferences you posted here, I'm able to Parse files, but only if I try something like @souramoo 's if docx then OOXMLParser type stuff. I don't want to have to do that for hundreds of file extensions / mime-types.
Has anyone got the AutoDetectParser working with .Net and IKVM.Maven.Sdk?
I suspect it's a problem with the DefaultDetector? (from Tika troubleshooting guide)
Hoping someone's already solved this problem and can share their solution! 🤞

Please let us know if you find a solution!

I Finally got a, somewhat, working project in .Net 6 with AutoDetectParser. But it doesn't work on all files. It seems to be working on .doc files, bout not .docx files e.g.
I guess this is because of missing parsers for those.
I've tried slamming in a bunch of references, but that doesn't help.
`

`
TikaOnDotNet does parse the files. So something must be missing.
@KevM what parsers or modules are included in TikaOnDotNet?

Hmm, interesting find.
AutoDetectParser works, if you create an instance of specific parsers, even if you are not using them:
new OOXMLParser();
new OfficeParser();

Hmm, interesting find.
AutoDetectParser works, if you create an instance of specific parsers, even if you're not using them:
new OOXMLParser();
new OfficeParser();

Nice find! That helps heaps! Here's my experiment so far - with your find included:
https://github.com/MattBayliss/TikaTest-IKVM.Maven.Sdk
I assume IKVM.Maven.Sdk has smarts not to include libraries that aren't directly referenced - I think that's what the IKVM.Maven.Sdk README is talking about in the Transitive Dependencies section... although I can't figure out / successfully duck-duck-go what a TFM is.
In the meantime I thought I needed to get logging working - because I assumed that would tell me where the problem was. Did you have the issue with SLF4J?
SLF4J: No SLF4J providers were found.
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See https://www.slf4j.org/codes.html#noProviders for further details.

The Tika documentation had pages on logging config, which I tried to emulate with MavenReferences in that repo of mine.
I abandoned Tika research to just get SLF4J working with MavenReferences, assuming I was having the same underlying issue - it not finding a library (SLF4J Simple Provider) that I had included as a MavenReference. I asked about that on StackOverflow to no success so far.
I'll keep working on my repo, and let you know if I have any further luck.

I ended up just preloading all parser assemblies like this:
string assemblyPath = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);
string[] assemblies = Directory.GetFiles(assemblyPath, "*.dll", SearchOption.AllDirectories);
var parserAssemblies = assemblies.Where(p => Regex.IsMatch(p, ".*\\.parser\\..*\\.dll$"));

foreach (string file in parserAssemblies)
{
    Assembly.LoadFile(file);
}

But, from my initial test, the parsing is very slow! and much slower than using TikaOnDotNet, for some reason. So i actually tried to find another way and came across Tika-Server.
I've already created a custom dockerfile and have it running in Docker Desktop using both the /tika (syncronious) and /async (asyncronious) parsing.
This article was very helpful.
It seems much faster and very easy to setup.

Yes SLF4J did not work for me either.

It's the same situation. Have to preload things that are dynamic.

Is this still ongoing? I'd loved to have it :(

Hi @isaaclisg! I submitted the initial pull request for .NET core support (nearly two years ago now 😳). I hope I'm not speaking incorrectly here (@KevM or anyone else for that matter), but I think this effort is more or less abandoned. Somebody please do correct me if I'm wrong!
My original use case for this was primarily motivated by simply not wanting to use the Tika library in Java or mess around with Tika Server. Really wanted to use C# but that use case ultimately ended up using the Tika library in Java, due primarily to performance issues with the .NET Core version which other's have mentioned. Without knowing you're use case that would be my general recommendation.
FYI, the .NET Core support was only made possible thanks the AMAZING efforts of https://github.com/ikvmnet/ikvm and primarily @wasabii. I haven't messed around with any of this in a long time now but when I did that documentation on that project was vital to get things even kind of working.

I would love to know more about that performance issue.

The use case I targeted was pretty niche and needed to run as Microservices in AWS. We had AWS Lambda functions written in both C# and Java at one point to evaluate the performance of the two on the same documents. The long and short of it is the Lambdas which used Tika via IKVM ended up being slower and using more cpu/memory. This was a .NET shop but we went with the Java version in the end because it would be cheaper for us to run. 🤷♂ Unfortunately all that work is protected under an NDA and I no longer have access to the code to share anyways. It was long enough ago now too that things may have changed so please take all this with a grain of salt.
I think IKVM is fantastic btw, thank you for your work!

My recollection is people had been working on ways to use Tika directly against IKVM.
On Fri, Jun 21, 2024, at 2:34 PM, Dylan wrote:

The use case I targeted was pretty niche and needed to run as Microservices in AWS. We had AWS Lambda functions written in both C# and Java at one point to evaluate the performance of the two on the same documents. The long and short of it is the Lambdas which used Tika via IKVM ended up being slower and using more cpu/memory. This was a .NET shop but we went with the Java version in the end because it would be cheaper for us to run. 🤷♂ Unfortunately all that work is protected under an NDA and I no longer have access to the code to share anyways. It was long enough ago now too that things may have changed so please take all this with a grain of salt.
I think IKVM is fantastic btw, thank you for your work!
—
Reply to this email directly, view it on GitHub https://github.com/KevM/tikaondotnet/pull/152#issuecomment-2183341984, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAMHPDNX36R64OXTL3S6DZIR55PAVCNFSM6AAAAABJWR2NJSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBTGM2DCOJYGQ.
You are receiving this because you were mentioned.Message ID: @.***>


Cannot find module '@angular/compiler-cli/ngcc'

I receive the following error : An unhandled exception occurred: Cannot find module '@angular/compiler-cli/ngcc' after trying to ng serve
Here is a copy of my package.json file:
{
  "name": "visualizer",
  "version": "0.0.0",
  "license": "MIT",
  "scripts": {
    "ng": "ng",
    "start": "ng serve",
    "build": "ng build --prod",
    "test": "ng test",
    "lint": "tslint 'src/**/*.ts'",
    "fixlint": "tslint 'src/**/*.ts' --fix",
    "e2e": "ng e2e"
  },
  "private": true,
  "dependencies": {
    "@angular-devkit/build-angular": "^0.901.1",
    "@angular/animations": "^6.0.3",
    "@angular/cdk": "^6.1.0",
    "@angular/common": "^6.0.3",
    "@angular/compiler": "^6.0.3",
    "@angular/core": "^6.0.3",
    "@angular/forms": "^6.0.3",
    "@angular/http": "^6.0.3",
    "@angular/material": "^6.1.0",
    "@angular/platform-browser": "^6.0.3",
    "@angular/platform-browser-dynamic": "^6.0.3",
    "@angular/router": "^6.0.3",
    "@ngrx/core": "^1.2.0",
    "@ngrx/effects": "^6.1.0",
    "@ngrx/store": "^6.1.0",
    "chart.js": "^2.7.2",
    "classlist.js": "^1.1.20150312",
    "core-js": "^2.5.6",
    "hammerjs": "^2.0.8",
    "npm": "^6.0.1",
    "rxjs": "^6.2.0",
    "rxjs-compat": "^6.2.0",
    "underscore": "^1.9.0",
    "web-animations-js": "^2.3.1",
    "zone.js": "^0.8.26"
  },
  "devDependencies": {
    "@angular/cli": "^9.1.1",
    "@angular/compiler-cli": "^6.0.3",
    "@angular/language-service": "^6.0.3",
<EMAIL_ADDRESS>"^2.7.18",
    "@types/chartjs": "0.0.31",
    "@types/jasmine": "^2.8.7",
    "@types/jasminewd2": "~2.0.2",
    "@types/node": "^10.1.2",
    "codelyzer": "^4.3.0",
    "jasmine-core": "~3.1.0",
    "jasmine-spec-reporter": "~4.2.1",
    "karma": "^2.0.2",
    "karma-chrome-launcher": "~2.2.0",
    "karma-coverage-istanbul-reporter": "^2.0.1",
    "karma-jasmine": "^1.1.2",
    "karma-jasmine-html-reporter": "^1.1.0",
    "protractor": "~5.3.2",
    "ts-node": "~6.0.5",
    "tslint": "~5.10.0",
    "typescript": "^2.7.2"
  }
}

When I then try install @angular/compiler-cli/ngcc using npm I get an error saying:
Could not install from "@angular/compiler-cli/ngcc" as it does not contain a package.json file.

You have a large mismatch between you Angular and CLI versions. If you update CLI to v9 then you cannot use Angular at v6


Remote device control

Hi!
I have a problem with setup remote scrcpy access with two Windows 10 machines. I have read readme file and googled  how to configure adb tunnel but no luck and looking for help.
My setup is simple. Two Windows 10 machines in local network. PC1 (<IP_ADDRESS>) has connected android device and i want to use scrcpy with it on PC2 (<IP_ADDRESS>). I can ping PCs back and forward (all firewalls are off)
I have downloaded scrcpy-win64-v1.23 archive on both PCs.
Now I setup server part like this (everyting like in tutorial):

Open PowerShell in scrcpy-win64-v1.23 directory from archive
./adb kill-server
./adb -a nodaemon server start
Left this console as is

Next is a client side. And here i already have problems:

Open PowerShell in scrcpy-win64-v1.23 directory from archive
./adb kill-server
export ADB_SERVER_SOCKET=tcp:<IP_ADDRESS>:5037      # this line doesn't work as export is unknown command. so i tried to replace 'export' with 'set'
./scrcpy --tunnel-host=<IP_ADDRESS>

And after this commands on client side i got this output:
scrcpy 1.23 <https://github.com/Genymobile/scrcpy>
INFO: Tunnel host/port is set, --force-adb-forward automatically enabled.
* daemon not running; starting now at tcp:5037
* daemon started successfully
ERROR: Could not find any ADB device
ERROR: Server connection failed

And that's all. I have no idea where to dig further. Hope for help.

export ADB_SERVER_SOCKET=tcp:<IP_ADDRESS>:5037 # this line doesn't work as export is unknown command. so i tried to replace 'export' with 'set'

# in bash
export ADB_SERVER_SOCKET=tcp:<IP_ADDRESS>:5037

:: in cmd
set ADB_SERVER_SOCKET=tcp:<IP_ADDRESS>:5037

# in PowerShell
$env:ADB_SERVER_SOCKET = 'tcp:<IP_ADDRESS>:5037'

(I didn't know about the PowerShell syntax)


login/continue as guest

have a login/password or register or continue as guest

here is my code for the login page and or continue as guest

Login For Movie Purchases

Username

Password
<input type="password" placeholder="Enter Password" name="psw" required>

<button type="submit">Login</button>

CONTINUE AS GUEST

Cancel

Login For Movie Purchases

Username

Password
<input type="password" placeholder="Enter Password" name="psw" required>

<button type="submit">Login</button>

CONTINUE AS GUEST

Cancel


Support for web?

Are there plans to support react native web?

Any pointers @howljs

Hi guys, what issues have you exactly encountered with web?
Today I've been experimenting with it, in an EXPO project, and I'd like to confront my experience with yours.
In my case, I wasn't even able to open the app in browser after including this component, as I was getting some JS errors in browser console about unrecognized tokens in the build.
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
|     zoom: props => {
|       var _timelineVerticalList;
>       let newHeight = (props === null || props === void 0 ? void 0 : props.height) ?? initialTimeIntervalHeight;
|       if (props !== null && props !== void 0 && props.scale) {
|         newHeight = timeIntervalHeight.value * props.scale;
    at<EMAIL_ADDRESS>(timeZone.js:2229:1)

After some research, I found out that there are those 'void 0' in the build sent to browser, which are causing the issue. This has been the first time I've encountered such issue in a react native component / library, and I'm not sure if it's caused by the way it has been built or something else (@howljs do you have any clues on this?)
However, I was able to circumvent the issue by adding the 'build' option below into the web section of Expo app.json config file. It seems to work, although I believe it would be better if the root cause was fixed, as the config flag I used seems even to be undocumented in the official Expo documentation.
    "web": {
      "build": {
        "babel": {
          "include": ["@howljs/calendar-kit"]
        }
      }
    }

Sorry. This library is not support react-native-web :(


Cache the computation of core toString predicates for cpp c# and java.

This improves the performance of running many queries when toStrings of results are computed.

Have you checked what the effect is on the cache size? I'm also curious whether we have queries that derive from AST classes and override toString, thereby invalidating the cache. I've certainly seen that in the past.

The dist-compare report for C# shows no significant change in performance.

I just did a bit of testing with some C++ queries and this PR.  On a sequence of typical (real) queries it makes no noticeable difference - which I think is what we'd expect since none of them produced many result rows to toString.  However on a short sequence of (artificial) queries with lots of results this made a big improvement to performance after the first query.  I think this is what we're expecting.
:+1:

This improves the performance of running many queries when toStrings of results are computed.

Testing for Java on our CI infrastructure suggests that overall analysis time for the standard LGTM query suite increases by a non-negligible amount on some projects, in particular 10-15% for JDK 11. It would be good to understand why. Perhaps @aschackmull could look into it? The outliers in the profiler result were UselessParameter.ql, BoxedVariable.ql and FLinesOfCommentedCode.ql (in that order).

Testing for Java on our CI infrastructure suggests that overall analysis time for the standard LGTM query suite increases by a non-negligible amount on some projects, in particular 10-15% for JDK 11. It would be good to understand why. Perhaps @aschackmull could look into it? The outliers in the profiler result were UselessParameter.ql, BoxedVariable.ql and FLinesOfCommentedCode.ql (in that order).

Fixes for those three queries are here: https://github.com/Semmle/ql/pull/2341

@aschackmull, thanks for the Java fixes, which have now been merged.
This has already been approved for C++ and C#, but there was a question from @jbj in https://github.com/Semmle/ql/pull/2204#issuecomment-546361174 without a reply, so I'll leave @jbj to decide whether/when to merge.

I've started a benchmark on the full suite and will leave it running over night.

I benchmarked the change on the whole suite, and all I got was a 5% slowdown. But it was on an Azure machine with a networked drive, and it was rebooted between runs, so it could just be wobble. I'll investigate.


Attempting to resolve issues in backend

@ct-clmsn this is an attempt to resolve issues in the backend introduced by recent changes to handling scopes. It does not fully resolve things as I wasn't sure how things are supposed to work. Please have a careful look.

FWIW, with this PR, I'm seeing this when compiling forall.chpl:
chplx : for identifier already defined in current scope
chplx : i identifier already defined in current scope
chplx : i identifier already defined in current scope
chplx : i identifier already defined in current scope
chplx : i identifier already defined in current scope
chplx : coforall identifier already defined in current scope

after which the compiler segfaults here: https://github.com/ct-clmsn/chplx/blob/edcdb26facc3b1ae33ce06475d9d5aef2a515aec/backend/src/programtreebuildingvisitor.cpp#L870 as curStmts is empty.

Huh, I ran some tests on my local system before doing the commit+merge and it looked like everything checked out. Will dig into this in the evening. Thanks for looking into it.

These issues have been resolved by other commits. Closing now.


Feature Request: Add link to wikipedia page for pokemon

Being able to visit the page for each Pokemon in an online resource would be helpful to fill in the missing information.
The following link is an example for the "Cofagrigus" Pokemon: https://bulbapedia.bulbagarden.net/wiki/Cofagrigus_(Pokémon)
This link could be automatically generated for each Pokemon.

Ticket rejected.
Beyond the scope of this project.

Please consider closing the issue as "not completed" instead of "closed as completed" if the item is rejected


Page content is not using all available space

I really like this themes. thank you for providing it.
I noticed page content is not using the entire space available especially on large monitors. It would be great if we can change it to use all available space.

opened an issue on docsify-themeable as it seems more appropriate place to address this issue
https://github.com/jhildenbiddle/docsify-themeable/issues/74


Latest extension fails to load in VS Code 1.22

Environment data

VS Code version: 1.22.2
Extension version (available under the Extensions sidebar): 2018.5.0
OS and version: Windows 7 x64
Python version (& distribution if applicable, e.g. Anaconda): 3.6
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): N/A
Relevant/affected Python packages and their versions: N/A

Actual behavior
Python extension installs, but never activates and constantly prompts to reload. Developer mode shows that it requires VS Code >= 1.23.
Expected  behavior
We should be able to use v 1.22 as we are only a couple of months behind. We are in a corporate environment and it takes time to get new versions of VS Code packaged and deployed to our machines
Steps to reproduce:

Install VS Code 1.22.2
Try installing or updating python extension

Logs
Extension is not compatible with Code 1.22.2. Extension requires: ^1.23.0.

Unfortunately the minimum version required is 1.23.
If you are unable to to upgrade VS Code, you have two solutions:

Download VS Code as a zip file instead of installing it
Download and install the older version of the extension and disable automatic updates of extensions (https://marketplace.visualstudio.com/_apis/public/gallery/publishers/ms-python/vsextensions/python/2018.4.0/vspackage).

Thanks for the link, I'd been googling for ages to work out how to get older extensions


[Print] Improve print template

It would be nice to have the same template as https://observatoire.maine-et-loire.fr/

Manage a second logo if included in the django configuration

Linked with #520


[BUG] - The file.log  upload to ftp server. then lost \n line breaks

The Upload code of Core
fn test() {
  let mut buffer = Vec::new();
  let mut f = fs::File::open(src).unwrap();
  let mut reader = f
    .read_to_end(&mut buffer)
    .map(|_| Cursor::new(buffer))
    .map_err(FtpError::ConnectionError)?;
  match upload_type {
    StorageUploadType::Append => {
      ftp.append_file(&fname, &mut reader)?;
    }
    StorageUploadType::Replace => {
      ftp.put_file(&fname, &mut reader)?;
    }
    StorageUploadType::Once => {
      if ftp.size(&fname).is_err() {
        ftp.put_file(&fname, &mut reader)?;
      } else {
        /* 已经存在 */
      }
    }
  };
}

I have found the problem, but please do not handle line breaks on different platforms; Adding the following code solves this problem;
So I have the following proposal: for FPT uploads, we can increase the recognition of the FTP server platform; Then process the data;
Secondly, increase the judgment of file types and only handle text types;
Let mut buffer=Vec:: new();
Let mut f=fs:: File:: open (src). unwrap();
Let mut reader=f
. readuto_end (&mut buffer)
. map (| _ |{
//Iterate over multiple references to each byte and replace '\ n' with '\ n'
//Use map to replace '\ n' with '\ n' and collect into a new Vec<u8>
Let modified_data: Vec<u8>=buffer
. iter()
. map (|&byte |{
If byte==b '\ n'{
Vec! [b '\ r', b '\ n']
}Else{
Vec! [byte]
}
})
. flatten()
. collect();

Cursor:: new (modified_data)
})
. map_err (FtpError: ConnectionError)?;

I have found the problem, but please do not handle line breaks on different platforms; Adding the following code solves this problem;
So I have the following proposal: for FPT uploads, we can increase the recognition of the FTP server platform; Then process the data;
Secondly, increase the judgment of file types and only handle text types;

Let mut buffer=Vec:: new();
Let mut f=fs:: File:: open (src). unwrap();
Let mut reader=f
. readuto_end (&mut buffer)
. map (| _ |{
//Iterate over multiple references to each byte and replace '\ n' with '\ n'
//Use map to replace '\ n' with '\ n' and collect into a new Vec<u8>
Let modified_data: Vec<u8>=buffer
. iter()
. map (|&byte |{
If byte==b '\ n'{
Vec! [b '\ r', b '\ n']
}Else{
Vec! [byte]
}
})
. flatten()
. collect();

Cursor:: new (modified_data)
})
. map_err (FtpError: ConnectionError)?;

I have found the problem, but please do not handle line breaks on different platforms;
Adding the following code solves this problem;
So I have the following proposal: for FPT uploads, we can increase the recognition of the FTP server platform;
Then process the data;
Secondly, increase the judgment of file types and only handle text types;
Let mut buffer=Vec:: new();
Let mut f=fs:: File:: open (src). unwrap();
Let mut reader=f
. readuto_end (&mut buffer)
. map (| _ |{
//Iterate over multiple references to each byte and replace '\ n' with '\ n'
//Use map to replace '\ n' with '\ n' and collect into a new Vec<u8>
Let modified_data: Vec<u8>=buffer
. iter()
. map (|&byte |{
If byte==b '\ n'{
Vec! [b '\ r', b '\ n']
}Else{
Vec! [byte]
}
})
. flatten()
. collect();

Cursor:: new (modified_data)
})
. map_err (FtpError: ConnectionError)?;


Struggling with multiprocess in JAX

Description
I'm trying to compile a piece of code on multiple CPUs and evaluate it in parallel for nested sampling.  I'm not trying to parallelize individual instances of the code; I just want to control these separate processes from the same script so I know when I can stop sampling.
I found I had to do a warmup call on each CPU in order to run the code in my nested sampling routine, otherwise it recompiled each time.  But I'm finding that the warmup call often stalls.
Here's a snippet that reproduces the error:
import jax
import jax.numpy as jnp
from jax import vmap

import multiprocessing

rho_g_vec = jnp.logspace(3,-14,263)

@jax.jit
def get_t(rho_g_vec):

    def k2(x):
        return jnp.where(x<3,
                    jnp.exp(-x) * (
                                    784/1615 + 2/x**2 + 3232/(1615*x) - 448*x/4845 + 5416744*x**2/190855665
                    ),
                    jnp.exp(-x) * jnp.sqrt(jnp.pi / 2) * (x**(-0.5) + (15/8)*x**(-1.5) + (105/128)*x**(-2.5)))

    k2_vec= vmap(k2)

    def int_3_2(a):
        k_plus_1_vec = jnp.arange(20) + 1
        K_terms = k2_vec(a*(k_plus_1_vec))
        return 3 * a**2 *  jnp.sum(((-1)**(k_plus_1_vec-1)) * K_terms / k_plus_1_vec**2)

    def rho_e(T):
        result = int_3_2(0.5/T)
        return result
    rho_e_v = vmap(rho_e)

    rho_tot_vec = rho_e_v(rho_g_vec)

    return rho_tot_vec

def warmup_process():
    return get_t(rho_g_vec)

if __name__ == "__main__":
    num_cores = 4
    processes = [multiprocessing.Process(target=warmup_process) for _ in range(num_cores)]

    for p in processes:
        p.start()
    for p in processes:
        p.join()
        print("warmed up on {0}".format(p))

I thought it might be a memory issue--changing the length of rho_g_vec to 262 lets everything run smoothly.  But increasing the memory per CPU from 10GB to 100GB did not make a difference.  The code runs smoothly on 1 CPU (and num_cores=1) with 10GB, but when I run on 4 CPUs as above it hangs.  Interrupting gives the following error:
Traceback (most recent call last):
  File "/scratch/cg3566/jaxBBNsampling/multiprocess_test.py", line 45, in <module>
    p.join()
  File "/ext3/miniconda3/lib/python3.11/multiprocessing/process.py", line 149, in join
    res = self._popen.wait(timeout)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ext3/miniconda3/lib/python3.11/multiprocessing/popen_fork.py", line 43, in wait
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ext3/miniconda3/lib/python3.11/multiprocessing/popen_fork.py", line 27, in poll
    pid, sts = os.waitpid(self.pid, flag)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt

This also has to be interrupted, giving
Exception ignored in atexit callback: <function _exit_function at 0x14d6171dfce0>
Traceback (most recent call last):
  File "/ext3/miniconda3/lib/python3.11/multiprocessing/util.py", line 357, in _exit_function
    p.join()
  File "/ext3/miniconda3/lib/python3.11/multiprocessing/process.py", line 149, in join
    res = self._popen.wait(timeout)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ext3/miniconda3/lib/python3.11/multiprocessing/popen_fork.py", line 43, in wait
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ext3/miniconda3/lib/python3.11/multiprocessing/popen_fork.py", line 27, in poll
    pid, sts = os.waitpid(self.pid, flag)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt:

What jax/jaxlib version are you using?
0.4.23 0.4.23
Which accelerator(s) are you using?
CPU
Additional system info?
No response
NVIDIA GPU info
No response

See, e.g., https://github.com/google/jax/issues/1805#issuecomment-561244991
JAX is incompatible with the fork strategy for multiprocessing.
Does that fix the problem?

Yes, thanks!  Sorry, looks like I missed that when I was searching


Ambiguous interface description with C bit-fields

Ambiguous bit-field interface description
Describing the hypervisor interface by the means of C bit-fields is ambiguous. The C specification, does not specify the order of allocation of bit-fields within a unit, therefore in the following example:
typedef union
{
   UINT64 AsUINT64;
   struct
   {
       UINT64 Reserved : 62;         // Reserved bits
       UINT64 CrashMessage : 1;      // P3 is the PA of the message
                                     // P4 is the length in bytes
       UINT64 CrashNotify : 1;       // Log contents of crash parameter
   };
} HV_CRASH_CTL_REG_CONTENTS;

It is not clear whether CrashNotify and CrashMessage are bits 0 and 1, or 63 and 62 (second is true, but has to be deduced from Linux sources).
I did not search the documentation for all occurrences of this syntax, I only provide this one as an example.
Link to doc: virtualization/hyper-v-on-windows/tlfs/datatypes/hv_crash_ctl_reg_contents.md

We hugely appreciate your contribution, feedback, and support to our technical documentation.
We’ve phased out GitHub Issues on this repository as the feedback mechanism for content and have replaced it with a new feedback system. For more information see, Content User Feedback. As part of this rollout, we are migrating your issue where our team will use your comments to improve the documentation experience. Unfortunately, this does mean you won't be able to access this issue after the migration is complete.
All issues will be migrated by Thursday 29th August 2024.
Once again, thank you for your valuable feedback, we hope you’ll continue to contribute using our new feedback system.


Cannot set dps, prec, workdps, or workprec

Setting or changing dps, prec, workdps, or workprec does not work.
Windows 10, Python 3.9. Python backend but tried gmpy2 also with the same result.
import mpmath as mp
from mpmath import mpf,pi
mp.workdps=10
print("Sqrt of 2 (10 decimal places): ",mpf(2) ** mpf('0.5'))
print("Twice of pi (10 decimal places)", 2*pi)

mp.workdps=40
print("Sqrt of 2 (40 decimal places): ",mpf(2) ** mpf('0.5'))
print("Twice of pi (40 decimal places)", 2*pi)

I get the same answer.
>> python mp_test.py

Sqrt of 2 (10 decimal places):  1.4142135623731
Twice of pi (10 decimal places) 6.28318530717959
Sqrt of 2 (40 decimal places):  1.4142135623731
Twice of pi (40 decimal places) 6.28318530717959

Update. Solved but very weird behavior. You should look into this.
Adding one line from mpmath import makes the code work. But it should work w/o it as above.
import mpmath as mp
from mpmath import *
mp.dps=10
print("Sqrt of 2 (10 decimal places): ",mpf(2) ** mpf('0.5'))
print("Twice of pi (10 decimal places)", 2*pi)

mp.dps=40
print("Sqrt of 2 (40 decimal places): ",mpf(2) ** mpf('0.5'))
print("Twice of pi (40 decimal places)", 2*pi)

@tirthajyoti, in your 1st code snippet you should use mp.mp.dps instead:
$ python -q
>>> import mpmath
>>> mpmath.dps
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'mpmath' has no attribute 'dps'
>>> mpmath.mp.dps
15

$ cat mp_test.py
import mpmath as mp
from mpmath import mpf, pi
mp.mp.dps=10
print("Sqrt of 2 (10 decimal places): ", mpf(2) ** mpf('0.5'))
print("Twice of pi (10 decimal places)", 2*pi)

mp.mp.dps=40
print("Sqrt of 2 (40 decimal places): ",mpf(2) ** mpf('0.5'))
print("Twice of pi (40 decimal places)", 2*pi)

$ python mp_test.py
Sqrt of 2 (10 decimal places):  1.414213562
Twice of pi (10 decimal places) 6.283185307
Sqrt of 2 (40 decimal places):  1.41421356237309504880168872420969807857
Twice of pi (40 decimal places) 6.283185307179586476925286766559005768394

The workdps() is a context wrapper, not a variable.  See more in docs.
Probably, there is no issue.

workdps should be used like this:
import mpmath as mp
from mpmath import mpf,pi
with mp.workdps(10):
    print("Sqrt of 2 (10 decimal places): ",mpf(2) ** mpf('0.5'))
    print("Twice of pi (10 decimal places)", 2*pi)

with mp.workdps(40):
    print("Sqrt of 2 (40 decimal places): ",mpf(2) ** mpf('0.5'))
    print("Twice of pi (40 decimal places)", 2*pi)

Adding one line from mpmath import * makes the code work. But it should work w/o it as above.

@tirthajyoti could you justify why this "should work"?  Nothing like that was documented in the documentation, e.g. here.
Meanwhile, I'll close this in favor of #657.


[Feature Request] ES modality

What feature or change would you like to see made?
Hello, may be question will not actuall, i'm not sured. But could you add play and show in cornestone ES modality? Because i'm tried load and show in my app, but without success. Thanks
Why should we prioritize this feature?
Because it's a very useful and significant thing

You mean Endoscopy? Do you have data to share?

it says it needs a password

pass 123

I tried multiple viewers, non can render it, do you have a baseline (viewer) that can render this at least?

https://dropmefiles.com/vf2q8
i did video from meddream

Hmmm interesting, ok we will triage it


Fix TypeUtils.hashPosition for structural types

Fixes hashcode generation for maps that contain complex types, and enables GROUP BY on them

Seems good to me. @cberner Please take a second look. It's one line of real change and a new test in AbstractTestQueries.

@cberner This fixes the production issue you mentioned this morning.

One comment, but looks good otherwise

Looks good.


Expand Submodel Documentation

Submodel docs could do with some expansion.
Needs python example(s).

Will be resolved by #75


504 Gateway Time-out

Hi,
I am experiencing this error with long running tasks:
504 Gateway Time-out

Where should I add this setting to the config?
http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout

Thanks!

Hi.
This project (JrCs/docker-letsencrypt-nginx-proxy-companion) "only" purpose is to:

enable the automatic creation of certificates from containers env var.
put them in the right place with the right name so they can be used by nginx-proxy docker-gen template.
reload the nginx proxy when needed (cert creation or renewal).

What you want to achieve depends entirely on the jwilder/nginx-proxy project, which does the actual proxying and is maintained by different peoples.
We only deal with issues related to letsencrypt-nginx-proxy-companion, so please submit your nginx-proxy issues directly to the nginx-proxy repo.


Oxigraph vs SPARQL 1.1 discrepency in DECIMAL parsing

The SPARQL 1.1 query grammar do not allow decimals to be written without decimal digits like 1. whereas Oxigraph SPARQL parser allows them. It might be nice to restrict Oxigraph parser to only parse decimals with decimal digits to avoid "interoperability traps".

Fixed in 0.4.0-alpha.1


Please, suicide, stop programming and DONT loose our time.

Also on ubuntu 18.04 there are no or is wrong ocl.hpp because it crash all the time saying "I'm not for programming by stupid western morons."/home/my/butterflow/butterflow/ocl.cpp:298:11: error: ‘DeviceInfo’ does not name a type; did you mean ‘CvTypeInfo’?
const DeviceInfo& currentDevice = Context::getContext()->getDeviceInfo();
^~~~~~~~~~
CvTypeInfo
/home/rado/butterflow/butterflow/ocl.cpp:299:32: error: ‘currentDevice’ was not declared in this scope
return PyString_FromString(currentDevice.deviceName.c_str());
When you will suicide, remove your deep amateur code from github and stop programming?

 Someone wrote this code and released it for free for others to learn from instead of keeping it to themselves. There is no guarantee of it working, or any other code on Github. If you don't like it, fix it.
 It may be a joke to you, but you don't know what the mindset of the person reading it will be, it may not be a joke to them.
The world doesn't owe you anything. Grow up and deal with your own problems instead of blaming them on other people. Reported this comment to Github for abusive behavior.
Finally - just want to say thank you to @dthpham for creating and maintaining this repo, even if it's not in perfect shape. It's been valuable to me and countless others.


Deprecate generating the .genome file

Description of feature
We found out today that the .genome file is superfluous. HiGlass has an option to draw the chromosome grid from the cool file directly.
We can therefore consider not generating the file at all in this pipeline, which makes things a bit simpler.

.genome is not necessary to be used to display on higlass with mcool map, if you only tend to display mcool, indeed you don't need that. We maintain it as it is used for coverage and other bigwig related tracks.

Further discussion with @yumisims in the office: it's a HiGlass UX thing. Some people find it simpler and more convenient to see the grid as a separate file, and find selecting the heatmap to then scroll down and select "2d chrom grid" too cumbersome. So maybe let's just close this ticket and keep generating both ?


BIN and JSON cached files

I would like to read the generated BIN and JSON (when cache=True) directly via simple function into a Wavefront object. That is, I would like to read these two files myself and feed them to Wavefront constructor. I wonder if this can be done via some auxiliary function?

It does already load from cached files if they exist and the cache parameter is True

@einarf that's totally correct, but I would like to use those streams in my application (stream of bytes and not as a file); in fact, I would like to store them in a single file with some other data; then when I read the file I would like to construct the OBJ from those streams. One way to do this is to load those BIN and JSON file and store them into my file, then, at the time of loading; save them as BIN and JSON files and construct the object with cache=True, but that would be cumbersome.
Anyways, could you please guide me where in the code the loading happens? Maybe I can tweak it to my case.

I don't think you can fully reconstruct the obj from the bin/json file. They are mainly made so users can easily dump them into graphics memory.
I do have a library overriding the cache loader that just uploads the byte streams directly to the gpu instead of reconstructing the obj itself (much faster)
https://github.com/moderngl/moderngl-window/blob/4c3cd385718517a8055e8c271195f9100eaf2a66/moderngl_window/loaders/scene/wavefront.py#L52-L69
Cache loading triggers from this : https://github.com/pywavefront/PyWavefront/blob/caa2013d7026dd484ac8c5ca8273f2d680196130/pywavefront/obj.py#L86-L90
Loader and saver is here : https://github.com/pywavefront/PyWavefront/blob/master/pywavefront/cache.py

What I wanted is to pass data instead of reading it via data = json.loads(fd.read()) (line 193) and  other info red via file handler fd = gzip.open(str(cache_name(self.file_name)), 'rb') (lines 109-119) to the class constructor, but I guess that would disrupt the integrity of PyWavefront.
Hence, the best way to do this by allowing my application to save the cached files (BIN and JSON), same for the .mtl file if used, before calling PyWavefront (I am thus removing the original .obj file, and). I did a few tests and it worked even after xx.obj file has been removed, as long as cache=True and xx.obj.bin and xx.obj.json are available.

Yup. that seems reasonable. You can even replace the cache loader / saver classes here if you need more flexibility : https://github.com/pywavefront/PyWavefront/blob/caa2013d7026dd484ac8c5ca8273f2d680196130/pywavefront/obj.py#L50-L51

Thanks you for the heads up! I don't think I need to do this now, but maybe later.
The only concern, although trivial, is that when the original file OBJ is cached and then removed, one still needs to use its name when constructing the wavefront .. i.e. via
pywavefront.Wavefront('earth.obj', ...)
Not sure if it would be a good practice to remove the .obj extension from the name, and then add it later if it is not there. In this case, as an example with earth file, one can either use,
pywavefront.Wavefront('earth', ...)
or
pywavefront.Wavefront('earth.obj', ...)
to construct the object.


Add a utility / builtin for numeric widening conversions

A concrete example of where this would be useful:
primitive Example
  fun assert_eq(h: TestHelper, a: F64, b: F64, variation: F64):Bool =>
      h.assert_true(@fabs[F64](a - b) < ( b * variation ))

If a, b or variation differ in type above, then explicit conversions are required.
In generic code, this would clearly fail to compile:
primitive CompilerSaysNo
  fun assert_eq[A](a: A, b: A, variation: F64):Bool =>
      h.assert_true(@fabs[F64](a - b) < ( b * variation ))

In most languages ( such as Java / JVM based ) that support widening conversions
a single implementation would suffice ( via F64 ). But as pony requires explicit conversion
this would require a considerable amount of explicit casting and conversion:
actor Main
  new create(env : Env) =>
    let a : U32 = I32(10).abs()
    let b : U64 = ( a.i64() - I64(-100) ).abs()
    let c : F64 = ( b.f64() - F64(1.234e2) ).abs()
    env.out.print("C: " + c.string())

Here, the fact that abs() converts signed to unsigned has a compounding effect on
the amount of code that needs to be written.
Generally, explicit casting/conversion is awesome. It documents intent clearly.
But sometimes, widening conversions can be very useful in the right places.
There may be constraints, concerns or monkey wrenches involved due to the
interaction of structural subtyping and algebraic types that would need careful
consideration though here. And certainly, any use of widening should be
cause for review. I cannot think of many cases where it is desirable in Pony
( or languages that support widening conversions by default, for that matter )
but this is one case where it would be quite useful to the lazy programmer.
Worth some investigation and discussion.

The problem with implicit size conversions of any kind is that it matters exactly when in the expression evaluation they occur and that can make a difference to the result. This then becomes a situation where the programmer has to know exactly what the compiler's doing and if they get it wrong then you have bugs.
Also it's worth pointing out that your example has a lot more casting than it needs, Pony inference is better than that.
You have:
let a: U32 = I32(10).abs()
let b: U64 = (a.i64() - I64(-100)).abs()
let c: F64 = (b.f64() - F64(1.234e2)).abs()

whereas you only need:
let a = I32(10).abs()
let b = (a.i64() - -100).abs()
let c = (b.f64() - 1.234e2).abs()

Yep. Precedence is a pretty big monkey wrench! :) This is why I like explicit precedence and casting in
pony the way it is. But there's always an exception to the general case, and this is an example of one where I think some kind of widening support would be beneficial.

Of course we do already have something that effectively achieves this, at least for your example.
primitive CompilerSaysNo
  fun assert_eq[A](a: A, b: A, variation: F64):Bool =>
      h.assert_true(@fabs[F64](a - b) < (b * variation))

This would actually fail anyway because the compiler doesn't know you can subtract an A from an A. You can put in a constraint to allow that, eg require A to be Real.
This will still fail because you can't multiply b by an F64, but an explicit cast fixes that. And a final cast makes the argument to fabs an F64 (I'm assuming you need that).
Then it works:
primitive CompilerSaysYes
  fun assert_eq[A: Real[A] val](a: A, b: A, variation: F64):Bool =>
      h.assert_true(@fabs[F64]((a - b).f64()) < (b.f64() * variation))

(Actually this still fails, but only because we haven't passed in h as a parameter.)

@andymcn I love it when you find a way without needing extra code! :)
This covers primitive builtin numerics, which I'd imagine is the 99.9% use case, so closing with Thanks!


Autosize does not resize textbox to smaller when border: 0

Unsure if this is due to the any other settings ('box-sizing: border-box' ?) but when the <textarea autosize> has a style border: 0 the text area does not become smaller when text is removed.
It still correctly increases the size of the box when text is added.
I solved my problem by setting the border to 1 px of the same color as background, but wanted to bring this bug to your attention.

@whyboris Thank you for bringing this to my attention and your workaround 👍
Still a strange thing^^

I wouldn't mind addressing this bug. Am I able to submit a PR for this?


Return Response as friendly public transport JSON

Currently when I print the response of the pyhafas client, I get a according object as defined in the types package. However I would like to retrieve the freindly public transport format JSON of e.g. an journey. Is that already possible (and I overlooked it somehow) or would it be feasible to implement ? I could also offer to submit a PR for that.

I just experimented a bit with jsonpickle which seems to be able to do the trick. It serializes object graphs and even allows to deserialize them again. Would this be an option? I guess it makes sense to use an existing library instead of implementing something on our own. However one could also argue, that it makes more sense to keep this in the software, that makes use of pyhafas. Or this this a feature which could fit into pyhafas?


Roll Flutter Engine from 2c3efa0beb24 to 256a41e4d9bb (1 revision)

https://github.com/flutter/engine/compare/2c3efa0beb24...256a41e4d9bb
2022-08-15<EMAIL_ADDRESS>Roll Skia from babb35ec754b to 060222bf1622 (1 revision) (flutter/engine#35410)
If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Please CC<EMAIL_ADDRESS>on the revert to ensure that a human
is aware of the problem.
To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose
To report a problem with the AutoRoller itself, please file a bug:
https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

Commit queue failed; closing this roll.


handle ' tag in ATOM

Hi,  I noticed that MWFeedItem's link property is nil, when link tag is like <link href="" />.
So changed implementation for processing link tag for ATOM

This is years old but I'd still like it to get merged...


Blog: adding tags, mainly for Dev portal and platform engineering

Description
Multiple articles relate to developer portals & platform engineering, but they need to be tagged to be found together.
The following tags will be added for SEO purposes (since they become part of the page itself) and for easy findability via tags.
- developer-portals
- platform_engineering
Update: I also added other missing tags  on other somewhat related articles
Checklist:

[ ] I have reviewed the style guide.
[ ] If blogging, I have reviewed the blogging guide.
[ ] I have manually confirmed that all new links work.
[ ] I added aliases (i.e., redirects) for all filename changes.
[ ] If making css changes, I rebuilt the bundle.

Note: I using the PR to also add Kat's video to her article


Initial code copy

Merging should be blocked on tests passing.

@alexlaurinmath is this meant to be merged into a single commit like this?
I see 7 commits in this PR, most of them are merges of the copybara branch. I thought this was preserving history?

I wasn't able to get history preservation to work. I think we see multiple commits because I'm using the same branch without deleting it? If this becomes a problem I'll either delete the branch or use a different one.


Pip support

As someone who stumbled across your module and wanted support for pip, as well as someone who was slightly confused on how this works, I'm requesting a few changes in this pull request.

Updated README - these updates were to provide clarity on the cwd nesting, as well as add a table that says exactly what files are supported and what the resulting commands will be.  Just makes it more readable (IMHO) as a new user of the module.

Added support for pip - Even though pip requires a -r flag to specify what the requirements file is, other deployment systems (like Docker) look for the existence of this file to run pip install on it. These changes add support for pip as well.

Thanks! :+1:
Sorry for the slow response...


Update landscape with Sedai - 2nd attempt

Adding just in Continuous Optimization category.  Logo should be referenceable now.
Signed-off-by: John Jamie<EMAIL_ADDRESS>Pre-submission checklist:
Please check each of these after submitting your pull request:

[ ] Are you only including a repo_url if your project is 100% open source? If so, you need to pick the single best GitHub repository for your project, not a GitHub organization.
[ ] Is your project closed source or, if it is open source, does your project have at least 300 GitHub stars?
[ ] Have you picked the single best (existing) category for your project?
[ ] Does it follow the other guidelines from the new entries section?
[ ] Have you added your SVG to hosted_logos and referenced it there?
[ ] Does your logo clearly state the name of the project/product and follow the other logo guidelines?
[ ] Does your project/product name match the text on the logo?
[ ] Have you verified that the Crunchbase data for your organization is correct (including headquarters and LinkedIn)?
[ ] ~30 minutes after opening the pull request, the CNCF-Bot will post the URL for your staging server. Have you confirmed that it looks good to you and then added a comment to the PR saying "LGTM"?

LGTM

Thanks @caniszczyk !


Types of functions that we mostly use .

Types of functions that we generally use and also added comments on how they are treated in the memory creation phase in execution.

@karnatisrinivas this is already added please add the unique concept


Blobs example a bit confusing

General information:

emcee version: 3.0.2
platform: Linux
installation method (pip/conda/source/other?): pip

Problem description:
Expected behavior:
There are actually a few things that are confusing me (based on the docs), and one thing that (given I understand correctly) could potentially be made better in the code. The first confusing thing in the docs is that (on the "Blobs" page) it gives the following example:

def log_prob(params):
    lp = log_prior(params)
    if not np.isfinite(lp):
        return -np.inf, -np.inf
    ll = log_like(params)
    if not np.isfinite(ll):
        return lp, -np.inf
    return lp + ll, lp

This looks like it's returning lp as the log-likelihood when ll is infinite. That doesn't seem right... if ll is infinite, then shouldn't the returned probability be -np.inf? Are the returns backwards?
Now the next example, for "named blobs and custom dtypes" has this log_prob:
def log_prob(params):
    lp = log_prior(params)
    if not np.isfinite(lp):
        return -np.inf, -np.inf
    ll = log_like(params)
    if not np.isfinite(ll):
        return lp, -np.inf
    return lp + ll, lp, np.mean(params)

Again, this has the same seemingly backwards return when ll is not finite. But it also has a different number of returns when either lp or ll are infinite (2 instead of 3). Is this really allowed? If so, great (though see below), but then I think it should be more clear that this is really allowed and encouraged.
However, in my own application, I have a log_prob that has dynamic blobs (I let the user specify a bunch of keys that they want to pull out of an object to store as blobs). This works perfectly fine, except that when I do a similar if not np.isfinite(ll): return -np.inf, -np.inf, emcee complains:
~/miniconda3/envs/halomod/lib/python3.7/site-packages/emcee/moves/red_blue.py in propose(self, model, state)
     91
     92             # Compute the lnprobs of the proposed position.
---> 93             new_log_probs, new_blobs = model.compute_log_prob_fn(q)
     94
     95             # Loop over the walkers and update them accordingly.

~/miniconda3/envs/halomod/lib/python3.7/site-packages/emcee/ensemble.py in compute_log_prob(self, coords)
    443                 except ValueError:
    444                     dt = np.dtype("object")
--> 445             blob = np.array(blob, dtype=dt)
    446
    447             # Deal with single blobs properly

ValueError: could not assign tuple of length 1 to structure with 3 fields.

Minimal Example(s)
Let data = np.random.normal(loc=0.2, scale=0.3, size=25) and init_pos = np.array([0.2, 0.3]) + 1e-4 * np.random.normal(size=(100, 2))
Example 1:
def log_prob(p, data):
    mu, sig = p

    # "prior"
    if not 0 < mu < 1:
        return -np.inf, -np.inf

    ll = np.sum(-(data - mu)**2 / (2 * sig**2)) - sig

    return ll, mu*sig, mu+sig

This is almost the same as the docs example -- returning two blobs, but only one blob if returning a -np.inf prior. Setup sampler with
sampler = emcee.EnsembleSampler(
    nwalkers = 100,
    ndim = 2,
    log_prob_fn = log_prob,
    args=(data,)
)

It exits with
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-32-c79fbf8d39e5> in <module>
----> 1 sampler.run_mcmc(init_pos, nsteps=1000, progress=True)

~/miniconda3/envs/halomod/lib/python3.7/site-packages/emcee/ensemble.py in run_mcmc(self, initial_state, nsteps, **kwargs)
    382
    383         results = None
--> 384         for results in self.sample(initial_state, iterations=nsteps, **kwargs):
    385             pass
    386

~/miniconda3/envs/halomod/lib/python3.7/site-packages/emcee/ensemble.py in sample(self, initial_state, log_prob0, rstate0, blobs0, iterations, tune, skip_initial_state_check, thin_by, thin, store, progress)
    341
    342                     # Propose
--> 343                     state, accepted = move.propose(model, state)
    344                     state.random_state = self.random_state
    345

~/miniconda3/envs/halomod/lib/python3.7/site-packages/emcee/moves/red_blue.py in propose(self, model, state)
     91
     92             # Compute the lnprobs of the proposed position.
---> 93             new_log_probs, new_blobs = model.compute_log_prob_fn(q)
     94
     95             # Loop over the walkers and update them accordingly.

~/miniconda3/envs/halomod/lib/python3.7/site-packages/emcee/ensemble.py in compute_log_prob(self, coords)
    443                 except ValueError:
    444                     dt = np.dtype("object")
--> 445             blob = np.array(blob, dtype=dt)
    446
    447             # Deal with single blobs properly

ValueError: setting an array element with a sequence.

Example 2:
New log_prob where one of the blobs is an array:
def log_prob(p, data):
    mu, sig = p

    # "prior"
    if not 0 < mu < 1:
        return -np.inf, -np.inf

    ll = np.sum(-(data - mu)**2 / (2 * sig**2)) - sig

    return ll, mu*sig, mu+sig, (data -mu)/sig

Exits with
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-35-c79fbf8d39e5> in <module>
----> 1 sampler.run_mcmc(init_pos, nsteps=1000, progress=True)

~/miniconda3/envs/halomod/lib/python3.7/site-packages/emcee/ensemble.py in run_mcmc(self, initial_state, nsteps, **kwargs)
    382
    383         results = None
--> 384         for results in self.sample(initial_state, iterations=nsteps, **kwargs):
    385             pass
    386

~/miniconda3/envs/halomod/lib/python3.7/site-packages/emcee/ensemble.py in sample(self, initial_state, log_prob0, rstate0, blobs0, iterations, tune, skip_initial_state_check, thin_by, thin, store, progress)
    341
    342                     # Propose
--> 343                     state, accepted = move.propose(model, state)
    344                     state.random_state = self.random_state
    345

~/miniconda3/envs/halomod/lib/python3.7/site-packages/emcee/moves/red_blue.py in propose(self, model, state)
    102
    103             new_state = State(q, log_prob=new_log_probs, blobs=new_blobs)
--> 104             state = self.update(state, new_state, accepted, S1)
    105
    106         return state, accepted

~/miniconda3/envs/halomod/lib/python3.7/site-packages/emcee/moves/move.py in update(self, old_state, new_state, accepted, subset)
     41                     "blobs at that position."
     42                 )
---> 43             old_state.blobs[m1] = new_state.blobs[m2]
     44
     45         return old_state

ValueError: shape mismatch: value array of shape (41,) could not be broadcast to indexing result of shape (41,3)

Example 3:
Using the same log prob as example 2, but with a HDF5 backend, and sampler defined as:
backend = emcee.backends.HDFBackend("backend.h5")
sampler = emcee.EnsembleSampler(
    nwalkers = 100,
    ndim = 2,
    log_prob_fn = log_prob,
    args=(data,),
    backend=backend,
    blobs_dtype=[('mu*sig', float), ('mu+sig', float), ('z', (float, 25))]
)

Exits with
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-43-c79fbf8d39e5> in <module>
----> 1 sampler.run_mcmc(init_pos, nsteps=1000, progress=True)

~/miniconda3/envs/halomod/lib/python3.7/site-packages/emcee/ensemble.py in run_mcmc(self, initial_state, nsteps, **kwargs)
    382
    383         results = None
--> 384         for results in self.sample(initial_state, iterations=nsteps, **kwargs):
    385             pass
    386

~/miniconda3/envs/halomod/lib/python3.7/site-packages/emcee/ensemble.py in sample(self, initial_state, log_prob0, rstate0, blobs0, iterations, tune, skip_initial_state_check, thin_by, thin, store, progress)
    341
    342                     # Propose
--> 343                     state, accepted = move.propose(model, state)
    344                     state.random_state = self.random_state
    345

~/miniconda3/envs/halomod/lib/python3.7/site-packages/emcee/moves/red_blue.py in propose(self, model, state)
     91
     92             # Compute the lnprobs of the proposed position.
---> 93             new_log_probs, new_blobs = model.compute_log_prob_fn(q)
     94
     95             # Loop over the walkers and update them accordingly.

~/miniconda3/envs/halomod/lib/python3.7/site-packages/emcee/ensemble.py in compute_log_prob(self, coords)
    443                 except ValueError:
    444                     dt = np.dtype("object")
--> 445             blob = np.array(blob, dtype=dt)
    446
    447             # Deal with single blobs properly

ValueError: could not assign tuple of length 2 to structure with 3 fields.

If I explicitly return -np.inf, -np.inf, -np.inf, np.zeros(25) when the prior is infinite, then it works fine. However, returning an array of zeros of length 1 but with the correct dtype does not work.
This makes it kind of hard to dynamically return blobs. What I could do is create the relevant blob dtype, then unpack it as a tuple, add the likelihood to the front of the tuple, and return that. In this case, I'd have to write the following:
def log_prob(p, data, blobs_dtype):
    mu, sig = p

    # "prior"
    if not 0 < mu < 1:
        ll = -np.inf
        blobs = np.zeros(1, dtype=blobs_dtype)
        return (ll, ) + tuple(blobs[name] for name in blobs.dtype.names)

    ll = np.sum(-(data - mu)**2 / (2 * sig**2)) - sig

    return ll, mu*sig, mu+sig, (data - mu)/sig

backend = emcee.backends.HDFBackend("backend.h5")
sampler = emcee.EnsembleSampler(
    nwalkers = 100,
    ndim = 2,
    log_prob_fn = log_prob,
    args=(data, blobs_dtype),
    backend=backend,
    blobs_dtype=blobs_dtype
)

This works, but is clearly a lot more brittle and complex. In cases like these, we know that the blobs will never actually be saved because the likelihood is -np.inf anyway. I wonder if you couldn't just ignore the blobs in these cases?

Thanks for reporting! You're absolutely right that there are several typos in the docs. The first example should read:
def log_prob(params):
    lp = log_prior(params)
    if not np.isfinite(lp):
        return -np.inf, None
    ll = log_like(params)
    if not np.isfinite(ll):
        return -np.inf, None
    return lp + ll, lp

And the second should read:
def log_prob(params):
    lp = log_prior(params)
    if not np.isfinite(lp):
        return -np.inf, None, None
    ll = log_like(params)
    if not np.isfinite(ll):
        return -np.inf, None, None
    return lp + ll, lp, np.mean(params)

In summary:

The number of returned objects should be the same, but if the log prob is -np.inf these returned objects can be anything.
You should definitely return -np.inf when the log likelihood fails - that was a typo in the docs!

Let me know if this helps your use cases and we can update the docs.

Ah, you're right. Returning None for each of the blobs works both with and without setting blobs_dtype, and that is easy to do (even dynamically). However, one strange case: setting the blobs to each be 0 (since you say it doesn't matter what they are) without the blobs_dtype does result in an error:
~/miniconda3/envs/halomod/lib/python3.7/site-packages/emcee/ensemble.py in compute_log_prob(self, coords)
    443                 except ValueError:
    444                     dt = np.dtype("object")
--> 445             blob = np.array(blob, dtype=dt)
    446
    447             # Deal with single blobs properly

ValueError: setting an array element with a sequence.

I'm not sure why, exactly, but it's easily avoidable, just by returning None. That should be made explicit in the docs though!

Can you post exactly the code you're running to get this, with a fixed seed and all the relevant imports and data, etc? I wouldn't expect this to ever be hit: it shouldn't inspect the blobs if logprob is -np.inf and it should throw an error earlier if your initial coordinates are invalid so this suggests that there might be a bug.

Sure!
import emcee
import numpy as np

def log_prob(p):
    mu, sig = p

    if not 0 < mu < 1:
        ll = -np.inf
        return -np.inf, 0,0,0

    ll = np.sum(-(data - mu)**2 / (2 * sig**2)) - sig

    return ll, mu*sig, mu+sig, (data - mu)/sig

np.random.seed(1234)
data = np.random.normal(loc=0.2, scale=0.3, size=25)

sampler = emcee.EnsembleSampler(
    nwalkers = 100,
    ndim = 2,
    log_prob_fn = log_prob,
)

init_pos = np.array([0.2, 0.3]) + 1e-4 * np.random.normal(size=(100, 2))

sampler.run_mcmc(init_pos, nsteps=1000, progress=True)

Hi, I'm very glad I found this issue after also being confused by the blobs documentation. It has a lot of info that could be useful to add in the docs. Here's what I gathered so far (assuming I understand the intended behaviour correctly):

If your log probability return more than 1 argument, the following arguments are automatically assumed to be blobs
If you don't provide blobs_dtype, the dtype of the extra args is automatically guessed on the first pass
If your log probability returns -np.inf, the blobs are also not inspected and can be anything (but you should return the same number of arguments)

Now regarding 3. I think there might be a bug indeed. Playing with the example from the previous post, it seems that return -np.inf, 0, 0, 0 fails because the the last return is not the same shape as (data - mu)/sig. This is surprising if it's true the blobs are not inspected?  Here's a MWE:
import emcee
import numpy as np

def log_prob(p):
    mu, sig = p

    if not 0 < mu < 1:
        #return -np.inf, None, None, None          # works
        #return -np.inf, 0, 0, np.zeros_like(data) # works
        return -np.inf, 0, 0, 0                    # fails (unless blobs_dtype='object' is explicilty set)

    ll = np.sum(-(data - mu)**2 / (2 * sig**2)) - sig

    return ll, mu*sig, mu+sig, (data - mu)/sig

np.random.seed(1234)
data = np.random.normal(loc=0.2, scale=0.3, size=25)

sampler = emcee.EnsembleSampler(
    nwalkers = 100,
    ndim = 2,
    log_prob_fn = log_prob,
    #blobs_dtype='object'
)

init_pos = np.array([0.2, 0.3]) + 1e-4 * np.random.normal(size=(100, 2))

state = sampler.run_mcmc(init_pos, nsteps=100, progress=True)

Some of this is the result of how numpy has changed their handling of dtypes (for the better!) since I originally wrote this forever ago. But either way, I'd say that the best approach in this particular case (whenever you have non-trival blob types) is to provide the dtype explicitly. And don't use object! Instead, here I would use:
blobs_dtype=np.dtype([("a", float), ("b", float), ("c", float, len(data))])

Then your resulting blobs array with be a structured array that will be easy to slice, etc.


Feat/mqtt executor

This version is a basic functioning mqtt executor. Update now supports multiple topic subscriptions and avoids the global default handler which could have caused unexpected test results.

@SimonT90poe I think you merged upstream changes into your branch. Please start afresh from the current master, since most probably they won't accept this PR, extract your two commits and add them on top of their latest master.


luci-app-uhttpd: certificate params hidden; allow user to define defaults from GUI

Issue template:
I installed this a while ago, and uhttpd config is old, but good. The new config defaults contain certificate generation defaults also. But for those who have older configs (which lack the config cert defaults section), the GUI shows nothing about certificates. Just uHTTPd Self-signed Certificate Parameters. This affects users and requires help interactions.
Steps to reproduce:
Remove the config cert defaults section from /etc/config/uhttpd. You're my prisoner now.
Expected behavior:
User should be able to define some defaults in the GUI if the defaults are absent from the config file.
Additional Information:
OpenWrt version information from system /etc/openwrt_release
DISTRIB_ID='OpenWrt'
DISTRIB_RELEASE='23.05.2'
DISTRIB_REVISION='r23630-842932a63d'
DISTRIB_TARGET='ath79/generic'
DISTRIB_ARCH='mips_24kc'
DISTRIB_DESCRIPTION='OpenWrt 23.05.2 r23630-842932a63d'
DISTRIB_TAINTS=''

cert defaults are present since lede v17, year 2017.


Fix deprecations reported in GH actions

It looks like set-output is now deprecated and needs to be replaces, see https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

Fixed by 1d1795bc53eecbd3eb4b89929054d180e422765b


dpi and bit depth change

when i read an image with 96 dpi, 24 bits then save it, the result is 72 dpi 32 bits
from PIL import Image
import pillow_avif
image = Image.open("img.jpg")

compression_quality = 90  # Example quality level

image.save("output.avif", format="AVIF", quality=compression_quality)

it turns out that even if another program is used it does the same things, the colors look a little bit different converting from jpeg to avif, but converting back to jpeg makes them look like the original jpeg


Reduce request size to gitlab groups api

Currently, the groups/${group_slug} API returns too big data. We have to find a way to reduce the size.

closed by #54


Feature/934 validators filters

Code looks good and working well for me, only other thing I noticed is that the HNT amount for the validator list item doesn't convert to local currency when using the toggle. Thats the way it works it is on the consensus list too though.

oh, good catch. I'll fix that.

Code looks good and working well for me, only other thing I noticed is that the HNT amount for the validator list item doesn't convert to local currency when using the toggle. Thats the way it works it is on the consensus list too though.

oh, good catch. I'll fix that.

Looks good to merge when you are ready!


WAIT with no commands has logical issues with testing

For testing the WAIT command: WAIT numreplicas timeout, when the number of numreplicas is reached, it can be returned, and any value from numreplicas to the maximum number of replicas is possible. However, the testing task requires reaching the maximum number of replicas to consider it the correct testing logic.
Specific command information can be viewed at this website address：https://redis.io/commands/wait/

Hey @youngk1019, thanks for reporting this. I think our expectation here adheres to the Redis spec:

Since WAIT returns the number of replicas reached both in case of failure and success, the client should check that the returned value is equal or greater to the replication level it demanded.

We run our tester against the official Redis implementation periodically to avoid bugs of this nature.
Feel free to re-open if you still think there's a mistake here!