[CLOSED] Java Unsatisfied Link Error Exception

 Issue by richardpayne
Monday Feb 08, 2016 at 15:51 GMT
Originally opened as https://github.com/into-cps/INTO-CPS_Application/issues/1

I get the following error when trying to launch the COE with 32-bit Java on Windows.
C:\Users\Richard\Desktop\threetank_co_3danim>"C:\Program Files (x86)\Java\jre1.8.0_73\bin\java.exe" -jar coe-0.0.1-SNAPSHOT-jar-with-dependencies.jar
COE(Command line options: [port]
INFO [main] (CoeMain.java:29) - Checking native FMI support
Failed to load FMI API
ERROR [main] (CoeMain.java:35) - Failed to load FMI API
java.lang.UnsatisfiedLinkError: C:\Users\Richard\AppData\Local\Temp\jnifmuapi-Windows-x86\fmuapi.dll: Operation did not complete successfully because the file contains a virus or potentially unwanted software
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(Unknown Source)
at java.lang.ClassLoader.loadLibrary(Unknown Source)
at java.lang.Runtime.loadLibrary0(Unknown Source)
at java.lang.System.loadLibrary(Unknown Source)
at org.intocps.fmi.jnifmuapi.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:61)
at org.intocps.fmi.jnifmuapi.NativeFmu.(NativeFmu.java:26)
at org.intocps.fmi.jnifmuapi.Factory.checkApi(Factory.java:30)
at org.intocps.orchestration.coe.CoeMain.main(CoeMain.java:31)
I also get a Windows Defender popup saying that Malwatre has been detected.
Cheers, Richard

 Comment by CThuleHansen
Wednesday Feb 10, 2016 at 06:59 GMT

Hi Richard,
Can you try running the COE as administrator?
Cheers,
Casper

 Comment by richardpayne
Wednesday Feb 10, 2016 at 09:59 GMT

I have done this (I now have to use Java 8.74 32 bit as Windows 10 refuses to run 8.73 since yesterday), and now I get a new error. I am still warned about running the COE in 32 bit, but I can accept that and move one.
The new error may be due to the 3DAnimation FMU:
C:\Users\Richard\Desktop\threetank_co_3danim>"C:\Program Files (x86)\Java\jre1.8.0_74\bin\java.exe" -jar coe-0.0.1-SNAPSHOT-jar-with-dependencies.jar
COE(Command line options: [port]
INFO [main] (CoeMain.java:29) - Checking native FMI support
Now running on port 8082
Hit Enter to stop.
INFO [Thread-1] (CoeRequestHandler.java:287) - Using Fixed-step size calculator with size = 0.02
[WaterTankController.fmu, WaterTanks1.fmu, WaterTanks2.fmu, 3DAnimationFMU.fmu]
[{8c4e810f-3df3-4a00-8276-176fa3c9f003}.controller.wt3_valve -> {25941611-1350-4ca2-8fe8-85d9193c14fd}.tank3.valveControl, {8c4e810f-3df3-4a00-8276-176fa3c9f003}.controller.wt3_valve -> {946f9503-6eb1-41de-8ad1-5e0b8fc5ee34}.3DAnimationFMU.animation.tank3.valve.control, {dcd729ec-423a-4a0d-8030-4c42a840abba}.tank1and2.Tank2OutFlow -> {25941611-1350-4ca2-8fe8-85d9193c14fd}.tank3.inFlow, {dcd729ec-423a-4a0d-8030-4c42a840abba}.tank1and2.Tank2OutFlow -> {946f9503-6eb1-41de-8ad1-5e0b8fc5ee34}.3DAnimationFMU.animation.tank2.outflow, {25941611-1350-4ca2-8fe8-85d9193c14fd}.tank3.level -> {8c4e810f-3df3-4a00-8276-176fa3c9f003}.controller.wt3_level, {25941611-1350-4ca2-8fe8-85d9193c14fd}.tank3.level -> {946f9503-6eb1-41de-8ad1-5e0b8fc5ee34}.3DAnimationFMU.animation.tank3.waterlevel, {dcd729ec-423a-4a0d-8030-4c42a840abba}.tank1and2.Tank1InFlow -> {946f9503-6eb1-41de-8ad1-5e0b8fc5ee34}.3DAnimationFMU.animation.tank1.inflow, {dcd729ec-423a-4a0d-8030-4c42a840abba}.tank1and2.Tank1WaterLevel -> {946f9503-6eb1-41de-8ad1-5e0b8fc5ee34}.3DAnimationFMU.animation.tank1.waterlevel, {dcd729ec-423a-4a0d-8030-4c42a840abba}.tank1and2.Tank2WaterLevel -> {946f9503-6eb1-41de-8ad1-5e0b8fc5ee34}.3DAnimationFMU.animation.tank2.waterlevel, {25941611-1350-4ca2-8fe8-85d9193c14fd}.tank3.Tank3OutFlow -> {946f9503-6eb1-41de-8ad1-5e0b8fc5ee34}.3DAnimationFMU.animation.tank3.outflow, {25941611-1350-4ca2-8fe8-85d9193c14fd}.tank3.puddle -> {946f9503-6eb1-41de-8ad1-5e0b8fc5ee34}.3DAnimationFMU.animation.drain.puddle]
[org.intocps.orchestration.coe.config.ModelParameter@6db05c, org.intocps.orchestration.coe.config.ModelParameter@1d03ea8]
INFO [Thread-1] (CoeInitialize.scala:54) - loading fmus
TRACE [Thread-1] (CoeInitialize.scala:209) - Set({8c4e810f-3df3-4a00-8276-176fa3c9f003}.controller, {dcd729ec-423a-4a0d-8030-4c42a840abba}.tank1and2, {25941611-1350-4ca2-8fe8-85d9193c14fd}.tank3)
INFO [Thread-1] (CoeInitialize.scala:213) - Outputs from: {8c4e810f-3df3-4a00-8276-176fa3c9f003}.controller = Set(wt3_valve)
INFO [Thread-1] (CoeInitialize.scala:213) - Outputs from: {dcd729ec-423a-4a0d-8030-4c42a840abba}.tank1and2 = Set(Tank2OutFlow, Tank1InFlow, Tank1WaterLevel, Tank2WaterLevel)
INFO [Thread-1] (CoeInitialize.scala:213) - Outputs from: {25941611-1350-4ca2-8fe8-85d9193c14fd}.tank3 = Set(level, Tank3OutFlow, puddle)
INFO [Thread-1] (CoeInitialize.scala:218) - Inputs for: {25941611-1350-4ca2-8fe8-85d9193c14fd}.tank3
INFO [Thread-1] (CoeInitialize.scala:220) -    {8c4e810f-3df3-4a00-8276-176fa3c9f003}.controller.wt3_valve --> valveControl
INFO [Thread-1] (CoeInitialize.scala:220) -    {dcd729ec-423a-4a0d-8030-4c42a840abba}.tank1and2.Tank2OutFlow --> inFlow
INFO [Thread-1] (CoeInitialize.scala:218) - Inputs for: {946f9503-6eb1-41de-8ad1-5e0b8fc5ee34}.3DAnimationFMU
INFO [Thread-1] (CoeInitialize.scala:220) -    {25941611-1350-4ca2-8fe8-85d9193c14fd}.tank3.level --> animation.tank3.waterlevel
INFO [Thread-1] (CoeInitialize.scala:220) -    {dcd729ec-423a-4a0d-8030-4c42a840abba}.tank1and2.Tank2WaterLevel --> animation.tank2.waterlevel
INFO [Thread-1] (CoeInitialize.scala:220) -    {25941611-1350-4ca2-8fe8-85d9193c14fd}.tank3.Tank3OutFlow --> animation.tank3.outflow
INFO [Thread-1] (CoeInitialize.scala:220) -    {dcd729ec-423a-4a0d-8030-4c42a840abba}.tank1and2.Tank2OutFlow --> animation.tank2.outflow
INFO [Thread-1] (CoeInitialize.scala:220) -    {8c4e810f-3df3-4a00-8276-176fa3c9f003}.controller.wt3_valve --> animation.tank3.valve.control
INFO [Thread-1] (CoeInitialize.scala:220) -    {dcd729ec-423a-4a0d-8030-4c42a840abba}.tank1and2.Tank1WaterLevel --> animation.tank1.waterlevel
INFO [Thread-1] (CoeInitialize.scala:220) -    {25941611-1350-4ca2-8fe8-85d9193c14fd}.tank3.puddle --> animation.drain.puddle
INFO [Thread-1] (CoeInitialize.scala:220) -    {dcd729ec-423a-4a0d-8030-4c42a840abba}.tank1and2.Tank1InFlow --> animation.tank1.inflow
INFO [Thread-1] (CoeInitialize.scala:218) - Inputs for: {8c4e810f-3df3-4a00-8276-176fa3c9f003}.controller
INFO [Thread-1] (CoeInitialize.scala:220) -    {25941611-1350-4ca2-8fe8-85d9193c14fd}.tank3.level --> wt3_level
warning: Function fmi2GetMaxStepsize not found in dll
warning: Function fmi2GetMaxStepsize not found in dll
warning: Function fmi2GetMaxStepsize not found in dll
--Launching with shared memory key: {8c4e810f-3df3-4a00-8276-176fa3c9f003}controller
Launching process: 'java' with arguments: 'java -jar crescendo-fmi-2.0.7-SNAPSHOT-jar-with-dependencies.jar -p {8c4e810f-3df3-4a00-8276-176fa3c9f003}controller' in folder 'C:\Users\Richard\AppData\Local\Temp\fmu-1455098131471\resources'
Process launched continuing main thread.
AnimationFrame::InitializeDLL(void)m_plotDLL->IsLoaded() == falsePlot Control could not be loaded.Starting Crescendo ShmServer with key: '{8c4e810f-3df3-4a00-8276-176fa3c9f003}controller'
DEBUG [main] (SharedMemory.java:27) - Loading library: lib/Windows-x86/sharedmemory.dll
DEBUG [main] (NativeLibraryLoader.java:37) - Extracting library from: lib/Windows-x86/sharedmemory.dll
java.nio.file.AccessDeniedException: C:\Users\Richard\AppData\Local\Temp\fmi-shm-Windows-x86\sharedmemory.dll
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source)
at java.nio.file.Files.deleteIfExists(Unknown Source)
at java.nio.file.Files.copy(Unknown Source)
at org.intocps.fmi.jnifmuapi.NativeLibraryLoader.extractNativeLibrary(NativeLibraryLoader.java:101)
at org.intocps.fmi.jnifmuapi.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:38)
at org.intocps.java.fmi.shm.SharedMemory.(SharedMemory.java:31)
at org.intocps.java.fmi.service.ProtocolDriver.(ProtocolDriver.java:20)
at org.crescendo.fmi.ShmServer.main(ShmServer.java:23)
ERROR [main] (SharedMemory.java:35) - Faild to load native Library file: lib/Windows-x86/sharedmemory.dll
java.io.FileNotFoundException: Unable to find the native lib: lib/Windows-x86/sharedmemory.dll
at org.intocps.fmi.jnifmuapi.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:42)
at org.intocps.java.fmi.shm.SharedMemory.(SharedMemory.java:31)
at org.intocps.java.fmi.service.ProtocolDriver.(ProtocolDriver.java:20)
at org.crescendo.fmi.ShmServer.main(ShmServer.java:23)
Exception in thread "main" java.lang.ExceptionInInitializerError
at org.intocps.java.fmi.service.ProtocolDriver.(ProtocolDriver.java:20)
at org.crescendo.fmi.ShmServer.main(ShmServer.java:23)
Caused by: java.lang.reflect.UndeclaredThrowableException
at org.intocps.java.fmi.shm.SharedMemory.(SharedMemory.java:36)
... 2 more
Caused by: java.io.FileNotFoundException: Unable to find the native lib: lib/Windows-x86/sharedmemory.dll
at org.intocps.fmi.jnifmuapi.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:42)
at org.intocps.java.fmi.shm.SharedMemory.(SharedMemory.java:31)
... 2 more
No 3D plot initialize yetExecution protection violation

A fatal error has been detected by the Java Runtime Environment:

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00000000, pid=7284, tid=10952

JRE version: Java(TM) SE Runtime Environment (8.0_74-b02) (build 1.8.0_74-b02)
Java VM: Java HotSpot(TM) Client VM (25.74-b02 mixed mode windows-x86 )
Problematic frame:
C  0x00000000

Failed to write core dump. Minidumps are not enabled by default on client versions of Windows

An error report file with more information is saved as:
C:\Users\Richard\Desktop\threetank_co_3danim\hs_err_pid7284.log

If you would like to submit a bug report, please visit:
http://bugreport.java.com/bugreport/crash.jsp
The crash happened outside the Java Virtual Machine in native code.
See problematic frame for where to report the bug.

 Comment by richardpayne
Wednesday Feb 10, 2016 at 10:03 GMT

Ignore that!
The COE seems to work ok now when bypassing the Windows security issues. I guess that this should still be resolved though.

 Comment by CThuleHansen
Wednesday Feb 10, 2016 at 10:05 GMT

Hi Richard,
That happens :)
Yes, it should definitely be resolved in the future, and it will be added to the list.
Before I close this  - It works when running as administrator?


Documentation and Packaging update (Pre mc/dc 0.0.9)

Changes:

Documentation
Paper
Github Action CIs
Packaging

Documentation

I added a contributors and users guide and made edits to reflect the changes
Added links on the side bar
Added MC/DC logo and logos of our institutions to the home page

Paper
I wrote a paper for JOSS submission and added the github workflow to compile it. The runner will be turned off after all updates are in. Please look over and see
Packaging
Packaging is provided through pip. This required a new .toml file and some creation of new accounts and API tokens to make it all work the CI runner which will deploy when a new version is detected. I also set up zenodo repo. Please note the new minimum requirements listed in the .toml file.
New build options
Due to the significant size of netgen (~300MB) and its dependencies I added visualization as an optional build. This can installed by
pip install mcdc[viz]

To make the dependencies optional I put the imports in a try and except phraseology with a warning message when going to install
I also configured it to install the documentation dependencies in the same manner
pip install mcdc[doc]

Github CIs
When pip install . is run in a local directory all the necessary dependence will be snagged as part of that operation. So I disabled those in the github runner CIs. I also included 2 new runners to build the paper and package mcdc for pypi.
README.md
I made significant changes to the README.md file to describe these changes
Next steps
After this PR is accepted I will release version 0.9.0 which will provide a final update to pip for the package
I will fix some links to the documentation.
I will submit that paper to JOSS

Please note I am an awful speller while I do have spell check in VS code, someone should really give my copy a once over.

@jpmorgan98, let's merge it in after we are back in black. Great work!

Ok we should be good note that it looks like some stuff in the black standard has changed and you might need to update your version to get it to work pip install black==24.1.0


global: max content length form parsing fix

@lnielsen it looks good. I will add just some styling comments.


fix(parking-sensors): changed from root to non-root devcontainer

⚠ BLOCKED BY: #644

Type of PR

Code changes

Purpose
In bug #646 we see a permission error on wsl when setting up the dev container with the root user
Author pre-publish checklist

[ ] Added test to prove my fix is effective or new feature works
[ ] No PII in logs
[ ] Made corresponding changes to the documentation

Validation steps
Run the devcontainer in wsl like described in the issue #646
Issues Closed or Referenced

Closes #646

Consider merging this with changes for #644

We have added this to other changes that we are rolling into updates on the DevContainer. #644


Change default of encoder_scaling_*

See title.

Coverage remained the same at 88.875% when pulling 19a67056cf1604158fd506580964cb7dcf8ee041 on RickCarlino:firmware_config_default into ea7f34c348940f22fd9a461e56cdaf28ab2a865b on FarmBot:staging.


Conditional comments will be removed from the code after obfuscation

https://github.com/javascript-obfuscator/javascript-obfuscator/issues/641

Coverage decreased (-0.04%) to 98.606% when pulling 731d7658ac301a1df40bb1be2fb8e6f82c18ab2f on remove-conditional-comments into bf7511b04e151b5974c7b90f7f6a51c61c0319c6 on master.


StructLayout trait and macro

If there is one thing that bothers me with Rust, it's that every single crate is expected to implement serde. Implementing it quickly makes the code messy, because the easiest way to implement it is using serde-derive, and that would require duplicating the struct code and changing the #[derive] line. Not fun.
Not only this, but it also gives serde an unfair heads up. Nobody can really make a new serialization library, because serde will always be the most supported.

To fix this, I am proposing a new standard trait: StructLayout.
With a single function which is expected to convert the struct to a StructMap, which is a recursive type that holds all data in a HashMap.
struct StructMap {
    name: &'static str,
    child_maps: HashMap<&'static str, StructMap>,
    current_map: HashMap<&'static str, String>
}

example representation of StructMap
This could be compared to reflection, but instead of always storing useless values in memory, you create them when necessary.
Playground Example

EDIT 1: Modified comment in playground example
EDIT 2: Ran RustFmt

Do you have a particular technical issue with Serde? Is there something Serde doesn't offer that makes you want an alternative serialization and deserialization library?
Additionally, it seems like the recommendation given here results in a lot of allocations, which many Rust programs would want to / need to avoid.

I have no particular issue with Serde apart from the fact that most libraries are expected to implement it separately as a feature which can easily become quite messy.
My idea does have a lot of allocations, yes. This is why you would preferrably not use it unless it becomes absolutely necessary. Sort of like with reflection. Slow, but sometimes a necessary evil.

I guess I just don't see where the need for such a generic representation is, nor do I think that the representation is the right one if a generic stdlib representation is needed.
Serde works really well, and is well tested.

A better way to solve this issue might be to make a way to derive stuff by default, then? So Serde could basically say "all types, no matter what crate, should implement serde-derive". And then of course optimize away unused derives.

If there is one thing that bothers me with Rust, it's that every single crate is expected to implement serde.

They're not. You can use whatever serialization crate you want. Serde just happens to be well polished. On top of that you don't need to use serialization for every crate, so that term is a bit broad here.

Implementing it quickly makes the code messy, because the easiest way to implement it is using serde-derive, and that would require duplicating the struct code and changing the #[derive] line. Not fun.

Why is this messy? I'm unsure what you're trying to say here

Not only this, but it also gives serde an unfair heads up. Nobody can really make a new serialization library, because serde will always be the most supported.

That's all libraries. If something better comes around I'm sure people will switch to it. Just look at Rocket which blew libraries like Iron and Nickel out of the water for webframeworks with Rust.

To fix this, I am proposing a new standard trait: StructLayout.
With a single function which is expected to convert the struct to a StructMap, which is a recursive type that holds all data in a HashMap.

struct StructMap {
    name: &'static str,
    child_maps: HashMap<&'static str, StructMap>,
    current_map: HashMap<&'static str, String>
}

example representation of StructMap

The amount of allocations on this would be gigantic. What if I don't want String? What if I don't want strings as keys. This isn't generic at all.  How is this going to handle ownership? I can already see quite a few things that will be hard to implement let alone get working with this. You'll likely need Box and some things that this doesn't account for.

EDIT 2: Ran RustFmt to not trigger anybody
Seriously? This kind of attitude and language is against the code of conduct.

To be honest I'm legitimately confused as to what the issue with serde-derive is. You only need it because you have to implement Serializable and DeSerializable for their libraries. It makes it easy to handle and work with. You'll have a similar issue even if it was another library. serde-derive just makes it easier. I just don't think there's a solid argument or design to add something like this to stdlib.

There are certain types which should never be serialized or deserialized. Automatically implementing these operations for all types is not a good idea.

@mgattozzi

Why is this messy? I'm unsure what you're trying to say here

Say I have
#[derive(Clone)]
pub struct Test {
    test: bool,
}

To optionally implement serde, I would have to
#[cfg(not(feature = "serde"))]
#[derive(Clone)]
pub struct Test {
    test: bool,
}
#[cfg(feature = "serde")]
#[derive(Clone, Serialize, Deserialize)]
pub struct Test {
    test: bool,
}

Seriously? This kind of attitude and language is against the code of conduct.

What did I say wrong? :(
I was a joke, but I removed it now ;-;
@AndrewBrinker
Examples?

Something like this should work for conditionally implementing serialize/deserialize.
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
#[derive(Clone)]
pub struct Test {
    test: bool,
}

@Mark-Simulacrum Heh, I now feel like the stupidest person on earth. Sorry for bothering all of you with yet another one of my lame ideas because I don't know enough Rust.

Well first, there are multiple items on the Common Weakness Enumeration list regarding serialization and deserialization:

CWE-499: Serializable Class Containing Sensitive Data
CWE-502: CWE-502: Deserialization of Untrusted Data

As these items make clear, you don't want to be able to accidentally serialize sensitive data. Similarly, you don't want to deserialize untrusted data if your deserialization system allows for code execution.

Is it possible that what you actually want is a better solution to the "I want Foo to implement serde::Deserialize but I don't own Foo" problem than the current de facto standard of "everyone who owns useful-to-serialize types implements serde::Serialize"?
There have been suggestions in the past about adding alternative workarounds to the orphan rules, such as allowing you to implement serde::Serialize for Foo in your own crate, and just having that implementation only exist within your one crate so it can't possibly cause coherence issues in anyone else's code.

@Ixrec Hmm... that'd be cool, but would it work with derive?

afaik it's completely orthogonal to whether a trait impl is written out by hand or generated by a #[derive()]

@legolord208, people don't test their code (certainly not well enough), and turning something from a static failure ("this type doesn't implement Serialize") into a dynamic failure ("you forgot to test everything to make sure this sensitive data doesn't get leaked") is definitely not desirable.

@AndrewBrinker I guess I just wish I could serialize any object without too much effort lol. But on the other hand you're right. Compile time errors FTW!


Log checks API result before status assertion

High-level description
This changes the checks API test to log the result before it attempts to assert on the statuses of the checks. If there are any issues, this makes it possible to look at the logs and see the output of any failed checks.
Corresponding DC/OS tickets (obligatory)
These DC/OS JIRA ticket(s) must be updated (ideally closed) in the moment this PR lands:

DCOS-51593

Checklist for all PRs

[X] Added a comprehensible changelog entry to CHANGES.md or explain why this is not a user-facing change:
[X] Included a test which will fail if code is reverted but test is not. If there is no test please explain here:
[X] Read the DC/OS contributing guidelines
[X] Followed relevant code rules Rules for Packages and Systemd

Checklist for component/package updates:
If you are changing components or packages in DC/OS (e.g. you are bumping the sha or ref of anything underneath packages), then in addition to the above please also include:

[ ] Change log from the last version integrated (this should be a link to commits for easy verification and review): example
[ ] Test Results: [link to CI job test results for component]
[ ] Code Coverage (if available): [link to code coverage report]

PLEASE FILL IN THE TEMPLATE ABOVE / DO NOT REMOVE ANY SECTIONS ABOVE THIS LINE
Instructions and review process
What is the review process and when will my changes land?
All PRs require 2 approvals using GitHub's pull request reviews.
Reviewers should be:

Developers who understand the code being modified.
Developers responsible for code that interacts with or depends on the code being modified.

It is best to proactively ask for 2 reviews by @mentioning the candidate reviewers in the PR comments area. The responsibility is on the developer submitting the PR to follow-up with reviewers and make sure a PR is reviewed in a timely manner. Once a PR has 2 ship-it's, no red reviews, and all tests are green it will be included in the next train.

@mesosphere-mergebot label Ready for review

@drewvanstone @janisz @makkes can you please review this?


testing github bot

@objmagic - are you sure this is the right title for the PR? the changes seem to be reflecting dry run.

@kramasamy just opening a random PR to test bot. bot is running on aws now and has nothing to do with the Heron project itself.

@objmagic - got it.


Benchmark faster whisper turbo v3

#WIP
Benchmark with faster-whisper-large-v3-turbo-ct2
For reference, here's the time and memory usage that are required to transcribe 13 minutes of audio using different implementations:

openai/whisper@25639fc
faster-whisper@d57c5b4

Large-v3 model on GPU

Implementation
Precision
Beam size
Time
Max. GPU memory
Max. CPU memory
WER %

openai/whisper-large-v3
fp16
5
2m23s
MB
MB

openai/whisper-turbo
fp16
5
39s
MB
MB

faster-whisper
fp16
5
52.023s
4521MB
901MB
2.883

faster-whisper
int8
5
52.639s
2953MB
2261MB
4.594

faster-distil-large-v3
fp16
5
26.126s
2409MB
900MB
2.392

faster-distil-large-v3
int8
5
22.537s
1481MB
1468MB
2.392

faster-large-v3-turbo
fp16
5
19.155s
2537MB
899MB
1.919

faster-large-v3-turbo
int8
5
19.591s
1545MB
1526MB
1.919

WER on librispeech clean val split.

We now support the new whisper-large-v3-turbo on Sieve!
Use it via sieve/speech_transcriber: https://www.sievedata.com/functions/sieve/speech_transcriber
Use sieve/whisper directly: https://www.sievedata.com/functions/sieve/whisper
Just set speed_boost to True. API guide is under "Usage Guide" tab.

Would be great if medium (or more sizes) are added for comparison! In OpenAI's implementation, turbo is 8x faster than v3, while medium is 2x and base is 7x, while offering similar WER as large-v2, which sounds surreal. Wonder how it translate to FW version.

Would be great if medium (or more sizes) are added for comparison! In OpenAI's implementation, turbo is 8x faster than v3 (medium is 2x and base is 7x), while offering similar WER as large-v2, which sounds surreal. Wonder how it translate to FW version.

In my test, with the same 10-minute audio, Medium took 52 seconds, and Turbo took 39 seconds.

You may find this discussion helpful:
https://github.com/openai/whisper/discussions/2363#discussion-7264254

Compared to Medium, Turbo and large-v3 has a more advanced timeline. Subtitles generated by Turbo appear earlier, but end precisely on time. Medium subtitles also appear early, but to a much lesser extent than Turbo. However, Medium subtitles are delayed in disappearing.

Compared to Medium, Turbo and large-v3 has a more advanced timeline. Subtitles generated by Turbo appear earlier, but end precisely on time. Medium subtitles also appear early, but to a much lesser extent than Turbo. However, Medium subtitles are delayed in disappearing.
I find that subtitles disappearing late is a better experience than them appearing early. I should still use medium.

or you could use a forced alignment model after the transcription, much better timings than whisper

Great, thanks for your efforts! I hope turbo will be officially added here soon!
https://github.com/SYSTRAN/faster-whisper/blob/d57c5b40b06e59ec44240d93485a95799548af50/faster_whisper/utils.py#L12C1-L29C2

I benchmarked the models on my laptop using the same audio file in both sequential and batched processing. Seems that large-v3-turbo generally performs exceptionally well, offering greater accuracy than the base model while maintaining efficient processing times.
System Specifications

CPU: Intel Core i7-12650H
GPU: NVIDIA GeForce RTX 3060 Laptop (6 GB VRAM)

Benchmark Details

All models were tested with int4 precision.
WER (Word Error Rate) was calculated by comparing the original French subtitles of a video with the transcriptions generated by the models.
The language was explicitly set to French to prevent any translation errors or incorrect transcriptions.

Sequential Processing Benchmark

Model
WER (%)
Total Time (s)
Transcribe Time (s)
Model Load Time (s)

tiny
24.1%
28.95
28.44
0.51

base
16.0%
33.42
32.72
0.70

small
10.5%
55.62
53.21
2.41

medium
10.7%
113.25
106.30
6.95

large
17.6%
240.52
227.31
13.20

large-v1
8.7%
168.58
155.14
13.44

large-v2
8.5%
178.28
164.74
13.53

large-v3
17.6%
230.77
217.43
13.34

large-v3-turbo
9.5%
46.14
38.99
7.15

Observations:

The large-v3-turbo model achieves a WER of 9.5%, which is significantly better than the base model and comparable to large-v2.
In terms of speed, large-v3-turbo completes transcription in 38.99 seconds, much faster than other large models.

Batched Processing Benchmark
For batched processing, I used 10 batches for each model. I tried to use 16 batches, but some models thrown out-of-memory (OOM) errors due to the 6 GB VRAM limit.

Model
WER (%)
Total Time (s)
Transcribe Time (s)
Model Load Time (s)

tiny
23.6%
5.48
4.56
0.92

base
16.5%
6.92
5.70
1.22

small
9.8%
12.45
9.98
2.47

medium
8.9%
26.33
19.47
6.86

large
7.9%
35.97
29.66
6.31

large-v1
12.1%
42.90
29.64
13.26

large-v2
8.8%
43.17
29.71
13.46

large-v3
7.9%
42.97
29.69
13.28

large-v3-turbo
7.7%
18.68
11.47
7.20

Observations:

With batched processing, large-v3-turbo achieves the best WER of 7.7%, outperforming all other models in both accuracy and speed.
The transcribe time for large-v3-turbo is 11.47 seconds, making it suitable for real-time applications even on a laptop GPU.

Conclusions

The large-v3-turbo model offers an excellent balance between accuracy and processing speed, especially evident in batched processing scenarios.
It outperforms the base model in terms of WER while maintaining significantly lower processing times compared to other large models.

Just chiming in, I've tried using v3-turbo for streaming and found that it hallucinates more/misses audio more than other faster-whisper models. For example for this 10 second audio clip of an obama speech: temp_audio_wav.zip Using v3-turbo:
from faster_whisper import WhisperModel
model = WhisperModel(model_size_or_path="deepdml/faster-whisper-large-v3-turbo-ct2", device="cuda", compute_type="float16")

segments, info = model.transcribe("temp_audio.wav", initial_prompt="",
                                                                max_new_tokens=224,
                                                                beam_size=5,
                                                                temperature=0,
                                                                language="en",
                                                                word_timestamps=True,
                                                                vad_filter=False)
for seg in segments:
    print("[%.2fs -> %.2fs] %s" % (seg.start, seg.end, seg.text))

produces

[0.00s -> 9.66s]  ...to give the president a chance.

Whereas using medium:
from faster_whisper import WhisperModel
model = WhisperModel(model_size_or_path="medium", device="cuda", compute_type="float16")

segments, info = model.transcribe("temp_audio.wav", initial_prompt="",
                                                                max_new_tokens=224,
                                                                beam_size=5,
                                                                temperature=0,
                                                                language="en",
                                                                word_timestamps=True,
                                                                vad_filter=False)
for seg in segments:
    print("[%.2fs -> %.2fs] %s" % (seg.start, seg.end, seg.text))

produces

[0.00s -> 1.70s]  So give the president a chance.
[2.00s -> 4.96s]  Governor Romney, I'm glad that you recognize that al-Qaeda is a threat.
[5.38s -> 9.66s]  Because a few months ago, when you were asked what's the biggest geopolitical threat facing America, you said...

Not sure if any of you are experiencing anything similar? Or maybe an official faster-whisper turbo-v3 release would perform better.

Haven't encountered that. I've tried the same audio and both models return the same transcription. I did notice that turbo model hallucinates more on noisy data than v3, but that's to be expected considering what we saw with common voice 15 benchmark.

Sharing the benchmarking results with Turbo model compared to other large whisper models on one of the biggest open source long-form ASR evaluation dataset.
Our tests were conducted on a subset of YouTube-commons: yotutube-commons-asr-eval ).

Model
WER
Speed

large v3-turbo
13.40%
129.5x

large v3
13.20%
55.3x

large v2
14.10%
54.6x

distill-large-v3 (en)
15.00%
142.9x

The whisper-turbo model achieves a Word Error Rate (WER) similar to large models and excels in speed among multilingual models.

Also it does not seem to support translation task, even though it is mentioned.
also tried it with Transformer large-v3 turbo, same behaviour.

Also it does not seem to support translation task, even though it is mentioned. also tried it with Transformer large-v3 turbo, same behaviour.

They specifically mentioned the translation task being excluded... https://github.com/openai/whisper/discussions/2363

excluding translation data, on which we don’t expect turbo to perform well.

oh okay. i looked at the huggingface page and there the translation task is mentioned.
But maybe it was just copy pasted from the original large-v3 model.
Thanks for pointing that out.

Just chiming in, I've tried using v3-turbo for streaming and found that it hallucinates more/misses audio more than other faster-whisper models. For example for this 10 second audio clip of an obama speech: temp_audio_wav.zip Using v3-turbo:
from faster_whisper import WhisperModel
model = WhisperModel(model_size_or_path="deepdml/faster-whisper-large-v3-turbo-ct2", device="cuda", compute_type="float16")

segments, info = model.transcribe("temp_audio.wav", initial_prompt="",
                                                                max_new_tokens=224,
                                                                beam_size=5,
                                                                temperature=0,
                                                                language="en",
                                                                word_timestamps=True,
                                                                vad_filter=False)
for seg in segments:
    print("[%.2fs -> %.2fs] %s" % (seg.start, seg.end, seg.text))

produces

[0.00s -> 9.66s]  ...to give the president a chance.

Whereas using medium:
from faster_whisper import WhisperModel
model = WhisperModel(model_size_or_path="medium", device="cuda", compute_type="float16")

segments, info = model.transcribe("temp_audio.wav", initial_prompt="",
                                                                max_new_tokens=224,
                                                                beam_size=5,
                                                                temperature=0,
                                                                language="en",
                                                                word_timestamps=True,
                                                                vad_filter=False)
for seg in segments:
    print("[%.2fs -> %.2fs] %s" % (seg.start, seg.end, seg.text))

produces

[0.00s -> 1.70s]  So give the president a chance.
[2.00s -> 4.96s]  Governor Romney, I'm glad that you recognize that al-Qaeda is a threat.
[5.38s -> 9.66s]  Because a few months ago, when you were asked what's the biggest geopolitical threat facing America, you said...

Not sure if any of you are experiencing anything similar? Or maybe an official faster-whisper turbo-v3 release would perform better.

Haven't encountered that. I've tried the same audio and both models return the same transcription. I did notice that turbo model hallucinates more on noisy data than v3, but that's to be expected considering what we saw with common voice 15 benchmark.

Interesting, did you try on the audio I provided here? It's actually remarkably consistent in how it's worse at short audio clips for me (with "deepdml/faster-whisper-large-v3-turbo-ct2)

Just chiming in, I've tried using v3-turbo for streaming and found that it hallucinates more/misses audio more than other faster-whisper models. For example for this 10 second audio clip of an obama speech: temp_audio_wav.zip Using v3-turbo:
from faster_whisper import WhisperModel
model = WhisperModel(model_size_or_path="deepdml/faster-whisper-large-v3-turbo-ct2", device="cuda", compute_type="float16")

segments, info = model.transcribe("temp_audio.wav", initial_prompt="",
                                                                max_new_tokens=224,
                                                                beam_size=5,
                                                                temperature=0,
                                                                language="en",
                                                                word_timestamps=True,
                                                                vad_filter=False)
for seg in segments:
    print("[%.2fs -> %.2fs] %s" % (seg.start, seg.end, seg.text))

produces

[0.00s -> 9.66s]  ...to give the president a chance.

Whereas using medium:
from faster_whisper import WhisperModel
model = WhisperModel(model_size_or_path="medium", device="cuda", compute_type="float16")

segments, info = model.transcribe("temp_audio.wav", initial_prompt="",
                                                                max_new_tokens=224,
                                                                beam_size=5,
                                                                temperature=0,
                                                                language="en",
                                                                word_timestamps=True,
                                                                vad_filter=False)
for seg in segments:
    print("[%.2fs -> %.2fs] %s" % (seg.start, seg.end, seg.text))

produces

[0.00s -> 1.70s]  So give the president a chance.
[2.00s -> 4.96s]  Governor Romney, I'm glad that you recognize that al-Qaeda is a threat.
[5.38s -> 9.66s]  Because a few months ago, when you were asked what's the biggest geopolitical threat facing America, you said...

Not sure if any of you are experiencing anything similar? Or maybe an official faster-whisper turbo-v3 release would perform better.

Haven't encountered that. I've tried the same audio and both models return the same transcription. I did notice that turbo model hallucinates more on noisy data than v3, but that's to be expected considering what we saw with common voice 15 benchmark.

Interesting, did you try on the audio I provided here? It's actually remarkably consistent in how it's worse at short audio clips for me (with "deepdml/faster-whisper-large-v3-turbo-ct2)

Yeah, i did specifically try it on your clip with the same model. But i also did it with batched processing and not a sequential one, so i'm not sure if this is a specific issue with it or not. Batched processing with 6 - 10 batches works the best on my setup and actually provides more accurate transcriptions as you can see from my little benchmark earlier in this thread, so i use it for everything.

Thank you so much for recording these benchmarks, I almost cannot believe the speed and quality of these models.
I have one request - for anyone reading this who is benchmarking.
Could some of the people performing benchmarks please record the hardware they are benchmarking on if possible?
GPU, CPU and RAM. Such information will help to estimate the minimal cost incurred for developing a real-time ASR application; e.g. the minimal budget for hardware possible using large-v3-turbo.
Thanks in advance.

It seems that long-duration audio files cannot be processed correctly.
When I tested with an 11-hour MP3 file, the memory usage quickly spiked above 26 GB. After 3 minutes, the cli displayed “Killed” and then exited.

It seems that long-duration audio files cannot be processed correctly.
When I tested with an 11-hour MP3 file, the memory usage quickly spiked above 26 GB. After 3 minutes, the cli displayed “Killed” and then exited.

Is it a problem only with the Turbo model?

No, using “large-v3” also doesn’t work. The memory usage spikes to 27GB, and then it shows “Killed.”
model = WhisperModel("large-v3", device="cuda", compute_type="float16") segments, info = model.transcribe("11hours.mp3", word_timestamps=True)

torch.stft cause GPU OOM? I know that in the old stft matrix multiplication mel_spec = self.mel_filters @ magnitudes for long audio files would use a large amount of memory. For this reason, I've written a batch version before.
https://github.com/ben91lin/faster-whisper/blob/mochi/faster_whisper/feature_extractor.py

Why does this repos use mobiuslabsgmbh/faster-whisper-large-v3-turbo and not deepdml/faster-whisper-large-v3-turbo-ct2? And why not something like Systran/faster-whisper-v3-turbo?
See Code: https://github.com/SYSTRAN/faster-whisper/blob/master/faster_whisper/utils.py
HF-Links:
https://huggingface.co/deepdml/faster-whisper-large-v3-turbo-ct2
https://huggingface.co/mobiuslabsgmbh/faster-whisper-large-v3-turbo

Because the deepdml conversion has wrong alignment heads and tokenizer config, this will mainly affect word timestamps, the mobius labs conversion is closer to the official one

Just chiming in, I've tried using v3-turbo for streaming and found that it hallucinates more/misses audio more than other faster-whisper models. For example for this 10 second audio clip of an obama speech:

@tjongsma I could reproduce your findings (with float32 due to my graphics card, but shouldn't make a difference).
Every single point of the the following fixes that in my experimenting:

Use initial_prompt="The following is a speech:"
Use word_timestamps=False
Use mobiuslabsgmbh/faster-whisper-large-v3-turbo instead of deepdml/faster-whisper-large-v3-turbo-ct2
(Adding 5 second silence on the beginning of the audio file somewhat makes it better, but then "to give the president a chance." is missed)

( @usergit @NilaierMusic @zxl777 ) Large-v3 does work on the clip of @tjongsma :

[0.00s -> 1.58s]  So give the president a chance.
[1.68s -> 4.76s]  Governor Romney, I'm glad that you recognize that al-Qaeda is a threat.
[4.92s -> 9.58s]  Because a few months ago, when you were asked what's the biggest geopolitical threat facing America, you said

(Due my graphics card I use float32, but it probably should be the same with float16 ). But I see that OpenAI uses Large-v2 themselves for their API, see here. But that could be just because they didn't test large-v3 enough.
Why wouldn't they use large-v2 for making the turbo instead if they really saw problems with v3?
@MahmoudAshraf97 The third point of the list above confirms what you're saying. But i still wonder, why we have not something like Systran/faster-whisper-v3-turbo like the other model sizes? Also, is mobiuslabsgmbh trustworthy enough (less downloads)?

Number of downloads doesn't imply trustworthiness, deepdml has much more downloads because it was uploaded first and was shared more widely than mobiuslabs, but when I chose between the two, I used the OpenAI model as the reference and mobiuslabs was identical unlike deepdml, the difference between the two conversions is subtle that almost no one will notice any difference in performance except for some edge cases, Systran didn't upload the new model because they are busy with internal projects so community took into their own hands, having a systran conversion wouldn't make any difference though because converting a model is a single line of code that anyone can execute

Just chiming in, I've tried using v3-turbo for streaming and found that it hallucinates more/misses audio more than other faster-whisper models. For example for this 10 second audio clip of an obama speech:

@tjongsma I could reproduce your findings (with float32 due to my graphics card, but shouldn't make a difference).
Every single point of the the following fixes that in my experimenting:

Use initial_prompt="The following is a speech:"
Use word_timestamps=False
Use mobiuslabsgmbh/faster-whisper-large-v3-turbo instead of deepdml/faster-whisper-large-v3-turbo-ct2
(Adding 5 second silence on the beginning of the audio file somewhat makes it better, but then "to give the president a chance." is missed)

( @usergit @NilaierMusic @zxl777 ) Large-v3 does work on the clip of @tjongsma :

[0.00s -> 1.58s]  So give the president a chance.
[1.68s -> 4.76s]  Governor Romney, I'm glad that you recognize that al-Qaeda is a threat.
[4.92s -> 9.58s]  Because a few months ago, when you were asked what's the biggest geopolitical threat facing America, you said

(Due my graphics card I use float32, but it probably should be the same with float16 ). But I see that OpenAI uses Large-v2 themselves for their API, see here. But that could be just because they didn't test large-v3 enough.
Why wouldn't they use large-v2 for making the turbo instead if they really saw problems with v3?
@MahmoudAshraf97 The third point of the list above confirms what you're saying. But i still wonder, why we have not something like Systran/faster-whisper-v3-turbo like the other model sizes? Also, is mobiuslabsgmbh trustworthy enough (less downloads)?
Edit: Question: What thresholds to set, to get less hallucination at low sound parts?

Thanks so much for verifying @DoS007! I was a bit suspect of the deepml model at the time, but unfortunately there were no alternatives. Wil use mobiuslabsgmbh/faster-whisper-large-v3-turbo going forward!

Hi,
Any idea when Turbo V3 will be available in https://pypi.org/project/faster-whisper/ ?
As, I am interested to try it out at https://github.com/runpod-workers/worker-faster_whisper/tree/main
Thank you for all your effort.


Update the game over message

Update the game over message so people know how to play again!

It seems very well!
You won't be able to approve or request changes to your own pull request.


Mount /var/run/docker.sock from Docker host

WHY
To run risu easily by Docker Compose. /var/run/docker.sock is required to run Docker on the container.
WHAT
Mount /var/run/docker.sock on the container from Docker host

LGTM!


Send correct replacement range information (use OptionalReplacementSpan)

Raised from: https://github.com/microsoft/vscode/issues/134328
and: https://github.com/microsoft/vscode/issues/134328#issuecomment-973593757
It looks like this plugin doesn't send back the proper range information. meaning replacements and edits are a bit screwed up. OptionalReplacementSpan could help with this.
Looking at CompletionInfo theres a new property OptionalReplacementSpan I think maybe this needs adding. It could help with this issue
You can see we don't send it here:
https://github.com/microsoft/typescript-styled-plugin/blob/cd5903097be3e879051e4282ae7005ab1b41f199/src/_language-service.ts#L379-L392

From speakin to Andrew branch it sounds like this needs to be set then checked here
https://github.com/microsoft/TypeScript/blob/9766757ee6b4df479330c170bec3a8951dd95744/src/server/session.ts#L1873

See https://github.com/microsoft/vscode/issues/134328#issuecomment-974632750 for update. ReplacementSpan needs to be set inside of completionEntry


script: give Builder ability to verify-ify opcodes

What do you envision this is useful for? It doesn't seem to be a good idea to encourage people to do operations on scripts they don't fully know beforehand.

By definition a script::Builder is a script the user "doesn't fully know". If they fully knew it it would be a Script.

I don't think many people build a script if they don't know what script they are going to build beforehand.. I mean that almost all usage of Builder is with hard-coded opcodes and variable data pushes.
If you know all your script opcodes upfront, you can just as well manually verify-ify the last opcode and if you don't know all upcodes upfront, you're doing dangerous stuff.

Updated to fix bug where slices that ended in a CHECKSIG byte (or similar) would be modified by the function.
@stevenroose This function is semantically equivalent to .push_opcode(opcodes::all::OP_VERIFY) except that it saves a byte when possible, something that is subtle and difficult and tedious to do by hand. Similarly we have .push_int which finds the most efficient way to push a given integer. I really don't understand how it's dangerous for the user not to want to deal with this accounting himself.


No event for PipelineValidationFailed

Expected Behavior
When try to run a pipelinerun, it failed with the following condition:
status:
  completionTime: 2019-08-16T06:07:38Z
  conditions:
  - lastTransitionTime: 2019-08-16T06:07:38Z
    message: 'invalid input params: missing values for these params which have no
      default values: [TESTCASE]'
    reason: PipelineValidationFailed
    status: "False"
    type: Succeeded
  startTime: 2019-08-16T06:07:38Z

In this situation, I expect that there may be event about this failure.
Actual Behavior
No event in this situation.
Steps to Reproduce the Problem

Additional Info

I would be happy to help out with this one @dibyom. Can I start to work on it?

@NikeNano I think this is gonna be fixed by https://github.com/tektoncd/pipeline/issues/2082 (cc @afrittoli )

@Letty5411 thanks for reporting this and @NikeNano thanks for contributing!
There is definitely an inconsistency in how we generate emit events today, I tried to capture the complete (!) matrix in https://github.com/tektoncd/pipeline/pull/2329
So indeed I was going to address this as part of #2082 / #2039.

@Letty5411 thanks for reporting this and @NikeNano thanks for contributing!
There is definitely an inconsistency in how we generate emit events today, I tried to capture the complete (!) matrix in #2329
So indeed I was going to address this as part of #2082 / #2039.

Ok, cool. Then I see if I can find some other issues to look at :)

@afrittoli are you tackling both k8s events and cloudevents in your work?

@afrittoli are you tackling both k8s events and cloudevents in your work?

Yes, I am - first PR fixes k8s events, second PR will add cloudevents

@afrittoli is this fixed ? :upside_down_face:

Is this still the case ? (cc @afrittoli)

Is this still the case ? (cc @afrittoli)


include the json schema of funder object

The funder object was being overwritten as a string in the modify_schema class method.  I looked at the other properties that use modify_schema and they are all strings so this is the only object affected.

@Maurier - The uischema likely needs to be updated as well.  Would you mind taking a look at it?

@Maurier - The uischema likely needs to be updated as well.  Would you mind taking a look at it?

Got it. I'll taka care of it.

eh, this isn't quite fixing it.  I have to move things around a little bit

@Maurier - fixed for real this time.

@sblack-usu It looks like the overwrites are removing the required specifications in these fields. For example, the resulting schema for this field does not specify the name property as required.
"funder": {
            "title": "Funding Organization",
            "description": "The organization that provided the funding or sponsorship.",
            "type": "object",
            "properties": {
              . . .
            }
          }

@Maurier - I pushed a fix for the required issue you found.  It injects the whole schema in and then overwrites the title.


samples: n5010-ddr-test: support device selection with argsfilter

Running the n5010-ddr-test utility on machines containing multiple
accelerators requires a way to select which device to use. Implement this
using the argsfilter library.
Signed-off-by: Martin Hundebøll<EMAIL_ADDRESS>

Coverage decreased (-0.03%) to 65.749% when pulling 2e4adaad1ca9e6a33bc0704b9ba6a5898b8cc8ea on n5010-ddr-test-args-filter into 27fa6f9c78648669701102c5bc7ee361683bc45a on master.

@michelleyho @r-rojo ping


(Vulkan / RDNA2) Runtime Error with RDNA2 devices on some batch matmuls.

What happened?
Hi,
I've run into an issue on RDNA2 cards (6900xt and users report the same for other RDNA2 cards) with running the VAE for SHARK's stable diffusion upscaler app (tested on SHARK-runtime as well as stock IREE).
The specific HF model id
I've tracked down the failure to a specific dispatch which contains a batch matmul of dims 1x512x512x256.
Here is the error output:
(iree.venv) PS C:\V\s> .\iree-build\tools\iree-benchmark-module.exe --module=upscaler_vae_27.vmfb --device=vulkan
C:\V\s\runtime\src\iree\hal\drivers\vulkan\native_executable.cc:157: UNKNOWN; VkResult=4294967283; while invoking native function hal.executable.create; while calling import;
[ 1]   native hal.executable.create:0 -
[ 0] bytecode module@1:254 -

This is what the .vmfb was compiled with:
(iree.venv) PS C:\V\s> .\iree-build\tools\iree-compile.exe "module_forward_dispatch_27_vulkan_spirv_fb.mlir" --iree-input-type=tm_tensor --iree-vm-bytecode-module-output-format=flatbuffer-binary --iree-hal-target-backends=vulkan --mlir-print-debuginfo --mlir-print-op-on-diagnostic=false --mlir-pass-pipeline-crash-reproducer=./iree_tmp/core-reproducer.mlir --iree-llvmcpu-target-cpu-features=host --iree-vulkan-target-env="#vk.target_env<v1.3, r(120), [VK_KHR_16bit_storage, VK_KHR_8bit_storage, VK_KHR_shader_float16_int8, VK_KHR_spirv_1_4, VK_KHR_storage_buffer_storage_class, VK_KHR_variable_pointers, VK_EXT_subgroup_size_control], AMD:DiscreteGPU, #vk.caps< maxComputeSharedMemorySize = 65536, maxComputeWorkGroupInvocations = 1024, maxComputeWorkGroupSize = dense<[1024, 1024, 1024]>: vector<3xi32>, subgroupSize = 64, subgroupFeatures = 255: i32, minSubgroupSize = 32, maxSubgroupSize = 64, shaderFloat16 = unit, shaderFloat64 = unit, shaderInt8 = unit, shaderInt16 = unit, shaderInt64 = unit, storageBuffer16BitAccess = unit, storagePushConstant16 = unit, uniformAndStorageBuffer16BitAccess = unit, storageBuffer8BitAccess = unit, storagePushConstant8 = unit, uniformAndStorageBuffer8BitAccess = unit, variablePointers = unit, variablePointersStorageBuffer = unit >>" --iree-stream-resource-index-bits=64 --iree-vm-target-index-bits=64 --iree-vm-bytecode-module-strip-source-map=true --iree-util-zero-fill-elided-attrs -iree-vulkan-target-triple=rdna2-unknown-windows -o upscaler_vae_27.vmfb

The dispatch .mlir can be downloaded here: module_forward_dispatch_27_vulkan_spirv_fb.mlir
The .vmfb is here: upscaler_vae_27.vmfb
The e2e .mlir for the 128x128 fp32 upscaler VAE model lowered through torch-mlir can be found here: core-input.mlir
It should be noted that this error can only be reproduced on rdna2 cards. Using an RDNA3 device with the rdna2-unknown-windows triple fails to reproduce the issue.
In fact, the same .vmfb as above running on an A100 with Vulkan also succeeds without error. Hence I believe the issue is specific to RDNA2 devices.
Steps to reproduce your issue
Download the .mlir / .vmfb for dispatch 27 from above and run on a system with an RDNA2 device using the given CLI inputs.
What component(s) does this issue relate to?
Compiler, Runtime
Version information
5ad8800039323162aa1790bca142a32acf59a242
Additional context
No response

This is tracked on the SHARK repo here: https://github.com/nod-ai/SHARK/issues/1230

It seems to work with AMD_ENABLE_LLPC = 1. Closing this issue.

Can you please also attach the SPIR-V code? Anush said there is a reduced SPIR-V kernel that make the repro eaiser.

This issue can reproduced navi21 with old driver: 2023-02-16_012720-xgl_#7a3f6c1_pal_#d1cacef_sc_#b15b855.
And issue is gone with new xgl driver: 2023-05-28_143120-xgl_#99a1541_pal_#a955b3b_sc_#ac2c4ad.
Please try the newest driver.


Adds RFC on build reproducibility

Summary

Readable
Checklist

[x] I have viewed, signed, and submitted the Contributor License Agreement.
[ ] I have linked issue(s) that this PR should close using keywords or the Github UI (See docs)
[ ] I have added an integration test, if necessary.
[x] I have reviewed the styleguide for guidance on my code quality.
[x] I'm happy with the commit history on this PR (I have rebased/squashed as needed).

@ryanmoran I'm in favor of this change. Do you know why the built_at field was added originally? If so, can we add a section to the RFC detailing why it was added and why it's ok to remove it now?

Do you know why the built_at field was added originally?

This field was added in a number of buildpacks to assist in testing layer reuse at the integration level. Outside of that usecase, I don't know of any consumers of this field. Its a field that is exposed in the image metadata, but not one that we've ever really considered or treated as a stable API.


[Tooltip] Persistently set ignoreNonTouch when device is a touch device

While working with the speed dial, I found that tooltips were still appearing small in on mobile devices if they were automatically set to open because the logic for making them large was for some reason encapsulated in the onTouchStart and onTouchEnd methods. Not exactly sure why that is, but this proposes that if there is any case where a device responds to touch, show the larger tooltips, without seemingly unnecessarily toggling the values.

Aside from the solution that needs to be changed. I would challenge the requirement in the first place. After quickly benchmarking what Google Agenda and Google Inbox is doing, I wouldn't change a thing.

@oliviertassinari I do not understand why that timeout logic is needed. It is causing an issue where initially the tooltips are small, but when a speed dial action is tapped, it shows large for a second - looks very glitchy. See attached from the docs website.

As for whether the size should change or not, the material design spec says the size should be different for desktop and mobile:
https://material.io/design/components/tooltips.html#spec

I do not understand why that timeout logic is needed.

@hashwin Then maybe we could add a comment about it. The core idea is that people can have an hardware that supports both touch and mouse events. It's not a change we can make as easily.
One of the alternatives is to provide a property to force a rendering "mode".
But as I said, I have been benchmarking what Google is doing with its products. I would challenge the usage of the Tooltip of this use case. What we win at using the Tooltip? Using the Grow + Paper + Typography components might be simpler. I don't know, I'm just saying it's a path worth exploring.
https://material.io/design/components/buttons-floating-action-button.html#types-of-transitions

then the existing logic in the tooltips MUST be changed instead to use the WIDTH rather than whether touch is supported or not.

Large screens can support touch interactions.

it accurately sets the position in relation to the actual speed dial item

It can be done with CSS too, we can save a lot of CPU cycles by not doing it with JavaScript.

Large screens can support touch interactions.
That is precisely my point. The size of the tooltip should not depend on whether the device is touch enabled or not, rather it should depend on the width of the screen.

@oliviertassinari Fair enough I guess. I will make a PR with your suggested approach.


Sort parkinglot list according to user's location

It'd be cool to be able to sort the list of parking lots showing the nearest ones at the top. Not quite sure how to work with the sections? Or maybe sort the sections and then the stops in them?
Best way to go would probably be to leave out the sections entirely in this case. Go with just a long list of parkinglots sorted accordingly.

Having a settings button in the top left corner would be pretty nice. This option could be listed there.
And then throw the button for the about page right into that as well.


JasonRpc Api does not work (using Release 18.06.4)

Hey!
I've still got a problem consuming the JsonRpc api, also after installing a younger release than Chaos Calmer (see closed ticket 3050: https://github.com/openwrt/luci/issues/3050 )
I've now got a VM using OpenWrt 18.06.4 (r7808-ef686b7292).
I use this virtualisation to play around with it before using the "real world application".
I've installed LuCi -> works fine.
I've installed luci-mod-rpc (git-19.241.48373-38f5f4b-1)  and restarted the uhttpd.
I'm able to get an auth token, sending with Postman a Post statement to
http://x.x.x.x/cgi-bin/luci/rpc/auth
with json-data
{
"id": 1,
"method": "login",
"params": [
"root",
"xxxxxxx"
]
}
Fine, but using that token, I'm not able to get any response. No matter what library I try to reach.
Sadly, the library links in this Wiki are broken, so I have to Trial & Error :(
First question: do I send a Post or a Get statement? I've tried both, but any response...
So I tried:
http://x.x.x.x/cgi-bin/luci/rpc/fs?auth=f78bf762b59533713db182b45d7fb7e2
with json-data
{
"jsonrpc": "2.0",
"method": "dir",
"params": ["/"],
"id": 1
}
The response is
Could not get any response
There was an error connecting to .
So how to get the jsonrpc work?
Is there anywhere a running documentation for the exported libraries? All links in https://github.com/openwrt/luci/wiki/JsonRpcHowTo are broken :(
Thanks in advance!

@melaLetics Has your problem been solved?
Now i have similar issues with rpc api. It seems that nothing but auth method works...
http://<IP_ADDRESS>/cgi-bin/luci/rpc/fs
{ "jsonrpc": "2.0", "id": 2, "method": "dir", "params": ["/"] }

response:
{
    "id": null,
    "error": {
        "message": "Parse error.",
        "code": -32700
    },
    "jsonrpc": "2.0"
}

@melaLetics my just Has been solved.
Take a look https://github.com/openwrt/luci/issues/3470

Yeah, thanks. That worked for me!


primitives/sr25519: Enable generating secret keys from ed25519 keys

I'm still against using private keys from scheme A in schemes A and B.

This would not be the case, the key would only be used in one scheme, not both. But the derivation process is something that is needed for compatibility with a particular hardware device. Also see related implementation in Rust's schnorrkel for reference.

Some test vectors obtained directly from the Ledger's derivation for the test mnemonic:
PIN: 5555
Mnemonic: equip will roof matter pink blind book anxiety banner elbow sun young

Algorithm
Number
Public Key
Address

sr25519-adr8
0
l+cuboPsOeuY1+kYlROrpmKgiiELmXSw9xl0WEg8cWE=
oasis1qpl4axynedmdrrgrg7dpw3yxc4a8crevr5dkuksl

sr25519-adr8
1
VOmOqK/PEyHt3SyR7nH3+SN8OL2MMkIFe+XHzj9Gq70=
oasis1qrlps0l9g607535rgshz8urjvpwzuujmgyjm766h

sr25519-adr8
2
fRChHhpO9a3qM+ufMzLG0iHBLUYSmd4y0Q5s//zXdtg=
oasis1qr6j5zx4eyvsfdpe6wslnt7nj9cdhst38czrrrmr

sr25519-adr8
3
APOgBQkpM+jClW1+zmLL05cYZ441vypzcMNE6edVvBg=
oasis1qpc3uhcleepuztfwu709mhs5rfz5wvt60cg9w27e


poll: the code judgment condition is always true

In the source code of poll.c,

The num_events is a signed variable and 0U is the unsigned variable,  this condition compares these two variables,  the num_events will change the type to an unsigned variable, and I'm designing a negative test case, and I try to set the num_events as -1, an error "num_events too large" will return. It seems the condition is always true. So I think num_events >= 0U is always true, whatever the value of num_events.
And this changs is showed in commit bbbc38ba8f023bb2f1451e55568a7f3f4a273586.
The test case runs in zephyrproject/zephyr, the branch is mainline.
The log:
when the num_events is -1 and the second part is 0U:
E: syscall z_vrfy_k_poll failed check: num_events too large
(the branch is not tested, the error is reported by other conditions)
when the num_events is -1 and the second part is 0:
E: syscall z_vrfy_k_poll failed check: num_events >= 0
(the branch is tested)

Would your test work after removing U?

Would your test work after removing U?

Hi @dcpleung, yes, I have tested this, and there is a existing test case named test_k_poll_user_num_err to test this point, and I find that the output is not expected, so I dig into it. As the bug description said:
The output of this test case is:
START - test_k_poll_user_num_err
E: syscall z_vrfy_k_poll failed check: num_events too large
PASS - test_k_poll_user_num_err in 0.2 seconds
and after removing U, the log:
START - test_k_poll_user_num_err
E: syscall z_vrfy_k_poll failed check: num_events >= 0
PASS - test_k_poll_user_num_err in 0.2 seconds
So need I to remove U in poll.c?


[game crash] game crashes when closing audio offset calibration

the game will crash when you exit out if the offset calibration menu.

Duplicate of #291 and #396


Stack overflow in mrb_vm_exec

Description
We found a stack-overflow case in the latest commit (as of Sep 25, 2023). Unlike the closed (unreproducible) issue #6050, this PoC does not seem to involve any JSON.
Proof of Concept
$ mruby <poc_file>

poc.zip
Stack Trace
Due to the length of the stack trace, we've attached the stack trace along with the attached PoC.
Environment
Ubuntu 20.04
Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz
Memory: 64 GB
Affected Version
v3.2.0 (commit 15bb6a9f9b, latest as of 2023-09-25)
v3.2.0 (commit 595649626)

The following is the simplified version of poc.rb and you have intentionally created a circular reference in line 3.
v2 = {}
v3 = v2.lazy()
v3.args=(v3)
v5 = v3.filter_map()

We are going to address this issue in the future, but the priority is low.


Fix install dev

We need to add pip install --editable . just before pip install --editable .[all] because mlserver is listed in requirement of other sub-modules and this cause to install mlserver non-editable and from pypi.

Good catch! This one has been nagging me for a while.
I think that the main "culprits" here are the individual inference runtime packages, which list mlserver there. Therefore, would it be enough to just have a single pip install -e .[all] after the runtimes have been installed (i.e. just move it towards the end)?

I agree, that's better solution.

Hey @Salehbigdeli ,
Just noticed you undid the latest changes. Did you find any issue with installing the editable version at the end?

The main reason was the pipeline and I couldn't reproduce the behavior on my local machine. Other than that:

they aren't that different.
placing pip install --editable . in the beginning installs mlserver once.

Hey @Salehbigdeli ,
My main concern is that the approach suggested in this PR installs MLServer twice, one after the other:
pip install --editable .
pip install --editable .[all]

So I guess the main questions from my side would be:

How does this fix the issue if the individual runtimes still get installed afterwards? And why is it required to install it twice?
Related to 1. - to avoid someone else wondering the same in the future, could you add a comment to explain why it's done that way?
What issues did you find when you changed it to a single pip install -e .[all] and moved it at the end?

Hey @Salehbigdeli ,
My main concern is that the approach suggested in this PR installs MLServer twice, one after the other:
pip install --editable .
pip install --editable .[all]

So I guess the main questions from my side would be:
1. How does this fix the issue if the individual runtimes still get installed afterwards? And why is it required to install it twice?

2. Related to `1.` - to avoid someone else wondering the same in the future, could you add a comment to explain why it's done that way?

3. What issues did you find when you changed it to a single `pip install -e .[all]` and moved it at the end?

I tried installing everything from scratch several time to see what exactly happens. here is what I found:
First, The [all] in pip install --editable .[all] is obsolete according to log WARNING: mlserver 1.2.0.dev6 does not provide the extra 'all'
Second, The root of the problem is one of requirements in huggingface when we want to install huggingface runtime there is other requirement that is compatible with mlserever==1.1.0 and pip decides to uninstall current version of mlserver==1.2.0.dev6 and install older. this conclusion come from this log: INFO: pip is looking at multiple versions of mlserver to determine which version is compatible with other requirements. This could take a while. Collecting mlserver
Third, I couldn't reproduce pipeline error on my local machine.
My conclusion:
You are right about placing pip install --editable . after installing runtimes.
I'll move pip install --editable . after installing runtimes, and if pipeline fail I can't do anything about it.

Hey @Salehbigdeli ,
Thanks for those insights and thanks for taking the time to explore this one!
Good spot about the all extras not being there anymore. It seems that was removed a couple weeks ago (which went under the radar). RE the pipeline, I must have missed the previous pipeline error - I thought all tests were green? Now that a new run has been kicked off, let's see what the error is (it may have been an unrelated one).


Suggestion for Attributes: Shannon & Brute Force Thresholds Revisited

Your treatment of entropy per letter in the attributes (and as explained in your blogs) is interesting, but I think that to ensure a passphrase will not be too short, one must consider the number of words in the passphrase (as Reinhold did when he came up with a 19-letter minimum for a 6-word passphrase).  Basically, the character-based entropy must be at least as large as the word-based entropy.  This leads to a required lower bound for the minimum average word length in a given passphrase, from which one can determine the minimum number of letters required in a passphrase that has a specified number of words (by multiplying the minimum average word length by the number of words).  So my suggestion is that the attributes table should report this lower bound for the required average word length.
If the actual entropy per letter is E (e.g., 2.62 bits for Shannon, 4.70 bits for brute-force), and the number of entries in the word list is W (e.g., 7776 for a diceware list using 5 dice), then it can be shown that the required lower bound for the average word length in a generated passphrase is given by the expression (log2W)/E.
The above expression is for the case of no separators, but if separators are included, the resulting increase in the character-based entropy relative to the word-based entropy will be minimal.  The calculation becomes much more complex, so I think it would make most sense to just report the conservative value obtained for the assumption of no separators.
One could compare the average word length for the entire word list to the computed minimum required average word length, but to guarantee that none of the generated passphrases have an insufficient number of characters, the shortest word length in the word list would have to be greater than or equal to the lower bound for the average word length.
It may be possible to compute or estimate the overall entropy reduction caused by rejecting passphrases that are too short, but I would have to think more about how this could be done.

Ah, I did this work a few years, so this is a bit of test for me. Very cool to have someone interested in this stuff!

Basically, the character-based entropy must be at least as large as the word-based entropy.

Agree.
I think you're suggesting the attributes table include a value called "minimum average word length". Let's assume 4.7 bits of entropy for a moment (putting Shannon aside for now). I think the meaning of this value would be saying to the user "Hey, whenever you generate passphrase using this list, make sure the average length of the words in your passphrase is above X". Do I have that right?
If so, I object to that metric for a few reasons. First of all, for the EFF long list, this new "minimum average word length" would be 12.925/4.7 or 2.75. Since the shortest word is 3 characters, there's no possible way to generate a passphrase below the minimum length.
But secondly, I have intended to make Tidy a tool to make word lists that are good and safe with wide room for user error. By this I mean, I do not want to pass caveats to the user. "Ensure your passphrase's average word length is above X" would be one of these caveats. Basically, I want to encourage list-makers to make lists that are as "fool-proof" as possible. (Making it easy to remove prefix or suffix words is another example of this: that way, the passphrase generator need not require separators between words).
My method for preventing these caveats is to take conservative approaches.
Let's look at the metrics that Tidy currently displays. Here they are for the EFF long list:
List length               : 7776 words
Mean word length          : 6.99 characters
Length of shortest word   : 3 characters (aim)
Length of longest word    : 9 characters (zoologist)
Free of prefix words      : true
Free of suffix words      : false
Entropy per word          : 12.925 bits
Efficiency per character  : 1.849 bits
Assumed entropy per char  : 4.308 bits
Above brute force line    : true
Above Shannon line        : false

The key lines for our discussion are the last three. I'll use W as word list length (7,7776 in the case of the EFF long list).
"Assumed entropy per char(acter)" is log2(W) divided by length of the shortest word on the list (here's the code). For the EFF list, it's log2(7776)/3 which is 4.308.
I use the shortest word on the list to be conservative. Users could get a passphrase completely made up of words that are of the shortest length.
Once we've calculated "assumed entropy per character", we compare it to two constants to make the metric more human-understandable to users. If "assumed entropy per character" <= 4.7, Tidy reports that the list is "above" the "brute force line". For the EFF list, 4.308 is indeed less than 4.7, so we're safely "above" the brute force line (reported in the attribute table as "Above...": "true"). If it's less than 2.62, we're under the "Shannon line". Since 4.308 is greater the 2.62, the EFF list does not clear this line.
In general, I want Tidy to encourage the creation of lists that are as safe as possible, with as few caveats as possible. I hope that makes sense in connection to your comment.

Yep, I gotcha. The "lines" are admittedly strict/unforgiving in the current set-up... maybe too strict to be useful? Haha I also haven't gotten much feedback from tidy users who are not me, so this is a good discussion.
I guess we could do like "Chance that a 6-word passphrase is below the 'Shannon line'", expressed as a percentage? For the EFF list, we'd count the number of words with fewer than 5 characters and go from there?

I have a feeling that the calculation will not be trivial. My initial thought is that it may involve a 6-dimensional joint probability distribution...

Ah, maybe you're right. If you want to work out some kind of formula, I might be able to implement it for Tidy in Rust. Maybe we could create a sort of general 0 to 100 "score"?

Not exactly on-topic, but I did add two formulas and two tables to the bottom of the readme that might be of interest here (and would welcome a double-check of my math).
Maximum word list lengths to clear the Brute Force Line
Formula:
Where S is the length of the shortest word on the list, 26 is the number of letters in the English alphabet, and M is max list length: 2S * log2(26) = M
(or in Python: max_word_list_length = 2**(S*4.7))

shortest word length
max list length

2
675

3
17559

4
456419

5
11863283

Maximum word list lengths to clear the Shannon Line
Formula:
Where S is the length of the shortest word on the list and M is max list length: 2S * 2.62 = M
(or in Python: max_word_list_length = 2**(S*2.62))

shortest word length
max list length

2
37

3
232

4
1428

5
8779

6
53974

Before creating this table, I hadn't realized just how "strict" the Shannon line. Makes me wonder if it's so strict as to not be helpful.

For the first table, your max list length values are only accurate to 2-3 significant digits, as a result of rounding errors in taking the logarithm.  You can make the max list length values more accurate by taking advantage of the fact that S·log2(26) = log2(26S ) and that for any base B, by definition of the logarithm, we get BlogBX = X.  As a result, we get M = 26S, which can be exactly evaluated without rounding errors; thus, your table entries should be 676, 17576, 456976, and 11881376.

AMAZING! I've updated the brute force table in the readme.
Fixing the actual brute forcetest in the code itself is a bitter trickier, thanks to some rounding tomfoolery in Rust.
2.0_f64.powf(26_f64.log2()) // evaluates to 25.999999999999993, rather than 26

(Code playground, for those interested.)
So, unless you object, I'm just going to hard-code the decimal in the test: assumed_entropy_per_character <= 4.700439718141093

However, since the claimed average word length is only given to 2 significant digits, the ratio can also be known only to 2 significant digits — i.e., 2.6 bits/letter...  Thus, the corresponding values of the maximum list length should not be given to a precision greater than 2 or 3 digits.

Fascinating work! Going to change the Shannon test to 2.6 for now. Maybe we should just... do a bit of our own analysis with some more modern word corpus?

I found some oddities in the math of that 1951 Shannon paper... Admittedly, the linked PDF contains some scanning/OCR artefacts, so some of the mathematical expressions may not be correctly represented. I should probably check out the original version of the 1951 article, and also read the 1948 Shannon paper that is cited as Reference 1.

Again, super impressed you went this far here. Hope it's just a scanning issue? Let me know if I can help. The PDF I linked to was just a first search result. I should really get into the papers myself sometime.

Can you explain why the limited precision of Rust's floating point calculation causes a problem in the code? Not sure I understand why there is a need to calculate 2log26 in the code, or why an error on the order of 10-15 will cause issues.

I was confused as well until just now -- have resolved the rounding issue.
Think the culprit was this function:
pub fn calc_entropy_per_word(list_size: usize) -> f64 {
    (list_size as f64).ln() / (2_f64.ln() as f64)
}

If I change it to
(list_size as f64).log2()

The rounding issue I found resolves itself.
Here's a Rust code playground the shows the original issue:
Here's a Rust playground that lays it out. https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=e8234d2494140adf09272282a7c1ee20

For the brute force case, this should require no floating-point arithmetic, and therefore would be expected to yield a more precise result.

Very smart. THink it's working.

G = 22.6 = 6.1 (randomly drawn letters from written English)

Just want to confirm you like 6.1 here, rather than having more digits. Guessing we're set on limiting Shannon's calculations to two significant digits, but just checking.
NOTE: Using your new method of calculating the Shannon line disrupts the table in the readme, as it's currently calculated. Now, a list with shortest_word-length of 3 can be 227 words and still be above the Shannon line. Does that sound right?

you'll have to decide whether you consider 2.6 or log26.1 (or even something else, like 2.63, or 11.82/4.5) to be the canonical representation of Shannon's limit.

Glad we finally got to "We're stuck cuz we can't tag Claude Shannon"!
I guess I'll leave the Shannon calculations using 6.1 for now. Version 0.2.66 has this math in it -- we can make that a "release" if we don't find more things to fix in next few hours/days. Thanks again for your insisght and encouragement.
Related: I just ordered two information theory textbooks: Elements of Information Theory 2nd Edition by Cover and Thomas; and Information and Coding Theory by Jones and Jones. The former gets better reviews it seems like. Maybe they'll have some clues for us, both on these entropy calculations and on uniquely decodable stuff.

to me, this suggests that Shannon's notion of "entropy per letter" is not suited to characterizing the vulnerability of word lists to letter-based brute-force attacks.

I think you're asking and answering the right question here.
TBH when I picked that 2.62 number out of that Shannon paper (without reading far beyond that section), I wasn't attuned to all these ideas and Shannon's limitations. My goal was to provide a response to a (hypothetical) question of: "OK, this list is above the brute-force line that assumes all letters are equally likely in random order, but what about the different, slightly more enlightened attack of brute-force-guessing words rather than letters?" (And I think I figured it'd be cool to cite a 70-year-old paper from the father of information theory.)
I've now got enough of a handle on this whole entropy thing that it makes sense to me that the goals and methods Shannon used are distinct from what I was/am looking for when setting up this second, "tougher" line calculation.
Thus, I'm open to removing the Shannon line concept from Tidy entirely, at least for the time being.

I think I follow you, but I want to learn/know more about G, with an eye toward hopefully (a) improving its name or (b) mathematically converting into something more intuitive without losing any of its descriptive power or accuracy.
Two questions for now:

Is G more accurately described as assumed guesses per character?
Is G related to a value I label as "assumed entropy per character"? If so how?

Notes on assumed entropy per character
Tidy currently calculates a variable named assumed_entropy_per_character to measure a list against the brute force line.
assumed_entropy_per_character = log2(list_length) / shortest_word_length
As an example, for the EFF long list (7,776 length, shortest word is 3 characters), "assumed entropy per character" comes out to 4.308 bits. This is the number Tidy formerly compare against 4.7001 to see if the list is above or below the brute force line.
Thanks to your help, Tidy no longer uses this method, due to rounding inconsistencies, but in principle, "assumed entropy per character" still is tightly associated with the brute force line, hence my wondering if it's useful in our discussion of G and creation of a replacement for the Shannon Line.

A separate, wild thought
In the assumptions behind our construction of the brute force line, are we wrong to assume a brute-force attack that must go through all 26 characters? More generally: Won't, on average, an attacker guess correctly about halfway through the given space? As a contrived example, if I tell you I'm thinking of a randomly chosen (lowercase) letter from a to z, won't you, on average, guess it on or around your 13th guess?

What I'm less clear on right now, is how to interpret all of this when there is a non-uniform frequency distribution in the corpus.

I've seen some references to entropy definitions other than Shannon's, e.g. Tsallis Entropy and Renyi Entropy. While I have no idea if either of those two concepts are helpful to our problem(s), I'm encouraged that there are other conceptions of entropy out there. Wonder if something fits our needs just right.


🚀 Feature: Default share expiration configurable in administration

🔖 Feature description
Default share expiration configurable in administration
🎤 Pitch
I'm likely to use pingvin for a group of friend's in a private discord, but we have no things to be automatically deleted. Currently the default behavior is for a share to expire in 1 day, but it would be nice if this default could be configured from never to whenever.

Hello,
This feature would be extremely useful, even for professional use.
Thank you very much for your work.

Indeed the default settings. I am absolutely fine with our guests to lower this to a single day if they so desire. However now most of the clients dont think about it and thus forget to set it longer than a single day.
So my wish would be that the server default expiry can be changed. Whatever people set afterwards is up to them.


[SPARK-13648] Add Hive Cli to classes for isolated classloader

What changes were proposed in this pull request?
Adding the hive-cli classes to the classloader
How was this patch tested?
The hive Versionssuite tests were run
This is my original work and I license the work to the project under the project's open source license.

Can one of the admins verify this patch?

Copy of recent developemrnt mailing list comment.
I have been testing 1.6.1RC1 using the IBM Java SDK.
I notice a problem ( with the org.apache.spark.sql.hive.client.VersionsSuite tests ) after a recent Spark 1.6.1 change.
Pull request - https://github.com/apache/spark/commit/f7898f9e2df131fa78200f6034508e74a78c2a44
The change introduced a dependency on org.apache.hadoop.hive.cli.CliSessionState in org.apache.spark.sql.hive.client.ClientWrapper.
In particular the following test was added
if (originalState.isInstanceOf[CliSessionState]) {
The problem is that VersionsSuite test uses an isolated classloader in order to test various versions of Hive. However the classpath of the isolated classloader does not contain CliSessionState.
The behaviour of isInstanceOf[CliSessionState]) is JVM vendor specific. In particular whether this code causes the CliSessionState class to be loaded. ( It does not for openjdk, but does for IBM JDK ). Hence this call can throw a classnotfound exception.

cc @marmbrus

ok to test

LGTM pending tests

Test build #52418 has started for PR 11495 at commit 79911ff.

Test build #52418 has finished for PR 11495 at commit 79911ff.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52418/
Test PASSed.

Thanks, merging to master and 1.6.

Why we need hive-cli here? This is not version specific.

Because ClientWrapper calls isInstanceOf[CliSessionState] and so needs to load the class CliSessionstate. In the testcase ClientWraper has been loaded by the isolated class loader.


@JamesRamm My cols value is ,cols={{ lg: 32, md: 32, sm: 10, xs: 6, xxs: 4 }} , I want only 1/3, 2/3, 3/3 widgets on a column,

@JamesRamm My cols value is ,cols={{ lg: 32, md: 32, sm: 10, xs: 6, xxs: 4 }} , I want only 1/3, 2/3, 3/3 widgets on a column,

suppose total cols width is 33, If I want,
1/3 of widgets then total widget size will be 33
2/3 then 1st widget can be 11 and 2nd can be 22 or 1st can 22 and 2nd can 11
3/3 then all 3 widgets will be of size 11 each.
As in diagram. I want widgets with config 2/3, and 2nd widget is of size 22,and 1st if of 8 and there is some remaining space with size 3. So, I want the first widget get snapped with the size 11 automatically even if I resize it to 7 or 8. and if try to increase the size of 2nd widget (which is 22 now) then 2nd widget should occupy whole column space with size 33, and 1st widget should be moved to next column.

Originally posted by @PriyankaBhise in https://github.com/react-grid-layout/react-grid-layout/issues/1399#issuecomment-801616176

@PriyankaBhise
This is an application-specific use-case, so you should manage it using the onResizeStop callback which you can pass to the RGL Grid component

@PriyankaBhise
Take a look at this demo which implements more complex rules for widget sizes:
https://react-grid-layout.github.io/react-grid-layout/examples/10-dynamic-min-max-wh.html


Refactoring location of java sources

This refactoring is the first of several changes intended to unify and simplify the layout of code in this repository and allow for upcoming refactoring to be achieved with minimal disruption.
The code layout is now structured in line with Maven/Gradle conventions and the Java artifact is now primarily defined by its role (Android) and not it's language (Java).
Note that this removes the ant file as a way to run unit tests. Going forward there will be a way to run non-UI tests via gradle (but temporarily that functionality is no longer present).

Also adds the top level LICENSE file, which seemed to have gone missing somewhere.


video formatting

Wondering if you have also encountered the
assert len(video) == self.num_frames
error in torchvision/datasets/vid_utils.py .
I tried converting from avi to mp4 but it did not help, Thanks again.

Unfortunately, I haven't seen this issue before. Maybe the video is too short and there aren't enough frames?
Or, to debug, you could try reading in the video yourself as a numpy array, and then save it using something like scikit-video or cv2, since I know mp4 files saved from those should definitely have no issues be read using VideoClips

thanks, tried that, still nothing. Ended up changing
/torchvision/datasets/video_utils.py
'''python
assert len(video) == self.num_frames, "{} x {}".format(
video.shape, self.num_frames
)
'''
to:
'''python
if len(video) != self.num_frames:
video = video[:self.num_frames]
'''

thanks, tried that, still nothing. Ended up changing
/torchvision/datasets/video_utils.py
assert len(video) == self.num_frames, "{} x {}".format(
video.shape, self.num_frames
)

to:
if len(video) != self.num_frames:
video = video[:self.num_frames]


[stable/wordpress] Readiness and liveness probe failes with multisite

Is this a request for help?: No
Is this a BUG REPORT or FEATURE REQUEST?
bug report
Version of Helm and Kubernetes:
kubernetes 1.12.7
helm: 2.13.1
Which chart:
Wordpress
What happened
With a lot of struggling i managed to get a multisite worpdress up and running, the problem now is with the readiness and liveness probes. When i enable multisite urls get hardcoded into the database (that's what i understood from the internet), thats why it's more difficult to get a multisite wordpress running locally, because you have to change all those urls to localhost, but then it's not going to work online anymore.
So when the readiness or liveness probe kicks in, it sends a request to /wp-login.php which wordpress is redirecting to http://subdomain.domain.com which is down until the pod gets ready because the ingress is waiting for the pod to start and sends 503. It's an endless loop.
Here is what i enabled for multisite
define( 'WP_ALLOW_MULTISITE', true );
define('MULTISITE', true);
define('SUBDOMAIN_INSTALL', false);
define('DOMAIN_CURRENT_SITE', 'subdomain.domain.com'); this is a placeholder
define('PATH_CURRENT_SITE', '/');
define('SITE_ID_CURRENT_SITE', 1);
define('BLOG_ID_CURRENT_SITE', 1);
define('WP_SITEURL', 'http://subdomain.domain.com/'); this is a placeholder
define('WP_HOME', 'http://'subdomain.domain.com/'); this is a placeholder

And my htaccess
<Directory "/opt/bitnami/wordpress/">
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]

# add a trailing slash to /wp-admin
RewriteRule ^([_0-9a-zA-Z-]+/)?wp-admin$ $1wp-admin/ [R=301,L]

RewriteCond %{REQUEST_FILENAME} -f [OR]
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule ^ - [L]
RewriteRule ^([_0-9a-zA-Z-]+/)?(wp-(content|admin|includes).*) $2 [L]
RewriteRule ^([_0-9a-zA-Z-]+/)?(.*\.php)$ $2 [L]
RewriteRule . index.php [L]
</Directory>

# Askimet plugin
<Directory "/opt/bitnami/wordpress/wp-content/plugins/akismet">
  # Only allow direct access to specific Web-available files.

# Apache 2.2
<IfModule !mod_authz_core.c>
        Order Deny,Allow
        Deny from all
</IfModule>

# Apache 2.4
<IfModule mod_authz_core.c>
        Require all denied
</IfModule>

# Akismet CSS and JS
<FilesMatch "^(form\.js|akismet\.js|akismet\.css)$">
        <IfModule !mod_authz_core.c>
                Allow from all
        </IfModule>

        <IfModule mod_authz_core.c>
                Require all granted
        </IfModule>
</FilesMatch>

# Akismet images
<FilesMatch "^logo-full-2x\.png$">
        <IfModule !mod_authz_core.c>
                Allow from all
        </IfModule>

        <IfModule mod_authz_core.c>
                Require all granted
        </IfModule>
</FilesMatch>
</Directory>

I hacked through this by adding a random header to the readiness probe and then i created a rule in the htaccess to return 200ok if the header exists. This is hacky solution to get the pod up and running, but renders the probes useless.
I might have a bad configuration, i'm not sure, but i couldn't find any useful information on the internet, and i'm new to kubernetes.
What you expected to happen:
The readiness and liveness probes to succeed with multisite.
Anything else we need to know:
I opened this issue #13161 and then closed it because i thought i fixed the problem, but it's kind of the same.

If the issue is not with my configuration then this will be fixed with kubernetes 1.14
https://github.com/kubernetes/kubernetes/issues/73172
https://github.com/kubernetes/kubernetes/pull/75416

Hi @andreisergiu98
I think this error was solved on this PR: https://github.com/helm/charts/pull/9390
Could you please take a look? You can modify the httpHeaders to avoid the redirection.

I enabled the following headers, but it's still following the redirects and ends with 503
livenessProbeHeaders:
- name: X-Forwarded-Proto
  value: https
readinessProbeHeaders:
- name: X-Forwarded-Proto
  value: https

Hi @andreisergiu98
That httpHeaders fix the issue when you're being redirected from http to https. However, it doesn't fix the issue in your case (you're being redirected to a specific domain by WordPress).
I guess you could substitute the httpGet probe with a tcpSocket (checking the port is active) probe or an exec probe (using a curl call to check wether Apache is responding as expected).
@javsalgar @tompizmor do you know what other alternative to use?

Yes, I guess the problem is because of the redirection to a different domain. Maybe what it makes more sense is to have the tcpSocket test.

Did you have a chance to try the tcpSocket test @andreisergiu98 ?

Sorry for the late response, i was out of my country. It works with tcpSocket.  Thanks!
livenessProbe:
  initialDelaySeconds: 120
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 6
  successThreshold: 1
  httpGet: null
  tcpSocket:
    port: 80
readinessProbe:
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 6
  successThreshold: 1
  httpGet: null
  tcpSocket:
    port: 80

@andreisergiu98 can you share your process to get multisite working with Helm please? The steps I've taken are :

Enabled liveness and readiness probes as above
Enabled allowOverrideNone=true and an empty customHTAccessCM=custom-htaccess configmap
Add define( 'WP_ALLOW_MULTISITE', true ); to wp-config.php
enabled the network and updated wp-config.php with details provided by network setup. Database is updated.
Deleted and re-created configmap with new values. Deleted pods so they get new configmap.

The result is I get really broken redirects...login link points to http://wp-login.php/
I'm using nginx ingress controller like :
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: wp-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/rewrite-target: /
    # required to upload plugin
    nginx.ingress.kubernetes.io/proxy-body-size: 50m
    # limits access to app from specified IP
    nginx.ingress.kubernetes.io/whitelist-source-range: "<IP_ADDRESS>/32"
spec:
  tls:
  - hosts:
    - www.example.com
    secretName: wordpress.tls-local
  rules:
  - host: www.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: poc-wordpress
          servicePort: 80

Any ideas?

@hilt86 Yeah i stumbled across the same issue. If i remember correctly i used the relocate flag. https://wordpress.org/support/article/changing-the-site-url/

Note: When the RELOCATE flag is set to true, the Site URL will be automatically updated to whatever path you are using to access the login screen. This will get the admin section up and running on the new URL, but it will not correct any other part of the setup. You’ll still need to alter those manually.

You can also try to manually set the base urls:
define( 'WP_HOME', 'http://example.com' ); define( 'WP_SITEURL', 'http://example.com' );

I noticed WORDPRESS_CONFIG_EXTRA environment variable is not being appended to the wp-config.php file even when it is set. Is that by design, if not what I am missing?
Is there an alternate way to enable multisite using this chart? I do not seem find it in the values.yaml file.

Hi @bsakweson
At this moment we don't provide a way to modify that setting in the wp-config.php using environment variables.
We have an opened issue to support any kind of custom configuration to the wp-config.php as you can find in the link below:

https://github.com/bitnami/bitnami-docker-wordpress/issues/167

You can build your own WP image (based on bitnami/wordpress image ) adding this feature as it's explained in this link.

I am not provisioning a standalone instance. I am rather automating the whole installation process and that does not cut it for me. I need that flexibility. I noticed the ticket you shared has been on hold since April last year, is there any reason for the long hold? In any case, I went back to you use the original WordPress image and implement my own chart. This chart is not usable for me as I must automate multisite deployment.