GDScript performance improvement via Just-in-time (JIT) compilation

What about adopting just-in-time (JIT) compilation for GDscripts?
JIT compilation has no disadvantages over interpreted code (aside from slower startup, in some cases) and many advantages, mainly in terms of performance.
For JIT compilation, there are at least two popular ways to implement it in a cross-platform way without writing any code from scratch:

LLVM Intermediate Representation
DynASM, used in LuaJIT

There is also the option of using PyPy, the JVM or the CLR, but those are quite heavier, using GNU Lightning or libjit (not sure if the license is compatible with Godot's) and of course, writing a code generator from scratch (which could take years).
So, the choice is pretty much between LLVM IR and DynASM. Both have very good performance, but DynASM is written in Lua and LLVM has a fairly large footprint (~20MB), but also offers other features.
Of course, compiling GDScript directly to C++ could also be a valid alternative - the only game engine which does it that I know of (if I understood it correctly) is Enigma, which compiles its own scripting language (EDL) to C++.
What do you think about this?

JIT compilation has no disadvantages over interpreted code

There is one: portability…

@Calinou Portability isn't a big concern, since we could just run the GDScript as we do now on unsupported platforms.

static typing can be optionally jitted too

@Calinou portability isn't a big concern if those libraries are used - they support plenty of architectures (LLVM probably supports more architectures than Godot) and besides, the interpreter could always be used as a fallback (as @bojidar-bg said).
LLVM IR would actually make static typing easier to adopt (on the back-end, of course GDScript would still need either type inference or explicit type signatures).
If C# support is implemented by linking either against Mono or CoreCLR, those both include JITs so this issue will be solved automatically. I can understand why C# (it's a better language than Java and Unity already uses it) but for those purposes, the JVM would probably bring better performance than either Mono or CoreCLR (especially because of the JVM garbage collector, which is well suited for games). After Java 8, I personally don't think most people would object to it instead of C# (as the advantages brought by C# at this point mostly come down to Generics and LINQ, both not as useful in game development, which is mostly imperative).
Also LLVM makes it possible to use C++ in a JITted fashion. Unreal Engine also uses C++ for scripting, so I don't think that would be too weird (provided segfaults will be more common, but GDScript could still stay for less experienced programmers).

I would support JIT compiling for GDscript as long as it doesn't mean the end of the language (or its respective built-in editor) as it is now,
My proposal.

Add a new static variable type, which in turn can be created by way of doing something like
static var float(5.5) or something (the core difference would be higher performance and better parsing due to the expectation of it being a specific type. This method should allow the use of static typing while keeping the advantages of dynamic typing and eliminate any need for rewriting scripts if you want to use them
After that, add a JIT compiling routine that converts GDscript to C++ behind the scenes (ie. invisible to the user), but visible in the form of performance when the game is run. The idea of using C++ here would mean that the Godot developers would not have to add a bunch of new libraries to the source.

JIT in dynamically typed languages like Lua is done by doing type inference
from the entry point through the call tree, going as deep as possible. I'm
not saying this wouldn't be possible in Godot, as pretty much every entry
point is typed.  The code completion in Godot does something similar, which
is why it's able to guess so much type information.
However there are many cases where the type is not obvious, and while the
code completion works great by guessing the get_node() calls on the tree
being edited, this information might be different or change in run-time.
I'm honestly not sure how efficient type inference might be.
To help on this, it would be easy to allow GDScript to allow the user to
"force" the type of a variable (something not possible in lua). But if you
are going to do this, then you might as well just go full static typing.
I'm not convinced of any approach at the moment.
On Sun, Jun 5, 2016 at 5:37 PM, Ace-Dragon<EMAIL_ADDRESS>wrote:

I would support JIT compiling for GDscript as long as it doesn't mean the
end of the language (or its respective built-in editor) as it is now,
My proposal.

Add a new static variable type, which in turn can be created by
way of doing something like
static var float(5.5) or something (the core difference would be
higher performance and better parsing due to the expectation of it being a
specific type. This method should allow the use of static typing while
keeping the advantages of dynamic typing and eliminate any need for
rewriting scripts if you want to use them
After that, add a JIT compiling routine that converts GDscript to
C++ behind the scenes (ie. invisible to the user), but visible in the form
of performance when the game is run. The idea of using C++ here would mean
that the Godot developers would not have to add a bunch of new libraries to
the source.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/issues/5049#issuecomment-223836149,
or mute the thread
https://github.com/notifications/unsubscribe/AF-Z2znHs58L-KDgtIjHYYjgHVUGXgwpks5qIzOngaJpZM4IuWZe
.

Haxe uses type inference even though it is type safe under the hood. Personally I like the fact that I don't have to define the type of every variable but that you can do so if you need it. That way you have the best of both worlds: you don't need to specify types (as long as the compiler can figure them out) and writing "generic"/"template" methods results in a much cleaner syntax as in statically typed languages like C++ or C#, but effectively the code is type safe and you can specify the requested type whenever you want to use it to make the code more readable.
In Haxe it's done using the syntax
var  [: ] [=]
which looks very weird coming from typed languages but makes perfect sense coming from a language like GDScript.
But I have no idea how this affects complexity and effectiveness of JIT compilation, I merely wanted to point out that this would be kind of a middle ground between dynamic typing as it is now and full static typing as it is in languages like C++.

I used Haxe for 1~2 years long time ago.
I like it to be able to define type optionally for variable and function return as @Warlaan said.

I'd prefer the c++ type declaration style before the typescript/haxe one:
var x = 50
int y = 60
float z =70

Or maybe we could mirror export's style:
type(int) var y = 60
type(float) z =70

@bojidar-bg Personally I prefer c++ style more (a lot less typing).

There are a lot of issues that can arise from putting the type before the variable. Go was written by some of the most experienced C implementers ever (including Ken Thompson) and they went with Postfix notation (like Pascal and Haxe).
I think this discussion belongs in a separate discussion. I'm not proposing a vote because of course most people are used to typed languages like C++, C# and Java but merely to gather research materials and opinions to find the solution that is best suited for a language like GDScript (which is clearly not C-like in any other aspect).

I agree in all four aspects:

The C++-syntax is nicer to read, but
imho the postfix notation makes more sense here
which I would prefer to discuss (not decide by majority)
in another thread. Sorry, I didn't mean to hijack the thread when I mentioned the topic. Let's return to the issue of JIT compilation.

Would implementing JIT via LLVM mean Clang would be preffered over GCC . Is it  possible to compile LLVM JIT using GCC?
Unlike Java , Python, C# etc, GD Script works as part of a bigger C++ native code engine. In fact both GD Script & the rest of the engine are one unit.  Some game components are C++ and some are GD Script. Overall the game performance depends on creating a balance between the two. Writing a Godot game solely in script would make it slower but writing it mostly via nodes with minimal gd script would make it faster.
So my question is . Is this small boost in performance really worth the effort?
I think C# addition may in fact be better for the engine than implementing JIT in GD Script. C# is probably much more optimized and efficient language thus probably as fast as GD Script + JIT but  not as integrated as GD Script.

Would implementing JIT via LLVM mean Clang would be preffered over GCC . Is it possible to compile LLVM JIT using GCC?

Yes, but then LLVM would become a dependency so it would make more sense to use LLVM for everything.
I don't think the boost in performance would be small for larger games. Maybe for arcade games it would not be noticeable, but in games with a lot of loops and function calls it definitely would.
Also, is there a discussion about why C# over Java or even C++? I think C# is a very good language, but the CLR and tooling is definitely inferior to the JVM (especially on other platforms than Windows).

@paper-pauper you should make a forum post about Java, JIT ...  I'd actually prefer Java > C# tbh. I think C# was a not a choice but more of an opportunity that devs decided to take.

so that in the end we decided? will JIT or not?

How about the ability to compile GDScript to static C++ code? Would that even be possible?

@trollworkout : it would probably be simpler and more optimized to write your game logic directly in C++ then, wouldn't it ?

@SuperUserNameMan  Not necessarily. First you would be using C++ not GDScript and second you still need to figure out a way to load a binary object as a dll and plug it into the engine without having to recompile the engine with ur C++ changes.

@trollworkout Thanks to #3936 it could be possible to compile GDScript to C and load it dynamically, eventually.

it is not necessary to translate the script in c ++. @SuperUserNameMan rights

or need JIT or c# and all will be happy

nah the other topic @paper-pauper  linked is actually much more interesting. The idea is to use C binary as a type of scripting language and call code directly via reflection rather than being interpreted making it as fast as C++ native code I think that would solve the ABI issue . In fact with this you don't need ABI anymore.

I was thinking a lot about this, and my main concern about supporting something like C# is that most of the people who would use it would be Unity users coming to Godot. Clearly, we would not include C# with the official templates, but I highly doubt they would build the module themselves...
It could end up being a big effort for something that would be used by just a few.

I am currently experimenting with implementing a GDScript JIT using RPython. The first (very preliminary) results are promising, but I don't know how hard it will be to integrate it into the engine.
Is there still interest in a JITed GDScript or are people more interesting in c# or static GDScript?

Considering that you're actually trying out a possible solution, I say go for it and see if there's enough performance gain in games to be worth it.
What I mean by that is such a thing would only be worth pursuing if there's a rather large performance gain to be had for complex logic (or otherwise such gains might be better obtained from just doing general optimization work on the language itself).

Performance is crucial for me, I just can't choose Godot for the script
Performance. But I think a faster language(Lua, Java, c#) is better than Jit, since Jit doesn't work on iOS. And compiling to c++ loses the possibility to update game logic on iOS

Isn't Lua faster just because of JIT? Or does it have some kind of AOT compilation too?
I still prefer a statically typed GDScript (maybe an optional typing system, like TypeScript is for JavaScript). Also having GDScript -> C++ transpilation would solve a lot of performance issues, but I can see this is far from trivial.

Statically typed GDScript with optimized VM would be also be good, is there any plan for this ?

Optional typing system for GDScript as @vnen suggested would be perfect :)

While researching about this, I've found the great SLJIT. I have been trying to figure out how could I integrate it to allow JIT compilation at least at the GDScript function level.
After some hours of reading/thinking I've come to the conclusion that it would be of little benefit because the biggest bottle neck, now I have studied the code deeplly, is probably all the Variant stuff.
Finishing my task for a test would be overkill so I've abandoned it. Anyway a branch with my initial work is in my repository in case someone wishes to play with SLJIT and have it already integrated into the build system.
So I sum myself to the group that believes that static typing would remove a lot of calling and branching thus improving performance notably.

@RandomShaper
I agree that the JIT is not feasible currently, but the problem is Variant and not dynamic typing.
I've experimented with writing GDScript JIT using RPython, which is the base for PyPy and specifically made to create JITs for dynamic languages.
I disagree that the problem is static typing, it would be possible to have a fast GDScript JIT if we wouldn't have to work with Variant. Variant does replicate some of the functionality of dynamic languages, and unfortunately that is the part where RPython's JIT gets its speed boost. RPython needs to know what types it currently has and can then speed up those parts.
Variant is so deeply embedded into Godot however, that it's really hard to go without it. I would have to basically reimplement all of GDScripts data types and transform to Variants afterwards.
If anybody has an idea how we can get by without using Variants, a fast GDScript would be possible (a proof of concept GDScript JIT of mine that didn't use Variant was promising. A (silly) loop that in GDscript took 20 seconds was done in under a second when JITted)

AIUI Variant make possible to use C++ in Godot as if it were a dynamically typed language. Especially in the return type for some methods. I'm not that acquainted with the source, but I'm not sure if there's a clean way around it.

@brakhane Care to share your RPython experiment? I think everybody here would benefit from taking a look at it.

Variant is so deeply embedded into Godot however, that it's really hard to go without it. I would have to basically reimplement all of GDScripts data types and transform to Variants afterwards.

In theory, what would be the drawback to this (aside from the time needed to implement it)? Casting is not that expensive, as far as I understand it.

@paper-pauper look at PyPy and RPython - Python is strictly dynamically typed, and PyPy can speed-up Python code very well thought... RPython is very good for development of interpreters (with almost-free JIT)

A really interesting article has been published about (currently in development) new Ruby JIT.
The key part here is the Ruby choose to implement their JIT by generating C code and calling an external C compiler to build a .so then load it. Needless to say this has created a lot of interrogations and this article starts to give some answers.
Basically the advantage to use and external compiler (as opposed to integrate a JIT framework) are:

independence to a single compiler (can use GCC or LLVM depending on what's available/best)
JIT frameworks like libGCCJit are young so their API can be much more fragile that C

Of course this needs to be taken with a lot of precautions (the article benchmarks compare a prototype with production ready projects such as JRuby), but it shows that, provided you use clever optimizations such as precompiled headers, using an external compiler can be a viable solution.
From the Godot point of view, this technique could be useful to also provide a static compilation of GDscript.

I'm working on JIT compilation for my master's thesis and am planning on implementing something suited for Godot. So far I've mostly just read a lot of papers on the topic, specifically on how to deal with the dynamically typed nature of GDScript. I've now started working on a prototype, and hopefully have something to show relatively soonish.

Some time ago I started an experiment about JIT-ting GDScript. I did setup some foudation for it.
In case you want to have a look... https://github.com/RandomShaper/godot/tree/exp-gdscript-jit-master

@vnen I'm actually very excited to see how much of a difference in performance static typing can bring, as opposed to the type profiling I'm planning on doing. Since I'll also have to produce some actual text for my master's, I don't think I'll get much more done than a prototype, on which there'll be a lot of room for improvement.
@RandomShaper That setup looks nice, and after looking at the available options I'll probably go with SLJIT as well. Thanks for that!

May 2018 poll results have arrived, and this proposition appeared to be on the roadmap priority.
I personally don't think JIT GDScript would be good idea:

JIT is not allowed on mobile platforms (at least on IOS, not sure about Android), which are the one needing the most this kind of optimisation...
JIT is a big mess. There is multiple approaches (tracing vs method JIT), some of them not even deterministic :'-(
JIT is a BIG BIG mess. Debugging is complex (involving dumps of jited code and memory...), so is profiling, so is maintaining the code :'-(
JIT is a special kind of mess. JIT programmers is a scarce resource so maintaining such a thing would be tricky. This could lead to making GDScript evolve becoming harder since the interpreter and the JIT would be tightly coupled.

The luaJIT project illustrate well those points: the project is simply brilliant (code base really good, performances crazy good), it solve a real usecase, it is used by commercial companies.
And yet it is slowly dying since it main author has step back, unable to keep up with the new lua version (luaJIT stopped at 5.1, arguing the add of integer on top of float in the language made by the 5.2 would be really complex to add), even if the lua community has a medium size and luaJIT it best known project.
Obviously GDScript is order of magnitude simpler than Python or even lua, but the burden of making a JIT stays an important one, especially considering it is not a core feature of the Godot project.
The only acceptable solution here would be to rely on llvm which would take care of all the crazy parts and let us just output llvm-ir code to it. However, as said before, llvm is a big dependency (at this point we could almost ship v8 with Godot and write a GDScript to Javascript transpiler instead :trollface: ), so additional research to understand how much would be added is required here before going anywhere. And again this wouldn't be a viable solution for at least IOS platform (damn you Apple !).
I'm sorry to sound so gloomy here, but I wouldn't post this without a proposition ;-)
I guess what poll voters understand by "JIT GDScript" is "GDScript, but fast as native code". So in the end it doesn't really matter what it the mean to reach this goal right (AOT, JIT, transpiling + compilation etc.) ?
From what I saw of GDScript, a really cool (lack of) feature is it is not a dynamic language. There is no exec keyword, no monkey patching, you cannot modify variable during debugging etc.
This makes the language suitable for static compilation (and in fact it is again one more reason not to go with JIT). On top of that the new static typing system should allow easy optimisation (compare to having to do type inference).
So my idea would be to start by making a simple (no optimisations, just use Variant everywhere) GDScript-to-C transpiler, that would available as godot --gdcc <myfile.gd>. Using the existing GDScript compiler and GDNative api should make this relatively easy.
The users are then left to deal with the rest of the toolchain (for example compiling each C file as a .so and replace the original .gd file by a .gdns file pointing on the .so).
In a second time, we could start wondering about optimizations and, more importantly, a way to ship a compiler into the Godot.
Considering this second point, tinycc seems a really interesting solution given it low weight, it number of platforms supported (x86, amd64, arm, arm64 for linux, win32 and osx), it libtcc.h api that allow to integrate it into a program and it cross compilation mode. On top of that version 0.9.27 has been released last December (so the project seems stile alive ^^).
Finally the same way we currently turn .gd files into .gdc or .gde at export time, we could create a new type (.gdso ?) that would be nothing but a regular nativescript (so the GDNative resource loader would call Nativescript on it automatically).

I agree it would be overkill to write a JIT compiler for GDScript. As you mention, JIT compilation is not allowed on iOS (Android does allow it), so you would either have to fallback to the slow interpreted on that platform or write an AOT compiler (look! now you have to maintain two compilers as well as the interpreter).
I said many times that I consider the GDScript2C transpiler to be the best option now that we have GDNative.
For binaries built with the mono module, you could write a GDScript2IL compiler and then you would have both JIT and AOT compilation.

From what I saw of GDScript, a really cool (lack of) feature is it is not a dynamic language. There is no exec keyword, no monkey patching, you cannot modify variable during debugging etc.
This makes the language suitable for static compilation (and in fact it is again one more reason not to go with JIT).

Actually, in GDScript you can modify the source code for a script at runtime.

Actually, in GDScript you can modify the source code for a script at runtime.

Not as much as "modify". You can set the source_code property of Script with a whole new code and then call reload() to recompile it. Technically this is implemented for C# as well, since it's part of the Script API.
@touilleMan your idea is similar to what's discussed in #11068.

Problem with transpiling GDScript, or compiling it ahead of time, is that even with the static type system all method calls still have to be virtual. Anything could hide behind a Node parameter. So you'd run into similar performance problems as Java, which uses JIT on its bytecode for (in part) the same reasons.
In any case, since I need a practical part for my master's thesis I'm going to implement a JIT for GDScript anyway, so after that we can see how well it fares and how much it destroys the code base. I honestly wouldn't mind even if it stays a prototype forever and never gets merged.
Also, for the dynamic things, you can load script files from anywhere on the disk and use them for anything you can use scripts in your project for. Which is a very nice feature for DLC and/or user created content (though I don't know if anyone is actually using it).

@vnen The source code in C# is only used for scripting. It's not compiled on reload.

@touilleMan

Considering this second point, tinycc seems a really interesting solution given it low weight, it number of platforms supported (x86, amd64, arm, arm64 for linux, win32 and osx), it libtcc.h api that allow to integrate it into a program and it cross compilation mode. On top of that version 0.9.27 has been released last December (so the project seems stile alive ^^).

Bad idea, the project is more likely dead than alive!
@neikeq

For binaries built with the mono module, you could write a GDScript2IL compiler and then you would have both JIT and AOT compilation.

But this is an interesting option. And perhaps comparable to the labor costs with the addition of JIT.
In general, if there is already a person who will be engaged in this in any case, then let him do it. The team does not lose from this. I propose to relax and wait, suddenly something will turn out (although I do not count on it)

@DoctorAlpaca considering the virtual method dispatching, I guess things are already this way when using the C++ API. So your goal here is to make GDScript faster than C++ ? ;-)
Anyway, I'm sorry if my previous post had appeared to you in a "it's useless, you shouldn't do it" style.
Experience is immensely more valuable than supposition so I can only support you no this, on top of that there is great chances your work will bring you to improve various part of the engine (bug fixing, cleaning, or even finding optimizations for the too-dynamic-to-be-fast Variant system) so nothing is never lost !
BTW if you're really interested in JIT, I suggest you come to the Europython (end of July in Edinburgh, I'll be there ^^), the Pypy team is there every year and you can do sprint coding on Pypy with them during the whole weekend. There is usually not much people and they are really kind and pedagogic so it's basically like taking a 2 days class on JIT with world class experts ;-)
@ret80 I agree the tinycc project is not really active. On the other hand we are talking about a C compiler, the most used standard is still C99 (with C89 still really present) so there is not much evolutions to do once the project is stable enough (I guess we could say evolutions could be done in optimization, but this would likely add to the size and the speed of the compiler, so not the goal of this project).
My biggest concern for this compiler is more the support of arm architectures and cross compiling .dll and .dynlib because they are more exotic feature (so potentially less tested).
Finally given we would use this compiler with autogenerated code (always same constructions and gdnative as single dependency), we should be relatively safe not to hit hidden bugs once the project is on track.
@neikeq I find really elegant the idea of using the mono runtime for the JIT ! This would keep low the number of dependencies (and improve the number of people involved in the mono module which is always better) and they would save us from making yet another release.

Honestly I would not JIT but rather AOT compile.  Something like TerraLang (a lua frontend to LLVM designed to dynamically build languages) would be useful for both development hot-reloading of source as well as compilation out to binary native objects for compilation linking (no interpretation, pure LLVM compiled) would be useful as it would be fairly trivial to specialize for even specific Node types.  Virtual overhead would likely still be an issue but it would be a great deal faster than how it is now, especially once typing information starts to exist.  Personally I'd just embed LLVM into the editor/compiler, yes it would increase the size but it also means that games themselves could end up even smaller, especially if you include the object files and/or source of godot with the editor and let it LTO it all into a smaller and faster final output.

The luaJIT project illustrate well those points: the project is simply brilliant (code base really good, performances crazy good), it solve a real usecase, it is used by commercial companies.
And yet it is slowly dying since it main author has step back, unable to keep up with the new lua version (luaJIT stopped at 5.1, arguing the add of integer on top of float in the language made by the 5.2 would be really complex to add), even if the lua community has a medium size and luaJIT it best known project.

I'm not sure I would call it dead, it was forked on github a long while ago and is being updated to modern LUA versions and so forth.

I find really elegant the idea of using the mono runtime for the JIT ! This would keep low the number of dependencies (and improve the number of people involved in the mono module which is always better) and they would save us from making yet another release.

Still not allowed on iOS and so forth though.

I still think JIT idea should be merged with GDNative somehow so that GD Script gets pre-compiled a kind of 3rd party C opcode and GDNative should be able to parse this opcode  and translated into C++ code.

@OvermindDL1 thanks a lot for mentioning TerraLang, it seems like a really interesting project ;-)

I'm not sure I would call it dead, it was forked on github a long while ago and is being updated to modern LUA versions and so forth.

I'm not really close to the lua community so my understanding can be wrong on this topic. Can you point me to the forked version of luaJIT implementing the new lua versions ?

@OvermindDL1 thanks a lot for mentioning TerraLang, it seems like a really interesting project ;-)

They are very PR friendly, it's a really nice project that just isn't really noticed by many as of yet.  It is quite nice to whip up a language inside, built in parsers, LLVM integration, can output fully compiled and optimized binaries, etc...
Even considering that, if I were to write a native backend compiler for GDScript, I'd probably just use the latest version of LLVM's C++ API.  The C API is stable, but the C++ API just has so much more power, a lot of power that is needed for implementing a language and not just scanning, especially if you want to use libclang for the full optimization suite.  However, TerraLang is entirely functional in that way as well and is significantly easier to use if less well tested and using an older LLVM version (C++ API, though they want to update, so of course PR's welcome).

I'm not really close to the lua community so my understanding can be wrong on this topic. Can you point me to the forked version of luaJIT implementing the new lua versions ?

I ran across a couple of interesting ones at https://github.com/LuaJIT/LuaJIT/network at one point, it's been forked so many times and there is so much development work happening on it on a few big forks.  I'm not really big in the Lua community either, I implement luajit in a few projects to get scripting support but don't really write lua myself (apparently lua has a package manager nowadays, luarocks or something).

I admit that I kinda lost track of all the stuff people suppose here, but there seem at least some that say GDScript needs to become statically typed or the JIT should be based on LLVM.
Recently I started working on JIT for GDScript by using RPython (the toolkit PyPy uses to create their JIT), and I've looked into using LLVM as an alternative also.
To make a long story short: LLVM is not suitable for JIT, I've read quite a few blog posts coming to that conclusion, and my experiements with it also seem to confirm it. There are two main reasons:

LLVM** is slow. I do not mean the code generated by LLVM, I mean the time it takes for LLVM to create that code creates a bottleneck. Webkit's JIT switched from a LLVM to a homebrew solution, and that was one of the reasons

LLVM assumes your code is statically typed, and is just a bad fit for compiling dynamically typed languages.

This is why I have given up on LLVM, and am working with RPython. It has pros and cons:
The pros

RPython is a great system to build tracing JITs. For dynamically typed languages a tracing JIT seems to be the best solution. Implementing a tracing JIT is hard work and requires to add measuring points to non jitted code. RPython now exists for over 10 years, has very intelligent people working on it, and it requires you to do very little to get a running JIT. Basically, it expects you to write a bytecode interpreter in RPython and tell it which bytecodes are jump instructions and it does the rest.

The JITted code is quite fast, and it supports x86, x64 and ARM backends

The cons

Variant.cpp needs to be rewritten in RPython. GDScript's solution of dynamic typing, the Variant class, makes it hard to create good JITTed code. One of the things the JIT pipeline does is noticing when a function is often called with an int parameter, for example, and then optimizes for this case. If we would just use Variant as opaque types, tests suggest we can only get a 2x speed improvement, because all we end up effectively is compiling GDScript into C++ code that effectively calls Variant for every single operation, like adding or substracting.

Therefore, to get actual use out of the JIT, at least some parts of Variant have to be duplicated as RPython code so the JIT knows whats going on. It isn't very pretty, but seems to work.

RPython creates C code, not C++. Kinda unfortunate, but we can make it work, especially with the GDNative interface

We have code written in two languages, C++ and RPython. However, Godot already uses SCons, which is Python, so strictly speaking we're not adding a new language

RPython is not fun to program in. Basically you end up writing in a language that's somewhat more powerful than C, but not as powerful as C++, and it takes some getting used to what Python constructs you can use and which you can't

The translation process to turn the RPython code into C (which will be the fully working JIT) takes ages. In this early stage of development, it "only" takes 1-2 minutes, but I'm expecting it to take 20-30m for the somewhat complete implementation. This isn't as horrible as it seems however, because the C code would only needed to be generated when the JIT is modified itself, otherwise, the Godot compilation process can just use the generated C files.

I'm hoping to get a PoC working in 4-8 weeks for more feedback.

@baekdahl   Too much reading. GDScript is going to become typed regardless of what you think.    The decision came from the very top and is gonna happen in the near future.

@CarlGustavAlbertDwarfsteinYung GDScript is becoming optionally typed. Big difference for this problem.

To make a long story short: LLVM is not suitable for JIT, I've read quite a few blog posts coming to that conclusion, and my experiements with it also seem to confirm it. There are two main reasons:

That's because the JIT should be at dev time.  When making a release then a full static native compilation should happen.  LLVM may not be the fastest JIT, but it is a JIT while also being able to fully native compile to standalone libraries/objects/programs.

LLVM assumes your code is statically typed, and is just a bad fit for compiling dynamically typed languages.

Mmm, actually it has some quite fantastic dynamically typed interfaces.
Regardless, if GDScript is becoming typed then that's not an issue anyway, the 'dynamic' parts of GDScript would just become a variant anyway.

This is why I have given up on LLVM (which is a shame, because I really like it), and am working with RPython. It has pros and cons:

Is it capable of native compilation?  If it is not then it seems like a non-starter as you cannot JIT on platforms like iOS, in addition that a JIT at runtime will slow loading and spool-up where AOT has no such cost upon releases.

GDScript is becoming optionally typed. Big difference for this problem.

Yes, but even the 'untyped' parts are variants, which is rather trivial to represent in LLVM.

That's because the JIT should be at dev time.

Then it is not JIT, but normal ahead-of-time compilation.

Is it capable of native compilation? If it is not then it seems like a non-starter as you cannot JIT on platforms like iOS, in addition that a JIT at runtime will slow loading and spool-up where AOT has no such cost upon releases.

No, it's a JIT. And if iOS doesn't allow JITs, it's not going to work there. 🤷

Yes, but even the 'untyped' parts are variants, which is rather trivial to represent in LLVM.

But you don't get that much speed improvement out of it, as I mentioned.
It is rather easy to compile GDScript like this:
while i<10:
    foo(i)
    i = i + 1

to something equivalent to the following C++
// i, foo are Variant
while (i.op("<", Variant_10_const)) {
    foo.call(i);
    i = i.op("+", i, Variant_1_const);
}

(or something like that, don't remember the exact Variant API, it's been some time)
But this will only get you around a 2 times improvement in speed. Turning it into a while (i<10) {foo.call(new Variant(i)));i += 1} is much more difficult to achieve.

Then it is not JIT, but normal ahead-of-time compilation.

A JIT may not produce quite as fast of code in general as AOT, but it is much faster turn-around time during development so it is still useful there.  However I'd be quite good with AOT even at dev time since if the individual modules are compiled separately then it is not much of a speed issue regardless (<1s easy).

No, it's a JIT. And if iOS doesn't allow JITs, it's not going to work there.

Ah yeah that would not be useful then.  iOS requires either a full slow interpreter, or it requires AOT.  In addition AOT loads faster, requires a smaller runtime, can perform LTO, etc...

But you don't get that much speed improvement out of it, as I mentioned.

Except Typed GDScript will be able to infer 'most' types regardless (it could even be able to decorate the variant on the get_node calls to create a happy path for the expected node type based on the scene with standard variant fallback).  However, in the great majority of cases, even in non-typed code, it will still be able to infer the great majority of types, thus gaining significant performance enhancements and generated code reduction.

(or something like that, don't remember the exact Variant API, it's been some time)

Eh, that is not even remotely how I would suggest doing it honestly.  Performing a native < operation via a visitor would be significantly more performant than string testing as just one example.  However I'm not sure this example is very valid as I'd imagine that the type of i would be inferred as integer up above where it is set (unshown code) and even if it is not then there could be a single type check before the loop to check if it works with such integral comparisons and if it does not then throw an exception or error in some other way, otherwise acquire the integral/float value out of the variant (likely cast to something useful to minimize code gen, LLVM already has passes to optimize this though) and the loop would then become fully optimized regarldess.  You want to minimize variant time entirely and make checks for it's access early and often.  Falling back to visitors is not that slow either especially since the majority of the visitor would just throw an error (like calling a method on an integer).

But this will only get you around a 2 times improvement in speed. Turning it into a while (i<10) {foo.call(new Variant(i)));i += 1} is much more difficult to achieve.

This seems pretty trivially optimizable though?  Everything would be quite native right down to calling a trampoline for foo to dispatch appropriately or so (simple wrapper function to perform the untyped to type dispatch-or-error), but even then if the type of foo is known (possible also inferrable if not explicitly typed) then that would be optimized out as well.

I'd love to hear about those.

Other than inline native-code-generating JIT that LLVM itself has (which would work perfectly with all the above and would be fantastic for dev time edit-code-while-running while still able to do AOT for Releases) that allows for a direct JIT translation, there are also libraries such as pyston, fastlua, among others built on LLVM that have various forms of tracing JIT's for on-demand improvement, which I personally find fairly useless for dev time and you can't use at a reliable Release time anyway because of platforms like iOS, so I don't see the point.

@OvermindDL1 You are still talking about AOT that gets automatically triggered when saving an edited GDScript.
And yes, if GDScript gets optional static typing, those parts where types are enforced can be converted into pretty effective machine code. GDScript is also pretty limited with what it can do, so I don't think even making it strictly static everywhere wouldn't be that much of a loss.
And if people want to implement such a thing, they can. I have already written some static compilers and that problem isn't interesting for me.
What's interesting to me is creating a JIT for a dynamically typed GDScript that runs on PCs. It would be nice if other people would end up using it too, but it's not a requirement for me. I appreciate the "you might be wasting your time" warning though.

Eh, that is not even remotely how I would suggest doing it honestly. Performing a native < operation via a visitor would be significantly more performant than string testing as just one example.

IIRC, it's not really a "<" but more of a Operation.LT parameter that's given.

This seems pretty trivially optimizable though?

It isn't though, unless you want to reimplement a complete new GDScript interpreter. And even then, you will have a hard time inferring types except for trivial cases.
If you simply want to replace the opcode interpreter part, it's not as trivial (fun fact: the ".gdc" files are not bytecode, but more like the tokenized source code that gets interpreted and turned into in-memory bytecode later). At this point in time, you know the following:

There's a variable i, which gets compared ("less than") to another variant whose contents are opaque (although you could use the internal API to see it's an integer with value 10)
There's another variable "foo", which gets called with i as the parameter
i gets assigned a new value, the result of adding i to another opaque variant

If you want to be able to do AOT compilation, you basically have to reimplement the whole GDScript interpreter. Which is not as trivial as it sounds, as a significant part of the interpreter is part of the Variant class. And Variant is what holds godot together, even the parts that are not GDScript heavily rely on Variant.
It really isn't as easy as it sounds at first; I was also surprised when I first dived into it.

You are still talking about AOT that gets automatically triggered when saving an edited GDScript.

Not everywhere, rather only on Export.  Now I would prefer if it was AOT on save but it doesn't have to be, you could just AOT on Release/Export and JIT when running it via the editor or so to allow for hot-swapping of code during runtime.

And yes, if GDScript gets optional static typing, those parts where types are enforced can be converted into pretty effective machine code. GDScript is also pretty limited with what it can do, so I don't think even making it strictly static everywhere wouldn't be that much of a loss.

Based on the current spec there is a lot of inference, 'most' code should in general by automatically typed by inference.  I'm quite excited about that.  (I'm a fan of HM typing, have implemented it a dozen times in a variety of ways over the years!)

What's interesting to me is creating a JIT for a dynamically typed GDScript that runs on PCs. It would be nice if other people would end up using it too, but it's not a requirement for me. I appreciate the "you might be wasting your time" warning though.

Yeah a JIT absolutely should not be mandatory as that means it would not be exportable to all supported platforms (in addition to adding overhead).  AOT or interpretation should always be the primary method, and considering that most platforms that don't support JIT (iOS, webassembly, etc...) also tend to be the platforms that are the least efficient, then AOT should absolutely be the default, not interpretation, which is even better since typing is being added.

It isn't though, unless you want to reimplement a complete new GDScript interpreter. And even then, you will have a hard time inferring types except for trivial cases.

I'm not talking about for interpretation, I'm talking about for compilation.  If entirely untypeable code gets compiled then it would just become variants in machine code (tagged sum type), and any access of it, like say an operator< function would just perform a vistor call on that, which involves a single static jump based on the dynamic type value, then execution of the static code (which if it is, say, an integer is inlined, if it is, say, a custom type then that would likely end up performing a virtual call, or even a static call if it is a large type implemented directly in the variant like a vector).

If you simply want to replace the opcode interpreter part, it's not as trivial (fun fact: the ".gdc" files are not bytecode, but more like the tokenized source code that gets interpreted and turned into in-memory bytecode later).

Not at all, opcodes should not exist whatsoever by the time that AOT is complete, it should be pure machine code lowered to at the largest of variant dispatches on machine code or to the smallest of perfectly known and typed efficient machine code.

There's a variable i, which gets compared ("less than") to another variant whose contents are opaque (although you could use the internal API to see it's an integer with value 10)

If the variable i's type is known then you can generate machine code that tests the type of the opaque variant and if it is not an immediate compatible match then immediately error, else then perform the comparison based on the i type.  If the variable i's type is not known then it will be a full visitor dispatch on the two variants (a statically known function call) to acquire the type tags of the variants and jump to the appropriate machine code for the given types.

There's another variable "foo", which gets called with i as the parameter

If the call type of foo is known, as an example in ML notation, int -> int, then if i's type is also known and is also an int then perform efficient machine code, if i's type is not known then test the type tag if it matches, if it does not then error, if it does then extract the value and perform efficient machine code.  If the call type of foo is not known but the type tag states it is callable (else error) then if i's type is known (for all definitions of known here I mean both known at compile-time and is consequently unboxed) then wrap it in a variant (efficient inline setting of the proper type tag and placing i's bits into the proper bits) and perform a variant call to the wrapped trampoline function, which then verifies the arguments and calls the actual internal function (which would have been called directly if foo's call type was statically known).

i gets assigned a new value, the result of adding i to another opaque variant

All relies on what the known static type of i is now, and perform the appropriate above described native code generation as necessary.

If you want to be able to do AOT compilation, you basically have to reimplement the whole GDScript interpreter.

Which is still less work than creating a JIT specifically for GDScript unless you are wrapping an existing JIT (of which it's expected operations may not match GDScript perfectly).  AOT generation itself is not much more difficult than creating an interpretor, especially on LLVM as it does the significantly hard work for you (optimizations, LTO, machine code generation, debugger, etc... etc...).

Which is not as trivial as it sounds, as a significant part of the interpreter is part of the Variant class.

But still not as hard as implied (I've written quite a variety of languages over the decades, many down to machine code, both directly to x86 and via LLVM).

And Variant is what holds godot together, even the parts that are not GDScript heavily rely on Variant. So when you start messing with it, you suddenly are risking breaking backwards compability for the GDNative interface.

Which is precisely why variants should be baked straight into the machine code.  No opcode interpreter or anything of the sort, instead do full and proper visitor dispatching (whether via a local jump based on the type tag or even via a longcall from a static array (dispatch table)).  This would not only keeps backwards compatibility but also allows for the GDNative interface to then directly load the fully AOT compiled libraries (or if Godot gets the ability to LTO the engine to the GDNative interfaces via binary objects then LTO optimizations could be applied, which could then signficiantly reduce the output size of the final program as then known unused code could be removed, though that would be a later task).

It really isn't as easy as it sounds at first; I was also surprised when I first dived into it.

It's certainly a non-trivial amount of work, and in fact most of it is a large amount of busy work to translate the previous GDScript compilation pipeline to generate LLVM Assembler Code (of which it has a fantastic API to do so though) instead of opcodes.  But it is entirely doable, and it not quite a long a time as one might expect given a language implementation.

BTW, all Variant operators are treated in here: https://github.com/godotengine/godot/blob/d2b75557a5dedf951ee036ca01af4f94bc059069/core/variant_op.cpp#L391-L392
Which is quite optimized already.

But you don't get that much speed improvement out of it, as I mentioned.
It is rather easy to compile GDScript like this:
while i<10:
foo(i)
i = i + 1
to something equivalent to the following C++
// i, foo are Variant
while (i.op("<", Variant_10_const)) {
foo.call(i);
i = i.op("+", i, Variant_1_const);
}
(or something like that, don't remember the exact Variant API, it's been some time)
But this will only get you around a 2 times improvement in speed.

So wouldn't it already make compiled GDScript faster than C# in most cases?

So wouldn't it already make compiled GDScript faster than C# in most cases?

Likely not, given that it would have to pass through the whole Variant layer of indirection. Looking at languages such as Wren, it seems that using NaN-tagging or unboxed types is more detrimental to performance than compiling things to machine code. C# has both unboxed types and JIT compilation, so compiling GDScript to machine code won't magically make it faster than C#.
Note that GDScript already has a pretty optimized bytecode dispatcher, having machine code to do the same would not speedup the bottleneck.

Well, perhaps a good solution would be something that generates the unoptimized c++ code from the gdscript (perhaps the sconscript file too) and then the user would optimize and compile it manually.
It's not perfect but it's faster already than doing everything by hand.

If you are going to write the code twice, why not write it in C++ directly?

That's another thing about translating GDScript directly to LLVM, can AOT a binary for any platform, not just on the one you are currently on and cross-compileables.

This issue reminds me of the Boo language. A python inspired language running on .NET.
I don't know what it's worth (I haven't really used it tbh), and the project seems dead (5 years? Damn...) but I'm tossing it in anyway.

I prefer Kotlin's type declaration. It's more readable
var name:String = "Bob"

Ruby has a new JIT compiler: https://bugs.ruby-lang.org/projects/ruby/wiki/MJIT#MJIT-organization

Just curious, is this planned for 4.0?

@girng, there is a suspicion that this will not happen in Godot 4.0. But it is not exactly :)

I think v4.0 could have preliminary GDScript-to-C transpiling, just like v3.0 had preliminary C# support.
(if we're lucky enough).

It’s easier to make support for the ZIG language. Promising language. Although I really like the GDScript, and if it worked faster it would be very cool. By the way, it would be great if it were not for the file itself to be a class; the class was written in the file, as is done in other languages. Because of this feature, some difficulties arise which could easily be eluded.

Godot 4.0 most likely won't have JIT, but it will have performance improvements in GDScript.

I have done significant work toward compiling GDScript to C with GDNative. I have arrived at a point where I am not convinced that it is a worthwhile endeavor to complete. Theoretically the performance of AOT code could be orders of magnitude faster than GDScript in execution, but in practice the results are more disappointing. The costs of going through the GDNative layer are significant, and without major changes to GDScript itself the compiled code is very brittle (GDScript bytecode is not generated in a way that is safe to run in a later execution because it expects the environment to be static, including positions of things in the engine’s internal arrays).
I am however convinced that by implementing a minimal amount of back-end optimizations that some real performance improvements could be realized.
I am working on a fork that includes a number of optimizations that require no significant changes to GDScript at all, but already nets some performance gains.
So far I have implemented the boilerplate data structures to perform code optimization, as well as a few basic optimization passes:

construction and destruction of control flow graph
data flow analysis including gens and kill sets, live ranges
dead code elimination
jump threading (important optimization) as GDScript creates much unneeded branching

As a POC I plan to implement a few more passes prior to pushing up my fork:

common subexpression elimination
type inference of temporaries and native arithmetic for temporaries (I am also not convinced typed arithmetic will be a huge win)

You can see my progress on the above mentioned optimizations in my godot fork, https://github.com/pchasco/godot/tree/gdscript-optimization/modules/gdscript/optimizer

@pchasco You may want to talk to @vnen, who is currently reworking GDScript for Godot 4.0 - IMO, this would be a good time to make sure to get such low-hanging fruit in :)
See https://godotengine.org/article/gdscript-progress-report-writing-tokenizer

@pchasco You may want to talk to @vnen, who is currently reworking GDScript for Godot 4.0 - IMO, this would be a good time to make sure to get such low-hanging fruit in :)
See https://godotengine.org/article/gdscript-progress-report-writing-tokenizer

http://blog.moblcade.com/?p=114

Well, the VM is gonna change to integrate typed instructions so I'm not sure if doing optimization work now is the best. I do believe that those ideas can be used in the new VM, but it'll take a while until it's completed.
I'm not sure if we'll ever have JIT though, so maybe this issue could be closed in favor of something else. There's an idea of doing AOT compilation which would be easier to manage than this.

@vnen Yeah the issue title isn't fitting anymore, if you read the above they found JIT not to be worth it mostly and instead went for targeted optimizations in the current interpreter.
It may be worthwhile to keep those in mind while reworking that, but yeah, you'll know way better than me when a good time is to see about those :) just wanted to connect you two

http://blog.moblcade.com/?p=114

@pchasco You may want to talk to @vnen, who is currently reworking GDScript for Godot 4.0 - IMO, this would be a good time to make sure to get such low-hanging fruit in :)
See https://godotengine.org/article/gdscript-progress-report-writing-tokenizer

http://blog.moblcade.com/?p=114

I see you found my blog!

Well, the VM is gonna change to integrate typed instructions so I'm not sure if doing optimization work now is the best. I do believe that those ideas can be used in the new VM, but it'll take a while until it's completed.
I'm not sure if we'll ever have JIT though, so maybe this issue could be closed in favor of something else. There's an idea of doing AOT compilation which would be easier to manage than this.

I agree that JIT is not a workable solution given the number of platforms, varying hardware, and amount of work necessary to maintain. AOT is definitely the simplest and the most portable solution for native performance with GDScript. In fact Unity transpiles .NET to C++ for its AOT also. I would be interested in contributing with the GDScript rewrite.

@pchasco I'd suggest joining the #godotengine-devel IRC channel and get in contact there :)

As I mentioned before it's very unlikely that we'll add JIT. There are some other ideas in the discussion but no conclusion.
Since we are moving proposal to the GIP tracker, I'll close this one and if someone has an idea, please open a new proposal (after looking if nobody else did it first). This issue can be linked in relevant proposals to keep the discussion history.
Some contributors have different ideas for AOT compilation, but I'll let them open a proposal with what they have in mind.

@vnen
Then please update the Roadmap, it's misleading people.

Well, I even forgot the roadmap exist, it isn't updated in a long time.

@vnen
You forgot, but other users didn't.  There are plans to optimize GDScript in many ways, including compilation to native code, just not with JIT. Also if you compare with C++ even a JITed runtime will be slow.

@vnen: Just sayin' that LuaJIT is nearly up with C/C++ on performance benchmarks - but there is a widespread belief that magic is involved :smile:

JIT is not possible on all Major platforms. It is better to focus efforts
where all can take advantage of them. Ahead of time compilation is the only
100% compatible technology.
On Sun, Aug 9, 2020 at 8:46 AM Zireael07<EMAIL_ADDRESS>wrote:

@vnen https://github.com/vnen: Just sayin' that LuaJIT is nearly up
there with C/C++ on performance benchmarks - but there is a widespread
belief that magic is involved 😄
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/issues/5049#issuecomment-671047838,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAUFIAGK3S7RA3DRXW6NUC3R72LCVANCNFSM4CFZMZPA
.

Aren't there downsides to AoT compared to JIT? For example what about dynamically generated or loaded scripts? JIT shouldn't have a problem optimizing them, but AoT?
By the way, on what major platform is not JIT possible? I thought Java with its JIT runs pretty much everywhere.

iOS does not allow JIT for applications distributed through the App Store
unless it is using Apple’s browser component.
AOT would not be possible for dynamic scripts.
On Sun, Aug 9, 2020 at 11:15 AM monnef<EMAIL_ADDRESS>wrote:

Aren't there downsides to AoT compared to JIT? For example what about
dynamically generated or loaded scripts? JIT shouldn't have a problem
optimizing them, but AoT?
By the way, on what major platform is not JIT possible? I thought Java
with its JIT runs pretty much everywhere.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/issues/5049#issuecomment-671064117,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAUFIACWWCWDXO4NZ7GCL63R724RHANCNFSM4CFZMZPA
.

iOS, Web and a few console platforms do not support JIT compilation.

There's a rumor that AOT is now throwт out of the deck with JIT. I hope, that's not true. Somebody, please tell me, is it true?

@Calinou @neikeq @reduz @akien-mga

Are you saying there is a rumor AOT would no longer be allowed on some
devices? That’s literally impossible, unless I’m misunderstanding something?
On Wed, Sep 30, 2020 at 5:47 AM Teashrock<EMAIL_ADDRESS>wrote:

@Calinou https://github.com/Calinou @neikeq https://github.com/neikeq
@reduz https://github.com/reduz @akien-mga
https://github.com/akien-mga
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/issues/5049#issuecomment-701313134,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAUFIADVHAXMGBOL3YEDVILSIMEEFANCNFSM4CFZMZPA
.

@pchasco
No. I mean, somebody told me that GDScript will have neither AOT nor JIT.

Maybe it will not be officially supported, but I can confirm that the new
GDScript architecture outlined by vnen is capable of supporting it. An
enterprising developer could implement a custom module to do AOT to
GDNative C.
On Wed, Sep 30, 2020 at 11:16 AM Teashrock<EMAIL_ADDRESS>wrote:

@pchasco https://github.com/pchasco
No. I mean, somebody told me that GDScript will have neither AOT nor JIT.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/issues/5049#issuecomment-701494412,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAUFIACAKZVJQK2CHTDJT63SINKXDANCNFSM4CFZMZPA
.

@pchasco

maybe

So, you don't know.

@Teashrock We don't know for certain if GDScript will feature JIT or AOT compilation in 4.0, but it's not very likely. Maybe in 4.1 or later…

@Calinou
It's just that they talked about 4.0 regarding JIT/AOT. And now you're saying me about 4.1. Why's that? How's that? Somebody, just give a clear public anwser, which won't be somewhere deep on GitHub.


Enable all output to go to a directory

This allows all output to be written/read to a user specified output directory via a new command line option -o <dir> or --output_directory <dir>.
This includes loading restarts as well.  So users will have to use the same output directory command line option if they want to restart.
Things that were moved:

Inlet docs + input file value file
StateManager state files
Summary file
Field data files

Note: I removed the SLIC logging stream that wrote to a file for two reasons:

it's not a full capture of all output to the screen because not all things are going through SLIC
it was created pre command line options and therefore slightly complicated to fix though doable.

Codecov Report

Merging #527 (dbd8826) into develop (84744c1) will increase coverage by 0.20%.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##           develop     #527      +/-   ##
===========================================
+ Coverage    96.14%   96.35%   +0.20%
===========================================
  Files          103      103
  Lines         6457     6467      +10
===========================================
+ Hits          6208     6231      +23
+ Misses         249      236      -13

Impacted Files
Coverage Δ

src/drivers/serac.cpp
96.29% <100.00%> (+0.29%)
:arrow_up:

src/serac/infrastructure/cli.cpp
97.56% <100.00%> (+2.96%)
:arrow_up:

src/serac/infrastructure/output.cpp
94.00% <100.00%> (+2.00%)
:arrow_up:

src/serac/physics/base_physics.cpp
98.33% <0.00%> (+0.83%)
:arrow_up:

src/serac/infrastructure/logger.cpp
96.77% <0.00%> (+16.12%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 84744c1...dbd8826. Read the comment docs.

Comments should be addressed.  Really ready for review this time.

Added a new simple sphinx page for command line options here:
https://serac.readthedocs.io/en/bugfix-white238-fix_overlapping_output_files/sphinx/user_guide/command_line_options.html


Update AI21 model on extension

Hi, So, I wanted to try out AI21 on codegpt but found two model j2-grande-instruct and j2-jumbo-instruct. However, AI21 documentation states that there are only 3 variants of ` encompasses three variants - Ultra, Mid, and Light. Can this be updated for the extension?

@grizzlybearg Do you have the AI21 Documentation for this models. Could you share the link?

@grizzlybearg
We already have the j2 light, mid and ultra models!
They come in the next version. Thank you so much!

Thanks


fix(docs): "Edit this page" URLs to correctly point to GitHub repository structure

What this PR does:
This PR fixes the "Edit this page" URLs in the documentation pages to correctly point to the appropriate locations in the GitHub repository structure.
Why we need it:
The current "Edit this page" URLs are incorrect, pointing to a wrong path in the repository (missing .md extension).
Which issue(s) this PR fixes:
Fixes #5287
Does this PR introduce a user-facing change?:

How are users affected by this change:
Is this breaking change:
How to migrate (if breaking change):

@ntheanh201
Thank you so much.
Would you commit with -s to pass the DCO check?
cf. https://github.com/pipe-cd/pipecd/pull/5289/checks?check_run_id=31926903751


adding trailing slash for service calls

👍 , will merge when tests pass.

This broke dcos-commons tests

There is a change coming in DCOS which would not accept the lack of a trailing slash.  that would have affected all clients.   This was the best course of action.

Sorry about that @mgummelt -- this was definitely not intended to be a breaking change!  Can you provide details on why this broke dcos-commons?  I'd like to add a test for this scenario so that something similar doesn't happen again.

Ken, what DCOS change?
On Mar 14, 2017 2:56 PM, "Scott Schneider"<EMAIL_ADDRESS>wrote:

Sorry about that @mgummelt https://github.com/mgummelt -- this was
definitely not intended to be a breaking change! Can you provide details on
why this broke dcos-commons? I'd like to add a test for this scenario so
that something similar doesn't happen again.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dcos/shakedown/pull/129#issuecomment-286573768, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAJYhsnd8wXSmzaujnmlsjjvB3LwUxYqks5rlw0kgaJpZM4MQLHL
.


Bug: The header in the details layout blinks when the cursor is hovered over

Description
As the title describes. It occurs only when the theme is light and the cursor is hovered over the header for the first time after opening a tab.
Steps To Reproduce

Set the theme to light.
Open a new tab.
Open a folder with the details layout.
Hover over the header.
The header blinks only once.

Requirements
Prevent blinking.
Files Version
<IP_ADDRESS>
Windows Version
10.0.19045.3208
Log File
n/a

@hishitetsu thank you for reporting this issue.


Deprecate k8s 1.5.x

What this PR does / why we need it:
Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #2395
Special notes for your reviewer:
Release note:

@rjtsdl Thanks for this! To be clear the PopulateClassicModeDefaultValue FuncMap is not strictly related to 1.5 and should be retained?

@jackfrancis it is not :)
PopulateClassicModeDefaultValue is tied with RPv1.

@jackfrancis  thx !


Add tilesMissingThisFrame to ViewUpdateResult, rather than putting unrenderable tiles in tilesToRenderThisFrame

Tile with ContentLoaded usually receives the engine's render resources from the worker thread. Whereas, a tile with Done status has render resources from the main thread. Those resources are different. Engine receives ViewUpdateResult when calling Tileset::updateView() will expect render resources from the main thread, and reintepret_cast<> accordingly. However, currently, there are a mix of ContentLoaded and Done tiles in the ViewUpdateResult which will lead to undefined behavior when reintepret_cast

Can this be reproduced trivially with just CWT or something? Or does it only happen in certain circumstances?

Some "notes" from talking about that quicky:

The exact type of the pRendererResources is completely governed by the client (i.e. the Unreal Engine or other clients): The client implements the IPrepareRendererResources, and the client receives the ViewUpdateResult, and has to know what that void* pointer is, based on its IPrepareRendererResources implementation

An ambiguity is introduced by the fact that the pRendererResources can have different types, depending on the LoadState of the tile. At the core, with some abstraction, there is pseudocode like
HalfLoaded halfLoaded = static_cast<HalfLoaded>(tile.pRendererResources);
FullyLoaded fullyLoaded = process(halfLoaded);
tile.pRendererResources = fullyLoaded; // The type changes here!!!
tile.loadState = Done;

When the state transition from ContentLoading to Done takes place, the type of the object that is stored in the void* pointer changes.

The fact that the LoadState determines the type of the data is not documented clearly. (It may be something that can ~"implicitly be derived" from reading lots of code and Doxygen, but ... nobody does that, and it would still require lots of time, and maybe some guesses)

This could be solved by more prominent documentation, but even this would not solve the more general issue: The state transitions are not clear (c.f. https://github.com/CesiumGS/cesium-native/issues/126 ). It is not clear when a tile goes from ContentLoaded into Done (and there seem to be some oddities related to non-Done tiles being in the ViewUpdateResult, but that has to be investigated further)

The solution for now is that the client explicitly has to check the load state, as it is done in cesium-unreal, before casting the pRendererResources to the expected target type. (Or specicially: The type that is expected based on the current LoadState).

In order to make this clearer and less error-prone, there are several options. Some "non-technical" ones are

Documentation of the traversal process
Documentation of the state space
Documentation of the state transitions

Some technical ones are:

The Tile could not only have a void *_pRendererResources, but two void * _pRawRendererResources and void *_pFinalRendererResources to differentiate the different types that these resources may have. One reason to not do this could be the additional memory overhead of another pointer that this implies.

The Tile currently offers a method to access the renderer resources, namely void* getRendererResources() that just returns the internal pointer. This could be split into two functions that abstract away the dependency to the load state, roughly as in
void* getRawRendererResources() {
  if (this.state == ContentLoaded) return this->_pRendererResources;
  else return nullptr;
}
void* getFinalRendererResources() {
  if (this.state == Done) return this->_pRendererResources;
  else return nullptr;
}

This (with more sensible state checks, and explicit and elaborate documentation) could be OK, and free callers from the burden of knowing details about our state transitions, and prevent callers from receiving an unexpected type.

The ViewUpdateResult currently has the std::vector<Tile*> tilesToRenderThisFrame;, which apparently contains tiles that are not really "tiles to render", but rather "tiles that may be in Done or ContentLoaded state". Why this is the case has to be examined. But if there is a reason to return "non-Done" tiles to clients, this could be separated into two vectors, namely tilesThatAreDoneAndShouldReallyBeRenderer, and tilesThatAreInTheContentLoadingStateButGivenToYouNevertheless.

@baothientran In my testing with Unreal today there haven't been any ContentLoaded tiles in the render list, but there have been ContentLoading tiles and Unloaded tiles. In Melbourne particularly, it seems that sometimes Unloaded render list tiles can stay unloaded for entire seconds in some cases before beginning to load, even when there are no ContentLoading render list tiles. I'm saying that anecdotally based on printing the number of tiles in each stage (Done, ContentLoaded, etc.).
As we talked about in standup earlier, it would be strange and worrying to see ContentLoaded tiles since it implies that either:

They finished loading before the tick and were ignored by the tileset traversal,
or that they somehow became loaded after the tile traversal but before the construction of the render list (this is probably impossible since these happen in one continuous main thread execution).

Anyways, it's still worrying that ContentLoading and Unloaded tiles are showing up in the render list, so I'll dig deeper there.

Just to be clear, it is normal and expected for non-Done tiles to be included in the render list. It should be mostly ContentLoading tiles but also some Unloaded when a lot of tiles are needed at once and they need to wait their turn before they can start loading. ContentLoaded is unexpected, for the reasons Nithin mentioned, but not inherently problematic.
This behavior is perhaps unintuitive, but it's intentional. The idea is that the rendering engine may choose to render "something" in place of the not-yet-renderable tiles rather than a hole. Is this very likely? Probably not. And since it's not likely should we change this unintuitive behavior? Probably. Marco's suggestion of two lists sounds reasonable to me. Probably name them thing something like tilesToRenderThisFrame and tilesMissingThisFrame?

This could be split into two functions that abstract away the dependency to the load state

This is a great idea. Naming is hard, as usual. Maybe getLoadThreadRendererResources and getMainThreadRendererResources, corresponding to the prepareInLoadThread and prepareInMainThread methods on IPrepareRendererResources that return each of these void pointers.

I updated the description and label to reflect the desired action here.


Add test packages and clickOutside tests

This PR adds initial test setup and clickOutside tests (#5).
I picked mocha/jsdom/sucrase since that's what the main svelte repo uses. This should make it easy to port over the tests if these actions become official. sinon isn't used in the main svelte repo but it makes callback verification easy.
This PR also refactors lazyload to only init IntersectionObserver when the action is used (though still using a shared IntersectionObserver). Without this change, tests were failing with an "IntersectionObserver is not defined" ReferenceError. This kicks that can down the road until we go to write lazyload tests.
I wanted to get feedback on test structure and setup before writing tests for the other actions.

that's awesome!

that's awesome!

that's awesome @geoffrich! thanks so much for getting the ball rolling on this, it is sorely needed :)
i was thinking that this would happen someday and its probably time - what do you think about colocating the test to the action?

this would make it easy to find/modify/delete, and then we would manually manage index.ts and test.ts to require and export them.
i dont feel SUPER strongly about this, just proposing it as a straw man. honestly i dont intend this project to be that big that folder structure would ever really be a problem

that's awesome @geoffrich! thanks so much for getting the ball rolling on this, it is sorely needed :)
i was thinking that this would happen someday and its probably time - what do you think about colocating the test to the action?

this would make it easy to find/modify/delete, and then we would manually manage index.ts and test.ts to require and export them.
i dont feel SUPER strongly about this, just proposing it as a straw man. honestly i dont intend this project to be that big that folder structure would ever really be a problem

@sw-yx that makes sense to me! I moved each action into its own file and moved the tests into the src folder as described. The Action type is in its own file now too so the individual actions can import it. Let me know what you think.

@sw-yx that makes sense to me! I moved each action into its own file and moved the tests into the src folder as described. The Action type is in its own file now too so the individual actions can import it. Let me know what you think.

i like it :)

i like it :)


School Library: Feature/decorate

Added corrector class and its functionality
Added an instance of corrector into person constructor
Added a method to person class that class constructor instance

Hi @mateo951 👋,
Your project is complete!  There is nothing else to say other than... it's time to merge it :shipit:
Congratulations! 🎉

Cheers and Happy coding!👏👏👏
Feel free to leave any questions or comments in the PR thread if something is not 100% clear.

As described in the Code reviews limits policy  you have a limited number of reviews per project (check the exact number in your Dashboard). If you think that the code review was not fair, you can request a second opinion using this form.


Hover effect on the ListView

ITS LITERALLY TWO LINES OF CODE; DO IT ALREADY.
`    public partial class MainForm : Form
{
public MainForm()
{
InitializeComponent();
}
    [DllImport("uxtheme.dll", ExactSpelling = true, CharSet = CharSet.Unicode)]
    private static extern int SetWindowTheme(IntPtr hwnd, string pszSubAppName, string pszSubIdList);

    private void MainForm_Load(object sender, EventArgs e)
    {

        SetWindowTheme(this.listView1.Handle, "Explorer", null);

}
}

}
`

If your so admit about this perhaps submit a pull request? from there @NYAN-x-CAT will either accept or deny it

here is a POC for those wanting to know what this does
https://youtu.be/hU89jJIrkWQ


Add ZenHub to readme?

Hey, I'm on the ZenHub.io team. I noticed this repo uses ZenHub task boards to organize issues.
To help contributors know to check the task boards for current priorities, what do you think about adding the ZenHub badge to your ReadMe (or adding a mention to your contributor guidelines)? I'm happy to create a PR. A lot of people don't know ZenHub is free for open-source users :smile:
Thoughts?

Hi @feluraschi, I'm afraid we're not actually using it. I tested out the new version for about 10minutes the other day and have it installed, but we haven't used it properly in over a year.  Worth noting, I scoured the site the other day looking for info on whether it was free for OSS and couldn't find that info :wink:


Add changeName

So that I can pre deploy events when the gas price is low.

Nice! This is what we talked about at lunch right? Can we limit it to before the event starts or is that not possible?

@jefflau currently there is no notion of "event start". I just changed so that it allows name change only if no one has registered yet.


Develop library for common IP address allocation  functionalities

Develop library for CRUD operations (getters. setters etc) for IP allocation functions.

done


SOT-127: FRI implementation

[ ] Remember to squash all commits before merging and resolve all the comments


Can't find explicit IV "over the wire" - AES 128, CBC mode, TLS 1.2

Hello, to provide more information and details, from #11266, I am opening a new issue to ask why I can't find the explicit IV where I think it should be.
As mentioned in the other issue, according to the TLS 1.2 spec, the IV field is not part of the encrypted record for CBC ciphers; the definition of the GenericBlockCipher being:
struct {
    opaque IV[SecurityParameters.record_iv_length];
    block-ciphered struct {
        opaque content[TLSCompressed.length];
        opaque MAC[SecurityParameters.mac_length];
        uint8 padding[GenericBlockCipher.padding_length];
        uint8 padding_length;
    };
} GenericBlockCipher;

I've made a dummy aes_128_cbc engine for printing debug information (the IV, the key, and processed blocks; plain and cipher texts). I've temporarily uploaded it here: https://github.com/arielm33/dummy-engine.
When I run the following command:
$ echo "GET /" | openssl s_client -ign_eof -engine dummy -debug -tls1_2 -cipher AES128-SHA -connect www.google.com:443 -servername www.google.com 1> s_client.1.txt 2> s_client.2.txt

In the debug output of the engine I see the IV with which the cipher was initialized:
DUMMY engine DEBUG: dummy.c:121:cipher_aes_128_cbc_dummy_init(): key != iv != NULL, enc == [01], inicialización de ctx.
DUMMY engine DEBUG: dummy.c:129:cipher_aes_128_cbc_dummy_init(): enc = 1
DUMMY engine DEBUG: dummy.c:136:cipher_aes_128_cbc_dummy_init(): key = 3284888A774866A9F47BB76B2C64F786
DUMMY engine DEBUG: dummy.c:156:cipher_aes_128_cbc_dummy_init(): iv = 86F9C69D825D2660B7EF99E8BCBDE0F0
DUMMY engine DEBUG: dummy.c:167:cipher_aes_128_cbc_dummy_cipher(): se cifrará/decifrará vía cipher.
DUMMY engine DEBUG: dummy.c:178:cipher_aes_128_cbc_dummy_cipher(): dummy_ctx->iv = 86F9C69D825D2660B7EF99E8BCBDE0F0
DUMMY engine DEBUG: dummy.c:179:cipher_aes_128_cbc_dummy_cipher(): dummy_ctx->key = 3284888A774866A9F47BB76B2C64F786
DUMMY engine DEBUG: dummy.c:181:cipher_aes_128_cbc_dummy_cipher(): pre - entrada: 64 bytes.
DUMMY engine DEBUG: dummy.c:181:cipher_aes_128_cbc_dummy_cipher():  0: 35630DB4E6B0D4F88B839A79961595CE
DUMMY engine DEBUG: dummy.c:181:cipher_aes_128_cbc_dummy_cipher():  1: 1400000C8CFAFB8AB1F9C5213D58DB91
DUMMY engine DEBUG: dummy.c:181:cipher_aes_128_cbc_dummy_cipher():  2: F794B75AC1C5D2A5943AEDDCCF5F31AC
DUMMY engine DEBUG: dummy.c:181:cipher_aes_128_cbc_dummy_cipher():  3: 4CCD1E160B0B0B0B0B0B0B0B0B0B0B0B
DUMMY engine DEBUG: dummy.c:185:cipher_aes_128_cbc_dummy_cipher(): se invoca para cifrado de 64 bytes.
DUMMY engine DEBUG: dummy.c:193:cipher_aes_128_cbc_dummy_cipher(): post - salida: 64 bytes.
DUMMY engine DEBUG: dummy.c:193:cipher_aes_128_cbc_dummy_cipher():  0: 239C85D786CEEC3EFEF897465163287A
DUMMY engine DEBUG: dummy.c:193:cipher_aes_128_cbc_dummy_cipher():  1: C2D808EF9DFF9FCCA26A07AA029E1840
DUMMY engine DEBUG: dummy.c:193:cipher_aes_128_cbc_dummy_cipher():  2: 51F2298CBC4786CE296B127456A8A1BE
DUMMY engine DEBUG: dummy.c:193:cipher_aes_128_cbc_dummy_cipher():  3: 0CE6C716B7F417965970E919AB9ED92E

Immediately after initialization, the first encrypted block produced is 239C85... But searching for it in wireshark:

In what I thought would be the GenericBlockCipher, I don't see the (explicit) IV. It does not start with the value of the IV that was used to initialice the cipher (86F9C6...). Instead, I find the result of the encryption of the IV xored with the first block of the plain text. The encryption of (86F9C6... xor 35630D...) results in 239C85... (with 328488... as key of course).
Also, in the following records, I see that the last encrypted block is used as IV.
Sorry for being lengthy, and thank you.

Interesting ,
I think that this is duplicate to "cipher failures in master 2019-08-03" (https://github.com/openssl/openssl/issues/9524)
arielm33 wrote:

Hello, to provide more information and details, from #11266, I am opening a new issue to ask why I can't find the explicit IV where I think it should be.
As mentioned in the other issue, according to the TLS 1.2 spec, the IV field is not part of the encrypted record for CBC ciphers; the definition of the GenericBlockCipher being:
struct {
     opaque IV[SecurityParameters.record_iv_length];
     block-ciphered struct {
         opaque content[TLSCompressed.length];
         opaque MAC[SecurityParameters.mac_length];
         uint8 padding[GenericBlockCipher.padding_length];
         uint8 padding_length;
     };
} GenericBlockCipher;

I've made a dummy aes_128_cbc engine for printing debug information (the IV, the key, and processed blocks; plain and cipher texts). I've temporarily uploaded it here: https://github.com/arielm33/dummy-engine.
When I run the following command:
$ echo "GET /" | openssl s_client -ign_eof -engine dummy -debug -tls1_2 -cipher AES128-SHA -connect www.google.com:443 -servername www.google.com 1> s_client.1.txt 2> s_client.2.txt

In the debug output of the engine I see the IV with which the cipher was initialized:
DUMMY engine DEBUG: dummy.c:121:cipher_aes_128_cbc_dummy_init(): key != iv != NULL, enc == [01], inicialización de ctx.
DUMMY engine DEBUG: dummy.c:129:cipher_aes_128_cbc_dummy_init(): enc = 1
DUMMY engine DEBUG: dummy.c:136:cipher_aes_128_cbc_dummy_init(): key = 3284888A774866A9F47BB76B2C64F786
DUMMY engine DEBUG: dummy.c:156:cipher_aes_128_cbc_dummy_init(): iv = 86F9C69D825D2660B7EF99E8BCBDE0F0
DUMMY engine DEBUG: dummy.c:167:cipher_aes_128_cbc_dummy_cipher(): se cifrará/decifrará vía cipher.
DUMMY engine DEBUG: dummy.c:178:cipher_aes_128_cbc_dummy_cipher(): dummy_ctx->iv = 86F9C69D825D2660B7EF99E8BCBDE0F0
DUMMY engine DEBUG: dummy.c:179:cipher_aes_128_cbc_dummy_cipher(): dummy_ctx->key = 3284888A774866A9F47BB76B2C64F786
DUMMY engine DEBUG: dummy.c:181:cipher_aes_128_cbc_dummy_cipher(): pre - entrada: 64 bytes.
DUMMY engine DEBUG: dummy.c:181:cipher_aes_128_cbc_dummy_cipher():  0: 35630DB4E6B0D4F88B839A79961595CE
DUMMY engine DEBUG: dummy.c:181:cipher_aes_128_cbc_dummy_cipher():  1: 1400000C8CFAFB8AB1F9C5213D58DB91
DUMMY engine DEBUG: dummy.c:181:cipher_aes_128_cbc_dummy_cipher():  2: F794B75AC1C5D2A5943AEDDCCF5F31AC
DUMMY engine DEBUG: dummy.c:181:cipher_aes_128_cbc_dummy_cipher():  3: 4CCD1E160B0B0B0B0B0B0B0B0B0B0B0B
DUMMY engine DEBUG: dummy.c:185:cipher_aes_128_cbc_dummy_cipher(): se invoca para cifrado de 64 bytes.
DUMMY engine DEBUG: dummy.c:193:cipher_aes_128_cbc_dummy_cipher(): post - salida: 64 bytes.
DUMMY engine DEBUG: dummy.c:193:cipher_aes_128_cbc_dummy_cipher():  0: 239C85D786CEEC3EFEF897465163287A
DUMMY engine DEBUG: dummy.c:193:cipher_aes_128_cbc_dummy_cipher():  1: C2D808EF9DFF9FCCA26A07AA029E1840
DUMMY engine DEBUG: dummy.c:193:cipher_aes_128_cbc_dummy_cipher():  2: 51F2298CBC4786CE296B127456A8A1BE
DUMMY engine DEBUG: dummy.c:193:cipher_aes_128_cbc_dummy_cipher():  3: 0CE6C716B7F417965970E919AB9ED92E

Immediately after initialization, the first encrypted block produced is 239C85... But searching for it in wireshark:

In what I thought would be the GenericBlockCipher, I don't see the (explicit) IV. It does not start with the value of the IV that was used to initialice the cipher (86F9C6...). Instead, I find the result of the encryption of the IV xored with the first block of the plain text. The encryption of (86F9C6... xor 35630D...) results in 239C85... (with 328488... as key of course).
Also, in the following records, I see that the last encrypted block is used as IV.
Sorry for being lengthy, and thank you.

Thanks for replying @petrovr. Unfortunately I do not fully follow what was discussed in the referred issue. However something I didn't say before, sorry, is that the openssl version command on my system returns OpenSSL 1.1.0l 10 Sep 2019.
As mentioned by the #11266 opener, in the tls 1.2 rfc (5246), GenericBlockCipher is defined as:
       struct {
          opaque IV[SecurityParameters.record_iv_length];
          block-ciphered struct {
              opaque content[TLSCompressed.length];
              opaque MAC[SecurityParameters.mac_length];
              uint8 padding[GenericBlockCipher.padding_length];
              uint8 padding_length;
          };
      } GenericBlockCipher;

And in the rfc for tls 1.1 (4346):
       block-ciphered struct {
           opaque IV[CipherSpec.block_length];
           opaque content[TLSCompressed.length];
           opaque MAC[CipherSpec.hash_size];
           uint8 padding[GenericBlockCipher.padding_length];
           uint8 padding_length;
       } GenericBlockCipher;

Regarding the IV, rfc 5246 says:
      The Initialization Vector (IV) SHOULD be chosen at random, and
      MUST be unpredictable.  Note that in versions of TLS prior to 1.1,
      there was no IV field, and the last ciphertext block of the
      previous record (the "CBC residue") was used as the IV.  This was
      changed to prevent the attacks described in [CBCATT].  For block
      ciphers, the IV length is of length
      SecurityParameters.record_iv_length, which is equal to the
      SecurityParameters.block_size.

But rfc 4346:
   IV
      Unlike previous versions of SSL and TLS, TLS 1.1 uses an explicit
      IV in order to prevent the attacks described by [CBCATT].  We
      recommend the following equivalently strong procedures.  For
      clarity we use the following notation.

      IV
         The transmitted value of the IV field in the GenericBlockCipher
         structure.

      CBC residue
         The last ciphertext block of the previous record.

      mask
         The actual value that the cipher XORs with the plaintext prior
         to encryption of the first cipher block of the record.

      In prior versions of TLS, there was no IV field and the CBC
      residue and mask were one and the same.  See Sections 6.1,
      <IP_ADDRESS>, and 6.3, of [TLS1.0] for details of TLS 1.0 IV handling.

      One of the following two algorithms SHOULD be used to generate the
      per-record IV:

      (1) Generate a cryptographically strong random string R of length
          CipherSpec.block_length.  Place R in the IV field.  Set the
          mask to R.  Thus, the first cipher block will be encrypted as
          E(R XOR Data).

      (2) Generate a cryptographically strong random number R of length
          CipherSpec.block_length and prepend it to the plaintext prior
          to encryption.  In this case either:

          (a) The cipher may use a fixed mask such as zero.
          (b) The CBC residue from the previous record may be used as
              the mask.  This preserves maximum code compatibility with
              TLS 1.0 and SSL 3.  It also has the advantage that it does
              not require the ability to quickly reset the IV, which is
              known to be a problem on some systems.

          In either (2)(a) or (2)(b) the data (R || data) is fed into
          the encryption process.  The first cipher block (containing
          E(mask XOR R) is placed in the IV field.  The first block of
          content contains E(IV XOR data).

So I think what I'm seeing "over the wire" in the first 16 bytes of the record is the result of (2)(b), instead of the IV my aes-128-cbc cipher is using. I should actually call this IV "mask", following the rfc terminology for tls 1.1. And then the IV (not the mask) is obtained ciphering a random block R that is still prepended to the plaintext, to avoid the problems of tls 1.0...
Although all the latter is not defined in the rfc for tls 1.2. What rfc 5246 does say (and 4346 does not) is that the IV SHOULD be chosen at random, and MUST be unpredictable. Would it be correct to assume that these conditions, even if interpreted in a "per-record" basis, are met?


Osram Lightify Tunable Bulbs : Temperature cannot be controlled via Home app

Osram lightify bulbs ( tunable bulbs) do not respond to changes in color temperature settings made through the Home app (iOS) but do so through the Hassio frontend.
The bulbs support several colors throughout the white spectrum (color temperature) but default to white when controlled through homebridge.
I have used this setup directly through Homebridge (run directly on Rasbian) with the Osram-lightify platform and this feature worked as expected before. This does not work when using the homebridge addon (2.0.0) on the Hassio build (0.56.1). Unfortunately, I didnt test this with earlier versions, so I dont know if the new upgrades broke some support or if this issue was preexisting

Thank you for reporting your issue. Nevertheless, the add-on itself does not do anything with the logic of Homebridge or the Home Assistant plugin.
I would suggest you'd create an issue with the Home Assistant Homebridge plugin repository. Please be sure to attach some logs when creating that issue.


U10: The player can discard an item of his choice from his hand

I wasnt able to implement discarded locally, because to differentiate Drawn and Discarded state in ping data locally, the only way is to keep track of all discarded tiles locally. However, the record will get lost if I refresh the page.
To solve this problem, i make backend keep track of a boolean attribute "discarded" which evaluates to true if the tile is discarded. Also adjust code everywhere to match the change.
Might need to create a custom trash can icon
Let me know if you have any design suggestions (like where to place the trash icon or how it should look like)

I could do option 1 because its easier to implement
Do you mean place it "over" the bottom right corner of tile?
And no, sorry i couldnt find any icon without background, all the images are downloaded in square shape, all i can do is minimize the white background.

Yeah I mean over:) Okay so we'll do an trash icon ourselfes

Set tile state to Discarded is now done in backend, frontend just makes the call and receives tile information from ping


Fetch Limit not added to any query for DB2 Database

Description
I recently have been experiencing an issue the seems related to #22345 #21936 and #21940 but those mention some strange functions and a possible issue with the parser.
I think this issue is unrelated as my queries originate from 'main' in "View Table" or "View View"
I'm expecting to see LIMIT 200 added to all my queries, but that doesn't seem to be happening for any of them.
One of our Views is particularly large and without the LIMIT being added to the SQL, the query times out and is unable to return any results (self-canceled in the examples below but would be timed out if I let them run).
Settings:

DBeaver Version
Community Edition <IP_ADDRESS>312031432
Operating System
Microsoft Windows 10 Enterprise
Database and driver

IBM DB2 v<IP_ADDRESS>
Driver Name: DB2 for Linux/Unix/Windows driver
IBM Data Server Driver for JDBC and SQLJ V4.19

Steps to reproduce

Select "Read data in SQL console" or "View Table"/"View View"
Expect to see "LIMIT 200" added to the SQL Query in Query Manager"

Additional context
sql.log:
!ENTRY org.jkiss.dbeaver.model 1 22 2024-03-13 12:09:44.651
!MESSAGE SELECT x.* FROM ECW_V.CLAIM_HEADER x
!SUBENTRY 1 org.jkiss.dbeaver.model 4 -952 2024-03-13 12:11:15.324
!MESSAGE Processing was cancelled due to an interrupt.. SQLCODE=-952, SQLSTATE=57014, DRIVER=4.21.29

!ENTRY org.jkiss.dbeaver.model 1 24 2024-03-13 12:26:30.502
!MESSAGE SELECT x.* FROM ECW_V.CLAIM_HEADER x
!SUBENTRY 1 org.jkiss.dbeaver.model 4 -952 2024-03-13 12:28:36.453
!MESSAGE Processing was cancelled due to an interrupt.. SQLCODE=-952, SQLSTATE=57014, DRIVER=4.21.29

.log:
!ENTRY org.jkiss.dbeaver.model 2 0 2024-03-13 12:11:15.454
!MESSAGE SQL Warning (DataSource: ECWPROD; Code: 0; State: 00000): Statement processing was successful.. SQLCODE=0, SQLSTATE=00000, DRIVER=4.21.29

!ENTRY org.jkiss.dbeaver.model 4 0 2024-03-13 12:11:15.938
!MESSAGE SQL Error [57014]: Processing was cancelled due to an interrupt.. SQLCODE=-952, SQLSTATE=57014, DRIVER=4.21.29
!SUBENTRY 1 org.jkiss.dbeaver.model 4 0 2024-03-13 12:11:15.938
!MESSAGE Processing was cancelled due to an interrupt.. SQLCODE=-952, SQLSTATE=57014, DRIVER=4.21.29
!STACK 0
com.ibm.db2.jcc.am.SqlTimeoutException: Processing was cancelled due to an interrupt.. SQLCODE=-952, SQLSTATE=57014, DRIVER=4.21.29
	at com.ibm.db2.jcc.am.kd.a(kd.java:812)
	at com.ibm.db2.jcc.am.kd.a(kd.java:66)
	at com.ibm.db2.jcc.am.kd.a(kd.java:140)
	at com.ibm.db2.jcc.am.ResultSet.completeSqlca(ResultSet.java:4194)
	at com.ibm.db2.jcc.am.ResultSet.earlyCloseComplete(ResultSet.java:4176)
	at com.ibm.db2.jcc.t4.bb.a(bb.java:929)
	at com.ibm.db2.jcc.t4.bb.p(bb.java:894)
	at com.ibm.db2.jcc.t4.bb.k(bb.java:261)
	at com.ibm.db2.jcc.t4.bb.d(bb.java:55)
	at com.ibm.db2.jcc.t4.p.c(p.java:44)
	at com.ibm.db2.jcc.t4.vb.j(vb.java:157)
	at com.ibm.db2.jcc.am.sp.mb(sp.java:2234)
	at com.ibm.db2.jcc.am.sp.a(sp.java:3303)
	at com.ibm.db2.jcc.am.sp.e(sp.java:1111)
	at com.ibm.db2.jcc.am.sp.execute(sp.java:1090)
	at org.jkiss.dbeaver.model.impl.jdbc.exec.JDBCStatementImpl.execute(JDBCStatementImpl.java:330)
	at org.jkiss.dbeaver.model.impl.jdbc.exec.JDBCStatementImpl.executeStatement(JDBCStatementImpl.java:131)
	at org.jkiss.dbeaver.model.impl.jdbc.struct.JDBCTable.readData(JDBCTable.java:189)
	at org.jkiss.dbeaver.ui.controls.resultset.ResultSetJobDataRead.lambda$0(ResultSetJobDataRead.java:123)
	at org.jkiss.dbeaver.model.exec.DBExecUtils.tryExecuteRecover(DBExecUtils.java:190)
	at org.jkiss.dbeaver.ui.controls.resultset.ResultSetJobDataRead.run(ResultSetJobDataRead.java:121)
	at org.jkiss.dbeaver.ui.controls.resultset.ResultSetViewer$ResultSetDataPumpJob.run(ResultSetViewer.java:5151)
	at org.jkiss.dbeaver.model.runtime.AbstractJob.run(AbstractJob.java:105)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)

!ENTRY org.jkiss.dbeaver.model 4 0 2024-03-13 12:28:36.537
!MESSAGE SQL Error [57014]: Processing was cancelled due to an interrupt.. SQLCODE=-952, SQLSTATE=57014, DRIVER=4.21.29
!SUBENTRY 1 org.jkiss.dbeaver.model 4 0 2024-03-13 12:28:36.537
!MESSAGE Processing was cancelled due to an interrupt.. SQLCODE=-952, SQLSTATE=57014, DRIVER=4.21.29
!STACK 0
com.ibm.db2.jcc.am.SqlTimeoutException: Processing was cancelled due to an interrupt.. SQLCODE=-952, SQLSTATE=57014, DRIVER=4.21.29
	at com.ibm.db2.jcc.am.kd.a(kd.java:812)
	at com.ibm.db2.jcc.am.kd.a(kd.java:66)
	at com.ibm.db2.jcc.am.kd.a(kd.java:140)
	at com.ibm.db2.jcc.am.ResultSet.completeSqlca(ResultSet.java:4194)
	at com.ibm.db2.jcc.am.ResultSet.earlyCloseComplete(ResultSet.java:4176)
	at com.ibm.db2.jcc.t4.bb.a(bb.java:929)
	at com.ibm.db2.jcc.t4.bb.p(bb.java:894)
	at com.ibm.db2.jcc.t4.bb.k(bb.java:261)
	at com.ibm.db2.jcc.t4.bb.d(bb.java:55)
	at com.ibm.db2.jcc.t4.p.c(p.java:44)
	at com.ibm.db2.jcc.t4.vb.j(vb.java:157)
	at com.ibm.db2.jcc.am.sp.mb(sp.java:2234)
	at com.ibm.db2.jcc.am.sp.a(sp.java:3303)
	at com.ibm.db2.jcc.am.sp.e(sp.java:1111)
	at com.ibm.db2.jcc.am.sp.execute(sp.java:1090)
	at org.jkiss.dbeaver.model.impl.jdbc.exec.JDBCStatementImpl.execute(JDBCStatementImpl.java:330)
	at org.jkiss.dbeaver.model.impl.jdbc.exec.JDBCStatementImpl.executeStatement(JDBCStatementImpl.java:131)
	at org.jkiss.dbeaver.model.impl.jdbc.struct.JDBCTable.readData(JDBCTable.java:189)
	at org.jkiss.dbeaver.ui.controls.resultset.ResultSetJobDataRead.lambda$0(ResultSetJobDataRead.java:123)
	at org.jkiss.dbeaver.model.exec.DBExecUtils.tryExecuteRecover(DBExecUtils.java:190)
	at org.jkiss.dbeaver.ui.controls.resultset.ResultSetJobDataRead.run(ResultSetJobDataRead.java:121)
	at org.jkiss.dbeaver.ui.controls.resultset.ResultSetViewer$ResultSetDataPumpJob.run(ResultSetViewer.java:5151)
	at org.jkiss.dbeaver.model.runtime.AbstractJob.run(AbstractJob.java:105)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)

@ldsalomone Hello. I can't reproduce it. Can you show me driver properties for this connection?

Also would the result be the same if you disable these two options

It's been a while since no update here. If the issue is still actual please let me know and provide additional information.


add ability to configure and run preflight checks during install, upgrade, and uninstall operations

Description:

Adds a new Preflight interface and ability to configure a slice of Preflights to run during the actionClient install, upgrade, and uninstall operations.
Updates actionClient unit tests to test the preflight check implementation

Motivation:

https://github.com/operator-framework/operator-controller/issues/741

Codecov Report
Attention: Patch coverage is 74.00000% with 13 lines in your changes are missing coverage. Please review.

Project coverage is 85.31%. Comparing base (08ab7fb) to head (bd9d7f3).
Report is 14 commits behind head on main.

Files
Patch %
Lines

pkg/client/actionclient.go
74.00%
10 Missing and 3 partials :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #327      +/-   ##
==========================================
+ Coverage   85.06%   85.31%   +0.24%
==========================================
  Files          19       19
  Lines        1346     1559     +213
==========================================
+ Hits         1145     1330     +185
- Misses        125      149      +24
- Partials       76       80       +4

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Closing in favor of putting this directly in operator-controller/rukpak