[Draft] Linux Porting

This is just a draft PR. No guarantee of its availability.
I've finally gathered all the necessary parts for the Linux port, even though there's still a lot that's unfinished, needs to be added (other Unreal Engine versions), and needs testing (hook parts, AOB scanning, string compatibility, and compatibility with WIN32 code, etc.).
Bringing up some of the issues encountered might be beneficial for Linux porting. Even if this PR doesn't get merged, it could provide some insights for future attempts.
As for UE version support for linux, I don't think it'd be meaningful to support any game < 5.1 as that is the first version epic really starts to ship pre-built binaries for Linux.. Although devs can do it on their own, I'd not expect new games will use such an old engine versions..
Changes

Made some adjustments to CMake.

Disabled GUI, Input, and other features under Linux, using stderr for output, especially since Input has a high degree of coupling with Windows. It might be more reasonable to inject into UE's own input module.

Used LD_PRELOAD for loading and reading ELF segment information with dl_iterate_phdr under Linux.

Modified a lot of code that depends on MSVC features, such as member function pointers. However, there are some issues that couldn't be addressed, for example:

MemberVariableLayout_HeaderWrapper_ULocalPlayer.hpp:4:54: error: non-defining declaration of enumeration with a fixed underlying type is only permitted as a standalone declaration [-Welaborated-enum-base]

(clang wants all enum used by template get at least a definition for its size)
and the TypeAccessor being used twice in the DECLARE_VIRTUAL_TYPE_BASE macro, I'm unsure of the intention here.
Also, FUNCDNAME in GetDispatchMap(), which I think could be replaced with typeid(), but I'm unsure of the consequences.
-fms-extension is requred and to compile this and so clang is a must. (which is fine I guess, since UE also used clang for its linux build)

Modified GeneratedSourceFile template code to deal with a clang tuple bug https://github.com/llvm/llvm-project/issues/17042 .

Modified macros to suppress Windows-specific things like dllexport, which maybe should be controlled in CMake.

Modified a lot of string-related macros and functions, distinguishing between UE's character types and the C/C++ runtime character types using UEStringType and SystemStringType. This is mainly because wchar_t is 32-bit on Linux, and the standard library only provides compatibility for std::string (utf8)/wstring (unicode), while compatibility for u16string is missing.
I'm not satisfied with these changes, and my suggestion is maybe to introduce fmtlib, so u16string could be used globally, except for the Lua part, to minimize the parts that need encoding conversion.

Used DWARF information to generate memory and virtual function layouts and related headers under Linux.

Added AOBs and necessary xref/xcall features related to Linux binaries in patternsleuth because Linux uses non-pie binaries, leading to many addresses being 32-bit, and the Linux calling convention is different from Windows'.
Covered most resolvers, the rest I couldn't find in my binary files.
Since Linux's ELF doesn't load section information by default, used segment as a substitute.
Under Linux, UE's ELF doesn't contain any internal symbols, so this part of the scanning was removed.

Polyhook2 should be able to hook under Linux, but this hasn't been tested yet.

Fixed throw crash problem with UE.. The default std::throw used by UE has a CLANGC++ symbol and will be used by us to throw to glibc's c++ runtime.

I put all changes on this PR, although some of them are on other submoduels.

This is really cool but we are working on moving over to UE's platform system for generic platform strings, etc. So, a lot of your systemstring work will be covered by that.

We really appreciate the work you're doing for UE4SS and for Patternsleuth.
To keep you properly in the loop, you can find the UEPlatform stuff here. Additionally, you may benefit from taking a look at an existing attempt at a linux support which you can find here. Though as above you might need to have proper discussions with localcc/narknon regarding their specific plans with UEPlatform.

This is really cool but we are working on moving over to UE's platform system for generic platform strings, etc. So, a lot of your systemstring work will be covered by that.

that's actually nice to move away from std::string. I'm not happy with the current string handling either.

Yeah, we're trying to move the UE submodule to be as close to UE as possible so we can take advantage of their abstractions for different platforms and other optimizations more easily.  And it will make porting code for additional functionality much easier.

Made some progress and this porting is now working on the palworld linux server at least for NotifyOnNewObject, RegisterHook, StaticFindObject and FindFirstOf .
It is still missing some functions signature that's scanned in C++ code, but should not be a big deal to add them into it..
I'll take a look at the UEPlatform branch to see how to integrate current code into there.

That's awesome! UEPlatform doesn't have the generic platform string abstractions yet but I'll prioritize that. I previously just tried bringing over the string stuff so hopefully I can find those files and it's an easy port now

That's awesome! UEPlatform doesn't have the generic platform string abstractions yet but I'll prioritize that. I previously just tried bringing over the string stuff so hopefully I can find those files and it's an easy port now

So, I think for most code just need to unify the char to u16 instaed of wchar and make use of String:: like Format and maybe comparing etc in the case of containes..
But, I think even in this case, there are still some early output and use of FString where before the UE get located, what is the plan for those? use system string (utf8 on linux and u32 on win?)

My plan is to basically use the exact systems UE uses to cross compile. The UE platform branch is essentially porting all of that over for the generic platform abstractions and the windows platform. After that's fully ported, adding additional platforms will be as simple as porting over UE's headers for that platform as far as making types compatible. Of course that won't cover scanning and hooking though.

Note for reviewers in the future: At one point in time, several workflow files were removed in this PR (b03265f) so if this PR makes it to the point where it's going to be merged, please verify that they still exist and work before merging. Also be aware that we might run into githubs limitations on actions if we start running more CI (i.e. Linux build) for every commit.

I think I'll use a new branch when it get merged, because for example a lot of the changes to string handling here I think will be discarded or rework.

That's awesome! UEPlatform doesn't have the generic platform string abstractions yet but I'll prioritize that. I previously just tried bringing over the string stuff so hopefully I can find those files and it's an easy port now

I'd like to make sure that the plan here is to copy the source from UE FString/FStringFormatter and compile as it, instead of trying to locate FString::Format at the runtime?

That's awesome! UEPlatform doesn't have the generic platform string abstractions yet but I'll prioritize that. I previously just tried bringing over the string stuff so hopefully I can find those files and it's an easy port now

I'd like to make sure that the plan here is to copy the source from UE FString/FStringFormatter and compile as it, instead of trying to locate the address of FString::Format at the runtime? Also things like FFileHelper?

Trying to locate the address of any function is a last resort, or if it's trivial, e.g. if it's a virtual function and we have handy access to an instance.
I think it's pretty safe to say that we shouldn't be trying to locate the address of FString::Format.

I think multiple character types are still needed in every sense, with or without FString.
This is because of the following platform limitations:

UE will always use u16, however on Windows this is wchar_t, on Linux it's char16_t.
The wide character Windows API can only handle wchar_t and does not accept u16, even though they are essentially the same;
3.Linux can accept all kinds of u8;
wchat_t and char16_t can't be implicitly converted by the platform's compiler on Windows;
It's not possible to deal with UTF-8 directly through the Windows API (i.e., it's not possible to submit the correct string to the Windows api unless it is a binary stream)
All local files are essentially utf8
c++ std::exceptions require char
There are no u16-related library functions, such as isdigit, on the Linux platform, although they are not difficult to implement.
std::format does not support u16 formatting on Linux, so either use all ANSI characters, or use the wide character version on Windows and the u8 version on Linux.

Therefore, regardless of the future of FString, we need to make a distinction between the following three character types.

FileString -> always u8
UEString -> always u16, u16string on Linux, wstring on Windows, used by UE, header and etc..
SystemString -> Unicode encoded string supported by the platform API, wstring on Windows, using utf16, string on Linux, using utf8, used by API interface.
std::string is used for interaction with the c++ api and for interaction with Lua, using utf8.

And for different modules:

Obviously all files will use FileString, and I'll keep UEString as a compatibility backup to the original wstring interface:
All interactions with UE will use UEString.
All interactions with Lua use string.
All interactions with the GUI use UEString.
SDK Generators, JSON dumper uses UEString, but save as utf8
SDK uses UEString
Output uses SystemString for format parameters and interoperability with devices, and can accept any type of string as input.
Parser ... my opinion is to use UEString, because there will be more operations between the parser and the UE, so you only need to convert to/from the FileString once when it enters/leaves the parser, which is more efficient than converting the result of each parse.

@Yangff
Glad to see someone interested in linux port!
I tried to do some testing of this implementation but I got stucked. In case this is trivial to answer can you help me with it?
I'm already able to inject the program into the before launching a game (for example Palworld) using LD_PRELOAD=/path/to/libUE4SS.so /path/to/server/binary. The server boots up except that the program doesn't load any mod. It finds all the folders (root, game and mods) and even detects if there is a mods.txt or not. How did you managed to make the program load mods correctly?
My asumption is that I'm missing correct UE4SS-Signatures, because when I used them (different versions of windows) the program actually logged (after PS scan) a LUA scan attempt and then crashed with signal 11 (segmentation fault), and without the signatures this never happens, therefore, no mod is loaded.
Again, in case this is trivial to answer I'll appreciate any help on it, and in case its possible I can help with documentation of how to setup and load the program when this PR is ready to be merged (or before, if you want more testers to try it)
Thx for the amazing job done already!

@Yangff
Glad to see someone interested in linux port!
I tried to do some testing of this implementation but I got stucked. In case this is trivial to answer can you help me with it?
I'm already able to inject the program into the before launching a game (for example Palworld) using LD_PRELOAD=/path/to/libUE4SS.so /path/to/server/binary. The server boots up except that the program doesn't load any mod. It finds all the folders (root, game and mods) and even detects if there is a mods.txt or not. How did you managed to make the program load mods correctly?
My asumption is that I'm missing correct UE4SS-Signatures, because when I used them (different versions of windows) the program actually logged (after PS scan) a LUA scan attempt and then crashed with signal 11 (segmentation fault), and without the signatures this never happens, therefore, no mod is loaded.
Again, in case this is trivial to answer I'll appreciate any help on it, and in case its possible I can help with documentation of how to setup and load the program when this PR is ready to be merged (or before, if you want more testers to try it)
Thx for the amazing job done already!

Can you send your directory tree and UE4SS.log under the same folder as the .so?
In genreal, I'd suggset you start from this simple hello world to see if it works.
print(string.format("Hello world from Lua Mod!!!"))

One thing to notice is the scripts folder needs to be all lowercase in order to work.

Thx for the quick response!
The issue was with the "Scripts" folder, which I've created with capital case. Renaming to "scripts" did the trick, now the mods are loading correctly. (Thank you, Windows case insensitiveness)
Now that I can test this, do you want me to write a .md file with requirements (GLIBC >= 2.35, GLIBCXX >= 3.4.32, etc), how to upgrade stdc++ lib in case is needed and how to setup a script to run a game server with UE4SS injected?
This will bring more testers, and reveal any issue that is not already considered for linux implementation

Regarding file & path case sensitivity problems, should we maybe implement a case-insensitive function ?
Instead of just checking if the path exists, we'd have to iterate the path instead and case-insensitively compare each directory (and file when appropriate) in the path to the path (and/or file) we're looking for, and we'd need a counter that increases with every match so that we can generate a runtime error for the user to solve because if there are multiple directories with the same name but different casings then we can't possibly tell which one should be used.
Upsides:

Most of the time, this will just work for everyone, no need to think about the casing, and this also solves the enabled.txt/Enabled.txt problem.

Downsides:

It should be slower but do we do any path or file lookups in any hotpaths, does it matter if it's a bit slower ?
We'd effectively be reserving any directories with the names that we look for on Linux.
Users wouldn't be able to have both a "scripts" and a "Scripts" directory or a "mods" and a "Mods" directory or whatever other directories that happens to be in the path for any reason.

@Sasurtio Regarding the libc version, I think maybe the version requirement could be lowered to 2.17, but I'm not so sure that because ue4ss itself uses a lot of c++17/20 features, a lower version of glibc might not work correctly?
I don't recommend upgrading the system's glibc, if users are using a Linux distro with rolling upgrades like archlinux they should be well aware of such issues, and conversely, upgrading glibc on a distro with a fixed glibc version such as Ubuntu can lead to even more problems.
It might make more sense (as many Linux palworld servers currently do) to use docker/podman + mount an external mods directory to load it.
@UE4SS In the recent commit I tried to detect both scripts and Scripts, I guess arbitrary case combinations don't really make sense?
A similar problem occurs with lua looking for import files, I have it looking for scripts and Scripts.
I can only hope no one has written LogicMods as logicMods or something ......
I haven't changed the logic for enabled.txt yet, but I could similarly add a check for Enabled.txt.
As for the Mods directory itself, currently it will only check the Mods directory, and checking all mods directories might result in the user having two mods.txts in their respective directories? Seems like it would require changes to the loading logic and result in mods.txt being loaded in an uncontrollable order?
Considering that the Mods dir should be distributed with UE4SS, this doesn't seem to be a serious problem?

@Yangff I tried with default libc in Ubuntu 20.04 (glibc 2.30) and it crashed with seg fault because of it. Also the process to recompile a higher glibc version and then patch the program to use it was not possible (I tried with patchelf) because the ".interp" property is missing in the loader, meaning it is trying to use a fixed interpreter.
Regarding libstdc++ there is no issue going up, but the glibc is another story for static libc distros.

recompile a higher glibc version

You can install glibc to another location and execute its ld.so directly to bypass the interp, or modify the .interp of PalServer via patchelf.
Regarding lowering the version dependency of glibc, I meant to do it by linking https://github.com/wheybags/glibc_version_header at compile time, but I'm not sure if that would cause any other problems.

As for the Mods directory itself, currently it will only check the Mods directory, and checking all mods directories might result in the user having two mods.txts in their respective directories? Seems like it would require changes to the loading logic and result in mods.txt being loaded in an uncontrollable order? Considering that the Mods dir should be distributed with UE4SS, this doesn't seem to be a serious problem?

Avoiding the described problem should be trivial, either error if both Mods and mods exist or prioritize Mods, meaning ignore mods if both exist, but regardless, it shouldn't be a problem for the reason you stated.
The reason I prefer a more standardized way to deal with directories is so that devs don't have to consider both Window & Linux for every single file and path, and for that same reason I think we should probably change to std::filesystem::path everywhere.
We could have our own path_exists function for example, and our File system could implicitly do the proper checks as well.
For example, instead of this:
current_paths.append(std::format(";{}" LUA_DIRSEP "{}" LUA_DIRSEP "scripts" LUA_DIRSEP "?.lua", to_string(m_program.get_mods_directory()).c_str(), to_string(get_name())));
current_paths.append(std::format(";{}" LUA_DIRSEP "{}" LUA_DIRSEP "Scripts" LUA_DIRSEP "?.lua", to_string(m_program.get_mods_directory()).c_str(), to_string(get_name())));

We could have something like this:
// Assuming 'get_mods_directory()' returns std::filesystem::path.
auto scripts_path = File::get_path_if_exists(m_program.get_mods_directory() / get_name() / "Scripts");
current_paths.append(scripts_path.string().c_str());

The new File::get_path_if_exists function would check for both Scripts and scripts return the correct one based on what exists, and it would probably throw or return an empty path or something if the path doesn't exist.
If you think doing a fully case-insensitive implementation isn't worth it, the abstraction itself can still be implemented with just the first character being changed, and both dirs being found doesn't have to be an error, it could just return whichever it finds first or whatever other priority you think is best.
If you were to open a file, we could also avoid having to explicitly call get_path_if_exists, and just do it inside File::open instead.

What's the status of the GUI ? Why is there a need to use ImTui instead of ImGui ?
We're already using GLFW for Windows, is there a reason we can't use it for Linux ?
Also, would it be feasible to use GLFW for input handling instead of our own custom solution ? That would be preferable since our own solution is not cross-platform compatible.
I'm not sure how input handling is done in GLFW, if you can make use of it even while not rendering anything or if it's intrinsically linked to the UI.

I think multiple character types are still needed in every sense, with or without FString.

@Yangff I will defer to you on how best to handle this, as it seems like you have a much better understanding of what would be required.  The one thing I'd like to be kept in mind is that we want to move the UE module to be as close to UE code as possible, so where UE has a system or abstraction or optimization for different platforms, we'd like to use their systems if it will also work for our scenario.
My end goal is to have coding plugins for UE4SS as similar as possible to coding for UE while allowing them to be pseudo-universal across as many games/engine versions as possible.  And I think for maintainability's sake, using as much of the UE code base as possible will make it much easier to update alongside UE.  There are a lot more features in UE's Core engine source to move over and use.

Have you not been using clang-format

I've not formated the code yet.

What's the status of the GUI ?

I do not have a desktop env for linux right now. So I'm using TUI to allow access to the UI interface without the need of having a desktop or so. It'd be more accessible when using ssh compared to X11-forwarding?

3. use GLFW for input handling

The current input handle used by UE4SS is to get keyboard status directly from the Windows API. It's because for the gaming you want to get the key while the UE4SS Window is not on focus?
On linux it is kinda of tricky, you can interact with X say using X Record to get something like the windows api.. but I'm not sure how avialable is that if the wayland is used for the desktop which might have different result depends on what window management they're using.. I think the same situation for whatever underlaying API used by GLFW..
But for current linux user that might be fine, as there will be no gaming window for dedicated server anyway.. but on Windows that will prevent the player from accessing the shotcuts when playing I think because GLFW will only subscribe to its own message queue.

There are a lot more features in UE's Core engine source to move over and use.

yeah, and I think switching to UEPlatform means at least to clean up all that code marked as using SystemString so far.. they're marked as SystemString mostly beucase they're using stl for string manupilation... like all the codes involve in the parser and header dumping..
Once FString::Format is ready and compatible with other STL container currently in use (which might need more work, or also replace them with UE's) this should not be too hard I think.

I do not have a desktop env for linux right now. So I'm using TUI to allow access to the UI interface without the need of having a desktop or so. It'd be more accessible when using ssh compared to X11-forwarding?

ImTui instead of ImGui

It's still using the ImGui for rendering, but just using text +necurs as its input/output deivice. Mostly just need to have a scale factor becuause in TUI everything take space line by line instead of pixel by pixel. Unfortunately it has to modify part of imgui so far so it's using its own imgui.

I didn't consider non-desktop environments but this sounds very cool.
As long as we don't have to duplicate any UI elements or specialize our code specifically for ImTui, I think it's fine if you intend to keep it in.

I would leave enough abstraction space for the Input/Handler so that you can use any InputSource to add event sources to it.
You could have whatever InputSource running behind the scene and either pumping events into its queue or polling when the handler requested, and the input handler takes those inputs and act accordingly.
The benefit for that is, even if we're not using the TUI, you can still receive keyboard events from ssh to allow some functionality like reload mods or interact with lua mods.

This sounds fine.

Btw, I'm not sure https://github.com/UE4SS-RE/RE-UE4SS/blob/main/deps/first/Input/include/Input/Handler.hpp#L38 why the m_key_sets is defined as a vector of a map of a vector..
a map should be enough for mapping keys to the array of bind events on that keys with different modifier keys requirement?
It seems that in registration function it will create a new map for each key and push it to the vector which.. I'm not seeing any usage for it?

Btw, I'm not sure https://github.com/UE4SS-RE/RE-UE4SS/blob/main/deps/first/Input/include/Input/Handler.hpp#L38 why the m_key_sets is defined as a vector of a map of a vector.. a map (map<key,vector <key data >> for example ) should be enough for mapping keys to the array of bind events on that keys with different modifier keys requirement? It seems that in registration function it will create a new map for each key and push it to the vector which.. I'm not seeing any usage for it?

The input system was written years ago when I first started learning C++ and I no longer remember the details of it so I can't answer your question unfortunately, and the quality of that code is probably subpar in many ways.
It does seem to me that an unordered_map should be sufficient, but you'd have to test this yourself to make sure.

The Handler's interface to Lua remains unchanged, but its Handler now only contains logic of processing of generic input events  of key+modifier and dispatch them to the registered functions...
I'm trying my best to keep the handling logic on Windows unchanged but since the code are re-structured and simplified there might be some inconsistent..

pushed code that's formatted. Basically everything I wanted to implement under this PR is here. The CI scripts can currently be used to build the Linux version, and it's not restored to the way it was.
I haven't tested cppmod but I'm enabling it anyway... I think the linux abi problem will become even severe for it to be widely used compared to windows..
A tested GLFW implementation is added, But maybe there's something wrong with my wslg, it's rendering a window size that doesn't match the actual size.
Tested built-in keybinding and etc.. I think the basic functionality should be there.
There are some limitations regarding input, first of all not all key combinations are available, Ctrl+something only has A-Z and some function keys, while something like Ctrl+C is used by the terminal for other roles. enter and Ctrl+J will use the same codepoint etc.
Shift is recognized by distinghishing from a and A, it might have different result on different keyboard and I'm only talking US keyboard layout here..
Also TUI is sending user input as characters, but not keyup, so press will be is doing multiple keydowns, where I can only limit the frequency of responses a bit.
I'd be more concern if it break anything on the Windows side given the amount of code changed.

I will do a review of the code in a while, my above comment is not likely to be the last one before this gets merged, please be patient as this is a large PR.

@narknon @localcc @trumank @Buckminsterfullerene02 Because this PR is very pervasive, it will need significant testing even on Windows to make sure nothing has been broken, and I think we all need to help with this and perhaps even do an experimental release at some point so that the community can test it.
If code review passes, community testing can probably be done after merging to make the merge happen sooner which should simplify the workflow for everyone.

I'm not all too familiar with linux things, but I've noticed there's quite a few steam deck users who want to use UE4SS for mods. Currently the "workaround" is to rename the dwmapi.dll to msacm32.dll. And after that users don't seem to come back to complain it's not working. So I'm thinking that we could also have our build scripts make a seperate release for steam deck users (non dev as obviously people aren't going to be using steam deck to make mods) with that dll renamed (though Idk if raw dogging the rename of a dll can cause issues).

If we do an official release, can we not use proxy_generator to generate msacm32.dll instead of renaming ?

I'm not all too familiar with linux things, but I've noticed there's quite a few steam deck users who want to use UE4SS for mods. Currently the "workaround" is to rename the dwmapi.dll to msacm32.dll. And after that users don't seem to come back to complain it's not working. So I'm thinking that we could also have our build scripts make a seperate release for steam deck users (non dev as obviously people aren't going to be using steam deck to make mods) with that dll renamed (though Idk if raw dogging the rename of a dll can cause issues).

If we do an official release, can we not use proxy_generator to generate msacm32.dll instead of renaming ?

Oh so that's what proxy gen is? lol, I just never knew before

I'm not all too familiar with linux things, but I've noticed there's quite a few steam deck users who want to use UE4SS for mods. Currently the "workaround" is to rename the dwmapi.dll to msacm32.dll. And after that users don't seem to come back to complain it's not working. So I'm thinking that we could also have our build scripts make a seperate release for steam deck users (non dev as obviously people aren't going to be using steam deck to make mods) with that dll renamed (though Idk if raw dogging the rename of a dll can cause issues).

If we do an official release, can we not use proxy_generator to generate msacm32.dll instead of renaming ?

Oh so that's what proxy gen is? lol, I just never knew before

Yeah the syntax is this: proxy_generator.exe <input_dll_name> <output_path> where <input_dll_name> for example could be C:\Windows\System32\dwmapi.dll.
I did a cursory search and couldn't find any documentation for proxy_generator anywhere unfortunately but it seems the usage should be obvious.

@Buckminsterfullerene02 The proxy_generator.exe file will generate three files, and then those files need to be compiled into the dll.
So it's not as simple as I thought it was but it's definitely preferred over renaming the dll.

The easiest way to generate the proxy dll is probably to set UE4SS_PROXY_PATH to CMake and then build.
This can be done by passing -DUE4SS_PROXY_PATH=C:\Windows\System32\msacm32.dll when running CMake.

The HAS_GLFW definition

I'm not sure if we want to keep the GLFW backend for the linux build by even use it as a default setup.. so I'm just leaving there and look for some comment.
Because imtui reuqires a modified imgui, I can only build either one with UE4SS, not both.
(Another idea I had was to compile them each into a .so and load them on demand though)
The advantage of GLFW is that it's supposed to be more compatible with today's code, basically you don't need to modify any existing ui code, whereas TUI has to be tweaked in all the places where a fixed length or width is indicated.
In local GLFW doesn't seem to recognize the actual width of the window correctly and can't get it fixed unless I directly modify the result of getting the windows size, I suspect it's a wslg issue but I don't have an actual Linux desktop to test it on right now.

If imtui is going to be kept, then libncurses5-dev must be added as a requirement for the Linux build.

If imtui is going to be kept, then libncurses5-dev must be added as a requirement for the Linux build.

on ubuntu 22.04 I'm using libncursesw6-dev, w for its wide char version because the icon from nerd font using non-ansi characters.
Beside it requires a higher version of libstdc++ for the c++20 support, I think maybe something comes with gcc-13?
Some people have responded that they can't compile it within the Ubuntu 22.04 docker image, even though GitHub action is based on 22.04 as well.
And on 22.04 they need to install https://ubuntu.pkgs.org/22.04/ubuntu-main-amd64/libstdc++6_12-20220319-1ubuntu1_amd64.deb.html

If imtui is going to be kept, then libncurses5-dev must be added as a requirement for the Linux build.

on ubuntu 22.04 I'm using libncursesw6-dev, w for its wide char version because the icon from nerd font using non-ansi characters. This should come by default with ubuntu 22.04.
Beside it requires a higher version of libstdc++ for the c++20 support, I think maybe something comes with gcc-13? Some people have responded that they can't compile it within the Ubuntu 22.04 docker image, even though GitHub action is based on 22.04 as well. And on 22.04 they need to install https://ubuntu.pkgs.org/22.04/ubuntu-main-amd64/libstdc++6_12-20220319-1ubuntu1_amd64.deb.html

I'm using Mint 21.2 so I'm stuck with GCC 11 for now, so I'm building fully with Clang 17 instead.
I'm just trying to experience the window size problem that you're having.
I've gotten it to build, now I just need to set up SteamCMD so that I can install the Palworld server because patternsleuth times out on my test game.

so I'm building fully with Clang 17 instead.

yeah. gcc will never work as far as I tried. there are more inconsistent with msvc styled grammar I think. clang should work fine.

To build with clang, I had to add the following includes to the following files.
You have maintainer write-access turned off for this PR so I can't push these changes myself.
In UE4SS:
Function/include/Function/Function.hpp: <utility>
Input/include/Input/KeyDef.hpp: <initializer_list>
LuaMadeSimple/include/LuaMadeSimple/LuaMadeSimple.hpp: <type_traits>
include/GUI/GUI.hpp: <stop_token>
In Unreal:
src/VirtualFunctionHelper.cpp: <string>

maintainer write-access turned off

I think it's on? is there any other settings?

but I've added them

I should clarify as well what I meant by "fully with clang".
I had to add -stdlib=libc++ to the compile options, this is so that it would use Clangs lib implementation since my libstdc++ is too old.

Yes, the standard library that comes with older versions of gcc does not support the newer c++ standard. The library itself should be fine as long as it's gcc 12 I think?

Yes, the standard library that comes with older versions of gcc does not support the newer c++ standard. The library itself should be fine as long as it's gcc 12 I think?

Yes it was mostly just to let others know because they, like me, might also be running an old version of GCC.

In local GLFW doesn't seem to recognize the actual width of the window correctly and can't get it fixed unless I directly modify the result of getting the windows size, I suspect it's a wslg issue but I don't have an actual Linux desktop to test it on right now.

I can't seem to reproduce the window size problem, but I do have a desktop environment so that might have something to do with it.

It appears that exceptions are not caught properly in the UE4SSProgram constructor.
The function that looks up the settings file can throw, and if it does, there's nothing that tells the user what happened.

can't seem to reproduce the window size problem

then it must caused by wslg

exceptions are not caught properly in the UE4SSProgram

It's most likely because you're using libc++, clang's exceotion is not compatible with glibc's in abi.
unreal uses a static linked libc++ from clang, and they exported taht symbol globally and contaiminated to ue4ss when build with gcc's stdc++.
because of that, in __libc_start_main inside EntryLinux.cpp, I've some code to replace all __cxa_throw to gcc's.
But because you're using libc++, it's likely that you will need to remove those code.

exceptions are not caught properly in the UE4SSProgram

It's most likely because you're using libc++, clang's exceotion is not compatible with glibc's in abi.

No this isn't the problem, exceptions do work.
The problem is actually a preexisting problem that was introduced a long time ago here: https://github.com/UE4SS-RE/RE-UE4SS/commit/150f1543#diff-d09b4d74acaafd685651cb6c059177fe0e29aa7f5e570f04509c17470d6e9975
We used to create UE4SSProgram and immediately check for exceptions because the constructor can throw, but now, we create it and don't check for exceptions until we're in a new thread, which doesn't work.
Of course, this bad code was then understandably copied into the Linux entry file.
I think we should leave this bug alone for now and fix it after this is merged
I've made an issue for it here: #408

OutputDevice on Linux to one that uses stderr

I think the output device is using stderr, unless the tui is open which will suppress all Stderr output until it's closed. (Not the case for GUI).
There might be some code using printf for output though.

maybe workaround like this?
template <int32_t optional_arg, typename... FmtArgs>
auto send(std::string_view content, FmtArgs&&... fmt_args) -> void {
    auto args_tuple = std::make_tuple(to_system(std::forward<FmtArgs>(fmt_args))...);
    auto format_and_print = [&content](auto&&... args) {
        auto formated = std::vformat(content, std::make_format_args(args...));
        printf("%s\n", formated.c_str());
    };
    std::apply(format_and_print, args_tuple);
}

if make_format_args only accept lvalue we will have to create a "holder" inside our function..

maybe workaround like this?
template <int32_t optional_arg, typename... FmtArgs>
auto send(std::string_view content, FmtArgs&&... fmt_args) -> void {
    auto format_and_print = [&content](auto&&... args) {
        auto formated = std::vformat(content, std::make_format_args(args...));
        printf("%s\n", formated.c_str());
    };
    format_and_print(to_system(std::forward<FmtArgs>(fmt_args))...);
}

if make_format_args only accept lvalue we will have to create a "holder" inside our function... and not forwading the arguments.. but it feels weird anyway.. because even make_format_args(std::string("xxx")) become illegal.. I'm trying to avoid creating a formatter for wstring in string or string in wstring but hmm.. idk maybe that would be better in this case? (or buggy https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112607)

I think for now, your workaround is fine.
We can work towards something more ideal later if needed.
Right now we should just get this thing working.
I would like to change the factoring a bit though to something like this, don't implement this, just let me know if you agree, and I will implement it myself:
// Top of Output.hpp
constexpr auto apply_formatting(auto&& content, auto&&... args)
{
    // Must switch on the 'content' type at compile-time, otherwise std::string formatting
    // won't work on Windows since RC_STD_MAKE_FORMAT_ARGS is set to std::wstring.
    if constexpr (not_std_string_like_t<std::decay_t<decltype(content)>>::value)
    {
        return std::vformat(content, std::make_wformat_args(args...));
    }
    else
    {
        return std::vformat(content, std::make_format_args(args...));
    }
}

template <int32_t optional_arg, typename... FmtArgs>
auto send(std::string_view content, FmtArgs&&... fmt_args) -> void {
    auto formated = apply_formatting(content, to_system(std::forward<FmtArgs>(fmt_args))...);
    printf("%s\n", formated.c_str());
}

RC_STD_MAKE_FORMAT_ARGS is set to std::wstring

looks good to me. just out of curious... I think SystemString will always be wstring on windows and using RC_STD_MAKE_FORMAT_ARGS here should be fine?

RC_STD_MAKE_FORMAT_ARGS is set to std::wstring

looks good to me. just out of curious... I think SystemString will always be wstring on windows and using RC_STD_MAKE_FORMAT_ARGS here should be fine?

It's a problem for the fmt function in Helpers/Format.hpp, which appears to be used exclusively for formatting strings for std::runtime_error, which always uses std::string.
Effectively, this is a convenience function, and it keeps the code cleaner by requiring less explicit type conversions.

Regarding the comment about settings in 5a5a2e8cdd53ae7f83323ad07dae621c50edfcd0:
The way we usually deal with settings in other modules is to retrieve the value in UE4SSProgram and then pass it to whatever module needs it.
That's how we do it with the Unreal module.

59f68c4 is trying to fix https://github.com/UE4SS-RE/RE-UE4SS/issues/397 but I'm honest if it's right to use localobject on fstring here. feel free to revert it if there is other plan on it.

59f68c4 is trying to fix #397 but I'm honest not usre if it's right to use localobject on fstring here. feel free to revert it if there is other plan on it.
My feeling is it's will prevent parameter modifying on registerhook, so maybe this is not the solution here.

This fix doesn't belong in this PR, but nevertheless here are my thoughts:
A LocalObject is supposed to be transferred to Lua, meaning that Lua should own it completely and is therefore responsible for freeing any memory related to the object, including calling C++ destructors if needed.
A RemoteObject is owned by anything other than Lua and therefore Lua isn't responsible for freeing any memory or calling any C++ destructors.
For FString, I believe LocalObject makes sense, though you should verify that your fix actually works.
For the future, please make a new PR if the change is unrelated.

no worry I will revert this commit, I was just trying to trigger the ci for someone who want to test it.

add an issue.. the UE4SS maybe destruct before the game..
but if I delay the destruct the ui thread maybe running when the opengl has gone..
need to figure out how to destruct them in order..
libUE4SS.so!_ZN2RC3GUI23FLiveViewDeleteListener20NotifyUObjectDeletedEPKNS_6Unreal11UObjectBaseEi(+0x193)
PalServer-Linux-Test!FUObjectArray::RemoveObjectFromDeleteListeners(UObjectBase*) [C:/works/Pal-UE-EngineSource/Engine/Source/./Runtime/CoreUObject/Private/UObject/UObjectArray.cpp:275]
PalServer-Linux-Test!UObject::ConditionalFinishDestroy() [C:/works/Pal-UE-EngineSource/Engine/Source/./Runtime/CoreUObject/Private/UObject/Obj.cpp:1054]
PalServer-Linux-Test!IncrementalPurgeGarbage(bool, double) [C:/works/Pal-UE-EngineSource/Engine/Source/./Runtime/CoreUObject/Private/UObject/GarbageCollection.cpp:1767]
PalServer-Linux-Test!StaticExit() [C:/works/Pal-UE-EngineSource/Engine/Source/./Runtime/CoreUObject/Private/UObject/Obj.cpp:4893]
PalServer-Linux-Test!TBaseStaticDelegateInstance<void (), FDefaultDelegateUserPolicy>::ExecuteIfSafe() const [C:/works/Pal-UE-EngineSource/Engine/Source/Runtime/Core/Public/Delegates/DelegateInstancesImpl.h:737]
PalServer-Linux-Test!TMulticastDelegate<void (), FDefaultDelegateUserPolicy>::Broadcast() const [C:/works/Pal-UE-EngineSource/Engine/Source/Runtime/Core/Public/Delegates/DelegateSignatureImpl.inl:975]
PalServer-Linux-Test!FEngineLoop::AppPreExit() [C:/works/Pal-UE-EngineSource/Engine/Source/./Runtime/Launch/Private/LaunchEngineLoop.cpp:6286]
PalServer-Linux-Test!FEngineLoop::Exit() [C:/works/Pal-UE-EngineSource/Engine/Source/./Runtime/Launch/Private/LaunchEngineLoop.cpp:4592]
PalServer-Linux-Test!GuardedMain(char16_t const*) [C:/works/Pal-UE-EngineSource/Engine/Source/./Runtime/Launch/Private/Launch.cpp:217]
PalServer-Linux-Test!CommonUnixMain(int, char**, int (*)(char16_t const*), void (*)()) [C:/works/Pal-UE-EngineSource/Engine/Source/Runtime/Unix/UnixCommonStartup/Private/UnixCommonStartup.cpp:269]
libc.so.6!UnknownFunction(0x25ccf)
libc.so.6!__libc_start_main(+0x89)
libUE4SS.so!__libc_start_main(+0x264)
PalServer-Linux-Test!_start()

@Yangff How does github.com/Re-UE4SS/UEPseudo/pull/87 affect this ? It introduces UEs GenericPlatformString which might be a nicer solution to needing all these different string types and macros.

@Yangff How does github.com/Re-UE4SS/UEPseudo/pull/87 affect this ? It introduces UEs GenericPlatformString which might be a nicer solution to needing all these different string types and macros.

Put Linux port aside I'm think there are following problems that may prevent us from using platform string as the major string type.

we will still at least make sure the uestring can work with stl. This maybe trivial but certainly need some development.
ue's allocator won't be available until we finished binary scanning, but the strings need to be ready to initialize the containers that are using strings as the key.
Similarly, if we want to avoid 1 by using ue's container, we also need to delay the initiation until the ue's allocator is available.

Specifically, for Linux we are going to facing the problem when interacting with the stl, because they only accept either utf8 or utf32, again unless we get rid of all use of all in current codebase.. largely doing the formatting and the exception message.. as I have not seen the implement of any being added to that pr..

I didn't mean we should use FString, I should've been more clear, sorry.
I think we should have full support for FString eventually, but not for internal usage, only to make it easier for C++ mods and to simplify interacting with strings that come from UE itself.
I've also realized that none of this has much to do with the GenericPlatformString that's being added soon.
But regardless, here are some thoughts:
I think we should use UEs compile-time platform-specific string handling system, like using TEXT instead of STR, and the TCHAR type.
UE uses that type and those macros for every platform for most things as far as I know and I don't see why we shouldn't also do that, or at least that should be the goal.
Obviously we'll still have to use something separate for Lua since that always uses the char type, but that's fine.
I believe fmtlib has support for u16.
I originally didn't want to use fmtlib because I prefer stl whenever possible (I hate dependencies), but if it offers something that we can't get otherwise then we should probably use it.
Because of the amount of effort it might take to do all this though, I'm completely fine with waiting until after this PR is finished and merged before further work is done on simplifying strings.

Thanks for the clarification.

like using TEXT instead of STR, and the TCHAR type.
This should be easy, just convert the parts that currently use UEString to use the UE defiend macro.
Also it might be possible to remove some of the custom string handling functions for u16.

fmtlib has support for u16

Yes, fmtlib does support u16. Since the main consumers of string formatting are: std::exception (which only accepts char*) and headers generated by UE4SS, both of which seem to use UTF-8 encoding, using UTF-8 for string encoding ensures that all encoding conversions are directed towards UTF-8. (Since the need to eventually they will have to be transcoded to utf-8).
Config files  are  still using system string. for parser switching to uestring I don't think there is much problem if we use fmtlib.
However, even fmtlib supports u16, it intentionally does not support mixed u8/u16 usage, so explicit conversion is still necessary.
The other reason I use char here is that it can be handled by the same type std::string on all platforms (assume UTF-8), which is a bit better handled than the u16string vs wstring (you need to always be careful of two distinct string types), but I don't think that's a huge problem.

Status of xmake rebasing
I can't fix (help request):

When building with clang xmake f -v -D --UI=n --toolchain=clang -m "Game__Shipping__Linux" -y I get the following error that I don't understand (proxy gen has been disabled in linux, no reason it even tries to build rc)

error: @programdir/core/main.lua:329: @programdir/actions/build/main.lua:148: @programdir/modules/async/runjobs.lua:338: ...amdir/core/sandbox/modules/import/core/tool/compiler.lua:37: cannot get program for rc
stack traceback:
    [C]: in function 'error'
    [@programdir/core/base/os.lua:957]:
    [...amdir/core/sandbox/modules/import/core/tool/compiler.lua:37]: in function 'load'
    [@programdir/rules/rust/build/target.lua:42]: in function 'build_sourcefiles'
    [@programdir/rules/rust/build/target.lua:83]:
    [@programdir/actions/build/build.lua:52]: in function 'jobfunc'
    [@programdir/modules/async/runjobs.lua:256]:

stack traceback:
        [C]: in function 'error'
        @programdir/core/base/os.lua:957: in function 'base/os.raiselevel'
        (...tail calls...)
        @programdir/core/main.lua:329: in upvalue 'cotask'
        @programdir/core/base/scheduler.lua:406: in function <@programdir/core/base/scheduler.lua:399>

config is as follow
configure
{
    ndk_stdcxx = true
    profilerFlavor = Tracy
    buildir = Intermediates
    host = linux
    ue4ssIsBeta = true
    TUI = false
    plat = linux
    ue4ssBetaIsStarted = true
    arch = x86_64
    ccache = true
    GUI = false
    UI = false
    kind = static
    scriptsRoot = /home/yangff/Project/RE-UE4SS/tools/xmakescripts
    ue4ssRoot = /home/yangff/Project/RE-UE4SS
    mode = Game__Shipping__Linux
    toolchain = clang
    Input = false
}

xmake does not respect the set_defaultmode("Game__Shipping__Linux") and keep using release or debug unless specify them with xmake f

Workaround:
maybe a xmake bug that it is not passing zydis location to cmake correctly, or it failed to meet the cmake find_package, anyway I have to add the follow:
        if is_plat("linux") then
            table.insert(configs, "-DZYDIS_INCLUDE_DIR=" .. package:dep("zydis"):installdir("include"))
            table.insert(configs, "-DZYCORE_INCLUDE_DIR=" .. package:dep("zycore"):installdir("include"))
            table.insert(configs, "-DZYCORE_LIBRARY=" .. package:dep("zycore"):installdir("libzycore.a"))
            table.insert(configs, "-DZYDIS_LIBRARY=" .. package:dep("zydis"):installdir("libzydis.a"))
        end

The repo's base xmake.lua has set_allowedmodes(get_compilation_modes()) which is defined in /tools/xmakescripts/build_configs.lua.
Perhaps the set_defaultmode() has no effect if you try and set it to a mode that isn't explicitly allowed by set_allowedmodes()?

The repo's base xmake.lua has set_allowedmodes(get_compilation_modes()) which is defined in /tools/xmakescripts/build_configs.lua.
Perhaps the set_defaultmode() has no effect if you try and set it to a mode that isn't explicitly allowed by set_allowedmodes()?

Never mind... I see that you added ["Linux"] to the platforms table so this shouldn't be an issue. I'll try and investigate further

Do you have any documentation about the prereqs required to make this branch compile?
I tried on WSL Ubuntu 20.04 with xmake, clang, rust, etc installed, but still haven't been able to compile or configure?
bitonality@DESKTOP-STUDIO:~/RE-UE4SS$ xmake f --mode="Game__Shipping__Linux" -v
checking for platform ... linux
checking for architecture ... x86_64
checking for gcc ... /usr/bin/gcc
checking for zig ... no
checking for zig ... no
checking for nim ... no
checking for nim ... no
checking for GUI ... no
checking for unzip ... /usr/bin/unzip
checking for git ... /usr/bin/git
checking for gzip ... /usr/bin/gzip
checking for tar ... /usr/bin/tar
/usr/bin/git rev-parse HEAD
checking for xmake::ImGuiTextEdit ... no
checking for pacman ... no
checking for emerge ... no
checking for brew ... no
checking for pkg-config ... no
checking for pkgconf ... no
checking for clang ... /usr/bin/clang
checking for the c compiler (cc) ... clang
checking for /usr/bin/clang ... ok
checking for flags (-fPIC) ... ok
checking for flags (-fdiagnostics-color=always) ... ok
checking for clang++ ... /usr/bin/clang++
checking for the linker (ld) ... clang++
checking for /usr/bin/clang++ ... ok
checking for flags (-fPIC) ... ok
/usr/bin/ld: cannot find -lImGuiTextEdit
clang: error: linker command failed with exit code 1 (use -v to see invocation)
> checking for c links(ImGuiTextEdit)
> checking for c snippet(find_package/ImGuiTextEdit)
checking for ImGuiTextEdit ... no
checking for cmake ... no
checking for cmake ... /snap/bin/cmake
checking for xmake::zycore ... no
/usr/bin/ld: cannot find -lzycore
clang: error: linker command failed with exit code 1 (use -v to see invocation)
> checking for c links(zycore)
> checking for c snippet(find_package/zycore)
checking for zycore ... no
checking for xmake::zydis ... no
/usr/bin/ld: cannot find -lzydis
clang: error: linker command failed with exit code 1 (use -v to see invocation)
> checking for c links(zydis)
> checking for c snippet(find_package/zydis)
checking for zydis ... no
checking for xmake::polyhook_2 ... no
/usr/bin/ld: cannot find -lpolyhook_2
clang: error: linker command failed with exit code 1 (use -v to see invocation)
> checking for c links(polyhook_2)
> checking for c snippet(find_package/polyhook_2)
checking for polyhook_2 ... no
checking for xmake::Tracy ... no
/usr/bin/ld: cannot find -lTracy
clang: error: linker command failed with exit code 1 (use -v to see invocation)
> checking for c links(Tracy)
> checking for c snippet(find_package/Tracy)
checking for Tracy ... no
/usr/bin/ld: cannot find -lSuperluminal
clang: error: linker command failed with exit code 1 (use -v to see invocation)
> checking for c links(Superluminal)
> checking for c snippet(find_package/Superluminal)
checking for Superluminal ... no
checking for cargo::patternsleuth_bind ... no
note: the following packages were not found in any repository (check if they are spelled correctly):
  -> ImGuiTextEdit

Do you have any documentation about the prereqs required to make this branch compile?
I tried on WSL Ubuntu 20.04 with xmake, clang, rust, etc installed, but still haven't been able to compile or configure?

There is some build information for the linux port here: https://github.com/Yangff/RE-UE4SS/wiki

Do you have any documentation about the prereqs required to make this branch compile?
I tried on WSL Ubuntu 20.04 with xmake, clang, rust, etc installed, but still haven't been able to compile or configure?
bitonality@DESKTOP-STUDIO:~/RE-UE4SS$ xmake f --mode="Game__Shipping__Linux" -v
checking for platform ... linux
checking for architecture ... x86_64
checking for gcc ... /usr/bin/gcc
checking for zig ... no
checking for zig ... no
checking for nim ... no
checking for nim ... no
checking for GUI ... no
checking for unzip ... /usr/bin/unzip
checking for git ... /usr/bin/git
checking for gzip ... /usr/bin/gzip
checking for tar ... /usr/bin/tar
/usr/bin/git rev-parse HEAD
checking for xmake::ImGuiTextEdit ... no
checking for pacman ... no
checking for emerge ... no
checking for brew ... no
checking for pkg-config ... no
checking for pkgconf ... no
checking for clang ... /usr/bin/clang
checking for the c compiler (cc) ... clang
checking for /usr/bin/clang ... ok
checking for flags (-fPIC) ... ok
checking for flags (-fdiagnostics-color=always) ... ok
checking for clang++ ... /usr/bin/clang++
checking for the linker (ld) ... clang++
checking for /usr/bin/clang++ ... ok
checking for flags (-fPIC) ... ok
/usr/bin/ld: cannot find -lImGuiTextEdit
clang: error: linker command failed with exit code 1 (use -v to see invocation)
> checking for c links(ImGuiTextEdit)
> checking for c snippet(find_package/ImGuiTextEdit)
checking for ImGuiTextEdit ... no
checking for cmake ... no
checking for cmake ... /snap/bin/cmake
checking for xmake::zycore ... no
/usr/bin/ld: cannot find -lzycore
clang: error: linker command failed with exit code 1 (use -v to see invocation)
> checking for c links(zycore)
> checking for c snippet(find_package/zycore)
checking for zycore ... no
checking for xmake::zydis ... no
/usr/bin/ld: cannot find -lzydis
clang: error: linker command failed with exit code 1 (use -v to see invocation)
> checking for c links(zydis)
> checking for c snippet(find_package/zydis)
checking for zydis ... no
checking for xmake::polyhook_2 ... no
/usr/bin/ld: cannot find -lpolyhook_2
clang: error: linker command failed with exit code 1 (use -v to see invocation)
> checking for c links(polyhook_2)
> checking for c snippet(find_package/polyhook_2)
checking for polyhook_2 ... no
checking for xmake::Tracy ... no
/usr/bin/ld: cannot find -lTracy
clang: error: linker command failed with exit code 1 (use -v to see invocation)
> checking for c links(Tracy)
> checking for c snippet(find_package/Tracy)
checking for Tracy ... no
/usr/bin/ld: cannot find -lSuperluminal
clang: error: linker command failed with exit code 1 (use -v to see invocation)
> checking for c links(Superluminal)
> checking for c snippet(find_package/Superluminal)
checking for Superluminal ... no
checking for cargo::patternsleuth_bind ... no
note: the following packages were not found in any repository (check if they are spelled correctly):
  -> ImGuiTextEdit

right now please use --UI=n to disable all UI related code..

okay.. that's because xmake has priority problem with add_files and remove_files. you can't add files back once they are removed by remove_files.
now it should compile without ui. let me see how to make the ui works..

it may or may not work, xmake is too buggy to harm my brain, I'm done

Maybe @localcc has some ideas.

it may or may not work, xmake is too buggy to harm my brain, I'm done. I'd say if a build system can't even figure out and update when source code changed. it should not even exist on earth. EVEN MAKEFILE CAN DO.

https://github.com/UE4SS-RE/RE-UE4SS/discussions/471

I added
        print(package:get("links"))
        package:add("links", "PolyHook_2")
        package:add("links", "asmjit")
        package:add("links", "asmtk")
        print(package:get("links"))

in polyhook's xmake.lua to force the link order, I believe this is not the documented way of that and it may just work by accident. But add_linkorders is ignored and this at least is doing something.
at the same time I'm a little worry there maybe other link order problem because it will only show error at the runtime when it can't find the symbol, instead of when linking.. but can check this later.
now there is an error code at exit my becasue of my rebase.. will do another git diff to make sure but at least it is something that compile and run..

Does this look like the expected invocation of clang?
[ 19%]: compiling.Game__Debug__Linux deps/first/Unreal/src/UnrealVersion.cpp
/usr/bin/clang -c -Qunused-arguments -m64 -O0 -std=c++20 -Ideps/first/Unreal/include -Ideps/first/Unreal/generated_include -Ideps/first/Unreal/include/Unreal -Ideps/first/Unreal/include/Unreal/Core -Ideps/first/SinglePassSigScanner/include -Ideps/first/Profiler/include -Ideps/first/Function/include -Ideps/first/ASMHelper/include -Ideps/first/DynamicOutput/include -Ideps/first/File/include -Ideps/first/Helpers/include -Ideps/first/Constructs/include -D_UNICODE -DUNICODE -DRC_UNREAL_EXPORTS -DRC_UNREAL_BUILD_STATIC -DRC_ASM_HELPER_BUILD_STATIC -DRC_CONSTRUCTS_BUILD_STATIC -DRC_DYNAMIC_OUTPUT_BUILD_STATIC -DRC_FILE_BUILD_STATIC -DRC_FUNCTION_BUILD_STATIC -DRC_HELPERS_BUILD_STATIC -DRC_PROFILER_BUILD_STATIC -DRC_SINGLE_PASS_SIG_SCANNER_BUILD_STATIC -DZYDIS_STATIC_BUILD -DZYCORE_STATIC_BUILD -DIS_TRACY=1 -DIS_SUPERLUMINAL=0 -DUE_GAME -DUE_BUILD_DEBUG -DPLATFORM_LINUX -DPLATFORM_UNIX -DLINUX -DOVERRIDE_PLATFORM_HEADER_NAME=Linux -DUBT_COMPILED_PLATFORM=Linux -Dprintf_s=printf -DDLLEXT=.so -fexceptions -fcxx-exceptions -isystem /home/bitonality/.xmake/packages/z/zydis/v4.1.0/407615077e1e4d0289baec649ac615fd/include -isystem /home/bitonality/RE-UE4SS/Intermediates/.packages/p/polyhook_2/latest/1a6f9c81b2d54fd0a3486f9b60fcbfbb/include -isystem /home/bitonality/.xmake/packages/z/zydis/v4.1.0/535d6f96e7084a5db9ad4044a6b3f6a6/include -isystem /home/bitonality/.xmake/packages/z/zycore/v1.5.0/701cb606e55343bfa8b72dac63855b13/include -isystem /home/bitonality/.xmake/packages/t/tracy/v0.10/fd845d78d0544faaa37f475d2e360e4f/include -fno-delete-null-pointer-checks -fms-extensions -g -fcolor-diagnostics -Wno-unknown-pragmas -Wno-unused-parameter -Wignored-attributes -fPIC -o Intermediates/.objs/Unreal/linux/x86_64/Game__Debug__Linux/deps/first/Unreal/src/UnrealVersion.cpp.o deps/first/Unreal/src/UnrealVersion.cpp
error: In file included from deps/first/ASMHelper/src/ASMHelper.cpp:4:
In file included from deps/first/DynamicOutput/include/DynamicOutput/DynamicOutput.hpp:9:
In file included from deps/first/DynamicOutput/include/DynamicOutput/FileDevice.hpp:10:
In file included from deps/first/File/include/File/File.hpp:6:
deps/first/File/include/File/FileType/StdFile.hpp:4:10: fatal error: 'format' file not found
#include <format>
         ^~~~~~~~
1 error generated.

I'm running into issues with getting the libstd includes to work. I followed the link mentioned in the brief (https://ubuntu.pkgs.org/22.04/ubuntu-main-amd64/libstdc++6_12-20220319-1ubuntu1_amd64.deb.html), but I'm still having issues. I want to make sure the invocation of clang looks correct before I dig into issues with my local linux setup.

Does this look like the expected invocation of clang?
[ 19%]: compiling.Game__Debug__Linux deps/first/Unreal/src/UnrealVersion.cpp
/usr/bin/clang -c -Qunused-arguments -m64 -O0 -std=c++20 -Ideps/first/Unreal/include -Ideps/first/Unreal/generated_include -Ideps/first/Unreal/include/Unreal -Ideps/first/Unreal/include/Unreal/Core -Ideps/first/SinglePassSigScanner/include -Ideps/first/Profiler/include -Ideps/first/Function/include -Ideps/first/ASMHelper/include -Ideps/first/DynamicOutput/include -Ideps/first/File/include -Ideps/first/Helpers/include -Ideps/first/Constructs/include -D_UNICODE -DUNICODE -DRC_UNREAL_EXPORTS -DRC_UNREAL_BUILD_STATIC -DRC_ASM_HELPER_BUILD_STATIC -DRC_CONSTRUCTS_BUILD_STATIC -DRC_DYNAMIC_OUTPUT_BUILD_STATIC -DRC_FILE_BUILD_STATIC -DRC_FUNCTION_BUILD_STATIC -DRC_HELPERS_BUILD_STATIC -DRC_PROFILER_BUILD_STATIC -DRC_SINGLE_PASS_SIG_SCANNER_BUILD_STATIC -DZYDIS_STATIC_BUILD -DZYCORE_STATIC_BUILD -DIS_TRACY=1 -DIS_SUPERLUMINAL=0 -DUE_GAME -DUE_BUILD_DEBUG -DPLATFORM_LINUX -DPLATFORM_UNIX -DLINUX -DOVERRIDE_PLATFORM_HEADER_NAME=Linux -DUBT_COMPILED_PLATFORM=Linux -Dprintf_s=printf -DDLLEXT=.so -fexceptions -fcxx-exceptions -isystem /home/bitonality/.xmake/packages/z/zydis/v4.1.0/407615077e1e4d0289baec649ac615fd/include -isystem /home/bitonality/RE-UE4SS/Intermediates/.packages/p/polyhook_2/latest/1a6f9c81b2d54fd0a3486f9b60fcbfbb/include -isystem /home/bitonality/.xmake/packages/z/zydis/v4.1.0/535d6f96e7084a5db9ad4044a6b3f6a6/include -isystem /home/bitonality/.xmake/packages/z/zycore/v1.5.0/701cb606e55343bfa8b72dac63855b13/include -isystem /home/bitonality/.xmake/packages/t/tracy/v0.10/fd845d78d0544faaa37f475d2e360e4f/include -fno-delete-null-pointer-checks -fms-extensions -g -fcolor-diagnostics -Wno-unknown-pragmas -Wno-unused-parameter -Wignored-attributes -fPIC -o Intermediates/.objs/Unreal/linux/x86_64/Game__Debug__Linux/deps/first/Unreal/src/UnrealVersion.cpp.o deps/first/Unreal/src/UnrealVersion.cpp
error: In file included from deps/first/ASMHelper/src/ASMHelper.cpp:4:
In file included from deps/first/DynamicOutput/include/DynamicOutput/DynamicOutput.hpp:9:
In file included from deps/first/DynamicOutput/include/DynamicOutput/FileDevice.hpp:10:
In file included from deps/first/File/include/File/File.hpp:6:
deps/first/File/include/File/FileType/StdFile.hpp:4:10: fatal error: 'format' file not found
#include <format>
         ^~~~~~~~
1 error generated.

I'm running into issues with getting the libstd includes to work. I followed the link mentioned in the brief (https://ubuntu.pkgs.org/22.04/ubuntu-main-amd64/libstdc++6_12-20220319-1ubuntu1_amd64.deb.html), but I'm still having issues. I want to make sure the invocation of clang looks correct before I dig into issues with my local linux setup.

I'm getting the same error and I've been able to compile this branch previously.
I also can't generate cmakelists with xmake project -k cmakelists -m "Game__Debug__Linux" because it gives the error release is not a valid complation mode for this project

Update regarding <format> missing.
I completely forgot that this branch doesn't currently use libc++ by defualt, but instead uses stdlibc++.
After switching setting -stdlib=libc++, I no longer get that error.

Note that if you're using libc++, you must also use -fexperimental-library.
This is the command I use currently:
xmake f -m "Game__Debug__Linux" --UI=n --cxflags="-stdlib=libc++ -fexperimental-library" -y
It compiles successfully for me but I have some linker errors to fix still.

some linker errors

My observation is that changing cflags doesn't seem to cause packages to be recompiled, as evidenced by the lack of compilation information for third-party libraries such as polyhook, zydis, etc., as well as rust, on xmake f.
If this is the case, then this may result in trying to link glibc and clang's libc, which can lead to some problems

Note that if you're using libc++, you must also use -fexperimental-library (because of jthread). This is the command I currently use: xmake f -m "Game__Debug__Linux" --UI=n --cxflags="-stdlib=libc++ -fexperimental-library" -y It compiles successfully for me but I have some linker errors to fix still.

This unblocked me!! I know there's an xxmake way to leverage set_runtimes("c++_static") to accomplish the -stdlib flag part, but I need to do more research. This workaround seems sufficient for now.
Now I'm getting rust errors
[  9%]: linking.Game__Debug__Linux libpatternsleuth_bind.a
/home/bitonality/.cargo/bin/rustc -C debuginfo=2 -C opt-level=0 --edition 2018 -L dependency=/home/bitonality/LinuxUE4SS/Intermediates/.packages/c/cargo_patternsleuth_bind/latest/19f630f7508a43dba17124f457725bfa/lib -C debuginfo=2 --extern patternsleuth=/home/bitonality/LinuxUE4SS/Intermediates/.packages/c/cargo_patternsleuth_bind/latest/19f630f7508a43dba17124f457725bfa/lib/libpatternsleuth-793969ca9c36fda1.rlib --crate-type=staticlib -o Binaries/Game__Debug__Linux/patternsleuth_bind/libpatternsleuth_bind.a deps/first/patternsleuth_bind/src/lib.rs
error: error[E0786]: found invalid metadata files for crate `patternsleuth`
 --> deps/first/patternsleuth_bind/src/lib.rs:5:5
  |
5 | use patternsleuth::resolvers::{
  |     ^^^^^^^^^^^^^
  |
  = note: failed to mmap file '/home/bitonality/LinuxUE4SS/Intermediates/.packages/c/cargo_patternsleuth_bind/latest/19f630f7508a43dba17124f457725bfa/lib/libpatternsleuth-793969ca9c36fda1.rlib': memory map must have a non-zero length

error[E0786]: found invalid metadata files for crate `patternsleuth`
   --> deps/first/patternsleuth_bind/src/lib.rs:101:15
    |
101 |     let exe = patternsleuth::process::internal::read_image()?;
    |               ^^^^^^^^^^^^^
    |
    = note: failed to mmap file '/home/bitonality/LinuxUE4SS/Intermediates/.packages/c/cargo_patternsleuth_bind/latest/19f630f7508a43dba17124f457725bfa/lib/libpatternsleuth-793969ca9c36fda1.rlib': memory map must have a non-zero length

error[E0433]: failed to resolve: use of undeclared type `UE4SSResolution`
   --> deps/first/patternsleuth_bind/src/lib.rs:105:34
    |
105 |     let resolution = exe.resolve(UE4SSResolution::resolver())?;
    |                                  ^^^^^^^^^^^^^^^ use of undeclared type `UE4SSResolution`

error: aborting due to 3 previous errors

Some errors have detailed explanations: E0433, E0786.
For more information about an error, try `rustc --explain E0433`.

Did you get these errors?

Did you get these errors?

I did not.
Are you sure you're on the latest patternsleuth ?

btw, using  --cxflags="-stdlib=libc++ -fexperimental-library" is causing some problem with cmake for me.. idk why but it seems to take the whole "-stdlib=libc++ -fexperimental-library" as one argument..

You could try --cxflags="'-stdlib=libc++' '-fexperimental-library'".
I had a problem while experimenting  with another xmake option that gets passed to the compiler (includedirs) and adding single quotes around each entry fixed it, but I'm not sure if it works the same for cxflags.
I don't know the official xmake way to pass multiple flags unfortunately.

Did you get these errors?

I did not. Are you sure you're on the latest patternsleuth ?

Should I be on the forked patternsleuth?
Out of the box, the ue4ss fork on linux-port branch points to commit 0d51753 in the forked patternsleuth repo. Is that what your repo state looks like?

btw, using  --cxflags="-stdlib=libc++ -fexperimental-library" is causing some problem with cmake for me.. idk why but it seems to take the whole "-stdlib=libc++ -fexperimental-library" as one argument..

You could try --cxflags="'-stdlib=libc++' '-fexperimental-library'". I had a problem while experimenting with another xmake option that gets passed to the compiler (includedirs) and adding single quotes around each entry fixed it, but I'm not sure if it works the same for cxflags. I don't know the official xmake way to pass multiple flags unfortunately.

The way I set it locally is adding those flags to build_configs.lua
["Linux"] = {
        ["defines"] = {
            "PLATFORM_LINUX",
            "PLATFORM_UNIX",
            "LINUX",
            "OVERRIDE_PLATFORM_HEADER_NAME=Linux",
            "UBT_COMPILED_PLATFORM=Linux",
            "printf_s=printf",
            "DLLEXT=.so"
        },
        ["cxflags"] = {
            -- "clang::xxx" only adds the flag if clang is toolchain chosen/specified by xmake.
            "clang::-fno-delete-null-pointer-checks",
            "clang::-stdlib=libc++",
            "clang::-fexperimental-library"
        }
    }
    ```

Ok unblocked on compiling.
Are you hitting the
/usr/bin/ld: /usr/bin/ld: DWARF error: invalid or unhandled FORM value: 0x23

as your linker error?

Ok unblocked on compiling.
Are you hitting the
/usr/bin/ld: /usr/bin/ld: DWARF error: invalid or unhandled FORM value: 0x23

as your linker error?

Yes, and lots of undefined reference messages.

/usr/bin/ld: /usr/bin/ld: DWARF error: invalid or unhandled FORM value: 0x23

This is fixed by adding --ldflags="-fuse-ld=lld" to your xmake configuration command.
After that, there's a bunch of undefined symbol errors.

Ok unblocked on compiling.
Are you hitting the
/usr/bin/ld: /usr/bin/ld: DWARF error: invalid or unhandled FORM value: 0x23

as your linker error?

Yes, and lots of undefined reference messages.

I managed to get a successful build/link by adding the following linker flags
xmake f -m "Game__Debug__Linux" --UI=n --cxflags="-stdlib=libc++ -fexperimental-library"  --ldflags="-stdlib=libc++ -fuse-ld=lld" --ccache=n
Without specifying -fuse-ld=lld it looks like xmake uses regular ol ld which doesn't have dwarf 5 support. Does adding those two linker flags unblock you?

Without specifying -fuse-ld=lld it looks like xmake uses regular ol ld which doesn't have dwarf 5 support. Does adding those two linker flags unblock you?

No it didn't.
I just tried to do a clean build and I'm now stuck at the same place as @Yangff where CMake won't compile polyhook because the options passed to --cxflags are passed as a single option instead of two separate ones.

This is how it gets passed to CMake: clang++   "-stdlib=libc++ -fexperimental-library" -m64 -DZYCORE_STATIC_BUILD
It has quotes around it for some reason.

According to this, we're doing it correctly so this might be an xmake bug.

Go ahead and promote these flags to a config value in build_configs.lua
    ["Linux"] = {
        ["defines"] = {
            "PLATFORM_LINUX",
            "PLATFORM_UNIX",
            "LINUX",
            "OVERRIDE_PLATFORM_HEADER_NAME=Linux",
            "UBT_COMPILED_PLATFORM=Linux",
            "printf_s=printf",
            "DLLEXT=.so"
        },
        ["cxflags"] = {
            "clang::-fno-delete-null-pointer-checks",
            "clang::-stdlib=libc++",
            "clang::-fexperimental-library"
        },
        ["ldflags"] = {
            "-stdlib=libc++",
            "-fuse-ld=lld"
        }
    }
    ```

    xmake f -m "Game__Debug__Linux" --UI=n
    xmake

Go ahead and promote these flags to a config value in build_configs.lua

They probably should be, but I don't want to mess with the build scripts, they'll be annoying to maintain so I'll wait until it gets done on the remote and then I'll pull the changes.

If you use --cxxflags twice, it won't work, it'll use only the last one.
However, there's also --cxflags (notice only one x), and if you use both --cxxflags and --cxflags, with one option each, it will actually work.
Unfortunately, doing a clean build didn't fix my problem.
I'm getting linker errors involving the standard libs, like:
ld.lld: error: undefined symbol: std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::~basic_string()

Go ahead and promote these flags to a config value in build_configs.lua

They probably should be, but I don't want to mess with the build scripts, they'll be annoying to maintain so I'll wait until it gets done on the remote and then I'll pull the changes.
 If you use --cxxflags twice, it won't work, it'll use only the last one. However, there's also --cxflags (notice only one x), and if you use both --cxxflags and --cxflags, with one option each, it will actually work.
Unfortunately, doing a clean build didn't fix my problem. I'm getting linker errors involving the standard libs, like: ld.lld: error: undefined symbol: std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::~basic_string()

Even with --ldflags="-stdlib=libc++ -fuse-ld=lld" ?

Even with --ldflags="-stdlib=libc++ -fuse-ld=lld" ?

Oh I didn't realize -stdlib was a linker option.
I had it as a compiler option.
It does fully build now but I can't do a clean build because -fuse-ld has the same bug as the compiler options where you can't pass multiple options.
That'll probably cause problems, but hopefully it will get added to this PR eventually.

@Yangff So I think the problem with xmake not picking up file changes might be limited to any external dependencies and for some reason patternsleuth.
For me, it picks up changes in all other files.
External dependencies + patternsleuth are compiled in the xmake configuration step, though even if you make a change, re-running the config command isn't enough to trigger a rebuild of the modified file.
I don't think it's appropriate for patternsleuth to be treated as an external dependency.
It even lives in the deps/first directory like the Unreal module.
@localcc Is there any way to make xmake treat patternsleuth as a first party dependency so that it builds it during the build step instead of the compilation step ? I would assume xmake would start picking up on file changes automatically and rebuild when necessary at that point.

Due to the way xmake works with rust dependencies, it treats them as external packages from cargo with a local Cargo.toml file. We could opt for rewriting the Cargo.toml inside of xmake, and using it completely to build rust (aside from dependencies of patternsleuth), xmake supports this use case. Here's a relevant example project page from xmake docs: https://xmake.io/#/guide/project_examples?id=add-cargo-package-dependences

Due to the way xmake works with rust dependencies, it treats them as external packages from cargo with a local Cargo.toml file. We could opt for rewriting the Cargo.toml inside of xmake, and using it completely to build rust (aside from dependencies of patternsleuth), xmake supports this use case. Here's a relevant example project page from xmake docs: https://xmake.io/#/guide/project_examples?id=add-cargo-package-dependences

This sounds good to me.
We should definitely do something.
Neither xmake clean nor xrepo clean causes patternsleuth to be rebuilt when using the xmake config command after a change has been made.

External dependencies

Indeed, and have no idea why polyhooks are cached in Intermediates while others are cached in ~/.xmake/.
Also changes to their xmake.lua won't take effect either. The logic here seems to be that it simply matches the name and version. And then assumes that the right dependency is found.
Additionally, the contamination of the global space is also a problem in that it can lead to some modifications being subtly out of sync between different developers (maybe even after a clean clone?)

Additionally, the contamination of the global space is also a problem in that it can lead to some modifications being subtly out of sync between different developers (maybe even after a clean clone?)

As far as I know, the dependencies also get a build hash which should prevent from global space being contaminated for other projects, but this needs to be tested and a bug should be filed to xmake if such a case is discovered.

build hash

What is the build hash based on? I mean at least deleting .xmake folder should indicate two different builds, but it is not enough to trigger a rebuild of global cached packages.

Thx for the quick response!
The issue was with the "Scripts" folder, which I've created with a capital case. Renaming to "scripts" did the trick, now the mods are loading correctly. (Thank you, Windows case insensitiveness)
Now that I can test this, do you want me to write a .md file with requirements (GLIBC >= 2.35, GLIBCXX >= 3.4.32, etc), how to upgrade stdc++ lib in the case is needed, and how to set up a script to run a game server with UE4SS injected?
This will bring more testers, and reveal any issue that is not already considered for Linux implementation
I'm asking because this will attract (I hope) A LOT more people to try it while this MR is still in draft

I would KILL to be able to use mods on my Linux dedi again. I HATE using Windows for my host, but I like to mod more than vanilla play.
 :D

External dependencies

Indeed, and have no idea why polyhook and patternsleuth are cached in Intermediates while others are cached in ~/.xmake/. Also changes to their xmake.lua won't take effect either. The logic here seems to be that it simply matches the name and version. And then assumes that the right dependency is found. Additionally, the contamination of the global space is also a problem in that it can lead to some modifications being subtly out of sync between different developers (maybe even after a clean clone?)

rewriting the Cargo.toml inside of xmake

I'd rather let xmak to simply re-run a cargo build everytime. Rust itself has an incremental building that should work there. Using xmake for rust can prevent the rust language server from working or complicate configuration (say you need to manage two rust dependencies at the same time)

The reason for no rebuild is that we are kind of hacking

Due to the way xmake works with rust dependencies, it treats them as external packages from cargo with a local Cargo.toml file. We could opt for rewriting the Cargo.toml inside of xmake, and using it completely to build rust (aside from dependencies of patternsleuth), xmake supports this use case. Here's a relevant example project page from xmake docs: https://xmake.io/#/guide/project_examples?id=add-cargo-package-dependences

This sounds good to me. We should definitely do something. Neither xmake clean nor xrepo clean causes patternsleuth to be rebuilt when using the xmake config command after a change has been made.

The correct command to run is
xmake require --force --extra="{configs={cargo_toml='~/LinuxUE4SS/deps/first/patternsleuth_bind/Cargo.toml'}}" cargo::patternsleuth_bind
This gesture force uninstalls the package and flushes the cache and then rebuilds from src and re-registers the package with the package cache.

@Yangff So I think the problem with xmake not picking up file changes might be limited to any external dependencies and for some reason patternsleuth.
For me, it picks up changes in all other files. External dependencies + patternsleuth are compiled in the xmake configuration step, though even if you make a change, re-running the config command isn't enough to trigger a rebuild of the modified file.
I don't think it's appropriate for patternsleuth to be treated as an external dependency. It even lives in the deps/first directory like the Unreal module.
@localcc Is there any way to make xmake treat patternsleuth as a first party dependency so that it builds it during the build step instead of the compilation step ? I would assume xmake would start picking up on file changes automatically and rebuild when necessary at that point.

I spend the last week doing a deep-dive into xmake to see what possible solutions we could leverage/implement.
Philosophical Question
Who is responsible for patternsleuth compilation? Is it the job of xmake to build patternsleuth because it's submoduled in this repo? Is it patternsleuth's job to provide lib/binaries for ue4ss? The crux of the problem resides somewhere in how we interpret patternsleuth as a dependency.
xmake package
xmake packages all implement some form of install(). In the case of precompiled binaries or pure headers, the install() might be something like
os.cp(package:get("binaries"), package:installdir())

In the case of a package containing source that must be compiled, the install() usually looks something like...
os.exec("mycompiler", package:get("sourcedir"), "--output", package:cachedir())
os.cp(package:cachedir(), package:installdir())

A key concept to understand is that once a package is installed, xmake pretty much considers it an immutable dependency that it won't have to build again. The use case for packages shines when packages are used as versioned dependencies from a remote source (i.e. cargo::clap v4.4.14, xrepo:7zip 21.02, etc).
Why don't changes to *.rs files cause patternsleuth to rebuild?
patternsleuth is currently configured as a package which means that once it is installed, xmake treats it as an immutable dependency that is already built AND installed.
How can I force a package to run the install gesture?
xmake require --force <package>
This forces a reinstallation of a package by essentially deleting the locally installed package and removing all references to the package from xmake dependency/package cache. It then runs the internal install() of the package (which under the hood is a cargo build with os.cp to the package:installdir())
Possible Solutions:
Manually reinstall the package when you change patternsleuth source
Basically just executing xmake require --force cargo::patternsleuth if you know you made changes.
Treat patternsleuth as a target instead of a package
Essentially have xmake manage the building of patternsleuth as a target. Under the hood this would execute the cargo build/rustc commands on the deps/first/patternsleuth/... dir. xmake already has some of this infrastructure (detecting if cargo is installed, versioning, dependency resolution, compiler flags, etc...). The general guidelines from the xmake maintainers are to use a form of os.exec("cargo build") to build a pure-rust project that has a self contained/defined Cargo.toml.
Use the package.install_always policy
Packages can be assigned the policy package.install_always. Packages that have this policy assigned will run their install scripts every time xmake config <any flags> is run. This has the downside of trampling the cargo cache every time the package installs which can result in a lot of unnecessary compilations of patternsleuth.
Should we support building UE4SS with a local/private/dev version of patternsleuth?
If we continue submoduling patternsleuth - we just point xmake to the dir with patternsleuth (this is currently happening)
If we want to consume commit based versions from the patternsleuth git - we should probably support also pointing to a local repository. This can be accomplished by an xmake flag like --patternsleuth_src=... or potentially even more spicy is changing patternsleuth_bind Cargo.toml to point to the patternsleuth repo and using the [patch] dependency feature of Cargo to point to a local instance of the patternsleuth repo.
Many ways to accomplish this and none of them seem particularly egregious.

patternsleuth should always be submoduled, if any source changes are to be made they could be made inside of the submodule that is just checked out at a different commit and/or pointing to a different origin.

Fixing the CI and.. for unknown reason the clang on CI can't find the #include <format>.
So, I tried to switch to clang's libc++, but with libc++ it cannot find jthread although I turn on -fexperimental-library. Any clue?

Fixing the CI and.. for unknown reason the clang on CI can't find the #include <format>. So, I tried to switch to clang's libc++, but with libc++ it cannot find jthread although I turn on -fexperimental-library. Any clue?
https://github.com/Yangff/RE-UE4SS/actions/runs/9222085361/job/25372512543

Not sure what could be causing this.
In the past, the format header not being found was fixed for me by using libc++, which you're already doing, and jthread was fixed by using -fexperimental-library which you're also already doing so I'm at a loss as to what could be causing this problem, and I was using clang 17 so that should be sufficient but we can move to clang 18 if there's a release available.
Maybe try add the verbose flag to confirm that the command line being used is correct ?

According to this, jthread (search for joining thread) is only available in clang 18 even with the experimental flag.

To use clang 18 in the CI script, use KyleMayes/install-llvm-action@v2 instead of KyleMayes/install-llvm-action@v1.

To use clang 18 in the CI script, use KyleMayes/install-llvm-action@v2 instead of KyleMayes/install-llvm-action@v1.

thanks.. it seems that there is no clang18 on x64 even on v2 https://github.com/KyleMayes/install-llvm-action/blob/v2/assets.json#L45
currently I'm pulling gcc13's libstd++ and that works https://github.com/Yangff/RE-UE4SS/actions/runs/9222647973
an option --libcxx is added to use libcxx.

To use clang 18 in the CI script, use KyleMayes/install-llvm-action@v2 instead of KyleMayes/install-llvm-action@v1.

thanks.. it seems that there is no clang18 on x64 even on v2 https://github.com/KyleMayes/install-llvm-action/blob/v2/assets.json#L45
currently I'm pulling gcc13's libstd++ and that works https://github.com/Yangff/RE-UE4SS/actions/runs/9222647973 an option --libcxx is added to use libcxx.

My bad, it was 2.0.1 that they added 18.

2.0.1

https://github.com/KyleMayes/install-llvm-action/blob/v2.0.2/assets.json#L47
still not there.. I'm going to again try to lower the requried glibc version with glibc_version_header to see if we can get rid of 22.04.. and with ubuntu 24.04 there are clang 18 and gcc 14 by default..

2.0.1

https://github.com/KyleMayes/install-llvm-action/blob/v2.0.2/assets.json#L47
still not there.. I'm going to again try to lower the requried glibc version with glibc_version_header to see if we can get rid of 22.04.. and with ubuntu 24.04 there are clang 18 and gcc 14 by default.. https://github.com/actions/runner-images/blob/main/images/ubuntu/Ubuntu2404-Readme.md

Oh I misread what you said earlier.
I guess they didn't add x64 yet for whatever reason.

I've confirmed that clang 18 with -fexperimental-librray works.
However, there appears to be some problem with deps/first/ArgsParser, as it won't link with clang.
I haven't bothered to look into the actual problem as we don't actually use this dependency and as such it should be removed.
It compiles and links successfully locally for me with clang 18.1.4 after removing ArgsParser.

Apparently this action supports clang 18: https://github.com/marketplace/actions/setup-llvm, I haven't tested it.

don't know why but the libcxx I added failed to link when using clang 18 + libc++.. https://github.com/Yangff/RE-UE4SS/actions/runs/9233858625
clang17 + gcc 13's libstdc++ can build on ci https://github.com/Yangff/RE-UE4SS/actions/runs/9222647973
zig cc can build and link using clang 18 + libc++
in both zigcc and clang17+gcc cases, the artifacts will link both libncurses and libncursesw.. even when I'm using my own ncursesw package trying to figure out why.. https://github.com/Yangff/RE-UE4SS/actions/runs/9233651993
locally, building with clang17+libstdc++ is all good. it only linkes to libncursesw

don't know why but the libcxx I added failed to link when using clang 18 + libc++.. https://github.com/Yangff/RE-UE4SS/actions/runs/9233858625

This is what my build_rules looked like when it succeeded to build & link:
if has_config("libcxx") then
    table.insert(CLANG_COMPILE_OPTIONS["cxflags"], "-stdlib=libc++")
    table.insert(CLANG_COMPILE_OPTIONS["cxflags"], "-fexperimental-library")
    table.insert(CLANG_COMPILE_OPTIONS["ldflags"], "-static-libstdc++")
    table.insert(CLANG_COMPILE_OPTIONS["ldflags"], "-l:libc++abi.a")
end

This is what yours looked like in that action:
if has_config("libcxx") then
    table.insert(CLANG_COMPILE_OPTIONS["cxflags"], "-stdlib=libc++")
    table.insert(CLANG_COMPILE_OPTIONS["ldflags"], "-fuse-ld=lld")
    table.insert(CLANG_COMPILE_OPTIONS["ldflags"], "-stdlib=libc++")
    table.insert(CLANG_COMPILE_OPTIONS["cxflags"], "-fexperimental-library")
end

Make of that what you will.

@Yangff Oh I just remembered this: https://github.com/UE4SS-RE/RE-UE4SS/pull/384#issuecomment-2129298547
TLDR: Remove ArgsParser.

ok, right now the CI can build with clang 18 + gcc13's stdc++.
The problem with zig cc seems to be that xmake will not use my cc config to build its external packages and result in mixed use of libstdc++ and libc++ symbols.. don't know how to fix that..
And applied the changes to libcxx flags .. and I'm giving both a retry..

okay I think now --zig=y may also work with llvm 18.
But one problem here, currently UE4SS has an undefined symbol
/home/yangff/.steam/SteamApps/common/PalServer/Pal/Binaries/Linux/PalServer-Linux-Test: symbol lookup error: /home/yangff/Project/RE-UE4SS/build/Output/Game__Shipping__Linux/UE4SS/bin/libUE4SS.so: undefined symbol: _ZN2RC6Unreal26FGenericWidePlatformString10GetVarArgsEPDsmRPKDsP13__va_list_tag

This function is not defined but somehow referred, the binary generated by zig will try to resolve it at the beginning for some reason... maybe some magic happened there...
This symbol is also missed on the clang version but that one can at least load without complaining about it.

Anyway, now it should build for all : clang + libstdc++; clang + libc++ and zig-cc + clang + libc++ statically

This is amazing work.
It'll be awesome when this finally gets merged, it'll make it so much easier to develop on a Linux machine, the current way I do it is super hacky.

hmm.. the ncursesw from xmake is not providing correct wide char support.. weird..

xmake's ncurses is not compiled with normal prefix, so it cannot find the term database after compilling.
for some reason setlocale(LC_ALL, "") is not correctly setting the locale to my system locale which is en_US.UTF-8, using LC_ALL=en_US.UTF-8 can workaround this. ncurses might need the upstream to somehow fix it..?

cross compile from windows to linux
xmake f -y --ue4ssUI=None --ue4ssInput=n --ue4ssIsBeta=y --zig=y -p linux -a x86_64-unknown-linux-gnu -v -D -m "Game__Shipping__Linux"

you need to wait untli this get fixed https://github.com/xmake-io/xmake/issues/4049#issuecomment-2132153442
and have zig 0.13 installed for clnag 18 on windows.
can't build ui because they need ./configure but this alone should allow modders to create mod all in windows.

And on linux this should give you a zig build which elimiate the glibc version requirement..
xmake f -y --ue4ssUI=TUI --ue4ssInput=y --ue4ssIsBeta=y --zig=y -m "Game__Shipping__Linux"
xmake build

cross compile from windows to linux
xmake f -y --ue4ssUI=None --ue4ssInput=n --ue4ssIsBeta=y --zig=y -p linux -a x86_64-unknown-linux-gnu -v -D -m "Game__Shipping__Linux"

you need to wait untli this get fixed xmake-io/xmake#4049 (comment)
and have zig 0.13 installed for clnag 18 on windows.
can't build ui because they need ./configure but this alone should allow modders to create mod all in windows.
(still need some fix to the script)

https://github.com/UE4SS-RE/RE-UE4SS/pull/516
This is another potential workaround for the cargo package issue. I'd need to plumb the cross-compile stuff in (if my understanding of the current issue is correct), but this is my preferred way of handling it.

cross compile from windows to linux
xmake f -y --ue4ssUI=None --ue4ssInput=n --ue4ssIsBeta=y --zig=y -p linux -a x86_64-unknown-linux-gnu -v -D -m "Game__Shipping__Linux"

you need to wait untli this get fixed xmake-io/xmake#4049 (comment)
and have zig 0.13 installed for clnag 18 on windows.
can't build ui because they need ./configure but this alone should allow modders to create mod all in windows.
(still need some fix to the script)

#516
This is another potential workaround for the cargo package issue. I'd need to plumb the cross-compile stuff in (if my understanding of the current issue is correct), but this is my preferred way of handling it.

I think there might be more bugs on the xmake side for cross compiling rust say it can't specify the -target matching the cargo (which is correctly set to arch), and cannot copy the build deps, not sure if this is going to be a problem for a fake cross compile.
I also want to let xmake to install its own zig but haven't figure out how to make it works..

Btw, I'm trying to install zig within xmake.. but the installed toolchain is not added to path when compiling packages..
It's not a must since user can install one in their system but having a installed one can help control it's version etc.. (right now we need 0.13 for clang 18..) so if anyone can see how the remote toolchain can work.. please!!!

cpp mod works..

okay... this is crazy... after switching to llvm and libc++ we're basically jumping into its eco system ...
BUT.. maybe not completely...
the result is .. for some reason.. we will use Unreal's allocator (FMemory::Malloc and FMemory::Free because they overriding operator new and delete)
This is causing error like this
LowLevelFatalError [File:.\Runtime/Core/Private/HAL/MallocBinned2.cpp] [Line: 1428]
FMallocBinned2 Attempt to realloc an unrecognized block 0x7f4986b40000   canary == 0x0 != 0xb7 or 0xb7
^CFUnixPlatformMisc::RequestExit

And it seems that here we even got some racing.. the result is .. sometimes we will be able to load the UE4SS and sometimes we failed immediately when start because (maybe.. I'm mostly guessing), UE4SS's allocator is not ready at that moment..

with that said.. it's a bit weird why this is not a problem for libstdc++.. the symbol name should be the same..

okay.. really confused me.. maybe not caused by libc++ but it happened when I load a cpp mod..

I don't really understand what causes the problem here.. the mod and UE4SS should use the same compile command but the result.. UE4SS will generate theses symbols
readelf -s -C --wide ./Binaries/Game__Shipping__Linux/UE4SS/libUE4SS.so | grep "operator delete"
  3688:<PHONE_NUMBER>b8dab0    10 FUNC    WEAK   DEFAULT   13 operator delete(void*, std::align_val_t)
  7305:<PHONE_NUMBER>747700     5 FUNC    GLOBAL DEFAULT   13 RC::Unreal::FUseSystemMallocForNew::operator delete(void*)
  9307:<PHONE_NUMBER>b8d9f0    10 FUNC    WEAK   DEFAULT   13 operator delete(void*)
  9825:<PHONE_NUMBER>b8da20    10 FUNC    WEAK   DEFAULT   13 operator delete[](void*)
 10059:<PHONE_NUMBER>b8dae0    10 FUNC    WEAK   DEFAULT   13 operator delete[](void*, std::align_val_t)
 12134:<PHONE_NUMBER>747720     5 FUNC    GLOBAL DEFAULT   13 RC::Unreal::FUseSystemMallocForNew::operator delete[](void*)
 20765:<PHONE_NUMBER>b8da00    10 FUNC    LOCAL  HIDDEN    13 operator delete(void*, std::nothrow_t const&)
 20766:<PHONE_NUMBER>b8da10    10 FUNC    LOCAL  HIDDEN    13 operator delete(void*, unsigned long)
 20767:<PHONE_NUMBER>b8da30    10 FUNC    LOCAL  HIDDEN    13 operator delete[](void*, std::nothrow_t const&)
 20768:<PHONE_NUMBER>b8da40    10 FUNC    LOCAL  HIDDEN    13 operator delete[](void*, unsigned long)
 20771:<PHONE_NUMBER>b8dac0    10 FUNC    LOCAL  HIDDEN    13 operator delete(void*, std::align_val_t, std::nothrow_t const&)
 20772:<PHONE_NUMBER>b8dad0    13 FUNC    LOCAL  HIDDEN    13 operator delete(void*, unsigned long, std::align_val_t)
 20773:<PHONE_NUMBER>b8daf0    10 FUNC    LOCAL  HIDDEN    13 operator delete[](void*, std::align_val_t, std::nothrow_t const&)
 20774:<PHONE_NUMBER>b8db00    13 FUNC    LOCAL  HIDDEN    13 operator delete[](void*, unsigned long, std::align_val_t)
 23898:<PHONE_NUMBER>b8d9f0    10 FUNC    WEAK   DEFAULT   13 operator delete(void*)
 27437:<PHONE_NUMBER>b8da20    10 FUNC    WEAK   DEFAULT   13 operator delete[](void*)
 29534:<PHONE_NUMBER>747700     5 FUNC    GLOBAL DEFAULT   13 RC::Unreal::FUseSystemMallocForNew::operator delete(void*)
 29536:<PHONE_NUMBER>747720     5 FUNC    GLOBAL DEFAULT   13 RC::Unreal::FUseSystemMallocForNew::operator delete[](void*)
 36311:<PHONE_NUMBER>b8dab0    10 FUNC    WEAK   DEFAULT   13 operator delete(void*, std::align_val_t)
 36312:<PHONE_NUMBER>b8dae0    10 FUNC    WEAK   DEFAULT   13 operator delete[](void*, std::align_val_t)

However, in the cppmod they become global even if I add -fvisibility=hidden
readelf -s -C --wide ./Binaries/Game__Shipping__Linux/TestMod/libTestMod.so | grep "operator delete"
     3:<PHONE_NUMBER>000000     0 FUNC    GLOBAL DEFAULT  UND operator delete(void*)
    34:<PHONE_NUMBER>000000     0 FUNC    GLOBAL DEFAULT  UND operator delete[](void*)
   746:<PHONE_NUMBER>3eb130     5 FUNC    GLOBAL DEFAULT   13 RC::Unreal::FUseSystemMallocForNew::operator delete(void*)
   971:<PHONE_NUMBER>3eb150     5 FUNC    GLOBAL DEFAULT   13 RC::Unreal::FUseSystemMallocForNew::operator delete[](void*)
 19360:<PHONE_NUMBER>000000     0 FUNC    GLOBAL DEFAULT  UND operator delete(void*)
 19521:<PHONE_NUMBER>3eb130     5 FUNC    GLOBAL DEFAULT   13 RC::Unreal::FUseSystemMallocForNew::operator delete(void*)
 19523:<PHONE_NUMBER>3eb150     5 FUNC    GLOBAL DEFAULT   13 RC::Unreal::FUseSystemMallocForNew::operator delete[](void*)
 22495:<PHONE_NUMBER>000000     0 FUNC    GLOBAL DEFAULT  UND operator delete[](void*)

So.. I can't be sure but something must be wrong to cause the messy..

hmm.. and I've no idea if visibility can fix this.. intuitively it should but when I add it to cxflags it is not doing anything.. not sure if it's xmake again or it won't work..

worked out a weird fix...
https://github.com/Yangff/RE-UE4SS/tree/linux-dlopen-loader
This will build a libLoader.so which (kinda like the proxy in windows) will be loaded by LD_PRELOAD and then, it will load libUE4SS.so with local and deepbind to prevent the symbol being overwritten by the game..
One problem here is, because mods will link UE4SS by default, it will pull in a dynamic operator new symbol from there.. which will again try to use the game's version.. I have to manually link the mod without -lUE4SS for now..

hmm.. no there are still weird problem..

okay.. figured out.. there is a HAS_UI macro ... and the mod is not getting this macro by default..

worked out a weird fix...
https://github.com/Yangff/RE-UE4SS/tree/linux-dlopen-loader
This will build a libLoader.so which (kinda like the proxy in windows) will be loaded by LD_PRELOAD and then, it will load libUE4SS.so with local and deepbind to prevent the symbol being overwritten by the game..
One problem here is, because mods will link UE4SS by default, it will pull in a dynamic operator new symbol from there.. which will again try to use the game's version.. I have to manually link the mod without -lUE4SS for now..

If we've pulled in UEs operator new, we should probably remove that completely, we don't need it and it appears to be causing problems.
@narknon Thoughts ?

worked out a weird fix...
https://github.com/Yangff/RE-UE4SS/tree/linux-dlopen-loader
This will build a libLoader.so which (kinda like the proxy in windows) will be loaded by LD_PRELOAD and then, it will load libUE4SS.so with local and deepbind to prevent the symbol being overwritten by the game..
One problem here is, because mods will link UE4SS by default, it will pull in a dynamic operator new symbol from there.. which will again try to use the game's version.. I have to manually link the mod without -lUE4SS for now..

If we've pulled in UEs operator new, we should probably remove that completely, we don't need it and it appears to be causing problems. @narknon Thoughts ?

Aha.. so it turns out that we are using ue's operator new but that's not causing problem. Instead, it's because the cppmod's header is using has_ui macro but mods are not getting that macro and this resulted in different class size and corrupted to the near by heap object..
Because it first write the tags so I thought it is mixing different allocators, which is not the case...

worked out a weird fix...
https://github.com/Yangff/RE-UE4SS/tree/linux-dlopen-loader
This will build a libLoader.so which (kinda like the proxy in windows) will be loaded by LD_PRELOAD and then, it will load libUE4SS.so with local and deepbind to prevent the symbol being overwritten by the game..
One problem here is, because mods will link UE4SS by default, it will pull in a dynamic operator new symbol from there.. which will again try to use the game's version.. I have to manually link the mod without -lUE4SS for now..

If we've pulled in UEs operator new, we should probably remove that completely, we don't need it and it appears to be causing problems. @narknon Thoughts ?

Aha.. so it turns out that we are using ue's operator new but that's not causing problem. Instead, it's because the cppmod's header is using has_ui macro but mods are not getting that macro and this resulted in different class size and corrupted to the near by heap object..
Because it first write the tags so I thought it is mixing different allocators, which is not the case...

Ah okay, that makes sense.
I didn't understand how that macro could cause this problem originally so I thought it was an unrelated problem, thanks for explaining.

The UI did some wrapping up and it looks like all the features are working now...
Before doing the last rebase and make sure it works on windows for review.. there is one more question I'd like to ask..
Inevitably there are some hardcoded numbers for TUI in the UI code now, which could conceivably be a problem for the mod. They would probably need to be compiled separately for GUI/TUI environments just because of the difference in those hardcoded numbers. It would also have implications for other people modifying the gui code afterwards. My idea is that a structural could be provided to abstract the differences between the different UI environments, such as minimum line heights and margins (which are also the main things that would need to be hardcoded). Another thing would be to change the handling of ICONs from macros to functions. I would like to know what you guys think about these issues?

Remind me why we can't use GenericPlatformString or UE's other platform abstractions over UEStringType? I understand we need to use certain strings before FString is found, but GenericPlatformString is just a type that we are compiling.

Remind me why we can't use GenericPlatformString or UE's other platform abstractions over UEStringType? I understand we need to use certain strings before FString is found, but GenericPlatformString is just a type that we are compiling.

What problem are you trying to solve with GenericPlatformString? As far as I can see it only has some convert functions and string compare. It's not a string container like any string type here..
uestringtype is a rename of std::u16string/wstring so work with stl is relatively easy in this case.. I don't see the GenericPlatformString seems only replace the codecvt. But you still have to convert it to uestringtype or say systemstringtype.

Well, I'd like to use the least amount of our own abstractions as possible. If UE does have an existing abstraction, that should be pulled in. And it seems odd to me that it would not give that it supports all platforms.
Essentially, the closer to UE and the less homebrew workarounds we use, the easier to maintain and add new platforms and we don't have to worry about making extra changes when pulling in other UE features.

Well, I'd like to use the least amount of our own abstractions as possible. If UE does have an existing abstraction, that should be pulled in. And it seems odd to me that it would not give that it supports all platforms.
Essentially, the closer to UE and the less homebrew workarounds we use, the easier to maintain and add new platforms and we don't have to worry about making extra changes when pulling in other UE features.

I agree but this should wait for another PR in the future as it's out of the scope of this PR and will just cause delays for Linux support.
We will need some conversions for Lua regardless but that's fine, and whatever string type we choose to use for system things (i.e. json and ini files) need to be considered.

I know this is a draft PR that you plan to move to another branch to finalize, but when doing so it should really be split into multiple PRs.  The string changes, for example, should be their own PR that is merged and tested in advance.

Worth review:
https://docs.unrealengine.com/4.27/en-US/ProgrammingAndScripting/ProgrammingWithCPP/UnrealArchitecture/StringHandling/CharacterEncoding/
https://github.com/Tencent/UnLua/blob/182efd9467939e26a733fda435d9cf525b1bf34e/Plugins/UnLua/Source/UnLua/Private/LuaEnv.cpp#L597

Worth review:
https://docs.unrealengine.com/4.27/en-US/ProgrammingAndScripting/ProgrammingWithCPP/UnrealArchitecture/StringHandling/CharacterEncoding/
https://github.com/Tencent/UnLua/blob/182efd9467939e26a733fda435d9cf525b1bf34e/Plugins/UnLua/Source/UnLua/Private/LuaEnv.cpp#L597

again, this is NOT the problem here.
unless you can remove all use of stl and all in UE, they make no sense.

I know this is a draft PR that you plan to move to another branch to finalize, but when doing so it should really be split into multiple PRs. The string changes, for example, should be their own PR that is merged and tested in advance.

yeah, I think it would be a string pr, linux without ui and input source + imgui but I'm not really sure how this will turn out as the history is a bit mess now..

yeah, I think it would be a string pr, linux without ui and input source + imgui but I'm not really sure how this will turn out as the history is a bit mess now..

Yeah, the commit history and scope of the PR are much too messy/significant for it to be appropriately reviewed and tested as it stands.  There are renames to classes in UEHeaderGenerator and other random things mixed in.

with that said, do you have any rebase tool recommendations? with rebase -i I can't quickly compare between versions. I may need to reorder, remove and split some of the commits, because for example the commits for resolving previous rebase conflicts and problems encountered when switching to clang18 etc. are in a newer position...

with that said, do you have any rebase tool recommendations? with rebase -i I can't quickly compare between versions (and the consequence of changes). I may need to reorder, remove and split some of the commits, because for example the commits for resolving previous rebase conflicts (and the problem it causes) and problems encountered when switching to clang18 etc. are in a newer position...
If it's just my commits my original plan is to just do a git diff and pick changes into new commits but since there are commits from others..

I wouldn't be surprised if @UE4SS didn't care about his commit history being preserved and if @localcc was OK with just a co-author commit, or not preserving at all, but up to them.

I wouldn't be surprised if @UE4SS didn't care about his commit history being preserved

That's correct, I don't care.

with that said, do you have any rebase tool recommendations? with rebase -i I can't quickly compare between versions (and the consequence of changes). I may need to reorder, remove and split some of the commits, because for example the commits for resolving previous rebase conflicts (and the problem it causes) and problems encountered when switching to clang18 etc. are in a newer position...
If it's just my commits my original plan is to just do a git diff and pick changes into new commits but since there are commits from others..

Do you care about preserving ALL commit history from this branch or would you prefer to be selective about what commit history remains?

for me I'd prefer just rewrite all history using the diff from base to HEAD and put changes with the same purpose into one commit and reorder them.. say
GUI: widht/height adjustment for terminal
Input: Generic Input Interface
String: type conversion
Linux: bootloader
Linux: UE platform specific

and if other people don't care then I can just do this..

for me I'd prefer just rewrite all history using the diff from base to HEAD and put changes with the same purpose into one commit and reorder them.. say
GUI: widht/height adjustment for terminal
Input: Generic Input Interface
String: type conversion
Linux: LD_PRELOAD
Linux: UE platform specific

and if other people don't care then I can just do this..

I think that would be the best solution as well, pending how CC wants to handle her commits.  However, there would need to be greater splits between the commits than the list there (I understand that is likely just an example).
For example, adding TUI and tweaking the existing GUI should be separate commits and likely separate PRs, changes to names of classes like HeaderGenerator/UEHeaderGenerator, changes to any functionality within classes should be their own, etc.

I'm alright with commit history being nuked, rewritten. I don't care about authorship getting preserved so do whatever makes it easier and nicer.

rebased to main again, I will try to create PRs and see how it works..

Try to PR some less platform-affected code first to avoid and overly long history that makes it impossible to ensure the correctness of the rebase..