Consider moving the buildsystem to Meson or another build system

Meson's website
I know, this is a bit of a hot topic, or at least a "nobody really wants to discuss it because nobody wants to actually do it" topic, so I'm putting massive emphasis here on the "consider" part. If the end result of this discussion is that Godot's build system shall stay the way it is, so be it.
So currently, Godot uses SCons, and I get why that is - it's a very powerful build system (since it's basically using Python to write build scripts), and the way it's setup in Godot makes it extremely easy to use. However, it's not without its downsides - primarily, that it's quite slow (being basically that SCons build files are Python scripts, and most Python interpreters do not particularly excel at execution speed), and because of its relative lack of popularity, difficult to find many resources or documentation about it.
Plus, I don't personally know how maintainable or easy-to-work-with the buildsystem code, but in my opinion it shouldn't be the sort of situation of "don't touch this code ever, or you risk breaking it and having to do boring and painful work just to get the project working again".
Now, I realize that Godot's project structure is not simple and that's a huge reason why SCons is what was chosen as the build system (and why there has been massive reluctance to move to another system), but I don't really believe that it's the only build system that can possibly satisify Godot's requirements. Feel free to prove me wrong, though.
So what could a migration to Meson offer?

Better baseline build speeds (and potentially that much better if unity or threaded builds are taken advantage of) [Some sample benchmarks]

The use of Meson's dependency system, wrap, would help reduce the repository size (third party dependencies would no longer have to be included with the project, and would certainly also help when said dependencies need to be updated to new versions. Use of wrap would even allow for modules (both official and custom) to be simply included as 'subprojects' (and offloaded to their own repositories), further reducing the main repo's size. [On wrap]

I don't know where SCons stands regarding this, but it seems like Meson is designed to play nicely with CI tools such as Travis and AppVeyor, which might be of note. [CI]

Meson also appears to some degree of support for localization, but it appears to be gettext based which I believe differs from the way Godot handles this, so I'm not sure how compatible this is here. [Localisation]

Right now Godot does not seem to make much use of unit tests (the godot-tests repo seems to be rather low-activity), but Meson has full support for performing such tests, which may be valuable. [Unit tests]

In addition to providing a way to generate the relevant files for both Visual Studio and XCode, Meson comes with an API that allows for integration with IDEs and other build tools, which again seems like something of value for those who don't use VS or XCode. [IDE integration]

This is all in addition to supporting (almost?) everything that Godot already uses and requires - cross-compilation support, source generation, running external commands, et cetera.
It's evidently a buildsystem that is capable of handling both small and big and complex projects - complex projects like systemd, GNOME, elementaryOS, Xorg, Mesa.
No promises, but I may be able to try and work on a proof-of-concept of this, to see how it fares. Also if there are any Meson experts/enthusiasts reading this, please feel free to chime in, as I probably got a few things wrong (both wrt Meson and SCons).
Cheers!

@mhilbrunner Nope it is still maintained https://github.com/bazelbuild/bazel

Their point was that just because google made it, doesn't mean it'll be around for a long time.

I'm not sure if requiring C# for compiling Godot is a good idea, considering that Godot can be compiled without C# support.

@Calinou you are probably right, although it seems everybody has jumped on the C# van-wagon, personally I'm using it for game scripting with Godot and productivity apps with Xamarin.Forms, also if I really want to squeeze performance I can try something like an IL2cpp transpiler or jump right into C++. I'm also thinking to use the mono/csharp REPL for things that I'd normally use JavaScript, Python or PHP for, like for instance web development. The thing is that I wouldn't mind to have a build system based on C# either, since I'd be already using it for pretty much everything.

I love CMake, and meson looks cool, but I tend to agree with this article: http://www.rojtberg.net/1481/do-not-use-meson.

I'm very interested in doing the footwork to move Godot to Meson. The only thing currently stopping me from contributing to Godot (and using it) is Scons. I had to use it years ago, and I've never had a more painful incremental development experience in my entire life.
The article linked above has a lot of strawman arguments, arguing that CMake already exists and we should just use that. Meson's build files are extremely easy to read, and use a syntax close to that of Python. Conversion can be quick. What I'm posting in this thread for is, if the footwork was done, would a pull request even be considered for merging?

What I'm posting in this thread for is, if the footwork was done, would a pull request even be considered for merging?

It will be considered, but it has to be proven advantageous over SCons.

Once you consider CMake just another build system and not the de facto standard, you end up with too many options. Why meson over SCons? Or bazel/buck are decent tools as well? But the biggest argument I have in favor of CMake is the tooling built to support that ecosystem. Clang compilation databases, support for packaging to multiple operating systems, etc. The only valid downsides I have heard is bad syntax and docs, but that isn't really strong enough for me to change my position on this.

Why meson over SCons?

Meson generates ninja files, which is much faster for incremental development. Additionally, on windows, you don't need to do any sort of visual studio path manipulation. Meson works out of the box. Using meson over scons would reduce a lot of the overhead of targeting windows, especially for those wanting to use visual studio (of which, I should note, meson can generate projects just as CMake)

Or bazel/buck are decent tools as well?

Given that bazel and buck are both written in Java (a dependency godot has only if you'll be targeting android, and even then one could technically use just the NDK), I would argue they're right out as a primary dependency for all users. Python is already in the dependency list, and you can easily install meson (and ninja) through a quick pip install meson

Clang compilation databases

Meson supports this by default, as this is a default feature built into ninja

support for packaging to multiple operating systems

Meson also has support for this, and relies heavily on pkg-config where appropriate

The only valid downsides I have heard is bad syntax and docs

Meson has a very python-like syntax, relying on variables, objects, and builtin functions. In my opinion, writing
my_meson_list = ['x', 'y', 'z']
message(my_meson_list)

vs
set(MY_CMAKE_LIST "x;y;z")
message(STATUS ${MY_CMAKE_LIST})

is a lot easier, especially given how cmake is sometimes case sensitive, and other times not. When you start to get into generator expressions, it can get extremely hairy.
# In one file
set(USE_STD_CXX11 "-std=c++11")
set(USE_STDLIB_LIBCXX "-stdlib=libc++")

# Possibly elsewhere
set(LIBCXX $<BOOL:${CAN_USE_STDLIB_LIBCXX}>,$<BOOL:${BUILD_WITH_LIBCXX}>)
set(NO_RTTI $<BOOL:${CAN_USE_NO_RTTI}>,$<BOOL:${DISABLE_RTTI}>)

# Later on...
target_compile_options(my-exe
  PUBLIC
  $<$<BOOL:${CAN_USE_STD_CXX11}>:${USE_STD_CXX11}>
  $<$<AND:${LIBCXX}>:${USE_STDLIB_LIBCXX}>
  $<$<AND:${NO_RTTI}>:${USE_NO_RTTI}>
  # Oh yeah, you're gonna want more of these, because you can't trust compiler interfaces
)

# Generator expressions mean you can't follow the flow of the build to see what is called and defined where. This is a completely valid use of CMake.
check_cxx_compiler_flag(${USE_STDLIB_LIBCXX} CAN_USE_STDLIB_LIBCXX)
check_cxx_compiler_flag(${USE_NO_RTTI} CAN_USE_NO_RTTI)

The equivalent meson looks like:
cxx = meson.get_compiler('cpp')
# This is an extremely simplified approach. One can do a different option when dealing with MSVC and gcc support.
args = compiler.get_supported_arguments('-std=c++11', '-stdlib=libc++')
add_project_arguments(args)

I think I should note, by the way, that of all the C++ build systems out there, Meson is the least terrible. Arguments against its use tend to be a picture of a web comic, some hand waving about how CMake is the most used, so you should just deal with it, and "hey at least CMake isn't autotools".
But above all (and I think this is a really important thing, especially for those using godot as for gdnative), Meson supports precompiled headers natively. CMake does not (and using cotire to hack the build can cause more problems than its worth).
I cannot tell you how much this could speed up the build for gdnative users, but it will be a significant boost on gcc and msvc (at the very least, a performance increase on msvc is inevitable). Add in the use of extern template for the very little amounts of templated code found in godot, and you could see decent build time improvements, which are important for iterative and incremental builds.

Meson generates ninja files, which is much faster for incremental development.

Note that CMake can also generate Ninja build files instead of traditional Makefiles; this is done by passing -G Ninja to its command line. It works quite well on most projects I've tried and is slightly faster.

Yes, however, CMake does not automatically install ninja for you, and it is not the default generator. Most of CMake's time is spent configuring the project, in my experience, and while recent versions of CMake have vastly improved performance, not everyone will have the latest CMake.

OK that is quite thorough! I'm glad you have a good idea of the features you value in a build system. I still think there ought to be a standard build system in the C++ world that projects can rely on. I just wish meson were a CMake transpiler as opposed to its own thing.

I still think there ought to be a standard build system in the C++ world that projects can rely on.

Trust me, no one wants this more than me. I've become sort of a build system trash goblin and even gave a talk on them at CppCon. I unfortunately (or fortunately?) care a lot about build systems, almost to the point of obsession. :)

I just want to say this, but every time I've did a long build of Godot on Windows, I've noted that Python consistently took 20% of my CPU. I doubt it's only slow to start.

Who's going to make a small prototype godot engine built with meson? I never got to the point of creating a small test for bazel.
A 2d only build of godot should be sufficient to test with all the options disabled.

The first roadblock I faced was core/make_binders.py. This is the autogenerated code that godot uses.

I made an initial prototype in the build system Bazel but I'll discuss the setup on another issue. https://github.com/godotengine/godot/issues/18518
Don't have time over the week, but others can expand on the Bazel BUILD.

Lol how did bazel get involved?

Since there might be a one time opportunity to switch to a different build system, shouldn't we looking for the best possible options? I for one wold be interested in seeing a Cake vs Bazel vs Scons comparison chart, although I have to admit I have zero experience with Cake to do the footwork for those tests.

Isn't that just for C#?

@isaachier I really don't know, all this time I was assuming Cake was capable of that, just like SCons being written in Python and having support for C++. Nevertheless, I just asked at their gitter chat, I'll re-post here the answer.

@rraallvv for the record I think the idea of a side-by-side comparison of major build systems is a great idea. Here is what I found online that I think are fairly unbiased (not written by the build system developers): https://carlosvin.github.io/posts/choosing-modern-cpp-stack, https://www.reddit.com/r/cpp/comments/6euc7b/build_systems_bazel_buck/die6g1y/, https://stackoverflow.com/a/12022652/1930331.
My personal opinion:

CMake: Pretty ugly language, but ubiquitous and practically the industry standard.
SCons: Slow, old, and IDK status. Uses Python.
meson: New, cool, Python-based build system that is not incredibly slow. Main disadvantage is trajectory. Unclear if this will just be another flavor of the week build system.
Autotools: Ancient, no tooling support (i.e. compilation databases for Clang autocomplete support or static analysis). No Windows support. Avoid at all costs.
bazel: A Google package-manager/build-system rolled into one. Doesn't add much over CMake with a plugin such as Hunter. Also, Google software has a tendency to focus on what Google needs, not the public at large. Too new to gauge long-term trajectory.

I try pushing CMake whenever I get a chance because I want a de facto standard for C++ and CMake has the biggest "market-share" right now.

I don't understand. C++ would probably be an awful build language. Slowness has more to do with the internal build system core AFAIK not with the details of the scripting language used.

@isaachier My guess is that SCons is so slow because it was written in Python, but if someone were to take the task of porting the entire build system to C++ and also make it JIT compile C++ scripts to run those builds, it would be orders of magnitude faster. Maybe the same could be true for every build system out there.

It's a nice idea but meson is not considered slow by most accounts, yet it is written in Python.

I completely agree @reduz that I dont really find SCons to be slow either.
As @slurps-mad-rips said:

Scons is slow not because it is written in python, but because it also executes the build

man scons:

scons supports building multiple targets in parallel via a -j option that takes, as its argument, the number of simultaneous tasks that may be spawned:
scons -j 4

I have a CPU with 8c16t, so use "scons p=x11 -j 18" is very fast for me.
It's 10x faster than use "scons p=x11" as the default setting "-j 1".
Do someone have a single core CPU at year 2018?
Let your CPU working fully.
please give it a try.

Not everyone has these large CPUs, but more importantly, meson (well, ninja) is much faster at incremental recompilations. Scons, by default, performs an MD5 hash sum to see if a file has changed. Godot uses the MD5-timestamp setting, which means it will only hash a file if a timestamp has changed.

I guess the best way to prove what is better is to write a new one and see :)

Scons isn't fast for many people, especially windows users running into problem with anything above -j 1.

@slurps-mad-rips it depends on the project. ninja seems to hype its speed a little too much. See this article: http://david.rothlis.net/ninja-benchmark/. Seems counting source files is your best bet here. If <20K won't make much of a difference.

@isaachier that benchmark is between make and ninja, not ninja and scons, so you'll have to forgive me if I don't want to pay attention to it. ;)

I guess the best way to prove what is better is to write a new on and see :)

I agree, there is no point in bike shedding. If you have an opinion on a better build system, implement it and prove it's better. Otherwise this is a discussion of opinion and will lead nowhere. @fire has already done a partial test with bazel: https://github.com/godotengine/godot/issues/18518.

@slurps-mad-rips ya totally different topic, not SCons. I just meant make vs. ninja isn't crazy difference.

Slight update: I'm giving up on trying to move to meson. It's been piecemeal and while the build files are slightly easier to read, staying up to date with the current system is where most of my (now rare and in short supply) free time is spent, instead of moving scons to meson.
Right now, a lot of code and operations are placed directly in the scons scripts and this makes moving to another build system more difficult. If anything I'd argue time should be spent decoupling the existing python scripts that generate code from scons, as this would make a move to another build system like meson, cmake, or even bazel a hell of a lot easier.

Good thing a patch was just merged to move all the code generation out. https://github.com/godotengine/godot/pull/17595 Don't give up!

SCons Project Co-Manager here.  A couple notes:
1 - SCons is under active development and since our migration to Github we're seeing an increased rate of pull requests.
2 - We've identified a few specific parts of SCons which are responsible for about 50% of the null incremental build time and are actively working on addressing that. (The Subst functionality)
3 - We've started supporting Python 3.5+ (as well as 2.7.x) with SCons 3.0
4 - Let me suggest that instead of attempting to port to N different build systems, contributing some time to help improve SCons might be a more effective use of developer time
5 - We identified and addressed via profiling some of the low hanging fruit in improving performance which was released in 3.0.0 and 3.0.1. If you're using an earlier version the null incremental build can be 5-15% slower. (assuming Python 2.7.x)
As always the SCons project is ready and willing to help projects who use SCons with issues they run into.

Some "stats" for SCons & godot.
I used (and still do) a Q6600 which is a 4 core cpu from 2008 or so with 2.4GHZ (and a lower instructions per cycle count) than the processors  we are "all" using today (i7s).
On Q6600, a built took between 3 to 5 minutes (or more, but exact num not that important), out of which ~35-40 seconds was spent by SCons parsing the tree of dependencies and stuff (the prep work)... so, 80% of the time spent running cl.exe (MSVC compiler) process over and over again.
So, performance & optimisation wise, unless SCons (python really, not SCons) has a terrible process calling system that is taking more time than it should or some other tools is not compiling one file per process (like SCons currently is) and the 1 process per one file overhead is significant (I'm not an expert on cl.exe compliation internals & startup overheads), the only performance optimisation a faster build system would produce is probably in the building of dependency tree phase, and other prepwork for complication.
So, we are optimizing those 35-40 seconds out of a 5min build (on i7's now we are probably optimising a 20 second single core setup step out of a 3 min multi core build).
So, to recap, if i remember correctly, 3 to 5 min is actually a 4 core multicore build, while a single core build takes around 10? I need to rerun my tests...
So, "optimizing" SCons away means more or less optimizing those first 35 seconds of the builds start...
That's as far as "optimizations" are concerned when it comes to SCons vs other faster build system... It's possible that the actual compilation could be optimised a bit by avoiding calling a single cl.exe for every obj file, but that's only if the process overhead is significant (I fear HDD will always be the actual IO bottleneck there (and not the CPU), which is not avoidable/mitigatable without an SSD)
All of these paragraphs above consider a "full build". That said, SCons "cheats" if everything is already built and passes it's heavy handed consistency checks so if you change a single file it will actually compile a "single file" and then link everything back in, making a "full build" into somewhat of an incremental build). This however requires those 35-40 seconds of full dependency tree precomputation...
So, an "incremental build" is around 45 seconds on my old computer, and I thought when I was a noob 2 years ago that it would be CPU bound, but actually, it might be IO bound (checking several thousand files on the hdd) for a full dependency sweep/tree rebuild...  So, if it IS IO bound, no "faster buidlsystem" is going to solve that one...
My hackish idea from 2 years ago was to try to attach a filesystem "watcher" to SCons, have SCons run & recompile a single changed file (and it's dependencies, that's automatic), and then just have it link everything back again... although I realise today that linking would still probably run a full dependency sweep... that said, the dependency tree sweep/rebuilding can be killed with an option and then SCons will use it's cached dep tree... which saves around 15-20 seconds of the 35 sec precomputation step (the first 15 seconds seem unavoidable) although that's might not guarantee a "perfect" build always like SCons does (though that might not be important for incremental builds in dev... if you decide that speed is a worthy tradeoff).
I now have enough knowledge that I could probably hack it through npm and npm watch... and or other system that I'm not currently familiar with... But that's for later, and when I have time (not currently).
This is getting a bit long so I'll end it here, but the point of these paragraphs was to give you some "statistical" information rather then to discourage anyone (go ahead if you like working on build systems), like I myself liked fixing some stuff in Godot's SCons. I hope some information was useful for you guys here.
If you wish to replicate/redo my statistics, just browse through SCons docs till you that turn on statistical info (timers stuff) and/or find a way to turn off dependency redolution (those first 20 to 30-35 seconds of "precomputation")...
Or for a slightly less manual solution, timer debugging info should already be present in the automated VS project generated by SCons (at the end of SConstruct file)... you can find info on how to generate a VS Project in the "Windows compilation tutorial" on the godot website. You can then run SCons from VS, I believe the generated project should still probably work, though I have not tested it in Godot 3... It's an option, but tailing scons to a > log.txt is also another option (etc)...
Here's hoping somebody found this info helpful.

@Griefchief Just to clear things up a bit, I think we aren't trying to optimize full-build times, since those are dominated by compiler time anyway. Instead, we want to optimize incremental builds where only a single source file is changed.

Indeed the "null incremental build" (nothing actually needs to be rebuilt, but scons will still need to process the whole dependency tree to ensure that) is the case we're working on.
Some profiling has pointed out some inefficiencies in processing the command lines (which have to be done for every target to ensure that they have not changed) have been identified.  The bulk of that is the Subst() logic.  This is some tricky code and thus far has been implemented in a simpler and safe, but not terrible efficiently way.
Another hot spot of time is the reading and writing of the sconsign files, currently implemented with cpickle and always read and written as a whole. We have some ideas on how to either speed up and/or make more incremental.  Anyway watch future releases for improvements.
On a side note, you might try MSVC_BATCH to see if that speeds up your windows builds:

MSVC_BATCH
When set to any true value, specifies that SCons should batch compilation of object files when calling the Microsoft Visual C/C++ compiler. All compilations of source files from the same source directory that generate target files in a same output directory and were configured in SCons using the same construction environment will be built in a single call to the compiler. Only source files that have changed since their object files were built will be passed to each compiler invocation (via the $CHANGED_SOURCES construction variable). Any compilations where the object (target) file base name (minus the .obj) does not match the source file base name will be compiled separately.

@bojidar-bg yes, yes, I was actually talking about incremental builds as well, thank you for the tip :D
@bdbaddog Hi baddog, just for your info, I actually wrote this big comment of mine before I saw your comment, so, it doesn't actually refer to anything in you comment specifically... but you did manage mention some of the "concerns" I had (information for other people really) while we were both writing our comments at the same time...
And I didn't want to ping you in it cause I didn't wanna "waste" your time :D, but I guess I will if spend some more time on godot build/scons in the future (I do plan to, i'm just very busy))
Thank you for your input here!

@bdbaddog BD, since you're providing support, if I may ask here:
If I attach npm watch to the filesystem (godot's source) and know which exact file has been changed, would that bring me any benefits with SCons?
Now I can run scons through npm watch and push the file to it...
scons [godot options] the/file/that/was/changed.
this will obviously recompile that file, probably avoiding Godot's whole dep tree? correct? So now dep tree traversal won't take 20 sec, but less then 1 prob, but I get 1 updated .obj file (or more, depends on what you change). It won't actually go through the whole godot dep tree for that one file? I do have an advantage in less dep tree traversal in this example (and possibly other advantages in the precompute phase that can not be gained through reusing the dep tree cache, like less files overall to go through?)?
Now, if I do have an advantage, and if I wanted to just link everything again, would I need a full tree traversal for a "simple" linking phase?
I understand this usecase might not be currently supported by Scons (just link everything again), but I'm asking both from a practical and theoretical standpoint? I realize that cached dep tree could be reused with that "don't do dep tree" option (I forgot it, it's been two years), and that would work for some usecases, and that usecase is no new files added in, correct? (npm watch could alert me to new file addition and I could then do a full dep tree refresh if I detect that... in other words, kind of push the "ensure dep tree consistency" back to a filewatcher like npm watch, automate typing in that option by the user to a program like npm watch)
Is there something in my idea that would not work as an optimisation(practically and theoretically), and do you have other suggestions (that do have to guarantee a perfect build if something like npm watch can take care of stuff, it's fine as well)? Can you advise please? Thank you!

To recap, the usecase is similar to this:

run a filewatcher like npm watch on godot's source directory
user saves/modifies an old file
scons is immediately run for that file by npm.
if compilation is successful, npm instructs scons to run the linker and link the executable.

4.1) if dep tree is cashed (a full build was already run), npm can instruct the linker to just link at this point with a cached version of dep tree
4.1) if no cached dep tree is detected, npm watch runs a full godot build
4.2) if cache invalidation was detected, like adding a new file, perform a full dep tree traversal (npm won't use the "use cached dep tree" option with scons).
I hope this makes my idea a bit easier to understand, please let me know what's wrong with it at this point ;D

So a few things here (in no particular order, but I'm using numbers because yolo

Incremental null builds are basically nothing under any tool that generates ninja files. Chrome has, AFAIK, a less than 2 second null incremental build. This is of important note, as a 35 second drop to 15 under an npm watch is still very large compared to a meson or cmake based Ninja build. Ninja also does the same thing that Scons does in that it does perform a rebuild in the event that the command line changes (make, of course, does not do this)

CMake recently implemented the ability to use CONFIGURE_DEPENDS as an argument to the glob system. This has always been recommended against because tools like xcodebuild and msbuild have historically (and presently) not supported directory change detection. This is, afaik, one thing that Scons could improve upon, but I have neither the time nor patience to deep dive into the project to implement this. Basically, every OS updates a directory when its contents change, be it an added file, modified file, or removed file. When making incremental changes, one can simply check which directories have changed and glob only those on a reconfigure check. This can reduce work done as you're not globbing the entire tree, but smaller bits. For small projects this is fine. Whether this would benefit godot or not would require some testing and that's a lot of work to check if it would even help.

While it's nice to see that Scons is receiving more attention, I personally think it is better for the community at large for godot to move to a more used build system. I cannot think outside of godot of any large project that uses Scons. GNOME/GTK recently moved to meson (skipping CMake), KDE has been on CMake for quite some time. While it's anecdotal evidence (and thus a basic confirmation bias), I know of several C++ developers who would like to try godot but would rather not have to touch scons ever again. Removing the nasty experiences and thus the sour taste left in developers mouths is difficult, and I do honestly wish the Scons project the best. But I do know that the C++ community at large already hates long compile time. Long incremental null builds (or even long incremental builds of 1 file) are a nightmare in the modern world.

I do intend to try again, but I might target CMake for this. Moving from CMake to meson is a hell of a lot easier thanks to meson having a conversion script that can do a lot of work. CMake might make handling dependencies a bit easier with the new FetchContent module. We'll see how it works out. I will say, I'm glad these codegen scripts were moved to separate files. Using CMake will be nice if only because moving to a different C++ standard is as simple as target_compile_features(<target> cxx_std_<number>)
Obviously any build system related stuff is going to be contentious, but trying can't hurt.
To be quite honest, (and this might come off as callous) I have no desire to deep dive into the Scons project to improve it. Maybe after 2020 when Python 2.7 is officially finally really dead and the project can move forward to Python 3.5+ only it'll be worth investigating or improving, possibly with async operations. Until then, however, I'd rather not have to touch it.

Maybe after 2020 when Python 2.7 is officially finally really dead and the project can move forward to Python 3.5+ only it'll be worth investigating or improving

Scons 3 does support Python 3.5+ : https://scons.org/tag/releases.html .
I don't really understand how one can dislike Scons so much and like the configuration mess that is CMake, where it's even hard to define your own build settings if you are a user, hard to understand what are the options and their types and what are internal defines, it only brought pain to me, and I was actually relieved that Godot didn't use it.
In any case, if you feel like it's going to be a better solution go on and try, I might end up changing my view on CMake.

where it's even hard to define your own build settings if you are a user,

If that is asked then CMake is being used wrong.  All toolchain settings should be in a toolchain file and any project specific configurations are passed via flags (or gui or edit the cache file or whatever).  Everything is documented and typed, even the project authors can well document and type their exposed configuration.  You should read CGold and the modern CMake standards.

@Faless no one denies the CMake language is terrible. But the implementation is best in class. I've been considering the possibility of writing a transpiler from a better language to alleviate the issue.

@OvermindDL1 love the suggestion to read CGold.

If that is asked then CMake is being used wrong

Well I guess that all the programs I encountered where doing it wrong then.
To be honest, I hate having to change flags via -DI_LOVE_CMAKE_AND_MAKE_DEFINES_LONG=on (really? on?). Not to mention the idea of editing the cache file O_o . Or that, if I want to change a flag, I basically end up having to delete the build folder and start from scratch (because, you know, cache differs!)... so much for incremental builds...

95% of build files I have seen on GitHub are wrong. People get lazy about build systems and follow cargo cult practices to cobble something together.

@isaachier , just as a metric, could you provide 2 examples of a bad and good CMake file? How would you consider this: https://github.com/ARMmbed/mbedtls/blob/development/CMakeLists.txt ?

Better than most I've seen. Still, if you are not using CMake >= 3.0, then your experience with is not with modern CMake.

Well I guess that all the programs I encountered where doing it wrong then.

I tend to see it done well nowadays.

To be honest, I hate having to change flags via -DI_LOVE_CMAKE_AND_MAKE_DEFINES_LONG=on (really? on?). Not to mention the idea of editing the cache file O_o . Or that, if I want to change a flag, I basically end up having to delete the build folder and start from scratch (because, you know, cache differs!)... so much for incremental builds...

You don't have to use on, you can use true, 1, and a variety of other things.  It is to make it so you can choose an appropriate boolean name that flows well with flag name.
And you should not ever need to rebuild the build folder if you change options, cmake is exceptionally good at rebuilding when necessary.
Still though, everyone touching C/C++ build systems should read CGold as well as the official manual of CMake.  It is really easy to use CMake right, it is often a great deal shorter as well.  The ancient CMake2 methodologies should not be being used.  Really though, anyone that depends on a CMake 2.x version is absolutely doing things wrong (not using targets and toolchains properly, not setting options properly, etc... etc...).

Thanks for the explanations, while I remain sceptical, I'm looking forward to see how it would work for Godot :).

@slurps-mad-rips Your post with the supposed CMake example code for enabling C++11 mode is entirely non-standard CMake for a variety of reasons, including but not limited to:

Setting unexposed variables.
Toolchain arguments set in a non-toolchain file.
Absolutely not how you do something like enabled C++11.
Inefficient.

Given the shown self-described-as-simplified meson code in that post of:
cxx = meson.get_compiler('cpp')
# This is an extremely simplified approach. One can do a different option when dealing with MSVC and gcc support.
args = compiler.get_supported_arguments('-std=c++11', '-stdlib=libc++')
add_project_arguments(args)

The equivalant simplified (in that you can add more properties as well) CMake would be:
set_target_properties(godot PROPERTIES CXX_STANDARD 11)

And that will make sure that the toolchain used knows that it needs to set that mode (and you can specify how to handle failures if using a compiler that doesn't support it as well).  Assuming special compiler-specific command line flags in the build file is extremely poor form as those should only exist in toolchain files and the fact the meson example shown in the prior post actually specifies specific arguments essentially means you are hardcoding the supported compilers.
Considering there is a pre-built CMake Toolchain file for just about any toolchain you can think of (including all the standards of MSVC of various varieties, GCC/Clang, android, iOS, emscripten, etc... etc...) you should not be re-inventing the wheel.

@OvermindDL1
It is not non-standard CMake in the absolute slightest. The code sample targets CMake 3.0, a version in which CXX_STANDARD does not exist. You can easily see from here that you use target_compile_features with cxx_std_ and the number of the standard. This is the new and modern cmake way to set the standard version. set_target_properties is an older per-target approach.
In fact, the exact code taken from CMake's documentation is
target_compile_features(mylib PUBLIC cxx_std_11)

However, CXX_STANDARD does not set libc++ support for clang as clang uses libstdc++ by default on linux unless configured (and it's never configured to use libc++ by default on most linux distributions, as this would cause linker errors and ABI issues). The only way to do it at this time is to check if the compiler supports -stdlib=libc++ and pass it to the compiler via a generator expression. There is nothing involving unexposed variables regarding this.
Furthermore, I do not appreciate you dropping a comment to make corrections from a comment I made months ago, especially given that I will be targeting CMake over Meson in my next attempt. It lacks tact, and in my opinion adds nothing to the conversation.

CMake 3.0 is old enough to no longer be supported (June 10th, 2014).  No one should be running anything less than CMake 3.10 currently, though staying up to date with CMake 3.12 is preferred at the current moment.
And yes, the example you gave is entirely recommended against (I think even CGold mentions not doing that).
And yes, I did mean target_compile_features.  ^.^;

However, CXX_STANDARD does not set libc++ support for clang as clang uses libstdc++ by default on linux unless configured (and it's never configured to use libc++ by default on most linux distributions, as this would cause linker errors and ABI issues). The only way to do it at this time is to check if the compiler supports -stdlib=libc++ and pass it to the compiler via a generator expression. There is nothing involving unexposed variables regarding this.

The current toolchains all do that for clang just fine, at least with C++14 as that is what I compile with (and a couple C++17's).

Furthermore, I do not appreciate you dropping a comment to make corrections from a comment I made months ago, especially given that I will be targeting CMake over Meson in my next attempt. It lacks tact, and in my opinion adds nothing to the conversation.

I apologize, did not notice the date as I was reading the backlog (and there is a lot of backlog here).  I just noticed something that appeared woefully wrong for any kind of modern suggestions from what I've seen for a few years now and did not see any corrections in posts shortly after thus wanted the incorrect information to not propagate.  :-)

The only way to do it at this time is to check if the compiler supports -stdlib=libc++ and pass it to the compiler via a generator expression.

However yes, that is the job of the toolchain file, that should absolutely not ever appear in a build file, ever.

There is nothing involving unexposed variables regarding this.

I was just referencing the variables that are set then used elsewhere in only single locations, that is recommended against in CGold among others for a variety of reasons.

CMake 3.0 is old enough to no longer be supported (June 10th, 2014). No one should be running anything less than CMake 3.10 currently, though staying up to date with CMake 3.12 is preferred at the current moment.

They shouldn't, and yet some places are. For a game engine that tries to stay up to date and with the latest technology, staying on latest CMake is fine. However I am more than well aware of extremely active projects stuck on older versions such as 3.5 and in some cases 3.4. In fact, some well known C libraries still target CMake 2.8 (See: SDL2, libssh2, libgit2, etc.)

However yes, that is the job of the toolchain file, that should absolutely not ever appear in a build file, ever.

Making this assumption (that toolchain files are well written and cover all possible systems and all versions of CMake) is asking for trouble in the current general C++ ecosystem. When using clang, the option of compiling for libstdc++ or libc++ (especially when used via a FetchContent call) needs to be provided by the library. CheckIncludeCXX and CheckCXXCompileFlag are still necessary for even some flags that don't exist under GCC but do exist under Clang (and vice versa). The primary issue here is that compiler interfaces have diverged a large amount and instead of vendors spending time to facilitate compatibility options or even discussing a common interface, that behavior is thrust onto us, the developer when all we want to do is write code. Trust me, no one on this planet is more upset about the state of C and C++ build systems and dependency management than me, but pragmatism for existing libraries on CMake is needed. The nice thing about moving from a build system to CMake is that we can start out the gate with the latest and greatest features. The problem becomes stagnation of the build system because no one wants to touch it (except for that brave soul who kicked the code generation build scripts into separate scripts), but some work would be needed to make sure it doesn't get out of hand (which it will because... I mean it's a build system. They always do)

I must be the one guy who actually enjoys writing build scripts especially in CMake ;).

I've used a lot of different build systems over the years. The one that made most sense for me recently is CMake, but I can definately see the appeal of SCons and how flexible it is.
Even if SCons isn't perfect (what build system is?) it may be good enough.

Does it provide the required features?
How much overhead does it create?

If SCons scores "good enough" for both points then keep it.

Has any of you proposing the replacement of SCons by something else actually taken the effort to learn how Godot build system works?
As a hint, it's incredibly complex and I don't think there are many (or any) projects using CMake building for this many amount of platforms and allowing to have several builds targets configured/compiled at the same time the way we do.
All efforts by those who said "I will port Godot build system to somehting else" failed miserably so far when realizing the incredible complexity that SCons is taking care of.
To me, it's hands down the best tool for the job by far. Nothing is even close to it. If you guys complain about base compilation time (which is just 4/5 seconds on a medium to high end system with SSD), you really should first try to understand everything Godot does when building and see how it would perform in CMake or anything else.

I did take a look. I am not convinced it is undoable in CMake or elsewhere. But I'm also lazy ;). We'll see what happens.

As someone who did the bazel port. I was able to get to the point where the Godot editor started without icons.
I'm pretty sure CMake can reach that point.

I reproduced my bazel result on cmake.
https://github.com/fire/godot/tree/cmake
Assume visual studio 2017 is installed.
git clone https://github.com/fire/godot.git -b cmake
scons p=windows
Modify platform/register_platform_apis.gen.cpp
#include "register_platform_apis.h"

void register_platform_apis() {
}

void unregister_platform_apis() {
}

Install bazel
choco install cmake ninja -y
# Open visual studio command prompt amd 64 2017 native
# Go to godot source directory
cd ..
mkdir build
cd build
cmake .. -GNinja
ninja

Please play with it. The editor has the same problems as the bazel build (no icons), but you may find this sample helpful.

@fire Thanks for this. I see a lot of places for improvement that would make maintenance of these CMake files easier (and time spent adding new features or compiler flags). I might tackle this next weekend however, schedule pending.

Awesome work @fire. Glad to see someone here can produce something in more than just one build system :smile:.

Slight update. I had some downtime today to tackle building off of @fire's work. Unfortunately, the code generation scripts are still a little too embedded with the current Scons system and some work is needed to fully extricate them. That said, it'll probably be a separate issue + pull request. I did identify a few areas where an external template file (or series of template files) could be used. Python has support for template strings and this might make life a bit easier, instead of having to process strings, add them to a list, and then process more strings, and add them to a list, etc. etc.
Ideally, the code generation scripts should be able to be run as if they were executables without knowledge of the build. As long as the proper files were passed to them, we'd be good to go for fully porting to any other build system.

@slurps-mad-rips  Extricating that from your SCons system is a good idea even in SCons.  You can hang up the GIL by having too much logic in process and hurt your build performance.  Occasionally it's lead to some file close/open race conditions when building in parallel (though often resolved by opening the file with in a context).

Hello all,
I'm going to use this space to keep people updated on the port to CMake. If requested, I'll make a separate issue and keep track of changes there, though this is a nice area given it's got a massive backlog of discussion and I think it'd be a waste to lose so much context.
I digress. I'm going to have to take a break from the CMake port for the next week while I attend a conference, however you can see the efforts of my initial porting attempt here. Please keep in mind that I'm going to be shooting to stay 1:1 with the current master branch, and as such I'll be force pushing changes as they come in so I can keep a rebase. As I approach stability, I'll file a pull request to keep rebases easier to work with, as well as to make it easier to comment on changes or ask questions. I think this port to CMake is not only doable, but at the very least it'll make one more person (me) well versed in how Godot is actually built and constructed and I might be able to put that into words so others can get a bird's eye view of the build system. This port also might possibly make everyone's workflow easier in the long run, especially with releases. I'm not gonna lie, having the CONFIGURE_DEPENDS globbing of files mixed with Ninja is actually quite nice.
One last thing, I highly suggest you all take a peak at the helper (CMake Modules](https://github.com/slurps-mad-rips/godot/tree/master/cmake/modules) I've written just because if you are writing other CMake project's, these modules might come in handy.

Were you able to update to the latest 3.1 alpha? The alpha could be a good target. Currently trying the build.

This issue's title is Consider moving the buildsystem to Meson. I think it is inappropriate to have CMake discussion here. Discussion about CMake should have it's own issue. Comparison of Meson vs anything else is fine.

@zaniar There's been a lot of discussion in this thread regarding build systems in general. Just because the title hasn't been updated doesn't mean we're going to suddenly create a new issue and lose all that context for discussion and start all over again. I think asking us to suddenly throw away several months worth of discussions is inappropriate.
@fire I've been busy with work the last few weeks, I've got a large deadline for Friday, and then after that I might have some time to get my fork up to date. I've got a few changes locally I forgot to push up but I don't have the time right now. (Between this build system port, my fork of gnu m4, 15 proposals I'm writing for the C++ standards San Diego meeting and CppCon in the last week of the month, I'm absolutely swamped, and I have no one to blame but myself)

Yeah, Meson was the original discussion target, but I think it's kind of natural that other build systems have been brought up and discussed since, and it probably makes sense to keep the discussion within this issue here instead of spreading it out, like @slurps-mad-rips said.

I agree with moving to CMake. It has a lot of support with its toolchains. Android builds for example become very easy compared to what Godot is doing in its SCsub file which looks 'unmaintainable' to me.
As an open source project, Godot should go with solutions that are easy to maintain and that contributors are more likely to know and be able to contribute to.
Also, I think this is important for GDNative adoption. Whichever build tool is used, there needs to be an example or standard project with easily modifiable build script that supports building for all platforms, including iOS, Android, etc.
Also normal modules should be able to be compiled as shared libraries. This is related to #19486 so that the engine can be separated into multiple downloadable components.

I'm actually tackling these. I haven't pushed my changes up in a while because I've been busy with CppCon, the upcoming C++ standards meeting, and we're doing a release next week at work. Suffice to say, I've not had much time to finish up the port, but most of my time is going to be spent ripping the python scripts that generate C++ code out going forward. I'm hoping to resume work on my port this weekend. I'm currently evaluating whether to automatically acquire third party dependencies and apply patches stored in the repo or not. One question I do have for the core developers is how open they would be to breaking all the third_party repos out of the main repo and into their own repositories on the organization. I'm relying fairly heavily on FetchContent in my own projects, and I'm currently wishing I could do that for these third party dependencies that have been slightly forked. This would also let me work on each one in turn to clean up their cmake files while also doing incremental changes to the core repo once each third party library is ported.
Additionally  should note, is that I'm trying to also modularize the engine itself as well, so these components could technically be updated separate from each other if someone wanted it. This might knock out the issue that @chanon mentioned.
Lastly, I'm adding support for a few additional build tools for the core developers if they're found on the system. Things like clang-format, clang-tidy, ccache or sccache, distcc, clang-check, address sanitizer, ubsanitizer, thread sanitizer, etc. These are mostly linters (and might make some people's heads explode with the warnings and various errors they've been detecting) but they'll be optional to enable. At the very least using sccache or ccache will give some build improvement when switching between builds and vice versa. Waiting for CMake 3.13 to be released would be nice as this would solve some of the kludge-work I've had to rely on, but the important thing is that I'm setting it up so that the current "getting started" workflow is to still run pip install via Python 3.5 or later, however it'll be inside of a virtualenv so people don't have to mess with their system's environment. (This would also allow others to experiment with using additional python libraries as dependencies for various code generation scripts and not have to remember to uninstall them later)
Anyways, sorry for the update sudden info dump y'all. I'm dedicated to getting this port finished, but life has gotten in the way recently 😅

I'm relying fairly heavily on FetchContent in my own projects, and I'm currently wishing I could do that for these third party dependencies that have been slightly forked.

You could also use Hunter and make a godot-specific code repository (a special git repo, like on github for example) and access all the dependencies that way as it handles the building, cache'ing, etc... as necessary and properly (or submit all the dependencies to the 'main' Hunter package repo and just use it straight).  Hunter is just a single cmake file (HunterGate.cmake) and a call to it to acquire the package repository information (whether the official one or a custom one).

A humble request: this issue's comment thread has gotten huge - would someone who's been following it be able to summarize the general discussion-so-far in a comment (and maybe it could be linked to from @RiverMesa's initial post)?

Hello everyone. I am planning to use Godot as my next engine for making a small game.
I saw this conversation and as a person who has experience in both CMake and Meson (and autotools, tup, plain make, waf, SCons, trying full projects in all of these tools...) I wanted to give my feedback.
The systems I used more intensively have been Autotools, CMake and Meson. I have been using Meson and CMake for the past years, since I already discarded all the other build systems in production setups.

Biggest pros of cmake

Mature project generators if you care about Visual Studio (I think Meson does support it well nowadays but did not try lately) and especially XCode (pretty sure it is more lacking here).
Wider adoption, more support in IDEs

Pros of Meson

many useful targets by default: sanitizers, unity builds for free, precompiled headers for free
the documentation kicks the ass out of CMake documentation: http://mesonbuild.com/
I found cross compilation to be much easier
Meson has an obvious way to do each thing

Which one would I choose? I would choose Meson if cross-compilation is going to be intensive and you care about fast compile-times (unity builds and precompiled headers are supported by default).
I would consider CMake since it is the standard and contributions maybe are important for the team. I would choose myself Meson personally, but that is just a personal opinion, since I find the tool just better and that saves my own time (CMake scripting, space handling, generator expressions, etc... I am looking at you).
I have a small series of articles about more or less basic Meson here if anyone is curious (4 articles):

https://medium.com/@germandiagogomez/getting-started-with-meson-build-system-and-c-83270f444bee
https://medium.com/@germandiagogomez/getting-started-with-meson-in-c-part-2-58150354ff17
https://medium.com/@germandiagogomez/getting-started-with-meson-in-c-part-3-70b9bc419957
https://medium.com/@germandiagogomez/getting-started-with-meson-part-4-8bceec6149e1

I have a (bit outdated) reply on build systems in StackOverflow:
https://stackoverflow.com/questions/5837764/autotools-vs-cmake-for-both-windows-and-linux-compilation/24953691#24953691
Just my two cents :)

@germandiago That is a great writeup -- I've been following this issue for a while and it would be wonderful if I could work with godot + CLion, which is sadly CMake only.

Yeah most modern IDE's are either CMake-only, and/or some Custom Build System, and/or 'raw calls' that lose a lot of features.  CMake really has become standard, whether it is merited or not.

@dorkbox if I can get these changes I'm working on in, you'd definitely be able to work in a CLion (cmake + ninja) workflow.
@germandiago
I'm actually friends now (more or less 😛), with the author of meson as of CppCon 2018. We both ended up on the same paper for the upcoming meeting regarding modules titled "Remember the Fortran", and I'll be presenting his paper at said meeting (It's next week, actually!), and we also were on the same build systems panel at CppCon.
I would love to use Meson. If you read further up in the thread, I had tried a meson port, but things were too integrated with Scons at the time. I'm still pulling out the current code generation scripts to be separate 'executable' scripts so that if, in the future, a move to Meson would not be out of the question. However it's slow work, and I have taken to writing a few jinja templates to help with the code generation so scons can be dropped entirely. That said, while my public fork is out of date, the one I have at home is a bit further ahead (and synced to a private fork). Work has unfortunately gotten in the way, and San Diego is unfortunately going to be 'stealing' more of my time. After that, however, I should be free to remove some of the local experiments I have on my private fork and finally get back on the godot cmake train. I am committed to making sure a full pull request can go in without also having a massive disruption in the current workflow process.
(As an aside: thanks everyone for the patience you've had while I work on this. I know it's been a while since I started it, but I represent my employer for the US National Body for the C++ standards committee now so my schedule has been a bit more full lately, especially as we hit important topics like modules, coroutines, and ranges)

All and every build system have its major flaws, more than that, there is no reason to change that from technical point of vew, be it scons, autotools, cmake, meson, or whatever make vs ninja your religion is. I think it would be great to close all other build system change bugs as duplicate of this one.

<reduz> iFire: no, you did not, all you did was build it but did not port any of the dozens of custom scripts that generate code
<iFire> reduz: the other person worked on the little python script generation on her branch
<iFire> so they directly imported the python env
<iFire> it's workable
<reduz> iFire: that is what everyone said that attempted it, and failed :P
<reduz> there are too many things in there

Proof reduz wrong @slurps-mad-rips :).

800lb gorilla in the room that hasn't been mentioned yet. One of Godot's claims to fame is cross platform compatibility. It should follow that whatever build system that is a potential for migrating to should be available and tested on all platforms that Godot targets. Whoever is behind such a migration should be inclined to test the migration on all supported platforms, or at least be part of a team that is doing so. Otherwise I would see such a move as contrary to Godot philosophy and generally just a bad idea.

@lazybullfrog I have access to about 5 separate machines for desktop, and several mobile devices. Part of my port to cmake is making sure there are cmake toolchains available for all the current platforms.

@slurps-mad-rips Is one of those Haiku? If not, I'd be willing to help test. Thanks.

Part of my port to cmake is making sure there are cmake toolchains available for all the current platforms.

Don't forget as well that if you need cmake toolchains for a platform then there is probably a prebuilt one at https://github.com/ruslo/polly and if there is not then you should PR it.  :-)

@slurps-mad-rips How has it been going? I noticed you updated the code on Jan 14th. I tried compiling on Windows but it didn't find pkgconfig.

@fire life decided to throw a few curveballs my way and I had to take care of them. Personal stuff, unfortunately. I've been focusing on work and finishing the IXM library that I'm using to reduce the amount of work that would be copied into godot. Luckily I'm taking a vacation in a week (I'm heading to Kona for the WG21 ISO meeting for this quarter this weekend) so I'll have some spare time to work on this, and (hopefully!) I'll have a working build for macOS, Linux, and Windows soon™. Android and iOS, followed by FreeBSD and Haiku (sorry @lazybullfrog, though you're more than welcome to test early/often) will be my next priority after that.
The biggest chunk of work/blocker thus far has been extracting the python scripts into "installable" tools that are separate from Scons and executed like processes. The actual grabbing of dependencies, settings, etc. has been easier in comparison.

@slurps-mad-rips, life has thrown me a few curve balls too. Perhaps we should start a baseball team. 😂
Fortunately one of those curve balls since I last commented was a 8 core Xeon with 32gb ram with Haiku and FreeBSD installed. Perhaps soon I'll get some time to help test both.
I'm actually ecstatic that both of these platforms are getting attention. Look forward to testing soon.

I think it is reasonable to have multiple build-systems. Perhaps an official build-system that is well supported and the others provided on a "you figure it out" basis. Users will want to use their own build-system regardless, so why not pool their work into one place?
I would be willing to contribute a Bazel port if things go that way.

I think it is reasonable to have multiple build-systems. Perhaps an official build-system that is well supported and the others provided on a "you figure it out" basis. Users will want to use their own build-system regardless, so why not pool their work into one place?

As a Linux packager, I have seen hundreds of projects with various buildsystems. I've seen a few which did have several buildsystems available. I've never seen one where it was not a complete mess, especially when the alternative buildsystems were contributed by first-time "please add CMake it's so cool" contributors who then disappeared from the project, and maintainers didn't bother updating it.
There's no value at all in supporting several buildsystems, and I won't merge any PR going in that direction. I'm one of the main buildysytem maintainers, and I definitely don't want to have the burden of maintaining alternative ones whenever we need to change something.
We currently have a buildsystem that works for everyone and is well supported, and no intent to change. If someone really wants to investigate other options, feel free to, and if the final result also works for everyone, can be well supported and provides added value without big drawbacks, we'll consider it.

That's what I've been doing with my CMake port. Currently a no-op or single file change under scons takes upwards of 10 seconds on my current machine (which has 64GB of RAM, comparable processor, etc.) With ninja + cmake the entire configure + generate process takes less time, and the ninja build finishes in under 0.2 seconds in a no-op build (on windows, where launching a process is expensive). I'm also able to use the sccache tool from mozilla if its installed, and that's been extremely useful for editing the codegen parts/running a regenerated build from scratch to make sure nothing is depending on my local environment.
most of my time has been spent on extracting the code generation scripts into python scripts that could run on their own. If that was the only thing merged into godot, my time spent on this wouldn't have been a total waste. It also solves the run_in_subprocess hack that currently exists to solve issues on windows with parallelization.
I just returned from my trip and was able to get a bit of work done. Right now I still need to copy a few files over from a manual scons run, but I've been chipping away at the generated sources as time goes on. The license.gen.h and method_bind.inc headers are currently my biggest target, but I'm taking a few days to deal with jetlag + work before I'll tackle them.
(also just gonna cc @fire since he's been interested in this progress for a bit)

From my experience Meson is totally f*ed up when building Android on Windows, I just cannot get it work to compile one of third party libraries that uses it (libepoxy). And I think it's too much tied to concrete compiler stuff.

All I desire is a build system that can generate project files and database files separately from the build command.
Every time I check out a new branch with major changes to files, I have to run Scons from the command-line to regenerate my vsproj files, which also causes a full build to occur anyways. In Scons from what I have seen, there is no project generation stage of any sorts. This is also why it doesn't support generating compilation databases out of the box for usage in clang-tools.
Currently, the only viable solution to coding Godot on windows (IDE-wise) is with Visual Studio, otherwise be prepared to just use a text-editor.
Scons is wonderful and all, but when I have a 12 core threadripper and 128gb of ram and I have to wait 8+ seconds on windows to compile a single line change (most of that is Scons thinking), it gets a bit annoying.
But the real downside to Scons (and this could be my laziness in searching) is that I cannot compile individual cpp files. Being able to compile individual cpp files without building the entire project is a huge productivity booster and sanity checker.

If you want to build only a single target file, then specify that on the command line
scons -j24 num_jobs=24 vsproj=true platform=windows <path  to targetfile>/targetfile.obj

Just wanted to add that the latest version of CMake is adding some of the missing features people mentioned earlier as reasons to not use it.
Unity builds: https://cmake.org/cmake/help/latest/prop_tgt/UNITY_BUILD.html
Precompiled headers: https://cmake.org/cmake/help/latest/command/target_precompile_headers.html

@Nosliwnayr Both of those have been supported by addons for many years, they are well tested and wanted enough now to be included.  Those were never reasons not to use CMake as you've been able to do both in CMake for a very long time.

@bdbaddog
i don't understand, do i have to specificy every single file manually? what does that .obj thing do?
how do i apply it to the whole project? so that i can recompile individual files?
i hope the compile times can be optimized further, being able to compile c++ code near instantly with no tricks would be a god send, after seeing how fast godot compiles, i'm really struggling to go back to the ue4

@RaTcHeT302 - The command line specified is if you wanted to explicitly only compile a single target (in this case an object file) (and of course the files it depends on)
Indeed if all the folks complaining about issues with SCons would contribute to helping fix them rather than suggesting switching a complicated build system to another newer shinier one.. well many of such concerns would be fixed..

@bdbaddog Scons does not generate intermediate stage build files. It does the "project dependency" step in the same step as it does compile, and they cannot be separated.
Its like telling CMake to generate the project files everytime before you hit your build tool, whether it be MSVC, ninja-build, make, llvm, pick your poison.
If I switch git branches, I cannot update visual studio without performing a scons command line build to regenerate the proper .vsproj and .sln files - and I almost always clean build branch changes since I have ran into issues in the past.

When converting my fork to CMake most of the complexity was in the code generation files. The idea that the build is complicated is a fallacy. The build doesn't need scons because its complex. It's complex because it uses scons.
With that said, the reason I halted work is because I met with the lead devs at GDC in 2019 (at the GitHub meetup) and they said they have absolutely no intention of switching to something more usable, citing that they'd prefer to do the build entirely on linux and indeed from what I've seen they're violating the EULA that Apple requires acceptance of to install build tools for cross compiling.
The mentality of those in charge is very much "ends justify the means" and there doesn't seem to be any care about treating the build as software, and instead godot treats it as a series of scripts that happens to produce a build.
I will not be contributing the changes I made to the code generation to godot (despite it being more readable and faster), and I've given up on this engine entirely and moved on with my life. I'm much happier if I'm being honest. The engine already has considerable tech debt and I see it being an issue in about 3 years, especially as the C++ standard committee is working on a tooling technical report (I'm involved in this and have been pushing for things like options files being standardized to a degree, but it's just a technical report so some folks will diverge from it).
It's a shame too. My CMake build was able to catch a bunch of static analysis errors that have led to tons of issues and are easily solvable, but I'm too burned out from having argued with the devs about this at GDC to file bugs or even attempt to fix them. My energy and my mental health are better spent elsewhere.
There is no world where scons will be able to handle the upcoming C++20 modules design, as much as I have shouted how it will break many tools. It can't handle the dependency tracking for that, and would require it to have a pre-scan step, which will slow the dependency tracking down even further.
Lastly, I'm going to suggest people not try to improve a tool that can only accept minor changes. Scons would need a complete rewrite from the ground up and a large number of breaking changes to get the improvements people here seek.
I suggest this issue be closed as the discussion of changing the build system was answered at GDC 2019, but the leads don't even think its worth wasting their time to comment on this issue, which I think shows how much they care about the ergonomics of development.

I feel as though if the existing build system were 100% converted in a plug and play fashion there would be no reason from the devs not to switch.
Building only on Linux is useless. Having a game engine that doesn't run on windows will deter any serious game developers.

IIRC they use Wine to run the MSVC build tools. Still doesn't change your point, and I agree.

IIRC they use Wine to run the MSVC build tools. Still doesn't change your point, and I agree.

Official Windows builds are compiled using MinGW, we don't use WINE to run MSVC on Linux.

@marstaik - if you switch builds and run scons just to generate the MSVS Project files, you should be able to specify those files as the target and not have to do a full build.
so try:
scons <path to created MSVS project file>/VSPROJECTFILENAME (replace with actual paths and file name which are generated)

SCons should be able to do this without doing a complete build and also it should do it correctly without you needing to do a clean first.
(BTW. I'm the SCons project maintainer)

@bdbaddog I had to add vsproj=True to that command, but it seems to have worked
From the godot src:
scons godot.vxcproj num_jobs=x vsproj=true

@bdbaddog I had to add vsproj=True to that command, but it seems to have worked
From the godot src:
scons -j24 godot.vxcproj num_jobs=x vsproj=true
Edit: For some reason you need to add the -j flag also...

A lot of such flags aren't vanilla SCons, but part of the way various packages (Godot in this case) implement their build system with SCons.
Glad you found a way!

Why use SConstruct and insist on not switching to other build system???

@tjysdsg The discussion above sums that up very well :slightly_smiling_face:

I'll close this because we've made it clear over the years that we have no intention a priori to change the buildsystem. We're happy with SCons and any of its shortcomings can and should be assess individually, both within our own SCons usage and upstream if it's due to the tooling itself.
Switching to a "cooler" buildsystem for the sake of it will not happen. The vast majority of core contributors is happy using SCons and it's the one we now know best. Switching to a new buildsystem has both an implementation and a learning cost, and it hasn't been proven so far that those would be worth the trouble.
All in all, a buildsystem change is not something that we'd accept from new contributors. It has to come from the inside, and all core contributors must see it as a positive change for it to make any sense. We won't change the buildsystem just to please new users who attempt to build Godot from source and somehow don't like that they have to type scons instead of cmake or meson which they are used to.
Of course we're open to changing our minds if clear gains can be listed (and shown with a working proof of concept) that outweigh the costs of changing the buildsystem, but as of today and after two years of discussion here, I personally don't see the incentive.

@slurps-mad-rips said it well.

It's complex because it uses scons.

and

My CMake build was able to catch a bunch of static analysis errors that have led to tons of issues and are easily solvable

Flat out disgusting that CMake isn't given serious consideration, I can think of many reasons to ditch SCons, but SPECIFICALLY that CMake finds errors Scons does not find makes it's "worth it" to use something else.

Switching to a "cooler" buildsystem for the sake of it will not happen

CMake isn't "cooler", it's the standard. If you actually used it you would immediately understand why.

Of course we're open to changing our minds if clear gains can be listed (and shown with a working proof of concept) that outweigh the costs of changing the buildsystem

I'm not convinced that anyone is open to changing their minds. Again, @slurps-mad-rips showed clear gains, and she even did all the work... It's already done.
It is shameful, representing the official stance of the Godot team, to completely disregard her efforts improving the development process.

I quite understand your pain but I still agree with others - all the build
systems have their
pros and cons and generally the same. There is rarely any technical reason
to change it.
As for political reasons they will not do you anything good in the long
run. Everybody
prefers something different. Anyway nobody stops anyone from doing forks
for whatever
reasons, so I guess this one is as good as any.
On Fri, Feb 21, 2020 at 1:38 PM dorkbox<EMAIL_ADDRESS>wrote:

@slurps-mad-rips https://github.com/slurps-mad-rips said it well.
It's complex because it uses scons.
and
My CMake build was able to catch a bunch of static analysis errors that
have led to tons of issues and are easily solvable
Flat out disgusting that CMake isn't given serious consideration, I can
think of many reasons to ditch SCons, but SPECIFICALLY that CMake finds
errors Scons does not find makes it's "worth it" to use something else.
Switching to a "cooler" buildsystem for the sake of it will not happen
CMake isn't "cooler", it's the standard. If you actually used it you would
immediately understand why.
Of course we're open to changing our minds if clear gains can be listed
(and shown with a working proof of concept) that outweigh the costs of
changing the buildsystem
I'm not convinced that anyone is open to changing their minds. Again,
@slurps-mad-rips https://github.com/slurps-mad-rips showed clear gains,
and she even did all the work... It's already done.
It is shameful, representing the official stance of the Godot team, to
completely disregard her efforts improving the development process.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/issues/16014?email_source=notifications&email_token=AAABPU2PBIEK6PFKQ2LDTHLRD6VKVA5CNFSM4ENJ6NN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMSI4JY#issuecomment-589598247,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAABPU746R4YFHAI73Z57RTRD6VKVANCNFSM4ENJ6NNQ
.

@dorkbox - cmake found static analysis errors? or was it the compilation database output from cmake run through llvm's static analysis tools which found them?

@bdbaddog, @slurps-mad-rips did the work on this, and it would be best to ask her. (I think it would be the output from llvm's static analysis, but I'm not the one who did the work to build via cmake)

@slapin

There is rarely any technical reason to change it.

How about static analysis tools and modern language features support?

@bdbaddog @dorkbox I simply turned on clang-tidy to look for things like "returning references to non-static locals" and "buffer overflows" via CMake. I also ran clang-check on a few things and got a few more results. I was able to get include-what-you-use to run as well, and it found quite a few unnecessary header inclusions.
That said, I didn't actually "finish" the cmake port. The part that never got ported was all the work I did with trying to change the current python scripts that simply call f.write on some file to generate a file from JSON. I was able to remove a lot of that logic and replace it with a generic jinja2 based template generator. This also solved a hack that the current scons build does on windows where it launches a script as an executable.
However I had hoped to keep parity with the scons build and at the very least get my changes to the code generation merged in. This didn't happen because the python scripts used to generate source files are constantly changing, and keeping those changes in sync with my jinja2 templatization was an exercise in frustration.
Given the very emphatic and hard "no" I got from the lead developers at the GDC GitHub meetup, I decided I no longer wanted to contribute to godot. My changes were unwanted, are not desired, and they are apathetic to anyone else's opinion on this matter. I won't be contributing to godot, nor to any forks of godot. I wish you all luck in trying to convince the lead devs in the matter but it's like talking to a brick wall. Unyielding, immovable, and apathetic to the plight of everyone in this thread.

So if SCons can spit out the compilation database which is input to the llvm tools then this would also be able to be done from SCons.

So if SCons can spit out the compilation database which is input to the llvm tools then this would also be able to be done from SCons.

It looks like there is the sunk-cost fallacy going on with SCons. Why not use something that is easier, streamlined, and already part of existing build tools, like cmake?
From the scons website:

When compared to scons, CMake is :

Faster
Requires less code for common tasks
Arguably more stable
Supports outputting to projects like Code::Blocks, Xcode, etc. which scons does not

@dorkbox  - I'm the SCons project co-manager. I'm just keeping an eye on desired functionality.
The last few releases and the forthcoming one have had performance improvements.
Additionally we have PR's for ninja file generation and some community members have compilation database tools. So some of the issues are likely to be addressed (at least partially, if not fully) in the next release.
I'm not aware of any stability issues affecting the godot project which we haven't addressed. If there are any outstanding please let us know.
(Reach us via: https://scons.org/contact.html)
I'm not sure cmake really requires less code for common tasks, I'd be interested in reasonable comparison.

Well, what mostly annjoys me with scons is inability to generate plain old
GNU make makefile. Lack of ninja - I can live with it.
Also handling of cross-compile utility like CMake toolchains and autotools
cross-compiling capabilities is strong cmake weak point.
The rest is ok as I'm not IDE person. My IDE is UNIX environment.
On Sun, Feb 23, 2020 at 10:05 PM William Deegan<EMAIL_ADDRESS>wrote:

@dorkbox https://github.com/dorkbox - I'm the SCons project co-manager.
I'm just keeping an eye on desired functionality.
The last few releases and the forthcoming one have had performance
improvements.
Additionally we have PR's for ninja file generation and some community
members have compilation database tools. So some of the issues are likely
to be addressed (at least partially, if not fully) in the next release.
I'm not aware of any stability issues affecting the godot project which we
haven't addressed. If there are any outstanding please let us know.
(Reach us via: https://scons.org/contact.html)
I'm not sure cmake really requires less code for common tasks, I'd be
interested in reasonable comparison.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/issues/16014?email_source=notifications&email_token=AAABPU5TQQFET5RS2JDRY7TRELCHDA5CNFSM4ENJ6NN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMWEOMI#issuecomment-590104369,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAABPU6UHPX3HJTWPAFTPATRELCHDANCNFSM4ENJ6NNQ
.


[bn]  Localization of Content/bn/releases/version skew policy.md

[bn]  Localization of Content/bn/releases/version skew policy.md
Resolves: #44993

after complete the review I will remove reviewer option from the file (lines 2-8)

@sajjadrahman56 no need to remove reviewer list . comment out them like this

@sajjadrahman56 please fix them

/lgtm

To fix netlify build  errors follow this steps :

1 ) go to your fork repo .
2) go to dev-1.24-bn.1 branch
3 ) sync fork

go to PR section and create new PR

take your own forked repo as a base repository

take your PR branch as base

take dev-1.24-bn.1 as compare and create PR

merge this PR

10 ) and done

@sajjadrahman56  Kubernetes is one of the largest open-source repositories in the world, and as such, we encounter many obstacles along the way. It's our responsibility to address and overcome them  as a contributor .

/lgtm great work

/approve


Appointments not showing in Schedule [June 2023]

I have downloaded latest pluralsight-ddd-fundamentals-main.zip and started the apps with docker. However when I start ddd.frontdesk.blazor in browser the Schedule is displayed for 30-09-2030 but no appointments. There are multiple rows of data in dbo.Appointments for this date. If I select a Client and Patient and double click in a blank space the Appointment Data appears in a modal but when Save clicked a notification appears but nothing still displays in the Schedule.
This might not be related but in the browser DevTools there are 2 messages under console:
 DevTools failed to load source map: Could not load content for http://localhost:5100/css/kendo/kendo.common.min.css.map: HTTP error: status code 404, net::ERR_HTTP_RESPONSE_CODE_FAILURE
DevTools failed to load source map: Could not load content for http://localhost:5100/css/kendo/kendo.default.min.css.map: HTTP error: status code 404, net::ERR_HTTP_RESPONSE_CODE_FAILURE

dbo.Appointments.txt

Those files shouldn't matter too much. Are you able to see the DB calls being made? That should tell you something about what the app is trying to do under the covers. Troubleshooting steps:

Is EF using the SQL provider and talking to SQL?
What query is EF using?
What do you get when you run that query yourself?
Can you hit the API endpoint yourself and get data that way?

Basically the data needs to make it from the database to the API server to the Blazor Client code and finally be rendered to HTML. Any one of those transitions could fail so start from the database and try to find the failure point. HTH.

I have started from scratch and created both migrations and database updates.
I can confirm that I can get data via Swagger for all endpoints.
From Docker Desktop when I open ddd.frontdesk.blazor the Appointments page is displayed with Client: David Batten and Patient: Max.
No appointments appear for Monday, September 23, 2030. [The seeded data does correspond to this date].
However when I click 'Agenda' all the appointments are displayed correctly for Exam Room 1 / 2 /3.
Exam Room 1: 23 Monday September 2030 11:30 PM - 12:00 AM (WE} Darwin - Steve Smith
That might be a clue - When I click the 'Day' option the time slots displayed start at 7:00am and end 5:00pm.
So maybe the seeded appointments are outside this range and will not be displayed?
Also if I double click in any clear space in the schedule a modal pops up with Appointment Data Title: (WE) Max - David Batten etc. - I cannot see how I can add an appointment - also if I try and delete the appointment that pops up I get an error:
Access to fetch at 'http://localhost:5200/api/schedule/00000000-0000-0000-0000-000000000000/appointments/00000000-0000-0000-0000-000000000000' from origin 'http://localhost:5100' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
Thanks for taking the time to review this issue.

Adding an appointment is done by first choosing an owner/patient at the top and then just clicking (or maybe double-clicking) on a time in a room. That should bring up a modal window where you choose the appointment type. In the example you're giving with Max - David Batten that's trying to create a Wellness Exam (WE) and I assume you have Max/David chosen at the top. You should be able to save the modal and it should create the appointment, unless there's a problem (like CORS...).
The CORS issue is probably the root of your problem. Since this is a demo app you can just disable CORS by going into the middleware of the web API app and modifying CORS to allow anything.
See: https://learn.microsoft.com/en-us/aspnet/core/security/cors?view=aspnetcore-7.0
Here's where to make the change:
https://github.com/ardalis/pluralsight-ddd-fundamentals/blob/main/FrontDesk/src/FrontDesk.Api/Startup.cs#L87-L97
Add builder.AllowAnyHost() and remove the WithOrigins and SetIsOriginAllowed lines. See if that gets you past the CORS issue and gets other things working too hopefully.

I think I have narrowed the primary issue to being TimeZone related. I am operating in Adelaide Australia which is UTC+9:30.
In looking through the source code it appears that the TestDate was being initialised for UTC-4.00.
Thus appointments were being generated in non-available days/hours and therefore not appearing outside the available time slots. I adjusted the UTC time settings in several places and now am getting close and the appointments are appearing close to where they should be.
I have not used Docker much before but I note that if I run the apps via Visual Studio it uses my SQL Database and if I run Docker it uses another database within the containers - I am a slow learner so it took a while for the penny to drop!
The CORS problem was in Docker only and I made the suggested changes [note: I added builder.AllowAnyOrigin() not builder.AllowAnyHost() ]. I also need to reboot my PC and run Docker from a clean start for the FrontDesk to load the appointment data successfully.
So now I can get on with it!
Once again thanks for taking the time to review this issue.


iptables-restore with no flush leads to duplicate rules

When run with r=true, the first think kenc does is restore rules from the checkpointed file using iptables-restore --noflush:
https://github.com/coreos/kenc/blob/master/main.go#L130
https://github.com/coreos/kenc/blob/master/pkg/util/iptables/iptables.go#L320
Then it starts checkpointing rules.
The problem with this approach is that the kubelet also creates its own rules by default, which are then checkpointed by kenc:
https://github.com/kubernetes/kubernetes/blob/0dff56b4d88ec7551084bf89028dbeebf569620e/pkg/kubelet/kubelet_network.go#L289
The kubelet will certainly start up before kenc does. This means that on every system reboot and kenc pod restart:

kubelet has added a rule A
checkpointer restores checkpoint that contains rule A, so now there are duplicate rules A A
checkpointer checkpoints A A into its checkpoint file

If there are many kenc restarts or perhaps other iptables issues then this leads to linear rule growth. In a degenerate case (not sure exactly how) I ended up with a 26mb iptables ruleset with 100,000s of copies of rules like:
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING

A better approach would be to be more selective about what is checkpointed, or better yet, to use the EnsureChain / EnsureRule logic that the kubelet uses instead of invoking iptables-restore

@diegs
kenc code tries to simulate the kube-proxy logic, where we put all rules in one transaction and restore it with noflush. We do not need to do ensureRule or ensureChain when we try to flush out the chains anyway.
The problem is we accidentally checkpointed rules that are not inside the chains that we will flush in the restore logic. Only the rules not in the chains we flushed are duplicated.

@diegs if you still keep the huge file, probably you can help me to double check that only rules in
these chains:
PREROUTING
INPUT
OUTPUT
POSTROUTING

are duplicated and nothing else gets duplicated.

Here's the massive file.
ipt.save.gz

@diegs Checked the file. Confirmed my pervious assumption.


Markdown headings, typos and lead-in for additional identities

Signed-off-by: Steve Lasker<EMAIL_ADDRESS>

@gokarnm, can we queue the additional changes to another PR? This was just a set of typos and heading fixes.


Unable to call staticmethod on class from qualified import

When importing a class with a qualified import, it is not possible to call staticmethods. It is possible to work around by importing the exact class with a from X import Y and then it is possible to call the staticmethod
In module foo:
class Foo(object):
    def __init__(self):
        pass

    @staticmethod
    def new():
        return Foo()

in main module:

import foo

actor main(env):
    f = foo.Foo.new()
    env.exit(0)

and we get:
kll@Kristians-MacBook-Air imported_staticmethod % acton build
Building project in /Users/kll/dt/acton/test/compiler/imported_staticmethod
  Compiling foo.act for release
   Finished compilation in   0.004 s
  Compiling m.act for release

ERROR: Error when compiling m module: Type error

  |
5 |    f = foo.Foo.new()
  |        ^^^^^^^
The type of the indicated expression (inferred to be () -> foo.Foo) must have an attribute new with type _(+, ++) -> _; no such type is known.

 I don't remember exactly why it was. I couldn't find an existing issue, so opening this...

Fixed by #1960.


A file can't be renamed the same as a folder name

Description
If you have a folder named [folder] something.json and try to rename a file to the same name ([file] something.xyz -> [file] something.json), the file name will auto increment ([file] something.json (2)).
Steps To Reproduce

Create folder named something.json
Create file named example.txt
Rename example.txt to something.json
Result: file becomes something.json (2)

Steps to fix

eg. Fix crash that occurs when...
eg. Correct action happens when clicking...

Files Version
<IP_ADDRESS> (Stable latest Microsoft store version)
Windows Version
Windows 11 22H2
Relevant Assets/Logs

Interestingly, even the file manager that comes with Windows does not support consistent names for folders and files.

The file system isn't able to determine what is and isn't a folder when using names. Files won't be able to support it.