EEVblog Electronics Community Forum
Products => Computers => Programming => Topic started by: westfw on November 15, 2022, 12:58:54 am
-
Is there somewhere a description of sort-of the Philosophy and Purpose of CMake (and how to use it)?
I keep seeing people using CMake to generate Makefiles, when it looks to me like it would have been trivial to generate the makefile manually (at least, if one hadn't created an awful directory structure apparently designed to match the way CMake thinks, or something.) (cf RPi Pico SDK.) And I just am not getting it. :-(
-
who wants to write makefiles by hand?
-
I only use makefiles. Make is versatile and powerful. And it's like everything else, once you've written makefiles for a few different projects with the structure and features that you want, you can reuse them as templates for other projects, only modifying a few things here and there. It's a bit bizarre to hear people talk about it as though they had to write each of their makefiles from scratch every time, thus spending a lot of time. It doesn't make sense. Once you have covered your bases, it's a matter of a couple minutes for every new makefile - nothing longer than when using CMake. But, you have this initial time investment.
Apart from the syntax, that you get used to pretty quickly, Make is a decent tool.
Most of the criticism (again apart from syntax) I've heard is about dependency handling. Make itself can follow any dependencies you define, but it can't create them on its own (or only in very limited ways), and it's seemingly one of the main reasons new build tools are designed and used. So, if you build C or C++ code, you usually use the compiler to generate dependencies, GCC for instance have had options to generate Make dependencies for a very long time. For other compilers, that may vary. And, it's always an extra step you need to handle in your makefiles. Again, once you have set up a few makefiles covering your common use cases, it's mostly just a matter of reusing them. But the process may look clunky to many people, and generating dependencies for some other languages can require writing your own tools, which is admittedly not that convenient. And, even for C and C++, generating dependencies using the compiler, while straightforward and fully automatable, can take a pretty long extra time if you deal with big projects. This step is not very optimized.
That is one of the reasons most of these tools are front-ends to Make (or, alternatively, ninja.) Dependencies.
I personally do not use CMake unless I'm forced to, but if you find it useful, go for it. Ultimately it's whatever is most convenient and productive for your use case.
-
I use CMake in my day job. I work on a team that develops software that's used for communications across the Internet. The same software must be built to run on Windows, Mac, and more than a dozen varieties of Linux/Unix. The software has evolved over more than a decade with a large core written in C/C++, plus various bits written in Ruby, Java, and quite a few other languages. The nice thing about CMake is that we can write the cmakelist.txt file one time and it supports all our platforms. Before we used CMake, we had different build tools on Windows (dev studio with its project and solution files) than on Mac/Linux/Unix. Keeping build changes in sync across all platforms has become easier now.
-
Before we used CMake, we had different build tools on Windows [vs Mac/Linux]
How is it easier with CMake? I have makefiles that work on all three platforms; it wasn't even necessary to install unix-shell-like tools on Windows (though it was easier!)
-
I have never understood the real fundamental purpose of CMake, and no one has been able to properly explain it to me. It seems the only reason to use CMake is because CMake is used, or needs to be used, because someone chose to use CMake.
And make is quite kinda crappy tool for its job, but it gets the job done, not too bad. I don't see how CMake relates, because it's no replacement for make.
-
I have never understood the real fundamental purpose of CMake, and no one has been able to properly explain it to me. It seems the only reason to use CMake is because CMake is used, or needs to be used, because someone chose to use CMake.
And make is quite kinda crappy tool for its job, but it gets the job done, not too bad. I don't see how CMake relates, because it's no replacement for make.
cmake makes makefiles so of course it is not a replacement ...
-
Cmake is the reason why I developed myown builder
-
GCC for instance have had options to generate Make dependencies for a very long time
I have developed myown dependency builder.
It also checks for deadcode.
My-c independency from everything.
-
GCC for instance have had options to generate Make dependencies for a very long time
I have developed myown dependency builder.
It also checks for deadcode.
My-c independency from everything.
Ah. That's good!
-
Gcc, old versions, made me crazy with projrects because didn t behave the same way.
Makefile with kernel v2.4 and <2.6.16 is incompatible with modern tools.
Yet again hours wasted to understand and fix
-
I don't much like cmake. When it works it's very very good, but when it fails it's horrid.
It's predecessor was autoconf. That was a huge pile of scripts but it was robust. If it broke - which it didn't often do - you could find what was wrong, and mostly fix it.
Cmake is like the systemd of making. Apparently simpler because there aren't so many bits - but in reality the workings are hard to find, so when it doesn't work it's not easy to fix.
-
Every interaction I've ever had with cmake has been trying to fix it on other people's projects. My eyes glass over as I try and understand the wall of chaos and dependency detection systems. A programming language to generate a programming language to compile your programming language.
I don't know if cmake is the cause or just a symptom of complex build processes.
Is cmake an operating system?
-
Interestingly, browsing through the official website doesn't necessarily answer your questions: https://cmake.org/ - all they basically say is that you only need to put some text file in each source directory, and bim bam boom. ;D
I also find the "success" page odd: https://cmake.org/success/
As though they were trying to sell something. You usually put those testimonial things when you have something to sell. Right?
-
I really think a big part of the confusion stems from the assumption that CMake is somehow intended as a replacement for Make. It is really more of a replacement for autotools.
CMake does not just manage internal build dependencies, it also manages external library dependencies, e.g. it can check that the correct version of a library is actually available in the build environment and find the correct paths. It also provides an interface for configuring compile time options.
Apart from that, I think one of the central ideas behind CMake is to somewhat decouple the "what" (source files, targets, dependencies etc.) from the "how" (like the actual build system, OS, toolchain, paths and command). The latter could vary wildly for the same project, like you could have a standard make (or ninja) build on Linux, generate Visual Studio projects on Windows and then maybe only later decide to cross-compile for ARM. That also makes it possible for IDEs (and other tools) to understand CMakeLists.txt files and hopefully interpret #includes and #ifdefs correctly for a given target which can improve navigation and code completion quite a bit. And it tends to make life easier for Linux package maintainers too.
It is quite possible that none of that matters to you, and that is completely fine. Many of the features provided by CMake could be hand-coded for each project using Makefiles and maybe some scripts. But for larger and more complex projects that approach doesn't really scale.
Finally, I think a lot of people will use it for smaller stuff too because they are used to it anyway.
-
A programming language to generate a programming language to compile your programming language.
I don't know if cmake is the cause or just a symptom of complex build processes.
yup, exactly.
-
As though they were trying to sell something. You usually put those testimonial things when you have something to sell. Right?
As you can see from the front page, they are selling support and training.
-
CMake does not just manage internal build dependencies, it also manages external library dependencies, e.g. it can check that the correct version of a library is actually available in the build environment and find the correct paths.
Which is why I truly cannot understand pushing CMake for embedded microcontroller projects, but I have seen that happen.
-
Cmake is very old (created in 2000). Many open source projects these days seem to move to Meson https://en.wikipedia.org/wiki/Meson_(software)
I haven't really used either (other than messed around with a few projects), so I can't tell which is better, but it sounds like Meson has improved some things. Supposed to be more user friendly (cmake build definitions feel obscure to me).
-
And yes, if you have a very simple project it doesn't make sense to use a build system like cmake.
-
I use makefiles to automate many tasks, not just for compiling programs.
I use them to create compressed project files, automatically create and crop images from PDFs, run Python macros on source files and templates to create final files, create thumbnails, etc.
But better than CMake, I prefer to use GNU Make.
Edit:
https://www.gnu.org/software/make/ (https://www.gnu.org/software/make/)
https://www.gnu.org/software/make/manual/make.html (https://www.gnu.org/software/make/manual/make.html)
And be careful to use tabs instead of spaces !!!
-
And yes, if you have a very simple project it doesn't make sense to use a build system like cmake.
Sure, but unfortunately, in the embedded world, more and more vendors are switching to CMake for their SDKs, so even for a small project using a modern MCU, you may be faced with CMake.
I have personally run into two such vendors for MCUs that I've used: RPi with the RP2040 and NXP with the iMXRT series. In both cases, the CMake file hierarchies were so intricate that it drove me nuts. In both cases, I wrote my own makefiles for using the SDKs. That took a bit of time (mostly due to figuring out where SDK files were and handling the hierarchy). You can of course do without the SDKs altogether. But while this is absolutely doable for something like the RP2040, good luck with something as complex as a iMXRT106x.
-
Many open source projects these days seem to move to Meson https://en.wikipedia.org/wiki/Meson_(software)
I have looked at Meson and it seems much better crafted than CMake. As I said, I have no interest in those build tools so far, but it's always nice to have alternatives.
-
Sure, but unfortunately, in the embedded world, more and more vendors are switching to CMake for their SDKs, so even for a small project using a modern MCU, you may be faced with CMake.
I have personally run into two such vendors for MCUs that I've used: RPi with the RP2040 and NXP with the iMXRT series. In both cases, the CMake file hierarchies were so intricate that it drove me nuts. In both cases, I wrote my own makefiles for using the SDKs. That took a bit of time (mostly due to figuring out where SDK files were and handling the hierarchy). You can of course do without the SDKs altogether. But while this is absolutely doable for something like the RP2040, good luck with something as complex as a iMXRT106x.
I had the opposite experience a while ago. The (legacy) nRF5 SDK comes with plain Makefiles. But it consists of many small modules that each depend on other modules. In the end I had to:
1. make
2. find each missing include and add path
3. for each linker error find source file that defines the symbol and add to sources
4. goto 1 until build succeeds
And that can take quite a few iterations, so I was really wishing they had used some kind of build system to handle the dependencies. And in fact it turned out someone else was so annoyed by this that they put together... a set of CMake files (https://github.com/Polidea/cmake-nRF5x).
-
But the nRF52 SDK is a total disaster anyway. I tried for a week to make anything happen with it. Didn't get even Hello World working, because that alone meant debugging their 10000-lines-of-code NIH logging library which did not log out of box. Blinking a freaking LED was through a nearly mythological beast called LBS - LEDs And Buttons Service, so no wonder there were a lot of makefiles and a lot of hieararchy. IIRC, just the LED blinker compiled and linked over 200 files. The nRF52 hardware itself is very easy to use.
-
Well, yes, I wouldn't disagree with that. I mostly figured it out in the end and managed to port some non-trivial examples to a custom board as well (including logging over RTT and serial). But it definitely wasn't fun and involved several hours of methodically working through the source files because the documentation was not exactly helpful.
BTW The basic SDK blinky actually needs just 4 source files (including main.c). But I think they ran into the same problem I had and they didn't have time to figure out all the dependencies manually for each example. So instead they decided to build the whole kitchensink every time and let the linker throw most of it away. Of course, as soon as you switch to something more complex with a wireless stack, you DO need a lot more files but that seems unavoidable.
-
Sure, but unfortunately, in the embedded world, more and more vendors are switching to CMake for their SDKs, so even for a small project using a modern MCU, you may be faced with CMake.
I have personally run into two such vendors for MCUs that I've used: RPi with the RP2040 and NXP with the iMXRT series. In both cases, the CMake file hierarchies were so intricate that it drove me nuts. In both cases, I wrote my own makefiles for using the SDKs. That took a bit of time (mostly due to figuring out where SDK files were and handling the hierarchy). You can of course do without the SDKs altogether. But while this is absolutely doable for something like the RP2040, good luck with something as complex as a iMXRT106x.
I had the opposite experience a while ago. The (legacy) nRF5 SDK comes with plain Makefiles. But it consists of many small modules that each depend on other modules. In the end I had to:
1. make
2. find each missing include and add path
3. for each linker error find source file that defines the symbol and add to sources
4. goto 1 until build succeeds
And that can take quite a few iterations, so I was really wishing they had used some kind of build system to handle the dependencies. And in fact it turned out someone else was so annoyed by this that they put together... a set of CMake files (https://github.com/Polidea/cmake-nRF5x).
Well, this is actually quite a similar point, but with a different tool. As soon as you want to build in a slightly different way than what the vendor provides - sometimes requiring the whole shebang just for building a trivial test program - it becomes an intractable mess. In the same vein, good luck with the ESP32 SDK as well. It's not badly done as long as you strictly follow the way they want you to use it, but any change in the way you want to build stuff is a major headache.
I hate this new trend (which is often seen with CMake, but admittedly not only) of those SDKs with very deep hierarchies down to sometimes 1 source file per directory, with even a separate subidrectory for include files. :palm: It sure makes compiling it in any different way than exactly what the vendor provides you as examples or automatically generated stuff, a major PITA.
But ultimately this is down to the dreaded software bloat that is now infecting embedded development as well.
To paraphrase a famous saying which goes like "war is much too serious a thing to be left to the military", I will just say that software is much too serious a thing to be left to software engineers.
-
I, personally, like CMake.
At work, in one morning I was able to use it effectively, if not of course efficiently and smartly, after having struggled for days with the baroque mess that autotools is.
In the end I had to cave in (as in having someone else do it...) since autotools (plus bitbake plus yocto) is our standard toolchain.
The RP2040/pico SDK is a bit of a red herring, they overcomplicated it "a little bit" - though it sports some nice automation: when you add a library, the include path for the relevant .h is automatically added (had they avoided their maze of twisty little directories, all alike, this would not be a point... |O CMake does not force you in that direction).
I use it for my my projects, if only because I really dislike Makefile syntax* and builds with ninja are insanely fast (and you can't write ninja by hand)
For the same project, about 200 kLOC, AMD 5900X: make 12 s, CMake+ninja - including build system regeneration - 2 s (I know, 1 sample is not data).
CMake integration in VS Code (and VS, too!) is extremely good and helpful, this is of course not a virtue of CMake, but still it helps and make for a good experience.
The language is not "pleasant", a bit COBOLesque in verbosity, but it's very flexible, multi platform - my builds are unchanged across Linux, Windows, macOS - and easy to create templates with.
I'm starting to study meson - I like what I've seen so far and might prefer it in the end, but I literally started a handful of hours ago.
* it's slightly ironic to see a number of python "indentation is semantic" bashers sing the praise of makefiles...
-
* it's slightly ironic to see a number of python "indentation is semantic" bashers sing the praise of makefiles...
There's only "indented" or "not indented" -- same as many assembly languages -- which doesn't really seem comparable to Python.
I don't like makefiles much. They're ok for simple tasks, but some people overdo it by far.
Makefiles do two very different tasks:
1) describe the structure of your project and sub-projects, the dependencies between things, and how to build things from their dependencies
2) a breadth-first traversal of the dependency graph, executing the actions
Traditional e.g. GNU make does the second one tolerably well, though ninja is better.
It does the first part really quite badly, once projects get complex. That's where cmake and others come in.
A basic makefile format with completely expanded and explicit dependencies and build actions is a reasonable interface between the first and second tasks. It doesn't matter these days if it is a few MB in size.
-
Makefiles do two very different tasks:
1) describe the structure of your project and sub-projects, the dependencies between things, and how to build things from their dependencies
2) a breadth-first traversal of the dependency graph, executing the actions
That's what my tools do for my-c.
I haven't yet completed the implementation of the second point you mentioned, threads + processes in parallel, in the right order.
Anyway the dependency engine is up and running.
-
Pascalish syntax, begin, end, why not? ;D
-
Pascalish syntax, begin, end, why not? ;D
Sure, why not. I personally have nothing against curly brackets to delimit blocks of whatever, though. Simple and effective.
-
Digressing a bit there, but I've heard a lot about ninja being so much faster than make. My few (and admittedly limited) tests so far have not shown much difference. I have for instance built KiCad with either ninja or make as a back-end. Build time was nearly identical. It's not a huge project, but it still takes a few minutes to build either way, so it's not small.
One very common misconception is to compare make and ninja using default options for both. This makes no sense: ninja is by default multi-threaded. Make is not. So obviously the time difference with default will be huge. But make can be multi-threaded using the "-j" option, and from again my limited testing, in this case both perform relatively similarly, with differences being marginal.
-
Yup multi threading
For my-c builder I also want clustering capability.
(modules compiled on different computers in parallel)
Distcc like, I mean
It will be next step
-
One very common misconception is to compare make and ninja using default options for both. This makes no sense: ninja is by default multi-threaded. Make is not. So obviously the time difference with default will be huge. But make can be multi-threaded using the "-j" option
You don't say :-)
My .profile has a line export MAKEOPTS="-j64" so I don't have to tyoe it every time.
I suppose I haven't deliberately run a single-threaded make (except maybe to debug something) since I got a dual 2.0 G5 Mac in 2002 or so, but it's completely pointless paying big bucks for a 32 core Threadripper and then using single-threaded make!
Ninja has some other very useful capabilities, such as being able to limit the number of parallel jobs of any one type, for example with the LLVM_PARALLEL_LINK_JOBS Cmake option when building LLVM (only available with ninja back end).
That lets you run 64 parallel gcc jobs, but limit parallel links (which use 10 or 12 GB RAM each) to 8 or 10 or whatever to avoid serious VM thrashing.
-
Ninja has some other very useful capabilities, such as being able to limit the number of parallel jobs of any one type, for example with the LLVM_PARALLEL_LINK_JOBS Cmake option when building LLVM (only available with ninja back end).
That lets you run 64 parallel gcc jobs, but limit parallel links (which use 10 or 12 GB RAM each) to 8 or 10 or whatever to avoid serious VM thrashing.
OK, I'll look into that. Sure make blindly parallelizes stuff when it can without any distinction. So that one could be useful.
-
One very common misconception is to compare make and ninja using default options for both. This makes no sense: ninja is by default multi-threaded. Make is not. So obviously the time difference with default will be huge. But make can be multi-threaded using the "-j" option, and from again my limited testing, in this case both perform relatively similarly, with differences being marginal.
In my case at least, same -j option (24) for both. It might be something related to file access and how processes are spawned on Windows, but the difference was surprising. My friend reported a similar improvement on his Mac.
It has to be said that the Makefiles were the ones generated by that MCUxpresso turd, the CMakeLists.txt files were my doing.