EEVblog® Electronics Community Forum
Products => Computers => Programming => Topic started by: DiTBho on November 05, 2025, 10:02:35 pm
-
I know the person behind this project was hired at Apple to work on the JS garbage collector, a great working experience which I am sure inspired his C compiler
See here (https://github.com/pizlonator/fil-c) :D :D :D
Ok, it's not memory safety at compile time (like Rust?), but it's a step head with C.
-
I've been wondering when this will get discussed here :-) I've been playing with it for a few weeks.
In many cases, running Fil-C with -O1 gives very comparable performance to using GCC with -O0. E.g. on my primes benchmark on my i9 laptop:
1964ms gcc primes.c -o primes -O
3723ms fil-c primes.c -o primes -O
3753ms gcc primes.c -o primes
16334ms fil-c primes.c -o primes
Getting full (runtime) memory safety from unmodified C/C++ code -- at a really pretty low speed penalty -- is a pretty nice option to have, and a great alternative to the huge labour of a complete rewrite.
He's found a heck of a lot of serious packages work with zero modifications. And a lot more with very minor ones which I think could be described as fixing latent bugs.
2:1 slowdown on primes is actually getting towards the common worst case. Looking at his chart of slowdowns on thousands of different programs 4:1 is getting rather outlier and there is a huge cluster between 1:1 and 1.5:1.
-
I expected more interest in this topic... :-//
-
I think the problem is that for code that's engineered to be as safe as possible (e.g. MISRA) you're not getting much extra safety in exchange for using an experimental compiler. OTOH for something that's traditionally been unsafe, say a web browser, you probably want to compile with all the optimisation you can get because those things are slow and bloated enough as it is. So that leaves it as a cool proof-of-concept but a bit difficult to justify fitting into a real-world deployment.
-
you're not getting much extra safety in exchange for using an experimental compiler
It's Clang/LLVM, with just a little extra code added around pointer dereferences.
-
you're not getting much extra safety in exchange for using an experimental compiler
It's Clang/LLVM, with just a little extra code added around pointer dereferences.
Sure, but it's not mainstreamed into LLVM which means there's a risk development may stop at some point. I have a machine with around half a dozen obsolete versions of LLVM installed that are (or at least were) needed to work with various safety/security tools whose development stopped at that version. For example STACK (https://github.com/xiw/stack) needs LLVM 3.4 to work. This has made me a bit gun-shy around such tools.
-
you're not getting much extra safety in exchange for using an experimental compiler
It's Clang/LLVM, with just a little extra code added around pointer dereferences.
Sure, but it's not mainstreamed into LLVM which means there's a risk development may stop at some point. I have a machine with around half a dozen obsolete versions of LLVM installed that are (or at least were) needed to work with various safety/security tools whose development stopped at that version. For example STACK (https://github.com/xiw/stack) needs LLVM 3.4 to work. This has made me a bit gun-shy around such tools.
It doesn't change the language. The absolutely worst thing that can happen is you go back to compiling with gcc or clang.
-
you're not getting much extra safety in exchange for using an experimental compiler
It's Clang/LLVM, with just a little extra code added around pointer dereferences.
That's still in "experimental" stage and you don't know what this extra code could introduce in terms of bugs. Judging from a few of the reported issues ( https://github.com/pizlonator/fil-c/issues ), it looks far from being production-ready.
I personally agree with 5U4GB, I wouldn't trade a robust and standard compiler for something experimental. Unless, of course, for experimental use. No problem with that.
The performance hit seems pretty severe too, as expected. Yes, you said that it's probably not in real-life cases, but do we know? Are there actual benchmarks on real code? I suspect that it probably has a significant impact on compression and decompression code, which is ubiquitous.
I think (again my take here) that it may be more interesting as a test tool than as a "production" compiler. See what it's able to catch if anything, and fix code accordingly. There are actually tons of tools, both static and dynamic, that can do this, and I'm always surprised seeing how very few people use them.
Now, we know rewriting software has a high cost and almost invariably introduces a whole series of new bugs, so keeping code as is and using a different compiler producing "safer" executables looks like a reasonable idea. But it has been tried before and has never caught on, so, we'll see (I wouldn't bet a dollar on Fil-C, but that's just my opinion at the moment).
One thing that should IMO be integrated into compilers for ANY language is to force developers to add input validation of every single function (with a possibiity to escape that for specific performance reasons with extra keywords, making it annoying enough not to be "default"). Of course, that's a different approach, not a "drop-in" replacement: that would require writing better code and re-writing existing code that doesn't comply. The lack of input validation accounts for a gigantic proportion of bugs in general and security-related bugs in particular. I don't even quite remember the last time when I saw a security report that wasn't linked to a lack of input validation.
-
One thing that should IMO be integrated into compilers for ANY language is to force developers to add input validation of every single function (with a possibiity to escape that for specific performance reasons with extra keywords, making it annoying enough not to be "default"). Of course, that's a different approach, not a "drop-in" replacement: that would require writing better code and re-writing existing code that doesn't comply. The lack of input validation accounts for a gigantic proportion of bugs in general and security-related bugs in particular. I don't even quite remember the last time when I saw a security report that wasn't linked to a lack of input validation.
Yes. Input validation related bugs are common because input validation is a lot of work - and not just implementation work or boilerplate, but actual design work. People are lazy so they don't do it. Since early 1990's language designers have been promising languages that somehow make this easier and automatic, and result is that input validation is still not being done - because it cannot be automated. Yet people want to believe that stuff like exceptions or memory safety somehow even relates to input validation (which it doesn't), or even worse, believe it solves input validation somehow so that they don't have to think about it.
Result is that programs still
A) do wrong things (logically), which can be serious, including security issues
B) crash, which is also serious, downtime can be costly.
All that is needed is designing in proper input validation. Which takes time and effort, but can't be avoided. Main purpose is to prevent logical misbehavior and crashes. As a side effect, those nasty memory corruption bugs people like to talk about also basically just vanish.
-
To me the big obstacle here is (aside from the experimental nature) is the lack of compatibility with anything else. It's an ABI breaking change that can't really interoperate with anything else.
That, plus the performance hit I guess would make it a hard sell for many applications. While the most attractive applications are likely either using development techniques that reduce the impact or are better served by using other languages.
One are where something like this could be useful is fuzz testing. This looks much faster than UBSAN and valgrind, and catches more errors than valgrind. So if you have a particular module you want to extensively fuzz test you could use this compiler.
-
Yes. Input validation related bugs are common because input validation is a lot of work
Design by contract and not using text based API's get you most of the way there. Text based API's being broken by design is one of the least recognised major sources of exploits.
SQL is wrong, shell is wrong, html is wrong, printf is wrong, etc. Passing them strings from untrusted sources is too failure prone for humans to handle.
https://cs.ru.nl/E.Poll/papers/strings2018.pdf
Since early 1990's language designers have been promising languages that somehow make this easier and automatic, and result is that input validation is still not being done - because it cannot be automated.
A couple huge exploits in VPN software in recent years were caused by a web monkey being just too plain lazy to use prepared statements (some might say the problem was the lack of input sanitation, but that's confusing the symptom with the disease). The automation was there, the willingness of management to tell them to always do it the annoying way was not.
All that is needed is designing in proper input validation. Which takes time and effort, but can't be avoided. Main purpose is to prevent logical misbehavior and crashes. As a side effect, those nasty memory corruption bugs people like to talk about also basically just vanish.
The get good method. Doing that consistently is of course for lesser mortals who mistakes, if you're really good you can only do it where necessary.
-
One thing that should IMO be integrated into compilers for ANY language is to force developers to add input validation of every single function (with a possibiity to escape that for specific performance reasons with extra keywords, making it annoying enough not to be "default").
There have been attempts in the past to create tools that do taint-tracing, so the tool will tell you where potentially attacker-controlled data will end up. They've been distributed as patches to a particular compiler version (see my previous post about needing to maintain ancient compiler builds) with academic-quality code (works on all three test cases presented in the paper, with the authors mostly being able to understand the diagnostic output if they squint at it just right), so enough to get the conference paper published and that's it. Development then stopped once the paper was published.
It's a pity because every now and then you see tools pop up that go beyond the standard static source code analysis or dynamic checking (valgrind, fuzzers), but they never get past the proof-of-concept stage.
-
One are where something like this could be useful is fuzz testing. This looks much faster than UBSAN and valgrind, and catches more errors than valgrind. So if you have a particular module you want to extensively fuzz test you could use this compiler.
+1, it'd be cool to see it integrated into AFL or similar.
-
It's Clang/LLVM, with just a little extra code added around pointer dereferences.
[...]
It doesn't change the language. The absolutely worst thing that can happen is you go back to compiling with gcc or clang.
In fact, and as it doesn't change the C language it can be used to cook a whole GNU/Linux stage4: experimental, but usable :D
Comparing this compiler with my toy "myC" , well... at least the "Fil-C" compiler is based on a tested framework, while mine is written from scratch from the lexer up, and doesn't require completely rewriting your sources to comply with a more rigid syntax.
myC accepts nothing more than a subset of C/89, which must be strictly MISRA-C/95, with some modifications -> It cannot be used to "cook" any gnu/linux rootfs, as each source must be modified first.
At runtime, thanks to fact that types are monads, the compiler then adds checks not only on the valid range of a pointer, but also on the range of any type -> excellent way to catch bugs!
It makes the binary code 4 times heavier (worse than gcc -o0) and 5 times slower (but better than any instrumentation code). But that's not a problem, since its job is to help squash bugs.
-
I expected more interest in this topic... :-//
It's not a system programming language any more, it's cute but has little use case. For anything old suddenly introducing GC/RC is mostly unacceptable, for anything new it's competing with Go and C# ... and going to lose.
-
Yes. Input validation related bugs are common because input validation is a lot of work
Design by contract and not using text based API's get you most of the way there.
Design by contract is a good thing that unfortunately got too little traction. Of course, it's in Ada and was in Eiffel before that. I don't think Rust has this as a core feature? (Unless I missed it.) Just saw some people came up with "crates" to emulate contracts, which looks kind of like a band-aid.
-
I expected more interest in this topic... :-//
It's not a system programming language any more, it's cute but has little use case. For anything old suddenly introducing GC/RC is mostly unacceptable, for anything new it's competing with Go and C# ... and going to lose.
In my opinion, Fil-C has all the potential to compete with Rust.
-
Rust appeals to language nerds and is a system programming language (ie. not dependent on GC/RC). Neither is true for Fil-C.
Swift embedded is a bigger competitor, with non copyables and borrow checking.
-
One thing that should IMO be integrated into compilers for ANY language is to force developers to add input validation of every single function (with a possibiity to escape that for specific performance reasons with extra keywords, making it annoying enough not to be "default").
I don't know what that would even look like. The problem with input validation is that valid input is if you are lucky in the intent of the developer, but let's be honest the biggest bugs come when the developer is mistaken about what constitutes valid input. Invalid input handling is even harder than validation itself.
It's obviously important an a lot of tools have been developed to help with it. I just don't know how you make it a generic compiler check and still be useful. One of the main goals of OOP, especially as implemented in say C++ is that after construction an object is supposed to be valid, and and stay valid for the lifetime of the object. Of course valid as "internally consistent" is not the same thing as "usable for my current purpose" and even then, maintaining useful meanings of valid is not always possible. At least you can often use it to force error handling when trying to manipulate an invalid object.
Languages with Error/Maybe return types can force you to explicitly check if a return object exists and prevent accessing it if it's not valid but they can't force you to do something sensible if it's not.
Languages with memory safety features can prevent you from accessing an invalid pointer or out of bounds access. But again there are lots of other ways input can be invalid that cant be detected.
C++ tried to implement one of the more powerful input validation systems in the form of contracts, which give explicit syntax for pre- and post- condition checks, but I'm not sure how useful it really is going to be. I somehow doubt that if you made a rule "every c++ function must specify at least one pre-condition that references each input value" that it would meaningfully improve things beyond what other languages get by default.
-
In my opinion, Fil-C has all the potential to compete with Rust.
Why? How? The overhead is much worse than rust, and the guarantees are basically the same. The major difference is that fil-c does runtime checking which obviates the need for "unsafe" overrides but defers detecting detectable errors.
If they can make fil-C play nicely with other languages (it's totally unclear to me if that's possible) the it could be a nice way to check existing code, but if you want that kind of protection I don't know why you wouldn't use a language that does so with less overhead and earlier detection of errors.
IMHO, the languages that have a much better approach than rust are chandlar caruth's Carbon and herb sutters cpp2. These languages preserve extremely high levels of compatibility with unchanged C/C++ code, and heavily leverage existing developer knowledge while letting you develop new code that easily uses the "best practices" that we have learned over the last 30 years. They both avoid my main complaint about rust that its developers think they need to reinvent absolutely everything.
-
Carbon (https://herecomesthemoon.net/2025/02/carbon-is-not-a-language/)
-
Speaking of Carbon and many similar ideas, it's like roleplaying games in the 1980s, almost everyone who played them at some point thought "you know, this system has a number of problems, I bet I could do a better one!", after which they'd spend anything from a few weeks through to months and occasionally even erratic effort over years futzing around with their new improved universal system that got more and more complex as it progressed (or failed to) until eventually enthusiasm waned and it fell by the wayside. My nephew goes out RPG'ing every week or two and do you know what he plays? D&D, the Dartmouth BASIC of RPGs. I don't know how he'd even find anyone playing the fancier systems, and none of the revolutionary we-can-do-it-better-ourselves ones have survived.
-
In my opinion, Fil-C has all the potential to compete with Rust.
Why? How? The overhead is much worse than rust, and the guarantees are basically the same. The major difference is that fil-c does runtime checking which obviates the need for "unsafe" overrides but defers detecting detectable errors.
If they can make fil-C play nicely with other languages (it's totally unclear to me if that's possible) the it could be a nice way to check existing code, but if you want that kind of protection I don't know why you wouldn't use a language that does so with less overhead and earlier detection of errors.
IMHO, the languages that have a much better approach than rust are chandlar caruth's Carbon and herb sutters cpp2. These languages preserve extremely high levels of compatibility with unchanged C/C++ code, and heavily leverage existing developer knowledge while letting you develop new code that easily uses the "best practices" that we have learned over the last 30 years. They both avoid my main complaint about rust that its developers think they need to reinvent absolutely everything.
Makese no sense.
Fil_C is "C", and it's an attempt to improve C and does not need to play anything with "other languages".
In my opinion, the overhead is not that worse than Rust
The Fil_C also implements a garbage collector.
This may be a problem for things like "Kernels" and "embedded stuff".
But you can disable it.
-
Speaking of Carbon and many similar ideas, it's like roleplaying games in the 1980s, almost everyone who played them at some point thought "you know, this system has a number of problems, I bet I could do a better one!", after which they'd spend anything from a few weeks through to months and occasionally even erratic effort over years futzing around with their new improved universal system that got more and more complex as it progressed (or failed to) until eventually enthusiasm waned and it fell by the wayside. My nephew goes out RPG'ing every week or two and do you know what he plays? D&D, the Dartmouth BASIC of RPGs. I don't know how he'd even find anyone playing the fancier systems, and none of the revolutionary we-can-do-it-better-ourselves ones have survived.
well ... improving C@KISS(1) is no easy feat; every branch of development could contribute something, or contribute nothing at all.
(1) and there's always the risk of betraying the initial "KISS philosophy" ...
and generating very complicated things
-
Makese no sense.
Fil_C is "C", and it's an attempt to improve C and does not need to play anything with "other languages".
Maybe in your world. In the real world C is used to build libraries called from other languages, and has to call external code as well. Fil-C implements some of those for you such a system calls, and does provide a foreign function interface that can be used for some purposes (they suggest constant time crypto primatives) but it comes with a bunch of warnings.
In my opinion, the overhead is not that worse than Rust
That's not an opinion it's an incorrect fact. If you need the memory integrity guarantees of Fil-C, you care about performance, and you are writing new code I can't imagine choosing Fil-C over rust for technical rather than political reasons. The big advantage Fil-C has is the ability to keep using existing battle tested code with a tolerable overhead.
The Fil_C also implements a garbage collector.
This may be a problem for things like "Kernels" and "embedded stuff".
But you can disable it.
The GC is fine, but as it stands Fil-C can't be used for embedded development and I doubt will be adapted to do so. First off, right now it only supports 64 bit platforms with a full hosted libc. Looking over the implementation, it looks like it could be adapted to 32 bit, and maybe with a free standing implementation, but it relies on malloc pretty heavily.
As it stands it would need a way to run unsafe code in an operating system or embedded platform in order to do hardware access. That could obviously be added to the runtime in the same way that the runtime currently provides a syscall wrapper but that would require a major change to embedded software that currently assumes it can just cast a hard coded integer to a pointer and start poking hardware registers.
Memory overhead would be a big issue on embedded platforms as well. Every data structure that exists in memory (including on the stack, but excluding purely local variables) that ever contains a pointer ends up with a dynamically allocated shadow object of equal size that contains the capability information. Note that this means that a 1024 byte stuct with a single pointer has an entire 1024 byte shadow object. Those objects are lazily allocated with malloc, and while you could statically allocate them, then a great many more objects would have that overhead. I'm pretty sure this would be a complete non starter for most freestanding embedded applications.
Finally, if you are already required to confirm to something like MISRA, the benefits of the memory protection afforded by either Rust or Fil-C are not that big. Especially Fil-C with it's focus on runtime checking is really targeting applications where runtime panics are considered acceptable. The real focus is providing protection from security vulnerabilities caused by memory corruption, not ensuring correct opeation.
-
Maybe in your world. In the real world
well, "myC" is a researching object in many ways, a toy that as such makes no sense in development environments.
True, but if "myC" is hobby, when you say "real world" ... I write and test avionics software for a living, working only with C and Ada assembly.
"my world" is weird and different from the typical C application, and our "fly applications" are usually written in Ada and rarely written in C, while the bootloader, the BSP, the iBSP, all the drivers and the kernel all written in C and compiled by GreenHills C/99 cc.
I think you thought about the "use case" in which Python calls things written in C.
Never don, but I think you need a wrapper between them, so it would be the same for Fil-C, I guess.
Nerver thought about that, I don't have much experience in "mixing" various languages.
I have a rough answer when I compile a piece of application in Erlang and link it with modules written in C because I had to debug it from the bottom, and to write a wrapper.
The same between ghdl and C, but it worked better than ErLang and C.
-
I can't imagine choosing Fil-C over rust for technical rather than political reasons. The big advantage Fil-C has is the ability to keep using existing battle tested code with a tolerable overhead.
The goal I had in mind is to take a minimal GNU/Linux rootfs (all written in C) and recompile it from scratch with Fil-C!
It's something I do regularly for routers.
The result is a slightly slower rootfs, but one that doesn't suffer from memory leaks (Fil-C with the garbage collector enabled) or off-by-one bugs.
-
"my world" is weird and different from the typical C application, and our "fly applications" are usually written in Ada and rarely written in C, while the bootloader, the BSP, the iBSP, all the drivers and the kernel all written in C and compiled by GreenHills C/99 cc.
How good is GreenHills in terms of preserving source code semantics? For example if you compile 'for(i=0; i<10; i++)' will it rewrite it to 'for(i=0; i!=10; i++)' like gcc will?
-
The Fil_C also implements a garbage collector.
This may be a problem for things like "Kernels" and "embedded stuff".
But you can disable it.
I don't think you can. At least not if you want to free memory. The way it works is that when you call free() the objects capability block marked "freed". That causes any future access to panic. But the memory can't be reclaimed yet. When doing a mark and sweep, any pointers found that reference that capability are redirected to the global "freed object". Only after a full sweep guarantees all such ponters at cleaned can the memory be reclaimed. Some without GC, even explicitly freed memory will not be reclaimed.
I can't find in the documentation how this works with stack allocations, but I think it means that all local variables and alloca() allocations that potentially escape the scope must get promoted to dynamic allocations, and they are automatically freed (but not yet reclaimed) when the function returns.
-
I can't imagine choosing Fil-C over rust for technical rather than political reasons. The big advantage Fil-C has is the ability to keep using existing battle tested code with a tolerable overhead.
The goal I had in mind is to take a minimal GNU/Linux rootfs (all written in C) and recompile it from scratch with Fil-C!
DJB (Daniel J Bernstein) is attempting to recompile all of Debian with Fil-C:
https://cr.yp.to/2025/fil-c.html (https://cr.yp.to/2025/fil-c.html)
Scroll down to "Debian using Fil-C" to see his notes on that effort.
Other links:
https://news.ycombinator.com/item?id=45788040 (https://news.ycombinator.com/item?id=45788040)
https://www.theregister.com/2025/11/05/djb_tries_filc_and_approves/ (https://www.theregister.com/2025/11/05/djb_tries_filc_and_approves/)
-
That makes a lot of sense to me. I'd much rather use a fil-C compiled coreutils than a rust re-implementation for instance even if the latter is faster. Much of that code was already fast enough for me on a 486 and I'd rather not have weird incompatibilities and bugs from a new implementation. Stable and widely used C code pretty much by definition has no bugs that I care about, but it could have security issues in cases way outside normal operation. Compiling with Fil-C would protect against some of those while preserving the functionality.
I just don't think anyone writing new code is going to want to pay the performance price when they have alternatives that are faster.
I do hope this leads to more reconsidering about whether memory safety is even possible in C/C++. I think a lot of people would have said that it's not, or that the overhead would be a python like-50x, or that it would be at best statistical like pointer tagging.
-
Qt can be built with fil-c
https://git.qt.io/cradam/fil-qt
-
"my world" is weird and different from the typical C application, and our "fly applications" are usually written in Ada and rarely written in C, while the bootloader, the BSP, the iBSP, all the drivers and the kernel all written in C and compiled by GreenHills C/99 cc.
How good is GreenHills in terms of preserving source code semantics? For example if you compile 'for(i=0; i<10; i++)' will it rewrite it to 'for(i=0; i!=10; i++)' like gcc will?
Not sure how this can be true, what if 'i' were modified >= 10 within the loop?
-
"my world" is weird and different from the typical C application, and our "fly applications" are usually written in Ada and rarely written in C, while the bootloader, the BSP, the iBSP, all the drivers and the kernel all written in C and compiled by GreenHills C/99 cc.
How good is GreenHills in terms of preserving source code semantics? For example if you compile 'for(i=0; i<10; i++)' will it rewrite it to 'for(i=0; i!=10; i++)' like gcc will?
Not sure how this can be true, what if 'i' were modified >= 10 within the loop?
Well quite obviously in that case GCC won't implement it with non-equality comparison.
This only matters if you want to harden the application against memory corruption, which (e.g. rad-hard software) is actually pretty complicated thing to do and I'm not sure if micromanaging that through C + non-optimizing compiler is a good idea at all.
-
"my world" is weird and different from the typical C application, and our "fly applications" are usually written in Ada and rarely written in C, while the bootloader, the BSP, the iBSP, all the drivers and the kernel all written in C and compiled by GreenHills C/99 cc.
How good is GreenHills in terms of preserving source code semantics? For example if you compile 'for(i=0; i<10; i++)' will it rewrite it to 'for(i=0; i!=10; i++)' like gcc will?
Not sure how this can be true, what if 'i' were modified >= 10 within the loop?
Well quite obviously in that case GCC won't implement it with non-equality comparison.
This only matters if you want to harden the application against memory corruption, which (e.g. rad-hard software) is actually pretty complicated thing to do and I'm not sure if micromanaging that through C + non-optimizing compiler is a good idea at all.
It clearly isn't and it looks off-topic as Fil-C is only about "memory safety" as far as I've understood, not about general "hardening" of compiled code.
Fil-C being a modification of Clang (if I got it right), there's nothing it can do on the optimization level as optimization is not handled in the front-end when using LLVM, and I think the same applies to GCC. So I'm not sure any amount of massaging in the front-end would generate an IR that guarantees that the optimizer won't replace a "greater or equal" with just an "equal" test if they are functionally identical.
That question of hardening code on the assembly level is interesting though but I don't know of many compilers that actually bother with that, at least not general-purpose, non-commercial ones.
-
"my world" is weird and different from the typical C application, and our "fly applications" are usually written in Ada and rarely written in C, while the bootloader, the BSP, the iBSP, all the drivers and the kernel all written in C and compiled by GreenHills C/99 cc.
How good is GreenHills in terms of preserving source code semantics? For example if you compile 'for(i=0; i<10; i++)' will it rewrite it to 'for(i=0; i!=10; i++)' like gcc will?
Not sure how this can be true, what if 'i' were modified >= 10 within the loop?
Then the loop will run forever, and off into the weeds doing who knows what sort of damage, since it's been rewritten from "make sure i stays within bounds" to "allow i to be any value except this one out of four billion". So if you've got any situation where data corruption or faults can occur, which in practice is all instances where a computer is used, then gcc will transform code with safety checks into unsafe code. It's not just that case, it will remove all manner of safety and bounds checks if it decides they're unnecessary, in the worst case doing things like outputting code that segfaults when run because it's decided that a null pointer check is redundant. No warning that it's done it, just an instant crash when run.
Thus my question about how good GreenHills is for preserving code semantics.
-
"my world" is weird and different from the typical C application, and our "fly applications" are usually written in Ada and rarely written in C, while the bootloader, the BSP, the iBSP, all the drivers and the kernel all written in C and compiled by GreenHills C/99 cc.
How good is GreenHills in terms of preserving source code semantics? For example if you compile 'for(i=0; i<10; i++)' will it rewrite it to 'for(i=0; i!=10; i++)' like gcc will?
Not sure how this can be true, what if 'i' were modified >= 10 within the loop?
Then the loop will run forever, and off into the weeds doing who knows what sort of damage, since it's been rewritten from "make sure i stays within bounds" to "allow i to be any value except this one out of four billion". So if you've got any situation where data corruption or faults can occur, which in practice is all instances where a computer is used, then gcc will transform code with safety checks into unsafe code.
No, that's not true at all.
If there is a statement within the loop that potentially makes i >= 10 then either the main loop control will not be modified, or the interior assignment will have it's own exit test.
It is the prime job of compilers to maintain the meaning of the code, as written according to the rules of C, and gcc and llvm are very good at doing that.
It's not just that case, it will remove all manner of safety and bounds checks if it decides they're unnecessary, in the worst case doing things like outputting code that segfaults when run because it's decided that a null pointer check is redundant. No warning that it's done it, just an instant crash when run.
That can't happen unless you change the value of a pointer behind the compiler's back e.g. by an asynchronous process such as hardware or another CPU changing a memory location that you have incorrectly not marked as "volatile".
-
Thus my question about how good GreenHills is for preserving code semantics.
No, it's not. All compilers preserve code semantics.
You are asking about something different: code that is resistant to external corruption. Might be a useful tool, although I share the intuition that doing so with C is nonsensical. And I meant that literally. I don't mean a "dumb" or "bad" idea, but one that is logically inconsistent and cannot be made to correspond to anything sensible.
-
It's not just that case, it will remove all manner of safety and bounds checks if it decides they're unnecessary, in the worst case doing things like outputting code that segfaults when run because it's decided that a null pointer check is redundant. No warning that it's done it, just an instant crash when run.
That can't happen unless you change the value of a pointer behind the compiler's back e.g. by an asynchronous process such as hardware or another CPU changing a memory location that you have incorrectly not marked as "volatile".
Try compiling something like the following with gcc and see what happens:
int foo( struct thing *ptr ) {
const int value = thing->value;
if(ptr==NULL) return -1;
if(is_inverse(thing->data_in)) [...]
if(value == 0) [...]
}
No compiler warning, it just segfaults when run because the NULL pointer check has been removed because gcc sees the deref at the start, long before the value is used (which follows the NULL pointer check), and decides that since you've deref'd it there it can remove the NULL pointer check and emit code that it knows will crash when run.
In this case the behaviour is extremely obvious because it's been written to be so, but in a lot of code it's not obvious at all, or the segfault is extremely rare and hard to diagnose. In once case several folks spent some hours trying to figure out how gcc had decided to remove a NULL pointer check and could never figure it out. There was no obvious cause for it, it simply output code that had the NULL pointer check removed. The code eventually got rewritten with some volatile hack to make sure the pointer check wasn't quietly deleted.
And then there's the nonnnull attribute, which in gcc does the opposite of what you'd expect: It doesn't warn if you pass in a NULL value for a nonnull parameter, but it will remove NULL pointer checks in your code.
Expected behaviour: Either see that the value isn't used until after the pointer check and emit code as expected, or warn that there's a reference/check conflict in the code.
-
Thus my question about how good GreenHills is for preserving code semantics.
No, it's not. All compilers preserve code semantics.
How is a compiler that rewrites for(i=0;i<10;i++) into for(i=0;i!=10;i++) preserving code semantics?
-
Try compiling something like the following with gcc and see what happens:
int foo( struct thing *ptr ) {
const int value = thing->value;
if(ptr==NULL) return -1;
if(is_inverse(thing->data_in)) [...]
if(value == 0) [...]
}
No compiler warning, it just segfaults when run because the NULL pointer check has been removed because gcc sees the deref at the start, long before the value is used (which follows the NULL pointer check), and decides that since you've deref'd it there it can remove the NULL pointer check and emit code that it knows will crash when run.
OK, but an imaginary "literal compiler that does exactly what is written with no 'optimization'" (which again is a poorly formed concept) would _also_ segfault. Only a very weird optimizing compiler which defers evaluation of `const int value = thing->value;` to after the null check but still performs the null pointer check would return successfully here.
The code you wrote is dangerous, but not for the reason you say. The problem is that even if "thing" is NULL, "thing->value" can result in a valid memory address. And yes, this sort of thing _would_ be caught by Fil-C. Not that it would prevent a trap, since the code is defective, but if the supplied pointer is NULL, it would have empty capabilities. There would be no danger that it would inadvertently (due to pointer arithmetic) turn into a valid memory address and cause corruption to an unrelated object. Instead the code will either return -1 or trap without allowing memory corruption.
Expected behaviour: Either see that the value isn't used until after the pointer check and emit code as expected, or warn that there's a reference/check conflict in the code.
That doesn't make sense. The value *is* used before the pointer check.
In once case several folks spent some hours trying to figure out how gcc had decided to remove a NULL pointer check and could never figure it out. There was no obvious cause for it, it simply output code that had the NULL pointer check removed. The code eventually got rewritten with some volatile hack to make sure the pointer check wasn't quietly deleted.
Deciding to leave your code defective is not a good choice. The obvious reason for gcc to remove a check is that previous code already used the value, but did so silently. By plastering volatile around until it doesn't crash, you have simply decided to leave the real bug in place. There are a ton of tools available that will help diagnose the real problem, for instance ubsan would likely find it immediately. Or use a different language! Most languages don't have C/C++ style undefined behavior which is the problem here, not the compiler optimziing.
And then there's the nonnnull attribute, which in gcc does the opposite of what you'd expect: It doesn't warn if you pass in a NULL value for a nonnull parameter, but it will remove NULL pointer checks in your code.
That's funny because I that's exactly what I expect. It's an optimization flag to tell the compiler to optimize assuming the value passed is not NULL. And it _does_ emit a warning if you explicitly pass NULL to a non-null parameter if you use -Wnonnull or -Wall.
I could certainly see an argument for an attribute that requested the warnings but didn't add the optimization. But it is not what I would expect that attribute to do. In any case, nonnull is not part of C, it's a controlled non-conformity for gcc. A compiler running in a conformant mode will only remove the check when it's redundant based on the program text.
How is a compiler that rewrites for(i=0;i<10;i++) into for(i=0;i!=10;i++) preserving code semantics?
Well, to be annoyingly precise, it doesn't do that. It chooses a possible implementation of the former in the target machine code, and it does so within the constraints of the language (you aren't allowed to modify memory from an independent context). It may pick an implementation that "looks" more like the latter, but technically that's only in your imagination. And that in a nutshell is my point about people who complain about C optimizer "changing" their program. Compilers _have_ to change your program for it to be able to run.
-
Try compiling something like the following with gcc and see what happens:
int foo( struct thing *ptr ) {
const int value = thing->value;
if(ptr==NULL) return -1;
if(is_inverse(thing->data_in)) [...]
if(value == 0) [...]
}
No compiler warning, it just segfaults when run because the NULL pointer check has been removed because gcc sees the deref at the start, long before the value is used (which follows the NULL pointer check), and decides that since you've deref'd it there it can remove the NULL pointer check and emit code that it knows will crash when run.
That is not even close to being valid compilable code. I'm a compiler engineer and I have no idea what you think you're demonstrating there.
In this case the behaviour is extremely obvious because it's been written to be so, but in a lot of code it's not obvious at all, or the segfault is extremely rare and hard to diagnose. In once case several folks spent some hours trying to figure out how gcc had decided to remove a NULL pointer check and could never figure it out. There was no obvious cause for it, it simply output code that had the NULL pointer check removed. The code eventually got rewritten with some volatile hack to make sure the pointer check wasn't quietly deleted.
Volatile is not a hack, it is essential information about your code.
And then there's the nonnnull attribute, which in gcc does the opposite of what you'd expect: It doesn't warn if you pass in a NULL value for a nonnull parameter, but it will remove NULL pointer checks in your code.
It is a promise from you to the compiler that the passed value will not be NULL. On the caller side gcc will warn you if it knows that the value is null, but it is frequently not possible to know, so it comes down to being the programmer's responsibility. On the called side, naturally it can assume that the pointer is not null, because you told it so.
-
Then the loop will run forever, and off into the weeds doing who knows what sort of damage, since it's been rewritten from "make sure i stays within bounds" to "allow i to be any value except this one out of four billion".
Yes, the design of the C language is as such that random memory corruption can lead to more random memory corruption. The primary way to deal with this is avoidance of the original random memory corruption, in which case Fil-C is actually very relevant.
Rad-hard (or memory corruption hardened) defensive programming is a valid field of study (and practice) on its own, but I'm also sure that just throwing C at a non-optimizing compiler isn't the right way to do it at all - your amateurism is showing. For starters, non-optimizing compiler generates larger code, and as such, more surface area for corruption. Optimizations, like any other implementation choice, increase or decrease safety against memory corruption depending on what they exactly do.
Defensive programming against memory corruption is hard, with a lot of case-by-case thinking, and proving paths of program execution. Expecting a more literal and less optimized compiler to automatically achieve any of that is a textbook example of false sense of security.
Compilers preferred in the safety-critical fields likely optimize less only because for safety-critical, complexity is enemy in general and keeping things unchanged is positive because then you don't introduce new bugs. Therefore, they avoid trying to squeeze last bits of performance, not because there is something fundamentally wrong in that, but because the risk of any new feature having bugs is real.
-
I'm not aware of any programming language that is designed to deal with hardware data corruption. That is what ECC, Lock-step processing, redundant voting are designed to mitigate. If you want C to assume that hardware corruption may be an issue, then all variables must be considered volatile, but this doesn't buy you much from a functional stand point, the program is still going to act incorrectly with corruption.
-
Try compiling something like the following with gcc and see what happens:
int foo( struct thing *ptr ) {
const int value = thing->value;
if(ptr==NULL) return -1;
if(is_inverse(thing->data_in)) [...]
if(value == 0) [...]
}
No compiler warning, it just segfaults when run because the NULL pointer check has been removed because gcc sees the deref at the start, long before the value is used (which follows the NULL pointer check), and decides that since you've deref'd it there it can remove the NULL pointer check and emit code that it knows will crash when run.
In my mind it is crazy to think that this code would not segfault given a null pointer passed in. You are literally de-referencing it before the null check (assume thing and ptr are supposed to be the same)
-
I don't know what 5U4GB was expecting. (Bruce: let's assume the code is fixed and compilable.)
The pointer is dereferenced first before checking it against NULL, so of course the code will crash when executed (at least on a platform where derefencing a NULL pointer causes a crash).
So what were you expecting? A warning about the check against NULL happening after dereferencing? That could be nice, but way above what GCC (and most compilers) are designed to warn about.
That's where using an external static analyzer comes in handy. Even just the open-source CppCheck catches it with no issue:
ohmynull.c:18:21: warning: Either the condition 'ptr==NULL' is redundant or there is possible null pointer dereference: ptr. [nullPointerRedundantCheck]
const int value = ptr->value;
^
ohmynull.c:20:9: note: Assuming that condition 'ptr==NULL' is not redundant
if(ptr==NULL) return -1;
^
ohmynull.c:18:21: note: Null pointer dereference
const int value = ptr->value;
And of course I also highly recommend to validate function arguments and validate them first thing before doing anything with the arguments. But admittedly, even if the above example is simplified, this is the kind of things that happen all the time on non-trivial code if one doesn't strictly follow this rule of validating all inputs FIRST THING upon function entry rather than on an "ad-hoc" manner, which is bound to cause overlooks like the above. It definitely happens.
-
Expected behaviour: Either see that the value isn't used until after the pointer check and emit code as expected, or warn that there's a reference/check conflict in the code.
Yes, it has been discussed recently. Indeed, every optimization option of this kind is also accompanied by a matching diagnostic option. It is just that in the current implementation the optimization code is out-of-sync with the diagnostic-issuing code. The latter is somewhat behind. But once it catches up, the compiler should issue a diagnostic when it performs this optimization.
-
Expected behaviour: Either see that the value isn't used until after the pointer check and emit code as expected, or warn that there's a reference/check conflict in the code.
Yes, it has been discussed recently. Indeed, every optimization option of this kind is also accompanied by a matching diagnostic option. It is just that in the current implementation the optimization code is out-of-sync with the diagnostic-issuing code. The latter is somewhat behind. But once it catches up, the compiler should issue a diagnostic when it performs this optimization.
The "expected behaviour" is just plain wrong.
The definition of C says that when things are separated by semicolons then they must be performed "as if" in the order written. The "const int value = ptr->value;" must be done before the null pointer check, and either crash or else if it doesn't crash then the pointer can henceforth be assumed to not have been NULL.
Deferring that access until after the NULL check would be a violation of the semantics of C.
-
The "expected behaviour" is just plain wrong.
Agreed, it's not just wrong, it's nonsensical.
The definition of C says that when things are separated by semicolons then they must be performed "as if" in the order written. The "const int value = ptr->value;" must be done before the null pointer check, and either crash or else if it doesn't crash then the pointer can henceforth be assumed to not have been NULL.
Well, "crashing" isn't observable behavior so it's perfectly acceptable for the compiler to optimize by defer the read access. But it has to be clear that it's an as-if optimization: if the pointer is valid it's OK to to defer, if it's invalid it's not defined anyway.
-
Well, "crashing" isn't observable behavior so it's perfectly acceptable for the compiler to optimize by defer the read access. But it has to be clear that it's an as-if optimization: if the pointer is valid it's OK to to defer, if it's invalid it's not defined anyway.
It's not really about crashing being an observable behavior or not, "crashing" is just UB.
But yes, since 'value' is a local variable, the compiler would be free to optimize accesss until just before value is actually used in the code, but this would of course be optimization, not *guaranteed* behavior by any means. Expecting the compiler to generate code that derefences the pointer *just right* before the dereferenced value is used is the unfortunately nonsensical part.
Only if the pointer was volatile-qualified would we have a *guarantee* that it's dereferenced right where 'value' is declared. Otherwise, you just can't know.
It's interesting to see that quite a few people tend to get annoyed with C compilers when these do not generate compiled code that strictly matches the source code (for instance by reordering statements when it can), and that there are others (maybe even the same ones) that will get annoyed with C compilers when these actually do respect the original order, if this isn't convenient for the developer.
IMO the only reasonable thing to do here for a compiler is to emit a warning, and as I said, any decent static analyzer will, and I highly recommend using that. The reason both GCC and Clang do not emit any warning for such a piece of code is that optimizations happen at a later stage after code transformation to IR, and at this point, it's too late to emit "static analysis" warnings. It's a problem that I have also raised a ticket for in the past and just got answered that it wasn't possible to emit warning about the code at the optimization stage. Now of course, nothing prevents a compiler to have better static analysis before optimization, and as I showed, devent static analysis will catch this kind of issue, but that would also 1/ take a lot more compile time and 2/ probably annoy a lot of users, since more thorough static analysis almost invariably means a lot more warnings that they have to go through. So the question becomes more like: "why doesn't GCC behave exactly like I would like it to behave in my particular use cases?".
Anyway, for those who haven't, I really suggest trying CppCheck on their code, just to see what it comes up with. You'll probably be surprised.
-
The definition of C says that when things are separated by semicolons then they must be performed "as if" in the order written. The "const int value = ptr->value;" must be done before the null pointer check, and either crash or else if it doesn't crash then the pointer can henceforth be assumed to not have been NULL.
That's the definition of the behavior of "abstract C machine". Implementations are never required to literally implement an "abstract C machine". "Abstract C machine" is only used to define the canonical observable behavior, nothing else.
As it has been stated many times, an equivalent definition of UB is "The compiler is permitted to translate the code under assumption that conditions that lead to UB never occur". For this reason, the code that you quoted is a direct order to the compiler to translate it under assumption that pointer is never null. This is a hard guarantee that the author of the code gives to the compiler: "I will never call it with a null argument". Which means that the `if(ptr==NULL) return -1;` can be immediately discarded as a "conditional statement with an 'always false' condition".
We have many optimization-oriented features in C, which are based purely on promises that the code's author gives to the compiler (and might accidentally break, leading to UB): `restrict`, `const`, `[static]` etc. These are quite explicit. But there are also many more implicit promises: not overruning array sizes, not making null pointer accesses and so on. But formally, it is the same thing. There's a hard promise hiding behind any UB, which the compiler is allowed to rely upon the same way it is allowed to rely on `restrict`.
Every time you write a function that begins with
void foo(T *ptr) {
*ptr;
...
for virtually all means and purposes it is the same as declaring the function as
void foo(T ptr[static 1]) {
...