Inconvenient, I think.
Me fail English often, so I wouldn't know
.
My point is, I guess, that the application should not make that kind of determinations, and instead only allocate what it actually needs.
That's a very general statement, but doesn't fit all needs.
I meant that for userspace applications (category GUI-based tools, as opposed to services) exclusively. For services and applications dealing with massive datasets, I mentioned other options in a prior message.
I personally am against the approach that considers letting processes just crash instead of giving the developer an opportunity to handle things gracefully. YMMV. If anything, even if the developer doesn''t know what to do, they can always at least give the user a message about the app not having enough memory available to run. A lot more user-friendly than them having to debug core dumps. Anyway. As you mentioned, in some safety- or security-critical apps, that could be completely unacceptable.
I myself am ambivalent about this. I know why it is done, and that it works for many users, but I have been bitten by it in practice. I do not know if banning overcommit makes sense. I do support users and admins who disable it on their own systems, and believe that doing so on machines with plenty of RAM is a good idea for several different reasons; but as a
global default or non-user-tunable, I am not sure.
And by "not sure", I mean it literally: I am on the fence on this. I never use "not sure" as a weaselly "I disagree".
As to the C standard, I've re-read the section about memory management in C99, and from my interpretation, the discussed behavior, to my humble eyes, looks non-conforming. Read it carefully, and tell me what you think.
It is really the OS kernel (other OSes besides Linux do this; I recall seeing this first on an Unix box) that fails to uphold its promise.
In other words, I believe you can say that these kernels make it impossible for any implementation on them to fulfill the C standard.
I do not blame the C compiler or standard library though, because it is the kernel that is failing in its promise of service, not anything in C; all processes, even those written in raw assembly, are affected by this.
The proper management function is mlock()/mlock2()mlockall().
That's kind of different, then, isn't it? Being aware of that and not using those before accessing the memory is pretty stupid, ISTM. On that basis, I think that disqualifies this example from the discussion of whether NULL is meaningful or not in a pointer context.
Me fail English, so let me re-state my position. Apologies for the wall of text, but I am trying to write in a manner that lets even new C programmers to follow the discussion.
Typical userspace applications, both GUI and non-GUI should not care about overcommit or work around it. If an application can benefit from preallocating memory, or switch between using very little memory but being slow and using a lot of memory to speed up things, I think it should have an easy user tunable to determine that – combining it into a single sliding scale value, from "minimal memory" to "maximum performance", with a bit of tuning how that maps to different details available at compile/build time for different HW arches. The only applications that
should ever need to use mlock() are security-sensitive services, and perhaps some very important low-level services (that'd use mlockall() if the admin so desires, based on some config setting) like remote access services, or critical network infrastructure services.
The problem is that from the userspace perspective, including what the standard C library sees, the interfaces used to request new memory from the OS kernel, and thus passed to the malloc() caller, have fulfilled the promise. It is just that these kernels have learned to LIE because some human users WANT/NEED THAT, and simply kill the target process if they catch the kernel in that lie (having promised memory, but not being able to fulfill that promise).
The file-backed mmap() approach can be used to sidestep the entire issue. This is useful for simulators and other programs dealing with large and/or persistent datasets, and it can provide restart/continue support too in case the process keels over for any reason.
The mlock()/mlockall() approach can be used to enforce the kernel promise. On current Linux, Android, Mac, iOS, and various *BSD systems, the RLIMIT_MEMLOCK resource limit (managed by the system administrator and/or user running the process) defines this limit, per-process.
It is possible for a Linux process to detect and catch this via SIGBUS/SIGSEGV signal handlers, in a per-thread manner, but as it involves manipulating the processor and thread state (particularly including the instruction pointer when the un-fulfillable memory access occurs), it is too complicated to do for standard programs. I would only consider this approach if the process can safely undo/ignore the failure, and throw away (return back to the kernel) large-ish swathes of memory if that ever occurs; an "opportunistic memory-hog that nevertheless behaves nicely". The issue then is that if the kernel runs out of memory after providing the RAM backing to such a nicely behaved memory-hog, it can detect the situation when trying to fulfill the promise it made to some other process. There is no way – and I do not think C even provides any possible way! (POSIX signals could, I guess) – for the kernel to ask back memory, or even to inform processes about memory pressure. So, even this SIGBUS/SIGSEGV handler way is really no solution at all.
It should be noted that this – memory provided by the kernel not really being there – ONLY ever happens when the system is overburdened already.
Okay, so we know there are lots of use cases, where we'd really like to know how much memory we should speculatively allocate, and still be reasonably sure (or completely sure) that the allocations will be fulfilled by the kernel. And also things like how many threads we should use.
My opinion is that this is where we'd need a helper library. Not to ensure the memory we allocate is there, but to give the resource use hints to the process. Perhaps the library could also provide an interface to ensure memory allocations will be there if both the application requests it, and the library allows.
This library would be a
policy manager. It would be the extension of the human users'/admins' will, telling the application how to behave. It should also have trivial tuning ability, something like "Application properties", and it could include CPU and I/O priority tuning.
Unfortunately, that is unlikely to happen at all. It might be incorporated into some mutated form into systemd, so you can do a dbus query for these things, which basically defeats the entire idea, but that is unfortunately the direction current Linux distributions are "progressing".
(If anyone is interested, the entire Init system debacle and all related problems, including parallel and asynchronous service startup, would be
trivial to solve, if system service developers would just add calls to a library that is responsible for providing the service status updates, and provides an interface to manage inter-service dependencies. We've known that for decades, but that hasn't happened, because coordinating something like that among software developers is like herding cats: it won't work, and just leads to hissing (or bikeshedding, really). It seems the best we can do, is cobble together a super-privileged superdaemon that forcibly inserts its tentacles into all over the userspace, rewriting many of those service daemons, with honestly speaking quite poor quality code, and dropping it in from above, using Linux distribution human politics.)
Similarly, it would be
trivial for Linux kernel developers to provide a new syscall, say
mpopulate(), that would take as a parameter a memory range, for the userspace to request the kernel to populate those pages, or return a failure code. A flag parameter would tell the kernel how important the userspace process believes this memory to be, so the kernel can decide whether to try evicting other pages and do other make-more-ram-available stuff or not.
Problem is,
most users do not care. The users who care, can simply tune or turn off overcommit. Many users never encounter this thing, and even if they do, they think the crash occurred for some other reason ("shit this is badly coded" is one). Developers claiming they want this feature is not taken seriously, because their users are not demanding these features. (And the users don't listen to the developers, because humans. I'm a dev too, and I hate being between a stone and a hard place like this, so I fully understand the frustration others have expressed about overcommit issues.)
A number of years ago, I discussed with various compiler and C library developers whether we could provide a function, say
memfill() or
memrepeat(), that would repeat an initial part of the specified buffer into the entire buffer. (It is essentially the third part of the memcpy() - memmove() - memrepeat() triangle; all three functions are implemented in a very similar fashion at the hardware level, and thus actually belong to compiler support rather than C library; i.e. to be available in freestanding environments also, not just in hosted environments.) That failed, for the same reason, even though it alone changes the result of certain Fortran - C comparison benchmarks. The main use of memrepeat() is to initialize arrays with structures or floating-point values, faster than copying those structures or floating-point values in a loop; because it deals with the storage representation alone, none of the value requirements (signaling NANs and such) apply. It is particularly advantageous for odd-sized elements in a continous array.
In summary, whether one can trust malloc() to return NULL when at some point later on, the system cannot fulfill that promise anymore, is not really a C question, but a question about OS kernel services, and how to implement
policy (providing human operator
intent and wishes, and human administrator limits and overrides, to applications).
Home
Help
Search
Login
Register
Poll
