Several quanstions about Linux and memory management

[Logan]

Hi guys.
One thing I really hate about Linux is the OOM-killer and memory-over-commit, it makes me feel unsettled, unknown which process will get a sudden murder in next second. (an accurate joke here: https://lwn.net/Articles/104185/)
I tried to disable over-commit, but any program won't start when there's more than 1GB free RAM. I guess all programs for Linux just make use of this stupid design and commit as much as the developer can ever imaging.
So I came up another idea: keep over-commit disabled, use ZRAM as swap, and set over-commit ratio to 100. So the commit limit will be much larger, with actual space in case they use them, and no wasted HDD space at all.
Giving the average ZRAM compression ratio of 3:1, I should set the SWAP space limit to 3x free RAM(or less) when idle.
But I still have some questions:
1. Will my method work? Any cautions for it?
2. Does anyone know the average ratio of [used/commited] memory for common usage?
3. What's the standard/easy way to setup ZRAM and set it as the only SWAP space?
I'm still rather new to Linux, hoping someone can help.
Thank you all.

[Siwastaja]

So you are rather new to linux, but you know it has a bad design flaw, and even further, you have outsmarted all the designers of this OS that is widely in use in desktop, servers and mobile? And you think you could fix it by a few simple changes yet for some reason such chances don't come in by default.

You are aware that the link you posted is from 2004? The OOM killer was indeed infamous, but are things still that bad?

I suggest we take one step back, and you'd start by describing what your actual problem is, when running with defaults. Are you seeing out-of-memory errors, are you seeing random apps killed?

Then we could work towards solving the actual problem, if that exists, that is.

[golden_labels]

Logan:
So, instead of letting the OOM killer eliminate the offending process as soon as possible, you are willing to bring your whole system to its knees by forcing a memory leak into swap. After which the OOM killer will kick in anyway and kill the process. What’s the point?

The mention of HDD in a context of normal system operation suggests something even worse: misunderstanding what swap is and treating it as “free RAM”. Which it is not. At least not since when ancient dinosaurs were roaming around.

As Siwastaja asked: what’s the actual problem you are facing right now? Exactly.

Over-commit causes no problems in normal situations and allows efficient memory management. It does not cause any new uncertainity; at most shifts existing one in time — to the future. In either case that uncertainity should not be of concern. In a healthy system memory exhaustion doesn’t normally happen, except for a  runaway leak in which case killing the misbehaving process is exactly what you want.

[Logan]

Thank you guys for replying.
There's no problem in normal operation yet, but if someone just use everything default, and don't care what happens to their system, then he/she will just use IOS on the phone, and will not be in this forum.
Further, some OSes (e.g, Windows, Solaris) don't have this OOM-killer thing at all, and they works fine.

[Siwastaja]

I have hard time deciphering that last post.

Are you saying that by surfing web with linux under default overcommit and swap settings, the system and/or important applications crash due to out-of-memory so regularly (and you are expecting to see that soon, just not yet), that they switch to another OS due to that?

Did I get that correctly?

All systems must deal with the actual memory running out in one way or another. It's always a difficult situation, and really the only good way of dealing with this is have enough memory for the task you are doing, and fix buggy programs.

It is true linux used a fairly "stupid" way of dealing with this, some 15 years ago. I haven't seen any discussion about it in maybe last 5 years. I assume they have fixed it to behave better, a long time ago. I.e., the OOM killer now has enough intelligence of having good chances to kill the application causing the problem, not some random app.

I remember Windows also behaving stupidly, such as the whole system crashing in out-of-memory situation, but that is also a long time ago and I think they pretty much fixed it as well. It may be at some point in history, the behavior was better in Windows than it was in linux, but I'm not sure. Currently, I think basic memory management works quite well in all major operating systems (Windows, linux, mac OS) because it's such fundamental part and while non-trivial, not rocket science either.

[bd139]

Overcommit is a commonly used feature which is why stuff falls over fairly quickly, including core bits of the glibc library which mmap files into RAM. In fact I use it quite regularly for sparse data. I can allocate a 48Gb array with intent to use only 1Gb of it on a 4Gb node.  It'll only blow the process away if you hit a page that can't be mapped into real RAM.

Also regarding swap, it's there to hold one kernel core dump and that's about it these days. If you're touching it then you need to buy more RAM or fix whatever is eating it.

OOM killer is there only to protect the kernel. Without the kernel, go home.

[Logan]

I don't know which happened first, but seems in Linux world, programs commit insane amount of memory as a habit, kernels tolerate and connive them, good cooperation right? 
This is one particular thing Windows do better, by not over commiting, forcing those developers to claim only what they need. And if OOM do happens, just refuse giving out more. Much better and easier.

[bd139]

That’s not better. Just different. There are trade offs.

Everything will crap itself when it runs out of memory.

How and when it is mapped is the only difference between overcommit and conservative memory management.

[Siwastaja]

And if OOM do happens, just refuse giving out more. Much better and easier.

Guess what this leads to?

Let's imagine your "simple" allocation system in a multitasking OS.

Yes, your PC runs gazillion of software, all the time, some of which are important background tasks. All these do memory allocations every now and then, during normal operation.

Now some badly designed runaway software allocates basically all of your memory. You have say 1K left.

Now it happens that some important, well written process, by poor luck, happens to step in the middle and need 2K, so it tries to allocate it, but you don't have that available.

Guess which process crashes due to OOM in your "easier and better" "refuse to give out more" scheme?

Once you are able to answer this simple question, you understand already a bit more, and also find why the linux OOM killer sucked 20 years ago, and was replaced. No, Windows doesn't behave in your "easier and better" way either.

Out of memory is a serious and difficult-to-handle error condition, has always been. It's not something that you can think as, "just don't give more memory to applications and hope they cope". The fundamental role of the OS is to give the resources to the programs, and lack of necessary resources is a serious error condition. This is like food disappearing from the stores and the population fighting over it. Does it work if you say, "Easy! Just last one at the store gets no food!"? No, the problem is complex, and really the right solution is to have the resources you need.

Sorry to sound rude but this is a problem of you having no idea whatsoever but assuming you know how things work, despite the fact they don't work that way, and further assuming you know better.

[golden_labels]

There's no problem in normal operation yet, but if someone just use everything default, and don't care what happens to their system, then he/she will just use IOS on the phone, and will not be in this forum.

The exact reason you were asked to produce a specific example of the issue you are experiencing. There is no issue.

You seem to be making some assumptions about how things work and, facing the reality, you reject it as “done wrong”. And until you make an effort to climb out of perspective you forced upon yourself, you may actually never understand the situation.

Further, some OSes (e.g, Windows, Solaris) don't have this OOM-killer thing at all, and they works fine.

Because they expect programs to commit suicide. Instead of being killed, they are being forced to kill themselves; chosen even more randomly than the OOM killer would do it, because instead of a likely culprit being eliminated and some processes being protected, any unlucky process that tried to use more memory is facing that fate.

I don't know which happened first, but seems in Linux world, programs commit insane amount of memory as a habit, kernels tolerate and connive them, good cooperation right? 

You have some weird ideas about what overcommit is, what it does and why it is being used.

Judging by the tone and content, I would guess you must have read some articles of self-appointed prophets of overcommit catastrophe, who cover their lack of effort to understand the situation with unparalleled zeal and strong opinions. Isn’t that true?

Kernel doesn’t “tolerate” something. There is no bad behavior to be tolerated. It’s intentional feature that allows efficient memory management. It works faster, it wastes less memory, it improves system stability. That’s the reason behind it.

This is one particular thing Windows do better, by not over commiting, forcing those developers to claim only what they need. And if OOM do happens, just refuse giving out more. Much better and easier.

Windows — just like Linux — does overcommit. If it can’t provide a page, it even may show you a blue screen of death, murdering everything instead of a single process. Why you never heard of that? I suspect the causes are two. While Linux allows some flexibility with OOM killer and overcommiting policies, Windows offers no configuration and therefore no relevant documentation to raise people’s fears. “Out of sight, out of mind.” Second: due to how memory provision is performed in Windows (compared to Linux), different types of malfunction lead to that condition. In Windows it’s likely removal or corruption of medium containing the page file, which people intuitively consider to be “abnormal situation” and are fine with their process failing in such a scenario. In Linux those situations are equally abnormal and no one should expect processes to be able to work reliably if they happen, but understanding, why it is so, requires more mental effort and many not notice how bad they are. Being able to explicitly issue MEM_COMMIT under Windows, which will give a reliable error status, may also give false impression that accessing the page will not fail: but it is not so, as MEM_COMMIT doesn’t cause a page to be provided as one would expect: it only updates counters so commit calls from other processes will fail. The actual provision is done exactly the same way as in Linux: on first page access.

[SiliconWizard]

Memory overcommitment is a non-trivial topic. Whereas it has proven to be a good approach for general-purpose OSs, it's not the only approach nor even the best one for *all* use cases.

As a simplification, you can think of it as dynamic dynamic memory allocation - if you get the sense. So, very generally speaking, you can say that it's as useful on the OS level as dynamic allocation is on the process level. You can't deny it is for a very large range of applications and contexts. (Of course that can be applied at even higher levels, like an onion. Virtualization for instance can use overcommitting, while virtualizing OSs that themselves use overcommitting...)

But there surely are specific contexts for which overcommitting is not a good idea, just like there are contexts (for instance safety-critical, embedded stuff) for which dynamic allocation itself is often avoided.

[DiTBho]

Code: [Select]

sh invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
CPU: 0 PID: 434 Comm: sh Not tainted 5.4.128-fon2 #125
Stack : 1000bc01 00000001 809717c0 8009d7d0 809717c0 00000000 00000000 a6510b3a
        8073110c 81fa9b64 80770000 80771a47 806f70c8 1000bc00 81fa9b08 a6510b3a
        00000000 00000000 807f0000 000000f3 00003235 8033aa8c 00000010 00000000
        436f6d6d 000000f2 00000000 3a207368 80770000 80000000 806f8c28 80770000
        ffffffff 80771b60 00200000 00000001 000000f3 8033bdc8 00000000 807c0000
        ...


this bad guy  ?

Code: [Select]

# cat /proc/meminfo
MemTotal:          24716 kB
MemFree:            4500 kB
MemAvailable:       8760 kB

i am having a lot of fun with kernel 5, as they consume more ram and you need to find a solution to make your apps survive the reaper 

[golden_labels]

you need to find a solution to make your apps survive the reaper 

Buy more RAM. There is no other good solution to hardware limitations than upgrading it.

The message indicates you had oom_score_adj=0. Was  that shell a process that caused excessive memory usage? If no, it pushing the score lower could have saved it. If yes, I would say OOM killer getting rid of it did good job. Have you considered putting the troublesome process into a cgroup with memory.max set? This way it will be denied memory before it can affect the rest of the system.

[Logan]

Guess which process crashes due to OOM in your "easier and better" "refuse to give out more" scheme?

Because they expect programs to commit suicide. Instead of being killed, they are being forced to kill themselves; chosen even more randomly than the OOM killer would do it, because instead of a likely culprit being eliminated and some processes being protected, any unlucky process that tried to use more memory is facing that fate.

Why must a program crash or suicide after failing to get more memory? The programs should either find a way to continue working, or just pause, or at least, properly shutdown.
But when getting killed, it's died immediately.

[sleemanj]

In my experience the OOM killer is only a problem when /proc/sys/vm/min_free_kbytes is too small, which is seemingly the case by default on low memory systems, I forget now but I think the kernel defaults to a tiny percentage of ram, and it just isn't enough on small memory machines.

I run a lot of websites on 1G ram EC2 and Lightsail instances, my setup/boot scripts always do this...

Code: [Select]

  if [ $(cat /proc/sys/vm/min_free_kbytes) -lt 20000 ]
  then
    echo 20000 >/proc/sys/vm/min_free_kbytes
  fi

to ensure that the system will keep at least 20M free at all times for allocation.

With this, I, touch wood, very very seldom see the OOM killer step in, even under heavy loads (with respect to the system capabilities).  Without at least that 20M, the OOM killer gets a bit angry.

[Siwastaja]

Why must a program crash or suicide after failing to get more memory? The programs should either find a way to continue working, or just pause, or at least, properly shutdown.
But when getting killed, it's died immediately.

That's actually a good question. You'll find an answer to it if you try to write some software. Basically, if you depend on dynamic memory management all over your program, what can you do without memory? Very little. Is there a big difference to print "out of memory, sorry" then exit using exit(1), or having the program killed by the OS? I don't think so.

Assuming you can allocate memory, and handling not being able to do that as worst case, non recoverable catastrophic error (or not handling it at all), makes programming complex software much, much simpler. Because "not enough memory" circumvention logic would end up being a colossal task, and because such code never "normally" runs, it likely ends up a bugnest like any rarely used feature. It's not something you would have a simple one-size-fits-all solution. No, you'd need to decide if you need to write the current state into a file, but what if your file saving routine (say, a compressing algorithm) needs to allocate some memory, too?

So programmers just assume memory is like air to breathe, a resource that has to exist. You could do the programmatical equivalent of carrying an oxygen bottle and mask with you all the time, but the cost is high and your daily life is easier if you just know the air does not suddenly disappear.

When reliability matters, say in a lung ventilator or airbag controller, then you just either allocate statically so that allocation cannot fail run-time, or guarantee the availability of memory otherwise. None of such high-reliability systems create circumvention logic for out-of-memory errors, that would be just too risky. Also such systems do not use linux, or Windows.

Memory management as used in modern desktop general purpose OSes like Windows, Mac and linux all overcommit AFAIK, because that's most efficient use of resources in the real world, given the real-world software which does not carefully control all allocations to bare minimum. Without such overcommiting, out-of-memory problems would be far more common, though you could argue that being able to give a guarantee that memory allocated by succesful return code of malloc() will be available, that way you could implement software that "pause" and ask you to close other software while waiting for more memory to appear. Maybe it could be done but I don't quite believe it would be superior in practice.

[golden_labels]

Why must a program crash or suicide after failing to get more memory? The programs should either find a way to continue working, or just pause, or at least, properly shutdown.
But when getting killed, it's died immediately.

Because for a real world program⁽¹⁾ inability to acquire memory nearly always makes it impossible to continue: obtaining memory is a prerequisite for code being executed later. The exception are programs that allocate huge chunks of memory in one run (imagine: hundreds megabytes to gigabytes), but those are extremely rare — and while they don’t kill themselves, they bail out completely from a task, so kinda perform a partial suicide. It’s simply impractical to react to OOM. Sometimes even impossible in any sane way: handling OOM is often itself susceptible to OOM.

You can’t pause, as not only that would be a busy loop, but you can’t even know what’s the cause of failure. The condition may be permanent, giving you an infinite loop. That would also not solve anything, as inability to acquire memory indicates the system is already entered a mode in which normal operation⁽²⁾ is no longer possible. Hammering will make situation worse. It’s like smashing F5 like mad in a browser while receiving information that a request can’t be handled due to heavy load on a website.
____
⁽¹⁾ That is: not students’ toy programs or software made for programming research only.
⁽²⁾ Which is what you may be missing, judging by what you wrote so far and prior knowledge of people expressing similar attitudes. Memory starvation is not a normal mode of operation of an operating system; it’s a breakage of the underlying environment.

[SiliconWizard]

Why must a program crash or suicide after failing to get more memory? The programs should either find a way to continue working, or just pause, or at least, properly shutdown.
But when getting killed, it's died immediately.

As you can see from opinions in this thread, and pretty much everywhere else, this was, is, and will keep being a hot debate.
There are good points from both sides - at least, there can be.

Again simplifying things a bit, if you think about it, the whole idea behind overcommitting and the associated behaviors is that the OS should rule over processes, which, in general-purpose applications, is the most sensible thing to do. Kinda in the same way as preemptive multitasking now rules, and has many distinct benefits compared to cooperative multitasking.

So, in that regard, that the OS be able to kill - or ask it to kill itself - a process that "misbehaves" does make sense. It's, on average, something safe to do, at least much safer than risking the whole system to collapse.

Now, I still partly agree with Logan. I have read again, and again, and again... that letting processes crash is fine and even the best way to handle "memory allocation errors". I've even read that it was the best way of handling any kind of error. Yes. Fortunately, not everywhere. But programmers tend to be pretty lazy.

There ARE situations where giving a process the ability to know whether it can, or cannot, allocate a certain amount of memory with the guarantee of being able to use it would be extremely useful. No no no, not all applications just blindly require given amounts of memory and would not be able to operate with less anyway. That's simply not true. It all depends. And sometimes it's much better to be able to handle this gracefully. But here, the OS has all powers, so to speak. So it gets to decide what is good for a given program,  and what isn't. The program (so: the programmer) doesn't get to decide. It of course has benefits and drawbacks. For overall long-term stability of a system, that looks like the best way to handle it. But the real rationale behind it is that a given program crashing is better than the OS crashing, or at least that all other programs running can keep running. Which is something most of us would find an obvious behavior these days. How annoying was it that a given program could crash a whole system, back in the days... But guess what? Sometimes, individual applications crashing may be as bad as the whole system crashing. There is no absolute here. I think it's again all in what is considered having the better benefit/risk ratio for general-purpose use IMHO.

[bd139]

Let’s go up a level and purposely ignore these microscopic problems. It’s better to build systems that tolerate failure than systems that are designed not to fail. Thus the problem scope is larger than a program or an OS but the system as a whole.

[golden_labels]

There ARE situations where giving a process the ability to know whether it can, or cannot, allocate a certain amount of memory with the guarantee of being able to use it would be extremely useful. (…) Sometimes, individual applications crashing may be as bad as the whole system crashing. There is no absolute here. I think it's again all in what is considered having the better benefit/risk ratio for general-purpose use IMHO.

That is technically impossible to do, though. There is no way an operating system can guarantee to a program that the environment, in which the program operates, will certainly not become unreliable. It has no crystal ball.

This is the thing I am repeating here. The thing consistently overlooked by people preaching about the OOM-killer catastrophe. Not being able to provide a memory page is not a simple “something went wrong” situation in a normally functioning system. If that occurs, the system is already in a state not different — in qualitative terms — to what it would be in if you pull out a RAM stick, memory contents are getting corrupted, a CPU starts to misinterpret instructons or power is lost. The underlying environment is no longer reliable. And it is like that before memory starvation actually happens. If there is not enough memory to be provided, it means that memory usage must have reached nearly 100% already. At which point what you experience is — depending on the workload — somewhere between everything getting barely usable to a complete lockup lasting a quarter hour. At this point a service requiring high availability and reliabity should face a reset to minimize losses and bring system back to operational state as soon as possible.

This is the scale of the problem. Yet it happens to be presented as if that was some minor I/O read error or a TCP connection being dropped. No, it’s the whole world on fire, everything around collapsing. Which should also explain why this is not a reliabity issue much different from a power loss. You account for it the same way as you would for all other failures.

As for avoiding the condition, Linux actually allow you to provide backing storage for your memory. Instead of mapping anonymous pages, you can use a pre-allocated file. As long as other conditions will not kill your program, it will not fail. You can allocate that file on tmpfs, if you are worried about writeback. Of course that requires you to deal diectly with memory acquisition from the system, not through neat abstractions provided by the programming environment. It may also not work nicely (e.g. Python, Java, C#) or not work at all with some languages (e.g. JavaScript), for some others being completely outside of the language’s scope (C, C++). And you are required to do your own memory management. Since most programmers lack knowledge and time to do that properly, that solution is likely to end up wasting both memory and time. Windows may allow a similar thing, but I do not know the exact semantics on that platform, so I can’t confirm that with certainity. It also lacks equivalent to tmpfs, at last built-in.

IMO the best way is, however, to configure the system in a way that ensures stability. With how configurable is Linux, with cgroups and namespaces, nowadays you can control processes pretty well. And if coupled with systemd it becomes effortless compared to what it was even a decade ago.

[Logan]

Thank you everyone!

[golden_labels]

Logan:
Here is a program you may wish to think about. Run it with overcommit on (either full or heuristic) and with overcommit off. Consider not doing anything important while running with overcommit off, as some other random program may die.

Code: [Select]

#include <stdlib.h>
#include <stdio.h>
#include <stdint.h>
#include <unistd.h>
#include <sys/wait.h>

int main(void) {
    static size_t const allocSize = UINT32_C(268435456);
    char* dummy = malloc(allocSize);
    int counter = 1000;
    int waitTime = 20;
   
    for (size_t i = 0; i < allocSize; ++i) {
        dummy[i] = 'a';
    }
   
    while (counter > 0) {
        switch (fork()) {
            default:
                counter = 0;
                wait(NULL);
                break;
               
            case -1:
                perror("fork");
                abort();
               
            case 0:
                --counter;
                printf("%d: %c\n", counter, dummy[counter]);
                fflush(stdout);
                if (0 == counter) {
                    printf("Waiting %ds...", waitTime);
                    fflush(stdout);
                    sleep(waitTime);
                }
                break;
        }
    }
   
    free(dummy);
    return EXIT_SUCCESS;
}
The program is never using or allocating more than around 260MB of memory and should flawlessly run on a computer with few hundred MiB of RAM. With overcommit off it will not. Can you spot what important feature of many *nix systems you have lost while turning off overcommit?

This is of course a synthetic example to show a problem, but the issue is quite realistic and has nothing to do with programmer’s laziness or not caring about memory usage. In reality the situation may happen for example if a memory-demanding program tries to invoke another program or on a server spawning many children workers.

[nctnico]

Hi guys.
One thing I really hate about Linux is the OOM-killer and memory-over-commit, it makes me feel unsettled, unknown which process will get a sudden murder in next second. (an accurate joke here: https://lwn.net/Articles/104185/)
I tried to disable over-commit, but any program won't start when there's more than 1GB free RAM. I guess all programs for Linux just make use of this stupid design and commit as much as the developer can ever imaging.
So I came up another idea: keep over-commit disabled, use ZRAM as swap, and set over-commit ratio to 100. So the commit limit will be much larger, with actual space in case they use them, and no wasted HDD space at all.
Giving the average ZRAM compression ratio of 3:1, I should set the SWAP space limit to 3x free RAM(or less) when idle.

Using swap is a simple way to get more memory if the system is not going to use it. It is true some software will claim much more memory than it needs to have. Without a small amount of physical ram (like many embedded systems have) you can quickly run out of memory. I'd put the swap on an external memory device though and not the memory itself. Ofcourse you need to test your system whether it is actually using the swap at some point. If not then it is OK. If it is, you have a memory leak somewhere that needs fixing. For this reason I have a swap space on embedded Linux systems. Again: not due to lack of memory but due to processes claiming more than they actually need.

[golden_labels]

nctnico: please, update your knowledge. It’s no longer 1990s. Under normal conditions⁽¹⁾ swap is not “free memory” for people, who can’t afford actual RAM. It can’t be used that way and is not used like that in Linux for a very long time. Windows is moving away from that archaic model too since at least version 6.0 of the kernel. Simplifying things a bit: nowadays swap is used as an ad-hoc backing storage for anonymous pages, so the kernel can remove them from RAM and use that memory for something useful, like storing file cache. This is a performance increasing use, unlike “free RAM” which is greatly decreasing performance. A secondary use is suspend-to-disk. That is why swap is still defended by some against being completely removed from modern systems.

If you want some introduction to the situation from a more technical point of view, see a summary on Linux swap use from Chris Down, a kernel dev.
____
⁽¹⁾ There are some exceptions like specialized pieces of software writen in a manner that allows the to take advantage of disk to extend memory storage. But that requires code carefully crafted for that exact purpose and such programs will often maintain their own swap instead ouf interferring with the system’s one.

[Siwastaja]

Simplifying things a bit: nowadays swap is used as an ad-hoc backing storage for anonymous pages, so the kernel can remove them from RAM and use that memory for something useful, like storing file cache.

Maybe simplify it a bit more and elaborate what "anonymous page" means.

Asking for a friend

Edit: and will reply to myself, for example memory you allocated by malloc(), then wrote something to it making the OS actually commit it to RAM; this will be an "anonymous page". "Anonymous" refers to the fact that OS doesn't know, or have to know, what the purpose of this memory is, for example it is not a cached physical file, just whatever memory for the app. (Often simple things are complicated by weird names that are not initially very descriptive.)