Jump to content


 


Register a free account to unlock additional features at BleepingComputer.com
Welcome to BleepingComputer, a free community where people like yourself come together to discuss and learn how to use their computers. Using the site is easy and fun. As a guest, you can browse and view the various discussions in the forums, but can not create a new topic or reply to an existing one unless you are logged in. Other benefits of registering an account are subscribing to topics and forums, creating a blog, and having no ads shown anywhere on the site.


Click here to Register a free account now! or read our Welcome Guide to learn how to use this site.

Photo

Prevent memory leaks from crashing the system


  • Please log in to reply
6 replies to this topic

#1 Taoki

Taoki

  • Members
  • 53 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Romania, Bucharest
  • Local time:11:29 AM

Posted 11 July 2014 - 01:48 PM

The problem has reached a point where I'm no longer willing to let it slip, so I wish to find a solution to this ASAP. What happens precisely is: Occasionally, programs have memory leaks. Usually they tend to be harmless, as they are slow and / or don't fill quite a lot of RAM. In a few cases however, a process fills the memory with Gigabytes of RAM in a matter of seconds, until every bit of it is taken up. When this happens, the system becomes unusable (nothing can be clicked, mouse pointer freezes, etc) and the computer needs a hard restart. This is both a major problem as well as a security risk, because such a thing can cause loss of data or put the system in danger (consider it were to happen in the middle of a Kernel update).

I was hoping that at this day, the Linux Kernel has a minimal protection against this sort of thing, and enforces a pocket within the available RAM which user applications cannot simply take up... in order to save the system if memory is filled abusively. But since it doesn't appear to, I need to add one manually. Problem is I don't know how exactly, and I'm hoping someone can clarify me better.

What is the best way to limit memory usage for normal processes, in order to disallow memory leaks from bringing the system down by blocking all the RAM? I'm thinking something that restricts non-root processes to only part of the total memory. For example, I have 9GB of RAM. If it were to solve those crashes, I am okay with allowing 1GB to be used only by root and system processes, whereas normal programs I run may only have access to the other 8 GB.

The right path seems to be the ulimit command and the /etc/security/limits.conf file. But ulimit seems to have a lot of parameters and memory types it addresses (which aren't clarified either) and I'm not sure exactly what to set it to for this scenario. I basically seek the ulimit settings that require me to give up as little RAM as possible, in exchange for guaranteeing a space that memory leaks cannot touch to keep the system safe. Also, I'd prefer using percentages rather than fixed values, so I don't have to re-configure everything if I gain or lose RAM... like for example use "90%" instead of "8000 MB".

One clarification: I believe that in the past, I've heard people say that if a process of low priority has a limitless memory leak, it shouldn't actually take down the system because the Kernel knows to handle it, so maybe something else is happening. I've had the problem numerous times and can confirm this is false! If any badly written program fills up all the memory in a few seconds (which I get to see in KSysGuard before the system dies) it will render the system unusable and the user has to unplug the computer and start it up again. Also, I do have a SWAP partition... and large one at that (8GB). Even so, such leaks do bring down the system.



BC AdBot (Login to Remove)

 


#2 Guest_Kaosu_*

Guest_Kaosu_*

  • Guests
  • OFFLINE
  •  

Posted 11 July 2014 - 11:35 PM

What application is causing you so much grief? You should really be trying to isolate the underlying issue instead of trying to set hard limits to work around it. Honestly, setting a hard limit is mostly for shared multi-user environments, so you can limit a particular group or user so they cannot exhaust system resources. If you're using this system as a desktop or even single-user server, you shouldn't be having these kinds of problems unless you're unintentionally executing a fork bomb or something similar.
 
Please provide more detailed information about the program(s) that are causing issues so we can try to address the underlying problem. If the problem is the software just being this poorly written, you should replace the software because it probably has all kinds of other issues as well. Sloppy coding can result in these kinds of issues, which tend to be plagued by security issues as well.
 
If you truly just want a quick fix then I can probably help you achieve that goal with the following methods.

 

Limits Method

Limit memory usage of each process that belongs to a particular group of users. Since this is on a per-process basis, all processes ran by a user within the specified group will be able to use a total of 8GB of memory. This is likely not the method you want to use to achieve your goal, but I added it here for completeness.

 

1) Open /etc/security/limits.conf and add:

@usergroup hard as 8388608

This will place a hard address space limit for all users in the group "usergroup". So, if you added your normal user accounts to the group "usergroup", they would automatically be limited upon logging in. The "as" keyword literally stands for "address space" and is measured in kilobytes. So, limiting the address space to 8388608 kilobytes should result in user processes being limited to only 8GB of available memory. Since root wouldn't be in the group "usergroup", it would not be affected and could use all 9GB, so the reamining 1GB would be reserved for those operations.
 
You could also run problematic programs under a different user, place that user in "usergroup" and then limit its address space to 2GB or something, ensuring your regular user/root account has at least 7GB to use. Just play with it and see what solution works best for you.
 
 

CGroups Method (Recommended)
Limit the memory usage of every process that is ran by a specific user. This will limit a specific user to only 8GB of memory for all processes they want to run. Use this method to achieve your goal, because it is the most efficient and likely what you originally wanted to do.

 

1) Open /etc/cgconfig.conf and add:

group memlimit {
    memory {
        memory.limit_in_bytes = "8G";
    }
}

This will create the cgroup "memlimit" and define the task "memory" to limit the total amount of memory used at 8GB.

 

You can also limit the swap usage by adding memory.memsw.limit_in_bytes="2G"; under memory.limit_in_bytes = "8G";

 

2) Open /etc/cgrules.conf and add:

@usergroup memory memlimit

This will place the memory limit on all users that belong to the group "usergroup". Likewise, you could also replace "@usergroup" with a users name if you only wanted to set the restriction for a particular user instead of having to add him/her to a new group.

 

3) Start the cgconfig/cgred services and configure these services to start automatically when you reboot

 

In most cases you should just be able to install the cgroups-bin package, follow my instructions above and then start the services. If your distribution does things differently, you can do an online search to quickly get up and running with the rules I have provided for you.


Edited by Kaosu, 12 July 2014 - 03:21 AM.


#3 Taoki

Taoki
  • Topic Starter

  • Members
  • 53 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Romania, Bucharest

Posted 12 July 2014 - 04:29 AM

Thank you, that info is very helpful! I agree there's something wrong with the application in cause of course, but what really bothers me is that the OS allows itself to be vulnerable to applications doing this.

 

What's causing the problem: I don't get this sort of issue happening to me all the time, it's actually very rare and many programs caused it over the years. Even so however, it's not a risk I'm happy to realize my system is under all the time. What happened yesterday due to which I made this post was an issue in Firefox. The official Shumway extension (experimental builtin flash player) appears to cause severe memory leaks when opening even some simple flash files (1MB large). Typically it causes Firefox to go up by some 2 GB of memory... but eventually the impossible happened, and the FF process was thrown from using about 400 MB of RAM to using 6.3 GB in only a few seconds! Honestly I didn't even know you could write quite that fast to DDR3 at all, let alone that such a leak could be left uncontrolled by the Kernel.

 

I'll remember your suggestions in regard to limiting user space memory, and will activate that if there is no other way. For now however, if I suspect a program might cause memory leaks, I shall run it as "ulimit -v 2048000;myprogram" to limit it to 2GB only... which I found is a good way to test something suspected of leaking memory endlessly.



#4 Taoki

Taoki
  • Topic Starter

  • Members
  • 53 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Romania, Bucharest
  • Local time:11:29 AM

Posted 12 July 2014 - 02:51 PM

Alright everyone, I have (hopefully) found the solution to my problem at last. Someone clarified me what it is I'm looking for: Memory overcommit settings. From what I read, the Kernel allows applications to overcommit memory in some cases by default... meaning they can take more than is available. Although officially the defaults should prevent system crashes like mine, they clearly don't all the time. So after digging a bit through the Kernel settings involved, I found the parameters which will hopefully fix this problem for good:

 

vm.overcommit_memory = 2
vm.overcommit_ratio = 90

I added the above to sysctl.conf and applied them via sysctl command for now, those values being what I felt is best. Everything seems to be working well so far. What that basically says is that an application can only allocate 90% of the physical RAM + SWAP partition size, never more. Since I have 9GB of RAM (triple-channel before anyone asks why it's an impair number) and 8GB of SWAP, this means 8.1 GB + 8 GB, which should be more than enough! I'll see if such leaks grief me again after adding those Kernel settings.

A warning to everyone who considers playing with those: Never set vm.overcommit_ratio to 0! I read in several posts that the people's machines would no longer boot after doing that. That's because people assumed they could limit the system to 0% of the RAM because they have SWAP. But the SWAP partition isn't activated before these settings are, so using 0 there will basically tell the system there is no available memory at startup. I like to keep people out of trouble so I thought to also include this note.

#5 NickAu

NickAu

    Bleepin' Fish Doctor


  • Moderator
  • 13,250 posts
  • ONLINE
  •  
  • Gender:Male
  • Location:127.0.0.1 Australia
  • Local time:11:29 AM

Posted 12 July 2014 - 05:37 PM

Thank you Taoki and Kaosu I have learned something.


Arch Linux .
 
 Come join the fun, chat to Bleeping computer members and staff in real time on Discord.
 
The BleepingComputer Official Discord Chat Server!


#6 Guest_Kaosu_*

Guest_Kaosu_*

  • Guests
  • OFFLINE
  •  

Posted 25 July 2014 - 09:25 PM

 

Alright everyone, I have (hopefully) found the solution to my problem at last. Someone clarified me what it is I'm looking for: Memory overcommit settings. From what I read, the Kernel allows applications to overcommit memory in some cases by default... meaning they can take more than is available. Although officially the defaults should prevent system crashes like mine, they clearly don't all the time. So after digging a bit through the Kernel settings involved, I found the parameters which will hopefully fix this problem for good:

vm.overcommit_memory = 2
vm.overcommit_ratio = 90
I added the above to sysctl.conf and applied them via sysctl command for now, those values being what I felt is best. Everything seems to be working well so far. What that basically says is that an application can only allocate 90% of the physical RAM + SWAP partition size, never more. Since I have 9GB of RAM (triple-channel before anyone asks why it's an impair number) and 8GB of SWAP, this means 8.1 GB + 8 GB, which should be more than enough! I'll see if such leaks grief me again after adding those Kernel settings.

A warning to everyone who considers playing with those: Never set vm.overcommit_ratio to 0! I read in several posts that the people's machines would no longer boot after doing that. That's because people assumed they could limit the system to 0% of the RAM because they have SWAP. But the SWAP partition isn't activated before these settings are, so using 0 there will basically tell the system there is no available memory at startup. I like to keep people out of trouble so I thought to also include this note.

 

 

Cool solution to your problem. Thanks for coming back and sharing the information.

 

Setting vm.overcommit_memory to 2 seems to provide better stability for systems that may be memory exhausted due to poorly written or extremely demanding applications. To be honest, I wouldn't trust the OOM killer in a mission critical environment, since it could easily kill processes that have been doing computational tasks for weeks just because some crappy program decided to go crazy. So a setting of 2 seems sensible in a situation where you don't want processes randomly killed off.

 

Some light reading for anyone that would like to learn more about this feature:

 

https://www.kernel.org/doc/Documentation/vm/overcommit-accounting

http://www.win.tue.nl/~aeb/linux/lk/lk-9.html#ss9.6

http://www.etalabs.net/overcommit.html


Edited by Kaosu, 25 July 2014 - 09:58 PM.


#7 Taoki

Taoki
  • Topic Starter

  • Members
  • 53 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Romania, Bucharest
  • Local time:07:29 PM

Posted 06 August 2014 - 06:09 PM

Ok... apparently vm.overcommit_memory isn't all that safe to use. After approximately 2 days of uptime, the KDE screen locker eventually breaks... with an error saying "unable to connect to authentication system" or something like that, whenever I type my password and press enter to unlock. And yes, I analyzed it several times... this only happens with vm.overcommit_memory = 2 (didn't try 1). I'm probably not going to bother bug reporting this... since it's likely one of those things that are hard to test and only happens to me and a few other people out there. I wish Linux (especially KDE) wasn't so buggy all the time though...






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users