[Mauiusers] Bug in memory limit enforcement after maui restart

Chris Samuel csamuel at vpac.org
Sat Dec 22 00:20:24 MST 2007

----- Martin Kleinschmidt <mk at theochem.uni-duesseldorf.de> wrote:
> On Fr, 21 Dez 2007, Gareth.Williams at csiro.au wrote:
> >At our site, we only use vmem limits.  In fact I thought torque would
> >track mem, but neither torque or maui/moab would actually do anything
> >with it. We have never had a problem quite like yours.
> Before activating the limit enforcements, I have tested with mem and
> vmem limits, both work very well, except fot this restarting issue. 

My guess is that Gareth is talking about why the ulimits
set by Torque on jobs via mem or pmem are not enforcemed,
when they are with vmem.

This is down to the way that glibc and the Linux kernel
do memory allocations, and frankly it's a bit of a mess
which I mentioned on the torquedev list.

I tracked this down to the fact that somewhere around
glibc 2.3 the malloc() implementation was ripped out
and replaced with one that uses mmap() for allocations
of 128KB or more. The kicker is that the kernel mmap()
implementation only cares about the virtual memory
ulimit (RLIMIT_AS) for enforcement, the others are
just ignored!

So currently an application which uses small allocations
(<128KB) will find malloc() failing when they hit their
max mem / data seg size ulimit whereas an application
that grabs RAM in larger chunks will sail happily past
that without a care in the world (unless you've set the
max virtual mem limit as well)..

Hey ho..

Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

More information about the mauiusers mailing list