[Mauiusers] Bug in memory limit enforcement after maui restart

Gareth.Williams at csiro.au Gareth.Williams at csiro.au
Sun Dec 23 15:31:18 MST 2007

Thanks Martin and Chris,

You have brought together the puzzle pieces I had not previously
connected.  I think there are good arguments for using vmem limits
(predictability, support in both torque and maui/moab, works), but had
not realized I could turn on mem limits in maui/moab. Perhaps in review
the documentation will be clearer...

Sorry I have no fix for your problem Martin, but I have not had the same
problem with vmem, so you could compare the relevant sections of code.

- Gareth

-----Original Message-----
From: Chris Samuel [mailto:csamuel at vpac.org] 
Sent: Saturday, 22 December 2007 6:20 PM
To: Martin Kleinschmidt
Cc: Williams, Gareth (HPSC, Melbourne - HPSC); mauiusers
Subject: Re: [Mauiusers] Bug in memory limit enforcement after maui

----- Martin Kleinschmidt <mk at theochem.uni-duesseldorf.de> wrote:
> On Fr, 21 Dez 2007, Gareth.Williams at csiro.au wrote:
> >At our site, we only use vmem limits.  In fact I thought torque would
> >track mem, but neither torque or maui/moab would actually do anything
> >with it. We have never had a problem quite like yours.
> Before activating the limit enforcements, I have tested with mem and
> vmem limits, both work very well, except fot this restarting issue. 

My guess is that Gareth is talking about why the ulimits
set by Torque on jobs via mem or pmem are not enforcemed,
when they are with vmem.

This is down to the way that glibc and the Linux kernel
do memory allocations, and frankly it's a bit of a mess
which I mentioned on the torquedev list.

I tracked this down to the fact that somewhere around
glibc 2.3 the malloc() implementation was ripped out
and replaced with one that uses mmap() for allocations
of 128KB or more. The kicker is that the kernel mmap()
implementation only cares about the virtual memory
ulimit (RLIMIT_AS) for enforcement, the others are
just ignored!

So currently an application which uses small allocations
(<128KB) will find malloc() failing when they hit their
max mem / data seg size ulimit whereas an application
that grabs RAM in larger chunks will sail happily past
that without a care in the world (unless you've set the
max virtual mem limit as well)..

Hey ho..

Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

More information about the mauiusers mailing list