[torqueusers] Memory resource Limits

Chris Samuel csamuel at vpac.org
Sun Aug 3 22:06:52 MDT 2008

----- "Jim Kusznir" <jkusznir at gmail.com> wrote:

> We've tried mem= with no results; pvmem resulted
> in killing all jobs unless it was actually using
> an extremely small amount of ram.

Whilst PBS does monitor a job to try and work out
when it's breaking the set limits it is also complicated
by ulimits and glibc's malloc() I'm afraid.

By default Torque sets two different ulimits depending
on what you request.

If you set a mem or pmem limit then pbs_mom sets the RLIMIT_DATA
size which is described as:

   The  maximum  size  of  the  process’s data segment (initialized
   data, uninitialized data, and heap).  This limit  affects  calls
   to  brk(2)  and  sbrk(2),  which fail with the error ENOMEM upon
   encountering the soft limit of this resource.

If you set vmem or pvmem then it sets the RLIMIT_AS size
which is described as:

   The maximum size of the process’s virtual memory (address space)
   in  bytes.   This  limit  affects  calls  to brk(2), mmap(2) and
   mremap(2), which fail with the error ENOMEM upon exceeding  this

Notice that the first only affects brk()/sbrk() whilst
the second affects mmap() et. al as well as brk() ?

The kicker is that glibc's malloc() will switch between
using brk() and mmap() depending on the size of allocation
that you're trying to make.  So if you request pmem and
your program makes large allocations it will never hit
RLIMIT_DATA as glibc will be making calls to mmap().

Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

More information about the torqueusers mailing list