[torqueusers] Re: Memory Limits with torque/maui & enforcement

Åke Sandgren ake.sandgren at hpc2n.umu.se
Mon Sep 22 09:29:29 MDT 2008

On Sat, 2008-09-20 at 15:44 -0400, Tom Pierce wrote:
> Åke Sandgren wrote:
> On Thu, 2008-09-18 at 13:18 -0600, Josh Butikofer wrote:
> >>You may want to use pvmem as opposed to just pmem. Sometimes Linux has a hard time enforcing pmem
> >>because of the way it handles virtual memory. In my experience, pvmem behaves more as expected. I
> >>would try both out and see which seems to work better in your case.>>
> >To be specific the linux kernel does NOT enforce pmem (RLIMIT_RSS,
> >ulimit -m) at all. There was pieces of such code in the kernel up to
> > 2.4.15 or so when they removed it completely.
> >There is still no such code in at least the 2.6 based kernels.
> So to have torque manage the memeory needs of a parallel job, I need to use
> #PBS -l nodes=2:ppn=4: pvmem=1500m     would require that a node have
> 4 processors, and that each process needs 1500 megabytes of memory, so
> the node would have to have 4*1500 = 6000m or 6 Gb per node BEFORE it
> would "qualify" the node as having adequate resources to run. And then
> if the node only had 6 Gb of physical memory, the next job would have
> to wait for this node to be freed-up.
> Is that how torque resource manager works with the memory requirement?
>  or does pvmem allow for swap space as well?

pvmem is virtual memory
pmem is physical memory (almost)
So in your example the node would have to have 6G of virtual memory
Depending on your scheduler (maui?) you can use pmem to have it choose
nodes with 6G of physical memory but the kernel will only enforce the
virtual memory limit.

We always use pmem AND pvmem together.
And usually set pvmem slightly higher or sometimes a lot higher due to
shared library overheads

Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden
Internet: ake at hpc2n.umu.se   Phone: +46 90 7866134 Fax: +46 90 7866126
Mobile: +46 70 7716134 WWW: http://www.hpc2n.umu.se

More information about the torqueusers mailing list