[torqueusers] Limiting max vmem+pmem

Coyle, James J [ITACD] jjc at iastate.edu
Tue Aug 10 09:16:08 MDT 2010


  vmem should  always be greater than pmem. I think that Torque corrects
this should it be specified.

In the csh, the limit command will show  you the resource limits in a job.

  If you submit several jobs like:
echo 'csh -c limit' | qsub -l vmem=18GB,pmem=14GB,mem=2GB,nodes=1:ppn=1

you can see the effects of combinations of vmem, pmem and mem

For me, running Torque version 2.3.6:

it seems as though
   mem= does not have an effect on the csh limits,

  datasize and memoryuse are  set to pmem
   vmemoryuse is set to max(vmem , pmem)

  I use a web form where users request for cpus, walltime time and memory per process.
The result is a Torque script with the  #PBS -l  line to use.

 1) For the cluster that uses OpenMPI, I calculate:


   If this exceeds the memory on a node, I calculate how many processes can fit
on a single node and use that for ppn, and adjust the number of nodes accordingly.

   I do this because it seems as though OpenMPI reports back the sum of the vmem
for all OpenMPI processes on a node.

 2)   For the cluster that uses MPICH, I set vmem=pmem.

  since it seems that MPICH does not report back

   I've found that there is no such thing as a small amount of swapping.
    if a node is shared, the swapping is unfair to other users on the node,
  as it make their programs swap also.
    if no nodes are shared, the single user of the node should just get all
 the memory on the node.

 Users do sometimes over-allocate arrays, but then the space is just allocated in virtual space
and never used.

  You can try out the web form for the cluster using OpenMPI at:


  I can't say that it is perfect, but it gets the job done.

-          Jim

 James Coyle, PhD
 High Performance Computing Group
 115 Durham Center
 Iowa State Univ.
 Ames, Iowa 50011           web: http://www.public.iastate.edu/~jjc

From: torqueusers-bounces at supercluster.org [mailto:torqueusers-bounces at supercluster.org] On Behalf Of Rushton Martin
Sent: Tuesday, August 10, 2010 8:46 AM
To: Chris Vaughan; Torque Users Mailing List
Subject: Re: [torqueusers] Limiting max vmem+pmem

Are you sure you are asking the right question?  The physical memory used by a process (its working set) is mapped to its virtual memory, and is therefore a part of it.

Martin Rushton
Weapons Technologies
Tel: 01959 514777, Mobile: 07939 219057
email: jmrushton at QinetiQ.com
QinetiQ - Delivering customer-focused solutions

Please consider the environment before printing this email.

From: torqueusers-bounces at supercluster.org [mailto:torqueusers-bounces at supercluster.org] On Behalf Of Chris Vaughan
Sent: 10 August 2010 14:24
To: torqueusers
Subject: [torqueusers] Limiting max vmem+pmem

Is there a way to limit the total combined size of pmem and vmem within TORQUE?  The user is comfortable with some swapping but would like to limit the combined amount of vmem+pmem without setting two different individual limits for both.



Chris Vaughan  |  Technical Consultant - EMEA

3000 Cathedral Hill | Guildford GU2 7YB  |  United Kingdom


This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. QinetiQ may monitor email traffic data and also the content of email for the purposes of security. QinetiQ Limited (Registered in England & Wales: Company Number: 3796233) Registered office: 85 Buckingham Gate, London SW1E 6PD http://www.qinetiq.com<http://www.QinetiQ.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/torqueusers/attachments/20100810/866acbde/attachment-0001.html 

More information about the torqueusers mailing list