[torqueusers] Limiting max vmem+pmem
Coyle, James J [ITACD]
jjc at iastate.edu
Tue Aug 10 09:16:08 MDT 2010
vmem should always be greater than pmem. I think that Torque corrects
this should it be specified.
In the csh, the limit command will show you the resource limits in a job.
If you submit several jobs like:
echo 'csh -c limit' | qsub -l vmem=18GB,pmem=14GB,mem=2GB,nodes=1:ppn=1
you can see the effects of combinations of vmem, pmem and mem
For me, running Torque version 2.3.6:
it seems as though
mem= does not have an effect on the csh limits,
datasize and memoryuse are set to pmem
vmemoryuse is set to max(vmem , pmem)
I use a web form where users request for cpus, walltime time and memory per process.
The result is a Torque script with the #PBS -l line to use.
1) For the cluster that uses OpenMPI, I calculate:
If this exceeds the memory on a node, I calculate how many processes can fit
on a single node and use that for ppn, and adjust the number of nodes accordingly.
I do this because it seems as though OpenMPI reports back the sum of the vmem
for all OpenMPI processes on a node.
2) For the cluster that uses MPICH, I set vmem=pmem.
since it seems that MPICH does not report back
I've found that there is no such thing as a small amount of swapping.
if a node is shared, the swapping is unfair to other users on the node,
as it make their programs swap also.
if no nodes are shared, the single user of the node should just get all
the memory on the node.
Users do sometimes over-allocate arrays, but then the space is just allocated in virtual space
and never used.
You can try out the web form for the cluster using OpenMPI at:
I can't say that it is perfect, but it gets the job done.
James Coyle, PhD
High Performance Computing Group
115 Durham Center
Iowa State Univ.
Ames, Iowa 50011 web: http://www.public.iastate.edu/~jjc
From: torqueusers-bounces at supercluster.org [mailto:torqueusers-bounces at supercluster.org] On Behalf Of Rushton Martin
Sent: Tuesday, August 10, 2010 8:46 AM
To: Chris Vaughan; Torque Users Mailing List
Subject: Re: [torqueusers] Limiting max vmem+pmem
Are you sure you are asking the right question? The physical memory used by a process (its working set) is mapped to its virtual memory, and is therefore a part of it.
Tel: 01959 514777, Mobile: 07939 219057
email: jmrushton at QinetiQ.com
QinetiQ - Delivering customer-focused solutions
Please consider the environment before printing this email.
From: torqueusers-bounces at supercluster.org [mailto:torqueusers-bounces at supercluster.org] On Behalf Of Chris Vaughan
Sent: 10 August 2010 14:24
Subject: [torqueusers] Limiting max vmem+pmem
Is there a way to limit the total combined size of pmem and vmem within TORQUE? The user is comfortable with some swapping but would like to limit the combined amount of vmem+pmem without setting two different individual limits for both.
Chris Vaughan | Technical Consultant - EMEA
3000 Cathedral Hill | Guildford GU2 7YB | United Kingdom
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. QinetiQ may monitor email traffic data and also the content of email for the purposes of security. QinetiQ Limited (Registered in England & Wales: Company Number: 3796233) Registered office: 85 Buckingham Gate, London SW1E 6PD http://www.qinetiq.com<http://www.QinetiQ.com>
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the torqueusers