[torqueusers] Job Allocation on Nodes

Bill Wichser bill at Princeton.EDU
Wed Mar 7 17:47:31 MST 2012

On 3/7/2012 7:30 PM, Gareth.Williams at csiro.au wrote:
>> Perhaps this question has been answered before.  I have users who want to distribute jobs equally amongst nodes.  What I am observing at the moment is that when a user submits a job with nodes=12:ppn=3, the job uses three nodes with 12 cores per node.  Is there a way to make the job use only three cores per node.  How can I prevent this or setup some kind of affinity for following the user's job requirements?
> Hi Randall,
> Why would you want to do such a thing?  If the user submits four of the jobs they will align, and you will get worse contention.  I would suggest: if you need to spread jobs to access memory then you should schedule memory and/or if you need to avoid contention, say for memory bandwidth, then get the users to request whole nodes (all the available ppn) and only run as many processes as their scaling permits (they may need custom mpirun options).
> Gareth
> _______________________________________________
> torqueusers mailing list
> torqueusers at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torqueusers
I have users who desire to do just this -- maximize memory bandwidth for 
their application.  It turns out that sharing the node with others 
always provides better memory bandwidth than running the full node with 
the job.  This can be reproduced quantitatively while looking at 
walltime only.  Sometimes allocating multiple cores to cover memory use 
is required but the --bynode flag for openmpi is always used.

So memory contention is overcome and the node can be shared even with 
the same user's jobs as this contention tends to run in cycles instead 
of overlapping.


More information about the torqueusers mailing list