[torqueusers] RE: node access policy

Gabe Turner gabe at msi.umn.edu
Tue Mar 24 10:44:47 MDT 2009

On Tue, Mar 24, 2009 at 09:13:47AM -0700, Sarah Mulholland wrote:
> I'm still trying to find the syntax for a job to hog the nodes.
> According to the docs, the default nodeaccesspolicy is shared.  I am on a cluster with 8 nodes, 2 processors each.  I'd like to launch a job that uses 5 processors, 1 per node, and doesn't allow any other jobs to use the nodes.
> When I submit the job with
>         qsub -l nodes=5:ppn=1:naccesspolicy=singlejob
> or
>         qsub -l nodes=5:ppn=1:naccesspolicy=singletask
> I get the message:
>         Job exceeds queue resource limits MSG=cannot locate feasible nodes

You would want

    qsub -l nodes=5:ppn=1,naccesspolicy=singlejob

Note the , and not : between nodes and naccesspolicy.  Regardless, though,
this may only work with Moab, now Maui.

Another thing you can do, and what we often recommend our users do (because
it simplifies accounting), is just request the whole node and only run one
process per node.  There are multiple ways to do this, depending on how
your application needs to be launched.  If you're running 5 serial
processes, you could use pbsdsh -n to put one process on each node.  If
you're running an MPI job, you have a couple of options depending on how
mpirun/mpiexec obtains it's hostfile.  If you're passing $PBS_NODEFILE to
it directly, just run it through uniq first:

    cat $PBS_NODEFILE | uniq > newnodefile
    mpirun -np 5 --hostfile newnodefile .... ./a.out

If your mpirun/mpiexec is getting it's node list directly from torque
(OpenMPI can do this, for example), then you may be able to use mpirun
flags to have it only spawn one process per node:

    mpirun -np 5 -bynode ....  ./a.out

I _believe_ the above should work for OpenMPI, though I haven't tested it.
I'm going with what I found in the mpirun manpage.


Gabe Turner                                             gabe at msi.umn.edu
UNIX System Administrator,
University of Minnesota
Supercomputing Institute                          http://www.msi.umn.edu

More information about the torqueusers mailing list