[torqueusers] RE: node access policy

Sarah Mulholland smm at rincon.com
Tue Mar 24 10:13:47 MDT 2009

I'm still trying to find the syntax for a job to hog the nodes.

According to the docs, the default nodeaccesspolicy is shared.  I am on a cluster with 8 nodes, 2 processors each.  I'd like to launch a job that uses 5 processors, 1 per node, and doesn't allow any other jobs to use the nodes.

When I submit the job with

        qsub -l nodes=5:ppn=1:naccesspolicy=singlejob


        qsub -l nodes=5:ppn=1:naccesspolicy=singletask

I get the message:

        Job exceeds queue resource limits MSG=cannot locate feasible nodes

Showq indicats I have 8 nodes, 16 processors.  If I call pbsnodes, they are all online.

Any hints?


-----Original Message-----
From: pat.o'bryant at exxonmobil.com [mailto:pat.o'bryant at exxonmobil.com]
Sent: Friday, March 20, 2009 10:33 AM
To: Sarah Mulholland
Cc: Gareth.Williams at csiro.au; torqueusers at supercluster.org;
torqueusers-bounces at supercluster.org
Subject: RE: [torqueusers] Hogging Nodes

    Two points which I recently learned that control how nodes are
allocated to jobs.

1) Your maui.cfg should have "nodeaccesspolicy=shared" or whatever
"wide-open" is. You can only make requests as a user that are more
restrictive than the global policy. Since the global policy is wide-open
the users can specify thru Torque what they want.

2) Thru Torque you can code something like the following:

#PBS -l nodes=10,tpn=1,naccesspolicy=singlejob

What this statement means (at least in my Moab world and I would assume a
similar situation for Maui) is "give me 10 tasks with 1 task/node. Also, I
want to be the only job on any of the 10 nodes.  For us, understanding that
"nodes" needs to be interpreted as "tasks" made all the difference. So one
more example:

#PBS -l nodes=64,tpn=4,naccesspolicy=uniqueuser

This statement says I want 64 tasks with 4 tasks/node. I will get a total
of 16 nodes (64/4). Also, I only want 1 of my jobs on the nodes. Others are
allowed on the nodes but they too will be restricted to 1 job as well.

Check this link:
And this one:

Hope this helps,

J.W. (Pat) O'Bryant,Jr.
Business Line Infrastructure
Technical Systems, HPC

             <smm at rincon.co                                             To
             m>                       "Gareth.Williams at csiro.au"
             Sent by:                 <Gareth.Williams at csiro.au>
             torqueusers-bo                                             cc
             unces at superclu           "torqueusers at supercluster.org"
             ster.org                 <torqueusers at supercluster.org>
                                      RE: [torqueusers] Hogging Nodes
             03/20/09 09:55

There are a lot of great suggestions, but none seem to fit our situation
exactly.  Yes, we only want to affect the usage in question (a particular
program or two). In the general case we want all processors on all nodes to
be available.

The ppn=2 solution might be the best approach if we could write a little
bit of software to make half the jobs do nothing.

Thank you to everyone who replied.


-----Original Message-----
From: Gareth.Williams at csiro.au [mailto:Gareth.Williams at csiro.au]
Sent: Thursday, March 19, 2009 5:07 PM
To: Sarah Mulholland
Cc: torqueusers at supercluster.org
Subject: RE: [torqueusers] Hogging Nodes

> From: Prakash Velayutham [mailto:prakash.velayutham at cchmc.org]
> Sent: Friday, 20 March 2009 1:45 AM
> To: Sarah Mulholland
> Cc: torqueusers at supercluster.org
> Subject: Re: [torqueusers] Hogging Nodes
> May be
> will do?
> Prakash
> On Mar 18, 2009, at 7:47 PM, Sarah Mulholland wrote:
> I sent this question to the maui group over a week ago, but there was no
> answer.  Perhaps this question is more appropriate to the torque group.
> I am running the maui scheduler 3.2.6 patch level 16 with torque 2.1.6.
> am looking for a way to submit a job on some number of nodes, say 10.  In
> addition to running on 10 nodes, I want exclusive use of those nodes.  Is
> there a property I can set to allow that kind of scheduling?   So far I'm
> submitting jobs with
>             "-l nodes=10:ppn=1"
> Specifically I have a job that needs to run on 10 processors all on
> different nodes.  We have two processors per node.  The
> EXACTNODE makes sure our job gets what it needs, but I want to prevent
> anything else from running on the second processor of the 10 nodes.
> Is there a "don't-share-the-nodes" modifier that I can set on the job
> submission?  In the maui.cfg?

Hi Sarah,

You seem to have the solution now, but this will globally affect all jobs
and this may not be desirable if you have a mixed workload.  You could use
the softer approach of submitting the jobs in question with
"-l nodes=10:ppn=2" and then just only running 10 processes.  You may need
to customize your options used with mpi in this case, but it would only
affect the usage in question and not all users.


torqueusers mailing list
torqueusers at supercluster.org

More information about the torqueusers mailing list