[torqueusers] procs resource advice and docs

[Martin Siegert]

Hi Gareth,

just to expand a little bit on what Roman already said ...

On Wed, Jan 20, 2010 at 09:55:12PM +1100, Gareth.Williams at csiro.au wrote:
> Hi All,
> 
> After finding out that there is a 'procs' resource a while back, I've run
> some tests (with a 2.3 torque and moab) specifying procs instead of nodes
> and think I'd like to recommend it to our user community (particularly on
> our 128p numa system). It complements the 'nodes' resource specification
> nicely for when you just want allocated cores and don't care about the
> how they are arranged on nodes.
> 
> Is anybody already using the feature?

We are using this together with the

JOBNODEMATCHPOLICY EXACTNODE

in moab. This way the -l nodes=n:ppn=m specification is interpreted the
way users expect it to work and at the same time we have a method available
to users who do not care how their processes are distributed accross nodes.
After gaining some experience with this we now recommend to users to
use -l procs=N unless they have a specific reason to use the -l nodes=n:ppn=m
syntax: the waiting time in the queue with -l procs=N is much, much
shorter.

> Is there a downside apart from the lack of documentation?

None. Only benefits. The usage percentage of the cluster increases
dramatically.

>  I see it's documented in the pbs_resources_unicos8 man page so I guess
> it was developed by or for CRAY, possibly long ago, but it seems to work
> fine on our linux systems. The pbs_resources_unicos8 man page does not
> mention that it conflicts with the nodes resource syntax but I guess this
> is obvious.

Actually, -l procs is fairly new - we requested that feature :-)
Thus, I do not believe that whatever is mentioned under CRAY applies.

>  moab just uses the nodes resource request if it is present (ignoring
> the procs resource)

Actually it is the other way round: if you specify procs then nodes are
ignored, see:

http://www.clusterresources.com/products/mwm/docs/13.3rmextensions.shtml#procs

(I am confused here: that page existed a few days ago, but now it is gone).

> and I guess maui would too but that is a function of the scheduler
> and doesn't really have a place in the torque docs per-se.

I am not sure whether maui has support for procs.

> Can the linux pbs_resources man page should be updated to include
> the procs resource? 

There is one more issue: if you are using OSC's mpiexec you need
to patch the code in get_hosts.c to add support for procs. I can
email you the patch, if you need it.

Cheers,
Martin

-- 
Martin Siegert
Head, Research Computing
WestGrid Site Lead
IT Services                                phone: 778 782-4691
Simon Fraser University                    fax:   778 782-4242
Burnaby, British Columbia                  email: siegert at sfu.ca
Canada  V5A 1S6