[torqueusers] NCPUS environment variable?

Martin Siegert siegert at sfu.ca
Tue Jul 12 12:06:54 MDT 2005

On Mon, Jul 11, 2005 at 09:24:37PM -0400, Robin Humble wrote:
> On Mon, Jul 11, 2005 at 05:12:02PM -0700, Martin Siegert wrote:
> >I am wondering whether (how?) the number of processors for a MPI
> >job is available in a PBS script in an environment variable (or
> >otherwise).
> >
> >Example: I have a script
> >
> >#!/bin/bash
> >#PBS -S /bin/bash
> >#PBS -l walltime=0:12:00
> >#PBS -l nodes=4:ppn=2
> >mpiexec -machinefile $PBS_NODEFILE -np 8 ./mpi_program
> >
> >I would prefer having an environment variable (e.g., PBS_NCPUS) set by
> >torque so that I could use
> if you use an MPI that is PBS aware (eg. LAM) then you can just do:
>   lamboot
>   mpirun C ./mpi_program
>   lamhalt
> and LAM queries PBS/torque for the node list via the tm interface and
> "it just works". you can also put lamboot/lamhalt into the torque
> prologue/epilogue scripts so you that users don't need those two lines
> in their scripts.
> I presume there's a tm aware MPICH front-end somewhere that will also do
> the right thing by default.

Unfortunately that is not an option: we support a bunch of different
architectures, most systems come with the vendor's MPI distribution
(e.g., the p595 IBM systems), other's offer more than one MPI implementation
to the user or run on MPI distribution that does not support this (e.g.,
the example above corresponds to MPICH2 with the mpi daemon started at
boot time by root).

> >#!/bin/bash
> >#PBS -S /bin/bash
> >#PBS -l walltime=0:12:00
> >#PBS -l nodes=4:ppn=2
> >mpiexec -machinefile $PBS_NODEFILE -np $PBS_NCPUS ./mpi_program
> you could set $PBS_NCPUS from $PBS_NODEFILE in a torque prologue script.

Yes, I usually do that, but it works only, if the no. of lines in the
$PBS_NODEFILE corresponds to the no. of processors. It is also more an
ugly workaround: if torque can supply $PBS_NODEFILE why can it not
supply $PBS_NCPUS as well? (my guess is that the answer here is:
historic reasons)

> >Similarly on a shared memory machine I would like to use
> >
> >#!/bin/bash
> >#PBS -S /bin/bash
> >#PBS -l walltime=0:12:00
> >#PBS -l ncpus=8
> >mpiexec -machinefile $PBS_NODEFILE -np $PBS_NCPUS ./mpi_program
> >
> >Is this possible? I found that torque provides $PBS_NODENUM and $PBS_TASKNUM
> >but those always seem to get set to 0 and 1, resp. Am I missing something?
> I can't see the ncpus=8 notation in the torque docs :-/
>   http://www.clusterresources.com/products/torque/docs/4.2jobsubmission.shtml

It may not be om the docs, but it is in the code ... :-)
As far as I can tell ncpus=8 works like nodes=1:ppn=8, but I am not sure
whether it only works on certain architectures.

> >On a cluster I can cound the number of lines in $PBS_NODEFILE, but on
> >a IBM shared memory machine $PBS_NODEFILE only contains a single line
> >regardless of how many processors were requested in the
> >#PBS -l ncpus=...
> >line.
> odd. does 'pbsnodes -a' say you have the expected number of cpus on
> that shared memory node?

I did not try that. But I know that I did get (at least) the number of
processes that I requested for the MPI job.

> >On a somewhat unrelated topic:
> >On a cluster with dual processor nodes is it possible to request 8 cpus
> >without specifying the number of processors per node? I do not care
> >whether I get 2 processors per node or just one (or a mix), I just want
> you can do nodes=8:ppn=1, and in maui.cfg set 
> and make sure JOBNODEMATCHPOLICY is unset, and that'll work.
>   http://www.clusterresources.com/products/maui/docs/a.fparameters.shtml

Thanks! I try that. But in principle torque/maui (or torque/moab in our case)
should support 3 different cases:

1) nodes=4:ppn=2
2) nodes=8:ppn=1
   (e.g., for MPI jobs that require the full network bandwidth between
   processes, but can tolerate a serial job on the other processor in the
3) ncpus=8
   (the don't care choice)

As far as I can tell your solution supports 1 and 3, but not 2, correct?


Martin Siegert
Head, HPC at SFU
WestGrid Site Manager
Academic Computing Services                        phone: (604) 291-4691
Simon Fraser University                            fax:   (604) 291-4242
Burnaby, British Columbia                          email: siegert at sfu.ca
Canada  V5A 1S6

More information about the torqueusers mailing list