[Mauiusers] Reservations for GPUs

[Burkhard Bunk]

Hi Henrik,

I had a similar problem recently.
Since "consumable resources" didn't work for me, I used standing 
reservations to "split the nodes":

For the GPU-related processors (=cores):
#
# maui.cfg
#
SRCFG[gpu1] HOSTLIST=node01
SRCFG[gpu1] PERIOD=INFINITY
SRCFG[gpu1] TASKCOUNT=1
SRCFG[gpu1] RESOURCES=PROCS:2
SRCFG[gpu1] CLASSLIST=gpu
...

I wrote one such stanza for each node, in order to make sure that the
allocation goes to the right nodes.

Then for the cpu queue, I wrote a "complementary reservation", e.g. for
node0[1-4]:
#
# maui.cfg
#
SRCFG[cpux] HOSTLIST=node0[1-4]
SRCFG[cpux] PERIOD=INFINITY
SRCFG[cpux] TASKCOUNT=4
SRCFG[cpux] RESOURCES=PROCS:6
SRCFG[cpux] CLASSLIST=cpu

If you have other nodes for queue "cpu" without any GPUs, you should
prevent the gpu queue from sending jobs elsewhere. Give the gpu nodes
a "node property" like GPU (in server_priv/nodes) and restrict the queue
(in qmgr) accordingly:

set queue gpu resources_default.neednodes = GPU

For your last remark (more CPUs for gpu jobs if available), I can't see
a simple solution. The jobs, however, may simply try to use more CPUs than
requested from the batch system - PBS does not really force a job to
use the requested processors only.

Hope that helps.

Regards,
Burkhard Bunk.
----------------------------------------------------------------------
  bunk at physik.hu-berlin.de      Physics Institute, Humboldt University
  fax:    ++49-30 2093 7628     Newtonstr. 15
  phone:  ++49-30 2093 7980     12489 Berlin, Germany
----------------------------------------------------------------------

On Thu, 11 Nov 2010, Dr. Henrik Schulz wrote:

> Dear all,
>
> in our cluster we have several nodes containing 2 GPUs and 8 CPU-cores. Since most of the jobs, which utilize the graphics cards, only need 1 CPU-core per GPU-job, we have some free CPU-cores while all GPUs of the node are used.
>
> Is it possible (maybe using standing reservations) to have two queues for that:
>
> - a queue "gpu" which provides 2 CPU-cores per node
> - a queue "cpu" which provides the other 6 CPU-cores of the node.
>
> The most important fact is that it must not be possible to allocate more than 6 CPU-cores in the queue "cpu" even if there is no GPU-job running on that node. The other way round it would be nice to be able to use more than 2 CPU-cores in the queue "gpu" if some of the other 6 cores are available.
>
> Do You have any ideas to configure these two queues?
>
> Thanks in advance!
> Henrik
>
> _______________________________________________
> mauiusers mailing list
> mauiusers at supercluster.org
> http://www.supercluster.org/mailman/listinfo/mauiusers
>