[Mauiusers] Reservations for GPUs

Burkhard Bunk bunk at physik.hu-berlin.de
Thu Nov 11 02:01:47 MST 2010

Hi Henrik,

I had a similar problem recently.
Since "consumable resources" didn't work for me, I used standing 
reservations to "split the nodes":

For the GPU-related processors (=cores):
# maui.cfg
SRCFG[gpu1] HOSTLIST=node01

I wrote one such stanza for each node, in order to make sure that the
allocation goes to the right nodes.

Then for the cpu queue, I wrote a "complementary reservation", e.g. for
# maui.cfg
SRCFG[cpux] HOSTLIST=node0[1-4]

If you have other nodes for queue "cpu" without any GPUs, you should
prevent the gpu queue from sending jobs elsewhere. Give the gpu nodes
a "node property" like GPU (in server_priv/nodes) and restrict the queue
(in qmgr) accordingly:

set queue gpu resources_default.neednodes = GPU

For your last remark (more CPUs for gpu jobs if available), I can't see
a simple solution. The jobs, however, may simply try to use more CPUs than
requested from the batch system - PBS does not really force a job to
use the requested processors only.

Hope that helps.

Burkhard Bunk.
  bunk at physik.hu-berlin.de      Physics Institute, Humboldt University
  fax:    ++49-30 2093 7628     Newtonstr. 15
  phone:  ++49-30 2093 7980     12489 Berlin, Germany

On Thu, 11 Nov 2010, Dr. Henrik Schulz wrote:

> Dear all,
> in our cluster we have several nodes containing 2 GPUs and 8 CPU-cores. Since most of the jobs, which utilize the graphics cards, only need 1 CPU-core per GPU-job, we have some free CPU-cores while all GPUs of the node are used.
> Is it possible (maybe using standing reservations) to have two queues for that:
> - a queue "gpu" which provides 2 CPU-cores per node
> - a queue "cpu" which provides the other 6 CPU-cores of the node.
> The most important fact is that it must not be possible to allocate more than 6 CPU-cores in the queue "cpu" even if there is no GPU-job running on that node. The other way round it would be nice to be able to use more than 2 CPU-cores in the queue "gpu" if some of the other 6 cores are available.
> Do You have any ideas to configure these two queues?
> Thanks in advance!
> Henrik
> _______________________________________________
> mauiusers mailing list
> mauiusers at supercluster.org
> http://www.supercluster.org/mailman/listinfo/mauiusers

More information about the mauiusers mailing list