[torqueusers] requesting gpus

Gareth.Williams at csiro.au Gareth.Williams at csiro.au
Fri Feb 3 14:45:24 MST 2012


Matt Ismail at Warwick in the UK knew the problem/solution.

> I reported this issue to Adaptive in August last year and it got fixed in torque-3.0.3-snap.201111071556. From the CHANGELOG: "Fixed a problem in qsub where you could not submit a job in interactive mode with gpus in the resource list."

> If it is the same issue you're seeing it'll only be affecting interactive job submissions, i.e. qsub -I.

I can confirm that in our current setup non-interactive jobs are OK - and we'll upgrade to make interactive jobs work too.

Thanks,

Gareth

> -----Original Message-----
> From: Gareth.Williams at csiro.au [mailto:Gareth.Williams at csiro.au]
> Sent: Friday, 3 February 2012 8:08 PM
> To: torqueusers at supercluster.org
> Subject: Re: [torqueusers] requesting gpus
> 
> > -----Original Message-----
> > From: Craig West [mailto:cwest at vpac.org]
> > Sent: Friday, 3 February 2012 4:55 PM
> > To: torqueusers at supercluster.org
> > Subject: Re: [torqueusers] requesting gpus
> >
> >
> > Hi Gareth,
> >
> > > However when I run a job with the recommended syntax:
> > > http://www.adaptivecomputing.com/resources/docs/torque/3-0-
> > 3/3.7schedulinggpus.php
> > >
> > > I get:
> > >>  qsub -I -q viz -l nodes=1:ppn=1:gpus=1
> > > qsub: Job exceeds queue resource limits MSG=cannot locate feasible
> > nodes
> > >
> > > The torque version is 3.0.3-snap.201108261653
> > >
> > > Note that this is _/not/_ the --enable-nvidia-gpus functionality.
> > > Also note that the server has not been restarted.
> > > The scheduler is moab but I'm pretty sure the job gets rejected
> well
> > > before moab comes into the picture.
> > >
> > > Does anyone have such a setup working or can anyone see what is
> wrong
> > > (or have an idea of where to look)?
> >
> > Your pbsnodes output looks correct and similar to our systems.
> >
> > Few questions for you:
> > 1. What version of Moab are you running?
> > 2. Does the Viz queue have the ability to schedule to that node?
> > 3. What is in the "Configured Resources" line of "checknode n121"?
> >     It should have a "GPUS: 2" parameter.
> >
> > Cheers,
> > Craig.
> -snip-
> 
> 1) Moab Version: 6.0.2 - due for an upgrade anytime
> 2) yes  - and I can get jobs there with gpus as a gres but that doesn't
> count them right
> 3) > checknode n121 | grep Configu
> Configured Resources: PROCS: 12  MEM: 94G  SWAP: 96G  DISK: 137G  GPUS:
> 2
> 
> But I think moab is not getting to play a role. I've looked at logs but
> confess that I've not turned up the logging level yet.
> 
> Gareth



More information about the torqueusers mailing list