[torqueusers] (no subject)
Ronny T. Lampert
telecaadmin at uni.de
Wed Apr 26 04:16:56 MDT 2006
> Users, select either infiniband or myrinet nodes as follows:
> > qsub -l nodes=X:infiniband jobs.script
> > qsub -l nodes=X:myrinet jobs.script
> I would like users to be able to not specify myrinet or infiniband, if
> they wish, since VMI (from NCSA) allows users to compile binaries that
> will execute both on myrinet and infiniband. If users DO NOT specify
> infiniband or myrinet, PBS should pick the default as infiniband (if
> enought nodes are available) OR myrinet (if enough infiniband nodes are
> not available) but NEVER a mix of two. Right now if users do not specify
> "infiniband" or "myrinet" PBS picks a mix of two nodes.
You have to wrap your head around the pbs notion a bit. I think you'll need
One queue that takes all jobs and routes them to a specific destination.
One queue that only executes jobs on myrinet nodes.
One queue that only executes jobs on infiniband nodes.
I can _only_ help you for the strict separating of the nodes, so no mixed
nodes are allocated.
You will have to see how to get the rest to work with maui, I can't help you
there as I have zero experience, sorry.
First I would alter the nodes file a bit:
(and so on)
The use qmgr to create the 2 queues:
#> create queue myrinet
#> set queue myrinet queue_type=Execution
#> set queue myrinet resources_default.neednodes=is_myrinet
#> set queue infiniband resources_max.nodect=<NR_OF_MYRINET_NODES>
#> create queue infiniband
#> set queue infiniband queue_type=Execution
#> set queue infiniband resources_default.neednodes=is_infiniband
#> set queue infiniband resources_max.nodect=<NR_OF_INFINIBAND_NODES>
Then create the "front-end queue" that will route and set it as the default
#> create queue batch
#> set queue batch queue_type=Route
#> set queue batch route_destinations = "infiniband,myrinet"
#> set server default_queue = batch
Maybe also set the default resources to 1 node with 1 cpu:
#> set server resources_default.nodes = 1:ppn=1
I hope that will point into the right direction. Read in the torque/maui
docs about routing queues.
More information about the torqueusers