[torqueusers] (no subject)

[Ronny T. Lampert]

Hi,

> Users, select either infiniband or myrinet nodes as follows:
>  > qsub -l nodes=X:infiniband jobs.script
>  > qsub -l nodes=X:myrinet jobs.script
[...]
> I would like users to be able to not specify myrinet or infiniband, if
> they wish, since VMI (from NCSA) allows users to compile binaries that
> will execute both on myrinet and infiniband. If users DO NOT specify
> infiniband or myrinet, PBS should pick the default as infiniband (if
> enought nodes are available) OR myrinet (if enough infiniband nodes are
> not available) but NEVER a mix of two. Right now if users do not specify
> "infiniband" or "myrinet" PBS picks a mix of two nodes.

You have to wrap your head around the pbs notion a bit. I think you'll need
3 queues:

One queue that takes all jobs and routes them to a specific destination.
One queue that only executes jobs on myrinet nodes.
One queue that only executes jobs on infiniband nodes.

I can _only_ help you for the strict separating of the nodes, so no mixed
nodes are allocated.
You will have to see how to get the rest to work with maui, I can't help you
there as I have zero experience, sorry.

First I would alter the nodes file a bit:

ibnode1212 is_infiniband
myrnode0101 is_myrinet
(and so on)

The use qmgr to create the 2 queues:

#> create queue myrinet
#> set queue myrinet queue_type=Execution
#> set queue myrinet resources_default.neednodes=is_myrinet
#> set queue infiniband resources_max.nodect=<NR_OF_MYRINET_NODES>

#> create queue infiniband
#> set queue infiniband queue_type=Execution
#> set queue infiniband resources_default.neednodes=is_infiniband
#> set queue infiniband resources_max.nodect=<NR_OF_INFINIBAND_NODES>

Then create the "front-end queue" that will route and set it as the default
queue:

#> create queue batch
#> set queue batch queue_type=Route
#> set queue batch route_destinations = "infiniband,myrinet"
#> set server default_queue = batch

Maybe also set the default resources to 1 node with 1 cpu:
#> set server resources_default.nodes = 1:ppn=1


I hope that will point into the right direction. Read in the torque/maui
docs about routing queues.

Cheers,
Ronny