[torqueusers] (no subject)

Amitoj G. Singh amitoj at cs.uh.edu
Mon Apr 17 10:53:29 MDT 2006

Dear all,

Torque: Version 2.0.0p2
Maui: 3.2.6
Linux: Red Hat EL 4
Kernel: 2.6.12
Cluster Size: 520 Infiniband + 128 Myrinet nodes

I have defined the 520 Infiniband nodes with the "infiniband" and the
Myrinet nodes with the "myrinet" property respectively, as follows in the
/var/spool/PBS/server_prive/nodes file ..

ibnode0101 infiniband
ibnode1212 infiniband
myrnode0101 myrinet
myrnode1013 myrinet

Users, select either infiniband or myrinet nodes as follows:

 > qsub -l nodes=X:infiniband jobs.script

 > qsub -l nodes=X:myrinet jobs.script

Under /usr/local/maui/maui.cfg I have defined as follows:

NODESETPOLICY                   ONEOF
NODESETLIST                     myrinet infiniband

so that no job can have nodes that are a mix of myrinet+infiniband.

Queue properties:

Queue workq
        queue_type = Execution
        total_jobs = 34
        state_count = Transit:0 Queued:2 Held:0 Waiting:0 Running:31 Exiting:1
        resources_max.walltime = 36:00:00
        resources_default.walltime = 01:00:00
        resources_assigned.nodect = 484
        max_user_run = 40
        enabled = True
        started = True

Server properties (partial):

        default_queue = workq
        log_events = 511
        mail_from = adm
        query_other_jobs = True
        resources_default.walltime = 01:00:00
        resources_assigned.nodect = 492
        scheduler_iteration = 60
        node_check_rate = 150
        tcp_timeout = 6
        node_pack = False

I would like users to be able to not specify myrinet or infiniband, if
they wish, since VMI (from NCSA) allows users to compile binaries that
will execute both on myrinet and infiniband. If users DO NOT specify
infiniband or myrinet, PBS should pick the default as infiniband (if
enought nodes are available) OR myrinet (if enough infiniband nodes are
not available) but NEVER a mix of two. Right now if users do not specify
"infiniband" or "myrinet" PBS picks a mix of two nodes.

Any help in this regard will be appreciated. thank you in advance.

- Amitoj.

More information about the torqueusers mailing list