[Mauiusers] Network Topoligy Aware Scheduling

Roy Dragseth roy.dragseth at cc.uit.no
Mon Oct 11 05:10:26 MDT 2010


We have a bit simpler setup with only two levels of switches and maui seems to 
support that rather well.

In torque one can set node properties that describes what switch each node are 
connected to, for instance c3-4 has a property named switch3 and so on:

# pbsnodes c3-4
c3-4
     state = job-exclusive
     np = 8
     properties = ib,switch3,highmem
     ntype = cluster


Then we put this in maui.cfg

NODESETPOLICY    ONEOF
NODESETATTRIBUTE FEATURE 
NODESETDELAY     96:00:00
NODESETLIST       switch1 switch2 switch3 switch4 switch5 switch6 switch7 
switch8 switch9 switch10 switch11 switch12 switch13 switch14 switch15 switch16 
switch17 switch18 switch19 switch20 switch21 switch22 switch23 switch24 
switch25 switch26 switch27 switch28 switch29 switch30 switch31 switch32 
switch33 switch34 switch35 switch36 switch37 switch38 switch39 switch40 
switch41 switch42 switch43 switch44

This will make maui schedule jobs within one switch if it can do so within 96 
hours.  If you set NODESETDELAY to zero then maui will only schedule within 
one switch if it is immediately possible within available resources.

I do not know if it is possible to extend this to deeper levels of switching.

We turned away from this setup as we couldn't measure any difference in 
performance for our apps running within one switch or across switches and 
rather started scheduling strictly on the type of network the nodes are 
connected to.  Some of our nodes have infiniband and others only gigabit 
ethernet.

NODESETPOLICY    ONEOF
NODESETATTRIBUTE FEATURE 
NODESETDELAY     96:00:00
NODESETLIST      gige ib 

This will prevent any job running across the different network types which 
solved some issues we saw with open-mpi applications.

Anyway, hope this helps.

Regards,
r.





More information about the mauiusers mailing list