[torqueusers] infiniband advice

Troy Baer tbaer at utk.edu
Fri Apr 3 08:51:42 MDT 2009

On Fri, 2009-04-03 at 10:32 -0400, Steve Young wrote:
> 	I'm trying to figure out how to implement using multiple interfaces  
> in torque on the clients. The cluster is running fine now over gig-e  
> but I need to enable use of the Infiniband interface for MPI types of  
> jobs on it. Is it just as simple as making MPI aware of the interfaces  
> and just making sure the users call this version of MPI over IB? Or  
> are there some settings I should be looking at in torque/maui to  
> inform it of the IB interface and bind it to a queue or something? I'm  
> interested in hearing how some others handle it. Any advice on  
> infiniband in general would be great =). Thanks,

In my experience, TORQUE doesn't need to know about what high-speed
network you use... unless you want it to, that is.  If you have some
nodes with IB and some without, you'll probably want to add a node
property (e.g. "ib") to the nodes with.  You may need to do some clever
scheduling tricks as well.

Other than that, it's a matter of putting out an IB-friendly MPI
implementation, making that the default for users, and making sure MPI
startup in TORQUE jobs goes through the TM interface (i.e. using the OSC
mpiexec for MVAPICH or the native TM bindings for OpenMPI).

Troy Baer, HPC System Administrator
National Institute for Computational Sciences, University of Tennessee
Phone:  865-241-4233

More information about the torqueusers mailing list