[torqueusers] IB epilogoue/prolog, and any other concerns

Walid walid.shaari at gmail.com
Fri May 30 01:39:02 MDT 2008

2008/5/29 Brock Palen <brockp at umich.edu>:

> Strange, I thought CISCO was now pushing OpenMPI,  they only employ one of
> the main devs to make sure it works on their TopSpin IB.

CISCO is big, and most likely you will get different views from the inside,
however is there an officel statement from CISCO or known Roadmap that shows
that they are favouring OpenMPI for its technical merits?

> We run Cisco IB with OFED for the VERBS and OpenMPI.  Works well little if
> any issues.  Just be sure your pbs_mom starts with enough locked memory.

Thanks, we are aware of the issue, however does not like the workaround to
have it set in both limits.conf, and the init script of pbs_mom, is there a
better standard way to ease maintance as the turnaround of sysadmins where i
work is a bit high, and if not following standard operating system/distro
procedures means going back to docuemantion that could be missing/outdated

> If you do choose to go with the cisco VAPI verbs, note they are planed to
> be deprecated.  everything is going OFED.
> They all work well, I use OpenMPI because it integrates with TM on torque
> and can support myrinet as well as IB and TCP all in one lib with no
> recompiles needed.  Many options are tweak-able at runtime, which I love my
> self.  All of those will work though.
> Brock Palen
> www.umich.edu/~brockp <http://www.umich.edu/%7Ebrockp>
> Center for Advanced Computing
> brockp at umich.edu
> (734)936-1985
> On May 29, 2008, at 10:46 AM, Walid wrote:
> Hi,
> I do not have a problem yet, however in the next couple of days i might be
> :)
> We are deploying an IB large cluster, and with IB we have a choice of
> openMPI, mvapich, mvapich, among other commercial mpi implementations, our
> code used to run on mpich-gm over myrinet, and CISCO advice is to go with
> mvapich, our developers would like to have mvapcih2, and quick research on
> the net shows that openMPI is easier to depoly and have lesser issues
> My question is if you used any of the above and have any hints,gotches? or
> be aware of advice, could you please pass it on, or any useful URLs?
> Where is the best way to set the limits of memory unlocked so that pbs mom
> honours it?
> can you please share any epilogue/prolog that is needed for this kind of
> environments?
> regards
> Walid.
> _______________________________________________
> torqueusers mailing list
> torqueusers at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torqueusers
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/torqueusers/attachments/20080530/d8576a5e/attachment-0001.html

More information about the torqueusers mailing list