[torqueusers] TM improvements

Garrick Staples garrick at usc.edu
Tue Nov 22 23:06:56 MST 2005

I think I need more specifics.  It's obvious to me that you know a lot
more about using TM than I do.

On Tue, Nov 22, 2005 at 10:50:33PM -0500, Jeff Squyres alleged:
> --> Mark this one as only partially solved.  Yes, we can keep a TM 
> connection open for the duration of the MPI job, but you still can't 
> have that mpirun disconnect from a running job and still retain the 
> ability to tm_kill() any of the tm_spawned() processes later -- perhaps 
> even from something other than mpirun.  This would be *extremely* 
> useful (think "screen" for MPI jobs).

I'm lost on this one.  If you have two different MPI processes launching
different sets of tasks, and they both exit and reattach, even if we
supported retrieving the list of running tasks, how do the two mpiruns
know which tasks belong to which MPI job?  How do the two mpiruns know
which is itself?

With 'screen', I can specify a pid, tty, or a named session.  But I
can't think of an equivalent for mpirun.

> 3. If you tm_spawn() something that fails to launch properly (e.g., 
> file not found on the remote node), there is no error notification sent 
> back to the process that invoked tm_spawn().


$ pbsdsh -c 1 lkjahdsfljahdf
pbsdsh: task 0 exit status 254

$ pbsdsh -c 2 lkjahdsfljahdf
pbsdsh: task 0 exit status 254
pbsdsh: task 1 exit status 254

> 4. It would also be nice to have a "group" spawn -- where mpirun can 
> issue a single tm_spawn() and have it launch multiple processes at 
> once.  Even something simple to handle the common SPMD case (e.g., 
> "mpirun -np 1024 a.out") would be nice.  This pushes the scalability 
> issues down into TM.  True, you might simply do a simple linear loop, 
> but it at least allows for the *possibility* of a scalable launch 
> (where scalable = something better than linear).  With 
> uni-tm_spawn()'s, there is no possibility of anything better than 
> linear.

Pete Wycoff and I were talking about this at SC05 last week.  We never
came up with a decent interface that lets us specify different args/env
for each task.

> >>As a consumer of the TM interface, it would be *really great* if there
> >>was only *one* set of these things to check against.  If we have to
> >>splinter our configure script to check for different vendors and
> >>different variants, it will be a complete and total nightmare (well,
> >>more than the nightmare that our configure script already is! ;-) ).
> >
> >I really can't comment on what other PBS implementations are doing.  I
> >don't have access to their commercial software, nor would I want to
> >cause any misunderstandings.  To be honest, I have no idea what kind of
> >feature-parity we have PBSpro, SGE, etc.  I'm really just focusing on
> >TORQUE at this time.
> Ok.  Can you initiate discussions with them?  Consider the TM 
> consumers' perspectives (including mine): we absolutely do not want N 
> different TM implementations out that there are different and have 
> different tests to establish how they're different.  That becomes a 

I'll talk to Dave about this.  He hangs out with commercial peeps.

Can you get all the MPI vendors to use the same launch protocol? :)

Garrick Staples, Linux/HPCC Administrator
University of Southern California
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://www.supercluster.org/pipermail/torqueusers/attachments/20051122/8ed87c5a/attachment.bin

More information about the torqueusers mailing list