[torqueusers] RE : How to run mpirun of intel on torque

David Roman David.Roman at noveltis.fr
Wed Dec 19 06:59:01 MST 2012


I think I solved my problem.
To prevent that users use more than their resources requested, I use cpuset with torque.
On each executable nodes, all process are running under PBS have /proc/<PID>/cpuset with the definition of cpuset, other process have only / in this file.
In crontab I check all user process and I kill all jobs are not running under PBS .


-----Message d'origine-----
De : torqueusers-bounces at supercluster.org [mailto:torqueusers-bounces at supercluster.org] De la part de David Roman
Envoyé : samedi 15 décembre 2012 10:49
À : Torque Users Mailing List
Objet : [torqueusers] RE : How to run mpirun of intel on torque

Thank for your reply, I think that it can be my solution.
Could you show an example of script please ?


De : torqueusers-bounces at supercluster.org [torqueusers-bounces at supercluster.org] de la part de Brian Haymore [brian.haymore at utah.edu] Date d'envoi : vendredi 14 décembre 2012 19:56 À : Torque Users Mailing List Objet : Re: [torqueusers] How to run mpirun of intel on torque

There are no quotas present on the scratch file systems. So for you to be getting a quota limited  message tells me that somehow the tool you're using is trying to write something to your user home directory. You should look  to clean up some of your home directory to bring yourself below your quota limit .
Brian D Haymore
University of Utah
Center for High Performance Computing
155 South 1452 East, RM 405
Salt Lake City, Utah 84112
Phone: 801-558-1150, Fax: 801-585-5366

"Coyle, James J [ITACD]" <jjc at iastate.edu> wrote:

  I have a script that I run from /var/spool/mom_priv/prologue whenever a josb that requires all the cores on the node.  This script kills any user level processes that are not the job owner or a torque manager.

  This gets rid of any effect of users who should not be logged on to the compute node, that is who have just ssh'd in rather than using qsub -I

   I clean out scratch space on exit with epilogue if it is a dedicated job.

    You could couple this with techniques to prevent ssh logins durin the job (For jobs which dedicate the node, have prologue disallow logins from other users, and restore normal function on exit.)

From: torqueusers-bounces at supercluster.org [mailto:torqueusers-bounces at supercluster.org] On Behalf Of David Roman
Sent: Friday, December 14, 2012 6:53 AM
To: 'torqueusers at supercluster.org'
Subject: [torqueusers] How to run mpirun of intel on torque


I am sorry, but my english is really sad. I thank you for your patience for that.
I installed TORQUE 4.1.0, with MAU 3.3.1. I compiled OPENMPI with option --with-tm=/usr/local/torque.
I disallowed ssh connections for my users on executable nodes. In /etc/ssh/sshd_config I set AllowGroups root admin This works fine.

But now, I install MPI Intel Librarie and Ifort.
When I open a interactive pbs session:
qsub -I -l nodes=2:ppn=8

I am connected on a node.

I run mpi job

mpirun -genv I_MPI_FABRICS_LIST tmi ./my_program

But it can  not start, because it cannot connect on the other node.

If I append users in AllowGroups  of  /etc/ssh/sshd_config it works.

But if I do this, all users can connect on executable nodes, without use torque, and this is bad.

How can I do to disallow ssh connection without torque or make mpirun of intel works like openmpi, without ssh connection allowed for users ?

Thank you


torqueusers mailing list
torqueusers at supercluster.org

More information about the torqueusers mailing list