[torqueusers] qsub and limits.conf

Javier Lazaro flg869 at gmail.com
Sun May 18 06:53:45 MDT 2008


I have install torque-2.3.0 and openmpi-1.2.3.
I make tests and I have discovered that the work launched with the
parameter '-hostfile' or '-machinefile' stops are to exceed the limits
in the file /etc/security/limits.conf
More details:

file hola.c

#include <stdio.h>
#include <unistd.h>
#include "mpi.h"
int main(int argc, char *argv[]){
        int rank;
        int size;
        int i;
        int namelen;
        char pn[MPI_MAX_PROCESSOR_NAME];

        MPI_Init(&argc,&argv);
        MPI_Comm_size(MPI_COMM_WORLD,
&size);
        MPI_Comm_rank(MPI_COMM_WORLD,&rank);
        MPI_Get_processor_name(pn,&namelen);

        sleep(rank);
        system("bash -c 'ulimit -a'");

        for (i=0;;i++) {
                if (i%100000000==0) {
                        printf("--> %i --> Hola desde %d, de un total
de: %d. estoy en %s\n",i, rank, size,pn);
                }
        }
        MPI_Finalize();

        return 0;

}

##

> mpicc hola.c

file mpi3.sh

#!/bin/sh

#PBS -l nodes=3:ppn=1
#PBS -N pruebaMPI3
#PBS -o 3outpruebaMPIout3
#PBS -e 3errpruebaMPIerr3

cat ${PBS_NODEFILE}

mpirun -hostfile ${PBS_NODEFILE} /home/javier/mpi_hola/a.out

##

launch job with torque
> qsub mpi3.sh

##

termined

file 3outpruebaMPIout3
maquina3b
maquina2b
maquina1b
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
pending signals                 (-i) 8185
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited            #limit maquina3b
max user processes              (-u) 8185
virtual memory          (kbytes, -v) 2511840
file locks                      (-x) unlimited
--> 0 --> Hola desde 0, de un total de: 3. estoy en maquina3b
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
pending signals                 (-i) 8185
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) 880005
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) 60        #limit maquina2b
max user processes              (-u) 8185
virtual memory          (kbytes, -v) 2511840
file locks                      (-x) unlimited
--> 0 --> Hola desde 1, de un total de: 3. estoy en maquina2b
--> 100000000 --> Hola desde 0, de un total de: 3. estoy en maquina3b
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
pending signals                 (-i) 8185
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) 880005
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) 60        #limit maquina1b
max user processes              (-u) 8185
virtual memory          (kbytes, -v) 2511840
file locks                      (-x) unlimited
--> 0 --> Hola desde 2, de un total de: 3. estoy en maquina1b
--> 100000000 --> Hola desde 1, de un total de: 3. estoy en maquina2b
--> 200000000 --> Hola desde 0, de un total de: 3. estoy en maquina3b
--> 100000000 --> Hola desde 2, de un total de: 3. estoy en maquina1b
--> 200000000 --> Hola desde 1, de un total de: 3. estoy en maquina2b
........
--> -500000000 --> Hola desde 1, de un total de: 3. estoy en maquina2b
1 additional process aborted (not shown)
1 process killed (possibly by Open MPI)

##

file 3errpruebaMPIerr3

mpirun noticed that job rank 0 with PID 10839 on node maquina3b exited
on signal 15 (Terminated).

---------------------------
I have limited time of cpu at 60 seconds in all nodes. Torque modify
this limit only for maquina3b.
I think that torque should modify cpu's limit in the resf of nodes.
where is the error?


More information about the torqueusers mailing list