[torqueusers] The strange issue when submit job with pbs: libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes. and so on.

Lloyd Brown lloyd_brown at byu.edu
Mon Nov 14 07:41:23 MST 2011


I'm not sure I'd go so far as to say it's definitely unrelated to
Torque.  We ran into a similar problem, and in the end it was a
limitation imposed by the shell/environment when the pbs_mom started. 
Since jobs were child processes of the pbs_mom, they inherited the
memory limits.

Since we're using the RPM packages, and the contrib'd init script, the
best solution was to put the following lines in /etc/sysconfig/pbs_mom;
if you're using a different environment, etc., you may have to adjust
the method you use to do something similar:

> #installation-specific limits for pbs_mom
> ulimit -l unlimited
> ulimit -n 32768

There are a lot of opinions about what the right limits to put here are,
as well.

Lloyd Brown




On 11/13/11 2:25 AM, Stijn De Weirdt wrote:
> this is unrelated to torque, but here's the relevant faq entry for openmpi
>
> http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages
>
> stijn
>
> On 11/13/2011 09:21 AM, Hongsheng Zhao wrote:
>> Hi all,
>>
>> When I submit a job to my hpc with pbs script, I meet the following err
>> and the job cann't start at all:
>>
>> ---------------
>> zhaohongsheng at node32:~/work/Dr.Zhao/castep_test_myhome>  cat stderr
>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>       This will severely limit memory registrations.
>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>       This will severely limit memory registrations.
>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>       This will severely limit memory registrations.
>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>       This will severely limit memory registrations.
>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>>       This will severely limit memory registrations.
>> castepexe_mpi.exe: Rank 0:0: MPI_Init: ibv_create_cq() failed
>> castepexe_mpi.exe: Rank 0:0: MPI_Init: Can't initialize RDMA device
>> castepexe_mpi.exe: Rank 0:0: MPI_Init: MPI BUG: Cannot initialize RDMA
>> protocol
>> MPI Application rank 0 exited before MPI_Init() with status 1
>> forrtl: error (78): process killed (SIGTERM)
>> Image              PC                Routine            Line
>> Source
>> libpthread.so.0    00002B52C15F6C00  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B52C0F5FC8B  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B52C0F3B57A  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B52C0FAF03C  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B52C0FBB3AB  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  00000000004D6539  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  0000000000416404  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  0000000000412DDC  Unknown               Unknown  Unknown
>> libc.so.6          00002B52C171F184  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  0000000000412D0A  Unknown               Unknown  Unknown
>> forrtl: error (78): process killed (SIGTERM)
>> Image              PC                Routine            Line
>> Source
>> libpthread.so.0    00002B197B97CC00  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B197B06AAF7  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B197B05D2E1  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B197B05F757  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B197B067FCD  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B197B0641F6  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B197B067ACB  Unknown               Unknown  Unknown
>> libdl.so.2         00002B197BDD71FA  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B197B0641F6  Unknown               Unknown  Unknown
>> libdl.so.2         00002B197BDD758D  Unknown               Unknown  Unknown
>> libdl.so.2         00002B197BDD7171  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B197B36028A  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B197B2C198E  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B197B33503C  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B197B3413AB  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  00000000004D6539  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  0000000000416404  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  0000000000412DDC  Unknown               Unknown  Unknown
>> libc.so.6          00002B197BAA5184  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  0000000000412D0A  Unknown               Unknown  Unknown
>> forrtl: error (78): process killed (SIGTERM)
>> Image              PC                Routine            Line
>> Source
>> libpthread.so.0    00002B8D54E6DC00  Unknown               Unknown  Unknown
>> libpthread.so.0    00002B8D54E6C900  Unknown               Unknown  Unknown
>> libibverbs.so      00002B8D57B8C3A4  Unknown               Unknown  Unknown
>> libibverbs.so      00002B8D57B8D06A  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B8D547F1FA7  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B8D547EBE36  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B8D547D8F0C  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B8D547B3E1D  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B8D547B2999  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B8D5482603C  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B8D548323AB  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  00000000004D6539  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  0000000000416404  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  0000000000412DDC  Unknown               Unknown  Unknown
>> libc.so.6          00002B8D54F96184  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  0000000000412D0A  Unknown               Unknown  Unknown
>> forrtl: error (78): process killed (SIGTERM)
>> Image              PC                Routine            Line
>> Source
>> libpthread.so.0    00002B700FF46C00  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B700F634AF7  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B700F6272E1  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B700F627827  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B700F629987  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B700F631FCD  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B700F62E1F6  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B700F631ACB  Unknown               Unknown  Unknown
>> libdl.so.2         00002B70103A11FA  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B700F62E1F6  Unknown               Unknown  Unknown
>> libdl.so.2         00002B70103A158D  Unknown               Unknown  Unknown
>> libdl.so.2         00002B70103A1171  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B700F8A28EC  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B700F8A2A27  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B700F8E34BF  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B700F8C2C4B  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B700F8B230C  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B700F88CE1D  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B700F88B999  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B700F8FF03C  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B700F90B3AB  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  00000000004D6539  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  0000000000416404  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  0000000000412DDC  Unknown               Unknown  Unknown
>> libc.so.6          00002B701006F184  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  0000000000412D0A  Unknown               Unknown  Unknown
>> forrtl: error (78): process killed (SIGTERM)
>> Image              PC                Routine            Line
>> Source
>> libpthread.so.0    00002B92E5C7BC00  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B92E5369AF7  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B92E535C2E1  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B92E535C827  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B92E535E987  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B92E5366FCD  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B92E53631F6  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B92E5366ACB  Unknown               Unknown  Unknown
>> libdl.so.2         00002B92E60D61FA  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B92E53631F6  Unknown               Unknown  Unknown
>> libdl.so.2         00002B92E60D658D  Unknown               Unknown  Unknown
>> libdl.so.2         00002B92E60D6171  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B92E55D78EC  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B92E55D7A27  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B92E56184BF  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B92E55F7C4B  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B92E55E730C  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B92E55C1E1D  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B92E55C0999  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B92E563403C  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B92E56403AB  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  00000000004D6539  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  0000000000416404  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  0000000000412DDC  Unknown               Unknown  Unknown
>> libc.so.6          00002B92E5DA4184  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  0000000000412D0A  Unknown               Unknown  Unknown
>> forrtl: error (78): process killed (SIGTERM)
>> Image              PC                Routine            Line
>> Source
>> libpthread.so.0    00002AABE6311C00  Unknown               Unknown  Unknown
>> libpthread.so.0    00002AABE6310900  Unknown               Unknown  Unknown
>> libibverbs.so      00002AABE90303A4  Unknown               Unknown  Unknown
>> libibverbs.so      00002AABE903106A  Unknown               Unknown  Unknown
>> libmpi.so.1        00002AABE5C95FA7  Unknown               Unknown  Unknown
>> libmpi.so.1        00002AABE5C8FE36  Unknown               Unknown  Unknown
>> libmpi.so.1        00002AABE5C7CF0C  Unknown               Unknown  Unknown
>> libmpi.so.1        00002AABE5C57E1D  Unknown               Unknown  Unknown
>> libmpi.so.1        00002AABE5C56999  Unknown               Unknown  Unknown
>> libmpi.so.1        00002AABE5CCA03C  Unknown               Unknown  Unknown
>> libmpi.so.1        00002AABE5CD63AB  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  00000000004D6539  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  0000000000416404  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  0000000000412DDC  Unknown               Unknown  Unknown
>> libc.so.6          00002AABE643A184  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  0000000000412D0A  Unknown               Unknown  Unknown
>> forrtl: error (78): process killed (SIGTERM)
>> Image              PC                Routine            Line
>> Source
>> libpthread.so.0    00002B0D68F83C00  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B0D68671AF7  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B0D686642E1  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B0D68666757  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B0D6866EFCD  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B0D6866B1F6  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B0D6866EACB  Unknown               Unknown  Unknown
>> libdl.so.2         00002B0D693DE1FA  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B0D6866B1F6  Unknown               Unknown  Unknown
>> libdl.so.2         00002B0D693DE58D  Unknown               Unknown  Unknown
>> libdl.so.2         00002B0D693DE171  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B0D6896728A  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B0D688C898E  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B0D6893C03C  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B0D689483AB  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  00000000004D6539  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  0000000000416404  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  0000000000412DDC  Unknown               Unknown  Unknown
>> libc.so.6          00002B0D690AC184  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  0000000000412D0A  Unknown               Unknown  Unknown
>> forrtl: error (78): process killed (SIGTERM)
>> Image              PC                Routine            Line
>> Source
>> libpthread.so.0    00002B83A77EFC00  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B83A6ED34E0  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B83A6ED39B3  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B83A6ED5028  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B83A6EDB2A5  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B83A6ED71F6  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B83A6EDAACB  Unknown               Unknown  Unknown
>> libdl.so.2         00002B83A7C4A1FA  Unknown               Unknown  Unknown
>> ld-linux-x86-64.s  00002B83A6ED71F6  Unknown               Unknown  Unknown
>> libdl.so.2         00002B83A7C4A58D  Unknown               Unknown  Unknown
>> libdl.so.2         00002B83A7C4A171  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B83A714B8EC  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B83A714BA27  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B83A718C4BF  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B83A716BC4B  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B83A715B30C  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B83A7135E1D  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B83A7134999  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B83A71A803C  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B83A71B43AB  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  00000000004D6539  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  0000000000416404  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  0000000000412DDC  Unknown               Unknown  Unknown
>> libc.so.6          00002B83A7918184  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  0000000000412D0A  Unknown               Unknown  Unknown
>> forrtl: error (78): process killed (SIGTERM)
>> Image              PC                Routine            Line
>> Source
>> libpthread.so.0    00002AC2DC492C00  Unknown               Unknown  Unknown
>> libpthread.so.0    00002AC2DC491900  Unknown               Unknown  Unknown
>> libibverbs.so      00002AC2DF1B13A4  Unknown               Unknown  Unknown
>> libibverbs.so      00002AC2DF1B206A  Unknown               Unknown  Unknown
>> libmpi.so.1        00002AC2DBE16FA7  Unknown               Unknown  Unknown
>> libmpi.so.1        00002AC2DBE10E36  Unknown               Unknown  Unknown
>> libmpi.so.1        00002AC2DBDFDF0C  Unknown               Unknown  Unknown
>> libmpi.so.1        00002AC2DBDD8E1D  Unknown               Unknown  Unknown
>> libmpi.so.1        00002AC2DBDD7999  Unknown               Unknown  Unknown
>> libmpi.so.1        00002AC2DBE4B03C  Unknown               Unknown  Unknown
>> libmpi.so.1        00002AC2DBE573AB  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  00000000004D6539  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  0000000000416404  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  0000000000412DDC  Unknown               Unknown  Unknown
>> libc.so.6          00002AC2DC5BB184  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  0000000000412D0A  Unknown               Unknown  Unknown
>> forrtl: error (78): process killed (SIGTERM)
>> Image              PC                Routine            Line
>> Source
>> plugin002_004_002  00002B195294BA80  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B1951975108  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B19518D698E  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B195194A03C  Unknown               Unknown  Unknown
>> libmpi.so.1        00002B19519563AB  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  00000000004D6539  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  0000000000416404  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  0000000000412DDC  Unknown               Unknown  Unknown
>> libc.so.6          00002B19520BA184  Unknown               Unknown  Unknown
>> castepexe_mpi.exe  0000000000412D0A  Unknown               Unknown  Unknown
>> mpirun: Broken pipe
>> -----------------
>>
>> Could you please give me some hints?  Thanks in advance.
>>
>> Regards
> _______________________________________________
> torqueusers mailing list
> torqueusers at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torqueusers



More information about the torqueusers mailing list