[torqueusers] The strange issue when submit job with pbs: libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes. and so on.

Hongsheng Zhao zhaohscas at yahoo.com.cn
Sun Nov 13 01:21:02 MST 2011


Hi all,

When I submit a job to my hpc with pbs script, I meet the following err 
and the job cann't start at all:

---------------
zhaohongsheng at node32:~/work/Dr.Zhao/castep_test_myhome> cat stderr
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
     This will severely limit memory registrations.
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
     This will severely limit memory registrations.
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
     This will severely limit memory registrations.
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
     This will severely limit memory registrations.
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
     This will severely limit memory registrations.
castepexe_mpi.exe: Rank 0:0: MPI_Init: ibv_create_cq() failed
castepexe_mpi.exe: Rank 0:0: MPI_Init: Can't initialize RDMA device
castepexe_mpi.exe: Rank 0:0: MPI_Init: MPI BUG: Cannot initialize RDMA 
protocol
MPI Application rank 0 exited before MPI_Init() with status 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line 
Source
libpthread.so.0    00002B52C15F6C00  Unknown               Unknown  Unknown
libmpi.so.1        00002B52C0F5FC8B  Unknown               Unknown  Unknown
libmpi.so.1        00002B52C0F3B57A  Unknown               Unknown  Unknown
libmpi.so.1        00002B52C0FAF03C  Unknown               Unknown  Unknown
libmpi.so.1        00002B52C0FBB3AB  Unknown               Unknown  Unknown
castepexe_mpi.exe  00000000004D6539  Unknown               Unknown  Unknown
castepexe_mpi.exe  0000000000416404  Unknown               Unknown  Unknown
castepexe_mpi.exe  0000000000412DDC  Unknown               Unknown  Unknown
libc.so.6          00002B52C171F184  Unknown               Unknown  Unknown
castepexe_mpi.exe  0000000000412D0A  Unknown               Unknown  Unknown
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line 
Source
libpthread.so.0    00002B197B97CC00  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B197B06AAF7  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B197B05D2E1  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B197B05F757  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B197B067FCD  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B197B0641F6  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B197B067ACB  Unknown               Unknown  Unknown
libdl.so.2         00002B197BDD71FA  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B197B0641F6  Unknown               Unknown  Unknown
libdl.so.2         00002B197BDD758D  Unknown               Unknown  Unknown
libdl.so.2         00002B197BDD7171  Unknown               Unknown  Unknown
libmpi.so.1        00002B197B36028A  Unknown               Unknown  Unknown
libmpi.so.1        00002B197B2C198E  Unknown               Unknown  Unknown
libmpi.so.1        00002B197B33503C  Unknown               Unknown  Unknown
libmpi.so.1        00002B197B3413AB  Unknown               Unknown  Unknown
castepexe_mpi.exe  00000000004D6539  Unknown               Unknown  Unknown
castepexe_mpi.exe  0000000000416404  Unknown               Unknown  Unknown
castepexe_mpi.exe  0000000000412DDC  Unknown               Unknown  Unknown
libc.so.6          00002B197BAA5184  Unknown               Unknown  Unknown
castepexe_mpi.exe  0000000000412D0A  Unknown               Unknown  Unknown
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line 
Source
libpthread.so.0    00002B8D54E6DC00  Unknown               Unknown  Unknown
libpthread.so.0    00002B8D54E6C900  Unknown               Unknown  Unknown
libibverbs.so      00002B8D57B8C3A4  Unknown               Unknown  Unknown
libibverbs.so      00002B8D57B8D06A  Unknown               Unknown  Unknown
libmpi.so.1        00002B8D547F1FA7  Unknown               Unknown  Unknown
libmpi.so.1        00002B8D547EBE36  Unknown               Unknown  Unknown
libmpi.so.1        00002B8D547D8F0C  Unknown               Unknown  Unknown
libmpi.so.1        00002B8D547B3E1D  Unknown               Unknown  Unknown
libmpi.so.1        00002B8D547B2999  Unknown               Unknown  Unknown
libmpi.so.1        00002B8D5482603C  Unknown               Unknown  Unknown
libmpi.so.1        00002B8D548323AB  Unknown               Unknown  Unknown
castepexe_mpi.exe  00000000004D6539  Unknown               Unknown  Unknown
castepexe_mpi.exe  0000000000416404  Unknown               Unknown  Unknown
castepexe_mpi.exe  0000000000412DDC  Unknown               Unknown  Unknown
libc.so.6          00002B8D54F96184  Unknown               Unknown  Unknown
castepexe_mpi.exe  0000000000412D0A  Unknown               Unknown  Unknown
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line 
Source
libpthread.so.0    00002B700FF46C00  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B700F634AF7  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B700F6272E1  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B700F627827  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B700F629987  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B700F631FCD  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B700F62E1F6  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B700F631ACB  Unknown               Unknown  Unknown
libdl.so.2         00002B70103A11FA  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B700F62E1F6  Unknown               Unknown  Unknown
libdl.so.2         00002B70103A158D  Unknown               Unknown  Unknown
libdl.so.2         00002B70103A1171  Unknown               Unknown  Unknown
libmpi.so.1        00002B700F8A28EC  Unknown               Unknown  Unknown
libmpi.so.1        00002B700F8A2A27  Unknown               Unknown  Unknown
libmpi.so.1        00002B700F8E34BF  Unknown               Unknown  Unknown
libmpi.so.1        00002B700F8C2C4B  Unknown               Unknown  Unknown
libmpi.so.1        00002B700F8B230C  Unknown               Unknown  Unknown
libmpi.so.1        00002B700F88CE1D  Unknown               Unknown  Unknown
libmpi.so.1        00002B700F88B999  Unknown               Unknown  Unknown
libmpi.so.1        00002B700F8FF03C  Unknown               Unknown  Unknown
libmpi.so.1        00002B700F90B3AB  Unknown               Unknown  Unknown
castepexe_mpi.exe  00000000004D6539  Unknown               Unknown  Unknown
castepexe_mpi.exe  0000000000416404  Unknown               Unknown  Unknown
castepexe_mpi.exe  0000000000412DDC  Unknown               Unknown  Unknown
libc.so.6          00002B701006F184  Unknown               Unknown  Unknown
castepexe_mpi.exe  0000000000412D0A  Unknown               Unknown  Unknown
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line 
Source
libpthread.so.0    00002B92E5C7BC00  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B92E5369AF7  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B92E535C2E1  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B92E535C827  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B92E535E987  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B92E5366FCD  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B92E53631F6  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B92E5366ACB  Unknown               Unknown  Unknown
libdl.so.2         00002B92E60D61FA  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B92E53631F6  Unknown               Unknown  Unknown
libdl.so.2         00002B92E60D658D  Unknown               Unknown  Unknown
libdl.so.2         00002B92E60D6171  Unknown               Unknown  Unknown
libmpi.so.1        00002B92E55D78EC  Unknown               Unknown  Unknown
libmpi.so.1        00002B92E55D7A27  Unknown               Unknown  Unknown
libmpi.so.1        00002B92E56184BF  Unknown               Unknown  Unknown
libmpi.so.1        00002B92E55F7C4B  Unknown               Unknown  Unknown
libmpi.so.1        00002B92E55E730C  Unknown               Unknown  Unknown
libmpi.so.1        00002B92E55C1E1D  Unknown               Unknown  Unknown
libmpi.so.1        00002B92E55C0999  Unknown               Unknown  Unknown
libmpi.so.1        00002B92E563403C  Unknown               Unknown  Unknown
libmpi.so.1        00002B92E56403AB  Unknown               Unknown  Unknown
castepexe_mpi.exe  00000000004D6539  Unknown               Unknown  Unknown
castepexe_mpi.exe  0000000000416404  Unknown               Unknown  Unknown
castepexe_mpi.exe  0000000000412DDC  Unknown               Unknown  Unknown
libc.so.6          00002B92E5DA4184  Unknown               Unknown  Unknown
castepexe_mpi.exe  0000000000412D0A  Unknown               Unknown  Unknown
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line 
Source
libpthread.so.0    00002AABE6311C00  Unknown               Unknown  Unknown
libpthread.so.0    00002AABE6310900  Unknown               Unknown  Unknown
libibverbs.so      00002AABE90303A4  Unknown               Unknown  Unknown
libibverbs.so      00002AABE903106A  Unknown               Unknown  Unknown
libmpi.so.1        00002AABE5C95FA7  Unknown               Unknown  Unknown
libmpi.so.1        00002AABE5C8FE36  Unknown               Unknown  Unknown
libmpi.so.1        00002AABE5C7CF0C  Unknown               Unknown  Unknown
libmpi.so.1        00002AABE5C57E1D  Unknown               Unknown  Unknown
libmpi.so.1        00002AABE5C56999  Unknown               Unknown  Unknown
libmpi.so.1        00002AABE5CCA03C  Unknown               Unknown  Unknown
libmpi.so.1        00002AABE5CD63AB  Unknown               Unknown  Unknown
castepexe_mpi.exe  00000000004D6539  Unknown               Unknown  Unknown
castepexe_mpi.exe  0000000000416404  Unknown               Unknown  Unknown
castepexe_mpi.exe  0000000000412DDC  Unknown               Unknown  Unknown
libc.so.6          00002AABE643A184  Unknown               Unknown  Unknown
castepexe_mpi.exe  0000000000412D0A  Unknown               Unknown  Unknown
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line 
Source
libpthread.so.0    00002B0D68F83C00  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B0D68671AF7  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B0D686642E1  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B0D68666757  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B0D6866EFCD  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B0D6866B1F6  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B0D6866EACB  Unknown               Unknown  Unknown
libdl.so.2         00002B0D693DE1FA  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B0D6866B1F6  Unknown               Unknown  Unknown
libdl.so.2         00002B0D693DE58D  Unknown               Unknown  Unknown
libdl.so.2         00002B0D693DE171  Unknown               Unknown  Unknown
libmpi.so.1        00002B0D6896728A  Unknown               Unknown  Unknown
libmpi.so.1        00002B0D688C898E  Unknown               Unknown  Unknown
libmpi.so.1        00002B0D6893C03C  Unknown               Unknown  Unknown
libmpi.so.1        00002B0D689483AB  Unknown               Unknown  Unknown
castepexe_mpi.exe  00000000004D6539  Unknown               Unknown  Unknown
castepexe_mpi.exe  0000000000416404  Unknown               Unknown  Unknown
castepexe_mpi.exe  0000000000412DDC  Unknown               Unknown  Unknown
libc.so.6          00002B0D690AC184  Unknown               Unknown  Unknown
castepexe_mpi.exe  0000000000412D0A  Unknown               Unknown  Unknown
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line 
Source
libpthread.so.0    00002B83A77EFC00  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B83A6ED34E0  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B83A6ED39B3  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B83A6ED5028  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B83A6EDB2A5  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B83A6ED71F6  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B83A6EDAACB  Unknown               Unknown  Unknown
libdl.so.2         00002B83A7C4A1FA  Unknown               Unknown  Unknown
ld-linux-x86-64.s  00002B83A6ED71F6  Unknown               Unknown  Unknown
libdl.so.2         00002B83A7C4A58D  Unknown               Unknown  Unknown
libdl.so.2         00002B83A7C4A171  Unknown               Unknown  Unknown
libmpi.so.1        00002B83A714B8EC  Unknown               Unknown  Unknown
libmpi.so.1        00002B83A714BA27  Unknown               Unknown  Unknown
libmpi.so.1        00002B83A718C4BF  Unknown               Unknown  Unknown
libmpi.so.1        00002B83A716BC4B  Unknown               Unknown  Unknown
libmpi.so.1        00002B83A715B30C  Unknown               Unknown  Unknown
libmpi.so.1        00002B83A7135E1D  Unknown               Unknown  Unknown
libmpi.so.1        00002B83A7134999  Unknown               Unknown  Unknown
libmpi.so.1        00002B83A71A803C  Unknown               Unknown  Unknown
libmpi.so.1        00002B83A71B43AB  Unknown               Unknown  Unknown
castepexe_mpi.exe  00000000004D6539  Unknown               Unknown  Unknown
castepexe_mpi.exe  0000000000416404  Unknown               Unknown  Unknown
castepexe_mpi.exe  0000000000412DDC  Unknown               Unknown  Unknown
libc.so.6          00002B83A7918184  Unknown               Unknown  Unknown
castepexe_mpi.exe  0000000000412D0A  Unknown               Unknown  Unknown
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line 
Source
libpthread.so.0    00002AC2DC492C00  Unknown               Unknown  Unknown
libpthread.so.0    00002AC2DC491900  Unknown               Unknown  Unknown
libibverbs.so      00002AC2DF1B13A4  Unknown               Unknown  Unknown
libibverbs.so      00002AC2DF1B206A  Unknown               Unknown  Unknown
libmpi.so.1        00002AC2DBE16FA7  Unknown               Unknown  Unknown
libmpi.so.1        00002AC2DBE10E36  Unknown               Unknown  Unknown
libmpi.so.1        00002AC2DBDFDF0C  Unknown               Unknown  Unknown
libmpi.so.1        00002AC2DBDD8E1D  Unknown               Unknown  Unknown
libmpi.so.1        00002AC2DBDD7999  Unknown               Unknown  Unknown
libmpi.so.1        00002AC2DBE4B03C  Unknown               Unknown  Unknown
libmpi.so.1        00002AC2DBE573AB  Unknown               Unknown  Unknown
castepexe_mpi.exe  00000000004D6539  Unknown               Unknown  Unknown
castepexe_mpi.exe  0000000000416404  Unknown               Unknown  Unknown
castepexe_mpi.exe  0000000000412DDC  Unknown               Unknown  Unknown
libc.so.6          00002AC2DC5BB184  Unknown               Unknown  Unknown
castepexe_mpi.exe  0000000000412D0A  Unknown               Unknown  Unknown
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line 
Source
plugin002_004_002  00002B195294BA80  Unknown               Unknown  Unknown
libmpi.so.1        00002B1951975108  Unknown               Unknown  Unknown
libmpi.so.1        00002B19518D698E  Unknown               Unknown  Unknown
libmpi.so.1        00002B195194A03C  Unknown               Unknown  Unknown
libmpi.so.1        00002B19519563AB  Unknown               Unknown  Unknown
castepexe_mpi.exe  00000000004D6539  Unknown               Unknown  Unknown
castepexe_mpi.exe  0000000000416404  Unknown               Unknown  Unknown
castepexe_mpi.exe  0000000000412DDC  Unknown               Unknown  Unknown
libc.so.6          00002B19520BA184  Unknown               Unknown  Unknown
castepexe_mpi.exe  0000000000412D0A  Unknown               Unknown  Unknown
mpirun: Broken pipe
-----------------

Could you please give me some hints?  Thanks in advance.

Regards
-- 
Hongsheng Zhao <zhaohscas at yahoo.com.cn>
School of Physics and Electrical Information Science,
Ningxia University, Yinchuan 750021, China


More information about the torqueusers mailing list