[torqueusers] The strange issue when submit job with pbs: libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes. and so on.
Lloyd Brown
lloyd_brown at byu.edu
Mon Nov 14 07:41:23 MST 2011
I'm not sure I'd go so far as to say it's definitely unrelated to
Torque. We ran into a similar problem, and in the end it was a
limitation imposed by the shell/environment when the pbs_mom started.
Since jobs were child processes of the pbs_mom, they inherited the
memory limits.
Since we're using the RPM packages, and the contrib'd init script, the
best solution was to put the following lines in /etc/sysconfig/pbs_mom;
if you're using a different environment, etc., you may have to adjust
the method you use to do something similar:
> #installation-specific limits for pbs_mom
> ulimit -l unlimited
> ulimit -n 32768
There are a lot of opinions about what the right limits to put here are,
as well.
Lloyd Brown
On 11/13/11 2:25 AM, Stijn De Weirdt wrote:
> this is unrelated to torque, but here's the relevant faq entry for openmpi
>
> http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages
>
> stijn
>
> On 11/13/2011 09:21 AM, Hongsheng Zhao wrote:
>> Hi all,
>>
>> When I submit a job to my hpc with pbs script, I meet the following err
>> and the job cann't start at all:
>>
>> ---------------
>> zhaohongsheng at node32:~/work/Dr.Zhao/castep_test_myhome> cat stderr
>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>> This will severely limit memory registrations.
>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>> This will severely limit memory registrations.
>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>> This will severely limit memory registrations.
>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>> This will severely limit memory registrations.
>> libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
>> This will severely limit memory registrations.
>> castepexe_mpi.exe: Rank 0:0: MPI_Init: ibv_create_cq() failed
>> castepexe_mpi.exe: Rank 0:0: MPI_Init: Can't initialize RDMA device
>> castepexe_mpi.exe: Rank 0:0: MPI_Init: MPI BUG: Cannot initialize RDMA
>> protocol
>> MPI Application rank 0 exited before MPI_Init() with status 1
>> forrtl: error (78): process killed (SIGTERM)
>> Image PC Routine Line
>> Source
>> libpthread.so.0 00002B52C15F6C00 Unknown Unknown Unknown
>> libmpi.so.1 00002B52C0F5FC8B Unknown Unknown Unknown
>> libmpi.so.1 00002B52C0F3B57A Unknown Unknown Unknown
>> libmpi.so.1 00002B52C0FAF03C Unknown Unknown Unknown
>> libmpi.so.1 00002B52C0FBB3AB Unknown Unknown Unknown
>> castepexe_mpi.exe 00000000004D6539 Unknown Unknown Unknown
>> castepexe_mpi.exe 0000000000416404 Unknown Unknown Unknown
>> castepexe_mpi.exe 0000000000412DDC Unknown Unknown Unknown
>> libc.so.6 00002B52C171F184 Unknown Unknown Unknown
>> castepexe_mpi.exe 0000000000412D0A Unknown Unknown Unknown
>> forrtl: error (78): process killed (SIGTERM)
>> Image PC Routine Line
>> Source
>> libpthread.so.0 00002B197B97CC00 Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B197B06AAF7 Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B197B05D2E1 Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B197B05F757 Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B197B067FCD Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B197B0641F6 Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B197B067ACB Unknown Unknown Unknown
>> libdl.so.2 00002B197BDD71FA Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B197B0641F6 Unknown Unknown Unknown
>> libdl.so.2 00002B197BDD758D Unknown Unknown Unknown
>> libdl.so.2 00002B197BDD7171 Unknown Unknown Unknown
>> libmpi.so.1 00002B197B36028A Unknown Unknown Unknown
>> libmpi.so.1 00002B197B2C198E Unknown Unknown Unknown
>> libmpi.so.1 00002B197B33503C Unknown Unknown Unknown
>> libmpi.so.1 00002B197B3413AB Unknown Unknown Unknown
>> castepexe_mpi.exe 00000000004D6539 Unknown Unknown Unknown
>> castepexe_mpi.exe 0000000000416404 Unknown Unknown Unknown
>> castepexe_mpi.exe 0000000000412DDC Unknown Unknown Unknown
>> libc.so.6 00002B197BAA5184 Unknown Unknown Unknown
>> castepexe_mpi.exe 0000000000412D0A Unknown Unknown Unknown
>> forrtl: error (78): process killed (SIGTERM)
>> Image PC Routine Line
>> Source
>> libpthread.so.0 00002B8D54E6DC00 Unknown Unknown Unknown
>> libpthread.so.0 00002B8D54E6C900 Unknown Unknown Unknown
>> libibverbs.so 00002B8D57B8C3A4 Unknown Unknown Unknown
>> libibverbs.so 00002B8D57B8D06A Unknown Unknown Unknown
>> libmpi.so.1 00002B8D547F1FA7 Unknown Unknown Unknown
>> libmpi.so.1 00002B8D547EBE36 Unknown Unknown Unknown
>> libmpi.so.1 00002B8D547D8F0C Unknown Unknown Unknown
>> libmpi.so.1 00002B8D547B3E1D Unknown Unknown Unknown
>> libmpi.so.1 00002B8D547B2999 Unknown Unknown Unknown
>> libmpi.so.1 00002B8D5482603C Unknown Unknown Unknown
>> libmpi.so.1 00002B8D548323AB Unknown Unknown Unknown
>> castepexe_mpi.exe 00000000004D6539 Unknown Unknown Unknown
>> castepexe_mpi.exe 0000000000416404 Unknown Unknown Unknown
>> castepexe_mpi.exe 0000000000412DDC Unknown Unknown Unknown
>> libc.so.6 00002B8D54F96184 Unknown Unknown Unknown
>> castepexe_mpi.exe 0000000000412D0A Unknown Unknown Unknown
>> forrtl: error (78): process killed (SIGTERM)
>> Image PC Routine Line
>> Source
>> libpthread.so.0 00002B700FF46C00 Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B700F634AF7 Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B700F6272E1 Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B700F627827 Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B700F629987 Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B700F631FCD Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B700F62E1F6 Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B700F631ACB Unknown Unknown Unknown
>> libdl.so.2 00002B70103A11FA Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B700F62E1F6 Unknown Unknown Unknown
>> libdl.so.2 00002B70103A158D Unknown Unknown Unknown
>> libdl.so.2 00002B70103A1171 Unknown Unknown Unknown
>> libmpi.so.1 00002B700F8A28EC Unknown Unknown Unknown
>> libmpi.so.1 00002B700F8A2A27 Unknown Unknown Unknown
>> libmpi.so.1 00002B700F8E34BF Unknown Unknown Unknown
>> libmpi.so.1 00002B700F8C2C4B Unknown Unknown Unknown
>> libmpi.so.1 00002B700F8B230C Unknown Unknown Unknown
>> libmpi.so.1 00002B700F88CE1D Unknown Unknown Unknown
>> libmpi.so.1 00002B700F88B999 Unknown Unknown Unknown
>> libmpi.so.1 00002B700F8FF03C Unknown Unknown Unknown
>> libmpi.so.1 00002B700F90B3AB Unknown Unknown Unknown
>> castepexe_mpi.exe 00000000004D6539 Unknown Unknown Unknown
>> castepexe_mpi.exe 0000000000416404 Unknown Unknown Unknown
>> castepexe_mpi.exe 0000000000412DDC Unknown Unknown Unknown
>> libc.so.6 00002B701006F184 Unknown Unknown Unknown
>> castepexe_mpi.exe 0000000000412D0A Unknown Unknown Unknown
>> forrtl: error (78): process killed (SIGTERM)
>> Image PC Routine Line
>> Source
>> libpthread.so.0 00002B92E5C7BC00 Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B92E5369AF7 Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B92E535C2E1 Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B92E535C827 Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B92E535E987 Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B92E5366FCD Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B92E53631F6 Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B92E5366ACB Unknown Unknown Unknown
>> libdl.so.2 00002B92E60D61FA Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B92E53631F6 Unknown Unknown Unknown
>> libdl.so.2 00002B92E60D658D Unknown Unknown Unknown
>> libdl.so.2 00002B92E60D6171 Unknown Unknown Unknown
>> libmpi.so.1 00002B92E55D78EC Unknown Unknown Unknown
>> libmpi.so.1 00002B92E55D7A27 Unknown Unknown Unknown
>> libmpi.so.1 00002B92E56184BF Unknown Unknown Unknown
>> libmpi.so.1 00002B92E55F7C4B Unknown Unknown Unknown
>> libmpi.so.1 00002B92E55E730C Unknown Unknown Unknown
>> libmpi.so.1 00002B92E55C1E1D Unknown Unknown Unknown
>> libmpi.so.1 00002B92E55C0999 Unknown Unknown Unknown
>> libmpi.so.1 00002B92E563403C Unknown Unknown Unknown
>> libmpi.so.1 00002B92E56403AB Unknown Unknown Unknown
>> castepexe_mpi.exe 00000000004D6539 Unknown Unknown Unknown
>> castepexe_mpi.exe 0000000000416404 Unknown Unknown Unknown
>> castepexe_mpi.exe 0000000000412DDC Unknown Unknown Unknown
>> libc.so.6 00002B92E5DA4184 Unknown Unknown Unknown
>> castepexe_mpi.exe 0000000000412D0A Unknown Unknown Unknown
>> forrtl: error (78): process killed (SIGTERM)
>> Image PC Routine Line
>> Source
>> libpthread.so.0 00002AABE6311C00 Unknown Unknown Unknown
>> libpthread.so.0 00002AABE6310900 Unknown Unknown Unknown
>> libibverbs.so 00002AABE90303A4 Unknown Unknown Unknown
>> libibverbs.so 00002AABE903106A Unknown Unknown Unknown
>> libmpi.so.1 00002AABE5C95FA7 Unknown Unknown Unknown
>> libmpi.so.1 00002AABE5C8FE36 Unknown Unknown Unknown
>> libmpi.so.1 00002AABE5C7CF0C Unknown Unknown Unknown
>> libmpi.so.1 00002AABE5C57E1D Unknown Unknown Unknown
>> libmpi.so.1 00002AABE5C56999 Unknown Unknown Unknown
>> libmpi.so.1 00002AABE5CCA03C Unknown Unknown Unknown
>> libmpi.so.1 00002AABE5CD63AB Unknown Unknown Unknown
>> castepexe_mpi.exe 00000000004D6539 Unknown Unknown Unknown
>> castepexe_mpi.exe 0000000000416404 Unknown Unknown Unknown
>> castepexe_mpi.exe 0000000000412DDC Unknown Unknown Unknown
>> libc.so.6 00002AABE643A184 Unknown Unknown Unknown
>> castepexe_mpi.exe 0000000000412D0A Unknown Unknown Unknown
>> forrtl: error (78): process killed (SIGTERM)
>> Image PC Routine Line
>> Source
>> libpthread.so.0 00002B0D68F83C00 Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B0D68671AF7 Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B0D686642E1 Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B0D68666757 Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B0D6866EFCD Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B0D6866B1F6 Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B0D6866EACB Unknown Unknown Unknown
>> libdl.so.2 00002B0D693DE1FA Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B0D6866B1F6 Unknown Unknown Unknown
>> libdl.so.2 00002B0D693DE58D Unknown Unknown Unknown
>> libdl.so.2 00002B0D693DE171 Unknown Unknown Unknown
>> libmpi.so.1 00002B0D6896728A Unknown Unknown Unknown
>> libmpi.so.1 00002B0D688C898E Unknown Unknown Unknown
>> libmpi.so.1 00002B0D6893C03C Unknown Unknown Unknown
>> libmpi.so.1 00002B0D689483AB Unknown Unknown Unknown
>> castepexe_mpi.exe 00000000004D6539 Unknown Unknown Unknown
>> castepexe_mpi.exe 0000000000416404 Unknown Unknown Unknown
>> castepexe_mpi.exe 0000000000412DDC Unknown Unknown Unknown
>> libc.so.6 00002B0D690AC184 Unknown Unknown Unknown
>> castepexe_mpi.exe 0000000000412D0A Unknown Unknown Unknown
>> forrtl: error (78): process killed (SIGTERM)
>> Image PC Routine Line
>> Source
>> libpthread.so.0 00002B83A77EFC00 Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B83A6ED34E0 Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B83A6ED39B3 Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B83A6ED5028 Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B83A6EDB2A5 Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B83A6ED71F6 Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B83A6EDAACB Unknown Unknown Unknown
>> libdl.so.2 00002B83A7C4A1FA Unknown Unknown Unknown
>> ld-linux-x86-64.s 00002B83A6ED71F6 Unknown Unknown Unknown
>> libdl.so.2 00002B83A7C4A58D Unknown Unknown Unknown
>> libdl.so.2 00002B83A7C4A171 Unknown Unknown Unknown
>> libmpi.so.1 00002B83A714B8EC Unknown Unknown Unknown
>> libmpi.so.1 00002B83A714BA27 Unknown Unknown Unknown
>> libmpi.so.1 00002B83A718C4BF Unknown Unknown Unknown
>> libmpi.so.1 00002B83A716BC4B Unknown Unknown Unknown
>> libmpi.so.1 00002B83A715B30C Unknown Unknown Unknown
>> libmpi.so.1 00002B83A7135E1D Unknown Unknown Unknown
>> libmpi.so.1 00002B83A7134999 Unknown Unknown Unknown
>> libmpi.so.1 00002B83A71A803C Unknown Unknown Unknown
>> libmpi.so.1 00002B83A71B43AB Unknown Unknown Unknown
>> castepexe_mpi.exe 00000000004D6539 Unknown Unknown Unknown
>> castepexe_mpi.exe 0000000000416404 Unknown Unknown Unknown
>> castepexe_mpi.exe 0000000000412DDC Unknown Unknown Unknown
>> libc.so.6 00002B83A7918184 Unknown Unknown Unknown
>> castepexe_mpi.exe 0000000000412D0A Unknown Unknown Unknown
>> forrtl: error (78): process killed (SIGTERM)
>> Image PC Routine Line
>> Source
>> libpthread.so.0 00002AC2DC492C00 Unknown Unknown Unknown
>> libpthread.so.0 00002AC2DC491900 Unknown Unknown Unknown
>> libibverbs.so 00002AC2DF1B13A4 Unknown Unknown Unknown
>> libibverbs.so 00002AC2DF1B206A Unknown Unknown Unknown
>> libmpi.so.1 00002AC2DBE16FA7 Unknown Unknown Unknown
>> libmpi.so.1 00002AC2DBE10E36 Unknown Unknown Unknown
>> libmpi.so.1 00002AC2DBDFDF0C Unknown Unknown Unknown
>> libmpi.so.1 00002AC2DBDD8E1D Unknown Unknown Unknown
>> libmpi.so.1 00002AC2DBDD7999 Unknown Unknown Unknown
>> libmpi.so.1 00002AC2DBE4B03C Unknown Unknown Unknown
>> libmpi.so.1 00002AC2DBE573AB Unknown Unknown Unknown
>> castepexe_mpi.exe 00000000004D6539 Unknown Unknown Unknown
>> castepexe_mpi.exe 0000000000416404 Unknown Unknown Unknown
>> castepexe_mpi.exe 0000000000412DDC Unknown Unknown Unknown
>> libc.so.6 00002AC2DC5BB184 Unknown Unknown Unknown
>> castepexe_mpi.exe 0000000000412D0A Unknown Unknown Unknown
>> forrtl: error (78): process killed (SIGTERM)
>> Image PC Routine Line
>> Source
>> plugin002_004_002 00002B195294BA80 Unknown Unknown Unknown
>> libmpi.so.1 00002B1951975108 Unknown Unknown Unknown
>> libmpi.so.1 00002B19518D698E Unknown Unknown Unknown
>> libmpi.so.1 00002B195194A03C Unknown Unknown Unknown
>> libmpi.so.1 00002B19519563AB Unknown Unknown Unknown
>> castepexe_mpi.exe 00000000004D6539 Unknown Unknown Unknown
>> castepexe_mpi.exe 0000000000416404 Unknown Unknown Unknown
>> castepexe_mpi.exe 0000000000412DDC Unknown Unknown Unknown
>> libc.so.6 00002B19520BA184 Unknown Unknown Unknown
>> castepexe_mpi.exe 0000000000412D0A Unknown Unknown Unknown
>> mpirun: Broken pipe
>> -----------------
>>
>> Could you please give me some hints? Thanks in advance.
>>
>> Regards
> _______________________________________________
> torqueusers mailing list
> torqueusers at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torqueusers
More information about the torqueusers
mailing list