gus at ldeo.columbia.edu
Wed Nov 30 12:46:19 MST 2011
For what it is worth, we use OpenMPI with Torque (TM) support,
where Torque is configured with cpusets.
As far as I know, Torque doesn't bind processes to CPUs.
Hence, we use the OpenMPI '-mca mpi_paffinity_alone 1 ' flag in the
mpiexec command to do that. I think '-bind-to-core' does pretty much the same,
see mpiexec man page.
Whether this speeds up or not the code may depend on the code itself, I guess.
Also, I guess it may get tricky if you are sharing the node among several jobs,
particularly if processes from different jobs share the different CPU sockets.
[We use single job node policy in Maui.]
Torque has also a configuration feature '--enable-geometry-requests' that *may*
be used to address this issue.
On Nov 30, 2011, at 2:14 PM, Martin Siegert wrote:
> we just recently started using cpusets and I do not have much experience
> with them. However, by now I noticed several times that MPI jobs
> (openmpi with TM) slow down dramatically: apparently two processes
> are using the same core (i.e., both only get 50% cpu usage) even though
> the number of cores in the cpuset equals the number of processes
> of the mpi job on the particular node.
> top - 11:05:24 up 42 days, 22:43, 2 users, load average: 6.99, 6.93, 6.68
> Tasks: 468 total, 8 running, 460 sleeping, 0 stopped, 0 zombie
> Cpu(s): 24.9%us, 0.2%sy, 0.0%ni, 74.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Mem: 24675188k total, 12099684k used, 12575504k free, 69968k buffers
> Swap: 16777208k total, 29932k used, 16747276k free, 9946292k cached
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 3717 user1 25 0 183m 91m 14m R 100.0 0.4 15:43.62 Clark
> 4526 user2 25 0 109m 36m 3088 R 100.0 0.2 2:02.43 mdrun
> 15863 user3 25 0 459m 163m 15m R 100.0 0.7 711:26.30 wrfm_arw.exe
> 15864 user3 25 0 452m 156m 15m R 100.0 0.6 688:28.80 wrfm_arw.exe
> 4562 user2 25 0 109m 36m 3088 R 99.7 0.2 0:23.02 mdrun
> 15861 user3 25 0 462m 165m 15m R 50.2 0.7 510:02.12 wrfm_arw.exe
> 15862 user3 25 0 465m 169m 15m R 49.9 0.7 446:21.37 wrfm_arw.exe
> root at b311:~> cat /proc/15861/cpuset
> root at b311:~> cat /proc/15862/cpuset
> (same for 15863, 15864) and
> root at b311:~> ls /dev/cpuset//torque/4913985.b0
> 68 cpu_exclusive memory_pressure notify_on_release
> 69 cpus memory_spread_page sched_relax_domain_level
> 70 mem_exclusive memory_spread_slab tasks
> 71 memory_migrate mems
> root at b311:~> cat /dev/cpuset/torque/4913985.b0/cpus
> Do processes within a cpuset get bound to a particular cpu?
> If yes, how do I find out which one?
> Anyway, if you have na idea what could be causing this and how to
> solve this problem, please let me know.
> Martin Siegert
> Simon Fraser University
> torqueusers mailing list
> torqueusers at supercluster.org
More information about the torqueusers