[torqueusers] cpusets

[Gustavo Correa]

Hi Martin

For what it is worth, we use  OpenMPI with Torque (TM) support,
where Torque is configured with cpusets.
As far as I know, Torque doesn't bind processes to CPUs.
Hence, we use the OpenMPI  '-mca mpi_paffinity_alone 1 ' flag in the
mpiexec command to do that.  I think '-bind-to-core' does pretty much the same,
see mpiexec man page.
Whether this speeds up or not the code may depend on the code itself, I guess.
Also, I guess it may get tricky if you are sharing the node among several jobs,
particularly if processes from different jobs share the different CPU sockets.
[We use single job node policy in Maui.]
Torque has also a configuration feature '--enable-geometry-requests' that *may* 
be used to address this issue.

Gus Correa

On Nov 30, 2011, at 2:14 PM, Martin Siegert wrote:

> Hi,
> 
> we just recently started using cpusets and I do not have much experience
> with them. However, by now I noticed several times that MPI jobs
> (openmpi with TM) slow down dramatically: apparently two processes
> are using the same core (i.e., both only get 50% cpu usage) even though
> the number of cores in the cpuset equals the number of processes
> of the mpi job on the particular node.
> 
> E.g.,
> 
> top - 11:05:24 up 42 days, 22:43,  2 users,  load average: 6.99, 6.93, 6.68
> Tasks: 468 total,   8 running, 460 sleeping,   0 stopped,   0 zombie
> Cpu(s): 24.9%us,  0.2%sy,  0.0%ni, 74.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
> Mem:  24675188k total, 12099684k used, 12575504k free,    69968k buffers
> Swap: 16777208k total,    29932k used, 16747276k free,  9946292k cached
> 
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 3717 user1     25   0  183m  91m  14m R 100.0  0.4  15:43.62 Clark
> 4526 user2     25   0  109m  36m 3088 R 100.0  0.2   2:02.43 mdrun
> 15863 user3     25   0  459m 163m  15m R 100.0  0.7 711:26.30 wrfm_arw.exe
> 15864 user3     25   0  452m 156m  15m R 100.0  0.6 688:28.80 wrfm_arw.exe
> 4562 user2     25   0  109m  36m 3088 R 99.7  0.2   0:23.02 mdrun
> 15861 user3     25   0  462m 165m  15m R 50.2  0.7 510:02.12 wrfm_arw.exe
> 15862 user3     25   0  465m 169m  15m R 49.9  0.7 446:21.37 wrfm_arw.exe
> 
> root at b311:~> cat /proc/15861/cpuset 
> /torque/4913985.b0
> root at b311:~> cat /proc/15862/cpuset 
> /torque/4913985.b0
> 
> (same for 15863, 15864) and
> 
> root at b311:~> ls /dev/cpuset//torque/4913985.b0
> 68  cpu_exclusive   memory_pressure     notify_on_release
> 69  cpus            memory_spread_page  sched_relax_domain_level
> 70  mem_exclusive   memory_spread_slab  tasks
> 71  memory_migrate  mems
> root at b311:~> cat /dev/cpuset/torque/4913985.b0/cpus
> 0-1,4,8
> 
> Do processes within a cpuset get bound to a particular cpu?
> If yes, how do I find out which one?
> 
> Anyway, if you have na idea what could be causing this and how to
> solve this problem, please let me know.
> 
> Thanks!
> 
> Cheers,
> Martin
> 
> -- 
> Martin Siegert
> Simon Fraser University
> _______________________________________________
> torqueusers mailing list
> torqueusers at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torqueusers