[torqueusers] Compilation options for geometry-requests and numa-systems

Gus Correa gus at ldeo.columbia.edu
Mon Sep 16 13:25:26 MDT 2013

On 09/16/2013 02:27 PM, James A. Peltier wrote:
> I've been meaning to follow up with regards to compilation
> options of cpuset related stuff.
> If I configure with
> --enable-cpuset --enable-geometry-requests --enable-numa-support
> will I then be able to configure individual nodes to be
> NUMA nodes by creating a mom.layout file or is this going
> to cause all sorts of weirdness on my nodes?

Hi James

For what it is worth,
in my experience, limited to a few builds of Torque 4.X.Y.Z
with the same options you mention above, --with-numa-support
did notplay nice with Maui.
It would not work with pbs_sched either.
[I can't say anything about Moab, I don't have it.]

With Maui, simple MPI jobs (using OpenMPI build with
the corresponding Torque support)
would mislocate the MPI processes and hang.
Pbs_sched would not even start the jobs up.

We have multi-core AMD Opteron processors,
which are presumably bona fide NUMA systems,
and are recognized as such by hwloc and lstopo.
So, I'd love to split my physical nodes into numa-nodes,
to achieve finer job granularity (which I guess is also your goal).

However, after several failed attempts,
I gave up building Torque with --enable-numa-support,
and retreated to
--enable-cpuset and --enable-geometry-requests

It would be nice to hear success stories/experiences/howtos
with --enable-numa-support from other Torque+Maui users, though.

My two cents,
Gus Correa

More information about the torqueusers mailing list