[torqueusers] MATLAB and cpusets

Gus Correa gus at ldeo.columbia.edu
Thu Dec 16 10:28:13 MST 2010


Martin Thompson wrote:
> On Thu, 2010-12-16 at 09:04 +1100, Gus Correa wrote:
> 
>> Hi Martin
>>
>> Not sure if this will help, but there it goes anyway.
>>
>> In March I asked MathWorks/Matlab how to control the number of threads
>> used by Matlab.
>> Their software engineer was very forthcoming, but I ended up with the
>> impression that there is no way to have a consistent control of the 
>> number of active threads for all Matlab operations.
>> It seems to depend on which Matlab functions are being called,
>> and on different mechanisms that Matlab uses to control
>> the number of threads.
>>
>> For instance, (most) Linear Algebra functions can be
>> controlled by  MaxNumCompThreads, which you set
>> inside Matlab (or the Matlab script).
>> However, for the FFTs functions, for instance,
>> you either get one thread only or all the cores/cpus
>> in the node/computer.
>> In this case the behavior seems to be selected through 
>> -singleCompThread, a flag that you set when you launch Matlab.
>>
>> Given this lack of master control on the number of threads,
>> I asked the users to request a full node
>> to run Matlab, to launch Matlab in batch mode through a wrapper
>> with nodisplay, nojvm, no-nothing.
>> To my joy and relief nobody is currently using Matlab in the cluster anyway.
>>
>> I wonder if what you described is a side effect of these
>> multiple mechanisms that Matlab seems to use to control the
>> number of threads, and perhaps how they interact with the
>> resources that Torque makes available to each job.
> 
> The main reason why I have started experimenting with the cpuset support
> in Torque is that I hoped it would solve our original problem of MATLAB
> grabbing all available cores.  My interpretation of the advice we
> received from Mathworks about that original problem is that fine-grained
> control of threads is no longer possible.  You can either restrict
> MATLAB to one thread (-singleCompThread) or apply no restrictions and
> MATLAB will use as many cores as it can.  The maxNumCompThreads()
> function is deprecated.
> 
> Anyway, I have isolated the cpusets problem a little further.  It
> appears that MATLAB will only behave correctly when it is assigned the
> first N cores.  So if my job requests 4 cores and it is allocated cores
> 0-3 then all 4 cores are used in my matrix multiply test.  However, if
> the job is allocated cores 4-7 then only one core will be used, even if
> there is no other job running on that node.
> 
> I'll post the problem on a MATLAB forum and report back here if I make
> any progress.
> 
> Thanks
> 
> Martin
> 
Hi Martin, Chris

1, Yes, my conversation with Mathworks last March
led to the same conclusion you wrote above.
Either use -singleCompThread to compute on a single core, or Matlab
will take over all resources available (or, as it turns out from
what you reported, whatever the cpusets allow it to use).

Indeed they claim maxNumCompThreads() is deprecated, but it still
works in our Matlab 2009-whatever, although it seems to control only
part of the Linear Algebra functions, and certainly doesn't provide
any control of FFTs, for instance.

The Mathworks folks were nice, but didn't have
a solution for the problem.
They said they would put my request for a single and effective
control of number of threads and number of cores as a
"feature request for a future release" ... (I have to smile :) )
Maybe you could make a similar feature request to them.

2. However, regardless of -singleCompThread, and no matter what I do, 
Matlab always launches 5 extra threads,
besides the one-or-all-cores that one can presumably control via 
-singleCompThread.
These extra threads don't seem to participate in the computations (top
doesn't show much of CPU activity on them).
I don't know what they do, maybe memory management, as they do use RAM.


Did you use "top -H" ( "-H" to show all threads) to check this?

Do you see the extra threads there?

Which cores do these extra threads use?
(Here these extra threads migrate across all cores,
regardless of using/not using -singleCompThread)

**

Here is the silly but useful cpu- and memory-intensive
test that I used, while monitoring via "top -H"
in a separate window (top's "1" "u" and "f" "j" switches
help get a clean display):

qsub -I bla bla ...

(now on the node)
setenv OMP_NUM_THREADS 1

matlab --nojvm --nosplash [-singleCompThread]

 >> N= 2^14; (whatever size of square matrix that will fit your RAM)
 >> b=zeros(N);
 >> c=ifft2(fft2(b)):

**

I didn't pursue this matter further because nobody is really using
Matlab in our cluster.

I wonder if these extra threads are somehow tangled to your guess
that Matlab only behaves correctly if it gest the first N cores.


3. Sorry, I have Torque 2.3.6, not configures with --enable-cpusets.
Hence, on all my tests Matlab raises the hell on the cores, doing 
whatever it wants with its 5 extra threads (and taking all cores
for computation if you don't launch it with -singleCompThread).


4. Yes, I would be interested to read any new info you
get from Mathworks about this.  Thank you.


Cheers,
Gus Correa
---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------


More information about the torqueusers mailing list