[Mauiusers] Re: [torqueusers] Maui?torque problems with NodeSet

Bas van der Vlies basv at sara.nl
Thu Dec 23 13:17:17 MST 2004


Bas van der Vlies wrote:
> Bas van der Vlies wrote:
> 
>> We have 256 node cluster and 4 switches with 64 nodes connected to it. 
>> In our nodes file of torque:
>>  gb-r1n1 np=2 switchA
>>  ...
>>  gb-r4n7 np=2 switchB
>>
>>  gb-r4n8 np=2 switchB
>>  ....
>>  gb-r7n14 np=2 switchB
>>
>> --- Maui config ---
>>
>> NODESETPOLICY           ONEOF
>> NODESETATTRIBUTE        FEATURE
>> NODESETDELAY            10
>> NODESETLIST             switchA switchB switchC switchD
>>
>> -------------------
>>
>> I only want to run jobs that are restricted to nodes that belong to 
>> one switch (therefore NODESETDELAY ).
>>
>> When i run an job with: qsub -I -lnodes=40
>>  It is scheduled to switchD
>>
>> Now i start another qsub -I -lnodes=40:
>>    scheduled to switchC
>>
>> The third one waits for ever and ever. If one of the first two jobs is 
>> completed then this jobs run. I expected that it would run on switchA 
>> or switchB. Is this an bug?
>>
>> I can acomplish it with the following command:
>>     qsub -I -lnodes=40:switchA
>>
>> Can this also be done automatically?
>>
>>         Regards
>>
>>
>> torque 1.1.0p4
>> maui 3.2.6p10
>>
>>
>>
> 
> I have also tried:
> qsub -I -lnodes=40 -l walltime=10:00:00 -W  \
>    x=\"NODESET=ONEOF:FEATURE:switchA:switchB:switchC:switchD\"
>     
> With this qsub command i only get two jobs running on switchD and 
> switchC and third one is waiting for ever and ever.
> 
>         Regards and Thanks for any help

I think i have found the problem. Each nodeset has 64 nodes. When i 
submit jobs that require 40 nodes. It runs on:
   switchD
   switchC
   .... hangs for ever. This is because on switchD there are 24 nodes
        free and als on switchC. This means 48 nodes so the job could
        run, but the "NODESETDELAY" prevents this.


To prove this i submit jobs that require 50 nodes and this runs as expected:
  - switchA
  - switchB
  - switchC
  - switchD

I have also upgraded maui to 3.2.6p11 and that have the same problem ;-(


-- 
--
********************************************************************
*                                                                  *
*  Bas van der Vlies                     e-mail: basv at sara.nl      *
*  SARA - Academic Computing Services    phone:  +31 20 592 8012   *
*  Kruislaan 415                         fax:    +31 20 6683167    *
*  1098 SJ Amsterdam                                               *
*                                                                  *
********************************************************************


More information about the mauiusers mailing list