[torqueusers] procs= not working as documented (or understood?)
Lance Westerhoff
lance at quantumbioinc.com
Thu Feb 23 10:26:24 MST 2012
Hi All-
Ok, I think I have it. The problem was the setting (or lack thereof) of the server resources_available.nodect variable.
I set it to the following, and things seem to be working as expected:
set server resources_available.nodect=144
(where 144 is the number of processors in my cluster)
I have also commented out the following line in maui.cfg.
# JOBNODEMATCHPOLICY EXACTNODE
Thanks to everyone for their help. Hopefully this problem is officially behind us!
-Lance
On Feb 23, 2012, at 12:05 PM, Lance Westerhoff wrote:
>
> Hi Gareth-
>
> I've tried with and without the JOBNODEMATCHPOLICY set, and still no luck.
>
> Whenever I try the following, I get the following error. That is even though I have 144 processors in the cluster. So somehow torque/maui is seeing my nodes as collections of 8 processors.
>
> $ qsub -I -lnodes=46
> qsub: Job exceeds queue resource limits MSG=cannot locate feasible nodes
>
> I found that someone else that has set up their user documentation to show precisely the setting that I want, so it sounds like it can be done. Specifically, as noted on the following link, "#PBS -l nodes=1 asks PBS for one CPU. This means that when your jobs starts you will have exclusive access to one CPU. But if you want something like 4 nodes each with exactly 2 CPUs (total of 8 CPUs), then you would use something like -l nodes=4:ppn=2. Instead, if you just want any 8 CPUs in the cluster, you would request like just -l nodes=8." http://bmi.cchmc.org/resources/software/torque/examples
>
> But I can't find a lot in the email archive to help. Part of the problem is finding the right terms to look under though, so I'll keep searching.
>
> Thanks for any further information that can be provided.
>
> -Lance
>
>
> On Feb 14, 2012, at 6:27 PM, <Gareth.Williams at csiro.au> <Gareth.Williams at csiro.au> wrote:
>
>>> -----Original Message-----
>>> From: Lance Westerhoff [mailto:lance at quantumbioinc.com]
>>> Sent: Wednesday, 15 February 2012 5:12 AM
>>> To: Torque Users Mailing List
>>> Subject: Re: [torqueusers] procs= not working as documented (or
>>> understood?)
>>>
>>>
>>> Hello All-
>>>
>>> We're still having trouble with this feature, and we are starting to
>>> shop around for a torque/maui replacement in order to be able to use
>>> it. Before we do that however, I wanted to see if anyone has any
>>> thoughts on how to address the problem within torque/maui. Perhaps I
>>> simply don't understand the feature. The versions of torque and maui we
>>> are using are.
>>>
>>> torque-3.0.2
>>> maui-3.2.6p21
>>>
>>> Yes, we have tried newer versions of maui, but then the option doesn't
>>> work at all.
>>>
>>> Here is the scenario (I also included the conversation from November
>>> below for more information).
>>>
>>> Conceptually, our software is almost infinitely scalable in the sense
>>> that there is very little overhead associated with interprocess
>>> communication. Therefore, we do not require that all of the processes
>>> reside on a small number of nodes. In fact, we can stretch the
>>> processors to any and all nodes in the cluster with ~zero loss in
>>> performance. So we can literally have one node that has a single
>>> process running and another node that has 8 processes running. Since we
>>> have that level of scalability, we don't want to have to lock ourselves
>>> into having to request resources using the "nodes=X:ppn=Y" style since
>>> this style requires that nodes open up or drain in order to use them.
>>> Since our users have a big mixture of single and multi-processor jobs,
>>> waiting for node drain can really waste a lot of resources.
>>>
>>> I saw the "procs=#" the Requesting Resources table (see
>>> http://www.clusterresources.com/torquedocs/2.1jobsubmission.shtml#resou
>>> rces for more). It *appears* that this option should be able to allow
>>> the user to request simply X*Y processors and the scheduler should be
>>> able to schedule them any way it can fit. So using the following #PBS
>>> note, we should be able to request 40 processors:
>>>
>>> #PBS -l procs=40
>>>
>>> Instead, we see that the scheduler seems to take this information, read
>>> it, and basically disregard it. The reason I know it reads it is
>>> because if I ask for say 40 processors and 40 processors are available
>>> in the cluster, it works as expected and all is right with the world.
>>> Where it gets a bit more choppy is when I ask for 40 processors and
>>> only 1 processor is available. The job doesn't wait in the queue for
>>> the remaining 39 processors to open up, and instead PBS simply just
>>> starts the job on that processor. I can't see how that is anything but
>>> a bug. If the user is asking for 40 processors, why isn't the scheduler
>>> waiting for all 40 processors to open up?
>>>
>>> I'll also post this to the maui list so I apologize if you receive it
>>> twice. I'm just not sure if this is a problem with torque, maui, or
>>> both. If answering this question will require additional information,
>>> please ask. We are at our wits end here.
>>>
>>> Thanks!
>>>
>>> -Lance
>>
>> Hi Lance,
>>
>> It is more-or-less equivalent to request -l nodes=40 and -l procs=40 _if_
>> you are using maui _and_ you don't set JOBNODEMATCHPOLICY to EXACTNODE
>>
>> You may need to 'fake' having a large number of nodes to make this work.
>>
>> There are old mailing list items describing such a setup and how to 'fake'
>> the number of nodes. We've never actually done this at our site so I'm
>> unsure on details. I don't like it :-) but it may suit/help you.
>>
>> Gareth
>>
>>
>>>
>>>
>>>
>>>
>>> On Nov 18, 2011, at 11:12 AM, Lance Westerhoff wrote:
>>>
>>>>
>>>> Hi Steve-
>>>>
>>>> Here you go. Here is the top few lines of the job script. I have then
>>> provided the output you requested long with the maui.cfg. If you need
>>> anything further, certainly please let me know.
>>>>
>>>> Thanks for your help!
>>>>
>>>> ===============
>>>>
>>>> + head job.pbs
>>>>
>>>> #!/bin/bash
>>>> #PBS -S /bin/bash
>>>> #PBS -l procs=100
>>>> #PBS -l pmem=700mb
>>>> #PBS -l walltime=744:00:00
>>>> #PBS -j oe
>>>> #PBS -q batch
>>>>
>>>> Report run on Fri Nov 18 10:49:38 EST 2011
>>>> + pbsnodes --version
>>>> version: 3.0.2
>>>> + diagnose --version
>>>> maui client version 3.2.6p21
>>>> + checkjob 371010
>>>>
>>>>
>>>> checking job 371010
>>>>
>>>> State: Running
>>>> Creds: user:josh group:games class:batch qos:DEFAULT
>>>> WallTime: 00:02:35 of 31:00:00:00
>>>> SubmitTime: Fri Nov 18 10:46:33
>>>> (Time Queued Total: 00:00:01 Eligible: 00:00:01)
>>>>
>>>> StartTime: Fri Nov 18 10:46:34
>>>> Total Tasks: 1
>>>>
>>>> Req[0] TaskCount: 26 Partition: DEFAULT
>>>> Network: [NONE] Memory >= 700M Disk >= 0 Swap >= 0
>>>> Opsys: [NONE] Arch: [NONE] Features: [NONE]
>>>> Dedicated Resources Per Task: PROCS: 1 MEM: 700M
>>>> NodeCount: 10
>>>> Allocated Nodes:
>>>> [compute-0-17:7][compute-0-10:4][compute-0-3:2][compute-0-5:3]
>>>> [compute-0-6:1][compute-0-7:2][compute-0-9:1][compute-0-12:2]
>>>> [compute-0-13:2][compute-0-14:2]
>>>>
>>>>
>>>> IWD: [NONE] Executable: [NONE]
>>>> Bypass: 0 StartCount: 1
>>>> PartitionMask: [ALL]
>>>> Flags: RESTARTABLE
>>>>
>>>> Reservation '371010' (-00:02:09 -> 30:23:57:51 Duration:
>>> 31:00:00:00)
>>>> PE: 26.00 StartPriority: 4716
>>>>
>>>> + cat /opt/maui/maui.cfg | grep -v "#" | grep "^[A-Z]"
>>>> SERVERHOST gondor
>>>> ADMIN1 maui root
>>>> ADMIN3 ALL
>>>> RMCFG[base] TYPE=PBS
>>>> AMCFG[bank] TYPE=NONE
>>>> RMPOLLINTERVAL 00:01:00
>>>> SERVERPORT 42559
>>>> SERVERMODE NORMAL
>>>> LOGFILE maui.log
>>>> LOGFILEMAXSIZE 10000000
>>>> LOGLEVEL 3
>>>> QUEUETIMEWEIGHT 1
>>>> FSPOLICY DEDICATEDPS
>>>> FSDEPTH 7
>>>> FSINTERVAL 86400
>>>> FSDECAY 0.50
>>>> FSWEIGHT 200
>>>> FSUSERWEIGHT 1
>>>> FSGROUPWEIGHT 1000
>>>> FSQOSWEIGHT 1000
>>>> FSACCOUNTWEIGHT 1
>>>> FSCLASSWEIGHT 1000
>>>> USERWEIGHT 4
>>>> BACKFILLPOLICY FIRSTFIT
>>>> RESERVATIONPOLICY CURRENTHIGHEST
>>>> NODEALLOCATIONPOLICY MINRESOURCE
>>>> RESERVATIONDEPTH 8
>>>> MAXJOBPERUSERPOLICY OFF
>>>> MAXJOBPERUSERCOUNT 8
>>>> MAXPROCPERUSERPOLICY OFF
>>>> MAXPROCPERUSERCOUNT 256
>>>> MAXPROCSECONDPERUSERPOLICY OFF
>>>> MAXPROCSECONDPERUSERCOUNT 36864000
>>>> MAXJOBQUEUEDPERUSERPOLICY OFF
>>>> MAXJOBQUEUEDPERUSERCOUNT 2
>>>> JOBNODEMATCHPOLICY EXACTNODE
>>>> NODEACCESSPOLICY SHARED
>>>> JOBMAXOVERRUN 99:00:00:00
>>>> DEFERCOUNT 8192
>>>> DEFERTIME 0
>>>> CLASSCFG[developer] FSTARGET=40.00+
>>>> CLASSCFG[lowprio] PRIORITY=-1000
>>>> SRCFG[developer] CLASSLIST=developer
>>>> SRCFG[developer] ACCESS=dedicated
>>>> SRCFG[developer] DAYS=Mon,Tue,Wed,Thu,Fri
>>>> SRCFG[developer] STARTTIME=08:00:00
>>>> SRCFG[developer] ENDTIME=18:00:00
>>>> SRCFG[developer] TIMELIMIT=2:00:00
>>>> SRCFG[developer] RESOURCES=PROCS(8)
>>>> USERCFG[DEFAULT] FSTARGET=100.0
>>>>
>>>> ===============
>>>>
>>>> -Lance
>>>>
>>>>
>>>> On Nov 18, 2011, at 9:47 AM, Steve Crusan wrote:
>>>>
>>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>>> Hash: SHA1
>>>>>
>>>>>
>>>>> On Nov 18, 2011, at 9:33 AM, Lance Westerhoff wrote:
>>>>>
>>>>>> The request that is placed is for procs=60. Both torque and maui
>>> see that there are only 53 processors available and instead of letting
>>> the job sit in the queue and wait for all 60 processors to become
>>> available, it goes ahead and runs the job with what's available. Now if
>>> the user could ask for procs=[50-60] where 50 is the minimum number of
>>> processors to provide and 60 is the maximum, this would be a feature.
>>> But as it stands, if the user asks for 60 processors and ends up with 2
>>> processors, the job just won't scale properly and he may as well kill
>>> it (when it shouldn't have run anyway).
>>>>>
>>>>> Hi Lance,
>>>>>
>>>>> Can you post the output of checkjob <jobid> of an incorrectly
>>> running job. Let's take a look at what Maui thinks the job is asking
>>> for.
>>>>>
>>>>> Might as well add your maui.cfg file also.
>>>>>
>>>>> I've found in the past that procs= is troublesome...
>>>>>
>>>>>>
>>>>>> I'm actually beginning to think the problem may be related to maui.
>>> Perhaps I'll post this same question to the maui list and see what
>>> comes back.
>>>>>>
>>>>>> This problem is infuriating though since without the functionality
>>> working as it should, using procs=X in torque/maui makes torque/maui
>>> work more like a submission and run system (not a queuing system).
>>>>>
>>>>> Agreed. HPC cluster job management is normally be set it and forget
>>> it. Anything else other than maintenance/break fixes/new features would
>>> be ridiculously time consuming.
>>>>>
>>>>>>
>>>>>> -Lance
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Message: 3
>>>>>>> Date: Thu, 17 Nov 2011 17:29:17 -0800
>>>>>>> From: "Brock Palen" <brockp at umich.edu>
>>>>>>> Subject: Re: [torqueusers] procs= not working as documented
>>>>>>> To: "Torque Users Mailing List" <torqueusers at supercluster.org>
>>>>>>> Message-ID:
>>> <20111118012930.C635E83A8026 at mail.adaptivecomputing.com>
>>>>>>> Content-Type: text/plain; charset="utf-8"
>>>>>>>
>>>>>>> Does maui only see one cpu or does mpiexec only see one cpu?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Brock Palen
>>>>>>> (734)936-1985
>>>>>>> brockp at umich.edu
>>>>>>> - Sent from my Palm Pre, please excuse typos
>>>>>>> On Nov 17, 2011 3:19 PM, Lance Westerhoff
>>> <lance at quantumbioinc.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hello All-
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> It appears that when running with the following specs, the procs=
>>> option does not actually work as expected.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ==========================================
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> #PBS -S /bin/bash
>>>>>>>
>>>>>>> #PBS -l procs=60
>>>>>>>
>>>>>>> #PBS -l pmem=700mb
>>>>>>>
>>>>>>> #PBS -l walltime=744:00:00
>>>>>>>
>>>>>>> #PBS -j oe
>>>>>>>
>>>>>>> #PBS -q batch
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> torque version: tried 3.0.2. in v2.5.4, I think the procs option
>>> worked as documented
>>>>>>>
>>>>>>> maui version: 3.2.6p21 (also tried maui 3.3.1 but it is a complete
>>> fail in terms of the procs option and it only asks for a single CPU)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ==========================================
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> If there are fewer then 60 processors available in the cluster (in
>>> this case there were 53 available) the job will go in an take whatever
>>> is left instead of waiting for all 60 processors to free up. Any
>>> thoughts as to why this might be happening? Sometimes it doesn't really
>>> matter and 53 would be almost as good as 60, however if only 2
>>> processors are available and the user asks for 60, I would hate for him
>>> to go in.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thank you for your time!
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -Lance
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> torqueusers mailing list
>>>>>> torqueusers at supercluster.org
>>>>>> http://www.supercluster.org/mailman/listinfo/torqueusers
>>>>>
>>>>> ----------------------
>>>>> Steve Crusan
>>>>> System Administrator
>>>>> Center for Research Computing
>>>>> University of Rochester
>>>>> https://www.crc.rochester.edu/
>>>>>
>>>>>
>>>>> -----BEGIN PGP SIGNATURE-----
>>>>> Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
>>>>> Comment: GPGTools - http://gpgtools.org
>>>>>
>>>>> iQEcBAEBAgAGBQJOxnAEAAoJENS19LGOpgqK2CEH/Ry+THjmhxdTzcIZ5d5YYCP/
>>>>> bYQY2QthvbaEkUhh+q26m2EWrmPGHRgW9zXOx/fRBE2ejZE+EycpRLMdWDTOxn28
>>>>> cK1qs+ITaiOevNbxufd7pt/P5hhvafQgsDtuy8RPGokgqSuRBEH9i8DZAFfIASQZ
>>>>> tQ9YE5MSqEfaoTSwOVP2PXJCgEJh2ZU5GHO2UvmxF4SX4+7HePUgQYzmzIBu2cW8
>>>>> JeeIpaf2AuNIvXjG3ZNA3FjHWQEZefiZhRTQxeE1PHuQCLWPnfTwz0nzquCHZBJv
>>>>> Ufc1wOGanDi+LosRldVIUgAyHGcAcOvZzFnxlfNrYa2xfJSCyuC86YB4XNfpO1c=
>>>>> =AGW7
>>>>> -----END PGP SIGNATURE-----
>>>>> _______________________________________________
>>>>> torqueusers mailing list
>>>>> torqueusers at supercluster.org
>>>>> http://www.supercluster.org/mailman/listinfo/torqueusers
>>>>
>>>
>>
>> _______________________________________________
>> torqueusers mailing list
>> torqueusers at supercluster.org
>> http://www.supercluster.org/mailman/listinfo/torqueusers
>
More information about the torqueusers
mailing list