[torqueusers] Specifying nodes which can be used in array job

Gustavo Correa gus at ldeo.columbia.edu
Wed Feb 15 10:42:10 MST 2012


Rainer:  Somehow I receive all your postings to the Torqueusers list in duplicate,
one from R.M.Krug [capitals] another from r.m.krug [lowercase] at gmail dot com.
I don't know why, and it is not a big deal either.
Did you register twice perhaps?

Gus Correa


On Feb 15, 2012, at 12:35 PM, Gustavo Correa wrote:

> 
> On Feb 15, 2012, at 11:38 AM, Rainer M Krug wrote:
> 
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>> 
>> OK - the feature is added, but I don't get it to only use the nodes
>> with this feature set.
>> 
>> I have the following simple script named test.sub:
>> 
>> 
>> #PBS -j oe
>> #PBS -m abe
>> #PBS -M Rainer at krugs.de
>> #PBS -V
>> #PBS -t 1-10
>> #PBS -l feature='Rnodeff'
>> echo "---------------------"
>> 
>> 
>> and I submit it via
>> 
>> qsub test.sub
>> 
>> and it always runs, whatever I put into the feature line, even if the
>> feature property "Rnodeff" does not exist. It always uses all nodes,
>> even when I use "Rnode" which is set only for a subset of nodes.
>> 
>> 
>> I even setup torque on my desktop and set the features to try it.
>> 
>> 
> 
> There is one way to do what I guess you want to do [i.e. launch jobs on nodes with different
> properties, right?] by setting up specific queues for each node property.
> You may need maui or moab for this, I am not sure if it works with pbs_sched.
> 
> Say, create a queue qRnodeff the usual way.
> Then add something like this:
> 
> qmgr -c 'set queue qRnodeff resources_default.neednodes = Rnodeff'
> 
> where Rnodeff is a property of some of your nodes listed in the $Torque/pbs_server/nodes file.
> 
> On your maui.cfg add this line:
> ENABLEMULTIREQJOBS   TRUE
> 
> Restart pbs_server and maui.
> 
> On your PBS job script just direct the job to the appropriate queue:
> #PBS -q qRnodeff
> 
> no need for "#PBS -l Rnodeff"
> 
> Anyway, for these non-default configurations, it is worth reading the 
> Torque and Maui Admin guides:
> http://www.adaptivecomputing.com/resources/docs/
> 
> I hope this helps.
> 
> Gus Correa
> 
> 
>> Any ideas?
>> 
>> Rainer
>> 
>> qsub
>> On 13/02/12 09:37, Rainer M Krug wrote:
>>> Thanks a lot - this definitely helps. I will get in contact with
>>> our admin to add the features to the nodes.
>>> 
>>> Cheers,
>>> 
>>> Rainer
>>> 
>>> On 12/02/12 16:53, Sreedhar Manchu wrote:
>>>> Hi Rainer,
>>> 
>>>> Like Ken wrote it is possible with feature property. I use this 
>>>> feature heavily to place jobs on specific nodes.
>>> 
>>>> To add feature to nodes
>>> 
>>>> for i in {0..5}; do qmgr -c "set node node0$i properties += 
>>>> arrays"; done
>>> 
>>>> Here feature is arrays. You can replace that with whatever you 
>>>> like.
>>> 
>>>> Once you've done this you can get array jobs placed on these
>>>> nodes by requesting this feature in qsub such as
>>> 
>>>>>>> qsub the_script.sub -t 1-10 -l feature='arrays'
>>> 
>>>> This would put your jobs on the nodes that have property arrays.
>>>> In this case the nodes are 0 to 5.
>>> 
>>>> In my case I wrote a qsub wrapper which goes through the pbs 
>>>> scripts and command line and adds this feature line such as #PBS
>>>> -l feature=<feature name> to the script so that they are placed
>>>> on right nodes. This comes very handy especially when you have
>>>> nodes with diiferent amounts of memory under the same queue.
>>> 
>>>> If your scheduler is moab you can do really cool stuff using this
>>>> feature property.
>>> 
>>>> Hope this helps.
>>> 
>>>> Sreedhar.
>>> 
>>> 
>>> 
>>>> On 10-Feb-2012, at 2:49 AM, Rainer M Krug <R.M.Krug at gmail.com 
>>>> <mailto:R.M.Krug at gmail.com>> wrote:
>>> 
>>>> On 09/02/12 23:39, Ken Nielson wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> ----- Original Message -----
>>>>>>>> From: "Rainer M Krug" <R.M.Krug at gmail.com 
>>>>>>>> <mailto:R.M.Krug at gmail.com>> To: 
>>>>>>>> torqueusers at supercluster.org 
>>>>>>>> <mailto:torqueusers at supercluster.org> Sent: Thursday, 
>>>>>>>> February 9, 2012 2:16:07 AM Subject: [torqueusers] 
>>>>>>>> Specifying nodes which can be used in array job
>>>>>>>> 
>>>>>>>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
>>>>>>>> 
>>>>>>>> Hi
>>>>>>>> 
>>>>>>>> assuming I have cluster of 10 nodes (node01, ...
>>>>>>>> node10), of which I am not the administrator.
>>>>>>>> 
>>>>>>>> Some nodes are setup slightly different, so that a
>>>>>>>> certain job only runs on nodes node01 to node05.
>>>>>>>> 
>>>>>>>> So I would like to submit an array job and specify "only 
>>>>>>>> use the node01, node02, node03, node04 or node05 to run
>>>>>>>> the each individual job".
>>>>>>>> 
>>>>>>>> How can I do that? I know that I can use -l to specify 
>>>>>>>> resource requirements, but if I specify nodes=...,
>>>>>>>> *each* job will allocate *all* nodes for the job, which
>>>>>>>> is not what I want - each individual job should run on
>>>>>>>> one of the nodes.
>>>>>>>> 
>>>>>>>> so:
>>>>>>>> 
>>>>>>>> qsub the_script.sub -t 1-10
>>>>>>>> 
>>>>>>>> and how do I specify the nodes?
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> 
>>>>>>>> Rainer
>>>>>>> 
>>>>>>> Rainer,
>>>>>>> 
>>>>>>> Are there feature (properties) in the nodes files of those 
>>>>>>> hosts which would allow you to specify a feature on the
>>>>>>> qsub line?
>>> 
>>>> No - unfortunately not.
>>> 
>>>>>>> 
>>>>>>> Ken
>>> 
>>>>> 
>>>>> _______________________________________________ torqueusers 
>>>>> mailing list torqueusers at supercluster.org 
>>>>> <mailto:torqueusers at supercluster.org> 
>>>>> http://www.supercluster.org/mailman/listinfo/torqueusers
>>> 
>>> 
>>>> _______________________________________________ torqueusers
>>>> mailing list torqueusers at supercluster.org 
>>>> http://www.supercluster.org/mailman/listinfo/torqueusers
>>> 
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1.4.11 (GNU/Linux)
>> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>> 
>> iEYEARECAAYFAk8734IACgkQoYgNqgF2egoY3QCfVqXzOLwjZ+DLqgVQdMU0eLjr
>> TVwAn0dQs8G0y+JTIc+lRtP/0oT7GCyo
>> =vpZF
>> -----END PGP SIGNATURE-----
>> 
>> _______________________________________________
>> torqueusers mailing list
>> torqueusers at supercluster.org
>> http://www.supercluster.org/mailman/listinfo/torqueusers
> 
> _______________________________________________
> torqueusers mailing list
> torqueusers at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torqueusers



More information about the torqueusers mailing list