[torqueusers] Specifying nodes which can be used in array job

Rainer M Krug R.M.Krug at gmail.com
Thu Feb 16 02:35:49 MST 2012 

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 15/02/12 18:42, Gustavo Correa wrote:
> Rainer:  Somehow I receive all your postings to the Torqueusers
> list in duplicate, one from R.M.Krug [capitals] another from
> r.m.krug [lowercase] at gmail dot com. I don't know why, and it is
> not a big deal either. Did you register twice perhaps?

not that I know of - but it might be that I hit reply-to-all, which
sends to the mailing list and you directly?

This one is only via mailinglist - do you also get it twice?

Rainer

> 
> Gus Correa
> 
> 
> On Feb 15, 2012, at 12:35 PM, Gustavo Correa wrote:
> 
>> 
>> On Feb 15, 2012, at 11:38 AM, Rainer M Krug wrote:
>> 
> OK - the feature is added, but I don't get it to only use the
> nodes with this feature set.
> 
> I have the following simple script named test.sub:
> 
> 
> #PBS -j oe #PBS -m abe #PBS -M Rainer at krugs.de #PBS -V #PBS -t
> 1-10 #PBS -l feature='Rnodeff' echo "---------------------"
> 
> 
> and I submit it via
> 
> qsub test.sub
> 
> and it always runs, whatever I put into the feature line, even if
> the feature property "Rnodeff" does not exist. It always uses all
> nodes, even when I use "Rnode" which is set only for a subset of
> nodes.
> 
> 
> I even setup torque on my desktop and set the features to try it.
> 
> 
>>> 
>>> There is one way to do what I guess you want to do [i.e. launch
>>> jobs on nodes with different properties, right?] by setting up
>>> specific queues for each node property. You may need maui or
>>> moab for this, I am not sure if it works with pbs_sched.
>>> 
>>> Say, create a queue qRnodeff the usual way. Then add something
>>> like this:
>>> 
>>> qmgr -c 'set queue qRnodeff resources_default.neednodes =
>>> Rnodeff'
>>> 
>>> where Rnodeff is a property of some of your nodes listed in the
>>> $Torque/pbs_server/nodes file.
>>> 
>>> On your maui.cfg add this line: ENABLEMULTIREQJOBS   TRUE
>>> 
>>> Restart pbs_server and maui.
>>> 
>>> On your PBS job script just direct the job to the appropriate
>>> queue: #PBS -q qRnodeff
>>> 
>>> no need for "#PBS -l Rnodeff"
>>> 
>>> Anyway, for these non-default configurations, it is worth
>>> reading the Torque and Maui Admin guides: 
>>> http://www.adaptivecomputing.com/resources/docs/
>>> 
>>> I hope this helps.
>>> 
>>> Gus Correa
>>> 
>>> 
> Any ideas?
> 
> Rainer
> 
> qsub On 13/02/12 09:37, Rainer M Krug wrote:
>>>>> Thanks a lot - this definitely helps. I will get in contact
>>>>> with our admin to add the features to the nodes.
>>>>> 
>>>>> Cheers,
>>>>> 
>>>>> Rainer
>>>>> 
>>>>> On 12/02/12 16:53, Sreedhar Manchu wrote:
>>>>>> Hi Rainer,
>>>>> 
>>>>>> Like Ken wrote it is possible with feature property. I
>>>>>> use this feature heavily to place jobs on specific
>>>>>> nodes.
>>>>> 
>>>>>> To add feature to nodes
>>>>> 
>>>>>> for i in {0..5}; do qmgr -c "set node node0$i properties
>>>>>> += arrays"; done
>>>>> 
>>>>>> Here feature is arrays. You can replace that with
>>>>>> whatever you like.
>>>>> 
>>>>>> Once you've done this you can get array jobs placed on
>>>>>> these nodes by requesting this feature in qsub such as
>>>>> 
>>>>>>>>> qsub the_script.sub -t 1-10 -l feature='arrays'
>>>>> 
>>>>>> This would put your jobs on the nodes that have property
>>>>>> arrays. In this case the nodes are 0 to 5.
>>>>> 
>>>>>> In my case I wrote a qsub wrapper which goes through the
>>>>>> pbs scripts and command line and adds this feature line
>>>>>> such as #PBS -l feature=<feature name> to the script so
>>>>>> that they are placed on right nodes. This comes very
>>>>>> handy especially when you have nodes with diiferent
>>>>>> amounts of memory under the same queue.
>>>>> 
>>>>>> If your scheduler is moab you can do really cool stuff
>>>>>> using this feature property.
>>>>> 
>>>>>> Hope this helps.
>>>>> 
>>>>>> Sreedhar.
>>>>> 
>>>>> 
>>>>> 
>>>>>> On 10-Feb-2012, at 2:49 AM, Rainer M Krug
>>>>>> <R.M.Krug at gmail.com <mailto:R.M.Krug at gmail.com>> wrote:
>>>>> 
>>>>>> On 09/02/12 23:39, Ken Nielson wrote:
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> ----- Original Message -----
>>>>>>>>>> From: "Rainer M Krug" <R.M.Krug at gmail.com 
>>>>>>>>>> <mailto:R.M.Krug at gmail.com>> To: 
>>>>>>>>>> torqueusers at supercluster.org 
>>>>>>>>>> <mailto:torqueusers at supercluster.org> Sent:
>>>>>>>>>> Thursday, February 9, 2012 2:16:07 AM Subject:
>>>>>>>>>> [torqueusers] Specifying nodes which can be used
>>>>>>>>>> in array job
>>>>>>>>>> 
>>>>>>>>>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
>>>>>>>>>> 
>>>>>>>>>> Hi
>>>>>>>>>> 
>>>>>>>>>> assuming I have cluster of 10 nodes (node01, ... 
>>>>>>>>>> node10), of which I am not the administrator.
>>>>>>>>>> 
>>>>>>>>>> Some nodes are setup slightly different, so that
>>>>>>>>>> a certain job only runs on nodes node01 to
>>>>>>>>>> node05.
>>>>>>>>>> 
>>>>>>>>>> So I would like to submit an array job and
>>>>>>>>>> specify "only use the node01, node02, node03,
>>>>>>>>>> node04 or node05 to run the each individual
>>>>>>>>>> job".
>>>>>>>>>> 
>>>>>>>>>> How can I do that? I know that I can use -l to
>>>>>>>>>> specify resource requirements, but if I specify
>>>>>>>>>> nodes=..., *each* job will allocate *all* nodes
>>>>>>>>>> for the job, which is not what I want - each
>>>>>>>>>> individual job should run on one of the nodes.
>>>>>>>>>> 
>>>>>>>>>> so:
>>>>>>>>>> 
>>>>>>>>>> qsub the_script.sub -t 1-10
>>>>>>>>>> 
>>>>>>>>>> and how do I specify the nodes?
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> 
>>>>>>>>>> Rainer
>>>>>>>>> 
>>>>>>>>> Rainer,
>>>>>>>>> 
>>>>>>>>> Are there feature (properties) in the nodes files
>>>>>>>>> of those hosts which would allow you to specify a
>>>>>>>>> feature on the qsub line?
>>>>> 
>>>>>> No - unfortunately not.
>>>>> 
>>>>>>>>> 
>>>>>>>>> Ken
>>>>> 
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> torqueusers mailing list torqueusers at supercluster.org 
>>>>>>> <mailto:torqueusers at supercluster.org> 
>>>>>>> http://www.supercluster.org/mailman/listinfo/torqueusers
>>>>>
>>>>>
>>>>>>
>>>>>>> 
_______________________________________________ torqueusers
>>>>>> mailing list torqueusers at supercluster.org 
>>>>>> http://www.supercluster.org/mailman/listinfo/torqueusers
>>>>> 
>>> 
>>> _______________________________________________ torqueusers
>>> mailing list torqueusers at supercluster.org 
>>> http://www.supercluster.org/mailman/listinfo/torqueusers
>> 
>> _______________________________________________ torqueusers
>> mailing list torqueusers at supercluster.org 
>> http://www.supercluster.org/mailman/listinfo/torqueusers

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk88zfUACgkQoYgNqgF2egpEtACgiqRtn07Qmqhsyrbfig6HESuR
gwMAn2Uho2jAHn5SN/6IdHEErol7BHkJ
=B1Fu
-----END PGP SIGNATURE-----

More information about the torqueusers mailing list