[torqueusers] Specifying nodes which can be used in array job

Gustavo Correa gus at ldeo.columbia.edu
Thu Feb 16 08:39:26 MST 2012 

On Feb 16, 2012, at 4:35 AM, Rainer M Krug wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 15/02/12 18:42, Gustavo Correa wrote:
>> Rainer:  Somehow I receive all your postings to the Torqueusers
>> list in duplicate, one from R.M.Krug [capitals] another from
>> r.m.krug [lowercase] at gmail dot com. I don't know why, and it is
>> not a big deal either. Did you register twice perhaps?
> 
> not that I know of - but it might be that I hit reply-to-all, which
> sends to the mailing list and you directly?
> 
> This one is only via mailinglist - do you also get it twice?
> 
> Rainer
> 

I received it twice.  
Same thing with the capitals/lowercase email addresses.
It doesn't bother  me, just wondering if it goes twice to all list subscribers.

Gus Correa

>> 
>> Gus Correa
>> 
>> 
>> On Feb 15, 2012, at 12:35 PM, Gustavo Correa wrote:
>> 
>>> 
>>> On Feb 15, 2012, at 11:38 AM, Rainer M Krug wrote:
>>> 
>> OK - the feature is added, but I don't get it to only use the
>> nodes with this feature set.
>> 
>> I have the following simple script named test.sub:
>> 
>> 
>> #PBS -j oe #PBS -m abe #PBS -M Rainer at krugs.de #PBS -V #PBS -t
>> 1-10 #PBS -l feature='Rnodeff' echo "---------------------"
>> 
>> 
>> and I submit it via
>> 
>> qsub test.sub
>> 
>> and it always runs, whatever I put into the feature line, even if
>> the feature property "Rnodeff" does not exist. It always uses all
>> nodes, even when I use "Rnode" which is set only for a subset of
>> nodes.
>> 
>> 
>> I even setup torque on my desktop and set the features to try it.
>> 
>> 
>>>> 
>>>> There is one way to do what I guess you want to do [i.e. launch
>>>> jobs on nodes with different properties, right?] by setting up
>>>> specific queues for each node property. You may need maui or
>>>> moab for this, I am not sure if it works with pbs_sched.
>>>> 
>>>> Say, create a queue qRnodeff the usual way. Then add something
>>>> like this:
>>>> 
>>>> qmgr -c 'set queue qRnodeff resources_default.neednodes =
>>>> Rnodeff'
>>>> 
>>>> where Rnodeff is a property of some of your nodes listed in the
>>>> $Torque/pbs_server/nodes file.
>>>> 
>>>> On your maui.cfg add this line: ENABLEMULTIREQJOBS   TRUE
>>>> 
>>>> Restart pbs_server and maui.
>>>> 
>>>> On your PBS job script just direct the job to the appropriate
>>>> queue: #PBS -q qRnodeff
>>>> 
>>>> no need for "#PBS -l Rnodeff"
>>>> 
>>>> Anyway, for these non-default configurations, it is worth
>>>> reading the Torque and Maui Admin guides: 
>>>> http://www.adaptivecomputing.com/resources/docs/
>>>> 
>>>> I hope this helps.
>>>> 
>>>> Gus Correa
>>>> 
>>>> 
>> Any ideas?
>> 
>> Rainer
>> 
>> qsub On 13/02/12 09:37, Rainer M Krug wrote:
>>>>>> Thanks a lot - this definitely helps. I will get in contact
>>>>>> with our admin to add the features to the nodes.
>>>>>> 
>>>>>> Cheers,
>>>>>> 
>>>>>> Rainer
>>>>>> 
>>>>>> On 12/02/12 16:53, Sreedhar Manchu wrote:
>>>>>>> Hi Rainer,
>>>>>> 
>>>>>>> Like Ken wrote it is possible with feature property. I
>>>>>>> use this feature heavily to place jobs on specific
>>>>>>> nodes.
>>>>>> 
>>>>>>> To add feature to nodes
>>>>>> 
>>>>>>> for i in {0..5}; do qmgr -c "set node node0$i properties
>>>>>>> += arrays"; done
>>>>>> 
>>>>>>> Here feature is arrays. You can replace that with
>>>>>>> whatever you like.
>>>>>> 
>>>>>>> Once you've done this you can get array jobs placed on
>>>>>>> these nodes by requesting this feature in qsub such as
>>>>>> 
>>>>>>>>>> qsub the_script.sub -t 1-10 -l feature='arrays'
>>>>>> 
>>>>>>> This would put your jobs on the nodes that have property
>>>>>>> arrays. In this case the nodes are 0 to 5.
>>>>>> 
>>>>>>> In my case I wrote a qsub wrapper which goes through the
>>>>>>> pbs scripts and command line and adds this feature line
>>>>>>> such as #PBS -l feature=<feature name> to the script so
>>>>>>> that they are placed on right nodes. This comes very
>>>>>>> handy especially when you have nodes with diiferent
>>>>>>> amounts of memory under the same queue.
>>>>>> 
>>>>>>> If your scheduler is moab you can do really cool stuff
>>>>>>> using this feature property.
>>>>>> 
>>>>>>> Hope this helps.
>>>>>> 
>>>>>>> Sreedhar.
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> On 10-Feb-2012, at 2:49 AM, Rainer M Krug
>>>>>>> <R.M.Krug at gmail.com <mailto:R.M.Krug at gmail.com>> wrote:
>>>>>> 
>>>>>>> On 09/02/12 23:39, Ken Nielson wrote:
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>> From: "Rainer M Krug" <R.M.Krug at gmail.com 
>>>>>>>>>>> <mailto:R.M.Krug at gmail.com>> To: 
>>>>>>>>>>> torqueusers at supercluster.org 
>>>>>>>>>>> <mailto:torqueusers at supercluster.org> Sent:
>>>>>>>>>>> Thursday, February 9, 2012 2:16:07 AM Subject:
>>>>>>>>>>> [torqueusers] Specifying nodes which can be used
>>>>>>>>>>> in array job
>>>>>>>>>>> 
>>>>>>>>>>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
>>>>>>>>>>> 
>>>>>>>>>>> Hi
>>>>>>>>>>> 
>>>>>>>>>>> assuming I have cluster of 10 nodes (node01, ... 
>>>>>>>>>>> node10), of which I am not the administrator.
>>>>>>>>>>> 
>>>>>>>>>>> Some nodes are setup slightly different, so that
>>>>>>>>>>> a certain job only runs on nodes node01 to
>>>>>>>>>>> node05.
>>>>>>>>>>> 
>>>>>>>>>>> So I would like to submit an array job and
>>>>>>>>>>> specify "only use the node01, node02, node03,
>>>>>>>>>>> node04 or node05 to run the each individual
>>>>>>>>>>> job".
>>>>>>>>>>> 
>>>>>>>>>>> How can I do that? I know that I can use -l to
>>>>>>>>>>> specify resource requirements, but if I specify
>>>>>>>>>>> nodes=..., *each* job will allocate *all* nodes
>>>>>>>>>>> for the job, which is not what I want - each
>>>>>>>>>>> individual job should run on one of the nodes.
>>>>>>>>>>> 
>>>>>>>>>>> so:
>>>>>>>>>>> 
>>>>>>>>>>> qsub the_script.sub -t 1-10
>>>>>>>>>>> 
>>>>>>>>>>> and how do I specify the nodes?
>>>>>>>>>>> 
>>>>>>>>>>> Thanks,
>>>>>>>>>>> 
>>>>>>>>>>> Rainer
>>>>>>>>>> 
>>>>>>>>>> Rainer,
>>>>>>>>>> 
>>>>>>>>>> Are there feature (properties) in the nodes files
>>>>>>>>>> of those hosts which would allow you to specify a
>>>>>>>>>> feature on the qsub line?
>>>>>> 
>>>>>>> No - unfortunately not.
>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Ken
>>>>>> 
>>>>>>>> 
>>>>>>>> _______________________________________________
>>>>>>>> torqueusers mailing list torqueusers at supercluster.org 
>>>>>>>> <mailto:torqueusers at supercluster.org> 
>>>>>>>> http://www.supercluster.org/mailman/listinfo/torqueusers
>>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>>> 
> _______________________________________________ torqueusers
>>>>>>> mailing list torqueusers at supercluster.org 
>>>>>>> http://www.supercluster.org/mailman/listinfo/torqueusers
>>>>>> 
>>>> 
>>>> _______________________________________________ torqueusers
>>>> mailing list torqueusers at supercluster.org 
>>>> http://www.supercluster.org/mailman/listinfo/torqueusers
>>> 
>>> _______________________________________________ torqueusers
>>> mailing list torqueusers at supercluster.org 
>>> http://www.supercluster.org/mailman/listinfo/torqueusers
> 
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.11 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
> 
> iEYEARECAAYFAk88zfUACgkQoYgNqgF2egpEtACgiqRtn07Qmqhsyrbfig6HESuR
> gwMAn2Uho2jAHn5SN/6IdHEErol7BHkJ
> =B1Fu
> -----END PGP SIGNATURE-----
> 
> _______________________________________________
> torqueusers mailing list
> torqueusers at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torqueusers

More information about the torqueusers mailing list