[torqueusers] Step Change in Job Arrays

Sreedhar Manchu sm4082 at nyu.edu
Thu Mar 1 10:37:25 MST 2012


Hello,

First I'm sorry for sending multiple emails. After sending the last message
I realized we can't put it in the qsub submit filter.  If we put it in qsub
filter it still complains bad array request. So I made another wrapper that
works from the system side.

We just have to create another wrapper so that wrapper goes through this
and changes arguments and resubmits it to qsub. You can use the code in my
previous email and put it in a file such as

/user/local/bin/qsub_wrapper.sh

on login node where users submit the jobs. Then add this line

/opt/torque/bin/qsub "$@"

to the end of the code.

This line submits the job with changed arguments. If you have submit filter
it works like always. Then we need to create an alias for our wrapper.

I have my alias as

[root at login-0-0 ~]# cat /etc/profile.d/qsub.sh
alias qsub='/usr/local/bin/qsub_wrapper.sh'

[root at login-0-0 ~]# ls -l $_
-rwxr-xr-x 1 root root 44 Mar  1 11:58 /etc/profile.d/qsub.sh

[root at login-0-0 ~]# cat /usr/local/bin/qsub_wrapper.sh
#!/bin/bash

args=("$@")
for((arg=0;arg<$#;arg++))
do
    if [ "${args[$arg]}" = "-t" ]
    then
        if echo "${args[$(($arg+1))]}" | egrep
"^(^0|^[1-9][0-9]*)-[1-9][0-9]*:[1-9][0-9]*$" > /dev/null 2>&1
        then
            array_arg="${args[$(($arg+1))]}"
            array_arg="$(seq -s, $(echo $array_arg | cut -f1 -d-) $(echo
$array_arg | cut -f2 -d:) $(echo $array_arg | cut -f1 -d: | cut -f2 -d-))"
            str="set -- \"\${@:1:$(($arg+1))}\" \"${array_arg}\"
\"\${@:$(($arg+3))}\""
            eval `echo "$str"`
            break
        fi
    fi
done
/opt/torque/bin/qsub "$@"
[root at login-0-0 ~]#

[root at login-0-0 ~]# ls -l $_
-rwxr-xr-x 1 root root 528 Mar  1 12:12 /usr/local/bin/qsub_wrapper.sh
[root at login-0-0 ~]#

I have tested and it works. See below for an example


[user at login-0-0 array_jobs]$ qsub -t 0-15:3 array.pbs
157713[].crunch.local

[user at login-0-0 array_jobs]$ qstat -t -u sm4082

crunch.local:

Req'd  Req'd   Elap
Job ID               Username Queue    Jobname          SessID NDS   TSK
Memory Time  S Time
-------------------- -------- -------- ---------------- ------ ----- ---
------ ----- - -----
157713[0].crunch     sm4082   s48      arraytest-0        2790     1   1
--  00:05 R   --
157713[3].crunch     sm4082   s48      arraytest-3        2804     1   1
--  00:05 R   --
157713[6].crunch     sm4082   s48      arraytest-6        2825     1   1
--  00:05 R   --
157713[9].crunch     sm4082   s48      arraytest-9        2849     1   1
--  00:05 R   --
157713[12].crunc     sm4082   s48      arraytest-12       2865     1   1
--  00:05 R   --
157713[15].crunc     sm4082   s48      arraytest-15      16818     1   1
--  00:05 R   --
[user at login-0-0 array_jobs]$


Sreedhar.


On Wed, Feb 29, 2012 at 10:51 PM, Sreedhar Manchu <sm4082 at nyu.edu> wrote:

> Hello,
>
> Putting the code below in qsub wrapper would make torque recognize step
> change like 1-20:2 or 0-19:3, etc. This works only when -t is mentioned in
> the command line. I made my script recognize the same even when users
> mention it inside the script as well. But it's quite long. I guess once we
> find #PBS -t 2-23:3, we just have to make the script change it to #PBS -t
> 2,5,8,11,14,17,20,23. Which can be easily done with Andy's method of using
> seq.
>
> Anyway, here is the code for wrapper for making it work with command line.
>
> #!/bin/sh
> args=("$@")
> for((arg=0;arg<$#;arg++))
> do
>     if [ "${args[$arg]}" = "-t" ]
>     then
>         if echo "${args[$(($arg+1))]}" | egrep
> "^(^0|^[1-9][0-9]*)-[1-9][0-9]*:[1-9][0-9]*$" > /dev/null 2>&1
>         then
>             array_arg="${args[$(($arg+1))]}"
>             array_arg="$(seq -s, $(echo $array_arg | cut -f1 -d-) $(echo
> $array_arg | cut -f2 -d:) $(echo $array_arg | cut -f1 -d: | cut -f2 -d-))"
>             str="set -- \"\${@:1:$(($arg+1))}\" \"${array_arg}\"
> \"\${@:$(($arg+3))}\""
>             eval `echo "$str"`
>             break
>         fi
>     fi
> done
>
> In case you already have a wrapper in place, it doesn't do any harm to
> keep it at the beginning of it.
>
> Sreedhar.
>
>
>
> On Wed, Feb 8, 2012 at 9:47 AM, Ibad Kureshi U0850037 <U0850037 at hud.ac.uk>wrote:
>
>> Thanks Glen, Andy
>>
>> Andy: Nice!
>>
>> -Ibad
>>
>>
>> ________________________________________
>> From: torqueusers-bounces at supercluster.org [
>> torqueusers-bounces at supercluster.org] On Behalf Of Andrew Caird [
>> acaird at umich.edu]
>> Sent: Wednesday, February 08, 2012 2:28 PM
>> To: Torque Users Mailing List
>> Subject: Re: [torqueusers] Step Change in Job Arrays
>>
>> On Wed, Feb 8, 2012 at 9:02 AM, Glen Beane <glen.beane at gmail.com<mailto:
>> glen.beane at gmail.com>> wrote:
>> On Wed, Feb 8, 2012 at 8:57 AM, Ibad Kureshi U0850037
>> <U0850037 at hud.ac.uk<mailto:U0850037 at hud.ac.uk>> wrote:
>> > Hello,
>> >
>> > I was wondering is someone could tell me how to adjust the step size in
>> a job array. We are running Torque 2.5.7 with the PBS_SCHEDD on a small
>> cluster and our users want to submit arrays.
>> >
>> > One the SGE and the Moab/Torque based systems
>> >
>> > $ -t 1-20:2
>> >
>> > or
>> >
>> > #PBS -t 1-20:2
>> >
>> > respectively, gives them 10 jobs with even ID numbers.
>> >
>> > How can this be done with Torque? It throws out "qsub: Bad Job Array
>> Request" error
>> >
>> > Have not been able to find much literature on this.
>> >
>> > Thanks
>>
>>
>> this is not currently supported, but it is a great feature request.
>>
>> unfortunately the only option would be to explicitly specify each array
>> ID:
>>
>> #PBS -t 2,4,6,8,10 ...20
>>
>> Or:
>>
>>  qsub -t `seq -s, 2 2 20` pbsfile.txt
>>
>> in case you don't want to type all the numbers.
>>
>> --andy
>>
>>
>> ---
>> This transmission is confidential and may be legally privileged. If you
>> receive it in error, please notify us immediately by e-mail and remove it
>> from your system. If the content of this e-mail does not relate to the
>> business of the University of Huddersfield, then we do not endorse it and
>> will accept no liability.
>> _______________________________________________
>> torqueusers mailing list
>> torqueusers at supercluster.org
>> http://www.supercluster.org/mailman/listinfo/torqueusers
>>
>
>
>
> --
> Sreedhar Manchu
> HPC Support Specialist
> New York University
> 251 Mercer Street
> New York, NY 10012-1110
>
>


-- 
Sreedhar Manchu
HPC Support Specialist
New York University
251 Mercer Street
New York, NY 10012-1110
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/torqueusers/attachments/20120301/0dfeea83/attachment.html 


More information about the torqueusers mailing list