[torqueusers] Interacting with Torque/Maui from Perl scripts

Coyle, James J [ITACD] jjc at iastate.edu
Wed Aug 17 14:51:28 MDT 2011


Anthony,

     Assuming that you are running the perl script on the head node of the cluster,
and assuming you have a newer torque (version 2.5.4 is sufficient) installed,
how about the following:  If t1 contains the pbs job:

perl -e '$A=`qsub -I -x -lnodes=1:ppn=1 t1; echo $?  `; print $A'

  Those are backticks before qsub and after echo $?

  The -I -x combination causes the qsub to print output to the screen, and to hold onto the
job while it is run, which is what I think that you want.
I am print the value of $A, but you could use it in your perl program.


E.g.for me:

t1 contains:

date
sleep 3
echo 'test'
sleep 3
date

Issuing the command

time perl -e '$A=`qsub -I -x -lnodes=1:ppn=1   t1; echo $?  `; print $A'

yields:

qsub: waiting for job 150965.hpc5 to start
qsub: job 150965.hpc5 ready

Wed Aug 17 15:50:20 CDT 2011
test
Wed Aug 17 15:50:26 CDT 2011

qsub: job 150965.hpc5 completed
0
0.027u 0.042s 0:06.26 0.9%      0+0k 0+16io 0pf+0w



-          Jim


James Coyle, PhD
High Performance Computing Group
 Iowa State Univ.
web: http://jjc.public.iastate.edu/<http://www.public.iastate.edu/~jjc>


From: torqueusers-bounces at supercluster.org [mailto:torqueusers-bounces at supercluster.org] On Behalf Of William Nelson
Sent: Wednesday, August 17, 2011 10:36 AM
To: Torque Users Mailing List
Cc: torquedev at supercluster.org
Subject: Re: [torqueusers] Interacting with Torque/Maui from Perl scripts

Hi Anthony,
I don't have a specific answer to your question but I can point you to some example code. InterProScan (http://www.ebi.ac.uk/Tools/pfa/iprscan/) is a PERL project that integrates with PBS, LSF, or SGE. We use it with Torque by making small modifications to the PBS conf files. It runs several analysis tool in parallel on the cluster and then merges the results.
-Bill

On Wed, Aug 17, 2011 at 1:33 AM, Anthony Ferrari <ferraria at gmail.com<mailto:ferraria at gmail.com>> wrote:

Hi all,

Basically, I would like to launch a job on a cluster with specific resources and dependencies, wait for the job to complete or fail, check that all has gone OK, then do something with the generated files ...

To launch jobs, I am using a customized version of PBS-Client-0.09 package written by Ka-Wai Mak.
I also saw that packages named PBS-0.03 by Todd Merritt and perl-PBS by Garrick Staples exist but they do not seem maintained or seem outdated (please correct me if I am wrong).
What do you personally use ?

My problem here is more after the job submission, to keep 'control' of the job.
Because with the PBS-Client module, once the job is submitted, Torque does what it has to do and that's it. It does not give any information in return to my Perl script.

I was thinking about several solutions to achieve this but I do not find them very smart ... like :
 - using qstat | grep '^myjobid' in a while loop (with open3 perl function)
 - waiting for the output & error files to be written where I expect them (hum...)
 - use an epilogue script in the job to do something useful...

But there may exist brighter answers to this question. Perhaps with some modules I don't know about.
Do you have any advice ?


And actually this is to know when the job has finished but how do you check that the job completed and did not crash ?
Do you always parse the created logs ?
Is there a way to propagate the exit code of the program launched with qsub to the perl script running on the master node ?


Thank you very much.

Cheers,
Anthony



_______________________________________________
torqueusers mailing list
torqueusers at supercluster.org<mailto:torqueusers at supercluster.org>
http://www.supercluster.org/mailman/listinfo/torqueusers



--
Bill Nelson
Bioinformatics Developer
TAIR
Carnegie Institution of Washington
Dept. of Plant Biology
260, Panama
Stanford, CA 94305
USA

Cell: (650) 720-5708
Phone: Phone: (650) 739-4343
Fax: (650) 325-6857

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/torqueusers/attachments/20110817/36912d29/attachment-0001.html 


More information about the torqueusers mailing list