[torqueusers] Re: Evaluating DRMAA + Torque

Darren Platt darren at 23andme.com
Wed Jun 4 15:57:18 MDT 2008

> Darren Platt wrote:
>> Thanks!  One of the attractions is the portability, so we don't have to
>> recode if we outgrow or change schedulers.  Part of the problem seems to be
>> the abstraction.  For example, setting the outputPath  field in the job to
>>  myhost:/some/path/to/some/file.txt    didn't deliver the result to the
>> headnode in the way -o   would.  The problem seemed to be in how the drmaa
>> layer was rewriting the path,  assumptions about whether the intermediate
>> path /some/path / etc existed on the node vs the submit host.
> These are the exact sorts of issues I ran into when I was playing with

That's a pity - I assume it's because most users are relying on file systems
for data staging.  I managed to get it to at least send results out to the
submit host but only by
patching the source which didn't seem like the intended mode of use ;)

>  So I assume most people are using either the shell or torque api to submit
>> then?
> Absolutely. You might be able to abstract this concept into "job templates"
> that then only need to be modified to fit a new scheduler, allowing you to
> NOT change your code. Generally though, the most common or even popular
> scheduler's accept in the PBS (TORQUE) style submit scripts, so I don't
> think you are taking a risk by using TORQUE.

We are hoping to run things via an API so we can  get better integration
between applications and  the jobs submitted.  Has anyone written a python
API for torque's native  API?

Darren Platt
Senior Director, Research
23andMe, inc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/torqueusers/attachments/20080604/0ee4fe04/attachment.html

More information about the torqueusers mailing list