[torqueusers] Re: Evaluating DRMAA + Torque

Darren Platt darren at 23andme.com
Wed Jun 4 15:57:18 MDT 2008


>
> Darren Platt wrote:
>
>> Thanks!  One of the attractions is the portability, so we don't have to
>> recode if we outgrow or change schedulers.  Part of the problem seems to be
>> the abstraction.  For example, setting the outputPath  field in the job to
>>  myhost:/some/path/to/some/file.txt    didn't deliver the result to the
>> headnode in the way -o   would.  The problem seemed to be in how the drmaa
>> layer was rewriting the path,  assumptions about whether the intermediate
>> path /some/path / etc existed on the node vs the submit host.
>>
>
> These are the exact sorts of issues I ran into when I was playing with
> DRMAA.


That's a pity - I assume it's because most users are relying on file systems
for data staging.  I managed to get it to at least send results out to the
submit host but only by
patching the source which didn't seem like the intended mode of use ;)


>
>  So I assume most people are using either the shell or torque api to submit
>> then?
>>
>
> Absolutely. You might be able to abstract this concept into "job templates"
> that then only need to be modified to fit a new scheduler, allowing you to
> NOT change your code. Generally though, the most common or even popular
> scheduler's accept in the PBS (TORQUE) style submit scripts, so I don't
> think you are taking a risk by using TORQUE.


We are hoping to run things via an API so we can  get better integration
between applications and  the jobs submitted.  Has anyone written a python
API for torque's native  API?


-- 
Darren Platt
Senior Director, Research
23andMe, inc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/torqueusers/attachments/20080604/0ee4fe04/attachment.html


More information about the torqueusers mailing list