[torqueusers] Overriding command line options in the Torquesubmitfilter ?

David Singleton David.Singleton at anu.edu.au
Fri May 23 16:50:50 MDT 2008

We came across a few problems with wrapping/modifying qsub:
   * qalter also had to be wrapped
   * people started using the PBS API, or Python wrappers for
     the API, including pbs_submit() and pbs_alterjob()
   * there were cases where we had to (or it was easiest to)
     allow a "vanilla" set of PBS commands including qsub to
     access our servers (Chris knows about the Aust. Grid
     gateway setup)

In the end we put all the stuff we had in wrappers into the
server proper as a pluggable module called userproj - the interface
to the server is just 4 functions.  Most of our active version of
userproj is related to ncpus-dependent per-user and per-project
queue resource limits but along the way it checks, and sets default,


Matney Sr, Kenneth D. wrote:
> Hi Chris,
> What we have done at ORNL for quite some time is to script qsub.  Our
> original implementation was done ca. 2004 for PBSPro (for Cray X1, which
> ran (IRIX 6.5-based) UNICOS/mp] and, at the time, was a fairly simple
> perl script.  Since then, we have moved on to Torque and the script has
> greatly increased in complexity, but the basic concept remains the same.
> If you move the installed qsub to a new location and write a script to
> transfer control to the original qsub, you can parse and re-write the
> command line.  This just seemed to be the simplest approach to
> overriding the basic PBS/Torque design.  (As the man page says, command
> line options override directives and the submitfilter is just a way to
> write or rewrite directives.)
> Of course, for the case in which the command line does not contain an
> account specification, you will still need the submitfilter to
> parse/rewrite batch scripts.  Something that we do not do, but I think
> is a good idea is to place the rewrite rules into a common script, so
> that the submitfilter and the scripted qsub always reference the same
> set of rules.  On the Cray X1, the overhead might have been a bit
> excessive.  But, in current Linux clusters, it should not be an issue.
> Also, keep in mind that for the end-case of this scenario, in which the
> user specifies a null directive prefix, you will want to force the
> default to the command line when transferring control to the original
> qsub.

Chris Samuel wrote:
 > Hi all,
 > Does anyone know if there is a way in the Torque
 > submit_filter to override the command line options
 > that are set as arguments to the filter ?
 > In the near future we're going to be automatically
 > specifying users default projects in their PBS scripts
 > using the submit filter, and if they make a typo and
 > pick someone elses project (or one they are not in any
 > more) in their PBS script we will just convert it to their
 > default.
 > Obviously should they specify a non-default project that
 > they are in then we'll leave it alone.
 > So far so good, and that all works.
 > Problem is that if they specify it on the command line
 > I reckon it'll override whatever we write into the script
 > and the best we could do is parse the command line arguments
 > and reject the job by exiting with -1 should they pick
 > a project they're not in.
 > We'd *much* rather be able to change the parameter, but
 > looking at the qsub code I can't see any way for that
 > to happen.
 > Any clues, or anything I've missed ?
 > cheers!
 > Chris

More information about the torqueusers mailing list