[torquedev] Looking for submit filter examples

Gareth.Williams at csiro.au Gareth.Williams at csiro.au
Sun Mar 20 16:45:29 MDT 2011


> -----Original Message-----
> From: torquedev-bounces at supercluster.org [mailto:torquedev-
> bounces at supercluster.org] On Behalf Of Ken Nielson
> Sent: Friday, 18 March 2011 8:53 AM
> To: Torque Developers mailing list
> Subject: Re: [torquedev] Looking for submit filter examples
> 
> On 03/17/2011 12:18 PM, Troy Baer wrote:
> > On Thu, 2011-03-17 at 09:52 -0600, Ken Nielson wrote:
> >> We would like to improve Appendix J of the TORQUE documentation to
> >> include more examples of submit filters and what they can
> accomplish.
> >>
> http://www.adaptivecomputing.com/resources/docs/torque/a.jqsubwrapper.p
> hp
> >>
> >> Is there anyone who would be willing to send us an example of their
> >> submit filters and what they try to accomplish with them. We would
> then
> >> add the example to the documentation.
> > I'd love to send in ours, but I fear that it's much too complex to
> make
> > a good example.  For one thing, it's ~1100 lines of fairly densely
> > written Perl...  If I get some free time, maybe I can sanitize and
> > simplify it to the point where it would be comprehensible to somebody
> > outside of NICS.
> >
> > 	--Troy
> That seems to be a common theme in the replies I have seen so far.
> Submit filters get long and complex. Probably what we need is to know
> what they are being used for in the real world and a few examples from
> the code showing how to do those things.
> 
> I appreciate the response
> 
> Regards
> 
> Ken

Hi Ken,

Here is the filter we have in place on one of our clusters.  It doesn't do anything right now, but at one point it ran some checks and gave feedback on nodes vs ncpus.  The structure is still illustrative, parsing command line options and the header, but there are just comments where you would handle errors or insert directives - and I don't have an example of altering or removing a directive.  You have limited (or no) options available to 'fix' problems in commands line options, so sometimes refusing to accept the job is the right action.

I've just noticed that the line 'shift @ARGV; # drop #PBS sentinel' would break if there is not a space between the sentinel and the options... but I think the use of Options helps readability.  Hmmf.  Well it's a starting point.  At least such scripts are easy to test as it's just a filter you can invoke stand-alone on the command line.

Cheers,

Gareth

#!/usr/bin/env perl
# $Id: qsub_filter 68 2008-05-27 12:52:02Z wil240 $
# $HeadURL: svn+ssh://cherax/usr/local/svn/sysadmin/burnet_xcat/pbs/qsub_filter $
# Gareth Williams CSIRO Fri Oct 28 2005
# http://www.clusterresources.com/wiki/doku.php?id=torque:appendix:j_submit_filter
# expects that server/queues are set up with:
#  resources_default.nodect = 1
#  resources_default.nodes = 1
# and the scheduler is setup with shared access to nodes
# May 2008, trimmed for new system
# _could_ give warnings about requesting ncpus and mem
use strict;
$^W = 1; # use warnings;
use Getopt::Long qw(:config pass_through); # for extracting -l options from ARGV
my $nodestring;
my $np_default = 1; # default number of ppn allocated in scheduling if not specified
my $np_header = 0;
my $ncpus_header = 0;
my @lresources = ();
my $lresource;
my $verbose = 0; # can embed line(s) with #verbose in script to turn verbosity up
my $debug = 0;
my $filter = 1;
my @savedargs = @ARGV;
print STDERR "qsub filter: filtering script through $0, to check resources requested\n" if $verbose;
# get requested resources from qsub args (which are also my args)
GetOptions('l=s' =>\@lresources);
foreach $lresource (@lresources) {
    print STDERR "qsub filter: in command line arguments found resourcestring '$lresource'\n" if $verbose;
    if ($lresource =~ /(?:^|,)nodes=([^,]+)/) {
        $np_header= &getnprocs($1); # np from nodes string
        print STDERR "  found nodestring '$1'\n" if $verbose > 1;
        print STDERR "  that would correspond to '$np_header' processes\n" if $verbose > 1;
    }
    if ($lresource =~ /(?:^|,)ncpus=(\d+)/) {
        $ncpus_header= $1; # ncpus if specified
        print STDERR "  found ncpus '$1'\n" if $verbose > 1;
        # this is the first opportunity to bail out if $ncpus_header >1
    }
}

my $head_done = 0; # only parse header - then add extra directives
my $np = $np_default;
my $ncpus  = 0;
my $lineno = 0;
my $header = "";
while (<STDIN>) {
    $lineno++;

    $debug = 1 if /^#debug/; # can embed line(s) with #debug to turn debug on
    $verbose++ if /^#verbose/; # can embed line(s) with #verbose in script to turn verbosity up
    $filter = 1 if /^#filter/; # can embed line(s) with #filter to turn filtering on
#    $filter = 0 if /^#nofilter/; # can embed line(s) with #nofilter to turn filtering off

    unless ($filter) { print; next }
    if ($head_done) { print; next } # do not filter after header
    if ($lineno == 1 and /^:/) { print; next } # skip special null command allowed in first line

    unless (/^ *#/ or /^ *$/) {
        $head_done=1;
        print STDERR "qsub filter: finished processing script header\n" if $verbose > 1;
        # could add directives/content here
    }
    $header .= $_;

    if (/^ *#PBS/) { # parse #PBS lines
        print STDERR "qsub filter: parsing a directive line:$_" if $verbose > 1;
        @ARGV = split; # reconstruct ARGV to use with GetOptions
        shift @ARGV; # drop #PBS sentinel
        @lresources = ();
        GetOptions('l=s' =>\@lresources);
        foreach $lresource (@lresources) {
            print STDERR "  found resourcestring '$lresource'\n" if $verbose > 1;
            if ($lresource =~ /(?:^|,)nodes=([^,]+)/) {
                $np = &getnprocs($1); # np from node string
                print STDERR "  found nodestring '$1'\n" if $verbose > 1;
                print STDERR "  that would correspond to '$np' processes\n" if $verbose > 1;
            }
            if ($lresource =~ /(?:^|,)ncpus=(\d+)/) {
                $ncpus = $1; # ncpus if specified
                print STDERR "  found ncpus '$1'\n" if $verbose > 1;
            }
        }
        @ARGV = ();
    }
    # could take action here...
    print;
}

sub getnprocs {
    my $nodestring = shift;
    my $nprocs = 0;
    my @chunks = split '\+', $nodestring; #nodestring like i:property+j:ppn=k+...
    foreach my $chunk (@chunks) {
        my ($nn, $np) = (1, $np_default);
        if ($chunk =~ /^(\d+)/) {
            $nn = $1;
        }
        if ($chunk =~ /ppn=(\d+)/) {
            $np = $1;
        }
        $nprocs += $nn*$np; # sum of  nodes*ppn  over chunks
    }
    return $nprocs;
}


More information about the torquedev mailing list