[torqueusers] accounting

Gus Correa gus at ldeo.columbia.edu
Tue Jun 23 18:44:46 MDT 2009


Hi Jon, Troy, list

Besides Troy's suggestions:

Ole Holm Nielsen wrote a suite of scripts to do Torque/PBS accounting
reports:

ftp://ftp.fysik.dtu.dk/pub/Torque/pbsacct-1.4.6.tar.gz

I never tried it,
but would guess a prologue script may be able to do #2 also,
say, by checking if the project number is on a list of projects,
and returning an abort exit code (1),
if the project is not valid:

http://www.clusterresources.com/torquedocs21/a.gprologueepilogue.shtml

However, this may reject the job only when it starts,
not right after qsub, like Troy's "qsub filter" would do.
In other words, the prologue solution defers the job deletion to run time.

My doubt is how to enforce the use of the "qsub filter",
instead of the actual qsub command.
I suppose the user could always write the full path to qsub, instead of 
the filter alias.

You could use both solutions also,
if you are an overzealous sys admin. :)

I hope this helps,
Gus Correa
---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------

Troy Baer wrote:
> On Tue, 2009-06-23 at 21:09 +0200, Jon Tegner wrote:
>> We need to monitor the usage of our cluster resources, and we need to 
>> now how much time is spent by different projects (a user can work on 
>> several projects). So what I need to do is the following:
>>
>> 1. Each user should supply a project number when submitting a job.
>> 2. If the project number is not a valid one the job should be rejected.
>> 3. We need to extract the relevant information from the 
>> server_priv/accounting/2009xxxx-file.
>>
>> I can do 1 and 3 if I let all nodes have "project-properties", but I 
>> haven't managed to find out how to achieve 2.
>>
>> Can this be done in torque? And surely there is a better approach than 
>> this one?
> 
> One way to handle #1 and #2 is to write a submit filter that checks for
> a valid project code rejects the user's job if none is found.
> 
> As far as #3, there are lots of examples of code out there of how to
> parse and analyze the TORQUE accounting records.  For instance, PBS
> Tools [1] includes a set of Perl scripts that parse the accounting logs
> and stuff them into a MySQL database.  It all depends on what you want
> to do with it, and it seems that every site is slightly different in
> that regard.
> 
> [1] http://www.osc.edu/~troy/pbstools/pbstools.tar.gz
> 
> 	--Troy



More information about the torqueusers mailing list