[gold-users] listing jobs charged to a specific allocation

[Kevin Van Workum]

On Tue, Mar 9, 2010 at 8:00 PM, Scott Jackson
<scottmo at adaptivecomputing.com> wrote:
> Kevin Van Workum wrote:
>>
>> On Mon, Mar 8, 2010 at 7:18 PM, Scott Jackson
>> <scottmo at adaptivecomputing.com> wrote:
>>
>>>
>>> Kevin,
>>>
>>> Kevin Van Workum wrote:
>>>
>>>>
>>>> I'm trying to create a command that will list jobs that have been
>>>> charged to a specific allocation. I realize I can use "glstxn -i Id -O
>>>> Job", but that lists the transactions.
>>>>
>>>
>>> Well, it is not that simple because a job can have any number of charges
>>> against it and each charge can be to one or more allocations. So, it is
>>> not
>>> possible to directly relate a one-to-one answer to this question. You
>>> basically are going to have to look at this in terms of the transactions,
>>> though this can certainly be narrowed down to just Charges.
>>>
>>> Ohh, I see -- you know that, you just want any job listed that has ever
>>> had
>>> a charge against a particular allocation...
>>>
>>> goldsh Transaction Query Show:=JobId,Amount Action==Charge Allocation==26
>>>
>>> or
>>>
>>> glstxn -A Charge -i 26 --show=JobId
>>>
>>> If you want the job info for these guys, you'll have to use goldsh to do
>>> a
>>> join:
>>>
>>> goldsh Job,Transaction Query
>>>
>>> Show:=Job.Id,Job.JobId,Job.User,Job.Project,Job.Machine,Job.Processors,Job.WallDuration,Job.Charge
>>> Transaction.Action==Charge Transaction.Allocation==26
>>> Transaction.JobId==Job.JobId
>>>
>>> Id  JobId        User    Project   Machine Processors WallDuration Charge
>>> --- ------------ ------- --------- ------- ---------- ------------ ------
>>> 32  2            root    catchall  colony             1               123
>>> 36  root.8       root    catchall  keko    1          465               1
>>> 37  root.9       root    catchall  keko    1          523               1
>>> 40  root.12      root    catchall  keko    1          362               1
>>> 43  RHEL3ASVM.15 root    catchall  keko    1          58              111
>>> 45  RHEL3ASVM.17 root    catchall  keko    1          17              111
>>> 51  80           scottmo catchall  keko    1          277             277
>>>
>>
>> Cool. I didn't know you could do that. Unfortunately, gold seems to
>> choke on this. The same command as above just hangs. Then I have to
>> restart postgresql to get things back to normal.
>>
>>
>
> Hmmmm..... maybe your Transaction table (and Job table) are ginormous. I
> can't think of any other reason for the hang. Just for curiosity's sake, how
> long have you waited for it to return?

About 1 minute. My /var/lib/pgsql/data dir is about 466M.

> I'm sorry if I've asked before, but
> do you have the indexes in place?

I never added them explicitly, but I'm using 2.1.9.0 which I believe
has them in place.

> How many rows are in your Transaction
> table? Job table.

Not sure. How do I figure that out? I'm nearly clueless with postgres.
How about this:

gold=> select count(*) from g_transaction;
 count
--------
 867776
(1 row)


> We are doing a join on it, so this may not be practical.
> If they are too large and postgres does not have memory for the join, we may
> need to take the iterative approach.

I've already implemented this in a way. Basically, I first do a
Transaction Query to get the list of JobId's, then do a Job Query.
Finally I grep for common JobId's.

> It would be interesting to try to do a
> direct join in Postgres to do the similar join query in the psql prompt. You
> can get the sql that gold is trying to enact in the goldd.log if you want a
> good starting point.
>
> Scott
>
>
>
>>>> I want to get the list of jobs.
>>>> I could first use glstxn, then use glsjob for each job found, but is
>>>> there a more efficient way. Maybe something like:
>>>>
>>>>  my $request = new Gold::Request(object => "Job", action => "Query");
>>>>  Gold::Client::buildSupplements($request);
>>>>  $request->setCondition("AllocationId", "123");
>>>>
>>>>
>>>
>>> To do this in the perl api, something like (I'm winging it here, you will
>>> have to tweak this a tad):
>>>
>>> my $request = new Gold::Request(
>>>  objects => [
>>>          new Gold::Object(name => "Job"),
>>>          new Gold::Object(name => "Transaction")
>>>            ], action => "Query");
>>>
>>> $request->setSelection("Id", "", "Job");
>>> $request->setSelection("JobId", "", "Job");
>>> $request->setSelection("User", "", "Job");
>>> ...
>>> $request->setCondition("Action", "Charge", "", "", "", "Transaction");
>>> $request->setCondition("Allocation", "26", "", "", "", "Transaction");
>>> $request->setCondition("JobId", "JobId", "", "", "", "Transaction",
>>> "Job");
>>>
>>>
>>>
>>> See perldoc for Request, Condition, Selection, Object, etc.
>>>
>>>
>>>
>>>
>>>>
>>>> --
>>>> Kevin Van Workum, PhD
>>>> Sabalcore Computing Inc.
>>>> Run your code on 500 processors.
>>>> Sign up for a free trial account.
>>>> www.sabalcore.com
>>>> 877-492-8027 ext. 11
>>>> _______________________________________________
>>>> gold-users mailing list
>>>> gold-users at supercluster.org
>>>> http://www.supercluster.org/mailman/listinfo/gold-users
>>>>
>>>>
>>>
>>>
>>
>>
>>
>>
>
>



-- 
Kevin Van Workum, PhD
Sabalcore Computing Inc.
Run your code on 500 processors.
Sign up for a free trial account.
www.sabalcore.com
877-492-8027 ext. 11