[gold-users] listing jobs charged to a specific allocation

[Scott Jackson]

Kevin Van Workum wrote:
> On Tue, Mar 9, 2010 at 8:00 PM, Scott Jackson
> <scottmo at adaptivecomputing.com> wrote:
>   
>> Kevin Van Workum wrote:
>>     
>>> On Mon, Mar 8, 2010 at 7:18 PM, Scott Jackson
>>> <scottmo at adaptivecomputing.com> wrote:
>>>
>>>       
>>>> Kevin,
>>>>
>>>> Kevin Van Workum wrote:
>>>>
>>>>         
>>>>> I'm trying to create a command that will list jobs that have been
>>>>> charged to a specific allocation. I realize I can use "glstxn -i Id -O
>>>>> Job", but that lists the transactions.
>>>>>
>>>>>           
>>>> Well, it is not that simple because a job can have any number of charges
>>>> against it and each charge can be to one or more allocations. So, it is
>>>> not
>>>> possible to directly relate a one-to-one answer to this question. You
>>>> basically are going to have to look at this in terms of the transactions,
>>>> though this can certainly be narrowed down to just Charges.
>>>>
>>>> Ohh, I see -- you know that, you just want any job listed that has ever
>>>> had
>>>> a charge against a particular allocation...
>>>>
>>>> goldsh Transaction Query Show:=JobId,Amount Action==Charge Allocation==26
>>>>
>>>> or
>>>>
>>>> glstxn -A Charge -i 26 --show=JobId
>>>>
>>>> If you want the job info for these guys, you'll have to use goldsh to do
>>>> a
>>>> join:
>>>>
>>>> goldsh Job,Transaction Query
>>>>
>>>> Show:=Job.Id,Job.JobId,Job.User,Job.Project,Job.Machine,Job.Processors,Job.WallDuration,Job.Charge
>>>> Transaction.Action==Charge Transaction.Allocation==26
>>>> Transaction.JobId==Job.JobId
>>>>
>>>> Id  JobId        User    Project   Machine Processors WallDuration Charge
>>>> --- ------------ ------- --------- ------- ---------- ------------ ------
>>>> 32  2            root    catchall  colony             1               123
>>>> 36  root.8       root    catchall  keko    1          465               1
>>>> 37  root.9       root    catchall  keko    1          523               1
>>>> 40  root.12      root    catchall  keko    1          362               1
>>>> 43  RHEL3ASVM.15 root    catchall  keko    1          58              111
>>>> 45  RHEL3ASVM.17 root    catchall  keko    1          17              111
>>>> 51  80           scottmo catchall  keko    1          277             277
>>>>
>>>>         
>>> Cool. I didn't know you could do that. Unfortunately, gold seems to
>>> choke on this. The same command as above just hangs. Then I have to
>>> restart postgresql to get things back to normal.
>>>
>>>
>>>       
>> Hmmmm..... maybe your Transaction table (and Job table) are ginormous. I
>> can't think of any other reason for the hang. Just for curiosity's sake, how
>> long have you waited for it to return?
>>     
>
> About 1 minute. My /var/lib/pgsql/data dir is about 466M.
>
>   
It is probably working then. This type of a join on two very large 
tables could take substantially longer than this.
>> I'm sorry if I've asked before, but
>> do you have the indexes in place?
>>     
>
> I never added them explicitly, but I'm using 2.1.9.0 which I believe
> has them in place.
>
>   
I would agree.
>> How many rows are in your Transaction
>> table? Job table.
>>     
>
> Not sure. How do I figure that out? I'm nearly clueless with postgres.
> How about this:
>
> gold=> select count(*) from g_transaction;
>  count
> --------
>  867776
> (1 row)
>
>
>   
>> We are doing a join on it, so this may not be practical.
>> If they are too large and postgres does not have memory for the join, we may
>> need to take the iterative approach.
>>     
>
> I've already implemented this in a way. Basically, I first do a
> Transaction Query to get the list of JobId's, then do a Job Query.
> Finally I grep for common JobId's.
>
>   
>> It would be interesting to try to do a
>> direct join in Postgres to do the similar join query in the psql prompt. You
>> can get the sql that gold is trying to enact in the goldd.log if you want a
>> good starting point.
>>
>> Scott
>>
>>
>>
>>     
>>>>> I want to get the list of jobs.
>>>>> I could first use glstxn, then use glsjob for each job found, but is
>>>>> there a more efficient way. Maybe something like:
>>>>>
>>>>>  my $request = new Gold::Request(object => "Job", action => "Query");
>>>>>  Gold::Client::buildSupplements($request);
>>>>>  $request->setCondition("AllocationId", "123");
>>>>>
>>>>>
>>>>>           
>>>> To do this in the perl api, something like (I'm winging it here, you will
>>>> have to tweak this a tad):
>>>>
>>>> my $request = new Gold::Request(
>>>>  objects => [
>>>>          new Gold::Object(name => "Job"),
>>>>          new Gold::Object(name => "Transaction")
>>>>            ], action => "Query");
>>>>
>>>> $request->setSelection("Id", "", "Job");
>>>> $request->setSelection("JobId", "", "Job");
>>>> $request->setSelection("User", "", "Job");
>>>> ...
>>>> $request->setCondition("Action", "Charge", "", "", "", "Transaction");
>>>> $request->setCondition("Allocation", "26", "", "", "", "Transaction");
>>>> $request->setCondition("JobId", "JobId", "", "", "", "Transaction",
>>>> "Job");
>>>>
>>>>
>>>>
>>>> See perldoc for Request, Condition, Selection, Object, etc.
>>>>
>>>>
>>>>
>>>>
>>>>         
>>>>> --
>>>>> Kevin Van Workum, PhD
>>>>> Sabalcore Computing Inc.
>>>>> Run your code on 500 processors.
>>>>> Sign up for a free trial account.
>>>>> www.sabalcore.com
>>>>> 877-492-8027 ext. 11
>>>>> _______________________________________________
>>>>> gold-users mailing list
>>>>> gold-users at supercluster.org
>>>>> http://www.supercluster.org/mailman/listinfo/gold-users
>>>>>
>>>>>
>>>>>           
>>>>         
>>>
>>>
>>>       
>>     
>
>
>
>