[gold-users] Questions about statistics (gstatement/glsjob)

Scott Jackson scottmo at adaptivecomputing.com
Thu Aug 12 11:25:26 MDT 2010


Hi,


Mehmet Soysal wrote:
> Hi,
> first of all, i wanna apologize for my email.
> A lot of typo in that mail (and missing words), i coded whole day perl 
> and after that it is hard to concentrate for a clean mail :).
>
> Thanks for your detailed explanations. I understand that Gold is 
> almost a whole booking system. But i was asking myself if there i not 
> such way, that i can show
> which jobs ended in a month,

goldsh
gold> Job Query EndTime>="2010-06-01" EndTime<"2010-07-01" 
Show:="Id,User,Project,Machine,Processors,WallDuration"

> with a little bit summarization which projects used how much cpu time.
>
For this part, start with a command like gusage and tweak it to iterate 
through Job data multiplying processors time wallDuration.
> Im already using the perl API to do this things for me. Jobs are 
> charged with detailed Data how much core/nodes are used and correct 
> start- and enddates.
>
> As far as i know, our batchsystem (slurm) is not able to communicate 
> directly with Gold, like others (torque), thats the reason why we fill 
> Gold with filtered logs.
>
I would bet that it could. I know Torque can. Any resource manager that 
has a prolog and epilog script can tie into Gold. You just make an 
appropriate greserve command in the prolog and an appropriate gcharge 
command in the epilog. If you are not using it to block jobs with 
insufficient funds, you would not even need the prolog/greserve part. 
Just put in a ton of funds and do the charge part. I have seen this done 
with LSF, Torque, PBSPro, etc.

> I think Gold is a great piece of software and the Perl API is not hard 
> to understand,
> it is written very cleanly (and it is hard to find cleanly written 
> perl Code). But i think some examples (or tutorials) would be great 
> and help other users, especially using goldsh.
>
I can appreciate that. Right now I am very resource constrained as I am 
developing the next version of Gold. Once a year we have a Moab-Con 
conference and I would be available to give a tutorial along these lines 
if you wanted to come. I just don't have time at the moment to create 
any tutorials or extended docs around goldsh. Sorry. Maybe I could do 
this at Supercomputing? Maybe near the holiday season I could arrange an 
impromptu web tutorial.

Scott


> Thx for your help.
> M.Soysal
>
>> Hi M.Soysal,
>>
>> A couple of things:
>>
>> First, CreationTime represents when a job record was first created or
>> when an import event actually happened, while StartTime and EndTime
>> represent when a job started or ended respectively. It is not
>> infrequently the case that some schedulers or resource managers do not
>> supply the start or endtime of the job in the reservation or charge so
>> these fields are sometimes not filled in.
>>
>> Assuming you are just looking at when jobs ran, and further assuming
>> that you set proper startTime and endTime values for the jobs when you
>> imported them, then you should be able to come up with an appropriate
>> query when querying jobs to get the job listing that you want.
>>
>> Secondly, Gold maintains a consistent historical state of the balance
>> and it does not allow you to arbitrarily change past balance states.
>> Just as in a real bank, the debits and credits are registered. If a
>> mistake is detected, this is corrected through a refund, or an
>> appropriate present action. You cannot just go back and change past
>> states thereby invalidating and reformulating previous balance sheets.
>> If a wrong amount was charged yesterday, you can correct it today by a
>> refund or deposit, but you cannot change yesterday's balance. Same for
>> the importing of new job data. This will generally not affect the
>> balance. If you decide to correlate these with charges, the charges will
>> hit present balances, not previous balances.
>>
>> The gstatement command looks back at previous states of the account
>> balance. It uses CreationTime on the Transactions to filter out what
>> balance affecting actions occurred during the specified time period.
>> Gstatement is really looking at changes to the state of the account
>> balances over that time. Gold will not provide you a way to import the
>> jobs in such a way that impacts the balances at a previous time. If you
>> want to try to do this on your own you will have to accept whatever
>> inconsistencies it may create in the journals.
>>
>> For your immediate question, you would have to either write a client
>> using the perl gold api to do what you want or use the goldsh prompt,
>> something like:
>>
>>
>> Job Query EndTime>='2010-06-01' EndTime<'2010-07-01'
>> Show:="GroupBy(Project),Sum(Charge)"
>>
>> I hope this helps,
>>
>> Scott
>>
>> Mehmet Soysal wrote:
>>   
>>> Hi,
>>> since a few weeks we are using Gold now and i have some questions about
>>> how to get some statistics from Gold.
>>>
>>> I wrote some script to fill Gold with all our old logs,
>>> everything looks good and works great so far.
>>>
>>> But struggling about "CreationTime" and Job EndTime
>>>
>>> In my case i trying to get a summarization for example:
>>> How many job finished in a month (incl. used cpu time)
>>>      
>>
>>   
>>> and maybe which projects consumed how many cpu hours in this month and
>>> so on ..
>>>
>>> gstatement command looks right for this, but it is using the
>>> Creationtime from  allocations. But if i fill Gold with old logs they
>>> all have actual dates for creationtime.
>>>
>>> With glsjob i can tell which jobs i want:
>>> glsjob  -s "2009-04-01 00:00:00" -e "2009-04-30 23:59:59"
>>> but there i no kind of summarize for users/projects/machines ........
>>>
>>>
>>> I added some options and a little bit for glsjob which does this now
>>> for me, but im wondering if there is an already working solution for
>>> this.
>>>
>>> MfG
>>> M.Soysal
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------ 
>>>
>>>
>>> _______________________________________________
>>> gold-users mailing list
>>> gold-users at supercluster.org
>>> http://www.supercluster.org/mailman/listinfo/gold-users
>>>
>>>      
>> _______________________________________________
>> gold-users mailing list
>> gold-users at supercluster.org
>> http://www.supercluster.org/mailman/listinfo/gold-users
>>    
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> gold-users mailing list
> gold-users at supercluster.org
> http://www.supercluster.org/mailman/listinfo/gold-users
>   


More information about the gold-users mailing list