[gold-users] importing old job data into gold

Mehmet Soysal mehmet.soysal at kit.edu
Wed Oct 6 03:23:53 MDT 2010


Hi Steve,
i had the same issue like you and wanted to modify modification and 
creation time directly in the db,
but i think this is a really bad idea, cause i think that you will run 
in more problems than you solve.

We are using Gold only for stats and not about debits and other stuff.

I just modified some original scripts with a few more lines for this 
renamed those,
that they do not affect the original ones.. If you work a little bit
with the Perl API you will see that is it quite simple.


The Output looks like that (i added option --showsum and modified 
behavior of --quiet):

kit_glsjob -m xc2 -s 2010-02-01 -e 2010-03-01  --showsum --quiet -h
########################################################
########################################################
Used CPU Time for Selection:
518540.60444444 hours
########################################################
Group XYZ has used: 210707.765555556
XYZ_USER has used:210707.765555556
########################################################
Group ABC has used: 32757.5288888889
ABC_USER1 has used:15.4266666666667
ABC_USER2 has used:32742.1022222222
...... and so on ......

kit_glsjob scripts acts almost like the normal glsjob.


MfG
M.Soysal



> Thanks for the response Scott. Gold is essentially a bank.
>
> I guess I need to clarify my statement a bit more then. I don't care about
> applying old charges to people's balances, but more about starting at a
> baseline with our old Maui logs. At this point I wouldn't care if after
> running a massive script if every user/project/etc was millions of hours in
> debt. I just want the historical usage numbers, with a good way to query
> them by time periods. Basically, my place would be to actually run gold on
> our cluster AFTER we do an import, change the balances, etc...
>
> We're not at the point yet where we want to start using Gold as a
> charging/CPU hours banking system, but more internally so we have easy to
> query numbers. I see an evolution coming soon, but let's not get ahead of
> ourselves.
>
> Let's say I create a fresh instance of Gold, import any
> users/groups/setting/etc, but do not have any job data. Now, just importing
> the Maui logs via a script would produce the behavior you specified before,
> (i.e job records get debited that day). Would there be any way of changing
> the times in the gold database itself to reflect the actual debit times? I
> guess it would be tricking the system if I changed all of the references in
> the right place, but would it be theoretically possible to change the values
> in the DB, so that some of the historical gold commands would reflect the
> actual job times?
>
>
>
> Maybe I'm just trying to dig too deep into this, because it seems I can use
> a combination of glsjob + start/EndTimes to achieve what I'd like after job
> imports, but I'd rather not rewrite your gusage/statement/etc commands,
> especially since they natively work with the db, and they will probably be a
> heck of a lot more efficient than my perl ;-)
>
>
>
>
>
> On 10/1/10 6:44 PM, "Scott Jackson"<scottmo at adaptivecomputing.com>  wrote:
>
>    
>> Steve,
>>
>> It was introduced in gold-2.2. If you are using 2.1 you will not have gmkjob.
>>
>> The imported jobs will not show up against the old statement, they will show
>> up against a current statement only. However, the jobs will be inserted into
>> the job table with the specified StartTime and EndTime (while CreationTime and
>> ModificationTime will be current dates). You have to realize that Gold tracks
>> the historical states of all allocations and objects and you can't just insert
>> debits into the past. This would disrupt and change every intervening state.
>> This would have effected balances causing other jobs not to have been able to
>> run, invalidating previously generated statements, etc. You would never see a
>> bank inserting credits or debits into their past balance sheets. If a bank
>> catches a mistake, they correct it in the present with a current refund or
>> withdrawal.
>>
>> When you import the job data, it does not affect balances. If you wish to
>> catch up for the past charges, the charges will occur in the present.
>>
>> Scott
>>
>>
>> ----- Original Message -----
>> From: "Steve Crusan"<scrusan at UR.Rochester.edu>
>> To: gold-users at supercluster.org
>> Sent: Friday, October 1, 2010 3:20:57 PM
>> Subject: [gold-users] importing old job data into gold
>>
>>
>> importing old job data into gold
>>
>> Quick question about importing old job data (from Maui logs) into Gold.
>>
>> I've noticed that it seems to be possible to do this using the goldsh shell
>> (in the manual it shows gmkjob, but I cannot seem to find it; is it
>> deprecated?):
>>
>>
>> goldsh Job Create JobId=PBS.$goldId.0 User=$netId Project=$account
>> Machine=$machine Charge=$charge Processors=$procs StartTime=$startTime
>> EndTime=$endTime WallDuration=$walltime
>>
>>      
>>>  From there, it seems one can use gcharge to charge the job:
>>>        
>> gcharge -p $account -u $netId -m $machine -P $procs -t $walltime -j $goldId -d
>> "charge imported from old data" -s $startTime -e $endTime
>>
>> Now, provided that I properly fill in the correct attributes, and use the
>> proper gold job id, would doing the above operations properly keep the
>> timestamps of the jobs? Meaning that if I have job logs from September 2009,
>> if I import them and use their epoch time (do all the conversions, etc), will
>> the gstatement commands and other querying operations reflect the proper
>> dates/times?
>>
>> Ex:
>> gstatement -s 2009-09-10 -e 2009-12-25
>>
>> Will the imported jobs from the Maui logs properly show up within that
>> statement, and NOT show up for a rather recent query?
>>
>> Thanks!
>>
>>
>>
>> ----------------------
>> Steve Crusan
>> System Administrator
>> Center for Research Computing
>>
>>
>> _______________________________________________
>> gold-users mailing list
>> gold-users at supercluster.org
>> http://www.supercluster.org/mailman/listinfo/gold-users
>> _______________________________________________
>> gold-users mailing list
>> gold-users at supercluster.org
>> http://www.supercluster.org/mailman/listinfo/gold-users
>>      
>
>
> ----------------------
> Steve Crusan
> System Administrator
> Center for Research Computing
> University of Rochester
> https://www.crc.rochester.edu/
>
> _______________________________________________
> gold-users mailing list
> gold-users at supercluster.org
> http://www.supercluster.org/mailman/listinfo/gold-users
>    


-- 
---------------------------------------+--------------------------------------
Mehmet Soysal                          | Tel   : +49 721 608-6347
Karlsruhe Institut of Technology (KIT) | Email : mehmet.soysal at kit.edu
Steinbuch Centre for Computing (SCC)   |
Zirkel 2                               |
D-76128 Karlsruhe                      |
---------------------------------------+--------------------------------------


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6349 bytes
Desc: S/MIME Cryptographic Signature
Url : http://www.supercluster.org/pipermail/gold-users/attachments/20101006/a29bd601/attachment-0001.bin 


More information about the gold-users mailing list