[gold-users] importing old job data into gold

Steve Crusan scrusan at ur.rochester.edu
Tue Oct 5 14:39:08 MDT 2010

Thanks for the response Scott. Gold is essentially a bank.

I guess I need to clarify my statement a bit more then. I don't care about
applying old charges to people's balances, but more about starting at a
baseline with our old Maui logs. At this point I wouldn't care if after
running a massive script if every user/project/etc was millions of hours in
debt. I just want the historical usage numbers, with a good way to query
them by time periods. Basically, my place would be to actually run gold on
our cluster AFTER we do an import, change the balances, etc...

We're not at the point yet where we want to start using Gold as a
charging/CPU hours banking system, but more internally so we have easy to
query numbers. I see an evolution coming soon, but let's not get ahead of

Let's say I create a fresh instance of Gold, import any
users/groups/setting/etc, but do not have any job data. Now, just importing
the Maui logs via a script would produce the behavior you specified before,
(i.e job records get debited that day). Would there be any way of changing
the times in the gold database itself to reflect the actual debit times? I
guess it would be tricking the system if I changed all of the references in
the right place, but would it be theoretically possible to change the values
in the DB, so that some of the historical gold commands would reflect the
actual job times?

Maybe I'm just trying to dig too deep into this, because it seems I can use
a combination of glsjob + start/EndTimes to achieve what I'd like after job
imports, but I'd rather not rewrite your gusage/statement/etc commands,
especially since they natively work with the db, and they will probably be a
heck of a lot more efficient than my perl ;-)

On 10/1/10 6:44 PM, "Scott Jackson" <scottmo at adaptivecomputing.com> wrote:

> Steve,
> It was introduced in gold-2.2. If you are using 2.1 you will not have gmkjob.
> The imported jobs will not show up against the old statement, they will show
> up against a current statement only. However, the jobs will be inserted into
> the job table with the specified StartTime and EndTime (while CreationTime and
> ModificationTime will be current dates). You have to realize that Gold tracks
> the historical states of all allocations and objects and you can't just insert
> debits into the past. This would disrupt and change every intervening state.
> This would have effected balances causing other jobs not to have been able to
> run, invalidating previously generated statements, etc. You would never see a
> bank inserting credits or debits into their past balance sheets. If a bank
> catches a mistake, they correct it in the present with a current refund or
> withdrawal.
> When you import the job data, it does not affect balances. If you wish to
> catch up for the past charges, the charges will occur in the present.
> Scott
> ----- Original Message -----
> From: "Steve Crusan" <scrusan at UR.Rochester.edu>
> To: gold-users at supercluster.org
> Sent: Friday, October 1, 2010 3:20:57 PM
> Subject: [gold-users] importing old job data into gold
> importing old job data into gold
> Quick question about importing old job data (from Maui logs) into Gold.
> I've noticed that it seems to be possible to do this using the goldsh shell
> (in the manual it shows gmkjob, but I cannot seem to find it; is it
> deprecated?): 
> goldsh Job Create JobId=PBS.$goldId.0 User=$netId Project=$account
> Machine=$machine Charge=$charge Processors=$procs StartTime=$startTime
> EndTime=$endTime WallDuration=$walltime
>> From there, it seems one can use gcharge to charge the job:
> gcharge -p $account -u $netId -m $machine -P $procs -t $walltime -j $goldId -d
> "charge imported from old data" -s $startTime -e $endTime
> Now, provided that I properly fill in the correct attributes, and use the
> proper gold job id, would doing the above operations properly keep the
> timestamps of the jobs? Meaning that if I have job logs from September 2009,
> if I import them and use their epoch time (do all the conversions, etc), will
> the gstatement commands and other querying operations reflect the proper
> dates/times? 
> Ex: 
> gstatement -s 2009-09-10 -e 2009-12-25
> Will the imported jobs from the Maui logs properly show up within that
> statement, and NOT show up for a rather recent query?
> Thanks! 
> ----------------------
> Steve Crusan 
> System Administrator
> Center for Research Computing
> _______________________________________________
> gold-users mailing list
> gold-users at supercluster.org
> http://www.supercluster.org/mailman/listinfo/gold-users
> _______________________________________________
> gold-users mailing list
> gold-users at supercluster.org
> http://www.supercluster.org/mailman/listinfo/gold-users

Steve Crusan
System Administrator
Center for Research Computing
University of Rochester

More information about the gold-users mailing list