[gold-users] importing old job data into gold

Hazlewood, Victor Gene vhazlewo at utk.edu
Wed Oct 6 07:52:22 MDT 2010

I don't know if this helps, but...

If all you are doing is wanting to have the job information in a table
and not use any of the other gold features (which I don't know if that
is what you want or not) why don't use just create a new Job table in
the Gold database by creating a new Gold Object with all the same
attributes as the Gold Job Object and then you could make your own load
program or modify the gmkjob to update your new Object instead of the
real one.

Or you could do it all with SQL commands of the database you are using
for Gold.   I have lots of extensions to Gold in our database both
created with Gold Objects and created by SQL commands.  I like creating
Gold Objects instead of using SQL commands because you get features like
when you load data it automagically gets creation times and
modifications times, etc.


-----Original Message-----
From: gold-users-bounces at supercluster.org
[mailto:gold-users-bounces at supercluster.org] On Behalf Of Steve Crusan
Sent: Tuesday, October 05, 2010 4:39 PM
To: Gold Users Mailing List
Subject: Re: [gold-users] importing old job data into gold

Thanks for the response Scott. Gold is essentially a bank.

I guess I need to clarify my statement a bit more then. I don't care
applying old charges to people's balances, but more about starting at a
baseline with our old Maui logs. At this point I wouldn't care if after
running a massive script if every user/project/etc was millions of hours
debt. I just want the historical usage numbers, with a good way to query
them by time periods. Basically, my place would be to actually run gold
our cluster AFTER we do an import, change the balances, etc...

We're not at the point yet where we want to start using Gold as a
charging/CPU hours banking system, but more internally so we have easy
query numbers. I see an evolution coming soon, but let's not get ahead

Let's say I create a fresh instance of Gold, import any
users/groups/setting/etc, but do not have any job data. Now, just
the Maui logs via a script would produce the behavior you specified
(i.e job records get debited that day). Would there be any way of
the times in the gold database itself to reflect the actual debit times?
guess it would be tricking the system if I changed all of the references
the right place, but would it be theoretically possible to change the
in the DB, so that some of the historical gold commands would reflect
actual job times?

Maybe I'm just trying to dig too deep into this, because it seems I can
a combination of glsjob + start/EndTimes to achieve what I'd like after
imports, but I'd rather not rewrite your gusage/statement/etc commands,
especially since they natively work with the db, and they will probably
be a
heck of a lot more efficient than my perl ;-)

On 10/1/10 6:44 PM, "Scott Jackson" <scottmo at adaptivecomputing.com>

> Steve,
> It was introduced in gold-2.2. If you are using 2.1 you will not have
> The imported jobs will not show up against the old statement, they
will show
> up against a current statement only. However, the jobs will be
inserted into
> the job table with the specified StartTime and EndTime (while
CreationTime and
> ModificationTime will be current dates). You have to realize that Gold
> the historical states of all allocations and objects and you can't
just insert
> debits into the past. This would disrupt and change every intervening
> This would have effected balances causing other jobs not to have been
able to
> run, invalidating previously generated statements, etc. You would
never see a
> bank inserting credits or debits into their past balance sheets. If a
> catches a mistake, they correct it in the present with a current
refund or
> withdrawal.
> When you import the job data, it does not affect balances. If you wish
> catch up for the past charges, the charges will occur in the present.
> Scott
> ----- Original Message -----
> From: "Steve Crusan" <scrusan at UR.Rochester.edu>
> To: gold-users at supercluster.org
> Sent: Friday, October 1, 2010 3:20:57 PM
> Subject: [gold-users] importing old job data into gold
> importing old job data into gold
> Quick question about importing old job data (from Maui logs) into
> I've noticed that it seems to be possible to do this using the goldsh
> (in the manual it shows gmkjob, but I cannot seem to find it; is it
> deprecated?): 
> goldsh Job Create JobId=PBS.$goldId.0 User=$netId Project=$account
> Machine=$machine Charge=$charge Processors=$procs StartTime=$startTime
> EndTime=$endTime WallDuration=$walltime
>> From there, it seems one can use gcharge to charge the job:
> gcharge -p $account -u $netId -m $machine -P $procs -t $walltime -j
$goldId -d
> "charge imported from old data" -s $startTime -e $endTime
> Now, provided that I properly fill in the correct attributes, and use
> proper gold job id, would doing the above operations properly keep the
> timestamps of the jobs? Meaning that if I have job logs from September
> if I import them and use their epoch time (do all the conversions,
etc), will
> the gstatement commands and other querying operations reflect the
> dates/times? 
> Ex: 
> gstatement -s 2009-09-10 -e 2009-12-25
> Will the imported jobs from the Maui logs properly show up within that
> statement, and NOT show up for a rather recent query?
> Thanks! 
> ----------------------
> Steve Crusan 
> System Administrator
> Center for Research Computing
> _______________________________________________
> gold-users mailing list
> gold-users at supercluster.org
> http://www.supercluster.org/mailman/listinfo/gold-users
> _______________________________________________
> gold-users mailing list
> gold-users at supercluster.org
> http://www.supercluster.org/mailman/listinfo/gold-users

Steve Crusan
System Administrator
Center for Research Computing
University of Rochester

gold-users mailing list
gold-users at supercluster.org

More information about the gold-users mailing list