[gold-users] historical data

Scott Jackson scott at clusterresources.com
Thu May 24 10:54:39 MDT 2007


That may be difficult to do. Gold functions very much like a bank. It's
various records are tightly intertwined. When you make a charge to Gold,
it adds records to the Transaction table for that time (such as Job
Charge, etc.), decreases the value in the Allocation table
(corresponding to the debit charge), adds or updates the Job record with
the charge amount, and other information. Additionally, all of the
affected Object journals (TransactionLog, JobLog, AllocationLog, ...)
are updated so that historical queries can be made against that
information (such as being able to ask what was the account balance for
an arbitrary time in the past, etc).

As you can imagine, making just a single small change in the past will
amount to an enormous set of changes to make things consistent forward
through time to the present. The Job record could be easily updated, but
the journal would be much harder to patch up. A charge transaction could
be added to the transaction log, but this would introduce a disconnect
between the transaction report and the historical balances as calculated
in the bank statement report. In order to patch up the historical state
of past balances, you would have to change every intermediate allocation
balance value moving from that time forward to the present.

This is why real banks would never let you do such a thing. Every thing
that happens is recorded. If a mistake occurs (like they charge you too
much or they forget to charge you for something) the correction is made
by fixing it in the present (a refund or debit annotated that it is to
fix a mistake in the past). You just can;t change the past state of the
bank. This would change all of the balance sheets that they had run up
to that time -- and given to management, shareholders, etc and render
them a falsehood.

If you just wanted to use Gold to maintain a simple balance and a record
of jobs and no more, then you COULD do this (you would ignore the
transaction commands, ignore the historical balance commands like
gstatement, etc). But you would lose the ability to get an accurate
report of the state or balance of the accounts in the past and the
ability to query the transactions. Gold tries to prevent you from using
it this way because it introduces internal inconsistencies, but in many
cases you can do this either via low level goldsh commands or by direct
table manipulation in SQL.

If it was not palatable to run the gcharges and have them show up as
current charges, then probably what I would do is to make a bunch of Job
records for the old LSF jobs via the goldsh command "Job Create ...".
There would be no corresponding Transaction entries or account balance
impacts. This should be fine since there is no Gold command that tries
to reconcile the Job table with other tables. It might be good to
annotate the jobs to say they were added post-mortem. If you needed
account balances to go down, you could also make corresponding lump
debits (with gwithdraw) to the affected accounts and annotate them
saying something like "Pre-Gold job charges".

Let me know how you would like to proceed and I will try to lend help in
how to do this.


On Thu, 2007-05-24 at 14:23 +0100, Tim Robinson wrote:
> Hi all
> I recently turned on gold accounting for our local HPC resource, and
> it
> is working well. 
> I then wanted to enter all historical data (from old LSF logs). I got
> the data into a suitable form (basically a very large number of
> gcharge
> lines) but because they all get the timestamp for today, rather than
> say
> "end time" then the historical data isn't really useful at all. For
> example, I can't query jobs over time periods in the GUI, etc, etc. 
> Any advice?
> Thanks
> Tim 

More information about the gold-users mailing list