[gold-users] Gold system getting bogged down

Hazlewood, Victor Gene vhazlewo at utk.edu
Thu Dec 3 15:37:15 MST 2009


Hi!

 

Got a gold issue...  We have a Cray XT5 system which is starting to run
about 7000 jobs a day.  I post process the Torque accounting log each
night and create gcharge records for posting to gold.  With 7,000
gcharge records and each taking approximately 5-8 seconds each it is
taking 15 or more hours to process the accounting records.  (bad!).  The
gold database and gold daemon are on an infrastructure server and the
gcharge is done on the Cray directly.

 

What can be done to the post processing process, gold daemon or the gold
database to speed up the processing of these gcharge records? I wish
these gcharge commands would take less than a second.  Maybe they have
always processed in more than 1 second, maybe even more but when we had
1000-2000 or less jobs is wasn't really noticed.

 

These records in Gold are being used to copy information to another
postgres database using gold commands to collect the info and then
subsequently these records are posted to an off site database.  With
these three things going on (charge posting, processing charges into the
other database, and sending the charges offsite) gold is doing quite a
bit and seems to get bogged down with these requests.   Also gold is
getting queried at the job submission to make sure there is time
available in the account.  All this is putting pressure on gold and it
seems to be slowing significantly (or noticeably I guess) under the
pressure.

 

Thanks for any help you can provide.

 

-Victor

 

Victor Hazlewood, CISSP

Senior HPC Systems Analyst

National Institute for Computational Science

University of Tennessee

http://www.nics.tennessee.edu/ <http://www.nics.utk.edu/> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/gold-users/attachments/20091203/16eb8fdf/attachment.html 


More information about the gold-users mailing list