[gold-users] moab-gold binding performance issue

Hu, Zongjun zhu at med.miami.edu
Thu Jun 4 13:21:42 MDT 2009


We use Gold-2.1.6 as allocation manager for Moab-5.2.0. We are trying
to move this configuration to production. We got several issues and
need help.

1. This binding is working as job reservation @ job start. Most of the
time, this mode works perfectly. If a job is accepted to start, a new
job will be created in gold and account balance will be deducted
according to requested cpu hours. If this job runs and finishes without
problem, this job will be charged finally according to real usage. The
previous charge in gold will be returned to account balance. However,
we got an interesting problem. We found some jobs are accepted to
start, and then moved to compute nodes. For some reason, these jobs do
not start successfully on compute nodes. They are then rejected and
moved back to blocked/waiting list. After a while, these jobs will be
evaluated again and repeat these steps. In this situation, multiple
jobs will be created in gold and account balance will be deducted
multiple times (to make things worse, user usually request much more
than they need). When these jobs are finally finished or canceled, gold
will only change the last job created to 'Charge' stage and put back
only the last deduction back to account. All previous balance
jobs/deductions created will stay in gold and the reserved balance
won't be restored. After a while, gold will have a huge amount of
'Reserved balance' even all jobs are completed. Can you give us
instruction to fix this problem and release all those unnecessary
reserved balance?

2. Sometimes, we have lot of small jobs (finish in 1 minute). Because
for each job, Moab has to contact gold server to reserve and then
charge job when it finishes. Those small jobs make moab repeat these
steps frequently and moab server is very busy. This slow down reponse
to user reqeust a lot and sometimes time out in user request. Is there
a way to speed up gold job processing? If not, can we configure Moab-
gold binding to Job charge @ job end time? Therefore we can save at
least half of the processing time. We did not find guidance in Moab or
Gold documents for this configuration.


Zongjun Hu
University of Miami
Center for Computational Science
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/gold-users/attachments/20090604/3278ae20/attachment.html 

More information about the gold-users mailing list