[gold-users] Re: bug in goldsh

Kevin Van Workum vanw at tticluster.com
Thu Mar 8 16:02:09 MST 2007


Scott,

I suspected something like that was probably the culprit. If that is the
intended behavior, then I have no desire to change it. Thanks for looking
into it.

What I was really wanted to do is get a list of all my jobs in a particular
time frame, showing the WallDuration of each job.  glsjob doesn't work
because the StartTime and EndTime attributes of all my jobs are empty.
glstxn doesn't work either because WallDuration is not an attribute of a
Transaction. I just realized right now that the Name attribute of a
Transaction with Object==Job and Action==Charge  is the  gold_job_id of the
job. So I can use glstxn to get the first and last gold_job_id and then use
glsjob to get my list of Jobs.

Or maybe I can just use CreationTime as the discriminator in the Job Query,
like this:

gold> Job Query User==vanw CreationTime>=2007-02-21 CreationTime<=2007-02-21

or will the lexicographic comparison cause problems for me here also? A
couple quick tests indicate that it seems to work.

Kevin

On 3/8/07, Scott Jackson <scott at clusterresources.com> wrote:
>
> Great,
>
> I understand the issue now.
>
> JobIds are strings, not integers, since many resource managers use
> characters in their job ids. Since Gold is sorting on a string type
> field, it is doing a lexicographic comparison rather than a numeric
> comparison.
>
> I guess you can either live with it, use elaborate means to define a
> lexicographic query that will accomplish your needs, wrapper the command
> you want, or change to JobId type in Gold to be understood as an
> Integer.
>
> If you really want to pursue the last route, I must warn you that I may
> not have caught all of the ramifications of this action. Gold was
> designed around the idea of JobId being a String. That said, you may
> have some success following an approach that I just tried and seemed to
> work for me.
>
> There are at least two database references to JobId:
>
> gold> Attribute Query Name==JobId
> Object      Name  DataType PrimaryKey Required Fixed Values DefaultValue
> Sequence Hidden Description
> ----------- ----- -------- ---------- -------- ----- ------ ------------
> -------- ------ -----------
> Transaction JobId String   False      False    False
> 210      False  Job Id
> Job         JobId String   False      True     False
> 20       False  Job Id
>
> # Since Gold does not allow you to change Attribute DataTypes (for good
> reason:), you will have to do this at the database level. You will have
> to try to find a way to get an sql prompt to run something similar to
> the following: (I wrote sqlite for you in $GOLDHOME/sbin for this
> purpose but I cannot vouch as to whether this will work for you or not
> -- do this at your own risk).
>
> gold=# update g_attribute set g_data_type='Integer' where
> g_name='JobId';
> UPDATE 2
>
> # Now, here is an example of how I went about changing the g_job_id
> column in the g_job table to have an integer column type instead of
> varchar. I highly doubt that sqlite even supports such functions.
>
> gold=# alter table g_job rename column g_job_id to g_job_id_string;
> ALTER TABLE
> gold=# alter table g_job add column g_job_id integer;
> ALTER TABLE
> gold=# update g_job set g_job_id=g_job_id_string::int;
> UPDATE 2
>
> After doing this, you would have to do the same steps for the
> g_transaction table.
>
> You will also probably need to do this in the log tables for both as
> well, g_job_log and g_transaction_log.
>
>
> A second possibility would be to backup the database, recreate a new
> gold database, mess with gold.sql to change all 4 references of g_job_id
> to type integer, and import whatever data manually that you want to
> carry over. I think you will also need to Change the Attribute for JobId
> to be of DataType Integer.
>
> A third, and probably better alternative, could be to backup the
> database to a file, tweak the file to have the 4 new values, modify the
> g_job info to take the quotes off the job_id field, change the Attribute
> JobId DataTypes to Integer, drop and recreate the database, import the
> new file, etc.
>
> If this makes you nervous, then I would recommend living with the way it
> is:)
>
> Good luck,
>
> Scott
>
> On Thu, 2007-03-08 at 15:20 -0500, Kevin Van Workum wrote:
> > Ok, I added TRACE to the logger. Attached are the gold and goldd log
> > files after I ran:
> >
> > Job Query JobId>=300 User==vanw Show:=Id,JobId,User
> >
> > The output missed a bunch of JobId's in the range [1000,2999],
> > presumably because the first 3 digits are less than 300:
> >
> > Id   JobId User
> > ---- ----- ----
> > 214  415   vanw
> > 215  416   vanw
> > 216  417   vanw
> > 217  418   vanw
> > 218  420   vanw
> > 1899 3445  vanw
> > 1900 3446  vanw
> > 1901 3447  vanw
> > 1902 3448  vanw
> > 1987 3543  vanw
> > 1991 3547  vanw
> > 1992 3548  vanw
> > 1993 3549  vanw
> > 1994 3550  vanw
> > 1996 3552  vanw
> > 2025 3582  vanw
> >
> >
> >
> > On 3/8/07, Scott Jackson <scott at clusterresources.com> wrote:
> >         Kevin,
> >
> >         There are some parts of this picture that are still missing to
> >         me.
> >
> >         When I do this (with Postgresql -- but it goes through the
> >         exact same
> >         Gold statements and should still enter these log entries
> >         regardless of
> >         db), I get lines like:
> >
> >         2007-03-08 12:47:14.217 DEBUG Gold::Proxy::prepare  Cooked
> >         request:
> >         (Query, scottmo, [(Job, , )], [(Id, , , ), (JobId, , , ),
> >         (User, , , )],
> >         [], [(JobId, 45, GE, , , , ), (User, scottmo, EQ, And, , , )],
> >         [], [],
> >         0, 0, 100000, Gold::Database=HASH(0x9d37468))
> >         2007-03-08 12:47:14.217 TRACE Gold::Proxy::execute  invoked
> >         with
> >         arguments: ()
> >         2007-03-08 12:47:14.217 TRACE Gold::Bank::execute  invoked
> >         with
> >         arguments: (Gold::Proxy=HASH(0x9d3709c))
> >         2007-03-08 12:47:14.218 DEBUG Gold::Bank::query  invoked with
> >         arguments:
> >         (Gold::Request=HASH(0x9d453a0), 1343)
> >         2007-03-08 12:47:14.218 DEBUG Gold::Base::query  invoked with
> >         arguments:
> >         (Gold::Request=HASH(0x9d453a0))
> >         2007-03-08 12:47:14.219 TRACE Gold::Database::select  invoked
> >         with
> >         arguments: (chunkSize => 100000, options => ARRAY(0x98f6c88),
> >         selections
> >         => ARRAY(0x98f6c10), objects => ARRAY(0x98f6b20), chunkNum =>
> >         1,
> >         conditions => ARRAY(0x98f6c4c))
> >         2007-03-08 12:47:14.230 TRACE
> >         Gold::Database::buildWhere  invoked with
> >         arguments: (objects => ARRAY(0x93f2538), conditions =>
> >         ARRAY(0x93f2640))
> >         2007-03-08 12:47:14.231 DEBUG Gold::Database::select  SQL
> >         Query: SELECT
> >         g_id,g_job_id,g_user FROM g_job WHERE ( g_job_id>='45' AND
> >         g_user='scottmo' ) AND g_deleted!='True' LIMIT 100000 OFFSET 0
> >         2007-03-08 12:47: 14.232 DEBUG Gold::Database::select  SQL
> >         Results: 6|45|
> >         scottmo
> >         7|46|scottmo
> >         8|47|scottmo
> >         9|48|scottmo
> >         10|49|scottmo
> >         11|50|scottmo
> >         12|51|scottmo
> >         13|52|scottmo
> >         14|53|scottmo
> >         15|54|scottmo
> >         16|59|scottmo
> >         17|60|scottmo
> >         19|62|scottmo
> >         20|63|scottmo
> >         18|61|scottmo
> >         21|64|scottmo
> >         22|65|scottmo
> >         23|PBS.1234.0|scottmo
> >         2007-03-08 12:47:14.234 TRACE Gold::Response::new  invoked
> >         with
> >         arguments: ()
> >         2007-03-08 12:47: 14.234 TRACE
> >         Gold::Response::success  invoked with
> >         arguments: (18, ARRAY(0x98f6d18))
> >         2007-03-08 12:47:14.236 TRACE Gold::Reply::new  invoked with
> >         arguments:
> >         (connection => IO::Socket::INET=GLOB(0x9d45da8))
> >         2007-03-08 12:47:14.237 TRACE Gold::Reply::sendChunk  invoked
> >         with
> >         arguments: (Gold::Chunk=HASH(0x9d90058))
> >         2007-03-08 12:47:14.237 TRACE
> >         Gold::Reply::marshallChunk  invoked with
> >         arguments: (Gold::Chunk=HASH(0x9d90058))
> >         2007-03-08 12:47:14.238 DEBUG Gold::Reply::sendChunk  Writing
> >         reply
> >         header (HTTP/1.1 200 OK^M
> >         Content-Type: text/xml; charset="utf-8"^M
> >         Transfer-Encoding: chunked^M
> >         ).
> >         2007-03-08 12:47:14.239 INFO  Gold::Reply::sendChunk  Writing
> >         reply
> >         payload (1262, <?xml version="1.0" encoding="UTF-8"?>
> >         <Envelope><Body><Response
> >
> actor="scottmo"><Status><Value>Success</Value><Code>000</Code></Status><Count>18</Count><Data><Job><Id>6</Id><JobId>45</JobId><User>scottmo</User></Job><Job><Id>7</Id><JobId>46</JobId><User>scottmo</User></Job><Job><Id>8</Id><JobId>47</JobId><User>scottmo</User></Job><Job><Id>9</Id><JobId>48</JobId><User>scottmo</User></Job><Job><Id>10</Id><JobId>49</JobId><User>scottmo</User></Job><Job><Id>11</Id><JobId>50</JobId><User>scottmo</User></Job><Job><Id>12</Id><JobId>51</JobId><User>scottmo</User></Job><Job><Id>13</Id><JobId>52</JobId><User>scottmo</User></Job><Job><Id>14</Id><JobId>53</JobId><User>scottmo</User></Job><Job><Id>15</Id><JobId>54</JobId><User>scottmo</User></Job><Job><Id>16</Id><JobId>59</JobId><User>scottmo</User></Job><Job><Id>17</Id><JobId>60</JobId><User>scottmo</User></Job><Job><Id>19</Id><JobId>62</JobId><User>scottmo</User></Job><Job><Id>20</Id><JobId>63</JobId><User>scottmo</User></Job><Job><Id>18</Id><JobId>61</JobId><User>scottmo</User></Job><Job><Id>21</Id><JobId>64</JobId><User>scottmo</User></Job><Job><Id>22</Id><JobId>65</JobId><User>scottmo</User></Job><Job><Id>23</Id><JobId>
> PBS.1234.0
> </JobId><User>scottmo</User></Job></Data></Response></Body></Envelope>
> >         ).
> >
> >         You seem to be missing all of the TRACE and DEBUG statements.
> >
> >         Ohh, I see it now. Your first line of the Log4perl
> >         configuration should
> >         say:
> >
> >         log4perl.logger = TRACE, Log, Screen
> >
> >         I am guessing you must have changed this at some point.
> >
> >         Can you set this back up, restart goldd, run your command
> >         again and send
> >         me the output?
> >
> >         Thanks,
> >
> >         Scott
> >
> >         On Wed, 2007-03-07 at 18:19 -0500, Kevin Van Workum wrote:
> >         > Scott,
> >         >
> >         > I had attached all the output that was generated after I
> >         issued the
> >         > Query. Perhaps you should know I'm using the SQLite
> >         interface, but I
> >         > plan to go to mysql prior to production. Or maybe I should
> >         adjust my
> >         > Log4perl configuration to get the logging you need to see?
> >         >
> >         > # Log4perl configuration
> >         > log4perl.logger = INFO, Log, Screen
> >         > log4perl.appender.Log.Threshold = TRACE
> >         > log4perl.appender.Screen.Threshold = FATAL
> >         > #log4perl.logger.Message = TRACE, Screen
> >         > #log4perl.appender.Log = Log::Log4perl::Appender::File
> >         > log4perl.appender.Log = Log::Dispatch::FileRotate
> >         > log4perl.appender.Log.filename
> >         = /usr/local/gold-2.1.1.1/log/goldd.log
> >         > log4perl.appender.Log.mode = append
> >         > log4perl.appender.Log.size = 10000000
> >         > log4perl.appender.Log.max = 7
> >         > log4perl.appender.Log.layout = PatternLayout
> >         > log4perl.appender.Log.layout.ConversionPattern = %
> >         d{yyyy-MM-dd
> >         > HH:mm:ss.SSS} %-5p %M  %m%n
> >         >
> >         >
> >         > On 3/7/07, Scott Jackson <scott at clusterresources.com> wrote:
> >         >         Kevin,
> >         >
> >         >         Thanks.
> >         >
> >         >         I actually need to see a much larger portion of the
> >         goldd.log
> >         >         that shows
> >         >         what the sql query is that it is using and the
> >         results.
> >         >
> >         >         Scott
> >         >
> >         >         On Tue, 2007-03-06 at 20:39 -0500, Kevin Van Workum
> >         wrote:
> >         >         > Same thing happens without &&. I've attached are
> >         the log
> >         >         files.
> >         >         >
> >         >         > Kevin
> >         >         >
> >         >         > On 3/6/07, Scott Jackson <
> >         scott at clusterresources.com> wrote:
> >         >         >         That IS strange. Could you send me the
> >         gold.log and
> >         >         goldd.log?
> >         >         >         Also, can
> >         >         >         you see what happens if you do not use the
> >         &&'s?
> >         >         >
> >         >         >         Thanks,
> >         >         >
> >         >         >         Scott
> >         >         >
> >         >         >         On Tue, 2007-03-06 at 13:33 -0500, Kevin
> >         Van Workum
> >         >         wrote:
> >         >         >         > Forgot to attach the following. The 344?
> >         JobId's
> >         >         aren't
> >         >         >         displayed:
> >         >         >         >
> >         >         >         > gold> Job Query JobId>=345 && User==vanw
> >         >         Show:=Id,JobId,User
> >         >         >         > Id  JobId User
> >         >         >         > --- ----- ----
> >         >         >         > 214 415   vanw
> >         >         >         > 215 416   vanw
> >         >         >         > 216 417   vanw
> >         >         >         > 217 418   vanw
> >         >         >         > 218 420   vanw
> >         >         >         >
> >         >         >         > On 3/6/07, Scott Jackson
> >         >         <scott at clusterresources.com> wrote:
> >         >         >         >         Kevin,
> >         >         >         >
> >         >         >         >         How do you figure that?
> >         >         >         >
> >         >         >         >         All of the JobIds listed below
> >         are greater
> >         >         than or
> >         >         >         equal to
> >         >         >         >         344.
> >         >         >         >
> >         >         >         >         Scott
> >         >         >         >
> >         >         >         >         On Tue, 2007-03-06 at 12:34
> >         -0500, Kevin
> >         >         Van Workum
> >         >         >         wrote:
> >         >         >         >         > It looks like goldsh is only
> >         using the
> >         >         first 3
> >         >         >         digits of the
> >         >         >         >         JobId.
> >         >         >         >         >
> >         >         >         >         > gold> Job Query JobId>=344 &&
> >         >         User==vanw
> >         >         >         >         Show:=Id,JobId,User
> >         >         >         >         > Id   JobId User
> >         >         >         >         > ---- ----- ----
> >         >         >         >         > 214  415   vanw
> >         >         >         >         > 215  416   vanw
> >         >         >         >         > 216  417   vanw
> >         >         >         >         > 217  418   vanw
> >         >         >         >         > 218  420   vanw
> >         >         >         >         > 1899 3445  vanw
> >         >         >         >         > 1900 3446  vanw
> >         >         >         >         > 1901 3447  vanw
> >         >         >         >         > 1902 3448  vanw
> >         >         >         >         >
> >         >         >         >         >
> >         >         >         >         > --
> >         >         >         >         > Kevin Van Workum, Ph.D.
> >         >         >         >         > Vice President
> >         >         >         >         > Senior System Administrator
> >         >         >         >         > www.clusterondemand.com
> >         >         >         >         > ONLINE COMPUTER CLUSTERS
> >         >         >         >
> >         >         >         >
> >         >         >         >
> >         >         >         >
> >         >         >         > --
> >         >         >         > Kevin Van Workum, Ph.D.
> >         >         >         > Vice President
> >         >         >         > Senior System Administrator
> >         >         >         > www.clusterondemand.com
> >         >         >         > ONLINE COMPUTER CLUSTERS
> >         >         >
> >         >         >
> >         >
> >         >
> >         >
> >         >
> >         > --
> >         > Kevin Van Workum, Ph.D.
> >         > Vice President
> >         > Senior System Administrator
> >         > www.clusterondemand.com
> >         > ONLINE COMPUTER CLUSTERS
> >
> >
> >
> >
> > --
> > Kevin Van Workum, Ph.D.
> > Vice President
> > Senior System Administrator
> > www.clusterondemand.com
> > ONLINE COMPUTER CLUSTERS
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/gold-users/attachments/20070308/3179071f/attachment-0001.html


More information about the gold-users mailing list