[gold-users] Re: bug in goldsh
Scott Jackson
scott at clusterresources.com
Thu Mar 8 15:14:00 MST 2007
Great,
I understand the issue now.
JobIds are strings, not integers, since many resource managers use
characters in their job ids. Since Gold is sorting on a string type
field, it is doing a lexicographic comparison rather than a numeric
comparison.
I guess you can either live with it, use elaborate means to define a
lexicographic query that will accomplish your needs, wrapper the command
you want, or change to JobId type in Gold to be understood as an
Integer.
If you really want to pursue the last route, I must warn you that I may
not have caught all of the ramifications of this action. Gold was
designed around the idea of JobId being a String. That said, you may
have some success following an approach that I just tried and seemed to
work for me.
There are at least two database references to JobId:
gold> Attribute Query Name==JobId
Object Name DataType PrimaryKey Required Fixed Values DefaultValue
Sequence Hidden Description
----------- ----- -------- ---------- -------- ----- ------ ------------
-------- ------ -----------
Transaction JobId String False False False
210 False Job Id
Job JobId String False True False
20 False Job Id
# Since Gold does not allow you to change Attribute DataTypes (for good
reason:), you will have to do this at the database level. You will have
to try to find a way to get an sql prompt to run something similar to
the following: (I wrote sqlite for you in $GOLDHOME/sbin for this
purpose but I cannot vouch as to whether this will work for you or not
-- do this at your own risk).
gold=# update g_attribute set g_data_type='Integer' where
g_name='JobId';
UPDATE 2
# Now, here is an example of how I went about changing the g_job_id
column in the g_job table to have an integer column type instead of
varchar. I highly doubt that sqlite even supports such functions.
gold=# alter table g_job rename column g_job_id to g_job_id_string;
ALTER TABLE
gold=# alter table g_job add column g_job_id integer;
ALTER TABLE
gold=# update g_job set g_job_id=g_job_id_string::int;
UPDATE 2
After doing this, you would have to do the same steps for the
g_transaction table.
You will also probably need to do this in the log tables for both as
well, g_job_log and g_transaction_log.
A second possibility would be to backup the database, recreate a new
gold database, mess with gold.sql to change all 4 references of g_job_id
to type integer, and import whatever data manually that you want to
carry over. I think you will also need to Change the Attribute for JobId
to be of DataType Integer.
A third, and probably better alternative, could be to backup the
database to a file, tweak the file to have the 4 new values, modify the
g_job info to take the quotes off the job_id field, change the Attribute
JobId DataTypes to Integer, drop and recreate the database, import the
new file, etc.
If this makes you nervous, then I would recommend living with the way it
is:)
Good luck,
Scott
On Thu, 2007-03-08 at 15:20 -0500, Kevin Van Workum wrote:
> Ok, I added TRACE to the logger. Attached are the gold and goldd log
> files after I ran:
>
> Job Query JobId>=300 User==vanw Show:=Id,JobId,User
>
> The output missed a bunch of JobId's in the range [1000,2999],
> presumably because the first 3 digits are less than 300:
>
> Id JobId User
> ---- ----- ----
> 214 415 vanw
> 215 416 vanw
> 216 417 vanw
> 217 418 vanw
> 218 420 vanw
> 1899 3445 vanw
> 1900 3446 vanw
> 1901 3447 vanw
> 1902 3448 vanw
> 1987 3543 vanw
> 1991 3547 vanw
> 1992 3548 vanw
> 1993 3549 vanw
> 1994 3550 vanw
> 1996 3552 vanw
> 2025 3582 vanw
>
>
>
> On 3/8/07, Scott Jackson <scott at clusterresources.com> wrote:
> Kevin,
>
> There are some parts of this picture that are still missing to
> me.
>
> When I do this (with Postgresql -- but it goes through the
> exact same
> Gold statements and should still enter these log entries
> regardless of
> db), I get lines like:
>
> 2007-03-08 12:47:14.217 DEBUG Gold::Proxy::prepare Cooked
> request:
> (Query, scottmo, [(Job, , )], [(Id, , , ), (JobId, , , ),
> (User, , , )],
> [], [(JobId, 45, GE, , , , ), (User, scottmo, EQ, And, , , )],
> [], [],
> 0, 0, 100000, Gold::Database=HASH(0x9d37468))
> 2007-03-08 12:47:14.217 TRACE Gold::Proxy::execute invoked
> with
> arguments: ()
> 2007-03-08 12:47:14.217 TRACE Gold::Bank::execute invoked
> with
> arguments: (Gold::Proxy=HASH(0x9d3709c))
> 2007-03-08 12:47:14.218 DEBUG Gold::Bank::query invoked with
> arguments:
> (Gold::Request=HASH(0x9d453a0), 1343)
> 2007-03-08 12:47:14.218 DEBUG Gold::Base::query invoked with
> arguments:
> (Gold::Request=HASH(0x9d453a0))
> 2007-03-08 12:47:14.219 TRACE Gold::Database::select invoked
> with
> arguments: (chunkSize => 100000, options => ARRAY(0x98f6c88),
> selections
> => ARRAY(0x98f6c10), objects => ARRAY(0x98f6b20), chunkNum =>
> 1,
> conditions => ARRAY(0x98f6c4c))
> 2007-03-08 12:47:14.230 TRACE
> Gold::Database::buildWhere invoked with
> arguments: (objects => ARRAY(0x93f2538), conditions =>
> ARRAY(0x93f2640))
> 2007-03-08 12:47:14.231 DEBUG Gold::Database::select SQL
> Query: SELECT
> g_id,g_job_id,g_user FROM g_job WHERE ( g_job_id>='45' AND
> g_user='scottmo' ) AND g_deleted!='True' LIMIT 100000 OFFSET 0
> 2007-03-08 12:47: 14.232 DEBUG Gold::Database::select SQL
> Results: 6|45|
> scottmo
> 7|46|scottmo
> 8|47|scottmo
> 9|48|scottmo
> 10|49|scottmo
> 11|50|scottmo
> 12|51|scottmo
> 13|52|scottmo
> 14|53|scottmo
> 15|54|scottmo
> 16|59|scottmo
> 17|60|scottmo
> 19|62|scottmo
> 20|63|scottmo
> 18|61|scottmo
> 21|64|scottmo
> 22|65|scottmo
> 23|PBS.1234.0|scottmo
> 2007-03-08 12:47:14.234 TRACE Gold::Response::new invoked
> with
> arguments: ()
> 2007-03-08 12:47: 14.234 TRACE
> Gold::Response::success invoked with
> arguments: (18, ARRAY(0x98f6d18))
> 2007-03-08 12:47:14.236 TRACE Gold::Reply::new invoked with
> arguments:
> (connection => IO::Socket::INET=GLOB(0x9d45da8))
> 2007-03-08 12:47:14.237 TRACE Gold::Reply::sendChunk invoked
> with
> arguments: (Gold::Chunk=HASH(0x9d90058))
> 2007-03-08 12:47:14.237 TRACE
> Gold::Reply::marshallChunk invoked with
> arguments: (Gold::Chunk=HASH(0x9d90058))
> 2007-03-08 12:47:14.238 DEBUG Gold::Reply::sendChunk Writing
> reply
> header (HTTP/1.1 200 OK^M
> Content-Type: text/xml; charset="utf-8"^M
> Transfer-Encoding: chunked^M
> ).
> 2007-03-08 12:47:14.239 INFO Gold::Reply::sendChunk Writing
> reply
> payload (1262, <?xml version="1.0" encoding="UTF-8"?>
> <Envelope><Body><Response
> actor="scottmo"><Status><Value>Success</Value><Code>000</Code></Status><Count>18</Count><Data><Job><Id>6</Id><JobId>45</JobId><User>scottmo</User></Job><Job><Id>7</Id><JobId>46</JobId><User>scottmo</User></Job><Job><Id>8</Id><JobId>47</JobId><User>scottmo</User></Job><Job><Id>9</Id><JobId>48</JobId><User>scottmo</User></Job><Job><Id>10</Id><JobId>49</JobId><User>scottmo</User></Job><Job><Id>11</Id><JobId>50</JobId><User>scottmo</User></Job><Job><Id>12</Id><JobId>51</JobId><User>scottmo</User></Job><Job><Id>13</Id><JobId>52</JobId><User>scottmo</User></Job><Job><Id>14</Id><JobId>53</JobId><User>scottmo</User></Job><Job><Id>15</Id><JobId>54</JobId><User>scottmo</User></Job><Job><Id>16</Id><JobId>59</JobId><User>scottmo</User></Job><Job><Id>17</Id><JobId>60</JobId><User>scottmo</User></Job><Job><Id>19</Id><JobId>62</JobId><User>scottmo</User></Job><Job><Id>20</Id><JobId>63</JobId><User>scottmo</User></Job><Job><Id>18</Id><JobId>61</JobId><User>scottmo</User></Job><Job><Id>21</Id><JobId>64</JobId><User>scottmo</User></Job><Job><Id>22</Id><JobId>65</JobId><User>scottmo</User></Job><Job><Id>23</Id><JobId> PBS.1234.0</JobId><User>scottmo</User></Job></Data></Response></Body></Envelope>
> ).
>
> You seem to be missing all of the TRACE and DEBUG statements.
>
> Ohh, I see it now. Your first line of the Log4perl
> configuration should
> say:
>
> log4perl.logger = TRACE, Log, Screen
>
> I am guessing you must have changed this at some point.
>
> Can you set this back up, restart goldd, run your command
> again and send
> me the output?
>
> Thanks,
>
> Scott
>
> On Wed, 2007-03-07 at 18:19 -0500, Kevin Van Workum wrote:
> > Scott,
> >
> > I had attached all the output that was generated after I
> issued the
> > Query. Perhaps you should know I'm using the SQLite
> interface, but I
> > plan to go to mysql prior to production. Or maybe I should
> adjust my
> > Log4perl configuration to get the logging you need to see?
> >
> > # Log4perl configuration
> > log4perl.logger = INFO, Log, Screen
> > log4perl.appender.Log.Threshold = TRACE
> > log4perl.appender.Screen.Threshold = FATAL
> > #log4perl.logger.Message = TRACE, Screen
> > #log4perl.appender.Log = Log::Log4perl::Appender::File
> > log4perl.appender.Log = Log::Dispatch::FileRotate
> > log4perl.appender.Log.filename
> = /usr/local/gold-2.1.1.1/log/goldd.log
> > log4perl.appender.Log.mode = append
> > log4perl.appender.Log.size = 10000000
> > log4perl.appender.Log.max = 7
> > log4perl.appender.Log.layout = PatternLayout
> > log4perl.appender.Log.layout.ConversionPattern = %
> d{yyyy-MM-dd
> > HH:mm:ss.SSS} %-5p %M %m%n
> >
> >
> > On 3/7/07, Scott Jackson <scott at clusterresources.com> wrote:
> > Kevin,
> >
> > Thanks.
> >
> > I actually need to see a much larger portion of the
> goldd.log
> > that shows
> > what the sql query is that it is using and the
> results.
> >
> > Scott
> >
> > On Tue, 2007-03-06 at 20:39 -0500, Kevin Van Workum
> wrote:
> > > Same thing happens without &&. I've attached are
> the log
> > files.
> > >
> > > Kevin
> > >
> > > On 3/6/07, Scott Jackson <
> scott at clusterresources.com> wrote:
> > > That IS strange. Could you send me the
> gold.log and
> > goldd.log?
> > > Also, can
> > > you see what happens if you do not use the
> &&'s?
> > >
> > > Thanks,
> > >
> > > Scott
> > >
> > > On Tue, 2007-03-06 at 13:33 -0500, Kevin
> Van Workum
> > wrote:
> > > > Forgot to attach the following. The 344?
> JobId's
> > aren't
> > > displayed:
> > > >
> > > > gold> Job Query JobId>=345 && User==vanw
> > Show:=Id,JobId,User
> > > > Id JobId User
> > > > --- ----- ----
> > > > 214 415 vanw
> > > > 215 416 vanw
> > > > 216 417 vanw
> > > > 217 418 vanw
> > > > 218 420 vanw
> > > >
> > > > On 3/6/07, Scott Jackson
> > <scott at clusterresources.com> wrote:
> > > > Kevin,
> > > >
> > > > How do you figure that?
> > > >
> > > > All of the JobIds listed below
> are greater
> > than or
> > > equal to
> > > > 344.
> > > >
> > > > Scott
> > > >
> > > > On Tue, 2007-03-06 at 12:34
> -0500, Kevin
> > Van Workum
> > > wrote:
> > > > > It looks like goldsh is only
> using the
> > first 3
> > > digits of the
> > > > JobId.
> > > > >
> > > > > gold> Job Query JobId>=344 &&
> > User==vanw
> > > > Show:=Id,JobId,User
> > > > > Id JobId User
> > > > > ---- ----- ----
> > > > > 214 415 vanw
> > > > > 215 416 vanw
> > > > > 216 417 vanw
> > > > > 217 418 vanw
> > > > > 218 420 vanw
> > > > > 1899 3445 vanw
> > > > > 1900 3446 vanw
> > > > > 1901 3447 vanw
> > > > > 1902 3448 vanw
> > > > >
> > > > >
> > > > > --
> > > > > Kevin Van Workum, Ph.D.
> > > > > Vice President
> > > > > Senior System Administrator
> > > > > www.clusterondemand.com
> > > > > ONLINE COMPUTER CLUSTERS
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Kevin Van Workum, Ph.D.
> > > > Vice President
> > > > Senior System Administrator
> > > > www.clusterondemand.com
> > > > ONLINE COMPUTER CLUSTERS
> > >
> > >
> >
> >
> >
> >
> > --
> > Kevin Van Workum, Ph.D.
> > Vice President
> > Senior System Administrator
> > www.clusterondemand.com
> > ONLINE COMPUTER CLUSTERS
>
>
>
>
> --
> Kevin Van Workum, Ph.D.
> Vice President
> Senior System Administrator
> www.clusterondemand.com
> ONLINE COMPUTER CLUSTERS
More information about the gold-users
mailing list