[gold-users] Re: bug in goldsh
Kevin Van Workum
vanw at tticluster.com
Thu Mar 8 16:02:09 MST 2007
Scott,
I suspected something like that was probably the culprit. If that is the
intended behavior, then I have no desire to change it. Thanks for looking
into it.
What I was really wanted to do is get a list of all my jobs in a particular
time frame, showing the WallDuration of each job. glsjob doesn't work
because the StartTime and EndTime attributes of all my jobs are empty.
glstxn doesn't work either because WallDuration is not an attribute of a
Transaction. I just realized right now that the Name attribute of a
Transaction with Object==Job and Action==Charge is the gold_job_id of the
job. So I can use glstxn to get the first and last gold_job_id and then use
glsjob to get my list of Jobs.
Or maybe I can just use CreationTime as the discriminator in the Job Query,
like this:
gold> Job Query User==vanw CreationTime>=2007-02-21 CreationTime<=2007-02-21
or will the lexicographic comparison cause problems for me here also? A
couple quick tests indicate that it seems to work.
Kevin
On 3/8/07, Scott Jackson <scott at clusterresources.com> wrote:
>
> Great,
>
> I understand the issue now.
>
> JobIds are strings, not integers, since many resource managers use
> characters in their job ids. Since Gold is sorting on a string type
> field, it is doing a lexicographic comparison rather than a numeric
> comparison.
>
> I guess you can either live with it, use elaborate means to define a
> lexicographic query that will accomplish your needs, wrapper the command
> you want, or change to JobId type in Gold to be understood as an
> Integer.
>
> If you really want to pursue the last route, I must warn you that I may
> not have caught all of the ramifications of this action. Gold was
> designed around the idea of JobId being a String. That said, you may
> have some success following an approach that I just tried and seemed to
> work for me.
>
> There are at least two database references to JobId:
>
> gold> Attribute Query Name==JobId
> Object Name DataType PrimaryKey Required Fixed Values DefaultValue
> Sequence Hidden Description
> ----------- ----- -------- ---------- -------- ----- ------ ------------
> -------- ------ -----------
> Transaction JobId String False False False
> 210 False Job Id
> Job JobId String False True False
> 20 False Job Id
>
> # Since Gold does not allow you to change Attribute DataTypes (for good
> reason:), you will have to do this at the database level. You will have
> to try to find a way to get an sql prompt to run something similar to
> the following: (I wrote sqlite for you in $GOLDHOME/sbin for this
> purpose but I cannot vouch as to whether this will work for you or not
> -- do this at your own risk).
>
> gold=# update g_attribute set g_data_type='Integer' where
> g_name='JobId';
> UPDATE 2
>
> # Now, here is an example of how I went about changing the g_job_id
> column in the g_job table to have an integer column type instead of
> varchar. I highly doubt that sqlite even supports such functions.
>
> gold=# alter table g_job rename column g_job_id to g_job_id_string;
> ALTER TABLE
> gold=# alter table g_job add column g_job_id integer;
> ALTER TABLE
> gold=# update g_job set g_job_id=g_job_id_string::int;
> UPDATE 2
>
> After doing this, you would have to do the same steps for the
> g_transaction table.
>
> You will also probably need to do this in the log tables for both as
> well, g_job_log and g_transaction_log.
>
>
> A second possibility would be to backup the database, recreate a new
> gold database, mess with gold.sql to change all 4 references of g_job_id
> to type integer, and import whatever data manually that you want to
> carry over. I think you will also need to Change the Attribute for JobId
> to be of DataType Integer.
>
> A third, and probably better alternative, could be to backup the
> database to a file, tweak the file to have the 4 new values, modify the
> g_job info to take the quotes off the job_id field, change the Attribute
> JobId DataTypes to Integer, drop and recreate the database, import the
> new file, etc.
>
> If this makes you nervous, then I would recommend living with the way it
> is:)
>
> Good luck,
>
> Scott
>
> On Thu, 2007-03-08 at 15:20 -0500, Kevin Van Workum wrote:
> > Ok, I added TRACE to the logger. Attached are the gold and goldd log
> > files after I ran:
> >
> > Job Query JobId>=300 User==vanw Show:=Id,JobId,User
> >
> > The output missed a bunch of JobId's in the range [1000,2999],
> > presumably because the first 3 digits are less than 300:
> >
> > Id JobId User
> > ---- ----- ----
> > 214 415 vanw
> > 215 416 vanw
> > 216 417 vanw
> > 217 418 vanw
> > 218 420 vanw
> > 1899 3445 vanw
> > 1900 3446 vanw
> > 1901 3447 vanw
> > 1902 3448 vanw
> > 1987 3543 vanw
> > 1991 3547 vanw
> > 1992 3548 vanw
> > 1993 3549 vanw
> > 1994 3550 vanw
> > 1996 3552 vanw
> > 2025 3582 vanw
> >
> >
> >
> > On 3/8/07, Scott Jackson <scott at clusterresources.com> wrote:
> > Kevin,
> >
> > There are some parts of this picture that are still missing to
> > me.
> >
> > When I do this (with Postgresql -- but it goes through the
> > exact same
> > Gold statements and should still enter these log entries
> > regardless of
> > db), I get lines like:
> >
> > 2007-03-08 12:47:14.217 DEBUG Gold::Proxy::prepare Cooked
> > request:
> > (Query, scottmo, [(Job, , )], [(Id, , , ), (JobId, , , ),
> > (User, , , )],
> > [], [(JobId, 45, GE, , , , ), (User, scottmo, EQ, And, , , )],
> > [], [],
> > 0, 0, 100000, Gold::Database=HASH(0x9d37468))
> > 2007-03-08 12:47:14.217 TRACE Gold::Proxy::execute invoked
> > with
> > arguments: ()
> > 2007-03-08 12:47:14.217 TRACE Gold::Bank::execute invoked
> > with
> > arguments: (Gold::Proxy=HASH(0x9d3709c))
> > 2007-03-08 12:47:14.218 DEBUG Gold::Bank::query invoked with
> > arguments:
> > (Gold::Request=HASH(0x9d453a0), 1343)
> > 2007-03-08 12:47:14.218 DEBUG Gold::Base::query invoked with
> > arguments:
> > (Gold::Request=HASH(0x9d453a0))
> > 2007-03-08 12:47:14.219 TRACE Gold::Database::select invoked
> > with
> > arguments: (chunkSize => 100000, options => ARRAY(0x98f6c88),
> > selections
> > => ARRAY(0x98f6c10), objects => ARRAY(0x98f6b20), chunkNum =>
> > 1,
> > conditions => ARRAY(0x98f6c4c))
> > 2007-03-08 12:47:14.230 TRACE
> > Gold::Database::buildWhere invoked with
> > arguments: (objects => ARRAY(0x93f2538), conditions =>
> > ARRAY(0x93f2640))
> > 2007-03-08 12:47:14.231 DEBUG Gold::Database::select SQL
> > Query: SELECT
> > g_id,g_job_id,g_user FROM g_job WHERE ( g_job_id>='45' AND
> > g_user='scottmo' ) AND g_deleted!='True' LIMIT 100000 OFFSET 0
> > 2007-03-08 12:47: 14.232 DEBUG Gold::Database::select SQL
> > Results: 6|45|
> > scottmo
> > 7|46|scottmo
> > 8|47|scottmo
> > 9|48|scottmo
> > 10|49|scottmo
> > 11|50|scottmo
> > 12|51|scottmo
> > 13|52|scottmo
> > 14|53|scottmo
> > 15|54|scottmo
> > 16|59|scottmo
> > 17|60|scottmo
> > 19|62|scottmo
> > 20|63|scottmo
> > 18|61|scottmo
> > 21|64|scottmo
> > 22|65|scottmo
> > 23|PBS.1234.0|scottmo
> > 2007-03-08 12:47:14.234 TRACE Gold::Response::new invoked
> > with
> > arguments: ()
> > 2007-03-08 12:47: 14.234 TRACE
> > Gold::Response::success invoked with
> > arguments: (18, ARRAY(0x98f6d18))
> > 2007-03-08 12:47:14.236 TRACE Gold::Reply::new invoked with
> > arguments:
> > (connection => IO::Socket::INET=GLOB(0x9d45da8))
> > 2007-03-08 12:47:14.237 TRACE Gold::Reply::sendChunk invoked
> > with
> > arguments: (Gold::Chunk=HASH(0x9d90058))
> > 2007-03-08 12:47:14.237 TRACE
> > Gold::Reply::marshallChunk invoked with
> > arguments: (Gold::Chunk=HASH(0x9d90058))
> > 2007-03-08 12:47:14.238 DEBUG Gold::Reply::sendChunk Writing
> > reply
> > header (HTTP/1.1 200 OK^M
> > Content-Type: text/xml; charset="utf-8"^M
> > Transfer-Encoding: chunked^M
> > ).
> > 2007-03-08 12:47:14.239 INFO Gold::Reply::sendChunk Writing
> > reply
> > payload (1262, <?xml version="1.0" encoding="UTF-8"?>
> > <Envelope><Body><Response
> >
> actor="scottmo"><Status><Value>Success</Value><Code>000</Code></Status><Count>18</Count><Data><Job><Id>6</Id><JobId>45</JobId><User>scottmo</User></Job><Job><Id>7</Id><JobId>46</JobId><User>scottmo</User></Job><Job><Id>8</Id><JobId>47</JobId><User>scottmo</User></Job><Job><Id>9</Id><JobId>48</JobId><User>scottmo</User></Job><Job><Id>10</Id><JobId>49</JobId><User>scottmo</User></Job><Job><Id>11</Id><JobId>50</JobId><User>scottmo</User></Job><Job><Id>12</Id><JobId>51</JobId><User>scottmo</User></Job><Job><Id>13</Id><JobId>52</JobId><User>scottmo</User></Job><Job><Id>14</Id><JobId>53</JobId><User>scottmo</User></Job><Job><Id>15</Id><JobId>54</JobId><User>scottmo</User></Job><Job><Id>16</Id><JobId>59</JobId><User>scottmo</User></Job><Job><Id>17</Id><JobId>60</JobId><User>scottmo</User></Job><Job><Id>19</Id><JobId>62</JobId><User>scottmo</User></Job><Job><Id>20</Id><JobId>63</JobId><User>scottmo</User></Job><Job><Id>18</Id><JobId>61</JobId><User>scottmo</User></Job><Job><Id>21</Id><JobId>64</JobId><User>scottmo</User></Job><Job><Id>22</Id><JobId>65</JobId><User>scottmo</User></Job><Job><Id>23</Id><JobId>
> PBS.1234.0
> </JobId><User>scottmo</User></Job></Data></Response></Body></Envelope>
> > ).
> >
> > You seem to be missing all of the TRACE and DEBUG statements.
> >
> > Ohh, I see it now. Your first line of the Log4perl
> > configuration should
> > say:
> >
> > log4perl.logger = TRACE, Log, Screen
> >
> > I am guessing you must have changed this at some point.
> >
> > Can you set this back up, restart goldd, run your command
> > again and send
> > me the output?
> >
> > Thanks,
> >
> > Scott
> >
> > On Wed, 2007-03-07 at 18:19 -0500, Kevin Van Workum wrote:
> > > Scott,
> > >
> > > I had attached all the output that was generated after I
> > issued the
> > > Query. Perhaps you should know I'm using the SQLite
> > interface, but I
> > > plan to go to mysql prior to production. Or maybe I should
> > adjust my
> > > Log4perl configuration to get the logging you need to see?
> > >
> > > # Log4perl configuration
> > > log4perl.logger = INFO, Log, Screen
> > > log4perl.appender.Log.Threshold = TRACE
> > > log4perl.appender.Screen.Threshold = FATAL
> > > #log4perl.logger.Message = TRACE, Screen
> > > #log4perl.appender.Log = Log::Log4perl::Appender::File
> > > log4perl.appender.Log = Log::Dispatch::FileRotate
> > > log4perl.appender.Log.filename
> > = /usr/local/gold-2.1.1.1/log/goldd.log
> > > log4perl.appender.Log.mode = append
> > > log4perl.appender.Log.size = 10000000
> > > log4perl.appender.Log.max = 7
> > > log4perl.appender.Log.layout = PatternLayout
> > > log4perl.appender.Log.layout.ConversionPattern = %
> > d{yyyy-MM-dd
> > > HH:mm:ss.SSS} %-5p %M %m%n
> > >
> > >
> > > On 3/7/07, Scott Jackson <scott at clusterresources.com> wrote:
> > > Kevin,
> > >
> > > Thanks.
> > >
> > > I actually need to see a much larger portion of the
> > goldd.log
> > > that shows
> > > what the sql query is that it is using and the
> > results.
> > >
> > > Scott
> > >
> > > On Tue, 2007-03-06 at 20:39 -0500, Kevin Van Workum
> > wrote:
> > > > Same thing happens without &&. I've attached are
> > the log
> > > files.
> > > >
> > > > Kevin
> > > >
> > > > On 3/6/07, Scott Jackson <
> > scott at clusterresources.com> wrote:
> > > > That IS strange. Could you send me the
> > gold.log and
> > > goldd.log?
> > > > Also, can
> > > > you see what happens if you do not use the
> > &&'s?
> > > >
> > > > Thanks,
> > > >
> > > > Scott
> > > >
> > > > On Tue, 2007-03-06 at 13:33 -0500, Kevin
> > Van Workum
> > > wrote:
> > > > > Forgot to attach the following. The 344?
> > JobId's
> > > aren't
> > > > displayed:
> > > > >
> > > > > gold> Job Query JobId>=345 && User==vanw
> > > Show:=Id,JobId,User
> > > > > Id JobId User
> > > > > --- ----- ----
> > > > > 214 415 vanw
> > > > > 215 416 vanw
> > > > > 216 417 vanw
> > > > > 217 418 vanw
> > > > > 218 420 vanw
> > > > >
> > > > > On 3/6/07, Scott Jackson
> > > <scott at clusterresources.com> wrote:
> > > > > Kevin,
> > > > >
> > > > > How do you figure that?
> > > > >
> > > > > All of the JobIds listed below
> > are greater
> > > than or
> > > > equal to
> > > > > 344.
> > > > >
> > > > > Scott
> > > > >
> > > > > On Tue, 2007-03-06 at 12:34
> > -0500, Kevin
> > > Van Workum
> > > > wrote:
> > > > > > It looks like goldsh is only
> > using the
> > > first 3
> > > > digits of the
> > > > > JobId.
> > > > > >
> > > > > > gold> Job Query JobId>=344 &&
> > > User==vanw
> > > > > Show:=Id,JobId,User
> > > > > > Id JobId User
> > > > > > ---- ----- ----
> > > > > > 214 415 vanw
> > > > > > 215 416 vanw
> > > > > > 216 417 vanw
> > > > > > 217 418 vanw
> > > > > > 218 420 vanw
> > > > > > 1899 3445 vanw
> > > > > > 1900 3446 vanw
> > > > > > 1901 3447 vanw
> > > > > > 1902 3448 vanw
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Kevin Van Workum, Ph.D.
> > > > > > Vice President
> > > > > > Senior System Administrator
> > > > > > www.clusterondemand.com
> > > > > > ONLINE COMPUTER CLUSTERS
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Kevin Van Workum, Ph.D.
> > > > > Vice President
> > > > > Senior System Administrator
> > > > > www.clusterondemand.com
> > > > > ONLINE COMPUTER CLUSTERS
> > > >
> > > >
> > >
> > >
> > >
> > >
> > > --
> > > Kevin Van Workum, Ph.D.
> > > Vice President
> > > Senior System Administrator
> > > www.clusterondemand.com
> > > ONLINE COMPUTER CLUSTERS
> >
> >
> >
> >
> > --
> > Kevin Van Workum, Ph.D.
> > Vice President
> > Senior System Administrator
> > www.clusterondemand.com
> > ONLINE COMPUTER CLUSTERS
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/gold-users/attachments/20070308/3179071f/attachment-0001.html
More information about the gold-users
mailing list