From s.prabhakaran at grs-sim.de  Tue Sep  4 06:14:01 2012
From: s.prabhakaran at grs-sim.de (Suraj Prabhakaran)
Date: Tue, 04 Sep 2012 14:14:01 +0200
Subject: [torquedev] Influencing the order of nodes in PBS_NODEFILE
Message-ID: <81443EB8-04E8-4ABA-8A09-779461D8F05F@grs-sim.de>

Hi all,

I have some nodes with 2 different properties, example:

node1 fast
node2 fast
node3 slow
node4 slow

When I request nodes in this following fashion

qsub -l nodes=2:fast+2:slow

I observe the $PBS_NODEFILE does not strictly have the nodes listed in the order that was requested but its random.
For instance, a strict ordering of the nodes in the order of request would lead to $PBS_NODEFILE with content

node1
node2
node3
node4

But I observed something like

node3
node2
node4
node1

--------

The question is, is it possible to influence a strict ordering of nodes according to the request? 
If not, could someone please show me some pointers to the torque code where this can be influenced?

Best regards,
Suraj

From samuel at unimelb.edu.au  Tue Sep  4 21:02:32 2012
From: samuel at unimelb.edu.au (Christopher Samuel)
Date: Wed, 05 Sep 2012 13:02:32 +1000
Subject: [torquedev] Influencing the order of nodes in PBS_NODEFILE
In-Reply-To: <81443EB8-04E8-4ABA-8A09-779461D8F05F@grs-sim.de>
References: <81443EB8-04E8-4ABA-8A09-779461D8F05F@grs-sim.de>
Message-ID: <5046C0C8.8080208@unimelb.edu.au>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/09/12 22:14, Suraj Prabhakaran wrote:

> The question is, is it possible to influence a strict ordering of
> nodes according to the request? If not, could someone please show
> me some pointers to the torque code where this can be influenced?

I'm not sure, but I think the ordering of the nodes is done by the
scheduler, not Torque itself (well, unless you use pbs_sched).  I
could be wrong though..

- -- 
 Christopher Samuel        Senior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.org.au/      http://twitter.com/vlsci

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iEYEARECAAYFAlBGwMgACgkQO2KABBYQAh9p1QCgkmfqeZdb0r6Yuq8Kz67Docns
oW4An2lXl+Pc0dMIXWc43S5usf2ERV8L
=Yjgx
-----END PGP SIGNATURE-----

From samuel at unimelb.edu.au  Tue Sep  4 21:05:32 2012
From: samuel at unimelb.edu.au (Christopher Samuel)
Date: Wed, 05 Sep 2012 13:05:32 +1000
Subject: [torquedev] [torqueusers] torque IRC channel
In-Reply-To: <5022773D.1040507@cyf-kr.edu.pl>
References: <5022773D.1040507@cyf-kr.edu.pl>
Message-ID: <5046C17C.4030401@unimelb.edu.au>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/08/12 00:27, Lukasz Flis wrote:

> Is there any IRC channel for Torque community and/or developers
> available?

I've not used it for decades now, but nothing to stop those that do
from congregating in #torque if they so want to.

cheers,
Chris
- -- 
 Christopher Samuel        Senior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.org.au/      http://twitter.com/vlsci

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iEYEARECAAYFAlBGwXwACgkQO2KABBYQAh/lhACfZvaJGzhm18CkzSEA8nYdrccZ
sa0AnRKl6gqeec6Nw+b0XZfheYJNPyOC
=tTRX
-----END PGP SIGNATURE-----

From s.prabhakaran at grs-sim.de  Mon Sep 10 12:40:17 2012
From: s.prabhakaran at grs-sim.de (Suraj Prabhakaran)
Date: Mon, 10 Sep 2012 20:40:17 +0200
Subject: [torquedev] Maui/Torque with node properties
Message-ID: <00DF894B-F5A2-4F73-BD12-1BFC0C6A562A@grs-sim.de>

Dear all,

I have 4 nodes with the following properties

node1 fast
node2 fast
node3 slow
node4 slow

Traditionally, torque allows to request nodes with different properties by

	qsub -l nodes=1:fast+1:slow

The above should allocate one fast node and one slow node and this works perfectly fine when pbs_sched is used. 

But when I use maui as my scheduler, I never get the nodes assigned and end up waiting infinitely. 

Is this feature supported in maui? Until now, I haven't read anywhere that this feature is not supported in maui. 
Or, am I just missing something here?

Best,
Suraj

From bugzilla-daemon at supercluster.org  Tue Sep 11 02:19:48 2012
From: bugzilla-daemon at supercluster.org (bugzilla-daemon at supercluster.org)
Date: Tue, 11 Sep 2012 02:19:48 -0600 (MDT)
Subject: [torquedev] [Bug 215] New: if qdel is used with "-t" option,
 qdel should add "[]" to JOBID.
Message-ID: <bug-215-38@http.www.clusterresources.com/bugzilla/>

http://www.clusterresources.com/bugzilla/show_bug.cgi?id=215

           Summary: if qdel is used with "-t" option, qdel should add "[]"
                    to JOBID.
           Product: TORQUE
           Version: 4.0.*
          Platform: PC
        OS/Version: Linux
            Status: NEW
          Severity: minor
          Priority: P5
         Component: clients
        AssignedTo: knielson at adaptivecomputing.com
        ReportedBy: sstcosaka01 at gmail.com
                CC: torquedev at supercluster.org
   Estimated Hours: 0.0


Created an attachment (id=118)
 --> (http://www.clusterresources.com/bugzilla/attachment.cgi?id=118)
qdel.c modification

We can not cancel jobs in a job array with qdel -t array_range JOBID with
errors like
  qdel: Unauthorized Request  MSG=must have operator or manager privilege to
use -m parameter 446.torque02

If we modify torque-4.1.1/src/cmds/qdel.c with patch.qdel.c so that "[]" is
added to job_id,
  we can cancel jobs in a job array with qdel -t.

There is a workaround that we can cancel jobs with qdel -t 
  adding "[]" to jobid at command line.
   (Good)  qdel -t 4-8 479[]  
   (Bad)   qdel -t 4-8 479

We have checked this with torque-4.1.1 and torque-4.1.0.

((qdel -t with modified qdel))
/dev/shm/usr/local/torque/bin/qdel:  modified qdel
/usr/local/torque/bin/qdel:          original qdel

[test01 at torque02 ~]$ qstat -t
Job id                    Name             User            Time Use S Queue
------------------------- ---------------- --------------- -------- - -----
445[5].torque02            STDIN-5          test01          00:00:00 C batch    
445[6].torque02            STDIN-6          test01          00:00:00 C batch    
445[7].torque02            STDIN-7          test01          00:00:00 C batch    
445[8].torque02            STDIN-8          test01          00:00:00 C batch    
446[1].torque02            STDIN-1          test01                 0 R batch    
446[2].torque02            STDIN-2          test01                 0 R batch    
446[3].torque02            STDIN-3          test01                 0 R batch    
446[4].torque02            STDIN-4          test01                 0 R batch    
446[5].torque02            STDIN-5          test01                 0 R batch    
446[6].torque02            STDIN-6          test01                 0 R batch    
446[7].torque02            STDIN-7          test01                 0 R batch    
446[8].torque02            STDIN-8          test01                 0 R batch    
[test01 at torque02 ~]$ /dev/shm/usr/local/torque/bin/qdel -t 5-8 446
[test01 at torque02 ~]$ qdel -t 1-4 446 
qdel: Unauthorized Request  MSG=must have operator or manager privilege to use
-m parameter 446.torque02

((workarround))
[test01 at torque02 ~]$ qstat -t
Job id                    Name             User            Time Use S Queue
------------------------- ---------------- --------------- -------- - -----
479[1].torque02            NPB01-1          test01                 0 Q batch    
479[2].torque02            NPB01-2          test01                 0 Q batch    
479[3].torque02            NPB01-3          test01                 0 Q batch    
479[4].torque02            NPB01-4          test01                 0 Q batch    
479[5].torque02            NPB01-5          test01                 0 Q batch    
479[6].torque02            NPB01-6          test01                 0 Q batch    
479[7].torque02            NPB01-7          test01                 0 Q batch    
479[8].torque02            NPB01-8          test01                 0 Q batch    
[test01 at torque02 ~]$ qdel -t 4-8 479
qdel: Unauthorized Request  MSG=must have operator or manager privilege to use
-m parameter 479.torque02.ahoaho
[test01 at torque02 ~]$ qdel -t 4-8 479[]
[test01 at torque02 ~]$ qstat -t
Job id                    Name             User            Time Use S Queue
------------------------- ---------------- --------------- -------- - -----
479[1].torque02            NPB01-1          test01                 0 R batch    
479[2].torque02            NPB01-2          test01                 0 R batch    
479[3].torque02            NPB01-3          test01                 0 Q batch    
479[4].torque02            NPB01-4          test01                 0 C batch    
479[5].torque02            NPB01-5          test01                 0 C batch    
479[6].torque02            NPB01-6          test01                 0 C batch    
479[7].torque02            NPB01-7          test01                 0 C batch    
479[8].torque02            NPB01-8          test01                 0 C batch  

thank you
go
---

-- 
Configure bugmail: http://www.clusterresources.com/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

From bugzilla-daemon at supercluster.org  Tue Sep 11 02:34:21 2012
From: bugzilla-daemon at supercluster.org (bugzilla-daemon at supercluster.org)
Date: Tue, 11 Sep 2012 02:34:21 -0600 (MDT)
Subject: [torquedev] [Bug 215] if qdel is used with "-t" option,
 qdel should add "[]" to JOBID.
In-Reply-To: <bug-215-38@http.www.clusterresources.com/bugzilla/>
References: <bug-215-38@http.www.clusterresources.com/bugzilla/>
Message-ID: <20120911083421.C29453EA8299@http.supercluster.org>

http://www.clusterresources.com/bugzilla/show_bug.cgi?id=215

sstcosaka01 at gmail.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
    Attachment #118|qdel.c modification         |patch.qdel.c
        description|                            |

-- 
Configure bugmail: http://www.clusterresources.com/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

From bugzilla-daemon at supercluster.org  Thu Sep 13 19:30:37 2012
From: bugzilla-daemon at supercluster.org (bugzilla-daemon at supercluster.org)
Date: Thu, 13 Sep 2012 19:30:37 -0600 (MDT)
Subject: [torquedev] [Bug 209] pbs_server rejects Obits with cray_enabled
In-Reply-To: <bug-209-38@http.www.clusterresources.com/bugzilla/>
References: <bug-209-38@http.www.clusterresources.com/bugzilla/>
Message-ID: <20120914013037.2E0A567810D@http.supercluster.org>

http://www.clusterresources.com/bugzilla/show_bug.cgi?id=209

Matt Ezell <ezellma at ornl.gov> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |FIXED

--- Comment #3 from Matt Ezell <ezellma at ornl.gov> 2012-09-13 19:30:34 MDT ---
I haven't seen this again.  Closing.

-- 
Configure bugmail: http://www.clusterresources.com/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

From bugzilla-daemon at supercluster.org  Thu Sep 13 19:42:02 2012
From: bugzilla-daemon at supercluster.org (bugzilla-daemon at supercluster.org)
Date: Thu, 13 Sep 2012 19:42:02 -0600 (MDT)
Subject: [torquedev] [Bug 112] 'qstat -a' 'tasks' column width too small
In-Reply-To: <bug-112-38@http.www.clusterresources.com/bugzilla/>
References: <bug-112-38@http.www.clusterresources.com/bugzilla/>
Message-ID: <20120914014202.5653E4121140@http.supercluster.org>

http://www.clusterresources.com/bugzilla/show_bug.cgi?id=112

Matt Ezell <ezellma at ornl.gov> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |ezellma at ornl.gov
         Resolution|                            |FIXED

--- Comment #1 from Matt Ezell <ezellma at ornl.gov> 2012-09-13 19:42:01 MDT ---
It looks like this has been fixed.  Right now, we have '#define DEFTASKSIZE 6'.
 Closing.

-- 
Configure bugmail: http://www.clusterresources.com/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

From bugzilla-daemon at supercluster.org  Sun Sep 16 06:08:28 2012
From: bugzilla-daemon at supercluster.org (bugzilla-daemon at supercluster.org)
Date: Sun, 16 Sep 2012 06:08:28 -0600 (MDT)
Subject: [torquedev] [Bug 216] New: array job crashes server if
 display_job_server_suffix = False
Message-ID: <bug-216-38@http.www.clusterresources.com/bugzilla/>

http://www.clusterresources.com/bugzilla/show_bug.cgi?id=216

           Summary: array job crashes server if display_job_server_suffix
                    = False
           Product: TORQUE
           Version: 2.5.x
          Platform: PC
        OS/Version: Linux
            Status: NEW
          Severity: enhancement
          Priority: P5
         Component: pbs_server
        AssignedTo: dbeer at adaptivecomputing.com
        ReportedBy: bunk at physik.hu-berlin.de
                CC: torquedev at supercluster.org
   Estimated Hours: 0.0


Hi,

after setting

        set server display_job_server_suffix = False

(such that JobID = job_number), any attempt to submit an array job with

        qsub -t <range> <script>

results in

        qsub: End of File

and the server crashes immediately, without any message in its logfile.

I'm running torque-2.5.11 (with maui-3.3.1) in Debian Squeeze.

A fix would be to define another suffix with

        set server job_suffix_alias = <something>

BTW, "job_suffix_alias" is not documented in the Admin Guide (2.5.12).
And admins should be warned that the settings of
        display_job_server_suffix
and     job_suffix_alias
should only be changed while the system is empty. The syntax change in JobID
hurts running (and queued?) jobs.

Regards,
Burkhard Bunk.

-- 
Configure bugmail: http://www.clusterresources.com/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

From bugzilla-daemon at supercluster.org  Sun Sep 16 06:13:11 2012
From: bugzilla-daemon at supercluster.org (bugzilla-daemon at supercluster.org)
Date: Sun, 16 Sep 2012 06:13:11 -0600 (MDT)
Subject: [torquedev] [Bug 216] array job crashes server if
 display_job_server_suffix = False
In-Reply-To: <bug-216-38@http.www.clusterresources.com/bugzilla/>
References: <bug-216-38@http.www.clusterresources.com/bugzilla/>
Message-ID: <20120916121311.467C6412137F@http.supercluster.org>

http://www.clusterresources.com/bugzilla/show_bug.cgi?id=216

--- Comment #1 from Burkhard Bunk <bunk at physik.hu-berlin.de> 2012-09-16 06:13:10 MDT ---
The same happened on CentOS 6.2,
running torque-2.5.5-7nec (and maui-3.3.2.nec1-14nec).

-- 
Configure bugmail: http://www.clusterresources.com/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

From bugzilla-daemon at supercluster.org  Thu Sep 20 14:32:29 2012
From: bugzilla-daemon at supercluster.org (bugzilla-daemon at supercluster.org)
Date: Thu, 20 Sep 2012 14:32:29 -0600 (MDT)
Subject: [torquedev] [Bug 217] New: Multiple job array dependencies are not
	honored
Message-ID: <bug-217-38@http.www.clusterresources.com/bugzilla/>

http://www.clusterresources.com/bugzilla/show_bug.cgi?id=217

           Summary: Multiple job array dependencies are not honored
           Product: TORQUE
           Version: 3.0.x
          Platform: All
        OS/Version: Linux
            Status: NEW
          Severity: major
          Priority: P5
         Component: pbs_server
        AssignedTo: dbeer at adaptivecomputing.com
        ReportedBy: tasbu at aol.com
                CC: torquedev at supercluster.org
   Estimated Hours: 0.0


I've boiled down a significant job dependency array bug to a simple test
script.
The problem is that job array dependencies are not honored when you have
more than one array dependency.

The script below creates 2 small sleep jobs of different lengths and submits
each as an array. A final wrap-up job is submitted as depending upon both
of the job arrays successfully finishing. However, the dependent job does not
wait for both arrays to exit successfully before starting, rather,
it starts early after the first array job finishes:

   sleep30_array           sleep15_array
        |                       |
        |                       |
        |                       -
        |                     done
        |                dep job starts
        |                    *bug*
        |
       ---
       done
dep job should start here


--------SCRIPT_START------
cat << EOF > sleeper15
sleep 15
EOF
cat << EOF > sleeper30
sleep 30
EOF

JOB1=`cat sleeper30 | qsub -t 1-2`
JOB2=`cat sleeper15 | qsub -t 1-2`
echo "cat sleeper15 | qsub -W depend=afterokarray:${JOB1}:${JOB2}"
cat sleeper15 | qsub -W depend=afterokarray:${JOB1}:${JOB2}
--------SCRIPT_END------

Running the script above will give the something like the following:

> sh SCRIPT
cat sleeper15 | qsub -W depend=afterokarray:55349[].madrid:55350[].madrid
55351.madrid

-- after 20 seconds --

> qstat
55349[].madrid             STDIN            tasbu                  0 R batch
55350[].madrid             STDIN            tasbu                  0 C batch
55351.madrid               STDIN            tasbu                  0 R batch

55351.madrid should not be running!

Torque version: 3.0.5

A solution to this problem is crucial to our pipeline and I would
appreciate any fixes / workarounds.

Thanks -
Tom

-- 
Configure bugmail: http://www.clusterresources.com/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

From bugzilla-daemon at supercluster.org  Thu Sep 27 10:56:40 2012
From: bugzilla-daemon at supercluster.org (bugzilla-daemon at supercluster.org)
Date: Thu, 27 Sep 2012 10:56:40 -0600 (MDT)
Subject: [torquedev] [Bug 216] array job crashes server if
 display_job_server_suffix = False
In-Reply-To: <bug-216-38@http.www.clusterresources.com/bugzilla/>
References: <bug-216-38@http.www.clusterresources.com/bugzilla/>
Message-ID: <20120927165640.81DD5678029@http.supercluster.org>

http://www.clusterresources.com/bugzilla/show_bug.cgi?id=216

Burkhard Bunk <bunk at physik.hu-berlin.de> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |bunk at physik.hu-berlin.de
           Severity|enhancement                 |major

-- 
Configure bugmail: http://www.clusterresources.com/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

From bugzilla-daemon at supercluster.org  Fri Sep 28 14:40:58 2012
From: bugzilla-daemon at supercluster.org (bugzilla-daemon at supercluster.org)
Date: Fri, 28 Sep 2012 14:40:58 -0600 (MDT)
Subject: [torquedev] [Bug 216] array job crashes server if
 display_job_server_suffix = False
In-Reply-To: <bug-216-38@http.www.clusterresources.com/bugzilla/>
References: <bug-216-38@http.www.clusterresources.com/bugzilla/>
Message-ID: <20120928204058.F30853EA800B@http.supercluster.org>

http://www.clusterresources.com/bugzilla/show_bug.cgi?id=216

David Beer <dbeer at adaptivecomputing.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |dbeer at adaptivecomputing.com
         Resolution|                            |FIXED

--- Comment #2 from David Beer <dbeer at adaptivecomputing.com> 2012-09-28 14:40:58 MDT ---
Checked in to 2.5.13 and 4.1.3.

-- 
Configure bugmail: http://www.clusterresources.com/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

From bugzilla-daemon at supercluster.org  Sun Sep 30 23:08:47 2012
From: bugzilla-daemon at supercluster.org (bugzilla-daemon at supercluster.org)
Date: Sun, 30 Sep 2012 23:08:47 -0600 (MDT)
Subject: [torquedev] [Bug 218] New: Jobs getting stuck in exiting "job
 recycled into exiting on SIGNULL/KILL"
Message-ID: <bug-218-38@http.www.clusterresources.com/bugzilla/>

http://www.clusterresources.com/bugzilla/show_bug.cgi?id=218

           Summary: Jobs getting stuck in exiting "job recycled into
                    exiting on SIGNULL/KILL"
           Product: TORQUE
           Version: 2.4.x
          Platform: PC
        OS/Version: Linux
            Status: NEW
          Severity: normal
          Priority: P5
         Component: pbs_mom
        AssignedTo: knielson at adaptivecomputing.com
        ReportedBy: chris at csamuel.org
                CC: torquedev at supercluster.org
   Estimated Hours: 0.0


We are running 2.4, but I note that this code is essentially unchanged through
to 4.1.x.

On one of our clusters we are seeing bursts of jobs that end up stuck in an
EXITING state on various nodes for no apparent reason.   For this month I see:

[root at merri-m ~]# xdsh compute -v 'fgrep -h "job recycled into exiting"
/var/spool/torque/mom_logs/201209*' | awk -F\; '{print $NF}' | sort | uniq -c
     10 job recycled into exiting on SIGNULL/KILL from substate 1
      9 job recycled into exiting on SIGNULL/KILL from substate 40
     45 job recycled into exiting on SIGNULL/KILL from substate 42
     50 job recycled into exiting on SIGNULL/KILL from substate 50
      6 job recycled into exiting on SIGNULL/KILL from substate 53
     76 job recycled into exiting on SIGNULL/KILL from substate 57

These messages are logged from line 2199 in
branches/2.4-fixes/src/resmom/requests.c and get triggered when signal 0 or 9
are sent to a task (either to see if it exists, or to kill it off,
respectively).

Currently our only solution we've found so far is to do either a momctl -c for
the job on its mother superior or qdel -p for the job.   It would be nicer if
we could just get this to handle this case.

-- 
Configure bugmail: http://www.clusterresources.com/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

From tomas at jikos.cz  Tue Sep 11 02:27:05 2012
From: tomas at jikos.cz (=?windows-1252?Q?Tom=E1=9A_Kouba?=)
Date: Tue, 11 Sep 2012 08:27:05 -0000
Subject: [torquedev] IPv6 status of torque
Message-ID: <504EF5D0.907@jikos.cz>

Hello,

I would like to try the torque in our ipv6-only environment and I found 
a thread in this mailing list about IPv6 branch of the torque. But it is 
based on 2.1 version and it looks quite dead. I have checked that the 
network code changed a lot between 2.1 and 4.0
Can anyone comment on the current status of torque regarding ipv6?
Is there any roadmap, someone working on it, hints what should be done?

Thank you,

-- 
Tom?? Kouba

From douglas.holt at st.com  Wed Sep 12 07:40:51 2012
From: douglas.holt at st.com (Douglas Holt)
Date: Wed, 12 Sep 2012 13:40:51 -0000
Subject: [torquedev] pbs_server hanging
Message-ID: <505090D9.3050301@st.com>

I'm seeing numerous socket timeouts when trying to use commands that 
interact with pbs_server (ver. 4.1.1) and saw this in my logs:

INFO: task pbs_server:57675 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
pbs_server    D 0000000000000002     0 57675      1 0x00000080
  ffff880802ebdbb8 0000000000000086 0000000000000000 ffff88102c7ae3c8
  0000000000000000 ffff88102c7ae3e0 00000000000000c3 0000000000000001
  ffff8807e41ba5f8 ffff880802ebdfd8 000000000000f4e8 ffff8807e41ba5f8
Call Trace:
  [<ffffffff8105fa50>] ? default_wake_function+0x0/0x20
  [<ffffffff814ee0ae>] __mutex_lock_slowpath+0x13e/0x180
  [<ffffffff814edf4b>] mutex_lock+0x2b/0x50
  [<ffffffff814bc9d1>] unix_dgram_recvmsg+0x71/0x380
  [<ffffffff811458fd>] ? page_add_new_anon_rmap+0x9d/0xf0
  [<ffffffff8141bf51>] sock_aio_read+0x181/0x190
  [<ffffffff8117641a>] do_sync_read+0xfa/0x140
  [<ffffffff81090bf0>] ? autoremove_wake_function+0x0/0x40
  [<ffffffff8120c3c6>] ? security_file_permission+0x16/0x20
  [<ffffffff81176ee1>] vfs_read+0x181/0x1a0
  [<ffffffff810d46e2>] ? audit_syscall_entry+0x272/0x2a0
  [<ffffffff81176f51>] sys_read+0x51/0x90
  [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b

From tom.asbury at sequentainc.com  Mon Sep 17 14:44:47 2012
From: tom.asbury at sequentainc.com (Tom Asbury)
Date: Mon, 17 Sep 2012 20:44:47 -0000
Subject: [torquedev] multiple job array dependency bug with simple test case
	script
Message-ID: <4424F658BE5DB042A4E55CDE3B50CA4923B32F5DD4@EX-BE-016-SFO.shared.themessagecenter.com>


Hi there - 

I've boiled down what I think to be a significant job dependency array bug to a simple test case.
The problem is that job array dependencies are not honored when you have
more than one array dependency.

The script below creates 2 small sleep jobs of different lengths and submits
each as an array. A final wrap-up job is submitted as depending upon both
of the job arrays successfully finishing. However, the dependent job does not
wait for both arrays to exit successfully before starting, rather,
it starts early after the first array job finishes:

   sleep120_array           sleep60_array
        |                       |
        |                       |
        |                       -
        |                     done
        |                dep job starts
        |                    *bug*
        |
       ---
       done
dep job should start here


#--------SH SCRIPT START------
cat << EOF > sleeper60
sleep 60
EOF
cat << EOF > sleeper120
sleep 120
EOF

JOB1=`cat sleeper120 | qsub -t 1-2`
JOB2=`cat sleeper60 | qsub -t 1-2`
echo "cat sleeper60 | qsub -W depend=afterokarray:${JOB1}:${JOB2}"
cat sleeper60 | qsub -W depend=afterokarray:${JOB1}:${JOB2}
# --------SH SCRIPT END------

Running the script above will give the something like the following:

> sh SCRIPT
cat sleeper60 | qsub -W depend=afterokarray:55349[].madrid:55350[].madrid
55351.madrid

-- after 90 seconds --

> qstat
55349[].madrid             STDIN            tasbu                  0 R batch
55350[].madrid             STDIN            tasbu                  0 C batch
55351.madrid               STDIN            tasbu                  0 R batch

55351.madrid should not be running!

Torque version: 3.0.5

A solution to this problem is crucial to our pipeline and I would
appreciate any fixes / workarounds.

Thanks -
Tom


From ezellma at ornl.gov  Fri Sep 21 13:27:31 2012
From: ezellma at ornl.gov (Ezell, Matthew A.)
Date: Fri, 21 Sep 2012 19:27:31 -0000
Subject: [torquedev] Testing TORQUE with unit tests
Message-ID: <CC8237DD.1056E%ezellma@ornl.gov>

Hello-

I am interested in trying out TORQUE's unit test framework.  I configured
--with-check and ran 'make check' but many files didn't compile.  Some
scaffolding files appear to be missing "helper" functions.  How are these
managed?  Is there a tool to automatically pull in functions/constants
that are needed, or is this a manual process?  What happens if a "helper"
function changes in the regular source tree - you have to go check all the
scaffolding files and make updates there also?

Thanks,
~Matt

---
Matt Ezell
HPC Systems Administrator
Oak Ridge National Laboratory