From bugzilla-daemon at supercluster.org  Mon Mar  5 21:18:44 2012
From: bugzilla-daemon at supercluster.org (bugzilla-daemon at supercluster.org)
Date: Mon,  5 Mar 2012 21:18:44 -0700 (MST)
Subject: [torquedev] [Bug 174] New: pbs_mom kills running jobs despite -p
	flag
Message-ID: <bug-174-38@http.www.clusterresources.com/bugzilla/>

http://www.clusterresources.com/bugzilla/show_bug.cgi?id=174

           Summary: pbs_mom kills running jobs despite -p flag
           Product: TORQUE
           Version: 2.5.x
          Platform: PC
        OS/Version: Linux
            Status: NEW
          Severity: critical
          Priority: P5
         Component: pbs_mom
        AssignedTo: knielson at adaptivecomputing.com
        ReportedBy: siegert at sfu.ca
                CC: torquedev at supercluster.org
   Estimated Hours: 0.0


I want to restart a pbs_mom on a node where it has died for whatever reason
without killing the jobs that are still running on the node. We used to be
able to do this by starting the pbs_mom with the -p flag, but apparently this
is not working anymore: everytime I start the mom using "pbs_mom -p" all
running jobs get killed. My feeling is that -p stopped working when we started
to use cpusets (I am not absolutely sure about this since we also upgraded
torque versions since then). We are currently running torque-2.5.10.

-- 
Configure bugmail: http://www.clusterresources.com/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

From bugzilla-daemon at supercluster.org  Tue Mar  6 13:08:28 2012
From: bugzilla-daemon at supercluster.org (bugzilla-daemon at supercluster.org)
Date: Tue,  6 Mar 2012 13:08:28 -0700 (MST)
Subject: [torquedev] [Bug 139] Negative value in 'Que' when using qstat
In-Reply-To: <bug-139-38@http.www.clusterresources.com/bugzilla/>
References: <bug-139-38@http.www.clusterresources.com/bugzilla/>
Message-ID: <20120306200828.C35C3678133@http.supercluster.org>

http://www.clusterresources.com/bugzilla/show_bug.cgi?id=139

Yury V. Zaytsev <yury at shurup.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |yury at shurup.com

--- Comment #6 from Yury V. Zaytsev <yury at shurup.com> 2012-03-06 13:08:28 MST ---
Same problem on my 2.5.9 installation: qterm -t quick and pbs_server -t hot
helps, but only for short time.

-- 
Configure bugmail: http://www.clusterresources.com/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

From bugzilla-daemon at supercluster.org  Tue Mar  6 14:41:47 2012
From: bugzilla-daemon at supercluster.org (bugzilla-daemon at supercluster.org)
Date: Tue,  6 Mar 2012 14:41:47 -0700 (MST)
Subject: [torquedev] [Bug 174] pbs_mom kills running jobs despite -p flag
In-Reply-To: <bug-174-38@http.www.clusterresources.com/bugzilla/>
References: <bug-174-38@http.www.clusterresources.com/bugzilla/>
Message-ID: <20120306214147.3C70641217F4@http.supercluster.org>

http://www.clusterresources.com/bugzilla/show_bug.cgi?id=174

--- Comment #1 from Martin Siegert <siegert at sfu.ca> 2012-03-06 14:41:47 MST ---
When I replace the line 214 in cpuset.c

        if (cpuset_delete(pdirent->d_name) == 0)

with "if (0)" then the jobs do not get killed when I restart pbs_mom.

-- 
Configure bugmail: http://www.clusterresources.com/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

From bugzilla-daemon at supercluster.org  Tue Mar  6 16:28:40 2012
From: bugzilla-daemon at supercluster.org (bugzilla-daemon at supercluster.org)
Date: Tue,  6 Mar 2012 16:28:40 -0700 (MST)
Subject: [torquedev] [Bug 174] pbs_mom kills running jobs despite -p flag
In-Reply-To: <bug-174-38@http.www.clusterresources.com/bugzilla/>
References: <bug-174-38@http.www.clusterresources.com/bugzilla/>
Message-ID: <20120306232840.22EE64121886@http.supercluster.org>

http://www.clusterresources.com/bugzilla/show_bug.cgi?id=174

--- Comment #2 from Ken Nielson <knielson at adaptivecomputing.com> 2012-03-06 16:28:39 MST ---
(In reply to comment #1)
> When I replace the line 214 in cpuset.c
> 
>         if (cpuset_delete(pdirent->d_name) == 0)
> 
> with "if (0)" then the jobs do not get killed when I restart pbs_mom.

I have added this to the AC internal ticketing system so we can get it fixed.

-- 
Configure bugmail: http://www.clusterresources.com/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

From bugzilla-daemon at supercluster.org  Wed Mar  7 06:35:57 2012
From: bugzilla-daemon at supercluster.org (bugzilla-daemon at supercluster.org)
Date: Wed,  7 Mar 2012 06:35:57 -0700 (MST)
Subject: [torquedev] [Bug 174] pbs_mom kills running jobs despite -p flag
In-Reply-To: <bug-174-38@http.www.clusterresources.com/bugzilla/>
References: <bug-174-38@http.www.clusterresources.com/bugzilla/>
Message-ID: <20120307133557.A6FC767819A@http.supercluster.org>

http://www.clusterresources.com/bugzilla/show_bug.cgi?id=174

Lukasz Flis <l.flis at cyf-kr.edu.pl> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |l.flis at cyf-kr.edu.pl

--- Comment #3 from Lukasz Flis <l.flis at cyf-kr.edu.pl> 2012-03-07 06:35:57 MST ---
Hi, 

I can confirm we experience the same issue since switching cpuset support on.

Currently we run 2.5.10 and the problem persists


Good thing is that CPUsets should ease process tracking is such case since all
child processess spawned by given job are available in:

/dev/cpuset/torque/<jobid>/tasks

cat /dev/cpuset/torque/19164583.batch.grid.cyf-kr.edu.pl/tasks
10807
10866
10875
10889
10956
10962
10965
10993
10994
10995
10996
10997
10998
10999
11000

Cheers
--
LKF

-- 
Configure bugmail: http://www.clusterresources.com/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

From bugzilla-daemon at supercluster.org  Wed Mar  7 20:11:37 2012
From: bugzilla-daemon at supercluster.org (bugzilla-daemon at supercluster.org)
Date: Wed,  7 Mar 2012 20:11:37 -0700 (MST)
Subject: [torquedev] [Bug 174] pbs_mom kills running jobs despite -p flag
In-Reply-To: <bug-174-38@http.www.clusterresources.com/bugzilla/>
References: <bug-174-38@http.www.clusterresources.com/bugzilla/>
Message-ID: <20120308031137.72E66678072@http.supercluster.org>

http://www.clusterresources.com/bugzilla/show_bug.cgi?id=174

Chris Samuel <chris at csamuel.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED
                 CC|                            |chris at csamuel.org

--- Comment #4 from Chris Samuel <chris at csamuel.org> 2012-03-07 20:11:37 MST ---
Looks like this has been fixed in SVN with commits 5855 and 5856.

The commit doesn't reference this BZ number unfortunately.

-- 
Configure bugmail: http://www.clusterresources.com/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

From bugzilla-daemon at supercluster.org  Thu Mar  8 09:49:36 2012
From: bugzilla-daemon at supercluster.org (bugzilla-daemon at supercluster.org)
Date: Thu,  8 Mar 2012 09:49:36 -0700 (MST)
Subject: [torquedev] [Bug 174] pbs_mom kills running jobs despite -p flag
In-Reply-To: <bug-174-38@http.www.clusterresources.com/bugzilla/>
References: <bug-174-38@http.www.clusterresources.com/bugzilla/>
Message-ID: <20120308164936.D3764257810F@http.supercluster.org>

http://www.clusterresources.com/bugzilla/show_bug.cgi?id=174

Ken Nielson <knielson at adaptivecomputing.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|                            |FIXED

--- Comment #5 from Ken Nielson <knielson at adaptivecomputing.com> 2012-03-08 09:49:36 MST ---
Fixed in 2.5.11 revision 5855

-- 
Configure bugmail: http://www.clusterresources.com/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

From jrosenquist at adaptivecomputing.com  Thu Mar  8 10:32:53 2012
From: jrosenquist at adaptivecomputing.com (John Rosenquist)
Date: Thu, 08 Mar 2012 10:32:53 -0700
Subject: [torquedev] command line -q option
Message-ID: <4F58ED45.1040509@adaptivecomputing.com>

This is John Rosenquist, I'm one of the developers at Adaptive Computing 
working on torque.

I was wondering if anyone uses the -q option on any of the commands 
(pbsnodes, etc). The purpose is to suppress all output from the command.

I would like to get rid of it.

Please let me know if anyone is using this feature.

John.

From tbaer at utk.edu  Thu Mar  8 10:40:50 2012
From: tbaer at utk.edu (Troy Baer)
Date: Thu, 8 Mar 2012 12:40:50 -0500
Subject: [torquedev] command line -q option
In-Reply-To: <4F58ED45.1040509@adaptivecomputing.com>
References: <4F58ED45.1040509@adaptivecomputing.com>
Message-ID: <1331228450.5702.479.camel@browncoat.jics.utk.edu>

On Thu, 2012-03-08 at 10:32 -0700, John Rosenquist wrote:
> This is John Rosenquist, I'm one of the developers at Adaptive Computing 
> working on torque.
> 
> I was wondering if anyone uses the -q option on any of the commands 
> (pbsnodes, etc). The purpose is to suppress all output from the command.
> 
> I would like to get rid of it.
> 
> Please let me know if anyone is using this feature.

I think you're going to have to be more specific about exactly which
commands you mean.  (For instance, if you remove the -q option from
qsub, you may have a riot on your hands...)

	--Troy
-- 
Troy Baer, Senior HPC System Administrator
National Institute for Computational Sciences, University of Tennessee
http://www.nics.tennessee.edu/
Phone:  865-241-4233



From jrosenquist at adaptivecomputing.com  Thu Mar  8 14:19:11 2012
From: jrosenquist at adaptivecomputing.com (John Rosenquist)
Date: Thu, 8 Mar 2012 14:19:11 -0700
Subject: [torquedev] command line -q option
In-Reply-To: <1331228450.5702.479.camel@browncoat.jics.utk.edu>
References: <4F58ED45.1040509@adaptivecomputing.com>
	<1331228450.5702.479.camel@browncoat.jics.utk.edu>
Message-ID: <CAOa9zhpe2e0s5gAs4FWCMW91pamFTkA2oEWagEM2ithbdR9x9g@mail.gmail.com>

Doh. My bad. Let me be more specific.
I'm only looking at removing the ones where -q means quiet.

pestat -q in contrib
pbsnodes -q (qnodes being an alias to pbsnodes)
tracejob


Thanks, John.

On Thu, Mar 8, 2012 at 10:40 AM, Troy Baer <tbaer at utk.edu> wrote:

> On Thu, 2012-03-08 at 10:32 -0700, John Rosenquist wrote:
> > This is John Rosenquist, I'm one of the developers at Adaptive Computing
> > working on torque.
> >
> > I was wondering if anyone uses the -q option on any of the commands
> > (pbsnodes, etc). The purpose is to suppress all output from the command.
> >
> > I would like to get rid of it.
> >
> > Please let me know if anyone is using this feature.
>
> I think you're going to have to be more specific about exactly which
> commands you mean.  (For instance, if you remove the -q option from
> qsub, you may have a riot on your hands...)
>
>        --Troy
> --
> Troy Baer, Senior HPC System Administrator
> National Institute for Computational Sciences, University of Tennessee
> http://www.nics.tennessee.edu/
> Phone:  865-241-4233
>
>
> _______________________________________________
> torquedev mailing list
> torquedev at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torquedev
>



-- 
-- 
John Rosenquist | Torque Developer
Direct Line: 801.341.4629 | Fax: 801.717.3738
1656 S. East Bay Blvd. Suite #300 | Provo, Utah 84601 | USA
Adaptive Computing, Ent.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/torquedev/attachments/20120308/fbabdd07/attachment.html 

From knielson at adaptivecomputing.com  Thu Mar  8 14:45:21 2012
From: knielson at adaptivecomputing.com (Ken Nielson)
Date: Thu, 8 Mar 2012 14:45:21 -0700
Subject: [torquedev] 2.5.11 release candidate
Message-ID: <CADvLK3dXbeJHDP3fpJP6aSe3GcZEU4HppnfizY2=c+95s40J7g@mail.gmail.com>

Hi all,

There is a release candidate for 2.5.11 now available for download. Please
try this and let us know if you find any problems.

You can download the tar ball at
http://www.adaptivecomputing.com/resources/downloads/torque/snapshots/
torque-2.5.11-snap.201203081434.tar.gz<http://www.adaptivecomputing.com/resources/downloads/torque/snapshots/torque-2.5.11-snap.201203081434.tar.gz>

Regards

Ken Nielson
Adaptive Computing
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/torquedev/attachments/20120308/0e7c9b5b/attachment.html 

From Gareth.Williams at csiro.au  Thu Mar  8 15:57:08 2012
From: Gareth.Williams at csiro.au (Gareth.Williams at csiro.au)
Date: Fri, 9 Mar 2012 09:57:08 +1100
Subject: [torquedev] command line -q option
In-Reply-To: <CAOa9zhpe2e0s5gAs4FWCMW91pamFTkA2oEWagEM2ithbdR9x9g@mail.gmail.com>
References: <4F58ED45.1040509@adaptivecomputing.com>
	<1331228450.5702.479.camel@browncoat.jics.utk.edu>
	<CAOa9zhpe2e0s5gAs4FWCMW91pamFTkA2oEWagEM2ithbdR9x9g@mail.gmail.com>
Message-ID: <007DECE986B47F4EABF823C1FBB19C620102D7AE2E65@exvic-mbx04.nexus.csiro.au>

It seems to me (without looking at code) that with tracejob, -q is effectively the same as redirecting stderr to /dev/null. So we would not be losing anything that couldn't be easily done anyway. Mostly I don't think I care.

Gareth

From: torquedev-bounces at supercluster.org [mailto:torquedev-bounces at supercluster.org] On Behalf Of John Rosenquist
Sent: Friday, 9 March 2012 8:19 AM
To: Torque Developers mailing list
Cc: Torque Users Mailing List
Subject: Re: [torquedev] command line -q option

Doh. My bad. Let me be more specific.
I'm only looking at removing the ones where -q means quiet.

pestat -q in contrib
pbsnodes -q (qnodes being an alias to pbsnodes)
tracejob


Thanks, John.
On Thu, Mar 8, 2012 at 10:40 AM, Troy Baer <tbaer at utk.edu<mailto:tbaer at utk.edu>> wrote:
On Thu, 2012-03-08 at 10:32 -0700, John Rosenquist wrote:
> This is John Rosenquist, I'm one of the developers at Adaptive Computing
> working on torque.
>
> I was wondering if anyone uses the -q option on any of the commands
> (pbsnodes, etc). The purpose is to suppress all output from the command.
>
> I would like to get rid of it.
>
> Please let me know if anyone is using this feature.
I think you're going to have to be more specific about exactly which
commands you mean.  (For instance, if you remove the -q option from
qsub, you may have a riot on your hands...)

       --Troy
--
Troy Baer, Senior HPC System Administrator
National Institute for Computational Sciences, University of Tennessee
http://www.nics.tennessee.edu/
Phone:  865-241-4233


_______________________________________________
torquedev mailing list
torquedev at supercluster.org<mailto:torquedev at supercluster.org>
http://www.supercluster.org/mailman/listinfo/torquedev



--
--
John Rosenquist | Torque Developer
Direct Line: 801.341.4629 | Fax: 801.717.3738
1656 S. East Bay Blvd. Suite #300 | Provo, Utah 84601 | USA
Adaptive Computing, Ent.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/torquedev/attachments/20120309/66f09c3c/attachment-0001.html 

From samuel at unimelb.edu.au  Thu Mar  8 21:23:11 2012
From: samuel at unimelb.edu.au (Christopher Samuel)
Date: Fri, 09 Mar 2012 15:23:11 +1100
Subject: [torquedev] command line -q option
In-Reply-To: <CAOa9zhpe2e0s5gAs4FWCMW91pamFTkA2oEWagEM2ithbdR9x9g@mail.gmail.com>
References: <4F58ED45.1040509@adaptivecomputing.com>
	<1331228450.5702.479.camel@browncoat.jics.utk.edu>
	<CAOa9zhpe2e0s5gAs4FWCMW91pamFTkA2oEWagEM2ithbdR9x9g@mail.gmail.com>
Message-ID: <4F5985AF.2040204@unimelb.edu.au>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/03/12 08:19, John Rosenquist wrote:

> Doh. My bad. Let me be more specific. I'm only looking at removing
> the ones where -q means quiet.
> 
> pestat -q in contrib pbsnodes -q (qnodes being an alias to
> pbsnodes) tracejob

We use -q routinely in tracejob.  In fact I think it should default to
on and there should be an option to show warnings (as I don't consider
complaining about not being able to read pbs_mom or pbs_sched logs on
the management node running pbs_server and moab as an error).

cheers,
Chris
- -- 
    Christopher Samuel - Senior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545
         http://www.vlsci.unimelb.edu.au/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk9Zha8ACgkQO2KABBYQAh+U7gCeJ8j7PbpGa6VuhSkdNttJ9QWx
0goAniS61H4Ve7OBFn10of9R5HIHAzuo
=vdh+
-----END PGP SIGNATURE-----

From cwest at vpac.org  Thu Mar  8 21:58:49 2012
From: cwest at vpac.org (Craig West)
Date: Fri, 09 Mar 2012 15:58:49 +1100
Subject: [torquedev] command line -q option
In-Reply-To: <4F5985AF.2040204@unimelb.edu.au>
References: <4F58ED45.1040509@adaptivecomputing.com>
	<1331228450.5702.479.camel@browncoat.jics.utk.edu>
	<CAOa9zhpe2e0s5gAs4FWCMW91pamFTkA2oEWagEM2ithbdR9x9g@mail.gmail.com>
	<4F5985AF.2040204@unimelb.edu.au>
Message-ID: <4F598E09.6090603@vpac.org>


On 09/03/12 15:23, Christopher Samuel wrote:
> On 09/03/12 08:19, John Rosenquist wrote:
>
>> Doh. My bad. Let me be more specific. I'm only looking at removing
>> the ones where -q means quiet.

Just wondering what the purpose of this would be? Are you wanting to 
reuse the "-q" flag?

>> pestat -q in contrib pbsnodes -q (qnodes being an alias to
>> pbsnodes) tracejob
>
> We use -q routinely in tracejob.  In fact I think it should default to
> on and there should be an option to show warnings (as I don't consider
> complaining about not being able to read pbs_mom or pbs_sched logs on
> the management node running pbs_server and moab as an error).

I will agree with Chris. I always use -q with tracejob and would prefer 
to have a default that doesn't show the errors. Perhaps a flag could be 
made to show warnings?


Craig.

-- 
Craig West                   Systems Manager
Victorian Partnership for Advanced Computing
110 Victoria Street, Carlton South  VIC 3053
P: +61 3 9925 4751         E: cwest at vpac.org
                          http://www.vpac.org

From jayavant.patil82 at gmail.com  Fri Mar  9 04:54:33 2012
From: jayavant.patil82 at gmail.com (Jayavant Patil)
Date: Fri, 9 Mar 2012 17:24:33 +0530
Subject: [torquedev] Solution or workaround for Mother Superior node failure
Message-ID: <CAPDKCffgUEamwy-E+XNC0Hb3hywojfzDk794DnTDWVkz3c+Kpw@mail.gmail.com>

Hi,

   I am using Torque 3.0.0 and Maui 3.3. When mother superior node fails,
the running processes (related to job) should get killed and job should get
requeued. But, I am not getting this desired behaviour. Is there any
solution or workaround available for this?

-- 

Thanks & Regards,
Jayavant Ningoji Patil
+91 9923536030.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/torquedev/attachments/20120309/bd77a2eb/attachment.html 

From knielson at adaptivecomputing.com  Fri Mar  9 10:47:50 2012
From: knielson at adaptivecomputing.com (Ken Nielson)
Date: Fri, 9 Mar 2012 10:47:50 -0700
Subject: [torquedev] New 2.5.11 snapshot available
Message-ID: <CADvLK3c_83pXKrfTEhPDEfWs7EU1wOTPdMprkO6xH9C7rpYfUQ@mail.gmail.com>

Hi all,

We fixed a segfault in pbs_mom where pbs_mom could not support 8 GPUs. The
new release candidate can be downloaded at
http://www.adaptivecomputing.com/resources/downloads/torque/snapshots/
torque-2.5.11-snap.201203091041.tar.gz<http://www.adaptivecomputing.com/resources/downloads/torque/snapshots/torque-2.5.11-snap.201203091041.tar.gz>

Please download and try this snapshot and let us know if you find any
problems.

Ken
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/torquedev/attachments/20120309/a356360d/attachment.html 

From samuel at unimelb.edu.au  Sun Mar 11 22:15:49 2012
From: samuel at unimelb.edu.au (Christopher Samuel)
Date: Mon, 12 Mar 2012 15:15:49 +1100
Subject: [torquedev] Solution or workaround for Mother Superior node
	failure
In-Reply-To: <CAPDKCffgUEamwy-E+XNC0Hb3hywojfzDk794DnTDWVkz3c+Kpw@mail.gmail.com>
References: <CAPDKCffgUEamwy-E+XNC0Hb3hywojfzDk794DnTDWVkz3c+Kpw@mail.gmail.com>
Message-ID: <4F5D7875.7070806@unimelb.edu.au>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/03/12 22:54, Jayavant Patil wrote:

> I am using Torque 3.0.0 and Maui 3.3. When mother superior node 
> fails, the running processes (related to job) should get killed
> and job should get requeued. But, I am not getting this desired 
> behaviour. Is there any solution or workaround available for this?

I'm puzzled - why do you think this should happen ?

If the pbs_mom dies on the mother superior the job should keep on
running and when you start pbs_mom with the -p option it should
inherit the jobs on that node.

cheers,
Chris
- -- 
    Christopher Samuel - Senior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545
         http://www.vlsci.unimelb.edu.au/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk9deHUACgkQO2KABBYQAh9RxACfUy7heWWXw2qsiF+tmJaEKvGf
6tAAnigLdYGCeeqJCfSCYWwndENjPRYZ
=tg3j
-----END PGP SIGNATURE-----

From alan at madllama.net  Mon Mar 12 14:38:41 2012
From: alan at madllama.net (Alan Wild)
Date: Mon, 12 Mar 2012 15:38:41 -0500
Subject: [torquedev] "Fixing" qsig -s USR1 and kill_delay on torque 2.5.x
Message-ID: <CAATyho7HnaSds-g07++1=MsrMfGwdcv4G11schzcc_ywzOzHhA@mail.gmail.com>

NOTE:  we are presently running 2.5.7, but I've confirmed that this change
is still applicable to 2.5.9.  I've not had a chance to look at 3.x or 4.x
in any way.

We recently wanted to change our kill_delay on our system to allow jobs
adequate time to properly clean up in the event of a qdel.  At the same
time I started playing with qsig and discovered that sending a USR1 signal
to process would cause it to terminate (even if the jobscript/job properly
handled SIGUSR1).

It tuns out that both issues are related to the same problem: The failure
of the user's shell (by default) to catch and properly handle signals.
This has been discussed here (and on torqueusers) several times in the past
and the general recommendation has always been to have the user add the
necessary "trap" statements to their .bashrc (or appropriate file) in
addition to putting them in their job script.

The reasons for these recommendations stems from the process hierarchy that
is created by pbs_mom:

pbs_mom,6488 -p
  `-bash,10919
     `-16398.hpdjsl001,10978 -l /var/spool/pbs/mom_priv/jobs/
16398.hpdjsl001.SC

pbs_mom launches a shell (in my case bash) which, in turn, invokes the job
script.  When the user executes a qsig or qdel... the server passes the
signal to the mom and the mom signals both of these processes.  If the job
script has the necessary trap calls in it... it, of course, handles the
signal properly, but the shell process will exit... and many shells will
exit even on on a seemingly innocuous SIGUSR1.

If the shell process exits... the pbs_mom believes the job to have died and
automatically enters into a mode where it sends a SIGTERM to the jobscript
and ~5seconds later a SIGKILL.  This happens whether regardless of the
singal the user sent (even SIGUSR1) or in the event of a qdel.  However,
given that the goal of a qdel is to remove a job... most Torque users are
probably none the wiser that it isn't going through the "correct"
termination sequence.

We have a large user community (and most are not technical enough) that I
don't reasonably expect them to be able to properly implement the changes
to their individual login files.  I've considering having our system
configuration files updated, but this would affect all users (even those
that don't submit jobs) and I we would be stuck maintaining a solution that
works for each of about five different shells we have installed.

So I wondered if there couldn't be a better way.

I looked at the pbs_mom source and found how the pbs_mom passes the script
command to invoked into the shell process.  It does so via a pipe which is
connected to the shell's stdin. So I thought, "why couldn't the shell
simply 'exec' the job script instead of running it as a simple command
line?"  It turns out that the pipe is closed shortly after the script's
path is passed to the shell so it's not like pbs_mom was going to talk to
the shell anymore... so why leave the shell running?  If the shell is no
longer running... that's one less process to have worry about catching
signals... and potentially it's less memory wasted on the compute node.

I threw together this rather small patch as a prototype:

diff -urN torque-2.5.7/src/resmom/start_exec.c
torque-2.5.7-new/src/resmom/start_exec.c
--- torque-2.5.7/src/resmom/start_exec.c        2011-06-17
17:15:57.000000000 -0500
+++ torque-2.5.7-new/src/resmom/start_exec.c    2012-03-12
13:29:13.000000000 -0500
@@ -1966,5 +1966,11 @@
                {
                int k;

+               if (strlen(buf)+5 <= MAXPATHLEN) {
+                       for (i=strlen(buf); i>=0; i--)
+                               buf[i+5] = buf[i];
+                       strncpy(buf, "exec ", 5);
+               }
+
                /* pass name of shell script on pipe */
                /* will be stdin of shell  */

...And found it to work as expected in our test environment (with
admittedly limited testing).  All this does, (if there is still space in
the buffer) is shifts everything over 5 characters and inserts "exec " at
the beginning of the command line. The shell invokes the process, which of
course, now exec's the script. The script inherits the pid of the shell as
well as its stdin/stdout/stderr so pbs_demux appears to function correctly.

Every shell I've investigated (sh, csh, ksh, bash, zsh) all appear to honor
the "exec" command in the same manner so this appears to be a viable
solution to this problem (premature shell termination) without requiring
users (or admins) to add "trap" statements to dotfiles to protect that one
process.  For the record, this doesn't get anyone off the hook about
installing trap's in the job scripts (or signal handlers in the processes
themselves), but this appears to remove one of larger barriers in
leveraging qsig(1) and extended kill_delay settings.

I'lll concede there could be a flaw in my logic, and as I stated above,
this has only had limited testing thus far, but I would love to hear what I
may have missed and why this couldn't be a viable change in Torque.

This was tested by qsub'ing the following perl script directly (no shell
job-script around it).  This code simply catches signals, prints the time
that they were received, and after the first signal is caught... prints the
time in 1 second intervals (since you'll never see the final SIGKILL you
can at least count of the seconds).

#!/usr/bin/perl -l
use constant CATCH => qw/USR1 USR2 HUP TERM INT QUIT ABRT ILL FPE SEGV ALRM
PIPE CHLD/;
my $stop;
$|=1;

@SIG{(CATCH)} = (sub { $stop||=1; print join ' ', shift, '@', scalar
localtime }) x CATCH;

sleep unless $stop;
print (scalar localtime), sleep 1 while 1;

When tested with a qdel, you'll see a TERM signal logged at the time
invocation, followed by the number of printouts which correspond with your
kill_delay setting (defaults to 2 seconds).  Finally you see a second
SIGTERM and then ~5 seconds later the output stops (because the process
receives a SIGKILL).  For the unfamiliar, when the server asks a mom to do
a SIGKILL... it is hard coded to SIGTERM first and then ~5 seconds later to
try a SIGKILL.

Without my patch above (and without adding trap statements to your .bashrc)
this script will output two SIGTERM's (typically within the same second)
with about 5 more seconds of printouts (before the final kill). mom_logs
will confirm that the initial SIGTERM terminated the shell process, and
that the mom then automatically initiated a job termination (via the second
TERM and KILL).

I also won't take any offense if someone wants to implement the patch more
efficiently, I was just trying to do what I wanted with the minimal amount
of change to the torque code.

Thanks,

-Alan

-- 
alan at madllama.net http://humbleville.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/torquedev/attachments/20120312/e67cb5f4/attachment-0001.html 

From alan at madllama.net  Tue Mar 13 08:53:52 2012
From: alan at madllama.net (Alan Wild)
Date: Tue, 13 Mar 2012 09:53:52 -0500
Subject: [torquedev] "Fixing" qsig -s USR1 and kill_delay on torque 2.5.x
In-Reply-To: <CAATyho7HnaSds-g07++1=MsrMfGwdcv4G11schzcc_ywzOzHhA@mail.gmail.com>
References: <CAATyho7HnaSds-g07++1=MsrMfGwdcv4G11schzcc_ywzOzHhA@mail.gmail.com>
Message-ID: <CAATyho5o0uFpNV=y042d+3u5xKOb2Yuj1qXV3HKR1NA5Mm+xAw@mail.gmail.com>

(wipes egg off face)... if it isn't obvious.. I haven't had to do any heavy
C-coding in a few months.

I wrote the loop in the previous pass knowing fully that strcpy() doesn't
support overlapping memory regions.  All of a sudden last night... I
recalled memmove() did.  Here's an updated patch against the latest 2.5.11
snapshot using memmove() and no loop.

Yes.. and I also realize I screwed up the diff.  That's what I get trying
to hand-tune it to not have it bounded by whatspace.  This one should apply
cleanly with  patch -p1

-Alan

diff -rN -U2 torque-2.5.11-snap.201203081434-old/src/resmom/start_exec.c
torque-2.5.11-snap.201203081434/src/resmom/start_exec.c
--- torque-2.5.11-snap.201203081434-old/src/resmom/start_exec.c 2012-03-08
15:34:57.000000000 -0600
+++ torque-2.5.11-snap.201203081434/src/resmom/start_exec.c     2012-03-13
09:47:55.000000000 -0500
@@ -1997,4 +1997,9 @@
     int k;
 +    if (strlen(buf)+5 <= MAXPATHLEN) {
+        memmove(buf+5,buf,strlen(buf)+1);
+        strncpy(buf, "exec ", 5);
+    }
+
     /* pass name of shell script on pipe */
     /* will be stdin of shell  */


On Mon, Mar 12, 2012 at 3:38 PM, Alan Wild <alan at madllama.net> wrote:


> NOTE:  we are presently running 2.5.7, but I've confirmed that this change
> is still applicable to 2.5.9.  I've not had a chance to look at 3.x or 4.x
> in any way.
>
> We recently wanted to change our kill_delay on our system to allow jobs
> adequate time to properly clean up in the event of a qdel.  At the same
> time I started playing with qsig and discovered that sending a USR1 signal
> to process would cause it to terminate (even if the jobscript/job properly
> handled SIGUSR1).
>
> It tuns out that both issues are related to the same problem: The failure
> of the user's shell (by default) to catch and properly handle signals.
> This has been discussed here (and on torqueusers) several times in the past
> and the general recommendation has always been to have the user add the
> necessary "trap" statements to their .bashrc (or appropriate file) in
> addition to putting them in their job script.
>
> The reasons for these recommendations stems from the process hierarchy
> that is created by pbs_mom:
>
> pbs_mom,6488 -p
>   `-bash,10919
>      `-16398.hpdjsl001,10978 -l /var/spool/pbs/mom_priv/jobs/
> 16398.hpdjsl001.SC <http://16398.hpdjsl001.sc/>
>
> pbs_mom launches a shell (in my case bash) which, in turn, invokes the job
> script.  When the user executes a qsig or qdel... the server passes the
> signal to the mom and the mom signals both of these processes.  If the job
> script has the necessary trap calls in it... it, of course, handles the
> signal properly, but the shell process will exit... and many shells will
> exit even on on a seemingly innocuous SIGUSR1.
>
> If the shell process exits... the pbs_mom believes the job to have died
> and automatically enters into a mode where it sends a SIGTERM to the
> jobscript and ~5seconds later a SIGKILL.  This happens whether regardless
> of the singal the user sent (even SIGUSR1) or in the event of a qdel.
> However, given that the goal of a qdel is to remove a job... most Torque
> users are probably none the wiser that it isn't going through the "correct"
> termination sequence.
>
> We have a large user community (and most are not technical enough) that I
> don't reasonably expect them to be able to properly implement the changes
> to their individual login files.  I've considering having our system
> configuration files updated, but this would affect all users (even those
> that don't submit jobs) and I we would be stuck maintaining a solution that
> works for each of about five different shells we have installed.
>
> So I wondered if there couldn't be a better way.
>
> I looked at the pbs_mom source and found how the pbs_mom passes the script
> command to invoked into the shell process.  It does so via a pipe which is
> connected to the shell's stdin. So I thought, "why couldn't the shell
> simply 'exec' the job script instead of running it as a simple command
> line?"  It turns out that the pipe is closed shortly after the script's
> path is passed to the shell so it's not like pbs_mom was going to talk to
> the shell anymore... so why leave the shell running?  If the shell is no
> longer running... that's one less process to have worry about catching
> signals... and potentially it's less memory wasted on the compute node.
>
> I threw together this rather small patch as a prototype:
>
> diff -urN torque-2.5.7/src/resmom/start_exec.c
> torque-2.5.7-new/src/resmom/start_exec.c
> --- torque-2.5.7/src/resmom/start_exec.c        2011-06-17
> 17:15:57.000000000 -0500
> +++ torque-2.5.7-new/src/resmom/start_exec.c    2012-03-12
> 13:29:13.000000000 -0500
> @@ -1966,5 +1966,11 @@
>                 {
>                 int k;
>
> +               if (strlen(buf)+5 <= MAXPATHLEN) {
> +                       for (i=strlen(buf); i>=0; i--)
> +                               buf[i+5] = buf[i];
> +                       strncpy(buf, "exec ", 5);
> +               }
> +
>                 /* pass name of shell script on pipe */
>                 /* will be stdin of shell  */
>
> ...And found it to work as expected in our test environment (with
> admittedly limited testing).  All this does, (if there is still space in
> the buffer) is shifts everything over 5 characters and inserts "exec " at
> the beginning of the command line. The shell invokes the process, which of
> course, now exec's the script. The script inherits the pid of the shell as
> well as its stdin/stdout/stderr so pbs_demux appears to function correctly.
>
> Every shell I've investigated (sh, csh, ksh, bash, zsh) all appear to
> honor the "exec" command in the same manner so this appears to be a viable
> solution to this problem (premature shell termination) without requiring
> users (or admins) to add "trap" statements to dotfiles to protect that one
> process.  For the record, this doesn't get anyone off the hook about
> installing trap's in the job scripts (or signal handlers in the processes
> themselves), but this appears to remove one of larger barriers in
> leveraging qsig(1) and extended kill_delay settings.
>
> I'lll concede there could be a flaw in my logic, and as I stated above,
> this has only had limited testing thus far, but I would love to hear what I
> may have missed and why this couldn't be a viable change in Torque.
>
> This was tested by qsub'ing the following perl script directly (no shell
> job-script around it).  This code simply catches signals, prints the time
> that they were received, and after the first signal is caught... prints the
> time in 1 second intervals (since you'll never see the final SIGKILL you
> can at least count of the seconds).
>
> #!/usr/bin/perl -l
> use constant CATCH => qw/USR1 USR2 HUP TERM INT QUIT ABRT ILL FPE SEGV
> ALRM PIPE CHLD/;
> my $stop;
> $|=1;
>  @SIG{(CATCH)} = (sub { $stop||=1; print join ' ', shift, '@', scalar
> localtime }) x CATCH;
>  sleep unless $stop;
> print (scalar localtime), sleep 1 while 1;
>
> When tested with a qdel, you'll see a TERM signal logged at the time
> invocation, followed by the number of printouts which correspond with your
> kill_delay setting (defaults to 2 seconds).  Finally you see a second
> SIGTERM and then ~5 seconds later the output stops (because the process
> receives a SIGKILL).  For the unfamiliar, when the server asks a mom to do
> a SIGKILL... it is hard coded to SIGTERM first and then ~5 seconds later to
> try a SIGKILL.
>
> Without my patch above (and without adding trap statements to your
> .bashrc) this script will output two SIGTERM's (typically within the same
> second) with about 5 more seconds of printouts (before the final kill).
> mom_logs will confirm that the initial SIGTERM terminated the shell
> process, and that the mom then automatically initiated a job termination
> (via the second TERM and KILL).
>
> I also won't take any offense if someone wants to implement the patch more
> efficiently, I was just trying to do what I wanted with the minimal amount
> of change to the torque code.
>
> Thanks,
>
> -Alan
>
> --
> alan at madllama.net http://humbleville.blogspot.com
>
>
>

-- 
alan at madllama.net http://humbleville.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/torquedev/attachments/20120313/1f86db4f/attachment.html 

From ramon.bastiaans at sara.nl  Tue Mar 13 09:05:17 2012
From: ramon.bastiaans at sara.nl (Ramon Bastiaans)
Date: Tue, 13 Mar 2012 16:05:17 +0100
Subject: [torquedev] 2.5.11 release candidate
In-Reply-To: <CADvLK3dXbeJHDP3fpJP6aSe3GcZEU4HppnfizY2=c+95s40J7g@mail.gmail.com>
References: <CADvLK3dXbeJHDP3fpJP6aSe3GcZEU4HppnfizY2=c+95s40J7g@mail.gmail.com>
Message-ID: <4F5F622D.50709@sara.nl>

Hi,

I have a little remark.

I read in the Changelog that this bug:
  * http://www.clusterresources.com/bugzilla/show_bug.cgi?id=168

Should be fixed now in 2.5.11. This is great and thanks for that.

However; it would be nice (for me the bug reporter) if you guys would 
actually log/close this in the Bugzilla ticket.

So that I may know I should download 2.5.11 ;)


Kind regards,
- Ramon.

On 8-3-2012 22:45, Ken Nielson wrote:
> Hi all,
>
> There is a release candidate for 2.5.11 now available for download. 
> Please try this and let us know if you find any problems.
>
> You can download the tar ball at 
> http://www.adaptivecomputing.com/resources/downloads/torque/snapshots/torque-2.5.11-snap.201203081434.tar.gz 
> <http://www.adaptivecomputing.com/resources/downloads/torque/snapshots/torque-2.5.11-snap.201203081434.tar.gz>
>
> Regards
>
> Ken Nielson
> Adaptive Computing

-- 
ing. R. Bastiaans, B.ICT
* Senior Systems Programmer
* Operations, Support and Development

SARA
Science Park 140     PO Box 94613
1098 XG Amsterdam NL 1090 GP Amsterdam NL
P.+31 (0)20 592 3000 F.+31 (0)20 668 3167

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/torquedev/attachments/20120313/275c6520/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4573 bytes
Desc: S/MIME Cryptographic Signature
Url : http://www.supercluster.org/pipermail/torquedev/attachments/20120313/275c6520/attachment-0001.bin 

From dbeer at adaptivecomputing.com  Tue Mar 13 10:04:41 2012
From: dbeer at adaptivecomputing.com (David Beer)
Date: Tue, 13 Mar 2012 10:04:41 -0600
Subject: [torquedev] "Fixing" qsig -s USR1 and kill_delay on torque 2.5.x
In-Reply-To: <CAATyho5o0uFpNV=y042d+3u5xKOb2Yuj1qXV3HKR1NA5Mm+xAw@mail.gmail.com>
References: <CAATyho7HnaSds-g07++1=MsrMfGwdcv4G11schzcc_ywzOzHhA@mail.gmail.com>
	<CAATyho5o0uFpNV=y042d+3u5xKOb2Yuj1qXV3HKR1NA5Mm+xAw@mail.gmail.com>
Message-ID: <CAFUQeZ1bpV-uzgm-qgQvAZnDZVY5J3bPvHSnkPOh-La4_hP+3g@mail.gmail.com>

Having done the work it takes to configure these signals to work in the
system's current state, I'm all for addressing this issue. I'm wondering if
there are any community concerns about this change? Do you see any possible
regressions? What are the risks? Should we make this change something that
only happens if you turn it on in the mom config file? In some ways I like
this option because it is easy to turn off if there are regressions, but on
the other hand the kill_delay functionality is so cumbersome to set up its
essentially broken now. I'm interested to hear the community's input on
this patch.

David

On Tue, Mar 13, 2012 at 8:53 AM, Alan Wild <alan at madllama.net> wrote:

> (wipes egg off face)... if it isn't obvious.. I haven't had to do any
> heavy C-coding in a few months.
>
> I wrote the loop in the previous pass knowing fully that strcpy() doesn't
> support overlapping memory regions.  All of a sudden last night... I
> recalled memmove() did.  Here's an updated patch against the latest 2.5.11
> snapshot using memmove() and no loop.
>
> Yes.. and I also realize I screwed up the diff.  That's what I get trying
> to hand-tune it to not have it bounded by whatspace.  This one should apply
> cleanly with  patch -p1
>
> -Alan
>
> diff -rN -U2 torque-2.5.11-snap.201203081434-old/src/resmom/start_exec.c
> torque-2.5.11-snap.201203081434/src/resmom/start_exec.c
> --- torque-2.5.11-snap.201203081434-old/src/resmom/start_exec.c 2012-03-08
> 15:34:57.000000000 -0600
> +++ torque-2.5.11-snap.201203081434/src/resmom/start_exec.c     2012-03-13
> 09:47:55.000000000 -0500
> @@ -1997,4 +1997,9 @@
>      int k;
>  +    if (strlen(buf)+5 <= MAXPATHLEN) {
> +        memmove(buf+5,buf,strlen(buf)+1);
> +        strncpy(buf, "exec ", 5);
> +    }
> +
>      /* pass name of shell script on pipe */
>      /* will be stdin of shell  */
>
>
> On Mon, Mar 12, 2012 at 3:38 PM, Alan Wild <alan at madllama.net> wrote:
>
>
>> NOTE:  we are presently running 2.5.7, but I've confirmed that this
>> change is still applicable to 2.5.9.  I've not had a chance to look at 3.x
>> or 4.x in any way.
>>
>> We recently wanted to change our kill_delay on our system to allow jobs
>> adequate time to properly clean up in the event of a qdel.  At the same
>> time I started playing with qsig and discovered that sending a USR1 signal
>> to process would cause it to terminate (even if the jobscript/job properly
>> handled SIGUSR1).
>>
>> It tuns out that both issues are related to the same problem: The failure
>> of the user's shell (by default) to catch and properly handle signals.
>> This has been discussed here (and on torqueusers) several times in the past
>> and the general recommendation has always been to have the user add the
>> necessary "trap" statements to their .bashrc (or appropriate file) in
>> addition to putting them in their job script.
>>
>> The reasons for these recommendations stems from the process hierarchy
>> that is created by pbs_mom:
>>
>> pbs_mom,6488 -p
>>   `-bash,10919
>>      `-16398.hpdjsl001,10978 -l /var/spool/pbs/mom_priv/jobs/
>> 16398.hpdjsl001.SC <http://16398.hpdjsl001.sc/>
>>
>> pbs_mom launches a shell (in my case bash) which, in turn, invokes the
>> job script.  When the user executes a qsig or qdel... the server passes the
>> signal to the mom and the mom signals both of these processes.  If the job
>> script has the necessary trap calls in it... it, of course, handles the
>> signal properly, but the shell process will exit... and many shells will
>> exit even on on a seemingly innocuous SIGUSR1.
>>
>> If the shell process exits... the pbs_mom believes the job to have died
>> and automatically enters into a mode where it sends a SIGTERM to the
>> jobscript and ~5seconds later a SIGKILL.  This happens whether regardless
>> of the singal the user sent (even SIGUSR1) or in the event of a qdel.
>> However, given that the goal of a qdel is to remove a job... most Torque
>> users are probably none the wiser that it isn't going through the "correct"
>> termination sequence.
>>
>> We have a large user community (and most are not technical enough) that I
>> don't reasonably expect them to be able to properly implement the changes
>> to their individual login files.  I've considering having our system
>> configuration files updated, but this would affect all users (even those
>> that don't submit jobs) and I we would be stuck maintaining a solution that
>> works for each of about five different shells we have installed.
>>
>> So I wondered if there couldn't be a better way.
>>
>> I looked at the pbs_mom source and found how the pbs_mom passes the
>> script command to invoked into the shell process.  It does so via a pipe
>> which is connected to the shell's stdin. So I thought, "why couldn't the
>> shell simply 'exec' the job script instead of running it as a simple
>> command line?"  It turns out that the pipe is closed shortly after the
>> script's path is passed to the shell so it's not like pbs_mom was going to
>> talk to the shell anymore... so why leave the shell running?  If the shell
>> is no longer running... that's one less process to have worry about
>> catching signals... and potentially it's less memory wasted on the compute
>> node.
>>
>> I threw together this rather small patch as a prototype:
>>
>> diff -urN torque-2.5.7/src/resmom/start_exec.c
>> torque-2.5.7-new/src/resmom/start_exec.c
>> --- torque-2.5.7/src/resmom/start_exec.c        2011-06-17
>> 17:15:57.000000000 -0500
>> +++ torque-2.5.7-new/src/resmom/start_exec.c    2012-03-12
>> 13:29:13.000000000 -0500
>> @@ -1966,5 +1966,11 @@
>>                 {
>>                 int k;
>>
>> +               if (strlen(buf)+5 <= MAXPATHLEN) {
>> +                       for (i=strlen(buf); i>=0; i--)
>> +                               buf[i+5] = buf[i];
>> +                       strncpy(buf, "exec ", 5);
>> +               }
>> +
>>                 /* pass name of shell script on pipe */
>>                 /* will be stdin of shell  */
>>
>> ...And found it to work as expected in our test environment (with
>> admittedly limited testing).  All this does, (if there is still space in
>> the buffer) is shifts everything over 5 characters and inserts "exec " at
>> the beginning of the command line. The shell invokes the process, which of
>> course, now exec's the script. The script inherits the pid of the shell as
>> well as its stdin/stdout/stderr so pbs_demux appears to function correctly.
>>
>> Every shell I've investigated (sh, csh, ksh, bash, zsh) all appear to
>> honor the "exec" command in the same manner so this appears to be a viable
>> solution to this problem (premature shell termination) without requiring
>> users (or admins) to add "trap" statements to dotfiles to protect that one
>> process.  For the record, this doesn't get anyone off the hook about
>> installing trap's in the job scripts (or signal handlers in the processes
>> themselves), but this appears to remove one of larger barriers in
>> leveraging qsig(1) and extended kill_delay settings.
>>
>> I'lll concede there could be a flaw in my logic, and as I stated above,
>> this has only had limited testing thus far, but I would love to hear what I
>> may have missed and why this couldn't be a viable change in Torque.
>>
>> This was tested by qsub'ing the following perl script directly (no shell
>> job-script around it).  This code simply catches signals, prints the time
>> that they were received, and after the first signal is caught... prints the
>> time in 1 second intervals (since you'll never see the final SIGKILL you
>> can at least count of the seconds).
>>
>> #!/usr/bin/perl -l
>> use constant CATCH => qw/USR1 USR2 HUP TERM INT QUIT ABRT ILL FPE SEGV
>> ALRM PIPE CHLD/;
>> my $stop;
>> $|=1;
>>
>>  @SIG{(CATCH)} = (sub { $stop||=1; print join ' ', shift, '@', scalar
> localtime }) x CATCH;
>  sleep unless $stop;
> print (scalar localtime), sleep 1 while 1;
> When tested with a qdel, you'll see a TERM signal logged at the time
> invocation, followed by the number of printouts which correspond with your
> kill_delay setting (defaults to 2 seconds).  Finally you see a second
> SIGTERM and then ~5 seconds later the output stops (because the process
> receives a SIGKILL).  For the unfamiliar, when the server asks a mom to do
> a SIGKILL... it is hard coded to SIGTERM first and then ~5 seconds later to
> try a SIGKILL.
>
> Without my patch above (and without adding trap statements to your
> .bashrc) this script will output two SIGTERM's (typically within the same
> second) with about 5 more seconds of printouts (before the final kill).
> mom_logs will confirm that the initial SIGTERM terminated the shell
> process, and that the mom then automatically initiated a job termination
> (via the second TERM and KILL).
>
> I also won't take any offense if someone wants to implement the patch more
> efficiently, I was just trying to do what I wanted with the minimal amount
> of change to the torque code.
>
> Thanks,
>
> -Alan
>
> --
> alan at madllama.net http://humbleville.blogspot.com
>
>
>
>
> --
> alan at madllama.net http://humbleville.blogspot.com
>
>
> _______________________________________________
> torquedev mailing list
> torquedev at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torquedev
>
>


-- 
David Beer | Software Engineer
Adaptive Computing
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/torquedev/attachments/20120313/68bdfe0f/attachment.html 

From bugzilla-daemon at supercluster.org  Tue Mar 13 10:07:36 2012
From: bugzilla-daemon at supercluster.org (bugzilla-daemon at supercluster.org)
Date: Tue, 13 Mar 2012 10:07:36 -0600 (MDT)
Subject: [torquedev] [Bug 168] 2.5(.9) qsub does not seem to accept comma
 seperated -W argument
In-Reply-To: <bug-168-38@http.www.clusterresources.com/bugzilla/>
References: <bug-168-38@http.www.clusterresources.com/bugzilla/>
Message-ID: <20120313160736.7E6264121046@http.supercluster.org>

http://www.clusterresources.com/bugzilla/show_bug.cgi?id=168

Ken Nielson <knielson at adaptivecomputing.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |FIXED

--- Comment #6 from Ken Nielson <knielson at adaptivecomputing.com> 2012-03-13 10:07:36 MDT ---
Fixed in 2.5.11 revision 5803

-- 
Configure bugmail: http://www.clusterresources.com/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

From knielson at adaptivecomputing.com  Tue Mar 13 10:24:26 2012
From: knielson at adaptivecomputing.com (Ken Nielson)
Date: Tue, 13 Mar 2012 10:24:26 -0600
Subject: [torquedev] "Fixing" qsig -s USR1 and kill_delay on torque 2.5.x
In-Reply-To: <CAATyho7HnaSds-g07++1=MsrMfGwdcv4G11schzcc_ywzOzHhA@mail.gmail.com>
References: <CAATyho7HnaSds-g07++1=MsrMfGwdcv4G11schzcc_ywzOzHhA@mail.gmail.com>
Message-ID: <CADvLK3dhjWYNqjYgficcs9EEuey+KW7h=9HBvgq9axg9tDx8FA@mail.gmail.com>

Alan,

I'm sorry I did not get to this sooner. I would like add this and test it.
We are up against a deadline to get torque 2.5.11 out. I would like to put
this in for 2.5.12 and get some testing for it. We can do a 2.5.12 release
when we feel confident in this fix.

Regards

Ken

On Mon, Mar 12, 2012 at 2:38 PM, Alan Wild <alan at madllama.net> wrote:

> NOTE:  we are presently running 2.5.7, but I've confirmed that this change
> is still applicable to 2.5.9.  I've not had a chance to look at 3.x or 4.x
> in any way.
>
> We recently wanted to change our kill_delay on our system to allow jobs
> adequate time to properly clean up in the event of a qdel.  At the same
> time I started playing with qsig and discovered that sending a USR1 signal
> to process would cause it to terminate (even if the jobscript/job properly
> handled SIGUSR1).
>
> It tuns out that both issues are related to the same problem: The failure
> of the user's shell (by default) to catch and properly handle signals.
> This has been discussed here (and on torqueusers) several times in the past
> and the general recommendation has always been to have the user add the
> necessary "trap" statements to their .bashrc (or appropriate file) in
> addition to putting them in their job script.
>
> The reasons for these recommendations stems from the process hierarchy
> that is created by pbs_mom:
>
> pbs_mom,6488 -p
>   `-bash,10919
>      `-16398.hpdjsl001,10978 -l /var/spool/pbs/mom_priv/jobs/
> 16398.hpdjsl001.SC
>
> pbs_mom launches a shell (in my case bash) which, in turn, invokes the job
> script.  When the user executes a qsig or qdel... the server passes the
> signal to the mom and the mom signals both of these processes.  If the job
> script has the necessary trap calls in it... it, of course, handles the
> signal properly, but the shell process will exit... and many shells will
> exit even on on a seemingly innocuous SIGUSR1.
>
> If the shell process exits... the pbs_mom believes the job to have died
> and automatically enters into a mode where it sends a SIGTERM to the
> jobscript and ~5seconds later a SIGKILL.  This happens whether regardless
> of the singal the user sent (even SIGUSR1) or in the event of a qdel.
> However, given that the goal of a qdel is to remove a job... most Torque
> users are probably none the wiser that it isn't going through the "correct"
> termination sequence.
>
> We have a large user community (and most are not technical enough) that I
> don't reasonably expect them to be able to properly implement the changes
> to their individual login files.  I've considering having our system
> configuration files updated, but this would affect all users (even those
> that don't submit jobs) and I we would be stuck maintaining a solution that
> works for each of about five different shells we have installed.
>
> So I wondered if there couldn't be a better way.
>
> I looked at the pbs_mom source and found how the pbs_mom passes the script
> command to invoked into the shell process.  It does so via a pipe which is
> connected to the shell's stdin. So I thought, "why couldn't the shell
> simply 'exec' the job script instead of running it as a simple command
> line?"  It turns out that the pipe is closed shortly after the script's
> path is passed to the shell so it's not like pbs_mom was going to talk to
> the shell anymore... so why leave the shell running?  If the shell is no
> longer running... that's one less process to have worry about catching
> signals... and potentially it's less memory wasted on the compute node.
>
> I threw together this rather small patch as a prototype:
>
> diff -urN torque-2.5.7/src/resmom/start_exec.c
> torque-2.5.7-new/src/resmom/start_exec.c
> --- torque-2.5.7/src/resmom/start_exec.c        2011-06-17
> 17:15:57.000000000 -0500
> +++ torque-2.5.7-new/src/resmom/start_exec.c    2012-03-12
> 13:29:13.000000000 -0500
> @@ -1966,5 +1966,11 @@
>                 {
>                 int k;
>
> +               if (strlen(buf)+5 <= MAXPATHLEN) {
> +                       for (i=strlen(buf); i>=0; i--)
> +                               buf[i+5] = buf[i];
> +                       strncpy(buf, "exec ", 5);
> +               }
> +
>                 /* pass name of shell script on pipe */
>                 /* will be stdin of shell  */
>
> ...And found it to work as expected in our test environment (with
> admittedly limited testing).  All this does, (if there is still space in
> the buffer) is shifts everything over 5 characters and inserts "exec " at
> the beginning of the command line. The shell invokes the process, which of
> course, now exec's the script. The script inherits the pid of the shell as
> well as its stdin/stdout/stderr so pbs_demux appears to function correctly.
>
> Every shell I've investigated (sh, csh, ksh, bash, zsh) all appear to
> honor the "exec" command in the same manner so this appears to be a viable
> solution to this problem (premature shell termination) without requiring
> users (or admins) to add "trap" statements to dotfiles to protect that one
> process.  For the record, this doesn't get anyone off the hook about
> installing trap's in the job scripts (or signal handlers in the processes
> themselves), but this appears to remove one of larger barriers in
> leveraging qsig(1) and extended kill_delay settings.
>
> I'lll concede there could be a flaw in my logic, and as I stated above,
> this has only had limited testing thus far, but I would love to hear what I
> may have missed and why this couldn't be a viable change in Torque.
>
> This was tested by qsub'ing the following perl script directly (no shell
> job-script around it).  This code simply catches signals, prints the time
> that they were received, and after the first signal is caught... prints the
> time in 1 second intervals (since you'll never see the final SIGKILL you
> can at least count of the seconds).
>
> #!/usr/bin/perl -l
> use constant CATCH => qw/USR1 USR2 HUP TERM INT QUIT ABRT ILL FPE SEGV
> ALRM PIPE CHLD/;
> my $stop;
> $|=1;
>
> @SIG{(CATCH)} = (sub { $stop||=1; print join ' ', shift, '@', scalar
> localtime }) x CATCH;
>
> sleep unless $stop;
> print (scalar localtime), sleep 1 while 1;
>
> When tested with a qdel, you'll see a TERM signal logged at the time
> invocation, followed by the number of printouts which correspond with your
> kill_delay setting (defaults to 2 seconds).  Finally you see a second
> SIGTERM and then ~5 seconds later the output stops (because the process
> receives a SIGKILL).  For the unfamiliar, when the server asks a mom to do
> a SIGKILL... it is hard coded to SIGTERM first and then ~5 seconds later to
> try a SIGKILL.
>
> Without my patch above (and without adding trap statements to your
> .bashrc) this script will output two SIGTERM's (typically within the same
> second) with about 5 more seconds of printouts (before the final kill).
> mom_logs will confirm that the initial SIGTERM terminated the shell
> process, and that the mom then automatically initiated a job termination
> (via the second TERM and KILL).
>
> I also won't take any offense if someone wants to implement the patch more
> efficiently, I was just trying to do what I wanted with the minimal amount
> of change to the torque code.
>
> Thanks,
>
> -Alan
>
> --
> alan at madllama.net http://humbleville.blogspot.com
>
>
>
> _______________________________________________
> torquedev mailing list
> torquedev at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torquedev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/torquedev/attachments/20120313/22ab13b4/attachment-0001.html 

From knielson at adaptivecomputing.com  Tue Mar 13 10:43:31 2012
From: knielson at adaptivecomputing.com (Ken Nielson)
Date: Tue, 13 Mar 2012 10:43:31 -0600
Subject: [torquedev] TORQUE 2.5.11 available
Message-ID: <CADvLK3cikpvoMtZw00rAeSVwZiGWE=ybt_4dHvm9k-11KxdtNw@mail.gmail.com>

Hi all,

TORQUE 2.5.11 is now available.

Please read the CHANGELOG and Release_Notes for changes and updates.

TORQUE 2.5.11 can be downloaded at
http://www.adaptivecomputing.com/resources/downloads/torque/
torque-2.5.11.tar.gz<http://www.adaptivecomputing.com/resources/downloads/torque/torque-2.5.11.tar.gz>

Thanks to everyone who made this new revision possible.

Regards

Ken Nielson
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/torquedev/attachments/20120313/5f41acf5/attachment.html 

From alan at madllama.net  Tue Mar 13 13:16:47 2012
From: alan at madllama.net (Alan Wild)
Date: Tue, 13 Mar 2012 14:16:47 -0500
Subject: [torquedev] torquedev Digest, Vol 76, Issue 5
In-Reply-To: <mailman.1125.1331655869.3252.torquedev@supercluster.org>
References: <mailman.1125.1331655869.3252.torquedev@supercluster.org>
Message-ID: <CAATyho5NvBLaoYY8qFdfkjF3xhYhuv=LuoRJK9Z0RhiKhi2idw@mail.gmail.com>

I'm waiting to see if the community can find a hole in my logic.  I'll have
to admit that I'm relatively new to pbs and I realize there are a lot of
things done for historical (or platform compatibility reasons) I don't
appreciate.

One issue I see.. what happens if the user' shell accepts:

/path/to/some/job/script.SC

which Torque does today, but not

exec /path/tp/some/job/script.SC

.. which is what I'm proposing.  I checked the shells that I have on hand
(sh, ksh, bash, csh, tcsh, zsh) and all of these appear fine, but I'm only
checking on RHEL Linux and I fully appreciate that other OS's sh and csh
implementations can be quite different (but I would be quite surprised if
they can't exec).  While this covers the 95% case, there could be some
esoteric shell out there that proves incompatible.

That said, I find myself wondering why Torque uses the users's shell
doesn't just use /bin/sh (which is pretty much going to exist on any Unix
machine out there).  I'm guessing that there could be some subtle things in
the user's environment that make the job script work under their shell that
could fail if invoked under /bin/sh (assuming they aren't an sh-user)...
but at least Torque would be gauranteed of the shell's behavior (and it
would gaurantee that the above "exec" issue couldn't happen).  Personally,
I'm tempted to suggest that Torque should have a --always-use-sh compile
time option :)

Of course, the environment problem I'm describing should be quite familiar
to anyone that uses cron and people have worked around that for years
(decades?) so I would think some sites could live with that option.

Again, I'm relatively new and there may be some subtle interaction between
pbs_mom, job_scripts and pbs_demux I'm not picking up on, but in the
testing I've been able to do thus far... it works fine for me.

-Alan

On Tue, Mar 13, 2012 at 11:24 AM, <torquedev-request at supercluster.org>wrote:
----------------------------------------------------------------------

Message: 1
Date: Tue, 13 Mar 2012 10:04:41 -0600
From: David Beer <dbeer at adaptivecomputing.com>
Subject: Re: [torquedev] "Fixing" qsig -s USR1 and kill_delay on
       torque 2.5.x
To: Torque Developers mailing list <torquedev at supercluster.org>
Message-ID:
       <CAFUQeZ1bpV-uzgm-qgQvAZnDZVY5J3bPvHSnkPOh-La4_hP+3g at mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Having done the work it takes to configure these signals to work in the
system's current state, I'm all for addressing this issue. I'm wondering if
there are any community concerns about this change? Do you see any possible
regressions? What are the risks? Should we make this change something that
only happens if you turn it on in the mom config file? In some ways I like
this option because it is easy to turn off if there are regressions, but on
the other hand the kill_delay functionality is so cumbersome to set up its
essentially broken now. I'm interested to hear the community's input on
this patch.

David

On Tue, Mar 13, 2012 at 8:53 AM, Alan Wild <alan at madllama.net> wrote:

> (wipes egg off face)... if it isn't obvious.. I haven't had to do any
> heavy C-coding in a few months.
>
> I wrote the loop in the previous pass knowing fully that strcpy() doesn't
> support overlapping memory regions.  All of a sudden last night... I
> recalled memmove() did.  Here's an updated patch against the latest 2.5.11
> snapshot using memmove() and no loop.
>
> Yes.. and I also realize I screwed up the diff.  That's what I get trying
> to hand-tune it to not have it bounded by whatspace.  This one should
apply
> cleanly with  patch -p1
>
> -Alan
>
> diff -rN -U2 torque-2.5.11-snap.201203081434-old/src/resmom/start_exec.c
> torque-2.5.11-snap.201203081434/src/resmom/start_exec.c
> --- torque-2.5.11-snap.201203081434-old/src/resmom/start_exec.c 2012-03-08
> 15:34:57.000000000 -0600
> +++ torque-2.5.11-snap.201203081434/src/resmom/start_exec.c     2012-03-13
> 09:47:55.000000000 -0500
> @@ -1997,4 +1997,9 @@
>      int k;
>  +    if (strlen(buf)+5 <= MAXPATHLEN) {
> +        memmove(buf+5,buf,strlen(buf)+1);
> +        strncpy(buf, "exec ", 5);
> +    }
> +
>      /* pass name of shell script on pipe */
>      /* will be stdin of shell  */
> On Mon, Mar 12, 2012 at 3:38 PM, Alan Wild <alan at madllama.net> wrote:
>
>
>> NOTE:  we are presently running 2.5.7, but I've confirmed that this
>> change is still applicable to 2.5.9.  I've not had a chance to look at
3.x
>> or 4.x in any way.
>>
>> We recently wanted to change our kill_delay on our system to allow jobs
>> adequate time to properly clean up in the event of a qdel.  At the same
>> time I started playing with qsig and discovered that sending a USR1
signal
>> to process would cause it to terminate (even if the jobscript/job
properly
>> handled SIGUSR1).
>>
>> It tuns out that both issues are related to the same problem: The failure
>> of the user's shell (by default) to catch and properly handle signals.
>> This has been discussed here (and on torqueusers) several times in the
past
>> and the general recommendation has always been to have the user add the
>> necessary "trap" statements to their .bashrc (or appropriate file) in
>> addition to putting them in their job script.
>>
>> The reasons for these recommendations stems from the process hierarchy
>> that is created by pbs_mom:
>>
>> pbs_mom,6488 -p
>>   `-bash,10919
>>      `-16398.hpdjsl001,10978 -l /var/spool/pbs/mom_priv/jobs/
>> 16398.hpdjsl001.SC <http://16398.hpdjsl001.sc/> <
http://16398.hpdjsl001.sc/>
>>
>> pbs_mom launches a shell (in my case bash) which, in turn, invokes the
>> job script.  When the user executes a qsig or qdel... the server passes
the
>> signal to the mom and the mom signals both of these processes.  If the
job
>> script has the necessary trap calls in it... it, of course, handles the
>> signal properly, but the shell process will exit... and many shells will
>> exit even on on a seemingly innocuous SIGUSR1.
>>
>> If the shell process exits... the pbs_mom believes the job to have died
>> and automatically enters into a mode where it sends a SIGTERM to the
>> jobscript and ~5seconds later a SIGKILL.  This happens whether regardless
>> of the singal the user sent (even SIGUSR1) or in the event of a qdel.
>> However, given that the goal of a qdel is to remove a job... most Torque
>> users are probably none the wiser that it isn't going through the
"correct"
>> termination sequence.
>>
>> We have a large user community (and most are not technical enough) that I
>> don't reasonably expect them to be able to properly implement the changes
>> to their individual login files.  I've considering having our system
>> configuration files updated, but this would affect all users (even those
>> that don't submit jobs) and I we would be stuck maintaining a solution
that
>> works for each of about five different shells we have installed.
>>
>> So I wondered if there couldn't be a better way.
>>
>> I looked at the pbs_mom source and found how the pbs_mom passes the
>> script command to invoked into the shell process.  It does so via a pipe
>> which is connected to the shell's stdin. So I thought, "why couldn't the
>> shell simply 'exec' the job script instead of running it as a simple
>> command line?"  It turns out that the pipe is closed shortly after the
>> script's path is passed to the shell so it's not like pbs_mom was going
to
>> talk to the shell anymore... so why leave the shell running?  If the
shell
>> is no longer running... that's one less process to have worry about
>> catching signals... and potentially it's less memory wasted on the
compute
>> node.
>>
>> I threw together this rather small patch as a prototype:
>>
>> diff -urN torque-2.5.7/src/resmom/start_exec.c
>> torque-2.5.7-new/src/resmom/start_exec.c
>> --- torque-2.5.7/src/resmom/start_exec.c        2011-06-17
>> 17:15:57.000000000 -0500
>> +++ torque-2.5.7-new/src/resmom/start_exec.c    2012-03-12
>> 13:29:13.000000000 -0500
>> @@ -1966,5 +1966,11 @@
>>                 {
>>                 int k;
>>
>> +               if (strlen(buf)+5 <= MAXPATHLEN) {
>> +                       for (i=strlen(buf); i>=0; i--)
>> +                               buf[i+5] = buf[i];
>> +                       strncpy(buf, "exec ", 5);
>> +               }
>> +
>>                 /* pass name of shell script on pipe */
>>                 /* will be stdin of shell  */
>>
>> ...And found it to work as expected in our test environment (with
>> admittedly limited testing).  All this does, (if there is still space in
>> the buffer) is shifts everything over 5 characters and inserts "exec " at
>> the beginning of the command line. The shell invokes the process, which
of
>> course, now exec's the script. The script inherits the pid of the shell
as
>> well as its stdin/stdout/stderr so pbs_demux appears to function
correctly.
>>
>> Every shell I've investigated (sh, csh, ksh, bash, zsh) all appear to
>> honor the "exec" command in the same manner so this appears to be a
viable
>> solution to this problem (premature shell termination) without requiring
>> users (or admins) to add "trap" statements to dotfiles to protect that
one
>> process.  For the record, this doesn't get anyone off the hook about
>> installing trap's in the job scripts (or signal handlers in the processes
>> themselves), but this appears to remove one of larger barriers in
>> leveraging qsig(1) and extended kill_delay settings.
>>
>> I'lll concede there could be a flaw in my logic, and as I stated above,
>> this has only had limited testing thus far, but I would love to hear
what I
>> may have missed and why this couldn't be a viable change in Torque.
>>
>> This was tested by qsub'ing the following perl script directly (no shell
>> job-script around it).  This code simply catches signals, prints the time
>> that they were received, and after the first signal is caught... prints
the
>> time in 1 second intervals (since you'll never see the final SIGKILL you
>> can at least count of the seconds).
>>
>> #!/usr/bin/perl -l
>> use constant CATCH => qw/USR1 USR2 HUP TERM INT QUIT ABRT ILL FPE SEGV
>> ALRM PIPE CHLD/;
>> my $stop;
>> $|=1;
>>
>>  @SIG{(CATCH)} = (sub { $stop||=1; print join ' ', shift, '@', scalar
> localtime }) x CATCH;
>  sleep unless $stop;
> print (scalar localtime), sleep 1 while 1;
> When tested with a qdel, you'll see a TERM signal logged at the time
> invocation, followed by the number of printouts which correspond with your
> kill_delay setting (defaults to 2 seconds).  Finally you see a second
> SIGTERM and then ~5 seconds later the output stops (because the process
> receives a SIGKILL).  For the unfamiliar, when the server asks a mom to do
> a SIGKILL... it is hard coded to SIGTERM first and then ~5 seconds later
to
> try a SIGKILL.
>
> Without my patch above (and without adding trap statements to your
> .bashrc) this script will output two SIGTERM's (typically within the same
> second) with about 5 more seconds of printouts (before the final kill).
> mom_logs will confirm that the initial SIGTERM terminated the shell
> process, and that the mom then automatically initiated a job termination
> (via the second TERM and KILL).
>
> I also won't take any offense if someone wants to implement the patch more
> efficiently, I was just trying to do what I wanted with the minimal amount
> of change to the torque code.
>
> Thanks,
>
> -Alan
>
> --
> alan at madllama.net http://humbleville.blogspot.com
>
>
>
>
> --
> alan at madllama.net http://humbleville.blogspot.com
>
>
> _______________________________________________
> torquedev mailing list
> torquedev at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torquedev
>
>


--
David Beer | Software Engineer
Adaptive Computing
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://www.supercluster.org/pipermail/torquedev/attachments/20120313/68bdfe0f/attachment-0001.html

------------------------------

Message: 2
Date: Tue, 13 Mar 2012 10:07:36 -0600 (MDT)
From: bugzilla-daemon at supercluster.org
Subject: [torquedev] [Bug 168] 2.5(.9) qsub does not seem to accept
       comma seperated -W argument
To: torquedev at supercluster.org
Message-ID: <20120313160736.7E6264121046 at http.supercluster.org>
Content-Type: text/plain; charset="UTF-8"

http://www.clusterresources.com/bugzilla/show_bug.cgi?id=168

Ken Nielson <knielson at adaptivecomputing.com> changed:

          What    |Removed                     |Added
----------------------------------------------------------------------------
            Status|NEW                         |RESOLVED
        Resolution|                            |FIXED

--- Comment #6 from Ken Nielson <knielson at adaptivecomputing.com> 2012-03-13
10:07:36 MDT ---
Fixed in 2.5.11 revision 5803

--
Configure bugmail:
http://www.clusterresources.com/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.


------------------------------

Message: 3
Date: Tue, 13 Mar 2012 10:24:26 -0600
From: Ken Nielson <knielson at adaptivecomputing.com>
Subject: Re: [torquedev] "Fixing" qsig -s USR1 and kill_delay on
       torque 2.5.x
To: Torque Developers mailing list <torquedev at supercluster.org>
Message-ID:
       <CADvLK3dhjWYNqjYgficcs9EEuey+KW7h=9HBvgq9axg9tDx8FA at mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Alan,

I'm sorry I did not get to this sooner. I would like add this and test it.
We are up against a deadline to get torque 2.5.11 out. I would like to put
this in for 2.5.12 and get some testing for it. We can do a 2.5.12 release
when we feel confident in this fix.

Regards

Ken

On Mon, Mar 12, 2012 at 2:38 PM, Alan Wild <alan at madllama.net> wrote:

> NOTE:  we are presently running 2.5.7, but I've confirmed that this change
> is still applicable to 2.5.9.  I've not had a chance to look at 3.x or 4.x
> in any way.
>
> We recently wanted to change our kill_delay on our system to allow jobs
> adequate time to properly clean up in the event of a qdel.  At the same
> time I started playing with qsig and discovered that sending a USR1 signal
> to process would cause it to terminate (even if the jobscript/job properly
> handled SIGUSR1).
>
> It tuns out that both issues are related to the same problem: The failure
> of the user's shell (by default) to catch and properly handle signals.
> This has been discussed here (and on torqueusers) several times in the
past
> and the general recommendation has always been to have the user add the
> necessary "trap" statements to their .bashrc (or appropriate file) in
> addition to putting them in their job script.
>
> The reasons for these recommendations stems from the process hierarchy
> that is created by pbs_mom:
>
> pbs_mom,6488 -p
>   `-bash,10919
>      `-16398.hpdjsl001,10978 -l /var/spool/pbs/mom_priv/jobs/
> 16398.hpdjsl001.SC <http://16398.hpdjsl001.sc/>
>
> pbs_mom launches a shell (in my case bash) which, in turn, invokes the job
> script.  When the user executes a qsig or qdel... the server passes the
> signal to the mom and the mom signals both of these processes.  If the job
> script has the necessary trap calls in it... it, of course, handles the
> signal properly, but the shell process will exit... and many shells will
> exit even on on a seemingly innocuous SIGUSR1.
>
> If the shell process exits... the pbs_mom believes the job to have died
> and automatically enters into a mode where it sends a SIGTERM to the
> jobscript and ~5seconds later a SIGKILL.  This happens whether regardless
> of the singal the user sent (even SIGUSR1) or in the event of a qdel.
> However, given that the goal of a qdel is to remove a job... most Torque
> users are probably none the wiser that it isn't going through the
"correct"
> termination sequence.
>
> We have a large user community (and most are not technical enough) that I
> don't reasonably expect them to be able to properly implement the changes
> to their individual login files.  I've considering having our system
> configuration files updated, but this would affect all users (even those
> that don't submit jobs) and I we would be stuck maintaining a solution
that
> works for each of about five different shells we have installed.
>
> So I wondered if there couldn't be a better way.
>
> I looked at the pbs_mom source and found how the pbs_mom passes the script
> command to invoked into the shell process.  It does so via a pipe which is
> connected to the shell's stdin. So I thought, "why couldn't the shell
> simply 'exec' the job script instead of running it as a simple command
> line?"  It turns out that the pipe is closed shortly after the script's
> path is passed to the shell so it's not like pbs_mom was going to talk to
> the shell anymore... so why leave the shell running?  If the shell is no
> longer running... that's one less process to have worry about catching
> signals... and potentially it's less memory wasted on the compute node.
>
> I threw together this rather small patch as a prototype:
>
> diff -urN torque-2.5.7/src/resmom/start_exec.c
> torque-2.5.7-new/src/resmom/start_exec.c
> --- torque-2.5.7/src/resmom/start_exec.c        2011-06-17
> 17:15:57.000000000 -0500
> +++ torque-2.5.7-new/src/resmom/start_exec.c    2012-03-12
> 13:29:13.000000000 -0500
> @@ -1966,5 +1966,11 @@
>                 {
>                 int k;
>
> +               if (strlen(buf)+5 <= MAXPATHLEN) {
> +                       for (i=strlen(buf); i>=0; i--)
> +                               buf[i+5] = buf[i];
> +                       strncpy(buf, "exec ", 5);
> +               }
> +
>                 /* pass name of shell script on pipe */
>                 /* will be stdin of shell  */
>
> ...And found it to work as expected in our test environment (with
> admittedly limited testing).  All this does, (if there is still space in
> the buffer) is shifts everything over 5 characters and inserts "exec " at
> the beginning of the command line. The shell invokes the process, which of
> course, now exec's the script. The script inherits the pid of the shell as
> well as its stdin/stdout/stderr so pbs_demux appears to function
correctly.
>
> Every shell I've investigated (sh, csh, ksh, bash, zsh) all appear to
> honor the "exec" command in the same manner so this appears to be a viable
> solution to this problem (premature shell termination) without requiring
> users (or admins) to add "trap" statements to dotfiles to protect that one
> process.  For the record, this doesn't get anyone off the hook about
> installing trap's in the job scripts (or signal handlers in the processes
> themselves), but this appears to remove one of larger barriers in
> leveraging qsig(1) and extended kill_delay settings.
>
> I'lll concede there could be a flaw in my logic, and as I stated above,
> this has only had limited testing thus far, but I would love to hear what
I
> may have missed and why this couldn't be a viable change in Torque.
>
> This was tested by qsub'ing the following perl script directly (no shell
> job-script around it).  This code simply catches signals, prints the time
> that they were received, and after the first signal is caught... prints
the
> time in 1 second intervals (since you'll never see the final SIGKILL you
> can at least count of the seconds).
>
> #!/usr/bin/perl -l
> use constant CATCH => qw/USR1 USR2 HUP TERM INT QUIT ABRT ILL FPE SEGV
> ALRM PIPE CHLD/;
> my $stop;
> $|=1;
>
> @SIG{(CATCH)} = (sub { $stop||=1; print join ' ', shift, '@', scalar
> localtime }) x CATCH;
>
> sleep unless $stop;
> print (scalar localtime), sleep 1 while 1;
>
> When tested with a qdel, you'll see a TERM signal logged at the time
> invocation, followed by the number of printouts which correspond with your
> kill_delay setting (defaults to 2 seconds).  Finally you see a second
> SIGTERM and then ~5 seconds later the output stops (because the process
> receives a SIGKILL).  For the unfamiliar, when the server asks a mom to do
> a SIGKILL... it is hard coded to SIGTERM first and then ~5 seconds later
to
> try a SIGKILL.
>
> Without my patch above (and without adding trap statements to your
> .bashrc) this script will output two SIGTERM's (typically within the same
> second) with about 5 more seconds of printouts (before the final kill).
> mom_logs will confirm that the initial SIGTERM terminated the shell
> process, and that the mom then automatically initiated a job termination
> (via the second TERM and KILL).
>
> I also won't take any offense if someone wants to implement the patch more
> efficiently, I was just trying to do what I wanted with the minimal amount
> of change to the torque code.
>
> Thanks,
>
> -Alan
>
> --
> alan at madllama.net http://humbleville.blogspot.com
>
>
>
> _______________________________________________
> torquedev mailing list
> torquedev at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torquedev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://www.supercluster.org/pipermail/torquedev/attachments/20120313/22ab13b4/attachment.html

------------------------------

_______________________________________________
torquedev mailing list
torquedev at supercluster.org
http://www.supercluster.org/mailman/listinfo/torquedev


End of torquedev Digest, Vol 76, Issue 5
****************************************



-- 
alan at madllama.net http://humbleville.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/torquedev/attachments/20120313/67b1d77f/attachment-0001.html 

From dbeer at adaptivecomputing.com  Tue Mar 13 13:42:36 2012
From: dbeer at adaptivecomputing.com (David Beer)
Date: Tue, 13 Mar 2012 13:42:36 -0600
Subject: [torquedev] TORQUE 4.0 Officially Announced
Message-ID: <CAFUQeZ3V161ZKUxK7dT2p6eqFh3D0BR87gTR8u0HzbBAQKmmrQ@mail.gmail.com>

All,

TORQUE 4.0 is officially here! Please check out Adaptive Computing's
official announcement here:
http://www.adaptivecomputing.com/adaptive-computing-offers-the-next-generation-of-high-performance-computing-with-moab-hpc-suite-7-0/

The tarball can be downloaded from here:
http://www.adaptivecomputing.com/resources/downloads/torque/torque-4.0.0.tar.gz


We have several sites currently using 4.0 and feedback has been positive.
These warnings are posted on the download site, but I am copying them here:

1. Make sure that you have openssl-devel (RedHat based) / libssl-dev
(Debian based) installed (the name may differ for different operating
systems) in order to be able to build TORQUE 4.0.
2. Make sure that you run the daemon trqauthd on machines that will be
running client commands. NOTE: there is an init.d script for it in
contrib/init.d/ but it needs customization (this includes Moab). One
problem is that it has a misspelling for PBS_DAEMON - it should be
/usr/local/sbin/trqauthd by default, not /usr/local/bin/trqauthd.
3. Moab needs to be started or restarted after installing TORQUE 4.0 (if
you are using Moab)

Please make sure to take all normal precautions for upgrading. Another
advisory (not on the website) is that TORQUE now uses hwloc to manage
cpusets, meaning you will need to install hwloc on your system if it isn't
already there and you wish to use it. It needs to be version 1.1 or higher.

The major features of the release are briefly described on the release, but
the CHANGELOG for 4.0 is copied at the end of this email.

This release has undergone more testing than any previous release of
TORQUE; to be fair, it also has more changes than any previous version of
TORQUE. Overall, we saw very good results in our beta program and most of
the sites using it have had good experiences. We are proud of the quality
of this release and hope that you'll try it out and let us know how it
works for you.

-- 
David Beer | Software Engineer
Adaptive Computing


4.0.0
  e - make a threadpool for TORQUE server. The number of threads is
      customizable using min_threads and max_threads, and idle time before
      exiting can be set using thread_idle_seconds.
  e - make pbs_server multi-threaded in order to increase responsiveness
and scalability.
  e - remove the forking from pbs_server running a job, the thread handling
the request just
      waits until the job is run.
  e - change qdel to simply send qdel all - previously this was executed by
a qstat and a qdel
      of every individual job
  e - no longer fork to send mail, just use a thread
  e - use hwloc as the backbone for cpuset support in TORQUE (contributed
by Dr. Bernd Kallies)
  e - add the boolean variable $use_smt to mom config. If set to false,
this skips logical
      cores and uses only physical cores for the job. It is true by default.
      (contributed by Dr. Bernd Kallies)
  n - with the multi-threading the pbs_server -t create and -t cold
commands could no longer
      ask for user input from the command line. The call to ask if the user
wants to continue
      was moved higher in the initialization process and some of the
wording changed to
      reflect what is now happening.
  e - if cpusets are configured but aren't found and cannot be mounted,
pbs_mom will now fail to
      start instead of failing silently.
  e - Change node_spec from an N^2 (but average 5N) algorithm to an N
algorithm with respect
      to nodes. We only loop over each node once at a maximum.
  e - Abandon pbs_iff in favor of trqauthd. trqauthd is a daemon to be
started once that can
      perform pbs_iff's functionality, increasing speed and enabling future
security
      enhancements
  e - add mom_hierarchy functionality for reporting. The file is located in
      <TORQUE_HOME>/server_priv/mom_hierarchy, and can be written to tell
moms to send
      updates to other moms who will pass them on to pbs_server. See docs
for details
  e - add a unit testing framework (check). It is compiled with
--with-check and tests
      are executed using make check. The framework is complete but not many
tests have
      been written as of yet.
  e - Mom rejection messages are now passed back to qrun when possible
  e - Added the option -c for startup. By default, the server attempts to
send the mom
      hierarchy file to all moms on startup, and all moms update the server
and request
      the hierarchy file. If both are trying to do this at once, it can
cause a lot of
      traffic. -c tells pbs_server to wait 10 minutes to attempt to contact
moms that
      haven't contacted it, reducing this traffic.
  e - Added mom parameter -w to reduce start times. This parameter wait to
send it's
      first update until the server sends it the mom hierarchy file, or
until 10
      minutes have passed. This should reduce large cluster startup times.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/torquedev/attachments/20120313/98915e9a/attachment.html 

From mej at lbl.gov  Tue Mar 13 17:18:00 2012
From: mej at lbl.gov (Michael Jennings)
Date: Tue, 13 Mar 2012 16:18:00 -0700
Subject: [torquedev] "Fixing" qsig -s USR1 and kill_delay on torque 2.5.x
In-Reply-To: <CAFUQeZ1bpV-uzgm-qgQvAZnDZVY5J3bPvHSnkPOh-La4_hP+3g@mail.gmail.com>
References: <CAATyho7HnaSds-g07++1=MsrMfGwdcv4G11schzcc_ywzOzHhA@mail.gmail.com>
	<CAATyho5o0uFpNV=y042d+3u5xKOb2Yuj1qXV3HKR1NA5Mm+xAw@mail.gmail.com>
	<CAFUQeZ1bpV-uzgm-qgQvAZnDZVY5J3bPvHSnkPOh-La4_hP+3g@mail.gmail.com>
Message-ID: <20120313231758.GF9750@lbl.gov>

On Tuesday, 13 March 2012, at 10:04:41 (-0600),
David Beer wrote:

> Having done the work it takes to configure these signals to work in the
> system's current state, I'm all for addressing this issue. I'm wondering if
> there are any community concerns about this change? Do you see any possible
> regressions? What are the risks? Should we make this change something that
> only happens if you turn it on in the mom config file? In some ways I like
> this option because it is easy to turn off if there are regressions, but on
> the other hand the kill_delay functionality is so cumbersome to set up its
> essentially broken now. I'm interested to hear the community's input on
> this patch.

I definitely agree that exec'ing the script is the correct way to
spawn it.  I think the patch is reasonable.

Would "exec " also need to be added to the shell command line in
TMomFinalizeChild() in the SHELL_USE_ARGV == 1 case?

Michael

-- 
Michael Jennings <mej at lbl.gov>
Senior HPC Systems Engineer
High-Performance Computing Services
Lawrence Berkeley National Laboratory
Bldg 50B-3209E        W: 510-495-2687
MS 050B-3209          F: 510-486-8615

From cwest at vpac.org  Sun Mar 18 23:40:14 2012
From: cwest at vpac.org (Craig West)
Date: Mon, 19 Mar 2012 16:40:14 +1100
Subject: [torquedev] [torqueusers] TORQUE 4.0 Officially Announced
In-Reply-To: <560DBE57F33C4C4C9FBF11C662951AF805ABC89E@ORSMSX106.amr.corp.intel.com>
References: <CAFUQeZ3V161ZKUxK7dT2p6eqFh3D0BR87gTR8u0HzbBAQKmmrQ@mail.gmail.com>
	<560DBE57F33C4C4C9FBF11C662951AF805ABC89E@ORSMSX106.amr.corp.intel.com>
Message-ID: <4F66C6BE.3010605@vpac.org>


Hi Steven,

I have just begun testing Torque 4.0, as hwloc has been a long awaited 
feature for me.

> It is unclear from this announcement text where hwloc has to be installed.
> Is it just on the server or on the nodes only?

It needs to be available on the BUILD server and the nodes. I tried to 
run pbs_mom on a node without the hwloc I had installed and it failed.

Note: I am running hwloc 1.4 from a directory in /usr/local
This was not automatically found by the TORQUE configure script, but you 
can specify the location using HWLOC_CFLAGS & HWLOC_LIBS.
It embeds the locations that you specify in the pbs_mom (and other 
files) but it seems you can set the LD_LIBRARY_PATH variable if it is 
not in the same location on the BUILD server as the compute nodes.
For simplicity installing them in the same location makes sense.

> More documentation about this would be greatly appreciated.

I agree, clearer and more detailed documentation would be useful.

Cheers,
Craig.

From Gareth.Williams at csiro.au  Mon Mar 19 00:37:39 2012
From: Gareth.Williams at csiro.au (Gareth.Williams at csiro.au)
Date: Mon, 19 Mar 2012 17:37:39 +1100
Subject: [torquedev] TORQUE 2.5.11 available
In-Reply-To: <CADvLK3cikpvoMtZw00rAeSVwZiGWE=ybt_4dHvm9k-11KxdtNw@mail.gmail.com>
References: <CADvLK3cikpvoMtZw00rAeSVwZiGWE=ybt_4dHvm9k-11KxdtNw@mail.gmail.com>
Message-ID: <007DECE986B47F4EABF823C1FBB19C620102D7AE2EBE@exvic-mbx04.nexus.csiro.au>

> TORQUE 2.5.11 is now available. 

Hi Ken,

What would you advise production users of 3.0.x, particularly those with a numa system (SGI UV or other)?

Will the fixes for this release be available in a 3 release or should we look to 4?

I think we've just hit the cpuset/restart bug.

Gareth

From A.Kaliazin at damtp.cam.ac.uk  Mon Mar 19 03:52:51 2012
From: A.Kaliazin at damtp.cam.ac.uk (Andrey Kaliazin)
Date: Mon, 19 Mar 2012 09:52:51 +0000
Subject: [torquedev] TORQUE 2.5.11 available
In-Reply-To: <007DECE986B47F4EABF823C1FBB19C620102D7AE2EBE@exvic-mbx04.nexus.csiro.au>
References: <CADvLK3cikpvoMtZw00rAeSVwZiGWE=ybt_4dHvm9k-11KxdtNw@mail.gmail.com>
	<007DECE986B47F4EABF823C1FBB19C620102D7AE2EBE@exvic-mbx04.nexus.csiro.au>
Message-ID: <4F6701F3.6070001@damtp.cam.ac.uk>

Hi Gareth

Which bug you are talking about and in which particular version of Torque?
I've been using Torque 3.0.2 without a hitch for more than a few months on a UV1000.
Did I miss something?

cheers,

Andrey
COSMOS System Manager


Gareth.Williams at csiro.au wrote:
>> TORQUE 2.5.11 is now available. 
> 
> Hi Ken,
> 
> What would you advise production users of 3.0.x, particularly those with a numa system (SGI UV or other)?
> 
> Will the fixes for this release be available in a 3 release or should we look to 4?
> 
> I think we've just hit the cpuset/restart bug.
> 
> Gareth
> _______________________________________________
> torquedev mailing list
> torquedev at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torquedev

From Gareth.Williams at csiro.au  Mon Mar 19 04:23:52 2012
From: Gareth.Williams at csiro.au (Gareth.Williams at csiro.au)
Date: Mon, 19 Mar 2012 21:23:52 +1100
Subject: [torquedev] TORQUE 2.5.11 available
In-Reply-To: <4F6701F3.6070001@damtp.cam.ac.uk>
References: <CADvLK3cikpvoMtZw00rAeSVwZiGWE=ybt_4dHvm9k-11KxdtNw@mail.gmail.com>
	<007DECE986B47F4EABF823C1FBB19C620102D7AE2EBE@exvic-mbx04.nexus.csiro.au>
	<4F6701F3.6070001@damtp.cam.ac.uk>
Message-ID: <007DECE986B47F4EABF823C1FBB19C620102D7AE2EBF@exvic-mbx04.nexus.csiro.au>

> -----Original Message-----
> From: torquedev-bounces at supercluster.org [mailto:torquedev-
> bounces at supercluster.org] On Behalf Of Andrey Kaliazin
> Sent: Monday, 19 March 2012 8:53 PM
> To: Torque Developers mailing list
> Subject: Re: [torquedev] TORQUE 2.5.11 available
> 
> Hi Gareth
> 
> Which bug you are talking about and in which particular version of
> Torque?
> I've been using Torque 3.0.2 without a hitch for more than a few months
> on a UV1000.
> Did I miss something?
> 
> cheers,
> 
> Andrey
> COSMOS System Manager

Hi Andrey,

Sorry to alarm you! This is a non-event until you restart pbs_mom with -p

http://www.supercluster.org/pipermail/torqueusers/2012-March/014236.html
http://www.clusterresources.com/bugzilla/show_bug.cgi?id=174
pbs_mom kills running jobs despite -p flag
fixed in 2.5.11

Gareth

> 
> 
> Gareth.Williams at csiro.au wrote:
> >> TORQUE 2.5.11 is now available.
> >
> > Hi Ken,
> >
> > What would you advise production users of 3.0.x, particularly those
> with a numa system (SGI UV or other)?
> >
> > Will the fixes for this release be available in a 3 release or should
> we look to 4?
> >
> > I think we've just hit the cpuset/restart bug.
> >
> > Gareth
> > _______________________________________________
> > torquedev mailing list
> > torquedev at supercluster.org
> > http://www.supercluster.org/mailman/listinfo/torquedev
> _______________________________________________
> torquedev mailing list
> torquedev at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torquedev

From A.Kaliazin at damtp.cam.ac.uk  Mon Mar 19 04:50:41 2012
From: A.Kaliazin at damtp.cam.ac.uk (Andrey Kaliazin)
Date: Mon, 19 Mar 2012 10:50:41 +0000
Subject: [torquedev] TORQUE 2.5.11 available
In-Reply-To: <007DECE986B47F4EABF823C1FBB19C620102D7AE2EBF@exvic-mbx04.nexus.csiro.au>
References: <CADvLK3cikpvoMtZw00rAeSVwZiGWE=ybt_4dHvm9k-11KxdtNw@mail.gmail.com>	<007DECE986B47F4EABF823C1FBB19C620102D7AE2EBE@exvic-mbx04.nexus.csiro.au>
	<4F6701F3.6070001@damtp.cam.ac.uk>
	<007DECE986B47F4EABF823C1FBB19C620102D7AE2EBF@exvic-mbx04.nexus.csiro.au>
Message-ID: <4F670F81.2040306@damtp.cam.ac.uk>

Thanks, that's fine then.
I have safely restarted MOM in 3.0.2 many times and no job was harmed in the process.

cheers,
Andrey


Gareth.Williams at csiro.au wrote:
>> -----Original Message-----
>> From: torquedev-bounces at supercluster.org [mailto:torquedev-
>> bounces at supercluster.org] On Behalf Of Andrey Kaliazin
>> Sent: Monday, 19 March 2012 8:53 PM
>> To: Torque Developers mailing list
>> Subject: Re: [torquedev] TORQUE 2.5.11 available
>>
>> Hi Gareth
>>
>> Which bug you are talking about and in which particular version of
>> Torque?
>> I've been using Torque 3.0.2 without a hitch for more than a few months
>> on a UV1000.
>> Did I miss something?
>>
>> cheers,
>>
>> Andrey
>> COSMOS System Manager
> 
> Hi Andrey,
> 
> Sorry to alarm you! This is a non-event until you restart pbs_mom with -p
> 
> http://www.supercluster.org/pipermail/torqueusers/2012-March/014236.html
> http://www.clusterresources.com/bugzilla/show_bug.cgi?id=174
> pbs_mom kills running jobs despite -p flag
> fixed in 2.5.11
> 
> Gareth
> 
>>
>> Gareth.Williams at csiro.au wrote:
>>>> TORQUE 2.5.11 is now available.
>>> Hi Ken,
>>>
>>> What would you advise production users of 3.0.x, particularly those
>> with a numa system (SGI UV or other)?
>>> Will the fixes for this release be available in a 3 release or should
>> we look to 4?
>>> I think we've just hit the cpuset/restart bug.
>>>
>>> Gareth
>>> _______________________________________________
>>> torquedev mailing list
>>> torquedev at supercluster.org
>>> http://www.supercluster.org/mailman/listinfo/torquedev
>> _______________________________________________
>> torquedev mailing list
>> torquedev at supercluster.org
>> http://www.supercluster.org/mailman/listinfo/torquedev

From knielson at adaptivecomputing.com  Mon Mar 19 09:29:03 2012
From: knielson at adaptivecomputing.com (Ken Nielson)
Date: Mon, 19 Mar 2012 09:29:03 -0600
Subject: [torquedev] TORQUE 2.5.11 available
In-Reply-To: <007DECE986B47F4EABF823C1FBB19C620102D7AE2EBE@exvic-mbx04.nexus.csiro.au>
References: <CADvLK3cikpvoMtZw00rAeSVwZiGWE=ybt_4dHvm9k-11KxdtNw@mail.gmail.com>
	<007DECE986B47F4EABF823C1FBB19C620102D7AE2EBE@exvic-mbx04.nexus.csiro.au>
Message-ID: <CADvLK3ckwYigDHVaZj9UEPw3ND=CYF96KOHn=qQVRXC+Z7g-4Q@mail.gmail.com>

On Mon, Mar 19, 2012 at 12:37 AM, <Gareth.Williams at csiro.au> wrote:

> > TORQUE 2.5.11 is now available.
>
> Hi Ken,
>
> What would you advise production users of 3.0.x, particularly those with a
> numa system (SGI UV or other)?
>
> Will the fixes for this release be available in a 3 release or should we
> look to 4?
>
> I think we've just hit the cpuset/restart bug.
>
> Gareth
>

A 3.0.5 release candidate will come out later today. It will have the
relevant fixes from 2.5.11.

Regards

Ken
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/torquedev/attachments/20120319/5daf1fde/attachment.html 

From knielson at adaptivecomputing.com  Thu Mar 29 15:47:59 2012
From: knielson at adaptivecomputing.com (Ken Nielson)
Date: Thu, 29 Mar 2012 15:47:59 -0600
Subject: [torquedev] Just a test
Message-ID: <CADvLK3ep+HcyTnwawVHJtqsYb9PSo2tMgzh=Gy2UOvXX2mFPWQ@mail.gmail.com>

The mail server seems to be down. This is testing to see if we are back up

Ken Nielson
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/torquedev/attachments/20120329/186e3ab6/attachment.html 

From l.flis at cyf-kr.edu.pl  Tue Mar 27 16:59:34 2012
From: l.flis at cyf-kr.edu.pl (Lukasz Flis)
Date: Wed, 28 Mar 2012 00:59:34 +0200
Subject: [torquedev] Problem with TM interface when using --enable-numa
Message-ID: <4F724656.2070205@cyf-kr.edu.pl>

Hi

It seems that TM interface in Torque 3.0.4 compiled with --enable-numa 
flag is broken.

Example:

qsub -I -l nodes=4:ppn=1
qsub: waiting for job 307.batch-xsmp to start
qsub: job 307.batch-xsmp ready


[@xsmp4-3-1 ~]$ cat $PBS_NODEFILE
xsmp4-3-1.local
xsmp4-2-4.local
xsmp4-2-3.local
xsmp4-1-2.local

#mpiexec from openmpi compiled with TM support
mpiexec uname -n
xsmp4-3-1.local
xsmp4-3-1.local
xsmp4-3-1.local
xsmp4-3-1.local


The job above had been allocated 4 different nodes.
However mpiexec or pbsdsh runs given command 4 times on the first of 
hosts from $PBS_NODE file

Is this desired behaviour? I haven't tested Torque 4.0 with numa but I 
suspect it could have the same problem.

Cheers
--
LKF

From alan at madllama.net  Wed Mar 28 13:47:57 2012
From: alan at madllama.net (Alan Wild)
Date: Wed, 28 Mar 2012 14:47:57 -0500
Subject: [torquedev] "Fixing" qsig -s USR1 and kill_delay on torque 2.5.x
In-Reply-To: <CAATyho7ph6F6JKWuwUSZfwcMme+o-VAnYAhO42f6m_fYmcUe2w@mail.gmail.com>
References: <CAATyho7ph6F6JKWuwUSZfwcMme+o-VAnYAhO42f6m_fYmcUe2w@mail.gmail.com>
Message-ID: <CAATyho4F5Q496Y_Psj2YQWkCJPRh7ZO0DmfDr=4kf8mGe6VJLA@mail.gmail.com>

We still don't have permission to install torque-4.0.0, even on our test
systems.  However, I thought I would take a look at the source for pbs_mom
to see how it works.   It appears, overall, very similiar to 2.5.11.  So I
attempted to port my patch to 4.0.0

This does compile, but I can't comment on whether or not it will run.  :)


-%<---%<--CUT
HERE---%<---%<---%<---%<---%<---%<---%<---%<---%<---%<---%<---%<---%<---%<---%<---%<--
diff -rN -U4 torque-4.0.0/src/resmom/start_exec.c
torque-4.0.0-new/src/resmom/start_exec.c
--- torque-4.0.0/src/resmom/start_exec.c        2012-02-21
17:43:51.000000000 -0600
+++ torque-4.0.0-new/src/resmom/start_exec.c    2012-03-28
14:34:37.000000000 -0500
@@ -2213,8 +2213,13 @@
   if (TJE->is_interactive == FALSE)
     {
     int k;

+    if (strlen(buf)+5 <= MAXPATHLEN) {
+        memmove(buf+5,buf,strlen(buf)+1);
+        strncpy(buf, "exec ", 5);
+    }
+
     /* pass name of shell script on pipe */
     /* will be stdin of shell  */

     close(TJE->pipe_script[0]);
@@ -3881,9 +3886,9 @@
       {
       arg[aindex] = calloc(1,
                           strlen(path_jobs) +
                           strlen(pjob->ji_qs.ji_fileprefix) +
-                          strlen(JOB_SCRIPT_SUFFIX) + 1);
+                          strlen(JOB_SCRIPT_SUFFIX) + 6);

       if (arg[aindex] == NULL)
         {
         log_err(errno,id,"cannot alloc env");
@@ -3894,9 +3899,10 @@

         return(-1);
         }

-      strcpy(arg[aindex], path_jobs);
+      strcpy(arg[aindex], "exec ");
+      strcat(arg[aindex], path_jobs);
       strcat(arg[aindex], pjob->ji_qs.ji_fileprefix);
       strcat(arg[aindex], JOB_SCRIPT_SUFFIX);

       arg[aindex + 1] = NULL;
-%<---%<--CUT
HERE---%<---%<---%<---%<---%<---%<---%<---%<---%<---%<---%<---%<---%<---%<---%<---%<--


-Alan


On Mon, Mar 26, 2012 at 11:15 PM, Alan Wild <alan at madllama.net> wrote:


> Sorry, real world intruded the last couple of weeks and I haven't had a
> chance to dive back into this.  Yes, for users building Torque with
> SHELL_USE_ARGV == 1, you would need to modify TMomFinalizeChild().
> However, we don't build Torque this way, so I haven't had a chance to
> really test this.  Regardless, I took a stab at making the patch more
> complete.
>
> This is against the released 2.5.11:
>
>
> diff -rN -U2 torque-2.5.11/src/resmom/start_exec.c
> torque-2.5.11-new/src/resmom/start_exec.c
> --- torque-2.5.11/src/resmom/start_exec.c       2012-03-08
> 15:34:57.000000000 -0600
> +++ torque-2.5.11-new/src/resmom/start_exec.c   2012-03-26
> 23:03:56.000000000 -0500
> @@ -1997,4 +1997,9 @@
>      int k;
>
> +    if (strlen(buf)+5 <= MAXPATHLEN) {
> +        memmove(buf+5,buf,strlen(buf)+1);
> +        strncpy(buf, "exec ", 5);
> +    }
> +
>      /* pass name of shell script on pipe */
>      /* will be stdin of shell  */
> @@ -3641,5 +3646,5 @@
>                            strlen(path_jobs) +
>                            strlen(pjob->ji_qs.ji_fileprefix) +
> -                          strlen(JOB_SCRIPT_SUFFIX) + 1);
> +                          strlen(JOB_SCRIPT_SUFFIX) + 6);
>
>        if (arg[aindex] == NULL)
> @@ -3654,5 +3659,6 @@
>          }
>
> -      strcpy(arg[aindex], path_jobs);
> +      strcpy(arg[aindex], "exec ");
> +      strcat(arg[aindex], path_jobs);
>        strcat(arg[aindex], pjob->ji_qs.ji_fileprefix);
>        strcat(arg[aindex], JOB_SCRIPT_SUFFIX);
>
> I would love to know if anyone other than me has played with this patch
> and whether or not it's looking viable.
>
> -Alan
>
>
> On Mon, Mar 19, 2012 at 4:52 AM, <torquedev-request at supercluster.org>
> wrote:
> >
> > I definitely agree that exec'ing the script is the correct way to
> > spawn it.  I think the patch is reasonable.
> >
> > Would "exec " also need to be added to the shell command line in
> > TMomFinalizeChild() in the SHELL_USE_ARGV == 1 case?
> >
> > Michael
>
> --
> alan at madllama.net http://humbleville.blogspot.com
>


-- 
alan at madllama.net http://humbleville.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/torquedev/attachments/20120328/c2d4cc2f/attachment-0001.html