[torqueusers] duplicate job processes create IO wait state on node

Steven Young chemadm at hamilton.edu
Thu Aug 9 09:53:51 MDT 2012


Hello,
	I'm running into an issue with a cluster where it seems like two copies of the same job is started which ends up making the node become unstable in an IO wait state since both copies appear to be trying to use the same files for the job. When I do a ps and trace through the PID's and PPID's it looks like the mom daemon fires up the login for the job (ie. -tcsh) and then it creates the SC file. Then that process spawns another copy of the same process. I haven't been able to figure out how to reproduce this problem but after one of the users runs a bunch of jobs eventually some of the nodes get into this state. It's a 12 core machine and the load average will be up around 14. When you look at the running processes there isn't anything running on the machine. Looking at top you see rpciod just churning away and creating this IO wait on the machine. Every time a node gets into this state I see the same thing... all the jobs scheduled on the machine have 2 processes "/bin/sh /var/spool/torque/mom_priv/jobs/<job_id>.<server>.SC" whereas on a working node you only see one of these and is what I would expect. When I look at the mom logs for that node it appears that there was some issues trying to remove the previous jobs from the node just before the jobs that end up being duplicates. Perhaps this is related? I'm hoping someone might be able to give me some hints as to why this could be happening? Thanks,

-Steve

OS: RedHat Enterprise 6
Torque: 3.0.2
Maui:  3.3.1


UID        PID  PPID  C STIME TTY          TIME CMD

root      2123     1  0 Aug05 ?        00:01:19 /usr/local/sbin/pbs_mom -q -d /var/spool/torque
<user> 10913  2123  0 03:54 ?        00:00:00 -tcsh
<user> 11079 10913  0 03:54 ?        00:00:00 /bin/sh /var/spool/torque/mom_priv/jobs/56290.<pbs_server>.SC
<user>  2612 11079  0 03:59 ?        00:00:00 /bin/sh /var/spool/torque/mom_priv/jobs/56290.<pbs_server>.SC


mom_logs on node (I changed the real server name with pbs_server_name):

08/07/2012 03:03:11;0002;   pbs_mom;Svr;pbs_mom;Torque Mom Version = 3.0.2, loglevel = 0
08/07/2012 03:08:11;0002;   pbs_mom;Svr;pbs_mom;Torque Mom Version = 3.0.2, loglevel = 0
08/07/2012 03:13:11;0002;   pbs_mom;Svr;pbs_mom;Torque Mom Version = 3.0.2, loglevel = 0
08/07/2012 03:18:11;0002;   pbs_mom;Svr;pbs_mom;Torque Mom Version = 3.0.2, loglevel = 0
08/07/2012 03:23:11;0002;   pbs_mom;Svr;pbs_mom;Torque Mom Version = 3.0.2, loglevel = 0
08/07/2012 03:28:11;0002;   pbs_mom;Svr;pbs_mom;Torque Mom Version = 3.0.2, loglevel = 0
08/07/2012 03:33:11;0002;   pbs_mom;Svr;pbs_mom;Torque Mom Version = 3.0.2, loglevel = 0
08/07/2012 03:38:11;0002;   pbs_mom;Svr;pbs_mom;Torque Mom Version = 3.0.2, loglevel = 0
08/07/2012 03:39:53;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: killing pid 1867 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: killing pid 2491 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: killing pid 7426 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: killing pid 1976 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: killing pid 2486 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: killing pid 6034 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: killing pid 2079 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: killing pid 2488 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: killing pid 13274 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: killing pid 2368 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: killing pid 2568 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: killing pid 18730 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: killing pid 3875 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: killing pid 3998 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: killing pid 21115 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: killing pid 3978 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: killing pid 4128 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: killing pid 29379 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: killing pid 4068 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: killing pid 4252 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: killing pid 13662 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: killing pid 4407 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: killing pid 4544 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: killing pid 31496 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: killing pid 5643 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: killing pid 5757 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: killing pid 10527 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: killing pid 5728 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: killing pid 5890 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: killing pid 27051 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: killing pid 5835 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: killing pid 6012 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: killing pid 30157 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: killing pid 6018 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: killing pid 6297 task 1 with sig 15
08/07/2012 03:39:53;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: killing pid 24477 task 1 with sig 15
08/07/2012 03:39:55;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: killing pid 1867 task 1 gracefully with sig 15
08/07/2012 03:39:55;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=1867/state=S) after sig 15
08/07/2012 03:39:55;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=1867/state=S) after sig 15
08/07/2012 03:39:55;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=1867/state=S) after sig 15
08/07/2012 03:39:56;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=1867/state=S) after sig 15
08/07/2012 03:39:56;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=1867/state=S) after sig 15
08/07/2012 03:39:56;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=1867/state=S) after sig 15
08/07/2012 03:39:56;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=1867/state=S) after sig 15
08/07/2012 03:39:57;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=1867/state=S) after sig 15
08/07/2012 03:39:57;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=1867/state=S) after sig 15
08/07/2012 03:39:57;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=1867/state=S) after sig 15
08/07/2012 03:39:57;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=1867/state=S) after sig 15
08/07/2012 03:39:58;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=1867/state=S) after sig 15
08/07/2012 03:39:58;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=1867/state=S) after sig 15
08/07/2012 03:39:58;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=1867/state=S) after sig 15
08/07/2012 03:39:58;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=1867/state=S) after sig 15
08/07/2012 03:39:59;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=1867/state=S) after sig 15
08/07/2012 03:39:59;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=1867/state=S) after sig 15
08/07/2012 03:39:59;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=1867/state=S) after sig 15
08/07/2012 03:39:59;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=1867/state=S) after sig 15
08/07/2012 03:40:00;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=1867/state=S) after sig 15
08/07/2012 03:40:00;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: killing pid 1867 task 1 with sig 9
08/07/2012 03:40:00;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: killing pid 2491 task 1 gracefully with sig 15
08/07/2012 03:40:00;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=2491/state=R) after sig 15
08/07/2012 03:40:00;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=2491/state=S) after sig 15
08/07/2012 03:40:00;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=2491/state=S) after sig 15
08/07/2012 03:40:01;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=2491/state=S) after sig 15
08/07/2012 03:40:01;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=2491/state=S) after sig 15
08/07/2012 03:40:01;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=2491/state=S) after sig 15
08/07/2012 03:40:01;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=2491/state=S) after sig 15
08/07/2012 03:40:02;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=2491/state=S) after sig 15
08/07/2012 03:40:02;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=2491/state=S) after sig 15
08/07/2012 03:40:02;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=2491/state=S) after sig 15
08/07/2012 03:40:02;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=2491/state=S) after sig 15
08/07/2012 03:40:03;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=2491/state=S) after sig 15
08/07/2012 03:40:03;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=2491/state=S) after sig 15
08/07/2012 03:40:03;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=2491/state=S) after sig 15
08/07/2012 03:40:03;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=2491/state=S) after sig 15
08/07/2012 03:40:04;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=2491/state=S) after sig 15
08/07/2012 03:40:04;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=2491/state=S) after sig 15
08/07/2012 03:40:04;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=2491/state=S) after sig 15
08/07/2012 03:40:04;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=2491/state=S) after sig 15
08/07/2012 03:40:05;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=2491/state=S) after sig 15
08/07/2012 03:40:05;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: killing pid 2491 task 1 with sig 9
08/07/2012 03:40:05;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: killing pid 5020 task 1 gracefully with sig 15
08/07/2012 03:40:05;0008;   pbs_mom;Job;52769.pbs_server_name;kill_task: process (pid=5020/state=R) after sig 15
08/07/2012 03:40:05;0080;   pbs_mom;Job;52769.pbs_server_name;scan_for_terminated: job 52769.pbs_server_name task 1 terminated, sid=1867
08/07/2012 03:40:05;0008;   pbs_mom;Job;52769.pbs_server_name;job was terminated
08/07/2012 03:40:05;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: killing pid 1976 task 1 gracefully with sig 15
08/07/2012 03:40:05;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=1976/state=S) after sig 15
08/07/2012 03:40:05;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=1976/state=S) after sig 15
08/07/2012 03:40:06;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=1976/state=S) after sig 15
08/07/2012 03:40:06;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=1976/state=S) after sig 15
08/07/2012 03:40:06;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=1976/state=S) after sig 15
08/07/2012 03:40:06;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=1976/state=S) after sig 15
08/07/2012 03:40:07;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=1976/state=S) after sig 15
08/07/2012 03:40:07;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=1976/state=S) after sig 15
08/07/2012 03:40:07;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=1976/state=S) after sig 15
08/07/2012 03:40:07;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=1976/state=S) after sig 15
08/07/2012 03:40:08;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=1976/state=S) after sig 15
08/07/2012 03:40:08;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=1976/state=S) after sig 15
08/07/2012 03:40:08;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=1976/state=S) after sig 15
08/07/2012 03:40:08;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=1976/state=S) after sig 15
08/07/2012 03:40:09;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=1976/state=S) after sig 15
08/07/2012 03:40:09;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=1976/state=S) after sig 15
08/07/2012 03:40:09;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=1976/state=S) after sig 15
08/07/2012 03:40:09;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=1976/state=S) after sig 15
08/07/2012 03:40:10;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=1976/state=S) after sig 15
08/07/2012 03:40:10;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=1976/state=S) after sig 15
08/07/2012 03:40:10;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: killing pid 1976 task 1 with sig 9
08/07/2012 03:40:10;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: killing pid 2486 task 1 gracefully with sig 15
08/07/2012 03:40:10;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=2486/state=R) after sig 15
08/07/2012 03:40:10;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=2486/state=S) after sig 15
08/07/2012 03:40:11;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=2486/state=S) after sig 15
08/07/2012 03:40:11;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=2486/state=S) after sig 15
08/07/2012 03:40:11;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=2486/state=S) after sig 15
08/07/2012 03:40:11;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=2486/state=S) after sig 15
08/07/2012 03:40:12;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=2486/state=S) after sig 15
08/07/2012 03:40:12;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=2486/state=S) after sig 15
08/07/2012 03:40:12;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=2486/state=S) after sig 15
08/07/2012 03:40:12;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=2486/state=S) after sig 15
08/07/2012 03:40:13;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=2486/state=S) after sig 15
08/07/2012 03:40:13;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=2486/state=S) after sig 15
08/07/2012 03:40:13;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=2486/state=S) after sig 15
08/07/2012 03:40:13;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=2486/state=S) after sig 15
08/07/2012 03:40:14;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=2486/state=S) after sig 15
08/07/2012 03:40:14;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=2486/state=S) after sig 15
08/07/2012 03:40:14;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=2486/state=S) after sig 15
08/07/2012 03:40:14;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=2486/state=S) after sig 15
08/07/2012 03:40:15;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=2486/state=S) after sig 15
08/07/2012 03:40:15;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=2486/state=S) after sig 15
08/07/2012 03:40:15;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: killing pid 2486 task 1 with sig 9
08/07/2012 03:40:15;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: killing pid 5001 task 1 gracefully with sig 15
08/07/2012 03:40:15;0008;   pbs_mom;Job;52770.pbs_server_name;kill_task: process (pid=5001/state=R) after sig 15
08/07/2012 03:40:15;0080;   pbs_mom;Job;52770.pbs_server_name;scan_for_terminated: job 52770.pbs_server_name task 1 terminated, sid=1976
08/07/2012 03:40:15;0008;   pbs_mom;Job;52770.pbs_server_name;job was terminated
08/07/2012 03:40:15;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: killing pid 2079 task 1 gracefully with sig 15
08/07/2012 03:40:15;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2079/state=S) after sig 15
08/07/2012 03:40:16;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2079/state=S) after sig 15
08/07/2012 03:40:16;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2079/state=S) after sig 15
08/07/2012 03:40:16;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2079/state=S) after sig 15
08/07/2012 03:40:16;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2079/state=S) after sig 15
08/07/2012 03:40:17;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2079/state=S) after sig 15
08/07/2012 03:40:17;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2079/state=S) after sig 15
08/07/2012 03:40:17;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2079/state=S) after sig 15
08/07/2012 03:40:17;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2079/state=S) after sig 15
08/07/2012 03:40:18;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2079/state=S) after sig 15
08/07/2012 03:40:18;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2079/state=S) after sig 15
08/07/2012 03:40:18;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2079/state=S) after sig 15
08/07/2012 03:40:18;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2079/state=S) after sig 15
08/07/2012 03:40:19;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2079/state=S) after sig 15
08/07/2012 03:40:19;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2079/state=S) after sig 15
08/07/2012 03:40:19;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2079/state=S) after sig 15
08/07/2012 03:40:19;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2079/state=S) after sig 15
08/07/2012 03:40:20;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2079/state=S) after sig 15
08/07/2012 03:40:20;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2079/state=S) after sig 15
08/07/2012 03:40:20;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2079/state=S) after sig 15
08/07/2012 03:40:20;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: killing pid 2079 task 1 with sig 9
08/07/2012 03:40:20;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: killing pid 2488 task 1 gracefully with sig 15
08/07/2012 03:40:20;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2488/state=R) after sig 15
08/07/2012 03:40:21;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2488/state=S) after sig 15
08/07/2012 03:40:21;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2488/state=S) after sig 15
08/07/2012 03:40:21;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2488/state=S) after sig 15
08/07/2012 03:40:21;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2488/state=S) after sig 15
08/07/2012 03:40:22;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2488/state=S) after sig 15
08/07/2012 03:40:22;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2488/state=S) after sig 15
08/07/2012 03:40:22;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2488/state=S) after sig 15
08/07/2012 03:40:22;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2488/state=S) after sig 15
08/07/2012 03:40:23;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2488/state=S) after sig 15
08/07/2012 03:40:23;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2488/state=S) after sig 15
08/07/2012 03:40:23;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2488/state=S) after sig 15
08/07/2012 03:40:23;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2488/state=S) after sig 15
08/07/2012 03:40:24;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2488/state=S) after sig 15
08/07/2012 03:40:24;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2488/state=S) after sig 15
08/07/2012 03:40:24;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2488/state=S) after sig 15
08/07/2012 03:40:24;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2488/state=S) after sig 15
08/07/2012 03:40:25;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2488/state=S) after sig 15
08/07/2012 03:40:25;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2488/state=S) after sig 15
08/07/2012 03:40:25;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=2488/state=S) after sig 15
08/07/2012 03:40:25;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: killing pid 2488 task 1 with sig 9
08/07/2012 03:40:25;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: killing pid 5002 task 1 gracefully with sig 15
08/07/2012 03:40:25;0008;   pbs_mom;Job;52771.pbs_server_name;kill_task: process (pid=5002/state=R) after sig 15
08/07/2012 03:40:26;0080;   pbs_mom;Svr;preobit_reply;top of preobit_reply
08/07/2012 03:40:26;0080;   pbs_mom;Svr;preobit_reply;DIS_reply_read/decode_DIS_replySvr worked, top of while loop
08/07/2012 03:40:26;0080;   pbs_mom;Svr;preobit_reply;in while loop, no error from job stat
08/07/2012 03:40:26;0080;   pbs_mom;Job;52771.pbs_server_name;scan_for_terminated: job 52771.pbs_server_name task 1 terminated, sid=2079
08/07/2012 03:40:26;0080;   pbs_mom;Job;52769.pbs_server_name;obit sent to server
08/07/2012 03:40:26;0008;   pbs_mom;Job;52771.pbs_server_name;job was terminated
08/07/2012 03:40:26;0080;   pbs_mom;Svr;preobit_reply;top of preobit_reply
08/07/2012 03:40:26;0080;   pbs_mom;Svr;preobit_reply;DIS_reply_read/decode_DIS_replySvr worked, top of while loop
08/07/2012 03:40:26;0080;   pbs_mom;Svr;preobit_reply;in while loop, no error from job stat
08/07/2012 03:40:26;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: killing pid 2368 task 1 gracefully with sig 15
08/07/2012 03:40:26;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2368/state=S) after sig 15
08/07/2012 03:40:26;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2368/state=S) after sig 15
08/07/2012 03:40:26;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2368/state=S) after sig 15
08/07/2012 03:40:26;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2368/state=S) after sig 15
08/07/2012 03:40:27;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2368/state=S) after sig 15
08/07/2012 03:40:27;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2368/state=S) after sig 15
08/07/2012 03:40:27;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2368/state=S) after sig 15
08/07/2012 03:40:27;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2368/state=S) after sig 15
08/07/2012 03:40:28;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2368/state=S) after sig 15
08/07/2012 03:40:28;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2368/state=S) after sig 15
08/07/2012 03:40:28;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2368/state=S) after sig 15
08/07/2012 03:40:28;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2368/state=S) after sig 15
08/07/2012 03:40:29;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2368/state=S) after sig 15
08/07/2012 03:40:29;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2368/state=S) after sig 15
08/07/2012 03:40:29;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2368/state=S) after sig 15
08/07/2012 03:40:29;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2368/state=S) after sig 15
08/07/2012 03:40:30;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2368/state=S) after sig 15
08/07/2012 03:40:30;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2368/state=S) after sig 15
08/07/2012 03:40:30;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2368/state=S) after sig 15
08/07/2012 03:40:30;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2368/state=S) after sig 15
08/07/2012 03:40:31;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: killing pid 2368 task 1 with sig 9
08/07/2012 03:40:31;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: killing pid 2568 task 1 gracefully with sig 15
08/07/2012 03:40:31;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2568/state=R) after sig 15
08/07/2012 03:40:31;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2568/state=S) after sig 15
08/07/2012 03:40:31;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2568/state=S) after sig 15
08/07/2012 03:40:31;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2568/state=S) after sig 15
08/07/2012 03:40:31;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2568/state=S) after sig 15
08/07/2012 03:40:32;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2568/state=S) after sig 15
08/07/2012 03:40:32;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2568/state=S) after sig 15
08/07/2012 03:40:32;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2568/state=S) after sig 15
08/07/2012 03:40:32;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2568/state=S) after sig 15
08/07/2012 03:40:33;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2568/state=S) after sig 15
08/07/2012 03:40:33;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2568/state=S) after sig 15
08/07/2012 03:40:33;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2568/state=S) after sig 15
08/07/2012 03:40:33;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2568/state=S) after sig 15
08/07/2012 03:40:34;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2568/state=S) after sig 15
08/07/2012 03:40:34;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2568/state=S) after sig 15
08/07/2012 03:40:34;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2568/state=S) after sig 15
08/07/2012 03:40:34;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2568/state=S) after sig 15
08/07/2012 03:40:35;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2568/state=S) after sig 15
08/07/2012 03:40:35;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2568/state=S) after sig 15
08/07/2012 03:40:35;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=2568/state=S) after sig 15
08/07/2012 03:40:35;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: killing pid 2568 task 1 with sig 9
08/07/2012 03:40:35;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: killing pid 5000 task 1 gracefully with sig 15
08/07/2012 03:40:35;0008;   pbs_mom;Job;52772.pbs_server_name;kill_task: process (pid=5000/state=R) after sig 15
08/07/2012 03:40:36;0080;   pbs_mom;Job;52772.pbs_server_name;scan_for_terminated: job 52772.pbs_server_name task 1 terminated, sid=2368
08/07/2012 03:40:36;0080;   pbs_mom;Job;52770.pbs_server_name;obit sent to server
08/07/2012 03:40:36;0008;   pbs_mom;Job;52772.pbs_server_name;job was terminated
08/07/2012 03:40:36;0080;   pbs_mom;Svr;preobit_reply;top of preobit_reply
08/07/2012 03:40:36;0080;   pbs_mom;Svr;preobit_reply;DIS_reply_read/decode_DIS_replySvr worked, top of while loop
08/07/2012 03:40:36;0080;   pbs_mom;Svr;preobit_reply;in while loop, no error from job stat
08/07/2012 03:40:36;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: killing pid 3875 task 1 gracefully with sig 15
08/07/2012 03:40:36;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3875/state=S) after sig 15
08/07/2012 03:40:36;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3875/state=S) after sig 15
08/07/2012 03:40:36;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3875/state=S) after sig 15
08/07/2012 03:40:36;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3875/state=S) after sig 15
08/07/2012 03:40:37;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3875/state=S) after sig 15
08/07/2012 03:40:37;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3875/state=S) after sig 15
08/07/2012 03:40:37;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3875/state=S) after sig 15
08/07/2012 03:40:37;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3875/state=S) after sig 15
08/07/2012 03:40:38;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3875/state=S) after sig 15
08/07/2012 03:40:38;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3875/state=S) after sig 15
08/07/2012 03:40:38;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3875/state=S) after sig 15
08/07/2012 03:40:38;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3875/state=S) after sig 15
08/07/2012 03:40:39;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3875/state=S) after sig 15
08/07/2012 03:40:39;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3875/state=S) after sig 15
08/07/2012 03:40:39;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3875/state=S) after sig 15
08/07/2012 03:40:39;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3875/state=S) after sig 15
08/07/2012 03:40:40;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3875/state=S) after sig 15
08/07/2012 03:40:40;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3875/state=S) after sig 15
08/07/2012 03:40:40;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3875/state=S) after sig 15
08/07/2012 03:40:40;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3875/state=S) after sig 15
08/07/2012 03:40:41;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: killing pid 3875 task 1 with sig 9
08/07/2012 03:40:41;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: killing pid 3998 task 1 gracefully with sig 15
08/07/2012 03:40:41;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3998/state=R) after sig 15
08/07/2012 03:40:41;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3998/state=S) after sig 15
08/07/2012 03:40:41;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3998/state=S) after sig 15
08/07/2012 03:40:41;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3998/state=S) after sig 15
08/07/2012 03:40:41;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3998/state=S) after sig 15
08/07/2012 03:40:42;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3998/state=S) after sig 15
08/07/2012 03:40:42;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3998/state=S) after sig 15
08/07/2012 03:40:42;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3998/state=S) after sig 15
08/07/2012 03:40:42;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3998/state=S) after sig 15
08/07/2012 03:40:43;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3998/state=S) after sig 15
08/07/2012 03:40:43;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3998/state=S) after sig 15
08/07/2012 03:40:43;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3998/state=S) after sig 15
08/07/2012 03:40:43;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3998/state=S) after sig 15
08/07/2012 03:40:44;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3998/state=S) after sig 15
08/07/2012 03:40:44;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3998/state=S) after sig 15
08/07/2012 03:40:44;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3998/state=S) after sig 15
08/07/2012 03:40:44;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3998/state=S) after sig 15
08/07/2012 03:40:45;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3998/state=S) after sig 15
08/07/2012 03:40:45;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3998/state=S) after sig 15
08/07/2012 03:40:45;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=3998/state=S) after sig 15
08/07/2012 03:40:45;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: killing pid 3998 task 1 with sig 9
08/07/2012 03:40:45;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: killing pid 5006 task 1 gracefully with sig 15
08/07/2012 03:40:45;0008;   pbs_mom;Job;52773.pbs_server_name;kill_task: process (pid=5006/state=R) after sig 15
08/07/2012 03:40:46;0080;   pbs_mom;Job;52773.pbs_server_name;scan_for_terminated: job 52773.pbs_server_name task 1 terminated, sid=3875
08/07/2012 03:40:46;0080;   pbs_mom;Job;52771.pbs_server_name;obit sent to server
08/07/2012 03:40:46;0008;   pbs_mom;Job;52773.pbs_server_name;job was terminated
08/07/2012 03:40:46;0080;   pbs_mom;Svr;preobit_reply;top of preobit_reply
08/07/2012 03:40:46;0080;   pbs_mom;Svr;preobit_reply;DIS_reply_read/decode_DIS_replySvr worked, top of while loop
08/07/2012 03:40:46;0080;   pbs_mom;Svr;preobit_reply;in while loop, no error from job stat
08/07/2012 03:40:46;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: killing pid 3978 task 1 gracefully with sig 15
08/07/2012 03:40:46;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=3978/state=S) after sig 15
08/07/2012 03:40:46;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=3978/state=S) after sig 15
08/07/2012 03:40:46;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=3978/state=S) after sig 15
08/07/2012 03:40:46;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=3978/state=S) after sig 15
08/07/2012 03:40:47;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=3978/state=S) after sig 15
08/07/2012 03:40:47;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=3978/state=S) after sig 15
08/07/2012 03:40:47;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=3978/state=S) after sig 15
08/07/2012 03:40:47;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=3978/state=S) after sig 15
08/07/2012 03:40:48;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=3978/state=S) after sig 15
08/07/2012 03:40:48;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=3978/state=S) after sig 15
08/07/2012 03:40:48;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=3978/state=S) after sig 15
08/07/2012 03:40:49;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=3978/state=S) after sig 15
08/07/2012 03:40:49;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=3978/state=S) after sig 15
08/07/2012 03:40:49;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=3978/state=S) after sig 15
08/07/2012 03:40:49;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=3978/state=S) after sig 15
08/07/2012 03:40:50;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=3978/state=S) after sig 15
08/07/2012 03:40:50;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=3978/state=S) after sig 15
08/07/2012 03:40:50;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=3978/state=S) after sig 15
08/07/2012 03:40:50;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=3978/state=S) after sig 15
08/07/2012 03:40:51;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=3978/state=S) after sig 15
08/07/2012 03:40:51;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: killing pid 3978 task 1 with sig 9
08/07/2012 03:40:51;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: killing pid 4128 task 1 gracefully with sig 15
08/07/2012 03:40:51;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=4128/state=R) after sig 15
08/07/2012 03:40:51;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=4128/state=S) after sig 15
08/07/2012 03:40:51;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=4128/state=S) after sig 15
08/07/2012 03:40:51;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=4128/state=S) after sig 15
08/07/2012 03:40:52;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=4128/state=S) after sig 15
08/07/2012 03:40:52;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=4128/state=S) after sig 15
08/07/2012 03:40:52;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=4128/state=S) after sig 15
08/07/2012 03:40:52;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=4128/state=S) after sig 15
08/07/2012 03:40:53;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=4128/state=S) after sig 15
08/07/2012 03:40:53;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=4128/state=S) after sig 15
08/07/2012 03:40:53;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=4128/state=S) after sig 15
08/07/2012 03:40:53;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=4128/state=S) after sig 15
08/07/2012 03:40:54;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=4128/state=S) after sig 15
08/07/2012 03:40:54;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=4128/state=S) after sig 15
08/07/2012 03:40:54;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=4128/state=S) after sig 15
08/07/2012 03:40:54;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=4128/state=S) after sig 15
08/07/2012 03:40:55;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=4128/state=S) after sig 15
08/07/2012 03:40:55;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=4128/state=S) after sig 15
08/07/2012 03:40:55;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=4128/state=S) after sig 15
08/07/2012 03:40:55;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=4128/state=S) after sig 15
08/07/2012 03:40:56;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: killing pid 4128 task 1 with sig 9
08/07/2012 03:40:56;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: killing pid 4997 task 1 gracefully with sig 15
08/07/2012 03:40:56;0008;   pbs_mom;Job;52774.pbs_server_name;kill_task: process (pid=4997/state=R) after sig 15
08/07/2012 03:40:56;0080;   pbs_mom;Job;52774.pbs_server_name;scan_for_terminated: job 52774.pbs_server_name task 1 terminated, sid=3978
08/07/2012 03:40:56;0080;   pbs_mom;Job;52772.pbs_server_name;obit sent to server
08/07/2012 03:40:56;0008;   pbs_mom;Job;52774.pbs_server_name;job was terminated
08/07/2012 03:40:56;0080;   pbs_mom;Svr;preobit_reply;top of preobit_reply
08/07/2012 03:40:56;0080;   pbs_mom;Svr;preobit_reply;DIS_reply_read/decode_DIS_replySvr worked, top of while loop
08/07/2012 03:40:56;0080;   pbs_mom;Svr;preobit_reply;in while loop, no error from job stat
08/07/2012 03:40:56;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: killing pid 4068 task 1 gracefully with sig 15
08/07/2012 03:40:56;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4068/state=S) after sig 15
08/07/2012 03:40:56;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4068/state=S) after sig 15
08/07/2012 03:40:56;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4068/state=S) after sig 15
08/07/2012 03:40:57;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4068/state=S) after sig 15
08/07/2012 03:40:57;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4068/state=S) after sig 15
08/07/2012 03:40:57;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4068/state=S) after sig 15
08/07/2012 03:40:57;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4068/state=S) after sig 15
08/07/2012 03:40:58;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4068/state=S) after sig 15
08/07/2012 03:40:58;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4068/state=S) after sig 15
08/07/2012 03:40:58;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4068/state=S) after sig 15
08/07/2012 03:40:58;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4068/state=S) after sig 15
08/07/2012 03:40:59;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4068/state=S) after sig 15
08/07/2012 03:40:59;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4068/state=S) after sig 15
08/07/2012 03:40:59;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4068/state=S) after sig 15
08/07/2012 03:40:59;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4068/state=S) after sig 15
08/07/2012 03:41:00;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4068/state=S) after sig 15
08/07/2012 03:41:00;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4068/state=S) after sig 15
08/07/2012 03:41:00;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4068/state=S) after sig 15
08/07/2012 03:41:00;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4068/state=S) after sig 15
08/07/2012 03:41:01;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4068/state=S) after sig 15
08/07/2012 03:41:01;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: killing pid 4068 task 1 with sig 9
08/07/2012 03:41:01;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: killing pid 4252 task 1 gracefully with sig 15
08/07/2012 03:41:01;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4252/state=R) after sig 15
08/07/2012 03:41:01;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4252/state=S) after sig 15
08/07/2012 03:41:01;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4252/state=S) after sig 15
08/07/2012 03:41:01;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4252/state=S) after sig 15
08/07/2012 03:41:02;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4252/state=S) after sig 15
08/07/2012 03:41:02;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4252/state=S) after sig 15
08/07/2012 03:41:02;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4252/state=S) after sig 15
08/07/2012 03:41:02;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4252/state=S) after sig 15
08/07/2012 03:41:03;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4252/state=S) after sig 15
08/07/2012 03:41:03;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4252/state=S) after sig 15
08/07/2012 03:41:03;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4252/state=S) after sig 15
08/07/2012 03:41:03;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4252/state=S) after sig 15
08/07/2012 03:41:04;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4252/state=S) after sig 15
08/07/2012 03:41:04;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4252/state=S) after sig 15
08/07/2012 03:41:04;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4252/state=S) after sig 15
08/07/2012 03:41:04;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4252/state=S) after sig 15
08/07/2012 03:41:05;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4252/state=S) after sig 15
08/07/2012 03:41:05;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4252/state=S) after sig 15
08/07/2012 03:41:05;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4252/state=S) after sig 15
08/07/2012 03:41:05;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=4252/state=S) after sig 15
08/07/2012 03:41:06;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: killing pid 4252 task 1 with sig 9
08/07/2012 03:41:06;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: killing pid 5032 task 1 gracefully with sig 15
08/07/2012 03:41:06;0008;   pbs_mom;Job;52775.pbs_server_name;kill_task: process (pid=5032/state=R) after sig 15
08/07/2012 03:41:06;0080;   pbs_mom;Job;52775.pbs_server_name;scan_for_terminated: job 52775.pbs_server_name task 1 terminated, sid=4068
08/07/2012 03:41:06;0080;   pbs_mom;Job;52773.pbs_server_name;obit sent to server
08/07/2012 03:41:06;0008;   pbs_mom;Job;52775.pbs_server_name;job was terminated
08/07/2012 03:41:06;0080;   pbs_mom;Svr;preobit_reply;top of preobit_reply
08/07/2012 03:41:06;0080;   pbs_mom;Svr;preobit_reply;DIS_reply_read/decode_DIS_replySvr worked, top of while loop
08/07/2012 03:41:06;0080;   pbs_mom;Svr;preobit_reply;in while loop, no error from job stat
08/07/2012 03:41:06;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: killing pid 4407 task 1 gracefully with sig 15
08/07/2012 03:41:06;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4407/state=S) after sig 15
08/07/2012 03:41:06;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4407/state=S) after sig 15
08/07/2012 03:41:06;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4407/state=S) after sig 15
08/07/2012 03:41:07;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4407/state=S) after sig 15
08/07/2012 03:41:07;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4407/state=S) after sig 15
08/07/2012 03:41:07;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4407/state=S) after sig 15
08/07/2012 03:41:07;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4407/state=S) after sig 15
08/07/2012 03:41:08;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4407/state=S) after sig 15
08/07/2012 03:41:08;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4407/state=S) after sig 15
08/07/2012 03:41:08;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4407/state=S) after sig 15
08/07/2012 03:41:08;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4407/state=S) after sig 15
08/07/2012 03:41:09;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4407/state=S) after sig 15
08/07/2012 03:41:09;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4407/state=S) after sig 15
08/07/2012 03:41:09;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4407/state=S) after sig 15
08/07/2012 03:41:09;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4407/state=S) after sig 15
08/07/2012 03:41:10;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4407/state=S) after sig 15
08/07/2012 03:41:10;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4407/state=S) after sig 15
08/07/2012 03:41:10;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4407/state=S) after sig 15
08/07/2012 03:41:10;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4407/state=S) after sig 15
08/07/2012 03:41:11;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4407/state=S) after sig 15
08/07/2012 03:41:11;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: killing pid 4407 task 1 with sig 9
08/07/2012 03:41:11;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: killing pid 4544 task 1 gracefully with sig 15
08/07/2012 03:41:11;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4544/state=R) after sig 15
08/07/2012 03:41:11;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4544/state=S) after sig 15
08/07/2012 03:41:11;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4544/state=S) after sig 15
08/07/2012 03:41:11;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4544/state=S) after sig 15
08/07/2012 03:41:12;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4544/state=S) after sig 15
08/07/2012 03:41:12;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4544/state=S) after sig 15
08/07/2012 03:41:12;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4544/state=S) after sig 15
08/07/2012 03:41:12;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4544/state=S) after sig 15
08/07/2012 03:41:13;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4544/state=S) after sig 15
08/07/2012 03:41:13;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4544/state=S) after sig 15
08/07/2012 03:41:13;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4544/state=S) after sig 15
08/07/2012 03:41:13;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4544/state=S) after sig 15
08/07/2012 03:41:14;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4544/state=S) after sig 15
08/07/2012 03:41:14;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4544/state=S) after sig 15
08/07/2012 03:41:14;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4544/state=S) after sig 15
08/07/2012 03:41:14;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4544/state=S) after sig 15
08/07/2012 03:41:15;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4544/state=S) after sig 15
08/07/2012 03:41:15;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4544/state=S) after sig 15
08/07/2012 03:41:15;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4544/state=S) after sig 15
08/07/2012 03:41:15;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=4544/state=S) after sig 15
08/07/2012 03:41:16;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: killing pid 4544 task 1 with sig 9
08/07/2012 03:41:16;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: killing pid 5039 task 1 gracefully with sig 15
08/07/2012 03:41:16;0008;   pbs_mom;Job;52776.pbs_server_name;kill_task: process (pid=5039/state=R) after sig 15
08/07/2012 03:41:16;0080;   pbs_mom;Job;52776.pbs_server_name;scan_for_terminated: job 52776.pbs_server_name task 1 terminated, sid=4407
08/07/2012 03:41:16;0080;   pbs_mom;Job;52774.pbs_server_name;obit sent to server
08/07/2012 03:41:16;0008;   pbs_mom;Job;52776.pbs_server_name;job was terminated
08/07/2012 03:41:16;0080;   pbs_mom;Svr;preobit_reply;top of preobit_reply
08/07/2012 03:41:16;0080;   pbs_mom;Svr;preobit_reply;DIS_reply_read/decode_DIS_replySvr worked, top of while loop
08/07/2012 03:41:16;0080;   pbs_mom;Svr;preobit_reply;in while loop, no error from job stat
08/07/2012 03:41:16;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: killing pid 5036 task 1 gracefully with sig 15
08/07/2012 03:41:16;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5036/state=R) after sig 15
08/07/2012 03:41:16;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: killing pid 5643 task 1 gracefully with sig 15
08/07/2012 03:41:16;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5643/state=S) after sig 15
08/07/2012 03:41:16;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5643/state=S) after sig 15
08/07/2012 03:41:17;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5643/state=S) after sig 15
08/07/2012 03:41:17;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5643/state=S) after sig 15
08/07/2012 03:41:17;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5643/state=S) after sig 15
08/07/2012 03:41:17;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5643/state=S) after sig 15
08/07/2012 03:41:18;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5643/state=S) after sig 15
08/07/2012 03:41:18;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5643/state=S) after sig 15
08/07/2012 03:41:18;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5643/state=S) after sig 15
08/07/2012 03:41:18;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5643/state=S) after sig 15
08/07/2012 03:41:19;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5643/state=S) after sig 15
08/07/2012 03:41:19;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5643/state=S) after sig 15
08/07/2012 03:41:19;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5643/state=S) after sig 15
08/07/2012 03:41:19;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5643/state=S) after sig 15
08/07/2012 03:41:20;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5643/state=S) after sig 15
08/07/2012 03:41:20;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5643/state=S) after sig 15
08/07/2012 03:41:20;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5643/state=S) after sig 15
08/07/2012 03:41:20;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5643/state=S) after sig 15
08/07/2012 03:41:21;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5643/state=S) after sig 15
08/07/2012 03:41:21;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5643/state=S) after sig 15
08/07/2012 03:41:21;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: killing pid 5643 task 1 with sig 9
08/07/2012 03:41:21;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: killing pid 5757 task 1 gracefully with sig 15
08/07/2012 03:41:21;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5757/state=R) after sig 15
08/07/2012 03:41:21;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5757/state=S) after sig 15
08/07/2012 03:41:21;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5757/state=S) after sig 15
08/07/2012 03:41:22;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5757/state=S) after sig 15
08/07/2012 03:41:22;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5757/state=S) after sig 15
08/07/2012 03:41:22;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5757/state=S) after sig 15
08/07/2012 03:41:22;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5757/state=S) after sig 15
08/07/2012 03:41:23;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5757/state=S) after sig 15
08/07/2012 03:41:23;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5757/state=S) after sig 15
08/07/2012 03:41:23;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5757/state=S) after sig 15
08/07/2012 03:41:23;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5757/state=S) after sig 15
08/07/2012 03:41:24;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5757/state=S) after sig 15
08/07/2012 03:41:24;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5757/state=S) after sig 15
08/07/2012 03:41:24;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5757/state=S) after sig 15
08/07/2012 03:41:24;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5757/state=S) after sig 15
08/07/2012 03:41:25;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5757/state=S) after sig 15
08/07/2012 03:41:25;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5757/state=S) after sig 15
08/07/2012 03:41:25;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5757/state=S) after sig 15
08/07/2012 03:41:25;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5757/state=S) after sig 15
08/07/2012 03:41:26;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=5757/state=S) after sig 15
08/07/2012 03:41:26;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: killing pid 5757 task 1 with sig 9
08/07/2012 03:41:26;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: killing pid 6320 task 1 gracefully with sig 15
08/07/2012 03:41:26;0008;   pbs_mom;Job;52777.pbs_server_name;kill_task: process (pid=6320/state=R) after sig 15
08/07/2012 03:41:26;0080;   pbs_mom;Job;52777.pbs_server_name;scan_for_terminated: job 52777.pbs_server_name task 1 terminated, sid=5643
08/07/2012 03:41:26;0080;   pbs_mom;Job;52775.pbs_server_name;obit sent to server
08/07/2012 03:41:26;0008;   pbs_mom;Job;52777.pbs_server_name;job was terminated
08/07/2012 03:41:26;0080;   pbs_mom;Svr;preobit_reply;top of preobit_reply
08/07/2012 03:41:26;0080;   pbs_mom;Svr;preobit_reply;DIS_reply_read/decode_DIS_replySvr worked, top of while loop
08/07/2012 03:41:26;0080;   pbs_mom;Svr;preobit_reply;in while loop, no error from job stat
08/07/2012 03:41:26;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: killing pid 5054 task 1 gracefully with sig 15
08/07/2012 03:41:26;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5054/state=R) after sig 15
08/07/2012 03:41:26;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: killing pid 5728 task 1 gracefully with sig 15
08/07/2012 03:41:26;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5728/state=S) after sig 15
08/07/2012 03:41:27;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5728/state=S) after sig 15
08/07/2012 03:41:27;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5728/state=S) after sig 15
08/07/2012 03:41:27;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5728/state=S) after sig 15
08/07/2012 03:41:27;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5728/state=S) after sig 15
08/07/2012 03:41:28;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5728/state=S) after sig 15
08/07/2012 03:41:28;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5728/state=S) after sig 15
08/07/2012 03:41:28;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5728/state=S) after sig 15
08/07/2012 03:41:28;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5728/state=S) after sig 15
08/07/2012 03:41:29;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5728/state=S) after sig 15
08/07/2012 03:41:29;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5728/state=S) after sig 15
08/07/2012 03:41:29;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5728/state=S) after sig 15
08/07/2012 03:41:29;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5728/state=S) after sig 15
08/07/2012 03:41:30;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5728/state=S) after sig 15
08/07/2012 03:41:30;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5728/state=S) after sig 15
08/07/2012 03:41:30;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5728/state=S) after sig 15
08/07/2012 03:41:30;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5728/state=S) after sig 15
08/07/2012 03:41:31;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5728/state=S) after sig 15
08/07/2012 03:41:31;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5728/state=S) after sig 15
08/07/2012 03:41:31;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5728/state=S) after sig 15
08/07/2012 03:41:31;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: killing pid 5728 task 1 with sig 9
08/07/2012 03:41:31;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: killing pid 5890 task 1 gracefully with sig 15
08/07/2012 03:41:31;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5890/state=S) after sig 15
08/07/2012 03:41:32;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5890/state=S) after sig 15
08/07/2012 03:41:32;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5890/state=S) after sig 15
08/07/2012 03:41:32;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5890/state=S) after sig 15
08/07/2012 03:41:32;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5890/state=S) after sig 15
08/07/2012 03:41:33;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5890/state=S) after sig 15
08/07/2012 03:41:33;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5890/state=S) after sig 15
08/07/2012 03:41:33;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5890/state=S) after sig 15
08/07/2012 03:41:33;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5890/state=S) after sig 15
08/07/2012 03:41:34;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5890/state=S) after sig 15
08/07/2012 03:41:34;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5890/state=S) after sig 15
08/07/2012 03:41:34;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5890/state=S) after sig 15
08/07/2012 03:41:34;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5890/state=S) after sig 15
08/07/2012 03:41:35;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5890/state=S) after sig 15
08/07/2012 03:41:35;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5890/state=S) after sig 15
08/07/2012 03:41:35;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5890/state=S) after sig 15
08/07/2012 03:41:35;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5890/state=S) after sig 15
08/07/2012 03:41:36;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5890/state=S) after sig 15
08/07/2012 03:41:36;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5890/state=S) after sig 15
08/07/2012 03:41:36;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=5890/state=S) after sig 15
08/07/2012 03:41:36;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: killing pid 5890 task 1 with sig 9
08/07/2012 03:41:36;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: killing pid 6196 task 1 gracefully with sig 15
08/07/2012 03:41:36;0008;   pbs_mom;Job;52778.pbs_server_name;kill_task: process (pid=6196/state=R) after sig 15
08/07/2012 03:41:37;0080;   pbs_mom;Job;52778.pbs_server_name;scan_for_terminated: job 52778.pbs_server_name task 1 terminated, sid=5728
08/07/2012 03:41:37;0080;   pbs_mom;Job;52776.pbs_server_name;obit sent to server
08/07/2012 03:41:37;0008;   pbs_mom;Job;52778.pbs_server_name;job was terminated
08/07/2012 03:41:37;0080;   pbs_mom;Svr;preobit_reply;top of preobit_reply
08/07/2012 03:41:37;0080;   pbs_mom;Svr;preobit_reply;DIS_reply_read/decode_DIS_replySvr worked, top of while loop
08/07/2012 03:41:37;0080;   pbs_mom;Svr;preobit_reply;in while loop, no error from job stat
08/07/2012 03:41:37;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: killing pid 5063 task 1 gracefully with sig 15
08/07/2012 03:41:37;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=5063/state=R) after sig 15
08/07/2012 03:41:37;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: killing pid 5835 task 1 gracefully with sig 15
08/07/2012 03:41:37;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=5835/state=S) after sig 15
08/07/2012 03:41:37;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=5835/state=S) after sig 15
08/07/2012 03:41:37;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=5835/state=S) after sig 15
08/07/2012 03:41:38;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=5835/state=S) after sig 15
08/07/2012 03:41:38;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=5835/state=S) after sig 15
08/07/2012 03:41:38;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=5835/state=S) after sig 15
08/07/2012 03:41:38;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=5835/state=S) after sig 15
08/07/2012 03:41:39;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=5835/state=S) after sig 15
08/07/2012 03:41:39;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=5835/state=S) after sig 15
08/07/2012 03:41:39;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=5835/state=S) after sig 15
08/07/2012 03:41:39;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=5835/state=S) after sig 15
08/07/2012 03:41:40;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=5835/state=S) after sig 15
08/07/2012 03:41:40;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=5835/state=S) after sig 15
08/07/2012 03:41:40;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=5835/state=S) after sig 15
08/07/2012 03:41:40;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=5835/state=S) after sig 15
08/07/2012 03:41:41;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=5835/state=S) after sig 15
08/07/2012 03:41:41;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=5835/state=S) after sig 15
08/07/2012 03:41:41;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=5835/state=S) after sig 15
08/07/2012 03:41:41;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=5835/state=S) after sig 15
08/07/2012 03:41:42;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=5835/state=S) after sig 15
08/07/2012 03:41:42;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: killing pid 5835 task 1 with sig 9
08/07/2012 03:41:42;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: killing pid 6012 task 1 gracefully with sig 15
08/07/2012 03:41:42;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=6012/state=R) after sig 15
08/07/2012 03:41:42;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=6012/state=S) after sig 15
08/07/2012 03:41:42;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=6012/state=S) after sig 15
08/07/2012 03:41:42;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=6012/state=S) after sig 15
08/07/2012 03:41:43;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=6012/state=S) after sig 15
08/07/2012 03:41:43;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=6012/state=S) after sig 15
08/07/2012 03:41:43;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=6012/state=S) after sig 15
08/07/2012 03:41:43;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=6012/state=S) after sig 15
08/07/2012 03:41:44;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=6012/state=S) after sig 15
08/07/2012 03:41:44;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=6012/state=S) after sig 15
08/07/2012 03:41:44;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=6012/state=S) after sig 15
08/07/2012 03:41:44;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=6012/state=S) after sig 15
08/07/2012 03:41:45;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=6012/state=S) after sig 15
08/07/2012 03:41:45;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=6012/state=S) after sig 15
08/07/2012 03:41:45;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=6012/state=S) after sig 15
08/07/2012 03:41:45;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=6012/state=S) after sig 15
08/07/2012 03:41:46;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=6012/state=S) after sig 15
08/07/2012 03:41:46;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=6012/state=S) after sig 15
08/07/2012 03:41:46;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=6012/state=S) after sig 15
08/07/2012 03:41:46;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=6012/state=S) after sig 15
08/07/2012 03:41:47;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: killing pid 6012 task 1 with sig 9
08/07/2012 03:41:47;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: killing pid 3605 task 1 gracefully with sig 15
08/07/2012 03:41:47;0008;   pbs_mom;Job;52779.pbs_server_name;kill_task: process (pid=3605/state=R) after sig 15
08/07/2012 03:41:47;0080;   pbs_mom;Job;52779.pbs_server_name;scan_for_terminated: job 52779.pbs_server_name task 1 terminated, sid=5835
08/07/2012 03:41:47;0080;   pbs_mom;Job;52777.pbs_server_name;obit sent to server
08/07/2012 03:41:47;0008;   pbs_mom;Job;52779.pbs_server_name;job was terminated
08/07/2012 03:41:47;0080;   pbs_mom;Svr;preobit_reply;top of preobit_reply
08/07/2012 03:41:47;0080;   pbs_mom;Svr;preobit_reply;DIS_reply_read/decode_DIS_replySvr worked, top of while loop
08/07/2012 03:41:47;0080;   pbs_mom;Svr;preobit_reply;in while loop, no error from job stat
08/07/2012 03:41:47;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: killing pid 5062 task 1 gracefully with sig 15
08/07/2012 03:41:47;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=5062/state=R) after sig 15
08/07/2012 03:41:47;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: killing pid 6018 task 1 gracefully with sig 15
08/07/2012 03:41:47;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6018/state=S) after sig 15
08/07/2012 03:41:47;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6018/state=S) after sig 15
08/07/2012 03:41:48;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6018/state=S) after sig 15
08/07/2012 03:41:48;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6018/state=S) after sig 15
08/07/2012 03:41:48;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6018/state=S) after sig 15
08/07/2012 03:41:48;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6018/state=S) after sig 15
08/07/2012 03:41:49;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6018/state=S) after sig 15
08/07/2012 03:41:49;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6018/state=S) after sig 15
08/07/2012 03:41:49;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6018/state=S) after sig 15
08/07/2012 03:41:49;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6018/state=S) after sig 15
08/07/2012 03:41:50;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6018/state=S) after sig 15
08/07/2012 03:41:50;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6018/state=S) after sig 15
08/07/2012 03:41:50;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6018/state=S) after sig 15
08/07/2012 03:41:50;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6018/state=S) after sig 15
08/07/2012 03:41:51;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6018/state=S) after sig 15
08/07/2012 03:41:51;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6018/state=S) after sig 15
08/07/2012 03:41:51;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6018/state=S) after sig 15
08/07/2012 03:41:51;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6018/state=S) after sig 15
08/07/2012 03:41:52;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6018/state=S) after sig 15
08/07/2012 03:41:52;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6018/state=S) after sig 15
08/07/2012 03:41:52;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: killing pid 6018 task 1 with sig 9
08/07/2012 03:41:52;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: killing pid 6297 task 1 gracefully with sig 15
08/07/2012 03:41:52;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6297/state=R) after sig 15
08/07/2012 03:41:52;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6297/state=S) after sig 15
08/07/2012 03:41:52;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6297/state=S) after sig 15
08/07/2012 03:41:53;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6297/state=S) after sig 15
08/07/2012 03:41:53;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6297/state=S) after sig 15
08/07/2012 03:41:53;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6297/state=S) after sig 15
08/07/2012 03:41:53;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6297/state=S) after sig 15
08/07/2012 03:41:54;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6297/state=S) after sig 15
08/07/2012 03:41:54;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6297/state=S) after sig 15
08/07/2012 03:41:54;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6297/state=S) after sig 15
08/07/2012 03:41:54;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6297/state=S) after sig 15
08/07/2012 03:41:55;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6297/state=S) after sig 15
08/07/2012 03:41:55;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6297/state=S) after sig 15
08/07/2012 03:41:55;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6297/state=S) after sig 15
08/07/2012 03:41:55;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6297/state=S) after sig 15
08/07/2012 03:41:56;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6297/state=S) after sig 15
08/07/2012 03:41:56;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6297/state=S) after sig 15
08/07/2012 03:41:56;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6297/state=S) after sig 15
08/07/2012 03:41:56;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6297/state=S) after sig 15
08/07/2012 03:41:57;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=6297/state=S) after sig 15
08/07/2012 03:41:57;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: killing pid 6297 task 1 with sig 9
08/07/2012 03:41:57;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: killing pid 32585 task 1 gracefully with sig 15
08/07/2012 03:41:57;0008;   pbs_mom;Job;52780.pbs_server_name;kill_task: process (pid=32585/state=R) after sig 15
08/07/2012 03:41:57;0080;   pbs_mom;Job;52780.pbs_server_name;scan_for_terminated: job 52780.pbs_server_name task 1 terminated, sid=6018
08/07/2012 03:41:57;0080;   pbs_mom;Job;52778.pbs_server_name;obit sent to server
08/07/2012 03:41:57;0008;   pbs_mom;Job;52780.pbs_server_name;job was terminated
08/07/2012 03:41:57;0080;   pbs_mom;Svr;preobit_reply;top of preobit_reply
08/07/2012 03:41:57;0080;   pbs_mom;Svr;preobit_reply;DIS_reply_read/decode_DIS_replySvr worked, top of while loop
08/07/2012 03:41:57;0080;   pbs_mom;Svr;preobit_reply;in while loop, no error from job stat
08/07/2012 03:41:57;0080;   pbs_mom;Job;52779.pbs_server_name;obit sent to server
08/07/2012 03:41:57;0001;   pbs_mom;Job;52772.pbs_server_name;job recycled into exiting on SIGNULL/KILL from substate 53
08/07/2012 03:41:57;0001;   pbs_mom;Job;52773.pbs_server_name;job recycled into exiting on SIGNULL/KILL from substate 53
08/07/2012 03:41:57;0001;   pbs_mom;Job;52774.pbs_server_name;job recycled into exiting on SIGNULL/KILL from substate 53
08/07/2012 03:41:57;0001;   pbs_mom;Job;52775.pbs_server_name;job recycled into exiting on SIGNULL/KILL from substate 53
08/07/2012 03:41:57;0001;   pbs_mom;Job;52776.pbs_server_name;job recycled into exiting on SIGNULL/KILL from substate 53
08/07/2012 03:41:57;0001;   pbs_mom;Job;52777.pbs_server_name;job recycled into exiting on SIGNULL/KILL from substate 53
08/07/2012 03:41:57;0001;   pbs_mom;Job;52780.pbs_server_name;job recycled into exiting on SIGNULL/KILL from substate 57
08/07/2012 03:41:57;0080;   pbs_mom;Svr;preobit_reply;top of preobit_reply
08/07/2012 03:41:57;0080;   pbs_mom;Svr;preobit_reply;DIS_reply_read/decode_DIS_replySvr worked, top of while loop
08/07/2012 03:41:57;0080;   pbs_mom;Svr;preobit_reply;cannot locate job that triggered req
08/07/2012 03:41:57;0080;   pbs_mom;Job;52769.pbs_server_name;removed job script
08/07/2012 03:41:57;0080;   pbs_mom;Job;52770.pbs_server_name;removed job script
08/07/2012 03:41:57;0080;   pbs_mom;Job;52771.pbs_server_name;removed job script
08/07/2012 03:41:57;0080;   pbs_mom;Job;52772.pbs_server_name;removed job script
08/07/2012 03:41:57;0080;   pbs_mom;Job;52773.pbs_server_name;removed job script
08/07/2012 03:41:57;0080;   pbs_mom;Svr;preobit_reply;top of preobit_reply
08/07/2012 03:41:57;0080;   pbs_mom;Svr;preobit_reply;DIS_reply_read/decode_DIS_replySvr worked, top of while loop
08/07/2012 03:41:57;0080;   pbs_mom;Svr;preobit_reply;cannot locate job that triggered req
08/07/2012 03:41:57;0080;   pbs_mom;Svr;preobit_reply;top of preobit_reply
08/07/2012 03:41:57;0080;   pbs_mom;Svr;preobit_reply;DIS_reply_read/decode_DIS_replySvr worked, top of while loop
08/07/2012 03:41:57;0080;   pbs_mom;Svr;preobit_reply;cannot locate job that triggered req
08/07/2012 03:41:57;0080;   pbs_mom;Svr;preobit_reply;top of preobit_reply
08/07/2012 03:41:57;0080;   pbs_mom;Svr;preobit_reply;DIS_reply_read/decode_DIS_replySvr worked, top of while loop
08/07/2012 03:41:57;0001;   pbs_mom;Job;52774.pbs_server_name;preobit_reply, unknown on server, deleting locally
08/07/2012 03:41:57;0080;   pbs_mom;Job;52774.pbs_server_name;removed job script
08/07/2012 03:41:57;0080;   pbs_mom;Svr;preobit_reply;top of preobit_reply
08/07/2012 03:41:57;0080;   pbs_mom;Svr;preobit_reply;DIS_reply_read/decode_DIS_replySvr worked, top of while loop
08/07/2012 03:41:57;0001;   pbs_mom;Job;52775.pbs_server_name;preobit_reply, unknown on server, deleting locally
08/07/2012 03:41:57;0080;   pbs_mom;Job;52775.pbs_server_name;removed job script
08/07/2012 03:41:57;0080;   pbs_mom;Svr;preobit_reply;top of preobit_reply
08/07/2012 03:41:57;0080;   pbs_mom;Svr;preobit_reply;DIS_reply_read/decode_DIS_replySvr worked, top of while loop
08/07/2012 03:41:57;0001;   pbs_mom;Job;52776.pbs_server_name;preobit_reply, unknown on server, deleting locally
08/07/2012 03:41:57;0080;   pbs_mom;Job;52776.pbs_server_name;removed job script
08/07/2012 03:41:57;0080;   pbs_mom;Req;req_reject;Reject reply code=15001(Unknown Job Id REJHOST=node0013 MSG=cannot locate job to delete), aux=0, type=DeleteJob, from PBS_Server at pbs_server_name
08/07/2012 03:41:57;0080;   pbs_mom;Req;req_reject;Reject reply code=15001(Unknown Job Id REJHOST=node0013 MSG=cannot locate job to delete), aux=0, type=DeleteJob, from PBS_Server at pbs_server_name
08/07/2012 03:41:57;0080;   pbs_mom;Req;req_reject;Reject reply code=15001(Unknown Job Id REJHOST=node0013 MSG=cannot locate job to delete), aux=0, type=DeleteJob, from PBS_Server at pbs_server_name
08/07/2012 03:41:57;0080;   pbs_mom;Svr;preobit_reply;top of preobit_reply
08/07/2012 03:41:57;0080;   pbs_mom;Svr;preobit_reply;DIS_reply_read/decode_DIS_replySvr worked, top of while loop
08/07/2012 03:41:57;0080;   pbs_mom;Svr;preobit_reply;in while loop, no error from job stat
08/07/2012 03:41:57;0080;   pbs_mom;Svr;preobit_reply;top of preobit_reply
08/07/2012 03:41:57;0080;   pbs_mom;Svr;preobit_reply;DIS_reply_read/decode_DIS_replySvr worked, top of while loop
08/07/2012 03:41:57;0080;   pbs_mom;Svr;preobit_reply;in while loop, no error from job stat
08/07/2012 03:41:57;0080;   pbs_mom;Job;52780.pbs_server_name;obit sent to server
08/07/2012 03:41:57;0080;   pbs_mom;Job;52777.pbs_server_name;obit sent to server
08/07/2012 03:41:57;0080;   pbs_mom;Job;52777.pbs_server_name;removed job script
08/07/2012 03:41:57;0001;   pbs_mom;Job;52778.pbs_server_name;job recycled into exiting on SIGNULL/KILL from substate 53
08/07/2012 03:41:57;0001;   pbs_mom;Job;52779.pbs_server_name;job recycled into exiting on SIGNULL/KILL from substate 53
08/07/2012 03:41:57;0001;   pbs_mom;Req;obit reply;Job not found for obit reply
08/07/2012 03:41:57;0080;   pbs_mom;Job;52779.pbs_server_name;removed job script
08/07/2012 03:41:57;0080;   pbs_mom;Job;52778.pbs_server_name;removed job script
08/07/2012 03:41:57;0080;   pbs_mom;Svr;preobit_reply;top of preobit_reply
08/07/2012 03:41:57;0080;   pbs_mom;Svr;preobit_reply;DIS_reply_read/decode_DIS_replySvr worked, top of while loop
08/07/2012 03:41:57;0080;   pbs_mom;Svr;preobit_reply;cannot locate job that triggered req
08/07/2012 03:41:57;0080;   pbs_mom;Svr;preobit_reply;top of preobit_reply
08/07/2012 03:41:57;0080;   pbs_mom;Svr;preobit_reply;DIS_reply_read/decode_DIS_replySvr worked, top of while loop
08/07/2012 03:41:57;0080;   pbs_mom;Svr;preobit_reply;cannot locate job that triggered req
08/07/2012 03:41:57;0080;   pbs_mom;Job;52780.pbs_server_name;removed job script
08/07/2012 03:43:11;0002;   pbs_mom;Svr;pbs_mom;Torque Mom Version = 3.0.2, loglevel = 0
08/07/2012 03:48:11;0002;   pbs_mom;Svr;pbs_mom;Torque Mom Version = 3.0.2, loglevel = 0
08/07/2012 03:53:11;0002;   pbs_mom;Svr;pbs_mom;Torque Mom Version = 3.0.2, loglevel = 0
08/07/2012 03:54:12;0001;   pbs_mom;Svr;pbs_mom;LOG_DEBUG::mom_checkpoint_job_has_checkpoint, FALSE
08/07/2012 03:54:12;0001;   pbs_mom;Job;TMomFinalizeJob3;job 56282.pbs_server_name started, pid = 7265
08/07/2012 03:54:12;0001;   pbs_mom;Svr;pbs_mom;LOG_DEBUG::mom_checkpoint_job_has_checkpoint, FALSE
08/07/2012 03:54:12;0001;   pbs_mom;Job;TMomFinalizeJob3;job 56283.pbs_server_name started, pid = 7369
08/07/2012 03:54:12;0001;   pbs_mom;Svr;pbs_mom;LOG_DEBUG::mom_checkpoint_job_has_checkpoint, FALSE
08/07/2012 03:54:12;0001;   pbs_mom;Job;TMomFinalizeJob3;job 56284.pbs_server_name started, pid = 7454
08/07/2012 03:54:12;0001;   pbs_mom;Svr;pbs_mom;LOG_DEBUG::mom_checkpoint_job_has_checkpoint, FALSE
08/07/2012 03:54:12;0001;   pbs_mom;Job;TMomFinalizeJob3;job 56285.pbs_server_name started, pid = 7562
08/07/2012 03:54:12;0001;   pbs_mom;Svr;pbs_mom;LOG_DEBUG::mom_checkpoint_job_has_checkpoint, FALSE
08/07/2012 03:54:12;0001;   pbs_mom;Job;TMomFinalizeJob3;job 56286.pbs_server_name started, pid = 8959
08/07/2012 03:54:12;0001;   pbs_mom;Svr;pbs_mom;LOG_DEBUG::mom_checkpoint_job_has_checkpoint, FALSE
08/07/2012 03:54:12;0001;   pbs_mom;Job;TMomFinalizeJob3;job 56287.pbs_server_name started, pid = 9080
08/07/2012 03:54:12;0001;   pbs_mom;Svr;pbs_mom;LOG_DEBUG::mom_checkpoint_job_has_checkpoint, FALSE
08/07/2012 03:54:12;0001;   pbs_mom;Job;TMomFinalizeJob3;job 56288.pbs_server_name started, pid = 9174
08/07/2012 03:54:12;0001;   pbs_mom;Svr;pbs_mom;LOG_DEBUG::mom_checkpoint_job_has_checkpoint, FALSE
08/07/2012 03:54:12;0001;   pbs_mom;Job;TMomFinalizeJob3;job 56289.pbs_server_name started, pid = 9431
08/07/2012 03:54:13;0001;   pbs_mom;Svr;pbs_mom;LOG_DEBUG::mom_checkpoint_job_has_checkpoint, FALSE
08/07/2012 03:54:13;0001;   pbs_mom;Job;TMomFinalizeJob3;job 56290.pbs_server_name started, pid = 10913
08/07/2012 03:54:13;0001;   pbs_mom;Svr;pbs_mom;LOG_DEBUG::mom_checkpoint_job_has_checkpoint, FALSE
08/07/2012 03:54:13;0001;   pbs_mom;Job;TMomFinalizeJob3;job 56291.pbs_server_name started, pid = 11034
08/07/2012 03:54:13;0001;   pbs_mom;Svr;pbs_mom;LOG_DEBUG::mom_checkpoint_job_has_checkpoint, FALSE
08/07/2012 03:54:13;0001;   pbs_mom;Job;TMomFinalizeJob3;job 56292.pbs_server_name started, pid = 11109
08/07/2012 03:54:13;0001;   pbs_mom;Svr;pbs_mom;LOG_DEBUG::mom_checkpoint_job_has_checkpoint, FALSE
08/07/2012 03:54:13;0001;   pbs_mom;Job;TMomFinalizeJob3;job 56293.pbs_server_name started, pid = 11247
08/07/2012 03:58:11;0002;   pbs_mom;Svr;pbs_mom;Torque Mom Version = 3.0.2, loglevel = 0
08/07/2012 04:03:11;0002;   pbs_mom;Svr;pbs_mom;Torque Mom Version = 3.0.2, loglevel = 0
08/07/2012 04:08:11;0002;   pbs_mom;Svr;pbs_mom;Torque Mom Version = 3.0.2, loglevel = 0
08/07/2012 04:13:11;0002;   pbs_mom;Svr;pbs_mom;Torque Mom Version = 3.0.2, loglevel = 0
08/07/2012 04:18:11;0002;   pbs_mom;Svr;pbs_mom;Torque Mom Version = 3.0.2, loglevel = 0
08/07/2012 04:23:11;0002;   pbs_mom;Svr;pbs_mom;Torque Mom Version = 3.0.2, loglevel = 0
08/07/2012 04:28:11;0002;   pbs_mom;Svr;pbs_mom;Torque Mom Version = 3.0.2, loglevel = 0
08/07/2012 04:33:11;0002;   pbs_mom;Svr;pbs_mom;Torque Mom Version = 3.0.2, loglevel = 0
08/07/2012 04:38:11;0002;   pbs_mom;Svr;pbs_mom;Torque Mom Version = 3.0.2, loglevel = 0
08/07/2012 04:43:11;0002;   pbs_mom;Svr;pbs_mom;Torque Mom Version = 3.0.2, loglevel = 0



More information about the torqueusers mailing list