[torqueusers] Torque-2.1.6 Problem with pbs_mom logging - get_proc_stat

[Bill Wichser]

As a followup, I'm also seeing many syslog entries like this as well 
(from multiple hosts):

pbs_mom: Success (0) in cput_sum, 6426: get_proc_stat
pbs_mom: Success (0) in cput_sum, 6779: get_proc_stat
pbs_mom: Inappropriate ioctl for device (25) in mem_sum, 6779: get_proc_stat
pbs_mom: Inappropriate ioctl for device (25) in mem_sum, 6426: get_proc_stat
pbs_mom: Success (0) in cput_sum, 6221: get_proc_stat
pbs_mom: Success (0) in cput_sum, 6247: get_proc_stat
pbs_mom: Inappropriate ioctl for device (25) in resi_sum, 6779: 
get_proc_stat
pbs_mom: Success (0) in cput_sum, 5836: get_proc_stat
pbs_mom: Success (0) in cput_sum, 5645: get_proc_stat
pbs_mom: Success (0) in cput_sum, 5562: get_proc_stat
pbs_mom: Success (0) in cput_sum, 6267: get_proc_stat
pbs_mom: Success (0) in cput_sum, 6905: get_proc_stat
pbs_mom: Success (0) in cput_sum, 6293: get_proc_stat
pbs_mom: Success (0) in cput_sum, 6284: get_proc_stat
pbs_mom: Inappropriate ioctl for device (25) in mem_sum, 6221: get_proc_stat
pbs_mom: Inappropriate ioctl for device (25) in mem_sum, 6247: get_proc_stat
pbs_mom: Inappropriate ioctl for device (25) in mem_sum, 5836: get_proc_stat

Again, only since a kernel update have we been seeing these log entries. 
  Is it possible that something else is logging these events and not the 
pbs_mom directly?

An strace on the syslog daemon finds this:
select(1, [0], NULL, NULL, NULL)        = 1 (in [0])
recvfrom(0, "<27>Jul  3 09:39:42 pbs_mom: Suc"..., 1022, 0, NULL, NULL) = 73
writev(1, [{"Jul  3 09:39:42", 15}, {" ", 1}, {"woodhen-004", 11}, {" ", 
1}, {"p
bs_mom: Success (0) in cput_sum"..., 53}, {"\n", 1}], 6) = 82
fsync(1)                                = 0

Thanks, Bill

Bill Wichser wrote:
> Since upgrading this morning to a new kernel (2.6.9-55.0.2.ELsmp) and IB 
> drivers, the pbs_mom on the nodes have been logging (to syslog) a 
> constant barrage of successes.
> 
> pbs_mom: Success (0) in sessions, 5720: get_proc_stat
> pbs_mom: Success (0) in sessions, 5720: get_proc_stat
> pbs_mom: Success (0) in nusers, 5720: get_proc_stat
> 
> every 15 seconds.  In the mom_priv/config I have turned back the logging 
> from 255 to 128 to 127 with no change.  I'm not sure why this has 
> suddenly started but an surely open for any suggestions on how to 
> squelch these succeses.
> 
> Thanks,
> Bill
> _______________________________________________
> torqueusers mailing list
> torqueusers at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torqueusers