[torqueusers] pbs_server not keeping up

Tony Schreiner anthony.schreiner at bc.edu
Wed Aug 29 08:38:33 MDT 2012


On my smallish cluster with torque 2.5.7.

A user submitted about 8000 jobs to a routing queue, which feeds to an execution queue with 200 runnable slots.

At the moment, bps_server is unable to handle it,  pbsnodes returns no nodes found, qstat -q takes a long time and shows nothing.
This is the tail of the latest server_logs file

08/29/2012 10:26:03;0004;PBS_Server;Svr;check_nodes;node linux24 not detected in 1346250363 seconds, marking node down
08/29/2012 10:26:03;0004;PBS_Server;Svr;check_nodes;node linux25 not detected in 1346250363 seconds, marking node down
08/29/2012 10:26:03;0004;PBS_Server;Svr;check_nodes;node linux27 not detected in 1346250363 seconds, marking node down
08/29/2012 10:26:03;0004;PBS_Server;Svr;check_nodes;node linux29 not detected in 1346250363 seconds, marking node down
08/29/2012 10:26:40;0008;PBS_Server;Job;NULL;on_job_exit valid pjob: 0xe3d4180 (substate=51)
08/29/2012 10:27:18;0008;PBS_Server;Job;NULL;on_job_exit valid pjob: 0xe3d4180 (substate=51)
08/29/2012 10:27:18;000d;PBS_Server;Job;797357.portal;Post job file processing error; job 797357.portal on host linux29/1
08/29/2012 10:27:18;0008;PBS_Server;Job;NULL;on_job_exit valid pjob: 0xe3d4180 (substate=53)
08/29/2012 10:28:33;0004;PBS_Server;Svr;check_nodes;node linux20 not detected in 1346250513 seconds, marking node down

here are some server settings

set server log_events = 511
set server mail_from = adm
set server query_other_jobs = True
set server resources_default.ncpus = 1
set server resources_default.nodect = 1
set server scheduler_iteration = 600
set server node_check_rate = 150
set server tcp_timeout = 6
set server mom_job_sync = True
set server keep_completed = 86400
set server log_keep_days = 365
set server next_job_number = 808497
set server job_log_keep_days = 365


is there anything I can change to help move things along.
Thanks

Tony Schreiner


More information about the torqueusers mailing list