[torqueusers] jobs with nodes=x x>1 are packing on single node

[P Spencer Davis]

Hi,
    In a nutshell, I'm trying to configure Torque(2.1.6)/Maui(3.2.6p19) 
so that jobs will be spread over multiple nodes instead of bunched up on 
a single node.

Currently when I submit

echo "cat $PBS_NODESFILE" | qsub -l nodes=4

I get back something like this

n31
n31
n31
n31

I've tried a variety of options in my maui.cfg file, such as setting the 
BACKFILLPOLICY to NONE (no effect) and NODEALLOCATIONPOLICY to 
MAXBALANCE (which caused jobs requesting more than one node/cpu to 
hang), but honestly I'm not even sure that was the right place to start 
after reading some of the previous posts around this same topic.

Any ideas on what needs to be tweaked?

                        Spencer Davis


Qmgr: list server
Server xxx.xxx.xxx
         server_state = Active
         scheduling = True
         total_jobs = 11
         state_count = Transit:0 Queued:0 Held:0 Waiting:0 Running:11 
Exiting:0
         managers = xxx at xxx.xxx.xxx,
         operators = xxx at xxxx.xxx.xxx
         default_queue = batch
         log_events = 511
         mail_from = adm
         resources_default.mem = 4mb
         resources_assigned.mem = 46137344b
         resources_assigned.nodect = 11
         scheduler_iteration = 600
         node_check_rate = 150
         tcp_timeout = 6
         node_pack = False
         pbs_version = 2.1.6

Qmgr: q