[torqueusers] Does any one have a Qsub filter/wrapper that they would share

Sreedhar Manchu sm4082 at nyu.edu
Mon Sep 30 10:46:18 MDT 2013


Hi Jeffrey,

I use this wrapper on our systems at NYU. This works well for us. It is easy to modify it as all the functions start from the end in their order of execution. Find the attachment and modify the line 727 from 'done' to done < $bps_script_name for testing. I have put some examples below where I tested it on my mac desktop.

Please let me know if you have questions about it. It has 859 lines but it is a very easy to follow and does nothing but checking and grabbing node count, ppn count , memory, walltime and adds some lines based on that. If users don't follow the rules we specified it rejects their job and prints some helpful information at the job submission.


Sreedhar.
New York University.

727c727
< 	done
---
> 	done < $pbs_script_name

Now an example:

If we were to submit: qsub -I this is what we get:


sreedhars-imac:Desktop manchu$ ./submit.sh -I
#PBS -l mem=2048MB
#PBS -l feature=serial

If I were to submit a pbs script with qsub test.pbs this is what happens:

sreedhars-imac:Desktop manchu$ cat test.pbs 
#!/bin/sh

#PBS -l nodes=2:ppn=12,walltime=5:00,mem=2GB
#PBS -q s48

hostname
sreedhars-imac:Desktop manchu$ ./submit.sh test.pbs 
#!/bin/sh

#PBS -l nodes=2:ppn=12,walltime=5:00,mem=2GB
#PBS -q s48


###-----Wrapper Added-----###


Maximum node count for a serial job is 1.

For more information please refer to HPC wiki at
https://wikis.nyu.edu/display/NYUHPC/Queues

sreedhars-imac:Desktop manchu$ 

Since I put it under serial 48 hour queue it gets rejected as script has requested 2 nodes.

If I correct this:

sreedhars-imac:Desktop manchu$ cat test.pbs 
#!/bin/sh

#PBS -l nodes=1:ppn=12,walltime=5:00,mem=2GB
#PBS -q s48

hostname
sreedhars-imac:Desktop manchu$ ./submit.sh test.pbs 
#!/bin/sh

#PBS -l nodes=1:ppn=12,walltime=5:00,mem=2GB
#PBS -q s48


###-----Wrapper Added-----###

#PBS -l feature=serial

###-----Wrapper Done-----###

hostname
sreedhars-imac:Desktop manchu$

On a submit host we see a job id as it went through and the content shown below gets added to the script just like shown above.

###-----Wrapper Added-----###

#PBS -l feature=serial

###-----Wrapper Done-----###

Memory example:

sreedhars-imac:Desktop manchu$ cat test.pbs 
#!/bin/sh

#PBS -l nodes=1:ppn=12,walltime=5:00,mem=200GB
#PBS -q s48

hostname
sreedhars-imac:Desktop manchu$ ./submit.sh test.pbs 
#!/bin/sh

#PBS -l nodes=1:ppn=12,walltime=5:00,mem=200GB
#PBS -q s48


###-----Wrapper Added-----###


Maximum memory available for serial jobs under s48
queue is 46GB. If you need more than this you can
use bigmem queue.

For more information please refer to HPC wiki at
https://wikis.nyu.edu/display/NYUHPC/Queues

sreedhars-imac:Desktop manchu$

Just to show you I am pasting here what our server configuration looks like:

[root at hpc0 ~]# qmgr -c 'p s'
#
# Create queues and set their attributes.
#
#
# Create and define queue sysadm
#
create queue sysadm
set queue sysadm queue_type = Execution
set queue sysadm Priority = 10000000
set queue sysadm acl_user_enable = True
set queue sysadm acl_users = ciancia
set queue sysadm acl_users += <snipped>
set queue sysadm acl_users += <snipped>
set queue sysadm acl_users += <snipped>
set queue sysadm acl_users += <snipped>
set queue sysadm acl_users += <snipped>
set queue sysadm acl_users += <snipped>
set queue sysadm acl_users += <snipped>
set queue sysadm enabled = True
set queue sysadm started = True
#
# Create and define queue batch
#
create queue batch
set queue batch queue_type = Execution
set queue batch resources_default.nodes = 1
set queue batch resources_default.walltime = 01:00:00
set queue batch enabled = False
set queue batch started = False
#
# Create and define queue p12
#
create queue p12
set queue p12 queue_type = Execution
set queue p12 max_user_queuable = 1000
set queue p12 resources_max.mem = 46gb
set queue p12 resources_max.walltime = 12:00:00
set queue p12 resources_min.nodect = 2
set queue p12 resources_default.neednodes = parallel
set queue p12 resources_default.walltime = 01:00:00
set queue p12 disallowed_types = interactive
set queue p12 enabled = True
set queue p12 started = True
#
# Create and define queue bigmem
#
create queue bigmem
set queue bigmem queue_type = Execution
set queue bigmem Priority = 100000
set queue bigmem max_user_queuable = 50
set queue bigmem resources_max.mem = 1000gb
set queue bigmem resources_max.nodect = 8
set queue bigmem resources_max.walltime = 48:00:00
set queue bigmem resources_min.mem = 47gb
set queue bigmem resources_min.nodect = 1
set queue bigmem resources_default.neednodes = fat
set queue bigmem resources_default.walltime = 01:00:00
set queue bigmem enabled = True
set queue bigmem started = True
#
# Create and define queue route
#
create queue route
set queue route queue_type = Route
set queue route max_user_queuable = 100
set queue route resources_max.mem = 46gb
set queue route resources_default.walltime = 01:00:00
set queue route route_destinations = s48
set queue route route_destinations += s96
set queue route route_destinations += p12
set queue route route_destinations += p48
set queue route enabled = True
set queue route started = True
#
# Create and define queue s96
#
create queue s96
set queue s96 queue_type = Execution
set queue s96 max_user_queuable = 500
set queue s96 resources_max.mem = 46gb
set queue s96 resources_max.nodect = 1
set queue s96 resources_max.walltime = 96:00:00
set queue s96 resources_min.walltime = 48:01:00
set queue s96 resources_default.neednodes = serial
set queue s96 resources_default.walltime = 01:00:00
set queue s96 checkpoint_defaults = enabled
set queue s96 enabled = True
set queue s96 started = True
#
# Create and define queue p48
#
create queue p48
set queue p48 queue_type = Execution
set queue p48 max_user_queuable = 200
set queue p48 resources_max.mem = 46gb
set queue p48 resources_max.walltime = 48:00:00
set queue p48 resources_min.nodect = 2
set queue p48 resources_min.walltime = 12:01:00
set queue p48 resources_default.neednodes = parallel
set queue p48 resources_default.walltime = 01:00:00
set queue p48 disallowed_types = interactive
set queue p48 enabled = True
set queue p48 started = True
#
# Create and define queue interactive
#
create queue interactive
set queue interactive queue_type = Execution
set queue interactive Priority = 1500
set queue interactive resources_max.mem = 46gb
set queue interactive resources_max.nodect = 2
set queue interactive resources_max.walltime = 04:00:00
set queue interactive resources_default.walltime = 01:00:00
set queue interactive max_user_run = 2
set queue interactive enabled = True
set queue interactive started = True
#
# Create and define queue gpu
#
create queue gpu
set queue gpu queue_type = Execution
set queue gpu max_user_queuable = 10
set queue gpu resources_max.mem = 94gb
set queue gpu resources_max.nodect = 2
set queue gpu resources_max.walltime = 48:00:00
set queue gpu resources_default.neednodes = gpu
set queue gpu resources_default.walltime = 01:00:00
set queue gpu enabled = True
set queue gpu started = True
#
# Create and define queue s48
#
create queue s48
set queue s48 queue_type = Execution
set queue s48 max_user_queuable = 1500
set queue s48 resources_max.mem = 46gb
set queue s48 resources_max.nodect = 1
set queue s48 resources_max.walltime = 48:00:00
set queue s48 resources_min.walltime = 00:01:00
set queue s48 resources_default.neednodes = serial
set queue s48 resources_default.walltime = 01:00:00
set queue s48 checkpoint_defaults = enabled
set queue s48 enabled = True
set queue s48 started = True
#
# Set server attributes.
#
set server scheduling = True
set server acl_host_enable = False
set server acl_hosts = hpc0.abudhabi.nyu.edu
set server acl_hosts += hpc0.local
set server managers = root at hpc0.abudhabi.nyu.edu
set server operators = root at hpc0.abudhabi.nyu.edu
set server default_queue = route
set server log_events = 511
set server mail_from = adm
set server query_other_jobs = True
set server scheduler_iteration = 600
set server node_check_rate = 150
set server tcp_timeout = 6
set server log_level = 1
set server mom_job_sync = True
set server keep_completed = 86400
set server submit_hosts = login-0-1
set server submit_hosts += login-0-2
set server submit_hosts += login-0-3
set server submit_hosts += login-0-4
set server allow_node_submit = False
set server log_file_max_size = 51200
set server log_file_roll_depth = 10
set server log_keep_days = 7
set server next_job_number = 391817
[root at hpc0 ~]#
 
This is where I have it on the login nodes:

[root at login-0-1 ~]# cat /opt/torque/torque.cfg 
SUBMITFILTER /share/apps/admins/torque/submit.sh 
[root at login-0-1 ~]#

This is how our nodes file in /opt/torque/server_priv/nodes looks like:

[root at hpc0 ~]# cat /opt/torque/server_priv/nodes
compute-1-1 np=12 serial interactive
compute-1-2 np=12 serial interactive
compute-1-3 np=12 serial interactive
compute-1-4 np=12 serial interactive
compute-1-5 np=12 serial interactive
compute-1-6 np=12 serial interactive
compute-1-7 np=12 serial interactive
compute-1-8 np=12 serial interactive
compute-1-9 np=12 serial interactive
compute-1-10 np=12 serial interactive
compute-1-11 np=12 serial interactive
compute-1-12 np=12 serial interactive
compute-1-13 np=12 serial interactive
compute-1-14 np=12 serial interactive
compute-1-15 np=12 serial interactive
compute-1-16 np=12 serial interactive
compute-1-17 np=12 serial interactive
compute-1-18 np=12 serial interactive
compute-1-19 np=12 serial interactive
compute-1-20 np=12 serial interactive
compute-1-21 np=12 serial interactive
compute-1-22 np=12 serial interactive
compute-1-23 np=12 serial interactive
compute-1-24 np=12 serial interactive
compute-1-25 np=12 serial interactive
compute-1-26 np=12 serial interactive
compute-1-27 np=12 serial interactive
compute-1-28 np=12 serial interactive
compute-1-29 np=12 serial interactive
compute-1-30 np=12 serial interactive
compute-1-31 np=12 serial interactive
compute-1-32 np=12 serial interactive
compute-2-1 np=12 serial interactive
compute-2-2 np=12 serial interactive
compute-2-3 np=12 serial interactive
compute-2-4 np=12 serial interactive
compute-2-5 np=12 serial interactive
compute-2-6 np=12 serial interactive
compute-2-7 np=12 serial interactive
compute-2-8 np=12 serial interactive
compute-2-9 np=12 serial interactive
compute-2-10 np=12 serial interactive
compute-2-11 np=12 serial interactive
compute-2-12 np=12 serial interactive
compute-2-13 np=12 serial interactive
compute-2-14 np=12 serial interactive
compute-2-15 np=12 serial interactive
compute-2-16 np=12 serial interactive
compute-2-17 np=12 serial interactive
compute-2-18 np=12 serial interactive
compute-2-19 np=12 serial interactive
compute-2-20 np=12 serial interactive
compute-2-21 np=12 serial interactive
compute-2-22 np=12 serial interactive
compute-2-23 np=12 serial interactive
compute-2-24 np=12 serial interactive
compute-2-25 np=12 serial interactive
compute-2-26 np=12 serial interactive
compute-2-27 np=12 serial interactive
compute-2-28 np=12 serial interactive
compute-2-29 np=12 serial interactive
compute-2-30 np=12 serial interactive
compute-2-31 np=12 serial interactive
compute-2-32 np=12 serial interactive
compute-3-1 np=12 chassis3 interactive serial
compute-3-2 np=12 chassis3 interactive serial
compute-3-3 np=12 chassis3 interactive serial
compute-3-4 np=12 chassis3 interactive serial
compute-3-5 np=12 chassis3 interactive serial
compute-3-6 np=12 chassis3 interactive serial
compute-3-7 np=12 chassis3 interactive serial
compute-3-8 np=12 chassis3 interactive serial
compute-3-9 np=12 chassis3 interactive serial
compute-3-10 np=12 chassis3 interactive serial
compute-3-11 np=12 chassis3 interactive serial
compute-3-12 np=12 chassis3 interactive serial
compute-3-13 np=12 chassis3 interactive serial
compute-3-14 np=12 chassis3 interactive serial
compute-3-15 np=12 chassis3 interactive serial
compute-3-16 np=12 chassis3 interactive serial
compute-3-17 np=12 chassis3 interactive serial
compute-3-18 np=12 chassis3 interactive serial
compute-3-19 np=12 chassis3 interactive serial
compute-3-20 np=12 chassis3 interactive serial
compute-3-21 np=12 chassis3 interactive serial
compute-3-22 np=12 chassis3 interactive serial
compute-3-23 np=12 chassis3 interactive serial
compute-3-24 np=12 chassis3 interactive serial
compute-3-25 np=12 chassis3 interactive serial
compute-3-26 np=12 chassis3 interactive serial
compute-3-27 np=12 chassis3 interactive serial
compute-3-28 np=12 chassis3 interactive serial
compute-3-29 np=12 chassis3 interactive serial
compute-3-30 np=12 chassis3 interactive serial
compute-3-31 np=12 chassis3 interactive serial
compute-3-32 np=12 chassis3 interactive serial
compute-4-1 np=12 parallel chassis4 interactive
compute-4-2 np=12 parallel chassis4 interactive
compute-4-3 np=12 parallel chassis4 interactive
compute-4-4 np=12 parallel chassis4 interactive
compute-4-5 np=12 parallel chassis4 interactive
compute-4-6 np=12 parallel chassis4 interactive
compute-4-7 np=12 parallel chassis4 interactive
compute-4-8 np=12 parallel chassis4 interactive
compute-4-9 np=12 parallel chassis4 interactive
compute-4-10 np=12 parallel chassis4 interactive
compute-4-11 np=12 parallel chassis4 interactive
compute-4-12 np=12 parallel chassis4 interactive
compute-4-13 np=12 parallel chassis4 interactive
compute-4-14 np=12 parallel chassis4 interactive
compute-4-15 np=12 parallel chassis4 interactive
compute-4-16 np=12 parallel chassis4 interactive
compute-4-17 np=12 parallel chassis4 interactive
compute-4-18 np=12 parallel chassis4 interactive
compute-4-19 np=12 parallel chassis4 interactive
compute-4-20 np=12 parallel chassis4 interactive
compute-4-21 np=12 parallel chassis4 interactive
compute-4-22 np=12 parallel chassis4 interactive
compute-4-23 np=12 parallel chassis4 interactive
compute-4-24 np=12 parallel chassis4 interactive
compute-4-25 np=12 parallel chassis4 interactive
compute-4-26 np=12 parallel chassis4 interactive
compute-4-27 np=12 parallel chassis4 interactive
compute-4-28 np=12 parallel chassis4 interactive
compute-4-29 np=12 parallel chassis4 interactive
compute-4-30 np=12 parallel chassis4 interactive
compute-4-31 np=12 parallel chassis4 interactive
compute-4-32 np=12 parallel chassis4 interactive
compute-5-1 np=12 parallel chassis5 interactive
compute-5-2 np=12 parallel chassis5 interactive
compute-5-3 np=12 parallel chassis5 interactive
compute-5-4 np=12 parallel chassis5 interactive
compute-5-5 np=12 parallel chassis5 interactive
compute-5-6 np=12 parallel chassis5 interactive
compute-5-7 np=12 parallel chassis5 interactive
compute-5-8 np=12 parallel chassis5 interactive
compute-5-9 np=12 parallel chassis5 interactive
compute-5-10 np=12 parallel chassis5 interactive
compute-5-11 np=12 parallel chassis5 interactive
compute-5-12 np=12 parallel chassis5 interactive
compute-5-13 np=12 parallel chassis5 interactive
compute-5-14 np=12 parallel chassis5 interactive
compute-5-15 np=12 parallel chassis5 interactive
compute-5-16 np=12 parallel chassis5 interactive
compute-5-17 np=12 parallel chassis5 interactive
compute-5-18 np=12 parallel chassis5 interactive
compute-5-19 np=12 parallel chassis5 interactive
compute-5-20 np=12 parallel chassis5 interactive
compute-5-21 np=12 parallel chassis5 interactive
compute-5-22 np=12 parallel chassis5 interactive
compute-5-23 np=12 parallel chassis5 interactive
compute-5-24 np=12 parallel chassis5 interactive
compute-5-25 np=12 parallel chassis5 interactive
compute-5-26 np=12 parallel chassis5 interactive
compute-5-27 np=12 parallel chassis5 interactive
compute-5-28 np=12 parallel chassis5 interactive
compute-5-29 np=12 parallel chassis5 interactive
compute-5-30 np=12 parallel chassis5 interactive
compute-5-31 np=12 parallel chassis5 interactive
compute-5-32 np=12 parallel chassis5 interactive
compute-6-1 np=12 parallel chassis6 interactive
compute-6-2 np=12 parallel chassis6 interactive
compute-6-3 np=12 parallel chassis6 interactive
compute-6-4 np=12 parallel chassis6 interactive
compute-6-5 np=12 parallel chassis6 interactive
compute-6-6 np=12 parallel chassis6 interactive
compute-6-7 np=12 parallel chassis6 interactive
compute-6-8 np=12 parallel chassis6 interactive
compute-6-9 np=12 parallel chassis6 interactive
compute-6-10 np=12 parallel chassis6 interactive
compute-6-11 np=12 parallel chassis6 interactive
compute-6-12 np=12 parallel chassis6 interactive
compute-6-13 np=12 parallel chassis6 interactive
compute-6-14 np=12 parallel chassis6 interactive
compute-6-15 np=12 parallel chassis6 interactive
compute-6-16 np=12 parallel chassis6 interactive
compute-6-17 np=12 parallel chassis6 interactive
compute-6-18 np=12 parallel chassis6 interactive
compute-6-19 np=12 parallel chassis6 interactive
compute-6-20 np=12 parallel chassis6 interactive
compute-6-21 np=12 parallel chassis6 interactive
compute-6-22 np=12 parallel chassis6 interactive
compute-6-23 np=12 parallel chassis6 interactive
compute-6-24 np=12 parallel chassis6 interactive
compute-6-25 np=12 parallel chassis6 interactive
compute-6-26 np=12 parallel chassis6 interactive
compute-6-27 np=12 parallel chassis6 interactive
compute-6-28 np=12 parallel chassis6 interactive
compute-6-29 np=12 parallel chassis6 interactive
compute-6-30 np=12 parallel chassis6 interactive
compute-6-31 np=12 parallel chassis6 interactive
compute-6-32 np=12 parallel chassis6 interactive
compute-7-1 np=12 parallel chassis7 interactive
compute-7-2 np=12 parallel chassis7 interactive
compute-7-3 np=12 parallel chassis7 interactive
compute-7-4 np=12 parallel chassis7 interactive
compute-7-5 np=12 parallel chassis7 interactive
compute-7-6 np=12 parallel chassis7 interactive
compute-7-7 np=12 parallel chassis7 interactive
compute-7-8 np=12 parallel chassis7 interactive
compute-7-9 np=12 parallel chassis7 interactive
compute-7-10 np=12 parallel chassis7 interactive
compute-7-11 np=12 parallel chassis7 interactive
compute-7-12 np=12 parallel chassis7 interactive
compute-7-13 np=12 parallel chassis7 interactive
compute-7-14 np=12 parallel chassis7 interactive
compute-7-15 np=12 parallel chassis7 interactive
compute-7-16 np=12 parallel chassis7 interactive
compute-7-17 np=12 parallel chassis7 interactive
compute-7-18 np=12 parallel chassis7 interactive
compute-7-19 np=12 parallel chassis7 interactive
compute-7-20 np=12 parallel chassis7 interactive
compute-7-21 np=12 parallel chassis7 interactive
compute-7-22 np=12 parallel chassis7 interactive
compute-7-23 np=12 parallel chassis7 interactive
compute-7-24 np=12 parallel chassis7 interactive
compute-7-25 np=12 parallel chassis7 interactive
compute-7-26 np=12 parallel chassis7 interactive
compute-7-27 np=12 parallel chassis7 interactive
compute-7-28 np=12 parallel chassis7 interactive
compute-7-29 np=12 parallel chassis7 interactive
compute-7-30 np=12 parallel chassis7 interactive
compute-7-31 np=12 parallel chassis7 interactive
compute-7-32 np=12 parallel chassis7 interactive
compute-8-1 np=12 parallel chassis8 interactive
compute-8-2 np=12 parallel chassis8 interactive
compute-8-3 np=12 parallel chassis8 interactive
compute-8-4 np=12 parallel chassis8 interactive
compute-8-5 np=12 parallel chassis8 interactive
compute-8-6 np=12 parallel chassis8 interactive
compute-8-7 np=12 parallel chassis8 interactive
compute-8-8 np=12 parallel chassis8 interactive
compute-8-9 np=12 parallel chassis8 interactive
compute-8-10 np=12 parallel chassis8 interactive
compute-8-11 np=12 parallel chassis8 interactive
compute-8-12 np=12 parallel chassis8 interactive
compute-8-13 np=12 parallel chassis8 interactive
compute-8-14 np=12 parallel chassis8 interactive
compute-8-15 np=12 parallel chassis8 interactive
compute-8-16 np=12 parallel chassis8 interactive
compute-8-17 np=12 parallel chassis8 interactive
compute-8-18 np=12 parallel chassis8 interactive
compute-8-19 np=12 parallel chassis8 interactive
compute-8-20 np=12 parallel chassis8 interactive
compute-8-21 np=12 parallel chassis8 interactive
compute-8-22 np=12 parallel chassis8 interactive
compute-8-23 np=12 parallel chassis8 interactive
compute-8-24 np=12 parallel chassis8 interactive
compute-8-25 np=12 parallel chassis8 interactive
compute-8-26 np=12 parallel chassis8 interactive
compute-8-27 np=12 parallel chassis8 interactive
compute-8-28 np=12 parallel chassis8 interactive
compute-8-29 np=12 parallel chassis8 interactive
compute-8-30 np=12 parallel chassis8 interactive
compute-8-31 np=12 parallel chassis8 interactive
compute-8-32 np=12 parallel chassis8 interactive
compute-9-1 np=12 parallel chassis9 interactive
compute-9-2 np=12 parallel chassis9 interactive
compute-9-3 np=12 parallel chassis9 interactive
compute-9-4 np=12 parallel chassis9 interactive
compute-9-5 np=12 parallel chassis9 interactive
compute-9-6 np=12 parallel chassis9 interactive
compute-9-7 np=12 parallel chassis9 interactive
compute-9-8 np=12 parallel chassis9 interactive
compute-9-9 np=12 parallel chassis9 interactive
compute-9-10 np=12 parallel chassis9 interactive
compute-9-11 np=12 parallel chassis9 interactive
compute-9-12 np=12 parallel chassis9 interactive
compute-9-13 np=12 parallel chassis9 interactive
compute-9-14 np=12 parallel chassis9 interactive
compute-9-15 np=12 parallel chassis9 interactive
compute-9-16 np=12 parallel chassis9 interactive
compute-9-17 np=12 parallel chassis9 interactive
compute-9-18 np=12 parallel chassis9 interactive
compute-9-19 np=12 parallel chassis9 interactive
compute-9-20 np=12 parallel chassis9 interactive
compute-9-21 np=12 parallel chassis9 interactive
compute-9-22 np=12 parallel chassis9 interactive
compute-9-23 np=12 parallel chassis9 interactive
compute-9-24 np=12 parallel chassis9 interactive
compute-9-25 np=12 parallel chassis9 interactive
compute-9-26 np=12 parallel chassis9 interactive
compute-9-27 np=12 parallel chassis9 interactive
compute-9-28 np=12 parallel chassis9 interactive
compute-9-29 np=12 parallel chassis9 interactive
compute-9-30 np=12 parallel chassis9 interactive
compute-9-31 np=12 parallel chassis9 interactive
compute-9-32 np=12 parallel chassis9 interactive
compute-10-1 np=12 parallel chassis10 interactive
compute-10-2 np=12 parallel chassis10 interactive
compute-10-3 np=12 parallel chassis10 interactive
compute-10-4 np=12 parallel chassis10 interactive
compute-10-5 np=12 parallel chassis10 interactive
compute-10-6 np=12 parallel chassis10 interactive
compute-10-7 np=12 parallel chassis10 interactive
compute-10-8 np=12 parallel chassis10 interactive
compute-10-9 np=12 parallel chassis10 interactive
compute-10-10 np=12 parallel chassis10 interactive
compute-10-11 np=12 parallel chassis10 interactive
compute-10-12 np=12 parallel chassis10 interactive
compute-10-13 np=12 parallel chassis10 interactive
compute-10-14 np=12 parallel chassis10 interactive
compute-10-15 np=12 parallel chassis10 interactive
compute-10-16 np=12 parallel chassis10 interactive
compute-10-17 np=12 parallel chassis10 interactive
compute-10-18 np=12 parallel chassis10 interactive
compute-10-19 np=12 parallel chassis10 interactive
compute-10-20 np=12 parallel chassis10 interactive
compute-10-21 np=12 parallel chassis10 interactive
compute-10-22 np=12 parallel chassis10 interactive
compute-10-23 np=12 parallel chassis10 interactive
compute-10-24 np=12 parallel chassis10 interactive
compute-10-25 np=12 parallel chassis10 interactive
compute-10-26 np=12 parallel chassis10 interactive
compute-10-27 np=12 parallel chassis10 interactive
compute-10-28 np=12 parallel chassis10 interactive
compute-10-29 np=12 parallel chassis10 interactive
compute-10-30 np=12 parallel chassis10 interactive
compute-10-31 np=12 parallel chassis10 interactive
compute-10-32 np=12 parallel chassis10 interactive
compute-11-1 np=12 parallel chassis11 interactive
compute-11-2 np=12 parallel chassis11 interactive
compute-11-3 np=12 parallel chassis11 interactive
compute-11-4 np=12 parallel chassis11 interactive
compute-11-5 np=12 parallel chassis11 interactive
compute-11-6 np=12 parallel chassis11 interactive
compute-11-7 np=12 parallel chassis11 interactive
compute-11-8 np=12 parallel chassis11 interactive
compute-11-9 np=12 parallel chassis11 interactive
compute-11-10 np=12 parallel chassis11 interactive
compute-11-11 np=12 parallel chassis11 interactive
compute-11-12 np=12 parallel chassis11 interactive
compute-11-13 np=12 parallel chassis11 interactive
compute-11-14 np=12 parallel chassis11 interactive
compute-11-15 np=12 parallel chassis11 interactive
compute-11-16 np=12 parallel chassis11 interactive
compute-11-17 np=12 parallel chassis11 interactive
compute-11-18 np=12 parallel chassis11 interactive
compute-11-19 np=12 parallel chassis11 interactive
compute-11-20 np=12 parallel chassis11 interactive
compute-11-21 np=12 parallel chassis11 interactive
compute-11-22 np=12 parallel chassis11 interactive
compute-11-23 np=12 parallel chassis11 interactive
compute-11-24 np=12 parallel chassis11 interactive
compute-11-25 np=12 parallel chassis11 interactive
compute-11-26 np=12 parallel chassis11 interactive
compute-11-27 np=12 parallel chassis11 interactive
compute-11-28 np=12 parallel chassis11 interactive
compute-11-29 np=12 parallel chassis11 interactive
compute-11-30 np=12 parallel chassis11 interactive
compute-11-31 np=12 parallel chassis11 interactive
compute-11-32 np=12 parallel chassis11 interactive
compute-12-1 np=12 parallel chassis12 interactive
compute-12-2 np=12 parallel chassis12 interactive
compute-12-3 np=12 parallel chassis12 interactive
compute-12-4 np=12 parallel chassis12 interactive
compute-12-5 np=12 parallel chassis12 interactive
compute-12-6 np=12 parallel chassis12 interactive
compute-12-7 np=12 parallel chassis12 interactive
compute-12-8 np=12 parallel chassis12 interactive
compute-12-9 np=12 parallel chassis12 interactive
compute-12-10 np=12 parallel chassis12 interactive
compute-12-11 np=12 parallel chassis12 interactive
compute-12-12 np=12 parallel chassis12 interactive
compute-12-13 np=12 parallel chassis12 interactive
compute-12-14 np=12 parallel chassis12 interactive
compute-12-15 np=12 parallel chassis12 interactive
compute-12-16 np=12 parallel chassis12 interactive
compute-12-17 np=12 parallel chassis12 interactive
compute-12-18 np=12 parallel chassis12 interactive
compute-12-19 np=12 parallel chassis12 interactive
compute-12-20 np=12 parallel chassis12 interactive
compute-12-21 np=12 parallel chassis12 interactive
compute-12-22 np=12 parallel chassis12 interactive
compute-12-23 np=12 parallel chassis12 interactive
compute-12-24 np=12 parallel chassis12 interactive
compute-12-25 np=12 parallel chassis12 interactive
compute-12-26 np=12 parallel chassis12 interactive
compute-12-27 np=12 parallel chassis12 interactive
compute-12-28 np=12 parallel chassis12 interactive
compute-12-29 np=12 parallel chassis12 interactive
compute-12-30 np=12 parallel chassis12 interactive
compute-12-31 np=12 parallel chassis12 interactive
compute-12-32 np=12 parallel chassis12 interactive
compute-13-1 np=12 parallel chassis13 interactive
compute-13-2 np=12 parallel chassis13 interactive
compute-13-3 np=12 parallel chassis13 interactive
compute-13-4 np=12 parallel chassis13 interactive
compute-13-5 np=12 parallel chassis13 interactive
compute-13-6 np=12 parallel chassis13 interactive
compute-13-7 np=12 parallel chassis13 interactive
compute-13-8 np=12 parallel chassis13 interactive
compute-13-9 np=12 parallel chassis13 interactive
compute-13-10 np=12 parallel chassis13 interactive
compute-13-11 np=12 parallel chassis13 interactive
compute-13-12 np=12 parallel chassis13 interactive
compute-13-13 np=12 parallel chassis13 interactive
compute-13-14 np=12 parallel chassis13 interactive
compute-13-15 np=12 parallel chassis13 interactive
compute-13-16 np=12 parallel chassis13 interactive
compute-13-17 np=12 parallel chassis13 interactive
compute-13-18 np=12 parallel chassis13 interactive
compute-13-19 np=12 parallel chassis13 interactive
compute-13-20 np=12 parallel chassis13 interactive
compute-13-21 np=12 parallel chassis13 interactive
compute-13-22 np=12 parallel chassis13 interactive
compute-13-23 np=12 parallel chassis13 interactive
compute-13-24 np=12 parallel chassis13 interactive
compute-13-25 np=12 parallel chassis13 interactive
compute-13-26 np=12 parallel chassis13 interactive
compute-13-27 np=12 parallel chassis13 interactive
compute-13-28 np=12 parallel chassis13 interactive
compute-13-29 np=12 parallel chassis13 interactive
compute-13-30 np=12 parallel chassis13 interactive
compute-13-31 np=12 parallel chassis13 interactive
compute-13-32 np=12 parallel chassis13 interactive
compute-14-1 np=12 parallel chassis14 interactive
compute-14-2 np=12 parallel chassis14 interactive
compute-14-3 np=12 parallel chassis14 interactive
compute-14-4 np=12 parallel chassis14 interactive
compute-14-5 np=12 parallel chassis14 interactive
compute-14-6 np=12 parallel chassis14 interactive
compute-14-7 np=12 parallel chassis14 interactive
compute-14-8 np=12 parallel chassis14 interactive
compute-14-9 np=12 parallel chassis14 interactive
compute-14-10 np=12 parallel chassis14 interactive
compute-14-11 np=12 parallel chassis14 interactive
compute-14-12 np=12 parallel chassis14 interactive
compute-14-13 np=12 parallel chassis14 interactive
compute-14-14 np=12 parallel chassis14 interactive
compute-14-15 np=12 parallel chassis14 interactive
compute-14-16 np=12 parallel chassis14 interactive
compute-14-17 np=12 parallel chassis14 interactive
compute-14-18 np=12 parallel chassis14 interactive
compute-14-19 np=12 parallel chassis14 interactive
compute-14-20 np=12 parallel chassis14 interactive
compute-14-21 np=12 parallel chassis14 interactive
compute-14-22 np=12 parallel chassis14 interactive
compute-14-23 np=12 parallel chassis14 interactive
compute-14-24 np=12 parallel chassis14 interactive
compute-14-25 np=12 parallel chassis14 interactive
compute-14-26 np=12 parallel chassis14 interactive
compute-14-27 np=12 parallel chassis14 interactive
compute-14-28 np=12 parallel chassis14 interactive
compute-14-29 np=12 parallel chassis14 interactive
compute-14-30 np=12 parallel chassis14 interactive
compute-14-31 np=12 parallel chassis14 interactive
compute-14-32 np=12 parallel chassis14 interactive
compute-15-1 np=12 parallel chassis15 interactive
compute-15-2 np=12 parallel chassis15 interactive
compute-15-3 np=12 parallel chassis15 interactive
compute-15-4 np=12 parallel chassis15 interactive
compute-15-5 np=12 parallel chassis15 interactive
compute-15-6 np=12 parallel chassis15 interactive
compute-15-7 np=12 parallel chassis15 interactive
compute-15-8 np=12 parallel chassis15 interactive
compute-15-9 np=12 parallel chassis15 interactive
compute-15-10 np=12 parallel chassis15 interactive
compute-15-11 np=12 parallel chassis15 interactive
compute-15-12 np=12 parallel chassis15 interactive
compute-15-13 np=12 parallel chassis15 interactive
compute-15-14 np=12 parallel chassis15 interactive
compute-15-15 np=12 parallel chassis15 interactive
compute-15-16 np=12 parallel chassis15 interactive
compute-15-17 np=12 parallel chassis15 interactive
compute-15-18 np=12 parallel chassis15 interactive
compute-15-19 np=12 parallel chassis15 interactive
compute-15-20 np=12 parallel chassis15 interactive
compute-15-21 np=12 parallel chassis15 interactive
compute-15-22 np=12 parallel chassis15 interactive
compute-15-23 np=12 parallel chassis15 interactive
compute-15-24 np=12 parallel chassis15 interactive
compute-15-25 np=12 parallel chassis15 interactive
compute-15-26 np=12 parallel chassis15 interactive
compute-15-27 np=12 parallel chassis15 interactive
compute-15-28 np=12 parallel chassis15 interactive
compute-15-29 np=12 parallel chassis15 interactive
compute-15-30 np=12 parallel chassis15 interactive
compute-15-31 np=12 parallel chassis15 interactive
compute-15-32 np=12 parallel chassis15 interactive
compute-16-1 np=12 parallel chassis16 interactive
compute-16-2 np=12 parallel chassis16 interactive
compute-16-3 np=12 parallel chassis16 interactive
compute-16-4 np=12 parallel chassis16 interactive
compute-16-5 np=12 parallel chassis16 interactive
compute-16-6 np=12 parallel chassis16 interactive
compute-16-7 np=12 parallel chassis16 interactive
compute-16-8 np=12 parallel chassis16 interactive
compute-16-9 np=12 parallel chassis16 interactive
compute-16-10 np=12 parallel chassis16 interactive
compute-16-11 np=12 parallel chassis16 interactive
compute-16-12 np=12 parallel chassis16 interactive
compute-16-13 np=12 parallel chassis16 interactive
compute-16-14 np=12 parallel chassis16 interactive
compute-16-15 np=12 parallel chassis16 interactive
compute-16-16 np=12 parallel chassis16 interactive
compute-16-17 np=12 parallel chassis16 interactive
compute-16-18 np=12 parallel chassis16 interactive
compute-16-19 np=12 parallel chassis16 interactive
compute-16-20 np=12 parallel chassis16 interactive
compute-16-21 np=12 parallel chassis16 interactive
compute-16-22 np=12 parallel chassis16 interactive
compute-16-23 np=12 parallel chassis16 interactive
compute-16-24 np=12 parallel chassis16 interactive
compute-16-25 np=12 parallel chassis16 interactive
compute-16-26 np=12 parallel chassis16 interactive
compute-16-27 np=12 parallel chassis16 interactive
compute-16-28 np=12 parallel chassis16 interactive
compute-16-29 np=12 parallel chassis16 interactive
compute-16-30 np=12 parallel chassis16 interactive
compute-16-31 np=12 parallel chassis16 interactive
compute-16-32 np=12 parallel chassis16 interactive
compute-17-1 np=12 fat chassis17 interactive
compute-17-2 np=12 fat chassis17 interactive
compute-17-3 np=12 fat chassis17 interactive
compute-17-4 np=12 fat chassis17 interactive
compute-17-5 np=12 fat chassis17 interactive
compute-17-6 np=12 fat chassis17 interactive
compute-17-7 np=12 fat chassis17 interactive
compute-17-8 np=12 fat chassis17 interactive
gpu-18-1 np=12 gpus=1 gpu cuda chassis18
gpu-18-2 np=12 gpus=1 gpu cuda chassis18
gpu-18-3 np=12 gpus=1 gpu cuda chassis18
gpu-18-4 np=12 gpus=1 gpu cuda chassis18
gpu-18-5 np=12 gpus=1 gpu cuda chassis18
gpu-18-6 np=12 gpus=1 gpu cuda chassis18
gpu-18-7 np=12 gpus=1 gpu cuda chassis18
gpu-18-8 np=12 gpus=1 gpu cuda chassis18
gpu-18-9 np=12 gpus=1 gpu cuda chassis18
gpu-18-10 np=12 gpus=1 gpu cuda chassis18
gpu-18-11 np=12 gpus=1 gpu cuda chassis18
gpu-18-12 np=12 gpus=1 gpu cuda chassis18
gpu-18-13 np=12 gpus=1 gpu cuda chassis18
gpu-18-14 np=12 gpus=1 gpu cuda chassis18
gpu-18-15 np=12 gpus=1 gpu cuda chassis18
gpu-18-16 np=12 gpus=1 gpu cuda chassis18
login-0-1
login-0-2
login-0-3
login-0-4
compute-19-1 np=32 fat superfat
[root at hpc0 ~]#

This is what we have in our moab.cfg to force parallel jobs to fit on a same chassis if possible:

################################################################################

NODESETPOLICY ONEOF      # tells Moab to allocate nodes within a single attribute set.
NODESETATTRIBUTE FEATURE # Base node set boundaries on node's node feature attribute. Features are the same ones set in Torque (/opt/torque/server_priv/nodes) 
NODESETISOPTIONAL FALSE # will force the job to always run in a complete nodeset regardless of any start delay this imposes.
NODESETLIST serial,interactive,gpu,chassis3,chassis4,chassis5,chassis6,chassis7,chassis8,chassis9,chassis10,chassis11,chassis12,chassis13,chassis14,chassis15,chassis16,parallel,fat,superfat,chassis17,chassis18 # tells Moab to only use node sets based on the listed feature values. Moab attempts to fit jobs on node sets in the order they are specified in the NODESETLIST. These are nested nodesets from smallest to largest.
NODESETPRIORITYTYPE BESTFIT # When resources are available in more than one resource set, the NODESETPRIORITYTYPE parameter allows control over how the best resource set is selected. BESTFIT selects the smallest resource set possible. Minimizes fragmentation of larger resource sets.

################################################################################

Finally, please find the submit.sh as an attachment. I have changed just one line as shown above.


To employ it on the login nodes, it should be like before. Modifying this line helps only for debugging.

Hope this helps. Actually, another simple way of testing real environment is by installing pbs server and pbs_mom on some linux host/laptop and turning off pbs_mom. With the torque.cfg pointing to submit.sh or putting submit.sh in /usr/local/bin ( I think) would work as in real environment.

Thanks to you I looked into it and now it also helps me a lot to have it tested on my mac it self. Don't know why but never occurred to me to test it on it before.

Please let me know if you have any questions. 

Sreedhar.
On Sep 27, 2013, at 10:51 AM, Jeffrey Lang <jrlang at uwyo.edu> wrote:

> I need to build a "qsub" wrapper/filter script to verify that the user includes certain parameters as part of there job.
> 
> Does anyone have a wrapper/filter that they could share that I could use as a starting point for creating one for out system?
> 
> 
> <jrlang.vcf>_______________________________________________
> torqueusers mailing list
> torqueusers at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torqueusers

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/torqueusers/attachments/20130930/ee702e5b/attachment-0002.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: submit.sh
Type: application/octet-stream
Size: 29566 bytes
Desc: not available
Url : http://www.supercluster.org/pipermail/torqueusers/attachments/20130930/ee702e5b/attachment-0001.obj 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/torqueusers/attachments/20130930/ee702e5b/attachment-0003.html 


More information about the torqueusers mailing list