| TORQUE Resource Manager |
1.3 Advanced Configuration
1.3.1 Customizing the Install
The TORQUE configure command has several options available. Listed below are some suggested options to use when running ./configure.
- By default, TORQUE does not install the admin manuals. To enable this, use --enable-docs
- By default, TORQUE does not enable syslog usage. To enable this, use --enable-syslog
Full Configure Options List
Optional features:| Option | Description | ||
|---|---|---|---|
| --disable-clients | directs torque not to build and install the TORQUE client utilities such as qsub, qstat, qdel, etc. | ||
| --disable-dependency-tracking | directs TORQUE build system to only rebuild changed source files and not rebuild dependent files. | ||
| --disable-FEATURE | do not include FEATURE (same as --enable-FEATURE=no). | ||
| --disable-gcc-warnings | Disable gcc strictness and warnings. If using gcc, default is to error on any warning. | ||
| --disable-gui | do not include the GUI-clients. | ||
| --disable-libtool-lock | avoid locking (might break parallel builds). | ||
| --disable-mom | do not include the mom daemon. | ||
| --disable-mom-checkspool | Don't check free space on spool directory and set an error. | ||
| --disable-posixmemlock | disable the moms use of mlockall. Some versions of OSs seem to have buggy POSIX MEMLOCK. | ||
| --disable-privports | disable the use of privileged ports for authentication. Some versions of OSX have a buggy bind() and cannot bind to privileged ports. | ||
| --disable-qsub-keep-override | do not allow the qsub -k flag to override -o -e. | ||
| --disable-rpp | By default, TORQUE uses RPP/UDP for resource queries from the PBS server to the MOMs. If disabled, TCP is used. This does not affect general node/job status messages, job launch/exit messages, inter-mom messages, etc. | ||
| --disable-server | do not include server and scheduler. | ||
| --disable-shell-pipe | give the job script file as standard input to the shell instead of passing its name via a pipe. | ||
| --disable-spool | if disabled, TORQUE will create output and error files directly in $HOME/.pbs_spool if it exists or in $HOME otherwise. By default, TORQUE will spool files in $TORQUEHOME/spool and copy them to the users home directory when the job completes. | ||
| --disable-xopen-networking | With HPUX and GCC, don't force usage of XOPEN and libxnet. | ||
| --enable-acct-x | enable adding x attributes to accounting log. | ||
| --enable-array | setting this under IRIX enables the SGI Origin 2000 parallel support. Normally autodetected from the /etc/config/array file. | ||
| --enable-autorun | allows jobs to run automatically as soon as they are queued if resources are available (available in TORQUE 2.3.1 and later). | ||
| --enable-blcr | enable BLCR support. | ||
| --enable-cpa | enable Cray's CPA support. | ||
| --enable-cpuset | enable Linux 2.6 kernel cpusets (in development). | ||
| --enable-debug | turn on the compilation of DEBUG code. | ||
| --enable-dependency-tracking | do not reject slow dependency extractors. | ||
| --enable-drmaa | build the DRMAA 1.0 library. | ||
| --enable-fast-install[=PKGS] | optimize for fast installation [default=yes]. | ||
| --enable-FEATURE[=ARG] | include FEATURE [ARG=yes]. | ||
| --enable-filesync | Open files with sync on each write operation. This has a negative impact on TORQUE performance. This is disabled by default. | ||
| --enable-force-nodefile | forces creation of nodefile regardless of job submission parameters. Not on by default. | ||
| --enable-geometry-requests | TORQUE is compiled to use procs_bitmap during job submission. | ||
| --enable-high-availability | this is for the autoconf utility and tells autoconf to enable so called rebuild rules. See maintainer mode for more information. | ||
| --enable-maintainer-mode | enable make rules and dependencies not useful and sometimes confusing to the casual installer. | ||
| --enable-maxdefault | turn on the RESOURCEMAXDEFAULT flag.
|
||
| --enable-nochildsignal | turn on the NO_SIGCHLD flag. | ||
| --enable-nodemask | enable nodemask-based scheduling on the Origin 2000. | ||
| --enable-pemask | enable pemask-based scheduling on the Cray T3e. | ||
| --enable-plock-daemons[=ARG] | enable daemons to lock themselves into memory: logical-or of 1 for pbs_server, 2 for pbs_scheduler, 4 for pbs_mom (no argument means 7 for all three). | ||
| --enable-quickcommit | turn on the QUICKCOMMIT flag. | ||
| --enable-shared[=PKGS] | build shared libraries [default=yes]. | ||
| --enable-shell-use-argv | enable this to put the job script name on the command line that invokes the shell. Not on by default. Ignores --enable-shell-pipe setting. | ||
| --enable-sp2 | build PBS for an IBM SP2. | ||
| --enable-srfs | enable support for SRFS on Cray. | ||
| --enable-static[=PKGS] | build static libraries [default=yes]. | ||
| --enable-syslog | enable (default) the use of syslog for error reporting. | ||
| --enable-tcl-qstat | setting this builds qstat with Tcl interpreter features. This is enabled if Tcl is enabled. | ||
| --enable-unixsockets | enable the use of Unix Domain sockets for authentication. |
Optional Packages:
| Option | Description |
|---|---|
| --with-cpa-include=DIR | include path for cpalib.h. |
| --with-cpa-lib=DIR | lib path for libcpalib. |
| --with-debug | compile with debugging symbols. |
| --with-default-server=HOSTNAME | set the name of the computer that clients will access when no machine name is specified as part of the queue name. It defaults to the hostname of the machine on which PBS is being compiled. |
| --with-environ=PATH | set the path containing the environment variables for the daemons. For SP2 and AIX systems, suggested setting is to /etc/environment. Defaults to the file "pbs_environment" in the server-home. Relative paths are interpreted within the context of the server-home. |
| --with-gnu-ld | assume the C compiler uses GNU ld [default=no]. |
| --with-maildomain=MAILDOMAIN | override the default domain for outgoing mail messages, i.e. "user@maildomain". The default maildomain is the hostname where the job was submitted from. |
| --with-modulefiles[=DIR] | use module files in specified directory [/etc/modulefiles]. |
| --with-momlogdir | use this directory for MOM logs. |
| --with-momlogsuffix | use this suffix for MOM logs. |
| --without-PACKAGE | do not use PACKAGE (same as --with-PACKAGE=no). |
| --without-readline | do not include readline support (default: included if found). |
| --with-PACKAGE[=ARG] | use PACKAGE [ARG=yes]. |
| --with-pam=DIR | Directory that holds the system PAM modules. Defaults to /lib(64)/security on Linux. |
| --with-pic | try to use only PIC/non-PIC objects [default=use both]. |
| --with-qstatrc-file=FILE | set the name of the file that qstat will use if there is no ".qstatrc" file in the directory where it is being invoked. Relative path names will be evaluated relative to the server home directory (see above). If this option is not specified, the default name for this file will be set to "qstatrc" (no dot) in the server home directory. |
| --with-rcp | one of "scp", "rcp", "mom_rcp", or the fullpath of a remote file copy program. scp is the default if found, otherwise mom_rcp is used. Some rcp programs don't always exit with valid error codes in case of failure. mom_rcp is a copy of BSD rcp included with this source that has correct error codes, but it is also old, unmaintained, and doesn't have largefile support. |
| --with-sched=TYPE | sets the scheduler type. If TYPE is "c", the scheduler will be written in C. If TYPE is "tcl" the server will use a Tcl based scheduler. If TYPE is "basl", TORQUE will use the rule based scheduler. If TYPE is "no", then no scheduling is done. "c" is the default. |
| --with-sched-code=PATH | sets the name of the scheduler to use. This only applies to BASL schedulers and those written in the C language. For C schedulers this should be a directory name and for BASL schedulers a filename ending in ".basl". It will be interpreted relative to srctree/src/schedulers.SCHD_TYPE/samples. As an example, an appropriate BASL scheduler realtive path would be "nas.basl". The default scheduler code for "C" schedulers is "fifo". |
| --with-scp | In TORQUE 2.1 and later, SCP is the default remote copy protocol. See --with-rcp if a different protocol is desired. |
| --with-sendmail[=FILE] | sendmail executable to use. |
| --with-server-home=DIR | set the server home/spool directory for PBS use. Defaults to /var/spool/torque. |
| --with-server-name-file=FILE | set the file that will contain the name of the default server for clients to use. If this is not an absolute pathname, it will be evaluated relative to the server home directory that either defaults to /usr/spool/PBS or is set using the --with-server-home option to configure. If this option is not specified, the default name for this file will be set to "server_name". |
| --with-tags[=TAGS] | include additional configurations [automatic]. |
| --with-tcl | directory containing tcl configuration (tclConfig.sh). |
| --with-tclatrsep=CHAR | set the Tcl attribute separator character this will default to "." if unspecified. |
| --with-tclinclude | directory containing the public Tcl header files. |
| --with-tclx | directory containing tclx configuration (tclxConfig.sh). |
| --with-tk | directory containing tk configuration (tkConfig.sh). |
| --with-tkinclude | directory containing the public Tk header files. |
| --with-tkx | directory containing tkx configuration (tkxConfig.sh). |
| --with-tmpdir=DIR | set the tmp directory that pbs_mom will use. Defaults to "/tmp". This is a Cray-specific feature. |
| --with-xauth=PATH | specify path to xauth program. |
1.3.2 Server Configuration
1.3.2.1 Server Configuration Overview
There are several steps to ensure that the server and the nodes are completely aware of each other and able to communicate directly. Some of this configuration takes place within TORQUE directly using the qmgr command. Other configuration settings are managed using the pbs_server nodes file, DNS files such as /etc/hosts and the /etc/hosts.equiv file.
1.3.2.2 Name Service Configuration
Each node, as well as the server, must be able to resolve the name of every node with which it will interact. This can be accomplished using /etc/hosts, DNS, NIS, or other mechanisms. In the case of /etc/hosts, the file can be shared across systems in most cases.
A simple method of checking proper name service configuration is to verify that the server and the nodes can "ping" each other.
1.3.2.3 Configuring Job Submission Hosts
When jobs can be submitted from several different hosts, these hosts should be trusted via the R* commands (such as rsh and rcp). This can be enabled by adding the hosts to the /etc/hosts.equiv file of the machine executing the pbs_server daemon or using other R* command authorization methods. The exact specification can vary from OS to OS (see the man page for ruserok to find out how your OS validates remote users). In most cases, configuring this file is as simple as adding a line to your /etc/hosts.equiv file, as in the following:
/etc/hosts.equiv:#[+ | -] [hostname] [username] mynode.myorganization.com .....
Please note that when a hostname is specified, it must be the fully qualified domain name (FQDN) of the host. Job submission can be further secured using the server or queue acl_hosts and acl_host_enabled parameters.
Using the submit_hosts Server Parameter
Trusted submit host access may be directly specified without using RCmd authentication by setting the server submit_hosts parameter via qmgr as in the following example:
> qmgr -c 'set server submit_hosts = login1' > qmgr -c 'set server submit_hosts += login2' > qmgr -c 'set server submit_hosts += login3'
 |
Use of submit_hosts is potentially subject to DNS spoofing and should not be used outside of controlled and trusted environments. |
Allowing Job Submission from Compute Hosts
If preferred, all compute nodes can be enabled as job submit hosts without setting .rhosts or hosts.equiv by setting the allow_node_submit parameter to true.
1.3.2.4 Configuring TORQUE on a Multi-Homed Server
If the pbs_server daemon is to be run on a multi-homed host (a host possessing multiple network interfaces), the interface to be used can be explicitly set using the SERVERHOST parameter.
1.3.2.5 Architecture Specific Notes
1.3.2.5.1 Mac OS/X Specific Notes
With some versions of Mac OS/X, it is required to add the line $restricted *.<DOMAIN> to the pbs_mom configuration file. This is required to work around some socket bind bugs in the OS.
1.3.2.6 Specifying Non-Root Administrators
By default, only root is allowed to start, configure and manage the pbs_server daemon. Additional trusted users can be authorized using the parameters managers and operators. To configure these parameters use the qmgr command, as in the following example:
> qmgr Qmgr: set server managers += josh@*.fsc.com Qmgr: set server operators += josh@*.fsc.com
All manager and operator specifications must include a user name and either a fully qualified domain name or a host expression.
|
To enable all users to be trusted as both operators and administrators, place the + (plus) character on its own line in the server_priv/acl_svr/operators and server_priv/acl_svr/managers files. |
1.3.2.7 Setting Up E-mail
Moab relies on e-mails from TORQUE about job events. To set up e-mail, do the following:
- Use the --with-sendmail configure option at configure time. TORQUE needs to know where the email application is. If this option is not used, TORQUE tries to find the sendmail executable. If it isn't found, TORQUE cannot send e-mails.
> ./configure --with-sendmail=<path_to_executable>
- Set mail_domain in your server settings. If your domain is clusterresources.com, execute:
> qmgr -c 'set server mail_domain=clusterresources.com'
- (Optional) You can override the default mail_body_fmt and mail_subject_fmt values via qmgr:
> qmgr -c 'set server mail_body_fmt=Job: %i \n Name: %j \n On host: %h \n \n %m \n \n %d' > qmgr -c 'set server mail_subject_fmt=Job %i - %r'
By default, users receive e-mails on job aborts. Each user can select which kind of e-mails to receive by using the qsub -m option when submitting the job. If you want to dictate when each user should receive e-mails, use a submit filter.
See Also