2.2 Specifying Files and Directories

2.2.1 Default Behavior
2.2.2 Job Environment Variables
2.2.3 Redirecting Output/Error Data
2.2.4 Staging Input and Output Files
2.2.5 Interactive Jobs
2.2.6 Specifying Working Directory
2.2.7 TORQUE Configuration Options

2.2.1 Default Behavior

The batch job submitted by a user will execute from within a specific directory. It will create both output and error files containing info written to stdout and stderr respectively. The directory in which the job executes will be the same as the directory from which the qsub command was executed.

2.2.2 Job Environment Variables

When a job is submitted, aspects of the calling environment are saved. When the job is actually executed, this environment may be restored (see the v and V qsub options)

Variable Description
PBS_O_HOST the name of the host where the qsub command was run
PBS_JOBID the job identifier assigned to the job by the batch system
PBS_JOBNAME the job name supplied by the user.
PBS_O_WORKDIR the absolute path of the current working directory of the qsub command

See the qsub manpage for full list of available variables.

Example

torquejob.cmd

#!/bin/bash

echo "starting $PBS_JOBNAME" > /tmp/output.$PBS_JOBID

2.2.3 Redirecting Output/Error Data

By default, job output data will be placed in the file <SUBMITDIR>/<JOBNAME>.o<SEQUENCENUMBER> and job error data will be placed in the file <SUBMITDIR>/<JOBNAME>.e<SEQUENCENUMBER> These files will be populated and available to the user once the job completes.

This behavior can be changed by specifying the '-o' or '-e' arguments on the command line or within the command script as in the following example:

torquejob.cmd

#!/bin/bash

#PBS -e /tmp/error.$PBS_O_JOBID
#PBS -o /tmp/output.$PBS_O_JOBID
...

Note If desired, the output and error streams can be joined by specifying the '-j' option.

Note By default, output and error file information will be removed from the compute hosts when the job is complete. If desired, these files can be retained by specifying the '-k' option.

See the qsub manpage for more information on the '-o', '-e', '-j' and '-k' options.

2.2.4 Staging Input and Output Files

If an input file is required by the job which is not by default available on the compute nodes, then the file can be staged in within the script or the script can request that the batch system copy the file on behalf of the user. If the latter approach is desired, the stagein option should be specified.

If one or more output files are produced by the job and must be staged to an accessible location, then the script can explicitly copy this data or can request the the batch system copy the file on behalf of the user. If the latter approach is desired, the stageout option should be specified.

FIXME from here down

torquejob.cmd

#!/bin/bash

#PBS -W stagein=dataset13442 
#PBS -W stageout=/localscratch/data1.$PBS_O_JOBID
#PBS -W stageout=/localscratch/data2.$PBS_O_JOBID
...

See the qsub manpage for more details.

2.2.5 Interactive Jobs

A job run with the interactive option will run normally, but stdout and stderr will be connected directly to the users terminal. This also allows stdin from the user to be sent directly to the application.

qsub -I

> qsub -I /tmp/script

See the interactive option in the qsub manpage for more details.

2.2.6 Specifying Working Directory

The -d qsub option is used to define the working directory path to be used for the job. If the -d option is not specified, the default working directory is the home directory. This option sets the environment variable PBS_O_INITDIR and launches the job from the specified directory.

torquejob.cmd:

#!/bin/bash

#PBS -d /tmp/working_dir
...

2.2.7 TORQUE Configuration Options

2.2.7.1 Disabling Spooling - Allowing Users to Watch Output/Error Data in Real-time

Spooling can be disabled by using the qsub '-k' option. With this option, job output and error streams can be sent directly to a file in the job's working directory bypassing the intermediate spooling step. This is useful if the job is submitted from within a parallel filesystem or if the output is small and the user would like to view it in real-time. If the output is large and the remote working directory is not available via a high performance network, excessive use of this option may result in reduced cluster performance.

Example

qsub

> qsub -l nodes=1,walltime=1:00:00 -k oe file.cmd

Disabling Spooling at a Global Level

When the --disable-spool configure option is used, TORQUE will create the output and error files directly in $HOME/.pbs_spool if it exists, or in $HOME if it does not exist. By default, TORQUE will spool files in $TORQUEHOME/spool and copy them to the users home directory when the job completes.

Note If desired for disk space or performance reasons, the default TORQUE spool directory can be changed by making $TORQUEHOME/spool a symbolic link to the desired target directory.

2.2.7.2 Creating Per-Job Temporary Directories

The tmpdir mom config option sets the directory basename for a per-job temporary directory. Before the job is launched, MOM will append the jobid to the tmpdir base-name and create the directory. After the job completes and exits, MOM will recursively delete that temporary directory. The env variable TMPDIR will be set for all pro/epilog scripts, the job script, and TM tasks.

Directory creation and removal is done as the job owner and group, so the owner must have write permission to create the directory. If the directory already exists and is owned by the job owner, it will not be deleted after the job. If the directory already exists and is NOT owned by the job owner, the job start will be rejected.

mom_priv/config:

$tmpdir /localscratch