[torqueusers] PBS_O_WORKDIR on prologue

Troy Baer troy at osc.edu
Tue Jul 24 09:50:19 MDT 2007


On Tue, 2007-07-24 at 15:34 +0200, Arnau Bria wrote: 
> we're trying to set PBS_O_WORKDIR value in prologue.
> is this possible?
> We have tried a simple export and it did not work. 
> 

No.  The prologue runs as root and does not share any environment with
the job itself.  Also, this sort of violates the definition of
PBS_O_WORKDIR as the directory from which the job was submitted and to
which output will be delivered.

> Looking for how people sets var values in prologue did not find anythig
> well, someone who recomends setting vars in config file:
> /var/spool/pbs/mom_priv/config or just in job submission script file...
> 
> We're looking for something more dynamically: we're creating a dir
> in a scratch zone and we'd like to set PBS_O_WORKDIR
> to /scratch_zone$i/$jod_id where $i is not static and $job_id is $1 ...
> 
> Could someone help us with his experience?

You can use prologue and epilogue to set up and tear down this
directory, but you need something inside the job to set environment
variables.  We do something like this at OSC.  Our TORQUE installs are
configured with the flags --disable-shell-pipe --enable-shell-use-argv,
which causes the job scripts to act more or less like login shells.
Then we put something in the shell environment that detects if the shell
is a job and sets variables accordingly.  We set two variables:  TMPDIR,
which is a scratch directory local to each node; and PFSDIR, which is a
shared scratch directory on a parallel file system (currently PVFS).
Here's how we do that for sh/ksh/bash and csh/tcsh on our Linux
clusters:
-----
#!/bin/sh
# /etc/profile.d/tmpdir.sh
# If PBS_ENVIRONMENT exists and is "PBS_BATCH" or "PBS_INTERACTIVE",
# set TMPDIR
if [ -n "$PBS_ENVIRONMENT" ]
then
  if [ "$PBS_ENVIRONMENT" = PBS_BATCH -o \
       "$PBS_ENVIRONMENT" = PBS_INTERACTIVE ]
  then
    jobid=`echo $PBS_JOBID | sed 's/\..*//'`
    export TMPDIR=/tmp/pbstmp.$jobid
    export PFSDIR=/pvfs/pbsjobs/$jobid
    export ENVIRONMENT="BATCH"
  fi
fi
-----
#!/bin/csh
# /etc/profile.d/tmpdir.csh
# If PBS_ENVIRONMENT exists and is "PBS_BATCH" or "PBS_INTERACTIVE",
# set TMPDIR
if ( $?PBS_ENVIRONMENT ) then
  if ( "$PBS_ENVIRONMENT" == "PBS_BATCH" || \
       "$PBS_ENVIRONMENT" == "PBS_INTERACTIVE") then
    setenv jobid `echo $PBS_JOBID | sed 's/\..*//'`
    setenv TMPDIR /tmp/pbstmp.$jobid
    setenv PFSDIR /pvfs/pbsjobs/$jobid
    setenv ENVIRONMENT BATCH
  endif
endif
-----
The directories themselves are created using {pro,epi}logue and
{pro,epi}logue.parallel.

Hope this helps,
	--Troy
-- 
Troy Baer                       troy at osc.edu
Science & Technology Support    http://www.osc.edu/hpc/
Ohio Supercomputer Center       614-292-9701




More information about the torqueusers mailing list