[torqueusers] local scratch space allocation

Steve Traylen s.traylen at rl.ac.uk
Fri Oct 21 03:09:30 MDT 2005


On Thu, Oct 20, 2005 at 04:26:18PM -0700 or thereabouts, Garrick Staples wrote:
> On Mon, Aug 08, 2005 at 05:35:17PM -0600, Dave Jackson alleged:
> > Steve,
> > 
> >   This patch looks broadly useful.  TORQUE 1.2.0patch 6 is already fully
> > populated with a full slate of enhancements (thanks to USC and several
> > other sites!) and is in testing.  We have included this patch in the
> > 'contrib' directory and will get it rolled into the patch 7 release.
> 
> I suddenly have need for this patch, but it seems that it never got
> rolled in.  Unfortunately it doesn't apply anymore.
> 
> I'd to revamp it for current TORQUE with one minor change:  as written,
> only MS will make the tmpdir, instead I'd like it to check if the
> filesystem is local or remote.  If it is remote, then only MS will make
> it, if it is local, all MOMs will make it.
> 
> "remote" will be defined as any filesystem with a ":" in the name and
> the hostname isn't the local hostname, or the fstype is "smbfs", the
> device starts with "//", and the hostname portion isn't the local
> hostname.
> 
> Does that sound reasonable to everyone?

Hi Garrick,

 Attached is a patch that builds against 1.2.0p6.

 Steve
> 
> -- 
> Garrick Staples, Linux/HPCC Administrator
> University of Southern California



> _______________________________________________
> torqueusers mailing list
> torqueusers at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torqueusers


-- 
Steve Traylen
s.traylen at rl.ac.uk
http://www.gridpp.ac.uk/
-------------- next part --------------
diff -urN torque-1.2.0p3/README.tmpdir torque-1.2.0p3-tmpdir/README.tmpdir
--- torque-1.2.0p3/README.tmpdir	1970-01-01 02:00:00.000000000 +0200
+++ torque-1.2.0p3-tmpdir/README.tmpdir	2005-05-18 11:12:23.000000000 +0300
@@ -0,0 +1,190 @@
+
+Readme.tmpdir - Enabling transient temporary directories for jobs
+------------------------------------------------------------------------------
+
+The transient-tmpdir patch enables the per-job TMPDIR functionality 
+such as found in the CRAY systems and in PBS Pro version 5.2 and
+above. This document described the way to enable this patch and how
+to configure the behaviour of TMPDIR using the pbs_mom configuration
+file.
+Although the functionality is the same, the way this patch works and
+is configured may be different from PBS Pro or CRAY.
+
+
+Compilation
+-----------
+
+The transient-tmpdir patch is enabled by defining the TRANSIENT_TMPDIR
+symbol when compiling pbs-mom (in src/resmom). When using 
+autoconf, define CFLAGS as follows
+
+CFLAGS="$CFLAGS -DTRANSIENT_TMPDIR=1"
+export CFLAGS
+
+./configure --set-tmpdir=<tmpdir_default> [ other configure options ]
+
+and run "make" and "make install". The default for tmpdir is defined
+by configure, and set to "/tmp" unless spoecified otherwise.
+
+
+How it works
+------------
+
+Transient job directories are only created when a non-null value
+is set for TMP_DIR (either using configure or by using the
+"$tmpdir <path>" directive in mom's config file), and the TMPDIR
+environment variable is not set in the pbs_environment file.
+
+Before the job prologue script (or the job itself) is started, 
+a temporary, per-job, directory name is generated based on the
+value of TMP_DIR (as set in configure or using the $tmpdir directive
+in Mom's configuration file) and the job ID. This name is
+passed to the job in the $TMPDIR environment variable.
+
+The Mother-Superior (MS) responsible for the job will actually
+create the directory; other mom's will only define the value for the
+job and refrain from creating it, although they will ensure that 
+TMPDIR is pointing to a directory and not a (special) file.
+
+This directory is created before the prologue script is run, and
+TMPDIR will be available to the prologue script as well.
+
+When the job finished (the job is sent to the Obituary), this
+directory is removed by Mother Superior after the epilogue script
+has been run and after the PBS_NODEFILE has been removed. 
+
+
+Disabling the transient-tmpdir functionality
+--------------------------------------------
+
+If your copy of (Scalable) OpenPBS has been built with transient-tmpdir
+functionality, but you would like to disable the creation of transient
+directories, you can do so by explicitly setting TMPDIR in
+the pbs_environment file ("TMPDIR=/tmp"). If TMPDIR is set explicitly,
+no directories will be created (or removed).
+
+
+Caveats
+-------
+
+When jobs are using multiple nodes for execution, and the 
+transient-tmpdir functionality is used, the basename for the temporary 
+directories (TMP_DIR or the $tmpdir directive in the mom config file) 
+must be the same for all mom's, and must be pointing to a file system 
+shared by all nodes.
+
+
+Changes
+-------
+
+The transient-tmpdir patch only affects the following files
+
+  src/resmom/mom_main.c
+  src/resmom/start_exec.c
+  src/resmom/catch_child.c
+
+
+Acknowledgements
+----------------
+
+This patch has been provided by the Scientific Computing project 
+(DataGrid Team) of the Dutch National Institute for Nuclear and 
+High-Energy Physics (NIKHEF), Amsterdam, The Netherlands.
+                      David Groep <davidg at nikhef.nl>, 2003-11-19.
+
+
+
+Readme.tmpdir - Enabling transient temporary directories for jobs
+------------------------------------------------------------------------------
+
+The transient-tmpdir patch enables the per-job TMPDIR functionality 
+such as found in the CRAY systems and in PBS Pro version 5.2 and
+above. This document described the way to enable this patch and how
+to configure the behaviour of TMPDIR using the pbs_mom configuration
+file.
+Although the functionality is the same, the way this patch works and
+is configured may be different from PBS Pro or CRAY.
+
+
+Compilation
+-----------
+
+The transient-tmpdir patch is enabled by defining the TRANSIENT_TMPDIR
+symbol when compiling pbs-mom (in src/resmom). When using 
+autoconf, define CFLAGS as follows
+
+CFLAGS="$CFLAGS -DTRANSIENT_TMPDIR=1"
+export CFLAGS
+
+./configure --set-tmpdir=<tmpdir_default> [ other configure options ]
+
+and run "make" and "make install". The default for tmpdir is defined
+by configure, and set to "/tmp" unless spoecified otherwise.
+
+
+How it works
+------------
+
+Transient job directories are only created when a non-null value
+is set for TMP_DIR (either using configure or by using the
+"$tmpdir <path>" directive in mom's config file), and the TMPDIR
+environment variable is not set in the pbs_environment file.
+
+Before the job prologue script (or the job itself) is started, 
+a temporary, per-job, directory name is generated based on the
+value of TMP_DIR (as set in configure or using the $tmpdir directive
+in Mom's configuration file) and the job ID. This name is
+passed to the job in the $TMPDIR environment variable.
+
+The Mother-Superior (MS) responsible for the job will actually
+create the directory; other mom's will only define the value for the
+job and refrain from creating it, although they will ensure that 
+TMPDIR is pointing to a directory and not a (special) file.
+
+This directory is created before the prologue script is run, and
+TMPDIR will be available to the prologue script as well.
+
+When the job finished (the job is sent to the Obituary), this
+directory is removed by Mother Superior after the epilogue script
+has been run and after the PBS_NODEFILE has been removed. 
+
+
+Disabling the transient-tmpdir functionality
+--------------------------------------------
+
+If your copy of (Scalable) OpenPBS has been built with transient-tmpdir
+functionality, but you would like to disable the creation of transient
+directories, you can do so by explicitly setting TMPDIR in
+the pbs_environment file ("TMPDIR=/tmp"). If TMPDIR is set explicitly,
+no directories will be created (or removed).
+
+
+Caveats
+-------
+
+When jobs are using multiple nodes for execution, and the 
+transient-tmpdir functionality is used, the basename for the temporary 
+directories (TMP_DIR or the $tmpdir directive in the mom config file) 
+must be the same for all mom's, and must be pointing to a file system 
+shared by all nodes.
+
+
+Changes
+-------
+
+The transient-tmpdir patch only affects the following files
+
+  src/resmom/mom_main.c
+  src/resmom/start_exec.c
+  src/resmom/catch_child.c
+
+
+Acknowledgements
+----------------
+
+This patch has been provided by the Scientific Computing project 
+(DataGrid Team) of the Dutch National Institute for Nuclear and 
+High-Energy Physics (NIKHEF), Amsterdam, The Netherlands.
+                      David Groep <davidg at nikhef.nl>, 2003-11-19.
+
+
diff -urN torque-1.2.0p3/src/resmom/catch_child.c torque-1.2.0p3-tmpdir/src/resmom/catch_child.c
--- torque-1.2.0p3/src/resmom/catch_child.c	2005-03-20 00:25:25.000000000 +0200
+++ torque-1.2.0p3-tmpdir/src/resmom/catch_child.c	2005-05-18 11:12:23.000000000 +0300
@@ -85,6 +85,9 @@
 #include <dirent.h>
 #include <limits.h>
 #include <assert.h>
+#ifdef TRANSIENT_TMPDIR
+#include <stdlib.h>
+#endif /* TRANSIENT_TMPDIR */
 #include <ctype.h>
 #include <sys/stat.h>
 #include "dis.h"
@@ -126,6 +129,9 @@
 extern struct connection svr_conn[];
 extern int		resc_access_perm;
 extern char		*path_home;
+#ifdef TRANSIENT_TMPDIR
+extern char		tmpdir_basename[];
+#endif /* TRANSIENT_TMPDIR */
 
 extern int              LOGLEVEL;
 
@@ -329,6 +335,9 @@
   char		*svrport;
   char		*cookie;
   unsigned int	port;
+#ifdef TRANSIENT_TMPDIR
+  char          tmpdir[MAXPATHLEN];
+#endif /* TRANSIENT_TMPDIR */
   u_long	gettime		A_((resource *pres));
   u_long	getsize		A_((resource *pres));
   task         *task_find	A_((job	*pjob,tm_task_id taskid));
@@ -763,6 +772,25 @@
       pjob->ji_flags &= ~MOM_HAS_NODEFILE;
       }
 		
+#ifdef TRANSIENT_TMPDIR
+  /* this is the set of condition for a transient tmpdir as used at
+   * creation time in start_exec.c 
+   * we already know we are Mother Superior */
+    if ( (getenv("TMPDIR") == NULL) && *tmpdir_basename ) {
+      snprintf(tmpdir, sizeof(tmpdir), "%s/%s", tmpdir_basename, pjob->ji_qs.ji_jobid);
+      sprintf(log_buffer,"Removing transient job directory %s",tmpdir);
+      log_record(PBSEVENT_DEBUG, PBS_EVENTCLASS_JOB,
+		 pjob->ji_qs.ji_jobid, log_buffer);
+      
+      if ( rrmdir(tmpdir) != 0 ) {
+	(void)sprintf(log_buffer,
+		      "recursive remove of job transient tmpdir %s failed",
+		      tmpdir);
+	log_err(errno, "recursive (r)rmdir", log_buffer);
+      }
+    }
+#endif /* TRANSIENT_TMPDIR */
+
     /* send the job obiturary notice to the server */
 
     preq = alloc_br(PBS_BATCH_JobObit);
@@ -810,6 +838,69 @@
 
 
 
+
+#ifdef TRANSIENT_TMPDIR
+/* this routine will recursively remove everything below <path> */
+int rrmdir(path)
+	char *path;
+{
+
+#define SUBPATHLENCHUNK 2048
+
+	int rc;
+	DIR *dir;
+	struct dirent *dent;
+	struct stat sb;
+	char *subpath,*p;
+	int subpathlen=0;
+
+	if ( *path == (int)'0' ) return ENOENT;
+	if ( (dir=opendir(path)) == NULL ) return errno;
+
+	if ( (subpath=malloc(SUBPATHLENCHUNK)) == NULL )
+		return ENOMEM;
+	subpathlen=SUBPATHLENCHUNK;
+
+	while ( dent=readdir(dir) ) {
+		/* save me and my parent; and myself from a loop */
+		if( *(dent->d_name)=='.' && *(1+dent->d_name)==0 ) continue;
+		if( *(dent->d_name)=='.' && *(1+dent->d_name)=='.' && *(2+dent->d_name)==0 ) continue;
+
+		if(2+strlen(path)+strlen(dent->d_name) > subpathlen )  {
+			if( (p=realloc(subpath,subpathlen+SUBPATHLENCHUNK)) == NULL ) {
+				free(subpath);
+				return ENOMEM;
+			}
+			subpath=p;
+			subpathlen+=SUBPATHLENCHUNK;
+		}
+		strcpy(subpath,path);
+		strcat(subpath,"/");
+		strcat(subpath,dent->d_name);
+
+		if ( (rc=lstat(subpath,&sb)) != 0 ) {
+			free(subpath);
+			return rc;
+		}
+
+		if(S_ISDIR(sb.st_mode)) 
+			rc=rrmdir(subpath);
+		else
+			rc=unlink(subpath);
+
+		if(rc) {
+			free(subpath);
+			return rc;
+		}
+	}
+	closedir(dir);
+	free(subpath);
+	rc=rmdir(path);
+	return rc;
+}
+#endif /* TRANSIENT_TMPDIR */
+
+
 /*
  * obit_reply - read and process the reply from the server acknowledging
  *	the job obiturary notice.
diff -urN torque-1.2.0p3/src/resmom/mom_main.c torque-1.2.0p3-tmpdir/src/resmom/mom_main.c
--- torque-1.2.0p3/src/resmom/mom_main.c	2005-04-04 20:04:19.000000000 +0300
+++ torque-1.2.0p3-tmpdir/src/resmom/mom_main.c	2005-05-18 11:12:23.000000000 +0300
@@ -230,6 +230,9 @@
 
 extern void     add_resc_def(char *,char *);
 
+#ifdef TRANSIENT_TMPDIR
+extern char	tmpdir_basename[];
+#endif /* TRANSIENT_TMPDIR */
 
 #define PMOMTCPTIMEOUT 60  /* duration in seconds mom TCP requests will block */
 
@@ -1110,6 +1113,16 @@
   }  /* END setserver() */
 
 
+#ifdef TRANSIENT_TMPDIR
+static        u_long
+settmpdir(char *name)
+{
+  static  char    id[] = "settmpdir";
+  
+  strcpy(tmpdir_basename, name);
+  return 1;
+}
+#endif /* TRANSIENT_TMPDIR */
 
 
 
@@ -1677,6 +1690,9 @@
       { "loglevel",     setloglevel },
       { "max_load",     setmaxload },
       { "prologalarm",  prologalarm },
+#ifdef TRANSIENT_TMPDIR
+      { "tmpdir",       settmpdir },
+#endif /* TRANSIENT_TMPDIR */
       { "restricted",   restricted },
       { "jobstartblocktime", jobstartblocktime },
       { "usecp",        usecp },
@@ -4880,6 +4896,10 @@
   if (gethostname(ret_string,ret_size) == 0)
     addclient(ret_string);
 
+#ifdef TRANSIENT_TMPDIR
+  settmpdir(TMP_DIR);
+#endif /* TRANSIENT_TMPDIR */
+
   if (read_config(NULL)) 
     {
     fprintf(stderr,"%s: config file '%s' failed\n",
diff -urN torque-1.2.0p3/src/resmom/start_exec.c torque-1.2.0p3-tmpdir/src/resmom/start_exec.c
--- torque-1.2.0p3/src/resmom/start_exec.c	2005-05-12 20:32:52.000000000 +0300
+++ torque-1.2.0p3-tmpdir/src/resmom/start_exec.c	2005-05-18 11:12:23.000000000 +0300
@@ -154,6 +154,10 @@
 int              mom_reader_go;		/* see catchinter() & mom_writer() */
 struct var_table vtable;		/* for building up Job's environ */
 
+#ifdef TRANSIENT_TMPDIR
+char             tmpdir_basename[MAXPATHLEN];
+#endif /* TRANSIENT_TMPDIR */
+
 /* Local Varibles */ 
 
 static int	 script_in;	/* script file, will be stdin	  */
@@ -173,6 +177,9 @@
   "PBS_NODENUM",
   "PBS_TASKNUM",
   "PBS_MOMPORT",
+#ifdef TRANSIENT_TMPDIR
+  "TMPDIR",
+#endif /* TRANSIENT_TMPDIR */
   "PBS_NODEFILE" };
 
 static	char *variables_env[NUM_LCL_ENV_VAR];
@@ -1319,7 +1326,13 @@
   job                  *pjob;
   task                 *ptask;
 
-  struct passwd        *pwdp;
+#ifdef TRANSIENT_TMPDIR
+  struct stat          sb;
+  int                  rc;
+  char                 tmpdir[MAXPATHLEN];
+#endif
+
+struct passwd        *pwdp;
 
   pjob  = (job *)TJE->pjob;
   ptask = (task *)TJE->ptask;
@@ -1453,7 +1466,11 @@
       path_home, 
       pjob->ji_qs.ji_jobid);
 
+#ifdef TRANSIENT_TMPDIR
+    bld_env_variables(&vtable,variables_else[12],buf);
+#else
     bld_env_variables(&vtable,variables_else[11],buf);
+#endif
 
     if ((nhow = fopen(buf,"w")) == NULL) 
       {
@@ -1532,6 +1549,63 @@
 
 #endif  /* (PENABLE_CPUSETS || PENABLE_DYNAMIC_CPUSETS) */
 
+#ifdef TRANSIENT_TMPDIR
+  /* only when TMPDIR is not set by someone else */
+  if ( getenv("TMPDIR") == NULL ) {
+    *buf=0;
+    snprintf(tmpdir, sizeof(tmpdir), "%s/%s", tmpdir_basename, pjob->ji_qs.ji_jobid);
+    rc=stat(tmpdir,&sb);
+    if (rc) rc=errno;
+    switch (rc) {
+    case ENOENT:
+      break;
+    case 0:
+      if( ! S_ISDIR(sb.st_mode)) {
+	sprintf(log_buffer, "Job transient tmpdir %s exists, but is not a directory", tmpdir);
+	log_err(errno, id, log_buffer);
+        starter_return(TJE->upfds,TJE->downfds,JOB_EXEC_FAIL1,&sjr);
+      }
+      break;
+    default:
+      sprintf(log_buffer, "Cannot name job tmp directory %s (on stat)", tmpdir);
+      log_err(errno, id, log_buffer);
+      starter_return(TJE->upfds,TJE->downfds,JOB_EXEC_FAIL1,&sjr);
+      break;
+    }
+    
+    /* only Mother Superior shoudl create the directory */
+    if ( pjob->ji_qs.ji_svrflags & JOB_SVFLG_HERE ) {
+      if (mkdir(tmpdir,0700) == -1 ) {
+	sprintf(log_buffer, "cannot mkdir tmpdir %s", tmpdir);
+	log_err(errno, id, log_buffer);
+        starter_return(TJE->upfds,TJE->downfds,JOB_EXEC_FAIL1,&sjr);
+      }
+      if (chown(tmpdir,pjob->ji_qs.ji_un.ji_momt.ji_exuid,pjob->ji_qs.ji_un.ji_momt.ji_exgid) == -1 ) {
+	sprintf(log_buffer, "cannot chown tmpdir %s to %d:%d", tmpdir,
+		pjob->ji_qs.ji_un.ji_momt.ji_exuid,
+		pjob->ji_qs.ji_un.ji_momt.ji_exgid);
+	log_err(errno, id, log_buffer);
+        starter_return(TJE->upfds,TJE->downfds,JOB_EXEC_FAIL1,&sjr);
+      }
+      if (chmod(tmpdir,0700) == -1 ) {
+	sprintf(log_buffer, "cannot chmod tmpdir %s", tmpdir);
+	log_err(errno, id, log_buffer);
+        starter_return(TJE->upfds,TJE->downfds,JOB_EXEC_FAIL1,&sjr);
+      }
+    }
+    
+    sprintf(log_buffer,"using transient tmpdir %s",tmpdir);
+    log_record(PBSEVENT_DEBUG, PBS_EVENTCLASS_JOB,
+	       pjob->ji_qs.ji_jobid, log_buffer);
+    
+    bld_env_variables(&vtable, variables_else[11], tmpdir);
+  } else {
+    *tmpdir=0; /* did not create a unique directory */
+    bld_env_variables(&vtable, variables_else[11], getenv("TMPDIR"));
+  }
+#endif /* TRANSIENT_TMPDIR */
+  
+
   /* specific system related variables */
 
   j = set_mach_vars(pjob,&vtable);


More information about the torqueusers mailing list