[torqueusers] qsub environment variable limits

Garrick Staples garrick at usc.edu
Mon Jul 11 17:37:28 MDT 2005


On Mon, Jul 11, 2005 at 11:48:31AM -0700, Josh Lauricha alleged:
> Specifically, what are they? I have a user doing something along the
> lines of:
> 
>  $ find -type f -maxdepth 1 | xargs -n 250 echo | while read INPUT; do
>         export INPUT; qsub foo.sh; done
> 
> with foo.sh something like:
> #!/bin/sh
> #PBS -v INPUT
> 
> for file in $INPUT; do
>     something_to $file
> done
> 
> after around 2-4k of data, the buffer used to send the data overflows.
> It might be helpful for an error message to be printed when they are
> going to be truncated.

It is definitely not checking bounds, in fact I'm getting segfaults in qsbu
trying to set insanely large variables.  Feel free to file a bug in
http://clusterresources/bugzilla

But in all honestly, you are fighting an uphill battle here with really large
env vars.  You'd be better off putting the list of input files into the job
script.

find -type f -maxdepth 1 | xargs -n 250 echo | \
   while read INPUT; do
      sed "s|%INPUT%|$INPUT|" < foo.sh | qsub -N foo.sh
   done


-- 
Garrick Staples, Linux/HPCC Administrator
University of Southern California
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://www.supercluster.org/pipermail/torqueusers/attachments/20050711/85a3eaf4/attachment.bin


More information about the torqueusers mailing list